-*- mode: org -*-
* general
** DONE libtbx_env
   A discussion about this is available on github
   https://github.com/cctbx/cctbx_project/issues/460

  - pytest load libtbx.env_config qui unpickle the build environment.
    We need to patch the code and install the pickled under /etc.

    /usr/lib/python3/dist-packages/libtbx/env_config.py:2627: in
    unpickle build_path = os.environ["LIBTBX_BUILD"]
    /usr/lib/python3.8/os.py:675: in __getitem__ raise KeyError(key)
    from None E KeyError: 'LIBTBX_BUILD'

    for now the path could be: `/var/lib/libtbx/libtbx_env`

  - [X] create a python script which fix the libtbx_env for final
    installed version. this script will be run during the build
    process and installed in place of the original one. We keep the
    original one for the build tests.

** DONE take care of the python modules.
   - [X] patch to remove pathlib2
** TODO bin wrappers.
   the generated wrappers looks like this.

   #+begin_src sh
     LIBTBX_PYEXE="/usr/bin/$LIBTBX_PYEXE_BASENAME"
     export LIBTBX_PYEXE
     if [ -n "$LIBTBX__VALGRIND_FLAG__" ]; then
     exec $LIBTBX_VALGRIND "$LIBTBX_PYEXE" "$LIBTBX_BUILD/../../../../modules/cctbx_project/libtbx/command_line/show_env.py" "$@"
     elif [ $# -eq 0 ]; then
     exec "$LIBTBX_PYEXE" "$LIBTBX_BUILD/../../../../modules/cctbx_project/libtbx/command_line/show_env.py"
     else
     exec "$LIBTBX_PYEXE" "$LIBTBX_BUILD/../../../../modules/cctbx_project/libtbx/command_line/show_env.py" "$@"
     fi
   #+end_src

   the first problem is the path of the script, which is relative to
   $LIBTBX_BUILD.

   This variable is extracted from

   #+begin_src sh
     # ----------------------------------------------------------------------------
     # The shellrealpath function resolves an absolute physical path of its
     # first argument and stores it in a global shell variable RESULT.
     # The function returns nonzero for unreadable or invalid symlinks
     # and resets the RESULT to an empty string.

     shellrealpath() {
      local ORGDIR="$PWD"
      local TARGET="$1"
      RESULT=""
      # This test fails for a symlink loop.  We can do without resolution
      # of symlinks that point to existing unreadable files.
      [ -r "$TARGET" ] || return $?
      # Check if the readlink command exists.
      type readlink >/dev/null || return $?
      while true; do
          cd "$(dirname "$TARGET")"
          TARGET="$(basename "$TARGET")"
          if [ -L "$TARGET" ]; then
              TARGET="$(readlink "$TARGET")"
              continue
          fi
          RESULT="$(pwd -P)/$TARGET"
          break
      done
      cd "$ORGDIR"
      }
      # ----------------------------------------------------------------------------
      LIBTBX_BUILD="$(shellrealpath "$0" && cd "$(dirname "$RESULT")/.." && pwd)"

   #+end_src

   the value of this variable is "/usr" our case.

   Another problem is that once installer on the system, the source path is wrong.

   "$LIBTBX_BUILD/../../../../modules/cctbx_project/libtbx/command_line/show_env.py"

   we install the modules here

   /usr/lib/python3/dist-packages/libtbx/command_line/show_env.py

   an easy fix (but fragile) would be to replace the values in all the
   bin scripts via sed, but this does not guarantees that these wrong
   path ar not use somewhere else in the code.

   the interest of this solution is that we do not touch the upstream
   generated code and the tests can be used out of the box.

   beware there is two cases. the first on is for the cctbx_project
   modules and the other one is for the dials modules (without the
   cctbx_project)

   it is necessary also to remove the LD_LIBRARY_PATH part, PATH and
   PYTHONPATH.
   #+begin_src sh
     :~/Debian/dials/dials$ dials.find_spots
     Traceback (most recent call last):
       File "/usr/lib/python3/dist-packages//dials/command_line/find_spots.py", line 9, in <module>
         from dials.array_family import flex
       File "/usr/lib/python3/dist-packages/dials/array_family/flex.py", line 137, in <module>
         from dials.array_family.flex_ext import (  # noqa: F401; lgtm
       File "/usr/lib/python3/dist-packages/dials/array_family/flex_ext.py", line 18, in <module>
         import cctbx.miller
       File "/usr/lib/python3/dist-packages/cctbx/miller/__init__.py", line 13, in <module>
         from cctbx import maptbx
       File "/usr/lib/python3/dist-packages/cctbx/maptbx/__init__.py", line 17, in <module>
         import libtbx.load_env
       File "/usr/lib/python3/dist-packages/libtbx/load_env.py", line 5, in <module>
         libtbx.env = libtbx.env_config.unpickle()
       File "/usr/lib/python3/dist-packages/libtbx/env_config.py", line 2630, in unpickle
         libtbx_env = open(op.join(build_path, "libtbx_env"), "rb")
     FileNotFoundError: [Errno 2] No such file or directory: '/usr/libtbx_env'
   #+end_src

   This is because the LIBTBX_BUILD variable is exported in all the
   wrappers. When LIBTBX_BUILD is exported it is used to find the
   libtbx_env pickled environment. This is why it does not work. In
   that case build_path=/usr

   It seems that we must use libtbx.refresh in order to regenerate all
   the scripts.

   But it does not work because my script dos not install the non
   =.py= files. So I missed all the .sh scripts.

** TODO script use tcsh instead of sh
   - [X] create setup.py for cctbx and dials
   - [ ] modify them to install also the data.
** TODO hardening
   - [X] all the libraries and python extensions are aware of the
     hardening flags.
   - [ ] upstream the patche used to fix a bunch of missing flags.
** TODO build the documentation
** TODO autocompletion
   autocompletion are generate under /build/dials/autocomplete
** TODO dials.find_spots does not work
   https://github.com/dials/dials/issues/1262 I need to recompile
   bitschuffle with hdf5-serial. It seems that there is a strange
   interaction between OpenMP and dials/cctbx. I think that hdf5
   should not be forgotten in the equation.  I learn that cctbx
   provide an omptbx which is a stub, which should not be used because
   it breaks OpenMP code. This assumption should be verified.

   Speaking with Jerome, it seems that using openMP and hdf5 can cause
   some troubles. bitshuffle openMP version cause a lot of slowdown on
   beamlines since it create an OMP scheduler per thread an this
   invalidate the cache, producing very pour performances.

   But it seems that in my case there is a dead lock. I do not know
   for now how to determine what is the cause of this lock. (hdf5 ?).

   I rebuilt bitstream without openMP and with hdf5-serial. now it
   works.  It would be nice to confirm the reactivating openMP, cause
   some trouble.
** TODO backport all this on buster.
* embedded libraries
  All the un-bundled libraries are listed in the d/copyright file
  under the Files-Excluded section.
** DONE boost
  Files-Excluded: build_dials/modules/boost
** DONE mmdb
  Files-Excluded: build_dials/modules/ccp4io/mmdb
** DONE ssm
  Files-Excluded: build_dials/modules/ccp4io/ssm
** DONE antlr3
  Files-Excluded: build_dials/modules/cctbx_project/ucif/antlr3
** DONE clipper
  Files-Excluded: build_dials/modules/clipper
** DONE eiger
  Files-Excluded: build_dials/modules/eigen
** DONE gl2ps
  Files-Excluded: build_dials/modules/gui_resources/gl2ps
** DONE msgpack
  Files-Excluded: build_dials/modules/msgpack-3.1.1
** DONE scons
  Files-Excluded: build_dials/modules/scons
** DONE tntbx (header only library)
  Files-Excluded: build_dials/modules/tntbx/include
** TODO ccp4
   The sources of this library is patched during the build process.
   So we can not use the system one for now.
** TODO annlib
   Same thing, two version of the library are used, on with a variable
   set to TRUE and the another one with FALSE. the namespace of the
   second library is changed to avoid symbol collision.
** TODO pycbf
  - [X] packaging de python-pycbf #681997 -> get rid of the cbflib part.
  - [ ] The pycbf embedded is compiled with
    SWIG_PYTHON_STRICT_BYTE_CHAR which is not the case of the official
    pycbf. Their is two way to deal with this. recompile the cbflib
    with this option, or fix the dials code to provide normal python3
    string instead of bytes. In fact the problem seems to be with the
    filepath part. On unix filepath are in reality bytes. so the file
    patch should be considere bytes but not the other strings. to my
    opinin, the python strings should not be considere strict byte all
    the time. only path on linux system should. this is why types are
    good when programming (it not enough, right types is important).
    https://github.com/cctbx/cctbx_project/issues/475
** TODO gui_resources
   need to check if the icons are specifiques or generic from the crystal_project
** TODO javascript minified libraries
   - [ ] E: dials source: source-is-missing modules/cctbx_project/crys3d/hklview/ngl.js line length is 32768 characters (>512)
   - [ ] E: dials source: source-is-missing modules/dials/static/js/MathJax.js line length is 32123 characters (>512)
   - [ ] E: dials source: source-is-missing modules/dials/static/js/bootstrap.min.js
   - [ ] E: dials source: source-is-missing modules/dials/static/js/jquery-1.12.0.min.js
   - [ ] E: dials source: source-is-missing modules/dials/static/js/plotly-latest.min.js
   - [ ] E: dials source: source-is-missing modules/dials/static/katex/contrib/auto-render.min.js
   - [ ] E: dials source: source-is-missing modules/dials/static/katex/katex.min.js
* tests
** pytest
  - pytest, missing extension.

    ImportError while importing test module
    '/usr/lib/python3/dist-packages/cctbx/examples/merging/test_cc_half.py'.
    Hint: make sure your test modules/packages have valid Python
    names.
    Traceback:
    /usr/lib/python3/dist-packages/boost/python.py:31: in import_ext
    try: mod = __import__(name)
    E   ModuleNotFoundError: No module named 'scitbx_examples_bevington_ext'

    During handling of the above exception, another exception occurred:
    /usr/lib/python3/dist-packages/cctbx/examples/merging/__init__.py:2: in <module>
    from scitbx.examples import bevington # import dependency
    /usr/lib/python3/dist-packages/scitbx/examples/bevington/__init__.py:4: in <module>
    ext = boost.python.import_ext("scitbx_examples_bevington_ext")
    /usr/lib/python3/dist-packages/boost/python.py:40: in import_ext
    raise ImportError(
    E   ImportError: __import__("scitbx_examples_bevington_ext"): No module named 'scitbx_examples_bevington_ext'
    E     sys.path:
    E       /usr/lib/python3/dist-packages
    E       /usr/bin
    E       /usr/lib/python38.zip
    E       /usr/lib/python3.8
    E       /usr/lib/python3.8/lib-dynload
    E       /usr/local/lib/python3.8/dist-packages
    E       /usr/lib/python3/dist-packages
    E       /usr/lib/python3.8/dist-packages

    this extension is missing due to the removal of eigen. scitbx_examples_bevington_ext
** packaging
  - packaging dials-data pypi, maybe better if integrated in the git repository.
  - packaging de chimera http://www.cgl.ucsf.edu/chimera/
** runtimeError
*** RuntimeError: detector library not found
  ______________________________________________________________ ERROR collecting algorithms/indexing/test_assign_indices.py ______________________________________________________________
  algorithms/indexing/test_assign_indices.py:25: in <module>
  from dxtbx.model.experiment_list import Experiment, ExperimentList
  ../dxtbx/model/experiment_list.py:35: in <module>
  from dxtbx.serialize import xds
  ../dxtbx/serialize/xds.py:16: in <module>
  from dxtbx.model.detector_helpers_types import detector_helpers_types
  ../dxtbx/model/detector_helpers_types.py:87: in <module>
  detector_helpers_types = detector_helpers_types()
  ../dxtbx/model/detector_helpers_types.py:29: in __init__
  raise RuntimeError("detector library not found")
  E   RuntimeError: detector library not found


  We need to install also the data of dials.
  Here the dxtbx detector library locvated here
  dxtbx/data/detectors.lib

*** AssertionError: No models registered with dxtbx.scaling_model_ext entry point
___________________________________________________________________ ERROR collecting algorithms/scaling/test_model.py ___________________________________________________________________
algorithms/scaling/test_model.py:10: in <module>
    from dials.algorithms.scaling.model.model import (
algorithms/scaling/model/model.py:973: in <module>
    assert (
E   AssertionError: No models registered with dxtbx.scaling_model_ext entry point
test/command_line/test_cosym.py:10: in <module>
    from dials.algorithms.symmetry.cosym._generate_test_data import (
algorithms/symmetry/__init__.py:24: in <module>
    from dials.util.resolutionizer import Resolutionizer, phil_defaults
util/resolutionizer.py:12: in <module>
    from dials.algorithms.scaling.scaling_library import determine_best_unit_cell
algorithms/scaling/scaling_library.py:25: in <module>
    from dials.algorithms.scaling.model.model import KBScalingModel
algorithms/scaling/model/model.py:973: in <module>
    assert (
E   AssertionError: No models registered with dxtbx.scaling_model_ext entry point


this code is triggered by

model_phil_scope = phil.parse("")
_dxtbx_scaling_models = {
    ep.name: ep for ep in pkg_resources.iter_entry_points("dxtbx.scaling_model_ext")
}
assert (
    _dxtbx_scaling_models
), "No models registered with dxtbx.scaling_model_ext entry point"


so we need to find what are the entry point of the dxtbx.scaling_model_ext.

These entry_points are stored in the egg files, which are created by
the build system. Is it a good idea to write a setup.py file which
take care of this.

I found three of them in the original build directory:

$ find . -name *egg*
./lib/libtbx.scitbx.egg-info
./lib/libtbx.dxtbx.egg-info
./lib/libtbx.dials.egg-info


* for upstream
** TODO os.environ[FLAG]
   some code use os.environ[FLAG] but does not guard agains the
   KeyError exception.  It is better to use os.environ.get(FLAG) which
   return None instead of throwing an exception.
