Compilation Overview

OpenDiS is a python-based framework that relies on core libraries to implement calculation modules, see Project Overview. While some OpenDiS scripts can be run without requiring compilation of the code, most of the simulations would require compilation to enjoy high-performance modules.

Hint

All scripts/simulations that use modules from the pyexadis library require the code to be compiled before they can be run. The pyexadis library is built during compilation of the ExaDiS core library, included in the OpenDiS compilation tree.

CMake build

OpenDiS is compiled using CMake. Core libraries and modules that need to be compiled are included in the OpenDiS compilation tree. At the moment, a build includes compilation of the ExaDiS core library (including the pyexadis library), and some modules of the PyDiS library.

A typical installation follows the steps below:

Step 1: Prepare the build directory

Make sure any previous build directory is removed:

cd ${OPENDIS_DIR}
rm -rf build/

Step 2: Configure the build

Configure the build for your system by passing build options to the configure.sh script, including options to build for CPU or GPU architectures. (See list of options in the Build Options section below.)

Below are various examples of how OpenDiS can be configured:

  • Example: default build with ExaDiS SERIAL and OPENMP backends

./configure.sh

  • Example: GPU build with ExaDiS CUDA backend and device architecture VOLTA70

./configure.sh \
    -DKokkos_ENABLE_CUDA=On \
    -DKokkos_ENABLE_CUDA_LAMBDA=On \
    -DKokkos_ARCH_VOLTA70=On

  • Example: use a pre-defined build options defined in cmake/sys.cmake.<system>, and then pass build argument -DSYS=<system>. E.g., to build for SYS=mac (i.e. using options set in file cmake/sys.cmake.mac):

./configure.sh -DSYS=mac

For more detailed instructions on pre-defined build options available in OpenDiS, see the next pages of this section.

  • Example: create your own build options by setting options specific to your system in file e.g. cmake/sys.cmake.mysystem, and then pass build argument -DSYS=mysystem:

./configure.sh -DSYS=mysystem

Tip

If no SYS option is passed to configure.sh, e.g. as for a default build

./configure.sh

then by default the configuration will attempt to read build options from file cmake/sys.cmake.ext if it exists. Thus, file cmake/sys.cmake.ext can be used as a sandbox to test different build configurations.

Step 3: Build and install the codes

Once configuration is completed, build and install the codes using the following command:

cmake --build build -j 8 ; cmake --build build --target install

A successful compilation will produce various files in ${OPENDIS_DIR}/lib, and shared library file ${OPENDIS_DIR}/core/exadis/python/pyexadis.cpython-*.so.

Note

Building for GPU (e.g. with nvcc or hipcc) may be pretty slow, please be patient!

Step 4: Test your installation by running some examples

Once compilation is completed, run examples from the examples/ folder, e.g.:

cd examples/02_frank_read_src
python3 test_frank_read_src_pydis.py
python3 test_frank_read_src_exadis.py

More information about these two examples can be found in the Frank-Read Source section, and more examples are detailed in the Tutorials section.

If errors occur when trying to run the examples, please refer to the Troubleshooting section.

Build options

Besides the standard CMAKE options and the SYS option, most OpenDiS build options are related to the ExaDiS/Kokkos core library. See the list of ExaDiS build options.

Troubleshooting

This section reports common errors encountered by OpenDiS users. If you encounter other issues or bugs, please also browse the Issues section and/or open a new issue.

pyexadis import errors

Issue: Python is not being able to import pyexadis (python binding to the ExaDiS core library), with errors such as

import pyexadis
ModuleNotFoundError: No module named 'pyexadis'

or

raise ImportError('Cannot import pyexadis')
ImportError: Cannot import pyexadis

This typically happens when the python installation used for compilation is not the same as the python installation used to run the code.

When running the ./configure.sh script, the python executable to be used for compilation should be indicated, e.g.:

-- Found PythonInterp: /path/to/python3

Then after compilation one should be able to run the scripts using the same python executable, e.g.

/path/to/python3 test_frank_read_src_exadis.py

Alternatively, one can explicitly specify the python executable to be used by passing build option -DPYTHON_EXECUTABLE=... to the configure.sh script, e.g.

./configure.sh \
    -DKokkos_ENABLE_CUDA=On \
    -DKokkos_ENABLE_CUDA_LAMBDA=On \
    -DKokkos_ARCH_VOLTA70=On \
    -DPYTHON_EXECUTABLE=$(which python3)

in which case one should be able to run the scripts with, e.g.

python3 test_frank_read_src_exadis.py

Hanging simulations

Issue: Simulations appear to be hanging after initialization, e.g. after diplaying

Run for X steps

The cause of this issue is generally an inter-operability issue between the compilers, or between python and the compiler(s).

First, identify whether this issue is specific to pyexadis or also occurs for regular ExaDiS simulations. To do so, rebuild the code with option -DEXADIS_BUILD_TESTS=On, e.g.

cd ${OPENDIS_DIR}
rm -rf build/
./configure.sh \
    -DKokkos_ENABLE_CUDA=On \
    -DKokkos_ENABLE_CUDA_LAMBDA=On \
    -DKokkos_ARCH_VOLTA70=On \
    -DEXADIS_BUILD_TESTS=On
cmake --build build -j 8 ; cmake --build build --target install

This should create a few executables in directory ${OPENDIS_DIR}/build/core/exadis/tests. Run these executables to help identify the source of the issue:

cd ${OPENDIS_DIR}/build/core/exadis/tests
./test_cuda # if compiled with -DKokkos_ENABLE_CUDA=On
./test_kokkos
./test_exadis 0
./test_exadis 1
...
./test_exadis 8

  • If none of the tests fail but the hanging still happens when running the python-based examples, then the issue is with the compilation of pyexadis. First make sure that the python installation used to run is the same as the python installation used for compilation, e.g. see previous issue. If the issue remains, try using a different combination of compilers and python installations.

  • If test_cuda fails then the issue is likely with the CUDA compiler

  • If test_kokkos fails then the issue is with the compilation of Kokkos

  • If test_exadis fails then the issue is with the compilation of ExaDiS

Try using a different combination of compilers to resolve compatibility issues. Some known issues are:

  • Compatibility issues between gcc11 and some versions of cuda11

  • Compatibility issues have been reported between Kokkos and gcc12

  • Newest versions of cuda (e.g. 12.5+) have not been fully tested with Kokkos and pybind11 and may trigger compatibility issues