Compile on MC3

Mc3.stanford.edu is a Linux (Rocky Linux 8) cluster.

Create conda environment (opendis)

This is for installing Python3 executable as well as modules (such as matplotlib) needed to run test case

eval "$(/opt/ohpc/pub/compiler/anaconda3/2024.02-1/bin/conda shell.bash hook)"
conda init
conda create --name opendis python=3.9
conda activate opendis
conda install matplotlib
conda install networkx

To ensure matplotlib runs properly on MC3, we need to specify its backend to be tkagg in the matplotlibrc file

vi ~/.config/matplotlib/matplotlibrc
backend: tkagg

Load modules

Put the following lines in your ~/.bash_profile file, exit and login again

module load cmake/3.24.2
module load gnu9/9.4.0
module load cuda/nvhpc/23.7

Install FFTW

Follow the instructions at this wiki page to install FFTW in your home directory. The library and include files should be put in paths consistent with what’s specified in the cmake/sys.cmake.mc3_cpu file.

Note

FFTW is only required when compiling for CPU/OMP. When compiling for cuda GPU, FFT transforms will be performed using the cuFFT library.

Build ExaDiS/KOKKOS (OMP version)

cd ${OPENDIS_DIR}
conda activate opendis
rm -rf build/; ./configure.sh -DSYS=mc3_cpu
cmake --build build -j 8 ; cmake --build build --target install

Alternatively, you can also copy the cmake/sys.cmake.mc3 file to cmake/sys.cmake.ext and configure without -DSYS. The cmake/sys.cmake.ext file is not tracked by git so you can feel free to experiment with the settings.

cp cmake/sys.cmake.mc3_cpu cmake/sys.cmake.ext
rm -rf build/; ./configure.sh 
cmake --build build -j 8 ; cmake --build build --target install

Hint

If you encounter the error pyexadis not found, it is likely that the Python used for compiling and the Python used for running are different. The Python executable used in compiling is specified in the cmake/sys.cmake.mc3_cpu file. Check which Python version you are running by e.g. python3 --version. Make sure you are in the opendis conda environment using the following command before both compiling and running.

conda activate opendis

When compilation is successful, you should see a file like pyexadis.cpython*.so in the core/exadis/python folder.

Run test case (OMP version)

export OMP_NUM_THREADS=8
cd ${OPENDIS_DIR}
cd examples/02_frank_read_src
conda activate opendis
python3 -i test_frank_read_src_exadis.py

Build ExaDiS/KOKKOS (GPU version)

# first we need to get on a GPU node (gpu-ampere) in order to use nvcc
srun -p gpu-ampere -n 1 --x11 --pty bash
cd ${OPENDIS_DIR}
conda activate opendis
rm -rf build/; ./configure.sh -DSYS=mc3_ampere
cmake --build build -j 8 ; cmake --build build --target install

Hint

The Python executable used in compiling is ${HOME}/.conda/envs/opendis/bin/python, as specified in the cmake/sys.cmake.mc3_tesla file. Make sure you use the same Python version when running as that used for compiling by using the following command.

conda activate opendis

Run test case (GPU version)

export OMP_NUM_THREADS=8
cd ${OPENDIS_DIR}
cd examples/02_frank_read_src
conda activate opendis
python3 -i test_frank_read_src_exadis.py