Nsight Systems

• System-wide performance analysis
• CPU + GPU
• Visualise a timeline of events
• CUDA API information, kernel block sizes etc
• Pascal GPUs or newer (SM 60+)

nsys profile -o timeline ./celer-g4 input.mac
nsys-ui timeline.nsys-rep

Code Annotation

• NVIDIA Tools Extension (NVTX)
• AMD ROCTX

void some_function() {

    for (int i = 0; i < 6; ++i) {

        std::this_thread::sleep_for(std::chrono::milliseconds{100});

    }

}

Code Annotation

• NVIDIA Tools Extension (NVTX)
• AMD ROCTX

#include <nvtx3/nvToolsExt.h>
void some_function() {
    nvtxRangePush(__FUNCTION__);
    for (int i = 0; i < 6; ++i) {
        nvtxRangePush("inner")
        std::this_thread::sleep_for(std::chrono::milliseconds{100});
        nvtxRangePop();
    }
    nvtxRangePop();
}

Nsight Compute

• Detailed GPU performance metrics
• Compile with -lineinfo for line-level profiling
• Capture full metrics via --set=full
• Replays GPU kernels many times - significant runtime increase
• Reduce captured kernels via filtering, -s, -c etc.
• SM 70+ (Volta)

# All metrics, skip 64 kernels, capture 128.
ncu --set=full -s 64 -c 128 -o metrics celer-g4 input.mac
ncu-ui metrics.ncu-rep

• May require --target-processes
• Nvidia profiler counters require root or security mitigation disabling since 418.43 (2019-02-22). See ERR_NVGPUCTRPERM.

Spack Compute Capability

• Spack only accepts a single cuda_arch value
• Requires a full dependency rebuild (~90 mins)

Dockerfile nsys

nvidia/cuda:11.8.0-devel-ubuntu22.04 does not include nsys

Nsys 2022.4.2 (CUDA 11.8.0):

# Install nsys for profiling. ncu is included
RUN if [ "$DOCKERFILE_DISTRO" = "ubuntu" ] ; then \
  apt-get -yqq update \
  && apt-get -yqq install --no-install-recommends nsight-systems-2022.4.2 \
  && rm -rf /var/lib/apt/lists/* ; \ 
fi

• Note: nsys and ncu will be removed from the -ci containers, which are smaller for bandwidth reasons.

Docker to Apptainer / Singularity

• apptainer/singularity build can convert docker files to apptainer images
• from a registry via apptainer build img.sif docker://registry/image:tag
• locally via deameon apptainer build img.sif docker-deamon:registry/image:tag
• locally via docker archive files
• https://apptainer.org/docs/user/main/docker_and_oci.html

# Build the appropriate container 
cd celeritas/scripts/Docker
# Build the cuda Docker container, sm_70. Wait ~90 minutes.
./build.sh cuda
# If the image hasn't been pushed to a registry, apptainer requires a local path, so save the image
rm -f docker-temp.tar && docker save $(docker images --format="{{.Repository}} {{.ID}}" | grep "celeritas/dev-jammy-cuda11" | sort -rk 2 | awk 'NR==1{print $2}') -o image.tar
# Convert to an apptainer container in the working dir
apptainer build -F celeritas-dev-jammy-cuda11.sif docker-archive:image.tar

Docker to Apptainer / Singularity

• Docker and Apptainer have different defaults when executing images
  • Default directory bindings
  • environment variable mapping
  • in-container user
  • entrypoints
• Likely need to run with various flags to achieve similar behaviour

Docker to Apptainer / Singularity

# ephemeral, does not bind home dir by default
docker run --rm -ti --gpus all -v .:/src celeritas/dev-jammy-cuda11:2023-06-19
# apptainer, runs as the current user, with the calling users env vars and default bidnings
apptainer run --nv --bind ./:/celeritas-project celeritas-dev-jammy-cuda11-2023-06-19.sif 
# TUoS HPC - this is not perfect
apptainer run --nv --bind ./:/celeritas-project /mnt/parscratch/users/ac1phey/celeritas-dev-jammy-cuda11-sm90.sif

lineinfo

Add -lineinfo to

mkdir build-lineinfo && cd build-lineinfo
cmake .. -DCMAKE_CUDA_ARCHITECTURES=70 -DCMAKE_BUILD_TYPE=Release -DCMAKE_CUDA_FLAGS_RELEASE="-O3 -DNDEBUG -lineinfo" -DCELERITAS_DEBUG=OFF 
cmake --build . -j `nproc`