Um Jupyter-Notebooks effizient und ohne Port-Forwarding im Cluster zu nutzen gibt es einen JupyterHub unter https://jupyter.hpc.rz.uni-duesseldorf.de
Dort können Jobs mit vordefinierten Ressourcen abgeschickt werden, innerhalb derer dann das Notebook gestartet wird.
Derzeit werden dort nur Shells und Python 3 - Notebooks angeboten, aber die Liste soll erweitert werden.
Install packages
Install individual Python kernel
Jupyter allows you to work in your own virtual environment using the Python modules installed on the HPC. Start by loading the Python interpreter/module you want to work with, e.g.:
module load Python/3.11.3
Create a new virtual environment, e.g. in /gpfs/project/$USER, then update pip, setuptools and wheel (replace $USER with your username):
python -m venv --prompt py311 --system-site-packages /gpfs/project/$USER/py311 PIP_CONFIG_FILE=/software/python/pip.conf pip install --user --upgrade pip setuptools wheel
Activate this environment and install all packages you want to work with - at least ipykernel must be installed
source /gpfs/project/$USER/py311/bin/activate PIP_CONFIG_FILE=/software/python/pip.conf pip install ipykernel
Create a new file "kernel.sh" in the main directory of your environment and make it executable
cd /gpfs/project/$USER/py311 vi kernel.sh kernel.sh ---------------------------------------------------- #!/bin/bash source /gpfs/project/$USER/py311/bin/activate exec python -m ipykernel $@ ---------------------------------------------------- chmod a+x kernel.sh
Create a new directory for your kernel in your /home/.local/share folder
mkdir -p /home/$USER/.local/share/jupyter/kernels/py311 cd /home/$USER/.local/share/jupyter/kernels/py311
Create a new file "kernel.json" with contents
{
"argv": [
"/gpfs/project/$USER/py311/kernel.sh",
"-f",
"{connection_file}"
],
"display_name": "Venv (py311)",
"language": "python",
"metadata": {
"debugger": true
}
}
In your next jupyterhub session a new kernel with the name "venv (py311)" will then be available.
