Um Jupyter-Notebooks effizient und ohne Port-Forwarding im Cluster zu nutzen gibt es einen JupyterHub unter https://jupyter.hpc.rz.uni-duesseldorf.de
Dort können Jobs mit vordefinierten Ressourcen abgeschickt werden, innerhalb derer dann das Notebook gestartet wird.
Derzeit werden dort nur Shells und Python 3 - Notebooks angeboten, aber die Liste soll erweitert werden.
Install packages
Install individual Python kernel (Wait ... under construction .....)
Jupyter allows you to work in your own virtual environment using the Python modules installed on the HPC. Start by loading the Python interpreter/module you want to work with, e.g.:
module load Python/3.10.5
Create a new virtual environment, e.g. in /gpfs/project/$USER, then update pip, setuptools and wheel (replace $USER with your username):
python -m venv --prompt py310 --system-site-packages /gpfs/project/$USER/py310 PIP_CONFIG_FILE=/software/python/pip.conf pip install --user --upgrade pip setuptools wheel
Hint: it may be neccessary to install virtualenv as a user if this fails: PIP_CONFIG_FILE=/software/python/pip.conf pip install --user virtualenv
Activate this environment and install all packages you want to work with - at least ipykernel must be installed
source /gpfs/project/$USER/py310/bin/activate PIP_CONFIG_FILE=/software/python/pip.conf pip install ipykernel
Create a new file "kernel.sh" in the main directory of your environment and make it executable
cd /gpfs/project/$USER/py310 vi kernel.sh kernel.sh ---------------------------------------------------- #!/bin/bash module load Python/3.10.5 source /gpfs/project/$USER/py310/bin/activate exec python -m ipykernel $@ ---------------------------------------------------- chmod a+x kernel.sh
Create a new directory for your kernel in your /home/.local/share folder
mkdir -p /home/$USER/.local/share/jupyter/kernels/py310 cd /home/$USER/.local/share/jupyter/kernels/py310
Create a new file "kernel.json" with contents
{
"argv": [
"/gpfs/project/$USER/py310/kernel.sh",
"-f",
"{connection_file}"
],
"display_name": "Venv (py310)",
"language": "python",
"metadata": {
"debugger": true
}
}
In your next jupyterhub session a new kernel with the name "venv (py311)" will then be available.
