Tekst źródłowy strony Prometheus:Basics

__NOTITLE__

== Logging in ==

Access to the Prometheus cluster is facilitated by the access host at:
<pre>
prometheus.cyfronet.pl (alias for the following hosts):

login01.prometheus.cyfronet.pl
login02.prometheus.cyfronet.pl
</pre>

eg.

<pre>
ssh <plgtestowy>@prometheus.cyfronet.pl
</pre>

You can also use the short alias '''pro.cyfronet.pl'''

{| class="wikitable"  style="text-align:center;"
! Login node !! Description
|-
| '''prometheus.cyfronet.pl'''<br>  '''pro.cyfronet.pl''' || Default login node for [https://plgrid.pl PLGrid] users
|-
| '''prace.pro.cyfronet.pl''' || Dedicated login node for [http://www.prace-ri.eu PRACE] users
|-
| '''prace-int.pro.cyfronet.pl''' || Alias of PRACE login node within [http://www.prace-ri.eu PRACE] internal network
|-
|}

An SSH client is required for login.

SSH fingerprints:
<pre>
 (ECDSA)   SHA256:tKk4RpOzWSTf+H8oN7COX1o5xMGSX7vMk956oHcdSB8
 (ED25519) SHA256:roerSF6HBUEqJpoPk36BXwz7NsPRIrTguoApYoYK+ro
 (RSA)     SHA256:v/iUvEh8faHN7IoZ5Vpm0kU6OYKXkGyWp9O9ZFkbeDw

 (ECDSA)   MD5:16:e5:7f:69:45:d0:0f:6c:49:8d:c0:99:0b:e1:e9:dd
 (ED25519) MD5:33:4e:15:cf:71:07:b3:6d:62:0e:b3:7a:9e:7c:7e:13
 (RSA)     MD5:f0:ea:1e:85:79:e0:29:f7:de:64:7a:df:ed:c4:f4:fa
</pre>

Please ensure you accept a valid key.

== Available software ==

A list of available software can be found [[Software|here]].

Access to software is provided through the [[Modules:en|modules]] tool.

== Partitions ==

SLURM replaces PBS queues with the concept of partitions. The following paritions are currently available for Prometheus users:

{| class="wikitable"  style="text-align:center;"
! Parition name !! Max task <br/> duration !! Description
|-
| '''plgrid''' || 72:00:00 || Default partition
|-
| '''plgrid-testing''' || 1:00:00 || For testing and preparing computations (e.g. building applications, copying files etc.)
|-
| '''plgrid-short''' || 1:00:00 || For short jobs (walltime shorter than 1h); provides an increased likelihood that your job will be quickly scheduled for execution
|-
| '''plgrid-long''' || 168:00:00 || For long jobs (as specified in the relevant grant)
|-
| '''plgrid-large''' || 24:00:00 || For jobs requiring at least 432 cores (18 full nodes); contains additional resources not available through the '''plgrid''' partition
|-
| '''plgrid-gpu''' || 72:00:00 || For jobs requiring GPGPU computations (note that access to GPGPU must be allowed under the relevant grant)
|-
| '''plgrid-gpu-v100''' || 12:00:00 || For jobs requiring GPGPU V100 computations (note that access to GPGPU must be allowed under the relevant grant)
|-
| '''prace''' || 72:00:00 || Partition for [http://www.prace-ri.eu PRACE] projects
|-
|}

Further information regarding any specific partition may be obtained by running the following command:

<code>
scontrol show partition [<partition_name>]
</code>

'''Notes:'''
* Selecting the right partition and specifying the expected execution time greatly increase the chances that your job will be quickly scheduled for execution. We recommend that jobs be preferentially submitted to partitions with short maximum execution times (e.g. '''plgrid-short''' where feasible).
* Failing to specify a partition will cause your job to be submitted to the default partition, i.e. '''plgrid'''.
* Jobs submitted to '''plgrid''' (or submitted without specifying a partition) may be automatically transferred to '''plgrid-large''' or '''plgrid-short''' by the queue manager, if this will cause them to be executed faster.
* Currently, all users may access '''plgrid''', '''plgrid-testing''', '''plgrid-short''' and '''plgrid-large'''. Access to other partitions is regulated by grants and provided on a case-by-case basis in the course of evaluating proposed grants.
* If you wish to access '''plgrid-long''', this must be clearly expressed in your grant application.
* Access to '''plgrid-gpu''' requires applying for a separate grant which is wholly dedicated to GPGPU computations.
* '''plgrid-testing''' must not be used for large-scale computations. This partition is specifically meant for tests and other maintenance tasks (e.g. building applications, interactively copying data etc.)

== Disk storage ==

{| class="wikitable" style="text-align:center;"
! Location !! Volume quota !! Files quota !! Purpose !! Access !! Throughput !! Available space !! Backups !! '''Support for intensive <br/> writes and/or reads''' !! Availability !! Remarks
|-
|<tt>$HOME</tt> || 40 GB || 1 000 000 || [[Podstawy#Katalogi_domowe|user home directory]] || NFS || 1 GB/s || ∞ || yes || no || global ||
|-
|<tt>$SCRATCH</tt> || 100 TB || 1 000 000 || [[Podstawy#Przestrzeń tymczasowa|storage for temporary job data]] || Lustre || 120 GB/s || 5 PB || no || yes || global || data older than '''30 days'''<br/>will be automatically deleted
|-
| <tt>$PLG_GROUPS_STORAGE/<team_name></tt>  ||colspan="2"| grant total || Storage for PL-Grid team datasets || Lustre || 30 GB/s || 2.5 PB || no || no || global ||
|-
|<tt>$SCRATCH_LOCAL</tt> || 512 GB || ∞ || [[Podstawy#Lokalna_przestrze.C5.84_typu_scratch|local scratch storage]] || Lustre || 120 GB/s || - || no || yes || limited to single node || unavailable following<br/>completion of job;<br/>requires '''-C localfs'''
|}

=== Environmental variables ===

Personal:
* $HOME, $PLG_USER_STORAGE, eg. '/people/plgtestowy' - points to your home directory
* $PLG_USER_SCRATCH_SHARED, $SCRATCH eg. '/net/scratch/people/plgtestowy/' - points to your private scratch storage - '''this is where data should be stored during computations'''

Team-wide (append team name to each variable):
* PLG_GROUPS_STORAGE, PLG_GROUPS_SHARED, eg. '/net/archive/groups' - points to where teams are expected to deposit their data (including input data, output data and data shared between team members)

=== Local scratch storage ===

For computations which require a large quantity of metadata operations (open/close/create/remove) it is possible to set up dedicated scratch storage. To do so, please supply the <code>-C localfs</code> parameter when submitting your job (i.e. at the resource allocation stage, before the job ID is assigned).

For example, when submitting jobs in batch mode, use the following directive:
<code>#SBATCH -C localfs</code>

Any job submitted in this fashion will be assigned a separate scratch volume, available on the computational nodes and referenced by the '''$SCRATCH_LOCAL''' environmental variable (using the /localfs/<task_id> naming convention). '''$SCRATCH_LOCAL''' space is optimized for processing small files. Each computational node possesses a separate '''$SCRATCH_LOCAL''' volume, available only during execution of the given job.

The size of '''$SCRATCH_LOCAL''' is 512GiB per computational node assigned to your job. Please contact the administrators if you require more scratch space.

'''Caution:''' The '''$SCRATCH_LOCAL''' volume is available only on those computational nodes which have been allocated to your job, and only while the job is executing. Furthermore, it is not shared across nodes (i.e. each node possesses a separate volume). Following completion of your job, you will lose any data saved in '''$SCRATCH_LOCAL''' - if this data is relevant, it must first be copied to your team storage ($PLG_GROUPS_STORAGE/<team_name>) or to your home directory ($HOME).

=== MEMFS RAM storage ===

In order to allocate RAM for storage of temporary files, please add the "-C memfs” parameter to your job specification. 
For example, use the following directive in your batch script:
<code>#SBATCH -C memfs</code>

A storage volume will be set up for your job, referenced by the '''$MEMFS''' environmental variable. Please note that memory allocated to MEMFS storage counts towards the total memory allocated for your job and declared through "--mem" or "--mem-per-cpu".

'''Caution:''' When using MEMFS for file storage, be aware of the following limitations:
* this method is only available for single-node jobs
* the total amount of memory consumed by your job, including any MEMFS storage, must not exceed the value declared through "--mem" or "--mem-per-cpu".
* this method may only be used if the total memory requirements of your job (including MEMFS storage) do not exceed the memory available on a single node (120 GB per standard Prometheus node).
* when using MEMFS, it is recommended to request allocation of a full node for your job.

== Running jobs ==

The SLURM scheduling system is used to manage jobs on the Prometheus cluster.

A job may be executed in batch or interactive mode.

'''<span style="color: crimson">Before running any jobs, please familiarize yourself with the [[System_Grant%C3%B3w|grant system]]</span>'''

=== Batch mode ===

To run a job in batch mode, use the <tt>sbatch</tt> command.
Usage:
<code>sbatch script.sh</code>.<br/>
All scheduler options should be preceded by the '''#SBATCH''' keyword (do not forget the #!).<br/>
For more information see: <code>man sbatch</code> and <code>sbatch --help</code>.

'''Sample script:'''

'''<span style="color: crimson">CAUTION! This sample script is provided as a reference and SHOULD NOT be executed as is. Before running any jobs, make sure that the scheduler options accurately reflect your requirements e.g. with regard to grant ID, partition name, execution time etc.</span>'''

<syntaxhighlight lang="bash">
#!/bin/bash -l
## Job name
#SBATCH -J ADFtestjob
## Number of allocated nodes
#SBATCH -N 1
## Number of tasks per node (by default this corresponds to the number of cores allocated per node)
#SBATCH --ntasks-per-node=1
## Memory allocated per core (default is 5GB)
#SBATCH --mem-per-cpu=1GB
## Max task execution time (format is HH:MM:SS)
#SBATCH --time=01:00:00 
## Name of grant to which resource usage will be charged
#SBATCH -A <grant_id>
## Name of partition
#SBATCH -p plgrid-testing
## Name of file to which standard output will be redirected
#SBATCH --output="output.out"
## Name of file to which the standard error stream will be redirected
#SBATCH --error="error.err"

## change to sbatch working directory
cd $SLURM_SUBMIT_DIR

srun /bin/hostname
module load plgrid/apps/adf/2014.07 
adf input.adf

</syntaxhighlight>

==== MPI jobs ====

Running MPI jobs on Prometheus is similar to running them on other SLURM clusters with integrated MPI libraries. Prometheus provides various versions of MPI libraries (mostly OpenMPI and IntelMPI) for a variety of popular compilers. When running MPI jobs, the <code>-n</code> and <code>-np</code> scheduler options should not be explicitly specified - instead, the system will select optimal values based on allocation parameters expressed in the startup script. Usage of <code>-n</code> and <code>-np</code> should be restricted to exceptional cases which call for nonstandard configurations (e.g. local threading).

'''<span style="color: crimson">CAUTION! This sample script is provided as a reference and SHOULD NOT be executed as is. Before running any jobs, make sure that the scheduler options accurately reflect your requirements e.g. with regard to grant ID, partition name, execution time etc.</span>'''

<syntaxhighlight lang="bash">
#!/bin/bash -l
## Job name
#SBATCH -J MPITest
## Number of allocated nodes
#SBATCH -N 2
## Number of tasks per node (by default this corresponds to the number of cores allocated per node)
#SBATCH --ntasks-per-node=24
## Memory allocated per core (default is 5GB)
#SBATCH --mem-per-cpu=5GB
## Max task execution time (format is HH:MM:SS)
#SBATCH --time=01:00:00 
## Name of grant to which resource usage will be charged
#SBATCH -A <grant_id>
## Name of partition
#SBATCH -p plgrid-testing
## Name of file to which standard output will be redirected
#SBATCH --output="output.out"
## Name of file to which the standard error stream will be redirected
#SBATCH --error="error.err"

srun /bin/hostname

## Load the IntelMPI module
module add plgrid/tools/impi

## change to sbatch working directory
cd $SLURM_SUBMIT_DIR

mpiexec ./calcDiff 100 50
</syntaxhighlight>

It is considered good practice to compile and execute applications in an environment comprising the same set of modules. We recommend using the <code>mpiexec</code> wrapper. Computational resource allocation - in this case 2 nodes, 24 cores each - results in 48 MPI tasks available for execution. It is permissible to run a series of MPI applications with a single script, but given their execution time (which is usually long) this approach is discouraged. Running multiple MPI applications in parallel within a single allocation is not allowed due to resource contention issues.

==== Simple parallelization of independent tasks ====

Parallel processing is a typical use case scenario for computing clusters. One representative example involves processing multiple images with a shared algorithm. SLURM provides a way to easily parallelize such tasks with the '''srun''' command. A sample script which runs multiple computations in parallel (depending on their specifications) is provided below.

'''<span style="color: crimson">CAUTION! This sample script is provided as a reference and SHOULD NOT be executed as is. Before running any jobs, make sure that the scheduler options accurately reflect your requirements e.g. with regard to grant ID, partition name, execution time etc.</span>'''

<syntaxhighlight lang="bash">
#!/bin/bash -l
## Job name
#SBATCH -J testjob
## Number of allocated nodes
#SBATCH -N 2
## Number of tasks per node (by default this corresponds to the number of cores allocated per node)
#SBATCH --ntasks-per-node=24
## Memory allocated per core (default is 5GB)
#SBATCH --mem-per-cpu=5GB
## Max task execution time (format is HH:MM:SS)
#SBATCH --time=01:00:00 
## Name of grant to which resource usage will be charged
#SBATCH -A <grant_id>
## Name of partition
#SBATCH -p plgrid-testing
## Name of file to which standard output will be redirected
#SBATCH --output="output.out"
## Name of file to which the standard error stream will be redirected
#SBATCH --error="error.err"

module load plgrid/tools/imagemagick

## change to sbatch working directory
cd $SLURM_SUBMIT_DIR

ls *.tif | xargs -t -d "\n" -P ${SLURM_NTASKS} -n 1 srun -n 1 -N 1 --mem=5gb mogrify -format png
</syntaxhighlight>

The above script runs the <code>mogrify</code> application for each *tif file in the working directory, converting the image to the png format. We begin by listing all input files, and feed the output into the <code>xargs</code> command, which (through the '''-P''' flag) ensures parallel execution of each subsequent <code>srun</code> instance. Each instance is supplied with a single input parameter, with the maximum level of parallelism defined as '''${SLURM_NTASKS}'''. Thus, each invocation of <code>srun</code> runs the <code>mogrify</code> application on a single core, with 5GB of assigned memory.

=== Interactive mode ===

To run shell commands in interactive mode, use the following (sample) command:
<code>srun -p plgrid-testing -N 1 --ntasks-per-node=1 -n 1 -A <grant_id> --pty /bin/bash -l</code>

This will run a job in the plgrid-testing partition, on a single node, using a single core.

The <tt>srun</tt> command is typically used to execute the specified operation on the assigned resources.<br> However, if resources have not been allocated in advance, <tt>srun</tt> also takes care of performing the allocation.

'''<span style="color: crimson">CAUTION: Graphic mode (using X11) is only available on the Prometheus access host. </span>'''

=== Array jobs ===

Array jobs provide a way to schedule large numbers of similar coputational tasks. Each instance of an array job is assigned an index referenced by the '''$SLURM_ARRAY_TASK_ID''' environmental variable, which can be used to parameterize the given instance. Submitting array jobs is similar to submitting ordinary batch jobs. The following sample script submits an array job with the task ID range defined by 
the <code>--array</code> (or <code>-a</code>) scheduler option:

'''<span style="color: crimson">CAUTION! This sample script is provided as a reference and SHOULD NOT be executed as is. Before running any jobs, make sure that the scheduler options accurately reflect your requirements e.g. with regard to grant ID, partition name, execution time etc.</span>'''

<syntaxhighlight lang="bash">
#!/bin/bash -l
## Job name
#SBATCH -J testjob
## Number of allocated nodes
#SBATCH -N 1
## Number of tasks per node (by default this corresponds to the number of cores allocated per node)
#SBATCH --ntasks-per-node=1
## Memory allocated per core (default is 5GB)
#SBATCH --mem-per-cpu=5GB
## Max task execution time (format is HH:MM:SS)
#SBATCH --time=01:00:00 
## Name of grant to which resource usage will be charged
#SBATCH -A <grant_id>
## Name of partition
#SBATCH -p plgrid-testing
## Name of file to which standard output will be redirected
#SBATCH --output="output.out"
## Name of file to which the standard error stream will be redirected
#SBATCH --error="error.err"
## Array index range
#SBATCH --array=0-100

## change to sbatch working directory
cd $SLURM_SUBMIT_DIR

myCalculations $SLURM_ARRAY_TASK_ID

</syntaxhighlight>

=== GPGPU jobs ===

==== Accessing GPGPU nodes ====

A special partition named '''plgrid-gpu''' has been set up for jobs which require access to GPGPU hardware. In order to schedule such jobs, it is first necessary to obtain a grant specifically dedicated to GPGPU processing. This grant should not be used to run any other types of computational tasks.

When applying for your grant, please specify that you require access to '''plgrid-gpu'''. It is also recommended (although not strictly required) to put '''gpu''' in your grant name (e.g. '''gpucomputations''') - this will help prevent potential mix-ups. 

All applications for access to '''plgrid-gpu''' are evaluated on a case-by-case basis by the resource provider.

==== General information ====

The SLURM scheduler treats GPUs as generic resources (GRES) identified by the "gpu" codeword.<br/> You can find out which nodes/partitions provide such resources by using the <tt>sinfo</tt> command:

<code>sinfo -o '%P || %N || %G'</code>

Jobs are submitted by adding <code>--partition=plgrid-gpu --gres=gpu[:count]</code> to your list of scheduler options. If the <tt>count</tt> argument is omitted, the system will default to allocating a single GPU adapter per processing node.

Following submission of your job, the scheduler automatically sets the <tt>$CUDA_VISIBLE_DEVICES</tt> environmental variable and permits access to allocated GPUs.

==== Interactive jobs ====

For example, to run a single-node interactive job with 2 GPUs, type:

<code>srun -p plgrid-gpu -N 1 -n 24 -A <grant_id> --gres=gpu:2 --pty /bin/bash -l</code>

==== Interactive MPI jobs ====

Caution: running interactive MPI jobs (e.g. for testing) is a special case. Due to the fact that SLURM treats GPUs differently than general-purpose processors, launching interactive computations calls for a nonstandard procedure.

For example, to run an interactive job on 2 nodes with 2 GPUs per node (4 GPUs total) for 1 hour, specify the following:

<code>salloc -p plgrid-gpu -N 2 --ntasks-per-node 24 -n 48 -A <grant_id> --gres=gpu:2 --time 1:00:00</code>

The <tt>salloc</tt> command allocates our job and returns its ID - for example, 1234.

Next, we use <tt>srun</tt> requesting 0 GPUs for the job whose ID was returned by <tt>salloc</tt>:

<code>srun --jobid=1234 --gres=gpu:0 -O --pty /bin/bash -l</code>

This gives us shell access on one of the computing nodes. Now, after loading the appropriate MPI module, we may run our actual application via <tt>mpirun</tt> or <tt>mpiexec</tt>. It is important to avoid exporting environmental settings - e.g., when using IntelMPI, remember to specify the <code>-genvnone</code> flag. The application will retain access to all GPUs allocated via <tt>salloc</tt> regardless of the <tt>gpu:0</tt> parameter in <tt>srun</tt>.

Once the computation has concluded, the allocation should be manually cleared by typing:

<code>scancel 1234</code>

==== Batch jobs ====

No special procedure is required for batch jobs. For example, to request one GPU adapter per computing node, simply add the following to your script:

<syntaxhighlight lang="bash">
#SBATCH --partition=plgrid-gpu
#SBATCH --gres=gpu
</syntaxhighlight>

'''Note:''' Further details regarding SLURM commands may be obtained by typing <code> man sbatch</code>

== Monitoring queues, partitions, nodes, jobs and resources ==

*; Dedicated Prometheus scripts
** '''[[Prometheus:pro-jobs|pro-jobs]]''' - lists jobs queued by or running for the current user
** '''[[Prometheus:pro-jobs-history|pro-jobs-history]]''' - lists completed jobs for the current user
** '''[[Prometheus:pro-fs|pro-fs]]''' - returns disk usage statistics for the current user
** '''[[Prometheus:pro-groupdir-fix|pro-groupdir-fix]]''' - used to adjust access permissions for the team data directory
** '''pro-grant-jobs''' - lists jobs scheduled under the user's active grants (including jobs whch belong to other users sharing these grants)

*; SLURM system commands
**<code>squeue</code> - list of currently queued/running jobs
**<code>scontrol show job [<job_ID>]</code> - details of a given job
**<code>sstat -a -j <job_ID>/<job_ID.batch></code> - resource consumption for each step of a currently running job
**<code>scancel <job_ID></code> - cancel a queued or running job
**<code>sacct</code> - resource consumption for jobs/steps which have already been completed
**<code>scontrol show partition [<partition_name>]</code> - details of a given partition
**<code>sinfo</code> - list of nodes
**<code>scontrol show node [<node_name>]</code> - details of a given node

<!-- == Properties of computing nodes == -->
<!-- == Using the Infiniband network == -->

== General rules for using the Prometheus cluster ==

* Any computations launched on the access host will be killed without notice. The access host should only be used to prepare input files and submitting jobs to the scheduler. Since transferring large quantities of data may cause significant load on the access host, it is recommended to allocated a computing node for this purpose (in interactive mode, e.g. <code>srun -p plgrid -A grant_id -n 1 --pty /bin/bash -l</code>). The same goes for compiling applications - this should be performed on a computing node, not on the access host.
* Computations which are out of scope of the subitted research agenda or which consume a large fraction of the cluster's resources will be killed without notice.
* Working datasets and intermediate results should be stored and processed in the Lustre filesystem available under $SCRATCH.
* Computations '''should not''' make use of the /tmp directory to store temporary data - instead, please use directories referenced by $SCRATCH, $SCRATCHDIR or $TMPDIR.
* The access host imposes a 6GB memory limit for all processes belonging to the current user. Exceeding this limit will cause the offending process to be automatically killed.

== Contact ==
Assistance for [http://www.plgrid.pl PL-Grid] users is available at: [https://helpdesk.plgrid.pl/ Helpdesk PLGrid] <br /> System administrators may be contacted by e-mail at: [mailto:prometheus@cyfronet.pl prometheus@cyfronet.pl], or by phone at +48 12 632 33 55 int. 603.