JackNeely NC State University College of PAMS

pco@pams.ncsu.edu

PAMS is moving its High Performance Computing Beowulf cluster to a new set of software called NPACI Rocks and phasing out the very old version of the Beowulf software from Scyld. The new software has all the same benefits as the old but works very differently. The goal of this short document is to give users of our new Beowulf software a quick start to computing on the cluster and pointers for where to get more information. The most noticeable difference is that we have changed the batch system we use to the Sun Grid Engine and now require users to submit all jobs via this batch system. (Jobs not submitted through the batch system or run on the head node will be terminated by the system administrators for the Beowulf.) Another noticeable change is that the software is now based on Red Hat Linux 7.3 and much more modern.

On to submitting jobs. Each job is nothing more than a shell script. You submit this shell script via qsub and the Sun Grid Engine (SGE) will schedule it and then run it. For non parallel compute jobs its pretty simple. For example: [jjneely@beo-test jjneely]$ cat job.csh #!/bin/tcsh echo "Running job job.csh" date sleep 20 date echo "WOOT!" [jjneely@beo-test jjneely]$ qsub job.csh your job 6 ("job.csh") has been submitted [jjneely@beo-test jjneely]$ The script job.sh is my "compute job" and I've used the qsub command to submit it to the system. After the job has been scheduled and run it will leave its output as files in your home directory on the cluster. [jjneely@beo-test jjneely]$ ls job.* job.csh job.csh.e6 job.csh.o6 [jjneely@beo-test jjneely]$ Here we see the script and the output files. The files always end in a numerical ID -- that's the job ID to SGE. The "e" and the "o" specify the standard error, and the standard out outputs respectively. [jjneely@beo-test jjneely]$ cat job.csh.o6 Running job job.csh Mon Feb 10 18:45:14 GMT 2003 Mon Feb 10 18:45:34 GMT 2003 WOOT! [jjneely@beo-test jjneely]$ There are many, many options to qsub and different ways of running jobs. (Such as "array jobs" which run the same script multiple times to get different output.) A handy one is -cwd which runs the job from the same directory that you submitted it from and puts your output files in that directory. All of the details can be found in the documentation for the Sun Grid Engine.

Next, how do you run an MPI job? Yes, MPI jobs must also be submitted via qsub and SGE knows exactly how to work with them. There are three things you'll need to run an MPI job. First, you will have to request a parallel environment and specify how many (can be a range) of compute nodes you need for your MPI job. Secondly, you'll need to use the $NSLOTS variable so that SGE will tell MPI how many processes to use. Finally, you'll need to use the path $TMPDIR/machines for your machines file to tell MPI to run on which nodes. That's a lot easier than it looks. You can request the environment and number of nodes to use either on the command line or in your job script. You can use -pe mpich <range> to qsub, or to embed that in your job script start a newline with "#$" and then type in the same arguments you would give to qsub. The "mpich" is the environment you are requesting. (That's also the only MPI environment we have right now.) The range can be a single number for a definite number of nodes or a string like "2-4,8,16" to specify you would like to use 2, 3, 4, 8, or 16 nodes depending on what's available to the system at that time. Let's look at an example that runs the popular Linpack benchmark. (You'll need your HPL.dat in your home directory.) Here's my job: [jjneely@beo-test jjneely]$ cat linpack.csh #!/bin/tcsh #$ -pe mpich 4 echo "I have $NSLOTS slots." /opt/mpich/ethernet/gcc/bin/mpirun -np $NSLOTS -machinefile $TMPDIR/machines /opt/hpl-eth/bin/xhpl [jjneely@beo-test jjneely]$ You can see how I've requested the environment and used the supplied variables as command line arguments to the mpirun program. That's pretty much it. Use qsub to submit the script and off it goes. One more detail to note with MPI. Our Beowulf uses MPICH's ethernet device for its communications layer and GCC for its compiler. So when running and building MPI programs you'll need to use the MPI scripts, binaries, and libraries found in /opt/mpich/ethernet/gcc as we did in the above job script. As compilers are added to the system you may find other directiories in /opt/mpich/ethernet that contain MPI tools specific to those compilers.

How do you see what the status of the jobs and queues are on the cluster? This is done with the qstat command. Running qstat without any options will list out any jobs that are currently running. If there is no output from the command then there are no jobs running. The man page for qstat explains all the options. [jjneely@beo-test jjneely]$ qstat job-ID prior name user state submit/start at queue master ja-task-ID --------------------------------------------------------------------------------------------- 8 0 linpack.csh jjneely t 02/11/2003 15:13:32 compute-0- SLAVE 8 0 linpack.csh jjneely t 02/11/2003 15:13:32 compute-0- MASTER 0 linpack.csh jjneely t 02/11/2003 15:13:32 compute-0- SLAVE 8 0 linpack.csh jjneely t 02/11/2003 15:13:32 compute-0- SLAVE 8 0 linpack.csh jjneely t 02/11/2003 15:13:32 compute-0- SLAVE [jjneely@beo-test jjneely]$ Some of the more useful options to qstat are -f for full output, and -u to specify a user. With these options you can get detailed information about your currently running jobs. Also, if you remember your job ID number you can use the -j option to see detailed information about just that job.

If you have problems use the PAMS Beowulf or just need more information about using the system please email us as pams_hpc@help.ncsu.edu.