Running on Supermike
--------------------

SuperMike has 4 head nodes and 512 compute nodes. Hostnames for the head nodes
are mike1.lsu.edu  mike2.lsu.edu mike3.lsu.edu and mike4.lsu.edu
The head nodes  are identical. Access to the SuperMike cluster is via 
secure shell client (ssh) to one of the head nodes.

1. To allow communications among the nodes type:  
------------

cp -f /etc/skel/.ssh/known_hosts ~/.ssh/known_hosts


2. Compilers available include:
-------------------------------
gcc,g77 ,g++ (GNU)

pgcc,pgCC, pgf77,pgf90 (portland Group)

icc, ifc (Intel)
 
We, CACTUS users want to use intel.

3. MPI
------
There are three versions of MPICH compiled for each of
the compilers installed , GNU, Portland Group and 
Intel 

Depending on the version of MPICH you are going to use,
ionly one of the these three lines should be uncommented out
in your .bashrc file:
#export MPI_HOME=/usr/local/gmpi_pgi
#export MPI_HOME=/usr/local/gmpi_gcc
#export MPI_HOME=/usr/local/gmpi_intel

mpicc mpif77 mpif90 are the commands to compile your mpi programs.

We CACTUS users want to uncomment the third line.

4. PBSPro is our queuing system.
--------------------------------
Submitting a job to the PBS queue requires a PBS script.
 
A working script example using 32 nodes, two nodes per processor is:

#!/bin/bash
#PBS -m b
#PBS -m e
#PBS -q c32
#PBS -l nodes=32:ppn=2
#PBS -l walltime=24:00:00
#PBS -o <output-file>
#PBS -j oe
#PBS -N <job-name>
#PBS
#cd $PBS_O_WORKDIR
# date
cd <your exec dir>
#  a MPI 64 processors run
mpirun.ch_gm  -machinefile  $PBS_NODEFILE  -np 64 <executable> <par-file>
# date

you just need to fill with your need the names in < >'s.

5. Many  queues are configured:
------------------------------
Queue            Memory CPU Time Walltime Node   Run   Que   Lm  State
---------------- ------ -------- -------- ---- ----- ----- ----  -----
reserved_big       --      --       --     256     0     0    1   E R
workq              --      --    48:00:00    8     0     0    4   E R
c128_1             --      --    24:00:00  128     1     3    1   E R
c64                --      --    40:00:00   64     2     5    5   E R
c32                --      --    24:00:00   32     4     0    6   E R
c16                --      --    24:00:00   16    10     3   10   E R
fast               --      --    01:00:00    2     0     0   --   E R
glabtest           --      --       --       2     0     0    3   E R
c64_tmp            --      --    80:00:00   64     0     0   --   E R
                                               ----- -----
                                                  17    11

6. Running

You have to type in the command line:

qsub -q <queue-name> ./your-pbs-script

e.g qsub -q c16  ./your_pbs_script

will launch your job 

6. interactive runs
-------------------
you can use the fast queue for interactive runs:
a. qsub -q fast -I
b. gmpiconf > machines
c. mpirun.ch_gm -machinefile machines -np 2 ./executable parfile

7. COMPILING CACTUS
------------------

An options file would be the following, you can also use the cactus cpp:

CPP=/home1/tradke/bin/cpp
CC=icc
CXX=icc
F90=ifc
                                                                                                                                                   
HDF5=yes
HDF5_DIR=/exports/local/packages/hdf5-1.6.1-intel/serial
LIBSZ_DIR=/exports/local/packages/hdf5-1.6.1-intel/serial
                                                                                                                                                   
# cannot use MPI=MPICH because it needs another external library
# change MPI path according to the compilers used
MPI=CUSTOM
MPI_LIB_DIRS=/usr/local/gmpi_intel/lib /usr/gm/lib
MPI_INC_DIRS=/usr/local/gmpi_intel/include
MPI_LIBS=mpich gm