Compiling Cactus on Our Machines

NOTE: Please also see the page of machine configurations and architectures on the Cactus web pages:

Configurations

Architectures

The Cactus pages are supposed to be a subset of these pages, but they might get out of sync.

This page contains configuration options for the different machines we have access to. The best way to apply these options is to make a file config.<machine_name> containing these options and then configure Cactus using

      gmake <name>-config options=config.<machine_name>
     

Also contained here are RSL files for submitting jobs with globus

Please tell us about any mistakes, updates, enhancements you find!

Machines:

Aurora

sv1.npaci.edu

# NO MPI
ARFLAGS=ruc
INTEGER_PRECISION=8
      

Berte

berte.zib.de

# NATIVE MPI
MPI=NATIVE
HDF5_DIR=/usr/local/hdf5/devel
HDF5=yes
PTHREADS=yes
      

# MPICH-G2
GLOBUS_LIB_DIR=/usr/local/globus/1.1.4/development/alpha-cray-unicosmk2.0.5.X_nothreads_mpi_debug/lib
MPICH_DIR=/usr/local/mpich/1.2.1
MPICH_DEVICE=globus
MPI=MPICH
HDF5_DIR=/home/bvpp03th/install/hdf5
HDF5=yes
      

Bifrost

bifrost.aei.mpg.de

# MPICH shmem device
MPI=MPICH
MPICH_DIR=/usr/local/packages/mpich/ch_shmem
HDF5=yes

# MPICH p4 device
MPI=MPICH
MPICH_DIR=/usr/local/packages/mpich/ch_p4
HDF5=yes
      

Blue Horizon

horizon.sdsc.edu

# NATIVE MPI
MPI=NATIVE
HDF5=yes
      

# MPICH-G2 (NOT WORKING)
CC=/usr/local/apps/globus-1.1.4/mpich-g2-1.2.1/bin/mpicc
CXX=/usr/local/apps/globus-1.1.4/mpich-g2-1.2.1/bin/mpiCC
F90=/usr/local/apps/globus-1.1.4/mpich-g2-1.2.1/bin/mpif77
C_DEPEND_OUT=> /dev/null ; mv $(basename $(basename $@)).u $@
CXX_DEPEND_OUT=$(C_DEPEND_OUT)
MPI=MPICH
MPICH_DIR=/usr/local/apps/globus-1.1.4/mpich-g2-1.2.1
MPICH_DEVICE=globus
GLOBUS_LIB_DIR=/usr/local/apps/globus-1.1.4/development/rs6000-ibm-aix4.3.3.0_nothreads_mpi_debug/lib
HDF5=yes
HDF5_DIR=/paci/sdsc/ux452883/install/hdf5
      

Denali

denali.mcs.anl.gov

# NATIVE MPI
FILL ME IN!
      

# MPICH-G2
GLOBUS_LIB_DIR=/soft/pub/packages/globus/globus-1.1.4/development/mips-sgi-irix6.5-64_nothreads_standard/lib
MPICH_DIR=/usr/local/mpi
MPICH_DEVICE=globus
MPI=MPICH
HDF5_DIR=/homes/radke/install/hdf5
HDF5=yes
      

Golden

t3e.npaci.edu

# NATIVE MPI
MPI=NATIVE
      

Harpo

harpo.wustl.edu

# NATIVE MPI
IRIX_BITS=32
MPI=NATIVE 
HDF5_DIR=/usr/local/hdf5_dev
HDF5=yes
PTREADS=no
      

# MPICH-G2 
IRIX_BITS=32
GLOBUS_LIB_DIR=/usr/local/globus-1.1.4_n32-install/development/mips-sgi-irix6.5-n32_nothreads_mpi_debug/lib/
MPICH_DIR=/usr/local/mpich-g2/
MPICH_DEVICE=globus
MPI=MPICH 
HDF5_DIR=/usr/local/hdf5_dev
HDF5=yes
        

Notes:

Pthreads:

Native MPI on SGI's uses sproc threads which won't work together with Pthreads.
Do not configure with both MPI=NATIVE and PTHREADS !

Jalapeno

jalapeno.cactuscode.org

# MPICH WITH P4 DEVICE
HDF5=yes 
MPI=MPICH 
MPICH_DIR=/usr/local/mpich/
      

# MPICH WITH SHMEM DEVICE
HDF5=yes 
MPI=MPICH 
MPICH_DIR=/usr/local/mpich/ch_shmem
      

Lemieux

lemieux.psc.edu

CXX=cxx  
  
MPI=CUSTOM  
MPI_LIBS= mpi elan   
HDF5_DIR=/usr/users/1/tradke/install/hdf5  
HDF5=yes  
    
#for LAPACK set:     
LAPACK=yes 
LAPACK_LIBS = dxml  
LAPACK_DIR = /usr/include/cxx 
      

lin-dec

lin-dec.aei.mpg.de

F77=fort
F90=fort
CC=ccc
CXX=cxx
# also possible: gcc (version 2.95.4) or gcc-3.0 (version 3.0.4)
HDF5=yes
PTHREADS=yes
      

Lonestar

t3e-utexas.npaci.edu

# NATIVE MPI
MPI=NATIVE
      

Los Lobos

loslobos.alliance.unm.edu

# MPICH WITH MYRINET DEVICE
MYRINET_DIR=/usr/local/ll-gm
MPICH_DEVICE=ch_gm
MPICH_DIR=/usr/parallel/ll/mpich-gm.pgi
MPI=MPICH
F90=pgf90
CXX=pgCC
CC=pgcc
CXXFLAGS=-tused
HDF5_DIR=/users/tradke/install/hdf5
HDF5=yes
PTHREADS=yes
      

Notes:

qs2:

It is recommended to submit jobs with the qs2 script on loslobos, since it uses a special mpirun command, that delays the activation of the next node for a second, so that the communication channels can be set up properly.

.cshrc:

Make sure you have the following lines in your .cshrc (most if not all will probably already be there):

setenv MPIHOME /usr/parallel/ll/mpich-gm.pgi
setenv PATH $MPIHOME/bin:$PATH
setenv MANPATH $MPIHOME/man:$MANPATH
setenv GM_HOME /usr/local/myrinet
setenv PGI /usr/pgi
setenv LM_LICENSE_FILE 7496@sage.alliance.unm.edu
setenv PATH $PGI/linux86/bin:$PATH
setenv MANPATH $PGI/man:$MANPATH
setenv PATH /usr/local/pbs/bin:$PATH
setenv MANPATH /usr/local/pbs/man:$MANPATH

Modi4

modi4.ncsa.uiuc.edu

# NATIVE MPI
MPI=NATIVE
HDF5=yes
HDF5_DIR=/u/ac/tradke/install/hdf5
PTHREADS=no

# MPICH with shared memory device
MPI=MPICH
MPICH_DIR=/u/ac/tradke/install/mpich
HDF5=yes
HDF5_DIR=/u/ac/tradke/install/hdf5
PTHREADS=yes

# MPICH-G2
MPI=MPICH
MPICH_DIR=/usr/local/MPICH-G2/mips-sgi-irix6.5-64_nothreads_mpi
MPICH_DEVICE=globus
GLOBUS_LIB_DIR=/usr/local/globus-install-1.1.4/development/mips-sgi-irix6.5-64_nothreads_mpi/lib
CC=/usr/local/MPICH-G2/mips-sgi-irix6.5-64_nothreads_mpi/bin/mpicc
CXX=/usr/local/MPICH-G2/mips-sgi-irix6.5-64_nothreads_mpi/bin/mpiCC
LD=/usr/local/MPICH-G2/mips-sgi-irix6.5-64_nothreads_mpi/bin/mpiCC
F90=f90
HDF5=yes
HDF5_DIR=/u/ac/tradke/install/hdf5
      

Notes:

Pthreads:

Native MPI on SGI's uses sproc threads which won't work together with Pthreads.
Do not configure with both MPI=NATIVE and PTHREADS !

NCSA NT Supercluster

FILL ME IN!

# HPVM
MPI=CUSTOM 
MPI_LIBS="mpi.lib fm.lib advapi32.lib kernal32.lib wsock32.lib" 
MPI_INC_DIRS="d:/apps/hpvm/include"
      

Origin

origin.aei.mpg.de

# NATIVE MPI
MPI=NATIVE
HDF5=yes
PTHREADS=no
# ports available for HTTPD and data streaming: 7000 .. 8000

# MPICH with shared memory device
MPI=MPICH
MPICH_DEVICE=ch_shmem
HDF5=yes
PTHREADS=yes
# ports available for HTTPD and data streaming: 7000 .. 8000

# MPICH-G2
GLOBUS_LIB_DIR=/opt/globus/deploy/lib
MPICH_DIR=/opt/mpich-1.2.1
MPICH_DEVICE=globus
MPI=MPICH
HDF5=yes

#to use LAPACK set:
LAPACK=yes
LAPACK_DIR=none
LAPACK_LIBS=complib.sgimath

      

Notes:

MPICH

If you use MPICH use mpirun from /usr/local/apps/mpich/bin/mpirun.ch_shmem

Pthreads:

Native MPI on SGI's uses sproc threads which won't work together with Pthreads.
Do not configure with both MPI=NATIVE and PTHREADS !

PETSc:

PETSC_DIR /usr/local/apps/petsc-2.0.24
PETSC_ARCH IRIX64
      

Platinum (intel compilers)

platinum.ncsa.uiuc.edu

CPP=$${PERL} $${CCTK_HOME}/lib/sbin/cpp.pl

# Intel compilers:
CXX=icpc
CC=icc
F77=ifc
F90=ifc

# native MPI:
MPI=CUSTOM
MPI_LIBS=mpich vmi pthread dl
MPI_LIB_DIRS=/usr/local/vmi/mpich/lib/icc
MPI_INC_DIRS=/usr/local/vmi/mpich/include

# HDF5:
HDF5=yes
HDF5_DIR=/usr/apps/hdf5/5-1.4.4-intel

PTHREADS=yes

# -O1 can give better performance for Fortran:
C_OPTIMISE_FLAGS=-O2
CXX_OPTIMISE_FLAGS=-O2
F90_OPTIMISE_FLAGS=-O1 -override_limits
F77_OPTIMISE_FLAGS=-O1 -override_limits
      

PC

pc.rzg.mpg.de

MPI=NATIVE
HDF5=yes
HDF5_DIR=/u/tradke/install/hdf5
PTHREADS=yes
      

POSIC IA32 Linux Cluster

posic.ncsa.uiuc.edu

MPI=CUSTOM
MPI_INC_DIRS=/usr/local/mpich/include/
MPI_LIB_DIRS=/usr/local/mpich/lib/
MPI_LIBS="mpich vmi dl pthread"
HDF5=yes
      

Quad

quad.mcs.anl.gov

# NATIVE MPI
MPI=NATIVE
HDF5=yes
HDF5_DIR=/homes/radke/install/hdf5
      

Regatta (Garching)

psi.rzg.mpg.de

# NATIVE MPI
MPI=NATIVE
HDF5=yes
HDF5_DIR=/u/tradke/install/hdf5
PTHREADS=yes
CFLAGS=-q64
F90FLAGS=-q64 -qmaxmem=-1
CXXFLAGS=-q64

#for LAPACK set:
LAPACK=yes
LAPACK_DIR=/afs/rzg/@sys/lib
LAPACK_LIBS="lapack-essl essl f"
      

Road Runner

rr.alliance.unm.edu

# MPICH WITH MYRINET DEVICE
MYRINET_DIR=/usr/local/rr-gm
MPICH_DEVICE=ch_gm
MPICH_DIR=/usr/parallel/rr/mpich-gm.pgi
MPI=MPICH
F90=pgf90
CXX=pgCC
CC=pgcc
CXXFLAGS=-tused
HDF5_DIR=/users/tradke/install/hdf5
HDF5=yes
PTHREADS=yes
      

sp-umich

sp-umich.npaci.edu

# NATIVE MPI
MPI=NATIVE
      

SR8000

sr8000.lrz-muenchen.de

# NATIVE MPI
MPI=NATIVE
HDF5=yes
PTHREADS=yes
# ports available for HTTPD and data streaming: 1030 .. 1040
#
# In order to use the standard fortran output format for floating point numbers
# in scientific notation you need to set the environment variable
# HF_90OPTS to '-Fport(realedt)'.
# Otherwise you might get spurios spaces between the mantissa and the exponent
# which can confuse postprocessing tools.
      

t90

t90.npaci.edu

FILL ME IN!
      

Titan

titan.ncsa.uiuc.edu

CPP=$${PERL} $${CCTK_HOME}/lib/sbin/cpp.pl

# Intel compilers:
CXX=ecpc
CC=ecc
F77=efc
F90=efc

# SGI compilers:
#CXX=sgicc
#CC=sgicc
#F90=sgif90

# native MPI:
MPI=CUSTOM
MPI_LIBS=mpich vmi pthread dl
MPI_LIB_DIRS=/usr/local/vmi/mpich/lib/intel
MPI_INC_DIRS=/usr/local/vmi/mpich/include

# mpich MPI:
#MPI=MPICH
#MPICH_DEVICE=ch_shmem

# HDF5:
HDF5=yes
HDF5_DIR=/u/ncsa/tradke/install/hdf5

PTHREADS=yes

# -O1 can give better performance for Fortran:
C_OPTIMISE_FLAGS=-O2
CXX_OPTIMISE_FLAGS=-O2
F90_OPTIMISE_FLAGS=-O1 -override_limits
F77_OPTIMISE_FLAGS=-O1 -override_limits
      

Ufer

ufer.zib.de

# MPICH WITH P4 DEVICE
MPI=CUSTOM 
MPI_LIB_DIRS=/client/lib/solaris/ch_p4 
MPI_INC_DIRS=/client/include 
MPI_LIBS=mpich 
CC=/opt/SUNWspro/bin/cc 
CXX=CC 
OPTIMISE=no
      

Xeon workstations at AEI

xeonXX.aei.mpg.de

# Using the Intel compilers:
# (Note, for quicker compiles, set:
#    F77_OPTIMISE_FLAGS=-O0
#    F90_OPTIMISE_FLAGS=-O0
# )

F77=ifc
F90=ifc
CC=icc
CXX=icc
# also possible: gcc (version 2.96) or gcc3 (version 3.0.2)
HDF5=yes
MPI=LAM
# also possible: MPI=MPICH MPICH_DEVICE=ch_shmem
PTHREADS=yes

# to use LAPACK set:
LAPACK=yes
LAPACK_EXTRA_LIBS=g2c
LAPACK_EXTRA_LIB_DIRS=/usr/lib/gcc-lib/i386-redhat-linux/2.96
      

Supermike Linux Cluster at LSU

supermike.lsu.edu

# NOTICE THAT HDF5 IS NOT YET SORTED OUT
F77=ifc
F90=ifc
CC=mpicc
CXX=mpicc
CPP=$${PERL} $${CCTK_HOME}/lib/sbin/cpp.pl
#F77_OPTIMISE_FLAGS=-O2
#F90_OPTIMISE_FLAGS=-O2
#HDF5=yes
#HDF5_DIR=/usr/local/apps/hdf5
MPI=MPICH
MPICH_DIR=/usr/local/gmpi_intel
MPI_DEVICE=ch_gm
MVRINET_DIR=/usr/gm
PTHREADS=yes
      

This page last modified: $Date: 2004/02/19 18:35:52 $