Berkeley UPC and Intrepid GCCUPC
Cray XT3 Special Release, September 5 2006
Compiler installation instructions
----------------------------------
This source archive contains everything required to build the Berkeley UPC
compiler and Intrepid GCCUPC+UPCR compiler on the Cray XT3, and consists of
the following components:
* berkeley_upc-2.3.16.tar.gz: Berkeley UPC Runtime (UPCR) v2.3.16 source
Beta release of version 2.4 of the Berkeley UPC runtime.
Includes XT3-specific fixes relative to the 2.2.2 public release,
and XT3 cross-compilation support.
Intended for release to "Cray-friendly" XT3 customers ONLY.
* berkeley_upc_translator-2.2.4.tar.gz: Berkeley UPC translator v2.2.4 source
Also available from http://upc.lbl.gov/download/beta
* upc-3_4_6_1-cray-xt3.tar.gz: GCCUPC compiler source (version 3 branch)
Special preview release version of GCCUPC for the XT3
* README.CrayXT3: These instructions
===============================================================================
SELECTING A CONFIGURATION TO BUILD
There are two different configurations for using Berkeley UPC on the Cray XT3:
1) 'BUPC+UPCR': the Berkeley runtime is used with our UPC-to-C translator.
2) 'GCCUPC+UPCR': the Berkeley runtime is used with the GCCUPC binary
UPC compiler.
Instructions for each are provided.
None of this software requires root permissions to install. The only
requirement is that the installing user have write permissions to the install
directories specified by the prefix options described in the instructions
below. We do *not* recommend installing these products directly into a prefix
of an existing directory (eg / or /usr), as that might inadvertently overwrite
system-specific files. We recommend creating an empty directory owned by an
unprivileged user for each product to be installed (eg
/usr/local/upc/bupc_translator), and perform the install operations as that
user, targeting that directory as the install --prefix.
CrayXT3 login nodes use a shared, read-only root that can only be written to
from the boot node; the software included in this package should *not* be
installed on the shared root; it should be installed on some other file system,
available from login nodes, such as /ptmp, or /usr/local, assuming they do not
exist as part of the shared root. Also, it's a good idea to do the install
under a user ID other than root.
===============================================================================
BUPC+UPCR: Instructions for the Berkeley UPC compiler
===============================================================================
Berkeley UPC can be configured to use the public UPC-to-C translation server
provided at http:///upc.lbl.gov (the default), or you can build the Berkeley UPC
translator to run on the XT3 frontend (login) node and use that locally (recommended for
compiling classified UPC application code).
1. Build the Berkeley UPC translator: (if desired)
* unpack the translator source tarball:
gunzip -c berkeley_upc_translator-2.2.4.tar.gz | tar xvpf -
* cd into the source directory:
cd berkeley_upc_translator-2.2.4/
* build the translator, using the *FRONTEND* (login node) gcc compiler
(CC and CXX MUST reference the gcc for the login host,
NOT the cross-compiling gcc for compute-node targets):
gmake CC=/usr/bin/gcc CXX=/usr/bin/g++
* install the translator to an appropriate empty directory:
gmake install PREFIX=/prefix/to/use/for/bupc
* make note of the translator path printed at the end of the last step, eg:
"To use this translator LOCALLY, insert the following pathname:
/prefix/you/chose/targ
..."
you'll need that path below.
2. Build the Berkeley UPC runtime:
* decide whether to use PrgEnv gnu or pgi, and load the appropriate module:
module unload PrgEnv-pgi
module load PrgEnv-gnu
BUPC+UPCR supports either PrgEnv when using mpi-conduit, but it must be
configured, built AND used all with the same PrgEnv loaded.
The new portals-conduit which provides native communication services ONLY
supports PrgEnv-gnu (due to limitations in Cray Portals), therefore it
must be configured, built AND used all with PrgEnv-gnu loaded.
* unpack the runtime source tarball:
gunzip -c berkeley_upc-2.3.16.tar.gz | tar xvpf -
* symlink the cross-configure script into the top-level of the source tree:
cd berkeley_upc-2.3.16
ln -s gasnet/other/contrib/cross-configure-crayxt3 .
* open the cross-configure script in an editor:
vi cross-configure-crayxt3
inspect the settings at the top for accuracy:
- CC and MPI_CC should both be set to the compute-node MPI compiler
to find the MPI compiler, enter 'which mpicc'
- HOST_CC should be set to the frontend gcc compiler (/usr/bin/gcc)
- MPIRUN_CMD might need tweaking to match on your job system setup
- The C++ compiler setting is ignored on the XT3
* create a build directory, and cd into it:
cd ..
mkdir bld && cd bld
* select optional configure options. Many options are available: a complete
list is available with:
../berkeley_upc-2.3.16/configure --help
Here are several potentially useful options to consider:
--with-translator=/prefix/you/chose/targ
(change default UPC translator to the one built in step 1)
--enable-sptr-struct Use the 128-bit shared pointer representation
(currently required for runs using > 1024 nodes)
--enable-debug Build UPC Runtime/GASNet in global debug mode
(globally disables optimization and enables extra sanity checks)
* configure the runtime using the cross-configure script:
../berkeley_upc-2.3.16/cross-configure-crayxt3 \
--prefix=/path/to/install/runtime ...any optional settings...
--prefix is the path where the runtime will be installed, which includes the upcc
compilation driver script and upcrun job launch script that end-users invoke.
* build the runtime:
gmake
* test the compiler:
gmake tests-local
./upcrun -np 2 ./libupcr-mpi-seq-test (may need to be in a PBS session)
./upcrun -np 2 ./libupcr-portals-seq-test (only available if portals-conduit enabled)
* install the runtime:
gmake install
the compiler should now be ready for use in: $prefix/bin/upcc
for documentation, see: $prefix/man and http://upc.lbl.gov
* run comprehensive correctness tests (optional):
- check the job queue settings in tester configuration file:
../berkeley_upc-2.3.16/harness/guppy
and ensure they match the job system for your machine
- run the test harness - note this builds and runs hundreds of tests,
and takes a *long* time:
cd harness/ && harness -sysconf=guppy
See 'harness -help' for additional options
for GCCUPC+UPCR, include option: -compiler_spec=gcc-upc-3-upcr.spec
- check the test results, during and after the run:
cd harness && checkfail -u
there are a handful of expected failures on the XT3,
but most tests should pass
===============================================================================
GCCUPC+UPCR: Instructions for Intrepid GCCUPC w/ Berkeley UPC Runtime:
===============================================================================
This configuration utilizes the Intrepid GCCUPC v3 compiler which runs on the
frontend and compiles UPC code to binary object code with calls to the runtime
system. This is linked with the Berkeley UPC runtime library and GASNet, which
handle all cross-node communication.
1. Build the GCCUPC compiler:
* unpack the translator source tarball:
gunzip -c upc-3_4_6_1-cray-xt3.tar.gz | tar xvpf -
* cd into the unpacked directory:
cd upc-3_4_6_1-cray-xt3/
* create a build directory, and cd into it:
mkdir bld && cd bld
* Set environment variable CC to ensure the host gcc is used for
config/build process:
setenv CC /usr/bin/gcc (csh shells)
CC=/usr/bin/gcc ; export CC (bourne shells)
* configure the compiler:
../configure --prefix=/path/to/install/gccupc --disable-multilib
* build the compiler:
gmake
* install the compiler:
gmake install
2. Build the Berkeley UPC Runtime, configured for use with GCCUPC:
* Load the PrgEnv-gnu module:
module unload PrgEnv-pgi
module load PrgEnv-gnu
GCCUPC+UPCR does *not* support PrgEnv-pgi. It must be be configured, built
AND used with PrgEnv-gnu loaded.
* Follow the instructions above in BUPC+UPCR - Step 2 for configuring,
building and installing the Berkeley UPC runtime.
During the configure step, be sure to provide configure with the full path
to the 'upc' compiler executable created in the previous step:
--with-gccupc=/path/to/install/gccupc/bin/upc
Also, as in the instructions for BUPC+UPCR, be sure to specify --prefix, the
full path for installing the UPCR executables, upcc (the script invoked by
end-users to compile all UPC programs, even those using GCCUPC+UPCR)
and upcrun (yod wrapper for launching UPC programs) that will be created with
'gmake', and 'gmake install':
--prefix=/path/to/use/for/UPCR/executables
We recommend this prefix be a separate, empty directory, ie not the same
directory as /path/to/install/gccupc.
Do not use --with-translator or --enable-sptr-struct with GCCUPC.
===============================================================================
Known Issues and limitations:
===============================================================================
* BUPC+UPCR supports either PrgEnv-gnu or PrgEnv-pgi, but the runtime component
must be configured, built and used with all the same setting (otherwise
non-obvious malfunctions can occur). The best solution is probably to build
and install two separate copies of the runtime, one for each PrgEnv, then
setup modules to select the correct one automatically when PrgEnv is loaded
(the translator component is insensitive to PrgEnv).
* GCCUPC+UPCR currently supports only PrgEnv-gnu, and must be configured, built
and used with PrgEnv-gnu loaded. Similarly, use of the new portals-conduit
communication layer also requires PrgEnv-gnu. Modules should probably be
setup to disallow loading a GCCUPC+UPCR module (or BUPC+UPCR module
w/portals-conduit support) under PrgEnv-pgi.
* Harmless compiler warnings of the form:
gcc: -lpapi: linker input file unused because linking not done
gcc: -lperfctr: linker input file unused because linking not done
can be avoided by unloading the papi module.
portals-conduit:
---------------
* portals-conduit is a new native implementation of GASNet (and hence UPC)
targeting Portals which delivers improved communication performance on the
XT3. The current release includes only an initial prototype of
portals-conduit, where some of the less performance-critical services are
still implemented using portions of mpi-conduit.
* This preview release of portals-conduit is only supported for use on XT3/Catamount
compute nodes with Berkeley UPCR (Berkeley UPC or GCC/UPC+UPCR).
Support for additional operating systems and GASNet clients is planned.
* See gasnet/portals-conduit/README and gasnet/README in the UPCR distribution
for information on environment variables that can be used to tweak the runtime
behavior of portals-conduit and GASNet.
mpi-conduit:
-----------
* mpi-conduit is a fully portable implementation of GASNet that implementes
all communication services needed by UPC using a one-sided communication
layer built on MPI. Due to limitations of MPI, UPC communication performance is
non-optimal when using mpi-conduit. XT3 users are advised to use the native
portals-conduit for improved communication performance.
* The lack of robust backpressure in the MPI implementation can lead to crashes
under certain heavy communication patterns for mpi-conduit. The crashes look like the
following:
MPIDI_PortalsU_Request_PUPE(334): exhausted unexpected receive queue
buffering increase via env. var. MPICH_UNEX_BUFFER_SIZE
- or -
[0] MPIDI_Portals_Progress: dropped event on unexpected receive queue, increase
[0] queue size by setting the environment variable MPICH_PTL_UNEX_EVENTS
In addition to the workarounds suggested in the error messages, increasing
GASNET_NETWORKDEPTH or decreasing AMMPI_CREDITS_PP may also help, although
none of these provide a foolproof solution. The native portals-conduit
implementation currently under development is intended to avoid these problems
by design (although the current beta version might theoretically still
encounter these errors in rare cases).
* See gasnet/mpi-conduit/README and gasnet/README in the UPCR distribution
for information on environment variables that can be used to tweak the runtime
behavior of mpi-conduit and GASNet.