More on ScaLAPACK Installation

Introduction

Documentation Tips

Setting Up

Compiling and Running the Code

Some Simple Trouble Shooting

Return to User Information

Return to Beowulf Home Page

Introduction

ScaLAPACK is a parallel implementation of the block-oriented LAPACK linear algebra routines. These FORTRAN libraries are installed on each node in /opt/SCALAPACK. Parallel routines are available to solve dense linear systems and to calculate eigenvalues for a variety of matrix types. The information in this page will describe some SCALAPACK example codes that we have modified, and which you can modify to solve additional linear algebra problems.

Documentation Tips

The official ScaLAPACK website has a lot of information but looking up something specific can be difficult. So here are some tips for using the page efficiently:

1. The ScaLAPACK routines have a systematic naming scheme which tells you alot about a function. For example PDDBSV is a double precision routine (PD) that solves a linear system (SV).

2. The ScaLAPACK routines are grouped into single precision, double precision, complex, complex16 categories at the bottom of the page. These links connect to an index of routines. So to see the code for PDDBSV click on the double precision link, and search for PDDBSV or to find single precision eigenvalue routines go to the single precision index and search on the key word eigenvalues.

3. The ScaLAPACK User's Guide is difficult to use, since the index and table of contents are not complete and often the active links don't have any contents. I found that doing a search on google was often an effect way to find information.

4.Finally, the site gives instructions for getting started with ScaLAPACK; Please note that all necessary libraries have already been installed on the cluster, so it is not necessary for you to compile any libraries in your home directory. Our installation of ScaLAPACK uses the ATLAS libraries installed on the cluster (these are optimized BLAS libraries).

Setting Up

We have made some modifications to existing ScaLAPACK example routines, that will allow you to get started with a minimal amount of time. To get your copy of these files on beowulf run this script on the master node. It will get the tar file from /data/tmp and create a new directory called $HOME/scalapack with everything you need. NOTE: If you decide to alter the name of your directory to something other than $HOME/scalapack you will need to change the variable TESTINGdir in the SLmake.inc file accordingly.

SLmake.inc : This defines the necessary variables for the Makefile and will be the same for any ScaLAPACK program that you write. It contains all the library and header information that is used by the Makefile when linking your program.

Makefile : This make file has targets that create the executable xsmsolve xsolve xeig. If you are new to makefiles you can easily modify this simple example to accommodate any ScaLAPACK program that you create.

pdlaread.f: One specified process reads a matrix stored as a text file and either writes the data to its own submatrix or broadcasts the data to the appropriate process. On completion each process in the process grid has a submatrix. The global matrix exists only as a collection of submatrices in the grid, in other words no process in the grid ever has the whole global matrix as defined in the file -- it is a 2D block cyclically distributed matrix.

pdlawrite.f : If a 2D block cyclically distributed matrix is stored on the processes in a process grid one can use pdlawrite to have one process receive the values of the matrix from the other process and write them to a file. The result is the whole global matrix goes into one file.

pdscaexinfo.f : This simple program uses one process in the grid to read the header information from HEADER.dat and broadcasts this information to all other processes in the grid.

solve.f : This code calls pdlaread and scaexinfo to get the data needed to solve a large square system. It solves the system with a call to pdgesv, and writes the answer to the file sol.dat with a call to pdlawrite. The answer is also checked and the result is reported in the log file. It also shows how to make ScaLAPACK calls to calculate norms and perform Matrix vector multiplications.

smallsolve.f : This code does the same things as solve.f, but it also writes the whole matrix A, b and x to the log file returned at runtime. Memory problems occur if this is used for large files.

eig.f : This code calls pdlaread and scaexinfo to get the Matrix. It finds some of the eigenvalues for a general dense matrix. It calls PDGEHRD to reduce a real general matrix A to upper Hessenberg form H by an orthogonal similarity transformation H = Q^T*A*Q. It then calculates the eigenvalues of H with a call to the subroutine PDGEHQR. PDGEHQR takes the upper Hessenberg form H of the real general matrix A and calculates H's eigenvalues which are the same as those of A. The eigenvalues are written to two files realeig.dat and imageig.dat where each file contains the respective parts of a complex vector;
eigenvalue = realeig + i*imageig.

ScaLAPACK documentation requires the block size (NB) >= 6 otherwise PDGEHQR doesn't work properly.

RHS.dat: This contains the write hand side of the system of equations. The first line contains the row and column size of the vector, each additional line contains one element from the vector.

MAT.dat: This contains the matrix of the linear system or the matrix whose eigenvalues are to be computed. The first line contains the row and column size of the Matrix, each additional line contains one element from the Matrix, organized a column at a time; the first n elements are the first column of A, the second n elements are the second column of A,....

HEADER.dat: A small file that contains basic information needed by all codes listed here, including the process grid info. The file format is self explanatory and an explanation of the input/output parameters follows:
N : The order of the matrix A.
NRHS : The number of right hand sides.
NB : The size of the square blocks the matrices A and B are split into.
P : The number of process rows.
Q : The number of process columns.

write.m: This is a simple matlab code that will take a matrix or vector from Matlab and write it to a file that is readable by pdlaread.f

Compiling and Running the Code

Compiling: make [eig, solve, smsolve]

Running: scasub -mpimon -np [process columns * process rows] [xeig, xsolve, xsmsolve]

Some Simple Trouble Shooting

Error Codes: When a ScaLAPACK routine returns the error code is stored in the variable INFO. The programs listed above print out this code in the log file for each of the major ScaLAPACK computations. INFO = 0 means the call was successfully completed, otherwise something is wrong.

For example you might get the message:

PDGESV para 605 illegal value

This mens that INFO = -605 = -(100*i) + j so i = 6 and j = 5 implies that there is a problem with the 5th element of the array DESCA which is the 6th argument passed to PDGESV.

Work Space: A segmentation violation and INFO = 0 indicates that one of the ScaLAPACK routines needs more workspace. A fast and reliable method is to have the routine do an LWORK Query.


Created February 21, 2003 by Janine Kennedy.