How to use the PaStiX solver in Kratos

The PaStiX (Parallel Sparse matriX package) is a open-source scientific library that provides a high performance parallel solver for very large sparse linear systems based on direct methods. Numerical algorithms are implemented in single or double precision (real or complex) using LLt, LDLt and LU with static pivoting (for non symmetric matrices having a symmetric pattern). This solver provides also an adaptive blockwise iLU(k) factorization that can be used as a parallel preconditioner using approximated supernodes to build a coarser block structure of the incomplete factors ([1]).

It can be linked to the Kratos by the ExternalSolversApplication. The following steps have to be made in order to use be able to use it in Ubuntu version 14.04 LTS 64 bit or 16.04 LTS 64 bit.

Compile the Scotch library (shared memory parallelism)

PaStiX needs the Scotch library as a prerequisite to work.

You can download version 6.0.4 here:

http://gforge.inria.fr/projects/scotch/

Copy it to a folder of your choice. In the following we are using "~/compiled_libraries" as a reference.

Extract the files by the command:

tar -xf scotch_6.0.4.tar.gz

Navigate into the source directory by:

cd scotch_6.0.4/src

Copy the correct MakeFile.inc for the version of your system to the source directory. Ín the case of the Ubuntu versions mentioned above it is the Makefile.inc.x86-64_linux2:

cp ./Make.inc/Makefile.inc.x86-64_linux2 Makefile.inc

Additional libraries are necessary, which can be installed by:

sudo apt-get install flex bison zlib1g-dev

In order to install Scotch locally the install path has to be set. Since the "prefix" command is not working this has to be done manually in the Makefile, which is also located in the /src folder. You can see an example for the part of the file to be modified below:

...some code...
include Makefile.inc
prefix		?= ~/software/scotch_6.0.4-install
bindir		?= $(prefix)/bin
includedir	?= $(prefix)/include
libdir		?= $(prefix)/lib
datarootdir	?= $(prefix)/share
mandir		?= $(datarootdir)/man
...some code...

Now Scotch can be compiled by (we are still in the /src directory):

make scotch

And can be installed to the directory specified in the MakeFile by:

make install

Compile the PaStiX library (shared memory parallelism)

Download the PaStiX version 5.2.3 from:

https://gforge.inria.fr/frs/?group_id=186

Copy it to a folder of your choice. In the following we are using "~/compiled_libraries" as a reference.

Extract the files by the command:

tar -xf pastix_5.2.3.tar.bz2

Navigate into the source directory by:

cd pastix_5.2.3/src

The different config files can be found in the confi folder. You have to copy the one fitting to your system to the /src folder. In the case of the systems mentioned above it ist the LINUX-GNU.in file. Copy it to the source directory as config.in by:

cp ./config/LINUX-GNU.in config.in

You have to edit or uncomment the following blocks of the config.in file:

###################################################################
#                          INTEGER TYPE                           #
###################################################################
# Uncomment the following lines for integer type support (Only 1)
#VERSIONINT  = _long
#CCTYPES     = -DFORCE_LONG -DINTSIZELONG
#---------------------------
VERSIONINT  = _int32
CCTYPES     = -DINTSIZE32
#---------------------------
#VERSIONINT  = _int64
#CCTYPES     = -DINTSSIZE64

###################################################################
#                          MPI/THREADS                            #
###################################################################
# Uncomment the following lines for sequential (NOMPI) version
VERSIONMPI  = _nompi
CCTYPES    := $(CCTYPES) -DFORCE_NOMPI
MPCCPROG    = $(CCPROG)
MCFPROG     = $(CFPROG)
MPCXXPROG   = $(CXXPROG)
...some code...
#CCPASTIX   := $(CCPASTIX) -DCUDA_SM_VERSION=20
#NVCCOPT    := $(NVCCOPT) -arch sm_20

###################################################################
#                      GRAPH PARTITIONING                         #
###################################################################
# Uncomment the following lines for using metis ordering
#VERSIONORD  = _metis
#METIS_HOME  = ${HOME}/metis-4.0
#CCPASTIX   := $(CCPASTIX) -DMETIS -I$(METIS_HOME)/Lib
#EXTRALIB   := $(EXTRALIB) -L$(METIS_HOME) -lmetis
# Scotch always needed to compile
SCOTCH_HOME = ${HOME}/compiled_libraries/scotch_6.0.4
SCOTCH_INC = $(SCOTCH_HOME)/include
SCOTCH_LIB = $(SCOTCH_HOME)/lib
# Uncomment on of this blocks
#scotch
CCPASTIX   := $(CCPASTIX) -I$(SCOTCH_INC) -DWITH_SCOTCH
EXTRALIB   := $(EXTRALIB) -L$(SCOTCH_LIB) -lscotch -lscotcherrexit
#ptscotch
#CCPASTIX   := $(CCPASTIX) -I$(SCOTCH_INC) -DDISTRIBUTED -DWITH_SCOTCH
#if scotch >= 6.0
EXTRALIB   := $(EXTRALIB) -L$(SCOTCH_LIB) -lptscotch -lscotch -lptscotcherrexit
#else
#EXTRALIB   := $(EXTRALIB) -L$(SCOTCH_LIB) -lptscotch -lptscotcherrexit

###################################################################
#                Portable Hardware Locality                       #
###################################################################
# By default PaStiX uses hwloc to bind threads,
# comment this lines if you don't want it (not recommended)
#HWLOC_HOME = ~/compiled_libraries/hwloc-1.11.5 #/opt/hwloc/
#HWLOC_INC  = $(HWLOC_HOME)/include
#HWLOC_LIB  = $(HWLOC_HOME)/lib
#CCPASTIX   := $(CCPASTIX) -I$(HWLOC_INC) -DWITH_HWLOC
#EXTRALIB   := $(EXTRALIB) -L$(HWLOC_LIB) -lhwloc