How to use the PaStiX solver in Kratos

From KratosWiki
(Difference between revisions)
Jump to: navigation, search
(Compile the Scotch library)
(Compile the PaStiX library (shared memory parallelism))
 
(18 intermediate revisions by one user not shown)
Line 5: Line 5:
 
It can be linked to the Kratos by the '''ExternalSolversApplication'''. The following steps have to be made in order to use be able to use it in Ubuntu version 14.04 LTS 64 bit or 16.04 LTS 64 bit.
 
It can be linked to the Kratos by the '''ExternalSolversApplication'''. The following steps have to be made in order to use be able to use it in Ubuntu version 14.04 LTS 64 bit or 16.04 LTS 64 bit.
  
== Compile the Scotch library ==
+
== Compile the Scotch library (shared memory parallelism)==
  
 
PaStiX needs the Scotch library as a prerequisite to work.
 
PaStiX needs the Scotch library as a prerequisite to work.
Line 31: Line 31:
 
  sudo apt-get install flex bison zlib1g-dev
 
  sudo apt-get install flex bison zlib1g-dev
  
In order to install scotch locally the install path has to be set. Since the "prefix" command is not working this has to be done manually in the '''Makefile''', which is also located in the /src folder. You can see an example for the part of the file to be modified below:
+
In order to install Scotch locally the install path has to be set. Since the "prefix" command is not working this has to be done manually in the '''Makefile''', which is also located in the /src folder. You can see an example for the part of the file to be modified below:
  
 
  ''...some code...''
 
  ''...some code...''
Line 43: Line 43:
 
  ''...some code...''
 
  ''...some code...''
 
   
 
   
Now scotch can be compiled by (we are still in the /src directory):
+
Now Scotch can be compiled by (we are still in the /src directory):
  
 
  make scotch
 
  make scotch
Line 51: Line 51:
 
  make install
 
  make install
  
== Compile the PaStiX library ==
+
== Compile the PaStiX library (shared memory parallelism)==
  
== Export your variables ==
+
Download the PaStiX version 5.2.3 from:
  
In order to make your variables visible, you will need to change its declaration with the define name from the step 1.
+
https://gforge.inria.fr/frs/?group_id=186
We provide a couple of replacements for do that:
+
  
KRATOS_DEFINE_VARIABLE
+
Copy it to a folder of your choice. In the following we are using "~/compiled_libraries" as a reference.
KRATOS_DEFINE_3D_VARIABLE_WITH_COMPONENTS
+
  
will become:
+
Extract the files by the command:  
  
  KRATOS_DEFINE_APPLICATION_VARIABLE
+
  tar -xf pastix_5.2.3.tar.bz2
KRATOS_DEFINE_3D_APPLICATION_VARIABLE_WITH_COMPONENTS
+
  
For example:
+
Navigate into the source directory by:  
 +
 +
cd pastix_5.2.3/src
  
KRATOS_DEFINE_VARIABLE(int, VAR_1)
+
The different config files can be found in the config folder. You have to copy the one fitting to your system to the /src folder. In the case of the systems mentioned above it ist the '''LINUX-GNU.in''' file. Copy it to the source directory as '''config.in''' by:
  
will become
+
cp ./config/LINUX-GNU.in config.in
  
KRATOS_DEFINE_APPLICATION_VARIABLE(FOO_APPLICATION, int, VAR_1)
+
You have to edit the following blocks of the '''config.in''' file:
  
== Export your classes ==
+
###################################################################
 +
#                  SETTING INSTALL DIRECTORIES                    #
 +
###################################################################
 +
<span style="color:red">ROOT          = /your/pastix/installation/path</span>
 +
INCLUDEDIR    = ${ROOT}/include
 +
LIBDIR        = ${ROOT}/lib
 +
BINDIR        = ${ROOT}/bin
 +
PYTHON_PREFIX = ${ROOT}
  
Finally you will also need to indicate which classes are available to be used from other applications.
+
###################################################################
In order to do it you will need to use the ''KRATOS_API()'' macro with the define name of your applications
+
  #                          INTEGER TYPE                          #
in the declaration of the class.
+
###################################################################
 +
# Uncomment the following lines for integer type support (Only 1)
 +
#VERSIONINT  = _long
 +
#CCTYPES    = -DFORCE_LONG -DINTSIZELONG
 +
#---------------------------
 +
<span style="color:red">VERSIONINT  = _int32
 +
CCTYPES    = -DINTSIZE32</span>
 +
#---------------------------
 +
#VERSIONINT  = _int64
 +
#CCTYPES    = -DINTSSIZE64
  
For example
+
###################################################################
 +
#                          MPI/THREADS                            #
 +
###################################################################
 +
# Uncomment the following lines for sequential (NOMPI) version
 +
<span style="color:red">VERSIONMPI  = _nompi
 +
CCTYPES    := $(CCTYPES) -DFORCE_NOMPI
 +
MPCCPROG    = $(CCPROG)
 +
MCFPROG    = $(CFPROG)
 +
MPCXXPROG  = $(CXXPROG)</span>
 +
''...some code...''
 +
<span style="color:red">#CCPASTIX  := $(CCPASTIX) -DCUDA_SM_VERSION=20
 +
#NVCCOPT    := $(NVCCOPT) -arch sm_20</span>
  
   class myclass() {
+
###################################################################
   }
+
#                      GRAPH PARTITIONING                        #
 +
###################################################################
 +
# Uncomment the following lines for using metis ordering
 +
#VERSIONORD  = _metis
 +
#METIS_HOME  = ${HOME}/metis-4.0
 +
#CCPASTIX   := $(CCPASTIX) -DMETIS -I$(METIS_HOME)/Lib
 +
#EXTRALIB   := $(EXTRALIB) -L$(METIS_HOME) -lmetis
 +
# Scotch always needed to compile
 +
<span style="color:red">SCOTCH_HOME = ${HOME}/compiled_libraries/scotch_6.0.4
 +
SCOTCH_INC = $(SCOTCH_HOME)/include
 +
SCOTCH_LIB = $(SCOTCH_HOME)/lib</span>
 +
# Uncomment on of this blocks
 +
#scotch
 +
<span style="color:red">CCPASTIX  := $(CCPASTIX) -I$(SCOTCH_INC) -DWITH_SCOTCH
 +
EXTRALIB  := $(EXTRALIB) -L$(SCOTCH_LIB) -lscotch -lscotcherrexit</span>
 +
#ptscotch
 +
<span style="color:red">#CCPASTIX  := $(CCPASTIX) -I$(SCOTCH_INC) -DDISTRIBUTED -DWITH_SCOTCH</span>
 +
#if scotch >= 6.0
 +
EXTRALIB  := $(EXTRALIB) -L$(SCOTCH_LIB) -lptscotch -lscotch -lptscotcherrexit
 +
#else
 +
#EXTRALIB  := $(EXTRALIB) -L$(SCOTCH_LIB) -lptscotch -lptscotcherrexit
  
will become:
+
###################################################################
 +
#                Portable Hardware Locality                      #
 +
###################################################################
 +
# By default PaStiX uses hwloc to bind threads,
 +
# comment this lines if you don't want it (not recommended)
 +
<span style="color:red">#HWLOC_HOME = ~/compiled_libraries/hwloc-1.11.5 #/opt/hwloc/
 +
#HWLOC_INC  = $(HWLOC_HOME)/include
 +
#HWLOC_LIB  = $(HWLOC_HOME)/lib
 +
#CCPASTIX  := $(CCPASTIX) -I$(HWLOC_INC) -DWITH_HWLOC
 +
#EXTRALIB  := $(EXTRALIB) -L$(HWLOC_LIB) -lhwloc</span>
  
  class KRATOS_API(FOO_APPLICATION) myclass() {
+
Now you can compile the PaStiX library (we are still in the /src directory):
  }
+
 
 +
make
 +
 
 +
You can install it to the directory defined in the config.in file by:
 +
 
 +
make install
  
If your class depends on a template parameter, you will also need to instantiate such class, as no code exists in compile time and otherwise will be impossible to export:
+
In order to be able to compile the Kratos with PaStiX now, you have to fix a little bug in one of the header files in the installation directory. In the header file '''pastix_int32_double_real.h''' you have to add:
  
 +
#include <stdint.h>
  
For instance, if you want to use class ''myclass'' with '''double''':
+
== Compile the Kratos with PaStiX (shared memory parallelism)==
  
  temaplte<class T>
+
Finally we can compile the Kratos with the PaStiX library. In order to do this, you have to add the following lines to you configure.sh file:
  class myclass() {
+
    ...
+
  };
+
  
becomes
+
-DINCLUDE_PASTIX=ON                                      \
 +
-DPASTIX_INSTALL_DIR= "/path/to/your/pastix/installation" \ (e.g. "~/software/pastix_5.2.3-install/install")
 +
-DSCOTCH_INSTALL_DIR= "/path/to/your/scotch/installation" \ (e.g. "~/software/scotch_6.0.4-install/lib")
  
  temaplte<class T>
+
You can now use the direct and iterative PaStiX solver by adding the following block to your '''ProjectParameters.json''' file:
  class KRATOS_API(FOO_APPLICATION) myclass() {
+
"solver_type": "PastixSolver",
    ...
+
"solution_method": "Direct",
  };
+
"tolerance": 0.000001,
 
+
"max_iteration": 100,
  ...
+
"gmres_krylov_space_dimension": 100,
 
+
"ilu_level_of_fill": 1,
  template class KRATOS_API(FOO_APPLICATION) myclass<double>;
+
"is_symmetric": false,
 +
"verbosity": 0,
 +
"scaling": false,
 +
"block_size": 1,
 +
"use_block_matrices_if_possible": true

Latest revision as of 12:01, 3 January 2017

Contents

How to use the PaStiX solver in Kratos

The PaStiX (Parallel Sparse matriX package) is a open-source scientific library that provides a high performance parallel solver for very large sparse linear systems based on direct methods. Numerical algorithms are implemented in single or double precision (real or complex) using LLt, LDLt and LU with static pivoting (for non symmetric matrices having a symmetric pattern). This solver provides also an adaptive blockwise iLU(k) factorization that can be used as a parallel preconditioner using approximated supernodes to build a coarser block structure of the incomplete factors ([1]).

It can be linked to the Kratos by the ExternalSolversApplication. The following steps have to be made in order to use be able to use it in Ubuntu version 14.04 LTS 64 bit or 16.04 LTS 64 bit.

Compile the Scotch library (shared memory parallelism)

PaStiX needs the Scotch library as a prerequisite to work.

You can download version 6.0.4 here:

http://gforge.inria.fr/projects/scotch/

Copy it to a folder of your choice. In the following we are using "~/compiled_libraries" as a reference.

Extract the files by the command:

tar -xf scotch_6.0.4.tar.gz

Navigate into the source directory by:

cd scotch_6.0.4/src

Copy the correct MakeFile.inc for the version of your system to the source directory. Ín the case of the Ubuntu versions mentioned above it is the Makefile.inc.x86-64_linux2:

cp ./Make.inc/Makefile.inc.x86-64_linux2 Makefile.inc

Additional libraries are necessary, which can be installed by:

sudo apt-get install flex bison zlib1g-dev

In order to install Scotch locally the install path has to be set. Since the "prefix" command is not working this has to be done manually in the Makefile, which is also located in the /src folder. You can see an example for the part of the file to be modified below:

...some code...
include Makefile.inc
prefix		?= ~/software/scotch_6.0.4-install
bindir		?= $(prefix)/bin
includedir	?= $(prefix)/include
libdir		?= $(prefix)/lib
datarootdir	?= $(prefix)/share
mandir		?= $(datarootdir)/man
...some code...

Now Scotch can be compiled by (we are still in the /src directory):

make scotch

And can be installed to the directory specified in the MakeFile by:

make install

Compile the PaStiX library (shared memory parallelism)

Download the PaStiX version 5.2.3 from:

https://gforge.inria.fr/frs/?group_id=186

Copy it to a folder of your choice. In the following we are using "~/compiled_libraries" as a reference.

Extract the files by the command:

tar -xf pastix_5.2.3.tar.bz2

Navigate into the source directory by:

cd pastix_5.2.3/src

The different config files can be found in the config folder. You have to copy the one fitting to your system to the /src folder. In the case of the systems mentioned above it ist the LINUX-GNU.in file. Copy it to the source directory as config.in by:

cp ./config/LINUX-GNU.in config.in

You have to edit the following blocks of the config.in file:

###################################################################
#                  SETTING INSTALL DIRECTORIES                    #
###################################################################
ROOT          = /your/pastix/installation/path
INCLUDEDIR    = ${ROOT}/include
LIBDIR        = ${ROOT}/lib
BINDIR        = ${ROOT}/bin
PYTHON_PREFIX = ${ROOT}
###################################################################
#                          INTEGER TYPE                           #
###################################################################
# Uncomment the following lines for integer type support (Only 1)
#VERSIONINT  = _long
#CCTYPES     = -DFORCE_LONG -DINTSIZELONG
#---------------------------
VERSIONINT  = _int32
CCTYPES     = -DINTSIZE32
#---------------------------
#VERSIONINT  = _int64
#CCTYPES     = -DINTSSIZE64
###################################################################
#                          MPI/THREADS                            #
###################################################################
# Uncomment the following lines for sequential (NOMPI) version
VERSIONMPI  = _nompi
CCTYPES    := $(CCTYPES) -DFORCE_NOMPI
MPCCPROG    = $(CCPROG)
MCFPROG     = $(CFPROG)
MPCXXPROG   = $(CXXPROG)
...some code...
#CCPASTIX   := $(CCPASTIX) -DCUDA_SM_VERSION=20
#NVCCOPT    := $(NVCCOPT) -arch sm_20
###################################################################
#                      GRAPH PARTITIONING                         #
###################################################################
# Uncomment the following lines for using metis ordering
#VERSIONORD  = _metis
#METIS_HOME  = ${HOME}/metis-4.0
#CCPASTIX   := $(CCPASTIX) -DMETIS -I$(METIS_HOME)/Lib
#EXTRALIB   := $(EXTRALIB) -L$(METIS_HOME) -lmetis
# Scotch always needed to compile
SCOTCH_HOME = ${HOME}/compiled_libraries/scotch_6.0.4
SCOTCH_INC = $(SCOTCH_HOME)/include
SCOTCH_LIB = $(SCOTCH_HOME)/lib
# Uncomment on of this blocks
#scotch
CCPASTIX   := $(CCPASTIX) -I$(SCOTCH_INC) -DWITH_SCOTCH
EXTRALIB   := $(EXTRALIB) -L$(SCOTCH_LIB) -lscotch -lscotcherrexit
#ptscotch
#CCPASTIX   := $(CCPASTIX) -I$(SCOTCH_INC) -DDISTRIBUTED -DWITH_SCOTCH
#if scotch >= 6.0
EXTRALIB   := $(EXTRALIB) -L$(SCOTCH_LIB) -lptscotch -lscotch -lptscotcherrexit
#else
#EXTRALIB   := $(EXTRALIB) -L$(SCOTCH_LIB) -lptscotch -lptscotcherrexit
###################################################################
#                Portable Hardware Locality                       #
###################################################################
# By default PaStiX uses hwloc to bind threads,
# comment this lines if you don't want it (not recommended)
#HWLOC_HOME = ~/compiled_libraries/hwloc-1.11.5 #/opt/hwloc/
#HWLOC_INC  = $(HWLOC_HOME)/include
#HWLOC_LIB  = $(HWLOC_HOME)/lib
#CCPASTIX   := $(CCPASTIX) -I$(HWLOC_INC) -DWITH_HWLOC
#EXTRALIB   := $(EXTRALIB) -L$(HWLOC_LIB) -lhwloc

Now you can compile the PaStiX library (we are still in the /src directory):

make

You can install it to the directory defined in the config.in file by:

make install

In order to be able to compile the Kratos with PaStiX now, you have to fix a little bug in one of the header files in the installation directory. In the header file pastix_int32_double_real.h you have to add:

#include <stdint.h>

Compile the Kratos with PaStiX (shared memory parallelism)

Finally we can compile the Kratos with the PaStiX library. In order to do this, you have to add the following lines to you configure.sh file:

-DINCLUDE_PASTIX=ON                                       \
-DPASTIX_INSTALL_DIR= "/path/to/your/pastix/installation" \ (e.g. "~/software/pastix_5.2.3-install/install") 
-DSCOTCH_INSTALL_DIR= "/path/to/your/scotch/installation" \ (e.g. "~/software/scotch_6.0.4-install/lib")

You can now use the direct and iterative PaStiX solver by adding the following block to your ProjectParameters.json file:

"solver_type": "PastixSolver",
"solution_method": "Direct",
"tolerance": 0.000001,
"max_iteration": 100,
"gmres_krylov_space_dimension": 100,
"ilu_level_of_fill": 1,
"is_symmetric": false,
"verbosity": 0,
"scaling": false,
"block_size": 1,
"use_block_matrices_if_possible": true
Personal tools
Categories