How to add automatic benchmarking to your example

From KratosWiki
Jump to: navigation, search

Kratos contains an automatic benchmarking feature which can be used to automatically check the status of nightly build.

Contents

Automatic benchmarking

To use the automatic benchmarking feature you need to import benchmarking module from Kratos root/benchmarking. To do this add the lines printed in bold to your code:

##################################################################
##################################################################
## ATTENTION: here the order is important

#including kratos path
kratos_libs_path = '../../../../libs' ##kratos_root/libs
kratos_applications_path = '../../../../applications' ##kratos_root/applications
kratos_benchmarking_path = '../../../../benchmarking' ##kratos_root/benchmarking

import sys
sys.path.append(kratos_libs_path)
sys.path.append(kratos_applications_path)
sys.path.append(kratos_benchmarking_path)

#importing Kratos main library
from Kratos import *
kernel = Kernel()   #defining kernel

#importing applications
import applications_interface
applications_interface.Import_IncompressibleFluidApplication = True
applications_interface.Import_PFEMApplication = True
applications_interface.ImportApplications(kernel, kratos_applications_path)

import benchmarking
## from now on the order is not anymore crucial
##################################################################
##################################################################

Specifying the values to be benchmarked

The principle of the benchmarking module is to first marking some values in your program which need to be examined, or some part of the code which needs to be checked for amount of the time it takes to run. Then, the program is run (using the benchmarking module) in a special mode, called benchmarking mode, and the required data are collected and stored in a file. These data are called the reference data, and will be used to verify the functionality of the program later. This also is done using the benchmarking module.

To specify a value for benchmarking, pass it to benchmarking.Output. The format of using benchmarking.Output is:

benchmarking.Output(Var, Label = "", AbsErr = None, RelErr = None)

In the above, Var is the variable you want to benchmark. This variable can either be a floating number, an integer number, a string, or even a boolean. The label is an string, which you can later use to identify the source of a difference, if something goes wrong. For floating point numbers and integers, you may specify the amount of the absolute or relative errors allowed. These default to None, which means that the values should be exactly the same every time the program runs. Please note that you cannot specify a relative error tolerance for zero reference data! The reason should be clear. Another important point to remember is that the variables should be outputted in the same order every time the code runs. This means that the user need to perform some additional efforts in parallel cases or where the order of outputing the variables may change.

Determining if the program is running in benchmarking or build reference mode

Another useful function is benchmarking.InBenchmarkingMode, which tells if the program is being run in benchmarking mode. This is useful if you need to calculate some results only for benchmarking use:

if (benchmarking.InBenchmarkingMode()):
      calculate some results...
    benchmarking.Output(...)

You may note that the benchmarking.Output also uses this function to avoid any outputting of any benchmark data during a normal run.

You can also check if you are in build reference mode using InBuildReferenceMode function. This can be useful if you want to differentiate between the data collected as reference data and the data to be checked, for example in case you know the exact value for a quantity. Please note that this should not change the order or type of variable outputted for benchmarking. Typical use of this function can be:

if (benchmarking.InBenchmarkingMode()):
    if (benchmarking.InBuildReferenceMode()):
        # output exact value
        benchmarking.Output(...)
    else:
        # output value generated by the program
        benchmarking.Output(...)

Checking time consumed by some part of your code

The benchmarking module can also check the time consumed by some part of your code and see if it is running much faster or slower than the original run. The parts checked can also be nested. To use this feature, use benchmarking.StartTiming and benchmarking.StopTiming, as specified in the code below:

t = benchmarking.StartTiming()
  the code to be checked goes here...
benchmarking.StopTiming(t, AbsDiff, RelDiff)

AbsDiff and RelDiff are absolute and relative tolerance for the consumed time, respectively.

Getting notification of changes via email

The benchmarking module features a function which can be used to send an email, notifying a group of recipients from changes in benchmarked data and/or problems in them. The function can be used as:

benchmarking.NotifyViaEmail(Subject, Text, Recipients)

In the above code, Subject and Text are the subject of the email and the text of it. The recipients will always get the email from no-reply-kratos-benchmarking@cimne.upc.es, so they may need to add this address to their address book/white list to avoid its delivery to Spam/Bulk folder. The Recipients is a list of email addresses as strings, obviously.

Building the reference data

Building the reference data is very simple. To do this, just call benchmarking.BuildReferenceData like this:

benchmarking.BuildReferenceData(PathToThePythonScript, PathToTheReferenceDataFile)

This will invoke python to run the script and use grep to filter the benchmarking data into the specified reference data file. It is recommended that you use the above code from the same directory as the example itself, as the Kratos example are very sensitive to the path they are executed from. A systematic way to do this will be presented later in this article.

Running a benchmark

The way of running a benchmark is quite the same as making the reference data. To do this use the code below:

print benchmarking.RunBenchmark(PathToThePythonScript, PathToTheReferenceDataFile)

This will also run the example in benchmarking mode and filter the benchmarking data to a temporary file, name BenchTemp.txt. Then, the contents of this file will be checked against the specified reference data file, and any difference will be reported. If no differences is found, the function will return True.

The structure of automatic benchmarking in Kratos

For a systematic checking of the examples in Kratos, a special structure is employed. The below lines describe a sample Kratos structure (abridged):

Kratos root
  + applications
  |  + PFEMapplication
  |  |  + test_examples
  |  |     |  + pfem_benchmarks.py               The script to run all examples in test_examples directory in benchmarking mode
  |  |     |  + pfem_build_references.py         The script to build reference data for all examples in test_examples directory
  |  |     + dam2d.gid
  |  |     |  + dam2d.py                         The example
  |  |     |  + dam2d_benchmark.py               The script to run dam2d.py example in benchmarking mode
  |  |     |  + dam2d_build_reference.py         The script to build the reference data for dam2d.py example
  |  |     |  + dam2d_ref.txt                    The reference data
  |  |     + dam3d.gid
  |  |        + dam3d.py                         The example
  |  |        + dam3d_benchmark.py               The script to run dam3d.py example in benchmarking mode
  |  |        + dam3d_build_reference.py         The script to build the reference data for dam3d.py example
  |  |        + dam3d_ref.txt                    The reference data
  |  + structural_application
  |     + test_examples
  |        |  + structural_benchmarks.py         The script to run all examples in test_examples directory in benchmarking mode
  |        |  + structural_build_references.py   The script to build reference data for all examples in test_examples directory
  |        + cantilever2d.gid
  |        |  + cantilever2d.py                  The example
  |        |  + cantilever2d_benchmark.py        The script to run cantilever2d.py example in benchmarking mode
  |        |  + cantilever2d_build_reference.py  The script to build the reference data for cantilever2d.py example
  |        |  + cantilever2d_ref.txt             The reference data
  |        + cantilever3d.gid
  |           + cantilever3d.py                  The example
  |           + cantilever3d_benchmark.py        The script to run cantilever3d.py example in benchmarking mode
  |           + cantilever3d_build_reference.py  The script to build the reference data for cantilever3d.py example
  |           + cantilever3d_ref.txt             The reference data
  + benchmarking
  |  + benchmarking.py                           The benchmarking module 
  |  + run_all_benchmarks                        The script to run all examples in benchmarking mode
  |  + build_all_references                      The script to build all reference data for all examples
  + bin
  + doc
  + external_libraries
  + kratos
  + libs

This structure helps to generate a single status email be sent on every nightly build, using the run_all_benchmarks script. Also, the script build_all_references can be use to build all reference data files, as the name suggests. A sample script is provided for each kind of the script you need to provide/change in order to have your example be benchmarked automatically on every nightly build. Also, adding benchmarking and a sample reference data from dam2d example is provided.

  • Sample run_all_benchmarks.py
import os
import sys
import benchmarking

sys.path.append(".")

Text = "Status of Kratos examples:\n\n"
os.chdir("../applications")

# PFEM
os.chdir("PFEMapplication/test_examples")

import pfem_benchmarks
Text += pfem_benchmarks.Run()

os.chdir("../..")

# Add other directories here

print Text

benchmarking.NotifyViaEmail("Status of Kratos examples", Text, ["pooyan@cimne.upc.edu", "rrossi@cimne.upc.edu", "mossaiby@yahoo.com"])
  • Sample build_all_references.py
import os
import sys
import benchmarking

sys.path.append(".")

os.chdir("../applications")

# PFEM
os.chdir("PFEMapplication/test_examples")

import pfem_build_references

os.chdir("../..")

# Add other directories here
  • Sample pfem_benchmarks.py
import os

def Run():
	Msg = ""
	Text = "== PFEM ==========\n"

	# dam2d

	Text += "dam_2d: "
	os.chdir("dam2d.gid")	
	
	import dam2d_benchmark
	Msg = dam2d_benchmark.Run()
	
	if (Msg == True):
		Text += "OK\n"
	else:
		Text += "FAILED\n"
		Text += Msg
		Text += "\n\n"

	os.chdir("..")

	# Add other examples here

	return Text
  • Sample pfem_build_references.py
import os

# dam2d

os.chdir("dam2d.gid")	

import dam2d_build_reference

os.chdir("..")

# Add other examples here
  • Sample dam2d_benchmark.py

import benchmarking

def Run():
	print "Running dam2d.py..."
	return benchmarking.RunBenchmark("dam2d.py", "dam2d_ref.txt")
  • Sample dam2d_build_reference.py
import benchmarking

print "Building reference data for dam2d.py..."
benchmarking.BuildReferenceData("dam2d.py", "dam2d_ref.txt")
  • Sample change in dam2d.py example to add support for benchmarking
def FindNode(node_list,x,y,z):
    for node in node_list:
        if ((node.X - x) ** 2 + (node.Y - y) ** 2 + (node.Z - z) ** 2 < 0.0000001):
            return node

def BenchmarkCheck(time, node1, node2):
    benchmarking.Output(time, "Time")
    benchmarking.Output(node1.GetSolutionStepValue(PRESSURE), "Node 1 pressure", 1.0)
    benchmarking.Output(node2.GetSolutionStepValue(PRESSURE), "Node 2 pressure", 1.0)
:
:
node_1 = FindNode(model_part.Nodes, 0.5, 0.0, 0.0)
node_2 = FindNode(model_part.Nodes, 0.24, 0.0, 0.0)
:
:
  in the main loop
  BenchmarkCheck(time, node_1, node_2)
  • Sample dam2d_ref.txt (created as described above)
KRATOS_BENCHMARK | Float | Time | 0.02 | None | None
KRATOS_BENCHMARK | Float | Node 1 pressure | 0.0 | 1.0 | None
KRATOS_BENCHMARK | Float | Node 2 pressure | 0.0 | 1.0 | None
KRATOS_BENCHMARK | Float | Time | 0.03 | None | None
KRATOS_BENCHMARK | Float | Node 1 pressure | 0.0 | 1.0 | None
KRATOS_BENCHMARK | Float | Node 2 pressure | 0.0 | 1.0 | None
KRATOS_BENCHMARK | Float | Time | 0.04 | None | None
KRATOS_BENCHMARK | Float | Node 1 pressure | 0.0 | 1.0 | None
KRATOS_BENCHMARK | Float | Node 2 pressure | 0.0 | 1.0 | None
KRATOS_BENCHMARK | Float | Time | 0.05 | None | None
KRATOS_BENCHMARK | Float | Node 1 pressure | 0.0 | 1.0 | None
KRATOS_BENCHMARK | Float | Node 2 pressure | 0.0 | 1.0 | None
KRATOS_BENCHMARK | Float | Time | 0.06 | None | None
KRATOS_BENCHMARK | Float | Node 1 pressure | 333.406773695 | 1.0 | None
KRATOS_BENCHMARK | Float | Node 2 pressure | 2458.18608537 | 1.0 | None
KRATOS_BENCHMARK | Float | Time | 0.07 | None | None
KRATOS_BENCHMARK | Float | Node 1 pressure | 312.122179189 | 1.0 | None
KRATOS_BENCHMARK | Float | Node 2 pressure | 3463.08976427 | 1.0 | None
KRATOS_BENCHMARK | Float | Time | 0.08 | None | None
KRATOS_BENCHMARK | Float | Node 1 pressure | 299.833235691 | 1.0 | None
KRATOS_BENCHMARK | Float | Node 2 pressure | 3281.962115 | 1.0 | None
KRATOS_BENCHMARK | Float | Time | 0.09 | None | None
KRATOS_BENCHMARK | Float | Node 1 pressure | 383.857316791 | 1.0 | None
KRATOS_BENCHMARK | Float | Node 2 pressure | 3114.42161813 | 1.0 | None
KRATOS_BENCHMARK | Float | Time | 0.1 | None | None
KRATOS_BENCHMARK | Float | Node 1 pressure | 570.167983978 | 1.0 | None
KRATOS_BENCHMARK | Float | Node 2 pressure | 2998.74435582 | 1.0 | None
KRATOS_BENCHMARK | Float | Time | 0.108789867663 | None | None
KRATOS_BENCHMARK | Float | Node 1 pressure | 1010.36409701 | 1.0 | None
KRATOS_BENCHMARK | Float | Node 2 pressure | 3026.6533133 | 1.0 | None
Personal tools
Categories