How to run or add tests

In addition to automatically compiling the code, the Buildbot also runs a suite of short tests to verify that iSALE is functioning correctly. Of course, the rigour of this automated testing depends on the number and breadth of the tests. For this reason it is important that when new functionality is added developers also add a (short) test to check that their new function is (still) working.

Running the test suite

To run all the tests: in the the root directory of your working copy, simply type:

make tests

Note that you must have iSALEPlot, VIMoD and ANEOS enabled (./configure --with-isaleplot --with-vimod) to run all the tests.

To run a subset of the tests you can run one of a number of test suites.

  • make tests-2D Will make all iSALE2D tests (tests tagged iSALE2D)
  • make tests-3D Will make all iSALE3D tests (tests tagged iSALE3D)
  • make tests-ANEOS Will make all ANEOS tests (tests tagged ANEOS)

Alternatively, you can omit tests from suite using the commands:

  • make tests-no-iSALEPlot Will make all tests except those tests tagged iSALEPlot
  • make tests-no-vimod Will make all tests except those tests tagged vimod
  • make tests-no-viz Will make all tests except those tests tagged iSALEPlot or VIMoD
  • make tests-no-aneos Will make all tests except those tests tagged ANEOS

The directory containing all the tests is:


Adding a new test

To add a new test you need three things:

  1. The input files for the iSALE problem you want to run, as well any input files for iSALEPlot if you want to use this to post-process.
  2. A Makefile, which is just a recipe for running your test problem
  3. An .xml file which includes instructions for how you are going to test that iSALE ran correctly

In the following example we are going to add the first iSALE test, which is called planar_shock. In this test, two very short, low-resolution planar impact simulations are run; in one the impact velocity is 600 m/s in the other the impact velocity is 800 m/s. The results are processed with iSALEPlot to produce pressure and density profiles across the shock. The test compares the shock pressures and densities to known values from the equation of state.

Step 1: Create a new subdirectory for your test

gsc@hellas:trunk$ pwd
gsc@hellas:trunk$ cd iSALE/tests/
gsc@hellas:tests$ mkdir planar_shock
gsc@hellas:tests$ cd planar_shock

Step 2: Copy all necessary input files into the test directory

gsc@hellas:planar_shock$ pwd
gsc@hellas:planar_shock$ cp <path/to/input/files/input_files.inp> .

This is the content of the planar_shock directory:

gsc@hellas:planar_shock$ ls -l
total 28
-rw-r--r-- 1 gsc esestaff 2108 2011-03-31 15:29 exportPre.inp
-rw-r--r-- 1 gsc esestaff 3725 2011-03-31 15:29 planar600.inp
-rw-r--r-- 1 gsc esestaff 3725 2011-03-31 15:29 planar800.inp
-rw-r--r-- 1 gsc esestaff  832 2011-03-31 15:29 planar.mat
-rw-r--r-- 1 gsc esestaff  975 2011-03-31 15:35 planar_shock_plot.gnuscript
  • exportPre.inp is the iSALEPlot input file
  • planar600.inp is the iSALE input file for the 600 m/s planar impact simulation
  • planar800.inp is the iSALE input file for the 800 m/s planar impact simulation
  • planar.mat is the material input file for both the planar impact simulations
  • planar_shock_plot.gnuscript is a gnuscript script for plotting the shock pressure vs. shock density data

Step 3: Create Makefile with instructions for how to run your problem

Let's look at the one in planar_shock

gsc@hellas:planar_shock$ pwd
gsc@hellas:planar_shock$ more Makefile

It contains
inputfiles=planar800.inp planar600.inp
include ../

run: prerun
    @echo **********Running 800 m/s simulation
    $(ISALE2D) -i planar800.inp -M planar.mat
    cd Planar-800mps; $(ISALEPLOT) -f ../exportPre.inp -m Planar-800mps.dat
    @echo **********Running 600 m/s simulation
    $(ISALE2D) -i planar600.inp -M planar.mat
    cd Planar-600mps; $(ISALEPLOT) -f ../exportPre.inp -m Planar-600mps.dat
    gnuplot planar_shock_plot.gnuscript

    rm -rf Planar-600mps Planar-800mps *.eps

The Makefile must start with a list containing all model-input-files in this directory. This is usually just 'asteroid.inp', but for the tests in this directory we need two different input-files called 'planar800.inp' and 'planar600.inp':

inputfiles=planar800.inp planar600.inp

This list helps to identify all input-files and allows to keep all input-files up-to-date. The command 'make convert' allows you to convert all input-files (if they have an older format) to the most recent version.
If your test does not require any iSALE-input-files, you can skip this step.

The next line is definitely required. It includes the file located in /trunk/iSALE/tests:
include ../

This file defines some useful variables, namely:
$(ISALE2D) location of the iSALE-2D executable
$(ISALE3D) location of the iSALE-3D executable. If iSALE-3D is configured for parallel calculations, $(ISALE3D) starts iSALE-3D by using two processes.
$(ISALEPLOT) location of the iSALEPlot executable

The Makefile must have the instruction "run" as this is run by the automatic test harness. "run" is the instruction for running the iSALE test problem and any post-processing script that might be necessary. "clean-usr" is an additional instruction for convenience, which deletes the non-input files.

Let's take a look at the "run" instructions in more detail:

run:  prerun

This first step prerun is defined in and performs two actions: First, it runs the "clean" instructions. This makes sure that if the test has been run before, the output from the last run is deleted. Second, it creates a symbolic link called eos that points to the equation of state subdirectory in your working copy ../../eos. This will direct iSALE to equation of state files that it needs to run your problem.

    @echo **********Running 800 m/s simulation

This line is just printing to the screen some information to tell the observer that we are first going to run the 800 m/s planar impact simulation.

    $(ISALE2D) -i planar800.inp -M planar.mat

This is the instruction to run the first simulation. We are instructing iSALE2D that it should use planar800.inp as the main input file and planar.mat as the material input file.

    cd Planar-800mps; $(ISALEPLOT) -f ../exportPre.inp -m Planar-800mps.dat

The final step in running the first simulation is actually two steps: we first change into the Planar-800mps directory. Then we run iSALEPlot on the output file. Note that ../exportPre.inp is the iSALEPlot input file that tells iSALEPlot we want to export the pressure and density profiles to text files (the ../ is required because we want to keep this input file with the iSALE input files in the planar_shock directory).

The next lines just repeat this procedure for the 600 m/s simulation:

    @echo **********Running 600 m/s simulation
    $(ISALE2D) -i planar600.inp -M planar.mat
    cd Planar-600mps; $(ISALEPLOT) -f ../exportPre.inp -m Planar-600mps.dat


    gnuplot planar_shock_plot.gnuscript

runs a gnuplot script to create a .eps plot of the shock pressure vs. shock density data. This is not really necessary for a test---it is not used to verify the results of the simulation---but it is helpful to have this for debugging purposes.

Once you have created your Makefile, you will probably want to test it. Type "make run" to perform all the instructions; check that everything has worked; then type make clean to make sure you tidy up afterwards.

Step 4: Create XML file with test instructions

OK. This is the really clever bit. The test harness procedure (developed by the Applied Computational Modelling Group) that runs all the tests uses an XML test case file to provide it with all the information it needs to decide how to tell if the test should pass or fail. Importantly, the test case XML file should share the same name as the test directory: so our XML file will be: planar_shock.xml. Have a look at the full version in your working copy of the repository. Don't let the syntax put you off; it's really very straightforward and we'll go through bits of it in detail below:

gsc@hellas:planar_shock$ pwd
gsc@hellas:planar_shock$ more planar_shock.xml 
<?xml version="1.0" encoding="UTF-8" ?>
<!DOCTYPE testproblem SYSTEM "regressiontest.dtd">

The first two funny-looking lines are a necessary evil that you don't ever need to understand or change.

  <name>Planar Impacts (600 m/s and 800 m/s)</name>
  <owner userid="gsc"/>
  <problem_definition length="short" nprocs="1">
    <command_line>make run</command_line>
    <!-- Trial test case for iSALE -->

The above section tells the test harness what the name of the problem is (Planar Impacts), who the owner is (me!), how long it takes to run, how many processors it uses, and the command line to execute. Let's take them in turn:

  • The name: used for printing out the results of the test case. There are no particular requirements for it.
  • The owner: username of the person who wrote and will maintain the test
  • The tags: you can associate arbitrary labels with tests, so that you can group them later. All tests should have at least one tag (e.g. "iSALE2D", "iSALE3D", "ANEOS"), depending on how the problem is set up.
  • The length: can be one of "short", "medium", "long" and "special". In future, this will be used in the buildbot to decide how and when to run your problem automatically. For now, all tests should be "short", which means they should run in less than a minute. If your test takes longer, it is too complicated!
  • The number of processors: here this is a simple serial job, so the number of processors (nprocs) is 1. For all short or medium tests this should be 1.
  • The command line: what command to execute to run your problem. This should always be make run, so that the recipe in your Makefile is followed.

In this test we are going to check that the shock pressure and density in the quartz material in our planar impact simulation is correct. To do this we need to define some variables. This happens in the the part of the XML file within:


Let's look at just one variable definition. The part of the XML file below describes how to find the shock pressure in one quartzite cell of the mesh at a time when the first shock wave from the 600 m/s impact has arrived.

    <variable name="shockpressure600a" language="python">
from numpy import loadtxt
data  = loadtxt('Planar-600mps/export/PreDen00030.txt', comments='#')
shockpressure600a = data[35,2]*1.E-9

Each variable has a name; in the above example we have defined the name as shockpressure600a (this is because it is the shock pressure in the 600 m/s simulation for the first (a) shock wave. We have then specified that the language of the code we will use to define the variable is "python". The python code inside the variable tags is very simple. We first import the function "loadtxt" from the python module "numpy"---we need this to read the text file with the pressure and density data that iSALEPlot exported. The second line uses loadtxt to read the data written in the file Planar-600mps/export/PreDen00030.txt into an array called "data". Note that the comments='#' option just tells loadtxt to ignore any line in the .txt file that begins with a '#'. The last line defines shockpressure600a as a single value in the data array (converted into GPa); it takes the entry in the 36th row (python starts indexing from 0) and the 3rd column. NOTE: the file exported by iSALEPlot comprises four columns of data [X position, Y position, Pressure, Density], all in SI units.

If you examine the planar_shock.xml file in full you will see that there are several other variables defined: a shock pressure and density for each simulation, for both the first direct shock wave and the second reflected shock wave (the quartzite is buffered by an iron plate of higher impedance).

After the variables are defined, we next need some rules to test whether these variables are the ones we expect. This is done in the section of the XML file between:


Again, let's just look at one of the tests in the planar_shock example:

    <test name="check of shock pressure (600 m/s; direct wave)" language="python">
print 'Shock pressure (GPa) = ',shockpressure600a
assert abs(shockpressure600a/5.3546-1.) &lt; 1.E-3 

The test name should uniquely identify the test to aid the debugger. The example above is a test of the shock pressure value. Again the test code is written in python. The first line just prints the shock pressure out (for debugging purposes if there is a problem). The second line is the test. It is checking whether the shock pressure (in GPa) is within a certain tolerance (.1 percent) of a prescribed value: 5.3546. Note that if you use > or < in your tests, you have to replace them with &gt; or &lt; respectively, because > and < are part of the XML syntax.

The test passes if no assertion is raised, and it fails if the assertion is not true.

Step 5: Run your test manually

Now that you have the Makefile (specifying how to run your problem) and your .xml file (specifying what to check) written, run your test case with:

gsc@hellas:planar_shock$ pwd
gsc@hellas:planar_shock$ ../ -f planar_shock.xml

This will run the simulation, fetch the variables and check they meet all the conditions you specify.

If you have already run the test case and so have dump files etc available, but want to play with what is checked in the xml file, you can run the tests without having to run the simulation with iSALE, etc., all over again. To do this type:

gsc@hellas:planar_shock$ pwd
gsc@hellas:planar_shock$ ../ -f planar_shock.xml  --just-test

Step 6: Commit your test!

Once you are happy that your test is working properly, please commit your test to the repository so that it can be added to the suite of tests run automatically. However, please only commit the files that the test needs to run, not all of the output data as well. So, before you commit, type make clean, if you have specified rules for this, or clean up your directory manually. When I committed the planar_shock example I did this:

gsc@hellas:trunk$ pwd
gsc@hellas:trunk$ svn up
At revision 574.
gsc@hellas:trunk$ cd iSALE/tests/; pwd
gsc@hellas:tests$ svn add planar_shock
A         planar_shock
A         planar_shock/exportPre.inp
A         planar_shock/planar600.inp
A         planar_shock/planar800.inp
A         planar_shock/planar.mat
A         planar_shock/Makefile
A         planar_shock/planar_shock.xml
A         planar_shock/planar_shock_plot.gnuscript
gsc@hellas:tests$ svn ci planar_shock -m"Adding planar shock test case" 
Adding         planar_shock
Adding         planar_shock/exportPre.inp
Adding         planar_shock/planar600.inp
Adding         planar_shock/planar800.inp
Adding         planar_shock/planar.mat
Adding         planar_shock/Makefile
Adding         planar_shock/planar_shock.xml
Adding         planar_shock/planar_shock_plot.gnuscript
Transmitting file data ...........
Committed revision 575.

What will happen next is that the Buildbot will notice a change to the code has been committed; it will update its version of the repository; compile all subprojects; run all short tests (including your new one). If any of these steps fails, a failure email will be sent out and the results will be reported here