How to run or add tests¶
In addition to automatically compiling the code, the Buildbot also runs a suite of short tests to verify that iSALE is functioning correctly. Of course, the rigour of this automated testing depends on the number and breadth of the tests. For this reason it is important that when new functionality is added developers also add a (short) test to check that their new function is (still) working.
Running the test suite¶
To run all the tests: in the the root directory of your working copy, simply type:
Note that you must have iSALEPlot, VIMoD and ANEOS enabled (
./configure --with-isaleplot --with-vimod) to run all the tests.
To run a subset of the tests you can run one of a number of test suites.
make tests-2DWill make all iSALE2D tests (tests tagged iSALE2D)
make tests-3DWill make all iSALE3D tests (tests tagged iSALE3D)
make tests-ANEOSWill make all ANEOS tests (tests tagged ANEOS)
Alternatively, you can omit tests from suite using the commands:
make tests-no-iSALEPlotWill make all tests except those tests tagged iSALEPlot
make tests-no-vimodWill make all tests except those tests tagged vimod
make tests-no-vizWill make all tests except those tests tagged iSALEPlot or VIMoD
make tests-no-aneosWill make all tests except those tests tagged ANEOS
The directory containing all the tests is:
Adding a new test¶
To add a new test you need three things:
- The input files for the iSALE problem you want to run, as well any input files for iSALEPlot if you want to use this to post-process.
- A Makefile, which is just a recipe for running your test problem
.xmlfile which includes instructions for how you are going to test that iSALE ran correctly
In the following example we are going to add the first iSALE test, which is called
planar_shock. In this test, two very short, low-resolution planar impact simulations are run; in one the impact velocity is 600 m/s in the other the impact velocity is 800 m/s. The results are processed with
iSALEPlot to produce pressure and density profiles across the shock. The test compares the shock pressures and densities to known values from the equation of state.
Step 1: Create a new subdirectory for your test¶
gsc@hellas:trunk$ pwd /home/gsc/svn/iSALE/trunk gsc@hellas:trunk$ cd iSALE/tests/ gsc@hellas:tests$ mkdir planar_shock gsc@hellas:tests$ cd planar_shock
Step 2: Copy all necessary input files into the test directory¶
gsc@hellas:planar_shock$ pwd /home/gsc/svn/iSALE/trunk/iSALE/tests/planar_shock gsc@hellas:planar_shock$ cp <path/to/input/files/input_files.inp> .
This is the content of the
/home/gsc/svn/iSALE/trunk/iSALE/tests/planar_shock gsc@hellas:planar_shock$ ls -l total 28 -rw-r--r-- 1 gsc esestaff 2108 2011-03-31 15:29 exportPre.inp -rw-r--r-- 1 gsc esestaff 3725 2011-03-31 15:29 planar600.inp -rw-r--r-- 1 gsc esestaff 3725 2011-03-31 15:29 planar800.inp -rw-r--r-- 1 gsc esestaff 832 2011-03-31 15:29 planar.mat -rw-r--r-- 1 gsc esestaff 975 2011-03-31 15:35 planar_shock_plot.gnuscript
exportPre.inpis the iSALEPlot input file
planar600.inpis the iSALE input file for the 600 m/s planar impact simulation
planar800.inpis the iSALE input file for the 800 m/s planar impact simulation
planar.matis the material input file for both the planar impact simulations
planar_shock_plot.gnuscriptis a gnuscript script for plotting the shock pressure vs. shock density data
Step 3: Create Makefile with instructions for how to run your problem¶
Let's look at the one in
gsc@hellas:planar_shock$ pwd /home/gsc/svn/iSALE/trunk/iSALE/tests/planar_shock gsc@hellas:planar_shock$ more Makefile
inputfiles=planar800.inp planar600.inp include ../maketest.in run: prerun @echo **********Running 800 m/s simulation $(ISALE2D) -i planar800.inp -M planar.mat cd Planar-800mps; $(ISALEPLOT) -f ../exportPre.inp -m Planar-800mps.dat @echo **********Running 600 m/s simulation $(ISALE2D) -i planar600.inp -M planar.mat cd Planar-600mps; $(ISALEPLOT) -f ../exportPre.inp -m Planar-600mps.dat gnuplot planar_shock_plot.gnuscript clean-usr: rm -rf Planar-600mps Planar-800mps *.eps
The Makefile must start with a list containing all model-input-files in this directory. This is usually just 'asteroid.inp', but for the tests in this directory we need two different input-files called 'planar800.inp' and 'planar600.inp':
This list helps to identify all input-files and allows to keep all input-files up-to-date. The command '
make convert' allows you to convert all input-files (if they have an older format) to the most recent version.
If your test does not require any iSALE-input-files, you can skip this step. The next line is definitely required. It includes the file
maketest.inlocated in /trunk/iSALE/tests:
This file defines some useful variables, namely:
|$(ISALE2D)||location of the iSALE-2D executable|
|$(ISALE3D)|| location of the iSALE-3D executable. If iSALE-3D is configured for parallel calculations,
|$(ISALEPLOT)||location of the iSALEPlot executable|
The Makefile must have the instruction "run" as this is run by the automatic test harness. "run" is the instruction for running the iSALE test problem and any post-processing script that might be necessary. "clean-usr" is an additional instruction for convenience, which deletes the non-input files.
Let's take a look at the "run" instructions in more detail:
This first step
prerun is defined in makefile.in and performs two actions: First, it runs the "clean" instructions. This makes sure that if the test has been run before, the output from the last run is deleted. Second, it creates a symbolic link called
eos that points to the equation of state subdirectory in your working copy
../../eos. This will direct iSALE to equation of state files that it needs to run your problem.
@echo **********Running 800 m/s simulation
This line is just printing to the screen some information to tell the observer that we are first going to run the 800 m/s planar impact simulation.
$(ISALE2D) -i planar800.inp -M planar.mat
This is the instruction to run the first simulation. We are instructing iSALE2D that it should use
planar800.inp as the main input file and planar.mat as the material input file.
cd Planar-800mps; $(ISALEPLOT) -f ../exportPre.inp -m Planar-800mps.dat
The final step in running the first simulation is actually two steps: we first change into the
Planar-800mps directory. Then we run iSALEPlot on the output file. Note that
../exportPre.inp is the iSALEPlot input file that tells iSALEPlot we want to export the pressure and density profiles to text files (the
../ is required because we want to keep this input file with the iSALE input files in the
The next lines just repeat this procedure for the 600 m/s simulation:
@echo **********Running 600 m/s simulation $(ISALE2D) -i planar600.inp -M planar.mat cd Planar-600mps; $(ISALEPLOT) -f ../exportPre.inp -m Planar-600mps.dat
runs a gnuplot script to create a
.eps plot of the shock pressure vs. shock density data. This is not really necessary for a test---it is not used to verify the results of the simulation---but it is helpful to have this for debugging purposes.
Once you have created your Makefile, you will probably want to test it. Type "make run" to perform all the instructions; check that everything has worked; then type
make clean to make sure you tidy up afterwards.
Step 4: Create XML file with test instructions¶
OK. This is the really clever bit. The test harness procedure (developed by the Applied Computational Modelling Group) that runs all the tests uses an XML test case file to provide it with all the information it needs to decide how to tell if the test should pass or fail. Importantly, the test case XML file should share the same name as the test directory: so our XML file will be:
planar_shock.xml. Have a look at the full version in your working copy of the repository. Don't let the syntax put you off; it's really very straightforward and we'll go through bits of it in detail below:
gsc@hellas:planar_shock$ pwd /home/gsc/svn/iSALE/trunk/iSALE/tests/planar_shock gsc@hellas:planar_shock$ more planar_shock.xml <?xml version="1.0" encoding="UTF-8" ?> <!DOCTYPE testproblem SYSTEM "regressiontest.dtd">
The first two funny-looking lines are a necessary evil that you don't ever need to understand or change.
<testproblem> <name>Planar Impacts (600 m/s and 800 m/s)</name> <owner userid="gsc"/> <tags>iSALE2D</tags> <problem_definition length="short" nprocs="1"> <command_line>make run</command_line> <!-- Trial test case for iSALE --> </problem_definition>
The above section tells the test harness what the name of the problem is (Planar Impacts), who the owner is (me!), how long it takes to run, how many processors it uses, and the command line to execute. Let's take them in turn:
- The name: used for printing out the results of the test case. There are no particular requirements for it.
- The owner: username of the person who wrote and will maintain the test
- The tags: you can associate arbitrary labels with tests, so that you can group them later. All tests should have at least one tag (e.g. "iSALE2D", "iSALE3D", "ANEOS"), depending on how the problem is set up.
- The length: can be one of "short", "medium", "long" and "special". In future, this will be used in the buildbot to decide how and when to run your problem automatically. For now, all tests should be "short", which means they should run in less than a minute. If your test takes longer, it is too complicated!
- The number of processors: here this is a simple serial job, so the number of processors (nprocs) is 1. For all short or medium tests this should be 1.
- The command line: what command to execute to run your problem. This should always be
make run, so that the recipe in your Makefile is followed.
In this test we are going to check that the shock pressure and density in the quartz material in our planar impact simulation is correct. To do this we need to define some variables. This happens in the the part of the XML file within:
Let's look at just one variable definition. The part of the XML file below describes how to find the shock pressure in one quartzite cell of the mesh at a time when the first shock wave from the 600 m/s impact has arrived.
<variable name="shockpressure600a" language="python"> from numpy import loadtxt data = loadtxt('Planar-600mps/export/PreDen00030.txt', comments='#') shockpressure600a = data[35,2]*1.E-9 </variable>
Each variable has a name; in the above example we have defined the name as
shockpressure600a (this is because it is the shock pressure in the 600 m/s simulation for the first (a) shock wave. We have then specified that the language of the code we will use to define the variable is "python". The python code inside the variable tags is very simple. We first import the function "loadtxt" from the python module "numpy"---we need this to read the text file with the pressure and density data that iSALEPlot exported. The second line uses loadtxt to read the data written in the file
Planar-600mps/export/PreDen00030.txt into an array called "data". Note that the comments='#' option just tells loadtxt to ignore any line in the .txt file that begins with a '#'. The last line defines
shockpressure600a as a single value in the data array (converted into GPa); it takes the entry in the 36th row (python starts indexing from 0) and the 3rd column. NOTE: the file exported by iSALEPlot comprises four columns of data [X position, Y position, Pressure, Density], all in SI units.
If you examine the planar_shock.xml file in full you will see that there are several other variables defined: a shock pressure and density for each simulation, for both the first direct shock wave and the second reflected shock wave (the quartzite is buffered by an iron plate of higher impedance).
After the variables are defined, we next need some rules to test whether these variables are the ones we expect. This is done in the section of the XML file between:
Again, let's just look at one of the tests in the planar_shock example:
<test name="check of shock pressure (600 m/s; direct wave)" language="python"> print 'Shock pressure (GPa) = ',shockpressure600a assert abs(shockpressure600a/5.3546-1.) < 1.E-3 </test>
The test name should uniquely identify the test to aid the debugger. The example above is a test of the shock pressure value. Again the test code is written in python. The first line just prints the shock pressure out (for debugging purposes if there is a problem). The second line is the test. It is checking whether the shock pressure (in GPa) is within a certain tolerance (.1 percent) of a prescribed value: 5.3546. Note that if you use > or < in your tests, you have to replace them with
< respectively, because > and < are part of the XML syntax.
The test passes if no assertion is raised, and it fails if the assertion is not true.
Step 5: Run your test manually¶
Now that you have the Makefile (specifying how to run your problem) and your .xml file (specifying what to check) written, run your test case with:
gsc@hellas:planar_shock$ pwd /home/gsc/svn/iSALE/trunk/iSALE/tests/planar_shock gsc@hellas:planar_shock$ ../testharness.py -f planar_shock.xml
This will run the simulation, fetch the variables and check they meet all the conditions you specify.
If you have already run the test case and so have dump files etc available, but want to play with what is checked in the xml file, you can run the tests without having to run the simulation with iSALE, etc., all over again. To do this type:
gsc@hellas:planar_shock$ pwd /home/gsc/svn/iSALE/trunk/iSALE/tests/planar_shock gsc@hellas:planar_shock$ ../testharness.py -f planar_shock.xml --just-test
Step 6: Commit your test!¶
Once you are happy that your test is working properly, please commit your test to the repository so that it can be added to the suite of tests run automatically. However, please only commit the files that the test needs to run, not all of the output data as well. So, before you commit, type
make clean, if you have specified rules for this, or clean up your directory manually. When I committed the planar_shock example I did this:
gsc@hellas:trunk$ pwd /home/gsc/svn/iSALE/trunk gsc@hellas:trunk$ svn up At revision 574. gsc@hellas:trunk$ cd iSALE/tests/; pwd /home/gsc/svn/iSALE/trunk/iSALE/tests gsc@hellas:tests$ svn add planar_shock A planar_shock A planar_shock/exportPre.inp A planar_shock/planar600.inp A planar_shock/planar800.inp A planar_shock/planar.mat A planar_shock/Makefile A planar_shock/planar_shock.xml A planar_shock/planar_shock_plot.gnuscript gsc@hellas:tests$ svn ci planar_shock -m"Adding planar shock test case" Adding planar_shock Adding planar_shock/exportPre.inp Adding planar_shock/planar600.inp Adding planar_shock/planar800.inp Adding planar_shock/planar.mat Adding planar_shock/Makefile Adding planar_shock/planar_shock.xml Adding planar_shock/planar_shock_plot.gnuscript Transmitting file data ........... Committed revision 575.
What will happen next is that the Buildbot will notice a change to the code has been committed; it will update its version of the repository; compile all subprojects; run all short tests (including your new one). If any of these steps fails, a failure email will be sent out and the results will be reported here