Unified UPC Test Framework (UUTF) HOWTO

Unified UPC Test Framework (UUTF) is bunches of bash  scripts and gmake  templates 
to enhance portability of UPC test benches.

Overview

Suppose the top dir of the test bench is ${TEST_SUITE}, the directory structure for the 
workloads looks like this:

${TEST_SUITE}
	|- config/
	|	|- template/
	|	|	|- Makefile.include.SGI
	|	|	|- Makefile.include.BerkeleyUPC
	|	|	|- … …
	|	|- Makefile.include
	|	|- Makefile.common
	|	|- sh.common
	|- upc-io_test/
	|	|- src/ (optional)
	|	|- doc/ (optional)
	|	|- include/ (optional)
	|	|- README (optional)
	|	|- Makefile
	|	|- Makefile.config (optional)
	|	|- UpcTest.sh
	|	|- Clean.sh(optional)
	|- upc_test/
	|- collectives_test/
	|- … …

config holds the global configuration for all workloads. 

There are several Makefile.include templates in conifg/template. Each of them defines 
the platform-dependent variable, such as static/dynamic mode, CC, UPCC and 
UPCFLAGS, to target one specific architecture/compiler. Users are free to add more 
templates there to address emerging architectures. Before any workload is built, it is the 
user’s responsibility to make a symbol link of Makefile.include to the target platform 
defined in config/template directory.

Makefile.common defines implicit rules how to build the object file from source code, 
and how to link the object files to the executive binary. All commands and flags are 
platform-independent variables that have been defined in the Makefile.include.

Sh.common and sh.test consist miscellaneous functions used in the UpcTest.sh script.

Let’s take a close look at one workload, and explain how this test framework in detail. 

In collectives_test, src/, doc/ and include/directories are optional. We just recommend the 
developers to put their source code in src/, documentation in doc/ and header files in 
include/. The Makefile may not need to modify if vpath  directive and header files search 
directory is configured in config/Makefile.common, 

Makefile.config is also optional. This file setups the workload-specific configuration. 
Developers may define extra linking library, header file or any other platform dependent 
settings in this file. For example, the upcio_test depends on the UPC-IO functions; we 
need to build the specific UPC-IO implementation from the 3rd party vendor. It is the 
users’ responsibilities to put all support libraries or object files into the current working 
directory. In this package, we provide a simple example to build GWU UPC-IO reference 
implementation on BUPC platform for your convenience.

Makefile is the last but the most important file in this test framework; all the previous 
work we have done is to simplify the Makefile. Generally speaking, the developer just 
define the ${TARGETS} and dependencies, done.

UpcTest.sh is in charge of building the targets, running the test suite, and summarizing 
the test results. Developers are free to customize it to meet their own needs.

It is quite flexible for the developers to integrate the previous effort to the current test 
framework. You may want to check out how the upc_test is implemented. In this 
workload, we do not have Makefile, but we do collect information from 
Makefile.include, and we compile the test case in the UpcTest.sh. Our test framework 
tries to help, but it’s up to you to decide how to do the test.

Integration
How to fit your test cases into this test framework? 

First, make sure your test case has been built and run in one platform successfully. Then 
check whether this platform has been supported in the conifg/ directory. If so, you are 
lucky, just replace your hard coded platform-depended variable in the Makefile with 
generic variables; if not, please develop the Makefile.include template for that platform 
and send a copy to us if you would like share your work with other developers. 

After that, the Makefile is platform/compiler independent. We also need to modify 
Makefile.config if your application depends on different library in various platforms.

Then copy the dependency to the new Makefile, and build the project. We might iterate 
the previous operations several time to get the execute test case. 

If you are ambitious to develop your own UpcTest.sh, please take a look at the existed 
example and stop here.

 If you would like to re-use the UpcTest.sh developed by GWU UPC working group, you 
might need to modify your test cases to comply with the interface between the test cases 
and test framework.

Your test case should output the following message to stdout or stderr to indicate the exit 
status of the test case:
   
   PASS: This test case exits normally, and the result complies to the corresponding 
specification.
   FAIL: This test case exits normally, but the result violates the corresponding 
specification.
   EROR: The test case exits abnormally due to unexpected error BEFORE the test case 
is executed, or the specific test condition is not satisfied. 

The reason we does not check the exit status of the test case is BUPC can not return the 
correct exit status of upc program. BUPC uses upcrun to launch the upc program. For 
example:

	upcrun  –n 4 foo

The exit status is the exit status of upcrun, instead of foo. The exit status of foo is 
discarded during the transition from the UPC world to the POSIX world.

Run
Now it is time to run the test bench, just call 
	./UpcTest.sh  your_test_case THREADS
You can build the specific test case, run it and collect the information. Enjoy.


  BASH  www.gnu.org/software/bash/bash.html
  GNU Make -  www.gnu.org/software/make/
  GNU Make Manual 4.5.2 The vpath Directive 
ttp://www.gnu.org/software/make/manual/html_node/make_32.html#SEC36