Home · About · A-Z Index · Search · Contacts · Press · Register · Login

The Open Group Verification Suite Development Process

Certified Products

Certification Guides

Documents

Conformance Statements

Links

You are here: The Open Brand >Testing Test procedures >The Open Group Verification Suite Development Process

Introduction

The process for test development is based on experience gained through over twelve years of X/Open test development. This involves a multi-step process where objectivity, and traceability of test code to a written specification are the key. It is the written specification that is the definitive reference point and in cases of doubt assertions and tests are secondary and defer to the specification. Quality , correctness and fitness-for-purpose are achieved through a methodology that places regular checkpoints and peer-review feedback within the process.

Process overview

The first step is development of a test specification. Secondly, a set of formal test assertions are developed. Thirdly, the test suite based on the specification and the assertions is developed and taken through a controlled development cycle before general availability. Lastly after completion of the test suite, the test suite is moved to support and maintenance mode where customer support questions, and interpretations are processed, and regular test suite updates made.

The Open Group has developed a set of criteria that the test specification, assertions and the test suite need to be meet in order to be accepted. At regular points formal reviews are held and feedback is obtained from the intended customer and specification owners to verify correct assumptions. A technical expert group is used to advise on specification interpretations during development (questions from developers) during Beta (issues raised by beta testers) and during test suite maintenance. This expert group is usually the owners of the specification. Such issues are dealt with in a way that ensures an archived set of referencable interpretations for other test suite users and potential brandees.

Test Suite Specification

A test suite specification is developed which provides a high level description of the test suite. This describes the test methodology, the functional coverage, the test architecture and states any assumptions about the test environment . It also includes a schedule of review points for the project.

Test Assertions

Creation of a set of test assertions is a necessary step to bridge the gap between natural language specification and test suite code. The goal of the test suite code is to produce an executable representation of the requirements of the specification under test.

The Open Group adopts an assertion based technique for test specifications. This follows the IEEE Standard for Test Methods (1003.3). An assertion is developed for each definitive statement in the specification under test. Each assertion is designed to determine whether the statement is true or false for the implementation under test.

Typically, a definitive statement within a specification contains or implies a shall, should or may. A shall statement is an unconditional requirement. A should or may statement is a conditional requirement, based on support of an option. Some judgement needs to be exercised by the assertion writer to ensure that the assertions generated are based upon the requirements of specification, and not in some cases based on non-normative advice to programmers or such like.

For example, a specification might state:

Required files for /bin
If the C compiler exists, then the following commands shall exist in /bin.
{A list of 34 commands}

This is written as 34 separate test assertions , for example

Reference 3.1-3 (C)
If the implementation provides a C compiler:
	The XYZ command will exist in /bin.

etc.

Assertions also need to consider the level of testing. POSIX.3 defines three levels:

1. Exhaustive : Full testing of all aspects of a specification requirement. For example, testing every possible value for an argument to a function.
2. Thorough : Testing sufficientto validate that the implementation under test matches a specification requirement without exhaustive testing. For example, a subset of the possible values of an argument to a function, exercising boundary and representative conditions.
3. Identification : Testing that the implementation-under-test provides the mechanism necessary to meet the specification requirement. For example, that a function exists without validating its processing of any arguments.

Testing occurs at the exhaustive level where practical. However in many areas it may not be practical to do so due to the time and system resources required. Most test suites perform thorough testing which often serves as a fair compromise in the coverage/cost ratio. Identification testing is rarely used unless the specification requires it, by stating merely that a feature exists and defining no behaviours or values.

Reviews

Once the test suite specification has been developed it needs validating to assure that its a correct representation of the requirements of the specification under test.

The Open Group has developed methods for managing review feedback which allow development to proceed in an objective and timely fashion.

This is achieved using review guidelines and a pro-forma responses. Comments should be specific , and always include a specified action. Just stating that there is a problem, without stating a solution is not useful or procedurally acceptable.

The same applies for the test assertions.

At this point an iterative process occurs as test assertions are validated or shown to be false. This may be due to an incorrect assumption by the author of the test assertion, or in some cases what is now considered an error in the underlying specification. The latter should only be decided by the authors of the specifications, and not the test suite author, and these issues should be fed back to the specification owners so that corrective action can be taken. The test suite will note the issue in its output , but the final correct strategy cannot be undertaken until the specification owners have issued corrections or possibly a revised version of the specification.

Once a test suite has completed the development cycle The Open Group operates a formal process with guaranteed response times for resolving issues concerning the validity of individual tests. This process is confidential, objective, repeatable and fair, making use of industry consensus as appropriate. Where a test is found to give false 'fails' after all company and product details are removed, the formal interpretation is published on the open group web server for reference by other test suite support customers.

Test Development

The Open Group has developed formal procedures for test acceptance. This involves up to two review cycles:

1. Alpha test cycle: This is an optional stage, not required if a sequence of test suite snapshots are delivered. At this stage a complete test framework would be delivered with over 50% of tests complete. This early stage allows early validation of both the general test approach and also for issues to be identified with the assertions and underlying specification.
2. Beta test cycle: This is mandatory. At this stage the complete test suite has been coded and this is the formal review of correctness. The Open Group has developed procedures for formal acceptance of test suites where severity and numbers of problem reports are logged and used to objectively determine acceptance.

It is recommended that participants in the test program supply equipment and sample implementations for use by the Open Group during the development cycle. This can greatly reduce the effort for debugging and ensure maximum portability of the resulting test suite.

Test Suite Maintenance

Once test development is complete a test suite goes into a maintenance cycle. A support service is established using the world wide web and electronic mail. A database of sanitised interpretations is maintained that can be searched by other test suite users. Regular maintenance releases are produced up to twice a year if necessary, addressing customer issues identified by using the test suite live.