Home · About · A-Z Index · Search · Contacts · Press · Register · Login

The Open Brand -- Test Suite Acceptance Criteria

Certified Products

Certification Guides

Documents

Conformance Statements

Links

You are here: The Open Brand >Testing > Test procedures >Test Suite acceptance criteria

This document is a set of advisory criteria to be used in evaluating the suitability of test suites for inclusion in the Open Brand program. 

The document classifies criteria as being either mandatory or desirable:

• Mandatory A test suite must meet the criterion. Dispensations will be granted only in exceptional circumstances.
• Desirable The criterion may not in all circumstances be met. Failure to meet a desirable criterion will not necessarily result in exclusion, but will require justification.

The funding model may alter the weighting and rigour applied to the criteria during test suite evaluation. These criteria may be used to define what work needs to be done to convert an existing test suite into one acceptable for inclusion in the Brand program.

1. The Test Specification

1.1. The test shall be based on a written specification (Mandatory)

1.2. The test specification upon which the test suite is based shall confirm to POSIX .3 (Desirable)

2. The Test Suite Design

2.1. The test suite is aligned with the current draft/issue of the relevant specification (Mandatory)

2.2. The test suite must be able to establish the correctness of all claims of support of any optional or implementation-defined features of the specification under test (Mandatory)(That is, where the specification under test allows optional behaviour or there is a range in the way a particular feature is supported, for example, limits)

2.3. Each individually executable test must provide its own setup and clean up functionality (Mandatory)

2.4. No individually executable test shall introduce side effects that will affect the results of any other tests (Mandatory)

2.5. The unit of test execution should cover no more than a single interface function (Desirable)

2.6. The test suite shall have a capability to check complete and correct configuration and installation. This may include a set of defined confidence checks (Mandatory)

2.7. Configuration and installation checking facilities should be automated (Desirable)

2.8. The test suite must include adequate user and programmer documentation (Mandatory)

2.9. The test suite documentation should follow The Open Group test suite documentation style guide (Desirable)

2.10. The test suite shall be sufficiently documented and structured to facilitate the debugging of an implementation under test (Mandatory)

2.11. Test system design should conform to POSIX .3 (Desirable)

2.12. The test suite must be fully integrated with TET for non-distributed testing or TET3 for distributed testing (Desirable)

2.13. The test suite supplier shall produce a questionnaire designed to elicit all extra information needed to parameterize the test suite and determine the testability of the implementation (Desirable)

3. Test Suite Portability

3.1. If applicable, the test suite must be capable of testing its target specifications in XPG4 Base environments and in Windows NT and Windows 95 systems, provided TET is installed. An acceptable alternative is a Java environment. (Mandatory)

3.2. The test suite shall not require any resources or functionality outside TET and the specification under test; that is, the test suite must itself be a fully conforming application (Desirable)

3.3. Over and above TET, XPG4 Base and the specification against which conformance is to be assessed, any extensions that are required to build, configure, or run the test suite must be documented and must be freely available on equitable terms (Mandatory)

3.4. Any activity required by the implementor in order to port the test suite to the implementation under test shall be clearly and unambiguously documented (Mandatory)

3.5 The test suite shall be written in C and/or C++ without use of the following features: (Desirable)

• Run Time Test Information (RTTI)

3.6 The test suite shall compile without change using the following compilers: (Mandatory)

• Microsoft C++ 8.0 and later
• GNU C++ compiler (version TBD) and later
• Solaris C++ compiler (version TBD) and later

4. Test Suite Commercial Terms

4.1. The test suite licensing terms must allow it to be used for The Open Brand (Mandatory)

4.2. The test suite must be made available on fair and equitable licensing terms (Mandatory)

4.3. Commercial terms must be such that The Open Group can recover from commercial revenue any costs it incurs in respect of ownership, support, maintenance, and upgrade (Mandatory)

4.4. The test suite must be distributed in source code (Mandatory)

4.5. A support and maintenance service must be in existence or The Open Group must be enabled to set one up (Mandatory)

5. Assignment of Test Results

In this section the term "results" is used to mean the overall results (sometimes known as "verdicts") produced by a test campaign. The production of such results may in some circumstances require the output of automated testing to be analyzed by automated and/or manual processes. For this reason, it is the combination of test software, analysis tools, and manual procedures that must together be considered as the test suite and assessed for suitability against these criteria.

5.1. The test suite shall meet the criterion of repeatability, in that repeated testing of the same implementation shall produce results that are consistent with those produced on the first occasion (Mandatory)

5.2. The test suite shall meet the criterion of reproducibility, in that it shall produce objective rather than subjective results, so that testing of the same implementation by two different test operators who do not collude shall lead to the results produced by the one operator being consistent with those produced by the other (Mandatory)

5.3. The test suite shall meet the criterion of impartiality, in that the results produced for different tested implementations are free from bias. Thus if different implementations exhibit the same characteristics with regard to some objective test purpose, then when the corresponding test case is run against the different implementations the results shall be consistent with one another (Mandatory)

5.4. If it is necessary for expert analysis of the observations of the test to take place in order to interpret the results or determine results, then there shall be documented, objective procedures sufficient to ensure that repeatability, reproducibility, objectivity, and impartiality are maintained. The procedures shall be such as to ensure that there can be no ambiguity over the test result to be assigned for each test (Mandatory)

5.5. If a test specifies two possible sets of observations either of which could result from the same behaviour on the part of the implementation under test, then the two sets of observations shall give consistent test results. It is not acceptable that one observation shall give rise to a pass result and the other a fail (Mandatory)

5.6. The test suite shall have the property that all test results shall be assigned objectively with respect to the specification against which conformance is being tested (Mandatory)

5.7. The test suite shall be supported by utilities and or documentation that shall ensure that there are objective criteria employed to determine whether or not each test is to be run (Mandatory)

5.8. If expert analysis of the observations leads to modification or qualification of the automatically produced results, then the test report shall be structured in such a way that there can be no ambiguity concerning the test outcome. Such modification or qualification of the results shall not conflict with the test result given by the relevant test. The test report shall be structured such that the justification for each such modification, together with an endorsement by the test operator, shall appear in the output (Desirable)

5.9. The test suite shall be configurable such that it will not produce test results for functions or behavior outside the scope of the specification against which conformance is being assessed (Desirable)

6. Test Suite Usability

6.1. Technical staff skilled in the technology and the implementation under test, but not necessarily in the test technology, should be able to install the delivered test suite source code, build it, and initiate a test run within one working day (Desirable)

6.2. Technical staff skilled in testing, but not necessarily in the technology and the implementation under test, should be able to install the delivered test suite source code, build it, and initiate a test run within one working day (Desirable)

6.3. English shall be the language used for documentation (Mandatory)

7. Test Report

7.1. The result of each test must appear in the report (Mandatory)

7.2. It should be possible to deduce that the test report was produced from a single run of the test suite (Mandatory)

7.3 Test suite documentation should define the mapping between claimed support for any optional or implementation-defined features of the specification, the parameterization of the test suite, and the test results (Mandatory) (that is, where the specification under test allows optional behavior or there is a range in the way a particular feature is supported; for example, limits)

7.4 The test report automatically produced by the test tool shall include as a minimum the following information: (Mandatory)

• the date and time that the test campaign commenced
• the date and time that the test campaign completed
• an unambiguous description of the implementation that was tested
• any abnormalities from standard conditions
• an unambiguous definition of the specification against which conformance is being assessed.

7.5. The test report shall be structured such that any test results for functions or behavior not within the scope of the specification against which conformance is being assessed are clearly differentiated from those that are within scope (Mandatory)

7.6. The format of automatically generated test reports shall conform to POSIX .3 (Desirable)

7.7. The test suite must generate a formatted report in accordance with the Test Development Style Guide. If TET-based, a report writer must be provided to take journal file output from the TET and produce a formatted report (Desirable)

7.8. There should be a results summary to enable the conformance status of the tested implementation to be readily apparent, preferably on one page (Mandatory)

7.9. Verdicts other than PASS should cause the relevant part of the test specification (assertion), the test strategy, and the observed results to be included in the report (Mandatory)

7.10. Test reports should include information on the settings of all parameters and configuration options (Mandatory)

8. Test Suite Acceptance

8.1. The Open Group acceptance procedures shall be applied for acceptance and the supplier shall meet the requirements therein (Desirable)
See http://www.opengroup.org/testing/testprocs/

8.2. The test suite supplier shall assert compliance or otherwise against each of the requirements whether mandatory of desirable herein. The Open Group shall at its discretion audit such compliance assertions (Mandatory)

8.3. The test suite supplier shall supply support and maintenance during the acceptance period free-of-charge (Mandatory)

8.4. The test suite supplier shall have a direct Internet connection and shall use this medium for all routine communications and shall deliver the test suite and documentation by ftp (Mandatory)