The LSB-FHS Verification Suite Development Process

The LSB-FHS test suite, tests the filesystem hierarchy aspects of the Linux Standard Base, as defined in the FHS 2.0 specification.

The process for developing the test suite involves a multi-step process where objectivity, and relationship to a written specification are the key.

The first step is development of a test specification which should undergo formal review. Secondly, a test suite based on that specification. There are criteria for both the specification and the test suite that should be met (see criteria). At regular points there should be feedback from the customer audience / underlying specification owners to verify the correct assumptions.

In the case of the LSB-FHS test suite, the test specification is based on the FHS 2.0 specification, see www.pathname.com, as modified by the requirements placed by the LSB specification (for example LSB mandates presence of the X Window system which is optional in the FHS).

The test specification follows the POSIX 1003.3 guidelines and consists of a set of assertions , that is a set of statements which evaluate as true or false based on the specification under test. This can be thought of the translation between the natural language specification and the testable specification.

For example, the FHS has a statement in section 3.1:

Required files for /bin
The following commands have been included because they are essential. A few are present because of their traditional placement in /bin.
{A list of 34 commands}

This is written as 34 separate test assertions , for example

Reference 3.1-3 (A)
The implementation provides an exec-able version of the cat 
utility in the /bin directory. 
Result: PASS/FAIL

Reference 3.1-4 (A)
The implementation provides an exec-able version of the chgrp 
utility in the /bin directory. 
Result: PASS/FAIL

etc.
The above example is a simple case, more complex examples are for conditional tests etc. The key is to produce logical, testable statements that can be reasoned about, these are called test assertions. By producing test assertions, we can then produce tests.

In order for development to proceed in an objective fashion , review guidelines and a pro-forma response are used for getting feedback from the community. A one month review was held to check the correctness of the assertions with respect to the two specifications. The review group was the union of the lsb-spec and lsb-test mailing lists.

Review comments are then collated and then a set of proposed resolutions produced . Reviewers review the comments in the light of the specifications being tested and consider where applicable using one of the following pro-forma resolutions:

  1. Accepted as stated in action.
  2. Accepted with the following changes to the suggested action....
  3. Rejected: The LBS requires that this feature be present ....
  4. Rejected: The FHS requires that this feature is present .... (for example /var/mail is /var/mail).
  5. Rejected: the problem statement is not specific

    The review comments together with the proposed resolutions are then circulated to the review group for confirmation.

    This allows test development to proceed based on the specifications. At this point an iterative process occurs as test assertions are validated or shown to be false. This may be due to one an incorrect assumption by the author of the test assertion, or in some cases what is now considered an error in the underlying specification. The latter should only be decided by the authors of the specifications, and not the test suite author, and these issues should be fed back to the specification owners so that corrective action can be taken. The test suite will note the issue in its output , but the final correct strategy cannot be undertaken until the specification owners have issued corrections or possibly a version of the specification.

    The timeline of development has been as follows:

    1. Dec 1 1998 Review LSB and FHS specifications
    2. Develop Draft 1 test assertions
    3. Dec 11th 1998 Commence public review of test assertions
    4. Dec 16th 1998 Commence initial test development
    5. Jan 11th 1999 Test assertion review period ends
    6. Jan 11-18 1999 Sort review comments by section and generate initial proposed resolutions. Circulate proposed resolutions to the LSB test and LSB specification mailing lists for feedback
    7. Jan 18 1999 Release initial test suite (0.52) based on the draft 1.0 test assertions. The benefit of releasing real tests is that it tends to concentrate reviewers minds on any specification issues.
    8. Jan 18 - Feb 2 incorporate feedback from LSB specification team to proposed resolutions. Let discussions continue for contentious issues.
    9. Jan 31 - Send specification issues to Dan Quinlan as feedback for possible FHS revision
    10. Feb 2 - incorporate concensus changes into test suite , some tests will generate warnings for now until LSB/FHS spec changes decided. Release 0.55 version.
    11. Feb 19 - the beta 0.9 is released.
    12. We are looking for test data on as many platform/architecture combinations to identify any platform dependencies/bugs in the testcases during the next 4-6 weeks while we lead up to the 1.0 release.

    Criteria for Test Development


    Review Instructions

    The following review instructions were used (these are based on the X/Open Aardvark bug eater format used for some specification development).

    Comments should take the following format, which allows us to
    collate comments by section.
    
    Subject: BUG in LSB-FHS-TS_SPEC_V1.0
    
    @ section ,  assertion_reference #
    
    Problem:
    
    Explain why here.
    Be sure to add sufficient explanation for someone not
    familiar with the problem to be able to make a decision.
    
    Action:
    Be specific, that is give precise editorial instructions for change.
    
    For example
    
    @ /foo,  X.13-1
    
    Problem:
    /foo/dp may not exist if the dp extension is not
    supported. The LSB spec says clearly that the dp extension
    is optional.
    
    Action:
    
    Make this a conditional test, replace the existing text with
    
    "If the implementation provides the dp extension then
    	the /foo/dp directory exists and is searchable"