Guide to COE Platform Certification

Copyright © 2002-2004 The Open Group, All Rights Reserved

Issue 2.1, November 2 2004

1.  Introduction

1.1  Summary of Web References

2.1  Understanding the Program
2.2  Confidentiality
2.3  Preparation for Certification
2.4  Certification
2.5  The Open Brand Trademark License Agreement (TMLA)
2.6  The Open Group COE Platform Certification Agreement

3.1  The First Step - Completing the Trademark License Agreement
3.2  The COE Platform Conformance Statement
3.3  Submitting Initial Certification Information
3.4  Initial Submission Check
3.5  Completing Additional Conformance Statements
3.6  Formal Testing for Certification
3.7  Completion of Certification
3.8  The Appeals Process

4.1 The Renewals Process
4.2 Certified Product Updates

8 May 2003, Issue 1.0, Initial Publication.

20 October 2003, Issue 2.0, Add in COE Linux Platform.

2 November 2004, Issue 2.1, Add in COE Platform V2.

1. Introduction

This document is a guide for suppliers who want to submit products for COE Platform certification within the Open Brand Certification Program. This document guides you through all the steps required for certification.

COE Platform certification is given to those products or range of products that have been certified by The Open Group as having demonstrated their conformance to the applicable Conformance Requirements. The detailed requirements for conformance are identified in the applicable COE Platform Product Standards and their subsidiary Product Standards. Only suppliers of Certified Products may use the Open Brand certification trademark in connection with their product.

The COE Platform Certification Policy governs COE Platform certification. The use of the Open Brand certification trademark is governed by the Open Brand Trademark License Agreement (TMLA). The COE Platform Certification Agreement covers the terms and conditions of the certification service. You must agree to the terms and conditions outlined in all three documents prior to certification. For your convenience, significant highlights of the COE Platform certification process are outlined below, and the following sections give more comprehensive information.

1. The Certification Authority for COE Platform certification is The Open Group.
2. You will be granted a license to use the Open Brand certification trademark in connection with your product when your product has passed the applicable certification Test Suite(s) and you have formally agreed to all the required terms.
3. The TMLA must be signed by both an authorized signatory from your company and The Open Group in order to have a product certified.
4. A certified product is subject to re-certification on a periodic basis as stated in the COE Platform Certification Policy.
5. All product information supplied to the Certification Authority will be treated as confidential, as required by the COE Platform Certification Agreement.
6. The Open Group Certification Authority may, from time to time, request proof that your product remains in compliance with the requirements outlined in the COE Platform Conformance Requirements, as required by the TMLA.

At the end of this document is a Certification Checklist to help you make sure that your product registration is complete.

Note: In the event of any conflicts, this guide defers to the COE Platform Certification Policy, the Open Brand Trademark License Agreement (TMLA), the COE Platform Product Standards, and the COE Platform Certification Agreement.

1.1 Summary of Web References

A summary of the external web references is given below:

2. How to Achieve COE Platform Certification

This guide details the steps you are required to take to achieve certification, and provides information on how to perform each of these steps.

The process involves:

• Understanding the certification program and process
• Testing the product informally
• Applying for certification
• Formal testing and submission of results

Note: Certification cannot commence until all the requirements for a COE Platform are met, including the porting of the GSKS and other associated preparation.

2.1 Understanding the Certification Program

To become familiar with the program, you should read the following program documents:

• The COE Platform Certification Policy
• The COE Platform Product Standards
• The Practical Guide to The Open Brand

(Link to Diagram describing the Elements of the COE Platform Certification program )

(Link to Diagram describing the Elements Mapping from the former DISA COE Program to the COE Platform Certification program )

2.1.1 The COE Platform Certification Policy

It is best to start with the COE Platform Certification Policy, which is the foundation of the program. It provides information on the types of products that can be certified in the program, what it means to be certified, what is required to get a product certified, and how to make sure that a product remains certified.

The COE Platform Certification Policy defines two classes of product for COE Platform certification:

• POSIX® -based systems
• Linux® -based systems

These definitions are based on the DISA Common Operating Environment (COE) Cross Application Platform Compliance Criteria document available at http://www.disa.mil/ges/coe_kpc/index.html.

Certification in this program applies to families of binary-compatible systems, from which specific platforms may then also be certified to a higher degree of assurance which involves an on-site witness test campaign , if required by the applicant. The two levels of certification are referred to in the program as:

• Product Family certification
• Specific Platform certification

2.1.2 The COE Platform Product Standards (POSIX® -based systems)

The COE Platform Product Standards should be read to understand the detailed Conformance Requirements against which a product can be certified. Product Standards provide a mapping between certification, specifications, and the Test Suites needed to demonstrate conformance.

There are two available product standards, Version 1 and Version 2. The primary difference is that Version 2 is an update of Version 1 to include realignment with the Single UNIX Specification Version 3.

The COE Platform Product Standard is built out of the following three subsidiary Product Standards:

• COE Platform Government-Supplied Kernel Source Product Standard
• COE Platform Security Product Standard
• COE Platform Standards Product Standard

(Link to Diagram describing the COE Platform Product Standards)

You should also review Section 3, Conformance of the COE Platform Certification Policy.

2.1.3 The COE Linux Platform V1 Product Standard (Linux® -based systems)

The COE Linux Platform V1 Product Standard should be read to understand the detailed Conformance Requirements against which a Linux-based product can be certified.

The COE Linux Platform V1 Product Standard is built out of the following three subsidiary Product Standards:

• COE Linux Platform Government-Supplied Kernel Source Product Standard
• COE Platform Security Product Standard
• COE Linux Platform Specifications Product Standard

(Link to Diagram describing the COE Platform Product Standards)

You should also review Section 3, Conformance of the COE Platform Certification Policy.

2.1.4 The Practical Guide to The Open Brand

COE Platform certification is part of the Open Brand certification program. A link to the Practical Guide to the Open Brand is available at http://www.opengroup.org/openbrand/. The Guide is designed primarily for those who wish to apply to register products within the Open Brand certification program, although it will be of value to those who, for procurement purposes, wish to examine the requirements and establish the exact terms for product registration. The Guide provides a description of the operation of the Open Brand for all available Product Standards. For readers unfamiliar with the Open Brand, we recommend reading the Introduction, referring as necessary to the individual chapters for an explanation of Product Standards, Conformance Statements, Indicators of Compliance, Testing, and so on.

2.2 Confidentiality

Your company and product details are absolutely confidential between you and The Open Group. This is effective once you accept the COE Platform Certification Agreement, which is a prerequisite to registering a product. The Open Group does not make any certification information available to any third party, including consultants or members, without written permission of the supplier.

Upon successful completion of the certification process, the certified product will be included in the publicly available Certification Register at http://www.opengroup.org/regproducts/. However, to enable you to launch a Certified Product, you may request that certification remain confidential for up to six months from the date of written notification by the Certification Authority that a product has achieved certification. See Section 11.3 of the COE Platform Certification Policy for further information.

2.3 Preparation for Certification

The objective of The Open Group COE Platform certification program is to encourage and facilitate the development and market availability of products that meet the COE specifications. The Test Suites and test procedures have a dual role in this context: first, they help developers and quality assurance teams to ensure the product meets the specifications and, second, they are used to ensure that only the eligible products are marketed as compliant.

Prior to applying for certification you should perform internal development. This includes porting the GSKS code, tuning the implementation to meet the security and other requirements, and preparing the COE supplier documentation.

You should read the COE Platform Product Standards referring to the Indicators of Compliance section to identify the required certification tests. The certification tests comprise automated Test Suites, as well as manual validation procedures. You should then perform quality assurance testing using the conformance Test Suite(s), test procedures, checklists, and any additional verification methods you deem appropriate, to ensure the product meets the applicable Conformance Requirements and is ready for entry into the certification program.

2.3.1 Authorized Test Suites

You should note that, for formal testing, only currently authorized versions of the COE Platform certification Test Suites and procedures can be used. So, for informal testing it is vital that you use an authorized version. The Open Group maintains a list of the currently authorized certification Test Suites at http://www.opengroup.org/openbrand/coe/docs/testsuites.html. This document also includes information on how to obtain the test suites.

Below is information on how to find further information on the Test Suites and where to report any bugs.

2.3.2 Problems and Problem Reporting

If you have problems with running the certification Test Suites or Manual Validation Procedures, first read the documentation provided with the certification Test Suites and Manual Validation Procedures . If this does not answer your questions, check the Frequently Asked Questions file at http://www.opengroup.org/openbrand/coe/docs/faq.html. If you still cannot find an answer, report the problem to the Test Suite Maintenance Authority (TSMA) via http://www.opengroup.org/testing/support/.

If you are using one of the authorized Test Suites or Manual Validation Procedures and getting failures but are certain that your product conforms to the COE Platform specifications, check the Problem Report information database held at http://www.opengroup.org/openbrand/PR. The online Problem Report database contains known information on existing bugs and interpretations of the specification. The issue may already be resolved by a COE Platform specification Interpretation, or by an agreed Test Suite Deficiency. If it is not, you should raise a new Problem Report; see http://www.opengroup.org/openbrand/PR.

2.4 Certification

Certification is a formal process. Under the terms of the TMLA, you, as a supplier of a Certified Product, warrant and represent that the product meets all the Conformance Requirements applicable to the Product Standard against which it is certified and implements all the features claimed in the Conformance Statement.

The obligations, terms, and conditions of certification are fully set out in the following documents:

• The COE Platform Certification Policy
• The Open Brand Trademark License Agreement (TMLA)
• The COE Platform Certification Agreement

These documents are available via The Open Group COE Certification web page at http://www.opengroup.org/openbrand/coe/.

It is necessary to read these documents before you start the certification process in order to fully understand the policies and requirements.

2.5 The Open Brand Trademark License Agreement (TMLA)

The Open Brand Trademark License Agreement requires signature. It is recommended that you commence the process to complete the TMLA as soon as possible to save delays later in the certification process. A TMLA must be completed before a product can complete the certification process and be entered onto the Certification Register.

The TMLA only needs to be entered into once per organization. When it is in place, multiple products can be certified. If you are an existing customer in the Open Brand program, your company has already entered into this agreement. To confirm this, contact the Certification Authority.

2.6 The COE Platform Certification Agreement

You are required to agree to the COE Platform Certification Agreement for each product registration.

The COE Platform Certification Agreement defines the certification service and the legal commitment to the conditions of the service. If the type of certification you are applying for includes a Specific Platform certification, then you should select that option on the Certification Agreement. This will involve scheduling an on-site witness test procedure with The Open Group.

3. Product Registration

3.1 The First Step: The TMLA

Prior to commencing a product registration, you should submit a completed Open Brand Trademark License Agreement (TMLA) to The Open Group.

3.2 The COE Platform Conformance Statement

The next step is to complete the COE Platform Conformance Statement to describe your product and how it meets the Conformance Requirements. Your Conformance Statement will be linked into the Certification Register entry for the product once it is certified. The template Conformance Statement Questionnaire (CSQ) for the COE Platform V1 Product Standard is available from http://www.opengroup.org/csq/csqdata/blanks/CP1.html. The template Conformance Statement Questionnaire (CSQ) for the COE Platform V2 Product Standard is available from http://www.opengroup.org/csq/csqdata/blanks/CP2.html. The template Conformance Statement Questionnaire (CSQ) for the COE Linux Platform V1 Product Standard is available from http://www.opengroup.org/csq/csqdata/blanks/CL1.html.

You must complete the CSQ to create a Conformance Statement for your product. See Sections 1.2, Section 2, and Section 3.3 of the COE Platform Certification Policy for further information on the purpose of the Conformance Statement.

You must enter your Conformance Statement online in The Open Group CSQ system at http://www.opengroup.org/csq/. Please contact the Certification Authority for information on establishing a corporate account in this system (note that this is a separate account to any personal accounts you might already have on The Open Group Web server).

The Conformance Statement includes information about the product and product family to be certified, which includes at least the following:

• The product name, version/release number, and product supplier for each product required to meet the conformance requirements.
• A description of all the members of a binary-compatible family of products for which this certification applies, including configuration instructions.
• The type of certification requested, whether it is for a Specific Platform within a Product Family certification
• The platform with which the interoperability validation procedures will be executed (which should be an existing COE Certified Platform)
• Additional information related to the test campaign and platform options.

3.3 Submitting Initial Certification Information

Applying for certification requires you to submit some initial information to The Open Group. The initial submission involves sending a covering letter (including your company details and contact details), completing a Registration Form for the COE Platform Product Standards and constituent Product Standards, submitting a completed and signed COE Platform Certification Agreement, and supplying payment details (a cheque or purchase order number).

You must also submit a COE Platform Test Campaign Definition form. Where a Product Standard allows multiple Indicators of Compliance, you must select your choice of the Indicator of Compliance, and if required select the Witness Test option for a Specific Platform certification.

3.4 Initial Submission Check

Once the above stages have been completed, the Certification Authority will perform an initial check of the completeness and correctness of the submission, including the Conformance Statement.

3.4.1 Confirmation of the Test Campaign Definition

If your submission is complete and correct, the Certification Authority will confirm within six working days the definition of the specific test campaign for your product registration. If you have selected that your certification include a Specific Platform certification, an on-site witness test procedure will also be scheduled to take place. If this is a first test campaign for a Product Family then the on-site witness test procedure is scheduled after the full submission for the Product Family certification has been received and provisionally assessed.

3.5 Completing Additional Conformance Statements

If you have selected Indicators of Compliance that require additional Conformance Statements to be completed, the Certification Authority will advise you at this stage.

The next steps are to perform formal testing and then submission of the test results. Where test result data is available in electronic form it should be uploaded via ftp to The Open Group (please contact the Certification Authority who will establish an ftp account for this purpose). The Manual Validation Procedures should be submitted in hardcopy.

3.6 Formal Testing for Certification

For a Product Family certification, formal testing is a self-test activity with results submitted to the Certification Authority for audit. At this point, you need to make sure you understand the required test campaign that has been defined by the Certification Authority; that is, which Test Suites and procedures are required to be used for certification testing (see below for an outline of a typical campaign). You also need to make sure that you have obtained a currently authorized version of the certification Test Suite(s), since other versions will not be valid for formal testing.

You will need to complete and submit to the Certification Authority, a Candidate Platform Initial Validation Procedure and a Validation Host Initial Validation Procedure. A template is supplied for these procedures.

For a Specific Platform certification, formal testing is self-testing partly witnessed on-site by The Open Group. A prerequisite is that the system under test has completed a full submission for a Product Family certification with all issues resolved. At this point the Specific Platform certification can then be undertaken in accordance with its associated on-site test witness procedures. Ordinarily only a subset of the complete campaign may be witnessed in this way, but you should be prepared for all the testing to be witnessed or audited on-site.

If the type of certification you are applying for includes a Specific Platform certification, then the 25 working days for completion of the audit of the full submission (which includes the Product Family submission and the Specific Platform submission) commences at completion of the on-site witness test procedure.

You can obtain help and information as described in Preparation for Certification.

3.6.1 Test Journals from Automated Test Suites

Test results submitted from the automated Test Suites should always include the full journal output from an uninterrupted run of the applicable Test Suite. If this is not possible, a Problem Report must be raised stating the nature of the problem and you should wait for a response from the Certification Authority before proceeding further.

When submitting test results for audit, you are required to upload test journals for each applicable Test Suite. If there are failed test results, then you need to provide a text file giving references to granted Problem Reports. For automated Test Suites that produce a test journal in a format other than the Test Environment Toolkit (TET) journal, the Certification Authority will define the exact requirements in the test campaign definition.

The rest of this section applies to automated Test Suites that produce a TET Journal.

The failed test results in a test journal that require resolution are those with a status of FAIL, UNINITIATED, UNRESOLVED, or UNREPORTED. The resolution process requires you to provide a valid reference to an approved Problem Report from the Problem Reporting database for each failed test case. An approved Problem Report is one which has resulted in an agreed Interpretation of the specification, a Test Suite Deficiency, or a Certification System Deficiency. These apply to specific releases of the specification, Test Suite, or certification system.

Any other test results in the journal that require manual resolution are denoted FIP (further information provided). The resolution process requires you to provide an assertion that the test case resolves to a PASS result for your system, and optionally to provide brief commentary. No reference is required to an approved Problem Report for FIP results.

3.6.2 Reports from a Manual Validation Procedure

Many of the tests for COE Platform certification involve a Manual Validation Procedure. In these cases, you are required to produce a report in accordance with the validation procedure and submit it to the Certification Authority for audit.

For each manual validation procedure, the test procedures must each be executed in the sequence specified in the procedure's table of test procedure items. The "Observed Results" of each test item in the test procedure table must be recorded in the table, and the completed tables are submitted as a component of the certification process.

Determination of Overall Test Results for Manual Validation Procedures

The overall PASS/FAIL result for a manual test procedure is determined as follows: The overall test result is "PASS" if and only if all test items that have PASS/FAIL condition in the "Observed Result" column, have a test "PASS" result.

Determining a Manual Validation Procedure Test Item's Results

For each test item in a manual validation test procedure table with a "Circle One: PASS/FAIL" in the Observed Result column, the tester shall compare the directly observed Candidate Platform behavior with the "PASS" criteria. Unless other PASS criteria are noted in the test item, the test item's result is "PASS" if and only if the Candidate Platform presents all of the behaviors and conditions described in the "Expected Result" column for that test procedure item. The test procedure item's result is "FAIL" if the PASS criteria have not been satisfied.

If the test item results are "PASS", the tester will circle "PASS" and note the actual behaviors and conditions presented in the "Observed Result" column. If the observed result precisely matches the expected result, the Observed Result column for the test item may contain the statement "as expected."

If the test procedure item results are "FAIL", the tester will circle "FAIL" in the Observed Result column and requires a valid reference to an approved Problem Report from the Problem Reporting database. An approved Problem Report is one which has resulted in an agreed Interpretation of the specification, a Test Suite Deficiency, or a Certification System Deficiency. These apply to specific releases of the specification, Manual Validation Procedure, or certification system.

3.6.3 Test Campaign Definition

For an initial certification, a standard set of Indicators of Conformance are defined in the Product Standards. The test campaign definitions are drawn up from this set by the Certification Authority, and will have been supplied earlier in the certification process. A summary follows of the Indicators of Conformance by Product Standard (this also notes whether the Test Campaign is applicable to POSIX or Linux-based systems or both):

For renewals and certification updates, you should refer to Renewals and Certified Product Updates for policy on how the Certification Authority will define the applicable test campaigns.

3.7 Completion of Certification

If your submission is complete, which includes the TMLA and payment details being in place, you will be notified of successful certification within 25 working days. If the type of certification you are applying for includes a Specific Platform certification, then this 25 working days is from completion of the on-site witness test procedure.

If for any reason the submission was not complete, you will be notified so that any corrections can be made and resubmitted. Such a resubmission needs to be done within 60 calendar days unless an extension has been granted by the Certification Authority. The initial fee covers one set of corrective actions.

The details of the Certified Product will then be put on the Certification Register, which is a public document, unless you have requested that it remains confidential. To keep the certification confidential, this option should be selected on the Registration Form at the time the product is submitted.

When a product certification has been made public, you will be sent a certificate by electronic mail.

You should note that certification is valid for a defined period, as stated in Section 8.2, Duration of Certification of the COE Platform Certification Policy. At the end of that period, if you wish your product to remain certified, you will need to renew your product's certification (see Renewals and Certified Product Updates for more information).

3.8 Appeals Process

You may appeal decisions made by The Open Group as stated in Section 10, Appeals Process of the COE Platform Certification Policy. Any request for appeal should be sent by electronic mail to conformance@opengroup.org. Receipt of such a request will be acknowledged within six working days by the Certification Authority and the appeals process invoked.

4. Renewals and Certified Product Updates

4.1 The Renewals Process

A certified product has a defined period for its initial certification, after which it must be renewed or the product will no longer continue to be certified. The Certification Authority will notify you approximately two months in advance by electronic mail (so it is important to maintain your contact information) when a renewal is due.

Renewal implies that your product continues to conform and that you will continue to support your product for the duration of the renewal period.

Renewals are undertaken by contacting the Certification Authority, who will supply a renewal form for you to complete.

You may be required to re-test your product. This depends on whether the Certification Authority is aware of any issues that would highlight the need to re-test. If you are required to re-test, the Certification Authority will inform you of the test requirements, and allow you to upload new test results for your product.

You can then elect to renew for the next certification period.

4.2 Certified Product Updates

If you make any changes to your Certified Product, you may be required to re-test or re-certify as required by Section 7, Scope of Certification and Requirements for Product Re-Test and Re-Certification of the COE Platform Certification Policy. You should read that section thoroughly since it describes multiple scenarios related to product changes, some of which require a certification-related activity. In some cases, a complete "full certification" may be required if the update is considered to be equivalent to a new product in certification terms.

 5. Certification Checklist

The following checklist is provided to assist with completion of your product registration:

POSIX is a registered trademark of the IEEE