The Open Group : Making Standards Work
About The Open Group The Open Group Forums Certification Collaboration Services
HOME   |   SITE MAP  
Sponsor an Event Become a Member of The Open Group The Open Group Member Area
Government Programs Events Bookstore & Downloads Newsroom Contact The Open Group
You are here: Home >Testing >Test Suites >VSX4 Overview
Test Suites
 
VSX4 Overview

The XPG4 Core Operating System Calls and Libraries Test Suite

Summary

  • Verifies the core operating system calls and libraries.
  • Built upon the proven VSXgen harness, yielding a configurable and productive tool.
  • Using the VSXgen architecture, this suite is provided as a VSXgen Testset Package.
  • Over 10,000 tests.
  • Test Coverage: IEEE POSIX 1003.1-1990 (ISO 9945-1:1990), FIPS 151-2, XPG3, XPG4, UNIX 98.
  • Test Coverage also available for POSIX 1003.1,2003 and UNIX 03 in the Westwood version.
  • Mandatory for testing XPG4 Base, XPG4 Base 95 and the Single UNIX® Specification (including Versions 1, 2 and 3).
  • An embedded test capability is also available in the form of the derivative VSX4-PSE test suite
  • Derivative test suites providing a subset of the full VSX4 test suite are: VSX-PCTS1999, VSX-PCTS2003 and the LSB-VSX.

This test suite is one of the most widely used test suites in the industry. If your organization is developing an operating system supporting at least POSIX.1 then this test suite gives significant test coverage and is recommended for in-house quality-assurance as well as part of your branding campaign.

Top



Introduction

VSX4 is the descendant of X/Open Company's first test suite, VSX2. Developed continuously since 1988 to test the System Call interfaces, VSX4 is a mature and dependable test suite that verifies the core operating system calls and libraries.

It comprises over 10,000 tests (approx 750,000 lines of source code), giving coverage of the IEEE POSIX 2003.1 test assertions as well as providing extensive tests for the X/Open extensions beyond POSIX, including internationalization, and header file namespace tests. It also includes tests for traditional ``Common Usage'' C language compiler syntax and tests for the IEEE POSIX 1003.2 C Language Binding interfaces. VSX4 is mandatory for testing XPG4 Base, XPG4 Base 95, UNIX 95, UNIX 98 and UNIX 03.

It has the following test modes:

  • IEEE POSIX 1003.1-1990 (also known as ISO/IEC 9945-1)
  • X/Open Portability Guide Issue 3 (XPG3)
  • X/Open Portability Guide Issue 4 (XPG4)
  • UNIX 98

Additionally the Westwood version provides:

  • IEEE POSIX 1003.1, 2003 Edition(also known as ISO/IEC 9945-1:2003)
  • UNIX 03

In POSIX mode VSX4 can be run as a POSIX Conformance Test Suite (PCTS), and as such is the recognized certification test tool in Europe. For the Open Brand, the test suite is run in one of the X/Open modes (XPG3 , XPG4 , UNIX 98 or UNIX 03) which are supersets of the POSIX test mode (the X/Open XPG and UNIX Specifications defer to POSIX.1).

An enhanced version known as VSX4 +XNFS is available for testing the Network File System (NFS) specification, including tests for system interfaces over NFS, as well as the NFS protocol itself.

VSX4 uses the robust VSXgen test harness, which in turn uses the industry standard Test Environment Toolkit (TET) . The benefits of using VSXgen, which is derived from an earlier version of VSX4, is that the harness has proven portability and robustness characteristics and it allows the VSX4 tests to be integrated with other testsets that run under VSXgen, making the test suites easily useable for in-house quality assurance and automated regresssion testing.

The TET is described on the TETworks web site.

Top


Disk Space Requirements

Disk space requirements on a typical CISC based system are as follows:

  • Source code - 25 Mbytes
  • Source and tests built - 100 Mbytes (on a CISC system)

Top


Numbers of tests, and time to run

  • POSIX.1-1990, POSIX.1-2003 mode - over 6000 tests
  • XPG4/UNIX 98/UNIX 03 Mode - over 10,000 tests

A typical test run from start to finish takes approximately 15 hours on an Intel Pentium based system.

Top


Test Coverage

The VSX test suite verifies compliance of systems to The Open Group Base Specifications for core kernel and library interfaces.  VSX features include:

  • Complete coverage of the IEEE POSIX 2003.1 test assertions.
  • Extensive tests for the XSI extensions over POSIX, including internationalisation, and header file namespace tests.
  • Extensive tests for traditional C language compiler syntax (K&R C).
  • Test coverage for the IEEE POSIX 1003.2 (1003.1-2003) C Language Binding interfaces.

The source tree and the executable tree are organised in a hierarchical design, being divided into sections corresponding approximately but not exactly to boundaries between various standards that is POSIX, ANSI C [ISO C], and X/Open.  The testsets may be classified as follows:

  • System Interface Tests
  • Header File Tests
  • Commands Tests
  • Programming Language Tests

Top


Modular Design

VSX4 is modular, both at the implementation level and at the user level. The testing process is divided into clear stages -

Each stage is clearly documented in the VSX User and Installation Guide, which accompanies the product. 

Preparation

This stage includes setting up user-ids and groups for testing, and reading the tape. This stage includes configuring and installing the TET harness, which is used to control building, execution and cleanup of VSX tests.

Configuration and Installation

VSX provides a configuration script that interrogates the user and the system, checking for availability of certain resources. If these resources are not present then stub routines are supplied. This enables a non-compliant system to be tested.

After configuration, the installation stage builds the test harness and associated libraries.

Test Building

VSX separates the building of the test harness and tools from the testsets themselves. The test suite may be built completely or in sub-sections, which decrease in granularity from test sections and areas down to individual testsets.

Test Execution

Once built, the test suite may be run as a whole in one pass, or in sub-sections which decrease in granularity from test areas down to individual tests within testsets. If you are producing an X/Open branding application the test suite must run in a single uninterrupted pass.

Test Reporting

VSX generates an intermediate level journal file in a TET compliant format. This is sufficient for debugging purposes. When a formal report is required, such as for X/Open branding then the formal report writers provided with VSX4 are used to process the journal file.

Test Organisation

The source tree and the executable tree are organised in a hierarchical design, being divided into sections corresponding approximately but not exactly to boundaries between standards. There are ten sections, the name suffix giving an indication of the class of tests contained therein. Within the test section, there is a further subdivision into areas, which then contain related testsets.

For example the section that tests the IEEE POSIX assertions is denoted POSIX.os.

The testsets may be classified as follows:

  • System Interface Tests
  • Header File Tests
  • Commands Tests
  • Programming Language Tests

For each of the classifications the sections of the test suite are described together with a list of testsets for each section. Testsets denoted _X are for testing X/Open extensions beyond POSIX 1003.1-1990. VSX4 contains tests for macro versions of interfaces. In this case the subsection begins with a capital M, for example: POSIX.os/MCsupport/tzset tests the macro version of tzset.

    System Interface Tests

    The sections to test system call and library interfaces have the suffix os.

    There are four os sections within VSX4:

    • POSIX.os
    • XOPEN.os
    • ANSI.os
    • XPG4.os

    These are described in detail in the following sections.

    POSIX.os

    This includes all the POSIX 2003.1 assertions excluding the majority of the P2003.1 chapter Language-Specific Services for the C Programming Language assertions which are located in the ANSI.os section. There are additional assertions that test X/Open extensions to POSIX (denoted by the suffix _X) and these are automatically excluded if POSIX only testing is required.

    The POSIX tests are divided into the following areas:

    • POSIX.os/Csupport

      This contains some of the assertions in the P2003.1 chapter Language-Specific Services for the C Programming Language. Assertions which are in both in this chapter and the ISO C standard are included in the ANSI.os section. The testsets in this area follow:


       fdopen  fileno  siglongjmp  sigsetjmp  tzset
       tzset_X
    • POSIX.os/dataform

      This contains the assertions in the P2003.1 chapter, Data Interchange Format, which includes tests for the following functions:


       cpio  tar

    • POSIX.os/devclass

      This contains the assertions in the P2003.1 chapter, Device- and Class-Specific Functions which includes the following testsets.


       c_cc  c_cflag  c_iflag  c_iflag_X  c_lflag
       c_oflag  c_oflag_X  cfgetispee  cfgetospee  cfsetispee
       cfsetospee  i_access  i_canon  i_close  i_ctty
       i_inproc  i_modem  i_noncanon  i_spchars  tcdrain
       tcflow  tcflush  tcgetattr  tcgetpgrp  tcsendbrea
       tcsetattr  tcsetpgrp

    • POSIX.os/files

      This contains the assertions in the P2003.1 chapter, Files and Directories which includes the following testsets:


       access  access_X  chdir  chmod  chown
       chown_X  closedir  creat  creat_X  fpathconf
       fstat  getcwd  link  mkdir  mkfifo
       open  open_X  opendir  pathconf  readdir
       rename  rename_X  rewinddir  rmdir  stat
       stat_X  umask  unlink  unlink_X  utime

    • POSIX.os/ioprim

      This contains the assertions in the P2003.1 chapter, Input and Output Primitives, which includes the following testsets :


       close  dup  dup2  fcntl  fcntl_X
       lseek  pipe  read  write

    • POSIX.os/procenv

      This contains the assertions in the P2003.1 chapter, Process Environment, which includes the following testsets:


       ctermid  getegid  getenv  geteuid  getgid
       getgroups  getlogin  getpgrp  getpid  getppid
       getuid  isatty  isatty_X  setgid  setpgid
       setsid  setuid  sysconf  sysconf_X  time
       times  ttyname  uname

    • POSIX.os/procprim

      This contains the assertions in the P2003.1 chapter, Process Primitives, which includes the following testsets:


       _exit  alarm  exec  exec_X  fork
       fork_X  kill  kill_X  pause  sigaction
       sigaddset  sigconcept  sigdelset  sigemptyse  sigfillset
       sigismembe  sigpending  sigprocmas  sigsuspend  sleep
       wait  waitpid

    • POSIX.os/sysdb

      This contains the assertions in the P2003.1 chapter, System Databases, which includes tests for the following functions:


       getgrgid  getgrnam  getpwnam  getpwuid

    ANSI.os

    This includes all the C Languagage specific P2003.1 assertions plus many additional assertions based on the text of the XPG and ISO C (was ANSI C).

    The ANSI tests are divided into the following areas:

    • ANSI.os/charhandle

      This area contains tests for character handling routines and has the following testsets:


       isalnum  isalnum_X  isalpha  isalpha_X  iscntrl
       iscntrl_X  isdigit  isgraph  isgraph_X  islower
       islower_X  isprint  isprint_X  ispunct  ispunct_X
       isspace  isspace_X  isupper  isupper_X  isxdigit
       tolower  tolower_X  toupper  toupper_X

    • ANSI.os/diagnostics

      This contains a single testset.

      assert

    • ANSI.os/genuts

      This sections contains tests for general functionality and has the following testsets:

       abort  abs  atof  atoi  atol
       bsearch  calloc  exit  free  malloc
       qsort  rand  realloc  srand  strtod_X
       strtol_X  system_X

    • ANSI.os/jump

      This section contains "jmp" related functionality.

       longjmp  setjmp

    • ANSI.os/locale

      This section contains a testset for the setlocale function.

      setlocale

    • ANSI.os/maths

      This contains testsets for the mathematical functions.

       acos  asin  atan  atan2  ceil
       cos  cosh  exp  fabs  floor
       fmod  frexp  ldexp  log  log10
       modf  pow  sin  sinh  sqrt
       tan  tanh

    • ANSI.os/signal

      signal_X

    • ANSI.os/streamio

      This contains testsets for the stdio-streams related functionality.

       clearerr  fclose  feof  ferror  fflush
       fgets  fopen  fopen_X  fputs  fread
       freopen  freopen_X  fseek  ftell  fwrite
       getc  gets  perror  printf  printf_X
       putc  puts  remove  remove_X  rewind
       scanf  scanf_X  setbuf  setvbuf_X  tmpfile
       tmpnam  ungetc  vprintf  vprintf_X

    • ANSI.os/string

      This contains testsets for string related functionality.

       memchr_X  memcmp_X  memcpy_X  memset_X  strcat
       strchr  strcmp  strcoll_X  strcpy  strcspn
       strerror_X  strlen  strncat  strncmp  strncpy
       strpbrk  strrchr  strspn  strstr  strtok
       strxfrm_X

    • ANSI.os/time

      This contains testsets for time related functionality.

       asctime  ctime  ctime_X  gmtime  localtim_X
       localtime  mktime  mktime_X  strftime  strftime_X

    XOPEN.os

    This includes interfaces that are not contained in the POSIX or ANSI standards. For example, popen, ftw, getpass, getopt. Where groups of functions have similar functionality to areas within POSIX.os and ANSI.os, then the areas have been given a similar name, e.g. XOPEN.os/files. This section is for XPG3 based funtionality, there also exists an XPG4.os.

    • XOPEN.os/charhandle

       _tolower  _toupper  isascii  toascii

    • XOPEN.os/files

       dev_null  dev_tty  fsync  seekdir  telldir

    • XOPEN.os/genuts

       crypt  drand48  encrypt  ftw  getopt
       getpass  hsearch  lsearch  pclose  popen
       regexp  setkey  swab  tsearch

    • XOPEN.os/locale

       catclose  catgets  catopen  nl_langinf

    • XOPEN.os/maths

       erf  gamma  hypot  isnan  j0
       lgamma  y0

    • XOPEN.os/procenv

       chroot  cuserid  nice  putenv  ulimit

    • XOPEN.os/streamio

       getw  putw  tempnam

    • XOPEN.os/string

      memccpy

    • XOPEN.os/time

      clock

    XPG4.os

    This section contains tests for XPG4 specific features. This section is not run in XPG3 mode.

    The XPG4 specific system interface tests are divided into the following areas:

    • XPG4.os/genuts

       atexit  div  fnmatch  getopt  glob
       labs  ldiv  regcomp  strfmon  strtoul
       wordexp

    • XPG4.os/ipc

       msgctl  msgget  msgrcv  msgsnd  semctl
       semget  semop  shmat  shmctl  shmdt
       shmget

    • XPG4.os/locale

      localeconv

    • XPG4.os/procenv

      confstr

    • XPG4.os/signal

      raise

    • XPG4.os/streamio

       fgetpos  fsetpos

    • XPG4.os/string

       mblen  mbstowcs  mbtowc  memmove  wcstombs
       wctomb

    • XPG4.os/time

       difftime  strptime

    • XPG4.os/wcharhandle

       iswalnum  iswalpha  iswcntrl  iswctype  iswdigit
       iswgraph  iswlower  iswprint  iswpunct  iswspace
       iswupper  iswxdigit  towlower  towupper  wctype

    • XPG4.os/wgenuts

       iconv  iconv_clos  iconv_open  wcstod  wcstol
       wcstoul

    • XPG4.os/wprintpos

       wcswidth  wcwidth

    • XPG4.os/wstreamio

       fgetws  fputws  getwc  putwc  ungetwc

    • XPG4.os/wstring

       wcscat  wcschr  wcscmp  wcscoll  wcscpy
       wcscspn  wcslen  wcsncat  wcsncmp  wcsncpy
       wcspbrk  wcsrchr  wcsspn  wcstok  wcswcs
       wcsxfrm

    • XPG4.os/wtime

       wcsftime  wcsftime_X

    Header file tests.

    The sections that test the header files have the suffix hdr.

    There are four header file test sections, which are organised in a similar manner to the os tests, except that each section may test the same header file to check for definitions mandated by the different standards.

    The sections are:

    • POSIX.hdr

      This contains testsets for the POSIX header files.

       dirent  dirent_X  errno  errno_X  fcntl
       fcntl_X  general  grp  limits  limits_X
       pwd  signal  signal_X  stat  stat_X
       tar  termios  termios_X  times  types
       unistd_1  unistd_2  unistd_3  unistd_X  utime
       utsname  wait

    • ANSI.hdr

      This contains testsets for the ANSI header files.

       assert  ctype  ctype_X  locale  locale_X
       math  math_X  setjmp  stdio_1  stdio_2
       stdio_X  stdlib  stdlib_X  string  string_X
       time  time_X

    • XOPEN.hdr

      This contains testsets for header files that are defined by XPG3 only.

       cpio  ftw  langinfo  nl_types  regexp
       search  ulimit  varargs

    • XPG4.hdr

      This section contains tests for XPG4 header files. This test section is not run in XPG3 mode.

      The header files are:

       float  fnmatch  glob  iconv  ipc
       monetary  msg  regex  sem  shm
       stdarg  stddef  wchar  wordexp

    Commands Tests

    The sections that test command interfaces have the suffix cmd. Presently only a single command is tested.

    XOPEN.cmd

    This section tests the gencat command.

    Programming Language Tests

    The sections for language specific tests have the prefix lang. Presently there is only one language tested, the "C" language.

    lang.C

    This section contains the C language syntax tests (Volume 4 of XPG3). This does not test for ISO C syntax but Common Usage C (K&R). There are no tests for ISO C, since X/Open recognises NIST validated ISO C test suites.

    Top


Further Information

Top

   
 

 

 

 
   |   Legal Notices & Terms of Use   |   Privacy Statement   |   Top of Page   Return to Top of Page