Kyua 0.13: Stability fixes

Released on August 26th, 2016.

• Fixed execution of test cases as an unprivileged user, at least under
  NetBSD 7.0. Kyua-level failures were probably a regression introduced
  in Kyua 0.12, but the underlying may have existed for much longer:
  test cases might have previously failed for mysterious reasons when
  running under an unprivileged user.
• Issue #134: Fixed metadata test broken on 32-bit platforms.
• Issue #139: Added per-test case start/end timestamps to all reports.
• Issue #156: Fixed crashes due to the invalid handling of cleanup
  routine data and triggered by the reuse of PIDs in long-running Kyua
  instances.
• Issue #159: Fixed TAP parser to ignore case while matching TODO and
  SKIP directives, and to also recognize Skipped.
• Fixed potential crash due to a race condition in the unprogramming of
  timers to control test deadlines.

Kyua 0.12: Parallel execution of tests

Released on November 22nd, 2015.

This is a huge release and marks a major milestone for Kyua as it finally
implements a long-standing feature request: the ability to execute test
cases in parallel. This is a big deal because test cases are rarely
CPU-bound: running them in parallel yields much faster execution times for
large test suites, allowing faster iteration of changes during development.

As an example: the FreeBSD test suite as of this date contains 3285 test
cases. With sequential execution, a full test suite run takes around 12
minutes to complete, whereas on a 4-core machine with a high level of
parallelism it takes a little over 1 minute.

Implementing parallel execution required rewriting most of Kyua's core and
partly explains explains why there has not been a new release for over a
year. The current implementation is purely subprocess-based, which works
but has some limitations and has resulted in a core that is really complex
and difficult to understand. Future versions will investigate the use of
threads instead for a simplified programming model and additional
parallelization possibilities.

• Issue #2: Implemented support to execute test cases in parallel when
  invoking kyua test. Parallel execution is only enabled when the new
  parallelism configuration variable is set to a value greater than 1.
  The default behavior is still to run tests sequentially because some test
  suites contain test cases with side-effects that might fail when run in
  parallel. To resolve this, the new metadata property is_exclusive can
  be set to true on a test basis to indicate that the test must be run on
  its own.
• Known regression: Running kyua debug on a TAP-based test program does
  not currently report the output in real time. The output will only be
  displayed once the test program completes. This is a shortcoming of
  the new parallel execution engine and will be resolved.
• Removed the external C-based testers code in favor of the new built-in
  implementations. The new approach feels significantly faster than the
  previous one.
• Fixed the handling of relative paths in the fs.* functions available
  in Kyuafiles. All paths are now resolved relative to the location of
  the caller Kyuafile. Kyuafile.top has been updated with these
  changes and you should update custom copies of this file with the new
  version.
• Changed temporary directory creation to always grant search
  permissions on temporary directories. This is to prevent potential
  problems when running Kyua as root and executing test cases that require
  dropping privileges (as they may later be unable to use absolute paths
  that point inside their work directory).
• The cleanup of work directories does not longer attempt to deal with
  mount points. If a test case mounts a file system and forgets to unmount
  it, the mount point will be left behind. It is now the responsibility of
  the test case to clean after itself. The reasons for this change are
  simplicity and clarity: there are many more things that a test case can
  do that have side-effects on the system and Kyua cannot protect against
  them all, so it is better to just have the test undo anything it might
  have done.
• Improved kyua report --verbose to properly handle environment
  variables with continuation lines in them, and fixed the integration
  tests for this command to avoid false negatives.
• Changed the configuration file format to accept the definition of
  unknown variables without declaring them local. The syntax version
  number remains at 2. This is to allow configuration files for newer Kyua
  versions to work on older Kyua versions, as there is no reason to forbid
  this.
• Fixed stacktrace gathering with FreeBSD's ancient version of GDB.
  GDB 6.1.1 (circa 2004) does not have the -ex flag so we need to
  generate a temporary GDB script and feed it to GDB with -x instead.
• Issue #136: Fixed the XML escaping in the JUnit output so that
  non-printable characters are properly handled when they appear in the
  process's stdout or stderr.
• Issue #141: Improved reporting of errors triggered by sqlite3. In
  particular, all error messages are now tagged with their corresponding
  database filename and, if they are API-level errors, the name of the
  sqlite3 function that caused them.
• Issue #144: Improved documentation on the support for custom properties
  in the test metadata.
• Converted the INSTALL, NEWS, and README distribution documents to
  Markdown for better formatting online.

Kyua 0.11: Major cleanup release (bug fixes, doc improvements)

Changes in version 0.11

Released on October 23rd, 2014.

• Added support to print the details of all test cases (metadata and
  their output) to 'report'. This is via a new '--verbose' flag which
  replaces the previous '--show-context'.
• Added support to specify the amount of physical disk space required
  by a test case. This is in the form of a new "required_disk_space"
  metadata property, which can also be provided by ATF test cases as
  "require.diskspace".
• Assimilated the contents of all the kyua--tester(1) and
  kyua--interface(7) manual pages into more relevant places. In
  particular, added more details on test program registration and their
  metadata to kyuafile(5), and added kyua-test-isolation(7) describing
  the isolation features of the test execution.
• Assimilated the contents of all auxiliary manual pages, including
  kyua-build-root(7), kyua-results-files(7), kyua-test-filters(7) and
  kyua-test-isolation(7), into the relevant command-specific manual
  pages. This is for easier discoverability of relevant information
  when reading how specific Kyua commands work.
• Issue 30: Plumbed through support to query configuration variables
  from ATF's test case heads. This resolves the confusing situation
  where test cases could only do this from their body and cleanup
  routines.
• Issue 49: Extended 'report' to support test case filters as
  command-line arguments. Combined with '--verbose', this allows
  inspecting the details of a test case failure after execution.
• Issue 55: Deprecated support for specifying test_suite overrides on
  a test program basis. This idiom should not be used but support for
  it remains in place.
• Issue 72: Added caching support to the getcwd(3) test in configure so
  that the result can be overriden for cross-compilation purposes.
• Issue 83: Changed manual page headings to include a "kyua" prefix in
  their name. This prevents some possible confusion when displaying,
  for example, the "kyua-test" manual page with a plain name of "test".
• Issue 84: Started passing test-suite configuration variables to plain
  and TAP test programs via the environment. The name of the
  environment variables set this way is prefixed by TEST_ENV_, so a
  configuration variable of the form
  'test_suites.some_name.allow_unsafe_ops=yes' in kyua.conf becomes
  'TEST_ENV_allow_unsafe_ops=YES' in the environment.
• Issue 97 and 116: Fixed the build on Illumos.
• Issue 102: Set TMPDIR to the test case's work directory when running
  the test case. If the test case happens to use the mktemp(3) family
  of functions (due to misunderstandings on how Kyua works or due to
  the reuse of legacy test code), we don't want it to easily escape the
  automanaged work directory.
• Issue 103: Started being more liberal in the parsing of TAP test
  results by treating the number in 'ok' and 'not ok' lines as optional.
• Issue 105: Started using tmpfs instead of md as a temporary file
  system for tests in FreeBSD so that we do not leak md(4) devices.
• Issue 109: Changed the privilege dropping code to start properly
  dropping group privileges when unprivileged_user is set. Also fixes
  testers/run_test:fork_wait__unprivileged_group.
• Issue 110: Changed 'help' to display version information and clarified
  the purpose of the 'about' command in its documentation.
• Issue 111: Fixed crash when defining a test program in a Kyuafile that
  has not yet specified the test suite name.
• Issue 114: Improved the kyuafile(5) manual page by clarifying the
  restrictions of the include() directive and by adding abundant
  examples.

Kyua 0.10: Single unified package to rule them all

Changes in version 0.10

Experimental version released on August 14th, 2014.

• Merged 'kyua-cli' and 'kyua-testers' into a single 'kyua' package.
• Dropped the 'kyua-atf-compat' package.
• Issue 100: Do not try to drop privileges to unprivileged_user when we
  are already running as an unprivileged user. Doing so is not possible
  and thus causes spurious test failures when the current user is not
  root and the current user and unprivileged_user do not match.
• Issue 79: Mention kyua.conf(5) in the "see also" section of kyua(1).
• Issue 75: Change the rewrite__expected_signal__bad_arg test in
  testers/atf_result_test to use a different signal value. This is to
  prevent triggering a core dump that made the test fail in some platforms.

Kyua Testers 0.3: Improved TAP handling

Changes in version 0.3

Experimental version released on August 8th, 2014.

• Made the testers set a "sanitized" value for the HOME environment
  variable where, for example, consecutive and trailing slashes have
  been cleared. Mac OS X has a tendency to append a trailing slash to
  the value of TMPDIR, which can cause third-party tests to fail if they
  compare ${HOME} with $(pwd).
• Issues 85, 86, 90 and 92: Made the TAP parser more complete: mark test
  cases reported as TODO or SKIP as passed; handle skip plans; ignore
  lines that look like "ok" and "not ok" but aren't results; and handle
  test programs that report a pass but exit with a non-zero code.

Kyua CLI 0.9: Test actions no more

Changes in version 0.9

Experimental version released on August 8th, 2014.

Major changes:

The internal architecture of Kyua to record the results of test suite
runs has completely changed in this release. Kyua no longer stores all
the different test suite run results as different "actions" within the
single store.db database. Instead, Kyua now generates a separate
results file inside ~/.kyua/store/ for every test suite run.

Due to the complexity involved in the migration process and the little
need for it, this is probably going to be the only release where the
'db-migrate' command is able to convert an old store.db file to the
new scheme.

Changes in more detail:

• Added the 'report-junit' command to generate JUnit XML result files.
  The output has been verified to work within Jenkins.
• Switched to results files specific to their corresponding test suite
  run. The unified store.db file is now gone: 'kyua test' creates a
  new results file for every invocation under ~/.kyua/store/ and the
  'kyua report*' commands are able to locate the latest file for a
  corresponding test suite automatically.
• The 'db-migrate' command takes an old store.db file and generates one
  results file for every previously-recorded action, later deleting the
  store.db file.
• The '--action' flag has been removed from all commands that accepted
  it. This has been superseded by the tests results files.
• The '--store' flag that many commands took has been renamed to
  '--results-file' in line with the semantical changes.
• The 'db-exec' command no longer creates an empty database when none
  is found. This command is now intended to run only over existing
  files.

Kyua ATF Compat 0.2: Miscellaneous fixes

Changes in version 0.2

Experimental version released on December 28th, 2013.

• Fixed two problems regarding configuration parsing in atf-run: one
  to properly handle tab characters and one that caused an error with
  NetBSD's /bin/sh.
• Fixed the HTML output of atf-report to include all tests after
  kyua-cli-0.7 stopped listing passed tests by default.

Kyua Testers 0.2: New TAP tester

Changes in version 0.2

Experimental version released on December 7th, 2013.

• Issue 74: Added the kyua-tap-tester, a new backend to interact with
  test programs that comply with the Test Anything Protocol.
• Issue 69: Cope with the lack of AM_PROG_AR in configure.ac, which
  first appeared in Automake 1.11.2. Fixes a problem in Ubuntu 10.04
  LTS, which appears stuck in 1.11.1.
• Issue 24: Improve test case isolation by confining the tests to their
  own session instead of just to their own process group.

Kyua CLI 0.8: Dynamic support for testers

Changes in version 0.8

Experimental version released on December 7th, 2013.

• Added support for Lutok 0.4.
• Issue 24: Plug the bootstrap tests back into the test suite. Fixes
  in kyua-testers-0.2 to isolate test cases into their own sessions
  should allow these to run fine.
• Issue 74: Changed the kyuafile(5) parser to automatically discover
  existing tester interfaces. The various *_test_program() functions
  will now exist (or not) based on tester availability, which simplifies
  the addition of new testers or the selective installation of them.

Kyua CLI 0.7: Support for Lua 5.2 and miscellaneous fixes

Changes in version 0.7

Experimental version released on October 18th, 2013.

• Made failures from testers more resilent. If a tester fails, the
  corresponding test case will be marked as broken instead of causing
  kyua to exit.
• Added the '--results-filter' option to the 'report-html' command and
  set its default value to skip passed results from HTML reports. This
  is to keep these reports more succint and to avoid generating tons of
  detail files that will be, in general, useless.
• Switched to use Lutok 0.3 to gain compatibility with Lua 5.2.
• Issue 69: Cope with the lack of AM_PROG_AR in configure.ac, which
  first appeared in Automake 1.11.2. Fixes a problem in Ubuntu 10.04
  LTS, which appears stuck in 1.11.1.