Skip to content

Testing

Jump to bottom
Chris Lilley edited this page Jul 30, 2018 · 15 revisions

Testing SVG

The SVG 2 test suite will be hosted in Web Platform Tests.

Web Platform Tests

Web Platform Tests is a massive repository of W3C tests. The aim is to bring all tests into one place so they can be used by everyone rather than each implementation having their own test suites.

Web Platform Tests (WPT) is, by design, very focused on web platform implementations (i.e. browsers). Tools, such as image editors, that don't implement the full stack of web platform technologies may run into problems trying to run the tests and require additional bespoke steps.

WPT is hosted on Github: https://github.com/web-platform-tests/wpt

How to get a copy and submit tests

  1. Fork the web-platform-test repository
  2. Clone locally onto your machine to make changes
  3. After making changes, submit a Pull Request to web-platform-tests
  4. To keep your fork in sync with the upstream repository https://help.github.com/articles/syncing-a-fork/
    1. Configure a remote that points to the upstream repository git remote add upstream https://github.com/w3c/web-platform-tests.git
    2. Fetch the upstream commits - they will go into the upstream/master branch git fetch upstream
    3. Merge changes from upstream/master to local master. From local master: git merge upstream/master

Lint tests prior to submission

There is a lint tool in the root directory of web-platform-tests. It is not totally intuitive how the tool works; If you pass a path, the path name is tested, not the files within that directory. Therefore, you have to pass the file name of the single file you want to test. Or, to test all the SVG files you can do:

find svg | xargs ./lint

Reviewing tests

The SVG WG will review tests, but tests may be reviewed by anyone with access who thinks they are capable. This makes it difficult to set strong guidelines on what the test format should be, unless that guideline is in line with the guidelines for the entire repository.

Test suite structure

Tests are stored within web-platform-tests in web-platform-tests/svg/{chapter}/ The chapter name must be the same as the name of the html file for that chapter.

We will not further split the tests by 'reftest', 'scripted' test, or 'manual' tests because web-platform-tests includes a manifest generator that can create lists by test type.

Test Formatting

See http://testthewebforward.org/docs/writing-tests.html

Naming

See http://testthewebforward.org/docs/css-naming.html

The following naming convention should be used as a guide: web-platform-tests/svg/{chapter}/{element}-{attribute}-###.svg

### is a zero filled number starting at 001 and ascending as high as needed to implement all required tests for that feature.

'-manual' added to the end of the name means the test is a manual test '-ref' added to the end of the name means the file is a reference for a reftest

Metadata

Metadata is useful, but forcing too much can reduce motivation to contribute tests.

Past plans for the SVG 2 test suite included the use of Shepherd, which required a lot of metadata. Without Shepherd, this metadata is less useful. A lot of the metadata is redundant because it can be deduced from the git repository. E.g.:

  • copyright notice is now repository wide
  • license is also repository wide
  • author can be deduced from git
  • reviewer can be deduced from git
  • what's being tested can be deduced from file name

We don't want to force metadata, for the following reasons:

  • We want the bar to submitting tests to be as low as possible. If someone has a test from some other test suite, we don't want to make them do a lot of work to submit it to WPT.
  • It may not be SVG WG members who review tests, we don't want to complicate the review process

Having said that, some additional information beyond what's present in the file name may be useful, so we do encourage a short description and a link to the point in the specification that states the assertion being tested is.

This metadata should be supplied using the following elements, which are in the HTML namespace (so declare the namespace in your root SVG element):

  • <html:link rel="help" href="${5:Link to tested section}">
  • <html:meta name="assert" content="${8:Description of what you're trying to test}">

Reftests

See https://web-platform-tests.org/writing-tests/reftests.html

The preferred test format. A reftest consists of a test case and a reference. We prefer SVG test files and SVG references where possible, as this allows non browser implementations (e.g. InkScape) to use the tests. The reference is not an image file, it is an alternative representation of the visual output of the test that can be rendered by the user agent itself, and compared to the test rendering. The reference must use different features to achieve the result (usually features that are simpler and well tested elsewhere).

Templates: https://web-platform-tests.org/appendix/test-templates.html

testharness.js

The test format when the result can be verified using Javascript:

  • Primarily for testing DOM features
  • Time efficient
  • Doesn't allow for visual verification
  • Can't be run with the file: protocol because link must be to /resources/testharness.js

Templates: https://web-platform-tests.org/appendix/test-templates.html

Manual test

  • The recommended way to test the fundamental basic features of SVG (e.g. rendering a rect, rendering a path) that will be used as references for the more complex features
  • For things like animation that need to be manually checked
  • Probably won't be run very often
  • SVG 1.1 tests are currently included as manual tests - we want to mostly convert these to reftests

Running tests

See https://github.com/w3c/web-platform-tests/blob/master/README.md

Both web-platform-tests and csswg-test use the same local test runner. This is launched by running ./serve from the test repo directory.

Other notes

General notes, things we've learnt while investigating testing SVG 2, and questions we might have.

Shepherd

Shepherd is a test suite manager that includes issue tracking, etc. The functionality of Shepherd has been replaced by Github and the SVG WG will not be using Shepherd for managing test issues or reviewing.

How Bikeshed integrates with Shepherd

Bikeshed uses the Shepherd API to discover CSS test suites in WPT. It then injects some script that adds annotations to the spec listing test results at each section heading where tests are present in the test suite. This relies on the rel=help link.

Most CSS specs now use this. The injected script no longer needs to be removed when publishing to /TR.

Clone this wiki locally