Revamp the test system #12439

ariya · 2014-08-09T17:57:24Z

The plan is as follows:

There will be a master test runner.
The test runner executes every test script with a fresh instance of PhantomJS.
The exit code of every script defines the success or failure.
If necessary, the output (stdout, stderr) can be compared against a defined baseline.

This was previously discussed in the mailing-list.

#12439

#12439 #10522

#11845 #11558 #11746 #10315 #12439

#12439

#12431 #12439

This Python script serves as a master test runner. It will run each test script with a fresh instance of PhantomJS. The exit code of each script determines its success or failure. The existing tests (run-tests.js) are wrapped as one of the test scripts. This permits a gradual migration to the new test system. #12439

#12439

The server runs on port 9180 and serves the content from test/www directory. Any tests can use this base URL to load test content and to exercise certain PhantomJS API. #12439

#12439

The HTTP server launched by the test runner has a new endpoint called /echo. This simply echoes back some information (current on the made request, in the future it will be expanded to further information) formatted as JSON. This echo system facilitates easy testing for request header manipulation feature. #12439

This reduces the total test duration from 72 seconds to 54 seconds. #12439

#12439

This means that there is no longer a need to add every individual test to the list; only new *directories* have to be mentioned. New tests can be added just by dropping .js files into appropriate directories. Tests which are not meant to be run by default now live in tests/manual/. Currently there is one such test, tests/manual/standards/ecma-test262.js, which downloads and runs the ECMAScript conformance suite. Ongoing work on issue ariya#12439.

We already have 'base_path' as a global variable, so use that (or rather, a new derivative of that) instead of inspect.getfile(inspect.currentframe()). Rather than splitting the path and then iterating over every component trying to figure out if it's "special", take advantage of the fact that posixpath.normpath() will reliably produce a path with all ../ and bare-/ components at the very beginning. Preserve the distinction between paths with and without a trailing /, as SimpleHTTPRequestHandler expects. So that paths containing unusual (e.g. non-ASCII, non-UTF-8) segments can safely be tested, unquote and then requote the entire path. This means that the *bytes* corresponding to the ASCII characters 0123456789 ABCDEFGHIJKLMNOPQRSTUVWXYZ abcdefghijklmnopqrstuvwxyz -_./ are guaranteed to be unquoted, and all other *bytes* are guaranteed to be %-quoted. (Note especially that *whether or not* the tests are being run on Windows, \ in the original URL will be represented as %5C, and will *not* be treated as a pathname separator.) So that the tests behave consistently whether or not the OS has a case- sensitive filesystem, after the above transformation, lowercase the entire path. Ongoing work on issue ariya#12439.

Rather than add all the special URLs we'll ever want to run-tests.py, the embedded HTTP server now supports scripts in test/www/. If you try to load http://localhost:9180/path and test/www/path doesn't exist but test/www/path.py does, then test/www/path.py is loaded as a module. That module must export one function, handle_request(), which is called to produce the response. handle_request() has the same semantics as SimpleHTTPRequestHandler.send_head(). That is, it takes one argument, the SimpleHTTPRequestHandler object, conventionally named 'req'. It should call the send_response(), send_header(), and end_headers() methods of that object as appropriate. And it should return a readable filelike whose contents are taken as the body of the response. If either module import or handle_request() throws any exception, the HTTP response will be a 500 Internal Server Error with body provided by cgitb. Ongoing work on issue ariya#12439.

If there are any positional arguments to test/run-tests.py, they are matched (verbatim, not as regular expressions) against each test's name. Only those tests whose names contain at least one of the positional arguments will be run. Thus, for instance, you can run just the webpage tests with ./test/run-tests.py module/webpage or one specific test with ./test/run-tests.py exact/name/of/test.js or the basics and standards tests with ./test/run-tests.py basics standards If no tests at all match, run-tests.py will print "ALL TESTS SKIPPED" and exit unsuccessfully. Ongoing work on issue ariya#12439.

Callback, clip-rect, confirm, object, prompt, scroll-position, viewport-size and window. ariya#12439

ariya#12439

ariya · 2015-02-01T06:23:27Z

Looks good for now. Further refinements will be filed as a separate ticket.

ariya#12439

If set, this hook is called when the main script (the one named on the phantomjs command line) completes evaluation. This allows a library (e.g. a test harness) to detect when no further code will execute except as a result of events already pending. ariya#12439

ariya#12439

If set, this hook is called when the main script (the one named on the phantomjs command line) completes evaluation. This allows a library (e.g. a test harness) to detect when no further code will execute except as a result of events already pending. ariya#12439

ariya#12439

If set, this hook is called when the main script (the one named on the phantomjs command line) completes evaluation. This allows a library (e.g. a test harness) to detect when no further code will execute except as a result of events already pending. ariya#12439

ariya#12439

If set, this hook is called when the main script (the one named on the phantomjs command line) completes evaluation. This allows a library (e.g. a test harness) to detect when no further code will execute except as a result of events already pending. ariya#12439

ariya#12439

* Eliminate the global variables. * More thorough error detection in the HTTP(S) server. * Test directive comments must start with //! and the parser is more robust. * Handle child process stderr separately from stdout (this will make it possible to test more things). * Execute tests within each directory in alphabetical order. (ariya#12439) * Support and use multi-level wildcards in TESTS. This means we don't have to touch run-tests.py every time we add a subdirectory to module/ or standards/. * Handle HTTP server errors during a test more gracefully. Errors caused by the client disconnecting in the middle of a query-response transaction are ignored. Other errors are reported and the test suite exits cleanly. * Add HTTPS and POST support to test server. * Add ability to run PhantomJS under a debugger on a test case. * Use argparse for command line parsing. * Accept -v as short for --verbose. * Repeating -v means more verbosity; at -vvv, phantomjs command lines are printed. (-vv is reserved for printing individual subtests, which will happen later.) * Fix bug where, with one or more -v, the pjs version number was printed twice.

If set, this hook is called when the main script (the one named on the phantomjs command line) completes evaluation. This allows a library (e.g. a test harness) to detect when no further code will execute except as a result of events already pending. ariya#12439

* Eliminate the global variables. * More thorough error detection in the HTTP(S) server. * Test directive comments must start with //! and the parser is more robust. * Handle child process stderr separately from stdout (this will make it possible to test more things). * Execute tests within each directory in alphabetical order. (ariya#12439) * Support and use multi-level wildcards in TESTS. This means we don't have to touch run-tests.py every time we add a subdirectory to module/ or standards/. * Handle HTTP server errors during a test more gracefully. Errors caused by the client disconnecting in the middle of a query-response transaction are ignored. Other errors are reported and the test suite exits cleanly. * Add HTTPS and POST support to test server. * Add ability to run PhantomJS under a debugger on a test case. * Use argparse for command line parsing. * Accept -v as short for --verbose. * Repeating -v means more verbosity; at -vvv, phantomjs command lines are printed. (-vv is reserved for printing individual subtests, which will happen later.) * Fix bug where, with one or more -v, the pjs version number was printed twice. Part of issue ariya#13478 (test suite overhaul).

If set, this hook is called when the main script (the one named on the phantomjs command line) completes evaluation. This allows a library (e.g. a test harness) to detect when no further code will execute except as a result of events already pending. ariya#12439

* Eliminate the global variables. * More thorough error detection in the HTTP(S) server. * Test directive comments must start with //! and the parser is more robust. * Handle child process stderr separately from stdout (this will make it possible to test more things). * Execute tests within each directory in alphabetical order. (ariya#12439) * Support and use multi-level wildcards in TESTS. This means we don't have to touch run-tests.py every time we add a subdirectory to module/ or standards/. * Handle HTTP server errors during a test more gracefully. Errors caused by the client disconnecting in the middle of a query-response transaction are ignored. Other errors are reported and the test suite exits cleanly. * Add HTTPS and POST support to test server. * Add ability to run PhantomJS under a debugger on a test case. * Use argparse for command line parsing. * Accept -v as short for --verbose. * Repeating -v means more verbosity; at -vvv, phantomjs command lines are printed. (-vv is reserved for printing individual subtests, which will happen later.) * Fix bug where, with one or more -v, the pjs version number was printed twice. Part of issue ariya#13478 (test suite overhaul).

If set, this hook is called when the main script (the one named on the phantomjs command line) completes evaluation. This allows a library (e.g. a test harness) to detect when no further code will execute except as a result of events already pending. ariya#12439

* Eliminate the global variables. * More thorough error detection in the HTTP(S) server. * Test directive comments must start with //! and the parser is more robust. * Handle child process stderr separately from stdout (this will make it possible to test more things). * Execute tests within each directory in alphabetical order. (ariya#12439) * Support and use multi-level wildcards in TESTS. This means we don't have to touch run-tests.py every time we add a subdirectory to module/ or standards/. * Handle HTTP server errors during a test more gracefully. Errors caused by the client disconnecting in the middle of a query-response transaction are ignored. Other errors are reported and the test suite exits cleanly. * Add HTTPS and POST support to test server. * Add ability to run PhantomJS under a debugger on a test case. * Use argparse for command line parsing. * Accept -v as short for --verbose. * Repeating -v means more verbosity; at -vvv, phantomjs command lines are printed. (-vv is reserved for printing individual subtests, which will happen later.) * Fix bug where, with one or more -v, the pjs version number was printed twice. Part of issue ariya#13478 (test suite overhaul).

If set, this hook is called when the main script (the one named on the phantomjs command line) completes evaluation. This allows a library (e.g. a test harness) to detect when no further code will execute except as a result of events already pending. ariya#12439

ariya added this to the Release 2.0 milestone Aug 9, 2014

ariya added Priority-High labels Aug 9, 2014

ariya added a commit that referenced this issue Aug 10, 2014

A simple assertion for the fortcoming revamped test system.

bbdaa9e

#12439

ariya added a commit that referenced this issue Aug 10, 2014

Some tests for JavaScript's Function.

3136898

#12439 #10522

ariya added a commit that referenced this issue Aug 16, 2014

More tests for the arguments object.

f532b73

#11845 #11558 #11746 #10315 #12439

ariya added a commit that referenced this issue Aug 17, 2014

Launcher for ECMA-262 test suite (test262.ecmascript.org).

fbe8eef

#12439

ariya added a commit that referenced this issue Aug 20, 2014

Add a simple test for exit behavior.

b5626bf

#12431 #12439

ariya self-assigned this Aug 24, 2014

ariya added a commit that referenced this issue Aug 28, 2014

Port the tests of 'system' module.

2c0364b

#12439

ariya added a commit that referenced this issue Sep 3, 2014

Port a few basic tests of global objects.

c883a80

#12439

ariya added a commit that referenced this issue Sep 4, 2014

Port the tests for error handling.

729f12c

#12439

ariya added a commit that referenced this issue Sep 4, 2014

Port the tests of JavaScript's date parsing.

e8598a8

#12439

ariya added a commit that referenced this issue Sep 6, 2014

The test runner also launches an HTTP server.

784a06c

The server runs on port 9180 and serves the content from test/www directory. Any tests can use this base URL to load test content and to exercise certain PhantomJS API. #12439

ariya added a commit that referenced this issue Sep 7, 2014

Port loading tests of web page.

2c10f60

#12439

ariya added a commit that referenced this issue Sep 8, 2014

Speed-up the tests: default auto-exit timeout is set to 1 second.

804f61b

This reduces the total test duration from 72 seconds to 54 seconds. #12439

ariya added a commit that referenced this issue Sep 10, 2014

Port the test for resource request error handling.

f29b76b

#12439

ariya added a commit that referenced this issue Sep 11, 2014

Port the test to check for resource received upon error.

f95ef19

#12439

ariya added a commit that referenced this issue Sep 11, 2014

Port the test to abort network request.

d52df7a

#12439

ariya added a commit that referenced this issue Sep 11, 2014

Port the test for onRepaintRequested handler.

dca6f77

#12439

ariya added a commit that referenced this issue Sep 13, 2014

Port the test to change the URL of network request.

3b972fb

#12439

ariya added a commit that referenced this issue Sep 14, 2014

Port the test to change the encoded URL of network request.

f852383

#12439

ariya added a commit that referenced this issue Sep 14, 2014

Port the test to check for plugin non-existence.

83f4baf

#12439

ariya mentioned this issue Sep 15, 2014

Python test runner features #12566

Merged

ariya mentioned this issue Dec 2, 2014

Fix timing issue in WebPage test. #12772

Closed

jakozaur added a commit to jakozaur/phantomjs that referenced this issue Dec 8, 2014

Port the tests from webpage-spec.js

3b8282c

Callback, clip-rect, confirm, object, prompt, scroll-position, viewport-size and window. ariya#12439

jakozaur mentioned this issue Dec 8, 2014

Port the tests from webpage-spec.js #12807

Merged

jakozaur added a commit to jakozaur/phantomjs that referenced this issue Dec 12, 2014

Port more tests from webpage-spec.

32d20b4

ariya#12439

jakozaur mentioned this issue Dec 12, 2014

Port more tests and fix passing date #12822

Merged

ariya closed this as completed Feb 1, 2015

ariya added the Resolved label Feb 1, 2015

kjelloe pushed a commit to kjelloe/phantomjs that referenced this issue Feb 2, 2015

Port more tests from webpage-spec.

8b2a2d5

ariya#12439

zackw added a commit to zackw/phantomjs that referenced this issue Jul 16, 2015

Execute tests within each directory in alphabetical order.

1033b2d

ariya#12439

zackw added a commit to zackw/phantomjs that referenced this issue Jul 16, 2015

Execute tests within each directory in alphabetical order.

3617a47

ariya#12439

zackw added a commit to zackw/phantomjs that referenced this issue Aug 6, 2015

Execute tests within each directory in alphabetical order.

e0ccc08

ariya#12439

zackw added a commit to zackw/phantomjs that referenced this issue Aug 10, 2015

Execute tests within each directory in alphabetical order.

93b03f4

ariya#12439

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Revamp the test system #12439

Revamp the test system #12439

ariya commented Aug 9, 2014

ariya commented Feb 1, 2015

Revamp the test system #12439

Revamp the test system #12439

Comments

ariya commented Aug 9, 2014

ariya commented Feb 1, 2015