Repository navigation
[Tracking Issue] Web Platform Tests revamp/update automation #23192
Copy link
Copy link
Closed
Labels
testIssues and PRs related to Node.js core tests and test infrastructure.Issues and PRs related to Node.js core tests and test infrastructure.
Description
It's been a while since we update the Web Platform Tests in test/parallel in bulk and there have been some recent updates upstream that make the tests easier to run in our test suite, so here is a tracking issue for a revamp of existing tests (hopefully building the foundation of more test integration in the future).
I have a WIP in node-core-utils that currently pulls the tests/harness, track the commit hashes to a file, and makes sure that the license stuff are in place. There is also a WIP branch for this repo with the revamp for URL tests. I was thinking about writing a design doc and get some feedback before opening PRs in both repo, but probably a GitHub issue may work better so here it goes.
- Tooling to pull tests from upstream automatically (right now they are copy-pasted by hand), maintain the commit hashes in a local file and in the git history, and makes sure the license stuff are in place. (In my WIP I put them all in
test/fixtures/wptso the linter will ignore them) - Have a test runner that uses the harness in WPT instead of our mocks in
test/common/wpt.js - Sort out a status file for test expectations (including requirement about intl/crypto, and expected failures, .etc). I have implemented this in my fork with a separate format, but we may also consider changing our python test runner to consume a status file with finer-granularity of control and communicate with Jenkins nicely in the future.
- Migrate the
.any.jstests upstream into a separatetest/wptdirectory with a test runner that creates a sandbox with the WPT harness to run the tests and use the status file to check expectations. - Work with the upstream to migrate more tests out of HTML and put them into JS files that are easier for us to pull down into our repository, as well as split out the parts that cannot be run by us (involving certain APIs that we have not implemented) from the parts that we can run.
------------ TODO ----------
- Fix existing test failures: mostly IDL failures. See status files
- Port encoding tests (need to figure out ICU requirements for the CI): test: port encoding WPT into core #25321
- Port timer tests and mark expected failures
- Port performance tests and mark expected failures
- Port messaging tests and mark expected failures
- Upstream/split existing
test/parallel/test-whatwg-url-custom-*.jsand remove them - Upstream/split existing
test/parallel/test-whatwg-encoding-custom-*.jsand remove them - Support running
.window.js(but we are not a window, so we need to add another scope for us in the upstream and potentially WebIDL) - Enable the driver to be ported out to another project so that it can be reused by our "official" modules: POC in [POC]: Porting WPT utilities in Node.js core whatwg-stream#2
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
testIssues and PRs related to Node.js core tests and test infrastructure.Issues and PRs related to Node.js core tests and test infrastructure.
Activity
joyeecheung commentedon Oct 1, 2018
cc @nodejs/url (not sure what other team I should cc but I guess they are all in this team?)
Also cc @devsnek since you seem to be intersted
test: use git node wpt to pull WPT into test/fixtures
test: remove WPT tests that are now .any.js in the upstream
test: use URL fixtures under test/fixtures/wpt/url/resources
test: initialize test/wpt to run URL and console .js tests
doc: remove legacy WPT integration guide
joyeecheung commentedon Nov 9, 2018
Updated the TODO list now that #24035 landed
test: use git node wpt to pull WPT into test/fixtures
test: remove WPT tests that are now .any.js in the upstream
test: use URL fixtures under test/fixtures/wpt/url/resources
test: initialize test/wpt to run URL and console .js tests
doc: remove legacy WPT integration guide
test: use git node wpt to pull WPT into test/fixtures
test: remove WPT tests that are now .any.js in the upstream
test: use URL fixtures under test/fixtures/wpt/url/resources
test: initialize test/wpt to run URL and console .js tests
doc: remove legacy WPT integration guide
jasnell commentedon Jun 26, 2020
Ping @joyeecheung ... should this remain open?
joyeecheung commentedon Jul 14, 2020
I think this can be closed in favor of a more updated TODO list. Thanks for the ping.