Skip to content

doc: How to run just the .out tests? #40920

Closed
#41352
Closed
doc: How to run just the .out tests?#40920
#41352
Labels
docIssues and PRs related to Node.js documentation.questionIssues asking questions about Node.js.testIssues and PRs related to Node.js core tests and test infrastructure.

Description

@GeoffreyBooth
GeoffreyBooth
opened on Nov 22, 2021

I’ve done some digging, and I’m sure if I kept looking I’d eventually find it, but could someone please tell me how to run just the tests that check expected output against the .out files? Those tests are annoying to update, as it’s a lot of trial-and-error to get the new output in, and it’s tedious to have to run the full make test for every attempt.

This feels like something that should be documented in https://github.com/nodejs/node/blob/master/BUILDING.md#running-tests.

Activity

  1. GeoffreyBooth
    added
    docIssues and PRs related to Node.js documentation.
    testIssues and PRs related to Node.js core tests and test infrastructure.
    on Nov 22, 2021

  2. Trott commented on Nov 22, 2021

    @Trott
    Trott
    on Nov 22, 2021
    Member

    These would be tests in test/messages and test/pseudo-tty, right? If so, you can run them with tools/test.py messages pseudo-tty. If you know which one specifically you want to run, you can specify it: tools/test.py test/message/test-esm-loader-obsolete-hooks.mjs.

    Does that help?

  3. GeoffreyBooth commented on Nov 22, 2021

    @GeoffreyBooth
    GeoffreyBooth
    on Nov 22, 2021
    MemberAuthor

    Yes! Thank you, that’s what I was looking for.

    I was frustrated by the fact that when these tests fail, the output tells me the command the test runner ran to generate the output, not to compare the output. In other words, if I edit test/message/test-esm-loader-obsolete-hooks.out so that it causes the test to fail, the output of this failing test includes this:

    Command: out/Release/node --no-warnings --throw-deprecation --experimental-loader ./test/fixtures/es-module-loaders/hooks-obsolete.mjs /Users/Geoffrey/Sites/node/test/message/test-esm-loader-obsolete-hooks.mjs

    Which for other types of tests is a one-liner to re-run that particular test; but for these “compare to .out file” tests, it’s not.

    Anyway because these tests are somewhat unusual, I think a callout explaining them in the docs might help others; it would at least have helped me. It’s something I’ve been wondering about for years.

  4. Mesteery
    added
    questionIssues asking questions about Node.js.
    on Nov 22, 2021

  5. Trott commented on Nov 22, 2021

    @Trott
    Trott
    on Nov 22, 2021
    Member

    Anyway because these tests are somewhat unusual, I think a callout explaining them in the docs might help others; it would at least have helped me. It’s something I’ve been wondering about for years.

    Since this has been a pain point for you and that doc is where you looked for information and didn't find it, I am +1 on adding the information there. Do you want to open a PR for that or would you prefer I do it?

  6. GeoffreyBooth commented on Nov 22, 2021

    @GeoffreyBooth
    GeoffreyBooth
    on Nov 22, 2021
    MemberAuthor

    Do you want to open a PR for that or would you prefer I do it?

    I don't understand these tests well, so I think it would be better if you wrote this.

    Could we add something to the output where if the command that was run is from one of these tests, another line follows the Command like with something like As part of: tools/test.py test/message/test-esm-loader-obsolete-hooks.mjs?

  7. Trott commented on Nov 23, 2021

    @Trott
    Trott
    on Nov 23, 2021
    Member

    I don't understand these tests well, so I think it would be better if you wrote this.

    Included in #40933.

    Could we add something to the output where if the command that was run is from one of these tests, another line follows the Command like with something like As part of: tools/test.py test/message/test-esm-loader-obsolete-hooks.mjs?

    I think the way to do this would be:

    Move the compare-output-to-.out-file stuff out of test/message/testcfg.py and test/pseudo-ttyp/testcfg.py, and into a module in test/common that all these tests load. Perhaps the module would add a listener for the 'exit' event on process and check the output there. (It would probably load the expected output at an earlier time, perhaps when the module is first required/imported.)

    Then running the test directly with node rather than test.py would still do the comparison, so the existing displayed command would work the way someone might expect.

    @nodejs/testing What do you think?

  8. GeoffreyBooth commented on Nov 23, 2021

    @GeoffreyBooth
    GeoffreyBooth
    on Nov 23, 2021
    MemberAuthor

    It seems to me that maybe all these .out-file tests for error messages, at least for the ESM subsystem that I work on, could have been written like https://github.com/nodejs/node/blob/master/test/es-module/test-esm-invalid-extension.js where the test itself is responsible for spawning the child process and validating the error message. As in, perhaps it was a mistake to use the .out file pattern for these ESM tests, and we should refactor them to be like test-esm-invalid-extension.js. If we did that, running python tools/test.py -J --mode=release es-module really would run all the ESM tests. Any thoughts on this @Trott or others? cc @JakobJingleheimer

  9. Trott commented on Nov 24, 2021

    @Trott
    Trott
    on Nov 24, 2021
    Member

    It seems to me that maybe all these .out-file tests for error messages, at least for the ESM subsystem that I work on, could have been written like https://github.com/nodejs/node/blob/master/test/es-module/test-esm-invalid-extension.js where the test itself is responsible for spawning the child process and validating the error message. As in, perhaps it was a mistake to use the .out file pattern for these ESM tests, and we should refactor them to be like test-esm-invalid-extension.js. If we did that, running python tools/test.py -J --mode=release es-module really would run all the ESM tests. Any thoughts on this @Trott or others? cc @JakobJingleheimer

    I have no opinion because I'm not familiar with the friction involved. No objection from me, certainly! If it simplifies things for you and others, then that's reason enough for me.

  10. nodejs-github-bot
    added a commit that references this issue on Nov 25, 2021

    doc: update BUILDING.md section on running tests

    a5d3449
  11. targos
    added a commit that references this issue on Nov 26, 2021

    doc: update BUILDING.md section on running tests

    130777a

  12. JakobJingleheimer commented on Dec 4, 2021

    @JakobJingleheimer
    JakobJingleheimer
    on Dec 4, 2021
    Member

    When I first started, these .out file tests were extremely confusing. As an implementor, these are very tedious as they break whenever the wind blows despite no substantive changes.

    I think moving to an approach like in the test Geoffrey cited would be a huge improvement 🙏

  13. nodejs-github-bot
    closed this as completedin #41352on Jan 6, 2022

  14. GeoffreyBooth commented on Jan 6, 2022

    @GeoffreyBooth
    GeoffreyBooth
    on Jan 6, 2022
    MemberAuthor

    Fixed by #40933.

  15. danielleadams
    added 2 commits that reference this issue on Jan 30, 2022

    doc: update BUILDING.md section on running tests

    308fe7d

    doc: update BUILDING.md section on running tests

    02e9c2e
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    docIssues and PRs related to Node.js documentation.questionIssues asking questions about Node.js.testIssues and PRs related to Node.js core tests and test infrastructure.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions