Skip to content

Control uses of rand() in test scripts #301

Open

Description

@plbossart

The test scripts use random number generator to e.g. change the sleep times.

This is wonderful for stress tests, but a really bad idea for CI and daily tests, since by definition these tests cannot be reproduced UNLESS the seed is known.

We should add one option to tests that use random numbers, so that by default we shall use the same seed - 42
https://en.wikipedia.org/wiki/Phrases_from_The_Hitchhiker%27s_Guide_to_the_Galaxy#Answer_to_the_Ultimate_Question_of_Life,_the_Universe,_and_Everything_(42)

When the user wishes more randomness, the seed shall be saved in the logs so that we can reproduce failures.

This can be done in bash with the RANDOM variable

Normal case:

$ T=42
$ RANDOM=$T; echo $RANDOM; echo $RANDOM; echo $RANDOM
17766
11151
23481
$ RANDOM=$T; echo $RANDOM; echo $RANDOM; echo $RANDOM
17766
11151
23481

stress-test:

$ T=$(date +%s)
$ echo $T
1596129567
$ RANDOM=$T; echo $RANDOM; echo $RANDOM;
19204
11943
$ RANDOM=$T; echo $RANDOM; echo $RANDOM;
19204
11943
$ RANDOM=$T; echo $RANDOM; echo $RANDOM;
19204
11943

cc:

Activity

  1. added
    P2Critical bugs or normal features
    on Jul 31, 2020
  2. marc-hb commented on Jul 31, 2020

    @marc-hb
    Collaborator

    As suggested by @xiulipan : the best option is a command flag that lets the caller choose between constant seed parameter (with a default value) versus truly random seed.

    My suggestions:

    • The test should always print the seed used in any case so it can be re-used.
    • clearly label CI results using a random seed, e.g. "stress test" not to surprise users.
  3. xiulipan commented on Jul 31, 2020

    @xiulipan
    Contributor

    thanks @marc-hb for the suggestion. I will take them with the new feature implement.
    Another feature:

    • use a parameter to accept seed, used for reproduce issue from stress test
  4. self-assigned this
    on Jul 31, 2020
  5. marc-hb commented on Jul 31, 2020

    @marc-hb
    Collaborator

    use a parameter to accept seed, used for reproduce issue from stress test

    Yes that's what I meant by "... so it can be re-used". Rephrasing the previous comment.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

P2Critical bugs or normal featurestype:enhancementNew framework feature or requesttype:test coverage gapThis requires a new test case, not just fixing one

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions