tests: actually randomise the order of tests#1613
Conversation
iscgar
force-pushed
the
iscgar/randomise-test-order
branch
from
d7ccf34 to
93ea9eb
Compare
|
This is kind of a bad idea. If a test fails because of running in a particular order, then random failures will not be reproducible. |
Indeed. The tests should use a P(seudo)RNG (not a T(rue)RNG). If the tests are to run with a different seed of the PRNG each time (in principle not a bad idea) then this seed should be displayed (to be logged during test runs) so that if the tests fail with a particular seed the failure can be reproduced. |
|
The Mersenne Twister is still a PRNG, just seeded from what might be a true random (not required by the standard). The original code always had a deterministic order because the RNG wasn't seeded, so the shuffle did nothing. If we're after reproducibility, I can just print the seed and add an option to supply a seed for running the tests to make it reproducible. Would that be acceptable? |
|
@iscgar Yes, printing the seed should make failures reproducible. |
iscgar
force-pushed
the
iscgar/randomise-test-order
branch
from
93ea9eb to
511f6a6
Compare
|
Done. |
| filter = args[1]; | ||
| filterPattern = args[1]; | ||
| } else if(args.size() == 3) { | ||
| seed = std::stoul(args[2]); |
There was a problem hiding this comment.
| seed = std::stoul(args[2]); | |
| filterPattern = args[1]; | |
| seed = std::stoul(args[2]); |
There was a problem hiding this comment.
Good catch, thanks! Fixed.
`std::random_shuffle` can use `std::rand` (and does so on Windows), and without intialising with `srand()` the order will always be the same, so we don't get the expected randomisation. Switch to using `std::shuffle` with a proper random number generator in order to fix this and get random order execution.
iscgar
force-pushed
the
iscgar/randomise-test-order
branch
from
511f6a6 to
fefce40
Compare
|
I'm fine with this. @phkahler ? |
3 participants
std::random_shufflecan usestd::rand(and does so on Windows), and without intialising withsrand()the order will always be the same, so we don't get the expected randomisation.Switch to using
std::shufflewith a proper random number generator in order to fix this and get random order execution.