output is nondeterministic #297

during #295 i noticed that there are crates for which rust-semverver output is not deterministic.

• i'm not sure how similar projects handle this, or if there's something that e.g. compiletest-rs do to make the output more deterministic
• i guess there might be things we could do on our side to make the output more deterministic, like iterate in sorted order over things etc.
• there are cases where the number of errors differs, not just the order. i'm not sure if this indicates a general problem with using compile errors/warnings for breaking/non-breaking changes; i.e. i'd assume that how many errors are shown once an initial error has been hit (and it's clear the "compile" won't be successful) is more of a best-effort thing to reduce the number of iterations, and the effort of making things deterministic goes more towards the compiled code than the error messages.

cases i've collected so far

• PATH=${PWD}/target/debug:$PATH cargo run --bin cargo-semver -- -S casbin:2.0.0 -C casbin:2.0.1
  • casbin-2.0.0-2.0.1.linux.1.txt
  • casbin-2.0.0-2.0.1.linux.2.txt
  • casbin-2.0.0-2.0.1.linux.3.txt
• PATH=${PWD}/target/debug:$PATH cargo run --bin cargo-semver -- -S kitchen-fridge:0.3.0 -C kitchen-fridge:0.4.0
  • kitchen-fridge-0.3.0-0.4.0.linux.1.txt
  • kitchen-fridge-0.3.0-0.4.0.linux.2.txt
  • kitchen-fridge-0.3.0-0.4.0.linux.3.txt

these are all from https://crates.io/crates/async-trait/reverse_dependencies, and i guess the common denominator is that they're big enough™