You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Add performance tests for scenarios we care about #20
We need to have a set of tests that contributors can run to verify that their PRs have desired impact on perf, i.e. they would run the tests on master, then on the branch with changes, and compare the results.
This project is not to do comparative benchmarking between different ML frameworks.
Sounds good, then we should probably rename the project/file to PerfRegressionTest.
Benchmarks are usually competitive.
Hopefully, in this case the benchmarks can be normalized against hardware, so that if the run is on a different hw platform it will not fail. Alternatively it should track the perf and the hw platform it runs against.
Hopefully, in this case the benchmarks can be normalized against hardware, so that if the run is on a different hw platform it will not fail. Alternatively it should track the perf and the hw platform it runs against.
The way we use such benchmarks is the following:
Run the benchmark on master branch.
Switch to PR branch
Run the tests again.
Post the diff in the PR description.
This "isolates" the hardware.
Having said that, the reports from BDN record information about the hardware, runtime, etc.
The PR just merged an infrastructure project and a "demo" test. I think it would be good to add more real tests, e.g. flight delay prediction. I would be more than happy to add it, if we think this is the right representative test and we add the data set to the repo.
Activity
KrzysztofCwalina commentedon May 9, 2018
glebuk commentedon May 10, 2018
Sounds good, then we should probably rename the project/file to PerfRegressionTest.
Benchmarks are usually competitive.
Hopefully, in this case the benchmarks can be normalized against hardware, so that if the run is on a different hw platform it will not fail. Alternatively it should track the perf and the hw platform it runs against.
KrzysztofCwalina commentedon May 10, 2018
The way we use such benchmarks is the following:
This "isolates" the hardware.
Having said that, the reports from BDN record information about the hardware, runtime, etc.
shauheen commentedon May 15, 2018
PR #62 merged this.
KrzysztofCwalina commentedon May 15, 2018
The PR just merged an infrastructure project and a "demo" test. I think it would be good to add more real tests, e.g. flight delay prediction. I would be more than happy to add it, if we think this is the right representative test and we add the data set to the repo.
mark get next pipeline test as ignore for now (dotnet#20)
code migration from machinelearning-automl to machinelearning repo fe…
mark get next pipeline test as ignore for now (dotnet#20)