Repository navigation
Enabling different settings for the TextLoader on train and test data might lead to incorrect metrics with no error #5
Description
The train and test data can be read with different TextLoaders, which makes it easy to introduce bugs. If there is a difference in the settings, the metrics are likely to be wrong but there are no errors.
Example:
string trainDataPath= "sentiment_data.tsv";
pipeline.Add(new TextLoader<SentimentData>(trainDataPath, useHeader: true));
// Later in the file, after completing the pipeline and training the model
string testDataPath = "sentiment_test.tsv";
var testData = new TextLoader<SentimentData>(testDataPath, userHeader: true, sep: ',');
var evaluator = new BinaryClassificationEvaluator();
BinaryClassificationMetrics metrics = evaluator.Evaluate(model, testData);Evaluating on the test data will result in incorrect metrics as the test file will be parsed incorrectly. However, the experiment runs successfully with no errors, so this might be difficult to detect.
Metadata
Metadata
Assignees
Labels
No labels
Activity
zeahmed commentedon May 23, 2018
Blocked by #216 and #119
Merge pull request dotnet#5 from dotnet/master
abgoswam commentedon Oct 15, 2018
DRI RESPONSE: this is about legacy API, which we no longer support, no reason to continue to look into this one
make params for MLContext data extensions match ML.NET default names …
code migration from machinelearning-automl to machinelearning repo fe…
make params for MLContext data extensions match ML.NET default names …