Repository navigation
Plans for lazy Loading? #199
Description
Hello ML.NET experts :D,
there is just one other point which was in my mind. In a time of cloud and big data there is always one important question : how to access data with a good performance and low memory?
I had a lot of trouble in past in simulations because the tools of our customers loaded (mostly) all measurement data at once. It took about 5-10 Minutes for loading and 20-30 GB of memory. Just let you know - this was in car simulation area and to be honest - this was low data - todays time such cars have TB of data in just 1 hour driving. So we reinvented their tools and included lazy loading.
We archived this lazy loading by included some tricks into import level. Every data source got one general class called "dataSrc" and was injected with an Dataprovider. When user called the property "DataSets" (property in C# meaning) the Getter invoke provider and the provider looked for all dataSet IDs (column names or general spoken - the ID of an measurement array) in the source. The DataSets property returned an Array of dataset objects. Each Set had an Array property (which was still not invoken) and a dictionary with Attributes (key - value). So we could search for meta informations (by the Dictionary) without loading the array (our true measurement data). Just when the User really was sure - okay I want the data from DataSet XYZ - then the Provider was invoken to look for my data.
With this strategy we was able to reduce the time and memory a lot. The users could search for attributes and take datasets they need - moreover - It was even possible to use Linq in the same way for every possible datasource.
One thing more to consider :
- If the arrays get too big maybe we could think about loading parts of the total measurement array (like Span? .
Activity
glebuk commentedon May 22, 2018
@dotChris90,
One of design goals of ML.NET is to be able to stream data without loading it all in memory.
It is capable of processing data in a streaming fashion.
In order for streaming to happen, couple things need to occur:
Did you have a chance to try training on a large dataset to see if streaming is achieved?
dotChris90 commentedon May 22, 2018
A very interisting design (want to say : brillant). Sure : streams are the typical solution for this Kind of Problems. With Streaming you can archiev both : load single Array of source and load parts of Array.
At Moment it is hard to get from Simulation Department since this year I am in a new Project (Hardware C/C++ Area). But since I am still a .NET fan watch this Project very closely ;)
Moreover I have to say : my question is already satisfied. Just remember the pain we had last year and wanted to share my experience to you.
just one last question maybe connected to this issue because curiousity : in the examples I usual see Trainings data columns in source are equal to the .... Memory Training class ... I don not know how to call it (in example https://www.microsoft.com/net/learn/apps/machine-learning-and-ai/ml-dotnet/get-started/windows it is the class IrisData and IrisData fits totally to the columns of our text file). The generic Parameter of Textloader - am I right if I think that this Parameter effect the columns which are loaded so that you could use one Training source for multiple different models?
dotChris90 commentedon May 23, 2018
@glebuk just saw your strategy : the IData Loader and Saver intface. If I am right (please correct me if i am wrong) your data representation is column focused so you can load just data of a column and you have also meta data. So the column Focus + Streamer idea : my lazy loading wish is totally satisfied. If you do not mind - I Close my ticket by myself. >.< maybe I should read your Github repo and understand the architecture before rise tickets ...
Include some missed logging in the generated code. (dotnet#199)
code migration from machinelearning-automl to machinelearning repo fe…
Include some missed logging in the generated code. (dotnet#199)
Added onnx export functionality for LpNormNormalizingTransformer (#4161)
Added Onnx export functionality to PCATransformer (#4188)