DataFrame.LoadCsv can not load CSV with duplicate column names #6182

Activity

1. torronen

   added

   enhancementNew feature or requestNew feature or request

   on May 3, 2022
2. ghost added

   untriagedNew issue has not been triagedNew issue has not been triaged

   on May 3, 2022
3. torronen commented on May 3, 2022

   

   torronen

   on May 3, 2022

   ContributorAuthor

   More actions

   If anyone has same problem renamed header names can put in the parameter. This solves my issue. I do not know if LoadCsv should have this functionality inbuilt or not (or, issue to be closed or not)

   LoadCsv with renamed columns:

     string line1 = File.ReadLines(TrainDatasetPath).First();
                   string[] arr = line1.Split(';');
                   var duplicatedItems = arr.GroupBy(a => a)
                                          .Where(g => g.Count() > 0)
                                          .ToDictionary(g => g.Key, g => g.Count());
   
                   for (int i = arr.Length - 1; i >= 0; i--)
                   {
                       string item = arr[i];
                       if (!duplicatedItems.ContainsKey(item))
                       {
                           arr[i] = item;
                           continue;
                       }
   
                       if (duplicatedItems[item] > 1)
                       {
                           arr[i] = String.Format("{0}_{1}", item, duplicatedItems[item]);
                       }
                       
                       duplicatedItems[item]--;
                   }
   
                   IDataView trainData = DataFrame.LoadCsv(TrainDatasetPath, separator: ';', header: true, columnNames: arr, guessRows: 100);
   

   👍React with 👍2Reacted by Luis Quintanilla and Mark
4. michaelgsharp commented on May 9, 2022

   

   michaelgsharp

   on May 9, 2022

   Contributor

   More actions

   @luisquintanilla @torronen is this something we think should be built in to load? I can see it going both ways honestly. I would probably lean towards having it built in somehow.
5. luisquintanilla commented on May 9, 2022

   

   luisquintanilla

   on May 9, 2022

   Contributor

   More actions

   @michaelgsharp by builtin do you mean duplicate columns are automatically renamed like the snippet @torronen shared?
6. beccamc commented on May 9, 2022

   

   beccamc

   on May 9, 2022

   Contributor

   More actions

   +1 - I also experienced pain around large data files with multiple columns sharing names. I only had less than 50 columns and it was still troublesome to deal with.

   👍React with 👍1Reacted by Luis Quintanilla
7. michaelgsharp

   added

   P2

   Microsoft.Data.Analysis

   on Jun 13, 2022
8. michaelgsharp

   added this to the ML.NET Future milestone on Jun 13, 2022
9. ghost removed

   untriagedNew issue has not been triagedNew issue has not been triaged

   on Jun 13, 2022
10. ghost added

    in-prA pull request is open to resolve this issueA pull request is open to resolve this issue

    on Jul 24, 2023
11. michaelgsharp

    closed this as completedin #6772on Aug 31, 2023
12. ghost removed

    in-prA pull request is open to resolve this issueA pull request is open to resolve this issue

    on Aug 31, 2023
13. ghost locked as resolved and limited conversation to collaborators on Sep 30, 2023