Repository navigation
Reinforcement learning #181
Description
I've looked into the available documentation and examples, but haven't been able to figure out if it is possible to use the ML.NET in its current state for (non-deep) reinforcement learning. If it is possible, I'd be thankful for any hints on how to implement a simple case. In case reinforcement learning is not possible atm, what exactly is missing and are there any plans on implementing the missing pieces?
Thanks!
Reacted by Jon Wood, zedLove, Steven-Mark-Ford, InDieTasten, Herman Eldering, chris3713, Faraj Farook, Henk7, sunn, xiety and 14 more
Reacted by Alxy Savin, Rustam Salakhutdinov and Caio Henrique Figurski
Activity
GalOshri commentedon May 18, 2018
Reinforcement learning is not yet available in ML.NET. One critical component that is currently missing is "exploration" (trying out different actions to see how the reward changes) and how this affects data collection and model training. This is something that would be interesting to explore in the future.
What is the scenario you have in mind?
jarnmo commentedon May 18, 2018
I see. My scenario is basically to use it as a part of building a game AI.
GalOshri commentedon May 18, 2018
Got it. A few more questions: what is the complexity of the game? How many actions does the AI have (the number of choices available to it at each step)? Does the list of actions change throughout the course of the game or is it static?
jarnmo commentedon May 19, 2018
Well, what I'm actually trying to do, is to build a framework which would be used to write game rules (gameplay code) and would make it as easy as possible to add AI actors as well. In the end anything but a trivial AI would probably have to be build of several components, some ML and some utilizing more traditional features of the framework. The end goal is to allow building games of any practical complexity, but no 'academic restrictions' apply here. RL can only be a piece in the AI puzzle and for example 'goal hacking' is certainly an option if it makes the AI behave in a desirable way :).
While a game AI can be extremely complex, I'm hoping to find ways to split the problem domains into smaller pieces that could then be handled by individual 'RL pipelines'. Maybe even just learn to estimate the value of taking each action in a given situation individually. I think the Microsoft's Mrs PacMan AI used this kind of an approach.
It's certainly common that the list of possible actions an actor can take changes throughout the course of a game, but if the value of each action (for each target) would be evaluated by it's own NN I guess the changes would be a matter of updating the list of evaluated NNs (and whatever logic coordinates between them).
This is all very much just an idea atm, but I guess even a very simple RL implementation would allow meaningful experimentation. I might look into CNTK and their C# bindings now, but as I have no practical experience from ML I wouldn't mind something a bit easier to approach..
axmand commentedon Jun 19, 2018
To Implement RL such as DQN, I need to train neural network after an observation.
however, i don't know how to split train data to train the neural network by 'TextLoader'.
Is there any way to gradually input training data?
//
well, i found this in:
machinelearning/test/Microsoft.ML.Tests/CollectionDataSourceTests.cs
Line 29 in f6c6f5b
API rev (dotnet#181)
code migration from machinelearning-automl to machinelearning repo fe…
codemzs commentedon Jun 2, 2019
CC: @wschin
chris3713 commentedon Jun 16, 2019
Would be nice too see this implemented some time.
codemzs commentedon Jun 30, 2019
We don't have RL support in the short-term but it is something we are considering for long-term.
32 remaining items
Bonifatius94 commentedon Mar 14, 2023
@luisquintanilla @jez9999 RL isn't only for game AI development, but also for any kind of automation that involves solving a Markov Decision Process. For example, I'm currently researching on training a self-driving e-scooter with the PPO algorithm regarding autonomous driving. But there are various other applications like continuous power plant / factory control. Also in robotics, there's a huge research field on autonomous agents for motion control and decision making. Or let's say you wanna develop any kind of personal assistant that helps to be more productive at work. There will be plenty of demand for RL training.
IMO, behavioral AI systems for decision making are the next step, especially in combination with ChatGPT-empowered expert systems. Every big company will start to enhance their workers' boring repetitive tasks with AI systems within the next 5-10 years. So, there should be a huge market for a well-performing training framework that allows to train a lot faster than this very slow Python crap due to the support for hardware intrinsics, etc. Just think about it for a moment. You've got a great opportunity.
decriptor commentedon Jun 6, 2023
@luisquintanilla does that mean that most of #5918 should be completed in ML.NET 3.0?
Ceeeeed commentedon Aug 29, 2023
I'm also interested in RL. This needs to be in ML.NET.
GeorgeS2019 commentedon Sep 28, 2023
@michaelgsharp
Sept 2023 Update for .NET Deep RL
Do add the deep learning label to this issue.
We are in 2023, this project has ONLY 3 issues labeled with deep learning :-)
sirhamsteralot commentedon Feb 23, 2024
Chipping in to say that I too am currently in the progress of planning out an AI for a game and would like to utilise ML.Net for Reinforcement learning. The inclusion would be mostly on the decision making front of the game AI for me however robotics/self driving rc cars and the like would also be applicable usecases.
Are there any updates? I see that it still has the label future, considering this seems like one of the more active topics on the roadmap I hope this can be picked up relatively soon
Flem100 commentedon Sep 22, 2024
I have been wanting to use ML.Net with Unity for a long time. I want to use it instead of https://github.com/Unity-Technologies/ml-agents, which I don't like to use.
I got ML.net 3.0 to work with Unity and then I learn that ML.net does not have reinforcement learning.
My current goal is to do the same as in this video:
https://www.youtube.com/watch?v=L_4BPjLBF4E
using ML.net in Unity.
GeorgeS2019 commentedon Sep 24, 2024
First, this is OUTSIDE the scope of Microsoft ML.NET team.
Second, this topic has been discussed in the Godot communities, those involved in Godot reinforcement learning, using c# with or without TorchSharp.
The video requires 3D Godot environment requirements which is outside the scope and of interest of ML.NET team, as well as outside the team working on TorchSharp.
InDieTasten commentedon Sep 24, 2024
@GeorgeS2019 Nobody expects ML.NET to feature a 3D environment for agent training.
What we want is for ML.NET to provide an interface to generate neural networks so we can run a fitness function on them and return that fitness value back to ML.NET to adjust the model. There are obviously many different algorithms that could be used under the hood.
With that kind of interface, it would be easy to use any simulation environment, no matter if it's Unity, Godot, or something else entirely. We can spawn agents into environments or entire environments entirely as part of the fitness function.
Flem100 commentedon Sep 24, 2024
@GeorgeS2019 I hope the ML.NET people have a different perspective. I would not think it's in Microsoft's best interest to send people Godot's way instead of Unity, because it's so heavily connected to .NET, which is one of the main reasons I prefer it over anything else. Everything I create in .NET, more less, natively works in Unity. Niiice :)
GeorgeS2019 commentedon Sep 24, 2024
The video suggests a solution similar to Unity for achieving the RL to deliver similar solution in c# beyond the Unity environment.
Unity RL is based on python, which means currently no satisfying RL framework for c# unless very serious effort being invested.
Flem100 commentedon Sep 24, 2024
@GeorgeS2019 Sadly yes, you are are right. But maybe this framework is what we need, at least for now:
https://github.com/asieradzk/RL_Matrix
He is actually going to have a Community standup tomorrow: https://www.youtube.com/watch?v=Aoyj7qzyung
He is going to talk about using his framework in Godot, but I just emailed him and asked about Unity and he responded that he will include an example for Unity as well :D
GeorgeS2019 commentedon Sep 24, 2024
@Flem100
Try using the RL_Matrix. I assume you have not tried Unity RL and know what is the learning experience and why there are so many Unity RL projects in github and why Unity is used in so many e.g. Master projects.
For c#, we need a solution, which will help the transition and easy adoption of an RL framework from people who have used the Unity RL.
If possible, it is feasible to port Unity RL to e.g. Godot RL.
The implementation of RL_Matrix, those seem the best, the design for future choice is to be discussed.
GeorgeS2019 commentedon Sep 24, 2024
@InDieTasten
Before we can get there, we need someone with expertise on how e.g. pytorch address that.
Is any of the Pytorch-based RL frameworks out there addressing that using an interface as you wish?
asieradzk commentedon Sep 27, 2024
We can use all of the Unity's ml-agents sensor components with RLMatrix, all you need to do is use the toolkit. I wrote a wrapper converting ray sensor component and its only couple dozen lines. If there's demand I'll convert all Unity's ml-agents sensor components - they are great one of my favourite things that came out of ml-agents.
https://x.com/adisiera/status/1839015234742210779
in this vid its the same sensor component and environment that was used with ml-agents before.