More recently, ml-agents have updated to v0.4. For those who do not know this is an open source plugin, which is an environment for training agents in unity. They can be trained using reinforcement learning, imitation learning, neuroevolution, or other machine learning techniques using the Python API. It also provides a number of modern algorithms (based on TensorSlow), which allow you to create smarter NPCs for your games.
What do you need to implement machine learning in Unity?
Download this plugin can
link . You will need the unity-envonment folder. Before this, create an empty project. In the Assets folder, add the contents of the Assets folder with unity-envonment. Do this with ProjectSettings. Please note that if you add this plugin to an existing project, first create an empty project, complete the above description above and create a package (Assets-ExportPackage) and then just import it into your existing project. This is necessary so that you do not lose the existing ProjectSettings settings in your project.
I will show you how to implement machine learning using the example of a soccer game (this scene already exists in the plugin). It has two teams, blue and red. Each team has a striker and a goalkeeper.
Let's go back to the plugin itself. In order for it to work on stage we need at least one Academy, Brain and Agent object. See the interaction diagram below.
On the stage are the object Academy and its subsidiaries GoalieBrain, StrikerBrain. As we can see from the diagram, each Academy can have several Brains, but each Brain should have one Academy.
Agent is our game objects we want to train.
Next, go to the code. The SoccerAcademy class, which is added to the GameObject Academy inherited from the Academy class, is preceded by adding a space using MLAgents. After that, we should override the methods.
public override void AcademyReset() { } public override void AcademyStep() { }
And add links to created Brain
public Brain brainStriker; public Brain brainGoalie;
Add a Brain component to GoalieBrain, StrikerBrain and assign them to Academy
The AgentSoccer class is inherited from Soccer. Add it to each player and indicate which Brain will be used.
In the AgentSoccer script we describe the logic of the players. In the overridden CollectObservations () method, we add a list of rays that will help players monitor and respond to gameplay.
public override void CollectObservations() { float rayDistance = 20f; float[] rayAngles = { 0f, 45f, 90f, 135f, 180f, 110f, 70f }; string[] detectableObjects; if (team == Team.red) { detectableObjects = new string[] { "ball", "redGoal", "blueGoal", "wall", "redAgent", "blueAgent" }; } else { detectableObjects = new string[] { "ball", "blueGoal", "redGoal", "wall", "blueAgent", "redAgent" }; } AddVectorObs(rayPer.Perceive(rayDistance, rayAngles, detectableObjects, 0f, 0f)); AddVectorObs(rayPer.Perceive(rayDistance, rayAngles, detectableObjects, 1f, 0f)); }
The overridden AgentAction method is an analogue of Update. In it, we call the move method and distribute the rewards.
public override void AgentAction(float[] vectorAction, string textAction) {
If you touch the ball, push it
void OnCollisionEnter(Collision c) { float force = 2000f * kickPower; if (c.gameObject.tag == "ball") { Vector3 dir = c.contacts[0].point - transform.position; dir = dir.normalized; c.gameObject.GetComponent<Rigidbody>().AddForce(dir * force); } }
Reset commands when the target is completed.
public override void AgentReset() { if (academy.randomizePlayersTeamForTraining) { ChooseRandomTeam(); } if (team == Team.red) { JoinRedTeam(agentRole); transform.rotation = Quaternion.Euler(0f, -90f, 0f); } else { JoinBlueTeam(agentRole); transform.rotation = Quaternion.Euler(0f, 90f, 0f); } transform.position = area.GetRandomSpawnPos(team.ToString(), agentRole.ToString()); agentRB.velocity = Vector3.zero; agentRB.angularVelocity = Vector3.zero; area.ResetBall(); }
Learning options
You can train agents using the unity editor or using
TensorFlow . As for unity, in Academy you need to specify the punishment and the reward for each Brain.
It is also necessary to specify Brain by its TypeBrain by setting the value to External. They also come in the following types:
- External - where decisions are made using the Python API. Here, the observations and rewards generated by Brain are redirected to the Python API via an external communicator. The Python API then returns the corresponding action the agent must perform.
- Internal - where decisions are made using the built-in model TensorFlow. The embedded TensorFlow model is a science policy, and Brain directly uses this model to determine actions for each Agent.
- Player - where decisions are made using real keyboard input or controller. Here, the human player controls the Agent, and the observations and rewards collected by the Brain are not used to control the Agent.
- Heuristic - where decisions are made using hard-coded behavior. This is similar to how the majority of characters' behavior is currently defined and can be useful for debugging or comparing how an Agent with hard-coded rules is compared with an Agent whose behavior has been trained.