Strategic Agent Navigation using Deep RL

01 Challenge

An agent has to collect targets in a closed arena, evade a pursuing opponent and avoid lethal walls – without rigid, rule-based logic.

02 Approach

Unity ML-Agents with PPO: 360° ray perception plus vector observations, an iteratively designed reward function and training over 2 million simulation steps.

03 Result

Average reward rose from -9.0 to over 70.0 (~34 pellets per life); the agent uses its dash tactically and “kites” the opponent.

Stack: Unity 2022 LTSML-AgentsPyTorchPPOC#ONNX

This project demonstrates the potential of Deep Reinforcement Learning to solve complex logistical challenges in dynamic environments. The focus is on developing autonomous agents that learn to efficiently reach goals, prioritize resources, and navigate hazards in real time — a scenario directly applicable to autonomous warehouse logistics or navigation systems.

Rather than relying on rigid, rule-based algorithms, this system uses a reward-based learning architecture. Through millions of simulation steps, the agent autonomously optimizes its strategies for maximum efficiency and risk minimization.

Training steps: 2M; PPO in Unity ML-Agents
Avg. reward: -9 → 70+; from phase 1 to phase 3
Pellets per life: ~34; once fully trained
Perception: 360°; Ray Perception Sensors 3D

Unity scene: arena with red walls, pink agent, green zombie and blue pellet — Fig. 1 — The training environment in Unity

Unity scene hierarchy with plane, four walls, agent, pellet and zombie — Fig. 1 — The training environment in Unity

The Scenario: Survive at All Costs

The simulation consists of three main actors in a closed arena:

The Agent (Pink): The AI protagonist. It can move and has a special “Dash” ability (a fast sprint) with a cooldown.
The Zombie (Green): A simple bot that constantly pursues the agent. A touch ends the episode immediately (-10 points).
The Pellet (Blue): The target object. Collecting it earns +5 points and respawns the pellet at a new random position.

Additionally, the walls are lethal. The agent had to learn not only to evade the zombie, but also to avoid being cornered.

The Technical Approach

The project is based on Unity and the ML-Agents Toolkit. The training algorithm used was PPO (Proximal Policy Optimization), a standard method for continuous control problems.

01PerceptionRays detect wall, pellet, zombie + distanceRay sensors
02ObservationPosition, velocity, target, dash statusVector obs
03DecisionNeural network picks movement and dashPPO policy
04ActionPhysics-based movement via the rigidbodyC#
05RewardReward/penalty feeds back into trainingReward

Fig. 2 — The agent's learning loop per simulation step

1. Perception (Sensors)

To allow the agent to “see” its environment, I equipped it with Ray Perception Sensors 3D. Think of this like a LIDAR system: the agent emits invisible rays in a 360-degree radius. These rays tell it whether a wall, a pellet, or the zombie is in a given direction, and how far away it is.

In addition, the neural network receives numerical data (vector observations):

its own position and velocity,
the relative position to the target,
the status of the “Dash” cooldown (ready or not).

2. The Reward System

The most critical part of reinforcement learning is the design of rewards. The final system:

Event	Reward	Intent
Existence	+0.001 per step	Rewards pure survival
Pellet collected	+5.0	The main goal
Emergency dash	+0.2	Dash while the zombie is very close
Death (wall or zombie)	-10.0	Catastrophic failure, episode ends

3. Training

Training ran for 2 million steps. Learning progress can be divided into three phases:

Phase	Steps	Avg. reward	Behaviour
1	0 – 200k	-9.0	Barely moves or runs randomly into walls
2	~500k	+17.0	Understands that pellets are good, but still gets caught often
3	1.5M+	over 70.0	Uses the dash tactically and “kites” the zombie efficiently

In phase 3 the agent reaches “superhuman” performance: it evades the zombie at the last moment with a dash and collects around 34 pellets per life on average without dying.

The Result

The following video shows the fully trained agent (with the ONNX model) in action. Notice how it waits until the zombie is close, then uses the Dash to escape through the gap.

Fig. 3 — The trained agent dashes past the zombie and collects pellets

Technologies Used

Engine: Unity 2022 LTS
ML Framework: Unity ML-Agents (PyTorch Backend)
Algorithm: PPO (Proximal Policy Optimization)
Language: C# (for game logic and agent control)
Inference: Barracuda / ONNX Runtime

Code Insight

Here is an excerpt from AgentController.cs, showing how the agent translates decisions (actions) into movement and distributes rewards:

public override void OnActionReceived(ActionBuffers actions)
{
    // 1. Dash logic (discrete action)
    int dashAction = actions.DiscreteActions[0];
    if (dashAction == 1 && m_DashCooldownTimer <= 0)
    {
        m_IsDashing = true;
        // Reward for tactical dash in danger situations
        if (zombie != null && Vector3.Distance(transform.localPosition, zombie.localPosition) < 2.5f)
        {
            AddReward(0.2f);
        }
    }

    // 2. Movement (continuous actions)
    float moveX = actions.ContinuousActions[0];
    float moveZ = actions.ContinuousActions[1];
    
    // Physics-based movement
    if (m_Rb != null)
    {
        Vector3 velocity = new Vector3(moveX * currentSpeed, 0, moveZ * currentSpeed);
        m_Rb.linearVelocity = velocity;
    }

    // 3. Collision handling (penalty)
    // Processed in OnCollisionEnter:
    // Wall -> AddReward(-10f); EndEpisode();
    // Zombie -> AddReward(-10f); EndEpisode();
}

Learnings

Reward design is the lever: small changes to the reward structure (wall penalty -1 → -10) fundamentally change the learned behaviour.
Targeted incentives for tactics: the small bonus for the emergency dash (+0.2) steers the agent towards using its special ability at the right moment.
Transferability: the combination of sensing, goal prioritization and hazard avoidance is directly applicable to autonomous warehouse logistics or navigation systems.