This project demonstrates the potential of Deep Reinforcement Learning to solve complex logistical challenges in dynamic environments. The focus is on developing autonomous agents that learn to efficiently reach goals, prioritize resources, and navigate hazards in real time — a scenario directly applicable to autonomous warehouse logistics or navigation systems.

Rather than relying on rigid, rule-based algorithms, this system uses a reward-based learning architecture. Through millions of simulation steps, the agent autonomously optimizes its strategies for maximum efficiency and risk minimization.

The Scenario: Survive at All Costs

The simulation consists of three main actors in a closed arena:

1. The Agent (Pink): Our AI protagonist. It can move and has a special “Dash” ability (a fast sprint), but with a cooldown.
2. The Zombie (Green): A simple bot that constantly pursues the agent. A touch means the immediate end of the episode (-10 points).
3. The Pellet (Blue): The target object. Collecting it earns points (+5) and respawns the pellet at a new random position.

Additionally, the walls are lethal. The agent had to learn not only to evade the zombie, but also to avoid being cornered.

The Technical Approach

The project is based on Unity and the ML-Agents Toolkit. The training algorithm used was PPO (Proximal Policy Optimization), a standard method for such continuous control problems.

1. Perception (Sensors)

To allow the agent to “see” its environment, I equipped it with Ray Perception Sensors 3D. Think of this like a LIDAR system: the agent emits invisible rays in a 360-degree radius. These rays provide information about whether a wall, a pellet, or the zombie is in a given direction, and how far away they are.

In addition, the neural network receives numerical data (vector observations):

• Its own position and velocity.
• The relative position to the target.
• The status of the “Dash” cooldown (ready or not).

2. The Reward System

The most critical part of reinforcement learning is the design of rewards. My final system looked like this:

• Existence: +0.001 per step (rewards pure survival).
• Pellet: +5.0 (the main goal).
• Death (wall/zombie): -10.0 (catastrophic failure).
• Emergency Dash: +0.2 (if the zombie is very close and the agent dashes, this is reinforced).

Initially, the penalty for walls was too low (-1). This caused the agent to prefer running into the wall (faster death) rather than face the stressful zombie situation. After increasing it to -10, it learned to fear walls as much as the zombie.

3. Training

Training ran for 2 million steps.

• Phase 1 (0 – 200k steps): The agent barely moves or runs randomly into walls. Average reward was negative (-9.0).
• Phase 2 (500k steps): The agent understands that pellets are good, but still gets caught often. Reward becomes positive (+17.0).
• Phase 3 (1.5M+ steps): “Superhuman” performance. The agent uses the Dash tactically to evade the zombie at the last moment and “kites” it efficiently to reach pellets. Average reward rose above 70.0 (equivalent to ~34 collected pellets per life without dying).

The Result

The following video shows the fully trained agent (with the ONNX model) in action. Notice how it waits until the zombie is close, then uses the Dash to escape through the gap.

Technologies Used

• Engine: Unity 2022 LTS
• ML Framework: Unity ML-Agents (PyTorch Backend)
• Algorithm: PPO (Proximal Policy Optimization)
• Language: C# (for game logic and agent control)
• Inference: Barracuda / ONNX Runtime

Code Insight

Here is an excerpt from AgentController.cs, showing how the agent translates decisions (actions) into movement and distributes rewards:

public override void OnActionReceived(ActionBuffers actions)
{
    // 1. Dash logic (discrete action)
    int dashAction = actions.DiscreteActions[0];
    if (dashAction == 1 && m_DashCooldownTimer <= 0)
    {
        m_IsDashing = true;
        // Reward for tactical dash in danger situations
        if (zombie != null && Vector3.Distance(transform.localPosition, zombie.localPosition) < 2.5f)
        {
            AddReward(0.2f);
        }
    }

    // 2. Movement (continuous actions)
    float moveX = actions.ContinuousActions[0];
    float moveZ = actions.ContinuousActions[1];
    
    // Physics-based movement
    if (m_Rb != null)
    {
        Vector3 velocity = new Vector3(moveX * currentSpeed, 0, moveZ * currentSpeed);
        m_Rb.linearVelocity = velocity;
    }

    // 3. Collision handling (penalty)
    // Processed in OnCollisionEnter:
    // Wall -> AddReward(-10f); EndEpisode();
    // Zombie -> AddReward(-10f); EndEpisode();
}