How We're Teaching Claude to Play Age Of Empires 2
The no-API ground rule, why we keep images out of the model entirely (a local-OCR strategist and a text executor), and how prompt caching keeps the cost low. The first post of the series.
Updated 10 min read
Foundation models and agent architecture evolved enormously in the past months. The agents are very capable of understanding a wide variety of inputs and they can manage complex workflows with dynamic decision points and tools. I was curious how good is this agentic framework to manage and control a live strategic game. I think Age of Empires made a lot of us childhood a bit more exciting and colourful. At least my first thought was this game when I started to think which game should I teach the AI to play.
I am planning to write a longer series about this project because it evolved quite a bit, it would be hard summarising it in one article. The first article is about the project introduction and a few early decision point.
What is AoE2
Age of Empires II is a real-time strategy game from 1999. I think I am not lying if I say, this was the most popular part of the Age of Empires series. You start the Dark Age with a Town Center, a handful of villagers, and limited resources. The game progresses through four ages (Dark, Feudal, Castle, Imperial), and you win by outbuilding and outfighting your opponent.
Three core loops run in parallel throughout the game:
- Economy: villagers gather food, wood, gold, stone. Idle villagers are wasted seconds.
- Military: train units, respond to threats, push objectives.
- Tech progression: research upgrades, age up, unlock better units and buildings.
A good player handles all three at the same time, and our agent has to do the same, based only on what it sees on the screen.
I chose AoE2 for four specific reasons: it's complex enough to be a real test, deterministic enough to measure progress across runs, it has no anti-cheat that blocks screen capture and a personal one, it was my favourite strategy game back then.
Playing without an API
My ground rule was to not modify or inject anything to the AoE2 binary to control the game but to build an agent or agent swarm that controls the game from outside, like a real human would. The inputs are the visual representation of the game state, the map, the indicators on the map, the same things you would observe to build up the game state in your head.
AoE2 is a real-time strategy game, so the game clock never stops. But the agent has to act in discrete steps, so I impose an artificial "turn" on top of the real-time loop: a tick where the pipeline captures the screen, runs perception, and lets the LLM decide on actions. There is a small fixed delay between ticks (0.3s), but most of a turn is spent waiting for the LLM call. The game keeps running between turns, the agent only takes a sample of it every turn. In this series, "turn" always means one of these agent decision cycles, not a pause in the game.
Every turn, the agent receives a structured text representation of the game world: a list of detected entities (sheep_0 at (450, 320), town_center_0 at (512, 400), villager_2 at (380, 290)) plus cached resource readings. It uses that to decide what to do: gather food, queue a villager, build a lumber camp, respond to a threat.
Perception comes from a YOLO model trained on 60 entity classes. Execution happens via pyautogui, which sends mouse clicks and key presses to the game window. The LLM works with text and chooses the actions, and the execution layer carries them out.
So perception, reasoning and execution all have to tolerate noise: the YOLO model misses things, the LLM occasionally makes bad calls, and the game doesn't always respond as expected.
Two-tier design
The simplest approach would be to send a screenshot to a vision model every turn and ask "what should I do?" I dropped this idea quite early for two reasons: speed and cost.
A vision API call with a full screenshot takes 3–8 seconds round-trip, and an RTS game does not pause while you wait. Villagers stay idle and attacks go unanswered in the meantime. So even with free API calls, this agent would be too slow to play the game.
About the cost: At $3.00 per million input tokens, with a screenshot-heavy context, you're looking at roughly $0.01–0.03 per vision call. Across a 30-minute game at 1 call/second, that's $18–54 in API costs per game. I wanted to be cost effective as much as possible.
So I decided to keep images out of the model completely. The screen is processed on my machine first, and the LLM work is split into two tiers that run at different speeds:
The strategist reads the game UI (resources, population, current age) straight off the resource bar with local OCR (RapidOCR), then passes those numbers to the model as plain text. It returns 3–5 prioritized goals. It doesn't run often: every 3 turns in the Dark Age, every 5 in Feudal, every 10 in Castle and Imperial, or right away when an alarm fires (enemy spotted, under attack).
The executor receives text only: the YOLO entity list, the strategist's goals, recent memory and game knowledge. It runs every turn. Routine turns resolve in a single fast structured-output call; combat and housing emergencies fall back to an agentic tool-use loop (up to 7 tool calls). A per-call effort setting (default low) keeps the common case quick.
Both tiers run the same model, claude-sonnet-4-6 (the executor started on Haiku and moved to Sonnet because it followed instructions more reliably). The reason for the split is that setting goals is slow and needed only occasionally, while execution has to be fast and happens every turn, so each tier runs at its own pace.
Resources, population and age, read with local OCR
Strategist
every 3–10 turns, or on an alarm
3–5 prioritized goals
YOLO entities, memory and game knowledge
Executor
every turn
Mouse clicks and key presses sent to the game
Prompt caching
The executor runs every turn. Its system prompt includes a full AoE2 hotkey reference, strategic rules, a per-turn decision checklist, and cross-game memories from prior runs. Together that's roughly 1400 stable tokens before any dynamic context.
Without caching, that block is priced at $3.00/MTok every turn. Anthropic's prompt caching (cache_control: ephemeral) prices cached tokens at $0.30/MTok, a 90% reduction on the static portion.
The cached part of the prompt is split into two blocks:
- Block 1 (~800 tokens):
core.md+hotkeys.md+ cross-game memories. It doesn't change during the game, so after turn 1 it is a cache hit almost every time. - Block 2 (~600 tokens): age-specific prompt (
dark.md,feudal.md, etc.). Cached too. It changes only ~3 times per game when the agent ages up, so within an age it's a cache hit every turn.
The dynamic per-turn context (entity list, resources, recent turns) is not cached, because it changes every turn. That part is small. (On combat turns that run the tool loop, a moving cache breakpoint keeps the growing conversation cached across iterations; more in Episode 5.)
After the cache warms up, an executor turn costs roughly $0.0006, and a full game run costs under a dollar in executor tokens. With this cost, I can run a lot of test games without worrying about the bill.
Screenshot to a vision model
Text executor, cached prompt
Cost per turn, log scale
Without prompt caching, the executor's per-turn costs would dominate and I'd be back to the cost problem from above. Episode 5 goes into the full caching implementation.
Where the agent is today
The agent gathers resources, queues villagers, builds lumber camps and houses, responds to alarm conditions, and advances into Feudal Age. It plays a recognizable Dark Age economy.
Cross-game memory injection means the agent that starts game 50 has patterns from previous runs already in its context, things like "don't build farms before the Mill in Dark Age" or "queue villagers more aggressively in the first 10 turns." Those patterns accumulate in a memory file that gets injected at the start of each game.
Military is weak. The agent understands it should train scouts or militia, but unit composition timing is inconsistent. Long games degrade as the working memory window fills and early-game state falls off.
Multiplayer is not attempted. Turn latency varies by path: routine turns resolve in a few seconds (a single executor call plus the 0.3s loop delay), but combat turns that fall back to the full tool loop can still reach ~20–30 seconds. In both cases the LLM calls take most of the time, which is too slow for real-time PvP.
Beating a skilled human player is not the goal, at least not for now. I am more interested in a working pattern for LLM agents in real visual environments: how to structure the perception-reasoning-execution loop, how to manage costs across many turns, how to accumulate knowledge across runs without fine-tuning. AoE2 is the test environment for that.
Next in the series
The next posts get into the implementation. We'll go through the full game loop step by step, with real timing numbers for each phase, to see where the time goes and which steps slow down a real-time agent.
A big chunk of the series is computer vision. AoE2 is an isometric game with overlapping sprites, dense unit clusters, and a UI that changes between ages. Training a YOLO model to detect dozens of entity classes in this environment needed a lot more than collecting data. We'll cover the taxonomy design, why the detection resolution has to match the size the model trained on (the tiling pipeline I built made real-world accuracy worse), and tracking detections across frames so the agent sees stable identities instead of flickering boxes. And because hand-labelling tens of thousands of game screenshots was not an option, we'll also cover how synthetic training data is generated from the game's own sprites, and what it took to make models trained on that data transfer to real gameplay.
On the agent side, the series digs into prompt engineering for real-time play: the two-block caching strategy, how prompts shift as the game progresses through the ages, and the per-turn decision checklist that keeps the executor from drifting. There's a separate post on memory: how the agent carries lessons from one game into the next without fine-tuning, using a feedback loop on its own actions and a cross-game memory chain.
There will be a post about the layer between the LLM and the game: translating "click on this villager" into mouse coordinates that hit the right pixel, handling the cases where the game doesn't respond, and composing multi-step actions into single tools the LLM can call. And finally, testing: how to iterate on the agent without sitting through full games, using a synthetic arena where you can fork a game state, run variants against each other, and change prompts to see which change helps.
Every post is based on the real implementation, with code and numbers, including the parts that didn't work. Next up is the full game loop, annotated with timing.
Tools and references3
- Ultralytics YOLO
The 60-class entity detector that gives the agent its "eyes"
- PyAutoGUI
Sends mouse clicks and keypresses to the game window
- Anthropic prompt caching
`cache_control: ephemeral`, 90% cheaper input tokens for the static prompt