Real vs Synthetic Training Data

I'm building an AI agent that plays Age of Empires II, a real-time strategy game. The agent has no access to the game's internal state. It only sees the screen, like a human player. On every turn, an object detector processes the screenshot and returns the visible objects with their class and position: villagers, animals, resources, buildings and military units. The agent plans its next action from that list.

This post is only about training that detector. It addresses a common question for small datasets: how much does synthetic training data help, and how is it best used?

The starting point

Labelling real screenshots is slow. A typical screenshot has 45 objects to box by hand, and I had 264 labelled screenshots. To fill the gap, the training set also had 2,400 synthetic images: sprites cut out of the game files and pasted onto generated backgrounds, with labels generated automatically. Two thirds of the training data was synthetic.

The goal of this work was to measure what those images are worth. Does the model get better with them, worse, or stay the same? And if they help, is there a better way to make or use them?

The first check was simple: compare the synthetic images with real ones. I measured the median box size of a few common objects:

The synthetic objects were 2–3× bigger than the real ones. The generator drew sprites at full size on a 1280×720 canvas, while real screenshots are 3024 pixels wide and get shrunk to 42% before the model sees them. Scale augmentation (0.5–1.5×) can't close a gap that wide.

The synthetic data could still help despite the size mismatch, or it could teach the model incorrect object sizes. To measure this, I ran several training rounds with and without it and compared the results.

The detector

The model is YOLO26n from Ultralytics: a 2.6M-parameter detector, pretrained on COCO and fine-tuned on my data. It's the smallest YOLO26, because the agent needs a new detection every ~100 ms.

It learns 65 classes: trees, gold and stone mines, berry bushes, animals, buildings, villagers and military units. The screenshots are 3024×1672 and the model sees them at 1280 pixels wide, so a villager is ~14 pixels tall at the model input. This small object size is the main constraint behind the input resolution and several of the training choices below.

The real data is 264 screenshots with 11,842 boxes. Villagers, trees and houses make up half of the boxes. Some classes, like kings or trebuchets, have no real example at all and only exist in the synthetic data.

How I compared the runs

The validation set (8 screenshots) was too small to show meaningful differences between models: two earlier models scored F1 0.771 and 0.772, a gap within noise. For evaluation, I built a separate test set of 40 screenshots (1,248 hand-reviewed boxes) from 10 game sessions not used in any training run. The training script excludes those sessions to prevent leakage.

Each model is scored with the same confidence thresholds. A detection counts as correct if it overlaps the label with IoU ≥ 0.5. I report precision (the share of detections that are right), recall (the share of objects found) and F1, which combines the two.

Screenshots from the same game look alike, so I compare models per game session, with a permutation test. It gives a p-value: how likely a difference this big is if the two models were actually equal. On this test set, anything below ~0.02 F1 is noise.

The training setup

Every run in this post starts from this configuration. Each round changes one or two lines of it.

Augmentation is close to the Ultralytics defaults, with two choices specific to a game:

The hardware is one RTX 4090 (24 GB) per run, rented on RunPod. Images are cached in RAM to keep the GPU busy. An epoch takes 20–90 seconds depending on the dataset and model size, so a run takes 45 minutes to 4 hours. I ran up to three at once, one per pod.

RAM limits differed between pods with the same GPU type: 31, 61 and 83 GB. Run E ran on the 31 GB pod with RAM caching and 16 data loader workers. At epoch 141 of 150, Ultralytics disables mosaic and rebuilds the data loader; worker memory briefly doubled and the pod terminated the run. Training can be resumed from the last checkpoint, with fewer workers if needed:

YOLO("runs/exp_e/weights/last.pt").train(resume=True, workers=8)

Round 1: four ways to use synthetic data

The first round changes only the synthetic data. Everything else is the configuration above.

The composites address the size problem. Instead of drawing a background, the generator takes a real training screenshot and pastes sprites into it at the correct size. Real ground, fog and game interface are included automatically, and every object already in the screenshot has a label. Units go in as groups, never on black unexplored map and never over a labelled object. Trees and mines are not pasted, because real labels often put one box around a whole forest.

Run C used batch 8 for its pretraining phase. Composites are dense: a mosaic of four contains hundreds of boxes, and the label assigner ran out of GPU memory at batch 16.

The results:

Removing the synthetic data reduced F1 by 0.056, the largest effect in this round. Composites improved on real-only training by ~0.04. Mixing (B) and pretraining (C) gave the same result, consistent with Burdorf et al.. Despite the correct scale, the composites did not outperform the old synthetic images.

Per-class results show where the difference comes from:

With the composites, the model found more animals but fewer trees and gold mines, the two classes the composites don't paste. The old synthetic images have trees and gold mines in every picture, even if at the wrong size.

The runs also differ in the number of training steps (last column of the first table): run A gets 9,900 and the baseline 32,400, because a smaller dataset means fewer batches per epoch. Round 1 did not control for this; Round 3 shows why it matters.

Round 2: combining both data sources

Before the next round I fixed a labelling error. Run B had seen 1,749 pasted goats but detected none of the 34 goats in the test set. Most of the "goat" labels turned out to be ibex, a wild animal that is hunted, rather than goats, which are herded; the model's predictions were correct. I relabelled them and trained ibex as deer, the class the agent uses for huntable animals:

# classes.yaml
label_aliases:
  - label: ibex # label what you see
    maps_to: deer # train the class the agent acts on

The next round combined both sources: the old images for trees and mines, and the composites for animals.

The label fix alone changed nothing (E equals the baseline). Adding the composites gave the best model so far. The bigger YOLO26s model on the old data didn't beat the small model on the better data. E-s used batch 8 to fit in memory, but Ultralytics accumulates gradients to the same nominal batch of 64, so the update size stays the same.

D detected fewer gold mines: 46 of 71, against 56 for E (p = 0.016).

On the training screenshots, D detects 92% of the gold mines and E only 56%. D fits its training data much more closely than E, which pointed to the training process rather than the data.

Round 3: separating the optimizer from the data

Every run so far used optimizer: auto. The training logs show which optimizer it selected:

E:  optimizer: AdamW(lr=0.000145, momentum=0.9)
D:  optimizer: MuSGD(lr=0.01, momentum=0.9)

Simplified from Ultralytics 8.4.54:

iterations = math.ceil(len(train_set) / max(batch, 64)) * epochs
if optimizer == "auto":
    if iterations > 10_000:
        name, lr0 = "MuSGD", 0.01
    else:
        name, lr0 = "AdamW", round(0.002 * 5 / (4 + num_classes), 6)  # 0.000145 for 65 classes

E has 3,456 images per epoch: 54 × 150 = 8,100 iterations, so AdamW. D has 4,456: 70 × 150 = 10,500, so MuSGD with a 70× higher learning rate. The limit is at ~4,225 images. D was the only run that crossed it. Every other run in this post, including all of Round 1, trained with AdamW at 0.000145.

As a result, the comparison between D and E changed two variables at once. All runs also used seed 0, so the run-to-run variance was unknown.

To separate them, I set the optimizer explicitly and ran a 2×2: two datasets (E, D) × two optimizers, each trained twice with different seeds. Everything else stayed the same:

warmup_bias_lr=0.0 matches what auto sets, so the new runs reproduce the earlier ones exactly. Two of the eight runs already existed (E and D), so this round was 6 new runs on three pods, ~20 GPU hours.

With AdamW at this learning rate, the model is undertrained. Switching to MuSGD and adding composites each improve F1 by about 0.02; combined, they add little more, which suggests that both address the same limitation. All Round 1 runs used AdamW, so Round 1 compared the synthetic datasets on an undertrained model.

The gold mine drop was within run-to-run variance. With seed 1, run D detects 73% of gold mines, the same as E with MuSGD. Overall F1 varies by about 0.005 between seeds, but gold mine recall varied by up to 0.085. The 71 gold mine boxes come from about 7 physical mines seen in consecutive screenshots, so a few mines determine the score. The earlier significance test compared two single runs and therefore did not account for training variance.

D also has more images, so at the same batch size it gets 29% more training steps (279 per epoch against 216). Part of the composite gain may therefore come from the additional steps.

The configuration I train with now

All settings are now explicit:

from ultralytics import YOLO
 
model = YOLO("yolo26n.pt")
model.train(
    data="dataset.yaml",          # real ×4 + 2,400 old synthetic + 1,000 composites
    imgsz=1280, epochs=150, batch=16,
    optimizer="MuSGD", lr0=0.01, momentum=0.9, warmup_bias_lr=0.0,
    seed=0, deterministic=True,
    patience=0,                   # no early stopping; keep the final epoch
    mosaic=1.0, close_mosaic=10, scale=0.5, translate=0.1, degrees=10,
    fliplr=0.5, flipud=0.0, mixup=0.1,
    hsv_h=0.015, hsv_s=0.7, hsv_v=0.4,
    cache="ram", workers=12,      # check the pod's RAM limit, not the GPU
)

Results are compared over at least two seeds.

The outcome

The model the agent runs now is run D. The second seed scored 0.003 higher, inside the noise, so I kept the run that had already passed the release checks. I also exported it differently: a 736×1280 input instead of 1280×1280, which fits a wide screenshot with less grey padding, and YOLO26's second output head with a separate duplicate-removal step (NMS). With the same weights, this export scores F1 0.873 and runs in 10 ms per image instead of 15 ms on an Apple M2 Pro.

Against the model the agent used before, on the same 40 screenshots:

The F1 improvement is +0.031 (p = 0.045), and the new model is better in 8 of the 10 test sessions. Per class:

It detects more animals and villagers, slightly fewer trees and mines, and produces more false detections: 77 against 36. The model is on Hugging Face as martondobos/aoe2-entity-detector.

What I'd do differently

Set the optimizer, learning rate and seed explicitly from the first run, and check the optimizer line in the training log. auto depends on the dataset size, so any experiment that changes the data can also change the optimizer. In this project the switch happened at ~4,225 images.

Keep the number of training steps equal across data experiments, or at least report it with the results.

Train every configuration at least twice. Overall F1 varied by 0.005 between seeds, while a single class varied by 0.085. A second seed would have settled the gold mine question immediately.

Measure the synthetic data before training on it. The size check took ten minutes, and the problem was not visible in the training curves.