How It Works

The game agent loop

Every decision the agent makes is a vision LLM call — no hardcoded game logic, no emulator hooks, no ROM hacking. The agent sees the screen and decides what buttons to press.

Each cycle:

1. Perceive — capture a screenshot via Win32 PrintWindow through MCP
2. Analyze — Phase 1 vision call extracts scene structure, dialogue text, character position, NPCs
3. Evaluate — scores progress relative to the previous frame and flags stuck states
4. Strategize — the Superego fires on events (screen change, dialogue end, stuck streak, checkpoint, heartbeat) and issues a directive
5. Plan — the Ego integrates analysis, evaluator feedback, directive, retrieved memories, and context-matched skills
6. Act — button presses are sent to the emulator

Event-driven Superego

In v1 the Superego ran every N cycles regardless of what happened. In v2 it is triggered by concrete events — so it thinks when something has changed, not on a clock. Each firing records its trigger_reason so the mix can be inspected afterwards.

A directive is a structured object — objective, priority, warnings, hints, quest_state — not a free-text dump. The Ego receives it as high-signal guidance.

Skill library — procedural knowledge

Skills are YAML-frontmatter markdown files describing generalizable procedures ("open the menu", "dismiss a dialogue box", "leave a shop"). They're indexed by context — screen type and location keywords — and the matching ones are injected into the Ego's prompt when they apply. Each skill tracks its success count and last-used timestamp, so the library improves over time.

Skills are stored per-(agent, model) on disk so one model's learnings never leak into another model's benchmark run. The Superego can also propose new skills when it recognizes a pattern worth generalizing.

CORE.md — curated notes

A bounded, agent-writable markdown file (max 2500 chars) that holds durable facts the agent decided were worth remembering: NPC quirks, map landmarks, recurring game mechanics. Separate from the episodic SQLite memory tiers.

CORE.md is loaded once at session start and frozen into the prompt prefix so it benefits from prompt caching. Every write is scanned for prompt-injection patterns before being accepted.

Three-tier memory with async consolidation

Episodic memory has three tiers — short-term, medium-term, long-term — and runs an asynchronous consolidation loop that compresses short-term observations upward as they accrue importance. Importance-weighted eviction keeps each tier bounded.

Two per-cycle health metrics surface on run pages: recall_topic_count (how many distinct topics the retrieval step pulled from) and recall_top_topic_share (what fraction of the retrieved memories clustered on one topic). Together they show whether memory is fragmenting or collapsing.

Per-(agent, model) isolation

Every benchmark run starts from a blank slate. Each (agent, model) pair has its own soul file, memory database, skill library directory, CORE.md, and log folder. Runs are fully comparable across models with no cross-contamination.

The harness supports Anthropic direct, OpenRouter, and LM Studio. The reasoning_budget field caps thinking tokens for extended-reasoning models.

Pure vision — no shortcuts

The agent has no access to game memory, no tile maps, no scripted paths. It sees pixels and presses buttons. This makes ChronoBench a true test of visual reasoning, spatial navigation, and long-horizon planning.