Scripted model & fake sandbox

Two test-kit pieces let you run a real agent loop in a unit test: scriptedModel / scripted_model plays back model replies you write, and fakeSandbox / fake_sandbox is an in-memory sandbox. Pair them with an in-memory store, sqlite(":memory:"), and a test touches no network and no disk.

A complete test

import { expect, test } from "bun:test";
import { agent, fakeSandbox, scriptedModel, sqlite } from "@threads/core";

test("the fixer runs the tests and reports", async () => {
  const model = scriptedModel({
    responses: [
      { error: { reason: "rate_limited", http_status: 429, retry_after_ms: 1 } },
      {
        content: [{ type: "tool_use", call_id: "c1", name: "bash", input: { command: "pytest -q" } }],
        stop_reason: "tool_use",
        usage: { input_tokens: 30, output_tokens: 6 },
      },
      {
        content: [{ type: "text", text: "All 12 tests pass." }],
        stop_reason: "end_turn",
        usage: { input_tokens: 50, output_tokens: 7 },
      },
    ],
  });
  const sandbox = fakeSandbox({
    tools: { pytest: { output: "12 passed in 0.4s\n" } },
  });
  const fixer = agent({ model, sandbox, permissions: { mode: "bypass" } });

  const result = await fixer.run("Run the tests", { store: sqlite(":memory:") });

  expect(result).toMatchObject({ status: "completed", output: "All 12 tests pass." });
  expect(model.remaining()).toBe(0);
  expect(sandbox.execs().length).toBe(1);
});

The first reply is a rate-limit error, so this test also covers the agent's retry path. The agent's bash call runs pytest -q in the fake sandbox, which answers with the scripted output.

The model script

responses is played in order, one entry per model call. Each entry is one of:

usage is { input_tokens, output_tokens }. A reply can mix text and several tool calls in content. Tool calls go through the same argument parsing, permissions and hooks as with a real model, so the test exercises your real tools.

After the run, check that the script was used up:

• TypeScript: model.remaining() counts replies never asked for; model.unexpected() counts calls past the end. A call past the end fails the run.
• Python: model.remaining is a property; model.sent lists the requests the model received. A call past the end raises ScriptExhaustedError.

The script is the same format saveCase writes to model.json, so a saved case doubles as a script.

The fake sandbox

fakeSandbox() keeps files in memory, so the built-in file tools such as read, write and edit work against it. For shell commands, script the output by the command's first word:

fakeSandbox({ tools: { pytest: { output: "12 passed in 0.4s\n" } } });

Add is_error: true to make the command exit with status 1. Any other command answers command not found. It snapshots like a real provider, so fork and saved cases work in tests too, and it blocks all network access.

In TypeScript, sandbox.execs() lists every command the agent ran and sandbox.creates() counts sandboxes created. In Python, sandbox.creates is an attribute.

Never hit a real model by accident

Turn on the model-request guard once for your whole test run. After that, any request to a model that isn't a scripted one throws before anything is sent.

// test/setup.ts, loaded by bunfig.toml: [test] preload = ["./test/setup.ts"]
import { blockRealModels } from "@threads/core";

blockRealModels();