EC2 Fleet Evaluations

See the generated ct run eval reference for the current EC2 and fleet options.

Isolated settings

Fleet jobs automatically use each isolated setting's dependencies and the shipped Control Tower checkout. No custom worker_setup script is needed to activate the setting.

Worker AMI (Speed Optimization)

Fleet workers boot ~3x faster with a pre-baked AMI that includes Docker and pre-pulled Docker images:

uv run ct cloud build-ami  # ~15-20 min, one-time setup

Without AMI: ~230s bootstrap. With AMI: ~65s bootstrap. See EC2 Sandbox docs for details.

Fleet Streaming Logs

When you run --ec2, the controller prints only a compact carriage-returned progress line ([N/M] X done, …). To follow per-trajectory results live — from another shell or an LLM — tail logs/fleet/<fleet_id>/events.jsonl. Every line is one JSON object; types include:

  • fleet_start — fleet_id, expected jobs, worker count
  • worker_online — instance_id, bootstrap_s
  • result — job_id, task, model, per-scorer scores, error, duration_s, instance_id. Emitted the instant each sample finishes mid-batch via an Inspect AI hook (worker.py's FleetSampleStreamer) that posts a sample_done SQS message — not at batch-end. So with --concurrency-per-worker 10, you see 10 separate result events as each of the 10 concurrent samples lands, not one burst when the batch completes.
  • job_done — job_id, status, instance_id, duration_s, failure_category, error_summary. Emitted at batch-end from the SQS done queue (the worker's existing flow). Useful for operational debugging.
  • fleet_summary — final counts

The startup banner prints the exact path.

Recipes:

# Watch failures land:
jq -c 'select(.type=="job_done" and .status!="completed")' \
  logs/fleet/<fleet_id>/events.jsonl

# Live per-task scores as each trajectory finishes:
jq -c 'select(.type=="result") | {task, status, scores, duration_s}' \
  logs/fleet/<fleet_id>/events.jsonl

# Running pass-rate across completed trajectories:
jq -s '[.[] | select(.type=="result")] | {n: length, passed: ([.[] | select(.scores.linux_scorer.value.main_task_success=="C")] | length)}' \
  logs/fleet/<fleet_id>/events.jsonl

# Pull a failed .eval for inspection (workers also upload full .eval at batch-end):
aws s3 cp s3://<bucket>/<fleet_id>/<job_id>.eval /tmp/job.eval
uv run inspect view /tmp/job.eval