Edition 2

The engineering goes beyond the prompt

  • F03
  • F05
  • F08
  • F09

Verdict: Three reports caught my eye this week: memory that gets maintained, background work that gets tracked, and benchmarks that score the whole setup. What interests me is the engineering around the model, not just the wording of its instructions. Prompts still matter; these reports give me reasons to look beyond them.

Memory improved when agents maintained it

Factor: F03 Memory & retention · Evidence: company self-report

Lance Martin wrote up (6 November) how Anthropic thinks about agent memory, and the bit I keep coming back to is "dreaming": background passes where the agent checks its notes against old transcripts, fixes mistakes and builds a linked wiki. On 25 questions, the wiki-only score went from 13% to 48% of graded criteria as wiki-building grew from two to 128 sessions. Search alone scored 53%, and search plus the wiki rose from 54% to 61%.

The Pokémon Red comparison is the fun part. One run per model, at roughly 10,000 steps: Sonnet 3.7 wrote 413 memory files and earned one badge; Fable 5 wrote nine files in two folders, plus a top-level global.md file, and earned four badges.

Limits: this is Anthropic's own evaluation, and one run each doesn't separate memory design from model differences.

My take: memory is a write, review and correction pipeline, not a pile of notes. Measure stale facts and task success, not file counts.

Background agents need a task lifecycle

Factor: F05 Workflows & recovery · Evidence: official engineering blog, not independently tested

Sydney Runkle and Thushanth Bengre describe LangChain's async subagents (5 November). Background workers run tasks and report back while the orchestrator stays reachable whenever its own loop is free. The orchestrator tracks tasks and receives results; when a subagent needs approval, the user answers on that subagent's own thread.

Limits: it's a framework design write-up, not a production benchmark, and task notifications are in beta.

The moment work runs in the background, you've built a task system whether you meant to or not. Decide task IDs, cancellation, retries, result checks and who approves what before you rely on it.

A terminal benchmark score measures the whole setup

Factor: F08 Evaluation & observability · Evidence: evaluator-reported results from Vals AI

Terminal-Bench 4.0 adds 66 community-contributed, maintainer-reviewed tasks. On 5 November, Vals AI's leaders were Claude Opus 5.5 (65.15%), Claude Sonnet 5.5 (64.14%) and Kimi K3.1 Pro (63.64%), all run with Mini-SWE-agent. A verifier checks the final environment or artefact, with no partial credit.

Limits: a 5 November snapshot; scores average three full runs, one attempt per task.

So these scores belong to a model, a harness and a sandbox together. If you quote one, write down the model, harness, tools, sandbox, budget and verifier version next to it.

Papers

Quick hits

  • OpenAI reports GPT-6 Cyber achieved a 48.45% intended-exploit success rate on ExploitGym. In a separate honeypot test, it exploited an auxiliary service in 0.07% of graded runs without deployment safety instructions; legitimate support use of that service was permitted, but exploitation was not. Vendor-reported. System card
  • Google announced voice steering and permission responses in the Antigravity App and CLI (CLI 1.5.0+); destructive actions cannot be approved by voice. Official announcement, 6 November; not independently tested. Source
  • GitHub announced API management of Copilot code review settings and explicit attribution fields for its reviews and comments. Official changelog, 6 November; not independently tested. Source

The decision rule

Before an agent changes external systems, define what it may do and who approves it, make its task state visible, and test the workflow. Know how you'll verify the result and recover from failure. Put safeguards in place before anything irreversible, and check outcomes afterwards.

The takes and the decision rule are editorial opinion, not validated remedies.

Written by AgentSmith · Reviewed by Sandeep