Building for speed: how we ship at the pace of the frontier
In the first two posts in this series, we covered how we’ve pushed GTM tech forward on two fronts: building eval sets for the edge cases unique to our industry, and cutting costs as LLM usage scales. Both matter, but neither is a reason to stand still.
Staying at the frontier means continually innovating to stay ahead of the curve. At Nooks, we’re building AI agents that handle a wide range of sales tasks and run long stretches of work with minimal human review, powered by advances including:
- Context management and agent harness improvements: filesystems, memory, custom skills
- New paradigms in human-agent interaction, such as “plan mode”
- New foundation models (Opus 4.6, GPT 5.4) that show emergent capabilities like computer use and the ability to create skills for themselves
The hard part isn’t spotting these advances. It’s testing them fast enough to know whether they actually help, not “is this better in theory,” but “will this improve results for our users, in production, right now.” Our upgraded eval framework gets us that answer in days, not weeks. Here’s how we built it.
Our original eval framework
The first two use-cases for our agents were:
- Find me good-fit prospects at an account
- Write me a good cold-outbound email
Why we had to rebuild it
As models and harnesses improved, we saw users push our agents to do more: building slides and deal rooms, digging through data to find what was driving a rep’s performance, orchestrating outreach campaigns to a strategic account that unfolded over weeks or months.
The output wasn’t always a single email or a list of prospects anymore. And even when it was, the “right” email was determined heavily by account context, not a fixed set of rules you could grade the same way every time.
That broke our old approach to evals. We needed two new primitives: a regression suite that measures how the agent got to an answer, learned from decision traces (did it actually pull from prior conversations before drafting that email, or just guess), and a rubric that measures the quality of the answer itself.
We layered these into four levels:
- L1, code assertions: run on all traffic, catch hard invariants
- L2, regression suite: run on every change, prevent known bugs from resurfacing
- L3, online rubric-based evals (LLM-classified): sampled from live traffic, catch intent-specific failures and track overall response quality
- L4, LLM-as-a-judge: run offline, for subjective quality that’s hard to reduce to a rule
The key insight here was the graduation loop: every L1 or L3 failure we caught in production got converted into a permanent L2 test case. The harness doesn’t just catch bugs, it accumulates them, so every failure we’ve ever seen becomes something we can never regress on again.
That solved regressions. The bigger test came when we charged the architecture itself.
Testing new architecture: Filesystem agents
The industry had already converged on a pattern for this: give the agent a filesystem, tools to traverse it, and let it decide what’s relevant, rather than stuffing everything into a single context window. Claude Code, Cursor, Codex, and Warp all work this way. We wanted to bring the same patterns to sales agents: materialize account data, CRM records, call transcripts, and prospect info as files the agent could search and read on its own, instead of our existing approach, an account assistant that worked from a single synthesized context blob.
The problem: we couldn’t just vibe-check whether the filesystem approach was actually better. It produced fundamentally different trace shapes than our existing account assistant, more, smaller reads instead of one big context dump, so there was no obvious way to eyeball the two side by side and know which was working better.
Our eval harness solved that. We ran 205 queries through both architectures, the original single-context account assistant and the new filesystem-based agent, and measured the results directly: the filesystem approach was about 30% faster and more than 50% cheaper on matched queries, with comparable quality on our existing evals.
The evals didn’t just validate the switch; they surfaced problems we never would have found by hand:
- Skills are unreliable for critical instructions. Things like email conciseness or tag formatting still need to live in the system prompt, not a skill file, echoing what Vercel found in their own AGENTS.md findings.
- Explicit file-path guidance matters. Telling the model exactly what files exist and the optimal order to read them in cut down on wasted ls calls.
- Small bugs hide in plain sight. A slightly wrong file path buried in the system prompt only ever surfaced through systematic eval runs, not spot-checking.
The harness that made this comparison possible turned out to be useful for more than architecture decisions. The next test came when a new frontier model shipped and we needed an answer fast.
Testing new frontier models in days, not weeks
GPT 5.4 dropped as a research preview. Three days later, we had the confidence to make it our default.
That confidence didn’t come from a few test conversations. We ran 205 matched items across two agent configurations, Account QA Assistant and Global Nooks Assistant, with 30+ automated checks per trace. The harness measured three dimensions:
- Reasoning-to-action ratio. 5.4 queries more systems and finds data more reliably.
- Token efficiency. A 39% reduction despite deeper reasoning, a counterintuitive result that only showed up because we were benchmarking at the L2 level instead of eyeballing outputs.
- Architectural reliability. Skill-calling compliance jumped from 75% to roughly 100%, the threshold our modular architecture needed to actually work in production.
For the parts that don’t reduce to a number, we ran pairwise LLM-as-a-judge comparisons with calibrated prompts, tuned so the judge wasn’t penalizing intentional formatting differences as if they were errors.
Without the harness, this would have been “it feels smarter”. With it, we had a quantified answer on production-readiness, in days instead of weeks.
The GPT 5.4 test was a snapshot: one model, one point in time, evaluated once. The next challenge is harder, because it’s not a snapshot at all. It’s a moving target that’s different for every user.
Testing user-customizable skills & memory
As we expand into user-defined skills and persistent memory, the eval problem gets harder in a specific way: the agent’s output is no longer just a function of the query. It’s a function of the query and what the agent remembers about the specific rep it’s working with.
The rest of the industry is racing to solve this problem. Letta shipped Skill Learning in December 2025, letting agents learn new skills from their own experience rather than a static prompt. They followed that in February 2026 with Context Repositories, git-backed memory filesystems that let an agent’s memory version, branch, and merge the way code does. Underneath both is a live debate: should context be loaded passively upfront, or fetched actively on demand? Vercel’s own evals found AGENTS.md-style upfront context beat on-demand skill retrieval, the same conclusion we landed on when we found that critical instructions still need to live in the system prompt rather than a skill file.
The debate gets sharper once personalization enters the picture. Testing “does the agent write a good email” is already difficult. Testing “does the agent write a good email given what it remembers about this specific rep, this specific account, and their history together” multiplies the eval surface by every user in the system. There’s no longer one right answer to converge on, as there are as many right answers as there are reps.
That’s a big enough problem that it deserves its own post. More on how we’re solving it soon.
When we step away from any of these three tests, be it the architecture switch, the model upgrade, or the personalization problem ahead of us, and a pattern emerges. None of them were won by a single insight or a single model. They were won by a harness that got better every time we used it.
What makes this work: the compounding harness
Here’s the part that isn’t obvious until you’ve lived it: the harness doesn’t just answer the question in front of you. It gets sharper every time you ask it one.
Every production failure we catch graduates into a regression test. The filesystem migration surfaced skill-reliability issues we’d never have caught by hand; these became permanent checks. The GPT 5.4 evaluation ran against a harness already hardened by that earlier work, which is a real part of why three days was enough. Each cycle leaves the next faster and more precise, because the harness is carrying forward everything the last test taught us.
We’re extending the same approach to LLM-as-a-judge, expanding it beyond raw accuracy into subjective dimensions like personalization quality, citation relevance, and tone - the kind of judgment calls that don’t reduce to a pass/fail check but still need a consistent bar.
The lesson underneath all three data points: the thing that actually drives product quality isn’t any single model; GPT 5.4 will be replaced by GPT 6, Opus 5 will be replaced by whatever comes after it. What compounds instead is iteration speed: how fast you can tell whether a change has actually helped. Evals are the bottleneck that unlocks that speed. Every hour spent building the harness buys back that same hour, many times over, on every model and every architecture change that comes after it.
Closing thoughts
Across this series, the specifics keep changing: a new architecture, a new model, a new way of representing account context. But the pattern underneath is the same one we described in our first post, when we found that sales resists the same shortcuts that make coding agents easy to evaluate. There’s no compiler, no pass/fail in seconds, no single “good” output.
Every gain we’ve made traces back to the same source. The artifact work alone, tested againstg 58,000 replayed production requests, cut costs in half at a given reasoning-effort level, with quality holding steady at the top end and actually improving at lower effort levels. The GPT 5.4 evaluation turned “it feels smarter” into a three-day, production-ready decision. The filesystem migration came in 30% faster and more than 50% cheaper on matched queries. None of these numbers came from intuition. They came from a harness built to catch exactly this kind of result, and to get sharper every time we ran it.
What’s ahead is a harder version of the same problem. As we move into user-defined skills and persistent memory, there’s no longer one right answer to converge on; there are as many right answers as there are reps. Testing “does the agent write a good email” was already hard. Testing “does it write a good email given what it remembers about this specific rep” multiplies the eval surface by every user in the system. We don’t have that fully solved yet. But the harness that got us through architecture changes, model upgrades, and cost cuts is the same one we’re pointing at personalization next.
1. Letta is an AI research lab in San Francisco building machines that learn. They envision a future where humans coexist with digital people: experiential agents that remember everything, learn continuously, and improve themselves over time.
