Solving for cost: cheaper agent context, better results
In my last blog, I discussed why agents struggle within the GTM use case. Sales isn’t like coding: whereas code is observable and has clear pass/fail parameters, sales runs on nuance and hidden information i.e. has a champion actually pitched leadership? Has the budget been earmarked elsewhere? The “correct” answer is much more complex to find. We rebuilt our models and eval sets to accommodate the ambiguity, reason with inconsistent data and help agents arrive at the right answers, ultimately better serving sales reps in their day-to-day workflow. Accuracy came at a price; getting every factual question right proved expensive and slow, especially when account information and context kept changing.
Cost-cutting through token reduction
As exponentially rising LLM usage has made reducing spend top of mind for many other organizations, we explored how to meaningfully reduce cost without sacrificing the quality of our model’s reasoning and output.
The most common LLM cost reduction approaches focus on lowering per-token cost by using open-source models for less complex, high-volume tasks and model routers to match the right model to a given task. This ensures you don’t end up using Claude Fable to write an email or other trivial task. However, with this approach you quickly reach a floor of how much you can save by reducing token costs alone. The question then becomes: how can you reduce the tokens required to complete a task?
Token reduction strategies generally fall into two buckets:
Improving agent harness ergonomics
A well-designed harness does more than sit quietly in the background. Cleaner tool interfaces and better feedback loops mean an agent can act on a prompt more precisely and use its tools more reliably, instead of losing effort to friction it shouldn't have to deal with. Improving agent harness ergonomics also does a lot to reduce token costs. One key way to do this is through context compression. Consider Headroom as an example. Just as a product like Bitly takes a massive, bloated URL and shrinks it down to a tiny slug without breaking the destination, Headroom takes massive, bloated text payloads and shrinks them to a fraction of their size, significantly improving token spend by filtering out verbose filler, repeated boilerplate, overly hedged phrasing, or anything else that consumes context window budget without meaningfully contributing to the task.
Context compression only reduces the cost of a single request; it doesn't change how often an agent has to pay that cost.
Caching so an agent doesn't start from zero
Caching works by reusing something the model has already computed instead of paying to compute it again. At the infrastructure level, this usually takes the shape of prefix caching: if a new request shares the same leading sequence of tokens as a recent one, the model skips reprocessing that shared prefix and picks up from where it left off. Think of it like a browser reloading a page you visited a minute ago. It skips re-fetching everything because it still holds a valid, recent copy. Prefix caching works the same way for an LLM request: it only pays off if the same prefix gets hit again soon, while the cached copy is still fresh.
That's exactly where it breaks down for account context. A sales account isn't touched every few seconds; it's touched sporadically, with a new call, email, or CRM update landing hours or days apart. By the time a request against that account comes around again, the account's true state has already moved, so the cached prefix is stale before it's ever reused. Caching didn't fail here as an idea, it just needed a continuity our access pattern doesn't have.
While both approaches can yield real reductions, neither changes the fact that agents need to re-discover their environment for every request.
Solving for agent amnesia
The LLM at the core of any agent has no persistent memory. Every request starts from a blank slate, so before it can act, the agent first has to reconstruct everything it would otherwise already know. At Nooks, that meant whether drafting an email, prospecting, or helping a rep prepare for a call, the agent had to read an account's entire history first: every call, email, CRM update, and meeting recording, then re-synthesize a coherent understanding of it, spanning different timeframes and stakeholders, before it could even begin answering the actual query. For an enterprise account, that raw context alone can run into the hundreds of thousands of tokens, so even the simplest request carries a large, fixed cost, paid in full every single time.
Our first attempt to fix this was caching: save a synthesized account summary so a future request could reuse it instead of reconsuming the raw context from scratch. But caching only pays off when the same content gets requested again soon, while it's still cached. Sales accounts don't work that way. They get touched sporadically, sometimes days or weeks apart, and by the time a request against an account came around again, its underlying data had usually already moved on. The cached summary was stale before it was ever reused.
The fix wasn't a better cache. It was to stop caching altogether. A cache only exists to make reuse of something recently computed cheap; that's exactly why it's temporary, and why its value disappears the moment access turns sporadic instead of frequent. So instead of trying to keep the synthesized summary alive in a cache, we just stored it: a plain-text brief, written once and persisted the same way any other file would be. Unlike a cache, it costs nothing while it sits unused, there's no eviction clock working against it, and it only costs something again once a future request actually reads it back in.
None of this is a new idea in the abstract. Persistent, incrementally updated memory for agents is an established category now, projects like Letta's memory blocks work on the same principle: replace re-derived context with a stored summary that gets patched over time instead of rebuilt from scratch. What's specific to our case is the shape of the problem that the summary has to survive. A sales account's context is unusually large, pulled from a dozen inconsistent sources, and touched on no predictable schedule: sometimes hours apart, sometimes months. The open question for us wasn't whether persistent memory works in principle. It was whether an artifact patched dozens of times over months still holds up as well as one built fresh at that same point.
Building the artifact
Creating the initial artifact was straightforward: we defined a prompt which outlined how we wanted to represent an account, provided an agent the necessary tools, and then saved the resulting agent output.
Building the artifact costs a lot upfront: it means consuming an account’s entire context once. That cost only pays off if it’s spread across many requests, not just one. So instead of rebuilding it from scratch, we keep it in sync: any new activity on an account, an email, a call, a CRM update, triggers a lightweight updater agent which generates and applies a diff to the artifact with any meaningful changes.
Evaluating artifact architecture against cost & quality
Before scaling the artifact, we needed to know if it was actually buying us anything: lower cost, better quality, or both. To evaluate the system we created a benchmark using 58,000 past requests from our email and dialer agents. These requests were replayed offline using the same production harness both with the artifact and without, essentially reconstructing the account context from raw data each time. Quality was scored by blinded pairwise preference versus the baseline configuration (GPT5.4-high with no artifact).
We evaluated the artifact vs no artifact over a range of reasoning effort, which provided us with two quality and cost curves. As one would expect, lowering the thinking level reduced the number of tokens and overall cost for a given setting while resulting in decreased quality.
What we found surprising was that using the artifact shifted the curve significantly to the left; cutting costs in half for a given effort level. Quality isn't measurably higher at the highest effort levels but does seem to improve more significantly for lower thinking levels, indicating that the agent is able to benefit from the cached work that the artifact brief provides.
Testing artifact durability
While the artifact cures our agent's amnesia, moving from a stateless to a stateful system introduces a new kind of complexity, one that has to be understood before it can be trusted in production. Our benchmark (Figure 1) already showed that a freshly bootstrapped artifact (a new account with no prior artifact to patch) reduces cost without hurting quality. But that benchmark only tested fresh artifacts. In production, most artifacts aren't fresh; they've been patched repeatedly as new activity comes in, and we hadn't yet shown that patching preserves the same quality.
So the real question became: does an artifact that's been updated many times still perform as well as one built from scratch at that same point? If yes, our earlier benchmark results carry over to the system as it actually runs. If no, the artifact could be degrading quietly with every patch, and our benchmark would only describe a best case that doesn't hold in practice.
More formally, we wanted to test if Bootstrap(t) + k updates ≈ Bootstrap(t + k); whether an artifact built fresh at time t and then patched k times (Bootstrap(t) + k updates), produces the same quality as an artifact built fresh directly at that later point (Bootstrap(t + k)). In other words: does patching an artifact k times get you to the same place as just rebuilding it from scratch at that point? We simulated this through our production harness, re-bootstrapping fresh comparison artifacts at intervals of k = 5, 10, 25, and 50 updates.
We replayed real account histories, applying 1, 5, 10, 25, and 50 updates to an existing artifact, then compared each updated version against a fresh bootstrap built from scratch at that same point in time. If patching is working, an artifact patched 25 times should be indistinguishable from one built fresh at 25.
That’s what we found, up to a point. Through 25 updates, patched and freshly bootstrapped artifacts were indistinguishable. At 50, both curves turned: quality started to drift, and the cumulative cost of all those small patches exceeded the cost of rebuilding just from scratch. So that’s the policy we shipped: patch incrementally, and re-bootstrap the artifact from scratch after 25 updates.
That policy answers the quality question: patching doesn't quietly erode the artifact, as long as it's rebuilt every 25 updates. It doesn't yet answer the question the whole post started with, whether any of this actually reduces cost once it's running for real, not just in a benchmark. So we shipped the artifact under this policy and measured what happened.
Evaluating artifact performance in production
Earlier this summer, we tested the artifact with control-vs-shadow pairs: real production traffic ran as-is (control), while a parallel shadow arm handled the same requests with the artifact added, without affecting what the user actually saw. Comparing matched pairs from one hour of traffic (3-4pm PDT, June 24), the shadow arm cut costs 36.9% for the dialer builder-agent and 33.1% for email, with median per-pair savings of 51% and 39% respectively:
.png)
Breaking the builder-agent results down by model tells a more specific story. This particular hour of data skewed toward gpt-mini traffic, so the percentage moves with model mix, over a longer three-hour window with more Claude traffic, the builder-agent number comes in closer to 23%.
What’s next
Looking back, we’ve learned a tremendous amount about the limits of caching synthesized account summaries, about why artifacts succeed where that approach fell short, and how much cost savings depends on agent harness ergonomics generally.
But there’s still a lot more to be done. Staying at the frontier means continuing to focus on context management and agent harness improvements, new paradigms for agent-human interaction, and new foundation models that show emergent capabilities, like the ability to create their own skills. More on that in our next post.
1. Headroom is a context compression tool that strips repeated boilerplate from tool calls, DB rows, file reads, and RAG payloads, reducing costs without sacrificing quality.
