Cracking the hard problem: sales agents that actually work
.png)
Like other GTM tools in the category, we initially built Nooks within the AI-as-copilot paradigm: AI that could augment parts of human workflows but did not own outcomes. We saw it as a starting point, not an endpoint. The models just weren't reliable enough yet to build much more.
By mid-2025, frontier models (Opus 4, OpenAI's o3, Gemini 2.5 Pro) had crossed a real threshold: reliable enough to carry a multi-step task for hours without needing constant human intervention. That was finally enough to build something closer to what we'd wanted all along. We shipped AI Assistant, an agent embedded in almost every surface of our product, capable of acting on its own instead of just assisting. We hill-climbed a rigorous evaluation benchmark built from a diverse set of common requests, tested it with the Nooks sales team and early customers, and got an overwhelmingly positive response.
When we dug into usage data post-launch, we saw a repeated pattern: reps were spending most of their time on edge cases the agent couldn't handle:
I see so much conflicting information cited. What technology does this company really use?
Just because the decision maker said now's not the right time doesn't mean we give up. What's the next best action for me to take on this account?
This email isn't my style at all. Can you make it better?
Reps had adopted AI as a reasoning aid, and loved it. But it turns out the bar for saving meaningful amounts of time and for a rep to trust AI to act without checking its work is much higher than the bar for a good copilot.
The three quotes above point to three hard problems that current agents don't handle well, and that off-the-shelf prompt engineering and evals don't solve:
- Fact resolution
- Strategy selection
- Style and taste
Each one is hard for the same underlying reason: sales isn't like code. Code is fully observable; you can read every line. Sales runs on hidden information; you can't see whether a champion has actually pitched leadership, or whether budget's already been earmarked elsewhere. Code gives you a pass/fail in seconds; sales gives you an answer in weeks, if at all. And code has one kind of output, text that either compiles or doesn't, while sales spans calls, emails, LinkedIn messages, slide decks, and gifts, each with its own notion of "good."
That's why the playbooks that make coding agents work (standard benchmarks, editable plans, engineered calibration, prompt-level style rules) kept failing when we applied them to sales. Rather than launch AI Assistant to all our customers with those problems unsolved, we spent the next few months building sales-specific answers to each one.
Here's how we got there and what we learned.
Building evals for sales
Fact resolution
We were hitting 97% accuracy on our eval set for factual sales questions, with zero hallucinations. But qualitatively, sales reps felt it was wrong “half the time” and always double-checked the output. Why the difference?
It turned out our benchmarks didn't capture what happens in production:
Unlabelled false negatives (the classic retrieval problem). The right answer might live in a LinkedIn conversation, one of many recorded calls, a note from two years ago, or a Slack thread. When we revisited our eval set, we realized that instances where AI Assistant didn’t know the answer weren’t because it wasn’t findable; it just required far more effort than the average data labeller might spend.
Conflicting information sources. If the call from two years ago disagrees with the note from two months ago, which is right? What if the note has typos and looks like it was written in a rush? What if the call is with someone who has since left the company? Information recency, relevancy, and source all need to be taken into account.
Human nuance. It’s common for reps to hear “We don’t have the budget” or “We already have a tool in place”; it’s up to them to interpret if this is a brush-off or a hard constraint. The difference is often found in tone and context, which can be tricky to model for.
Additionally, accuracy alone wasn't the problem: the model needed to know when it was guessing. We started by parsing through customers’ data traces to identify examples of three kinds of inconsistencies:
- Temporal drift: answers that change as information arrives
- Data source disagreement: two sources covering the same fact contradict each other
- Model disagreement: the agent reaches two different conclusions from the same evidence
We gave human labellers these hard cases along with the conflicting reasoning traces behind each answer i.e. why the model landed on A instead of B, so they could judge the model's reasoning directly instead of re-guessing from scratch, which made our eval-set ground truth far more reliable.
Better ground truth solved half the problem; the other half was knowing when the model itself wasn't sure. We had it output a confidence score for every fact and tried three methods of calibration: logprob-based methods, ensemble disagreement, and separate verifier models. All three lost to simply asking for a confidence score with a carefully worded prompt. That surprised us at the time, though it shouldn't have. Tian et al. (2023) had already shown that verbalized confidence from models fine-tuned with human feedback is better calibrated than the model's own conditional probabilities, cutting expected calibration error by roughly half. What wasn't obvious was whether it would hold on multi-source, temporally inconsistent sales data, where the model is reconciling contradictory evidence rather than answering trivia. It did. We've since moved to better calibration methods
Once we had harder test cases and calibrated confidence, we could optimize for two things at once: is the answer right, and does the model know when it's guessing?
We surfaced that confidence directly to reps: a flagged answer means double-check it, an unflagged one means trust it. Reps no longer felt they had to double-check everything, and they could focus on strategy rather than playing fact-checker.
We called this eval set SalesBench, and the 97% accuracy mentioned earlier was the original SalesBench score. We then built a subset, SalesBench Verified, specifically from adversarial, multi-source, and temporally ambiguous examples from customer data.
Our original system scored below 50% on Verified; after the recalibration, it reached 72–79%:
We had successfully built a model that reasoned with inconsistent data and arrived at the right answer, but this came at a price. Getting every factual question right proved to be expensive and slow, especially when the information kept changing. More to come on this.
Strategy selection
We knew strategy would be one of the hardest parts of building sales agents. Every company and every rep has a different strategy: a cadence that lands for a fast-moving SMB account reads as pushy at an enterprise account, where the same rep might need to wait weeks between touches. You can't create a global eval set for that, and without knowing the right strategy, simply executing actions is counterproductive.
We thought we could borrow some concepts from coding agents to solve this: where the agent asks questions in advance and creates editable plans a.k.a “plan mode”:
The analogy didn't transfer. Coding is intrinsically rule-based, whereas sales relies on intuitive reasoning. Reps found it difficult to articulate exactly why they would pick one strategy over another, and when editing plans, they made inconsistent choices. For example, when faced with the same situation, rep A would choose to forego a recommended action entirely while rep B chose to prioritize it. The reps were making judgement calls based on intuitive reasoning, making it difficult to extract the highest-order bits (the specific signal that actually drove each decision).
We solved this by creating a contextual learning loop based on a new filesystem architecture: letting the agent search globally across all accounts and learn what worked for a few similar ones. This yielded much better results than asking sales reps to describe their optimal strategy.
But that alone wasn't enough. In practice, there is a huge breadth of strategies to try: multithreading accounts, engaging on social media, personalized gifting, etc. Left unconstrained, the agent would try all of them regardless of fit; to combat this, we built a separate “strategy evaluator” tool, which estimates the chance a strategy has to close the deal for a given sales team by comparing it to how that kind of move played out for like-for-like accounts historically. When we let our agent try a range of strategies, then ran the evaluator to prune the ones that didn't work, the recommended strategies were grounded without sacrificing creativity.
Style and taste
Nooks users were already using AI as a copilot to write emails, texts, LinkedIn messages, and generate call scripts. The bar for an end-to-end agent is much higher: users need to trust the AI to execute autonomously without reviewing every artifact.
Awkward paragraph structure, mentioning irrelevant information, being too wordy — all forgivable in a copilot — but as the agent took on more of the work on its own, they became blockers. Beyond outright mistakes, style varied so much between users that it was hard to learn even by sampling their previous emails.
How did we solve this? First, we curated a high-signal negative set: 500+ examples of bad AI messaging, along with a clear rubric explaining and grading why each was bad. We used feedback signals from across multiple customers and design partners, including repeated situations where users wrote a different email than the AI generated, and situations where users gave repeated negative feedback directly via chat.
We found that prompt engineering or in-context learning against these negative examples wasn't very effective and could degrade overall performance (the “don't think of an elephant” problem: telling a model what not to do keeps that exact thing in its context). But using each example as a real-time correction guardrail — the “decision-time guidance” paradigm that Replit popularized — dramatically reduced those failure modes.
The negative set solved the floor: it caught what would read as bad output to any user. The ceiling was harder to define; there’s no single ‘good’ to aim for, since every user writes differently. To handle this, we built per-user contextual memory. After a chat session where an email was edited heavily, we'd prompt them with an AI-generated skill describing when and how to generalize those edits for future emails. Surprisingly, users found it intuitive to read and edit the skills (basic prompt engineering is much easier when you aren't starting from scratch), and once approved, the skills decreased email revision rates:
The result: We're now at 96% of emails sent autonomously without edits. Every time we increase the complexity of our strategies — LinkedIn DMs, personalized gifting campaigns — there's a new class of stylistic mistakes waiting to be found.
What's next
Looking back, when we built our first sales agents, we were naive. We saw how effective coding agents were at dramatically speeding up the time to take apps from zero to one, and assumed we'd see similar gains in sales right away.
But sales differs from engineering in several ways:
It's not a fully observable system. You can't see whether a champion has actually pitched leadership, or whether budget's already been earmarked elsewhere.
Delayed feedback loops make it hard to evaluate success. A code change gets a pass/fail in seconds; a sales strategy might take weeks to know if it worked.
A wide action space that includes calls, slide decks and collateral, in-person events, emails, LinkedIn messages, gifts, and ads, each with its own stylistic nuances.
By working each part of the problem rigorously, we've doubled meetings booked per sales rep for our own team within the last three months, and achieved comparable results with early beta customers across a range of sales motions, from high-volume SMB outbound to complex enterprise deals.
We have a lot more work to do. What does it take to get from 2x productivity gains to 10x? What language-model limitations and bottlenecks will we hit as we try to get there? We're hiring engineers, product managers, and designers who want to work on problems like these, where the hard part isn't optimizing a metric, but figuring out which metrics are actually worth optimizing.
This is the first post in a series on building sales agents. Up next: how we tamed the cost of fact resolution, and the backtesting system behind our strategy evaluator.
Want to keep up with what we’re building? Follow us on X. We’re hiring across engineering, product, and design at Nooks.
[1] Tian et al. (2023) were the first to show how for models fine-tuned with human feedback, verbalized confidence is typically better calibrated than the model's own conditional probabilities, cutting expected calibration error by roughly half on TriviaQA, SciQ and TruthfulQA. Their explanation is that RLHF degrades the log probabilities themselves. The RL objective pushes probability mass toward the preferred answer rather than matching how often each answer is actually right, so the proxy you'd normally reach for is the thing that got broken.
[2] Our new architecture materializes account data like CRM records, call transcripts, notes, prospect info, as files the agent can search and traverse with tools, instead of stuffing everything into a single context window. It follows the same filesystem-as-context pattern used by coding agents like Claude Code, Cursor, and Codex.
[3] Building this strategy evaluator required extensive backtesting on historical data. We'll share more on how we built that backtesting system in a later article.
[4] Replit describes this technique in "Decision-Time Guidance: Keeping Replit Agent Reliable" (Li et al., Replit, January 2026): a lightweight classifier watches the agent's trajectory and injects short, situational instructions only when they're relevant, instead of loading every rule into the system prompt upfront.
.png)
