Building an agent swarm for engineering

This post is a bit different from our usual ones, because it isn't about sales or about building Nooks. It's about the work that happens around the code: the routine jobs that keep an engineering team's tickets moving from one week to the next. Over the past few months we've handed most of that work to a small group of agents, and we wanted to write down how they fit together.

None of these agents write product code, and if you looked at any one of them on its own you probably wouldn't be impressed. Each does a single job that used to be somebody's chore. What turned out to be interesting is everything around them: how they pass work to one another, how each model call is set up, and where we decided a person still needs to make the call.

The jobs

When we looked at where an engineering team's non-coding time actually goes, most of it fell into four buckets.

The first is intake. Someone posts a bug in Slack, and it has to become a ticket with the right product area, a sensible priority and an owner who isn't on vacation. Doing that by hand takes whoever is on call a few minutes per report, and it's easy to file things in the wrong place when you're doing it between other work.

The second is hygiene. Tickets need consistent labels, mostly for product area and customer, so that we can count and filter them later. Nobody enjoys labeling, so it tends to slip, and once it slips every number built on top of those labels quietly becomes wrong.

The third is follow-through. A ticket goes past its SLA and nobody notices, or a feature ships and nobody announces it, or it gets announced but nobody updates the help center or the readiness checklist. These are all things everyone agrees should happen, but nobody in particular owns them.

The fourth is reporting. Customers want to know where their issues stand, leadership wants to know whether bugs are trending up or down, and the sales team wants to know what shipped this week. Each of those used to be a document someone pieced together by hand from the ticket tracker.

We built at least one agent for each of these jobs:

• Triage bot (intake, small model): Reads each new post in the bugs channel, files a ticket with an area, priority and owner, and replies in the thread
• Product-area labeler (hygiene, small model): Adds a product-area label to new tickets
• Customer labeler (hygiene, small model): Tags each ticket with the customer who reported it
• SLA nudger (follow-through, no model): Reminds the owner in the ticket's Slack thread once it's past due
• Release nudger (follow-through, mid-tier model): Asks the owner of a shipped ticket to announce it, if it's worth announcing
• Release reviewer (follow-through, mid-tier model): Reads each release announcement and files follow-ups for any rollout steps it missed
• Customer docs (reporting, large model): Updates a weekly "state of your issues" page for each key customer
• Bug health (reporting, no model): Publishes daily dashboard metrics on open and new bugs by priority and area
• Enablement deck (reporting, no model): Turns the week's release posts into slides for the sales team

How they stitch together

There's no orchestrator in this system. No agent calls another one, and none of them even knows the others exist. Instead, they coordinate through the same places the team already works, which are the ticket tracker and Slack.

Most of the handoffs happen in the tracker. The triage bot files a ticket, the labelers come along afterwards and add an area and a customer, and the reporting agents then query by those labels. When the customer docs agent wants to know which tickets belong to a customer, it doesn't ask the customer labeler anything; it just reads the label, the same way a person would.

That arrangement turns the labels into a kind of contract. Every report downstream groups tickets by area or by customer, so a wrong label doesn't stay hidden in the tracker. It ends up as a wrong number in a document that someone outside the team will read. That's the main reason the labelers got more tuning than anything else in the system.

Slack is where the context lives. Every ticket links back to the thread where the bug was first reported, and the nudgers post their reminders in that same thread, so the whole conversation stays in one place. The release reviewer works the same way in the releases channel, replying underneath each announcement, and the enablement deck builds its slides from that channel too.

The harness

The bulk of our effort didn't go into the agents themselves so much as into the harness around each model call, meaning what the model gets to see, what it's allowed to do, and what happens to its answer afterwards.

One call, no tools

When people hear "a swarm of agents," they often picture a set of models running in loops and calling tools. Ours, with one exception, are close to the opposite. Each agent makes a single model call per item, and the model has no tools at all. It's asked one question, such as "which product area does this belong to?" or "is this release worth announcing?", and it answers with either a short JSON object or a single label name. Ordinary code handles everything else, from fetching the data to creating the ticket, applying the labels, choosing an owner and posting to Slack.

We set it up this way deliberately. Each of these jobs needs judgment at exactly one point, and everything around that point is plumbing that ought to behave identically every time. Keeping the model away from the side effects means every action can be tested, logged and dry-run like any other code. The worst a bad answer can do is put the wrong label on a ticket; it can't take an action we didn't write code for.

Context is where the work is

Since the model has no tools, the prompt is the only thing it knows about the world, so that's where most of our care went. A few examples give a sense of what that looks like.

• The triage bot sees the bug report alongside the list of product areas, each with a short description. Those descriptions are read live from the tracker, so when someone on the team edits a label's description, the bot picks up the change on its next run. Each area also carries a few "not this" clauses, which we wrote after auditing tickets the bot had misrouted. Most of those misroutes had landed in a plausible neighbouring area, and spelling out the boundaries is what fixed them. The prompt also includes a handful of routing rules and a list of strategic accounts, and all of that configuration lives in the repo so changes get reviewed like code.
• The customer labeler gets the ticket and the current customer list from the CRM, along with a single firm instruction not to guess. If a customer's name or domain doesn't actually appear in the ticket, the right answer is "none."
• The release nudger gets the shipped ticket, the last three weeks of release posts (so it can tell whether the work has already been announced) and a closed list of the kinds of bug fix that are worth announcing. That closed list came from experience. When we gave the model open-ended criteria instead, it could argue that almost any fix was "user-visible." The prompt also includes real past decisions from the team, covering both what they chose to announce and what they didn't.
• The release reviewer gets the announcement, the replies underneath it and a description of what the readiness checklist and onboarding content currently cover. It also waits until a post has been up long enough to collect replies, so that it doesn't ask about something a teammate has already answered in the thread.

Don't trust the answer

Every answer gets checked against reality before code acts on it. If the triage bot names a product area that doesn't exist, the area is treated as unknown. If it cites a routing rule we never wrote, the rule is ignored. If the customer labeler returns a name that doesn't exactly match someone on the CRM list, the answer is thrown away.

We also log the different ways a model can fail to give a useful answer as separate cases. From the outside, "the model said it didn't know" looks exactly like "the model answered, but nothing matched its answer," unless you deliberately record them apart. Doing that once surfaced a bug that had been hiding for weeks, where the product-area labeler was only being shown a fraction of the labels it was meant to choose from.

The smallest model that works

Labeling and triage run on a small model, since there's a lot of volume and each question is narrow. The release agents run on a mid-tier model, because deciding whether something is worth announcing takes real judgment. Three of the agents don't use a model at all: the SLA nudger compares dates, bug health is a set of counts, and the enablement deck is a parser feeding a slide template. Our rule of thumb is that if plain code can do the job, we don't reach for a model.

The exception: customer docs

The weekly customer docs are the one place where we run a genuine agent loop. Here a large model is connected to both the ticket tracker and our docs tool. It begins by reading last week's page for the customer, which gives it a snapshot to compare against as well as the format it should follow. From there it works out which issues have moved, closed or appeared since then, and writes the new update.

This agent also carries the strictest rule in the whole system, which is never to fill a gap with invented content. That rule got its first real test on the first fully unattended run, when the agent lost access to the tracker partway through. Rather than improvise, it wrote nothing at all, which was exactly the right outcome. Since then we've run the docs in a session with an engineer watching, and that has turned out to suit a document that customers read anyway.

Where a person stays in the loop

This is the part we spent the most time thinking about. The rule we eventually settled on is that an agent can act on its own when a mistake is cheap to fix and nobody outside the team will see it. Once an answer costs someone's attention, or is going to be read by someone outside the team, a person needs to be involved somewhere.

In practice that gives us roughly four levels of autonomy.

Fully automatic. The labelers write labels directly, with nobody approving them. A wrong label is internal and takes one click to fix, and asking someone to approve hundreds of labels a week would simply bring back the chore we were trying to remove. The SLA nudger and the bug-health metrics are fully automatic as well, though neither of them uses a model.

Automatic, but it steps back when unsure. The triage bot files tickets straight away, without an approval step. When it isn't confident about where a bug belongs, though, it doesn't guess an owner. It leaves the ticket unassigned, mentions the other area it was considering, and lets the on-call engineer decide. Clear human signals also take priority over the model: if the reporter @-mentions an owner, that person gets the ticket, and anyone marked as out of office is skipped.

The agent asks, and a person acts. The release nudger never writes an announcement itself; it only asks the owner to write one. The release reviewer, when it thinks something is missing from an announcement, files a follow-up ticket for the author, who is free to close it. This works in the other direction too, since replying "file follow-up" to an announcement makes the reviewer file one even if it thought the post was fine.

A person reviews before anything is sent. The customer docs go to customers, so an engineer is in the session while they're being written.

We also learned to treat people's attention with the same care as accuracy. When the team told us the release nudger was too chatty, our first instinct was to make its prompt stricter, and that barely changed anything. What actually worked was giving it a budget of at most three nudges per run, ranked by impact, with anything over the limit waiting for the next run.

Safety around the edges

A few rules apply to every agent that takes actions:

• Dry-run by default. The triage bot won't post anything unless dry-run mode has been explicitly switched off, and it also sits behind a feature flag. The other agents have the same dry-run switch, so their first runs only log what they would have done.
• Do everything exactly once. Before acting, the triage bot and the release reviewer each write a claim record that fails if one already exists. The release nudger instead leaves a short marker phrase in every message it posts, and checks for that phrase before posting again.
• Retry once, then hand it to a person. If a triage run dies partway through, a sweeper picks it up and tries again once. If that also fails, it posts in the alerts channel asking someone to triage the bug by hand.
• Post as the on-call engineer. The release agents post through the account of whoever is on call that week. If they can't work out who that is, they stop rather than fall back to posting as an anonymous bot.

What this adds up to

Looked at one by one, none of these agents is especially clever. Most of them make a single model call, and three of them don't make any. What made the system useful was choosing the right jobs to hand over, letting the tracker and Slack carry the handoffs, putting real effort into what each model gets to see, and being deliberate about where a person stays in the loop.