Building an agent swarm for engineering

This post is a little different; it's not about AI in sales or building Nooks at all. It’s an inventory of the swarm of sub-agents we built to run a specific ticketing process for one eng team, and what we learned along the way.

None of these agents write product code. Rather, they automate the process metabolism of an eng team: getting bugs from Slack into the ticket tracker correctly classified and routed, keeping tickets labeled so dashboards emerged from, nudging humans at the exact moment something goes stale, and turning ticket churn into readable weekly narratives for customers and leadership. Each agent is small, single-purpose, and observable. That’s a deliberate choice. A swarm, not a monolith.
‍

The dependency graph

Here’s the shape of the whole system. The labelers are the roots. Every reporting agent downstream is only as good as the labels underneath it. Weekly customer docs, the bug-health dashboard, the state-of-bugs reports, none of them can be trusted if the ticket underneath is filed under the wrong customer or the wrong area. That's where we put the most effort, and it's the right lens for walking through the swarm itself: roots first, then the nudges that keep humans honest, then the reporting that turns it all into something readable.

The roots: getting the ticket right

• Bug triage bot. Live, triggered on every message in the bugs channel. One structured LLM call classifies each report into area, priority, reason, and summary against the product-area taxonomy. The prompt includes a description of each area plus explicit exclusion rules, refined across two rounds of auditing misrouted tickets. It resolves an owner from an eng-ownership doc, skips anyone OOO, honors explicit @-mentions, files the ticket with area and customer labels, and replies in-thread with the triage proposal.
• Request auto-routing. Live, and absorbed into the triage bot. We originally built this as a standalone router, then folded it in once it became clear the two problems were the sam. One agent, not two. 
• Product-area labeler.  Live, daily cron, small model. Infers the product-area label for new tickets from title and description, with label descriptions sourced from the eng-ownership doc and fuzzy matching to handle inconsistent label formatting.
• Customer labeler. Live, runs every 5 minutes, small model. Reconciles the customer label set against the live CRM list, so a new customer gets a label within minutes, and infers which customer reported each fresh ticket from title, description, and attachments.

Everything downstream reads these labels and assumes they’re right. The assumption is only safe because these four agents run first, run often, and were tuned against real misroutes instead of shipped once and left alone. 

The nudges: keeping humans honest

• SLA breach nudger. Live, runs daily. No LLM, just rules: find every ticket past its SLA date and post into its linked thread. Assigned tickets get “please leave an update,” unassigned tickets get “could someone pick this up?” This is deliberate. The swarm uses models only where classification is genuinely fuzzy, not where a date comparison isn’t. 
• Release-post nudger. Built, dry-run validated, rolling out. For each completed ticket, it decides whether the release is worth announcing and whether it’s already been announced. Unworthy or already-covered tickets get skipped; everything else gets exactly one nudge, deduped by a marker string. 
• Release-post reviewer.  Built, dry-run validated, rolling out, runs after the nudger. Reviews new release announcements and flags the rollout surfaces that historically get forgotten: : internal readiness dashboards, in-product onboarding content, customer-facing docs. It threads a reply tagging the author with whatever looks missing. 

Both release agents post as the current on-call human, resolved from the rotation via their own OAuth token, instead of as a bot. Nudges from a person get answered. Nudges from a bot get muted.

The narrative: turning tickets into something readable

• Weekly enterprise-customer docs. Running, agent-driven, weekly. A living “state of your issues” doc per key enterprise customer, built by diffing customer-labeled tickets against last week’s snapshot. Each doc preserves its own evolved format rather than being regenerated from scratch. This only works because the customer labeler exists; the labels make the underlying queries possible. 
• Weekly state of bugs and feature requests. Live, a weekly doc plus a daily metrics cron. The narrative half reports landed and open bugs and features for the team. The quantitative half emits bug-health gauges, open and new by priority and area, normalized by active work volume, to a dashboard. This runs after labelers, on purpose, so the by-area numbers are trustworthy by the time anyone reads them.
• Enablement deck generator (WIP). Written, not yet deployed. Reads the release-announcements channel weekly and builds a slide deck, one slide per release with owner, links, and screenshots, for GTM enablement.

None of these three agents does any classification on its own. They’re pure consumers of labels and announcements produced upstream. That’s the payoff of getting the roots right: the reporting layer gets to be simple. 

What holds it together 

Ten agents sharing a Slack workspace and a ticket tracker only works because of stringent conventions we committed to during the build.

• Idempotency via marker strings. Every nudging agent stamps a marker into its output and checks for that marker before posting again; this acts as a dedupe mechanism. No state store, and it survives identity rotation, since the marker lives in the message, not in whichever account posted it.
• Double-gated rollout. Every agent ships behind a feature flag and a dry-run environment variable. It runs read-only against production first, and only flips to live once the dry run looks right. Nothing goes live in one step.
• Post as a human, not a bot. We resolve the current on-call and post through their token rather than a service account. People answer people; they mute bots.
• A backfill runner and a simulator for every classifier. Before deploying a prompt change, we replay history through it and compare. This is how the triage bot’s per-area NOT-clauses got tuned across two misroute audits instead of one round of guessing. 
• Observability inside the classifier. We split "the model abstained" from "the model answered but matched nothing.” Without that split, an LLM cron job is a black box: a silent failure and a confident wrong answer look identical from the outside.
• Cheapest model that works, or none. Small models handle high-volume labeling. Mid-tier models handle judgmernt calls like release-worthiness. Plain rules handle date math. The SLA nudger doesn’t touch an LLM at all, because it doesn’t need to.
• Sequenced crons. Metrics run after labelers. The reviewer runs after the nudger. It’s an implicit DAG, expressed entirely in cron offsets rather than a workflow engine, and it works because the dependency graph above is small enough to hold in your head.

What this adds up to

None of these ten agents are sophisticated on their own: the SLA nudger is a date comparison, the customer labeler is a lookup against a CRM list, and so on. What makes the swarm work is the system they sit within, rather than the build of a single agent.

‍