For the past ~2 years I've volunteered to build websites for non-profits, using slack as the medium for communication. However, I've gotten a lot busier, and requests often come in sporadic bursts. While not particularly hard to address, I just don't have the time to extend to each of these tasks anymore.
So I built Caleb JR, a multi agent pipeline composed of 4 different agent calls, regulated by a few human bottle necks. It is optimized for minimal human effort and concurrently; maximum leverage across the flow.
On a high level, this is an automation chaining several agents together: context is gathered via Vercel CRON, triaged by a Claude instance, then pushed to a cloud agent which opens a PR. All of this is automated.
Context is gathered via a Vercel CRON which fires once a day. It pulls all new slack messages in channels the agent is added in. Inbound messages are then triaged via an instance of Claude Haiku and evaluated as "actionable" according to a confidence score dependant on:
1.
Ignores messages not aimed at me (random chatter ect)
2.
Tasks are then cross checked with pre existing tickets to prevent redundancies.
3.
If at this point the agent is confused, it pings for clarification
When I initially set my automation up, I just threw whatever model I was using onto the line. Part of creating good scalable automations is being highly intentional with each of the parts. So I did some research to try and determine if I was really getting the most “bang for my buck”. Also, in general I figured this would be a good exercise for understanding model benchmarks in general.
Note!! These benchmarks should not be used for general evaluations, and were evaluated (through subjective multipliers) to my very specific use cases! (ie: I can't afford to run expensive models, so model cost was factored generally higher ect..)
Triage was evaluated on BFCL (Berkeley Function Call Leaderboard), a public standard benchmark for function/tool calling from natural language — which is basically the job of the triage layer of the agent.
This was evaluated on 3 factors:
Speed (latency), measured by how quickly the model triaged decisions from BFCL
Cost, measured by the cost to run the entire BFCL
Triage, a weighted composite of irrelevance and live / non-live accuracy
Model | Triage | CPP ($/pt) | Cost $ | Latency |
|---|---|---|---|---|
Command A Reasoning (Cohere) | 84.2 | 0.036 | 3.04 | 3.4s |
GLM-4.6 thinking (Zhipu) | 84.5 | 0.055 | 4.64 | 4.3s |
Claude Haiku 4.5chosen | 83.6 | 0.170 | 14.23 | 1.7s |
Gemini 2.5 Flash (Google) | 85.3 | 0.309 | 26.36 | 3.0s |
Claude Sonnet 4.5 | 85.6 | 0.511 | 43.73 | 4.3s |
Claude Opus 4.5 | 84.4 | 1.026 | 86.55 | 4.4s |
Accuracy is composed of an irrelevance factor and the ability to parse structured prompts and messy non-structured prompts. Since we run a Vercel CRON and intake ~100–200 messages a day, and the workspace is primarily gen-z, it's imperative that chosen models are able to parse messy workspace chatter.
I weighed models according to Triage ability, which took into account model accuracy and ability to tool call successfully from both structured prompts and unstructured prompts (chatter). I gated the table to a triage benchmark of 83 — anything lower than that wouldn't be considered.
While Command A and GLM-4.6 feature higher triage scores and cheaper CPP, their latency was more than double Claude Haiku's, which rounded out to be the fastest model of the showing, as well as the 3rd lowest CPP (cost per performance). Not that sweating cost is that big of an issue since I'm operating on a relatively low intake scale, but latency and speed were much higher of a priority for me. The triage gate needs to quickly communicate to an impatient prompter what it's doing, and so speed being the most important factor steered me to use Haiku.
Other features that help with re-seeding context, communicating what the bot is doing to the user, and generally making it's uses cases more robust.
Responds when pinged
/commands to display tasks and to rescan tasks.
each ticket has a ticket number which can be used to append context via /agent call.
Responds to correspondents when the agent has finished task with description
Uses emojis to indicate ticket status (read, finished)