July 12, 2026
Creating a multi agent pipeline on slack
The problem
For the past ~2 years I've volunteered to build websites for non-profits, using Slack as the medium for communication. Since then, I've gotten a lot busier, and just don't have capacity to tend to all the incoming requests.
The solution
Inspired by the fleet of autonomous coding agents (Devin, Claude tag etc…), Caleb JR is a multi agent pipeline that operates at a fraction of the price with similar output capacity.
Self-serve
It's completely self-serve on Slack, and conversational (DeepSeek brain).
Cheap per run
~$0.10 per run, without compromising on code execution (Opus 5.5).
Always on
Cloud agent: operates independently of my laptop/phone/anything being on.
When testing Devin, my average run cost about $0.50–$2 per session and screenshot capture. My automation averages less than $0.10 per run, and mirrors similar capabilities.
This is primarily achieved through a combination of deterministic checks, cheap model triages, and using Claude cloud sessions (which consume usage limits, not pay per use) for the heavy coding. For those curious, I have meticulously documented how I built Caleb JR below.
How the automation works
On a high level, this is an automation which chains several different agent calls together to ultimately deploy web changes entirely through Slack.
1
Context is gathered by either Vercel CRONs or ping invocation. A cheap Haiku model with a fallback DeepSeek v4 brain work together to triage tasks, which can be approved / disapproved by a human handler.
2
Approved tasks are turned into GitHub issues, which trigger a Claude cloud session with an Opus agent.
3
Caleb JR renders the changes occurring in the cloud session in real time through Slack's Block Kit, and once finished executing — appends both a Vercel preview link, and a screenshot of the change.
4
Human handlers can then merge directly from Slack without ever needing to touch GitHub (they still need to make an account and be added to the organization for safety reasons etc.)
Design decisions and intricacies
In this section, I'll document the granularities behind every consideration I made while building Caleb JR. I did not follow any template, did not get inspo from any github repo -- and for the most part just built, tested, and then iterated on this flow based on my own experiences across doing web maintenance for clients.
Passively it uses a CRON job to make calls to slack's API to ingest messages every day. This is manageable at scale because there are usually less than 200 new messages to intake per day. This is further managed by only reading channels Caleb JR is in.
Messages are then triaged and either sorted into actionable tasks or not.
Alternatively, it can respond directly to ping invocation. (i.e. if you ping Caleb JR, it will read whatever you say)
The context of the message is triaged by Claude Haiku, using a basic scoring algorithm which assigns points on a 1 point scale.
The scale is scored depending on assumed relevancy to web topics, if it matches with an existing repo, is it a code change that can be made and aligns with base prompt, is it already changed, does it exist in another working session, is it currently being PR'd.
Anything above a 0.8 confidence score gets turned into a ticket.
Anything between 0.5 and 0.8 triggers Caleb JR to switch models from Haiku to DeepSeek and activate its “brain”.
The brain is a smarter model which can respond more robustly and dynamically to unclear or poor prompts. i.e. if someone asks Caleb JR to change “that” or fix “this page” (which is vague and unactionable), Caleb JR will ask for more context until the confidence score re-evaluates to 0.8 through clarification.
Model evaluation
When I initially set my automation up, I just threw whatever model I was using onto the line. Part of creating good scalable automations is being highly intentional with each of the parts. So I did some research to try and determine if I was really getting the most “bang for my buck”. Also, in general I figured this would be a good exercise for understanding model benchmarks in general.
Note!! These benchmarks should not be used for general evaluations, and were evaluated (through subjective multipliers) to my very specific use cases! (ie: I can't afford to run expensive models, so model cost was factored generally higher ect..)
Triage was evaluated on BFCL (Berkeley Function Call Leaderboard), a public standard benchmark for function/tool calling from natural language — which is basically the job of the triage layer of the agent.
This was evaluated on 3 factors:
Speed (latency), measured by how quickly the model triaged decisions from BFCL
Cost, measured by the cost to run the entire BFCL
Triage, a weighted composite of irrelevance and live / non-live accuracy
Command A Reasoning (Cohere)
Triage 84.2
CPP ($/pt) 0.036
Cost $ 3.04
Latency 3.4s
GLM-4.6 thinking (Zhipu)
Triage 84.5
CPP ($/pt) 0.055
Cost $ 4.64
Latency 4.3s
Claude Haiku 4.5
chosen
Triage 83.6
CPP ($/pt) 0.170
Cost $ 14.23
Latency 1.7s
Gemini 2.5 Flash (Google)
Triage 85.3
CPP ($/pt) 0.309
Cost $ 26.36
Latency 3.0s
Claude Sonnet 4.5
Triage 85.6
CPP ($/pt) 0.511
Cost $ 43.73
Latency 4.3s
Claude Opus 4.5
Triage 84.4
CPP ($/pt) 1.026
Cost $ 86.55
Latency 4.4s
Model | Triage | CPP ($/pt) | Cost $ | Latency |
|---|---|---|---|---|
Command A Reasoning (Cohere) | 84.2 | 0.036 | 3.04 | 3.4s |
GLM-4.6 thinking (Zhipu) | 84.5 | 0.055 | 4.64 | 4.3s |
Claude Haiku 4.5chosen | 83.6 | 0.170 | 14.23 | 1.7s |
Gemini 2.5 Flash (Google) | 85.3 | 0.309 | 26.36 | 3.0s |
Claude Sonnet 4.5 | 85.6 | 0.511 | 43.73 | 4.3s |
Claude Opus 4.5 | 84.4 | 1.026 | 86.55 | 4.4s |
Accuracy is composed of an irrelevance factor and the ability to parse structured prompts and messy non-structured prompts. Since we run a Vercel CRON and intake ~100–200 messages a day, and the workspace is primarily gen-z, it's imperative that chosen models are able to parse messy workspace chatter.
I weighed models according to Triage ability, which took into account model accuracy and ability to tool call successfully from both structured prompts and unstructured prompts (chatter). I gated the table to a triage benchmark of 83 — anything lower than that wouldn't be considered.
While Command A and GLM-4.6 feature higher triage scores and cheaper CPP, their latency was more than double Claude Haiku's, which rounded out to be the fastest model of the showing, as well as the 3rd lowest CPP (cost per performance). Not that sweating cost is that big of an issue since I'm operating on a relatively low intake scale, but latency and speed were much higher of a priority for me. The triage gate needs to quickly communicate to an impatient prompter what it's doing, and so speed being the most important factor steered me to use Haiku.
Features
Other features that help with re-seeding context, communicating what the bot is doing to the user, and generally making it's uses cases more robust.
Responds when pinged
/commands to display tasks and to rescan tasks.
each ticket has a ticket number which can be used to append context via /agent call.
Responds to correspondents when the agent has finished task with description
Uses emojis to indicate ticket status (read, finished)