Building an agent orchestrator without a framework

If you build enough LLM-powered chat systems, you notice the same shape appearing. There’s a message coming in. There’s a state machine deciding what to do with it. There’s a fleet of “tools” — arbitrary HTTP endpoints — that might get called. There’s a stream going back to the user. And somewhere, in the middle, the whole thing has to survive things like agent timeouts, dropped SSE connections, and slow background jobs.

We chose to build the orchestrator by hand rather than adopt an off-the-shelf agent framework. This post is why, and how the pieces fit together.

What the orchestrator does

It sits between the chat backend and the reasoning stack. On every user turn:

Terminate the client’s request (FastAPI, StreamingResponse over SSE).
Mint a transaction_id and workflow_id for correlation.
Get-or-create the conversation record.
Call the intent classification service to find out what the user wants.
Route based on the returned intent to a handler: direct answer, call a tool, ask a clarifying question, escalate to a ticketing system, produce a summary, or opt out.
Persist an audit row with intent_classified, final_response_source, fallback_reason, duration_ms, status.
Stream the answer back.

The mental model I keep is: policy-driven router with a fallback tree. Every branch has a well-defined fallback into “answer directly.” No user turn goes unanswered — the worst case is a plain LLM completion.

The state, split by heat

There are really two kinds of state, and they belong in different systems.

Hot state goes in Redis:

session:{id} — start time, message counts (24-hour TTL) for session-duration KPIs.
active_agent:{conversation_id} — the tool currently owning the thread, so follow-ups route straight back without reclassification.
enable_agents:{conversation_id} — feature flags.

Durable state goes in DynamoDB:

Conversations, messages — chat history, partition-keyed by conversation ID.
Agents — the tool catalog: entrypoint URL, transport, method, auth type, HMAC config, required_context, timeout.
Interactions — one row per user turn, the audit log.
Agent capabilities — the “what can this tool do” index the intent service reads.

Repositories implement application-layer Port interfaces so DynamoDB can be swapped for an in-memory implementation in tests. That single decision — abstract-repo-ports plus in-memory doubles — is the reason the test suite runs in seconds instead of minutes.

The tool invocation contract

Tools are modeled as agents, which are — deliberately — just arbitrary HTTP endpoints. A workflow engine, an MCP server, a bespoke microservice, a Lambda: they’re all the same shape from here.

Each agent record carries:

Entrypoint — transport, URL, method, and per-agent timeout_ms (with connect / read / write / pool timeouts configured separately, because a single global timeout lies to you).
RoutingHints — priority, allowed domains.
Auth — the strategy this agent expects.
Security — HMAC config, encryption flag.

The invocation itself:

Build a payload keyed by the agent’s declared required_context — e.g. chatInput, sessionId.
Attach recent conversation history.
Encrypt-then-sign. AES-256-GCM whole-body encryption, HMAC-SHA256 digest, RSA signature over the digest. Two layers because different downstreams check different things, and it costs microseconds.
Fire via an async HTTP client with the per-agent timeout.
Agent returns automation_available. If false, we fall back to a direct LLM answer and record fallback_reason=automation_unavailable. Users get an answer either way.
Long-running work returns out-of-band via a message queue (below). Terminal events (success, error) close the interaction row.

Streaming, and retrying a stream

The direct-answer path buffers a downstream SSE stream, parses data: / event: lines, tracks end_of_response, and — this is the fun part — supports stream retry.

If a stream drops mid-response, we retry up to twice. On retry, before re-sending the prompt, we fetch the last assistant message that partially wrote to storage and use it as a “resume from here” prefix. Most of the time the user never sees the retry. When they do, they see a completion, not a rug-pull.

That single feature came out of watching real logs. Streams die in production. The correct answer is “retry politely,” not “surface a 500 to the user.”

Out-of-band completion via SQS

Tools sometimes take minutes. The synchronous HTTP call to an agent doesn’t wait — it kicks off work and returns. Progress and completion arrive via events published to a message queue.

The orchestrator runs a background SQS poller as a FastAPI lifespan task: long-polls (20s wait, so the queue is quiet when idle), batch-receives (up to 10 messages), routes each event to the right conversation subscriber, updates the interaction row, and terminates polling on success / error.

On the frontend side, an in-memory SubscriptionManager (a defaultdict of user → conversations, asyncio-lock guarded) lets SSE consumers subscribe to conversation event streams. When a long-running tool finishes, its completion event finds its way to the right open SSE connection.

We skipped LangGraph

We looked at LangGraph. It’s a good library. We chose not to adopt it because at the time we were shipping this, we wanted:

Tight SLA control over serialization boundaries. The orchestrator’s SLOs are stricter than the frameworks default to.
No lock-in on prompt formats or persistence.
A codebase readable by engineers who’ve never seen a graph library.

But this is a LangGraph in miniature. If I had to describe our orchestrator in framework terms:

State = the DTO plus conversation history plus the Redis session hash.
Nodes = handler methods (_handle_answer_direct, _handle_call_tool, _handle_ask_clarify, etc.).
Edges = the intent-router switch on IntentType, with explicit fallback edges (agent-not-found or automation-unavailable → direct answer).
Checkpointer = DynamoDB conversation + messages tables.
Streaming = FastAPI StreamingResponse re-emitting parsed SSE events.
Interrupts = the ticketing-escalation and clarify paths — the run pauses waiting for the user or a human.

If I rebuilt from scratch today, I’d probably use LangGraph. Not because we made a wrong call — the hand-rolled version has been reliable — but because the ecosystem has moved. The interrupt() primitive alone would clean up the clarify/ticket paths meaningfully.

Observability that survives contact with production

Two rules that emerged from real incidents:

Monitoring code cannot crash the request. Every metric emission is wrapped in a safe_capture_event. If Datadog or Amplitude is having a bad day, users never see it.
transaction_id on every log line. Generated at ingress, echoed back in the response header, threaded through every downstream call, logged via extra=. If you can’t grep for a single transaction_id across every service touched by one request, you don’t have distributed tracing — you have distributed hope.

The metrics themselves are unglamorous: KPI counters (requests, resolution rate), custom spans (prompt_orchestration), latency distributions, LLM-specific spans. But because every one of them is wrapped, they can all be ripped out or replaced without touching business code.

What I’d change

Adopt LangGraph now — see above. The interrupt() model is cleaner than what we hand-rolled.
Move SSE parsing to a real streaming library. We roll our own data: / event: parser. It works. It also has three subtle bugs we’ve fixed over the years, all of which would have been someone else’s problem in a library.
Turn the audit log into a full event stream. Right now it’s a DynamoDB row per turn. A properly-shaped event log would let us reconstruct any conversation for evaluation.

The underrated part

The most valuable thing about this whole design isn’t the orchestration — it’s the fallback tree. Every branch has a “if this fails, do X instead” edge, and X is almost always “just answer the question directly with the LLM.” That means the platform degrades gracefully: agent broken? Direct answer. Ticketing down? Direct answer. Notification service returning 503? Direct answer.

Users can always get something back. That’s not a UX detail. It’s the whole product.