musings
All posts
architecturellmagentspythonfastapi

Designing an intent classification service for an LLM assistant

The front-of-funnel decision — direct answer, clarify, or call a tool — and why building it as a small, dedicated service pays off.

·6 min read

Every chat-first LLM assistant needs to answer one question before it does anything else: what should we actually do with this message? The user typed something. Do we generate a response? Ask them to clarify? Hand it off to a tool that can act on their behalf? That decision, per turn, is what an intent classification service exists to make.

I’ve spent the last while building one. This post is about how it’s structured, why it lives as a standalone service, and the handful of decisions I’d keep even if I had to rebuild it from scratch.

The three routes

The output is small on purpose. Given a raw user prompt plus context (conversation history, session metadata, user profile), the service returns exactly one of:

  • ANSWER_DIRECT — general knowledge, a code or text generation task, or “no registered tool matches this well enough.” The upstream chat surface answers with a plain LLM completion, optionally with retrieval.
  • ASK_CLARIFY — the intent is ambiguous, or a required parameter is missing. The response ships a clarifying question the frontend can render.
  • CALL_TOOL — the request maps to a registered agent. The response names the agent and returns the parameters extracted from context.

That’s it. The intent service deliberately does not execute tools, run retrieval, or produce the final answer. It’s the router. Everything else is downstream.

Why a separate service

The temptation is to fold this into whatever chat backend you already have. Don’t. Isolating routing gives you three things:

  1. A cheap, cache-friendly hot path. Every turn hits it. When it’s small and stateless-ish, you can scale it independently and reason about latency in isolation.
  2. A place for the agent registry to live. Agents are metadata: capabilities, required parameters, priorities, active flags. That registry needs to feed the classifier’s system prompt on every call. If it lives inside a giant chat backend it gets tangled with other state.
  3. A place to iterate on prompts. Prompt-versioning, evaluation harnesses, safety guards — all of it lands in one repo with a narrow surface area. When someone breaks it, you know who to talk to.

The layers

I default to Clean / Hexagonal Architecture for services like this — not because it’s fashionable, but because the LLM at the center of the flow is the single largest source of churn. You’ll swap providers. You’ll add caching. You’ll add a fallback classifier. The domain — “there is an intent, there is an agent, there is a conversation” — barely moves. Keeping infrastructure out of the domain means the churn is confined:

  • API layer (FastAPI). A single POST /decide route, middleware for timing and structured logs, a lifespan handler that fails fast if secrets can’t be fetched, and global exception mappers.
  • Application layer. Use cases (DecideIntentUseCase is the orchestrator), Pydantic DTOs, prompt builders, response parsers, and ports — abstract interfaces for every repo, the LLM client, the cache. This layer never imports from infrastructure/.
  • Domain layer. Pure entities and value objects: IntentDecision, Agent, Conversation, Message, RoutingRule. No I/O.
  • Infrastructure layer. Concrete adapters: LLM SDK clients, DynamoDB, Redis, secrets manager, monitoring. Ports bind to implementations through FastAPI’s dependency injection.

The request flow reads like the routing decision itself:

transaction_id ← header or generated
→ validate DTO
→ load active agents from registry
→ load-or-create conversation, pull recent history
→ filter agents by allowed capabilities for this channel
→ load channel-scoped routing rules + information sources
→ build a dynamic system prompt
→ call the LLM with a structured-output schema
→ parse into a domain object
→ (optional) second-pass source picker, in parallel with a category classifier
→ return typed response

Structured outputs, always

The single biggest quality improvement was moving from “ask the model for JSON and hope” to structured outputs — a Pydantic-derived JSON schema handed to the LLM via response_format. The response now parses deterministically, which means the retry / fallback code stops running.

There’s still a defensive fallback: if the model returns text with a JSON block inside, we extract the fenced block and try to parse. It runs maybe once a week. That’s the correct rate.

Prompt-versioning in the database

The system prompt is assembled per request from the live agent registry, routing rules, and information-source catalogue. Version numbers live in a table. Editing a prompt or adding an agent doesn’t require a deploy.

This sounds obvious once you say it, but the first version I built hardcoded prompts in Python. Every routing change was a PR. The moment you accept that agent metadata is prompt content, the shape changes.

Token budgeting is not optional

tiktoken counts tokens. A sliding-window truncator preserves the system prompt, reserves a response buffer, and drops the oldest history first to fit the target model’s context window.

You can skip this at first. It bites when you least want it to. A user drops in a long paste, the model returns a truncated response mid-JSON, your parser dies. Handle it once, up front.

Cost/latency levers

A few tricks that punch above their weight:

  • Cheaper model for secondary passes. The information-source picker runs on a smaller model at low temperature and a small max_tokens. The main classifier gets the good model.
  • Run secondary classifiers in parallel. Where a category classifier and the main classifier are both needed, they launch under a single asyncio.gather — the second call adds no wall-clock time.
  • Loop-breaker on repeated clarifications. If the model asks the same clarifying question N times in a row, override the route to CALL_TOOL with best-effort parameters. Users don’t want to be trapped in a clarification loop, and the model, left to itself, will happily trap them.

Auth: two modes running side by side

The service takes two kinds of callers: other services and humans. Both auth modes are available simultaneously through a chained “either wins” dependency:

  • HMAC (SHA-256 over method + path + body + timestamp + nonce, with a configurable clock-skew window). For service-to-service.
  • Hashed API key looked up against a secrets manager. For humans and lower-trust automations.

The keys are hashed, not compared against a static env value. That means rotating a key is a database write, not a redeploy.

Nonce replay protection rides on Redis SET NX EX — a nonce can be spent exactly once within its TTL; a repeat returns 409 Conflict. It’s four lines of code and it kills an entire category of attack.

PII-aware logging

Long user-provided strings are truncated in error payloads. Prompts are not logged in production. Nonces are partially masked. This isn’t glamorous but it’s the difference between “we had a bug” and “we had a bug and now Legal wants a meeting.”

What I’d change

Two things I’d do differently on version two:

  1. Streaming end-to-end. Currently the classifier waits for the full LLM response before parsing. For most turns that’s fine (a hundreds-of-milliseconds decision). But the moment you want to show the user anything mid-classification, this needs SSE.
  2. A first-class evaluation harness. Right now the integration tests double as evaluation. They should be separate — the eval set should live with the prompts and be part of every prompt-version review.

Wrapping up

The thing that keeps surprising me about this service is how small it is. Under 3k lines of Python. And yet it changes the ergonomics of the whole platform: adding a tool becomes a database write plus a prompt tweak, not a code change. That’s the whole reason to build it.