Durable log plus event bus: a notification service for AI agents

Every agent platform I’ve built eventually collides with the same problem. Tools finish. Workflows produce events. The system needs to tell people — through Slack, through a ticketing system, through email, through a mobile push — and it needs to do it reliably even when the receivers are having a bad day.

The lazy version of this is: let each agent talk to Slack directly. Don’t do that. You end up with N agents having M integrations, each service holding its own credentials, and no single place to answer the question “did we actually notify the user?”

The pattern I’ve settled on is one service that plays two roles at once: it is the durable audit log, and it is the event bus. Downstream consumers subscribe. They own delivery. This post is how it’s built.

The shape

Agents, orchestrators, and internal automations POST workflow events (progress, success, error, input_required) to a single endpoint. The service authenticates the sender, durably records the event, and fans it out. Consumers — Slack renderer, ticketing sync, email, audit — each own a subscription and their own retry logic.

Deliberately not in scope: delivery UI, channel-specific rendering, retry-for-Slack-specific-quirks. Those live in the consumers, one per delivery channel.

Persistence before fanout

The invariant that matters more than any other: the DynamoDB write completes before the event bus publish begins. A downstream outage should become a replay problem, not a data-loss problem.

The use case is simple to draw:

authenticate → (optionally decrypt) → persist to DynamoDB → publish to EventBridge

But the ordering is doing real work. If the bus publish fails, the event is still on disk. A backfill job replays it. If the DynamoDB write fails, the client sees a 5xx and retries — no partial state. If both succeed, we win.

Downstream, EventBridge fans out to per-consumer SQS queues. Each consumer runs at its own pace. The rendering service for Slack is doing something different from the ticketing sync, and they should not share failure modes.

Local dev without EventBridge

The single IS_LOCAL=true branch in the publisher swaps EventBridge for direct SQS in a LocalStack container. Same code path everywhere else. New engineers can docker compose up and see events flow. Nothing to configure, nothing to mock.

I keep this pattern in every service now. Environment-shaped conditionals in the outermost infrastructure adapter and nowhere else.

Two auth modes, side by side

Two callers, two auth strategies, both live simultaneously through a chained dependency:

• HMAC-SHA256 for service-to-service — canonical request signature (method + path + body + timestamp + nonce), replay-protected by a Redis nonce cache within a timestamp window.
• Hashed API key for humans and lower-trust automations — validated against a secrets manager, not compared against a static env value.

The API-key hash pattern is worth naming. It means key rotation is a database write, not a redeploy. When someone’s key leaks, you rotate. You don’t fill in a JIRA ticket.

Optional payload encryption

TLS handles the wire. But for events that carry sensitive payloads through multiple hops (bus → queue → consumer → downstream), TLS-per-hop is not the same as end-to-end. We optionally wrap payloads in AES-256-GCM with a shared key.

The hard rule that goes with encryption: plaintext payloads never touch logs. Only IDs, byte lengths, and structural fields. This is easy to write down and hard to enforce, so we put it behind a wrapper — the encrypt/decrypt module is the only code that sees plaintext, and its logging is id and nbytes and nothing else.

Job-ID backfill

Small trick that saves debugging hours: if inbound job_id is null, populate it from event_id. Every downstream consumer sees a stable correlation key, no matter which upstream emitted the event.

This kind of thing sounds fussy. Then you’re two hours into an incident triangulating across three services with three different ID schemes, and it stops sounding fussy.

Attachments: presigned URLs as zero-credential file transport

The tricky case is files. An agent produces a PDF. That PDF needs to reach Slack. It needs to reach a ticket. It needs to reach an archive. It’s produced by a workflow tool that shouldn’t have cloud credentials.

The pattern: expose a POST /attachments/presign endpoint that mints time-boxed S3 PUT URLs. The workflow tool PUTs the bytes directly to S3 with no long-lived credentials. The resulting s3_key rides in the event’s meta. When a downstream consumer needs the file, it mints its own time-boxed GET URL and renders.

Two nice properties fall out:

1. The event bus doesn’t carry file bytes. Events stay small. Fan-out stays cheap.
2. Every hop has short-lived, scoped credentials. No API key floating around a workflow tool.

What I’d change

• A pluggable persistence layer. DynamoDB has been fine. But at some point the log becomes valuable enough to be its own product, and it wants a proper event-sourced store.
• A schema registry. Right now event shapes are documented in the code. A registry with versions would make cross-team coordination less painful.
• Better replay tooling. We can replay from the DynamoDB log, but the tooling is a script, not a UI. On the day we needed it most, we didn’t wish for more capacity — we wished for better ergonomics.

Why this shape

The whole design starts to feel obvious once you accept a small idea: the notification service is not a delivery pipe. It’s a durable log with a dispatcher on top. Everything else follows from that.

Delivery lives in the consumers. Rendering lives in the consumers. Retry lives in the consumers. The notification service does the minimum job that no one else can: guarantee that when someone said “this happened,” we recorded it, and every downstream that cares got told.