Calling an LLM API is the easy part. The hard part is batching thousands of jobs within rate limits and surviving a worker crash. You also have to recover when a key runs out of quota and prove the output matches the schema.
Forge handles that layer. It runs LLM batch and realtime jobs across OpenAI and Anthropic with two Node daemons. Anvil builds batches and Hammer executes them. They share nothing but a Postgres database, and every coordination problem is solved inside it.
Postgres as the coordinator
The system turns on one handoff table. An upstream producer seeds pending rows. Anvil claims them and groups them into batches each provider accepts. Hammer moves those batches through its worker loops and the provider endpoints. Each job ends as done or failed.
The usual choice is a queue such as Redis, SQS or RabbitMQ. A queue holds a second copy of the job state, and two copies drift apart. Postgres already holds the state, so it coordinates too.
Claiming work safely
Several copies of Anvil can run at once without two of them grabbing the same row. Opening a batch is idempotent, so running it twice cannot create a duplicate. Workers claim batches by advancing their status.
So every daemon and worker is safe to run as several copies. Two Anvil replicas skip past each other's rows. A worker that crashes after claiming leaves a row another worker reclaims.
Leases instead of timestamps
Crash recovery uses an explicit lease column instead of a last-updated timestamp. A row untouched for five minutes is ambiguous, since the worker may be slow or dead. An expired lease is clear. Long realtime executions renew their lease while they run, and terminal states clear it. A reaper worker reclaims anything with a lapsed lease and either releases it for retry or fails it as an orphan.
Side effects that survive a crash
When a job completes, the database changes state and the outside world hears about it through a tag or a notification. If these are separate operations, a crash between them leaves the two out of step.
Forge writes the side effect in the same database write as the state change. A dedicated worker delivers it afterwards. A crash can delay a notification but never lose one.
Batch and realtime in one model
Batch jobs go through submitting, polling and collecting. Realtime jobs skip the provider batch path. Both use the same tables and leases.
Provider ports
OpenAI and Anthropic plug in through a provider registry that resolves per batch. OpenAI wants JSONL uploads and Anthropic takes inline batches. Those quirks live in adapters, so the core never depends on either provider.
A key that runs out of quota pauses. A recovery worker probes it with tiny requests that cost nothing and unpauses it on refill. Prometheus metrics track the loop.
Validation before acceptance
Every provider success line is checked against the processor's response schema before Forge accepts it. A transient error gets a retry. A persistent mismatch ends as failed instead of storing data nobody validated.
Prompts are versioned as production, canary or draft, previewed, then promoted.
Lessons learned
- State with one home made every coordination feature simpler.
- If a step cannot safely run twice, the claiming rules are wrong.
- A lease records who owns a job and until when, so crash recovery becomes a simple check.
- Provider responses are external input, so I validate them before they become my data.
This design does not fit every scale. LLM jobs are heavy, provider limits cap throughput, and correctness matters more than microseconds. For that work, Postgres alone removes a whole layer of infrastructure and its failure modes.
References and related work
- PostgreSQL explicit locking: the row-locking primitives behind worker coordination.
The Forge case study covers the daemons, the operator viewer and my role.