Long, reliable processes: why we orchestrate document flows with a workflow engine

When a document arrives at Loony, our document intelligence platform, it is not processed in a single step. It is split into sections, its text is read, OCR is applied when needed, it is classified, its data is extracted, it is added to a cabinet (archivero) and a review of the complete file is triggered. Some steps take seconds; others, minutes; others wait for a person to upload a document or review a result, and that can take days. In this post we explain why we orchestrate all of this with a workflow engine rather than with hand-built queues and retries.

The problem with hand-built queues and retries

The most common way to process documents in the background is well known: a service receives the file, publishes a message to a queue, a worker process picks it up and keeps a status updated in the database: queued, processing, processed, error. We have built systems like that ourselves, and they work well as long as the process is short and linear.

Problems appear when the process grows:

• State ends up scattered. Part of it lives in the queue, part in the database and part in the worker's memory. If the worker restarts halfway through, reconstructing which step it was on requires extra code.
• Retries are dangerous. Running a step again can mean paying twice for OCR or extraction, or writing the same result twice.
• Timeouts become handcrafted. Each step needs its own time limit and its own way of detecting that something got stuck.
• Nobody quite knows where each case is. Answering "where is this file?" means going through logs from several services.
• Every new flow reinvents the previous one. What was solved for one process has to be solved again for the next.

What a workflow engine gives us

In a workflow engine, the flow is described as a sequence of steps, and each step that touches the outside world (storage, OCR, a language model, a database) is a separate unit. The engine keeps the history of every execution. If a worker stops, another one resumes the flow exactly where it left off, without losing what was already done.

On that foundation we solve uniformly what used to be solved case by case:

1. Retries with a per-step policy. Network errors are retried with growing backoff; permanent errors, such as a missing file or an invalid configuration, fail immediately. Required steps stop the execution, and optional ones are recorded without stopping it.
2. Idempotency. Steps that cost money, such as OCR or extraction, are designed not to be resubmitted on a retry, and writes are identified by their destination: if the result already exists, the step does not repeat it.
3. Timeouts and heartbeats. Long tasks report their progress periodically; if they stop doing so, the engine knows and acts.
4. Capacity without blind retries. When an OCR or model provider has no room, work is not fired off only to fail and be retried: it waits its turn in an ordered, visible line and runs only when there is space.
5. Visibility. Every execution has its history, and each section of a large document appears separately, so it can be re-run on its own. In the interface, a document shows whether it is processing, under review or in line, and at what position.

Human steps inside the same flow

What we value most about this model is that a human wait is just another step in the process, not an exception living outside the system.

• Reviewing results. The review of a file produces a verdict: approved, rejected or, when the result falls in an intermediate band, needs review. In that case a person accepts or rejects it, from the interface or through the API, and the decision is recorded with date and author.
• Requested documents. A cabinet can ask for a document that was not planned. That request stays pending, waiting on a person, not on a server; when the document arrives, the completeness review updates by itself.
• Simultaneous uploads. If ten documents are uploaded at once to the same cabinet, ten versions are created, but the file review does not run ten times: it finishes the current pass and runs one more on the latest version.

For the engine, waiting a week for a person costs nothing and is not lost if the servers restart.

Decisions that keep it sustainable

A workflow engine does not organize the system by itself. These rules are what, in our experience, make it manageable:

• Few flows, a lot of configuration. We have two generic flows, one for documents and one for cabinets, that interpret a versioned configuration. A new capability is a new catalog entry, not a new flow.
• References in the flow, content in storage. The engine only sees identifiers and references, never the text of the documents. That preserves confidentiality and keeps the history light.
• Configuration is pinned at the start. Each execution records the versions it used and does not re-read them halfway through.
• Results are never overwritten. Reprocessing adds a new version next to the previous one; nothing is deleted.

A workflow engine does not remove the complexity of a long process; it concentrates it in one place, where it can be seen, tested and audited.

It also has costs: workflow code must be deterministic, the team needs to learn a different model and there is one more piece of infrastructure to operate. For processes that last minutes or days, involve people and must leave evidence, we find it a clearly favorable trade.

Do you have long document processes that need to be reliable? Write to us.