No Queue Until It Is Needed: A YOLO Vision Platform That Grew by Evidence

This post is based on the architecture case study, a documentation-only, generalized and anonymized release of an internal computer-vision platform. It names no organization, application field, site, hardware or measured result, and it contains no code.

The platform around the model

An earlier post covered how to evaluate whether YOLO with tiled inference (SAHI) detects small objects in 4K drone imagery. That is the part that usually gets told. The other part is the platform around it: how a small group of operators and researchers request a training run or process a batch of images without logging into a GPU machine, and how anyone can tell afterwards which model produced which result.

Two processes and one boundary

The first decision was to separate. A Django web layer serves people: forms, validation, administration, results. A separate FastAPI service is the only thing that touches the GPU: it trains, validates and infers. They talk over HTTP with JSON, share a storage volume where artifacts land (models, manifests, outputs), and the database belongs to the web layer only; the AI service has none of its own.

The reason is not aesthetic. A training run can hold the GPU and its memory for hours and has no business inside the process that answers a browser. The two sides also need different environments: multi-GPU training is very sensitive to the combination of operating system, drivers and CUDA version, which is why that service was pinned to Ubuntu, while the web layer can live wherever is convenient.

The uncomfortable decision

The first iteration had no task queue. When someone requested a training run, Django called the AI service and waited, connection open, until it finished. No Celery, no Redis, no message broker.

Put like that it sounds careless, and the case study does not hide it. With a request open for hours, the AI service becomes a bottleneck, a dropped connection loses the work and, if the service restarts, nothing records what was in flight. It was chosen anyway, for concrete reasons: it was the simplest thing to build, the easiest to debug (one request, one response, one visible error) and it let the platform be validated before investing in infrastructure that might never be needed.

Writing the limit down with its signal

What turns that decision into engineering rather than debt is how it was written down. Every limitation was documented together with the signal that would make it unacceptable: requests timing out, operators needing to see progress, failures that cannot be explained afterwards. And the roadmap ordered the responses: status records and polling first; a controlled worker only when jobs compete for the GPU; a broker only when retries, cancellation or several workers are needed.

When the signals arrived

They did. A batch of hundreds of high-resolution images outlived any reasonable timeout. Operators asked whether a job was still running, and the only way to know was to look at the GPU. And a restart left half-written outputs, with no record of what had been in flight.

The revision implemented the first step of the roadmap and stopped there. Submitting a job now returns an identifier immediately, and the job runs on a thread pool inside the same service. Every run leaves a durable record, written atomically, that the console polls every few seconds. On startup, the service reconciles whatever was left half done. There is still no broker: it was not needed yet.

The other responses followed the same pattern, each one to a limit that had been written down:

• a transactional model registry, with explicit promotion by a person, instead of a reference in a file;
• a single mount path in both containers, checked by a test, instead of translating paths across four coordinate systems;
• a key between the services, so nobody else could request GPU work;
• a test suite that runs without a GPU in continuous integration.

What stayed open

Not everything was resolved. The experiment-tracking tool was chosen early for the convenience of its cloud version, which meant data left the organization by default. It was withdrawn, a self-hosted alternative was decided, and that alternative is still not deployed. The case study records that just like everything else.

Three lessons

• Separate the GPU from the web from day one. It is the cheapest decision to make early and the most expensive one to undo later.
• A limit without its signal is debt; with its signal, it is a plan. Writing down when a decision stops working is what lets you make it without guilt.
• Build the first step, not the staircase. The revision needed status records and polling, not a distributed queue, and knew to stop there.

I designed and documented this architecture. The decisions (ADRs), limits, roadmap and the later revision are in the case study repository.