Bridging a COBOL Production System Without Touching It

This post is based on the Legacy Production System Integration case study — a documentation-only release of a proprietary industrial platform. It describes architecture, patterns and trade-offs; no source code, client data or business rules are disclosed.

The system that could not be replaced

The plant ran on a COBOL application that had been doing its job for decades: indexed files, scheduled batch executions, and a nightly export of flat files into a shared network directory. Nobody was going to switch it off. It held the operational truth — products, components, technical specifications, work orders — and every shift depended on it.

What it could not do was answer questions. The exports were fragmented: one file for products, another for operations, a third for specifications, with identifiers that referred to each other only by convention. There was no centralized reporting, synchronization was manual, and traceability stopped at the file boundary. Every new application that wanted production data had to reinvent the parsing.

The brief was therefore not "replace the legacy system" but something more interesting: build a bridge that the legacy system does not know exists. It keeps exporting files exactly as before; the platform on the other side turns them into a relational model that modern tools can use.

A modular monolith, on purpose

The platform is a single Django backend over PostgreSQL, organized into modules with clear responsibilities: ingestion, transformation and synchronization, an API layer, reporting, and — downstream of all that — production scheduling. A modular monolith was a deliberate choice over microservices: one deployment, one database, one transaction boundary, and an internal structure that leaves room to split things later if the load ever justifies it. For an integration that lives or dies on data consistency, keeping everything inside one atomic transaction was worth more than independent scalability.

Long-running work — batch imports, daily metric calculations, large synchronizations — runs outside the request cycle on Celery workers with a message broker, with retry logic for the transient failures that file-based integrations produce: a network share that is briefly unavailable, a file still being written, a temporary database lock.

The ingestion pipeline: detect, parse, validate, transform, persist

Each import follows the same five stages, and the boundaries between them are where most of the engineering went.

• Detection. New files are found by scheduled scans, by a manual trigger, or through an API call. The same code path serves all three, so an operator re-running an import by hand gets exactly the behaviour of the nightly job.
• Parsing. Legacy exports do not come with a schema. Encoding is detected rather than assumed, records are split by delimiter, and field formats are normalized before anything else looks at them.
• Validation. Required fields, data types, uniqueness constraints, and — critically — whether the entities a record refers to actually exist. A record that references an unknown product is a question for a human, not something to persist and hope.
• Transformation. Legacy identifiers are mapped onto relational entities. This is where fragmented files become a domain: products with technical specifications, specifications with components, work orders linked to both.
• Persistence. Everything lands inside transaction.atomic, using get_or_create and update_or_create so that processing the same file twice produces the same database, not duplicated rows.

That last property — idempotence — is the one that matters most in practice. Legacy exports get re-sent. Jobs get retried. Someone restores a backup and replays a week of files. If ingestion is not idempotent, every one of those ordinary events corrupts the data; if it is, they are non-events.

Scheduling on top of trusted data

Once the relational model was reliable, a second capability became possible: production planning. A scheduling module assigns work orders to machines and operators, computes task durations, manages time-slot availability, and enforces the sequential process constraints of the plant — the same kind of precedence constraints that make job-shop scheduling hard in theory, now with real availability windows that split and merge as assignments are made.

The important architectural point is the dependency direction. Scheduling consumes synchronized data; it is a downstream capability, not the system's primary function. Getting that order right kept the integration layer simple and let the planning features evolve without touching the ingestion contract.

What it cost, honestly

The case study lists the risks as plainly as the strengths, and they are the ones you would expect from this shape of system: integration logic somewhat coupled to the persistence layer, heavy database load during large batch imports, concurrency risk when several processes touch the same records, and a hard dependency on an external system's file exports — if the legacy batch fails at 2 a.m., the platform has nothing to ingest. The evolution path is equally unsurprising: event-driven ingestion, incremental synchronization, better observability, and eventually a dedicated scheduling service.

Three lessons

• Modernize the data, not the system. The legacy application never changed. The value came from giving its output a normalized, queryable home.
• Idempotence is the feature. In file-based integration, "what happens when this runs twice" is the first question, not the last.
• Put the transaction boundary where the business invariant is. A product, its specification and its components are one fact; they are persisted as one unit or not at all.

Architecture, data flow, domain model and known risks are documented in the public case study repository.