Legacy Production System Integration and Data Normalization Platform

This project focused on the design and implementation of a backend integration platform responsible for bridging a legacy industrial production system built in COBOL with a modern relational data environment. The primary objective was to enable reliable synchronization of operational data generated by legacy batch processes into a structured, normalized domain model capable of supporting reporting, traceability, and long-term system evolution.

The legacy system produced large volumes of operational data through scheduled batch executions, exporting information into flat and indexed files stored on shared network locations. These files contained fragmented records representing products, technical specifications, production operations, and related master data without consistent referential relationships. To address this, a robust ingestion and transformation pipeline was developed to detect new files, validate structural integrity, normalize field formats, and map legacy identifiers into relational entities using the Django ORM and PostgreSQL as the primary persistence layer.

A key architectural responsibility of the platform was ensuring data integrity and consistency across asynchronous workflows. The ingestion process incorporated transactional safeguards, idempotent record creation strategies, and controlled synchronization logic to prevent duplication and maintain referential relationships between products, components, technical sheets, and supporting catalogs. Background processing was implemented using Celery workers coordinated through a message broker, enabling scheduled imports, periodic data reconciliation, and operational metric calculations without blocking application workflows.

Significant emphasis was placed on domain normalization and schema evolution. Legacy attributes originally stored as raw text fields were progressively migrated into structured relational catalogs using incremental migration strategies designed to avoid service interruption or data loss. This approach followed industry-standard patterns for safe schema evolution, including staged transitions, controlled data migration, and validation of referential integrity before deprecating legacy fields.

The system also introduced centralized catalog management and automated master data synchronization, reducing reliance on manual data entry and improving consistency across operational records. Logging, validation, and audit mechanisms were incorporated to support traceability of data imports and to provide visibility into batch processing outcomes, enabling operational teams to detect anomalies and maintain confidence in the integrity of production data.

From an architectural perspective, the platform was designed as a modular backend service capable of supporting future system modernization initiatives. Conceptual extensions were explored for advanced production scheduling and maintenance management modules, including resource allocation logic and availability modeling; however, these components remained at the design and prototyping stage and were not deployed in the production environment.

Overall, this project represents a practical implementation of legacy system modernization through controlled data integration, emphasizing reliability, transactional consistency, and maintainable system evolution within an industrial production context.

Technology Stack:

Python, Django | Django REST Framework | PostgreSQL | Celery | RabbitMQ | Redis | CSV Processing | ETL Pipelines | ACUCOBOL-GT | Windows Task Scheduler | REST API | Data Normalization | Transaction Management | Background Jobs | File-Based Integration

About Team