Refactoring Legacy Python for Prod Reliability

Source: Dev.to | Jan 30, 2026 (UTC)

1. Problem

The nightly ingestion pipeline – a grab‑bag of legacy Python modules – ships as a monolith: main.py imports everything, re‑exports some globals and runs a handful of workers that talk to the database. The real source of chaos is twofold.

1. Leaky Imports – most modules import the top‑level package, which pulls in heavy, partially initialised data structures. When the orchestrator boots a new worker, that import chain runs again, forcing the runtime to recreate large in‑memory objects.
2. Bubbling Exceptions – every worker hides its own exception handling behind a try/except SystemExit: pass guard. If a critical error occurs, the process dies silently, the orchestrator restarts the worker and a surge of database writes storms in. The monitoring stack sees a blip and flags a prod outage.

These incidents mean the 4 pm delivery pipeline stalls, rollback chains flip and confidence in the automated playbook erodes.

2. Solution

The refactor is scoped around three pillars.

1. Lazy Imports via importlib – replace heavyweight import package with explicit, module‑level imports that pull in only the names required.
2. Dependency Injection for Globals – remove the global event‑loop and database handle; instead pass them into worker constructors. This enables mocking and unit tests.
3. Structured Error Propagation – rather than swallowing exceptions, log them, raise a custom WorkerError and let the orchestrator decide on retries.

The orchestrator evolves into a lightweight dispatcher that keeps a pool of long‑running worker objects. Each worker receives the database session it needs, performs its task and sits idle for the next batch. The new design eliminates repeated stack‑heavy imports, reduces memory churn and ensures that a single worker crash does not cascade into a full‑system restart.

During the refactor we also:

• Refactored the Christmas XML schema parser to use a streaming approach with xml.etree.ElementTree.iterparse. This cuts memory usage from 300 MB to 60 MB per job.
• Re‑placed global signal handling with context managers that clean up temporary files reliably.
• Added application metrics (latency, success rate) to a Prometheus exporter for live health checks.

Local testing on a 10 k‑record dataset shows a 70 % reduction in peak memory and a 4× faster restart cycle.

3. Code Block

python

before: global DB connection and shallow imports

from legacy import perform_task, db_session

def worker(event):
try:
perform_task(db_session, event)
except Exception:
pass # swallow – bad practice

after: explicit dependencies and proper error handling

import importlib
from typing import Callable

class Worker:
def init(self, db_conn, task_func: Callable):
self.db = db_conn
self.task = task_func

def run(self, event):
try:
self.task(self.db, event)
except Exception as exc:
log.error("Worker failed", exc_info=exc)
raise WorkerError("Task failed", exc) from exc


Enter fullscreen mode

Exit fullscreen mode