NGL Models Migration
Migrating R models to Python on Databricks.
I learned an unfamiliar legacy Natural Gas Liquids daily supply modeling system, architected its full replacement pipeline, and created the final code deployed to production. The migration moved the system from R on a legacy server to Python and PySpark on Databricks, with clearer documentation, better reliability, and observable daily runs.
01 / The Challenge
The existing system ran on a legacy server and depended on R code with little documentation and a structure that was difficult to read, understand, and safely modify.
The rebuild needed to reproduce the established daily modeling process while moving it into the company’s centralized cloud environment.
The migration was also an opportunity to make daily execution easier to monitor, with logs and visible success and failure tracking.
System view
02 / The Approach
- Learned and reverse-engineered the unfamiliar R modeling system, then architected the full replacement pipeline in Python and PySpark.
- Used Anthropic models as the primary code-generation tools while creating the final production implementation and owning the pipeline design and deployment.
- Created thorough documentation and made the modeling code substantially easier to read and understand.
- Deployed the completed pipeline to production in Databricks, with Airflow orchestration for the daily process.
- Added execution logging so each run can be tracked and failures can be located and investigated.
03 / The Impact
- Retired the product’s dependency on a legacy server and moved it into the centralized Databricks environment.
- Made the codebase substantially easier to understand, maintain, and modify.
- Improved reliability and operational visibility through Airflow run tracking and detailed logs.
- Preserved comparable processing time while meaningfully improving the system around the models.