Enterprises relying on legacy ETL tools and tightly coupled on‑prem pipelines face challenges in scale, cost, observability, and onboarding new vendor feeds. With rapidly growing data volumes and increased business demand for faster insights, modernizing the data ingestion and transformation stack became a strategic priority. This Blog outlines how a large enterprise modernized its legacy Informatica-based pipelines using a serverless, metadata-driven architecture on AWS—significantly improving scalability, governance, and operational efficiency.

• Legacy Informatica system with ~1100 workflows tightly coupled with Oracle driver tables.
• High license and infrastructure costs limiting scalability.
• Limited observability, manual scheduling, and minimal retry logic increased operational risk.
• Slow onboarding of new vendors and data formats.
• Lack of modularity across pipelines, making maintenance complex.
• Minimal governance and SLA monitoring across ingestion and transformation stages.

• Incremental migration strategy for zero-disruption transitions.
• Reusable pipeline templates using AWS Glue, Step Functions, and Lambda.
• SLA monitoring and automated recovery with DLQs and CloudWatch alerts.
• Secure ingestion and staging using AWS KMS, IAM roles, and segregated Landing–Staging–Prod layers.

1. Event-Driven Ingestion

• Ingest vendor files securely using AWS Transfer Family (SFTP → S3).
• Automated metadata-driven routing using DynamoDB.
• File encryption and role-based access control ensure secure transfer.

2. Transformation Processing

• AWS Glue (PySpark) executes complex transformations at scale.
• Lambda orchestrates lightweight compute and validation tasks.
• EventBridge coordinates state transitions and error handling.

3. Orchestration and Workflow Management

• Step Functions manage multi-step workflows across ingestion, validation, and load.
• DLQs and SQS queues ensure robust retry logic and traceability.

4. Validation and SLA Adherence

• Pre-load: Schema, format, and row-count checks.
• Post-load: Oracle/S3 reconciliation with results logged in DynamoDB or Aurora.
• Real-time monitoring integrated with Datadog for SLA breaches.

5. Data Loading to Oracle

• Lambda executes SQL-based INSERT/MERGE logic.
• Airflow orchestrates final loads and downstream triggers.

• Assessed legacy workflows and categorized based on complexity.
• Migrated simple workflows to Step Functions + Lambda.
• Migrated transformation-heavy logic to AWS Glue.
• Established metadata-driven pipeline generation to reduce manual effort.
• Implemented modular, auditable workflows with central logging and observability.

AWS S3, AWS Lambda, AWS Glue (PySpark), Step Functions, DynamoDB, EventBridge,

AWS Transfer Family, KMS, IAM, SQS, CloudWatch, Datadog, Airflow, Oracle.

• 70% reduction in ETL execution cost by eliminating license-heavy tooling.
• 2–3× improvement in pipeline performance.
• 100% SLA adherence with automated alerting and recovery workflows.
• Delivery of 250+ production-grade pipelines across DEV, BETA, and PROD.
• 60% improvement in vendor feed onboarding speed.
• Fully modular, auditable architecture supports future cloud expansion.

• Serverless architecture drastically improves scalability and cost efficiency.
• Metadata-driven orchestration accelerates onboarding and reduces errors.
• Built-in observability and automated recovery eliminate operational blind spots.
• Modular pipelines future-proof modernization efforts and support multi-cloud expansion.

The digital modernization initiative successfully replaced costly, monolithic ETL infrastructure with a flexible, serverless data pipeline framework on AWS. This transformation significantly enhanced performance, governance, security, and operational reliability empowering the enterprise to scale rapidly, onboard vendors with ease, and maintain strong SLA performance. The modernization framework now serves as a blueprint for future cloud-native migrations across the organization.