Blogs > Technology > Converting Legacy to Metadata Driven Framework
Converting Legacy to Metadata Driven Framework
Jun 01 2022 |6 min read
Problem Statement

The feeds were used by thousands of analytical jobs to calculate risk measures. Performance, stability and scalability were the main focus areas. The objective is to move to high end operating system that improves ease of operations as well as data quality implementation. 

Project Objectives
  • Migration from Solaris to Linux operating system 
  • Removing hardcoded security credentials from scripts
  • Creating a robust environment for ETL
  • Building different environments for dev & beta in addition to the production instance
  • Restricted access to ETL developers during business hours
  • Create a centralized metadata for: Source Vendor like connection string, username & password, whether be for FTP, HTTP or Email
  • Autosys job information including the business user details, SLA etc.
  • Create a separate code repository for ETL and move all code
Scope of Work Included
  • ~ 945 feeds were converted to new metadata driven framework
  • Dedicated environments for Dev, Beta, Prod, DR (Linux)
  • Parallel job execution supported on Dev, Beta, Prod, DR (on demand)
  • Dedicated file storage for each environment (Dev, Beta, Prod, DR)
  • Dedicated data schema in Oracle
Approach Followed

Step 1-Start enhancing the script on TEST environment and perform unit testing. Setup job in TEST autosys and execute.

Step 2 Compare data loaded in TEST tables vs PROD tables for a week using automated reporting tools.

Step 3- Stakeholders signoff on test reports.

Step 4- Production Release.

Technology Stack
  • Core Technologies: C++, Python, csh, Bash, Perl, Autosys
  • Databases: Oracle SQL Developer, SQL Server Management Studio
  • ETL Tool- Informatica, Talend
  • QA: Query Surge, ICEDQ
  • Reporting Tools: Micro-strategy, Power BI
Anmol Raheja

Anmol Raheja

Blogs you may like

There are no more Blogs for this Category