The client has extensive security-level data for large financial institutions and insurance companies from multiple data vendors which is refreshed weekly. The data is used to generate multiple KPIs and summarized datasets which helped the managers analyze investment portfolios. Performance, Availability and Scalability were main focus areas. The objective is to develop a web-based application which can be used to generate portfolio report for an institution in minimum time.

• Create ETL jobs to migrate data from Oracle to Snowflake
• Data and Quality checks during migration process 
• Ensure hassle-free go-live through end-to-end testing across various parameters- performance, integration and user acceptance testing
• Create a web-based application that allows user to select companies and run analysis in minimum time while producing accurate results
• Incident & problem management

• Set up Snowflake Database and create all the data tables
• Set up Data Quality checks to be run on weekly data received from vendor
• Create ETL jobs to migrate weekly large data feeds from Oracle to Snowflake tables in optimized way
• Build an offline process that calculates high level data statistics and caches them for further use.
• Create a web-based application that allows user to select companies and run analysis on minimum time.
• Ensure hassle-free go-live through end-to-end testing across various parameters- performance, integration and user acceptance testing

• Created an aggressive project plan to approach development goals in an organized and targeted approach  
• Python scripts used with Apache Airflow to migrate weekly data from Oracle to Snowflake  
• Designed a Python-based backend architecture using Tornado framework to host REST-APIs
• Stored frequently used datasets and summaries on S3 and Redis to minimize runtime and increase reusability

• Core Technologies: Python
• ETL Tools: Apache Airflow
• Databases: Oracle SQL Developer, Snowflake
• Object DB: AWS S3, Redis
• Web Framework: Tornado