Data Pipeline & ETL Migration (REMOTE) - ADPMN INC

Job Title:  Data Pipeline & ETL Migration 

Location: Remote

Duration:6%2Bmonths

(ONLY USC & GC)

JOB DESCRIPTION::

• Lead the migration of legacy data warehouse services (Oracle, SQL Server) into Snowflake for client-facing solutions, ensuring smooth data extraction, transformation, and loading(ETL).
• Implement Snowflake Time Travel and Snapshot DBs to replace outdated legacy services, enhancing data backup and recovery capabilities for clients.
• Design and build automated pipelines to extract, transform, and load (ETL) data from various systems, including manual systems and system of record (SoR).
• Ensure encryption protocols are implemented where necessary to protect PII and sensitive data during migration and post-migration processes.
• Support manual data collection processes that involve moving data to S3 buckets, automating them where it is possible to reduce reliance on manual interventions.
• Collaborate with the network and security teams to address potential security issues, particularly with data grabber solutions used for pulling data from client systems.
• Optimize data flow architectures and monitoring to support efficient data extraction from legacy systems to Snowflake, ensuring seamless integration and performance.
• Develop and deploy solutions to automate manual tasks and improve the scalability of data extraction processes for 40%2B channels.
• Troubleshoot issues related to data migration, such as system integration and data compatibility, while working closely with clients.
• Oversee the integration of security protocols into the network architecture, ensuring encryption standards (SSL/TLS, PGP) are adhered to at every stage of the data flow.

Skills & Qualifications:

• Strong experience in SQL Server to Snowflake migrations, with specific knowledge of Snowflake Time Travel and Snapshot DBs for data recovery.
• Expertise in building and optimizing ETL pipelines using S3 and Snowflake Tasks (or other DAG tools) for efficient data processing and transformation.
• Knowledge of data encryption techniques and secure handling of PII within cloud data platforms like Snowflake.
• Familiarity with legacy systems, particularly with Oracle and client-facing data services, and experience in automating manual data collection processes.
• Ability to troubleshoot complex data issues related to legacy dependencies, such as ETL tools and reporting systems.
• Hands-on experience with data security, particularly in addressing potential security issues related to external data pull processes (e.g., data grabber).
• Excellent collaboration skills to work with both internal teams and clients, ensuring seamless migration and minimal disruptions to ongoing operations.
• Strong problem-solving skills to resolve data migration challenges, such as system incompatibilities and data flow optimization.
• In-depth knowledge of data encryption, key management, and secure transmission methods (SFTP, FTPS) in cloud environments.
• Experience in cloud-native data extraction tools (AWS Glue, Snowflake Tasks) for building and optimizing ETL pipelines.