Quality Engineer Level 3 - Apidel Technologies

Job Description:

We are seeking a detail-oriented ETL Tester withexpertise in Python, Databricks, and Google Cloud Platform (GCP) to join ourdata engineering team. The ideal candidate will be responsible for validatingETL processes, ensuring data quality, and supporting data integrationinitiatives. You will work closely with data engineers, analysts, andstakeholders to ensure that our data pipelines are robust, accurate, andreliable.

Key Responsibilities:

ETL Process Validation:

Validate and verify ETL processes implemented in GCP,ensuring data integrity during extraction, transformation, and loading.

Develop and execute comprehensive test cases to confirmthat data transformations meet business requirements.

Data Quality Assurance:

Conduct data profiling and perform quality checks toidentify and resolve discrepancies in datasets.

Monitor data quality metrics and report on data integrityand quality issues.

Test Case Development:

Create and maintain detailed test plans and test casesbased on ETL specifications and business needs.

Ensure full coverage of all ETL processes, including dataextraction, transformation, and loading.

Collaboration:

Work closely with data engineers, data scientists, andother stakeholders to understand ETL workflows and data flows.

Participate in design reviews to provide input on testingstrategies and best practices.

Automation:

Use Python to develop automated testing scripts for ETLvalidation and data quality checks.

Leverage Databricks notebooks for testing and validatingETL processes efficiently.

Workflow Management:

Utilize Apache Airflow for scheduling, monitoring, andmanaging ETL workflows.

Collaborate with teams to troubleshoot and optimizeAirflow DAGs related to ETL processes.

Issue Tracking and Resolution:

Identify, document, and track defects and data qualityissues throughout the ETL process.

Work with engineering teams to diagnose and resolvedata-related problems quickly.

Documentation:

Maintain clear and comprehensive documentation of testingprocesses, test cases, and results.

Document data mappings, transformation rules, and dataflow diagrams for reference.

Continuous Improvement:

Contribute to the enhancement of ETL testingmethodologies and data management practices.

Stay updated on GCP, Databricks, and industry trends tocontinuously improve testing strategies.

Note to Vendors:

Qualifications:

Strong knowledge of Google Cloud Platform, specificallyBigQuery, GCS, and Airflow.

4%2B years of Experience with Databricks for dataprocessing and analysis.

Proficiency in Python for developing testing scripts andautomating testing processes.

Please go off market rate for this req.

Prescreen consists of 5 questions and a game

Remote candidates will be considered