Data Engineer / Consultant - Smartbridge
Houston, TX 77042
About the Job
3-month Contract
Start first week of November 2024
Remote
Must have experience with
- Azure Data Factory & Synapse
- Azure Databricks
Data Engineer (3-Month Engagement)
We are seeking an experienced Data Engineer for a 3-month assignment. This role requires a strong understanding of data & analytics solution architecture, encompassing data warehousing, data lakes, ETL/ELT workload patterns, and related BI & analytics systems.
Key Responsibilities:
- Develop and implement technical best practices for data ingestion, data quality, data cleansing, and other data integration/ETL/engineering-related activities.
- Manage and maintain a multi-terabyte enterprise data warehouse with incremental data pipelines.
- Build infrastructure for optimal extraction, transformation, and loading (ETL) of data from various sources using SQL and modern cloud technologies.
- Participate in meetings with key system owners to understand the current data and systems environments.
- Resolve source data issues and refine transformation rules, implementing data quality processes to assist with data cleansing requirements.
- Analyze source system data to define transformation logic and data quality through data profiling.
- Collaborate with technical and business representatives to determine strategies for handling data anomalies.
- Design ETL processes, develop source-to-target data mappings, integration workflows, and load processes.
- Develop, test, integrate, and deploy data pipelines using various tools and external programming/scripting languages as needed.
- Create technical documentation and other artifacts for data pipelines, ingestion, integration, or other data solutions.
- Identify problems, develop ideas, and propose solutions using analytical, evaluative, or creative thinking.
- Apply innovative thinking to identify possible reporting solution alternatives.
- Additional duties may be assigned based on project needs.
Requirements and Qualifications:
- 3 years of hands-on experience with the following data technologies:
- Azure Data Factory & Synapse
- Azure Databricks
- Lake-first approach in building data solutions
- On-premise data warehousing solutions
- ETL design and development, data marts, and star schemas
- Building data warehousing solutions in Azure
- Migrating data from on-premise to cloud environments
- Designing data warehouse solutions using Synapse or Azure SQL DB
- Building pipelines with Synapse or Azure Data Factory for data ingestion
- Understanding of integration run times available in Azure
- Lake-first approach solutions in Azure leveraging Databricks, pySpark, and delta format
- Advanced SQL skills with experience in relational databases and query authoring.
- Proficiency with scripting languages like Python, Pyspark, or Scala.
- Microsoft Azure Cloud platform certifications are a plus.
- Ability to travel to client locations as needed during the contract period.
Note: This is a contract position for 3 months, with the possibility of extension based on project needs.
Powered by JazzHR
Source : Smartbridge