Data Engineer with Azure and MS Fabric - Diverse Linx
Newark, NJ
About the Job
Role- Data Engineer with Azure and MS Fabric
Location- Newark, NJ
Job type- contract
A Data Engineer with Azure and MS Fabric is a specialized role focused on designing, building, and maintaining data infrastructures and systems to enable organizations to collect, store, process, and analyze large volumes of data effectively.
Core Responsibilities:
Data Ingestion: Designing and implementing data pipelines to extract data from various sources (databases, APIs, files, etc.) into Azure data storage solutions.
Data Transformation: Cleaning, transforming, and enriching data to prepare it for analysis.
Data Storage: Selecting and optimizing Azure storage solutions (Data Lake Storage, Blob Storage, Azure SQL Database) for different data types and workloads.
Data Processing: Utilizing Azure Data Factory, Azure Databricks, or Synapse Spark for data processing, transformation, and ETL/ELT pipelines.
Data Modeling: Creating data models and schemas to support reporting and analysis.
Data Quality: Ensuring data accuracy, consistency, and completeness through data profiling and cleansing.
Performance Optimization: Identifying and resolving performance bottlenecks in data pipelines and queries.
Cloud Infrastructure: Managing and maintaining Azure resources, including virtual networks, storage accounts, and compute resources.
MS Fabric: Leveraging the capabilities of MS Fabric for data engineering tasks, including data integration, transformation, and job orchestration.
Required Skills:
Programming Languages: Python, Scala, or SQL for data processing and analysis.
Cloud Platforms: Azure, including Azure Data Factory, Azure Databricks, Azure Synapse Analytics, Azure Data Lake Storage.
Data Engineering Tools: Experience with ETL/ELT tools and data integration patterns.
Data Warehousing and Modeling: Knowledge of data warehousing concepts and dimensional modeling.
Big Data Technologies: Understanding of distributed computing frameworks (Spark, Hadoop).
Cloud Infrastructure: Proficiency in managing cloud resources and costs.
MS Fabric: Familiarity with the components and capabilities of MS Fabric.
Challenges and Opportunities:
Data Volume and Velocity: Handling large and rapidly growing datasets.
Data Quality: Ensuring data accuracy and consistency.
Cloud Cost Optimization: Managing cloud resources efficiently.
Emerging Technologies: Staying updated with the latest data technologies and trends.
Diverse Lynx LLC is an Equal Employment Opportunity employer. All qualified applicants will receive due consideration for employment without any discrimination. All applicants will be evaluated solely on the basis of their ability, competence and their proven capability to perform the functions outlined in the corresponding role. We promote and support a diverse workforce across all levels in the company.
Location- Newark, NJ
Job type- contract
A Data Engineer with Azure and MS Fabric is a specialized role focused on designing, building, and maintaining data infrastructures and systems to enable organizations to collect, store, process, and analyze large volumes of data effectively.
Core Responsibilities:
Data Ingestion: Designing and implementing data pipelines to extract data from various sources (databases, APIs, files, etc.) into Azure data storage solutions.
Data Transformation: Cleaning, transforming, and enriching data to prepare it for analysis.
Data Storage: Selecting and optimizing Azure storage solutions (Data Lake Storage, Blob Storage, Azure SQL Database) for different data types and workloads.
Data Processing: Utilizing Azure Data Factory, Azure Databricks, or Synapse Spark for data processing, transformation, and ETL/ELT pipelines.
Data Modeling: Creating data models and schemas to support reporting and analysis.
Data Quality: Ensuring data accuracy, consistency, and completeness through data profiling and cleansing.
Performance Optimization: Identifying and resolving performance bottlenecks in data pipelines and queries.
Cloud Infrastructure: Managing and maintaining Azure resources, including virtual networks, storage accounts, and compute resources.
MS Fabric: Leveraging the capabilities of MS Fabric for data engineering tasks, including data integration, transformation, and job orchestration.
Required Skills:
Programming Languages: Python, Scala, or SQL for data processing and analysis.
Cloud Platforms: Azure, including Azure Data Factory, Azure Databricks, Azure Synapse Analytics, Azure Data Lake Storage.
Data Engineering Tools: Experience with ETL/ELT tools and data integration patterns.
Data Warehousing and Modeling: Knowledge of data warehousing concepts and dimensional modeling.
Big Data Technologies: Understanding of distributed computing frameworks (Spark, Hadoop).
Cloud Infrastructure: Proficiency in managing cloud resources and costs.
MS Fabric: Familiarity with the components and capabilities of MS Fabric.
Challenges and Opportunities:
Data Volume and Velocity: Handling large and rapidly growing datasets.
Data Quality: Ensuring data accuracy and consistency.
Cloud Cost Optimization: Managing cloud resources efficiently.
Emerging Technologies: Staying updated with the latest data technologies and trends.
Diverse Lynx LLC is an Equal Employment Opportunity employer. All qualified applicants will receive due consideration for employment without any discrimination. All applicants will be evaluated solely on the basis of their ability, competence and their proven capability to perform the functions outlined in the corresponding role. We promote and support a diverse workforce across all levels in the company.
Source : Diverse Linx