Senior Data Engineer - Xpertbizz
Norwell, MA
About the Job
We are on a quest for a highly skilled and passionate Senior Data Engineer to be a part of our dynamic team. The Data Engineer will be responsible for designing, implementing, and managing data solutions on the Microsoft Azure cloud platform. This role involves working with various data storage options offered by Microsoft Azure, developing, and implementing data integration processes, and using big data technologies to handle and analyze large volumes of data for multiple purposes. In addition, the role requires supporting AI implementations and Power BI reporting.
If you have a knack for problem-solving, a passion for data, and a desire to make a significant impact, we would love to hear from you. Together, we can shape the future of data strategy and the environmental services industry, Join us on this exciting journey!
Responsibilities
Provide thought leadership, strategic insight, and clear communication on Data Engineering and execution for complex problems.
Design data storage systems that scale, perform, and are economical enough to satisfy the organization's requirements.
Develop and implement data integration processes, frequently using Azure Data Factory, to extract, transform, and load (ETL) data from various sources into Azure data warehouses or data lakes.
Use big data technologies like Apache Spark to handle and analyze enormous volumes of data.
Collaborate with data scientists to create data sets and engineer features for AI Initiatives
Implement and optimize data retrieval for Power BI solutions.
Plan, coordinate, and oversee all technical aspects of our Azure data initiatives. Ensure that project goals align with the company's objectives and meet the needs of our stakeholders.
Design and implement data-driven solutions using Microsoft Azure's tools and services, such as Azure Machine Learning, Azure Databricks, Azure Data Factory and Azure databases or Fabric implementation. Ensure these solutions are scalable, reliable, and secure.
Lead a talented team of data scientists working in multiple regions. Foster a collaborative environment that encourages innovation and professional growth.
Work closely with cross-functional teams to understand their needs and challenges. Leverage your expertise to provide data-driven AI solutions that address these issues.
Leverage large volumes of data from various sources(Hybrid Cloud, OnPrem) to support teams building AI solutions and meaningful insights.
Drive feedback loop for the models implemented in production using metrics to ensure our whole MLOps pipeline is effective and continuously refreshed with the right data set and algorithms.
Stay updated with the latest trends and advancements in data science, machine learning, GenAI space and cloud computing.
Qualifications
A master's degree in computer science, Statistics, Mathematics, or a related field is required. A bachelor's degree with significant relevant experience would suffice.
5+ years of data engineering experience leveraging technologies such as Azure Data Factory, Azure Synapse Analytics, Azure Stream Analytics, Azure Event Hubs, Azure Data Lake Storage, Azure Databricks
Experience with data pipelines and engineering infrastructure: You should have experience designing and implementing data pipelines and engineering infrastructure for machine learning systems.
Solid knowledge of data processing languages, including SQL, Python, Scala.
Experience with AI implementations and Power BI.
Experience with Microsoft Azure's data services: You should have hands-on experience with various data services offered
by Microsoft Azure, such as Azure Machine Learning, Azure Databricks, and Azure Data Factory.
Familiarity with Azure Cognitive Services for integrating AI capabilities (like vision, language, and decision functionalities) directly into applications.
Ability to work with large datasets and experience with data processing tools like SQL, Pandas, and big data technologies such as Apache Spark.
Proficiency in Python, .NET C#, API development
Experience with SQL, NoSQL databases and Data Fabric architectures.
Strong communication skills to effectively collaborate with team members and stakeholders.
Strong problem-solving skills and ability to think algorithmically.
Ability to translate complex findings into a compelling narrative for non-technical stakeholders.
Certifications like Microsoft Certified: Azure Data Engineer Associate will be preferred.
Source : Xpertbizz
