Big Data Engineer - Career Portal

Overview

The Big Data Engineer plays a crucial role in the organization by designing, developing, and maintaining the big data infrastructure. They are responsible for implementing and managing large-scale data processing systems and platforms to support the company's data initiatives. They work closely with data scientists, analysts, and other stakeholders to ensure that the infrastructure meets the organization's data processing needs.

Key responsibilities

Design and build large-scale data processing systems.

Develop and maintain data ingestion pipelines.

Implement data storage solutions for structured and unstructured data.

Optimize and tune the performance of big data solutions.

Collaborate with data scientists and analysts to understand data requirements.

Create and maintain documentation for big data systems and processes.

Ensure data security and privacy are maintained in all big data solutions.

Monitor and troubleshoot big data solutions to ensure smooth operation.

Implement and manage big data tools and technologies.

Provide technical support and guidance to other teams using big data systems.

Required qualifications

Bachelor's degree in Computer Science, Information Technology, or related field.

Proven experience in big data technology stack (Hadoop, Spark, etc.).

Proficiency in SQL and NoSQL databases.

Strong programming skills in languages such as Python, Java, or Scala.

Experience in data modeling and data warehousing concepts.

Understanding of distributed computing and parallel processing.

Knowledge of data analysis and visualization tools.

Familiarity with cloud platforms and services, such as AWS, Azure, or Google Cloud.

Ability to work in a collaborative, team environment.

Excellent problem-solving and analytical skills.

Effective communication and interpersonal abilities.

Relevant certifications in big data technologies is a plus.

Understanding of agile and DevOps methodologies.

Experience in implementing data security and compliance measures.