Apache Spark Engineer - pro it

Role:&

Apache Spark Engineer

Location:

& Remote, PST Time

Duration:&

6 mo to start

Role Info:

Build an open source project and continually adding to the community

Successful candiate must be contributing to Spark and&

NOT

& using Spark

"Apache Spark community committer "

Guide the team and help them make decisions

Overview

The Apache Spark Engineer plays a crucial role in our organization by designing and implementing big data processing solutions using Apache Spark. They are responsible for developing and maintaining scalable and high-performance applications to process and analyze large volumes of data efficiently.

Key Responsibilities

Designing and implementing scalable and high-performance data processing applications using Apache Spark

Collaborating with data scientists and analysts to understand data processing requirements

Optimizing and tuning Apache Spark applications for maximum performance

Developing and maintaining ETL pipelines for data ingestion and processing

Performing debugging and troubleshooting of Apache Spark applications

Ensuring data security and integrity in Apache Spark applications

Implementing best practices for Apache Spark development and deployment

Participating in code reviews and contributing to continuous improvement of development processes

Utilizing Scala and Hadoop ecosystem technologies to enhance Apache Spark applications

Collaborating with cross-functional teams to integrate data processing applications with other systems

Required Qualifications

Bachelor's or Master's degree in Computer Science, Engineering, or a related field

Proven experience in developing and maintaining Apache Spark applications

Demonstrated proficiency in Scala programming language

Strong understanding of big data concepts and technologies

Experience with Hadoop ecosystem tools such as HDFS, MapReduce, and Hive

Proficiency in designing and optimizing ETL processes for large-scale data

Knowledge of data security and compliance standards in big data environments

Experience with performance tuning and optimization of Apache Spark applications

Ability to work in a fast-paced, collaborative environment with a focus on results

Excellent problem-solving and analytical skills

Strong interpersonal and communication skills to collaborate with diverse teams

Familiarity with cloud-based big data platforms such as AWS or Azure is a plus

Relevant certifications in big data and Apache Spark are preferred