AWS Spark Terraform Data Engineer - Kforce Inc.
San Francisco, CA 94111
About the Job
- Data Onboarding: Onboard various data sources into the data lake, ensuring seamless integration and data consistency
- Data Pipeline Development: Design, develop, and maintain scalable and efficient data pipelines using AWS services such as Lambda, Step Functions, and EMR
- Data Registration: Register data sources and manage metadata to ensure data discoverability and accessibility
- Data Quality Management: Implement data quality checks and transformations to ensure the accuracy and reliability of data
- Data Governance: Comply with data governance principles and best practices to ensure data security, privacy, and compliance
- Infrastructure as Code: Utilize Terraform scripting to manage and automate AWS infrastructure
- Data Processing: Leverage Spark and other big data technologies to process and analyze large datasets
- Orchestration: Use Airflow and Step Functions to orchestrate complex data workflows
- Data Modeling: Work with Snowflake, Iceberg table formats, and other data modeling tools to design and optimize data storage solutions
- Collaboration: Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and deliver high-quality data solutions
Requirements:
- AWS Services: Proficiency in AWS Lake Formation, Step Functions, Lambda (serverless), EC2, EMR, and EKS
- Scripting and Programming: Strong experience with Python and Terraform scripting
- Data Tools: Experience with Jupyter Notebook, RDS, Snowflake, and Iceberg table formats
- Big Data Technologies: Expertise in Spark and data pipeline orchestration tools like Airflow and dbt
- Data Engineering: Solid understanding of data engineering principles, including ETL processes, data warehousing, and data modeling
- Data Governance: Knowledge of data governance principles and best practices
- Problem-Solving: Strong analytical and problem-solving skills with the ability to troubleshoot and resolve data-related issues
- Communication: Excellent communication skills with the ability to collaborate effectively with cross-functional teams
- Certifications: AWS Certified Data Engineer or Analytics - Specialty, AWS Certified Solutions Architect, or other relevant certifications
- Experience: Previous experience in a similar role within a fast-paced, data-driven environment
The pay range is the lowest to highest compensation we reasonably in good faith believe we would pay at posting for this role. We may ultimately pay more or less than this range. Employee pay is based on factors like relevant education, qualifications, certifications, experience, skills, seniority, location, performance, union contract and business needs. This range may be modified in the future.
We offer comprehensive benefits including medical/dental/vision insurance, HSA, FSA, 401(k), and life, disability & ADD insurance to eligible employees. Salaried personnel receive paid time off. Hourly employees are not eligible for paid time off unless required by law. Hourly employees on a Service Contract Act project are eligible for paid sick leave.
Note: Pay is not considered compensation until it is earned, vested and determinable. The amount and availability of any compensation remains in Kforce's sole discretion unless and until paid and may be modified in its discretion consistent with the law.
This job is not eligible for bonuses, incentives or commissions.
Kforce is an Equal Opportunity/Affirmative Action Employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, pregnancy, sexual orientation, gender identity, national origin, age, protected veteran status, or disability status.
By clicking “Apply Today” you agree to receive calls, AI-generated calls, text messages or emails from Kforce and its affiliates, and service providers. Note that if you choose to communicate with Kforce via text messaging the frequency may vary, and message and data rates may apply. Carriers are not liable for delayed or undelivered messages. You will always have the right to cease communicating via text by using key words such as STOP.
