Databrick Engineer at Confidential company
About the Job
Hi,
Hope you are doing well?
I am looking for a candidate with experience as Databrick Engineer.
Role: Databrick Engineer
Location: Quincy, MA
Duration: Long term
Must be on our W2
Key skill- Databrick, AWS, Pyspark, Scala,
Education and experience Qualification
- Bachelor’s degree in Computer Science, Information Systems, or equivalent education or work experience
- Around 5+ years of experience as a developer on cloud technologies
- Any AWS and/or Databricks certification will be a plus
Roles & Responsibilities
- Recognize the current application infrastructure and suggest new concepts to improve performance
- Document the best practices and strategies associated with application deployment and infrastructure support
- Produce reusable, efficient, and scalable programs, and also cost-effective migration strategies
- Develop Data Engineering and ML pipelines in Databricks and different AWS services, including S3, EC2, API, RDS, Kinesis/Kafka and Lambda to build serverless applications
- Work jointly with the IT team and other departments to migrate data engineering and ML applications to Databricks/AWS
- Comfortable to work on tight timelines, when required.
Skill Sets Required
- Good decision-making and problem solving skills
- Solid understanding of Databricks fundamentals/architecture and have hands on experience in setting up Databricks cluster, working in Databricks modules (Data Engineering, ML and SQL warehouse).
- Knowledge on medallion architecture, DLT and unity catalog within Databricks.
- Experience in migrating data from on-prem Hadoop to Databricks/AWS
- Understanding of core AWS services, uses, and AWS architecture best practices
- Hands-on experience in different domains, like database architecture, business intelligence, machine learning, advanced analytics, big data, etc.
- Solid knowledge on Airflow
- Solid knowledge on CI/CD pipeline in AWS technologies
- Application migration of RDBMS, java/python applications, model code, elastic etc.
- Solid programming background on scala, python
- Experience with Docker and Kubernetes is a plus