Data Bricks developer at Infinite Computer Solutions Inc

Data Bricks developer

Location: Morris Plains, NJ (Remote and onsite both are fine)

Required skills:

• At least 4 years of experience in Scala, Python, Spark (Batch and Structured Streaming) development.
• Clear understanding of Hadoop Ecosystem & Spark Architecture.
• Experience building data pipelines using both Spark structured streaming and spark batch processes.
• In depth understanding of Databricks Lake house Architecture.
• Experience in building data pipelines using Databricks.
• Good experience in build dashboards using any BI/reporting tools.
• Working experience in AWS services (lambda, S3, Data sync).
• Experience in consuming and producing messages to Kafka topic.
• Strong understanding of Functional programming and RESTful APIs.
• Produce unit tests for Spark transformations and helper methods.
• Experience with design and implementation of ETL framework for complex warehouses, Knowledge of large data sets and experience with performance tuning and troubleshooting.
• Strong knowledge and hands-on experience in SQL.
• Basic UNIX commands.
• Experience using Git, Jenkins, Terraform, IDEs, Jira .