REMOTE Senior Data Engineer - Insight Global

Job Description We are looking for a fully remote Senior Data Engineer to join a client in the rail industry. In this role, you will be responsible for expanding and optimizing our data and data pipeline architecture, as well as optimizing data flow and collection for cross functional teams. The ideal candidate is an experienced data pipeline builder and data wrangler who enjoys optimizing data systems and building them from the ground up. The Data Engineer will develop and support a broad range of software capabilities including building data pipelines, managing the ETL/ELT processes, receiving, and delivering data through various interfaces, and processing significant amounts of data related to railcar movements, railcar liability, and financial calculations. The right candidate will be excited by the prospect of optimizing or even re-designing our company's data architecture to support our next generation of products and data initiatives. A successful candidate has experience working with Agile teams and developing data engineering solutions, Design and develop generic, scalable data pipelines in Azure Data factory and Databricks with python for on-prem and cloud data sources, as well as leverage your curiosity for solving unstructured data problems and ability to manipulate and optimize large data sets to advance business problem-solving. We are looking for a highly technical individual who is at the expert-level in coding with Python and has a background in data warehousing. We are a company committed to creating diverse and inclusive environments where people can bring their full, authentic selves to work every day. We are an equal opportunity/affirmative action employer that believes everyone matters. Qualified candidates will receive consideration for employment regardless of their race, color, ethnicity, religion, sex (including pregnancy), sexual orientation, gender identity and expression, marital status, national origin, ancestry, genetic factors, age, disability, protected veteran status, military or uniformed service member status, or any other status or characteristic protected by applicable laws, regulations, and ordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or recruiting process, please send a request to HR@insightglobal.com . To learn more about how we collect, keep, and process your private information, please review Insight Global's Workforce Privacy Policy: https://insightglobal.com/workforce-privacy-policy/ . Skills and Requirements Bachelor's degree in computer science, computer engineering, a related field, or equivalent experience. 5+ years of data engineering or equivalent experience. Expert-level technical experience with Apache Spark / Azure Databricks. Expert-level coding experience with PySpark and Python. Proficient in using and designing solutions on Azure Cloud infrastructure (particularly Azure Data Factory) and Azure DevOps. Proficient with core business intelligence and data warehousing technology. Proficient designing and developing data integration solutions using ETL tools such as Azure Data Factory and/or SSIS. Proficient with software development practices such as Agile, TDD, and CI/CD. 5+ years of hands-on experience in developing and deploying data architecture strategies or engineering practices. 5+ years of experience with complex SQL queries and knowledge of database technologies. Experience with graph databases or graph libraries Kafka or other streaming technologies Elastic Search Snowflake Prior experience in rail industry or other transportation-based commodities. null We are a company committed to creating diverse and inclusive environments where people can bring their full, authentic selves to work every day. We are an equal employment opportunity/affirmative action employer that believes everyone matters. Qualified candidates will receive consideration for employment without regard to race, color, ethnicity, religion,sex (including pregnancy), sexual orientation, gender identity and expression, marital status, national origin, ancestry, genetic factors, age, disability, protected veteran status, military oruniformed service member status, or any other status or characteristic protected by applicable laws, regulations, andordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or the recruiting process, please send a request to HR@insightglobal.com.