Senior Data Engineer - Karsun Solutions, LLC

As a Senior Data Engineer, you will be responsible for the maintenance, synchronization, cleaning, and migration of transactional data in a hybrid environment with both on-prem and highly modern cloud based microservices environment. You will work with the product teams to understand, analyze, document and efficiently implement to deliver streaming as well as batch oriented data for synchronizing legacy and modern data stores ensuring data integrity.  You will provide support to the application database design to aid in eliminating data duplication and other issues and enabling selective event based and scheduled based data transfer to endpoints within the cloud and legacy environment as required.  Using out of the box thinking, AWS native capabilities and CI/CD tools, you will drive towards programmatic pipeline generation and orchestration to enhance repeatability and rapid deployment while utilizing established design patterns and methods.  

The successful candidate will be able to rapidly develop technical solutions working closely with the integrated product teams and developers with minimal direction from senior or lead resources

• Understand data needs and be able to construct data pipelines for automating event driven bi-directional selective data replication, along with micro-batch and batch- based data pipelines
• Standardization of data processing modules to deliver modularity and enhance reusability
• Understand, maintain and perform operation on datasets stored in relational databases such as MySQL, SQL Server and Redshift, as well as in AWS S3
• Utilize data processing tools and services such as StreamSets, Python, Java and shell scripting, AWS tools such as Glue, Step functions and Lambda and DB tools such as MySQL Workbench and SQL Server Management Studio
• Create and maintain standards and best practices for data and pipeline standards
• Support a variety of structured, semi-structured and unstructured data in streaming and batch frameworks
• Design, create and support the data pipeline ETL processes across various data assets within the current scope of the system
• Monitor, troubleshoot and coordinate defect resolution related to data processing & preparation

Required:

• Typically requires a bachelor's degree or higher in Computer Science or related discipline and 8+ years of related experience including:
• 4+ years of hands-on experience in data transformation, creation of complex SQL queries and functions as well as data processing, cleanup and migration (MySQL and SQL Server preferred)
• 3+ years of hands-on experience in with Python and various Python toolkits and libraries for data processing and pipelines
• 3+ years of Database operations and administration (MySQL and SQL Server preferred)
• 2+ years of hands-on experience in with ETL tools such as Streamsets, AWS Glue, Pentaho etc.
• 1+ year of hands-on experience in AWS
• 1+ year of experience in Java and Linux scripting
• 1+ year experience working with CI/CD tools including Git
• Successful track record in data migrations, database operations and maintenance and ETL job design and development, as well as scripting and automation activities with minimal supervision.
• Proven skills in database operations, such as export/import, backup/restore etc. as well as proven experience with AWS data tools and services related to data processing.
• Ability to monitor, troubleshoot and coordinate defect resolution related to data processing & preparation.
• Ability to obtain and maintain a Public Trust clearance

Desired:

• Experience of working with multiple AWS tools and any AWS certification is highly desired
• Advanced Database Administration skills in SQL Server, MySQL and Redshift is highly desired
• Experience with big data tools such as EMR/Spark, Databricks/PySpark is an advantage
• Experience working with database versioning tools and tools like SSRS is an advantage
• Experience in Ansible and Jenkins scripting is an advantage
• Experience supporting US government customers
• Highly prefer candidates residing in the easter, central or mountain time zones

In accordance with pay transparency guidelines, the proposed salary range for this position is $125,000 to $165,000. Final salary will be determined based on various factors such as relevant skills, experience and certifications.