Senior Data Engineer

Essential Duties and Responsibilities:

• Create and maintain optimal data pipeline architecture
• Overseeing activities of the junior data engineering teams ensuring proper execution of their duties and alignment with business vision and objectives.
• Assemble large complex data sets that meet functional / nonfunctional business requirements.
• Identify design and implement internal process improvements: automating manual processes optimizing data delivery redesigning infrastructure for greater scalability etc.
• Work closely with the business’s Data and Analytics teams gathering technical requirements for exceptional data governance across the department and the business at large
• Learn how to write more efficient code and support the development of predictive datadriven solutions.
• Create systems and procedures to acquire organize and clean raw data that data scientists and MIS teams.
• Improves efficiency through workable thinking and the efficient use of new technologies and concepts
• Create UT cases to ensure that the codebase maintains system stability and documentation.
• Implement best practices and learn innovative approaches to keep codebases up to date on a regular basis.

Technical Skills:

• Working with cloud services data lake storage Hadoop Spark Python/ Scala Hive HDFS SQL and NoSQL databases.
• Knowledge on creating Batch and Realtime Data pipelines with different cloud services ETL tools and Kafka etc.
• Perspective to always create optimized complex ETL mappings for relational and nonrelational workloads.
• Hands on Unix scripting or PowerShell.
• Design DWH model ensuring data design follows the prescribed reference architecture framework while reflecting appropriate business rules built for logical physical and conceptual model.
• Knowledge on setting up code versioning with GitHub and CICD pipelines.
• Facilitate data integration data conformity data integrity and ensure data structures are designed for flexibility to support future business needs and enhancements.
• Understand the relationships across business information and units of data Reverse engineer data from various source systems.
• Exceptional data analysis critical thinking communication problem solving and crossgroup collaboration skills.