Description
Lead Azure Data Engineer Architect role is to strategically design develop and implement Azure Cloud based Data Warehouse Data Lake Data Lakehouse Data Marts & BI solution using modern data engineering techniques and advanced analytics methods to provide information & insights to end users. The role requires modeling collecting harmonizing aggregating storing and reconciling data from variety of data sources for our petroleum refineries IT/OT technology platform including time series Database SCADA data batch real time enterprise applications Rest API s streaming & IOT data. The role will help design and build data pipelines streams data warehouse data lake reporting application data generators and a whole range of tools to provide information and insight.
The ideal candidate will be responsible for designing developing and maintaining scalable data processing pipelines in a cloudbased environment. The candidate should have a deep understanding of Azure ADF Databricks streaming analytics data engineering best practices and a proven track record of delivering highquality real time solutions.
Responsibilities
Strategy & Planning
- Assess and cultivate longterm strategic goals for data warehouse data lake BI & advance analytics development in conjunction with end users managers clients and other stakeholders.
- Coordinate and work with other technical staff to develop Azure Data & BI architecture coding standards and quality assurance policies and procedures.
- Promote a culture of selfserve data analytics by minimizing technical barriers to data access and understanding.
- Is responsible for data collaboration and communication with various stakeholders.
Operational Management
- Lead the integration efforts for Azure data warehouse data lakehouse sourcing data from disparate enterprise systems including applications both on premises & cloud IOT devices Data Service (Rest API s) timeseries database semistructure unstructured data & external data sources.
- Organize and lead projects in the implementation and use of BI software tools and systems.
- Design code test and document all new or modified BI systems applications and programs.
- Develop the semantic layer metadata reports and report definitions.
- Assist in the design of databases to ensure interoperability with DW & BI solutions.
- Produce ELT/ETL design guidelines to ensure a manageable ELT/ETL infrastructure for the analytics system.
- Participate in design discussions about new features and ensure realtime technology integrates seamlessly with other pieces of the platform
- Applying technical knowledge to architect solutions that meet business needs AI/ML roadmaps and contributing to architectures that enable the ability to scale to support additional modelling use cases (e.g. Azure ML Azure Databricks Azure EventHub/IoT Hub etc)
- Build processes supporting data transformation data structures metadata dependency and workload management.
- Work with project managers to ensure that data entry retrieval change and delete functions meet business requirements for project completion.
- Act as the primary owner and maintainer of new and existing Lakehouse/DW/BI data models
- Design implement and maintain CI/CD pipelines using Azure DevOps to integrate analytical solutions into cloud based custom developed applications
- Support Cloud strategy team to integrate analytical capabilities into an overall cloud architecture and business case development
- Facilitate Master/ Reference Data Management and overall Data Governance process
- Conduct job duties and responsibilities according to the organization s business systems development methodology and its Systems Development Life Cycle (SDLC) methodology.
Design and deliver enduser training and training materials; provide technical support as necessary.
- Troubleshoot Data Integration & BI tools systems and software; performancetune these applications as necessary.
- Defines and lead proof of concept activity associated with cloud data & analytics technology assessment.
- Act as evangelist for Azure Cloud data & analytics benefits across the organization; increase usage to relevant departments.
- Act as a subject matter expert on all things cloud data architecture and pipeline development
Position Requirements
Knowledge & Experience
- Minimum of 12 to 15 years of Data Engineering Data Warehouse/ Business intelligence with handson experience leading and delivering multiple full life cycle implementations.
- Comprehensive data warehouse analysis and design experience with full knowledge of data warehouse methodologies and data modeling.
- Handson experience & deep understanding in designing development and implementation of Azure Data & Analytics architecture data modeling transformations and development.
- Indepth knowledge and minimum 5 to 7 years experience developing & implementing data engineering solutions using Azure services for Data Warehouse Data Lakehouse & Advance analytics Primarily Azure Data Factory ADLS Azure Databricks Azure Functions Logic Apps IoT Hub Event Hub & Azure SQL
- Experience with Data pipeline and workflow management tools & Azure DevOps processes.
- Strong proficiency in programming languages such as Python PySpark or Scala
- End to End Implementations expertise in Azure Data Warehouse & Data Lakehouse
- Solid understanding of database structure principles Master and Reference Data management
- Experience & expertise in optimizing the ETL/ELT data pipeline technology options to minimize the ongoing operational cost of Azure data engineering & storage solutions.
- 8 years experience in agile Data warehouse development methodology.
- 8 years experience designing and developing dimensional models for data warehousing
- 3 years experience in MSBI suite primarily SSIS
- 3 years of implementing operational data model is plus not essential
- 3 years of experience with big data batch/streaming processing.
- 5 years experience using Pyspark Python or Scala
- Experience delivering reporting solution using visualization tools like PowerBI
- Experience in implementing SAP data analytics solution in Azure is plus not essential
- Experience using data modeling tool like ERWin
- Ability to lead and work with remote & offshore teams.
- Experience integrating & analyzing data from timeseries databases SCADA data IOT voice to text images & videos for creating Industrial Intelligence solution.
- Knowledge/expertise on Azure ML/AI predictive modeling capabilities is a plus
- Experience using proven methods to solve business problems using Azure Data and Analytics services in combination with building data pipelines data streams and system integration
- Handson experience with analytics and big data technologies within Microsoft Azure Cloud platform in tools such as Azure Machine Learning & Azure Cognitive Services is plus not essential
- Experience with data profiling cataloguing and data mapping for technical design using a use casebased approach that drives the construction of technical data flows
- Experience with Azure Administration & Security (Connection setup User setup Object security) is plus
- Understanding of statistical and machine learning modeling with experience applying these modeling techniques to business problems is a plus not essential.
- Ability to set up data and experimental platforms using Azure cloud service
- Ability to work independently with business stakeholders to collect and understand requirements for reporting & analytics.
- Full project management and software development life cycle experience.
- Experience with data process flowcharting techniques.
- Oil & Gas industry experience preferred but not essential.
- Experience & knowledge of SAP S4 and SAP BW4HANA is a plus not essential.
- Excellent quantitative analytical & mathematical skills is a plus
- Strong understanding of the organization s goals and objectives
Formal Education & Certification
- Bachelor s degree in computer science information systems or a related field or equivalent work experience.
- Certifications in Azure Data Engineering and Analytics space is a plus
Personal Attributes
- Ability to work independently and collaboratively in a fastpaced environment
- Strong written and oral communication skills.
- Strong presentation and interpersonal skills.
- Exceptional analytical conceptual and problemsolving abilities.
- Able to prioritize and execute tasks in a highpressure environment.
Experience working in a teamoriented collaborative environment.
data lake,data governance,pyspark,iot,azure data factory,scada data,bi,azure databricks,data warehouse,power bi,azure devops,ci/cd,data process flowcharting,azure ml,statistical modeling,data mapping,microsoft azure cloud platform,analytics,iot hub,azure cognitive services,sap bw4hana,sap s4,data profiling,etl,streaming analytics,powerbi,azure sql,data engineering,python,azure eventhub,big data technologies,metadata,machine learning modeling,azure,data lakes,data modeling