Vidpro Consultancy Services is seeking a Senior Databricks Engineer / Databricks Technical Lead / Data Architect with 510 years of experience.
Location: Bangalore Chennai Delhi Pune Kolkata and Gurgaon
Work Type: Hybrid
Mandatory Skills: Python/ PySpark / Spark with Azure/ Azure Databricks / Azure DataFactory
Primary Roles and Responsibilities:
Creating Modern Data Warehouse solutions using Databricks and AWS/Azure Stack
Providing forwardthinking solutions in data engineering and analytics
Collaborating with DW/BI leads for new ETL pipeline development requirements
Troubleshooting issues in existing pipelines and addressing gaps
Understanding reporting layer needs and developing data models
Assisting team members with problem resolution and technical challenges
Leading technical discussions with client architects and team members
Orchestrating data pipelines in a scheduler via Airflow
Skills and Qualifications:
Bachelors and/or master s degree in computer science or equivalent experience
Minimum 3 years of experience in Data warehouse/ETL projects
Deep understanding of Star and Snowflake dimensional modelling
Strong knowledge of Data Management principles
Proficiency in Databricks Data & AI platform and Databricks Delta Lake Architecture
Handson experience in SQL Python and Spark (PySpark)
Experience with AWS/Azure stack and ETL with batch and streaming (Kinesis)
Building ETL/data warehouse transformation processes
Familiarity with Apache Kafka for use with streaming data/eventbased data
Working knowledge of Hadoop (incl. Hive Pig Impala) and nonrelational/NoSQL data repositories (incl. MongoDB Cassandra Neo4J)
Experience with structured and unstructured data including imaging & geospatial data
Proficiency in RDBMS complex SQL PL/SQL Unix Shell Scripting performance tuning and troubleshooting
Databricks Certified Data Engineer Associate/Professional Certification (Desirable)
Comfortable working in a dynamic fastpaced innovative environment with several ongoing concurrent projects
Experience working in Agile methodology
Strong verbal and written communication skills
Strong analytical and problemsolving skills with a high attention to detail
etl,circleci,rdbms,mongodb,hadoop,pyspark,pl/sql,databricks data & ai platform,azure,azure databricks,git,data warehouse,databricks delta lake architecture,sql,aws,pig,impala,apache kafka,unix shell scripting,data management,spark,neo4j,extract, transform, load (etl),hive,microsoft azure,azure datafactory,cassandra,python,pipelines,terraform