Sr GCP Data Engineer

Position : Sr. GCP Data Engineer

Location : Remote

FTE salary: $145K/Annum Tech M Standard benefits

We are seeking a highly skilled GCP Data Engineer to join our team. The ideal candidate will have extensive experience with Google Cloud Platform (GCP) services particularly Dataflow BigQuery Pub/Sub and related services. Additionally the candidate should be proficient in provisioning infrastructure on GCP using Infrastructure as Code (IaC) with Terraform. This role will be critical in building and managing data pipelines ensuring data quality and optimizing data storage and retrieval processes.

Responsibilities:

Design develop and maintain data pipelines using GCP services such as Dataflow BigQuery and Pub/Sub.

Implement and manage data warehouse solutions using BigQuery ensuring data is stored securely and efficiently.

Use Pub/Sub for realtime data ingestion and streaming analytics.

Provision and manage GCP infrastructure using Terraform ensuring best practices in IaC are followed.

Optimize data storage and retrieval processes to enhance performance and reduce costs.

Monitor and troubleshoot data pipeline issues ensuring high availability and reliability of data services.

Ensure data quality and integrity through robust testing and validation processes.

Stay updated with the latest GCP features and best practices integrating them into existing workflows.

Document data workflows infrastructure setups and processes for future reference and knowledge sharing.

Requirements:

Proven experience as a Data Engineer with a focus on GCP services.

Strong proficiency in GCP services such as Dataflow BigQuery and Pub/Sub.

Handson experience with Terraform for provisioning and managing GCP infrastructure.

Proficiency in SQL and Python for data manipulation and analysis.

Solid understanding of data warehousing concepts and ETL processes.

Experience with realtime data processing and streaming analytics.

Strong problemsolving skills and attention to detail.

Excellent communication and collaboration skills.

Preferred Qualifications:

Google Cloud Professional Data Engineer certification.

Experience with other GCP services like Cloud Storage Cloud Functions and Cloud Composer.

Familiarity with other IaC tools such as Ansible or Cloud Deployment Manager.

Experience with containerization technologies like Docker and Kubernetes.

Knowledge of data security and governance practices.

particularly Dataflow,BigQuery,Pub/Sub,and related services. Additionally,ensuring data quality,develop,and maintain data pipelines using GCP services such as Dataflow,BigQuery,integrating them into existing workflows. Document data workflows,infrastructure setups,BigQuery,Cloud Functions