GCP Data Engineer

Overview

Key Responsibilities

• Design and develop scalable data pipelines on GCP using tools such as Dataflow Apache Beam or Apache Spark.
• Implement and maintain data storage solutions on GCP including BigQuery Cloud Storage and Cloud Bigtable.
• Collaborate with data scientists and analysts to understand data requirements and develop solutions that enable efficient data analysis and modeling.
• Optimize data infrastructure for performance reliability and costefficiency.
• Ensure data security and compliance with regulatory requirements.
• Monitor and troubleshoot data pipeline and infrastructure issues.
• Document data architecture processes and best practices.
• Conduct performance tuning capacity planning and infrastructure scaling.
• Stay updated with GCP tools and technologies and make recommendations for continuous improvement.
• Implement automation for data infrastructure provisioning and maintenance.
• Collaborate with crossfunctional teams to support various datarelated initiatives.
• Provide technical guidance and mentorship to junior team members.
• Participate in oncall support as needed.
• Contribute to the overall data strategy and roadmap.

Required Qualifications

• Bachelors or Masters degree in Computer Science Engineering or a related field.
• Proven experience working as a data engineer or similar role.
• Expertise in building and optimizing data pipelines for big data processing.
• Proficiency in GCP services like BigQuery Dataflow Cloud Storage Pub/Sub and Data Proc.
• Strong programming skills in languages such as Python Java or Scala.
• Experience with data modeling ETL development and data warehousing concepts.
• Deep understanding of distributed computing and parallel processing.
• Excellent SQL skills for querying and data manipulation.
• Knowledge of data orchestration and workflow management tools like Apache Airflow.
• Ability to work in a fastpaced collaborative and agile environment.
• Strong problemsolving and analytical abilities.
• Good communication skills and the ability to work effectively with stakeholders across the organization.
• Relevant GCP certifications would be a plus.
• Understanding of machine learning and AI concepts would be advantageous.
• Experience with containerization and orchestration platforms like Kubernetes is a plus.

gcp,data engineering,big data,sql,python