01Overview
Cloud Data Engineer
Role Summary
This job involves creating data pipelines using a configurable framework (developed using Databricks / pyspark), which are used to load data from multiple sources to Google Cloud. It also involves extensive collaboration with other team members (India USA) to follow a plan, execute workflows, ensure data load accuracy and debug in case of issues.
Essential Responsibilities
Design, develop, and implement robust, high-performance data pipelines using Databricks/PySpark to meet diverse data integration requirements. Understand data and its relation / significance from business perspective
Leverage extensive SQL expertise for complex data manipulation, analysis, and query optimization.
Experience in Google Cloud services like Dataproc, BigQuery, Airflow, etc.
Create custom scripts in python / pyspark
Minor development and enhancement activities as needed to handle new / custom requirements
Validate data between source and target highlight issues and rerun jobs as needed
Lead other developers, review their code and ensure quality optimal code is being delivered
Provide regular updates on the tasks, status and risks to project manager
Required
Bachelors degree from a reputed university
7 to 12 years overall experience
Excellent SQL / data knowledge (6+ years) in creating complex SQL queries, Stored Procedures etc.
Experience in developing enterprise grade ETL applications (any tool)
Experience in Google BigQuery or any cloud database
Experience in Google Cloud services like Managed service for Apache Spark(Dataproc), Airflow etc.
Working knowledge in Pyspark / Python
Good communication and positive attitude
Attitude to take ownership and drive tasks without follow-ups
Good to have
Experience in Big data technologies like Hadoop, Hive, etc.
Knowledge in Iceberg, Starburst, DBT
Knowledge in US Healthcare domain
Knowledge in Agile .
