01Overview
Snowflake and Pyspark Developer (Data Engineer)
Section I - Overview
This is for a Development project using Snowflake and AWS.
Technologies: Snowflake, PYSPARK, AWS Glue, Snowpipe, Snowpark using Python.
Job Title: Snowflake and Pyspark Developer (Data Engineer)
Coding: Mandatory
Experience Level:
- Min: 4 years
- Max: 8 years
Location: Hyderabad/Bangalore/Pune
Work Type: Full-time
Section II - Job Evaluation
Evaluation based on the following topics:
Topic of Evaluation / Skills Mandatory / Non-mandatory Percentage Snowflake, PYSPARK, AWS Glue, Snowpipe, Snowpark using Python Mandatory (Not Specified) SQL programming Mandatory (Not Specified) (Note: Percentage values should sum up to 100%)
Section III - Job Requirements and Responsibilities
Job Requirements
Must-Have
- Proficient in SQL programming (stored procedures, user defined functions, CTEs, window functions), Design and implement Snowflake data warehousing solutions, including data modelling and schema designing Snowflake
- Able to source data from APIs, data lake, on premise systems to Snowflake.
- Must have minimum 2 yrs experience in AWS Glue
- Process semi structured data using Snowflake specific features like variant, lateral flatten
- Experience in using Snowpipe to load micro batch data.
- Experience in Snowpark using python.
- Good knowledge of caching layers, micro partitions, clustering keys, clustering depth, materialized views, scale in/out vs scale up/down of warehouses.
- Hands-on in creating reusable pipelines for handling SCD type1, type2 loads.
- Ability to implement data pipelines to handle data retention, data redaction use cases.
- Proficient in designing and implementing complex data models, ETL processes, and data governance frameworks.
- Strong hands on in migration projects to Snowflake
- Deep understanding of cloud-based data platforms and data integration techniques.
- Skilled in writing productive SQL queries and optimizing database performance.
- Ability to develop and imple .
