01Overview
Job Description
Location: Noida, UP, India
Core Technologies
Apache Spark (Core, SQL, Structured Streaming)
PySpark
Databricks (AWS/Azure)
Advanced SQL
DevOps CI/CD
Jenkins, Git/GitHub/Bitbucket
Programming
Python, SQL
Java/Scala (Nice to have)
Cloud Infrastructure
AWS (S3, EMR, EC2, IAM, CloudWatch)/Azure- Preferred
Databricks Runtime Cluster Management- Preferred
Streaming Integration
Apache Kafka
Snowflake integration
Airflow
Perform Spark performance tuning
DAG analysis
Shuffle optimization
Partition strategies
Executor/memory tuning
Experience in financial services, regulatory reporting, or enterprise data platforms.
Handson experience in Delta Lake optimization and incremental processing strategies.
Experience with Snowflake data warehousing.
Databricks Certification (Professional level preferred).
Strong understanding of distributed computing principles.
Key Responsibilities
Design and develop scalable batch and near-real-time ETL/ELT pipelines using Databricks (AWS) and Apache Spark (PySpark, Spark SQL, Structured Streaming).
Modernize legacy SQL/Hive/stored procedure workflows into distributed Spark-native architectures.
Perform Spark performance tuning, including:
Build structured streaming pipelines using Kafka and Spark Structured Streaming.
Design dimensional data models (Fact/Dimension, SCD Type 2).
Orchestrate pipelines using Databricks Workflows / Apache Airflow.
Integrate CI/CD pipelines using Jenkins, Git, Bitbucket/GitHub for automated deployment across DEV/UAT/PROD.
Technical Leadership
Lead end-to-end solution design for data platforms using Databricks (batch, streaming, ML workloads)
Define architecture patterns like Lakehouse, Medallion (Bronze/Silver/Gold)
Act as SME for Databricks, Spark, and data engineering best practices
Team Leadership
Lead and mentor a team of data engineers (typically 5-10 members)
Conduct code reviews, enforce best practices, and ensure delivery quality
Guide team in troubleshooting complex technical issues
Stakeholder Management
Collaborate with business stakeholders, architects, and product owners
Translate business requirements into technical designs and sprint tasks
Drive technical decisions (performance vs cost vs scalability)
Delivery Governance
Own end-to-end delivery of data projects
Ensure adherence to Agile processes, SLAs, and governance models
Performdesign reviews, estimations, and risk management
Mandatory Competencies
Data Science and Machine Learning - Data Science and Machine Learning - Databricks
Cloud - Azure - Azure Data Factory (ADF), Azure Databricks, Azure Data Lake Storage, Event Hubs, HDInsight
Big Data - Big Data - Pyspark
Database - Database Programming - SQL
Data Science and Machine Learning - Data Science and Machine Learning - Python
Beh - Communication and collaboration .
