01Overview
Candidates
ready to join immediately
can share their details via email for quick processing.
CCTC | ECTC | Notice Period | Location Preference
Act fast for immediate attention
Key Responsibilities
- Design, develop, and maintain scalable
data pipelines
using
PySpark
- Build and manage
batch and real-time data processing
systems
- Develop and integrate
Kafka-based streaming solutions
- Optimize Spark jobs for performance, cost, and scalability
- Work with cloud-native services to deploy and manage data solutions
- Ensure data quality, reliability, and security across platforms
- Collaborate with data scientists, analysts, and application teams
- Participate in code reviews, design discussions, and production support
Must-Have Skills
- Strong hands-on experience with
PySpark / Apache Spark
- Solid understanding of
distributed data processing
concepts
- Experience with
Apache Kafka
(producers, consumers, topics, partitions)
- Hands-on experience with
any one cloud platform
:
- AWS (S3, EMR, Glue, EC2, IAM)
or
- Azure (ADLS, Synapse, Databricks)
or
- GCP (GCS, Dataproc, BigQuery)
- Proficiency in
Python
- Strong experience with
SQL
and data modeling
- Experience working with
large-scale datasets
- Familiarity with
Linux/Unix environments
- Understanding of
ETL/ELT frameworks
- Experience with
CI/CD pipelines
for data applications
Valuable-to-Have Skills
- Experience with
Spark Structured Streaming
- Knowledge of
Kafka Connect
and
Kafka Streams
- Exposure to
Databricks
- Experience with
NoSQL databases
(Cassandra, MongoDB, HBase)
- Familiarity with
workflow orchestration tools
(Airflow, Oozie)
- Knowledge of
containerization
(Docker, Kubernetes)
- Experience with
data lake architectures
- Understanding of
security, governance, and compliance
in cloud environments
- Exposure to
Scala
or
Java
is a plus
- Prior experience in
Agile/Scrum
environments .
