01Overview
Role Description Key Responsibilities Lead the design and architecture of scalable big data solutions using Apache Spark and Hadoop ecosystem technologies.
Own end-to-end development of batch and streaming data pipelines.
Design, optimize, and tune Spark applications for performance, scalability, and reliability.
Provide technical leadership for Hadoop components including HDFS, Hive, HBase, and YARN.
Ensure data quality, data governance, security, and compliance standards are met.
Lead real-time streaming solutions using Spark Structured Streaming and Kafka.
Review code, conduct technical design reviews, and enforce engineering best practices.
Collaborate with architects, product owners, and business stakeholders to translate requirements into technical solutions.
Mentor and guide data engineers and developers.
Support production deployments, monitoring, and incident resolution.
Drive continuous improvement, automation, and adoption of recent big data technologies.
Required Technical Skills Robust experience with Apache Spark (Spark SQL, DataFrames, Datasets, Structured Streaming).
Hands-on experience with Hadoop ecosystem tools (HDFS, Hive, HBase, YARN).
Advanced SQL and data modeling knowledge.
Experience with data formats such as Parquet, ORC, and Avro.
Knowledge of distributed systems and performance tuning.
Experience with Kafka or similar messaging platforms.
Solid programming skills in Python, Scala, or Java.
Experience with CI/CD pipelines and DevOps practices.
Cloud & Platform Experience Experience with cloud-based big data platforms such as AWS EMR/Glue, Azure Databricks/Synapse, or GCP Dataproc.
Experience migrating Hadoop workloads to cloud-based platforms.
Understanding of cloud security, cost optimization, and scalability best practices.
Skills spark,cloud computing,data quality management,databricks, .
