01Key Responsibilities
Must be capable of handling existing or new Apache HDFS cluster having name node, data node & edge node commissioning & decommissioning.
Work closely with data architects and analysts to design technical solutions.
Integrate and ingest data from multiple source systems into big data environments.
Develop end-to-end data transformations and workflows, ensuring logging and recovery mechanisms.
Must able to troubleshoot spark job failures.
Design and implement batch, real-time, and near-real-time data pipelines.
Optimize Big Data transformations using Apache Spark, Hive, and Tez
Work with Data Science teams to enhance actionable insights.
Ensure seamless data integration and transformation across multiple systems.
4+ years of mandatory experience with big data.
4+ years mandatory experience in Apache Spark.
Proficiency in Apache Spark, Hive on Tez, and Hadoop ecosystem components.
Strong coding skills in Python & Pyspark.
Experience building reusable components or frameworks using Spark
Expertise data ingestion from multiple sources using APIs, HDFS, and NiFi.
Solid experience working with structured, unstructured, and semi-structured data formats (Text, JSON, Avro, Parquet, ORC, etc.).
Experience with UNIX Bash scripting and databases like Postgres, MySQL and Oracle.
Ability to design, develop, and evolve fault-tolerant distributed systems.
Strong SQL skills, with expertise in Hive, Impala, Mongo and NoSQL databases.
Hands-on with Git and CI/CD tools
Experience with streaming data technologies (Kafka, Spark Streaming, Apache Flink, etc.).
Proficient with HDFS, or similar data lake technologies
Excellent problem-solving skills you will be evaluated through coding rounds .
