01Overview
Key Responsibilities :
- Design, develop, and maintain scalable data pipelines and ETL/ELT workflows.
- Build and optimize data processing solutions using Databricks and cloud-based data platforms.
- Develop data integration solutions using Azure Data Factory (ADF) and other ETL tools.
- Work with structured and unstructured data from multiple data sources.
- Develop and optimize data models, ensuring data quality, integrity, and consistency.
- Collaborate with data scientists, analysts, and business teams to understand data requirements and deliver scalable solutions.
- Monitor, troubleshoot, and optimize data pipelines for performance and reliability.
- Implement data governance, security, and best practices across the data platform.
- Support data migration, transformation, and modernization initiatives.
- Participate in code reviews, testing, deployment, and documentation activities.
Required Skills :
- Strong hands-on experience with Databricks.
- Experience with Azure Data Factory (ADF) or equivalent ETL tools.
- Proficiency in Azure and/or AWS cloud platforms.
- Good understanding of Hadoop ecosystem and distributed data processing.
- Strong experience in designing and developing ETL/ELT pipelines.
- Proficiency in Python, SQL, or Scala for data engineering.
- Experience with relational and NoSQL databases.
- Knowledge of data warehousing concepts and dimensional data modeling.
- Familiarity with version control tools such as Git.
- Strong analytical, debugging, and performance optimization skills.
Preferred Skills:
- Experience with Apache Spark and PySpark.
- Knowledge of Delta Lake, Apache Kafka, or streaming data pipelines.
- Exposure to CI/CD pipelines using Azure DevOps, Jenkins, or similar tools.
- Familiarity with containerization technologies such as Docker and Kubernetes.
- Experience with data governance and cloud security best practices.
Educational Qualification:
- B.E./B.Tech in Computer Science, Information Technology, or a related discipline.
- MCA or M.Sc. in Computer Science, Information Technology, or a related field is also acceptable. .
