01Responsibilities
- Lead end-to-end design and delivery of enterprise data solutions on Databricks.
- Design and implement Medallion Architecture (Bronze, Silver, Gold layers).
- Build scalable ETL/ELT pipelines using PySpark and Spark SQL.
- Develop ingestion frameworks using Databricks Auto Loader for batch and streaming workloads.
- Design and implement Delta Lake based solutions including CDC, MERGE, Time Travel, and Schema Evolution.
- Develop data quality, validation, reconciliation, and monitoring frameworks.
- Implement orchestration using Databricks Workflows and external schedulers.
- Define data models, transformation logic, and integration patterns for insurance data platforms.
- Implement security and governance using Unity Catalog, RBAC, and data access controls.
Mandatory Skills:
- Strong expertise in Databricks, including data engineering and analytics. In-depth knowledge of the insurance domain, including industry standards and regulations.
- Proven experience in project management methodologies (Agile, Waterfall, etc.).
- Excellent leadership and team management skills. Robust analytical and problem-solving abilities.
- Exceptional communication and interpersonal skills.
Strong expertise in:
- Databricks Lakehouse Platform
- PySpark
- Spark SQL
- Delta Lake
- Auto Loader
- Databricks Workflows
- Unity Catalog
- Delta Live Tables (DLT) .
