01Responsibilities
Design, develop, and implement scalable ETL/ELT pipelines for extracting, transforming, and loading consumer data from various sources (e.g., CRM, marketing platforms, DCM, GA4, digital channels), with Databricks/AWS as the primary execution platform.
Develop and manage end-to-end solutions on Databricks including Unity Catalog, Delta Live Tables, Databricks Workflows, Databricks SQL, etc.; own platform governance covering schemas, permissions, and data lineage.
Design multi-hop lakehouse architectures (Bronze / Silver / Gold) using Delta Lake; optimize Spark compute, cluster configurations, and Auto Loader for performance and cost efficiency.
Leverage AWS data services S3, Glue, Lambda and Redshift in conjunction with Databricks to build reliable, end-to-end consumer data flows.
Architect and optimize data models and schemas to support complex analytical queries and reporting requirements related to consumer behaviour, preferences, and engagement.
Publish semantic layers (metrics definitions, certified datasets, business logic) consumed by downstream BI tools and AI agents; build and deploy agentic workflows using Databricks AI Functions or similar frameworks.
Ensure data quality, integrity, and governance across all consumer data assets by implementing validation rules, schema evolution controls, and monitoring processes through Unity Catalog.
Collaborate with data scientists, business analysts, and marketing teams to understand data needs and translate them into technical data engineering solutions; partner to productionize ML models and feature stores on Databricks.
Implement automation for data ingestion, processing, and delivery with a focus on efficiency, reliability, and SLA adherence.
Troubleshoot and resolve data-related issues, performing root cause analysis and implementing corrective actions.
Stay current with Databricks platform updates, AWS data services, and emerging best practices in data engineering and AI-driven analytics.
Job Qualifications:
Bachelor's or Master's degree in Computer Science, Engineering, Information Technology, or a related quantitative field.
3-8 years of data engineering experience with hands-on production experience on Databricks.
Deep knowledge of Databricks platform architecture Unity Catalog, Delta Lake, Databricks Workflows, Databricks SQL, and cluster/compute management. (Preferred)
Experience designing lakehouse architectures (medallion/multi-hop patterns) at scale and with ETL/ELT orchestration tools. (Preferred)
Experience with AWS data stack: S3, Glue, Redshift, Lambda, and IAM.
Strong proficiency in Python and PySpark; advanced SQL for complex analytical workloads.
Familiarity with Software Development Life Cycle (SDLC) practices including version control (Git), CI/CD pipelines, code reviews, and agile development methodologies. (Preferred)
Strong understanding of data warehousing concepts, dimensional modeling, and data governance principles.
Hands-on experience building semantic layers (e.g., dbt metrics, Databricks AI/BI semantic layer) and creating AI agents or agentic pipelines (Databricks AI Functions etc.) .
Excellent problem-solving, communication, and stakeholder collaboration skills.
Lilly is dedicated to helping individuals with disabilities to actively engage in the workforce, ensuring equal opportunities when vying for .
