Senior Data Engineer (Databricks / PySpark)

Skills Required Primary Skills Databricks and PySpark; Advanced SQLmedallion/lakehouse; Data vault 2.0 Job Description Work Location: Remote (India) Candidates should be flexible to work from an EXL office whenever business requires. While the role is primarily remote, it is not a permanent work-from-home position . Key Responsibilities Design, build, and productionise batch and incremental ETL/ELT pipelines using Databricks (notebooks, Jobs/Workflows, DLT), PySpark, and Spark SQL across bronze/silver/gold Delta Lake layers. Implement the medallion architecture with Delta Lake features ACID merges/upserts (SCD1/SCD2), schema evolution/enforcement, time travel, OPTIMIZE/Z-ORDER, vacuuming and manage data with Unity Catalog. Orchestrate workloads via Databricks Workflows and/or Azure Data Factory; parameterise pipelines, implement restartability, checkpointing, and idempotent re-runs. Tune Spark for performance and cost: partitioning strategy, join/broadcast optimisation, skew handling, caching, cluster sizing/autoscaling, Photon usage, and job-level cost monitoring. Build data quality checks (expectations, reconciliation against source, row/measure-level controls) and implement failure alerting, logging, and lineage-friendly design. Ingest from diverse sources RDBMS (SQL Server/Oracle), files (CSV/Parquet/JSON/XML), APIs, and streaming/CDC feeds into ADLS Gen2 landing zones. Contribute to CI/CD for data: Git-based development, pull-request reviews, Azure DevOps/GitHub Actions pipelines, environment promotion, and infrastructure/config as code. Collaborate with Data Modelers, BDAs, testers, and onshore leads in Agile ceremonies; convert mapping specifications into robust, reviewed code; mentor junior engineers. Provide L3 production support for owned pipelines: triage incidents, perform root-cause analysis, and drive permanent fixes within SLAs. Must-Have Skills & Experience 9 12 years overall; strong recent hands-on Databricks and PySpark delivery experience (3+ years Databricks preferred). Expert-level Spark (DataFrame API, Spark SQL, window functions, UDF trade-offs) and advanced Python for data engineering. Advanced SQL complex transformations, performance tuning, analytical/window queries on large volumes. Delta Lake internals and medallion/lakehouse architecture in production. Azure data stack: ADLS Gen2, Azure Data Factory, Key Vault, and Databricks administration basics (clusters, pools, jobs, secrets). Data warehousing concepts: star schemas, fact/dimension loading patterns, SCD handling, reconciliation. Git-based SDLC with CI/CD exposure (Azure DevOps or GitHub). Insurance or financial-services data experience ideally P&C (policy / claims / premium / reinsurance structures). Good-to-Have Delta Live Tables, Unity Catalog governance, Databricks Asset Bundles. Streaming (Structured Streaming, Auto Loader, Event Hubs/Kafka) and CDC tools. dbt on Databricks; Airflow; Terraform. Guidewire, Duck Creek, or London-market source systems; Snowflake or Synapse coexistence. Databricks Data Engineer Associate/Professional certification. Qualifications Bachelor s degree in Computer Science, Engineering, or equivalent practical experience. Professional & Communication Skills Clear written and spoken English; comfortable presenting design decisions to onshore architects and client stakeholders. Proven offshore-delivery discipline: crisp status reporting, proactive risk flagging, dependable overlap-hours availability. Ownership mindset drives issues to closure without follow-up. Disclaimer: This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying. .