01Responsibilities
-
Build and maintain multi-cloud Databricks infrastructure using Terraform and Databricks functionality across mostly AWS, and GCP.
Contribute to Unity Catalog administration including general best practices, metastores, workspaces, catalog, schema infrastructure and Databricks Asset Bundles.
Help build deployment accelerators, templates, and self-service tooling so other teams can onboard and ship on their own on Databricks platform.
Act as a platform engineer, building self-service tools and 'golden paths' for other teams, and actively contribute to community enablement through documentation, training, and cross-team knowledge sharing.
Required Skills
Cloud & Infrastructure:
5 to 8 years of software engineering experience, ideally in infrastructure or data platform roles.
Proficiency in Python/PySpark and strong knowledge of SQL.
Experience with Terraform or similar Infrastructure-as-Code (IaC) tools.
Strong knowledge of at least one cloud platform (GCP/AWS)
Solid CI/CD experience.
Data & MLOps Expertise:
Strong Background in Databricks, specifically Unity Catalog, medallion architecture, and Databricks Asset Bundles (DABs).
Proficiency in Python and API frameworks such as FastAPI or Flask.
Strong experience designing, developing, and maintaining RESTful APIs.
Valuable understanding of data engineering concepts such as data modeling and workflow orchestration.
Experience with MLOps and Data Engineering like having worked on Spark/Delta Lake/MLflow/ETL frameworks (e.g., Fivetran, Lakeflow Connect).
Operational Excellence:
Experience with Github workflows, including container versioning.
Proficiency with incident management and observability tools (e.g., PagerDuty, system monitoring).
Mindset & Collaboration:
An AI as a partner mindset.
A collaborative approach, including asking questions, navigating unclear requirements, and actively contributing to knowledge sharing/community enablement. .
