Senior Data Engineer

**How you'll make an impact** * Design and build production pipelines on Databricks using Spark Declarative Pipelines (SDP) and PySpark, from raw ingestion through business-ready data products. * Define all pipelines, jobs, and schedules as code in Databricks Asset Bundles, deployed to every environment through automated CI/CD. * Build data quality, monitoring, and lineage into pipelines so issues are caught and diagnosed before they reach consumers. * Turn recurring solutions into reusable frameworks, standards, and shared libraries that raise delivery speed across the team. * Own your pipelines in production — performance, cost, reliability, and incident response. * Partner with Digital Product Managers, architects, and business stakeholders to translate requirements into technical designs, and mentor engineers newer to the platform. **What you'll need (Required)** * Bachelor's degree in computer science, engineering, or a related technical field, plus five or more years of data engineering experience, including hands-on production experience on Databricks. * Demonstrated experience with Spark Declarative Pipelines (SDP / Delta Live Tables) — streaming tables, materialized views, expectations, Auto Loader, and CDC patterns. * Hands-on experience deploying Databricks workloads with Databricks Asset Bundles (DABs) across multiple environments. * Strong Spark and PySpark skills, including performance tuning, and production-quality Python beyond notebook scripting. * Working knowledge of Delta Lake, Unity Catalog, medallion architecture, and strong analytical SQL. * Experience with Git-based CI/CD in a shared repository — code review, automated validation, and promotion across environments. * Demonstrated ability to take ambiguous requirements through design to production independently, and to make and defend sound technical decisions. * "Experience with interoperable catalog architectures across Unity Catalog and Snowflake Horizon, including Iceberg REST Catalog and catalog-linked databases for cross-platform table access without data duplication." *  "Working knowledge of Apache Iceberg as a table format, including managed versus external Iceberg tables and the performance trade-offs of cross-engine reads." **What else we look for (Preferred)** * Hands-on experience with Spark Declarative Pipelines (SDP / Delta Live Tables), including streaming tables, materialized views, expectations, Auto Loader, and CDC patterns. * Hands-on experience deploying Databricks workloads with Databricks Asset Bundles (DABs) across multiple environments. * Strong Spark and PySpark development skills, including performance tuning, and production-quality Python beyond notebook scripting. * Working knowledge of Delta Lake, Unity Catalog, medallion architecture, and strong analytical SQL. * Experience with Git-based CI/CD for data platforms, including code review, automated validation, and promotion across environments. * Experience with cloud data platforms on AWS, metadata-driven ingestion frameworks, and infrastructure-as-code. * Experience with interoperable catalog architectures across Unity Catalog and Snowflake Horizon, including Iceberg REST Catalog and catalog-linked databases for cross-platform table access without data duplication. * Working knowledge of Apache Iceberg as a table format, including managed versus external Iceberg tables and the performance trade-offs of cross-engine reads. * Familiarity with the broader modern data ecosystem, such as Snowflake, dbt, Kafka, and Airflow. * Experience integrating enterprise and clinical source systems, such as Epic, SAP, or Salesforce, or migrating workloads from legacy ETL platforms onto a lakehouse. * Practical understanding of governance, quality, security, validation, and support expectations in a regulated enterprise environment. * Ability to provide technical guidance, coach team members, and contribute to reusable standards, documentation, and delivery practices.