Data Engineer with Databricks

We are seeking a highly skilled Data Engineer with strong hands-on experience in Databricks to build, optimize, and support enterprise-grade data pipelines. The ideal candidate will be an expert in scalable data engineering patterns, modern lakehouse architecture, and distributed data processing. This role is critical to expanding and maturing our data ecosystem, ensuring high-quality data availability for analytics, operations, and downstream applications. Key Responsibilities Data Engineering & Pipelines

· Build scalable, production-grade ETL/ELT pipelines using Databricks (PySpark, Spark SQL, Delta Live Tables, Workflows).

· Ingest structured, semi-structured, and streaming data into Bronze, Silver, and Gold layers.

· Develop optimized transformations, data quality rules, and reusable framework components.

· Strong usage / proven experience of leveraging inbuilt AI Genie code for faster delivery.

· Implement best practices for job orchestration, monitoring, alerting, and automation.

· Hands-on experience: Spark, Delta Lake, Workflows, Unity Catalog.

· Strong SQL programming and performance tuning skills.

· Experience with cloud environments (AWS/Azure/GCP).

· Experience with modern data lakehouse concepts and distributed systems.

Data Modeling:

· Design and implement dimensional models (star/snowflake) for analytical workloads.

· Apply normalization/denormalization strategies for performance and usability.

· Create physical data models aligned to Delta Lake and Medallion architecture.

· Ensure data quality, integrity, and alignment with enterprise governance policies.

Technology Stack:

· Advanced hands-on Databricks experience: Spark, Delta Lake, Workflows, Unity Catalog.

· Strong SQL programming and performance tuning skills.

· Experience with cloud environments (AWS/Azure/GCP).

· Experience with modern data lakehouse concepts and distributed systems.

Required Qualifications:

· 7–10+ years of experience in data engineering.

· Proven experience building and deploying large-scale Databricks pipelines.

· Strong understanding of Medallion architecture (Bronze/Silver/Gold).

· Proficiency with PySpark, SQL, ETL/ELT frameworks, and Delta Lake optimizations.

· Strong experience working with CI/CD, Git, and job orchestration tools.

· Deep understanding of data modeling principles for warehousing and analytics.

Preferred Skills:

· Knowledge of data governance, metadata management, and Unity Catalog.

· Experience with streaming technologies (Auto-Loader, Structured Streaming).

· Background in data security, compliance, and access models.

· Knowledge of performance tuning and cost optimization in cloud environments.

· Experience with tools like Airflow, Databricks Workflows, dbt, or similar.

Role Descriptions:

Skills – Databricks (Python and SQLPySpark) – Azure SQL Server (Managed Instance| Azure SQL DB| SQL Server VMs) – Data modeling and ETLELT design patterns – Docker and Azure Kubernetes Service (AKS) for select automation – Performance optimization and scalabilityExperience – 7 years enterprise data engineering – Healthcare data integration experience strongly preferred – Experience with large-scale data harmonization and normalization