prachi.chowhan@nytpcorp.com
Job Title – Sr. Data Engineer – MS Fabric, Azure
Location: Chicago, IL (2 days onsite in a week)
No H1B
Core Responsibilities
- Design and build end-to-end data platforms using Microsoft Fabric
o Lakehouse, Warehouse, OneLake, Dataflows Gen2
- Develop and optimize Spark workloads using PySpark and SparkSQL
- Develop MLOps pipelines for Advanced Analytics & AI
- Build scalable ETL/ELT pipelines using:
o Azure Data Factory (ADF)
o MS Fabric Data pipeline
o Dataflow gen 2
o SSIS (on-prem, Azure-SSIS IR, and migration scenarios)
- Implement data modeling patterns:
o Medallion (Bronze / Silver / Gold)
o Dimensional modeling (Star/Snowflake)
o Different data file management experience – Parquet, JSON, XML
- Integrate Microsoft Purview for:
o Data cataloging & classification
o Automated data lineage (ADF, Fabric, SQL, ADLS)
- Enforce data security and access controls:
o RBAC, column-level security, masking
o Fabric & Purview policy alignment
- Optimize performance, reliability, and cost across Fabric capacities
- Implement CI/CD and IaC for data pipelines and governance artifacts
- Partner with security, compliance, and BI teams to ensure trusted data delivery
Required Technical Skills
Microsoft Fabric (Must-Have)
- Fabric Data Engineering workloads
- Lakehouse & Warehouse
- OneLake architecture
- Fabric pipelines & notebooks
- Capacity planning and performance optimization
- Advanced PySpark (joins, windows, UDFs, optimization)
- Strong SparkSQL
- Strong MLOps & Feature Engg.
- Partitioning strategies, shuffle tuning, caching
- Large-scale data processing (TB+)
Azure Data Platform
- Azure SQL Database / SQL Server
- Azure Data Factory (ADF)
- SSIS / Azure-SSIS Integration Runtime
- ADLS Gen2
Data Lineage & Security (Microsoft Purview)
- Purview data catalog & scanning
- Automated lineage across ADF, Fabric, SQL, ADLS
- Business glossary management
- Integration with Azure RBAC & security policies