Sr Data Engineer

Role: Microsoft Fabric Data Lakehouse Lead / Senior Data Engineer RICEFW Technologies Inc. is seeking a senior Microsoft Fabric Data Lakehouse professional to support Bank’s enterprise-wide data modernization initiative. This role will lead the technical discovery, migration design, backlog execution, and engineering delivery needed to consolidate SQL Server–based data assets across Servicing, Originations, and Operations into a unified Microsoft Fabric and Azure-based Data Lakehouse platform. The resource will inventory all source SQL Server databases, including schemas, tables, views, stored procedures, row volumes, indexing strategies, and deprecated objects, establishing the migration source of truth. They will reverse-engineer existing T-SQL stored procedure logic, including ETL transformations, aggregations, cursor-based processing, and reporting logic, and translate that logic into equivalent PySpark notebooks or Dataflow Gen2 implementations. Stored procedures requiring redesign for performance, scalability, or maintainability will be flagged and documented. The role will involve leading requirement workshops with SQL report owners, business SMEs, and technical stakeholders to capture source-to-target mappings, transformation rules, scheduling requirements, and reporting dependencies. The resource will be responsible for producing and obtaining sign-off on Transformation Requirement Specifications and Data Mapping Documents. This position will also manage delivery execution through Azure DevOps by decomposing requirements into Epics, Features, Stories, and Tasks, defining data-grounded acceptance criteria, assigning work to Data Engineers based on skillset and dependency order, and tracking sprint progress and velocity. From an engineering standpoint, the resource will conduct pull request reviews for PySpark notebooks and pipeline JSON, validate deliverables against approved mapping documents, and escalate scope or technical blockers to the Data Lakehouse Architect. They will build reusable Spark-based ingestion templates for full-load and CDC incremental patterns from SQL Server into OneLake Delta tables using JDBC, SQL MI connector, and watermark-based change tracking. The resource will help architect the Silver transformation layer, including implementation of SCD Type 1 and Type 2 patterns through Delta MERGE, schema evolution strategy, and reusable PySpark libraries deployed as Fabric Environment custom libraries. They will also design Fabric Data Factory orchestration patterns, including master pipelines, dependency chaining, retry and error handling, parameterized multi-source SQL ingestion templates, and Azure DevOps pipeline trigger integration. The ideal candidate must have strong experience tuning Spark workloads, including custom pool sizing, Delta caching, Adaptive Query Execution, broadcast joins, partition pruning, and runtime monitoring through Fabric Monitoring Hub. The role will also enforce team engineering standards, including PySpark coding conventions, notebook parameterization, Delta naming standards, and unit test coverage through Azure DevOps PR policies.