Title: Azure Data Lakehouse Engineer with Iceberg Location: Raritan, NJ (Hybrid) Contract Job Description: Apache Iceberg Architecture & Engineering • Design and implement Iceberg table schemas and partitioning strategies optimized for performance, scalability, and query flexibility in petabyte-scale Lakehouse environments • Architect Iceberg catalog layers: Hive, AWS Glue, Nessie, or REST-based catalogs enabling ACID-compliant operations and consistent metadata propagation across multiple engines • Operationalize transactional inserts, upserts, deletes, and merges using Spark, Flink, or native Iceberg APIs supporting both streaming and batch workloads • Manage schema evolution, partition evolution, and snapshot lifecycle policies to optimize long-term performance and storage • Build custom tools or wrappers around Iceberg APIs in Java, Scala, or Python for metadata introspection, compaction automation, and lineage integration Multi-Engine Compatibility and Optimization • Enable multi-engine interoperability across Spark, Trino, Dromio, Flink, Snowflake, and Presto ensuring Iceberg tables are accessible, performant, and consistent regardless of compute layer • Collaborate with BI and ML teams to design read-optimized layouts for consumption by Power BI, Tableau, or notebooks while enforcing data freshness SLAs • Design table format version upgrade paths and coordinate with platform teams to validate compatibility across engines, connectors, and query runtimes Governance, Metadata & Observability • Integrate Iceberg tables with data catalogs (e.g., Unity Catalog, Alation, Amundsen) and data lineage platforms (e.g., Marquez, Collibra, DataHub) • Enforce data governance via column-level lineage, row-level ACLs, and policy-based retention rules leveraging metadata-rich Iceberg features • Automate compaction, snapshot expiration, and manifest pruning using orchestration tools (e.g., Apache Airflow, dbt, Dagster) for metadata hygiene • Implement Iceberg table telemetry using Prometheus-Grafana or integrate with enterprise observability tools for query stats, scan efficiency, and access patterns Automation, Scaling & Cloud Integration • Build IaC-driven infrastructure using Terraform, CloudFormation, or Pulumi to deploy Iceberg-backed Lakehouse components across AWS, Azure, or GCP • Automate Iceberg table provisioning pipelines as part of domain-specific data product deployments aligned to Data Mesh architecture • Optimize compute-storage configurations in S3, Azure Data Lake Gen2, or GCS to support efficient Iceberg table layouts, object sizing, I/O patterns, and partition pruning • Enable streaming ingestion: Spark Structured Streaming to land real-time CDC or log-based feeds