Senior Data Engineer Consultant (AI-Forward, Kafka + Databricks)

Job post summary Date posted: September 24, 2025 Pay: $75.00 - $105.00 per hour Job description: Who We Are Proactive Logic Consulting INC is a boutique technology consulting firm specializing in: Innovation, zero to one projects Assessments & Roadmaps Process Automation and AI Azure Cloud Migrations App Modernization (including Low-Code solutions) We bring together top-tier independent consultants—experts in their craft who thrive on flexibility and autonomy. Our consultants have a strong customer-obsessed mindset, high EQ, and deep technical skills. We’re entrepreneurial at heart and fully support individuals who are simultaneously building their own product ventures. What We’re Looking For We need an experienced Senior Data Engineer Consultant (AI-Forward, Kafka + Databricks) who can design scalable streaming and batch data platforms—and integrate Large Language Models (LLMs) and agentic workflows into real analytics and data operations. This role is a corp-to-corp, 1099 capacity. If you deliver high-quality outcomes, work independently, and leverage AI tools to multiply impact, we’d love to talk. Key Requirements 10+ Years in Data Engineering Building and operating reliable data platforms, ELT/ETL pipelines, and lakehouse architectures. Strong data modeling (medallion/bronze-silver-gold, dimensional/star schema, semantic layers). Kafka Expertise High-throughput, fault-tolerant streaming design (partitions, consumer groups, idempotency, EOS semantics). Kafka Streams/Flink/Spark Structured Streaming; Connectors & Schema Registry; CDC (e.g., Debezium). Confluent Cloud or self-managed; security (SASL/SCRAM, mTLS), governance, and capacity planning. Databricks / Spark PySpark/Scala/Spark SQL; Delta Lake, Delta Live Tables, Structured Streaming, MLflow, Unity Catalog. Job orchestration (Workflows), cluster sizing/tuning (incl. Photon), CI/CD, cost optimization. Azure Data Platform Azure Databricks, ADLS Gen2, Event Hubs (Kafka), Data Factory/Synapse pipelines, API Management, Key Vault. Azure DevOps/GitHub, IaC (Terraform/Bicep), Observability (App Insights/Log Analytics/OpenTelemetry). AI-Forward Data Engineering Daily practitioner with AI coding tools and agents: Claude Code, OpenAI Codex CLI (and similar). Prompt and instruction design; translate business goals into structured prompts, tools, and context. LLM integration for data: NL-to-SQL, documentation/enrichment, catalog search, semantic layers, and RAG. Embeddings and vector search (Azure AI Search, pgvector, Pinecone or equivalent) over lakehouse/catalog. Agentic workflows for ops: incident runbooks, pipeline triage, schema-change diffing, and code reviews with human-in-the-loop. Safety/compliance: PII/PHI handling, redaction, guardrails/moderation, data residency, secrets (Key Vault), HIPAA alignment. Measurement & cost: evals, latency/throughput tradeoffs, token/cost tracking, model/version pinning and resilient fallbacks. Governance & Quality Unity Catalog, lineage (OpenLineage/DataHub), data quality (Great Expectations/Deequ), SLAs/SLOs. Healthcare/Pharmacy Domain (Preferred) Experience in healthcare or pharmacy; understanding of HIPAA and PHI constraints. Consulting & EQ Excellent client-facing communication and requirements discovery; outcome-focused and pragmatic. Entrepreneurial Mindset Think like an owner; innovate and deliver measurable value. Business Structure Must have an active LLC or S-Corp; 1099 Only (Corp-to-Corp). What You’ll Do Architect & Design: Lakehouse and streaming architectures on Azure with Kafka + Databricks. Build Streaming & Batch: Kafka ingestion (incl. CDC), DLT pipelines, ELT to Delta, curated gold datasets. AI-Integrated Data: Implement NL-to-SQL, semantic catalog search, RAG over documentation/metadata, and automated doc generation. Operate & Optimize: Establish observability, data quality checks, cost/latency budgets, and performance tuning. Govern & Secure: Apply Unity Catalog policies, lineage, masking/row-level security, and HIPAA-aligned practices. Client Engagement: Translate business needs into data roadmaps; present trade-offs in cost, performance, and accuracy. Continuous Improvement: Track model/platform updates; evolve data and AI architecture as capabilities improve. Preferred Tech & Tooling Kafka, Confluent Cloud, Kafka Connect, Schema Registry, Debezium, Flink (plus) Databricks: Spark, Delta Lake, Delta Live Tables, Workflows, Unity Catalog, MLflow, Repos/dbx, SQL Orchestration: ADF/Synapse Pipelines, Databricks Workflows, Airflow Data Quality & Lineage: Great Expectations, Deequ, OpenLineage, DataHub, Azure Purview Azure: ADLS Gen2, Event Hubs (Kafka), Functions, API Management, Key Vault, App Insights/Log Analytics LLM platforms: Azure OpenAI, OpenAI, Anthropic; vector: Azure AI Search, pgvector, Pinecone Orchestration libs: Semantic Kernel, LangChain/LangChain4j, LlamaIndex Agentic tooling & IDEs: Claude Code, OpenAI Codex CLI, GitHub Copilot/Cursor (or similar) Why Join Proactive Logic Consulting? Flexible Engagements: Autonomy in how you work and deliver. Top-Tier Peers: Senior experts in cloud, data, and AI. Impactful Projects: Healthcare and modernization with measurable business outcomes. AI-Forward Culture: Operate at the leading edge of agentic data engineering. Entrepreneurial Community: Connect with builders and founders. How to Apply If this describes your ideal consulting engagement and you meet the qualifications, please send: Your resume or consultant profile highlighting data engineering, Kafka, Databricks, and AI/LLM experience. A brief description of your LLC or S-Corp (name, location). Links to data/AI-forward work (streaming pipelines, notebooks, DBX repos, MLflow runs, or technical write-ups). A short statement on how you use AI tools (Claude Code, OpenAI Codex CLI, etc.) to accelerate delivery while maintaining quality and safety. Job Type: Contract Compensation Package: 1099 contract Experience: Data engineering: 10+ years (Required) Kafka (Confluent or equivalent): 4+ years (Required) Databricks/Spark: 3+ years (Required) AI/LLM integration in data workflows: 1–2+ years (Preferred) Consulting: 5+ years (Required) Work Location: Remote Job Type: Contract Pay: $75.00 - $105.00 per hour Expected hours: 20 – 40 per week Work Location: Remote