Senior Data Engineer

Senior Data Engineer Hybrid- Minneapolis, MN Hourly Pay Rate Range $70-75/Hour This project supports the firm’s Fixed Income organization, which currently relies on highly manual, Excel-based workflows to capture, maintain, and analyze deal and market data. These processes are time-intensive, introduce potential data quality risks, and limit the ability to leverage advanced automation and AI-driven insights. The initiative focuses on modernizing this ecosystem by developing a scalable full-stack application that integrates seamlessly with Databricks and internal data platforms. The solution will incorporate LLM-powered capabilities to streamline data ingestion, enhance data quality, surface actionable insights, and enable intelligent, multi-agent workflows. Unstructured Data Platform • Design and implement pipelines to ingest, parse, and transform unstructured data — PDFs, earnings transcripts, deal documents, research reports, and pitch materials • Build scalable document processing workflows on Databricks using Delta Lake, AutoLoader, and Spark-based NLP/embedding pipelines • Integrate with vector stores (e.g., LanceDB, Pinecone, Databricks Vector Search) to support RAG architectures and semantic search • Partner with AI engineers to prep and ground document corpora for LLM-powered applications Genie Spaces • Architect and deploy Databricks Genie Spaces to enable natural language querying across deal, financial, and client data • Define and curate certified datasets, semantic models, and trusted metrics layers to ensure accuracy of AI-generated answers • Collaborate with business stakeholders to identify high-value use cases and onboard end users to Genie-powered analytics • Monitor and tune Genie Space performance, model grounding quality, and response fidelity API & Integration Layer • Build and maintain REST and Model Serving APIs on Databricks to expose data and ML model outputs to downstream applications • Integrate Databricks Model Serving endpoints with Azure API Management for governance, rate limiting, and routing • Develop robust Python/FastAPI microservices and connectors linking Databricks assets to Cosmos DB, DealCloud, and Azure services • Ensure API reliability, observability, and versioning standards across the platform Required Qualifications • 5+ years of data engineering experience with at least 2 years in Databricks (Unity Catalog, Delta Live Tables, Workflows) • Hands-on experience with unstructured data processing — text extraction, chunking strategies, embedding pipelines • Proficiency in PySpark, Python, and SQL; experience with Databricks SQL and serverless compute • Working knowledge of Databricks Genie Spaces or equivalent NL-to-SQL/AI BI tools • Experience building and deploying REST APIs; familiarity with FastAPI or equivalent frameworks • Familiarity with vector databases and RAG pipeline design • Strong understanding of data modeling, lineage, and governance best practices Preferred Qualifications • Experience integrating Databricks with Azure services (ADLS, API Management, Azure AI Foundry, Cosmos DB) • Exposure to LLM orchestration frameworks (LangChain, DSPy, LlamaIndex) • Background in financial services, investment banking, or professional services data environments • Familiarity with DealCloud, S&P Capital IQ, or PitchBook data structures • Databricks Certified Data Engineer Associate or Professional certification About Us: At Brooksource, relationships are the foundation of everything we do. Since 2000, we’ve built lasting partnerships with clients, consultants, and internal teams to deliver an exceptional experience across every engagement. As a trusted IT and Engineering services provider, Brooksource supports Fortune 500 organizations through Experience-Driven Staffing, Professional Services, and Elevate, our proprietary Workforce Transformation program. Whether you're hiring for software development, cloud computing, cybersecurity, data analytics, or enterprise IT, our customized staffing solutions are designed to align with your company’s unique goals, culture, and technology stack. We offer flexible hiring models, including contract, contract-to-hire, and direct placement to meet your evolving business needs. We are a certified partner of leading platforms, including Salesforce, AWS, Microsoft, and Google Cloud, enabling us to deliver scalable, end-to-end technology solutions. With a growing national footprint, Brooksource is redefining expectations in IT consulting, engineering services, and technology workforce solutions. EEO Statement: Brooksource is an equal opportunity employer that does not discriminate on the basis of actual or perceived race, color, creed, religion, national origin, ancestry, citizenship status, age, sex or gender (including pregnancy, childbirth, lactation and related medical conditions), gender identity or gender expression, sexual orientation, marital status, military service and veteran status, physical or mental disability, protected medical condition as defined by applicable state or local law, genetic information, or any other characteristic protected by applicable federal, state, or local laws and ordinances.