Senior Data Engineer

The position is part of a data-focused team working on the “T-Bar” (Transaction Books and Records) product, a critical data store for positions, transactions, and tax lots. The team is transitioning from a legacy data center to a cloud-native AWS environment, aiming to modernize infrastructure and processes by next summer. Senior Engineer Must-Have Requirements • Experience with Batch and Streaming Data Processing: Ability to handle intraday use cases with trading partners, including micro-batches. • CI/CD and Developer Discipline: Proficiency in code commit practices, including tying commits to Jira tickets for traceability and cherry-picking. • AWS Familiarity: Experience working within AWS environments, particularly with data pipelines and foundational data layers. • Python: Strong skills in Python for data processing and logic conversion (e.g., rewriting stored procedures from legacy systems). • Infrastructure as Code (IaC): Experience with Terraform for deploying and managing infrastructure, particularly for carving out separate AWS accounts. Nice-to-Have Skills • Orchestration Tools: Familiarity with Airflow, Dagster, or similar tools for centralized orchestration (current setup includes step functions, Eventbridge, and an in-house eventing system). • AI/ML Integration: Exposure to AI tools like GitHub Copilot or Cursor for code development, unit test generation, or data quality checks (e.g., anomaly detection). • Data Governance: Experience with data catalogs, producing/consuming assets, or tools like AWS DataZone for governance and contract-based testing. • Event-Driven Architecture: Understanding of event-driven systems, as the team aims to move toward this model in the future. #DICE