Spark Framework Engineer – Enterprise Data Platform

Outstanding contract opportunity! A well-known Financial Services Company is looking for an Ansible Automation Engineer (with Load Balancing experience) in Columbus or Charlotte (hybrid 3 days on-site). If you're passionate about advancing automation and working with cutting-edge load balancing technology within a collaborative, non-egotistical team environment, this is for you! Contract Duration: 12 Months We are seeking a Lead Software Engineer to drive the evolution of SparkFlow, an enterprise data processing framework built on Apache Spark. This role will focus on designing clean, composable abstractions, delivering new functional capabilities within the framework, and significantly improving developer experience. The engineer will also design and deploy AI enabled capabilities and lead the integration of the SparkFlow framework into the Unity control plane. Key Responsibilities • Lead the architectural evolution of the SparkFlow framework, emphasizing modularity, composability, and clear abstraction boundaries • Design and implement new functional features within the framework to expand supported use cases and capabilities • Balance feature delivery with framework quality, ensuring performance, backward compatibility, and governance standards • Improve developer experience by simplifying APIs, configuration models, extension points, and onboarding patterns • Design and implement AI driven solutions that assist developers and enhance framework usability, productivity, and operability • Integrate SparkFlow as a first class framework within the Unity control plane • Collaborate with data, platform, and infrastructure teams to ensure consistent adoption across environments • Provide technical leadership through design reviews, code reviews, and architectural guidance Required Experience • Deep experience with Apache Spark (Scala and/or Java; Python a plus) • Proven success building reusable frameworks or libraries, including ownership of functional feature development • Strong skills in API design, abstraction design, and extensibility patterns • Experience applying AI/ML or LLM based techniques to developer tooling or data frameworks • Experience working in large?scale, enterprise data environments Nice to Have • Experience integrating frameworks into control planes or internal developer platforms • Familiarity with Spark SQL–centric architectures and data governance patterns • Experience modernizing or replacing legacy ETL frameworks Posted By: Natalie DeWitt