Data Engineer

Only apply if you have worked with Previously Capital One in the last 10 years for over 2+ yrs. Sr Data Engineer Project Overview Discover’s historical credit card data will be integrated into the Capital One ecosystem within Capital Ones data stream pipeline & datalake. Engineer will be responsible for: • Extracting the data from OneLake • Transforming the data • Loading it into the target state, which could be Snowflake or AWS S3. Tech Lead Expectations • Accountability: Ensure team deliverables are completed on time. • Leadership: Lead sprint ceremonies and manage team workflows. • Problem Solving: Collaborate with senior managers & others to resolve issues. • Adaptability: Pivot quickly based on changing requirements from customer/client. • Talent Level: Must be exceptionally high-caliber professionals. • Communication: Strong written and verbal skills; confidently articulate points of view. Must be able to write out their thoughts. • PySpark optimization and troubleshooting as part of performance tuning and data pipeline efficiency. • Architectural Awareness: Understand available tools and how they integrate and engage effectively with architects (nominal architecture experience required). Tech Stack: Must Have Must Have AWS(EMR/Glue, S3, Lambda, Cloudwatch), Python, Spark, Databricks, SQL, Bash) Python use: scripting, building new data pipelines (not app dev) Data warehousing experience (Snowflake) Understanding of NoSQL databases (DynamoDB) APIs Agile engineering practices & JIRA usage Nice to Have - Kafka, Airflow, Open Table formats (Delta, Hudi, Iceberg), Splunk, NewRelic, Github, Jenkins, prior Capital One experience highly preferred Secondary Skills - Nice to Haves Business Drivers/Customer Impact DFS data migration effort - Capital One is undertaking a critical data migration initiative to ingest all historical Discover Card data into its internal systems. While the data already resides within the Capital One ecosystem, the challenge lies in efficiently ingesting it into its target state, ensuring completeness, accuracy, and usability across platforms. This migration is an enterprise imperative to enable downstream analytics, compliance, and operational continuity using DFS data. Success requires tight coordination across engineering, data governance, and platform teams to minimize disruption and maximize business value.