Data Engineer

Creospan is a growing tech collective of makers, shakers, and problem solvers, offering solutions today that will propel businesses into a better tomorrow. “Tomorrow’s ideas, built today!” In addition to being able to work alongside equally brilliant and motivated developers, our consultants appreciate the opportunity to learn and apply new skills and methodologies to different clients and industries. • • • • • • NO C2C/3RD PARTY, LOOKING FOR W2 CANDIDATES ONLY, must be able to work in the US without sponsorship now or in the future • • • Data Engineer San Francisco Bay Area and Chicago, IL Long term Contract Responsibilities • Design and implement scalable data ingestion frameworks capable of processing petabyte-scale datasets. • Develop automation scripts and orchestration workflows that transform manual data operations into self-service processes. • Operationalize ingestion processes by creating clear, repeatable runbooks and automation tools. • Collaborate with cross-functional teams to understand data sources, integration requirements, and performance bottlenecks. • Build monitoring, alerting, and reporting tools to ensure reliability and data quality across ingestion pipelines. • Continuously optimize system performance and resource utilization for large-scale data workflows. Required Qualifications • Proficiency in Python, SQL, and R, with a strong scripting and automation background. • Hands-on experience working with large-scale data processing (terabytes to petabytes). • Strong understanding of data pipeline orchestration, ETL frameworks, and workflow automation. • Experience with distributed data systems (e.g., Spark, Presto, Hive, or similar). • Ability to translate manual ingestion processes into automated, self-service tools. • Excellent problem-solving skills and attention to detail. Preferred Qualifications • Experience working in large enterprise data environments. • Familiarity with Airflow, Luigi, or other orchestration frameworks. • Background in data engineering, DevOps, or process automation. • Strong communication skills and ability to document and operationalize complex systems.