Data Engineer

Job Title: Data Engineer Location: Onsite – Denver, CO Inperson Interview is must. Employment Type: W2 Job Summary We are seeking an experienced Data Engineer IV to design, develop, and maintain scalable ETL pipelines and data infrastructure supporting the IIA Data Lake, anomaly detection models, AI agents, dashboards, and other downstream data consumers. The ideal candidate will have strong experience with Python, Apache Spark, ETL development, AWS data services, SQL, and large-scale distributed data processing. This role will focus on onboarding diverse network data sources, ensuring data quality and reliability, and building highly scalable data solutions capable of processing billions of events per day. Key Responsibilities • Design, develop, and maintain scalable ETL pipelines using Apache Spark and Scala to ingest, transform, and load network data into the IIA Data Lake. • Build ingestion pipelines for network telemetry, syslogs, SNMP traps, device configuration data, ticketing systems, and other data sources. • Implement monitoring and alerting solutions to ensure pipeline reliability, performance, and availability. • Develop and maintain CI/CD and deployment pipelines for data engineering solutions. • Manage and optimize data storage across distributed file systems, relational databases, flat files, APIs, and other external sources. • Implement data quality checks, validation rules, and automated testing to maintain data integrity. • Optimize batch and mini-batch processing pipelines handling billions of events per day. • Design and maintain data schemas, partitioning strategies, and efficient storage formats such as Parquet. • Perform data backfills and recovery activities when upstream issues or schema changes require reprocessing. • Collaborate with data scientists, AI/agent developers, BI teams, and other stakeholders to deliver reliable datasets for downstream applications. • Provide technical guidance on data engineering best practices, standards, and methodologies. • Document data engineering processes and communicate technical solutions to technical and non-technical stakeholders. • Continuously evaluate and improve tools, technologies, and approaches to enhance data pipeline performance and efficiency. • Perform other duties as assigned. Required Qualifications • Strong proficiency in Python and distributed data processing. • Strong hands-on experience with Apache Spark; experience with Scala is preferred. • Proven experience designing and maintaining large-scale ETL pipelines. • Hands-on experience with AWS data services, including S3, Glue, Athena, and EMR. • Strong understanding of relational databases and SQL. • Knowledge of data architecture, data warehousing, partitioning strategies, and columnar storage formats such as Parquet. • Experience implementing data quality checks, validation frameworks, and automated testing. • Experience with workflow orchestration tools such as Airflow. • Proficiency with Linux/Unix environments and shell scripting. • Experience with Git and collaborative software development workflows. • Strong communication, collaboration, problem-solving, and continuous-learning skills. Preferred Qualifications • Experience with streaming or mini-batch processing, including Spark Streaming or Structured Streaming. • Experience with Apache Kafka or similar messaging/streaming technologies. • Experience working with NoSQL databases. • Telecommunications or large-scale network operations experience. • Familiarity with network data sources such as telemetry, syslogs, SNMP traps, and device configuration data. • Experience integrating data through REST APIs and cloud SDKs, including boto3. • Experience developing automated tests for data pipelines. Position Details Title: Data Engineer IV Location: Denver, CO – Onsite Employment: W2 Only