Apache NiFi Engineer

Join our team as an Apache NiFi Engineer and play a vital role in designing, deploying, and maintaining high-performance data pipelines that power real-time analytics and industrial IoT solutions. You'll be responsible for managing, securing, and troubleshooting NiFi clusters, optimizing data flows, and ensuring system reliability across complex environments involving IoT devices, SCADA systems, and big data platforms. With your expertise in data integration, industrial protocols, scripting, and system monitoring, you'll help deliver scalable, secure, and efficient data solutions that drive innovation and operational excellence. Technical Requirements For Apache NiFi Resource • Strong understanding of Apache NiFi: • Experience designing, developing, and deploying data flows using NiFi. • Knowledge of NiFi processors, controller services, templates, and reporting tasks. • Ability to configure and optimize NiFi for performance and reliability. • Data Integration and ETL skills: • Experience with data ingestion, transformation, routing, and system integration. • Familiarity with common data formats (JSON, XML, CSV, Avro, Parquet). • Understanding of batch and real-time data processing concepts. • IoT Data Processing with NiFi: • Experience ingesting and processing IoT data from industrial sensors, PLCs (Programmable Logic Controllers), SCADA systems, and other ICS devices. • Familiarity with industrial communication protocols such as MQTT, OPC-UA, Modbus, DNP3, and Ethernet/IP. • Ability to design NiFi flows for real-time telemetry data collection, anomaly detection, filtering, enrichment, and routing within ICS environments. • Handling high-volume, time-sensitive data streams with strict latency and reliability requirements. • Integration with industrial IoT platforms, historians, time-series databases, and analytics/monitoring tools. • Understanding of ICS-specific security concerns, including network segmentation, secure communication (TLS), and role-based access control. • Programming and Scripting: • Proficiency in scripting languages such as Groovy, Python, or Shell for custom processors or extensions. • Knowledge of Java is a plus for developing custom NiFi processors. • Big Data and Streaming Technologies: • Experience with Kafka, Hadoop, Spark, or other big data ecosystems is beneficial. • Understanding of messaging systems and streaming data pipelines. • System and Network Knowledge: • Familiarity with Linux/Unix environments where NiFi typically runs. • Understanding of networking concepts, security (SSL/TLS), and authentication mechanisms. • Monitoring and Troubleshooting: • Ability to monitor NiFi flows, troubleshoot issues, and optimize resource usage. • Experience with NiFi logs, provenance data, and metrics. • Version Control and CI/CD: • Experience with Git or other version control systems. • Familiarity with CI/CD pipelines for deploying NiFi flows. • Soft Skills: • Strong problem-solving and analytical skills. • Good communication skills to collaborate with data engineers, architects, and stakeholders. Benefits • 401(k). • Dental Insurance. • Health insurance. • Vision insurance. • We are an equal-opportunity employer and value diversity, equality, inclusion, and respect for people. • The salary will be determined based on several factors, including, but not limited to, location, relevant education, qualifications, experience, technical skills, and business needs. Additional Responsibilities • Participate in OP monthly team meetings and participate in team-building efforts. • Contribute to OP technical discussions, peer reviews, etc. • Contribute content and collaborate via the OP-Wiki/Knowledge Base. • Provide status reports to OP Account Management as requested. About Us At OP, we help you harness the power of technology for maximum impact. A technology consulting and solutions company, we offer advisory and managed services, innovative platforms, and staffing solutions across a wide range of fields including AI, cyber security, enterprise architecture, and beyond. For nearly two decades, we’ve been challenging the status quo of the consulting industry, serving up fresh, ingenious thinking through a radically lean structure. Together, this strategy delivers unprecedented performance at an unparalleled pace for faster results that propel your business forward.