Technical Lead - Data Engineering

Minimum 14 years of exp is IT is required Technical skill sets :Python, Pyspark, AWS Lambda, S3,AWS glue, Kinesis,SQL, Apache Flink, Confluent Kafka, AI-LLM, Gen-AI , Java is optiona l Responsibilities :1. Advanced Architecture & System Desi gnA Tech Lead is primarily responsible for the overall platform vision and ensuring systems do not break under scal e. • Distributed Computing: Mastery of frameworks like Apache Spark or Ray for massive-scale parallel data processin g. • Streaming & Event-Driven Architecture: Deep understanding of real-time pipeline design using Kafka, Kinesis, or Flin k. • Cloud Infrastructure: Expertise in at least one major public cloud (AWS), specifically understanding storage/compute decoupling and cost optimizatio n.2. Core Programming & Database Manageme ntLeads set coding standards and review code, requiring complete fluency in the fundamental s. • SQL: Advanced mastery for metrics computation, window functions, and query performance tuning across relational and columnar databases (e.g., Snowflake, Redshift, BigQuery ). • Scripting Languages: High proficiency in Python or Scala for writing reusable pipeline code and interacting with API s. • Data Storage: Deep familiarity with both columnar/analytical stores and NoSQL databases (e.g., DynamoDb, Cassandra ).3. Pipeline Orchestration & DevO psEnsuring pipelines run smoothly, idempotently, and securely in productio n. • Workflow Orchestration: Ability to architect Directed Acyclic Graphs (DAGs) in tools like Apache Airflow or Prefec t. • CI/CD & Infrastructure as Code (IaC): Applying software engineering principles to data by using Docker, Kubernetes, and Terrafor m. • Data Governance & Security: Implementing Role-Based Access Control (RBAC), data masking, and compliance framework s.4. Leadership & Soft Skil lsTech leads also mentor junior engineers, estimate project timelines, and translate ambiguous business needs into concrete technical specification s. • Mentorship & Code Review: Fostering a collaborative development environment and enforcing style guideline s. • System Observability: Building logging, monitoring, and alerting mechanisms so the team knows exactly when and why pipelines fai l.