DevOps Engineer

Please Note: • Work Arrangement: This is a hybrid role, requiring employees to be onsite three (3) days per week at our New York City office. Candidates must currently reside in the NYC metropolitan area or demonstrate a clear willingness and ability to relocate prior to their start date. • Background Verification: All selected candidates will undergo a comprehensive background check, which may include employment history verification, education verification, reference checks, and criminal background screening. • Work Authorization: • We are not able to consider OPT, CPT, or H1B transfer candidates for this position. • C2C (Corp-to-Corp) arrangements are acceptable, provided the vendor company is compliant with U.S. labor and tax regulations. • At this time, we are only considering applications from U.S. Citizens and Green Card holders due to project requirements. • Additional Requirements: Candidates must have the ability to work independently in a hybrid environment, adhere to confidentiality and compliance standards, and be prepared to collaborate with cross-functional teams onsite when required. About the job We are seeking a highly skilled and experienced Azure Infrastructure and Databricks Specialist to join our team. The ideal candidate will have extensive experience in setting up and managing Azure infrastructure, Azure Databricks, and various other Azure services. The candidate should also have a strong background in Red Hat OpenShift or Kubernetes, as well as experience with Airflow instance setup and DAG deployment. Core Technical Skills Kubernetes (AKS/EKS): • Strong understanding of core components (API Server, Scheduler, Controller Manager, etcd, kubelet). • Experience with Deployments, StatefulSets, DaemonSets, ReplicaSets, and rolling updates with zero downtime. • Ability to roll back production deployments safely with access control in place. • Knowledge of AKS cluster upgrades with high availability strategies. • Terraform & Infrastructure as Code (IaC): • Proficient in Terraform modules, remote state management, and resolving state locking issues. • Experience building Databricks Terraform modules (user sync, workspace setup, cluster provisioning). • Strong understanding of GitHub/GitLab integrations with Terraform CI/CD pipelines. Databricks & Data Engineering: • Hands-on with Databricks REST APIs for automation and provisioning. • Experience with Unity Catalog replication/migration across regions. • Strong knowledge of Databricks workloads (ETL, ML, streaming, Delta Lake optimization). • Integration with Airflow for DAG orchestration (managing 10–20+ DAGs). • Troubleshooting queued tasks/delays in Airflow with Databricks jobs. • Azure Cloud Expertise: • Strong knowledge of Azure Resource Groups, Storage Accounts (types and use cases), and VM failover/failback. • Ability to secure internal-facing VMs (NSGs, firewalls, managed identity). • Deep understanding of Azure Service Principal vs Managed Identity. • AWS Cloud Expertise: • Cost optimization using EC2 right-sizing, Spot Instances, Savings Plans. • S3 lifecycle policies, deduplication, and Intelligent-Tiering. • Multi-tenant SaaS design with VPC isolation, IAM roles, KMS encryption, and workload segregation. Site Reliability & Automation • Proven experience implementing monitoring & observability (Prometheus, Grafana, Datadog, CloudWatch). • Expertise in incident diagnosis (DNS outages, Kafka corruption, pipeline failures). • Familiarity with chaos engineering practices and blameless post-mortems. • Building voice-based / chatbot-driven analytics for querying KPIs. • Automating provisioning using Ansible, Terraform, and GitOps workflows. System Design & Architecture • Designing elastic, cost-effective, public-facing applications that autoscale under unpredictable traffic spikes. • Experience with load balancing (NGINX, ALB, cross-zone balancing) and caching (Redis, Memcached). • Knowledge of OSI model layers and their relevance to network troubleshooting & security. • Resilient design for financial services and mission-critical applications. • Defense strategies for DDoS attacks (WAF, rate limiting, CDN-based protection). Soft Skills & Customer Engagement • Ability to gather requirements from customers, map them to cloud-native solutions, and translate business SLAs into technical implementations. • Strong experience guiding customers as an architect for multi-cloud deployments. • Excellent communication and documentation skills, particularly in runbooks, automation workflows, and post-incident reviews.