Generative AI Consultant

Role: Senior Databricks Data Architect (AWS) Location: Ridgefield Park, New Jersey 07660 12 Contract onsite position - This role requires 100% onsite presence Monday through Friday and will cover core business hours based on the EST timezone. JOB DESCRIPTION JOB TITLE: Senior Databricks Data Architect (AWS) - Hands-On Technical Lead PURPOSE OF THE JOB: Lead the strategic design and architectural roadmap of the Enterprise Data Platform on Databricks/AWS while remaining actively hands-on in design, prototyping, coding, and performance tuning. To provide high-level technical consultancy and solution design to internal business stakeholders and external partners, acting as the bridge between complex business requirements and technical implementation. To govern the technical quality of the platform by establishing design standards, conducting code reviews, and guiding the engineering team on best practices for Databricks Serverless and AWS integration. As part of the role, you will be required to provide technical assurance for all major data initiatives, ensuring that solutions are robust, cost-effective, and compliant with enterprise security standards. KEY ACCOUNTABILITIES: Lead all architecture design and review sessions for strategic Databricks initiatives. Actively contribute code to critical architectural components, building complex frameworks, reusable libraries, and core ETL/ELT pipelines using Python and PySpark. Lead by example by prototyping Proof of Concepts (PoCs) for new features (e.g., Databricks Asset Bundles, Delta Live Tables) before handing them off for wider team implementation. Perform hands-on performance tuning and debugging of high-latency jobs by deep-diving into Spark UI, execution plans, and spill logs. Defining and enforcing coding standards, data modeling best practices (Medallion Architecture), and CI/CD protocols for the development team to follow. conducting deep-dive technical reviews of designs and code produced by the engineering team to ensure efficiency, maintainability, and security. Designing the strategy for Unity Catalog governance, ensuring granular data access controls and compliance with data privacy regulations. Acting as the Technical Design Authority for the Databricks platform, making high-level decisions regarding the adoption of new Serverless features and AWS services. Mentoring and coaching Senior and Junior engineers, fostering a culture of technical excellence and continuous improvement. Own the technical relationship with key business stakeholders and translate business strategy into robust data architecture Engage and collaborate with Executive Stakeholders to present technical roadmaps and demonstrate the business value of data initiatives. Actively seeks opportunities to develop knowledge in emerging technologies (e.g., Generative AI on Databricks, Vector Databases) and evaluate their impact on the business. Provide guidance to the Change Management board regarding the risk and technical impact of proposed architectural changes. Develop strategies to architect resilience, high availability, and disaster recovery into the data platform. Develop clear communication channels with C-level executives and Product Managers to ensure technical strategies align with business goals. KEY LIAISONS: Head of Data & Analytics / CIO Business Unit Directors & Product Managers Lead Data Engineers and Data Scientists Enterprise Security & Infrastructure Architects DIMENSIONS: Ownership of the Technical Roadmap and Architecture for the Data Estate Maintain excellent relationships with all key liaisons Responsible for architectural decisions impacting critical data assets Primary Technical Advisor in the CST timezone EDUCATION: Educated to degree level in Computer Science, Engineering, or equivalent (Master's degree preferred) SKILLS AND ATTRIBUTES: Essential Exceptional "Customer Facing" skills: proven ability to lead workshops, manage stakeholder expectations, and present complex technical solutions to non-technical audiences. Minimum 8-10 years' experience in Data Engineering/Architecture, with at least 4 years specialized in Databricks and AWS . Expert-level knowledge of Data Lakehouse Architecture, Delta Lake internals, and Spark optimization strategies. Proven experience in Data Modeling (Dimensional, Vault, Relational) and designing complex ETL/ELT pipelines. Advanced hands-on coding proficiency in Python and SQL, with the ability to write clean, production-ready, and testable code. Strong experience with Databricks Serverless architecture, Unity Catalog implementation, and security patterns. Deep understanding of Cloud Architecture (AWS Well-Architected Framework), including networking, storage (S3), and IAM. Experience in Design Review and Code Review processes, capable of constructive critique and enforcing standards. Detailed knowledge of DevOps principles (IaC with Terraform, CI/CD pipelines). Knowledge of the MS Office suite (Word, Excel, PowerPoint, Outlook) specifically for creating architectural diagrams and presentations. Very strong relationship building and influencing skills at a senior level. Strategic thinking with the ability to balance immediate business needs with long-term architectural vision. EXPERIENCE: Essential - 8+ years of experience in Data Architecture and Engineering. Extensive experience designing solutions on the Databricks platform. Advantage - Strong experience working with large-scale enterprises or as a Consultant for big agencies. Certified Databricks Data Architect or AWS Solutions Architect (Professional). Excellent organizational and planning skills, with an ability to manage architectural backlogs. A best-in-class technical leader proactively driving business innovation and process improvements. Languages: Python, SQL, Scala. Experience designing APIs for data consumption. Experience with Enterprise Architecture frameworks (TOGAF). For applications and inquiries, contact: hirings@openkyber.com