Data Engineer

W2 ONLY - we are not able to provide sponsorship REMOTE What You'll Do • Address issues such as ML training pipelines not emitting metadata about upstream dependencies and training data being gathered directly within pipelines without declared upstreams. • Provide insight into the data used to train a model to ensure compliance with legal and regulatory obligations. • Add support to OpenLineage for data endpoint types (e.g., model training, feature generation, model inference, feature registration). • Improve ML pipeline lineage instrumentation by connecting feature lineage to upstream pipeline lineage. • Help user teams migrate their existing implementation to the new orchestration with lineage • Works autonomously, providing proactive updates on progress and obstacles encountered in creating the lineage features between data and training pipelines. Who You Are • You have 3+ years of hands-on experience implementing production ML infrastructure at scale in Python • 3+ years of experience working with a public cloud provider such as GCP, AWS, or Azure. Preferably GCP. • Must have Knowledge of deep learning fundamentals, algorithms, and open-source tools such as Flyte, PyTorch, Ray, Kubeflow, TensorFlow, and Huggingface • You have a general understanding of data processing for ML • You have experience with agile software processes and modular code design following industry standards.