Senior Data Engineer

Description

Title : Senior Data Engineer

Department : IT Data

Open Lending is seeking an experienced Senior Data Engineer to drive the continued evolution of our modern data platform leveraging Databricks, Unity Catalog, Lakehouse architecture, as well as assisting in the continued support and development of our data warehouse. As a Senior Data Engineer, you will be building scalable, secure, and high-performance data pipelines supporting analytics, business intelligence, and machine learning operations. As part of our modernization initiatives, you will design and deliver end-to-end data solutions with an emphasis on Databricks workflows, PySpark-based processing, and Spark structured streaming pipelines. In this role, you will be engaged in all phases of the data management lifecycle, including gathering requirements, processing, storing and securing of data, communication, and documentation. The ideal candidate is passionate about data engineering, data quality, governance, and automation, with experience or a strong desire to expand into ML Ops.

Responsibilities :

• Design, build, and maintain scalable data ingestion and transformation pipelines using Databricks (PySpark, Delta Lake, Workflows) and Azure Data Factory.
• Implement Lakehouse architecture best practices with Delta Lake for performance, reliability, and ACID compliance.
• Utilize Unity Catalog for centralized data governance, lineage tracking, and fine-grained access controls.
• Develop and maintain data validation frameworks within Databricks to ensure accuracy, completeness, and compliance across all data domains.
• Build both batch and streaming data pipelines using Spark Structured Streaming, Kafka, or Azure Event Hubs.
• Drive performance tuning, cost optimization, and observability of data pipelines and compute clusters
• Improve resiliency and maintainability through testing, alerting, and automation with Databricks workflows and Lakehouse monitoring.
• Partner with architects and data consumers to deliver curated, high-quality datasets optimized for downstream analytics and machine learning.
• Champion CI / CD and DevOps practices for data pipelines, leveraging Git-based workflows and automation within Databricks and GitHub Actions.
• Identify and implement process improvements to enhance speed of delivery while maintaining governance and compliance.
• Enable downstream analytics and reporting platforms such as Power BI through reliable, governed data delivery.
• Support MLOps processes for model deployment, feature management, and pipeline integration within Databricks.

Education & Experience :

Skillsets Used

Azure Databricks (PySpark, Delta Lake, Unity Catalog, Workflows, MLflow, feature stores), ADLS Gen2 Storage, Microsoft SQL Server, Kafka / Event Hubs, Spark SQL, Python, Azure Key Vault, GitHub, CI / CD, Infrastructure-as-Code (Databricks Asset Bundles).

Physical Requirements :