Senior Data Engineer - Spark, AirflowSigmaways Inc • Santa Rosa, CA, United States
Senior Data Engineer - Spark, Airflow
Sigmaways Inc • Santa Rosa, CA, United States
job_description.job_card.variable_days_ago
serp_jobs.job_preview.job_type
- serp_jobs.job_card.full_time
job_description.job_card.job_descriptionBachelor’s degree in Computer Science, Data Engineering, or related field (or equivalent experience). At least 7 years of experience in data engineering or big data development. Strong expertise in Apache Spark architecture, optimization, and job configuration. Proven experience with Airflow DAGs using authoring, scheduling, checkpointing, monitoring. Skilled in data shuffling, partitioning strategies, and performance tuning in distributed systems. Expertise in Python programming including data structures and algorithmic problem-solving. Hands-on with Spark DataFrames and PySpark transformations using joins, aggregations, filters. Proficient in shell scripting, including managing and passing variables between scripts. Experienced with spark submit for deployment and tuning. Solid understanding of ETL design, workflow automation, and distributed data systems. Excellent debugging and problem-solving skills in large-scale environments. Experience with AWS Glue, EMR, Databricks, or similar Spark platforms. Knowledge of data lineage and data quality frameworks like Apache Atlas. Familiarity with CI / CD pipelines, Docker / Kubernetes, and data governance tools.
We are seeking an experienced Data Engineer to design and optimize scalable data pipelines that drive our global data and analytics initiatives.
In this role, you will leverage technologies such as Apache Spark , Airflow , and Python to build high performance data processing systems and ensure data quality, reliability, and lineage across Mastercard’s data ecosystem.
The ideal candidate combines strong technical expertise with hands-on experience in distributed data systems, workflow automation, and performance tuning to deliver impactful, data-driven solutions at enterprise scale.
Responsibilities :
- Design and optimize Spark-based ETL pipelines for large-scale data processing.
- Build and manage Airflow DAGs for scheduling, orchestration, and checkpointing.
- Implement partitioning and shuffling strategies to improve Spark performance.
- Ensure data lineage, quality, and traceability across systems.
- Develop Python scripts for data transformation, aggregation, and validation.
- Execute and tune Spark jobs using spark-submit.
- Perform DataFrame joins and aggregations for analytical insights.
- Automate multi-step processes through shell scripting and variable management.
- Collaborate with data, DevOps, and analytics teams to deliver scalable data solutions.
Qualifications :
serp_jobs.job_alerts.create_a_job
Senior Data Engineer • Santa Rosa, CA, United States
Job_description.internal_linking.related_jobs If you’re hands on with modern data platforms, cloud tech, and big data tools and you like building solutions that are secure, repeatable, and fast, this role is for you.
As a Senior Data Engineer, ...serp_jobs.internal_linking.show_more 
If youre hands on with modern data platforms, cloud tech, and big data tools and you like building solutions that are secure, repeatable, and fast, this role is for you.
As a Senior Data Engineer, y...serp_jobs.internal_linking.show_more
Velocity Tech has partnered with an exciting tech start-up that is looking for multiple Backend Developers to join their team.
This role will be on-site in San Francisco.They are currently seed-fund...serp_jobs.internal_linking.show_more
Keysight is on the forefront of technology innovation, delivering breakthroughs and trusted insights in electronic design, simulation, prototyping, test, manufacturing, and optimization.Our ~15,000...serp_jobs.internal_linking.show_more
Our client is an innovative fintech startup based in the Bay Area, emerging as a key player in the financial software sector.
Since its product launch in summer 2024, this seed-stage organization ha...serp_jobs.internal_linking.show_more
We’re supporting a major global financial technology organization that’s making significant investments in AI innovation.
They’re scaling their engineering teams across North America to drive develo...serp_jobs.internal_linking.show_more
Our client is a leading force in advancing safer, smarter AI technology.Their work has been featured in.They’ve built a global community of expert contributors and have already paid out more ...serp_jobs.internal_linking.show_more
Senior Backend Engineer – AI Application Builder.SoMa, San Francisco, CA | 💼 Full-Time | Onsite.Salary : $200K–$325K + Meaningful Equity (1%+).
Join an early-stage AI SaaS start-up creating an.AI co...serp_jobs.internal_linking.show_more
Homebound is on a mission to make it possible for anyone anywhere to build a home using technology.Created by an experienced team of construction real estate design and technology experts Homebound...serp_jobs.internal_linking.show_more
Senior Backend Engineer AI Application Builder.SoMa, San Francisco, CA | 💼 Full-Time | Onsite.Salary : $200K$325K + Meaningful Equity (1%+).
Join an early-stage AI SaaS start-up creating an.AI coll...serp_jobs.internal_linking.show_more
We’re looking for a Lead Data Engineer to spearhead the design, implementation, and iteration of a world-class, modern data infrastructure that powers analytics, data science, and ML / AI systems.You...serp_jobs.internal_linking.show_more
Crossing Hurdles is a global recruitment firm partnering with, a fast-growing Clinical Data Intelligence platform built on 12+ years of advanced research in Machine Reading and Knowledge Graph tech...serp_jobs.internal_linking.show_more
We're the data team behind Midjourney's image generation models.We handle the dataset side : processing, filtering, scoring, captioning, and all the distributed compute that makes high-quality train...serp_jobs.internal_linking.show_more Midjourney is a research lab exploring new mediums to expand the imaginative powers of the human species.We are a small, self-funded team focused on design, human infrastructure, and AI.We have no ...serp_jobs.internal_linking.show_more 
Honey Health is the all-in-one AI back office for primary and specialty care.Our AI agents autonomously handle core back-office jobs, including aggregating patient data, processing orders and presc...serp_jobs.internal_linking.show_more
We are actively searching for a Data Infrastructure Engineer to join our team on a permanent basis.In this founding engineer role you will focus on building next-generation data infrastructure for ...serp_jobs.internal_linking.show_more
Honey Health is the all-in-one AI back office for primary and specialty care.Our AI agents autonomously handle core back-office jobs, such as aggregating patients data, processing orders and prescr...serp_jobs.internal_linking.show_more
Tiger Analytics is looking for experienced.Gen AI experience to join our fast-growing advanced analytics consulting firm.
Our employees bring deep expertise in Machine Learning, Data Science, and AI...serp_jobs.internal_linking.show_more
Senior Data Engineer
Sigmaways Inc • Santa Rosa, CA, United States
serp_jobs.job_card.full_time
serp_jobs.last_updated.last_updated_variable_days • serp_jobs.job_card.promoted
Senior Data Engineer (Santa Rosa)
Sigmaways Inc • Santa Rosa, CA, US
serp_jobs.job_card.part_time
serp_jobs.last_updated.last_updated_variable_days • serp_jobs.job_card.promoted
Senior Backend Engineer
Velocity Tech • Santa Rosa, CA, United States
serp_jobs.job_card.full_time
serp_jobs.last_updated.last_updated_variable_hours • serp_jobs.job_card.promoted • serp_jobs.job_card.new
Senior Software Engineer
Keysight Technologies • Santa Rosa, CA, United States
serp_jobs.job_card.full_time
serp_jobs.last_updated.last_updated_variable_days • serp_jobs.job_card.promoted
Senior Software Engineer (Ref : 194286) (Santa Rosa)
Forsyth Barnes • Santa Rosa, CA, US
serp_jobs.job_card.part_time
serp_jobs.last_updated.last_updated_variable_hours • serp_jobs.job_card.promoted • serp_jobs.job_card.new
Lead Generative AI Engineer
Madison-Davis, LLC • Santa Rosa, CA, United States
serp_jobs.job_card.full_time
serp_jobs.last_updated.last_updated_variable_days • serp_jobs.job_card.promoted
Engineer
Quality Talent Group • Santa Rosa, California, United States
serp_jobs.job_card.full_time
serp_jobs.filters_job_card.quick_apply
serp_jobs.last_updated.last_updated_variable_days
Senior Back End Engineer - AI Workflow / Application Builder
Ikuto • Santa Rosa, CA, United States
serp_jobs.job_card.full_time
serp_jobs.last_updated.last_updated_variable_days • serp_jobs.job_card.promoted
Senior or Staff AI Engineer
Homebound • Santa Rosa, California, USA
serp_jobs.job_card.full_time
serp_jobs.last_updated.last_updated_variable_days • serp_jobs.job_card.promoted
Senior Back End Engineer - AI Workflow / Application Builder (Santa Rosa)
Ikuto • Santa Rosa, CA, US
serp_jobs.job_card.full_time +1
serp_jobs.last_updated.last_updated_variable_days • serp_jobs.job_card.promoted
Lead Data Engineer
Mentor Talent Acquisition • Santa Rosa, CA, United States
serp_jobs.job_card.full_time
serp_jobs.last_updated.last_updated_variable_days • serp_jobs.job_card.promoted
Senior Data / AI / ML Software Engineer
Crossing Hurdles • Santa Rosa, CA, United States
serp_jobs.job_card.full_time
serp_jobs.last_updated.last_updated_variable_days • serp_jobs.job_card.promoted
Senior ML Data Engineer
Midjourney • Santa Rosa, CA, United States
serp_jobs.job_card.full_time
serp_jobs.last_updated.last_updated_variable_days • serp_jobs.job_card.promoted
Data Engineer
Midjourney • Santa Rosa, CA, United States
serp_jobs.job_card.full_time
serp_jobs.last_updated.last_updated_variable_days • serp_jobs.job_card.promoted
Senior Software Engineer - AI Agent Infrastructure
Honey Health • Santa Rosa, CA, United States
serp_jobs.job_card.full_time
serp_jobs.last_updated.last_updated_variable_days • serp_jobs.job_card.promoted
Data Platform Engineer / AI Workloads (Santa Rosa)
The Crypto Recruiters • Santa Rosa, CA, US
serp_jobs.job_card.part_time +1
serp_jobs.last_updated.last_updated_variable_days • serp_jobs.job_card.promoted
Senior Software Engineer - AI Agent Infrastructure (Healthcare)
Honey Health • Santa Rosa, CA, US
serp_jobs.job_card.full_time
serp_jobs.last_updated.last_updated_variable_days • serp_jobs.job_card.promoted
Azure AI Lead Engineer
Tiger Analytics • Santa Rosa, CA, United States
serp_jobs.job_card.full_time
serp_jobs.last_updated.last_updated_variable_days • serp_jobs.job_card.promoted