Talent.com
Data Engineer - AI & ML
Data Engineer - AI & MLTEPHRA • San Francisco, CA, United States
serp_jobs.error_messages.no_longer_accepting
Data Engineer - AI & ML

Data Engineer - AI & ML

TEPHRA • San Francisco, CA, United States
job_description.job_card.30_days_ago
serp_jobs.job_preview.job_type
  • serp_jobs.job_card.full_time
job_description.job_card.job_description

Description :

Location : San Francisco, CA

Responsibilities :

1.Design and Build Data Pipelines :

  • Develop, construct, test, and maintain data pipelines to extract, transform, and load (ETL) data from various sources to data warehouses or data lakes.
  • Ensure data pipelines are efficient, scalable, and maintainable, enabling seamless data flow for downstream analysis and modeling.
  • Work with stakeholders to identify data requirements and implement effective data processing solutions.

2. Data Integration :

  • Integrate data from multiple sources such as internal databases, external APIs, third-party vendors, and flat files.
  • Collaborate with business teams to understand data needs and ensure data is structured properly for reporting and analytics.
  • Build and optimize data ingestion systems to handle both real-time and batch data processing.
  • 3. Data Storage and Management :

  • Design and manage data storage solutions (e.g., relational databases, NoSQL databases, data lakes, cloud storage) that support large-scale data processing.
  • Implement best practices for data security, backup, and disaster recovery, ensuring that data is safe, recoverable, and complies with relevant regulations.
  • Manage and optimize storage systems for scalability and cost efficiency.
  • 4. Data Transformation :

  • Develop data transformation logic to clean, enrich, and standardize raw data, ensuring it is suitable for analysis.
  • Implement data transformation frameworks and tools, ensuring they work seamlessly across different data formats and sources.
  • Ensure the accuracy and integrity of data as it is processed and stored.
  • 5. Automation and Optimization :

  • Automate repetitive tasks such as data extraction, transformation, and loading to improve pipeline efficiency.
  • Optimize data processing workflows for performance, reducing processing time and resource consumption.
  • Troubleshoot and resolve performance bottlenecks in data pipelines.
  • 6. Collaboration with Data Teams :

  • Work closely with Data Scientists, Analysts, and business teams to understand data requirements and ensure the correct data is available and accessible.
  • Assist Data Scientists with preparing datasets for model training and deployment.
  • Provide technical expertise and support to ensure the integrity and consistency of data across all projects.
  • 7. Data Quality Assurance :

  • Implement data validation checks to ensure data accuracy, completeness, and consistency throughout the pipeline.
  • Develop and enforce data quality standards to detect and resolve data issues before they affect analysis or reporting.
  • Monitor and improve data quality by identifying areas for improvement and implementing solutions.
  • 8. Monitoring and Maintenance :

  • Set up monitoring and logging for data pipelines to detect and alert for issues such as failures, data mismatches, or delays.
  • Perform regular maintenance of data pipelines and storage systems to ensure optimal performance.
  • Update and improve data systems as required, keeping up with evolving technology and business needs.
  • 9. Documentation and Reporting :

  • Document data pipeline designs, ETL processes, data schemas, and transformation logic for transparency and future reference.
  • Create reports on the performance and status of data pipelines, identifying areas of improvement or potential issues.
  • Provide guidance to other teams regarding the usage and structure of data systems.
  • 10. Stay Updated with Technology Trends :

  • Continuously evaluate and adopt new tools, technologies, and best practices in data engineering and big data systems.
  • Participate in industry conferences, webinars, and training to stay current with emerging trends in data engineering and cloud computing.
  • Qualifications : (Please list all required qualifications) Click here to enter text.

    (Rationalizes basic requirements for candidates to apply. Helps w / rationalization when

    Requirements : -

  • Minimum of 7 years of total experience
  • 1.Educational Background :

    Bachelor's or Master's degree in Computer Science, Information Technology, Data Engineering, or a related field

    2.Technical Skills :

  • Proficiency in programming languages such as Python, Java, or Scala for data processing.
  • Strong knowledge of SQL and relational databases (e.g., MySQL, PostgreSQL, MS SQL Server).
  • Experience with NoSQL databases (e.g., MongoDB, Cassandra, HBase).
  • Familiarity with data warehousing solutions (e.g., Amazon Redshift, Google BigQuery, Snowflake).
  • Hands-on experience with ETL frameworks and tools (e.g., Apache NiFi, Talend, Informatica, Airflow).
  • Knowledge of big data technologies (e.g., Hadoop, Apache Spark, Kafka).
  • Experience with cloud platforms (AWS, Azure, Google Cloud) and related services for data storage and processing.
  • Familiarity with containerization and orchestration tools (e.g., Docker, Kubernetes) for building scalable data systems.
  • Knowledge of version control systems (e.g., Git) and collaboration tools (e.g., Jira, Confluence).
  • Understanding data modeling concepts (e.g., star schema, snowflake schema) and how they relate to data warehousing and analytics.
  • Knowledge of data lakes, data warehousing architecture, and how to design efficient and scalable storage solutions.
  • 3.Soft Skills :

  • Strong problem-solving skills with an ability to troubleshoot complex data issues.
  • Excellent communication skills, with the ability to explain technical concepts to both technical and non-technical stakeholders.
  • Strong attention to detail and a commitment to maintaining data accuracy and integrity.
  • Ability to work effectively in a collaborative, team-based environment.
  • 4.Experience :

  • 3+ years of experience in data engineering, with hands-on experience in building and maintaining data pipelines and systems.
  • Proven track record of implementing data engineering solutions at scale, preferably in large or complex environments.
  • Experience working with data governance, compliance, and security protocols.
  • 5.Preferred Qualifications

  • Experience with machine learning and preparing data for AI / ML model training.
  • Familiarity with stream processing frameworks (e.g., Apache Kafka, Apache Flink).
  • Certification in cloud platforms (e.g., AWS Certified Big Data - Specialty, Google Cloud Professional Data Engineer).
  • Experience with DevOps practices and CI / CD pipelines for data systems.
  • Experience with automation and orchestration tools (e.g., Apache Airflow, Luigi).
  • Familiarity with data visualization and reporting tools (e.g., Tableau, Power BI) to support analytics teams
  • 6.Work Environment :

  • Collaborative and fast-paced work environment.
  • Opportunity to work with state-of-the-art technologies.
  • Supportive and dynamic team culture
  • #LI-AD1

    serp_jobs.job_alerts.create_a_job

    Ai Ml Engineer • San Francisco, CA, United States

    Job_description.internal_linking.related_jobs
    Senior Data Platform Engineer – AI / ML Pipelines

    Senior Data Platform Engineer – AI / ML Pipelines

    Amazon • San Francisco, CA, United States
    serp_jobs.job_card.full_time
    A leading technology company in San Francisco is seeking an experienced Data Engineer to join Ring's Decision Sciences Platform Team. You will design and maintain robust data pipelines, ensuring the...serp_jobs.internal_linking.show_more
    serp_jobs.last_updated.last_updated_1_day • serp_jobs.job_card.promoted
    AI / ML Analytics & Metrics Frameworks Engineer

    AI / ML Analytics & Metrics Frameworks Engineer

    General Motors • San Francisco, CA, United States
    serp_jobs.job_card.full_time
    A leading automotive company is looking for an AI / ML Engineer to develop and optimize analytics frameworks as part of their autonomous vehicle development efforts. The role requires strong Python an...serp_jobs.internal_linking.show_more
    serp_jobs.last_updated.last_updated_variable_days • serp_jobs.job_card.promoted
    Senior Data / AI / ML Software Engineer

    Senior Data / AI / ML Software Engineer

    Crossing Hurdles • San Mateo, CA, United States
    serp_jobs.job_card.full_time
    Crossing Hurdles is a global recruitment firm partnering with, a fast-growing Clinical Data Intelligence platform built on 12+ years of advanced research in Machine Reading and Knowledge Graph tech...serp_jobs.internal_linking.show_more
    serp_jobs.last_updated.last_updated_variable_days • serp_jobs.job_card.promoted
    AI / ML Architect

    AI / ML Architect

    Cooley LLP • San Francisco, CA, United States
    serp_jobs.job_card.full_time
    Cooley is seeking an AI / ML Architect to join the Practice Engineering team within the Innovation department.As a leading technology law firm, Cooley is determined to become a leader in the digital ...serp_jobs.internal_linking.show_more
    serp_jobs.last_updated.last_updated_30 • serp_jobs.job_card.promoted
    ML Engineer — Build AI-Powered NLP CRM Features

    ML Engineer — Build AI-Powered NLP CRM Features

    Lightfield • San Francisco, CA, United States
    serp_jobs.job_card.full_time
    A leading AI-focused company in San Francisco is seeking an experienced AI / ML Product Developer to create innovative AI experiences and shape the company’s AI / ML strategy.The ideal candidate should...serp_jobs.internal_linking.show_more
    serp_jobs.last_updated.last_updated_variable_days • serp_jobs.job_card.promoted
    AI / ML Engineer

    AI / ML Engineer

    Krane • San Francisco, CA, United States
    serp_jobs.job_card.full_time
    Get AI-powered advice on this job and more exclusive features.Krane is building intelligent tools that power the future of construction operations. You’ll lead the design and deployment of intellige...serp_jobs.internal_linking.show_more
    serp_jobs.last_updated.last_updated_1_day • serp_jobs.job_card.promoted
    Senior ML Engineer - Backend, Data Pipelines & AI

    Senior ML Engineer - Backend, Data Pipelines & AI

    Rippling • San Francisco, CA, United States
    serp_jobs.job_card.full_time
    A leading technology company in San Francisco is seeking a Senior Software Engineer specializing in Machine Learning to design and develop backend software systems. Candidates should have over 6 yea...serp_jobs.internal_linking.show_more
    serp_jobs.last_updated.last_updated_variable_days • serp_jobs.job_card.promoted
    Generative AI ML Engineer — Production & Cloud

    Generative AI ML Engineer — Production & Cloud

    Adidev Technologies Inc • San Francisco, CA, United States
    serp_jobs.job_card.full_time
    A premier IT consulting firm is seeking a Machine Learning Engineer in San Francisco, CA.The role emphasizes deploying advanced ML models with a focus on Generative AI. Key responsibilities include ...serp_jobs.internal_linking.show_more
    serp_jobs.last_updated.last_updated_variable_days • serp_jobs.job_card.promoted
    ML / AI Engineer

    ML / AI Engineer

    Rillet • San Francisco, CA, United States
    serp_jobs.job_card.full_time
    Our customers are the financial brains of their companies.Our job is to help them run the numbers with impossible speed, accuracy, and insight. Today, we do that with powerful and elegant accounting...serp_jobs.internal_linking.show_more
    serp_jobs.last_updated.last_updated_30 • serp_jobs.job_card.promoted
    HPC / AI Data Performance Engineer

    HPC / AI Data Performance Engineer

    Lawrence Berkeley National Laboratory • Berkeley, CA, United States
    serp_jobs.job_card.full_time +1
    In this exciting role, you will serve as a Data Performance Engineer in NERSC's Application Performance Group, architecting HPC and AI data services that advance fundamental science.You'll optimize...serp_jobs.internal_linking.show_more
    serp_jobs.last_updated.last_updated_30 • serp_jobs.job_card.promoted
    Lead AI Engineer

    Lead AI Engineer

    1Five • San Francisco, CA, United States
    serp_jobs.job_card.full_time
    This is a leadership role at the intersection of.AI, technical architecture, and company vision.ML engineering and model development. Backflip’s core model, including architecture, data, training, a...serp_jobs.internal_linking.show_more
    serp_jobs.last_updated.last_updated_1_hour • serp_jobs.job_card.promoted • serp_jobs.job_card.new
    AI / ML Engineer

    AI / ML Engineer

    Keeper Tax Inc. • San Francisco, CA, United States
    serp_jobs.job_card.full_time
    K paying users ($199 to $399 per year).We're looking for an AI / ML engineer to build user-friendly features powered by cutting-edge tools. Here are some examples of projects we’re currently working o...serp_jobs.internal_linking.show_more
    serp_jobs.last_updated.last_updated_variable_days • serp_jobs.job_card.promoted
    Senior AI / ML Engineer - Product-Driven AI Workflows

    Senior AI / ML Engineer - Product-Driven AI Workflows

    TeamEx Inc. • San Francisco, CA, United States
    serp_jobs.job_card.full_time
    A technology company is seeking an AI Engineer to design and implement user-facing AI products.This role requires a blend of strong engineering and product thinking. Responsibilities include develop...serp_jobs.internal_linking.show_more
    serp_jobs.last_updated.last_updated_variable_days • serp_jobs.job_card.promoted
    Senior ML Data Engineer

    Senior ML Data Engineer

    Midjourney • Alameda, CA, United States
    serp_jobs.job_card.full_time
    We're the data team behind Midjourney's image generation models.We handle the dataset side : processing, filtering, scoring, captioning, and all the distributed compute that makes high-quality train...serp_jobs.internal_linking.show_more
    serp_jobs.last_updated.last_updated_variable_days • serp_jobs.job_card.promoted
    AI / ML Engineer

    AI / ML Engineer

    Stealth Company • San Francisco, CA, United States
    serp_jobs.job_card.full_time
    Get AI-powered advice on this job and more exclusive features.We're a well-funded stealth startup backed by proven unicorn founders, building the next generation of AI-powered consumer hardware.We'...serp_jobs.internal_linking.show_more
    serp_jobs.last_updated.last_updated_variable_days • serp_jobs.job_card.promoted
    ML Engineer, AI & Data Platform — GenAI / NLP

    ML Engineer, AI & Data Platform — GenAI / NLP

    Apple Inc. • San Francisco, CA, United States
    serp_jobs.job_card.full_time
    A leading technology company is looking for a Machine Learning Engineer in San Francisco.This role involves developing state-of-the-art AI models, implementing scalable ML infrastructure, and worki...serp_jobs.internal_linking.show_more
    serp_jobs.last_updated.last_updated_variable_days • serp_jobs.job_card.promoted
    Senior ML Engineer — AI-Driven Data Orchestration

    Senior ML Engineer — AI-Driven Data Orchestration

    Prophecy • San Francisco, CA, United States
    serp_jobs.job_card.full_time
    A tech company specializing in AI-native solutions is seeking a Senior Machine Learning Engineer to join their innovative team. This hybrid role offers a competitive salary of $250,000 to $350,000, ...serp_jobs.internal_linking.show_more
    serp_jobs.last_updated.last_updated_1_day • serp_jobs.job_card.promoted
    AI / ML Engineer

    AI / ML Engineer

    Rulebase • San Francisco, CA, United States
    serp_jobs.job_card.full_time
    This range is provided by Rulebase.Your actual pay will be based on your skills and experience — talk with your recruiter to learn more. We are building an autonomous factory for financial service a...serp_jobs.internal_linking.show_more
    serp_jobs.last_updated.last_updated_variable_days • serp_jobs.job_card.promoted