Data Engineer (Middle) ID43228

Job Description

AgileEngine is an Inc. 5000 company that creates award-winning software for Fortune 500 brands and trailblazing startups across 17+ industries. We rank among the leaders in areas like application development and AI / ML, and our people-first culture has earned us multiple Best Place to Work awards.

WHY JOIN US

If you're looking for a place to grow, make an impact, and work with people who care, we'd love to meet you!

ABOUT THE ROLE

As a Middle / Senior Data Engineer , you’ll help build a high-impact Thematic Research Platform that processes large-scale financial data and powers daily insights for researchers and data scientists. You’ll design and optimize ETL pipelines, develop cloud data warehousing solutions, and shape AWS-based infrastructure using tools like Airflow, Spark, and Terraform. This role offers the chance to work on complex data ecosystems, drive performance improvements, and collaborate closely with cross-functional teams in a technically rich, mission-driven environment.

WHAT YOU WILL DO

• Build and support ETL pipelines;
• Monitor data pipelines, identify bottlenecks, optimize data processing and storage for performance and cost-effectiveness;
• Collaborate effectively with cross-functional teams including data scientists, analysts, software engineers, and business stakeholders;
• Work with Terraform to build AWS infrastructure;
• Analyze sources and build Cloud Data Warehouse and Data Lake solutions.

MUST HAVES

NICE TO HAVES

PERKS AND BENEFITS

Requirements

3+ years of experience with Python; 3+ years of experience in a Data Engineering role; Proficiency in programming languages commonly used in data engineering such as Python, SQL, and optionally Scala for working with data processing frameworks like Spark and libraries like Pandas; Proficiency in designing, deploying, and managing data pipelines using Apache Airflow for workflow orchestration and scheduling; Ability to design, develop, and optimize ETL processes to move and transform data from various sources into the data warehouse, ensuring data quality, reliability, and efficiency; Knowledge of big data technologies and frameworks such as Apache Spark for processing large volumes of data efficiently; Extensive hands-on experience with various AWS services relevant to data engineering, including Amazon MWAA, Amazon S3, Amazon RDS, Amazon EMR, AWS Lambda, AWS Glue, Amazon Redshift, AWS Data Pipeline, Amazon DynamoDB; Deep understanding and practical experience in building and optimizing cloud data warehousing solutions; Ability to monitor data pipelines, identify bottlenecks, and optimize data processing and storage for performance and cost-effectiveness; Excellent communication skills to collaborate effectively with cross-functional teams; Bachelor’s degree in computer science / engineering or other technical field, or equivalent experience; Upper-intermediate English level.