Big Data Engineer

Our client is looking Big Data Engineer  for Long term  project in Multi Location (USA)  below is the detailed requirements.

Job Title : Big Data Engineer

Location : Multi Location (USA)

Duration : Contract / Full-Time

Primary Skill :

AWS Cloud

Python programming and AWS services (EC2, RDS, S3, Lambda, .

Airflow DAG

Python to support data operations.

Integrate various internal and external data sources (including SharePoint, One Drive, One Note,

Understanding of CI / CD tools and practices.

Job Description :

• Using AWS cloud services.
• Develop and maintain automated data pipelines, ensuring data accuracy and efficient processing using Airflow DAG.
• Write high-quality, maintainable code in Python to support data operations.
• Integrate various internal and external data sources (including SharePoint, One Drive, One Note, to create a unified data platform.
• Build tools for document ingestion, dynamic file previewing, and tabular data extraction.
• Manage graph database systems and support large text corpus management.
• Implement security controls to maintain data integrity and compliance.
• Use Large Language Models for keyword extraction and data analysis to gain insights.

Requirements

• Proven experience with Python programming and AWS services (EC2, RDS, S3, Lambda, .
• Proficient in managing and orchestrating workflows with Airflow.
• Solid understanding of CI / CD tools and practices.
• Demonstrated experience with data engineering and familiarity with graph database management.
• Expertise in data processing, corpus management, and keyword extraction.
• Strong problem-solving abilities, attention to detail, and organizational skills.
• Excellent communication and teamwork skills to collaborate with cross-functional teams.