Talent.com
PL / SQL PY Spark Developer

PL / SQL PY Spark Developer

ExperisHouston, Texas, US
job_description.job_card.variable_days_ago
serp_jobs.job_preview.job_type
  • serp_jobs.job_card.full_time
job_description.job_card.job_description

We are seeking a Senior Data Engineer to lead a critical pilot project focused on modernizing our enterprise customer data consolidation from SQL Server to our Databricks-based data lake. This role combines traditional Oracle / PL / SQL expertise with modern PySpark development to support our East region initiative across three billing platforms.

Key Responsibilities

Primary Project (Enterprise Customer Table Pilot)

  • Design and implement data consolidation solutions moving from SQL Server to Databricks data lake
  • Work with business stakeholders and cross-functional teams to define enterprise customer table specifications
  • Determine optimal approach for data processing - either within existing Oracle systems or in the data lake environment
  • Collaborate with enterprise data lake team to leverage existing PySpark resources and infrastructure
  • Produce modified data inputs for the new enterprise customer table consolidation process
  • Ensure data quality and consistency across three different billing platform feeds

Secondary Pilot (Code Conversion)

  • Convert existing Oracle / PL / SQL code to PySpark for data lake processing
  • Evaluate feasibility of migrating current data warehouse operations to PySpark
  • Provide proof-of-concept for future large-scale migration initiatives
  • Test and validate converted code performance in the data lake environment

    • Team Development & Knowledge Transfer

  • Train and mentor existing PL / SQL team members on PySpark technologies
  • Work independently with minimal supervision while collaborating effectively with stakeholders
  • Provide technical leadership and architectural guidance for data processing solutions
  • Document best practices and create knowledge base for future PySpark implementations

    • Required Technical Skills

    Core Requirements

  • 10+ years
  • of back-end development experience
  • Expert-level PL / SQL and Oracle
  • database development
  • 7+ years of PySpark
  • experience with data lake implementations
  • Strong experience with
  • Databricks
  • platform
  • Proficiency in
  • data modeling and schema design
  • Experience with
  • data pipeline development
  • Custom data warehouse development experience

    • Preferred Technologies

  • Delta Lake
  • experience
  • Apache Airflow
  • for job scheduling and pipeline orchestration
  • GCP (Google Cloud Platform)
  • our primary cloud environment
  • AWS or Azure
  • cloud experience (transferable)
  • Data warehouse and ETL / ELT processes
  • Experience with enterprise-scale data integration projects

    • Technical Environment

  • Current Stack
  • Oracle-based custom data warehouse, PL / SQL processing
  • Target Stack
  • Databricks data lake, PySpark, Delta Lake, GCP
  • Integration Points
  • Three separate billing systems, SQL Server consolidation layer
  • Data Volume
  • Enterprise-scale customer data across multiple regions

    • Business Context & Domain Knowledge

  • Support for multiple regions : East, Texas (largest), and Panera
  • Integration challenges across three separate billing systems with different data formats
  • Enterprise-level customer data consolidation and reporting requirements
  • Migration from legacy SQL Server data warehouse to modern data lake architecture
  • Sales reporting and sales count reporting focus
  • Experience with acquired company data integration challenges preferred

    • Required Competencies

    Technical Leadership

  • Ability to analyze existing systems and recommend architectural improvements
  • Experience designing scalable data processing solutions
  • Strong debugging and troubleshooting skills across multiple platforms
  • Code review and quality assurance capabilities

    • Business Acumen

  • Understanding of enterprise data warehouse concepts
  • Experience with customer data management and consolidation
  • Knowledge of sales reporting and business intelligence requirements
  • Familiarity with multi-system integration challenges

    • Communication & Collaboration

  • Excellent stakeholder management skills
  • Ability to translate technical concepts to business users
  • Experience working with cross-functional teams
  • Strong documentation and knowledge sharing abilities

    • Work Arrangement

  • Hybrid role
  • 3 days on-site (Monday, Tuesday, Thursday) in Houston, TX
  • 2 days remote work
  • Candidates willing to relocate to Houston will be considered
  • Office Location
  • Houston, Texas (specific location to be provided)
    • serp_jobs.job_alerts.create_a_job

    Developer • Houston, Texas, US