Databricks Admin

Job Description :

We are looking for a candidate to manage and maintain Databricks platform, ensuring it runs smoothly, securely, and efficiently. This role involves tasks like workspace and cluster management, user and access control, performance optimization, and integration with other systems.

Core Responsibilities :

Creating and configuring Databricks workspaces, managing access control, and setting up user roles and permissions.

Setting up, monitoring, and optimizing Databricks clusters for optimal performance and resource utilization.

Implementing and maintaining security best practices, including data encryption, access controls, and compliance with relevant regulations.

Troubleshooting performance issues, identifying bottlenecks, and fine-tuning configurations for efficient resource allocation.

Integrating Databricks with other cloud services and automating infrastructure provisioning using tools like Terraform.

Setting up monitoring tools, analyzing logs, and ensuring the overall health and stability of the Databricks environment.

Managing data access, ensuring data quality, and implementing data governance policies.

Working closely with data engineers, analysts, and other users to understand their needs and support their Databricks workflows.

Creating and maintaining documentation for the Databricks environment, including setup instructions, best practices, and troubleshooting guides.

Key Skills and Experience :

• Experience with Databricks administration, including workspace management, cluster configuration, and security.
• Proficiency in AWS cloud platform.
• Strong understanding of data governance principles and best practices.
• Experience with scripting languages like Python or Bash for automation.
• Knowledge of AWS Gitlab CI / CD pipelines and integration with Databricks.
• Excellent communication and collaboration skills.
• Problem-solving and troubleshooting skills.