Databricks Architect
Experience: 10+ Years
Work Mode: Remote
Job Summary
We are looking for an experienced Databricks Architect to design, develop, and implement scalable data engineering and analytics solutions using Databricks and cloud technologies. The ideal candidate should have strong expertise in data architecture, lakehouse design, data pipelines, performance optimization, and cloud-based data platforms.
Key Responsibilities
- Design and implement scalable data lakehouse architectures using Databricks.
- Develop and optimize data pipelines using Apache Spark, PySpark, and Spark SQL.
- Build and manage ETL/ELT workflows for batch and streaming data processing.
- Implement data solutions using Delta Lake, Delta Live Tables (Lakeflow Declarative Pipelines), and Unity Catalog.
- Design data architectures on AWS, Microsoft Azure, or Google Cloud Platform.
- Implement data governance, security, access controls, and data quality standards.
- Optimize Spark jobs, cluster configurations, and overall platform performance.
- Integrate Databricks with cloud storage, data warehouses, APIs, and enterprise applications.
- Establish CI/CD practices, automated deployments, and infrastructure-as-code standards.
- Collaborate with business stakeholders, data engineers, data scientists, and technical teams.
Required Skills
- 10+ years of experience in data engineering, data architecture, or related roles.
- Strong hands-on experience with Databricks and Apache Spark.
- Expertise in PySpark, Python, and Spark SQL.
- Experience designing Lakehouse architectures and implementing Delta Lake.
- Strong knowledge of ETL/ELT, data modeling, and distributed data processing.
- Experience with at least one cloud platform: Azure, AWS, or GCP.
- Knowledge of data governance, security, and access management.
- Experience with workflow orchestration, monitoring, and performance tuning.
- Familiarity with Git, CI/CD pipelines, and DevOps practices.
- Strong architecture, problem-solving, and communication skills.
Preferred Qualifications
- Databricks Certified Data Engineer Professional or Databricks Certified Data Engineer Associate.
- Experience with Unity Catalog, Lakeflow, Structured Streaming, and data governance frameworks.
- Exposure to dbt, Kafka, Airflow, Power BI, or other analytics tools.
- Experience leading technical teams and defining enterprise data architecture standards.
