JOB ID:QT - AJH -10-613
We are looking for an experienced Data Engineer with strong Azure Databricks expertise to build, deploy, and manage modern data platforms. The ideal candidate will own end-to-end data engineering delivery, including ingestion, transformation, CI/CD, monitoring, and production support, while leveraging AI-powered development tools to improve engineering productivity.
Key Responsibilities
- Build scalable batch and real-time data pipelines using Azure Databricks, PySpark, SQL, and Delta Lake.
- Implement Medallion Architecture (Bronze/Silver/Gold) and governed data products using Unity Catalog.
- Develop streaming pipelines using Structured Streaming and Azure Event Hubs.
- Work with ADLS Gen2, Azure Key Vault, and Azure DevOps.
- Own CI/CD pipelines, deployment automation, monitoring, and production troubleshooting.
- Optimize data pipelines for performance, reliability, and cost efficiency.
- Build analytical datasets and dashboards using BI and visualization tools.
- Develop data products that support Analytics, ML, Gen AI, and agentic applications.
- Use AI coding assistants, AI agents, and automation workflows for development, testing, documentation, and troubleshooting.
- Collaborate with architects, analysts, and business stakeholders to deliver production-ready solutions.
Required Skills
- 4+ years of experience in Data Engineering or Data Platforms.
- Strong hands-on experience with Python, PySpark, and SQL.
- Expertise in Azure Databricks, Delta Lake, and Medallion Architecture.
- Experience with batch processing and real-time data streaming.
- Knowledge of Unity Catalog, Databricks Jobs/Workflows, and Structured Streaming.
- Experience with ADLS Gen2, Azure Event Hubs, Azure DevOps, Git, and Key Vault.
- Understanding of CI/CD, cloud security, monitoring, and production support.
- Hands-on experience with BI/dashboarding tools.
- Familiarity with AI coding assistants, agentic automation, and AI-native engineering practices.
