Lead Databricks Data Engineer

Company: Qualcomm India Private Limited Job Area: Information Technology Group, Information Technology Group > IT Data Engineer General Summary: We are looking for a Senior Data Engineer to design, build, and operate a modern, scalable data platform with Databricks Lakehouse as a core foundation. In this role, you will focus on building reusable data frameworks, shared platform components, and standardized pipelines that enable teams to deliver data products efficiently and consistently. Your work will support analytics, reporting, and downstream advanced use cases (including AI and machine learning), with a strong emphasis on reliability, governance, developer productivity, and intelligent automation. Strong technical role with ownership across data engineering, framework development, AI driven automation, platform reliability, security, and cost management, while contributing to architectural decisions and data standards. Data Engineering, Frameworks & AI Driven Automation • Design, build, and maintain scalable batch and streaming data pipelines • Develop reusable data engineering frameworks, libraries, and templates for ingestion, transformation, validation, and publishing • Establish standardized patterns for data modeling, transformations, and pipeline orchestration • Implement end-to-end data workflows from raw ingestion to curated analytical datasets • Leverage AI based techniques to automate and optimize data engineering workflows, such as: o Intelligent schema inference and evolution o Automated data quality checks and anomaly detection o Pipeline failure detection and self-healing mechanisms o Experience building AI assisted or intelligent automation for: o Data quality monitoring o Pipeline observability o Cost or performance optimization • Ensure data quality, reliability, and performance across pipelines and shared frameworks • Support downstream consumers such as analytics, reporting, and AI/ML teams Reliability, Operations & Intelligent Automation • Define and monitor SLIs/SLOs for data pipelines, frameworks, and platform availability • Participate in incident response, on-call rotations, and post-incident reviews • Apply AI assisted monitoring and alerting to proactively detect performance issues, data drift, and operational anomalies • Implement security, compliance, and data governance controls across shared data assets • Drive performance tuning and cost optimization, including automated recommendations for resource utilization and workload optimization Collaboration & Technical Ownership • Partner with analytics, application, and platform teams to understand common data needs and platform gaps • Adopt standardized data frameworks, automation patterns, and best practices • Contribute to data architecture decisions, platform standards, and design guidelines - 7+ years of IT-related work experience in Data Platform with a Bachelor's degree. - 7+ years of strong Data engineering experience with mandatory AI assisted development - Hands-on technical experience with AWS, Databricks data platform ecosystem of Fivetran ingestion and Acceldata observability tools - Solid hands-on experience in Data Engineering, Data Analysis, Data Modelling and enabling Analytics solutions - Passion to explore emerging technologies and perform Proof Of Concepts (POC) - Ability to work independently and collaborate with business and cross functional teams - Accountability and Ownership of the delivery in driving the projects to completion Data Engineering, Frameworks & System Design • 7+ years of experience building and operating data platforms or distributed data systems • Proven experience designing and building reusable data engineering frameworks, libraries, or platform components • Strong experience designing scalable, reliable data pipelines using standardized patterns • Solid understanding of data modeling, storage formats, schema evolution, and query performance • Experience implementing automation in data pipelines, including rule based or AI assisted approaches • Ability to reason about architectural trade-offs across scalability, cost, reliability, and security Cloud & Data Platform Experience • Strong hands-on experience with AWS, including IAM, networking, and multi-account setups • Proven experience with Databricks Lakehouse, including Delta Lake and Unity Catalog • Strong proficiency in Python for framework development, data processing, and automation • Experience building data platforms that support multiple consumers and automated workflows Security & Communication • Understanding of cloud security best practices and data governance • Experience working in regulated or compliance-driven environments • Strong communication skills and the ability to drive adoption of shared frameworks and automation patterns across teams Minimum Qualifications: • 4+ years of IT-related work experience with a Bachelor's degree in Computer Engineering, Computer Science, Information Systems or a related field. OR 6+ years of IT-related work experience without a Bachelor’s degree. • 2+ years of work experience with programming (e.g., Java, Python). • 1+ year of work experience with SQL or NoSQL Databases. • 1+ year of work experience with Data Structures and algorithms. Applicants: Qualcomm is an equal opportunity employer. If you are an individual with a disability and need an accommodation during the application/hiring process, rest assured that Qualcomm is committed to providing an accessible process. You may e-mail disability-accomodations@qualcomm.com or call Qualcomm's toll-free number found here. Upon request, Qualcomm will provide reasonable accommodations to support individuals with disabilities to be able participate in the hiring process. Qualcomm is also committed to making our workplace accessible for individuals with disabilities. (Keep in mind that this email address is used to provide reasonable accommodations for individuals with disabilities. We will not respond here to requests for updates on applications or resume inquiries). Qualcomm expects its employees to abide by all applicable policies and procedures, including but not limited to security and other requirements regarding protection of Company confidential information and other confidential and/or proprietary information, to the extent those requirements are permissible under applicable law. To all Staffing and Recruiting Agencies: Our Careers Site is only for individuals seeking a job at Qualcomm. Staffing and recruiting agencies and individuals being represented by an agency are not authorized to use this site or to submit profiles, applications or resumes, and any such submissions will be considered unsolicited. Qualcomm does not accept unsolicited resumes or applications from agencies. Please do not forward resumes to our jobs alias, Qualcomm employees or any other company location. Qualcomm is not responsible for any fees related to unsolicited resumes/applications. If you would like more information about this role, please contact Qualcomm Careers.