Senior Data Engineer, Ingestion Framework - Data & AI Platform

Role Summary

As a Senior Data Engineer in the Data & AI Platform team, you will play a pivotal role in enabling our big data platform to operate seamlessly at a petabyte scale. The successful candidate will be an architect and custodian of a custom-built, Go-powered vehicle data processing framework, ensuring its smooth integration with our Databricks platform.

Responsibilities

• Framework Development & Maintenance: Design, develop, and enhance a cutting-edge, petabyte-scale vehicle data processing framework using Go. This framework is the backbone of our data operations; therefore, the ability to write clean, efficient, and maintainable code is essential.

• Databricks Integration: Collaborate closely with the big data platform team to ensure seamless integration of the framework with our Databricks environment. This involves understanding the intricacies of both systems and opti...