Mastering Data Pipelines with Apache Kafka
Table of Contents
- Introduction
- The Early Days of Analytics at Etsy
- 2.1 Building the Analytics Stack
- 2.2 Challenges Faced
- The Need for Change
- 3.1 Instrumenting the Native App
- 3.2 CD and Diversification Project
- Rewriting the ETL Layer
- 4.1 Migrating to Beacon Infrastructure
- 4.2 Switching to Scalding
- The Implications and Opportunities
- 5.1 Long-Term Implications of Choices
- 5.2 Fixing Issues Opens New Opportunities
- The Next Steps
- 6.1 Real-Time ETL and Streaming Infrastructure
- 6.2 Unifying Infrastructure for More Products
- 6.3 Expanding the Use of Kafka
- Takeaways
- 7.1 Long-Term Implications of Choices
- 7.2 Fixing Issues Creates New Opportunities
- Conclusion
The Evolution of Analytics at Etsy
In this article, we will take a closer look at the evolution of analytics at Etsy, an e-commerce company based in Brooklyn. We will delve into the early days of their analytics stack, the challenges they faced, the need for change, and the implications and opportunities that arose from their decisions. We will also discuss their next steps and provide takeaways that can be applied to other analytics projects.
1. Introduction
Etsy is a well-known e-commerce company that specializes in selling handmade goods. With over 600 employees, including 150 engineers, Etsy has experienced significant growth over the years. One of the key components of their success is their data engineering team, responsible for managing the data pipeline and analytics infrastructure.
2. The Early Days of Analytics at Etsy
2.1 Building the Analytics Stack
In the early days, Etsy faced the challenge of building an analytics pipeline from scratch. They needed real analytics to improve their search functionality and develop a new advertising product. To overcome this hurdle, they built a zero-impact analytics stack using JavaScript event loggers and a CDN-based infrastructure. This solution allowed them to track user actions and measure their response to different features.
2.2 Challenges Faced
While the initial analytics stack worked well, it had a few limitations. For instance, there was a 24-hour delay in processing events and a 48-hour latency in processing visit data. Additionally, they relied heavily on Google Analytics, which set limitations on their visit serialization logic. Despite these challenges, the analytics stack served its purpose and laid the foundation for future improvements.
3. The Need for Change
3.1 Instrumenting the Native App
As Etsy grew, it became essential to instrument their native app to gather analytics data. The team faced the challenge of integrating the app with the existing data pipeline. They developed a solution that involved bundling events and buffering them until they could be sent to the backend infrastructure. This allowed them to gather analytics data from the native app while maintaining the structure of their existing analytics stack.
3.2 CD and Diversification Project
Etsy embarked on a CDN and diversification project, which aimed to make their infrastructure more resilient and flexible. This project involved migrating away from their existing CDN provider and developing their own beacon infrastructure. The team also transitioned from using Apache access logs to using Elastic MapReduce jobs for processing data. These changes gave them more control over their data pipeline and improved their overall infrastructure.
4. Rewriting the ETL Layer
4.1 Migrating to Beacon Infrastructure
As Etsy's data engineering team grew, they realized the need to migrate from their existing infrastructure to an improved system. They moved away from the CDN-based approach and developed their own beacon infrastructure using Apache servers. This change allowed for better data collection and reduced data loss.
4.2 Switching to Scalding
Another significant decision the data engineering team made was to switch from using cascading JRuby to using Scalding for their big data processing. This decision was driven by the need for a more efficient and scalable solution. By utilizing Scalding and Scala, they were able to process data faster and provide better analytics capabilities.
5. The Implications and Opportunities
5.1 Long-Term Implications of Choices
Etsy's early decisions had long-term implications for their analytics stack. The initial choices they made, such as relying on Google Analytics and using a CDN-based infrastructure, set the trajectory for their analytics journey. These decisions shaped the limitations they faced and the challenges they needed to overcome in the future.
5.2 Fixing Issues Creates New Opportunities
While addressing the challenges they faced, Etsy's data engineering team found new opportunities for improvement. By instrumenting the native app and migrating to their own beacon infrastructure, they gained more control and flexibility over their data pipeline. This allowed them to explore new technologies and make further enhancements to their analytics infrastructure.
6. The Next Steps
6.1 Real-Time ETL and Streaming Infrastructure
One of the next steps for Etsy's data engineering team is to implement real-time ETL and develop a streaming infrastructure. By reducing latency and processing data in near real-time, they aim to provide more up-to-date and actionable insights for their analytics projects.
6.2 Unifying Infrastructure for More Products
With the improvements made to their analytics stack, Etsy plans to unify their infrastructure for more data products. By bringing them onto a single stack, they can streamline their operations and provide a consistent and reliable analytics platform for different teams within the company.
6.3 Expanding the Use of Kafka
Etsy has identified Kafka as a key technology for their analytics infrastructure. They plan to expand its usage and explore additional use cases to leverage its capabilities further. Kafka's scalability and resiliency make it an ideal choice for handling event streaming and processing.
7. Takeaways
7.1 Long-Term Implications of Choices
Etsy's experience highlights the importance of considering the long-term implications of the choices made during the development of an analytics stack. Decisions made early on can significantly impact the capabilities and limitations of the infrastructure in the future. It is crucial to strike a balance between addressing immediate needs and designing for scalability and flexibility.
7.2 Fixing Issues Creates New Opportunities
Fixing issues and addressing challenges can lead to new opportunities and improvements. By continuously reviewing and iterating on the existing infrastructure, organizations can uncover new ways to enhance their analytics capabilities. Embracing change and being open to new technologies can result in significant advancements in data processing and analysis.
8. Conclusion
The evolution of analytics at Etsy showcases the iterative nature of building and refining a data infrastructure. From their early days using a CDN-based approach to their current use of Kafka for event streaming, Etsy's data engineering team has faced challenges and embraced opportunities along the way. Their journey serves as a valuable case study for organizations looking to improve their analytics capabilities and build a scalable and flexible infrastructure.
