Your app needs an analytics platform, but your data is a mess?

Here's how we built reporting capabilities from scratch in a multi-layered product.

The Challenge

Find a scalable solution that will work regardless of further strategic decisions; design future-proof data storage and reporting that will serve different purposes.

#AdTech

#Snowflake

#BigData

#Kafka

#KafkaConnect

#Kubernetes

The Problem

Szczepan's team received a challenge that was one tough nut to crack - building a reporting and analytics part of the product from scratch. But how to do it when data historically was never properly stored nor prepared? On top of that, parts of the product were written in different languages and maintained by 4 different teams. And the analytics part stood right at the very end and required the data to be structured well to be properly analysed and reported back to the users.

Meet our Expert

Szczepan Cieślik, a devoted manager passionate about Asian cuisine and "nerding out," has been instrumental in the project's success. With a love for bacon sushi and a decade-long experience in the company, Szczepan's insights and expertise have been invaluable.

Our team faced numerous dependencies across teams. Aligning with the existing parts of the product and structuring the data was essential. The technical challenge was significant, but it was more about the architecture of things.

Szczepan

Head of Technology

The Journey

For a newly formed, small team, we needed a fast and reliable solution, Szczepan explained. Snowflake came to our rescue. His experience taught him the importance of storing data in three separate tiers: raw, refined, and analytics data. This method allows for the processing of vast amounts of data without losing the original information. It might slow things down, but it's a safer approach.

Using Snowflake was a strategic decision, especially for a small team. It allowed for the quick introduction of a solution that a custom approach wouldn't have permitted. Snowflake's ability to handle vast amounts of data and ability to connect it (via Kafka Connect) to existing Kafka streams were invaluable. The three-tier data storage system ensured that raw facts remained untouched and accessible, even if refined or changed later.

Beyond the technical aspects, the project faced challenges related to the structure of services and communication. With multiple teams and many dependencies, aligning data, code, and communication was a hurdle. The corporate environment and the involvement of numerous individuals (~20-30) added layers of complexity. While the project wasn't overly challenging technically, communication posed significant challenges.

The Outcome

The solution provided the client with reporting capabilities and access to data that was previously limited.

Conclusion

In the dynamic and ever-evolving AdTech industry, staying ahead of the curve and adapting to new challenges is crucial. Emerging trends such as connected TVs and campaigns linked to the internet are becoming hot topics. The shift towards cookieless tracking, which the client had already adopted, is also a significant trend. Building a robust analytics platform from scratch, especially with disorganized data, is a testament to the team's dedication and expertise. As businesses grow and evolve, the importance of a well-structured, reliable analytics system cannot be overstated.

Ready to enhance your analytics?

Does this resonate with your challenges? Let's discuss your needs. We're here to brainstorm and explore the potential of enhancing your analytics capabilities.

Written by: Natalia, on October 12, 2023

Tags:

ADTECH

EXPERT CASE