Your app needs an analytics platform, but your data is a mess?
Here's how we built reporting capabilities from scratch in a multi-layered product.
Meet our Expert
Our team faced numerous dependencies across teams. Aligning with the existing parts of the product and structuring the data was essential. The technical challenge was significant, but it was more about the architecture of things.
For a newly formed, small team, we needed a fast and reliable solution, Szczepan explained. Snowflake came to our rescue. His experience taught him the importance of storing data in three separate tiers: raw, refined, and analytics data. This method allows for the processing of vast amounts of data without losing the original information. It might slow things down, but it's a safer approach.
Using Snowflake was a strategic decision, especially for a small team. It allowed for the quick introduction of a solution that a custom approach wouldn't have permitted. Snowflake's ability to handle vast amounts of data and ability to connect it (via Kafka Connect) to existing Kafka streams were invaluable. The three-tier data storage system ensured that raw facts remained untouched and accessible, even if refined or changed later.
Beyond the technical aspects, the project faced challenges related to the structure of services and communication. With multiple teams and many dependencies, aligning data, code, and communication was a hurdle. The corporate environment and the involvement of numerous individuals (~20-30) added layers of complexity. While the project wasn't overly challenging technically, communication posed significant challenges.
Ready to enhance your analytics?
Does this resonate with your challenges? Let's discuss your needs. We're here to brainstorm and explore the potential of enhancing your analytics capabilities.