At bol.com (the biggest online retailer in the Benelux) we want to help the customer find what they wanted. To automate this process we need to understand what products/promotions we showed them and which of those made them happy. With the fine grained personalization that has been introduced over the last few years we see that just measuring ‘what page’ we showed (like all the standard web analytics systems do) is no longer enough. So we need something different. In order to get a solution that will support our business for the coming years we raised the bar to the top: Measure everything and analyze in near-realtime.

<!--break-->

In this talk Niels Basjes will explain the project “Measuring 2.0”, our next generation web analytics measuring and processing stack, that is to go live in the spring of 2016. Niels will go into

• the custom built measuring system that will produce over 50000 measurements per second
• the processing system and the algorithms implemented with Apache Flink
• why we did not choose Storm or Spark for this task
• the development and operational hurdles needed to make this type of solution run in production
• the architectural concepts to make this data available in the personalization services we have