It’s important to know your users’ preferences and behavior in the E-marketplace world. If you can quickly understand who your users are, you can optimize the user journey on the E-marketplace platform by presenting relevant products to the user, and by improving the relevance of search results. One way to leverage user preferences is to calculate a passive user profile based on the user’s interactions with the E-marketplace platform. At mobile.de, Germany's largest online vehicle marketplace, we include in the user profile information such as likelihood of a user to select different car colors, price distribution, mileage distribution, etc.
The real challenge is designing a scalable system that can calculate profiles for different users in real-time and serve those profiles via a REST API to other stakeholders.

At mobile.de, we reviewed some of the most popular open-source stream processing solutions to consider possible architecture designs for the problem. In a nutshell updating user profile in real time is actually a stateful stream processing system in which the state is the user profile, the state key could be the user ID, and the state update operation can be simple as counter increment or an average/variance update.

The main focus of this talk would be: how to store the stream state? Which is one of the most important choices to be made before designing a stateful stream processing system.

During this talk I would compare between a local storage option and an external (global) storage option, and would try to present the different tradeoffs as a result from selecting each of the mentioned options above.

This talk is sponsored by our Gold partner ebay tech.