Serving data analytics data by microservices

83 Views Asked by At

Suppose, I have a microservice that owns an OLTP database. The service collects data from event stream and aggregates it into a database. As a result of the aggregation, it's able to tag each user, for example: user has made its first purchase, user has activated a welcome bonus, user has filled and confirmed its email, etc. The microservice serves these tags via API to external "banner promotion service" that is able to promote a particular action to the user, based on the conveyed tags.

Now, time pass, and eventually, data analytics team is came to play. They offer a table in the DWH (in OLAP database) that contains aggregated purchase activity of each user: the user is in top 100 by turnover, the user have participated in 3 promotions in a row and etc. I want to include this data in my service as a new set of tags.

So, I have three options of doing this:

  1. I can eliminate the usage of microservice database and use data only from the DWH. But I will loose the ability to serve tags in near real-time. The data analytics team will need to prepare the data and load it into DWH. Another downside is that the production microservice is now under control of the different team, and the data analytics team will need to coordinate its work with the development team that owns the microservice.

  2. I can ignore the DWH and use the data from microservice database. But I will loose the ability to serve tags based on the data analytics team's work. The reproduction of DWH data is not only complex to accomplish in the microservice environment, but also it requires access to many other data sources. Not really an option.

  3. I can combine the data from both sources. But in this case, I'll have a microservice that works with two databases which doesn't look like a "microservice way".

Something tells me, that the third approach is correct. It can be improved by using a Data Mart that will be prepared by data analytics. But where the Data Mart should be prepared and constantly? If it's a separate database then it bothers me that the service will have two databases. And if it's a table in the microservice database, then it bothers me that the data analytics team can potentially affect the performance of the microservice by redelivering the data.

Overall, the question is: What is the most common way of solving this problem? What is the best practice? I have found tons of information about how to deliver raw data from OLTP to OLAP, but I haven't found any information about how to serve the results of data analytics team's work back to OLTP.

Maybe this question is a bit too broad, in this case I would be grateful for any links to the articles or books that cover this topic.

0

There are 0 best solutions below