> What on earth is this article trying to accomplish?

Most articles explain building event-driven systems from a greenfield point of view. This article is for when you want to build an event-driven system but you already have brownfield relational data.

Four years ago I heard "Kafka IS your database".

I thought maybe these insane people (probably parroting some tech company enterprise penetration propaganda e.g. Confluent) would have a better story, but... no.

Anyway, yeah, sure, keep logs. But a lot of that article about commands and events is something that only exists if you had one ubiquitous language, system, and OS. You know, the almost literal "seamless" where there aren't any seams.

Sure that will probably plug into some enterprise bus and enterprise integration and enterprise ... anyway.

Competent developers will understand what events to preserve and log and possibly allow retry/repeats.

Anyone who has looked at a Splunk bill will realize that just storing all the logs everywhere is very expensive, which is another way of saying "wasteful". But any generic enterprisey event system will basically start and end with splunk-level log aggregation and kinda-analysis.

Dealt with something like this when I joined a previous job years ago, Kafka was the main data store, microservices had their state in memory they would build at startup by processing their whole history of events (that was ridiculously slow for some of them). Then GDPR came in place and the whole “keep everything in Kafka forever” had to face the reality of “not allowed to keep PII for longer than 30 days” :)

(Before someone suggests it, no messages weren’t encrypted, just throwing the key away wasn’t an option)

> ... they would build at startup by processing their whole history of events (that was ridiculously slow for some of them)

There is a basic technique to solve this, you snapshot every "nth" event

Would be expensive but you can do a copy and replace to keep the data you need, on a new topic.

If you don't need the data, then you don't need it.

Eh. The model he describes is actually standard for analytics.

And because it's standard there are literally databases designed and optimized to do what he says. It's not madness when it already exists and is really common.

Think, redshift, snowflake, biq query, clickhouse..

Additionally their already exists user interfaces and web services that already do what he says.

Datadog, splunk, Google analytics... Anything related to logs, analytics and aggregation of those analytics. What he proposes actually already exists.

That being said I don't agree with the articles point to replace everything with this model. Usually these types of services target very specific use cases.

I think your reaction is a bit extreme here. I don't agree with his proposed model but I see where he's coming from and it's not that the model won't work... It's been proven to work from all the examples I gave above.

The problem with it is that it's just slower and much more complicated. But his proposal does increase the capabilities of your data.

You can increase speed by having a pre-caching layer for your aggregations. Basically what was originally your static store is now a caching layer where the developer or user pre specifies an aggregation that the system should count live as the events come in as well as throwing the events into the event db. If when querying for that aggregation you get a "cache miss" then it hits the event layer and has to do the aggregation job live.

So essentially if you build it like this you have all the capabilities and speed of your original static data store but now you have the ability to re aggregate events differently so you have MORE ways to deal with your data. It can work and it will have more features its just really really really complicated to make an entire system centered around events. Additionally theres also a boatload of extra data to deal with which is another engineering problem.

That's why when people do build these systems it's usually centered around some business requirement that absolutely needs this ability to dynamically query and aggregate events. Logs and analytics being the two big ones. Or some service to data scientists as well.

The theory behind it is attractive. All static data can be represented as a series of events. In fact static data is simply the result of a certain of aggregation query on an event database. It's attractive to use smaller primitives in programming and build higher level abstractions through composition so this style of event driven services seems more fundamental and proper. But of course like I said there's practical issues with it when you look past the theory such that this model is usually only applied to the specific use cases I mentioned above.

So there is a failure here. Not of your intelligence. Failure of your experience.

And as I side note I agree with you on the tone of the article. He's trying to be witty but he's trying too hard.

Relational and event-driven aren't exclusive concepts, that's the problem with the article. Also, it'd help to have a real example of the solution it proposes, since we all know the "old" way it describes is in Postgres/MySQL/whatever.