Optimize your data access by using CQRS Architecture Pattern — Part I
A theoretical and practical approach
Have you ever wondered how many times you have started to develop a new service with a simple CRUD architecture for a certain domain object from your bounded context and that was “ok” at that moment, whilst the surrounding ecosystem keeps growing and scaling over time till the point you start noticing either (or both):
- The need to execute queries around your object becomes complex to deal with (multiple HTTP calls to other services, expensive joins across tables, etc).
- Performance degradation on your writes since your service ended up having more reading operation than writes.
Certainly, some of the issues pointed out above could be sorted out in many ways. From the top of my head, I can come up with some data patterns such as sharding strategies, materialized views on DB, data reporting offload (data warehouse), different database access techniques for reads and updates, and there might be some others for sure by using a different combination of patterns and approaches that solves it too.
However, I will be focusing this writing on one of my favorites which is the CQRS pattern.
What problems does it solve?
When using traditional architectures like CRUD, where the same data model is used to update and query a database for large-scale solutions, it could end up being a burden. For example:
- The read endpoint may perform multiple queries on different sources on the query side to return complex DTOs mappings with different shapes. And we already know that mappings could become quite complicated.
- On the writes side, the model may implement multiple and complicated business rules to validate creation and update operations.
- We might want to query the model in other ways, maybe collapsing multiple records into one, or aggregating more information to the model that is not currently available on its domain, or just change the way the queries look through the records by using some secondary fields as a key.
As a result, our CRUD service around our model object starts doing too many things and gets worst when it grows. Then is when this pattern comes into our tool-belt to help us out to solve these scalability issues.
The Pattern!
CQRS acronym states for Command and Query Responsibility Segregation. Its main purpose is based on the simple idea of separating the data mutation operations (command) from the reading operations (query). To accomplish this, it separates reads and writes into different models, using commands for creations/updates, and queries to read data from them.
As revealed by the above diagram, you will notice there is a queue of events that connects both writes and reads worlds by pushing onto the topic an event, every time an instance of our domain is being created/updated on the write side. Then after that, the query service will read from the incoming event, denormalize, enrich, slice, and dice the data to create query-optimized models and store them to be read later on.
Particularly, I focus this article series on leveraging the CQRS pattern by adding Event Sourcing architecture into the mix. It suits properly when we want to keep this flow with a clear separation of concerns, asynchronous, and also to utilize the proper database engines aiming for query performance (let’s say a SQL database for writings and a NoSQL for querying operations over materialized view optimized for queries to avoid expensive joins).
In addition to this, when we use Event Sourcing architecture, the event topic will become our golden source of data, since it could be used at any time to replay the whole collections of events and reproduce the current state of the data. So that it is possible for us to asynchronously read the queue from the beginning and generate a new set of materialized views from the original data when the system evolves, or when the read model must change. The materialized views are in effect a durable read-only cache of the data.
Yet another benefit of having separated worlds is the chance to scale both separately, resulting in fewer lock contentions. Also by separating the models makes them more flexible and eases the maintenance due to most of the complex business logic goes into the write model. The read one can be relatively simple.
When is this pattern a convenient solution?
Like any pattern, CQRS is useful sometimes, but not always. We have learned on many occasions that there is no silver bullet to kill all our issues. But, it is going to be useful under these possible requirements:
- The performance of data reads must be fine-tuned separately from the performance of data writes, especially when the number of reads is much greater than the number of writes. In this scenario, you can scale out the read model, but run the write model on just a few instances.
- Scenarios where one team of developers can focus on the complex domain model that is part of the write model, and another team can focus on the read model and the user interfaces.
- The system is expected to evolve and might contain multiple versions of the model, or where business rules change regularly.
- Integration with other systems, especially in combination with event sourcing, where the temporal failure of one subsystem shouldn’t affect the availability of the others.
- Read data that is eventually consistent is allowed. Since the asynchronous nature of this pattern.
Summary
As I stated above, this is not the solution to all our scaling/querying issues. We should be very cautious when using the CQRS pattern. It is useful sometimes but not always. Many systems do fit and do work very well with just a CRUD model because they are simple enough and consequently would be a waste of time and turn something easy-to-manage out on a very complex architecture to implement and maintain if we were looking to switch to the CQRS pattern. So try to keep things simple as possible unless it is required. Remember that reading from a queue and making all the data transformations to denormalize the original input could be very expensive regarding time consumption and resource usage.
However, despite the complexity, I can easily find this pattern very useful when facing problems like the need to have multiple ways to query our model without affecting write performance. Not only that, but also it is worth highlighting the fact we can just replay the whole queue to create new views of the original data to make queries simpler and inexpensive because we could avoid table joins and data merges.
Next Steps
Certainly, there are many other aspects of this pattern to be addressed in a longer article, but I wanted to highlight here at least the core principles that rule this architecture. So that, let's jump right into the second part of this series to see a practical example of this pattern and how the moving part interaction is by coding some silly examples using GoLang, Kafka, MongoDB, and Cassandra. See you at the next one!
[UPDATE] Part II is ready for you guys!
