Lambda Architecture2046 2018-10-23 10:30
Why lambda architecture? #
To solve three problems introduced by big data
- Accuracy (好)
- Latency (快)
- Throughput (多)
e.g. problems with scaling a pageview service in a traditional way
- You start with a traditional relational database.
- Then adding a pub-sub queue.
- Then scaling by horizontal partitioning or sharding
- Fault-tolerance issues begin
- Data corruption happens
The key point is that X-axis dimension alone of the AKF scale cube is not good enough. We should introduce Y-axis / functional decomposition as well. Lambda architecture tells us how to do it for a data system.
What is lambda architecture? #
If we define a data system as
Query = function(all data)
Then a lambda architecture is
batch view = function(all data at the batching job's execution time) realtime view = function(realtime view, new data) query = function(batch view. realtime view)
Lambda architecture = CQRS (batch layer + serving layer) + speed layer