“After working for
a long time on many different distributed systems, you learn to
appreciate how complicated the littlest things can become.”
This is the opening
sentence of the blog which Yahoo research tells the story of how
ZooKeeper came into being. It not only precisely describes the
engineering effort needs to put into a mature distributed systems,
also provides the major motivation of system like ZooKeeper. Although
there exist various form of applications that can be distributed and
gain benefits like fault-tolerant and scalability, some very general
requirements are shared among them:
(1) Distributed
applications need some “ground truth”.
The truth can come
from an oracle embedded in the system or a service externally
available. Some examples of the “truth” are leader election and
failure detector. [Chandra et al. 1996] has shown that the weakest
failure detector to solve consensus in an asynchronous system is of
class $\diamond S$, eventual weak accuracy, thus it's safe to say we
only require “partial truth”.
(2) The service
needs to be simple.
Simple means easy to
understand and easy to use. This blog elaborates the reason behind
consensus is no longer used as a library is that it required
developer to be experts in distributed systems to understand the
operational characteristics of failure modes and how to debug them.
(3) The service
needs good performance.
Consensus is
expensive. Even so, a service should make full effort of it to
provide performance.
Therefore, after a
bunch of distributed projects with animal names, Yahoo designed
ZooKeeper which uses Zab (Paxos-like consensus) algorithm to
satisfy (1), uses hierarchal namespace and file system like
model to satisfy (2), uses non-blocking pipelined architecture
to satisfy (3).
Zab provides
different properties than Paxos. It's designed for coordination with
high availability thanks to primary ordered passive state machine
replication (SMR).
ZooKeeper's file
system like API model has three advantages: one, file system API is
easy to understand by more developers; two, hierarchal namespace
makes more use cases possible of the service (e.g. discovery); three,
partition the clients of different use. However, it is unlike a real
file system, because some features (e.g. rename) is not provided due
to hard to implement and possibly in the cost of system performance.
Pipelined
architecture is the different request processors we described in few
blogs earlier. Different request processor chain is the main reason
for different behavior of leader and follower nodes.
Early version of
ZooKeeper is written by one engineer in three months. To developer's
surprise, the service can actually provide all kinds of coordination
primitives than their initial request. Same story happened to
Google's distributed lock service Chubby, that can be used for more
tasks other than locks. Since release, more contributors have added
rich features into ZooKeeper, and years of work to implement,
optimize and test made ZooKeeper mature and popular.
So far, ZooKeeper is
the only consensus system that pass the test, whereas etcd, based on
Raft algorithm failed in partitions.
[Chandra et al. 1996] Chandra, Tushar Deepak, and Sam Toueg. "Unreliable failure detectors for reliable distributed systems." Journal of the ACM (JACM) 43.2 (1996): 225-267.
