Brahms: Byzantine resilient random membership sampling

Abstract

We present Brahms, an algorithm for sampling random nodes in a large dynamic system prone to malicious behavior. Brahms stores small membership views at each node, and yet overcomes Byzantine attacks by a linear portion of the system. Brahms is composed of two components. The first is an attack-resilient gossip-based membership protocol. The second component extracts independent uniformly random node samples from the stream of node ids gossiped by the first. We evaluate Brahms using rigorous analysis, backed by simulations, which show that our theoretical model captures the protocol’s essentials. We study two representative attacks, and show that with high probability, an attacker cannot create a partition between correct nodes. We further prove that each node’s sample converges to an independent uniform one over time. To our knowledge, no such properties were proven for gossip protocols in the past.

Introduction

We consider the problem of sampling random nodes (sometimes called peers) in a large dynamic system subject to adversarial (Byzantine) attacks. Random node sampling is important for many scalable dynamic applications, including neighbor selection in constructing and maintaining overlay networks [23], [32], [35], [37], selection of communication partners in gossip-based protocols [13], [18], [21], data sampling, and choosing locations for data caching, e.g., in unstructured peer-to-peer networks [34].

Typically, in such applications, each node maintains a set of random node ids that is asymptotically smaller than the system size. This set is called the node’s local view. We consider a dynamic system, subject to churn, whereby the set of active nodes changes over time. Local views in such a system must continuously evolve to incorporate new active nodes and to remove ones that are no longer active. By using small local views, the maintenance overhead is kept small. In the absence of malicious behavior, small local views can be effectively maintained with gossip-based membership protocols [1], [21], [22], [26], [43], which were proven to have a low probability for partitions, including under churn [1].

Nevertheless, adversarial attacks present a major challenge for small local views. Previous Byzantine-tolerant gossip protocols either considered static settings where the full membership is known to all [19], [33], [39], or maintained (almost) full local views [9], [28] (i.e., views that include all the nodes in the system), where faulty nodes cannot push correct ones out of the view (please see Section 2 for more detailed discussion of previous work). In contrast, small local views are susceptible to poisoning with entries (node ids) originating from faulty nodes; this is because in a dynamic system, nodes must inherently accept new ids and store them in place of old ones in their local views. In Section 3, we illustrate that traditional gossip-based membership is highly vulnerable to adversary attacks, which can quickly poison the entire views of correct nodes.

It is even more challenging to provide independent uniform samples in such a setting. Even without Byzantine failures, gossip-based membership only ensures that eventually the average representation of nodes in local views is uniform [1], [22], [26], and not that every node obtains an independent uniform random sample. Faulty nodes may attempt to skew the system-wide distribution, as well as the individual local view of a given node.

This paper addresses these challenges. In Section 4, we present Brahms, a membership service that stores a sub-linear number of ids (e.g., $\Theta \left(\sqrt[3]{n}\right)$ in a system of size n) at each node, and provides each node with independent random node samples that converge to uniform ones over time. The main ideas behind Brahms are (1) to use gossip-based membership with some extra defenses to make it viable (in the sense that local views are not solely composed of faulty ids) in an adversarial setting; (2) to recognize that such a solution is susceptible to attacks that may bias the views, i.e., cause certain nodes to be over-represented in views while others are under-represented (we precisely quantify the extent of this bias mathematically); and (3) to correct this bias at each node. Specifically, each node maintains, in addition to the gossip-based local view, an unbiased sample list of nodes.

To achieve the latter, we introduce Sampler, a component that obtains uniform samples out of a data stream in which elements recur with an unknown bias. Sampler uses min-wise independent permutations [14], and stores one element of the stream at a time. In Brahms, the data stream is comprised of gossiped ids, from which Samplers obtain independent uniformly random id samples, and store them in the sample list. By using such history samples from the sample list to update part of the local view, Brahms achieves self-healing from partitions that may occur with gossip-based membership. In particular, nodes that have been active for sufficiently long (we quantify how long) cannot be isolated from the rest of the system, with high probability. The use of history samples is an example of amplification, whereby even a small healthy sample of the past can boost the resilience of a constantly evolving view. We note that only a small portion of the view is updated with history samples, e.g., 10%. Therefore, the protocol can still deal effectively with churn.

In Section 5, we define the attacker’s goals and the corresponding attack strategies, under which we evaluate Brahms. We consider two possible goals for an attacker. First, we study attacks that attempt to maximize the representation of faulty ids in local views at any given time. This goal is achieved by a uniform attack, whereby the attacker equally divides its power among all correct nodes. Second, we consider an attacker that aims to partition the network. The easiest way to do so is by isolating one node from the rest [1]. Since samples help prevent isolation, we analyze the most adverse circumstances, where an attack is launched on a new node that joins the system when its samples are still empty, and when it does not yet appear in views or samples of other nodes. We further assume that such a _targeted attack on the new node occurs in tandem with an attack on the entire system, as described above.

One of the important contributions of this paper is our mathematical analysis, which provides insights to the extent of damage that an attacker can cause and the effectiveness of various mechanisms for dealing with them. Extensive simulations of Brahms with up to 4000 nodes validate the few simplifying assumptions made in the analysis. We first show (in Section 6) that whenever the set of nodes remains connected, the sample lists converge to independent uniformly random selections from among all nodes. We further show that if views are of size $\Omega \left(\sqrt[3]{n}\right)$, then the convergence rate is bounded independently of the system size. Section 7 then analyzes the local views generated by the gossip process and shows that under certain circumstances, they preserve the connectivity required for uniform samples.

Specifically, for the attack goal of maximizing the representation of faulty ids (Section 7.1), we show that under certain conditions on the adversary, even without using history samples, the portion of faulty ids in local views generated by Brahms’s gossip process is bounded by a constant smaller than one. (Recall that the over-representation of faulty ids is later fixed by Sampler; the upper bound on faulty ids in local views ensures Sampler has good ids to work with.)

Next, we consider the goal of isolating a node (Section 7.2). The key to proving that Brahms prevents, with high probability, an attacked node’s isolation is in comparing how long it takes for two competing processes to complete: on the one hand, we provide a lower bound on the expected time to poison the entire view of the attacked node, assuming there are no history samples at all. On the other hand, we provide an upper bound on how fast history samples are expected to converge, under the same attack. Whenever the former exceeds the latter, the attacked node is expected to become immune to isolation before it is isolated. We prove that with appropriate parameter settings, this is indeed the case.

Finally, we simulate the complete system (Section 8), and measure Brahms’s resilience to the combination of both attacks. Our results show that, indeed, Brahms prevents the isolation of attacked nodes, its views never partition, and the membership samples converge to perfectly random ones over time.