Freedomlayerhttps://www.freedomlayer.org/2017-08-29T15:18:00+00:00Fragmentos fragmentation system2017-08-29T15:18:00+00:002017-08-29T15:18:00+00:00realtag:www.freedomlayer.org,2017-08-29:/fragmentos-fragmentation-system.html<p><strong>work in progress!</strong></p>
<h2>Abstract</h2>
<p>Fragmentos is a fragmentation system allowing to send large datagrams over
protocols that support only small datagrams. Compared to traditional
fragmentation systems, Fragmentos sends about twice more data while achieving
improved delivery rate.</p>
<h2>Intro</h2>
<p>Given two nodes <span class="math">\(x\)</span> and <span class="math">\(y\)</span> on the internet, if <span class="math">\(x\)</span> attempts …</p><p><strong>work in progress!</strong></p>
<h2>Abstract</h2>
<p>Fragmentos is a fragmentation system allowing to send large datagrams over
protocols that support only small datagrams. Compared to traditional
fragmentation systems, Fragmentos sends about twice more data while achieving
improved delivery rate.</p>
<h2>Intro</h2>
<p>Given two nodes <span class="math">\(x\)</span> and <span class="math">\(y\)</span> on the internet, if <span class="math">\(x\)</span> attempts to send a large
<span class="caps">UDP</span> datagram <span class="math">\(y\)</span>, it is likely that the datagram will be fragmented, in the
lower <span class="caps">IP</span> layer, to a few <span class="caps">IP</span> packets. Those <span class="caps">IP</span> packets will be reconstructed
somewhere on a router on the way, or possibly at the endpoint <span class="math">\(y\)</span>.</p>
<p>Traditional <span class="caps">IP</span> Packet fragmentation works by splitting a large <span class="caps">IP</span> packet into a
few smaller packets. The problem with this method is that the original large
packet is less likely to arrive at its destination, as the loss of any of the
fragments will lead to the loss of the original large packet. </p>
<p>Thinking about a naive probabilistic model, if an <span class="caps">IP</span> packet sent from <span class="math">\(x\)</span> to
<span class="math">\(y\)</span> arrives at <span class="math">\(y\)</span> with probability <span class="math">\(p\)</span>, then a datagram which was fragmented
to two <span class="caps">IP</span> packets will arrive at probability <span class="math">\(p^2\)</span>. If <span class="math">\(p = \frac{3}{4}\)</span>, then
<span class="math">\(q := p^2 = \frac{9}{16}\)</span>, which represents much lower odds for successful delivery.</p>
<p>This document presents a different fragmentation method which increases the
likelihood of successful delivery given the model discussed above. For example,
Instead of splitting a large datagram into two packets, we create three shares
of the datagram in a way that every two shares are enough to reconstruct the
original datagram. </p>
<p>In this case, if <span class="math">\(x\)</span> wants to send the datagram to <span class="math">\(y\)</span>, <span class="math">\(x\)</span> sends all the three
shares of the datagram to <span class="math">\(y\)</span>. In the naive model described above where the
odds of successful delivery for one <span class="caps">IP</span> packet are <span class="math">\(p\)</span>, we obtain that the odds
for successful arrival for the complete datagram are
<span class="math">\(q := p^3 + 3\cdot p^2 \cdot (1-p)\)</span>. For <span class="math">\(p = \frac{3}{4}\)</span>, this will result in
<span class="math">\(q = 0.84375\)</span>. Surprisingly in this case <span class="math">\(q > p\)</span>.</p>
<h2>Splitting a small chunk</h2>
<p>Given a chunk of data <span class="math">\(W\)</span> of size <span class="math">\(b\)</span> bytes, we want to split it to <span class="math">\(2b-1\)</span>
shares, so that every <span class="math">\(b\)</span> shares are enough to reconstruct the original chunk.</p>
<p>Using information theoretic considerations we can
conclude that each share should be of size at least <span class="math">\(len(M) / b\)</span>, so that the
reconstruction algorithm gets at least <span class="math">\(b \cdot (len(M) / b) = b\)</span> bytes of input.</p>
<p>We will do this in a method similar to <a href="https://en.wikipedia.org/wiki/Shamir%27s_Secret_Sharing">Shamir’s secret
sharing</a>. We treat the
given chunk <span class="math">\(W\)</span> as the coefficients of a polynomial <span class="math">\(P_W\)</span> over the <a href="https://en.wikipedia.org/wiki/Finite_field#GF.2816.29">Galois
field</a> <span class="math">\(Gf(256)\)</span>. The
first byte of the message <span class="math">\(W\)</span> is considered to be the most coefficient of
monomial of the highest degree in the polynomial.</p>
<p>A share of the chunk <span class="math">\(W\)</span> is a pair of bytes <span class="math">\((i, P_W(i))\)</span>, for <span class="math">\(0 \leq i < 2b -
1\)</span>. In other words, a share is a point that the polynomial <span class="math">\(P\)</span> goes through.</p>
<p>Given <span class="math">\(b\)</span> shares of the chunk <span class="math">\(W\)</span> we can always reconstruct a polynomial <span class="math">\(Q\)</span> of
degree <span class="math">\(b-1\)</span> that goes through the set of points provided as shares. For
example, we could do this using <a href="https://en.wikipedia.org/wiki/Lagrange_polynomial">Lagrange
interpolation</a>.</p>
<p>As <span class="math">\(Q\)</span> and <span class="math">\(P_W\)</span> are two polynomials of degree <span class="math">\(b-1\)</span> that agree on <span class="math">\(b\)</span> points,
they must be equal. Therefore using any <span class="math">\(b\)</span> shares we can reconstruct the
<span class="math">\(P_W\)</span> and the chunk <span class="math">\(W\)</span>.</p>
<p>We picked <span class="math">\(Gf(256)\)</span> with modulo polynomial <span class="math">\(x^8 + x^4 + x^3 + x^2 + 1\)</span>
(Represented at <code>0x1D</code> in byte form) as the underlying field. <span class="math">\(GF(256)\)</span> has
exactly <span class="math">\(256\)</span> elements, which means that we have an exact mapping between a
byte value and an element of the field, thus avoiding various encoding and
decoding problems.</p>
<p>On the other hand, the choice of <span class="math">\(GF(256)\)</span> imposes a limitation on the amount
of shares we can produce for one chunk. As every share <span class="math">\((i, P_W(i))\)</span> should have
a unique <span class="math">\(i\)</span>, we can have no more than <span class="math">\(256\)</span> shares. This means that <span class="math">\(2b - 1
\leq 256\)</span>, hence <span class="math">\(b \leq 128\)</span>.</p>
<h2>Splitting a large message</h2>
<p>Given a message <span class="math">\(M\)</span> of certain size <span class="math">\(len(M)\)</span>, we would like to split it to
<span class="math">\(2b-1\)</span> data shares, so that every <span class="math">\(b\)</span> data shares could be used to reconstruct
the original chunk of data. Note that in the previous section we solved this
problem for <span class="math">\(M\)</span> with <span class="math">\(len(M) = b\)</span>. Here we want to solve this problem for a
message <span class="math">\(M\)</span> of arbitrary size.</p>
<p>First we divide <span class="math">\(M\)</span> into chunks of size <span class="math">\(b\)</span> bytes. If <span class="math">\(len(M)\)</span> is not
divisible by <span class="math">\(b\)</span>, we add padding <span class="math">\(0\)</span> bytes to the end of <span class="math">\(M\)</span>. This means that
the last chunk has possibly some <span class="math">\(0\)</span> bytes in the end.</p>
<p>For example, if <span class="math">\(M\)</span> is the message <code>12 34 56 78 90 ab cd ef 55</code> (Every two
characters are one byte, base 16) and <span class="math">\(b = 4\)</span>, we will get the following
chunks (Every line represents a chunk):</p>
<div class="highlight"><pre><span></span>12 34 56 78
90 ab cd ef
55 00 00 00
</pre></div>
<p>Note that the last chunk has three <span class="math">\(0\)</span> padding bytes.</p>
<p>Next, we divide each chunk <span class="math">\(W\)</span> of size <span class="math">\(b\)</span> bytes into <span class="math">\(2b-1\)</span> small shares. We
do this as described in the previous section “splitting a chunk”. For every
resulting small share <span class="math">\((i, P_W(i))\)</span> we take the value <span class="math">\(P_W(i)\)</span> and add it to
data share number <span class="math">\(i\)</span>.</p>
<p>Eventually we will have <span class="math">\(2b - 1\)</span> data shares, each one having <span class="math">\(v := \lceil len(M) /
b \rceil\)</span> bytes. Given that <span class="math">\(M\)</span> is divided to the chunks <span class="math">\(W_0, W_1, W_2
\dots W_{v-1}\)</span>, byte number <span class="math">\(k\)</span> in data share <span class="math">\(i\)</span> is <span class="math">\(P_{W_k}(i)\)</span>.</p>
<p>Continuing the previous example of the message <span class="math">\(M\)</span>: <code>12 34 56 78 90 ab cd ef
55</code>, every chunk is split into <span class="math">\(2b - 1 = 7\)</span> small shares:</p>
<div class="highlight"><pre><span></span> ds0 ds1 ds2 ds3 ds4 ds5 ds6
----------------------------
12 34 56 78 -> 78 08 94 88 ae ab 05
90 ab cd ef -> ef 19 0a bb 6d 14 8a
55 00 00 00 -> 00 55 92 24 e4 c1 3d
</pre></div>
<p>And we obtain the following <span class="math">\(2b - 1 = 7\)</span> data shares:</p>
<div class="highlight"><pre><span></span><span class="n">ds0</span><span class="o">:</span> <span class="mi">78</span> <span class="n">ef</span> <span class="mi">00</span>
<span class="n">ds1</span><span class="o">:</span> <span class="mi">08</span> <span class="mi">19</span> <span class="mi">55</span>
<span class="n">ds2</span><span class="o">:</span> <span class="mi">94</span> <span class="mi">0</span><span class="n">a</span> <span class="mi">92</span>
<span class="n">ds3</span><span class="o">:</span> <span class="mi">88</span> <span class="n">bb</span> <span class="mi">24</span>
<span class="n">ds4</span><span class="o">:</span> <span class="n">ae</span> <span class="mi">6</span><span class="n">d</span> <span class="n">e4</span>
<span class="n">ds5</span><span class="o">:</span> <span class="n">ab</span> <span class="mi">14</span> <span class="n">c1</span>
<span class="n">ds6</span><span class="o">:</span> <span class="mi">05</span> <span class="mi">8</span><span class="n">a</span> <span class="mi">3</span><span class="n">d</span>
</pre></div>
<p>Every <span class="math">\(b = 4\)</span> shares from these <span class="math">\(2b - 1 = 7\)</span> data shares are enough to
reconstruct the original message <span class="math">\(M\)</span>.</p>
<h2>Fragmentos messages structure</h2>
<p>A message <span class="math">\(M\)</span> from is sent from <span class="math">\(x\)</span> to <span class="math">\(y\)</span> by transmitting a few Fragmentos
messages. Each <strong>Fragmentos message</strong> is of the following form:</p>
<div class="highlight"><pre><span></span>- messageId [8 bytes]
- b [1 byte]
- shareIndex [1 byte]
- shareData [variable amount of bytes]
- errorCorrection [8 bytes]
</pre></div>
<p>The actual data that we split to data shares is: </p>
<p><code>T := nonce8 || paddingCount || M || padding</code></p>
<p>Where <code>||</code> means concatenation. <code>nonce8</code> is of size <span class="math">\(8\)</span> bytes, <code>paddingCount</code>
is of size <span class="math">\(1\)</span> byte.</p>
<p>The value <code>paddingCount := (b - ((8 + 1 + len(M)) % b)) % b</code> is the amount of <span class="math">\(0\)</span>
padding bytes at the end of the message <span class="math">\(M\)</span>. Recall that we need padding bytes
at the end of <span class="math">\(M\)</span> to make sure that the data we split to data shares, <span class="math">\(T\)</span>, is
of size divisible by <span class="math">\(b\)</span>. We use the value of <code>paddingCount</code> to be able to cut
the padding bytes at the receiving side.</p>
<p><code>nonce8</code> is a random 8 bytes nonce that is generated at the time of sending the
message <span class="math">\(M\)</span>. It is used as means for the receiver end to distinguish between
identical messages sent during the same short period of time. For two identical
messages, different <code>nonce8</code> values will generate different <code>messageId</code>-s.</p>
<p><code>messageId</code> is obtained by calculating <code>sha256</code> over <code>T</code>, taking only the first
<span class="math">\(8\)</span> bytes of the result.</p>
<p><code>b</code> represents the amount of shares that are required to reconstruct the
original message. It is necessary that <span class="math">\(b \in [1, 128]\)</span>. Any other value of <span class="math">\(b\)</span>
is illegal. A message with illegal <span class="math">\(b\)</span> value must be ignored.</p>
<p><code>shareIndex</code> is the index of the current share. There are <span class="math">\(2b - 1\)</span> shares,
therefore <span class="math">\(0 \leq shareIndex < 2b - 1\)</span>. Any other value of <span class="math">\(shareIndex\)</span> is
illegal. A message with illegal <span class="math">\(shareIndex\)</span> value must be discarded.</p>
<p><code>shareData</code> is the actual data of this share. Its length could be deduced by
subtracting the total size of all the other fields from the total size of the
datagram. It contains a share of the data <code>T</code>.</p>
<p><code>errorCorrection</code> is an <span class="math">\(8\)</span> bytes <a href="https://en.wikipedia.org/wiki/Reed%E2%80%93Solomon_error_correction">Reed
Solomon</a>
error correction code. Currently we use the implementation of the
<a href="https://docs.rs/reed-solomon/0.1.6/reed_solomon/">reed-solomon</a> rust crate for
this functionality. The error correction code is applied for whole contents of
the Fragmentos message. It should be able to detect <span class="math">\(8\)</span> bytes errors, and fix
the message given that at most <span class="math">\(4\)</span> byte errors have occurred.</p>
<h2>Maximum Fragmentos datagram</h2>
<p>We assume that we use a protocol where we can not send datagrams over <span class="math">\(n\)</span> bytes.
We begin by calculating the maximum size of datagram we can send using the
Fragmentos system over the given protocol.</p>
<p>We can split a message to at most <span class="math">\(256\)</span> shares, a restriction imposed by the
size of the field <span class="math">\(GF(256)\)</span>. Therefore <span class="math">\(2b - 1 \leq 256\)</span> and <span class="math">\(b \leq 128\)</span>.</p>
<p>Every message share resides inside a Fragmentos message. The rest of the fields
in a Fragmentos message (<code>messageId, b, shareIndex, errorCorrection</code>) take <span class="math">\(8 +
1 + 1 + 8 = 18\)</span> bytes. Hence we are left with <span class="math">\(n - 18\)</span> bytes for <code>shareData</code>.</p>
<p>Therefore we should be able to send a <code>T</code> data message of size at most <span class="math">\(128
\cdot (n-18)\)</span>. Recall that <code>T</code> contains <code>nonce8, paddingCount</code> which take
together <span class="math">\(8 + 1 = 9\)</span> bytes. This leaves <span class="math">\(d(n) := 128\cdot(n-18) - 9\)</span> bytes for <span class="math">\(M\)</span>.</p>
<p>If we use <span class="caps">UDP</span> as the underlying protocol for sending datagrams, we can pick as
an example the value <span class="math">\(n=512\)</span> as a safe size for a <span class="caps">UDP</span> packet. By safe we mean
that it is unlikely for the <span class="caps">UDP</span> packet to be fragmented by the lower <span class="caps">IP</span> layer.</p>
<p>As a result, we obtain <span class="math">\(d(n) = 128 \cdot (512 - 18) - 9 = 63223\)</span> bytes (close
to <span class="math">\(64KB\)</span>) as the maximum possible Fragmentos datagram.</p>
<h2>Sending a message</h2>
<p>We assume the value <span class="math">\(n\)</span> for the maximum size for a safe datagram in the
underlying datagram protocol.</p>
<p>Algorithm for sending a message <span class="math">\(M\)</span>:</p>
<ol>
<li>
<p>If <span class="math">\(len(M) > d(n)\)</span>, the message is too large. We abort.</p>
</li>
<li>
<p>Calculate <span class="math">\(b := \lceil\frac{8 + 1 + len(M)}{n - 18}\rceil\)</span>.</p>
</li>
<li>
<p>Construct <code>T := nonce8 || paddingCount || M || padding</code>.
<code>padding</code> should contain enough <code>\x00</code> bytes so that <code>T</code> will have length
that is divisible by <span class="math">\(b\)</span>. <code>paddingCount</code> is set to the amount of padding bytes.</p>
</li>
<li>
<p>Split <code>T</code> to <span class="math">\(2b-1\)</span> data shares.</p>
</li>
<li>
<p>For each data share of <code>T</code>, create a Fragmentos message and send it to the destination.
The message’s fields will be filled as follows:</p>
<ul>
<li><code>messageId = sha256(T)[0:8]</code></li>
<li><code>b</code></li>
<li><code>shareIndex</code> is the data share number.</li>
<li><code>shareData</code> is the data of the share.</li>
<li><code>errorCorrection</code> is calculated over all the previous fields of the message.</li>
</ul>
</li>
</ol>
<h2>Receiving a message</h2>
<h3>Structures to maintain in memory:</h3>
<ul>
<li>
<p><code>usedMessageIds</code>: A set of messageIds that were recently processed.
A messageId has to stay in this list for at least <code>PROCESSED_TIMEOUT</code>
seconds. Then it may be removed.</p>
</li>
<li>
<p><code>curMessages</code>: A dictionary for currently processed
messages, having messageIds as keys. Every entry contains:</p>
<ul>
<li>The value <code>b</code>.</li>
<li><code>shareLengh</code>: length (in bytes) of a share data. (All shares should have
exactly the same amount of bytes).</li>
<li>A set of the received data shares: <code>(shareIndex, shareData)</code> </li>
</ul>
</li>
</ul>
<h3>Upon receiving a Fragmentos message <code>F</code>:</h3>
<ol>
<li>
<p>Use <code>F.errorCorrection</code> error correction code to possibly correct the
message’s content. If not possible, discard the message.</p>
</li>
<li>
<p>If <code>F.messageId</code> is in <code>usedMessageIds</code>,
<code>usedMessageIds[F.massageId]</code> is refreshed, so that it will stay
another <code>PROCESSED_TIMEOUT</code> seconds before being cleaned up. The message is discarded.</p>
</li>
<li>
<p>If there exists an entry <code>entry</code> with the key <code>F.messageId</code> inside
<code>curMessages</code> but <code>entry.b != F.b</code> or <code>entry.shareLength !=
len(F.shareData)</code>, discard the message.</p>
</li>
<li>
<p>If there is no entry with the key <code>F.messageId</code> inside <code>curMessages</code>,
create a new entry with <code>messageId = F.messageId</code>, <code>b = F.b</code> and
<code>shareLength = len(F.shareData)</code>.</p>
</li>
<li>
<p>Given that the message was not discarded yet, denote by <code>entry</code> the relevant
entry from <code>curMessages</code> relevant for our message.</p>
</li>
<li>
<p>If <code>F.shareIndex</code> is already present in <code>entry</code>, discard the message.</p>
</li>
<li>
<p>Insert <code>(F.shareIndex, F.shareData)</code> as a new data share to
<code>entry</code>.</p>
</li>
<li>
<p>If the amount of shares in the entry is less than <code>b</code>, return.</p>
</li>
<li>
<p>Add <code>messageId</code> to <code>usedMessageIds</code>. </p>
</li>
<li>
<p>Remove <code>entry</code> from <code>curMessages</code>.</p>
</li>
<li>
<p>Reconstruct the shares to obtain <code>T := nonce8 || paddingCount || M || padding</code>. </p>
</li>
<li>
<p>If <code>sha256(T)[0:8] != entry.messageId</code>: remove the entry from <code>curMessages</code>
and return.</p>
</li>
<li>
<p>Extract <code>M</code> from <code>T</code> and return it as a received message.</p>
</li>
</ol>
<h3>Cleanup algorithm (Being ran periodically):</h3>
<ul>
<li>
<p>If an <code>entry</code> in <code>curMessages</code> is older than <code>PROCESSED_TIMEOUT</code> seconds,
remove it and add <code>entry.messageId</code> to <code>usedMessageIds</code>.</p>
</li>
<li>
<p>If a <code>messageId</code> in <code>usedMessagesIds</code> is older than <code>PROCESSED_TIMEOUT</code>
seconds, remove it.</p>
</li>
</ul>
<script type="text/javascript">if (!document.getElementById('mathjaxscript_pelican_#%@#$@#')) {
var align = "center",
indent = "0em",
linebreak = "false";
if (false) {
align = (screen.width < 768) ? "left" : align;
indent = (screen.width < 768) ? "0em" : indent;
linebreak = (screen.width < 768) ? 'true' : linebreak;
}
var mathjaxscript = document.createElement('script');
mathjaxscript.id = 'mathjaxscript_pelican_#%@#$@#';
mathjaxscript.type = 'text/javascript';
mathjaxscript.src = 'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.0/MathJax.js?config=TeX-AMS-MML_HTMLorMML';
mathjaxscript[(window.opera ? "innerHTML" : "text")] =
"MathJax.Hub.Config({" +
" config: ['MMLorHTML.js']," +
" TeX: { extensions: ['AMSmath.js','AMSsymbols.js','noErrors.js','noUndefined.js'], equationNumbers: { autoNumber: 'AMS' } }," +
" jax: ['input/TeX','input/MathML','output/HTML-CSS']," +
" extensions: ['tex2jax.js','mml2jax.js','MathMenu.js','MathZoom.js']," +
" displayAlign: '"+ align +"'," +
" displayIndent: '"+ indent +"'," +
" showMathMenu: true," +
" messageStyle: 'normal'," +
" tex2jax: { " +
" inlineMath: [ ['\\\\(','\\\\)'] ], " +
" displayMath: [ ['$$','$$'] ]," +
" processEscapes: true," +
" preview: 'TeX'," +
" }, " +
" 'HTML-CSS': { " +
" styles: { '.MathJax_Display, .MathJax .mo, .MathJax .mi, .MathJax .mn': {color: 'inherit ! important'} }," +
" linebreaks: { automatic: "+ linebreak +", width: '90% container' }," +
" }, " +
"}); " +
"if ('default' !== 'default') {" +
"MathJax.Hub.Register.StartupHook('HTML-CSS Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax['HTML-CSS'].FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"MathJax.Hub.Register.StartupHook('SVG Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax.SVG.FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"}";
(document.body || document.getElementsByTagName('head')[0]).appendChild(mathjaxscript);
}
</script>Passerby UDP Porter2017-08-24T20:18:00+00:002017-08-24T20:18:00+00:00realtag:www.freedomlayer.org,2017-08-24:/passerby-udp-porter.html<p><strong>work in progress!</strong></p>
<h2>Intro</h2>
<p>The Passerby routing protocol allows routing of messages in a network of nodes,
where every node <span class="math">\(v\)</span> is directly connected to a few other predefined nodes.
These nodes are called the direct neighbors of <span class="math">\(v\)</span>.</p>
<p>Passerby <span class="caps">UDP</span> Porter is a mechanism for maintaining direct neighbors for …</p><p><strong>work in progress!</strong></p>
<h2>Intro</h2>
<p>The Passerby routing protocol allows routing of messages in a network of nodes,
where every node <span class="math">\(v\)</span> is directly connected to a few other predefined nodes.
These nodes are called the direct neighbors of <span class="math">\(v\)</span>.</p>
<p>Passerby <span class="caps">UDP</span> Porter is a mechanism for maintaining direct neighbors for
Passerby as <span class="caps">UDP</span> connections.</p>
<h2>Initialization</h2>
<p>The passerby <span class="caps">UDP</span> porter of node <span class="math">\(v\)</span> begins with the following input:</p>
<ul>
<li>A list of known remote nodes: (nodePublicKey, ip, port)</li>
</ul>
<p>In every tuple of known remote node, ip and port are optional. This means that
it is possible for a <span class="math">\(a\)</span> to know about the public key of node <span class="math">\(b\)</span> without
knowing about his <span class="caps">IP</span> address. </p>
<p><span class="caps">TODO</span>: Fill in <span class="caps">UDP</span> related details:</p>
<ul>
<li>Possibly explain how to support larger datagrams?<ul>
<li>Sending multiple smaller datagrams using secret sharing?</li>
<li>How to verify secret sharing?</li>
<li>Possible error correction?</li>
</ul>
</li>
</ul>
<h2>Messages in a <span class="caps">UDP</span> porter link</h2>
<p><strong>Hello</strong> (L0)</p>
<ul>
<li>randValue (128 bit)</li>
</ul>
<p>This is the first message sent during the setup of a <span class="caps">UDP</span> porter link. It
introduces the identity of the remote host.</p>
<p>randValue is the first random value sent by this side of the connection. It
is required for sending EncMessage messages without the issue of replay attacks.</p>
<p><strong>EncMessage</strong> (L0)</p>
<ul>
<li>encSymmetricKey == Encrypt{key=recipientPublicKey}<ul>
<li>pastKeepaliveRandValue (A randValue from the near past)</li>
<li>symmetricKey</li>
<li>senderPublicKey</li>
<li>Sign{key=senderPublicKey}(keepaliveRandValue, symmetricKey, senderPublicKey)</li>
</ul>
</li>
<li>encDatagramData == Encrypt{key=symmetricKey, nonce=random}(
pastKeepaliveRandValue, L1Content)</li>
</ul>
<p>A message that contains an underlying Passerby routing engine message.</p>
<p><strong>KeepAlive</strong> (L1) // EncMessage</p>
<ul>
<li>randValue (128 bit)</li>
</ul>
<p>This message is sent periodically, to show that the link is still open.
It contains randValue, which is a random value to be used as a replay attack
countermeasure in EncMessage messages.</p>
<p>This message marks that the sender is actually the one that was stated in the
Hello message. It should be sent as the first EncMessage, immediately after
Hello message was received. It marks the beginning of the <span class="caps">UDP</span> porter link.</p>
<p><strong>UserMessage</strong> (L1) // EncMessage</p>
<ul>
<li>content</li>
</ul>
<p>Allows to send user messages over the <span class="caps">UDP</span> porter link.</p>
<h2>Forming a <span class="caps">UDP</span> Porter link</h2>
<p>If <span class="math">\(a\)</span> has the remote address of <span class="math">\(b\)</span> and there is not yet an open <span class="caps">UDP</span> porter
link between <span class="math">\(a\)</span> and <span class="math">\(b\)</span>, <span class="math">\(a\)</span> performs the following:</p>
<ol>
<li><span class="math">\(a\)</span> sends Hello message to <span class="math">\(b\)</span>. This message contains a random value.</li>
<li><span class="math">\(b\)</span> sends KeepAlive message to <span class="math">\(a\)</span> using the random value that is has
obtained from <span class="math">\(b\)</span><span class="quo">‘</span>s Hello message.</li>
</ol>
<p>Upon receipt of a KeepAlive message at <span class="math">\(a\)</span> from <span class="math">\(b\)</span>:</p>
<ol>
<li>
<p>If <span class="math">\(b\)</span><span class="quo">‘</span>s public key is not registered at the node <span class="math">\(a\)</span>, <span class="math">\(a\)</span> ignores the message.</p>
</li>
<li>
<p>If <span class="math">\(a\)</span> has opened a <span class="caps">UDP</span> porter link to <span class="math">\(b\)</span>:</p>
<ol>
<li><span class="math">\(a\)</span> updates the liveliness of <span class="math">\(b\)</span>.</li>
</ol>
</li>
<li>
<p>Else, if <span class="math">\(a\)</span> doesn’t yet have a <span class="caps">UDP</span> porter link to <span class="math">\(b\)</span>:</p>
<ol>
<li><span class="math">\(a\)</span> opens a <span class="caps">UDP</span> porter link to <span class="math">\(b\)</span>, initializing the liveliness value of
<span class="math">\(b\)</span>.</li>
<li><span class="math">\(a\)</span> sends a KeepAlive message to <span class="math">\(b\)</span> using the random value from the
last KeepAlive message sent by <span class="math">\(b\)</span>.</li>
</ol>
</li>
</ol>
<p>If <span class="math">\(a\)</span> has an open <span class="caps">UDP</span> porter link to <span class="math">\(b\)</span> and <span class="math">\(b\)</span> hasn’t sent a KeepAlive
message for a long enough time, the link is considered to be close.</p>
<script type="text/javascript">if (!document.getElementById('mathjaxscript_pelican_#%@#$@#')) {
var align = "center",
indent = "0em",
linebreak = "false";
if (false) {
align = (screen.width < 768) ? "left" : align;
indent = (screen.width < 768) ? "0em" : indent;
linebreak = (screen.width < 768) ? 'true' : linebreak;
}
var mathjaxscript = document.createElement('script');
mathjaxscript.id = 'mathjaxscript_pelican_#%@#$@#';
mathjaxscript.type = 'text/javascript';
mathjaxscript.src = 'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.0/MathJax.js?config=TeX-AMS-MML_HTMLorMML';
mathjaxscript[(window.opera ? "innerHTML" : "text")] =
"MathJax.Hub.Config({" +
" config: ['MMLorHTML.js']," +
" TeX: { extensions: ['AMSmath.js','AMSsymbols.js','noErrors.js','noUndefined.js'], equationNumbers: { autoNumber: 'AMS' } }," +
" jax: ['input/TeX','input/MathML','output/HTML-CSS']," +
" extensions: ['tex2jax.js','mml2jax.js','MathMenu.js','MathZoom.js']," +
" displayAlign: '"+ align +"'," +
" displayIndent: '"+ indent +"'," +
" showMathMenu: true," +
" messageStyle: 'normal'," +
" tex2jax: { " +
" inlineMath: [ ['\\\\(','\\\\)'] ], " +
" displayMath: [ ['$$','$$'] ]," +
" processEscapes: true," +
" preview: 'TeX'," +
" }, " +
" 'HTML-CSS': { " +
" styles: { '.MathJax_Display, .MathJax .mo, .MathJax .mi, .MathJax .mn': {color: 'inherit ! important'} }," +
" linebreaks: { automatic: "+ linebreak +", width: '90% container' }," +
" }, " +
"}); " +
"if ('default' !== 'default') {" +
"MathJax.Hub.Register.StartupHook('HTML-CSS Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax['HTML-CSS'].FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"MathJax.Hub.Register.StartupHook('SVG Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax.SVG.FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"}";
(document.body || document.getElementsByTagName('head')[0]).appendChild(mathjaxscript);
}
</script>Passerby Routing Engine2017-08-09T12:50:00+00:002017-08-09T12:50:00+00:00realtag:www.freedomlayer.org,2017-08-09:/passerby-routing-engine.html<p><strong>work in progress!</strong></p>
<h2>Abstract</h2>
<p>Passerby is an experimental datagram routing protocol. We present here the
ideas of how Passerby protocol works and provide details about how to
implement it.</p>
<p>The main goal of the Passerby protocol is passing encrypted messages between
nodes in a network with high probability of success …</p><p><strong>work in progress!</strong></p>
<h2>Abstract</h2>
<p>Passerby is an experimental datagram routing protocol. We present here the
ideas of how Passerby protocol works and provide details about how to
implement it.</p>
<p>The main goal of the Passerby protocol is passing encrypted messages between
nodes in a network with high probability of success.</p>
<h2>Core routing algorithm</h2>
<p>The Passerby protocol assumes a connected network of nodes. Every node is
directly connected to a few other nodes in some way. A direct connection could
be a <span class="caps">TCP</span> connection over the internet, a direct wifi connection, a direct
bluetooth connection or any other type of connection. It is assumed that the
direct connection two nodes is <span class="caps">TCP</span> like: It is reliable and sends messages in order.</p>
<p>Routing in the network is relies on the existence of Index nodes. Index
nodes are nodes that contain an approximate full map of the network, and they
provide short paths between nodes as a service. Whenever a node <span class="math">\(v\)</span> wants to
send a message to some remote node <span class="math">\(u\)</span>, <span class="math">\(v\)</span> first asks for a path from <span class="math">\(v\)</span> to
<span class="math">\(u\)</span> from one of the index nodes. <span class="math">\(v\)</span> then uses that path to send the message
to the node <span class="math">\(u\)</span>.</p>
<p>Index nodes perform difficult work. An index node deals with much more
bandwidth than a usual node, as it has to keep an updated view of the
network graph. In addition, an index node makes heavy use of its computation
resources to calculate short paths between nodes in the the large network graph.</p>
<h2>Index nodes setup</h2>
<p>As index nodes are crucial for the ability of nodes to route message in the
network, index nodes should be accessible to every node.
To make this work, every index node advertises its existence periodically. This
is done by sending a message to all his direct neighbors in the network,
declaring himself as an index node, signing over a timestamp.</p>
<p>Every node <span class="math">\(v\)</span> keeps at all times paths to the closest index nodes he knows of.
He keeps <span class="math">\(k\)</span> different paths to closest index nodes through every direct
neighbor he has. Hence even if some of <span class="math">\(v\)</span><span class="quo">‘</span>s direct neighbors give <span class="math">\(v\)</span> paths to
invalid index nodes, <span class="math">\(v\)</span> will still have some paths to valid nodes from his
other direct neighbors.</p>
<p>We denote <span class="math">\(v\)</span><span class="quo">‘</span>s list of closest index nodes and their paths as <span class="math">\(L(v)\)</span>.
Whenever any node in this list changes, <span class="math">\(v\)</span> informs all of his neighbors about
the change. Using this method changes in the state of index nodes in the
network propagates to all the relevant nodes in the network in a short time.</p>
<p>Changes to a nodes best known index nodes list can happen due to finding a new
closer index node, or the expiry of some index node, due to a timestamp being
too old.</p>
<h2>Network indexing</h2>
<p>Index nodes maintain an approximate view of the current structure of the
network. A node <span class="math">\(v\)</span> sends to some randomly chosen index nodes from <span class="math">\(L(v)\)</span> about
his direct connections to his neighbors. A message that contains information
about such connections also contains a signature. This signature is later used
as a proof by the index node. </p>
<p>As an example: If <span class="math">\(v\)</span> has a direct neighbor node <span class="math">\(u\)</span>, <span class="math">\(v\)</span> sends to some
randomly chosen index nodes from <span class="math">\(L(v)\)</span> the connection <span class="math">\(v \rightarrow u\)</span>,
together with a signature of <span class="math">\(v\)</span> over <span class="math">\(u\)</span>. The signature can later be used to
prove that <span class="math">\(v\)</span> claimed to be a direct neighbor of <span class="math">\(u\)</span>. In addition, this
message will contain a signed timestamp by <span class="math">\(v\)</span>, to show that this direct
connection is not too old. </p>
<p>Messages about direct connections between nodes are sent in the following occasions:</p>
<ul>
<li>
<p>Periodically. This allows indexing nodes to perform a periodic cleanup of
connections with old timestamps.</p>
</li>
<li>
<p>Whenever a connection is formed: If <span class="math">\(v\)</span> has a new direct neighbor <span class="math">\(u\)</span>, he
will immediately send a message about this connection to some random
collection of the index nodes in <span class="math">\(L(v)\)</span>.</p>
</li>
<li>
<p>Whenever a connection is lost: If <span class="math">\(v\)</span><span class="quo">‘</span>s direct neighbor <span class="math">\(u\)</span> has disconnected,
<span class="math">\(v\)</span> will send a message about this to some random collection of the index
nodes in <span class="math">\(L(v)\)</span>.</p>
</li>
</ul>
<p>Whenever an index node <span class="math">\(w\)</span> receives a message about a change in the network
structure, it first checks if he have seen this message before (Using some
local cache mechanism). If this message was seen before, it is discarded. If
this message is new, <span class="math">\(w\)</span> will send it to all of the index nodes in the list
<span class="math">\(L(w)\)</span>. We assume that this makes sure that with high probability the message
will arrive at all the index nodes in the network.</p>
<p><span class="caps">TODO</span>: Assuming a connected network, why do we think that the overlay graph of
index nodes is connected? Create an experiment to test this. This is a bit
different than the formation of colored towers. Add here references to our experiments.</p>
<p>An index node considers a direct connection to exist between two nodes <span class="math">\(v\)</span> and
<span class="math">\(u\)</span> if and only if both <span class="math">\(v\)</span> and <span class="math">\(u\)</span> have reported and signed over the existence
of this direct connection.</p>
<h2>Querying for paths</h2>
<p>Whenever a node <span class="math">\(v\)</span> wants to send a message to some remote node <span class="math">\(u\)</span>, <span class="math">\(v\)</span> first
needs to know about a path of nodes from <span class="math">\(v\)</span> to <span class="math">\(u\)</span>. This is done by asking
some index nodes for a path.</p>
<p><span class="math">\(v\)</span> chooses a random index node <span class="math">\(w\)</span> from <span class="math">\(L(v)\)</span> and sends him a request for a path
from <span class="math">\(v\)</span> to <span class="math">\(u\)</span>. The index node <span class="math">\(w\)</span> generates a path from <span class="math">\(v\)</span> to <span class="math">\(u\)</span> as follows:</p>
<ol>
<li>
<p>Choose a random node <span class="math">\(x\)</span> from all the nodes that he knows of.</p>
</li>
<li>
<p>Find a shortest path from <span class="math">\(v\)</span> to <span class="math">\(x\)</span>.</p>
</li>
<li>
<p>Find a shortest path from <span class="math">\(x\)</span> to <span class="math">\(u\)</span>.</p>
</li>
<li>
<p>Concatenate the paths, possibly removing cycles.</p>
</li>
<li>
<p>Send the resulting path to <span class="math">\(v\)</span>.</p>
</li>
</ol>
<p><span class="math">\(v\)</span> then checks the returned path from <span class="math">\(v\)</span> to <span class="math">\(u\)</span>, and verifies all the
signatures. If the path is valid, it can be used to send messages to <span class="math">\(u\)</span>.</p>
<p>The choice of returning a path from <span class="math">\(v\)</span> to <span class="math">\(u\)</span> through a random intermediate
node <span class="math">\(x\)</span> is made to to lower network congestion, and make the communication
more resilient. </p>
<h2>Main limitations</h2>
<ul>
<li>
<p>Synchronization:</p>
<ul>
<li>We use the time on node’s machines as a source of time for timestamps. The
time between different nodes might not be synchronized well. We plan to solve
this problem in future designs using a distributed and secure source of time.</li>
</ul>
</li>
<li>
<p>Scalability: </p>
<ul>
<li>It is possible that it might become very difficult for index nodes to
maintain the full structure of the network and find shortest paths in
it as the network becomes very large. However, it seems like this should
be manageable for pretty large networks.</li>
</ul>
</li>
<li>
<p>Incentives:</p>
<ul>
<li>
<p>It is bandwidth and <span class="caps">CPU</span> consuming to be an Index node, however, Index node
operators are not compensated for this extra effort. In addition, the direct
neighbors of an index node will probably suffer too from In the future we could
use some kind of monetary incentive to compenstate Index node owners.</p>
</li>
<li>
<p>Nodes that relay more data on the network are not compensated for their work.</p>
</li>
</ul>
</li>
<li>
<p>Denial of service</p>
<ul>
<li>It is possible to announce many index nodes that do nothing. If there are
enough such index node next to some node <span class="math">\(v\)</span>, <span class="math">\(v\)</span> might not be able to
obtain paths to remote nodes.</li>
</ul>
</li>
</ul>
<h2>Passerby router interface</h2>
<p>The passerby router is a state machine. It communicates using the following messages:</p>
<h3>Input messages:</h3>
<p><strong>Initialize</strong></p>
<ul>
<li>privateKey</li>
<li>currentTime</li>
<li>isIndexNode</li>
</ul>
<p>Initialize the state machine.</p>
<p><strong>TimeTick</strong></p>
<ul>
<li>currentTime</li>
</ul>
<p>Information about the current time.</p>
<p><strong>PortConnected</strong></p>
<ul>
<li>neighborIdHash</li>
</ul>
<p>A new port was connected to the passerby router.</p>
<p><strong>PortDisconnected</strong></p>
<ul>
<li>neighborIdHash</li>
</ul>
<p>A port was disconnected from the passerby router.</p>
<p><strong>MsgFromPort</strong></p>
<ul>
<li>portId</li>
<li>msg</li>
</ul>
<p>Message was received from a given port.</p>
<p><strong>SendMessage</strong></p>
<ul>
<li>remoteNodeIdHash</li>
<li>datagramData</li>
</ul>
<p>A request to send a message to a remote node id hash.</p>
<h3>Output messages:</h3>
<p><strong>MsgToPort</strong></p>
<ul>
<li>portId</li>
<li>msg</li>
</ul>
<p>A request to send message through port portId.</p>
<p><strong>MessageReceived</strong>
- remoteNodeIdHash
- datagramData</p>
<p>A remote message was received from remoteNodeIdHash.</p>
<p>Node id hashes are of size <span class="math">\(120\)</span> bits (<span class="math">\(15\)</span> bytes). This allows them to fit into
IPv6 address space of <a href="https://en.wikipedia.org/wiki/Unique_local_address">private
addresses</a> (All ipv6
addresses that begin with the byte 0xfc).</p>
<p>The structure of msg in MsgFromPort and MsgToPort is as follows:</p>
<ul>
<li>messageType</li>
<li>payload</li>
</ul>
<p>We rely on a service external to the Passerby router to cut the stream of bytes
into frames of discrete messages.</p>
<h2>Messages between routing engines</h2>
<p>Messages are of three possible layers, marked as L0, L1, L2.</p>
<p>A signed path from <code>srcNodeId</code> to <code>dstNodeId</code> is of the form:</p>
<div class="highlight"><pre><span></span>srcNodeId, timestamp
a_1, sign{nodeId}(a_1, prev)
a_2, sign{a_1}(a_2, prev)
...
dstNodeId, sign{a_{k-1}}(dstNodeId, prev)
</pre></div>
<p>A doubly signed path from <code>srcNodeId</code> to <code>dstNodeId</code> is of the form:</p>
<div class="highlight"><pre><span></span>srcNodeId,
sign{srcNodeId}(srcNodeId->a_1, timestamp)
sign{a_1}(a_1->srcNodeId, timestamp)
a_1
sign{a_1}(a_1->a_2, timestamp)
sign{a_2}(a_2->a_1, timestamp)
a_2
...
a_{k-1}
sign{a_{k-1}}(a_{k-1}->dstNodeId, timestamp)
sign{dstNodeId}(dstNodeId->a_{k-1}, timestamp)
dstNodeId
</pre></div>
<p><strong>UpdateIndexNode</strong> (L0)</p>
<ul>
<li>SignedPathFromIndex</li>
</ul>
<p>A message about a path to a local index node. This message originates from
index nodes and propagate through the network. If a node receives this
message and updates his list of best index nodes as a result, he forwards
this message to all of his direct neighbors.</p>
<p><strong>RegisterWithIndex</strong> (L0)</p>
<ul>
<li>SignedPathToIndex</li>
</ul>
<p>An index node registers with a local index, to get messages about network updates.</p>
<p><strong>MsgByPath</strong> (L0)</p>
<ul>
<li>nodeIdPath</li>
<li>L1content</li>
</ul>
<p>Where nodeIdPath is a path of nodeIds.
This message allows to send content to a remote node using a path of nodeIds.</p>
<p><strong>StructureUpdate</strong> // MsgByPath (L1)</p>
<ul>
<li>type: Connect/Disconnect</li>
<li>edgeSrc</li>
<li>edgeDest</li>
<li>nonce</li>
<li>timestamp</li>
<li>Signature: Sign{edgeSrc}(all message)</li>
</ul>
<p>Sent over MsgByPath. A node reports to an index node about Connect/Disconnect
event of the edge edgeSrc —> edgeDest. This message is also broadcasted
between index nodes.</p>
<p><strong>EncDatagram</strong> // MsgByPath (L1)</p>
<ul>
<li>encSymmetricKey == Encrypt{key=recipientPublicKey}(symmetricKey)</li>
<li>encDatagramData == Encrypt{key=symmetricKey, nonce=random}(
timestamp, senderPublicKey, L2content)</li>
</ul>
<p>Encrypted datagram being passed between nodes.</p>
<p><strong>UserDatagram</strong> // EncDatagram // MsgByPath (L2)</p>
<ul>
<li>content</li>
</ul>
<h2>Index node service messages</h2>
<p><strong>RequestPath</strong> // EncDatagram // MsgByPath (L2)</p>
<ul>
<li>srcNodeId or srcNodeHashId</li>
<li>dstNodeId or dstNodeHashId</li>
</ul>
<p>Sent from a node to a close index node, requesting a path from srcNodeId to dstNodeId.</p>
<p><strong>ResponsePath</strong> // EncDatagram // MsgByPath (L2)</p>
<ul>
<li>DoublySignedPath</li>
</ul>
<script type="text/javascript">if (!document.getElementById('mathjaxscript_pelican_#%@#$@#')) {
var align = "center",
indent = "0em",
linebreak = "false";
if (false) {
align = (screen.width < 768) ? "left" : align;
indent = (screen.width < 768) ? "0em" : indent;
linebreak = (screen.width < 768) ? 'true' : linebreak;
}
var mathjaxscript = document.createElement('script');
mathjaxscript.id = 'mathjaxscript_pelican_#%@#$@#';
mathjaxscript.type = 'text/javascript';
mathjaxscript.src = 'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.0/MathJax.js?config=TeX-AMS-MML_HTMLorMML';
mathjaxscript[(window.opera ? "innerHTML" : "text")] =
"MathJax.Hub.Config({" +
" config: ['MMLorHTML.js']," +
" TeX: { extensions: ['AMSmath.js','AMSsymbols.js','noErrors.js','noUndefined.js'], equationNumbers: { autoNumber: 'AMS' } }," +
" jax: ['input/TeX','input/MathML','output/HTML-CSS']," +
" extensions: ['tex2jax.js','mml2jax.js','MathMenu.js','MathZoom.js']," +
" displayAlign: '"+ align +"'," +
" displayIndent: '"+ indent +"'," +
" showMathMenu: true," +
" messageStyle: 'normal'," +
" tex2jax: { " +
" inlineMath: [ ['\\\\(','\\\\)'] ], " +
" displayMath: [ ['$$','$$'] ]," +
" processEscapes: true," +
" preview: 'TeX'," +
" }, " +
" 'HTML-CSS': { " +
" styles: { '.MathJax_Display, .MathJax .mo, .MathJax .mi, .MathJax .mn': {color: 'inherit ! important'} }," +
" linebreaks: { automatic: "+ linebreak +", width: '90% container' }," +
" }, " +
"}); " +
"if ('default' !== 'default') {" +
"MathJax.Hub.Register.StartupHook('HTML-CSS Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax['HTML-CSS'].FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"MathJax.Hub.Register.StartupHook('SVG Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax.SVG.FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"}";
(document.body || document.getElementsByTagName('head')[0]).appendChild(mathjaxscript);
}
</script>The connectivity of sqrt(n) towers2017-08-01T17:32:00+00:002017-08-01T17:32:00+00:00realtag:www.freedomlayer.org,2017-08-01:/the-connectivity-of-sqrtn-towers.html<h2>Abstract</h2>
<p>Suppose <span class=math>\(G\)</span> is a connected graph of <span class=math>\(n\)</span> vertices. Let <span class=math>\(k > 0\)</span> be some
integer, and let <span class=math>\(T_i \subseteq V(G)\)</span> be a set of vertices, for <span class=math>\(0 \leq i < k\)</span>.
We say that <span class=math>\(T_i\)</span> is the set of “towers” of color <span class=math>\(i\)</span>.</p>
<p>For every vertex <span class=math>\(v \in V …</span></p><h2>Abstract</h2>
<p>Suppose <span class=math>\(G\)</span> is a connected graph of <span class=math>\(n\)</span> vertices. Let <span class=math>\(k > 0\)</span> be some
integer, and let <span class=math>\(T_i \subseteq V(G)\)</span> be a set of vertices, for <span class=math>\(0 \leq i < k\)</span>.
We say that <span class=math>\(T_i\)</span> is the set of “towers” of color <span class=math>\(i\)</span>.</p>
<p>For every vertex <span class=math>\(v \in V(G)\)</span> we denote by <span class=math>\(t_i(v)\)</span> the tower <span class=math>\(x\)</span> of color <span class=math>\(i\)</span>
that is closest to <span class=math>\(v\)</span> in the graph. (If there is a tie between a few
candidates, we break it by picking the tower of color <span class=math>\(i\)</span> with lowest id).</p>
<p>Denote the set <span class=math>\(T = \cup_{i}{T_i}\)</span> of all the towers. We now define a new
directed graph <span class=math>\(Q\)</span> with <span class=math>\(V(Q) = T\)</span>, the set of all towers. For two towers <span class=math>\(u, v
\in V(Q)\)</span>, <span class=math>\((u,v) \in E(Q)\)</span> if and only if <span class=math>\(t_i(u) = v\)</span> for some <span class=math>\(0 \leq i < k\)</span>.</p>
<p>We want to know in which cases <span class=math>\(Q\)</span> is a <a href="https://en.wikipedia.org/wiki/Strongly_connected_component">strongly
connected</a> or weakly
connected (Connected as an undirected graph). In other words, we want to know
whether the connections towers form a connected graph.</p>
<p>Our experiments show that for <span class=math>\(k \geq \log_{2}{n}\)</span> the resulting overlay graph <span class=math>\(Q\)</span>
is probably weakly connected, but not necessarily strongly connected. </p>
<p>The experiments are written in <a href="https://www.rust-lang.org">Rust</a> and could be
found <a href="https://github.com/realcr/freedomlayer_code/tree/master/connectivity_towers">here
[github]</a>.</p>
<h2>Motivation</h2>
<p>Suppose that we are given a large network <span class=math>\(G\)</span>. Every node is connected directly
to a few other nodes, and because of the large size of the network, no node can
perceive the full structure of the network. We want to be able to route
messages in this network between any pair of nodes.</p>
<p>Our method to do this is by choosing a random set of about <span class=math>\(\sqrt{n}\)</span> nodes,
called the “towers”. Every tower out of the <span class=math>\(\sqrt{n}\)</span> towers has some color
<span class=math>\(c\)</span>, where <span class=math>\(0 \leq c < k\)</span>.</p>
<p>Every node keeps contact with the closest tower of color <span class=math>\(c\)</span> for <span class=math>\(0 \leq c <
k\)</span>. This amounts to a total of <span class=math>\(c\)</span> towers. Whenever a node <span class=math>\(x\)</span> wants to send a
message to a remote node <span class=math>\(y\)</span> in the network, <span class=math>\(x\)</span> sends the message to one of the
close towers he keeps contact with. The towers then route the message all the
way to one of the towers that keep contact with <span class=math>\(y\)</span>, and finally that tower
passes the message to <span class=math>\(y\)</span>.</p>
<p>For this scheme to work, we have to make sure that the towers know how to route
messages between every two towers. Every node maintains contact with <span class=math>\(k\)</span>
different towers, and this is also true for tower nodes. We use those
connections as an overlay graph between the towers. In this overlay directed
graph, every tower is connected to <span class=math>\(k-1\)</span> other towers and has a self edge
himself. The self edge happens because a tower <span class=math>\(x_c\)</span> of color <span class=math>\(c\)</span> has
<span class=math>\(t_c(x_c) = x_c\)</span>.</p>
<p><img src="https://www.freedomlayer.org/articles/freedomlayer/connectivity_towers/images/towers_and_nodes.svg" alt="Overlay directed graph of towers"></p>
<blockquote>
<p>In the picture: The overlay directed graph between the towers (Self edges are
omitted). Note that the graph is connected, but not strongly connected.
The black small dots represent regular nodes. For example: To route a message
from the node <span class=math>\(x\)</span> to the node <span class=math>\(y\)</span>, the message has to pass between various
towers along the way.</p>
</blockquote>
<p><strong>If we can assume that the overlay directed graph between towers is somehow
connected</strong>, we should be able to manage routing between the towers. As a simple
solution, as there are only <span class=math>\(\sqrt{n}\)</span> towers, this could be done by letting
every tower remember a shortest path to every other tower.</p>
<p>Note that in this setting a tower does more work than a usual node. A tower has
to remember shortest path to other <span class=math>\(\sqrt{n}\)</span> towers, and in addition maintain
contact with <span class=math>\(\sqrt{n}\)</span> nodes around him. This is not a fully decentralized
setting, but we believe that the work that a tower does should be manageable
and some might choose to do it for the right compensation.</p>
<p>In this document we try to evaluate which parameters for this
structure of towers (Amount of towers and amount of colors) we should use to
make sure that the resulting overlay directed graph between towers is connected.</p>
<h2>Experiment</h2>
<p>We experiment as follows: We generate different types of networks of different sizes.
For a network of size <span class=math>\(n\)</span> nodes we pick <span class=math>\(k = 2\cdot\log_{2}{n}\)</span> different
colors. For each color we pick <span class=math>\(\sqrt{n} / k\)</span> different towers (Towers of
different color may overlap). This is the code that picks the towers (written
in <a href="https://www.rust-lang.org">Rust</a>):</p>
<div class=highlight><pre><span></span><span class=sd>/// Choose nodes to be towers. We pick num_towers towers of every color. There are num_colors</span>
<span class=sd>/// different tower colors.</span>
<span class=k>pub</span><span class=w> </span><span class=k>fn</span> <span class=nf>choose_towers</span><span class=o><</span><span class=n>Node</span>: <span class=nc>Hash</span><span class=w> </span><span class=o>+</span><span class=w> </span><span class=nb>Eq</span><span class=w> </span><span class=o>+</span><span class=w> </span><span class=nb>Clone</span><span class=p>,</span><span class=w> </span><span class=n>R</span>: <span class=nc>Rng</span><span class=o>></span><span class=p>(</span><span class=n>net</span>: <span class=kp>&</span><span class=nc>Network</span><span class=o><</span><span class=n>Node</span><span class=o>></span><span class=p>,</span><span class=w> </span>
<span class=w> </span><span class=n>num_towers</span>: <span class=kt>usize</span><span class=p>,</span><span class=w> </span><span class=n>num_colors</span>: <span class=kt>usize</span><span class=p>,</span><span class=w> </span><span class=n>rng</span>: <span class=kp>&</span><span class=nc>mut</span><span class=w> </span><span class=n>R</span><span class=p>)</span><span class=w> </span>-> <span class=nb>Vec</span><span class=o><</span><span class=nb>Vec</span><span class=o><</span><span class=kt>usize</span><span class=o>>></span><span class=w> </span><span class=p>{</span><span class=w></span>
<span class=w> </span><span class=kd>let</span><span class=w> </span><span class=k>mut</span><span class=w> </span><span class=n>chosen_towers</span>: <span class=nb>Vec</span><span class=o><</span><span class=nb>Vec</span><span class=o><</span><span class=kt>usize</span><span class=o>>></span><span class=w> </span><span class=o>=</span><span class=w> </span><span class=nb>Vec</span>::<span class=n>new</span><span class=p>();</span><span class=w></span>
<span class=w> </span><span class=k>for</span><span class=w> </span><span class=n>_</span><span class=w> </span><span class=k>in</span><span class=w> </span><span class=mi>0</span><span class=w> </span><span class=p>..</span><span class=w> </span><span class=n>num_colors</span><span class=w> </span><span class=p>{</span><span class=w></span>
<span class=w> </span><span class=c1>// Pick random towers for a certain color:</span>
<span class=w> </span><span class=kd>let</span><span class=w> </span><span class=k>mut</span><span class=w> </span><span class=n>ctowers</span><span class=w> </span><span class=o>=</span><span class=w> </span><span class=n>choose_k_nums</span><span class=p>(</span><span class=n>num_towers</span><span class=p>,</span><span class=w> </span><span class=n>net</span><span class=p>.</span><span class=n>igraph</span><span class=p>.</span><span class=n>node_count</span><span class=p>(),</span><span class=w> </span><span class=n>rng</span><span class=p>)</span><span class=w></span>
<span class=w> </span><span class=p>.</span><span class=n>into_iter</span><span class=p>()</span><span class=w></span>
<span class=w> </span><span class=p>.</span><span class=n>collect</span>::<span class=o><</span><span class=nb>Vec</span><span class=o><</span><span class=kt>usize</span><span class=o>>></span><span class=p>();</span><span class=w></span>
<span class=w> </span><span class=c1>// Sort for determinism:</span>
<span class=w> </span><span class=n>ctowers</span><span class=p>.</span><span class=n>sort</span><span class=p>();</span><span class=w></span>
<span class=w> </span><span class=n>chosen_towers</span><span class=p>.</span><span class=n>push</span><span class=p>(</span><span class=n>ctowers</span><span class=p>);</span><span class=w></span>
<span class=w> </span><span class=p>}</span><span class=w></span>
<span class=w> </span><span class=n>chosen_towers</span><span class=w></span>
<span class=p>}</span><span class=w></span>
</pre></div>
<p>Then we calculate iteratively for every node <span class=math>\(v\)</span> in the network the closest
tower of color <span class=math>\(c\)</span> for <span class=math>\(0 \leq c < k\)</span>. This is how one iteration looks like:</p>
<div class=highlight><pre><span></span><span class=sd>/// Perform one iteration of calculating towers info.</span>
<span class=sd>/// Return whether any changed happen during this iteration.</span>
<span class=k>fn</span> <span class=nf>iter_towers_info</span><span class=o><</span><span class=n>Node</span>: <span class=nc>Hash</span><span class=w> </span><span class=o>+</span><span class=w> </span><span class=nb>Eq</span><span class=w> </span><span class=o>+</span><span class=w> </span><span class=nb>Clone</span><span class=o>></span><span class=p>(</span><span class=n>net</span>: <span class=kp>&</span><span class=nc>Network</span><span class=o><</span><span class=n>Node</span><span class=o>></span><span class=p>,</span><span class=w></span>
<span class=w> </span><span class=n>chosen_towers</span>: <span class=kp>&</span><span class=nb>Vec</span><span class=o><</span><span class=nb>Vec</span><span class=o><</span><span class=kt>usize</span><span class=o>>></span><span class=p>,</span><span class=w></span>
<span class=w> </span><span class=n>towers_info</span>: <span class=kp>&</span><span class=nc>mut</span><span class=w> </span><span class=nb>Vec</span><span class=o><</span><span class=nb>Vec</span><span class=o><</span><span class=nb>Option</span><span class=o><</span><span class=n>LocalTowerInfo</span><span class=o>>>></span><span class=p>)</span><span class=w> </span>-> <span class=kt>bool</span> <span class=p>{</span><span class=w></span>
<span class=w> </span><span class=kd>let</span><span class=w> </span><span class=k>mut</span><span class=w> </span><span class=n>changed</span><span class=w> </span><span class=o>=</span><span class=w> </span><span class=kc>false</span><span class=p>;</span><span class=w></span>
<span class=w> </span><span class=k>for</span><span class=w> </span><span class=n>node</span><span class=w> </span><span class=k>in</span><span class=w> </span><span class=n>net</span><span class=p>.</span><span class=n>igraph</span><span class=p>.</span><span class=n>nodes</span><span class=p>()</span><span class=w> </span><span class=p>{</span><span class=w></span>
<span class=w> </span><span class=k>for</span><span class=w> </span><span class=n>nei</span><span class=w> </span><span class=k>in</span><span class=w> </span><span class=n>net</span><span class=p>.</span><span class=n>igraph</span><span class=p>.</span><span class=n>neighbors</span><span class=p>(</span><span class=n>node</span><span class=p>)</span><span class=w> </span><span class=p>{</span><span class=w></span>
<span class=w> </span><span class=k>for</span><span class=w> </span><span class=n>tower_color</span><span class=w> </span><span class=k>in</span><span class=w> </span><span class=mi>0</span><span class=w> </span><span class=p>..</span><span class=w> </span><span class=n>chosen_towers</span><span class=p>.</span><span class=n>len</span><span class=p>()</span><span class=w> </span><span class=p>{</span><span class=w></span>
<span class=w> </span><span class=k>if</span><span class=w> </span><span class=n>towers_info</span><span class=p>[</span><span class=n>node</span><span class=p>][</span><span class=n>tower_color</span><span class=p>].</span><span class=n>is_none</span><span class=p>()</span><span class=w> </span><span class=p>{</span><span class=w></span>
<span class=w> </span><span class=k>continue</span><span class=w></span>
<span class=w> </span><span class=p>}</span><span class=w></span>
<span class=w> </span><span class=c1>// This is the candidate LocalTowerInfo for nei:</span>
<span class=w> </span><span class=kd>let</span><span class=w> </span><span class=k>mut</span><span class=w> </span><span class=n>candidate_info</span><span class=w> </span><span class=o>=</span><span class=w> </span><span class=n>towers_info</span><span class=p>[</span><span class=n>node</span><span class=p>][</span><span class=n>tower_color</span><span class=p>].</span><span class=n>clone</span><span class=p>().</span><span class=n>unwrap</span><span class=p>();</span><span class=w></span>
<span class=w> </span><span class=n>candidate_info</span><span class=p>.</span><span class=n>distance</span><span class=w> </span><span class=o>+=</span><span class=w> </span><span class=mi>1</span><span class=p>;</span><span class=w></span>
<span class=w> </span><span class=n>candidate_info</span><span class=p>.</span><span class=n>gateway</span><span class=w> </span><span class=o>=</span><span class=w> </span><span class=n>node</span><span class=p>;</span><span class=w></span>
<span class=w> </span><span class=c1>// Current nei's LocalTowerInfo:</span>
<span class=w> </span><span class=k>if</span><span class=w> </span><span class=n>towers_info</span><span class=p>[</span><span class=n>nei</span><span class=p>][</span><span class=n>tower_color</span><span class=p>].</span><span class=n>is_none</span><span class=p>()</span><span class=w> </span><span class=p>{</span><span class=w></span>
<span class=w> </span><span class=n>changed</span><span class=w> </span><span class=o>=</span><span class=w> </span><span class=kc>true</span><span class=p>;</span><span class=w></span>
<span class=w> </span><span class=n>towers_info</span><span class=p>[</span><span class=n>nei</span><span class=p>][</span><span class=n>tower_color</span><span class=p>]</span><span class=w> </span><span class=o>=</span><span class=w> </span><span class=nb>Some</span><span class=p>(</span><span class=n>candidate_info</span><span class=p>);</span><span class=w></span>
<span class=w> </span><span class=k>continue</span><span class=w></span>
<span class=w> </span><span class=p>}</span><span class=w></span>
<span class=w> </span><span class=kd>let</span><span class=w> </span><span class=n>nei_info</span><span class=w> </span><span class=o>=</span><span class=w> </span><span class=n>towers_info</span><span class=p>[</span><span class=n>nei</span><span class=p>][</span><span class=n>tower_color</span><span class=p>].</span><span class=n>clone</span><span class=p>().</span><span class=n>unwrap</span><span class=p>();</span><span class=w></span>
<span class=w> </span><span class=k>if</span><span class=w> </span><span class=p>(</span><span class=n>candidate_info</span><span class=p>.</span><span class=n>distance</span><span class=p>,</span><span class=w></span>
<span class=w> </span><span class=n>candidate_info</span><span class=p>.</span><span class=n>gateway</span><span class=p>,</span><span class=w></span>
<span class=w> </span><span class=n>candidate_info</span><span class=p>.</span><span class=n>tower_node</span><span class=p>)</span><span class=w> </span><span class=o><</span><span class=w></span>
<span class=w> </span><span class=p>(</span><span class=n>nei_info</span><span class=p>.</span><span class=n>distance</span><span class=p>,</span><span class=w></span>
<span class=w> </span><span class=n>nei_info</span><span class=p>.</span><span class=n>gateway</span><span class=p>,</span><span class=w></span>
<span class=w> </span><span class=n>nei_info</span><span class=p>.</span><span class=n>tower_node</span><span class=p>)</span><span class=w> </span><span class=p>{</span><span class=w></span>
<span class=w> </span><span class=n>changed</span><span class=w> </span><span class=o>=</span><span class=w> </span><span class=kc>true</span><span class=p>;</span><span class=w></span>
<span class=w> </span><span class=n>towers_info</span><span class=p>[</span><span class=n>nei</span><span class=p>][</span><span class=n>tower_color</span><span class=p>]</span><span class=w> </span><span class=o>=</span><span class=w> </span><span class=nb>Some</span><span class=p>(</span><span class=n>candidate_info</span><span class=p>);</span><span class=w></span>
<span class=w> </span><span class=p>}</span><span class=w></span>
<span class=w> </span><span class=p>}</span><span class=w></span>
<span class=w> </span><span class=p>}</span><span class=w></span>
<span class=w> </span><span class=p>}</span><span class=w></span>
<span class=w> </span><span class=n>changed</span><span class=w></span>
<span class=p>}</span><span class=w></span>
</pre></div>
<p>Finally, we create a directed overlay graph of the towers and check its
connectivity. We check two types of connectivity for the overlay graph: weak
connectivity and strong connectivity. Weak connectivity is checked using
<a href="https://en.wikipedia.org/wiki/Depth-first_search">dfs</a>, and strong
connectivity is checked using <a href="https://en.wikipedia.org/wiki/Kosaraju%27s_algorithm">kosaraju
algorithm</a> for finding
strongly connected components. Both of those algorithms were taken from the
<a href="https://github.com/bluss/petgraph">petgraph</a> rust crate.</p>
<p>This is the code that checks connectivity:</p>
<div class=highlight><pre><span></span><span class=sd>/// Check if overlay directed graph of towers is connected.</span>
<span class=sd>/// Returns (connected, strongly_connected)</span>
<span class=k>pub</span><span class=w> </span><span class=k>fn</span> <span class=nf>is_connected</span><span class=p>(</span><span class=n>chosen_towers</span>: <span class=kp>&</span><span class=nb>Vec</span><span class=o><</span><span class=nb>Vec</span><span class=o><</span><span class=kt>usize</span><span class=o>>></span><span class=p>,</span><span class=w> </span>
<span class=w> </span><span class=n>towers_info</span>: <span class=kp>&</span><span class=nb>Vec</span><span class=o><</span><span class=nb>Vec</span><span class=o><</span><span class=nb>Option</span><span class=o><</span><span class=n>LocalTowerInfo</span><span class=o>>>></span><span class=p>)</span><span class=w> </span>-> <span class=p>(</span><span class=kt>bool</span><span class=p>,</span><span class=w> </span><span class=kt>bool</span><span class=p>)</span><span class=w> </span><span class=p>{</span><span class=w></span>
<span class=w> </span><span class=c1>// An overlay directed graph of the towers in the network</span>
<span class=w> </span><span class=c1>// and the connections between them.</span>
<span class=w> </span><span class=c1>// A tower T is connected to a tower T' if T' is the closest tower to T of some color.</span>
<span class=w> </span><span class=kd>let</span><span class=w> </span><span class=k>mut</span><span class=w> </span><span class=n>towers_graph</span>: <span class=nc>graphmap</span>::<span class=n>DiGraphMap</span><span class=o><</span><span class=kt>usize</span><span class=p>,()</span><span class=o>></span><span class=w> </span><span class=o>=</span><span class=w> </span>
<span class=w> </span><span class=n>graphmap</span>::<span class=n>DiGraphMap</span>::<span class=n>new</span><span class=p>();</span><span class=w></span>
<span class=w> </span><span class=c1>// Add towers as nodes to the graph:</span>
<span class=w> </span><span class=k>for</span><span class=w> </span><span class=n>tower_color</span><span class=w> </span><span class=k>in</span><span class=w> </span><span class=mi>0</span><span class=w> </span><span class=p>..</span><span class=w> </span><span class=n>chosen_towers</span><span class=p>.</span><span class=n>len</span><span class=p>()</span><span class=w> </span><span class=p>{</span><span class=w></span>
<span class=w> </span><span class=k>for</span><span class=w> </span><span class=n>tower_index</span><span class=w> </span><span class=k>in</span><span class=w> </span><span class=mi>0</span><span class=w> </span><span class=p>..</span><span class=w> </span><span class=n>chosen_towers</span><span class=p>[</span><span class=n>tower_color</span><span class=p>].</span><span class=n>len</span><span class=p>()</span><span class=w> </span><span class=p>{</span><span class=w></span>
<span class=w> </span><span class=n>towers_graph</span><span class=p>.</span><span class=n>add_node</span><span class=p>(</span><span class=n>chosen_towers</span><span class=p>[</span><span class=n>tower_color</span><span class=p>][</span><span class=n>tower_index</span><span class=p>]);</span><span class=w></span>
<span class=w> </span><span class=p>}</span><span class=w></span>
<span class=w> </span><span class=p>}</span><span class=w></span>
<span class=w> </span><span class=c1>// For every tower, add all connections to closest local towers as nodes.</span>
<span class=w> </span><span class=kd>let</span><span class=w> </span><span class=n>graph_nodes</span><span class=w> </span><span class=o>=</span><span class=w> </span><span class=n>towers_graph</span><span class=p>.</span><span class=n>nodes</span><span class=p>().</span><span class=n>collect</span>::<span class=o><</span><span class=nb>Vec</span><span class=o><</span><span class=kt>usize</span><span class=o>>></span><span class=p>();</span><span class=w></span>
<span class=w> </span><span class=k>for</span><span class=w> </span><span class=n>tower_node</span><span class=w> </span><span class=k>in</span><span class=w> </span><span class=n>graph_nodes</span><span class=w> </span><span class=p>{</span><span class=w></span>
<span class=w> </span><span class=k>for</span><span class=w> </span><span class=n>tower_color</span><span class=w> </span><span class=k>in</span><span class=w> </span><span class=mi>0</span><span class=w> </span><span class=p>..</span><span class=w> </span><span class=n>chosen_towers</span><span class=p>.</span><span class=n>len</span><span class=p>()</span><span class=w> </span><span class=p>{</span><span class=w></span>
<span class=w> </span><span class=n>towers_graph</span><span class=p>.</span><span class=n>add_edge</span><span class=p>(</span><span class=n>tower_node</span><span class=p>,</span><span class=w> </span>
<span class=w> </span><span class=n>towers_info</span><span class=p>[</span><span class=n>tower_node</span><span class=p>][</span><span class=n>tower_color</span><span class=p>].</span><span class=n>clone</span><span class=p>().</span><span class=n>unwrap</span><span class=p>().</span><span class=n>tower_node</span><span class=p>,());</span><span class=w></span>
<span class=w> </span><span class=p>}</span><span class=w></span>
<span class=w> </span><span class=p>}</span><span class=w></span>
<span class=w> </span><span class=kd>let</span><span class=w> </span><span class=n>sconnected_comps</span><span class=w> </span><span class=o>=</span><span class=w> </span><span class=n>kosaraju_scc</span><span class=p>(</span><span class=o>&</span><span class=n>towers_graph</span><span class=p>);</span><span class=w></span>
<span class=w> </span><span class=p>(</span><span class=n>connected_components</span><span class=p>(</span><span class=o>&</span><span class=n>towers_graph</span><span class=p>)</span><span class=w> </span><span class=o>==</span><span class=w> </span><span class=mi>1</span><span class=p>,</span><span class=w> </span><span class=n>sconnected_comps</span><span class=p>.</span><span class=n>len</span><span class=p>()</span><span class=w> </span><span class=o>==</span><span class=w> </span><span class=mi>1</span><span class=p>)</span><span class=w></span>
<span class=p>}</span><span class=w></span>
</pre></div>
<p>The experiment is done for 5 different types of networks:</p>
<ul>
<li>rand: An approximated random <span class=math>\(G(n,p)\)</span> network with <span class=math>\(p = 1.5\log(n)/n\)</span></li>
<li>2d: a two dimensional grid.</li>
<li>rand + 2d: A sum of rand and 2d networks.</li>
<li>planar: We randomize nodes as points in a large plane. Every point is
connected to the closest <span class=math>\(1.5\log(n)\)</span> points <sup id=sf-the-connectivity-of-sqrtn-towers-1-back><a title=" The main bottleneck for this implementation is the generation of the planar type network. It is currently done in \(O(n^2)\), which takes a long time. " class=simple-footnote href=#sf-the-connectivity-of-sqrtn-towers-1>1</a></sup>.</li>
<li>tree: A tree network.</li>
</ul>
<p>Experiment results (<span class=math>\(k = 2\log_{2}{n}\)</span>)</p>
<div class=highlight><pre><span></span>$ cargo run --bin towers_scc --release <span class=p>|</span> tee ../results/towers_scc_g_2_weakly_2017_07_26.txt
Compiling net_coords v0.1.0 <span class=o>(</span>file:///home/real/projects/d/freedomlayer/freedomlayer_research/network_coords/net_coords<span class=o>)</span>
Finished release <span class=o>[</span>optimized<span class=o>]</span> target<span class=o>(</span>s<span class=o>)</span> in <span class=m>3</span>.30 secs
Running <span class=sb>`</span>target/release/towers_scc<span class=sb>`</span>
Checking <span class=k>if</span> <span class=nb>local</span> towers overlay graph is strongly connected
<span class=nv>g</span><span class=o>=</span> <span class=m>6</span><span class=p>;</span> rand <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>12</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>1</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span> <span class=m>6</span><span class=p>;</span> rand <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>12</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>1</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span> <span class=m>6</span><span class=p>;</span> 2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>12</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>1</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span> <span class=m>6</span><span class=p>;</span> 2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>12</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>1</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span> <span class=m>6</span><span class=p>;</span> rand+2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>12</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>1</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span> <span class=m>6</span><span class=p>;</span> rand+2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>12</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>1</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span> <span class=m>6</span><span class=p>;</span> planar <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>12</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>1</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span> <span class=m>6</span><span class=p>;</span> planar <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>12</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>1</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span> <span class=m>6</span><span class=p>;</span> tree <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>12</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>1</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span> <span class=m>6</span><span class=p>;</span> tree <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>12</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>1</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span> <span class=m>7</span><span class=p>;</span> rand <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>14</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>1</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span> <span class=m>7</span><span class=p>;</span> rand <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>14</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>1</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span> <span class=m>7</span><span class=p>;</span> 2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>14</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>1</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span> <span class=m>7</span><span class=p>;</span> 2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>14</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>1</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span> <span class=m>7</span><span class=p>;</span> rand+2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>14</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>1</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span> <span class=m>7</span><span class=p>;</span> rand+2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>14</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>1</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span> <span class=m>7</span><span class=p>;</span> planar <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>14</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>1</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span> <span class=m>7</span><span class=p>;</span> planar <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>14</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>1</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span> <span class=m>7</span><span class=p>;</span> tree <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>14</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>1</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span> <span class=m>7</span><span class=p>;</span> tree <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>14</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>1</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span> <span class=m>8</span><span class=p>;</span> rand <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>16</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>2</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span> <span class=m>8</span><span class=p>;</span> rand <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>16</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>2</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span> <span class=m>8</span><span class=p>;</span> 2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>16</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>2</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span> <span class=m>8</span><span class=p>;</span> 2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>16</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>2</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span> <span class=m>8</span><span class=p>;</span> rand+2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>16</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>2</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span> <span class=m>8</span><span class=p>;</span> rand+2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>16</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>2</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span> <span class=m>8</span><span class=p>;</span> planar <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>16</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>2</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span> <span class=m>8</span><span class=p>;</span> planar <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>16</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>2</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span> <span class=m>8</span><span class=p>;</span> tree <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>16</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>2</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span> <span class=m>8</span><span class=p>;</span> tree <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>16</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>2</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> X
<span class=nv>g</span><span class=o>=</span> <span class=m>9</span><span class=p>;</span> rand <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>18</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>2</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span> <span class=m>9</span><span class=p>;</span> rand <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>18</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>2</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span> <span class=m>9</span><span class=p>;</span> 2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>18</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>2</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span> <span class=m>9</span><span class=p>;</span> 2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>18</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>2</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span> <span class=m>9</span><span class=p>;</span> rand+2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>18</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>2</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span> <span class=m>9</span><span class=p>;</span> rand+2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>18</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>2</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span> <span class=m>9</span><span class=p>;</span> planar <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>18</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>2</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span> <span class=m>9</span><span class=p>;</span> planar <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>18</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>2</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span> <span class=m>9</span><span class=p>;</span> tree <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>18</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>2</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span> <span class=m>9</span><span class=p>;</span> tree <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>18</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>2</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> X
<span class=nv>g</span><span class=o>=</span><span class=m>10</span><span class=p>;</span> rand <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>20</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>2</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>10</span><span class=p>;</span> rand <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>20</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>2</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>10</span><span class=p>;</span> 2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>20</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>2</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>10</span><span class=p>;</span> 2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>20</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>2</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>10</span><span class=p>;</span> rand+2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>20</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>2</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>10</span><span class=p>;</span> rand+2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>20</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>2</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>10</span><span class=p>;</span> planar <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>20</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>2</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>10</span><span class=p>;</span> planar <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>20</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>2</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>10</span><span class=p>;</span> tree <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>20</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>2</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> X
<span class=nv>g</span><span class=o>=</span><span class=m>10</span><span class=p>;</span> tree <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>20</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>2</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>11</span><span class=p>;</span> rand <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>22</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>3</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>11</span><span class=p>;</span> rand <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>22</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>3</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>11</span><span class=p>;</span> 2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>22</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>3</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>11</span><span class=p>;</span> 2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>22</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>3</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>11</span><span class=p>;</span> rand+2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>22</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>3</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>11</span><span class=p>;</span> rand+2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>22</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>3</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>11</span><span class=p>;</span> planar <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>22</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>3</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>11</span><span class=p>;</span> planar <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>22</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>3</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>11</span><span class=p>;</span> tree <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>22</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>3</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> X
<span class=nv>g</span><span class=o>=</span><span class=m>11</span><span class=p>;</span> tree <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>22</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>3</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> X
<span class=nv>g</span><span class=o>=</span><span class=m>12</span><span class=p>;</span> rand <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>24</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>3</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>12</span><span class=p>;</span> rand <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>24</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>3</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>12</span><span class=p>;</span> 2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>24</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>3</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>12</span><span class=p>;</span> 2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>24</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>3</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>12</span><span class=p>;</span> rand+2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>24</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>3</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>12</span><span class=p>;</span> rand+2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>24</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>3</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>12</span><span class=p>;</span> planar <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>24</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>3</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>12</span><span class=p>;</span> planar <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>24</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>3</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>12</span><span class=p>;</span> tree <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>24</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>3</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>12</span><span class=p>;</span> tree <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>24</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>3</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> X
<span class=nv>g</span><span class=o>=</span><span class=m>13</span><span class=p>;</span> rand <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>26</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>4</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>13</span><span class=p>;</span> rand <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>26</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>4</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>13</span><span class=p>;</span> 2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>26</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>4</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>13</span><span class=p>;</span> 2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>26</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>4</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>13</span><span class=p>;</span> rand+2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>26</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>4</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>13</span><span class=p>;</span> rand+2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>26</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>4</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>13</span><span class=p>;</span> planar <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>26</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>4</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>13</span><span class=p>;</span> planar <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>26</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>4</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>13</span><span class=p>;</span> tree <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>26</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>4</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> X
<span class=nv>g</span><span class=o>=</span><span class=m>13</span><span class=p>;</span> tree <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>26</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>4</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> X
<span class=nv>g</span><span class=o>=</span><span class=m>14</span><span class=p>;</span> rand <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>28</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>5</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>14</span><span class=p>;</span> rand <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>28</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>5</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>14</span><span class=p>;</span> 2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>28</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>5</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>14</span><span class=p>;</span> 2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>28</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>5</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>14</span><span class=p>;</span> rand+2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>28</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>5</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>14</span><span class=p>;</span> rand+2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>28</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>5</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>14</span><span class=p>;</span> planar <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>28</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>5</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>14</span><span class=p>;</span> planar <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>28</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>5</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>14</span><span class=p>;</span> tree <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>28</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>5</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> X
<span class=nv>g</span><span class=o>=</span><span class=m>14</span><span class=p>;</span> tree <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>28</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>5</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> X
<span class=nv>g</span><span class=o>=</span><span class=m>15</span><span class=p>;</span> rand <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>30</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>7</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>15</span><span class=p>;</span> rand <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>30</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>7</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>15</span><span class=p>;</span> 2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>30</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>7</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>15</span><span class=p>;</span> 2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>30</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>7</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>15</span><span class=p>;</span> rand+2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>30</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>7</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>15</span><span class=p>;</span> rand+2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>30</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>7</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>15</span><span class=p>;</span> planar <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>30</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>7</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>15</span><span class=p>;</span> planar <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>30</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>7</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>15</span><span class=p>;</span> tree <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>30</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>7</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> X
<span class=nv>g</span><span class=o>=</span><span class=m>15</span><span class=p>;</span> tree <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>30</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>7</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> X
<span class=nv>g</span><span class=o>=</span><span class=m>16</span><span class=p>;</span> rand <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>32</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>9</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>16</span><span class=p>;</span> rand <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>32</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>9</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>16</span><span class=p>;</span> 2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>32</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>9</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>16</span><span class=p>;</span> 2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>32</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>9</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>16</span><span class=p>;</span> rand+2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>32</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>9</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>16</span><span class=p>;</span> rand+2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>32</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>9</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>16</span><span class=p>;</span> planar <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>32</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>9</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>16</span><span class=p>;</span> planar <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>32</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>9</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> X
<span class=nv>g</span><span class=o>=</span><span class=m>16</span><span class=p>;</span> tree <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>32</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>9</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> X
<span class=nv>g</span><span class=o>=</span><span class=m>16</span><span class=p>;</span> tree <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>32</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>9</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> X
<span class=nv>g</span><span class=o>=</span><span class=m>17</span><span class=p>;</span> rand <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>34</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>11</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>17</span><span class=p>;</span> rand <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>34</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>11</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>17</span><span class=p>;</span> 2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>34</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>11</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> X
<span class=nv>g</span><span class=o>=</span><span class=m>17</span><span class=p>;</span> 2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>34</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>11</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> X
<span class=nv>g</span><span class=o>=</span><span class=m>17</span><span class=p>;</span> rand+2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>34</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>11</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>17</span><span class=p>;</span> rand+2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>34</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>11</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>17</span><span class=p>;</span> planar <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>34</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>11</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>17</span><span class=p>;</span> planar <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>34</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>11</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> X
<span class=nv>g</span><span class=o>=</span><span class=m>17</span><span class=p>;</span> tree <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>34</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>11</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> X
<span class=nv>g</span><span class=o>=</span><span class=m>17</span><span class=p>;</span> tree <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>34</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>11</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> X
<span class=nv>g</span><span class=o>=</span><span class=m>18</span><span class=p>;</span> rand <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>36</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>15</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>18</span><span class=p>;</span> rand <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>36</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>15</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>18</span><span class=p>;</span> 2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>36</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>15</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> X
<span class=nv>g</span><span class=o>=</span><span class=m>18</span><span class=p>;</span> 2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>36</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>15</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>18</span><span class=p>;</span> rand+2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>36</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>15</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>18</span><span class=p>;</span> rand+2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>36</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>15</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>18</span><span class=p>;</span> planar <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>36</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>15</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>18</span><span class=p>;</span> planar <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>36</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>15</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>18</span><span class=p>;</span> tree <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>36</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>15</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> X
<span class=nv>g</span><span class=o>=</span><span class=m>18</span><span class=p>;</span> tree <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>36</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>15</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> X
<span class=nv>g</span><span class=o>=</span><span class=m>19</span><span class=p>;</span> rand <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>38</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>20</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>19</span><span class=p>;</span> rand <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>38</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>20</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>19</span><span class=p>;</span> 2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>38</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>20</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>19</span><span class=p>;</span> 2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>38</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>20</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>19</span><span class=p>;</span> rand+2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>38</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>20</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>19</span><span class=p>;</span> rand+2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>38</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>20</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V<span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
</pre></div>
<p>About the results table: <span class=math>\(g\)</span> means log in base 2 of <span class=math>\(n\)</span>. In other words, <span class=math>\(n
\approx 2^g\)</span>. <code>ni</code> means network iteration (We do two iterations for every
specific network type and network size). <code>num_colors</code> means the amount of
colors we have for towers. <code>num_towers</code> is the amount of towers we have of
every color. <code>connected</code> and <code>sconnected</code> indicate whether the resulting
overlay directed graph of towers is weakly connected or strongly connected
respectively. <code>V</code> means yes, <code>X</code> means no. Note that if the overlay directed
graph is strongly connected, it must be also weakly connected. This means that
if we see a <code>V</code> in the last column, we must also see a <code>V</code> in the preceding column.</p>
<p>We can see in the results that the resulting overlay directed graph of towers
is not always strongly connected, but it seems to be weakly connected in all of
the examples we have checked. Similar results apply for the choice of
<span class=math>\(k = \log_{2}{n}\)</span> (The overlay directed graph of towers seems to be weakly
connected, but not strongly connected). For <span class=math>\(k = 0.5\log_{2}{n}\)</span> the
overlay directed graph of towers is sometimes not weakly connected.</p>
<p>The following are the results for <span class=math>\(k = \log_{2}{n}\)</span>:</p>
<div class=highlight><pre><span></span>$ cargo run --bin towers_scc --release
Compiling net_coords v0.1.0 <span class=o>(</span>file:///home/real/projects/d/freedomlayer/freedomlayer_research/network_coords/net_coords<span class=o>)</span>
Finished release <span class=o>[</span>optimized<span class=o>]</span> target<span class=o>(</span>s<span class=o>)</span> in <span class=m>3</span>.34 secs
Running <span class=sb>`</span>target/release/towers_scc<span class=sb>`</span>
Checking <span class=k>if</span> <span class=nb>local</span> towers overlay graph is strongly connected
<span class=nv>g</span><span class=o>=</span> <span class=m>6</span><span class=p>;</span> rand <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>6</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>2</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span> <span class=m>6</span><span class=p>;</span> rand <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>6</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>2</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span> <span class=m>6</span><span class=p>;</span> 2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>6</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>2</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span> <span class=m>6</span><span class=p>;</span> 2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>6</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>2</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span> <span class=m>6</span><span class=p>;</span> rand+2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>6</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>2</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span> <span class=m>6</span><span class=p>;</span> rand+2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>6</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>2</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span> <span class=m>6</span><span class=p>;</span> tree <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>6</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>2</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span> <span class=m>6</span><span class=p>;</span> tree <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>6</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>2</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> X
<span class=nv>g</span><span class=o>=</span> <span class=m>7</span><span class=p>;</span> rand <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>7</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>2</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span> <span class=m>7</span><span class=p>;</span> rand <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>7</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>2</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span> <span class=m>7</span><span class=p>;</span> 2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>7</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>2</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span> <span class=m>7</span><span class=p>;</span> 2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>7</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>2</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span> <span class=m>7</span><span class=p>;</span> rand+2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>7</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>2</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span> <span class=m>7</span><span class=p>;</span> rand+2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>7</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>2</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span> <span class=m>7</span><span class=p>;</span> tree <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>7</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>2</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span> <span class=m>7</span><span class=p>;</span> tree <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>7</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>2</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> X
<span class=nv>g</span><span class=o>=</span> <span class=m>8</span><span class=p>;</span> rand <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>8</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>3</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span> <span class=m>8</span><span class=p>;</span> rand <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>8</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>3</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span> <span class=m>8</span><span class=p>;</span> 2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>8</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>3</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> X
<span class=nv>g</span><span class=o>=</span> <span class=m>8</span><span class=p>;</span> 2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>8</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>3</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span> <span class=m>8</span><span class=p>;</span> rand+2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>8</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>3</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span> <span class=m>8</span><span class=p>;</span> rand+2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>8</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>3</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span> <span class=m>8</span><span class=p>;</span> tree <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>8</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>3</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> X
<span class=nv>g</span><span class=o>=</span> <span class=m>8</span><span class=p>;</span> tree <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>8</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>3</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> X
<span class=nv>g</span><span class=o>=</span> <span class=m>9</span><span class=p>;</span> rand <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>9</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>3</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span> <span class=m>9</span><span class=p>;</span> rand <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>9</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>3</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span> <span class=m>9</span><span class=p>;</span> 2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>9</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>3</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> X
<span class=nv>g</span><span class=o>=</span> <span class=m>9</span><span class=p>;</span> 2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>9</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>3</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span> <span class=m>9</span><span class=p>;</span> rand+2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>9</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>3</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span> <span class=m>9</span><span class=p>;</span> rand+2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>9</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>3</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span> <span class=m>9</span><span class=p>;</span> tree <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>9</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>3</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> X
<span class=nv>g</span><span class=o>=</span> <span class=m>9</span><span class=p>;</span> tree <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>9</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>3</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> X
<span class=nv>g</span><span class=o>=</span><span class=m>10</span><span class=p>;</span> rand <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>10</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>4</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>10</span><span class=p>;</span> rand <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>10</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>4</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>10</span><span class=p>;</span> 2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>10</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>4</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> X
<span class=nv>g</span><span class=o>=</span><span class=m>10</span><span class=p>;</span> 2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>10</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>4</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> X
<span class=nv>g</span><span class=o>=</span><span class=m>10</span><span class=p>;</span> rand+2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>10</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>4</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>10</span><span class=p>;</span> rand+2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>10</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>4</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>10</span><span class=p>;</span> tree <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>10</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>4</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> X
<span class=nv>g</span><span class=o>=</span><span class=m>10</span><span class=p>;</span> tree <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>10</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>4</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> X
<span class=nv>g</span><span class=o>=</span><span class=m>11</span><span class=p>;</span> rand <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>11</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>5</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>11</span><span class=p>;</span> rand <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>11</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>5</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>11</span><span class=p>;</span> 2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>11</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>5</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> X
<span class=nv>g</span><span class=o>=</span><span class=m>11</span><span class=p>;</span> 2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>11</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>5</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>11</span><span class=p>;</span> rand+2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>11</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>5</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>11</span><span class=p>;</span> rand+2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>11</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>5</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>11</span><span class=p>;</span> tree <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>11</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>5</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> X
<span class=nv>g</span><span class=o>=</span><span class=m>11</span><span class=p>;</span> tree <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>11</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>5</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> X
<span class=nv>g</span><span class=o>=</span><span class=m>12</span><span class=p>;</span> rand <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>12</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>6</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>12</span><span class=p>;</span> rand <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>12</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>6</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>12</span><span class=p>;</span> 2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>12</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>6</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>12</span><span class=p>;</span> 2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>12</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>6</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> X
<span class=nv>g</span><span class=o>=</span><span class=m>12</span><span class=p>;</span> rand+2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>12</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>6</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>12</span><span class=p>;</span> rand+2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>12</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>6</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>12</span><span class=p>;</span> tree <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>12</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>6</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> X
<span class=nv>g</span><span class=o>=</span><span class=m>12</span><span class=p>;</span> tree <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>12</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>6</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> X
<span class=nv>g</span><span class=o>=</span><span class=m>13</span><span class=p>;</span> rand <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>13</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>7</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>13</span><span class=p>;</span> rand <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>13</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>7</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>13</span><span class=p>;</span> 2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>13</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>7</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> X
<span class=nv>g</span><span class=o>=</span><span class=m>13</span><span class=p>;</span> 2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>13</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>7</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> X
<span class=nv>g</span><span class=o>=</span><span class=m>13</span><span class=p>;</span> rand+2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>13</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>7</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>13</span><span class=p>;</span> rand+2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>13</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>7</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>13</span><span class=p>;</span> tree <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>13</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>7</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> X
<span class=nv>g</span><span class=o>=</span><span class=m>13</span><span class=p>;</span> tree <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>13</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>7</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> X
<span class=nv>g</span><span class=o>=</span><span class=m>14</span><span class=p>;</span> rand <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>14</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>10</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>14</span><span class=p>;</span> rand <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>14</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>10</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>14</span><span class=p>;</span> 2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>14</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>10</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>14</span><span class=p>;</span> 2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>14</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>10</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>14</span><span class=p>;</span> rand+2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>14</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>10</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>14</span><span class=p>;</span> rand+2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>14</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>10</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>14</span><span class=p>;</span> tree <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>14</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>10</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> X
<span class=nv>g</span><span class=o>=</span><span class=m>14</span><span class=p>;</span> tree <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>14</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>10</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> X
<span class=nv>g</span><span class=o>=</span><span class=m>15</span><span class=p>;</span> rand <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>15</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>13</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>15</span><span class=p>;</span> rand <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>15</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>13</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>15</span><span class=p>;</span> 2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>15</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>13</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>15</span><span class=p>;</span> 2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>15</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>13</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> X
<span class=nv>g</span><span class=o>=</span><span class=m>15</span><span class=p>;</span> rand+2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>15</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>13</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>15</span><span class=p>;</span> rand+2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>15</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>13</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>15</span><span class=p>;</span> tree <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>15</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>13</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> X
<span class=nv>g</span><span class=o>=</span><span class=m>15</span><span class=p>;</span> tree <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>15</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>13</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> X
<span class=nv>g</span><span class=o>=</span><span class=m>16</span><span class=p>;</span> rand <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>16</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>17</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>16</span><span class=p>;</span> rand <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>16</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>17</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>16</span><span class=p>;</span> 2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>16</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>17</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> X
<span class=nv>g</span><span class=o>=</span><span class=m>16</span><span class=p>;</span> 2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>16</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>17</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>16</span><span class=p>;</span> rand+2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>16</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>17</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>16</span><span class=p>;</span> rand+2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>16</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>17</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>16</span><span class=p>;</span> tree <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>16</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>17</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> X
<span class=nv>g</span><span class=o>=</span><span class=m>16</span><span class=p>;</span> tree <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>16</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>17</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> X
<span class=nv>g</span><span class=o>=</span><span class=m>17</span><span class=p>;</span> rand <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>17</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>22</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>17</span><span class=p>;</span> rand <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>17</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>22</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>17</span><span class=p>;</span> 2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>17</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>22</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> X
<span class=nv>g</span><span class=o>=</span><span class=m>17</span><span class=p>;</span> 2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>17</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>22</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> X
<span class=nv>g</span><span class=o>=</span><span class=m>17</span><span class=p>;</span> rand+2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>17</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>22</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>17</span><span class=p>;</span> rand+2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>17</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>22</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>17</span><span class=p>;</span> tree <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>17</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>22</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> X
<span class=nv>g</span><span class=o>=</span><span class=m>17</span><span class=p>;</span> tree <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>17</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>22</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> X
<span class=nv>g</span><span class=o>=</span><span class=m>18</span><span class=p>;</span> rand <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>18</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>29</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>18</span><span class=p>;</span> rand <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>18</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>29</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>18</span><span class=p>;</span> 2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>18</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>29</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> X
<span class=nv>g</span><span class=o>=</span><span class=m>18</span><span class=p>;</span> 2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>18</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>29</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>18</span><span class=p>;</span> rand+2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>18</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>29</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>18</span><span class=p>;</span> rand+2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>18</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>29</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>18</span><span class=p>;</span> tree <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>18</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>29</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> X
<span class=nv>g</span><span class=o>=</span><span class=m>18</span><span class=p>;</span> tree <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>18</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>29</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> X
<span class=nv>g</span><span class=o>=</span><span class=m>19</span><span class=p>;</span> rand <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>19</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>39</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>19</span><span class=p>;</span> rand <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>19</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>39</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>19</span><span class=p>;</span> 2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>19</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>39</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> X
<span class=nv>g</span><span class=o>=</span><span class=m>19</span><span class=p>;</span> 2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>19</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>39</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>19</span><span class=p>;</span> rand+2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>19</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>39</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>19</span><span class=p>;</span> rand+2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>19</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>39</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>19</span><span class=p>;</span> tree <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>19</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>39</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> X
<span class=nv>g</span><span class=o>=</span><span class=m>19</span><span class=p>;</span> tree <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>19</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>39</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> X
<span class=nv>g</span><span class=o>=</span><span class=m>20</span><span class=p>;</span> rand <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>20</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>52</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>20</span><span class=p>;</span> rand <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>20</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>52</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>20</span><span class=p>;</span> 2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>20</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>52</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> X
<span class=nv>g</span><span class=o>=</span><span class=m>20</span><span class=p>;</span> 2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>20</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>52</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> X
<span class=nv>g</span><span class=o>=</span><span class=m>20</span><span class=p>;</span> rand+2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>20</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>52</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>20</span><span class=p>;</span> rand+2d <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>20</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>52</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> V
<span class=nv>g</span><span class=o>=</span><span class=m>20</span><span class=p>;</span> tree <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>0</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>20</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>52</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> X
<span class=nv>g</span><span class=o>=</span><span class=m>20</span><span class=p>;</span> tree <span class=p>;</span> <span class=nv>ni</span><span class=o>=</span><span class=m>1</span> <span class=p>|</span><span class=nv>num_colors</span> <span class=o>=</span> <span class=m>20</span> <span class=p>|</span><span class=nv>num_towers</span> <span class=o>=</span> <span class=m>52</span> <span class=p>|</span> <span class=nv>connected</span> <span class=o>=</span> V <span class=p>|</span> <span class=nv>sconnected</span> <span class=o>=</span> X
</pre></div>
<p>Note that the results are a bit worse for strong connectivity, because <span class=math>\(k\)</span> is
smaller. However, all the resulting overlay graphs in this experiment are
weakly connected.</p>
<h2>Explaining the weak connectivity</h2>
<p>We currently don’t have any rigorous proofs for the weak connectivity of the
towers overlay directed graph for <span class=math>\(k = \log_{2}{n}\)</span>. We could apply
the following approximate explanation:</p>
<p>Every tower is connected in the overlay graph to a few other towers. If these
connections were random, then we would expect that for <span class=math>\(k > \log(\sqrt{n})\)</span> the
towers overlay graph <a href="https://en.wikipedia.org/wiki/Erd%C5%91s%E2%80%93R%C3%A9nyi_model">will start becoming weakly
connected</a>.</p>
<p>This is not exact, because the connections between the towers are not really
random, as a connection between two towers mean that they are close in the
original graph.</p>
<script type="text/javascript">if (!document.getElementById('mathjaxscript_pelican_#%@#$@#')) {
var align = "center",
indent = "0em",
linebreak = "false";
if (false) {
align = (screen.width < 768) ? "left" : align;
indent = (screen.width < 768) ? "0em" : indent;
linebreak = (screen.width < 768) ? 'true' : linebreak;
}
var mathjaxscript = document.createElement('script');
mathjaxscript.id = 'mathjaxscript_pelican_#%@#$@#';
mathjaxscript.type = 'text/javascript';
mathjaxscript.src = 'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.0/MathJax.js?config=TeX-AMS-MML_HTMLorMML';
mathjaxscript[(window.opera ? "innerHTML" : "text")] =
"MathJax.Hub.Config({" +
" config: ['MMLorHTML.js']," +
" TeX: { extensions: ['AMSmath.js','AMSsymbols.js','noErrors.js','noUndefined.js'], equationNumbers: { autoNumber: 'AMS' } }," +
" jax: ['input/TeX','input/MathML','output/HTML-CSS']," +
" extensions: ['tex2jax.js','mml2jax.js','MathMenu.js','MathZoom.js']," +
" displayAlign: '"+ align +"'," +
" displayIndent: '"+ indent +"'," +
" showMathMenu: true," +
" messageStyle: 'normal'," +
" tex2jax: { " +
" inlineMath: [ ['\\\\(','\\\\)'] ], " +
" displayMath: [ ['$$','$$'] ]," +
" processEscapes: true," +
" preview: 'TeX'," +
" }, " +
" 'HTML-CSS': { " +
" styles: { '.MathJax_Display, .MathJax .mo, .MathJax .mi, .MathJax .mn': {color: 'inherit ! important'} }," +
" linebreaks: { automatic: "+ linebreak +", width: '90% container' }," +
" }, " +
"}); " +
"if ('default' !== 'default') {" +
"MathJax.Hub.Register.StartupHook('HTML-CSS Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax['HTML-CSS'].FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"MathJax.Hub.Register.StartupHook('SVG Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax.SVG.FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"}";
(document.body || document.getElementsByTagName('head')[0]).appendChild(mathjaxscript);
}
</script><ol class=simple-footnotes><li id=sf-the-connectivity-of-sqrtn-towers-1> The main bottleneck for
this implementation is the generation of the <code>planar</code> type network. It is
currently done in <span class=math>\(O(n^2)\)</span>, which takes a long time. <a class=simple-footnote-back href=#sf-the-connectivity-of-sqrtn-towers-1-back>↩</a></li></ol>Approximating the size of a mesh network2017-07-28T14:33:00+00:002017-07-28T14:33:00+00:00realtag:www.freedomlayer.org,2017-07-28:/approximating-the-size-of-a-mesh-network.html<div class="math">$$
\newcommand{\E}{\mathrm{E}}
\newcommand{\Var}{\mathrm{Var}}
\newcommand{\Cov}{\mathrm{Cov}}
$$</div>
<h2>Abstract</h2>
<p>We present here a method for approximating the amount of nodes
in a mesh network. We also introduce an experiment to test our approximation
method. The size approximation is done by taking the minimum hashes of various …</p><div class="math">$$
\newcommand{\E}{\mathrm{E}}
\newcommand{\Var}{\mathrm{Var}}
\newcommand{\Cov}{\mathrm{Cov}}
$$</div>
<h2>Abstract</h2>
<p>We present here a method for approximating the amount of nodes
in a mesh network. We also introduce an experiment to test our approximation
method. The size approximation is done by taking the minimum hashes of various
node ids and performing a transformation over those values.</p>
<p>The experiments are written in <a href="https://www.rust-lang.org">Rust</a> and could be
found <a href="https://github.com/realcr/freedomlayer_code/tree/master/approximate_net">here
[github]</a>.</p>
<h2>Hashing node ids</h2>
<p>Let <span class="math">\(G\)</span> be a connected network of nodes. Every node is connected to a few other nodes
that we call his neighbors. How can a node get an approximation of the amount
of nodes in the network?</p>
<p>We will discuss here a distributed algorithm that allows all the nodes of the
network to calculate an approximation.</p>
<p>Assume that every node has a unique identifier. For example, a public key or a
hash of a public key. Let <span class="math">\(I\)</span> be the set of all possible node ids. Assume that
we have a hash functions <span class="math">\(h: I \rightarrow [0,1]\)</span>. This function takes a node id as
input, and returns a value between 0 and 1 as output. </p>
<p>How to practically obtain this kind of function? We could use some
cryptographic hash function, like
<a href="https://en.wikipedia.org/wiki/SHA-2"><span class="caps">SHA256</span></a>, as follows:</p>
<div class="math">$$h(x) = sha256("myhash" || x) / 2^{256}$$</div>
<p>Where || means concatenation. We divide by <span class="math">\(2^{256}\)</span> to normalize, so that we
get a value between <span class="math">\(0\)</span> and <span class="math">\(1\)</span>. </p>
<h2>Observing the minimum hash value</h2>
<p>Consider the minimum value of the function <span class="math">\(h\)</span> over the set <span class="math">\(V(G)\)</span>. (We
implicitly assume that <span class="math">\(V(G) \subseteq I\)</span>). In other words, we apply <span class="math">\(h\)</span> over
all the node ids in the network, and pick the minimum value of <span class="math">\(h\)</span>. We call
this value <span class="math">\(min\)</span>.</p>
<p><span class="math">\(min\)</span> has some relation to the amount of nodes in the network. If there are
more nodes in the network, we expect <span class="math">\(min\)</span> to be smaller. This is because it is
more likely that some node <span class="math">\(v \in V(G)\)</span> will get a small value for <span class="math">\(h(v)\)</span>.</p>
<p>If there is only one node in the network, we expect <span class="math">\(min\)</span> to be about <span class="math">\(1/2\)</span>, on
average. Extending this idea, if there are two nodes in the network, we expect
<span class="math">\(min\)</span> to be about <span class="math">\(1/3\)</span>. Generally for <span class="math">\(n\)</span> nodes, we expect <span class="math">\(min\)</span> to have the
value <span class="math">\(1/(n+1)\)</span>.</p>
<p>We can prove this idea more rigorously. Consider <span class="math">\(n\)</span> random node ids <span class="math">\(v_1,
\dots v_{n}\)</span>. What is the probability that <span class="math">\(min\)</span> is going to be larger than
<span class="math">\(t\)</span>? This requires all of the values <span class="math">\(h(v_j)\)</span> to be larger than <span class="math">\(t\)</span>. The
probability of this event is <span class="math">\(\Pr[min > t] = (1-t)^n\)</span>, because <span class="math">\(\Pr[h(v_j) > t] =
1-t\)</span>, and the values of <span class="math">\(h\)</span> for different nodes are independent. Therefore
<span class="math">\(F_{min}(t) = \Pr[min \leq t] = 1 - (1-t)^n\)</span>. This is also known as the cumulative
probability function of <span class="math">\(min\)</span>.</p>
<p><span class="math">\(F(t)\)</span> is differentiable, and so we can obtain the density function: <span class="math">\(f(t) =
F'(t) = n(1-t)^{n-1}\)</span>. To obtain the mean of <span class="math">\(min\)</span> we can then calculate:
</p>
<div class="math">$$\E[min] = \int_{0}^{1} tf(t)dt = \int_{0}^{1} nt(1-t)^{n-1}dt$$</div>
<p>To solve this we first calculate:</p>
<div class="math">$$\begin{split}
\int_{0}^{1}t(1-t)^{n-1}dt = & \left[-\frac{t(1-t)^n}{n}\right]_{0}^{1} -
\int_{0}^{1} 1 \cdot \frac{(1-t)^n}{-n}dt \\
& = \int_{0}^{1}\frac{(1-t)^n}{n}dt =
\left[\frac{(1-t)^{n+1}}{-n(n+1)}\right]_{0}^{1} \\
& = \frac{1}{n(n+1)}
\end{split}$$</div>
<p>Hence <span class="math">\(\E[min] = n\cdot\frac{1}{n(n+1)} = \frac{1}{n+1}\)</span>. This result is similar
to our intuitive idea earlier.</p>
<p>For the sake of completeness, we are adding here the computation for
<span class="math">\(Var[min]\)</span>. <span class="math">\(Var[min] = \E[min^2] - \E[min]^2\)</span>. We already know <span class="math">\(\E[min]\)</span>, we are
now left to calculate <span class="math">\(\E[min^2]\)</span>.</p>
<div class="math">$$\begin{split}
\E[min^2] = & \int_{0}^{1} nt^2(1-t)^{n-1}dt =
\left[\frac{nt^2(1-t)^n}{-n}\right]_{0}^{1} -
\int_{0}^{1} n\cdot 2\cdot t\cdot\frac{(1-t)^n}{-n}dt \\
& = 2\int_{0}^{1}t(1-t)^n dt =
\left[\frac{2t(1-t)^{n+1}}{-(n+1)}\right]_{0}^{1} -
2\int_{0}^{1} \frac{(1-t)^{n+1}}{-(n+1)}dt \\
& = 2\int_{0}^{1} (1-t)^{n+1} dt =
\left[2\cdot\frac{(1-t)^{n+2}}{-(n+1)(n+2)}\right]_{0}^{1} \\
& = \frac{2}{(n+1)(n+2)}
\end{split}$$</div>
<p>Therefore </p>
<div class="math">$$\begin{split}
\Var[min] = & \E[min]^2 - \E[min^2] =
\frac{2}{(n+1)(n+2)} - \left(\frac{1}{n+1}\right)^2 \\
& = \frac{2(n+1)}{(n+1)^2(n+2)} - \frac{n+2}{(n+1)^2(n+2)} =
\frac{n}{(n+1)^2(n+2)}
\end{split}$$</div>
<p>We can also obtain that <span class="math">\(\Var[min] \leq \frac{n+1}{(n+1)^2(n+2)} =
\frac{1}{(n+1)(n+2)} \leq \frac{1}{(n+1)^2}\)</span>.</p>
<h2>Using multiple hash functions</h2>
<p>Having the minimum of the hash function <span class="math">\(h\)</span> over all the node ids in the
network gives us some idea about the amount of nodes in the network. <span class="math">\(min\)</span> is
about <span class="math">\(1 / (n+1)\)</span>, so we approximate <span class="math">\(n\)</span>, the amount of nodes in the network,
to be <span class="math">\((1 / min) - 1\)</span>.</p>
<p>As we have seen above, the variance of <span class="math">\(min\)</span> is pretty large. This means that
there is a possibility for large errors in our approximation method. One idea
to deal with this problem is to add more hash functions. Instead of just one
hash function <span class="math">\(h\)</span>, we will have <span class="math">\(r\)</span> hash functions, defined for example as:</p>
<div class="math">$$h_i(x) = sha256("myhash" || x || i) / 2^{256}$$</div>
<p>Where <span class="math">\(0 \leq i < r\)</span>. With <span class="math">\(r\)</span> hash functions we can obtain <span class="math">\(r\)</span> different min
values: <span class="math">\(min_i\)</span> for <span class="math">\(0 \leq i < r\)</span>. Those <span class="math">\(r\)</span> minimum values hold more
information about the network size than just one minimum value, however we
still have to figure out how to combine all those minimum values into one
approximation of the network size.</p>
<h2>Experimenting with combining multiple minimums</h2>
<p>We wrote a basic experiment (In Rust) to check the various methods for combining minimum
values for approximating the amount of nodes. It can be found <a href="https://github.com/realcr/freedomlayer_code/tree/master/approximate_net">here
[github]</a>.</p>
<p>We have tried various methods:</p>
<ul>
<li>Averaging all the minimum values and then calculating <span class="math">\((1 / avg) - 1\)</span>.</li>
<li>Calculating <span class="math">\((1 / min_i) - 1\)</span> for every <span class="math">\(0 \leq i < r\)</span> and then averaging the values.</li>
<li>Calculating harmonic average over the minimum values and then calculating
<span class="math">\((1 / avg) - 1\)</span>.</li>
<li>Calculating <span class="math">\((1 / min_i) - 1\)</span> for every <span class="math">\(0 \leq i < r\)</span> and then calculating
harmonic average over the values.</li>
</ul>
<p>This is how the various combination schemes look like on code:</p>
<div class="highlight"><pre><span></span><span class="k">pub</span><span class="w"> </span><span class="k">fn</span> <span class="nf">approx_size_harmonic_before</span><span class="p">(</span><span class="n">mins</span>: <span class="kp">&</span><span class="p">[</span><span class="kt">u64</span><span class="p">])</span><span class="w"> </span>-> <span class="kt">usize</span> <span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">fmeans</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">mins</span><span class="p">.</span><span class="n">iter</span><span class="p">()</span><span class="w"></span>
<span class="w"> </span><span class="p">.</span><span class="n">map</span><span class="p">(</span><span class="o">|&</span><span class="n">m</span><span class="o">|</span><span class="w"> </span><span class="n">m</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="kt">f64</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="p">.</span><span class="n">collect</span>::<span class="o"><</span><span class="nb">Vec</span><span class="o"><</span><span class="kt">f64</span><span class="o">>></span><span class="p">();</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">hmean_min</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">harmonic_mean</span><span class="p">(</span><span class="o">&</span><span class="n">fmeans</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="p">(((</span><span class="kt">u64</span>::<span class="n">max_value</span><span class="p">()</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="kt">f64</span><span class="p">)</span><span class="o">/</span><span class="w"> </span><span class="n">hmean_min</span><span class="p">)</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="kt">usize</span><span class="p">)</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="mi">1</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="k">pub</span><span class="w"> </span><span class="k">fn</span> <span class="nf">approx_size_harmonic_after</span><span class="p">(</span><span class="n">mins</span>: <span class="kp">&</span><span class="p">[</span><span class="kt">u64</span><span class="p">])</span><span class="w"> </span>-> <span class="kt">usize</span> <span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">trans</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">mins</span><span class="p">.</span><span class="n">iter</span><span class="p">()</span><span class="w"></span>
<span class="w"> </span><span class="p">.</span><span class="n">map</span><span class="p">(</span><span class="o">|&</span><span class="n">m</span><span class="o">|</span><span class="w"> </span><span class="p">(</span><span class="kt">u64</span>::<span class="n">max_value</span><span class="p">()</span><span class="w"> </span><span class="o">/</span><span class="w"> </span><span class="n">m</span><span class="p">)</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="mi">1</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="p">.</span><span class="n">map</span><span class="p">(</span><span class="o">|</span><span class="n">x</span><span class="o">|</span><span class="w"> </span><span class="n">x</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="kt">f64</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="p">.</span><span class="n">collect</span>::<span class="o"><</span><span class="nb">Vec</span><span class="o"><</span><span class="kt">f64</span><span class="o">>></span><span class="p">();</span><span class="w"></span>
<span class="w"> </span><span class="n">harmonic_mean</span><span class="p">(</span><span class="o">&</span><span class="n">trans</span><span class="p">)</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="kt">usize</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="k">pub</span><span class="w"> </span><span class="k">fn</span> <span class="nf">approx_size_mean_before</span><span class="p">(</span><span class="n">mins</span>: <span class="kp">&</span><span class="p">[</span><span class="kt">u64</span><span class="p">])</span><span class="w"> </span>-> <span class="kt">usize</span> <span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">fmeans</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">mins</span><span class="p">.</span><span class="n">iter</span><span class="p">()</span><span class="w"></span>
<span class="w"> </span><span class="p">.</span><span class="n">map</span><span class="p">(</span><span class="o">|&</span><span class="n">m</span><span class="o">|</span><span class="w"> </span><span class="n">m</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="kt">f64</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="p">.</span><span class="n">collect</span>::<span class="o"><</span><span class="nb">Vec</span><span class="o"><</span><span class="kt">f64</span><span class="o">>></span><span class="p">();</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">mean_min</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">mean</span><span class="p">(</span><span class="o">&</span><span class="n">fmeans</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="p">(((</span><span class="kt">u64</span>::<span class="n">max_value</span><span class="p">()</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="kt">f64</span><span class="p">)</span><span class="o">/</span><span class="w"> </span><span class="n">mean_min</span><span class="p">)</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="kt">usize</span><span class="p">)</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="mi">1</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="k">pub</span><span class="w"> </span><span class="k">fn</span> <span class="nf">approx_size_mean_after</span><span class="p">(</span><span class="n">mins</span>: <span class="kp">&</span><span class="p">[</span><span class="kt">u64</span><span class="p">])</span><span class="w"> </span>-> <span class="kt">usize</span> <span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">trans</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">mins</span><span class="p">.</span><span class="n">iter</span><span class="p">()</span><span class="w"></span>
<span class="w"> </span><span class="p">.</span><span class="n">map</span><span class="p">(</span><span class="o">|&</span><span class="n">m</span><span class="o">|</span><span class="w"> </span><span class="p">(</span><span class="kt">u64</span>::<span class="n">max_value</span><span class="p">()</span><span class="w"> </span><span class="o">/</span><span class="w"> </span><span class="n">m</span><span class="p">)</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="mi">1</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="p">.</span><span class="n">map</span><span class="p">(</span><span class="o">|</span><span class="n">x</span><span class="o">|</span><span class="w"> </span><span class="n">x</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="kt">f64</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="p">.</span><span class="n">collect</span>::<span class="o"><</span><span class="nb">Vec</span><span class="o"><</span><span class="kt">f64</span><span class="o">>></span><span class="p">();</span><span class="w"></span>
<span class="w"> </span><span class="n">mean</span><span class="p">(</span><span class="o">&</span><span class="n">trans</span><span class="p">)</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="kt">usize</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</pre></div>
<p>These are the results:</p>
<div class="highlight"><pre><span></span>$ cargo run --release
Compiling approximate_net v0.1.0
Finished release <span class="o">[</span>optimized<span class="o">]</span> target<span class="o">(</span>s<span class="o">)</span> in <span class="m">1</span>.44 secs
Running <span class="sb">`</span>target/release/approximate_net<span class="sb">`</span>
Calculating error ratios <span class="k">for</span> approximation functions...
<span class="nv">num_iters</span> <span class="o">=</span> <span class="m">100</span>
<span class="nv">num_mins</span> <span class="o">=</span> <span class="m">40</span>
<span class="nv">num_elems</span> <span class="o">=</span> <span class="m">1000000</span>
err_ratio <span class="k">for</span> approximation functions:
approx_size_harmonic_before : <span class="m">10</span>.208276536189299
approx_size_harmonic_after : <span class="m">0</span>.14301947584857105
approx_size_mean_before : <span class="m">0</span>.14301974775883225
approx_size_mean_after : <span class="m">10</span>.208276278935292
</pre></div>
<p>Explaining the experiment:</p>
<p>We generate a set of size <code>num_elems</code>. Each element is of type <code>u64</code>: a 64 bit
number. We then use <code>num_mins</code> different hash functions to find the minimums
over all the elements. We then obtain <code>num_mins</code> different minimums for all the
different hash functions. Finally we use one of <code>4</code> methods to approximate the
amount of elements in the set.</p>
<p>We do this whole process <code>num_iters</code> times, to get a general idea of how much
deviation each of those methods have from the original amount of elements in
the set. To check if an approximation method works well, we calculate the error ratio.
It is calculated as the standard deviation divided by <code>num_elems</code>, the amount
of elements in the set. We approximate the standard deviation using this calculation:</p>
<div class="math">$$\sigma \approx \sqrt{\frac{1}{numIters}
\sum_{0 \leq j < numIters}(approx_j - numElems)^2}$$</div>
<p>And we then calculate <span class="math">\(errRatio = \frac{\sigma}{numElems}\)</span>.</p>
<p>Looking at the results, approx_size_harmonic_after and approx_size_mean_before
give very similar results, and they have the best results in this group. The
other two methods (approx_size_harmonic_before and approx_size_mean_after) have
very bad results.</p>
<p>A friend noted that <code>approx_size_mean_after</code> and <code>approx_size_harmonic_before</code>
should be identical (up to floating point math errors). This is because:
</p>
<div class="math">$$\begin{split}
mean_i{ \left(\frac{1}{a_i} - 1 \right) } & = \frac{\sum_{i}{\left(\frac{1}{a_i} - 1\right)}}{n} \\
& = \frac{\sum_{i}{\frac{1}{a_i}}}{n} - 1 = \frac{1}{\frac{n}{\sum_{i}{\frac{1}{a_i}}} } - 1 \\
& = \frac{1}{harmonic_{i}{(a_i)}} - 1
\end{split}$$</div>
<p>For similar reasons, <code>approx_size_mean_before</code> and <code>approx_size_harmonic_after</code>
should be pretty close.</p>
<p>Our choice for combining the min values is going to be the
<code>approx_size_mean_before</code>, as it is the simplest to reason about.</p>
<h2>Distributed algorithm for calculating the hashes minimums</h2>
<p>One of the reasons we chose the minimum hashes method for approximating the
amount of nodes in the network is that it is easy to calculate in a mesh network.</p>
<p>Every node in the network maintains an array of the nodes with lowest hashes
that he ever seen, as follows: <span class="math">\(minHash[i] = (nodeId, timestamp, sign[nodeId](timestamp))\)</span>
for <span class="math">\(0 \leq i < r\)</span>.</p>
<p><span class="math">\(minHash[i]\)</span> corresponds to the node with lowest hash value for the function
<span class="math">\(h_i\)</span>. <span class="math">\(sign[nodeId](timestamp)\)</span> is a digital signature by the node <span class="math">\(nodeId\)</span>
over some timestamp. This proves that <span class="math">\(nodeId\)</span> was recently alive.</p>
<p>Given that every node knows the set of minimums for all the <span class="math">\(r\)</span> different hash
functions, every node in the network can calculate an approximation for the
amount of nodes in the network.</p>
<p>We now describe the distributed algorithm for a given node <span class="math">\(x\)</span>:</p>
<p><strong>Initialization</strong></p>
<p>When a node <span class="math">\(x\)</span> enters the network, he first fills:
<span class="math">\(minHash_x[i] = (x, sign[x](timestamp))\)</span> for <span class="math">\(0 \leq i < r\)</span>. In other words, <span class="math">\(x\)</span>
fills himself as the node with lowest hash for all the hash functions.</p>
<p><strong>Periodic cleaning</strong></p>
<p>For every node <span class="math">\(x\)</span>, every constant period of time the following happens: </p>
<ol>
<li>
<p><span class="math">\(x\)</span> checks his list of <span class="math">\(minHash\)</span> nodes. If any timestamp is too old, the
entry <span class="math">\(minHash[i] = (nodeId, timestamp, sign[nodeId](timestamp))\)</span> is removed
from the list and <span class="math">\(x\)</span> puts himself instead: <span class="math">\(minHash_x[i] = (x ,
currentTimestamp, sign[x](currentTimestamp))\)</span>. </p>
</li>
<li>
<p><span class="math">\(x\)</span> sends all his direct network neighbors a message the contains the
updated <span class="math">\(minHash\)</span> entry: <span class="math">\(UpdateMinNodeId(minHash_x[i])\)</span></p>
</li>
</ol>
<p><strong>Receipt of UpdateMinNodeId message</strong></p>
<p>Assume that a node <span class="math">\(x\)</span> receives an UpdateMinNodeId message with <span class="math">\(minHash[i] = (nodeId,
timestamp, sign[nodeId](timestamp))\)</span>. Also assume that <span class="math">\(x\)</span> currently have
a current entry: <span class="math">\(minHash_x[i]\)</span> as the node with lowest value for <span class="math">\(h_i\)</span>.</p>
<ol>
<li>If the timestamp is too old, abort.</li>
<li>If the signature by <span class="math">\(nodeId\)</span> over the timestamp is not valid, abort.</li>
<li>If <span class="math">\(h_i(nodeId) > h_i(minHash_x[i].nodeId)\)</span> then abort. (<span class="math">\(x\)</span><span class="quo">‘</span>s node with
lowest <span class="math">\(h_i\)</span> value has lower value than the candidate).</li>
<li>If <span class="math">\(h_i(nodeId) = h_i(minHash_x[i].nodeId)\)</span> and the timestamp of the
candidate is older then abort.</li>
<li>Update <span class="math">\(minHash_x[i] = minHash[i]\)</span></li>
<li><span class="math">\(x\)</span> sends all his direct network neighbors a message that contains the
updated <span class="math">\(minHash\)</span> entry: <span class="math">\(UpdateMinNodeId(minHash_x[i])\)</span></li>
</ol>
<h2>Further research</h2>
<h3>Accuracy</h3>
<p>It is known that there are more accurate methods to approximate the unique
amount of elements in a set. However, not all of those methods are adequate to
work as a distributed algorithm.</p>
<p>We know that the arithmetic mean gives good enough results for approximating
the network size for our purposes, however it is possible that better methods
exist to approximate the network size only given various minimum values of
random hash functions over the node ids.</p>
<h3>Security</h3>
<p>If a node can choose his own node id, he may generate many node ids until his
node id gives a low value for one of the hashes. This can allow one participant
in the network to greatly affect the approximation for the amount of nodes in
the network. Currently the only way we know of to avoid this problem is to not
let nodes pick their own node ids, and to invalidate node ids from time to
time. See the article “How to spread Adversarial Nodes? Rotate!” by Christian
Scheideler for more about this idea.</p>
<h2>Further reading</h2>
<ul>
<li><a href="https://blog.demofox.org/2015/02/03/estimating-counts-of-distinct-values-with-kmv/">Estimating counts of distinct values with kmv</a></li>
<li><span class="dquo">“</span>HyperLogLog: the analysis of a near-optimal cardinality estimation
algorithm” by Philippe Flajolet, Éric Fusy, Olivier Gandouet, Frédéric Meunier</li>
<li><a href="http://tech.adroll.com/blog/data/2013/07/10/hll-minhash.html">HyperLogLog and MinHash</a></li>
<li><a href="https://en.wikipedia.org/wiki/Jaccard_index">Jaccard Index (wikipedia)</a></li>
<li><a href="https://en.wikipedia.org/wiki/MinHash">MinHash (wikipedia)</a></li>
<li><span class="dquo">“</span>How to spread Adversarial Nodes? Rotate!” by Christian Scheideler.</li>
</ul>
<script type="text/javascript">if (!document.getElementById('mathjaxscript_pelican_#%@#$@#')) {
var align = "center",
indent = "0em",
linebreak = "false";
if (false) {
align = (screen.width < 768) ? "left" : align;
indent = (screen.width < 768) ? "0em" : indent;
linebreak = (screen.width < 768) ? 'true' : linebreak;
}
var mathjaxscript = document.createElement('script');
mathjaxscript.id = 'mathjaxscript_pelican_#%@#$@#';
mathjaxscript.type = 'text/javascript';
mathjaxscript.src = 'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.0/MathJax.js?config=TeX-AMS-MML_HTMLorMML';
mathjaxscript[(window.opera ? "innerHTML" : "text")] =
"MathJax.Hub.Config({" +
" config: ['MMLorHTML.js']," +
" TeX: { extensions: ['AMSmath.js','AMSsymbols.js','noErrors.js','noUndefined.js'], equationNumbers: { autoNumber: 'AMS' } }," +
" jax: ['input/TeX','input/MathML','output/HTML-CSS']," +
" extensions: ['tex2jax.js','mml2jax.js','MathMenu.js','MathZoom.js']," +
" displayAlign: '"+ align +"'," +
" displayIndent: '"+ indent +"'," +
" showMathMenu: true," +
" messageStyle: 'normal'," +
" tex2jax: { " +
" inlineMath: [ ['\\\\(','\\\\)'] ], " +
" displayMath: [ ['$$','$$'] ]," +
" processEscapes: true," +
" preview: 'TeX'," +
" }, " +
" 'HTML-CSS': { " +
" styles: { '.MathJax_Display, .MathJax .mo, .MathJax .mi, .MathJax .mn': {color: 'inherit ! important'} }," +
" linebreaks: { automatic: "+ linebreak +", width: '90% container' }," +
" }, " +
"}); " +
"if ('default' !== 'default') {" +
"MathJax.Hub.Register.StartupHook('HTML-CSS Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax['HTML-CSS'].FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"MathJax.Hub.Register.StartupHook('SVG Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax.SVG.FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"}";
(document.body || document.getElementsByTagName('head')[0]).appendChild(mathjaxscript);
}
</script>Node coordinate discovery by uniform hashing2017-06-22T10:50:00+00:002017-06-22T10:50:00+00:00realtag:www.freedomlayer.org,2017-06-22:/node-coordinate-discovery-by-uniform-hashing.html<h3>Abstract</h3>
<p>Landmarks based routing allows efficient routing of messages between nodes in a
network. Given two nodes <span class="math">\(x, y\)</span> in a network <span class="math">\(G\)</span>, <span class="math">\(x\)</span> can send a message to <span class="math">\(y\)</span>
given that <span class="math">\(x\)</span> knows the network coordinate <span class="math">\(Coord(y)\)</span> of <span class="math">\(y\)</span>.</p>
<p>Hence we need some method of translating a node …</p><h3>Abstract</h3>
<p>Landmarks based routing allows efficient routing of messages between nodes in a
network. Given two nodes <span class="math">\(x, y\)</span> in a network <span class="math">\(G\)</span>, <span class="math">\(x\)</span> can send a message to <span class="math">\(y\)</span>
given that <span class="math">\(x\)</span> knows the network coordinate <span class="math">\(Coord(y)\)</span> of <span class="math">\(y\)</span>.</p>
<p>Hence we need some method of translating a node id <span class="math">\(y\)</span> into his
network coordinate, <span class="math">\(Coord(y)\)</span>.</p>
<p>We discuss here one method of translation: hashing of node id into uniform
valid coordinate.</p>
<p>The code for all the experiments is written in
<a href="https://www.rust-lang.org">Rust</a>, and could be found <a href="https://github.com/realcr/freedomlayer_code/tree/master/landmarks_discovery_hashing/net_coords">here [github]</a>.</p>
<h3>Hashing node ids into coordinates</h3>
<p>Assume that we had some deterministic way of hashing a node id, <span class="math">\(y\)</span>, into some
coordinate <span class="math">\(H(y)\)</span>. (Note that <span class="math">\(H(y)\)</span> has no relation to <span class="math">\(Coord(y)\)</span>).</p>
<p>The node <span class="math">\(y\)</span> will then calculate <span class="math">\(H(y)\)</span>, obtaining some network coordinate.
<span class="math">\(y\)</span> will then find the node <span class="math">\(z\)</span> with network coordinate closest to <span class="math">\(H(y)\)</span>, and
let <span class="math">\(z\)</span> keep <span class="math">\(y\)</span><span class="quo">‘</span>s coordinate: <span class="math">\(Coord(y)\)</span>.</p>
<p>If some node <span class="math">\(x\)</span> wants to send a message to <span class="math">\(y\)</span>, <span class="math">\(x\)</span> will first calculate
<span class="math">\(H(y)\)</span>. <span class="math">\(H\)</span> is deterministic, so <span class="math">\(x\)</span> will get the same value of <span class="math">\(H(y)\)</span> that <span class="math">\(y\)</span>
got. <span class="math">\(x\)</span> will then find the node <span class="math">\(z\)</span> with a coordinate closest to <span class="math">\(H(y)\)</span>, and
ask him for <span class="math">\(Coord(y)\)</span>.</p>
<p>Finally, knowing <span class="math">\(Coord(y)\)</span>, <span class="math">\(x\)</span> can send a message to <span class="math">\(y\)</span>.</p>
<p>There are some requirements to make this technique work:</p>
<ul>
<li>
<p>The process of applying the function <span class="math">\(H\)</span>, followed by finding a node with
closest coordinate to the result of <span class="math">\(H\)</span> should distribute close to uniformly
over the set of nodes. This is because we don’t want that few nodes will keep
the coordinates of most of the nodes in the network.</p>
</li>
<li>
<p>The coordinate produced by <span class="math">\(H\)</span> should look like a “real coordinate”. It
should satisfy the triangle inequalities, and hopefully look like other
coordinates in the network.</p>
</li>
</ul>
<h3>Generating random coordinates</h3>
<p>We want to create a function <span class="math">\(H: V(G) \rightarrow Coordinates\)</span> that produces
deterministic results, even when invoked from different nodes. For this to
happen, we need to rely on some common knowledge that all nodes have.</p>
<p>We will rely on the knowledge of the coordinates of all the landmarks:
<span class="math">\(Coord(l_i)\)</span> for <span class="math">\(1 \leq i \leq k\)</span>.</p>
<p>We examined a few algorithms of generating random coordinates that are uniform
over the set of nodes in the network. We introduce here two algorithms that
gave us good results.</p>
<h3><code>randomize_coord_landmarks_coords</code></h3>
<p>The first algorithm, called <code>randomize_coord_landmarks_coords</code>, does the
following to create a random coordinate: For entry <span class="math">\(j\)</span>, we pick a random entry
of a random coordinate from all the coordinates <span class="math">\(Coord(l_i)\)</span>, <span class="math">\(1 \leq i \leq k\)</span>.</p>
<p>Example rust code for this algorithm:</p>
<div class="highlight"><pre><span></span><span class="k">pub</span><span class="w"> </span><span class="k">fn</span> <span class="nf">randomize_coord_landmarks_coords</span><span class="o"><</span><span class="n">R</span>: <span class="nc">Rng</span><span class="o">></span><span class="p">(</span><span class="n">landmarks</span>: <span class="kp">&</span><span class="nb">Vec</span><span class="o"><</span><span class="kt">usize</span><span class="o">></span><span class="p">,</span><span class="w"> </span><span class="n">coords</span>: <span class="kp">&</span><span class="nb">Vec</span><span class="o"><</span><span class="nb">Vec</span><span class="o"><</span><span class="kt">u64</span><span class="o">>></span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">rng</span>: <span class="kp">&</span><span class="nc">mut</span><span class="w"> </span><span class="n">R</span><span class="p">)</span><span class="w"> </span>-> <span class="nb">Vec</span><span class="o"><</span><span class="kt">u64</span><span class="o">></span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">rand_landmark</span>: <span class="nc">Range</span><span class="o"><</span><span class="kt">usize</span><span class="o">></span><span class="w"> </span><span class="o">=</span><span class="w"> </span>
<span class="w"> </span><span class="n">Range</span>::<span class="n">new</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span><span class="w"> </span><span class="n">landmarks</span><span class="p">.</span><span class="n">len</span><span class="p">());</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">interval_size</span>: <span class="kt">u64</span> <span class="o">=</span><span class="w"> </span><span class="mi">2_</span><span class="k">u64</span><span class="p">.</span><span class="n">pow</span><span class="p">(</span><span class="mi">0_</span><span class="k">u32</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">coord</span>: <span class="nb">Vec</span><span class="o"><</span><span class="kt">u64</span><span class="o">></span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">vec</span><span class="o">!</span><span class="p">[];</span><span class="w"></span>
<span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="n">_</span><span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="p">..</span><span class="w"> </span><span class="n">landmarks</span><span class="p">.</span><span class="n">len</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">cur_value</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">0</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="n">_</span><span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="p">..</span><span class="w"> </span><span class="n">interval_size</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">cur_value</span><span class="w"> </span><span class="o">+=</span><span class="w"> </span><span class="n">coords</span><span class="p">[</span><span class="n">landmarks</span><span class="p">[</span><span class="n">rand_landmark</span><span class="p">.</span><span class="n">ind_sample</span><span class="p">(</span><span class="o">&</span><span class="k">mut</span><span class="w"> </span><span class="n">rng</span><span class="p">)]][</span><span class="n">rand_landmark</span><span class="p">.</span><span class="n">ind_sample</span><span class="p">(</span><span class="o">&</span><span class="k">mut</span><span class="w"> </span><span class="n">rng</span><span class="p">)];</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="n">coord</span><span class="p">.</span><span class="n">push</span><span class="p">(</span><span class="n">cur_value</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="n">coord</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</pre></div>
<p>This algorithm is not very smart. It does not produce coordinates that satisfy
the triangle inequalities, but it does produce coordinates that distribute
somewhat uniformly over all the nodes in the network.</p>
<p>We include here some results for this algorithm, for small sized networks.
Explanation about how to read the result: <span class="math">\(g\)</span> means log in base 2 of the size
of the network. (For example, <span class="math">\(g=9\)</span> means a network of <span class="math">\(512\)</span> nodes).
We check against <span class="math">\(4\)</span> types of networks:
- rand: A random network. Every node is connected to <span class="math">\(1.5 * \log(n)\)</span> other
nodes.
- 2d: A two dimensional grid.
- rand+2d: A sum of a random network with a two dimensional grid.
- planar: A planar graph: Nodes are randomized in a two dimensional plane, and
every node is connected to the closet <span class="math">\(1.5 * \log(n)\)</span> nodes.</p>
<p><span class="math">\(ni\)</span> means network iteration. In this experiment we make two iterations for
every type of network. We do this to have more numbers to look at.</p>
<p>In this experiment we generate <span class="math">\(n = |V(G)|\)</span> random coordinates, and for each
coordinate we find the node with a coordinate closets to the generated coordinate.</p>
<p>For each node we count the amount of times it was the closest node to the
generated coordinate. <code>max_nr</code> is the maximum number of times we visited a
single node. If <code>max_nr</code> is very large, it means that only a few nodes were
visited during the experiment, and the algorithm chosen for randomizing
coordinates is not very uniform.</p>
<p>We also for every randomly generated coordinate how many nodes are fit to be
the node with closest node to this coordinate (There could be more than one).
<code>average_min_indices</code> is the average of this amount, over all the randomly
generated coordinates. It is usually <span class="math">\(1\)</span>, because usually there is only one
closest node.</p>
<p>Parameters for this experiment:</p>
<ul>
<li>Amount of landmarks: <span class="math">\(\log(n)^2\)</span></li>
<li>Tie breaking by randomizing edge distances: Done with interval <span class="math">\([0x10000,
0x20000)\)</span>. (This helps with having a low <code>average_min_indices</code> value).</li>
</ul>
<div class="highlight"><pre><span></span>git clone https://github.com/realcr/freedomlayer_code
<span class="nb">cd</span> landmarks_discovery_hashing/net_coords
</pre></div>
<p>Edit <code>landmarks_discovery_hashing/net_coords/bin/randomize_coord_balanced</code> to
use <code>randomize_coord_landmarks_coords</code>.</p>
<p>Then run:</p>
<div class="highlight"><pre><span></span>cargo run --bin randomize_coord_balanced --release
</pre></div>
<p>Results:</p>
<div class="highlight"><pre><span></span>g=10; rand ; ni=0 |max_nr = 8| average_min_indices = 1
g=10; rand ; ni=1 |max_nr = 6| average_min_indices = 1.0009765625
g=10; 2d ; ni=0 |max_nr = 11| average_min_indices = 1
g=10; 2d ; ni=1 |max_nr = 7| average_min_indices = 1
g=10; rand+2d ; ni=0 |max_nr = 11| average_min_indices = 1.0009765625
g=10; rand+2d ; ni=1 |max_nr = 8| average_min_indices = 1.0009765625
g=10; planar ; ni=0 |max_nr = 14| average_min_indices = 1
g=10; planar ; ni=1 |max_nr = 15| average_min_indices = 1
g=11; rand ; ni=0 |max_nr = 9| average_min_indices = 1.00048828125
g=11; rand ; ni=1 |max_nr = 9| average_min_indices = 1.0009765625
g=11; 2d ; ni=0 |max_nr = 11| average_min_indices = 1
g=11; 2d ; ni=1 |max_nr = 8| average_min_indices = 1
g=11; rand+2d ; ni=0 |max_nr = 11| average_min_indices = 1.0004938271604937
g=11; rand+2d ; ni=1 |max_nr = 8| average_min_indices = 1
g=11; planar ; ni=0 |max_nr = 14| average_min_indices = 1
g=11; planar ; ni=1 |max_nr = 17| average_min_indices = 1
g=12; rand ; ni=0 |max_nr = 15| average_min_indices = 1.000732421875
g=12; rand ; ni=1 |max_nr = 16| average_min_indices = 1.000244140625
g=12; 2d ; ni=0 |max_nr = 14| average_min_indices = 1
g=12; 2d ; ni=1 |max_nr = 10| average_min_indices = 1
g=12; rand+2d ; ni=0 |max_nr = 8| average_min_indices = 1.000244140625
g=12; rand+2d ; ni=1 |max_nr = 11| average_min_indices = 1.00048828125
g=12; planar ; ni=0 |max_nr = 13| average_min_indices = 1.000244140625
g=12; planar ; ni=1 |max_nr = 21| average_min_indices = 1.000244140625
g=13; rand ; ni=0 |max_nr = 26| average_min_indices = 1
g=13; rand ; ni=1 |max_nr = 15| average_min_indices = 1.0001220703125
g=13; 2d ; ni=0 |max_nr = 14| average_min_indices = 1
g=13; 2d ; ni=1 |max_nr = 12| average_min_indices = 1.000246913580247
g=13; rand+2d ; ni=0 |max_nr = 18| average_min_indices = 1.0003703703703704
g=13; rand+2d ; ni=1 |max_nr = 18| average_min_indices = 1
g=13; planar ; ni=0 |max_nr = 44| average_min_indices = 1
g=13; planar ; ni=1 |max_nr = 19| average_min_indices = 1
</pre></div>
<h3><code>randomize_coord_rw_directional</code></h3>
<p>The first algorithm presented, <code>randomize_coord_landmarks_coords</code>, generates
random coordinates that don’t satisfy the triangle inequality with respect to
the landmarks coordinates.</p>
<p>Let <span class="math">\(d(x,y)\)</span> is the length of the shortest path between <span class="math">\(x\)</span> and <span class="math">\(y\)</span> in the
network, <span class="math">\(d\)</span> is a metric over <span class="math">\(V(G)\)</span>, the vertices of the network.
Let <span class="math">\(\{l_1, \dots , l_k\} \subseteq V(G)\)</span> be the chosen landmarks for the network.
For every node <span class="math">\(x\)</span>, we define: </p>
<div class="math">$$Coord(x) := \{d(x,l_1), \dots , d(x, l_k)\}$$</div>
<p>We then define:</p>
<div class="math">$$mdist(Coord(x), Coord(y)) := \max_{1 \leq i \leq k} {|Coord(x)_i - Coord(y)_i|}$$</div>
<p>The virtual distance between <span class="math">\(x\)</span> and <span class="math">\(y\)</span>. <span class="math">\(mdist\)</span> satisfies <span class="math">\(mdist(Coord(x),
Coord(y)) \leq d(x,y)\)</span>. This follows from the triangle inequality over the
metric <span class="math">\(d\)</span>.</p>
<p>We call a coordinate <span class="math">\(A\)</span> valid if it satisfies all the triangle inequalities
according to the metric <span class="math">\(d\)</span>. In other words, <span class="math">\(A_i + A_j \geq d(l_i,l_j) \geq
|A_i - A_j|\)</span>. Hence if two coordinates <span class="math">\(A\)</span> and <span class="math">\(B\)</span> are valid, <span class="math">\(\alpha \cdot A +
(1 - \alpha) \cdot B\)</span> is also a valid coordinate for <span class="math">\(\alpha \in [0,1]\)</span>.
Therefore the set of valid coordinates is convex.</p>
<p>In addition, if a coordinate <span class="math">\(A = (a_1, \dots , a_k)\)</span> is valid, then <span class="math">\(A +
\beta \cdot 1 = (a_1 + \beta, \dots, a_k + \beta)\)</span> is also valid. Hence the set
of valid coordinates has infinite volume.</p>
<p>If we add an artificial constraint of <span class="math">\(coord_i \leq const * \max_i
{Coord(l_j)_i}\)</span>, we get a bounded set of valid coordinates, with contraints
that are linear inequalities. This kind of shape is also known as a polytope.</p>
<p>It appears that the question of choosing a random point inside the volume of a
polytope was studied in the past. (Mostly for approximation of volume for high
dimensional bodies, and finding redundant equations in a set of linear inequalities). </p>
<p>See, for example:</p>
<ul>
<li><span class="dquo">“</span>Hit-And-Run algorithms for the identification of nonredundant linear
equations”
(<span class="caps">H.C.P.</span> Berbee, <span class="caps">C.G.</span>E Boender, <span class="caps">A.H.</span>G Rinnooy Kan, <span class="caps">C.L.</span> Scheffer, <span class="caps">R.L.</span> Smith,
J. Telgen)</li>
<li><span class="dquo">“</span>Random walks and an O*(n^5) volume algorithm for convex bodies” (Ravi
Kannan, László Lovász, Miklós Simonovits)</li>
</ul>
<p>We use an algorithm that is similar to the one presented in “Hit-AndRun
algorithms”. It works as follows:</p>
<p>We begin with some coordinate <span class="math">\(C\)</span>. We can possibly pick one of <span class="math">\(Coord(l_i)\)</span> for
some <span class="math">\(1 \leq i \leq k\)</span> as a starting point, or possibly the average of all the
coordinates of the landmarks (If we use distance randomization, the average
is usually a valid coordinate).</p>
<p>In every step we pick a random entry index <span class="math">\(j\)</span>, where <span class="math">\(1 \leq j \leq k\)</span>. We
check what is the possible range for <span class="math">\(C_j\)</span> so that it still satisfies the
triangle inequality. Let <span class="math">\(a\)</span> be the small possible value, and <span class="math">\(b\)</span> be the highest
possible value. We then pick a random value uniformly in <span class="math">\([a,b]\)</span>, and finally
set <span class="math">\(C_j\)</span> to be the new value of entry <span class="math">\(j\)</span> of the coordinate.</p>
<p>After enough steps, we expect to reach a random coordinate valid coordinate.</p>
<p>This is the code for this algorithm:</p>
<div class="highlight"><pre><span></span><span class="k">pub</span><span class="w"> </span><span class="k">fn</span> <span class="nf">randomize_coord_rw_directional</span><span class="o"><</span><span class="n">R</span>: <span class="nc">Rng</span><span class="o">></span><span class="p">(</span><span class="n">upper_constraints</span>: <span class="kp">&</span><span class="nb">Vec</span><span class="o"><</span><span class="kt">u64</span><span class="o">></span><span class="p">,</span><span class="w"> </span>
<span class="w"> </span><span class="n">landmarks</span>: <span class="kp">&</span><span class="nb">Vec</span><span class="o"><</span><span class="kt">usize</span><span class="o">></span><span class="p">,</span><span class="w"> </span><span class="n">coords</span>: <span class="kp">&</span><span class="nb">Vec</span><span class="o"><</span><span class="nb">Vec</span><span class="o"><</span><span class="kt">u64</span><span class="o">>></span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">rng</span>: <span class="kp">&</span><span class="nc">mut</span><span class="w"> </span><span class="n">R</span><span class="p">)</span><span class="w"> </span>-> <span class="nb">Vec</span><span class="o"><</span><span class="kt">u64</span><span class="o">></span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">entry_range</span>: <span class="nc">Range</span><span class="o"><</span><span class="kt">usize</span><span class="o">></span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">Range</span>::<span class="n">new</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span><span class="w"> </span><span class="n">landmarks</span><span class="p">.</span><span class="n">len</span><span class="p">());</span><span class="w"></span>
<span class="w"> </span><span class="c1">// Start from the average of all landmarks:</span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">cur_coord</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">average_landmarks</span><span class="p">(</span><span class="o">&</span><span class="n">landmarks</span><span class="p">,</span><span class="w"> </span><span class="o">&</span><span class="n">coords</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="c1">// Start from a random landmark:</span>
<span class="w"> </span><span class="c1">// let mut cur_coord = coords[landmarks[entry_range.ind_sample(rng)]].clone();</span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">good_iters</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">0</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="k">while</span><span class="w"> </span><span class="n">good_iters</span><span class="w"> </span><span class="o"><</span><span class="w"> </span><span class="n">landmarks</span><span class="p">.</span><span class="n">len</span><span class="p">().</span><span class="n">pow</span><span class="p">(</span><span class="mi">2</span><span class="p">)</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="mi">4</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">i</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">entry_range</span><span class="p">.</span><span class="n">ind_sample</span><span class="p">(</span><span class="n">rng</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="c1">// Get range of valid values for entry number i:</span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="p">(</span><span class="n">low</span><span class="p">,</span><span class="w"> </span><span class="n">high</span><span class="p">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">get_entry_rw_range</span><span class="p">(</span><span class="o">&</span><span class="n">cur_coord</span><span class="p">,</span><span class="w"> </span><span class="n">i</span><span class="p">,</span><span class="w"> </span>
<span class="w"> </span><span class="o">&</span><span class="n">upper_constraints</span><span class="p">,</span><span class="w"> </span><span class="n">landmarks</span><span class="p">,</span><span class="w"> </span><span class="n">coords</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="n">low</span><span class="w"> </span><span class="o"><</span><span class="w"> </span><span class="n">high</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">good_iters</span><span class="w"> </span><span class="o">+=</span><span class="w"> </span><span class="mi">1</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="c1">// Set the new random value to the entry:</span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">value_range</span>: <span class="nc">Range</span><span class="o"><</span><span class="kt">u64</span><span class="o">></span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">Range</span>::<span class="n">new</span><span class="p">(</span><span class="n">low</span><span class="p">,</span><span class="w"> </span><span class="n">high</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="mi">1</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="n">cur_coord</span><span class="p">[</span><span class="n">i</span><span class="p">]</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">value_range</span><span class="p">.</span><span class="n">ind_sample</span><span class="p">(</span><span class="n">rng</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="n">cur_coord</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</pre></div>
<p>We check the uniformity of the resulting generated coordinates, the same way we
did with the previous algorithm: <code>randomize_coord_landmarks_coords</code>.</p>
<div class="highlight"><pre><span></span>git clone https://github.com/realcr/freedomlayer_code
<span class="nb">cd</span> landmarks_discovery_hashing/net_coords
cargo run --bin randomize_coord_balanced --release
</pre></div>
<div class="highlight"><pre><span></span>g=10; rand ; ni=0 |max_nr = 21| average_min_indices = 1
g=10; rand ; ni=1 |max_nr = 26| average_min_indices = 1
g=10; 2d ; ni=0 |max_nr = 21| average_min_indices = 1
g=10; 2d ; ni=1 |max_nr = 58| average_min_indices = 1
g=10; rand+2d ; ni=0 |max_nr = 24| average_min_indices = 1
g=10; rand+2d ; ni=1 |max_nr = 26| average_min_indices = 1
g=10; planar ; ni=0 |max_nr = 33| average_min_indices = 1
g=10; planar ; ni=1 |max_nr = 16| average_min_indices = 1
g=11; rand ; ni=0 |max_nr = 39| average_min_indices = 1
g=11; rand ; ni=1 |max_nr = 78| average_min_indices = 1
g=11; 2d ; ni=0 |max_nr = 63| average_min_indices = 1
g=11; 2d ; ni=1 |max_nr = 47| average_min_indices = 1
g=11; rand+2d ; ni=0 |max_nr = 27| average_min_indices = 1
g=11; rand+2d ; ni=1 |max_nr = 32| average_min_indices = 1
g=11; planar ; ni=0 |max_nr = 72| average_min_indices = 1
g=11; planar ; ni=1 |max_nr = 28| average_min_indices = 1
</pre></div>
<h3><code>find_coord</code> experiment</h3>
<p>We perform an end-to-end experiment, that works as follows.
We pick a pair of nodes <span class="math">\(x,y \in V(G)\)</span> randomly. We then generate a random
coordinate using a chosen coordinate generation algorithm. Call that coordinate
<span class="math">\(C\)</span>.</p>
<p><span class="math">\(x\)</span> then searches for a node with coordinate close to <span class="math">\(C\)</span>, call that node <span class="math">\(z\)</span>.
Finally <span class="math">\(y\)</span> searches for <span class="math">\(z\)</span>, given the coordinate <span class="math">\(C\)</span>.</p>
<p><span class="math">\(y\)</span> searches for <span class="math">\(z\)</span> as follows: <span class="math">\(y\)</span> first chooses a random coordinate <span class="math">\(M\)</span> and
finds a node <span class="math">\(m\)</span> close to <span class="math">\(M\)</span>. From that node, <span class="math">\(y\)</span> tries to find the node <span class="math">\(z\)</span>,
given the coordinate <span class="math">\(C\)</span>. If many iterations pass and <span class="math">\(y\)</span> couldn’t find <span class="math">\(z\)</span>
from <span class="math">\(m\)</span>, <span class="math">\(y\)</span> generates a new random coordinate <span class="math">\(M\)</span> and tries again.</p>
<p>In the results tables:
<span class="math">\(g\)</span> is the log of the size of the network, <span class="math">\(ni\)</span> means the network iteration.
Again we check against <span class="math">\(4\)</span> types of networks: rand, 2d, rand+2d, planar.
<code>avg_path_len</code> is the average resulting path length for arriving from <span class="math">\(y\)</span> to
<span class="math">\(z\)</span>. </p>
<p><code>avg_num_attemps</code> is the most important number for this experiment: It
shows how many times <span class="math">\(y\)</span> had to generate a new <span class="math">\(M\)</span> and try to find <span class="math">\(z\)</span>.
Ideally this number should be as low as possible: 1.</p>
<p>Results for <code>randomize_coord_landmarks_coords</code>:</p>
<div class="highlight"><pre><span></span>git clone https://github.com/realcr/freedomlayer_code
<span class="nb">cd</span> landmarks_discovery_hashing/net_coords
</pre></div>
<p>Edit <code>landmarks_discovery_hashing/net_coords/bin/find_coords</code> to
use <code>randomize_coord_landmarks_coords</code>.</p>
<p>Then run:</p>
<div class="highlight"><pre><span></span>cargo run --bin find_coords --release
</pre></div>
<div class="highlight"><pre><span></span>max_visits = 2
num_pairs = 10
g= 8; rand ; ni=0 |avg_path_len = 17.100 |avg_num_attempts = 1.000 |
g= 8; rand ; ni=1 |avg_path_len = 15.900 |avg_num_attempts = 1.000 |
g= 8; 2d ; ni=0 |avg_path_len = 47.600 |avg_num_attempts = 2.200 |
g= 8; 2d ; ni=1 |avg_path_len = 55.500 |avg_num_attempts = 1.700 |
g= 8; rand+2d ; ni=0 |avg_path_len = 15.400 |avg_num_attempts = 1.000 |
g= 8; rand+2d ; ni=1 |avg_path_len = 17.700 |avg_num_attempts = 1.000 |
g= 8; planar ; ni=0 |avg_path_len = 25.100 |avg_num_attempts = 1.000 |
g= 8; planar ; ni=1 |avg_path_len = 23.300 |avg_num_attempts = 2.200 |
g= 9; rand ; ni=0 |avg_path_len = 21.200 |avg_num_attempts = 1.000 |
g= 9; rand ; ni=1 |avg_path_len = 13.900 |avg_num_attempts = 1.000 |
g= 9; 2d ; ni=0 |avg_path_len = 37.100 |avg_num_attempts = 1.900 |
g= 9; 2d ; ni=1 |avg_path_len = 41.700 |avg_num_attempts = 2.700 |
g= 9; rand+2d ; ni=0 |avg_path_len = 19.800 |avg_num_attempts = 1.000 |
g= 9; rand+2d ; ni=1 |avg_path_len = 17.400 |avg_num_attempts = 1.000 |
g= 9; planar ; ni=0 |avg_path_len = 48.500 |avg_num_attempts = 1.200 |
g= 9; planar ; ni=1 |avg_path_len = 50.400 |avg_num_attempts = 3.500 |
g=10; rand ; ni=0 |avg_path_len = 25.800 |avg_num_attempts = 1.000 |
g=10; rand ; ni=1 |avg_path_len = 16.500 |avg_num_attempts = 1.000 |
g=10; 2d ; ni=0 |avg_path_len = 48.300 |avg_num_attempts = 2.800 |
g=10; 2d ; ni=1 |avg_path_len = 126.100 |avg_num_attempts = 2.200 |
g=10; rand+2d ; ni=0 |avg_path_len = 22.400 |avg_num_attempts = 1.000 |
g=10; rand+2d ; ni=1 |avg_path_len = 15.400 |avg_num_attempts = 1.000 |
g=10; planar ; ni=0 |avg_path_len = 59.400 |avg_num_attempts = 1.200 |
g=10; planar ; ni=1 |avg_path_len = 81.900 |avg_num_attempts = 3.400 |
g=11; rand ; ni=0 |avg_path_len = 26.000 |avg_num_attempts = 1.900 |
g=11; rand ; ni=1 |avg_path_len = 23.400 |avg_num_attempts = 1.000 |
g=11; 2d ; ni=0 |avg_path_len = 64.900 |avg_num_attempts = 40.200 |
g=11; 2d ; ni=1 |avg_path_len = 56.100 |avg_num_attempts = 33.600 |
g=11; rand+2d ; ni=0 |avg_path_len = 51.200 |avg_num_attempts = 1.000 |
g=11; rand+2d ; ni=1 |avg_path_len = 25.700 |avg_num_attempts = 1.000 |
g=11; planar ; ni=0 |avg_path_len = 37.700 |avg_num_attempts = 5.800 |
g=11; planar ; ni=1 |avg_path_len = 39.700 |avg_num_attempts = 2.800 |
</pre></div>
<p>Results for <code>randomize_coord_rw_directional</code>:</p>
<div class="highlight"><pre><span></span>git clone https://github.com/realcr/freedomlayer_code
<span class="nb">cd</span> landmarks_discovery_hashing/net_coords
cargo run --bin find_coords --release
</pre></div>
<div class="highlight"><pre><span></span>max_visits = 2
num_pairs = 10
g= 8; rand ; ni=0 |avg_path_len = 15.400 |avg_num_attempts = 1.000 |
g= 8; rand ; ni=1 |avg_path_len = 15.100 |avg_num_attempts = 1.000 |
g= 8; 2d ; ni=0 |avg_path_len = 41.900 |avg_num_attempts = 1.200 |
g= 8; 2d ; ni=1 |avg_path_len = 38.200 |avg_num_attempts = 1.100 |
g= 8; rand+2d ; ni=0 |avg_path_len = 16.700 |avg_num_attempts = 1.000 |
g= 8; rand+2d ; ni=1 |avg_path_len = 15.200 |avg_num_attempts = 1.000 |
g= 8; planar ; ni=0 |avg_path_len = 26.400 |avg_num_attempts = 2.900 |
g= 8; planar ; ni=1 |avg_path_len = 26.800 |avg_num_attempts = 1.400 |
g= 9; rand ; ni=0 |avg_path_len = 17.100 |avg_num_attempts = 1.000 |
g= 9; rand ; ni=1 |avg_path_len = 21.100 |avg_num_attempts = 1.000 |
g= 9; 2d ; ni=0 |avg_path_len = 47.200 |avg_num_attempts = 2.400 |
g= 9; 2d ; ni=1 |avg_path_len = 91.300 |avg_num_attempts = 5.900 |
g= 9; rand+2d ; ni=0 |avg_path_len = 14.800 |avg_num_attempts = 1.000 |
g= 9; rand+2d ; ni=1 |avg_path_len = 15.500 |avg_num_attempts = 1.000 |
g= 9; planar ; ni=0 |avg_path_len = 33.100 |avg_num_attempts = 1.000 |
g= 9; planar ; ni=1 |avg_path_len = 29.400 |avg_num_attempts = 1.600 |
g=10; rand ; ni=0 |avg_path_len = 21.000 |avg_num_attempts = 1.000 |
g=10; rand ; ni=1 |avg_path_len = 20.200 |avg_num_attempts = 1.000 |
g=10; 2d ; ni=0 |avg_path_len = 71.200 |avg_num_attempts = 4.700 |
g=10; 2d ; ni=1 |avg_path_len = 132.300 |avg_num_attempts = 3.900 |
g=10; rand+2d ; ni=0 |avg_path_len = 33.900 |avg_num_attempts = 1.000 |
g=10; rand+2d ; ni=1 |avg_path_len = 19.100 |avg_num_attempts = 1.000 |
g=10; planar ; ni=0 |avg_path_len = 46.800 |avg_num_attempts = 10.100 |
g=10; planar ; ni=1 |avg_path_len = 54.500 |avg_num_attempts = 2.800 |
g=11; rand ; ni=0 |avg_path_len = 77.300 |avg_num_attempts = 1.000 |
g=11; rand ; ni=1 |avg_path_len = 19.000 |avg_num_attempts = 1.000 |
g=11; 2d ; ni=0 |avg_path_len = 184.200 |avg_num_attempts = 17.900 |
g=11; 2d ; ni=1 |avg_path_len = 101.100 |avg_num_attempts = 9.300 |
g=11; rand+2d ; ni=0 |avg_path_len = 24.100 |avg_num_attempts = 1.000 |
g=11; rand+2d ; ni=1 |avg_path_len = 18.400 |avg_num_attempts = 1.000 |
g=11; planar ; ni=0 |avg_path_len = 104.800 |avg_num_attempts = 39.900 |
g=11; planar ; ni=1 |avg_path_len = 108.900 |avg_num_attempts = 37.800 |
</pre></div>
<p>Note that <code>avg_num_attempts</code> is almost always <span class="math">\(1\)</span> for the networks rand,
rand+2d, but it blows up for the networks 2d, planar.</p>
<p>This means that the idea of node discovery by node id hashing is practical for
random looking network. However, at this point we don’t know how to make it
work for planar like networks.</p>
<script type="text/javascript">if (!document.getElementById('mathjaxscript_pelican_#%@#$@#')) {
var align = "center",
indent = "0em",
linebreak = "false";
if (false) {
align = (screen.width < 768) ? "left" : align;
indent = (screen.width < 768) ? "0em" : indent;
linebreak = (screen.width < 768) ? 'true' : linebreak;
}
var mathjaxscript = document.createElement('script');
mathjaxscript.id = 'mathjaxscript_pelican_#%@#$@#';
mathjaxscript.type = 'text/javascript';
mathjaxscript.src = 'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.0/MathJax.js?config=TeX-AMS-MML_HTMLorMML';
mathjaxscript[(window.opera ? "innerHTML" : "text")] =
"MathJax.Hub.Config({" +
" config: ['MMLorHTML.js']," +
" TeX: { extensions: ['AMSmath.js','AMSsymbols.js','noErrors.js','noUndefined.js'], equationNumbers: { autoNumber: 'AMS' } }," +
" jax: ['input/TeX','input/MathML','output/HTML-CSS']," +
" extensions: ['tex2jax.js','mml2jax.js','MathMenu.js','MathZoom.js']," +
" displayAlign: '"+ align +"'," +
" displayIndent: '"+ indent +"'," +
" showMathMenu: true," +
" messageStyle: 'normal'," +
" tex2jax: { " +
" inlineMath: [ ['\\\\(','\\\\)'] ], " +
" displayMath: [ ['$$','$$'] ]," +
" processEscapes: true," +
" preview: 'TeX'," +
" }, " +
" 'HTML-CSS': { " +
" styles: { '.MathJax_Display, .MathJax .mo, .MathJax .mi, .MathJax .mn': {color: 'inherit ! important'} }," +
" linebreaks: { automatic: "+ linebreak +", width: '90% container' }," +
" }, " +
"}); " +
"if ('default' !== 'default') {" +
"MathJax.Hub.Register.StartupHook('HTML-CSS Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax['HTML-CSS'].FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"MathJax.Hub.Register.StartupHook('SVG Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax.SVG.FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"}";
(document.body || document.getElementsByTagName('head')[0]).appendChild(mathjaxscript);
}
</script>Landmarks routing with lookahead experiment results2017-06-21T12:43:00+00:002017-06-21T12:43:00+00:00realtag:www.freedomlayer.org,2017-06-21:/landmarks-routing-with-lookahead-experiment-results.html<h2>Abstract</h2>
<p>We present here some experiment results for a variation of the landmarks
routing method. Those results show that landmarks routing could be used in
practice for efficient routing in a network.
The experiments are written in <a href="https://www.rust-lang.org">Rust</a> and could be
found on github, <a href="https://github.com/realcr/freedomlayer_code/tree/master/landmarks_lookahead">here [github]</a>.</p>
<h2>A Short reminder about …</h2><h2>Abstract</h2>
<p>We present here some experiment results for a variation of the landmarks
routing method. Those results show that landmarks routing could be used in
practice for efficient routing in a network.
The experiments are written in <a href="https://www.rust-lang.org">Rust</a> and could be
found on github, <a href="https://github.com/realcr/freedomlayer_code/tree/master/landmarks_lookahead">here [github]</a>.</p>
<h2>A Short reminder about network coordinates</h2>
<p>Given a network <span class="math">\(G\)</span> of <span class="math">\(n\)</span> nodes, we pick <span class="math">\(k\)</span> nodes randomly <span class="math">\(\{l_1, \dots ,
l_k\}\)</span> and call them the landmarks of the network. The amount of landmarks we
pick is about <span class="math">\(\log(n)^2\)</span>, where <span class="math">\(n = |V(G)|\)</span>, the amount of nodes in the network.</p>
<p>Every node <span class="math">\(x\)</span> maintains shortest paths to all the landmarks. Specifically,
every node should know his distances to each of the landmarks. If we order the
landmarks somehow, this results in every node having a coordinate, specifying
his location in the network. For some node <span class="math">\(x\)</span>, this coordinate consists of
the distances of <span class="math">\(x\)</span> to each of the landmarks <span class="math">\(l_i\)</span>.</p>
<div class="math">$$Coord(x) := \{d(x,l_1), \dots , d(x, l_k)\}$$</div>
<p>Where <span class="math">\(d(x,y)\)</span> is the length of the shortest path between the nodes <span class="math">\(x\)</span> and
<span class="math">\(y\)</span>.</p>
<p>Recall that landmarks based routing (Using random walking) was discussed
earlier in: <a href="https://www.freedomlayer.org/landmarks-navigation-by-random-walking.html">Landmarks navigation by random walking</a>.</p>
<p>We use the network coordinates to approximate distances between nodes in the
network. This should help us with routing messages between nodes.</p>
<p>The distance between two nodes <span class="math">\(x\)</span> and <span class="math">\(y\)</span> is approximated as follows: We take
the <span class="math">\(x\)</span><span class="quo">‘</span>s and <span class="math">\(y\)</span><span class="quo">‘</span>s coordinates, subtract the entries pairwise, take absolute
values and find the maximum: </p>
<div class="math">$$mdist(Coord(x), Coord(y)) := \max_{1 \leq i \leq k} {|Coord(x)_i - Coord(y)_i|}$$</div>
<p><span class="math">\(mdist\)</span> satisfies <span class="math">\(mdist(Coord(x), Coord(y)) \leq d(x,y)\)</span>. This follows from
the triangle inequality over the metric <span class="math">\(d\)</span>.
<span class="math">\(mdist\)</span> itself is a semi metric. It is symmetric and always satisfies the triangle
inequality. However, it is possible that <span class="math">\(mdist(C) = mdist(D)\)</span> for two
different coordinates <span class="math">\(C \neq D\)</span>.</p>
<h2>Using lookahead</h2>
<p>In <a href="https://www.freedomlayer.org/landmarks-navigation-by-random-walking.html">Landmarks navigation by random walking</a>
we ran an experiment of routing with network coordinates, relying on a random
walk. It didn’t gave us very good results.</p>
<p>In this experiment we will use a different strategy. Assume that we want to
route from a node <span class="math">\(x\)</span> to a node <span class="math">\(y\)</span>. We begin from node <span class="math">\(x\)</span>, and in every
iteration we pick a neighbor that is closest to <span class="math">\(y\)</span> according to <span class="math">\(mdist\)</span> over
the network coordinates.</p>
<p>Using this method we arrive quickly at some node <span class="math">\(z\)</span> with low <span class="math">\(mdist(z,y)\)</span>, and
every neighbor <span class="math">\(w\)</span> of <span class="math">\(z\)</span> has a higher value of <span class="math">\(mdist(w,y)\)</span>. This means that
we are stuck. </p>
<p>Whenever we are stuck at some node <span class="math">\(z\)</span>, we pick a random neighbor of <span class="math">\(z\)</span> and continue from there.</p>
<p>This algorithm can also get stuck. We can solve this by letting every node know
more about the network. For every node <span class="math">\(z\)</span>, instead of picking the best
neighbor every time, we will pick from all nodes of some distance from <span class="math">\(z\)</span>.</p>
<p>It appears that this method allows efficient routing in various types of
networks. We knows this from experiments. Unfortunately, we have no formal ways
of proving this.</p>
<h2>Comparing landmarks routing and chord virtual <span class="caps">DHT</span> routing</h2>
<p>The following experiment is used to check the performance of landmarks routing
with lookahead, and compare its performance with chord <a href="https://www.freedomlayer.org/experimenting-with-virtual-dht-routing.html">Virtual <span class="caps">DHT</span>
routing</a>.
Also see <a href="https://www.freedomlayer.org/articles/assets/chord_connected_routing.pdf">A globally connected overlay for Virtual Ring Routing [pdf]</a>.</p>
<p>How to read this table? </p>
<p><span class="math">\(g\)</span> is the log in base 2 of the size of the network. <span class="math">\(g=6\)</span> means a network of
size <span class="math">\(64\)</span>. The network column shows which type of network the routing is
performed on. There are three types of networks we check: </p>
<ul>
<li>rand: A random network where every node is connected to about <span class="math">\(1.5 * log(n)\)</span> other nodes.</li>
<li>2d: A two dimensional grid.</li>
<li>rand+2d: A sum of a two dimensional grid and a random network.</li>
</ul>
<p>Note that for every type of network we generate three networks, to have more
numbers to look at. <span class="math">\(ni\)</span> means network iteration. </p>
<p>There are three routing methods we consider: </p>
<ul>
<li>
<p>chord: Virtual <span class="caps">DHT</span> routing. We programmed the idea presented at <a href="https://www.freedomlayer.org/articles/assets/chord_connected_routing.pdf">A globally
connected overlay for Virtual Ring Routing [pdf]</a>.</p>
</li>
<li>
<p>landmarks nei^2: Landmarks routing where every node can see approximately all
nodes in radius 2 from himself. (His direct neighbors, and the neighbors of his neighbors).</p>
</li>
<li>
<p>landmarks nei^3: Landmarks routing where every node can see approximately all
nodes in radius 3 from himself.</p>
</li>
</ul>
<p>For every routing experiment, we keep the average path length (left), the
maximum path length (middle) and the ratio of routing success (right).</p>
<p>If the maximum path length becomes too large, we stop the specific experiment
and put stars (<code>***</code>) in all subsequent routing experiments of the same type.</p>
<p>The experiments are written in <a href="https://www.rust-lang.org">Rust</a>. The results are presented here.
You can get the same results yourself by running:</p>
<div class="highlight"><pre><span></span>git clone https://github.com/realcr/freedomlayer_code
cd landmarks_lookahead/net_coords
cargo run --bin full_matrix --release
</pre></div>
<div class="highlight"><pre><span></span> Network | chord | landmarks nei^2 | landmarks nei^3
---------------------+------------------------+------------------------+------------------------+
g= 6; rand ; ni=0 | 2.87, 9, 1.00 | 1.99, 3, 1.00 | 1.99, 3, 1.00 |
g= 6; rand ; ni=1 | 2.85, 9, 1.00 | 1.91, 3, 1.00 | 1.91, 3, 1.00 |
g= 6; rand ; ni=2 | 2.82, 9, 1.00 | 2.02, 3, 1.00 | 2.02, 3, 1.00 |
g= 6; 2d ; ni=0 | 9.19, 39, 1.00 | 5.51, 13, 1.00 | 5.51, 13, 1.00 |
g= 6; 2d ; ni=1 | 9.09, 30, 1.00 | 5.17, 10, 1.00 | 5.17, 10, 1.00 |
g= 6; 2d ; ni=2 | 8.93, 28, 1.00 | 5.34, 13, 1.00 | 5.34, 13, 1.00 |
g= 6; rand+2d ; ni=0 | 2.58, 8, 1.00 | 1.83, 3, 1.00 | 1.83, 3, 1.00 |
g= 6; rand+2d ; ni=1 | 2.53, 8, 1.00 | 1.89, 3, 1.00 | 1.89, 3, 1.00 |
g= 6; rand+2d ; ni=2 | 2.45, 7, 1.00 | 1.82, 3, 1.00 | 1.82, 3, 1.00 |
g= 7; rand ; ni=0 | 3.66, 10, 1.00 | 2.15, 3, 1.00 | 2.15, 3, 1.00 |
g= 7; rand ; ni=1 | 3.60, 11, 1.00 | 2.07, 3, 1.00 | 2.07, 3, 1.00 |
g= 7; rand ; ni=2 | 3.67, 10, 1.00 | 2.13, 3, 1.00 | 2.13, 3, 1.00 |
g= 7; 2d ; ni=0 | 15.57, 55, 1.00 | 6.91, 15, 1.00 | 6.91, 15, 1.00 |
g= 7; 2d ; ni=1 | 15.46, 54, 1.00 | 6.70, 16, 1.00 | 6.70, 16, 1.00 |
g= 7; 2d ; ni=2 | 15.49, 49, 1.00 | 7.62, 17, 1.00 | 7.62, 17, 1.00 |
g= 7; rand+2d ; ni=0 | 3.18, 9, 1.00 | 2.04, 3, 1.00 | 2.04, 3, 1.00 |
g= 7; rand+2d ; ni=1 | 3.00, 11, 1.00 | 1.91, 3, 1.00 | 1.91, 3, 1.00 |
g= 7; rand+2d ; ni=2 | 3.17, 9, 1.00 | 1.92, 3, 1.00 | 1.92, 3, 1.00 |
g= 8; rand ; ni=0 | 4.59, 13, 1.00 | 2.40, 3, 1.00 | 2.40, 3, 1.00 |
g= 8; rand ; ni=1 | 4.74, 12, 1.00 | 2.33, 3, 1.00 | 2.33, 3, 1.00 |
g= 8; rand ; ni=2 | 4.71, 13, 1.00 | 2.30, 3, 1.00 | 2.30, 3, 1.00 |
g= 8; 2d ; ni=0 | 32.81, 105, 1.00 | 9.75, 23, 1.00 | 9.75, 23, 1.00 |
g= 8; 2d ; ni=1 | 32.13, 95, 1.00 | 11.60, 27, 1.00 | 11.60, 27, 1.00 |
g= 8; 2d ; ni=2 | 30.22, 96, 1.00 | 10.11, 23, 1.00 | 10.11, 23, 1.00 |
g= 8; rand+2d ; ni=0 | 4.19, 12, 1.00 | 2.13, 3, 1.00 | 2.13, 3, 1.00 |
g= 8; rand+2d ; ni=1 | 4.18, 12, 1.00 | 2.08, 3, 1.00 | 2.08, 3, 1.00 |
g= 8; rand+2d ; ni=2 | 4.25, 15, 1.00 | 2.17, 3, 1.00 | 2.17, 3, 1.00 |
g= 9; rand ; ni=0 | 6.05, 18, 1.00 | 3.19, 11, 1.00 | 2.54, 3, 1.00 |
g= 9; rand ; ni=1 | 5.86, 14, 1.00 | 3.12, 9, 1.00 | 2.55, 3, 1.00 |
g= 9; rand ; ni=2 | 6.07, 16, 1.00 | 3.26, 13, 1.00 | 2.57, 4, 1.00 |
g= 9; 2d ; ni=0 | 61.83, 206, 1.00 | 14.72, 37, 1.00 | 14.72, 37, 1.00 |
g= 9; 2d ; ni=1 | 53.72, 165, 1.00 | 15.30, 32, 1.00 | 15.30, 32, 1.00 |
g= 9; 2d ; ni=2 | 60.82, 354, 1.00 | 13.41, 40, 1.00 | 13.41, 40, 1.00 |
g= 9; rand+2d ; ni=0 | 5.29, 14, 1.00 | 2.33, 3, 1.00 | 2.33, 3, 1.00 |
g= 9; rand+2d ; ni=1 | 5.27, 15, 1.00 | 2.25, 3, 1.00 | 2.25, 3, 1.00 |
g= 9; rand+2d ; ni=2 | 5.31, 14, 1.00 | 2.40, 3, 1.00 | 2.40, 3, 1.00 |
g=10; rand ; ni=0 | 7.48, 18, 1.00 | 4.55, 26, 1.00 | 2.62, 4, 1.00 |
g=10; rand ; ni=1 | 7.54, 21, 1.00 | 4.58, 27, 1.00 | 2.65, 3, 1.00 |
g=10; rand ; ni=2 | 7.48, 17, 1.00 | 4.20, 24, 1.00 | 2.69, 3, 1.00 |
g=10; 2d ; ni=0 | 130.53, 462, 1.00 | 20.86, 55, 1.00 | 20.86, 55, 1.00 |
g=10; 2d ; ni=1 | 129.02, 811, 1.00 | 19.58, 49, 1.00 | 19.58, 49, 1.00 |
g=10; 2d ; ni=2 | 156.51, 1003, 1.00 | 22.75, 52, 1.00 | 22.75, 52, 1.00 |
g=10; rand+2d ; ni=0 | 6.78, 16, 1.00 | 4.00, 15, 1.00 | 2.61, 3, 1.00 |
g=10; rand+2d ; ni=1 | 6.76, 17, 1.00 | 3.83, 19, 1.00 | 2.58, 3, 1.00 |
g=10; rand+2d ; ni=2 | 6.78, 18, 1.00 | 3.63, 15, 1.00 | 2.52, 3, 1.00 |
g=11; rand ; ni=0 | 9.57, 22, 1.00 | 7.52, 43, 1.00 | 2.87, 4, 1.00 |
g=11; rand ; ni=1 | 9.48, 21, 1.00 | 5.56, 22, 1.00 | 2.75, 4, 1.00 |
g=11; rand ; ni=2 | 9.43, 24, 1.00 | 6.00, 35, 1.00 | 2.76, 3, 1.00 |
g=11; 2d ; ni=0 | 294.56, 1116, 1.00 | 31.23, 71, 1.00 | 31.23, 71, 1.00 |
g=11; 2d ; ni=1 | 358.45, 1237, 1.00 | 28.14, 64, 1.00 | 28.14, 64, 1.00 |
g=11; 2d ; ni=2 | 355.77, 1242, 1.00 | 29.17, 75, 1.00 | 29.17, 75, 1.00 |
g=11; rand+2d ; ni=0 | 8.75, 18, 1.00 | 5.37, 25, 1.00 | 2.70, 3, 1.00 |
g=11; rand+2d ; ni=1 | 8.80, 20, 1.00 | 4.89, 23, 1.00 | 2.57, 3, 1.00 |
g=11; rand+2d ; ni=2 | 8.90, 24, 1.00 | 5.91, 29, 1.00 | 2.74, 3, 1.00 |
g=12; rand ; ni=0 | 12.55, 31, 1.00 | 11.09, 71, 1.00 | 2.93, 4, 1.00 |
g=12; rand ; ni=1 | 12.25, 42, 1.00 | 10.40, 141, 1.00 | 2.84, 4, 1.00 |
g=12; rand ; ni=2 | 12.39, 43, 1.00 | 8.59, 45, 1.00 | 2.94, 4, 1.00 |
g=12; 2d ; ni=0 | 817.58, 3433, 1.00 | 39.81, 91, 1.00 | 39.81, 91, 1.00 |
g=12; 2d ; ni=1 | 866.15, 3367, 1.00 | 40.21, 95, 1.00 | 40.21, 95, 1.00 |
g=12; 2d ; ni=2 | 844.80, 3937, 1.00 | 44.39, 115, 1.00 | 44.39, 115, 1.00 |
g=12; rand+2d ; ni=0 | 11.11, 26, 1.00 | 8.67, 94, 1.00 | 2.81, 4, 1.00 |
g=12; rand+2d ; ni=1 | 11.11, 26, 1.00 | 10.28, 108, 1.00 | 2.84, 4, 1.00 |
g=12; rand+2d ; ni=2 | 11.21, 24, 1.00 | 8.92, 42, 1.00 | 2.79, 4, 1.00 |
g=13; rand ; ni=0 | 16.69, 43, 1.00 | 15.19, 119, 1.00 | 3.05, 4, 1.00 |
g=13; rand ; ni=1 | 16.59, 38, 1.00 | 15.89, 145, 1.00 | 2.99, 4, 1.00 |
g=13; rand ; ni=2 | 16.46, 106, 1.00 | 16.47, 92, 1.00 | 3.10, 4, 1.00 |
g=13; 2d ; ni=0 | 3580.53, 34424, 1.00 | 63.29, 145, 1.00 | 63.29, 145, 1.00 |
g=13; 2d ; ni=1 |************************| 60.27, 137, 1.00 | 60.27, 137, 1.00 |
g=13; 2d ; ni=2 |************************| 54.35, 126, 1.00 | 54.35, 126, 1.00 |
g=13; rand+2d ; ni=0 | 14.64, 35, 1.00 | 12.60, 123, 1.00 | 2.96, 4, 1.00 |
g=13; rand+2d ; ni=1 | 14.78, 32, 1.00 | 14.51, 81, 1.00 | 2.91, 3, 1.00 |
g=13; rand+2d ; ni=2 | 14.69, 33, 1.00 | 12.94, 59, 1.00 | 2.91, 4, 1.00 |
g=14; rand ; ni=0 | 22.19, 187, 1.00 | 16.68, 171, 1.00 | 3.19, 4, 1.00 |
g=14; rand ; ni=1 | 23.04, 215, 1.00 | 18.86, 233, 1.00 | 3.17, 4, 1.00 |
g=14; rand ; ni=2 | 23.23, 497, 1.00 | 12.71, 162, 1.00 | 3.18, 4, 1.00 |
g=14; 2d ; ni=0 |************************| 99.64, 173, 1.00 | 99.64, 173, 1.00 |
g=14; 2d ; ni=1 |************************| 88.99, 213, 1.00 | 88.99, 213, 1.00 |
g=14; 2d ; ni=2 |************************| 84.10, 175, 1.00 | 84.10, 175, 1.00 |
g=14; rand+2d ; ni=0 | 20.40, 176, 1.00 | 23.61, 121, 1.00 | 3.08, 4, 1.00 |
g=14; rand+2d ; ni=1 | 20.07, 307, 1.00 | 19.42, 170, 1.00 | 3.09, 4, 1.00 |
g=14; rand+2d ; ni=2 | 20.64, 530, 1.00 | 18.85, 127, 1.00 | 3.03, 4, 1.00 |
g=15; rand ; ni=0 | 35.67, 1292, 1.00 | 22.96, 157, 1.00 | 4.14, 16, 1.00 |
g=15; rand ; ni=1 | 34.16, 949, 1.00 | 23.83, 239, 1.00 | 4.12, 17, 1.00 |
g=15; rand ; ni=2 | 35.81, 1940, 1.00 | 33.00, 381, 1.00 | 3.98, 16, 1.00 |
g=15; 2d ; ni=0 |************************| 122.06, 263, 1.00 | 122.06, 263, 1.00 |
g=15; 2d ; ni=1 |************************| 111.60, 278, 1.00 | 111.60, 278, 1.00 |
g=15; 2d ; ni=2 |************************| 114.55, 292, 1.00 | 114.55, 292, 1.00 |
g=15; rand+2d ; ni=0 | 41.51, 7527, 1.00 | 32.80, 270, 1.00 | 3.30, 4, 1.00 |
g=15; rand+2d ; ni=1 | 45.24, 7244, 1.00 | 31.98, 210, 1.00 | 3.26, 4, 1.00 |
g=15; rand+2d ; ni=2 | 38.32, 6505, 1.00 | 24.21, 190, 1.00 | 3.32, 4, 1.00 |
g=16; rand ; ni=0 | 57.93, 1565, 1.00 | 62.13, 619, 1.00 | 5.44, 21, 1.00 |
g=16; rand ; ni=1 | 69.82, 2856, 1.00 | 67.00, 507, 1.00 | 5.72, 40, 1.00 |
g=16; rand ; ni=2 | 58.75, 1675, 1.00 | 50.52, 255, 1.00 | 5.73, 29, 1.00 |
g=16; 2d ; ni=0 |************************| 179.36, 407, 1.00 | 179.36, 407, 1.00 |
g=16; 2d ; ni=1 |************************| 146.36, 410, 1.00 | 146.36, 410, 1.00 |
g=16; 2d ; ni=2 |************************| 159.79, 357, 1.00 | 159.79, 357, 1.00 |
g=16; rand+2d ; ni=0 | 215.08, 42298, 1.00 | 48.92, 294, 1.00 | 4.61, 26, 1.00 |
g=16; rand+2d ; ni=1 |************************| 46.13, 357, 1.00 | 4.79, 21, 1.00 |
g=16; rand+2d ; ni=2 |************************| 52.57, 421, 1.00 | 4.58, 15, 1.00 |
</pre></div>
<h2>Running Landmarks routing on large networks</h2>
<p>We could not run the previous experiment for very large networks, because Chord
virtual <span class="caps">DHT</span> routing is hard to simulate for large networks. Therefore we
created another experiment that only checks landmarks routing for nei^2 and nei^3.</p>
<p>In addition, we use here a technique of tie breaking with random distances:
Instead of having a plain distance <span class="math">\(1\)</span> between every two neighbors, we pick a
random number between 1000 and 2000. This distorts the geometry of the graph a
little bit, but it surprisingly gives better results in our routing
experiments. This was not done at the previous experiment.</p>
<p>You can run the following experiment by running:</p>
<div class="highlight"><pre><span></span>git clone https://github.com/realcr/freedomlayer_code
cd landmarks_lookahead/net_coords
cargo run --bin landmarks_weighted_matrix --release
</pre></div>
<div class="highlight"><pre><span></span> Network | landmarks nei^2 | landmarks nei^3
---------------------+------------------------+------------------------+
g= 6; rand ; ni=0 | 1.99, 3, 1.00 | 1.99, 3, 1.00 |
g= 6; rand ; ni=1 | 1.91, 3, 1.00 | 1.91, 3, 1.00 |
g= 6; rand ; ni=2 | 2.02, 3, 1.00 | 2.02, 3, 1.00 |
g= 6; 2d ; ni=0 | 5.51, 13, 1.00 | 5.51, 13, 1.00 |
g= 6; 2d ; ni=1 | 5.17, 10, 1.00 | 5.17, 10, 1.00 |
g= 6; 2d ; ni=2 | 5.34, 13, 1.00 | 5.34, 13, 1.00 |
g= 6; rand+2d ; ni=0 | 1.83, 3, 1.00 | 1.83, 3, 1.00 |
g= 6; rand+2d ; ni=1 | 1.89, 3, 1.00 | 1.89, 3, 1.00 |
g= 6; rand+2d ; ni=2 | 1.82, 3, 1.00 | 1.82, 3, 1.00 |
g= 7; rand ; ni=0 | 2.15, 3, 1.00 | 2.15, 3, 1.00 |
g= 7; rand ; ni=1 | 2.07, 3, 1.00 | 2.07, 3, 1.00 |
g= 7; rand ; ni=2 | 2.13, 3, 1.00 | 2.13, 3, 1.00 |
g= 7; 2d ; ni=0 | 6.91, 15, 1.00 | 6.91, 15, 1.00 |
g= 7; 2d ; ni=1 | 6.70, 16, 1.00 | 6.70, 16, 1.00 |
g= 7; 2d ; ni=2 | 7.62, 17, 1.00 | 7.62, 17, 1.00 |
g= 7; rand+2d ; ni=0 | 2.04, 3, 1.00 | 2.04, 3, 1.00 |
g= 7; rand+2d ; ni=1 | 1.91, 3, 1.00 | 1.91, 3, 1.00 |
g= 7; rand+2d ; ni=2 | 1.92, 3, 1.00 | 1.92, 3, 1.00 |
g= 8; rand ; ni=0 | 2.40, 3, 1.00 | 2.40, 3, 1.00 |
g= 8; rand ; ni=1 | 2.33, 3, 1.00 | 2.33, 3, 1.00 |
g= 8; rand ; ni=2 | 2.30, 3, 1.00 | 2.30, 3, 1.00 |
g= 8; 2d ; ni=0 | 9.75, 23, 1.00 | 9.75, 23, 1.00 |
g= 8; 2d ; ni=1 | 11.60, 27, 1.00 | 11.60, 27, 1.00 |
g= 8; 2d ; ni=2 | 10.11, 23, 1.00 | 10.11, 23, 1.00 |
g= 8; rand+2d ; ni=0 | 2.13, 3, 1.00 | 2.13, 3, 1.00 |
g= 8; rand+2d ; ni=1 | 2.08, 3, 1.00 | 2.08, 3, 1.00 |
g= 8; rand+2d ; ni=2 | 2.17, 3, 1.00 | 2.17, 3, 1.00 |
g= 9; rand ; ni=0 | 2.68, 4, 1.00 | 2.54, 3, 1.00 |
g= 9; rand ; ni=1 | 2.65, 4, 1.00 | 2.55, 3, 1.00 |
g= 9; rand ; ni=2 | 2.72, 4, 1.00 | 2.57, 4, 1.00 |
g= 9; 2d ; ni=0 | 14.74, 37, 1.00 | 14.72, 37, 1.00 |
g= 9; 2d ; ni=1 | 15.30, 32, 1.00 | 15.30, 32, 1.00 |
g= 9; 2d ; ni=2 | 13.41, 40, 1.00 | 13.41, 40, 1.00 |
g= 9; rand+2d ; ni=0 | 2.33, 3, 1.00 | 2.33, 3, 1.00 |
g= 9; rand+2d ; ni=1 | 2.25, 3, 1.00 | 2.25, 3, 1.00 |
g= 9; rand+2d ; ni=2 | 2.40, 3, 1.00 | 2.40, 3, 1.00 |
g=10; rand ; ni=0 | 2.76, 4, 1.00 | 2.62, 4, 1.00 |
g=10; rand ; ni=1 | 2.75, 4, 1.00 | 2.65, 3, 1.00 |
g=10; rand ; ni=2 | 2.78, 4, 1.00 | 2.69, 3, 1.00 |
g=10; 2d ; ni=0 | 20.88, 55, 1.00 | 20.88, 55, 1.00 |
g=10; 2d ; ni=1 | 19.62, 49, 1.00 | 19.58, 49, 1.00 |
g=10; 2d ; ni=2 | 22.89, 52, 1.00 | 22.75, 52, 1.00 |
g=10; rand+2d ; ni=0 | 2.71, 4, 1.00 | 2.61, 3, 1.00 |
g=10; rand+2d ; ni=1 | 2.73, 4, 1.00 | 2.58, 3, 1.00 |
g=10; rand+2d ; ni=2 | 2.62, 4, 1.00 | 2.52, 3, 1.00 |
g=11; rand ; ni=0 | 2.98, 5, 1.00 | 2.87, 4, 1.00 |
g=11; rand ; ni=1 | 2.87, 6, 1.00 | 2.75, 4, 1.00 |
g=11; rand ; ni=2 | 2.92, 5, 1.00 | 2.76, 3, 1.00 |
g=11; 2d ; ni=0 | 31.29, 71, 1.00 | 31.25, 71, 1.00 |
g=11; 2d ; ni=1 | 28.14, 64, 1.00 | 28.14, 64, 1.00 |
g=11; 2d ; ni=2 | 29.21, 75, 1.00 | 29.19, 75, 1.00 |
g=11; rand+2d ; ni=0 | 2.83, 6, 1.00 | 2.70, 3, 1.00 |
g=11; rand+2d ; ni=1 | 2.73, 4, 1.00 | 2.57, 3, 1.00 |
g=11; rand+2d ; ni=2 | 2.83, 4, 1.00 | 2.74, 3, 1.00 |
g=12; rand ; ni=0 | 3.21, 7, 1.00 | 2.93, 4, 1.00 |
g=12; rand ; ni=1 | 3.05, 9, 1.00 | 2.84, 4, 1.00 |
g=12; rand ; ni=2 | 3.10, 7, 1.00 | 2.94, 4, 1.00 |
g=12; 2d ; ni=0 | 40.93, 153, 1.00 | 39.89, 91, 1.00 |
g=12; 2d ; ni=1 | 40.23, 95, 1.00 | 40.21, 95, 1.00 |
g=12; 2d ; ni=2 | 44.49, 115, 1.00 | 44.39, 115, 1.00 |
g=12; rand+2d ; ni=0 | 2.98, 5, 1.00 | 2.81, 4, 1.00 |
g=12; rand+2d ; ni=1 | 2.92, 5, 1.00 | 2.84, 4, 1.00 |
g=12; rand+2d ; ni=2 | 2.94, 5, 1.00 | 2.79, 4, 1.00 |
g=13; rand ; ni=0 | 3.68, 12, 1.00 | 3.05, 4, 1.00 |
g=13; rand ; ni=1 | 3.30, 8, 1.00 | 2.99, 4, 1.00 |
g=13; rand ; ni=2 | 3.51, 10, 1.00 | 3.10, 4, 1.00 |
g=13; 2d ; ni=0 | 63.25, 145, 0.99 | 63.47, 145, 1.00 |
g=13; 2d ; ni=1 | 60.47, 137, 1.00 | 60.27, 137, 1.00 |
g=13; 2d ; ni=2 | 54.45, 126, 1.00 | 54.41, 126, 1.00 |
g=13; rand+2d ; ni=0 | 3.14, 7, 1.00 | 2.96, 4, 1.00 |
g=13; rand+2d ; ni=1 | 3.08, 6, 1.00 | 2.91, 3, 1.00 |
g=13; rand+2d ; ni=2 | 3.07, 5, 1.00 | 2.91, 4, 1.00 |
g=14; rand ; ni=0 | 4.26, 27, 1.00 | 3.19, 4, 1.00 |
g=14; rand ; ni=1 | 3.87, 15, 1.00 | 3.17, 4, 1.00 |
g=14; rand ; ni=2 | 4.07, 22, 1.00 | 3.18, 4, 1.00 |
g=14; 2d ; ni=0 | 99.51, 173, 0.99 | 99.68, 173, 1.00 |
g=14; 2d ; ni=1 | 89.35, 213, 1.00 | 89.03, 213, 1.00 |
g=14; 2d ; ni=2 | 84.24, 175, 1.00 | 84.14, 175, 1.00 |
g=14; rand+2d ; ni=0 | 3.83, 23, 1.00 | 3.08, 4, 1.00 |
g=14; rand+2d ; ni=1 | 3.52, 8, 1.00 | 3.09, 4, 1.00 |
g=14; rand+2d ; ni=2 | 3.61, 17, 1.00 | 3.03, 4, 1.00 |
g=15; rand ; ni=0 | 6.31, 51, 1.00 | 3.52, 5, 1.00 |
g=15; rand ; ni=1 | 5.56, 23, 1.00 | 3.48, 5, 1.00 |
g=15; rand ; ni=2 | 6.04, 31, 1.00 | 3.51, 5, 1.00 |
g=15; 2d ; ni=0 | 122.30, 263, 1.00 | 122.08, 263, 1.00 |
g=15; 2d ; ni=1 | 112.18, 278, 1.00 | 111.70, 278, 1.00 |
g=15; 2d ; ni=2 | 114.91, 292, 1.00 | 114.61, 292, 1.00 |
g=15; rand+2d ; ni=0 | 4.42, 22, 1.00 | 3.30, 4, 1.00 |
g=15; rand+2d ; ni=1 | 5.52, 67, 1.00 | 3.26, 4, 1.00 |
g=15; rand+2d ; ni=2 | 4.63, 23, 1.00 | 3.32, 4, 1.00 |
g=16; rand ; ni=0 | 7.40, 37, 1.00 | 3.67, 5, 1.00 |
g=16; rand ; ni=1 | 8.84, 87, 1.00 | 3.59, 5, 1.00 |
g=16; rand ; ni=2 | 8.21, 40, 1.00 | 3.75, 5, 1.00 |
g=16; 2d ; ni=0 | 178.18, 407, 0.99 | 180.36, 462, 1.00 |
g=16; 2d ; ni=1 | 146.28, 410, 0.99 | 147.32, 410, 1.00 |
g=16; 2d ; ni=2 | 159.23, 355, 0.99 | 159.85, 355, 1.00 |
g=16; rand+2d ; ni=0 | 7.10, 33, 1.00 | 3.70, 5, 1.00 |
g=16; rand+2d ; ni=1 | 6.31, 27, 1.00 | 3.54, 5, 1.00 |
g=16; rand+2d ; ni=2 | 7.04, 59, 1.00 | 3.59, 5, 1.00 |
g=17; rand ; ni=0 | 12.07, 61, 1.00 | 3.81, 5, 1.00 |
g=17; rand ; ni=1 | 11.60, 59, 1.00 | 3.74, 5, 1.00 |
g=17; rand ; ni=2 | 15.41, 99, 1.00 | 3.85, 7, 1.00 |
g=17; 2d ; ni=0 | 234.92, 653, 1.00 | 234.46, 653, 1.00 |
g=17; 2d ; ni=1 | 255.01, 522, 1.00 | 254.77, 522, 1.00 |
g=17; 2d ; ni=2 | 243.38, 568, 1.00 | 242.72, 568, 1.00 |
g=17; rand+2d ; ni=0 | 9.30, 77, 1.00 | 3.70, 5, 1.00 |
g=17; rand+2d ; ni=1 | 8.61, 40, 1.00 | 3.79, 5, 1.00 |
g=17; rand+2d ; ni=2 | 9.41, 79, 1.00 | 3.73, 5, 1.00 |
g=18; rand ; ni=0 | 23.53, 155, 1.00 | 3.96, 7, 1.00 |
g=18; rand ; ni=1 | 21.44, 117, 1.00 | 3.93, 5, 1.00 |
g=18; rand ; ni=2 | 23.41, 222, 1.00 | 3.86, 5, 1.00 |
g=18; 2d ; ni=0 | 364.25, 835, 1.00 | 363.13, 835, 1.00 |
g=18; 2d ; ni=1 | 319.11, 794, 1.00 | 318.03, 792, 1.00 |
g=18; 2d ; ni=2 | 335.69, 781, 1.00 | 334.95, 781, 1.00 |
g=18; rand+2d ; ni=0 | 18.00, 103, 1.00 | 3.85, 7, 1.00 |
g=18; rand+2d ; ni=1 | 15.77, 146, 1.00 | 3.92, 5, 1.00 |
g=18; rand+2d ; ni=2 | 18.08, 141, 1.00 | 3.89, 7, 1.00 |
g=19; rand ; ni=0 | 40.79, 219, 1.00 | 4.16, 7, 1.00 |
g=19; rand ; ni=1 | 41.68, 270, 1.00 | 4.24, 8, 1.00 |
g=19; rand ; ni=2 | 43.11, 229, 1.00 | 4.36, 11, 1.00 |
g=19; 2d ; ni=0 | 492.48, 1141, 1.00 | 492.08, 1141, 1.00 |
g=19; 2d ; ni=1 | 477.74, 1053, 1.00 | 476.40, 1053, 1.00 |
g=19; 2d ; ni=2 | 503.55, 1258, 0.98 | 501.88, 1258, 0.98 |
g=19; rand+2d ; ni=0 | 30.63, 193, 1.00 | 3.97, 7, 1.00 |
g=19; rand+2d ; ni=1 | 31.11, 280, 1.00 | 4.11, 7, 1.00 |
g=19; rand+2d ; ni=2 | 38.90, 287, 1.00 | 3.97, 7, 1.00 |
g=20; rand ; ni=0 | 71.98, 479, 1.00 | 4.96, 11, 1.00 |
g=20; rand ; ni=1 | 67.75, 566, 1.00 | 4.68, 10, 1.00 |
g=20; rand ; ni=2 | 67.08, 697, 1.00 | 4.36, 10, 1.00 |
g=20; 2d ; ni=0 | 653.86, 1626, 1.00 | 652.96, 1626, 1.00 |
g=20; 2d ; ni=1 | 762.23, 1482, 1.00 | 760.85, 1482, 1.00 |
g=20; 2d ; ni=2 | 637.62, 1608, 1.00 | 636.42, 1608, 1.00 |
g=20; rand+2d ; ni=0 | 58.23, 617, 1.00 | 4.24, 10, 1.00 |
g=20; rand+2d ; ni=1 | 53.61, 341, 1.00 | 4.22, 8, 1.00 |
g=20; rand+2d ; ni=2 | 56.92, 356, 1.00 | 4.58, 10, 1.00 |
</pre></div>
<script type="text/javascript">if (!document.getElementById('mathjaxscript_pelican_#%@#$@#')) {
var align = "center",
indent = "0em",
linebreak = "false";
if (false) {
align = (screen.width < 768) ? "left" : align;
indent = (screen.width < 768) ? "0em" : indent;
linebreak = (screen.width < 768) ? 'true' : linebreak;
}
var mathjaxscript = document.createElement('script');
mathjaxscript.id = 'mathjaxscript_pelican_#%@#$@#';
mathjaxscript.type = 'text/javascript';
mathjaxscript.src = 'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.0/MathJax.js?config=TeX-AMS-MML_HTMLorMML';
mathjaxscript[(window.opera ? "innerHTML" : "text")] =
"MathJax.Hub.Config({" +
" config: ['MMLorHTML.js']," +
" TeX: { extensions: ['AMSmath.js','AMSsymbols.js','noErrors.js','noUndefined.js'], equationNumbers: { autoNumber: 'AMS' } }," +
" jax: ['input/TeX','input/MathML','output/HTML-CSS']," +
" extensions: ['tex2jax.js','mml2jax.js','MathMenu.js','MathZoom.js']," +
" displayAlign: '"+ align +"'," +
" displayIndent: '"+ indent +"'," +
" showMathMenu: true," +
" messageStyle: 'normal'," +
" tex2jax: { " +
" inlineMath: [ ['\\\\(','\\\\)'] ], " +
" displayMath: [ ['$$','$$'] ]," +
" processEscapes: true," +
" preview: 'TeX'," +
" }, " +
" 'HTML-CSS': { " +
" styles: { '.MathJax_Display, .MathJax .mo, .MathJax .mi, .MathJax .mn': {color: 'inherit ! important'} }," +
" linebreaks: { automatic: "+ linebreak +", width: '90% container' }," +
" }, " +
"}); " +
"if ('default' !== 'default') {" +
"MathJax.Hub.Register.StartupHook('HTML-CSS Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax['HTML-CSS'].FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"MathJax.Hub.Register.StartupHook('SVG Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax.SVG.FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"}";
(document.body || document.getElementsByTagName('head')[0]).appendChild(mathjaxscript);
}
</script>The Trusted Supernode and Distributed Banking2015-12-08T12:43:00+00:002015-12-08T12:43:00+00:00realtag:www.freedomlayer.org,2015-12-08:/the-trusted-supernode-and-distributed-banking.html<h2>Abstract</h2>
<p>The Trusted Supernode is an abstract idea for a distributed secure and
efficient banking system. This system allows payment operations that disturb
only small amount of participants. It overcomes adversarial attacks by applying
a useful proof of work, combined with node mixing.</p>
<p>The Trusted Supernode bank system relies at …</p><h2>Abstract</h2>
<p>The Trusted Supernode is an abstract idea for a distributed secure and
efficient banking system. This system allows payment operations that disturb
only small amount of participants. It overcomes adversarial attacks by applying
a useful proof of work, combined with node mixing.</p>
<p>The Trusted Supernode bank system relies at its core on a special form of
trusted entity called the supernode. In addition to its ability to manage
payments, the supernode should allow to securely exchange computation and
storage services for money.</p>
<h2>Motivation</h2>
<p>Assume a <a href="https://en.wikipedia.org/wiki/Mesh_networking">mesh network</a> of
computers (nodes), working together to somehow navigate messages efficiently
across the network.</p>
<p>Life is all good, until at some point one computer (Let’s call him <span class="math">\(x\)</span>) starts
sending a lot of messages. He sends so many messages, that other computers have
problem sending and receiving messages. <span class="math">\(x\)</span><span class="quo">‘</span>s use of the network makes it hard
for other users to use the network correctly.</p>
<p><span class="math">\(x\)</span> is Bob’s computer. Bob is a cool guy, but is also a pretty heavy network
user. He transfers large video files, and he also has a digital business that
relies on the networking services of the network.</p>
<p>If this was the normal Internet, at this point Bob will get a call from his
<a href="https://en.wikipedia.org/wiki/Internet_service_provider"><span class="caps">ISP</span></a> representative,
telling him either to stop what he is doing with the internet, or maybe upgrade
his internet package to a business class client.</p>
<p>But Bob’s computer, <span class="math">\(x\)</span>, is part of a mesh network. Mesh networks are not
managed by any central authority, hence there is no <span class="caps">ISP</span> to supervise Bob’s
network usage.</p>
<p>It would be nice if the network itself could somehow supervise the network
usage of all the computers. If any computer uses too much bandwidth, he will
have to <strong>pay something of value</strong> to the other computers in the network. This
way, every user of the network could use the network as much as he wants, and
pay according to his use.</p>
<p>If we want to use payments in our network, we need to maintain some kind of a
bank: A distributed bank over the mesh network. In this document we discuss an
idea for creating such a bank. </p>
<p>This document will not really consider the structure of the mesh network.
Instead, we will assume some kind of underlying communication channel, and
explain how to create a distributed bank on top of it.</p>
<h2>Requirements for Distributed Banking</h2>
<p>Let’s be more specific about what we want to achieve with our bank. For every
user <span class="math">\(x\)</span>, a balance value <span class="math">\(v_x\)</span> is stored. <span class="math">\(v_x\)</span> is the amount of money that
<span class="math">\(x\)</span> owns.</p>
<p>Our bank should be able to perform the following operations <strong>efficiently</strong>:</p>
<ol>
<li>
<p>Money Transfer: A user <span class="math">\(x\)</span> can send user <span class="math">\(y\)</span> amount money <span class="math">\(u\)</span> if <span class="math">\(u \leq
v_x\)</span>. After the money transfer, <span class="math">\(x\)</span><span class="quo">‘</span>s balance value will be decreased by
<span class="math">\(u\)</span>, and <span class="math">\(y\)</span><span class="quo">‘</span>s balance value will increase by <span class="math">\(u\)</span>.</p>
</li>
<li>
<p>Inquiry: A user <span class="math">\(x\)</span> can get his current balance <span class="math">\(v_x\)</span>.</p>
</li>
</ol>
<p>Assume that there are <span class="math">\(n\)</span> nodes on the mesh network. Also assume that the
longest path between two nodes in the network is diam(G). (The
<a href="https://en.wikipedia.org/wiki/Distance_%28graph_theory%29">diameter</a> of the
network G). Whenever we write “efficiently” in this document, we mean that every
operation will disturb at most <span class="math">\(diam(G)\cdot polylog(n)\)</span> nodes of the network.</p>
<p>We also want our bank to be somewhat decentralized. We don’t have a formal
definition for that, but generally it means that we don’t want it to be managed
by a selected few nodes. We want it to be resilient to large amounts of
misbehaving nodes.</p>
<h2>Routing and Distributed Banking</h2>
<p>In mesh networks there is a special relation between the problem of message
Routing and Distributed Banking.</p>
<p>Recall that at this document we seek a solution for Distributed Banking in
order to make our message routing algorithm fair. If user <span class="math">\(x\)</span> makes a heavier
use of the network than user <span class="math">\(y\)</span>, then <span class="math">\(x\)</span> should pay more than user <span class="math">\(y\)</span>.</p>
<p>It is interesting to observe the opposite direction of this relation: <strong>If all
that we have is a solution for Distributed Banking, we can use it to implement
efficient routing.</strong></p>
<p>We roughly describe here this idea:
Assume that we have a mesh network with Distributed Banking, and let <span class="math">\(x\)</span> and
<span class="math">\(y\)</span> be two nodes. Also assume that <span class="math">\(v_x \geq 2\)</span>. We want to efficiently route a
message from <span class="math">\(x\)</span> to <span class="math">\(y\)</span>. For the sake of simplicity, assume that we want to
route a single bit <span class="math">\(b \in \{0,1\}\)</span> from <span class="math">\(x\)</span> to <span class="math">\(y\)</span>.</p>
<p>First, <span class="math">\(y\)</span> performs Inquiry operation, to obtain his current balance. Next, <span class="math">\(x\)</span>
will transfer to <span class="math">\(y\)</span><span class="quo">‘</span>s <span class="math">\(b+1\)</span> money. <span class="math">\(y\)</span> will perform another Inquiry. If his
balance has increased by <span class="math">\(1\)</span>, it means that <span class="math">\(x\)</span> has sent the bit <span class="math">\(0\)</span>. If his
balance has increased by <span class="math">\(2\)</span>, it means that <span class="math">\(x\)</span> has sent the bit <span class="math">\(1\)</span>.</p>
<p>We assumed that all the operations in the distributed bank are efficient.
Therefore the bit was sent from <span class="math">\(x\)</span> to <span class="math">\(y\)</span> efficiently.</p>
<h2>About using the Bitcoin network</h2>
<p>What about using the <a href="https://en.wikipedia.org/wiki/Bitcoin">Bitcoin</a> network
as a distributed bank for our mesh network?</p>
<p>In the Bitcoin network every money transfer between two nodes must be
broadcasted to all the nodes in the network. In other words, the Bitcoin
network does not allow to perform efficient money transfer.</p>
<p>Recall that our original motivation to implement a distributed bank is to be
able to route messages in a fair and efficient manner along the network. If our
distributed bank does not allow efficient money transfer, we can not rely on it
to implement efficient routing.</p>
<h2>Initial Attempts</h2>
<p>Consider a world of <span class="math">\(n\)</span> people. How could we create an economy in this world,
where every person can pay another person, and payments have some meaningful value?</p>
<p>A common method to achieve such an economy would be to use cash: Bills and
coins. Those are artifacts that are hard to produce. A limited amount of bills
are printed, and people can exchange those bills as payment.</p>
<p>In the digital world of computers, we don’t have an exact equivalent for cash.
This happens because digital objects can be duplicated easily. This problem is
sometimes referred to as <a href="https://en.wikipedia.org/wiki/Double-spending">Double
Spending</a>:
Assume that a user <span class="math">\(x\)</span> owns a digital bill. He could duplicate the digital bill
and pay the same bill simultaneously both to <span class="math">\(y\)</span> and <span class="math">\(z\)</span>.</p>
<p>Instead of using Cash, we could use some concept of a bank instead. A balance
will be kept for every person. People can pay each other by updating their
balances accordingly (Increasing the destination balance, and decreasing the
source balance).</p>
<p>Now we have the problem of where and how to keep the balances.</p>
<p>A naive solution would be to let every person <span class="math">\(x\)</span> keep his own balance value,
<span class="math">\(v_x\)</span>. This solution is also distributed: Every person has to remember just one
value. A transaction between people <span class="math">\(x\)</span> and <span class="math">\(y\)</span> only involves <span class="math">\(x\)</span> and <span class="math">\(y\)</span>. If
<span class="math">\(x\)</span> wants to transfer <span class="math">\(k\)</span> money to <span class="math">\(y\)</span>, <span class="math">\(x\)</span> decreases his own balance by <span class="math">\(k\)</span>,
and <span class="math">\(y\)</span> increases his own balance by <span class="math">\(k\)</span>.</p>
<p>The problem with this model is obvious: What if <span class="math">\(x\)</span> becomes greedy, and decides
to increase his own balance? Our economic system will be compromised.</p>
<p>At this point we understand that we can not trust all people to be honest.
Another solution would be to let just one special person <span class="math">\(z\)</span>, to keep the
balances for all the nodes in the network.</p>
<p>Whenever <span class="math">\(x\)</span> wants to send <span class="math">\(k\)</span> money to <span class="math">\(y\)</span>, <span class="math">\(x\)</span> will file a request
to <span class="math">\(z\)</span>, saying that he wants to send <span class="math">\(k\)</span> money to node <span class="math">\(y\)</span>.</p>
<p><span class="math">\(z\)</span> will inspect <span class="math">\(x\)</span><span class="quo">‘</span>s balance and make sure that <span class="math">\(x\)</span> has at least <span class="math">\(k\)</span> money.
If this is the case, <span class="math">\(z\)</span> will subtract <span class="math">\(k\)</span> from <span class="math">\(x\)</span><span class="quo">‘</span>s balance, and add <span class="math">\(k\)</span> to
<span class="math">\(y\)</span><span class="quo">‘</span>s balance.</p>
<p><img alt="A transaction processed by z" src="https://www.freedomlayer.org/articles/freedomlayer/dist_bank/images/centralized_bank_z.svg"></p>
<p><em>In the picture: A transaction as processed by a centralized bank <span class="math">\(z\)</span>.</em></p>
<p>This solution is somewhat equivalent to the usual centralized bank. (The banks
that we have in the real world).</p>
<p>The weak point of the centralized bank is with <span class="math">\(z\)</span>: The person that manages the
bank. He has full control over all the balances of all the participants.</p>
<p>In addition, every money transfer between two people <span class="math">\(x\)</span> and <span class="math">\(y\)</span> must go
through <span class="math">\(z\)</span>. Therefore the economy can be only as efficient as the ability of
<span class="math">\(z\)</span> to process transactions.</p>
<h2>Introducing the trusted subset</h2>
<blockquote>
<p>So the Lord scattered them abroad from there over the face of all the earth,
and they ceased building the city. (Genesis 11:8)</p>
</blockquote>
<p>Let <span class="math">\(S\)</span> be set of <span class="math">\(n\)</span> people. We have noticed that if we want to create an
economy between those people, we have to take into account the fact that some
people might try to cheat, given the opportunity.</p>
<p>To be able to talk formally, we need to have some model of when people might
cheat. We choose the following simplistic model: Every person in <span class="math">\(S\)</span> is either
<strong>Good</strong> or <strong>Bad</strong>. A good person always goes by the rules. All the Bad people
collude together to try to subvert the system.</p>
<p>We are going to assume that there are more good people than bad people in <span class="math">\(S\)</span>.
There is some philosophy behind this assumption and we won’t get to it now. For
now, think about it as giving the good people some kind of advantage. All
other things equal, it is hard to believe that a minority of good people in <span class="math">\(S\)</span>
will be able to hold together a secure distributed bank. </p>
<p>On the opposite side of things, it is reasonable to believe that a set with a
majority of good people is able to make correct decisions somehow with respect
to banking operations: Maybe through some kind of a smart voting system.</p>
<p>Let <span class="math">\(x\)</span> be some person in the network. We want to keep <span class="math">\(x\)</span><span class="quo">‘</span>s balance on the
network, somehow. We have already noted that keeping this balance with just one
person is too centralized and fragile. On the other end of the spectrum,
making all the people in <span class="math">\(S\)</span> remember <span class="math">\(x\)</span><span class="quo">‘</span>s balance at all times is very
resilient, but too inefficient. As a middle ground, we will choose a small
subset of people and let them remember <span class="math">\(x\)</span><span class="quo">‘</span>s balance.</p>
<p>To make sure the members of the small subset do not collude to do something
bad, we pick the small set randomly.</p>
<p>Let <span class="math">\(T \subseteq S\)</span> a random subset of <span class="math">\(S\)</span>. As there are more good people than
bad people in <span class="math">\(S\)</span>, we expect that <span class="math">\(T\)</span> will also contain more good people than
bad people. </p>
<p>Let’s put in some numbers. Assume that the total amount of people is
<span class="math">\(n=2^{r}\)</span>, and that the amount of bad people is <span class="math">\(\alpha\)</span> of all the people.
Next, we choose random sets <span class="math">\(T\)</span> of size <span class="math">\(\log{n} = r\)</span>. What is the probability
<span class="math">\(p_r\)</span> that the amount of bad people in <span class="math">\(T\)</span> is more than <span class="math">\(\beta\)</span> of <span class="math">\(T\)</span>?</p>
<p><span class="math">\(n\)</span> is very large with respect to <span class="math">\(|T|\)</span>, so for the ease of computation we will
assume that our population is infinite (And every person is bad with
probability of <span class="math">\(\alpha\)</span>). This will give us a slightly wrong result, but not
too wrong. Now that the draw of a person from the pool of infinite person is
independent of other draws, we can use the <a href="https://en.wikipedia.org/wiki/Chernoff_bound">Multiplicative form of
Chernoff bound</a> to
estimate <span class="math">\(p_r\)</span>.</p>
<div class="math">$$ Pr[X > (1+\delta)\mu] < {\left(\frac{e^{\delta}}{(1+\delta)^{(1 +
\delta)}}\right)}^\mu $$</div>
<p>In our case <span class="math">\(\mu = r\cdot\alpha\)</span>, which is the expected amount of bad people
in <span class="math">\(T\)</span>. We pick <span class="math">\(\delta = \frac{\beta - \alpha}{\alpha}\)</span> to get:</p>
<div class="math">$$ p_r \approx Pr[X > {\beta}\cdot r] < {\left(\frac{e^{\delta}}{(1+\delta)^{(1 +
\delta)}}\right)}^\mu $$</div>
<p>If one out of 9 people is bad (<span class="math">\(\alpha = 1/9\)</span>), the probability of having a
random subset <span class="math">\(T\)</span> of size <span class="math">\(r=256\)</span> with more than <span class="math">\(\beta=1/3\)</span> bad people is
(Calculation done with python):</p>
<div class="highlight"><pre><span></span><span class="n">In</span> <span class="o">:</span> <span class="n">math</span><span class="o">.</span><span class="na">log2</span><span class="o">(</span><span class="n">calc_bad_prob</span><span class="o">(</span><span class="mi">256</span><span class="o">,</span><span class="mi">1</span><span class="sr">/9,1/</span><span class="mi">3</span><span class="o">))</span>
<span class="n">Out</span><span class="o">:</span> <span class="o">-</span><span class="mf">53.176815513188735</span>
</pre></div>
<p>We got a probability of at most <span class="math">\(2^{-53}\)</span>. This is a very unlikely event. So
with huge probability such a random set <span class="math">\(T\)</span> has less than <span class="math">\(1/3\)</span> fraction of bad people.</p>
<p>This is the code used to obtain this result (Python3):</p>
<div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">math</span>
<span class="k">def</span> <span class="nf">calc_bad_prob</span><span class="p">(</span><span class="n">r</span><span class="p">,</span><span class="n">alpha</span><span class="p">,</span><span class="n">beta</span><span class="p">):</span>
<span class="sd">"""</span>
<span class="sd"> r - Size of a subset.</span>
<span class="sd"> alpha - Probability of a node to be bad.</span>
<span class="sd"> beta - Amount of bad nodes in a subset to turn it into a bad subset.</span>
<span class="sd"> Calculate upper bound for Pr[X > beta*r] using the Chernoff bound.</span>
<span class="sd"> This calculation assumes infinite sampling population.</span>
<span class="sd"> """</span>
<span class="n">delta</span> <span class="o">=</span> <span class="p">(</span><span class="n">beta</span> <span class="o">-</span> <span class="n">alpha</span><span class="p">)</span> <span class="o">/</span> <span class="n">alpha</span>
<span class="n">miu</span> <span class="o">=</span> <span class="n">alpha</span> <span class="o">*</span> <span class="n">r</span> <span class="c1"># miu = E[x]</span>
<span class="n">bound</span> <span class="o">=</span> <span class="p">(</span><span class="n">math</span><span class="o">.</span><span class="n">exp</span><span class="p">(</span><span class="n">delta</span><span class="p">)</span> <span class="o">/</span> <span class="p">((</span><span class="mi">1</span> <span class="o">+</span> <span class="n">delta</span><span class="p">)</span><span class="o">**</span><span class="p">(</span><span class="mi">1</span> <span class="o">+</span> <span class="n">delta</span><span class="p">)))</span> <span class="o">**</span> <span class="n">miu</span>
<span class="k">return</span> <span class="n">bound</span>
</pre></div>
<p>The above choice of the number <span class="math">\(1/3\)</span> was not arbitrary. There is a family of
algorithms called <strong>Asynchronous Byzantine Agreement</strong>. (We will call it <span class="caps">ABA</span> in
this document). An <span class="caps">ABA</span> algorithm can turn a set of <span class="math">\(M\)</span> of people with less than
<span class="math">\(1/3\)</span> bad people into one “virtual good person”.</p>
<p>How does it work? An <span class="caps">ABA</span> algorithm is some kind of a smart vote between the
members of the set. It allows the set of people to make correct decisions with
very high probability, despite the existence of a minority of bad people in the
set. (We will give the details of <span class="caps">ABA</span> in this document, but you can learn
about it from Ran Canetti’s Thesis: Studies in Secure Multiparty Computation and
Applications, chapter 4.)</p>
<p>You might be wondering: Why didn’t we use <span class="caps">ABA</span> from the first place, on the
whole set <span class="math">\(S\)</span>? The reason is efficiency: Making all the people in <span class="math">\(S\)</span> vote over
every money transfer operation is too inefficient. Instead, we pick small
subsets of size about <span class="math">\(\log(n)\)</span>, and run the <span class="caps">ABA</span> algorithm on them.</p>
<p>As a result we get “trusted subsets” of <span class="math">\(S\)</span>.</p>
<h2>Basic banking operations</h2>
<p>Now that we have the concept of trusted subsets, we can use it to store money
balances and perform money transfers. We describe here the general idea behind
the basic banking operations.</p>
<p>For every person <span class="math">\(x\)</span>, we match a random subset <span class="math">\(T_x \subseteq S\)</span>. The members
of <span class="math">\(T_x\)</span> will always keep the current balance of <span class="math">\(x\)</span>.
Note that it is possible for two different people, <span class="math">\(x\)</span> and <span class="math">\(y\)</span>, to have
intersecting random subsets. In other words, it is possible that <span class="math">\(T_x \cap T_y
\neq \emptyset\)</span></p>
<p><strong>Inquiry:</strong> <span class="math">\(x\)</span> can perform an Inquiry operation by sending a request to the
members of <span class="math">\(T_x\)</span>. The members of <span class="math">\(T_x\)</span> work together as one virtual good
person, using the <span class="caps">ABA</span> algorithm. Hence they will send back to <span class="math">\(x\)</span> his current balance.</p>
<p><strong>Money Transfer:</strong> If <span class="math">\(x\)</span> wants to send <span class="math">\(k\)</span> money to another person, <span class="math">\(y\)</span>, <span class="math">\(x\)</span>
can send a request to <span class="math">\(T_x\)</span> to transfer <span class="math">\(k\)</span> money to <span class="math">\(y\)</span>. The members of <span class="math">\(T_x\)</span>
will work together as one virtual good person: First they will check that the
current balance of <span class="math">\(x\)</span> is at least <span class="math">\(k\)</span>. Then <span class="math">\(T_x\)</span> will decrease the balance of
<span class="math">\(x\)</span> by <span class="math">\(k\)</span>. Next, <span class="math">\(T_x\)</span> will send a request to <span class="math">\(T_y\)</span> to increase the balance of
<span class="math">\(y\)</span> by <span class="math">\(k\)</span>. The members of <span class="math">\(T_y\)</span> <em>should be able</em> to trust <span class="math">\(T_x\)</span>, because <span class="math">\(T_x\)</span>
is a randomly chosen subset of <span class="math">\(S\)</span>, and it acts like a virtual good person.
Finally the members of <span class="math">\(T_y\)</span> will add <span class="math">\(k\)</span> to the balance of <span class="math">\(y\)</span>.</p>
<p>Note that we implicitly rely here on some method for supernodes to be able to
find each other. We will discuss it later.</p>
<p>The introduction of trusted subsets of <span class="math">\(S\)</span> creates a world of virtual
participants, where all the participants are good. This setting allows some new
possibilities that we will talk about in the future.</p>
<h2>The lifetime of a Supernode</h2>
<p>To make our discussion more exact, let’s go back to talk about computers.
We are given a set of <span class="math">\(S\)</span> nodes that can somehow communicate efficiently, and
we want to implement a distributed bank. Every node <span class="math">\(x\)</span> is matched with a
random subset <span class="math">\(T_x \subseteq S\)</span> that keeps his balance.</p>
<p>We begin by addressing an important issue with respect to computers: They are
not always online.</p>
<p>Let <span class="math">\(x\)</span> be some node. Assume that some of the nodes in <span class="math">\(T_x\)</span> get offline, or
end their lifetime (It happens to computers too!). Over time it is possible
that no nodes from <span class="math">\(T_x\)</span> will be online. How can we make sure that <span class="math">\(x\)</span><span class="quo">‘</span>s
balance will not be erased?</p>
<p>To solve this issue, <span class="math">\(x\)</span><span class="quo">‘</span>s subset <span class="math">\(T_x\)</span> must be dynamic. It must be able to
change its members over time, with respect to changes in the network. We will
call the dynamic set of nodes that keep <span class="math">\(x\)</span><span class="quo">‘</span>s balance <span class="math">\(sup_x\)</span>, or: <span class="math">\(x\)</span><span class="quo">‘</span>s
<strong>Supernode</strong>. This new name denotes the fact that <span class="math">\(x\)</span><span class="quo">‘</span>s balance is kept by a
dynamic entity, not by a static set.</p>
<p>If some node dies, it should be somehow replaced by a new randomly selected
node. In addition, members of <span class="math">\(sup_x\)</span> should be removed randomly from time to
time, to keep the random structure of <span class="math">\(sup_x\)</span>. (We don’t want bad nodes to stay
forever in <span class="math">\(sup_x\)</span> while good nodes leave). The supernode has the
responsibility of keeping itself alive.</p>
<p>However, The dynamic nature of Supernodes poses some new trust questions.
Consider this example: Let <span class="math">\(x\)</span> and <span class="math">\(y\)</span> be two nodes, and assume that <span class="math">\(x\)</span> wants
to transfer <span class="math">\(k\)</span> money to <span class="math">\(y\)</span>. <span class="math">\(x\)</span> contacts <span class="math">\(sup_x\)</span>, which subtracts <span class="math">\(k\)</span> from
<span class="math">\(x\)</span><span class="quo">‘</span>s balance, and then sends a request to <span class="math">\(sup_y\)</span> to add <span class="math">\(k\)</span> to <span class="math">\(y\)</span><span class="quo">‘</span>s balance.</p>
<p>How can <span class="math">\(sup_y\)</span> know that it can trust <span class="math">\(sup_x\)</span>? Maybe <span class="math">\(sup_x\)</span> is some
artificial set that was invented by <span class="math">\(y\)</span>, because <span class="math">\(y\)</span> wants to increase his balance? </p>
<p>We break this question to smaller parts.
Assume that a node <span class="math">\(z\)</span> once knew that a set <span class="math">\(G\)</span> of nodes is a supernode. After
a while while he encounters a different set <span class="math">\(G'\)</span>, which claims to be the same
supernode as the original set, <span class="math">\(G\)</span>. How can <span class="math">\(z\)</span> verify that <span class="math">\(G'\)</span> is the same
supernode as the original <span class="math">\(G\)</span>?</p>
<p>To “transfer the trust” from <span class="math">\(G\)</span> to <span class="math">\(G'\)</span>, we can use <a href="https://en.wikipedia.org/wiki/Digital_signature">digital
signatures</a>. Assume that each
node can cryptographically sign a block of data. </p>
<p>Let <span class="math">\(G_1\)</span> be the set of nodes of some supernode.
Assume that some node <span class="math">\(z\)</span> knows that the set of nodes <span class="math">\(G_1\)</span> is a supernode.
Now assume that some node has joined <span class="math">\(G\)</span>, or some node has left <span class="math">\(G\)</span>. We mark
the new structure of the supernode to be <span class="math">\(G_2\)</span>. Every node from <span class="math">\(G_1\)</span> signs the
new structure of the <span class="math">\(G_2\)</span>. The set of all those signatures is a proof that the
supernode has changed its structure from <span class="math">\(G_1\)</span> to <span class="math">\(G_2\)</span>.</p>
<p>The supernode keeps changing over time. <span class="math">\(G_2\)</span> changes to <span class="math">\(G_3\)</span> and so on.
Whenever it changes, a new set of signatures is added, as an evidence for the
structure change. The members of the supernode keep the chain of sets of
signatures at all times.</p>
<p>After a while, <span class="math">\(z\)</span> encounters a some set <span class="math">\(G' = G_u\)</span> of nodes. <span class="math">\(G'\)</span> can then
prove that it is the same supernode as the original <span class="math">\(G_1\)</span> by showing <span class="math">\(z\)</span> the
chain of all signatures, leading from the original structure <span class="math">\(G_1\)</span>, all the way
to <span class="math">\(G'\)</span>.</p>
<p><span class="math">\(z\)</span> knows that the original set, <span class="math">\(G_1\)</span>, was a supernode, therefore its members
were somehow sampled randomly from <span class="math">\(S\)</span>. Hence in the set of signatures proving
the change from <span class="math">\(G_1\)</span> to <span class="math">\(G_2\)</span>, there must be a signature of a good node. <span class="math">\(z\)</span>
doesn’t know which one of those signatures was signed by a good node, but he
knows that at least <span class="math">\(2/3\)</span> of those signatures were signed by good nodes.</p>
<p>A good node will not sign over fake structure changes. Therefore <span class="math">\(z\)</span> can be
sure that indeed the new set <span class="math">\(G_2\)</span> is also a supernode. If, for example, the
change was adding a new node, <span class="math">\(z\)</span> can be sure that this new node was sampled
randomly from <span class="math">\(S\)</span>.</p>
<p>In the same way, <span class="math">\(z\)</span> can be sure that the change from <span class="math">\(G_2\)</span> to <span class="math">\(G_3\)</span> is valid,
and so on. Finally, given a valid chain of signature sets, <span class="math">\(z\)</span> can be sure that
<span class="math">\(G'\)</span> is the same supernode as the original one, <span class="math">\(G_1\)</span>.</p>
<h2>An example for sets of signature chains</h2>
<p>The above explanation will not be complete without an example, so let’s create one.</p>
<p>Assume that a supernode <span class="math">\(sup\)</span> is made of the following nodes:</p>
<div class="highlight"><pre><span></span>(A,B,C,D,E)
</pre></div>
<p>Now a new node <span class="math">\(F\)</span> joins the supernode. The new structure of the supernode is
as follows:</p>
<div class="highlight"><pre><span></span>(A,B,C,D,E,F)
</pre></div>
<p>The signatures that prove the change from the old structure to the new
structure are as follows:</p>
<div class="highlight"><pre><span></span>sign_A((A,B,C,D,E,F))
sign_B((A,B,C,D,E,F))
sign_C((A,B,C,D,E,F))
sign_D((A,B,C,D,E,F))
sign_E((A,B,C,D,E,F))
</pre></div>
<p>Now assume that the node <span class="math">\(B\)</span> went offline, and this was recognized by the
supernode. The supernode removes <span class="math">\(B\)</span> from the set of its members. This is the
new structure of the supernode:</p>
<div class="highlight"><pre><span></span>(A,C,D,E,F)
</pre></div>
<p>And these are the signatures that prove the change from the old structure to
the new structure:</p>
<div class="highlight"><pre><span></span>sign_A((A,C,D,E,F))
sign_C((A,C,D,E,F))
sign_D((A,C,D,E,F))
sign_E((A,C,D,E,F))
sign_F((A,C,D,E,F))
</pre></div>
<p>Note that <span class="math">\(B\)</span> has not signed the new structure, because he went offline. The
signatures of the other nodes <span class="math">\(\{A,C,D,E,F\}\)</span> should be enough, as many of them
are good nodes.</p>
<p>If the supernode <span class="math">\(sup\)</span> wants to prove the changes that happened to him since
his initial structure <span class="math">\((A,B,C,D,E)\)</span> to his current structure <span class="math">\((A,C,D,E,F)\)</span>, he
could show the two sets of signatures from above.</p>
<h2>The birth of a Supernode</h2>
<p>The section about “lifetime of a supernode” explains how supernodes change over
time. However, it does not explain how they are initially created.</p>
<p>Assume that <span class="math">\(x\)</span> has some supernode <span class="math">\(sup_x\)</span>. <span class="math">\(y\)</span> can ask <span class="math">\(x\)</span> to create a new
supernode for him. In that case, <span class="math">\(x\)</span> will send a request to <span class="math">\(sup_x\)</span> to create a
new supernode <span class="math">\(sup_y\)</span> for <span class="math">\(y\)</span>.</p>
<p>The supernode <span class="math">\(sup_x\)</span> will then randomly select a subset of <span class="math">\(S\)</span>, and call it
<span class="math">\(sup_y\)</span>. It will also initialize some balance value for <span class="math">\(y\)</span>. All the members of
<span class="math">\(sup_x\)</span> will sign over the structure of the newly created supernode, <span class="math">\(sup_y\)</span>,
and hand those signatures to the members of <span class="math">\(sup_y\)</span>, together with all the
signatures of the history of changes <span class="math">\(sup_x\)</span>.</p>
<p>Let <span class="math">\(z\)</span> be some node. Assume that <span class="math">\(z\)</span> knew in the past that some set <span class="math">\(G\)</span> is
the set of members of the supernode <span class="math">\(sup_x\)</span>. If <span class="math">\(z\)</span> ever encounters a set of
nodes <span class="math">\(P\)</span> that claim to be the supernode <span class="math">\(sup_y\)</span>, <span class="math">\(z\)</span> will be able to verify
this statement by verifying the chain of signature sets all the way from the
supernode structure <span class="math">\(G\)</span> to the creation of <span class="math">\(sup_y\)</span>. </p>
<p><span class="math">\(z\)</span> will have to verify all the changes that happened to <span class="math">\(sup_x\)</span> since the
member set <span class="math">\(G\)</span>, the creation of the supernode <span class="math">\(sup_y\)</span>, and then all the changes
that happened to the supernode <span class="math">\(sup_y\)</span> until its current structure, <span class="math">\(P\)</span>.</p>
<p>It is hard to create a new fake supernode, because the signatures of members of
a real supernode are required, and a real supernode contains a majority of good
nodes, which will not be willing to sign.</p>
<p>We described here a way to create a new supernode, given that we have a
supernode already. We are going to assume that our network have some initial
supernode, and all the supernodes in the network will be his children, in some sense.</p>
<p>Every supernode <span class="math">\(sup\)</span> will keep in memory a chain of set of signatures,
describing everything that happened until its creation. This chain is used as a
proof that <span class="math">\(sup\)</span> is a valid supernode.</p>
<p>You must be worried right now about the size of the chains that every supernode
has to remember, and you are correct: The chains keep growing as time goes by
and the supernodes change. Generally, every supernode should remember a chain
which is of size about <span class="math">\(O(t)\)</span>, where <span class="math">\(t\)</span> is the time that has passed since the
beginning of the network. We will talk about a solution for this problem later.</p>
<h2>The death of a Supernode</h2>
<p>Given that <span class="math">\(x\)</span> has created a some supernode <span class="math">\(sup_x\)</span>, should this supernode stay
alive forever? </p>
<p>Managing a Supernode is not a full time job for its members, but it does take
some resources. The supernode members have to respond to various requests to
the supernode, find new members or remove old members from time to time.</p>
<p>To make sure that supernodes are not created carelessly, it is possible to
deduct some money from the supernode owner on a timely basis.
The supernode <span class="math">\(sup_x\)</span> will decrease <span class="math">\(x\)</span><span class="quo">‘</span>s balance by a small amount every few
minutes, for example. This is easy to implement, because the supernode <span class="math">\(sup_x\)</span>
is the supernode who maintains the <span class="math">\(x\)</span><span class="quo">‘</span>s money balance.</p>
<p>Using this method, the supernode will only keep living if <span class="math">\(x\)</span> has enough money
to “support” it. When <span class="math">\(x\)</span><span class="quo">‘</span>s balance becomes <span class="math">\(0\)</span>, the supernode will disassemble
and die.</p>
<p>Using this method we can also allow one node <span class="math">\(x\)</span> creating more than one
supernode. <span class="math">\(x\)</span> can use each supernode that he creates as a separate wallet, or
even, a separate instance of a virtual trusted computer.</p>
<h2>Introducing Chain Shortening</h2>
<p>It could be nice if every supernode <span class="math">\(sup\)</span> will have to remember just a short
proof for his realness, instead of remembering a full chain of sets of
signatures all the way to the first supernode ever created.</p>
<p>I searched for a magic cryptographic technology that will enable to compress a
chain of signature sets into a small structure, but I never found one. (I did
find something called <a href="https://eprint.iacr.org/2004/215.pdf">Transitive
Signatures</a>, but never managed to use it
to solve this problem)</p>
<p>Instead, we will use some communication. (It’s a general truth of life that
lack of cryptographic technology forces one to use more communication).</p>
<p>We begin by uniquely naming supernodes. Whenever a supernode creates a new
supernode, it gives the new supernode a new random name.</p>
<p>Let <span class="math">\(W\)</span> be a special supernode. It does not belong to any node in the network,
and it lives forever. Every node in the network was created by <span class="math">\(W\)</span> or one of
his descendent supernodes. The network is bootstrapped by creating <span class="math">\(W\)</span>, and
then letting <span class="math">\(W\)</span> create an initial <span class="math">\(sup_x\)</span> supernode for some node <span class="math">\(x\)</span>.</p>
<p>Being a special supernode, <span class="math">\(W\)</span> is the only supernode with the privilege of
broadcasting messages to the whole network. Every time a change happens to the
structure of <span class="math">\(W\)</span>, it broadcasts the new structure, including a proof: A set of
signatures over the new structure.</p>
<p>For every node <span class="math">\(y\)</span> on the network, on receipt of a message from <span class="math">\(W\)</span>,
broadcasts the message to all of his immediate neighbours. Every node in the
network should know the current structure of <span class="math">\(W\)</span>, and also remember the
structure of <span class="math">\(W\)</span> in the last <span class="math">\(C\)</span> changes. </p>
<p><span class="math">\(C\)</span> could be some large number, for example <span class="math">\(C = 50000\)</span>. This should not cost
too much memory for every node in the network, but at the same time allow every
node to remember all the history of changes of <span class="math">\(W\)</span> in the last half year.</p>
<p>Consider some supernode <span class="math">\(sup\)</span> in the network. <span class="math">\(sup\)</span> members maintain a chain of
signature sets that describe and prove the history of changes of <span class="math">\(sup\)</span> since
the beginning of the network. <span class="math">\(sup\)</span> can use the existence of the ever living
supernode <span class="math">\(W\)</span> to shorten this chain.</p>
<p><span class="math">\(sup\)</span> will send <span class="math">\(W\)</span> his current chain. The supernode <span class="math">\(W\)</span> will then inspect the
chain. If <span class="math">\(W\)</span> manages to recognize the beginning of the chain (The beginning of
the chain should be some ancient structure of <span class="math">\(W\)</span>), and all the signatures are
correct, then the members of <span class="math">\(W\)</span> will produce a new set of signatures over the
supernode <span class="math">\(sup\)</span>. </p>
<p>From now on, instead of using the old chain of signatures, <span class="math">\(sup\)</span> can use <span class="math">\(W\)</span><span class="quo">‘</span>s
signature over his current structure as a proof for his validity.</p>
<p>We call the above process <strong>chain shortening.</strong></p>
<p><img alt="Basic Chain Shortening" src="https://www.freedomlayer.org/articles/freedomlayer/dist_bank/images/basic_chain_shortening.svg"></p>
<p><span class="math">\(sup\)</span> should perform the chain shortening process regularly to keep his proof
chain short, and to stay alive. If <span class="math">\(sup\)</span> stops shortening his chain for a long
enough time, no node will be able to recognize him. This amount of time depends
on the choice of constant <span class="math">\(C\)</span>.</p>
<p>How can a node <span class="math">\(z\)</span> verify such a proof chain of a supernode <span class="math">\(sup\)</span>?
Every node <span class="math">\(z\)</span> knows the current and previous structure (Up to <span class="math">\(C\)</span> steps
backwards) of the supernode <span class="math">\(W\)</span>. <span class="math">\(z\)</span> will verify that the proof chain begins
with some known structure of the supernode <span class="math">\(W\)</span>, and that all the signatures up
to the current structure of <span class="math">\(sup\)</span> are valid.</p>
<p>The problem with the described mechanism is the large amount of work <span class="math">\(W\)</span> needs
to do. <span class="math">\(W\)</span> has to deal with chain shortening requests from all the supernodes
in the network. This would be too much for one supernode to deal with.</p>
<h2>Efficient Chain Shortening</h2>
<p>We need to take off some of work of chain shortening from the supernode <span class="math">\(W\)</span>. We
could, for example, let <span class="math">\(W\)</span> do the chain shortening for just a few supernodes.
Then each of those supernodes will do the chain shortening for a few other supernodes.</p>
<p>This kind of hierarchy can leave every supernode with a proof chain of length
at most <span class="math">\(O(log(N))\)</span>, where <span class="math">\(N\)</span> is the amount of supernodes in the network. At
the same time, every supernode has to perform the work of chain shortening only
for a few other supernodes, which is not very expensive (With respect to
computation or communication).</p>
<p>Assuming that we have some kind of efficient communication infrastructure, we
can arrange the supernodes into a <span class="caps">DHT</span> in order to achieve this kind of
hierarchy. We are going to use the <a href="https://www.freedomlayer.org/intro-to-distributed-hash-tables-dhts.html">Chord
<span class="caps">DHT</span></a> here.
Recall that a <span class="caps">DHT</span> is a structure where every participant is directly connected
to a logarithmic amount of other participants. In addition, every participant
can reach another participant using logarithmic amount of hops.</p>
<p>Every supernode is given a unique name on its creation time. This name will be
used as his <span class="caps">DHT</span> Identity. <span class="math">\(W\)</span> will also be on this <span class="caps">DHT</span>. Every supernode in the
<span class="caps">DHT</span> will perform chain shortening for all his connections in the <span class="caps">DHT</span> (If
possible), every constant amount of time.</p>
<p>This means that every supernode <span class="math">\(sup\)</span> on the <span class="caps">DHT</span> receives proposals for proof
chains every constant amount of time. <span class="math">\(sup\)</span> will always maintain the shortest
valid proof chain he was offered.</p>
<p>Hence, Every supernode in the <span class="caps">DHT</span> will have a proof chain which represents his
shortest path to <span class="math">\(W\)</span> on the <span class="caps">DHT</span>. In the chord <span class="caps">DHT</span> the longest distance is
logarithmic. Therefore the shortest proof chain should be of size at most
<span class="math">\(O(\log(N))\)</span>, where <span class="math">\(N\)</span> is the amount of supernodes in the network.</p>
<p>Note that the resulting <span class="caps">DHT</span> will be connected, because of the ability of <span class="math">\(W\)</span> to broadcast.</p>
<p>We can use this same <span class="caps">DHT</span> as a tool for <strong>discovery of supernodes</strong>. A supernode
<span class="math">\(sup_x\)</span> can find another supernode <span class="math">\(sup_y\)</span> by searching for <span class="math">\(sup_y\)</span><span class="quo">‘</span>s unique
name in the <span class="caps">DHT</span>.</p>
<h2>Network Stickiness</h2>
<p>Assume that some node <span class="math">\(x\)</span> connects to the network for the first time. <span class="math">\(x\)</span>
learns about the current structure of <span class="math">\(W\)</span>, and is given a copy of the history
of structure of <span class="math">\(W\)</span> (Of size <span class="math">\(C\)</span> steps backwards).</p>
<p>If <span class="math">\(x\)</span> ever connects again to the network after some not too long amount of
time, <span class="math">\(x\)</span> can show his last known structure of the supernode <span class="math">\(W\)</span> to any other
node <span class="math">\(y\)</span>, and receive back a proof chain from that old structure all the way to
the current structure of <span class="math">\(W\)</span>. This is possible because all the nodes in the
network remember a long enough history of changes to the structure of the
supernode <span class="math">\(W\)</span>.</p>
<p>Using this method, <span class="math">\(x\)</span> can also verify that he has arrived to the same network
like the one he saw in his first visit. <span class="math">\(x\)</span> can not be put into a different
network, and be fooled to believe that it is the original one.</p>
<p>We call this property <strong>Network Stickiness.</strong></p>
<h2>Limiting the amount of bad nodes</h2>
<p>Earlier we have chosen the following model: We have a world <span class="math">\(S\)</span> of <span class="math">\(n\)</span> nodes:
Most of them are good, and the rest are bad.</p>
<p>How can we enforce this kind of ratio between the nodes in the network? What
stops an adversary from inserting many bad nodes in the network? ( This kind of
attack is also known as <a href="https://en.wikipedia.org/wiki/Sybil_attack">Sybil
Attack</a>.)</p>
<p>The solution would be to require nodes joining the network to sacrifice some
rare resource. In some philosophical sense, this allows us to link real value
from the outside world into the network.</p>
<p>A simple idea would be to use <a href="https://en.wikipedia.org/wiki/Proof-of-work_system">Proof of
Work</a>. Every node that
wants to be part of a supernode must perform difficult calculations regularly.
Assuming that no adversary holds majority of computation power, we can assume
that it should be computationally difficult for any adversary to have majority
of bad nodes inside the network.</p>
<p>The main disadvantage of using this raw form of “Proof of Work” is that we
waste resources. </p>
<p>Another option would be to require an indirect proof of work: <strong>Instead of
requiring nodes to perform difficult calculations, we can ask nodes to pay.</strong>
As money in our bank stands for real value, we can use it as a rare resource
that nodes sacrifice.</p>
<p>In this method, every node <span class="math">\(x\)</span> that wants to join a supernode must lost money
regularly. For example: <span class="math">\(x\)</span> has to lose <span class="math">\(k\)</span> money every few minutes. Where
does the money <span class="math">\(x\)</span> pays go? It is just gone. This is equivalent of spreading it
equally between all the nodes of the network, but easier to do.</p>
<h2>The Supernode as a trusted general purpose computer</h2>
<p>Recall that the supernode is a set of nodes, with a majority of good nodes,
that work together like one virtual good node. So far we have talked about the
supernode as means for storing balance for one node, but the supernode can do
much more. We mention here some very abstract ideas of what can be done.</p>
<p>Let <span class="math">\(x\)</span> be some node, and <span class="math">\(sup_x\)</span> be a supernode owned by <span class="math">\(x\)</span>. The supernode
<span class="math">\(sup_x\)</span> can be used as a trusted remote computer owned by <span class="math">\(sup_x\)</span>, supplying
various services in exchange for <span class="math">\(x\)</span><span class="quo">‘</span>s money. Using the Asynchronous Byzantine
Agreement algorithm <span class="math">\(sup_x\)</span> acts a virtual one good node, so <span class="math">\(x\)</span> can trust
<span class="math">\(sup_x\)</span>. In addition, <span class="math">\(sup_x\)</span> has direct access to <span class="math">\(x\)</span><span class="quo">‘</span>s money balance. These
two facts make a perfect setting for <span class="math">\(x\)</span> and <span class="math">\(sup_x\)</span> to do some business.</p>
<p>Also note that <span class="math">\(sup_x\)</span> keeps living as long as <span class="math">\(x\)</span> still has money in its
balance. Therefore <span class="math">\(sup_x\)</span> can keep working even when <span class="math">\(x\)</span> is offline.</p>
<p>Letting the members of <span class="math">\(sup_x\)</span> perform computational or storage services might
not be a good idea, because <span class="math">\(sup_x\)</span> has only so many members, and they might be
limited in their computational or storage abilities.</p>
<p>Another possibility would be to let <span class="math">\(sup_x\)</span> manage workers for <span class="math">\(x\)</span>. The workers
will supply storage or computational services. The supernode <span class="math">\(sup_x\)</span> will
handle the communication. </p>
<p>Assume that <span class="math">\(x\)</span> wants to store some files. <span class="math">\(x\)</span> will first send a request to
<span class="math">\(sup_x\)</span> to allocate storage space. Next, <span class="math">\(sup_x\)</span> will hire some random nodes
from the network (That requested to be hired for storage purposes). <span class="math">\(x\)</span> will
then send blocks of the files he wants to store to <span class="math">\(sup_x\)</span>, and <span class="math">\(sup_x\)</span> will
store those blocks redundantly on the workers it has just hired.</p>
<p>The storage workers will be paid from <span class="math">\(x\)</span><span class="quo">‘</span>s balance, and they will be payed on
regularly (For example: Every few minutes/seconds) for their storage work.</p>
<p>At a later time, <span class="math">\(x\)</span> could request the blocks he stored earlier from <span class="math">\(sup_x\)</span>.
The supernode will then contact the workers and retrieve the stored blocks.
Finally, the stored blocks will be sent back to <span class="math">\(x\)</span>.</p>
<p>A similar idea could work for computation workers: The supernode <span class="math">\(sup_x\)</span> can
hire a few random nodes from the network (That requested to be hired for
computation purposes). Then it can run the same computation on all of those
nodes, and take the majority answer. Computation workers could be paid by the
instruction (Some kind of basic computational unit).</p>
<p><span class="math">\(x\)</span> can also use his supernode <span class="math">\(sup_x\)</span>, as some kind of a server. He can load
it with some code and data, and allow it to run and serve other nodes of the
network. <span class="math">\(sup_x\)</span> might be programmed to charge money for its services to other
nodes of the network. This way, <span class="math">\(sup_x\)</span> can stay alive as long as it keeps
giving value to users of the network, even long time after <span class="math">\(x\)</span> has died.</p>
<h2>Future Research</h2>
<p>The Trusted Supernode model of distributed bank was presented here as pretty
abstract idea. Many pieces are still missing to make this idea a working
system. We list some of them here:</p>
<ol>
<li>
<p>How to balance the economic system? Recall that a node has to lose money
when working inside a supernode. Where does new money come from into the network?</p>
</li>
<li>
<p>Incentives for nodes to work in a supernode: If working as part of a
supernode costs money for a node, why would a node want to do that?
Translating to the language of Bitcoin: If miners don’t make money from
mining, will they mine anyways?</p>
</li>
<li>
<p>Efficiency: The supernode works as one trusted general purpose computer, but
it is pretty slow, because it is spread across the whole network (His
members are chosen randomly from the network). How practical would it be to
run a supernode of ~<span class="math">\(256\)</span> members over the Internet? Are there other ways to
choose supernodes so that their members are not so far away, and still
maintain secure banking?</p>
</li>
<li>
<p>Connecting the Trusted Supernode bank system to the Mesh network: How to pay
for sending messages using the proposed banking system? How to avoid giving
money to nodes in the mesh that don’t route correctly? How can supernodes
communicate efficiently in a mesh network? How to deal with a network split?</p>
</li>
<li>
<p>Creating an efficient Asynchronous Byzantine Agreement algorithm. Some
efficient <span class="caps">ABA</span> algorithms are already known, but it takes some work to apply
those algorithms for our supernode use case.</p>
</li>
</ol>
<h2>Further reading</h2>
<ul>
<li>
<p><a href="https://www.cs.unm.edu/~saia/papers/swarm.pdf">Making Chord Robust to Byzantine
Attacks</a> By Amos Fiat, Jared
Saia and Maxwell Young</p>
</li>
<li>
<p><a href="http://www14.in.tum.de/personen/scheideler/papers/f85-scheideler.ps.gz">How to Spread Adversarial Nodes?
Rotate!</a> By
Christian Scheideler</p>
</li>
<li>
<p><a href="http://www.cs.tau.ac.il/~canetti/materials/thesis.ps">Studies in Secure Multiparty Computation and
Applications</a> By Ran Canetti</p>
</li>
<li>
<p><a href="https://www.zurich.ibm.com/~cca/papers/abba.pdf">Random Oracles in Constantinople: Practical Asynchronous Byzantine Agreement
using Cryptography</a> By
Christian Cachin, Klaus Kursawe, Victor Shoup</p>
</li>
<li>
<p><a href="https://bitcoin.org/bitcoin.pdf">Bitcoin: A Peer-to-Peer Electronic Cash
System</a> By Satoshi Nakamoto</p>
</li>
</ul>
<script type="text/javascript">if (!document.getElementById('mathjaxscript_pelican_#%@#$@#')) {
var align = "center",
indent = "0em",
linebreak = "false";
if (false) {
align = (screen.width < 768) ? "left" : align;
indent = (screen.width < 768) ? "0em" : indent;
linebreak = (screen.width < 768) ? 'true' : linebreak;
}
var mathjaxscript = document.createElement('script');
mathjaxscript.id = 'mathjaxscript_pelican_#%@#$@#';
mathjaxscript.type = 'text/javascript';
mathjaxscript.src = 'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.0/MathJax.js?config=TeX-AMS-MML_HTMLorMML';
mathjaxscript[(window.opera ? "innerHTML" : "text")] =
"MathJax.Hub.Config({" +
" config: ['MMLorHTML.js']," +
" TeX: { extensions: ['AMSmath.js','AMSsymbols.js','noErrors.js','noUndefined.js'], equationNumbers: { autoNumber: 'AMS' } }," +
" jax: ['input/TeX','input/MathML','output/HTML-CSS']," +
" extensions: ['tex2jax.js','mml2jax.js','MathMenu.js','MathZoom.js']," +
" displayAlign: '"+ align +"'," +
" displayIndent: '"+ indent +"'," +
" showMathMenu: true," +
" messageStyle: 'normal'," +
" tex2jax: { " +
" inlineMath: [ ['\\\\(','\\\\)'] ], " +
" displayMath: [ ['$$','$$'] ]," +
" processEscapes: true," +
" preview: 'TeX'," +
" }, " +
" 'HTML-CSS': { " +
" styles: { '.MathJax_Display, .MathJax .mo, .MathJax .mi, .MathJax .mn': {color: 'inherit ! important'} }," +
" linebreaks: { automatic: "+ linebreak +", width: '90% container' }," +
" }, " +
"}); " +
"if ('default' !== 'default') {" +
"MathJax.Hub.Register.StartupHook('HTML-CSS Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax['HTML-CSS'].FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"MathJax.Hub.Register.StartupHook('SVG Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax.SVG.FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"}";
(document.body || document.getElementsByTagName('head')[0]).appendChild(mathjaxscript);
}
</script>Multi Named VDHT Routing2015-10-17T10:49:00+00:002015-10-17T10:49:00+00:00realtag:www.freedomlayer.org,2015-10-17:/multi-named-vdht-routing.html<h2>Abstract</h2>
<p>We are given a connected mesh network of <span class="math">\(n\)</span> nodes, where each node has a unique
name from the range of <span class="math">\([0,2^l)\)</span>. We assume that <span class="math">\(n \leq 2^l\)</span>. Every node is
directly connected to a few other nodes.</p>
<p>We present an iterative distributed algorithm: MIterF, that …</p><h2>Abstract</h2>
<p>We are given a connected mesh network of <span class="math">\(n\)</span> nodes, where each node has a unique
name from the range of <span class="math">\([0,2^l)\)</span>. We assume that <span class="math">\(n \leq 2^l\)</span>. Every node is
directly connected to a few other nodes.</p>
<p>We present an iterative distributed algorithm: MIterF, that allows each node <span class="math">\(x\)</span>
to maintain remote contact with a few key nodes. The remote contact with those
key nodes allows efficient greedy routing of messages between any two nodes: At
each step the message is sent to a remote contact with a name that is closest to
the destination.</p>
<p>Given that the mesh network is connected, We prove that the presented iterative
distributed algorithm MIterF always converges to a solution that allows
efficient greedy routing.</p>
<h2>Previous attempts</h2>
<p>In <a href="https://www.freedomlayer.org/experimenting-with-virtual-dht-routing.html">Experimenting with Virtual <span class="caps">DHT</span>
Routing</a>
we have presented an iterative distributed algorithm that allows each node <span class="math">\(x\)</span>
to maintain contact with a few key nodes that are close to <span class="math">\(x \pm 2^t\)</span>. We used
experiments to show that this algorithm can work in some cases, but we didn’t
provide any rigorous proof for those claims.</p>
<p>In <a href="https://www.freedomlayer.org/about-cycles-in-stationary-states-of-vdht-routing-algorithms.html">About cycles in stationary states of <span class="caps">VDHT</span> routing
algorithms</a>
we presented a slightly different iterative distributed algorithm, IterF, that
also allows each node <span class="math">\(x\)</span> to maintain contact to a few key nodes that are close
to <span class="math">\(x \pm 2^t\)</span>. </p>
<p>We have managed to prove that after IterF converges, we end up with possibly a
few disjoint sets of nodes (OneRound Theorem). In each such set <span class="math">\(S\)</span>, for every
two nodes <span class="math">\(a,b \in S\)</span> it is possible to route messages efficiently between <span class="math">\(a\)</span>
and <span class="math">\(b\)</span>. However, we do not know how to efficiently route messages between nodes
from different disjoint sets.</p>
<p>We present here a modification to the IterF algorithm, MIterF (Multi named
IterF), which is proven to converge to just one set of nodes <span class="math">\(S\)</span>, such that for
every two nodes <span class="math">\(a,b \in S\)</span> it is possible to route messages efficiently between
<span class="math">\(a\)</span> and <span class="math">\(b\)</span>. This improvement comes at the cost of having every node maintain
paths to more nodes.</p>
<h2>Nodes that must share a cycle</h2>
<p>Recall that the IterF algorithm eventually converges to a stationary state. Also
recall that in a stationary state of the IterF algorithm the nodes are divided
to disjoint cycles (of one round each). Our analysis didn’t show which node will
end up at which cycle.</p>
<p>We will now discuss a few cases of pairs of nodes who we know for sure will end
up on the same cycle at a stationary state of IterF.</p>
<p><strong>Lemma (SameCycleA):</strong> If <span class="math">\(x\)</span> is connected directly to a node <span class="math">\(y\)</span> of the form
<span class="math">\(y = \pm 2^k \pm 2^t\)</span> where <span class="math">\(0 \leq k,t < l\)</span>, then <span class="math">\(x\)</span> and <span class="math">\(y\)</span> must be on the
same cycle on any stationary state of the IterF algorithm.</p>
<p>An example for a case of SameCycleA:</p>
<p><img alt="An example of SameCycleA case" src="https://www.freedomlayer.org/articles/freedomlayer/multi_named_vdht_routing/images/same_cycle_a_example.svg"></p>
<p>In this example, we know that the two nodes <span class="math">\(x\)</span> and <span class="math">\(x + 2^3 - 2^5\)</span> will end up
on the same cycle.</p>
<p><strong>Proof:</strong> For readability reasons we will only deal with the case of <span class="math">\(y = x +
2^k + 2^t\)</span>. The other cases can be proved in the same way, but with inverted
plus and minus symbols.</p>
<p>Assume a stationary state of the IterF algorithm.
We first show that <span class="math">\(\overleftarrow{(x + 2^k + 2^t)_{-2^k}} =
\overleftarrow{x_{2^t}}\)</span>. Suppose that <span class="math">\(\overleftarrow{(x + 2^k + 2^t)_{-2^k}}
\neq \overleftarrow{x_{2^t}}\)</span>. </p>
<p><img alt="Illustration of proving the SameCycleA
Lemma" src="https://www.freedomlayer.org/articles/freedomlayer/multi_named_vdht_routing/images/same_cycle_a_lemma.svg"></p>
<p>Then we can distinct between two cases:</p>
<p><em>Case 1:</em> <span class="math">\(\overleftarrow{(x + 2^k + 2^t)_{-2^k}} \in
(\overleftarrow{x_{2^t}},x+2^t]\)</span>. In that case in the next iteration of the
algorithm <span class="math">\(x\)</span> will update <span class="math">\(\overleftarrow{x_{2^t}}\)</span> to be <span class="math">\(\overleftarrow{(x +
2^k + 2^t)_{-2^k}}\)</span>, because it is closer to <span class="math">\(x+2^t\)</span> from the left (clockwise).
This is a contradiction to the stationary state of IterF.</p>
<p><em>Case 2:</em> <span class="math">\(\overleftarrow{(x + 2^k + 2^t)_{-2^k}} \in
(x+2^t,\overleftarrow{x_{2^t}})\)</span>. In that case in the next iteration of the
algorithm <span class="math">\(y = x + 2^k + 2^t\)</span> will update <span class="math">\(\overleftarrow{y_{-2^k}}\)</span> to be
<span class="math">\(\overleftarrow{x_{2^t}}\)</span>, because it is closer to <span class="math">\(x+2^t\)</span> from the left
(clockwise). This is a contradiction to the stationary state of IterF.</p>
<p>Thus it must be true that <span class="math">\(\overleftarrow{y_{-2^k}} = \overleftarrow{x_{2^t}}\)</span>.</p>
<p>By the Locality lemma (see <a href="https://www.freedomlayer.org/about-cycles-in-stationary-states-of-vdht-routing-algorithms.html">About cycles in stationary states of <span class="caps">VDHT</span> routing
algorithms</a>)
there is a direct path between <span class="math">\(x\)</span> and <span class="math">\(\overleftarrow{x_{2^t}}\)</span>, and so <span class="math">\(x\)</span> and
<span class="math">\(\overleftarrow{x_{2^t}}\)</span> must be on the same cycle. In the same way, there is a
direct path between <span class="math">\(\overleftarrow{y_{-2^k}}\)</span> to <span class="math">\(y\)</span>, therefore
<span class="math">\(\overleftarrow{y_{-2^k}}\)</span> and <span class="math">\(y\)</span> must be on the same cycle.</p>
<p>As <span class="math">\(\overleftarrow{y_{-2^k}} = \overleftarrow{x_{2^t}}\)</span>, we conclude that <span class="math">\(x\)</span>
and <span class="math">\(y\)</span> must be on the same cycle. <strong><span class="caps">QED</span></strong></p>
<p><strong>Lemma (SameCycleB):</strong> If <span class="math">\(x\)</span> is connected directly to a node <span class="math">\(y\)</span>, which is
connected directly to a node <span class="math">\(z\)</span> of the form <span class="math">\(z = x \pm 2^k\)</span>, where <span class="math">\(0 \leq k <
l\)</span>, then <span class="math">\(x\)</span> and <span class="math">\(z\)</span> must be on the same cycle on any stationary state of the
IterF algorithm.</p>
<p>An example for a case of SameCycleB:</p>
<p><img alt="An example of SameCycleB case" src="https://www.freedomlayer.org/articles/freedomlayer/multi_named_vdht_routing/images/same_cycle_b_example.svg"></p>
<p>In this example, we know that the two nodes <span class="math">\(x\)</span> and <span class="math">\(x + 2^7\)</span> will end up
on the same cycle. (The value of <span class="math">\(z\)</span> does not matter).</p>
<p><strong>Proof:</strong>
For readability reasons we will only deal with the case of <span class="math">\(z = x + 2^k\)</span>. The
other case could be proved in a similar way.</p>
<p>Assume a stationary state of the IterF algorithm. We first show that
<span class="math">\(\overleftarrow{x_{2^k}} =
x + 2^k\)</span>. Suppose that <span class="math">\(\overleftarrow{x_{2^k}} \neq x + 2^k\)</span>. Then in the next
iteration <span class="math">\(y\)</span> will tell <span class="math">\(x\)</span> about <span class="math">\(z = x + 2^k\)</span>, and <span class="math">\(x\)</span> will update
<span class="math">\(\overleftarrow{x_{2^k}}\)</span> to be <span class="math">\(x+2^k\)</span>, because it is closer to <span class="math">\(x+2^k\)</span> from
the left (It has distance 0!). This is a contradiction to the stationary state
of IterF. Therefore we conclude that <span class="math">\(\overleftarrow{x_{2^k}} = x + 2^k = z\)</span>.</p>
<p>By the Locality lemma there is a direct path between <span class="math">\(x\)</span> and
<span class="math">\(\overleftarrow{x_{2^k}} = z\)</span>. Therefore there is a direct path between <span class="math">\(x\)</span> and
<span class="math">\(z\)</span>, and so <span class="math">\(x\)</span> and <span class="math">\(z\)</span> must be on the same cycle. <strong><span class="caps">QED</span></strong>.</p>
<p>The two lemmas we have proved above tell us something about which nodes should
end up on the same cycle, but they are far from complete in describing the
structure of the cycles of a stationary state of IterF. For Example, currently
we don’t know to say much about which nodes will not be on the same cycle.</p>
<h2>On the applicability of SameCycleA and SameCycleB</h2>
<p>Given the two lemmas above (SameCycleA and SameCycleB), one might wonder how
often it is possible to apply those lemmas and conclude that all nodes will end
up on the same cycle.</p>
<p>We will perform a very hand wavy calculation to get a feeling for this likelihood.</p>
<p>For a specific node <span class="math">\(x\)</span> there are at most <span class="math">\((2l)^2\)</span> possible combinations of <span class="math">\(x
\pm 2^k \pm 2^t\)</span> for <span class="math">\(0 \leq k,t < l\)</span>, and at most <span class="math">\(2l\)</span> possible combinations of
<span class="math">\(x \pm 2^k\)</span>.</p>
<p>We assume that every node in the network is directly conected to at most
<span class="math">\(\log(n)\)</span> other nodes.</p>
<p><em>For the nodes directly connected to <span class="math">\(x\)</span></em>: Assume that those <span class="math">\(\log(n)\)</span> nodes were
chosen randomly. About <span class="math">\(\log(n) \cdot \frac{(2l)^2}{n}\)</span> out of these nodes will
be of the form <span class="math">\(x \pm 2^k \pm 2^t\)</span>. </p>
<div class="math">$$\log(n) \cdot \frac{(2l)^2}{n} = \frac{\log(n)\cdot (2l)^2}{n} \leq \frac{l\cdot
(2l)^2}{n} = \frac{4l^3}{n}$$</div>
<p>Hence for <span class="math">\(n > 4l^3\)</span> we expect that with high probability some node <span class="math">\(z\)</span> will not
have any direct connections on the network of the form <span class="math">\(z \pm 2^k \pm 2^t\)</span>.</p>
<p><em>For nodes that are two hops from <span class="math">\(x\)</span></em>: There are about <span class="math">\({\log(n)}^2\)</span> such
nodes. Assume that they were chosen randomly. About <span class="math">\({\log(n)}^2 \cdot
\frac{2l}{n}\)</span> out of these nodes will be of the form <span class="math">\(x \pm 2^k\)</span>.</p>
<div class="math">$${\log(n)}^2 \cdot \frac{2l}{n} = \frac{{2l}\cdot{\log(n)}^2}{n} \leq
\frac{2l\cdot l^2}{n} = \frac{2l^3}{n}$$</div>
<p>Hence for <span class="math">\(n > 2l^3\)</span> we expect that with high probability some node <span class="math">\(z\)</span> will not
have any node two hops from him of the form <span class="math">\(z \pm 2^k\)</span>.</p>
<p>As a final rough conclusion, we believe that for large enough values of <span class="math">\(n\)</span>, for
example <span class="math">\(n \gg 4l^3\)</span>, we expect that for some nodes non of the lemmas SameCycleA
or SameCycleB will be applicable. </p>
<p>We might need something else to be able to prove the convergence of IterF to a
single cycle.</p>
<h2>Multi Naming</h2>
<p>For every two nodes <span class="math">\(x\)</span> and <span class="math">\(y\)</span>, we say that <span class="math">\(x\)</span> and <span class="math">\(y\)</span> are <strong>FriendsA</strong> if <span class="math">\(y\)</span>
is of the form <span class="math">\(y = x \pm 2^k \pm 2^t\)</span>. (A special case would be <span class="math">\(y = x \pm
2^k\)</span>). Note that this property is symmetric (If <span class="math">\(x\)</span> and <span class="math">\(y\)</span> are FriendsA, then
<span class="math">\(y\)</span> and <span class="math">\(x\)</span> are FriendsA). We will denote that <span class="math">\(x\)</span> and <span class="math">\(y\)</span> are FriendsA by <span class="math">\(x
\sim y\)</span>.</p>
<p>Consider some two directly connected nodes <span class="math">\(x,y\)</span> on a mesh network. If <span class="math">\(x \sim
y\)</span> then by the SameCycleA lemma we can show that <span class="math">\(x\)</span> and <span class="math">\(y\)</span> will end up on the
same cycle. However, if <span class="math">\(x \nsim y\)</span>, we don’t have anything meaningful to say.
Unfortunately, in most cases we will have <span class="math">\(x \nsim y\)</span>.</p>
<p>Consider that case of <span class="math">\(x \nsim y\)</span>. Imagine for a moment that there were some
extra nodes between <span class="math">\(x\)</span> and <span class="math">\(y\)</span>: <span class="math">\(x = z_1,z_2,\dots,z_r = y\)</span> such that <span class="math">\(z_j \sim
z_{j+1}\)</span> for <span class="math">\(1 \leq j < r\)</span>. In that case we conclude that <span class="math">\(z_j, z_{j+1}\)</span> are on
the same cycle for <span class="math">\(1 \leq j < r\)</span>, and therefore <span class="math">\(x\)</span> and <span class="math">\(y\)</span> must end up on the
same cycle on every stationary state of the IterF algorithm.</p>
<p><img alt="Before and after adding imaginary extra nodes between \(x\) and
\(y\)" src="https://www.freedomlayer.org/articles/freedomlayer/multi_named_vdht_routing/images/imaginary_extra_nodes_x_y.svg"></p>
<p>In the picture: Before and after adding imaginary extra nodes between <span class="math">\(x\)</span> and
<span class="math">\(y\)</span>.</p>
<p>In the real network there are no such extra nodes, but nothing prevents <span class="math">\(x\)</span>
and <span class="math">\(y\)</span> to pretend the existence of those nodes. <span class="math">\(x\)</span> will play the role of
<span class="math">\(z_1,z_2,\dots,z_{\left\lceil r/2 \right\rceil}\)</span> in the network, and <span class="math">\(y\)</span> will
play the role of <span class="math">\(z_{\left\lfloor r/2 \right\rfloor},\dots,z_r\)</span> in the network.</p>
<p>We want to know how large can <span class="math">\(r\)</span> be. We know that <span class="math">\(0 \leq x,y < 2^l\)</span>, hence
both <span class="math">\(x\)</span> and <span class="math">\(y\)</span> could be written using at most <span class="math">\(l\)</span> bits. We want to get from
<span class="math">\(x\)</span> to <span class="math">\(y\)</span> using small changes of the form <span class="math">\(\pm 2^k \pm 2^t\)</span>. Every small change
introduces another imaginary extra node.</p>
<p>We start from <span class="math">\(z_1 = x\)</span>. We can reach <span class="math">\(y\)</span> by changing at most two bits at a
time. <span class="math">\(x\)</span> and <span class="math">\(y\)</span> will usually differ at about <span class="math">\(l/2\)</span> bits. We assume that worst
case, where <span class="math">\(x\)</span> and <span class="math">\(y\)</span> differ at <span class="math">\(l\)</span> bits. Hence we will need at most
<span class="math">\(\left\lceil l/2 \right\rceil\)</span> imaginary nodes. (At most two bits in every small
change. We add the ceiling operator for the case of odd <span class="math">\(l\)</span>). We conclude that
<span class="math">\(r \leq \left\lceil l/2 \right\rceil\)</span>.</p>
<p>Hence each of <span class="math">\(x,y\)</span> will play the role of at most <span class="math">\(\left\lceil l/4 \right\rceil\)</span>
extra imaginary nodes.</p>
<p>We can add imaginary extra nodes between every two nodes <span class="math">\(a,b\)</span> in the network
such that <span class="math">\(a \nsim b\)</span>. A node <span class="math">\(z\)</span> that is directly connected to <span class="math">\(q\)</span> nodes will
have to play the role of at most <span class="math">\(q\cdot \left\lceil l/4 \right\rceil\)</span> nodes.
This means that every node will maintain remote paths to about
<span class="math">\(q\cdot(l/4)\cdot(4l) = ql^2\)</span> nodes.</p>
<p>Note that originally, every node had to maintain remote paths to about <span class="math">\(2l\)</span>
nodes, which is much less. </p>
<p>Adding the extra imaginary nodes should not make message routing less efficient,
because routing messages between imaginary nodes does not cost significant time.
The modification does require, however, that nodes will maintain more “fingers”:
More paths to remote nodes. The more direct connections a node has, the more
remote connections he will have to maintain.</p>
<p><img alt="Before and after adding imaginary extra nodes between all pairs of connected
nodes" src="https://www.freedomlayer.org/articles/freedomlayer/multi_named_vdht_routing/images/imaginary_extra_nodes_all_pairs.svg"></p>
<p>In the picture: Before and after adding imaginary extra nodes between all pairs
of connected nodes. The imaginary nodes are shown with lighter color that
matches the color of the real node who plays their role in the algorithm.</p>
<h2>Practicality issues</h2>
<p><em>Regarding Network performance:</em> We require every node to maintain remote
paths to about <span class="math">\(ql^2\)</span> nodes. Assuming that a node <span class="math">\(x\)</span> has <span class="math">\(5\)</span> direct
connections, and we pick <span class="math">\(l=128\)</span>, this results in <span class="math">\(81920\)</span> (<span class="math">\(\approx 2^{15}\)</span>) which
is a lot of connections. I think that this number is on the edge of what is
practical these days with communication. I believe that there is a way to
slightly improve the MIterF algorithm to acheive more realistic numbers.</p>
<p><em>Who gets to pick the names for the nodes?</em> This is a major question, as name
collisions (two nodes with the same name) could disrupt correct routing. Also,
who gets to pick how the path of imaginary FriendsA nodes between two nodes
<span class="math">\(x,y\)</span> will be constructed? Where will the names come from?</p>
<p>We solve the problem of name collisions by using a large namespace. We pick a
large <span class="math">\(l\)</span>, for example <span class="math">\(l=128\)</span>. If names are chosen randomly, the probability of
a name collision should be very small (Even between the paths of imaginary
FriendsA nodes).</p>
<p><em>Security: Will this system survive a strong adversary?</em> Not in its current
form. We need to add some mechanisms to avoid the following possible attacks:</p>
<ul>
<li>
<p>Eclipse attack: An attacker might be able to obtain many names that are
close to a node <span class="math">\(x\)</span> in the namespace, and thus absorb all the messages that
should arrive at <span class="math">\(x\)</span>. Therefore <span class="math">\(x\)</span> will not be able to receive any messages.
The question of who gets to pick the names is tightly related to preventing
these kinds of attacks.</p>
</li>
<li>
<p><a href="https://en.wikipedia.org/wiki/Denial-of-service_attack">Denial of Service</a>:
A node might be able to send many messages and flood the network. A node might
also decide to stop routing certain messages, and thus disrupt routing of
messages in the network.</p>
</li>
</ul>
<h2>Further questions</h2>
<p>We know that the MIterF always converges to a stationary state where all the
nodes are in the same cycle (And the cycle has only one round). We still haven’t
answered the following questions:</p>
<ul>
<li>
<p>How long does it take for MIterF to converge? I think that the answer is
something like <span class="math">\(O(c\cdot diam(G))\)</span>, where <span class="math">\(c\)</span> is some constant and <span class="math">\(diam(G)\)</span> is the
diameter of the network (The longest distance between two nodes), but I don’t
know of a proof for this.</p>
</li>
<li>
<p>How long are the paths that we get from each node <span class="math">\(x\)</span> to his “fingers”? For
example, how long would be the path from <span class="math">\(x\)</span> to <span class="math">\(\overleftarrow{x_{2^t}}\)</span>, as
calculated by MIterF? How does it compare to the length of the shortest path in
the network between <span class="math">\(x\)</span> and <span class="math">\(\overleftarrow{x_{2^t}}\)</span>? Currently I have no idea
about how to estimate the length of paths calculated by MIterF. Note that this
question might be related to the shape of the network.</p>
</li>
</ul>
<h2>Summary</h2>
<p>We presented MIterF, an iterative distributed algorithm that builds a setting
for efficient routing of messages between nodes. Assuming that the network is
connected, we proved that MIterF always converges to stationary state that
allows efficient routing between all pairs of nodes in the network.</p>
<p>During the execution of the algorithm every node <span class="math">\(x\)</span> that has direct connections
to <span class="math">\(q\)</span> nodes must maintain remote paths to at most <span class="math">\(ql^2\)</span> nodes.</p>
<script type="text/javascript">if (!document.getElementById('mathjaxscript_pelican_#%@#$@#')) {
var align = "center",
indent = "0em",
linebreak = "false";
if (false) {
align = (screen.width < 768) ? "left" : align;
indent = (screen.width < 768) ? "0em" : indent;
linebreak = (screen.width < 768) ? 'true' : linebreak;
}
var mathjaxscript = document.createElement('script');
mathjaxscript.id = 'mathjaxscript_pelican_#%@#$@#';
mathjaxscript.type = 'text/javascript';
mathjaxscript.src = 'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.0/MathJax.js?config=TeX-AMS-MML_HTMLorMML';
mathjaxscript[(window.opera ? "innerHTML" : "text")] =
"MathJax.Hub.Config({" +
" config: ['MMLorHTML.js']," +
" TeX: { extensions: ['AMSmath.js','AMSsymbols.js','noErrors.js','noUndefined.js'], equationNumbers: { autoNumber: 'AMS' } }," +
" jax: ['input/TeX','input/MathML','output/HTML-CSS']," +
" extensions: ['tex2jax.js','mml2jax.js','MathMenu.js','MathZoom.js']," +
" displayAlign: '"+ align +"'," +
" displayIndent: '"+ indent +"'," +
" showMathMenu: true," +
" messageStyle: 'normal'," +
" tex2jax: { " +
" inlineMath: [ ['\\\\(','\\\\)'] ], " +
" displayMath: [ ['$$','$$'] ]," +
" processEscapes: true," +
" preview: 'TeX'," +
" }, " +
" 'HTML-CSS': { " +
" styles: { '.MathJax_Display, .MathJax .mo, .MathJax .mi, .MathJax .mn': {color: 'inherit ! important'} }," +
" linebreaks: { automatic: "+ linebreak +", width: '90% container' }," +
" }, " +
"}); " +
"if ('default' !== 'default') {" +
"MathJax.Hub.Register.StartupHook('HTML-CSS Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax['HTML-CSS'].FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"MathJax.Hub.Register.StartupHook('SVG Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax.SVG.FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"}";
(document.body || document.getElementsByTagName('head')[0]).appendChild(mathjaxscript);
}
</script>About cycles in stationary states of VDHT routing algorithms2015-10-15T17:23:00+00:002015-10-15T17:23:00+00:00realtag:www.freedomlayer.org,2015-10-15:/about-cycles-in-stationary-states-of-vdht-routing-algorithms.html<h2>Abstract</h2>
<p>We are given a connected network (graph) of named nodes. We discuss an iterative
distributed algorithm that allows each node in the graph to find a path to
several other key nodes. Those paths later allow efficient routing of messages
between any two nodes in the network.</p>
<p>We discuss …</p><h2>Abstract</h2>
<p>We are given a connected network (graph) of named nodes. We discuss an iterative
distributed algorithm that allows each node in the graph to find a path to
several other key nodes. Those paths later allow efficient routing of messages
between any two nodes in the network.</p>
<p>We discuss one specific iterative distributed algorithm (IterF). We show that it
converges for every given connected graph after a finite amount of steps, and we
discuss some properties of its output.</p>
<p>It turns out that the IterF algorithm returns one or a few disjoint sets of
nodes. Each of those sets has a structure that allows efficient routing between
the nodes inside that set. (But not to any node outside that set).</p>
<h2>Intro</h2>
<p>In <a href="https://www.freedomlayer.org/experimenting-with-virtual-dht-routing.html">Experimenting with Virtual <span class="caps">DHT</span> Routing</a>
we have discussed an idea of routing using a Virtual <span class="caps">DHT</span>.
follows: Every node in a mesh graph is given a random number from a “closed
ring”, and the nodes somehow try to find each other, iteratively. </p>
<p>We think about the following simplified routing problem: We are given a
connected graph of <span class="math">\(n = 2^l\)</span> vertices. Each verticle is given a number between
<span class="math">\(0\)</span> and <span class="math">\(2^l - 1\)</span> (Every two vertices are given different numbers). This
assignment of names is arbitrary, and we have no control over it. The vertices
of this graph represent the nodes in a mesh network. </p>
<p>We want a distributed algorithm that allows node <span class="math">\(x\)</span> to find a path to some key
nodes: <span class="math">\(x-2^i , x+2^i\)</span> for <span class="math">\(0 \leq i < l\)</span>. We know that this should be possible,
because the graph is connected, and those paths exist.</p>
<p>Given that every node of number <span class="math">\(x\)</span> knows paths to nodes of numbers <span class="math">\(x-2^i,
x+2^i\)</span>, every node <span class="math">\(y\)</span> should be able to send a message to any other node <span class="math">\(z\)</span> in
the network using the <a href="https://www.freedomlayer.org/intro-to-distributed-hash-tables-dhts.html">greedy routing algorithm of the Chord
<span class="caps">DHT</span></a>: In every
iteration send the message to the closest known node to the destination.</p>
<h2>Running the Basic Algorithm</h2>
<p>We begin with a basic algorithm, where every node <span class="math">\(x\)</span> tries to find paths to the
adjacent nodes on the circle: <span class="math">\(x-1\)</span> and <span class="math">\(x+1\)</span>. It will not give us an efficient
routing solution, but it will teach us something about the convergence of
similar algorithms.</p>
<p>Every node <span class="math">\(x\)</span> remembers at all times paths to two nodes: A node <span class="math">\(x^l\)</span> (Read:
<span class="math">\(x\)</span><span class="quo">‘</span>s left) that minimizes <span class="math">\(d(x^l,x-1)\)</span>, and a node <span class="math">\(x^r\)</span> that minimizes <span class="math">\(d(x+1,x^r)\)</span>.
Recall that <span class="math">\(d(a,b)\)</span> is the clockwise distance between <span class="math">\(a\)</span> and <span class="math">\(b\)</span>. Hence, In
other words, <span class="math">\(x\)</span> tries to find nodes that are closest to him clockwise and counter-clockwise.</p>
<p><img alt="x^l and x^r" src="https://www.freedomlayer.org/articles/freedomlayer/vdht_cycles_rounds/images/x_left_x_right.svg"></p>
<p>We describe here the algorithm, as executed by one node <span class="math">\(x\)</span>. The algorithm takes
as arguments the direct connections of <span class="math">\(x\)</span> to other nodes in the network. The
set of the direct connections of <span class="math">\(x\)</span> is called DirectCons.</p>
<p>Macro <span class="math">\(Known\)</span>: <span class="math">\(DirectCons \cup \{x^l,x^r\} \cup TempRecipients\)</span></p>
<ul>
<li>
<p>Initialization(<span class="math">\(DirectCons\)</span>):</p>
<ol>
<li>Set <span class="math">\(x^l, x^r\)</span> to be the most suitable nodes from the set <span class="math">\(DirectCons \cup
\{x\}\)</span>.</li>
<li><span class="math">\(TempRecipients \leftarrow \emptyset\)</span></li>
</ol>
</li>
<li>
<p>On receival of Update message with set <span class="math">\(S\)</span> from node <span class="math">\(y\)</span></p>
<ol>
<li>
<p>If <span class="math">\(y \not\in Known\)</span>:</p>
<p>1.1 Add <span class="math">\(y\)</span> to <span class="math">\(TempRecipients\)</span>.</p>
</li>
<li>
<p>Update <span class="math">\(x^l\)</span> and <span class="math">\(x^r\)</span> if better candidates were found inside <span class="math">\(S \cup
\{y\}\)</span>.</p>
</li>
</ol>
</li>
<li>
<p>On time tick:</p>
<ol>
<li>
<p>For every node <span class="math">\(y \in Known\)</span>:</p>
<p>1.1 Send <span class="math">\(y\)</span> an Update message with set <span class="math">\(Known\)</span>.</p>
</li>
<li>
<p>Set <span class="math">\(TempRecipients \leftarrow \emptyset\)</span></p>
</li>
</ol>
</li>
</ul>
<p>Experiments (Described in previous articles) have shown that the basic algorithm
does not always converge to a solution. That means: Sometimes a node <span class="math">\(x\)</span> will
not find <span class="math">\(x-1\)</span> and <span class="math">\(x+1\)</span>, even after running the algorithm for many iterations.
It is also noticeable that after a while the state of network graph is
stuck: further iterations of the Basic Algorithm do not change the state. We
call such state a stationary state. It is interesting to understand what is the
state of the network graph at this point.</p>
<h2>Analyzing the operation of the Basic Algorithm</h2>
<p>For every node <span class="math">\(x\)</span> we denote by <span class="math">\(x^l\)</span> (<span class="math">\(l\)</span> stands for left) the node closest to
<span class="math">\(x\)</span> counter-clockwise that <span class="math">\(x\)</span> knows of (That is not <span class="math">\(x\)</span>). We denote by <span class="math">\(x^r\)</span>
the node closest to <span class="math">\(x\)</span> clockwise that <span class="math">\(x\)</span> knows of (That is not <span class="math">\(x\)</span>).
(As a sanity check for understanding the notation, <span class="math">\(x^l\)</span> minimizes <span class="math">\(d(x^l,x)\)</span>
between all nodes that were ever known to <span class="math">\(x\)</span>)</p>
<p>We define the state of the algorithm to be <span class="math">\(S = \{(x,x^l,x^r) | 0 \leq x < n\}\)</span>
It is the set of all choices of left and right node for all the nodes. We say
that a state <span class="math">\(S\)</span> is stationary if any amount of algorithm iterations will leave
<span class="math">\(S\)</span> unchanged.</p>
<p>We first show that every run of the Basic Algorithm indeed eventually converges
to a stationary state. </p>
<p><strong>Lemma:</strong> After a finite amount of iterations of the Basic Algorithm, the
algorithm converges to a stationary state.</p>
<p><strong>Proof:</strong> For a state <span class="math">\(S\)</span> of the algorithm we define:
</p>
<div class="math">$$ e(S) := {\sum}_x {d(x+1,x^r) + d(x^l,x-1)}$$</div>
<p><span class="math">\(e\)</span> is an energy function of the current state of the algorithm. It decreases in
every iteration of the algorithm (When some node <span class="math">\(y\)</span> updates his <span class="math">\(y^l\)</span> or <span class="math">\(y^r\)</span>
to a better node). As the only possible values for <span class="math">\(e\)</span> are the natural numbers,
after a finite amount of iterations of the Basic Algorithm <span class="math">\(e\)</span> will not be able
to decrease anymore, and we will reach a stationary state of the algorithm. <strong><span class="caps">QED</span></strong></p>
<p><strong>Lemma:</strong> During a stationary state of the algorithm, for every node <span class="math">\(x\)</span> it
is true that <span class="math">\(x^{rl} = x\)</span> and <span class="math">\(x^{lr} = x\)</span>. (<span class="math">\(x^{rl}\)</span> is a short notation for
<span class="math">\((x^r)^l\)</span>)</p>
<p><strong>Proof:</strong> Let <span class="math">\(x\)</span> be some node. Consider <span class="math">\(x^r\)</span> and <span class="math">\(x^{rl}\)</span>. First note that <span class="math">\(x
\neq x^r\)</span>, because that would mean <span class="math">\(x\)</span> has no direct connections, and the graph
is not connected. Of a similar reason, <span class="math">\(x^r \neq x^{rl}\)</span>.</p>
<p>Suppose that <span class="math">\(x \neq x^{rl}\)</span>. Then we distinct between two cases:</p>
<p><img alt="Two cases for the location of x^{rl}" src="https://www.freedomlayer.org/articles/freedomlayer/vdht_cycles_rounds/images/x_left_right_lemma.svg"></p>
<p><em>Case 1</em>: <span class="math">\(x^{rl} \in (x,x^r)\)</span>. (This means: <span class="math">\(x^{rl}\)</span> is on the arc between <span class="math">\(x\)</span> and
<span class="math">\(x^r\)</span>, clockwise). In the next iteration <span class="math">\(x\)</span> will send <span class="math">\(x^r\)</span> an Update message.
In the following iteration, <span class="math">\(x^r\)</span> will send <span class="math">\(x\)</span> an Update message. This message
will contain a path to <span class="math">\(x^{rl}\)</span>. <span class="math">\(x\)</span> will then update <span class="math">\(x^r\)</span> to be <span class="math">\(x^{rl}\)</span>. This
is a contradiction to the given stationary state.</p>
<p><em>Case 2</em>: <span class="math">\(x^{rl} \in (x^r,x)\)</span>. In the next iteration <span class="math">\(x\)</span> will send <span class="math">\(x^r\)</span> an
Update message, and <span class="math">\(x^r\)</span> will update <span class="math">\(x^{rl}\)</span> to be <span class="math">\(x\)</span>. This is a
contradiction to the given stationary state.</p>
<p>Hence it must be true that <span class="math">\(x = x^{rl}\)</span>. In the same way, we can conclude that
<span class="math">\(x^{lr} = x\)</span>. <strong><span class="caps">QED</span></strong></p>
<p><strong>Lemma:</strong> During a stationary state of the algorithm, The function <span class="math">\(h(x) :=
x^r\)</span> (Or dually, <span class="math">\(t(x) := x^l\)</span>) is <a href="https://en.wikipedia.org/wiki/Injective_function">injective (one to one)</a></p>
<p><strong>Proof:</strong> Let <span class="math">\(x,y\)</span> nodes such that <span class="math">\(h(x) = h(y) = z\)</span>, for some node <span class="math">\(z\)</span>.
By the previous lemma we conclude that <span class="math">\(z^l = x^{rl} = x\)</span> and <span class="math">\(z^l = x^{rl} =
y\)</span>, therefore <span class="math">\(x=y\)</span>. <strong><span class="caps">QED</span></strong></p>
<h2>Division to cycles</h2>
<p>We now begin from the node <span class="math">\(x\)</span>, and apply the function <span class="math">\(h\)</span> over <span class="math">\(x\)</span> multiple
times. We get the nodes <span class="math">\(x, h(x), h(h(x)), \dots\)</span>. There is a finite amount of
nodes, so we expect that eventually some element <span class="math">\(h^k(x)\)</span> (<span class="math">\(h\)</span> applied <span class="math">\(k\)</span> times
over <span class="math">\(x\)</span>) will be equal to some other element <span class="math">\(h^q(x), q < k\)</span> that was already
on the series. Let’s assume that <span class="math">\(k\)</span> is the smallest such integer.</p>
<p>But since <span class="math">\(h\)</span> is injective, <span class="math">\(h^q(x) = h^k(x)\)</span> means that <span class="math">\(x = h^{k-q}(x)\)</span>. As
<span class="math">\(k\)</span> is the index of the first element that equals to a previously seen element,
we conclude that <span class="math">\(q=0\)</span> and so <span class="math">\(h^k(x) = x\)</span>, the first element of the series.
Therefore the series <span class="math">\(x, h(x), \dots, h^k(x)=x\)</span> form a cycle. </p>
<p><img alt="Cycles formed from the h function" src="https://www.freedomlayer.org/articles/freedomlayer/vdht_cycles_rounds/images/h_iterated_cycles.svg"></p>
<p>Note that this is a generic property of injective functions that operate inside
finite worlds.</p>
<p>If we now pick some node <span class="math">\(z\)</span> that has not appeared in the cycle of the node <span class="math">\(x\)</span>,
and start applying <span class="math">\(h\)</span> many times, we will get another cycle. We can continue in
this fashion to divide all the nodes to different cycles.</p>
<h2>The possibility of multiple cycles</h2>
<p>From the perspective of routing in the mesh, we don’t want to have different
cycles for different nodes, as nodes from different cycles can not reach each
other. We hope that all the nodes end up on the same cycle.</p>
<p>To convince you that the possibility of disjoin cycles in our solution is real,
we will look at a counter example: An input graph that results in a set of
disjoint cycles after running our basic algorithm:</p>
<div class="highlight"><pre><span></span>7 - 5 - 3 8 -10 -12
| | | |
9 -11 - 1 --- 6 - 4 - 2
</pre></div>
<p>If we run the Basic Algorithm it will converge after one iteration, and two
cycles will be formed: (7,5,3,1,11,9) and (8,10,12,2,4,6). As an example, <span class="math">\(2\)</span>
will not be able to route messages to <span class="math">\(3\)</span>, because it lives in another cycle.</p>
<p>For now we will forget about the disjoint cycles problem, and we will instead
try to understand how a cycle looks like.</p>
<h2>Multiple Rounds</h2>
<p>Assume that somehow the algorithm has converged and only one cycle was formed.
(In other words: All the nodes ended up on the same cycle). Does that mean that
<span class="math">\(x^r = x+1\)</span> for every node <span class="math">\(x\)</span>?</p>
<p>The answer is no. Let’s build a counter example: An input graph for the
algorithm. The input graph is a cycle size <span class="math">\(2^{3}\)</span>:</p>
<div class="highlight"><pre><span></span>0 - 2 - 4 - 6
| |
7 - 5 - 3 - 1
</pre></div>
<p>The basic algorithm will converge on the first iteration. Each node will have
his right and left nodes to be exactly the right and left nodes on the graph.</p>
<p>The wanted result from the algorithm is that <span class="math">\(x\)</span> will have paths to <span class="math">\(x-1\)</span>,<span class="math">\(x+1\)</span>.
This obviously doesn’t happen here, even though we did get one cycle that
contains all the nodes. </p>
<p>The extra intuitive requirement that we miss here is that the cycle will
complete only “one round”. We will now give a more rigorous definition for the
notion of round.</p>
<p><strong>Definition:</strong> The amount of rounds of a cycle <span class="math">\(C = (x, h(x), \dots ,
h^k(x)=x)\)</span> is <span class="math">\(R(C)\)</span>: the amount of local maximums (minimums) in the cycle. In
other words, it is the amount of <span class="math">\(0 \leq i < k\)</span> such that <span class="math">\(h^i(x) < h^{i+1}(x) >
h^{i+2}(x)\)</span>. (Note that in a cycle, the amount of minimums is always the same as
the amount of maximums)</p>
<p>For example, in the cycle drawn above, <span class="math">\(7\)</span> and <span class="math">\(6\)</span> are local maximums. Dually,
<span class="math">\(0\)</span> and <span class="math">\(1\)</span> are local minimums.</p>
<p><strong>Lemma:</strong> During a convergent state of the Basic Algorithm, If a cycle <span class="math">\(C\)</span>
contains all the nodes from <span class="math">\(0\)</span> to <span class="math">\(n-1\)</span> and <span class="math">\(R(C)=1\)</span> then for every node <span class="math">\(x\)</span> in
the cycle, <span class="math">\(h(x) = x+1\)</span> (Modulo <span class="math">\(n\)</span>)</p>
<p><strong>Proof:</strong> <span class="math">\(R(C)=1\)</span> means that the cycle has only one local minimum and one
local maximum. Those must be the nodes <span class="math">\(0\)</span> and <span class="math">\(n-1\)</span>. (Because they are also
global minimum and global maximum).</p>
<p>If <span class="math">\(0\)</span> is next to <span class="math">\(n-1\)</span> in the cycle (<span class="math">\(h(n-1) = 0\)</span> without loss of generality),
then the series <span class="math">\(b_k := h^k(0)\)</span>, <span class="math">\(0 \leq k < n\)</span> is monotonically increasing.
<span class="math">\(b_0 = 0\)</span> and <span class="math">\(b_{n-1} = n-1\)</span>, and also <span class="math">\((b_{i+1} - b_{i}) \geq 1\)</span> for all <span class="math">\(0
\leq i < n\)</span>. Therefore it must be true that <span class="math">\((b_{i+1} - b_{i}) = 1\)</span> for all <span class="math">\(0
\leq i < n\)</span>. This means that <span class="math">\(h(x) = x+1\)</span> (Modulo <span class="math">\(n\)</span>) for every node <span class="math">\(x\)</span> in
the cycle.</p>
<p>Now assume that node <span class="math">\(0\)</span> is not next to <span class="math">\(n-1\)</span> in the cycle. We will soon see
that this is not possible under our assumptions.</p>
<p>Beginning from <span class="math">\(0\)</span>, if we repeatedly apply the function <span class="math">\(h\)</span> we get the series
<span class="math">\(0, h(0), h^2(0), \dots\)</span>. If we ever get <span class="math">\(h^j(x) > h^{j+1}(x)\)</span> for some <span class="math">\(j\)</span>, it
means that <span class="math">\(h^j(x) = n-1\)</span>, the only local maximum in this cycle. Hence from this
point in the cycle applying the function <span class="math">\(h\)</span> gives decreasing values, until it
returns back to <span class="math">\(0\)</span>.</p>
<p>To summarize, we have seen that the cycle begins from <span class="math">\(0\)</span>, increases until it
reaches <span class="math">\(n-1\)</span> and then decreases to <span class="math">\(0\)</span>.</p>
<p>We observe the part of the cycle around the node <span class="math">\(0\)</span>: <span class="math">\(\dots,a,0,b,\dots\)</span>.
<span class="math">\(a_l > a\)</span>, therefore <span class="math">\(a_l\)</span> is a better choice for left node for <span class="math">\(0\)</span>. Hence in
the next iteration the node <span class="math">\(a\)</span> will tell the node <span class="math">\(0\)</span> about a path to <span class="math">\(a_l\)</span>,
and <span class="math">\(0_l\)</span> will be <span class="math">\(a_l\)</span>. This means that this state is not a convergent state of
the algorithm, which leads to a contradiction. (Note that we could not have
obtained this contradiction if <span class="math">\(a=n-1\)</span>)</p>
<p><strong><span class="caps">QED</span></strong></p>
<p>We can conclude that we want an algorithm that will leave us with just one cycle
<span class="math">\(C\)</span> with <span class="math">\(R(C) = 1\)</span> that contains all nodes. </p>
<p>Having all the nodes in one cycle of only one round allows efficient routing
using the following greedy method: On receipt of a message <span class="math">\(m\)</span> that is
designated to node <span class="math">\(y\)</span>, choose a node <span class="math">\(z\)</span> with a name closest to <span class="math">\(y\)</span> out of all the
known nodes, and send the message to <span class="math">\(z\)</span> (through the maintained path). We
expect the message to arrive to its destination after at most <span class="math">\(l\)</span> such hops.</p>
<h2>Restricting the amount of rounds</h2>
<p>We might be able to make some modification to our Basic Algorithm, to make sure
that each cycle <span class="math">\(C\)</span> of nodes we eventually get has just one round (<span class="math">\(R(C)=1\)</span>).
This will not be enough to converge to a real solution, because we might have
more than one cycle, but it’s a start.</p>
<h3>Adding two distant fingers</h3>
<p>Intuitively, thinking about some node <span class="math">\(x\)</span> inside a cycle <span class="math">\(C\)</span> of more than one
round, the existence of multiple rounds could not be detected because <span class="math">\(x\)</span> does
not see far enough. He can only see his immediate connections to the left and
right, but he can not see what is going on on the other side of the cycle.</p>
<p>We could try to add two more “fingers”: In addition to remembering <span class="math">\(x^r\)</span> and
<span class="math">\(x^l\)</span>, <span class="math">\(x\)</span> will also remember paths to the following nodes:</p>
<ul>
<li>
<p><span class="math">\(x^L\)</span> is the node that is closest (clockwise) to <span class="math">\(x + 2^{l-1}\)</span>. In other
words, it is the node <span class="math">\(t\)</span> that minimizes <span class="math">\(d(t,x+2^{l-1})\)</span>, of all the nodes that
<span class="math">\(x\)</span> has ever encountered.</p>
</li>
<li>
<p><span class="math">\(x^R\)</span> is the node that is closest (counter-clockwise) to <span class="math">\(x+2^{l-1}\)</span>, of all
the nodes that <span class="math">\(x\)</span> has ever encountered.</p>
</li>
</ul>
<p>I believe that using this method will still result in cycles with multiple
rounds. Some work is still required here. I didn’t manage to come up with a
simple example of a stationary state that contains a cycle of multiple rounds in
this setting. In addition, I never managed to prove that this method results in
a cycles that have only one round.</p>
<h3>Adding all powers of <span class="math">\(2\)</span> fingers</h3>
<p>On the quest of finding a strong enough setting to eliminate cycles with
multiple rounds, we add more fingers. A nice bet would be to add all the fingers
of sizes which are powers of <span class="math">\(2\)</span>, just like in the original
<a href="https://www.freedomlayer.org/intro-to-distributed-hash-tables-dhts.html">Chord <span class="caps">DHT</span> structure</a>.</p>
<p>To be more specific: We say that a node <span class="math">\(x\)</span> maintains a finger <span class="math">\(k\)</span> if <span class="math">\(x\)</span>
remembers a path to: </p>
<ul>
<li>
<p>Left side: A node <span class="math">\(\overleftarrow{x_k}\)</span>, which is the node closest to <span class="math">\(x+k\)</span>
from the left (clockwise) that <span class="math">\(x\)</span> knows of. Note that this node could be <span class="math">\(x+k\)</span>
himself. In other words, this node minimizes <span class="math">\(d(z,x+k)\)</span> between all nodes that
<span class="math">\(x\)</span> has ever known.</p>
</li>
<li>
<p>Right side: A node <span class="math">\(\overrightarrow{x_k}\)</span>, which is the node closest to <span class="math">\(x+k\)</span>
from the right (clockwise) that <span class="math">\(x\)</span> knows of. Note that this node could be <span class="math">\(x+k\)</span>
himself. In other words, this node minimizes <span class="math">\(d(x+k,z)\)</span> between all nodes that
<span class="math">\(x\)</span> has ever known.</p>
</li>
</ul>
<p>We create a version of the algorithm where each node <span class="math">\(x\)</span> maintains the following
fingers: <span class="math">\(\pm 2^i\)</span> for <span class="math">\(0 \leq i < l\)</span>. (Note that <span class="math">\(-2^{l-1}\)</span> is just
<span class="math">\(2^{l-1}\)</span> modulo <span class="math">\(n = 2^l\)</span>, the distant finger). We call this version IterF.</p>
<p>Note that <span class="math">\(x^l\)</span> is the left side of finger <span class="math">\(-1\)</span>, and <span class="math">\(x^r\)</span> is the right side of
finger <span class="math">\(+1\)</span>.</p>
<p>The following part assumes a stationary state of the algorithm. Using previous
Lemmas we know that at a convergent state all the nodes are divided into cycles
(with respect to <span class="math">\(x^l\)</span> and <span class="math">\(x^r\)</span> choices). We can deduce this fact using only
the convergent properties of all <span class="math">\(x^l,x^r\)</span> for every node <span class="math">\(x\)</span> (Ignoring the
existence of all the other fingers).</p>
<p><strong>Definition:</strong> A <strong>Path</strong> between nodes <span class="math">\(a\)</span> and <span class="math">\(b\)</span> (clockwise) is a sequence
<span class="math">\(a = z_1, z_2, \dots ,z_r = b\)</span>, where <span class="math">\(h(z_j) = z_{j+1}\)</span> for <span class="math">\(1 \leq j < r\)</span>.</p>
<p><strong>Definition:</strong> A <strong>Direct path</strong> between nodes <span class="math">\(a\)</span> and <span class="math">\(b\)</span> is a path <span class="math">\(a = z_1,
z_2, \dots, z_r=b\)</span> such that <span class="math">\({\sum}_{1 \leq j < r} d(z_j,z_{j+1}) = d(a,b)\)</span>.</p>
<p>Intuitively, a direct path is a path that doesn’t go more than one round around
the circle.</p>
<p><strong>Lemma (Locality):</strong> Let <span class="math">\(a\)</span> be a node, and assume <span class="math">\(0 \leq t < l\)</span>. Then
<em><span class="caps">DPATH</span>[t]</em>: there exist direct paths between <span class="math">\(a\)</span> to each of the following nodes:
<span class="math">\(\overleftarrow{a_{2^t}}, \overrightarrow{a_{2^t}}\)</span>, and there exist direct
paths from each of the nodes <span class="math">\(\overleftarrow{a_{-2^t}},
\overrightarrow{a_{-2^t}}\)</span> to the node <span class="math">\(a\)</span>.</p>
<p><strong>Proof: </strong> By induction of <span class="math">\(t\)</span>. We begin with the <strong>base case</strong>, for <span class="math">\(t=0\)</span>. Let
<span class="math">\(a\)</span> be some node, and consider the node <span class="math">\(\overrightarrow{a_1}\)</span>. If <span class="math">\(d(a,h(a)) =
1\)</span> then <span class="math">\(\overrightarrow{a_1} = h(h(a))\)</span>, hence there is a direct path:
<span class="math">\(a,h(a),h(h(a))\)</span> between <span class="math">\(a\)</span> and <span class="math">\(\overrightarrow{a_1}\)</span>.</p>
<p>If <span class="math">\(d(a,h(a)) > 1\)</span> then <span class="math">\(\overrightarrow{a_1} = h(a)\)</span>. Hence <span class="math">\(a,h(a)\)</span> is a
direct path between <span class="math">\(a\)</span> and <span class="math">\(\overrightarrow{a_1}\)</span>.</p>
<p>Next, consider the node <span class="math">\(\overleftarrow{a_1} = a\)</span>. This is a length 1 direct
path between <span class="math">\(a\)</span> and <span class="math">\(\overleftarrow{a_1}\)</span>.</p>
<p>We prove the base case for <span class="math">\(\overleftarrow{a_{-1}}, \overrightarrow{a_{-1}}\)</span> in
a similar manner.</p>
<p><strong>The inductive step</strong>: Assume that for some <span class="math">\(0 \leq t < l-1\)</span> that
<span class="caps">DPATH</span>[t]. Then <span class="caps">DPATH</span>[t+1].</p>
<p>Let <span class="math">\(a\)</span> be a node. First, we want to show that there is a direct path between
<span class="math">\(a\)</span> and <span class="math">\(\overleftarrow{a_{2^{t+1}}}\)</span>. Consider the node
<span class="math">\(\overleftarrow{a_{2^t}}\)</span>. We distinct between two cases:</p>
<p><img alt="\overleftarrow{a_{2^{t+1}}} and
\overleftarrow{a_{2^t}}" src="https://www.freedomlayer.org/articles/freedomlayer/vdht_cycles_rounds/images/locality_lemma.svg"></p>
<p><em>Case 1</em>: <span class="math">\(d(\overleftarrow{a_{2^t}},a+2^t) <
d(\overleftarrow{a_{2^{t+1}}},a+2^{t+1})\)</span>. Denote <span class="math">\(z :=
\overleftarrow{(\overleftarrow{a_{2^t}})_{2^t}}\)</span> We will show that in this case
<span class="math">\(z = \overleftarrow{a_{2^{t+1}}}\)</span>.</p>
<p>By definition <span class="math">\(z \in (\overleftarrow{a_{2^t}},\overleftarrow{a_{2^t}} + 2^t)\)</span>.</p>
<p>If <span class="math">\(z \in (\overleftarrow{a_{2^t}},\overleftarrow{a_{2^{t+1}}})\)</span> then
<span class="math">\(\overleftarrow{a_{2^{t+1}}}\)</span> is a better option for <span class="math">\(z\)</span>, which is a
contradiction to the stationary state of the algorithm.</p>
<p>If <span class="math">\(z \in (\overleftarrow{a_{2^{t+1}}},\overleftarrow{a_{2^t}} + 2^t)\)</span> then <span class="math">\(z\)</span>
is a better option for <span class="math">\(\overleftarrow{a_{2^{t+1}}}\)</span>, which is a contradiction
to the stationary state of the algorithm.</p>
<p>Therefore we have shown that <span class="math">\(z = \overleftarrow{a_{2^{t+1}}}\)</span>. By the inductive
assumption we have a direct path between <span class="math">\(a\)</span> and <span class="math">\(\overleftarrow{a_{2^t}}\)</span>, and
a direct path between <span class="math">\(\overleftarrow{a_{2^t}}\)</span> to <span class="math">\(z =
\overleftarrow{a_{2^{t+1}}}\)</span>. Those paths (together) are bounded in a size of
<span class="math">\(2^{t+1}\)</span>, hence their concatenation gives a direct path between <span class="math">\(a\)</span> and
<span class="math">\(\overleftarrow{a_{2^{t+1}}}\)</span>.</p>
<p><em>Case 2</em>: <span class="math">\(d(\overleftarrow{a_{2^t}},a+2^t) \geq
d(\overleftarrow{a_{2^{t+1}}},a+2^{t+1})\)</span>. Denote <span class="math">\(y :=
\overleftarrow{(\overleftarrow{a_{2^{t+1}}})_{-2^t}}\)</span>. We will show that in
this case <span class="math">\(y = \overleftarrow{a_{2^t}}\)</span></p>
<p>By definition <span class="math">\(y \in [a,\overleftarrow{a_{2^{t+1}}} - 2^t)\)</span> (If this range
collapses then <span class="math">\(y\)</span> is just <span class="math">\(a\)</span>).</p>
<p>If <span class="math">\(y \in [a,\overleftarrow{a_{2^t}})\)</span> then <span class="math">\(\overleftarrow{a_{2^t}}\)</span> is a
better option for <span class="math">\(y\)</span>, which is a contradiction to the stationary state of the algorithm.</p>
<p>If <span class="math">\(y \in (\overleftarrow{a_{2^t}},\overleftarrow{a_{2^{t+1}}} - 2^t)\)</span> then <span class="math">\(y\)</span>
is a better option for <span class="math">\(\overleftarrow{a_{2^t}}\)</span>, which is a contradiction
to the stationary state of the algorithm.</p>
<p>Therefore we have shown that <span class="math">\(y = \overleftarrow{a_{2^t}}\)</span>. By the inductive
assumption we have a direct path between <span class="math">\(a\)</span> and <span class="math">\(\overleftarrow{a_{2^t}}\)</span>, and
a direct path between <span class="math">\(\overleftarrow{a_{2^t}}\)</span> and
<span class="math">\(\overleftarrow{a_{2^{t+1}}}\)</span>.
Those two paths (together) are bounded in a size of <span class="math">\(2^{t+1}\)</span>, hence their
concatenation gives a direct path between <span class="math">\(a\)</span> and <span class="math">\(\overleftarrow{a_{2^{t+1}}}\)</span>.</p>
<p>The rest of <span class="caps">DPATH</span>[t+1] (We have 3 more direct paths to show) can be proved in a
similar way.</p>
<p><strong><span class="caps">QED</span></strong></p>
<p>Note that the inductive proof above makes use of all the powers of <span class="math">\(2\)</span> fingers.
It won’t work for the algorithm with just two near fingers and two distant
fingers mentioned above.</p>
<p><strong>Theorem (OneRound): </strong> Assume a stationary state of the IterF algorithm. Let
<span class="math">\(C\)</span> be a cycle of nodes. Then <span class="math">\(R(C) = 1\)</span>. (<span class="math">\(C\)</span> has only one round).</p>
<p><strong>Proof: </strong> Let <span class="math">\(C\)</span> be a cycle of nodes during a stationary state of the IterF
algorithm. Let <span class="math">\(x\)</span> be a node inside <span class="math">\(C\)</span>. By the Locality Lemma there is direct
path <span class="math">\(P\)</span> between <span class="math">\(x\)</span> and <span class="math">\(\overleftarrow{x_{2^{l-1}}}\)</span>. Also, there is a direct
path <span class="math">\(Q\)</span> between <span class="math">\(\overrightarrow{x_{-2^{l-1}}}\)</span> and <span class="math">\(x\)</span>.</p>
<p><img alt="The bath built from P and Q" src="https://www.freedomlayer.org/articles/freedomlayer/vdht_cycles_rounds/images/one_round_theorem.svg"></p>
<p>We can combine the above paths to a total path <span class="math">\(T := (P , Q)\)</span></p>
<p>It is possible that <span class="math">\(\overrightarrow{x_{-2^{l-1}}} =
\overleftarrow{x_{2^{l-1}}}\)</span>. In this case we omit one of them in the path
concatenation. If <span class="math">\(\overrightarrow{x_{-2^{l-1}}} \neq
\overleftarrow{x_{2^{l-1}}}\)</span> then <span class="math">\(h(\overrightarrow{x_{-2^{l-1}}}) =
\overleftarrow{x_{2^{l-1}}}\)</span>.</p>
<p>This is a path from <span class="math">\(a\)</span> to <span class="math">\(a\)</span> of length <span class="math">\(2^l\)</span>, which means that <span class="math">\(a\)</span> is part of
a cycle of only one round.</p>
<p><strong><span class="caps">QED</span></strong></p>
<h2>Code</h2>
<p>I wrote a short piece of code (Python3) that implements some of the things
discussed above. You can find it <a href="https://github.com/realcr/freedomlayer_code/blob/master/vdht_cycles_rounds/alg_state.py">here
[github]</a>.</p>
<p>Basically the code generates a random <span class="math">\(G(n,p)\)</span> graph with <span class="math">\(p=3\cdot\log(n)/n\)</span>
(So that the graph is connected with high probability), and then runs IterF
iterations on this graph until a stationary state is reached.
At this point the following properties are checked:</p>
<ul>
<li>
<p>Is <span class="math">\(h\)</span> injective on the resulting state?</p>
</li>
<li>
<p>Is every cycle on the resulting state of exactly one round?</p>
</li>
<li>
<p>Is there only one cycle of nodes in the resulting state?</p>
</li>
</ul>
<p>To run the code you will need python3 and the
<a href="https://networkx.github.io/">networkx</a> package. After you have those two just run:</p>
<div class="highlight"><pre><span></span>python3 alg_slate.py
</pre></div>
<p>This is an example of how the output looks like:</p>
<div class="highlight"><pre><span></span>next_node Injective: True | One round cycles: True | Is only one cycle: True
next_node Injective: True | One round cycles: True | Is only one cycle: True
next_node Injective: True | One round cycles: True | Is only one cycle: True
...
</pre></div>
<p>Every line in the output represents one run of the IterF algorithm over a
randomly generated graph. Each line tells about the properties of the stationary
state: Is the function <span class="math">\(h\)</span> injective, and if every cycle in the graph is of
one round, and is there only one cycle.</p>
<p>The code is not optimized, and thus somewhat slow. On my <span class="caps">PC</span> for <span class="math">\(n=2^9\)</span> it can
check one graph in about <span class="math">\(45\)</span> seconds.</p>
<p>I didn’t yet see a result with “Is only one cycle: False”, but that is probably
because of the choice of the randomized graphs. (Small <span class="math">\(G(n,p)\)</span> graphs). I think
it is possible to build an example for a graph that results in more than one
cycle of nodes after running the algorithm.</p>
<h2>Summary</h2>
<ul>
<li>
<p>If a cycle of nodes has exactly one round, then for each node <span class="math">\(x\)</span> in the
cycle, <span class="math">\(h(x)\)</span> is the node that is closest to <span class="math">\(x\)</span>, clockwise, between all the
nodes in the cycle. (In other words, a cycle that has only one round behaves
correctly with respect to our routing needs.)</p>
</li>
<li>
<p>The IterF algorithm (Every node <span class="math">\(x\)</span> maintains a set of fingers which are all
the powers of <span class="math">\(2\)</span> with respect to <span class="math">\(x\)</span>) always arrives at a stationary state
after a finite amount of iterations. A stationary sate always has a few disjoint
cycles of nodes, where each cycle is of exactly one round.</p>
</li>
</ul>
<h2>Further Questions</h2>
<ol>
<li>
<p>Are there other configurations of “fingers” for each node so that the
Locality Lemma is still true?</p>
</li>
<li>
<p>In which cases a run of IterF algorithm will leave us with just one
cycle? Could we improve IterF somehow, so that it will always end up with only
one cycle?</p>
</li>
</ol>
<h2>References</h2>
<p>The following articles discuss similar/relevant ideas:</p>
<ul>
<li>
<p><a href="http://research.microsoft.com/pubs/75325/virtualring.pdf">Virtual Ring Routing: Network Routing Inspired by
DHTs</a>, by Matthew
Caesar, Miguel Castro, Edmund B. Nightingale, Greg O’Shea and Antony Rowstron.</p>
</li>
<li>
<p><a href="https://gnunet.org/sites/default/files/fuhrmann06pushing.pdf">Pushing Chord into the Underlay: Scalable Routing for Hybrid
MANETs</a> by Thomas
Fuhrmann, Pengfei Di, Kendy Kutzner, and Curt Cramer</p>
</li>
<li>
<p><a href="https://en.wikipedia.org/wiki/Scalable_Source_Routing">Scalable Source Routing [Wikipedia]</a></p>
</li>
</ul>
<script type="text/javascript">if (!document.getElementById('mathjaxscript_pelican_#%@#$@#')) {
var align = "center",
indent = "0em",
linebreak = "false";
if (false) {
align = (screen.width < 768) ? "left" : align;
indent = (screen.width < 768) ? "0em" : indent;
linebreak = (screen.width < 768) ? 'true' : linebreak;
}
var mathjaxscript = document.createElement('script');
mathjaxscript.id = 'mathjaxscript_pelican_#%@#$@#';
mathjaxscript.type = 'text/javascript';
mathjaxscript.src = 'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.0/MathJax.js?config=TeX-AMS-MML_HTMLorMML';
mathjaxscript[(window.opera ? "innerHTML" : "text")] =
"MathJax.Hub.Config({" +
" config: ['MMLorHTML.js']," +
" TeX: { extensions: ['AMSmath.js','AMSsymbols.js','noErrors.js','noUndefined.js'], equationNumbers: { autoNumber: 'AMS' } }," +
" jax: ['input/TeX','input/MathML','output/HTML-CSS']," +
" extensions: ['tex2jax.js','mml2jax.js','MathMenu.js','MathZoom.js']," +
" displayAlign: '"+ align +"'," +
" displayIndent: '"+ indent +"'," +
" showMathMenu: true," +
" messageStyle: 'normal'," +
" tex2jax: { " +
" inlineMath: [ ['\\\\(','\\\\)'] ], " +
" displayMath: [ ['$$','$$'] ]," +
" processEscapes: true," +
" preview: 'TeX'," +
" }, " +
" 'HTML-CSS': { " +
" styles: { '.MathJax_Display, .MathJax .mo, .MathJax .mi, .MathJax .mn': {color: 'inherit ! important'} }," +
" linebreaks: { automatic: "+ linebreak +", width: '90% container' }," +
" }, " +
"}); " +
"if ('default' !== 'default') {" +
"MathJax.Hub.Register.StartupHook('HTML-CSS Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax['HTML-CSS'].FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"MathJax.Hub.Register.StartupHook('SVG Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax.SVG.FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"}";
(document.body || document.getElementsByTagName('head')[0]).appendChild(mathjaxscript);
}
</script>The Unified Challenge-Response: Secure Time inside the Mesh2014-12-31T18:49:00+00:002014-12-31T18:49:00+00:00realtag:www.freedomlayer.org,2014-12-31:/the-unified-challenge-response-secure-time-inside-the-mesh.html<!--
abs function.
-->
<div class="math">$$
\newcommand{\paren}[1]{\left({#1}\right)}
$$</div>
<h4>Abstract</h4>
<p>Let <span class="math">\(t\)</span> be a specific node in a mesh network. Assume that every node <span class="math">\(x\)</span>
knows the identity of <span class="math">\(t\)</span>, and wants to verify that <span class="math">\(t\)</span> was recently alive.
Also assume that <span class="math">\(t\)</span> has limited networking and computing power.</p>
<p>We review a few …</p><!--
abs function.
-->
<div class="math">$$
\newcommand{\paren}[1]{\left({#1}\right)}
$$</div>
<h4>Abstract</h4>
<p>Let <span class="math">\(t\)</span> be a specific node in a mesh network. Assume that every node <span class="math">\(x\)</span>
knows the identity of <span class="math">\(t\)</span>, and wants to verify that <span class="math">\(t\)</span> was recently alive.
Also assume that <span class="math">\(t\)</span> has limited networking and computing power.</p>
<p>We review a few algorithms to solve this problem: Simple Challenge-Response,
Digital Signature over a counter and Digital Signature over a time stamp. We
then discuss the flaws of each of those algorithms.</p>
<p>Finally we present the Unified Challenge-Response: A distributed algorithm that
simulates time stamp using Cryptographic Hash functions and periodic randomly
generated numbers. By Signing the simulated time stamp and broadcasting it, the
node <span class="math">\(t\)</span> can prove that he was recently alive.</p>
<p>The size of the broadcasted message by <span class="math">\(t\)</span> in the naive solution is
<span class="math">\(O(d\cdot s)\)</span>, where <span class="math">\(d\)</span> is the network diameter and <span class="math">\(s\)</span> is the amount of
immediate neighbours every node has. We mention two more improvements which
allow to reduce the size of the broadcasted message to <span class="math">\(O(d\log{s})\)</span> or to
<span class="math">\(O(d)\)</span>.</p>
<h4>Initial Ideas</h4>
<h5>The Challenge-Response</h5>
<p>Let <span class="math">\(x,y\)</span> be two nodes in distributed mesh network. Assume that there is a
path of nodes between <span class="math">\(x\)</span> and <span class="math">\(y\)</span>, known to <span class="math">\(x\)</span> and <span class="math">\(y\)</span>. By a path we
mean that <span class="math">\(x\)</span> is an immediate neighbour of some other node <span class="math">\(x_1\)</span>, which in
turn an immediate neighbour of some other node <span class="math">\(x_2\)</span> and so on, until we get
to some <span class="math">\(x_k\)</span>, which is an immediate neighbour of <span class="math">\(y\)</span>.</p>
<p><span class="math">\(x\)</span> and <span class="math">\(y\)</span> are not immediate neighbours, however using the path between
them they can communicate.</p>
<p>As the network changes, the path between <span class="math">\(x\)</span> and <span class="math">\(y\)</span> might not last
forever. (One of the nodes one the path might fail). Moreover, one of the nodes
<span class="math">\(x\)</span> or <span class="math">\(y\)</span> might fail at some point. In a case of failure, the nodes <span class="math">\(x\)</span>
and <span class="math">\(y\)</span> will want to be informed, so that they could find an alternative
path. (Or even just give up on the connection, in case one of them has failed).</p>
<p><img alt="A path of nodes between x and y" src="https://www.freedomlayer.org/articles/freedomlayer/unified_challenge_response/images/path_x_y.svg"></p>
<p>A path between of nodes between <span class="math">\(x\)</span> and <span class="math">\(y\)</span>. If any of the nodes along the
path fails, the communication between <span class="math">\(x\)</span> and <span class="math">\(y\)</span> will fail.</p>
<p>One idea would be that <span class="math">\(x\)</span> and <span class="math">\(y\)</span> will send periodic messages to
each other along the known path. Then if for example <span class="math">\(x\)</span> has not received a
message from <span class="math">\(y\)</span> for a long period of time, <span class="math">\(x\)</span> will assume that some node
along the path has failed, or <span class="math">\(y\)</span> itself has failed.</p>
<p>To make things a bit more secure we could use <a href="http://en.wikipedia.org/wiki/Public-key_cryptography">public key
cryptography</a>, with an
idea we call “challenge-response”. <span class="math">\(x\)</span> will send a periodic message to <span class="math">\(y\)</span>
(Along the path) which is a challenge to prove his identity. <span class="math">\(y\)</span> will respond
with a proof about his identity, sent along the path all the way to <span class="math">\(x\)</span>.
Using public key cryptography we hope that <span class="math">\(y\)</span><span class="quo">‘</span>s response could not be faked.</p>
<p>Note that the challenge sent by <span class="math">\(x\)</span> along the path to <span class="math">\(y\)</span> has to be a
different challenge every time. <span class="math">\(x\)</span> can not use the same challenge twice. The
next example explains why.</p>
<h5>The Replay Attack</h5>
<p>Consider the following case: <span class="math">\(x\)</span> and <span class="math">\(y\)</span> are connected by some path, and
<span class="math">\(e\)</span> is a node on the path between <span class="math">\(x\)</span> and <span class="math">\(y\)</span>. Assume that <span class="math">\(x\)</span> sends a
challenge message <span class="math">\(cm\)</span> along the path to <span class="math">\(y\)</span>. <span class="math">\(e\)</span> sees the messages
<span class="math">\(cm\)</span> and passes it to the next node on the path, until the message <span class="math">\(cm\)</span>
arrives at <span class="math">\(y\)</span>.</p>
<p><span class="math">\(y\)</span> creates a proof for his identity with respect to the sent challenge. We
call this proof <span class="math">\(pr\)</span>. <span class="math">\(y\)</span> then sends <span class="math">\(pr\)</span> along the path from <span class="math">\(y\)</span> to
<span class="math">\(x\)</span>. <span class="math">\(e\)</span> will see <span class="math">\(pr\)</span>, and pass it along to the next node on the path to
<span class="math">\(x\)</span>.</p>
<p>Assume that <span class="math">\(x\)</span> now sends the same challenge <span class="math">\(cm\)</span> along the path to <span class="math">\(y\)</span>.
Whenever the challenge <span class="math">\(cm\)</span> arrives at <span class="math">\(e\)</span>, <span class="math">\(e\)</span> will not pass it to the
next node on the path to <span class="math">\(y\)</span>. Instead, it will send back to <span class="math">\(x\)</span> the proof
message <span class="math">\(pr\)</span> that <span class="math">\(y\)</span> has generated in response to <span class="math">\(cm\)</span> the last time it
was sent.</p>
<p><span class="math">\(x\)</span> will receive the message <span class="math">\(pr\)</span> which is a valid proof, and so <span class="math">\(x\)</span> will
incorrectly assume that <span class="math">\(y\)</span> has sent it. At this point <span class="math">\(y\)</span> might be already
dead, but <span class="math">\(e\)</span> could keep sending convincing messages to <span class="math">\(x\)</span> about <span class="math">\(y\)</span><span class="quo">‘</span>s existence.</p>
<p><img alt="Replay Attack" src="https://www.freedomlayer.org/articles/freedomlayer/unified_challenge_response/images/replay_attack_x_y_e.svg"></p>
<p>In the picture: First (1), <span class="math">\(x\)</span> wants to verify that <span class="math">\(y\)</span> is alive. <span class="math">\(x\)</span> sends a
challenge message <span class="math">\(cm\)</span> to <span class="math">\(y\)</span> along the path between <span class="math">\(x\)</span> and <span class="math">\(y\)</span>, and
<span class="math">\(y\)</span> sends back a response, proving that he is alive. Next (2), <span class="math">\(x\)</span> sends
again the same challenge to <span class="math">\(y\)</span>, asking him to prove that he is alive. <span class="math">\(e\)</span>,
which is a node in the middle of the path, doesn’t forward <span class="math">\(x\)</span><span class="quo">‘</span>s challenge to
<span class="math">\(y\)</span>. Instead, it replays the last response sent by <span class="math">\(y\)</span>, fooling <span class="math">\(x\)</span> to
believe that <span class="math">\(y\)</span> is alive.</p>
<p>This kind of attack, performed by <span class="math">\(e\)</span>, is also known as <a href="http://en.wikipedia.org/wiki/Replay_attack">Replay
attack</a>. One way to avoid it is to
make sure that <span class="math">\(x\)</span> sends to <span class="math">\(y\)</span> a different challenge every time.</p>
<h5>One to Many Challenge-Response</h5>
<p>Assume that some node <span class="math">\(t\)</span> is a pretty important node in the network. It’s so
important that every node <span class="math">\(x\)</span> in the network keeps a path to <span class="math">\(t\)</span>. (An
example to this case could be found in <a href="https://www.freedomlayer.org/landmarks-navigation-by-random-walking.html">Landmarks Navigation by Random
Walking</a>,
where <span class="math">\(t\)</span> might be one of the landmarks).</p>
<p>Note that this case is a bit different from the one described above. It
asymmetric. Every node remembers a path to <span class="math">\(t\)</span>, but <span class="math">\(t\)</span> doesn’t remember a
path to each node in the network. <span class="math">\(t\)</span> doesn’t have the ability to do that, as
there are too many nodes in the network. Therefore we assume that every node
<span class="math">\(x\)</span> remembers a path to <span class="math">\(t\)</span>, however <span class="math">\(t\)</span> doesn’t know that path.</p>
<p>Every node <span class="math">\(x\)</span> that keeps a path to <span class="math">\(t\)</span> will want to know that <span class="math">\(t\)</span> is
still alive, and that his path to <span class="math">\(t\)</span> is still alive. </p>
<h5>Using simple challenge response</h5>
<p>One idea to do that would be to use many instances of the challenge-response
method that was introduced earlier. Every node <span class="math">\(x\)</span> will periodically send a
message with a <strong>unique</strong> challenge along the path to <span class="math">\(t\)</span>, then <span class="math">\(t\)</span> will
produce a proof and send a response message containing that proof somehow back
to <span class="math">\(x\)</span>.</p>
<p>The main issue with this idea is that <span class="math">\(t\)</span> will have to generate proofs for
challenges sent from every node in the network. As noted above, <span class="math">\(t\)</span> does not
have the ability to do that, because there are too many nodes in the network.</p>
<h5>Flooding a signed message</h5>
<p>Instead of using the challenge response mechanism, <span class="math">\(t\)</span> could just flood the
network with a message signed by <span class="math">\(t\)</span><span class="quo">‘</span>s key that says <span class="math">\(t\)</span> is alive.
The message will be something of the form: (“<span class="math">\(t\)</span> is alive”, Signature by <span class="math">\(t\)</span>).
(Note that the signature signs the content “<span class="math">\(t\)</span> is alive”).</p>
<p>Recall what flooding means: Every node <span class="math">\(x\)</span> that receives such a message and
thinks it is valid will pass the message to all of its immediate neighbours.</p>
<p>This method is vulnerable to the Replay Attack: Let <span class="math">\(e\)</span> be some node in the
network. <span class="math">\(e\)</span> receives all the messages flooded from <span class="math">\(t\)</span>. Therefore <span class="math">\(e\)</span>
will obtain the message (“<span class="math">\(t\)</span> is alive”, Signature) pretty soon.</p>
<p>Assume that after a while <span class="math">\(t\)</span> went offline. <span class="math">\(e\)</span> could then send the message
(“<span class="math">\(t\)</span> is alive”, Signature) again. This could trick all the nodes in the
network to believe that <span class="math">\(t\)</span> is still alive, although it is not online.</p>
<h5>Flooding a signed message with a counter</h5>
<p>To avoid the Replay attack introduced in the case of flooding signed message,
we could add something to <span class="math">\(t\)</span><span class="quo">‘</span>s message which will make it a bit different
every time.</p>
<p><span class="math">\(t\)</span> can include some kind of counter inside his messages, and
increase that counter by <span class="math">\(1\)</span> for every message sent. Then the message sent by
<span class="math">\(t\)</span> will look something like: (“<span class="math">\(t\)</span> is alive”, counter = 345, Signature by
<span class="math">\(t\)</span>), where the counter is increased for every message. (Note that the
signature signs the content and the counter together).</p>
<p>This time it is a bit harder for <span class="math">\(e\)</span> to replay <span class="math">\(t\)</span><span class="quo">‘</span>s messages.
That is because the messages have a counter, and every node in the network
remembers the last counter value sent by <span class="math">\(t\)</span>. If <span class="math">\(e\)</span> tries to replay some
message that was sent earlier by <span class="math">\(t\)</span>, the nodes in the network will recognize
that the message is not recent, and they will not accept it.</p>
<p>Explained in some more detail: Assume that <span class="math">\(t\)</span> sends a periodic signed
message with a counter, stating that he is alive. An example for a sequence of
messages sent from <span class="math">\(t\)</span> would be: </p>
<p>(“<span class="math">\(t\)</span> is alive”, counter = 103, Signature)</p>
<p>(“<span class="math">\(t\)</span> is alive”, counter = 104, Signature)</p>
<p>(“<span class="math">\(t\)</span> is alive”, counter = 105, Signature)</p>
<p>…</p>
<p>(“<span class="math">\(t\)</span> is alive”, counter = 171, Signature)</p>
<p>Recall that those messages have the purpose of proving to all the nodes in the
network that <span class="math">\(t\)</span> is still alive. </p>
<p>After the last message sent by <span class="math">\(t\)</span>, all the nodes in the network remember the
counter value <span class="math">\(171\)</span> for <span class="math">\(t\)</span><span class="quo">‘</span>s sequence.</p>
<p>Let <span class="math">\(e\)</span> be some node in the network that receives those messages from <span class="math">\(t\)</span>.
<span class="math">\(e\)</span> can try to send again one of <span class="math">\(t\)</span><span class="quo">‘</span>s old messages. For example, the
message (“<span class="math">\(t\)</span> is alive”,counter = 104,Signature). However, the nodes in the
network will not accept this message, because they know the current value of
the counter: 171. They know that the message with counter = 104 sent by <span class="math">\(e\)</span>
is not new.</p>
<p>However, if <span class="math">\(e\)</span> is really determined to replay <span class="math">\(t\)</span><span class="quo">‘</span>s messages, he can
wait longer. <span class="math">\(e\)</span> can wait until <span class="math">\(t\)</span> is offline, and then wait even more:
until all the nodes in the network have forgotten about <span class="math">\(t\)</span><span class="quo">‘</span>s existence.</p>
<p>Why do we think that <span class="math">\(t\)</span> will eventually be forgotten? For every node <span class="math">\(x\)</span>
in the network we assume that <span class="math">\(x\)</span> will forget <span class="math">\(t\)</span> at some point. That is
because it is not practical for <span class="math">\(x\)</span> to remember all the <span class="math">\(t\)</span>-s of the past
together with their counter number forever. <span class="math">\(x\)</span> will have to delete some old
information, to make space for new information.</p>
<p>At the point where the information about <span class="math">\(t\)</span> was long forgotten, <span class="math">\(e\)</span> will
be able to replay all of <span class="math">\(t\)</span><span class="quo">‘</span>s messages. <span class="math">\(e\)</span> could send all the messages
from the list above, one after the other, living the history of <span class="math">\(t\)</span> again.</p>
<h5>Using a Time Stamp</h5>
<p>We saw that adding a counter to the signed “alive” messages helps to deal with
replay attacks of the present, but it fails protecting against replay attacks
that take place in the far future.</p>
<p>To prove that a signed “alive” message was signed recently, we could add the
current time into the message, and sign it. A typical message would be:
(“<span class="math">\(t\)</span> is alive”, time=”Sat Dec 27 09:38:58 <span class="caps">UTC</span> 2014”,Signature), where the
signature is over the message’s content and the time. This method to overcome
replay attacks is sometimes called a
<a href="http://en.wikipedia.org/wiki/Timestamp">Timestamp</a>.</p>
<p>Whenever a node <span class="math">\(x\)</span> receives an “alive” message, he will verify the signature
and check the time stated inside the message. If the time stated is very long
ago (Maybe a few minutes ago), <span class="math">\(x\)</span> will discard the message. Otherwise, if
the time is recent and the signature is valid, <span class="math">\(x\)</span> will know that <span class="math">\(t\)</span> is alive.</p>
<p>In this case a node <span class="math">\(e\)</span> will not be able to replay an old “alive” message
sent by <span class="math">\(t\)</span>. If <span class="math">\(e\)</span> wants to generate a valid “alive” message, he needs to
have a recent time stamp inside the message, together with a signature of <span class="math">\(t\)</span>
over this message. Without cooperation from <span class="math">\(t\)</span>, this would be computationally
infeasible for <span class="math">\(e\)</span>.</p>
<p>There are some possible issues with this method, though.
First, we might not fully trust <span class="math">\(t\)</span>. Maybe it is important for us that <span class="math">\(t\)</span>
generates “alive” messages in real time, and doesn’t make many of those
messages ahead of time for later use.</p>
<p>With the Time-stamping mechanism introduced above, <span class="math">\(t\)</span> could generate many
messages for the future ahead of time, signing over future timestamps. <span class="math">\(t\)</span>
could then hand those future “alive” messages for someone else to deliver, in
the future.</p>
<p>The next issue is about time itself. <strong>How can we get a definition of time
inside a distributed mesh network?</strong> It seems to be a nontrivial question, both
because of relativistic time effects in distributed networks, and also because
we assume existence of some adversary players inside our network.</p>
<p>There are some solutions for this problem for the case of no adversaries or
weak adversaries. One thing you should probably check out is <a href="http://en.wikipedia.org/wiki/Lamport_timestamps">Lamport
Timestamps</a>.</p>
<p>I include here some naive approaches to get a mesh network definition of time,
and a short explanation of why they won’t fit for our case:</p>
<ul>
<li>
<p>We can set a special trusted node <span class="math">\(TS\)</span> (Time Server) that is responsible
for advertising time. <span class="math">\(TS\)</span> will flood the network with a message about his
current time every 1 second.</p>
<p>The message sent by <span class="math">\(TS\)</span> will be of the form (current_time, Signature by
<span class="math">\(TS\)</span>).</p>
<p>Then whenever <span class="math">\(t\)</span> wants to send an “alive” message, it will take the last
message sent by <span class="math">\(TS\)</span>, concatenate to that message the string “<span class="math">\(t\)</span> is alive”
and add a signature over everything. The end result will be:
(“<span class="math">\(t\)</span> is alive”,(current_time,Signature by <span class="math">\(TS\)</span>), Signature by <span class="math">\(t\)</span>).
(Note that using this method <span class="math">\(t\)</span> can not make “alive” messages ahead of time).</p>
<p>To make this work we need some trusted node with some globally trusted key.
This node has to live forever, or else the time in the network will “stop
moving. This is probably unacceptable for most distributed network settings.</p>
</li>
<li>
<p>Every node will have its own clock, and nodes will fix drift by asking each
other about the time. The time it takes a message to arrive can also be taken
into account.</p>
<p>This could probably work in a setting where we trust all the nodes to tell
the truth about time. However, in the case of adversarial nodes inside the
network, this will probably not work correctly.</p>
</li>
</ul>
<h4>The Unified Challenge-Response</h4>
<p>Let <span class="math">\(t\)</span> be a specific node inside a mesh network. We are looking for a way to
let <span class="math">\(t\)</span> broadcast a signed message to all the nodes in the network, so that
every node will know that the message sent was signed recently, and thus <span class="math">\(t\)</span>
was recently alive. </p>
<p>We mentioned earlier two main ideas:</p>
<ul>
<li>
<p>Challenge-Response: Every node <span class="math">\(x\)</span> sends a unique challenge to <span class="math">\(t\)</span>, and
<span class="math">\(t\)</span> responds with a proof. This proves to <span class="math">\(x\)</span> that <span class="math">\(t\)</span> is alive. We
couldn’t use this method because we can’t afford letting <span class="math">\(t\)</span> generate
proofs for challenges from all the nodes in the network.</p>
</li>
<li>
<p>Signing over some recent data: <span class="math">\(t\)</span> will sign some recent data, like the
current time, and broadcast it to all the nodes in the network. We couldn’t
use this method because we don’t have a reliable and secure way to generate
“recent data”, like time.</p>
</li>
</ul>
<p>The approach presented in this text is to somehow combine those two ideas
together. We will create something that we call the “Unified Challenge”: A
challenge that will be created together by all the nodes in the network, in a
distributed and efficient fashion. </p>
<p>This challenge will have somewhat similar role as the “current time” had in our
previous ideas. <span class="math">\(t\)</span> will sign the Unified challenge. Whenever a node
<span class="math">\(x\)</span> sees that unified challenge, he will be able to identify his own part
inside this challenge, thus <span class="math">\(x\)</span> will know that the message from <span class="math">\(t\)</span> is
recent, and hence <span class="math">\(t\)</span> was recently alive.</p>
<h5>Combining Challenges</h5>
<p>Recall how a simple
<a href="http://en.wikipedia.org/wiki/Challenge%E2%80%93response_authentication">challenge-response</a>
mechanism works. <span class="math">\(x\)</span> sends <span class="math">\(t\)</span> some random number <span class="math">\(r\)</span> (Sometimes called
a <a href="http://en.wikipedia.org/wiki/Cryptographic_nonce">nonce</a>). <span class="math">\(t\)</span>
concatenates some value of its own <span class="math">\(g\)</span> to <span class="math">\(r\)</span>, and then <span class="math">\(t\)</span> signs the
full message to obtain: (r,g,Signature by <span class="math">\(t\)</span>). (The signature is over (r,g)).</p>
<p>Side Note: If <span class="math">\(t\)</span> doesn’t concatenate its own value <span class="math">\(g\)</span> to the response, <span class="math">\(x\)</span>
could in fact ask <span class="math">\(t\)</span> to sign anything, which is not a good thing. Therefore
<span class="math">\(t\)</span> will add its own value <span class="math">\(g\)</span> before signing the message. You can think of
<span class="math">\(g\)</span> as some special constant.</p>
<p><img alt="Classic challenge response" src="https://www.freedomlayer.org/articles/freedomlayer/unified_challenge_response/images/classic_challenge_response.svg"></p>
<p>A <a href="http://en.wikipedia.org/wiki/Message_sequence_chart">Message sequence chart</a>
for a simple challenge response algorithm. <span class="math">\(x\)</span> sends <span class="math">\(t\)</span> a random number
<span class="math">\(r\)</span>. <span class="math">\(t\)</span> generates a random number <span class="math">\(g\)</span>, and then <span class="math">\(t\)</span> sends
(r,g,Signature by <span class="math">\(t\)</span>), where the signature is over (r,g). Finally <span class="math">\(x\)</span>
verifies the signature.</p>
<p>Observing this challenge-response process, we see that <span class="math">\(x\)</span><span class="quo">‘</span>s challenge could
be determined by some random number <span class="math">\(r\)</span>. In order to build a challenge that
combines are the challenges together, we should somehow combine all the numbers
<span class="math">\(r\)</span> from all the nodes in the network.</p>
<p>We use the following algorithm for each node <span class="math">\(x\)</span> in the network.
Assume that <span class="math">\(x\)</span> immediate neighbours are <span class="math">\(q_1,\dots,q_s\)</span>. Also assume that
<span class="math">\(H\)</span> is some agreed upon <a href="http://en.wikipedia.org/wiki/Cryptographic_hash_function">cryptographic hash
function</a>.</p>
<p>Initialize:</p>
<ul>
<li>
<p>Initialize current Iteration Number <span class="math">\(cit\)</span> to be <span class="math">\(0\)</span>
(We use the concept of “Iteration Number” instead of a clock)</p>
</li>
<li>
<p>For each neighbour <span class="math">\(q_j\)</span>:</p>
<ul>
<li>Set <span class="math">\(r_j\)</span> to be a random number.</li>
</ul>
</li>
</ul>
<p>Every few seconds: (Update Iteration)</p>
<ul>
<li>
<p>Increase the current iteration number <span class="math">\(cit\)</span> by <span class="math">\(1\)</span>. </p>
</li>
<li>
<p>Generate a random number <span class="math">\(r\)</span>.</p>
</li>
<li>
<p>Set <span class="math">\(w := (r,r_1,\dots,r_{s})\)</span>. </p>
<p>In other words, we construct a string that contains all the random
numbers, and includes the new random number <span class="math">\(r\)</span> that we have just generated.</p>
</li>
<li>
<p>for every <span class="math">\(j\)</span> between <span class="math">\(1\)</span> and <span class="math">\(s\)</span>:</p>
<ul>
<li>
<p>Remember <span class="math">\(w\)</span>. (It can be forgotten after long enough time).
In addition, remember the iteration number <span class="math">\(it\)</span> in which <span class="math">\(w\)</span> was generated.</p>
</li>
<li>
<p>Send <span class="math">\(H(w)\)</span> to <span class="math">\(q_j\)</span>.</p>
</li>
</ul>
</li>
</ul>
<p>On receipt of message <span class="math">\(m\)</span> from neighbour <span class="math">\(q_j\)</span>:</p>
<ul>
<li>Set <span class="math">\(r_j := m\)</span>.</li>
</ul>
<p><img alt="Combine challenges iteration" src="https://www.freedomlayer.org/articles/freedomlayer/unified_challenge_response/images/combine_challenges_iter.svg"></p>
<p>A basic iteration for “combining challenges”. The big blue circle is a
magnification of the node <span class="math">\(x\)</span> (So that you can see what <span class="math">\(x\)</span> does). The
nodes around: <span class="math">\(q_1,q_2,q_3,q_4,q_5\)</span> are <span class="math">\(x\)</span><span class="quo">‘</span>s immediate neighbours. The
picture shows the flow of values. <span class="math">\(r_1,\dots,r_5\)</span> are obtained from messages
from the neighbours <span class="math">\(q_1,\dots,q_5\)</span>. Then <span class="math">\(x\)</span> combines the values
<span class="math">\(r,r_1,\dots,r_5\)</span> to be <span class="math">\(w\)</span>. Finally, the hash operation <span class="math">\(H\)</span> is performed
over <span class="math">\(w\)</span>, and the result, <span class="math">\(H(w)\)</span>, is sent to all the immediate neighbours
<span class="math">\(q_1,\dots,q_5\)</span>.</p>
<p>Some Observations:</p>
<ul>
<li>
<p>The amount of traffic used: Every few seconds every node in the network sends
to all his neighbours data of constant size.</p>
</li>
<li>
<p>Memory usage: The node <span class="math">\(x\)</span> should remember the value
<span class="math">\(w\)</span> for every Update Iteration. We may assume that <span class="math">\(x\)</span> keeps track of the
last <span class="math">\(1024\)</span> values of <span class="math">\(w\)</span> generated. Whenever new values of <span class="math">\(w\)</span> are
generated, the old ones can be forgotten.</p>
</li>
<li>
<p>Assuming that the network is connected, after a while, every value <span class="math">\(w\)</span>
generated by a node <span class="math">\(x\)</span> depends on all the random numbers from all the nodes
in the network. To be more precise, for this to happen, we need at most <span class="math">\(d\)</span>
update iterations, where <span class="math">\(d\)</span> is the
<a href="http://en.wikipedia.org/wiki/Distance_%28graph_theory%29">diameter</a> of the
network. (<span class="math">\(d\)</span> is the biggest distance possible between two nodes in the network).</p>
</li>
</ul>
<p>Further explanation for the last observation: Consider a new calculated <span class="math">\(w\)</span>
value of some node <span class="math">\(x\)</span> at the network. <span class="math">\(w\)</span> depends on the newly generated
random value <span class="math">\(r\)</span> of <span class="math">\(x\)</span>, and on all the <span class="math">\(w\)</span> values from all of <span class="math">\(x\)</span><span class="quo">‘</span>s
neighbours. Consider some value <span class="math">\(w\)</span> that was accepted from a neighbour <span class="math">\(q\)</span>
of <span class="math">\(w\)</span>. <span class="math">\(w\)</span> depends on the random value <span class="math">\(r\)</span> generated by <span class="math">\(q\)</span>, and also
on all the <span class="math">\(w\)</span> values accepted from all the neighbours of <span class="math">\(q\)</span>. We can keep
going until we get that the original value <span class="math">\(w\)</span> we discussed depends on random
values <span class="math">\(r\)</span> generated by all the nodes in the network.</p>
<p><img alt="Combine challenges illustration" src="https://www.freedomlayer.org/articles/freedomlayer/unified_challenge_response/images/combine_challenges_illustration.svg"></p>
<p>In the picture: Illustration of the dependence of <span class="math">\(H(w)\)</span> on values <span class="math">\(r\)</span>
generated by all nodes in the network.</p>
<p>We denote by <span class="math">\(w_x\)</span> the value <span class="math">\(w\)</span> calculated by the node <span class="math">\(x\)</span>, and by
<span class="math">\(r_x\)</span> the random value <span class="math">\(r\)</span> generated by <span class="math">\(x\)</span>.</p>
<p>This is an example of a small graph of <span class="math">\(5\)</span> nodes, and the structure of
<span class="math">\(w_x\)</span> calculated by <span class="math">\(x\)</span>. On the first level, <span class="math">\(x\)</span> depends on <span class="math">\(r_x\)</span> and
on values from the neighbours <span class="math">\(y_2\)</span> and <span class="math">\(y_3\)</span>. On the next level, we can
see dependence on all the values <span class="math">\(r_x,r_{y_1},r_{y_2},r_{y_3},r_{y_4}\)</span>.</p>
<p>Side note: The value <span class="math">\(r_x\)</span> on the first level might be different from the
value <span class="math">\(r_x\)</span> on the last level. <span class="math">\(x\)</span> might have generated a new random value
<span class="math">\(r\)</span> by that time.</p>
<h5>Building the Response</h5>
<p>Let <span class="math">\(t\)</span> be a node in the network. <span class="math">\(t\)</span> wants to broadcast a proof of being
alive to all the nodes in the network.</p>
<p>This will be done as follows: <span class="math">\(t\)</span> will take its current value <span class="math">\(w_0 = w\)</span> and sign
it. The full message <span class="math">\(t\)</span> generates is: (“<span class="math">\(t\)</span> is alive”,<span class="math">\(w_0\)</span>, Signature by
<span class="math">\(t\)</span>). (The signature by <span class="math">\(t\)</span> is over the content and over the
value <span class="math">\(w\)</span>). This message will be sent to all of <span class="math">\(t\)</span><span class="quo">‘</span>s immediate neighbours. </p>
<p>In some sense, the value <span class="math">\(w_0\)</span> replaces the time stamp that we used in
previous solution proposals.</p>
<p>We describe here the verification algorithm for any node <span class="math">\(x\)</span> in the network
that receives such a message. </p>
<p>On receipt of a message of the form: (“<span class="math">\(t\)</span> is alive”,<span class="math">\(w_0\)</span>,Signature by <span class="math">\(t\)</span>,
<span class="math">\(w_1,\dots,w_p\)</span>):</p>
<p>(Note: This message has probably gone through a path of <span class="math">\(p\)</span> nodes before <span class="math">\(x\)</span>
has received it.)</p>
<ul>
<li>
<p>Verify <span class="math">\(t\)</span><span class="quo">‘</span>s signature. If it is invalid, discard the message.</p>
</li>
<li>
<p>Look for some record of <span class="math">\(w\)</span> in the past such that <span class="math">\(H(w)\)</span> is inside
<span class="math">\(w_p\)</span>. We call this record <span class="math">\(rc\)</span>. If there is no such record, discard this message.</p>
<p>Let <span class="math">\(it\)</span> be the iteration number in which <span class="math">\(w\)</span> was generated.</p>
</li>
<li>
<p>Verify that <span class="math">\(H(w_i)\)</span> is inside <span class="math">\(H(w_{i-1})\)</span> for every <span class="math">\(1 \leq i \leq p\)</span>.
If not, discard the message.</p>
</li>
<li>
<p>Accept the message. <span class="math">\(x\)</span> will conclude that <span class="math">\(t\)</span> was alive at time <span class="math">\(dt\)</span>.</p>
</li>
<li>
<p>Check if the record <span class="math">\(rc\)</span> contains <span class="math">\(t\)</span> (That means we already know that
<span class="math">\(t\)</span> was alive at iteration number <span class="math">\(it\)</span>). If <span class="math">\(rc\)</span> doesn’t contain <span class="math">\(t\)</span>:</p>
<p>(Note that we do this check to avoid loops in the flooding algorithm)</p>
<ul>
<li>
<p>Add <span class="math">\(t\)</span> to the record <span class="math">\(rc\)</span>.
This has the meaning of remembering <span class="math">\(t\)</span> was alive “after iteration
<span class="math">\(it\)</span><span class="dquo">“</span>.</p>
</li>
<li>
<p>Set <span class="math">\(w_{p+1} := w\)</span>. Send the following message to all the neighbours:
(“<span class="math">\(t\)</span> is alive”, <span class="math">\(w_0\)</span>, Signature by <span class="math">\(t\)</span>, <span class="math">\(w_1,\dots,w_p,w_{p+1}\)</span>)</p>
</li>
</ul>
</li>
</ul>
<p>We call the complete procedure described here (creating a combined challenge
from all the network nodes followed by verifying <span class="math">\(t\)</span><span class="quo">‘</span>s “alive” message) the
<strong>Unified Challenge-Response mechanism</strong>.</p>
<h5>The security of the Unified Challenge-Response</h5>
<p>We want to show that it is hard to forge a “proof of being alive” for node
<span class="math">\(t\)</span> without knowing <span class="math">\(t\)</span><span class="quo">‘</span>s private key.</p>
<p>Our assumptions over the adversary: The adversary is computationally bounded
(Can not forge a signature or invert a hash function), and has control over
some nodes inside the network. We call the nodes under the control of the
adversary the “corrupt” nodes. We call the rest of the nodes the “correct” nodes.</p>
<p>We assume that if we discard the set of corrupt nodes, the network
will still be connected. In a different formulation: We assume that there is a
path between every two correct nodes that doesn’t go through a corrupt node.</p>
<p>We first note that given the algorithm we described, <span class="math">\(t\)</span><span class="quo">‘</span>s messages will
arrive at all the correct nodes in the network. That is because for every
correct node <span class="math">\(x\)</span> there is a path between <span class="math">\(t\)</span> and <span class="math">\(x\)</span> that doesn’t go
through a corrupt node. (Note that we also assume here that <span class="math">\(t\)</span> is correct).</p>
<p><img alt="Correct path" src="https://www.freedomlayer.org/articles/freedomlayer/unified_challenge_response/images/correct_path.svg"></p>
<p>Illustration: A path of correct nodes between <span class="math">\(t\)</span> and <span class="math">\(x\)</span>. <span class="math">\(t\)</span><span class="quo">‘</span>s messages
will arrive at <span class="math">\(x\)</span>. The blue nodes are correct nodes. The purple nodes are
corrupt nodes.</p>
<p>Next, we want to show that if a node <span class="math">\(x\)</span> receives and accepts a message of
the form (“<span class="math">\(t\)</span> is alive”,<span class="math">\(w_0\)</span>, Signature by <span class="math">\(t\)</span>, <span class="math">\(w_1,\dots,w_p\)</span>), it
means that with high probability <span class="math">\(t\)</span> was alive after a specific iteration
<span class="math">\(it\)</span> of <span class="math">\(x\)</span>.</p>
<p>As we assumed the adversary is computationally bounded, he can not forge the
first part of the message: [“<span class="math">\(t\)</span> is alive”,<span class="math">\(w_0\)</span>,Signature by <span class="math">\(t\)</span>].
We conclude that the first part of the message was created by <span class="math">\(t\)</span>, and that
<span class="math">\(w_0\)</span> was the value of <span class="math">\(t\)</span><span class="quo">‘</span>s <span class="math">\(w\)</span> at the time of the message creation. In
other words: <span class="math">\(t\)</span> was alive after the value <span class="math">\(w_0\)</span> was computed.</p>
<p>Next, as the message is accepted by <span class="math">\(x\)</span>, it must be true that <span class="math">\(H(w_1)\)</span> is
inside <span class="math">\(w_0\)</span>. We assumed that with high probability the adversary can not
invert cryptographic hash functions, therefore the value <span class="math">\(H(w_1)\)</span> was
generated using the contents of <span class="math">\(w_1\)</span>.</p>
<p>Generally for every <span class="math">\(1 \leq i \leq p\)</span> we find that the value <span class="math">\(H(w_i)\)</span> was
generated relying on the contents of <span class="math">\(w_i\)</span>. As <span class="math">\(w_{i-1}\)</span> contains
<span class="math">\(H(w_i)\)</span>, we conclude that <span class="math">\(w_{i-1}\)</span> was computed only after <span class="math">\(w_i\)</span> was computed.</p>
<p>Therefore we find that <span class="math">\(t\)</span> was alive after the value <span class="math">\(w_p\)</span> was computed.
<span class="math">\(x\)</span> knows of a record <span class="math">\(rc\)</span> that contains <span class="math">\(w\)</span> such that <span class="math">\(H(w)\)</span> is inside
<span class="math">\(w_p\)</span>. That means that <span class="math">\(t\)</span> was alive after the record <span class="math">\(rc\)</span> was generated.
<span class="math">\(rc\)</span> was generated at iteration <span class="math">\(it\)</span> of <span class="math">\(x\)</span>. Therefore <span class="math">\(x\)</span>
knows that with high probability <span class="math">\(t\)</span> was alive after iteration <span class="math">\(it\)</span>.</p>
<h5>Shorter Messages</h5>
<p>If every node in the network has many immediate neighbours, the broadcasted
messages from <span class="math">\(t\)</span> of the form (“<span class="math">\(t\)</span> is alive”,<span class="math">\(w_0\)</span>, Signature by <span class="math">\(t\)</span>,
<span class="math">\(w_1,\dots,w_p\)</span>) could become large.</p>
<p>Assuming that the size of a signature or a hash value is <span class="math">\(b\)</span> and that every
node in the network has <span class="math">\(s\)</span> immediate neighbours, we get that <span class="math">\(w_i\)</span> is of
size <span class="math">\(b\cdot s\)</span> for every <span class="math">\(0 \leq i \leq p\)</span>. It means that a message from
<span class="math">\(t\)</span> could grow to a total size of <span class="math">\(d\cdot b \cdot s\)</span>, where <span class="math">\(d\)</span> is the
diameter of the network.</p>
<p>In our current algorithm every node <span class="math">\(x\)</span> combines all the values <span class="math">\(r_i\)</span> from his
neighbours together with his randomly generated value <span class="math">\(r\)</span> into one new value
<span class="math">\(H(w)\)</span>, where <span class="math">\(w = (r,r_1,\dots,r_s)\)</span>. <span class="math">\(w\)</span> is kept for later use, <strong>as
a proof</strong> that the value <span class="math">\(H(w)\)</span> was calculated after the values
<span class="math">\(r,r_1,\dots,r_s\)</span>.</p>
<p>We might be able to combine the values <span class="math">\(r,r_1,\dots,r_s\)</span> into a new value
<span class="math">\(g\)</span> using a different method. Without getting into too much details, I
present here two methods in which we could obtain much shorter proofs.</p>
<p>One method would be to use a <a href="http://en.wikipedia.org/wiki/Merkle_tree">Merkle Tree</a>.
Using this method <span class="math">\(g\)</span> will still have a constant size, however a proof that
<span class="math">\(g\)</span> depends on some <span class="math">\(r_i\)</span> will be of size <span class="math">\(c\cdot\log{s}\)</span> instead of
<span class="math">\(c\cdot s\)</span>. This could be done by using .</p>
<p>We could become even more efficient, and get a proof of size <span class="math">\(O(1)\)</span> using
constant size polynomial commitments. Read <a href="http://www.cypherpunks.ca/~iang/pubs/PolyCommit-AsiaCrypt.pdf">“Constant-Size Commitments to
Polynomials and Their
Applications”</a>
By A. Kate, G. Zaverucha, I. Goldberg for more information about this idea.</p>
<h4>Some thoughts about network time</h4>
<p>We managed to replace the time stamp we used in the beginning (See “Using a
Time Stamp” above) with a strange cryptographic object: A hash value <span class="math">\(H(w)\)</span> that
depends on random numbers generated by all the nodes in the network. </p>
<p><strong>That hash value, together with other data kept inside other nodes in the
network, are a proof that certain events happened in a certain order.</strong> This is a
proof that the value <span class="math">\(w\)</span> was calculated only after all the random numbers
were generated. </p>
<p>Note that We use real clocks in our solution only to invoke periodic
iterations. We need that every node in the network will have some machine that
ticks every once in a while. We call the time between two ticks “the iteration
period”. It different nodes might have different periods.</p>
<p>The period times of all nodes can be used to calculate how fast a generated
random number propagates through the network. Consider some random
number <span class="math">\(r\)</span> that was generated by a node <span class="math">\(x\)</span>. Assume that some node <span class="math">\(y\)</span> is
of distance <span class="math">\(D = dist(x,y)\)</span> from <span class="math">\(y\)</span>. Then the value <span class="math">\(w\)</span> of <span class="math">\(w\)</span> will
depend on the <span class="math">\(r\)</span> value generated by <span class="math">\(x\)</span> only after at least <span class="math">\(D\)</span>
iterations. (Maybe more, because the periodic ticks of all nodes in the network
are not synchronized).</p>
<p>Finally, note that a hash value <span class="math">\(H(w)\)</span> at some node <span class="math">\(x\)</span> is different from
the one at node <span class="math">\(y\)</span>. If we treat the value <span class="math">\(H(w)\)</span> as the “mesh network time”,
then in some sense every node in the network has a different view of time.</p>
<h4>Summary</h4>
<p>A node <span class="math">\(t\)</span> wants to broadcast a proof that he is alive to all the nodes in a
mesh network. We tried different methods to solve this problem: Flooding a
signed message, adding in a counter and using a time stamp of a global clock,
but we found some flaws in all of those methods.</p>
<p>We introduced a different method to simulate a time stamp using
cryptographic hash functions and periodically generated random numbers. A
signature of <span class="math">\(t\)</span> over that simulated time stamp serves as evidence for each
node <span class="math">\(x\)</span> in the network that <span class="math">\(t\)</span> was alive after a certain moment.</p>
<p>Finally, we showed some ideas to make our algorithm more efficient, by sending
shorter messages. We found that messages broadcasted by <span class="math">\(t\)</span> could be reduced
to be of size <span class="math">\(O(d\log{s})\)</span> using Merkle trees, and <span class="math">\(O(d)\)</span> using polynomial commitments.</p>
<script type="text/javascript">if (!document.getElementById('mathjaxscript_pelican_#%@#$@#')) {
var align = "center",
indent = "0em",
linebreak = "false";
if (false) {
align = (screen.width < 768) ? "left" : align;
indent = (screen.width < 768) ? "0em" : indent;
linebreak = (screen.width < 768) ? 'true' : linebreak;
}
var mathjaxscript = document.createElement('script');
mathjaxscript.id = 'mathjaxscript_pelican_#%@#$@#';
mathjaxscript.type = 'text/javascript';
mathjaxscript.src = 'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.0/MathJax.js?config=TeX-AMS-MML_HTMLorMML';
mathjaxscript[(window.opera ? "innerHTML" : "text")] =
"MathJax.Hub.Config({" +
" config: ['MMLorHTML.js']," +
" TeX: { extensions: ['AMSmath.js','AMSsymbols.js','noErrors.js','noUndefined.js'], equationNumbers: { autoNumber: 'AMS' } }," +
" jax: ['input/TeX','input/MathML','output/HTML-CSS']," +
" extensions: ['tex2jax.js','mml2jax.js','MathMenu.js','MathZoom.js']," +
" displayAlign: '"+ align +"'," +
" displayIndent: '"+ indent +"'," +
" showMathMenu: true," +
" messageStyle: 'normal'," +
" tex2jax: { " +
" inlineMath: [ ['\\\\(','\\\\)'] ], " +
" displayMath: [ ['$$','$$'] ]," +
" processEscapes: true," +
" preview: 'TeX'," +
" }, " +
" 'HTML-CSS': { " +
" styles: { '.MathJax_Display, .MathJax .mo, .MathJax .mi, .MathJax .mn': {color: 'inherit ! important'} }," +
" linebreaks: { automatic: "+ linebreak +", width: '90% container' }," +
" }, " +
"}); " +
"if ('default' !== 'default') {" +
"MathJax.Hub.Register.StartupHook('HTML-CSS Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax['HTML-CSS'].FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"MathJax.Hub.Register.StartupHook('SVG Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax.SVG.FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"}";
(document.body || document.getElementsByTagName('head')[0]).appendChild(mathjaxscript);
}
</script>Landmarks Navigation by Random Walking2014-12-23T19:19:00+00:002014-12-23T19:19:00+00:00realtag:www.freedomlayer.org,2014-12-23:/landmarks-navigation-by-random-walking.html<!--
abs function.
-->
<div class="math">$$
\newcommand{\abs}[1]{\left|{#1}\right|}
\newcommand{\paren}[1]{\left({#1}\right)}
$$</div>
<h4>Abstract</h4>
<p>We introduce the Network Coordinates: A dynamic method for assigning addresses
for nodes in a distributed mesh network. We show some properties of the Network
Coordinates. In particular, the Network Coordinates of adjacent nodes are similar.</p>
<p>We …</p><!--
abs function.
-->
<div class="math">$$
\newcommand{\abs}[1]{\left|{#1}\right|}
\newcommand{\paren}[1]{\left({#1}\right)}
$$</div>
<h4>Abstract</h4>
<p>We introduce the Network Coordinates: A dynamic method for assigning addresses
for nodes in a distributed mesh network. We show some properties of the Network
Coordinates. In particular, the Network Coordinates of adjacent nodes are similar.</p>
<p>We create a random walking algorithm that is based on the information from the
Network Coordinates. We use this algorithm to route messages from a source node
to a destination node in the network. </p>
<p>Next, we run some <a href="https://github.com/realcr/freedomlayer_code/tree/master/landmarks_navigation_rw">code experiments
[github]</a>
to check the performance of the random walk algorithm. Our results show that
this method of routing works for networks up to size of <span class="math">\(2^{15}\)</span> nodes. It
will probably not scale well for much larger networks.</p>
<p>We conjecture that there exists a more efficient decentralized algorithm that
relies on Network Coordinates to route messages in a mesh network.</p>
<h4>Motivation</h4>
<p>Given a mesh network, we want to be able to send a message between two
arbitrary nodes. We have already presented several possible solutions:
<a href="http://en.wikipedia.org/wiki/Flooding_%28computer_networking%29">flooding</a>,
<a href="https://www.freedomlayer.org/sqrtn-mesh-routing.html"><span class="math">\(\sqrt{n}\)</span> mesh
routing</a>, <a href="https://www.freedomlayer.org/experimenting-with-virtual-dht-routing.html">Virtual
<span class="caps">DHT</span>
routing</a>
and the <a href="https://www.freedomlayer.org/the-distributed-post-office-instant-hierarcy-for-mesh-networks.html">Distributed Post
Office</a>.</p>
<p>We can look at this question from another perspective: How can
a message inside the network navigate itself to a given destination?</p>
<h5>The Global Positioning System</h5>
<p>As human beings, the modern way of navigating oneself around the surface of the
planet is using the <a href="http://en.wikipedia.org/wiki/Global_Positioning_System">Global Positioning
System</a>, also known as
<span class="caps">GPS</span>. It works pretty well, and one might wonder why not apply it to messages
that travel inside mesh networks.</p>
<p>Some objections to use <span class="caps">GPS</span> to route messages in a mesh network might be as follows:</p>
<ul>
<li>
<p>Having a <span class="caps">GPS</span> receiver on a node might require extra hardware setup and cost.</p>
</li>
<li>
<p>Being dependent on <span class="caps">GPS</span> coordinates for message routing means being dependent
on a system of satellites on the sky. (If someone can take those satellites
down, he can take down our mesh network).</p>
</li>
<li>
<p><span class="caps">GPS</span> coordinates might not be so accurate, so we might have to complement the
<span class="caps">GPS</span> navigation with some other routing algorithm whenever the message gets
close enough to the destination.</p>
</li>
<li>
<p>The loss of Anonymity: If a node’s address is his <span class="caps">GPS</span> coordinates, it is easy
to find his location geographically. (There might be a way around this though).</p>
</li>
</ul>
<p>But even ignoring all the objections mentioned above, there is still one more
thing to consider. Navigating using <span class="caps">GPS</span> coordinates inside a mesh network has
an inherent theoretical flaw.</p>
<p>While the <span class="caps">GPS</span> system encapsulates a “good understanding” of physical location on
the Earth surface, generally it will not have a “good understanding” of the
mesh network layout. Put in other words: <strong>Two network nodes that are connected
by a direct network link might be very distant geographically</strong>, and thus have
very different <span class="caps">GPS</span> coordinates.</p>
<p>Work done by Jon Kleinberg, <a href="http://www.cs.cornell.edu/home/kleinber/swn.pdf">“The Small-World Phenomenon: An
Algorithmic Perspective”</a>,
puts this idea into a more formal argument. Kleinberg’s work hints that
navigation in a mesh using <span class="caps">GPS</span> coordinates can work efficiently only when the
network links are configured in a specific way. (If a link between every
two nodes <span class="math">\(x,y\)</span> exists with probability proportional to the inverse square
geographical distance between <span class="math">\(x\)</span> and <span class="math">\(y\)</span>).</p>
<p>In simpler words, routing algorithms that rely on <span class="caps">GPS</span> coordinates will not make
good use of long links between distant nodes, unless those links show up in some
specific probability. (Don’t let this make you skip the Small-World paper though :) )</p>
<p>We don’t know if this specific configuration happens in real mesh networks, but
it seems to be very specific, so we will assume that generally it doesn’t
happen for the rest of this text.</p>
<p><img alt="GPS Fails Long Links" src="https://www.freedomlayer.org/articles/freedomlayer/landmarks_navigation_rw/images/gps_fails_long_links.svg"></p>
<p>In the picture: <span class="math">\(x\)</span> routes a message to <span class="math">\(y\)</span> using <span class="caps">GPS</span> coordinates. As <span class="math">\(y\)</span>
is at the north, it seems reasonable to route the message to northern nodes.
However, it turns out that a faster path is possible if <span class="math">\(x\)</span> would send the
message to <span class="math">\(z\)</span> at the south, which has a direct link to the destination node
<span class="math">\(y\)</span>.</p>
<p>Using only knowledge from <span class="caps">GPS</span> coordinates, <span class="math">\(x\)</span> won’t be able to guess that.</p>
<p>Despite the flaws in <span class="caps">GPS</span> based routing, there is still something very
attractive about it. Routing a message in a mesh network is hard because every
node has only local knowledge of the network, which doesn’t give much
information about the network’s structure as a whole.
<span class="caps">GPS</span> gives us some approximation (although very incomplete) about
where we are in the network: It gives us some kind of “global information”,
something that could not be trivially achieved in a distributed mesh network.</p>
<p>We might be able to reproduce the effect we get from <span class="caps">GPS</span> using some other
means. We begin by understanding how <span class="caps">GPS</span> works. </p>
<h5>How Positioning Systems work?</h5>
<p>Described very roughly, the <span class="caps">GPS</span> system is based on a set of human made satellites
around earth. The satellites where positioned in a way that makes sure in every
time and place on earth, one can receive signal from a few of them. In order to
find out where you are on earth, you should receive signal from a few
satellites and find out how far you are from those satellites. Using the
obtained set of distances (And some information about current time and course
of those satellites), you can calculate where you are.</p>
<p><img alt="Satellites Distances" src="https://www.freedomlayer.org/articles/freedomlayer/landmarks_navigation_rw/images/satellites_distances.svg"></p>
<p>A very schematic illustration of finding location by data from a few satellites.</p>
<p>The navigators of old used <a href="http://en.wikipedia.org/wiki/Celestial_navigation">Celestial
Navigation</a> to find their
way using the stars. They looked at the stars and concluded information about
their location or direction. (Instead of measuring distance and time, they used
angular measurements).</p>
<p>An interesting thing to note is that in order to allow the navigation, both
the satellites and the stars don’t have to do much or be very smart. The
satellites transmit data continuously, and the stars always shine.</p>
<p>We might be able to make our own system of “satellites” or “stars” inside our
network, to allow messages find their way in the network.</p>
<h4>The Network Coordinates</h4>
<h5>The Landmarks</h5>
<p>In the end of <a href="https://www.freedomlayer.org/the-distributed-post-office-instant-hierarcy-for-mesh-networks.html">The Distributed Post
Office</a> article
we mentioned the idea of
Landmarks. Given a network of <span class="math">\(n\)</span> nodes, we choose a set of <span class="math">\(k\)</span> nodes to be
landmarks: <span class="math">\(\{l_1,l_2,\dots,l_k\}\)</span>. (Those are just regular nodes that were
given the “Landmark” title.) Every node <span class="math">\(x\)</span> in the network remembers
a shortest path to each of the landmarks.</p>
<p>Let’s assume that every node in the network has generated a key-pair of <a href="http://en.wikipedia.org/wiki/Public-key_cryptography">public
key and private key</a>. We
also assume that every node in the network knows a list of all the landmarks’
public keys. Every node that is a landmark would be able to prove it, by proving
that he owns his public key.</p>
<p>We currently do not discuss the method used to choose the set of landmarks.
Instead we will assume that it was chosen randomly somehow ahead of time. This
will be discussed in the future. (We showed one method to do this in “The
Distributed Post Office”: Choosing the nodes that maximize the value of some
cryptographic hash functions)</p>
<p>We also do not discuss the value we pick for <span class="math">\(k\)</span>. Currently you may assume
that we pick <span class="math">\(k\)</span> to be of some
<a href="http://en.wikipedia.org/wiki/Polylogarithmic_function">poly-logarithmic</a> size
with respect to <span class="math">\(n\)</span>.</p>
<h5>Maintaining contact with the Landmarks</h5>
<p>We begin by describing how each node <span class="math">\(x\)</span> in the network obtains a shortest
path to each of the landmarks: This is done by a few iterations where every
node in the network exchanges information with his immediate neighbours.</p>
<p>Find Shortest Paths (for node <span class="math">\(x\)</span>):</p>
<ul>
<li>
<p>Every few seconds:</p>
<ul>
<li>Send to all immediate neighbours the shortest path known to landmark
<span class="math">\(l_j\)</span> for each <span class="math">\(1 \leq j \leq k\)</span>.</li>
</ul>
</li>
<li>
<p>On receival of a set of paths:</p>
<ul>
<li>Update shortest paths to <span class="math">\(l_j\)</span> for each <span class="math">\(1 \leq j \leq k\)</span> accordingly.</li>
</ul>
</li>
</ul>
<p>As shown in “The Distributed Post Office”, this algorithm will calculate
shortest paths from every node <span class="math">\(x\)</span> to all the landmarks in at most <span class="math">\(d\)</span>
iterations, where <span class="math">\(d\)</span> is the
<a href="http://en.wikipedia.org/wiki/Distance_%28graph_theory%29">diameter</a> of the
network. Note that we don’t need to know the exact value of <span class="math">\(d\)</span>, because the
“Find Shortest Paths” algorithm keeps running as long as a node is in the network.</p>
<p>Next, Every node <span class="math">\(x\)</span> should verify periodically that his paths to the
landmarks are alive. Here is one way to do it: <span class="math">\(x\)</span> will periodically send a
message to each of the landmarks that asks them to prove their identity (The
message will be sent along the shortest path known to <span class="math">\(x\)</span>). In return, the
landmarks will respond with a proof that they are alive.</p>
<p><img alt="Landmarks Challenge Response" src="https://www.freedomlayer.org/articles/freedomlayer/landmarks_navigation_rw/images/landmarks_challenge_response.svg"></p>
<p><span class="math">\(x\)</span> sends a challenge message to one of the landmarks
(<span class="math">\(l_3\)</span>) through a shortest path. <span class="math">\(x\)</span> then waits for a proof of identity
from <span class="math">\(l_3\)</span>. If no answer arrives, <span class="math">\(x\)</span> discards his shortest path to <span class="math">\(l_3\)</span>
and finds another path.</p>
<p><span class="math">\(x\)</span> does the same for all of the other landmarks.</p>
<p>This method works, but it puts a lot of load on the landmarks. From the point of
view of one landmark <span class="math">\(l_j\)</span>, <span class="math">\(l_j\)</span> has to send proofs to all the nodes in the
network every period of time. Forget about this problem for a while. We will show how
to resolve it in the future.</p>
<p>We get that at all times, every node <span class="math">\(x\)</span> has a verified shortest path to each
of the landmarks in the network. <span class="math">\(x\)</span> can calculate his network distance from
each of the landmarks (It’s the length of the shortest path to those landmarks).
We denote the list of distances of <span class="math">\(x\)</span> from each of the landmarks by</p>
<div class="math">$$Coord(x) = (c_x^1,\dots,c_x^k) := (dist(x,l_1),\dots,dist(x,l_k))$$</div>
<p>and we call it <strong><span class="math">\(x\)</span><span class="quo">‘</span>s network coordinate</strong>.</p>
<p><img alt="Network Coordinates" src="https://www.freedomlayer.org/articles/freedomlayer/landmarks_navigation_rw/images/network_coordinate.svg"></p>
<p>The picture illustrates network coordinates.
<span class="math">\(x\)</span> maintains a shorest path to each of the landmarks
<span class="math">\(l_1,l_2,l_3,l_4,l_5\)</span>. The coordinate <span class="math">\(c_x^j = dist(x,l_j)\)</span> is the length
of the shortest path from <span class="math">\(x\)</span> to <span class="math">\(l_j\)</span> for some <span class="math">\(1 \leq j \leq 5\)</span>.</p>
<h5>Properties of the Network Coordinates</h5>
<p>By our construction we get a network coordinate for every node in the network.
We note here a few properties of the network coordinates.</p>
<h6>Coordinates of landmarks</h6>
<p>The landmark <span class="math">\(l_j\)</span> will have a coordinate of the form:</p>
<div class="math">$$(dist(l_j,l_1),\dots,dist(l_{j-1},l_j),0,dist(l_{j+1},l_j),\dots,dist(l_j,l_k)$$</div>
<p>That is because <span class="math">\(dist(l_j,l_j) = 0\)</span> (The network distance of <span class="math">\(l_j\)</span> from himself is 0$. </p>
<p>In addition, note that if a node <span class="math">\(x\)</span> has some <span class="math">\(0\)</span> coordinate, than <span class="math">\(x\)</span> must be a
landmark. (If the <span class="math">\(i\)</span><span class="quo">‘</span>s coordinate is <span class="math">\(0\)</span>, then this is the <span class="math">\(i\)</span><span class="quo">‘</span>s landmark.)</p>
<h6>Continuity</h6>
<p>For every two immediate neighbours <span class="math">\(x\)</span> and <span class="math">\(y\)</span> in the network, we get that
<span class="math">\(\left|c_x^j - c_y^j\right| \leq 1\)</span> for every <span class="math">\(1 \leq j \leq k\)</span>. (Whenever
we move to an adjacent node in the network, we change every entry in the network
coordinate by at most 1)</p>
<p>We can prove this <a href="http://en.wikipedia.org/wiki/Proof_by_contradiction">by
contradiction</a>. Assume
(Without loss of generality) that for some <span class="math">\(j\)</span> we get that <span class="math">\(c_x^j - c_y^j
\geq 2\)</span>. This means that <span class="math">\(dist(x,l_j) \geq 2 + dist(y,l_j)\)</span>: The shortest
path between <span class="math">\(x\)</span> and <span class="math">\(l_j\)</span> is two hops longer than the shortest path between
<span class="math">\(y\)</span> and <span class="math">\(l_j\)</span>. </p>
<p><img alt="Coord continuity" src="https://www.freedomlayer.org/articles/freedomlayer/landmarks_navigation_rw/images/coord_continuity.svg"></p>
<p>Illustration of <span class="math">\(c_x^j\)</span> and <span class="math">\(c_y^j\)</span>.</p>
<p>But <span class="math">\(x\)</span> and <span class="math">\(y\)</span> are neighbours, so <span class="math">\(x\)</span> could instead
use the following path to <span class="math">\(l_j\)</span>: First move to <span class="math">\(y\)</span>, and then continue using
the shortest path from <span class="math">\(y\)</span> to <span class="math">\(l_j\)</span>, which is of length <span class="math">\(dist(y,l_j)\)</span>. As
a result we get a path from <span class="math">\(x\)</span> to <span class="math">\(l_j\)</span> which is of total length
<span class="math">\(dist(y,l_j) + 1\)</span>, <span class="math">\(1\)</span> hop shorter than the path we assumed is the shortest
from <span class="math">\(x\)</span> to <span class="math">\(l_j\)</span>. This is a contradiction.</p>
<p>Therefore we conclude that for every two immediate neighbours <span class="math">\(x,y\)</span> in the
network, <span class="math">\(\left|c_x^j - c_y^j\right| \leq 1\)</span>. We call this property <strong>the
continuity of network coordinates.</strong></p>
<h6>The triangle inequalities</h6>
<p>The function <span class="math">\(dist(x,y)\)</span> calculates the length of a shortest path between two
nodes <span class="math">\(x,y\)</span> in the network. It satisfies a few properties for every <span class="math">\(x,y,z\)</span>
nodes in the network:</p>
<ul>
<li><span class="math">\(dist(x,y) \geq 0\)</span> (Non negativity)</li>
<li><span class="math">\(dist(x,y) = 0\)</span> if and only if <span class="math">\(x=y\)</span>.</li>
<li><span class="math">\(dist(x,y) = dist(y,x)\)</span>. (Symmetry)</li>
<li><span class="math">\(dist(x,z) \leq dist(x,y) + dist(y,z)\)</span> (The Triangle inequality).</li>
</ul>
<p>Make sure that you understand why the first three are correct.</p>
<p>We now show why the fourth one: The Triangle inequality, is correct.
The length of the shortest path between <span class="math">\(x\)</span> and <span class="math">\(y\)</span> is <span class="math">\(d(x,y)\)</span>. The
length of the shortest path between <span class="math">\(y\)</span> and <span class="math">\(z\)</span> is <span class="math">\(d(y,z)\)</span>. We can always
concatenate those two paths to obtain a path between <span class="math">\(x\)</span> and <span class="math">\(z\)</span> that is of
length <span class="math">\(d(x,y) + d(y,z)\)</span>. Therefore the shortest path between <span class="math">\(x\)</span> and <span class="math">\(z\)</span>
is at most of length <span class="math">\(d(x,y) + d(y,z)\)</span>.</p>
<p>The four properties of <span class="math">\(dist\)</span> mentioned above could be summarized by saying
that <span class="math">\(dist\)</span> is a
<a href="http://en.wikipedia.org/wiki/Metric_%28mathematics%29">metric</a>.</p>
<p>We use <span class="math">\(dist\)</span><span class="quo">‘</span>s properties to conclude the following inequality about the network
coordinates: For every node <span class="math">\(x\)</span> and <span class="math">\(1 \leq i,j \leq k\)</span>:</p>
<div class="math">$$\left|c_x^i - c_x^j\right| \leq dist(l_i,l_j) \leq c_x^i + c_x^j$$</div>
<p>In particular, this means that not every coordinate is valid. Some combinations
of numbers can never form a network coordinate. (With respect to a given set of
landmarks and distances between landmarks).</p>
<p><img alt="Coordinates triangle inequality" src="https://www.freedomlayer.org/articles/freedomlayer/landmarks_navigation_rw/images/coord_triangle_ineq.svg"></p>
<p>Illustration of the triangle inequality for network coordinates.</p>
<p>Proof: Recall that <span class="math">\(c_x^i = dist(x,l_i)\)</span>, and <span class="math">\(c_x^j = dist(x,l_j)\)</span>.
By the triangle inequality property of <span class="math">\(dist\)</span> we get that <span class="math">\(dist(l_i,l_j) \leq
dist(x,l_i) + dist(x,l_j) = c_x^i + c_x^j\)</span>, which proves the right part of the inequality.</p>
<p>For the left part of the inequality, note that by the triangle inequality
property of <span class="math">\(dist\)</span> we get that <span class="math">\(dist(l_i,l_j) + dist(l_j,x) \geq
dist(l_i,x)\)</span>. Therefore <span class="math">\(dist(l_i,l_j) \geq dist(l_i,x) - dist(l_j,x)\)</span>. In
the symmetric direction we will get that <span class="math">\(dist(l_i,l_j) \geq dist(l_j,x) -
dist(l_i,x)\)</span>. We conclude that <span class="math">\(dist(l_i,l_j) \geq \left|dist(l_i,x) -
dist(l_j,x)\right|\)</span>, which proves the left side of the inequality.</p>
<h6>The Uniqueness question</h6>
<p>Are network coordinates unique? </p>
<p>Assume that <span class="math">\(x\)</span> and <span class="math">\(y\)</span> are two different nodes in the network. Is it
possible that <span class="math">\(Coord(x) = Coord(y)\)</span>?</p>
<p>The answer is yes. It is possible to construct a network where <span class="math">\(x,y\)</span> are two
different nodes, but <span class="math">\(Coord(x) = Coord(y)\)</span>.</p>
<p><img alt="Non unique coordinates" src="https://www.freedomlayer.org/articles/freedomlayer/landmarks_navigation_rw/images/nonunique_coords.svg"></p>
<p>Example of different nodes having the same network
coordinate. Blue dots are regular nodes. Green dots are landmarks. Note that
the network coordinate (2,4,1) shows up twice.</p>
<p>However, this case is not the common case. It turns out that in many networks
where <span class="math">\(\paren{log{n}}^2\)</span> landmarks are chosen randomly, nodes’ coordinates are
unique. We observe this phenomenon later in our experiments. Note that at this
point we don’t know of a formal explanation in favor of this phenomenon.</p>
<p>As an exercise, think about the case of two dimensional grid. How many landmarks
are needed to make sure that network coordinates are unique?</p>
<h4>Navigation by Random Walking</h4>
<p>By now we managed to set Network Coordinates for every node. We have seen that
those coordinates, as opposed to <span class="caps">GPS</span> Coordinates, have a “real understanding” of
the network’s structure. (See for example the continuity property, which the <span class="caps">GPS</span>
coordinates lack). In particular, our new construction is not limited by the
<a href="http://www.cs.cornell.edu/home/kleinber/swn.pdf">small-world impossibility
results</a> result.</p>
<p>That said, it is still not obvious how to use the Network Coordinates to route
messages in the network.</p>
<p>We present here a relatively naive message routing method that gives nice
results and demonstrates the usefulness of Network Coordinates. It is based on
<a href="http://en.wikipedia.org/wiki/Random_walk">random walking</a>. </p>
<p>To rephrase our intentions: Given two arbitrary nodes <span class="math">\(x\)</span> and <span class="math">\(y\)</span>, <span class="math">\(x\)</span>
wants to send a message to <span class="math">\(y\)</span>. For this purpose <span class="math">\(x\)</span> is given <span class="math">\(y\)</span><span class="quo">‘</span>s
Network Coordinate <span class="math">\(Coord(y)\)</span>. We show an algorithm for routing a message from
<span class="math">\(x\)</span> to <span class="math">\(y\)</span>, given the knowledge of <span class="math">\(Coord(y)\)</span>.</p>
<h5>The Observable distance</h5>
<p>Assume that a message is to be routed from <span class="math">\(x\)</span> to <span class="math">\(y\)</span>. Also assume that the
current position of the message in the network is some node <span class="math">\(q\)</span>. If we could
know <span class="math">\(dist(q,y)\)</span> for any intermediate <span class="math">\(q\)</span>, we could easily route the message to
its destination <span class="math">\(y\)</span> using the following greedy algorithm: In every step pass
the message to the neighbour node with the smallest distance to the destination
<span class="math">\(y\)</span>.</p>
<p>However, we probably don’t have enough information to calculate <span class="math">\(dist(q,y)\)</span> for
every <span class="math">\(q\)</span> in the network. Instead, we might be able to calculate some
approximation to the <span class="math">\(dist\)</span> function.</p>
<p>Given two nodes <span class="math">\(a,b\)</span> in the network and a value <span class="math">\(1 \leq j \leq k\)</span>, it is
true that:</p>
<div class="math">$$ \abs{dist(a,l_j) - dist(b,l_j)} \leq dist(a,b) \leq dist(a,l_j) + dist(b,l_j)$$</div>
<p>This can be concluded from the triangle inequality for the metric <span class="math">\(dist\)</span>, as
we demonstrated earlier.</p>
<p>Using the information from distances to all the landmarks we can further
conclude that:</p>
<div class="math">$$ \max_{1 \leq j \leq k}\abs{dist(a,l_j) - dist(b,l_j)} \leq dist(a,b)
\leq \min_{1 \leq j \leq k}\paren{dist(a,l_j) + dist(b,l_j)}$$</div>
<p>We are more interested in the left part at this point. We call it the
Observable distance between the nodes <span class="math">\(a\)</span> and <span class="math">\(b\)</span>. We denote it by:</p>
<div class="math">$$odist(a,b) := \max_{1 \leq j \leq k}\abs{dist(a,l_j) - dist(b_l,j)}$$</div>
<p>As we have just seen, <span class="math">\(odist(a,b) \leq dist(a,b)\)</span> for every two nodes
<span class="math">\(a,b\)</span>. </p>
<p>We also emphasize that <span class="math">\(odist(a,b)\)</span> could be calculated just by
knowing the network coordinates of <span class="math">\(a,b\)</span>.</p>
<div class="math">$$odist(a,b) := \max_{1 \leq j \leq k}\abs{dist(a,l_j) -
dist(b,l_j)} = \max_{1 \leq j \leq k}\abs{c_a^j - c_b^j}$$</div>
<p>Next, we prove some properties of <span class="math">\(odist\)</span>. (Basically we are going to show
that <span class="math">\(odist\)</span> is some weaker notion of distance function. You can skip this if
you don’t have the mood for the math. The rest of the text doesn’t rely on
those results).</p>
<h6>odist is a Pseudometric</h6>
<p>we prove that <span class="math">\(odist\)</span> satisfies the following for every three nodes
<span class="math">\(x,y,z\)</span>:</p>
<ul>
<li><span class="math">\(odist(x,y) \geq 0\)</span> (Non negativity)</li>
<li><span class="math">\(odist(x,x) = 0\)</span> for every node x.</li>
<li><span class="math">\(odist(x,y) = odist(y,x)\)</span>. (Symmetry)</li>
<li><span class="math">\(odist(x,z) \leq odist(x,y) + odist(y,z)\)</span> (The Triangle inequality).</li>
</ul>
<p>This means that <span class="math">\(odist\)</span> is a <a href="http://en.wikipedia.org/wiki/Pseudometric_space">Pseudometric</a>.
Note that <span class="math">\(odist\)</span> is not a metric because as we have seen before, it is
possible that <span class="math">\(odist(x,y) = 0\)</span> but <span class="math">\(x \neq y\)</span>. (Compare the conditions we
presented for <span class="math">\(dist\)</span> with those we presented here for <span class="math">\(odist\)</span>).</p>
<p>Our plan to prove this is to first show that each of the functions
<span class="math">\(odist_j(a,b) := \abs{dist(a,l_j) - dist(b,l_j)}\)</span> (For <span class="math">\(1 \leq j \leq
k\)</span> are pseudometrics. </p>
<p>Then we use the fact that a maximum of pseudometrics is also a pseudometric
(See a hint <a href="http://math.stackexchange.com/questions/1020808/maximum-of-two-metrics-is-a-metric">here [Mathematics
StackExchange]</a>),
which will prove that <span class="math">\(odist\)</span> is a pseudo metric.</p>
<p>For some <span class="math">\(1 \leq j \leq k\)</span>, we get that <span class="math">\(odist_j(x,y) = \abs{dist(x,l_j) -
dist(y,l_j)} \geq 0\)</span>, and also that <span class="math">\(odist_j(x,x) = \abs{dist(x,l_j) -
dist(x,l_j)} = 0\)</span>. To show the Symmetry property, we have that <span class="math">\(odist_j(x,y)
= \abs{dist(x,l_j) - dist(y,l_j)} = \abs{dist(y,l_j) - dist(x,l_j)} =
odist_j(y,x)\)</span>.</p>
<p>We are left to show the triangle inequality for <span class="math">\(odist_j\)</span>.
</p>
<div class="math">$$odist_j(x,z) = \abs{c_x^j - c_z^j}$$</div>
<div class="math">$$odist_j(x,y) + odist_j(y,z) = \abs{c_x^j - c_y^j} + \abs{c_y^j -
c_z^j}$$</div>
<p>Assigning <span class="math">\(x' = c_x^j, y' = c_y^j, z' = c_z^j\)</span> we are left to prove:</p>
<div class="math">$$odist_j(x,z) = \abs{x' - z'} \leq \abs{x' - y'} + \abs{y' - z'} =
odist_j(x,y) + odist_j(y,z)$$</div>
<p>But <span class="math">\(\abs{x' - z'} = \abs{(x' - y') + (y' - z')} \leq \abs{x' - y'} +
\abs{y' - z'}\)</span>. So we proved the triangle inequality for <span class="math">\(odist_j\)</span>.</p>
<p>As <span class="math">\(odist(x,y) = \max_{1 \leq j \leq k}{odist_j(x,y)}\)</span>, we conclude that
<span class="math">\(odist\)</span> is a pseudometric.</p>
<h6>odist with $A^*$</h6>
<p>I deviate from our current track for a moment to show you another interesting
feature of the <span class="math">\(odist\)</span> function.</p>
<p><a href="http://en.wikipedia.org/wiki/A*_search_algorithm"><span class="math">\(A^*\)</span></a> (Pronounced A Star) is
a graph searching algorithm. Shortly described, it is an extension of Dijkstra
algorithm that uses heuristics to get better performance. A great explanation
of <span class="math">\(A^*\)</span> and graphs path-finding could be found
<a href="http://www.redblobgames.com/pathfinding/a-star/introduction.html">here</a>.</p>
<p>Please make sure you understand <span class="math">\(A^*\)</span> to some extent before you continue reading
this section.</p>
<p>While I don’t know of a way to use the <span class="math">\(A^*\)</span> algorithm in a distributed manner
(which is also secure), it is still interesting to find out what is possible
with <span class="math">\(A^*\)</span>, given the <span class="math">\(odist\)</span> function.</p>
<p>To use <span class="math">\(A^*\)</span>, we need some <a href="http://en.wikipedia.org/wiki/Admissible_heuristic">admissible heuristic
function</a> <span class="math">\(h\)</span>. That means,
if we try to find a path from node <span class="math">\(x\)</span> to node <span class="math">\(y\)</span>, we want some function
<span class="math">\(h\)</span> such that <span class="math">\(h(q) \leq dist(q,y)\)</span> for every node <span class="math">\(q\)</span>.</p>
<p>It is also known that if we manage to find such function <span class="math">\(h\)</span> that is also
<a href="http://en.wikipedia.org/wiki/Consistent_heuristic">consistent</a>, we will be
able to invoke <span class="math">\(A^*\)</span> more efficiently. A consistent heuristic is a function
<span class="math">\(h\)</span> that satisfies <span class="math">\(h(a) \leq dist(a,b) + h(b)\)</span> for every two nodes
<span class="math">\(a,b\)</span>.</p>
<p>Given a destination node <span class="math">\(y\)</span>, we can define <span class="math">\(h(q) := odist(q,y)\)</span> to be our
heuristic function. We show that <span class="math">\(h\)</span> is an admissible consistent
heuristic function.</p>
<p><span class="math">\(h\)</span> is admissible because <span class="math">\(h(q) = odist(q,y) \leq dist(q,y)\)</span>.
To be prove that <span class="math">\(h\)</span> is consistent, we first note that by <span class="math">\(odist\)</span><span class="quo">‘</span>s
triangle inequality property, we get:</p>
<div class="math">$$h(a) - h(b) = odist(a,y) - odist(b,y) \leq odist(a,b) \leq dist(a,b)$$</div>
<p>Therefore <span class="math">\(h(a) \leq dist(a,b) + h(b)\)</span>, which means that <span class="math">\(h\)</span> is consistent.</p>
<p>We showed that <span class="math">\(h(q) := odist(q,y)\)</span> is an admissible consistent heuristic.</p>
<h5>Random walking</h5>
<p>We want to send a message from some node <span class="math">\(x\)</span> to another node <span class="math">\(y\)</span> in the
network. We plan to let the message wander somewhat randomly around the
network, hoping that it will eventually find <span class="math">\(y\)</span>.
Given that at the current step the message is located at some node <span class="math">\(q\)</span>, in
the next step the message will be passed randomly to one of <span class="math">\(q\)</span><span class="quo">‘</span>s neighbours.</p>
<p>Without further information about the location of <span class="math">\(y\)</span> and the current
location of the message inside the network, we couldn’t expect this method to
yield any promising results. (The message might wander around and go through
almost all nodes before finding <span class="math">\(y\)</span>). However, using the <span class="math">\(odist\)</span> function
we could get some approximation for how close we are to the destination node
<span class="math">\(y\)</span>.</p>
<p>Assume that a message that is being sent from a source node <span class="math">\(x\)</span> to a destination node
<span class="math">\(y\)</span>. Also assume that currently the message is located at some node <span class="math">\(q\)</span> in
the network. We describe here the algorithm used to pass the message to one of
<span class="math">\(q\)</span><span class="quo">‘</span>s neighbours:</p>
<ul>
<li>
<p>For each <span class="math">\(r\)</span> which is a neighbour of <span class="math">\(q\)</span>:</p>
<ul>
<li>Calculate the weight <span class="math">\(w_r = {\beta}^{odist(q,y) - odist(r,y)}\)</span></li>
</ul>
</li>
<li>
<p>Calculate <span class="math">\(w = \sum_{r}{w_r}\)</span>, the sum of all weights.</p>
</li>
<li>
<p>Pass the message to a random neighbour of <span class="math">\(q\)</span>, where every neighbour <span class="math">\(r\)</span>
is chosen with probability <span class="math">\(\frac{w_r}{w}\)</span>.</p>
</li>
</ul>
<p>Some notes about neighbour’s probabilities:</p>
<ul>
<li>
<p>We pick <span class="math">\(\beta\)</span> to be some number bigger than 1. You can imagine that <span class="math">\(\beta
= 2\)</span>. In our experiments later we pick <span class="math">\(\beta = 22\)</span>.</p>
</li>
<li>
<p>The more a neighbour of <span class="math">\(q\)</span> is close to <span class="math">\(y\)</span>, the more likely it is that
we choose to pass the message to that neighbour.</p>
</li>
<li>
<p>We divide by <span class="math">\(w\)</span> to normalize the weights <span class="math">\(w_r\)</span> into a <a href="http://en.wikipedia.org/wiki/Probability_vector">probability
vector</a>.</p>
</li>
<li>
<p>Because of the continuity property of the coordinates, we can conclude that
<span class="math">\(odist(q,y) - odist(r,y) \in \{-1,0,1\}\)</span>. Therefore <span class="math">\(w_r \in \{{\beta}^{-1},1,\beta\}\)</span></p>
</li>
</ul>
<h5>Experiments results</h5>
<p>I wrote some code to test the efficiency of message routing using random walk,
as described above. It is written in Python3, and relies on the networkx library.</p>
<p>The code could be obtained <a href="https://github.com/realcr/freedomlayer_code/tree/master/landmarks_navigation_rw">here [github]</a></p>
<p>If you want to run it, make sure that you have networkx installed. It could be
installed by running:</p>
<div class="highlight"><pre><span></span>pip install networkx
</pre></div>
<p>There are a few files in the landmarks_navigation_rw folder. The file
graph_coord.py is the main graph coordinates library. It contains GraphCoord
class, which is most of the logic. The other python files rely on
graph_coord.py, and they check different things.</p>
<h6>Random Walking with odist</h6>
<p>We begin with random_walk_odist.py. This program checks the performance of
random walking with odist.</p>
<p>We generate random
<a href="http://en.wikipedia.org/wiki/Erd%C5%91s%E2%80%93R%C3%A9nyi_model">Erdos-Renyi</a>
networks of different sizes. We pick <span class="math">\(n = 2^i\)</span> to be the amount of nodes in the
network, and <span class="math">\(p = \frac{2i}{2^{i}}\)</span>: the probability for an edge to exist.
<span class="math">\(k=i^2\)</span> is the amount of landmarks. In other words, <span class="math">\(k =
\paren{\log{n}}^2\)</span>. </p>
<p>For the random walk we pick <span class="math">\(\beta = 22\)</span>. Why <span class="math">\(22\)</span>? It gave me good
results. You might find a better <span class="math">\(\beta\)</span> yourself. (Note that <span class="math">\(\beta\)</span> is
referred to as “base” inside the code.)</p>
<p>For every generated graph we simulate delivery of a few messages
using the random walk method.</p>
<p>These are the results: </p>
<div class="highlight"><pre><span></span>Random walking using odist
---------------------------
||| graph generation func = gen_gnp_graph
||| i's range = range(6, 16)
||| num_messages = 32
||| base = 22
i | k | Avg num hops | Max Node Visits | Max Coord Occur
----------------------------------------------------------------------
6 | 36 | 2.687500 | 5 | 1
7 | 49 | 3.031250 | 5 | 1
8 | 64 | 3.562500 | 4 | 1
9 | 81 | 5.781250 | 4 | 1
10 | 100 | 10.281250 | 6 | 1
11 | 121 | 15.437500 | 6 | 1
12 | 144 | 29.593750 | 12 | 1
13 | 169 | 34.531250 | 8 | 1
14 | 196 | 41.000000 | 7 | 1
15 | 225 | 53.593750 | 8 | 1
</pre></div>
<p>How to read this output?</p>
<p>The first lines show the parameters chosen for this run. graph_generation_func
is the function used to generate the graph. We chose gen_gnp_graph, which is an
Erdos-Renyi random graph. (Other possibility is gen_grid_graph. You can try
it). <span class="math">\(i\)</span> is a parameter related to the amount of nodes in the network. As
noted above, <span class="math">\(n = 2^{i}\)</span>. num_messages is the amount of messages we send to
approximate the average number of hops it takes for one message to get to its
destination. We chose 32.</p>
<p>Next, we have a table. Every row in this table represents the results for one
generated graph. We now explain every column in this table. <span class="math">\(i\)</span> is the
logarithm of the amount of nodes in the network. <span class="math">\(k\)</span> is the amount of
landmarks (Note that in this experiment, we chose <span class="math">\(k = i^2\)</span>).</p>
<p><span class="dquo">“</span>Avg num hops” is the average number of hops it takes for a message to get to
its destination. This number is approximate. It is deduced by sending just a
few messages, and calculating an average for the amount of hops.</p>
<p><span class="dquo">“</span>Max Node Visits” counts the maximum amount of times a node was visited by
wandering messages. For example, if this column shows <span class="math">\(5\)</span>, it means that
some node <span class="math">\(q\)</span> in the network has routed <span class="math">\(5\)</span> messages (Maybe even the same
message more than once), and no node in the network has routed more messages
than <span class="math">\(5\)</span>.</p>
<p><span class="dquo">“</span>Max Coord Occur” counts the occurrences of the most common coordinate. If it is
<span class="math">\(1\)</span>, it means that coordinates in the network are unique. If is is more than
<span class="math">\(1\)</span>, it means that some coordinate in the network occurs more than once,
hence the network coordinates are not unique.</p>
<p>The first thing to observe about the results is the value of “Max Coord Occur”.
It is always <span class="math">\(1\)</span> (At least until <span class="math">\(i=15\)</span>). This means that the network
coordinates are unique. This is important: We can not expect to route
messages successfully if nodes’ addresses aren’t unique. </p>
<p>If we pick <span class="math">\(i=k\)</span> the network coordinates will not be unique. If we pick
<span class="math">\(i=k^{1.5}\)</span> we sometimes get a <span class="math">\(2\)</span> at the “Max Coord Occur”.</p>
<p>Next, it is interesting to look at “Avg num hops”. It has some pretty small
values for <span class="math">\(i \leq 10\)</span>, but it increases very fast. At <span class="math">\(i=14\)</span> we already
get a value of <span class="math">\(68.9\)</span>. This is still practical, but at this growth rate when
we get to <span class="math">\(i=40\)</span> we expect to have a huge “Avg num hops”. My guess is that
this method won’t scale to very big networks.</p>
<p>Regarding “Max Node Visits”: It looks like it somehow grows, but I didn’t
manage to get any important conclusion from its values. I left it there because
you might have an idea.</p>
<h6>Measuring network load</h6>
<p>In the <a href="https://www.freedomlayer.org/the-distributed-post-office-instant-hierarcy-for-mesh-networks.html">Distributed Post
Office</a> we
managed to route messages very efficiently, however all the messages were
routed through a few specific nodes. This is unacceptable for large distributed
mesh networks.</p>
<p>We want to make sure that we managed to avoid this problem here. To measure the
load, we count the amount of messages that have passed through each of the
nodes in the network. Then we look at the nodes that routed the biggest amount
of messages.</p>
<p>For this run we generate a random network (Using the Erdos-Renyi model) of
size <span class="math">\(i=14\)</span> (<span class="math">\(n=2^{i})\)</span>. We pick <span class="math">\(p = \frac{2i}{2^{i}}\)</span> to be the
probability of an edge to exist. The amount of landmarks is <span class="math">\(k = i^{2}\)</span>, and
<span class="math">\(\beta = 22\)</span>.</p>
<p>After the graph generation, we simulate the delivery of <span class="math">\(4096\)</span> messages. For
each node <span class="math">\(x\)</span> we count the amount of times a message has passed through
<span class="math">\(x\)</span>. (Note that a message might pass more than once through <span class="math">\(x\)</span>).</p>
<p>The relevant code is inside measure_load_odist.py. Here are the results:</p>
<div class="highlight"><pre><span></span>||| graph generation func = gen_gnp_graph
||| i = 14
||| num_messages = 4096
||| base = 22
Generating graph...
Generating coordinates...
Simulating messages delivery...
most commonly visited nodes:
node | Times visited
-----------------------------------
1689 | 51
15541 | 47
13284 | 45
11387 | 44
8423 | 43
4256 | 41
6428 | 41
5203 | 40
13389 | 40
15753 | 40
6618 | 39
999 | 37
7084 | 37
9790 | 37
12962 | 37
1308 | 36
1548 | 36
5301 | 36
5592 | 36
9701 | 36
16201 | 36
573 | 35
1339 | 35
2628 | 35
4451 | 35
4615 | 35
7415 | 35
7492 | 35
7720 | 35
10367 | 35
12390 | 35
12492 | 35
</pre></div>
<p>The table shown in the output is the set of most commonly visited nodes. It
is ordered by the amount of visits.</p>
<p>The node column is the index of the node in the list of nodes inside the python
code. Times visited counts the amount of messages that have visited the
specific node.</p>
<p>None of the nodes you see in this table are landmark nodes. I know this because
if any of those nodes was a landmark, it was printed with two asterisks around
it. (See the source code for evidence). This means that the landmark nodes
aren’t being visited more than usual nodes.</p>
<p>We can also see that some nodes are more visited than others, though it seems
like there is no specific set of nodes that routes most of the messages.</p>
<h6>Naive random walking</h6>
<p>If you are not sure that we gained anything by having the <span class="math">\(odist\)</span> function,
let me show you what happens when we perform a naive random walk, without
using the information from <span class="math">\(odist\)</span>.</p>
<p>We use the same parameters from the “Random Walking with odist” section, except
for choosing <span class="math">\(\beta = 1\)</span>. Recall that the weight for moving to a neighbour
<span class="math">\(r\)</span> in the random walk is <span class="math">\(w_r = {\beta}^{odist(q,y) - odist(r,y)}\)</span>.
For <span class="math">\(\beta = 1\)</span> we always get <span class="math">\(w_r = 1\)</span>, therefore all the neighbours will
have the same probability of being chosen.</p>
<p>The code is at random_walk_naive.py. These are the results:</p>
<div class="highlight"><pre><span></span>Naive random walking
---------------------------
||| graph generation func = gen_gnp_graph
||| i's range = range(6, 16)
||| num_messages = 32
||| base = 1
i | k | Avg num hops | Max Node Visits | Max Coord Occur
----------------------------------------------------------------------
6 | 36 | 71.593750 | 59 | 1
7 | 49 | 172.187500 | 103 | 1
8 | 64 | 326.343750 | 82 | 1
9 | 81 | 449.625000 | 68 | 1
10 | 100 | 884.781250 | 61 | 1
11 | 121 | 2719.187500 | 85 | 1
12 | 144 | 3449.906250 | 58 | 1
13 | 169 | 10169.531250 | 88 | 1
14 | 196 | 18943.343750 | 82 | 1
15 | 225 | 36732.375000 | 78 | 1
</pre></div>
<p>Note that the average amount of hops is much larger in this case. This means
that we do gain something by using the information from the <span class="math">\(odist\)</span> function.</p>
<h4>Summary</h4>
<p>We introduced a way to assign Coordinates to nodes in any mesh network. We call
these coordinates The Network Coordinates, and we use them as the network
addresses of nodes. The Network coordinates have some “understanding” of the
structure of the mesh network which we want to exploit to route messages in the network.</p>
<p>We defined a function <span class="math">\(odist\)</span>, which is used here as approximation of
the network distance function (<span class="math">\(dist\)</span>). We also explored some other
interesting features of <span class="math">\(odist\)</span>. The value <span class="math">\(odist(x,y)\)</span> could be calculated
just by knowing the coordinates of <span class="math">\(x\)</span> and <span class="math">\(y\)</span>.</p>
<p>To route messages in the network we use random walking that takes into account
information from the <span class="math">\(odist\)</span> function: In every step, Nodes that are closer
to the destination are more likely to be chosen.</p>
<p>We ran the random walk algorithm and checked the results. It seems like the
random walk algorithm is practical for small networks (Probably up to
<span class="math">\(n=2^{14}\)</span>). We don’t know what happens for larger networks, but we
conjecture that the average amount of hops for message delivery becomes too large.</p>
<h4>Further thoughts</h4>
<p><strong>The random walk method we used is pretty naive</strong>. It was just a quick example of
what could be done with the Network Coordinates. Probably we can use the
Network Coordinates in a smarter way to route messages.</p>
<p>I suspect that routing messages using Network Coordinates on a two dimensional
grid graph should be easier than routing them on random Erdos-Renyi graph. This
is why all of the run results here deal with Erdos-Renyi graphs, which is the
harder case.</p>
<p><strong>Maintaining contact with the landmarks</strong>: The network changes (Nodes join and
leave). Therefore every node <span class="math">\(x\)</span> in the network should verify that his
shortest path to each of the landmarks is still alive, and also that the
landmarks themselves are still alive. <span class="math">\(x\)</span> could periodically send a message
to each of the landmarks, and expect a response. This method puts great
load over the landmarks. There is a way to overcome this issue by combining all
the challenges and responses in the network. We will discuss it in the future.</p>
<p>About <strong>choosing the Landmarks</strong>: We have already presented in “The Distributed
Post Office” a method to pick landmarks (We pick the nodes that maximize some
cryptographic hash functions with their public key). However, an adversary
could craft special public keys that happen to maximize the hash values, and
thus he could take control over all the landmarks. One approach would be to
“mine” landmarks. All the nodes in the network will make effort to get higher
hash values. Assuming that the adversary is computationally limited, we assume
that he won’t be able to gain control over most of the landmarks. This method
has the disadvantage of being computationally expensive.</p>
<p>A different approach to choosing landmarks would be to elect them. Every some
period of the time the landmarks will free some node from his landmark duty,
and elect a new landmark. </p>
<script type="text/javascript">if (!document.getElementById('mathjaxscript_pelican_#%@#$@#')) {
var align = "center",
indent = "0em",
linebreak = "false";
if (false) {
align = (screen.width < 768) ? "left" : align;
indent = (screen.width < 768) ? "0em" : indent;
linebreak = (screen.width < 768) ? 'true' : linebreak;
}
var mathjaxscript = document.createElement('script');
mathjaxscript.id = 'mathjaxscript_pelican_#%@#$@#';
mathjaxscript.type = 'text/javascript';
mathjaxscript.src = 'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.0/MathJax.js?config=TeX-AMS-MML_HTMLorMML';
mathjaxscript[(window.opera ? "innerHTML" : "text")] =
"MathJax.Hub.Config({" +
" config: ['MMLorHTML.js']," +
" TeX: { extensions: ['AMSmath.js','AMSsymbols.js','noErrors.js','noUndefined.js'], equationNumbers: { autoNumber: 'AMS' } }," +
" jax: ['input/TeX','input/MathML','output/HTML-CSS']," +
" extensions: ['tex2jax.js','mml2jax.js','MathMenu.js','MathZoom.js']," +
" displayAlign: '"+ align +"'," +
" displayIndent: '"+ indent +"'," +
" showMathMenu: true," +
" messageStyle: 'normal'," +
" tex2jax: { " +
" inlineMath: [ ['\\\\(','\\\\)'] ], " +
" displayMath: [ ['$$','$$'] ]," +
" processEscapes: true," +
" preview: 'TeX'," +
" }, " +
" 'HTML-CSS': { " +
" styles: { '.MathJax_Display, .MathJax .mo, .MathJax .mi, .MathJax .mn': {color: 'inherit ! important'} }," +
" linebreaks: { automatic: "+ linebreak +", width: '90% container' }," +
" }, " +
"}); " +
"if ('default' !== 'default') {" +
"MathJax.Hub.Register.StartupHook('HTML-CSS Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax['HTML-CSS'].FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"MathJax.Hub.Register.StartupHook('SVG Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax.SVG.FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"}";
(document.body || document.getElementsByTagName('head')[0]).appendChild(mathjaxscript);
}
</script>The Distributed Post Office: Instant Hierarcy for mesh networks2014-12-17T13:47:00+00:002014-12-17T13:47:00+00:00realtag:www.freedomlayer.org,2014-12-17:/the-distributed-post-office-instant-hierarcy-for-mesh-networks.html<h4>Abstract</h4>
<p>We introduce the Distributed Post Office, an idea for routing messages in a
decentralized mesh network. The Distributed Post Office is a method for
creating an instant hierarchical structure from a mesh network.</p>
<p>We describe the basic form of the Distributed Post Office, and mention two
improvements to its …</p><h4>Abstract</h4>
<p>We introduce the Distributed Post Office, an idea for routing messages in a
decentralized mesh network. The Distributed Post Office is a method for
creating an instant hierarchical structure from a mesh network.</p>
<p>We describe the basic form of the Distributed Post Office, and mention two
improvements to its structure.</p>
<p>Next, we run some <a href="https://github.com/realcr/freedomlayer_code/tree/master/dist_post_office">experiments
[github]</a>.
Our results show that the Distributed Post Office in its current form does not
scale well as a solution for routing messages in large networks.</p>
<p>In this text we do not discuss security or reliability topics related to the
Distributed Post office.</p>
<h4>The distributed post office question</h4>
<p>What if there there were no post offices in the world, and you wanted to send a
package to some far friend?</p>
<p>Assume that you meet some people you trust on a regular basis. More generally,
assume that every person <span class="math">\(x\)</span> in the world meets a few people he trusts on a
regular basis. We also call those trusted regulars of <span class="math">\(x\)</span> the <strong>neighbours</strong>
of <span class="math">\(x\)</span>. If <span class="math">\(y\)</span> is a neighbour of <span class="math">\(x\)</span>, we also say that <span class="math">\(x\)</span> is a
neighbour of <span class="math">\(y\)</span>. The neighbours relationships could be used to transfer packages. </p>
<p>For the feasibility of things, let’s also add in the assumption that
this system is connected. This means: We can get from any person to any other
person using a sequence of neighbours. Or, in other words: If we draw a graph
of all the people in the world (As nodes), and put edges between every person
and his neighbours, then we get a connected graph.</p>
<p><img alt="Connected Graph Path" src="https://www.freedomlayer.org/articles/freedomlayer/dist_post_office/images/connected_graph_path.svg"></p>
<p>In the picture: A connected network of people. There is a path of neighbours
that connects <span class="math">\(x\)</span> and <span class="math">\(y\)</span>.</p>
<p>Would these assumptions be enough to let us send a message to anyone in the world?</p>
<p>A few initial thoughts:</p>
<ul>
<li>
<p>Person <span class="math">\(a\)</span> can send a package to person <span class="math">\(B\)</span> only if there is a “path” of
neighbours between <span class="math">\(a\)</span> and <span class="math">\(b\)</span>. That means: <span class="math">\(a\)</span> has some neighbour
<span class="math">\(a_1\)</span>, and <span class="math">\(a_1\)</span> has some neighbour <span class="math">\(a_2\)</span> and so on, until finally some
<span class="math">\(a_k\)</span> has <span class="math">\(b\)</span> as a neighbour.</p>
</li>
<li>
<p>The lack of any centralized post office might create the urge to create one.
This artificial post office might be some person that everyone knows.
However, picking such a person that is agreed upon all the participants might
be difficult task.</p>
</li>
<li>
<p>What kind of addressing are we going to use for sending packages, if there is
no central post office? (What are we going to write on the package on the
“<span class="caps">TO</span>” field?) And what if the network layout changes? Will the address remain valid?</p>
</li>
</ul>
<h4>Basic Distributed Post office</h4>
<h5>Finding the highest person</h5>
<p>One way to solve this is finding some special person. This person will be used
as a reference point for sending messages. All our messages will pass through
this person. This centralized point of view is a bit different from our usual
decentralized point of view, but it will present some idea that we
might later be able to utilize in a decentralized way, so bear with me here.</p>
<p>Finding a special person could be achieved as follows: Every person <span class="math">\(x\)</span> will
maintain a link (Maybe through a few neighbours) to the highest person he knows
of (We assume that there are no two people of the exact same height). By
maintaining a link we mean that <span class="math">\(x\)</span> will remember some person <span class="math">\(y\)</span>, his
height and a path from <span class="math">\(x\)</span> to <span class="math">\(y\)</span> that goes through neighbours.</p>
<p>Initially, every person <span class="math">\(x\)</span> will look at his neighbours (And himself), and
find the highest person among them. Then <span class="math">\(x\)</span> will inform all his neighbours
of the highest person he knows (He will also send his path to that person).
Next, <span class="math">\(x\)</span> will look at all the highest known people he received from his
neighbours. He will update his highest known person accordingly.</p>
<p>The algorithm could be described simply as follows: In every iteration,
every person <span class="math">\(x\)</span> sends to all his neighbours the highest person <span class="math">\(y\)</span> that he
knows, and also the shortest path he knows from <span class="math">\(x\)</span> to <span class="math">\(y\)</span>. (Iteration
could happen every few seconds, for example).</p>
<p>After enough iterations, we expect that every person <span class="math">\(x\)</span> will find the
highest person in the world <span class="math">\(t\)</span>. In addition, <span class="math">\(x\)</span> will know a shortest path
possible from <span class="math">\(x\)</span> to <span class="math">\(t\)</span>.</p>
<p>Let’s explain this result: Why do we expect that the highest person <span class="math">\(t\)</span> will
always be found by all people, and also that a shortest path will be found?</p>
<p>Assume that <span class="math">\(x\)</span> is some person, and <span class="math">\(t\)</span> is the highest person in the world.
There is some shortest path of neighbours between <span class="math">\(x\)</span> and <span class="math">\(t\)</span> (However
<span class="math">\(x\)</span> doesn’t know that path yet). Let’s assume that this path of neighbours
have the people: <span class="math">\((x,x_1,x_2,\dots,x_{k-1},x_k,t)\)</span> in this order. </p>
<p><img alt="Shortest path x t" src="https://www.freedomlayer.org/articles/freedomlayer/dist_post_office/images/shortest_path_x_t.svg"></p>
<p>In the picture: A shortest path between <span class="math">\(x\)</span> and <span class="math">\(t\)</span>. Note that this is not
the only shortest path between <span class="math">\(x\)</span> and <span class="math">\(t\)</span>.</p>
<p>After the first iteration, <span class="math">\(x_k\)</span> will have <span class="math">\(t\)</span> as the highest person in the
world. <span class="math">\(x_k\)</span> will also have a shortest path to <span class="math">\(t\)</span>. In the next iteration,
<span class="math">\(x_{k-1}\)</span> will know <span class="math">\(t\)</span> as the highest person in the world (Because <span class="math">\(x_k\)</span>
has told him about it). <span class="math">\(x_{k-1}\)</span> will also know a shortest path to <span class="math">\(t\)</span>:
The path: <span class="math">\((x_{k-1},x_{k},t)\)</span>. What if this is not the shortest path? Then
there must be some path of length two, as follows: <span class="math">\(x_{k-1},t\)</span>. In that case,
there is also a shorter path between <span class="math">\(x\)</span> and <span class="math">\(t\)</span>:
<span class="math">\((x,x_1,x_2,\dots,x_{k-1},t)\)</span>. But this will contradict our assumption that
<span class="math">\(x,x_1,x_2,\dots,x_{k-1},x_k\)</span> is a shortest path between <span class="math">\(x\)</span> and <span class="math">\(t\)</span>.</p>
<p>We can continue this argument until we get to <span class="math">\(x\)</span>. (The formal way to do it
is using mathematical induction). Then we conclude that after <span class="math">\(k+1\)</span>
iterations, <span class="math">\(x\)</span> will have <span class="math">\(t\)</span> as the highest person in the world, and <span class="math">\(x\)</span>
will also have a shortest path from <span class="math">\(x\)</span> to <span class="math">\(t\)</span>.</p>
<p>Another way to think about it is that the amount of iterations needed until
every person finds the highest person in the world is not more than the
<a href="http://en.wikipedia.org/wiki/Distance_%28graph_theory%29">diameter</a> of the
neighbours graph.</p>
<p>We don’t really deal with security here, but I want to mention this question:
Whenever a person sends to all his neighbours the highest person he knows, how
can we know he doesn’t lie? We will deal with this later.</p>
<h5>Addressing and Drift</h5>
<p>So far we got some special person <span class="math">\(t\)</span> that every person can reach: Every
person <span class="math">\(x\)</span> knows a path to <span class="math">\(t\)</span>, and thus <span class="math">\(x\)</span> can send a package to <span class="math">\(t\)</span>.</p>
<p>How could <span class="math">\(x\)</span> send a message to some arbitrary person <span class="math">\(y\)</span>? <span class="math">\(x\)</span> already
knows the path from <span class="math">\(x\)</span> to <span class="math">\(t\)</span>. If <span class="math">\(x\)</span> knew a path from <span class="math">\(t\)</span> to <span class="math">\(y\)</span>,
he could first deliver his package to <span class="math">\(t\)</span>, and then ask <span class="math">\(t\)</span> to send his
package to <span class="math">\(y\)</span>.</p>
<p>This idea leads us to choose the address of an arbitrary node <span class="math">\(y\)</span> as a
shortest path between <span class="math">\(t\)</span> and <span class="math">\(y\)</span>. (This is the reversed path between <span class="math">\(y\)</span>
and <span class="math">\(t\)</span>). We will mark this as the address of <span class="math">\(y\)</span>, or <strong><span class="math">\(A(y)\)</span></strong>. </p>
<p>Given <span class="math">\(A(y)\)</span>, <span class="math">\(x\)</span> can send a package to <span class="math">\(y\)</span> by sending the package first
to <span class="math">\(t\)</span>, with the address <span class="math">\(A(y)\)</span> written on the package. <span class="math">\(t\)</span> will then use
the path <span class="math">\(A(y)\)</span> to deliver the message all the way to <span class="math">\(y\)</span>.</p>
<p>We ignore the centrality issues of this idea for a moment (<span class="math">\(y\)</span> has to deal
with all the packages sent in the world!!!), and try to think about the
addressing idea.</p>
<p><span class="math">\(y\)</span><span class="quo">‘</span>s address is induced from the structure of the network of neighbours.
Certain changes could invalidate <span class="math">\(y\)</span><span class="quo">‘</span>s address. In other words: There is a
drift in the addresses, as the network of neighbours changes.</p>
<p>One way to deal with this issue is to <strong>refresh</strong> the addresses from time to
time: If <span class="math">\(x\)</span> is in contact with <span class="math">\(y\)</span>, then <span class="math">\(y\)</span> will send his current
address to <span class="math">\(x\)</span> every few seconds. <span class="math">\(x\)</span> will also send his current address to
<span class="math">\(y\)</span> every few seconds. This might not be very reliable, but it’s an idea.</p>
<h4>Changing people into nodes</h4>
<p>Formally we solved the problem of delivering packages, however the solution is
not very satisfying. All the packages has to go through some special person.
This is not good for us because of security reasons (Could we trust the special
person), and also because of load issues (Just because he is the
highest person in the world, he has to deal with all the packages?)</p>
<p>Before we start proposing more ideas, it is probably a good time to change our
terminology to networks and nodes. We get the following question: Assume that
we are given a mesh network, where every node has a few neighbours. How can we
deliver messages between every two arbitrary nodes?</p>
<p>Note that we already tried to solve this question using
<a href="http://en.wikipedia.org/wiki/Flooding_%28computer_networking%29">flooding</a>,
<a href="https://www.freedomlayer.org/sqrtn-mesh-routing.html"><span class="math">\(sqrt{n}\)</span>
routing</a>
and
<a href="https://www.freedomlayer.org/experimenting-with-virtual-dht-routing.html">Virtual <span class="caps">DHT</span>
routing</a>.</p>
<p>Instead of checking the height of people, we can use some other properties of
nodes. We will use some <a href="http://en.wikipedia.org/wiki/Public-key_cryptography">public key
cryptography</a>.
Every node <span class="math">\(y\)</span> will create a key pair <span class="math">\(prv_x,pub_x\)</span> (Private and Public).
We assume that every node <span class="math">\(x\)</span> knows the public keys of all his neighbours.</p>
<p>Next, we choose some <a href="http://en.wikipedia.org/wiki/Cryptographic_hash_function">cryptographic hash
function</a> <span class="math">\(h\)</span>.
Instead of person <span class="math">\(x\)</span><span class="quo">‘</span>s height, we will take a look at <span class="math">\(h(pub_x)\)</span>. We will
call this value <span class="math">\(x\)</span><span class="quo">‘</span>s height. The highest person in the world will turn into
the node <span class="math">\(t\)</span> that maximizes the value <span class="math">\(h(pub_t)\)</span>. In that case we will also
say that <span class="math">\(t\)</span> is the “highest” node in the network with respect to the
cryptographic hash function <span class="math">\(h\)</span>.</p>
<p>While it was difficult to verify the height of a person from a
distance, we could verify the public key of a node from a distance.
If node <span class="math">\(x\)</span> is informed by node <span class="math">\(y\)</span> about some remote node <span class="math">\(t\)</span> that has a
certain <span class="math">\(h(pub_t)\)</span> value, <span class="math">\(x\)</span> can verify it himself. <span class="math">\(x\)</span> can send some
challenge all the way to <span class="math">\(t\)</span>, and <span class="math">\(t\)</span> will send back a response that proves
he owns the public key <span class="math">\(pub_t\)</span>. </p>
<p><img alt="Remote Verification" src="https://www.freedomlayer.org/articles/freedomlayer/dist_post_office/images/remote_verification.svg"></p>
<p>In the picture: <span class="math">\(x\)</span> wants to verify that <span class="math">\(t\)</span> owns the public key <span class="math">\(pub_t\)</span>.
Therefore <span class="math">\(x\)</span> sends a challenge to <span class="math">\(t\)</span>. If <span class="math">\(t\)</span> returns a correct
response, <span class="math">\(x\)</span> will believe that <span class="math">\(t\)</span> owns the public key <span class="math">\(pub_t\)</span>.</p>
<p>Note however that this challenge response idea is not a magic cure to all the
security problems in this model. It just helps a bit.</p>
<h4>Adding hierarchy</h4>
<h5>Highest node in some radius</h5>
<p>One approach to make things better is to create some kind of hierarchy. Earlier,
every node <span class="math">\(x\)</span> maintained contact to the “highest” node in the network (with
respect to some cryptographic hash function <span class="math">\(h\)</span>) through some path of nodes. </p>
<p>This time, instead of remembering just the “highest” node in the network, every
node <span class="math">\(x\)</span> will remember a few special nodes. Every node will be the “highest”
in some certain area around <span class="math">\(x\)</span>:</p>
<ul>
<li><span class="math">\(t_x^1\)</span>: The “highest” node of maximum distance <span class="math">\(1\)</span>.</li>
<li><span class="math">\(t_x^2\)</span>: The “highest” node of maximum distance <span class="math">\(2\)</span>.</li>
<li><span class="math">\(t_x^3\)</span>: The “highest” node of maximum distance <span class="math">\(3\)</span>.</li>
</ul>
<p>…</p>
<ul>
<li><span class="math">\(t_x^d\)</span> The “highest” node in the network.</li>
</ul>
<p><img alt="Circles Highest" src="https://www.freedomlayer.org/articles/freedomlayer/dist_post_office/images/circles_highest.svg"></p>
<p>In the picture: Distance rings around <span class="math">\(x\)</span>. The first ring is distance <span class="math">\(0\)</span>
from <span class="math">\(x\)</span>. It contains only <span class="math">\(x\)</span> itself. The next ring contains all nodes of
distance exactly <span class="math">\(1\)</span> from <span class="math">\(x\)</span>. The next ring contains all nodes of distance
exactly <span class="math">\(2\)</span>, and so on. Note that by definition, A node from a certain ring
can only be connected to nodes of adjacent rings, or nodes from the same ring.</p>
<p>We mark by <span class="math">\(t_x^j\)</span> The “highest” node of maximum distance <span class="math">\(j\)</span> from <span class="math">\(x\)</span>.
In our example, <span class="math">\(t_x^3 = t_x^4\)</span>.</p>
<p>Side question: How can we choose a good value for <span class="math">\(d\)</span>? One suggestion is to
keep increasing the distance until we stop getting new highest nodes. Another
suggestion would be to just assume that the graph diameter won’t be more than
some constant number.</p>
<p>First let’s assume that somehow we managed to have the above information for
every node in the network, and see what we can do with it. (Note that we didn’t
yet describe how to get this information. It will be described soon later). </p>
<p>We define <span class="math">\(x\)</span><span class="quo">‘</span>s address to be <span class="math">\(A(x) = (p_x^1,p_x^2,\dots,p_x^d)\)</span>, where
<span class="math">\(p_x^j\)</span> is the path from <span class="math">\(t_x^j\)</span> to <span class="math">\(x\)</span>. This definition of <span class="math">\(A(x)\)</span> is
an extension of our previous definition of <span class="math">\(A(x)\)</span>, where we only had the
<span class="math">\(t_x^d = t\)</span>. Also note that looking at some <span class="math">\(p_x^j\)</span>, one can conclude
<span class="math">\(t_x^j\)</span> (It is just the first node on the path).</p>
<p>Looking at two different nodes: <span class="math">\(x,y\)</span>, the first thing to note is that
<span class="math">\(t_x^d\)</span> and <span class="math">\(t_y^d\)</span> are the same, assuming that <span class="math">\(d\)</span> is large enough. Why?
Because <span class="math">\(t_x^d = t_y^d = t\)</span>, the highest node in the network. For other
distances, the nodes <span class="math">\(x\)</span> and <span class="math">\(y\)</span> have chosen might differ. For example,
<span class="math">\(t_x^1\)</span> and <span class="math">\(t_y^1\)</span> are likely to be different.</p>
<p>How to deliver messages using the address information? Assume that <span class="math">\(x\)</span> has
the address of <span class="math">\(y\)</span>: <span class="math">\(A(y)\)</span>, as described above. <span class="math">\(x\)</span> will compare his own
address: <span class="math">\(A(x)\)</span> with <span class="math">\(A(y)\)</span>. <span class="math">\(x\)</span> will try to find the smallest <span class="math">\(j\)</span> such
that <span class="math">\(t_x^j = t_y^j\)</span>. <span class="math">\(x\)</span> knows a shortest path from <span class="math">\(x\)</span> to <span class="math">\(t_x^j\)</span>.
<span class="math">\(x\)</span> also knows <span class="math">\(A(y)\)</span>, So <span class="math">\(x\)</span> knows <span class="math">\(p_y^j\)</span>, which is a shortest
path from <span class="math">\(t_y^j = t_x^j\)</span> to <span class="math">\(y\)</span>. Finally, <span class="math">\(x\)</span> can create a full path
from <span class="math">\(x\)</span> to <span class="math">\(y\)</span> that goes through <span class="math">\(t_y^j = t_x^j\)</span>. This path could be
used to send messages.</p>
<p><img alt="Message through mediator" src="https://www.freedomlayer.org/articles/freedomlayer/dist_post_office/images/msg_through_mediator.svg"></p>
<p>In the picture: The node <span class="math">\(t_x^4 = t_y^4\)</span> is a mediator between <span class="math">\(x\)</span> and
<span class="math">\(y\)</span>. <span class="math">\(x\)</span> and <span class="math">\(y\)</span> can route messages through <span class="math">\(t_x^4 = t_y^4\)</span>.</p>
<p>Addresses should not be too large if we want them to be practical to use. Let’s
estimate the size of a typical address, as defined above. For some node <span class="math">\(x\)</span>,
<span class="math">\(x\)</span><span class="quo">‘</span>s address is <span class="math">\((p_x^1,p_x^2,\dots,p_x^d)\)</span>. Every such <span class="math">\(p_x^j\)</span> is a
path. Assuming that the diameter of the network graph is <span class="math">\(d\)</span>, we get that
each path is of length no more than <span class="math">\(d\)</span>. Therefore we get at most <span class="math">\(d^{2}
q\)</span>, where <span class="math">\(q\)</span> is the size of a typical public key. This could become more
than a few kilobytes if the public key size and the network diameter are big.
(Much more than an <span class="caps">IP</span> address, unfortunately).</p>
<h5>Obtaining “highest” nodes</h5>
<p>We now explain how a node <span class="math">\(x\)</span> can obtain contact to the nodes
<span class="math">\(t_x^1,\dots,t_x^d\)</span>. (And also a shortest path to each of those nodes).</p>
<p>In every iteration, the node <span class="math">\(x\)</span> will ask every neighbour <span class="math">\(y\)</span> for the value
<span class="math">\(t_y^j\)</span> for every <span class="math">\(1 \leq j \leq d\)</span>. Then <span class="math">\(x\)</span> will update his values of
<span class="math">\(t_x^j\)</span> accordingly:
The value <span class="math">\(t_y^j\)</span> from <span class="math">\(y\)</span> is a candidate for <span class="math">\(t_x^{j+1}\)</span>. If <span class="math">\(t_y^j\)</span>
is “higher” than <span class="math">\(t_x^{j+1}\)</span>, then <span class="math">\(x\)</span> will replace it with <span class="math">\(t_y^j\)</span>.</p>
<p>Simply speaking: In iteration number <span class="math">\(k\)</span>, a node <span class="math">\(x\)</span> is aware of all the
“highest” nodes in the network until distance <span class="math">\(k\)</span>.</p>
<p>Formally, Using mathematical induction (Over the amount of iterations) it can
be shown that after <span class="math">\(k\)</span> network iterations, Every node <span class="math">\(x\)</span> knows the
correct value for <span class="math">\(t_x^j\)</span> for every <span class="math">\(1 \leq j \leq k\)</span>, and also a shortest
path to <span class="math">\(t_x^j\)</span>.</p>
<h5>Hierarchy benefits?</h5>
<p>Why would we want to have hierarchy from the first place? Earlier we complained
that all the messages will go through one special node <span class="math">\(t\)</span>, and <span class="math">\(t\)</span> won’t
be able to deal with the load. Maybe the hierarchy we have added can help a bit.</p>
<p>How often will a message go through the “highest” node <span class="math">\(t\)</span>? </p>
<p>Assume that <span class="math">\(x\)</span> wants to send a message to some node <span class="math">\(y\)</span>. <span class="math">\(x\)</span> checks
<span class="math">\(y\)</span><span class="quo">‘</span>s address, and tries to find the first <span class="math">\(j\)</span> such that <span class="math">\(t_x^j =
t_y^j\)</span>. (As described above). If any small enough such <span class="math">\(j\)</span> is found, the
message will be routed through <span class="math">\(t_x^j = t_y^j\)</span>, and not through the “highest”
node <span class="math">\(t\)</span>. We can think of <span class="math">\(t\)</span>, the “highest” node, as some kind of backup.
If nothing better was found, we can always route through <span class="math">\(t\)</span>.</p>
<p>But how do we know if <span class="math">\(x\)</span> will route his message to <span class="math">\(y\)</span> through <span class="math">\(t\)</span>, or
through some lower level node <span class="math">\(t_x^j = t_y^j\)</span>? Generally speaking, we expect
that the more <span class="math">\(x\)</span> and <span class="math">\(y\)</span> are close, the more their addresses <span class="math">\(A(x),A(y)\)</span>
are similar, and the more likely it is to route the message using a “high” node
that is not the “highest” node in the network.</p>
<h6>The “highest” node is still overloaded</h6>
<p>We expect that messages between close nodes are routed using a local “high”
node, while messages between very far nodes are routed using a globally “high”
node. Therefore <strong>idealy</strong> we expect that the “highest” node in the network
will not be so loaded, because the lower level “high” nodes will take part of
the load.</p>
<p><del>This somehow resembles the way physical post offices work. You have
the global post office which handles messages between countries, and smaller
post offices that handle messages between cities, and so on.</del>
<span class="caps">EDIT</span>: Thanks to rubygeek I now know that post offices work differently in some
countries :)</p>
<p>However, this is just the ideal. It is true that the local “high” nodes in the
network take part of the load from the “highest” node in the network, but
usually they only take a small part of it. Intuitively, this happens for a few reasons:</p>
<ul>
<li>
<p>In physical mail system people tend to send packages and letters to people
close to them geographically, so the structure of global post offices and local
post offices makes sense. However, this is not the case with mesh networks: In
a mesh network, any two arbitrary nodes <span class="math">\(x\)</span> and <span class="math">\(y\)</span> might want to
communicate. With high probability <span class="math">\(x\)</span> and <span class="math">\(y\)</span> are far away from each
other (With respect to the network graph), and so their messages will be routed
through the “highest” node in the network.</p>
</li>
<li>
<p>In a grid style graph (Or any <a href="http://en.wikipedia.org/wiki/Planar_graph">planar
graphs</a>), close nodes are expected
to have many “high” nodes in common. However, for other types of graphs, close
nodes might only have the “highest” node in common.</p>
</li>
</ul>
<p>I want to discuss the second reason with a bit more detail. (Though with a bit
of hand waving). You can skip directly to the code experiments results below.</p>
<p>For some node <span class="math">\(x\)</span> in the network, we denote by <span class="math">\(R_j(x)\)</span> the set of nodes of
distance no more than <span class="math">\(j\)</span> from <span class="math">\(x\)</span>. You can think about it as a ball around
<span class="math">\(x\)</span> of radius <span class="math">\(j\)</span>.</p>
<p>Consider two nodes <span class="math">\(x\)</span> and <span class="math">\(y\)</span> in the network. We observe the sets <span class="math">\(R_i(x)
\cap R_j(y)\)</span> and <span class="math">\(R_i(x) \cup R_j(y)\)</span>.</p>
<p><img alt="Two balls intersect" src="{images}two_balls_intersect.svg"></p>
<p>A schematic picture of the sets <span class="math">\(R_i(x)\)</span> and <span class="math">\(R_j(y)\)</span> intersecting.
<span class="math">\(R_i(x)\)</span> contains all the nodes of distance at most <span class="math">\(i\)</span> from <span class="math">\(x\)</span>.
<span class="math">\(R_j(y)\)</span> contains all the nodes of distance at most <span class="math">\(j\)</span> from <span class="math">\(y\)</span>.</p>
<p>Let <span class="math">\(w\)</span> be the “highest node in <span class="math">\(R_i(x) \cup R_j(y)\)</span>. If <span class="math">\(w\)</span> is inside
<span class="math">\(R_i(x) \cap R_j(y)\)</span> then <span class="math">\(t_x^i = w = t_y^j\)</span>. (In other words: <span class="math">\(w\)</span> is
the “highest” node in distance <span class="math">\(i\)</span> from <span class="math">\(x\)</span>, and the “highest” node in
distance <span class="math">\(j\)</span> from <span class="math">\(y\)</span>). In that case, <span class="math">\(x\)</span> and <span class="math">\(y\)</span> could route messages
through <span class="math">\(w\)</span>.</p>
<p>What are the odds that such <span class="math">\(w\)</span> exists? As the “highest” node in <span class="math">\(R_i(x)
\cup R_j(y)\)</span> could be any node in that set, the odds are:
[\frac{\left|R_i(x) \cap R_j(y)\right|}{\left|R_i(x) \cup R_j(y)\right|}]</p>
<p>I don’t know to calculate those odds for every type of graph, but let me leave
you with my intuition about it. For every type of graphs, the odds for the
existence of <span class="math">\(w\)</span> decrease as the distance between <span class="math">\(x\)</span> and <span class="math">\(y\)</span>
increases. However, for some graphs the odds decrease slowly, and for other
graphs, the odds decrease quickly.</p>
<p>The odds decrease slowly (Quadratically) when we consider a planar graph, like
a grid. However, for random graphs (Like the <a href="http://en.wikipedia.org/wiki/Erd%C5%91s%E2%80%93R%C3%A9nyi_model">Erdos-Renyi
model</a>) the
odds decrease quickly (exponentially).</p>
<p>This might be related to the fact that intersection between higher dimensional
spheres becomes smaller with respect to their union, as we increase the dimension.</p>
<h6>Experiments results</h6>
<p>I wrote some Python3 code to check the load over the “high” nodes in the
network. It can be found <a href="https://github.com/realcr/freedomlayer_code/tree/master/dist_post_office">here [github]</a></p>
<p>To run this code, you will need the python package <strong>networkx</strong>. I could be
installed as follows:</p>
<div class="highlight"><pre><span></span>pip install networkx
</pre></div>
<p>If you want to change any parameter in the code, check out the go() function.
All the parameters are there.</p>
<p>The code creates a network <span class="math">\(G\)</span> of nodes with random identity numbers. Using
iterations as described above (In “Obtaining highest nodes”), every node <span class="math">\(x\)</span>
finds the highest node in distance <span class="math">\(j\)</span>, for every distance <span class="math">\(0 < j \leq
diameter(G)\)</span>.</p>
<p>After creating the network and finding the “high” nodes, some large amount of
pairs of nodes are chosen randomly. Between every pair of nodes <span class="math">\(x,y\)</span> the best
mediator node is found. A mediator node is some “high” node that both <span class="math">\(x\)</span> and
<span class="math">\(y\)</span> know. A best mediator is a mediator that minimizes the sum of distances
from <span class="math">\(x\)</span> and <span class="math">\(y\)</span>.</p>
<p>For every mediator ever chosen, we count the amount of messages that were
routed through that mediator. We sort the mediators list by the amount of
messages they have routed, and show as output the mediators that routed the
largest amount of messages. Those are the mediators that had the highest load.</p>
<p>These are the results for a two dimensional grid graph of about <span class="math">\(2^{12}\)</span>
nodes, and simulation of <span class="math">\(2^{16}\)</span> messages.</p>
<div class="highlight"><pre><span></span>||| i = 12
||| num_hashes = 1
||| ident_bits = 32
Generating graph...
Generating Network...
Calculating specials...
Simulating 65536 messages delivery...
most common mediators:
mediator index | ratio | messages routed
------------------------------------------------
565 | 0.507568 | 33264
3251 | 0.210510 | 13796
3661 | 0.078995 | 5177
1573 | 0.058914 | 3861
3724 | 0.031265 | 2049
1806 | 0.022171 | 1453
3333 | 0.018906 | 1239
1341 | 0.006180 | 405
159 | 0.006027 | 395
1884 | 0.005585 | 366
2047 | 0.005035 | 330
978 | 0.003891 | 255
3109 | 0.003662 | 240
377 | 0.002518 | 165
26 | 0.002228 | 146
2269 | 0.001862 | 122
</pre></div>
<p>How to read this table? Mediator index is a unique number that identifies the
node used as a mediator. In our code, every node has a unique number. This
number doesn’t really matter to you. (If you really care, it is the index
number inside a python list)</p>
<p>The ratio is the amount of messages routed through a given node, divided by the
total amount of messages delivered. In our results the first ratio is
<span class="math">\(0.507\)</span>. This ratio could be calculated as <span class="math">\(33264/65536\)</span>. The last column
shows the amount of messages routed through a specific node.</p>
<p>It can be seen from the table that the first node (565) routes most of the
messages (about half of the messages). It is probably the “highest” node in the network.</p>
<p>Next, let’s look at the results for an Erdos-Renyi Random graph with <span class="math">\(2^{12}\)</span>
nodes, and <span class="math">\(p = (2\cdot 12)/{2^{12}}\)</span> (This is the probability for every
edge in the graph to exist). Again we simulate the delivery of <span class="math">\(2^{16}\)</span> messages.</p>
<div class="highlight"><pre><span></span>||| i = 12
||| num_hashes = 1
||| ident_bits = 32
Generating graph...
Generating Network...
Calculating specials...
Simulating 65536 messages delivery...
most common mediators:
mediator index | ratio | messages routed
------------------------------------------------
3425 | 0.918594 | 60201
2300 | 0.029877 | 1958
3935 | 0.012985 | 851
3232 | 0.006516 | 427
2453 | 0.005585 | 366
767 | 0.004410 | 289
2281 | 0.003174 | 208
943 | 0.002869 | 188
457 | 0.002625 | 172
2189 | 0.002319 | 152
3682 | 0.001694 | 111
3215 | 0.001648 | 108
641 | 0.001144 | 75
1049 | 0.000565 | 37
3469 | 0.000534 | 35
782 | 0.000519 | 34
</pre></div>
<p>It can be seen that there is a main difference between the results of the
Erdos-Renyi model and the two dimensional grid. In the Erdos-Renyi model the
ratios are more condensed to the upper part of the table. The most common
mediator routes 0.91 of the messages. (Compared to about half in the grid
case). Also note that the rest of the ratios decrease much faster in the
Erdos-Renyi model, compared to the two dimensional grid.</p>
<h4>Adding hash functions</h4>
<p>Another idea to take off the load from the “highest” node in the network would
be to add more cryptographic hash functions.</p>
<p>Recall that the “highest” node in the network, <span class="math">\(t\)</span>, is a node that maximizes
the value <span class="math">\(h(pub_t)\)</span>, for some cryptographic hash function <span class="math">\(h\)</span>. We could
add a few more cryptographic hash functions, to end up with a few “highest”
nodes, one for each hash function. The process of obtaining “highest” nodes
for different distances will be invoked independently for each of the hash functions.</p>
<p>With more hash functions, every two nodes <span class="math">\(x,y\)</span> are likely to have more
“high” nodes in common with respect to a few different hash functions. On the
other hand, having <span class="math">\(k\)</span> hash functions means having an address that is <span class="math">\(k\)</span>
times bigger. (Because it contains paths to highest nodes for <span class="math">\(k\)</span> different
hashes). It also means that every node has to maintain contact with <span class="math">\(k\)</span> times
more nodes.</p>
<p>You might be wondering where will we get all those cryptographic hash functions
from. If you have one cryptographic hash function <span class="math">\(h\)</span>, you can get more for
free by appending something to the input. For example, given a function <span class="math">\(h\)</span>
we can define <span class="math">\(h_i(x) = h(i . x)\)</span>, where <span class="math">\(.\)</span> is string concatenation. </p>
<p>This is a pretty simple example of how to do it, and it could be <a href="http://en.wikipedia.org/wiki/Length_extension_attack">not very
secure in some cases</a>, so
be careful. We are just experimenting here, so we don’t really care.</p>
<p>Let’s look at some run results with more than one hash function.
We show here the results for an Erdos-Renyi network with <span class="math">\(2^{12}\)</span> nodes, and
<span class="math">\(p = (2\cdot 12)/2^{12}\)</span>. We use <span class="math">\(4\)</span> hash functions:</p>
<div class="highlight"><pre><span></span>||| i = 12
||| num_hashes = 4
||| ident_bits = 32
Generating graph...
Generating Network...
Calculating specials...
Simulating 65536 messages delivery...
most common mediators:
mediator index | ratio | messages routed
------------------------------------------------
3685 | 0.189270 | 12404
2935 | 0.189026 | 12388
2010 | 0.187103 | 12262
3136 | 0.186569 | 12227
2466 | 0.025085 | 1644
3546 | 0.021988 | 1441
3886 | 0.021057 | 1380
1330 | 0.015961 | 1046
2039 | 0.010941 | 717
400 | 0.010269 | 673
761 | 0.009781 | 641
1057 | 0.008957 | 587
502 | 0.006882 | 451
2890 | 0.006760 | 443
2487 | 0.005554 | 364
3204 | 0.005341 | 350
</pre></div>
<p>We can see from the results that the first <span class="math">\(4\)</span> most common mediator nodes
route about the same amount of messages: About 0.18 of all the messages sent.
The rest of the nodes route much less messages. (The next one routes 0.02
messages out of the total amount).</p>
<p>It seems that having <span class="math">\(k\)</span> cryptographic hash functions approximately
divides the amount of work the “highest” node has to do by <span class="math">\(k\)</span>. It’s an
improvement, but it seems like each of the “highest” nodes still has to route a
constant fraction of all the message sent in the network, which is unacceptable
for a symmetric mesh network. (In other words: The average computer out there
can not deal with this amount of traffic).</p>
<h4>Final simplification</h4>
<p>We began with a simple idea of the “highest” node in the network, and later
added two improvements: First, every node <span class="math">\(x\)</span> had to remember the “highest”
node of distance <span class="math">\(j\)</span>: <span class="math">\(t_x^j\)</span>, for every <span class="math">\(0 < j \leq diameter(G)\)</span>. Next,
we added some more “height” properties. In other words: We added more hash functions.</p>
<p>We noticed that the first improvement: remembering local “high” nodes didn’t
help much. It took some of the traffic from the “highest” node in the network
to some other local “high” nodes, but still most of the traffic was routed by
the “highest” node in the network.</p>
<p>The second improvement: Changing to <span class="math">\(k\)</span> hash functions instead of one made a
bigger difference. Instead of having just one “highest” node in the network,
now there are <span class="math">\(k\)</span> of them, and the task of routing the network messages is
divided somewhat equally between those <span class="math">\(k\)</span> <span class="dquo">“</span>highest” node. </p>
<p>We could discard the first improvement (Remembering local “high” nodes), and
stay only with the second improvement (Adding more hash functions). That
means: Every node will have to remember one “highest” node <span class="math">\(l_j\)</span> for every
one of <span class="math">\(k\)</span> hash function. Note that <span class="math">\(l_j\)</span> is the “highest” node in the
network with respect to hash function number <span class="math">\(j\)</span>. All the messges sent in the
network will then be routed by the <span class="math">\(k\)</span> <span class="dquo">“</span>highest” nodes <span class="math">\(l_1,\dots,l_k\)</span>.
(One “highest” node for every cryptographic hash function).</p>
<p>Using this simpler method we are going to have results comparable to the more
complex method described above, and at the same time have much shorter
addresses for nodes. We call the <span class="math">\(k\)</span> <span class="dquo">“</span>highest” nodes in the graph <strong>the
<span class="math">\(k\)</span> landmarks</strong>.</p>
<p><img alt="K Landmarks Routing" src="https://www.freedomlayer.org/articles/freedomlayer/dist_post_office/images/k_landmarks_routing.svg"></p>
<p>In the picture: The nodes <span class="math">\(l_j\)</span> for <span class="math">\(1 \leq j \leq 5\)</span> are the landmarks
used for routing messages. <span class="math">\(l_j\)</span> is the “highest” node in the network with
respect to cryptographic hash function number <span class="math">\(j\)</span>.Each node in the network
maintains a shortest path to each of the landmarks.</p>
<p>Note that each message sent in the network will be routed through one of the
landmarks, therefore each landmarks is going to route about <span class="math">\(1/5\)</span> of all the
messages in the network.</p>
<p>In the example: Both <span class="math">\(x\)</span> and <span class="math">\(y\)</span> keep contact with each of the <span class="math">\(5\)</span>
landmarks, so messages between <span class="math">\(x\)</span> and <span class="math">\(y\)</span> could be routed using any of
those <span class="math">\(5\)</span> landmarks.</p>
<p>We end up with a routing method that is not very practical for large networks,
but it gives us some new ideas about routing. This method will serve as a
starting point for some of our next ideas.</p>
<h4>Summary</h4>
<p>The Distributed Post Office is an algorithm for creating a hierarchical
structure in a decentralized mesh network.</p>
<p>Our results show that the Distributed Post Office works, but not very
efficiently: A few nodes (The “highest” node with respect to every hash
function) have the responsibility of routing messages for the whole network.</p>
<p>In our last simplified solution, every node has to remember exactly <span class="math">\(k\)</span>
landmarks: The “highest” nodes with respect to <span class="math">\(k\)</span> different cryptographic hash
functions. In this solution we get that every landmark has to route <span class="math">\(1/k\)</span> of
the messages sent in the network.</p>
<p>A question to think about: Is there a way to route the messages in the network
using the knowledge about the <span class="math">\(k\)</span> landmarks, without actually routing the
messages through the landmarks themselves?</p>
<script type="text/javascript">if (!document.getElementById('mathjaxscript_pelican_#%@#$@#')) {
var align = "center",
indent = "0em",
linebreak = "false";
if (false) {
align = (screen.width < 768) ? "left" : align;
indent = (screen.width < 768) ? "0em" : indent;
linebreak = (screen.width < 768) ? 'true' : linebreak;
}
var mathjaxscript = document.createElement('script');
mathjaxscript.id = 'mathjaxscript_pelican_#%@#$@#';
mathjaxscript.type = 'text/javascript';
mathjaxscript.src = 'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.0/MathJax.js?config=TeX-AMS-MML_HTMLorMML';
mathjaxscript[(window.opera ? "innerHTML" : "text")] =
"MathJax.Hub.Config({" +
" config: ['MMLorHTML.js']," +
" TeX: { extensions: ['AMSmath.js','AMSsymbols.js','noErrors.js','noUndefined.js'], equationNumbers: { autoNumber: 'AMS' } }," +
" jax: ['input/TeX','input/MathML','output/HTML-CSS']," +
" extensions: ['tex2jax.js','mml2jax.js','MathMenu.js','MathZoom.js']," +
" displayAlign: '"+ align +"'," +
" displayIndent: '"+ indent +"'," +
" showMathMenu: true," +
" messageStyle: 'normal'," +
" tex2jax: { " +
" inlineMath: [ ['\\\\(','\\\\)'] ], " +
" displayMath: [ ['$$','$$'] ]," +
" processEscapes: true," +
" preview: 'TeX'," +
" }, " +
" 'HTML-CSS': { " +
" styles: { '.MathJax_Display, .MathJax .mo, .MathJax .mi, .MathJax .mn': {color: 'inherit ! important'} }," +
" linebreaks: { automatic: "+ linebreak +", width: '90% container' }," +
" }, " +
"}); " +
"if ('default' !== 'default') {" +
"MathJax.Hub.Register.StartupHook('HTML-CSS Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax['HTML-CSS'].FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"MathJax.Hub.Register.StartupHook('SVG Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax.SVG.FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"}";
(document.body || document.getElementsByTagName('head')[0]).appendChild(mathjaxscript);
}
</script>Experimenting with Virtual DHT Routing2014-12-14T15:51:00+00:002014-12-14T15:51:00+00:00realtag:www.freedomlayer.org,2014-12-14:/experimenting-with-virtual-dht-routing.html<h4>Abstract</h4>
<p>We introduce the Virtual <span class="caps">DHT</span> idea for routing messages in a mesh network. We
describe a crude algorithm that builds a Chord Virtual <span class="caps">DHT</span>. Next we run some
<a href="https://github.com/realcr/freedomlayer_code/tree/master/exp_virtual_dht_routing">experiments
[github]</a>
to measure the algorithm’s ability to create a Virtual <span class="caps">DHT</span> efficiently. We also
measure the expected amount of …</p><h4>Abstract</h4>
<p>We introduce the Virtual <span class="caps">DHT</span> idea for routing messages in a mesh network. We
describe a crude algorithm that builds a Chord Virtual <span class="caps">DHT</span>. Next we run some
<a href="https://github.com/realcr/freedomlayer_code/tree/master/exp_virtual_dht_routing">experiments
[github]</a>
to measure the algorithm’s ability to create a Virtual <span class="caps">DHT</span> efficiently. We also
measure the expected amount of hops required to deliver a message using this method. </p>
<p>Our experiments results show that Virtual <span class="caps">DHT</span> routing might work for networks
of certain size, but at this time we do not know how the results scale to
larger networks.</p>
<p>We don’t present any formal (mathematical) results regarding the idea of
Virtual <span class="caps">DHT</span>, and we don’t discuss reliability and security issues.</p>
<h4>The message routing problem and related work</h4>
<p>Assume that we have <span class="math">\(n\)</span> correct (honest) nodes (We will not consider any
Adversary at the moment), connected somehow in a mesh. Every node is linked to
a few other nodes. We call these nodes his immediate neighbours.<br>
Nodes in the mesh have only local knowledge. They only know about their
immediate neighbours. They don’t know about the general structure of the mesh network.</p>
<p>For any two nodes <span class="math">\(a,b\)</span> in the network, we want to be able to send a message
from <span class="math">\(a\)</span> to <span class="math">\(b\)</span>. We care that this message will arrive with high
probability, quickly enough, and also that it won’t disturb too many nodes in
the network.</p>
<p>We have seen so far a few ways of routing messages in a mesh network: Flooding
(See <a href="https://www.freedomlayer.org/the-mesh-question.html">The mesh
question</a> and
<a href="https://www.freedomlayer.org/sqrtn-mesh-routing.html">sqrt(n) routing</a>.</p>
<p>Flooding was pretty inefficient (However pretty robust and secure). The idea of
<span class="math">\(\sqrt{n}\)</span>-routing was a bit more efficient, though it required that every
node will maintain contact with <span class="math">\(\sqrt{n\log{n}}\)</span> virtual neighbours.</p>
<p>We will consider here a different approach for routing messages in a
decentralized mesh network: Using a “Virtual <span class="caps">DHT</span>”. This idea is based on the
articles <a href="http://research.microsoft.com/pubs/75325/virtualring.pdf">Virtual Ring Routing: Network Routing Inspired by
DHTs</a> and <a href="http://os.ibds.kit.edu/downloads/publ_2006_fuhrmann-ua_pushing-chord.pdf">Pushing
Chord into the Underlay: Scalable Routing for Hybrid
MANETs</a>.
This method is sometimes also called <a href="http://en.wikipedia.org/wiki/Scalable_Source_Routing">Scalable Source
Routing</a>.</p>
<p>A similar method apparently has a working implementation in the
<a href="http://en.wikipedia.org/wiki/Cjdns"><span class="caps">CJDNS</span></a> project. (Also see the
<a href="https://github.com/cjdelisle/cjdns/blob/master/doc/Whitepaper.md">whitepaper</a>)
I didn’t manage to figure out from <span class="caps">CJDNS</span> whitepaper the specific details of the
implementation, though it seems like it uses the Kademlia <span class="caps">DHT</span>. We are going to
use Chord in this text.</p>
<h4>Motivation for using a <span class="caps">DHT</span></h4>
<p>Recall that a <a href="https://www.freedomlayer.org/intro-to-distributed-hash-tables-dhts.html"><span class="caps">DHT</span></a> is
a special structure of links between nodes. This structure allows to find a
specific node (or key) quickly (In about <span class="math">\(\log{n}\)</span> iterations). If we
managed somehow to create a <span class="caps">DHT</span> between the nodes in the mesh network, we might
be able to find any wanted node quickly, and also pass messages quickly between nodes.</p>
<h4>Why can’t we just use a regular <span class="caps">DHT</span>?</h4>
<p>Why is this case different from our previous discussion about DHTs? We already
know how DHTs work, why can’t we just use a regular <span class="caps">DHT</span> to be able to navigate
messages in a network?</p>
<p>The answer is that creating a working <span class="caps">DHT</span> depends on our ability to send
messages “by address”. In other words, to have a working <span class="caps">DHT</span>, we need to rely
on some structure like the internet. Thus trying to build something like the
internet using a <span class="caps">DHT</span> is kind of cyclic, when you think about it.</p>
<p>Let me explain it in detail. Let’s recall how a <span class="caps">DHT</span> works. Whenever a node
<span class="math">\(x\)</span> wants to join the <span class="caps">DHT</span>, <span class="math">\(x\)</span> contacts some node <span class="math">\(y\)</span> that is already
part of the <span class="caps">DHT</span>. <span class="math">\(y\)</span> then finds the node closest to <span class="math">\(x\)</span> (By <span class="caps">DHT</span> Identity
distance) inside the <span class="caps">DHT</span>. Let’s call that node <span class="math">\(z\)</span>. <span class="math">\(y\)</span> then sends to <span class="math">\(x\)</span>
the address of <span class="math">\(z\)</span>. Finally, <span class="math">\(x\)</span> can connect to <span class="math">\(z\)</span> and join the <span class="caps">DHT</span>.</p>
<p><img alt="DHT Join" src="https://www.freedomlayer.org/articles/freedomlayer/exp_virtual_dht_routing/images/dht_join.svg"></p>
<p>In the picture: <span class="math">\(x\)</span> wants to join the network. Marked in gray is the place in
the ring where <span class="math">\(x\)</span> is supposed to be, right after <span class="math">\(z\)</span>.</p>
<p>This would work if we have a structure like the internet to support the idea of
address. However, if we are working in a simple mesh, how could <span class="math">\(y\)</span> connect
<span class="math">\(x\)</span> to <span class="math">\(z\)</span>? We don’t yet have any notion of address, so it is a hard task
for <span class="math">\(y\)</span> to introduce <span class="math">\(z\)</span> to <span class="math">\(x\)</span>.</p>
<p>The act of “introducing nodes to other nodes” is something that happens a lot
in a <span class="caps">DHT</span>. In particular, it happens whenever a node wants to join the network,
and also every time that a node performs the stabilize operation. </p>
<p>One naive method to introduce nodes in a mesh is using paths. Assume that <span class="math">\(y\)</span>
wants to introduce <span class="math">\(z\)</span> to the node <span class="math">\(x\)</span>. <span class="math">\(y\)</span> knows a path <span class="math">\(path(y,z)\)</span>
from <span class="math">\(y\)</span> to <span class="math">\(z\)</span>, and also a path <span class="math">\(path(y,x)\)</span> from <span class="math">\(y\)</span> to <span class="math">\(x\)</span>. <span class="math">\(y\)</span>
can use those two paths to construct a path <span class="math">\(path(x,y) + path(y,z)\)</span> from <span class="math">\(x\)</span> to <span class="math">\(z\)</span>.</p>
<p><span class="math">\(y\)</span> can send <span class="math">\(path(x,y) + path(y,z)\)</span> to <span class="math">\(x\)</span>. Then whenever <span class="math">\(x\)</span> wants to send a
message to <span class="math">\(z\)</span>, <span class="math">\(x\)</span> will use the path <span class="math">\(path(x,y) + path(y,z)\)</span>.</p>
<p><img alt="Concat Path" src="https://www.freedomlayer.org/articles/freedomlayer/exp_virtual_dht_routing/images/concat_path.svg"></p>
<p>In the picture: The path between <span class="math">\(x\)</span> and <span class="math">\(z\)</span> is the concatenation of paths
between <span class="math">\(x\)</span> and <span class="math">\(y\)</span>, <span class="math">\(y\)</span> and <span class="math">\(z\)</span>.</p>
<p>This method sounds like a solution, but it has a major flaw: The path
<span class="math">\(path(x,y) + path(y,z)\)</span> that we get from <span class="math">\(x\)</span> to <span class="math">\(z\)</span> is not optimal. It is
not the shortest path possible. What if <span class="math">\(x\)</span> wants to introduce <span class="math">\(z\)</span> to some
other node <span class="math">\(w\)</span>? If we keep using the naive paths methods, <span class="math">\(w\)</span> will end up
with a path <span class="math">\(path(w,x) + path(x,y) + path(y,z)\)</span> from <span class="math">\(w\)</span> to <span class="math">\(z\)</span>, which
could be pretty long.</p>
<p>After a while in the <span class="caps">DHT</span>, we might get to a situation where the paths become
too long to handle. In the next sections we will try to think of a solution.</p>
<h4>Setup</h4>
<p>To use a <span class="caps">DHT</span> we need <span class="caps">DHT</span> Identities for all the participants. For the rest of
this text we will use the Chord <span class="caps">DHT</span>, and we will choose a random Identities from
<span class="math">\(B_s = \{0,1,2,\dots,2^{s} - 1\}\)</span>, where <span class="math">\(s\)</span> is large enough. (In practice
we could choose the Identities in <a href="https://www.freedomlayer.org/basic-dht-security-concepts.html">some other
way</a>,
but let’s assume for now they are just random)</p>
<p>Between every two nodes <span class="math">\(x,y\)</span> (We use the notation <span class="math">\(x\)</span> to denote both the
Identity value and the node itself) we will define two notions of distance
(Which are unrelated):</p>
<ul>
<li>
<p>The Network Distance: The length of the shortest path between the nodes
<span class="math">\(x,y\)</span> in the network.</p>
</li>
<li>
<p>The Virtual Distance: The value <span class="math">\(d(x,y) = (y - x) \mod 2^{s}\)</span>. This is the
distance we when we described the Chord <span class="caps">DHT</span>.</p>
</li>
</ul>
<p><img alt="Mesh and Ring" src="https://www.freedomlayer.org/articles/freedomlayer/exp_virtual_dht_routing/images/mesh_and_ring.svg">
In the picture: On the right we can see an example for a mesh network. On the
left, we can see the Virtual <span class="caps">DHT</span>. Note that there is no relation between
distances on the mesh and distances in the Virtual <span class="caps">DHT</span>. The Virtual Distance
between <span class="math">\(x_{10}\)</span> and <span class="math">\(x_9\)</span> is pretty short, however in the mesh they are
very far.</p>
<p>Note that two nodes could have very short Network Distance though very long
Virtual Distance, and vice versa. It is crucial to understand that the Network
Distance and the Virtual Distance are unrelated.</p>
<h4>Converging the ring</h4>
<p>To maintain a Chord <span class="caps">DHT</span>, every node should at least know about its successor
with respect to the <span class="caps">DHT</span> Identity. (This will form the basic Chord ring).
We will begin by finding out some way in which every node will find a mesh path
to his successor and predecessor with respect to the <span class="caps">DHT</span> Identity. (Somewhat
surprisingly, finding just the successor is harder than finding both the
successor and predecessor at the same time).</p>
<p>Finding the immediate successor and predecessor for every node <span class="math">\(z\)</span> will allow
us to send a message (In a very inefficient way) between every two nodes
<span class="math">\(x\)</span>,<span class="math">\(y\)</span>. Assume that <span class="math">\(x\)</span> wants to send a message to <span class="math">\(y\)</span>. <span class="math">\(x\)</span> knows a
path to his <span class="caps">DHT</span> successor, <span class="math">\(x_1\)</span>. <span class="math">\(x\)</span> will send the message to <span class="math">\(x_1\)</span>, and
ask <span class="math">\(x_1\)</span> to pass the message to <span class="math">\(y\)</span>. <span class="math">\(x_1\)</span>, in turn, will send the
message through a known path to his successor on the ring, <span class="math">\(x_2\)</span>, and so on,
until the message will arrive at <span class="math">\(y\)</span>.</p>
<p>Why is this inefficient? First, only from the <span class="caps">DHT</span> (Virtual) perspective, the
message will go through about half the nodes in the <span class="caps">DHT</span>. Next, when looking at
the mesh, we find that passing the message from some node <span class="math">\(t\)</span> to his
successor goes through even more nodes in the mesh. How many nodes? It depends
on our mesh. If it’s a random graph, the distance between two random nodes
could be something like <span class="math">\(\log{n}\)</span>. If it’s more like a grid (This might
happen if you deploy wifi antennas on some large surface), then the average
path might be longer.
It seems like our message will go through more than <span class="math">\(n\)</span> nodes, which is too
much. Flooding could probably perform better.</p>
<p>But it is still a start. Maybe sending messages will be very inefficient, but
we still get a way to find a path from one node to another. It might give us
some ideas about how to move on. As a conclusion, our first task is finding the
successor and predecessor (With respect to the <span class="caps">DHT</span>) for every node <span class="math">\(x\)</span> in the network.</p>
<h5>Asking the neighbours</h5>
<p>Every node <span class="math">\(x\)</span> has only local knowledge of the network. In fact, a node <span class="math">\(x\)</span>
in the mesh network really knows nothing besides his immediate neighbours. We
may also assume that <span class="math">\(x\)</span> knows the <span class="caps">DHT</span> Identity of each of his neighbours.
(This information could be passed in a message between every two adjacent nodes
in the mesh).</p>
<p><img alt="Local Knowledge" src="https://www.freedomlayer.org/articles/freedomlayer/exp_virtual_dht_routing/images/local_knowledge.svg"></p>
<p>In the picture: The local knowledge of the node <span class="math">\(x_2\)</span>. All the unknown parts
of the network are colored gray.</p>
<p>For a node <span class="math">\(x\)</span>, we could traverse the whole network and find the best
successor and predecessor (With respect to the <span class="caps">DHT</span>), however this will be too
inefficient. Another idea would be to let <span class="math">\(x\)</span> stay in contact only with the
“best” nodes he knows. The nodes that have <span class="caps">DHT</span> Identity closest to <span class="math">\(x\)</span><span class="quo">‘</span>s <span class="caps">DHT</span> Identity.</p>
<p>How to do this? <span class="math">\(x\)</span> will begin from his neighbours. <span class="math">\(x\)</span> will pick <span class="math">\(k\)</span>
neighbours that minimize <span class="math">\(dist(x,z)\)</span>, and keep them in a set called
<span class="math">\(S_x\)</span>. Those are the best successors that <span class="math">\(x\)</span> knows. If you were wondering
about <span class="math">\(k\)</span>: we will have to choose it at some point. For now you may assume it
is some number like <span class="math">\(5\)</span>.</p>
<p>In the same way, <span class="math">\(x\)</span> will look at all of his neighbours, and pick <span class="math">\(k\)</span>
neighbours that minimize <span class="math">\(dist(z,x)\)</span>. (This is not the same as
<span class="math">\(dist(x,z)\)</span> !). We will denote those nodes as the set <span class="math">\(P_x\)</span>. This is the
set of the best predecessors <span class="math">\(x\)</span> knows so far.</p>
<p>Edge case: It might be possible that <span class="math">\(x\)</span> has less than <span class="math">\(k\)</span> neighbours. In
that case, the sets <span class="math">\(S_x,P_x\)</span> will both contain all the neighbours of <span class="math">\(x\)</span>.</p>
<p><span class="math">\(x\)</span> will also keep the path to each of the nodes inside the sets
<span class="math">\(S_x,P_x\)</span>. It doesn’t have much meaning in the first step of keeping the
immediate neighbours, but if <span class="math">\(x\)</span> wants to remember some other more distant
nodes, he will have to keep the path to them too.</p>
<p>After performing the initial step of filling in <span class="math">\(S_x\)</span> and <span class="math">\(P_x\)</span> from <span class="math">\(x\)</span><span class="quo">‘</span>s
neighbours, <span class="math">\(x\)</span> will keep performing the following step: For every node <span class="math">\(z\)</span>
in <span class="math">\(S_x \cup P_x\)</span>: <span class="math">\(x\)</span> will send to <span class="math">\(z\)</span> the set <span class="math">\(S_x \cup P_x\)</span>. In
other words: <span class="math">\(x\)</span> sends to all of his known best successor and predecessors
his set of best known successors and predecessors. (<span class="math">\(x\)</span> can send those
messages to all the nodes in <span class="math">\(S_x,P_x\)</span> because <span class="math">\(x\)</span> remembers a path to
those nodes.)</p>
<p>As every nodes performs this step, <span class="math">\(x\)</span> itself will get sets of nodes from
other nodes. <span class="math">\(x\)</span> will then look at the set of nodes he got and update his
sets <span class="math">\(S_x,P_x\)</span> accordingly. (For example: if better successor or predecessor
nodes were found. We prioritize “better” by first comparing virtual distance,
and then comparing network distance). </p>
<p>Note that every node <span class="math">\(z\)</span> that <span class="math">\(x\)</span> from some other node <span class="math">\(y\)</span> also contains
a path description, to make sure <span class="math">\(x\)</span> will be able to send messages to <span class="math">\(z\)</span>
in the future. This path might not be the shortest path from <span class="math">\(x\)</span> to <span class="math">\(z\)</span>.</p>
<p>We somehow expect that if every node performs this operation, after enough
iterations every node <span class="math">\(x\)</span> will have inside his sets <span class="math">\(S_x,P_x\)</span> the best
successor and the best predecessor (With respect to the <span class="caps">DHT</span>).</p>
<p>A simple words summary of what we have done here: Every node <span class="math">\(x\)</span> keeps at all
times a size-bounded set of the nodes closest to him with respect to the <span class="caps">DHT</span>.
Then at every iteration, the node <span class="math">\(x\)</span> sends to the set of nodes closest to
him the set of all nodes known to him. Finally, whenever a node <span class="math">\(x\)</span> gets a
set of nodes, he updates his set of closest nodes accordingly.</p>
<h5>An example</h5>
<p>Assume that <span class="math">\(k=3\)</span>, and that a node <span class="math">\(x\)</span> has the <span class="caps">DHT</span> Identity 349085. Also
assume that the set <span class="math">\(S_x\)</span> of <span class="math">\(x\)</span> currently contains:</p>
<p>$S_x = $ {<br />
(ident=359123,path_len=6),<br />
(ident=372115,path_len=4),<br />
(ident=384126,path_len=2)<br /> }</p>
<p>Next, assume that some node <span class="math">\(y\)</span> (of <span class="caps">DHT</span> Identity 384126) sent a message to
<span class="math">\(x\)</span> with the following set of nodes:</p>
<p>{ <br />
(ident=349085,path_len=2),<br />
(ident=372115,path_len=1),<br />
(ident=383525,path_len=2),<br />
(ident=391334,path_len=3),<br />
(ident=401351,path_len=4),<br />
(ident=412351,path_len=1) <br /> }</p>
<p><img alt="Before Iter Example" src="https://www.freedomlayer.org/articles/freedomlayer/exp_virtual_dht_routing/images/before_iter_example.svg"></p>
<p>In the picture: Schematic picture of the Virtual <span class="caps">DHT</span> ordering of all the nodes
mentioned above.</p>
<p>Some explanations: In the set of nodes we mention two values: ident and
path_len. Ident is the identity of the remote node, and path_len is the length
of the shortest network path known to the remote node. For clarity we don’t
show here the full description of the path, but that information should be kept too.</p>
<p>Another thing to note in this example is that <span class="math">\(y\)</span> is inside <span class="math">\(S_x\)</span>. You can
also find <span class="math">\(x\)</span> inside the set of nodes sent from <span class="math">\(y\)</span>. Although in this
example <span class="math">\(y \in S_x\)</span>, this will not always be the case. <span class="math">\(x\)</span> might get a
message from nodes that he doesn’t know, because of some referral from another
node that knows <span class="math">\(x\)</span>.</p>
<p>Let’s analyze what happens in the above example. <span class="math">\(x\)</span> receives the message,
and now he should update his sets <span class="math">\(S_x,P_x\)</span>. For clarity in this example we
only look at what happens to the set <span class="math">\(S_x\)</span>. The nodes with idents
391334,401351,412351 will not fit into <span class="math">\(S_x\)</span>, because all the nodes inside
<span class="math">\(S_x\)</span> have shorter virtual distance from <span class="math">\(x\)</span>. Therefore we are left to deal
with the nodes of <span class="caps">DHT</span> identities: 349085,372115,383525.</p>
<p>349085 is <span class="math">\(x\)</span><span class="quo">‘</span>s <span class="caps">DHT</span> Identity value, so <span class="math">\(x\)</span> can just discard it. 383525
seems to be a better choice than 384126 that <span class="math">\(x\)</span> currently have inside his
<span class="math">\(S_x\)</span> set. To put 383525 into <span class="math">\(S_x\)</span> we first have to calculate the path
from <span class="math">\(x\)</span> to 383525. <span class="math">\(x\)</span> knows the path from <span class="math">\(x\)</span> to <span class="math">\(y\)</span>. (It is of size
2 in this example). The length of the path known to <span class="math">\(y\)</span> from <span class="math">\(y\)</span> to 383525
is 2. (See the path_len argument of ident 383525). Therefore we get a total of
path length <span class="math">\(2+2 = 4\)</span>. Thus the new <span class="math">\(S_x\)</span> will now contain:</p>
<p><span class="math">\(S_x =\)</span> { <br />
(ident=359123,path_len=6), <br />
(ident=372115,path_len=4), <br />
(ident=383525,path_len=4) <br /> }</p>
<p>We are not done yet. We still have to deal with 372115. <span class="math">\(x\)</span> already has
372115 inside his set <span class="math">\(S_x\)</span>, but maybe <span class="math">\(x\)</span> could get a shorter path to
372115. <span class="math">\(y\)</span><span class="quo">‘</span>s network distance from 372115 is 1 (See the path_len argument
for 372115 inside the message sent from <span class="math">\(y\)</span>.) <span class="math">\(y\)</span><span class="quo">‘</span>s network distance from
<span class="math">\(x\)</span> is 2. Therefore we get a total path length of <span class="math">\(2+1=3\)</span> between <span class="math">\(x\)</span> and
372115. This new path length is shorter than the older path <span class="math">\(x\)</span> has to 372115.</p>
<p>In this case, <span class="math">\(x\)</span> will just update the path description to 372115 to be the
new shorter path. Finally we get the following <span class="math">\(S_x\)</span>:</p>
<p>$S_x = $ { <br />
(ident=359123,path_len=6), <br />
(ident=372115,path_len=3), <br />
(ident=383525,path_len=4) <br /> }</p>
<p>Note that after the transformation on the <span class="math">\(S_x\)</span> set, <span class="math">\(x\)</span> no longer has
<span class="math">\(y\)</span> inside <span class="math">\(S_x\)</span>. That’s the irony of life. <span class="math">\(y\)</span> sent <span class="math">\(x\)</span> so many “good”
nodes, that <span class="math">\(x\)</span> no longer needs <span class="math">\(y\)</span> as a node inside <span class="math">\(S_x\)</span>.</p>
<h5>Algorithm for a node: Converging the ring</h5>
<p>Let’s put our previous words into some more formal writing.
This is the algorithm for a node <span class="math">\(x\)</span> in the network:</p>
<p>Initialize:</p>
<ul>
<li>
<p>Initialize <span class="math">\(S_x\)</span> to be the <span class="math">\(k\)</span> nodes that minimize
<a href="http://en.wikipedia.org/wiki/Lexicographical_order">lexicographically</a> the
virtual distance <strong>from</strong> <span class="math">\(x\)</span> and the length of path known to <span class="math">\(x\)</span>.</p>
</li>
<li>
<p>Initialize <span class="math">\(P_x\)</span> to be the <span class="math">\(k\)</span> nodes that minimize
lexicographically the virtual distance <strong>to</strong> <span class="math">\(x\)</span> and the length of path
known to <span class="math">\(x\)</span>.</p>
</li>
</ul>
<p>Do every few seconds:</p>
<ul>
<li>For every node <span class="math">\(z\)</span> in <span class="math">\(S_x \cup P_x\)</span>:<ul>
<li>Send <span class="math">\(S_x \cup P_x\)</span> to <span class="math">\(z\)</span> (This includes paths description).</li>
</ul>
</li>
</ul>
<p>On arrival of a set of nodes <span class="math">\(T\)</span> from some node <span class="math">\(y\)</span>:</p>
<ul>
<li>For every node <span class="math">\(t\)</span> in <span class="math">\(T\)</span>:<ul>
<li>Figure out the full path from <span class="math">\(x\)</span> to <span class="math">\(t\)</span>.</li>
<li>Add <span class="math">\(t\)</span> to <span class="math">\(S_x\)</span> or <span class="math">\(P_x\)</span> if it is “better” than any other node in
one of those sets. Make sure that the sets <span class="math">\(S_x\)</span> and <span class="math">\(P_x\)</span> do not
exceed the size of <span class="math">\(k\)</span>.
Note that “better” means better with respect to the lexicographical order
of virtual distance and then network distance.</li>
</ul>
</li>
</ul>
<p>It could be a good point to show some code implementation, but hold on with
this for now. We first want to generalize this idea a bit. Also note that we
didn’t yet prove anything. This is generally an interesting idea, but we don’t
yet have any mathematical (Or even experimental) results that confirm the
correctness of this idea.</p>
<h4>Converging all the fingers</h4>
<p>Recall that to have a Chord <span class="caps">DHT</span> with a quick lookup function, every node <span class="math">\(x\)</span>
has to maintain not only his immediate successor and predecessor, but also the
best successor to <span class="math">\(x+2^{t}\)</span> for every <span class="math">\(0 \leq t < s\)</span>. (And symmetrically:
the best predecessor to <span class="math">\(x-2^{t}\)</span>). The value <span class="math">\(x+2^{t}\)</span> is called the
successor finger <span class="math">\(t\)</span> of <span class="math">\(x\)</span>. The value <span class="math">\(x-2^{t}\)</span> is called the
predecessor finger <span class="math">\(t\)</span> of <span class="math">\(x\)</span>. (Note: recall that the addition here is done
modulo <span class="math">\(2^s\)</span>).</p>
<p><img alt="DHT Fingers" src="https://www.freedomlayer.org/articles/freedomlayer/exp_virtual_dht_routing/images/dht_fingers.svg"></p>
<p>In the picture: The Chord <span class="caps">DHT</span> fingers. We can see both the successor fingers
and predecessor fingers.</p>
<p>So far we only dealt with the fingers <span class="math">\(x+1\)</span> and <span class="math">\(x-1\)</span>. Those are the
successor finger <span class="math">\(0\)</span> of <span class="math">\(x\)</span> and the predecessor finger <span class="math">\(0\)</span> of <span class="math">\(x\)</span>
respectively. We can probably generalize our idea to the rest of the fingers,
so that eventually every node <span class="math">\(x\)</span> will find the best successor of
<span class="math">\(x+2^{t}\)</span> and the best predecessor of <span class="math">\(x-2^{t}\)</span></p>
<p>Instead of letting every node <span class="math">\(x\)</span> keep the two sets <span class="math">\(S_x,P_x\)</span>, we will use
more sets, to keep the best candidates to every finger. We will call those sets
<span class="math">\(S_x^0,S_x^1,\dots,S_x^{s-1}\)</span> and <span class="math">\(P_x^0,P_x^1,\dots,P_x^{s-1}\)</span>. Every such
set will be bounded to size <span class="math">\(k\)</span>. Note that <span class="math">\(S_x^0\)</span> of the new notation is
exactly <span class="math">\(S_x\)</span> of the old notation (When we kept only the best successor).</p>
<p>Two examples: <span class="math">\(P_x^3\)</span> contains the best candidates to minimize
<span class="math">\(dist(z,x+2^{3})\)</span>. <span class="math">\(S_x^5\)</span> contains the best candidates to minimize
<span class="math">\(dist(x+2^{5},z)\)</span>.</p>
<p>This time, whenever a node <span class="math">\(x\)</span> gets a message from some other node <span class="math">\(y\)</span>
about a set of nodes, <span class="math">\(x\)</span> will update each of his <span class="math">\(S_x^i,P_x^i\)</span>
accordingly. (For <span class="math">\(0 \leq i < s\)</span>.</p>
<h5>Algorithm for a node: Converging the fingers</h5>
<p>We describe in a more detailed fashion the algorithm for fingers convergence
for one node. This algorithm is not very different from the one described
earlier. I write it here for completeness:</p>
<p>Initialize:</p>
<ul>
<li>
<p>for every <span class="math">\(0 \leq i \leq s\)</span>:</p>
<ul>
<li>Initialize <span class="math">\(S_x^i\)</span> to be the <span class="math">\(k\)</span> nodes that minimize
<a href="http://en.wikipedia.org/wiki/Lexicographical_order">lexicographically</a> the
virtual distance <strong>from</strong> <span class="math">\(x + 2^{i}\)</span> and the length of path known to <span class="math">\(x\)</span>.</li>
</ul>
</li>
<li>
<p>for every <span class="math">\(0 \leq i \leq s\)</span>:</p>
<ul>
<li>Initialize <span class="math">\(P_x^i\)</span> to be the <span class="math">\(k\)</span> nodes that minimize
lexicographically the virtual distance <strong>to</strong> <span class="math">\(x - 2^{i}\)</span> and the
length of path known to <span class="math">\(x\)</span>.</li>
</ul>
</li>
</ul>
<p>Do every few seconds:</p>
<ul>
<li>
<p>Denote <span class="math">\(close_x = \bigcup _{0 \leq i < s} \left(S_x^i \cup P_x^i\right)\)</span></p>
</li>
<li>
<p>For every node <span class="math">\(z\)</span> in <span class="math">\(close_x\)</span>:</p>
<ul>
<li>Send <span class="math">\(close_x\)</span> to <span class="math">\(z\)</span> (This includes paths description).</li>
</ul>
</li>
</ul>
<p>On arrival of a set of nodes <span class="math">\(T\)</span> from some node <span class="math">\(y\)</span>:</p>
<ul>
<li>For every node <span class="math">\(t\)</span> in <span class="math">\(T\)</span>:<ul>
<li>Figure out the full path from <span class="math">\(x\)</span> to <span class="math">\(t\)</span>.</li>
<li>for every <span class="math">\(0 \leq i < s\)</span>:<ul>
<li>Add <span class="math">\(t\)</span> to <span class="math">\(S_x^i\)</span> or <span class="math">\(P_x^i\)</span> if it is “better” than any other
node in one of those sets. Make sure that the sets <span class="math">\(S_x^i\)</span> and <span class="math">\(P_x^i\)</span>
do not exceed the size of <span class="math">\(k\)</span>.</li>
</ul>
</li>
</ul>
</li>
</ul>
<h4>Experimenting</h4>
<p>As noted above, we didn’t include any proof or confirmation of the correctness
of the presented ideas. Currently I don’t know of any rigorous mathematical
proof that shows that the methods here work.</p>
<p>You might be curious to know what could go wrong with the idea of converging
the fingers, for example. Let me address a few of my concerns here:</p>
<ul>
<li>
<p>Maybe the best successor (or predecessor) for some finger is never found: for
some node <span class="math">\(x\)</span> in the network, <span class="math">\(x\)</span><span class="quo">‘</span>s direct successor <span class="math">\(y\)</span> with respect
to the <span class="caps">DHT</span> is somewhere on the network, but <span class="math">\(x\)</span> never manages to find
<span class="math">\(y\)</span>.</p>
</li>
<li>
<p>It could be that the best successors and predecessors are eventually found, but the
paths to them are too long. If the paths are too long, we get higher
<a href="http://en.wikipedia.org/wiki/Latency_%28engineering%29">latency</a> for sending
messages, and it becomes harder to maintain a link to a remote node (As the
link existence depends on many nodes in the mesh being online at the same time).</p>
</li>
</ul>
<p>One thing that we can do to get some quick verification of our ideas is to
write simulation code. If the simulation code shows good results, it is
encouraging (Though it doesn’t prove anything mathematically). If, however, the
simulation shows bad results, we are pretty sure that we should try something else. </p>
<p>I wrote simulation code for the Fingers Convergence idea in Python3. <a href="https://github.com/realcr/freedomlayer_code/tree/master/exp_virtual_dht_routing">You can
find it here
[github]</a>
It’s not the fastest code out there, but it works, and it should be pretty easy
to understand and change.</p>
<p>You will need the <strong>networkx</strong> library to run this code. It could be obtained
by running:</p>
<div class="highlight"><pre><span></span><span class="n">pip</span> <span class="n">install</span> <span class="n">networkx</span>
</pre></div>
<p>Let’s begin with a quick summary of the results. </p>
<h5>Results for Random graphs</h5>
<p>I ran the code with networks of
sizes <span class="math">\(n=2^i\)</span> for <span class="math">\(i \in \{ 6,7,8,9,10,11\}\)</span>. (<span class="math">\(i=12\)</span> was taking too long,
so I terminated it). For all of those network sizes, the algorithm terminated
with one iteration of all nodes.</p>
<p>The networks were built as <a href="http://en.wikipedia.org/wiki/Erd%C5%91s%E2%80%93R%C3%A9nyi_model">Erdos
Renyi</a> random
graphs. Every edge has a probability <span class="math">\(p = \frac{2i}{2^i}\)</span> to exist. I chose
<span class="math">\(k = i\)</span>. Recall that <span class="math">\(k\)</span> is the size of set being maintained for each
finger. In the python source code, <span class="math">\(k\)</span> shows up as <span class="math">\(fk\)</span>.</p>
<p><img alt="gnp all fingers results" src="https://www.freedomlayer.org/articles/freedomlayer/exp_virtual_dht_routing/images/conv_gnp_all_fingers_results.svg"></p>
<p>Results for Random Erdos-Renyi networks. Horizontal axis is <span class="math">\(i\)</span>. Vertical
axis is the average network path length to the best finger candidate.</p>
<p>It’s hard to say what is going to happen for <span class="math">\(i=40\)</span> from those results, but
for the smaller size of <span class="math">\(i=11\)</span> this solution seems to be practical.</p>
<p>Considering a random graph of the Erdos-Renyi model with <span class="math">\(n=2^{11}\)</span> nodes
and <span class="math">\(p = \frac{2i}{2^i}\)</span>, we get that the average path from a node <span class="math">\(x\)</span> to
some best finger candidate is about <span class="math">\(3.3\)</span>. Routing a message in a <span class="caps">DHT</span> of size
<span class="math">\(2^{11}\)</span> will usually take no more than <span class="math">\(11\)</span> <span class="caps">DHT</span> hops. As each of those <span class="caps">DHT</span>
hops will go through an average of <span class="math">\(3.3\)</span> nodes in the mesh network, we get a
total network path length of <span class="math">\(3.3 \cdot 11 = 36.3\)</span>.</p>
<p>If our network is an overlay mesh network over the internet (Like in
<a href="https://bitcoin.org">Bitcoin</a>), then we expect every every hop should take no
longer than half a second. We get a total of <span class="math">\(36.3 \cdot 0.5 = 18.15\)</span> seconds.</p>
<p>Note that I only consider here overlay mesh network and not a real mesh
network, because the Erdos-Renyi Random graph model probably doesn’t fit a mesh
network deployed on some surface.</p>
<p><span class="math">\(18.15\)</span> sounds like a long time to me at this point, but it is probably
possible to get better results with some improvements or heuristics. (Also note
that I stopped the algorithm after it found all the best candidates for all the
fingers. Maybe if I let it run some more, I could get shorter path lengths).</p>
<h5>Results for Grid Graphs</h5>
<p>Real mesh networks will usually not be random graphs (As in the Erdos-Renyi
model). They will probably be more grid like. I ran the code with 2-dimensional
grid networks of estimated sizes <span class="math">\(n=2^i\)</span> for <span class="math">\(i \in \{ 6,7,8,9,10\}\)</span>. (The
sizes are estimated because a square grid has to be of size <span class="math">\(m\times m\)</span>. I
chose <span class="math">\(m\)</span> to be the integral part of <span class="math">\(\sqrt{n}\)</span>.</p>
<p><img alt="grid all fingers results" src="https://www.freedomlayer.org/articles/freedomlayer/exp_virtual_dht_routing/images/conv_grid_all_fingers_results.svg"></p>
<p>Results for grid networks. Horizontal axis is <span class="math">\(i\)</span>. Vertical axis is the
average network path length to the best finger candidate.</p>
<p>Note that in the grid networks we get much larger values for the average path
length. Generally, in grid networks the shortest path between two arbitrary
nodes could be pretty long. On the other hand, on a real mesh network we expect
that every network hop is much faster. (compared to the case of an overlay mesh
network that works above the Internet).</p>
<h5>Modifying parameters</h5>
<p>I want you to get a feeling of how the simulation code works, and how to make
modifications to this code, so that you can do your own experiments. You might
have some ideas that I didn’t think about. If you manage to get
some interesting results, or find bugs, please tell me about it. </p>
<p>Most of the things you will want to change are probably inside the go()
function. This is where you put the parameters for the simulation. This is how
it should look like:</p>
<div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">go</span><span class="p">():</span>
<span class="n">i</span> <span class="o">=</span> <span class="mi">11</span>
<span class="n">ident_bits</span> <span class="o">=</span> <span class="n">math</span><span class="o">.</span><span class="n">ceil</span><span class="p">(</span><span class="n">i</span><span class="o">*</span><span class="mf">2.6</span><span class="p">)</span>
<span class="n">fk</span> <span class="o">=</span> <span class="n">i</span>
<span class="c1"># Fingers we are interested in:</span>
<span class="n">succ_fingers</span> <span class="o">=</span> <span class="nb">list</span><span class="p">(</span><span class="nb">range</span><span class="p">(</span><span class="n">ident_bits</span><span class="p">))</span>
<span class="n">pred_fingers</span> <span class="o">=</span> <span class="nb">list</span><span class="p">(</span><span class="nb">range</span><span class="p">(</span><span class="n">ident_bits</span><span class="p">))</span>
<span class="c1"># succ_fingers = [0]</span>
<span class="c1"># pred_fingers = [0]</span>
<span class="k">print</span><span class="p">(</span><span class="s2">"||| i ="</span><span class="p">,</span><span class="n">i</span><span class="p">)</span>
<span class="k">print</span><span class="p">(</span><span class="s2">"||| ident_bits ="</span><span class="p">,</span><span class="n">ident_bits</span><span class="p">)</span>
<span class="k">print</span><span class="p">(</span><span class="s2">"||| fk ="</span><span class="p">,</span><span class="n">fk</span><span class="p">)</span>
<span class="k">print</span><span class="p">(</span><span class="s2">"||| succ_fingers = "</span><span class="p">,</span><span class="n">succ_fingers</span><span class="p">)</span>
<span class="k">print</span><span class="p">(</span><span class="s2">"||| pred_fingers = "</span><span class="p">,</span><span class="n">pred_fingers</span><span class="p">)</span>
<span class="k">print</span><span class="p">(</span><span class="s2">"Generating graph..."</span><span class="p">)</span>
<span class="n">g</span> <span class="o">=</span> <span class="n">gen_grid_graph</span><span class="p">(</span><span class="n">i</span><span class="p">)</span>
<span class="c1"># g = gen_gnp_graph(i)</span>
<span class="k">print</span><span class="p">(</span><span class="s2">"Generating Network..."</span><span class="p">)</span>
<span class="n">vd</span> <span class="o">=</span> <span class="n">VirtualDHT</span><span class="p">(</span><span class="n">graph</span><span class="o">=</span><span class="n">g</span><span class="p">,</span><span class="n">fk</span><span class="o">=</span><span class="n">fk</span><span class="p">,</span>\
<span class="n">dht_fingers</span><span class="o">=</span><span class="p">(</span><span class="n">succ_fingers</span><span class="p">,</span><span class="n">pred_fingers</span><span class="p">),</span>\
<span class="n">ident_bits</span><span class="o">=</span><span class="n">ident_bits</span><span class="p">)</span>
<span class="k">print</span><span class="p">(</span><span class="s2">"Initiating convergence...</span><span class="se">\n</span><span class="s2">"</span><span class="p">)</span>
<span class="n">vd</span><span class="o">.</span><span class="n">converge</span><span class="p">(</span><span class="n">max_iters</span><span class="o">=</span><span class="mh">0x80</span><span class="p">)</span>
</pre></div>
<p><strong>i</strong> is a parameter that represents the size of a network. For a given value
i, we say that the size of the network is <span class="math">\(n=2^i\)</span>.</p>
<p><strong>ident_bits</strong> is the amount of bits used to describe the identity of a node.
We could just pick a large number like <span class="math">\(160\)</span> bits and never change it (This
is what we are going to do in a real network), but it will make the simulation
pretty slow, as every node will have to maintain <span class="math">\(160\)</span> fingers. Therefore for
simulation purposes we are going to pick the value of ident_bits according to
the amount of nodes in the network. We choose it to be about <span class="math">\(i \cdot 2.6\)</span>.</p>
<p>Why <span class="math">\(2.6\)</span>? Because we don’t want two nodes to have the same <span class="caps">DHT</span> Identity
value. Choosing <span class="math">\(2\)</span> should have been a good bet (See the <a href="http://en.wikipedia.org/wiki/Birthday_problem">birthday
paradox</a>, but I wanted to be on
the safe side of things, so I picked a bigger value: <span class="math">\(2.6\)</span>.</p>
<p>The <strong>fk</strong> parameter is the same as <span class="math">\(k\)</span> we mentioned in our discussions
earlier. It determines the size of the sets <span class="math">\(S_x^j,P_x^j\)</span>. In other words, it
determines the size of the candidates set for each finger. For example: if
<span class="math">\(fk=1\)</span> then every node <span class="math">\(x\)</span> only remembers the best candidate he has seen
for every finger.</p>
<p><strong>succ_fingers</strong> and <strong>pred_fingers</strong> are the set of fingers every node is
going to maintain. If we pick [0] for both of those parameters (It is commented
out in the code), then every node will only maintain his immediate successor
and predecessor. This will give us the basic Chord ring, as we discussed in
“Convering the ring” part.</p>
<p>In the code above we picked the value list(range(ident_bits)),
which is all the fingers. We need to know all the best candidates for all the
fingers if we want to be able to route messages in the resulting <span class="caps">DHT</span>.</p>
<p><strong>g</strong> is network graph. Remember that at the results above we mentioned both
Random graphs and Grid graphs? Changing g lets you change the network layout.
There are already two graph models inside the code: gen_gnp_graph() and
gen_grid_graph(), However you can use any other model of graph that you want to
test. Just create some networkx graph, and assign it to g.</p>
<h5>Understanding the output</h5>
<p>The vd.converge() function will put some output to the console. I want to
explain that output. Let’s see an example:</p>
<div class="highlight"><pre><span></span>||| i = 9
||| ident_bits = 24
||| fk = 9
||| succ_fingers = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23]
||| pred_fingers = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23]
Generating graph...
Generating Network...
Initiating convergence...
iter number | Fingers verified | path_len verified | avg path_len
--------------------------------------------------------------------
0 | False | False | 19.380424
1 | True | False | 15.708721
</pre></div>
<p>The first lines (Beginning with 3 pipe characters) show some of the parameters
used. Sometime I run experiments for days, and it is useful to know what
parameters caused the results I see on my terminal.</p>
<p>Next there are some notifications about Generation of the graph, network and
initialization of convergence process. We have already seen those print
functions in the code above.</p>
<p>Finally there is a table that describes the results from the iterations. Every
row in this table represents one iteration of the convergence algorithm. (Every
node sends to all the nodes he knows a set of all the nodes he knows, and in
turn every node updates his best known finger candidates accordingly).</p>
<p>The row “iter number” just counts the iterations. (It begins from 0). “Fingers
verified” tells you if every node has found the best finger candidate already.
The algorithm is designed to stop as soon as “Fingers verified” becomes True, but
you can change this behaviour. (Check the converge() function if you want to do that).</p>
<p><span class="dquo">“</span>path_len verified” means that every for every node <span class="math">\(x\)</span>, if <span class="math">\(x\)</span> knows some
best candidate <span class="math">\(y\)</span> for some finger <span class="math">\(f\)</span>, then <span class="math">\(x\)</span> also knows the shortest path
possible to <span class="math">\(y\)</span>. I added it to check if I manage to get the shortest path
possible using some methods. Most of the time (Maybe except for some really
small networks), this value will be False.</p>
<p><span class="dquo">“</span>avg path_len” is the average path length from some node <span class="math">\(x\)</span> to his best
candidate for finger <span class="math">\(f\)</span>. This value is not exact. It is calculated by
sampling some known amount of distances, and finding their average. (This means
that you might see small changes in this value, but in fact nothing changes in
the network).</p>
<h5>Some modification attempts</h5>
<p>I introduce here some modifications I tried. It might be of interest.</p>
<ul>
<li>
<p>If <span class="math">\(fk=1\)</span>. (<span class="math">\(fk\)</span> is the size of the set of candidates for
some finger), the algorithm doesn’t find a solution for large enough networks.</p>
</li>
<li>
<p>If <span class="math">\(fk=\frac{i}{2}\)</span> the simulations run a bit faster, but we get longer
average path lengths. (We might not get a solution for large enough networks,
but I haven’t seen any evidence for that).</p>
</li>
<li>
<p>Choosing succ_fingers = [0], pred_fingers = [] doesn’t converge, however
choosing succ_fingers = [0] and pred_fingers = [0] does converge. (This is
true both for gnp and grid graphs). In the first case: Every node searches
for his best successor. In the second case: Every node searches for his best
successor and best predecessor at the same time.</p>
</li>
</ul>
<h4>Summary</h4>
<p>We wanted to route messages in a mesh network. For this purpose, we gave every
node some random name and created a <span class="caps">DHT</span> from those random names. The random
names given to the nodes on the network are not related to the underlying
structure of the mesh network. We end up with two distances: A network
distance, which is the physical amount of hops between nodes in the mesh, and a
Virtual Distance: A <span class="caps">DHT</span> namespace distance between names of two nodes.</p>
<p>We introduced a pretty crude way of finding the best Chord <span class="caps">DHT</span> finger
candidates for every node in the network. It seems to be working, though we are
not sure at this point if it scales well to bigger networks. In its raw form it
does seems to be on the edge of “practical” for networks of size <span class="math">\(2^{11}\)</span>.</p>
<p>We don’t have any formal justification for this method. Some more research is required.</p>
<p>Things we haven’t discussed here: Dealing with security and reliability when
using a Virtual <span class="caps">DHT</span>. For example: Can we overcome nodes that fail? How to deal
with Sybil attacks?</p>
<script type="text/javascript">if (!document.getElementById('mathjaxscript_pelican_#%@#$@#')) {
var align = "center",
indent = "0em",
linebreak = "false";
if (false) {
align = (screen.width < 768) ? "left" : align;
indent = (screen.width < 768) ? "0em" : indent;
linebreak = (screen.width < 768) ? 'true' : linebreak;
}
var mathjaxscript = document.createElement('script');
mathjaxscript.id = 'mathjaxscript_pelican_#%@#$@#';
mathjaxscript.type = 'text/javascript';
mathjaxscript.src = 'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.0/MathJax.js?config=TeX-AMS-MML_HTMLorMML';
mathjaxscript[(window.opera ? "innerHTML" : "text")] =
"MathJax.Hub.Config({" +
" config: ['MMLorHTML.js']," +
" TeX: { extensions: ['AMSmath.js','AMSsymbols.js','noErrors.js','noUndefined.js'], equationNumbers: { autoNumber: 'AMS' } }," +
" jax: ['input/TeX','input/MathML','output/HTML-CSS']," +
" extensions: ['tex2jax.js','mml2jax.js','MathMenu.js','MathZoom.js']," +
" displayAlign: '"+ align +"'," +
" displayIndent: '"+ indent +"'," +
" showMathMenu: true," +
" messageStyle: 'normal'," +
" tex2jax: { " +
" inlineMath: [ ['\\\\(','\\\\)'] ], " +
" displayMath: [ ['$$','$$'] ]," +
" processEscapes: true," +
" preview: 'TeX'," +
" }, " +
" 'HTML-CSS': { " +
" styles: { '.MathJax_Display, .MathJax .mo, .MathJax .mi, .MathJax .mn': {color: 'inherit ! important'} }," +
" linebreaks: { automatic: "+ linebreak +", width: '90% container' }," +
" }, " +
"}); " +
"if ('default' !== 'default') {" +
"MathJax.Hub.Register.StartupHook('HTML-CSS Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax['HTML-CSS'].FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"MathJax.Hub.Register.StartupHook('SVG Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax.SVG.FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"}";
(document.body || document.getElementsByTagName('head')[0]).appendChild(mathjaxscript);
}
</script>Sqrt(n) mesh routing2014-12-01T20:22:00+00:002014-12-01T20:22:00+00:00realtag:www.freedomlayer.org,2014-12-01:/sqrtn-mesh-routing.html<!--
ceil and floor latex macros:
-->
<div class="math">$$
\newcommand{\ceil}[1]{\left\lceil{#1}\right\rceil}
\newcommand{\floor}[1]{\left\lfloor{#1}\right\rfloor}
$$</div>
<h4>Abstract</h4>
<p>We introduce a message routing mechanism in which every node has to maintain
contact with a bit more than <span class="math">\(\sqrt{n}\)</span> random nodes in the network. Given
that set up, every two nodes …</p><!--
ceil and floor latex macros:
-->
<div class="math">$$
\newcommand{\ceil}[1]{\left\lceil{#1}\right\rceil}
\newcommand{\floor}[1]{\left\lfloor{#1}\right\rfloor}
$$</div>
<h4>Abstract</h4>
<p>We introduce a message routing mechanism in which every node has to maintain
contact with a bit more than <span class="math">\(\sqrt{n}\)</span> random nodes in the network. Given
that set up, every two nodes can form a short communication path between them
that allows to send and receive messages. This mechanism is a bit more
efficient than flooding, but it is expected to work well only on some kinds of networks.</p>
<p>This article was inspired by the ideas in <a href="http://pdos.csail.mit.edu/papers/sybil-dht-socialnets08.pdf">A sybil proof one hop
<span class="caps">DHT</span></a> by Chris Lesniewski-Laas. </p>
<h4>The message routing problem</h4>
<p>Assume that we have a set of <span class="math">\(n\)</span> correct nodes (No adversary). Each
node is directly connected to some other nodes (About <span class="math">\(log(n)\)</span> direct
connections). For a node <span class="math">\(x\)</span>, we call every other node <span class="math">\(y\)</span> that is directly
connected to <span class="math">\(x\)</span> an immediate <strong>neighbour</strong> (We will sometimes write only “neighbour”).</p>
<p>Given two arbitrary nodes <span class="math">\(a\)</span> and <span class="math">\(b\)</span> in the network, we want to be able to
route a message from <span class="math">\(a\)</span> to <span class="math">\(b\)</span>. As <span class="math">\(a\)</span> and <span class="math">\(b\)</span> are possibly not
neighbours, we might need to transfer the message through some intermediate
nodes until it finally reaches its destination, <span class="math">\(b\)</span>.</p>
<p>In <a href="https://www.freedomlayer.org/the-mesh-question.html">The mesh question
article</a> we already
discussed the
<a href="http://en.wikipedia.org/wiki/Flooding_%28computer_networking%29">flooding</a>
solution. We have seen that it works, but it is not very efficient: Every
message sent in the network has to pass through all the nodes in the network.</p>
<h4>Virtual Neighbours</h4>
<p>We noted above that in our network, every node <span class="math">\(x\)</span> has a small set of
immediate neighbours. We can extend this idea of neighbours by keeping contact
with farer nodes that are not necessarily immediate neighbours.</p>
<p>We say that a <strong>path</strong> is a consecutive set of connected nodes.
If a node <span class="math">\(x\)</span> knows a path to some other node <span class="math">\(y\)</span>, <span class="math">\(x\)</span> could send
messages to <span class="math">\(y\)</span> through that path, assuming that all the nodes on that path cooperate.</p>
<p><img alt="Path from x to y" src="https://www.freedomlayer.org/articles/freedomlayer/sqrt_n_routing/images/path_x_to_y.svg"></p>
<p>In the picture: Nodes are are blue points. Immediate links between nodes are
drawn as black lines. The green lines mark a chosen path of links between the
nodes <span class="math">\(x\)</span> and <span class="math">\(y\)</span>.</p>
<p>It might be very likely that <span class="math">\(y\)</span> will want to send a message back to <span class="math">\(x\)</span>.
(After all, they are having a conversation). In that case, <span class="math">\(y\)</span> will need to
know some path to send the message back to <span class="math">\(x\)</span>. It could be exactly the same
path from <span class="math">\(x\)</span> to <span class="math">\(y\)</span> (But reversed), or some other path.</p>
<p><span class="math">\(x\)</span> and <span class="math">\(y\)</span> will maintain contact by sending periodic
<a href="http://en.wikipedia.org/wiki/Heartbeat_%28computing%29">heartbeats</a> to each
other, making sure that the remote side is alive. In that case we will say that
<span class="math">\(x\)</span> and <span class="math">\(y\)</span> are <strong>virtual neighbours</strong>. (<span class="math">\(x\)</span> is a virtual neighbour of
<span class="math">\(y\)</span>, and vice versa).</p>
<p>The longer the path between <span class="math">\(x\)</span> and <span class="math">\(y\)</span>, the harder it becomes for them to
stay in contact. That is because the link between <span class="math">\(x\)</span> and <span class="math">\(y\)</span> relies on all
the nodes on the path used between <span class="math">\(x\)</span> and <span class="math">\(y\)</span>.</p>
<h4>A Common Virtual Neighbour</h4>
<p>Let’s assume that every node in the network has some virtual neighbours.</p>
<p>In addition, let’s imagine that somehow, every two nodes <span class="math">\(a\)</span> and <span class="math">\(b\)</span> in the
network have a virtual neighbour <span class="math">\(z\)</span> in common. In that case we can route
messages between any two nodes <span class="math">\(a\)</span> and <span class="math">\(b\)</span> in that network:</p>
<p>Assume that we are at node <span class="math">\(a\)</span>, and we want to send a message to node <span class="math">\(b\)</span>.
We first send a message to each of <span class="math">\(a\)</span><span class="quo">‘</span>s virtual neighbours, asking them if
they know <span class="math">\(b\)</span> (As their virtual neighbour). One of <span class="math">\(a\)</span><span class="quo">‘</span>s virtual
neighbours, <span class="math">\(z\)</span>, should have <span class="math">\(b\)</span> as a virtual neighbour. Then we could send
a message to <span class="math">\(z\)</span>, and <span class="math">\(z\)</span> will forward that message to <span class="math">\(b\)</span>.</p>
<p><img alt="Z Common virtual neighbour" src="https://www.freedomlayer.org/articles/freedomlayer/sqrt_n_routing/images/z_common_virtual_neighbour.svg"></p>
<p>In the picture: <span class="math">\(a\)</span> and <span class="math">\(b\)</span> are two nodes in the network. The lines from
<span class="math">\(a\)</span> and <span class="math">\(b\)</span> are paths to virtual neighbours of <span class="math">\(a\)</span> and <span class="math">\(b\)</span>. <span class="math">\(z\)</span> is
both a virtual neighbour of <span class="math">\(a\)</span> and a virtual neighbour of <span class="math">\(b\)</span>. With the
help of <span class="math">\(z\)</span>, <span class="math">\(a\)</span> and <span class="math">\(b\)</span> can communicate.</p>
<p>To make this solution valid, we have to make sure somehow that every two nodes
in the network have a virtual neighbour in common.</p>
<h4>Using Random Virtual Neighbours</h4>
<p>One simple approach would be to try randomize a set of virtual neighbours for
every node on the network, and hope for the best. We should still decide how
many virtual nodes every node is going to have, and how to randomize them.</p>
<p>Assume for a while that we do have a method of choosing a random virtual
neighbour. Given that ability, all that is left is to choose how many random
virtual neighbours every node will have. Knowing about the <a href="http://en.wikipedia.org/wiki/Birthday_problem">birthday paradox</a>, we can assume that we are going
to need at least <span class="math">\(\sqrt{n}\)</span> virtual neighbours per node.</p>
<p>Assume that every node has a set of <span class="math">\(r\)</span> virtual neighbours. Let <span class="math">\(a\)</span> be some
node in the network, and we want to send a message from <span class="math">\(a\)</span> to <span class="math">\(b\)</span>. What
are the odds of finding a path to <span class="math">\(b\)</span> using the random virtual neighbours?</p>
<p><span class="math">\(a\)</span> will ask each of his virtual neighbours if they know <span class="math">\(b\)</span>. <span class="math">\(a\)</span> has
<span class="math">\(r\)</span> virtual neighbours, and each of those neighbours knows about <span class="math">\(r\)</span>
virtual neighbours. The probability that some specific virtual neighbour of
<span class="math">\(a\)</span> doesn’t know about <span class="math">\(b\)</span> is:</p>
<div class="math">$$1 - \frac{r}{n}$$</div>
<p>Thus the probability that all of <span class="math">\(a\)</span><span class="quo">‘</span>s virtual neighbours don’t know <span class="math">\(b\)</span> is: </p>
<div class="math">$$p = \left(1 - \frac{r}{n}\right)^r \leq \left(e^{-\frac{r}{n}}\right)^r =
e^{\frac{-r^2}{n}}$$</div>
<p>The first inequality is derived from the inequality <span class="math">\(1 + x \leq
e^x\)</span>.</p>
<p>We want to get a value for <span class="math">\(p\)</span> that is very close to <span class="math">\(0\)</span>, and becomes
smaller as <span class="math">\(n\)</span> increases.
Choosing <span class="math">\(r=\sqrt{n}\)</span> will give us a constant bound over <span class="math">\(p\)</span>, which is not
what we want. We should increase <span class="math">\(r\)</span> a bit. We could pick <span class="math">\(r=\sqrt{n
\log{n}}\)</span>. In that case we get that <span class="math">\(p \leq e^{\frac{-r^2}{n}} = e^{-\log{n}}
= \frac{1}{n}\)</span>, which is what we wanted.</p>
<p>We conclude that to be able to send a message from any node to any other node
in the network, it is enough to make sure that every node in the network
maintains <span class="math">\(r = \sqrt{n \log{n}}\)</span> randomly chosen virtual neighbours.</p>
<h4>Randomizing Virtual Neighbours</h4>
<p>As one of the nodes in the network, Getting random virtual neighbours is
nontrivial. This is mostly because a node in the network doesn’t have a full
view of the network. He only knows about his immediate neighbours.</p>
<p>One approach to get a random virtual neighbour is <strong>random walking</strong>. Random
walking in a graph (Or a network) means that we begin from some node, and in
every iteration we pick a neighbour randomly, and move to that neighbour.</p>
<p><img alt="A random walk in a graph" src="https://www.freedomlayer.org/articles/freedomlayer/sqrt_n_routing/images/random_walk_graph.svg"></p>
<p>In the picture: An example for a random walk on a graph. Follow the numbers to
get a feeling of the random walk. (Begin from 1). Note that some vertices have
more than one number next to them. That is because the random walk has visited
those vertices more than once.</p>
<p>Can we really get to random nodes in the graph using this method?
It turns out that if we walk a really long time on a graph, the <a href="http://www.cs.elte.hu/~lovasz/erdos.pdf">probability to
end up on a node is proportional to his graph
degree</a>. This is pretty random if most
nodes have the same amount of immediate neighbours.</p>
<p>Next, we want to figure out how many steps we should random walk before we get
close enough to the stated distribution. The amount of steps required is also
known as the <a href="http://en.wikipedia.org/wiki/Markov_chain_mixing_time">mixing
time</a> of the system. </p>
<p>For some graphs we will have to random walk for a very long time before we get
to somehow random nodes. In other words: Some graphs have a very large mixing
time. We will be generous and assume that our network has a small mixing time.
About <span class="math">\(\log{n}\)</span> steps. In another formulation, we will say that it is <strong>Fast
Mixing</strong>. How can we know that our network is really fast mixing? We don’t
know, but we will assume it is for now, and see what we can do with it.</p>
<h5>Getting Random nodes by random walking</h5>
<p>Given the fast mixing assumption on our network, we can obtain random virtual
neighbours using the following method:</p>
<p>A node <span class="math">\(x\)</span> will generate a “random walking message” <span class="math">\(m\)</span>. It will then
include his identity on the message, and send <span class="math">\(m|x\)</span> to one of his immediate
neighbours (He will pick one randomly). Assume that the immediate neighbour
<span class="math">\(x_1\)</span> was picked. <span class="math">\(x_1\)</span> will then forward the message to some random
immediate neighbour of his, <span class="math">\(x_2\)</span>. <span class="math">\(x_1\)</span> will send <span class="math">\(x_2\)</span> the message
<span class="math">\(m|x|x_1\)</span>. <span class="math">\(x_2\)</span> will send the message <span class="math">\(m|x|x_1|x_2\)</span> to a random
immediate neighbour of his, <span class="math">\(x_3\)</span>, and so on.</p>
<p>The message forwarding will stop after <span class="math">\(\log{n}\)</span> iterations. (The message
contains the full path history, so it is possible to know the path length).
Then the last node that got the random walking message, <span class="math">\(x_w\)</span>, will send a
message to <span class="math">\(x\)</span> along the reversed path. <span class="math">\(x_w\)</span> and <span class="math">\(x\)</span> will then become
virtual neighbours.</p>
<p>As we assume that the network is fast mixing, <span class="math">\(x_w\)</span> should be a random node
in the network, and therefore we found a way to get random virtual neighbours
for the node <span class="math">\(x\)</span>.</p>
<p><span class="math">\(x\)</span> will repeat the same process, to obtain a total of <span class="math">\(r =
\sqrt{n\log{n}}\)</span> virtual neighbours. <span class="math">\(x\)</span> will also have to maintain those
virtual neighbours by sending (and responding to) periodic heartbeats. If any
of those virtual neighbours fail, <span class="math">\(x\)</span> will find a new virtual neighbour.</p>
<h5>Putting everything together</h5>
<p>Let’s review our full solution. Every node sends “random walk messages” to
obtain a total of <span class="math">\(r = \sqrt{n\log{n}}\)</span>. Then whenever a node <span class="math">\(a\)</span> wants to
send a message to some other node <span class="math">\(b\)</span>, <span class="math">\(a\)</span> will ask all of its virtual
neighbours if they have <span class="math">\(b\)</span> as a virtual neighbour. With high probability
(more than <span class="math">\(1 - \frac{1}{n}\)</span>) one of <span class="math">\(a\)</span><span class="quo">‘</span>s virtual neighbours, <span class="math">\(x\)</span>, will have
<span class="math">\(b\)</span> as a virtual neighbour. Then <span class="math">\(a\)</span> already knows a path of nodes from
<span class="math">\(a\)</span> to <span class="math">\(x\)</span>, and <span class="math">\(x\)</span> knows a path of nodes from <span class="math">\(x\)</span> to <span class="math">\(b\)</span>. Using
information from <span class="math">\(x\)</span>, <span class="math">\(a\)</span> could learn about a path from <span class="math">\(a\)</span> to <span class="math">\(b\)</span>.
That path could be used to send messages between <span class="math">\(a\)</span> and <span class="math">\(b\)</span>. Note that the
path between <span class="math">\(a\)</span> and <span class="math">\(b\)</span> will be of length about <span class="math">\(2\cdot\log{n}\)</span>, which
is not very long.</p>
<h4>Practicality questions</h4>
<p>In the proposal above we get that every node has to maintain contact with about
<span class="math">\(r = \sqrt{n\log{n}}\)</span> virtual nodes. That means that every node <span class="math">\(x\)</span> in the
network has to receive and send heartbeats periodically to <span class="math">\(r\)</span> nodes.</p>
<p>In addition, whenever a node <span class="math">\(a\)</span> wants to set up a path to some other node
<span class="math">\(b\)</span>, <span class="math">\(a\)</span> will first ask all of its <span class="math">\(r\)</span> virtual neighbours if they know
<span class="math">\(a\)</span>. </p>
<p>For big values of <span class="math">\(n\)</span>, this could get impractical. For example,
If <span class="math">\(n = 2^{40}\)</span>, we get that <span class="math">\(r = 2^{20}\cdot 20 \approx 2^{24}\)</span>. Sending
<span class="math">\(2^{24}\)</span> messages to form connection with a remote node could be too much. In
addition, every node has to keep sending periodic heartbeats to his <span class="math">\(2^{24}\)</span>
virtual neighbours. This could be expensive.</p>
<p>A further question might be regarding the fast mixing nature of networks. You
might be wondering if the network in our case is expected to be fast mixing.
(In other words - It has the property that random walks produce random nodes
pretty quickly). We don’t have an exact answer for this one.</p>
<p>If our network looks like a <a href="http://en.wikipedia.org/wiki/Random_graph">random
graph</a>, we expect it to be fast
mixing. In addition, it seems like some researchers believe typical social
networks are fast mixing. (There are many articles about it, just search for
“social networks fast mixing” and see what you get). However, If our networks
looks like a grid (This could happen if we just connect close devices using
wireless), it is probably not very fast mixing.</p>
<p><img alt="Random Graph" src="https://www.freedomlayer.org/articles/freedomlayer/sqrt_n_routing/images/random_graph.svg"></p>
<p>In the picture: An
<a href="http://en.wikipedia.org/wiki/Erd%C5%91s%E2%80%93R%C3%A9nyi_model">Erdos-Renyi</a>
random graph with <span class="math">\(n=2^6\)</span> and
<span class="math">\(p=\frac{2\cdot 6}{2^6}\)</span>. This is an example of a fast mixing graph. You can
browse for the code <a href="https://github.com/realcr/freedomlayer_code">here
[github]</a>.</p>
<p><img alt="A grid graph" src="https://www.freedomlayer.org/articles/freedomlayer/sqrt_n_routing/images/grid.svg"></p>
<p>A grid is an example of a non fast mixing graph.</p>
<p>Going backwards a bit, why do we care at all about the network being fast
mixing? What if it is “slow mixing”? Couldn’t we just use very long random
walks until we get a random node? The answer is that we could take longer
random walks, however the longer the path between a node <span class="math">\(x\)</span> and a virtual
node <span class="math">\(y\)</span>, the less reliable the connection. We haven’t considered adversarial
nodes in this case, but note that the longer the path, the more likely it is
that it contains some corrupt node that belongs to the adversary.</p>
<p>Longer paths also mean higher latency (It takes longer time to send and receive
messages), but we have more serious issues to care about at this point.</p>
<h4>Summary</h4>
<p>In a fast mixing network, given that every node maintains connections to <span class="math">\(r
= \sqrt{n\log{n}}\)</span> virtual nodes, every node <span class="math">\(a\)</span> in the network can form a short
path to any other node <span class="math">\(b\)</span>. Using that path <span class="math">\(a\)</span> could send messages that
will arrive at <span class="math">\(b\)</span>. Forming the path from <span class="math">\(a\)</span> to <span class="math">\(b\)</span> requires <span class="math">\(a\)</span> to
send about <span class="math">\(r = \sqrt{n\log{n}}\)</span> messages first, though.</p>
<p>This result might be enough for small networks, however for larger networks
This is not a very efficient result. It is better than flooding, though, which
is a step forward.</p>
<p>As a note about further reading, the article <a href="http://pdos.csail.mit.edu/papers/sybil-dht-socialnets08.pdf">A sybil proof one hop
<span class="caps">DHT</span></a>
by Chris Lesniewski-Laas also talks about dealing with an adversary using the
scarcity of network connections (The adversary might find it difficult to get
links to many correct nodes), which is a pretty interesting idea.</p>
<script type="text/javascript">if (!document.getElementById('mathjaxscript_pelican_#%@#$@#')) {
var align = "center",
indent = "0em",
linebreak = "false";
if (false) {
align = (screen.width < 768) ? "left" : align;
indent = (screen.width < 768) ? "0em" : indent;
linebreak = (screen.width < 768) ? 'true' : linebreak;
}
var mathjaxscript = document.createElement('script');
mathjaxscript.id = 'mathjaxscript_pelican_#%@#$@#';
mathjaxscript.type = 'text/javascript';
mathjaxscript.src = 'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.0/MathJax.js?config=TeX-AMS-MML_HTMLorMML';
mathjaxscript[(window.opera ? "innerHTML" : "text")] =
"MathJax.Hub.Config({" +
" config: ['MMLorHTML.js']," +
" TeX: { extensions: ['AMSmath.js','AMSsymbols.js','noErrors.js','noUndefined.js'], equationNumbers: { autoNumber: 'AMS' } }," +
" jax: ['input/TeX','input/MathML','output/HTML-CSS']," +
" extensions: ['tex2jax.js','mml2jax.js','MathMenu.js','MathZoom.js']," +
" displayAlign: '"+ align +"'," +
" displayIndent: '"+ indent +"'," +
" showMathMenu: true," +
" messageStyle: 'normal'," +
" tex2jax: { " +
" inlineMath: [ ['\\\\(','\\\\)'] ], " +
" displayMath: [ ['$$','$$'] ]," +
" processEscapes: true," +
" preview: 'TeX'," +
" }, " +
" 'HTML-CSS': { " +
" styles: { '.MathJax_Display, .MathJax .mo, .MathJax .mi, .MathJax .mn': {color: 'inherit ! important'} }," +
" linebreaks: { automatic: "+ linebreak +", width: '90% container' }," +
" }, " +
"}); " +
"if ('default' !== 'default') {" +
"MathJax.Hub.Register.StartupHook('HTML-CSS Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax['HTML-CSS'].FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"MathJax.Hub.Register.StartupHook('SVG Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax.SVG.FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"}";
(document.body || document.getElementsByTagName('head')[0]).appendChild(mathjaxscript);
}
</script>Basic DHT security concepts2014-11-27T10:03:00+00:002014-11-27T10:03:00+00:00realtag:www.freedomlayer.org,2014-11-27:/basic-dht-security-concepts.html<!--
ceil and floor latex macros:
-->
<div class="math">$$
\newcommand{\ceil}[1]{\left\lceil{#1}\right\rceil}
\newcommand{\floor}[1]{\left\lfloor{#1}\right\rfloor}
$$</div>
<h4>Abstract</h4>
<p>We are going to talk about some very basic techniques of securing a <span class="caps">DHT</span>
(Distributed Hash Table). We begin by discussing the kind of Adversary are we
going to defend against. We then …</p><!--
ceil and floor latex macros:
-->
<div class="math">$$
\newcommand{\ceil}[1]{\left\lceil{#1}\right\rceil}
\newcommand{\floor}[1]{\left\lfloor{#1}\right\rfloor}
$$</div>
<h4>Abstract</h4>
<p>We are going to talk about some very basic techniques of securing a <span class="caps">DHT</span>
(Distributed Hash Table). We begin by discussing the kind of Adversary are we
going to defend against. We then move on to some ideas regarding the choice of
Identities in a <span class="caps">DHT</span>. Finally we present a simple idea about bounding the amount
of adversarial nodes that manage to get into our network.</p>
<p>Most of the talk is going to be pretty informal, though it serves as a useful
down to earth introduction to the subject.</p>
<h5><span class="caps">DHT</span> Reminder</h5>
<p>Before we move on to talk about security , We review again the basic functions
of our <span class="caps">DHT</span>. In the lower level of the <span class="caps">DHT</span> structure, it means the ability to
search for node <span class="math">\(\floor{v}\)</span> given the value <span class="math">\(v\)</span>. In the higher level
(Thinking about a Distributed Hash table), it is the ability to store pairs of
(key,value), or retrieve a correct value given a key.</p>
<h4>The Adversary</h4>
<p><span class="famous_quote">
<span class="dquo">“</span>It is not cowardly, quite the contrary, to seek to meet the adversary and know
his intentions. However, it is cowardly, shameful and treasonable to lay down
arms.”</span> (King Hassan <span class="caps">II</span>.)</p>
<p>The first thing we want to do when talking about security is to define the
adversary. We will discuss it in a very informal manner, because this is just an
introduction to this subject. Our informal definitions here will be good enough
for this text.</p>
<p>Generally speaking, in our case the adversary is some entity that
wants to disrupt the <span class="caps">DHT</span> functions. By entity we don’t just mean one node, or
one computer, or even one person. The adversary could mean many things. More
than something realistic, it is an entity that helps us think about security.
It’s like a game. We give the adversary certain powers, and see if our model can
deal with it. In some sense, our Adversarial model is really our understanding
of reality. If all of this sounds too abstract to you, don’t worry, we are soon
going to see some examples.</p>
<p>Our general model of the adversary in the following text will be as follows.
The adversary manifests itself in the network in the form of “corrupt” nodes.
Those are nodes that are under the full control of the adversary. We then make
a distinction: For every node in the network, it is either a node controlled by
the adversary - A “corrupt node”, or a “correct” node. By “correct” we mean
honest, or “not corrupt”. It’s a node that follows the rules as we defined
them. (From now on we will not add the quotes when referring correct or corrupt nodes).</p>
<p>Another important thing to note about the two types of nodes is that <strong>it is not
possible to tell if a node is a correct node or a corrupt node.</strong> Every node
knows if he himself is correct or corrupt, but he can never know for sure if
another node is correct or corrupt. You could try to ask a node if he is
correct, but then, if it’s a corrupt node, he might lie. You could also try to
watch a node for a really long time, and then try to conclude that it is a
correct node. However that node might be just pretending, waiting for you to
look away.</p>
<p>Of those reasons, (And several others that will become clearer in the future),
we will never try to directly conclude if a node is correct or corrupt. We just
know that some nodes are corrupt and some are correct. This is the way of life.</p>
<p>One more thing to note is that for the sake of simplicity, we are going to
forget for a while the Churn and Reliability issues. We will not think about
nodes that fail in the article. We assume that all the correct nodes are
perfectly reliable, and our main problem to deal with is the Adversary.</p>
<p>Let’s talk about what the adversary can do, or at least what we “allow” him to
do in this model. </p>
<h5>The Sybil Attack</h5>
<p>We begin with an adversary that can get into the network as many corrupt
nodes as he likes. To make things easy to think about, forget about reliability
issues (Nodes that fail). We will assume that the correct nodes never fail.</p>
<p>Try to think for a moment what could be done with this ability of inserting as
many corrupt nodes as you want into the <span class="caps">DHT</span>.</p>
<p>One very philosophical way of thinking about this situation is as follows: Does
the network really belong to us if there are more corrupt nodes than correct
nodes? Could we really enforce our structure on the network if the nodes we
control are in a minority? And if we could, why couldn’t the adversary do
exactly the same? Thinking about it, what makes us and the adversary so different?</p>
<p>Going back to our physical world, let’s try to see some concrete examples of
things the adversary can do:</p>
<ul>
<li>
<p>Disconnecting the <span class="caps">DHT</span>: The adversary could insert many nodes into the network,
and let them run by the <span class="caps">DHT</span> rules. Correct nodes will not be able to
distinguish between the corrupt nodes and the correct nodes, and they will
form connection to the corrupt nodes. Then in one sudden moment, the
adversary will make all the corrupt nodes halt. That means: All the corrupt
nodes will stop responding. As the adversary has many nodes (Much more than
corrupt nodes), With a pretty high probability, the <span class="caps">DHT</span> will become disconnected. </p>
</li>
<li>
<p>Blocking or Changing specific keys: Assume that the adversary doesn’t want
some key to be accessible (It will not be possible to get the value <span class="math">\(v\)</span> for
a key <span class="math">\(k\)</span>). To do this, The adversary could insert many nodes into the
network with random IDs (Remember that every node has some <span class="caps">ID</span>). If the
adversary has a very big amount of nodes, with some high probability the key
<span class="math">\(k\)</span> will be under the responsibility of a corrupt node. Then whenever that
key is asked for, the adversary could return some other value <span class="math">\(v'\)</span> instead
of <span class="math">\(v\)</span>, or just not return anything.</p>
</li>
</ul>
<p>As we mentioned above though (In the philosophical paragraph), if the adversary
wanted he could make anything out of this network, because really, he owns this
network. He was a majority of “corrupt” nodes.</p>
<p>This kind of attack where an adversary inserts a large amount of “corrupt” nodes
into a network is also known as <a href="http://en.wikipedia.org/wiki/Sybil_attack">Sybil Attack</a>
This kind of attack is not specific to DHTs. It is related to any distributed
network where all the nodes have symmetric roles. It is probably the first
attack to consider when you hear about a fully distributed technology.</p>
<p>The Sybil attack is one of the fundamental issues we have to deal with when
designing anything that is really distributed. We will not solve this at this
point, but we are going to talk about it more in the future.</p>
<h5>Node bounded Adversary</h5>
<p>We have seen that we can’t defend ourselves at this point from an adversary that
inserts unlimited amount of nodes into the network. So let’s put a limit to the
amount of nodes the adversary can insert.</p>
<p>Assuming that there are <span class="math">\(n\)</span> nodes in the network, a reasonable bound might be
<span class="math">\(\alpha \cdot n\)</span>, where <span class="math">\(0 < \alpha < 0.5\)</span>. In other words, the maximal amount
of corrupt nodes is a constant fraction of the total size of the network. Why do
we pick <span class="math">\(\alpha\)</span> smaller than half? That is because (As we noted above) if the
adversary has more than half the nodes im the network, he pretty much owns the
network. Just as a sanity check, note that the amount of correct nodes in the
network is <span class="math">\((1 - \alpha)n\)</span>.</p>
<p>The adversary has a collection of <span class="math">\(\alpha n\)</span> corrupt nodes. He can insert
each of those nodes into the <span class="caps">DHT</span>, and also remove each of them from the <span class="caps">DHT</span>.
The adversary has much freedom to move the corrupt nodes around, however he
owns only <span class="math">\(\alpha n\)</span> nodes.</p>
<p>You might be thinking how come we get to choose the abilities of our adversary.
For example - How can we allow ourselves to bound the amount of corrupt nodes?
After all, the adversary doesn’t work by our rules. There are two answers. The
first is that we can’t, but it helps us to think. The second answer is that we
might actually create some mechanism where the adversary will not be able to
issue too many corrupt nodes. In that case, it could be easier to first think
about the <span class="caps">DHT</span> construct, and only later about the other mechanism to bound the
amount of corrupt nodes.</p>
<p>Just to make sure I don’t leave you with abstract promises, let’s consider a
very naive mechanism to bound the amount of possible corrupt nodes. For every
node who wants to join the network, we will ask the computer owner to show up
in person and hand over his id card. (In the real world). We will let one node
in for every unique id card that we get. In this example, hopefully the
adversary will not be able to get too many id cards, and therefore he will not
be able to have too many nodes inside the network.</p>
<p>We are not going to use this mechanism (It is not very distributed or
efficient), but it serves as a good example of how to bound the amount of
corrupt nodes in a network. Another formulation of that example would be:
We used the scarcity of id cards in the real world to make Sybil attacks hard
to perform.</p>
<h4>Security of Identities’ choice</h4>
<p>In the following sections we will try to deal with various security issues that
might arise in a <span class="caps">DHT</span>. We have met the <a href="https://www.freedomlayer.org/intro-to-distributed-hash-tables-dhts.html">Chord
<span class="caps">DHT</span></a> but this might
apply to a wider range of <span class="caps">DHT</span> structures.</p>
<p>Regarding the Adversarial model: We are going to assume a Node Bounded Slow
Changing Adversary. To avoid writing those 5 words every time I want to refer
the Adversary, we will call it just Adversary on the rest of this text.</p>
<p>Most of our security considerations presented here will relate to the way we
choose the Identities of nodes in the network. I remind you that in Chord, every
node has an Identity number which is from a set <span class="math">\(B_s := \{0,1,2,\dots,2^{s}-1
\}\)</span> (A number of size <span class="math">\(s\)</span> bits). Identities in a <span class="caps">DHT</span> are very important, as
they determine the range of keys a node has responsibility over.</p>
<p>Given that we know the set of possible identities, we are left with the task of
choosing the Identity for new nodes who join the network. We will observe a few
methods here.</p>
<h5>Random Identities</h5>
<p>A simple solution for the Identity choice would be to let every new node in the
network pick a random Identity number. If we are talking about Chord, that random
Identity number will be a random bit string of size <span class="math">\(s\)</span> bits. If all the nodes
are correct and <span class="math">\(s\)</span> is <a href="http://en.wikipedia.org/wiki/Birthday_problem">large
enough</a>, we can be pretty sure
that no two nodes are going to have the exact same Identity number.</p>
<p>However, if we take into consideration an Adversary as discussed above, we are
expected to have problems. One thing the adversary can do, for example, is to
block or modify the value for any wanted key. Assume that the Adversary wants
to block the value of key <span class="math">\(k\)</span>. It is enough that the Adversary will insert
just one corrupt node <span class="math">\(z\)</span> into the network, choosing for him the Identity
<span class="math">\(k\)</span>.</p>
<p>Then for sure the node <span class="math">\(z\)</span> will have the responsibility of keeping the key
<span class="math">\(k\)</span> and its corresponding value. Whenever some other node <span class="math">\(x\)</span> will search
for the key <span class="math">\(k\)</span> and ask for its value, the node <span class="math">\(z\)</span> could not respond (Thus
blocking access to the key <span class="math">\(k\)</span>), or give a different value <span class="math">\(v'\)</span>.</p>
<p>The idea of Random Identities gets us started, but obviously it can not deal
with our Adversary.</p>
<h5>Public keys as Identities</h5>
<p>Following the previous attempt, we want to make sure the Adversary could not
choose any arbitrary <span class="caps">DHT</span> Identity that he wants. We want to make it hard for the
adversary to choose arbitrary <span class="caps">DHT</span> identities. </p>
<p>One possibility for Identity choice is Public keys, using any form of
<a href="http://en.wikipedia.org/wiki/Public-key_cryptography">asymmetric
cryptography</a>. (Also
known as Public key Cryptography).</p>
<p>In this method, every node that joins the network first generates a key pair of
public key and private key. Then the node uses his own public key as his <span class="caps">DHT</span>
Identity. Whenever a node <span class="math">\(y\)</span> comes in contact with a node <span class="math">\(x\)</span>, <span class="math">\(y\)</span> will
ask <span class="math">\(x\)</span> to prove his claimed identity. <span class="math">\(x\)</span> will then prove his identity
using the Public Key Cryptography system. This way the Adversary will have to
use a valid Identity for his corrupt nodes, if he wants to be able to
communicate with correct nodes.</p>
<p>We somehow hope that it is going to be hard for the Adversary to generate an
Identity that is both valid and close to a specified wanted number.</p>
<h6>Some words about Public Key Cryptography</h6>
<p>All those Cryptography stuff might sound very strange to you if you have never
heard about Public Key Cryptography before. I will try to explain shortly what
Public Key Cryptography feels like, however if you want to really understand
things, please take the time to read more about it from some more serious
sources. It will really make a difference in your understanding of things.</p>
<p>Public Key cryptography is a system with a few participants. Every participant
initially generates somehow two keys: A public key and a private key. Those keys
are usually generated randomly (By the participants), and they are related
somehow to each other. Then every participant shares his public key with the
rest of the participants, but keeps his private key a secret.</p>
<p>What could be done with those public and private keys? Using the public key one
can encrypt data. The corresponding private key could be used to decrypt that data.</p>
<p>If for example Bob generates a key pair of Public key and private key, he will
share the public key with all the participants in the system. Then every
participant can encrypt messages with Bob’s public key, but only Bob could
decrypt those messages with his secret private key.</p>
<p>Another feature that some Public Key cryptography systems has is the <strong>ability to
sign</strong>. Bob could sign some message, and then every participant could verify the
signature Bob’s signature, and be sure that Bob wrote the message. It’s pretty
much like signatures in the real world (Though a bit more secure :) )</p>
<p>In our case we are really interested in the signing ability that we get from
Public Key Cryptography. (Hint: We are going to use it later to prove that we own an
Identity in the <span class="caps">DHT</span>).</p>
<p>I don’t want to get too abstract on you, so let me follow with a real example of
a Public Key Cryptography system. A famous example is
<a href="http://en.wikipedia.org/wiki/RSA_%28cryptosystem%29"><span class="caps">RSA</span></a>. If Alice wants to
generate a key pair, she has to generate two big random prime numbers. Let’s
call those <span class="math">\(p,q\)</span>. Then Alice multiplies them, to get the value <span class="math">\(N=pq\)</span>. <span class="math">\(N\)</span>
is then called the public key, and <span class="math">\(p,q\)</span> are the private keys. Alice can share
the public key (<span class="math">\(N\)</span>) with anyone, however she will never tell <span class="math">\(p,q\)</span>. This is
the private (Or secret) key. Note that concluding the numbers <span class="math">\(p,q\)</span> knowing
only <span class="math">\(N\)</span> is considered to be a difficult computational problem. (See <a href="http://en.wikipedia.org/wiki/Integer_factorization">Integer
Factorization</a>)</p>
<p>In <span class="caps">RSA</span> the public key could be used to Encrypt messages, and the private key
could be used to decrypt messages. In a similar way, we could use the private
key to sign messages, and the public key to verify those messages. For the
interested, the operation of signing a message in <span class="caps">RSA</span> is the same as decrypting
a message using the private key. The operation of verifying an <span class="caps">RSA</span> signature is
the same as encrypting a message using the public key.</p>
<h6>Using Public keys as Identities</h6>
<p>As we mentioned in the beginning of this section, we planned that every node
<span class="math">\(x\)</span> will generate a key pair, and use the Public key as a <span class="caps">DHT</span> Identity.
When a node <span class="math">\(x\)</span> joins the network and presents his Public key (Which is his
claimed <span class="caps">DHT</span> Identity), we will send him to prove his identity.</p>
<p>How could we ask <span class="math">\(x\)</span> to prove it his identity? A naive method would be to send
him some random value <span class="math">\(t\)</span>, and ask him ask him to sign it. If <span class="math">\(x\)</span> manages to
create a valid signature over <span class="math">\(t\)</span>, we will conclude that his identity is confirmed.</p>
<p>Again let’s assume that the Adversary wants to block some key <span class="math">\(k\)</span> in the <span class="caps">DHT</span>,
and see what happens in this case. The Adversary has to position some corrupt
node with an Identity very close to <span class="math">\(k\)</span> (But not bigger than <span class="math">\(k\)</span>) inside the
network. If we are dealing with <span class="caps">RSA</span>, then all the Adversary needs to do is find
some two prime number <span class="math">\(p,q\)</span> such that <span class="math">\(pq\)</span> is very close to <span class="math">\(k\)</span>. Then the
adversary will use <span class="math">\(N=pq\)</span> as the corrupt node’s Identity (Public Key), and
<span class="math">\(p,q\)</span> as the private key. </p>
<p>There are many prime numbers, so this should not really be a problem. It’s a bit
harder than the previous case, where the Adversary could just pick and Identity
that he wants for his corrupt nodes, however it is not much harder.</p>
<p>(One could also say that it might be not needed to supply a value <span class="math">\(N\)</span> that is
really made of a multiplication of two primes, but we don’t need to get into
this. Even if we follow the rules and create <span class="math">\(N\)</span> as a multiplication of two
primes, it seems to be pretty easy to get close as we want to <span class="math">\(k\)</span>).</p>
<h6>Using Hashed Public Keys as identities</h6>
<p>We could make it a bit harder, though. For a node <span class="math">\(x\)</span> with public key <span class="math">\(N\)</span>
and private key <span class="math">\(p,q\)</span>, we could declare <span class="math">\(x\)</span><span class="quo">‘</span>s identity to be <span class="math">\(f(N)\)</span>, where
<span class="math">\(f\)</span> is some <a href="http://en.wikipedia.org/wiki/Cryptographic_hash_function">cryptographic hash
function</a>.</p>
<p>Let’s again go over all the process of <span class="math">\(x\)</span> joining the network to make sure
that this makes sense. <span class="math">\(x\)</span> first generates a pair <span class="math">\(p,q\)</span> of random big
primes, and then derives <span class="math">\(N=pq\)</span> to be his public key. Next, <span class="math">\(x\)</span> calculates
<span class="math">\(Id=f(N)\)</span>, and this is <span class="math">\(x\)</span><span class="quo">‘</span>s <span class="caps">DHT</span> Identity.</p>
<p>When <span class="math">\(x\)</span> joins the network, he has to confirm his identity. He will claim his
identity to be <span class="math">\(Id=F(N)\)</span>, and he will also supply the value <span class="math">\(N\)</span>. (The verifier
will check that <span class="math">\(Id=F(N)\)</span>). Next, the verifier will send <span class="math">\(x\)</span> some random
value <span class="math">\(t\)</span>, and ask <span class="math">\(x\)</span> to sign it. <span class="math">\(x\)</span> will create a signature over <span class="math">\(t\)</span>
and send it back to the verifier. The verifier will then make sure that the
signature is correct. If it is, <span class="math">\(x\)</span><span class="quo">‘</span>s identity is confirmed (With respect to
this verifier).</p>
<p>Now let’s see what happens if the adversary wants to take control over a key
<span class="math">\(k\)</span>. The adversary has to create a corrupt node with Identity close to <span class="math">\(k\)</span>
but no bigger than <span class="math">\(k\)</span>. Eventually, the Identity is <span class="math">\(f(N)\)</span>. <span class="math">\(N\)</span> has to be
created in a specific way (Probably a multiplication of two primes), however we
could be generous and assume that the Adversary could get to any <span class="math">\(N\)</span> that it
wants. All that is left is finding <span class="math">\(N\)</span> such that <span class="math">\(f(N)\)</span> is close to <span class="math">\(k\)</span>.
If the adversary could do that, he will be able to take control over the key
<span class="math">\(k\)</span>.</p>
<p>We assume that the adversary is computationally bounded. (He can only compute
things in polynomial time). So far we didn’t discuss this property of our
adversary, however this might be a good place to add this assumption. Basically
it means that we assume that adversary doesn’t have too much computation power.</p>
<p>Without thinking about any specific properties of the cryptographic hash
function <span class="math">\(f\)</span>, a good idea to find a correct <span class="math">\(N\)</span> will to be generate random
numbers and check if <span class="math">\(d(f(a),k)\)</span> is small enough. (Recall that <span class="math">\(d(a,b)\)</span> is
the distance between two identity numbers on the ring). As we expect <span class="math">\(f\)</span> to be
somewhat random, we expect that for some number <span class="math">\(a\)</span>, we will get that
<span class="math">\(d(f(a),k)\)</span> is distributed uniformly on <span class="math">\([0,2^s)\)</span>. In simpler words, it
means that <span class="math">\(f(a)\)</span> has the same likelihood of being anywhere on the ring.
By creating lots of numbers <span class="math">\(N=pq\)</span> for many different primes <span class="math">\(p,q\)</span>, we can
generate many values <span class="math">\(f(N)\)</span> that distribute uniformly on the ring.</p>
<p>We have some interval that we want to “land on”, when getting a random value
<span class="math">\(f(a)\)</span>. This interval is exactly between <span class="math">\(\floor{k}\)</span> and <span class="math">\(k\)</span> - Right
between the node that is currently responsible over the key <span class="math">\(k\)</span>, and the key
<span class="math">\(k\)</span> itself.</p>
<p><img alt="Wanted interval floor k" src="https://www.freedomlayer.org/articles/freedomlayer/dht_basic_security/images/wanted_interval_floor_k.svg"></p>
<p>In the picture: The interval that the Adversary wants to land on to take over
the key <span class="math">\(k\)</span> is marked with the green color.</p>
<p>Now we want to know how many different values <span class="math">\(N=pq\)</span> we have to generate
before we get that <span class="math">\(f(N)\)</span> is inside the wanted interval on the ring.
A simple calculation shows that the expected number of tries is going to be The
size of the ring divided by the size of the interval. (Think if it makes sense
for an interval of size <span class="math">\(\frac{1}{4}\)</span> of the ring, for example).</p>
<p>We still don’t know the size of the interval, though. We want to evaluate
somehow the value <span class="math">\(d(\floor{k},k)\)</span> for some key <span class="math">\(k\)</span>. We could get a rough
estimation of this value by thinking about the easy case, where all the nodes
are distributed evenly on the ring. In that case, if there are <span class="math">\(n\)</span> nodes, we
expect that the distance between two consecutive nodes will be exactly <span class="math">\(\frac{2^s}{n}\)</span>.</p>
<p>Finally to get the amount of tries, we divide the size of the ring by the our
rough estimation of the size of the interval. We get
<span class="math">\(\frac{2^s}{\frac{2^s}{n}}=n\)</span>. Therefore we expect about <span class="math">\(n\)</span> tries before we
get a number <span class="math">\(N\)</span> such that <span class="math">\(d(f(N),k) < d(\floor{k},k)\)</span>. Recall that <span class="math">\(n\)</span>
is the amount of nodes in the network. This number is probably not larget than
<span class="math">\(2^40\)</span> in most cases.</p>
<p>As a short summary, the adversary could gain control over any specific key <span class="math">\(k\)</span>
using this algorithm:</p>
<ol>
<li>Generate two random primes <span class="math">\(p,q\)</span>. Calculate <span class="math">\(N=pq\)</span>.</li>
<li>Calculate <span class="math">\(f(N)\)</span>.</li>
<li>If <span class="math">\(d(f(N),k) < d(\floor{k},k)\)</span> then return (p,q). Else go back to 1.</li>
</ol>
<p>By our estimations we expect this algorithm to loop about <span class="math">\(n\)</span> times, before a
suitable pair of <span class="math">\(p,q\)</span> is found.</p>
<p>Also note that the Adversary doesn’t have to use more than one corrupt node in
the network to gain control over the key <span class="math">\(k\)</span>. Most of the calculation is done
offline, ahead of time.</p>
<p>We can conclude that this method can not help us deal with this adversary, but
we did make some progress. Recall that in the previous sections the adversary
could gain control over some key <span class="math">\(k\)</span> without much effort. Here the adversary
has to do some effort to take control over some key.</p>
<h5>Hashing IPs</h5>
<p>We have already considered the idea of hash functions to make it harder for the
adversary to control the Identity of corrupt nodes. Another idea would be to
use the hash of a the <span class="caps">IP</span> address of a node as its <span class="caps">DHT</span> Identity. This solution
relies on the hierarchical structure of the Internet and the fact that it is
hard to obtain many different <span class="caps">IP</span> addresses (At least for simple Adversaries).</p>
<p>Let’s be more detailed. For a new node <span class="math">\(x\)</span> with <span class="caps">IP</span> Address <span class="math">\(x_a\)</span> that wants
to join the <span class="caps">DHT</span>, we define <span class="math">\(x\)</span><span class="quo">‘</span>s Identity to be <span class="math">\(f(x_a)\)</span>, where <span class="math">\(f\)</span> is a
known cryptographic hash function. Every other node <span class="math">\(y\)</span> on the network that
contacts <span class="math">\(x\)</span> knows <span class="math">\(x\)</span><span class="quo">‘</span>s <span class="caps">IP</span> Address (Or else, how could <span class="math">\(y\)</span> contact <span class="math">\(x\)</span>
from the first place). Therefore <span class="math">\(y\)</span> knows automatically the value <span class="math">\(f(x_a)\)</span>
which is <span class="math">\(x\)</span><span class="quo">‘</span>s <span class="caps">DHT</span> Identity.</p>
<p>This time, if an Adversary wants to take control over a key <span class="math">\(k\)</span> in the <span class="caps">DHT</span>,
he will have to create a corrupt node with <span class="caps">IP</span> address <span class="math">\(a\)</span> such that <span class="math">\(f(a)\)</span>
is close to the value <span class="math">\(k\)</span> (But not bigger than <span class="math">\(k\)</span>).</p>
<p>In IPv4 There are no more than <span class="math">\(2^{32}\)</span> possible addresses. By the calculations
we have seen in the idea of Public keys as Identities, we conclude that about
<span class="math">\(\frac{2^{32}}{n}\)</span> Addresses will be suitable for the adversary if he wants
to take control over some specific key <span class="math">\(k\)</span>. </p>
<p>Obtaining specific IPs could be pretty hard these days (Though not impossible).
Therefore this method does make it hard to take control over a specific key in
the <span class="caps">DHT</span>. Thinking about it, the change to IPv6 (Where there are <span class="math">\(2^{128}\)</span>
possible addresses) might make this method less effective. </p>
<p>Generally speaking, in a network where every node has the freedom to choose his
Address in some fast manner (And there are many possibilities for addresses),
this method might not work well. The Adversary could generate many Addresses
(We have seen that about <span class="math">\(n\)</span> addresses should be enough), until one address
has a hash value in the correct range. Assuming that <span class="math">\(n\)</span> is not too big, this
shouldn’t be a hard calculation.</p>
<h4>Bounding the Adversary</h4>
<p>We talked earlier about our assumption of a Node Bounded Adversary - An
Adversary that can insert only so many corrupt nodes into the network. At this
point I want to show an example of achieving this property - Making sure that
the Adversary doesn’t have too many corrupt nodes in the network.</p>
<p>Recall that the very naive solution we initially proposed for this was using
some kind of <strong>external scarcity</strong>, like real world <span class="caps">ID</span> cards. Every node that
wants to join the network should first be represented by a person in the real
world, handing over his (real world) <span class="caps">ID</span> card to some central authority.</p>
<p>There is probably something deeper about the scarcity idea. I say probably,
because we don’t have any rigorous theory yet about it, but it seems like most
distributed systems today that are able to deal with Sybil attacks are based
on some kind of scarcity. That scarcity makes it hard for any participant to
insert many nodes into the network, as every node is linked to some amount of
that scarce resource.</p>
<p>In this example we are going to use Computing power as a scarce resource. This
kind of scarcity is widely used in
<a href="http://en.wikipedia.org/wiki/Bitcoin">Bitcoin</a> based crypto currencies.</p>
<p>Recall that a <span class="caps">DHT</span> structure is based on links between nodes. Every node is
linked to a few other nodes, allowing strong network connectivity and fast
network traversal (Fast searching) at the same time. Also recall that to
maintain a link between two nodes, we proposed the idea of
<a href="http://en.wikipedia.org/wiki/Heartbeat_%28computing%29">heartbeats</a>. Messages
are sent in constant time interval between two nodes, to make sure that the
remote node is alive.</p>
<p>Every node in the network has to maintain a few links with other nodes.
Maintaining a link requires sending periodic heartbeat messages, which is not
very computationally expensive. It would be interesting if we could make the task
of maintaining a link computationally expensive. In that case, the more nodes the
adversary inserts into the network, the more links he will have to maintain. A
computationally bounded Adversary will not be able to insert too many nodes, as
he won’t be able to maintain all the links.</p>
<p>One way to make the task of maintaining links more difficult is using a <a href="http://en.wikipedia.org/wiki/Proof-of-work_system">proof of
work</a> puzzle. These are
special riddles that are very easy to generate, very hard to solve but it is
very easy to verify their solution.</p>
<p>We will extend the idea of heartbeat, by adding a difficult riddle to
the heartbeat. If <span class="math">\(x\)</span> and <span class="math">\(y\)</span> are two linked nodes in the <span class="caps">DHT</span>, <span class="math">\(x\)</span> will
send a riddle to <span class="math">\(y\)</span> every 10 seconds, for example. <span class="math">\(x\)</span> will then expect
<span class="math">\(y\)</span> to answer the riddle in a short time. If <span class="math">\(y\)</span> doesn’t manage to solve
the riddle, <span class="math">\(x\)</span> will disconnect the link to <span class="math">\(y\)</span>. The same happens the other
way: <span class="math">\(y\)</span> will send <span class="math">\(x\)</span> periodic riddles, and ask <span class="math">\(x\)</span> to solve those
riddles. If <span class="math">\(x\)</span> doesn’t manage to solve the riddles in time, <span class="math">\(y\)</span> will
disconnect <span class="math">\(x\)</span>.</p>
<p><img alt="Riddle Exchange" src="https://www.freedomlayer.org/articles/freedomlayer/dht_basic_security/images/riddle_exchange.svg"></p>
<p>In the picture: Illustration of the riddles exchange between two linked nodes.
Each node sends periodically a riddle to the other, and waits for a correct solution.</p>
<p>This is somehow like the alarm clock applications that make you solve a hard
problem, to prove that you are awake and present. Here we use a riddle to make a
remote computer prove that he is present, and not scattered as many different
nodes in the network.</p>
<p>You might be wondering about the kind of “riddles” we are going to use. As we
are dealing with computers here, it must be a hard riddle. We also have to make
sure that the riddle is hard to solve, but easy to verify. We could use
cryptographic hash functions to create some pretty hard riddles. Assuming that
we have a cryptographic hash function <span class="math">\(f\)</span>, A famous riddle is
to find a value <span class="math">\(a\)</span> such that <span class="math">\(f(a|b)\)</span> begins with <span class="math">\(k\)</span> binary zeroes. (By
| we mean concatenation of strings). We could call this riddle riddle<span class="math">\(_b\)</span>. </p>
<p>As <span class="math">\(f\)</span> is some general cryptographic hash function, we expect that the best
strategy to solve this would be to randomize many values <span class="math">\(a\)</span> until we get
that <span class="math">\(f(a|b)\)</span> begins with <span class="math">\(k\)</span> binary zeroes. We should generate about
<span class="math">\(2^k\)</span> random values <span class="math">\(a\)</span> until we get a good result for <span class="math">\(f(a|b)\)</span>. For a
large enough <span class="math">\(k\)</span>, this would be a hard question for a computer.</p>
<p>Note that it is pretty easy to verify that a solution <span class="math">\(s\)</span> is a correct
solution to a given riddle<span class="math">\(_b\)</span>. We just make sure that <span class="math">\(f(s|b)\)</span> begins with
<span class="math">\(k\)</span> binary zeroes.</p>
<p>Now let’s go back to the Adversary point of view. We want to check if we managed
to bound the amount of corrupt nodes an adversary can insert into the network.
Assume that some adversary wants to insert many corrupt nodes into the network.
Each of those corrupt nodes has to maintain a few links, to be considered as
part of the network. If the adversary wants to insert <span class="math">\(m\)</span> corrupt nodes, and
maintaining links of some node inside the network costs <span class="math">\(t\)</span> basic calculation
units per second, then the adversary will need about <span class="math">\(mt\)</span> basic calculation
units per second to maintain <span class="math">\(m\)</span> corrupt nodes inside the network.</p>
<p>Assuming that the Adversary is capable of <span class="math">\(Q\)</span> calculation units per second, we
get that the maximum amount of corrupt nodes that could stay inside the network
is about <span class="math">\(\frac{Q}{t}\)</span>. Note that if the cost of maintaining a node inside the
network, <span class="math">\(t\)</span>, is increased, the adversary can insert less corrupt nodes into
the network, however at the same time it becomes less comfortable for a correct
node to stay linked in the network. There is some trade-off here.</p>
<p>One question that I leave you to think about - How could we avoid the following
situation (Riddle redirection): Assume that a corrupt node <span class="math">\(z\)</span> is connected
to two correct nodes <span class="math">\(x\)</span> and <span class="math">\(y\)</span>. <span class="math">\(x\)</span> sends a riddle <span class="math">\(R\)</span> to <span class="math">\(z\)</span>, but
<span class="math">\(z\)</span> (As a corrupt node) doesn’t want to invest the time in solving the riddle
<span class="math">\(R\)</span>. Therefore <span class="math">\(z\)</span> forwards the riddle <span class="math">\(R\)</span> to <span class="math">\(y\)</span>, asking him to solve
that riddle. <span class="math">\(y\)</span> is an innocent correct node, and he solves the riddle <span class="math">\(R\)</span>,
sending back the solution <span class="math">\(S\)</span> to <span class="math">\(z\)</span>. <span class="math">\(z\)</span> then returns the solution <span class="math">\(S\)</span>
back to <span class="math">\(x\)</span>, and <span class="math">\(x\)</span> accepts the solution. This way <span class="math">\(z\)</span> doesn’t have to
solve any riddles that <span class="math">\(x\)</span> sends. </p>
<p><img alt="Riddle Redirection" src="https://www.freedomlayer.org/articles/freedomlayer/dht_basic_security/images/riddle_redirection.svg"></p>
<p>In the picture: The riddle redirection attack illustrated. How can you solve it?</p>
<p>As a final note about this example - We managed to find a way to bound the
amount of corrupt nodes in the network, assuming that our adversary is
computationally bounded. We used scarcity of computing power, and linked it to
maintenance of a node in the network.</p>
<h4>Summary</h4>
<p>We described shortly our network enemy - The Adversary, and talked about the
Sybil attack: An attack that happens when too many corrupt nodes are inside the
network. Next we discussed different methods of choosing the Identities for a
<span class="caps">DHT</span> from the security perspective. We found that using Hashed <span class="caps">IP</span> addresses could
be practical as a <span class="caps">DHT</span> Identity, but they rely on the structure of the Internet,
somehow. Finally we showed a simple method of bounding the amount of corrupt
nodes in our network, relying on the scarcity of computing power. Generally
speaking, we have seen that <span class="caps">DHT</span> are pretty hard to secure in the wild.</p>
<script type="text/javascript">if (!document.getElementById('mathjaxscript_pelican_#%@#$@#')) {
var align = "center",
indent = "0em",
linebreak = "false";
if (false) {
align = (screen.width < 768) ? "left" : align;
indent = (screen.width < 768) ? "0em" : indent;
linebreak = (screen.width < 768) ? 'true' : linebreak;
}
var mathjaxscript = document.createElement('script');
mathjaxscript.id = 'mathjaxscript_pelican_#%@#$@#';
mathjaxscript.type = 'text/javascript';
mathjaxscript.src = 'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.0/MathJax.js?config=TeX-AMS-MML_HTMLorMML';
mathjaxscript[(window.opera ? "innerHTML" : "text")] =
"MathJax.Hub.Config({" +
" config: ['MMLorHTML.js']," +
" TeX: { extensions: ['AMSmath.js','AMSsymbols.js','noErrors.js','noUndefined.js'], equationNumbers: { autoNumber: 'AMS' } }," +
" jax: ['input/TeX','input/MathML','output/HTML-CSS']," +
" extensions: ['tex2jax.js','mml2jax.js','MathMenu.js','MathZoom.js']," +
" displayAlign: '"+ align +"'," +
" displayIndent: '"+ indent +"'," +
" showMathMenu: true," +
" messageStyle: 'normal'," +
" tex2jax: { " +
" inlineMath: [ ['\\\\(','\\\\)'] ], " +
" displayMath: [ ['$$','$$'] ]," +
" processEscapes: true," +
" preview: 'TeX'," +
" }, " +
" 'HTML-CSS': { " +
" styles: { '.MathJax_Display, .MathJax .mo, .MathJax .mi, .MathJax .mn': {color: 'inherit ! important'} }," +
" linebreaks: { automatic: "+ linebreak +", width: '90% container' }," +
" }, " +
"}); " +
"if ('default' !== 'default') {" +
"MathJax.Hub.Register.StartupHook('HTML-CSS Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax['HTML-CSS'].FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"MathJax.Hub.Register.StartupHook('SVG Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax.SVG.FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"}";
(document.body || document.getElementsByTagName('head')[0]).appendChild(mathjaxscript);
}
</script>Stabilizing Chord2014-11-13T19:41:00+00:002014-11-13T19:41:00+00:00realtag:www.freedomlayer.org,2014-11-13:/stabilizing-chord.html<!--
ceil and floor latex macros:
-->
<div class="math">$$
\newcommand{\ceil}[1]{\left\lceil{#1}\right\rceil}
\newcommand{\floor}[1]{\left\lfloor{#1}\right\rfloor}
$$</div>
<h3>Abstract</h3>
<p>In the <a href="https://www.freedomlayer.org/intro-to-distributed-hash-tables-dhts.html">Intro to
DHTs</a>
article we presented a very basic idea of the chord <span class="caps">DHT</span> (See also <a href="http://pdos.csail.mit.edu/papers/chord:sigcomm01/chord_sigcomm.pdf">original
article</a>
and generally how to search it. We treated it pretty much as a …</p><!--
ceil and floor latex macros:
-->
<div class="math">$$
\newcommand{\ceil}[1]{\left\lceil{#1}\right\rceil}
\newcommand{\floor}[1]{\left\lfloor{#1}\right\rfloor}
$$</div>
<h3>Abstract</h3>
<p>In the <a href="https://www.freedomlayer.org/intro-to-distributed-hash-tables-dhts.html">Intro to
DHTs</a>
article we presented a very basic idea of the chord <span class="caps">DHT</span> (See also <a href="http://pdos.csail.mit.edu/papers/chord:sigcomm01/chord_sigcomm.pdf">original
article</a>
and generally how to search it. We treated it pretty much as a static network
of computers. We did not invest much time on thinking about what happens when
nodes join or leave the network (Or what happens when nodes fail).</p>
<p>In this article we are going to discuss (Very briefly) some ideas regarding how
to preserve the connectedness and structure of the network, despite all the
frequent changes that happen, such as: Nodes joins, node leaves and failures.</p>
<p>Particularly, we are going to present an operation called Stabilize (Which all
nodes perform periodically). This operation allows the network to preserve its
structure. (We note that Stabilize as presented here is not the same
as Stabilize presented in the original Chord article.)</p>
<h3>The origin of Churn</h3>
<p>Before we get into the details, I want us to have some general picture of what
we are going to deal with, when I’m saying “Stabilize”. At this point we are
not going to deal with some malicious powerful adversary that tries to destroy
our <span class="caps">DHT</span> (We will meet this adversary later at some point, but not today). You
may assume that all the nodes are inside some laboratory, and we own all of them.</p>
<p>There are some events that could happen. First of all, there are the usual <span class="caps">DHT</span>
events of nodes joining and leaving. How should we handle those joins and
leaves, so that the network keeps its “original structure”? So that it stays
searchable, and even more than that: Stays connected?</p>
<p>Besides joins and leaves, we should also deal with nodes that fail. Why would a
node fail? There could be a few reasons. Maybe the computer broke, or someone
tripped over the <span class="caps">LAN</span> cable connecting the node to the rest of the computers. It
might be that power went offline, or someone pushed the restart button. We
don’t really know. We just know that the failure was immediate, and the node
had no chance at all to report its failure to the network.</p>
<p>We are going to find at some point that this failed node is dead, but not
immediately. One inherent characteristic of the Internet, and maybe networks
generally, is that <strong>you can not know immediately that a remote node died</strong>.
You will find out about it after some time, eventually.</p>
<h4>Heartbeats</h4>
<p>How would you check if someone is dead on the real world? A good idea might be
to check if he can talk. If someone is talking, he must be alive.</p>
<p>A similar solution can be used over the Internet. To make sure that a remote
node is alive, you just check if he recently sent a message.
Going further with this idea, we can have the following arrangement in our
network: If <span class="math">\(x\)</span> is a node that is linked to <span class="math">\(y\)</span>, <span class="math">\(x\)</span> is expecting to get
a message from <span class="math">\(y\)</span> at least as frequent as every 10 seconds. If it doesn’t
happen, <span class="math">\(x\)</span> will assume that <span class="math">\(y\)</span> is dead. <span class="math">\(y\)</span> expects the same from
<span class="math">\(x\)</span>.</p>
<p>But what if <span class="math">\(y\)</span> doesn’t have anything important to say to <span class="math">\(x\)</span>, and 10
seconds are soon going to pass? In that case, <span class="math">\(y\)</span> will send some kind of a
dummy message to <span class="math">\(x\)</span>. A dummy message that is used to prove that <span class="math">\(y\)</span> is alive.</p>
<p>This technique is sometimes referred to as <em>Heartbeat</em> messages.</p>
<p>Two notes here:</p>
<ul>
<li>
<p>For the Transport Protocols savvy people: Some claim that a <span class="caps">TCP</span> connection
would be enough to hold a link. It might be true in some cases, though some
NATs will be happy to close your <span class="caps">TCP</span> connection if some time passes and no
communication occurs, so sending Heartbeats is generally a good practice
(At least in my opinion).</p>
</li>
<li>
<p>To see the theoretical incompleteness of this method of understand the
state of a remote node, I recommend you to read about the <a href="http://en.wikipedia.org/wiki/Two_Generals%27_Problem">Two Generals’
Problem</a>. (You could also
just look at the arbitrary “10 seconds” constant, and realize that there is
nothing very deep here).</p>
</li>
</ul>
<h4>Leaving nodes don’t care</h4>
<p>One might wonder what should a node do when he wants to leave the network. It
might seem reasonable to have some kind of predefined procedure for a node that
wants to leave the network. For example - Maybe that node would have to notify
some of his neighbours about his intentions to leave.</p>
<p>A way to simplify things would be to not distinguish between the case of a node
dying and a node leaving the network intentionally. We will just assume that
every node <span class="math">\(x\)</span> that want to leave the network will just stop responding. After some
time will pass, all the nodes that are linked to <span class="math">\(x\)</span> will assume that <span class="math">\(x\)</span>
has left the network.</p>
<p>This is also a good security assumption (that prepares us for our future
discussions). Generally, we assume that <strong>Leaving nodes don’t care</strong> about our
network procedures. That is because generally they have no incentives to behave
well. Joining nodes do have incentives to behave well. If they don’t, we won’t
help them join the network. But leaving nodes really don’t care. Most likely
they have already got what they want from the network, and they can just pull
the network plug.</p>
<h3>Stabilizing the Ring</h3>
<p>I remind you that the chord network is just a ring of nodes (Each node
connected to the next one), with some extra links. Those extra links make
searches much faster, though they are not really needed for the search
operation to function. So lets forget about them for a moment, and go back to
the basic ring idea.</p>
<p>Now we are left with a very simple ring of nodes, where every node is connected
to the next node. This structure is pretty fragile. Every node that dies will
destroy the ring structure. </p>
<p>One simple improvement would be that each node will remember two nodes, instead
of one. Each node will remember the previous and the next node on the ring.
Generalizing this idea: Every node will be linked to the closest node and to
the farthest node, with respect to the distance function <span class="math">\(d\)</span>. (I remind you
that for <span class="math">\(x,y \in B_s\)</span>, <span class="math">\(d(x,y) = y - x\)</span> if <span class="math">\(y > x\)</span>, and <span class="math">\(d(x,y) = 2^s
+ (y - x)\)</span> otherwise.</p>
<p><img alt="single vs double link" src="https://www.freedomlayer.org/articles/freedomlayer/chord_stabilize/images/single_vs_double_link.svg"></p>
<p>(On the left: A one way link ring network. Every node is connected to the next
one. On the right: A two way link ring network. It is a bit more robust.)</p>
<p>This time, if one node dies, the network is still fixable. Take a few moments
to think about how to fix it.</p>
<h4>Basic Stabilize</h4>
<p>We could add some kind of a periodic maintenance job to each node, called
“stabilize”. Every node will invoke it every once in a while (Every 40 seconds,
for example). For every node <span class="math">\(x\)</span> we denote <span class="math">\(x.Links\)</span> to be the set of links
from <span class="math">\(x\)</span>. In this case, we have that <span class="math">\(x.Links = \{x.next,x.pref\}\)</span>.</p>
<p>We now define the Stabilize algorithm:</p>
<p><strong>Stabilize</strong> [Performed by node <span class="math">\(x\)</span> in the case of Doubly linked ring]</p>
<ol>
<li>Initialize set <span class="math">\(x.Known = \{x.prev,x.next\}\)</span></li>
<li>Ask <span class="math">\(x\)</span>.next for his <span class="math">\(x.next.Links\)</span> set, and add those links to
<span class="math">\(x.Known\)</span>. </li>
<li>Ask <span class="math">\(x\)</span>.next for his <span class="math">\(x.prev.Links\)</span> set, and add those links to
<span class="math">\(x.Known\)</span>. </li>
<li>Remove <span class="math">\(x\)</span> from the set <span class="math">\(x.Known\)</span> (If he was there at all).</li>
<li>Let <span class="math">\(new\_prev = argmin_{z \in x.Known}{d(z,x)}\)</span>. (In other words: Get
the node <span class="math">\(z \in x.Known\)</span> such that <span class="math">\(z\)</span> minimizes the value <span class="math">\(d(z,x\)</span>.)</li>
<li>Let <span class="math">\(new\_next = argmin_{z \in x.Known}{d(x,z)}\)</span></li>
<li>Set <span class="math">\(x\)</span>.next = <span class="math">\(new\_next\)</span>. Set <span class="math">\(x\)</span>.prev = <span class="math">\(new\_prev\)</span>.</li>
</ol>
<p>Let’s try to understand how Stabilize works. Every node <span class="math">\(x\)</span> asks his neighbours for
their neighbours. He then takes a look at all the nodes he know (Except for
himself), and try to find the two most suitable next and previous nodes.
The most suitable “next node” will be the node <span class="math">\(z\)</span> such that <span class="math">\(d(x,z)\)</span> is
minimized. The most suitable “previous node” will be the node <span class="math">\(z\)</span> such that
<span class="math">\(d(z,x)\)</span> is minimized.</p>
<p>In the usual case, performing stabilize will not change the node’s next and
previous node links, however it could be useful if something has changed in the
network layout. Let’s begin with the case of some node <span class="math">\(y\)</span> that dies.</p>
<p>Let’s assume that <span class="math">\(y\)</span><span class="quo">‘</span>s previous node was <span class="math">\(x\)</span>, and <span class="math">\(y\)</span><span class="quo">‘</span>s next node was
<span class="math">\(z\)</span>. Therefore at some point <span class="math">\(x\)</span> and <span class="math">\(z\)</span> will notice that <span class="math">\(y\)</span> has died.
Both of them will miss a link at this point. <span class="math">\(x\)</span> will miss the “next” link,
and <span class="math">\(z\)</span> will miss the “previous” link. It could have been nice if <span class="math">\(x\)</span> and
<span class="math">\(z\)</span> could just somehow connect to each other, resulting in <span class="math">\(x\)</span>.next = <span class="math">\(z\)</span>
and <span class="math">\(z\)</span>.prev = <span class="math">\(x\)</span>, but in our case <span class="math">\(x\)</span> and <span class="math">\(z\)</span> do not have enough
information to do that. They just don’t see far enough.</p>
<p><img alt="y disconnected" src="https://www.freedomlayer.org/articles/freedomlayer/chord_stabilize/images/y_disconnected.svg"></p>
<p>(On the left: The original state of the network. On the right: The state of the
network after <span class="math">\(y\)</span> has failed.</p>
<p>Now let’s investigate what happens to <span class="math">\(x\)</span> and <span class="math">\(z\)</span> after a while. For
reasons of readability, let’s just think about <span class="math">\(x\)</span>, and conclude the same for
<span class="math">\(z\)</span> later. They are symmetric. <span class="math">\(x\)</span> will invoke the Stabilize operation
after a while. Let’s mark <span class="math">\(x\)</span>.prev = <span class="math">\(x_1\)</span>. We will also mark the next
nodes counter-clockwise to be <span class="math">\(x_2, x_3, \dots\)</span>.</p>
<p>So <span class="math">\(x\)</span> will ask <span class="math">\(x_1\)</span> (His previous link) for all his neighbours. Those
will be <span class="math">\(x\)</span> and <span class="math">\(x_2\)</span>. <span class="math">\(x\)</span> will finally set <span class="math">\(x\)</span>.next = <span class="math">\(x_1\)</span>, and
<span class="math">\(x\)</span>.prev = <span class="math">\(x_1\)</span>. (This one was left unchanged). Please follow the
Stabilize algorithm and make sure that you understand why).</p>
<p><img alt="y disconnected stabilize 1" src="https://www.freedomlayer.org/articles/freedomlayer/chord_stabilize/images/y_disconnected_stb1.svg"></p>
<p>In this picture: The state of the network after one Stabilize iteration. We
mark by purple arrows the links that are not yet optimal. (They are going to
change in the next iteration of Stabilize)</p>
<p>Note that all other nodes will in the ring will perform Stabilize too, however
for every node that is not <span class="math">\(x\)</span> or <span class="math">\(z\)</span> the links are already optimized, so
they will not change. </p>
<p>After one iteration of Stabilize we get that <span class="math">\(x\)</span>.prev = <span class="math">\(x_1\)</span>.
<span class="math">\(x\)</span>.next = <span class="math">\(x_2\)</span>. In the next iteration we will get that <span class="math">\(x\)</span>.prev =
<span class="math">\(x_1\)</span>, and <span class="math">\(x\)</span>.next = <span class="math">\(x_3\)</span>, and so on. After <span class="math">\(O(n)\)</span> iterations of
Stabilize, we expect that <span class="math">\(x\)</span>.next = <span class="math">\(z\)</span>.</p>
<p>We expect the same from <span class="math">\(z\)</span>, but in the other direction. In the beginning we
have that <span class="math">\(z\)</span>.prev is dead, and <span class="math">\(z\)</span>.next = <span class="math">\(z_1\)</span>. In the next
iteration we have <span class="math">\(z\)</span>.prev = <span class="math">\(z_2\)</span>, and <span class="math">\(z\)</span>.next = <span class="math">\(z_1\)</span> (Left
unchanged). After <span class="math">\(O(n)\)</span> iterations of Stabilize, we will get that
<span class="math">\(z\)</span>.prev = <span class="math">\(x\)</span>.</p>
<p>It really takes a long time for this network to fix itself after one node dies,
However it is interesting to see that it fixes itself eventually. As opposed to
the one dying node case, if two nodes die the network is not connected anymore,
and there is no hope to fix it.</p>
<p><img alt="y disconnected stabilize all" src="https://www.freedomlayer.org/articles/freedomlayer/chord_stabilize/images/y_disconnected_stb_all.svg"></p>
<p>In the picture: The self fixing process of the network using iterations of
Stabilize (After a node <span class="math">\(y\)</span> failed). It is very slow.</p>
<p>Generally, if the network becomes not connected (There are two nodes <span class="math">\(a,b\)</span>
for which there is no path of links leading from <span class="math">\(a\)</span> to <span class="math">\(b\)</span>), there is no
hope to fix it.</p>
<p>Joining the network is simple in this case, and could be done without requiring
any Stabilize iterations. Think how to do it.</p>
<h3>Thickening the Ring</h3>
<p>One simple way to make the simple ring more robust is to add links to
more neighbours. Instead of having node <span class="math">\(x\)</span> linked to his immediate next
and previous neighbours on the ring, we will link <span class="math">\(x\)</span> to his <span class="math">\(k\)</span> immediate
next neighbours on the ring, and <span class="math">\(k\)</span> immediate previous neighbours on the ring.</p>
<p><img alt="ring 2" src="https://www.freedomlayer.org/articles/freedomlayer/chord_stabilize/images/ring_2.svg"></p>
<p>In the picture: An example ring with <span class="math">\(k = 2\)</span>. In this network one failed node
will be fixed quickly.</p>
<p>We have already discussed this structure in <a href="https://www.freedomlayer.org/intro-to-distributed-hash-tables-dhts.html">Intro to
DHTs</a>
as a naive approach for increasing search speed. This time we want to use this
structure to make the network more resilient to network churn.</p>
<p>As before, we denote <span class="math">\(x\)</span>.Links to be the set of links from <span class="math">\(x\)</span>.
We describe here the Stabilize algorithm for this case.</p>
<p><strong>Stabilize<span class="math">\(_k\)</span></strong> [Performed by node <span class="math">\(x\)</span> in the case of <span class="math">\(k\)</span>-linked ring]</p>
<ol>
<li>Initialize set <span class="math">\(x.Known\)</span> to be the set of links <span class="math">\(x\)</span>.Links.</li>
<li>for all <span class="math">\(t \in x.Links\)</span> Ask <span class="math">\(t\)</span> for his set <span class="math">\(t.Links\)</span>.
Add the contents of this set to <span class="math">\(x.Known\)</span>.</li>
<li>Remove <span class="math">\(x\)</span> from the set <span class="math">\(x.Known\)</span> (If he was there at all).</li>
<li>Out of all nodes in <span class="math">\(x.Known\)</span>, find the best <span class="math">\(k\)</span> nodes that minimize
<span class="math">\(d(z,x)\)</span> for <span class="math">\(z \in x.Known\)</span>, and set those nodes to be
<span class="math">\(x.prev_1,\dots,x.prev_k\)</span>. (If there are not enough nodes in <span class="math">\(x.Known\)</span>,
the last prev link will be Null).</li>
<li>Out of all nodes in <span class="math">\(x.Known\)</span>, find the best <span class="math">\(k\)</span> nodes that minimize
<span class="math">\(d(x,z)\)</span> for <span class="math">\(z \in x.Known\)</span>, and set those nodes to be
<span class="math">\(x.prev_1,\dots,x.prev_k\)</span>.</li>
</ol>
<p>This Stabilize algorithm is not very different from the doubly connected ring
one. (In fact, the previous Stabilize algorithm is just this one with <span class="math">\(k=1\)</span>.
This version of Stabilize communicates with more nodes (Usually with <span class="math">\(2\cdot
k\)</span> nodes). At all times a node <span class="math">\(x\)</span> tries to maintain a list of <span class="math">\(k\)</span> nodes
that are closest as possible to <span class="math">\(x\)</span> but not bigger than <span class="math">\(x\)</span>, and also a
list of <span class="math">\(k\)</span> nodes that are closest as possible to <span class="math">\(x\)</span> but bigger than
<span class="math">\(x\)</span>.</p>
<p>In the k-connected ring with Stabilize<span class="math">\(_k\)</span>, if any <span class="math">\(k-1\)</span> nodes die the ring
will fix itself quickly. (It is enough to wait for one iteration of
Stabilize<span class="math">\(_k\)</span> done by all nodes in the ring). However if <span class="math">\(k\)</span> consecutive
nodes on the ring die at the same time, it will take longer for the ring to fix
itself, just like the case of 1 node dying in the doubly linked ring.</p>
<p>If <span class="math">\(2\cdot k\)</span> nodes die at the same time, <span class="math">\(k\)</span> consecutive nodes in one part
of the ring, and other <span class="math">\(k\)</span> consecutive nodes in another part, we expect the
network to become disconnected. </p>
<h3>Adding the Far links</h3>
<p>I remind you that the fast search operation was obtained by adding links for
far nodes. We could extend the <span class="math">\(k\)</span>-connected ring to have those links too.
That way we get a more reliable network in which we have a fast search operation. </p>
<p>Given that the name of nodes come from the set <span class="math">\(B_s := \{0,1,2,\dots,2^{s}-1
\}\)</span>, the links we want to add to node <span class="math">\(x\)</span> will be:
<span class="math">\(\ceil{x + 1}\)</span>,
<span class="math">\(\ceil{x + 2}\)</span>, <span class="math">\(\dots\)</span>,
<span class="math">\(\ceil{x + 2^{s-1}}\)</span>. For symmetry we will also add:
<span class="math">\(\floor{x - 1}\)</span>,
<span class="math">\(\floor{x - 2}\)</span>,<span class="math">\(\dots\)</span>,
<span class="math">\(\floor{x - 2^{s-1}}\)</span>.</p>
<p>I also remind you that by <span class="math">\(\ceil{y}\)</span> we mean the first node (clockwise) that has a
name bigger or equal to <span class="math">\(y\)</span>. By <span class="math">\(\floor{y}\)</span> we mean the first node
(counter-clockwise) that has a name smaller or equal to <span class="math">\(y\)</span>.</p>
<p><img alt="floor and ceil" src="https://www.freedomlayer.org/articles/freedomlayer/chord_stabilize/images/floor_ceil.svg"></p>
<p>In the picture: Some value <span class="math">\(v\)</span>, and the nodes <span class="math">\(\floor{v}, \ceil{v}\)</span></p>
<p>For simplicity’s sake we will call the original links from the <span class="math">\(k\)</span>-connected
ring <em>The Local links</em>. The new links we have just added will be called <em>The
Far links</em>. We denote <span class="math">\(x.Local\)</span> to be <span class="math">\(x\)</span><span class="quo">‘</span>s local links,
<span class="math">\(x.Far\)</span> to be the far links, and <span class="math">\(x.Links = x.Local \cup
x.Far\)</span> to be all the nodes <span class="math">\(x\)</span> is linked to.</p>
<p>The new stabilize algorithm is as follows:</p>
<p><strong>Stabilize<span class="math">\(^*_k\)</span></strong> [Performed by node <span class="math">\(x\)</span> in the case of <span class="math">\(k\)</span>-linked ring with Far nodes.]</p>
<ol>
<li>Initialize <span class="math">\(x\)</span>.Known to be the <span class="math">\(x\)</span>.Links.</li>
<li>for all <span class="math">\(t \in x.Links\)</span> Ask <span class="math">\(t\)</span> for his set <span class="math">\(t\)</span>.Links.
Add the contents of this set to <span class="math">\(x.Known\)</span>.</li>
<li>Remove <span class="math">\(x\)</span> from the set <span class="math">\(x.Known\)</span> (If he was there at all).</li>
<li>Out of all nodes in <span class="math">\(x.Known\)</span>, find the best <span class="math">\(k\)</span> nodes that minimize
<span class="math">\(d(z,x)\)</span> for <span class="math">\(z \in x.Known\)</span>, and set those nodes to be
<span class="math">\(x.prev_1,\dots,x.prev_k\)</span>. (If there are not enough nodes in <span class="math">\(x.Known\)</span>,
the last prev link will be Null).</li>
<li>
<p>Out of all nodes in <span class="math">\(x.Known\)</span>, find the best <span class="math">\(k\)</span> nodes that minimize
<span class="math">\(d(x,z)\)</span> for <span class="math">\(z \in x.Known\)</span>, and set those nodes to be
<span class="math">\(x.prev_1,\dots,x.prev_k\)</span>.</p>
</li>
<li>
<p>For all <span class="math">\(1 \leq j < s\)</span>: Set <span class="math">\(x.next\_far_{j} := \arg\min_{z \in
x.Known}{d(x+2^{j},z)}\)</span>. (In other words: We take the first (Clockwise) <span class="math">\(z \in
x.Known\)</span> that has a name bigger than <span class="math">\(x + 2^{j}\)</span>.)</p>
</li>
<li>For all <span class="math">\(1 \leq j < s\)</span>: Set <span class="math">\(x.prev\_far_{j} := \arg\min_{z \in x.Known}{d(z,x-2^{j})}.\)</span></li>
</ol>
<p>A short explanation of the Stabilize<span class="math">\(^*_k\)</span> algorithm:
Every node <span class="math">\(x\)</span> first asks all the nodes he is linked to for all the nodes
they are linked to. Just to make sure you follow me here, <span class="math">\(x\)</span> should be
linked to about <span class="math">\(c := \left|x.Links\right| = 2\cdot k + 2\cdot (s-1)\)</span> nodes.
He might be actually linked to less nodes, because some Local links coincide
with Far links.</p>
<p><span class="math">\(x\)</span> collects a set of known links. This set consists of <span class="math">\(x\)</span> own immediate
links, and the links sets of <span class="math">\(x\)</span> immediate links. This will result in no more
than <span class="math">\(c^2 + c\)</span> links to unique nodes.</p>
<p>Out of this collected <span class="math">\(x.Known\)</span> set, <span class="math">\(x\)</span> sets up the new Local and Far
links. The Local links will be the <span class="math">\(k\)</span> closest nodes to <span class="math">\(x\)</span> from the set
<span class="math">\(x\)</span>.Known. (From both sides of <span class="math">\(x\)</span>). To find the new Far links, <span class="math">\(x\)</span> will
try to find the nodes closest to <span class="math">\(x + 2^{j}\)</span> (and <span class="math">\(x - 2^{j}\)</span> for the
symmetric side).</p>
<h4>Constructing the Far from the Local</h4>
<p>One easy way to think about this new structure (k-linked ring + Far links) is
to separate the analysis of Local links and Far links. To get away with it, we
will first show that the Far links could always be built from the Local links
quickly. A good start would be to think about it yourself. Given a <span class="math">\(k\)</span>-linked
ring, how can you build the Far links to enable fast search?</p>
<p>A good idea would be to run Stabilize<span class="math">\(^*_k\)</span> a few times. It seems like
Stabilize<span class="math">\(^*_k\)</span> deals with the Far links. Maybe it will manage to build them
correctly. We hope that running some iterations of Stabilize<span class="math">\(^*_k\)</span> will
converge to a structure where all the Local and Far links are optimal.</p>
<p>To simplify things, we assume a perfect world where all the nodes in the ring
perform Stabilize<span class="math">\(^*_k\)</span> exactly at the same time. (I will not discuss here
the asynchronous case, but it’s a good thought experiment). We call it a
Stabilize<span class="math">\(^*_k\)</span> iteration.</p>
<p>I claim that in about <span class="math">\(O(log(n))\)</span> iterations of Stabilize<span class="math">\(^*_k\)</span> the Far
links will be fully constructed. I thought that I know how to prove it, but it
seems like I can’t find an elegant way to explain it. If you have any
interesting proof (Even with conditions that are a bit different), please tell
me about it.</p>
<h5>Stabilize code example</h5>
<p>I don’t want you to leave empty handed (You got so far already), so I
wrote a simple simulation for the Stabilize<span class="math">\(^*_k\)</span> algorithm in Python. It
shows how Stabilize<span class="math">\(^*_k\)</span> can create the Far links from the Local links in
about <span class="math">\(O(\log(n))\)</span> iterations. You can find it <a href="https://github.com/realcr/freedomlayer_code/blob/master/chord_stabilize/build_far_from_local.py">here
[github]</a>.</p>
<p>Some words about the code. It is not very fast. (You could optimize it though).
On my computer you could simulate a ring of 2^10 nodes, though you will have to
wait a few minutes to get a result. To use it, just run it with python. (I use
Python3). If you want to tweak the parameters, go to the last function and
change them. You can change both <span class="math">\(n\)</span> and <span class="math">\(k\)</span>. <span class="math">\(n\)</span> is the amount of nodes
in the ring, and <span class="math">\(k\)</span> is the ring constant. (Every node is connected to his
<span class="math">\(k\)</span> closest neighbours from both sides).</p>
<p>If you just want to see how Stabilize<span class="math">\(^*_k\)</span> looks like in the code, I
include it here: <%text></p>
<div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">stabilize</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span><span class="n">node_name</span><span class="p">):</span>
<span class="sd">"""</span>
<span class="sd"> Try improving immediate links by asking first level links.</span>
<span class="sd"> Done according to the Chord Stabilizing article.</span>
<span class="sd"> """</span>
<span class="c1"># Get the node's class:</span>
<span class="n">nd</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">nodes</span><span class="p">[</span><span class="n">node_name</span><span class="p">]</span>
<span class="c1"># Initialize the Known nodes set:</span>
<span class="n">known</span> <span class="o">=</span> <span class="nb">set</span><span class="p">(</span><span class="n">nd</span><span class="o">.</span><span class="n">links</span><span class="p">)</span>
<span class="k">for</span> <span class="n">ln</span> <span class="ow">in</span> <span class="n">nd</span><span class="o">.</span><span class="n">links</span><span class="p">:</span>
<span class="n">known</span><span class="o">.</span><span class="n">update</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">nodes</span><span class="p">[</span><span class="n">ln</span><span class="p">]</span><span class="o">.</span><span class="n">links</span><span class="p">)</span>
<span class="c1"># Remove myself from the known set:</span>
<span class="n">known</span><span class="o">.</span><span class="n">discard</span><span class="p">(</span><span class="n">node_name</span><span class="p">)</span>
<span class="c1"># Find the optimal local links:</span>
<span class="n">known_lst</span> <span class="o">=</span> <span class="nb">list</span><span class="p">(</span><span class="n">known</span><span class="p">)</span>
<span class="n">nd</span><span class="o">.</span><span class="n">local_links</span> <span class="o">=</span> <span class="nb">set</span><span class="p">()</span>
<span class="c1"># Find "before nd" k best local links:</span>
<span class="n">known_lst</span><span class="o">.</span><span class="n">sort</span><span class="p">(</span><span class="n">key</span><span class="o">=</span><span class="k">lambda</span> <span class="n">z</span><span class="p">:</span><span class="bp">self</span><span class="o">.</span><span class="n">name_dist</span><span class="p">(</span><span class="n">z</span><span class="p">,</span><span class="n">node_name</span><span class="p">))</span>
<span class="n">nd</span><span class="o">.</span><span class="n">local_links</span><span class="o">.</span><span class="n">update</span><span class="p">(</span><span class="nb">set</span><span class="p">(</span><span class="n">known_lst</span><span class="p">[:</span><span class="bp">self</span><span class="o">.</span><span class="n">k</span><span class="p">]))</span>
<span class="c1"># Find "after nd" k best local links:</span>
<span class="n">known_lst</span><span class="o">.</span><span class="n">sort</span><span class="p">(</span><span class="n">key</span><span class="o">=</span><span class="k">lambda</span> <span class="n">z</span><span class="p">:</span><span class="bp">self</span><span class="o">.</span><span class="n">name_dist</span><span class="p">(</span><span class="n">node_name</span><span class="p">,</span><span class="n">z</span><span class="p">))</span>
<span class="n">nd</span><span class="o">.</span><span class="n">local_links</span><span class="o">.</span><span class="n">update</span><span class="p">(</span><span class="nb">set</span><span class="p">(</span><span class="n">known_lst</span><span class="p">[:</span><span class="bp">self</span><span class="o">.</span><span class="n">k</span><span class="p">]))</span>
<span class="c1"># Find optimal far links:</span>
<span class="n">nd</span><span class="o">.</span><span class="n">far_links</span> <span class="o">=</span> <span class="nb">set</span><span class="p">()</span>
<span class="k">for</span> <span class="n">j</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">NODE_NAMESPACE</span><span class="p">):</span>
<span class="c1"># First direction:</span>
<span class="n">opt</span> <span class="o">=</span> <span class="nb">min</span><span class="p">(</span><span class="n">known_lst</span><span class="p">,</span><span class="n">key</span><span class="o">=</span><span class="k">lambda</span> <span class="n">z</span><span class="p">:</span><span class="bp">self</span><span class="o">.</span><span class="n">name_dist</span><span class="p">(</span>\
<span class="p">(</span><span class="n">node_name</span> <span class="o">+</span> <span class="p">(</span><span class="mi">2</span><span class="o">**</span><span class="n">j</span><span class="p">))</span> <span class="o">%</span> <span class="n">NODE_NAMESPACE_SIZE</span><span class="p">,</span><span class="n">z</span><span class="p">))</span>
<span class="n">nd</span><span class="o">.</span><span class="n">far_links</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="n">opt</span><span class="p">)</span>
<span class="c1"># Second direction:</span>
<span class="n">opt</span> <span class="o">=</span> <span class="nb">min</span><span class="p">(</span><span class="n">known_lst</span><span class="p">,</span><span class="n">key</span><span class="o">=</span><span class="k">lambda</span> <span class="n">z</span><span class="p">:</span><span class="bp">self</span><span class="o">.</span><span class="n">name_dist</span><span class="p">(</span>\
<span class="n">z</span><span class="p">,(</span><span class="n">node_name</span> <span class="o">-</span> <span class="p">(</span><span class="mi">2</span><span class="o">**</span><span class="n">j</span><span class="p">))</span> <span class="o">%</span> <span class="n">NODE_NAMESPACE_SIZE</span><span class="p">))</span>
<span class="n">nd</span><span class="o">.</span><span class="n">far_links</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="n">opt</span><span class="p">)</span>
<span class="n">nd</span><span class="o">.</span><span class="n">links</span> <span class="o">=</span> <span class="n">nd</span><span class="o">.</span><span class="n">local_links</span><span class="o">.</span><span class="n">union</span><span class="p">(</span><span class="n">nd</span><span class="o">.</span><span class="n">far_links</span><span class="p">)</span>
</pre></div>
</%text>
<h5>Local Links are enough</h5>
<p>By showing that the Far links could be constructed from the Local links in a
short time, we can conclude that as long as the k-linked ring could be
reconstructed, the whole structure of Far Links could be reconstructed.</p>
<h3>Final notes</h3>
<p>We showed that a k-linked ring survives and quickly
recovers from the sudden failure of any <span class="math">\(k-1\)</span> nodes. Therefore this result
applies also to a k-linked ring with Far links (Because the Far links could
always be reconstructed quickly from the Local links). This is a pretty weak
theoretical result, as we only considered the Local links when deriving it.</p>
<p>However, the Stabilize algorithm we have presented is much stronger than we
managed to prove here. The Far links probably add much more to the stability of
the structure. At this point we don’t bother to fully analyze or prove it,
though I’m pretty sure that there many articles and research about this subject
out there.</p>
<p>Another interesting thing that we have found is that it is enough to invoke the
Stabilize<span class="math">\(^*_k\)</span> operation periodically (by all nodes) to preserve the
total structure (k-linked ring + Far links). In fact, this is the only
operation we ever have to invoke to keep up with network events. (Node failures
and joins). We don’t have to remember lots of rules that apply in different cases
or times. It might not be the most optimal thing to do, however it is a very
safe way of constructing a simple and stable structure that stays both
connected and searchable.</p>
<p>As a thought exercise, follow what happens when a new node joins. (See how many
Stabilize Iterations it takes for him to obtain all the optimal Far links).</p>
<p>Another thing to mention is what we haven’t discussed here: How to preserve the
information inside the nodes in the network, if we use it as a <span class="caps">DHT</span>. In other
words: If a few nodes fail, we already know how to reconstruct the network.
What we still have to think about is how to deal with losing the data inside
those failed nodes.</p>
<script type="text/javascript">if (!document.getElementById('mathjaxscript_pelican_#%@#$@#')) {
var align = "center",
indent = "0em",
linebreak = "false";
if (false) {
align = (screen.width < 768) ? "left" : align;
indent = (screen.width < 768) ? "0em" : indent;
linebreak = (screen.width < 768) ? 'true' : linebreak;
}
var mathjaxscript = document.createElement('script');
mathjaxscript.id = 'mathjaxscript_pelican_#%@#$@#';
mathjaxscript.type = 'text/javascript';
mathjaxscript.src = 'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.0/MathJax.js?config=TeX-AMS-MML_HTMLorMML';
mathjaxscript[(window.opera ? "innerHTML" : "text")] =
"MathJax.Hub.Config({" +
" config: ['MMLorHTML.js']," +
" TeX: { extensions: ['AMSmath.js','AMSsymbols.js','noErrors.js','noUndefined.js'], equationNumbers: { autoNumber: 'AMS' } }," +
" jax: ['input/TeX','input/MathML','output/HTML-CSS']," +
" extensions: ['tex2jax.js','mml2jax.js','MathMenu.js','MathZoom.js']," +
" displayAlign: '"+ align +"'," +
" displayIndent: '"+ indent +"'," +
" showMathMenu: true," +
" messageStyle: 'normal'," +
" tex2jax: { " +
" inlineMath: [ ['\\\\(','\\\\)'] ], " +
" displayMath: [ ['$$','$$'] ]," +
" processEscapes: true," +
" preview: 'TeX'," +
" }, " +
" 'HTML-CSS': { " +
" styles: { '.MathJax_Display, .MathJax .mo, .MathJax .mi, .MathJax .mn': {color: 'inherit ! important'} }," +
" linebreaks: { automatic: "+ linebreak +", width: '90% container' }," +
" }, " +
"}); " +
"if ('default' !== 'default') {" +
"MathJax.Hub.Register.StartupHook('HTML-CSS Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax['HTML-CSS'].FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"MathJax.Hub.Register.StartupHook('SVG Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax.SVG.FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"}";
(document.body || document.getElementsByTagName('head')[0]).appendChild(mathjaxscript);
}
</script>Intro to Distributed Hash Tables (DHTs)2014-11-11T13:07:00+00:002014-11-11T13:07:00+00:00realtag:www.freedomlayer.org,2014-11-11:/intro-to-distributed-hash-tables-dhts.html<!--
ceil and floor latex macros:
-->
<div class="math">$$
\newcommand{\ceil}[1]{\left\lceil{#1}\right\rceil}
\newcommand{\floor}[1]{\left\lfloor{#1}\right\rfloor}
$$</div>
<h3>Abstract</h3>
<p>We introduce the idea of the Chord <span class="caps">DHT</span> from scratch, giving some intuition for
the decisions made in the design of Chord.</p>
<h3>Building a phone list</h3>
<p>I want to begin with an example …</p><!--
ceil and floor latex macros:
-->
<div class="math">$$
\newcommand{\ceil}[1]{\left\lceil{#1}\right\rceil}
\newcommand{\floor}[1]{\left\lfloor{#1}\right\rfloor}
$$</div>
<h3>Abstract</h3>
<p>We introduce the idea of the Chord <span class="caps">DHT</span> from scratch, giving some intuition for
the decisions made in the design of Chord.</p>
<h3>Building a phone list</h3>
<p>I want to begin with an example from life. You might want to read it even if you
have some general knowledge about DHTs, because it might give you some new
ideas about where DHTs come from.</p>
<p>On your cellphone, most likely you have a list of contacts. Could you maintain
contact with all your friends without having this list?
More specifically - What if every person in the world could remember only about
40 phone numbers. Given that structure, could we make sure that every person in
the world will be able to call any other person in the world?</p>
<p>In the spirit of no hierarchical related solutions, we will also want to have a
solution where all the participants have more or less symmetric roles.</p>
<h4>First solution - Phone ring</h4>
<p><br/></p>
<h5>General structure</h5>
<p>A simple solution would be as follows: We sort the names of all the people in
the world into a very big list. (Assume that people have unique names, just for
this article :) ). Next, every person will have the responsibility of
remembering one phone number: The phone number of the next person on the list.</p>
<p>As an example, if the list is as follows:</p>
<div class="highlight"><pre><span></span>Benito Kellner
Britney Antonio
Cassi Dewolfe
Cleotilde Vandyne
Colene Kaufmann
Cordell Varley
Denae Fernandez
Donnette Thornberry
Edwin Peters
Georgine Reneau
</pre></div>
<p>Then Britney will keep the phone number of Cassi. Cassi, in turn,
keeps the phone number of Cleotilde. Cleotilde keeps the phone
number of Colene, and so on. </p>
<p>The list is cyclic. You can think of it as a ring, more than as a list. The
last person on the list will remember the phone number of the first person on
the list.
(In our list, it means that Georgine keeps the phone number of Benito).</p>
<p><img alt="The phone list drawn as a ring, with lines representing the connection between
people on the list." src="https://www.freedomlayer.org/articles/freedomlayer/dht_intro/images/linear_dht_circle.svg"></p>
<p>The phone list drawn as a ring, with lines representing the connection between
people on the list.</p>
<p>Now assume that Benito wants to call Edwin. How can he do that? He will first
call Britney, because he knows her phone number. He will ask Britney for the
name and phone number of the next person on the list. That would be Cassi.</p>
<p>Next Benito will call Cassi, and ask her for the name and phone number of the
next person on the list. That would be Cleotilde. At this point Benito can
forget the name and phone number of Cassi, and move on to calling Cleotilde.
Benito will keep advancing in the list until he finally finds Edwin.</p>
<p>We call this operation of finding someone on the list a query, or a search.</p>
<h5>Joining the ring</h5>
<p>Assume that some person <span class="math">\(X\)</span> wants to join the phone list. How can we
add <span class="math">\(X\)</span> so that the structure is preserved?</p>
<p><span class="math">\(X\)</span> will first contact some person <span class="math">\(Y\)</span> that is already on the list. Let us
assume that <span class="math">\(X\)</span> contacts Denae for example. Denae will then search for a
suitable place for <span class="math">\(X\)</span> on the cyclic list, so that the list will stay sorted.
If in our example <span class="math">\(X\)</span> is Gary Jablonski, Then Denae search will yield that
Gary should be put between Edwin and Georgine.</p>
<p>After <span class="math">\(Y\)</span> Finds a place for <span class="math">\(X\)</span> on the list, <span class="math">\(Y\)</span> will tell <span class="math">\(X\)</span> about
his designated location in the list. Then <span class="math">\(X\)</span> will join the list at this
place. (We assume that <span class="math">\(X\)</span> is a good person, and he will just go to his
designated place without giving us any trouble.)</p>
<p>Following our example of Gary Jablonski joining the list, the new list will
look somehow like this:</p>
<div class="highlight"><pre><span></span>Benito Kellner
Britney Antonio
Cassi Dewolfe
Cleotilde Vandyne
Colene Kaufmann
Cordell Varley
Denae Fernandez
Donnette Thornberry
Edwin Peters
Gary Jablonski
Georgine Reneau
</pre></div>
<p>Of course that in the new setting, Edwin for example now has to remember only
Gary’s phone. He shouldn’t keep remembering Georgine’s phone number, because
it is not needed anymore.</p>
<p><img alt="The new state of the list, after Gary has joined" src="https://www.freedomlayer.org/articles/freedomlayer/dht_intro/images/join_linear_dht_circle.svg"></p>
<p>The new state of the list, after Gary has joined.</p>
<h5>Analysis</h5>
<p>Whenever person <span class="math">\(A\)</span> wants to find person <span class="math">\(B\)</span> on the list, he will have to
traverse the list of people one by one until he finds <span class="math">\(B\)</span>. It could take a
very short time if <span class="math">\(A\)</span> and <span class="math">\(B\)</span> are close on this list, however it could
also take a very long time if <span class="math">\(A\)</span> and <span class="math">\(B\)</span> are very far (In the cyclic
sense. In the worst case, <span class="math">\(B\)</span> is right before <span class="math">\(A\)</span> on the list).</p>
<p>However we could find the average time it takes for <span class="math">\(A\)</span> to contact <span class="math">\(B\)</span>. It
would be about <span class="math">\(\frac{n}{2}\)</span>, where <span class="math">\(n\)</span> is the amount of people on the list.</p>
<p>In addition, we can also measure the amount of memory used for each of the
people on the list. Every person is responsible for remembering exactly one
people’s name and phone number. (The next one on the list). </p>
<p>Whenever a person wants to call someone, he will have to remember an additional
phone number, which is the next person he is going to call. This is not much to
remember though.</p>
<p>In more mathematical terms, we say that a search (or a query) costs <span class="math">\(O(n)\)</span>
operations, and every person on the list has to maintain memory of size
<span class="math">\(O(1)\)</span>.</p>
<p>Joining the network also costs <span class="math">\(O(n)\)</span> operations. (That is because joining
the network requires a search).</p>
<h4>Improving search speed</h4>
<p>So far we managed to prove that we could live
in a world without contact lists. We just have to remember a few names and
phone numbers (In the simple solution above: only one name and one phone
number) to be able to call anyone eventually. Though “eventually” is usually
not enough. We don’t want to call half of the world to be able to contact one
person. It is not practical.</p>
<p>Just imagine this: Every time that someone in the world wants to call someone
else, there is a probability of <span class="math">\(\frac{1}{2}\)</span> that he will call you on the
way! Your phone will never stop ringing.</p>
<p>What if we could somehow arrange the phone list so that we will need to call
only a few people for every search? Maybe if we remember a bit more than one
people’s phone number, we could get a major improvement in search performance.</p>
<h5>Adding more immediate links</h5>
<p>A first idea for improving the phone list would be that each person will
remember more of his list neighbours phone numbers.
Instead of remembering just the next on the list, why not remember the two next
people on the list?</p>
<p>In this structure, every person has to remember <span class="math">\(2\)</span> names and phone numbers,
which is not so much more than the <span class="math">\(1\)</span> that we previously had. However, the
improvement in the search operation is major: A search operation will now cost
an average of <span class="math">\(\frac{n}{4}\)</span> operations, instead of <span class="math">\(\frac{n}{2}\)</span> that we
had previously. (Implicitly, it also improves the cost of joining the network).</p>
<p>We can add more and more records to remember for each of the people on the
phone list, to get further improvement in the speed of one search operation.
If each person on the list remembers <span class="math">\(k\)</span> neighbors forward
on the list, then the search operation will be <span class="math">\(k\)</span> times
faster. As <span class="math">\(k\)</span> can’t be so big (Generally we will assume that people on the
list can not remember more than <span class="math">\(O(\log(n))\)</span> stuff), we can only get so far
with this method.</p>
<p>Maybe if we choose to remember only specific people on the list in some special
way, we could get better results.</p>
<p><img alt="The list with \(k=2\). Search operation is twice as fast." src="https://www.freedomlayer.org/articles/freedomlayer/dht_intro/images/double_linear_dht_circle.svg"></p>
<p>The list with <span class="math">\(k=2\)</span>. Search operation is twice as fast.</p>
<h3>Chord</h3>
<p>So far we have discussed a very nice phone list game, and you might not
understand why care about it at all.
Let me formulate the question differently. Assume that we have a set of
<span class="math">\(n\)</span> computers, or nodes, connected to the Internet (The good old internet that
you know and use). Each computer has some kind of unique name. (The unique name
is not his Internet Address.) </p>
<p>We want to create a communication structure (Or an overlay network) that
satisfies the following requirements: </p>
<ol>
<li>Each computer will able to “contact” each of the other computers.</li>
<li>Every computer can remember the addresses of only about <span class="math">\(O(\log(n))\)</span>
other computers’ addresses.</li>
<li>Computers might join or leave the network from time to time. We would like
to be able to allow that while preserving the general structure of the network.</li>
</ol>
<p>Before dealing with solving this problem, I want to discuss some of the
requirements. Lets begin with the first requirement. What does it mean to be
able to “contact” other computers? Let me give you a simple use case. Lets
assume that every computer holds some chunk of information, some kind of a very
big table. Maybe this table is a distributed database. Maybe part of a file
sharing protocol. Maybe something else. We want to make sure that every
computer can reach any other computer, to obtain data for example.</p>
<p>Regarding the second requirement - Every computer can remember only a few
addresses. Why can’t every computer keep the addresses of all the other
computers? Well, there are a few practical reasons for that. First - There
might be a lot of computers. <span class="math">\(n\)</span> might be very large, and it might be heavy
for some computers to remember a list of <span class="math">\(n\)</span> addresses. In fact, it might be
more than remembering <span class="math">\(n\)</span> addresses. A <a href="http://en.wikipedia.org/wiki/Transmission_Control_Protocol"><span class="caps">TCP</span>
connection</a> between
two computers, for example, has to be maintained somehow. It takes effort to
maintain it.</p>
<p>But there is another reason. Probably a more major one. We want that this set of
computers will be able to change with time. Some computers might join, and
others might leave from time to time. If every computer is to remember all the
addresses of all the other computers, then every time a computer joins this
set, <span class="math">\(n\)</span> computers will have to be informed about it. That means joining the
network costs at least <span class="math">\(O(n)\)</span>, which is unacceptable.</p>
<p>If we want computers in this set to be able to bear the churn of computers
joining and leaving, we will have to build a structure where every computer
maintains links with only a small number of other computers.</p>
<h4>Adapting the phone ring solution</h4>
<p>As you have probably noticed, this problem is not very different from the phone
list problem. Just replace Computers with People, Computers’ unique identities
with the people’s unique names, and Computer’s Internet Addresses (IPs) with
People’s phone numbers. (Go ahead and do it, I’m waiting :) )</p>
<p>So the solution for the Computer’s case is as follows:
First we sort the node’s names somehow. (If the nodes’ unique names are numbers, we
just use the order of the natural numbers). Then we build a ring that contains
all the nodes, ordered by their name. (We just think about it as ring, we don’t
really order the nodes physically in a ring, just like we didn’t order the
people in a circle when we dealt with the phone list problem)</p>
<p>Every node will be linked to the next node on the ring.
Searching a node (By his unique name) will be done by iteratively asking the
next node for the name and address of the next next node, until the wanted node
is found.</p>
<p>Joining the network is as described in the phone list case. (Leaving the
network is a subject we will discuss in a later time.)</p>
<p>Here, just like in our description of the previous problem (The phone list), we
could also improve the speed of search if every node will keep more links to
direct neighbours. However, as we have seen before, we can only get so much
improvement in this method, and we would like to find a better idea for link
structures between the nodes.</p>
<h4>Improving the Search</h4>
<p>The following leap of thought could be achieved in more than one way. One way
to begin with it to think ituitively about how we manage to find things in the
real world.</p>
<h5>Intuition from real world searching</h5>
<p>Lets assume that you want to get to some place, and you are not sure where it
is. A good idea would be to ask someone how to get there. If you are very far
from your destination, most likely the person you asked will give you a very
vague description of how to get there. But it will get you starting in the
correct direction.</p>
<p>After you advance a while, you can ask somebody else. You will get another
description, this time more a detailed one. You will then follow this
description, until you get closer.</p>
<p>Finally when you are really close, you will find someone that knows exactly
where is that place you are looking for. Then your search will end.</p>
<p>This might lead us to think that maybe the network of links between nodes
should be arranged as follows:</p>
<ul>
<li>
<p>Every node <span class="math">\(X\)</span> is “linked” to nodes with names closest to his name. (His two
immediate neighbors on the ring, for example).</p>
</li>
<li>
<p>Every node <span class="math">\(X\)</span> is connected to other nodes from the ring: As the distance
<span class="math">\(X\)</span> becomes greater, <span class="math">\(X\)</span> is connected to less and less nodes.</p>
</li>
</ul>
<p>Generally: <span class="math">\(X\)</span> knows a lot about his close neighbourhood, however he knows
little about the parts of the rings that are far.</p>
<h5>Binary Search</h5>
<p>A different way to look at the search problem is from the angle of a more
common method: <a href="http://en.wikipedia.org/wiki/Binary_search_algorithm">Binary
search</a>. Given a sorted
array, we could find an element inside the array in <span class="math">\(O(log(n))\)</span> operations,
instead of the naive <span class="math">\(O(n)\)</span>.</p>
<p>How could we apply Binary Search to our case? In the binary search algorithm in
every iteration we cut the array to two halves, and then continue searching in
the relevant half. We can do that because we have <a href="http://en.wikipedia.org/wiki/Random_access">random
access</a> to the elements of the
array. That means - We could access any element that we want immediately. We
could access the middle element immediately.</p>
<p>In the simple ring setting (Every node is connected to the next and previous
nodes) we don’t have random access. However we could obtain something similar
to random access if we added the right links from every node. Take some time to
think about it. How would you wire the nodes to obtain the “random access ability”?</p>
<h5>Binary search Wiring</h5>
<p>To explain the next structure of links I want to discuss some notation stuff
first. We assume that the names of all the nodes are numbers that could be
represented using <span class="math">\(s\)</span> bits. In other words, the names of nodes are from the
set: <span class="math">\(B_s := \{0,1,2,\dots,2^{s}-1 \}\)</span>. The details here don’t really matter.
All that matters is that <span class="math">\(2^{s} \geq n\)</span>, so that there are enough possible
unique names for all the nodes in the network.</p>
<p>We also want to treat the set <span class="math">\(B_s\)</span> as cyclic
<a href="http://en.wikipedia.org/wiki/Modulo_operation">modulo</a> <span class="math">\(2^{s}\)</span>.</p>
<p>Let <span class="math">\(x\)</span> be some node on the ring. (<span class="math">\(x\)</span> is the name of this node. <span class="math">\(x \in
B_s\)</span>). We will connect <span class="math">\(x\)</span> to the following nodes on the ring:</p>
<ul>
<li><span class="math">\(\ceil{x + 1}\)</span></li>
<li><span class="math">\(\ceil{x + 2}\)</span></li>
<li><span class="math">\(\ceil{x + 4}\)</span></li>
</ul>
<p><span class="math">\(\cdots\)</span></p>
<ul>
<li><span class="math">\(\ceil{x + 2^{s-1}}\)</span></li>
</ul>
<p>The notation <span class="math">\(\left\lceil{y}\right\rceil\)</span> means the first node
that his name is bigger than <span class="math">\(y\)</span>.</p>
<p><img alt="log wiring" src="https://www.freedomlayer.org/articles/freedomlayer/dht_intro/images/log_wiring.svg"></p>
<p>In the picture: The ring represents the set <span class="math">\(B_s\)</span> of possible names for
nodes. (With <span class="math">\(s = 6\)</span>). Blue points are existing nodes. Their location on the ring represents
their name. Cuts on the ring represent the exact locations of <span class="math">\(x+1,
x+2,\dots,x+2^{s-1}\)</span>. The nodes of the form <span class="math">\(\left\lceil{x +
2^{q}}\right\rceil\)</span> are marked on the ring.
The green lines represents links from the node <span class="math">\(x\)</span> to other nodes.</p>
<p>Follow the picture and make sure you understand what <span class="math">\(\ceil{x +
2^{q}}\)</span> means - It is the “first” (clockwise) node with a name
bigger than the number <span class="math">\(x + 2^{q}\)</span> on the ring.</p>
<p>This idea of wiring is also known as a <a href="http://en.wikipedia.org/wiki/Skip_list">Skip
list</a>.</p>
<h5>New Search Algorithm</h5>
<p>Let’s describe the searching process with the new links structure.
Assume that node <span class="math">\(x\)</span> (<span class="math">\(x \in B_s\)</span> is the name of the node) wants to reach
node <span class="math">\(y\)</span>. Node <span class="math">\(x\)</span> will first check his own list of links, and see if he is
already connected directly to <span class="math">\(y\)</span>. If this is the case, <span class="math">\(x\)</span> can reach
<span class="math">\(y\)</span>.</p>
<p>But <span class="math">\(x\)</span> will not be that lucky every time. if <span class="math">\(y\)</span> is not in <span class="math">\(x\)</span><span class="quo">‘</span>s links
list, then <span class="math">\(x\)</span> will choose the “closest” option - a node <span class="math">\(x_1\)</span> that is the
closest <span class="math">\(x\)</span> knows to <span class="math">\(y\)</span>. By “closest” we mean the closest when walking
clockwise. (As an example, the node just before <span class="math">\(x\)</span> on the ring is the farest
node from <span class="math">\(x\)</span>).</p>
<p><span class="math">\(x\)</span> will ask <span class="math">\(x_1\)</span> if he knows <span class="math">\(y\)</span>, and if he doesn’t, <span class="math">\(x\)</span> will ask
<span class="math">\(x_1\)</span> what is the closest node to <span class="math">\(y\)</span> known to <span class="math">\(x_1\)</span>? Let that node be
<span class="math">\(x_2\)</span>.</p>
<p><span class="math">\(x\)</span> will keep going, until he eventually finds <span class="math">\(y\)</span>. We should analyze this
algorithm to make sure that indeed <span class="math">\(x\)</span> eventually finds <span class="math">\(y\)</span>, and also how
many iterations it takes to find <span class="math">\(y\)</span>.</p>
<p><img alt="x searches y" src="https://www.freedomlayer.org/articles/freedomlayer/dht_intro/images/x_search_y.svg"></p>
<p>Illustrated search process</p>
<h5>Analysis</h5>
<p>Let us start with the simple things. How many links every node has to maintain?
By the definitions of links earlier, we know that not more than <span class="math">\(s\)</span> links. We
said that the size of the set <span class="math">\(B_s\)</span> must be more than <span class="math">\(n\)</span>, therefore
<span class="math">\(2^{s} \geq n\)</span>, which means <span class="math">\(s \geq \log(n)\)</span>. Therefore every node
maintains about <span class="math">\(\log(n)\)</span> links. This is generally a reasonable number, even
for very large <span class="math">\(n\)</span>-s.</p>
<p>Next, we want to know how long does it take for a node <span class="math">\(x\)</span> to find some
random node <span class="math">\(y\)</span>. In fact, we want to be sure that <span class="math">\(x\)</span> always manages to
find <span class="math">\(y\)</span> eventually.</p>
<p>If you are not in a mood for some math symbols, I give here a short description
of what is going to happen. We are soon going to find out that in every
stage of the search algorithm we get twice as close to <span class="math">\(y\)</span>. As the size of
the set <span class="math">\(B_s\)</span> is <span class="math">\(2^{s}\)</span>, we are going to have no more than <span class="math">\(s\)</span> stages
before we find <span class="math">\(y\)</span>. This also proves that we always manage to find <span class="math">\(y\)</span>.</p>
<p>Now let’s do some math.
We define the distance (going clockwise) between two nodes <span class="math">\(a\)</span> and <span class="math">\(b\)</span> to
be <span class="math">\(d(a,b)\)</span>. If <span class="math">\(b > a\)</span> then <span class="math">\(d(a,b) = b-a\)</span>. Otherwise <span class="math">\(d(a,b) = 2^{s} +
b - a\)</span>. (Think why).</p>
<p>Back to the searching algorithm, we can note that at every stage we are at
point <span class="math">\(x_t\)</span> on the ring, and we want to reach <span class="math">\(y\)</span>. We will pay attention to
the amount <span class="math">\(d(x_t,y)\)</span> at any stage of the algorithm.</p>
<p>We begin from <span class="math">\(x\)</span>. If <span class="math">\(x\)</span> is not directly connected to <span class="math">\(y\)</span>, then <span class="math">\(x\)</span>
finds the closest direct link he has to <span class="math">\(y\)</span>. Let that node be <span class="math">\(x_1\)</span>. As
<span class="math">\(x\)</span> is linked to <span class="math">\(\ceil{x + 1}, \ceil{x +
2}, \ceil{x + 4} \dots ,\ceil{x + 2^{s-1}}\)</span>, we conclude that
<span class="math">\(d(x_1,y) < \frac{1}{2}\cdot d(x,y)\)</span>. </p>
<p>Let me explain it in a more detailed fashion:
Assume that <span class="math">\(y = x +q\)</span> for some <span class="math">\(q\)</span> (The addition of <span class="math">\(x + q\)</span> might be
modulo the set <span class="math">\(B_s\)</span>). There is some integer number <span class="math">\(r\)</span> such that <span class="math">\(2^{r}
\leq q < 2^{r+1}\)</span>. (You could understand it by counting the amount of bits in the
<a href="http://en.wikipedia.org/wiki/Binary_number">binary representation</a> of <span class="math">\(q\)</span>
for example). Therefore the closest link from <span class="math">\(x\)</span> to <span class="math">\(y\)</span> would be
<span class="math">\(\ceil{x + 2^{r}} = x_1\)</span>.
And indeed, we get that <span class="math">\(d(x_1,y) = d(x_1,x+q) \leq d(x+2^r,x+q) \leq q - 2^r
< \frac{q}{2} = \frac{d(x,y)}{2}\)</span>. So we get that <span class="math">\(d(x_1,y) < \frac{d(x,y)}{2}\)</span>.</p>
<p>The same is true at the next stages of the algorithm (When finding
<span class="math">\(x_2,x_3,\dots\)</span>, therefore we conclude that on every stage we get
twice closer to <span class="math">\(y\)</span>, compared to the previous stage. Finally we get that
<span class="math">\(d(x_q,y) < \frac{1}{2}\cdot d(x_{q-1},y) < \frac{1}{4}\cdot d(x_{q-2},y) <
\dots < \frac{1}{2^{q}}\cdot d(x,y)\)</span></p>
<p>We know that the initial distance <span class="math">\(d(x,y)\)</span> is no more than <span class="math">\(2^{s}\)</span>,
therefore in at most <span class="math">\(s\)</span> stages we will reach distance <span class="math">\(0\)</span>, which means we
have found <span class="math">\(y\)</span>.</p>
<p>If you are a careful reader, you might be worried at this point that <span class="math">\(s\)</span>
might be much more than <span class="math">\(\log(n)\)</span>. This is in fact true. It is also true that
in some worst case scenarios the amount of stages for the search algorithm will
actually be <span class="math">\(s\)</span>, even if <span class="math">\(log(n)\)</span> is much smaller.</p>
<p>However if the names of the nodes are chosen somehow uniformly from the set
<span class="math">\(B_s\)</span>, we should expect better results which are much closer to <span class="math">\(log(n)\)</span>.</p>
<h4>Some words about Chord</h4>
<p>Congratulations, you now know how to wire a collection of <span class="math">\(n\)</span> nodes so that
they can contact each other quickly, and at the same time each node doesn’t have to
remember too many addresses of other nodes.</p>
<p>The construct we have described is related to an idea called <a href="http://en.wikipedia.org/wiki/Chord_(peer-to-peer)">The Chord
<span class="caps">DHT</span></a>. You are encouraged to
check the original article: <a href="http://pdos.csail.mit.edu/papers/chord:sigcomm01/chord_sigcomm.pdf">Chord: A scalable Peer-to-peer Lookup Service for
internet
Applications</a>.</p>
<h3>Distributed Hash Tables (DHTs)</h3>
<p>Lets discuss an important use case for the structure we have found so far.
We want to be able to store a large table of keys and values over a large set
of computers. This is usually called a <a href="http://en.wikipedia.org/wiki/Distributed_hash_table">Distributed Hash Table
(<span class="caps">DHT</span>)</a>.</p>
<p>The main operations that we want to be able to perform are as follows:</p>
<ul>
<li>
<p>set_value(key,value) - Sets the value of “key” in the table to be “value”.</p>
</li>
<li>
<p>get_value(key) - Reads the value of “key” from the table.</p>
</li>
</ul>
<p>The cool part is that we can invoke those operations from any of the computers,
as all the computers have a symmetric role in the network. Instead of letting
just one computer deal with requests from client, theoretically we could use
all the computers on the network. (Though we might have to deal with some
synchronization stuff, which are outside the scope of this document).</p>
<p>There are still some questions to be asked here. What kind of values can the
keys be? Must they be numbers, or could they be something else? Maybe strings?</p>
<p>Lets begin with the case in which keys are also from the set <span class="math">\(B_s\)</span>. This is
not always very realistic, but it would be easier to solve at this point. In that
case, the keys are in the same “space” as the names of nodes.</p>
<p>We could let node <span class="math">\(\floor{k}\)</span> keep the value of key <span class="math">\(k\)</span>,
where <span class="math">\(\floor{k}\)</span> is the “last” node (clockwise) that has
a name not bigger than the number <span class="math">\(k\)</span>.</p>
<p><img alt="responsible keys" src="https://www.freedomlayer.org/articles/freedomlayer/dht_intro/images/responsible_keys.svg"></p>
<p>In the picture: The node <span class="math">\(z\)</span> (A blue dot), and some keys that <span class="math">\(z\)</span> is
responsible to keep (Small orange dots). The keys and node names are of the
same kind (Both are from <span class="math">\(B_s\)</span>, so we can also draw them on the ring
according to their value. The next node (clockwise) after <span class="math">\(z\)</span> marks the end
of the domain <span class="math">\(z\)</span> has responsibility over.</p>
<p>To invoke set_value(key=k,value=v), we first search (Using our search
algorithm) for the node that is responsible for keeping the key <span class="math">\(k\)</span>. This is
done by searching for the value <span class="math">\(k\)</span>. We are going to find the node <span class="math">\(z =
\floor{k}\)</span>, which is exactly the node that has the
responsibility to keep the key <span class="math">\(k\)</span>. Then we just ask the node <span class="math">\(z\)</span> to update
<span class="math">\(k\)</span> to have the value <span class="math">\(v\)</span>.</p>
<p>To invoke get_value(key=k), again we search for <span class="math">\(k\)</span>, and find the node <span class="math">\(z =
\floor{k}\)</span>. We then ask <span class="math">\(z\)</span> what is the value that
corresponds to the key <span class="math">\(k\)</span>. <span class="math">\(z\)</span> will then tell us the value <span class="math">\(v\)</span>.</p>
<h4>Dealing with complex keys</h4>
<p>But what if our keys are not from the set <span class="math">\(B_s\)</span>? Maybe the keys are strings?
Maybe they are names of files, or people? In that case all we need is some
function <span class="math">\(f: K\rightarrow B_s\)</span>, where <span class="math">\(K\)</span> is the world of keys. Hopefully
the function <span class="math">\(f\)</span> will also be some kind of a random function, which means
a few things:</p>
<ul>
<li>
<p>It is very unlikely for two keys <span class="math">\(k_1,k_2\)</span> to satisfy <span class="math">\(f(k_1) = f(k_2)\)</span>.
(A property also known as <a href="http://en.wikipedia.org/wiki/Collision_resistance">Collision Resistance</a>).</p>
</li>
<li>
<p>The keys will map evenly as possible between all the elements inside the set
<span class="math">\(B_s\)</span>. We don’t want to have too much load of a few of the computers.</p>
</li>
</ul>
<p>If you were wondering where you can get such a function, don’t worry. We have a
few of those functions. They are called <a href="http://en.wikipedia.org/wiki/Cryptographic_hash_function">Cryptographic Hash
Functions</a>.</p>
<p>Now that we have the function <span class="math">\(f\)</span>, we will define two operations:</p>
<ul>
<li>
<p>set_key_generic(key=k,value=v) will invoke
set_key(key=<span class="math">\(f(k)\)</span>,value=v).</p>
</li>
<li>
<p>get_key_generic(key=k) will invoke get_key(key=<span class="math">\(f(k)\)</span>)</p>
</li>
</ul>
<p>And we get a <span class="caps">DHT</span> for a generic key space.</p>
<h3>Final Notes</h3>
<p>We have introduced a special way to wire a set of computers, so that we don’t
use too many wires, and at the same time it is easy to find any computer
quickly. A major use case of this construct is the idea of <span class="caps">DHT</span>.</p>
<p>Our main construction follows the idea of the <a href="http://en.wikipedia.org/wiki/Chord_(peer-to-peer)">The Chord
<span class="caps">DHT</span></a>, however there are
other possible designs for <span class="caps">DHT</span> which we haven’t talked about. Our space of
names was a ring, with a distance function of walking clockwise. There are
other spaces with different distance functions that give nice results. One
notable example is the <a href="http://en.wikipedia.org/wiki/Kademlia">Kademlia <span class="caps">DHT</span></a>,
which uses <span class="caps">XOR</span> as a metric. You can (And probably should) read the original
Kademlia article: <a href="http://pdos.csail.mit.edu/~petar/papers/maymounkov-kademlia-lncs.pdf">Kademlia: A peer-to-peer Information System based on the <span class="caps">XOR</span>
metric</a>,
by Petar Maymounkov and David Mazières.</p>
<p>We discussed the problem generally, but we didn’t address a few important
issues. We didn’t address stability issues (What happens if some node on the
way goes offline just when we want to search for some key?) and security
issues. (What happens if a node gives us a wrong value for the key? Could an
adversary block users from getting the value of a specific key in the <span class="caps">DHT</span>?)</p>
<p>We will think about those topics and how to deal with some of them in the next articles.</p>
<script type="text/javascript">if (!document.getElementById('mathjaxscript_pelican_#%@#$@#')) {
var align = "center",
indent = "0em",
linebreak = "false";
if (false) {
align = (screen.width < 768) ? "left" : align;
indent = (screen.width < 768) ? "0em" : indent;
linebreak = (screen.width < 768) ? 'true' : linebreak;
}
var mathjaxscript = document.createElement('script');
mathjaxscript.id = 'mathjaxscript_pelican_#%@#$@#';
mathjaxscript.type = 'text/javascript';
mathjaxscript.src = 'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.0/MathJax.js?config=TeX-AMS-MML_HTMLorMML';
mathjaxscript[(window.opera ? "innerHTML" : "text")] =
"MathJax.Hub.Config({" +
" config: ['MMLorHTML.js']," +
" TeX: { extensions: ['AMSmath.js','AMSsymbols.js','noErrors.js','noUndefined.js'], equationNumbers: { autoNumber: 'AMS' } }," +
" jax: ['input/TeX','input/MathML','output/HTML-CSS']," +
" extensions: ['tex2jax.js','mml2jax.js','MathMenu.js','MathZoom.js']," +
" displayAlign: '"+ align +"'," +
" displayIndent: '"+ indent +"'," +
" showMathMenu: true," +
" messageStyle: 'normal'," +
" tex2jax: { " +
" inlineMath: [ ['\\\\(','\\\\)'] ], " +
" displayMath: [ ['$$','$$'] ]," +
" processEscapes: true," +
" preview: 'TeX'," +
" }, " +
" 'HTML-CSS': { " +
" styles: { '.MathJax_Display, .MathJax .mo, .MathJax .mi, .MathJax .mn': {color: 'inherit ! important'} }," +
" linebreaks: { automatic: "+ linebreak +", width: '90% container' }," +
" }, " +
"}); " +
"if ('default' !== 'default') {" +
"MathJax.Hub.Register.StartupHook('HTML-CSS Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax['HTML-CSS'].FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"MathJax.Hub.Register.StartupHook('SVG Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax.SVG.FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"}";
(document.body || document.getElementsByTagName('head')[0]).appendChild(mathjaxscript);
}
</script>The Mesh Question2014-11-09T18:31:00+00:002014-11-09T18:31:00+00:00realtag:www.freedomlayer.org,2014-11-09:/the-mesh-question.html<h3>Abstract</h3>
<p>We discuss motivation for creating a distributed network instead or on top of
the internet. Then we write down a list of specific requirements from this
network. Finally we introduce the Flooding idea as a naive solution for this
problem, and explain why it won’t do.</p>
<h3>Motivation for …</h3><h3>Abstract</h3>
<p>We discuss motivation for creating a distributed network instead or on top of
the internet. Then we write down a list of specific requirements from this
network. Finally we introduce the Flooding idea as a naive solution for this
problem, and explain why it won’t do.</p>
<h3>Motivation for change</h3>
<p>In <a href="https://www.freedomlayer.org/intro-to-the-internet-and-current-issues.html">Intro to the Internet</a>
we generally discussed how the internet works, and some of the drawbacks of the
current situation. I remind you the main ones: </p>
<ol>
<li>
<p>Addressing - A usual participant in the internet usually has no control over his
address. It is assigned to him. In most cases it also changes from time to time. </p>
</li>
<li>
<p>NATs - Most of the computers in the internet these days are behind a <span class="caps">NAT</span>.
If two computers are each behind a <span class="caps">NAT</span>, they might experience great
difficulty in the task of initiating conversation. (Though some ad hoc
methods exist today today. See <a href="http://en.wikipedia.org/wiki/STUN">Stun</a> as
an example.)</p>
</li>
<li>
<p>Operators Greed - The Internet is hierarchical, which gives much power to
its operators. They might be tempted to find new and interesting ways to
get more money from the end users of the Internet.</p>
</li>
<li>
<p>Privacy and Censorship: The hierarchical structure of the internet causes a
situation where much traffic is routed through few computers. The owners of
those computers can leverage their advantage to read end users traffic or
censor specific content.</p>
</li>
</ol>
<h3>The search for alternatives</h3>
<p>While some work on solutions inside the Internet itself to solve some of the
issues presented above, it might be interesting to discuss an alternative
structure of connecting computers together.</p>
<p>The main idea proposed in this project is the creation of a new network with a
different structure, that will serve the same purposes as the original Internet.</p>
<p>As many issues that we mentioned before relate to the hierarchial structure of
the Internet, we will avoid it. We will try to find some other more
decentralized structure, where all the computers have more or less a symmetric
role in the network. This kind of network is sometimes referred to as a <a href="http://en.wikipedia.org/wiki/Mesh_networking">mesh
network</a>.</p>
<p>We will also want to utilize the current infrastructure of the Internet, as it
is already there, and it works. One idea would be to start a new network on the
Internet itself, and slowly take it out.</p>
<p>Next, we care about security. The Internet was built in the <a href="http://en.wikipedia.org/wiki/Internet#History">sixties and
seventies</a>, and its creators
didn’t expect it to be deployed on all the computers in the world. They didn’t
design it with security in mind - They just wanted it to work.</p>
<p>Finally, we want the new network to be efficient, or maybe - scalable. We want
it to be able to connect all the computers in the world, and still have good
performance. Performance that is comparable with today’s Internet.</p>
<h3>Isolating the question</h3>
<p>We have a <span class="math">\(n\)</span> computers scattered around the world. (Or maybe in some other
places, it you like). <span class="math">\(n\)</span> is a large number. You may assume <span class="math">\(n =
2^{40}\)</span>. We want to wire them somehow and choose a program that
all of them run (The same program for all of them!), so that we get the
following properties:</p>
<ul>
<li>
<p><strong>Addressing</strong>: We can give names or addresses to different computers on
the network. Then those names could be used to initiate communications with
remote computers on the network.</p>
</li>
<li>
<p><strong>Routability</strong>: We are able to send messages to remote computers on the network,
given their address. Those messages arrive with some constant high enough
probability, in amount of time poly logarithmic in <span class="math">\(n\)</span>.
(<span class="math">\(O(P(log n))\)</span> where <span class="math">\(P\)</span> is some non evil Polynomial).</p>
</li>
<li>
<p><strong>Symmetry</strong>: All the computers in the network have the same role, or at least
the potential to be in the same role. All the computers in the network run
the same program.</p>
</li>
<li>
<p><strong>Scalability</strong>: Sending a message in the network disturbs no more than <span class="math">\(P(log
n)\)</span> computers. Basically it means that routing a message in the network can
not involve too many computers in the network.</p>
</li>
<li>
<p><strong>Network Justice</strong>: A computer in the network that sends lots of data will not
be able to disturb other computers in the network to send and receive
messages. (Of course we will have to define somehow what is lots of data,
though this is just a general idea of a property that we want).</p>
</li>
<li>
<p><strong>Security</strong>: Given an adversary that has limited computing power and
bandwidth, the network will keep working and all the participants will be
able to continue their life in the network normally. (This is a bit vague,
but we will discuss it in the future. It’s hard to discuss security without
having a system).</p>
</li>
</ul>
<p>There are some other things that will be nice to have. We would like to be able
to deal with <a href="http://en.wikipedia.org/wiki/Denial-of-service_attack">Denial Of Service
(DoS)</a> in some way.</p>
<p>This description is far from perfect (It is not even close to being formal),
though it is a start, and we can work with it at this point.
As a summary, if somebody on the street asks you what I
search for, it would be a <strong>secure distributed and scalable mesh network</strong>.</p>
<h3>First Naive solution</h3>
<p>Like every good researcher, we begin by trying a few simple solutions to the problem.</p>
<p>In all the following solutions we will begin by wiring every computer to a few
computers close to him (Maybe geographically). We call those few computers
close to him immediate <strong>neighbours</strong>, or just neighbours. Regarding the
participants of the network - As they might be pretty general entities, we
sometimes call them <strong>nodes</strong> instead of computers.</p>
<p>We will usually wonder how to send a message from some random node <span class="math">\(A\)</span> to
another random node <span class="math">\(B\)</span> on the network, given that <span class="math">\(A\)</span> knows the address of
<span class="math">\(B\)</span>. (How <span class="math">\(A\)</span> knows <span class="math">\(B\)</span><span class="quo">‘</span>s address from the first place is another
question that we are going to deal with later, don’t worry about it now).</p>
<p><img alt="a_to_b" src="https://www.freedomlayer.org/articles/freedomlayer/mesh_question/images/a_to_b.svg"></p>
<p>A generic schema of a mesh network. We want to send a message from <span class="math">\(A\)</span> to
<span class="math">\(B\)</span>.</p>
<h4>Flooding</h4>
<p>We program every node (Computer) in the network to forward every message it
gets to all of his neighbors.</p>
<p>To avoid loops, every node will remember the messages that it sees somehow,
and if he gets again a message that he has already seen, he will not forward it again.</p>
<p>Let <span class="math">\(A,B\)</span> be two nodes in the network. We want to send a message from
<span class="math">\(A\)</span> to <span class="math">\(B\)</span>. To do that, <span class="math">\(A\)</span> will send the message to all of his
neighbours. In turn, as we programmed all those nodes, every nodes that
gets the message will forward it to all of his neighbours, and so on.
Eventually the message will arrive at <span class="math">\(B\)</span>.</p>
<p>This solution will not be enough for us, because it is not scalable. Note that
sending a message from <span class="math">\(A\)</span> to <span class="math">\(B\)</span> involves all the nodes in the
network. This means that as a node sitting in this network, you will have
to process every message that every node sends! This is too much for large
amount of network participants.</p>
<p>Flooding is one of the oldest tricks in the book. Although it is considered to
be not very scalable, it is a very common solution. (See for example
<a href="https://bitcoin.org/">bitcoin</a> or
<a href="http://en.wikipedia.org/wiki/Gnutella">Gnutella</a>)</p>
<p>The Flooding solution has some features that are hard to achieve in other methods:</p>
<ul>
<li>
<p>Flooding is very simple to design and program. No special rules, just forward
every message to all of your neighbors.</p>
</li>
<li>
<p>Flooding manages to route a message to its destination with a very high
probability. Also, almost any addressing scheme will work with flooding, as
every message arrives at all the nodes in the network.</p>
</li>
<li>
<p>Flooding has important security features. For example: It is very hard for an
adversary to block a node from recieving a message destined to him. (Bitcoin,
for example, relies on this feature for its security).</p>
</li>
</ul>
<h3>Summary</h3>
<p>We made some progress in defining the kind of network we want, and introduced
the Flooding solution. We also explained why it will not be suitable as a
solution to our problem.</p>
<script type="text/javascript">if (!document.getElementById('mathjaxscript_pelican_#%@#$@#')) {
var align = "center",
indent = "0em",
linebreak = "false";
if (false) {
align = (screen.width < 768) ? "left" : align;
indent = (screen.width < 768) ? "0em" : indent;
linebreak = (screen.width < 768) ? 'true' : linebreak;
}
var mathjaxscript = document.createElement('script');
mathjaxscript.id = 'mathjaxscript_pelican_#%@#$@#';
mathjaxscript.type = 'text/javascript';
mathjaxscript.src = 'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.0/MathJax.js?config=TeX-AMS-MML_HTMLorMML';
mathjaxscript[(window.opera ? "innerHTML" : "text")] =
"MathJax.Hub.Config({" +
" config: ['MMLorHTML.js']," +
" TeX: { extensions: ['AMSmath.js','AMSsymbols.js','noErrors.js','noUndefined.js'], equationNumbers: { autoNumber: 'AMS' } }," +
" jax: ['input/TeX','input/MathML','output/HTML-CSS']," +
" extensions: ['tex2jax.js','mml2jax.js','MathMenu.js','MathZoom.js']," +
" displayAlign: '"+ align +"'," +
" displayIndent: '"+ indent +"'," +
" showMathMenu: true," +
" messageStyle: 'normal'," +
" tex2jax: { " +
" inlineMath: [ ['\\\\(','\\\\)'] ], " +
" displayMath: [ ['$$','$$'] ]," +
" processEscapes: true," +
" preview: 'TeX'," +
" }, " +
" 'HTML-CSS': { " +
" styles: { '.MathJax_Display, .MathJax .mo, .MathJax .mi, .MathJax .mn': {color: 'inherit ! important'} }," +
" linebreaks: { automatic: "+ linebreak +", width: '90% container' }," +
" }, " +
"}); " +
"if ('default' !== 'default') {" +
"MathJax.Hub.Register.StartupHook('HTML-CSS Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax['HTML-CSS'].FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"MathJax.Hub.Register.StartupHook('SVG Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax.SVG.FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"}";
(document.body || document.getElementsByTagName('head')[0]).appendChild(mathjaxscript);
}
</script>Intro to the Internet and current issues2014-11-09T10:34:00+00:002014-11-09T10:34:00+00:00realtag:www.freedomlayer.org,2014-11-09:/intro-to-the-internet-and-current-issues.html<h3>Abstract</h3>
<p>We discuss the general idea of communication between computers, the structure
of the internet and some issues with the current structure.</p>
<h3>Information delivery between computers</h3>
<p>Assume that there many computers, and we want them to be able to communicate
somehow with each other. Maybe from a human point of …</p><h3>Abstract</h3>
<p>We discuss the general idea of communication between computers, the structure
of the internet and some issues with the current structure.</p>
<h3>Information delivery between computers</h3>
<p>Assume that there many computers, and we want them to be able to communicate
somehow with each other. Maybe from a human point of view, there are many human
beings in different locations, and they all have computers. Then those human
beings want to communicate somehow, effectively, using those computers.</p>
<p>The simple thing to do would be to connect those computers to each other, using
wires or some kind of wireless setting. Though it seems like whatever way we
pick, To connect every two computers we have to do some kind of work. If they
are very far geographically, it would be harder. We might need a very long
wire, or a way to create a very strong signal. </p>
<p>We could also use some kind of an intermediate entity to transfer information
between two computers. It could be a satellite, or a big tower retransmits
information to long distances. However if do that, somebody has manage the
creation of those entities, and everyone else will have to pay to maintain it,
somehow. This setting can work, however there are a few reasons for which it
might not be the solution by its own.</p>
<h3>The Internet</h3>
<p>The most famous solution to the question posed above is called the Internet. It
is a structure that connects many computers together, so that information could
be transferred from one computer to another, in a pretty simple fashion.
(A structure that connect many computers together is also called a network.)</p>
<p>For many, the idea of the Internet is some kind of mysterious thing. They think
about green numbers flying in the air, the cyberspace and other bombastic
words. Basically it is just a method for wiring many computers together, so
that routing information from a source to the destination is easy.</p>
<p>The internet is heirarchial. There are a few central computers that have the
sole job of routing information around. We call those routers. Then there are
some lower level routers that are connected to those routers. Finally the end
user of the internet connects to one of those low level routers. Those low
level routers in the end are sometimes called ISPs, or internet service providers.</p>
<p><img alt="A very simplistic view of the internet" src="https://www.freedomlayer.org/articles/freedomlayer/intro_internet/images/internet_scheme.svg"></p>
<p>A Very simplistic view of the internet</p>
<p>Every computer on the network gets an address. In the Internet terms we call it
<span class="caps">IP</span> (Internet Protocol) address. This address somehow describes the location of
the computer in the Internet heirarchy of computers. Computers can’t really
choose their address. Their addressed is assigned by a computer from the higher heirarchy.</p>
<p>Whenever we want to send information from one computer to another, we first
have to know the address of the destination computer. Then we build a message
(It contains the destination address and some other information), and then we
pass it to the closest router. In turn, this router passes the message on to
the next router, and so on, until the destination is found.</p>
<p>How can routers know how to forward the message given only the address of the
destination? They use the heirarchial structure of the Internet. The address is
a description of where the destination is, inside this structure.</p>
<p>It is not very different than any other message delivery mechanism that the
human race has invented. If you think about it for a moment, the post office
(The offline post office) is not much different.</p>
<p>How can the post office send your letter to a given destination? If the
destination is in the same city, the local post office could do that. However
if the destination is in a different city, the letter will be passed to a
higher level post office. Finally if the destination is somewhere in another
country, the letter will be transferred to higher level post office, which will
then send the message to another high level post office on the destination
country. The phone system hierarchy is also not much different.</p>
<p>That said, there is much more to learn about how the internet works. There are
many tactical algorithms for making routing more efficient. (but this is
probably also true for post offices and phone companies), and also many more
protocols built upon the basic message delivery to make it more reliable.
(packages could be lost on the way, and we might need to deliver them again etc.)</p>
<p>My main claim here though, is that the internet method of delivering messages
from a source to destination is dependant on the hierarchical structure of the
internet. There is no magic.</p>
<h3>Obstacles with the current method</h3>
<p>I wouldn’t bother telling you all that about the Internet without a point. I
want to discuss a few drawbacks of the currently used hierarchical method for
deliving information:</p>
<h4>1. The Addressing problem and <span class="caps">NAT</span></h4>
<h5>More about Internet Addressing</h5>
<p>As we mentioned above, every computer in the Internet is assigned a unique
address. This address somehow marks its location inside the hierarchical.</p>
<p>To be more specific, in the Internet every computer is assigned a number of
size 32 bits, or 4 bytes, according to the IPv4 protocol. This means that there
are about <span class="math">\(2^{32} = 4294967296\)</span> possible addresses for computers in the internet.</p>
<p>When the end user (It might be you) wants to join the internet, he first pays
for an account at some <span class="caps">ISP</span> (Internet Service Provider). You can think about <span class="caps">ISP</span>
as some low level router on the internet. Your computer is then directly
connected to the <span class="caps">ISP</span>. In fact, the <span class="caps">ISP</span> will be the only entity that your
computer is directly connected to.</p>
<p>The <span class="caps">ISP</span> is responsible for assigning an address to your computer, and also to
handle the routing of information from or to you computer. The <span class="caps">ISP</span> can’t just
pick random addresses for end users, as there might be a collision with
another address on the internet. (Addresses should be unique). Therefore
there is a higher level division of the addresses between ISPs.</p>
<p>The <span class="caps">IANA</span> (Internet Assigned Numbers Authority) orginization is the top level
entity responsible for assigning addresses. Basically speaking, ISPs buy ranges
of addresses from <span class="caps">IANA</span>.</p>
<h5>Shortage of Addresses</h5>
<p><span class="math">\(2^{32}\)</span> addresses is a pretty large amount, though if you consider it again
for a moment, you might find out that we could finish it pretty quickly.
To begin with, there are more people in the world than <span class="math">\(2^{32}\)</span>. In addition,
we probably have more computers than people. It’s not only the desktop
computers that need to have an address. It’s also all the laptops, the
cellphones, the stoplights on the streets, and very soon your refrigirator.</p>
<p>One solution that was proposed was increasing the address space. It was
proposed in the IPv6 protocol (Together with other adjustments that we won’t
discuss here). In the IPv6 protocol, every address is of size 128 bits (16
bytes). Just so that you get a prespective, <span class="math">\(2^{128} =
340282366920938463463374607431768211456\)</span>. It’s really a lot.</p>
<p>As IPv4 and IPv6 are not compatible, (and because of other reasons which are
probably related to human nature,) it is a difficult task to transition all the
computers connected to the internet to IPv6. </p>
<h5>The <span class="caps">NAT</span> idea</h5>
<p>Meanwhile, other temporary solution was introduced. It is called <span class="caps">NAT</span> (Network
Address Translation). The basic idea behind <span class="caps">NAT</span> is to overcome the shortage of
IPv4 addresses by hiding many computers behind one address.</p>
<p>Imagine for example that your local post office doesn’t want to give you a new
address, because there is a shortage of addresses in the world. If you really
want to get mail, you could just supply the address of your neighbour Joe, and ask
him to give you your mail. If other people on your block do the same and ask
Joe to recieve their mails, Joe becomes a <span class="caps">NAT</span>. (A human <span class="caps">NAT</span>).</p>
<p>Going back to the digital world, a <span class="caps">NAT</span> looks like a small box. You connect all
your computers to the <span class="caps">NAT</span>, and then you connect your <span class="caps">NAT</span> to the <span class="caps">ISP</span>. We say
that the computers in the internal part of the <span class="caps">NAT</span> are “behind the <span class="caps">NAT</span>”.</p>
<p><img alt="A picture of a Cisco router. It probably has the NAT abilities." src="https://www.freedomlayer.org/articles/freedomlayer/intro_internet/images/cisco_router.png"></p>
<p>A picture of a Cisco router. It probably has the <span class="caps">NAT</span> abilities.</p>
<p>To the outside internet, the <span class="caps">NAT</span> feels like one computer. It has one Internet
Address and it can receive or send messages.</p>
<p>Behind the <span class="caps">NAT</span>, every computer is assigned an internal address by the <span class="caps">NAT</span>.
Every computer from the internal part of the <span class="caps">NAT</span> can send a message to the
outside internet. Receiving messages is more difficult (Remember that this was
the original problem).</p>
<p>To recieve a message, a computer behind the <span class="caps">NAT</span> should arrange it with the <span class="caps">NAT</span>
first. (In real world words, the system administrator has to enter the
administration interface of the <span class="caps">NAT</span> and “forward a port”). Then whenever the
<span class="caps">NAT</span> receives a message, it checks if it is destined to one of the computers
behind the <span class="caps">NAT</span>. If it is, the <span class="caps">NAT</span> can forward the message to the apropriate computer.</p>
<p><img alt="A general sketch of the network behind a NAT, and the external Internet" src="https://www.freedomlayer.org/articles/freedomlayer/intro_internet/images/nat_scheme.svg"></p>
<p>A general sketch of the network behind a <span class="caps">NAT</span>, and the external Internet</p>
<p>NATs have become very common these days, mostly because they quick solve the
address shortage problem without having to change the Internet itself. The <span class="caps">NAT</span>
actually fools the external network to think that it is one computer, and it
fools the computers behind the <span class="caps">NAT</span> to think that they are connected to the
real internet.</p>
<p>NATs are so common these days, that most likely you are behind a <span class="caps">NAT</span> right now.
Maybe behind even more than one <span class="caps">NAT</span>, so far from the “real” Internet.</p>
<h5>Addressing issues</h5>
<p>Why should you care about NATs? It seems that everything works pretty smoothly.
Your web browser works just fine, and all the websites load quickly. As a more
advanced user you can create a website and upload it to some hosting service. </p>
<p>But let me tell you about the things that you can’t do so easily. </p>
<h6>Problems with Self hosting</h6>
<p>As a simple user, most likely you can’t host a website on your own computer if
you are behind a <span class="caps">NAT</span>. If you own the <span class="caps">NAT</span> and you know some stuff, you could
configure it so that outside computers will be able to connect to your
computer, behind the <span class="caps">NAT</span>. But if you don’t own that <span class="caps">NAT</span>, or you don’t know much
about NATs, your computer is just a guest in the internet. Outside computers
can not connect it, it can only connect to remote hosts.</p>
<p>Just as an example, most likely you can’t host a website on your cellphone.
That is because your cellphone is behind a <span class="caps">NAT</span>. The <span class="caps">NAT</span> belongs to the phone
company, and you have no control over it. You can connect to external computers
in the internet, however external computers can not connect to your cellphone.</p>
<p>Some of those issues were present even before the <span class="caps">NAT</span>. If you want to host a
website on your own computer, you will need some kind of permanent Address, so
that surferes can get to your website. You will need a permanent <span class="caps">IP</span> Address. If
you are connected to the Internet as a usual end user, most likely your <span class="caps">ISP</span>
assigns you a random <span class="caps">IP</span> address from the range of addresses it owns. As this
address changes every once in a while, you can not rely on it.</p>
<p>To have real independence you will have to buy an <span class="caps">IP</span> address of your own and
some direct connection to the internet.</p>
<h6>Issues with finding your own address</h6>
<p>If your computer is behind a <span class="caps">NAT</span>, he can not know is own address. If you try to
check your own address it is very likely that it will be something like
192.168.x.y, or 10.0.0.x. This is an internal address, assigned to you by your
<span class="caps">NAT</span>. </p>
<p>You can check it on your own computer.
On a windows box, enter at the command line:</p>
<div class="highlight"><pre><span></span>ipconfig
</pre></div>
<p>On a Linux box, enter at the terminal:</p>
<div class="highlight"><pre><span></span>ifconfig
</pre></div>
<p>Here is the output on my linux computer:</p>
<div class="highlight"><pre><span></span><span class="o">[</span>real@freedom:~<span class="o">]</span>$ ifconfig
eth0 Link encap:Ethernet HWaddr f4:6d:04:0e:8b:89
inet addr:10.0.0.9 Bcast:10.0.0.255 Mask:255.255.255.0
inet6 addr: fe80::f66d:4ff:fe0e:8b89/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:7838193 errors:0 dropped:0 overruns:0 frame:0
TX packets:5607954 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:9336295938 <span class="o">(</span><span class="m">9</span>.3 GB<span class="o">)</span> TX bytes:850374026 <span class="o">(</span><span class="m">850</span>.3 MB<span class="o">)</span>
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:214589 errors:0 dropped:0 overruns:0 frame:0
TX packets:214589 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:23187729 <span class="o">(</span><span class="m">23</span>.1 MB<span class="o">)</span> TX bytes:23187729 <span class="o">(</span><span class="m">23</span>.1 MB<span class="o">)</span>
</pre></div>
<p>As you can see, my address is 10.0.0.9. This is an internal address given to my
computer by the <span class="caps">NAT</span>. My computer doesn’t really know how to introduce himself
to the global Internet.</p>
<p>If you still want to know your global address, you could ask an external
computer. It’s one of those cases in which to know who you are you have to ask
someone else. An example for <span class="caps">IP</span> discovery service is
<a href="http://whatismyip.com">whatismyip</a>. You can try it to find out your own <span class="caps">IP</span> address.</p>
<p>While still possible, it could be a real pain to find your own address. You
have to trust some remote computer to find it out and send it to you. As an
example - If you and a friend are on two remote computers, each behind a <span class="caps">NAT</span>,
you can’t initiate any communication without first finding out your “real” addresses.</p>
<p>You might be wondering why can’t the <span class="caps">NAT</span> tell you your address. After all, the
<span class="caps">NAT</span> is directly connected to the Internet, and he knows his own address. Many
NATs could tell you your address, however each <span class="caps">NAT</span> will do it in its own way.
Therefore you can not rely on a single way to find your address. (In other
words, there is no agreed upon <span class="caps">API</span> to ask for your address).</p>
<p>In other cases you might be behind two NATs. Then only the external <span class="caps">NAT</span> knows
his own address, while the internal one doesn’t.</p>
<h4>2. Operator’s Greed</h4>
<p>Some think about the Internet as an ad-hoc distributed network of computers,
not controlled by anybody. (By now you probably know that it doesn’t work this
way). At the same time, as an Internet user you must know that being connected
to the Internet costs money.</p>
<p>You might be wondering where this money ends up eventually. You pay the money
directly to the <span class="caps">ISP</span>. The <span class="caps">ISP</span> uses the money to maintain the structure (For
example, the wires that connect you to the <span class="caps">ISP</span>), to buy <span class="caps">IP</span> addresses from <span class="caps">IANA</span>
and to pay other expenses and salaries.</p>
<p>The <span class="caps">ISP</span> and also other central entities on the chain of hierarchy could charge
more money, and in most cases as the end user you will have to comply. </p>
<p>You could get away by changing your <span class="caps">ISP</span>, but you can’t move to another <span class="caps">IANA</span> for
example. There is only one <span class="caps">IANA</span>. You can’t get away from paying for an <span class="caps">IP</span> address.</p>
<p>Other things that you will have to deal with as an end user is the bandwidth
allocation (You are only allowed to send/receive so much data per second to the
internet through your <span class="caps">ISP</span>). There is some logic behind this limitation. If
too many users of the internet will send lots of data at the same time, it will
harm the Internet experience of other users. However, in some cases the speed
limitation is arbitrary, and is related to pricing strategies. </p>
<p>Another thing is that many of the Internet users are not very tech savvy. ISPs
sometimes come up with strange schemes to make more money. Lately I also heard
of some ISPs that plan on breaking the <a href="http://en.wikipedia.org/wiki/Net_neutrality">net
neutrality</a> by blocking some
websites and slowing down others. Website owners that want to keep their
website accessible will have to pay those ISPs.</p>
<p>I do not claim that I know the real costs and expenses of managing an <span class="caps">ISP</span>,
though I do claim that from the position of <span class="caps">ISP</span> owners (Or other entities on
the hierarchy) it is very tempting to try making more money in questionable ways.</p>
<h4>3. Privacy and Censorship</h4>
<p>Being connected to the internet as an end user, you are connected directly to
your <span class="caps">ISP</span>. That means that every information you send or receive is routed
directly through your <span class="caps">ISP</span>. Your <span class="caps">ISP</span> sees everything.</p>
<p>The <span class="caps">ISP</span> could save any suspicious network activity for later use, or even block
certain websites that your government don’t want you to visit. In some
countries in the world ISPs block and monitor much of the information end users
send or receive. See for example <a href="http://en.wikipedia.org/wiki/Golden_Shield_Project">The Golden Shield
project</a> in China.</p>
<p>It is also hard to be anonymous in the Internet. To be connected, you have to
register with an <span class="caps">ISP</span>. Usually you will give the <span class="caps">ISP</span> your name and details. They
deliver communication to your home, so they also know where you live.</p>
<p>Some users use <a href="http://en.wikipedia.org/wiki/Encryption">Encryption</a> to evade
snooping from the <span class="caps">ISP</span>’s side, and also methods of data obfuscation or
steganographic channels to be able to access blocked websites and stay
anonymous online. (See for example the <a href="https://www.torproject.org/">tor
project</a>)</p>
<p>It’s pretty much a constant silent war, where the <span class="caps">ISP</span> has the initial advantage
of access to the Internet.</p>
<h3>Final Summary</h3>
<p>That was a <strong>very</strong> general summary of what is the internet, how it basically
works and what problem it tries to solve. We also pointed out some issues with
the current structure. Some of those issues arise from the hierarchical
structure of the Internet.</p>
<script type="text/javascript">if (!document.getElementById('mathjaxscript_pelican_#%@#$@#')) {
var align = "center",
indent = "0em",
linebreak = "false";
if (false) {
align = (screen.width < 768) ? "left" : align;
indent = (screen.width < 768) ? "0em" : indent;
linebreak = (screen.width < 768) ? 'true' : linebreak;
}
var mathjaxscript = document.createElement('script');
mathjaxscript.id = 'mathjaxscript_pelican_#%@#$@#';
mathjaxscript.type = 'text/javascript';
mathjaxscript.src = 'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.0/MathJax.js?config=TeX-AMS-MML_HTMLorMML';
mathjaxscript[(window.opera ? "innerHTML" : "text")] =
"MathJax.Hub.Config({" +
" config: ['MMLorHTML.js']," +
" TeX: { extensions: ['AMSmath.js','AMSsymbols.js','noErrors.js','noUndefined.js'], equationNumbers: { autoNumber: 'AMS' } }," +
" jax: ['input/TeX','input/MathML','output/HTML-CSS']," +
" extensions: ['tex2jax.js','mml2jax.js','MathMenu.js','MathZoom.js']," +
" displayAlign: '"+ align +"'," +
" displayIndent: '"+ indent +"'," +
" showMathMenu: true," +
" messageStyle: 'normal'," +
" tex2jax: { " +
" inlineMath: [ ['\\\\(','\\\\)'] ], " +
" displayMath: [ ['$$','$$'] ]," +
" processEscapes: true," +
" preview: 'TeX'," +
" }, " +
" 'HTML-CSS': { " +
" styles: { '.MathJax_Display, .MathJax .mo, .MathJax .mi, .MathJax .mn': {color: 'inherit ! important'} }," +
" linebreaks: { automatic: "+ linebreak +", width: '90% container' }," +
" }, " +
"}); " +
"if ('default' !== 'default') {" +
"MathJax.Hub.Register.StartupHook('HTML-CSS Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax['HTML-CSS'].FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"MathJax.Hub.Register.StartupHook('SVG Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax.SVG.FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"}";
(document.body || document.getElementsByTagName('head')[0]).appendChild(mathjaxscript);
}
</script>