Fault Tolerance doesn't come out of the box - Alchemy 101: Part 3

<h3>Alchemy 101: Part 3 - Fault Tolerance doesn&rsquo;t come out of the box</h3> <p>One of the biggest selling points of Elixir is the means it gives you to write fault tolerant applications via its concurrency model. Processes can broadcast their failure to dependant processes which can take appropriate action. <strong>You decide how processes should respond to failure based on your use case. There is no single solution.</strong> I&rsquo;ll give you an example in which not handling failure led to, you guesed it, more failure.</p> <p>First install and start <a href="https://www.rabbitmq.com/download.html">RabbitMQ</a>.</p> <p>Then create a new project. </p> <pre class="codehilite"><code>mix new rabbit --module Rabbit <span style="color: #0086B3">cd </span>rabbit </code></pre> <p>Then add the amqp dependency to mix.exs.</p> <pre class="codehilite"><code><span style="color: #000000;font-weight: bold">defp</span> <span style="background-color: #f8f8f8">deps</span> <span style="color: #000000;font-weight: bold">do</span> <span style="background-color: #f8f8f8">[{</span><span style="color: #990073">:amqp</span><span style="background-color: #f8f8f8">,</span> <span style="color: #d14">"</span><span style="color: #d14">~&gt; 0.1.5"</span><span style="background-color: #f8f8f8">}]</span> <span style="color: #000000;font-weight: bold">end</span> </code></pre> <p>Now add the following to lib/rabbit.ex.</p> <pre class="codehilite"><code><span style="color: #000000;font-weight: bold">defmodule</span> <span style="color: #008080">Rabbit</span> <span style="color: #000000;font-weight: bold">do</span> <span style="color: #000000;font-weight: bold">use</span> <span style="color: #008080">GenServer</span> <span style="color: #000000;font-weight: bold">require</span> <span style="color: #008080">Logger</span> <span style="color: #000000;font-weight: bold">def</span> <span style="background-color: #f8f8f8">start_link</span> <span style="color: #000000;font-weight: bold">do</span> <span style="color: #008080">GenServer</span><span style="color: #000000;font-weight: bold">.</span><span style="background-color: #f8f8f8">start_link</span><span style="background-color: #f8f8f8">(</span><span style="color: #999999">__MODULE__</span><span style="background-color: #f8f8f8">,</span> <span style="color: #008080">nil</span><span style="background-color: #f8f8f8">)</span> <span style="color: #000000;font-weight: bold">end</span> <span style="color: #000000;font-weight: bold">def</span> <span style="background-color: #f8f8f8">init</span><span style="background-color: #f8f8f8">(</span><span style="background-color: #f8f8f8">_</span><span style="background-color: #f8f8f8">)</span> <span style="color: #000000;font-weight: bold">do</span> <span style="background-color: #f8f8f8">{</span><span style="color: #990073">:ok</span><span style="background-color: #f8f8f8">,</span> <span style="background-color: #f8f8f8">connection</span><span style="background-color: #f8f8f8">}</span> <span style="color: #000000;font-weight: bold">=</span> <span style="color: #008080">AMQP</span><span style="color: #000000;font-weight: bold">.</span><span style="color: #008080">Connection</span><span style="color: #000000;font-weight: bold">.</span><span style="background-color: #f8f8f8">open</span><span style="background-color: #f8f8f8">(</span><span style="color: #d14">"</span><span style="color: #d14">amqp://localhost"</span><span style="background-color: #f8f8f8">)</span> <span style="background-color: #f8f8f8">{</span><span style="color: #990073">:ok</span><span style="background-color: #f8f8f8">,</span> <span style="background-color: #f8f8f8">channel</span><span style="background-color: #f8f8f8">}</span> <span style="color: #000000;font-weight: bold">=</span> <span style="color: #008080">AMQP</span><span style="color: #000000;font-weight: bold">.</span><span style="color: #008080">Channel</span><span style="color: #000000;font-weight: bold">.</span><span style="background-color: #f8f8f8">open</span><span style="background-color: #f8f8f8">(</span><span style="background-color: #f8f8f8">connection</span><span style="background-color: #f8f8f8">)</span> <span style="background-color: #f8f8f8">{</span><span style="color: #990073">:ok</span><span style="background-color: #f8f8f8">,</span> <span style="background-color: #f8f8f8">channel</span><span style="background-color: #f8f8f8">}</span> <span style="color: #000000;font-weight: bold">end</span> <span style="color: #000000;font-weight: bold">def</span> <span style="background-color: #f8f8f8">publish</span><span style="background-color: #f8f8f8">(</span><span style="background-color: #f8f8f8">pid</span><span style="background-color: #f8f8f8">,</span> <span style="background-color: #f8f8f8">payload</span><span style="background-color: #f8f8f8">)</span> <span style="color: #000000;font-weight: bold">do</span> <span style="color: #008080">GenServer</span><span style="color: #000000;font-weight: bold">.</span><span style="background-color: #f8f8f8">cast</span><span style="background-color: #f8f8f8">(</span><span style="background-color: #f8f8f8">pid</span><span style="background-color: #f8f8f8">,</span> <span style="background-color: #f8f8f8">{</span><span style="color: #990073">:publish</span><span style="background-color: #f8f8f8">,</span> <span style="background-color: #f8f8f8">payload</span><span style="background-color: #f8f8f8">})</span> <span style="color: #000000;font-weight: bold">end</span> <span style="color: #000000;font-weight: bold">def</span> <span style="background-color: #f8f8f8">handle_cast</span><span style="background-color: #f8f8f8">({</span><span style="color: #990073">:publish</span><span style="background-color: #f8f8f8">,</span> <span style="background-color: #f8f8f8">payload</span><span style="background-color: #f8f8f8">},</span> <span style="background-color: #f8f8f8">channel</span><span style="background-color: #f8f8f8">)</span> <span style="color: #000000;font-weight: bold">do</span> <span style="color: #008080">AMQP</span><span style="color: #000000;font-weight: bold">.</span><span style="color: #008080">Basic</span><span style="color: #000000;font-weight: bold">.</span><span style="background-color: #f8f8f8">publish</span><span style="background-color: #f8f8f8">(</span><span style="background-color: #f8f8f8">channel</span><span style="background-color: #f8f8f8">,</span> <span style="color: #d14">"</span><span style="color: #d14">"</span><span style="background-color: #f8f8f8">,</span> <span style="color: #d14">"</span><span style="color: #d14">"</span><span style="background-color: #f8f8f8">,</span> <span style="background-color: #f8f8f8">payload</span><span style="background-color: #f8f8f8">)</span> <span style="color: #008080">Logger</span><span style="color: #000000;font-weight: bold">.</span><span style="background-color: #f8f8f8">info</span><span style="background-color: #f8f8f8">(</span><span style="color: #d14">"</span><span style="color: #d14">Published </span><span style="color: #d14">#{</span><span style="background-color: #f8f8f8">payload</span><span style="color: #d14">}</span><span style="color: #d14">"</span><span style="background-color: #f8f8f8">)</span> <span style="background-color: #f8f8f8">{</span><span style="color: #990073">:noreply</span><span style="background-color: #f8f8f8">,</span> <span style="background-color: #f8f8f8">channel</span><span style="background-color: #f8f8f8">}</span> <span style="color: #000000;font-weight: bold">end</span> <span style="color: #000000;font-weight: bold">end</span> </code></pre> <p>That&rsquo;s all you need, fetch the dependencies and test Rabbit.</p> <pre class="codehilite"><code><span style="background-color: #f8f8f8">mix</span> <span style="background-color: #f8f8f8">deps</span><span style="color: #000000;font-weight: bold">.</span><span style="background-color: #f8f8f8">get</span> <span style="background-color: #f8f8f8">iex</span> <span style="color: #000000;font-weight: bold">-</span><span style="color: #008080">S</span> <span style="background-color: #f8f8f8">mix</span> <span style="background-color: #f8f8f8">iex</span><span style="background-color: #f8f8f8">(</span><span style="color: #009999">1</span><span style="background-color: #f8f8f8">)</span><span style="color: #000000;font-weight: bold">&gt;</span> <span style="background-color: #f8f8f8">{</span><span style="color: #990073">:ok</span><span style="background-color: #f8f8f8">,</span> <span style="background-color: #f8f8f8">pid</span><span style="background-color: #f8f8f8">}</span> <span style="color: #000000;font-weight: bold">=</span> <span style="color: #008080">Rabbit</span><span style="color: #000000;font-weight: bold">.</span><span style="background-color: #f8f8f8">start_link</span><span style="background-color: #f8f8f8">()</span> <span style="background-color: #f8f8f8">{</span><span style="color: #990073">:ok</span><span style="background-color: #f8f8f8">,</span> <span style="color: #999988;font-style: italic">#PID&lt;0.139.0&gt;}</span> <span style="background-color: #f8f8f8">iex</span><span style="background-color: #f8f8f8">(</span><span style="color: #009999">2</span><span style="background-color: #f8f8f8">)</span><span style="color: #000000;font-weight: bold">&gt;</span> <span style="background-color: #f8f8f8">for</span> <span style="background-color: #f8f8f8">i</span> <span style="color: #000000;font-weight: bold">&lt;-</span> <span style="color: #009999">1</span><span style="color: #000000;font-weight: bold">..</span><span style="color: #009999">3</span><span style="background-color: #f8f8f8">,</span> <span style="color: #000000;font-weight: bold">do</span><span style="background-color: #f8f8f8">:</span> <span style="color: #008080">Rabbit</span><span style="color: #000000;font-weight: bold">.</span><span style="background-color: #f8f8f8">publish</span><span style="background-color: #f8f8f8">(</span><span style="background-color: #f8f8f8">pid</span><span style="background-color: #f8f8f8">,</span> <span style="color: #d14">"</span><span style="color: #d14">message </span><span style="color: #d14">#{</span><span style="background-color: #f8f8f8">i</span><span style="color: #d14">}</span><span style="color: #d14">"</span><span style="background-color: #f8f8f8">)</span> <span style="color: #009999">19</span><span style="background-color: #f8f8f8">:</span><span style="color: #009999">21</span><span style="background-color: #f8f8f8">:</span><span style="color: #009999">31.471</span> <span style="background-color: #f8f8f8">[</span><span style="background-color: #f8f8f8">info</span><span style="background-color: #f8f8f8">]</span> <span style="color: #008080">Published</span> <span style="background-color: #f8f8f8">message</span> <span style="color: #009999">1</span> <span style="color: #009999">19</span><span style="background-color: #f8f8f8">:</span><span style="color: #009999">21</span><span style="background-color: #f8f8f8">:</span><span style="color: #009999">31.471</span> <span style="background-color: #f8f8f8">[</span><span style="background-color: #f8f8f8">info</span><span style="background-color: #f8f8f8">]</span> <span style="color: #008080">Published</span> <span style="background-color: #f8f8f8">message</span> <span style="color: #009999">2</span> <span style="color: #009999">19</span><span style="background-color: #f8f8f8">:</span><span style="color: #009999">21</span><span style="background-color: #f8f8f8">:</span><span style="color: #009999">31.471</span> <span style="background-color: #f8f8f8">[</span><span style="background-color: #f8f8f8">info</span><span style="background-color: #f8f8f8">]</span> <span style="color: #008080">Published</span> <span style="background-color: #f8f8f8">message</span> <span style="color: #009999">3</span> </code></pre> <p>Head to the <a href="http://localhost:15672/#/exchanges/%2F/amq.default">default exchange</a> and you should see some activity on the Message rates chart. <strong>Keep the above IEx shell open, you will need it again soon</strong>.</p> <p>Time to introduce a problem. Restart RabbitMQ.</p> <pre class="codehilite"><code>brew services restart rabbitmq </code></pre> <p>Go back to the IEx shell, notice how there is an OTP error report. Looks serious, the main takeaway from it is below. The socket to RabbitMQ closed causing a process to crash.</p> <pre class="codehilite"><code>19:22:57.994 [error] GenServer #PID&lt;0.142.0&gt; terminating ** (stop) :socket_closed_unexpectedly Last message: :socket_closed </code></pre> <p>But it still appears like you can publish messages. Try it.</p> <pre class="codehilite"><code><span style="color: #555555">iex(3)&gt; </span><span style="color: #000000;font-weight: bold">for </span>i &lt;- 1..3, <span style="color: #000000;font-weight: bold">do</span>: Rabbit.publish<span style="color: #000000;font-weight: bold">(</span>pid, <span style="color: #d14">"message #{i}"</span><span style="color: #000000;font-weight: bold">)</span> 19:21:31.471 <span style="color: #000000;font-weight: bold">[</span>info] Published message 1 19:21:31.471 <span style="color: #000000;font-weight: bold">[</span>info] Published message 2 19:21:31.471 <span style="color: #000000;font-weight: bold">[</span>info] Published message 3 </code></pre> <p>But if you head to the <a href="http://localhost:15672/#/exchanges/%2F/amq.default">default exchange</a> there appears to be no new messages coming in. But why? </p> <p>First of all we don&rsquo;t see any failures in the Rabbit process because AMQP.Basic.publish/4 ultimately leads to a <a href="http://erlang.org/doc/man/gen_server.html#cast-2">:gen_server.cast/2</a> (amqp_channel.erl in rabbit_common) being called with channel.pid. :gen_server.cast/2 will not return or throw an error if the PID (in this case channel.pid) does not exist. This means the failure was hard to detect. Now imagine if your application was running in the background, this could have been even more difficult to spot.</p> <p>Here comes the good part, how to handle the failure. We want Rabbit to be sent a message when the socket to RabbitMQ is closed. To do this we need to link to the channel process (to receive an exit message if it stops) we started and trap exits i.e. not crash if we receive an exit signal. To do this add the following code.</p> <pre class="codehilite"><code> <span style="color: #000000;font-weight: bold">def</span> <span style="background-color: #f8f8f8">init</span><span style="background-color: #f8f8f8">(</span><span style="background-color: #f8f8f8">_</span><span style="background-color: #f8f8f8">)</span> <span style="color: #000000;font-weight: bold">do</span> <span style="background-color: #f8f8f8">{</span><span style="color: #990073">:ok</span><span style="background-color: #f8f8f8">,</span> <span style="background-color: #f8f8f8">connection</span><span style="background-color: #f8f8f8">}</span> <span style="color: #000000;font-weight: bold">=</span> <span style="color: #008080">AMQP</span><span style="color: #000000;font-weight: bold">.</span><span style="color: #008080">Connection</span><span style="color: #000000;font-weight: bold">.</span><span style="background-color: #f8f8f8">open</span><span style="background-color: #f8f8f8">(</span><span style="color: #d14">"</span><span style="color: #d14">amqp://localhost"</span><span style="background-color: #f8f8f8">)</span> <span style="color: #008080">Process</span><span style="color: #000000;font-weight: bold">.</span><span style="background-color: #f8f8f8">flag</span><span style="background-color: #f8f8f8">(</span><span style="color: #990073">:trap_exit</span><span style="background-color: #f8f8f8">,</span> <span style="color: #008080">true</span><span style="background-color: #f8f8f8">)</span> <span style="background-color: #f8f8f8">{</span><span style="color: #990073">:ok</span><span style="background-color: #f8f8f8">,</span> <span style="background-color: #f8f8f8">channel</span><span style="background-color: #f8f8f8">}</span> <span style="color: #000000;font-weight: bold">=</span> <span style="color: #008080">AMQP</span><span style="color: #000000;font-weight: bold">.</span><span style="color: #008080">Channel</span><span style="color: #000000;font-weight: bold">.</span><span style="background-color: #f8f8f8">open</span><span style="background-color: #f8f8f8">(</span><span style="background-color: #f8f8f8">connection</span><span style="background-color: #f8f8f8">)</span> <span style="color: #008080">Process</span><span style="color: #000000;font-weight: bold">.</span><span style="background-color: #f8f8f8">link</span><span style="background-color: #f8f8f8">(</span><span style="background-color: #f8f8f8">connection</span><span style="color: #000000;font-weight: bold">.</span><span style="background-color: #f8f8f8">pid</span><span style="background-color: #f8f8f8">)</span> <span style="background-color: #f8f8f8">{</span><span style="color: #990073">:ok</span><span style="background-color: #f8f8f8">,</span> <span style="background-color: #f8f8f8">channel</span><span style="background-color: #f8f8f8">}</span> <span style="color: #000000;font-weight: bold">end</span> <span style="color: #000000;font-weight: bold">def</span> <span style="background-color: #f8f8f8">handle_info</span><span style="background-color: #f8f8f8">({</span><span style="color: #990073">:EXIT</span><span style="background-color: #f8f8f8">,</span> <span style="background-color: #f8f8f8">from</span><span style="background-color: #f8f8f8">,</span> <span style="color: #990073">:socket_closed_unexpectedly</span><span style="background-color: #f8f8f8">},</span> <span style="background-color: #f8f8f8">channel</span><span style="background-color: #f8f8f8">)</span> <span style="color: #000000;font-weight: bold">do</span> <span style="color: #008080">Logger</span><span style="color: #000000;font-weight: bold">.</span><span style="background-color: #f8f8f8">warn</span><span style="background-color: #f8f8f8">(</span><span style="color: #d14">"</span><span style="color: #d14">Received :EXIT from </span><span style="color: #d14">#{</span><span style="background-color: #f8f8f8">inspect</span><span style="background-color: #f8f8f8">(</span><span style="background-color: #f8f8f8">from</span><span style="background-color: #f8f8f8">)</span><span style="color: #d14">}</span><span style="color: #d14"> for </span><span style="color: #d14">#{</span><span style="color: #990073">:socket_closed_unexpectedly</span><span style="color: #d14">}</span><span style="color: #d14">"</span><span style="background-color: #f8f8f8">)</span> <span style="background-color: #f8f8f8">{</span><span style="color: #990073">:stop</span><span style="background-color: #f8f8f8">,</span> <span style="color: #990073">:lost_rabbitmq_connection</span><span style="background-color: #f8f8f8">}</span> <span style="color: #000000;font-weight: bold">end</span> </code></pre> <p>Kill and start the IEx shell and then run the following.</p> <pre class="codehilite"><code>brew services restart rabbitmq </code></pre> <p>Jump to the IEx shell and observe the output.</p> <pre class="codehilite"><code>19:27:33.497 [warn] Received :EXIT from #PID&lt;0.142.0&gt; for :socket_closed_unexpectedly </code></pre> <p>Perfect, now when the socket is closed the dependent process, Rabbit, is informed. Now to respond to the failure. <strong>This is something you must decide, no one including myself, library/framework developers and others can tell what to do. This is in your hands. It is also the reason why Elixir (and Erlang) has such a good reputation when it comes to building fault tolerant systems.</strong> You decide on how to respond to failure, there is no single silver bullet. For the demo though, I have choosen to simply stop then Rabbit GenServer when I receive an :EXIT for :socket_closed_unexpectedly. </p> <p>Hope you enjoyed reading. Feel free to learn more about <a href="http://www2.erlang-solutions.com/l/23452/2017-03-29/4jd1f8">RabbitMQ</a> and please give me your feedback.</p>

Permalink

How Would the World Embrace Video Collaboration if it Couldn’t Scale?

<h3>“How would the world embrace video collaboration if it couldn’t scale?”</h3> <p>This is a big question that Vidyo spotted early on, way back in 2005, as it set out to pioneer the next generation of video conferencing. </p> <p>Video collaboration at that time was primarily used for communicating between one traditional conference room to another. Since then, the world has changed. Today, in a time when “the office” can increasingly mean “pretty much anywhere” for millions of workers, (and consumer-facing apps are turning to video to service millions of customers), building a scalable video platform presents a new set of requirements. </p> <p>We are happy to share what we’ve learned over the years with developers who are interested in the next wave of video communications. At 1 PM EST on 3/29, <a href="http://www2.erlang-solutions.com/l/23452/2017-03-27/4jbrbg">join us</a> for our <strong><a href="http://www2.erlang-solutions.com/l/23452/2017-03-27/4jbrbg">“Building a Global Video Chat Platform for Developers”</a></strong> webinar, as we discuss the challenges faced and solved when building a developer platform that delivers real-time, mobile, multiparty video. </p> <h1>Today’s video collaboration landscape</h1> <p>Today, clear and reliable video collaboration has lept out of the conference room and into our pockets, letting us hop on a video call from our smartphone or tablet while on the train, in the airport or on the road - with more or less the same audio/visual quality as a dedicated room system that enjoys a perfect ethernet connection and high bandwidth. </p> <p>When hundreds of people can reliably connect face-to-face at the same time, from wherever they happen to be, this method of communicating is proving to be transformative for critical fields like healthcare, banking, government and education. This is especially true as we work to make nuisances like dropped calls, lag, grainy or blurry visuals and frozen screens a thing of the past - and stunning levels of clarity and consistency become more of the norm, from basically anywhere. </p> <h1>Behind the scenes</h1> <p>Vidyo has diligently developed patented <a href="https://vidyo.io/platform/sfu-video-router/?utm_source=erlang&amp;utm_campaign=erlang&amp;utm_medium=webinar">adaptive video layering</a> and routing technologies that have ultimately given us a reputation as the go-to enterprise-grade video service, especially among industries with tight regulations and high security demands. </p> <p>For years, global enterprises have video-enabled their applications using our APIs and SDK. Now, however, the video conferencing market is shifting to embedded video - with the vision that developers should be able to easily video-enable any application on virtually any device - effectively “video-enabling the world”. When we decided to make the shift to embedded video and roll out our video API platform, <strong><a href="http://www.vidyo.io/?utm_source=erlang&amp;utm_campaign=erlang&amp;utm_medium=webinar">Vidyo.io</a></strong>, we knew that our customers would continue to demand Vidyo’s specific brand of mobile, multiparty video that is highly scalable, without sacrificing its quality. </p> <h1>Spoiler: it’s not all plain sailing</h1> <p>One of the challenges today of offering high-quality video collaboration at scale, is signaling. Initially, we built our own signaling protocol. We had a very scalable platform that was ideal for our enterprise customers, which could easily scale to tens of thousands of users. </p> <p>But, with Vidyo.io, we were building a global video API platform to be used to video-enable any concept or idea that application developers could think of, whether for enterprise business or consumer-facing applications. We chose MongooseIM because we needed signaling that could scale to millions of users. </p> <h1>Why MongooseIM was the solution for us</h1> <p>We found MongooseIM to be simple, flexible and reliable, and designed for this type of global Internet scale. XMPP is the battle-hardened standard, with the interoperability and classical benefits (like chat, historical chat, etc.) that we, as a uniquely enterprise-grade service, needed for an architecture that could get to 99.999% uptime.</p> <p>This is how video collaboration has lept out of the conference room and into the pockets of millions of people around the world. </p> <p>As we brought <strong><a href="http://vidyo.io/?utm_source=erlang&amp;utm_campaign=erlang&amp;utm_medium=webinar">Vidyo.io</a></strong>, to market, our priorities included geo-scalable distribution, easily working through firewalls and accounting for compiled code along with native code and browser access (as WebRTC is becoming increasingly important to the video space). At the time, we evaluated all XMPP platforms. We found that Erlang was the best solution for us, and it has gone on to help us significantly on our mission to offer embedded video to the world that “just works.”</p> <h1>Ask me anything</h1> <p>I’m proud to say that today, Vidyo.io has been featured on DevOps.com, ProgrammableWeb, App Developer Magazine, Software Development Times and elsewhere. Video is now becoming ingrained in everything people do. </p> <p>I’ll be speaking more in depth on the ups and downs of building a real-time, multiparty video platform with Erlang Solutions. <strong><a href="http://www2.erlang-solutions.com/l/23452/2017-03-27/4jbrbg">Register for the webinar</a></strong> if you are interested in learning more about my experience, the technology, and to take part in the Q&amp;A panel afterwards. You can ask me anything!</p> <p>Learn more about <strong><a href="http://www2.erlang-solutions.com/l/23452/2017-03-27/4jbrdl">MongooseIM</a></strong>, Erlang Solutions’ high volume messaging solution. </p>

Permalink

Why I Stayed With Elixir

Last weekend I attended the wroc_love.rb conference in the beautiful city of Wrocław. I got a chance to be a member of the “Ruby vs Elixir” panel session and spent hours discussing related topics during the afterparties.

I love Ruby, I’m a “Ruby native” - it was the first language I really learned, and the first one I used to make money as a programmer. But for the time being, I decided to stick with Elixir. In this article I share some of my reasoning that I expressed during the panel, the discussions, and some other thoughts that I had since. I’ll also include links to other resources that explain the points I’m making probably better and to a greater extent.

Different use case

First thing, I’d like to state is that, I don’t think you have to choose one of Elixir or Ruby and completely condemn the other. They are both useful languages, and while there’s a significant overlap in the use cases, I’m convinced there’s a place for both of them.

Talking with people during various conferences and on the Elixir’s Slack channel or IRC, I see a lot of people leveraging the power of Elixir in new and exciting use cases, far beyond the regular web development - something that remains mostly inaccessible for the mainstream Ruby community. And that’s probably the reason more and more people are looking somewhere else, to languages like Elixir or Go. Because more and more applications are not “just” business web apps, more and more applications require processing a lot of data, providing some level of real-time features and require massive parallelism. Additionally, projects like Nerves open completely new areas to developers - I’m pretty sure we’re going to see more of those in the coming years.

Performance & scalability

Whenever Elixir and Ruby are mentioned in the same sentence, the next one almost certainly includes performance claims and comparisons. That’s true - in many cases, Elixir is widely faster (within one or even several orders of magnitude), but I don’t think it’s that important.

It’s an easy thing to include in a blog post, or mention in 140 characters of a tweet. Everybody loves a good benchmark. But performance is why people come to Elixir, not why they stay. It’s just the icing on the cake.

At this point I also need to mention the Elixir - different kind of promises article by Hubert Łępicki, where he explains his view on how Elixir and Ruby differ and why it might be a good idea to stay with Elixir.

Fault tolerance and isolation

The biggest advantage of the entire BEAM platform is the isolation and fault tolerance guarantees it provides. It’s the only technology, that I know of, that truly allows separating different parts of the system, so that they don’t influence one another. This happens on, at least two, levels:

  • limited error propagation - the supervision system allows for isolating failure and collectively managing the unexpected failure scenarios without excessive code. A bug in one part of the system does not affect other parts;
  • progress guarantee - the preemptive scheduling of processes is a true life changer. With it, we never have to worry, we’re going to “block” our thread pool (which is a major concern in other actor system implementations), we never have to be concerned about starving some part of the system. This is where the soft-realtime guarantees of the platform come from.

To learn more about the fault tolerance and isolation of the Erlang ecosystem, I recommend reading the excellent The Zen of Erlang essay by Fred Hebert.

Developer happiness

Ruby and Rails are praised for the “developer happiness”. And that’s true - starting an application with Rails is a breeze, allows you to deliver features fast and solve problems quickly. It’s extremely enjoyable and satisfying to see a complex system arise from nothing in such a short time. Unfortunately, I feel like the whole regard for developer happiness goes completely out the window, when we start considering longer projects. The defaults promoted by the language and framework, make it easy to create code that is hard to maintain and makes you want to cry - and those are not tears of happiness. It’s true you can use your own patterns that make this a non-issue, but for a majority of projects, the sad truth is, the defaults are what is used.

The “developer happiness” is equally a goal for the Elixir community. But the notion is regarded much more widely - not just in the initial phase of the project, but also in the long term. Various libraries are willing to sacrifice some of the initial “velocity” for the long-term maintainability. The changes introduced in Ecto 2.0 (focused primarily on the removal of the features that are known to create maintenance burdens - e.g. callbacks), and recent changes in code organisation philosophy from Phoenix 1.3, are all a testament to that.

One thing about the Phoenix 1.3 changes to note, is that those changes are not sudden or unexpected - various discussions about the structure of a Phoenix application were happening in the blog articles, forum posts, and conference talks for at least a year. The leaders of the community are not afraid of listening to others, admitting to errors when necessary, and correcting them.

Tools

A common worry for new platforms is the lack of tooling. But Elixir is not a new platform - the tooling for the Erlang ecosystem was constructed over the last 20 years into an impressive set of debugging and diagnostic facilities. The most profound - tracing, sometimes seems like a true superpower. The language and the platform cooperate nicely providing basic low-level facilities, which make it easy to build extremely useful tools on top of them - Recon, Redbug and observer just to name a few. And there’s more coming (!) with projects like wobserver and Erlang Performance Lab putting the bar even higher.

That said, on the front of editor tooling we have a long way to go to match other languages. Hopefully, we’ll get a student to implement the Language Server Protocol for Elixir during the Google Summer of Code, which has the potential to improve the support quickly across various editors (if you’re a student, please consider applying for GSoC!).

It’s all about simplicity

If I were to pick one thing about Elixir, that makes me like it so much, it would be simplicity. In the excellent talk Simple Made Easy, Rich Hickey - the creator of Clojure, defines the difference between those two, at first glance similar terms. In short, something is easy when it is short, concise and familiar, but something is simple when it is not complex when it is easy to understand and decompose.

I feel like many libraries and solutions in the Ruby community, overly focus on making it easy - “just include this one line and you can launch rockets to the moon” is such a common claim in READMEs of many gems. The mainstream Ruby community focuses on building solutions to problems and only then trying to make them reusable in some way.

On the other hand, most of the core Elixir libraries - Plug, Phoenix, Ecto, and countless other ones are primarily tools for building solutions. They are more low-level, which means you need to write some of the code yourself, but it’s much easier to switch parts or customise their behaviour. They allow you to spend time solving your problem, instead of trying to make a solution to somebody else’s problem solve yours as well.

Community

With all of that out of the way, I love one thing about both Elixir & Ruby - the people. You rarely find such open and welcoming souls as amongst people attending conferences for either language. The honest love for learning, simple human kindness, and willingness to discuss are otherwise unheard of. Even when having arguments on such emotional subjects as the choice of the language, the discussion is respectful and factual. I thank very much my co-panelist for a great discussion Hubert, Andrzej, Robert and Maciej during the panel, and all the other people that spoke to me on the subject during the afterparties. Thank you.

Permalink

Scaling RabbitMQ on a CoreOS cluster through Docker

<h2>Introduction</h2> <p>RabbitMQ provides, among other features, clustering capabilities. Using clustering, a group of properly configured hosts will behave the same as a single broker instance.</p> <p>All the nodes of a RabbitMQ cluster share the definition of vhosts, users, and exchanges but not queues. By default they physically reside on the node where they have been created, however as from version 3.6.1, the queue node owneriship can be configured using <a href="https://www.erlang-solutions.com/blog/take-control-of-your-rabbitmq-queues.html">Queue Master Location policies</a>. Queues are globally defined and reachable, by establishing a connection to any node of the cluster. </p> <p>Modern architectures often involve container based ways of scaling such as Docker . In this post we will see how to create a dynamic scaling RabbitMQ cluster using <a href="https://coreos.com/">CoreOS</a> and <a href="https://www.docker.com/">Docker</a>:</p> <p><img src="https://esl-website-production.s3.amazonaws.com/uploads/image/file/298/coreos_cluster.png" alt="Alternative Text"></p> <p>We will take you on a step by step journey from zero to the cluster. </p> <h2>Get ready</h2> <p>We are going to use different technologies, although we will not get into the details of all of them. For instance it is not required to have a deep CoreOS/Docker knowledge for the purpose of executing this test.</p> <p>It can be executed using your pc, and what you need is: </p> <ul> <li><a href="https://www.vagrantup.com/">Vagrant</a></li> <li><a href="https://www.virtualbox.org/">VirtualBox</a></li> <li><a href="https://git-scm.com/">Git</a> <br><br></li> </ul> <p>What we will do:</p> <ol> <li><a href="#configure-coreos-cluster-machines">Configure CoreOS cluster machines</a></li> <li><a href="#configure-docker-swarm">Configure Docker Swarm</a> </li> <li><a href="#configure-rabbitmq-docker-cluster">Configure RabbitMQ docker cluster</a> </li> </ol> <h2>Configure CoreOS cluster machines</h2> <p>First we have to configure the CoreOS cluster:</p> <p><strong>1.</strong> Clone the vagrant repository:</p> <pre class="codehilite"><code><span style="color: #555555">$ </span>git clone https://github.com/coreos/coreos-vagrant <span style="color: #555555">$ </span><span style="color: #0086B3">cd </span>coreos-vagrant </code></pre> <p><strong>2.</strong> Use the user-data example file:</p> <pre class="codehilite"><code><span style="color: #555555">$ </span>cp user-data.sample user-data </code></pre> <p><strong>3.</strong> Configure the cluster parameters:</p> <pre class="codehilite"><code><span style="color: #555555">$ </span>cp config.rb.sample config.rb </code></pre> <p><strong>4.</strong> Open the file then uncomment <code>num_instances</code> and change it to 3, or execute:</p> <pre class="codehilite"><code> sed -i.bk <span style="color: #d14">'s/$num_instances=1/$num_instances=3/'</span> config.rb </code></pre> <p><strong>5.</strong> Start the machines using <code>vagrant up</code>:</p> <pre class="codehilite"><code><span style="color: #555555">$ </span>vagrant up Bringing machine <span style="color: #d14">'core-01'</span> up with <span style="color: #d14">'virtualbox'</span> provider... Bringing machine <span style="color: #d14">'core-02'</span> up with <span style="color: #d14">'virtualbox'</span> provider... Bringing machine <span style="color: #d14">'core-03'</span> up with <span style="color: #d14">'virtualbox'</span> provider… </code></pre> <p><strong>6.</strong> Add the ssh key:</p> <p><code>ssh-add ~/.vagrant.d/insecure_private_key</code></p> <p><strong>7.</strong> Use vagrant <code>ssh core-XX -- -A</code> to login, ex:</p> <pre class="codehilite"><code><span style="color: #555555">$ </span>vagrant ssh core-01 -- -A <span style="color: #555555">$ </span>vagrant ssh core-02 -- -A <span style="color: #555555">$ </span>vagrant ssh core-03 -- -A </code></pre> <p><strong>8.</strong> Test your CoreOS cluster, login to the machine core-01:</p> <p><code>$ vagrant ssh core-01 -- -A</code></p> <p>Then</p> <pre class="codehilite"><code>core@core-01 ~ <span style="color: #008080">$ </span>fleetctl list-machines MACHINE IP METADATA 5f676932... 172.17.8.103 - 995875fc... 172.17.8.102 - e4ae7225... 172.17.8.101 - </code></pre> <p><strong>9.</strong> Test the etcd service:</p> <pre class="codehilite"><code>core@core-01 ~ <span style="color: #008080">$ </span>etcdctl <span style="color: #0086B3">set</span> /my-message <span style="color: #d14">"I love Italy"</span> I love Italy </code></pre> <p><strong>10.</strong> Login to vagrant ssh core-02:</p> <pre class="codehilite"><code><span style="color: #555555">$ </span>vagrant ssh core-02 -- -A core@core-02 ~ <span style="color: #008080">$ </span>etcdctl get /my-message I love Italy </code></pre> <p><strong>11.</strong> Login to vagrant ssh core-03:</p> <pre class="codehilite"><code>vagrant ssh core-02 -- -A core@core-03 ~ <span style="color: #008080">$ </span>etcdctl get /my-message I love Italy </code></pre> <p>As result you should have:</p> <p><img src="https://esl-website-production.s3.amazonaws.com/uploads/image/file/297/etcd.png" alt="Alternative Text"></p> <p><strong>12.</strong> Test Docker installation using <code>docker -v</code> :</p> <pre class="codehilite"><code>core@core-01 ~ <span style="color: #008080">$ </span>docker -v Docker version 1.12.3, build 34a2ead </code></pre> <p><strong>13.</strong> (Optional step) Run the first image with docker run :</p> <pre class="codehilite"><code>core@core-01 ~ <span style="color: #008080">$ </span> docker run ubuntu /bin/echo <span style="color: #d14">'Hello world'</span> … Hello world </code></pre> <p>The CoreOS the cluster is ready, and we are able to run Docker inside CoreOS. Let’s test our first RabbitMQ docker instance:</p> <p><strong>14.</strong> Execute the official RabbitMQ docker image:</p> <pre class="codehilite"><code>core@core-01 ~ <span style="color: #008080">$ </span>docker run -d --hostname my-rabbit --name first_rabbit -p 15672:15672 rabbitmq:3-management </code></pre> <p><strong>15.</strong> Check your <code>eth1</code> vagrant IP (used to access the machine) :</p> <pre class="codehilite"><code>core@core-01 ~ <span style="color: #008080">$ </span>ifconfig | grep -A1 eth1 eth1: <span style="color: #008080">flags</span><span style="color: #000000;font-weight: bold">=</span>4163&lt;UP,BROADCAST,RUNNING,MULTICAST&gt; mtu 1500 inet 172.17.8.101 netmask 255.255.255.0 broadcast 172.17.8.255 </code></pre> <p>Go to <code>http://&lt;your_ip&gt;:15672/#/</code> in this case: <a href="http://172.17.8.101:15672/#/">http://172.17.8.101:15672/#/</a>.</p> <p>You should see the RabbitMQ management UI as (<code>guest</code> <code>guest</code> ):</p> <p><img src="https://esl-website-production.s3.amazonaws.com/uploads/image/file/296/http_ui.jpg" alt="Alternative Text"></p> <p>In order to scale up the node above, we should run another container with <code>--link</code> parameter and execute <code>rabbitmqctl join_cluster rabbit@&lt;docker_host_name&gt;</code>. In order to scale down we should stop the second container and execute <code>rabbitmqctl forget_cluster_node rabbit@&lt;docker_host_name&gt;</code>.</p> <p>Turn into more positive. e.g. This is one of the areas where further enhancements on automation would be helpful.</p> <p>We need docker orchestration to configure and manage the docker cluster. Among the available orchestration tools, we have chosen <a href="https://www.docker.com/products/docker-swarm">Docker swarm</a>.</p> <p>Before going ahead we should remove all the running containers:</p> <pre class="codehilite"><code>core@core-01 ~ <span style="color: #008080">$ </span>docker rm -f <span style="color: #000000;font-weight: bold">$(</span>docker ps -a -q<span style="color: #000000;font-weight: bold">)</span> </code></pre> <p>And the images:</p> <pre class="codehilite"><code>core@core-01 ~ <span style="color: #008080">$ </span>docker rmi -f <span style="color: #000000;font-weight: bold">$(</span>docker images -q<span style="color: #000000;font-weight: bold">)</span> </code></pre> <h2>Configure Docker swarm</h2> <p>Docker Swarm is the native clustering mechanism for Docker. We need to initialize one node and join the other nodes, as: </p> <p><strong>1.</strong> Swarm initialization: to the node-01 execute <code>docker swarm init --advertise-addr 172.17.8.101</code>.</p> <p><code>docker swarm init</code> automatically generates the command (with the token) to join other nodes to the cluster, as: </p> <pre class="codehilite"><code>core@core-01 ~ <span style="color: #008080">$ </span>docker swarm init --advertise-addr 172.17.8.101 Swarm initialized: current node <span style="color: #000000;font-weight: bold">(</span>2fyocfwfwy9o3akuf6a7mg19o<span style="color: #000000;font-weight: bold">)</span> is now a manager. To add a worker to this swarm, run the following <span style="color: #0086B3">command</span>: docker swarm join <span style="color: #d14">\</span> --token SWMTKN-1-3xq8o0yc7h74agna72u2dhqv8blaw40zs1oow9io24u229y22z-4bysfgwdijzutfl6ydguqdu1s <span style="color: #d14">\</span> 172.17.8.101:2377 </code></pre> <p>Docker swarm cluster is is composed by leader node and worker nodes. </p> <p><strong>2.</strong> Join the core-02 to the cluster <code>docker swarm join --token &lt;token&gt; &lt;ip&gt;:&lt;port&gt;</code> (you can copy and paste the command generated to the step 1) :</p> <p>In this case:</p> <pre class="codehilite"><code>core@core-02 ~ <span style="color: #008080">$ </span>docker swarm join <span style="color: #d14">\</span> --token SWMTKN-1-3xq8o0yc7h74agna72u2dhqv8blaw40zs1oow9io24u229y22z-4bysfgwdijzutfl6ydguqdu1s <span style="color: #d14">\</span> 172.17.8.101:2377 This node joined a swarm as a worker. </code></pre> <p><strong>3.</strong> Join the core-03 to the cluster <code>docker swarm join --token &lt;token&gt; &lt;ip&gt;:&lt;port&gt;</code> :</p> <pre class="codehilite"><code>core@core-03 ~ <span style="color: #008080">$ </span>docker swarm join <span style="color: #d14">\</span> --token SWMTKN-1-3xq8o0yc7h74agna72u2dhqv8blaw40zs1oow9io24u229y22z-4bysfgwdijzutfl6ydguqdu1s <span style="color: #d14">\</span> 172.17.8.101:2377 This node joined a swarm as a worker. </code></pre> <p><strong>4.</strong> Check the swarm cluster using <code>docker node ls</code> :</p> <pre class="codehilite"><code>core@core-01 ~ <span style="color: #008080">$ </span>docker node ls ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS 07m3d8ipj2kgdiv9jptv9k18a core-02 Ready Active 2fyocfwfwy9o3akuf6a7mg19o <span style="color: #000000;font-weight: bold">*</span> core-01 Ready Active Leader 8cicxxpn5f86u3roembijanig core-03 Ready Active </code></pre> <h2>Configure RabbitMQ docker cluster</h2> <p>There are different ways to create a RabbitMQ cluster:</p> <ul> <li>Manually with <code>rabbitmqctl</code></li> <li>Declaratively by listing cluster nodes in a config file</li> <li>Declaratively with <code>rabbitmq-autocluster</code> (a plugin)</li> <li>Declaratively with <code>rabbitmq-clusterer</code> (a plugin) <br><br></li> </ul> <p>To create the cluster we use the rabbitmq-autocluster plugin since it supports different services discovery such as <a href="https://www.consul.io/">Consul</a>, <a href="https://github.com/coreos/etcd">etcd2</a>, DNS, AWS EC2 tags or <a href="https://aws.amazon.com/autoscaling/">AWS Autoscaling Groups</a>.</p> <p>We decided to use etcd2, this is why we tested it on <a href="#configure-coreos-cluster-machines"><strong>Configure CoreOS cluster machines</strong></a> see step 8.</p> <h4>Ready to the final round, create the RabbitMQ cluster.</h4> <p><strong>1.</strong> Create A Docker network:</p> <pre class="codehilite"><code><span style="color: #555555">core@core-01~$ </span>docker network create --driver overlay rabbitmq-network </code></pre> <p>The swarm makes the overlay network available only to nodes in the swarm that require it for a service</p> <p><strong>2.</strong> Create a Docker service:</p> <pre class="codehilite"><code>core@core-01 ~ <span style="color: #008080">$ </span>docker service create --name rabbitmq-docker-service <span style="color: #d14">\</span> -p 15672:15672 -p 5672:5672 --network rabbitmq-network -e <span style="color: #008080">AUTOCLUSTER_TYPE</span><span style="color: #000000;font-weight: bold">=</span>etcd <span style="color: #d14">\</span> -e <span style="color: #008080">ETCD_HOST</span><span style="color: #000000;font-weight: bold">=</span><span style="color: #000000;font-weight: bold">${</span><span style="color: #008080">COREOS_PRIVATE_IPV4</span><span style="color: #000000;font-weight: bold">}</span> -e <span style="color: #008080">ETCD_TTL</span><span style="color: #000000;font-weight: bold">=</span>30 -e <span style="color: #008080">RABBITMQ_ERLANG_COOKIE</span><span style="color: #000000;font-weight: bold">=</span><span style="color: #d14">'ilovebeam'</span> <span style="color: #d14">\</span> -e <span style="color: #008080">AUTOCLUSTER_CLEANUP</span><span style="color: #000000;font-weight: bold">=</span><span style="color: #0086B3">true</span> -e <span style="color: #008080">CLEANUP_WARN_ONLY</span><span style="color: #000000;font-weight: bold">=</span><span style="color: #0086B3">false </span>gsantomaggio/rabbitmq-autocluster </code></pre> <p><strong>Note</strong>: The first time you have to wait a few seconds.</p> <p><strong>3.</strong> Check the service list using <code>docker service ls</code></p> <p><strong>4.</strong> You can check the RabbitMQ instance running on <code>http://&lt;you_vagrant_ip&gt;:15672/#/</code> most likely <a href="http://172.17.8.101:15672/#/">http://172.17.8.101:15672/#/</a></p> <p><strong>5.</strong> Scale your cluster, using <code>docker service scale</code> as:</p> <pre class="codehilite"><code>core@core-01 ~ <span style="color: #008080">$ </span>docker service scale rabbitmq-docker-service<span style="color: #000000;font-weight: bold">=</span>5 rabbitmq-docker-service scaled to 5 </code></pre> <h4>Congratulations!! You just scaled your cluster to 5 nodes!</h4> <p>Since the 3 CoreOS machine are in cluster, you can use all the 3 machines to access the cluster, as:</p> <ul> <li><a href="http://172.17.8.101:15672/#/">http://172.17.8.101:15672/#/</a> </li> <li><a href="http://172.17.8.102:15672/#/">http://172.17.8.102:15672/#/</a> </li> <li><a href="http://172.17.8.103:15672/#/">http://172.17.8.103:15672/#/</a><br> <br><br> Where you should have:</li> </ul> <p><img src="https://esl-website-production.s3.amazonaws.com/uploads/image/file/295/ui_cluster.jpg" alt="Alternative Text"></p> <p><strong>6.</strong> Check the cluster status on the machine:</p> <pre class="codehilite"><code>core@core-01 ~ <span style="color: #008080">$ </span>docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES b480a09ea6e2 gsantomaggio/rabbitmq-autocluster:latest <span style="color: #d14">"docker-entrypoint.sh"</span> 1 seconds ago Up Less than a second 4369/tcp, 5671-5672/tcp, 15671-15672/tcp, 25672/tcp rabbitmq-docker-service.3.1vp3o2w1eelzbpjngxncb9wur aabb62882b1b gsantomaggio/rabbitmq-autocluster:latest <span style="color: #d14">"docker-entrypoint.sh"</span> 6 seconds ago Up 5 seconds 4369/tcp, 5671-5672/tcp, 15671-15672/tcp, 25672/tcp rabbitmq-docker-service.1.f2larueov9lk33rwzael6oore </code></pre> <p>Same to the other nodes, you have more or less the same number of containers. </p> <p>Let’s see now in detail the <code>docker service</code> parameters:</p> <style type="text/css"> .tg {border-collapse:collapse;border-spacing:0;} .tg td{font-family:Arial, sans-serif;font-size:14px;padding:10px 10px;border-style:solid;border-width:1px;overflow:hidden;word-break:normal;} .tg th{font-family:Arial, sans-serif;font-size:14px;font-weight:normal;padding:10px 10px;border-style:solid;border-width:1px;overflow:hidden;word-break:normal;} .tg .tg-9hbo{font-weight:bold;vertical-align:top} .tg .tg-yw4l{vertical-align:top} </style> <p><table class="tg" style="undefined;table-layout: fixed; width: 806px"> <colgroup> <col style="width: 300px"> <col style="width: 560px"> </colgroup> <tr> <th class="tg-9hbo">Command</th> <th class="tg-9hbo">Description</th> </tr> <tr> <td class="tg-yw4l"><code><FONT style="BACKGROUND-COLOR: #F7F7F7">docker service create</font></code></td> <td class="tg-yw4l">Create a docker service</td> </tr> <tr> <td class="tg-yw4l"><code><FONT style="BACKGROUND-COLOR: #F7F7F7">--name rabbitmq-docker-service</font></code></td> <td class="tg-yw4l">Set the service name, you can check the services list using <code><FONT style="BACKGROUND-COLOR: #F7F7F7">docker service ls</font></code></td> </tr> <tr> <td class="tg-yw4l"><code><FONT style="BACKGROUND-COLOR: #F7F7F7">-p 15672:15672 -p 5672:5672</font></code></td> <td class="tg-yw4l">map the RabbitMQ standard ports, 5672 is the AMQP port and 15672 is the Management_UI port</td> </tr> <tr> <td class="tg-yw4l"><code><FONT style="BACKGROUND-COLOR: #F7F7F7">--network rabbitmq-network</font></code></td> <td class="tg-yw4l">Choose the docker network</td> </tr> <tr> <td class="tg-yw4l"><code><FONT style="BACKGROUND-COLOR: #F7F7F7">-e RABBITMQ_ERLANG_COKIE=&#39;ilovebeam&#39;</font></code></td> <td class="tg-yw4l">Set the same erlang.cookie value to all the containers, needed by RabbitMQ to create a cluster. With different erlang.cookie it is not possible create a cluster.</td> </tr> </table><br><br></p> <p>Next are the auto-cluster parameters: </p> <style type="text/css"> .tg {border-collapse:collapse;border-spacing:0;} .tg td{font-family:Arial, sans-serif;font-size:14px;padding:10px 10px;border-style:solid;border-width:1px;overflow:hidden;word-break:normal;} .tg th{font-family:Arial, sans-serif;font-size:14px;font-weight:normal;padding:10px 10px;border-style:solid;border-width:1px;overflow:hidden;word-break:normal;} .tg .tg-9hbo{font-weight:bold;vertical-align:top} .tg .tg-yw4l{vertical-align:top} </style> <p><table class="tg" style="undefined;table-layout: fixed; width: 860px"> <colgroup> <col style="width: 300px"> <col style="width: 560px"> </colgroup> <tr> <th class="tg-9hbo">Command</th> <th class="tg-9hbo">Description</th> </tr> <tr> <td class="tg-yw4l"><code>-e AUTOCLUSTER_TYPE=etcd</code></td> <td class="tg-yw4l">set the service discovery backend = etcd</td> </tr> <tr> <td class="tg-yw4l"><code>-e ETCD_HOST=${COREOS_PRIVATE_IPV4}</code></td> <td class="tg-yw4l">The containers need to know the etcd2 ip. After executing the service you can query the database using the command line etcdctl ex: <code>etcdctl ls /rabbitmq -recursive</code> or using the http API ex: <code>curl -L <a href="http://127.0.0.1:2379/v2/keys/rabbitmq">http://127.0.0.1:2379/v2/keys/rabbitmq</a></code></td> </tr> <tr> <td class="tg-yw4l"><code>-e ETCD_TTL=30</code></td> <td class="tg-yw4l">Used to specify how long a node can be down before it is removed from etcd&rsquo;s list of RabbitMQ nodes in the cluster</td> </tr> <tr> <td class="tg-yw4l"><code>-e AUTOCLUSTER_CLEANUP=true</code></td> <td class="tg-yw4l"><li>Enables a periodic check that removes any nodes that are not alive in the cluster and no longer listed in the service discovery list.</li><br><li>Scaling down removes one or more containers, the nodes will be removed from <code>etcd</code> database, see, for example: <code>docker service scale rabbitmq-docker-service=4</code></li></td> </tr> <tr> <td class="tg-yw4l"><code>-e CLEANUP_WARN_ONLY=false</code></td> <td class="tg-yw4l">If set, the plugin will only warn about nodes that it would cleanup. <code>AUTOCLUSTER_CLEANUP</code> requires <code>CLEANUP_WARN_ONLY=false</code> to work.</td> </tr> <tr> <td class="tg-yw4l">gsantomaggio/rabbitmq-autocluster</td> <td class="tg-yw4l">The official docker image does not support the auto-cluster plugin, in my personal opinion they should. I created a docker image and registered it on docker-hub.</td> </tr> </table><br><br></p> <p><code>AUTOCLUSTER_CLEANUP</code> to true removes the node automatically, if <code>AUTOCLUSTER_CLEANUP</code> is false you need to remove the node manually.</p> <p><strong>Scaling down and <code>AUTOCLUSTER_CLEANUP</code> can be very dangerous</strong>, if there are not <a href="https://www.rabbitmq.com/ha.html">HA policies</a> all the queues and messages stored to the node will be lost. To enable HA policy you can use the command line or the HTTP API, in this case the easier way is the HTTP API, as:</p> <pre class="codehilite"><code>curl -u guest:guest -H <span style="color: #d14">"Content-Type: application/json"</span> -X PUT <span style="color: #d14">\</span> -d <span style="color: #d14">'{"pattern":"","definition":{"ha-mode":"exactly","ha-params":3,"ha-sync-mode":"automatic"}}'</span> <span style="color: #d14">\</span> http://172.17.8.101:15672/api/policies/%2f/ha-3-nodes </code></pre> <p><strong>Note</strong>: Enabling the mirror queues across all the nodes could impact the performance, especially when the number of the nodes is undefined. Using <code>&quot;ha-mode&quot;:&quot;exactly&quot;,&quot;ha-params&quot;:3</code> we enable the mirror only for 3 nodes. So scaling down should be done for one node at time, in this way RabbitMQ can move the mirroring to other nodes.</p> <h2>Conclusions</h2> <p>RabbitMQ can easily scale inside Docker, each RabbitMQ node has its own files and does not need to share anything through the file system. It fits perfectly with containers.</p> <p>This architecture implements important features as:</p> <ul> <li>Round-Robin connections</li> <li>Failover cluster machines/images </li> <li>Portability </li> <li>Scaling in term of CoreOS nodes and RabbitMQ nodes <br><br></li> </ul> <p>Scaling RabbitMQ on Docker and CoreOS is very easy and powerful, we are testing and implementing the same environment using different orchestration tools and service discovery tools as kubernetes, consul etc, by the way <strong>we consider this architecture as experimental</strong>. </p> <p>Here you can see the final result:</p> <p><a href="https://www.youtube.com/embed/i9KH_FWJ2ek" target="_blank"><img src="http://img.youtube.com/vi/i9KH_FWJ2ek/0.jpg" alt="RabbitMQ on CoreOS" width="240" height="180" border="10" /></a></p> <p>Enjoy!</p> <p>At Erlang Solutions we can help you design, implement, operate and optimise a system utilising RabbitMQ. We provide tier 3 (most advanced) level RabbitMQ support for Pivotal`s customers and we work closely with Pivotal support tier 1 and 2. We also offer RabbitMQ customisation if your system goes beyond the typical requirements, and bespoke support for such implementations. </p> <p><a href="https://www.erlang-solutions.com/products/rabbitmq.html">Learn more about RabbitMQ</a>, Erlang Solutions&rsquo; the only fast and dependable open-source message server you&rsquo;ll ever need. <br></p>

Permalink

Copyright © 2016, Planet Erlang. No rights reserved.
Planet Erlang is maintained by Proctor.