Docs in the shell 2: Man-Erlang interface

<p><strong>Learn more about how Erlang Solutions&rsquo; <a href="http://www2.erlang-solutions.com/l/23452/2017-05-23/4llxcl">Erlang and Elixir Solutions</a> or <a href="http://www2.erlang-solutions.com/emailpreference">sign up to our mailing list</a> to be the first to know about our future blog posts.</strong></p> <hr> <h1>Docsh update</h1> <p>It&rsquo;s been a while since <a href="https://medium.com/@erszcz/docsh-status-update-117f39bb68df">the last update on docsh</a> and over a year since the <a href="http://www2.erlang-solutions.com/l/23452/2017-05-16/4lcdmx">idea of shell access to Erlang docs originally emerged in my mind</a>. Believe it or not, the project has seen a huge progress, though one might argue that at a pace a tad slower than the speed of light. By the way, if you ask me, it&rsquo;s &ldquo;doc-es-age&rdquo;, not &ldquo;doc-shhh&rdquo;.</p> <p>Let&rsquo;s recap what the original goals were, whether they&rsquo;ve shifted, and how we fared realising them.</p> <h1>Require as little fuss as possible to use</h1> <p>If you&rsquo;re just beginning your Erlang journey, there&rsquo;s an install script available, which will take care of the default setup in a few seconds. Install Erlang, <a href="https://github.com/erszcz/docsh#installation">clone docsh, run <code>install.sh</code></a> and there you are!</p> <p>For the veterans: the install script won&rsquo;t mess up your precious Erlang development environment settings, it&rsquo;s going to balk at touching your config files, but will provide hints on how you can enable doc access manually.</p> <h1>Provide value to the developer no matter whether they use community or OTP libraries</h1> <p>Once docsh is installed, when you run <code>erl</code> it will greet you with a reminder that docsh is enabled. From this point, the shell is equipped with a new set of helper functions for accessing the documentation of modules and functions, function specs, and type definitions. These helpers will look familiar to those who had a sip of Elixir, but have been adapted to Erlang&rsquo;s syntax. If in doubt, refer to <code>h(docsh).</code> in the Erlang shell - that is, use docsh to view its own documentation. <a href="https://github.com/erszcz/docsh#usage">The README</a> and <a href="https://github.com/erszcz/docsh/blob/master/examples.md">some examples</a> might also whet your appetite some more if you&rsquo;re not yet convinced enough to install the tool.</p> <p align="center"> <img src="https://cdn-images-1.medium.com/max/1600/0*JUUdbxbnccP5Pq6N."> </p> <p>There&rsquo;s one caveat at the moment, though. The so called <em>Erlang/OTP source bundles</em> (i.e. the .tar.gz archives you can get from <a href="http://www.erlang.org/?utm_source=Erlang%20Solutions&amp;utm_medium=Blog">erlang.org</a>), you use to <em>install from source</em>, aren&rsquo;t really exactly the thing they&rsquo;re supposed to be. The C source code included within is compiled on your machine during installation, but the standard Erlang modules come precompiled on the tarball packager&rsquo;s machine.</p> <p>When docsh inspects a module, in some cases it tries to find its source code. Given the source file path is not valid on your machine, documentation extraction will fail. All in all, unless you take extra steps when installing Erlang, currently for the standard OTP modules docsh can only provide function specs, not the inline EDoc documentation one might expect. Despite that, the specs can be more than enough to tell what a function does, and are an invaluable help when determining its parameters&rsquo; meaning and order. <a href="https://github.com/erszcz/docsh/issues/7">This <em>EDoc extraction from OTP modules thing</em> is a known issue</a> and will be fixed in the future.</p> <h1>Make docsh documentation compatible with Elixir&rsquo;s format for IEx to use</h1> <p>This goal was twofold: to make the internal format compatible with Elixir and to enable IEx (Elixir shell) to access Erlang documentation. While the internal format compatibility is shelved for now, the access from the Elixir aspect has progressed somewhat.</p> <p>However, there are more languages on the BEAM (the Erlang VM) than just Elixir and Erlang. For instance <a href="http://lfe.io/">LFE</a>, <a href="http://efene.org/">Efene</a> and <a href="https://github.com/alpaca-lang/alpaca">Alpaca</a> come to mind. <a href="https://github.com/alpaca-lang/alpaca/issues/134">With Alpaca having an open issue on the doc string system</a> they&rsquo;re designing, there&rsquo;s some space for research and maybe even for future collaboration.</p> <p>Back to Elixir, though - there&rsquo;s a project called <a href="https://github.com/philosophers-stone/ehelper">Ehelper</a>, whose goals overlap a bit with those of docsh. Ehelper aims to extend the Elixir shell with programmable plugins. Thanks to this architecture, docsh could be an Erlang documentation provider for Ehelper. It might be the way to go, but tinkering directly with Elixir&rsquo;s <code>.iex.exs</code> might be more straightforward. I have to spend more time on finding the most convenient approach for everyday use. Let&rsquo;s see how it unfolds.</p> <p>In the <a href="http://www2.erlang-solutions.com/l/23452/2017-05-16/4lcdmx">original blog post</a>, I also highlighted some issues which were yet to be solved back then. The first one was documentation format diversity: at least Markdown, AsciiDoc and EDoc are used for documentation in the Erlang community, while the OTP Team also uses non-EDoc XML out-of-source docs for the standard library (the Erlang manpages are generated from it). This is addressed in docsh by defining a <code>docsh_reader</code> behaviour, which implementations can consume different inputs and produce a docsh-internal representation. This internal format is written out by a <code>docsh_writer</code>.</p> <p>Currently, there are two readers: one for reading the Erlang abstract syntax tree (stored in modules if <code>debug_info</code> option is passed to the compiler), and one for reading EDoc comments from source files. That&rsquo;s why even if the source files aren&rsquo;t available the function specs can still be extracted from Erlang modules. Only one writer is available for now - it embeds the documentation into the module .beam file for later recovery.</p> <p>Whew, that was a long one! <a href="https://github.com/erszcz/docsh#usage">Check out how docsh makes your everyday life easier</a> and <a href="https://twitter.com/erszcz/">let me know</a> where it could use some improvements!</p> <hr> <p><strong>Learn more about how Erlang Solutions&rsquo; <a href="http://www2.erlang-solutions.com/l/23452/2017-05-23/4llxcl">Erlang and Elixir Solutions</a> or <a href="http://www2.erlang-solutions.com/emailpreference">sign up to our mailing list</a> to be the first to know about our future blog posts.</strong></p>

Permalink

Erlang 19.0 Garbage Collector

<p><strong>Learn more about how Erlang Solutions&rsquo; <a href="http://www2.erlang-solutions.com/l/23452/2017-05-23/4llxcl">Erlang and Elixir Solutions</a> or <a href="http://www2.erlang-solutions.com/emailpreference">sign up to our mailing list</a> to be the first to know about our future blog posts.</strong></p> <hr> <p>Disclaimer: This blog post describes the state of the Erlang garbage collector in Erlang/OTP 19.0. The things that will change in 19.0 are clearly marked in each of the sections. The Erlang/OTP master branch is a moving target and some of the things may have changed by the time 19.0 is released.</p> <p>Erlang manages dynamic memory with a <a href="https://en.wikipedia.org/wiki/Tracing_garbage_collection" style="box-sizing: border-box; color: rgb(64, 120, 192); text-decoration: none; background-color: transparent;">tracing garbage collector</a>. More precisely a per process generational semi-space copying collector using Cheney&rsquo;s&nbsp;[1] copy collection algorithm together with a global large object space.</p> <h2>Overview</h2> <p>Each Erlang process has its own stack and heap which are allocated in the same memory block and grow towards each other. When the stack and the heap <a href="https://github.com/erlang/otp/blob/OTP-18.0/erts/emulator/beam/beam_emu.c#L387" style="box-sizing: border-box; color: rgb(64, 120, 192); text-decoration: none; background-color: transparent;">meet</a>, the garbage collector is triggered and memory is reclaimed. If not enough memory was reclaimed, the heap will grow.</p> <h3>Creating Data</h3> <p>Terms are created on the heap by evaluating expressions. There are two major types of terms; <a href="https://github.com/erlang/otp/blob/OTP-18.0/erts/emulator/beam/erl_term.h#L88-L97" style="box-sizing: border-box; color: rgb(64, 120, 192); text-decoration: none; background-color: transparent;">immediate terms</a> which require no heap space (small integers, atoms, pids, port ids etc) and cons or <a href="https://github.com/erlang/otp/blob/OTP-18.0/erts/emulator/beam/erl_term.h#L106-L120" style="box-sizing: border-box; color: rgb(64, 120, 192); text-decoration: none; background-color: transparent;">boxed terms</a> (tuple, big num, binaries etc) that do require heap space. Immediate terms do not need any heap space because they are embedded into the containing structure.</p> <p>Let&rsquo;s look at an example that returns a tuple with the newly created data.</p> <div class="highlight highlight-source-erlang" style="box-sizing: border-box; margin-bottom: 16px; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px;"> <pre> <span style="color:rgb(121, 93, 163)">data</span>(Foo) <span style="color:rgb(167, 29, 93)">-&gt;</span> Cons <span style="color:rgb(167, 29, 93)">=</span> [<span style="color:rgb(0, 134, 179)">42</span>|Foo], Literal <span style="color:rgb(167, 29, 93)">=</span> {<span style="color:rgb(0, 134, 179)">text</span>, <span style="color:rgb(24, 54, 145)">"hello world!"</span>}, {<span style="color:rgb(0, 134, 179)">tag</span>, Cons, Literal}.</pre> </div> <p>In this example we first create a new cons cell with an integer and a tuple with some text. Then a tuple of size three wrapping the other values with an atom tag is created and returned.</p> <p>On the heap tuples require a word size for each of its elements as well as for the header. Cons cells always require two words. Adding these things together, we get seven words for the tuples and 26 words for the cons cells. The string <code>"hello world!"</code> is a list of cons cells and thus requires 24 words. The atom <code>tag</code> and the integer <code>42</code> do not require any additional heap memory since it is an <em>immediate</em>. Adding all the terms together, the heap space required in this example should be 33 words.</p> <p>Compiling this code to beam assembly (<code>erlc +S</code>) shows exactly what is happening.</p> <div class="highlight highlight-source-erlang" style="box-sizing: border-box; margin-bottom: 16px; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px;"> <pre> ... {<span style="color:rgb(0, 134, 179)">test_heap</span>,<span style="color:rgb(0, 134, 179)">6</span>,<span style="color:rgb(0, 134, 179)">1</span>}. {<span style="color:rgb(0, 134, 179)">put_list</span>,{<span style="color:rgb(0, 134, 179)">integer</span>,<span style="color:rgb(0, 134, 179)">42</span>},{<span style="color:rgb(0, 134, 179)">x</span>,<span style="color:rgb(0, 134, 179)">0</span>},{<span style="color:rgb(0, 134, 179)">x</span>,<span style="color:rgb(0, 134, 179)">1</span>}}. {<span style="color:rgb(0, 134, 179)">put_tuple</span>,<span style="color:rgb(0, 134, 179)">3</span>,{<span style="color:rgb(0, 134, 179)">x</span>,<span style="color:rgb(0, 134, 179)">0</span>}}. {<span style="color:rgb(0, 134, 179)">put</span>,{<span style="color:rgb(0, 134, 179)">atom</span>,<span style="color:rgb(0, 134, 179)">tag</span>}}. {<span style="color:rgb(0, 134, 179)">put</span>,{<span style="color:rgb(0, 134, 179)">x</span>,<span style="color:rgb(0, 134, 179)">1</span>}}. {<span style="color:rgb(0, 134, 179)">put</span>,{<span style="color:rgb(0, 134, 179)">literal</span>,{<span style="color:rgb(0, 134, 179)">text</span>,<span style="color:rgb(24, 54, 145)">"hello world!"</span>}}}. <span style="color:rgb(0, 134, 179)">return</span>.</pre> </div> <p>Looking at the assembler code we can see three things; The heap requirement in this function turns out to be only six words, as seen by the <code>{test_heap,6,1}</code> instruction. All the allocations are combined to a single instruction. The bulk of the data <code>{text, "hello world!"}</code> is a <em>literal</em>. Literals, sometimes referred to as constants, are not allocated in the function since they are a part of the module and allocated at load time.</p> <p>If there is not enough space available on the heap to satisfy the <code>test_heap</code> instructions request for memory, then a garbage collection is initiated. It may happen immediately in the <code>test_heap</code> instruction, or it can be delayed until a later time depending on what state the process is in. If the garbage collection is delayed, any memory needed will be allocated in heap fragments. Heap fragments are extra memory blocks that are a part of the young heap, but are not allocated in the contigious area where terms normally recide. See The Young Heap for more details.</p> <h3>The Collector</h3> <p>Erlang has a copying semi-space garbage collector. This means that when doing a garbage collection, the terms are copied from one distinct area, called the from space, to a new clean area, called the to space. The collector starts by <a href="https://github.com/erlang/otp/blob/OTP-18.0/erts/emulator/beam/erl_gc.c#L1980" style="box-sizing: border-box; color: rgb(64, 120, 192); text-decoration: none; background-color: transparent;">scanning the root-set</a> (stack, registers, etc).</p> <p><img alt="gc-start" src="https://esl-website-production.s3.amazonaws.com/uploads/content_1.jpg" /></p> <p>It follows all the pointers from the root-set to the heap and copies each term word by word to the to space.</p> <p>After the header word has been copied a <a href="https://github.com/erlang/otp/blob/OTP-18.0/erts/emulator/beam/erl_gc.h#L45-L46" style="box-sizing: border-box; color: rgb(64, 120, 192); text-decoration: none; background-color: transparent;"><em>move marker</em></a> is destructively placed in it pointing to the term in the to space. Any other term that points to the already moved term will <a href="https://github.com/erlang/otp/blob/OTP-18.0/erts/emulator/beam/erl_gc.c#L1125" style="box-sizing: border-box; color: rgb(64, 120, 192); text-decoration: none; background-color: transparent;">see this move marker</a> and copy the referring pointer instead. For example, if the have the following Erlang code:</p> <div class="highlight highlight-source-erlang" style="box-sizing: border-box; margin-bottom: 16px; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px;"> <pre> <span style="color:rgb(121, 93, 163)">foo</span>(Arg) <span style="color:rgb(167, 29, 93)">-&gt;</span> T <span style="color:rgb(167, 29, 93)">=</span> {<span style="color:rgb(0, 134, 179)">test</span>, Arg}, {<span style="color:rgb(0, 134, 179)">wrapper</span>, T, T, T}.</pre> </div> <p>Only one copy of T exists on the heap and during the garbage collection only the first time T is encountered will it be copied.</p> <p><img alt="gc-rootset-scan" src="https://esl-website-production.s3.amazonaws.com/uploads/content_2.jpg" /></p> <p>After <a href="https://github.com/erlang/otp/blob/OTP-18.0/erts/emulator/beam/erl_gc.c#L1089" style="box-sizing: border-box; color: rgb(64, 120, 192); text-decoration: none; background-color: transparent;">all terms</a> referenced by the root-set have been copied, the collector scans the to space and copies all terms that these terms reference. When scanning, the collector steps through each term on the to space and any term still referencing the from space is copied over to the to space. Some terms contain non-term data (the payload of a on heap binary for instance). When encountered by the collector, these values are simply skipped.</p> <p><img alt="gc-heap-scan1" src="https://esl-website-production.s3.amazonaws.com/uploads/content_3.jpg" /></p> <p>Every term object we can reach is copied to the to space and stored on top of&nbsp;the <em>scan stop</em> line, and then the scan stop is moved to the end of the last object.</p> <p><img alt="gc-heap-stop.png" src="https://esl-website-production.s3.amazonaws.com/uploads/content_4.jpg" /></p> <p>When <em>scan stop</em> marker <a href="https://github.com/erlang/otp/blob/OTP-18.0/erts/emulator/beam/erl_gc.c#L1103" style="box-sizing: border-box; color: rgb(64, 120, 192); text-decoration: none; background-color: transparent;">catches up</a> to the <em>scan start</em> marker, the garbage collection is done. At this point we can <a href="https://github.com/erlang/otp/blob/OTP-18.0/erts/emulator/beam/erl_gc.c#L1206" style="box-sizing: border-box; color: rgb(64, 120, 192); text-decoration: none; background-color: transparent;">deallocate</a>&nbsp;the entire from space and therefore reclaim the entire young heap.</p> <h2>Generational Garbage Collection</h2> <p>In addition to the collection algorithm described above, the Erlang garbage collector also provides generational garbage collection. An additional heap, called the old heap, is used where the long lived data is stored. The original heap is called the young heap, or sometimes the allocation heap.</p> <p>With this in mind we can look at the Erlang&rsquo;s garbage collection again. During the copy stage anything that should be copied to the young to space is instead copied to the old to space <em>if</em> it is <a href="https://github.com/erlang/otp/blob/OTP-18.0/erts/emulator/beam/erl_gc.c#L1127" style="box-sizing: border-box; color: rgb(64, 120, 192); text-decoration: none; background-color: transparent;">below the <em>high-watermark</em></a>.</p> <p><img alt="gc-watermark.png" src="https://esl-website-production.s3.amazonaws.com/uploads/content_5.jpg" /></p> <p>The <a href="https://github.com/erlang/otp/blob/OTP-18.0/erts/emulator/beam/erl_process.h#L1021" style="box-sizing: border-box; color: rgb(64, 120, 192); text-decoration: none; background-color: transparent;"><em>high-watermark</em></a> is placed where the previous garbage collection ended and we have introduced a new area called the old heap. When doing the normal garbage collection pass, any term that is located below the high-watermark is copied to the old to space instead of the young.</p> <p><img alt="gc-watermark-2" src="https://esl-website-production.s3.amazonaws.com/uploads/content_6__1_.jpg" /></p> <p>In the next garbage collection, any pointers to the old heap will be ignored and not scanned. This way the garbage collector does not have to scan the long-lived terms.</p> <p>Generational garbage collection aims to increase performance at the expense of memory. This is achieved because only the young, smaller, heap is considered in most garbage collections.</p> <p>The generational hypothesis [2] predicts that most terms tend to die young, and for an immutable language such as Erlang, young terms die even faster than in other languages. So for most usage patterns the data in the new heap will die very soon after it is allocated. This is good because it limits the amount of data copied to the old heap and also because the garbage collection algorithm used is proportional to the amount of live data on the heap.</p> <p>One critical issue to note here is that any term on the young heap can reference terms on the old heap but <em>no</em> term on the old heap may refer to a term on the young heap. This is due to the nature of the copy algorithm. Anything referenced by an old heap term is not included in the reference tree, root-set and its followers, and hence is not copied. If it was, the data would be lost, fire and brimstone would rise to cover the earth. Fortunately, this comes naturally for Erlang because the terms are immutable and thus there can be no pointers modified on the old heap to point to the young heap.</p> <p>To reclaim data from the old heap, both young and old heaps are included during the collection and copied to a common to space. Both the from spaces of the young and old heap are then deallocated and the procedure will start over from the beginning. This type of garbage collection is called a full sweep and is triggered when the size of the area under the high-watermark is larger than the size of the free area of the old heap. It can also be triggered by doing a manual call to&nbsp;<a href="http://erlang.org/doc/man/erlang.html#garbage_collect-0" style="box-sizing: border-box; color: rgb(64, 120, 192); text-decoration: none; background-color: transparent;">erlang:garbage_collect()</a>, or by running into the young garbage collection limit set by <a href="http://erlang.org/doc/man/erlang.html#spawn_opt-4" style="box-sizing: border-box; color: rgb(64, 120, 192); text-decoration: none; background-color: transparent;">spawn_opt(fun(),[{fullsweep_after, N}])</a>&nbsp;where N is the number of young garbage collections to do before forcing a garbage collection of both young and old heap.</p> <h2>The Young Heap</h2> <p>The young heap, or the allocation heap, consists of the stack and heap as described in the Overview. However it also includes any heap fragments that are attached to the heap. All of the heap fragments are considered to be above the high-watermark and part of the young generation. Heap fragments contain terms that either did not fit on the heap, or were created by another process and then attached to the heap. For instance if the bif binary_to_term created a term which does not fit on the current heap without doing a garbage collection, it will create a heap-fragment for the term and then schedule a garbage collection for later. Also if a message is sent to the process, the payload may be placed in a heap-fragment and that fragment is added to young heap when the message is matched in a receive clause.</p> <p>This procedure differs from how it worked prior to Erlang/OTP 19.0. Before 19.0, only a contiguous memory block where the young heap and stack resided was considered to be part of the young heap. Heap fragments and messages were immediately copied into the young heap before they could be inspected by the Erlang program. The behaviour introduced in 19.0 is superior in many ways - most significantly it reduces the number of necessary copy operations and the root set for garbage collection.</p> <h2>Sizing the Heap</h2> <p>As mentioned in the Overview the size of the heap <a href="https://github.com/erlang/otp/blob/OTP-18.0/erts/emulator/beam/erl_gc.c#L247" style="box-sizing: border-box; color: rgb(64, 120, 192); text-decoration: none; background-color: transparent;">grows</a> to accommodate more data. Heaps grow in two stages, first <a href="http://github.com/erlang/otp/blob/OTP-18.0/erts/emulator/beam/erl_gc.c#L199-L208">a variation of Fibonacci sequence</a>&nbsp;is used starting at 233 words. Then at about 1 mega words the heap only <a href="https://github.com/erlang/otp/blob/OTP-18.0/erts/emulator/beam/erl_gc.c#L215-L227" style="box-sizing: border-box; color: rgb(64, 120, 192); text-decoration: none; background-color: transparent;">grows in 20% increments</a>.</p> <p>There are two occasions when the young heap grows:</p> <ul class="dotted"> <li>if the total size of the heap + message and heap fragments exceeds the current heap size.</li> <li>if after a fullsweep, the total amount of live objects is greater than 75%.</li> </ul> <p>There are two occasions when the young heap is shrunk:</p> <ul class="dotted"> <li>if after a young collection, the total amount of live objects is less than 25% of the heap and the young heap is &ldquo;big&rdquo;</li> <li>if after a fullsweep, the total amount of live objects is less than 25% of the heap.</li> </ul> <p>The old heap is always one step ahead in the heap growth stages than the young heap.</p> <h2>Literals</h2> <p>When garbage collecting a heap (young or old) all literals are left in place and not copied. To figure out if a term should be copied or not when doing a garbage collection the following pseudo code is used:</p> <div class="highlight highlight-source-c" style="box-sizing: border-box; margin-bottom: 16px; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px;"> <pre> <span style="color:rgb(167, 29, 93)">if</span> (is_literal(ptr) || (on_old_heap(ptr) &amp;&amp; !fullsweep)) { <span style="color:rgb(150, 152, 150)">/* literal or non fullsweep - do not copy */</span> } <span style="color:rgb(167, 29, 93)">else</span> { <span style="color:rgb(0, 134, 179)">copy</span>(ptr); }</pre> </div> <p>The is_literal check works differently on different architectures and operating systems.</p> <p>On 64 bit systems that allow mapping of unreserved virtual memory areas (most operating systems except Windows), an area of size 1 GB (by default) is mapped and then all literals are placed within that area. Then all that has to be done to determine if something is a literal or not is two quick pointer checks. This system relies on the fact that a memory page that has not been touched yet does not take any actual space. So even if 1 GB of virtual memory is mapped, only the memory which is actually needed for literals is allocated in ram. The size of the literal area is configurable through the +MIscs erts_alloc option.</p> <p>On 32 bit systems, there is not enough virtual memory space to allocate 1 GB for just literals, so instead small 256 KB sized literal regions are created on demand and a card mark bit-array of the entire 32 bit memory space is then used to determine if a term is a literal or not. Since the total memory space is only 32 bits, the card mark bit-array is only 256 words large. On a 64 bit system the same bit-array would have to be 1 tera words large, so this technique is only viable on 32 bit systems. Doing lookups in the array is a little more expensive then just doing the pointer checks that can be done in 64 bit systems, but not extremely so.</p> <p>On 64 bit windows, on which erts_alloc cannot do unreserved virtual memory mappings, a special tag within the Erlang term object is used to determine if something is a literal or not. This is very cheap, however the tag is only available on 64 bit machines, and it is possible to do a great deal of other nice optimizations with this tag in the future (like for instance a more compact list implementation) so it is not used on operating systems where it is not needed.</p> <p>This behaviour is different from how it worked prior to Erlang/OTP 19.0. Before 19.0 the literal check was done by checking if the pointer pointed to the young or old heap block. If it did not, then it was considered a literal. This lead to considerable overhead and strange memory usage scenarios, so it was removed in 19.0.</p> <h2>Binary Heap</h2> <p>The binary heap works as a large object space for binary terms that are greater than 64 bytes (from now on called off heap binaries). The binary heap is <a href="https://en.wikipedia.org/wiki/Reference_counting" style="box-sizing: border-box; color: rgb(64, 120, 192); text-decoration: none; background-color: transparent;">reference counted</a> and a pointer to the off-heap binary is stored on the process heap. To keep track of when to decrement the reference counter of the off heap binary, a linked list (the MSO - mark and sweep object list) containing funs and externals as well as off heap binaries is woven through the heap. After a garbage collection is done, the&nbsp;<a href="https://github.com/erlang/otp/blob/OTP-18.0/erts/emulator/beam/erl_gc.c#L2299" style="box-sizing: border-box; color: rgb(64, 120, 192); text-decoration: none; background-color: transparent;">MSO list is swept</a> and any off-heap binary that does not have a <a href="https://github.com/erlang/otp/blob/OTP-18.0/erts/emulator/beam/erl_gc.c#L2325" style="box-sizing: border-box; color: rgb(64, 120, 192); text-decoration: none; background-color: transparent;">move marker</a> written into the header words has its reference&nbsp;<a href="https://github.com/erlang/otp/blob/OTP-18.0/erts/emulator/beam/erl_gc.c#L2344-L2367" style="box-sizing: border-box; color: rgb(64, 120, 192); text-decoration: none; background-color: transparent;">decremented and is potentially freed</a>.</p> <p>All items in the MSO list are ordered by the time they were added to the process heap, so when doing a minor garbage collection, the MSO sweeper only has to sweep until it <a href="https://github.com/erlang/otp/blob/OTP-18.0/erts/emulator/beam/erl_gc.c#L2369" style="box-sizing: border-box; color: rgb(64, 120, 192); text-decoration: none; background-color: transparent;">encounters an off heap binary that is on the old heap</a>.</p> <h3>Virtual Binary Heap</h3> <p>Each process has a virtual binary heap associated with it that has the size of all the current off heap binaries that the process has references to. The virtual binary heap also has a limit and grows and shrinks depending on how off heap binaries are used by the process. The same growth and shrink mechanisms are used for the binary heap and for the term heap, so first a Fibonacci like series and then 20% growth.</p> <p>The virtual binary heap exists in order to <a href="https://github.com/erlang/otp/blob/OTP-18.0/erts/emulator/beam/beam_emu.c#L364" style="box-sizing: border-box; color: rgb(64, 120, 192); text-decoration: none; background-color: transparent;">trigger</a> garbage collections earlier when potentially there is a very large amount of off heap binary data that could be reclaimed. This approach does not catch all problems with binary memory not being released soon enough, but it does catch a lot of them.</p> <h2>Messages</h2> <p>Messages can become a part of the process heap at different times. This depends on how the process is configured. We can configure the behaviour of each process using <code>process_flag(message_queue_data, off_heap | on_heap | mixed)</code> or we can set a default for all processes at start using the option <code>+xmqd</code>.</p> <p>What do these different configurations do and when should we use them? Let&rsquo;s start by going through what happens when one Erlang process sends a message to another. The sending process needs to do a couple of things:</p> <p>&nbsp; &nbsp; &nbsp; 1. calculate <a href="https://github.com/erlang/otp/blob/OTP-18.0/erts/emulator/beam/erl_message.c#L1031" style="box-sizing: border-box; color: rgb(64, 120, 192); text-decoration: none; background-color: transparent;">how large</a> the message to be sent is</p> <p>&nbsp; &nbsp; &nbsp; 2.&nbsp;<a href="https://github.com/erlang/otp/blob/OTP-18.0/erts/emulator/beam/erl_message.c#L1033" style="box-sizing: border-box; color: rgb(64, 120, 192); text-decoration: none; background-color: transparent;">allocate enough space</a> to fit the entire message</p> <p>&nbsp; &nbsp; &nbsp; 3.&nbsp;<a href="https://github.com/erlang/otp/blob/OTP-18.0/erts/emulator/beam/erl_message.c#L1040" style="box-sizing: border-box; color: rgb(64, 120, 192); text-decoration: none; background-color: transparent;">copy</a> the message payload</p> <p>&nbsp; &nbsp; &nbsp; 4. allocate a message container with some meta data</p> <p>&nbsp; &nbsp; &nbsp; 5.&nbsp;<a href="https://github.com/erlang/otp/blob/OTP-18.0/erts/emulator/beam/erl_message.c#L502" style="box-sizing: border-box; color: rgb(64, 120, 192); text-decoration: none; background-color: transparent;">insert</a> the message container in the receiver process&rsquo; <a href="https://github.com/erlang/otp/blob/OTP-18.0/erts/emulator/beam/erl_process.h#L1042" style="box-sizing: border-box; color: rgb(64, 120, 192); text-decoration: none; background-color: transparent;">message queue</a></p> <p>&nbsp;</p> <p>The process flag <code>message_queue_data</code>, of the receiver process, controls the message allocating strategy of the sender process in step 2 and also how the message data is treaded by the garbage collector.</p> <p>The procedure above is different from how it worked prior to 19.0. Before 19.0 there was no configuration option, the behaviour was always very similar to how the <code>on_heap</code> option is in 19.0. This part of how the message queues can be configured is as of now quite new in master and may not be exactly the way it is released in 19.0. For instance there are discussions about removing the <code>mixed</code> option completely.</p> <h3>Message Allocating Strategies</h3> <p>If set to <code>mixed</code> or <code>on_heap</code>, the sending process will first attempt to allocate the space for the message directly on the young heap block of the receiving process. This is not always possible as it requires taking the <em>main lock</em> of the receiving process. The main lock is also held when the process is executing. The possibility for a lock conflict is thus likely in an intensely collaborating system. If the sending process cannot acquire the main lock, a heap fragment is instead created for the message and the message payload is copied onto that. With the <code>off_heap</code> option the sender process always creates heap fragments for messages sent to that process.</p> <p>What is the difference between <code>mixed</code> and <code>on_heap</code>?</p> <p>The difference is what happens on the receiving process side when it starts looking for through messages when garbage collecting. When using <code>on_heap</code>, any message in the internal message queue is considered to be part of the root-set when doing the garbage collection. When using <code>mixed</code>, messages that are in heap fragments will not be part of the root-set and thus not copied by the garbage collector.</p> <p>There are a bunch of different tradeoffs that come into play when trying to figure out which of the strategies you want to use.</p> <p>Using <code>off_heap</code> may seem like a nice way to get a more scalable system as you get very little contention on the main locks, however allocating a heap fragment is more expensive than allocating on the heap of the receiving process. So if it is very unlikely that contention will occur, it is more efficient to try to allocate the message directly on the receiving process&rsquo; heap.</p> <p>Using <code>on_heap</code> will force all messages to be part of on the young heap which will increase the amount of data that the garbage collector has to move. So if a garbage collection is triggered while processing a large amount of messages, they will be copied to the young heap. This in turn will lead to that the messages will quickly be promoted to the old heap and thus increase its&rsquo; size. This may be good or bad depending on exactly what the process does. A large old heap means that the young heap will also be larger, which in turn means that less garbage collections will be triggered while processing the message queue. This will temporarly increase the throughput of the process at the cost of more memory usage. However, if after all the messages have been consumed the process enters a state where a lot less messages are being received. Then it may be a long time before the next fullsweep garbage collection happens and the messages that are on the old heap will be there until that happens. So while <code>on_heap</code> is potentially faster than the other modes, it uses more memory for a longer time. This mode is the legacy mode which is almost how the message queue was handled before Erlang/OTP 19.0.</p> <p>Using <code>mixed</code> mode is a compromise between the two. It is potentially not as efficient as <code>on_heap</code> and may increase contention on the main lock of the receiving process when compared to <code>off_heap</code>. This option may be dropped before the release of Erlang/OTP 19.0, as it is not clear if it brings any benefit outside synthetic benchmarks.</p> <p>Which one of these strategies is best depends a lot on what the process is doing and how it interacts with other processes. So, as always, profile the application and see how it behaves with the different options. Also note that how these modes work may change before the release of Erlang/OTP 19.0.</p> <h2>Bibliography</h2> <p>&nbsp;</p> <p>[1]: C. J. Cheney. A nonrecursive list compacting algorithm. Commun. ACM, 13(11):677–678, Nov. 1970.</p> <p>[2]<!--2-->: D. Ungar. Generation scavenging: A non-disruptive high performance storage reclamation algorithm. SIGSOFT Softw. Eng. Notes, 9(3):157–167, Apr. 1984.</p> <hr> <p><strong>Learn more about how Erlang Solutions&rsquo; <a href="http://www2.erlang-solutions.com/l/23452/2017-05-23/4llxcl">Erlang and Elixir Solutions</a> or <a href="http://www2.erlang-solutions.com/emailpreference">sign up to our mailing list</a> to be the first to know about our future blog posts.</strong></p>

Permalink

Optimizing Your Elixir and Phoenix projects with ETS

Many Elixir programmers have probably heard references to "ets" or "ETS" in talks, or may have seen calls to the Erlang :ets module in code, but I would wager the majority haven't used this feature of Erlang in practice. Let's change that.

ETS, or Erlang Term Storage, is one of those innovative Erlang features that feels like it has been hiding in plain sight once you use it. Projects across the community like Phoenix.PubSub, Phoenix.Presence, and Elixir's own Registry take advantage of ETS, along with many internal Erlang and Elixir modules. Simply put, ETS is an in-memory store for Elixir and Erlang terms with fast access. Critically, it allows access to state outside of a process and message passing. Before we talk when to ETS (and when not to), let's begin with a basic primer by firing up iex:

First, we'll create an ETS table with :ets.new/2:

1
2
iex> tab = :ets.new(:my_table, [:set])
8211

That was easy enough. We created a table of type :set, which will allows us to map unique keys to values as a standard key-value storage. Let's insert a couple items:

1
2
3
4
iex> :ets.insert(tab, {:key1, "value1"})
true
iex> :ets.insert(tab, {:key2, "value1"})
true

Now with a couple items in place, let's do a lookup: elixir iex> :ets.lookup(tab, :key1) [key1: "value1"]

Notices how we aren't re-binding tab to a new value after inserting items. ETS tables are managed by the VM and their existence lives and dies by the process that created them. We can see this in action by killing the current iex shell process:

1
2
3
4
5
6
7
iex> Process.exit(self(), :kill)
** (EXIT from #PID<0.88.0>) killed

Interactive Elixir (1.4.4) - press Ctrl+C to exit (type h() ENTER for help)
iex> :ets.insert(8211, {:key1, "value1"})
** (ArgumentError) argument error
    (stdlib) :ets.insert(12307, {:key1, "value1"})

Once iex crashes, we lose our previous bindings, but we can pass the ets table ID returned from our call to :ets.new/2. We can see that when we tried to access the table after its owner crashed, an ArgumentError was thrown. This automatic cleanup of tables and data when the owning process crashes is one of ETS's great features. We don't have to be concerned about memory leaks when processes terminate after creating tables and inserting data. Here, we also got our first glimpse of the esoteric :ets API and its often unhelpful errors, such as ArgumentError with no other guidelines on what the problem may be. As you use the :ets module more and more, you'll undoubtably become famliar with frustrating argument errors and the :ets documentation is likely to end up in your bookmarks.

This just barely scratched the surface of what features ETS provides. We'll only be using a fraction of its capabilities, but just remember where it really shines is fast reads and writes to key-value storage, with the ability to efficiently match on most erlang terms stored within the table (excluding maps). In our examples, we'll only be storing simple key-values with basic lookups, but you should consult the docs to explore the breadth of provided features.

Optimizing GenServer access with an ETS table

Optimizing code is a rewarding experince, but doubly so when you don't have to change your public interface. Let's see a common way ETS is used to optimize data access for state wrapped in a GenServer.

Imagine we're writing a rate limiter GenServer which is a process in our app that counts user requests and allows us to deny access once a user exceeds their allotted requests per minute. We know right away that we'll need to store the request count state for our users somewhere, as well as a process that periodically sweeps the state once per minute. A naive first-pass with a plain-old GenServer might look something like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
defmodule RateLimiter do
  use GenServer
  require Logger

  @max_per_minute 5
  @sweep_after :timer.seconds(60)

  ## Client

  def start_link do
    GenServer.start_link(__MODULE__, [], name: __MODULE__)
  end

  def log(uid) do
    GenServer.call(__MODULE__, {:log, uid})
  end

  ## Server
  def init(_) do
    schedule_sweep()
    {:ok, %{requests: %{}}}
  end

  def handle_info(:sweep, state) do
    Logger.debug("Sweeping requests")
    schedule_sweep()
    {:noreply, %{state | requests: %{}}}
  end

  def handle_call({:log, uid}, _from, state) do
    case state.requests[uid] do
      count when is_nil(count) or count < @max_per_minute ->
        {:reply, :ok, put_in(state, [:requests, uid], (count || 0) + 1)}
      count when count >= @max_per_minute ->
        {:reply, {:error, :rate_limited}, state}
    end
  end

  defp schedule_sweep do
    Process.send_after(self(), :sweep, @sweep_after)
  end
end

First, we defined start_link/0 which starts a GenServer, using our RateLimiter module as the callback module. We also named the server as our module so we can reference it later in our call to log/1. Next, we defined a log/1 function which makes a synchronous call to the rate limiter server, asking it to log our user's request. We expect to receive either :ok back, to indicate our request can proceed, or {:error, :rate_limited}, to indicate the user has exceeded their allotted requests, and the request should not proceed.

Next, in init/1, we called a schedule_sweep/0 function which simply has the server send itself a message one per minute to clear out all request data. Then we defined a handle_info/2 clause to pickup the :sweep event and clear out the request state. To complete our implementation, we defined a handle_call/3 clause to track request state for the user and return an :ok, or {:error, :rate_limited} response for our caller in log/2.

Let's try it out in iex:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
iex> RateLimiter.start_link()
{:ok, #PID<0.126.0>}
iex> RateLimiter.log("user1")
:ok
iex> RateLimiter.log("user1")
:ok
iex> RateLimiter.log("user1")
:ok
iex> RateLimiter.log("user1")
:ok
iex> RateLimiter.log("user1")
:ok
iex> RateLimiter.log("user1")
{:error, :rate_limited}

13:55:44.803 [debug] Sweeping requests
iex(9)> RateLimiter.log("user1")
:ok

It works! Once our "user1" exceeded 5 requests/minute, the server returned the expected error response. Then after we waited we observed the debug output of the state sweep, and we confirmed we were no longer rate limited. Looks great right? Unfortunately, there's some serious performance issues in our implementation. Let's see why.

Since this feature is for rate limiting, all user requests must pass through this server. Since messages are processed in serial, this effectively limits our application to single-threaded performance, and creates a bottleneck on this single process.

ETS to the rescue

Fortunately for us, Erlangers solved these kinds of problems for us. We can refactor our rate limiter server to use a publicly accessible ETS table so clients can log their requests directly in ets, and our owning process can be responsible only for sweeping and cleaning up the table. This allows concurrent reads and writes against ETS without having to serialize calls through the single server. Let's make it happen:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
defmodule RateLimiter do
  use GenServer
  require Logger

  @max_per_minute 5
  @sweep_after :timer.seconds(60)
  @tab :rate_limiter_requests

  ## Client

  def start_link do
    GenServer.start_link(__MODULE__, [], name: __MODULE__)
  end

  def log(uid) do
    case :ets.update_counter(@tab, uid, {2, 1}, {uid, 0}) do
      count when count > @max_per_minute -> {:error, :rate_limited}
      _count -> :ok
    end
  end

  ## Server
  def init(_) do
    :ets.new(@tab, [:set, :named_table, :public, read_concurrency: true,
                                                 write_concurrency: true])
    schedule_sweep()
    {:ok, %{}}
  end

  def handle_info(:sweep, state) do
    Logger.debug("Sweeping requests")
    :ets.delete_all_objects(@tab)
    schedule_sweep()
    {:noreply, state}
  end

  defp schedule_sweep do
    Process.send_after(self(), :sweep, @sweep_after)
  end
end

First, we modified our init/1 function to create an ETS table with the :named_table and :public options so that callers outside of our process can access it. We also used the read_concurrency and write_concurrency options to optimize access. Next, we changed our log/1 function to write the request count to :ets directly, rather than going through the GenServer. This allows requests to concurrently track their own rate-limit usage. Here we used the update_counter/4 feature of ETS, which allows us to efficiently, and atomically, update a counter. After checking rate limit usage, we return the same value to the caller as before. Lastly, in our :sweep callback, we simply use :ets.delete_all_objects/1 to wipe the table for the next rate limit interval.

Let's try it out:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
iex> RateLimiter.start_link
{:ok, #PID<0.124.0>}
iex> RateLimiter.log("user1")
:ok
iex> RateLimiter.log("user1")
:ok
iex> RateLimiter.log("user1")
:ok
iex> RateLimiter.log("user1")
:ok
iex> RateLimiter.log("user1")
:ok
iex> RateLimiter.log("user1")
{:error, :rate_limited}
iex> :ets.tab2list(:rate_limiter_requests)
[{"user1", 7}]
iex> RateLimiter.log("user2")
:ok
iex> :ets.tab2list(:rate_limiter_requests)
[{"user2", 1}, {"user1", 7}]

14:27:19.082 [debug] Sweeping requests

iex> :ets.tab2list(:rate_limiter_requests)
[]
iex> RateLimiter.log("user1")
:ok

It works just as before. We also used :ets.tab2list/1 to spy on the data in the table. We can see our users requests are tracked properly, and the table is swept as expected.

That's all there to it. Our public interface remained unchanged and we vastly improve the performance of our mission-critical feature. Not bad!

Here Be Dragons

This just scratched the surface on what's possible with ETS. But before you get too carried away and extract out all your serialized state access from GenServer's and Agent's to ETS, you need to think carefully about which actions in your application are atomic, and which require serialized access. You can easily introduce race conditions by allowing concurrent reads and writes in the pursuit of performance. One of the beautiful things about Elixir's process model is the serial processing of messages. It lets us avoid race conditions exactly because we can serialize access to state that requires atomic operations. In the case of our rate limiter, each user wrote to ets with the atomic update_counter operation so concurrent writes are not a problem. Use the following rules to determine if you can move from serial access to ETS:

  1. The operations must be atomic. If clients are reading data from ets in one operation, then writing to ETS based on the result, you have a race condition and the fix is serial access in a server

  2. If the operations are not atomic, ensure different processes write to distinct keys. For example, Elixir's registry uses the Pid as key and allows only the current process to change its own entry. This guarantees there are no races as each process work on a different row of the ETS table

  3. If none of the above apply, consider keeping the operation serial. Concurrent reads with serial writes is a common ETS pattern

If you're curious about using ETS for a dependency-free in-memory cache, check out Saša Jurić's excellent ConCache library.

Permalink

Functional Geekery Episode 95 – Dan Friedman

In this episode I talk with Dan Friedman. Dan gives shares stories of his history working with Lisp, including topics such as the power of Lisp, and tips for introducing a language.

Our Guest, Dan Friedman

Dan’s Faculty Page

Conference Announcements

Elm Europe will be taking place June 8th and 9th in Paris, France. Visit http://elmeurope.org/ for more information and to register.

Erlang User Conference 2017 will be taking place June 8th and 9th in Stockholm, Sweden. Visit http://www.erlang-factory.com/euc2017 for more information and to register.

ZuriHac 2017 will be taking place in Zurich on the 9th-11th of June. For more information, and to register visit https://zurihac.info/

Oslo Elm Day is a one-day conference about Elm taking place June 10th in Oslo, Norway. Visit https://osloelmday.no/ for more information and to register.

Curry On Barcelona will be taking place June 19-20th. For more information, visit http://www.curry-on.org/2017/.

O’Reilly Fluent Conference will be taking place June 19–22, 2017 in San Jose, California. Visit http://www.oreilly.com/pub/cpc/61309 to find out more and to register, and use code USRG for 20% off.

EuroClojure will be taking place in Berlin, Germany on July 20th & 21st. Visit http://2017.euroclojure.org/ for more information and to keep updated.

BusConf will be taking place the 3rd-5th of August in Frankfurt, Germany. Registration is open, and more information can be found at http://www.bus-conf.org/.

The Strange Loop coming! It will be held in St. Louis, MO on September 28-30, 2017 at the Peabody Opera House. To submit your CfP, visit http://thestrangeloop.com/.

Open FSharp will be taking place the 28th-29th of September in San Fransisco, California. Visit openfsharp.org for more information and to register.

LambdaWorld will be taking place in Cadiz, Spain on October 26th and 27th. For more information visit and to keep updated visit http://www.lambda.world/.

If you have a conference related to functional programming, contact me, and I will be happy to announce it.

Announcements

Some of you have asked how you can support Functional Geekery, in that vein,
Functional Geekery now has a Patreon Page.

If that is one of the ways you would like to show your support, you can
find out more at https://www.patreon.com/fngeekery.

Topics [@5:03]

About Dan
Dan’s early entry into teaching
Elliott Organick
MAD (Michigan Algorithmic Decoder) language
Multics
LISP 1.5 Programmer’s Manual
Indiana University
The Little Lisper origins
Selectric
Smith Corona
PDP-11
Ben Schneiderman
Getting Tenure
David Wise
CONS should not Evaluate its Arguments
Mitch Wand
Essentials of Programming Languages
Kanren
Oleg Kiselyov
Grasp
Bill Cohagan
Grope
Bob Baron
IBM 360
What about Lisp that appeals now and then to Dan
Brian Smith
Planner
Carl Hewitt
Eugene Charniak
Gerald Sussman
Conniver
Scheme Report (1975)
Higher Order Functions
Coda language
Bob Filman
William Byrd
Simplicity as the over arching model
miniKanren
Prolog
Dan’s 60th Birthday Celebration
Chung-chieh Shan
The Reasoned Schemer
Richard Salter
Implementing languages from scratch in a couple of evenings
Dick Kieburtz
Marigold
Power of macros in Lisp
Matthias Felleisen
Building an Operating System in a week
Racket
Jason Hemann
µKanren
Advice and tips for teaching
Convince students in first two lectures they will be changed
2 or 3 cases max when dealing with recursion
Ackermann function
Carl Eastlund
The Little Prover
David Christiansen
“Little Book” in the works on dependent types
Pie language

As always, a giant Thank You goes to David Belcher for the logo design.

Permalink

Elixir's Secret Weapon

I recently began using a new(ish) feature of Elixir that completely transformed the way I build programs. I'm talking about the special form with. It can feel unfamiliar at first, but it is extremely powerful and flexible. This article will explain how with works and how it can be used to make your code more robust to errors. First, though, let's look at the problem with is trying to solve.

Tagged Tuples

The Elixir community has borrowed an idiom from the Erlang community called tagged tuples. Using this approach, our functions will return a tuple where the first value is an atom describing the "type" of the result. A common pattern is returning {:ok, value} for a successful response and {:error, some_error} for an error response.

defmodule Math do  
  def divide(_, b) when b == 0, do: {:error, "divide by zero"}
  def divide(a, b), do: {:ok, a / b}
end

case Math.divide(1, 2) do  
  {:ok, result} -> "success: #{result}"
  {:error, error} -> "error: #{error}"
end  
# => "success: 0.5"

This is nice for a few reasons. First, it allows us to treat errors as values. We know that our function will always return even when provided semantically invalid data. Second, we can use pattern matching to act explicitly on both the success and failure case.

This patterns starts to break down when we want to perform multiple, dependent actions that can all fail. Suppose, for example, that we'd like to divide two numbers and, if successful, divide the result by a third number.

case Math.divide(1, 2) do  
  {:ok, result} ->
    case Math.divide(result, 4) do
      {:ok, result} -> "success: #{result}"
      {:error, error} -> "error: #{error}"
    end
  {:error, error} -> "error: #{error}"
end  
# => "success: 0.125"

This works, sure, but it's becoming challenging to read. You can easily imagine the mess this becomes with an arbitrarily long list of dependent operations.

What should we do? We want to continue using this pattern because it is safe and explicit but it feels ugly and unreadable with real-world examples. The answer, of course, is to use with.

Using with

The with special form allows you to define a set of operations to perform and associated patterns to match their results against. Each operation can use bindings from the pattern match of the previous operations. If any of the matches fail, the entire with form stops and that non-matching result is returned. Let's explore by rewriting the above example using with.

with {:ok, a} <- Math.divide(1, 2),  
     {:ok, b} <- Math.divide(a, 4) do
  "success: #{b}"
end  
# => "success: 0.125"

Let's walk through the execution. First, we call Math.divide(1, 2). The result is {:ok, 0.5}. The with form checks to see if this matches the pattern on the lefthand side of <-. It does, so the variable a is bound to 0.5 and execution continues. On the second line, we run Math.divide(0.5, 4) (because a is now bound to 0.5). This returns {:ok, 0.125}. We check if it matches the pattern on the lefthand side of its <-. It does, so b is bound to 0.125. There are no more operations to perform, so the body of the do block is executed. This do block can use any of the bindings from the with operations above. It uses b to return the string "success: 0.125".

Now, let's try walking through an error case.

with {:ok, a} <- Math.divide(1, 0),  
     {:ok, b} <- Math.divide(a, 4) do
  "success: #{b}"
end  
# => {:error, "divide by zero"}

First, we call Math.divide(1, 0). The result is {:error, "divide by zero"}. We check to see if this matches the patter on the left of <-. It doesn't! As soon as this mismatch occurs, the with form immediately stops executing further operations and returns the result that did not match. Therefore, the return value of the form is {:error, "divide by zero"}.

Better Error Handling

We're already in a better spot by using with but we can go even further. The with form allows us to describe an else clause that handles any non-matching values rather than simply returning them. Let's add one.

with {:ok, a} <- Math.divide(1, 0),  
     {:ok, b} <- Math.divide(a, 4) do
  "success: #{b}"
else  
  {:error, error} -> "error: #{error}"
end  
# => "error: divide by zero"

Now, when the first operation returns {:error, "divide by zero"} and it doesn't match the pattern, the value is passed to the else block. Next, we check each clause of the else block in order (in this example there is only one clause). The first clause matches {:error, "divide by zero"}, so the string "divide by zero" is bound to error. Finally, the body of that clause is executed with the bindings from the match. It returns the string "error: divide by zero".

More

The with form allows even more flexibility, including guard clauses in each of the operations. For more examples see the docs.

Permalink

Lessons Learned Building in Next.js

Part 1: Introduction and Initial Gotchas

Last Updated: 5/9/2017

The theme for this series will be crabs! For reasons!

Current Versions:

  • React: v15.4
  • next.js: v2.3

Introduction

next.js is a server-side rendering framework for React apps (and it’s pretty fast, too)! It also offers a slightly more opinionated approach to building your React apps by asking you to stick to SOME slight conventions regarding your directory structure and how you structure your components. While this may sound scary at first, this actually ends up being a big help to get next.jsyour application structured a little bit more soundly.

So how does it differ from create-react-app? In my mind, it is best described via a bit of a Venn diagram. There’s some overlap and some areas where one surpasses (or at least focuses more) the other. create-react-app feels like it has a slightly nicer developer experience: a lot of the testing setup is already ready to go and doesn’t require much tinkering. In addition, you can add redux to it or do dynamic routes with react-router with no extra effort, which is definitely a pretty nice bonus. However, if you want to do any neat stuff with server-side rendering, you’re on your own, and changing the pipelines to include SASS or other post-css-style plugins get a lot rougher. next.js is more on the spectrum of giving you a good framework, a good foundation, and getting your app a little more love on the server side of things.

Before I go any further, however, there is one thing I’d like to note: this is a deviation from my usual style of posts that focus deeply on tutorial-style lessons. This is instead a little bit more geared towards the things I learned in the process of converting a side project from create-react-app into next.js. Depending on the various levels of interest I may venture I a little more into a tutorial covering more bits of this approach (as well as the API side of things via Elixir/Phoenix).

As of a few days ago, I shipped my first approach to a next.js application, and one that I feel will be a precursor for others as well, so the tl;dr for this post is that next.js is pretty good, and I’m pretty happy with it overall! That being said, I ran into some issues during the process of converting my full create-react-app over, and I thought it might be nice to document them out should someone else run into similar scenarios!

Also, before I proceed any further, I want to give a huge shout-out to the create-react-app team and Zeit/the next.js team. These are some fantastic tools that significantly increase the quality of life for all of us! Thanks for all the hard work you’ve all poured into these products, and please know that we appreciate and respect you and all that you’ve done! You all rock!

Initial Gotchas

So one thing that is worth mentioning before proceeding at all is that you can’t just straight up copy and paste your create-react-app code into next.js and hope for the best. There aren’t a ton of differences between the two but there enough that you can’t do a straight-up 1:1 translation.

Project Structure

The first little gotcha (or at least, the first thing I needed to acclimate to) was next.js’ intended project structure. At a very minimum level, you should be splitting your app apart into two separate directories: pages and components.

pages are what you’d think of as the actual pages that your users can navigate to, and should be composed from components inside of your components directory. This also informs next.js about the URL and routing structure of your application through the names of files. So, if under pages, you had a few files: index.js, store.js, and about.js, next.js would assume your route structure was, by default, [yoursite] (the index.js acts as the index or base route for your app), [yoursite]/store, and [yoursite]/about. These are really the only major decisions that next.js will inform about your project structure; beyond this it is dealer’s choice.

Uh, this thing is sort of a screen, so I’ll create a screens/ directory, and this thing is a container, so I’ll create containers/, and this is stuff related to the Redux store, so I’ll create store/

This gets very unsustainable very quickly. Trust me. If you’re looking for an alternative structure, you can also check out Redux ducks!

I personally structured my application into these directories:

  • pages — The top-level components and routes for my app
  • components — The smaller components used to build the larger pages
  • services — Any of the library/utility JS functions that provided the layer to communicate with my API
  • __tests__ — This is more Jest-specific, but this is where my tests went for my app. From there, the directory structure mirrored the root of my next.js project, so I had: __tests__/pages, __tests__/components, etc.

This is more in-depth than what you’ll get or have suggested as part of building an app in create-react-app, so it took a little bit of restructuring of my original app to make it all work. That being said, I like it; it forced me to rethink my structure a bit and focus more composability.

Redux

Redux was a slightly more interesting case and one that originally caused me to drop my excursions into converting my app from create-react-app to next.js. Specifically, I had a number of container components that embedded Redux via react-redux’s connect function.

next.js has a slightly different idea. Instead of using connect(mapStateToProps, mapDispatchToProps)(Component), you can instead use a library (next-redux-wrapper) that provides a withRedux(store, mapStateToProps, mapDispatchToProps)(Component) function instead. Instead of picking up the store through context and higher-order components like Provider, you’re using a function and hooking the store up to each page individually. The idea is not to bundle the redux store and functionality for your components, as this again encourages bad design patterns and bad habits.

This component needs something from the store, but it’s a lot of work to pass that property down from the parent here, so screw it, I’ll just hook it up to the store with connect!

Yeah, don’t do this. You can get into some wacky scenarios and you’re making your tests harder to write (and your life more difficult in the process)!

My original concern with this approach was two-fold.

I didn’t want to get locked into framework-specific knowledge that wouldn’t be applicable elsewhere. This is still a concern of mine, for what it’s worth, but I decided that the things I had to learn to make my app work weren’t significant enough departures from standard React knowledge that I’d be “specializing” in a single framework.

I’d rather be a React developer than a next.js developer!

You could just be both, or you could just be a javascript developer! Both are great options. You do you.

My app was built in a pretty inefficient way that originally had me attempting to jump through hoops to still be able to use withRedux with smaller components (not full pages). Again, I was clinging to the original way I was attempting to architect my app, which was not necessarily the best option. I’ve instead modified my components and pages to pass state down when appropriate, and my tests for my components are about a million times easier. Yes, it gets annoying passing props down three, four, even five levels, but it gives a level of predictability that I appreciate.

Conclusion

We’ve already covered a lot of ground, and this is only about 1/3rd of the outline I put together for this series, so we’ll wrap up here and keep this short and sweet. Ultimately, next.js has some really cool aspects of it, and I hope that this series of posts can help you tackle adopting next.js with a little more certainty and information! Stay tuned for the next couple of posts where we’ll discuss AJAX, Custom Routing, and Deployment (plus more)!

Next Post in this Series

Lessons Learned Building in Next.js

Permalink

Copyright © 2016, Planet Erlang. No rights reserved.
Planet Erlang is maintained by Proctor.