Planet = erlang.

Strong arrows: a new approach to gradual typing

This is article expands on the topic of gradual set-theoretic typing discussed during my keynote at ElixirConf US 2023.

There is an on-going effort to research and develop a type system for Elixir, lead by Giuseppe Castagna, CNRS Senior Researcher, and taken by Guillaume Duboc as part of his PhD studies.

In this article, we will discuss how the proposed type system will tackle gradual typing and how it relates to set-theoretic types, with the goal of providing an introduction to the ideas presented in our paper.

Set-theoretic types

The type system we are currently researching and developing for Elixir is based on set-theoretic types, which is to say its operations are based on the fundamental set operations of union, intersection, and negation.

For example, the atom :ok is a value in Elixir, that can be represented by the type :ok. All atoms in Elixir as represented by themselves in the type system. A function that returns either :ok or :error is said to return :ok or :error, where the or operator represents the union.

The types :ok and :error are contained by the type atom(), which is an infinite set representing all atoms. The union of the types :ok and atom() can be written as :ok or atom(), and is equivalent to atom() (as :ok is a subset of atom()). The intersection of the types :ok and atom() can be written as :ok and atom(), and is equivalent to :ok.

Similarly, integer() is another infinite set representing all integers. integer() or atom() is the union of all integers and atoms. The intersection integer() and atom() is an empty set, which we call none(). The union of all types that exist in Elixir is called term().

The beauty of set-theoretic types is that we can model many interesting properties found in Elixir programs on top of those fundamental set operations, which in turn we hope to make typing in Elixir both more expressive and accessible. Let’s see an example of how a type system feature, called bounded quantification (or bounded polymorphism), can be implemented with set-theoretic types.

Upper and lower bounds

The identity function is a function that receives an argument and returns it as is. In Java, it would be written as follows:

static <T> T identity(T arg) {
    return arg;
}

In TypeScript:

function identity<T>(arg: T): T {
  return arg;
}

Or in Haskell:

id :: a -> a
id arg = arg

In all of the examples above, we say the function receives an argument of type variable T (or type variable a in Haskell’s case) and return a value of the same type T. We call this parametric polymorphism, because the function parameter - its argument - can take many (poly) shapes (morphs). In Elixir, we could then support:

$ a -> a
def identity(arg), do: arg

Sometimes we may want to further constrain those type variables. As example, let’s constraint the identity function in Java to numbers:

static <T extends Number> T identity(T arg) {
    return arg;
}

Or in TypeScript:

function identity<T extends number>(arg: T): T {
    return arg;
}

In Haskell, we can constrain to a typeclass, such as Ord:

id :: Ord a => a -> a
id x = x

In order words, these functions can accept any type as long as they fullfil a given constraint. This in turn is called bounded polymorphism, because we are putting bounds on the types we can receive.

With all that said, how can we implement bounded polymorphism in set-theoretic types? Imagine we have a type variable a, how can we ensure it is bounded or constrained to another type?

With set-theoretic types, this operation is an intersection. If you have a and atom(), a can be the type :foo. a can also be the type atom(), which represents all atom types, but a cannot be integer(), as integer() and atom() will return an empty set. In other words, there is no need to introduce a new semantic construct, as intersections can be used to place upper bounds in type variables! Therefore, we could restrict Elixir’s identity function to numbers like this:

$ a and number() -> a and number()
def identity(arg), do: arg

Or course, we can provide syntax sugar for those constraints:

$ a -> a when a: number()
def identity(arg), do: arg

But at the end of the day it will simply expand to intersections. The important bit is that, at the semantic level, there is no need for additional constructs and representations.

Note: for the type-curious readers, set-theoretic types implement a limited form of bounded quantification à la Kernel Fun. In a nutshell, it means we can only compare functions if they have the same bounds. For example, our type system states a -> a when a: integer() or boolean() is not a subtype of a -> a when a: integer().

We also get lower bounds for free. If intersections allow us to place an upper bound on a type variable, a union is equivalent to a lower bound as it specifies the type variable will always be augmented by the union-ed type. For example, a or atom() says the result will always include atoms plus whatever else specified by a (which may be an atom, atom() itself, or a completely disjoint type such as integer()).

Elixir protocols, which is an Elixir construct equivalent to Haskell Typeclasses or Java interfaces, is another example of functionality that can be modelled and composed with set-theoretic types without additional semantics. The exact mechanism to do so is left as an exercise to the reader (or the topic of a future blog post).

Enter gradual typing

Elixir is a functional dynamic programming language. Existing Elixir programs are untyped, which means that a type system needs mechanisms to interface existing Elixir code with future statically typed Elixir code. We can achieve this with gradual typing.

A gradual type system is a type system that defines a dynamic() type. It is sometimes written as ? and sometimes known as the any type (but I prefer to avoid any because it is too short and too lax in languages like TypeScript).

In Elixir, the dynamic() type means the type is only known at runtime, effectively disabling static checks for that type. More interestingly, we can also place upper and lower bounds on the dynamic type using set operations. As we will soon learn, this will reveal interesting properties about our type system.

It is often said that gradual typing is the best of both words. Perhaps ironically, that’s true and false at the same time. If you use a gradual type system but you never use the dynamic() type, then it behaves exactly like a static type system. However, the more you use the dynamic() type, the fewer guarantees the type system will give you, the more the dynamic() type propagates through the system. Therefore, it is in our interest to reduce the occurrences of the dynamic() type as much as possible, and that’s what we set out to do.

Interfacing static and dynamic code: the trouble with dynamic()

Let’s go back to our constrained identity function that accepts only numbers:

$ a -> a when a: number()
def identity(arg), do: arg

Now imagine that we have some untyped code that calls this function:

def debug(arg) do
  "we got: " <> identity(arg)
end

Since debug/1 is untyped, its argument will receive the type dynamic().

debug/1 proceeds to call identity with an argument and then uses the string concatenation operator (<>) to concatenate "we got: " to the result of identity(arg). Since identity/1 is meant to return a number and string concatenation requires two strings as operands, there is a typing error in this program. On the other hand, if you call debug("hello") at runtime, the code will work and won’t raise any exceptions.

In other words, the static typing version of the program and its runtime execution do not match in behaviour. So how do we tackle this?

One option is to say that’s all behaving as expected. If debug/1 is untyped, its arg has the dynamic() type. To type check this program, we specify that identity(dynamic()) returns the dynamic() type, the concatenation of a string with dynamic() also returns dynamic(), and consequently debug/1 gets the type dynamic() -> dynamic(), with no type errors emitted.

The trouble is: this is not a very useful choice. Once dynamic() enters the system, it spreads everywhere, we perform fewer checks, effectively discarding the information that identity/1 returns a number, and the overall type system becomes less useful.

Another option would be for us to say: once we call a statically typed function with dynamic(), we will ignore the dynamic() type. If the function says it returns a number(), then it will surely be a number! In this version, identity(dynamic()) returns number() and the type system will catch a type error when concatenating a string with a number.

This is similar to the approach taken by TypeScript. This means we can perform further static checks, but it also means we can call debug("foobar") and that will return the string "we got: foobar"! But how can that be possible when the type system told us that identity returns a number()? This can lead to confusion and surprising results at runtime. We say this system is unsound, because the types at runtime do not match our compile-time types.

None of our solutions so far attempted to match the static and runtime behaviors, but rather, they picked one in favor of the other.

But don’t despair, there is yet another option. We could introduce runtime checks whenever we cross the “dynamic <-> static” boundaries. In this case, we could say identity(dynamic()) returns a number(), but we will introduce a runtime check into the code to guarantee that’s the case. This means we get static checks, we ensure the value is correct at runtime, with the cost of introducing additional checks at runtime. Unfortunately, those checks may affect performance, depending on the complexity of the data structure and on how many times we cross the “dynamic <-> static” boundary.

Note: there is recent research in using the runtime checks introduced by a gradual type system to provide compiler optimizations. Some of these techniques are already leveraged by the Erlang VM to optimize code based on patterns and guards.

To summarize, we have three options:

• Calling static code from dynamic code returns dynamic(), dropping the opportunity of further static typing checks (this is sound)

• Calling static code from dynamic code returns the static types, potentially leading to mismatched types at runtime (this is unsound)

• Calling static code from dynamic code returns the static types with additional runtime checks, unifying both behaviours but potentially impacting performance (this is sound)

Introducing strong arrows

I have always said that Elixir, thanks to Erlang, is an assertive language. For example, if our identity function is restricted to only numbers, in practice we would most likely write it as:

$ a -> a when a: number()
def identity(arg) when is_number(arg), do: arg

In the example above, identity will fail if given any value that is not a number. We often rely on pattern matching and guards and, in turn, they helps us assert on the types we are working with. Not only that, Erlang’s JIT compiler already relies on this information to perform optimizations whenever possible.

We also say Elixir is strongly typed because its functions and operators avoid implicit type conversions. The following functions also fail when their input does not match their type:

$ binary() -> binary()
def debug(string), do: "we got: " <> string

$ (integer() -> integer()) and (float() -> float())
def increment(number), do: number + 1

<> only accepts binaries as arguments and will raise otherwise. + only accepts numbers (integers or floats) and will raise otherwise. + does not perform implicit conversions of non-numeric types, such as strings to number, as we can see next:

iex(1)> increment(1)
2
iex(2)> increment(13.0)
14.0
iex(3)> increment("foobar")
** (ArithmeticError) bad argument in arithmetic expression: "foobar" + 1

In other words, Elixir’s runtime consistently checks the values and their types at runtime. If increment fails when given something else than a number, then it will fail when the dynamic() type does not match its input at runtime. This guarantees increment returns its declared type and therefore we do not need to introduce runtime type checks when calling said function from untyped code.

When we look at the identity, debug, and increment functions above, we - as developers - can state that these functions raise when given a value that does not match their input. However, how can we generalize this property so it is computed by the type system itself? To do so, we introduce a new concept called strong arrows, which relies on set-theoretical types to derive this property.

The idea goes as follows: a strong arrow is a function that can be statically proven that, when evaluated on values outside of its input types (i.e. its domain), it will error. For example, in our increment function, if we pass a string() as argument, it won’t type check, because string() + integer() is not a valid operation. Thanks to set-theoretic types, we can compute all values outside of the domain by computing the negation of a set. Given increment/1 will fail for all types which are not number(), the function is strong.

By applying this rule to all typed functions, we will know which functions are strong and which ones are not. If a function is strong, the type system knows that calling it with a dynamic() type will always evaluate to its return type! Therefore we say the return type of increment(dynamic()) is number(), which is sound and does not need further runtime checks!

Going back to our debug function, when used with a guarded identity, it will be able emit warnings at compile-time, errors at runtime, without introducing any additional runtime check:

$ a -> a when a: number()
def identity(arg) when is_number(arg), do: arg

def debug(arg) do
  "we got: " <> identity(arg)
end

However, if the identity function is not strong, then we must fallback to one of the strategies in the previous section.

Another powerful property of strong arrows is that they are composable. Let’s pick an example from the paper:

$ number(), number() -> number()
def subtract(a, b) do
  a + negate(b)
end

$ number() -> number()
def negate(int), do: -int

In the example above, negate/1’s type is a strong arrow, as it raises for any input outside of its domain. subtract/2’s type is also a strong arrow, because both + and our own negate are strong arrows too. This is an important capability as it limits how dynamic() types spread throughout the system.

Errata: my presentation used the type integer() instead of number() for the example above. However, that was a mistake in the slide. Giving the type integer(), integer() -> integer() to subtract and integer() -> integer() to negate does not make subtract a strong arrow. Can you tell why?

Luckily, other gradually typed languages can also leverage strong arrows. However, the more polymorphic a language and its functions are, the more unlikely it is to conclude that a given function is strong. For example, in other gradually typed languages such as Python or Ruby, the + operator is extensible and the user can define custom types where the operation is valid. In TypeScript, "foobar" + 1 is also a valid operation, which expands the function domain. In both cases, an increment function restricted to numbers would not have a strong arrow type, as the operator won’t fail for all types outside of number(). Therefore, to remain sound, they must either restrict the operands with further runtime checks or return dynamic() (reducing the number of compile-time checks).

There is one last scenario to consider, which I did not include during my keynote for brevity. Take this function:

$ integer() -> :ok
def receives_integer_and_returns_ok(_arg), do: :ok

The function above can receive any type and return :ok. Is its type a strong arrow? Well, according to our definition, it is not. If we negate its input, type checking does not fail, it returns :ok.

However, given the return type is always the same, it should be a strong arrow! To do so, let’s amend and rephrase our definition of strong arrows: we negate the domain (i.e. the inputs) of a function and then type check it. If the function returns none() (i.e. it does not type check) or a type which is a subset of its codomain (i.e. its output), then it is a strong arrow.

Gradual typing and false positives

There is one last scenario we must take into consideration when interfacing dynamic and static code. Imagine the following code:

def increment_and_remainder(numerator, denominator) do
  rem(numerator, increment(denominator))
end

$ (integer() -> integer()) and (float() -> float())
def increment(number), do: number + 1

The increment_and_remainder/2 function is untyped, therefore both of its arguments receive type dynamic(). The function then computes the remainder of the numerator by the denominator incremented by one. For this example, let’s assume all uses of increment_and_remainder/2 in our program passes two integers as arguments.

Given increment/1 has a strong arrow type, according to our definition, increment(dynamic()) will return integer() or float() (also known as number()). Here lies the issue: if increment(dynamic()) returns integer() or float(), the program above won’t type check because rem/2 does not accept floats.

When faced with this problem, there are two possible reactions:

1. It is correct for the function to not type check given increment may return a float

2. It is incorrect for the function to not type check because the error it describes never occurs in the codebase

Another interesting property of gradual set-theoretic types is that we can also place upper bounds on the dynamic() type. If a function returns number(), it means the caller needs to handle both integer() and float(). However, if a function returns dynamic() and number(), it means the type is defined at runtime, but it must still verify it is one of integer() or float() at compile time.

Therefore, rem/2 will type check if its second argument has the type dynamic() and number(), as there is one type at runtime (integer()) that satisfies type checking. On the other hand, if you attempt to use the string concatenation operator (<>) on dynamic() and number(), then there is no acceptable runtime type and you’d still get a typing violation!

Going back to strong arrows, there are two possible return types from a strong arrow:

1. A strong arrow, when presented with a dynamic type, returns its codomain

2. A strong arrow, when presented with a dynamic type, returns the intersection of the codomain with the dynamic() type

The second option opens up the possibility for existing codebases to gradually migrate to static types without dealing with false positives. Coming from a dynamic background, false positives can be seen as noisy or as an indication that static types are not worth the trouble. With strong arrows and gradual set-theoretic types, we will be able to explore different trade-offs on mixed codebases. Which of the two choices above we will adopt as a default and how to customize them is yet to be decided. It will depend on the community feedback as we experiment and integrate the type system.

Erlang and Elixir developers who use Dialyzer will be familiar with these trade-offs, as the second option mirrors Dialyzer’s behaviour of no false positives. The difference here is that our semantics are integrated into a complete type system. If no type signature is present, the dynamic() type is used, and we will leverage the techniques described here to interface dynamic and static code. If a function has a type signature, and no dynamic() type is present, then it will behave as statically typed code when called with statically typed arguments. Migrating to static types will naturally reduce the interaction points between dynamic and static code, removing the reliance on the dynamic() type.

Summary

Set-theoretic types allow us to express many typing features based on set operations of union, intersection, and negation.

In particular, we have been exploring a gradual set-theoretic type system for Elixir, paying special attention to how the type system will integrate with existing codebases and how it can best leverage the semantics of the Erlang Virtual Machine. The type system will also perform limited inference based on patterns and guards (as described in the paper), which - in addition to strong arrows - we hope to bring some of the benefits of static typing to codebases without changing a single line of code.

While our efforts have officially moved from research into development, and we have outlined an implementation plan, we haven’t yet fully implemented nor assessed the usability of set-theoretic types in existing Elixir codebases, nor large nor small. There is much to implement and validate, and we don’t rule the possibility of finding unforeseen deal breakers that could send us back to square one. Yet we are pleased and cautiously excited with the new developments so far.

The development of Elixir’s type system is sponsored by Fresha (they are hiring!), Starfish* (they are hiring!), and Dashbit.

Permalink

Erlang/OTP 26.1 Release

OTP 26.1

Erlang/OTP 26.1 is the first maintenance patch package for OTP 26, with mostly bug fixes as well as improvements.

For details about bugfixes and potential incompatibilities see the Erlang 26.1 README

The Erlang/OTP source can also be found at GitHub on the official Erlang repository, https://github.com/erlang/otp

Download links for this and previous versions are found here

• https://www.erlang.org/downloads

Permalink

An Introduction to RabbitMQ | Erlang Solutions

Permalink

What Should FinTech Learn From Telecom? Portfolio Conference, Banking Technology 2021

Permalink

Erlang Solutions Live Stream

Permalink

Language Design: When Less is More

Yesterday I was invited to do an interview with the man behind Erlang Punch, Mathieu Kerjouan. Mathieu is a truly great programmer, an absolutely excellent interviewer, and it was a lot of fun to get to do something task-focused with him like this. He asked some pretty broad questions about Erlang as a language, as an ecosystem, and its role in my professional development and journey as a programmer.

He asked what I like and what I don’t like about the language, and as anyone who knows me well as a programmer may imagine, I gave a rather extensive answer. In the course of giving my answer the main theme that emerged was that Erlang’s greatest trait as a language is its simplicity and consistency, and the worst parts are where this simplicity and consistency is violated. Most of the complexity of the system is actually in OTP as a system, rather than Erlang itself as a language, and this seems to be a profoundly beneficial tradeoff.

One point of inconsistency I mentioned as a dislike was the the shadowing of variable names declared in an outer scope in the heads of lambda functions, and another was the introduction of the alien ?= operator instead of relying on functional constructs that are far more flexible (such as adding a pipeline library module to the standard library). I went into some detail in my answer about the exact reasons why these are examples of stupid warts in the language and the history behind them so I won’t recount them here, but it is sufficient to say that there is a cardinal sin being committed in these two cases: the crime of inconsistency.

To expand on that point a bit, I would like to illustrate briefly how it would actually be possible to reduce Erlang’s syntax to bring even further consistency to it. I’m not saying this is something I’m going to write an EEP about any time soon (the juice just isn’t worth the squeeze), but it is illustrative of how easily a minimalist approach can be applied to language design in ways that comply with core elements of the language’s existing fundamentals. In fact the point of thinking this way is to explore and discover what those fundamentals might actually be as this is not always immediately obvious, even to someone experienced with using it or (ironically) the language’s original designer.

Let’s consider the syntax of Erlang’s “send” and “receive” operations.

Currently there is a magic token ! for “send” and another magic token, the word receive, for “receive”. Imagine if instead of these magical tokens we had the following functions as BIFs:

send(Recipient, Message) ->
    Recipient ! Message,
    ok.

receive() ->
    receive Any -> Any end.

receive(Timeout) ->
    receive
        Any           -> {ok, Any};
        after Timeout -> timeout
    end.

Suddenly we realize that these could actually be BIFs, privileged with whatever magical implementation or compilation cheats we might come up with, and invoked very simply as:

ok = send(Recipient, Message),

Message = receive(),

case receive(10000) of
    {ok, Message} -> % …
    timeout       -> % …
end,

It would be possible to implement the above functions exactly as demonstrated above and use them without any interruption or loss of meaning. This would reduce the syntactic complexity of the language, and in fact in the case of send/2 there actually is a function in the standard library that has turned the magic ! syntax into an optional function for exactly this reason: because it is useful in a functional context to have functions with which to work!

Is there anything else we can (and in a better world, probably should) adapt in this way?

Why yes.
Yes, in fact there is: the notorious try which we somehow got along just fine without once upon a time…

try(Operation) ->
    try {ok, Operation()}
    catch C:E:S -> {C, E, S}
    end.

This could be written any number of other ways (in the old days this would have been speculative execution in a monitored process spawned specifically for this reason, for example, which you can still do and parallel map implementations actually do), and of course we would want to make it into a magic BIF almost certainly, but my point here is that the complexity can be pushed back into the compiler and system rather than remaining a point of syntactic complexity that has its own little magic (and totally unique) notation that even graybeards occasionally forget the details of and have to look up on a cheatsheet from time to time (“wait, can we still call a function to get a stacktrace or is that in the… ohhhh yeah, it’s a syntax thing now… [typing noises]”).

Protip for language designers: the more things a programmer has to remember about your language’s little quirks the less mental bandwidth they have remaining for whatever problem they are actually trying to solve.

Why even have such inconsistency in the language when other features of the language that take advantage the existing syntax so well could simply have been implemented instead (and often retrospectively actually are, such as with send/2)?

In a word: habit

This is a bad habit. Languages in general, not just Erlang, would benefit greatly from a reduction in magic syntax and glyphy operators and we should move to minimize languages rather than add every allegedly “good idea” that the Fairy of Warped Ego poops out on our shoulders.

Try it out…

It’s really not all that insane.
C’mon, man, everyone’s doing it…

-module(should_be_bifs).

-export([send/2,
         recv/0,
         recv/1,
         tryy/1]).

send(Recipient, Message) ->
    Recipient ! Message,
    ok.

recv() ->
    receive Any -> Any end.

recv(Timeout) ->
    receive
        Any           -> {ok, Any}
        after Timeout -> timeout
    end.

tryy(Operation) ->
    try {ok, Operation()}
    catch C:E:S -> {C, E, S}
    end.

Permalink

Type system updates: moving from research into development

A year ago, at ElixirConf EU 2022, we announced an effort to research and develop a type system for Elixir (video presentation) (written report).

This work is happening under the lead of Giuseppe Castagna, CNRS Senior Researcher, and taken by Guillaume Duboc as part of his PhD studies, with further guidance from myself (José Valim).

This article is a summary of where we are in our efforts and where we are going.

Out of research

Our main goal during research is to find a type system that can model most of Elixir’s functional semantics and develop brand new theory on the areas we found to be incompatible or lacking. We believe we were able to achieve this goal with a gradual set-theoretic type system and we are now ready to head towards development. Over the last 2 months, we have published plenty of resources on our results:

• A technical report on the design principles of the Elixir type system
• A technical presentation by Guillaume Duboc at ElixirConf 2023 on the work above
• An informal discussion with Giuseppe Castagna, Guillaume Duboc, and José Valim on the SmartLogic podcast
• An informal Q&A with Guillaume Duboc, José Valim, and the community on Twitch

Our focus so far has been on the semantics. While we have introduced a new syntax capable of expressing the semantics of the new set-theoretic type system, the syntax is not final as there are still no concrete plans for user-facing changes to the language. Once we are confident those changes will happen, we will have plenty of discussion with the community about the type system interface and its syntax.

The work so far has been made possible thanks to a partnership between the CNRS and Remote, with sponsorships from Fresha, Supabase, and Dashbit.

Into development

While there is still on-going research, our focus for the second semester of 2023 onwards is on development.

Incorporating a type system into a language used at scale can be a daunting task. Our concerns range from how the community will interact and use the type system to how it will perform on large codebases. Therefore, our plan is to gradually introduce our gradual (pun intended) type system into the Elixir compiler.

In the first release, types will be used just internally by the compiler. The type system will extract type information from patterns and guards to find the most obvious mistakes, such as typos in field names or type mismatches from attempting to add an integer to a string, without introducing any user-facing changes to the language. At this stage, our main goal is to assess the performance impact of the type system and the quality of the reports we can generate in case of typing violations. If we are unhappy with the results, we still have time to reassess our work or drop the initiative altogether.

The second milestone is to introduce type annotations only in structs, which are named and statically-defined in Elixir codebases. Elixir programs frequently pattern match on structs, which reveals information about the struct fields, but it knows nothing about their respective types. By propagating types from structs and their fields throughout the program, we will increase the type system’s ability to find errors while further straining our type system implementation.

The third milestone is to introduce the (most likely) $-prefixed type annotations for functions, with no or very limited type reconstruction: users can annotate their code with types, but any untyped parameter will be assumed to be of the dynamic() type. If successful, then we will effectively have introduced a type system into the language.

This new exciting development stage is sponsored by Fresha (they are hiring!), Starfish* (they are hiring!), and Dashbit.

Permalink

Embrace Complexity; Tighten Your Feedback Loops

Permalink

Elixir v1.15 released

Elixir v1.15 has just been released. 🎉

Elixir v1.15 is a smaller release with focused improvements on compilation and boot times. This release also completes our integration process with Erlang/OTP logger, bringing new features such as log rotation and compression out of the box.

You will also find additional convenience functions in Code, Map, Keyword, all Calendar modules, and others.

Finally, we are glad to welcome Jean Klingler as a member of the Elixir Core team. Thank you for your contributions!

Compile and boot-time improvements

The last several releases brought improvements to compilation time and this version is no different. In particular, Elixir now caches and prunes load paths before compilation, ensuring your project (and dependencies!) compile faster and in an environment closer to production.

In a nutshell, the Erlang VM loads modules from code paths. Each application that ships with Erlang and Elixir plus each dependency become an entry in your code path. The larger the code path, the more work Erlang has to do in order to find a module.

In previous versions, Mix would only add entries to the load paths. Therefore, if you compiled 20 dependencies and you went to compile the 21st, the code path would have 21 entries (plus all Erlang and Elixir apps). This allowed modules from unrelated dependencies to be seen and made compilation slower the more dependencies you had. With this release, we will now prune the code paths to only the ones listed as dependencies, bringing the behaviour closer to mix release.

Furthermore, Erlang/OTP 26 allows us to start applications concurrently and cache the code path lookups, decreasing the cost of booting applications. The combination of Elixir v1.15 and Erlang/OTP 26 should also reduce the boot time of applications, such as when starting iex -S mix or running a single test with mix test.

As an example, I have benchmarked the Livebook application on a M1 Max MacStudio across different Elixir and Erlang/OTP versions. At the time of benchmarking, Livebook had ~200 source .ex files and ~35 dependencies. Compilation-times were improved by 16%:

Livebook saw an improvement of 30% on boot times:

Different application will see different results. Our expectations are the gains will be more meaningful the more dependencies you have, the more files you have, and the more cores you have. We have even received reports of up to 40% faster compilation times, although it is yet unclear how generalizable this will be in practice. Note this work does not improve the time to compile slow individual files.

The compiler is also smarter in several ways: @behaviour declarations no longer add compile-time dependencies and aliases in patterns and guards add no dependency whatsoever, as no dispatching happens. Furthermore, Mix now tracks the digests of @external_resource files, reducing the amount of recompilation when swapping branches. Finally, dependencies are automatically recompiled when their compile-time configuration changes, providing a smoother development experience.

Potential incompatibilities

Due to the code path pruning, if you have an application or dependency that does not specify its dependencies on Erlang/OTP and core Elixir applications, which has always been erroneus behaviour, it may no longer compile successfully in Elixir v1.15. You can temporarily disable code path pruning by setting prune_code_paths: false in your mix.exs, although doing so may lead to runtime bugs that are only manifested inside a mix release.

Compiler warnings and errors

The Elixir compiler can now emit many errors for a single file, making sure more feedback is reported to developers before compilation is aborted.

In Elixir v1.14, an undefined function would be reported as:

** (CompileError) undefined function foo/0 (there is no such import)
    my_file.exs:1

In Elixir v1.15, the new reports will look like:

error: undefined function foo/0 (there is no such import)
  my_file.exs:1

** (CompileError) my_file.exs: cannot compile file (errors have been logged)

A new function, called Code.with_diagnostics/2, has been added so this information can be leveraged by editors, allowing them to point to several errors at once. We have currently ongoing work and contribution to further improve the compiler diagnostics in future Elixir releases.

Potential incompatibilities

As part of this effort, the behaviour where undefined variables were transformed into nullary function calls, often leading to confusing error reports, has been disabled during project compilation. You can invoke Code.compiler_options(on_undefined_variable: :warn) at the top of your mix.exs to bring the old behaviour back.

Integration with Erlang/OTP logger

This release provides additional features such as global logger metadata and file logging (with rotation and compression) out of the box!

This release also soft-deprecates Elixir’s Logger Backends in favor of Erlang’s Logger handlers. Elixir will automatically convert your :console backend configuration into the new configuration. Previously, you would set:

config :logger, :console,
  level: :error,
  format: "$time $message $metadata"

Which is now translated to the equivalent:

config :logger, :default_handler,
  level: :error

config :logger, :default_formatter,
  format: "$time $message $metadata"

To replace the default console handler by one that writes to disk, with log rotation and compression:

config :logger, :default_handler,
  config: [
    file: ~c"system.log",
    filesync_repeat_interval: 5000,
    file_check: 5000,
    max_no_bytes: 10_000_000,
    max_no_files: 5,
    compress_on_rotate: true
  ]

Finally, the previous Logger Backends API is now soft-deprecated. If you implement your own backends, you want to consider migrating to :logger_backends in the long term. See the new Logger documentation for more information on the new features and compatibility.

Learn more

For a complete list of all changes, see the full release notes.

Check the Install section to get Elixir installed and read our Getting Started guide to learn more.

Happy compiling!

Permalink

Erlang/OTP 26.0 Release

Erlang/OTP 26 is a new major release with new features, improvements as well as a few incompatibilities.

For details about new features, bugfixes and potential incompatibilities see the Erlang 26.0 README or the Erlang/OTP 26.0 downloads page.

Below are some of the highlights of the release:

There is also a Blogpost about the highlights.

Parsetools

• Leex has been extended with optional column number support.

Stdlib

• The family of enumeration functions in module lists has been extended with enumerate/3 that allows a step value to be supplied.
• Update Unicode to version 15.0.0.
• proc_lib:start*/* has become synchronous when the started process fails. This requires that a failing process use a new function proc_lib:init_fail/2,3, or exits, to indicate failure. All OTP behaviours have been fixed to do this.

The Shell

There are a lot of new features and improvements in the Erlang shell:

• auto-complete of variables, record names, record fields names, map keys, function parameter types and filenames.
• Open external editor in the shell to edit the current expression.
• defining records (with types), functions, specs and types in the shell.

New terminal

• The TTY/terminal subsystem has been rewritten. Windows users will notice that erl.exe has the same functionality as a normal Unix shell and that werl.exe is just a symlink to erl.exe. This makes the Windows Erlang terminal experience identical to that of Unix.

Compiler and JIT optimizations:

• Creation and matching of binaries with segments of fixed sizes have been optimized.

• Creation and matching of UTF-8 segments have been optimized.

• Appending to binaries has been optimized.

• The compiler and JIT now generate better code for creation of small maps where all keys are literals known at compile time.

• Thanks to the optimizations above the performance of the base64 module has been significantly improved. For example, on an x86_64 system with the JIT both encode and decode are almost three times faster than in Erlang/OTP 25.

Maps

• Map comprehensions as suggested in EEP 58 has now been implemented.

• Some map operations have been optimized by changing the internal sort order of atom keys. This changes the (undocumented) order of how atom keys in small maps are printed and returned by maps:to_list/1 and maps:next/1. The new order is unpredictable and may change between different invocations of the Erlang VM.

• Introducing the new function maps:iterator/2 for creating an interator that return the map elements in a deterministic order. There are also new modifiers k and K for the format string in io:format() to support printing map elements ordered.

Dialyzer

• Added the new built-in type dynamic() introduced in EEP 61, PR introducing EEP 61 improving support for gradual type checkers.

• Dialyzer has a new incremental mode that be invoked by giving the --incremental option when running Dialyzer. This new incremental mode is likely to become the default in a future release.

Misc ERTS, Stdlib, Kernel, Compiler

• Multi time warp mode is now enabled by default. This assumes that all code executing on the system is time warp safe.

• Support for UTF-8 atoms and strings in the NIF interface including new functions enif_make_new_atom, enif_make_new_atom_len and enif_get_string_length.

• The BIFs min/2 and max/2 are now allowed to be used in guards and match specs.

• Improved the selective receive optimization, which can now be enabled for references returned from other functions. This greatly improves the performance of gen_server:send_request/3, gen_server:wait_response/2, and similar functions.

• New trace feature call_memory. Similar to call_time tracing, but instead of measure accumulated time in traced functions it measures accumulated heap space consumed by traced functions. It can be used to compare how much different functions are contributing to garbage collection being triggered.

• It is no longer necessary to enable a feature in the runtime system in order to load modules that are using it. It is sufficient to enable the feature in the compiler when compiling it.

• inet:setopts/2 has got 3 new options: reuseport, reuseport_lb and exclusiveaddruse.

• Fix so that -fno-omit-frame-pointer is applied to all of the Erlang VM when using the JIT so that tools, such as perf, can crawl the process stacks.

• In the lists module, the zip family of functions now takes options to allow handling lists of different lengths.

• Added the zip:zip_get_crc32/2 function to retrieve the CRC32 checksum from an opened ZIP archive. gen_server optimized by caching callback functions

• The modules Erlang DNS resolver inet_res and helper modules have been updated for RFC6891; to handle OPT RR with DNSSEC OK (DO) bit.

• Introduced application:get_supervisor/1.

• Cache OTP boot code paths, to limit how many folders that are being accessed during a module lookup. Can be disabled with -cache_boot_path false.

SSL

• Change the client default verify option to verify_peer. Note that this makes it mandatory to also supply trusted CA certificates or explicitly set verify to verify_none. This also applies when using the so called anonymous test cipher suites defined in TLS versions pre TLS-1.3.

• Support for Kernel TLS (kTLS), has been added to the SSL application, for TLS distribution (-proto_dist inet_tls), the SSL option {ktls, true}.
• Improved error checking and handling of ssl options.

• Mitigate memory usage from large certificate chains by lowering the maximum handshake size. This should not effect the common cases, if needed it can be configured to a higher value.

• For security reasons the SHA1 and DSA algorithms are no longer among the default values.

• Add encoding and decoding of use_srtp hello extension to facilitate for DTLS users to implement SRTP functionality.

For more details about new features and potential incompatibilities see the readme

Many thanks to all contributors!

Download links for this and previous versions are found here

• https://www.erlang.org/downloads

Permalink