Check out and sign up for Ruby Remote Conf!

02:45 - Francesco Cesarini Introduction

03:08 - Erlang Programming Language

08:23 - Francesco and Erlang

10:49 - Building a Company Around a Language (Erlang Solutions)

16:00 - The Erlang Programming Language

24:25 - Error Handling Semantics

30:23 - Getting Started with Erlang

34:23 - Elixir

35:28 - Erlang and Polyglot Architecture

37:01 - WombatOAM

38:57 - Erlang Pros and Cons

40:44 - TDD (Test-Driven Development)

46:10 - Languages/Technologies on the Horizon (for Francesco)

48:21 - The Erlang Community

50:24 - Writing Apps with Erlang / IoT?

Picks

Avdi Grimm: A Personal Programming Language Roadmap (Avdi)
Pharo (Avdi)
Avdi Grimm: In Which I Make You Hate Ruby in 7 Minutes (Avdi)
Babel-17 / Empire Star by Samuel R. Delany (Coraline)
Orson Welles (Coraline)
John Hughes: QuickCheck Evolution @ CodeMesh 2014 (Jessica)
Vehicles: Experiments in Synthetic Psychology by Valentino Braitenberg (Jessica)
Zero to One: Notes On Startups, or How to Build the Future by Peter Thiel (Francesco)
CodeNewbie Podcast (Chuck)
Ask Me Another (Chuck)
Startups For the Rest of Us (Chuck)

Special Guest: Francesco Cesarini.

 

I refer to it as The Ramones of the programming languages, setting the beat in the background.

All the features new languages now are picking up.

Hopefully it has more chords than The Ramones.

Not really. It’s basic and simple. But it’s got the right chords.

Hey everybody and welcome to episode 208 of the Ruby Rogues Podcast. This week on our panel, we have Jessica Kerr.

Good morning.

Avdi Grimm.

Hello from Pennsylvania. Wait, no, Tennessee.

Coraline Ada Ehmke.

Good morning, listeners.

I’m Charles Max Wood from DevChat.tv. I just want to put a quick reminder out. We have the schedule mostly finalized for Ruby Remote Conf. So, if you want to come and listen to some great speakers including some of the people on this show, go to RubyRemoteConf.com and sign up.

We also have a special guest this week, and that is Francesco Cesarini.

Hi there, from London, England.

Do you want to introduce yourself really quickly?

Yeah, I’m Francesco, technical director, and founder of Erlang Solutions. We’ve been working with Erlang since 1994. And I’ve coauthored two of the O’Reilly books around Erlang as well.

Oh, that’s cool. I didn’t know Erlang has been around that long. Actually, I think I did. But it seemed to kind of get… people seemed to know more about it over the last maybe 5 or 10 years (is when I really started to hear about it).

Yes. It’s with multicore and yeah, with the types of problems we’re solving today, I think it’s been getting more and more attention.

Well, it also started out in-house and then was open sourced. Is that correct?

Yes, but open source started in… yeah, it was open sourced in 1998. So, it was still when people didn’t really know what open source was. But it got open sourced without any budget, without any promotional material and PR.

Mmhmm.

So, it slowly spread organically. And I think it started picking up and ending up on everyone’s radar around 2006 with multicore. And then with Joe Armstrong’s book and then mine came out in 2009.

And from there, it’s been, more and more people have been hearing about it. And it’s being used in massively scalable fault-tolerant systems.

Francesco you talked about multicore. What exactly is that?

Well, the ability for the virtual machine to fully utilize all of the cores. So, you run a program on a machine with a single core. And you then take the same Erlang program running on a machine with 4 or 8 or 16 cores. It’s going to run 8 or 16 times faster. And that has to do with some of the features in Erlang, which makes the scalability on multicore architectures much easier.

Now at a low-level Erlang handles that very differently from just about any other programming languages. Can you fill us in on some of the details there?

Not sure if it handles it differently. It’s…

Well, I guess as compared to the typical threading model that most people think of, like [OS] threads.

Exactly, exactly. So, what we’ve got in Erlang is lightweight concurrency. And that means that the virtual machine is in effect an operating system on top of the operating system. And it handles its own processes. We call them lightweight processes. And so, a VM is able to handle millions of processes. They take sub-microseconds to create and use up very little memory.

Isn’t it like 304 words or something like that per…?

Exactly, something very, very little. Yes, exactly. It’s something in that space. But obviously as they do more things, and they need more memory it gets allocated. But the initial footprint is very, very small. And what differs with Erlang is it’s based on the Actor model. So, processes do not share memory. And they communicate with each other through message-passing, so message-passing which is asynchronous. And the biggest challenge in scaling a multicore architecture is memory lock contention. And so, by making sure that there is no shared memory you get rid of the memory lock contention. And you get almost linear scalability. So, it simplifies the problem and it allows those working on the VM to then focus on the more difficult things which are internal locks, looking at the schedulers, looking at your various approaches which you have today in scaling.

Where does Erlang come from? Why was it created this way?

So, it was invented back in the mid-80s. The computer science laboratory at Ericsson, so the telco infrastructure provider, were trying to figure out how to program the next generation of telecom switches. And they tried and prototyped finite state machines and telecom applications with most of the prevailing languages which were available at the time. And with most of the languages which were available at the time. And they quickly came to the conclusion that there were a lot of great features in these languages but there was no one language which had all of the features they required.

And if you think of it, the problems they were trying to solve were the problems of systems which were distributed. They were massively concurrent. That means you’ve got thousands of phone calls going on at the same time. They had to behave in a predictable fashion under extreme loads. So, at New Year’s Eve when everyone used to call each other you’d pick up the phone. You’d still expect to hear the [inaudible] on the other end. And they had to be fault-tolerant. So, even in the case of disasters you still had to be able to dial 999. So, this was the type of problem they were trying to solve. And they came to the conclusion after extensive prototyping that the solution to this problem was a programming language. So, they didn’t set out to invent the programming language. It just happened to be the solution.

And with the internet coming about, I think a lot of the problems I’ve described are very similar to what we’re seeing today. Think of mobile apps. Think of the Internet of Things. Even think of the web. We usually joke that Erlang predates the web even though it was designed for the web. [Chuckles]

What initially attracted you to Erlang?

So, I was studying in university. It was a parallel computing course. And the teacher comes in and holds up the first edition of the Erlang book and said, “Read it. These are the exercises. Do them.” And then off you went and started describing the horrors of parallel computing. So, deadlocks, semaphores, race conditions. And yeah, pretty much we were young and impressionable and got scared at what it was saying. And we started working on the exercise.

And it was a simulated world. We had carrot patches growing in this world. And then we had rabbits going around eating the carrots. And if a rabbit ate a lot of carrots, it would split in two and you’d have two rabbits. And if he didn’t eat enough carrots he would run out of energy and die. Then you had wolves roaming this virtual world. And wolves would go out hunting for rabbits. And if they found a rabbit, they’d eat it. And there was also a bit of communication in between the rabbits. If a rabbit found a carrot patch it would broadcast it to the rabbits within its vicinity. And the same with the wolves, if the wolves found a rabbit they would broadcast to the other wolves. And rabbits seeing wolves would start telling each other, start running away.

And the way we designed it was that every rabbit was an Erlang process. Every wolf was an Erlang process. Every carrot patch was an Erlang process. It was a really fun exercise. It took me about 40 hours to complete. And I didn’t think about it. I completed exercises, handed them in. And Erlang at the time was one of about 10 programming languages we had to use in our four years it took to take the degree. Until a few months later when we were studying [Eiffel] which was an object-oriented language, so we were studying object-oriented programming. And we were given the exact same exercise.

And despite me having already solved that exercise once and despite working with a classmate, in ended up taking us 60 hours to finish that exercise. So, three times the time it took us to do the original, probably. And at that point, that’s when I realized, right tool for the job. There is always a right tool for the job which makes you much more productive. And at that point soon after, I picked up the phone, called Joe Armstrong at the computer science lab and asked if he had any internships open. Two weeks later I went down for an interview and I never looked back. So yeah, that’s how I got into Erlang.

So, building a company around a language seems a little bit interesting to me. What’s the focus there? Are you consulting? Or are you ore focused on the language and the community around it?

So yeah, that’s an interesting question. I agree it’s pretty unusual to build a company around a programming language, especially a programming language we don’t own or control. Because still today, Ericsson has a team of about 20 people working on the VM and working on the libraries. And we work very closely with them and obviously influence them. But we don’t have the exact, the direct control. What happened was we started off as, well it started off as me doing consulting. And from there we moved on and started providing training courses. And soon after that we started going in-house systems development. And before we knew it, we were providing support and we then started working on our own products internally.

So we have today, MongooseIM which is an instant messaging stack being used widely for online chat by a lot of high profile customers. We’ve got Wombat which is an operation and maintenance tool. And we’re also resellers of Riak, the NoSQL database. So, as a consultancy and a body shop selling hours, we’re slowly making the migration over to a product company, obviously still doing a lot of the consultancy and a lot of what’s generating the revenue. As a split I would say we’re probably 30% products today, 70% consultancy and professional services.

And yeah, we are focused on the community. We run a lot of events. And where possible we try to let others run the events but help them from behind the scenes. So, I think some of the largest events we run is the Erlang User Conference. Ericsson asked us to take it over back in 2009. And it attracts about 350 people. We run the Erlang Factory in San Francisco which is the largest Erlang event in the US. And that attracts about 250 people. And then we’ve got a lot of smaller Erlang Factory [inaudible]. And more recently we started Code Mesh in London which is focused on distributed and functional programming where my heart has been for quite a while now.

What are some of the more interesting projects you’ve done through Erlang Solutions?

Oh, I wouldn’t know where to start. [Chuckles] I think if you go back in time, every other year we usually manage to get a project which allowed us to double in size. From my end I think the most exciting ones were in the very early days, I worked on T-Mobile’s SMS gateway. So, it was a gateway which was used for all inbound and outbound SMS’s. So, all the short codes, all of the TV voting, all went through this gateway. And it was, when it went live, one of the fastest SMS gateways out there, handling more SMS’s than what the network would actually carry. And that was really good fun, taking over proof of concept and getting it production ready.

More recently, fast forward a few years we built a massively scalable instant messaging gateway. This was before smartphones. So, this was 2008, 2009. You still had a few [Linux] phones. But most of the phones were very static. And what we did is a gateway which allowed users to connect to instant messaging providers such as AOL Aim, ICQ, Google Talk, the MSN network which existed at the time. And at its peak it carried about 30% of all of Microsoft’s wireless instant messaging traffic. And that was a [inaudible] project of about eight man years. And it was just a race against time to go live. And yet that was, you don’t get to experience that many projects like that where speed to market is critical, else the competition will overtake you. And it was highly optimized for mobile. So, it was incredibly, it was a lot of lessons learned which we’re still applying today over battery life, reducing loads, reducing the data you transfer.

And well more recently, we’ve been trying to package all of the lessons we’ve learned in ensuring your systems are up all the time. So, looking at preemptive [support], monitoring metrics, and alarming, into a generic product we’re calling Wombat. And that’s been a real fun product to work on as well. So yeah, it’s never dull. And this is just the tip of the iceberg. I think we’ve worked from… we worked with WhatsApp even before they had their first office, in helping them scale. We’ve worked with everything from single one-man startups all of the way to Fortune 100 companies.

I want to hear a little more about the language. Of course, this is a Ruby show but we’re very big on being polyglot and learning from other languages. And yesterday I put out a little screencast to show some people some of the power of the Pharo Smalltalk environment. And I was particularly focusing on things that the average Ruby programmer would look at and it would just blow them away. They don’t have anything like this in Ruby. And I actually had somebody comment and say that they were in their office and they actually yelled, “No way!” at the screen. I’m curious. If you wanted to show someone, like a Ruby programmer, something about Erlang that’s going to melt their face, is there something, is there some example that you would show them?

Absolutely. First of all, the concurrency model. Just on a laptop I could probably create about 10 million processes which each send a message to each other, receive the message, and terminate. And it would take a couple of seconds to create them all and run that script. The debugging, you’re able to turn on debugging on local functions and global functions without trace compiling your code. And you’re also able to…

Wait, what? You can turn on debugging without recompiling?

Yes. So, what you do is… it runs on a VM. It runs on bytecode. So, you want to go in and say, “I want to trace for all of the lists:sort functions. Every time I call lists:sort I want a trace event generated for it. And you’re basically able to go in and troubleshoot live systems.

So, you do this on production?

Absolutely. You do it on production. You can do it in test plants. And it’s incredibly powerful. Incredibly dangerous as well, because…

You go in and start tracing a function which is called all the time, you end up getting so much I/O in your shell that you end up killing the whole virtual machine. I’ve actually managed to sink a whole mobile data network by being [chuckles] careless with the tracer.

Oops.

And cause a minor outage. I was… yeah exactly, that was what I said, “Oops.” [Laughter]

And the support guys…

So, that was you.

Yes, the support guys were sitting probably three or four desks behind me. And I was like, start it again, start it again. And you’d hear them, “Did you see that?” “Yeah, yeah, let me retry.” And I managed to restart that particular system I just sunk. And, “No, it seems to work.” “Oh, it must have been a firewall.” And yeah, they blamed it on the system administrators which they always did. [Chuckles]

So, I got away with it that time around. But it’s incredibly powerful. And being able to trace in a live environment really cuts down the turnaround time to locating bugs and solving them.

You can do more than trace, right? You can deploy code?

Yes. So, you’re also able to do live upgrades in real-time. And this actually comes from Smalltalk. It’s an idea they took from Smalltalk where you’re running a version of the module. And any one time your virtual machine can have two versions of the module. You load in the new version of the module and you then do what we call a fully qualified function call which basically appends the module name to the function name. And every time you do a fully qualified function call you check, “Am I running the latest version of the code?” “No, you’re not”? The pointer is swapped from the old version to the new version of the code. So, retaining your process, retaining its state, retaining all the variable values, you’re running your updated version, your updated code.

And this was a requirement with telecom systems where you had to upgrade your switches, your phone switches, without taking them down. You needed to have [inaudible] availability including upgrades. So, shutting down the switches was not an option. So, live code upgrade is another really, really cool feature which I think a lot of people in the Ruby world are very interested in. And it’s something available in Elixir as well.

And I want to dig into that a little bit, because I’ve experienced this a little bit. And there are some details there that you don’t think about until you actually start trying to do it. Like in Ruby, when we think about live updating something, one of the issues that you can run into is for instance if you’re upgrading to a new version of a class that actually uses a different set of instance variables, or where an instance variable has been renamed. And you’re not throwing out all the objects and just creating new objects. Then you could have stuff blow up because the code is expecting a different instance variable to exist. But you actually have some solutions for that in this Erlang system, right?

That is correct. So, Erlang comes with something called OTP which provides the framework which we use to do software upgrades. So, you’ve got the basic semantics which are part of the language. But I’ve simplified it. And as you say, I made it appear like it’s very simple. The truth is software upgrades are not for the faint of heart.

If you think of it, you need to change… you might change the state of your process. You might change APIs you’re calling. You might change actual protocol used by processes when communicating with each other. And you’re doing it on a single node. What happens if you start doing it in the distributed environment as well? So, it’s not as easy as it sounds. But there are frameworks which allow you to test and do your upgrades in a controlled way. So, it’s not just done manually. But the key to it is actually, testing, testing, testing, and making sure that whatever you’ve done is backward compatible.

Right.

And then nothing blows up.

But you’re able to do effectively almost like a… we think a lot about database migrations in Ruby on Rails applications. But you can almost…

Yes.

You’re effectively doing process migrations.

Process or thread… yeah, the Ruby equivalent would be a thread migration. That’s correct, yes.

Mmhmm.

And in that migration yeah, you might also go in and upgrade your database and update your database, your functions to traverse it in case the schema changes. And to add to that, it’s incredibly powerful. But it’s useful if you need to do an upgrade to a product you’ve shipped in tens of thousands of instances. So, a company like Ericsson might have a team of a hundred Erlang developers working on a particular project. The cost of adding four or five developers just to focus on software upgrade is not an issue.

But if you’re then dealing with, if you’re a startup in Silicon Valley pumping out code and trying to get users but not generating any revenue, it doesn’t always make sense to do live upgrades. The cost of doing it might be too high. So, all you do is you take down your system and you restart it. And in most cases that will also work. So, it depends. You need to do an analysis and see if it’s worthwhile, the investment or not. But the infrastructure is there and the toolchain to do it is there as well.

And another thing which I think is pretty cool in Erlang is something called OTP, which what OTP does, it gives you a framework which describes the architecture of a single node. So, you were asking about things blowing up. What we do is we put processes or as we call them, behaviors, under a supervision tree. Behaviors could be a finite state machine. It could be event handlers or client-servers. And they’re all monitored by a supervisor. And if a process terminates or crashes, the supervisor is configured to take a certain number of actions which could restart the process or do nothing or terminate all the processes in the subtree and restart them. Or if the supervisor realizes that it has handled five crashes in the last minute, it comes to the realization, “Okay I can’t handle this issue because I’m getting cyclic restarts here. Let me terminate myself and see if my supervisor can handle it.”

And so, you have a system of escalation where the higher up you go in the supervision hierarchy the more parts of the system you restart. So, it’s a way of trying to isolate error handling. And if the isolation doesn’t work and you still have that issue, you escalate and you try to restart a larger part. And in the end it reaches the top-level supervisor. And if that terminates, your whole node goes down and then gets restarted.

In Ruby and other OO languages we talk about circuit breaker patterns for stuff like that. Is that something that Erlang has given a lot of thought to?

So, the thought has been put into the error handling semantics of the language. I think that’s really where… and the error handling semantics in the language is very easy. You’ve got two processes.

And two processes can monitor each other.

Mmhmm.

Through links or monitors. Links are bidirectional monitors or unidirectional. And if a process terminates, the process which you’re linked to will receive an exit signal. And based on the exit signal it can decide what to do. So, that’s it. That’s all there is to it. And using this simple semantics you build much more complex frameworks. So, I suspect the circuit breaker is more like realizing that you’ve got an issue and…

I think it’s probably one possible semantic that you could use.

Yeah.

You were talking about realizing that a process has gone down five times in a row and handling that differently. And that’s really important, because when, I think when I and maybe other people about, “Oh let it crash and then restart it,” we immediately start thinking about infinite restart loops. But it’s interesting. You’re talking about having supervisors that have a bit more intelligence than that.

That’s correct, yes. So, their supervisors which… by the way, it’s a library module. So, it will behave the same respective of what your system does. So, the intelligence there actually boils down to, how many restarts will you allow per minute before terminating? When a process terminates, what do you do with all the other processes in your supervision tree? Do you terminate them all? Do you only terminate the process which was started after the process which was terminated? Or do you do nothing? You just restart this process which terminated. And finally, when do you as a supervisor terminate yourself and escalate it to your top-level supervisor?

And so, what this does is it allows a programmer to only program for the correct case. And what this means is that they are not going in and putting any funny error handling in the code trying to cater for… if your process crashes, it crashes for two reasons. Either your state is corrupt or there’s a bug in your code. In Erlang what you do is you program for the correct case. You avoid putting error handling in your code unless it’s expected error handling. So, if a user types something in the shell and you need to parse it and what they typed is incorrect and you return an error message to the user, that’s part of the semantics of your program. But if there’s a bug in your code or your state gets corrupted you don’t really know what to handle and how to do it. Because A, you don’t know what the bug is, and B you don’t know how your state got corrupted.

So, what Erlang programmers are encouraged to do is instead of trying to handle these special cases, let your process fail. So, let it terminate and let your supervisor deal with it instead. And so, this actually reduces the complexity of the code greatly. It reduces it in that all of the error handling code is actually moved to a library module. And studies they’ve done where they’ve rewritten C++ code over to Erlang, they went in and counted every single line of C++ code and looked at what each line did. The error handling in the C++ code was about 25% of the whole codebase. The equivalent in the Erlang program was about 1%. So, just with the ‘let it crash’ approach, you wipe out 25% of your code. So, an incredibly powerful semantics and construct, yeah.

That’s definitely a pretty radical approach to error handling and a pretty compelling feature. So, aside from the actor and supervisor patterns, what are some of the other core metaphors or patters in Erlang?

Well, we’ve got the OTP behavior. So, we’ve talked about the supervisor. Another very common pattern is the client-server behavior. And the client-server behavior is a library module which handles all of the tricky parts of concurrency. But it does so behind the scenes. So, you can very, very easily go in and start sending messages without having to worry about your server crashing for example. You send a message to your server and the server has crashed before you send the message or has crashed when it’s waiting to handle your message, or even when it’s handling the message. All of this is handled behind the scenes. So, concurrent programming tends to get tricky. And what these patterns do is try to hide all of the complexity of the concurrency.

Another very common pattern is the event handler. Imagine you are actually… you need to log and you might want to log things to file or in certain cases filter through your logs and send out an SMS. It’s a behavior which allows you to hook modules. And every time you send it a request, it will apply particular functions in those modules. So, one module could go in and log everything to file. The other could send SMS. And you can add or remove these, we call them event handlers, in runtime as well.

And then finally you’ve got finite state machines. It’s a library module which allows you to very efficiently with very little code implement your own finite state machines as well. And once again, what they’ve done is they leave all of the generic code to the developers. Sorry, all of the generic code is written in the library module and the only thing that the developers need to think about is the specific code. So, the whole principle, I think the most common pattern or the pattern of all patterns is taking your process code and dividing it into generic code and into specific code. The generic code is given to you. All you need to worry about is the specific code.

So, if somebody decides, “Oh my goodness. This is something that I definitely need to go and try out,” how do people get started with Erlang?

So, one thing. Starting with Erlang is very different from working with most languages. Toolchains are different. Ways of thinking, of reasoning are different. And what I would suggest is the best way to start off is probably Joe Armstrong’s book called ‘Programming Erlang’. And Joe is very enthusiastic. And what he does is he does a great job on explaining why you should be using Erlang and giving you a very good high level overview over a lot of ideas and features and the philosophy behind it. And once you’re comfortable with that, there’s actually a MOOC right now with the University of Kent which is being trialed and which I recommend people to sign up with. It’s Simon Thopmson, Joe Armstrong, and I. We’ve gone in and recorded some master classes. And that’s a great way to get into it. And another…

Just to clarify, MOOC is massively open online class. Anyway, it’s a big class online that you can go take.

Exactly. And it’s done by universities. And they usually open up the course material to a much wider audience. And they’re doing the trials right now getting feedback which will go back into the real MOOC when it goes live in a few months. And from there I’d also recommend, there’s a great online resource called Learn You Some Erlang which Fred Hebert wrote. And it’s a great website. And if you like it, feel free to buy the book as well.

And at that point when you’re ready to do some harder stuff, I would recommend for ‘Designing for Scalability with Erlang/OTP’ which is a book I’m currently writing with Steve Vinoski. What we’re doing here is looking at what is it you need to think about when designing for massive scale and designing for fault tolerance. So, you want to write a system which will scale to millions of simultaneous users and never go down. What are the approaches? What is the philosophy you need to take and how do you tackle it?

I’m surprised you didn’t mention your book, ‘Erlang Programming’.

Oh, ‘Erlang Programming’ as well. Yes, that’s just as good as well. I wrote it together with Simon Thompson. And yeah, it can be used in parallel with Joe Armstrong’s book and Fred Hebert, ‘Learn You Some Erlang’. All the books, all the Erlang books out there actually are very complementary with each other. It’s been a while since we wrote it. But yeah, it’s just as valid as well.

What we do which the others don’t maybe is go very much into the way you need to start thinking concurrently. If you’re working with a system which has millions of processes the approach you need to take is usually slightly different. And I’d say there are three major hurdles to learning Erlang which we regularly see. The first is that of understanding tail recursion and using it optimally, and pattern matching. So, these are features which come from functional programming. The second hurdle is understanding concurrency and being able to think and reason concurrently. So, ensuring that you actually have a process for every truly concurrent activity in your system, and instead of serializing everything through a process. And then the third hurdle is understanding the fault tolerance and the error handling and the philosophy behind it.

So, it’s a slightly different way of thinking and reasoning to what you might be used to. But it really helps. And even if you don’t do Erlang on a daily basis, learning to think and reason that way will make you a better programmer no matter what technologies you use.

Absolutely. That sounds really fascinating, especially the book about what you need to think about when you’re programming for a hundred percent or as close as you can get to a hundred percent uptime. Francesco, I don’t know how much time you’ve spent on Elixir. But do you know whether learning Elixir will have the same effect as far as teaching you about tail recursion and pattern matching and concurrency and error handling?

I’ve not had enough time as I’d like to spend with Elixir. But I actually love what I’m seeing. And what Jose’s doing is creating a much easier path for, especially Ruby programmers to migrate from Ruby to Elixir. So, it makes… he’s making Erlang much more approachable in that he’s providing a much more modern toolchain around it. But all of the key concepts in Erlang are in there as well.

To be clear, Elixir is another programming language that compiles to Erlang, right?

That is correct. So, it’s taken a lot of the great features from Erlang and some good ideas also from Clojure, so macros from Clojure, with a Ruby-like syntax. And a lot of work is being done around the package manager and the whole toolchain as well.

So, in the Rails world there’s been a lot of interest in taking apart monoliths and breaking them down into services. And oftentimes this is taken as an opportunity for some resume-driven development with teams applying different languages for different services. How do you see Erlang fitting into a polyglot architecture?

It’s always been part of a polyglot architecture. I think you’ll see most Erlang developers advocating for the right tool for the job. And in many, many occasions, Erlang is actually beating in the middle of it all as it acts as the glue you use to integrate all of these different languages together. So, it’s an ideal language which will act as… just take a messaging bus. RabbitMQ is one of the most used Erlang open source applications out there. It’s an AMQP messaging bus being used by probably most Fortune 100 companies. And all it does is enables and facilitates the whole concept of publish/subscribe and service-oriented architectures.

I didn’t realize that RabbitMQ was Erlang.

Oh yeah, it is, yes. Another very well-known messaging app is both ejabberd and MongooseIM which are now… once again, they’re not maybe tying different programming languages together but they’re tying the internet together as we’re seeing an increased adoption of XMPP around the internet of things. You’ll also have MQTT and you have a lot of messaging buses. A lot of proprietary ones as well which have been written, out there.

You mentioned Wombat as a product that your company’s working on. Is that specific to Erlang?

Currently it is, yes. So, we were working on a lot of different Erlang projects. And what we realized is that we were solving the same problems time after time. And it was the problems of monitoring, the problems of preemptive support. And once issues had occurred, the whole problem of going back and having enough visibility to try to figure out what led up to the outage, fix it and ensure it wouldn’t happen again. And so, what we started doing as part of a research project was put together all of this code into a generic Erlang node. And what this node does is it connects itself to an Erlang node and then goes out and does an auto-discovery of all the nodes in the cluster which Wombat then connects to.

And once Wombat’s connected to it, it goes in and actually discovers what application, what libraries you are running in your virtual machine. And if certain libs are supported it injects code and starts pulling back metrics, alarms, and logs for these particular libraries. And it will work just out of the box without having to do anything with your code. So, you just develop your code as you would. Or you could have had code which has been in production for a few years. And yeah, you can just transparently just connect to it. And then all of a sudden it [inaudible] different memory usages in real-time. You can start getting data on the latency of your HTTP requests. And very soon you’ll also be able to start creating certain function calls as well.

But it’s very Erlang-specific. And it was possible because of the whole dynamic code uploading nature we’ve got in Erlang where we’re able to go in and inject agent’s codes. And it’s got agents who start running in the Erlang VM. There are plans to make it widely available for more, for other programming languages. But we first want to get it 100% right for Erlang and Elixir.

Wow. That sounds really cool.

You’ve talked a lot about some of the things that Erlang is really good for. Are there particular applications or types of things that you wouldn’t want to build in Erlang?

Absolutely. So, Erlang was invented for a specific type of problem. And yeah, embedded systems, massively scalable [inaudible] real-time. What it’s not good at are things such as number crunching and usually problems we refer to as parallelism, or parallel problems. So, video encoding, analysis of complex data sets, for example. And what we tend to do is we tend to use a lot of C. If we do need to do number crunching or encoding, we tend to use C. And there are excellent interfaces towards C where you can just embed your C threads in your Erlang VM. We call them NIFs. And we then use Erlang’s concurrency model to handle all of the orchestration, all of the delegation of the tasks and the jobs, and the aggregation of the results. Whilst the really CPU-intensive work is done with machine code or with C, with programming languages compiled to machine code.

Other areas, graphics. Go to any Erlang website or any website run by a company which does Erlang, you’ll realize we’re not that good at web development. We’re not that good at frontends. Even though it’s ideal for the web, I think there are very few tools and libraries which allow you to do nice, elegant frontends and web frontends. It’s ideal for REST APIs. There’s a really good library called Webmachine which can scale massively. And in those areas Cowboy is another excellent example of using it for web, but not for interfaces or websites itself.

So, you mentioned the toolchain a couple of times and you talked about real-time debugging. But I’m wondering if Erlang lends itself to TDD. What are the frameworks like?

There are two frameworks which are widely used. And the bigger framework is a framework called Common Test. And it’s actually used by Ericsson to test C code, so to test all of its mobile base stations. And that’s more to test systems. You then have EUnit which is a lightweight unit testing framework which you use on a module basis, or even on an application basis. And thirdly you’ve got a tool called QuickCheck. There is an open source variant called PropEr. QuickCheck and PropEr, what they do is provide property based testing where you go in and you create a model of your system.

And so, assume you’re testing an API. What you do is you say, “Okay, this API will take integers between 1 and 100 as the first argument and it will take a string of up to 255 characters.” And what QuickCheck and Proper do is it’ll start generating a random sequence of tests. So, instead of having one or two tests, every time you run it you’ll run tens of thousands of tests. And when it becomes really powerful is when it starts… you start running and picking up on real borderline test cases you wouldn’t use with conventional approaches. Do you have anything similar for Ruby?

There are a few property based testing frameworks for Ruby. Generative is one, and Rantly is another. But they’re not to the same level.

And I’d say they’re not widely used.

Yeah, yeah.

The idea of property based testing isn’t widely used in Ruby.

No. if you look at QuickCheck, it is a commercial product. You’ve got QuickCheck CI available for open source projects. But then again, you’ve had university professors work on it for well over a decade now. And it’s solid. It will pick up race conditions, concurrency errors. It’s being used… [inaudible] automotive industry to test code. It’s being used to test radio base stations. So, it is incredibly powerful but it’s not for the faint of heart. You pick up a lot of bugs which you would otherwise not see. But it’s almost as much effort in writing your QuickCheck properties as it is in writing your code, the code itself.

I would argue that most of that effort goes into the thought of figuring out the properties. And that all of that effort pays off in the code itself.

Oh, absolutely, absolutely. What happens is you usually… you should have two people looking at the particular protocol or problem. And so, you end up getting two views of your system. And when you’re testing and you’re running your QuickCheck properties, I’d say about half of the issues you discover are in the QuickCheck properties themselves. The other half are in the code. So yeah, code-wise it’s not that much. But the thought process which goes into it is [excellent]. And it actually forces people to think and reason about their programming in a completely different way.

And that’s ideal. That’s how we do test-driven development here at Erlang Solutions.

Yeah, in that sense property testing fits with the Erlang model of not ignoring failures, not ignoring edge cases, reasoning about far more than the happy path.

Exactly, that’s correct. Yeah. And I think we could go into stories over the strangest borderline cases which were discovered through QuickCheck where you ended up generating buffer overflows when following very particular patterns. Or even bugs in Mnesia.

So, Mnesia is a distributed database which comes with Erlang. Mnesia’s been in production for I’d say 15 years. It’s been used for over 15 years. You’ve got millions of users and it’s millions of queries. Their systems which handle millions of queries a day. And despite that, as they were looking at running Erlang on machines with more cores, you actually started getting true parallelism in your programming, that processes, each running on their individual cores were running in parallel. And that gave life to very strange race conditions which no one looking at the code was able to address and discover. We wrote QuickCheck properties and they immediately discovered race conditions among three processes which developers in over a decade had not been able to figure out and address and solve.

And I think one of the things we have not mentioned when it comes to QuickCheck is the ability to do shrinking. When QuickCheck finds a system not behaving the way it should, so returning a value different than the intended one, you say it could be a string of any size, you go and you start. You pass in a string with a thousand characters. What it does is it tries to shrink that string to the minimal common denominator. And once it’s done it, it will give you back that minimal common denominator. So, assume that a string with the character ‘00’ causes your function to crash. There’s a good chance QuickCheck will manage to reduce a string with a thousand characters which, there might be two ‘00’s in there, down to the ‘00’s.

Clearly, most of your programming life is Erlang. I’m curious if there’s anything else that’s on your horizon. Are there any other languages or technologies that you find yourself fascinated by right now?

Right now, very much Elixir. And that’s one area where we’re doing a lot of (well we’re looking into), which is built on the Erlang VM. And I think it’s maybe not Erlang per se which is fascinating, but I think the semantics of the Erlang programming language. So yeah, another thing which really fascinates me is large-scale distributed computing. And that’s really where my head is these days as well. We’re trying to think, “What do we do with the Erlang ecosystem and the Erlang VM and where do we want it to be in 2020?”

And one of the ideas we’re playing with is actually adding a soft switch to the Erlang virtual machine. And then using software-defined networking techniques such as OpenFlow to set up and tear down connections among different Erlang VMs. So, try to picture spawning off 30,000 Erlang VMs. It takes a few milliseconds if you’re using Erlang on Xen and have enough hardware to do it. And as these Erlang VMs start communicating with each other, you optimize the connectivity between the VMs based on layer two and layer three traffic. With this you’re basically able to create networks with tens of thousands of Erlang nodes and optimize the data traffic between them based on either latency or bandwidth. And I think we’re putting in a research proposal. So, we try to get grant and funding to make them a reality.

And we’re working on a project called FlowForwarding which is an OpenFlow switch, the LINCSwitch. And that has been integrated in the LING Erlang virtual machine which is an instance of Erlang on Xen running. And they’re doing tests in that space right now. But still, a lot of work and a lot of research still need to go into it. But the idea is there. And it’s the idea of massively scalable distributed systems. So, that’s really where my passion lies these days.

Francesco, I was wondering. As the founder of Erlang Solutions you mentioned that you don’t have to worry about language maintenance and you don’t have to finance that, which must be really nice compared to Clojure and Scala. But you do, do a lot of work in the community. What is the Erlang community like?

It’s, if you go to any Erlang events what you’ll see is you’ll find, especially in Sweden, you’ll find students studying in university really excited about Erlang all the way up to people who used to work with Erlang at Ericsson back in the late 80s, early 90s, who’ve now retired. So, it’s incredibly diverse and it’s incredibly wide. And also, Erlang being so specific… just take Erlang Solutions. I think we’re about 100 employees right now. And I would guess we have about 20 nationalities represented within the company. So, it’s a very open, very friendly community.

I think the focus is around the Erlang mailing list where a lot of the debates and discussions take place and where a lot of people go in and ask for help. There is a website called Erlang Central with a chat and forums for those who don’t, who prefer alternative ways. It also has a huge collection of tutorials. And also, very [inaudible] we have a lot of researchers and a lot of academics attending our events. You will have speakers and attendees from many universities. And I would guess probably 10 to 15% of all the events are attended by academics, PhD students, and regular university students. It’s an incredibly high number compared to most of the other events I’ve been to.

Cool. That’s pretty [nice].

Yeah. Everyone gets described as friendly and approachable. And I think yeah, it’s up to you to judge when you start looking at Erlang, playing around with it. See if you get the help you need.

So, when am I going to be able to write iPhone apps in Erlang?

I’m not sure we really want to go down that route.

I do admit, played a lot with the idea. And we’ve played with the idea of having a virtual machine where you have only one instance of the Erlang VM running on an iPhone or on a cellphone, on a smartphone. And then all the different apps do a bit of timesharing on that VM, which would mean you add security in there. But I think it’s probably a race we’ve lost right now. It’s partially because it has to do with user interfaces. On the other hand you buy a Parallella board, you’ll find Erlang installed on it. It’s ideal for the Raspberry Pi and a lot of people are using it on Raspberry Pi’s and smaller embedded devices. That’s where maybe user interfaces aren’t critical. So, you want to maybe run a web server on your Raspberry Pi. Erlang’s ideal for it.

I was only half serious but that was a great answer.

Yeah.

Yeah, so you can write the backend of your iPhone app in Erlang and have it be very highly available.

Yeah, having said that a lot of the server side code is, in a lot of iPhone apps, popular iPhone apps out there, is Erlang. And I could mention Whisper, the secret social network. Or WhatsApp is another, just to pick two examples out of there. All of their backend is all written in Erlang.

I actually really like the fact that you don’t jump immediately to, “Well, we have this language. So, we have to use it for all of the things.” You’re very focused it sounds like on tackling the problems that it’s best suited to.

That’s correct. I think we’ve been around the block a few times. The language has been around for 20 years. So, I’d say we’ve burnt our fingers a few times and we’ve done our mistakes and we’ve learned from them. And we’re now looking forward and try to leverage our strengths and whilst being aware of our weaknesses.

That sounds like a great conclusion. [chuckles]

Yeah. Let’s go ahead and get to some picks. Avdi, do you have some picks for us?

I guess my first pick is sort of a practice rather than an individual thing. There is more out there that we can learn about as programmers than we could ever have time for. There are all these languages, Erlang one of many. And there are all these technologies. And lately I’ve been more and more impressed with the idea that I’m never going to have time to learn all of the things that I want to learn or that I’ve found interesting or I thought, “Oh, I should dabble with that sometime. Or I should try that out.” And so, the other day I started thinking really pragmatically and seriously about this. And went over my mental list of programming languages that I’ve always thought, “Oh yeah, I’ll learn that language someday.”

And I really went down it very critically and did some triage. I did some mental triage and decided, “Okay. This is something that I need to [learn].” I came up with some broad ideas of what my goals are as a technologist. And then went down that list and thought, “Is this something that learning it will advance me towards my goals? Or is this something where I just think I should learn it because it looks nifty or because my friends like it or something like that?” And a lot of the decisions are based not on ‘What practical project am I going to use this for?’ but more like ‘In what way will learning this language stretch my brain?’ Is it a way that I need my brain stretched right now? Or is it a way that I’ve already had my brain stretched and maybe it’s time to stretch in a different direction now? So, I spent some time on that.

I wrote a blogpost about it where I listed the languages I wound up with. And yes, Erlang is on that list. And I also, and then a longer sampling of a lot of the stuff that I had to toss off the list because looking at it very frankly, I realized it was not something that was going to advance me towards my goals. So, I guess I’m not sure how to sum this up. But it’s basically the practice of being honest with yourself about what you really have time for and getting serious about sitting down and studying a few things rather than playing with this and playing with that and never really going anywhere with it. So, I’m really trying to do that now. And I’ll put a link to the blogpost in the show notes.

And related to that, I mentioned this a little bit earlier in the episode but the list that I came up with, I hadn’t actually intended it to be sequential. But Smalltalk happened to be at the top of the list. I’ve always wanted to get deeper into Smalltalk and understand Alan Kay’s ideas that really poured into Ruby. And it wound up just sort of randomly at the top of the list. But I decided, what the heck, let’s start at the top. So, for the past few days I’ve been studying Smalltalk via Pharo Smalltalk. And Pharo just had a release 4.0. And it’s getting really nice. It’s an open source Smalltalk VM that you can run on any platform. And there are some good resources out there for learning it.

And if anybody’s familiar with Squeak, it’s based on the Squeak project. But rather than being a big happy sandbox where everybody threw every possible little toy into, it’s more focused on professional developers. And I’ve really, really been enjoying that. I also did a little blogpost with a video of some of the more exciting things that I’ve found so far. And I’ll put a link to that in the show notes as well. So, that’s Pharo Smalltalk. And that’s it for me.

Alright, Coraline, what are your picks?

I have a couple of picks today. The first is a Nebula award-winning sci-fi novel called ‘Babel-17’. It was written by Samuel Delany and it’s kind of old. It was published in 1966. But one of the things that attracted me to the book is that it provides… it does an exploration of the Sapir-Whorf hypothesis which I’ve talked about in a couple of talks. And that basically posits the theory that spoken language influences and constrains our thinking. The book deals with a language that is actually weaponized. It’s called Babel-17 and it figures in a large-scale conflict between a federation style government and a rebel movement. The language itself is interesting. And we get to see over the course of the novel how it influences the thought and ability of the story’s hero. Interestingly, the language lacks pronouns and it has no conception of the word ‘I’. The idea of a weaponized language that Delaney explored in this novel was later used also in Neal Stephenson’s ‘Snow Crash’. So, I’ll provide a link to the Amazon page for ‘Babel-17’.

My other pick, anyone who knows me knows that I’m a really big fan of classic movies. And Turner Classic Movies this month is featuring a celebration of Orson Welles who would have turned 100 on May 6th of this year. So, their series includes his famous noir films as well as Shakespeare and other literary adaptations. And I strongly recommend that you take a look at some of the roots of great cinema and check out Orson Welles on Turner Classic Movies.

Alright. Jessica, do you have some picks for us?

I have two picks. One of them is a really quick repeat pick because we talked about QuickCheck and Erlang not long ago. And Francesco mentioned Erlang CI. So for that, I recommend John Hughes’ talk at CodeMesh last year which I’ll provide a link to.

Okay. Now, we’ve got the important pick and this is a pick that I’ve been waiting on for an episode that was worthy of it. but if you like to stretch your brain, then you should get the book ‘Vehicles’. It’s a tiny little book. You can get it on Amazon. It was written a couple of decades ago by Valentino Braitenberg. The book is a though experiment. It asks, “What if we built some sort of little car with a sensor and a wheel, or two sensors and a wheel, or we [cross] the input between the sensor and the wheels so that they go at different speeds under different stimuli?” And it builds this up, just these simple little imaginary vehicles. It builds them up until you’re modeling a human brain, really. And at every step of the way, the progression makes sense. It illustrates its principle of downhill synthesis, uphill analysis. It’s easier to understand what you’re making than to come later and figure out how it works, as we have all seen if we’ve ever gotten a job working on a legacy application.

This book, it’s a thought experiment. So, it depends where you are, what you’ll get out of it. But after 25 pages, I suddenly understood religion, which is kind of impressive. Okay, that’s my favorite pick ever. The end.

From my end, I’ve got a book which Chad Fowler actually recommended. We were having a conversation at CodeMesh last year over how as CTOs you have to have crazy ideas and turn these crazy ideas into reality. And I was… we were discussing my vision of Erlang in 2020. And he went in and actually suggested a book called ‘Zero to One’ by Peter Thiel. And it’s basically looking at innovation and thinking out of the box and how you go in and you create something new, so the whole idea of going from zero to one and not from one to N. At least the parts I’ve read so far are very inspiring. And I think it’s a must-read for any entrepreneur or anyone who wants to get into the startup world.

Alright. I’ve got a couple of podcasts that I want to pick for the show. the first one I’ve picked on the show before. But it’s just gotten so good. It’s the CodeNewbie Podcast. Saron’s not on today but she does that podcast and it is awesome.

Another one that’s really fun to listen to is an NPR show called Ask Me Another. And I’ve been enjoying that. It’s just a word game kind of show but it’s fun to listen to and get stuff out of.

And the last podcast I’m going to pick is called Startups For the Rest of Us. And that’s by Rob Walling and Mike Taber. And it’s pretty terrific as well. If you’re an entrepreneur and you want to hear two experienced people talking about awesome stuff related to their businesses and what they’ve learned and giving good advice on how to push your business ahead, then I have to recommend Startups For the Rest of Us. So, those are my picks.

I also want to remind you to go check out Ruby Remote Conf, RubyRemoteConf.com. And unless we have any other announcements, let’s wrap up the show. And we’ll catch you all next week.