In this episode of Elixir Mix, Chris Dosé joins us to talk about some of the open source work that he has done at Peek. He also talks to us about his exciting Nerves projects Xebow and AFK.

Hello, everybody. Welcome to today's episode of Elixir Mix. I am Sophie de Benedetto, and I am joined by some of our still relatively new panelists. We have Lars Wickman. Hey.

Did I say your last name right? Close enough. Close enough? Okay. Cool.

We have Alex Koutnos. Howdy. Howdy. Hey, Alex. And we have Steven Nunez.

Hello. Hey. And we are very excited to welcome today's guest, Chris Dose. I said that right. Right?

Yes. That's correct. Hello, and thanks for having me. Awesome. Welcome.

So, Chris, we're super happy to have you. Do you want to kick us off by introducing yourself? Yeah. Well, I've been, a software engineer for a little while now. I was I had to, you know, check back at my LinkedIn to see how far back it actually goes.

Looks like I've been doing this for about 13 years now. I started out down in, Louisiana, at LSU. I did some I taught workshops there as a student, in, software, primarily things like HTML and CSS and JavaScripts, but, you know, a little bit of PHP as well. That was my first language. And, from there, I got some jobs in, in Louisiana, but decided that that wasn't the place to really, propel the career I wanted.

So I moved to San Francisco and, got a job with Peak, as a software engineer there, and have been with Peak, since then for about, a little over six and a half years. And at peak is where I, got into Elixir and, started learning that. Super cool. I I haven't heard of peak just in terms of it being, a company that is using Elixir, so it's always exciting to add a new entry to, like, my mental list of companies that are using Elixir in production. Were they working with Elixir before you got there?

No. We, we were using Ruby primarily, before, before I got there and when I got there. So for the first, 2 years while I was there, we were primarily writing in Ruby, kind of going in, service oriented architecture direction. And then and I wasn't aware of Elixir, actually. And we we we hired a new engineer who was a big proponent of Elixir, you know, like an evangelist, and said, hey, you should really check this out.

And I did, and he completely converted me. So I became an evangelist for Elixir as well, and I suggested that we start exploring it for, for future services. Since we were going in a service oriented architecture direction, we could, you know, experiment with smaller services and say, hey. Does this work? And, thankfully, it did, and we kind of got everybody on board with that.

And, and we use, Elixir now primarily even though we've got plenty of legacy, systems around that are still in Ruby. We don't tend to build new things in Ruby. We we build all of our new stuff in Elixir. How did you find the process of either hiring Elixir engineers or ramping up, you know, Ruby developers and and getting them, familiar with, Elixir and the Beam? How did that process kinda go?

Well, thankfully, I guess, we're we're a pretty small, engineering team. So, the the ramp up process for the existing engineers on Ruby was was pretty pretty easy for the for the engineers that we had. We, you know, like, we've got some smart people. They just kind of got it, you know, and we we all kind of learned together, you know. I mean, we were all new to it.

And, and yeah, that that worked out pretty well. As far as, hiring, hiring new Elixir engineers, it definitely is harder. The pool is is much smaller. But we we've always been open to, you know, hiring anybody from any, programming background as long as they're interested and willing to, learn Elixir. Like, we you know, we're we're perfectly happy to have, you know, people that want to learn Elixir, but haven't had an opportunity yet, you know, in a in a professional setting to, to use Elixir.

So, you know, we were looking for people like that. Unfortunately, of course, the the the recent pandemic and stuff has drastically slowed us down, and, we've been on a hiring freeze ever since then. So that's kind of that's kind of been on hold. I'm always really curious about sort of teams that transition into Elixir. So where I'm at now, we, same kind of story predominantly went from, like, Ruby Rails background to, some psychopath came in and convinced everyone to go and look at this Elixir thing.

It was me. Who was that again? I don't I forget. It's a I I yeah. So we started building more and more apps of it with it, and then it's kind of like interesting because the journey is not just like, hey.

There's pipes here in gen servers. It's kind of like trying to figure out what are the learning materials you need to get in front of the team. How do you sort of organize it? Like, we have smart people too. We also realize that if you don't if you're not careful about, like, how to convert an o o mindset to a functional mindset, you wind up with some really gnarly looking code.

Did you guys run into that at all? And if so, what resources did you sort of use to, books, guidance, any sort of like, general tips? Well, yes. Definitely. The early experiments in Elixir were definitely a little weird.

Especially since we tried to kind of out the gate try some, try some funny business that, like, would have been a lot harder to do in Ruby, but we're like, but we're using Elixir now. We can do this. And those kinds of things turn out to be bad ideas in the long run, but, you know, just like as an example, we had, we had one of these integrations that we were building where we needed to, like, the the external API we were integrating with was asynchronous in some places. But our API is synchronous. So, you know, okay, fine.

So why don't we just have, you know, a little layer here where the internal synchronous request, HTTP request comes in, we spawn up a gen server that holds on to that HTTP request and then does the async actions with the external API. And then whenever that async action is done, then we can release the response, to the, you know, to the HTTP client there. And, yeah. It's it's funky, and, you know, ended up just being really hard to debug, and, I I wouldn't necessarily recommend it. I feel like Of course.

You me? I think we did that same thing in one of our earlier Oh, yeah. I'm not maybe not exactly the same, but very much a situation where you have, like, an incoming web request, and then you're just doing something kind of overly complex with gen servers and, you know, you're blocking a response until 15 different things, resolve. And, yeah, I think the idea of it being extremely hard to debug with absolutely, yeah, resonates. That That is sort of interesting that because Elixir gives you those tools to do it, like, you can be like, oh, I can kind of imagine how, like, a set of persistent servers running in, like, the background can hit this is perfect.

And then then it hurts hurts hurts so bad. But then that's that's part of learning, but I think I think worth exploring because I think it helps you understand sort of, like, the edges of, you know, what these tools can do for you. One thing that I know I was, excited to ask you about, Chris, is it looks like peak at peak, you guys have 1 or 2 or a couple different Elixir open source libraries that you maintain. And I think it's really exciting not only to see another company using Elixir in production, but actually supporting and giving back to the Elixir community in that way. So I would love to hear a little bit about the open source libraries that you guys developed.

What did they do? Why did you build them? What's it like to maintain them, you know, as you use them in production? Yeah. I am also excited about being, given the opportunity to develop and maintain these open source, libraries.

Thankfully, like, Peak is very open to that idea of, you know, giving back to the community. And so it was it was largely my initiative to, to to carve out some of the important bits or, you know, some of the things that we thought could make good libraries and, and, put them up on this, on this, GitHub organization, page here. And, you know, it it might not be a surprise that a lot of the libraries that, we do have available are specific to, what we do at peak, which is, building software for the tours and activities industry. And naturally that involves scheduling, availability on calendars. So I've got 2 libraries.

The first that I developed is the one that we are actually using in production. It's it's live, a library around, generating date recurrence, date date instances based off of date recurrence rules, so that you can, you know, schedule, things on your calendar in complex weird ways that, you know, people have. And we just needed, you know, we needed a way to to go from some, like, you know, small set of rules that, like, define a schedule and be able to expand that out into concrete instances of time. Right? So that's that's one of the libraries, that I wrote there.

It's called Cocktail. I have a similar one, called XCal, which, which is is another interesting thing to talk about. And then and then we've got a couple of other little small libraries that were just kind of like little tools that we needed for various things. The the most notable is probably Ecto diff that, that gives us a way of, getting, getting, like, a a a data structure that describes the changes that you performed after the fact, to to an Ecto schema. Especially in the context of, deeply nested Ecto schemas using, associations, like has many and belongs to associations.

And that's and that's been very helpful for us. We we definitely do use it in production, and it is useful. How have you found, participating in the open source community in Elixir compared to I don't know if you did any open source, let's say, in Ruby, which it sounds like you had a background in, or if anyone else has comparison open source contributions to share? Well, me personally, I didn't really contribute much to open source other than my own libraries, even before Elixir. And and these libraries that I've got here, for peak are not, you know, not exactly super popular.

So I don't get a lot of the, you know, like, I don't there's not a lot of, like, you know, action going on there. So I I don't I don't have a lot of experience with that yet, but I would I would like to have it. That is changing a little bit with a a side project that I'm working on where I've actually do have, you know, multiple people from the community involved now, and that's just starting out. So we'll see how that goes. And that would be the work with the what's it called?

KiboKit? Yes. That's the, the our our our initiative to build Elixir powered mechanical keyboard firmware. Yeah. So could you dive into a bit what you've been doing there and what your end goal would be?

Because the keyboard kit, as far as I understand it, is a mechanical keypad powered by Raspberry Pi. And for people who aren't familiar with NURBS, NURBS runs on Linux devices, such as the Raspberry Pi. So that's a good target for NURBS. And the keyboard kit is fairly, let's call it a constraint space. It's a small keypad rather than a full keyboard.

So I'm very curious what the what the both what the project is right now and where you hope to be headed with it. Yeah. Yeah. Like, like you said, it's it's a it's a small keypad. People also call them macro pads because the intent is that you, you know, it's it's like a little additional pad next to your keyboard in addition to your primary keyboard that you can assign various macros to and, like, you know, have various little functions that you have on your keys, that you perform, you know, daily and you want to automate.

That's kind of the purpose of of macro pads, but we're using this, Raspberry Pi powered, Kibo, as kind of like a prototyping, air, a prototyping device for us to see how, you know, how to develop a a NURBS based firmware for for what could eventually be a full a full sized keyboard. As far as I know though, there are no full sized keyboards that are powered by something like a Raspberry Pi. It it has to NURBS only targets, Linux machines, like embedded devices that run embedded Linux. You can't go all the way down to something like an Arduino, a microcontroller. And most mechanical keyboards, unsurprisingly, I think are powered by very small microcontrollers that are very, unsurprisingly, I think are powered by very small micro controllers that are very, very definitely not powerful enough to to run Linux and the Beam.

So this is definitely an experiment. It's, it's going really well, I think. We've, we've got a lot of really great libraries development, and, and it's fun to just, play with this thing and see what we can do. It's got, you know, it's got a grid of 12, RGB LEDs on it. We've been playing around a lot with, building out fun animations and stuff.

You know, if you've seen if you've seen some of the fancy mechanical keyboards that you can that you can buy out there, they've got, you know, full full backlight RGB LEDs that, you know, animate in all sorts of fancy ways and stuff like that. So that's kinda what we're shooting for. The the the the end goal being that we want to actually design, you know, the the keyboard itself. You know, we we need to design the the PCB itself, and and get that get that printed and, and actually, you know, design a case, around it, and, you know, hopefully, have that available, at some point. This is a super interesting project.

I'm also I'm also very interested to see where it goes. But, as of today, what could the people that are interested do with the Kibo kit and, the things you've been working on? From what I see, you have separate libraries for, like, the xibo. Feel free to correct the pronunciation there. And then the a f k library, which seems a little bit more general and for the for the wider mechanical keyboard ID.

Is that about right? Yeah. That's, that's correct. The idea is to use the the Kibo as our staging ground for the libraries that we're developing. So one of those libraries is called, AFK.

That is, it's the keyboard state management library because, it it it can actually get kind of complex, if you actually take a minute to think about how keyboards work and, you know, how you have to manage, like, which keys are being held down at which times and which ones get released at which times. And, and then when you throw in the the the fancier features that we want, like for example, layer layer keys. You hold down a key and it switches the entire keyboard to a different layer. And things like macros and and, other other fancy features like, like this feature that we call tap keys, where you can actually have a key, a physical key act differently depending on whether or not you tapped it versus whether or not you held it. These are all features that come from the wider mechanical keyboarding community.

But the by far the most popular, firmware out there, open source firmware for mechanical keyboards is called QMK. And it's all written in c. And I'm not a c programmer, but and and therefore, it's very difficult for me to understand what's going on in there. And to me, it kind of looks like, it's it's like a it's a it's a big, you know, very large project with many many contributors that has has kind of, like, gone and sprawled. So it's very hard to contribute to it and understand it.

And so my goal was, well, I'm not a c programmer, and but I'm a an Elixir programmer. I can I can do this in Elixir, and maybe in the process also make it a little simpler and easier to contribute and and be able to build higher level features, you know, in a higher level language without having to drop down into c, to to do that kind of stuff? In that same spirit, is, is this something that, like, you know, someone new to NERV's could jump into and start, contributing? You know, if not, maybe what are some some starter projects that really help kind of get you ramped up with what NERV's is, what the development process looks like, and, you know, how how to get familiar with all that? Yeah.

Well, there are definitely things that you can jump in on on the Xibo, firmware, which is what we've been developing. There's there are things that, you know, someone who's not familiar with nerves can jump in and start contributing with. I wanted to, you know, mention a a couple of other features that are actually on the, the in the Xibo firmware that I that I forgot to mention earlier, which is that, it it it, it runs Phoenix from the device itself. And the USB cable that you use to plug it into your computer with actually, you know, opens up a a virtual ethernet device. So you know, the device is actually connected to your computer as a USB hid gadget, you know, like as keyboards present themselves.

But it also presents itself as an ethernet device so you can SSH to it. You can get into a shell that's running directly on it. You can run a Phoenix web server from it which we are doing. And so you can use your browser to visit the Phoenix live view interface that we're building for it and you can, you know, change various settings about, about your your keyboard keypad. You can you can, configure the animations.

You can you can even preview the animations, in the browser itself. And we got this we just, we just finished this last week. It's, it's very interesting. It was a very fun project doing this where we have the animation engine running on the device itself that's, you know, painting the keys various colors. And and but the but the actual physical LEDs are just an abstraction to that.

And we have the Phoenix Live View interface, which is also just an abstraction on top of that. So really, like, we have a a common engine that's performing the animations, but it draws both to the physical keyboard and to the live view page at the same time, and it's and the events are synchronized. That's nice. That's pretty cool. Yeah.

I would have never thought about having a an Ethernet device over USB. So now next question is, when when is the Kickstarter going up and how can I sign up for one of these custom PCBs that you'll be printing? Yeah. Good question. We are definitely still in very early days here.

We're still prototyping the software to make sure it makes sense. And of course, the, PCB is completely not designed yet. I have not I do not have a background in electronics. I don't know how to design PCBs. So there are many tutorials in my future, on, learning this stuff and figuring out how to design it right.

And, you know, we're gonna need to go through several rounds of prototypes. That's not to say that I'm not, you know, completely new to this scene. I, I have, you know, I did kind of jump head first into the mechanical keyboard rabbit hole, a little while ago before trying it with Elixir. So I have, built several keyboard kits myself. But but the actual step of designing your own PCB from scratch, that's that's the next hurdle that I don't really know how to get over.

Definitely let us know because I'll give you my money. Yeah. It seems like there would be a lot of interesting things to a project like this even, looking at if you're using a Raspberry Pi for a keyboard, you actually have a computer in the keyboard, a full computer in the keyboard, which sort of makes me think of the some Amiga computers I've seen, which are a computer and a keyboard. But also, I started wondering, could you use an existing keyboard design and mostly hook up the correct electronics to actually drive an existing design? How feasible would that be?

Have you looked into it? Yes. Actually, the the Kibo is actually not my first attempt at this. I I did actually build, an existing PCB design, that I found online and got, I I sent off the the designs over to a lab in China and got 5 of them printed because that was the minimum number that you can print. And I, I put together this keyboard, kit like completely from scratch myself.

I like got all the, you know, the metal plates laser cut and and soldered the PCB together and soldered all the switches on it. But I was originally following a guide that, used an Arduino, microcontroller. So that was my first keyboard. I put it together with the Arduino microcontroller and it's running the QMK firmware written in c. The second time I had, and of course I had a couple of extra PCBs lying around.

So I took the I took the existing PCB and I did connect a, I originally connected a Pocket Beagle, to it from, from BeagleBone. So that's another that's another development board that I I would also highly recommend. BeagleBone, Black, and Pocket Beagle. They they also run NURBS. So I I used one of those, with the existing PCB.

Of course, you know, like the PCBs were designed for the Arduino, so the the the pin holes didn't line up. So I, you know, I I hacked it together. And and that is also that was my first first ever prototype. That one is also running Elixir and NURBS. It's not running the Zibo firmware.

It's running a different one, and it's a little older now. But a lot of the learnings that I did from that one, went into the the Xibo firmware. So what you're saying is you have a you have a full working mechanical keyboard, with nerves on it? Yes. I do.

But, it doesn't have the full RGB LED backlight, so I'm sad. That's no good then. You're only half as productive without the LEDs going. So that's why I got that's why I gotta take it to the next level with the LEDs. It seems like you might have a lot of options down the road as well.

Have you looked at the GRIPS 2 board? I don't think that's done entirely yet. Or and then there's also, of course, Lumen, which I tend to mention on every episode, which will build to web assembly. And as far as I know, web assembly does run on the ESP 32, which is a microcontroller. That is very interesting.

I will definitely keep my eyes on that. I was not aware that, Lumen was targeting, targeting embedded hardware. So that's that's very interesting to hear. I, as far as, the Grist 2, yeah. We've we've been aware of it for a while.

I actually have. One one of the other community members who's part of this, Kibo project, pre ordered 2 of the GRISP 2 boards. And they were supposed to have been here, you know, 6 months ago. But they've been, severely delayed. So we're we're still we're still waiting on those to see, you know, what the viability, of those boards are.

Maybe better to build with something something that actually exists for now. And in the case of Looman, the target the first priority is basically web web assembly for web, but I don't see any reason why, the end web assembly product wouldn't work on microcontrollers. So changing gears back to the calendar library you were talking about, I've always looked into, like, Google Calendar and seeing that you can set a calendar event to repeat according to a rule set that can be pretty intricate. And it just figures out when it should place these events on your timeline, and in your calendar. And the thing I've always thought about a little bit and wondered, how do they do that effectively and efficiently is when you remove or change singular events in a repeating one or move one of them.

It it seems like a terrible headache, and I imagine this is what you've been getting familiar with. Can you speak to that? Yeah. Definitely. Like I said, I've built 2 of these libraries now.

Although, one of them maybe doesn't count as much, but we'll we'll we'll get back to that. This, this concept of calendars and date recurrence rules and things like that has been around for quite a while now, and it's, it's it's, an official, IETF proposal. I've I've pulled it up here so that I get the numbers right. This is actually RFC 5545. It's called the Icalendar spec.

Right? It's it's it's like, you know, a lot of a lot of smart people and a lot of smart companies have come together to kind of say, this is how you do it. Right? This is the spec. This is how you do calendars with recurring recurring events.

So this spec is huge. It's a very very large spec and, you know, like, reading through hundreds of pages of specs is not, you know, not anybody's idea of a fun time, I don't think. And, but but I did, you know, spend a good amount of time reading the sections on how they define recurrence rules and how they're meant to be interpreted. So the basic idea behind this is that you've got a concept of an event, and that event can have any number of recurrence rules in it and the recur the recurrence rules act as like a set union. And and then within that event, you can also have any number of exception dates.

And those exception dates just just like set minus. They just remove that that recur that occurrence from the set. Right? So if you got a simple event that's like every Wednesday off into infinity, but then you wanna like cancel this one on this one specific Wednesday, you just put that specific date time recurrence into your exception set. And then you can also, of course, add arbitrary dates into the set as well.

So that's the basic mechanics that you got there. And but, like, all the complication really lies in the recurrence rule, construct. Thanks. I figured it had to be something like that, but I've always wondered. One of the few things that I have not enjoyed about working with Elixir is working with dates and datetime.

I don't think that's unique to Elixir. I think that can be one of the biggest headaches, really, of programming in general. But in particular, I feel like it always takes me longer than I want it to to do anything with date or datetime. So I'm curious if you ran into, like, any headaches or gotchas when devbing on these calendar libraries. Yes.

It is generally a universal, thought that date and datetime is hard in most programming languages. And, and you know, we at peak, of course, deal with it a lot. So anytime a bug comes up around dates or time zones specifically, we know that it is gonna be a hard bug. Right? Time zones are one of the hardest things to get right and to and to figure out.

Right? But, that being said, I honestly think that Elixir may not have gotten it completely right, but Elixir is one of the languages that has, you know, in my opinion, some of the some of the best date and date time handling out there. It they they approached it, you know, they didn't they didn't approach it like all at once. Like some languages, like, have to have like a fully fledged date time implementation all upfront. Elixir, like, backed off and said, no, we're gonna give you some, like, basic building blocks and it really doesn't do much, you know.

So if you need to do something fancier with, like, time zones and date math and date addition, then you gotta go to another library that handles it. Like, and and and, you know, for the longest time we had Timex, which, you know, is a is a good library. I dealt with it a lot, while building, my library cocktail. And I dealt with the performance problems of it a lot. I opened several issues with the maintainer of Timex, wondering, you know, what's going on with all of this, with all of this slowdown.

And and really, you know, the the ultimate answer is just that this is hard, And and, and dealing with time zones is hard. And and like, you know, adding to date, or you know, like adding time to one date time, is a very hard question to answer. Especially if it's like in an in a specific zone. Because in in specific time zones, sometimes don't exist. Sometimes exist twice.

It it gets it gets very complicated and hairy. But, but the the library that I've I've written, for the recurrence rules, it's called cocktail, and it's open source on our, on the peak travel, GitHub organization. It it, as far as I know, works pretty well, has good test coverage, and and I'm I'm not sure if there I mean, I actually, okay. There is there is a couple there are a couple of file bugs, but, I'll take care of those at some point. But the way it works, internally is that you you build up your recurrence rule sets that I described earlier.

And you you put them together into into a into an event. And then from that, you can generate an Elixir stream. A lazy stream. Right? That's that's what it all comes down to.

You want you want to set up your rules, you put it together, into a data structure and you put that into a a lazy stream. And then from that stream, you can you can take. Right? You can take as many instances as you want, and you can you can, you know, if you wanna say, like, give me the next 30 occurrences, you just enum.take30 out of that stream and it gives them to you. That's pretty cool.

I can definitely relate to the, the date time problem. I was working at a logistics company a few years ago, and we had, like, we're tracking drivers. And there were some cases where, like, a driver on daylight's, savings time, like, that that time would drive across the time zone. And so, like, we would, for whatever reason, you know, super edge case, we would negate their time. And it's like implementing the fix for that particular case was just like if statement after if statement after if statement to try to make sure, like, their time bookkeeping was totally right for that, you know, block of 6 hours that was just ridiculous.

But now I could totally feel that pain. Yeah. I actually have some advice, I think, to anybody who runs into these kinds of problems. What we ended up doing, you know, as like our, you know, I don't know, 3rd approach, to to this problem was just to remove time zones completely out of the mix. Like originally, like some of our legacy systems, right, every date time instance in the system was a a zoned time.

It was like this time in Pacific time or this time in Central time. And and you run into all sorts of issues like exactly what you described. Right? You you run into daylight savings times issues, you know, like this time was valid, you know, you know, like this time that was, you know, persisted in the database is a is a daylight savings time, but, you know, it's no longer daylight savings time. You gotta take that into account.

It gets really hairy and and really confusing, very fast. So our our our last approach when we switched to Elixir and, you know, rewrote a large chunk of our system in Elixir, we said we're taking time zones completely out of the mix. Every single time, in our system is a naive date time, until you need the zone. So you you what you wanna do is you wanna apply the time zone to your naive time as late in the process as possible, like, right when it goes out the API to be displayed to the user or, you know, right when you need it to know whether or not you have to, like, enforce booking cutoff rules, you know, or something like that. So but throughout the entire rest of the system, we just store naive date times and that has drastically simplified, the working with dates and times for us.

Yeah. I'll definitely plus one that. That's that's kinda the route we went down. Everything was being after the fact, we started everything as UTC and then it was just presentation layer. How do you convert to the time zone that makes sense for the viewer of that, of that data.

So, yeah. Definitely definitely a good way to go. Right. And actually, just a small detail there. That's one of the things that I think, you know, Elixir did so well.

Elixir has a naive date time, construct. And and that is not the same as a UTC date time. Right? A naive date time is, you know, like 5 o'clock on Wednesday. It's not 5 o'clock on Wednesday in UTC.

It's it's just a abstract concept of a time anywhere. And that's that's what we deal with. Rather than like having actual like UTC date times everywhere and then converting, It's more of, it's more of having a naive construct of a date time and applying a time zone to it after the fact. I definitely know I've seen my fair share of time based bugs. The most traditional probably being a cron job that was running nightly.

And, at a certain point, the client gets in touch and, like, yeah, the sync didn't go this this evening. Something's wrong. And you check, no, that time just never occurred. The actual hour did not exist because it was set to go at maybe 2 in the morning. Yeah.

That's true. 2 AM sometimes doesn't exist. Perfect. All all this talk about time has given me a 1,000 yard stare. Definitely, like, little PTSD from all the the bugs and stuff introduced that.

I don't know. Elixir 1.11 is introducing even more, thanks to the calendar class and date, time class. We finally get stripped time, so that's pretty cool. But I there's something interesting about sort of, like you said, Chris, delaying building out the full library, sort of, like, giving it time to breathe. What does the community need?

What does it what does it actually mean to be part of the, I guess, the standard library, which has been pretty cool. I just wanna punctuate this, conversation about time with one of the keynotes I saw at CodeBeam Stockholm in 2019, which was, why time is evil in distributed systems, which I found enlightening at the time, and, it seems appropriate for now. We'll link that in the show notes. It's funny because the conclusion, like, Chris, that you're recommending and, Alex, that you're saying you've also come to, right, which is that, you know, daytime is naive and you can apply time zone as you get later is exactly the conversation that my team had on at GitHub is having about, you know, time zones being applied to a product that we're devbing on. And, like, the very last thing I did work wise before joining this call was, like, thumbs up the resolve discussion thread that came to this exact conclusion.

So, yeah, definitely with you. Rewind you a bit to running, Elixir in production at peak. How have your experiences been with running, Elixir in production? Do you guys do, like, hot code upgrades? Do you target the VMs?

Do you target, you know, containers? How what does that story look like if you can divulge detail there? Yeah. All in all, I our our move to Elixir in production, was was pretty successful. Especially from like a performance perspective.

We definitely got to massively cut down on the number of, you know, instant application instances we had to run, in Ruby compared to Elixir. And Elixir is just, you know, a lot more efficient. So from, yeah, so from performance perspective, it's it's been really great. We do have we we are targeting containers. We're running in Kubernetes.

We are not doing, hot code, up updating. And, running Elixir applications inside of containers inside of like a container orchestration system like that, does, you know, sometimes, you know, people feel that that kinda goes against, the the the ideas behind the beam. It goes against some of the concepts. And and we felt that. And we we felt that, you know, that kind of like push and pull and and had to kinda work through that.

Like getting, getting all of our, Elixir application instances talking to each other, you know, clustering them together so that they form a, form a node cluster, is kind of annoying and difficult. And making sure making sure that that cluster is stable. We've had problems in the past where we we wanted to rely on, we wanted to rely on, you know, cluster global, states, cluster global processes, for for the purposes of, you know, global locking and and things like that. And and it it works like, you know, most of the time, but then we get like a bad deploy and the cluster never reforms correctly, and now there's 2 schedulers running instead of 1, across the whole cluster, and and that causes disasters. So it has been a bit of a struggle, you know, getting getting that stuff worked out.

But in the end, we just kind of we we just reach for some of the more standard tools. Like, if you need something, that's, you know, like an, like, you know, a a a a a singleton, like a global something or other, then okay, we can just use Redis, or we can use postgres, or something like that. And that works for us. You know, so maybe that's sad that we're not, you know, leveraging some of the, you know, fun capabilities of the beam, but, but it definitely is a lot more reliable, going this direction. But, other than that, you know, one of the things that I think just like is one of the best properties of the Beam itself is just the, the way the scheduling works, the the internal scheduling.

Right? You know, it can, we can have in in in our Ruby applications, if we had, you know, somebody triggers something in our system that was just a little too much for it, you know, we've got a lot of issues where like, people will try to like, massively reschedule a 3 year long calendar or something like that and then it takes our system down. Right? You know, because because Ruby when when one piece of of Ruby gets overloaded, it kind of like affects the entire system. It it, you know, like you start dropping all other requests and there's too much garbage collection going on and and things like that.

Elixir just has been able to handle things like that so much better. And that's because of the, you know, the property of the bean that everything it's it's the the fair scheduler that allows all processes a fair amount of time. So you don't you can't have one request or one part of the system, you know, go completely haywire and take down the rest of it. It's like, you know, if if the system's overloaded, then the entire system's overloaded, which which is which is really nice. A quick follow-up to that.

For, for the tools for for creating the, the cluster in in Kubernetes, are you using Swarm and LibCluster, I'm guessing, or Hard? Which one? We're using yeah. We're using LibCluster, for the for the clustering, aspect of it. I I I was, we we are in Kubernetes, but we we didn't decide to use the the Kubernetes lib cluster, behavior.

We wanted to do it, through DNS, and at the time, the DNS behavior the the DNS implementation didn't exist. So I was actually the one that wrote the DNS, lib cluster behavior and contributed that. It's changed a lot since the last time I looked at it, but, that was that was what we wanted and needed at the time. Because, we we're not exclusively in Kubernetes or rather I mean, we are in production, but we also run the entire stack, like locally on dev machines. And that's using Docker Compose.

So we need a we needed a mechanism that works in both places. So the lib cluster DNS, mechanism is is pretty good for that. We were using Swarm at one time too on top of that, but I don't, we're not using it anymore. We don't really have a need for it. We just we just needed the nodes to cluster together so that a couple of a couple of things throughout the system, you know, had access to the cluster.

When it comes to when it comes to things like, the hot code reloading or hot updates, I seem to find that most people aren't using it because it does add significant significant overhead for just making sure that you're doing perfect, well behaving updates. I haven't tried it. I've never heard of I haven't spoken to many people that have tried it. So I I think the normal case and, like, the 90% case, the 99% case is that people can just use Elixir and Phoenix and all that just as normal, a pretty normal run time. Running into Docker is fine.

There's nothing particularly strange about it, but you can do a bunch of interesting things. And as you say, there are some advantages you can have from clustering up and letting things talk to one another, but you don't even have to. Yeah. Just, we should chat sometime. I have a system that runs with hot code upgrades in production, and, we we started deploying some with, like, regular blue green start stop.

It is so less you know, the stress level is, like, on a different level. Having to manage, you know, the state the shape of my data when upgrading a genservor that's has, like, clients interfacing with it. It's, the different set of problems. It is cool, and I think we get a lot of value out of it. I think for the system that we use it for, we sort of need it to never go down.

But it is, it comes with a different set of problems. Yeah. It's a unique feature which doesn't thinks that most systems languages can't, or couldn't really do safely. And Elixir and the Beam and Erlang allows us to do it, but it comes with a set of problems. I can imagine.

We should have you on the show sometime. I'd love to be on. I think that would make a great blog post actually, kind of like a hot take, you know, what I hate about hot code upgrades. Because it's definitely on the list of, like, shiny elixir features that people just kinda rattle off when they're talking about how great it is. But, Yeah.

I feel like it kinda falls into 1 of 2 cancellers. Like you said, do you really need it? And then, Steven, like you said, if you do need it, it introduces a whole host of considerations that, you know, you're gonna have to deal with. And I think it's a topic that doesn't really get talked about a lot beyond the, like, oh, cool thing that it looks like you do. So someone should totally write that, and then we could maybe have them on the show and talk all about it.

I'm I'm doing this at work right now. So I'll let you guys know in a few weeks how miserably I failed or how awesomely I've succeeded. I'm sure you succeed awesomely, but have so many learnings for us. For sure. Alright.

So we are coming to the end of our time together. So why don't we move it over to picks if anyone has any recommendations on things to share? Could be Elixir related, could be programming related, could be something totally else. And, we'll go down the list here, and we'll start with Steven. So I have a coding one and then a non coding one.

So, I've sort of been having a bit of an existential crisis. Do I hate mocking? Am I, like, an anti mock person? I just sort of feel like the maybe maybe the way that I'm doing it is not right. I know Alex wrote a really great blog post about databaseless, you know, repos or integrating integrating without actually hitting the database, which is kind of interesting and made me think a lot about some things.

Blog post by James Shore, testing without mocks was really good. Sort of a cross between a, blog post and a Wiki. Very beefy if you know his writing. Yeah. I'm sort of, like, having that, that discussion in my head.

So that's that's my pick. Go check that out, and then tweet at me. Let me know what you think. Underscore Steven Nunez. Let me know, if I'm a total idiot for not liking mocks or if, I'm right and you wanna take up arms against the mockists.

My other pick, this is gonna be kinda basic, but Hamilton. So I watched Hamilton, and I really, really enjoyed it. It's very cool. I then realized that I went to a terrible school. I know nothing about American history or, you know, other reasons.

So my subsequent follow-up was a combination of watching, Crash Course US History on YouTube, which is amazing. And then the John Adams HBO series, which is also really good, though. America. Thanks for those. How about you, Lars?

So I wanna recommend a podcast that mostly helped me when I was starting my own business, but I think is very relevant to, to people right now because it's about slowing down. At the start, it claimed to be a productivity podcast, and to some extent, it is. It's gone a little wider in, later seasons. But it's very much about intentional work and slowing down to make sure your ideas are have room to breathe, that you actually think through what you're doing, doing the right things rather than all the things, that sort of thing. So I think, I really recommend it.

Jocelyn Ling, Kate Lai, is the creator. So Her Slowly podcast, strong recommend. Thank you for those. Yeah. I'm excited to check that out.

How about you, Chris? Do you have any picks for us? Yeah. So we didn't, touch on this topic, during during this session, but, one thing that I'm still very interested in, in Elixir is property testing. I find the concept very interesting and I have done some property testing both, in, you know, my, libraries and in, you know, for for peak.

And what really got me started on it was this talk at ElixirConf 2018, about how to pick properties, for property based testing. And I I I really recommend that talk. It was it was very, it was a very fun talk to to to watch, and, and it it got me thinking about how to do property testing well and and has has let me, you know, experiment with that kind of stuff. Yeah. I'll actually plus 1 your pick.

Michael Stalker, who gave that talk, is awesome, super smart guy. I think great speaker, great teacher. He's also one of the contributors to Elixir School, which is a free open source online Elixir curriculum. And he and I teamed up last year to do some workshops at ElixirConf, and he built, of course, all of this excellent property based testing content and lessons. And I hadn't done anything with property based testing before having a chance to work with him, and I learned a ton from him.

So if you haven't seen this talk, definitely check it out. Let's see. Do I have any picks? I have the opposite of picks, which is a request for picks. I received a birthday present recently that I have no idea what to do with.

It is from my dad. It is a stand mixer, so, like, big mixing, you know, for baking attached to a bowl But, it's like an industrial grade like this thing belongs in like a commercial bakery. It has horsepower It's like refurbished. I don't know where he got into his head that this is something that I would actually Need to use. So I would like to receive, if anyone, you know, is into this kind of thing, easy baking recipes that somebody that isn't necessarily making a wedding cake could do with their, enormous stand mixer that they could also use to, I don't know, power a small car or perhaps a golf cart.

So hit me up on twitter with your easy baking recipes, sm_dibenedetto. Oh, come on. You're a software developer. Sourdough bread. I don't know.

That seems like a lot. You can't find yeast anywhere too because everyone's doing their quarantine baking. So that was the point where the growth is at. Yeast. It's wild.

You don't need yeast. You can just make it. Yeah. That's true. I have heard that.

Thank you so much, Chris. That was awesome. I actually can't believe this is you mentioned this was the first time you had done maybe a programming podcast, and, Yeah. That was great. Yeah.

Well, thanks a lot. Yeah. Thanks a lot for the invite. Thanks a lot for having me. It's it was a very good experience.