In this episode the panel talks to Danny Hawkins - CTO at Quiqup - and his team's journey at Quiqup with Elixir. Danny explains how some of the first things Quiqup built were using Elixir and how they then left Elixir behind in favor of TypeScript, only to come back to Elixir.

The panel considers how these choices rarely are purely driven by technological qualities but instead have to factor in cultural- and knowledge-aspects of a team, and how a top-down dictated technology decision - even if there are good reasons for it - can be harmful to a team's morale.

Get in touch with Danny via email!  Click here.

Links

Hey everybody and welcome to another episode of Elixir Mix and this week on the panel we have Alan Weimar.

 

Hello.

 

and me, Sascha Wolff, and we have a special guest this week, and that is Daniel Hawkins. Daniel, why don't we tell the audience why we invited you and what we are going to talk about today, and why we think you are amazing.

 

Hi, thanks for having me. So, um, yeah, I'm, I'm Danny. I'm a CTO and co-founder of a company called QuickUp. Um, we started using, uh, Elixir around about 2015 for some specific use cases. And, uh, I'm going to talk about the kind of move to Elixir in the first place, the move away from Elixir and then the move back to Elixir and the pain and glory that we've experienced along that journey. Um, so yeah, that's, that's basically what we're going to talk about today.

 

Yes, and I'm very excited about it. We had some episodes in the past where we talked about, okay, when not to use Elixir or like how to onboard people to Elixir, all those kinds of things. And from what I've gathered, all of those are angles we can explore today on like the concrete example, in your case, Danny. So

 

I hope so. I hope so.

 

yeah, really looking forward to that. So, Danny, why don't you give us like a quick whirlwind tour of how you came personally to Elixir and how you ended up

 

Yeah. So, uh, back in 2014, um, a bunch of, uh, well, my co-founders and I were working in a company together and, uh, we, we kind of felt like we could do something together individually, like as, as a, as a company, we had this bit, this idea that we wanted to try and bring to life, which was based around the on-demand economy that was, that was coming up at the time. And we had this idea, what if you could have anything that you wanted delivered within one hour? and to your door, not just food, but anything. And so we built an application, mobile application that allowed you to say, hey, go and pick up this thing from this store for me, bring it to me within the hour and I'll pay you a bit of money for doing that. And we had quite some success with that. The business has changed dramatically now, but if we stay on that for the moment, we had this need for real time data. the tracking locations of the drivers on our dispatch software and also being able to send that to our consumers so that they could track where the driver was and actually see in real time. Around that time, that was still novel. I mean, it's everywhere now, but in 2014, the ability to track the driver, see where they are from a consumer standpoint and from an operational standpoint was pretty, pretty mind blowing. So we initially created a real time. distribution like WebSocket server that was in Node.js with socket IO. And we didn't really like the performance of it. We didn't really like the way that it worked and started reading about Elixir, Erlang, WhatsApp we're using at the time, this whole 2 million concurrent connections on a single server was like, oh wow, this is some technology that can, that can scale. Also, I was a Rubyist. We were using Ruby at that time. And so. you know, there was this natural progression from Ruby to, um, to Elixir for people that were pissed off with Ruby basically.

 

Hahaha!

 

So, um, that, that got me into the functional world pattern matching. All that was like, this is amazing. This is the way code should be written. Immutable data just felt really great. So, so we. The very first contact that we really had with Elixir was our real-time server, which still exists today. And it's kind of weak. We refer to it as the beating heart of QuickUp. If the ML is the brain, the real-time server is the thing that sends that out to all concerned consumers. And it's been very solid all the way through. So that was our entry into the Elixir world, but, you know, around in 2014, the talent pool was not huge. It was still early days really, when you talk relatively. And so we didn't really, we built a few other kind of small applications in Elixir that were just kind of Phoenix, Restful, Gratuel, Absinthe kind of stuff. But then. around about, I would say 2018, yeah, around 2018, because the, the talent pool in the company was largely, um, Ruby or JavaScript, TypeScript specifically. Um, we said, well, why don't we just focus on TypeScript? Cause we can use that on the backend and the front end. Um, it's a, it's a pretty good general use language. It's a very, very nice experience in the IDE from VS code. You know, the. The type safety and the type checking almost gave you a safety barrier before even tests came into it. Um, because it would just tell you if something wasn't right. I mean, it was a nightmare to debug. Oh my God. The debugging promise changes the worst thing in the world. It's, it's horrible. Um, but we had some success with a framework called Nest.js, which is a backend framework in TypeScript. And on the front end, we were using React and we actually found that our developers were able to. Um, moved to some extent across the front end and the backend. We had a fair amount of shared code and the nest.js framework encouraged pretty good modularity. Um, it was based on, uh, Angular actually, it was based on Angular from the front end, which had a dependency injection. Um, modularity is really, really nice. So that was kind of our strategy. Let's use TypeScript as a general use language at quick up. And Elixir will be special use case, which in the case of the real time server and anything else that we felt would need more scale than we could achieve with the TypeScript stuff. Um, so that was, that was, we were in a pretty good place. There was the one really, um, any big problem there. But around about that time, actually long before that, we had been looking into event sourcing. We felt it was a good pattern for the company because having to deal with real world events that are happening all the time, parcels moving between custodians, drivers moving along the road, drivers moving in and out of geo fences, all of this was useful information that we felt should be events. So We got to the point at the beginning of 2019 and we said, okay, now's the time to re-architect into event sourcing the core concerns of the business. But we felt like we should do that in TypeScript. We'd made this decision and it was a decision that we felt that we should get behind. And so we started looking around for first of all, you know, are there any kind of frameworks that can help us to do our event sorting and... There wasn't really that much. There were some good guidelines on CQRS and how to do this and how to do that. And we felt like there was enough that we could, that we could make a go of it.

 

When exactly

 

So,

 

was that?

 

sorry, that was the beginning

 

Wha-

 

of 2019.

 

Okay, okay, it's a pretty nice ending, okay. Got you.

 

Yeah, around the beginning of 2019. Yeah. So we did some kind of architecture qualification. We, we used a event store DB initially from Greg Young.

 

Yeah, yeah, yeah, I'm aware.

 

Yeah. And it was, it was all going reasonably well. But it was, there was a lot of work involved in just getting the basics going. Um, and we started to realize, Oh, you know, crap. I didn't think about. this thing, we didn't think about snapshotting, we didn't think about upcasting, we didn't think about like a whole stream of things that we

 

Yeah,

 

didn't think about that started to bite us in the ass, right?

 

all of these event sourcing things will then crop

 

Yeah,

 

up as soon as you put this thing

 

yeah,

 

into production. Yeah, yeah, yeah. Okay, I want

 

exactly.

 

to change this event. Like, okay, but now what? What about the old ones, huh?

 

Yeah, exactly. So, so it really started to bite us in the ass. And so, and then I remembered I did a POC one time, um, with Commanded, uh, the, the, um, event sourcing framework from Ben Smith. Um, and it felt really good when I used it. I remember like feeling like this, this guy spent some time on this. He really understands it. Um, you know, you look through the documentation and it's got so many. Like, oh, if you're in this situation, you can do this, you know, uh, regenerating projections, upcasting events. Um, you know, uh, uh, I forgot what he called it now, but like event handlers and everything, we're all there in a really nice way. So what we decided to do was, um, try to replicate those things. Uh, so we're like, okay, let's go and look. Uh, basically our answers to everything was if we didn't have something in TypeScript. that we needed for event sourcing. Let's go and look at how commanded today. It was became like the de facto, um, mode of operation for, for quite some time. Oh, we can't do this. Let's go and look how commanded. Um,

 

Sounds like a meme to be honest.

 

I know, right. So, so that we did that and we had again, some success, um, you know, snapshotting work and actually a certain point. We moved away from events store DB and went to Postgres mainly because of commanded like if they're using it and it works for them, why don't we use it? Because we understand it much better than events store DB. And it's a bit more forgiving if we want to go and make some changes. And like, if we do some bad stuff, we can go and fix it because we're still relatively new in the event source world. Um, but then more and more of these things were coming up, uh, day in, day out. And. We realized that the majority of our engineering time wasn't spent on delivering the value to the product team. It was like building a framework for event source. And it was at that point we'd spent a significant amount of time on like six to eight months or something on a relatively simple feature, which was based around driver pay. So we decided we wanted to change the driver pay to a split between hourly pay and order rewards. So bonuses per the amount of orders that they did. So we architected this whole system with event modeling. We started to develop that through this mechanism. But then, like I said, we realized that the majority of our time was spent on building the basics or dealing with situations that required the basics to be in place. So at that point I was sitting there and I was thinking, ah, if we go and we do this with Elixir and Commanded, we could have most of everything we've been working on for the six months is already there and we can actually start to focus on the business value. Um, so I actually got one of my lead engineers, uh, Lewis, Lewis Valera. Um, I asked him, you know, off the books, just. to a PSE, to see what you can do. Um, because I'd, I'd put a lot of, I put a lot of effort behind TypeScript and a lot of confidence in the team around TypeScript. And even with my co-founders, if I went to them and I said, guys, I think we made the wrong move, we should be going to Elixir. Um, they're not going to be super happy with me at that point. So I needed something to show for it. I needed something to actually demonstrate this is the right move. And so. Lewis went ahead and he did this kind of POC. And within like a week he'd got 50% of the original system done, which had taken us like six months. And it was like, obviously there's some value in that, you know, some of that, some of that effort is in discovering how to solve the problems,

 

Mm-mm.

 

but not six months to two weeks, that's too much. That's.

 

Yeah, yeah,

 

clearly

 

yeah,

 

an indication

 

definitely.

 

that, yeah. So after that POC, I was able to speak to the rest of the company and my co-founder and say this is the right move. And we doubled down on that effort. And we were really, really happy. And, you know, a lot of the reasons is, a lot of the reason behind that is that Elixir and what you can effectively call the actor, actor pattern, right? what Ben Smith has done with Commanded is each of your aggregates is an actor. And you don't have to rematerialize the actor every single time you want to send a message to them because they're a gen server. So they're alive for as long as you want them to be alive. And this, this just meant that OTP and Elixir and Commanded was such a good fit for this, um, for this event sourcing problem. And, and actually, you know, in the previous incarnation when we're trying to use TypeScript, we actually ended up writing two different mechanisms to try and emulate that. One was a cache. So the aggregate would literally be living in Redis and you would still have to pull that back from Redis and then do whatever you want with it. And the other one was snapshotting as well. But the GenServer approach is just so elegant. with aggregate as an actor. And so, yeah, now we've been running like that. We've, we've actually started on the second, um, service. So we were adopting a kind of SOA service oriented architecture at the moment. We're not going down the microservices route specifically, but we're looking at specific domains in our business that makes sense to be isolated, um, like free operations and order management, for example. And we have. now the event sourcing, um, SOA, uh, services that we, that we use commandedly. And then integration events move backwards and forwards between those via Kafka. But the effect on the team has been quite interesting. This is one of the most interesting things of all, I think, is that both Lewis and I, when we were talking about doing this, we had concerns. We're the only elixir people in the company, just me and you. Like...

 

So how many people were you in developers at that point?

 

around eight, around eight.

 

And they're

 

So

 

basically

 

not,

 

all of them more or less

 

yeah,

 

TypeScript devs, right?

 

pretty much. Yeah. Yeah. Some Python experience, no Ruby experience. And, and then me and Lewis, or he was just the only kind of Phoenix and Alexa people there. And so that was, that was a worry. But at the same time, I've been reading about what other companies have been doing, how it's out in conversations with and some other CTOs and technical guys that were using Elixir. And that gave me a bit more confidence because they said, you know, at the end of the day, you can get a lot more done with one Elixir developer than you can with a TypeScript front end and back end potentially, as long as you find the right people. And, you know, it seems to be pretty accurate, you know, because you, I don't know, I don't know what it is exactly. Maybe it's the testing framework, maybe it's, you know, X unit or maybe it's Phoenix or maybe it's Elixir as a whole. But your ability to be productive with it is, is pretty high in my opinion.

 

Yeah, I think, I mean, it's like some of the roots Jose has taken from Ruby, right? With like focus on developer happiness, but also

 

Yeah.

 

that the beam as a platform is, is very, very pragmatic, you know, like it's

 

Yeah.

 

really, it's been, it's basically built by pragmatism. Like it's been built by a concrete need and it's not been designed in an ivory tower, but it's, and

 

Yeah.

 

all of that combines to a platform which, I mean, let's, let's face it, it's focused on getting shit done. Hahaha

 

Exactly. Exactly. Exactly. Yeah. And that was actually one of the other small battles that we faced along the way is, you know, so we've gone to this TypeScript world, we decided we're going to come back to this Elixir world. And along the way, Golang has become like, everybody's

 

Hmm

 

like, Golang is the best thing in the world, let's go to Golang, Golang, Golang. And I was like, okay, it's great. But it's not really focused for me on productivity. It's focused on specific. specifically like performance or small microservices or

 

ease of onboarding.

 

yeah, yeah, and a lot of the things that I was hearing about Golang is like, it's great, but you have to do everything yourself. Like there's not that much that you can get for free. And I'm like,

 

I mean,

 

no,

 

yeah,

 

I want everything for free.

 

and they didn't even have a package manager at the beginning, right? I mean, this

 

In

 

is basically

 

the beginning,

 

like an

 

yeah.

 

ongoing

 

And I,

 

thing.

 

yeah. In

 

They

 

fact, at the time

 

make.

 

we were considering it, I was blown away by the fact that I couldn't just drop stuff in there, you know.

 

Yeah, it's actually it's an interesting perspective to be taken because I mean, Golang and Kubernetes and all of these things are topics which which evolved from from Google, but to Google's specific needs, right? Like, I mean,

 

Yeah.

 

you don't we as most software developers don't solve problems on the scale as Google tries to solve them, right? So like, Golang

 

Right.

 

as a language itself is not even designed for like a small team, necessarily, but really for We have super big teams, you want people to get ease on board, you want to have simplicity, and that is what it is designed around. That might be your use case, but it's definitely not the holy grail of language development.

 

Yeah. I mean, with a small team, like what we have, what we still have in fact, is we're looking to be lean and we're looking to be effective and

 

Mm-hmm.

 

productive. And that means that you have to have something in place that lends itself well to that as we saw when we tried to build the underlying framework stuff in TypeScript from our side, it completely stopped our ability to deliver any company

 

That's

 

value,

 

quite

 

any

 

the

 

productive

 

opposite

 

value.

 

of lean.

 

Exactly. Exactly. And so. When we went back to Elixir, well, first of all, we did so many good contributions in the community. I think it's probably one of the best communities I've seen. And so especially considering that it's a smaller language or a smaller community than say, in Golang or something like that. You know, you go to the Elixir Lang Slack and there's 34,000, 36,000 odd people in there. If you've got a problem, you go and ask it. And those like people have really helped. They come back to you, they try to solve problems and there's so many libraries available at HexPM to help you with problems. You know, some great ones like Guardian, you know, all of the OmniAuth, UberAuth, use UberAuth heavily. Like these are, there's just tons of libraries that are available. They can help you focus on the business problem you're

 

Thanks

 

facing

 

for

 

and not

 

watching!

 

the underlying, the underlying scaffold. that you need to get that stuff done.

 

Yeah, yeah. It even goes for big names. I mean, you just mentioned that a lot of the things you did was trying to replicate what Commando does. And even Ben Smith, I still remember, I also worked at a project where we used Commando for event sourcing purposes. And at one point we were running on Azure and we had an issue that our subscriptions, our event handlers stopped consuming new events. And we were like, well, what's going on? We had really tried to dig down into that. And at some point I actually reached out to Ben and was like, hey, like through Slack, like through the Xlinks, like, hey, you know, like I was at Gitter, I don't remember, but some platform reached out, I was like, hey, we have this weird problem, we don't know what it is, can you help us? And yeah, like he basically immediately

 

Yeah.

 

jumped and he came to

 

Where

 

our home

 

else does

 

and

 

that

 

like,

 

happen?

 

yeah, exactly.

 

You know, where?

 

Like, and he figured it out, like he figured out that actually the Postgres implementation of the subscribe and notify in Azure at that point was broken. That's why basically the event handlers never actually got the notification of new events. So, yeah.

 

Yeah.

 

Crazy story.

 

Yeah. We actually, we actually convinced Ben Smith to come to the office in London one time. Um,

 

And nice.

 

yeah. So it was a time that we were first looking at Commanded for actually specifically for Chain of Custody, the POC that I mentioned earlier, the thing that actually brought me onto Commanded in the first place. And he came into the office and he talked us through how it all worked and everything. Like he did it as a what we I think we had a lunch and learn kind of session. I'm really good guy, really nice guy. Um, and yeah, like you said, and it's the same, it's the same as like, uh, you know, uh, Chris, Chris McCord and, and, um, Oh, who's the one from absence? I always forget his name. Uh, from cargo sense. Uh,

 

Ben Williams.

 

it's another, yes, he's around. They're all around, like all the creators of these big, big, um, libraries and frameworks that are used, they're all around and so active.

 

And same with Jose.

 

and

 

I mean, if you post a question on the Elixir Forum, the good chances are that you might even get a response from Jose. So.

 

Yeah, yeah, it's very, very cool. Yeah, so the experience, so we're back in a happy place, so quick up in the dev team. But what I was mentioning earlier is about the worries about the TypeScript devs and the fact that there was just two of us, two Elixir devs in the company. So we actually decided, you know, let's see if any of the TypeScript devs are interested. Like, do you guys want to do some Elixir? And, one of our guys, Hussein, um, and also another guy, Nada, they're both working in that we had to, we have like a dev team in Beirut. It's a strange place to have a dev team, but we do. Um, and they, they were like, oh yeah, I want to have a go. I want to try this thing out. So the first thing we did, and I think it's super important for anybody that's thinking about doing this is make sure you get some kind of course where tutorials, materials, something that isn't just scraps from the internet. I think it's better to find something that's a bit more well put together. And, and, uh, there was one from the coding gnome. Um, I forget what the guy's name is, but the coding gnome, very, very

 

Dave Thomas.

 

simple. Yes, exactly. Yeah.

 

It's elixir for programmers. I've also heard very good things about that.

 

Ie, mae'n dda iawn iawn.

 

I'm just for all listeners, I'm going to put a link to that in the show notes so you can check it out.

 

Perfect, perfect. So very, very approachable and nice guided through something that's like a real world example, starts with a tech space client, moves gradually on to the more complex things and eventually you end up in the Phoenix world with your whole kind of web app side. So we got this and we had the guys go through that and they were... But they were falling in love with the language, like straight away. Hussain, we can't tear him away from it now. We're like, we've got some TypeScript stuff for you to work on. He's like, really? Oh, he wants to stay in the Elixir

 

Hahaha!

 

world. In fact, everybody does. Everybody that's been exposed to it, they just want to, they just want to use it more. And, and we found that Hussain was able to be productive in less than two weeks, really. I obviously, he didn't understand GenService or OTP or the, the beam or anything like that. But I think for anybody that's used specifically Phoenix as well is if you understand how, um, you know, MVC works or the basics of like rest, you can get fairly, uh, fairly far ahead, especially with an existing, um, code base. Cause you end up. you find yourself going in there and being able to find stuff and make those changes. And it's a bit more approachable. Sorry, my cat is me. So,

 

No worries.

 

so yeah, it's a bit more, it's a bit more approachable and easy to get into. And then you can gradually learn that with a 20% which will give you, um, you know, the more, uh, the things like gen server and, and, and whatnot. So, Hussein has actually been doing gen services and stuff now, and that's around four months after his initial onboarding.

 

I have

 

The

 

a question for you about learning, right? Sorry, but this

 

Yeah.

 

is like, actually I run into this a lot, like because I bring in Elixir to different companies. Of course, it's easy for me and all of us to get started and get moving on new projects with Elixir, but to be honest, it's quite different, right? There's no what I would call dot notation where it's like, you know, object dot function, right? You don't have methods. Like you talked about TypeScript, you talked about Golang, all these languages we've been talking about in comparison. are all OOP based, right? Where you have an object, you

 

Yeah,

 

say

 

it's

 

dot

 

a completely

 

notation

 

different

 

to it.

 

paradigm.

 

I mean, that's actually where I get tripped up a lot on is that for teaching new people, not for myself right now, but they're not used to modules with functions and passing data in, data that doesn't change. They have to understand that, hey, okay, I transformed this thing, I have to catch or, I say catch, but assign that to a new variable. to use that one. The one I had originally is not actually mutated. And also OTP itself, GenServers, message passing. I mean, this is like tons of new stuff. And if you compare somebody going from TypeScript to Elixir, it's a gigantic jump. Like, yeah, okay, if you look at it, it's not that difficult, but it is really a mind twist. Like, do you, did you have problems with kind of bringing him on board with this? I mean, you said two weeks, but was those two weeks hell for him? Or was he actually able to kind of grab his mind around? Because for me, it took like a year to really get my mind around it. I was productive in a couple of weeks, but I think really a year to two years to really get into it.

 

Yeah, I definitely think there is, I think you're going to get a mixed experience with that and it's a very individualistic thing, right? It depends on your previous experiences, it depends on your ability to absorb new ways of doing things. Also you know, how polyglot you are. If you've had experience of many different languages and how they work, your ability to switch between how a specific language works becomes... becomes easier, in my opinion. Um, but I think also, you know, in the, in the, the, the coding and own, um, guide, you know, they go, if they touch on the basics, it goes through the basics at the beginning. And also if you just look at Elixir Langley, you can, you can start to get a grip on it. I think the two most important or maybe the, some of the most important things are the assignment operator isn't an assignment operator, right? It's a. That's the most, one of the most important things to get to grips with. But that also leads you onto pattern matching or you can go either way from that. You start with pattern matching and then you realize that it's used there as well. But yeah, he just seemed to gel well with it. And in fact, both of the guys that are now doing elixir that started with TypeScript, they just seem to gel very well with it. But I would expect that other people would find it much more of an uphill struggle. Um, no, but I agree with you 100% as well. You get, you can do stuff. I mean, I remember my first experience with Elixir. I didn't really understand it that well, but I could do stuff with it. I could understand it well enough to, um, to use Phoenix, but it took me. Probably a year, year and a half to actually be able to go and write a Gen server without scratching my head. And I'm thinking. What is this doing? What is Handlecast? What is Handlecall? How the hell do I do this thing? And only probably, you know, after a year and after two years, I was comfortable with GenService. I could understand what they were doing. I'm still not comfortable with GenStage. Still need to figure that one out, but I don't need to use it at the moment. But when I need to use it, I'll start to broaden my, my understanding further. But for me, it's a testament to. I guess the underlying goal is that you can be productive without knowing everything. You can get stuff done and then gradually you can learn the stuff that you need to progress as you go. And that's one thing that I really enjoy about Elixir.

 

Yeah, and I also think that the course we just mentioned, at Exif Programmers, actually does a very good job at covering these basics. All of the things, all of the stepping, all of the pits, I guess, and you just mentioned, Ellen, are kind of covered by that course, from what I know, from what I've heard people who did it talk about it, I'd say about it. So you can actually... The basics and all of this new paradigm idea is something which I think the existing learning material actually covers quite well. When it gets to the nitty-gritty parts, especially of OTP, I mean we talked about it in the past, then it becomes a bit more spotty in the sense of that there's a lot of wisdom also from the early on community, which is not as readily accessible as the wisdom from the early on community, which is more focused on... making documentation, making information, making teaching material accessible to newcomers. But still, and that's something, as you also just said, Danny, right? Like the community is super helpful. So like if I bet money on, if you like go to the XFORUM, like, hey, I don't really grok GenService, I don't really grok OTP, can somebody help me? Probably takes like 10 minutes and you get a response with some helpful resources. I would bet 50 bucks on that.

 

Yeah, I'd agree with you. I'd agree with you there, for sure.

 

But yeah, I would be interested to hear, I mean, like, as you just said, you add like this team of like mostly TypeScripters, which now jump into Elixir. So what's the status quo now? Like how much, uh, is how much are you using Elixir right now? How much are you still using TypeScript? Do you have any plans on like getting rid of a TypeScript stuff at some point? Or is it like this, this twofold scenario of, of, of Elixir and TypeScript, and you're rolling good with that?

 

Okay. So we could talk like my personal long-term plan is to eradicate TypeScript from the company completely and just have everything running on Elixir. But let's say that's not, that's kind of just because I love the language, but it's not practical. And there are some

 

Yeah,

 

things

 

fair enough.

 

where using TypeScript and Nest.js are really good. And also we're going to keep on using React for the foreseeable future. for, especially for our, for our external, um, external stuff. And I'm going to touch on live view, um, in a, in a, in a short while. Um, but the status quo now is that so there's Lewis and I are still like with the, uh, original, uh, Elixir developers that are using Elixir on a, on a daily basis. Um, we have Nada and Hussein, which are very capable. And they came from the TypeScript world. And we actually hired two new Elixir devs just recently. And that's another thing is a hiring has got a lot easier. And it used to be quite difficult, but now there's a lot of options for hiring, especially if you're looking globally, which we are, we, you know, we operate mainly in Dubai, Dubai is not exactly known as a tech hub, so. Um, to find Elixir developers in Dubai might be a bit of a challenge. And so once we widened it and said, okay, we're going to be remote anyway, we found it much easier to find an attractive talent. And so now the view is that all of our, um, core services, all backend stuff will move to Elixir, um, uh, using event sourcing where applicable and, um, uh, potentially some of the internal tooling will gradually move to Live View. But we're very early days on Live View at the moment.

 

Yeah, okay, that's interesting to hear. I mean, you talked about this journey of like first having elixir, then typescript all the way and now back to elixir. If you could travel back in time, would you do it again like that?

 

No, I think I would have, I mean, hindsight's a wonderful thing, right?

 

Yeah, yeah, yeah,

 

Um, uh,

 

sure.

 

but, but like knowing what I know now, of course, I would have committed to the Elixir route, um, fully on the back end and, um, invested more in not necessarily finding Elixir people if they were, if we couldn't find Elixir people, fine, let's find people that are good engineers and give them a track for Elixir if they're interested. I've heard of several companies that have done that and have had great success. And one of the, some of the people that I spoke to actually reported that as well. Um, so yeah, I think I wouldn't have wasted that time. I'm going down the TypeScript route. Like I'm not, I'm not trashing on TypeScript or JavaScript. They have their place there. And it's, I'm using TypeScript and VS code with some of the, it's been so enjoyable, so fast to get things up and running and it's so lightweight. Ond nid oes gennym ni'r hyn a oes gennym ni pan oedd yn cael gweithio'n dda.

 

Yeah, fair enough. I mean, it's also I think like one thing you said at the very beginning that you had the first prototype with a bit of socket IO in Node.js. And I mean, it's interesting to hear that because from where my experience with Node.js also was really when it was back in beta and I was actually part of my bachelor thesis about Node.js. Because from what I can remember, like Node and like JavaScript, the JavaScript ecosystem was one of the very first ones which really... embraced this idea about WebSockets, right? Like that really pushed forward there. And socket IO was kind of like a, what? You have like this persistent connection between the server and the browser. That is absolutely insane. Yeah,

 

Yeah, it was crazy. It was crazy.

 

and like my best thesis was literally mostly about WebSockets, how you could use it to have like some kind of real time interaction on websites and in this case, it was about experimental sensoric measurements being visualized. But yeah, all of that was in Node.js. So I find it interesting to hear that even then, back in 2015, right? That was, you already realized that Phoenix and Elixir were so much better suited for that. What version of Elixir was that and what version of Phoenix was it? It must have been relatively early, right?

 

It was relatively early. It was, oh god, now you're asking me. It must have been like 1.3 or something like that, around 1.3, 1.2. Yeah, it was quite early days. But it was amazing. Seeing that work and how little resource it was using for these connections and how easy it was to manage those connections as well. Then, you know, being able to, so that was my first experience also of clustering, like distributed airline. So we use lib cluster.

 

Oh, okay.

 

And later on, we actually wrote, one of our guys in our company wrote a library called lib election, which is on GitHub. And that was because we started to use Kafka as well. And with Kafka, you would want to have a single consumer. across your cluster, unless you're partitioning Kafka, you don't want to have multiple consumers listening to the same partition because it messes everything up. So libElection was just a way to elect one of the libCluster nodes as a leader. And so we actually wrote a library that went on to GitHub and I felt great about that as well. Although it never really got used that much, but it was a good feeling to actually do something like the rest of the Elixir community. which is putting stuff back out there, you know. I'm.

 

Yeah, that's nice. But I guess I mean, we just said that an insight is always right. But when you consider that it was 1.3 at that point, the Elixir community was probably even smaller than it is now. And that's not going to lie. Elixir is still pretty niche. I think it makes sense that you said, okay, we want to open up our hiring pool. We didn't yet have made the experience about Elixir being actually quite easy to onboard. With all the information at hand, yeah, it was a sound decision I guess.

 

It felt like a sound decision. And up until the point where we encountered stuff where Elixir really would have helped us, and I guess the main thing that I would have changed, uh, you know, is the ability to have made that distinction sooner. You know, we, we made, we spent a lot of time looking at Commanded and saying, this is great, let's reproduce that. When really it should have been obvious, um, that we could just build in Commanded. Why? Why were we deciding as a team that we needed to have this general use language that everybody obviously there's a benefit to doing that because you have the developers working on it. But, you know, there's a lot of places where you say, don't, don't, uh, just say we're going to use this tool because this is a tool that we know and love.

 

Mm-hmm.

 

You know, you should be looking at what is the right tool for the problem you have, what is the right as a tool to solve that problem in a, in a, in a sufficient way. And if I'd done that, it would have probably happened sooner. Um, and we would have made the discovery sooner, but, uh, you know, you learn.

 

Yeah, it's pretty much what we also when Alan, Adi and me had the episode on like when not to use Elixir, that is what it boiled down to, right? Like figure out what your problem is, figure out what your use case is, and then make a choice based on that. And sometimes that might not be Elixir, right? Like if you have a specific problem where Elixir maybe doesn't bring a huge advantage to the table and you have a team, for example, in your case, of types of developers, why the hell would you choose Elixir? But in your case, well, there was compelling reason. to do something else. I would like to know, Danny, how was it that I mean, like you said, okay, the people were pretty quick to pick up Elixir, but how was it for Morel, I guess, because I mean, you just said they had this proof of concept, which was built in like two weeks, implemented 50% of what you needed. And then you had this TypeScript thing where you like worked on for over half a year. And now you said, okay, let's jump ship, right? Let's switch over to Elixir. How did you handle that with the team and how did the team work team feel about that because I mean, at the end of the day, I guess, yeah, you have some learnings and like a better understanding of how like an event source and security system works. But still, you're going to throw away code, right? Throw away code people worked on hard. So how was that?

 

Yeah, it's a good question actually. So the first thing I should mention is that I have a bit of a reputation for being the guy that jumps on new things too quickly. I'm like, wow, this thing's awesome. Let's use it. And then,

 

Are you using Cassandra?

 

no, no,

 

MongoDB!

 

no, that's one I avoided. I'm okay. I'm not a hype driven guy. I'll definitely

 

Okay.

 

be hype. hype driven as long as it makes sense. Right. So if, if I think it's cool.

 

Fair enough, fair enough.

 

Um, so, so when, so Elixir and like, when I started using it, I was like, this is, this is awesome, it's cool. I can do amazing stuff with it. Um, and so, you know, there's been many of these things in the past where I've been like, oh, this is great, let's jump on it. And gradually over time, I've had to get a bit more disciplined about that and, you know, not be the guy that disrupts the team because I think something's cool. So there was. That was a bit difficult for me because all of a sudden I'd realized that this was the right choice. But I didn't just want to say guys were changing everything we're doing this. I wanted to be exactly like you said, considerate of the guys that have been doing this work. And we've all been slaving away in TypeScript to produce these things. So the first thing that was the I mentioned like Hussein and Nada, those were the guys that were working in the TypeScript area. So it was a chance for them to actually experience something. that knew and to actually take the knowledge, you know, the not just the knowledge that they've been gaining through the process so far, but also the pain and have the ability to see if this reduced the pain and gave them a new experience that was better. And definitely I feel that that was the case. I mean, we would have to ask them themselves, but my, my, I feel quite good that they, Yeah, we're on board with that. And the car.

 

So you already were at the point where parts of the team had realized that the way you currently were doing things were producing overhead, which wasn't maybe as great. I mean, you mentioned that you realized that you mostly spent time on rebuilding a framework for events hosting. That was a realization which also was already. which the team also already had at that point, I assume.

 

Absolutely. Yeah. So

 

Yeah, OK.

 

there was already,

 

That makes sense.

 

you know, we could see the sprint board, we could see the backlog and we could see

 

Mm-mm.

 

that a lot of it was solving underlying stuff. Um, so didn't necessarily deliver business value straight away. Don't get me wrong. There was some overhead in the whole concept of event sourcing in general, like event modeling and understanding aggregates. And, and

 

Yeah, it's

 

so

 

not for free.

 

it would be. Yeah, it would be unfair to have like a direct comparison between what, what it would have been like. Um, you have to compensate with some of that, but yes, Jeff, definitely the backlog was mainly full of stuff that we didn't end up needing to do. Once we moved to commanded and elixir. So yeah, having those devs be the guys that were involved in that. I think it's crucial. I think if you were to just like. bring in two Elixir devs and say, okay, guys, we're going to do an Elixir and everything you did is nothing, that would be very, very uncool. So that was one big element of it. The other element was that... I'm sorry, go on.

 

I would even go further and say it's a recipe for disaster.

 

Exactly. It would be a recipe for disaster. Um, and then the other thing was we had our senior front end guy at the time. Um, Sam Tutton, uh, he's since left the company to go and do some coffee. Online coffee brewing or something. He's coffee nut. And he decided to go and do some coffee business. He had a midlife crisis. Anyway, Sam Tutton, uh, he was like the senior JavaScript TypeScript guy. And he had always wanted to get into Elixir. And he had touched on it in certain places, but he was always too busy with the front end stuff that he was responsible for in the company. So there wasn't really capacity for him to ever get to that point, but he liked it and he understood it and he saw the value in it. So it wasn't like I was just putting something in front of people that didn't make sense to them. It made sense to them. And we just had to prove that it actually did the things that we expected it would do. And this was the whole idea of that POC, proof of concept, like, and qualification of that architecture revealed that in fact did look good. But I did, so the first time I said that we should move to Elixir was probably around three months before we actually did that POC. And actually what happened was my co-founders were like, ah, come on, you can't do this. You've told us this is the way forward and it should be the way forward. And that actually caused me to say, okay, no, I should keep on with the TypeScript route. And then I did for a while until it just became so apparent that it's really a bad choice. And I had to say, guys, no, this is, we've got to make this change. And it was definitely the right one, but it was a difficult one to. to make because of all the factors that you mentioned. And also the fact is that I put my money behind the TypeScript route, quite literally the money of the company behind the TypeScript route. So, yeah,

 

Hahaha.

 

but you have to be able to learn from, learn from these experiences. Right. Um, so we, we learned how to approach things a bit better the next time around make sure that the, you know, not necessarily say, I said to you, I believe we should be using Elixir for everything, but given the lessons of the past, that's not the case. If there is a specific problem we come across that we're like, actually, this is perfect for Golang, or this is perfect for, you know, Python, then we should look into that.

 

Yeah, I'm also a big believer of choosing the right tool for the job. It just so happens that I think that Elixir is a very good tool for most of the problems we as

 

I agree.

 

average Joe web developers back in developers face. You know, it's just a

 

Yeah,

 

very good overlap.

 

I definitely agree with that. That's the main thing is that it's a general all-around good tool.

 

Yeah, but if I don't know if I had to build like some some number crunching or even though with like an X I think you could make the argument that this is getting better in elixir But in the past if you said okay, I want to like in some high performance and number crunching thing then yeah I mean that I don't know like some physics engine or whatever right? I'm not gonna use elixir for that But that's

 

It's

 

not my day-to-day job

 

actually, it's that, it's, it's, that's actually a really good point. So really early on in the company, we also had this, this other kind of case where we were doing some routing optimization kind of stuff, some ML kind of stuff. I know that at that time we were using node sock IO and it was like, we should use node for this, but actually that was never a good choice. Pandas, Python. All of the numeric optimizations that were made in Python and the ML tools that are available in Python meant that that was the right tool for the job. So yeah, it's always good to look at what is the right tool for the job.

 

Yeah, definitely. So yeah, all things considered, I mean, you also said that you don't regret now making the move you maybe should have done it earlier. I guess one thing which is important to consider is, yeah, you had all of these learnings, but you also made all of these mistakes, I guess more on the TypeScript side, like learning event sourcing first, realizing how do you do event versioning? How do you do replay projections? All of those were learnings, I guess. You then brought into this new era of Elixir. Let's call it that. So yeah, I mean, I think as software engineers, we sometimes forget that the learnings we make along the way are maybe just as valuable as some of the business value we are delivering, maybe not for the company itself, but for future endeavors of delivering business value.

 

Yeah. And also I would say that, I mean, we say, would you go back and change anything? But I actually see a lot of value in the route that we took because we really appreciate the value and the complexity of what we're taking on, both from a perspective of if you try and do it yourself and if you're using a framework that gives you some support there, and what difference it can make. So it's quite valuable.

 

Okay. Thank you. Thank you for these insights, Danny. And like, it's, I'm excited to, maybe, who knows, maybe, maybe we can, we can have another episode and like one year down the road and he said like, yeah, we went back to a TypeScript.

 

No, I don't think that's going to happen. I especially because now we also have the added fun of live view, which we're just starting to scratch. Like we're just starting to play with a little bit internally. And we were thinking to be able to use it to, for some of the internal applications that we run,

 

Hmm?

 

and we started to do so in some cases and wow. Wow. Live view is a game changer in my opinion. The only way to get away from JavaScript fatigue.

 

Yeah, fair enough. Is this something where you say, I mean, like you now said, okay, you want to potentially move forward with more elixir on the backend. You also mentioned that you do TypeScript on the front end of React. Is that then something where you say, well, who knows? And like, I don't know, six months down the line, we no longer do TypeScript on the front end as well.

 

Potentially, because I've also got another thing that I'm involved in. So actually, early on in the process of doing TypeScript and event sourcing and NestJS and this whole project, I was also involved with a gaming community called the IVRL, International Virtual Reality League. And this was a community that me and some other people set up after not being too satisfied with the existing. and leaks that were there.

 

Mm-hmm.

 

So I started to use this technology to build a platform there because it was much lower risk. There was like a few thousand users. There was no revenue. There was no, like if it went down for a day, it doesn't really matter. If QuickUp goes down for a day, end of story, you know, it can't, it can't go down. So I was using this, um, agrarail.org as a way to kind of prove this technology. and the event sourcing and actually I did the whole thing in TypeScript and then converted it into Elixir with CommandAdapter. We did our initial POC and started to do more kind of evolutionary stuff on there, which includes LiveView. So that whole platform is, it's using LiveView. And so that's been my kind of playground, my training experience, my ability to understand what can be done with LiveView has been on this much lower risk project that I've been working on. uh, in the gaming community. And I'd say I really believe that it's the future. I don't see why it couldn't be because, uh, it just reduces so much overhead. Um, don't have to think about like API, don't have to be like API authentication. Don't have to think about rest and GraphQL and, uh, separate front end stack and separate back end stack and do you combine them together? Would you have separate projects? All these kinds of things that you, uh, You have to deal with in the current de facto, which is like an API in a front end. Um, and, and that includes the JavaScript fatigue and web pack bundler parcel bun, like all of the different reacts next JS, create react app. There's just so, uh, so many ways that you can go that you can almost get like a paralysis of choice, um, and then having to keep those updated, having to. Um, You know, there's just constantly moving a rapid pace. It's always a better tool to do something in the JavaScript world. Um, and I just got super exhausted by, by that, honestly. And one of the things when I saw live view and actually committed to using it for the first time, it just was a game changer. It felt like way back to the days where a single developer, you can just create a POC. You can just make an app and then, you know, you add Tailwind on top

 

Yeah,

 

of that.

 

yeah.

 

And Alpine, you get to the petal stack and it's like. The holy grail of development, in my opinion.

 

It's kind of like the good old days, the lamp stack, right? Like suddenly you could actually develop like a backend and HTML based application and all of that in one language and like for one stack, holy shit. So yeah.

 

Yeah. And on top of that, it's like real time. Yeah.

 

In real time, yeah, indeed.

 

And the way that it does it as well with the diffs, you know, being able to send the diffs over the web socket instead of the full payload. So it's so efficient.

 

Yeah, I actually was on the Elixir U conference back in, when was that, 2019? When Chris McCord basically showcased this Flappy Bird thing. He was a Flappy Bird, no, he didn't build Flappy Bird, that came later, I think. But he had this, even an animation he did with

 

Yeah, the

 

wobbling,

 

animation.

 

yeah.

 

And

 

Where

 

that

 

he

 

was

 

even

 

all

 

said,

 

coming.

 

this is not a good use case for it, but it just showcases what it could do. And...

 

crazy.

 

That was great. I still remember the audience being like, whaaat?

 

Yeah. And then of course there's the live book that came off the back of that.

 

Yeah.

 

And live book I've been, I've been using that as well. I actually did another medium post on connecting live book to Kubernetes. So to a running node in Kubernetes. So you can

 

Interesting.

 

actually, yeah, it's very, very cool. But like live book, I, we, we don't want to go straight out to client facing production apps with that. Um, although, like I said, I've already with the IVL.org I've already, I'm already doing that. Um, but the reality is we have specific use cases which wouldn't fit well with that. So for example, our driver app is a mobile app, native app, and there are some offline capabilities that are required there, push notifications, things like that. And so,

 

Mm-hmm.

 

you know, that's gonna, that's going to remain, uh, uh, as, as is, and some of the, uh, the web app experiences, um, are there already. We've invested so much in react and the, the tolling around react to build our own kind of solutions that, um, it wouldn't be a very cool thing to just go in and just like swap all that out. Um,

 

Yeah, especially if there's

 

the way

 

no

 

that

 

need for it, right? If it works, if it does the

 

Exactly.

 

job.

 

Exactly. So our path to, to potentially becoming a LiveView first kind of company is to start with internal tools where it makes sense. And so there are several tools that we, that we run internally, which we found already that we can be more productive by using LiveView, but one, I think the biggest challenge with adopting a LiveView is convincing your existing front end developers that it's a good choice.

 

Yeah, I can imagine.

 

One, right, because they're in this JavaScript, Reacts or view or whatever world and that great ways to develop front end apps, components, you know, passing your props and it's a they're really nice ways to working with front end in the virtual DOM and all that kind of stuff. And I wouldn't necessarily say it's broken. The part that's broken is this the backend front end side. And that always ends up with front end, not being able to progress as fast as they want to because they're dependent on something changing in the backend. And I've never been a fan of this front end backend divide. And so I think if live view is to be successful, there needs to be some options there. So there already is one option that I think is kind of interesting. I haven't looked into fully, which is the live view react library. Uh, so I think it's on hex PM where instead of the front end being like H E X F E X H T M L, um, it's, uh, it's, you can mount react components in there. And actually what the live view sensors are props. So the props comes back and forth over the web socket. So you can still use react, but live view is powering your props. And then of course the surface that there's that library as well, which can make it a bit more approachable to people that are used to. more component driven interfaces. But I do see that

 

Mm-hmm.

 

as a battle that's going to be difficult.

 

Yeah, yeah. I mean, in general, I think it's super interesting to hear, I guess, this case study of like Elixir in the Wild, right? Because I mean, especially in the community and also in this podcast, we tend to talk about these things more in like these green field, perfect world scenarios, right? Like, okay, you have this new exciting thing, and it solves all your problems. I'm exaggerating here. But most of it... developers out there do not operate in that context. Most people in their day-to-day jobs, they don't work every day with Greenfield projects. They have existing applications at Brownfield. A Brownfield scenario where you have to build in and inside of an existing application, inside of an existing architecture with existing choices you can't just easily, nilly-nilly get rid of. I'm in the same position right now. We have a bit of a microservice architecture going on, but more like microservices going wrong. And we have also internal applications and app interfaces, which I've written in Vue for no good reason, except for back when it was built, the front-end people didn't have anything to do, so they built their app interface. That's the whole reason behind it. And in that existing context, you have these things, they work, and you might even have good reasons to say, hey, you might want to potentially switch technologies, but at the end of the day, you don't get built to... you don't get paid to build the perfect clean system, you get built to solve business problems. And then you have to find room and create room and figure out, okay, what's the most responsible and pragmatic way forward, or you can kind of achieve both. Because at the end of the day, we don't just want clean systems and easy to maintain systems because it's nicer to work with them, but also allows us to move faster. So there is a tangible impact, but it's more like a long term versus short term scenario. So yeah. Great to have here your real world experience in some of these cases.

 

Yeah, thank you. I think yeah, agility, that's the key for everybody's trying to achieve, right? It's agility.

 

Okay, Daniel, I would actually be interested, who knows, maybe one year down the road, we can have a chat again and see

 

I would

 

where

 

love

 

you've

 

to do

 

been

 

that.

 

moving.

 

Yeah, I would love to do that.

 

It's a recurring theme and figure, okay, 12 months later, how does it look like? But it was a pleasure having you here.

 

Yeah, thank you for having me. It's been a pleasure talking to you.

 

Okay, so if people want to get in touch with you, how would they do that? And I mean, on that note, are you hiring?

 

Um, we're not hiring right at the moment, but it would be good to, if anybody is interested in working with us to, we can have a conversation because there will be hiring opportunities come up as long as our plans go to, are successful over the coming months. And so definitely if anybody is interested in, in, in, uh, working with us and you can reach me at, uh, Danny at quickup.com that's Q U I Q U P.com. It's my email. Um, and also on, um, on Twitter. Danny Hawkins, if anybody wants to add me there, all good.

 

Nice. Then, as I mentioned before, we paid record, we are going to transition to picks. So I only have one pick this week. And that is a conference I'm going to attend in wait, not let me lie. In beginning of November and of October. And that's the Kandinsky. It's written with

 

smart.

 

Yeah, and it's Germany's, as far as I know, first and only Domain Driven Design conference. It takes place in Berlin. I'll be there. It will be a pleasure to meet a lot of these people. And if anyone, one of you, is interested and maybe lives in or around Germany and wants to learn more about Domain Driven Design, which I think you should, then you can find me there and should definitely take a look at the event. As usual, there's going to be a link. show notes. So Danny, what are your picks?

 

I've got three picks. I've got one for reading, one for health and one for fun. So for, we'll go backwards for fun. If anybody wants to try something that is completely mind blowing and a lot of fun, I started playing a virtual reality first person shooter game called Onward VR around two years ago and I'm addicted to it. So if you want. a kind of paintballing experience that you can just put on your head and, and then come back to your living room afterwards. Definitely recommend looking into that. It's a really, really fun thing to do. For health, I recently invested in a treadmill for my standing desk. And I can tell you it's the most, the best investment I've had for my health. And I can walk eight kilometres while I'm reading or working. And it took around a week to get used to, but it definitely would recommend, um, anybody that is sat at a desk all day and works as hard as me, having a treadmill under your desk is very useful. Uh, and for reading, I'm sure a lot of people already read this, but Accelerate is an amazing book. Um, it's based on a four year, uh, study of, um, uh, the way product and tech teams work. and the insights gained from that and what actually works to drive teams forward. It's one of the best reads that I've had in the past decade, so I definitely recommend that to anybody.

 

Good to know, it's on my ever-growing reading list, but I haven't yet gotten to it.

 

I was the same, I was the same for around a year and I regret not reading it sooner.

 

Okay, do you have a concrete recommendation for a treadmill or is it just whatever treadmill works?

 

So yeah, I got a SparNod. I got a SparNod one on Amazon, SparNod, S-P-A-R-N-O-D. It was only like $400, $350. It collapses so you can put it up against the wall and then you just pull it out when you need it for the standing desk. So yeah.

 

Nice, nice. That might be something I also should be getting. I don't have a standing desk yet, but I've been eyeing on getting one. Because I'm also working fully remote now, so... Eh!

 

It's

 

You

 

so

 

know?

 

good, it's so good, it's so good. I was just having the battle, like I've got a gym downstairs but I'm not going there, because I go down there and I think I want to go upstairs and do my work.

 

Yeah, yeah, yeah. Ah, yeah. All the things, all the things. Okay, what was a pleasure having you, Danny. What was a pleasure talking to you. It was really fun.

 

Ie, mae'n dda i chi. A rwy'n edrych ar y cyfle i ddod yn unig, neu efallai yn dda, a ddod yn fawr, rwy'n bwysig i wneud hynny.

 

Yeah, yeah. So, sounds good, sounds good. Okay, and it was also a pleasure, I hope it was a pleasure listening to us, folks. And I hope you tune in next time when we have another episode of AlexiMix. Bye-bye!