Allen_Wyma:
So hello and welcome to another episode of Elixir Mix. I'm your host, Alan Wima. Today on the panel, we also have Adi Iyengard.
Adi_Iyengar:
Hello.
Allen_Wyma:
Did I actually say that right?
Adi_Iyengar:
Yes he did.
Allen_Wyma:
I guess I just kind of like ear memory because I never have to say your name usually. And we also have Herman Ver
Herman_Verschooten:
verschoten. Ja, ja, ja, close enough.
Allen_Wyma:
Schoten. Did I say that okay? I thought I had to a little bit close enough. Herman, how are you?
Herman_Verschooten:
Fine, thank you. Thank you for inviting me.
Allen_Wyma:
Yeah, like I said, I think I was the one that clicked the button, say invite. So I'm happy that you took the invitation because as we were just discussing in the show, you do deploy on bare metal. Oh, what happened? I mean, no, I'm not trying to make fun of you. But you know, as we're talking for the show, I barely hear about that anymore. And it's good to talk about that. But I think maybe we can start a little bit more about your past. And then that can kind of start to answer that question for us. So when did you get into programming and what were you doing first?
Herman_Verschooten:
Well, I got in programming some time ago, it was I think somewhere around 1983, 84. So I'm a bit older than most. And as usual it was then I started programming BASIC. And got very heavily into a computer system which was called Adam Computer System. And went from there to PC and worked my way through different languages. going from Pascal C to eventually the.NET family of languages, Visual Basic around 2005, made Ruby, continued with
Allen_Wyma:
Sorry,
Herman_Verschooten:
Ruby.
Allen_Wyma:
I wanted to stop you for a second.
Herman_Verschooten:
Yeah.
Allen_Wyma:
You said Adam, right? Isn't that the zip? Maybe I got this wrong in my history isn't it related to a corn and arm or am I wrong?
Herman_Verschooten:
No, the Atom computer
Allen_Wyma:
Oh,
Herman_Verschooten:
was
Allen_Wyma:
so embarrassing.
Herman_Verschooten:
a computer that was built based upon a game console that was released by Coleco Industries somewhere around 1982-83, which was a competitor for the Atari game console at that time. go way, way
Allen_Wyma:
Okay,
Herman_Verschooten:
back.
Allen_Wyma:
for some reason I thought Adam was but maybe I guess I obviously remember wrong I must be thinking about a corn in arm rather than
Herman_Verschooten:
No, no,
Allen_Wyma:
Adam.
Herman_Verschooten:
you didn't really get those in Belgium. So,
Allen_Wyma:
I
Herman_Verschooten:
uh...
Allen_Wyma:
didn't know Kalika was over there. I had Kalika vision at home somewhere, everything from my parents days.
Herman_Verschooten:
Well,
Allen_Wyma:
So
Herman_Verschooten:
it was
Allen_Wyma:
I
Herman_Verschooten:
very strange for us too, because Coligo, as far as we knew, was a toy manufacturer. They were most known in Belgium for their cabbage patch dolls. And then all of a sudden, we were offered... My father had a game in a toy store at Model, trains and things like that. And we already sold the Atari computer and they were offered to sell this Coligo Vision 2. with the promise that you could expand it to a real computer system. And that was the Atom computer. And that's how I really got started. That was somewhere really programming to programming and IT support. That was somewhere around 1984 when I was like, well, at that time, around 14, 15 years old.
Allen_Wyma:
So that time you could just homebrew stuff, right? You didn't have to have, I mean, because I have a clicker vision at home and it's basically all cartridge based. So there was some kind of way you can add a storage to it and then start actually just doing your own thing. You don't need to apply for a dev kit or license or whatever like you do nowadays.
Herman_Verschooten:
No, you just have a console that you could click into the front board and that turned it into a real computer system, which had a daisy wheel printer, had a tape drive, but not like the Commodore used to have, where you had your start and stop and rewind that you had to do. This was a system that was directly controlled by the computer, and that would just, where you could just get a catalog of the files on it and say, okay, I want to load that file. It was very novel and ahead of its time. It was a very fun system to program. I worked on it for years. I even wrote my own basic interpreter, but I was using Z80 assembler. And that was when I got into PCs because the Atom wasn't powerful enough to compile the source code anymore. So I had to convert the disks to PC format, do the compiling on PC, convert it back to the Atom format, and then try and see what I've done worked. So it was usually
Allen_Wyma:
That sounds
Herman_Verschooten:
a round trip
Allen_Wyma:
absolutely
Herman_Verschooten:
of several
Allen_Wyma:
horrible.
Herman_Verschooten:
hours.
Allen_Wyma:
That sounds horrible.
Herman_Verschooten:
Nah, it's fun, it's a hobby.
Allen_Wyma:
I guess we're all kind of masochist, we wouldn't be doing programming because debugging is like more than half your life, I think, right?
Herman_Verschooten:
Yeah, you have to remember that at that time we were running systems that had like a speed of 3.5 MHz. So it's not like what we have today.
Allen_Wyma:
Yeah,
Herman_Verschooten:
But
Allen_Wyma:
this is true.
Herman_Verschooten:
essentially I left the atom behind and kept working on the PCs. And as I just said, I also started moving languages, worked with Ruby for about 10 years, learned about Elixir, and tried it and fell in love with it.
Allen_Wyma:
What made you fall in love with it? Because I also came from Ruby. And I think I first tried Elixir, and it just didn't click because it's so different when you're actually working with it rather than from Ruby.
Herman_Verschooten:
I think the difference for me is that I never really liked all the object-oriented stuff. The languages I used at first, that was even before object-oriented really became known. I learned C, not C++. And I also worked with a language when I was working at a bank which was called SAS, which had like data steps in which you do transformation, read something from a file, do transformation, things like that manipulate the output. And that functional programming that you do there was something that I felt very much returning even when I started working with Elixir. So for me it felt like more like coming home. Ruby was fun, but Ruby always carried the danger that somebody could change something underneath you and all of a sudden something would crash. You wouldn't know why because you'd installed some kind of gem that would have changed. string for example and I really
Allen_Wyma:
Yeah.
Herman_Verschooten:
prefer I think that's maybe I still can do with H2 but I really prefer things to be more clear and not too much magic anymore.
Allen_Wyma:
Okay, I see.
Herman_Verschooten:
But after working at the bank, I started as an ISP in Belgium. So I was an Internet Service Provider from 1994 till the end of 2004. And that meant that we not only provided Internet access to our customers, first from Dialog Modems in Indian DSL, but I also provided hosting. And from that hosting, I've always run my own servers. I've never had service that were rented from somebody else. So when I stopped at the end of 2004, I sold it to my colleagues with a company called NIT4ALL. I still had my own service running, but then in a data center or just around the corner. And I still do. So I still have my own service. I still offer hosting to a number of customers. So when I need to deploy something, for me it's only logical to do it on bare metal, because I'm not going to rent some space in somebody else's server when I have my own. So
Allen_Wyma:
OK,
Herman_Verschooten:
it's not
Allen_Wyma:
yeah,
Herman_Verschooten:
that
Allen_Wyma:
this
Herman_Verschooten:
I'm
Allen_Wyma:
is,
Herman_Verschooten:
that kind of muscle-hissed,
Allen_Wyma:
yeah.
Herman_Verschooten:
but still.
Allen_Wyma:
No, this is this is true. But I mean, there's a lot of setup involved when you have to run your own kind of stuff, right? Like, I think nowadays with fly.io, or these Heroku like push, push services, I mean, life has just become easier where it's just like, okay, I don't want to have to set up a server because yes, you have to install image magic and whatever else kind of dependencies you need, or Erlang, etc. Nowadays, you can just take a base image that has what you need. And kick it over and then it just kind of runs, right? I mean, with a bare metal, you still have to set up everything. I mean, are you using a puppet or anything to kind of automate that process? Or?
Herman_Verschooten:
No, no, no, but I'm using Proxmox, which is a virtualization platform. So the basic installation of Linux is not something I really have to do. I just choose the image and I've got an image with most of the things already installed. And I just need to add those things that I need, like for example, a Postgres. I don't need to install Erlang because that's what I do from my deploy.
Allen_Wyma:
Wait, so.
Herman_Verschooten:
I spent a long time perfecting my GitHub action, which is in itself already a continuous of what I used to use on CircleCI.
Adi_Iyengar:
Yeah, I think that's one of the, I mean, having deployed Elixir applications like this for five years, first five years, first four years of my life as an Elixir engineer, I mean, one of the biggest advantages is that you don't need Erlang runtime, right? Especially like build tools like distillery and stuff. I think one of the problems that very early that I ran into, I'm sure you did too and have a solution for that, is that the build environment in which you build the app, the release, should match the runtime environment. And in some cases, it's not just the operating system version, but packages need to also be the same. And that was such a big black box for me. But that was one of the biggest reasons why I wanted to go towards containerization and deploying as a container. So how do you manage stuff like that?
Herman_Verschooten:
Well, I must say I haven't run into too many problems. I use Ubuntu as a batch image, which is kind of stable. And I also only use LTS versions, so the long-time support versions, because that makes it much more easy to avoid those kind of issues. I know when I first started doing my deployments, it was really copying your code to the server, installing Erlang and Elixir and doing your compilation on the server, which was a problem because every time you did an update, it would install a new version of Erlang and everything would break. And I also made the foray into containerization by having it run as a just mixed Phoenix server in a Docker image and things like that. But I never felt happy with it. I always felt it was like, yeah, it does work. But every time you needed to deploy something, it was like, how do I do this again? Okay, I'd make a script, I'd run my own Docker, what do you call it? Registry. Because the images of my source, I just don't want to have somewhere
Allen_Wyma:
Yeah.
Herman_Verschooten:
which I had to remember which server it was on and things like that. And it always, it always was more than a hassle to me than what it was worth. Um, it's a difference of opinion. It's a difference of style. I know. Um, I never went into Kubernetes or things like that. I looked at it, but one of my issues always was what would I do with my database? Now for me,
Allen_Wyma:
Yeah.
Herman_Verschooten:
it's one image, it's one server, which gets back up every night. My databases get back up to every hour. I'm quite comfortable with that. If I need to set up another server, spin up another server, well, it takes me not that long. Setting up the Proxmox server takes a couple of minutes and installing the rest of the packages I need, it's like 10, 15 minutes more. It's the first thing I do. when I start a new project is getting my deployment fixed.
Allen_Wyma:
Hmm.
Herman_Verschooten:
And that will take me, like, say, like an hour.
Adi_Iyengar:
Yeah, I do think Kubernetes would have been an overkill for you, right? You don't need, you have this one singular app that you're deploying that you want, you know, the fact that you're focused on one app, you might not need the level of orchestration that Kubernetes, you know, comes with. But I think setting up is one part, right? But then there's maintenance, right? And for you, it's like, you know, something you're used to, you know, making sure the server is updated, making
Herman_Verschooten:
Mm-hmm.
Adi_Iyengar:
sure Postgres versions are, you know, updated and the data is compatible, right? Stuff like that, right? As you manage all of those things. How has been your experience of managing, updating security related updates or performance related updates through the lifetime of an app?
Herman_Verschooten:
Well, I know it's more than a single app I run about, I think about 30 or 40 apps in total.
Adi_Iyengar:
that depend on each other.
Herman_Verschooten:
Some depend on each other.
Adi_Iyengar:
Okay.
Herman_Verschooten:
I, outside of this hosting and the IT support business, I have another business which is called Gratwifi, which is a hotspot system, in which I write all the software. That means also the software runs on the routers and things like that, which I That's everything I write. And the backend exists of three Phoenix applications that talk to each other using PubSub and using clustering. And they all use the same database server. So there's sometimes I have to make sure, but it's. I remember when I was using Ruby and you do an upgrade on one server and you for example you'd upgraded your MySQL, you could get into trouble that the Ruby MySQL client wouldn't talk to the MySQL server you had, which is something I haven't run into with Postgres. It's much more stable. I usually try my upgrades first on one of the lesser important projects. Once I'm sure it's up and running and causes no problem, then I'll go on to the others. Security updates is something I do, let's say, weekly, bi-weekly. And it's still something I do by hand. I know that you can do that there are scripting tools that you can use, but my experience is that every server is like a person and they never do the same. You can install three servers at the same time, install the same packages, and one of the three will give you an error. Using the same base image from the save server, it's just the way IT works.
Adi_Iyengar:
Right. Yeah. And I think it really this, yeah, I think this whole approach really speaks to, like you said, your experience before of being in ISP and managing being a place where other people use your services to deploy their applications. So yeah, I think that's, like I said, that is the reason why you're able to set up, maintain, and also cross that psychological bridge to. do weekly maintenance and security updates to people like Alan and me. It's like, oh, man, I want this work to be managed by an external service and minimize the discrepancies and whatever things that might break that we have less knowledge about. Right? But yeah, that's where you're different, and you do this all on your own. I guess one last query I have at a high level would be an advantage of using even a simple hosted service like Heroku, not even Kubernetes, not even AWS. Something simple. Say you have your SLA start changing. So say a subset of your data needs a lot of reads. To create a follower database that's a read replica is two seconds work. And even this most simplest managed services platform, how do you deal with such more advanced scalability or availability-based or consistency-based requests?
Herman_Verschooten:
Well, I haven't yet, so it's never been an issue. I used to run my... My gotwifi project started out as a Ruby application in something like 2008. And there I used to use MySQL, which ran on the server in the data center and which had a slave running on my machine here at home. With Postgres, I've never done this. I've just set a command that syncs over and it's... It's all I need at the moment. It's one of the issues I have with Postgres, that there is no clear way for me to do an easy replication, to do
Adi_Iyengar:
Right.
Herman_Verschooten:
an easy failover setup and things like that. And that's one of the reasons I have some looked at those kinds of hosting solutions, but not to use them, but I'm to run them myself. Because that's always, that's my nature. So in the time when I was still doing Ruby, I looked at some of the projects that mimic Heroku, that would allow you to do the same, but none of them were stable enough to use it in production. So I just kept plotting on the way I've been doing for the past, well, let's see, since 1994, which is how long? Too long.
Adi_Iyengar:
Hahaha!
Herman_Verschooten:
But it's also part of my hobby.
Adi_Iyengar:
Hmm.
Herman_Verschooten:
So I'm one of those lucky guys that can combine his work with his hobby, with his passion.
Adi_Iyengar:
Great.
Herman_Verschooten:
It's also a part of my creativity.
Adi_Iyengar:
Yeah. Yeah, I think another, and this is something, just the guy in me who likes to do more root cause analysis. I think another thing that you, I'm starting to slowly see in you as a person, is the whole reasoning behind Ruby versus C. You mentioned C, right?
Herman_Verschooten:
Yeah?
Adi_Iyengar:
Is I think visibility and control over things that you're building, right? The light, the level of control you in the whole 100% control to 0% control spectrum, you're closer towards 100% control. And I can very much relate to that. The first time I used Kubernetes, I actually went and got the CKA certificate, installed Kubernetes admin on a machine, and learned how to manage that before going to manage Kubernetes. So I can totally relate to that. But it's also, like I said, a hobby, where you like to do it. which is why where you get the motivation to continuously do it week after week. So that's really really cool.
Herman_Verschooten:
Yeah, it's... You know, having finished, well not finished because they're never finished, but having my deploy workflows in each project and having them kind of standardized for myself so I easily know what to change where and now having the possibility of just doing a push and having my deploy run saves me a lot of time of course when I'm doing my development. But toying and tinkering with those scripts is fun too.
Adi_Iyengar:
Totally.
Herman_Verschooten:
I recently had to update most of my scripts because there's a deprecation with the Node.js in the GitHub workers. So I could change most of the scripts on all my projects. It's a hassle, but it's something I also like to do. When the time is there, I'll do them and I'll do all my projects. Even those that haven't changed in like more than a year in code, I'll update them. So that if I need to change something, I'll be sure I can continue. And that's one of the other things that I like in our environment that we have now is when I try to go back to a Ruby project of like three, four years ago, most of the time I can't even run it anymore. That's when I use my Docker experience to try and create a Docker image that will allow me to run that project from a couple of years back to make a change to be able to test it. But I have other projects that I wrote in Alexiv that I wrote like four, five, six years ago. I can still run them. I'm not gonna say there's no issue at all, but the issues that are there are easily fixed.
Adi_Iyengar:
Right.
Herman_Verschooten:
It's not like my brother also does programming. He also makes website, but he's still in the Ruby ecosystem. And he complains about this all the time. He needs to do something on projects for one customer. He needs to have a certain version of a gem. He installs it. He goes to another project, and the other project doesn't work anymore because it needs the same gem, but another version. And it just doesn't work. And... With us, with the mix log file, it all works as it should. especially now with the changes that have recently been made with getting rid of webpack. That's such a blessing.
Adi_Iyengar:
Yeah. Totally.
Herman_Verschooten:
That was usually the thing that wouldn't work anymore.
Adi_Iyengar:
Yeah.
Herman_Verschooten:
Goodbye and good riddance.
Adi_Iyengar:
100%.
Herman_Verschooten:
Really?
Adi_Iyengar:
100%. Yep.
Herman_Verschooten:
One of the other languages I use is Elm.
Adi_Iyengar:
Hmm.
Herman_Verschooten:
And I've been using Elm for a couple of years, for almost as long as Elixir. And that used to be a problem because you want to do something in a project that has Elm in it, and you'd get stuck for two, three days getting webpacking back up and running.
Adi_Iyengar:
Right?
Herman_Verschooten:
And now with ESBuild, it's no problem. And I just add a script. that compiles my LM application, uses ESBuild, and it works.
Allen_Wyma:
It's
Adi_Iyengar:
Actually,
Allen_Wyma:
interesting
Adi_Iyengar:
I'm
Allen_Wyma:
that you're
Adi_Iyengar:
looking
Allen_Wyma:
using,
Adi_Iyengar:
at your...
Allen_Wyma:
sorry, as I say, it's really interesting you're using Elm because I want to use Elm. But I always have a lot of concerns because it's not being actively pushed out new versions. And it's kind of run by one guy almost to a certain extent. I haven't really looked at it in a while. But I love the ideas. I love all this kind of stuff. It just seems like it's been stagnant. And
Herman_Verschooten:
Well,
Allen_Wyma:
it's very niche in terms of.
Herman_Verschooten:
that's not entirely
Allen_Wyma:
I
Herman_Verschooten:
true.
Allen_Wyma:
just said, that's why I said it seems.
Herman_Verschooten:
Yeah,
Allen_Wyma:
It doesn't mean that
Herman_Verschooten:
yeah,
Allen_Wyma:
that's
Herman_Verschooten:
yeah, but
Allen_Wyma:
what it is.
Herman_Verschooten:
I think... We're at a point in Elm where we don't need any changes anymore. It's like, it's like Jose said a couple of years back. Um, I don't think there will be an Elixir version two because our language is stable. And it's the same thing with Elm. There are some things that could be improved and there are some projects that do a better compilation or do an optimization of the compiled code. But the language in itself is kind of finished. So there's no need to keep pushing out updates because FN doesn't want change. And if there's no need to change, we can do. My brother uses L much more than I do because most of his front-end, he tries to use L back-end Ruby. And he's... and over the past like five years is not running to a situation where you couldn't do something with Elm. So, well, it's, every time I'm getting on the phone because of your questions, I really love Elm, especially the compiler. Compiler is so friendly.
Adi_Iyengar:
I love the AMP compiler for sure.
Allen_Wyma:
Now, now, now the other side too is that's kind of go back to the analogy or the comparison you did. Okay, yes, Jose made that com or sorry, Jose made that comment. But at the same time, we've since increased the minor version of elixir several times, adding new features, adding new compilation fixes, et cetera. Although he said, yes, it's basically done. Uh, I don't see any new version of Elm coming out. And that gives me some worry about if we should still be looking at it, right? Even like Phoenix and stuff, it's basically done, but you know, okay. Now they add the tailwind generators. There's geek stuff is coming in. Like there's more things coming to the community. They said, maybe it's because I'm in that community. I see that
Herman_Verschooten:
Yeah,
Allen_Wyma:
I
Herman_Verschooten:
but
Allen_Wyma:
haven't looked
Herman_Verschooten:
I think
Allen_Wyma:
a lot at
Herman_Verschooten:
the...
Allen_Wyma:
Elm.
Herman_Verschooten:
How should I translate this? The field in which you use alum is much smaller than the field in which you can use Alexir or even Phoenix. It's used to make applications in your browser. It's used to have a dependable language that you can use to create UIs. It's not an all-purpose language like Alexiris. So there's much less room to add things to it. And... adding things to add things and bringing out releases to bring out releases to say hey we're still alive. I don't know what's gonna happen to Elm, I don't know what Evan's plans are. And I think everybody agrees that we haven't heard much of him over the past couple of years. But it's not like we're screaming for a new version. This version works. As long as there is no real problem with something that comes along that really says, okay, now we can't use it anymore because this or this has changed in the browsers and it will not work anymore. I don't see much of a problem. маях и ним.
Adi_Iyengar:
I am not very, I haven't seriously coded an Elm in last three and a half years. I'm gonna, I don't think I deserve to have an opinion here, but I'm gonna very not smoothly transition this conversation back to the Beam world.
Herman_Verschooten:
Yeah.
Adi_Iyengar:
I was looking, I think one way that you do things differently than others who use, you know, like bare metal is at least from a while ago when I remember. knowing people who used to use Parametric is that you use SystemD to run the app, right?
Herman_Verschooten:
Mm-hmm.
Adi_Iyengar:
You don't just run the app. And I think you want to talk more about that and especially maybe also have you experimented with the hot upgrade, hard code upgrade, and how would it work with SystemD? I don't really know if it will. So I don't know. Yeah, I would love to get some thoughts on that.
Herman_Verschooten:
Uh. I didn't fully get your question.
Adi_Iyengar:
Yeah, so just it's a twofold question. The first one is how, what made you choose SystemD to run the app and manage the app? And second, have you played with hard code upgrades? I think Distillery with eDeliver, I think it was. They used to support the hard code updates. I'm not sure how it will work with the way your app being managed by SystemD. I don't know if it will call the exec stop, invoke the exec stop or not. I'm not sure. Learn more about that.
Herman_Verschooten:
Well, I used to use hard code upgrades a couple of years back, the time when I was still doing my compilations on the server itself. I used to use hard code upgrades, but I'm not doing it anymore because my upgrade process with using this is so fluid that it isn't really necessary. And
Adi_Iyengar:
Mm-hmm.
Herman_Verschooten:
the reason for using systemd is because it's the native way of doing things in Ubuntu since a couple of years. Before it used to be initd, and I used to write initd scripts. But to be honest, I prefer systemd. It's a bit easier. OK, it's not really code you write. It's just statements. But I still have to write initd, for example, for the software that I run on openwrt. So it's not like I don't do it anymore. But I can. I like the system D, the way it works, the way it can automatically restart an application when something fails, works very nicely, the way you can easily put in your environment and put in everything that you need, which was always a hassle when using any D, which is also a bit of a problem if you try to run your own scripts. And why would you? I mean, there's no, no reason, there's no extra in running your own script or to run your application. So I use this, it works for me. The only thing that's something that I had a discussion about a couple of years back, trying to remember his name, who wrote Distillery, that we had a discussion about,
Adi_Iyengar:
Just Bitwalker is this GitHub,
Herman_Verschooten:
yeah,
Adi_Iyengar:
I
Herman_Verschooten:
Bitwalker,
Adi_Iyengar:
know that.
Herman_Verschooten:
Paul Schoenfeld, I think he's called. And we had a discussion about something that's common in the Ruby world, which they call rolling upgrades. which we don't usually have because we have a single application that's powerful enough to run everything. But one of the issues I ran into was that I need to stop my application, unpack my tar file and restart my application. Now, I haven't tested it with the latest versions of the release scripts that Mix makes. One of the problems I had at that time was that in the stopping of your application, Distillery would write the version of the Erlang runtime and of your LXZ. In the file it uses to boot them up. So if you tried to upgrade your application while it was running, it would copy the new files in place, it would put the new versions of the ERTs and of AlexXel if they've been upgraded in that start file, but once you did the shutdown of your application to restart the application, it would go back to the previous versions, then all kinds of nastiness would happen. Because one of the things that I don't do or that often do is increase my version numbers. because in the way that I deploy it isn't really necessary. And of course for me that would make it more difficult because every time I deploy a new version it would create a new lib tree or add files to the lib tree and that would create a growth on my server that would at a certain moment be difficult to manage. So I did do it sometimes, but most of the times it was more to solve a problem with assets than with Elixir code.
Adi_Iyengar:
Got it. Yeah, this is ringing a bell. Again, I haven't done this in like three or four years, but I think the file you're talking about, is that the VM.args file is that what you're talking about? Or is it a different one that stored
Herman_Verschooten:
I think it
Adi_Iyengar:
all
Herman_Verschooten:
was
Adi_Iyengar:
those?
Herman_Verschooten:
start of Earl or something like that.
Adi_Iyengar:
Gotcha, gotcha, OK. Cool. I mean, that's a fair problem. So yeah, and like I said, you haven't had the need to do those updates, so there's no need for that. I guess this reminds me of another problem I had, was I think logs. And this is again, like in 2019. I know the logs would get erased between each update. Is it like a central place where you drain all the logs before you copy a new binary
Herman_Verschooten:
Well,
Adi_Iyengar:
to the server?
Herman_Verschooten:
I don't have that problem because all my logging is coming in CIS log.
Adi_Iyengar:
Alright, that's a good point.
Herman_Verschooten:
So that's an issue.
Adi_Iyengar:
Yeah. Yeah.
Herman_Verschooten:
I've been playing around with trying to get my logs more structured, JSON logging and getting logs into Grafana, but I'm still,
Adi_Iyengar:
Mmm.
Herman_Verschooten:
it's more like a toy project at the moment than something I'm really working on. But I have most of the pieces in place. I already have two or three products that do it, but I haven't had... time yet to go into Grafana and to really interact with the logs to see if it's really worth it. But it's an extension and it's a logical extension of what we're doing. And especially in the project for GradWifi, where I have three different servers running, it's nice to be able to create your logs at one place and also see, because I, for example, when one of my hotspot owners in his dashboard, for example, somebody who's online, who needs to go offline, he clicks on the button to kick him out. It goes through to the other server. So it sends a message by PubSub to the API server, which will then use Phoenix Channel to talk with the C code on the OpenWRT router to kill its session. And that'll call back and say, okay, it's done. And then it'll come back to the first application using PubSub and then he will go offline. And these interactions would also be nice if you could see the logging, like interwoven with each other.
Adi_Iyengar:
All right.
Herman_Verschooten:
And that's one of the reasons I want to... try and get this up and running and see, okay. One of the other things that's on my agenda to learn, which I still can't get my head around is telemetry.
Adi_Iyengar:
Hmm. That's called distributed tracing, right? That's what I talk about, right? Yeah, cool. Yeah, telemetry is really well built for that. But awesome. Very good. Alan, do you have any more questions for Herman?
Allen_Wyma:
No, I mean, it's definitely interesting. Of course I know that not every everything is meant for everybody, right? And we all don't do the same thing. So it's interesting to hear how you handle it and how you feel about it. I think a lot of it's about, like, you know, kind of what I took away today was a lot of it's kind of history, hobby, as you also said, interest, and I forgot what else is the word, and yeah, I think those are probably the two big ones. There's another one that's in my head,
Adi_Iyengar:
Control.
Allen_Wyma:
but I lost it. Yeah, control is another one. I forgot. That's a pretty big one. That's something I used to have. I think everybody, I think most programmers have that. They want to control everything because they know how bad things can happen. And only we know how bad it can happen. Only we know what has happened to us so we can stop all these bad things from happening, but then I think sounds like, um, I share the kind of same thing with audio is like, okay, I'm not going to use Kubernetes until I at least understand it admin moment, maybe try to run my own cluster. And it's like, okay. I know how this thing works now. I know how all the bad stuff. So let me just kind of slowly let go and let Amazon handle this. And then now you can focus more on the other issues that are more pressing rather than running a pseudo apt-get update, which I sometimes do and sometimes not. It just depends if I remember or not. I just like that feeling that I don't have to worry about another heart bleed or these kind of things. Usually, they have so many people handle that for me. But
Adi_Iyengar:
Yep.
Allen_Wyma:
at the same time, it's good to know, oh. you know, wait for these guys to fix it. Who knows when that's going to happen. And it's like, okay, I could just pop patch it. Because I just heard about it just broke out on the CV just came in. I know. And the fix is already done. Right. That's kind of a good feeling too.
Herman_Verschooten:
Yeah, it happens
Adi_Iyengar:
I think it's
Herman_Verschooten:
at
Adi_Iyengar:
all sus.
Herman_Verschooten:
5 o'clock in the morning, I'm already behind my computer. They say, okay, update on the server, update, server update. So then when my customer arrives in his office, he calls me and says, did you hear that? Yeah, the server's already upgraded.
Adi_Iyengar:
Yeah, it's also a symptom of, I think, how much technology has evolved, too, right? Like, Herman has been coding 10 years before I was even born, right? And a lot of what he was saying was, my dad was also an engineer. He keeps telling me all these stories. If I used to do anything a certain way, yes, you evolve as time goes. But it's harder for you to let go of control and knowledge that you know, right? I know how this works. So why should I not do it? Right. And Alan, you and I will hit that too in like six, seven years. Like we will hit it. You know, you have this chat GPT thing come in, and the next crop of engineers will be like, hey, just write 5% of the code that you deploy. And you'll be like, no, I want to write that. I want to have control over that. Right. So it's, again, I think a symptom of the fact that technology is just growing at an exponential rate. And we keep building abstractions on top of abstractions. And people are used to. lower level of abstraction want to retain some control over how to do their business.
Herman_Verschooten:
Yeah, because one of the other things I also run software on is on nerves. there you need to have some knowledge of the running system because
Adi_Iyengar:
Yeah.
Herman_Verschooten:
well it's easy to deploy a feed and tap to to nurse devices that's like nothing but if you want to do something more then you need to know more about running system and you need to know more about how it works and which files to edit or which extra kernel modules you need and things like that and use menu which is fun, to me it's fun. But I understand
Allen_Wyma:
I'm gonna
Herman_Verschooten:
people
Allen_Wyma:
give a
Herman_Verschooten:
that say, well, I don't care for that, I just want to be able to push the button and my application is live. I understand this perfectly, certainly. I stand in awe of the guys at Fly, and I love reading their blog posts. They have so much knowledge about all the interconnection and things like that. I love the way in which they've made a system that you can deploy your app around the world without any hassle. But sadly, I don't have that kind of application to run. So for me at the moment, it's not something I'm going to use. But I will read what they write and I will take their knowledge and try to apply it in situations where it works for me.
Allen_Wyma:
What I wanted to say is I kind of have another flip side to this, if I understand kind of what you guys are talking about. When I was working in the bank, we did have one part of the bank that gave us a JSON API. So actually, I'm sorry, I forgot the air quotes, JSON API. It was kind of interesting. So everything, you know, JSON, the keys have to be strings, right, double quoted strings. Well, all of their values, including no, were also double quoted strings. And they were missing commas here and there. Um, so, and, and basically, you know, the, the bank, which I think you said you worked in finance or the bank before. At least some of this, what I heard from my manager at the time, who's much senior than me was telling me that back then they would just make their own libraries, you couldn't bring it outside code, you had to make your own stuff. So it's very common to have a lot of people having this kind of homebrew kind of library that just was just not up to par. I mean, the positive is that you had the maintainer next to you. The negative was that he's busy maintaining his own stuff, and he's not spending time on the actual core business stuff that needs to happen. Instead, he's trying to figure out where to put commas and not to double quote null. So this kind of stuff. Sorry, I just want to give the flip side to kind of what
Herman_Verschooten:
No, no,
Allen_Wyma:
I understood you guys were talking about.
Herman_Verschooten:
it's true. It's true. It's something that I sometimes notice when I'm looking at something, I'm looking for some kind of functionality. And there's somebody wrote a library. And I'm looking at the library. I don't really like the API he's made. And my next step then normally would be, OK, I'll write my own library for this. And sometimes I really have to tell myself, no. Fix this so everybody can enjoy it instead of wasting your time on creating something that only you are going to use. I've been dying for years to create a PDF library. I make a lot of PDFs, I do my own invoicing and things like that, so I need to generate PDFs. I hate shelling out. So I do not want to use things like PDFTK or anything else that you need to run a command line application to do something. I want these things to be my code. Because if something goes wrong, then in my code, I can have my logging and things like that. That will give me some kind of clue while the other applications, most of the time, it's like, eh, tough luck, it didn't work. I've been helping Andrew on this PDF library. Still, it's like, hmm, sometimes I've got somewhere here, my desk, the specification of PDF, it's like this. It's like, what is it, 750 pages, Vector Verso, two-sided, with just the specification. It's like crazy, but still there's something, every, at least every year. I'm like two weeks working on a PDF library and then I drop it again. Well, it's, it's, it's, I think it's, it has something to do with age, certainly with the, the, the, the, the history we have, as you say, because when we started out, there was nothing. When I used to program in BASIC on the other computer, there was nothing. There was no community like there is now, especially not in our country. Maybe it was in the States or something like that, but real communities... For me, the Elixir community is my first real community that I take part in. I never felt at home in the Ruby community. When I went to the Rizikov in the States in 2016, it was my first ever conference I went to. So then if we look at 2016, I was like 46 at that time. So it's certainly something to do with age, with history, with the way we used to do things, but it's fun. When it gets boring, I'll try to find something that does it for me.
Adi_Iyengar:
Hehehe
Herman_Verschooten:
Yep.
Allen_Wyma:
OK, yeah, that definitely gave me another perspective on this. And I think I have a book about system D somewhere on my Kindle library. I might need to correct it open still. Adi, do you have anything else, or shall we start the transition over to PIX?
Adi_Iyengar:
All set.
Allen_Wyma:
OK. OK. Shall we have Adi go first since hosting?
Adi_Iyengar:
I am still trying to think. Give me like 30 seconds, dude.
Allen_Wyma:
Good thing I thought mine out many years ago. Since we're kind of talking a lot about infrastructure today, a tool that I've used for a long time, because again, like I said before, because I use a lot of Kubernetes, I use this tool called Kubernetes, which is a paid solution. But I think I paid like $30 one time, or $60 I forgot. But it's definitely saved a lot of time, and it's a lot easier to use than it is. command line, right? So I work with a lot of, I have a lot of, how do I say this? Most of the people I work with don't have as much Kubernetes experience as me, so I don't quite understand it. So having a visual tool that I can click around is super useful for them, and also for me, because I like to click with my mouse rather than type with my keyboard. I'm kind of a weirdo like that. So this tool has been super helpful for me. I think it's worth it. I think it's similar to kind of Lens and. They're all about the same because basically, Kubernetes is about the same. But I think this one's pretty good compared to lens. That's my pick. Shall we go to Herman then?
Herman_Verschooten:
Yeah, okay. So I have three picks. First one is the books by the late Sir Terry Pratchett, most commonly known the Discworld novels and recently one of the books that he co-wrote with Neil Gaiman has become very popular which is Good Omens, which now has a series on Amazon Prime and a second series in the make. I don't know whether you've heard of Terry Pratchett and the Discworld. He's a fun writer. It's stories of a world that is flat as a pancake, and is carried on the backs of four elephants that themselves stand on a giant turtle that swims through space. So it's kind of different. It has its share of witches and trolls and things like that. So it's a... It's a kind of unique world, but it's very fun. I think he wrote about 30 books about it. I've been enjoying them for the past, like, I think 25 years. My second pick is a library by Ossasha Jovic, which is called Site Encrypt. which comes back a bit to my thing about doing everything yourself. And since learning of this library, I kind of dropped using Nginx as a reverse proxy in front of my applications to do SSL and things like that. And I just have Phoenix do it using this library and it works extremely well. And for my third pick, people who know me and people who've met me on conferences and things like that know that I like to drink beer. And I live in Antwerp in Belgium, and Belgium is kind of known for beer. And not far away from me is a company called the Antwerp Brewery Company, which translates as the Antwerp Brew Company. And they have a number of very, very, very nice local beers. I don't know whether they export, but a reason more to come to Antwerp once in a while and indulge yourself. Those were my picks.
Allen_Wyma:
Great.
Adi_Iyengar:
Awesome. I have a pic ready now. So this is a resource I've been using for a little over a year now. It's called Dribble, D-R-I-B-B-L-E. It's a great place to just go and get inspiration for UIs. I know a lot of people who use Elixir are early stage startups or have a lot of side projects and stuff. Tailwind UI is great to get. initial you know set of template but if you are working on a you know for example a weird search feature you know that searches across multiple types of records how should it look like just go to dribble type search in the search box and get inspiration a lot of times you can also get to the website and like also just go get the source code right so but it's really cool has saved me a lot of time the startup that i was at When I built their product, which is closing on series A, most of their product was Dribbble-inspired.
Herman_Verschooten:
I'm
Adi_Iyengar:
So
Herman_Verschooten:
sorry.
Adi_Iyengar:
yeah, you can build a profitable, huge product using inspirations from other websites. I also have like, it's not quite a big, it's like a life update. I just started working at the score. As a staff engineer last week, I was sick. Last week, I didn't get a lot done, but this week I'm talking to more people. It's a great place to work. Everyone's really nice. So if anyone's looking for a good elixir position, pedal stack, work with a great bunch of people. And the code base is pretty good too, from what I've seen. I can highly recommend this score already. So yeah, that's pretty much it.
Allen_Wyma:
So no horror stories like we talked about last time.
Adi_Iyengar:
Y-yeah, no, yeah, uh,
Allen_Wyma:
No
Adi_Iyengar:
yeah
Allen_Wyma:
process dictionary being used everywhere.
Herman_Verschooten:
Thank you. Bye.
Adi_Iyengar:
No, not at all. Yeah, the code base is actually really good, unlike many of the big companies that adopted Elixir early stage. The score has really, I think, they've been very intentional about keeping some quality. Their quality is way above my bare minimum, and that's great. It's not where I would like an Elixir code base to be, obviously. But that's where I come in, and hopefully I can help them improve their quality.
Allen_Wyma:
That's awesome.
Herman_Verschooten:
Sounds good.
Allen_Wyma:
Great.
Herman_Verschooten:
Okay.
Allen_Wyma:
Well, and with that, I forgot how we usually end this. But that's the end of this. That's how the elixir was drink. So feels good. See you guys next time on another episode of elixir mix. Bye