Adi, Allen, and Sascha join this week's panelist episode to tackle different deployment applications in Elixir. Allen leads the show as he talks about the application "Fly.io". He describes how he used the software, how it works, and its benefit to users. On the other hand, Adi explains why he prefers to use Heroku. He explains its process and why it is also user-friendly. Lastly, they highlight the tools they think developers should utilize in 2023.

Hey everybody and welcome to another episode of Elixier Mix. This week on the panel we have Adi Aijonger,

 

Hello.

 

and Alan Weimar,

 

Hello.

 

and Sasha Wolf, that's me. And we have, no, special guest, just us today. And I think Alan wants to talk about something with flies, I didn't quite get it. Alan, what do you want to talk about today?

 

I don't get it either, to be honest. But no, actually what I wanted to talk about is I think the state of deployments has changed in the past couple of years. Maybe for you guys, it may have now changed. But I've been doing a lot of deployments off of AWS EKS. But finally, I ran into a project where they needed to deploy something to make it super simple. Back in the day, it used to be Heroku. And just like any other project, every project I have is low budget. So they're looking for something free or cheap. So like the $7 option, like Heroku, that can probably work, right? But Heroku, since they removed their free tier, I feel like they've kind of gone a little bit long on the tooth, as they say, right? A little bit past its prime. And now there's so many different choices. And we went through quite a few, and we ended up going with FlyIO for a couple of different reasons. One is the developer I'm working out with, he recommended it. Two is I keep hearing about it, but never actually tried it. I think Chris McCord's working there now still. And they're also committing a loss up to the Alexa community. And of course, there's Gigalexr, Render. There's so many different options. So we're doing FlyIO, and it's been a pretty enjoyable experience, but definitely something that takes time and get used to. I mean, I think, Adi, you said you worked with FlyIO before?

 

Yeah, a while ago. It was cool. I think at that time, the big draw for that was the whole edge kind of the game, right? I don't know if that's still the big reason. It was not easy to set things up. This was like 2020, beginning of 2020. So yeah, I've heard it's changed quite a bit. And I know a couple of friends of mine, they were able to set up a project and run a Phoenix project very quickly 10 minutes or 15 minutes it sounded like. So, yeah, I would love to learn about your experiences with that, Alan.

 

Yeah, I mean, it definitely is a little bit, takes a little bit of time to get used to, right? Because you're using the command line. And there's a lot of commands to deal with, right? And you have to run this like flyctl launch. That one will basically set up everything that you need for you. It's cool. Like it reads your project, read your Docker file, gets everything ready to go. The only thing that I had a little bit of issue, which was I needed to create a test environment. And. So I created it and I created on somebody else's account because you can only have one free app on your account. And so I managed to connect and manage to do everything but like deploying the app to that one took a little bit of time because if you use the fly CTL launch, it tries to create another Tom file, which is not what I want to do. I don't want to necessarily write it. I mean, so I couldn't try to figure I couldn't quite figure it out. So I had to go this long route of trying to redo everything that it does for you. other box like creating the phew checks host, creating the database, connecting it. How do you look confused?

 

No, I'm just confused that it was that hard to do it. Sorry,

 

Yeah,

 

continue,

 

yeah.

 

yeah.

 

Yeah, I was surprised. Maybe because I'm doing this with another account and another app name, and the way I was doing is probably not the best way to do it right. It must be a way that you can probably just clone it and say, OK, clone this environment and make another one for me. But I mean, this one needs to be moved to another account anyway, so we just created another account. So yeah, so again, maybe I'm not doing it the right way. So it took me some time, but I figured it out eventually, just doing everything a long way.

 

Does Fly have Terraform support? It didn't have it earlier when I was using it, but

 

No.

 

that would be the right way to do it if they do it.

 

Oh, yeah. That would be interesting if it does, right? But I mean, there's not much there. You just basically say, OK, here's my app, read the Docker file, and then it knows, OK, this is a Phoenix app. So that creates like the typical phx host, 8080 port, everything you need, attaches the Postgres instance, all that kind of stuff. But if you do it by hand, it's not that simple. But

 

Right.

 

yeah, the most difficult part is like, OK, which which region do I use?

 

Right. But that's where Terraform would help, right? Like if you have a production version of the app and you want to just like a QA or test version, just like copy the file and rename and just run one command and hopefully you're all set, right? That's like the advantage. I think for me, and this is gonna be a very quick tangent, we'll talk about Heroku more, like a huge advantage of Heroku is obviously the ease of setting and stuff, but Terraform makes it like even better if you know how to use Heroku with Terraform, which took us a couple attempts to use. It made life very easy to build many apps, not just for testing QA, but I also advise so many startups and I build their initial MVP. It just allows me to take this template app and within 10 minutes have an app, production ready app, with staging QA production environment. I'm sure you can do it with FlyIO too, but I just haven't done it yet. And I don't know if they have Terraform adapter.

 

Well, I mean, if you can order pizza with terraform, you should be able to create flyo apps with terraform, right?

 

I'm sorry, what? You can order pizza with terrafoam.

 

You didn't know about this? I thought everybody knew about this one. You

 

I mean,

 

get both of you don't know this or just

 

I hadn't heard of this particular thing, but I mean, depending on what you use Terraform for, you should be able to do anything with Terraform, right? As long as you write an adapter for it, so.

 

Yeah, yeah, there's a terraform. So for dominoes, it's in terraform registry, you can order pizza with it. I'll drop it into our chat if you guys are curious about this. Sasha, you look like

 

That

 

you want

 

is

 

to

 

funny.

 

use it.

 

I'm not even surprised to be honest. Hahaha.

 

Yeah. And they got the, is that the Nink? I forgot what the heck they are. The whiz? Or what do they call that little character? I forgot that the character of Domino's. Or the nonk? Maybe you don't remember. The Noid, sorry. The Noid. So Domino's little mascot is actually called the Noid. But yeah, I mean, I think I heard people actually using it for like, if your app deploys fine, then it orders a pizza on a Friday or something like that to kind of celebrate the week or whatever.

 

That's not a bad idea actually.

 

Yeah, it's not bad. I mean, it's just like, okay, you know, I guess it works. But it's I think it's officially only us I guess, because I don't think I have dominoes out here. Sure, not out here. We have Pizza Hut, but no dominoes.

 

They're a dominant sort of US too, but Sasha, hey, we need to write one for a vegan pizza place.

 

Stükkvärk on jõumeneid ehe fiegenpizza. So you were also saying, like, I mean, basically talking about the experience with Fly, right? Isn't there also Render now, which is not also a thing? At least I've seen it in like the Twitter sphere, thrown

 

Yeah,

 

around.

 

I haven't played around with it. But sorry, I just started working with Fly. And it looks like there is a Terraform for FlyIO. I'm pretty happy with FlyIO. I think it's pretty nice. But yeah, I just find if you wanted to create another environment, it's not easy just to say, OK, give me an app, a database, whatever. you actually have to like have like a blank app ready to go in. So it's like sometimes the convenience of things also makes it more complicated. Right. So again, the first time I set up read my app, saw everything creative and I knew that was simple. I want to create another app somewhere else. That one's more complicated because it was trying to override the file and everything, which maybe I did something wrong.

 

You can have a copied repost story which just follows the main

 

Yeah.

 

of a different branch.

 

Yeah, I could

 

That'd

 

do

 

be

 

that,

 

like the

 

but

 

simplest way to solve that.

 

I don't want to do things simple. Why would I want to do something simple?

 

That's what we engineers like, simple solutions to complex problems, right? That's what engineering is all about.

 

No, we like

 

No.

 

to design that, but we don't like to use that, I think.

 

Nice.

 

Isn't it like complex solution to simple problems? That was always my impression.

 

Yeah, yeah,

 

Hahaha

 

that sounds like exactly what I'm doing with this with this app right now is trying to give it a complex solution. It's a very simple problem.

 

Peace.

 

Yeah, but I do like that you can that everything is versioning up, which is kind of nice.

 

Can you elaborate on that? What do you mean by everything is versioning up?

 

Oh, like, so every time you set a secret, it basically increments the version number. So it's very much like a what do you call that like a read only kind of system to the most part.

 

Oh god, I don't know that's a fly manager subversion for you, of your app.

 

Yeah. And the other thing, too, is I couldn't figure out a way to roll back, actually. In fact, I think they don't have that. If

 

Hmm.

 

I tried looking a few times, and people were saying that they wished they had it, the only thing I could find is that because it builds like a Docker image, is that you can say, OK, deploy, and then you can name the Docker

 

Old version,

 

image.

 

yep.

 

And it would just basically bump up the version again, and then just use the old image. I thought it was a little bit weird. Because it's like, well, why can't I just

 

Got

 

go

 

it.

 

back one? Just point to the back one. Why do I got to make a new deployment, which is a little

 

Yeah,

 

bit weird.

 

I guess come to think of it, I think Heroku does that too, right? It also

 

Today.

 

like implicitly versions each release or each update you make. If you add a new container or a new secret, like you said, it adds a new release. But but that's interesting. I fly does it too. It's also interesting that they don't allow rollback. It's probably because you know, for like failures like handling failures and stuff, but

 

Maybe.

 

it's also yeah. So you said you have to name your Docker image. Do you not have access to the registry? that fly comes with a can you not get

 

I

 

a previous

 

think you,

 

build.

 

I think you may be able to like, all I know is that I wanted to roll back and the only thing I could find is somebody said that you could say, okay, fly CTL deploy dash i and then you can name the image.

 

Hmm.

 

And it would pull that image and make a new deployment with that image. So I'm effectively rolling back but actually rolling forward with an old image. So that was the only way that I could fix this problem. So could I pull the image? Probably, but I don't really need to run the image locally. I just need to fix something because we had a bug in a later version. I need to roll back. I remember before, like when you use the rails, right, you could just if you used what was the name of that deployment one, do you remember? Have

 

Capistrano.

 

you ever deployed this? Yeah, Capicciano, right? That one had like five or 10 versions,

 

Yeah.

 

you may would just point that that's what I thought it would could do. But apparently not. I was a little bit surprised.

 

Yeah. I guess Rails, you can assume, I mean, it does make assumptions. Capastrano makes assumptions about structure of your app compared to Fly, which is more containerized. And you could have

 

Yeah.

 

a set of other containers it could depend on. So I guess I can see that to some extent.

 

So I guess that what does that make the landscape of deployments for like the options right now? There's fly obviously, there's render, which we can't really say anything about right now. There's still Heroku around.

 

Yeah.

 

And of course there's the classic role yourself. What if you were to choose, are the element like kind of like starting something from scratch or maybe also in the long run, what would your choice there be at the moment?

 

It depends. I mean, if it's just for myself, I would look at some other options. If it's for another company, I think I would probably look at FlyIO. The nice part about FlyIO, sorry, I think I've talked about my words. The nice part about FlyIO is that you can multi-region deploy, which I think is something that any app that needs to scale needs. I don't know of anybody else that has ease of use for that one other than FlyIO.

 

Wow, did you just... Ali, did you also see that guy walking into Alan's room with a suitcase full of money? I think it said flyer

 

I wish I got money

 

on the

 

from

 

suitcase.

 

them. Again, maybe I'm wrong because I've just been deploying to Kubernetes for a while, but they're the only ones I know that can do it.

 

Well, let me now

 

And

 

do

 

it's in

 

something.

 

the easy way.

 

Right, right, yeah, now let me also do something because the Heroku guy walked in my room too, right?

 

Hahaha!

 

So Heroku has something called Edge as well. It's not as cool as Fly, right, or as snappy as Fly. But again, what comes with Heroku for me is the ease of management, right? And obviously, it's easy. Everything is like with the preface, like what Alan said, it's like, it depends. Generally, when I think of Elixir, at least at this point, building a new app, I rarely think of like a big company, like generally a startup or a small scale company,

 

Mm-hmm.

 

which

 

Mm-hmm.

 

is hoping to scale up to some extent more than Ruby on Rails, but like to get somewhere. Heroku took us so far in my previous startup, like we were scaling 50,000 new user accounts a month. And we were like, able to do it easily, just like by increasing the number of pods that were running. It's very easy. Easy to add a replica database, add a follower database for read replica and other purposes. Very simple to manage. And I think the load test we did was, I think, 24,000 or 2.4 million. I can't remember exactly. It's been a while that we were able to scale within a few minutes requests that we were able to respond to. So. I think what stood out was the effort it took for me to initially do the Heroku. That was for a very early stage startup. Once it's set up, the effort to get it to scale up to that point was negligible, just through the Heroku UI, right? And yeah, and like you said, Alan Edge, we did not get a chance to use that. But I know it works pretty good with Heroku as well though. I think it's maintained by Heroku itself or maybe it might be an add-on. Yeah, I'm a huge Heroku fan. Unless it's like your deployments are so complex that you need a container orchestration, and that's the way the Kubernetes would be a better alternative.

 

Yeah, I can only say we are still sitting with communities, but that is also like something inherited. So it didn't really make sense to move away from that if it's already set up, it works. And at that point there was not much reason to like kind of pull that out. If I were to be started, I would also probably consider something I can roll over fly or render. for my personal use case, I would probably just put something on my home lab, which is sitting on my desk. It's like a system deservice. But I'm actually wondering, because that was something we were struggling with at a previous place we were working at, that was kind of deliberately doing like a microservice architecture. That was a good choice, that stand on its own, right? And but there we actually had... issues with Heroku because that got pretty expensive pretty quickly because Heroku's even smallest pot I think, what are they called, I forgot, Dino, Dino right,

 

Yeah,

 

even

 

yep.

 

smallest Dino was, is not cheap. If you spin up like I don't know 15, 20 of those that adds up.

 

In my experience, at least it has been still cheaper than a full time DevOps engineer or you know if you're able to make lives easy for your actual application developers right like to develop the application layer side of things and minimize DevOps and platform. In a small startup, it is really worth it. Obviously, as you grow and you know. I don't think there's like a number, but generally I feel like, you know, there's like 20 engineers, 20 productive engineers, you know, you're making huge releases every other week. And you have like scale, which is like, you know, in in millions of, you know, requests a day. I think that's in my experiences when there's like an inflection point we should start thinking, you know, is it worth moving to like a more convenient architecture cheaper in that sense and also Easier to also easier from a security perspective, right? The RBAC stuff too. That's also, Heroku doesn't have a lot of rule-based access control features. So I think that's where it starts getting complex. And I don't think Fly has a lot of features there either, but correct me if I'm wrong, Alan.

 

Yeah, the role-based access, right? I know that you can add people, but I didn't actually see. Let me take a look and see what kind of actions you have. You can see who the last deployed or set secrets, right? But

 

Right.

 

I don't think you can actually say, OK, this guy can do what.

 

This

 

see

 

question

 

anything.

 

is born out of ignorance, but I guess there are probably other people out there which might ask the same thing. Back when I was working with Heroku, they're like source of truth, so to speak. That wasn't like a container-based deployment, it was something Heroku-based. Is that still the case or do they also support container-based deployments?

 

I only use container-based deployments.

 

Okay, fair enough.

 

Yeah, I know they have the app, what's called, like platformer service side of things, right? Like where they have one for Elixir too. It just doesn't give you that level of flexibility. Another thing is horizontal scaling doesn't work as well with some of them either, because they have to write adapters for those. There's

 

Yeah.

 

like

 

Mm-hmm.

 

weird adapters for each. But with containers, when Heroku starts supporting containers, I think 2020, right? Something

 

Okay,

 

like that.

 

fair enough.

 

That's when it... I think it was a clear choice for me at least just to move to containers. Even a simple, a very simple HTML only project, if I do something as a side project, I still use containers and keep the same kind of deployment.

 

for enough of that, thanks. Because I still remember us, like when we were moving away from Heroku and like getting some of the stuff squished into containers from like

 

Mm.

 

a big Heroku diner was actually, was not fun, let's say that. We had like

 

Mm.

 

this big, huge, like ridiculously large Ruby Monula, the two of them was the biggest and the most annoying co-op plays I've ever worked with. But getting that into a container, that was hard. Ha ha ha.

 

Yeah, I can imagine. I think one more thing I like about Heroku is just, again, like people said earlier, it's like a battle tested and tested with a lot of big loads and stuff. So I think with Heroku as well, I like that. It's not just tested in terms of what the app or the dynos themselves can handle, but also from a user perspective. It's evolved based on feedback from people. Primarily engineers don't care about DevOps, right? So it's evolved to make life easier for engineers like that. So I think that's another thing I like about it. But again, it doesn't mean I'm like, you know, not open for evaluating what's good. It just worked so well. I mean, I'm actually losing count of how many applications I have built, deployed, and are still running on Heroku in production. And that's, I think it says something.

 

And just for completeness sake, I think there's also to be sure that these mentions are an option. GigaLixia, which still exists. I've only played with them, I've never used them in anger and in production, but they are also there. I think probably from all the options we've mentioned, they're the only ones which support out of the box hotcode upgrades. Which if you actually want to go for that, then that's probably the way to go, unless you kind of want to double with managing things yourself.

 

Yeah, great call out.

 

Yeah, I have used Gigaluxer. I don't know about FlyIO, but Gigaluxer does support hot code upgrades, which is pretty cool. I don't think many of these things actually support that, but they for sure do, because I've actually used it. Just playing around with it, and it's worked pretty well. Scaling also works pretty well. But they're, of course, actually, they're run on, I think, AWS and GCP, and they have. Yeah, and I think you can actually connect to your own AWS stuff too. There's a lot of stuff you can do with it. I believe it's one guy named Jesse who's running everything. And he does a great job. Yeah, actually, that's funny too, because I haven't heard Gigaluxer in a while. I mean, I think it's still around, but nobody's been talking about it. I'm just curious if it's actually still around. Should be. Yeah.

 

I honestly am not surprised. I mean a lot of mindshare at least from my bubble I feel has been around Fly.io especially since now Chris McCord has working there. So Fly has definitely been pushing into the Elixir, into Elixir's sphere as like a hosting solution that also ships and like serves towards Elixir engineers. And Gigalixir I guess just doesn't have the same level of... Broadcasting power? Let's say that. Hahaha.

 

Yeah, I mean, they're pretty small team, right? I thought it was just Jesse himself, but I just went to the About page for GigaLixer. You see one, two, three, about five, six people here. People, most of them working at Google. So I guess it's probably how they met each other. Probably all work in Google. That's what it seems to be. Yeah.

 

So maybe beyond the immediately obvious, okay, you will deploy the things on there. Like how, let's also look at some of the related areas of engineering work. So like, for example, for you, I don't know how does usually a setup would then look like if you deploy your stuff to something like Flyer or are you then using something like GitHub actions for projects or how's the deployment pipeline looking there, including maybe CI integrations.

 

Yeah, I'm still using GitLab CI. And I usually

 

Hahaha.

 

just

 

Fair

 

write. Yeah.

 

enough.

 

I don't know why everybody goes crazy about GitHub actions. I feel so annoying because you always have to look for an action for everything. I mean, I suppose you could make your own, but I feel like I spent a lot of time trying to find the right action. With GitLab CI, I don't know. I just feel easy because it's like, OK, I know all the bash commands. I could just do it. I'm sure you could probably do the same thing with an action. But I don't know. I just find it more complicated. Adi, you kind of bit your lip for a second. Maybe I struck a nerve.

 

Hehehehehehe

 

No, no, not at all. I was a big GitLab CI guy. I think the big reason for me to move to GitHub Actions was just convenience. I do prefer GitHub over GitLab overall in terms of interface. And just having something right there was easy. And yeah, also from what I remember, 2019, I don't know if that's still the case. There's limited parallelism in GitLab CI, right? And I think I was starting to work in projects where CI was taking more than eight, nine minutes. And that really tests my patience. So GitHub actions, parallelism was another big reason. But yeah, sorry, continue. Explain how you use FlyCIO with GitLab.

 

Well, I tried using their container. It doesn't work in the CI. I made a complaint and they're like, Oh, that's a GitLab problem. So our container works fine.

 

Hahaha!

 

And then disclosed it. I wasn't very happy about that. But I understand they're probably busy or whatever else. So I tried for like a week, couldn't figure out how to make it work. So I just did what actually they have an article about how to do it. Basically, I just took an Ubuntu container and app get That way, and it just works. So I just, OK, fine. Then if it works, so here's a simple solution for a complex problem. Or simple problem, simple solution for a simple problem, I would say, for this one, actually. So that one just works. And then set the environment variables. This is my fly app, and this is my fly access token, and then it just works. So I'm pretty happy with it. The only other issue is like one of the guys I work with, um, he made his account later for GitLab. So when he does stuff, to say I won't run for him if you're using shared runners, because I guess people were, were taking advantage of GitLab before and running, uh, cryptocurrency stuff on there on the CI. So he's got to attach a credit card, but you didn't do so. Usually he does some work and I take a look and then I'll merge it in and my account will also work. So when I merge it in, then it gets deployed. So that's kind of the run around way we do. But usually for this kind of stuff, I just deploy a runner. Because usually we're using EKS. So just deploy a runner to EKS and then turn off the shared runners and it just works every time. So it's pretty straightforward, I think.

 

Ari, what about you? Do you just rely on like a road post, a built pipeline there or what is your approach?

 

Yeah, completely GitHub actions. So for the main production ones, depending on what the situation is, some people don't like CD. They just like image to be built and the last action to be manual by engineer so they can do an engineering QA as well. So depending on that, the image is built when you merge to main or push to main. the image also gets released. I think what I use, I don't think GitHub Actions has something, or at least didn't have something for Heroku when I was doing this. So I just used the regular Docker commands to push and publish image to Heroku. And then all you need is the Heroku, just one Heroku command from your terminal. Something that I did that I thought was cool was for. deploying your individual PRs to a QA or a staging site, was a simple solution was have staging and QA1, QA2, QA3 branches that correspond to sites. And with Heroku sites being dynamic now, you can be like, OK, I'll leave it on for a day and then stop. You can have those sites only run upon merge. And as soon as something gets merged, the site gets deployed and you can do a queue for a day or whatever time interval you want the site to be up for. So that was another cool thing with like a combination of Heroku, you know, an ability to bring a site down after a specific amount of time that Heroku has and using the same GitHub actions, the build and publish the production from main is the same one for like different branches that correspond to different environments. Yeah, one thing you need is like to reset your branches right every now and then because the conflicts will arise. So maybe that's something that was a problem. But yeah, depending on the use case, it could have a branch per Heroku app, or it could even have a manual trigger per PR that corresponds to a Heroku app. But I've done multiple of those. I think that's where it starts getting a little tricky. And I think with Kubernetes, it's so easy to tie it to a GitHub or a PR with AWS and just have a like a new site deployed with the same configuration. We haven't also used Terraforms with that part. I think I've always been curious what would it look like just to use Terraform per pull request to set up a site, if you need to. But that's in my to-do to try out someday.

 

Nice that actually does sound also fairly useful. I never got around to do something like feature, feature based page deployment, especially when it's involved non-trivial page management, which usually was the case because they need like, okay, how do you do it database? Do you do like a smaller copy

 

Exactly.

 

or it gets, gets

 

Yeah.

 

difficult fast for the,

 

Yeah.

 

we have, we have actually feature based deployments for our website. So that, that like place I'm looking at, but that is just connected into the state staging. infrastructure for

 

Right.

 

you.

 

I think seeds, having good seeds for your QA and staging sites is the key there. Then you can just release it to your database. But you're right, if you deploy a migration that changes a column in a feature branch

 

Yeah, exactly.

 

and you decide not to merge it to main and then try to deploy something else, it breaks. Exactly. The database state is a key thing that breaks. And so is any external services. So yeah.

 

I wonder if that is something, I mean, now we're getting into the speculation part, but where my understanding is that the SQLite support now also for Ecto is pretty decent at this point. So if you could actually set something up where you say if you have decent seeds, right, like just deploy the SQLite alongside and just say, hey, I'm just going to keep this on the machine because I'm not going to do anything with it anyway.

 

That's a very interesting idea, yeah.

 

But I have- what- this is just- I'm- I'm better at pops than my ads, so I have no freaking

 

Yeah.

 

clue. This is a good- good idea, so.

 

Right. I think the only thing would be obviously some fragments wouldn't work with Ecto, right? Surprisingly, every company I've worked for have done something weird enough with Jason that will not work in SQLite.

 

Yeah, probably.

 

But with that exception or similar exception, I think that's a very brilliant idea, actually.

 

It's then like, I mean, you would probably still want to have something like a staging environment where you actually do run the database migrations like for real, because you would have to recreate the database for the late scenario every time, which might work while the real migration might not, right? I think that's always kind of a gist of things, but yeah, that's just, that's just migration puckery. Okay, then any anything else beyond that? I mean, like now, now, what have we talked about? We have talked about getting that stuff built, talking about getting that stuff deployed in 2023. It is I nearly wanted to say 2022. And now what's the other thing getting that stuff monitored? So what's the state of monitoring maybe in 2023? What is what what are you doing using Ali?

 

I guess I've been using AppSignal. It's worked really well for basic monitoring and tracing stuff and for individual logs. Oh my god, it's all over the place. There's log DNA that I really enjoyed. That I think it's query language is very simple and ability to define a subset of logs based on a few expressions was very, very cool.

 

Yeah.

 

I guess Splunk is nice too, right? But that's expensive. And also, what is it, like BetterStack? Have you guys heard of the BetterStack? They have something called the BetterStack. I was doing some research. But I'm trying to find their logging solution. But anyway, that's another one that I, Logtail was the one that's part of the better stack. That was pretty good too. But LogDNA was most convenient. And for individual errors and monitoring and tracing, AppSignal was pretty good. It also has overlying support. If you set up, I think, just OTP app name environment variable and make sure it gets sent to AppSignal, you'll have an overlying trace as well, like processes and how many processes get spawned and memory usage of

 

Oh,

 

each

 

interesting.

 

process. Yeah, so it's pretty cool. And I think some like PagerDuty to make sure. Make sure you set your triggers in SLA and AppSignal that trigger like an on-call routine. Although, I've been wanting to use on-call built by, man, who was it built by? My GitHub stars are going Grafana, right? Grafana built on-call. I've been wanting to use that. I've heard it's a lot better than PagerDuty's interface where it's like, oh, it's a little bit It's almost impossible to set up rotations between people. But on call, can you see your Google Calendar and other to sync the rotations? And it's easy to manage, basically. It's a bit of a need to use that for on call shifts. But I would love to hear what Alan uses.

 

log DNA. Usually I just hook it in because they have a runner for what do you call it the collector for EKS or for Kubernetes in general. They also have some kind of Kubernetes integration thing before, but I haven't looked at in a while. Otherwise, Sentry works pretty well. I mean, you can just hook it in. I think there's some integration already. Yeah, those are the two things I use. I think I tried some other stuff, but those are the two I usually go for. Yeah, I've heard PagerDuty and all those other ones. I think we've had somebody on here that worked at one of those companies, right? And they there seems pretty good. So yeah, I mean, to be honest, they're all about the same. All you need to know is when an error happens, I think you can hook into like the sassle or something and get a bunch of information. So this in general, but you need something. Since Sasha was like, oh, what are you using on flyo? Actually, nothing. That's probably something we need to do next. We're just doing the old try and true method. Keep the browser window open at the log lines and look for errors. That's what we've been doing for the meantime. But we don't have this one deployed to production yet. So. We got some time.

 

Fair enough, it's a form of monitoring as well. The

 

Yeah.

 

same is like, angry customers calling you is also a form of monitoring.

 

Well, the good thing is there I know I'm in Hong Kong and they're in like Eastern time in US. So it's like we're on opposite times of the world. So they can call as much as they like. My phone's in sleep mode, so it's OK. I'll get to it later. But anyway, we're not in production yet. So that's definitely something that we're going to be looking at. I think Century is probably a good option.

 

One thing about AppSignal that reminds me is it has periodic checks you can set up based on context. You can add context to it and you can generate a status page using that and host it on a subdomain under your domain which gives you a very easy status page which is always cool.

 

That sounds useful. Yeah. I mean, the funniest thing is always are these status pages, which are kind of. Hosted on like the same infrastructure as the rest. And then when something goes down, it's actually a status pages down as well. Great. And maybe to throw in my perspective there, um, what we've been doing with like this new backend project with within a new code base, we are went full in on like open telemetry, like 100% by and there. And. The moment everything gets shipped to Datadog, just by the nature of us having a Datadog contract before with the previous stuff we were running. But the nice thing is that we, with OpenTelemetry setup, we can also, when we did that, try out some other platforms. It's still all very, very early. So I wouldn't even say that I myself have like a direct preference, but I would definitely, if I would be building an Elixir project nowadays, I would always go with OpenTelemetry. You have a little bit of... extra effort because you need those telemetry collectors, which they pull in your stuff and then they send it somewhere. So that is the thing you need to take care of as well. But honestly, the flexibility it gives you to then ship it to Datadog, to Honeycomb, or probably also something like Splunk or AppSignal. Man, I wouldn't want to miss it. I really wouldn't want to miss it. It's just so convenient. But you don't get that level of log in from these providers. So that's what I've been doing basically saying, do open telemetry and then figure out which platform you like best. That is my approach at the moment.

 

I like it a lot. I like that approach a lot. Yeah, totally a great open telemetry. I should pretty much always use it with Elixir. Once you use it, the flexibility it gives you to even move from one platform to another and download the actual contents. And I think what AppSignal has is importers now, right? It allows you to import old open telemetry logs, too. So yeah, it's pretty cool.

 

Yeah, it's also an open telemetry, I feel really is at a point where they have enough mind share or platforms like Datadog, AppSignal, Splunkable, the stream as well, so on and so forth, where they can't ignore it. You know? So it's actually, it's convenient. I'm really fond of it. It works pretty well. Okay, now we have application built, we have it deployed, we have it monitored. Anything else in the context of building and running and managing Lixi applications in 2020-2023?

 

How do you guys usually handle promoting your instances to production? I mean, for me, what I usually do is we always branch and do feature branches and then check out the work. And if it looks good, we merge it in and we have a staging or pre-production environment. And then once the client says, OK, this looks good, then we merge it over to production. And everything's running the CI. Do you guys have the same style?

 

I guess it depends. You do a lot of client work, right? So you need their feedback. I guess you can replace them with in a bigger company, like a QA kind of a team or a product team, right?

 

Yeah.

 

But yeah, I think from a branch perspective, I've worked with branches that merge, deploy main branch to staging and then promote to production, like you said. But also, they have a separate staging branch, which follows main on a weekly basis gets reset, that you merge two first, and then you get staging separately. And then when you merge to main, or develop, or depending on what you do, it's staged for the next release to be set. And you have a release branch that corresponds to the next production release. That's more like a Git flow-ish approach. But yeah, I like keeping it simple until the team grows. Right, I think it was too complex for a team of like five or less It just adds unnecessary complexity which I've seen people like over gonna go out for a good flow teams of like three four engineers I'm like why?

 

To be honest, I'm not even sure you need Gitflow nowadays anymore. Like it solved the problem back then. And I was a fan of it, but I'm now a big fan of just... If I were to decide freely, like we do it slightly different at my work, but that is also for historical reasons. But I'm a big fan of deploying on merge on main to like something like a staging system and then promoting one of these releases to production. I feel that it just brings a lot of... kind of sane guarantees around it, especially when you, when you really do that promotion of like the same image, basically this was deployed on staging,

 

Right.

 

it worked, the migrations ran, and now we take that same image and we deploy it to production. It's just a very, it just feels very nice and safe also. And we've never had any issues with that, that we actually, we make migrations again, right? But because data and production is different staging. That is like a different story for like maybe you want to synchronize from the database from production to stadium and anonymizing

 

Yeah.

 

But in general, it's just very sane approach and then you would only have very short lived feature branches The question of how you went then might want to do feature based feature branch based deployment That's a different one. You kind of have to tackle that on your

 

Right.

 

own The

 

Yeah.

 

one thing I've never got around to do which I would have loved would it be left to do all the time is like really proper blue green deployments where you really get that thing deployed. It really doesn't run some, some sanity checks on some smoke tests. And only then it kind of really goes fully live or kind of always the poor man's version of like a health check and pointing this case, Kubernetes, but they never, they get kept spinning and just failing. And the number one realization I had on that front is that like your whole deployment and CI pipeline, you kind of want to optimize for failure. want to optimize for the bad path, basically for the sad path. That, that, that every scenario there should actually end up in something concrete, in a concrete action step should post somewhere. Like there should be something happening and the happy path is more of a by-product of that. Yeah. Sorry, I kind of went around with that, but that's my perspective on this nowadays, of like how I would want to do the deployment pipeline and how I would want to deploy to staging of the production. I wouldn't want to do anything. Like staging a production from my perspective, I find I've not worked in a scenario where you really want like a third environment. I've not yet seen that. This

 

what we've always had

 

is necessary.

 

was we had like a dev and then we had a QA and then we had production. So our dev was kind of like the staging to a certain extent. And we would you know, this environment that we could mess around with and we would just kind of test stuff. But that was Yeah, I don't know, I felt a little bit weird to have a dev environment because I was like, well, why wasn't our local machine a dev environment? But to be honest, I mean, you need something that because we're I was working in the bank at the time, right. And the weird part about working in the bank is that we're developing on Windows. but we're deploying to Linux.

 

Hahaha

 

And it was like always weird, right? And I think that's why you actually needed that environment because you need one step before pre-prod or one step before prod, and one step before pre-prod because like you don't want to deploy this and say, okay, business, check this out. And like you forgot to replace the slashes correctly because that was actually one thing we had. We actually had really VMs that we were running on and we actually had like a code translation. If Windows do this, if... If Linux do this, something like that or so, like that was a painful part.

 

I actually want to also re-deck my statements, because I could think of an example. If your staging environment is actually used by someone else in the backend engineering team. So if there's maybe for a mobile app, you have like point to staging, you wanna keep be able to test that or anything like that. But anything else which also is, has a dependency on that system and like doesn't want it to be braked to their own tests. At that point, you might want something like a third environment where it's okay to break stuff.

 

That was basically a dev to a certain extent. Yeah.

 

Yeah, yeah, because I mean, at that point is also, yeah, locally, you can have death depending on the complexity of your system. It's just not reasonable to have your full system running on a machine. Um, especially when it's like more of a distributed nature where you actually might want to test out the, the, the interconnected parts and their full deployment.

 

Here's a good question too. Do you always call it staging or no? Because I always call it staging, but then I'm working with a French guy who likes to call it pre-prod.

 

I always call it staging. But, eh, I'll do what about you.

 

Yeah, I guess staging, I've called it dev like you were doing Alan. There is, yeah, yeah, staging is I think the most appropriate name.

 

Yeah, well, pre-prod also makes sense, to be honest. Staging is kind of unclear to a certain extent. I guess you're staging that to deploy to production, but not really, because you know what I'm saying? It is kind of a weird name, although I do prefer that name. Pre-prod does make more sense to me, because before you go to production, this is what you use. So I find staging the name itself is a little bit strange, but I do like it more than pre-prod. Maybe because staging and pre-prod, I don't know, I find pre-prod more hard to say. that makes sense.

 

It shouldn't be staged, right?

 

staged.

 

Right, this production.

 

Is this like a pun on Git or something?

 

No, no. I'm just saying it's like, I guess, like staging is like

 

Yeah.

 

a verb, right? And production is a noun. So it should be like, there's this production,

 

Yeah.

 

and this is staged. I don't know.

 

Yeah,

 

That's just,

 

could be.

 

yeah.

 

language. It's weird.

 

Well, I never thought about it until Alan mentioned it and I'm like, Hmm, what's the better name for staging? But yeah.

 

I don't know, but I think I'm drunk looking at Sasha right now.

 

Yeah, my camera was just my little focus. I looked very blurry. Let's do pigs. Is there anything else you would like to add before we do pigs? No? No? Okay. Then, Adi.

 

All right, so I got a few picks today. So one is a video game. Well, there's two video games. The one everyone knows, which is Hogwarts Legacy. And there's some controversy around it. But it looks epic. And any Harry Potter fan, you should definitely try it. There is controversy around it. I get it. But oh, man, the game looks epic. I'm super excited for that. There is another game that I played recently. It's called Vampire. I don't know if it's a PS exclusive, but I think it came out in 2018, which its graphics aren't like great, but its gameplay is quite interesting. So yeah, if you guys are interested in a dark vampire like game, we should give that a try. And I have a couple really awesome people who are looking for jobs. They're not engineers, but they are two of the best people I have ever worked with. One is a customer success expert. So like she has like 10 plus years of startup experience and very effective and entertaining at communication and making something boring sound super interesting. She's like a marketing genius. And yeah, a jack of all trades on top of it. So her name is Delaney, and I will add her contact in the show notes. She is looking for a job. She recently got laid off, so just helping out here. Another one of my friends is really, really great, is an operations expert. And she has a history of building and scaling startups from zero to one successfully multiple times. Again, 10 plus years of experience. great at many things, jack of all and master of some, which is the case in both of these people. Her name is Olivia. And I will again add her contact details in the show notes. Any company would be extremely lucky to have either of these people.

 

Nice, I love how you consistently do that, that props to you seriously. Evan, what are your picks for this week?

 

Yeah, for me, I just have one pick. I was just playing a game recently called Bright Memory Infinite. You guys heard of this one before. It's actually, they say it only takes you about two hours to complete the game, but the game looks really great. I was quite happy with it. It's a really quick game made by just a couple of Chinese developers. I was quite surprised. This is the second version, the second game in this, so it's called Bright Memory. The name is a little bit weird. Yeah, Bright Memory. Its name is really weird, but to be honest, the game is pretty decent. It looks really good. I'm playing on my Steam Deck. The only issue is sometimes it freezes. Maybe this is a Steam Deck Linux issue, not an issue with the game if you're playing on Windows. But yeah, it's kind of cool. Like you have powers where you can push people and make them explode and stuff, and a sword and guns. And yeah, it's been a lot of fun. So this is the game I'm playing recently, and I think it's something people should check out. It's on sale right now for this. I think it's just a few dollars US. I don't know. My store is in Hong Kong dollar. So it's like 27 Hong Kong dollars. It's like, I don't know, it's like three bucks, I think. $3, $4.

 

Question is if it's still on sale when this episode releases.

 

Probably not. But I think in retail, it's like 20 US dollar. It's not too bad, but it is a little bit short. So I think it's better if you buy it when it's on when it's on sale, because it's quite short game.

 

I'm also gonna pick a game because I have nothing, no interesting reading at the moment happening, but I want to pick a game. I just recently tried out kind of by accident. It got gifted to me around Christmas from an online acquaintance. And then we were wondering what to play one evening with some folks and we were like, hey, I still have this game. Someone gifted it to me. Let's try it out. And it's Stick Fight, the game. And it's... It's exactly what you would expect. It's like the over the top crazy games about stick figures fighting. It's nothing I would consider like serious, I don't know, combat, kind of like a fighting game. Although it has a surprising level of depth to it, but it just delivers on this, on this premise of like stick figures fighting so well. I'm not sure if you've been around for like the early days of YouTube. There were these videos of like. animated stick figures fighting epically. There's also like one finger death punch and so on like we stick figure fighting games and it kind of delivers on that in like a multiplayer fashion. And the neat thing about that is also it's the rounds are insanely quick and when you actually finish a round it just drops you immediately into the next map and then into the next map, into the next map. So just because keep playing one map after the other like this crazy stick fighting game sense. So it's a lot of fun with friends and voice chat. because there's a lot of screaming like, AHHHHH, WHAT ARE YOU DOING NOW, NICE, DON'T DO THAT, OH NO, KEEP THAT BUCK, MMMM and so on and so forth, so it was a lot of fun. And it's also, seriously, it's under five bucks. It's regularly on sale and sometimes even below two bucks, so... Even if you only play for one night with some friends, it's totally worth it, it was a lot of fun. So stick by the game. Okay, folks. It was a pleasure, as always, talking with you about this. And then, Adi, I expect next time that you pick a vegan pizza terraform adapter. Yeah. Then we're going to

 

Nice.

 

order some pizza. So we can eat pizza while recording a podcast. All right. Then thanks for listening and tune in next time when we have another episode of Elixir Mix. Bye.