Allen and Sascha join this week's panelist episode to talk about updating web application dependencies. They dive into the things you should consider in web application updates. Moreover, they share their professional experience and how they deal with errors and bugs after updating software.

What came into my mind is like that you build an application right and you deploy it somewhere and it actually delivers business. You. It does what's supposed to, and then it keeps running for months and it's fine And maybe you have like a little book here and there You fix it, But it happens, runs for two months. It runs for four months. It runs for six months, and at some point you like, I should probably update this Ight, I should probably check what kind of dependencies have been outdated Now. Maybe there's a new elixir version, O P. Version Should date this And a lot of resources at the talk about Okay, like this is how you built something in Alexia, but don't not resource to talk about this is how you keep things running Lix you. This is how you kind of tackle the boring, boring stuff of keeping things running up to date. So Ellen, how do you usually tackle this? What do you do when thing has been running for a while And you think maybe you should updatethis?

 

Yeah, just just kind of raally pops into my head that you know maybe I should take a look because nobody else in the team usually takes a look at that. They just kind of used to just bit, just runs, and luckily with a lixerusually works okay for some time and just you know, take a look an see what's the latest version and hings like that usually stay up to date about latest version of Elixir and laying, but for each project I run a d. F. So just kind of keep each one and each one separate

 

Hm

 

right Because we do have some issues, Were Versions of some library are not updated yet, M. Yeah, and then also, yeah, I just run. He outdated. See how it's going? I try to do that. I don't know. Maybe once every couple of months I don't know about you like. Do you guys have like a set schedule or a job or something that runs to see this kind of stuff?

 

You what we've been introducing for while I a process what recorded review day? And that is just a regularly scheduled thing in our calendars. I think it's currently at a one month cadence. We might have reduced the two weeks now. Not entirely sure. something like that where we basically all all the bacon engineers sit together for a day and just we have a check list. We go. Though that check list includes Okay, Let's also look at cloud costs, right as something may be noticing like an increase in a certain area, but it also includes dependency. Look at our dependency. check. what's out date. Check what that means. Is this the an upgrade we've got to do? We also have depend about integrated as a thing which is open in port. because automatically there's still a bit of discussion if you want to automat automat those or not. I. I'm not entirely sure because automaton can be working if you have a decent test set up, which also test integration level. I say, maybe I have, for example La installed, which is used for I request to an external system. You better have an integration test for that kind of thing if you actually want to auto updateit. right, So those kind of things, but in general we have this review day. We call it what. we go through the checkers and do the maintenance of things. And it also includes looking at documentation. for example, write documents. documentation still up to date. Is there something we need to change their anything outdated house? Our observability set up. Is there something we've been missing the past few weeks? That kind of thing? All those there poring questions you got to ask. At some point that has been working fairly well, but just literally just yesterday we had little incident wasn't very big because like it was part of a system that relatively new, hasn't that much exposure to you to users. But basically what happened as we uptated a bunch of dependencies. Also among them Tesla. Much There Tests were all fine deployed there to staging was running fine, deployed with production. And then it turns out this particular up date to test actually broke our integration with Google. pops up in very, in a very subtle way, In that Publishing, things still worked perfectly fine. Consuming things also still perfectly fine, but like processing the response of the publishing that broke, so our system was assuming. Hey, this publishing here over there didn't work. so let me try again at the customer in Pick, at that point where some people got a whole lot of email or a few days, because the system kept kept trying Like Okay, This didn't publishing. didn't work like, let me publish again. Let me publish again. I like. I don't know. Like forty. It's a day or somthglike that it's not end of a day right, but it's kind of annoying and it actually bore down to one line change they did in the the decompression middle Whare, because previously that didn't set any any accepting coding had S. and then the new version, the the acceptngcoding had the decompression layer. Actually turns out that whatever Google Pup sub sense of response that we haven't figured out what exactly happened, but we send response apparently can't be handled by that particular decome Passion layer of test, even though says like Hey, I accept Jesup, and deflayed one of those two right. whatever cup returns can't be decoded like that. I'm not entirely sure why we haven't dug into that. It's tack it on our back like a okay. Let's figure out what exactly happened there for now, because the traffic on that particular part of a system is very low. Just okay, let's turn the compression of for now and then It works. since it actually turned out that before the compression was also not working, The compression also wasn't being done because That hero wasn't there. So yeah, but but exactly kind of thing that can happen right like you have you up to the dependency. It does a little subtle change, not even worth mentioning all in all, but because of an integration. Because of integration in like a thing which communicates with another, another part of a system or some external thing, it breaks in unexpected ways, and that is a hard thing to solve for, I mean, like a now down we solved the immediate issue. We captured some some tickets. About Okay, we need to figure out how we can avoid this happening again, and also how we can cap. capture this earlier. But at that point it's also a unites wouldn't have caught that. You know, I don't see as writing a unites for that partcicar thing. This was actually like the combination of this, her being there, and then Google ups up sending something response, which tester in that particular thing didn't expect, So we would have to have an integration level test, or like an enter in test. Kind of what we say. K. This is actually going through a whole. See bang and publishing something ups up. maybe a test topic, whatever you know, but actually going through at and then you can kind of kind of make sure that this is working. I think that's always the old struggle about. In the perfect world You would have that set up and now we actually have to use. So we also can pointed and say, maybe we should invest a bit more time. there very much assume versus a topic for one of our architecture fixes, where we just talk about bigger picture things like this. Um, but yeah, I don't have already made answer for that for those kind of things that can always happen.

 

Yeah, then it's kind of like. Well, I mean, if you start testing everything, then how do you actually get worked on? You know what I mean, Like

 

Yeah,

 

you only have so much time and that kind of accident is really a freak accident, freak accident. I mean a freak bug or something where nobody was expecting this kind of stuff.

 

Yeah, I'm also, I'm not entirely sure. Like how to best handle this is his. like. Is this something you want to capture in an integration lever test? Were, just say hey, we have this one thing which integrating with an external system. Let's assume that it's running a fine, at least like a few examples. probably, but you might also, and that is like that is the point were I'm not certain how to best tec. That for example application is running a community cluster right in the, That has ready probes and healthy probes, So we could maybe also have For ready in point. Actually, say hey, this is publishing. for example, something to Google Pups to again to attest topic, Just making sure that this publishing just work. so it doesn't accept traffic before it actually kind of verified that we can do that. I'm not sure. Honestly not

 

Yeah, I've thought about this too, because I have like a micro service architecture for one project that were slowly kind of bringing into all into Elixer, and I'm thinking like, because I don't know how to explain, but I mean something is travelling through out the system and it'd be good to kind of trace where things go bad. Like at which step right, And that's probably a really good case for something like telemetry. Maybe where you can kind of tag it and then figure out you know, and I guess you'd probably have to suck that into. What is that one called? There's M yager or something like that? Sure how you would do this kind of stuff?

 

Connect you you. How you would do this kind of stuff Because this is exactly what we're doing. So

 

Yeah,

 

we

 

please.

 

are integrating open telemetry and we have an open telemetry collector running in our cluster, and that, collecting the things from from the Ms. from our services of particular service, and publishing to data, dog, data dog, as supports for locks for metrics and for traces. In this case, what you've been thinking of is traces right where you kind of can see like each individual step in between. how lo That quarter tache locks, For that that is actually pretty made like that, prolucs very well with Datadog and we also have a bunch of alert set up there especially for business critical things. But in this particular case, like as email being being triggered through this event, that was that was not like. not like a. It's not like a thing that was kind of caught by by an alert because we have. basically we have alert set up that these kind o e mails of a state get people that use us alive in this state. Because if nobody arrived in this state for like a longer period of time, then something is wrong. We have alert for that, but we don't have alert for you always now said I don't know if it emailsbutofthe. same us, Because nobody thought of that before that that could kind of be the outcome of things. So I feel it's always the age old struggle of you. Try to predict what could go wrong, but you can possibly predict what what possibly can go wrong. So s always going to be that one thing which is kind of caught you blind side, and in the best possible case, you at least call catch it by your own obobservability set up, and not like in our case, Because use a rod to support He. I got first forty times.

 

I mean, you should be useless because you have kids right, So you just get surprised by a lot of things that they do know.

 

Yeah, yeah, yeah,

 

You're just like. Oh, I never would have predicted they'd be doing that or whatever. Now?

 

Yeah, yeah, that is. that is definitely true. I mean the number of unknown states on trivial system, like just the ones we are building nowadays, especially how we integrate in a bunch of other systems. The number of possible states is so ridiculously large, and most of them are unknown states and most of them are probably broken in weird ways, and you can't possibly predict all of them, And in this case it's also like learning would be taking away from that Is also okay. How often should We re try things like this Because like I said, it's a very relatively new part of a system, so not all ites have been roughed out, but probably should not. We try this indefinitely for four days. You know. At some point it should kind of go into datleticue. Assume, and some should should be telling you, Hey, this is broken. Look at this and we, we have kind of similar set ups and other places of a system where we consume events. And if like that consumption Fails, then after a few times it goes into the A. But this particular thing was not trgebysechan. Events like that mechanism didn't catch, Didn't work out. But yeah and all because we up there, the dependency Nd like there were some changes in there, but like it was one line being eddied in a decompression met ware. That's I'm not even sure we could have caught that in this process we have set up right now because even if we looked at the difwhchin, this case, we didn't but look at the different detail. Okay, the cool right. I don't think. what. Wait, if they didn't the head. Now, what did they do before when they're using that? our Google pops up. Integrate some one and so forth like I don't think they would of cross my mind. And then again, I kind of have to figure out kick. How can you can you make sure that this integration. these changes don't break any integrations in unexpected ways? So how does it? How is it for you? An how do you notice when stuff breaks? And especially how do you notice when stuff breaks? Potentially after updating dependencies.

 

Yeah, that's a. that's a good question. I mean we just hook up to. I mean, I guess it depends on what kind of breaks you're talking about. Right things can kind of break. but you don't even know. Like maybe a number doesn't get rounded up properly Right. That could be a potential bog where you're Ot. gonna get an error Necessarily For sure. we have errortracking. We use Log Dan, we look for log statements coming in, then tacked with error warning, and then we blast those out with an email or notification Is slack. Looking to the law, We can see a context you know. Like what's the context? Like what else was happening before that? Get kind of an idea like. for instance, there was something that changed in absence where they actually start requiring a specific data type or something. I can't remember what it was. But something got more strict with absinthe have to, and I noticed an error in there and like we had to somehow handle that one because like we tested everything we were running, flutter around the whole apporo. everything worked fine. But one of the previous versions of the Ap didn't have the proper date type on it and it would crap out and we found it out in production. Basically so we just kind of released like a hot fix and just told people. Luckily, the wasn't that popular at the time, Like

 

Yeah,

 

super popular now, but that was like a short fix.

 

Yeah, that can happen. I mean, it's especially bad when it happens in an Ap, where the up date it's not as easy to push out a quick up date. Fix things right, because in the

 

Yeah,

 

Ourcenario, like I said, we just as able to the middle weapon now because every reasonably certain that it didn't deep compress before that anyway, and it doesn't serve that much traffic, so you're like Okay. It's just able to a ticket. Let's figure out later out to fix this proper. That's That was a thing of like half an hour right now Fixed. Not as easier than at

 

So you depatched the library and then sent up a P. R. Or how did you kind

 

You.

 

of make yourself

 

That

 

it?

 

is where the topic now we have to figure out like we wanted to proper a ticket on the back lock, which is gonna get pulled. Where we look at what exactly and wrong there? What? Why He couldn't this compressed layer not de compress the response from Gogpupswts. What's kind of the issue there? and then look at that potential. Depending what we learn, there, open a pre to test to fix it. but immediate problem was fixed right now. like I said, Since that's not big Traffic thing, there was no harm in his airing the compression.

 

Yeah,

 

but

 

then

 

that's

 

the other

 

not

 

question

 

always

 

too

 

possible.

 

is or is this an issue on Google side?

 

Maybe maybe I don't know. Like in a scenario like that, I like you have an active incident. The first thing you try to go for fixing the right, and we have a reasonably reasonable understanding of what exactly went wrong, but not the details of mechanisms and how it particularly broke. That is, I think, Find that is also professional professional ofengining to a certain degree, where you you have to choose your battles. I mean, at the end of the day, that Thing that needs to keep running and the thing needs to continue to serve business value And do we want compression at some point again? Yes, definitely, but it's not going to hurt us now, so fix the thing now, capture in a ticket, write proper post mortem. Figure out how we can catch this earlier next time or a customer of writing us. Why Why you receive the thirty times and then continue? I'm not sure you can do anything more than that.

 

Kind of. I'm just thinking like you know if it wasn't issue and govleside at, it's possible, right because everybody

 

It's possitiveyah.

 

makes issues. For sure. I've discovered an issue with is because we're talking. You know about flutter. Little bit before the show for Flutter. There was actual issue with is opening up like Well, it's Cal, quick links or heart links or something like that, Where you can you go to a specific link in the in the browser and it's goin to open up the app for you and go to specific page. I forgot the name for that

 

He

 

deep

 

blinks,

 

linking deep links. There was a Deep Lake bug, like between patch release versions of Is its crazy and we had to put a patron because my client was like you guys. You don't know what you're doing. It's like, No, like, Like I was on the latest patch release. He was on the path release in the middle, That was not working. and then another developer over here was on the path release before that one. So it was like he had. like my developer had two. I had four and he had three and it worked for me and the other guy and it didn't work for him. And it's always Well, it doesn't work for him. Maybe he's doing something wrong. you know, usually one of the first thoughts you have, but in general like, I think, I ended up having wanted over here somewhere or something. Somehow I figured out that there was actual an issue. I think I actually just asked him to record the screen Is probably what I did And then I saw

 

Hm,

 

the bug and I was like that's that's weird, and then I went looking and it's like other people are reporting the same thing, so even things could be totally outside of your control. And

 

Yeah,

 

that's also depend to see upgrade. Technically, Kind of.

 

Yeah,

 

I guess

 

Yeah, yeah, it is. I mean, especially when when you are integrating with an external service, which, honestly, nowadays we all kind of to a certain degree, even even if you're running. I don't know, Even if you're running your software on a machine at home, I don't assume you wrote the operating system rights. At some point you're integrating with somebody someone else did, and at some point you're probably going to want to upgrade that, especially if it's exposed to the internet. So Yeah, at some point you're going to have to trust somebody to fuck up because you can't possibly order everything. I don't think that, but that's that's humanly possible. And then you better have safeguards in place. And like, I mean, like I said for us, it's another question. Okay, We actually now have have a case. so he said we upguedependency. The tests were still running. Find over units Ave, pretty good test covers for that, but we don't have a lot of like testing integrations of externasist, And now like, how can we make sure that something like this doesn't happen again? Or at least we catch it earlier. The best possible case. I would assume that we could. I think we could have like a like a set up where you probably with review day right like you, go to a dependency. Is you up date and then deployed to staging and on stage, And you maybe have somethin to say. Okay, This is actually running for some additional and to end kind of test scenarios and going to alert you when something breaks. That is, probably we have to In four. And but yeah, it's It's difficult to find the right balance between pragmatism getting things out of the door and, and making sure that things don't break unexpectedly, and at least throughout my career, it always ended up being. There's a certain base level you can always go for, but everything beyond that you need good good arguments. You need to say. this is something we shud Doing because hey look at this. it. actually, it actually broke in this way, so we better capture it and a lot of a lot of companies. Are there more reactive on that? A while ago we had, We had the Michal lovers right like well from Porliaxial, which is about security. And there you kind of maybe make the argument. A security is maybe something you won't want to be a bit more pro active instead of reacted. But for like, I just like that, for for bucks you're kind of forced to always be reactive. You can can apply experience and you can apply best practices, but that's always going to be something which is catching you off guard. And unless you have an s. O. A, it's life. Have you ever built anything with an S. A Allen? Where were? you kind of

 

I was

 

had

 

in

 

to

 

a company

 

look it

 

that

 

up

 

had

 

time.

 

he had an So, but I wasn't part of that product, so I did see them scrambling. It was quite interesting to see because the whole office was abasically support the real developers ractury in New York, and there's support over here in Hong Kong, and for some reason they had one developer, which was me here. So like every time when there was a bug or an error happening all hands on deck, all these messages going out over email and I'm just like thumbs like this, just kind of enjoying and eating pop Corn and watching everybody run around like chicken with the head cut off. And just

 

Uh,

 

and there's not much I could do

 

uh,

 

right, because like the training, I didn't even get the training to even do that because they're like, Oh, you don't need this because you're a developer and I was like, All right, cool. not a problem. And yeah, I was. It's interesting to see. And and actually, the the We kind of talk about that reactive right.

 

Yeah,

 

I do remember one time like I don't know why they did this Mean they're making. They're getting like tens of millions of dollars from one client and they just had to cheap out and just get like one data base server and run Mango B on it before we had the set compliency on it, and they would just lose data all the time Like,

 

But Mogmongaibis Webscaleellan,

 

Yeah, Well, it wasn't financial scale I. this, so we're losing financial data. So

 

What can possibly go wrong there? right on't? see an issue.

 

yeah, exactly.

 

Boy.

 

I'm guessing they could probably really do it. I don't know, but I don't want to say exactly who it is because it could get me in trouble, but in any case like that kind of stuff is definitely preventative right like every, like all the d B as know that we were not ready. but it was like you know when you would peal for more money to solve the problem, you were just care like, Oh no, you know you can make it happen. And like you know what do you do right, they would and they're just beating down there. Just okay, fine, then you know, let the boss

 

Yeah,

 

handle it. when that we don't meet the S. A. I just do the best I can

 

I actually was once in a not quite that similar situation, but something similar where I also, I have to be very vague because I don't want to step on anybody's toes, but basically we had a customer project going on in. The customer wanted to do something, which it the very least was like a legal gray area, you know, and

 

And at

 

an

 

the

 

out.

 

very most was high treason. You're trying to say.

 

No, not like that, but it was knowingly ignoring some things which you probably shouldn't be ignoring Like that's that and bow down to me saying Hey, I'm not going to build this. You can ask somebody else to build this. We can, you kin put it production, but I want to have it on record. I want to have an email. A kind of point to say, Later on, I was not involved in this and I deliberately spoke up and said like Hey, this is. this is not what I would. What what I want to see what I want people to be doing And that is also what happened. Like. basically got got to cut like some. something in written. something in written form. In this case was just email right like my boss was like. Yes, Okay, duly noted. Never came anything out of it. It was, wasn't really a big. Big thing, was something small, but still At some point where just like now mentioned, if be re not ready. At some point it boils down where like we, as professions will sometimes have to say Hey, I don't agree with this. I want to keep. have like noted down. I want to have, like, maybe even a written record of that because if if things then, if the hammer comes down, so to speak, then I at least want to be able to say hey man, I want you don't put this on me. And maybe it doesn't. maybe it doesn't.

 

Now, I'm kind of curious, right because I think this is actually a discussion I had with quite a few developers before Where it's like Okay if somebody asked you do something that is that you know is not good. You know you should say no, Right

 

Yeah, Yeah,

 

and I always get the reply from body like, Oh, but like, But this is your job and your boss asked you to do it, or the client asked you to do it. It's like Well, you know you're kind of the gate keeper. to a certain extent.

 

Yes, I think I, too. To make an extreme example is ould. you write softer for rocket guide and software for like miscles right, like being shot on in war areas. I think a lot of people would say no to that. I wouldn't do that. but if I, unless you say yes, I don't care, Would write anything as long as I get my money, Then okay, cool. You always say yes and never say no. But if you say no to, would you write missal guidance? Softer then well, whether as the line, what would you do Right? And that is kind of what. it's down to say. Okay, Where is the point where you say? Now? the example said I supwasnot, Not really like a. Like a big thing. I was like a data privacy where we kind of are obliged to offer something here. but we can probably get away like a use. thought Bly could probably get away with not doing it right away. you know, So that was kind of the situation

 

Yeah,

 

to be in.

 

that reminds you of her client. Where talk bout g. D. P. And there is like a very long weight period where you can just not do

 

Yeah,

 

something

 

yeah, so like it's yeah. but in this case like I said, I said like I don't want to any part of that.

 

Well. I mean, I think it depends on what it is that you're going to do right. Like for that one, I think he was like. Yeah, let's delay on that one because we had to first come up with a plan about how we want to handle it right Like that's fine. There's within reason right. But if it's like now, let's just be shady and like pretend we delated this person, but we know who they are because I think you can like dele their personal information, but still keep their data or something. So it's like let's you know, let's not just put r X over here and then let's have a transcription layer of M. X Is who exactly Data base right, Like this kind of stuff,

 

Yeah,

 

But but speaking about like shady things right, sorry, but I mean you're German. I think this happened with a German car company quite a few years ago where they faked out the emissions right,

 

Yeah, yeah,

 

and I think it was the engineers that got most of the blame even though management asked them to do it. Isn't

 

Ah,

 

that

 

that

 

correct?

 

is bolted.

 

That's

 

I'm

 

that's

 

not.

 

what happened

 

I'm

 

right.

 

not entirely sure what exactly happened, but it's possible that it happened like that and that is. also. I think. I mean what you just said. I, if it, boss tells you to, Well, if it goes against my value, or if it even potentially like, If you're concerned, it's like an individual, even though you're not a lawyer that this is going against the law right then, I think it's your, your. You're right to say no, but also sometimes maybe your obligation to say no. At the end of the day, we are softer engineers. When it comes to altimating, things were the kind of a last line of defense. so to speak, right, you're the last person in a chain of people which can possibly say no, and at the very least what you can do is at least say he. At least I didn't do this right. I didn't build this particular thing. Somebody else is probably going to build it, and I'm very much aware that this is a position of luxury because depending on what your living circumstances are, you might not have the luxury of saying No, right. You Actually be concerned that somebody fires you if you said no, and that then your livelihood is engage. That is not not not like, not a privilege everybody can have. But if you have that privilege, If you are in a position to say no to something you think is wrong, then I would actually argue that your kind of the moral obligation to do so. But yeah, it's uncomfortable. I mean, I'm not

 

Yeah,

 

everybody who

 

I kind o wanted

 

who's like

 

sorry, Go ahead,

 

everybody who's ever been in a room full of people when you discuss something and then said Like people, I don't think we should do that. I think that's that's wrong. That is like a very vulnerable position to be in, but it's important.

 

But there's another thing too, where people may not know. It's It's like a bad thing to do like Like, If there's like a way that you can track people or something like, do

 

Hm.

 

something that's more than what you should be able to do right. Like, let's say that you, the guy asked you to be able to remotely take pictures of user, because they just want to get more data. Like what color hair or the male female, et cetera

 

Boy,

 

Like it's a very weird like thing, but like, I'm just trying to think of something Okay. Maybe it's not the best example, but I'm trying to tink of something that Could be benign and somebody's head like. Oh, I just want to know what the user looks like, Because then I can have an idea about how we can target them. Et cetera You know how old are they? Etcetera? they'll say, Can we can you remotely take a picture of them and put it on our service? It's like well that now, and you have to explain why that is a bad idea And I've

 

Yeah,

 

had conversations where people are like. Can you do this stuff? and it's like Well, probably, but it's not good because of these reasons and they just didn't know any better Right. They're just more obsessed trying to get their software out there and make money, so they're not thinking about these kind of things, So like, I don't

 

Yeah,

 

think that everybody s going to be nefarious in their request. I mean, some people are, but some people just may, does not know and may not think about it.

 

I mean, there's this saying that never assume malice when it came in. You can explain it with incompetency

 

Yeah,

 

and there's a lot of truth to that to you, But yeah, at some point along with chain, I mean, I would not expect from a form pomasophi engineer to for example, have like deep understanding of what gdpormeans in detailed and practice, you should have a rough understanding about it because it's probably going to impact all of our lives as soon as you build a product which is in ternational. then you probably will have some people from Europe in Union, So ugly applies to you unless you take explicit measures to exclude them right, And since some companies are doing, they were saying Hey, we don't serve customers from Europe Because well, Grippe doesn't appen very often, though, and of course, if you actually are based in the Uropenioniunion, If your customs of companies basin opinion, then you, ll you are subject to that, so soft engineers we feel that we benefit from having at least a superficial understanding of those kinds of things and having a rough idea. but I gain. his. S. probably not a good idea. This is probably not okay to do, But I would not expect from anybody. I don't think it's also reasonable to expect from anybody that you have a deeper understanding of every possible law that could be relevant to to to to the things you are building. Because at the end of the day that it's I mean, that's kind of a different job then.

 

Yeah,

 

but yeah, but if you, if you know, if you, if you think it's not the right thing to do, then I think if you can, you should, we should speak up about it. And if that

 

Even

 

potentially

 

even if you're wrong

 

means

 

in the

 

losing

 

end

 

autumn

 

think

 

revenue,

 

it's it's better raise the issue even if you.

 

Yeah,

 

if you're wrong, I think it's better to raise it shows

 

Yeah,

 

you care.

 

Yeah, I mean like you also have to pick your battles right, because I personally, I'm not that fond of Google. That's just me. I don't like them very much. I think what the pilot things are doing are problematic, So if I would, for example, now build a company, I would never ever reach for something like Google takes. But I'm a gor employer by using Google, not gonna on't know. I'm not gonna raise hell now right so that is also something where you have to pick your battles. But yeah, I'm not sure we ended up here from from talking about dependencies and up grading. It's an interesting

 

Yeah,

 

topic

 

it's

 

none

 

still

 

the less,

 

a good topic though, because it's happened a couple of times in my life. I don't know if you had something interesting in your life. I thought you did early. You had something happened, So it comes up. I don't know. It's important to me at least.

 

And I mean the depending on you could even make a cross over here. It like, depending on what your processes are, Upgndepdencies. If you have for example, integrated, new upside might be possible that like late newer version, they are even trceracking. even more things right. I mean, modern use tracking solutions there, tracking a ridiculous number of things. Don't even pretend to be aware of all the stuff that racking. I mean, tracking clicks that tracking, like how way curserisas Staying for a while, ere tracking how long you're staying on particular pages and so on of four of its the level of information we gather impressive, let's say that, and even there, you could be saying Okay, if you have a process for upgrading those dependencies on the regular, you might even include something which maybe doesn't even line with your own values without knowing you know. I mean, there's also this wasn't here. This case A while ago were like an M. P. packet's got cut Taking over maliciously and then it. it included,

 

Yeah.

 

like a bit coin, Minor kind of thing. Right like

 

yeah,

 

that Doesn't mean it's the more obvious one which you don't want to happen. But still yeah, It's a difficult problem to solve, and at the very least, I mean elixer to mave, To come back to that elixsiscoming with a few tools to help you make it a little bit easier. Mean you. earlier you mentioned he outdated. There's also he ordered, so there are some some tools you can be using especially for hexitypackages, which make it a little bit easier to do. Yeah to to get a more informed, more more information before making a decision to upgrade something To come back to what we talked Natural. How do you do language version up dates? like, How often do you do that? Because that is also something I don't feel. It's as necessary as a dependency upgrade. At least not of the same in the same cadence. right. I think dependencies you should probably keep up to date fairly regular. but language versions. That's the one thing which I don't know that for me that that's always been happening once in the blue Moon kind of thing.

 

Yeah, I mean, I guess hey around the same time, but I mean it depends what you're saying, Right if you're going from like P, twenty four to twenty five, that's a pretty big jump right. I think twenty five had some breaking changes

 

Hm,

 

in it. That one depends. I mean, I may just tried like mixed test to see if it works okay and read the notes to see what's changed, because sometimes things were taken out. Yeah, I basialydthm, about the same time because I don't. I mean, for the most part things are pretty stable. I think pretty stable. For the most part, I, I don't know. I don't know about you, but like patreleases, I mean, like I haven't really had anything of a break necessary before even

 

No,

 

feet your branches.

 

no, no, we had one time. Me. How exactly was that? I think we had one time a breakage where we updated an T P version and that had had an odd um. interaction. Were he? I think like something? something. I don't remember the details. but but something broke there in like a weird way, so we actually were forced to revert back to an all the T P version and that, of course, also was only called our card production. And and, at that point so not sure what to do best like. In the best possible case, you might maybe want like a set up where you can really do like a partial deployment right, Like can say, do actually doing a big upgrade here, and maybe free replicas of this thing running, and maybe one replica now this new version. If if you can do that right like, there's not a big softer difference between between the versions, and they can reasonably serve all traffic. And then you kind of see your case Like an increased are rate for things from the new department, But that's also that's not trivial to do. M, gonna be honest, I haven't done that yet. I also haven't worked on the system yet where you could make the argument that it's necessary if you have like an s, for example, or if you have like a very very big use of a base and a very large, very big amount of traffic than something like that might be more necessary, But If I could choose, I would love to have something like that, but you can say hey, I want to deploy a new version of this, but I want to want to keep it running alongside the old one for a while. Maybe like See if it's not a ring out in unexpected ways. Have you

 

Yeah,

 

ever done anything like that? Sorry,

 

that canary deployment.

 

yeah,

 

Sadly, No, all my stuff is pretty low traffic. but yeah, I mean, I've looked at and looked into it doesn't seem too difficult. Especial, with cubernesthat, they kind of is able to do all that kind of stuff. I think So that's something that I was looking into, but yeah, it's always just kind of like bring up the new one and then its traffic just switches over and spins down the old one right. That's how we usually

 

Yeah,

 

do it like a rolling

 

this

 

up

 

is

 

grade.

 

also

 

I think

 

what what I've been doing for the past years, nearly exclusively. The one thing I feel which is always not getting enough attention is the whole process of rolling back again like I have not yet worked at a place where that has been working smoothly out of a box like we've been. I've been involved in getting a bit of functionality there. It like a second. Maybe I want to be able to roll back more easily because again, actively something happened where you said he. It would have been nice if you could have been there Doing that. Um, but if it's always kind of been a, been a pain to be honest, but like you deploy something, it seems to be working in an staging deployed to production, and also, maybe it looks fine and then then you realize like shit, this is actually breaking in a way, an unexpected way. It was never easy and never moved to, just to press one button and say, roll back.

 

Yeah, I've head where we just like, go to a previous version that we deployed and then we just click on deploy, and that one is going to override the old deployment. But the tricky part, the really tricky part is. If that one like there's a data based migration in between, that's probably

 

Oh

 

the

 

yes,

 

more tricky part.

 

yeah, yeah, yeah. Well, I would actually say here, because if any of you who's listening has has been doing anything like like having having worked on more Complex system which required more of your safeguards, So to speak, for example, because you have an l. l, or anything like that, and especially in combination with Elixer, reach out because I think that that could have been been very interesting conversation to be had about actually keeping things running and actually keeping things evolving. I think that's a kind of challenge. Her right, Like to keep keep a system evolving over time while maintaining its stability Because we have heard these amazing stories of Ealing systems which haven't Up dated in thirty years, and they just keep chugging along And that is amazing. That is great, but that's not the reality for most of people Right. like most people, it's build a system and then keep it running and keep updating it and keep etching it And soon I often keep it secure. and if you have any of you have experience on that, and like a bigger picture scenario, then shoots message, Because I would love to hear of your experience. Okay, anything else, Allen, If you feel what we should be talking about,

 

Yeah, I mean, it's a kind of interesting that you broths topic, because I'm literally just doing an update today Like I try to keep all those updates of depths into one commit. So we kind of know where they come from, which you know. Maybe some people say Well, maybe you should do like Each depends should be a separate commit. Then you know if that commit broke the next depends is the problem right? That's one way to do it.

 

Want me to do it. I would probably say unless you got a good reason again, So I omething uptitindependencie, s, and pushing that out is like one release. Probably not the worst idea, because at the very least then you can roll back, but very easily again, right, no business. our business rules changes, no potential migration, So if there is a break in the, you can just say okay, let's roll back. Let's figure out which dependency broke out and then then we can do it again. But yeah, If you have very good reason to do so differently, then so old it differently, but el pobably. Remember by my suggestion that you're like. If you do upgrade dependencies, then maybe do that to de cover from any business business rule changes for any business logic changes in your application.

 

Is that how you guys do?

 

Yeah, because we have, because a first review day, Um, an outcome of cold very much being release. Also, obviously, after a bunch of dependency to put it on staging, maybe now at additional test cases for things like that to capture at, seems to be running a final staging. Let's push the production, Because then were rolling back is a whole lot easier when you actually realize it's breaking. That, probably my suggestion. at least so to make it a bit more sane,

 

Yeah, sounds sounds like a plan sounds good.

 

And in general beyond that, it's not let specific, but honestly like having this review day on a regular cadence in our calendars with a check list, just has been taking so much mental load of all of us because you don't have to keep in back of your head. Oh dependencies dated may hold up. dated now later, right, like all of these kind of things, or like, Okay, we have a fixed date on the calendar where we look at all those kind of things, And that also means that for all of the other days I don't know, Look at it and just ignore it. for now. I don't need look at costs and cloud. I don't need to look at. I don't know. Like the documentation that, Up to date we have, we have a thing at a point in time we look at it and it's fine. That is like another big recommendation we've Ben making because since since we've been integrating that like yea, you can then make the argument. Do you actually get all of the things done? You may be discover that context you get. Do you get them in the time Man? But that's a different process, But like that's about how to use schedepiorotize. man ance work that Different thing entirely, but just having this spot on the calendar the day when we say, we all just sit down and look though those checkles to those things, check it off on regular cadet's just helps of peace of mind. To be honest, I'll

 

But

 

probably

 

you

 

do

 

say

 

that

 

you do

 

everywhere.

 

this once a month, right, but then you talk

 

I

 

about

 

think we've

 

moving

 

Yeah,

 

up to

 

I think we've

 

two weeks.

 

I think we've been doing it once a month, and that is a very recent change because I was sick at the time, so I'm not entirely sure, but I think we change the cadence to every second week. Now might be wrong though, But at the end of the day, whatever works for you right,

 

Yeah, I was just curious about why he went from one every month to every

 

I think

 

every

 

we.

 

other week.

 

It was motivated. We did some some changes where our resource request on our community resources. But so that we wanted to scale down our cluster because we figured out, Kay, we actually do pay a lot on compute costs more than we have to. So how that's figure out how we can save money there, and that, In that one of points on the agenda, and we integrated a tool which I now actually Pitch. I'm gonna do it as a pick. Gonna it this pick. so we use a little

 

A.

 

too there to help us figure out. Okay, how much? what is like a good resource request thing here for the individual part of our system, So that like the auto scale for cloud manage abilities that it can do it so. But yeah, I think that was the motivation behind reducing the cadence, So like having a bit more often to see Kay. How hae been impacting our cost estimations And can we can? we do better from from what I've seen so far we are already saving. I think like something like two thousand bucks per month were just nice, so Yeah, Actually wanted to ask our C to if we could kind of say hey, we did this. We save quite quite a bit of money per month. Maybe we can. I don't now, like take ten per cent of that for some cool tooling, Because maybe we did that on our own volition. So like nobody told us to do that, so we saved saved money of a company without anybody especially telling us to do so. So maybe we can have some for cool, too, But let's see Okay, Alan. then I would say, let's talk about picks.

 

Yep. So that's on me right?

 

Yeah,

 

Yeah, saw, I've been playing a new game called Sleeping dogs.

 

Isn't that super old,

 

Well, it's is new to me.

 

But it was originally released on box three, Sixty, Right sleeping dogs?

 

Yeah, it's released twenty fourteen or something, or you know is one twenty fourteen, Defend edition. Yeah, like I told you before, I just got steam deck right, so I'm starting to catch up on all the old games. Well, actually, well, the rason that I wanted to pick this one is, it's based in Hong Kong, where I live right, so it's quite interesting to see some stuff and and hear the Cantonese and recognize some of it. So I thought that was kind of interesting. Kind of has a has a thing for me and they have expansion pack that actually takes place in my district, which I, though It's really interesting. Also, so yeah, I just been playing it since yesterday. I'm like this is actually kind o. a lot of fun Like it's based on like a real gang Over here. They just switched the names around. Have you ever played the game?

 

No, I'm aware I is,

 

Okay,

 

but were what it's about about under cover, Cop, Like going into

 

Yeah,

 

this into this

 

he's

 

gang

 

under cover

 

right?

 

crop. He grew up in Hong Kong. Went to us. Became like a Us. cop in like San Francisco or something, And he came back and he's under cover, Nd most of his friends like he was in a gang when he was younger. Um, I just started playing so I don't really know all of it, but yeah, the only thing that sucks about is that it's not you. Don't you don't have cloud cloud sink for the said games, which sucks because I'm playing on different platforms sometimes, so that's the thing I don't like about. Otherwise it's been Of fun. The combat is not that great, but I just think the story is quite interesting and he's got a lot of character to him. So think it's a pretty cool game. So if you're like games, I think the history of this game is that it was supposed to be the third, and like you know, remember the was called Streets of La, or something like that.

 

No, sorry.

 

I forgot the name of the game. Now there was actually three games. I forgot the first one, but the second on, I think was called like Streets of La, Or I have the cover in my head, but I forgot the name of the game. This is the third one, But then They decided to do their own thing. I think they lost the license or something, so it's interesting to see that, but in general, yeah, it's It's quite a lot of fun. So if you like t, t, a kind of like. Well, not even ta. Because Yu on't really steal carse. S far as I know, Yeah, I don't check it out. I think it's petty cool. It's a old game. It's on steam for. I don't know what that is. A hundrfiftyfour Kong dollars. So it's like what? Twenty something, Us dollars. I think I think it's fun so far.

 

Nice. I've actually on on little story there. I recently gifted

 

M.

 

my old expoxfresixty away. We have this little like a Facebook group Here,

 

Oh

 

My wife is in, which is basically you can put in things to give to people because you say I don't need it any more and needs to be given anything like. I had the thing right lying around for like, Don't have three years. Because piece for on our piece five, I don't need it any more and I did have sleeping dogs as one of the game, but I never played it

 

Yeah,

 

lying.

 

you bought the game. It never played it.

 

It never happens to me like it's a

 

How's

 

very

 

everybody?

 

rare occurrence that I buy a game and then don't play it

 

Yeah, it's It's a lot of fun. I quite like it is. like you said. It brings little bit more to me because I live over here so some of the stuff is like, Definitely true. So

 

Cool.

 

about your side,

 

What is my pick for this? as I want to pick one little soft tour, which actually has been helping us a lot on this costiplemization I mentioned earlier, and that is gold locks And goldilocks is basically like a thing you integrate into your cluster, and it's monitoring resource useargh for the individual components for the indeldeployments of the pots, and then helps you figure out what are the right resource requests you use You. Can you can set up for these components for for fobes pieces in your cluster, because it just looks at them over a while. and okay spikes how much memory does it use on every La. So that helps you find in a sweet spot. And honestly, the name is just amazing because I'm also spacenord and the Golylock zone is also, the is, the word for the area around a star in which a planet needs to be to have liquid water, which is at the moment the assumption necessary to sustain life. you know, So like liquid water is one of the things we assume is necessary that life can develop on a planet, so kind of. The Golylokzone is the zone in which a planet could Potentially sustain life outside of that. It's even call to hot liquid water. right. So yeah,

 

M.

 

called golylocks for cubinators, kind of kind of makes sense. I love the naming of this one. so check it out, because honestly before that, getting these resource requests right is like I don't know a dark dark art, black magic. I don't know how much do I need

 

Yeah,

 

right. So having a tool like this which just monitors

 

yeah,

 

you applicate Sons over aduration that helps you. Hey, based on what I've seen, so

 

Oh

 

of this is probably like a good set up. That's just nice. Yeah, that is

 

yeah,

 

my engineering pick for this week

 

Yeah,

 

and beyond that, To say anything else, I'd like to pick. No, I know, I don't think so. I mean like I picked. I'm just going to re pick something from last week because I still have lot of fun with it, which is sef game. I've been playing a lot lately. It's also

 

Oh,

 

not not unsimilar to to sleeping dogs in the sense that it's also like a Very

 

h.

 

combat heavy and also very martial

 

h.

 

arts heavy game And sea was basically, and last week I did name wrong. It's

 

H.

 

it's Brucey, Especially like

 

Yeah.

 

a Brucey movie For playing yourself, because it's like it's Cook Fu revenge story, kind of clearly thing, but it just has this combat system which is super deep and you can really dig your teeth into, and it has this neat little rock light mechanic where the premise is basically to the maker, The hats like this metrical pendent, which, when they die revives them, but ages them from the beginning one year, and then move about. more of you die. the more ages yureticfor each death you get like an additional years, like th first time, one year than you get second years older, three years older for your ponsofor. But there's also a check points in the game. Contuctgets, Really used by one. Um, And you kind of need to defeat the bad guy throughout multiple levels without kind of dying of old age, so to speak, So yeah, it's it's fun. It's a lot of fun. It's also fun to see how you get better because inevitably you will have to play old levels and it's just fun to see like the first time you got proved level your character was, What Like forty five, and then the next time you are thirty three, and then at some point like just you start, you start at twenty, Nd, like twenty, and then you go out twenty like easy. Psy, laments Greasy. Just because you, you kind of Better at the game, So yeah, if you, you're looking for something which is just mechanically appealing. Not that, not a big story, but has a very nice and very, very fluid and very dynamic combat system. And then this is maybe something you want to look out for. It's a lot of fun I picked at last week already, but still very much love in love with it. So yeah, those are my two picks for the week. Okay, Ellen? Its usual was a pleasure. What are you doing on laptop? I see. I see your clicking

 

I

 

things.

 

was just looking to see the sipo game.

 

Uh, yeah, it's also very stylish,

 

See if

 

so

 

you're right,

 

like.

 

S. I f.

 

S. if yeah, it's also

 

Yeah.

 

super stylish, so

 

It's

 

it

 

not

 

gets

 

out

 

very

 

yet.

 

very trippy in later levels, but like in a good way,

 

It's coming soon On steam. You got on steam. We got somewhere else,

 

I got it on P. five.

 

okay?

 

Okay and thank you folks for listening. Thank you for for sticking around for so long and listening to Ellen and me rambling about dependencies and ethics and morals and saying no, and tune in next time when we've another episode of eliximics by.