Sascha_Wolf:
Hey everybody and welcome to another episode of Elixir Mix. This week on the podcast we have Adi Angar. Oh, he's drinking
Adi_Iyengar:
Hello.
Sascha_Wolf:
coffee, sorry. He was drinking coffee, I didn't see that. We have Alan Weimar.
Allen_Wyma:
Hello
Sascha_Wolf:
And me, Sasha Wolff. And we have a special guest this week. And that is, now bear with me, Nikola Petrochich. Nikola, why
Nikola_Petrušić:
Hey
Sascha_Wolf:
don't you
Nikola_Petrušić:
guys.
Sascha_Wolf:
tell people why you are here? And what are we going to talk about today?
Nikola_Petrušić:
Yeah, sure. Hey guys, I'm Nikola. I'm an Elixir developer from Croatia. Yeah, I've mostly worked, I worked with two Elixir companies so far, both in the sports betting industry. So Adi asked me if I could come a bit and talk about the Elixir ecosystem in the betting industry since I do have some experience there.
Sascha_Wolf:
Nice. And you said it's like the second time you have the sports betting interest?
Adi_Iyengar:
Sasha is... we lost Sasha I think.
Allen_Wyma:
Yeah, so you're working in the sports betting, I heard you're the expert in sports betting, right? Is that because you've been working in this industry for a while?
Nikola_Petrušić:
Sorry, I mean, yeah, I've been working in the industry for about four or five years. I wouldn't say I'm the absolute expert. I've just had like multiple, yeah, I had multiple betting experiences, interestingly, quite different architectures on both sides.
Sascha_Wolf:
Was it a coincidence that you kind of ended up in that industry or I don't was it something like you seeked out deliberately?
Nikola_Petrušić:
So my first experience with the industry was just a coincidence. I was contacted by a friend who worked at the company, said they were looking for back-end developers. I worked at Python before and yeah, I came to the interview. I like sports. I'm a big basketball guy. So I was interested in working in that domain. Yeah, after that. At my second company, it wasn't as much of a coincidence. As you know, there aren't very many Elixir companies in the world, especially in Croatia or Europe. So I was just looking for Elixir because I really enjoyed it. I really liked it more than I did the languages I used before. And yeah, betting companies like Elixir. Betting companies like people who worked in betting companies before, so it was kind of a good match for me.
Sascha_Wolf:
Yeah, that makes sense. Just sort of got to go ahead.
Adi_Iyengar:
Yeah, I was gonna ask why do you think it is so Nikola? Like why do betting companies like Elixir?
Nikola_Petrušić:
I mean, it's, I would say it's just like the properties of the, like all the positives you hear or read when you go to, when you read an Elixir book or go to like a medium article on Elixir. It's yeah, Elixir supports concurrency really well. And I think that's like the biggest part of it. In betting systems, betting systems are usually like... large systems with a lot of external data coming in, different kinds of requests being processed all at the same time. And usually like in, as you know, in like small bursts of time, you have huge loads and yeah, so, and you want to do that all concurrently. You want to do it all parallely. So Elixir really has a nice support for that. Some other things would be like Elixir's It's really important that, uh, that betting systems are reliable as, uh, you're dealing with people's money. You're dealing with regulations. Uh, so you really want as, as a reliable system as possible to handle as much of a load as possible as you can.
Allen_Wyma:
I'm quite surprised that elixir and Erlang would be interesting for this. I mean, of course, we know it's relatively quick. Um, but it's not like as fast as C, the C++, Rust, these kind of languages. Right. So, um, I guess you can handle lots of bets, lots of people connecting. Is this the reason why elixir and Erlang would be looked at for a betting organization?
Nikola_Petrušić:
Yeah, I mean, of course, yeah, like C and C++ are always going to probably going to be faster than most other languages except like maybe Rust or something like that but yeah, you always have to make a trade-off I think it's it's it's a lot less tedious of a task to build a huge system in Elixir rather than in C or C++ and Yeah for the more performant parts You would use C or C++ and just call those procedures from Elixir or in recent times like Rust or something.
Sascha_Wolf:
Yeah, I also think, I mean, like the complexities of something like a, like a sports betting system, which is inherently, I feel often a very event driven system. Right. Because I mean, you're basically waiting on things to happen in the real life. That data comes in and then based on that, like you do business reevaluation, there's not a lot of number crunching in there. It's just a lot of data routing and figuring out what goes where and like, which numbers go up, which numbers go down, but in general, it's not like. complex scientific equations or simulations that need to be run very fast concurrently. If you have those, then yeah, I guess C++ and Rust and whatever, that's a better weapon of choice. But if you want to have a resilient system which handles a lot of concurrent throughput, I mean, it kind of sounds like the Elixir and Erlang written all over, you know?
Adi_Iyengar:
Yeah, I think it's also the ease of scalability. It's not that
Sascha_Wolf:
Mm-hmm.
Adi_Iyengar:
if you do the C, C++ route, you could still scale it, but then you'd be spending so much more resources in figuring out ways to scale it in a first designing it concurrently and then figuring out horizontal and vertical scaling, which elixir, at least vertically, is very easy. And with some of the clip cluster, horizontally it's also become so much easier. And I think the added bonus is also with all these sports betting things. especially if you need like a back office, like B2B kind of system, Phoenix and LiveView make it very simple and easy to quickly extend that to simpler interfaces without having to create an amazing front end or something. Not that you can't do an amazing front end with Phoenix LiveView, but it's especially easy to do a simple B2B kind of a front end or a back office front end.
Nikola_Petrušić:
Yeah. And the thing is like Elixir is, I feel like Elixir is, Elixir apps are often written like, yeah, you can crash, just start again. And that's a very useful,
Adi_Iyengar:
Yeah.
Nikola_Petrušić:
very useful thing in betting specifically. You'll get a crash every now and then. You just want it to start again. You want to keep it running. You can't have downtime. Yeah. You can't have latency. Those are all very expensive when you're betting.
Adi_Iyengar:
Totally, totally. Yeah, I think I totally, totally missed that point. Yeah, the reliability is so huge too. I guess like one more thing I wanted to say, like there's like one of the companies I advise on the side, how I sold Elixir to them, and they do a lot of concurrent processing, but it's like the scalability within a pod or within a node, the vertical scalability, with the limits that we have today, memory limits, you can. pretty much increase memory to 200, 300 gigs and do anything in one Elixir container or at least a node with multiple contains. But that is so huge. Building a C, C++ application that can do that is a lot of work, right? Because these days, you can have, again, limits to memories like, you know, I think you can even have a terabyte memory, right? now and in clouds and that's like you know you can run Twitter with one elixir node and a terabyte memory yeah it's expensive probably not the best architecture but it's true.
Sascha_Wolf:
I mean, I honestly think that the value of vertical scaling has kind of been lost a little bit, like in recent days of cloud and communities becoming the norm, because honestly, I mean, you said it's not a good architecture, but why though? I mean, you lose a whole lot of complexity in the sense of, well, if you actually want to do horizontal scaling, that's usually not, usually, there's exceptions, usually not such a simple thing. Well, it's been a lot of those things and it just works, you know? Sometimes it is and then way great, but often it isn't. And then you actually have the option to say, hey, we can kind of save on like complexity here and just buy a bigger beefier machine, maybe with a back off, you know? Why not? Honestly, why not? Because I still remember that like when we had kind of a similar project, that was more fantasy football thing, but there we actually... were doing horizontal scaling, but because we were using commanded, that was like years ago, probably they made that easier at this point, but we wanted only one node to actually do writes, like to send out commands because then otherwise it could get finicky to deal like the implications of two nodes issuing commands. So then we had to like do routing rules. So we say, like, if it's this request, right, then can they can do to the replicated read replicas kind of, and if it's this request, well, then it kind of goes to this one writing node. And I feel if we just said, well, just buy a bigger B4M machine, that might've been easier, honestly.
Adi_Iyengar:
But that's not fun as engineers, right? Like as engineers, we have an obligation to ourselves to not stop at good enough. We want
Sascha_Wolf:
Eh.
Adi_Iyengar:
to go above and beyond. We're like, oh, vertical scaling always has a hard limit. Horizontal scaling on paper might not, right?
Sascha_Wolf:
paper,
Adi_Iyengar:
So
Sascha_Wolf:
yeah blah
Adi_Iyengar:
yeah.
Sascha_Wolf:
blah blah.
Adi_Iyengar:
But
Sascha_Wolf:
I mean.
Adi_Iyengar:
that's what an engineer thinks, right?
Sascha_Wolf:
I honestly, I want to pitch one little tagline we have at work currently where we are replacing a microservice landscape with a single modular application, not modular, right? They kind of still apply microservice thinking about having separate contexts, but it's one big code base hosted as one big thing. And you have this tagline of all that. We call it delightfully boring. And honestly, that is a very, I feel like that is a very healthy design approach. Deletrily boring.
Nikola_Petrušić:
Yeah, I mean, you do make good points. I have seen horizontal scaling being done really well with Elixir and Kafka. So Kafka really makes it... If you use Kafka in a lot more places than you would usually use it, you could actually leverage it to have horizontal scaling as easy as, okay, let's just turn up some more pods. Everything will be fine.
Adi_Iyengar:
Yeah, I think that's the key, right? Like a nice async architecture within any kind of like bus, which is which can support that is the key for horizontal scaling for sure. But I mean, again, the complexity in design architecture also kind of reflects and complexity in DevOps and right. Like it's just it's just the costs like don't go up. The cost go up exponentially. If you
Sascha_Wolf:
Yeah.
Adi_Iyengar:
if you increase the number of microservices, if you have
Sascha_Wolf:
Yeah,
Adi_Iyengar:
like
Sascha_Wolf:
yeah.
Adi_Iyengar:
four to four to five, it's not the you know. It goes up exponentially. It doesn't quite go up linearly. They need a lot more people to manage five microservices.
Nikola_Petrušić:
Yeah, it's definitely, especially in the start, it's a lot more expensive than having like one monolith application. But depending on the domain, depending on the context, I mean, if you're a betting company and you have crashes because your monolith can't handle something that horizontally scaled application would, you're probably losing a lot more money on those crashes than you would paying for those pods. So yeah, everything's a healthy trade-off, I guess.
Sascha_Wolf:
I also want to like, just to like underline again, like that, even when you opt for something, like a kind of like a more monolithic approach, that doesn't mean you have to deploy it as one big monolith, right? That is, I think something people sometimes mix up, like say I have one code base, I need to deploy it as one thing. And I mean, there's no, there's no reason for having to do that. You can very much have multiple releases, like which build different versions of your app. And maybe one of those is something which you want to horizontally scale because you say, okay, this is like an awful read heavy thing and I actually wanted to scale it out. So I have these kind of deployed on 10 machines, but like the, the core of the business, which does background processing, whatever, I don't know, right? Like that is a thing where like I have a lot of write heavy things. That is like this other thing I deploy, but it's still the same code base, you know, that is One thing I often see people mix up when they say microservices or monoliths or modulus, where like the one is deployed in one big thing, the other is deployed in multiple, but there's no reason for having to do that.
Adi_Iyengar:
Yeah, it's actually, that's exactly how it works in the current team, the client I work in. We have this one, I guess you can call it a monologue. I don't consider it a monologue, it's a big app, but not
Sascha_Wolf:
Hehehe
Adi_Iyengar:
big compared to other monologues I've worked in. But yeah, we have different deployment configurations, for lack of a better word. And yeah, the responsibility is somewhat segregated at the time of deployment. But just not a bad architecture at all, like I said.
Sascha_Wolf:
It's also, I feel, doubly so. I don't know. It's also very comparatively easy in runtime, like V-Orlong-VM, where you actually think about your applications in terms of a supervision tree, right? And you can actually say, well, in this deployment over there, that's the part of a supervision tree, which is going to spun up and this deployment over there, this other part is going to be spun up. And like you have this very, very useful abstraction to say, well, those are the things that happen over here. Those are the things that happen over here. So honestly, if
Adi_Iyengar:
Yeah.
Sascha_Wolf:
any of you listeners out there like are considering, hello, should we do microservices, should we do modules? I would, unless you have a very specific and very good reason to go with microservices from the get go, and I wanna hear that, reach out to me if you wanna hear it because I have not heard it yet.
Adi_Iyengar:
Suck me.
Sascha_Wolf:
Then start with like a modular thick application like where you still kind of apply the same things and you make it easier for future selves. If you then maybe your team grows, right? And you have multiple teams, well then cut a thing out and put it in a separate repository. Nothing is stopping you from doing that, but at least in the very beginning, you don't have the whole devopsy overhead of having to maintain multiple responsibilities, multiple deployments and so on and so forth. Because chances are when you start off a product, you're going to get the separations wrong anyway. I'm not sure how you ended up here, but... From thoughts
Adi_Iyengar:
Elixir
Sascha_Wolf:
battling, but it's
Adi_Iyengar:
and
Sascha_Wolf:
interesting.
Adi_Iyengar:
sports betting. I guess I'm going to take us back to the elixir and sports betting then, right? So Nikola, let's also stay in the same vicinity of the same architecture. So you've worked at two companies. The first company you worked at, I'm not sure if you can name the companies here or not, but the first company you worked at, did they also have this? microservice the approach like horizontally scaled or lang clusters I can talk to each other versus just scaled. Clusters that don't talk to each other.
Nikola_Petrušić:
Yeah, so it was a very distributed system with a lot of microservices and very, in a true meaning of the word, microservices, very specific things. It was an event-driven system. I would say the idea was very similar to what Commandad is doing, for instance, with the bit different execution, but that same approach. But yeah, there were like, I think well over a hundred microservices communicating through Kafka. And Kafka was also leveraged for storage. So there weren't really like SQL databases anywhere on the stack.
Adi_Iyengar:
Interesting. What? I'm trying to think how it would work. Ha.
Nikola_Petrušić:
Yeah, so I can tell you exactly. Yeah. So, uh, we, uh, you would have like two different types of topics, right? So, uh, one would be like, uh, we, uh, we could call them like a command topic, which is similar to, which is used for communicating between services. Like send, uh, if you're using an event during system, you would send commands or events or whatever through, through those topics. And those would be like regular Kafka topics with low retention, a couple of days probably, or something like that. And those would get, those get processed, processed immediately. And then you would have domain topics, which would be, domain topics would be like an analog to a table in a database. So your one, one, your domain topic would be one message in your domain topic would be one row of your database table for some entity in, let's say, in betting for sports events. So you would have a topic with just a bunch of messages with sports events. And the last one is always the last one. The last message for the event is always the current state. And then you can have you can have vertically scaled. microservices that literally load or read through in the entire domain topic and load it into an ETS table or something like that. And you would be surprised how big of a load you can load in an ETS table and keep it in Kafka like that with not like terrible, terrible memory consumption. And yeah, it works really well. It's like I said, it's... If you get a crash in the app, it would restart, read the entire. So those domain topics are read from the beginning every time. So you don't keep missing messages and stuff like that. And yeah, it works really well. The APIs were really fast since you don't have database retrieval. Everything's in memory. But yeah, of course it has its problems. And everything you mentioned was true, completely true. Like it's a lot easier to deploy. Yeah. One bigger app than hundreds of smaller.
Adi_Iyengar:
I do like that the individual responsibilities of each app, it sounds like that itself wasn't distributed, like distributed or non-distributed. It looks like you relied on vertical scaling for the individual deployments, at least. So I think this is actually pretty good. It is consistent with what Sasha was saying, that it's simplifying. based on responsibilities instead of trying to optimize for horizontal scaling. And if you, for example, clusterize the application that is consuming domain from the domain topic, domain commands or domain events, whatever you said, that would make your deployments a lot harder. Because then there's conflicts and stuff. You have to worry about what Sasha said. Make sure only one. message gets consumed at one time and stuff like that. So I really like the vertical scaling approach with using kind of the context or domain to split services into microservices and horizontally scale those. That is not the case in our current team, unfortunately.
Nikola_Petrušić:
Yeah,
Sascha_Wolf:
Um.
Nikola_Petrušić:
I mean, yeah, the split, the split makes it easier to see when you need horizontal scaling, when you need vertical scaling. So let's say, for instance, the API we were talking about, if the number of messages in Kafka gets way larger than it was before, it's pretty, it's reasonable what you need to do. You need to increase the memory, you scale vertically, and that's it. If for instance, you can't process the number of messages coming in from like a command topic or from a command topic or something then you can scale it horizontally add a new pod which could help the load Kafka keeps everything consistent. So yeah, you get that benefit for instance there. I mean, like I said, there are benefits there. There are definitely drawbacks and everything. It's all a big game of weighing. Bros and Colts.
Sascha_Wolf:
That's engineering in a nutshell, I feel, honestly. It's also what experience comes in and it becomes valuable. But I'm actually curious because I mean, you just said, I agree with what you just said, Adi, with not being unlike what I said earlier, but it's also what is most important in a case like that is that you get these responsibilities and these separations right. Like, how do you do that? And I would actually be curious if you know, Nikolaj, how those, how those areas of responsibility were identified in this previous company, how they figured out, okay, this is like one deployable, right, and this is another. Do you know?
Nikola_Petrušić:
Yeah, so it's I wasn't there when it was decided that it was already the architecture was set
Sascha_Wolf:
Mm-hmm.
Nikola_Petrušić:
when I came in, but I would assume it was something seem like I don't think anyone there first came up with event sourcing or CQRS. But I would say like responsibilities were mostly you would segregate them based either first on like technical responsibilities for, let's say in command that you, command is segregated by, you have projectors, you have aggregators, you have commands events. And then, and then you would also, depending on the, of course, the, of your, on your case, you would segregate them on, on a domain level. So I may have a service, I can have an API service just for, just for I don't know, events, I can have an API service specifically just for bets and stuff like that. So, and yeah, it's a lot of, you can never know in advance, you can only assume and look over the results and reiterate and try to make it better on the second go.
Sascha_Wolf:
But that is something I feel like in conversations is often glossed over, right? Like, do microservices get the segregation and the separations right? And everything will be fine, but then nobody tells you how you do that. And of course, I also don't think there's a recipe you can follow and then you just end up with the right kind of separation. But it is a point of... of application design, of system design that is usually, I feel, in discussions very much pushed aside. Some actually was curious how you managed to do that there, because it's been a very involved process at the place I'm currently in, because we are working on a new product now, which hopefully will launch throughout this year. And there was a big part of that was figuring out how do we cut... these concerns like which responsibility falls in which kind of area, because we also wanted to cut the teams accordingly, having actually teams responsible for certain parts of the application. And getting that right, of course, there will always be iteration, there will always be something you learn down the road, but hopefully you start with a design which is not totally off. And in our case, we... We applied a lot of domain-driven design principles. We did quarterly mapping workshops, we did event stormings. And at the end of the day, we arrived at something we felt, okay, this makes sense. This sounds like it's reason. So far, it's been holding up, but like I said, I'm curious to hear how other teams and other companies go about this.
Nikola_Petrušić:
Yeah, if I were, if I haven't yet had an opportunity to start, start the system, designing its architecture from the, from the beginning. But if I were to have a go at something like that, I would, or if I needed to suggest someone how to start, I would say start with, yeah, exactly. Like you mentioned, a simpler monolithic architecture you can, everyone can understand and work on and then based on your. on your requirements based on like new information or the size of your team, your company or whatever, introduce if something needs to be offloaded to a separate service, you can offload it to a separate service. And you can build, you will either get to like a really well-made monolithic application, which didn't need separate services, or you will get to a a microservice architecture that was actually segregated on based on your actual needs and requirements at the time you were doing them. So it would be probably okay. You can never say it would be perfect, but yeah, yeah, that would be my general suggestion. Other than if you have like, if you have a client who says, look, we want this done with microservices because we had a consultant tell us. we should use microservices then yeah use microservices and that's pretty much it.
Adi_Iyengar:
Yeah, I totally agree with what both of you said. I think one of the key points I want to highlight again that you guys mentioned is there really isn't a perfect design right away, right? Because like Nicola said, it's based on the usage, which also changes, right? So I advise startups a lot on the side, and having advised a bunch of startups, or at least they start building the systems. I think one thing to optimize for is simplicity and flexibility, right? If you build a system which is not resilient to change, early stage, you're doing something wrong. If you overcomplicate, you're doing something wrong. So simplicity and flexibility also being the key because things will change that you just there's no way you will know how your system will be used. So yeah, totally agree
Sascha_Wolf:
Yeah.
Adi_Iyengar:
there.
Sascha_Wolf:
Yeah. I feel that is also something I can tell and kind of have some experience in my career. Like in the beginning, like when you kind of start out, at least it was for me like that, you kind of figure out, oh, maybe I can find the perfect abstraction, right? And then it never needs to change again and everything will fit in neatly. And then like the customer comes around and says, well, I want to square in the shape of a kitten. And they're like... Excuse me? How do I put this in here? My abstractions don't fit. I feel that is like something a lot of engineers learn painfully. I think it doesn't have to be that way that a maintainable software system is a system that's easy to change. And usually that means it's simple and simplicity. Coming back to what I said earlier, like delightfully boring, right? Delightfully boring systems are something which, well, you look in there and you're like, yeah, of course, this is how it's written. I would have not expected anything else, but the bare minimum to make it work like that. One thing I especially find very relevant in that context is something I also, I repeat very often throughout my day as a technical lead now is the last responsible moment principle. I'm not sure if you're familiar with that. But the basic idea is that you want to postpone making a decision until the last responsible moment to make that decision. Because very often, a lot of these complexities and these weird abstractions come in a moment in time where you make a decision, even though, honestly, you don't have to make that decision in that moment. And you make a decision in a moment where you don't have. the maximum amount of context to make that decision. That is kind of what the whole last responsible moment principle is about. Like you wait until the last responsible moment to make the decision, because then you have a maximum amount of context to actually make a good decision. And very often you can just say, well, duplicate that code over there, like once or twice, that's okay, right? Maybe flag it, like if a comment, that is a duplicate thing. And then if we need to duplicate it the fourth time, then we can make a decision how we abstract it, for example. But don't do it immediately when you need to do it a second time.
Adi_Iyengar:
Yeah, it sounds like another version of basically the cost benefit analysis, right? Like make sure the benefits outweigh cost. But again, like this is what's key is, like you said, benefits should also include your confidence in that benefit, right? So like if I think, oh, the benefit will be 10 and cost will be eight random units. But my confidence is the benefit will be 10 is very low, then we should not make that decision, right? So it sounds like cost benefit confidence analysis is what it sounds like last responsible moment. Awesome. I had not heard of this before, but it's pretty cool. I like it.
Nikola_Petrušić:
Yeah, basically
Sascha_Wolf:
It was very
Nikola_Petrušić:
it
Sascha_Wolf:
catchy.
Nikola_Petrušić:
comes down to it comes down to trading risk analysis.
Adi_Iyengar:
Yep, exactly.
Sascha_Wolf:
You want to minimize risk with that. I mean, like you have to figure out like how much risk is there and not making the decision right now. And honestly, very often the answer is not much to be fair.
Adi_Iyengar:
Yep.
Sascha_Wolf:
And still people jump to making decisions immediately.
Adi_Iyengar:
Right. I do want to highlight, I think, because a lot of our listeners are junior, mid-level people too, I do want to encourage them to over-engineer their side projects and their things that are.
Sascha_Wolf:
Yeah, yeah.
Adi_Iyengar:
If you over-engineer something and cost of that is low, like side projects or something very simple that you're managing, I want to encourage this because to get to a place where you can analyze cost and benefit of a production system, you need that experience. Right? But you won't get that experience. There aren't enough opportunities to do that. I learned this by having hundreds of side projects and over-engineering those. That's the only way, I think, the best way to learn, because there aren't enough opportunities at your regular work, unless you work at a really awesome company. There aren't enough opportunities. I'd want to encourage people to think through architecture, if not implement that, and the side projects are less... whatever not important projects so they can further understand what is the cost and benefit, right? So it's super important.
Sascha_Wolf:
I think that's fair. I think that's a fair advice. But you don't have to. I mean, we talked in the past podcast episode, I don't really do side projects. So even then you can become a decent software engineer.
Adi_Iyengar:
Well, I think Sasha are an exception. I think some people just learn things and visualize things very fast and some people don't. I am in the latter category. I need to do a lot of side projects to be able to understand things. Some people are like Sasha where they're good at visualizing and thinking through a problem and solution quickly. But most people in my experience are like that. But anyway.
Allen_Wyma:
Yeah, that's a good point. I've had a lot of interviews where I went to and they were like, oh, well, you've never done this, you've never done that. I'm like, listen, I've worked a lot of startups. I learned a lot of my stuff by myself. And just so happens that most startups in general don't ever get traction. So I know the theory, but I haven't done it before and I'm more than happy to have a try. And if I make a mistake, I have people I can call upon or I can research. I mean, there's ways around it. But sorry, not everybody has scaled Facebook. You need to understand that
Adi_Iyengar:
Yeah.
Allen_Wyma:
this kind of thing. So yeah, sometimes I am a little bit like Adi, where I just over-engineer it. But the problem is sometimes all my effort gets wasted. Like I said, I did a lot of work for a couple of startups, and they all went kaput. But at least now I know how to do it. So it's always nice to bring up an interview.
Adi_Iyengar:
Totally.
Nikola_Petrušić:
Yeah, in the end, you're an engineer. So yeah, it's not, we're not really, our jobs aren't really to do the same thing all over and over and over again. Yeah, we should be, we should be challenged.
Allen_Wyma:
But I mean, how do you handle this kind of problem where it's like, because one of my clients right now, they're telling me, oh yeah, we need to run this stuff through Kafka. I'm like, are you serious? You have like four traders, and you think you need to run that through Kafka? I think we could just shoot that straight to an Elixir service and then just send out the trades. Like, I don't. No, no, no, we trade a lot, really. How many trades do you, you know, like 60 a day? I'm like.
Sascha_Wolf:
HAHAHAHA
Allen_Wyma:
So I don't know if he was looking for an excuse or what's this deal was, but it was very, yeah, very weird. Like that's never come up ever again. But you know, like, I think there's gotta be a limit of when to over engineer. Right. So like if you're having 60 trades over an eight hour period with six people, it's not worth it to put in Kafka when you can just cue it in memory with Elixir or even put it into open or something like that. Like there's so many ways around it.
Sascha_Wolf:
I'm gonna put another little technique I like to use in cases like that here. It's called nine why's. And basically you ask why nine times. So like when they say, we want to use Kafka. Why? Then they give you a bullshit answer. Then you ask why? And then at some point you actually, when you can't really say anymore, well, that's, I don't know, it's the fundamental property of the universe, you know? Basically when it ends up, it goes... talking about money because at the end of the day, right, like if businesses make decisions for monetary and financial reasons, when you end up at the money, why? That is where the truth lies. And then often it's, oh, why do you think you need Kafka? Well, we have so many trades, we won't be able to do more and then that will hurt our financial bottom line and then you can kind of start educating. Well, honestly, Kafka is the system designed for like thousands. of messages per second. You're talking about 60 per day. You don't need this.
Nikola_Petrušić:
Yeah, I think it's just, oh, sorry. Yeah. I think it's, uh, there's a lot of, uh, I mean, let's call it ignorance. It's not from a bad place. Like a lot of people, uh, who are clients to IT companies or, uh, single developers. Aren't really IT people. Uh, they haven't really worked with all this. They, they, they heard of it somewhere or someone told them, or in the end they hired a consultant. uh, which told them that would be good for whatever reason they thought at that point, which you would never know as a, as an external hire after that. So, uh, yeah, I think it's, I mean, uh, I would say like probably most jobs deal with difficult clients sometimes. And this is just like the IT part.
Adi_Iyengar:
Yeah, I do want to also mention, Sasha, I think great call out, the Nine Wise root cause analysis. I think there is a cost to the whole Socratic questioning. People get frustrated very quickly. And I've experienced that people don't like to be questioned when they deeply know what they're asking is irrational. And that's why I think
Sascha_Wolf:
Hahaha
Adi_Iyengar:
building that equity, like if you're talking to a client especially, you don't want them to look at you and. be frustrated or this guy again. You have to pick your battles too. But yeah, I just want to mention that because I used to live by Socratic Questioning. And then I worked at a startup for a few years. And oh, man, people hate that.
Sascha_Wolf:
Honestly, I mean, that is part of the reason why I went into product development, because then you have this baseline of trust that you're all kind of working towards a shared goal. You know, like they don't assume that you kind of want to get plus one over them, you know, but that you actually are here to solve problems. And I honestly, that is maybe just me. I personally enjoy being this annoying guy which asks these silly questions, you know?
Adi_Iyengar:
Oh it's a lot of fun, it's a lot of fun!
Sascha_Wolf:
Why do you want to do that? Tell me, tell me again, okay. But yeah, but why? I don't understand. And usually, I mean, I have not yet had a session where I did that, which didn't end up in learning something very interesting. And also often, sometimes you actually realize, okay, what they're asking makes sense, but more than often not. Because more than often in my experience, people come to you with a solution. They have a solution. and they don't tell you what their problem is. And that is an issue because the people who usually come to you with that solution, they are not software engineers, which is fine, right? Like I mean, that everybody can be a software engineer, but how would you expect somebody who doesn't have this background to come up with a good solution, like an appropriate solution, one that makes sense, that has like the best balance between cost and effect, you know? So they come to you with like telling you, I want a button here. And maybe the problem they're having is, well, the CSV export they've been using actually has a bug in there because a field is missing, right? Like something like that. Something silly like that. But they kind of come with you, well, I need a button to get the data so I can then put it in Excel and merge it myself. And then everything is solved. Yeah. What you want to get to is the problem people are having and then you can figure out what is actually the thing we can solve this. And in the best possible case, and that has happened rarely, but it has happened, in the best possible case you arrive at the problem and you figure out you don't need code to solve this. Because code is expensive to write, it's expensive to maintain, and it's expensive to, I don't know, to keep around. So in the best possible case, you end up asking a bunch of why's. You figure out this is your problem. But you don't need code. You need, I don't know, you need a piece of paper.
Adi_Iyengar:
Yeah. Love
Sascha_Wolf:
I
Adi_Iyengar:
that.
Sascha_Wolf:
don't have a good example right now, but yeah.
Adi_Iyengar:
Totally
Nikola_Petrušić:
Yeah,
Adi_Iyengar:
agree there.
Nikola_Petrušić:
it's similar to like coming to a doctor's office and saying, yeah, I need this pill, this pill,
Sascha_Wolf:
Yes,
Nikola_Petrušić:
this
Sascha_Wolf:
exactly.
Nikola_Petrušić:
pill, and this pill.
Sascha_Wolf:
Yes, exactly. That's a good analogy, actually.
Adi_Iyengar:
Great analogy.
Sascha_Wolf:
Yeah, usually people come to software engineers telling you, I need VR ground. You're like, wait, wait a moment. Have you seen an advertisement for this?
Adi_Iyengar:
Yeah, I also really, yeah, I think we've talked about that a couple of times, Sasha, but yeah, any, any code is tech that kind of mentality where even if you write perfect code on top of perfect code that is increasing overall complexity, maintainability of the application. So being selective about what features to add is also such a good approach to product engineering and in general.
Sascha_Wolf:
And honestly, you can really make it a completely egoistic thing, because if I get paid for a system that is like, I don't know, 10,000 lines of code, but it still does the job, like great, right? I mean, amazing.
Adi_Iyengar:
Yeah, yeah.
Sascha_Wolf:
Bye-bye.
Nikola_Petrušić:
Yeah, it's similar like with music. Yeah, it's a, it's the one thing to write something very complicated and that that sounds amazing because it's very complicated and it's a different thing to write something very, very simple, which sounds amazing even though it's
Sascha_Wolf:
Mm-hmm.
Nikola_Petrušić:
very simple.
Adi_Iyengar:
Yeah.
Sascha_Wolf:
Yeah. So how simple is the second system you've been working on, Nicola?
Nikola_Petrušić:
How simple is the second system? It's not super simple.
Sascha_Wolf:
I think he's laughing.
Nikola_Petrušić:
Yeah, I mean, yeah, we've been using commended, leveraging commended. And yeah, it has its ups and downs. Yeah, I mean, I'm still fairly certain everything can be done, everything can run smoothly. So I don't think there's any like major issues, but... Yeah, I would definitely consider the goal scale of the project before I start using Commended. It's not even Commended itself before I start using any framework that hasn't been really tested on such high-scale usage.
Adi_Iyengar:
Yeah, I think what I add to what, well, Nicola and I work in the same team, by the way, for people who didn't catch that, but I think one thing Nicola said, like his confidence in the system is high, and that's the same for me. And I think that, for me, the reason is, I think, literally, I didn't create that, I've not worked in a team where every single engineer is so proactive about design and engineering issues. There isn't really an engineer, there isn't a blind spot in the application, you know, that part of the application that's owned by a specific engineer who is not proactively, you know, finding ways to make the application better, not just doing the job, but also thinking through how the application will work. And I think even though the application we work in is not perfect by any means, there's a lot of work to do. But I think just that quality of proactive engineering is like a huge reason why I feel confident in the application sustaining a good amount of load and stuff in the future. I think that's also a key thing. On top of architecture, it's also team building, building a culture within your team where people can feel confident in bringing ideas and discussing it with the rest of the team.
Nikola_Petrušić:
Yeah, I agree. It's very important to feel like you can suggest something or say something. Say you think you don't think something is very, very well made or things like that. I don't think you really say anything bad about the things that you want to improve or the person who made it before you. Yeah, everything that's been done before has been done in some other circumstances than you're looking at right now. So I don't think that's why people should encourage it. If I say, look, there's this part of the code and it sucks, that doesn't really mean I mean I'm saying, look, the person owning this part of the code sucks at what they do. It's more like... Yeah, the probably we made this call a part of the call like five years ago, two years ago, whatever and our use case has dramatically changed since then or our load has dramatically changed since then or Our style has dramatically any other reason is fine as well. So yeah, it's very important for people to To be able to express their concerns their suggestions and stuff
Sascha_Wolf:
Yeah, especially when it's like an older piece of code. Honestly, I think it should, it's kind of a compliment to it, right? Like the older it is, it's more of a, well, this thing has served as well for what? Six years now and circumstances have changed and change is really the only constant in an, in an, in an actively used system. So now it's time to say goodbye, right? I mean, the place we're working at, we are slowly getting rid of a Ruby monolith. Honestly, it's not even a very bad RWBY Monolith. I've seen a lot worse RWBY Monoliths than this. It's comparatively tame in its Monolith-y-ness, let's say that. But still, I mean, the thing has been around for seven-ish years now, and it's been becoming a clutch in the past few years. It's becoming like a stone, kind of. You kind of have to carry with us all the time. But it's still a testament to, well, that thing has been shipping business value. It's still shipping business value, but times have changed. And let's get rid of that thing and let's replace it with something where we can now make a more informed decision of what we need at the moment. And honestly, I've been working with one of my colleagues at the current company. I really adored this guy because he has like zero ego, like absolutely no ego. So when I tell, like when I basically, when we go through these and like architecture discussions, we realize, hey, that that thing over there, that's honestly like, it's not really serving as well. He's like, yeah, it makes sense. You're right. Let's get rid of it. Even though he wrote that thing himself, you know, like he's been the guy who built most of those things. And he's still, well, okay, let's reduce complexity. I feel like he kind of had to learn it the hard way that like a lot of complexity is not a good thing, but quite the opposite. So now he's very open. to simplifying things and having an environment where, and having a team where being able to talk about those things openly. I mean, it gets also a matter of language, how you articulate those things, but in general being around people which don't really have a lot of ego in the things they do, but maybe mostly driven by pragmatism. That is, it's kind of liberating to be honest.
Adi_Iyengar:
Totally.
Sascha_Wolf:
Maybe you sometimes hear these horror stories and honestly, luckily I've never had to experience it myself, but like from these people which are like you change a line of code and then somebody comes around the corner like, why did you change my code? Right? I mean, I've heard stories like that. I've luckily never seen it happen. I've never had to experience it, but
Adi_Iyengar:
Oh, I
Sascha_Wolf:
oh
Adi_Iyengar:
have
Sascha_Wolf:
boy.
Adi_Iyengar:
had to experience it. I mean, I can talk about block fires, bankrupt. They're not going to sue me. But yeah, I have stories over there. A lot of people in the team I was in were really awesome. But I think teams outside, yeah, I have plenty examples, Sasha, where you're talking about. I think ego does come in the way. People associating their work with their self-respect. And if you critique their work, you're critiquing them even though you're trying to phrase it in the best way possible. Yeah, that's a big hurdle generally for any kind of progress or any kind of improvement in code. So yeah, glad you have an experience with Sasha.
Allen_Wyma:
I usually only get upset when they change my code into something very stupid. That's usually what upsets me, which you see sometimes, right? People don't, sometimes it's that they don't understand why code's written that way. Maybe that's a problem that we should have written, wrote, we should have written something there to make it clear, right? Why it's like this. And sometimes it's just, I don't know, I know, is ignorance the right word to use? They just don't know any better. So it just happens. But yeah, sometimes I just look at, what is that? Why would you do that? And I think we have a conversation with them. What happened over here? And sometimes
Nikola_Petrušić:
Yeah, actually.
Allen_Wyma:
I find out because there's a bug, because that happened. Sorry, go ahead.
Nikola_Petrušić:
No, I actually just wanted to, yeah, I had a very, very similar situation to this. I was working on some code and there was this line, like it was in the whole module, like repeating in every function at the end. There was this, it was like a test module actually. There were, there was this line and I just couldn't understand. It was like re-rendering something for no reason or something like that. And at first I just went ahead and like removed it from everywhere because I would use... What I would usually do is ask someone who wrote it for an explanation, but the case here was that it was written by a colleague that left the company long ago. And yeah, I removed it, but just in case I asked another colleague and a few hours later got back that that would have been a really dumb move on my part. So yeah.
Sascha_Wolf:
I hope you edit a comment explaining why that line makes sense.
Adi_Iyengar:
Hope you had a test that breaks if you remove that line.
Sascha_Wolf:
Oh, that, even better. A test would also makes it clear why that line is there. That's the only worthwhile use for comments explaining why and not what, right? Because sometimes you have to write really weird code. When you, often when you have to work around an issue, like an dependency or something like that, like where there is something you're calling into, which does odd stuff because there's a bug or something like that. And then you have to write this really odd. looking cohort which takes care of that. And that's the only circumstance in which I actually had a common experience. I have to do it this way because of that. Because there's this issue over there. I mean, for example, recently we, opposite again, like we, right, we did a minor version upgrade to a dependency, like our HTTP client, and that actually broke our HTTP integration. And as it turns out, it's just that, that was, it was a Tesla, I think, Alexey, I think it was Tesla. The compression middleware of Tesla that they didn't add the accept compression header before that, and like in the minor version, they added that. And as it turned out, like the, it was Google PubSub API we were integrating with. I don't know exactly where the issue is lying there, like who's at fault, but basically what the compression that was returned by it, Tesla middleware was not able to decompress it. We didn't really dig into it deeply because well, we had actually an instant active internet because of that, right? So we just, okay, from the scale we're having right now, let's just turn off compression for the moment. I mean, it's taking what? Like a few hundred messages per day. Eh, it doesn't really matter, you know? But that was the case. We said, disabled it, we commented it out, and voted for comment. Well, this is disabled because actually, they... Google pops up and Tesla do not agree here. And we don't have the time to figure it out right now. It's on the backlog now to actually fix it properly, but might've even happened by now. But yeah, those are the moments where you have to write some odd looking code sometimes to solve an immediate problem at hand.
Adi_Iyengar:
Do we want to transition to PIX?
Sascha_Wolf:
Yeah, sounds about right. So, Nikola, before we do that, if people want to reach out, I guess maybe Adi, also you, I guess, in this case, how can they get in touch?
Nikola_Petrušić:
If people want to reach out, they can reach me on my email or through LinkedIn.
Sascha_Wolf:
old school.
Nikola_Petrušić:
Yeah, I guess I do have an Instagram, but it's mostly like clips of me playing guitar. So if anyone wants to reach out for that, they can reach out to my Instagram.
Adi_Iyengar:
I need a link to that.
Sascha_Wolf:
So what's your email? What's your LinkedIn? What's
Nikola_Petrušić:
Ah,
Sascha_Wolf:
your
Nikola_Petrušić:
yeah.
Sascha_Wolf:
Instagram?
Nikola_Petrušić:
So, email is nicola.petruzsic with regular letters without the accents above. 221 at gmail.com. And yeah, my LinkedIn is Nikola Petrović and I can send Instagram to Adi privately. I don't think the viewers want the guitar playing.
Sascha_Wolf:
Okay, then let us transition to picks. So, Alan, you've been very quiet, so I'm just gonna put you on the spot here. What are your picks?
Allen_Wyma:
So I just have one old game that I just started playing just before we started the podcast. And it's called Full Throttle. I don't know if you guys ever played that game.
Sascha_Wolf:
What
Allen_Wyma:
It's
Sascha_Wolf:
was
Allen_Wyma:
basically
Sascha_Wolf:
the title?
Allen_Wyma:
like, it's called Full Throttle. So I'll
Sascha_Wolf:
No.
Allen_Wyma:
drop into the chat. But basically, I don't know how you even say it. It's just kind of like a point and click kind of game to a certain extent. It's made by LucasArts. And it's been remastered. And now it's just kind of. You can flip through like the old style and the new style, like everything's been remastered. And I thought it's kind of cool to see how they updated because this is a game I used to play way long time ago. So I was just blasting my way through about half of it in about half hour, just because it's kind of like a what do you call that? You know, something from your childhood, right? So yeah, it's just a fun game. If you grew up in with Windows 95, I think you may enjoy it. It's a very simple game. What kind of fun?
Sascha_Wolf:
It's from Double Fine, Double Fine does weird stuff, but like in a good way.
Allen_Wyma:
Sounds like you like that.
Sascha_Wolf:
Adi, Adi, what are your picks?
Adi_Iyengar:
I guess I have a video game pick. I've been playing a bit of Days Gone. I have avoided this game for a while just because of reviews. And I just realized what I've been missing. It's actually a pretty good game with all the bugs, most of the bugs resolved. And it's available for, I think, $10 on the PS2, at least. So I think it's a pretty good bargain. Still looks great on PS5. So if someone wants to play that, So it's a PlayStation but a person wants to play that game. They should definitely give it a try Have a couple jobs Well, if you want to work with Nikola He's pretty awesome and me. I'm not as awesome, but I'm still I'm still okay our team specifically at the score is looking for senior engineer Along with a bunch of other jobs discourse So I'll leave a link in the description, but you guys can feel free to reach out to me, too If you have any questions about the company So yeah Yeah, we're still hiring a few engineers. Yeah, a couple of people who reached out to me last, like three weeks ago, they're still in the interview process. The process is still ongoing, just FYI. And as I mentioned last week, still looking for a founding engineer for a really, really awesome startup. The startup, I can't speak much there in stealth mode, but they are commercial real estate, commercial estate management tool. completely in Elixir. I built that application, 100% code coverage. It'd be a lot of fun to work in that. They are already raising a good amount of money. Founding engineers have a good amount of equity. I don't know if you love Elixir, you're a startup person, you want to work in a high likelihood successful startup, early phase startup, it's perfect fit. So reach out if that sounds exciting.
Sascha_Wolf:
Nice. If you, I mean, if I would be looking for a job, then maybe I would even consider. I have three picks today. I'm going to start with a tame one. I don't think I've picked that particular book before it, because we kind of cut into like, how do you get your separations and your segregations and your responsibilities in microservices or honesty and any kind of code base, right? I sincerely believe that domain-driven design is a very effective tool at that. Recently, there has been a book published which kind of is now the de facto default for getting into domain-driven design. It is called Learning Domain-Driven Design, very aptly named. It's written by Vlad Kononov, probably butchered that name. And from... I honestly, I have to give you a disclaimer here. I have not read it myself, but I am a colleague which was starting to look into domain-different-design, he has read it and it's also been mentioned on the Kandinsky, which is like a DDD conference in Germany. Um, as like, now really if you want to get into domain-different-design, that's the book to start. Um, in the past there honestly was not really a deep book to do that. There were like multiple little different things you could look into. So it's really nice now that you actually have this one book you can point to and say, hey, you want to learn about Mind of Indesign? That's the book you want to check out. So learning to Mind of Indesign, it's published by O'Reilly. And the second pick is a repick. I'm going to do a repick now because I just finished that game. I just beat that game yesterday. It's Sifu. It's the Bruce Lee movie as a game. And honestly, it's really nice. Like it's, it's. You really could feel that the studio knew what they wanted to deliver and that is exactly what the game is about. It's like kind of this playable revenge Bruce Lee movie, like Kung Fu as a game. It's really hard. Like the final boss. Good gosh, he slapped my butt a few times. But it's just a pleasure to play, honestly. Like it's just so slick in the way it does combat and the slick in the way it does this Kung Fu emulation nearly. It's a very fun game to play. So if you haven't played it yet and you like deep combat systems, really check it out. It's not easy, but it's a lot of fun.
Adi_Iyengar:
Is it like Dark Souls hard or is it like Sikiro
Sascha_Wolf:
It's like dark...
Adi_Iyengar:
hard?
Sascha_Wolf:
I haven't played Sekiro, so I don't know. I would assume it's more of a Sekiro spectrum, because it's a very fast game.
Adi_Iyengar:
Gotcha, nice.
Sascha_Wolf:
So yeah, I really enjoyed it.
Adi_Iyengar:
Awesome. So I don't have it in me to do a Dark Souls-like game right now, but yeah. I mean, if it's like Sekiro, then I'll give it a try. That's awesome.
Sascha_Wolf:
I can really recommend it. It's a lot of fun. And then the last but not least, it's like a super weird pick for this week. Regular listeners to the show know that I'm also into tabletop role-playing games. And one of my favorite role-playing games is Blades in the Dark. And I've recently discovered a hack and I'm also going to hack off it. And the hack for everybody who's not into role-playing games is basically that when you take a rule system and you do slightly change it to like, tell a different kind of story, right? And I discovered a hack to... play over the edge basically. So you can play little critters and which go out into the evil little city from humans trying to steal food. That's the premise. I'm about to run it for some friends and it's honestly it's a silly fun just playing these little kind of critters and which are... defying humanity's boot and stealing the tasty food. So yeah, Eat Trash Be Free, that is what it's called. It's on itch.io, it's only a few bucks. You need the Blades in the Dark system to run it. Like it's really just a kind of re-flavor and a re-skin with like some slightly different moves, but it's a lot of fun, Eat Trash Be Free. And I just see Adi, Adi Bordroot, the game I recommended I think last time, right?
Adi_Iyengar:
Yeah, about to play this weekend. It's artist really cool, but
Sascha_Wolf:
Yeah, it's amazing.
Adi_Iyengar:
very excited to try it out.
Sascha_Wolf:
Okay, Nicolas, order your picks.
Nikola_Petrušić:
Yeah, so I have two picks, I guess. So first for anyone starting or wanting to start Elixir, I'm sure it's been picked on the show before, but I would very much recommend picking up Elixir in action from Sascha Juric, a great stand-up guy who wrote an amazing book. It got me started and I'm sure it can get anyone started. And yeah, that was the useful pick and now the video game picks since everyone had one. I recently played the new God of War Ragnarok, no news there, it's one of the best games I've ever played. The story is amazing, it's really hard on the hardest difficulties, so it's really, I would really recommend it to anyone who can play it.
Adi_Iyengar:
Yeah, it's amazing actually to add it is actually really hard and the hardest difficulty unlike the 2018 God of War which was not that hard and the hardest difficulty. Yes, so yeah plus one to that pick Nikla. Great game.
Sascha_Wolf:
Nice. Okay, Nikola, it was a pleasure talking to you. It was a pleasure having you.
Nikola_Petrušić:
Yeah, it was a great coming on. Thanks for inviting me. Thanks for having me.
Sascha_Wolf:
And I hope all of you enjoyed listening to us rambling about all kinds of things. And I hope you tune in next time when we have another episode of ElecCimics. Bye.