Building Applications with Flexibility and Scalability in Mind - EMx 227

Adi, Allena, and Sascha join this week's panelist episode. They dive into the world of engineering approaches for startups and delve into the ongoing debate of whether software engineering is really engineering. They discuss a unique structure for building a big elixir application, where separate bound contexts are responsible for their own supervision trees. They share their insights on the benefits and challenges of this approach, exploring the balance between pragmatism and forward-thinking. From discussing microservices to exploring new programming languages and patterns, there is plenty of valuable information for developers of all levels.

Hosted by:

Adi Iyengar •

Allen Wyma •

Sascha Wolf

RSS Spotify Apple Podcasts YouTube Amazon Music

Show Notes

Picks

Transcript

Sascha Wolf (00:01.366)

Hey everyone, welcome to another episode of Elixir Mix. This week on the panel we have Adi Angar, Alan Weimar.

Adi (00:06.592)

Hello.

Allen (00:09.496)

Hello.

Sascha Wolf (00:10.642)

and me, your host, Sascha Wolf. And we have a special guest. No, we don't have a special guest. I'm lying. Uh, this week is the panelist episode and I'm coming to you this week, or rather I'm coming to Adi and Ellen, um, to, for some butt kicking because we recently had a discussion at work and we want to try out a very, I would say unconventional structure for building our big Elixir application. And that is the topic of today's podcast. And I'm looking forward to hearing Adi.

Yeah, tell me why I'm wrong. So, um, to get you, to give you a gist folks, the idea is what we're currently building and if you've been listening to the show for a while, you might remember an episode we had a while ago on poncho apps and umbrella apps and all that kind of thing. And that was also driven by a discussion we had at work at my job and long story short at work, we decided to build one single OTP application and inside of that OTP application.

we have separate what we call bound context that's kind of following the DDD teachings and each of those bound context is responsible for its own supervision tree. So basically, in the application file, we start up the supervisors of the distinct bound context supervisors and then in there that's implementation detail whatever they need they start. Something that came up recently in the discussion is how we when want to structure a code that

we talks to the outside world, so to speak, right? Because we have our business logic that is all nice and dandy and followers basically usually the usual Elixir conventions. But we also have like event buses. We have like a thing that's talking to like an external API, kind of going doing an integration through that, and so on and so forth. And what we agreed on is for each bounded context is now to have something we call a ports folder. And if you're familiar with hexagonal architecture, I think it's a...

terminology and from there, or only an architecture or clean architecture. Honestly, I can't really, they're all like the same in my head, but having a ports folder for each bounded context and inside of that ports folder have all the stuff that then actually talks to the outside world. And that includes, like I said, event buses that also includes an actor repos. But, and now the comes to be a very controversial thing. Uh, the decision we made is to also move any kind of web logic in there.

Sascha Wolf (02:32.33)

So any web controllers, any Phoenix views, although they are no longer called views, right? They are now called, what are they called? No, not templates, formats. They are now called formats. And yeah, any plugs that are specific to that particular bounded context, all that stuff would then live inside of that Pulse folder, port slash web. And we have, on the higher level, we have the actual Phoenix endpoints.

and a router there. And from there, we delegate to either routers or the controllers directly of those bounded context. I would presume that we would actually end up having separate routers in that structure. And just to give a heads up, we haven't done it yet. The decision was just made. And honestly, I like it because so far we already had inside of the folders for the bounded context, everything that belonged to the bounded context, including a readme, like every bounded context has a readme laying out the basic structure of the thing.

And we have all the event handlers in there. We have anything you kind of need to work with that bound context is inside of that folder, except for the controllers and all the API stuff. That was in a separate folder, like higher up web. And there we have two endpoints, piece endpoints, we have one for an internal API, only reachable for like some internal applications. And we have one for public API, but that's honestly detailed and doesn't really matter. But yeah, that is the structure we're going with. And...

The idea is also to maybe have a bit more structure in terms of not only having a supervisor, but maybe having something akin to an application module for each Boundary Context. Probably we're going to call it a context module that has some callbacks we expect it to have. That is a bit vague at the moment. But basically, the idea is that each Boundary Context is a unit of its own. And that is a thing we want to be able to, in the midterm, to also be able to ship releases and to store

decide, OK, this release might, for example, actually start these three bounded contexts over there, while that release over there only starts this one bounded context, maybe for scaling reasons and other topics. So yeah, that is the direction we're going with. And I'm curious to hear what you, Adi, and Alan think about that.

Adi (04:44.355)

Yeah, it's interesting. I have so many questions. I guess a good place to start would be, what was the primary motivation to go this route instead of an umbrella application with separate deployments or completely separate applications and just talk through HTTP? What was the motivation to do? What's the advantage of going this route versus either of those?

Sascha Wolf (05:11.506)

Yeah, that's a very good question. And I, in a nutshell, the answer is complexity. Um, because in our company, we are a team of four backend engineers. So the amount of people that actually need to be able to maintain the system is relatively low and the amount of changes we perform on that system where nobody else is involved and like where you need to decouple yourself from other people is also like everybody in the team can on a high level grok the entire system.

But at the same time, we are split up into separate smaller product teams. So we do have the desire and the need to actually be able to separate concerns a bit. But we honestly don't really meet the complexity overhead of having, for example, separate deployable microservices because it's more work. It's more flexibility, yes, but it's also more work. And at the scale where we are at, of four engineers working on this project, this seems to be the sweet spot.

And like one reason why we are going for this structure is also in the back of our head. Like if we ever decide to say, you know what, that thing over there actually has diverged from the needs of the remaining application so much. Let's cut it out and put it into its own application. But because we already from the get go kind of went with this, okay, this is a self-contained unit, right? Like that concludes everything that needs to be started in the supervisor that in theory, should be relatively straightforward.

And so yeah, does that answer your question?

Adi (06:41.719)

Yeah, follow up. So I guess to understand this correctly, you did not go separate umbrella apps and separate microservices because you didn't want to put the complexity and deployments and the whole infrastructure side. But you also did not want easily accessible functions between contexts. That's why you chose to go through message passing as a route.

and instead of just calling the functions directly from each.

Sascha Wolf (07:12.626)

Sometimes we do. I mean, it's a case by case basis. Like sometimes it's message parsing. Sometimes it's calling explicitly. And so it's a use case topic, honestly. Like it depends on the specifics of a use case. But like all of the interaction that is directly function calling always goes through a top level module. Yeah.

Adi (07:20.828)

Interesting.

Adi (07:33.951)

That makes sense. So why, okay, I guess I should have asked this question earlier. When you start a bounded context in a supervisor, what exactly are you starting then, if you can just call the margin and the function?

Sascha Wolf (07:48.97)

In the supervisors, it's mostly event handlers. That's currently what's mostly in the actor repos, that kind of thing. For we have one bounded context that is basic. I mean, we are building an app that's also delivering content to end users, right? But we are not building the content management system ourselves, we're integrating with a third party provider. So we also have one bounded context that is kind of providing like a nice interface to the rest of the application. And that under the hood, that also starts a Finch pool.

Adi (08:12.607)

Gun it.

Sascha Wolf (08:16.482)

that is connected to that software service provider, right? Like that kind of shit.

Adi (08:18.96)

Oh, got it.

Got it, got it, got it. I should have clarified earlier. So I thought when you said you start bounded context as well, that means the process that's responsible for communication. But no, it's just a module and anything related to that bounded context, you start over there. OK, got it. So the process isn't overhead. Everything doesn't go through a process. OK, that makes sense. OK.

Sascha Wolf (08:42.916)

No.

Adi (08:47.203)

Yeah, I think that's like my high level questions. I mean, I have more, but Alan, do you have any?

Allen (08:54.954)

No, I'm still trying to see like I don't see a big deal with it. If it works better for you, then, you know, seems fine for me. And I guess if you have an issue like with architecture, obviously you can benchmark that and see where the problems are. I mean, everybody knows that processes can get bogged down with messages. Right. Yeah, I mean, have you done any kinds of checking to see, you know, like how does it perform? Does it perform well? Have you had to reverse some of your choices?

Sascha Wolf (09:23.654)

I mean, just as a reminder here, like this is not necessarily all about individual processes talking about each other. This is more about code structure and like distributing responsibility, right? Not every bound context is one process, but every bound context manages its own supervision tree, which might include sometimes nothing. I think we have one bound context which literally starts nothing, but...

Adi (09:43.707)

Yeah. I actually think this is relatively not that uncommon. Sasha, I have seen applications, other applications that the score does it too. I think it's a very good way of, so just to be clear, you just have a supervisor per bounded context, right? That's basically like a context of process per bounded context that you put in the overall supervision tree.

Sascha Wolf (10:01.549)

Yes.

Adi (10:07.411)

good thing about that is you can turn it on and off with environment variables or whatever configurations. And eventually, when you're like, oh, I want to scale individual context separately, all you can do is turn off a switch and literally deploy different deployments without really complicating your infrastructure. That's actually, I think it's closer to, what's the word, more mainstream, believe it or not, for companies who anticipate...

Sascha Wolf (10:10.634)

Yeah, yeah, that's the idea.

Adi (10:38.511)

splitting into microservices, but not wanting to put that overhead. So yeah, initially when you mentioned modern context and supervised tree, I actually thought everything is process driven, but this is actually a lot more status quo for like a better word, yeah, cool.

Sascha Wolf (10:50.898)

No, no, no. But I think, I mean, what's been very odd for us in terms of feeling, feeling like we go against the mainstream here is also like moving anything that's rep-related in there, right? Like kind of making this its own self-contained thing. And like, when I talk through this, also talk with the team about this, and when I'm talking about this with you, it kind of sounds like the idea that was flown around a while ago by, gosh, I'm...

missing the name right now, but like there was a talk a few years back about iterating on the concept of applications and OTP, right? Like having components and having those be self-contained. Can you remind me who gave that talk? Because I can't think of it right now. You recently mentioned it, Alan.

Adi (11:33.124)

No idea.

Allen (11:39.478)

mention something that you enjoyed. I have to think about this for a second. Distribution of what world was it again?

Adi (11:49.855)

So like ability to use application as a notion and elixir in a more contextual way, right? So I shall like, yeah, using it as like a, yeah.

Sascha Wolf (12:01.718)

Yeah, I mean, it went a bit further in terms of also challenging what is an application in the first place, right? Like should we even... Dave Thomas, exactly. Yeah, it was with Dave Thomas talk. Yes.

Allen (12:02.911)

I'll come at some own.

Allen (12:09.998)

Oh, Dave Thomas. Yeah. Sorry, you're speaking so poetic. I didn't know what the heck you're talking about. OK, yeah. Challenging the norms and trying different architectural patterns and removing those stupid module files that just sit in your project.

Adi (12:11.363)

Oh, right. Oh. Oh, I see. That was a very small part of the talk, though.

Sascha Wolf (12:16.91)

I'm sorry.

Sascha Wolf (12:24.738)

By the way, should we just take a break here and cut this out? And then I'm going to remention the Dave Thomas talk. Basically, I'm going to repeat from the part where I said, talking for all of us, blah, blah. So hello, whoever listens to this needs to be edited.

Adi (12:32.651)

Sure.

Allen (12:33.77)

Yeah.

Sascha Wolf (12:47.646)

So what has been kind of interesting for us and like with the more controversial side is also moving all of web stuff inside, like having all of this, like the self-contained bounded context in there. And it actually reminded me of a talk I heard a few years back, and you also mentioned it recently by Dave Thomas. And like one part of that talk was about challenging OTP applications and how we build them in general and like making them more of like the self-contained components and I'm not sure how you see, but it feels like...

Like this structure we are now aiming for kind of goes into that direction.

Allen (13:24.818)

Yeah, I think you should. I mean, obviously, I enjoyed the talk from Dave Thomas. I'm not sure if everything made sense in his talk, but for sure, he had some interesting ideas. And I think that, yeah, we should try new things and see how they work, because, I mean, there's always new programming languages, right? How many do we have in the world? Like, thousands?

So there could be new patterns out there that we didn't yet discover that make more sense. And I think it also kind of makes sense, like you said. So when you start up your applications, they have their own supervision structure, right? So it's kind of like separate is what you're saying, rather than kind of combining into one app and then starting up the tree, right? That's somewhat common. It happens quite a bit. I've actually had an issue. Adi, remember a couple of months ago, I tried to meet with you to talk about something? That was because of that had its own.

Adi (14:15.466)

Yeah.

Allen (14:18.506)

startup structure actually. So it was like, even though I started up the calendar, what was that? The TZ data app. And then I started up the other one. It didn't quite register, right? So that made a problem. I'm kind of speaking a little bit unclear, but I had a problem where there was a library that started up a supervision tree that would do a refresh on certificates on a like interval.

Adi (14:32.712)

Yeah.

Allen (14:48.046)

And for some weird reason, even though we started up TZData before and we set TZData to be the calendar database in Elixir, it didn't see that. And it kept crashing all the time because it couldn't figure out what time zone it was in or something. So I forgot how I solved it. I think I manually set the database before I started up that thing. But yeah, actually, yeah, so.

Some apps actually do run like that, where you just kind of turn it on, and it just turns on like that and does its own thing. I mean, it is kind of nice, but then as you can see, it can also have a negative side, like what it had for me, where I couldn't quite handle that dependency because it had its own structure. I mean, what about that kind of issue, right? Where like basically that app actually does become its own app, and it has a problem, like how are you gonna solve that?

Allen (15:42.774)

Am I speaking too vague again, Sasha?

Adi (15:45.379)

No, I guess I can follow. I think that's the whole goal. Eventually, if one part of the supervision tree, in Sasha's case context, they want to separate that into its own app, you have already defined the dependencies of that application under that application, which is itself the supervisor.

So I think that's the whole idea. And I think your example is good, Alan, about applications and applications that have their own supervision tree. But Sasha is taking it to the extreme, where you have probably your own Ecto repos and maybe talking to the same database, different database. I don't know. And also your Phoenix or maybe whatever web server you're using or web framework you're using.

Sascha Wolf (16:28.402)

Yes.

Sascha Wolf (16:40.19)

It's Phoenix, yeah.

Adi (16:43.3)

OK, OK. So maybe the endpoint itself, different endpoints or router, depending on how you define to separate it, in their own context. So you'll probably have like, well, because it's endpoint, you'll have probably multiple ports when you actually start the application with everything turned on in the supervisor, right? Am I characterizing this correct, Sasha?

Sascha Wolf (17:03.338)

Yeah, we're trying to hit like the balance there, right? Because you could kind of go down that route of actually having one endpoint for each bounded context, which all of those are different ports. That is something we deliberately decided against because we don't need it. That is the thing. We don't need it. That is kind of like where all of this structure in general comes from. It's informed by...

by making decisions that make the least amount of assumptions on how this thing is going to evolve in the midterm, but giving us still some flexibility and some options to distinguish. So in this particular case, we actually do have two endpoints, but that is driven by kind of, yeah, by maybe not business needs, but by infrastructure needs, because some of our endpoints are purely internal. Like we have an admin interface that is communicating inside of a Kubernetes cluster, right?

So one of our Phoenix endpoints is only reachable from inside of a cluster. And it's an internal API, basically an admin API. And we have a separate endpoint, which is actually exposed to the internet. And at the moment, the configuration is basically the same, but there are some slight differences, right? Like for example, we have like all of the JWT and authentication stuff really only happens for the public API because well, yeah, it was an admin API. You first need to get access to the app in the panel in the first place.

But those are really pragmatic choices, I would say, and things that are driven by, okay, what is the status quo and what makes the most sense of the structure in a way. And we try to avoid doing structural decisions and things because, oh, it seems like the next logical step, like doing things just for doing them for the sake, right? Like we try to avoid that pitfall.

And honestly, I'm very grateful for the team I'm in because sometimes I can get my head stuck into clouds a little bit about like, okay, what kind of structure does make sense? Again, I mean, the podcast we had a while ago about Umbrella apps, poncho apps. I'm personally fond of the idea of poncho apps. I think they have the benefits and the attractive. But that was also the proposal I went forward with to the team and the team very rightfully questioned, honestly.

Sascha Wolf (19:22.774)

Do we need it? And yeah, honestly, we don't. Maybe we do in the midterm, but the way we're going forward now with one application and structuring this way, we can still do it. We can still reach for it. And we have a very good balance of people that are very pragmatic-driven and folks that like to think about the mid and the long term.

Adi (19:47.067)

That makes sense. I guess one more question, just so I can see where you guys may be going with this more. Do you have the same database and the same repo, or same database, different repos for each context? Yeah.

Sascha Wolf (20:01.194)

We have a same, we have different repos for each bundle context. And actually each of the repos also points to a different database that is in the same cluster. It's the same that preposterous database instance, kind of, but it has, all of us have separate databases.

Adi (20:16.175)

Gotcha. OK. That's very cool. So you're already building each context in a way. I mean, you call it context, so it makes sense. But you're already building each context in a way that's like you're not relying on transactional safety and stuff like that. Which, because of how people define context, what I'm using air quotes, context in Phoenix, that's not necessarily usually the case. People do add that the transactional safety is

Sascha Wolf (20:32.606)

Exactly, yeah.

Sascha Wolf (20:41.191)

Yeah, exactly.

Adi (20:45.275)

across context and they have trouble separating them in databases or even separate repos, right? So that makes sense. Yeah, I think building something like that from the very start, I think is, I actually think it's very, yeah, I think it's going to solve a lot of problems down the road when you do separate. Yeah, I think you probably will have to over-engineer early on a little bit more, obviously, right? It's always a trade-off. Yeah. So...

Sascha Wolf (21:08.563)

Yeah we did.

Adi (21:10.863)

And we'll probably still have to, even building initial features, right, as the application is building. You could, again, transaction is like one of the examples. You might have to like hack up some kind of sagas or something, or 2PC, just between, even if it's within the same even process, you might have to do that. But I think if you do anticipate the application needing to be separated very soon, or, you know, in the near future, two years or so.

then I think it's definitely good. Yeah, I think it's a fair price to pay.

Sascha Wolf (21:44.106)

I would even, I mean, that is one thing I would honestly challenge in that I think it's worthwhile to go down that road, even if you don't expect it to be separated. And because they're like something we already see now, and I mean, we only started building this what, like half a year ago, right? Something we already see now is that like with some of the conventions we're having with, okay, if you reach outside of your bounded context, then you should call like top level APIs only, right?

And the level of separation we have there is, it's just, it's very nice to work with. For example, we have like one part of the application that is integrating with a payment provider, like because we call it subscription based service. And that thing that takes care of all the integrity details about, okay, that has a web hooks, it tracks what kind of subscriptions people have. If they're in a free trial or not. But on the top level, that thing has an API, which is called, does this, this is user premium. And.

Like that part of the system is something I'm not very deeply familiar with because it's a responsibility of another product team. But I do have the need in my part of the system to say, okay, some of the features here that we behave differently on Revu or not, the user has premium access. And because we kind of had this upfront investment about like figuring out what are the cuts and what are the contexts we want to have and that have this up.

this agreement on like only using top level APIs. And it becomes a whole lot easier to reason about a part of the system without having to know the internal details of another one. And in theory, of course, nobody would stop us from saying I'm going to reach into that bounded context over there and like load data from a database directly. Um, there's no hard separation between that, but again, because we are a small enough team, that's not a big problem. So.

Maybe circling back to a question at the beginning is, I would, this structure is good for us where we are right now, and also the size where we are right now. I would never argue, and this is going out to you, dear listeners, this is not the best structure you can go with. It really always depends. But for the company size where we're at and for the product size we're at, it seems so far to hit a sweet spot.

Adi (24:07.407)

Yeah, and I think it's actually, like we discussed, I think it's like a very, you know, like zero to 100 life cycle of an app. Let's just imagine, you know, like a x-axis zero to 100. This is like, I think, probably a solution that can sustain you from like, you know, almost zero.

to 80. Like we talked about, you could use simple flags to turn different contexts on and off, scale them independently, manage them independently, trace them independently. It allows you to do all of that. Yes, you load the entire application in the build, you compile the entire application for every worker, for lack of a better word. But yeah, I think...

Sascha Wolf (24:32.214)

Yep, yep.

Adi (24:54.867)

That's the only flaw that I can think of right now. But it can definitely sustain you for a while. I think it's a pretty good architecture.

Sascha Wolf (25:01.618)

Yeah. And honestly, like, I mean, like if we wanted to, that would require a bit more investment. But like, I mean, you have the mix file in theory, we could go down and say, hey, you know what, depending on Flex, we only compile parts of the application. But yeah, I, I honestly don't think it's a good idea because, uh, yeah, it can become very difficult very quickly. But that is what I, what I really like about this Verge tech, about the Oling VM and about Lixi in general.

Adi (25:14.543)

Ooh, that's a tricky one to do. I've tried that. Ha ha.

Sascha Wolf (25:30.87)

Like those tools are available to you. And like, I know that a lot of people are very fond of Go for example, because it's a simple language to pick up. Me personally, there's like one of the design principles behind Go, which is basically, it's designed to be easy to pick up and it's designed to be easy to grok, but it also has this mindset where there are tools available to the language.

builders of Go, like in generic, I think generics now is a thing in the latest Go version, but for a very, very longest time you didn't have generics in Go, but like in some parts of the language you did. And that was like a tool that was exclusively reserved for the language designers. But it was never available in user land. The Elixir is literally the opposite of that, right? Elixir is, you know what, we have a very relatively small kernel.

And everything that kind of is part of a core language is built on top of that. And it's just available in user and you can use all of that yourself, right? Like with again, being an example of you could build the riff macro yourself if you wanted to.

So yeah, basically saying, I like the flexibility we get here in the platform. And I like that we are able to make this decision at this structure right now. And that the platform is not getting in our way of choosing it to go down a different route in the long run.

Adi (26:51.079)

I do think I think Go supervision is a little bit more configurable, more manageable. I think it's going to be done. I mean, obviously the whole website, which is like very not developed on Go, right? Like not even goes to Phoenix. But yeah, I think there is like the state management between processes. So I think this is very much doable in Go. Actually, it's one of my picks is going to be a Go pick today. But yeah.

Sascha Wolf (27:23.134)

I mean, like I said, I understand why people enjoy the ecosystem and the language. It's, I just, I honestly, like if I would have to put it bluntly, I feel like ghost, presuming I'm stupid. I don't like it.

Sascha Wolf (27:41.346)

But yeah, I'm actually curious, Adi, because this is the decisions we've been making in our team. And you've been part of a whole bunch of startups also, and of beginning new projects. We talked about it a few times in the past, but the road now we are going down is, what would you do differently? If you could be in a position where we are at, would you say, you know, I would do it exactly like you do, or is there something you would do different?

Adi (28:12.255)

I mean, it's hard to know without knowing all the constraints, especially the business side of things, right? And that's what Alan said right away early on, right? No, nothing is ideal. Your solution probably works best for you, right? Like, you know, it's very kind of depends on your use case. I haven't seen a startup that I advise that is going that approach. And that's probably because they don't have the luxury to.

Sascha Wolf (28:17.299)

Mm-hmm. Yeah, that's true.

Adi (28:39.323)

invest that much time into thinking about engineering architecture. Like it's, you know, startup, of the ones I advise, there's only one that's like mature enough to be post-series A. They're very, you know, early stage, they're like scrambling every day. They deal with scale as it comes, right? So yeah, it's very, I think the architecture is decent in those startups, but like,

They haven't had the luxury to put that much thought into building an application from scratch. It's usually like one of the startups had to build a generative AI application in three weeks, operating at a scale of hundreds of thousands of requests every 15 minutes. So they don't really have too much luxury to spend weeks thinking about that. I do anticipate them reaching to a same

point that you guys have, right? Because it's not that much effort, assuming you've done a few things correctly. It's not that much, it's not a crazy amount of effort, like separating Microsoft, but it's not a crazy amount of effort to get to a place where you can conditionally start different applications as part of a container, right? Or how I deploy application as part of a build. So I do anticipate them reaching that point very soon, because I think...

Sascha Wolf (29:43.493)

Hahaha.

Adi (30:04.103)

as I mentioned, some of them are operating with a lot of scale. And having the luxury to have the option to conditionally scale each component independent of each other would be very important for them, because cost is also a big deal for them. They can't be scaling up everything 100x, right? So yeah, I think that's where I see a lot of value in this design, right? From my startup perspective, like that, you know,

Sascha Wolf (30:18.998)

Mm-mm.

Adi (30:32.387)

with very little work, literally with it. I mean, if the repos are also not even separated, if you have an application that's just like everything is roughly separated, but as long as it's in the supervision tree, with just adding dependencies to each application, you can literally conditionally start them and conditionally scale them. That just buys you a lot.

longer of a runway until you have to redesign your application. So I think you guys have likely hit a good spot without again without knowing all against me. That sounds like yeah.

Sascha Wolf (31:11.55)

Yeah, of course. It's also, it's also not just to be honest here, like it's all not flowers and rainbows, right? I mean, some of the big upfront investment we made was like how we kind of want to structure our event subscriptions and publishing. And at the moment it's a lot more coupled to Google PubSub than I would honestly like. So like the local startup story is a little bit finicky.

That is like one of the downsides, but again, because we do have this abstraction layer between the actual event subscribers and publishers and Google PubSub, but in that abstraction layer, there's like a bit of a leakage one way or another. So I would, there's the idea to like in the midterm for local, for running it locally to basically swap it out with Google, with Phoenix PubSub, right? Like to just use Phoenix PubSub locally, but when it's deployed to production, use Google PubSub, but we're not there yet. So like.

The local run story is a bit finicky at the moment. Same about configuration. Honestly, at an application this size, I haven't found a structure where configuration is easy to read and easy to maintain. It's all working, it's all there, but there is a fair bit of boilerplate between some of the configurations. For example, the actor repos.

And it's kind of hard to distinguish between, okay, this is the actual critical configuration over there and this is just the stuff that needs to be written somewhere. And that is a bit of on the uglier side, I would say. But all in all, the amount of pain we're having with this system is a lot less than what we had with the previous one. And I think this is also what you were getting at earlier, right? We are not a startup. We are a, what do you say it, a grow. I forgot what the business term for that is. But like a grow up.

And we are at a scale where we are a profitable company, and we are now starting a new value proposition with a new product. So we are in a very luxurious position where we obviously don't twiddle our thumbs and just do nothing. We still have a bit of a deadline ahead of us, but we don't have this financial pressure in the back of our necks that most startups have. So we do have the luxury of being able to invest a little bit more time into thinking, okay, how do we wanna structure this in the long run?

Sascha Wolf (33:27.318)

Yeah. And one of the nice side effects is already that the existing system that is shipping business value is still around, but we were able to reduce the complexity, for example, from the previous payment handling that was very much a home brune significantly, like the service is still around, but the last few pull requests were all removing more code than adding, and that is nice.

Adi (33:53.275)

I'm curious what is Adam smiling about.

Allen (33:56.97)

type what I wanted to say.

Sascha Wolf (33:59.458)

I'm just gonna read it out. I'm in the pre-scandal stage, Sasha. By the way, he miswrote Sasha. He wrote ZA-ZA-HA. Ha ha ha.

Allen (34:08.862)

I did that on purpose. Gold star for you. Huh? Well, you have the stage where you're struggling as a startup, and then once you get some money, then people start being bold and they get into scandals.

Sascha Wolf (34:12.826)

What do you mean? Why am I in a pretty scandal stage? You mean where-

Sascha Wolf (34:25.383)

Yeah. Actually, it turns out the company I'm working at is not really taking a big investment money. So like where we are growing slowly or more organically, we have a, we have bigger company in the background that is like financially backing us, but it's not like hedge fund money, you know?

Adi (34:34.147)

That's...

Adi (34:39.795)

That's a big reason why you're able to invest so much time. Yeah. That's like, I have not experienced a lot of closer to bootstrap company experience. But the only one experience I've had with a bootstrap company, that's where even if the revenue was getting very close to expenses, people still had more of a say. Where someone external coming and telling you what to do. But yeah.

Sascha Wolf (34:43.059)

Ha ha

Sascha Wolf (35:05.495)

Yeah.

Allen (35:14.326)

like four people right like how did this come up I mean because it wasn't your original idea you said you're looking at umbrellas you're looking at poncho projects and then you're now all of a sudden you have this other structure right this is somebody proposed it and you guys tried to build something small then it just kind of made sense and just kept going with it

Sascha Wolf (35:33.482)

I mean, this is kind of where, very specific to our situation, but in general, like the colleagues that I work with among the, what we call the backend squad, we have like product teams and we have basically what we call squads are the groups of engineers that belong to a particular stack. And what we have in backend squad are all engineers that feel comfortable with autonomy and ownership.

So we don't really have anybody which would be considered a backend lead. If you would want to give the title out, that would mostly go to me. But that means that, for example, the initial suggestion I came to them with like, okay, hey, you know what this is, what I think makes sense, having that as a puncture app. And that was exactly that. It was a suggestion and we had an open discussion about it for like an hour. And then the team decided, you know, I see your point of view, but the majority was for the

doing something simpler and having one particular application that kind of follows the same ideas but having that separation of concerns, but not having the complexity, the infrastructure and deployment complexity of having separate services. I did work, pretty much my previous employer actually, in a company where we went all in on microservices. It has its benefits.

But again, we were at a scale that was a bit bigger than what we had here, engineering-wise. We had, what was it, 10-ish, 12-ish developers. But again, we had a whole bunch of microservices, and that works good for building the system. But for example, there was one point where there was a critical security issue in our Docker-based images, basically. And that...

meant we had to go to every freaking repository, take the Docker file, update it in, blah, blah. And honestly, that's annoying. And that, like, the level of flexibility we got there from having invested separate deployables. I didn't, I don't feel it was worth the effort. I think, and I've said it in the past, and I'm going to say it again, a microservices is not really a pattern that solves a technical problem. It's a pattern that solves an organizational problem.

Sascha Wolf (37:51.154)

And the organizational problem is when you have an engineering team that is too big to really be like one or two teams, but rather you need to be able to decouple. Yeah, I see Adi shaking his head, but I'm gonna stand on that ground. I'm gonna die on this hill. When you want to be...

Adi (38:07.946)

Well, define what you mean by microservices. Can you do that?

Sascha Wolf (38:11.714)

I guess that could be a better term would be self-contained services. If you ask engineers for a definition of a microservices, you get like, if you have five engineers, you get six definitions. But let's maybe use the word self-contained services, like the run of their own, manageable, deployable, blah, blah. And that is something I feel is solving an organizational need.

Adi (38:24.335)

Yeah, yep, that's why I asked.

Sascha Wolf (38:39.054)

why you have separate teams that need to want to run and be iterate and be able to change things while not being constrained by changes from other teams. Of course, it's never achievable in the perfect world. But if you actually have your own code base, you can reign freely over. That's more achievable than if you have a big monorepo, which doesn't mean it's not impossible in that context, but there are more considerations you have to make. So yeah. Basically, the hill I'm going to die on

I don't think self-contained services make sense below a certain organization size. That is my perspective on it.

Allen (39:21.214)

Yeah, I just want to talk a little bit about microservices. I mean, I don't know. I've heard there was, I think, a microservices pandemic, I think, what, 10 years ago or so, where everybody's going crazy about this whole entire thing. I don't know. I had this project that I got given, or that my English is getting worse, this project that I was given responsibility for.

that I was told was microservices and CQRS and all this stuff. I don't think the guy could even spell CQRS when he was telling me what CQRS. But anyways, I was going to say, what I found out is that we didn't have a microservices-based architecture, but it was basically a distributed monolith. If one of those pieces ever went down,

The whole thing was fucked basically. That's basically what it was. And it was just horrible to debug. There was no telemetry, there's no tracing, there's no anything. And you know, like that whole microservices thing, like, I don't know. That project really changed my thought process about how to build stuff. And that's why I don't usually like to build like distributed broken up pieces and deploy them separately for that reason. It's such a pain to like.

turn on all the things, make sure they talk to each other through some kind of thing. And this kind of really makes sense, but to be honest, there's not a lot of services that can be deployed like separately, like self-contained services. Unless of course you architect it that way, which is the big problem is that people don't architect that way. Oh, I need authentication service. Oh, I need this service. And they all have to talk to each other. And if one goes down, then the chain breaks and nobody knows what's going on.

Sascha Wolf (40:54.899)

Yeah, yeah.

Sascha Wolf (41:07.294)

Yeah, that is also, I mean, like, um, at the end of the day, it's about knowledge, right? It's about knowledge and especially about knowing the domain you're in and about, um, using that knowledge to then determine what are the boundaries and what are the separate distinct things inside of it and domain. And I think that is my personal theory. I don't really have anything to back it up.

But from where I'm sitting, I think a lot of what we consider legacy and how to work systems come from either not really caring about boundaries in the first place or from presuming, okay, these are the boundaries that exist in our domain and that's the end all be all of wisdom. And the thing is, especially when you work on a new project or a new product, you know very little at the beginning about that particular domain.

And I mean, I think this is an experience most engineers can relate to when you start building something and you maybe have an idea in your head, this is how it's supposed to work. And then you actually start working on it and you realize, oh damn, this is a whole lot more complex than I thought it would be. There's a whole lot more nooks and crannies I need to consider. And honestly, like big picture architecture and big picture problem domains are not even similar. They are exactly the same, but exacerbated.

Um, so if you go all out on, for example, like self-contained services from the get go and cut your boundaries, the chances that you are right with those particular boundaries are honestly very slim. Um, unless you have like a whole slew of like insanely smart people working on this. And chances are that maybe some of those are right and some of those are subtly wrong. And some of those are probably going to be completely wrong. Um, but if you have like.

separate deployables and even maybe separate languages, then shifting those boundaries around is going to be freaking expensive. But if you have a single repository where you kind of follow the same principles of trying to have boundaries, but it's the same language, it's one application, in the beginning it just starts up at one big thing, it's still going to be expensive to shift those boundaries, but at least you don't have to jump through all of these hoops of potentially bridging technologies, of potentially moving a code from one repository to another.

Sascha Wolf (43:30.654)

It just removes friction. It gives you flexibility. And then if you actually continue to build your application this way, the certainty that you kind of figured your boundaries out correctly, and if you also have a process to iterate on that, that certainty is going to increase over time. And then again, Adi, what you said earlier, right? And at some point, you can say, you know what? We are relatively certain at this point that those boundaries make sense. The system kind of works.

And now maybe it's a good point in time because we also grow as a team to cut that out into really separate deployables. There is a principle I like to use very often and I found not that many people are very familiar with it. It's the last responsible moment principle. The last responsible moment principle basically dictates is that you make it that you should not make a decision until the last responsible moment because the idea is...

that if you wait for making a decision until the last responsible moment, then you have the maximum amount of knowledge available to you. Then you know the most things, and then you can make a decision that's hopefully, not always, but hopefully the right one or going into the right direction. But if you make premature decisions, if you make decisions before that, you make them with less knowledge than if you could. And that is, if you look at the structure of the strategy we're going with,

And then you look at that principle, it honestly basically all follows that principle. The whole idea of going this way is trying to make the least amount of decisions where possible and when we have to make the decisions to make them as late as possible.

Allen (45:13.774)

I think I followed that pattern of thinking when I proposed to my wife. I waited until the last possible moment when she forced me to do that. Then I made the best choice to live and proposed.

Sascha Wolf (45:28.852)

I honestly think it is a very powerful principle.

Allen (45:35.478)

Yeah, but honestly, I think I kind of follow the irresponsible one where like I made a choice that I knew everything and then I said, oh, shit, I missed a time. And I made a choice at a time when I knew it was the worst time, even though I knew everything. You know what I mean? Like, because you do have a limit, right? If you kind of go back to what it is, it's like, usually I end up making a choice after that time. So, yeah, I knew when I'm like, oh, wait a minute, I think I knew I should have made a choice before. So

Sascha Wolf (45:52.138)

Yeah.

Sascha Wolf (46:00.874)

Yeah, I mean, the danger there is like making a decision too late. And like what that entails honestly, that depends on the decision in terms of like engineering that often means that you might take the level of refactoring becomes more expensive you need to do. Because I mean, for example, if you make it look on a very small scale, like when it comes to the dry principle, right? Like don't repeat yourself. I try to, I guess I have a rule of thumb that I like one time repetition is okay.

Second time repetition may be also okay, depends on the context, but when it comes to the third time and repeating something, then I look into like what kind of abstractions we can have, but sometimes you realize, you know, what we should have just went with like an abstraction layer from the get-go and we had a good idea in the beginning, but you win some, you lose some. In general, I found following the principle to be more helpful than harmful. And that at the end of the day is like the hallmark of a good principle.

Allen (47:06.61)

Adi, do you follow that principle?

Adi (47:09.479)

Yeah, and pretty much everything.

Adi (47:15.155)

I think it's also similar to asking a lot of questions when you get like, whenever you do something, right? Like you will have the most amount of knowledge only after you collect knowledge, right? So it's also, yeah, before doing anything significant, the more significant decision you're making in product engineering or life, the more data and more information you should have about that decision, so.

Sascha Wolf (47:19.67)

Yeah.

Sascha Wolf (47:46.634)

Yeah. And I think it's interesting, like there's a whole slew of principles that you can kind of derive from that. Um, are you too familiar with like the walking skeleton? A walking skeleton is basically, it's more of a, like a shipping idea, but the walking skeleton is the minimum amount of things you can build to have, to ship a feature that is like all the moving parts are there, but it might be a walking skeleton, you know.

Adi (47:58.121)

No.

Adi (48:15.821)

I love the name, that's awesome.

Sascha Wolf (48:17.61)

So yeah, and then it's like, for example, like a principle I'm trying to like ingrain our engineering culture because in the past we had, there was a tendency to like, over commit to the scope of features, right? And to really only ship if everything was done. And then now, uh, something we're not trying to do is like ship things early and often, and like even if they're shitty, right? Um, for example, for the product we're building right now, there, there's meant to be some, some way of content suggestions, right? Like recommendations.

very first version we're shipping right now is already the structure of how we expect them to be and they're going to appear in the app, but they're literally going to be hard coded in the backend for everybody because then at least the thing is already there. We can iterate on the next iteration is probably going to be to put the suggestions in our content management system, just say, hey, somebody swaps them out. But at that point already...

The feature is there, right? We can see, do people interact with those suggestions? How do they interact with that? And based on that, can make decisions on how we want to build the thing in the future. Do we want to have some kind of recommendation engine? How specifically, what kind of content pieces do people actually click on? And give us more information and more knowledge to make the right call.

Yeah. So if you would think of it from that day, the walking skeleton is kind of like one way of like going for this last response at the moment, like how you can implement it. But yeah, I'm also very fond of the mental image.

Allen (49:48.238)

All right, can you define the difference between walking skeleton, prototype, or MVP? Because I'm a little bit confused.

Sascha Wolf (49:57.707)

I think that MVP, from my understanding, MVP is like a bigger scope than a walking skeleton. Like an MVP is big, could potentially be like a collection of walking skeletons. But like a walking skeleton can also be shitty in the sense of that you don't really, that you don't even expect it to stay this way from the get-go. It can be hard-coded data. It can be, I don't know, like it can be...

like a UI that you know is suboptimal, but where all the moving parts are already where all of the things you know are going to make a difference.

Adi (50:30.744)

Right.

I think that's the key, right? I think walking skeleton, it looks like it's a, I think it's more architectural than features. MVP is usually when we say something, it's like a product list of features that are like minimum, you know, it could be really well built MVP, whereas actual product could be a walking skeleton product. It's like how you look at it, it's like features versus architecture, product versus engineering.

Sascha Wolf (50:42.155)

Yeah.

Sascha Wolf (50:48.136)

Yeah.

Sascha Wolf (50:52.204)

Yeah.

Sascha Wolf (50:58.698)

Yep, yep, yep.

Yeah. And like prototyping from where I'm standing is like a blocking skeleton is something I expect to actually go to production. And I expect it to stick around and then to be iterated upon while a prototype is something and that is something I've been very vocal about and every job I worked at is something I throw away. I expect to throw away. People don't like that. And it's far too often it doesn't happen. But that is my take on it. I expect

Adi (51:30.268)

It never happens.

Sascha Wolf (51:31.19)

Honestly, I've been working very well with being very vocal and being very upfront about it. And so far, every product that I've been building has been thrown away.

Adi (51:42.959)

Oh, wow. Okay, it's never happened to me. It's never happened to me. I guess it's like being in a venture-backed area where it's just like having more flexibility, right? But the reason why I was laughing is like, I was gonna say like, you know, a lot of MVPs are, a lot of production products can be walking skeleton and a lot of MVPs can be walking sumo wrestlers, right?

Sascha Wolf (51:46.73)

But yeah.

Sascha Wolf (52:05.639)

Yes, fucking sumo indeed.

Sascha Wolf (52:12.478)

Yeah. Okay, folks, is there anything else we you think makes sense to address in the context of structuring Elixir applications and building products responsibly?

Adi (52:26.919)

No, I think we're good. I think the conclusion just went up. We said it multiple times. It is subjective. It is, like you guys said, based on your requirements, based on research you do. And even after the same amount of data given to people in the same team, people might come to a different conclusion. Yeah, it's subjective. It's good to talk about it, think about it, try to present a case in an as empirical way as you can. But people might still.

disagree.

Sascha Wolf (53:00.002)

Yeah, if I would have to give one, like to kind of come to a conclusion here, but if I have to give one advice to you listeners is also that don't buy into cargo calls, right? Don't presume that this one technology over there is going to solve all of your problems or that this one language over there, including Elixir, is going to solve all of your problems. But be very curious when it comes to the problem domain you're in. And then

and find and try to get them a maximum amount of information. Like I said earlier, ask all the questions, right? And try to make disinformed decisions. And like I said, I find the last responsible moment principle helpful there. But so if I would have to distill my advice to a single sentence, I would say, make decisions at a point where you know the most

most things without incurring additional risk. That is, I guess, like how, if I would have to distill it down into a single sentence. That is, I feel like also the essence of what we've been trying to do.

Allen (54:18.454)

here.

Sascha Wolf (54:18.486)

Okay, folks, then let's move on to picks. Ah, you wanted to do a go pick.

Adi (54:30.095)

Yeah, so just out of curiosity, I was load testing a script that I wrote in Rust with Elixir. And I was just surprised to see the difference not being a lot. It was also a web component to it. But I was not counting the network latency as part of it. Just how efficiently Elixir uses the CPU time

even concurrent Rust, right? Context switching is a huge, huge time eater, right? Rust was on like three or four times more productive. I don't know the number is irrelevant unless I explain to you the problem, but maybe some other day. I was very surprised to see Go do so well. And I guess that's the way the pure computation power meets good process time management is probably a good place to maybe consider Go as.

and option and the web framework that I use, which was actually very easy to use and supports hardcore reloading and all of that stuff, it's called Revel. They haven't pushed a significant commit in a while, so I would not push in production or something, but if you really want to see the power that Go can give in certain use cases, I can use Alexa for most of them, Rust for some and maybe very specific use cases, I think Go is good.

where you do need good, like I mentioned, good process time management. But yeah, it's worth considering. It's actually pretty cool. So yeah, check it out. Also, what I like about it, I think many web frameworks in Go are very state heavy. They store states in so many weird ways. But I think this is actually truly stateless. So it helps you build a more functional, simple web application.

And it's very performant. It actually, again, something that performed almost 2.5 times better than Rust in parallel heavy computation. So yeah, that's a pick. My video game pick, actually, I just learned about Rice of the Ronin. I think it's a PS5 exclusive. I'm a PlayStation guy. But it looks amazing. It looks like a.

Adi (56:54.471)

a marriage between Neo and Ghost of Tsushima. The world looks like authentic, more authentic than many of the Japanese games that have been built, which don't have a lot of very authentic world. It's a bit more dark than Ghost of Tsushima, but it doesn't look as hard as Neo. So I don't know if you guys have PS5, keep your eye on that. I'm super excited about this game.

Sascha Wolf (57:26.318)

Nice, I personally really enjoyed Ghost of Tsushima. It was one of a few games I actually bought close to release and I do that very rarely.

Adi (57:34.987)

Yeah, I mean, just since I mentioned it, if anyone listening hasn't played it, if you have a PS4 or PS5, just drop what you're doing right now and play it. It's one of the best games of all time. I finished it a day before it was supposed to come out because I was one of the people who actually pre-pre-ordered the Gold Edition and I could not take my hands off of the controller. I literally did an all-nighter.

because I could not sleep. I wanted to sleep, but it was such a good game. So it's one of those very rare games that I would get. People should play Ghost of Sashima. Before Rise of Lorraine comes out, because I have a feeling this will help me hit that level.

Sascha Wolf (58:13.28)

But if you're driving a car right now, please don't drop off your doing. Please drive the car. Okay, Alan, what are your picks?

Adi (58:17.749)

Oh, who cares about the car? Play the game.

Allen (58:26.166)

Sasha's going to like this one. Let's start off. I have one Elixir pick. And there's a new newsletter that just started up. I'm not. I saw somebody start up about Elixir called Elixir Merge. You guys seen this one or not? So we have the only ones I know about is we have Elixir Weekly. I believe it's done by. I forgot his name now.

That's done by the guy who did the Lentor. What the heck is that one called again? Sasha, you know I'm talking about your friend. Yeah. Well, you guys are Germans. I must be friends. No, I'm just kidding. Yeah, but you met him already. Anyways, and then we have the one from I think they used to work together with Jose. I forgot his name now. Yeah.

Sascha Wolf (58:58.214)

Rene Föhring. He's not a friend of mine, but I've met him in person. It's Rene. Ah, okay, all Germans are friends. You heard it here, Biebs.

Adi (59:17.204)

Harmon?

Allen (59:19.366)

So I don't know. It's a little bit late over here. I haven't been saving too well. So anyway, there's two of those. This is the third one, right? So I think it just got released a couple of days ago, or like yesterday or something. So I signed up. I haven't seen any newsletter come out yet, but it'd be fun. The second pick I have is a Rust one. As I said, you'd be happy.

Allen (59:43.682)

They're trying so there's this company called Ferris Systems. They're trying to make a specific high assurance, safe kind of version of Rust. So they're taking the existing Rust out there and then they're adding onto it. So that way you can prove that the code that you write and compiles to the right code that you want. Because you have different factors, right? You got your operating system, the version of the compiler, some other things.

mix them together sometimes you get inconsistent behavior. And so what they're working on is basically making very, very safe Rust so that you can use it in cars, airplanes, whatever. So I think it's pretty interesting what they're doing, if anything at all, to see how they kind of are working with this big open source project, Rust, and then how they're trying to make that work for safe systems that require all this kind of stuff, like proving that the code you write will generate the...

the right code that you want. So it's quite cool overall, otherwise I'm kind of out and make sure that we can start getting rid of some of this C, C++ stuff out there. Because, yeah, it's pretty nasty stuff sometimes. So that's my two picks.

Sascha Wolf (01:00:58.998)

That's actually interesting. Um, I've, because, um, I feel like the, the whole category of a whole problem domain of building software that has to adhere to this level of correctness, right. Um, uh, that is a lot closer to traditional engineering than what most software engineers do in their day to day. Um, I'm actually going to, I'm going to do a pick inspired by that. And there is an article series by Hillel Wayne where.

he went out and actually interviewed people that used to work as traditional engineers, so like civil engineering, but also other types of engineering, and then moved into software engineering. And like, because he started with the premise and the assumption that software engineering isn't really engineering. And I'm not going to spoil what he what conclusion he comes throughout the article series, but let's just say that his

premise, it's not quite, at the end of the interview, CC doesn't quite have the same perspective. But this view is a lot more nuanced. So yeah, I'm going to, that is one of the picks I want to shout out because it's a really, really interesting read and also super interesting to hear about people that used to work in those very highly regulated industries and now work as software developers or engineers.

And to like, to hear Hillel's musings on that, because I feel like he has a very sharp mind. And my second pick is purely for entertainment. I've recently watched Vinland Saga. It's a Netflix anime, and it's like basically about Vikings. And it's about the Danish invasion of England, in that I think it was at the 13th century, or was it the 11th century? Honestly, I don't remember. But it's pretty...

great animation, like it just looks gorgeous and I'm always a sucker for good animation, honestly. But it's also like it's an interesting story, I feel. And so yeah, it's a bit of more on the bloody side. So if that is not your cup of tea, then I would suggest to maybe skip on this one. But if you're not bothered by bloody animes and you like animation as much as I do, then check out Vinland Saga on Netflix. It's been really fun watch.

Sascha Wolf (01:03:25.81)

Okay folks, then was a pleasure as usual.

Sascha Wolf (01:03:32.386)

And I hope you all have a great day and I hope you enjoyed listening to us rambling about architecture and structure and best practices and continue to drive your car if you're in a car. I hope you found our voices also soothing to fall asleep and hope you all have a great day and tune in next time with another episode of Elixir Mix.

Building Applications with Flexibility and Scalability in Mind - EMx 227

0:00

50:04

Playback Speed:

Show Notes

Sponsors

Picks

Transcript