And now, welcome back to another episode of the Ruby Rogues podcast. I'm your host today, Valentino Stoll, and I'm joined by a very special guest today, Kirill Kutsnokov. I'm sorry, I totally butchered that. Kutsnokov. We invited you on, because you wrote this incredible article on MRSK, which is Rails' new kind of deployment tool, amongst many other things.
Kir Kuznetsov (00:13.742)
Do you want to just introduce yourself? You know what you're best known for and You know they give us a quick little backdrop about this lovely thing. We're we're talking about today
Kir Kuznetsov (00:38.318)
Sure, sure. Nice to meet you, and I'm happy to be here with you today. Again, my name is Kir. I am the head of SRE department at Evil Martians. We are a kind of small product development shop. We lately focus our general efforts on helping developer tools mature and grow and become better selves.
There are about like 50 of us. We are mostly known for open source projects over the years. And we are known for our Ruby projects among the Ruby community, how we helped Ruby on Rails startups grow too. But obviously, we are not doing only that. And we help our clients with all kinds of stuff. And one of the stuff is...
their infrastructures. And that's why I am here for them, for Evil Martians, to do my magic and make some infrastructure to host those projects, whatever. And yeah, that's probably a quick recap like that. I can add a little bit more later, because I have some cards up my sleeve.
Yeah, that's great. We're at Doximity, we're a happy customer of Evil Martians. So thank you for contributing to that. Yeah, I mean, so what we're talking about today is deployment tooling and Rails now has some out of the box tooling to support that.
Kir Kuznetsov (02:15.326)
You're very welcome.
I'm familiar with Capystrono. That's kind of where I, my bread and butter from where I came from deployments and then kind of stopped doing it. And now, you know, the good old days and now I don't understand it. I mean, I understand enough to get by and help where I can, but it definitely is more and more complex, and maybe rightfully so, the apps are getting bigger.
Kir Kuznetsov (02:43.662)
Good old days, good old days.
Things are more complex in the cloud. So why don't you give us kind of like a big picture? You know, how is MRSK helping, you know, and how is it? How is it evolved from just the standard? I have this box I'm going to deploy to it with Capistrano and be done with it.
Kir Kuznetsov (03:22.634)
Okay, okay. Let's just pause here for a bit and step a little bit back because to speak about that, we need to define some kind of context and the context here is like I've said, the infrastructure today and helping people understand where to live or where to search for and how to process the ideas that have to be brought to life.
what infrastructure to use, why do people need to think about that? And for example, have you ever had a problem where you yourself were faced with a question, how the heck we should run our project or what should we do at this moment yourself?
Yeah, that's fair. I mean, I'm always trying to just like cheat and right, pick one and then use that where really I should be thinking about it. So what is your process for that? I'm curious. Like, how do you like, say you're starting with a new project, we can go with there and then evolve, right? Like, you're starting fresh with a brand new app. Maybe it's like for a startup environment, right? You're trying to just
get something working for your deployment process that's smooth and will help with the other developers that you have. Like, what do you instantly go for there?
Kir Kuznetsov (04:53.61)
Well, yeah, first of all, you're absolutely right. As we're kind of focused on startups, we deal with like 20 to 30 startups a year. And we see a lot of issues with the infrastructures. And when we see a fresh startup that needs our help, our first idea is always to evaluate the thought that it is a startup. It should focus on the business and the project,
the product, delivering that MVP. So the first usual suggestion that we give to our clients is to use platforms like Heroku or Fly.io, because they are the only thing that offers a startup this bit of magic that helps them focus on the product.
And this is the starting point with this investigation, where should we go next? And controversial thing here is that probably some startups shouldn't move any further, but some should. And this is always context dependent. Each startup should define its own path. And if we're returning back to MRSK a bit here, for example,
MRSK can be already investigated here because this is a nice new tool. Let's switch to it for a moment. This is a nice new tool. We actually are happy that we see more solutions that help you somehow partially manage the infrastructure. Basically, yes, it's a deployment tool, but...
It's kind of like stepping into a bit of like, let me do the infrastructure for you. You get where I'm talking? Yeah, where it spins off all those containers, figures out the seamless deploys by shifting the traffic in between those containers somewhere on the servers, like automatically. Yeah.
Kir Kuznetsov (07:16.77)
So this is a nice step because people started to investigate new ways to do the old stuff. Because we had Docker for years and we still struggle with running Docker containers.
Kir Kuznetsov (07:34.134)
What do we have right now? We have not even Heroku completely suited to run in Docker images, to my taste. We have some services as ECS, which are way more complex than Heroku. We have Kubernetes, which is scary to most of the people, but we'll return to that. And we have a.
tools like Docker Compose and other orchestrators that are way more rare for people to be used.
Kir Kuznetsov (08:31.046)
And I again say that we do love that point of view. Unfortunately, I see the problem that MRSK kind of, like the aura around MRSK and the presentation of MRSK kind of possesses a problem, presents a problem. Because MRSK is just like a tool that solves only a single part of the problem, but is presented like.
in the context where Basecamp and DHH are moving to bare metal setups, massive bare metal setups with their business. And that's an excellent solution for them, but they are missing that part. How do they go from the bare metal servers to the MRSK setup?
Yeah, I'm curious about that.
Um, okay. So where is, I guess.
In your in your article, it's pretty great. You have this like breakdown of kind of like, you know, how do you decide, you know, what is right for you? Right? Like, and
I'm curious just like what your point blank, you know, what is MRSK good for like team wise and app wise? Like what are the like top three things that it does excellent and why people should use it?
Kir Kuznetsov (10:07.118)
Excellent question. The first thing that comes out of my mind is prototyping and small experiment setups, which are not meant to grow exponentially, as they are at the moment of their birth. Or if you have an absolute niche situation,
where you can benefit from the exact set of the decisions that will include MRSK, like Basecamp does, for example. It's not the only solution we can discuss a lot of those edge cases. But let me just give you a downside for this situation on the Basecamp side, on the description of the Basecamp side. Let's do it.
like imaginary exercise. Imagine that I will provide you with like five bare metal servers, each has 48 scores, tons of memory, whatever, I even like screw them into the racks for you and power up, what's your next steps?
Lock it down. Ha ha.
Kir Kuznetsov (11:33.49)
Yeah, they're undefined. This is not the solution to manage the infrastructure. This is just a deployment tool, you see? And this is its limitation. This is the main limitation. So if you just need to deploy to a limited set of virtual servers or any service, basically, but better virtual servers, you can't use it.
for the tasks outside of that situation. You have to do your own homework. You have to do your own preparations. You have to basically in the situation that I gave you, you have to either do a manual configuration for those servers. And when I mean manual, you obviously expect, you're obviously expected to create some infrastructure as code for yourself.
that will save your future. Yeah, but you basically will invent something manual there. And when we start to grow, you continue to do the manual things until you'll basically be done with that. And you either.
in reinvent your own virtual server orchestrator, or you'll say to hell with this all and just set up a different solution that will spin up the virtual machines for you. You see? But when you get the... I don't know, you need to create yourself a test environment that should be...
shut down and spun up continuously, you can use MRSK for that. For example, on DigitalOcean, Amazon using just their bare virtual machines. And yeah, this will give you the feeling of using just the old favorite Capistrano sauce for your project. But...
Kir Kuznetsov (13:54.038)
You can't build the infrastructure around the MRSK. You need to build your infrastructure against MRSK, just general infrastructure, not against, but how should I say it? Anyway, you will have to build your own infrastructure in advance, and you have to think about the infrastructure in general. And it's way more than just a deploy.
So if I'm getting this right.
MRSK is less about like propping up new servers and configuring them and more about like the getting the app on the servers aspect of it.
Kir Kuznetsov (14:40.674)
It's a correct statement. I would have added it's a tool that gets your app on the virtual servers that you already have, because this is a crucial part. That you already have.
that you already have? Yeah, I mean, that you already have. I mean, I think that makes a lot of sense. It makes me think, I mean, I'm a Linode user, and they have recipes for propping up new servers, new virtual containers, things like that, with those resources locked down and things already set up with existing recipes, right? Like, I feel like they're not the only platform that does this kind of thing, right? Like Digital Ocean has their droplets.
You know, you can create a new resource, a new whatever that you're trying to get propped up and have it ready to go to use to deploy an app to, right? And I think Heroku kind of works like that where you have to still manually create, you know, the Heroku resources before you can start to use them, right, and make sure that they have all the... It's, yeah, it's a little more magic, right? But it's still the same idea, you know, you...
Kir Kuznetsov (15:42.298)
Just with a bit more magic.
set up the resources, then you can use them kind of thing. So I mean, I feel like that makes a lot of sense to me still, right? This is still slotting more into the Capistrano realm, which I like personally. So I mean, that's good. It seems good. The Docker thing, I'm not complete, you know, Docker is a, you know, it's a magic that I like.
Kir Kuznetsov (15:51.86)
Kir Kuznetsov (16:01.754)
Yeah. Yeah, yeah.
You know, will it survive forever? I don't know, but, uh, it's definitely what is used. Right. Um, and so it seems like that has been like MRSK uses Docker. Right. There's no other, right. Okay. So, uh,
Kir Kuznetsov (16:29.355)
Kir Kuznetsov (16:33.194)
It's crazy that MRSK hadn't appeared like 5-6 years ago. And it's just only now that we see it.
Yeah, I mean, I'm so, I guess I'm trying to.
visualize in my head how much different it is from campus Toronto and We had another episode in our SK where Dave and I kind of dove into pieces of that Let's see what episode was that I think it was 592 but You know, how does how does what about MRS K is different from campus Toronto and like Maybe what ways is it simpler? But also, you know
What convenience is, is it giving it to you in that deployment process? Or like, let's take the step back. Okay. It's not like the Kubernetes, right? It's not the orchestration of those resources, right? Which is what Kubernetes is great at. If you're trying to, if you want to quickly spin up a new server, right? Like a new, create a new resource, like Kubernetes is great at that. And the, MRSK is decidedly not toward that, right? Like you have a tool for that. You use that.
Kir Kuznetsov (17:33.223)
Kir Kuznetsov (17:53.61)
So where is MRSK like?
You know, how is it different from Capistrano is basically what I'm getting at.
Kir Kuznetsov (18:08.302)
Tricky question you have for me. Okay, let's try to figure that out. It's like right now. Well, in my mind, this is the same Capistrano, but with Docker containers. Yeah, because do you know what MRSK in general is in 2023?
Kir Kuznetsov (18:35.938)
Can you imagine that? No? OK, OK. Yeah, that's why you have me here. OK, be prepared. The next statement is technically incorrect, but it gives a nice picture or perspective to what MRSK is. MRSK is well thought, well prepared solution that's
that is basically an Ansible with an appropriate role and some tweaks. That does just the thing of deploying those containers. Because we saw clients who did basically that same thing a year ago successfully without any struggle with the just Ansible. Same thing, like literally.
Kir Kuznetsov (19:32.226)
Literally, same thing, yeah. Yeah, and it's great that people start to polish the tools and make them more comfortable for the particular task. It's the only problem that when you polish the tool to perform the only task that it was designed to, you should definitely fit into the task. And do not step.
Kir Kuznetsov (20:01.49)
even a step outside.
Gotcha. So I guess my follow up question is, with the cleared separation of like the app deployment process to the like resource deployment process, have you seen like any issues with that divide like being like the line drawn in the sand, right? Like as an example, like I'm thinking of Chef or Ansible or something like that.
Kir Kuznetsov (20:33.999)
where the two are together and because they're together, there are a lot of benefits to it, right? Have you seen any issues really with that being separate?
Kir Kuznetsov (20:39.822)
Kir Kuznetsov (20:47.656)
Can you give me some more context for me to understand the question better? Maybe some short example?
Yeah, like as an example, you can have some chef scripts that set up the resources and set up all the different communication between those resources. And then they also will handle the Capistrano if you wanted to encapsulate that in the same processes in their own recipes. And so having it shared as a whole,
Kir Kuznetsov (21:13.058)
the app grows, the infrastructure grows with it, and you wanna scale things or handle auto scaling, those things can be handled easier because it's all kind of centralized using the same tooling. So have you seen any issues with that being separated where you have maybe two different tools, MRSK for your deployments of the app, and then something else for the management of the resources?
Kir Kuznetsov (21:29.143)
Kir Kuznetsov (21:32.429)
Kir Kuznetsov (21:45.734)
I do and I don't see the issues simultaneously because, yes, it's kind of easier to understand what's going on if you have a single source of truth. And probably you can work excellent solutions around that. But actually, Valentina, I think the question is kind of a bit still...
to the question is still a bit too messed or mixed to the point that there is no exact answer to that. I think we can rephrase the question to the point is how to make the infrastructure. And to that, I steer it towards that idea because the deployment of the app,
is just one part. And you're talking about this shift around the separation or adding all the things into one place. It's about managing the whole infrastructure, right? Yeah, so.
Right. So, yeah, let's hone in on a specific thing then, right? Like, let's say we're introducing something new to the application deployment process that is like an addition of an email server or something like that, right? Where it's another resource that we've added that we need to integrate with the app that has a new deployment process coming into it. Maybe Redis is a better example, right? Where we didn't have Redis before, now we have Redis and the deployment process needs to, I don't know,
Kir Kuznetsov (23:19.2)
also introduce some new configurations, settings that were configured externally or are managed externally, but the application still needs to set up that link itself in the deployment process, right? So like having that separation, right? Like, is that really like a bad thing? Like, is it really harder to manage than if it were, it is?
Kir Kuznetsov (23:31.416)
Kir Kuznetsov (23:35.831)
Kir Kuznetsov (23:41.314)
Kir Kuznetsov (23:53.794)
Yes, it is harder to manage. You can forget about something. Like with that same MRSK problem, there is a separation between the real virtual servers, pun intended, that you have on your setup and the configuration file for MRSK itself. They are in separate places. And you need to figure out yourself this...
this connection thingy that will allow you to be solid on your path here. And the solution for that always was...
Kir Kuznetsov (24:41.882)
service discovery integrates it into the core idea for your infrastructure, at least partial service discovery. When we had no Docker or when we had no Kubernetes or any other orchestrators, per se, we had to rely...
on different things and there was Chef discovery magic inside the code while the Chef server. Ansible did some facts aggregation from the ground up. We had excellent tooling from, and we still have them from, HashiCorp, the whole HashiCorp stack was born to...
Highlight the problem that people you are doing something wrong. You're you're splitting the parts too far away from each other You're you're just Driving all this cruise ships manually pulling all the strings This is wrong, you know, not cruise ships like this freight ships with a lot of Goods on them, you know
Kir Kuznetsov (26:07.634)
This is a disaster. You need some tooling. And that's why we like, for example, Kubernetes. Let's give a different perspective here on what's happening on the other side. In Kubernetes, if you got yourself a Kubernetes, obviously there is some separation too, because you have to get yourself a Kubernetes somehow, either on a cloud or on a bare metal service. And it's obviously hard.
uh on bare metal it's harder on the bare metal you have to have some level of expertise there too but uh when you
Can you elaborate on that? Because I've never had to set that up. Let's.
Kir Kuznetsov (26:47.442)
Well, I can, but probably just a bit later, because we have a different topic right now. Yeah, we can jump into that. I can give you some perspective, for sure. Yeah, in Kubernetes, there is an idea that you have, it's not a single source of truth, but you have a unified way to address.
Kir Kuznetsov (27:16.702)
things around yourself. You have a unified API which is extendable. You have the idea of service discovery using that API just like from the ground up. It was built around this. And it was built like that on purpose to make things know... Let me restart the phrase.
And it was built like that in purpose to make you be able to walk around and to make your application walk around and see things like people do. Like, you know, that we, we need, for example, we understand that we need to go to the shop. We look for a shop, we go there. Yeah. Same way you can.
do your stuff inside like a Kubernetes API. It's a crude example, but it is what it is, yeah, for us to understand the idea. And this is what we like, because that way we can simplify not only the deployment process, but the deployment process across multiple systems, because they start to know something about each other's. Or.
the scale up or scale down situations or the self-healing situations, which is advertised by Kubernetes. But that's basically because there is some additional knowledge incorporated in the surrounding infrastructure there, which you don't have when you just talk about a single tool.
So yeah, it's kind of, MRSK is kind of like Capistrano for Docker and that's it. It's or Ansible, which was stripped of the additional capabilities to just streamline your application to the dedicated virtual machine.
Gotcha. Yeah, I mean, I think that makes a lot of sense. I mean, I like that in this case, because I, you know, because I come from campus Toronto. So it's easier for me to, you know, it's easier for me to reason about, you know, and mostly because I don't have much Kubernetes experience, which, you know, maybe people don't need anyway, right? I feel like introducing Kubernetes as a startup is probably a bad decision, right?
Kir Kuznetsov (29:35.655)
Yeah, yeah. Me too, me too.
Kir Kuznetsov (29:45.555)
Yeah, yeah, yeah.
in most cases.
Kir Kuznetsov (29:57.042)
Interesting question, interesting question. Maybe we can just a suggestion, maybe we can just describe this like the general idea of how you choose your infrastructure just a bit further, if you will.
Is it? Ha ha ha.
Yeah, yeah, I mean, I'd love I'd love to know your thought process, right? Like, as you're coming into it, what do you weigh?
Kir Kuznetsov (30:16.734)
Yeah, so you see we have, first of all, we have some experience. We see what problems people face around us. And yeah, we are experienced with Hiroko deploys, Capistrano deploys, Ansible deploys, and Kubernetes deploys, and deploys to tools like ECS or whatever. Yeah, you name it. So.
Like I was talking before, we recommend startups to start their life from Hiroko, because this allows them to keep their focus. And basically, they should keep their focus as long as they can. This is the general suggestions from us when we investigate and research the situation, how to help in each particular situation. Obviously,
there may be some problems on this way. For example, the costs are skyrocketing, like the most popular example. But this is not a problem. You see, this is a situation, and the problem is somewhere inside, underneath. And we need to always figure out that problem, because usually, like 99.9% of the time,
It's some kind of like a technical limitation that makes you just drop your money there and empty your credit cards. You know? So what?
I'm curious, just briefly on that, what are some of the most common wallet emptying things that you've seen?
Kir Kuznetsov (32:06.918)
Oh, a number of them. I don't even have the common one here. First that comes to my mind is the bloat of the database. Like, see? OK, we can tackle this example. Excellent one, actually. For example, you start to pay enormous amount of money for your database.
It's a technical problem. It's not a financial problem. You need to investigate what's happening with your product. And for example, you really do need that kind of data, that kind of amount of data that you're processing. But that raises the second question. Why didn't the business?
Kir Kuznetsov (33:02.946)
investigated that on its side, because maybe the technical realization, technical solution is not what really business wants to, or vice versa. Yeah. And this needs to be in sync. This is the first bell that rings that is crucial for any business and technical team of that business to watch for too, to tell the business.
There's something wrong. It's always the problem of that sync between the business goals and the technical solutions for these goals. If they aren't in sync, the problem is not money. The problem is with business always here.
This is, this is so true that what a great point. I mean, I don't know how many companies I've been at where, you know, a product decision has been made and that it's like, nobody asked why are we making this decision and they just build it. And then, and then it turns out, you know, a year or so later, Oh, we didn't really need that. We needed this other like tiny little thing. Uh, and you know, technically speaking and you know, we didn't even need to build it to begin with.
Kir Kuznetsov (34:00.088)
You know, and we could have we could have used like S3 or just a Google spreadsheet, right? Like, for the front end part of the process to pipeline it in to where they needed it, right? Like, there's so many things that can be just like saved from like asking the why question, right? Like
Kir Kuznetsov (34:15.787)
Kir Kuznetsov (34:33.798)
Yeah, five times, five times. Remember that this is a good framework.
Yeah, I'm serious. I'm dead serious. Yeah. But it's real. It's, it's, I forgot the name of this idea, but you have to ask why at least five times and go deeper with each iteration. Like why do we pay so much money? Because the database is wrong. Why the database is wrong? Because for example, we have a database bloat. Why do we have the database bloat? For example, it's technical error or.
That's so funny.
Kir Kuznetsov (35:09.974)
We do really have this kind of load that we're not just dealing with correctly. Why aren't we dealing with this load correctly? Why don't we process the data that way that we can, that the database can keep up and clean up itself, et cetera, et cetera, et cetera. Yeah. Why isn't business aware of this problem? You see where I'm going, but I can imagine the second example for you. This is especially for this episode.
because it involves MRSK, Myersk probably, yeah. Imagine you have Linos, you said you were using Linos, yeah? Imagine you have like, I don't know, 20 servers. You did deployments of Ruby application across all those servers.
with MRSK, you're absolutely happy with it, it's running perfectly suddenly, Sunday evening, it fails what do you do? how do you know it failed?
Kir Kuznetsov (36:23.924)
I don't hear you, I'm sorry.
I would say, you know, probably like a bug snag alert or something, right?
Kir Kuznetsov (36:32.598)
Well, you need some kind of instrumentation and monitoring solution. Exactly. Let's be general and say monitoring solution, at least like web checks. What monitoring solution can you implement for this kind of project? Like a back snack, like a data doc, like a new relic.
Yeah, the instrumentation. Yeah.
Kir Kuznetsov (36:57.502)
whatever, and you basically, we can simplify that and say that you basically have to buy yourself monitoring solution service. And now you have like, I don't know, maybe 20 servers were too much for this example, but like, I don't know, five servers, and you have a bill for like...
300 bucks for your Linode virtual machines. And you can get yourself in a situation where you get, on the other hand, a bill for like $5,000 to $8,000 for the Datadog. Again, why? And yeah, again, this is kind of like a technical problem. And how should we tackle this problem?
Kir Kuznetsov (37:52.778)
we need to always think about the infrastructure. And the only part of the life of any project that can skip a bit on it is when you start and you try to move on Heroku as far as you can. But already, you will face some technical limitations eventually. Or restrictions. For example,
you need some compliances to pass some compliance tests or security requirements, new security requirements, whatever that restricts you from using Heroku. This is the second example. At this point, you already have to be prepared to know your next steps, how you will evolve your infrastructure. Right? Right.
Kir Kuznetsov (38:50.366)
And this is kind of like a stop point for us always. And we try to emphasize here for any project that comes to us, guys, find yourself an SRE team. We don't care how. I don't know, grow them. I don't know, get them from the moon, from the Mars, from a...
Ha ha ha.
Kir Kuznetsov (39:19.214)
Get your debt to learn AWS or Kubernetes. We don't care. Get yourself a team that will think about that, that will have a task of thinking about the overall situation of your infrastructure. You can hire basically us. We do that. We provide infrastructure support with 24-7 on call.
Kir Kuznetsov (39:47.682)
We can do this, and we especially do this with Kubernetes, because it's basically easier for us to provide some more reliable infrastructure, even on a smaller scale on Kubernetes, with all the support, with all the smaller to bigger SLAs, I mean, in the scope of these SLAs. Because we already invested a lot of our time to figure our ins and outs.
If you don't want that, you can go anyway, but get yourself a T. Because we, for example, had a lot of situations where we already see a project running, for example, on AWS, ECS, do not have enough monitoring. They just pay ridiculous amount of money to DataDoc or New Relic, and they still don't get enough insights. Yes, these solutions provide you with excellent...
instrumentation options, but still any step outside of the general configuration. And you either have to pay more money, even more money, and nobody thought about it, or you are not allowed because you are limited by the money, for example, or I don't know. Any other situation is applicable here, too. And you can't implement.
additional insights, for example. I don't know. It's hard out of the box to know if your container ECS was restarted a number of times, for example, so it's failing basically. You don't have this. And a lot of projects are in this situation and they're afraid to move on. They are afraid because they are afraid to invest in the infrastructure team, at least one person.
hire it, it will be cheaper for you or hire us, the price will be comparable. Yeah, it will be additional one. You need to think, you need to check the price. So one of the metrics, if you should move outside Heroku or if you should just really tackle the money situation, for example, with this metrics solutions or with Heroku itself, does Heroku or DataDoc place whatever they are?
Kir Kuznetsov (42:13.59)
costs you more than the new infrastructure that you can imagine, plus the team of the size that is needed to manage this infrastructure. If it's higher, then you're in the spot. This is like a solution to check yourself, the question to check yourself. If this question is, yes, it's higher, the first one is higher, the Heroku price is higher.
You definitely need to start rethinking your situation in your next steps, you know?
But that's a great analysis. I never thought about it like that. Like, you know,
I guess most people won't... Sure, go ahead.
Kir Kuznetsov (42:54.014)
Just a sec. I'm really sorry. And yeah, this will be hard thing to swallow, but if it's cheaper than that, you either have a business problem or just don't complain about money. Pay them and do your business.
Right. Yeah, I feel like that's probably one of the hardest decisions as a business is to spend the money on the infrastructure because it is kind of like a behind the scenes costs that if you're not technical, right, like it does just seem like why am I paying this right? Like, and it has to be paid. I mean, especially as things like if you if you run into a business problem and it's the problem, you know, even still like as a business person, you're not going to understand that.
Kir Kuznetsov (43:34.753)
the reason technically why, right? Like that the problem exists and that it needs to be fixed. And so it does, it's probably hard for a business owner that's not technical to make that decision, right? And so I think that it's definitely a thing that's missed though is adding up that cost like, and maybe this is something that should be planned for when you're first starting out, right? Like what is the cost of an infrastructure person to hire? Right? And you know, what, at what point?
Kir Kuznetsov (43:58.743)
does that cost and I mean, your equation is perfect. You know, adding up all the costs of, okay, what's my break even point for that infrastructure team, right? And like, okay, once we get there, we need to at least start thinking about it or starting right to transition. And that's perfect. And I feel like people don't plan for it. It's kind of like, it becomes like an emergency, right? Like, oh no, we're spending too much money. Now we're bleeding.
Kir Kuznetsov (44:29.043)
Kir Kuznetsov (44:36.823)
Kir Kuznetsov (44:42.866)
You're taking words out of my mouth, literally. Yeah. We actually had a situation where a client came to us with the ALBS ECS infrastructure and some services around it. And they had a strict restriction where they said, we don't want to change the way the infrastructure is. I'm really sorry. The way that the infrastructure is right now.
Kir Kuznetsov (45:12.234)
And we literally spent more money introducing the workarounds for the problems that they had. They spent more money on us than if they would have hired us to introduce them the full-scale Kubernetes cluster with a support worth like 10 months or nine months, 24-7 on call.
80 hour, 40 or 80 hours a month, something like that. So yeah, you need to always do these calculations in your head.
Yeah. So I'm curious, trying to bring the conversation back to like MRSK, right? Like thinking about these costs and infrastructure setups and transitions, right? Like where does MRSK provide value in that, right? Like if you start with MRSK, or even if you adopt it at like tail ends, like if you weaned off of...
I guess we have two different directions, right? If you transition from something else like Heroku to MRSK, where's that value there? And what value does it provide if you've started there?
Kir Kuznetsov (46:33.726)
Actually, I really like the point in time where you asked this question to return to MRSK, but I'll be pressing the general idea here. I'll try to incorporate MRSK as much as I can here. Yeah, but you see, I've just talked about find yourself a necessary team. After that, you probably should not care.
We don't have to be specific about it.
Kir Kuznetsov (47:02.15)
what do they do for the infrastructure? Like, I mean, you don't need to listen for the hyped topics like, OK, we have a brand new platform. We need to go there. Or we have a brand new database. We need to go there. Or just move to bare metal. It will be cheaper. This is wrong. This is a mess, plus the technical solutions that your team is capable of doing.
If your team is capable of doing a perfectly functioning solution based on the Merse key, like for example Basecamp does, because they have a brilliant team. Yes, it's not big, but they're brilliant. And they have a very specific situation where their business is pretty stable. It's huge and it's stable. And so...
Look at my hands. They have a stable business. This is a math business slash math problem. That is resolved. They know their money. They have a team figured out beforehand, which can select any solutions that fit their needs and solve any problems.
Kir Kuznetsov (48:28.094)
that is not automatically solved by this solution. I don't know if they choose Kubernetes, they may figure out they need to write some operators, additional operators just for them. We saw that in our experience too. Or it's too complex. We don't want to figure out Kubernetes. But we already have the expertise to run our own virtual machines like Basecamp does.
and just run MRSK on top of them, this is a perfect solution for them. So just let your team do the magic there. Keep your focus on the whys. Why are you doing that? Get the data. Ask your whys and do your math on how much does it cost. Compare the situation. If it fits into your budget.
Kir Kuznetsov (49:26.858)
you're probably doing right. If you can do something more efficient, that comes from more wise. Why do we need Kubernetes? Because, for example, in our case, we can provide you with a stable Kubernetes way cheaper. Then we will basically write yourself custom infrastructure on ECS for that particular situation. And this will have much more perks for you. Or.
We just say you that, yeah, stay on Heroku and don't bother with that for at least next three months. Pay us like $300 for this consultation and see you in three months or half a year or a year. You're welcome. We saved you like a few, like a few, 10 to 20,000 bucks. Yeah. In a short term.
This is it. And back to Marskey one more time.
Kir Kuznetsov (50:33.57)
Actually, this is just a tool. You see, this is just a tool. For example, you can use your notebook for editing video. And you can use your notebook just for typing code. And obviously, in each situation, you have a different setup. And you put your effort to tinker that setup for laptop, obviously. My bad.
Kir Kuznetsov (51:08.794)
and you put your efforts into those two different directions. But your starting point was the same. It was only the task that defined your actions.
Kir Kuznetsov (51:33.23)
Based on this task, you bought the exact hardware included in that laptop. For example, more RAM or a dedicated graphics card, whatever. Yeah. You can combine the solutions. You can do both tasks, but again, this is the goal that you've set for yourself.
so you figured out how to solve this goal. That's as simple as it is. You don't need to overthink it.
Yeah, you bring up a really good point of like local development that I think is solved by Docker in some ways, right? Is the more you can virtualize and have everybody on a production like system, right? Probably the, the less problems you'll have infrastructure related when it comes time to using that same app that you've been using locally. And they're obvious, uh, you know, caveats to that statement. Um,
But for the most part, I feel like that's true. Like, having worked in a Docker container environment, it has definitely solved some infrastructure-related issues that would have eventually happened, caught earlier, because I was using the same kind of infrastructure that it was being deployed to. But I had some questions about your, like, how you think about pricing and doing those equations, right? Like,
I think one of the biggest problems business owners face is like kind of like the hidden costs of everything, right? Like as a perfect example, everybody's like, oh, Amazon is so cheap, like just use Amazon, right? Like, and people go to use Amazon and then they get their bill and they're just like, whoa, I thought this was cheap, right? And you know, they don't add up all the additional costs of all the different resources that they needed at the time until they realized, you know, what everything it was that they needed.
and maybe that they've configured it, you know, not ideally for the case that they are transitioning to, which is actual the business running and popular, right. And now they're stuck with their setup and because of how they set it up, it's costing them more, right? Like how do people kind of, how do you like help people think about those costs without knowing that they exist already, right?
Kir Kuznetsov (54:05.25)
Um, I love the twist. I love the twist. I have, I have to admit that this is excellent. Yeah. We, we, we like talking, we're like talking in general here and forgetting about the Mariskey, but yeah, this is an, this actual, uh, brilliant twist. Um, you see, there are no cheap solutions. No cheap solutions. You, you will be hit.
Kir Kuznetsov (54:34.794)
with bills everywhere. And the only thing that you can do is to figure out your own way. And how do you figure out your own way? Like all startups do, basically it's the same idea. You do experiments as fast as you can. You define the goals. You get the limitations. You figure out the limitations to get to these goals.
and you'd start to work around those limitations. Again, it's all about investigating your situation. Always expect to get a huge amount of costs and always expect to work around them and try to reduce them gradually. And how do you do these experiments?
This is actually one of the next parts of that framework inside my head, and that we present to our clients. Figure out the limitations. Like, for example, the simple one. You need to do three business experiments in the next two months, where you'll be hit by a huge amount of loads.
on your infrastructure, for example, you will run some ad campaigns just to experiment, will your business get up to the speed? Will you be able to use bare-metal servers there?
Yeah, I mean, it's a good question.
Kir Kuznetsov (56:18.742)
The clock is ticking, the clock is ticking.
Kir Kuznetsov (56:24.298)
OK, I'll reduce the anxiety here. No, you won't. You will have to use public clouds. You will have to figure out something. And you have to do it, because this will help you figure out your costs on public cloud. Yes, you'll lose some money, but you'll know the data. And you'll be able to step back, because public clouds, then you just shut them down in an hour or two, and that's it.
You had your experiments, and you have the data that you can work on. You can decide, yeah, I was right about the ads. I was wrong about the, I don't know, the technical solutions or vice versa. So I should continue using public clouds. I'll be able to squeeze more of them. I can optimize here and there and get the situation up and running because I have this limitation and I know what to do with it. Or you're...
For example, your application, we actually had a project where we just had bare-metal servers because we had one interesting technical limitation. We needed extremely powerful database server because we were doing, I guess, some cohort analysis. And it was like seven years ago. So the cost of the same machine on a public cloud would be skyrocketing. And we had limited budget.
And that was our technical limitations. We figured out that exact solution. And we worked around how we can invest the rest of the money to make it work. We hired two guys. I joined the project. And we hired another guy who worked on this project full time, basically. Like we shared. I think it was like one and a half person per month.
working on this project. And we figured how to squeeze into the budget because we had the strict technical limitations and the power of the old systems were lower, especially on public clouds. Same goes vice versa. You may, like I've said earlier, you may be in charge of a project. We have clients.
Kir Kuznetsov (58:48.09)
and they organize public events online, like during the COVID. And they have load for like a few hours a month.
No, you won't be buying yourself bare metal servers. And here comes some risk. If you have your limitations figured, for example, you have a person who thinks about it, who figured out how you will do the deployment stuff, how you will do the monitoring stuff, how you will do the log segregation stuff.
how to, I don't know, organize your infrastructure as a code and link all those things together. It may be wise to pay him like to invest like a few additional thousand bucks per month into him, into his time, and get yourself a stable infrastructure, which will
which will be run on MRSK, but you will need only like one person. For example, like, for example, like you will need only one person because it will be easier for him to understand the whole infrastructure. He'll be able to spend more time, more of his time figuring it out, figuring out how he will work around the limitation that he has. And, uh,
Yeah, he may not be the Kubernetes expert, but he will be paid to think around the situation that you have. And you will benefit from that. On the other hand, you may decide to invest your money, for example, in the solution that we sometimes suggest from our side. Yeah? Like just.
Kir Kuznetsov (01:00:57.174)
Give us a few weeks to like a month of time initially. We'll spin up your new Kubernetes infrastructure that you never thought of. We'll do some magic there. We'll give you our prepared internal framework, which is based on absolutely open source projects. So it will be accessible for you 100%. And it's possible for you to understand its ins and outs on yourself and hire yourself.
your own specialist to manage it if you're afraid of. But after this period, you'll be able to just rely on us to keep it up and running reliably, knowing all the limitations. Because we were thinking about these limitations for years and this will cost you compared to hiring a specialist. But if you don't hire anyone, expect to work around the limitations on your own.
And why are you complaining, guys?
Yeah, I feel like most people will just, you know, they don't want to, they know that there's costs coming for Heroku or something like this. And they know enough to, you know, prop up their own resources. And they'll, you know, go to deploy and then they, you know, the next step comes. Well, how are you monitoring this? And you know, who is monitoring that? And it's you. And oftentimes.
Kir Kuznetsov (01:02:26.73)
Wrong, wrong, I'm sorry to interrupt you. Wrong. In these situations, often nobody is the correct answer.
No, but yeah, that's true. Yeah, I mean, eventually, because you're doing other things, right? Yeah. Yeah, you're doing your business. That's so true. But yeah, I mean, so in that way, I mean, I know I have tested Docker containers before, like just playing with, oh, like if I move this app over to this new provider, do I save any money on that?
Kir Kuznetsov (01:02:40.778)
You're doing your business or you're doing the product. You're making your product.
And it's definitely an experiment worth testing out if you have a deployment tool in place that can make that easy to do. And so that's where I'm kind of like hopeful of MRSK providing that ability to like make the experimentation easier. And I don't mean like resource, like infrastructure setting up and configuration, right? Because if you're already in that space of using MRSK, I feel like you're already
know kind of how to, what is involved in setting up resources. No, you don't think that's mostly the case?
Kir Kuznetsov (01:03:41.234)
I'm sorry, I don't think so, because that's why I was like throwing the word experimentation and test environments, because when people start to do that MVP, for example, or
Kir Kuznetsov (01:04:03.494)
or a setup that tests their idea, they're still focused on the idea. They're not thinking about the infrastructure that will sustain your project for long. So it's still the mindset where you're just like, where you just think, but I can deploy the application. The problem is solved. No, the problem is not solved. And this is where...
Kir Kuznetsov (01:04:32.77)
it gets bloody because you have to think about the whole situation. Who, like we were just talking about this before, who will be in charge of the problems for that infrastructure? How these problems should be solved? How you can work around the possible
Kir Kuznetsov (01:05:01.034)
There is a huge amount of topics there. And the tool that solves only the deployment situation is just a tool that solves the deployment situation. And this is a great way to start your experiment. You can work around this tool to grow a bit or to a huge amount, but you need to do.
the work around the whole situation. Never forget that your project is not just like running in a thin air, just hanging there. There are business problems around and there are technical problems around that are outside of the scope of the.
Yeah, that makes a lot of sense. I mean, it keeps circling back to, you know, weighing your costs, I guess. And it's funny that you do mention that, like what I was getting at with like experimentation, and I'm gonna take, we keep talking about Basecamp. There's this article that they have on Signalverse noise on using spot instances for like specific, you know, background tasks that they knew.
Kir Kuznetsov (01:05:50.086)
I'm sorry, uh...
Kir Kuznetsov (01:05:54.292)
Kir Kuznetsov (01:06:10.263)
that they were just like spending too much money on for the amount of processing that it was doing. And they switched to using spot instances and it came with its own problems. But because that but because they had thought it through and had like kind of the people there to work through those problems and think through them, you know, it became less of a problem for them. But it's funny because
Kir Kuznetsov (01:06:20.781)
Kir Kuznetsov (01:06:26.982)
Yeah, of course, of course.
Kir Kuznetsov (01:06:40.392)
You know in the end they you know they will probably cannibalize that all of their work on this And I think that's a good point to bring up is that these? experimentations you know May lead to like nothing and that's okay, right like it's not always worth the time to Invest in these experiments right where you're testing You know what?
what you can and can't do with your infrastructure. And I feel like that's a common thing that's not done, right? And maybe there's probably a framework or something I'm not thinking of that, not necessarily coding wise, but where there's a process for testing, these specific goals or maybe you have some pricing that you're trying to cut down a certain amount, right? Where you can go through a process
Kir Kuznetsov (01:07:37.74)
Okay, we have this long to test this. And we know that we're willing to spend this much money investigating, right? No matter what. So if we eat it, that's fine. But if not, we can save, you know, we hope to save this, right? And in the end, I feel like more often than not, your infrastructure is better for it. Right, like you will save more money just from that investigation process because people have learned.
Kir Kuznetsov (01:08:03.795)
and investigated and gotten to all those contact points that you ignore. Right. Because I know when I do a, you know, a one-off project or something, like, you know, I, it's not my business, but like, you know, as most startups, like you, the prop up the infrastructure, whatever you start with. And then you just like, well now I need to do the product, right? Like you're saying. And so, you know, it does, it goes in the background and then like,
Kir Kuznetsov (01:08:16.13)
Kir Kuznetsov (01:08:32.36)
If you could do these sprints, it's always worth it. Spend the money, you're gonna be spending it anyway. The sooner you can run the experiments, the better. But it's hard, I guess, to decide to do that over product. So, I don't know, do you have any closing statements on that? How do you know the right time to invest in infrastructure versus product development?
Kir Kuznetsov (01:08:41.982)
Kir Kuznetsov (01:09:04.205)
Actually, I already told you about that.
Right, it's the formula, right? Like...
Kir Kuznetsov (01:09:08.55)
Yeah, always watch out for the technical limitations, which are highlighted by money sometimes. Expect there are some technical limitations behind the money that you see, the amount of money that you see on the bill. And do that math periodically. Will you be able to reduce the costs?
of the infrastructure if you hire yourself a proper team which knows ins and outs of a different kind of infrastructure or grow it yourself or hire a company to free to help you there. You may be, you may be.
you may get some additional value there, because sometimes the answer may be unexpected.
Kir Kuznetsov (01:10:04.034)
first, because the answer is still stay with your old infrastructure and figure out more business things. For now, maybe as valuable as how should we reduce the costs. Or yes, you can do those experiments. Actually, I have one great thing to add to your final thought.
I love this thing that calls growth hacking. We actually really had a great experience with this concept and implementing it on some of our projects where the clients were open to some new ideas and those were pretty huge clients and yeah, this helps sometimes drastically and it can be applied in all the ways like in business.
And in the technical stuff, like, do the, like, define, uh, the, the basic idea is there, there's, you create an assumption. I don't know the right words in English, how they, how they, they call it in the framework, but you, uh, you, you figure out some assumption, you set the metric which you will, which, uh, will be monitored by you to evaluate that assumption.
And you set a time limit after which you do this evaluation. If it works for you, you have a win. If it doesn't, just stop, try the next assumption. You can always try to use this framework. It has strict rules how not to go circles, but this is outside of the current topic. But yeah, this is a valuable framework.
for resolving some problems.
Yeah, I'm a big fan of growth hacking. It reminds me of AppSummo. They used to have a, I don't know what you call it, like a workshop. And they would have you, if you were interested about building your own company or your business, they were like, well, just try and sell something on Craigslist, right? And see if you can get people to pay you money for something that you were trying to sell. It doesn't even have to exist, right?
Kir Kuznetsov (01:12:25.814)
and you can refund the money and like they just did this experiment like if you can get somebody to buy something from you then that proves it's valuable right like and if you if you like think that you have this great idea that's going to make you know thousands of people like you have to prove it right and I feel and you know that's true like of not just selling right like it's true of the infrastructure too like if you decide to use MRSK right like
Kir Kuznetsov (01:12:48.407)
you have to prove that there's value in that for you, you know, for you, right? And I think that's kind of like an underlying thing here is like, it's not for everybody. It's not like a one-all, this is the deployment tool you're gonna use for Rails, right? And that's fair because it's, you know, nothing is, you know, even setting up your business, you can follow the growth hacking and still not make any money, you know.
Kir Kuznetsov (01:13:04.505)
Kir Kuznetsov (01:13:21.499)
Yeah, yeah, yeah.
Kir Kuznetsov (01:13:29.055)
Yep, for sure.
Kir Kuznetsov (01:13:32.454)
I guess this was our first peek about gross hacking.
Yeah, so funny. Yeah, speaking of which, you know, we're, we're coming to the end of the show here. Are there, is there anything else you wanted to touch on briefly here before we move into picks?
Kir Kuznetsov (01:13:52.006)
I don't know, maybe I can tell you that we'll be glad to help anyone with infrastructure. Yeah, because we see that a lot of people around us are struggling with that and we urge people to try and think about this just a bit more and we're ready to help with that. Just a small advertisement, but it's really honest here.
Yeah, I mean, I've followed quite a few of the Evil Martians blog posts that have helped me in my deployment struggles. So, you know, thank you for that. But, yeah, let's move into picks. I could go first, just to set the stage here, make it easy for you. Let's see, I've been on, I just got back from sabbatical and.
Kir Kuznetsov (01:14:30.314)
Kir Kuznetsov (01:14:38.993)
Yeah, please do, please do.
I've been playing Zelda, Tears of Kingdom, which is just a phenomenal game. Very impressive. I can't speak highly enough of it. Probably wasted way too much time just on the couch. Outside of that, there was one other thing I wanted to share here. This is really fun. I try and do a funny pick every time.
Kir Kuznetsov (01:14:51.263)
Kir Kuznetsov (01:15:02.289)
that somebody has used AI to have Elvis sing, baby got back, which is a, a popular song from the nineties. But it's, it's so funny. And I'm kind of looking forward to more remixes like this. So, so, so check it out.
Kir Kuznetsov (01:15:23.28)
Kir Kuznetsov (01:15:28.326)
Kir Kuznetsov (01:15:37.786)
Let me save the link. Yeah, I need to save it. Just give me a second. I'll give you my pics. Okay. If we're talking, can I continue? Yeah. Is it, are you finished? Yeah. Thank you. If we're talking about games, I'm not like a huge gamer myself, but I recently stumbled upon this game that is called Rain World.
Yeah, yeah, what do you got?
Kir Kuznetsov (01:16:06.074)
I was fascinated how organic it is, even if it's a pixelated game. I really do want to spend some time playing it. If you haven't seen the game process and the game physics do so, it's amazing. It's a totally different level for a platformer to be like that.
Yeah. And my actual pick for today, I'm recently into some 3D printing and I do believe, it's not like for everyone, but it's like a huge world and it allows my inner engineer to not only tackle kind of software engineering, but actual hardware engineering, and I can give you some example.
For example, I used a keyboard like that, which is mostly 3D printed by me. And the other thing that is amazing that these are the open projects. So it's like real physical open source inside this community. And I don't know how the frame will be cropped. But
I literally have like a 3D printer frame here that I'm assembling right now. Step by step, it will take a lot of time, whatever. But yeah, these are the open projects that you open and you start to assemble and you obviously modify them. And this is huge, like it solves some problems. Like I needed a compact ergonomic keyboard. I did it myself. I needed like...
That is so cool.
Are you printing the keycaps too?
Kir Kuznetsov (01:17:59.826)
Not these exact ones, but on a different keyboard, I've printed caps and they look way better on that keyboard than the original ones. Yeah, because they sculpted in a different way. That keyboard is flat and they are sculpted to make it like 3D, three-dimensional, kinda, so they are more comfortable.
Oh really? I didn't think about that. Okay.
Oh, that's cool. That's great. Yeah, I'm looking forward to that link. I'm gonna have to play with that as well.
Kir Kuznetsov (01:18:32.366)
Yeah, I can probably give you some links to this Keyboards, for example. It's a project by the guy whose nickname is bastard KB. He does amazing job. And I can give you the link to the printer that I am trying to build. Kinda at the moment. These are open source. These are open source bunkers.
I know it's mind blowing.
Kir Kuznetsov (01:18:59.262)
Yeah, I can touch open source. It's not bits and bytes.
Yeah, that reminds me of a I should add another pick here. You just reminded me of Tom's hardware or Tom's guide. They have like a 3D printing like best of 3D printing for the year like roundup. And it's all just stuff you can download and use and like, some of the composition I like I didn't even think like you could build parts of stuff that like fit together and like make like 3D sculptures and stuff.
Kir Kuznetsov (01:19:20.999)
Uh huh, uh huh.
I, you know, the more you're like, you get into like, they have this great roundup of just like the wildest things that you can print all open and you just click download and print it is what is wild. Yeah.
Kir Kuznetsov (01:19:40.606)
Kir Kuznetsov (01:19:45.882)
Yeah, it's a mind-blowing community. I don't know how it can exist, but it exists and it blossoms. Amazing.
Well, uh, thank you for coming on. This was a great conversation. Uh, you know, I'm glad we didn't dive too heavily into the, you know, technical discussion of MRSK because, uh, you know, we, we've done that another episode, but also, you know, I love the, how the conversation evolved, uh, from pricing and investigation and just all of the infrastructure related things that, you know, aren't MRSK that you should be thinking about.
while you're diving in and, you know, it is definitely kind of a breath of fresh air to think about and talk about. So thank you for that. Yep.
Kir Kuznetsov (01:20:31.678)
Thank you so much. Yeah. I'm really glad that you called me to this podcast. I'm honored to be here and this was a great conversation. Thank you for your questions and for like allowing me to steer it a bit to this part, because I had this like thought process. I thought this will be valuable because more people can relate to it.
Yeah, for sure. I mean, I'm not, you know, I'm not hands down as much as you are. So it was great to get your insights on, you know, what people should be thinking about and not, and, you know, what to watch out for sure. So thanks again for coming on.
Kir Kuznetsov (01:21:11.958)
Yeah, thank you so much. Thank you so much. I hope we'll see each other again.
All right, well, until next time. Yep.
Oh yeah, for sure. You know, I, I attend a lot of virtual events and unfortunately those have like gotten less and less, but you know, hopefully I can get out of my house. I've, I've set up my own 3d printer in the background and a laser cutter and I have too many tools now to keep me busy to travel to a conference or something. But you know, we'll see each other again for sure.
Kir Kuznetsov (01:21:40.962)
Sure. Thank you so much. See you.
Alright, sounds good.
Kir Kuznetsov (01:21:49.994)
Do we just drop from now or?
All right, cool.