Returning guest, Omer Hamerman is the Principal DevOps Engineer at Zesty. He joins the show to discuss cost optimization. He dives into the company's strategies on how they reduce cost and share his experience when it comes to managing it.  Moreover, he talks about identifying methods that impact cost optimization and application performance. 

What's going on everybody? Welcome to another episode of Adventures in DevOps. I'm your host today, Will Button, and joining me in the studio as usual, our co-host Jonathan Hall,

 

Hey, hey, hey.

 

live from the Neverlands, and today our special guest, Omar Hammerman. How are you Omar?

 

Hello,

 

doing

 

how are you

 

well.

 

guys?

 

So today we're going to be talking about cost optimization, which given how 2023 has started off seems to be a pretty hot topic for everyone. So before we jump into that, tell us a little bit about your background and what you do, Amir.

 

Okay, so at SESTI, I'm a principal engineer, mainly in the, well, we're a DevOps company, quote unquote, but I do DevOps again, quote unquote, in

 

Yeah.

 

the company. Before SESTI, I used to be a DevOps consultant for something like six years. So I got to see lots of companies, all different sizes, use cases, deployment scales, et cetera, et cetera. But I wanted to be of a team and do something not exactly of my own, but be part of something. And I found my nice place at Zesty. That's

 

Right on.

 

it.

 

I can totally relate to that because I prior to joining Polygon, I was a DevOps consultant on my own and there's just something cool about working with a team of people with similar skill sets. It seems like you, um, it seems like the, the things you're able to accomplish increase exponentially with a team rather than incrementally, I think.

 

Definitely, definitely. And in terms of experience, it's an amazing bootcamp. But at some point you just feel you're constantly being the external part of other teams. And I wanted to be part of it. It's something that I guess, you know, how the saying goes, the neighbor's grass is always greener. So I've always wanted to have everyone already, what the vast majority of engineers have, but I didn't, I didn't want to be a consultant.

 

Yeah,

 

I want to be part of it.

 

yeah, for

 

Yeah.

 

sure. Because as a consultant, you're always, you're like the friend of the friend at the party. You weren't invited directly, you

 

Exactly.

 

just got to go because you knew someone. All right.

 

I'm taking that. Thank

 

Thank

 

you.

 

you.

 

That will help me explain that in the future. Yes.

 

Cool, so let's talk about cost optimization. How do you approach that?

 

All right. All right. Ha ha

 

Yeah.

 

ha, big topic. So, obviously there are lots of approaches. Everything we're going to talk about today, my experience is mainly around AWS, but I believe everything we talk about can be taken to any cloud provider. And really much to, well, that's the biggest debate in the world, I think, whether do you want to be part of the cloud or take things on premise by your own servers, or maybe do some kind of, of a hybrid deployment. That's not what I'm doing, but it's a debate and it must be said. In the way we approach it, we try to structure a plan to begin with. So, well, I need to say that even to begin with. Zesty, that's what we do. We have a line of products. We have our customers save money, run the resources as efficient as possible on different cloud platforms. That's what we do. But internally, other than dog-fooding our own products and using them to save money, whether that's using reserved instances on AWS, buying saving plans to plan our future and get the saved costs that you can get by planning ahead and committing to the future, whether to one year or three years. So we do that. And then another area we specialize at is handling storage. on AWS, that's mainly block storage as in EBS volumes. And what we can do or what we do for our customers is kind of install different, we call them eyes, it's agents basically that can see the storage, understand whether that's reading and writing too fast to quickly understand the trends and being able to dynamically shrink and extend them so that you can save money whether if you're over provisioned and on the other end We all know how systems tend to clog because of huge data or log files or everything around that. And instead of going to downtime, it'll help you just stay alive by increasing your storage. Obviously, that's just the tip of the iceberg. As far as cost reduction go, you can go to, well, to begin with, I'm going back to our eyes, reserved instances. You can make a lot of commitments and a lot of plans where Cloud Resources, if we're again going back to AWS, you can talk about RDS instances, Elastic Cache, there are many ways you can commit to the future by yourself saving plans or reserved instances and make sure you make the most out of the discounts that you can get. And other tiers I can think of, S3 for example, you can have the automated tiering or do it on your own and make sure that things that are less, I forgot the term, but data that you're not reaching as often can be reduced to a lower tier. And we try to do that kind of method with many other systems. For example, Elasticsearch, which we use a lot, there's a way to manage policies, rotate your data, make sure you have these different tiers. There's different data tiers. if that's infrequent access, that's the term I was looking for, and you're not reaching a lot, you can cut down the replicas the data have, etc. etc. I can think of a lot more, but these are just the

 

Yeah,

 

basics.

 

I had a few years ago, I had a surprise addition to our AWS bill of about $50,000 because of an S3 bucket. The company I was working for, we had music licensing contracts with the major music labels in the world. And part of their integration process is... have to download our entire catalog. So we were using a very specific type of music, but the way that their policies were written, they're like, you got to consume it all. So we ended up downloading every piece of music ever recorded from every record label. And

 

Wow.

 

someone who will just, will just leave them as an anonymous person for the sake of this and send that stuff to Glacier until the following month when we got the AWS bill. Thank you. Bye.

 

That's incredible. And Glacier is also, that can also be a double-edged sword because you can transfer your things to Glacier, but I don't remember the exact tier, but if you try to pull them, something sooner than three months, I think, then you're getting kind of a penalty for doing that. And then you don't know how did you actually increase your cost instead of reducing them accessing that. Yeah, so you really need to know your stuff on AWS. And then actually, something that we try to tell our customers or think of internally is how do we take this cloud experience quote-unquote and turn it into something a little bit more autonomous and what tools can you use or utilize or leverage to turn that experience of yours as easy as an end as smooth as possible. It goes to Kubernetes clusters, ECS clusters, running instances, everything is very, manual and if you don't know your stuff like the examples that you just gave you tend to lose a lot of money and that's sometimes it

 

You're

 

can get

 

right.

 

very sad. I have a ton of examples. Very niche stuff like NAT gateways starting to rise two or 10x in their price. Same goes for things you never thought of

 

Yeah.

 

of surprises.

 

For

 

Yes.

 

sure, those are the, I think from my experience, like the big three are EC2 instances, EBS volumes and S3. But then, and I think those are pretty easy to manage if you dedicate the time to it. And that's where you can get the big wins and the big savings. But then after that, yeah, things like CloudTrail, like you'll be in an AWS console and say, wow, I wish I could see what's going on here. And then there's a little pop up that says, turn on cloud trail and you're like, ah, sweet. And then a couple months later, you get hit with this bill and you're like, dang, dude, that would have been cool to include in the pop-up notification too. And then you get hit with this bill. And you're like, dang, dude, that would have been cool to include in the pop-up notification too. And then you get hit with this bill. And you're like, dang, dude, that would have been cool to include in the pop-up notification too. And then you get hit with this bill. And you're like, dang, dude, that would have been cool to include in the pop-up notification too. And then you get hit with this bill.

 

So, so here's the recent example. I went through what you just said and then a few of our engineers said, okay, we've got this Athena thing and that's incredible. And now Cloudray trail have released something called cloud lake or trail lake or something like that that you don't have to use Athena. You can just duplicate the data and run SQL queries on that. And then they started another trail and another trail. And apparently if you log duplicates events management events on Amazon, twice

 

Thank

 

on

 

you.

 

each

 

Bye.

 

one. And then yes, we had the first log, but then they said, okay, cool, you can launch another another log trail here and then query that and that'll be really nice and helpful. And what they didn't include in the popup is what I just said, and we found ourselves paying something like $7,000 for two weeks of usage on another two trails. So yes, tons of surprises and that always gets back to understand what you're doing and read the pricing tab as closely as possible.

 

Yeah, as best you can because AWS pricing

 

as

 

is

 

best you

 

um,

 

can.

 

it's an odd strategy. I'll just leave it at that. All

 

Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you.

 

Enough

 

right.

 

said, yes.

 

And without burning any bridges, let's hope that this episode isn't sponsored by

 

Exactly.

 

AWS

 

Yeah.

 

or

 

Thank you. Bye.

 

this is going to be a really short podcast.

 

Even if it was, it's

 

Right.

 

not anymore, I guess, after that part. Yes.

 

Cool, so let's talk about like some specific tactics for anyone listening. They know that they need to do something with their bill because that's the state of the environment that we're in in this early part of 2023. What's the first thing that you recommend they look at?

 

So the easiest thing to do and probably the smartest thing to do would be start looking at your bill. AWS do a great work by trying to being as transparent as they can with the bill. So that's structured and you can filter that based on different services and regions and understand really what you're paying for. But you have to look at that closely and what I'm saying here is actually doing manual work. But there's no way around it. You have to start by doing that. And you can probably learn a little bit about the cost exporter, and that'll help you gain more visibility into what's going on and understand different trends. And maybe, again, like the examples we mentioned earlier, maybe someone's doing something they shouldn't. Maybe there is an excessive permissions somewhere that helps people spend money more than they should. So I'd start doing that. I'd start looking closely at the

 

Thank you.

 

bill and understanding getting and understanding of what's going on in the system. Then once you've done that, you can probably utilize things like CloudTrail and the Cloud Lake to help you actually run different queries and understanding even further what's going on. As an example, then that way that we mentioned, it's hard to understand where things go. So if you run queries, you can actually understand different parts in the system that are running. Maybe too many queries, maybe you're doing overload of work, maybe you're accessing endpoints where they shouldn't, stuff like that. So I'd start with that. Taking that a step further, you can use automated tooling. It doesn't have to be us. I mean, I'm just saying that I work at a commercial company.

 

But if

 

That's

 

you're looking

 

what we

 

for

 

do.

 

a recommendation...

 

You can go to GitHub. Yes, if you're looking for recommendation, that'll be great. I do suggest you go on GitHub and look for cost optimization. That's probably different categories, and you'll find all sorts of open source stuff from CNCF and not from CNCF. Lots of great tooling that you can install either in your Kubernetes clusters. Maybe it's tooling that you can just run against your cloud platform and understand what's going on, getting reports, stuff like that. At the end of the day, all of that is manual work. Some of it can probably be installed and is a long running workload, for example, I think cube cost, the specific names, but you can install like stuff services that will constantly run and measure your cloud performance or your cluster performance and help you gain again visibility on where you spend, where you can save. But

 

Thank

 

you

 

you.

 

have to do some work, maybe build stuff around it on your own. I'd start with

 

Yeah,

 

that.

 

for me, the thing that I think has been helpful, like you're doing that manual work and then the thing that I've always encountered with every company I've worked for is you identify the stuff that's running, and you're like, wow, that seems kind of high, but then you don't know what that thing does or who built it or where it came from. And that kind of pushes me towards a really strict tagging policy. So that you can, yeah, so

 

That's

 

you

 

a great

 

can

 

point.

 

start breaking

 

Yes.

 

things down, not only in the cost explorer, breaking things down by tag to see the teams that own them, but whenever you stumble across this thing, you can get an idea of who should I talk to before turning this off.

 

Mm-hmm.

 

Yeah,

 

Who's the owner here?

 

right,

 

Who did

 

right.

 

that? Yeah, yeah, very much so. And another thing that comes to mind when you just said that is lots of times we will find the tagging and we will understand what's going on if I'm going back to the NAT instance, for example, or a NAT gateway. That's something that you install on your private subnets for them to gain access to the public internet. And then you see that the costs go way too high. And maybe reaction can be, ooh, maybe we take that down, not actually use a service, install that on an EC2, build our own gateway. As it used to be on Amazon, that was the only option. And that will save us some costs. There's a hidden cost in cloud architecture, which is engineering times, engineering time and hours and salaries. And I don't think people acknowledge or managers acknowledge that enough. You pay salaries and you pay salaries for what you're doing, obviously create some kind of build automation and basically improve your infrastructure and improve your workloads and help you deliver your product. And if they'll be constantly working on stuff that can be mostly managed, maybe it's not the best decision, even if you save 100, 200, $1,000 a month. So I'm not saying don't do it. I'm just saying consider this idea. When something isn't managed, it not only needs to be done. it needs to be maintained and monitored and backed up. So obviously, and that gateway is an easy example. Take that to databases. If you're not using RDS because that's expensive and you start using your own MySQL instance, you'd have to patch it and update it and rotate it and monitor and backup and snapshot, et cetera, et cetera. So. So. So. So. So. So. So. So. So. So. So.

 

Yeah, yeah, because all of those costs do add up. Even if you choose not to do it, if you choose not to update on a regular basis, you're not getting out of that cost, you're just delaying it to a future date whenever it will, it will like present

 

Yeah.

 

itself to be for that bill to be paid, so to speak.

 

Yeah, and again, if your only lens is the AWS bill and you will get rid of your RDS for an EC2 instance, you obviously your bill will shrink. But I think it requires looking at the bigger picture and understanding that there are engineering hours invested into doing that workload. And a lot of times people tend to look through this lens of a budget. So if I'm part of the R&D department, I'm only looking through the lens of that budget. about salaries because that's maybe HR

 

That's

 

or finance,

 

somebody else's

 

right?

 

problem.

 

Exactly, exactly. And we smile here and we joke about it, but I've seen it happen sometimes, especially while being a consultant. And it's a little bit sad to see because at the end of the day, we're the same team, same company, we try to deliver the same product, and we're doing that for the company to succeed and make money, right?

 

Yeah, for

 

So,

 

sure. That's one of the rules

 

yeah.

 

that we have over at Polygon is we're only gonna spend time and resources engineering things that are unique to the thing we do for a living. If it's a commodity thing, then we're just gonna buy it from the experts for that commodity.

 

So I really tried to follow that buy versus build. And sometimes that's even hard, well, not even, I guess that would be straightforward. It's hard to explain to founders and C-level managers because they want

 

Yeah.

 

to reduce the costs and they look through specific lenses. So something I do try to use a lot. Utilizes many services as I can for many reasons, not only for costs and for that. I mean, it patches up security issues, networking issues. utilize better. I mean, it's their thing, right? It's not our core product to build databases or manage them. It's not our core business to run servers and handle them in some ways it is. But mostly you want to kind of outsource what's not your core business.

 

Yeah,

 

That's what

 

agreed.

 

I believe.

 

Jonathan, what do you think? Build versus buy? Because you're a pretty heavy duty builder. Buzzer. Buzzer. Buzzer. Buzzer. Buzzer. Buzzer. Buzzer. Buzzer. Buzzer. Buzzer. Buzzer. Buzzer. Buzzer. Buzzer. Buzzer. Buzzer. Buzzer. Buzzer. Buzzer. Buzzer. Buzzer.

 

Yeah, yeah. Can I build it? Yes, I can.

 

Ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha

 

Ha ha ha.

 

No, I actually,

 

Thank you. Bye.

 

whatever possible, I like to buy. I mean, I work with a lot of small companies where it usually just doesn't make sense to build something on your own. You know, a question of what it is, you want to build your core product, generally speaking. But yeah, I'm a big advocate of outsourcing anything possible. And you know, kind of my rule of at least a quarter of a person's salary or maybe half on a tool, don't build it or don't even manage it yourself. Let's talk about managing something simple like GitLab. You couldn't run

 

Mm-hmm

 

that yourself, maybe in the cloud, right? I'm not talking about in your server, in your server closet, but you could also hire GitLab to do that. And it's almost always cheaper in the long run to hire GitLab or someone else to do because

 

Exactly. And it would be rather easy to install

 

It is.

 

it on your

 

I've done

 

own,

 

it.

 

right? But what happens in the day that that thing

 

Well,

 

is down? Nobody's working.

 

that's a thing, but honestly, the bigger thing is that when you have it installed on your own AWS instance or whatever, then you're responsible for security updates, which happen all the time. Or you have not even security updates, just general updates. GitLab puts out new

 

Yeah,

 

stuff

 

an axis?

 

every few

 

Mm-hmm.

 

months. And unless you're happy using the old stuff from six years ago, you want to keep up to date with that. And that's a lot of toil. And I was responsible for doing that at a previous company. That's why this example comes to mind. And it was a huge waste of time for me and others to be managing GitLab when we could hire somebody. I mean literally, I think we hired GitLabhost.com or something like that for 200 bucks a month. That's two engineering hours, less than that really when you consider benefits and other

 

Mm-hmm.

 

costs of hiring somebody. Unless we're spending less than an hour a month doing that, we're throwing money down the toilet. And we were spending a lot more than an hour a month. you.

 

Yeah.

 

And it's so hard to explain, so hard to show and present people that that's what they're doing. But yes, I totally agree. Yeah. I can take it to another level and speak about Kubernetes a little bit

 

Oh, let's

 

because

 

do it. Everybody

 

you have to mention

 

right?

 

Kubernetes at some point. You can go through one hour speaking about infrastructure without mentioning

 

Well,

 

Kubernetes.

 

if we did, they would fire us as the co-host of the podcast.

 

Right.

 

Yeah.

 

Why does it call this Adventures in Kubernetes? It's just a little bit of a joke. I'm not sure. I don't know. 

 

So one thing we kind of saw along the way that, well, Kubernetes helps us all. It's not, it has a good reason for being that successful. And that's for how well it utilizes containers and being able to orchestrate them and manage different, you know, taking the nodes liability onto its own and having something manage it all for you. Again, quote unquote, you have to work a lot to make it happen. Even if you are using a service like EKS or AKS or GS. but still a lot of the work that we use to have as engineers with all the wiring and networking and security management, everything kind of, it's kind of offloaded onto the cluster as long as you know how to use it. But then we are starting to see another issue where we've kind of offloaded so much onto the cluster that we tend to forget about resource utilization. And when I say that, I mean, are we using the best nodes we can? source allocated are the, I mean, the resource, different resources allocated to different pods. Is it on the best threshold? Is it on the sweet spot? Are we over provisioning? I'll take that again to storage, which is even if kind of adding another layer of obstruction, because when you have just EBS volumes, it's easy to see on Kubernetes that's turning into the PVCs. And then it's not harder to see, but you do need to have something in place monitoring that. be that Prometheus or whatever other type of monitoring system, but you need to take a look at it and understand whether you're utilizing your resource or not. And we as a company, we look at our customers and we see that just, that's just isn't the case. The vast majority of customers don't really care or aren't really aware of what's going on, especially when it comes to storage, but also when it comes to pods and containers, resource utilization. So that's another point to think of. And I'd say you first want to, at least first have a monitoring system, just see what's going on, have the visibility, so that when you're asked, you can provide answers and shows the managers or the engineers running it what's going on, or even engineers, developers, they wanna know why their application is failing, maybe find a memory leak, maybe understand why the CPU is failing to run their application. So at least having monitoring in place, that's the base level, and then from there, maybe you can apply some tooling to dynamically change the resources says, we talked about elastic storage management, like we provide things like that. So that's another huge money pit where companies tend to lose money and then pay a lot of cash without actually understanding why.

 

So when it comes to that, do you have a preference as far as one large Kubernetes cluster to handle everything, or do you have Kubernetes clusters that are dedicated to different teams or environments or segmented some way?

 

That's a good question, a good approach question. I think my approach, it's a hybrid answer

 

Yeah.

 

and a hybrid solution. When it comes to environments, I do try to separate things in all levels, not only in Kubernetes, even when it comes to AWS accounts, just because of the blast radius and the potential harm

 

Yeah.

 

that people can do, even me, right? We tend to run processes that can be really harmful daily. And when you, your kind of removing part of the risk. It's easier to avoid issues and to create human errors. And actually when it comes to segmenting applications and stuff that are kind of encapsulated within the same environment, I tend to, at least in the Kubernetes context, I tend to go for namespaces and that kind of separation. Obviously this comes with RBAC rules that segment the access teams or groups have different resources on the cluster and their permissions to do stuff. That's kind of how we manage stuff.

 

Right on, yeah, I like the Kubernetes RBAC model a lot. It's of all the different permission systems I've worked with throughout my career. I think that Kubernetes, it's actually reasonable. Like you can read it and understand it and implement it and have a fairly high level of confidence that the rules are being followed the way you think that they're being followed.

 

Yeah, definitely. I wasn't a big fan to begin with, only when I got to really understand it, I can only agree. Yeah, but it has to come with that. In many places, while being a consultant and now with customers, I see that people just neglect to manage that

 

Right,

 

completely.

 

run everything in default.

 

Everything is just star, star, and then just go ahead and do whatever you want. And just there's a long list of users that can access the cluster, but it ends there, right? no segmentation, that would actually force you to create a lot of clusters. And if we were, if we're talking about cost reduction, cost allocation, what happens when you manage a lot of clusters, at least if you're consuming it as a service, you're paying like you're billed by the hour, I think, at least with the EKS and AKS. Uh, so that's another thing to consider. Just if you can, you can manage one large cluster as long as you're managing the access correctly. And by the way, the, the larger the cluster gets, I guess it gets more efficient. because if you're running two applications, well, you shouldn't run two applications on a Kubernetes cluster. But if you are, it's not as efficient as running 200 or 2000 pods. I think as it grows, the efficiency

 

Yeah,

 

grows with it in some way. And

 

for

 

so,

 

sure, for sure, because you can just, like there's a, there's just economies of scale.

 

Yes,

 

So you mentioned something,

 

exactly.

 

it's outside of Kubernetes, but it's a really solid point I wanna bring up for cost tracking. You mentioned segmenting AWS accounts, and I think that's an underrated feature of AWS is using control tower and having separate AWS accounts for dev and staging and production.

 

Mm-hmm. Yes, so that's, again, the term I always have in the back of my mind is blast radius. And the way we work with AWS credentials, by we, I mean all of us, that's the way it builds, whether you're running AWS CLI locally or one, you know, working with Boto3, you have to kind of provide the certain combination of credentials and that those credentials to a region and an account. And you're limiting yourself by design to work against a certain workload. And then it's kind of hard to make mistakes. It's hard there to make mistakes, right? I try to keep it as visible as possible. But I mean, I have a plugin on my CLI that when I run against a specific set of credentials in a region, it's highlighted in bright orange that I can understand what I'm working against. Because if I'm running, you know, deleting a bucket or I don't know, a Route 53, zone and I'm doing that in production, I want to understand that

 

Thank

 

exactly

 

you.

 

what's going to happen. I want to know the environment I'm working against and if you have everything working on the same account, you need to be far more diligent with the exact values that you provide and you need to be a lot more careful and it's easier to make mistakes. So I have a nice story about that. When I used to work as a consultant, it was I think seven years ago, we had this Jenkins cluster that used to manage everything. We had separations of accounts, but this Jenkins cluster had

 

Thank

 

access

 

you. Thank you.

 

to everything. And we had a nightly job that deletes CloudFormation stacks. And one of the engineers just ran it and he didn't provide what to delete in terms of the environment. And then this, I don't know, script was designed in a way that if it doesn't have an input, it just

 

What?

 

uses star. And this thing that started deleting stacks all over the place, we did have like, 15 or 16 accounts, but they were all starting to delete themselves. And it was stopped midway, but you can imagine the blast radius of that. And how much we had to work to rebuild everything. And when you have different, even a CI, um, CI servers or systems that, uh, serve a certain environment, we try to duplicate everything, even though it's harder, right? Six secret management, CI management, CI runners, everything around that. It's, it's, it's kind of hard. don't really want to do the work because you have one thing that works. Why won't I just peer everything together and let that do its thing? And that where plast radius comes in, you can make your errors and you can even feel more confident in making mistakes and failing faster when you know that you're kind of segmented in this way and your bounds are very limited. That's what I'm trying to say.

 

Yeah, it's like having a set of guard rails in place. You know you can go a little bit faster because if you lose control, you're gonna hit the guard rails rather than careening off the cliff.

 

very much so, yes.

 

Whereas they say, your ability to go fast depends on how good your brakes are.

 

Yeah, right.

 

Yeah, it's basically a breaking system. Yes. And I also wanted to add that it's, it feels like we as DevOps or ops engineers, we do that to protect developers, but not only developers, it's ourselves, right? We're doing so much work and we are the ones that usually utilize the admin credentials and we can do anything up to deleting an AWS account. So I'm kind of protecting me in the process just as well. It's not only developers and other engineers. It's a process that helps everyone. So that's how I feel confident working.

 

Yeah, for sure.

 

So that's how I feel confident working.

I would like to talk about the, I'm sure you've thought about this before. You probably talked about it before, but DHH made a big deal about leaving the cloud recently to save money.

 

Hmm,

 

I'm sure

 

yeah.

 

you've read his post on that. I don't remember the numbers, but he made

 

Definitely.

 

these some pretty outrageous claims about how much money they're saving by doing that. What's your take? I'm sure you've thought about this before. You probably talked about it before, but DHH made a big deal about leaving the cloud recently to save money.

 

If it helps, he's probably

 

I'm

 

never

 

laughing

 

going to

 

because...

 

listen to this podcast.

 

Right. Ha ha ha ha.

 

Oh yeah, that helps a lot. I'm saying that because I had a similar take back like five, six years ago and I stood on a stage on Tel Aviv and I said something about the cloud versus running on-prem. There were two huge companies sponsoring the event which they weren't a

 

Thank

 

big

 

you.

 

fan.

 

Bye.

 

Let's just say that. And yeah, it got

 

What

 

a

 

did

 

little

 

you

 

heated.

 

say?

 

But anyway, it brings me back to the hidden cause. of engineering hours. I think, yeah, great. Of course, you can save money by removing yourself from the cloud and deploying your stuff on physical servers that you have in Iraq somewhere, maybe in the offices, maybe in a shared space. There are a few issues with that. First of all, you need to have the in-house skills to manage that, which is not very common today, I think. I only see cloud engineers coming to interviews. I don't know whether that's a bias I have, interview, but I don't meet a lot of, you know, the old operations guys managing physical servers in the storage house. So you need to have the in-house scale not only now, but also in the future. So it probably comes with a lot of education and building structuring teams inside. That's for one. And the next is, I always ask the question, okay, but what about scale? So part of your scale, of course, you can anticipate. What happens with the parts you can't? And then one valid answer can be, okay, I don't buy the 100% I need in my physical rack in the warehouse or in the basement. I buy, I don't know, 50% more. And I say, okay, so you're actually paying for that to begin with. So you are trying to anticipate something, which you're not really, you don't really know what. So it's interesting. And I read his post and I still couldn't get my head, my head, sorry, around the idea. How do you make those calculations? Fine. You made the bottom line. You made the math and it makes sense for you to move away. But how do you anticipate the scale? How do you know what spikes are coming? How do you know if you're going to have this, especially for startups, right? We can sign a deal tomorrow that triples our workload. Or we get a 10x traffic. I cannot live with a rack in the basement to handle that because I don't know how much to purchase. Maybe at the size of the DHH's company it works. I guess for startups, very much not the case, at least nine out of ten times. That's a good question.

 

Yeah,

 

I tend to

 

I...

 

agree. I mean, my biggest complaint about his post wasn't even the content of the post. It was more the way people interpret it. And I think some of that might be his sort of cult of personality that surrounds him, that people read him and they think, oh, I have to do what he does. You know, the sort of, you're not Google syndrome.

 

for a good reason. I mean, he has his... for a very good

 

Yeah,

 

reason, but still...

 

but yeah, what he said, I'll give him the complete benefit of the doubt that it makes sense for his company, but it doesn't make sense for most companies or at least not all companies, maybe some. I'm still not convinced that it, I mean, when I look at the numbers, like those numbers don't quite add up to me. Like I didn't see him talking anything about the salaries of people managing his servers. I hear him talk about the equipment cost and bandwidth cost and that was

 

Yeah.

 

it. And like I didn't see anything or replacing them when the hard drives crash and all those sorts of things.

 

It's

 

Yeah.

 

called math and that's, it's hard. There are different aspects. Even just my engineering brain told me, okay, what if I want to create some type of automation? Right, I'm building now, I'm literally these days building a product to support storage on top of Kubernetes and Kubernetes in the cloud because obviously I have to speak to AWS's cloud with their APIs and the SDKs for all sorts of different languages. What happens if it's, you know, on-prem resources, on-prem system, do I need to start speaking to that? as AWS or Microsoft or Google's probably not. I don't know. It feels like I wouldn't want to be part of the team. And if I would, it would be for high salaries reasons, which I don't know if these are even scalable. So yeah. So yeah. So yeah. So yeah. So yeah. So yeah. So yeah. So yeah. So yeah. So yeah. So yeah. So yeah. So yeah. So yeah. So yeah. So yeah. So yeah. So yeah. So yeah.

 

Go.

 

Yeah, that was the big takeaway I got from it as well. His logic seemed to be very specific to his exact business and was not something that could be broadly applied to tech companies in general. And then the other part, I just had flashbacks to driving out to the data center at 2 o'clock in the morning and grabbing my coat and going in there and pulling servers and popping those tiles floors and it's like dang did he talk with his team before signing them up for that? I mean that's assuming that they even

 

I mean,

 

live

 

just think

 

near

 

of the...

 

wherever their data center is.

 

Yeah. And just think of the ops engineers, right? You're used to work with AWS for what, five, six, seven, eight years. And then C-level gets you into a meeting and they say, okay, we're going off the cloud. I think the first place

 

Yeah.

 

I go is LinkedIn. I'm going to look for another job. I'm not even thinking about it. That's not an option. And this is the type of engineers that wouldn't have this much problem to find a new job. So what are you doing? How are you harnessing them into this adventurous

 

Yeah.

 

project? I don't have the answer. The first thing I wanted to do is to ask the engineers kind of hearing their standpoint. I don't know.

 

Yeah, because for me, that's a long forgotten skill set, like the skill set of building out a sand array and putting in the monitoring in place to identify failed disks and having an appropriate number of redundant disks online. Like, I ditched those skills years and years ago.

 

Yeah.

 

Yeah. And what about the physical skills, right? IT engineers used to be those people that fix wiring and electricity and ventilation. What happens to them? Today, I think they mostly deal with managing, you know, Google directories and fixing physical stuff or hardware within the office. What happens to those skill sets? That's

 

I built my

 

a good question.

 

desktop computer a couple of years ago. I built a new one. And it was such a weird experience. I used to build computers all the time, right? You know, I mean, back in the late 90s, that was the thing.

 

Mm-hmm.

 

You know, oh, look, I built a Pentium II.

 

Yeah.

 

And now, you know,

 

All

 

I'm

 

right.

 

going to upgrade the Miram to 32 gigs or whatever it was back then. I don't remember

 

Mm-hmm.

 

it anymore.

 

Mm-hmm.

 

Everything's changed. Power supply, the power supply has changed. My goodness. And not just the connectors, but like the power supply technology. have a fanless hours applied that I don't think that existed back then. And and memory is changed. And your hard drive snaps into your motherboard these days. What the hell?

 

Thank you. Bye.

 

Ha ha ha!

 

So yeah, if I was going to do this at the servers, I don't I don't have I don't even I don't know what it would look like. I literally don't know what I would expect to see.

 

Yeah.

 

I'm used

 

Yeah.

 

to those you know, scuzzy disk, radar rays and so on. You know, I don't know that I'm sure those still exist somewhere. But what is this? What how do you install 30 different SSDs? I don't even know what that looks like. I have no no idea what it looks like. So yeah, I'm sure I could learn it again. But it's a completely different skill set. I might as well be learning how to use DOS all over again.

 

Yeah.

 

Yeah. Yeah. And there are aspects we're not even thinking about. What about physical

 

Oh yeah.

 

security? The type of access that AWS have on top of their warehouses is unbelievable. I mean, it's tiered and leveled in so many ways. I think the gatekeeper doesn't even know how to enter the main road. And that guy doesn't know the key to the, uh, to the warehouse door. And that guy doesn't know how to turn off the ventilation, et cetera, et cetera, et cetera. What are you going to do? Keep the rack in the basement? I mean, sure you can, the largest cloud in the world. It's a lot of questions to ask, and it feels like it's going kind of against the current, which is understandable if it makes sense to you, but it feels like the world is kind of shifting towards the cloud, and it's a slow shift, but it feels like that's the general direction, and doing the opposite probably has a lot of more aspects we're not even thinking of, recruiting people using tooling. I mean, GitHub is exponentially growing with tooling around the cloud. I'm assuming not around proprietary systems that manage on-prem resources. So those are a lot of questions to ask the

 

I've

 

HH.

 

been reading the book, Learning Wardley Mapping by Simon Wardley. I don't know if you're familiar with the concept, but it is basically a way to map a value stream against, this is really over some application, but a way to map a value stream over, it's a two dimensional map. So you have like the customer is at the top of the map and whatever is it, the farthest away from the customer is the bottom of the map. And then, and then the X axis is the, sort of, of the maturity of the thing. So if your

 

Mm-hmm.

 

value stream is a cup of coffee, then the top is the person buying the coffee, right below that is the barista at Starbucks, and then you have the coffee machine and the coffee grinder, and near the bottom you have the farmers growing the coffee somewhere in Ecuador or whatever. And generally at the bottom of the map you have things that are commodities, and they're far to the right. So the electricity commodity. And it's all, you know, it's clear far to the right. You don't care who you buy it from. The only differentiator between one provider and another, usually is the price. Nowadays, some are, you know, that you can get, you know, is it renewable or not, depending on where you live, you might have that option. But that's the only differentiator, right? And then the farther up the chain, you go, depending on, you know, different things, some things are, are like, they're highly experimental, they're far on the left. And like, we don't even know if this works yet or what it looks like, we're going to try something and see what it works. Once you do that, it starts to move to the right. And one of the examples he uses throughout the book is cloud computing. And when he started coming up with this process of mapping, it was back in the mid 2005, 2007, something like that, when cloud computing was

 

Mm-hmm.

 

just sort of people were thinking about it. AWS hadn't really been announced

 

Yes.

 

yet when he started doing this.

 

I think it was just S3 on 2006

 

Yeah. Yeah.

 

or something.

 

Um, so he's talking all through the whole book using Cloud Compete as an example of commoditization and, and, and more than that, utilization, you know, making something a utility. And I think that, I mean, that does a good job of explaining, I think, the way that industry has progressed, you know, back in 2000, servers were special. We gave him pet names, usually

 

Right.

 

named after Star Wars figures or Looney Tunes characters.

 

Yeah, that's a topic of

 

Thank

 

its

 

you.

 

own,

 

Bye.

 

yes.

 

And now, you know, I don't know if this transition is complete. And just because, and even if it is, um, that doesn't mean that everybody's using the cloud. I mean, some people still have their own solar panels and their own even diesel generators or whatever, you know, so even though electricity is definitely utility, it doesn't mean that in all circumstances it is. So, you know, it's appropriate to have your own data centers in some cases or your own server rack. If you need something really fast, close to your, your, your rendering engines or whatever you're doing. But yeah, in general, the trend seems to be that everything's going to the cloud and you're fighting the trend if you don't. The same way you're fighting the trend if you decide to install your own coal plant in your backyard to generate your electricity. Ha ha ha. 

 

Yeah, exactly. I just thought of five new things that we didn't even talk about, which I mean kind of a branching out of what we're talking about, but you know, cleaning the place, getting new hardware, getting spare parts for our hardware, AC, backup, AC, lighting, so many things that we're used to consume as commodity because I'm only launching an EC2. I don't care about all these things around it. And yes, it costs a little bit more, but for, I mean, I'm not trying to promote Amazon provider here, but for a good reason. There's so many things that come into that price of the easy two instance that I'm launching that I'm not even thinking of. And I probably don't want to think about, I don't want to clean the space around the server. Someone's doing that for me. So yes, very much so. It is a commodity today. And to your analogy with the pets and the names for the servers, there's this notion of running a kettle versus pets, which is probably translating in our world. running containers versus instances or whatever other workload. It doesn't have to be containers, but it's easier to run immutable things and it's easier to get rid of certain servers and large new ones where they're larger or smaller or faster or ones with GPU. It's just like you said, a commodity. And when I'm kind of downgrading again quote unquote, because maybe I'm not, but I'm downgrading myself to running from something that is so much elastic and is again with a limit, in terms of how much I can provision, it feels weird. It feels like going against the current and I don't understand

 

Thank you.

 

it.

 

but it does make for a spicy blog post.

 

Thank you. Bye.

 

Oh, definitely, definitely. If you want

 

Right?

 

traffic, that's the way to go.

 

Well, cool. Anything else we should talk about for cost optimization? or slamming

 

Um

 

on DHH.

 

Ha ha ha ha.

 

No, no, no, I think that's enough.

 

I want to stay under the radar with this.

 

Yeah. No, I think, again, going back to the basic point, it's first, it's most important to understand what's going on in your cloud bill. There's a good reason why there are so many consultants out there and consultants that will only come in to help you reduce costs. And I think some companies, it's even a title. It's a job. Like, being the, whatever that is, I think it's already a C level. I forgot the term, but it's the one that manages the Phenops, right? The Phenops engineer and a Phenops manager, and maybe there's a Phenops C level.

 

Yeah, and I think that's

 

Yeah.

 

a really good point to not necessarily end on, but that's a great takeaway. Whenever you look at your bill, it can seem overwhelming and it can look like there's some big wins or it can seem really confusing. So like bring in somebody external who's done this a lot because there's a lot of hidden places where, you know, things may look expensive, but actually that's a reasonable amount or things that look reasonable to you that who's done this multiple times may say, no, you're paying way too much for that. So bringing in someone to take a look at it can be a fresh set of eyes to help you gain perspective, especially after those quick wins. Cause in my experience, you know, I've always, it seems like I've always looked at the AWS bill and there's been something at the top that's way, way expensive. And it's like, oh, okay, well, here's a quick win. But then like once, once you knock that out, bringing in expert help can be really effective, especially like something we talked about early on but then didn't come back to with the savings plan and the reserved instances. Those are really, really tricky

 

Mm-hmm.

 

and you can do them the wrong way where they actually end up hurting you in the long run.

 

Definitely, definitely. I think Zesty was founded around the understanding that yes, you can bring that professional, but what happens in areas that you need that professional every month or every week, why don't you utilize something that's installed probably costs less than getting that professional and dynamically shifting your infrastructure as you go. And your infrastructure doesn't have to be nodes and pods, although that's something we do. That's also reserved instances and saving plans and utilizing them, telling them and having everything running autonomous for you. So you don't have to worry about it. And I think that's how the, at least I want to see the future cloud experience. That's what it should look like. It should be as automatic as possible and providing you with all those services. They're honestly from AWS perspective are rather easy to develop and provide as a service. I understand it's not their main goal because they want to make money and not save money for customers per se. But I hope at some point we'll get to a good enough baseline that's easy to work with without managing those hassle tasks running around and trying to understand what's what's popping

 

Yeah.

 

now in my build. I hope that'll be the future.

 

Cool. Right on. Well, should we move on to some pics?

 

True.

 

Jonathan, have you got a pick for us this week?

 

I do. I'm going to pick the book I just mentioned.

 

All right.

 

It's at warley maps.com. It's a free book. You can download it in English or Spanish. There's also other resources there.

 

Is one of them how to learn Spanish?

 

And it's a really cool I mean,

 

Is one of them how to learn Spanish?

 

sorry.

 

Ha ha ha. 

 

I don't know. I don't know. I don't know.

 

Ha ha ha.

 

I don't know. I don't know. I don't know. I don't know. I don't know. I don't know. I don't know. I don't know. I don't know. I don't know. I don't know. I don't know. So I recommend it. If you don't want to read the whole book, there are other resources there. You can watch videos by Simon Wordly or you can find him interviewed on podcasts. I recommend it just about everybody become at least familiar with the concept of a Wordly map and what it does. So that's my pick for the week.

 

right on. I bet you, Omer, got a pick for us.

 

Yeah,

 

Alright.

 

two, actually. One is pretty standard. I think Netflix's book No Rules Rules, which is amazing, at least to me. So many concepts that you'd never think about of talent density and how to manage people and how to manage people's approach towards the company and towards an organization, how to structure teams. It's an amazing management book. And just bring in new concepts. never thought of. So that's a book and another cool thing, kind of a niche thing, I need to transcribe a lot of content for many reasons. YouTube podcasts, stuff like that. And there are many paid resources online, but apparently OpenAI, which are responsible to chat GPT, have open source a tool that's called Whisper. And Whisper is an easy command line tool. It has different models for all sorts of languages, but with English it has even different sizes of models. So it's very easy, very fast, and you can just transcribe, and it works a lot better than online paid services. And I think most of them are actually trying to adopt it now because it's so far more accurate than the others.

 

Oh wow.

 

So I highly recommend it to anyone looking for that kind of service, yes. Yeah. 

 

Ah,

 

Yeah. Yeah.

 

sweet.

 

Yeah. 

 

Definitely going to check that out. Cool. For me, my pick this week is going to be a book. I feel like I've been picking a lot of books lately, but I'm not going to change that today. I just finished reading this book called Born to Run by Christopher McDougal. And it's actually a really, really cool story that's like, it's about running, obviously. Big surprise there. But it's about this tribe in Mexico, Mexico called the Tara Humara that they stay away from civilization and they're just these mysterious endurance runners and so the whole story was how this guy heard about these people and then went down to Mexico trying to find them and was able to find them and he puts together this race down in Mexico of like the top endurance runners from the US against this tribe in Mexico. And the whole story of how he pulls it off and how it almost doesn't happen. And the area of Mexico where it is is like the homelands of the cartel. So they're trying to navigate Mexico without running across the drug cartels. And...

 

probably running barefoot

 

Yeah,

 

too, right?

 

yeah. It talks a lot about that.

 

about.

 

And that was one of the cool aspects of it. I like this style of book because it was like one chapter would be an entertaining section of this whole story. And then the next chapter would be like an educational component where they talk about the different

 

Thank you.

 

shoes

 

Bye.

 

and running techniques and endurance characteristics. And then back to the story. And so you just flip back and forth where you're entertained

 

Wow, that's cool.

 

but the story itself is super cool, very, very entertaining, and I'm surprised it's not a movie now, or if it is, let me know and I'll check out the movie too, but Born to Run, it's a really, really good read. So even if you're not a runner, it's just a super cool story about how this guy goes wandering through the Mexican desert to put together this race.

 

I don't know if it's a movie, but it's a

 

Alright,

 

song at least,

 

I'm intrigued.

 

right?

 

Yeah, Bruce

 

song?

 

Springsteen,

 

Oh yeah.

 

right? Yeah, the boss. Alright, cool. Well, I think we have ourselves an episode. Omer, thanks for joining us. It was great to have you on again. And look forward to having you back

 

Thank

 

again.

 

you guys for having me, it was lovely. Yeah,

 

Alright,

 

hopefully.

 

cool. Thanks for listening everyone. We'll see you all next week.

 

Next.

 

Thanks guys.