Leon Wright is a Senior DevOps Engineer at Pet Circle. Omer Hamerman is the Principal DevOps Engineer at Zesty. They join the show alongside Will to talk about the similarities and differences between AWS and GCP in DevOps. Additionally, they discuss their experiences working and launching projects as DevOps Engineers for their respective companies.

What's going on everybody? I am today's host for Adventures in DevOps. I'm Will Button. I've been left here solo and abandoned by Jillian and Jonathan. They both have prior commitments and for whatever reason, they trusted me to run the show solo. So we'll check the wisdom of that decision here by the end of this episode. But joining me today, I have not one, but two guests on the show today and this is going to be a cool conversation. I have Leon Wright. Leon, you wanna give us a little bit about your background?

 

Yeah, hi, I'm Leon in a late night time zone over here in Perth, Western Australia. Senior DevOps engineer for a company that's on the other side of Australia, Pet Circle. I work in most of the DevOps team and spend most of my time shouting at things in GCP.

 

right on and our other guest for today, Omer, how are you?

 

Yes, hi, thank you for having me. Omer, I'm based out of London, working for Zesty, which is actually in Israel. Principal engineers, spending most of my time in AWS and systems around that, building the product that we deliver to our customers.

 

Right on. Cool, so today's talk, we're going to talk about AWS and GCP and the differences and similarities between the two. Because our guests today, between the both of them, have quite a bit of experience across both of those. And so it's going to be a cool chat to. To just understand the differences and similarities between the two and hopefully walk away with a little bit more info that may help you in deciding where you want to host your cloud services, or it might be enough to scare you away and say, you know what, forget it. I'm just going to build my own data center.

 

Or rent one from

 

Not

 

AWS.

 

a wise decision.

 

Right?

 

If you've got enough money for the... you can.

 

Bye.

 

Time and money, I think, yeah.

 

Yeah,

 

Ha ha ha.

 

yeah,

 

Yeah.

 

yeah. Now you can get a rack from AWS but I did look into it because Perth does not have a local point of presence and yeah, yeah, too much money for the previous company I was working for.

 

Yeah, I did that recently. We were considering looking at a bare metal server to do some really high-end load testing to see what the cost performance of the hypervisor was. And to meet the specs that we needed, the AWS bare metal server we were looking at was $18,000 a month.

 

Poor boy. Ha ha ha

 

Yeah.

 

ha ha.

 

Needless to say, we were like, ah, you know, we're OK with that vSphere or the hypervisor

 

Yeah.

 

performance penalty. Yeah.

 

Ha ha ha 

So let's talk about AWS and GCP. One of the things we were talking about before we started recording was each of them has their own built-in deployment tool. For AWS, it's CloudFormation. And for GCP, I didn't even actually know this till we started having this conversation. They have GDP, is that what it's called?

 

GDM, Google Deployment

 

GDMA.

 

Manager.

 

Yeah.

 

I don't know how maintained it still is. I know they're still maintaining it or keeping it running, but I do know that Google themselves have included Terraform snippets in their documentation now. It's in the, you got like the console, the CLI, and then you've got Terraform as a tab of how you can configure a thing. So that's just...

 

That's an interesting sign for something.

 

Yeah, it kind of like,

 

Hahahaha

 

early on I'm like, oh man, should we be going down the Terraform path? And I'm like, because I had to make a choice. And then I started seeing it in the documentation and I'm like, yeah, I can't guess that kind of settles it. You know, GDM's not there, but Terraform is. Ha ha

 

Hahaha

 

ha.

 

Hahaha! So on the AWS side, we have CloudFormation.

 

Yeah, yeah, okay. So CloudFormation is the solution by AWS. To be honest, I'm actually using Terraform for most of our high-level stuff. Our developers, by extension, use CloudFormation. By extension, that's because they're using a wrapper for functions that's called serverless. So it's a framework that you kind of declare what you wanna deploy using YAML files, and that's creating under the hood a CloudFormation template. So it's kind of nice because we have the segregation between Terraform that manages our, let's call it DevOps infrastructure and to whatever is going on with developers applications and functions in the cloud.

 

Yeah, I think.

 

Right, so you're using Terraform for the infrastructure and then the development, the code that the developer ship gets built and deployed using CloudFormation.

 

Yes, I mean, part of the infrastructure actually is being deployed with CloudFormation, but it's right around the application, like the application themselves, the API gateways, whatever else that's hooked to the application and isn't like high level infrastructure.

 

Gotcha. Are they using something like AWS SAM for that?

 

No, so Sem is AWS as a framework. We use serverless.com. That's the framework we

 

Okay,

 

chose to work

 

gotcha.

 

with. So it's a little bit different. Yeah.

 

Gotcha. And so you guys are using serverless, obviously, as hinted by serverless.com.

 

Yes.

 

I'm quick. You know,

 

Yeah,

 

it's

 

yeah.

 

kind of

 

Kind

 

early here

 

of

 

in

 

early

 

this

 

here

 

morning

 

in this morning. Ha

 

for me, but

 

ha ha.

 

just give me a few minutes for the coffee to kick in, and I'll catch up to speed.

 

Yeah,

 

Alright.

 

I had to make

 

So.

 

a choice. So do I have coffee and not sleep and be alert? Or do I not have coffee and just be a little bit wired

 

Yeah, right?

 

for the episode? Ha ha ha ha

 

Do you take the penalty now or in two hours when you're really wishing you were asleep?

 

ha.

 

Hehehehe

 

Yeah, exactly. Ha ha ha ha ha.

 

For sure. So in the Google world, what is the deployment framework for deploying serverless functions over there?

 

You've got a few choices. A lot of the examples will have cloud build. But it's very tied to Google source. I think you can use it via GitHub Actions. I haven't really looked into that side of it, so we're using pretty much Terraform all the way down. Via the CDK.

 

Gotcha.

 

Or the Terraform CDK, that is.

 

Okay, right on. And you all doing serverless functions as well?

 

We're I mean it's the platform itself was a sort of a lift and shift through sort of like on prem to Hosted to GCP. So there's still a lot of you know, VMS But we're growing out in you know cloud functions and cloud run and all of those other sort of serverless technologies as the platform grows and matures and developers get used to that cloud first way of doing things.

 

Are you running containers at all in between? Sorry to barge in out

 

Ah,

 

of interest.

 

no. The Cloud Run is containers. The Cloud Functions are kind of containers underneath you. You can even, the way, I mean, GCP is kind of Kubernetes. from the bottom up. Even our VMs, underneath, I'm pretty sure that Borg is essentially

 

Mm hmm.

 

Kubernetes. The VM boot up time compared to AWS is like seconds.

 

Right,

 

You

 

okay.

 

turn on a VM. So the driver to containers is less so in

 

Mm-hmm.

 

GCP, I found, when I made the switch. There's still benefits to running containers. I wish we were doing more of it. personal goals is to have more containers, but you know, that the journey for DevOps with the software engineers and developers is sort of like, just take them on the journey rather than just like, oh, here's a container, get on with life. Ha ha ha.

 

the

 

Yeah, that's a really interesting point to me, especially since you mentioned y'all did the lift and shift from on-prem to VMs to now trying to do functions and a serverless type model. There's really, like the way that you write code for those is very different. I've always felt like you have to consider how your application is gonna be run. in production and that is going to influence some decisions that you make when you're actually writing the code. What's been the transition for educating and collaborating with the development team to work with that model?

 

Uh, there's been like, it's sort of, I guess, kind of picking your battles, right? Uh, you, you'll get brought a problem or questions about how do we deploy this? And we're like, well, do you want to do it that way? Have you thought about doing it this way? And then sort of like really putting that, uh, uh, sometimes having some quiet, robust discussions around that and not always like winning those discussions, but sometimes. uh... the the challenges being made taken up and then really run with. So some of our biggest successes in the last year have come from the we want to do a set of VMs and going, well have you thought about doing it via some a bunch of cloud functions and some PubSub? Because this seems like a very event-driven, ripe for that kind of that was absolutely like flawless. I went to, it was for a, doing our integration with our own fleet for doing deliveries. And I was there for, in the warehouse for release day, just hanging out with a few of the team. And it was pretty quiet, you know? And I've been doing this sort of stuff, like sort of Greenfield's deployment stuff for years and years and day one is never that quiet.

 

Frank, were you a bit nervous? Ha ha ha ha!

 

I mean, I'm just a DevOps guy. I just sit in the background and watch people do the work. Like we have tools

 

Yeah

 

for that, right?

 

Hahaha

 

Like my job was done, the code was already deployed. I was a little bit nervous because it was a new pattern, but the software engineers did a really amazing job. They get the. the credit of taking that up and running with it. I just sort of built the tools to sort of make that relatively easy to deploy, if you will.

 

Right on. I think that's one of the vanity metrics of DevOps is just watching developers do their thing seamlessly and quickly day after day because of the tools that you've given them.

 

How many of you like engineers like, uh, where you're working, Oma? Like,

 

Um, we have a team

 

using

 

of,

 

these

 

I

 

sort

 

think

 

of tools.

 

70 engineers, um, but to be honest, while you were talking, I was thinking about serverless. So it has its pros and cons. Uh, one of the benefits is you as a DevOps engineer, don't really have to do anything. As long as you're monitoring that everything's fine and they don't mess, uh, with the configuration or anything funky is going on. They just build it on their own. And it's super easy to just create your own YAML deployment. Uh. run serverless deploy or whatever other framework you're running and everything just magically happens. And there's nothing much to handle. So it's been built for you and you can focus your time and energy on building infrastructure, managing, I don't know, secrets management and all other components. That's pretty nice. Um, that said, serverless can be a problem at times. Um, Like Will just said, you need to know your environment. You need to know where you're running. You need to know the limits. Serverless is nice, but it's not a container or a VM. It has a limited runtime. You're paying per second. It has other limitations, obviously. CPU, memory, deployment times, hot and cold starts, all kinds of other metrics that you, you need to start monitoring and understanding when you move there. So pros and cons. I don't think it's a... one tool fits all, you need to manage your, like spread your workload between things that can run and need to run on functions and other things probably better run on VMs or containers. So we're kind of the opposite way, right? We started serverless only and are trying to kind of slowly offload whatever needs offloading onto containers.

 

What are the key metrics that you're or key indicators you're looking for that's telling you this probably shouldn't be a serverless function?

 

The easiest one is functions that are hitting the capacity of runtime, which is 15 minutes, if I'm not mistaken today on AWS Lambda. But long before that, I think Lambdas that are running for long seconds, let's call it, if a Lambda is running more than 30 seconds, I think it's a problem because it means that it's doing something it shouldn't. And I did find functions just polling queues and waiting for responses from HTTP requests and things like that, but not really the place to do that. And that's a problem of, like you mentioned before, education. Developers are just running their code, expecting it'll work as it should anywhere else, but don't really understand the caveats of the system it's running on. So if you're pulling a queue rather than being triggered by a queue message, it means that you don't fully understand the path and, and it means a lot of education and monitoring. So it's hard to say what metrics, but I can say runtime is the first. The next one is a bit harder to understand and debug, but it's hot and cold starts. So a function is always starting from somewhere. And like Leon just mentioned, it's at the end of the day, it's a container. And container takes a few milliseconds to seconds to start and it'll be kept there alive. So it differs between cloud providers. I think on Amazon, it's something somewhere around 30 seconds that the container is kept alive for the next like the subsequent run. And if you're not running, fine, the container dies. And then a new one. starts, but that's a cold start because it takes time to provision the infra. Whereas if you have a lot of incoming requests one after the other, you can utilize the already created container, maybe use the cache inside it and have a lot of a faster startup time. So if a function is running only in cold starts, probably something's not going all that well over there. Maybe it should be a job, maybe it should be a standalone task, etc.

 

Yeah, I think within GCP, I can't recall across Cloud Functions, but Cloud Runs is an option for us to keep them warm. I think even the Cloud Functions, we can keep them warm. AWS has that option, if I recall.

 

So you have two options, the paid one, you can keep it provisioned. I think it's called provision capacity for lambdas,

 

Yep.

 

but then you're paying for them to actually keep the containers alive.

 

Yeah.

 

So you're paying for the time, the benefit is the performance. The other way are lambda wormers, I think they're called, or function wormers, which

 

Yeah,

 

are just...

 

it was like a little primer that just went and tickled

 

You just

 

it every

 

ping

 

now and then.

 

your application

 

Yeah,

 

and keep

 

that's...

 

poking at it, so

 

Hehehehe

 

it's kept

 

I

 

alive.

 

have had to do that in the past before the option was to do that. Because, I mean, a lot of the places where I've worked the performance far outweighed the sort of cost, you know, we're e-commerce, so customer responsiveness when they're hitting the site is kind of really important.

 

I find warmers the funniest solution ever. Just poking at the lambda in order to keep it alive.

 

Yeah.

 

A friend of mine just wrote a function that is called insomnia for that reason. And... And...

 

That is fantastic, I love it.

 

It reminds me, the warmers reminds me of a long time ago, there used to be this little desktop thing that was like a bird that had the liquid filled in it and it would just tip back and forth. It was actually in a Simpsons episode where he put

 

Yeah,

 

it over

 

yeah.

 

his keyboard so it

 

Yeah.

 

hit the enter key every so often.

 

Hahaha

 

Vent, vent, vent gas or vent nuclear gas or something it was. Yeah, I

 

Yeah,

 

remember

 

yeah,

 

that

 

yeah,

 

episode.

 

that's right.

 

Yeah.

 

Ha ha ha ha. Cool, so whenever it comes to launching a new project. What's your, how does this information come to you guys being in DevOps teams? Do you get integrated with the software teams early on or do you find out about this project as it's getting ready to go live?

 

It kind of depends, it's changed a lot over my time. We've got a architect team now, so a lot of it, we work closely with the architects and we're small enough, it's not quite 70, I think we're around 50 software engineers. It's small enough that we know most people and most people are kind of interested in how to utilize, like the tooling is now the new Shiny and it's sort of proven. people are coming to us going how do we do this and we're like ah yeah maybe go talk to the architects they're gonna have some ideas about it or we are doing a lot of brownfield shifting over so it's just the grunt work of getting off this sort of more manual processes that things were and sort of moving them across because what you could get away with setting up by hand you can't always get away with when you you put it onto IAC and that's kind of one of the big been one of the biggest challenges whittling away those corner cases and the things that have been done in a way that kind of were okay on a static instance that just don't fit when you know it gets auto-healed or auto-scaled or you do a rollover and and all the data goes away.

 

I can share the zesty side that we're probably on the same scale. And I wanted to say that Leon, it's good for you to have an architect's team because on our end, we're the architects. We're deploying the infrastructure and we're the architects. And so, yes, we're kind of involved pretty early on in the process, but we need to take a look at everything. And I find myself going through the infra, but then going through the code to understand that it's utilizing either the container or the function as it should. Again, The good thing, the easy thing, at least in the function side of things, the IAC kind of creates itself. So all you have to do is just make sure that everything's in place. And again, nothing funky is going on, like no security or for some reason, too many open stuff that shouldn't be there. But it kind of creates itself. But we do have a nice utility we just added to Terraform that's called Atlantis. Don't know if you've heard that. of a nice integration that helps you. It manages basically merge requests into your code base. So when a developer needs a new feature or a component in your infrastructure, you can now tell them, uh, dude, read the docs, uh, build the feature, open a pull request and we'll manage it. And then Atlantis will create a plan and we'll try to run it, uh, with a dry

 

Ah,

 

run.

 

okay.

 

Yeah. And maybe fail it in the process and kind of let you. create your own discussion between you and the DevOps or the developers and the ops engineers on what's going on. And then you iterate until it's fixed and it gets to the level that you want it. And then once you approve, it's integrated into the code base and deployed automatically through CI. So

 

Yeah,

 

that's

 

I

 

a

 

think

 

nice thing we added.

 

either Cloudflare or Slack used something similar. I read something, what are the, what, they did a talk, wrote up a couple articles recently. I can't remember which one I read it in, but I have heard one of those companies is using something like that. It might have been Cloudflare.

 

Yeah, that sounds pretty cool. Um.

 

So you're using straight terraform like HCL?

 

Yes. Like as opposed to using the CDK.

 

Yeah, yeah, yeah,

 

Yeah, I

 

I

 

started

 

must profess.

 

looking at...

 

Sorry, go ahead.

 

No, no, no, I mean, it... Pretty new, isn't it? It hasn't been there all...

 

Uh, maybe 18 months, two years?

 

Yeah,

 

Maybe

 

yeah.

 

a bit longer, maybe three years. I know I've been using it for about 18 months now.

 

Okay, so...

 

And it was in very much alpha, beta when I started using it.

 

Okay, you're an early adopter.

 

Yeah

 

Okay, so I remember starting looking at it and it looked pretty cool. And I tend to trust HashiCorp with everything they do. Hadn't failed

 

Yeah.

 

me yet. So we definitely wanna go there, but we're still writing HCL templates, yes.

 

Yeah, it's been an interesting journey. Like, it uses JSII or JSI or JSSI. It's the integration, it's the layer that takes TypeScript to native binding. So you can use it in Python, C Sharp, JavaScript. a whole bunch of different libraries via native bindings. And

 

Mm-hmm.

 

they say native bindings, but it's very, using it, it feels like you're writing TypeScript in Python. Okay.

 

Hahaha

 

That's an interesting experience.

 

Yeah, so because it was such a moving target, I kinda, the way of architected our tooling is that we don't actually use the, CDK until we need the template, right? So

 

Mm-hmm.

 

we build all our objects in Python and we've got some sort of common classes that underpin resources and stacks and when we go synth stack it pulls all the objects in it instantiates them as formal CDK classes and then throws them out to generate the template

 

I have to actually make it easier for developers to integrate themselves into the process or contribute code.

 

Yeah, so It keeps it sort of very Pythonic if you will but it's mostly the DevOps team and a couple of the developers because we're mostly Java in-house so

 

Okay.

 

Coming along to see this this Python they're like this is all a bit foreign and we're using

 

bit

 

You

 

dynamic

 

know abstract

 

to my taste.

 

abstract base classes

 

Yeah.

 

and mix ins and this that and the other thing they're

 

Yeah.

 

like what

 

Hahaha

 

earth is going on here?

 

I'm wondering though, since you are actually using a programming language rather than HCL, which is more of a declarative script, whether you can actually use a language server in your IDE and then just kind of like you normally get all the methods and classes from a certain object, maybe that helps you write the code? Get suggestions? Might be interesting.

 

Maybe, I haven't thought about it too much. So sort of like, it's, there's a lot of work to do. There's just

 

Yeah, yeah.

 

so much work to do. When you don't, like my former role was, you know, basically, it was a startup. There wasn't too much tech debt. infrastructure as code sort of came about as a natural sort of process as as the as we grew it became a necessity because it was uh just myself and uh my boss at the time that we're working on the the team to support you know hundreds of customers in a you know small and it was very consulting it was very uh it was a lot of work um for Tilvis to be managing you know hundreds of that and the other thing and deploying that all and management platforms and ERP that in-house it was a lot so you just had to do it with DevOps

 

Yeah, yeah.

 

to come here where you know we're building all the tooling we're kind of putting the rails down while the trains are still running you know

 

Hahaha.

 

Hahaha

 

And it's just pick the next annoying problem that you need to get out of the way next and work on that. Ha ha ha ha ha.

 

Have either of you used Pulumi?

 

I did a little bit in the past, not much.

 

Yeah.

 

I know it exists and on my article that I wrote that was like, I suggested at least 30 times. Have you checked out Pulumi? I'm like,

 

Yeah,

 

uh, it

 

Pulumi.

 

was...

 

I've used Troposphere in the past, which is like Python CDK, but only

 

Yeah.

 

Python. And I couldn't see the big benefit over using just a native template. So I ditched that, but...

 

Yeah,

 

right?

 

well, I came from troposphere. So I must profess that, or confess that, when I went into infrastructure as code, I didn't see CloudFormation or any of those tools as code, which is a bit judgy of what is code or is not code. But I've kind of taken it literally. And for me, code was a programming language like Python or Perl or... JavaScript,

 

Yeah.

 

something like

 

Yeah.

 

that. And I was like, oh, you know, once we get past this YAML stuff, we'll actually do infrastructure as code. And my boss found Tropersphere. I'm like, oh, code finally.

 

Yeah, code, nice.

 

Yeah.

 

And then I came to this job and like there was actual definitions for infrastructure as code. And, and cause there was no Tropersphere for Google or Google deployment manager. I was like. What the heck? And then I found out, you know, infrastructure as code, the way I saw it was actually called a cloud development kit. And I was just like completely wrong. So it was a exciting and challenging way to start a new DevOps job where you're the one that's sort of like, okay, now I'm gonna build all this stuff and I didn't know what it was called.

 

Right? Yeah.

 

I'm learning. I hadn't used any GCP before and I hadn't used Terraform much before. You know, it was... the way I jump into a lot of things it's like jumping at the deep end, sink or swim, or just flounder mostly.

 

That's the job.

 

See

 

That's

 

you.

 

what I like about it, basically.

 

Yeah, absolutely. That really is the job is you just, there's a saying like, we the unwilling led by the unknowing doing the impossible or something like that.

 

Ha ha.

 

And that's kind of DevOps in a nutshell.

 

That should be a t-shirt.

 

Hahaha!

 

I'm surprised I don't have it on a t-shirt somewhere.

 

Yeah,

 

Ha ha ha

 

yeah.

 

ha.

 

I'll find the exact quote and send it to you. We can have t-shirts made.

 

All right.

 

Excellent.

 

So yeah, Leon, you said something a few minutes ago that I want to highlight because I think it's really, really, it's like one of those hidden nuggets. We talked about how you integrate with the development team and when you get involved. And you said, after we had a few successes, they started coming to us. to work with us. And I think that's huge, because a lot of times when you're trying to do something like implement DevOps or change the DevOps strategy, you have to get everyone on board. And there's a couple of ways to do it. But the one. that you mentioned that I accidentally did at one point was you just deliver this thing that actually works for the developers and they're like, oh wow, this is great. This makes it so much easier for us. And then all of a sudden they just come, they come to find you and say, hey, we're gonna do this. Will this work with that thing that you built versus the alternative where you're trying to roll this thing out and you're having to continuously go out and see what they're doing and change their course of action path.

 

Yeah, I mean, the first year of the job, it was, I thought, you know, I thought it was a case of build it and they will come. And I built it and they didn't come.

 

Ha ha ha

 

Hahaha

 

So, so it was like, I had to change sort of tax this year. And... I realized that DevOps is a lot more about bringing people on a journey, because I was encountering people where, you know, I've done this stuff, I've done it for years, and I've... Especially, you know, a pet circles had an explosion of growth through the pandemic. You know, everyone went out and got a dog or a cat or a pet of some description. They couldn't go to the shops and buy their food, so where did they go? Us. And, you know, that's taken the platform to the point where... it's big and it's important that it stays reliable because people depend on it and You know Growth is a really good way of highlighting what tech debt you have

 

Right? Ha ha ha ha ha.

 

So, you know So there's a and when things processes are manual things aren't as consistent and it increases risk. So, you know, deployments comes with risk and risk is fear. So there's a lot of, you know, very well... the right word but you know the fear was real and understandable you know because gone from smaller to large and it's really important and I can't just lift out my experience out of my head and give it to the software engineers and say hey yeah are we fine I seem overly relaxed about what I'm doing here because you know I've done it for years I've seen it work and I trust the process trust it. But it's really hard to trust if you haven't been living that life. And translating that experience when... has been the thing I've been focusing on this year. And that's kind of worked to pick your champions and even pick your critics. Like there have been people that were probably my biggest challenges that have turned around and gone, hey, no, this is awesome. Once they sort of understood what it could do and they've become the biggest allies from being the person I'll be like. to the person are like, yes, cool, you're gonna be doing that.

 

Yeah, I totally understand that. Definitely when the developer can just click the button and see no problems whatsoever and his code just magically gets to production, it just becomes an amazing process and then there's no other way.

 

Yeah, I think the biggest challenge I've had to, in communicating this, is going from code being separate to infrastructure to, if you're working in the cloud and your infrastructure is not intrinsically something you think about in your code, are you really leaning on the cloud that hard? So it's getting that mindset shift of going, no, my code isn't just code. My code is, it's PubSub, it's a cloud function, it's a VM, it's a memory store, it's all these components, but they all belong with this project. And one of my toolings should be opinionated, I'm opinionated. And one of my strong opinions is config and code live together. Because it's like, unless you're selling a product, right? That's a little bit different. But if you're deploying an internal service, your code and your infrastructure are intrinsically linked. So they should live together. But

 

Yeah,

 

I'm

 

agreed.

 

happy to hear your thoughts on that. And throwing out a random strong opinion.

 

When you say code part of the application, as in code is, sorry, code and the environment are intrinsically one part of the other. Do you mean, how do you mean by that?

 

Um, so... Instead of having a configuration repo for your service or services,

 

Mm-hmm.

 

your configuration living with the application in the same repo. So one of the things we've done here is with the CDK, it allowed us to have a very light shim config.

 

Mm-hmm.

 

So every single repo has the same like you've got your.github folder for your workflows and whatnot, and we've got our own tooling directory that has out a small config file. and it is YAML, yes, but you know, say a VM is the, I think the minimum number of lines for a VM config is like seven lines or maybe less now, but it's like you know a very small declaration of the service name and maybe some environment variables and secrets and whatnot all in this little parcel and that gets passed to the tooling and then the tooling explodes it out to like a three or four actually it's a non or maybe about a thousand lines of HCL when it gets passed off to terraform down the track.

 

Yeah, so I totally get that. We get that with functions that have their own serverless YAML and obviously with Kubernetes pods that can describe their own YAMLs as part of their own code. However, I'm separating config between infrastructure configuration, which would be the templates and the actual configurations. You mentioned secrets and parameters, which I do agree that shouldn't reside in their own config repo. And we do still do that because the team grew so fast that we're still running processes as if it's four people in a garage and that's the hard part of DevOps in terms of education. We have to explain why it's a bad idea to keep everything in one place. And you don't have to explain too hard because suddenly you can start seeing secrets just pushed into this Git repo. and distributed among the applications wherever they go, staging, production, every application. And that's what you mentioned is a goal of ours. And it's part of the hard process that we're going through with educating developers.

 

Yeah, it is a, I think a challenge and something I noted in the sort of the examples or the go-to ways of doing terraform. So it kind of lent itself to that sort of separate repo.

 

It brings me back with follow hashi corp and you'll be fine. At least that's how I

 

Yeah

 

feel.

 

So speaking of HashiCorp and Secrets, are you using Vault? Or do you use the AWS Secrets Manager?

 

So we use all kinds of things. Started from the Secrets Manager, I think it's a little bit expensive if you consider what it does. Then you have the option in AWS to use the Parameter Store and use like your own key or KMS key to encrypt the parameters.

 

Yeah, the parameter store was like, I looked at Secret Manager when it came out on AWS. I'm like, oh, Secret Manager, what, 50 cents a month per secret?

 

And

 

right?

 

then, yeah,

 

Like,

 

why? I mean...

 

it's, and then you look at the parameter store and it's like, if you only, if you're doing less than 10,000 accesses per day, or I can't remember specifically, but like, it's essentially free.

 

Yeah, it's the same. You provide two services that provide the same value, one for free and one for cost. Why would

 

I mean,

 

I use that?

 

the key rotation part of Secret Manager kind of looks cool, but

 

Right.

 

you know, if you've got a thousand secrets, that's a non-trivial amount of money in secrets.

 

And that's why we find ourselves in Vault. Exactly for that. And then Vault manages everything for you. Key rotation, temporary access, everything.

 

Yeah, we've mostly lent on GCP Secret Manager, which is pretty cheap, I think. It's so cheap that I don't know what the costs are, but they don't show up on the bill.

 

All right, that's good

 

Like

 

for you.

 

they're not

 

That's

 

in

 

good for you.

 

the category of things I need to care about on the bill. So...

 

Nice. Nice. Uh, yeah, amazing.

 

When it comes to delivering those secrets, do you push them into the runtime through environment variables, or do you allow the application to connect to the secret store and pull them in when the application starts or something else?

 

So on our side, that's kind of a debate because most of our workload, like I mentioned, is with functions and functions are being built by the second. And if with everyone, with every run, you'll start talking to your secrets manager, decrypting whatever keys you need, then pulling them, setting them up, whatever else you're doing in order to use them. That has a cost. What you can do in Lambda is add a layer. And then that layer is kind of an... and in its service and it's run, it runs as part of the provisioning process. And then the secrets are already there when the, when the function starts running. But more importantly, the next runs, the hot runs, the hot starts. So the subsequent runs will have that as part of the cache and, and then it's seamless, completely seamless for them.

 

Yeah, GCP gives a few options, especially in Cloud Functions and Cloud Running. You can either use environment variables or mount them as a directory within the process. We've mostly opted for making it fairly seamless for the developers. So I think at some point, maybe baking it into the application may make sense. But. Getting them into Secret Manager and having them centrally controlled is more of a priority. And making it seamless so that when the applications have come across, it's just easy.

 

Yeah, for sure. Yeah. Cause with GCP being influenced heavily by Kubernetes. One of the common Kubernetes paths for that is to just mount your secrets as a read-only volume, right?

 

Yep. Yep.

 

Yeah, which is strange in their Kubernetes deployment. It's not very seamless to bring stuff in from Secrets Manager.

 

Yeah!

 

Like I do think they provide a service within Kubernetes that you can configure that is acts as a bridge, but it's not like using a Cloud Run or Cloud Functions. We just define it. And it's there. Right?

 

right.

 

That's not Google-like, I think, is it?

 

It's well, it's a bit weird there. They're hope they manage Kubernetes kind of feels separate if you know what I mean. Well, it

 

Hmm.

 

feels like straight Kubernetes, right? Whereas everything else kind of feels like it was built on top of Kubernetes, but you don't

 

Haha,

 

need to think

 

okay.

 

about it. And

 

Yeah.

 

that's where I'm like, the argument for Kubernetes in GCP is not quite as strong, because a lot of the caveats you face on the other vendors and cloud providers of use, you just don't sort of face in Google. like VMs start up fast, Cloud Functions and Cloud Run are so close to being containers and Kubernetes that they're really nice management pieces that you get. Just make it so easy that you've really, because running stuff in Kubernetes you've got to think about how it's going to run to take advantage of Kubernetes. Whereas on AWS, like... the caveats of running as VMs and stuff like that Drove you to something like Kubernetes because it

 

Yeah,

 

was faster

 

it feels more of

 

and

 

a managed

 

it

 

service, I guess, on AWS. We did recently get the solution for the secrets, so you can now manage sealed secrets using KMS to encrypt them, which is pretty nice. The thing is with EKS, that it's pretty slow to adapt new versions, whereas I think with Google, you get them

 

Yeah,

 

a couple

 

and

 

of weeks later.

 

yeah, it's quick, it's slick. They've got autopilot, which even spins up the infrastructure behind it automatically. So it's pretty nice. And there's probably a lot of things I don't know about it yet, because it hasn't been my focus. I've done a few deep dives into it, but that's my knowledge. There's probably a lot of things I don't know about it. But... The rest of the stuff works so seamless and so quick that the driver just isn't there.

 

Right on. So what do you guys think the next stage of DevOps looks like for each of you?

 

Um, as in for me as a roller for us as an industry.

 

Yeah, from like an industry perspective, like what's the direction that we're going, the next big thing that we're gonna go, oh, dang.

 

Hopefully automation, I'm a big believer in automation around the infrastructure. And I mean, infrastructure as code is one part of it. I think serverless is, I can say it's the next big leap and everything is going there because at the end of the day, like we said more than once here, it's just containers, which I think will keep being just containers, but slowly offloading, actually managing servers and offloading that to the cloud provider. I think is the next thing. And that's at least what I'm trying to do. So you don't really, you get to manage containers and applications, but the servers are slowly transitioning in their responsibility to the cloud provider. And even when you do need to pick, sometimes you do need access to your servers and you need to manage some lower level stuff. So the entire thing around auto scaling them, provisioning them, creating and destroying them, everything becomes a little bit more automated. And I think that's kind of the near future. not having to manage nodes of anything and just caring about your application, everything around that. I think that's improved. That improves the process of building an application and creating it because you're like laser focused on that rather than needing to manage nodes and, and storage and everything around, um, using a server as if it was in your basement.

 

Right.

 

Yeah, I think... I think the rise of the cloud development kit has been really interesting. And something I've been doing for maybe four or five years now, because I was doing it with Troposphere before the terms sort of come about. But since AWS had their CDK, there was Troposphere, and then HashiCorp come along for the CDK for Terraform. It's sort of showing that there's an appetite to do more than just a declarative approach to our infrastructure as code, right? So having that GoGA, having this seems to be a lot of energy behind it. I think. the world where we've got thousands of lines of YAML has kind of made us all a bit tired.

 

Yeah.

 

I don't know

 

I

 

whether

 

think.

 

you've been on the backend of trying to debug like a massive cloud formation

 

Yeah.

 

stack and it's got into some weird state and like there were times that I just went home because I'm like, well, I don't know what I've screwed up but

 

Hahaha

 

Right.

 

I can't fix it today and I'm tired, I'm going home.

 

Coming to think of it, now that you mention it, I think CDK is the perfect description of DevOps because at the end of the day, it shouldn't be a job title. It should be this culture of using developers and operations. And when you're like CDK, that's what it is. It's basically taking a programming language where developers can manage their infrastructure.

 

Yeah,

 

That's a nice analogy.

 

my former boss wrote an article recently. I'll have to look it up at some point. But it was basically, what did he call it? I've got to find it now. But it was basically that, that there should be no DevOps engineers.

 

Yep. Yeah.

 

He sent it to me.

 

I wrote something like that more than once.

 

Yeah.

 

The hit I got back.

 

There is no such thing as a DevOps engineer. Ha ha ha

 

Yeah,

 

ha.

 

yeah,

 

I- ha ha

 

that

 

ha

 

was

 

ha

 

my exact title, by the way.

 

ha ha ha ha ha ha ha ha

 

Hahaha!

 

ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha

 

It

 

ha ha

 

didn't

 

ha ha ha ha ha

 

go

 

ha ha

 

well.

 

ha ha ha 

 

I'll

 

ha ha

 

find

 

ha ha

 

it

 

ha

 

for

 

ha ha

 

you,

 

ha ha ha

 

I'll

 

ha ha ha

 

share

 

ha ha

 

the

 

ha ha

 

link.

 

ha ha ha ha ha ha

 

It

 

ha

 

was

 

ha ha

 

two

 

ha ha

 

years

 

ha ha

 

back,

 

ha ha

 

but

 

ha ha

 

yeah.

 

ha ha ha 

 

Okay.

 

Yeah.

 

It's funny though, how many of us who have the title of DevOps engineers are all advocates that this should not be a job. It's like, we're all clamoring to get ourselves fired.

 

Yeah, I mean, I, so I was a technician at a school and then I was a technician at a mining company and then I moved in sysadmin and then I got a software development role and what I'm doing hasn't really, it's just a more mature version of what I was doing as a sysadmin. Like the tooling that is available now compared to, you know, a decade ago when I was an assist admin, it's just so leaps and bounds better. And a lot of that has come with the cloud. But I mean, early in my career, I was using Zenworks desktop management because I was I was doing deployment

 

Yeah.

 

management for Windows desktops.

 

Yeah.

 

So that was kind of the same sort of process to what I'm doing now. It's just now I'm thinking about instead of, you know, installing Outlook, I'm installing, sporting up 50 cloud functions. I know which one's easier by the way.

 

Hahaha.

 

Right? Hahaha.

 

Yeah, a week, a decade ago, you were literally the CI server. You're taking

 

Yeah.

 

each step hand

 

Hahahaha

 

by hand, doing it manually.

 

Well,

 

And today I have systems

 

well,

 

doing it for you.

 

well, we had, I guess I was building the packages by hand, but we had systems that would. like you'd log into your workstation after it got imaged and it would spin up all your applications on demand. So it was kind of the same thing. A lot of the things and processes I use now haven't really changed that much. It's just the tools and the maturity of the tools have really uh changed significantly.

 

and their stability,

 

Like

 

I think.

 

yeah, yeah stability, and like git having really good open source

 

Yeah.

 

distributed version control

 

I actually had this conversation with someone the other day that was asking about, you know, is DevOps a sustainable career? And that was almost verbatim what I told them. I said, look, I've been doing this for almost 30 years now, and I'm solving the exact same problems I was solving 30 years ago. I'm just using different tools. So will the title

 

I got to

 

stick

 

tell

 

around?

 

you

 

Who knows?

 

when serverless started and we started working with it, the notion or the title was, it's a no ops framework, right? No more

 

Ha ha ha

 

DevOps engineers,

 

Ha ha ha

 

you have serverless. That's it, it's all fine. And then I came in and said, okay, what am I doing here then? Nothing to do, right? Everything's serverless. So you're basically solving the same things with different frameworks and different technologies. It's hard to get rid of. I'll be the first one jumping on the train, the automation train that takes away 90% of my job. Still hasn't happened.

 

Yeah, I mean,

 

Yeah.

 

my entire career has like, before I was diagnosed with ADHD, like automation was a coping strategy because I find it hard to do the same thing twice. And I find it really hard to do the same thing twice consistently. Like

 

Haha, yeah.

 

Yeah.

 

I'll do the same thing 50 times and I guarantee I've probably done it differently 55 times.

 

Ha ha.

 

Yeah.

 

Somehow I did it an extra five times and I don't know how.

 

Ha ha.

 

But it was a coping strategy before, but now it's... you know, automation is not even a goal, I don't think. It's a side effect of really good process. Like all the stuff I built first, I did by hand. I was running by hand. And then I got it to the point where it was so boring to run by hand. Well, I'm like, well, it now can go into a GitHub action. And now it's, you know, in the last year, it's run over 9,000 times across all of the different projects.

 

Right on. Well, we're coming up on an hour here. Is there anything else we need to cover? Anything else you'd like to discuss?

 

I mean, I have a habit of being able to talk until I've got no voice left.

 

Yeah!

 

Get me on a roll and you just can't

 

Ha

 

shut

 

ha ha

 

me up.

 

ha.

 

Or just find the right button to push. Ha ha ha.

 

Find something that annoys me, that's a guaranteed

 

Hahaha!

 

way.

 

Right?

 

I mean, I find most of what I do is, and like DevOps is solving the... The problems that are taking up all your time, you just keep iterating forwards. And I call it annoyance driven development, or ADD.

 

Yes!

 

Okay, that should be a t-shirt.

 

Hahaha! That should

 

I'm going

 

be

 

to

 

a

 

need

 

TED

 

to make

 

Talk.

 

a list

 

Ha ha

 

of... Yeah!

 

Yeah.

 

ha.

 

I mean, I do it in my house. Like I've got a build system from my home automation firmware for all my IoT devices. which I've put in place because like I forget to turn things off and they either break or they use a lot of power so They're all automated, but I also don't want you know to yell at things and say okay google uh turn a thing on

 

Yeah.

 

Apologies for all the things that just got turned on just then but so mine's a little like offline and uses uh, you know Uh various offline automation stuff like openhab or home assistant So my house just reacts to my presence, but a lot of the process is there because I'm not going to do them if they're hard. They're all sort of like GitHub actions and workflows and...

 

Uh, yeah, on the other hand, I find that I just talked about it with a colleague the other day, um, that sometimes you're too lazy to automate a process because it seems something you either not going to repeat enough or maybe this automation is going to take an hour or two and he's just too lazy and you keep repeating yourself for days and weeks and months. And if you just invest this one hour or two as a method of work, it'll solve so much pain in the life later on. So that's what I'm trying to teach myself. Um, It's not hard to implement, at least not for me. I tend to be too lazy with too many stuff. We can reach at some point you'd say, maybe laziness is why you do automation, but on the other hand, you need to make yourself start. You need to force yourself into action. And that's not always easy.

 

Yeah, one of my previous bosses was, laziness is a sign of a good sysadmin.

 

Yeah, yeah, yeah, I know that. And I agree, definitely. As long as you ship it in a good way, you turn it into automation rather than just not doing something. It depends for me.

 

Yeah, well, the fear of disappointing people is a big one for me. People depend on me to get stuff done,

 

Hmm.

 

and I don't remember to do stuff. So, like, automation

 

Yeah

 

just like, if the machines are doing it, they can't forget. Ha ha ha ha.

 

I think one bad sign is when you start having cron jobs on your local machine doing things for you. That's a bad signal.

 

Yeah!

 

I see I don't have them on my local machine it's all backed into my home automation

 

Me neither, but

 

Right?

 

once when I did, that was like a red light. Hey, something's wrong.

 

Yeah!

 

Oh, oh, I see what you're saying. Are you cron into... Oh, yeah,

 

Hahaha

 

no, I've tried to obliterate the world of cron. I love

 

Mm-hmm.

 

cron, it's a great tool, but

 

For certain things, yeah.

 

yeah, it has like, batch processes are interesting and they have their place, but are too often overused.

 

Mm-hmm.

 

I'd like

 

Yeah,

 

to

 

when

 

see

 

something becomes

 

more...

 

too easy to automate and to re-trigger based on like a time-based series

 

Yeah, event driven, like things.

 

Yeah, yeah, yeah,

 

Yeah.

 

exactly.

 

Yeah, that's key identifying the right reason to respond. You know, it's often in. It often should be in response to something happening versus the clock being a particular time.

 

That is one of the key concepts when building an architecture that's, or not serverless, functions-based. You want to be triggered by events rather than doing something or repetitive, like based on time or just waiting for something to happen. You want something to trigger the fact that that function was created.

 

sure. All right, let's move on to some picks. So...

 

Um, yeah.

 

Do you have something to pick Homer?

 

So I have a problem because I have too many of them

 

Ha ha ha!

 

Alright.

 

to the point where I figured I have so much links and articles and things I find on GitHub that I created my own Telegram channel. I can send you the link

 

Yeah

 

later. And I try not to spam because really I have like three or four a day. So I just try to pick two and send them like bi-weekly so people stay there and don't run away. I can share a recent one. I found a CNCF project. Maybe it's common knowledge. I wasn't familiar with it. It's called Buildpacks, if you ever heard of them. So Buildpacks is kind of a way to build container images without

 

Yeah, I

 

actually

 

have heard

 

providing

 

of that.

 

a Docker file. So when I saw that, that was mind blowing. So they just read your code and if you have a package JSON or a PIP file or whatever else, they identify the environment you're running on, they will analyze it. provision their own container template and build the container and comes out production grade, like ready to go. So that was mind blowing and I've seen it the other day. And I figured, I found out that GitLab for their GitLab DevOps and I think GitHub are also using it as part of the automation because they won't guess and sometimes you won't be the one providing a Dockerfile. So they have to do something and that's what they use. So that was pretty amazing.

 

Interesting. Have you had a play with it yet? I know one of my colleagues has had a play with it.

 

I did locally only. I find that the images are, I mean, again, it's automated and it analyzes based on certain rules. It's not as good as if you will build your own Dockerfile. You'll probably get a linear image, probably more precise. But for someone who either doesn't know how to write Dockerfile or needs an automation system that provisions different kinds of them, it was pretty good, not bad.

 

Yeah, because I mean, I've been using Docker quite a while now. And the last workplace I worked in was the entire environment where it was container based. So I've broken it in some interesting ways over the years.

 

For sure.

 

Hahaha! That should be another episode for us, all the ways that I've broken things.

 

Yeah,

 

That's

 

that'd

 

a series,

 

make...

 

not an episode.

 

Oh, oh

 

Yeah.

 

boy. All the hacks I've done to keep production up. I...

 

Uh, you know, you know what they say is better than a cup of coffee in the morning, that deleting a production database table.

 

Hahaha

 

I so far touch all of the wood in my office. Haven't done that yet.

 

Ha ha ha.

 

But I

 

I got

 

did

 

you covered.

 

get a...

 

Ha ha ha ha ha ha.

 

Hahaha!

 

I did have a colleague reach out to me a little while ago, because they were looking to ship, like, 10 years ago, I wrote a sort of a QA HSC app on top of Jira. Like, I hacked Jira, the interface that looked like it, because, you know, it was a quality tracking system. And I took one look at the custom one that was written and didn't work, and went, this is just a ticketing system. and then after a year of not being able to make that other thing work, I'm like, can you just give me two months to rewrite it in like on top of JIRA? And my boss said, okay. And I did.

 

Yeah

 

So like a year ago, so that was like, yeah, nine years ago when I did the first iteration. and I've not been at that company for six years. I get a message saying, oh yeah, the IT guys are doing some updates to it before we move on. They found your hack and they

 

Hahaha!

 

said, it was very clever because I wrote on the hack, on the piece of code, this is a hack but it works. And they're like,

 

Hey.

 

it was neat and clever and they appreciate it. And I'm like,

 

Unbelievable.

 

I don't even remember what it was. I did eventually remember because Jira, the issue types active, but we didn't want people to be able to select them. So I put some jQuery in the footer that just removed them from the list.

 

Yeah.

 

The world needs you. I mean, Gira is the most over glorified ticketing system I've ever seen

 

Hahaha

 

and had the pleasure to work with.

 

Ha ha ha!

 

Was

 

Air

 

that

 

quotes,

 

pleasure in air quotes?

 

yeah,

 

Hahahaha

 

definitely air quotes.

 

Cool,

 

I

 

all

 

just

 

right Leon,

 

gotta...

 

If you

 

what

 

have

 

have you

 

time,

 

got

 

if

 

for

 

you

 

a

 

have

 

pick?

 

time one more visit iheyjiro.com. That's a real thing. So

 

Yes!

 

I'll tell my friends that work at Atlassian.

 

Yeah

 

Yeah

 

My pick. So it's the IoT firmware that I use on all my devices, Tasmota, they've just added IPv6 integration, like to the project. So I've got my laptop sitting beside me to add IPv6 to my IAT VLAN and start bringing up most of my devices on IPv6. So I'm kind of excited by that.

 

Right on, nice. So this week I'm going to pick a book called The Metaverse by Matthew Ball. And you know, like the metaverse is like one of those buzzwords for the year, can mean anything. But the reason I picked this book is because I think the author did a really good job of saying, look, it's just another buzzword. But then they break down some historical context that's probably missing for a lot of people just getting started in the tech industry. They use some really great stories about how things went from mainframes to desktops to server side and how all these iterations led to the current state of what we have and provides some historical context and uses that to kind of like reframe the metaverse for what it really is, which is just another. client server application. And so it's really, really well written. And I think it's, it provides some good, much needed context around that whole buzzword

 

I have to say

 

and

 

that

 

its thing.

 

if he's wrong, he's one of the biggest money pitfalls in the history of the universe. So I hope he's right.

 

Right?

 

Ha ha ha

 

No doubt. Cool, well, thank you both so much for joining me on the show today. This has been a great conversation. I've really enjoyed it.

 

Yeah, likewise. It's

 

Me too, thanks

 

been a

 

you

 

fun

 

guys,

 

time.

 

it was nice meeting you.

 

Yeah, right on. And thanks for listening, everyone. And we will see you all next week.

 

Thank you.

 

Catch yous!