The GitOps Framework with Tanmay Bhat - DevOps 128
GitOps is an operational framework that takes DevOps best practices used for application development such as version control, collaboration, compliance, and CI/CD, and applies them to infrastructure automation. Today on the show, Will and Jonathan interview Tanmay Bhat to discuss the GitOps framework, advantages and disadvantages, and favorite tools.
Special Guests:
Tanmay Bhat
Show Notes
GitOps is an operational framework that takes DevOps best practices used for application development such as version control, collaboration, compliance, and CI/CD, and applies them to infrastructure automation. Today on the show, Will and Jonathan interview Tanmay Bhat to discuss the GitOps framework, advantages and disadvantages, and favorite tools.
In the episode…
- GitOps vs CDIC automated deployment
- GitOps vs infrastructure as code
- Who should use GitOps?
- Favorite tools and recommendations
- Getting started with GitOps
- Flux and RODC
- Pitfalls with GitOps
Sponsors
Links
Picks
- Jonathan- Technology Connections
- Jonathan- Join the American Red Cross to Sound the Alarm. Save a Life.
- Tanmay- ElectroBOOM
- Will- Drive | Daniel H. Pink
Transcript
Will_Button:
What's going on everybody? I'm today's host on Adventures in DevOps. I'm Will Button. Joining me in the studio, the co-host of the day, Jonathan Hall.
Jonathan_Hall:
Hey guys, how's it going?
Will_Button:
and we have our special guest for today Tanmay Bhatt. Tanmay, how are you bud?
Tanmay_Bhat:
Hey guys, what's cooking?
Will_Button:
You know, just, uh, just stuff.
Jonathan_Hall:
Risotto. My wife's making risotto right now, so that's what's cooking.
Tanmay_Bhat:
Alright.
Jonathan_Hall:
Ha ha!
Will_Button:
Oh, so you've got to sit there in your office and listen and smell the wonderful smell of risotto knowing
Jonathan_Hall:
That's
Will_Button:
that
Jonathan_Hall:
right.
Will_Button:
you're stuck here with me.
Jonathan_Hall:
Yeah, if I go quiet after a little while, you know why, it's because I'm eating some risotto.
Will_Button:
Right?
Jonathan_Hall:
Ha ha.
Tanmay_Bhat:
That's great.
Will_Button:
All right, so today I'm looking forward to this. We're going to talk about GitOps. But before we start talking about that and figure out what that is, why it's such a cool word, and things like that, Tanmay, do you want to give us a little bit of info on your background?
Tanmay_Bhat:
Yeah sure, so you know I mean DevOps engineer slash you know SRE slash cloud engineer or however you want to tag it out
Will_Button:
Heheheheheheh
Tanmay_Bhat:
because
Jonathan_Hall:
You're
Tanmay_Bhat:
people
Jonathan_Hall:
a guy
Tanmay_Bhat:
call me.
Jonathan_Hall:
who
Tanmay_Bhat:
Yeah
Jonathan_Hall:
does stuff.
Tanmay_Bhat:
yeah so you know been doing that for three years you know right now you know working for a you know like you know US based company where as a DevOps engineer again you know DevOps slash cloud slash SRE whatever you want to tag it out. So been doing that for since three years and doing. working as an extra consultant also. Yep. Taking up freelancing projects here and there.
Will_Button:
Right on. That's a great intro, because that comes up. I talk a lot with people who are just starting their DevOps career. And many of them are like,
Tanmay_Bhat:
Mm-hmm.
Will_Button:
oh, I don't know if I should focus on DevOps or SRE or cloud engineering. I'm like, it's kind of all the same, man.
Tanmay_Bhat:
All the same, the static works out.
Will_Button:
If you can get a little more money using one word instead of the other, go with that route.
Jonathan_Hall:
Exactly.
Tanmay_Bhat:
Thank
Will_Button:
But
Tanmay_Bhat:
you.
Will_Button:
the actual work is the same.
Tanmay_Bhat:
So
Will_Button:
Cool,
Tanmay_Bhat:
basically
Will_Button:
so tell
Tanmay_Bhat:
just writing
Will_Button:
us.
Tanmay_Bhat:
in the documentation all the day, that's what you do.
Will_Button:
Right? So give us an introduction to GitOps.
Tanmay_Bhat:
Alright, yeah, cool. So pretty recently when it comes to DevOps, right, so basically what, when DevOps came to the picture from the Google site, so the primary target of their job work was basically to handle the deployments. So there, there was mostly, there was a lot of universal way of deploying stuff automatically, because DevOps is basically the guy who does automation for most of the releases part of the deployment. There are other deployments. The problem is, there is no one way or one tool where you can, you know, or one probably how do you call it is like one theory from which you can deploy stuff into production. That's why then GitOps came up and as right now DevOps has high cost. So basically everything should be as infra as code right. So you know even if someone deletes it everything should be now passable. Everything should be now readily deployable. That's what the idea is. That's why GitOps comes in. And GitOps follows a principle called everything should be in Git. Whatever you do, whatever you do to show in applications, everything should be in a format of Git. The idea is a regular developer pushes his code into its own repository in the Git and you deploy GitHub's tool, and there are numerous ways, and that tool automatically pulls that Git repository every three minutes. That's the default value. Once that pulls it, if there's a new change, it will automatically apply that into the cluster. So there is basically no manual way of deploying stuff. It's all automatic. And since everything is stored as git, you can go back, now you can version control, you can control how things are applied, what kind of version you are deploying, all those stuff, all the manual details.
Will_Button:
Right on. The thing that I really like about that, there's
Tanmay_Bhat:
and
Will_Button:
actually two things. One is it follows the exact same process that a lot of software engineering teams use for deploying their code. So you get familiar with that process, and you also get to eat your own dog food. Like you're telling your software developers that you're supporting, hey, you've got to use this
Tanmay_Bhat:
Exactly.
Will_Button:
Git workflow. and now you're having to use it too for your own infrastructure. And then you
Tanmay_Bhat:
Thank
Will_Button:
get
Tanmay_Bhat:
you.
Will_Button:
the other part I like about that is just you get the benefits of opening pull requests and seeing like side-by-side changes of what's about to be deployed, or you get the get history to see what happened to this infrastructure in the past.
Tanmay_Bhat:
Exactly. So now in a primitive architecture consider example like Jonathan, right? So you as a manager tell to Jonathan hey Jonathan go to the production and deploy this stuff Now there is no hundred percent way of verifying that is Jonathan deploying the stuff correctly or not Because you're basically testing Jonathan's instincts, right? And this you know, he is Knowledge and skills of basically how he knows efficiently is deploying stuff So that's not a way to mention in the software deployment. So that's where the github comes in Yeah, thank you since it's automated it will work. That's the idea
Jonathan_Hall:
So what difference do you see between GitOps and just CI-CD and automated deployment? What's the difference there?
Tanmay_Bhat:
So in Git apps, basically the idea is you should not manually deploy anything into production or staging, whatever the environment is. Nobody should touch it on a note, deploy stuff manually. Everything should be automated.
Jonathan_Hall:
Mm-hmm.
Tanmay_Bhat:
That's the idea. And everything
Jonathan_Hall:
And is that...
Tanmay_Bhat:
should be automated only via Git. Git is
Jonathan_Hall:
Is
Tanmay_Bhat:
the
Jonathan_Hall:
that,
Tanmay_Bhat:
source of growth. That's the idea.
Jonathan_Hall:
okay. So is that any different than just a standard deployment, deployment automation that we see in CI-CD before the GitOps term was popularized?
Tanmay_Bhat:
I would say yes because in the primitive CI, CD, Watt you might have used mostly the Jenkins part, right? So Jenkins is not a polling mechanism, right? So it's kind of like a pushing mechanism. You trigger something, then only the Jenkins does some, does some, triggers some work flow or something. It's not an automated polling mechanism for a work flow.
Jonathan_Hall:
Okay, so you just make the distinction between whether Git initiates the or whether it's a pull I guess you're making a distinction between pulling versus push right?
Tanmay_Bhat:
Yeah, exactly. Yep.
Jonathan_Hall:
Alright. And then a related question. What's the relationship between GitOps and infrastructure as code? Those seem like similar concepts too, right?
Tanmay_Bhat:
Yeah, exactly. Both are same. So in infra as code, you keep all the infra part and in GitHub you keep all the application configuration part. Now how you want your application to be deployed. That's the only difference. Both follow the same principle.
Will_Button:
I want to expand on that a little bit too. One key difference in my opinion between Infra as Code and GitOps is you can do Infra as Code and run Terraform apply from your desktop, but if you're
Jonathan_Hall:
Mm-hmm.
Will_Button:
doing GitOps, you can't do Terraform apply from your desktop. You have to let the CI-CD server do that.
Jonathan_Hall:
Hmm.
Tanmay_Bhat:
Exactly. Yeah, so that's a myth. So there are a couple of tools that are right now popular One of the major one is Argo CD and the second, you know, the right now the popular tool is Flux CD So both of the tools, you know comes with their own CLIs where you can trigger the you know Deployments or the releases manually, but that's that's what you know people they don't You know prefer to do that because if you're doing doing it manually, you know going against the you know Gitops way because you're not letting the kid do the deployments. You're doing it manually That's the idea. But both come with a manual override.
Will_Button:
I hadn't heard of FluxCD before, I'm just checking out their website.
Jonathan_Hall:
Yeah,
Tanmay_Bhat:
Yeah?
Jonathan_Hall:
I've used
Tanmay_Bhat:
Is
Jonathan_Hall:
that
Tanmay_Bhat:
that
Jonathan_Hall:
at a previous
Tanmay_Bhat:
cool?
Jonathan_Hall:
company.
Will_Button:
Yeah.
Jonathan_Hall:
And they take it one step further, where you're actually committing Kubernetes manifests to Git. So it's not even like Helm commands or kubectl commands. You literally store your Kubernetes manifests in Git, and whenever you make a change, they get
Tanmay_Bhat:
Yeah.
Jonathan_Hall:
deployed that way.
Tanmay_Bhat:
So they use customize as their backends to generate the config maps and environment variables, et cetera, et cetera, and the back end.
Jonathan_Hall:
Mm-hmm.
Tanmay_Bhat:
Customize if you don't know, it's a very famous, you know, Kubernetes cloud native, you know, template generation tool. It's time for K for Customize.
Will_Button:
I've seen customize a lot. I haven't used it though. I keep
Tanmay_Bhat:
Yeah.
Will_Button:
seeing the name pop up.
Jonathan_Hall:
Same.
Tanmay_Bhat:
So
Jonathan_Hall:
I haven't
Tanmay_Bhat:
basically
Jonathan_Hall:
used it
Tanmay_Bhat:
it's
Jonathan_Hall:
either.
Tanmay_Bhat:
an alternative to Helm in a way where you can substitute variables, you know, temperatize them. Yeah. But Helm
Will_Button:
Ah.
Tanmay_Bhat:
got weird.
Will_Button:
right on.
Jonathan_Hall:
Cool. So, who should use GitOps, in your opinion?
Tanmay_Bhat:
Ideally, people should start implementing because now it's 2022 and everything is right now cloud native way. But the problem is, let's say you're running a company, let's say you're running 20 clusters and you want to implement suddenly your CTO says, hey man, this looks pretty cool, let's implement it. But it's not that much damn easy. The problem is, in your team, you should have someone who knows this GitHub principle. Otherwise, it's just proper jackass stuff. Yeah, because nobody knows what's happening. You know, if the reconsideration fails, if the deployment fails, you don't know where to debug because there are 15 ways to look at large. Yeah, so if you don't have someone who knows guitar apps in your company, I would say you'd rather stay away from it. Yeah. Because it looks fancy on the front end, but when things fail, it's true
Will_Button:
Hehehehe
Tanmay_Bhat:
pain.
Jonathan_Hall:
Hahaha. Yeah, okay. Good advice.
Will_Button:
It looks great until you get it up to speed and then you've got to figure out where the brakes are.
Jonathan_Hall:
So you've already started to answer what was going to be my next question, which is who should not use GitOps. But it sounds like your answer is similar to my answer, who should use Kubernetes. If
Tanmay_Bhat:
Exactly,
Jonathan_Hall:
somebody on
Tanmay_Bhat:
yeah.
Jonathan_Hall:
your team understands Kubernetes, go for it. It's a great, powerful tool.
Tanmay_Bhat:
Go for
Jonathan_Hall:
But if you
Tanmay_Bhat:
it.
Jonathan_Hall:
don't understand it, it's going to drive you crazy.
Tanmay_Bhat:
Exactly because if you don't know kubernetes ma'am, I would say why are you not know a lot of people that's my first question in 2022 If you're in devops field, you should be sure otherwise
Will_Button:
Yeah
Tanmay_Bhat:
Otherwise, there's no difference between a devops and ss admin. So yeah
Will_Button:
Hahaha.
Jonathan_Hall:
Yeah, nice. So what about use cases? Let's imagine that you do have somebody with GitOps on your team or in your company. Are there still
Tanmay_Bhat:
Mm-hmm.
Jonathan_Hall:
use cases where it doesn't make sense, where you shouldn't use GitOps?
Tanmay_Bhat:
Okay, so it doesn't make sense when you have a release cycle which are very extremely strict. Let's say in a corporate companies, right, in a big agencies, so they have a know the release management extremely strict. So it comes from a couple of manager approvals. First your team leader approves it, next your engineering manager approves it, next your CTO approves it, then only know it will go to the production. So at this point in time, so that considers in a couple of workflows, right, at least two to three approvals. So GitHub doesn't have a way to have that much kind of generality for approvals. So you just push it to the Git repo. If it has a proper tag and a proper contract, that's it. Bam. It's going to apply to the production. Yeah.
Jonathan_Hall:
Okay.
Tanmay_Bhat:
So if you have that much
Jonathan_Hall:
Yeah.
Tanmay_Bhat:
generality, I would say the R-back based mechanism, so it's kind of going to hurt you.
Jonathan_Hall:
Fair enough. And what, do you have any favorite tools that you like to use for GitOps? I mean, I think it's kind of an umbrella term, so you can use it with all sorts of things. I mean, you wouldn't even have to use Git, right? I mean, you could do subversion ops and it would be the same principles.
Will_Button:
I'm gonna
Jonathan_Hall:
Not
Will_Button:
go.
Jonathan_Hall:
that I would recommend that, but
Tanmay_Bhat:
Okay.
Jonathan_Hall:
it could be possible. So what, what tools do you usually use or recommend for doing GitOps?
Tanmay_Bhat:
So if someone is a very newbie and you know, he's just coming to the DevOps and he's just reading all the fancy words man, hey man, the new hiring, you know, Recruited manager says, you know, I should know get out. What should I learn? I would say just go for the Argo CD. It's fairly simple. You can learn it in another 5-10 minutes. It's very easy. Just install it and then bam man, it's good to go. And the important stuff about Argo CD is for the newbies, it has a UI where you can learn and see what exactly it's doing. And you can even do a lot of operations. You can even sync the application, that is basically new releases, you can delete, refresh, you can do all kinds of operations, whatever you're doing from the CLI or the automated way with GitOps, you can do it manually via the UI. So I would say that's one best way to learn how GitOps actually works in real. So because Flux is 100% GitOps, they don't have anything, they have nothing, no, that's not UI, nothing else. So that principle is you commit something to Git repo, if it's correct, we'll automatically pull it and apply it to the repo. cluster that's the idea so I would say for the newcomers go for the algo CD because it has a UI and you'll learn better
Jonathan_Hall:
Okay. Good. Good advice.
Will_Button:
Let me ask this, because I've been on the fence with this. Do you feel that, you know, so we mentioned that Argo has the ability to go in and change things from the UI. Do you feel that that
Tanmay_Bhat:
Mm-hmm.
Will_Button:
violates the GitOps core principle and the fact that you didn't do that through making a Git change?
Tanmay_Bhat:
Exactly. That's 100% true. That's basically, so that's the reason, you know, in our current company, we're changing from ArgoCity to Flux. If somebody is able to just click on delete some deployments from the UI, it's basically, that's not good, right?
Will_Button:
Yeah. All right, cool.
Tanmay_Bhat:
That makes sense.
Will_Button:
So
Tanmay_Bhat:
Yeah,
Will_Button:
we're on the same page there.
Tanmay_Bhat:
exactly. So that's the reason actually Flux was born. I mean, on
Will_Button:
Ah.
Tanmay_Bhat:
a major, major hand, yeah, because you don't have any UI, you can't do something like that. Yeah.
Jonathan_Hall:
If you want a UI, by the way, for Flex, you can use Weave. It's
Tanmay_Bhat:
Yeah,
Jonathan_Hall:
the same
Tanmay_Bhat:
yeah.
Jonathan_Hall:
company, but it's their paid
Tanmay_Bhat:
Same
Jonathan_Hall:
offering
Tanmay_Bhat:
company.
Jonathan_Hall:
that does provide a UI for it. But that
Tanmay_Bhat:
Yeah.
Jonathan_Hall:
UI, any change you make through the UI is manifested in Git. So it's not like you have two choices. You can do it the Git way or you can magically muck around in the UI. Everything in the UI
Tanmay_Bhat:
Yeah.
Jonathan_Hall:
is committed to Git. So,
Will_Button:
Nice.
Jonathan_Hall:
yeah.
Tanmay_Bhat:
Nice. So I tried it, but the one problem I found is basically it's not a centralized way. For example, in Argo CD, right? So let's say you have five clusters and you want to implement GitOps and all of them, and you want to obviously install Argo CD in all of them. So you can have one Argo CD as a centralized mechanism where everything is sent to the other four clusters from one server. Basically like a proper cluster architecture, one master and other workers. And you can connect all of those clusters into one single UI. So you can manage everything. and see what each cluster in our place is going to start. That's not possible with real guitars right now. That's the only problem that I face. In the GUI part by the way.
Jonathan_Hall:
Nice.
Tanmay_Bhat:
I mean I have a solution for the new ways, so if someone is hearing GitOps as a really cool keyword or something like that if you are seeing it and you are pushing it out, just don't do it just because of the sake of wanting to get hired or get some good package or something like that you just gotta understand why do we even need GitOps in the PENU 22 like why doesn't your company even push for GitOps or what's the drawback of the previous multi-deployment and write on the GitOps way then do it Just don't google how to get into DevOps, put it into and don't do that. That's it, it doesn't
Jonathan_Hall:
Hehehe
Tanmay_Bhat:
matter.
Will_Button:
Hahaha! All right, so if we're gonna go down the GitOps path,
Tanmay_Bhat:
Mm-hmm.
Will_Button:
let's say that we've got, let's say that we've, for years and years, we've been operating on the craft beer brewery microartisanal approach where everything's handcrafted and aged in bourbon barrels, and we're gonna go down to GitOps. What's your recommendation on how to get started with that?
Tanmay_Bhat:
I would say start with small. So just first learn proper GitHub. Again, the same principles of GitHub first one should understand before going for any tools. So if you just, again, Google how to get into GitHub, there'll be first two to three results. Each will try to sell their own products. Even though they try to make it in the cloud-native bullshit, it doesn't work. So just first understand why GitHub, why GitHub, and then try to figure out. Is it even a requirement for your company? It's not like everyone is doing it up, just do it, apply it. It doesn't work like that, right? Does it even require your company? And when you apply it as principles to your company, does it actually help to develop a lot of engineering productivity? Is it going to help or not? That's the first point you have to consider, I would say. So after that you start very small, try to onboard one or two applications in the dump cluster or something like that. See how it behaves, try to make a couple of other releases, see how it changes, see how it reconciles. So once you have proper strategy, then try to implement granular way.
Will_Button:
Yeah, so that's gonna be a time consuming process, right?
Tanmay_Bhat:
Yeah, exactly. So, unboarding
Will_Button:
Yeah.
Tanmay_Bhat:
is usually something that's on the GitHub. It's actually a pain in the ass. The problem is, you have to define manifest for each of the applications you're basically unboarding. That's how GitHub works. There's no automation for that. So let's say if you're running a company like Netflix, where you have 200 microservices, you have to define 200 YAML documents. That's another way.
Will_Button:
There's no right-click save as get ops. Ha ha ha ha.
Tanmay_Bhat:
I'm sorry.
Will_Button:
Now I think that's
Tanmay_Bhat:
Yeah.
Will_Button:
an important thing to point out over someone starting out in GitHub is it is, if you're converting your existing practices over, it's a long, tedious
Tanmay_Bhat:
Long journey.
Will_Button:
process. Yeah, so make sure that you set expectations with your management and the stakeholders accordingly that this is not a Friday afternoon type project.
Tanmay_Bhat:
Yeah, and exactly and one more, you know, everyone should take a note of it if they're trying to implement get up It's basically let's say someone someone running in a double score deploy and it is good pipeline instead of our CD or flux CD as a get up stool The thing is she seems an already AWS customer even if you face something, you know, you know Like heckling the deployment because this open a double support ticket and ask them hey man Why is this not working right as a dog in here? but the problem with you know, get out which is basically an open source tool is You can ask someone, there is no support. You gotta learn this rule completely before implementing it in the production. Because no one else will have the same view except community support where someone will reply after a week or something like that. That's the thing. That's the drawback of using GoPros and Stores, I'll say. But yeah.
Will_Button:
Yeah, for sure. All right, so we've got Git as our backend repository for our infrastructure code, pushing it out using Flux or Argo CD.
Tanmay_Bhat:
Mm-hmm.
Will_Button:
Do Flux and ArgoCD or possibly any of the other tools do live polling to verify that the last deployed state is actually what's running out in production?
Tanmay_Bhat:
Yeah, so they both run know they internally run the get diff to see what's the current manifest But that's actually the core principle of Kubernetes also if you look at it So they have each Kubernetes manifest will have three layers the current one the actual one and the last modified version They'll try to match it. That's what it does also and in Kubernetes also We will just do cubes it'll apply I can have some deployment document That's what the deployment does if there's already a deployment running. So if if you don't know hell 3 which is basically the newest version of Hel exactly does the same way so Hel does 4 way git diff before applying something
Will_Button:
Wait, there's Helm 3 now? How long has that been out?
Jonathan_Hall:
Well, like two or three years or something.
Tanmay_Bhat:
I'm gonna
Jonathan_Hall:
It's
Tanmay_Bhat:
get
Jonathan_Hall:
been a while.
Tanmay_Bhat:
through this.
Will_Button:
Okay, okay, cool.
Jonathan_Hall:
Yeah.
Will_Button:
I thought I was I thought I was behind again
Jonathan_Hall:
Don't worry, you will be soon.
Tanmay_Bhat:
Yeah. Sit.
Jonathan_Hall:
November
Tanmay_Bhat:
Ah.
Jonathan_Hall:
2019, yeah, it's been a little while.
Will_Button:
Okay, good, good, good. I should be okay.
Jonathan_Hall:
Ha ha!
Tanmay_Bhat:
So here's the thing, so what do you guys think about how does one survive, one should survive in the DevOps industry because everyday there's a new tool coming out, so a startup like that you know, they just want every new tool to be implemented in the company which is basically in the name of improving company productivity. So how one should survive in the flashing blazingly fast DevOps industry.
Will_Button:
So for me, I take the Forrest Gump approach. I just kind of wander through. Ha ha
Tanmay_Bhat:
I'm
Will_Button:
ha.
Tanmay_Bhat:
sorry.
Jonathan_Hall:
Oh, I thought you were gonna say you never know what you're gonna get.
Will_Button:
I was trying to come up with something clever there, but I totally failed.
Tanmay_Bhat:
Thank you.
Will_Button:
We're gonna have to get a new comedy host on the show, because I'm bombin'. Ha ha ha.
Tanmay_Bhat:
I mean that's the reality because you know like me you know being in the dev option industry for three years right and every day I see a new tool and after let's say the tool has been launched I'll get a little bit of you know publicity so cloud cloud foundation you know cloud cloud foundation will just take it as an incubator project now push it you know give it a way bigger marketing budget and BAM that's it that tool is famous now so everybody's supposed to use it
Will_Button:
Yeah, for sure. There's a lot of... That happens a lot, and I think it contributes a lot to the imposter syndrome because as you're learning
Tanmay_Bhat:
Exactly.
Will_Button:
stuff, the key takeaway of your learning when you first start doing DevOps is you learn about all the stuff that you don't know, and it just gets to feel overwhelming. I've been doing this for a really, really long time, and my approach on it is just don't worry about it. It's sort of like the old, a twist on the old philosophy statement, when the student is ready, the teacher will appear, and I kind of treat treat my technical skills the same way, you know, I continue to work on them, but I don't really freak out over what I don't know, because whenever I need it, I'll go learn it. And so I don't try to learn things that I don't have an immediate use case for.
Tanmay_Bhat:
Exactly, that's a perfect answer because for example, just to give an example like how things are So you have Teleform as I know, an infra code tool, right? Which has been there in the market since like two or five years and a lot of even big companies have implemented that That's easy for them to know, right? The problem is there are at least 10 to 15 tools which is born out of Teleform Like TeraGrant, TeraCode, blah blah blah stuff. So these are the new companies which are being launched every month So the name, it's again a new process syndrome. If somebody, for example, if someone is approaching as a DevOps engineer interview for a smart startup, so if the engineering manager, whoever is taking the interview, if he's like, you know, found out new tools, he'll definitely ask, hey, since you have worked on Terraform, do you know this thing? We are using that in the company. But that guy doesn't even answer, because that tool just launched two months ago.
Will_Button:
Right? Yeah. Jonathan, what's your approach on that?
Jonathan_Hall:
Sorry, what was the question? Approach on which part?
Will_Button:
for staying up to date with all
Jonathan_Hall:
Oh.
Will_Button:
of the latest tools and technologies and yeah.
Jonathan_Hall:
Don't even try.
Tanmay_Bhat:
That's the best approach, yeah. That's how DevOps
Jonathan_Hall:
Yeah.
Tanmay_Bhat:
works. Don't even try, man.
Jonathan_Hall:
I mean, there are a few places like I kind of keep up on Go. Go 1.19 was just released. So, you know, I kind of keep up with that, but that's because I use that day to day.
Will_Button:
Yeah.
Jonathan_Hall:
Um, I keep up on, you know, the things I use all the time, I'll more or less keep up on, uh, I try not to be more than a few weeks out of date, uh, for security reasons and you know, stuff like that. Um, but like I don't know what the latest version of Kubernetes is. Don't care. Don't know what features are added. When I when I need that, then I'll figure it out. I don't know what the latest version of Helm is. Three point something, I don't know beyond that.
Tanmay_Bhat:
Exactly.
Jonathan_Hall:
So yeah, I just don't, I don't even try.
Tanmay_Bhat:
See it makes no sense right, so even if you keep on doing that you end up learning you know 10% of every single tool out there which makes no sense because if someone wants to hire you, they want you to learn something really good right not like now you learn 5% of everything and all like 50 tools that makes no sense
Will_Button:
Yeah, one filter that I apply it through is looking at it from the perspective of the customers of our business. You know, like if I'm considering a new tool or a new implication or a new framework or whatever, I think about it like, do our customers using our app actually care if I do this or not? Is it gonna benefit them in some way? And so that helps me like. kind of like put it into perspective and where it falls for a priority.
Tanmay_Bhat:
Exactly, yeah. That's also the same perspective as mine because whenever a new tool comes, I'll just first think Is there even requirement for this? What's the problem statement for this? Yeah, like and how do I even if I apply this to my company does it really affect something or does it change something? Or this you know, this is a waste of time. So that's the thing
Jonathan_Hall:
So I'm curious
Tanmay_Bhat:
Thank you.
Jonathan_Hall:
to hear
Tanmay_Bhat:
Have a great day.
Jonathan_Hall:
if you have opinions on some like traps that get ops might throw at somebody. If you're getting started or maybe you've been doing it for a while, what are some things that are easy to do wrong that people should be aware of?
Tanmay_Bhat:
Hmm, okay. So the first thing is it for example, if you let's say if someone is working in a really small startup and And I've seen a lot of in the startups They don't have a diverse aging environment because I have one dev you know one production cluster Well, it because consider a company like you know 10 15 engineering members, right? They don't need a dev environment because they're getting started out The problem is if you as a doctor, you know, once you know, and you know implement get up there The first thing would be in get up right now by out of the box things are by default and they apply automatically. For example, if you're, if let's say if right now V1.0 is running via GitOps and developer by mistake which is V1.2.0 and it was not supposed to go to production, it will go to the production unless you stop it. So that's the thing. So he has to know about what's the, you know, polling intervals. That's the thing. So if you keep it to too delayed, you know, that's it. Bam, you're gone. will bang their heads as to see why deployments are not getting pushed to the production. And in a lot of times, one has to figure out how do I configure the secrets or basically the git secrets into the, for example, Argo CD has a feature called Argo CD image updater. It's an additional package which comes with Argo CD. So what it solves is basically in a traditional GitHub mechanism, if there is no image updation basically. the technology exists. Consider that. So you have an application repository where you show your application configs, including Argo series configs. Now, a developer has now pushed a new Docker image into the end of container registry, be it Docker Hub, ECR, GCR, whatever it is. So v1.0 was running, he pushed v2.0. The problem is, he didn't update the newer version image stack in the Argo CD configs in the Git. He was expecting that, you know, hey, my new version of application code will be deployed, but it's not. That's what Argosy did automatically without image updater is it will try to match the configurations of which is in the Git into the running cluster. That's the principle GitOps. So whatever is right now in the Git should exactly match right now in whatever the cluster is being implemented on. The problem right now is how do you update the image stacks automatically. because otherwise it will be a manual task, right? So every time you push a new image tag with a container registry, you have to go to the Git repository, manual and update. So that's to figure out that problem, both Flux and ArgoCD has image operator tools. Now for ArgoCD, it's an additional add-on, for Flux, it comes by default. So what exactly it's trying to solve is, if your developer pushes new image tag into the container registry, so that image operator, it has a separate container controller, What it does is at a specified interval, let's say every 2 minutes or every 5 minutes, it will try to pull the ECR and see, hey, for this container image, is there a new image stack? If yes, what it will do is it will pull that image stack and it will try to push a new git commit into the Argo series repo saying that hey, this is the image stack you need to apply because this has been found newly into the ECR repo. So then what happens is since there is a new config change, RGOCD will pull that and apply it into the cluster. Bam! And you got v2.0 in your cluster. And it sounds a little complicated, but it's trying to solve a really quick problem. How do you automate image changes?
Jonathan_Hall:
Yeah.
Tanmay_Bhat:
This is what I would say the common problem of DevOps is. So unless you have a really good document, half of the people won't be understanding what exactly you're talking about, because it's a new one. For the most time, mostly we'll be hearing about.
Jonathan_Hall:
Right.
Will_Button:
Yeah, for sure. Documentation and communication across the team, not only your team, but the teams that you support. How do you communicate
Tanmay_Bhat:
Exactly.
Will_Button:
that effectively?
Tanmay_Bhat:
Exactly. Yep.
Jonathan_Hall:
Another sort of pitfall I've thought of or I've run into is that of reproducible builds. Whether it's a Docker image or a compiled program, you lose some of the benefit of GitOps or even just automated deployment if you can't roll back to a known version. So, this is a big problem with certain. programming libraries or languages, like Node.js, where the dependency management tends to pull in the latest version every time you build. So you might build and get one artifact right now, and an hour later you might get something else without changing anything in your own code, right? So
Tanmay_Bhat:
Yeah.
Jonathan_Hall:
GoTry is pretty hard not to do that with a different dependency resolution algorithm. You can work around that with Node also by pinning specific. dependencies that usually solves the problem with Node or other
Tanmay_Bhat:
Yeah.
Jonathan_Hall:
languages where that's a problem. But if one of the reasons you're using GitOps is so that you can say, where I've seen this happen is you get a bug report
Tanmay_Bhat:
Amen.
Jonathan_Hall:
that happened like a week ago, and you've already deployed 15 times since then or a thousand times since then or whatever, but you can see in the log that this happened on that specific revision, and you want to go back and reproduce that bug. you kind of need to be able to reproduce your environment. So if you don't have reproducible builds, that's really hard.
Tanmay_Bhat:
Yeah,
Jonathan_Hall:
It's
Tanmay_Bhat:
yeah.
Jonathan_Hall:
also really hard to get reproducible builds, even when you have the right tools. Like Go makes it more or less easy. It's still really hard because... as soon as you put in a Docker image, you have the same dependency problem again.
Tanmay_Bhat:
Exactly.
Jonathan_Hall:
And then you have to keep your old Docker images around potentially, or at least be able to rebuild them from the same dependencies. And so that is a pitfall. It's a more advanced one, but it can be something to keep in mind for some people.
Tanmay_Bhat:
Yeah, yeah, far sure.
Will_Button:
this episode sponsored by Go.
Jonathan_Hall:
Exactly. Go, 1.19 has been just released. Choko, check
Will_Button:
Hahaha
Jonathan_Hall:
it out. It has lots of exciting new changes that, actually, not really any changes this time.
Will_Button:
All right, anything else we should talk about on GitOps?
Tanmay_Bhat:
I'd say we covered pretty much because yeah.
Will_Button:
Yeah. All right.
Tanmay_Bhat:
If you have any doubts, keep them coming.
Will_Button:
Yeah,
Jonathan_Hall:
Yeah, cool.
Will_Button:
now I feel like we covered it there.
Tanmay_Bhat:
Yeah.
Will_Button:
If people want to get more info from you on this, how can they, what social media do you hang out on?
Tanmay_Bhat:
I have an impulse syndrome on social media but I am amdible. Mostly the best one would be the twitter mechanism. Emails are the best I would say. Because that's the best way to reach out and get a quick reply. But yeah, man, twitter would work.
Will_Button:
Alright, cool. We'll include that in the show notes down below. And let's move on to picks. You got us some picks, Jonathan?
Jonathan_Hall:
Yeah, so I like to watch YouTube sometimes when I'm trying to fall asleep or trying not to fall asleep or whatever the case is. And there's a channel I watch sometimes called technology connections, which is kind of cool. So I guess it'll be my first pick. It's a it's a guy who sits in a room and talks about technology VCRs or water, lawn sprinklers, just whatever. Usually mechanical things, but sometimes like CRTs, it just kind of breaks it down and tells you how it works and maybe shows you how it works. Toasters, everything. Anyway, he did a recent one. I don't actually know. Maybe this is an old one. I watched it recently about smoke detectors. And he turned me on to the difference between two types of common smoke detectors. And so there's optical ones. which he recommends and I think he convinced me it makes sense. And there's others, I don't remember what they're called, but they basically detect radioactivity in a little thing, a little gel. Watch the video, look for it if you want. But anyway, the point is, we didn't have smoke detectors in our house until I watched this video. I was like, you know, I should get some smoke detectors in
Will_Button:
Hahaha!
Jonathan_Hall:
our house. So I just bought a pack of eight smoke detectors. We have them in all the bedrooms and the hallways now. So my pick. aside from the channel Technology Connections, is smoke detectors. Get a smoke detector in your house. So if something happens bad, maybe you won't regret it so much.
Will_Button:
Right? Seems reasonable.
Jonathan_Hall:
Yeah. Do you guys have smoke detectors?
Tanmay_Bhat:
Eh, no. Not in the
Jonathan_Hall:
No.
Tanmay_Bhat:
house, but yeah.
Will_Button:
I do and so
Jonathan_Hall:
Have you checked
Will_Button:
to
Jonathan_Hall:
it
Will_Button:
pit,
Jonathan_Hall:
recently?
Will_Button:
yeah I was going to say to follow up on that if you do have smoke detectors make sure you've changed the batteries recently
Jonathan_Hall:
Exactly.
Will_Button:
because one of my dogs like when on ours when the battery gets low it will start beeping
Jonathan_Hall:
Mm-hmm.
Will_Button:
and for some reason our one of our dogs knows what the smoke detector is she knows that it's
Jonathan_Hall:
Uh-huh.
Will_Button:
like some kind of alert to get out of the house have no idea how she knows this we've never had a fire. But if it goes off, she will freak out. So I always try to make sure I change the batteries before it goes off, because it will take hours to calm her down
Jonathan_Hall:
Oh my.
Will_Button:
once that smoke detector goes off.
Tanmay_Bhat:
So is it like fireworks for the dogs? Like, you know, that's how we know, is it a fact? Yeah, yeah.
Will_Button:
Yeah,
Tanmay_Bhat:
And then it freaks
Will_Button:
yeah, for
Tanmay_Bhat:
them
Will_Button:
sure.
Tanmay_Bhat:
out. Yeah.
Will_Button:
Alright, so my pick for the week, I can't remember if I picked this last time or not, I don't think I did. It's a book from Daniel H. Pink called Drive, the surprising truth about what motivates us and it's really cool because he's breaking down the myth that if you've been working in corporate America for very long, you're kind of familiar with all the different things that we use to try and motivate people. And he breaks it down and shows with a lot of studies that have been done how many of those are wrong. And then, I haven't finished the book yet, but supposedly at the end of the book, he replaces those myths with things that actually work. So I'm enjoying reading that. It's been pretty insightful. And yeah, so that's my pick for the week.
Tanmay_Bhat:
All right. So.
Jonathan_Hall:
I don't think I've read that book, but I've referenced it before because... ..I... ..I... ..I... ..I...
Will_Button:
Yeah.
Jonathan_Hall:
There's also
Will_Button:
I'm sorry.
Jonathan_Hall:
a YouTube video, I think, a three-minute version for those of us who are too stupid to read.
Will_Button:
Right?
Tanmay_Bhat:
Yeah.
Jonathan_Hall:
We can watch that. So...
Will_Button:
Or just hold out for the TikTok and get it in
Jonathan_Hall:
There
Will_Button:
60
Jonathan_Hall:
we go.
Will_Button:
seconds.
Jonathan_Hall:
There we go. That's what I really need.
Tanmay_Bhat:
Yeah, yeah.
Will_Button:
All right, time may you have any picks for us.
Tanmay_Bhat:
Yeah, so it's kind of funky one. So in YouTube there's a channel called Electro Boom. And this guy from the Middle East and what he does is basically he teaches you what not to do with electricity. Yeah, so he just... So last week he just took out a haze wire and he just teaches everyone what not to do. Like you know, how do you take it out? How do you make some thermocol cutter from a haze wire? He literally gets electrocuted at least 10 times in a video. It's really cool to watch it, it's really fun. Yeah, and it teaches you literally about how, you know, physics often works. And like, you know, what's the formula for, you know, some voltage, you know, for exactly, you know, like current flow, blah, blah, blah, blah, stuff. Prop, you know, one-on-one on physics and some fun. Go check it out, guys. It's really fun. Yeah.
Will_Button:
Right on, that's awesome.
Jonathan_Hall:
Awesome.
Will_Button:
All right, cool. Well, I think we've got ourselves an episode. Tanmay, thank you so much for joining us. It's been a great
Jonathan_Hall:
It was a
Will_Button:
time
Jonathan_Hall:
pleasure.
Will_Button:
chatting with you.
Tanmay_Bhat:
My pleasure man, yep.
Will_Button:
All right, so in the words of the famous philosopher Forrest Gump, that's all I got to say about that. We'll see you all next time.
Jonathan_Hall:
Cheers.
Tanmay_Bhat:
Yeah, sure. Have a nice day, guys.
The GitOps Framework with Tanmay Bhat - DevOps 128
0:00
Playback Speed: