Helm is the package manager for Kubernetes that helps you define, install, and upgrade even the most complex Kubernetes application.  Matt Butcher, core creator of Helm, joins the show today to share about his experience building and implementing the program.

Hello everybody and welcome to another exciting episode of Adventures in DevOps. I am your host for the day, Jonathan Hall, and here in the virtual studio I have Will Button.

 

Howdy everyone!

 

And we're excited today to have our special guest, Matt Butcher. Did I say that correctly or did I butcher it?

 

Oh

 

Uh, I've never heard that one before. Yeah. I've gotten into the habit of introducing myself when people ask how to spell my name. Like a meat cutter. Yeah,

 

Yeah.

 

Matt the meat cutter. Yeah.

 

Cleaver, right?

 

hahahaha

 

Matt Cleaver.

 

Yeah

 

Cleaver, I've got a new online handle now. I'm gonna start using that one. No

 

Great.

 

I

 

one

 

hate

 

will ever

 

that.

 

guess that Mark Cleaver is really Matt Butcher.

 

Welcome, Matt. I'm glad you're here.

 

Thanks,

 

I think

 

Rob.

 

this is going to be a fun episode. Would you tell us a little bit about yourself and what you do?

 

Yeah, yeah, sure. Currently, I'm the CEO of a startup called Fermion. We're building the next wave of cloud computing with WebAssembly. I got here sort of through this nice roundabout way of going back and forth between startups and megacorps. I've really kind of got my first taste of cloud computing, which is went from Drupal development to cloud computing when I was at HP Cloud and they were really, really into OpenStack at the time. And once I kind of got a taste of infrastructure computing, I have never looked back.

 

Wow. Cool. So you were at Microsoft until, what was it, a year ago? A couple of years ago?

 

Yeah, until November 1st was when we started Fermion. And prior to that, I'd been at Microsoft for four or five years.

 

Cool. And what does Fermion do?

 

We are building a cloud service for WebAssembly. We were looking into what we thought the next wave of cloud computing was going to look like and what some of the seemingly intractable problems with cloud computing were, particularly having been deeply steeped in the Kubernetes and Docker ecosystem, we were saying, all right, so there's some things we've learned here. How do we build something that's going to solve some of the edge cases and some of the... the non-starting cases for containers and Kubernetes. And that got us really looking at a variety of different technologies. And WebAssembly was the one that really stood out to us. I know it's a browser thing. Most people when they think of WebAssembly, think of running Rust or C or something in the web browser, but that

 

Mm-hmm.

 

same model that makes it really interesting for the browser is what caught our attention. When you think about... characteristics of running code in your browser. You think, well, you really want a good security sandbox. You want really fast startup times. You want cross architecture, cross operating system support, and of course, support for a variety of languages. Those were all the same kinds of things that when we were looking at the cloud, we thought it would be really great if we could have a good architecture that's gonna sit sort of alongside virtual machines and containers as

 

Mm-hmm.

 

sort of like this next wave that's gonna be able to do things like scaling up and down. you know, in milliseconds and running Rust applications or Swift applications or JavaScript or whatever, kind of in this neutral security sandbox that can be managed well as a cloud service.

 

Awesome. During that whole spiel, three words jumped out at me. Web, WebAssembly, Kubernetes, and milliseconds. So you're telling me we can run Kubernetes in the browser and it only takes milliseconds, right?

 

HAHAHAHAHAHA

 

Hahaha

 

Wait, cancel everything, you

 

RIP

 

know, start all over again. Yeah. When I seriously, so I worked at Microsoft for Brendan Burns, who is the creator of Kubernetes. And I scheduled a meeting with him and I must have set the topic to be something like WebAssembly and Kubernetes, but he starts the meeting going, you better not be telling me you did some hipster thing where you're running Kubernetes in the web browser. No, I swear that's

 

Hahaha!

 

not what I was doing.

 

Somebody's gonna do it, and if they hadn't thought of it yet, now they have, unfortunately,

 

Yeah, there you go.

 

and they will do it.

 

I'm sure

 

Just...

 

it's a weekend project. You should have no trouble knocking that out between Friday afternoon and Monday

 

Knock

 

morning.

 

that out in

 

No, no.

 

half an

 

Yeah.

 

hour, it's easy.

 

Just

 

Yeah.

 

make sure you tag all of us on Twitter whenever you

 

Yeah.

 

release

 

Yeah.

 

it.

 

So I think we've sort of beat around the bush a little bit. Today we're gonna talk, I think we're gonna talk about Helm. You used to work on Helm, I guess. I imagine you still

 

Yep.

 

use it, if you're using Kubernetes. And I understand you were there at the inception, I guess, and sort of know the origin story

 

Yeah,

 

of

 

yeah.

 

how Helm came to be. You wanna tell

 

Yeah,

 

us about that?

 

yeah. So to kind of frame this out, right, prior to starting Fermion, we were at Microsoft. Prior to Fermion, I was at a company called Dais, and Dais was one of the very early players in the Kubernetes space. Primarily, we were building a platform as a service. An open source competitor to Heroku is basically what we were trying to build. And in the course of building that, we were experimenting with a lot of different container orchestration systems, because the way the system worked is you supplied your source code, checked it into a Git repo. That kicked off a workflow that built the binary, put it in a container, and ran it somewhere. And the ran it somewhere part was rather substantial. So we started looking at different orchestrators for doing this. We had been using one called Fleet from CoreOS, which has been gone for, unmaintained for quite a long time at this point. But at that point, it was kind of the new shiny thing. And along comes this like out of Google sort of project with a name we couldn't pronounce called Kubernetes.

 

I'm glad

 

And

 

he is dead.

 

we went, I don't know. I mean, I had worked at Google before and I'm like, well, you know, Borg was pretty cool. Maybe we should take a look at this. And so the engineering team at Deus, started playing around with Kubernetes. We liked it. At this point, it was very early. It was just prior to 1.1 when we started experimenting with it. And so as we began the process of sort of experimentally

 

experimentally

 

lifting

 

in

 

and

 

the

 

shifting

 

day-to-day.

 

the dais workflow paths onto Kubernetes, we started to notice, you know,

 

started

 

there's

 

to notice.

 

a lot to be gained here and a lot to be done. So we convinced the upper management that we should sort of pivot into Kubernetes from fleet. And in revenge, the management said, okay, well, we're going to have an all company meeting and we're

 

Thanks

 

going

 

for

 

to

 

watching!

 

fly everybody into Boulder, Colorado. And it's up to you to explain how all of this works to the company. And I said to the company like all the engineers and they said no to the company like marketing and finance and all of them as well.

 

No.

 

And then we're gonna have like a hackathon to kind of get people going on.

 

And you were thinking, nobody's ever been fired at an offsite before, so I

 

Yeah!

 

can probably

 

It's

 

save? Hahahaha

 

Alright.

 

like, it's kind of like a crossover between a shark tank and an afternoon meeting,

 

Yeah.

 

yeah. And they gave me this slot right after lunch. I'm like, come on, you know, people are going to be

 

Wow.

 

falling asleep and I'm supposed to describe

 

At least you can

 

them.

 

blame

 

Ready?

 

them for- blame the lunch for their comatose rather than your speech. Ha ha ha.

 

Yeah, so on that one, I decided, well, I've got to do something that captures people's attention. So I went and got my daughter's stuffed animals and took a couple pictures of them. It was like a giraffe, the go-gofer, you know, an owl. And I wrote a PowerPoint slideshow called The Illustrated Children's Guide to Kubernetes and put that up there and

 

Is that

 

used

 

still

 

that.

 

available somewhere? That sounds amazing.

 

Yeah, the marketing department took it and actually published a book

 

Nice.

 

form of it. So,

 

Nice.

 

yeah.

 

I'm sorry.

 

And I think those characters are like now the CNCF mascots. But we did that in the afternoon. And then after that, they started a hackathon project that was supposed to be like, hey, now that you heard a little bit about this, you should go try building some kind of Kubernetes tool. So I joined a group with Jack, Francis, and Remus, and the three of us. brainstormed a little bit and said, you know what we should do? We should try creating like the NPM for Kubernetes. And so we sat around and hacked on this thing for an afternoon and we came up with the best name ever. We decided, all right, well, you know, Kubernetes gets shortened to K8S, K8s. Let's go with like a coffee shop motif and let's call it K8s place. And so we did this whole

 

Hahaha!

 

coffee shop motif about K8s place, the, you know, the. the package repository and package manager for Kubernetes. And we worked on it, you know, we went out for our team dinner and everybody, you know, had dinner, then we, you know, bolted and worked on it a little bit more that evening. And then the next morning and really didn't pay attention to anybody else's sessions. And by the end we had this demo. And so we did sort of, again, like a shark tank kind of thing where everybody came up and presented their demo and the CEO, CTO, and maybe it was the architect from. something else or something they they judged kind of the the entries in here and at stake was a $75 Amazon gift card and

 

High

 

so

 

stakes,

 

we did our whole

 

rolling

 

pitch

 

here.

 

yeah yeah and we we won we we won the $75 gift card which we had to split three ways that's the sum total of money i have ever gotten out of helm

 

Hahaha

 

uh i yeah

 

That's more than I've gotten.

 

That's more than most open source projects have made, so.

 

You know, it's the fame totally makes it worth it. I'm yeah,

 

Right?

 

it's definitely great. Hey, can you fix my bug? Yeah, so the next day I went back to went back to life as usual and I'm sitting at my desk hacking away on some code and the CEO and CTO call me and I'm like, oh crap, I am in big trouble. I don't know what I

 

Here

 

did,

 

comes

 

but yeah.

 

the meeting.

 

And they said, so about that demo thing you did yesterday, We kind of like it. We're thinking maybe it's a good idea and maybe we should keep working on it. There's just one caveat. You've got to change the name.

 

Hahaha

 

Hahaha

 

They didn't like the coffee shop motif much either.

 

Yeah.

 

So called back Jack, whom I'd been working with and the two of us sat there with a nautical dictionary and on opposite sides of a call, same one

 

such

 

pulled up

 

nerds.

 

between the two of us and we're just reading words back and forth to

 

Yeah

 

each other. Winners such as various knots, I don't know, slip knot, you know, things that just really don't convey anything terribly, you know,

 

Mm-hmm.

 

package manager. And my version of the story is that Jack said helm. Jack's version of the story is that the two of us simultaneously said helm, but I'm pretty sure he's the one who said it first. I'm like, yeah, that would be great. And you know, with this whole idea of like taking the helm and steering something and then having charts that are like nautical charts where you use that to navigate. And so that was how we kind of decided on that. And then for the next several months, that was what we did while we were at DAIS. I think, let's see, the first commit to Helm must've been somewhere in like October of 2015, and then the first KubeCon was a month and a half later. So we debuted it at the very first KubeCon.

 

Nice.

 

To, I would say, neutral reactions. A lot of people were like... No, you don't understand Kubernetes. We definitely don't need a package manager. You can just write YAML. And at the time, maybe so. I mean, your average Kubernetes application probably was about 40 lines of YAML. But as Kubernetes matured, it became clear that managing a group of, you know, associated YAML files in and of itself was very difficult. And so Helm kind of grew in popularity and then grew in sophistication. think it must have been maybe four months after that, maybe January, that Google showed up and said, hey, this is a really good idea. They had been working on a competing project called Kubernetes DM. And I don't even remember what DMs stood for at the time. And they just rolled Helm into that repository. And that was actually how we got from a DeusLab repository into Kubernetes itself, was just kind of rolling the project directly into theirs. And we worked on it there and other companies started joining that NAMI was next. And before long, we realized it is such that shocking moment in open source where you wake up one morning and see that somebody is using your thing in production. And you're like, whoa, you know, table stakes are a lot higher now than they were yesterday. So that was pretty exciting and terrifying at the same time.

 

Yeah.

 

And then from there, I mean, it wasn't long after that, that that Microsoft came along and So Brendan Burns, head of the Kubernetes project, had started at Google. He'd moved over to Microsoft, and Microsoft had sort of seen the vision, right? They understood what Kubernetes was gonna offer to operators from small to gigantic enterprises. And so they really wanted to build out their Kubernetes presence, and Brendan Burns and John Gossman at Microsoft came and approached Daes and ended up acquiring us. And it was great. I honestly... working at Microsoft was a fantastic experience and they split Deus into sort of two teams, two basic teams.

 

Thanks for watching!

 

One of them went and built AKS, which is Azure's

 

Thanks

 

main

 

for

 

Kubernetes

 

watching!

 

offering now. And that team is just a phenomenal team of low-level system engineers. And then the other half of us became sort of like the open source team and Helm became like my day job. And so in a

 

Hmm.

 

way, I mean, I joke about only making $25 off of Helm, but actually, you know, it was my full-time job. And that meant that we had time to focus on building the community. Karen Chu was our community manager and she was amazing. And it meant we had time to develop out the features and to, uh, you know, we did two Helm Summit conferences

 

Thanks for watching!

 

that were specific to Helm. Uh, and it was just a very heady time and very exciting to have been part of a piece that played a very big role in Kubernetes adoption and Kubernetes is continued success.

 

Well, thank you. I'm a fan of Helm, so I appreciate the efforts that you've put into it.

 

Most of the good stuff was not me. Ha ha ha ha. Uh oh.

 

Hahaha

 

I tell people that I have a pull request that was accepted by Helm. So that's one of my miniature claims to fame. I added

 

Nice.

 

an if statement around a standard output error.

 

HAHAHAHAHAHA

 

I'm pretty sure it's removed because it was related to Tiller. So I think in version three that probably was ripped out.

 

Yeah, yeah, there were a couple decisions, you know, you really learn as you go and one of the fascinating things about Kubernetes is that so many, and I'll be honest about this, so many of my early assumptions about how people would use Kubernetes were just wrong. I mean, like flat out wrong. I really, really thought that, you know, a team would have a cluster, right? Not an enterprise would have a cluster. I thought a three node Kubernetes cluster was gonna be average and a five node was gonna be really large. I had not seen it coming that people would deploy hundreds and thousands of nodes into their Kubernetes cluster. So as we were designing Tiller originally, so for those of you not familiar with Helm, the Helm 1 was a client-side tool and only lasted a very little time because as we got going on it, I kind of said, you know, maybe we should try this sort of client server architecture where we've got Tiller running as what we would now call a controller. At that time, there was no such thing, but Tiller running as an in-cluster service. and Helm just being sort of like an API client that would send everything to Tiller and Tiller would do all the work in the cluster and then tell the Helm client, okay, everything's good. And this seemed like a good idea at the time, as so many ideas do.

 

Yeah.

 

But because we thought, you know, we weren't gonna run into security issues with Tiller because it would be small teams using these things. And the teams would of course have essentially

 

essentially

 

root permission on their cluster. And, you know, I was also kind of using the metaphor and you see this everywhere in the old Helm documentation. We keyed into the way that we use this message, right? Kubernetes is going to be an operating system, right? And an operating system has a package manager. And we surveyed the landscape of package managers. We were deeply inspired by apt-get. You can see that all over the place by Snap, by Homebrew. and by these package managers that were operating system level package managers. And all of those require root to execute, to install packages. And so of course, when we were deploying this, there didn't seem at that point to be any problem with giving one process like Tiller, essentially cluster wide access to all this stuff. We even ran it in the cube system area, and we just thought that was gonna work out fine. We were wrong, right? The core

 

Okay.

 

is some... that clusters would be small and team specific did not hold at all. And the way that the Kubernetes access control model, you know, developed around us was not compatible with my original vision. much to the credit of the people who developed it. And I did not have a good vision on that. And so when we realized as a team, our assumptions about security and Helm and Kubernetes were not gonna bear out over the long run. And our assumptions about usage patterns in Kubernetes were not gonna bear out over the long run. We had to make the hardest pivot that I hope Helm ever has to make. And that was going from the client server model back to a client model. And that was, I mean, as you can imagine, it's contentious anytime you have to make a major architectural change. But more so when the community at large is going, you need to move on to this new security model, you know, and then

 

Mm-hmm.

 

core developers are saying, but this is so much work. I mean, it's so much. So many things we have to rewrite and features that we honestly had to drop and things we just couldn't figure out how to replicate from the Tiller world into the Helm 3 world. But, and it took us a long time, I think a year and a half total.

 

Mm-hmm.

 

maybe a little bit longer than that even start to finish. But the outcome with Helm 3 was something that I have been very pleased with, right? I think we, it was a hard lesson, right? That we should not have waited as long to listen to the community when the community started telling us our assumptions about security were incorrect. But once we figured it out, you know, the entire Helm team, which is comprised of people from a whole composed of people from a whole bunch of different companies, how quickly everybody sort of gelled on the mission. And then Adam Reese, who was on my team there, is also at Fermion now. He really kind of led this effort to say, okay, we know we did something wrong. We know how we're gonna course correct. It's gonna take a lot of work. I'll triage the PRs, everybody go. And it took a while, but it came through. You know, one of the funny things is we talk in the helm. I'm still involved in helm. And we talk on occasion, well, what's going to be in Helm 4? And in

 

Hehehe

 

a funny sort of way, we're kind of waiting for a moment that says, yeah, you know how you had to get rid of Tiller in Helm 3? Well, now there's something you got to do that's going to really push forward Helm 4. And for better or worse, we haven't seen one. A lot of the issues that are Helm 4 issues are things like, we'd like to tweak this a little, we'd like to refactor that a little. But we're hoping we will never again see that kind of major

 

Mm-hmm.

 

architectural overhaul like we did between Helm 2 and Helm 3.

 

Yeah. few questions I have. I don't know how much this relates to the origin story, but I don't know the things that came up to me. Why did you choose to write it in Go? Not that that's a strange choice, because given that Kubernetes is written in Go,

 

Great.

 

but you could have written in anything, I imagine. So why

 

Yep.

 

did you choose Go?

 

So I think the first application I'd written in Go was was several years earlier. And actually, the first time I tried Go was the day it came out. And

 

Okay.

 

Hahaha.

 

I was at a conference, and tried it and said to the guy who was sitting next to me while we're at a lull in the conference booth, why in the world would anybody use this language? It's like, see you again, I don't know, didn't we get past this?

 

Hahaha

 

And then you know, Fast forward a couple years later and I'm like, I'm going to write everything in Go. You know, it was the simplicity of the language was very appealing to me. The ability to compile it to a static binary, which was at the time as we were trying to deploy everything in containers was very attractive to me. And I think attractive to most of the team. It would have been conceivable given the languages we were using at the time that when we started Helm, we would have used Python instead. Bunch of us had used Python, bunch of us really liked Python. But the catch at the time had been Python package management itself was in a little bit of disarray time and it would have been somewhat ironic I suppose but to have a package manager that had dependency issues when you were building the package manager but for us really it was more a matter of well let's just go with something that's going to be simple and go check a lot of the simplicity boxes. Incidentally we have talked here and there about well what would it mean if we rewrote some of the things you can do with the Rust type system are more amenable to the kinds of things we wanted to be able to do in Helm but couldn't figure out how to do well. But

 

Mm-hmm.

 

that's never gonna happen because rewriting a big code base is not anybody's idea. To my knowledge, not anybody's idea of a good time. So...

 

Oh, I know some people who think it's a great idea.

 

Ha ha ha ha! Let me give you access to the Helm repo. Now, I'm... Ha ha ha ha!

 

Hey.

 

Yeah,

 

But you know...

 

those people never think it's a good idea after they try it. Only until

 

Yeah,

 

they

 

that is

 

try

 

pretty

 

it the

 

much,

 

first

 

yeah.

 

time.

 

It seems like

 

Yeah.

 

a good idea for the first day, then an okay idea for the first month, and then after that you're going, this is way more work. Also, then you have to look at all the ugly parts of your code and acknowledge that they're there for a reason

 

Hahaha.

 

and determine whether you want to fix that reason or accept the fact that your brand new shiny code is also gonna look scary.

 

I just rewrote a function today, not even a code base, a single function I rewrote today. And there was this little weird thing that it did. It just did some search replace on some text. And like, this looks broken. I can't imagine this is correct. Like maybe, I mean, I'm sure it solves the problem, but this is not the right way to solve this problem. But I don't know what it's solving and I don't know what it's gonna break if I remove it. So I had to copy

 

Fuck

 

and paste

 

Yeah.

 

that brokenness into my new version of the thing.

 

Yep. Yep. Oh, man. I hesitate to admit how much that resonates with me. One of the most peculiar functions I ever saw, I'm looking at a function in this back in the Drupal days, looking at this function that is supposed to print a string. And it's doing all these things like adding on CSS padding. you know, per character. And I'm looking at this going, how do you, you know, why, why, why is it doing this? And finally I went, some, but clearly this is a mistake. And I just backed it out and, and, and exchanged it for a print string thing. And I got a, got a bug issue a few hours after I had merged it in from one of the designers saying, the kerning is off on this font. And I'm like, kerning, kerning, I gotta go look up kerning. What is kerning?

 

Hahaha

 

Apparently a previous developer had gone, well, the designer doesn't like the kerning fonts so I'm just going to use CSS to pad it out so he was like manually checking for the characters that the designer didn't like and readjusting the font padding around or the the white space around them to

 

Wow.

 

get them to line up well and I thought okay I've seen it all now seen it all.

 

Yeah, lovely.

 

Hahaha

 

Went back to the old version never touched that function again.

 

Yep,

 

Great.

 

yep. At least add a comment so that the next person who stumbles upon that and thinks, what the hell,

 

Yeah,

 

knows not to touch it.

 

yeah, the following lines are done for purely aesthetic purposes. Do not touch. Yeah.

 

Do not remove this code until so-and-so has been fired,

 

Yeah,

 

or has retired, or changed

 

Hahaha

 

teams.

 

yeah. Oh, there's the strange things we do in service of making everything the way we want it to be.

 

Exactly.

 

And they're definitely more than our fair share of those in Helm. In fact, I think you interviewed Taylor Thomas a couple of weeks ago.

 

Yeah.

 

Taylor was also on the Helm core team with me. And he ran the Helm watch part of the code base. That was the

 

Ah,

 

one

 

okay.

 

that he supervised. And of all the pieces, he jumped in fresh-eyed and ready to go on the Helm project, inherited that piece of the project. And shortly, after is going, well, okay, now I see why I'm responsible for this piece.

 

Hahaha

 

It is, you know, trying to, it is the one part that seems like it ought to be really straightforward, right? The watch

 

Oh yeah,

 

piece.

 

I could write it in 30 minutes.

 

Yeah, yeah, install this

 

Thanks for

 

thing

 

watching!

 

and tell me when it's all ready to go, right? How

 

Yeah.

 

hard can that be? I mean, you know, but. The individual pieces of Kubernetes are very nuanced. And the difference between a pod coming up and being ready to serve and a service being ready to serve and all of these different things meant that there's just ridiculous branching logic with special cases for everything. And it is the gnarliest piece of code inside of Helm, and pretty much by necessity. I mean, we've tried to simplify it many times, but the fact of the matter is. the cost of an abstraction layer like Kubernetes is that at some point you have to say, okay, well on this platform, it's gonna do this. When you're using this as a load balancer, it's gonna do that. And then just continue to special case things in there.

 

So Helm is kind of the jQuery of Kubernetes.

 

Oh my

 

Ha!

 

gosh, yeah.

 

Hahaha

 

I mean, in a way, yes. This is the Tower of Babel we've been building all along. We think if we just add one more abstraction layer, it'll solve all the problems. But every abstraction layer is just leaky enough that the tower tilts just a little bit to the left every time you add another brick. And before long, you're looking at it going, well, it seemed like a good idea at the time. And also, Nobody can touch the bottom of this tower ever again because, you know, a camel's hair might tip the whole thing over.

 

Do you have any regrets about Helm? About the design choices or mistakes you've made, anything, however you want to interpret that.

 

I mean, yeah, the tiller part was a regret. And one that in hindsight, you know, you have to give yourself some grace on some of these things and say, all right, well, you know, I didn't know what I didn't know at the time. I think one of the things that was difficult for us early on was really figuring out how to build the community around the project. And I think we had some missteps. early on that just we got it going and then we didn't quite know what to do and how to build a community and created some environments that weren't particularly open. I think from January of 2016 on to maybe mid of that year, there were some times in there where the community was just, we were just malfunctioning, right? We weren't doing a good job of listening to users. Out of that, actually, we came up with a good model that ultimately has led... that we've used repeatedly on open source things. So it was a regret because I know feelings were hurt. There were people that were early contributors to Helm that never came back after that because we just mishandled things. But we learned really that there are periods in an open source project where you really have to kind of look inward and focus on the core developers getting the core project done, right? And in

 

Mm-hmm.

 

those points, you know, trying to spend a lot of time doing evangelism of your project can really misfire because you don't really, you don't have enough of it to keep the vision going. And if people are sidetracked on that. So, so we came up with this model where we said, you know, at the beginning, you're like a circle of engineers who are all facing sort of inward and looking at the thing in the middle and saying, how do we build this? But then at some point you have to switch focus and say, okay, now it would be, you know, we really want to start building up a user base. Also, it would be really helpful to get other people involved. And so, you know, the inward facing circle suddenly kind of stops being inward facing and everybody kind of turns around and invites other people into the community. So we had this sort of visual that was accompanied by our team saying, okay, now is the time when we go out on Slack and we answer every question out there and we get active on the GitHub issues and we get active on stack overflow and anywhere we can. And you start just. reaching out and building up. And of course, occasionally, sticking with this metaphor, you have to turn back inward and say, okay, here's a really difficult bug, how are we gonna handle that? But for the most part, you're encouraging other people to start contributing and writing good documentation. And then what happens is you start to build up this concentric ring outside of your inner ring of people who come in and they're familiar with your project and are engaged in the project and... That's the kind of magical moment. And that's what we didn't understand the first time we saw it. But once you've got that second layer consecutive ring, then you can say, all right, time to build Helm V2 or time to build Helm V3. We're gonna shuffle a little bit. Who wants to be in the inner ring focusing inward on getting the project done? And who wants to be the outer ring facing outward and bringing in yet another concentric ring of users? And we ended up with this sort of like target shaped community model where it was about You know keeping some people very focused on getting the day-to-day job done As far as the code base goes and other people focused on Being assistive to the community meeting them where they were at, you know helping them out And we would see as this as we built up this model people would come into the model and actually it worked really well So Taylor's a great example Taylor started by showing up in slack and asking a bunch of questions I think he was at Nike at the time asking a bunch of questions about how Helm worked, and then he got really involved, and then he started doing PRs, and then he started answering other people's questions. And so we saw him go from sort of like an outer observer to that outer ring of users, where he was the one outward-facing saying, hey, I saw you showed up here yesterday and asked a question, I was asleep, but I wanted to get back to you, you're still having this problem, anything I can do to help, and triaging issues and just knocking it out of the park. He was phenomenal in the community. Then when the helm two to three transition happened, Taylor ended up sort of shifting his way into the inner ring and taking on a lot of the core work because he had a lot of experience already in the outer work. And while I could never say there's kind

 

Thank

 

of a

 

you.

 

one-to-one replacement, but everything Taylor was doing before, an IBM developer named Martin Hickey sort of jumped in and said, hey, I'll pick up the slack here. And he started going to town on a lot of the community stuff with us. And Taylor really became one of the core engineers. I started drifting back. out to one of the outer rings. And we watched this model sort of play out in real time. And so even though that kind of came out of some high tension in the community initially, when we misfired and we're not sure how we were supposed to manage the people who wanted to be core contributors versus the people who were just kind of coming in and trying to do a quick PR and hurt some feelings early on, that model ended up being sort of like a big learning experience for us. Consequently, when we started building other projects like OAM, CNAB, Brigade, Draft, all of these other open source projects that are many now in CNCF, we used that model intentionally at the beginning and said, okay, well, first few months, we're really just focused in, then we turn around, we build the next circle, and then we do, and it has worked out really well. And it's one of the very few things that I could honestly say I'm proud, really proud about in my career is kind of. discovering a model that would work well to help open source grow. You know, most of the things we do, code is a hard one, right? Because we know that we pour energy into it and we work really hard on it, but it's going to last a few years before somebody PR is in a better version, or maybe 10 years, you know, if your project is really successful. But some of these models, they make a difference in the growth of a community. They make a difference in real people's lives, right? People are find a sense of belonging in a community. And that means something beyond just, you know, writing out some code that helps them get a job done, right? And so that's one of those few things I'm kind of proud of in my career. And again, it's not all me. It was a concerted effort by that whole Helm team in saying, how do we understand what's happening around us?

 

That's great. I mean, I've never been involved at that level of an open source project, but I have worked on several. I maintain a small library and I'm a member of the CouchDB committee. So I've seen some of that, but it sounds like you've really... I mean, I suppose it helps that you are involved in a project that got a lot of momentum. A lot of open source projects, of course, don't. They aren't

 

Yeah.

 

nearly as central to sort of the tech universe as something related to Kubernetes.

 

Yeah.

 

uh...

 

Yeah.

 

but uh... yeah that's that's great it's a really encouraging story

 

Yeah, and it... sorry, go ahead.

 

I was just going to say that last two or three minutes of that was pretty inspirational. Have you got that documented or available somewhere? Like is there a TEDx version of this? Because I know so many people who are like, oh, I'm going to make my project open source, you know, and they have this misunderstanding that if they mark their repo as public in GitHub, that all of these people are going to come in and just start contributing code. And they totally missed that. that whole part that you just talked about, about building your core group and focusing on the product, you know, and I've never heard the rings analogy before, but I think it's just fantastic and worthy of sharing.

 

I think the only other time we've really talked publicly about it, Karen Chu and I talked about the rings model briefly at KubeCon LA. What was that? 2021? Yeah, I think that was the last one. But you're right, that would be good blog post material.

 

for sure.

 

Yeah, definitely. It would be

 

Uh.

 

a good material for a podcast episode just to talk about how to run an open source project too. That would be great. I don't know if it's quite for this audience, but it would be a great

 

Yeah,

 

topic.

 

yeah, yeah. Karen, who ran the community with us, also has this depth of understanding of how developers think, really, more. I think, as developers, it is easy for us, as technologists in general, I think it's easy for us to see. We want to see ourselves. as purely rational actors, right? I write logic for a living, you know? I'm good at math,

 

Hahaha.

 

you know?

 

Yeah.

 

And yet we're actually just as emotional as everyone else. And some of the biggest hurdles for us being happy in our career are things like. sense of belonging in the community, right? A sense of enthusiasm about what I'm working on. Passion is definitely drives the vast majority of open source development. Why in the world would any of us code on a Saturday if we didn't feel some passion about what we were doing? And it's funny that we try to edit that, edit that emotional part out of our descriptions of ourselves, right? And I think one of the things that Karen, always point always was was right in tapping into was the fact that if you can meet people there, even when they refuse to admit that that's a need for them, right, but when you meet people there, they deeply, deeply appreciate it. And and consequently, that means you pay attention to branding right because your logo says something about the tenor of the project you pay attention to your documentation because The tone of voice you use there can either turn somebody off or get somebody excited within within moments and so many of those little details that we tend to not want to talk about an open source because again we are thinking of ourselves as pure rational actors are absolutely key to creating open source communities that can be successful. Helm was sort of a test bed of this. We tried to employ a lot of this technique in Helm to just create welcoming spaces. We had a formula for example for when we received a PR that we knew we couldn't merge. where it was always the first comment somebody sees has got to be positive and encouraging, even if it's just, thanks for doing this PR, we know it took you a long time to put together. Because if we could lead with that, then saying things like, you know, something's not right in this particular set of lines of code, right, doesn't come across immediately as, oh my gosh, they hate it, they don't like me, I should just close this PR and move on. It comes across

 

Mm-hmm.

 

as, oh, they care about the fact that I put this work in here, and they're trying to figure out, you know, how to help me get this merged. We didn't always do a great job of that, but it was a pattern that we tried very early on

 

you

 

just to kind of build some positive rapport with the community. And it does seem to be something that people really, really appreciate. Whether they acknowledge it, or whether it just results in people continuing to work on their PRs, it appeared to work quite well.

 

So what you're telling me is that you're not taking this stack overflow approach is down vote and close

 

Hahaha!

 

Honestly, one of the hardest things about becoming a big project is exactly the Stack Overflow problem. That at some point you have so many issues coming in and many of them are... things that are just so far out of scope or whatever that you can't even sort of, we definitely later in the project, six months ago from now really, suffered with the fact that we didn't have the number of people necessary to stay on top of the issue queue. And consequently, our earlier patterns just sort of fell by the wayside. And it was like, clothes won't fix, clothes won't fix, clothes won't fix because we just couldn't handle the volume. And that's a problem. I know Kubernetes as a whole is experiencing it, is that as demand goes, it's hard to scale up the number of open source developers who are willing to spend time, particularly doing the chore-like work. Cause there's nothing glamorous

 

Mm-hmm.

 

about triaging an issue queue. Ha ha ha.

 

Yeah, definitely. You wanna talk about how Helm became part of the CNCF?

 

Yeah, yeah. Helm was one of the first projects accepted into the CNCF. And it was an interesting process. So Helm, the original version, Helm Classic, was a dais project. Had nothing to do with official Kubernetes at all at that point. It was just us doing a thing. And then as we merged with the DM deployment manager, that's what DM stood for. As we merged with the Google Kubernetes deployment manager project, we were sort of like automatically merged into the Kubernetes org. So for a while Helm was officially part of Kubernetes itself, which was an awkward fit and didn't sit well with a number of people that were in the Kubernetes core saying, Why is the package manager part of the, in the same project as the Kubelet, right? Doesn't quite make sense. The governance model doesn't match and it was true. The way we did things was very different from the way the rest of Kubernetes did, more by necessity than anything else, right? As we were building up package chart repositories, we had a completely different set of guidelines we needed to apply. As we made decisions about the stability and robustness of the project, release cadence didn't match with Kubernetes. And so there, we had a number of points that were sort of like friction points. And, you know, I mentioned we had Helm summits, which were small conferences specific to Helm. And at one of them, Brian Grant and I went out for coffee. At the time, Brian Grant was heading the Kubernetes project and we sat down for coffee and sort of like... mutually landed on the same topic as the opening topic. Like, does Helm really belong inside of the Kubernetes project? And Brian's going, no, you don't match our process and you don't match our expectations about how people are building the software because you've got a different problem to solve. And I'm going, and no, because we need to be able to build a charts repository,

 

Bye.

 

which is totally different than what Kubernetes Core is doing. And our governance model is a little different. And so out of like a 30, 45 minute coffee session, we kind of came out going, okay, We gotta figure out where home belongs and it was clear to everybody that home belongs in the ecosystem and it was equally clear that didn't belong inside of kubernetes. So as cncf was

 

and

 

beginning

 

Seattle

 

to

 

was

 

really

 

beginning

 

formalize

 

to really

 

their

 

formalize

 

own process

 

their own

 

for

 

process

 

how they

 

starting

 

would manage

 

with

 

projects.

 

managed projects. Um...

 

We all kind of looked at it as an opportunity to say, would a project with the scope of Helm be a good fit for the Cloud Native Computing Foundation and the way they were trying to structure things? And it turns out, yeah, it was a much better fit. And it was a little bit of a painful move because separating out of Kubernetes required everything from figuring out how we were gonna do CI-CD again, because we were relying on a lot of their infrastructure, to then having to really answer in earnest, how are we going to govern this project? We need to write up guidelines for how we're going to govern this project. Even while we were sort of bulking against the way Kubernetes was running things, we were also actively relying on them to provide us a lot of the sort of organizational structure. So it was a little bit of a difficult move just because there were so many things we had to think through. And you really want to nail some of those things, right? We want to nail governance for sure. But you really want to nail CI, CD as well. And so we had, you know, deeply

 

Hahaha.

 

technical problems that were outside of the expertise of outside of my expertise, right? That we knew we needed to solve. And then, you know, sort of like this philosophical part where we had to say, OK, now for sure, we have to describe this. We have to. talk about how we're going to enforce the code of conduct. CNCF, incidentally, has solved many of these problems in sort of a generic way since then. But early on, those were sort of our trials. But once we did that initial separation, things just sort of fell into place. And other projects landed alongside us in CNCF. Some of them outpaced us. I think Helm graduated after, I forget what the project was that beat us through that process. Might have been, was it Envoy maybe? But the main point being, you know, we might have taken a little longer to learn some of the ways to do things, but it was always a very supportive environment. And I mean, when you look at CNCF now, after several years of diligent work on their part to create a foundation that could meet projects where they're at from tiny to huge. they have done an absolutely remarkable job. The whole sandbox way of doing things is just a testament to solving a problem that I was afraid was gonna sync CNCF. When Rocket, so Rocket was one of the alternative container runtimes, RKT I think was the way you spelled it. one of my favorite pieces of engineering to ever come out of CoreOS. So it is a very, very elegant piece of software. But it didn't, as OCI emerged and the container ecosystem sort of solidified around Mobi and ContainerD and technologies like that, it turned out that Rocket didn't really fit a niche, didn't really fill a niche. And the developers had sort of lost interest. And when I saw this happen, my first thought was... CNCF is not gonna be able to cope with this, right? We're gonna end up with CNCF being, and I'm glad to be wrong about this, but at the time I'm like, CNCF is gonna be a dumping ground for dead projects. And yet they reacted well, and they solved the problem with Rocket, but moreover, they applied the learnings from that and started separating out sandbox projects from incubated projects, and that was the change they needed to make, and now... there are definite milestones where you say, I'm gonna contribute this project to CNCF. We've contributed at Microsoft, I think we did six, seven, maybe eight projects. And you put them in incubator and then you see, you

 

Thanks

 

develop

 

for watching!

 

them and you try and build an audience and if you just don't succeed, then there's a clear path, there's a clear off ramp, right? And that, as we all know from open source development is very important because when we start losing interest in the project... it's unfair to not signal to your potential users that this project is not maintained anymore. So I love the way that CNCF ended up doing that. And Helm, of course, has found a very happy home there. But I kind of like to think that some of the things we learned in those early days, and we there being everybody involved in CNCF, have over time really led to a better, more robust foundation that has a really good set of guidelines that I think works very well. I can't even remember what the original question was. Ha ha ha

 

Oh,

 

ha!

 

just, yeah, that's fine. Just how Helm became part of CNCF.

 

Oh right, yeah, yeah, I don't... Did I answer that anywhere in there?

 

So I think you've answered it and then some.

 

Ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha

 

Yeah, that's really good.

 

Now this is literally turning into a masterclass of how to build open source software.

 

Yeah.

 

Oh You know, and it's funny because Helm is a big success, right, for me. For me personally, it's something I take a lot of pride in. But I have had literally over a hundred not successful open source projects in my career. And you never know what things will be successful and what won't. And you have outliers like I read the PHP HTML5 parser with Matt Farina. The two of us built this as a sort of passion side project. It was wildly successful. I mean, it was getting, it probably still does get a lot of usage. We had zero community. We had one person come in and help us out and he was amazing. But we never really, the issue queue never really spiked up and we didn't have like PHP, HTML5 con thing.

 

Hahaha

 

But it was by all, by the GitHub metrics, it was a totally successful project. But it had no community. And

 

Yeah.

 

Matt Farina, also another Helm maintainer, he works at SUSE now, excellent engineer, we sat down one time and said, why did Helm get a community and the HTML parser didn't? And maybe the answer is obvious to everybody else, but the HTML5 parser, we were just implementing a specification. There was nothing really to be decided or debated or anything like that. We either succeeded 100% or we failed and had a bug, right? And that was really... Our success criterion was implementing the standard, and the closer we approached to implementing the standard, the less community involvement we saw. The fewer issues were filed, the fewer people were talking about us because parsing HTML is a decidedly unglamorous task. Helm on the flip side was solving a human problem that required innovation and required thoughtfulness and a high degree of flexibility and... Every person has a different unique use case, even if it's only

 

even if it's

 

a hair

 

only

 

off

 

in our

 

of

 

hair

 

somebody

 

or

 

else's.

 

somebody else's.

 

And that meant that we had to continually form this sort of forum where we could discuss how it should operate or how it might operate. And community, that seems to be one of the key ingredients for forming a community that just never obtained for an HTML5 parser. So, you know, success in

 

success

 

open source

 

in

 

is

 

open

 

such

 

source

 

a strange

 

is such a strange

 

thing. because

 

thing because

 

you can be

 

it can be

 

successful

 

successful

 

as an

 

as

 

individual

 

a

 

person,

 

individual person.

 

scratching your own itch and discovering that other people like it and use it. You can be successful as a small team of people who create a project that gets relied upon by a bunch of big PHP projects, but never really forms much of a community. And then there are successes like Kubernetes itself, right? Probably one of them, I think at this point, it is the most active project in GitHub. And you've got a community that is probably at this point. well into the hundreds of thousands of people who have interacted at some point via an issue, via a PR or Slack messages. And, you know, all of those are successes, but each one is different in kind, right? And consequently, the way you as the core maintainer of the project go is gonna vary rapidly. You know, Brian Grant's stress level was much higher than my stress level has ever been because, you know, he had to... worry about production outages at major cloud providers all around the world. Whereas Helm might have been like something went down for a little while in the PHP HTML5 parser. As far as I know, nobody ever got angry with me and threw stuff at my screen

 

Right.

 

image or something like that.

 

You did write your HTML parser with regular expressions, right?

 

Oh man, I had really nerded out at that point about, you know, recursive descent parsers and things like that. And there, I tried to write it without regular expressions and in the end, it's always like one case where you're like, fine, fine.

 

Yeah.

 

Whereas

 

Yeah

 

PCRE, you know, quick reference and let me spend 45 minutes trying to figure out how many asterisks and dots I need in this thing.

 

Yeah. Yeah. Yeah. Yeah.

 

So the question that I think we have to ask, because we're talking about Helm, what's coming up in the future? I don't know how closely you're involved with the project anymore, but I imagine you have at least some finger on the pulse there.

 

Yeah, by way of editorializing, which apparently is all I do, being a CEO is kind of neat, but I don't get to write code ever anymore. I mean,

 

Yeah.

 

like, yeah. And I do miss that. So while I'm involved in Helm still, it's more as an observer and occasional. I'm sort of like the person who explains why we did it that way in the past now.

 

Hahaha!

 

But yeah, still go to the Helm calls, the weekly Helm calls as frequently as I can and still on the governing board for Helm. You know, one of the big transitions Kubernetes as a whole and consequently Helm has made over the years is going from, you know, the cutting edge, highly experimental. proof of work sort of thing where we can say, look, we can MVP, you know, go from proof of concept to a minimum viable product, which is probably Kubernetes 1.2. And then people start using it and expectations change. I think we got away with a lot in the helm two to three migration. And a lot of that was because people like Remus and Martin jumped in and helped smooth the transition between the two. But it's unlikely that we'll be able to pull off large scale feature changes in Kubernetes or or helm. over time because so many people rely on it and so many organizations are dependent on the way they function. So at some point in Helm's not too distant future, maybe a year and a half ago, we transitioned over to a model that was designed to foster stability at the expense of introducing new features. So no backward compatibility breaking changes, no new things that are just totally different from what the user expects. very strict guidelines about what command line arguments have to look like and how to do feature flagging and stuff like that. That's a deeply pervasive part of the community psychology from our core maintainers now. And so as we've talked about Helm 4, we're not sure sort of

 

Thanks

 

what

 

for watching!

 

we would wanna do. And occasionally we've opened up forums and said, okay, what are big features that people would like to see? This is our one chance to introduce breaking changes. What do people want to see? And we'll get a few ideas, and people do have some good ideas, but nothing that's really pushed us to say, okay, now is time to start development. We intended to announce development right before, I think it was in 2021, right before KubeCon LA, but we couldn't muster enough of a feature list to justify announcing it there, and it's been sort of perpetually backburnered now. That said. Red Hat in particular, but a number of other organizations have become more active in the Helm community recently, and it's been great. You get some new ideas in there, and some people who have seen things that we have not seen. And I wouldn't be surprised if 2023 is the year that people say, okay, here's a feature set we want to see in Helm 4. So I'm kind of holding out the option here that that discussion could start up again early next year and really kind of go forward from there. Again, I don't think we'll see a sea change like we did from Helm 2 to 3 where we changed from a client server model to a peer client model. But what I do think we'll see is more features that are aligning better with the way people are actually using Kubernetes and we see patterns drop in. The perennial problem with Helm 3 has been how we handle CRDs. And that's probably a three hour podcast in its own right. It's such a complicated topic. And we've taken a very risk averse. approach to it. But now I'm starting to hear interesting ideas and people might come up with good ways to deal with the operator use case, the CRD use case, and that might be the catalyst that kicks off a Helm 4. But for the remainder of this year, the big item on our schedule is like the Helm booth at KubeCon, so

 

Hehehe.

 

nothing boat rocking there, just

 

Yeah.

 

a fun chance to get together with the community and connect again.

 

Nice.

 

Which is not a bad thing. I mean, that's actually a great place for a project to be in. You've got a solid supporting community, and the product you've built is meeting their needs.

 

Yeah. In fact, I think one of the most dangerous assumptions we sometimes fall into making is that for a project to continue to be relevant, its feature set must continually change. And to some extent, that part might be true. But then we add on to that the thing saying, well, this project seems to be not adding new features. Therefore, it must be stagnant and irrelevant. when in a lot of times we avoid adding on new features because we want to encourage stability. In particular, infrastructure is that way, right? We used to, during COVID, right? When we're all trying to find ways to distract ourselves, during the worst of it, when we're all sitting at home, the team got into the habit of posting the JavaScript framework of the day, which was just sort of a joke that

 

Hahaha

 

there are so many JavaScript. But part of the reason there are is because the nimbleness of front end development really requires it. Design is constantly changing. The specs are constantly evolving. In operations and platform engineering world where we all live, that would be a bad thing if we had a new Kubernetes alternative every week, right? We just go mad, you know, continually trying to port our stuff from one platform to another. So systems engineering is very different in the sense that we strive for stability. And so maybe you're right. Maybe there is something to be said for the fact that when a project slows down a little bit in this space, it's because we are, to use like Eric Raymond's phrase, we're scratching the itches that are out there, right? We're meeting people where they're at and bug fixing is in that sense, the most correct thing you can do for the community and introducing dangerous new features might be the worst thing you could do for your community.

 

Yeah, for sure.

 

Well, I've always said that boring is good in this industry.

 

Yep. Yep.

 

Maybe it doesn't sell conference tickets or whatever

 

No!

 

other measure of commercial success you might be looking for, but it is what we're looking for.

 

Yep, yep. Yeah, libraries like OpenSSL, man, there's no glamour involved

 

Right.

 

at all in maintaining OpenSSL, but the internet would be a vastly different place if it weren't for that one library.

 

The only famous people on that on that library are the ones who make introduce the bugs, right?

 

Yeah!

 

They get discovered

 

Yeah.

 

20 years later.

 

Number one way to turn your name into a curse word is to introduce a bug

 

Yeah

 

and open a cell.

 

Yeah. Cool. Well, this has been a fun conversation for me. I enjoy using Helm. I haven't been for a while, but just because the project I'm working on mostly right now isn't using Kubernetes, but I've enjoyed Helm and Kubernetes, so I look forward to getting back to it at some point. Maybe by then Helm 4 will be out and we'll be using quantum Kubernetes or whatever the new

 

Ha ha

 

feature set will be.

 

ha ha! Quantum Coover, I felt a shiver go down my spine when you said quantum. I

 

Yeah.

 

don't know what

 

Hahaha!

 

that shiver means, but it was a

 

Okay,

 

shiver.

 

I won't ask for that yet. I'll just ask for Kubernetes in the browser. That's all I really

 

Ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha

 

need. Ha ha

 

Good.

 

ha. There we go.

 

Speaking of which, I do want to follow up with one question on that, because we had Taylor on the podcast, whose former was with the Helm project and is now working on Wasm. You obviously were deeply ingrained in the Helm project. Now you're working on WebAssembly. Is there a pattern here that I should be picking up on?

 

I would say yes. I mean, a lot of us are interested in figuring out there are definitely problems that Kubernetes and containers did not solve. And there are a lot of us trying to figure out how to do that.

 

What?

 

Oh! Hahahaha!

 

You know, I know, you're gonna like bleep out when I say that and then the final three.

 

Oh yeah, the editors are definitely cutting that out. Ha ha ha ha ha.

 

There are a lot of Yeah. And as we, there's room now and we start, technological innovation and evolution happens when we start to say, hey, we solved this batch of problems, what's the next batch of problems? And you can solve it, can we solve it with existing tools? And if the answer is no, then the next question is, then what do we need to build? And Taylor and I are well aligned on the idea that We hit some limitations in the way we wanted to use cloud services. They're expensive, getting more expensive, bursty workloads are hard to deal with. There's still a lot of tension about building cross-platform, cross-architecture, Docker containers, things like that. And we said, okay, there's got to be a way to solve this. And WebAssembly is a functional piece. that opens up some opportunities. And Taylor at Cosmonic, they're trying one way of doing this. Us at Fermion, we're trying another way. Hopefully both of us will be successful in this. And there are many more people trying this as well. I'm really optimistic about that. And I think that this is just one more sort of evolutionary cycle where we solve a few more problems today. Tomorrow we'll discover that even this solution didn't quite cut it all the way and there will be another thing that comes along. But that's what makes this industry great is that we never get bored and we will all be employed until the heat death of the universe because there's always something else to do.

 

Even after that, somebody's going to come back and ask for us to fix a bug.

 

This was not this this software does not function at below zero degrees Kelvin can

 

Right?

 

you?

 

Hmm. Ha

 

Right?

 

ha ha. Well, I'm a little bit disappointed that you were wishing Taylor well because I was hoping to have you both back on for a deathmatch episode, but

 

Yeah!

 

I guess that won't be happening.

 

I mean, we can fake it. I'm sure we can come

 

Okay.

 

up with something.

 

Yeah.

 

We'd probably disagree more on which musicians we like than we do about which technologies we like, but yeah.

 

All right,

 

Look,

 

Adventures

 

if WWE

 

in DevOps Music

 

can pull

 

Edition.

 

it

 

Ha

 

off,

 

ha

 

we can pull it off.

 

ha.

 

Taylor and I, I think, are speaking together. The plan is that we'll speak together at WebAssembly Day, which is one of the WASM Day, which is one of the pre-days at KubeCon in Detroit. So if you're around in Detroit, you can catch us there. I think that's the Monday of KubeCon. And if not, all the videos will get posted later. So if we get into some kind of pugilistic death match, it'll all be there for you viewing online

 

Okay,

 

or in person.

 

get your tickets now ladies and gentlemen.

 

I'd be ashamed if there were a bunch of folding chairs on stage when you two were speaking.

 

Hahaha!

 

In this corner we have...

 

Right?

 

As long as I get to wear the big feathered boa cape thing that I can throw off my shoulders dramatically, I'll be happy.

 

I would pay to see that,

 

Ha ha ha

 

totally.

 

ha

 

Absolutely.

 

ha! There's no place

 

I think you should

 

for

 

do that regardless.

 

a cape. Ha ha ha ha ha

 

Because

 

ha!

 

he may not catch this episode, but it'll be even funnier if you just walk out on stage with that and throw it off with no context.

 

We do have

 

You're

 

a joke

 

in the-

 

that Taylor and I have very different conference attire. I usually come wearing a tie and a sweater and a collared shirt, not necessarily in that order, usually shirt, tie, then sweater. And he comes wearing that.

 

Hahaha

 

So

 

Well, if you don't have exciting new features to announce for Helm 4, you have to do something to get attention. It sounds

 

That's,

 

like you figured it out.

 

yeah, that's right. Yeah, things I learned from the Kardashians, there's got to be something

 

HAHAHAHA

 

I hope that's the only time the Kardashians have ever been mentioned on this podcast.

 

Probably so. Certainly in my memory.

 

Open source community building with the Kardashians.

 

Seven things

 

And

 

you need

 

on that

 

to know.

 

note, it's

 

You

 

time

 

won't

 

to

 

believe

 

wrap up.

 

number

 

Thanks

 

three.

 

everybody for attending.

 

I

 

Oh

 

Is there anything else we should talk about? Is there anything we forgot

 

Yeah

 

to ask about related to Helm or the Kardashians or anything else?

 

Uh. I mean, you know, and going back to the WebAssembly thing, I really, I'm excited for the future of cloud commuting because I think we're going to see another wave of innovation. I think innovation tends to go in waves and I think we're just on the uptick of one. I'm really optimistic about the next few years and the things we'll see both, you know, in the Kubernetes container ecosystem as they continue to flesh out the things that they're working on. My interests now are kind of moving on from Kubernetes and containers into WebAssembly, which is a deeply promising space. layers upon layers of innovation there. And that's really, you know, again, that's kind of what excites me and what gets me up every morning, that and coffee. But yeah, I think there's some exciting stuff coming ahead. We're excited about the stuff we'll announce at KubeCon. We have a cat game at KubeCon to explain how WebAssembly works. So, you know, anybody who's there, stop by, play a cat game, get a t-shirt, it'll be great.

 

Sound awesome?

 

I can't tell you, but I'm just a little bit disappointed to hear that because I've been interested in WebAssembly and kept putting off learning it. Then we had Taylor on the show. And at the end of that show, I was excited about WebAssembly. And now after talking with you, I'm even more excited about infrastructure with WebAssembly. And it's like, dang it, I did not need another project on my

 

Ha

 

list

 

ha ha.

 

right now. But honestly, it's a great opportunity. I'm intrigued and I want to figure out how this plays into what we do for a living because I think it's pretty exciting.

 

Yeah.

 

All right, Will, I'm looking forward to you releasing Kubernetes in the browser.

 

Hehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehe

 

We have to start all over again, take it from the top.

 

Right?

 

Alright Matt, how can people get in touch with you if they're interested? Where can we follow you?

 

I am Technosophos everywhere. T-E-C-H-N-O-S-O-P-H-O-S. I'm Technosophos on Twitter, Technosophos on GitHub, and other places that I can't remember. Ah, ha, ha, ha, ha.

 

Hahaha

 

That's definitely the easiest way to find me online. I'm on Twitter a lot, I'm on GitHub every day. So those are good places. And then we hosted Discord, Fermion has a Discord server. I'm around in there quite frequently. And occasionally I answer messages in the Kubernetes Helm Slack, but that's getting rarer and rarer as the days go by.

 

And your company, do you have, are you, you have products we can buy yet? Or, or is it still, uh, in the pre pre launch phase or whatever.

 

Fermion has a number of open source projects that are out. The developer tool, Spin, is the best way to get started. You know, we like to say you can go from blinking cursor to deployed app in two minutes or less. Radu, the CTO, keeps saying, it's really one minute or less. I'm like, I know, but two sounds so catchy.

 

Hahaha

 

But Spin's the developer tool, all open source. It always will be. We have a Fermion platform, which is an open source platform you can install on AWS, DigitalOcean, Azure, whatever you want to kind of try out what we think the next wave of cloud computing is going to look like. It's powered by Nomad as its scheduler instead of Kubernetes. We've really been enjoying working with HashiCorp on. using the Nomad scheduler for WebAssembly. It's just very well, they're very well aligned as far as project goals go. So that's something else you can try. Then we'll have big announcement coming up at KubeCon that we're really excited about that really sort of sets the vision for where we're going from here on out. But we've been around,

 

Awesome.

 

November 1st will be our birthday. And so we've been around less than a year at this point. And it has been astonishing to watch a group of highly skilled engineers. realize their vision like right in front of me, right, as they're building out this platform. So we're really excited about the announcement coming up at KubeCon.

 

Any hints on the announcement or is it a top secret?

 

Uh,

 

Sounds like a secret.

 

uh, I have some kind of PR guidelines I'm supposed to be able to use to hint at stuff, but you

 

Uh-huh.

 

know, we're

 

Yeah

 

shooting for the kind of platform that'll make it really easy for people to get started on this and get going

 

Awesome.

 

and see something up on the internet very fast. So that's

 

Very

 

probably

 

cool.

 

way more than the PR people would tell me I'm supposed to say, but you know, we're friends. We'll

 

We won't tell them

 

Okay.

 

that you said it.

 

only tell all the people who listen to this podcast.

 

Well, when is the Detroit KubeCon?

 

I believe it's the last week of October. I've

 

Last

 

been

 

week

 

pushing

 

of...

 

for a Halloween

 

okay.

 

theme. I want a reason to get candy at KubeCon, but yeah.

 

Right

 

I'm going up soon.

 

now I was trying to figure out if this episode would be released prior close enough to then where it Wouldn't get you into trouble, but um,

 

Hahaha

 

no we'll be

 

Hehehe

 

we'll be This episode will be live a few weeks before that. So

 

Yeah.

 

Okay, all right. That sounds safe then.

 

Yeah, yeah. Want to keep it safe so that you have nice things to say about us and you're willing to come back on the show a second time

 

Yeah!

 

so we can have a deeper conversation about Wasm and WebAssembly infrastructure.

 

Yeah, and to learn more about WebAssembly and what Fermion's building, fermion.com is a great place, or you can follow fermion at fermiontech on Twitter. And we write lots of blog posts ranging from, you know, very practical ones about what we're doing to rather pie in the sky ones about why we are excited about WebAssembly in general.

 

Awesome.

 

pie in the sky? Is that really a thing or

 

I

 

seems like something

 

don't

 

vaguely?

 

know. I hear the phrase. I don't know what it possibly means.

 

Yeah,

 

Hahaha

 

it's a legitimate

 

I mean...

 

phrase, but I have no idea what, like, if there's

 

If you

 

pie

 

know what pie

 

in the

 

in

 

sky.

 

the sky means, come on and be a guest on the show and explain

 

Hahaha!

 

it to us. All right, shall we do some picks? Let's

 

Let's do some picks.

 

Yep.

 

do some picks. I completely forgot to get picks. Do you have a pick ready, Will?

 

I do this week I have an anti-pick and a pro-pick. So my anti-pick I started reading a book. I keep wanting to say Necrocomicon or Necronomicon but that's not the right book. That's actually the book of the dead or something. But I keep messing

 

Oh.

 

up the title. The book I started reading was Crypto-nomicon from Neal Stephenson. And this is my anti-pick because I just couldn't get into it. You know, it starts off where there's this guy who he's bike riding with Alan Turing, and then he's learning about how the mathematical relationships between the pipe organ makes the different sounds. And then there's this other guy who's flying to the Philippines. And the first few chapters. It took me three chapters to figure out that the previous three chapters were all about different people. And so it was really hard for me to get into. So that's my anti-pick.

 

So kind of like the book version of Inception.

 

Yeah, yeah, exactly.

 

The interesting thing about that book is he sort of saw cryptocurrency coming a long time before anybody else did.

 

That's what triggered me to read it. I'd read that same summary about it, and I was like, oh, that's cool. But I just couldn't get past enough of the intro chapters to get into the plot. So my actual pick for the week then is I put that down and picked up Stephen King's latest book. I think it's his latest book, Fairy Tale. And within the first three or four paragraphs, I was just sucked down deep into that world. and spent way too much time last night reading that book.

 

Hehehe

 

Which a lot of times I find Stephen King books very similar, you know the first few chapters are hard to get into but this one for whatever reason just grabbed me by the shirt and jerked me straight down the tail. So check that one out. Those are

 

Awesome.

 

my picks.

 

Cool. Well, my pick for the week is some shameless self-promotion. My new YouTube channel is up and running and has a few videos on it. And I have a couple in the pipeline will be coming out all about programming in Go. So if you want to learn

 

Hmm.

 

to hack on Helm or write that new Kubernetes in the browser thing in Go that's compiled to WebAssembly or just do normal things even, that's fine too. My channel called Boldly Go is up and running, waiting for your subscription. Be sure to like and subscribe and all those cheesy things that everybody says on the YouTube videos. So yeah, check out Boldly Go on YouTube. I'll have a link in the description.

 

That's a clever name. I like it. I'm

 

Got

 

hopeful

 

a neat little

 

that there's

 

Star

 

some

 

Trek

 

Star Trek.

 

sort

 

Yeah.

 

of vibe going to it. Yeah.

 

Yes. Excellent.

 

I think mine, now I'm kind of inspired for a mini pick before giving my real pick, which was

 

Awesome.

 

we discovered, and this is mini in multiple dimensions, tiny go is an alternate go compiler that can compile to embedded devices and small devices and also to WebAssembly. And so we have been using that quite a bit to compile Argo code to WebAssembly. Very, it allows you to do all kinds of neat little optimization tricks. It's a lot of fun. But my bigger pick would be Notion. I, I, tried Notion a long time ago and just didn't get it. And then thought, oh, this is yet another Wiki-ish kind of platform. I don't understand what I'd use it for. But then on a whim, I went back and actually watched through the video tutorial. And once I started to understand the model behind it, I got really interested. And then, of course, it became the new hammer, and everything looks like a nail, and I'm trying to put everything in Notion. But the whole idea of a database, structured database, and a documenting system has really sort of resonated with me. And I've been having a really good time working on Notion, everything from helping me keep track of meetings and to-do lists and things like that to organizing Dungeons and Dragons games. Cough. So.

 

Hahaha!

 

Nice. Which rule set do you use though? That's the real question.

 

Oh no, now I'm gonna be embarrassed, because I'm using 5e. I have not tried the new candidate for 1DND, and also 5e was the first one I ever played, so I've never tried any of the previous rulesets.

 

Oh well.

 

I'm a mainliner!

 

Yeah!

 

I make it sound like I'm gonna make fun of somebody for not being a purist, but I've never actually played D&D in person. I've only ever done the Baldur's Gate in other online series. They're made by the same company. So I guess they're official. They use the official rules, but I've never worn a costume or sat there and rolled a D12 in person. So

 

I've

 

I'm

 

never

 

not really

 

worn

 

a

 

a

 

geek

 

costume

 

either

 

either.

 

yet, but you will at the next conference, right?

 

Yeah,

 

At Wasm Days.

 

if we've got a KubeCon that's Halloween themed, I'll come dressed as a D&D character. Just putting that out there for any of you influencers out there who can make it happen.

 

Awesome.

 

It seems kind of like it's got to go hand in hand. You're in Detroit right before Halloween, for KubeCon. How can you not add costumes?

 

Yep.

 

Will, he only asked for influencers to make it happen. Aren't you an influencer?

 

Oh

 

Just declare it so.

 

I don't think I have that kind of influence. I can influence other things. Or, yeah.

 

haha

 

We won't talk about that on a family friendly show. Ha ha ha.

 

Right, yeah.

 

All right, guys, I think it might be time to wrap this up. It's been a lot of fun. Any last words before we hang up?

 

No, you know, every time someone says we should talk about Helm, I think I have nothing to say about Helm. And then every time

 

I mean...

 

I open my mouth, I discover I have a whole lot to say about Helm, and I'm surprised anybody will ever listen. So thank you very much for listening. Ha ha ha ha ha.

 

Well, thank you for sharing.

 

No.

 

I found it quite enjoyable.

 

Thanks.

 

Yeah, this was great. Thank you so much.

 

Thanks for having me on the show.

 

Thanks and until next time, enjoy your DevOps. I don't have