Nicholas Hughes is the CEO of EITR Technologies. He joins the show to talk about "Salt". Salt is used for deploying, configuring, and managing complex IT systems. He begins by explaining what Salt is, its useful features, advantages, and why he loves it. He also shares the difference between salt and other similar frameworks. Moreover, he talks about its process, set up, how it is being implemented, and many more!

Jonathan (00:02.67)

Ladies and gentlemen, welcome to another exciting episode, a salty episode of Adventures in DevOps. I'm your host for the day, Jonathan Hall. In the virtual studio, I also have co-hosts, Will Button and Jillian Rowe. I almost forgot your last name there, Jillian. Yeah.

 

Jillian (00:03.78)

Oh no! I really do need to show up to the podcast more often, don't I? What? Is that how that works? I have to talk to people for them to remember me? Like, what is this?

 

Will (00:10.8)

Thanks for watching!

 

Will (00:22.13)

What? Is that how that works?

 

Will (00:29.969)

Shit.

 

Jonathan (00:30.762)

And we have a very special guest today. I scoured all of my social media contacts and accounts looking for somebody to come teach me about salt. And we found Nicholas Hughes. Welcome Nicholas.

 

Nicholas Hughes (00:41.982)

Thanks for having me.

 

Jillian (00:42.216)

Thanks for having me. Thanks for coming on. Why don't you tell us a little bit about yourself, what you do, and maybe a relationship to solve in DevOps, and then we'll start a conversation. Sure. Nicholas Hughes, I'm 6'3". I like long walks on the beach. Are you Nicholas Hughes, or is this just the art of something beautiful to somebody else? Yeah, but I don't know what demographics you think are on this show. But um.

 

Jonathan (00:43.278)

Thanks for coming on. Why don't you tell us a little bit about yourself, what you do and maybe a relationship to Salt and DevOps, and then we'll start a conversation.

 

Nicholas Hughes (00:53.302)

Sure, uh, Nicholas Hughes, I'm 6'3", I like long walks on the beach. Um... Heh. Uh...

 

Will (00:59.786)

I'm sorry.

 

Jonathan (01:01.563)

Are you single too? Because this might be the start of something beautiful with somebody listening. Ha ha ha.

 

Nicholas Hughes (01:10.666)

laughter

 

Jillian (01:12.24)

I'm not sure this is going like the way that you might want for it to go. Just throw, just throw on that out there. I like being on the dating game, but I'm sure my wife would not like.

 

Nicholas Hughes (01:18.683)

Uh, I would start joking about like being on the dating game, but I'm sure my wife would not like that. So, um, yeah, so I run a consulting company called EITR Technologies. We've been in business for about five years now. Um, and broadly we, we do automation, right? So, we work with a lot of different automation systems. Um, but I have a special place in my heart for salt.

 

Jonathan (01:23.423)

Yeah, all right.

 

Will (01:23.473)

Yeah

 

Jillian (01:25.84)

Yeah, so I run a consulting company called EITR Technologies. We've been in business for about five years now. And broadly, we do automation, right? So we work with a lot of different automation systems.

 

Nicholas Hughes (01:45.11)

because I've probably been using that close to eight years and been really tied into the open source community. You know, we do a lot of work with, you know, just the open source project as a whole, customers can come to us and say like, hey, you know, this functionality is broken or we would love this feature or something like that. And we'll work with them, not only on their implementation, but also as sort of like a,

 

Jillian (01:45.76)

I've probably been using that for close to eight years.

 

Jillian (01:50.884)

been really tied into the open source community.

 

Jillian (01:58.163)

if you know just

 

the open source project as a whole, customers can come to us and say like, hey, this functionality is broken or we would love this feature or something like that. We'll work with them, not only on their implementation, but also as sort of like, I don't know, open source developer for hire. So we're really well integrated into that ecosystem and it's just a big part of what we do. So when Jonathan reached out, looking for somebody, he said,

 

Nicholas Hughes (02:14.43)

I don't know, open source developer for hire. So we're really well integrated into that ecosystem and it's just a big part of what we do. So when Jonathan reached out, looking for somebody and said, hey, I might know a thing or two.

 

Jonathan (02:30.222)

That's awesome. So I don't know. I mean, Jillian knows what salt is. I must be in the minority here, and that's OK. I'm happy to be the stupid one in the room. But I only learned about salt recently when a new client I started working with was using salt. And I'm still very much in the dark. Would you assume that there's at least one other person in our audience who's never even heard of salt and give them a brief description of what it is?

 

Jillian (02:32.36)

So I don't know, I mean, Jillian knows what salt is. I must be the minority here, and that's okay. I'm happy to be the stupid one in the room. But I only learned about salt recently when I...

 

Jillian (02:46.688)

and I'm still very much in the dark. Would you assume that there's at least one other person or audience who's never even heard of Salt and give them a brief description of what it is? Sure. Salt is an open source Python project and it can do anything. So it's like Emacs. No, it's like Fim. So.

 

Nicholas Hughes (02:57.214)

Sure. Salt is an open source Python project, and it can do anything. Exactly. So really, Salt was developed by Thomas Hatch back in 2011 as an event-based.

 

Jonathan (03:06.318)

Haha, so it's like Emacs.

 

Jillian (03:15.868)

You know, really, it's always developed by Thomas Hatch back in 2011 as an event based automation and orchestration system. So really the event bus is the first party citizen in this. And that's one of the reasons that I say that it can do anything. Right? Like a lot of...

 

Nicholas Hughes (03:24.874)

automation and orchestration systems. So really the, the event bus is the, the first party citizen in this. And that's one of the reasons that I say that it can do anything, right? Like a lot of salts, competitors might be like chef or puppet or ansible or whatever. Right. And like a lot of those systems were first party configuration management, right? Like that's the thing that they set out to accomplish. We're going to,

 

Jillian (03:37.04)

salt competitors might be like chef or puppet or ansible or whatever right and like a lot of those systems were first party configuration management right like that's the thing that they set out to accomplish we're gonna you know

 

Jonathan (03:50.933)

Mm-hmm.

 

Nicholas Hughes (03:54.942)

you know, throw down the contents of this file and make sure that this service is started and all of those things. Salt can do all of that, but at its core, it was really the event bus and the remote execution system that enables it to do everything that it does, including all of the configuration management things. So some use cases for Salt might be

 

Jillian (03:57.64)

down the contents of this file and make sure that this service is started and all of those things. Salt can do all of that, but at its core it was really the event bus and the remote execution system that enables it to do everything that it does including all the configuration management things. So some use cases for Salt might be...

 

know installing applications right like making sure that nginx is installed and the services started and certain websites are enabled maybe it's like system security right you know you're beholden to NIST or any other framework right you know you can write security lockdowns

 

Nicholas Hughes (04:27.683)

installing applications, right? Like making sure that NGINX is installed and the services started and certain websites are enabled. Maybe it's like system security, right? You're beholden to NIST or any other framework, right? You can write security lockdowns.

 

and roll those out with salt. You can perform remote execution, which always tends to make security folks nervous when you say that, because it's not the nefarious remote code execution. This is more like an equivalent to like PSSH or something like that, right? Like I can reach out to all of my systems and do a thing.

 

Jillian (04:55.732)

roll those out.

 

Jonathan (05:08.099)

Hehehe

 

Jonathan (05:19.863)

Okay.

 

Nicholas Hughes (05:24.906)

and like execute a command and get results back and things like that. So, you know, that's what you tend to see salt used for the most, but it can do a lot of things. And that's one of the reasons that it's often considered to be a little confusing.

 

Jonathan (05:45.08)

Hmm

 

Nicholas Hughes (05:45.338)

starting out because there isn't necessarily one way to do anything in Salt, right? Like, you can do things multiple ways and it really depends on your use case, which is the exact reason that it's valuable to me as an automator, just because, you know, any particular thing that I want to do, I can generally do it. It's like having a really big

 

Jonathan (05:51.87)

Mm-hmm.

 

Nicholas Hughes (06:13.958)

Swiss Army knife in my pocket.

 

Jonathan (06:15.874)

Okay.

 

Jillian, you've used salt before apparently, and had some interesting things to say about it before we could record. Since you have a little more experience than I do, do you want to, I don't know, can you chime in on your experience with salt and how it compares to what Nicholas just said?

 

Jonathan (06:44.534)

You're on mute if you didn't know that.

 

Nicholas Hughes (06:49.45)

And this is right when the contractors show up.

 

Jonathan (06:51.136)

Will's also on mute.

 

Jonathan (06:57.978)

You're unmuted and muted again, Will. Jillian's unmuted now, but not speaking. There she is! That's alright.

 

Jillian (06:58.656)

You unmuted and muted again, Will. Hehehehe Killian's unmuted now, but not yet. I'm here! I'm here! Sorry! Killian is killing us!

 

Nicholas Hughes (07:01.066)

Yeah.

 

Will (07:04.315)

My god, this technical stuff is killing us today.

 

Nicholas Hughes (07:09.494)

Hehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehe hehehe hehehe

 

Jillian (07:13.808)

Sorry, I have a... No, no, I didn't. I'm sorry. You asked just as contractors came in and I had to let them in.

 

Jonathan (07:14.006)

Did you hear my question, Jillian? Okay.

 

All right. Yeah, we figured. All right. So what I said, well, that might mean that you missed part of what I was going to ask about anyway. But my question was, as somebody who's you salt before, how does your experience line up with the description that Nicholas just gave? Or would you add anything? You know, as somebody a little bit maybe less experienced than Nicholas, is there something else that stands out to you as relevant?

 

Jillian (07:48.902)

Nicholas, is there something else that stands out to you that's relevant?

 

Jillian (08:08.116)

that could do that, because I remember at the time that I started using Salt, I'd been using Puppet before that. And I want to say it couldn't respond to events. I don't know. But this was a while ago, you guys. It's all getting kind of fuzzy. All the tech just sort of bleeds together.

 

Jonathan (08:18.349)

Mm-hmm.

 

Nicholas Hughes (08:26.578)

Yeah, Puppet eventually did have sort of like attack on event based service, but it was not capable of the same type of event driven automation and orchestration that SALT has been for years.

 

Jonathan (08:43.702)

So when I hear you talk about this, a lot of the things you say kind of remind me of Kubernetes, like event driven. I know Kubernetes isn't per se an event driven thing, but it has an event driven component to it, right? Can you help me compare and contrast? Like, I think that the answer is that salt and Kubernetes are completely different, but help paint that picture, because I imagine there's confused people listening just like me.

 

Jillian (08:43.999)

So when I hear you.

 

I know

 

Jillian (08:51.456)

I mean, it isn't per se anything but a thing that it has in it.

 

Jillian (08:57.172)

Can you help me compare and contrast? I think that the answer is that salt and Kubernetes are completely different. But help me in that picture.

 

Nicholas Hughes (09:10.83)

Yeah, so Kubernetes definitely can be sort of an event-driven platform, right? And it can feel sometimes similar because generally you're going to be writing your manifest in YAML and it's declarative, right? You write this YAML and you say, this is the state that I want things in and you apply that.

 

Jillian (09:10.984)

Yeah, so Kubernetes definitely can be sort of an event-driven platform, right? And it can feel sometimes similar because generally you're going to be writing your manifest in YAML and it's declarative, right? You know, you write this YAML and you say this is the state that I want things in and you...

 

Nicholas Hughes (09:38.162)

against your Kubernetes platform and it's there. And it's exactly how you expect it to be, right? Salt is similar in that regard, right? Salt has a declarative piece to it where it does stateful enforcement of things, right? Like file states, services, package, installations and things like that.

 

Jillian (09:38.432)

against your Kubernetes platform and it's there. And it's exactly how you expect it to be, right? Salt is similar in that regard, right? Salt has a declarative piece to it where it does stateful enforcement of things, right? Like file states, services.

 

Jonathan (10:06.375)

Mm-hmm.

 

Nicholas Hughes (10:06.566)

So it definitely has a similarity in the declarative nature of how it works. Salt also by default is going to leverage YAML for everything that you would write in it. There are other ways that you can write those files for Salt, but generally YAML is the most popular and the one that you see the most.

 

Jillian (10:06.74)

So it definitely has a similarity in the declarative nature of how it works. Salt also by default is going to leverage YAML for everything that you would write in it.

 

Jonathan (10:21.172)

Mm-hmm.

 

Jillian (10:24.832)

ways that you can write those files for salt, but generally, yaml is the most popular.

 

Nicholas Hughes (10:38.447)

Um, those similarities aside, right? Like Kubernetes is a whole platform, right? Like a lot of people call it a cloud operating system because it, um, you know, can run applications in a distributed nature. Um, salt is not that deep of a platform, right? Like running things at a core level. Um,

 

Jillian (10:39.404)

Similarities aside right like Kubernetes is a whole platform right like a lot of people call it a cloud operating system it's You know can run applications in a distributed nature Salt is not that deep of a platform right like running things at a core level

 

Jonathan (10:44.758)

Mm-hmm.

 

Right.

 

Nicholas Hughes (11:05.618)

You know, it can.

 

Jillian (11:05.896)

you know, it can...

 

Nicholas Hughes (11:09.222)

It can be used for rolling out applications. It can be used for ad hoc imperative things like, I want to get the disk usage of these systems right now, or something like that. It can be used for orchestration across machines. Say you have a web server and a database server, and you want to make sure that the database server is installed before the web server.

 

Jillian (11:09.536)

can be used for rolling out applications, it can be used for ad hoc imperative things, like I wanna get the disk usage of these systems right now, or something like that. If the...

 

Jonathan (11:22.537)

Mm-hmm.

 

Nicholas Hughes (11:36.83)

You can do those high level abstractions, where you make sure that this machine is done before you roll on to the rest of your orchestration. But there's nothing that's really hosted per se inside of Salt. It's more of a tool to get a job done as opposed to Kubernetes, which is hosting things.

 

Jonathan (11:51.189)

Mm-hmm.

 

Jonathan (11:59.754)

Yeah, yeah. Okay.

 

So maybe we could, I don't know the best way to go here. Will, feel free to jump in with any questions or comments you have too. I don't want to dominate this necessarily. Yeah, good.

 

Will (12:11.05)

Yeah, I do want to jump in real quick to, so with salt, you know, Puppet uses an agent, Ansible uses SSH. What's the method for salt to talk to the different things that it's controlling?

 

Nicholas Hughes (12:26.494)

How do you want to run it? Yeah, so the default setup for Salt is what they call a master and minion setup, it's agent based. So you would have a server component that's running and that's where all of your configurations are housed, like that's where you write the things that the minions are going to do. And then,

 

Will (12:28.334)

All right, perfect.

 

Nicholas Hughes (12:55.818)

Minion software would be installed on all of your systems. And, you know, it either checks in intermittently through a schedule or you can do event-based lockdowns, right? Where like, oh, I see that this file was just changed by Will on this server, right? We don't want that, you know, we're gonna slam down something on there, right? So like we can trigger events that then

 

Jonathan (13:20.106)

We really don't.

 

Nicholas Hughes (13:24.15)

come down and lock down that file again, right? Make sure that like local changes aren't being performed. So that's like a really valuable way to do things. The other ways that you can install Salt, you can use a component called Salt SSH that's very similar to how Ansible runs.

 

It uses SSH as the transport. Essentially, it creates like a thin little package of what needs to be done and transports that over to the system. And then it runs and then the result is shipped back over the SSH transport so you can see what happened. So if you want more of an agentless type of setup, you can certainly do that with SALT as well.

 

Nicholas Hughes (14:21.214)

There are folks that have been running Salt masterless. So think of an agent installed on the system, but it's not calling out to any sort of head end, right? So like, it's a really popular way to implement things in the cloud because you could have these ephemeral resources coming and going. And so what you can do is as long as your configuration artifacts are deployed to

 

whatever Minion is doing the work, it can just run locally and say like, all right, well, I'm gonna lock this system down. I'm gonna install this application or this or that, whatever, and then it's all there. And if that system goes poof, then there's nothing to clean up in a master, right? There's nothing like, oh, well, you know, we used to have Minion 12, where did that go, right? And then finally there's,

 

a really cool project called Heist that was sort of born out of some technology that the creator of Salt sort of championed. He came up with this paradigm in Python called plug-in oriented programming. And essentially, it's kind of the concept that

 

was born out of salt but abstracted and made better. And it allows you to write these plugin modules that can be cobbled together to do something greater than the sum of its parts. So it's, I don't know, you can think of it forming Voltron. You can put all these little plugins together, and then all of a sudden this application has a bunch of capabilities that weren't necessarily native to it.

 

Jonathan (16:07.84)

Mm-hmm.

 

Nicholas Hughes (16:17.926)

And so one of the applications that was written in POP is called Heist. And essentially, it's a way to deploy ephemeral agents over SSH. So think of it as I have a Saltmaster. I have Heist, sort of like somebody attached to it.

 

I reach out over SSH with Heist to some number of minions and what happens is when it reaches out it deploys an ephemeral agent that links back with the master and so it then acts exactly like a normal minion would. Except when you're done and Heist is brought down the ephemeral agents disappear off the system so they aren't.

 

like long running software, and then they just go poof, and everything's magic. So I guess the answer to the question is, yeah, there's lots of ways to do that.

 

Jonathan (17:26.958)

Ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha

 

Will (17:28.902)

Alright.

 

Jonathan (17:30.75)

I think this kind of goes back to the point you made early, Nicholas, that salt does so many things that it can be confusing. And so it's not a simple answer to some of these questions, right?

 

Nicholas Hughes (17:39.762)

Yeah, exactly. Like, you know, some of it has to do with, you know, people's personal preferences. Some of it has to do with what best practices might be for folks who have been using it for quite a while and know what works and what doesn't, especially at scale. For instance, like Salt has this concept of pillar where you can distribute

 

Will (17:56.046)

Thanks for watching!

 

Nicholas Hughes (18:09.17)

sensitive information like secrets to minions in a secure manner. So at the core of SALT, it kind of uses a PKI infrastructure with public and private keys. And so you can think of it as like, okay, a minion has a public key that the master can use to encrypt certain information. And then the only thing that can decrypt it is the private key for that minion, right? So.

 

Jonathan (18:22.353)

Thank you.

 

Nicholas Hughes (18:38.406)

with pillar it's a really good way to distribute that type of information but if you start overusing pillar uh... and put a ton of configuration information in it uh... and you say have thousands of minions you know five or ten thousand systems all hooked up to one master at scale pillar can be problematic and

 

Jonathan (19:04.23)

Hmm.

 

Nicholas Hughes (19:06.998)

That's really because the cryptographic operations to sort of like take that information and encrypt it and distribute it to the minion, that's all on the master, right? So like the more minions you have and the more information you have to do that with, it can definitely slow things down on the master. So, you know, best practice is to lightly use Pillar. But...

 

You know, it really depends, right? It depends on what you're doing, what's in there, and like how many systems you have. If you have 50 systems, like, okay, go nuts. You know, you use Pillar as much as you want, but if you have 5,000, like, I don't know, maybe don't do that.

 

Jillian (19:37.136)

But it really depends, right? It depends on what you're doing, what's in there, and how many systems you have. If you have 50 systems, okay, go nuts. You use Pillar as much as you want. But if you have 5,000, maybe don't do that.

 

Jonathan (19:45.495)

Mm-hmm.

 

Jonathan (19:50.146)

Alright, okay.

 

Jonathan (19:53.654)

So I want to maybe just tell you about how I learned about Salt and why I was so curious to learn about it. And then maybe the answer to my question is going to be very simple, and we can keep talking about other aspects. So I recently started helping a new client who is using Salt with Jenkins to deploy their software. It's sort of a.

 

I guess it's a monolith, but they kind of treated it like a microservice. That doesn't really matter. There's like two or three Docker containers they're deploying with salt. And the team that set this up, uh, I think they're in China or something, and they've left the company that was an agency or something. They're gone. I don't remember. I don't know the details. They're gone. The team now managing this doesn't have the first clue about salt. Um, and they're eager to.

 

Will (20:44.874)

Thanks for watching!

 

Jonathan (20:47.502)

to get rid of salt and switch to something else. And I sort of had this thought like, that might be a good idea, but I'd rather know what salt is for and whether we're using it correctly or not before I potentially throw the baby out with the bathwater. You know, the whole idea that, you know, don't do a refactor just because you don't understand something, come to understand it first, and then maybe refactor if it still makes sense. So that's kind of the situation I'm in here. Now,

 

I guess we're doing config management of some sort with salt, but the main thing it's doing is executing Docker update or Docker run commands over SSH essentially. I know that you don't know the details, but based on what I've said, is this ridiculous? Is this great? Or somewhere in between?

 

Nicholas Hughes (21:41.239)

It's probably somewhere in between. So this actually isn't a unique problem, right? Like I've also run into folks that I've dealt with that, you know, somebody has used salt in a pipeline and it's doing things in some sort of integral build process and deployment.

 

Jonathan (21:43.178)

Okay.

 

Jonathan (21:51.712)

Uh huh.

 

Nicholas Hughes (22:08.514)

And so when that person moves on all of that specific knowledge kind of leaves and You know now what you know, maybe we rip all that stuff out and we start over and that's kind of what people want to push towards because they They don't understand or appreciate salt right? So like the people who understand it and have used it

 

Jonathan (22:31.93)

Right.

 

Nicholas Hughes (22:36.342)

tend to be fans just because of what it's capable of. Even the people that complain about it a lot, like, oh, it doesn't do this or that, they're still not gonna use anything else because it's the best solution for the other 99% of their problems. That said, because people can become fans, all right, like, let's not forget that fan is short for fanatic, right? So...

 

Jonathan (22:42.71)

Hehehe

 

Jonathan (22:51.671)

Mm-hmm.

 

Jillian (22:55.45)

I'm sorry.

 

Jonathan (23:03.78)

Right

 

Will (23:03.976)

Jack.

 

Nicholas Hughes (23:06.683)

People will sometimes try to use it for everything because it can do everything when maybe it might not have been the best possible solution for something. So that's why I say that we at EITR are our automators. At a high level, we concentrate on automation. We aren't like a salt consultancy, but it is a big part of what we do because salt is an automation platform.

 

Jonathan (23:18.094)

Sure. Yeah.

 

Jillian (23:30.841)

It is a big part of what we do because Solve is an automation class.

 

Nicholas Hughes (23:34.622)

But if you come in and say like, hey, I want salt in my pipeline, or, you know, I want to rip salt out of my pipeline, we aren't going to necessarily be prescriptive on technology until we understand what your outcome is. Right? Like we need to understand how things currently are working, how you want them to work and what are the best tools we can put in place in order to achieve that goal. Um, and so without seeing it at all.

 

Jonathan (23:34.731)

Mm-hmm.

 

Jillian (23:35.981)

and say like, hey, I won.

 

Jonathan (23:47.882)

Mm-hmm.

 

Jillian (24:01.101)

seeing it at all, I'd imagine that it's somewhere in that realm.

 

Nicholas Hughes (24:02.878)

I'd imagine that it's somewhere in that realm, right? Like more than likely, it does the job. And there's a good chance that it is the right tool for the job, but there's also an equally likely chance that somebody just got carried away and maybe could have done this a little bit more simple.

 

Jonathan (24:02.99)

course.

 

Jonathan (24:06.795)

Yeah.

 

Jillian (24:18.313)

I really like you.

 

Jonathan (24:24.02)

Yeah. All right. So how would you advise somebody, myself or anybody listening, to get started learning about SALT? If you're interested in a new project or learning, you've just been onboarded onto a team that's using SALT and they don't understand it and you want to learn it, how do you get started?

 

Nicholas Hughes (24:44.31)

So the Salt open source community has a spectacular documentation lead right now in Elisa Rock and she is spearheading a lot of great documentation rewrites and revamps and has been for a number of months now. So there is.

 

more or less a brand new user guide that's currently housed in GitLab, which is apart from where the Salt code base is in GitHub. So we have that documentation spread out a little bit, but I think it's all accessible through links on the Salt Project website. It should be saltproject.io.

 

Jonathan (25:27.91)

Okay.

 

Nicholas Hughes (25:40.218)

And so that documentation, I think, is a great starting point for new folks. The other thing that I'll say is that we have a really good community. And so any mechanisms you can use to interact with that community, so like Slack, Reddit, things like that.

 

you can definitely get in there and ask questions, right? Like, you know, read through the docs, see what you can see, and then like pop over to Slack and, you know, ask a question. You know, there's a lot of great folks that are a part of the community that will help you out. There's also some stuff on YouTube. So, Sol Project has a YouTube channel and there's...

 

a lot of information from like past years and then also some recent stuff as well that go over some basic concepts. You know, Thomas Hatch did this like short little segment called Salt Air where, you know, like, oh, you want to understand the state system? Let's talk about it for, you know, five, 10 minutes or something like that. And so it's nice little like bite size consumable chunks.

 

Jonathan (26:55.559)

Okay.

 

Nicholas Hughes (27:03.094)

for folks to sort of pick up a little bit more information on salt and then Finally the EITR Technologies blog, you know Shil for that for a second. So the our website EITR.tech TCH We often write about salt in our interactions like whether it just be like, oh, yeah here's this new feature that we rolled out or You know

 

Jonathan (27:17.271)

Hehehe

 

Nicholas Hughes (27:32.906)

there was just a new release and we had like 40 PRs in it and here's what they all do or hey, we found this cool feature that was kind of like hidden in salt and it's not really documented anywhere so we're going to write about it. Those types of things. So between all those resources there's a lot of great information out there.

 

Jonathan (27:46.35)

Okay.

 

Jonathan (27:56.87)

Awesome. So you just touched on another question I wanted to ask. We'll do it now. What is your official capacity of any with regard to the salt project? It sounds like your company creates PRs. Are you an official contributor or what does that what would that even mean on that project? What's your relationship to salt proper, so to speak?

 

Nicholas Hughes (28:20.046)

Um, let's see. So, uh, I was the, uh, cloud working group captain, um, for about two years. Um, so salt does have capability to do cloud infrastructure automation as well. Um, salt cloud is the mechanism to do most of like, it can bootstrap VMs.

 

in a particular cloud and tie them into the Saltmaster, which is really nice. So you can do sort of like event-based automation in the sense that like I could deploy 10 VMs, right? And once they start spinning up, they're sending start service messages back on the event bus and saying like, hey, I'm here now. And then you can do automation on that. Like, okay, you know, Larry, Moe and Curly, my new VMs are all...

 

ready to go, let's go ahead and what we call high state them and basically slam down all the configurations that apply to them. So there's that. And then there's also execution modules and state modules that do cloud things inside of SALT. So like you wanna spin up some resource group and a virtual network in Azure. You can definitely do that using

 

Jonathan (29:38.431)

Mm-hmm.

 

Nicholas Hughes (29:41.042)

the same declarative YAML configuration syntax that you would with plopping a file down on a system and starting a service and things like that. So I sort of had that semi-formal role for a little while. And now, so SaltStack, the company, was acquired by VMware.

 

And VMware has rules when it comes to people outside the company being super hands-on in their repositories, I guess. So there's only so much that I can do, but I am sort of considered a contributor in the sense that I can't merge PRs, but I can do everything up to it. I can change labels on stuff. I can help people through things. I can improve PRs.

 

Jonathan (30:21.647)

Uh, okay.

 

Jonathan (30:34.143)

OK.

 

Jonathan (30:38.9)

Mm-hmm.

 

Nicholas Hughes (30:39.81)

So I'm tied into that quite a bit And I'm kind of a fixture out of all the open hours and things like that So, you know, I've just been around enough that I'm kind of like I Don't know this the salt mascot at this point

 

Jonathan (30:56.49)

All right. Very cool.

 

Will (30:56.675)

Yeah.

 

Will (31:01.45)

So we talked about how there's so many different things that salt can do and by extension of that, there's also a lot of other tools that can solve that problem. But based on your experience, what are your top go-to solutions where you see this problem, you're like, that's salt, all the way, all the way, every day.

 

Nicholas Hughes (31:25.464)

Um...

 

Nicholas Hughes (31:28.662)

So I think it really has to do with the scale of operations. So there's a certain point at which, once we get past that, I tend to throw out other solutions. If you want to do certain things at a certain level, I'm not going to have you. Even Ansible, right? Ansible's.

 

super popular and things like that and great, you know, but it is so incredibly slow at scale, right? Because it's using SSH as a transport, like, if I have to go and do a thing across 5,000 machines, I'm not going to pick Ansible. I'm just not. And so, you know, the core of how

 

Will (32:04.03)

What?

 

Nicholas Hughes (32:25.958)

it tries to put as little stress on the master as possible. And so essentially if I needed to go do a thing on 5,000 machines, like I want to see disk usage on the root volume for 5,000 machines all at once, right? I could certainly SSH into each one of those one by one. That would be super fun.

 

Will (32:52.599)

Right?

 

Jonathan (32:53.669)

I've done that.

 

Nicholas Hughes (32:53.674)

Or you could do something like PSSH, right? And you're generally going to be batching up SSH sessions and trying to probably dump that into a file or parse it coming off of the terminal. But with Salt, I could do that same thing. And the way it accomplishes it is not to reach out into those systems and run the commands and return it.

 

Essentially all it does is it's got the event bus, it publishes a job and it says like, hey, everybody, give me disk usage on slash. And they're all subscribed to that. And so through that pub sub relationship, the minions just say like, oh, I have a job to do. I'm gonna do this thing. And so you've distributed all of that work at that point, right? So then they've all finished. They're all running essentially at the same time.

 

And then they start shooting information back to the master. Like, hey, here's my info, here's my info. And all the master is doing is aggregating that information. And so all of a sudden, in the span of seconds, you have all of that information back. And I can output that as JSON and then do weird things to it with JQ or whatever I want to do. So it's really just the scale of operations that then like, OK, yeah, this is a salt thing.

 

The other thing is when you have like really unique event-based automation things that you want to accomplish, right? Orchestration is a great example of that. I mentioned earlier having a website and a database that are on separate systems. That type of orchestration isn't necessarily as easily implemented with other systems,

 

Nicholas Hughes (34:54.866)

It really depends. Some customers will be opinionated on their tooling. And so then you end up living inside of whatever it is that they want to be writing things in.

 

Will (35:09.294)

around.

 

Jonathan (35:11.458)

How does Salt work with quote the cloud? Like it sounds like it works really well if you have either physical servers or even just VMs, more static type VMs, but does it work well with container-based workflows and installations? And if so, how, or is it really a different, solving different problems?

 

Nicholas Hughes (35:32.822)

So we're kind of at an inflection point with cloud right now and salt. So.

 

Nicholas Hughes (35:46.222)

I think what I'll do is I'll talk a little bit about another future possibility and then we'll come back and we'll talk about the differences. So I already mentioned Heist is a pop application, right? There's this other one called Item. It's I-D-E-M, short for Item Potent. And so essentially what this started out as was the...

 

Jonathan (35:54.291)

Okay.

 

Nicholas Hughes (36:14.89)

the salt state system abstracted. So it can do stateful enforcement of stuff. And then I think it was basically like, we don't necessarily wanna write like, rewrite a salt competitor. But the one thing that is really dated inside of salt are the cloud APIs, right? Like...

 

They're just, they're not being kept up to date as quickly as they need to be. And there's reasons for that. So with item, it's a pop application. You can plug in different clouds, right? Like, so AWS was the first one that they really worked hard on, well, besides like VMware stuff, right? Like who was one of the...

 

Jonathan (37:09.742)

Sure, naturally.

 

Nicholas Hughes (37:11.918)

the main modules that they came out with. But you have VMware and you have AWS and Azure and stuff like that, right? And item can do stateful enforcement, similar to Terraform, but it's in YAML. It's a little bit cooler in the types of things that it can do. I think Terraform just actually recently implemented a feature that's been in item for a while. So it might be that.

 

that Terraform's starting to take notice of Item. But Item has this describe capability where you can essentially like, say I don't have any infrastructure as code whatsoever, I can describe my cloud environment and essentially spits out all of the YAML that does stateful enforcement, all of those things. And so now I've captured my entire environment in...

 

in YAML and I can throw that in and get repo and I can reach out and make sure that there's no drift intermittently and things like that. All the stuff that you really want IAC to do. So, item's a thing. There are integrations that are built into SALT to be able to leverage item. So, going forward, there's a good chance that what we're going to end up with is the

 

duo of salt and item to be able to do, you know, like second day operations things on your deployed systems. But item is your cloud management tool. That said, there are cloud things that are in salt currently. One of the reasons that they get out of date is that

 

Jonathan (38:56.835)

Okay.

 

Nicholas Hughes (39:05.826)

SALT's releases are somewhat infrequent when we talk about things in the context of cloud, right? Like cloud moves fast. There's constantly new services being released, new versions of APIs and things like that. And so when you only release a couple of times a year, that's not so great. I wrote

 

Jonathan (39:13.47)

Mm-hmm. Yep.

 

Nicholas Hughes (39:35.642)

All of the Microsoft Azure functionality back in the beginning of like 2018. Insult and so.

 

That did not make it into a stable release until February of 2019. So we're talking about a year, a little bit over a year, between, like, hey, I got this thing to, hey, it's there for people to use. And that's just a lifetime in cloud. So.

 

Jonathan (39:55.806)

Mm-hmm.

 

Jonathan (40:06.867)

Mm-hmm.

 

Nicholas Hughes (40:09.778)

A parallel initiative that we have going on right now to the development of item is the concept of salt extensions. And essentially what that is, is kind of adopting the item plugin mentality. So you have this salt extension where you take some of the code that relates to a certain thing and you put it out in its own repository, which can iterate as fast as you like. And then all you have to do is like pip install that.

 

thing into salt and you have all of the functionality. So in version 3007, which is the next major release, all Azure functionality will be completely gone from salt. And if you want to use Azure in salt, the salt extension is now the way to go because I own that extension and I can roll a release in five minutes, right? Like you have a bug.

 

Jonathan (40:52.69)

Mm-hmm.

 

Nicholas Hughes (41:08.946)

and needs to be fixed now, we can do that and we can roll out a release, we can get it up on PyPy and you're good to go. So that's a much more effective way to do cloud in Saltland. And so we're starting to see this mass exodus of functionality that requires fast moving capabilities into Salt extensions. You also mentioned containerized environments. There's been a lot of discussion in doing the same thing with

 

Jonathan (41:34.097)

Mm-hmm.

 

Nicholas Hughes (41:38.55)

Kubernetes. So there is currently Kubernetes functionality in the salt code base, but again, it's kind of dated. So it needs to be revamped and carved out into an extension so that we keep up with Kubernetes. And then we can manage Kubernetes things in salt.

 

Jonathan (42:00.11)

Cool.

 

Will (42:07.744)

And that's it.

 

Nicholas Hughes (42:07.762)

I should have said that I'll just keep talking indefinitely if somebody asks me questions.

 

Jonathan (42:10.64)

That's alright. I do have more questions, but I want to give someone else a chance, if they want.

 

Will (42:11.283)

Hahaha

 

Nicholas Hughes (42:15.31)

Yeah.

 

Will (42:16.454)

Yeah, I just wanted to comment that seems like a really reasonable approach to me, you know, because you have like your core salt functionality that you want to be stable and reliable. But then adding an extension ecosystem allows you to add new features and adopt to things that are outside of your control without having to...

 

you know, change the iteration cycle or the workflow for your core product. So I think there's a lot of logic behind that decision.

 

Jonathan (42:43.424)

I agree.

 

I like it too.

 

Nicholas Hughes (42:46.118)

Yeah, I mean, the core team is only so big, right? And Salt is a big sprawling code base. And what we've had happen over the years is that people will contribute modules to Salt that were important to them at the time. And then maybe they move on and they aren't in that role anymore. And that thing has become inconsequential. And so we have things that just sort of languish in the code base.

 

Will (42:49.111)

right.

 

Nicholas Hughes (43:16.298)

just because nobody really cares about them. We don't think anybody is using them, right? And then like in the future, sometimes somebody will go to use it and like, oh, well it's there and we'll, yeah, but this really is a community endeavor to keep all these things up to date. There's certainly core functionality that people are always gonna use, right? And we'll probably always be in the core of salt, like file management and service management, things like that.

 

Jonathan (43:28.663)

Hehehe

 

Nicholas Hughes (43:44.698)

And certainly all of the fundamental building blocks of Sol will always be in that core repository. But there's a lot of things that make sense to carve out as separate entities and then we can sort of see how dated those things really are when they're in their own repository.

 

Jonathan (44:10.43)

So I want to ask the opposite question from what Will asked earlier. He asked about typical use cases for salt. When are the, what are some examples that you might think of that are a bad fit for salt that you've seen? I mean, I'm sure we could think of ridiculous things, but like, have you seen salt used in really bad ways that it shouldn't have been used before?

 

Nicholas Hughes (44:31.36)

Um...

 

Nicholas Hughes (44:34.966)

That's a great question.

 

So.

 

Nicholas Hughes (44:42.294)

I think the times that have really stood out to me that were kind of in that vein, it's when people have business processes that have been adapted to other tool sets and then they're unwilling to adopt different...

 

processes when they transition to salt. So like say you're a puppet shop right and you know like you do operations the puppet way and you expect these things to be done in this mechanism. If in transition you try to use salt in the way that you've used puppet it's not necessarily

 

bad but it's certainly not the most efficient way to do salt things, right? Like salt wants to be used in the way that salt is best used. And when you do things apart from that in a process, you end up with not ideal situations, right? Like whether it be performance problems.

 

Jonathan (45:43.941)

Mm-hmm.

 

Nicholas Hughes (46:05.418)

or potentially workflow problems and things like that. Like there's definitely.

 

I've seen chicken and egg problems in Salt before, especially around grains. So grains are kind of like puppet facts, right? They're information about a system that are generally not changed that much, right? Like how many CPUs does the system have? How much memory does it have? What operating system is it running? So some of that is pretty static.

 

right, like system bios type and things like that. Okay, like how often is that gonna change? Probably gonna retire the system before you're gonna change the bios of the system. So, some of that stuff's not gonna change at all. And some of it's like operating system, maybe you upgrade from like Debian 10 to 11 or something like that.

 

Jonathan (46:41.41)

Mm-hmm.

 

Jonathan (46:47.885)

Hehehe

 

Jonathan (47:04.339)

Mm-hmm.

 

Nicholas Hughes (47:08.094)

That's not going to happen every day, right? So like grains are kind of meant for that. But it's very much like independently derived information. That information is what it is. Some folks try to use grains in this layered aggregate way, where it's like, OK, well, all of this information about a system.

 

means that it's this type of system. So let's make a grain that gives it like this role of whatever, right? And then, you know, oh, well systems with this and this and that role then have this other thing. So let's make that another grain. And right, like you have this grain sprawl and you know, that information as grains it's probably not the best way to do things generally, right? Like,

 

Jonathan (47:53.279)

Mm-hmm.

 

Nicholas Hughes (48:06.566)

And I sometimes have trouble convincing people of that. Just because it's not really meant to be used in that way. And once you start to use it in that way, things start to fall apart pretty quickly.

 

Jonathan (48:26.574)

How ubiquitous is salt? I just heard of it recently, a couple months ago. You've been using it for years. How frequently do you find people who've never heard of it versus those who've been using it, maybe casually? I don't know really how to answer the question in any sort of objective way, but I'm just curious how common is salt actually out there in the wild?

 

Nicholas Hughes (48:53.574)

It's surprisingly common because it keeps such a low profile.

 

Nicholas Hughes (49:01.974)

I haven't found that many people that just have never heard of it. Uh, I have, but it's, it's not that many. Generally, um, I hear from folks that, you know, it was like used in this one part of their organization or like on this one project that they were on and things like that, and like, there's these little pockets of it just everywhere. Um, and

 

Jonathan (49:28.5)

Mm-hmm.

 

Nicholas Hughes (49:31.046)

I don't know if it's like the equivalent of Johnny Appleseed. It's just one dude traveling around to like literally every organization and just like, I like salt. I'm going to put it here. But it feels that way sometimes just because you see all these like, oh yeah, this thing over here is doing it. This thing's over here is doing it. And I think the biggest wake up call that there was this.

 

Jonathan (49:37.198)

Ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha

 

Nicholas Hughes (49:59.262)

silent majority of people who are using it and we just didn't know about it, was a couple of years ago there was a fairly bad vulnerability that was found by some researchers. And

 

So there was this, you know, like notification process and things like that. You know, the Solcore team's trying to make sure that everybody knows that a patch is gonna come out at a certain time, like, hey, you know, it's gonna be disclosed at this time. Make sure that you patch your systems. But the notification channels for these people who are just like using it and not really engaging in the community, like how do we get...

 

word to them, right? And so what ended up happening was the vulnerability was disclosed. Some nefarious folks on the interweb started scanning for open salt masters, which is a terrible thing to do, by the way, like, you shouldn't, I don't care how secure you think that is the salt master is the keys to your castle, right? Like if you get into that,

 

Jonathan (50:49.376)

Mm-hmm.

 

Jonathan (51:09.351)

Mm-hmm. Hehehe.

 

Jonathan (51:14.978)

Yeah, right.

 

Nicholas Hughes (51:16.338)

It's literally touching all of your systems. Don't connect your salt master to the internet ever. But evidently a lot of people were, right? Like to manage offline systems like laptops and stuff like that without having to VPN into an organization. There's like, oh, well it's cryptographically secure, right? Like it's using PKI like, maybe don't do that. So anyway,

 

Will (51:40.654)

Yeah, right.

 

Nicholas Hughes (51:47.562)

They found like 2,500 unsecured systems on the internet. And so there was this mass influx of folks into Slack and other places saying like,

 

Jonathan (51:54.091)

Wow, okay.

 

Nicholas Hughes (52:09.798)

my salt systems are acting weird. Does anybody know what's going on? I think that there's like some sort of virus and everybody's like, yeah, there's just like the CV is disclosed and there was like this big master thread that I think somebody memorialized up on Git somewhere, but it was basically like all of the discovery that went into figuring out how people were infecting the systems, right? And then

 

some thoughts on like, okay, you know, between these time periods, it was this and then you can remove it and everything's good. If it's a little bit later, they had more time to like do really bad things and you're probably looking at like reinstalling all of your systems. Like it's a really good read and it's kind of scary. But

 

You know, it was interesting to see all of the different people that came out of the woodwork that were using salt that we had never heard from before. And it was all because of this CVE and you know you find things like, okay, well, you know, like certain governments are using salt, right, like we didn't know that, you know.

 

Jonathan (53:19.33)

Hehehe

 

Jonathan (53:28.137)

Mm-hmm.

 

Nicholas Hughes (53:28.902)

Um, you know, certain large organizations, like multinational corporations were using salt. That was never like publicized. You know, some folks have publicized it, right? Like LinkedIn has been a user of salt since like almost the very beginning. Um, you know, there have been some other big ones, like I think Liberty Mutual had like a really great use case on network automation that they were publicizing for a while.

 

Nicholas Hughes (53:59.134)

thousands and thousands of switches and routers and stuff with salt but then there was all these other ones that like you know never heard that they were using it until you know something went terribly wrong so yeah I think that it's kind of like it's definitely a sleeper tool right like it's got a lot of fans out there it's got a lot of really great use cases

 

Jonathan (54:20.024)

Hehehehe

 

Nicholas Hughes (54:26.73)

but it definitely flies under the radar.

 

Jonathan (54:36.799)

I don't know if you're actually really connected or not. You've been quiet, okay.

 

Jillian (54:39.904)

I am. I'm back. It went... I don't know. Now I'm having technical difficulties with the Wi-Fi, but for now I'm here.

 

Will (54:41.215)

Hahaha!

 

Jonathan (54:47.534)

All right, I was curious if you wanted to talk about your experience with Salt. It looked like before we started recording, we might have a sort of arm wrestling match going on here. I wanna see if we can start some sparks here. Do you wanna tell us your thoughts on Salt?

 

Jillian (55:04.189)

Oh, I'm always happy to start drama on the podcast. So, you know, sure. Why not? Uh, so when, so I was using salt a while ago when I was still working on HPC centers that were, um, you know, like in-house, so not on the cloud. And salt was really great for that because it had the event driven architecture. And for HPC, you know, like you, you start a job, node comes up, uh, job finishes. Node goes back down. That's, you know, that's generally how it's supposed to work.

 

Jonathan (55:06.318)

hahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahah

 

Nicholas Hughes (55:06.954)

Yeah.

 

Jillian (55:32.457)

And it was really great for being able to configure like all these different workflows for all the different use cases. We would have a different, um, like depending on how the jobs were submitted, they would mount different storage or they would have different services start on them. Uh, being able to configure like MATLAB servers. I remember that was probably the biggest one because MATLAB is like, you know, just, it's really tricky to configure because of all the licenses and whatnot, especially if you want to have it on demand.

 

because you have like a shared environment. Like typically an HPC system is, you have like one login node and then you have, however many compute nodes, I think this was like around 200, if I remember correctly. And you have lots of different users with lots of different needs and demands and they all wanna have different things. So being able to have that kind of event driven architecture, we were saying, okay, when this node comes up, if it is deployed with these parameters,

 

um, startup Matlab or startup, you know, the server for the microscope or the sequencer, just, you know, like whatever. It was really great for that. And then I hadn't used it. Um, you know, since I got more into cloud stuff and then I tried to use it again. And I think I was trying to use it with the AWS parallel cluster where AWS manages the events and like, I know like there's events in there.

 

somewhere with the AWS parallel cluster, but I don't like really know where they are. And so I think I was having a hard time getting salt to kind of like match up with what I expected. And it also seemed to be more complicated than I remember, but that might also just be like, I have fewer brain cells than I used to too. So it's a bit of like, where, where does the problem lie here? Is the problem, is the technology the problem or am I the problem? Because at any given time it's like 50 50.

 

Jonathan (57:18.716)

Hehehehe

 

Will (57:22.253)

Thanks for watching!

 

Nicholas Hughes (57:29.938)

Yeah, you know, troubleshooting those types of scenarios are super fun, right? You know, especially when you start to involve the event bus, because, you know, you can see things going on in the event bus, right? Depending on how busy your event bus is, it could be flying by at breakneck speed, though, right? Like if you're trying to pick out a particular event, it's tough.

 

Jillian (57:30.473)

cough

 

Jillian (57:52.265)

to pick out a particular.

 

It's tough.

 

Nicholas Hughes (57:57.334)

Sometimes right like there's a lot of okay And I'm gonna try and like dump this to a file and then like grep stuff out or you know Maybe look at it at the file with an editor after the fact But if you don't have a noisy one, yeah, sometimes you can pick stuff out and event bus that's going on the complexity is It's tough right because you know if you're doing things across systems, right like

 

Jillian (58:18.577)

The complexity is tough, right? Because if you're doing things across...

 

Nicholas Hughes (58:27.358)

Logging occurs on your master and your minion. And so you have the event bus, which can be globally viewable from either side. But when actual code is being run by a minion, that's not necessarily viewable by the master. So you have to bounce back and forth sometimes to figure out exactly.

 

Jillian (58:48.393)

necessarily.

 

Jillian (58:53.181)

sometimes to figure out exactly.

 

Nicholas Hughes (58:56.478)

what's happening. And the best early stage example I can see of that is, there's a salt command on the master that you can use to target a minion and run stuff on it. Like I was talking about running a disk usage command on thousands of machines before.

 

Jillian (59:01.733)

best early stage example I can see.

 

assault command on the master that you can use to target a minion.

 

Nicholas Hughes (59:21.054)

that would be via that, right? Like I'm targeting all systems, I'm targeting one system, it doesn't matter, right? It's just a target. And so when I do that, that command, all it's doing is submitting the job to the event bus and then waiting for all systems to return. That's all it does. So it'll sit there and it'll block the whole time. And if people try and run that command in debug or look at the...

 

Jillian (59:23.878)

all systems.

 

Jillian (59:28.701)

So when I do that, that command, all it's doing is submitting the job to the event bus.

 

Jillian (59:42.633)

So it'll sit there and it'll block the whole time. And if people try and run that command in debug or look.

 

Nicholas Hughes (59:52.255)

the log files on the master, all you're going to find is stuff related to that process, which was job submission and waiting for returns. The actual execution of the commands, the stuff that's being done, all of that's being logged over on the minions. So the distributed nature of how things work in SALT sometimes make it problematic to troubleshoot in that regard.

 

Jillian (59:56.813)

is stuff related to that process, which was job submission, the way to return. The actual execution of the commands, the stuff that's being done, all of that's being logged.

 

So the distributed nature of how things.

 

Nicholas Hughes (01:00:18.666)

The other thing that I'll say is that there are so many ways to configure it. It can definitely get complex that way as well. So that's job queue.

 

Jillian (01:00:18.985)

The other thing that I'll say is that...

 

Jillian (01:00:23.411)

There are so many ways to configure it.

 

Jillian (01:00:28.593)

definitely get complex that way as well. So that's job Q.

 

Nicholas Hughes (01:00:38.523)

it natively lives in the default.

 

Jillian (01:00:38.762)

it natively lives in the defaults.

 

Nicholas Hughes (01:00:44.102)

Event bus implementation that salt has and that's zero MQ for Pub Sub. But if you want to keep the job queue someplace else, you can do that. So like, say you have two masters and you're running them like more or less independently, but you want them to share a job queue. Um, apologize for the, the shake there. I have three dogs in my room. Um,

 

Jillian (01:00:57.661)

So you have two masts.

 

Jonathan (01:01:08.438)

Hehehe

 

Will (01:01:09.412)

I'm sorry.

 

Nicholas Hughes (01:01:13.85)

So, you know, you, I forgot, they completely threw me off. What was I talking about here? These little fur mongers in my room.

 

Jillian (01:01:14.173)

So.

 

Jillian (01:01:20.533)

completely threw me off. What was I talking about? These little fur mongers in my room. I'm like that too and my kids are around. My mom's side of my brain and my work's side of my brain, they can't coexist. They have to remain separate at all times.

 

Will (01:01:22.148)

Hahaha!

 

Jonathan (01:01:23.159)

Dogs will do that.

 

Nicholas Hughes (01:01:34.311)

Yeah

 

Yeah.

 

Will (01:01:38.176)

Now you were talking about multiple masters sharing the same pub sub queue.

 

Nicholas Hughes (01:01:42.298)

Yeah, so if you wanted to share the same queue so that minions connected to either master could see those jobs, you can hook Redis up to your masters and outsource the job queue there, right? Like, say you want to share configuration artifacts between systems, you can share through a configured

 

Jillian (01:01:59.965)

say you want to.

 

Nicholas Hughes (01:02:12.19)

back-end file system. So normally a master will share files with a minion off of its local file system. You could set up like GitFS or something like that so that your files, artifacts, whatever, are all in some Git repo that is gonna pull down and serve out to the minions. And so then you start to centralize some of the components outside of Salt.

 

Jillian (01:02:37.641)

start to like centralize.

 

side of salt but it still gives you all the glues.

 

Nicholas Hughes (01:02:42.67)

Um, but it still gives you all of the glue to do that. Um, so yeah, I think I'm just reaffirming that yes, it is complex. What can be complex. Um, and it can sometimes be difficult to troubleshoot. It really requires you to know how things fit together. Um, and so I don't even know if they're, they're still.

 

Jillian (01:02:48.425)

So, yeah, I think I'm just reaffirming that, yes, it is complex. What can be?

 

Nicholas Hughes (01:03:12.682)

giving out this certification, but there was like a certified salt stack engineer certification and like literally a large portion of that exam was just where does this run? Where does this render? Where does this? Yeah, it's like, okay, you got to know where things happen because of the distributed nature of it.

 

Jillian (01:03:14.185)

but there was like a certified SaltStack engineer certification and like literally a large portion of that exam was just where does this run? Where does this render? Where does this... It's like, okay, you gotta know where things happen because of the distributed nature of it. Yeah, I think that's definitely true of all distributed systems. The first thing that I figure out is like where does it log to? How do I just get a shell?

 

Jonathan (01:03:25.96)

Hehehe.

 

Jillian (01:03:43.165)

Um, you know, how do I execute a command remote, like locally and remotely? How do I check those things? So that's, that's definitely true.

 

Jonathan (01:03:55.314)

If anybody's still listening who hasn't decided they want to try using salt, um, what, what would be your, uh, your, your like elevator sales pitch? Like if this is your problem, use salt for this, or, or you should try salt because of X.

 

Jillian (01:04:01.134)

Never.

 

Nicholas Hughes (01:04:11.41)

Um... I don't know, like if I had to come up with a pitch for it, it would probably just be me singing Vanilla Ice, right? Like, if you got a problem, yo, Saldo, solve it.

 

Will (01:04:21.578)

Yes.

 

Jonathan (01:04:21.742)

HAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHA

 

Nicholas Hughes (01:04:27.302)

Um, yeah, I mean, I don't think I can do much better than that. Quite frankly, like it, it's, it, it's worth looking into, right? Like, you know, like I said, we don't use salt on every project that we have. Um, but it's kind of like a decision point that's really high up in the process when we're trying to work out tooling, right? Like, is this a job for assault? Like a lot of times, yes, but.

 

Jillian (01:04:28.485)

Yeah, I mean, I don't think I can do much better than that, quite frankly. Like it's worth looking into, right? Like, you know, like I said, we don't use salt on every project that we have. But it's kind of like a decision point.

 

Will (01:04:32.538)

That's gonna be hard to top.

 

Jillian (01:04:47.441)

that's really high up in the process when we're trying to work out tooling. Right? Like, is this a job for salt? Like, a lot of times, yes, but, you know, it's worth being considered for the job and then, you know, if it doesn't make sense, doesn't make sense.

 

Nicholas Hughes (01:04:56.57)

You know, it's worth being considered for the job. And then, you know, if it doesn't make sense, it doesn't make sense.

 

Jonathan (01:05:08.366)

I'm going to be checking out those resources you suggested earlier. I'm going to try to learn some salt and see if I can make sense of what we're doing with this client and make it make sense.

 

Nicholas Hughes (01:05:18.878)

Yeah, absolutely. If you make your way onto the Slack workspace, you can hit me up. I'm Nicholas M. Hughes. Not hard to find.

 

Jillian (01:05:21.349)

make your way onto the...

 

slack workspace. You can hit me up. I'm Nicholas M Hughes.

 

Jonathan (01:05:31.042)

Cool. That's awesome. Anything else we should talk about? Anything we should have asked that we didn't before we move on to closing out?

 

Jillian (01:05:34.993)

anything else we should talk about, anything we should have asked but we didn't before it moves on.

 

Nicholas Hughes (01:05:41.485)

Um...

 

Nicholas Hughes (01:05:44.762)

No, I don't think there's anything that you asked that you didn't ask that we should have covered. I think the only thing that I'll sort of like throw out there, right, is that I kind of hinted at that is really cool community. It's very accepting of new folks. You know, I haven't seen too many people be like super smarty. I can maybe think of one. But, you know, you're always going to have

 

Jonathan (01:06:10.73)

Hehehehehe

 

Nicholas Hughes (01:06:13.778)

somebody like that hanging around just looking to stir up trouble. Yeah, it's Jonathan. But, you know, it's a really great community and it's got great folks in it. And if there's anybody out there that is just starting out in IT, just starting out in software development and just starting out in systems administration or DevOps or anything,

 

Jonathan (01:06:14.882)

There's always that line. Yeah, it's usually me.

 

Nicholas Hughes (01:06:42.81)

it's a really great place to learn, right? Like eight years ago, when I first got introduced to Salt, I wrote zero Python code. And a lot of the way that I learned more about writing Python was submitting bug fixes and like working on issues in Salt. So there's definitely a lot of work out there to be done. It's also...

 

Advice to I give I give to folks that are in school You know like you're gonna learn how to code in school a little bit, but you know in general Find an open source project. There's tons of them find one that speaks to you and Go get involved in it right like it's going to up your development game so much more than Anything that you could do at school?

 

Jonathan (01:07:33.483)

Mm-hmm.

 

Jonathan (01:07:41.254)

Awesome. Good advice.

 

Nicholas Hughes (01:07:45.534)

That was my pre-T section of the...

 

Jonathan (01:07:47.602)

Yeah, that's good. If people would like to hear another sermon, is there any way that they can reach out to you on social media and get in touch?

 

Nicholas Hughes (01:07:52.926)

Hehehe

 

Will (01:07:53.729)

Hahaha!

 

Will (01:07:58.062)

Grab your credit card and call this number. Ha ha ha.

 

Nicholas Hughes (01:07:58.362)

Yeah. Yeah, so I'm pretty active on LinkedIn. So Nicholas Hughes on LinkedIn, CEO of EITR Technologies. I'm Nicholas M. Hughes on pretty much everything, right? So like if you got the LinkedIn URL, it's just Nicholas M. Hughes at the end of it.

 

Jillian (01:08:05.639)

Yeah, so I'm pretty active on LinkedIn. So Nicholas Hughes on LinkedIn, CEO of EIGI.

 

Jillian (01:08:16.588)

and views of

 

Nicholas Hughes (01:08:27.486)

Yeah, I think that's probably the best way to stand tough with me. As far as sermons and such, EITR Technologies does have a YouTube channel. And so if you just like listening to me talk about weird random crap, and you also like kind of funny skits for people that are being ridiculous, then go ahead and subscribe to that YouTube channel and check us out.

 

Jonathan (01:08:49.58)

Hehehe

 

Jonathan (01:08:57.774)

Awesome. Well, let's move on to pics then. Who would like to go first today?

 

Nicholas Hughes (01:09:03.818)

Not it.

 

Jonathan (01:09:05.002)

Alright.

 

Will (01:09:07.118)

All right, I'll take it. I'll take a swing. So my pick today, for those who don't know, I've been training to run a hundred mile marathon or ultra marathon at the end of October. And so I've been running quite a bit to get ready for that. And so there's this thing that I didn't know was real or salt stick tablets or salt tablets for replacing electrolytes that you lose while sweating.

 

I've been taking them for a couple of weeks now and had zero issues with cramping, with like, soreness.

 

and like water retention because I live out in the desert where today it's going to be 113 degrees. Still got to get my run in and so yeah, salt stick tablets. They're actually pretty, they don't taste like salt either, which is a huge plus since we're talking about salt here. They taste like, you remember the vitamin C tablets that you ate as a kid?

 

Jonathan (01:10:06.746)

Mm-hmm.

 

Jonathan (01:10:11.422)

The like fake orange flavored ones? Okay.

 

Will (01:10:12.658)

Yeah, yeah, they taste exactly like that. But they work really well for muscle cramping issues and also water retention so that as you drink the water, your muscles and your body actually retains it instead of just sweating it right out. So there you go, on the topic of salt.

 

Jonathan (01:10:30.798)

So salt really can solve any problem.

 

Will (01:10:34.078)

Right? It is just the universal tool.

 

Nicholas Hughes (01:10:34.762)

See?

 

Jillian (01:10:35.329)

See? It's just the universe of two worlds. I like how thematic that pic was. That was good.

 

Nicholas Hughes (01:10:41.81)

Yeah.

 

Will (01:10:43.158)

I wish I could say I thought that out, but I was halfway through giving that spiel and I was like, oh wait, this totally ties in.

 

Jonathan (01:10:51.158)

Now the rest is gonna look bad. Yeah.

 

Jillian (01:10:51.344)

Missed opportunity for so many puns!

 

Will (01:10:54.143)

No.

 

Jonathan (01:10:58.562)

So I guess I have a pick but it's not salt related. I can't figure out how to tie it in. Sorry.

 

Will (01:11:04.622)

Yeah

 

Nicholas Hughes (01:11:06.953)

We'll ruin it for everybody.

 

Jillian (01:11:07.209)

We'll ruin it for everybody. I guess I could pick the first season of Star Trek. The original.

 

Jonathan (01:11:08.734)

I guess I could pick the first season of Star Trek, the original series, episode one, or like the second or third episode is where they go to the planet where the alien creature sucks all the salt out of the bodies of the victims. So there's that one. If she just had these tablets you're talking about, well, she wouldn't have needed to do that. Could have saved some redshirt lives. So my pick today is a book I'm listening to, an audio book. I usually use physical or eBooks for...

 

Will (01:11:08.886)

Thanks.

 

Will (01:11:20.128)

Hahahaha

 

Jillian (01:11:23.153)

You just had to establish your tongue.

 

Will (01:11:25.928)

It would at least drug the process out.

 

Nicholas Hughes (01:11:28.274)

Hehehehe

 

Jillian (01:11:30.909)

My pick today is a book I'm listening to, an audiobook.

 

Jonathan (01:11:37.374)

software-related books, but this one I decided to try on audiobook and it's working well. It's Modern Software Engineering by Dave Farley. So he wrote Continuous Delivery, I don't know, close to 10 years ago with Jez Humble, and he has this new one out that came out I think end of last year. And it's really good. It's just about, you know, how to, the practice is more than the code of writing modern software and how to test it and the mentality you need around.

 

Jillian (01:11:51.165)

Yes, it's me when I.

 

Jonathan (01:12:04.298)

making modular code and cohesion and all this sort of stuff. So it's a good book. I was kind of hoping that Dave had narrated it himself since he has a YouTube channel, he has a good voice. It's narrated by a woman, no ill feelings towards her but I was wishing for Dave's voice. So anyway, it's still a great book. I recommend Modern Software Engineering by Dave Farley whether you listen to it or read it with your eyes.

 

Jillian (01:12:23.017)

Still a great book.

 

Jillian (01:12:34.169)

Uh, so it's summer where I'm at, so I'm basically reading, you know, trashy romance novels and space operas. And that's really it. I really appreciate the fact that you guys are still reading, you know, actual books. So I'm just going to pick Lindsay Broker. She's an author. She's on, you know, like Amazon. You can go to Amazon and get any of her books. She writes a lot of sci-fi and fantasy, and I've been binge reading pretty much everything by her, and they're all really good summer reads. So I'm out on my deck.

 

Will (01:12:39.736)

Yeah.

 

Jillian (01:13:01.989)

with my books and that's it. That's the pic.

 

Will (01:13:04.703)

Nice.

 

Jonathan (01:13:05.986)

Awesome.

 

Nicholas Hughes (01:13:07.03)

Cool, I guess that brings it to me. So after Will went, I was scrambling to find something that was salt related. And the best I could come up with was before I started EITR, I told you that I had been involved in the community. I went to SaltConf in 2017.

 

Jonathan (01:13:14.734)

Hehehehehehehe

 

Nicholas Hughes (01:13:32.606)

So Saltstack used to have a conference around salty things. And when I went there, you could sign up for an opportunity to meet with senior leadership of Saltstack, the company. And I was looking at the list of people and I'm like, I don't wanna talk to any of these guys, except for Thomas Hatch, who was the CTO and co-founder of Saltstack and the guy who started the whole thing.

 

And so I signed up for an opportunity to sit with him. And most people had an agenda that they wanted to go over. They're like, oh, Tom, check out my thing. Or like, hey, our companies need to work together, or whatever. I was just a dude who worked at a consultancy. And I was just like, it would be fun to talk to him. And I guess one of his assistants reached out to me prior. They're like, OK, so what are you going to be talking to Tom about? And I was like, I don't know. I'm just going to wing it. And

 

Jonathan (01:14:19.37)

Hehehe

 

Will (01:14:29.887)

Yeah.

 

Jonathan (01:14:30.117)

I'm gonna nerd out.

 

Nicholas Hughes (01:14:31.89)

And so he comes into the room during Salkov, during our session, and he sits down and he introduces himself. And he's like, so on my little sheet, it said that we're winging it. Am I winging it? Are you winging it? Are we winging it together? Like, what are we doing? And so that was my first face to face interaction with Tom. And one of the things that I asked him was really like, how do you get...

 

Jonathan (01:14:50.104)

Hehehe

 

Nicholas Hughes (01:14:59.242)

from where I am as a technical implementer to a leadership type role in a company. And he recommended a book to me that I now recommend to everybody else who is interested about that type of path. And that is called The Manager's Path by Camille Fournier. So that is my pick.

 

It's somewhat salt related because Thomas Hatch recommended it to me and now I'm recommending it to everybody else.

 

Will (01:15:30.293)

Nice.

 

Jonathan (01:15:30.422)

Very nice. Cool.

 

Well, thanks again, Nicholas, for coming on and helping to educate me and our entire audience about the amazing benefits of salt. I'm looking forward to learning more about it. So yeah, just a big shout out and thanks for taking the time.

 

Jillian (01:15:35.261)

Thanks again, Nicholas, for coming on and helping to educate me and our entire audience about

 

Nicholas Hughes (01:15:49.49)

Yeah, absolutely. Thanks for having me. It was blessed.

 

Jillian (01:15:50.441)

Thanks for having me, it was a blast. Great. We will see you all next week on Adventures in DevOps. Until then, have a good week. Bye.

 

Jonathan (01:15:51.906)

Great. We will see you all next week on Adventures in DevOps. Until then, have a good week.

 

Will (01:16:00.672)

and we're