In this episode, the Rogues talk about DevOps with Nathen Harvey of Chef.

Special Guest: Nathen Harvey.

 

It sounds like it might be a tricky DevOps kind of problem.

Uh-oh!

That’s awesome.

If only we had someone to advise us on how we could do that...!

Step one: acquire a tricky DevOp.

Yes. Step two: stop using that word wrong.

Hey everybody, and welcome to Episode 113 of the Ruby Rogues podcast. This week on our panel, we have James Edward Gray.

Good morning, everyone.

Josh Susser.

Hi. I'm in solidarity with San Franciscans by working from home today.

David Brady.

I'm apparently in solidarity with San Franciscans today. I didn’t know that.

[Chuckles] Me too! I'm Charles Max Wood from DevChat.TV and I have a PhD in horribleness.

But guess what I watched last night?

But do you have the goggles and gauntlets?

I need to order those off of Amazon.

Yeah!

[Chuckles] Get on it, man.

We also have a special guest and that’s Nathen Harvey.

Hey, how’s everyone doing today?

Terrific.

How are you, Nathen?

I'm doing great. I'm calling in, also working from home today. So, I'm one those also sitting in solidarity with those folks in San Francisco, although I'm doing it from Annapolis, Maryland.

Oh!

Wow!

So, can you introduce your self for the folks who don’t know who you are or don’t listen to your podcast?

Sure. As Chuck mentioned, my name is Nathen Harvey. I'm the co-host on a podcast called the Food Fight Show. The Food Fight Show is the podcast where DevOps chefs do battle. Fightfully, we talk a lot about Chef, we talk a lot about DevOps topics. But unfortunately, we don’t actually have a lot of babbling that happens on the shows. I'm also a Community Manager at Opscode which is the company behind the open-source framework Chef which is not about cooking although we speak in recipes and cookbooks all the time. And then, I'm also a co-organizer of a couple of meet-up groups including DevOpsDC which is in Washington, DC.

Awesome.

So, Josh, now is the part where you ask for a definition.

Oh, right. So, what is a DevOp?

Yeah. That’s a great question.

I recently gave a lightning talk at RubyNation and then followed that up with an ignite talk at DevOps Days Silicon Valley. My first slide, I asked for some audience participation. I asked folks in the room to stand up if any of the statements I'm about to make rang true with them. So, those statements were like ‘I leaping hate DevOps’, ‘Which DevOps tools should we use if there's someone at your company who’s a DevOps Engineer or a DevOps Lead or if your company has a DevOps team’. When I said that, invariably, a significant portion of the room stood up; at which point, I had to inform them that they keep using that word but I don’t think it means what they think it means. So, that is not what DevOps is. But I guess you want to know what DevOps actually is.

Yeah.

Yes, please.

Yes.

Alright. So, DevOps is a cultural and professional movement. It’s all about development and operations working together toward a common goal. And in fact, it goes beyond development and operations. It’s really about the entire business working together to deliver value to your customers. And the DevOps movement is really looking at leveraging ideas and processes from other industries, things like lean manufacturing. And we use DevOps practices and policies and this idea of DevOps to enable things like continuous delivery to enable agile development across the board, not just within the development team but also within your operations team and things like that.

That sounds really interesting. I use the word ‘DevOps’ and I'm being actually serious about this, this is not a joke. I use the word ‘DevOps’ to say the things that I do when I'm typing in the terminal when I'm not writing application code.

That’s interesting because I use the word wrong as well and it’s anytime a developer is doing IT work. The key DevOp move is, “Ah! I can't write to the Apache directory cd /; chmod -R 777 *. You know, just make the entire hard drive writable and that’s DevOps. [Chuckles]

You know, the thing that really made me both cry and laugh and shake my head and put my palm to my face was I saw a tweet awhile back about the Rails Devops Cheatsheet. And the Rails Devops Cheatsheet is, in fact, a bunch of Unix commands or Linux commands that do essentially that. They tell you how to do things like list all of the files in a directory, how to kill a process. These things are important. They're definitely things that we should all know how to do but system administration or infrastructure engineering is not DevOps.

I was going to say I worked as an Operations Analyst and then as a Server Administrator between the two for like six years when I was in college, then moved into tech support and then programming. But it was kind of an interesting thing for me to see it come up because a lot of the DevOps tools are really just, in some ways, a better or a different approach to the way that I did my job before the movement started. And so, it’s kind of fun to see it come up that way. I also want to comment briefly on Dave’s cheatsheet for Rails developers for Devops because I've seen a few of those. And about half of them are, “Here's how to circumvent the security that was set up around your app, around your server.” [Chuckles] And then, that just makes me laugh my head off and then cry. But it’s really interesting to find that intersection between the two and to allow the two to collaborate. And I know that Dave, in particular, can sympathize with developers that find themselves binging their heads against the IT wall trying to get stuff done on their servers versus more of the -- I've read a few books on XP and Agile where basically, you'll have members on your team who are responsible for providing you with what it is you need, whether that’s direction or resources. And to have an IT resource on your team whether he’s using kind of a DevOps approach or whether he’s actually doing classical server administration for you, it just makes a huge difference to have that collaboration and some of the onboard that’s invested in your team because he’s a part of it.

Chuck has been on a team with me when the proper definition of DevOps, meaning the interaction between dev and the operations people, could best be described as a cold war.

Oh yeah, absolutely. I mean, think it definitely spawns from just having misaligned objectives. Operations, their objectives typically are to keep the site up and keep it running. Whereas development’s objectives are to build new cool features that customers want. And the best way for an operator to keep a site up and running is to make no changes to it.

Immutability!

Absolutely! On the operations side, it’s within -- we’re kind of incented to keep changes from happening. So, the whole idea of just saying ‘no’ to development, that’s the right thing to do. And as on the development side, you bring in things like agile practices and you start iterating much, much more quickly, that just means you're throwing more and more stuff over the wall that operations wants to put up lockers against and say, “No, we can't keep pace with the velocity that you guys on the development side are generating. It’s impossible for us to keep up.”

When I worked at a big company, managing a team there, we had a pretty fast turnaround time on development. And then, there would be a three-week deployment process.

Yeah. Literally, you would push the button in the web console for deploying software and that would file a ticket in somebody’s queue and three weeks later, your app will get updated.

And it’s cool when that happens because three weeks later, when there's a bug in production, you know exactly what that bug is because you were just in that code, right?

Yeah. But then, it takes three weeks to fix it.

Yeah.

Yeah, three weeks to deploy the bug fix. I worked in another company that had a dedicated ops team. And that company, the CEO was an Engineer and he had started a previous software startup. And so, ops worked there a little bit differently. They weren’t part of the development team but at the same time, their job description was to keep the site secure and to keep all the data secure. And then, their job was also to basically make things as seamless as possible for the engineers so that they could push the new code up. And so, whenever anyone wanted to deploy, you kind of got it checked off by the lead developer and then it was good to go. Later, we had QA practices and stuff on to of it, but even then, it didn’t take more than a week to get things out.

My impression is that ops people tend to look at developers the way a proprietor of a china shop looks at a bull. [Chuckles]

Or a small [inaudible]. [Laughter]

Right.

Is that the case, Nathen? Or are developers just like the bane of the existence of a running deployment? “No, don’t log in. Don’t type anything, please!”

I think that the real change comes when you do start working together and try to have some empathy for one another and understand what's going on. I think that traditionally, yes, operations absolutely look at development as the bull in that china shop. But I think that we start to see real change and real improvement when we start working together. Myself, as an operations person, I spent most of my -- my first job in operations was sitting in the same room with the development team building out the product with them over time. And that is the right way to do things. And that definitely is the way to ensure that you can get to things like continuous delivery or something close to that, and really understand the problems that each other are facing.

Most of the time, when I run into issues with the operations or IT staff, it’s because they're not as involved. And so, what happens is when I come to them and say, “I need this library on the server,” it’s a surprise to them. They don’t even see it coming. And so, again, having somebody on the team that knows what's going on, knows that this dependency is coming up, by the time that has to be deployed, they should be on top of whether or not it’s a good idea to put it on the machine.

I have a topic here. And that’s system architecture versus application architecture. That’s the thing that I think the interaction between the app development and DevOps can be incredibly valuable. I've seen that be really awesome on a project before. Nathen, is that the bread and butter of DevOps or is that something special that happens?

Maybe I’ll ask another question in response to it. I think there's oftentimes a desire on the development side not to care about what the system architecture actually looks like. I don’t care what my deployment environment looks like. And I think that that definitely is the path towards destruction. When you're developing an application, if you're building it to run on something like Heroku, you're going to build it slightly differently than you would, say if it’s running on Amazon’s EC2 or if it’s running within your own data center. And understanding some of the tradeoffs between those various architectural points is definitely an important thing that you have to bring in to your application design and into your application architecture.

I listened to a few of the Food Fight episodes in preparation for this episode. And my favorite was I listened to the one where you guys interviewed the Netflix team over their open source stuff. It was a really great episode. I didn’t realize like, I think I remember when they put their open source stuff out up there and I’d lurked. And there was like one or two things. And now, if you go look, it’s like a billion things. [Chuckles] Ruby is like all of their different infrastructure and that was really interesting to see all their different pieces and hearing you guys talk about how they're used. But there were some great examples of what you were just talking about and they're of like how one team makes these, the monkeys, the Chaos Monkey being the most famous one that runs around killing things. But they actually said the one that’s more helpful is the Latency Monkey that goes around and just inserts random latency between the systems. And they said, “You wouldn’t believe how many problems that flushes,” when you insert latency because what they find is that the developers have the timeout set way too high. They have a timeout of something like three seconds where it ought to be like a hundred milliseconds or something. And so, once the Latency Monkey throws some random latency between these two systems, everything falls apart because that one system with a high timeout is still waiting. And I just thought it was an awesome example of ops guys helping developers. They were like, “We built this tool that will show you when you're making mistakes that could really hurt us.”

Yeah. And that was certainly a really fun episode. The stuff that Netflix does is pretty incredible. They also probably have the prettiest open source center homepage on GitHub.

Yeah, it’s awesome. You should go look at it if you haven't. We’ll put a link in the show notes.

The first time I looked at that, I wondered if there was some weird browser redirect that happened that I didn’t notice. [Chuckles]

Right. You’ll think, “Wait, did I land on the wrong site?” [Chuckles]

I said, “This isn't GitHub. What's going on?”

I want to talk a little bit about Ruby and DevOps. First, I think I really want to talk about some of the tools that are out there. So, there's Chef and Puppet, both of which are written in Ruby. There's a whole host of different deployment gems you can pick up to deploy your application. Why is it that these things tend to proliferate in Ruby as opposed to some of the other languages out there? Is there something about the Ruby language that makes it easy to build these or was it just because people like them?

I think what you'll find, and certainly coming from my Chef background, I use both Puppet and Chef professionally and obviously, now I work for Opscode, the company behind Chef. But I think that the real thing about Ruby and DevOps tools, it’s really no different than some of the reasons you came to Ruby. There's a system administrator being able to automate things, number one, just makes me really happy. Being able to automate those things and use Ruby in order to do so, Ruby is all about developer happiness and I think that getting into Ruby in these automation tools like Chef, I just feel really, really comfortable in that space, I feel really happy, I can be very productive using Ruby. And I think that that’s really a big draw for Ruby and why Ruby is used so often when it comes to automating your infrastructure.

I think one of the challenges, from my experience, one of the challenges of doing that kind of work is having to work at a lot of different levels at the same time. There's an incredibly low level bitwidth/length kind of things that you have to do with directory permissions or things like that. But then, there's also very high level things that you have to do that are like, “Oh, I have to keep track of a thousand servers and make sure that they're all organized in the right way and monitored the right way.” So, there's like very low level and very high level things. And I guess Ruby is pretty good at spanning those levels of abstraction pretty seamlessly. So, that’s my take on why Ruby gets used like that a lot.

Yeah. That’s definitely the case, that number one, you do have to think about things at so many different levels. And Ruby absolutely is a great way to abstract a lot of that away especially with meta-programming and things like DSLs, Domain Specific Languages. I can make a system administrator very comfortable managing not only a single server environment, but again, managing that entire infrastructure and managing that infrastructure as a cohesive unit not as a collection of a thousand different things.

I'm going to be slightly pessimistic here and say that perhaps these tools rose out of Ruby because in the beginning, deploying Rails was really complicated and kind of a pain in the butt. And we threw a lot of energy at that to try to make it better, things like Capistrano and stuff like that and maybe that’s what got this ball rolling and pushed us to build these better and better tools until it gets in there.

I will follow that by saying Chef and Puppet came into existence because James has stopped maintaining Capistrano.

I could totally argue the history there.

Sure.

Because I don’t think that -- I think that that’s an interesting take on it. I think that certainly, tools like Chef and Puppet came into the fore in Ruby in Rails and in that community through the things like Capistrano and whether or not they're still maintained. But in any case, I think that it’s very comfortable for a Ruby developer to switch or to start implementing something like Chef or Puppet specifically Chef because the DSL is so comfortable there. I think that there were things like CFEngine which came before Puppet and Chef and really kind of started the whole current wave of infrastructure automation. But I think that the big thing that sort of really driven the tool sets or the tooling around DevOps really are things like the Cloud, the ability for anyone to easily spin up infrastructure. I think frankly that a lot of what happens is developers will work around the IT Department because the IT Department is simply saying ‘no’ all the time. But I have functionality that I need to deploy and I need to manage those systems. And I can easily work around you with a credit card and leverage something like Chef and now, my application is live and you don’t even know about it.

Do you think the recent interest in things like SOA architectures to scale applications, it seems like maybe that’s another sweet spot why DevOps is becoming really important because it does require attention at both levels. The SOA requires some pretty savvy on the upside to build out the structure. And then in your programs, you're going to have to be aware of that because as the Netflix guys like to say, “Something’s going to go away eventually.” That service is going to go away and you got to handle it.

Yeah, that’s absolutely the case. I think that also, obviously with SOA, you move away from or you start to see this quick proliferation of services and therefore, servers that you need to run those services on. And so, the problem just gets bigger and bigger very quickly. And you quickly understand that managing things by hand is certainly not going to work ever.

One other question I have for you regarding DevOps is that it seems like a lot of people get into it by using some of these other tools. Not Chef or Puppet but something like Capistrano, or Mina, or Moonshine even which is sort of a dumb down version of a DevOps tool that uses Puppet so you can do simple deployment [inaudible]. Do you think that people eventually move up to Chef from those or are there tradeoffs to using tools like Moonshine that kind of do both things?

I would say that when I first came to operations and was managing my first sort of Ruby or Rails application of Scale, I definitely started with Capistrano and was using Capistrano almost exclusively to manage that application. The downside of tools like Capistrano is that in order to build in idempotence, you have to do a lot of stuff. You basically get that for free with Chef. And by idempotence, I mean things like if I want to, say install Nginx. I can easily write a script in Capistrano to build Nginx from source and install it on a server. The thing that I would then need to add in to se my Capistrano recipes is to check first to make sure that Nginx isn't there, so that I don’t install Nginx every time I run that particular script. Whereas things like Chef, you get that built-in for free. It knows and it’s smart enough to know, is Nginx here? If so, I don’t need to take any action; otherwise, I do need to take action to make something happen. And what we really get into then is we’re modeling sort of the desired state of our infrastructure which is a slightly different way of approaching the problem. It definitely requires some mindset shifts than something like Capistrano where I'm just going to script. These are the steps I take to build this infrastructure.

That kind of leads me into an interesting side question. I don’t know if you saw recently, Nathen, this blog post by Chad Fowler on Immutable Deployments. But this isn't the first time I've heard of this. Lots of people going to structures where when you build and deploy a server, then that’s it. And that server should never ever be changed again. So in other words, you would never re-Chef the server where it would need the idempotence feature. And then the idea is there, that if you need to deploy something else then bring up a new server, load your image on that or whatever with the changes and you put that in. And then, swap them out at the HAProxy level or whatever, your load balancer. Anyway, I was just wondering if you’ve heard of this kind of trend that seems to be -- I've seen multiple people doing it now where a server is a one-time build thing and then if that needs to be done for any reason at all, then it’s totally replaced. I was wondering what you think about that.

It’s interesting that you mentioned Chad’s article. We’re actually working to get him scheduled to come on the Food Fight Show. Perhaps we can have an episode where we actually do some battle.

[Laughs] Awesome.

I think you see actually, James, you said you listened to the Netflix episode. That’s a lot of the way Netflix manages their infrastructure. In fact, the application developers, user application use itself its own AMI or Amazon Machine Image that gets launched. And when you need to change the application, you're actually building a new AMI and launching that. And that’s the way that Netflix manages much of their infrastructure. I think that what we’ve learned in terms of system administration over the years is that golden images and essentially, that’s what we’re talking about here are golden images, aren’t necessarily a great way to manage your infrastructure. I think that you end up doing something like Netflix where you have to build a lot of overhead around how do I manage those golden images or how do I manage those AMIs and make sure that I can, in fact, get to a point where I have immutable servers. But even in the example that you just gave, that HAProxy server is going to have to change. So, the configuration of which application server sits behind the load balancer, that’s going to change over time. And so, I think that it becomes challenging to sort out, what are some small changes that potentially need to be made and what is their impact on having things like golden images. So, in the case where I have a HAProxy server, if I need to change the configuration of which backend services is it proxing loads to, do I want to rebuild the golden image or do we build the machine every time I change like that needs to happen. And I would argue that you certainly don’t want to. You need to have something that’s a little bit more flexible. And that’s where something like Chef or Puppet comes in.

So, I think I hear what you're saying there. I will give another interesting example though, that I heard recently. I believe it was at Ruby Midwest and I’ll try to find the link to the video. But one of the talks there, the developer works for basically a shopping application company. And they have very strict compliance that they have to keep due to the financial aspects of their business. So, one of that is that basically, they can't deploy a line of code that hasn’t been inspected by them. So literally, every time they update a gem, they have to go back through all those commits that led up to that gem and go through them and approve all those changes and verify that they're safe. And so, by using an architecture like this and they too build these kind of ready-to-go images, they knew that whatever was on that image, they had been through that code. Whereas if they just allow willynilly updates or whatever, it’s possible a code could sneak in there that would break their compliance that they have to meet. I think it was an interesting use case. I think there might be some value to the approach though. I think I understand what you're saying about some things are going to have to change.

I definitely agree of their value to the approach. And I think that it comes down to, how do you build those images? And something like Chef where you're modeling your infrastructure, of course, you're modeling it as code and it’s all in your GitHub repository or your git repository or whatever source code control you use. That also helps with auditability of what has changed on these servers over time, how have we been building out these AMIs. You can potentially give your Chef recipes with an explanation to your auditor and say, “Look, this is how bits get installed and configured on this server.” And this is the only way that those bits get installed and configured on this server.

I have another question. Chef Server, I believe, started out written in, I don’t remember if it was Rails or Sinatra. But I'm aware that recently, they’ve moved over to writing it in Erlang.

Yeah, that’s right. The Chef Server definitely -- Chef basically utilizes a client server model where you have on each one of the servers that you're managing or that are managed by Chef, they're running the Chef client which will just connect to an API server. So, the Chef Server itself basically has a couple of roles. It is an API server that your Chef clients will connect to. It also includes a searchable index of data about your infrastructure. So, when we talk about things like the load balancer, you can query the Chef Server and ask it, “What application servers should I configure to be behind this load balancer?” Or in a Rails application, you could use a query against the Chef Server to determine, “How do I write out my database.yml?” And then, the other thing that the Chef Server does is it provides essentially a publishing platform. So, locally on my workstation, I’ll write my cookbooks and recipes which are essentially the policy of the desired state of my infrastructure. And I will publish those up to the Chef Server and it will take care to distribute those recipes and cookbooks out to the nodes or the servers that are being managed that actually need to apply those policies. And you're absolutely right. The Chef Server itself was originally written in Ruby. And what Opscode has done over the past couple of years is we’ve launched a hosted Chef service where essentially, we’re running the server side of that architecture as software is a service. And what we saw in building that out in Ruby was that the API server really wasn’t scaling in a way that we wanted it to. And so, not only for our own needs in terms of hosted Chef, but also for some of our larger client needs. So, we went about rewriting that entire API server or the entire Chef Server in Erlang. And what we’ve found with that was that we got incredible increases in performance and speed, much more consistent memory and CPU utilization, and just awesome huge rapid games there.

Awesome.

Therefore, Ruby doesn’t scale.

I would say that one of the challenges that we do run into with folks is we are essentially asking system administrators to start modeling their infrastructure as code. And for a lot of system administrators, they feel like that is a huge step. They're not programmers, or at least, they don’t view themselves as programmers. They're used to writing Bash Scripts and things along those lines. So, coming to something like Chef can become a little bit scary for them. We tried to do our best and reassured them that they are in fact programmers, they just have really shitty tools. And so, we give them Chef and Ruby and that’s much, much better for them. And they can be much more happy there.

The team that I worked on where I characterize it as a cold war was like that. And the VP of IT was a Bash Scripting master. I mean, he taught me some amazing things in Bash. Every time I kept suggesting that he go do something like Puppet or Chef, he’s like, “What does that give us?” And I would describe another feature of Puppet or another feature of Chef. And he was like, “Well, I can do that.” And he would go off for a week and he would come back with a collection of Bash Scripts that did that for him. He just kept splacking on another layer of Bash to this mud ball that they use for their deployment. About a year after I got laid off then that particular guy quit, the other guy at IT who was also wicked sharp, I met up with him at a conference. And I'm like, “So, how is deployment going?” And he says, “Oh, it’s fantastic since so and so left. I moved everything over to Puppet and it’s easy now.”

[Chuckles] Yeah, that’s not uncommon. Oftentimes, when we go and teach system administrators, one of the things that Chef provides as a resource -- a resource is kind of the primitive in Chef that you will use to define your policy. And so, a resource might be a package or a user or a directory or something like that. But one of the resources that we have is an execute resource which is basically the punt resource. So, we don’t really have a great way for you to model which you want to model. You can just throw in some shell out into this execute resource. So, oftentimes, when you get someone like that with hundreds or thousands of lines of Bash code, the first thing they do is they move all that Bash code into an execute resource in their recipe and then they're done.

They're using Chef now.

It’s kind of like taking your entire -- I gave a talk about this on refactoring horrible Ruby scripts. And I said, “The first thing you do is you open up that entire script then you type class Application; def self.run; and then your whole application and you say, end; end; if __FILE__ == $0; Application.run and you're done. You’ve converted it.

And now, my program is object-oriented.

Object-oriented, that’s right!

That’s right, absolutely. And be sure to check it in the version control. I often remind folks that version control does not end in .back or .timestamp. That is not version control. [Laughter]

I had a coworker who would rename things to .org and that’s an organizer mode in Emacs and all my Ruby files started turning into planner sheets. I'm like “Chris, stop this!” [Laughs]

Yeah. I have another question. With Chef, you have your Chef client and I'm assuming that that can send shell commands because it really can.

Sure.

What power tools do you find yourself using on the server to get the work done from the shell, in particular?

I'm not sure I fully understand the question.

Grep, awk, sed…

Oh, sure, sure, sure. I think that for the most part, Chef enables you to not have to worry too much about some of those underlying shell commands like grep, awk, and sed. And we provide resources like for example, you might want to use awk or sed to manipulate a configuration file. With Chef, you get essentially templates. So, you have ERB templates that you can write out on to the file systems. Instead of managing a particular bit within a file, say a configuration parameter within your, maybe an Apache directory or something. Instead of managing just that directory, you will manage the entire file and you manage that with Chef through a template. I think that as a system administrator, you certainly are always using command line tools. But with Chef, the idea is really to get away from ever having to SSH-ing to a server. And then, any configuration change that you want to make really starts with a commit into your first repository.

That’s an awesome idea, actually. Developers have similar practices, like you shouldn’t really go into the server and edit the Rails file there and restart the thing so you can see what happens.

Absolutely. I think that the really exciting thing in the Chef community over the last 18 to 24 months is a lot of tools are being built up around Chef, basically an ecosystem of tools around testing Chef. And this gives us things like unit testing through chefspec, TDD through Cucumber-chef and basically everything in between. And the beauty of this, of course, is we can start to build deployment pipelines just like you would build an applications deployment pipeline wherein I check in some code to my git repository that’s an infrastructure code change. Something like Jenkins or Travis CI can pick that up, execute my tests to make sure that my build is clean. And then, can push out those changes out to production and you can actually start to manage your infrastructure in the same way that you manage your applications.

I have a question that’s always nagged me about managing servers. And perhaps, you can be the voice of reason at bridging the divide. And that’s installing stuff the Ruby way versus installing it through package management systems.

Then you should just package everything. Next question. [Laughter]

Okay, [inaudible].

Are you talking about there's, on certain systems, like a Ruby-MySQL versus installing the MySQL gem?

Yes, that kind of thing. In fact, I was trying to deploy into somebody’s data center and the ops guys there would not let me install any Ruby gems. They like had their own packaged-up gems that went in through their, I don’t know if it was Ubuntu or what, but they had their own repackaging of all of the Ruby gems as native packages.

That sounds really familiar. Doesn’t it, Dave?

Yeah. The cold war IT Department was the same way. Every single gem that we wanted to install, we had to give it to them and they would [inaudible] it and they would build an RPM package out of it and use that to deploy it.

Obviously, they have something that they care about there that doing that extra work is supposed to make it happen.

Yeah. We had classified data on those servers. And so, if the cows got out of the barn, the whole farm would be raised to the ground by the government. And so, they -- I kind of tortured that metaphor a little bit. But anyway, they took a really dim view of security breaches. And so, for them, they weren’t just being anal, they were doing their jobs.

I think it really comes down to that communication. In some cases and in some environments, you simply can't get out to the Internet. In that case, I need to either run my own gem server. In which case, I want to make sure I know what gems are on that server. But likely, that means I'm also running my own package repository because there are system level packages that I can get from vendors or that I can create on my own. And so, going to this idea of, now we’re complicated because we have some things that are gems that can be installed and some things that are packages that get installed and they don’t exactly work the same way. And so, how do we streamline that process and maybe packages is the right answer there. But really what it comes down to, I think, is having that communication between the operations team and really talking about, “Alright. Here's how our system architecture works and let’s work with the development team to make sure that we’re deploying their applications in a way that’s going to work for them.” Should I bundle with a Rails application? Should I bundle install all of the gems, or should they be there already as part of the system?

The hilarious thing about that was this was long enough ago that it was pre-Bundler. And it was like, “Oh, just give us a list of all of the gems that you need installed and we’ll install packages for them.”

Sure, that was easy. [Chuckles]

[Chuckles] It was like, “Well, how do w find that out?” [Chuckles] There was no obvious way to do that back then. It wasn’t too hard to figure it out.

Yeah. It’s certainly easier now. But you also have tools like FPM. I don’t know if you guys are familiar with FPM. That’s a really interesting tool. I’ll give you a link for the show notes. But it stands for Effing Package Managers. It’s a nice little Ruby scripts that you can use to build a package for basically anything that you want. Well, I’ll put RPMs or DEBs based on what you want. And so, typically you'll find folks are using that to build Ruby itself. I would never use something like RVM or rbenv in a production environment. I would have a package that is Ruby and that’s how Ruby gets installed on this machine.

Why not use something like RVM or rbenv in a production environment?

The reason that I would shy away from that is because I think that the real benefit you get from RVM or rbenv is being able to switch out which Ruby I'm using and test out various versions of Ruby. In a production system, I'm only ever going to need one version of Ruby. And if I need to change the version of Ruby, I should just reinstall all the package. Or probably, even more so, I should blow away that server and provision a new one with a proper Ruby version installed on it.

So, you’ve never run a deployment that had MRI and JRuby on the same server?

I have not.

[Chuckles] Just saying, it happens.

As a hacker, I have a system that belongs to me that comes off my personal credit card and it’s got Ruby 1.8 and Ruby 1.9 and Ruby 2.0 and JRuby. And Lord help me there, all running on that same box under the same Nginx instance, just depending on which virtual host it hits.

Yeah. I've got the same kind of thing going on with Apache and Passenger. I think it’s the prerelease version of Passenger lets you specify which Ruby per app. But yeah, same thing here. And it’s just because I don’t want to buy ten servers for ten apps that are on different versions of Ruby.

There are also much lighter tools than RVM and rbenv like chruby which seems to be gaining a lot of grounds like a hundred-line Bash scripts that just changes environment variables, basically.

Sure. And I think that the big argument against those, like you said, you're running many applications on the same server. That’s totally acceptable and a great use case for something like RVM or chruby. But as you get into scaling out those applications, and granted in your case, in your own personal applications, they aren’t [crosstalk] scale.

I mean, some of those servers get dozens of page we use per day.

Holy cow!

I know, I know.

And you have time to spend on a podcast?

No, actually I don’t. In fact, one of those requests is coming in right now. Hang on, I've got to tell that [crosstalk]. [Laughter]

Man, you really respond. I did have an application that needed two Ruby for some Java [inaudible]. But the way we did handle that in our production environment is we did put that on a separate machine so that we only had to worry about Java and those dependencies on that particular machine and then we communicated it with it when we needed those kinds of things. And that worked out pretty well, I think.

I will say that another bit of DevOps, if you will, is really all about sharing information. And we’ve talked a lot about how development operations really need to understand some of the constraints. And as they're developing applications, what does the system architecture look like. I think the other big place where that communication really comes into play is when things go wrong. And that really comes down to things like Post Mortems and the way that you run Post Mortems. And including both development and operations in a Post Mortem, I think, can serve both sides tremendously.

I actually worked at a company where we had a Post Mortem and the bus came in and very literally started a meeting with somebody is getting fired today. That’s not helpful.

That’s always [inaudible], assess blame.

Once we’ve done that, we can move on and actually figure out what went wrong.

The first step is admitting somebody else has a problem.

That’s right. [Laughter]

I think that a really good point. I mean, you talked about having both teams on there and stuff. It’s like if you had a DBA and it was a database problem that happened, you’d want them there. And if the developers, if there was, how the application was using the database, you’d want the developers there.

Yeah, that’s absolutely the case. And I've been in Post Mortems back when I used to run production systems not when I was Community Manager. But I've been in Post Mortems where on the operations side, we could absolutely come up with a way to prevent that thing from happening again. And it was super complex and would require a lot of work from us. Whereas having a developer in the room meant that a small change to the way the application worked could solve that problem and would take a tenth of the time in resources and it was actually the right solution anyhow.

I think you actually hit on something really key right there. I was involved in an application at one point where the client was asking for a very unusual feature. They were asking us, the developers, and it was going to be really complicated to implement and we were pushing back pretty hard because we didn’t think there was a big game for a development required. But it turned out, the ops guy, he basically did it by changing the web server config a little bit. He was able to get them basically what they want and it was a very simple thing to do. And I think that’s what you're hitting on right there is that, sometimes, things are very hard from one layer but not hard at all from the other layer. If you have all of these different services in a service-oriented architecture, and you need to handle some off globally across, maybe for third parties, one way to invent a complicated system that all of those services use programmatically that manages the [inaudible]. Another might be to put a proxy in there with some off-settings on it or something like that. There's sometimes an easier way to skin the cat from the other side.

That’s exactly right. And I think that kind of the other side of that is that DevOps doesn’t mean developers do all of operations. It doesn’t mean that we all need to start specializing in the other’s field. It’s simply that we work together and really understand how do we work together as a team and come up with solutions that are most appropriate for the challenges that we face.

It’s the impedance mismatch problem all over again. I mean, solve the problem at the level the problem is occurring. And that’s the easiest way to do it.

One other thing I want to just toss in with Post Mortems is that in a lot of cases, in fact, I've only seen one or two cases where it was clearly like a mis-configuration of the database or clearly something that was going wrong in the application. If it’s a major adage, it’s usually a conglomeration of things that come together that cause the problem. And so, you can reconfigure the database but you can also rewrite some of the application code and both help to alleviate the problem. And so, I really like the idea of pulling everybody together and saying, “Okay. Here’s the problem. How do we solve it?”

Yeah. I think you hit on another great point there. And that is that there rarely is a single root cause. It’s always a bunch of highly improbable things that will never happen together, all happening at the same time.

Yup. You get that wacky convergence of circumstances that’s traffic and whatever and whatever else.

You can read good books about how airplanes don’t crash because one thing goes wrong. We’ve made them so redundant and carefully balanced and monitored and everything. We have practices in place. Airplanes crash because ten things go wrong.

At the same time, yeah.

And that’s the other part of DevOps is taking learnings from other industries like the airline industry, like manufacturing. And also, just taking things like our Post Mortems and sharing them with the world. GitHub does a great job with this. Post Mortem happens, it happens in the public. It happens where people can see and read about everything that happened. As a professional, we can all learn from each other and make our own infrastructures even better.

I have a question about process and team composition. When Elisabeth Hendrickson was on a couple of weeks ago and we were talking about the role of testing and exploratory testing in the application development process, she was big on, “Oh, you want to have people who are doing your testing involved very early in the process,” so they can have some input into the requirements or to look at the requirements and say whether they’re going to work or not. How early does it make sense to get ops people and DevOps type stuff going in the app development process? Do you want people there from the very beginning?

I really think you do. I think that if one of your goals is to get to something like continuous delivery, then you really do need operations involved at the beginning. And there are tools out there, things like Vagrant that allow you to very easily spin up a development environment that’s going to look just your production environment. Let’s say you're deploying to something like Ubuntu. Instead of developing on your Mac locally, developing a Vagrant VM that’s running Ubuntu that’s configured with something like Chef and we’re going to use the same recipes to configure your production environment, build out that delivery pipeline. Start with your Capistrano scripts. Day two, can I deploy to a staging environment? Deploying code should not be an exciting thing that happens only once every two weeks or once every six months or what have you. It should happen all the time.

That was something that was brought up in one of our Book Club books. The first step is testing your deployment. It’s the Object-Oriented…

GOOS?

Yeah, GOOS.

Growing Object-Oriented Software, blah…blah…blah.

Yeah, that one. I’ll put a link to that in the show notes. [Laughter]

I want to follow up on that last question. Elisabeth talked about a couple of finesse moves that she had figured out to get the project owners to pay attention to the input from testing. Do you have similar or other kinds of moves that you do to get people to pay attention to your input early enough in the process to have it be useful at all?

I think that that, in part, comes down to where you are in the life cycle of an application. So, if we’re talking about building a new application from the ground up, I think that it makes sense to immediately put an operations person in the same room and working together with that development team hoping to build out that infrastructure from the onset. If that’s not the case, if you're looking at an application or a team that’s already together, I think that it makes sense to sort of, in either case, put operations in the same room with development. Have them working together from the beginning and really looking at -- I think the other kind of big thing is looking at the definition of ‘done’ for a developer. I found that in a previous life, done meant I did a git push. That’s it. My code is in the repository that everyone can see, therefore I'm done. And what you really need to do is redefine done for developers to mean that your code is in production. And I think that the operations team, our goal and sort of our idea is to build the tools that are required to allow the developer to get his or her code all the way through to production without needing to have an operations person there. “I shouldn’t be a merge monkey, I shouldn’t be the only person who knows how to press the ‘deploy’ button.” Everyone should be enabled to do that.

Let me get this straight. You actually want us to talk to each other?

I know. It’s crazy. It’s like a ludicrous idea but occasionally, I think it is required.

I want to ask you a question. Do you think this is getting better over time? I mean, my impression is that it probably is. I do have my horror story of the separate IT Department for me and the horrible things I had to do because of them. But that was a long time ago. And a lot of the clients I’d worked with in recent years seem to get it much more. In my current project, the ops guy definitely works day-to-day in the same room with us programmers and we solve each other’s problems and it’s great. And then, we have things like the DevOps movement and stuff. It seems like this is going in the right direction, do you think?

I think this is definitely going in the right direction. Not only are we seeing a prevalence of operations and dev working together a lot more, but I think that we’re also starting to see it in much larger companies. I think that you can now start to look at enterprises that are starting to embrace the idea of DevOps. And I do think that the way that they're embracing the idea of DevOps is rather similar to the way that they embraced Agile when it first came out. So, there's an edict from on high that ‘now we’re doing DevOps’ just like there was, at one point, ‘from now we’re doing Agile’. And so, you see some teams that have a standup everyday or that start to put an operations person in the room with the developers. That in and of itself doesn’t get you to DevOps or doesn’t get you to [inaudible] just standing up everyday. But there are pockets within the enterprise or pockets within companies that are well-established that are starting to actually understand the ideas and adopt them properly. I think that we have a long way to go to really help people make the transition, help people start talking to one another, help people start looking at things across the value chain. But I definitely agree with you that we are getting better. The future is quite bright and we’re moving in the right directions.

Awesome.

Is there something equivalent to like the Agile Manifesto that talks about the fundamental principles of DevOps?

That’s a great question.

For me, the thing that comes closest is a blog post that John Willis wrote on, it was actually on the Opscode Blog that defined DevOps as being about CAMS - Culture, Automation, Measurement, and Sharing. And he talks about things like people in process first and how do you automate things and so forth. I’ll put a link for that in the show notes as well. DevOps itself, you will find many, many definition for online and in books and so forth. But I think, for me, this is a really good summation of what the DevOps Movement is all about.

Cool. Okay.

Awesome. Let’s go ahead and get into the picks. James, you want to start us off?

Sure. I've been picking Jesse Storimer stuff lately because Jesse Storimer is kicking ass. But I picked his Unix class, maybe that was last week. And if you went and checked that out, then you probably found this video. But I'm worried that some of you didn’t go check that out and didn’t find this video. So, I'm going to link to it in a different way. He has a little screen cast here where he basically walks you through showing you how tools like Spork, Spin, Zeus [inaudible] work. And he does that by taking the Ruby Gems repository that we’ve talked about on this show before. And going in and just taking a test that takes about five seconds to run, and just building up infrastructure, that he cuts it down to, he shaves off two seconds, I think. It’s really interesting how he does that. And it shows you some cool Unixisms and stuff like that and why learning this kind of stuff is valuable. So, it seemed kind of DevOps and I like that. So, check that out. And then, for totally fun. A board game that my group of friends has been playing recently is called Flash Point. And it’s a pretty cool game where you basically show up as a team of firefighters at a building that’s burning down and you have to save seven of ten victims that are trapped inside. And each person plays a different firefighter that has different abilities. So, some are better at getting people out and others are better at containing the fire and stuff. It’s a cooperative game where you all work together and the board’s working against you trying to blow the place up before you can get everybody out. It’s a lot of fun. It’s been really enjoyable. So, check that out.

Awesome. Dave, what are your picks?

I have four but three of them are just links. So, just really quickly. The first one is from Mountain West this year. It’s related to the podcast today. Mountain West Ruby Conference ‘Escalating Complexity: DevOps learnings from Air France 447’. This is a really compelling talk about what happens when human beings are surrounded by tons and tons and tons of complexity, how to assign blame when very competent and well-meaning humans are actors in a highly complicated system. The second link I have is a little bit of fun. It’s from my old live journal that I don’t update anymore. But I was updating it infrequently when I was working at that place where we had the IT cold war. It’s a little post called ‘What I Do at my Job’. And it includes the sentence ‘I write programs. Making servers go is not my problem. But that doesn’t mean I can't write programs to watch servers and let me know when they stop going.’ And it’s just a fun little true tales of DevOps kind of thing that happened to me a couple of years ago. The third one even faster is from BuzzFeed. It’s ‘21 Jokes Only Nerds Will Understand’. A Roman walks into a bar and orders a martinus. The bartender says, “Don’t you mean a martini?” And he says, “If I had wanted a double, I would have asked for one.” There’s 20 more on that page. They're fantastic. [Laughter]

And then, the last one is an iPad game - Ridiculous Fishing: A Tale of Redemption. This is a $2.99 game. I bought it for two reasons. One, it won a design award for 2013 from Apple. And the second one is there are no in-app purchases at all. For $3.99, you get the whole freaking game. You’ll get tired of it in less than five or six hours. There's only about $7 or $8 worth of entertainment there but what a fantastic $8 worth of entertainment. You drop a fish hook down into a trench and you have to avoid the fish all the way down until you hit a fish. At which point, the fish hook starts coming up. And now, you try and hit as many fish as you can. And of course, because this is how fishing works in real life, when they get to the surface, you fling all the fish in the air and you have to shoot them with a shotgun.

Or a bazooka.

Or a bazooka or you can upgrade to dual machine guns. It’s ridiculous fishing. The title says it all. I highly recommend it. It is well worth the $3 that shell out for it.

I'm pretty sure I need a faster iPad to really do the game. [Laughter]

You're catching too many fish. Yeah, exactly.

Remind me not to go fishing with that guy.

Dude, if we find this guy in real life, I want to be on that boat. I don’t want to be hunkered down on that boat but I want to be on that boat.

David, I think James was referring to you.

[Laughs] I was.

Oh, I was the guy. Exactly. I had a friend from Georgia who referred to hunting is being drunk with a gun in a tree and fishing was being drunk with a gun in a boat.

At least he knows the difference.

Yes. Those are my picks.

Josh, what are your picks?

My first pick is Ryan Bates. We’ve had him on the show, we love him. He’s a little burned out right now. He put up an announcement on Railscast saying he was taking the summer off. So, good for you. Ryan has been a mainstay of the Ruby community for years. He’s given a lot to the community. And I think now is a great time to just give him some unconditional love and support because he deserves it.

Yey! Absolutely.

And then, I have a slightly more practical pick. I had to level up my YAML recently. I did some crazy stuff in a YAML file. I think I picked Middleman a month ago or so and I used it for the GoGaRuCo website. And then I did some crazy stuff in YAML to simplify what I was doing in the Ruby code that was displaying pages. I just realized, I have all that code up publicly so I can put a link to that repo and this as well. But I found a pretty cool page on the YAML website that explained all of the bizarre stuff that YAML can do. Format is actually understandable but the basic YAML specification on YAML.org is pretty impossible to understand. [Chuckles] But there's a great page that simplifies a lot of that stuff by showing an example in YAML and a corresponding form in Ruby. So, this is YAML for Ruby and I think it’s a great way to understand what the YAML code is doing, and if you're trying to do something advanced to get an example that’s actually useful. I think that’s it for me this week.

Alright. Well, I’ll go ahead with my picks then. My first pick is if you're trying to get started with Chef, go try out Hosted Chef. You can get a free account. It lets you manage a limited number of machines and stuff but it’s really cool. And it’s a good way to get going and then you don’t have to set up your own Chef server so you can kind of just get started there. Also, PeepCode’s Chef Videos are really good. I picked those up. They get you off to a decent start. And so, I really, really like those. And then, Gregg Pollack, Jesse [inaudible] and I did a Rails and Ruby Courses Roundtable on Google Hangouts. I'm uploading the video right now. For some reason, we couldn’t get the Hangouts on Air to work. Anyway, it was a good discussion and if you're interested in learning Ruby or Rails or leveling up that way, it’s a great way to go. And I’ll have the link in the show notes for that as well. Nathen, what are your picks?

Sure. I just want to plus one your Hosted Chef. It is a great way to get going with Chef. System administrators always want to start by installing a server. That’s how we want to learn things. The problem with installing a Chef server is when you're done, you have a Chef server. You don’t actually have any way that you’ve managed or automated your infrastructure. So, starting off with Hosted Chef is definitely the way to go. There's a sister site or a site that will also help you get going and that’s LearnChef.com that will help you with things like Vagrants and you can get basically from zero to first Chef converge within about an hour or so. On to my picks. Actually, I'm going to start with a pick that isn't my own but my co-host and the founder of the Food Fight Show, Bryan Berry couldn’t be with us today but he sent along his pick in absentia and that is Elixir. He’s been playing with Elixir a lot lately and is falling in love with it. So, he wanted to be sure that I pass that along to you. For my picks, the first pick that I will give is the Chef community itself. I know that on the Rogues, the last episode was all about the Ruby community and that was awesome. I would say that everything about the Ruby community that is awesome, you will also find within the Chef community. So, if you're interested in getting started with Chef, come on in. We are an extremely welcoming community, we’d love to have you and we will help you with anything that you need help with. Also, if you want to learn more about DevOps, I have a couple of resources for you. The first is a book called ‘The Phoenix Project’. This is a novel about DevOps and of course, having a novel about something technical sounds kind of wishy-washy. I will say that it is the after-school special of DevOps. You run into it, things start off really terrible, like the child just swallowed the poison under the sink. And by the end, you know everything’s going to be okay and we’re going to be living in this awesome DevOps world. And that’s exactly it. But you should totally read it. It’s a fun, easy read and you'll learn a lot about the ideas and tenets behind DevOps.

And it features drinking bleach.

Indeed, and Misty Eye stickers.

I'm in.

Also, if you're looking for additional podcasts in addition to the Food Fight Show, I have to recommend The Ship Show which is all about DevOps and Release Engineering. So, check that one out. And then finally, for something fun. If you want to have fun experiencing what a lot of operations and dev people feel like during an outage, I highly recommend this game for the iPhone. It’s called Spaceteam. Spaceteam is this awesome game that you have to play with two to four players. You find that you're in front of a console, shouting out instructions to one another trying to make it on to the next level. It’s super awesome, super geeky. I challenge you not to have fun with Spaceteam.

Is it like the Star Trek where you have one person is the captain and one person is the science officer? Is it like that?

It is similar to that except, of course, no one is in-charge. Everyone is just thrown into chaos.

And everyone has a different view into the system?

Yes. Everyone has their own control panel in front of them and instructions come up like ‘make toast’ might be one of your instructions, or ‘set the gamma ray to six’ might be one of the instructions. And you have no idea whose panel that control is from. It might be on your panel, it might be on one of your neighbor’s panels. You have no idea. So, you all just shout out instructions.

And they're getting instruction commands too, right?

Absolutely. Just like as if, “Holy crap, the site is down. I'm going to restart Apache, you go and redo the database,” Or, “You roll back this code.” So, it’s all very, very fun.

I'm installing that on mine and my wife’s iPad right now. If I still have a marriage next week, I’ll let you guys know.

It’s also fun for kids. We had probably 13 12-year-olds sleep over and I taught them all the beauty of Spaceteam.

You know, we’ll probably be playing that at the Rogues Retreat, it will be terrible. [Laughter]

Oh, dude. Yes!

“How was the retreat?” “It was fun, it was lot of fun.”

Highest score.

“How was the book coming?” “Book?” [Laughs]

“Book?” [Laughs] “Was it one of the commands? What have you got? Write a book?” [Laughter]

Awesome. Let’s go ahead and wrap the show up. Thanks for coming, Nathen.

Thanks for having me. I certainly had a whole lot of fun. I hope you learned a thing or two.

I learned a lot. Thank you, Nathen.

Yes, thank you.

I learned I need to listen to a new podcast.

Yeah. I wish Bryan could have made it. But we totally understand the whole ‘having a baby’ thing.

He’s having a baby? [Crosstalk]

I don’t understand how Bryan had a baby.

I didn’t say Bryan had a baby. I said he was involved in having a baby.

No, you said Bryan had a baby.

No. I said he couldn’t be here and we understand the whole ‘having a baby’ thing.

I assert, I assert that you said. [Chuckles]

David, you're misconstruing how I'm misconstrued this.

I know. But I'm having a lot of fun so I'm not going to stop.

Make it end! [Laughs]

Can somebody kick the plug?

See you all next week.

Bye!

Thanks so much.