In this episode, the Rogues talk to André Arko about Bundler.

Special Guest: André Arko.

 
[Laughter]

Oh, we were devolving so freaking fast. [Laughter]

Yeah, that may be the worst train wreck ending ever. [Laughter]

Well, I've been writing these conference talks. I was up until like one o’clock last night, and then I only got like 4-5 hours of sleep, so…

That’s hilarious.

I didn’t even know what I was saying half the time.

Hey, and welcome back to the Ruby Rogues podcast. This is episode 45, and this week on our panel, we have Avdi Grimm.

ANDRÉ: Hello, and lately I've been exploring ancient Unix tools, and so moving forward, you can expect to and [unintelligible] see from me. [Laughter]

We also have James Edward Gray.

Actually, James has been replaced this episode. Due to a bug in the Ruby Rogues site, he was overwritten and will be played by some random internet person this time.

[Chuckles] I don’t know what that means, but I'm afraid.

We also have David Brady.

Hi, I'm David Brady. I'm the chief metaphor officer at Slide Rule Labs. And according to my most recent twitter poll, I am the world’s most favorite slow motion train wreck.

[Chuckles] That’s awesome. We also have Josh Susser.

Okay, I'm just trying to keep from laughing here. [Chuckles] Hey, good morning from San Francisco. I’d miss a little bit of the introducing yourself on here, so I'll just say a couple of things. I ran the GoGaRuCo conference here in San Francisco with Leah and Jim. I am doing some startup stuff that is too early to talk about in much detail. I spent a long time at Pivotal Labs, and an old Smalltalker. So that’s reintroducing me for now.

Alright, I'm Charles Max Wood from teachmetocode.com. I've got a few things in the works, and when I'm ready to launch them, I'll tell you about them. And we also have a guest rogue, and that is André Arko.

ANDRÉ: Hey there.

Do you wanna introduce yourself for people who don’t know you?

ANDRÉ: Sure, absolutely. I'm André Arko, also known on Twitter and GitHub as ‘indirect.’ I maintain Bundler. I'm also the author of the somewhat popular jQuery Rails gem. And during the day, I mostly work for Plex Media Center app, doing web back end stuff for them. And pretty much all my spare time goes to maintaining bundler – at least right now.

Sweet!

And can we talk about bundler today?

ANDRÉ: I thought you'd never ask, Josh.

Oh, I don’t believe that. [Chuckles]

Why? Do we have a bundler expert here?

What’s bundler? I want a definition.

ANDRÉ: Okay, well they are these wonderful things called Ruby gems and…

Wait, wait, wait, I can test your initial premise. [Laughter] It's going to be one of those shows, isn’t it guys? [Laughter]

Sorry André. Please proceed. These are one of those thing called Ruby gems. Why now? We'll take it as posited.

ANDRÉ: Sure. I can roll with it. So, Ruby gems let you package Ruby libraries up into relatively compact packages, that contain relatively easily accessed metadata, like any other libraries that they depend on, etcetera. Bundler looks at Ruby gems and says, “This is fantastic, this is wonderful, but we don’t have an application level way to manage all those gems.” Like Ruby gems is great, but you run around running gem install foo, and when you have an application that depends on the API for that gem being exactly version 1.2.3. Then basically what happens is you end up with a read me in your application that you hold your fingers crossed, and hope that everyone remembered to update which versions and which gems the app needed. And you run gem install at times and then you run the app until it crashes to find out which gems weren’t in the readme, or which gems changed their API and the version number on the readme needs to be changed.

You are talking about the way I develop in almost a negative tone. And I don’t like that.

ANDRÉ: Well, I'm here to give you hope for a brighter future.

Okay. Tell me about the dream.

I'm a big fan of “Why the hell doesn’t it work” driven development.

ANDRÉ: Yeah, precisely. So into these dark days of insufficient documentation – to put it kindly -Bundler was conceived. It's absolute need was driven by development of Merb way back in the day, because Merb decided to go with a many many gems approach to library development, they very quickly realized that at an application level, just running gem install was not going to cut it. As probably anyone who’s used Ruby with Ruby gems for a while has discovered, you will eventually reach one of the dreaded activation errors, where you have somehow managed to load one version of one gem, but your app actually need this other version of that same gem -- and now you are stuck. I go speaking for personal experience on projects that I worked on way back in the battle days.

So yeah, the way that comes about is if you have several libraries you depend on, and then you load one of those libraries, and it loads something that both libraries depended on, and it says, “I need version… maybe just says greater than 1.” And so it loads the highest thing that it can. We'll say two, then you try to load the other one, and it says “I need version greater than one, but less than two.” And at that point, you are basically in trouble because you’ve already loaded two and you failed to meet the requirement.

ANDRÉ: I'm fixing that require as either elaborate “load these versions of these…” you would have to basically become a human dependency graph resolver and put the correct gem requirement, earlier in whatever it was that was loading gems.

Or, you could use gem installer.

ANDRÉ: Or you could use gem installer -- which I admit I never actually used.

What is gem installer?

Okay. So a couple of years ago, Chad Woolley who I worked with at pivotal wrote this tool called gem installer that you could put together a yaml file that described all of the gem dependency for your application, and it would install them all. But it didn’t use the internals of Ruby gems; it was using it as a client of Ruby gems. And so it had a lot of the same dependency resolution issues. The main advantage was that you had one file where you could lock down all for the particular versions of all of the gems. So you could at least get… I guess an equivalent of the gemfile.lock… you could down the particular versions that you wanted, and you can be pretty assured that your application atleast be deployable using that set. The thing that it didn’t do is it didn’t let you deal with… you wanted to change one dependency, and have ripple effects. And so you then had to revert to human dependency resolution machine.

So basically, I had all the disadvantages and all the drawbacks of taking something implicit and making It explicit.

I never knew this existed.

Wow. [Chuckles]

I'm with James on this, and I feel bad, because I met Chad and I think he's awesome. And now, I felt bad that he wrote something cool that I didn’t know about. Let me ask you this; is it still worth checking out?

No, Chad totally got on the bundler train.

Okay.

He was a big supporter of bundler, because doing gem installer, he learned all of the problems with that.

ANDRÉ: Yeah, it's actually kind of funny, I'm just looking at the gem installer webpage, and it actually says right at the top “If you think bundler is too complex, then you probably don’t understand the problem.”

Yeah, can we talk about that? Isn’t the problem of resolving those dependencies, isn't that actually like an np hard problem?

ANDRÉ: It is in fact np complete. I mean, you know, just like any other dependency graph problem. So we do the best we can, but every once in a while, someone manages to craft a gem file that has such a wide… in your gem file, you can list a range of versions that you will accept as a valid resolution. And some just people don’t put version numbers in at all; they just list all their gems, and that of course is an implicit greater than or equal to zero version number. And if you’ve got enough gems in your gem file, with a greater than or equal to zero, and there are enough versions of each of those gems, for example, I hear there are a lot of versions of like the Rails gem. But if you don’t require any specific version at all, and there are many, many versions of each of those gems that you’ve hosted in your gem file, of course this means that sometimes we get bug reports that, “Bundler has been attempting to resolve for the last hour, and will it ever finish. And why don’t you fix this terrible, terrible bug…”

You know André, I actually would have accepted, if you would had just with completely straight face said, “Eh, we just proved p=np, and then we just solve it in linear time. That’s how bundler works.”

ANDRÉ: That is bundler in a nut shell. But that only works during the correct phase of the moon, so I'm not going to make any promises.

[Chuckles] That’s awesome.

I have a question for you André -- and it goes back to this np problem -- how does bundler do it? ANDRÉ: [Chuckles]

Go for it. [Laughter]

Is it time for another dramatic code reading?

ANDRÉ: Oh dear. It’s probably more helpful at this point to say with great tenacity and stubbornness, [chuckles] the high level overview, is that it tries as hard as it can to narrow down the problem space in advance, which I think is how most people’s gem files do in fact resolve. And it's actually the time that it takes to download and unpack all of those Ruby gems that occupies the vast majority of bundler’s time, rather than you actually running the resolver. It's a very unusual gem file, that actually manages to make it into that crazy problem space. And as far as I know, all of those people had been able to solve their problem by specifying a more accurate, like a more narrow version number in their gem file and they are running bundle install again.

Can I stop you right there briefly. Will that be a good best practice when working with Bundler, would it be a good idea, now that kind of lock those versions down when you are specifying your gems?

ANDRÉ: So very briefly the officially best practice considered version of putting gems and version numbers into your gem file is to use the ~> operator -- also sometimes known as the ‘spermy operator.’ [Laughter]

Or the ‘twiddle wakka’.

I didn’t know that.

ANDRÉ: I'm just quoting the Ruby gems documentation here. They call it the spermy operator and describes its function.

Oh André, every person who has a conversation with David in the future is going to curse your name.

Yes they are. I will see to it personally.

Are you kidding? He's speaking at a conference next week. We are going to hear about it then.

Yikes. There will be pictures of sperm on all of his slides.

Pictures, cartoons, audio files, oh yeah.

I don’t know. Twiddle wakka has more euphemism potential.

I don't know. I can totally see Dave putting sperm on his slides and hearing the collective intake of air at Mountain West Ruby conference, and half of it is going to be like, “[Gasps] That’s awful.” And the other half is going to be “[gasp] That is so funny.”

I was in London, and going through a gift shop, and they had red dwarf t-shirts, and one of their catchphrases was, “Let’s get out there and twat it.” And twat is in 700 point font across the shirt. And I'm like, “I have to have this shirt.” Unfortunately, I was shopping with a friend who jump on the grenade and said, “No. You cannot have this shirt.” [Laughter] “You cannot walk around wearing a shirt that says twat on it.” Have we lost our family-friendly rating yet? [Laughter]

I'm just wondering if we lost all of our listeners.

Wait, what were we talking about? Bundler? Okay, back to André.

ANDRÉ: Just to update everyone…

This is a great demonstration about the dependency management problem.

Yeah, because you’ve got a David Brady gem in there somewhere, that’s trying to pull in everything and be weird.

Yeah, Ruby Rogues dependency resolver has still been running since the first episode. So far it's not found a good mix.

Anyway, sorry about that André. You were saying.

ANDRÉ: So just to update everyone , I ran over to GitHub quickly and apparently, the correct modern term for the ~> operator is either ‘approximate’, or ‘pessimistic version requirement.’

I like that.

ANDRÉ: As of a few weeks ago, Ruby gems actually uses ‘approximate’ or ‘pessimistic’ instead of ‘spermy’ in all of its code.

I reject that. [Laughter] I just discovered this wonderful thing -- you can't take it away from me now.

ANDRÉ: Considering it's been spermy for years, I think that you can probably keep it for a while longer.

“Dear Ruby gems, please let me have this.”

ANDRÉ: [Chuckles] So anyway, you guys are probably all familiar with the pessimistic -- as it is now known -- operator, but the short version is it allows the last stated column in the version number to increment up to whatever, but it keeps the next highest version number from incrementing. So if you say, 1.1.1, it means I need at least 1.1.1 but less than 1.2.

So if you go back to the versioning episode where basically Josh Susser yelled at all of us for an hour explaining the correct way to do versioning, that means that it accepts basically patches or bug fixes, or tiny releases, but it doesn’t accept even minor releases which would be the…

I didn’t actually understand that. I thought if you said ~> 1.1.1, I'm calling it ‘spermy’ I love that. Sorry. If you say ~> 1.1, what you are saying is it's okay to go all the way up to 1.9, is that correct?

Yes.

I thought what that meant was ‘1.1.anything’ everything you stated had…

ANDRÉ: Correct, so it's actually pretty flexible, because if for example you know that you need

1.1.6, but you are willing to accept patches above that, then you can say ~> 1.1.6, and that means that it will never install anything lower than that, but it will accept everything that is above that, but lower than 1.2.

And if you are doing versioning correctly, then that should not break your program… ANDRÉ: Exactly.

…because those patches are supposed to be safe.

So I'm a terrible person, and it makes me wonder what happens is you do ~> 1?

ANDRÉ: It's the same as greater than or equal to 0. Also the same as not putting anything.

Oh. So I have some bundler files that say ‘Rails ~> 3.0’ and I just figured out now why none of my projects work. [Laughter]

ANDRÉ: So that does mean though that you are safe from Rails master, which is currently at 4.0.

Now If you say 3.0.2, it will not give you 3.0.1, correct?

ANDRÉ: It will. It will not give you 3.1.0.

Or 3.0.1.

David, it's equivalent to… so ~> 3.0.1, is the same thing as saying >= 3.0.1 and < 3.1

Got it.

And I'll do that. There's some gems that I'll specify major minor, I won’t do patch, because I want to be able to… I'm trusting that they are doing semantic versioning, and the minor releases are going to be backward compatible and not breaking.

ANDRÉ: Exactly.

This is the same trust that leads you later to have outrage, right? [Laughter]

Sometimes. So André, here's a slow pitch over to play it for you. In the past, we felt bundler is a little slow.

ANDRÉ: Whoa, I have extremely good news for you theb because….

Can I interject real quick?

ANDRÉ: Yes.

In.. the.. past, we… felt…

You'll see.

45 am last night -- is much, much faster.

Woohoo!

ANDRÉ: Yeah. So funny story about why bundler was so slow. As I mentioned, it's almost never the resolver. Everyone is like, “Well it's an np complete problem. It will be really slow, right?” Well, it turns out that for the vast majority of gem files. And honestly, if you have a gemfile to unlock, there is no problem left, right? The graph had already been resolved. So the vast majority of the time, all of the slowness of bundler came from the fact that it had to download and sort through a vast amount of data from Ruby gems. Basically how that works is Ruby gems offered an index -- and when I say ‘Ruby gems,’ I mean RubyGems.org, the master Ruby gems server that provides the source of all public gems to rubyists -- and there's an index on that server that lets you know the name, and version number of every single gem that has ever been released in the history of gems. And of course, bundler has to go and download that entire thing. At least bundler 1.0 have to go download that entire thing, because your gem file could have quite literally, any gem in it. Needless to say, that index keeps getting bigger every day and it's never going to get smaller -- for sure. At this point, even gzipped as a marshalled set of arrays, it's almost a megabyte. And downloading that over the internet, un-gzpping it and then loading the entire thing into RAM so that we can go through it, is not the fastest thing ever.

So you sped it up by killing my neighbor’s Netflix subscription?

ANDRÉ: We tried to be a little more subtle than that. What we actually wound up doing… Nick, who I believe has been here before, he came up with an API that we have… I should say he added to the rubygems.org server, that is actually pretty awesome. It's backed by Redis -- although I guess that’s implementation detail -- but the short version is, you can hit that API with a comma, list of gems and it will give you back only the information about those gems, and those gems’ child dependencies. Since that’s all the information that bundler actually wanted in the first place, this is significantly faster than downloading everything about every gem ever.

That’s awesome. I think you mentioned Nick and his efforts there. I think Evan Phoenix may have helped a little bit.

ANDRÉ: He did. He helped with some of the debugging and getting it all talking to bundler correctly. Nick masterminded and implemented the initial version. So I am extremely happy to report that on our hello world gemfile just has gem Sinatra in it. And running bundle install on that hello world gemfile takes 18 seconds in bundler 1.0 -- and takes 3.2 seconds in bundler 1.1.

Sweet!

ANDRÉ: Yeah, so that’s a pretty big improvement. And obviously, if you have an extremely large gemfile, that benefit is reduced, but not eliminated. So it is definitely faster.

Have you ever considered doing like local caching? Because I'll go in and I'll say bundle… my gemfile will say Rails 3.2, and it will go get Rails 3.2 and download everything over the internet. And we'll go another project and do the same thing and it downloads it all from the internet. And I'm like, “Gosh, guys, this is on my SSD drive already.” ANDRÉ: Yeah.

Actually, bundler does use if you have like for example, if you have it in your gem set already, and then you bundle, doesn’t it just use it directly from there?

ANDRÉ: It does. So there's a somewhat subtle thing about these that I think most bundler users don’t realize. Bundler actually defaults to not talking to the internet. That’s a very important caveat. If your gemfile is specific about its requirements, bundler will just try to satisfy those requirements directly off of disc. The catch of course is that your gemfile has to actually say, “I want Rails 3.2,” not “I want Rails greater than 3.2.0”. And when you run bundle install, it's important that bundle satisfy your gemfile correctly, because if people want 3.2.1 knowing that it's out there, and bundler doesn’t get it, that would be very… like user expectations inverting behavior, so we don’t want to do that. That said, if you know that what you have locally on your machine is what you are perfectly fine, for filling your bundle out of, there's a switch bundle install –local, and it just says to bundler, “Don’t bother that Ruby gem server; just try to make it all work on my machine.”

I may be inflicting this damage on myself though, because I always, when I start new project, I create a new gem set and my global gem set has nothing in it except for bundler.

ANDRÉ: So, to be perfectly honest, I used to use gem sets and then I started using bundler.

That’s right, yay! I was so hoping somebody was going there.

ANDRÉ: So basically, gem sets existed to obviate the pain of working with multiple applications without bundler available. Because obviously if you have two applications, they might require mutually exclusive versions of some gems. So gem sets was basically hack to avoid those activation errors that James described earlier, where this application means that you have some high version number available and it gets activated in the wrong order, and then app blows up. So happily, with bundler, that doesn’t actually happen anymore. Because bundler resolves the graph in advanced, it's a runtime dependency resolver; it's a free runtime dependency.

Almost the compile time, yeah?

ANDRÉ: Right, it sort of compile. So because the graph is already resolved before your application starts to load any gems, as long as bundler gets loaded, it then locks down all of the gems that can be loaded to the pre-approved correct versions. As a result, that means that your system gems can in fact contain the gems that all five of your apps with mutually conflicting versions. They can all be in that same file, meaning that you don’t have to install multiple copies of the ones that are in fact shared, but there's no downside to leaving them all in system gems together.

Nice.

That’s right.

All though the combination of gem sets with gem installer was actually pretty effective.

ANDRÉ: Yeah, as long as you are cool being the human dependency resolver, gem installer was actually pretty great.

I actually still use gem sets and bundler side by side, mostly because it really just makes me crazy having to try bundle exec all the time… I've also got a couple of apps and I wanna say like a rake task, it's a legacy thing that I inherited from somebody, and I haven’t gone in with the tree surgery pruning shears to amputate. But you launch it up, and it spawns, like system calls to Ruby, and it calls user bin end Ruby, it doesn’t call user bin end bundle exec Ruby. And so, the sub process is that it spawns ends up being outside of bundler scope. And so putting on gem set was the only way that I can resolve that.

I don’t know how RVM handles this, but I can tell you if you are using RVM, you can drop the bundle exec rvm handles that correctly.

ANDRÉ: Yes. We actually even have… rvm is actually pretty awesome that it handles all of that. We even have some proposals on the table for upcoming versions of bundler that will hopefully provide something similar for people who don’t use RVM, and just want to have all my bundle to gem to their path environment variable. So we're working on that. It will probably get in bundler 1.2 or 1.3. But the other hopefully good news is that we're not going to stay on an 18 month release cycle. Bundler 1.2 will come out much more quickly.

Awesome.

Sweet.

ANDRÉ: Yeah.

I have a question about including bundler into your own applications. I mean, I think a lot of people are using it just with Rails, it's sort of built in; you don’t really think about how bundler gets integrated into the app, but when I'm explicitly calling like bundler.setup in my app, it's never been clear to me. Is bundler merely setting up the load path or is it also loading those gems?

ANDRÉ: It's not quite merely setting up the load path, but it is not requiring. Setup does the load path set up, which is important. It also goes in an undoes some Ruby gems tweaks. For example, it restores require to its original load path only version, rather than the gem version of require. Actually, requiring the gems is handled by another top level function, bundler.require. Rails actually calls bundler.require, I believe at the top of application.rb by default, although I’d have to go in and look to check. But basically, Rails developers, sort of in contrast to Ruby developers, became used to always having everything required all the time. And so, Rails says, “Well, let’s not bother people by making them actively require each individual gem that they use in their app. Let’s just have bundler require all of them, since we have the list already available in the gemfile.”

Gotcha. Yeah, because I often write spec helpers that are as lightweight as possible, and I don’t want them to spend a lot of time loading up any libraries that the tests aren’t actually going to be using, so I want them to have the app’s gem set… you know, the bundle available, the exact libraries available but I don’t want it to spend some time loading them all at the beginning of running the tests.

ANDRÉ: Absolutely. On a note very related to that, actually, 1.1 does something else that is very cool, and this is another way that Bundler 1.1 is a lot faster than 1.0. Basically, in Bundler 1.0, we added a very important feature –atleast it seemed that way to us at the time -- that lets you edit your gemfile, and then immediately start your application. You don’t have to run ‘bundle install’ every time you add a gem, change a version, do any of those things; bundler is actually smart enough to run ‘bundler.setup’ called check your gemfile and say, “Oh, you just changed a version number, but that other version is here too, so I'll just use it instead.” This was fantastic for reducing the number of bug reports that we got. There was an older version of bundler that actively require that you unbundle install every single time that you touch your gemfile, and people just didn’t like… that’s a very hard thing to always remember. It even bugged us as we were developing it, which is why we made the change. The downside to that is that it means that Bundler 1.0 actually reresolves your gemfile every time bundler.setup gets called. And so it uses the lock -- if you already have one -- as the basis for that resolution, which speeds things up a lot. I mean, you guys are probably aware that bundler. setup is much, much faster than running bundle install itself. But, it meant that every time a bundle app starts up, it pays that ‘convenience tax’ I guess, so that developers can edit their gemfile, and still have things work. So, Bundler 1.1 said, “Well, that’s silly because in the vast majority of cases, you haven’t actually edited your gemfile. So why don’t we just skip invoking the result or if nothing has change that is important.” So you would think that you could do that really obviously by like saving a hash at the gemfile. The problem is bundler provides lots of features that mean that doesn’t work at all. The naïve approach was too simplistic, and probably cost some problems. But, we managed to come up with a solution that basically just very quickly checks to make sure that everything that’s in your gemfile, is met by the things that are already written in your lock file. And if that’s true, it skips the resolve step completely. In my tests this sped up running bundler.setup, in a bigger rails app, about 120 gems total. It sped it up from a few seconds down to like 1.5 seconds. So it was a very significant improvement. And that of course is faster every single time you start up your app, or every single time you run your tests in a new process.

That’s exciting news for me, because that’s definitely something I've noticed is that adding bundler.setup would add a few seconds to my light spec helper file.

ANDRÉ: Yeah, absolutely.

I'm looking forward to trying that out.

ANDRÉ: Right. I think I was trying to tell James about the best practices for including gems in your gemfile. Yeah, so the best practice is to use that pessimistic -- or as spermy as they have it -operator, so that bundler knows which range of versions you'll accept, and you can run bundle install and bundler will basically give you the… since the version constraint is pessimistic, it will give you the newest version that it can, while still satisfying that pessimistic requirement.

Yeah, and I wanna throw in my support there. I actually got rubygems.org to add a little copy me thing for showing how to put the declaration in your gemfile.

ANDRÉ: It's fantastic and amazing and thank you so much.

[Chuckles] So I used to just say ‘gem install bundler’ for the command line or ‘gem install’ or whatever. And now, I say ‘gem whatever’ and it uses the pessimistic operator as the example, so people don’t even have to think about it; they just click the button, go into their gemfile and go paste them. It's done.

ANDRÉ: That’s totally fantastic. So, it's not super obvious, but there's even an auxiliary benefit to using the version constraints in this way. Not only does that mean that resolver run will be much faster because you cut down on combinatorial number of options that bundler has to check, but it also means that you can run bundle update to automatically apply any patch or minor releases you’re your pessimistic constraints will allow. This is pretty awesome, because it means that if I have like a pessimistic constraint on say rspec in my bundle, and there's a new release, I can just run bundle update rspec and know that I will get the newest version that I'm pretty sure won’t break my app, and then I can run my tests.

Sweet. Can you talk about bundle update in general, because I know that can be a little tricky knowing that when to type update. And sometimes it feels like I type that and I've gone over a cliff and can't get back.

ANDRÉ: Yeah, so fortunately, the way to get back off of the cliff, I'll start there. If you go over that cliff and you want to go back, just check out your old gemfile.lock -- that’s it. You have inversion control already. If you run bundle update and things do not go well, you can just go back to your old lock and everything is how it was before. That said, bundle update can be an extremely useful tool if you actually want to change things. So the way bundle update works is when you run bundle update, you can pass in an argument of a gem name or multiple gem names, and bundler will allow those gems to be updated while trying to keep all of the other gems in the same versions that they already are. I think Yehuda calls this ‘pessimistic version updating’ or something.

‘Conservative.’

ANDRÉ: Yes, thank you.

I find that I actually want that more often that I want just clean bundle update, right?

ANDRÉ: Exactly. So there's basically three levels of installation of gems with bundler -- and they get progressively less conservative. So install all is the most conservative, and all it does is it grabs gems that stated requirements, make sure that none of them conflict with each other, and then writes out the lock file so you are now set with set of gems that you know you want. In the future, once you have a lock file, bundle install is the most conservative option. It doesn’t change anything -- unless you change your gemfile yourself. So everything’s locked, running bundle install just gets you the gems that you already know work with your app. The next level that is slightly less conservative is running bundler update gem name, and that only allows that gem and child dependencies of that gem to update. So conservatively, it's allowing you to get all the new gems that you need to update that particular gem, without hopefully breaking any of the other complex web of the dependencies in your app. And then lastly, and least conservatively, you are running bundle update with no arguments, which says, “Give me the absolute newest versions of everything that you possibly can.” It's similar in function just to deleting your gemfile.lock, and running bundle install again, but it's faster and doesn’t require much resolve time, because it starts from the basis of a last known working resolve.

So what does a bare bundle do? It does the install if there's new lock file and update if there is, is that right?

ANDRÉ: It actually just runs install, but I feel a little weird about having no command invoke install. And I think it's probably in the future release, but right now, just bundle is alias directly to bundle install.

I always type bundle. I never type bundle install.

Coming from the land of Emacs where we have an entire suite of methods that end in DWIM, which stands for ‘do what I mean’, I love just typing ‘bundle’ and it does what I mean. It really install, it will update. And so, count me in as one vote against discontinuing that. And I'm okay if I type bundle, and it's does the most aggressive possible thing. If I load the gun and point it at my foot and pull the trigger, the most surprising thing that can happen I do not shoot my foot. [Laughter]

ANDRÉ: Fortunately, Bundler is preconfigured to not shoot your foot. It will only update if you actively and explicitly add update to the command that you are in.

“Your gun is pointed at your foot. You might have a problem.”

So I don’t really have a question about this, but I felt like the conversation wouldn’t be complete without talking about what I think is my favorite feature of bundler, which is just the fact that I can switch while I'm in development, I can switch between pointing at… like if I have a project that depends on other gems that I'm also actively developing, or that somebody I'm working with is actively developing, I can quickly switch between pointing at a local copy of that other gem, and pointing at the Git repo of that other gem, and pointing at the Ruby gems version of that or some other gem repo version of that gem. And it's just switching the source and specification. I love that. And I think for me, that's like the killer app of bundler. And like I said, I don’t really have a question about that, unless there's any interesting news in that department. I just wanna say it's great.

I have a wrinkle to add to that. Another wrinkle, a variation of that, and that’s that if you have a .gem file, that has been built by building a gem somewhere, you can take that file and copy it into the bundler cache directory, and bundler will find that there, rather than downloading it from any source

Ooh, so you could [unintelligible] with your dev to your dev gem.

Basically if it's not on RubyGems.org and it's not up on GitHub, and it just doesn’t work with any of those other ways, you can just build the gem directly, put it in your repo, check it in, and it's there and you can deploy it.

Cool.

ANDRÉ: Those are handy. It's actually even possible to do that with your entire bundle if you want to, there's a bundle pack command, that would just download and throw into that bundle cache every single gem that you need to install in the entire app. And I personally don’t do that a lot, but for people who need to be able to ship their app off somewhere else and just have it work, it can be pretty handy.

Yeah, it's great for being able to deploy, knowing that your dependencies will always be satisfiable, that you don’t have to worry about if GitHub down, or rubygems.org is down. But it's also really just handy for doing development when none of those other linking approaches work.

Yeah, I was going to point out that I love doing what Avdi suggested there. But one wrinkle that I run into, is that I end up with a gemfile.mine, and a gemfile.shareable with the team and they are in... And I always forget to swap them when I publish the… And that kind of headache. But yeah, if you can simulate the .gemfile, that's brilliant.

ANDRÉ: So we are actually working on making that less annoying in future version of bundler. One approach that is currently in a path that some people are testing, is basically providing a higher priority local only alternative for a gem, so you can have a gem that’s in your gemfile. But then, tell Bundler about if this directory exists, you should use Git instead of that gem. So in testing, it's incredibly awkward to put that directory into your gemfile as you are pointing out, you end up forgetting to swap it back when you release. Different people have different directories that gem check out might be in. So our preferred approach that I think we are going to try and implement and get some people to test is to allow you to set those alternate gem locations using the bundle config command, so that it's a local setting, rather than a [unintelligible]. So the one caveat there is of course that you will probably do that and then probably forget that you are now using the code that [unintelligible] push to GitHub. And then you'll push your app, and you'll deploy it to production, because everything works so great, and you'll discover that code that you are depending on doesn’t actually exist anywhere except your laptop.

Yeah, that’s why we need continuous integration.

ANDRÉ: Yes, that is a super strong argument for continuous integration. And we will probably also try and come up with some sort of solution that basically, ask bundler to do a quick check and say, “Hey, do you have uncommitted code or code that you haven’t pushed?” Well, you should probably think about the fact that you are committing or you are… by the way, you are actively developing against code that no one else has.

Yeah, to be fair, the existing mechanic also blows up immediately upon publication, because nobody else has a Home-dbrady on their…

ANDRÉ: That’s the upside to the current exclusive configuration option, where they spend a lot of time commenting and uncommenting lines in your gemfile. But, you can be positive that it's not even… you are not going to end up with the wrong code; you are going to end up with no code.

Well, one thing that I've seen, since I've been playing with my Casandra ORM that I've been working on, I set it up to use a relative path instead of Home-Chuck whatever. So it's just ../Sandra/SandraRails. And that actually… I checked that in and then I was working on it on my laptop, and it did kind of bite me, because it's like, “Oh yeah, I got to go update those other repos.” [Chuckles]

ANDRÉ: Exactly, so we are going to try and come up with a way that lets you have the same advantage, but it also gives you a little nudge in the direction of making sure that all your code ends up deployed together.

Right. I think the local.gemfile could actually work really, really well as long as you are careful with your version. Like you bumped the version number in your gem, as soon as you fork not at the end of the fork…

ANDRÉ: And of course, people never do that. There are a vast number of tests in the bundler suite that exists only because of conflicts between gems with the same version number, but there's a Git repo that has this gem with this version number, and there's a published Ruby gem that has this gem with the same version number, but of course, a completely different set of code. And we have to make it so that people get the code that they expect to get, which can be a very tricky problem.

Speaking of tricky problems, one of the things that is often hard to manage when you are dealing with gems is gems that are tricky to build. And it's particularly ones that with native C extensions or some of the database gems needs particular environment settings, so that they can find their dependencies in the whatever their make file world is. And so, can you talk a little bit about what's the best practices for dealing with those sort of things?

ANDRÉ: Sure, absolutely. To be honest, our preferred solution to that is to find a different gem that's easier to compile. [Laughter]

Okay, my first response was to say something that would lose our family-friendly rating. [Laughter]

Bundler’s response is “Why do you keep hitting yourself?”

ANDRÉ: So that is actually a realistic option for say the MySQL gem. Mysql2 gem is so much better at compiling itself, that you can basically stop worrying about that problem where the MySQL gem required basically that everyone who ever installed it, manually tell it where their MySQL headers were, so that it can compile. But, for all the people who don’t have a replacement gem that is smarter about where to look for headers, it's actually really easy. There is a way to tell bundler where your headers are on your machine, that bundler will just then automatically apply every single time they’ve installed that gem moving forward. It's just their bundle config command, has a specific syntax that you can use to set build options. You run ‘bundle config build.(name of the gem)’ then you pass it in the flags that you would normally pass the gem install after the first set of double dashes. So for example, if you were still using the original MySQL gem instead of mysql2, you could just say ‘bundle config build.mysql ---with –mysql –config = (your path to your MySQL config).’ And once you’ve run that command, bundler can then everywhere on your system, in any bundle, handle building the MySQL gem for you.

Nice.

And that ports across different systems? [overlapping talks]

Okay, so it's a local thing, so it doesn’t get checked in to the gemfile or gemfile.lock?

ANDRÉ: Yeah, exactly. Because of course, the MySQL config or whatever headers you need, could be in a different place on every single developer’s machine.

Yeah, most of us develop on a FreeBSD machine, but deployed to a Linux-based machine.

ANDRÉ: Right, exactly.

Is there some sort of workflow pattern that you’ve worked out that makes managing all that stuff across different OSes, easier to deal with?

ANDRÉ: To be perfectly honest, I don’t actually have to set any build options with the gems that I use in my apps. MySQL and the PostgreSQL gem, both actually just build because the headers are in places that they know to look for by default.

Right. They’ve gotten much better now.

ANDRÉ: Yeah, absolutely. So I'm actually happy to report that bundler appears to have had a positive impact on a lot of those gems, because people suddenly having lots of problems getting them installed meant that the gem authors were inundated with requests to make it work better.

I did run into a situation recently where I wasn’t up on the bundle config thing, so I ended up doing like… set up the environment variable that the GCC build needed, bundle install.

ANDRÉ: Uh-huh. That is one way that you can make it happen.

The other thing is that it will find the gem locally if it's there, so one other thing that I've done is I've just done a gem install with the options and then bundled after that, and that works fine too.

ANDRÉ: It does. That doesn’t work as well if you have gem sets limited for every apps, since you need to run that little more every time.

That’s true.

ANDRÉ: But if you have one set of system gems, that is a totally usable way to do it.

“Don’t take away my gems, they’re part of my sanity.”

ANDRÉ: [Chuckles] So obviously as one of the maintainers of bundler, I recommend using bundler config. It's actually pretty straightforward; the description of how to use it is right there at the top, if you run bundle help config. And hopefully, that makes installing those things easier.

So, I have a question that I'm a little curious about. Bundler was written by Carl and Yehuda, and then how did you wind up taking over the development of it?

ANDRÉ: Like I said a little while ago, our initial development was sort of driven by the necessity of Merb having so many gems in such a complicated dependency graph, that human dependency solution was no longer really practical. And so Karl and Yehuda basically wrote bundler way back in the day for Merb, and then updated it, so that it could work with the Rails 3/Merb merge that broke Rails into a lot more gems and made things sort of both easier and harder at the same time, I guess. So I started… this was a while before Rails 3 came out, but when it was in pretty heavy beta testing, I was working at Engine Yard, and I just wanted to try out Rails 3 and build a new app. And so, I sat down and said, “I'm going to try Rails 3. I'm going to try this cool bundle thing and it will be awesome.” And I made it about 15 minutes in before I have caused exceptions in bundler, like 2 or 3 different times. So I patched the exceptions. And later on, I saw Yehuda and I said, “Hey, Yehuda, you should apply these patches that I made to make bundler actually work for me.” And he said, “No, I'm too busy working on Rails 3 right now. I just gave you commit to the bundler repo. You should fix it.”

[Chuckles] Nice.

ANDRÉ: So I fixed it, and a few patches later, I noticed a ticket where someone else has the same problem, so I replied to the ticket and said, “Hey, I just fixed this.” And then Yehuda said, “Hey, I saw that you are handling tickets on the bundler repo. That's awesome. Thanks.” [Laughter] ANDRÉ: And it kind of snowballed from there.

Best PDI ever.

ANDRÉ: Exactly.

PDI?

“Please do investigate.”

or “Please do it.” [Laughter]

ANDRÉ: That’s a pretty good interpretation of that, I think.

This is terrible about the evils about contributing to open source.

We're not bitter.

ANDRÉ: No, not at all. Since there's no money in it, I have to say I just do it for the fame and the girls.

So André, I know Chuck wants to get us to the picks, but I got one last question for you. When do we get our hands on this shiny 1.1 thing?

ANDRÉ: Well, assuming that I can coordinate with Terrence -- which I'm guessing will happen when he wakes up, since he was up until 3 am, committing bug fixes -- I really have my fingers crossed that it will be today. Of course, don’t let that stop you. We already have a slightly buggier version of 1.1 up on RubyGems.org. You can run ‘gem install –pre’ to get that version. And the final with slightly less bugs will be out just as soon as we can get it.

So the geek in me is going, “Today? Today?” And then it's going, “That’s 3 am in what time zone?” ANDRÉ: Uh-huh.

What time zone is the internet in? [Laughter]

Mine.

“Mine? Mine? Mine?”

I have one quick thing which is relevant to the new release coming out, I've seen a lot of people do this and I tend to do this a lot of the time, is I actually put in my gem file, gem bundler and then a version number to say, “Oh this gem file is only going to work if you have atleast this version of bundler.” foo: git (some path to GitHub) -- that is very repetitive. You can now say gem rails:github rails/rails.

Hallelujah!

I'm a fan.

So since we are talking about mild annoyances, a couple of times -- and maybe this is like some older version because it's been a while since I've seen this -- but a couple of times, when I'm doing bundler in non-Rails projects, so I end up making the gemfile, I forget to throw in source:ruby gems…

ANDRÉ: So, there's actually a solution for that. It's the bundle init commit.

Bundle init. Yes. [Overlapping talk]

Gotcha. But it would be kind of nice if source Ruby gems was assumed if there are no sources. Seems kind of logical.

ANDRÉ: The problem is, if you don’t put any source lines in, there are still some sources. Those sources are in order that they are checked. The system gems installed on your machine, and the vendor cache directory.

I see.

ANDRÉ: There are – surprisingly enough – at least very vocal people, I don’t know what percentage usage they are, but there are definitely people who want to be able to not use Ruby gems. And so, that’s why we have bundle init commit.

Yeah, but I still feel like if they are the exception rather than the rule, that there should be like unsource line, so you can unsource it.

Yeah, or they should have to pass a flag or something.

ANDRÉ: Sure. I’d be open to investigating something along those lines. Back in the day, it didn’t seem like a strong of a concern, since we are worried about people actually getting the gems that they asked for.

André, this is your opportunity to PDI James.

ANDRÉ: There we go, yes James. That would be wonderful to investigate.

Cautionary tales.

Yeah, you might get bundled in the bundler.

[Chuckles] André, can we finish up about the putting the gem bundle whatever in the gem file, and is there downside to that or is there anything you got to watch out for?

ANDRÉ: Yeah, well I mean the downside is that you'll have to manually allow bundler 1.1, right? But it's a perfectly reasonable way to make sure that people know which version of bundler they need to install a project.

But there's no conflict with bundler falling down in infinite tower of turtles or anything like that?

ANDRÉ: No, not at all. There's a special case code to make sure that bundler simply checks to make sure that it itself meets that requirement.

And if it doesn’t, does it look if there's an older version installed on the machine, or does it just kind of give up?

ANDRÉ: It tells you that the version that you have run cannot successfully install the gem file and leaves it up to you to find the right version and run that one instead.

Right. So then you have to go dig in to your gem sources and look at the bin folder for that?

ANDRÉ: No, you run bundle_version_(the correct version). Do you not know about… Ruby gems has this cool…

A what?

[Chuckles] Yeah, that's what I was…

ANDRÉ: …has this very interesting feature where if you put a version number in underscores after the name of the command, it will use that version of the gem.

So you could do bundle_1_0?

It's _1.0_

ANDRÉ: Right. And then you would get the 1.0 release. So right now, you would wanna say something like, bundle_1.0.22_install.

Can we put an example of that in the show notes, because I'm lost.

Yeah, I think I just heard David’s head break.

Yeah.

It actually works for any Ruby gems. It's a Ruby gems feature. I used to run to it all the time with Capistrano. I used to have certain old apps that would only deploy with certain versions of Capistrano, so then I would do cap_ and whichever weird version that particular app needed, and then the command.

It seems that would make a lot of sense as an environment variable, that you can set on before the command.

ANDRÉ: It seems like putting it as the first argument with underscores is just a really weird way to do it. [Laughter]

There's absolutely nothing weird about Ruby gems.

ANDRÉ: [Laughs]

So with this feature, I have to ask one more question, that is if you don’t know the full version, like 1.0.22, if you just know that you just want the latest 1.0 version, can you do that? Can you do 1.0_...

ANDRÉ: I think it will tell you that 1.0 isn’t installed.

It has to be an exact version. And you can get all the versions installed just using gem list, right? You can get the versions that you have installed.

You mean there's no ‘under wakka’?

[Laughs] I'm trying to figure out what that would look like ~_~?

Yeah, we've made the new operator for Ruby gems.

Have we been getting complaints?

No. [Chuckles] In fact, when we were keeping it to an hour, I would get complaints sometimes saying, “You cut off the conversation!” but anyway. So picks, let’s start with Josh this week.

Okay, I have a quick pick and it's part of bundler. So if you go in your project and you type bundle viz, and you have graphviz installed, you get a pretty picture of your dependency tree.

V-i-z, right?

Yeah, viz. And the great thing about that is you don’t have to look at… if you look at the gemfile.lock, you get this indented tree structure. It's not actually a tree, it's a [unintelligible] of graph. You hope it's async. [Chuckles] Anyway, it's really a graph, but it's laid out in an outline, so it's much nicer to look at it graphically as the tree. So there you go. It was a tough week for me, so it's all I got.

Alright, Avdi what are your picks?

So I am deep in eBook generation mode this week. So my picks are both related to that. Couple of tools that I find handy; number one, I just started using… I've known about it for a while, I just started using it is pandoc. Pandoc is a utility written in Haskell, that converts between various document formats. And it supports just about anything you can throw at it. I'm not going to list them all off, but just as an example, I had a desire to incorporate some markdown sections in the objects on Rails book, but I write most of my stuff in Emacs org mode, so I added like 3 or 4 lines to my make file, that just said, “If you have a markdown file in the list of sections, run it through pandoc.” And pandoc takes that and generates a nice org mode file out of the markdown file. And it's pretty much just does what you want it to do; you feed it a format and an output format, and it figures out how to make it happen. The other one is Calibre, which is like this whole Swiss army knife tool set for reading and converting and managing eBooks. And it's not like a whole section of it is about just like managing your eBook collection and reading them and stuff like that, but it also comes with some great command line tools for from one format to another. And I use that a lot, and it's very, very helpful in my tool chain. So Calibre.

Alright, David what are your picks?

I just have one today and that’s The Developer’s Code by Ka Wai Cheung. You can get it from the Pragmatic Programmers. This book feels kind of like a new version of the Pragmatic Programmer. It's 150 pages of just one tip after another tip, after another tip of how to be a programmer, how to be a developer. And it starts with real simple technical things like how to keep a to do list, and how to set your pace and how to hack your own wetware. But then there's a whole chapter in complexity. How to smell it, how to reduce it. Towards the end, it starts getting on to career advice like how to deal with tough customers and understand the difference between features that are nice to have but will be horrible to write. I'm just really enjoying it. I picked it up last thanks giving. They do the turkey day sale, and I basically just swept my entire wish list into a shopping cart and bought it. And the dead tree version just came this week. And I'm really enjoying it.

Alright, James what are your picks?

So my picks recently have been like all series useful stuff, and that’s really starting to bother me. So I have that reputation to maintain. So, in the spirit of none useful, everybody knows I'm a huge fan of Song of Ice and Fire, George Martine series. Yesterday, aside from being super Tuesday – which who cares – the important thing happened was Game of Thrones, HBO’s series based on that book was released on DVD and Blu-ray. So you can now get that series and start watching it. Obviously, I haven’t watched it all yet since it just came out yesterday, but it's extremely highly rated, George R. Martin is one of the executive producers for the series, and he directs I think, one of the episodes and stuff. It's supposed to be very loyal, so I'm looking forward to that. And also I discovered some cool music I'm liking Glenn Kotche, he's like the drummer for Wilco, but he has his own album, which is kind of this really cool percussion piece. And it's just all kinds of great percussion stuff. Really good if you listen to it with headphones; it's got a lot of subtlety to it, and that makes for cool coding music. So those are what I'm doing for fun right now.

Cool. I guess I'll go next. So my wife got a fit bit for Christmas, and she’s really been enjoying it. Every time she’d use it, I get a little jealous because I thought it was really cool. And we've been doing this weight loss thing and I've already lost almost 10 pounds. So anyway, I kept wanting one, and I wound up getting her something really nice for her birthday, and I guess she decided that that entitled to get me something nice for our anniversary, which was last weekend. So she got me a fit bit, and I've been playing with it and using it. And it is really kind of a cool device. It basically will track how active you are, how many steps you’ve taken. For example, this morning, I got up, walked in to the bathroom, did what I needed to do in there. And I checked it and I have taken like 280 steps from my bed to my bathroom.

How many calories did you burn in the bathroom?

I just wanted to know, does he go to the bathroom in his neighbor’s house or what?

[Chuckles] I was wondering the same thing when I looked at it.

Did you get lost on the way?

“Where did these groceries come from?”

I thought I had a long hallway. Man. [Chuckles]

Right. Are you skipping the blackout period that happened in there?

[Chuckles] Yeah, went downstairs, got a snack. But anyway, it's really cool. It tracks all that stuff. You can also use it to track your food intake. Not the device, but the website. And so it's pretty nice and it just syncs up through your computer.

Hey chuck, my pick last week about like the Wi-Fi scale…

Yeah, I want one of those.

So if you get the scale, the software can talk to the fit bit data, and you can track your weight against your activity.

Oh, interesting.

So you have integration there.

Yeah, I tried to order one, but they were out of stock everywhere I looked. But yeah, that’s my pick. André?

ANDRÉ: This was not one of my picks, but I use fit bit too, and I think it's totally awesome. My picks are a little bit across the board. My first pick is also part of bundler, although we didn’t actually mentioned it today, and it's the bundle gem command. Basically, what that does is it sets you up a skeleton Ruby gem with a gem stack a lib directory, and a gem spec that is almost entirely filled out. But I have found that just ridiculously useful when I want to take some code that I have that is a library and turn it into a gem, I can basically just run bundle gem, copy the Ruby file over and then fill in new fields, and I have a completely working gem that I can then add to the gemfile and whatever it was that I just extracted that gem out of. That’s very handy. My sort of technically related, but for fun pick is the Hypercritical Podcast. Maybe I'll get in trouble for mentioning another podcast on Ruby Rogues.

No, we do that all the time.

Yeah.

ANDRÉ: Along with you guys, hypercritical is a five by five podcast with John Siracusa, who is somewhat infamous for writing ridiculously technically detailed reviews of each version of OS X as it comes out.

Which are amazingly great to read.

ANDRÉ: They are amazingly great. And I love hypercritical because it's like getting those amazingly great reviews, but for lots of other technical fix. There's an episode about televisions and DVRs that’s pretty amazing. There's a more recent episode about file systems. And I just really love having someone else who is interested in all those like technical nit-picky details who talk about them. And then my last pick is my coding music. I figured I should throw that in. Lately, what I've been doing is I've been listening to JWZ’s mix tapes, which JWZ used to work for Netscape, and retired to start his own nightclub. But one of the cool things that he does at that intersection of nightclub and technology is create mix tapes of obscure, but interesting music, and he releases theme very couple of weeks. But that’s mostly what's I've been listening to when I program, because it's sort of continuous stream of interesting and new music that I've almost entirely never heard of before.

Some people when faced with a problem say, “I know, I'll find some good music.” Now they have a problem a mix tape.

ANDRÉ: [Chuckles]

[Chuckles] Yeah, one thing I noticed while you were talking, it just occurred to me, I saw your last name ‘Arko’, and I realized it's one letter off from character from He-man called Orko. ANDRÉ: [Chuckles]

I don’t know why my brain made that connection, but anyway.

Chuck, as someone with a potentially funny name, you shouldn’t poke at others.

I know. [Laughter]

No, you got to be you, chuck. You either don’t stones or throw them at everybody. You know what I mean?

[Chuckles] So real quick just to wrap things up. We are in iTunes. If you’ve been listening, go ahead and go in to iTunes and subscribe. And if you are there and you wanna leave us a review or a rating, that would be terrific. It really helps us out to get the word out and stuff. Also, if you are listening on some other device, you can always go to rubyrogues.com and click on the subscribe link, and that should pull it in whatever pod catcher you are using like on the android phone or whatever. I just wanna thank André for coming on to the podcast again.

ANDRÉ: Yeah, absolutely. It's been great.

Thanks, André.

We are reading Crafting Rails Applications. And I have to say that I've spent about 45 minutes reading it and I'm already a quarter of the way through it. So it's a pretty quick read, but really interesting stuff if you wanted to know how the internals of Rails work. So go pick that up, we'll be talking to José here in like three weeks.

Do we have a discount code for the readers?

Not at this time.

Okay. It's worth it.

And Chuck, the date on that is March 28th we are doing the episode.

So that is in three weeks.

ANDRÉ: Cool. I'm looking forward to it.

And I just wanna let the readers know, if you haven’t read the book, you are not allowed to listen to the episode yet.

Ooh! Are we going to do spoilers?

[Chuckles] Yeah. We'll give away the ending.

The bundler did it.

Oooh!

That was pretty good. It was on topic. [Laughter]

Yeah, but the girl, she marries the vampire.

Oh, but he sparkles, right?

Yeah, they just put glitter. Anyway…

[Chuckles] On that note…

Anyway, we'll catch you next week. Thanks!

Bye!

Bye.

ANDRÉ: Bye, guys.