The panelists discuss changes in the npm package manager with Forrest Norvell, Rebecca Turner, Ben Coe, and Isaac Z. Schlueter.

Special Guests: Ben Coe, Forrest Norvell, Isaac Schleuter, and Rebecca Turner.

 

  There can be only one.

  Hey everybody and welcome to episode 127 of the JavaScript Jabber Show. This week on our panel, we have Joe Eames.

  Hey there.

  Dave Smith.

  Greetings.

  Jamison Dance.

  Hello friends.

  I’m Charles Max Wood from DevChat.TV. And this week we also have as guests several people from npm. We have Isaac Schlueter.

  Hello.

  Ben Coe.

  Yeah, hello.

  Rebecca, I don’t see a last name.

  Rebecca Turner. Hi.

  And Forrest…

  Forrest Norvell, that’s me. Hey, everybody.

  Alright. Well, you guys want to introduce yourselves really quickly? We’ve had Isaac on the show, but I don’t think we’ve had everybody else.

  I’ll go first. This is Isaac. You’ve talked to me before. I am the isaacs of npm and Node. So, I think probably most people know who I am. We’ll pass the mic to Ben Coe.

  Hey, I’m Ben Coe as Isaac says. And I am part of the ops team at npm. And most recently I’ve been a lot of work on our npm Enterprise, which is our first paid offering that we’re putting out.

  I will go next. I’m Forrest Norvell. My official title is ‘very good developer in charge of the npm command line interface’. And I’ve been doing a bunch of work for npm 2, which is going to be released this week. And I will hand off the mic to my esteemed colleague, Rebecca Turner. 

  Hi. Yeah, so I joined npm very recently to help Forrest help with working on the command line interface. And we’ve got lots, a major refactor coming, so excited about that. Really help a lot of the issues we’ve had.

  We love people who help issues. 

  Issues, woohoo!

  So many issues.

  [Chuckles] So many issues.

  Forrest, oh I just wanted to say I’m excited to know you’re a real person and not a red pony.

  I can be both.

  Because that’s the only way I’ve ever [interactively]…

  [Laughs]

  Seen you on GitHub.

  Yes.

  His avatar is remarkably, it’s a pretty striking resemblance actually. [Chuckles]

  Oh. So, npm 2, huh?

  Oh yeah. Who wants to know more about npm 2? So, the first thing to know about npm 2 is that it actually is not a big, huge deal. I know it’s been several years since npm 1 came out. And so, it’s natural to look at a major version bump as being a fairly striking change. The most important thing to know is that one of the things that we are trying to get right with npm 2 is how we deal with and advocate SemVer itself. There have been some exciting conversations about SemVer recently. But predating all of that, we realized that we were going to make some breaking changes with npm 2 so that justified the major version bump. And we ended up piling about three or four major breaking changes into one release, which we probably will try to avoid in the future. 

So, even though it took several years between npm 1 and npm 2, it’s going to be much, much, much faster on the order or probably weeks or maybe months at the outside before we make it to npm 3. What are those major breaking changes? Well, the most significant one is a feature that we added both to support npm Enterprise and future changes we’re going to make to the registry, which is scoped packages. So, in a way, similar to the way that you could have repositories scoped to a user or organization on GitHub, we are now offering the ability to scope the packages to a given organization. And I should point out this actually is only a forward-compatible change. This isn’t a backwards-compatible change. But there are a number of semantics about how it deals with the registry that need to be supported as well. So…

  Can I stop you for a minute?

  Sure.

  So, you said that npm 2 is faster than npm 1?

  I did not say that, I believe. [Laughs]

  I can say that. I’m an executive now. I can say anything. Yeah, it’s way faster. It’s better. It will make you more attractive.

  [Laughs]

  So, it’s actually likely to be slightly slower, but that is because of one of its other very important features, which is that it should be significantly more stable, particularly in a number of use cases when you’re driving it from other tools. 

So, one of the big use cases that’s relevant to more client side developers is that there are now quite a few tools that drive npm from the command line interface. And that includes Yeoman. It includes the Ember CLI. In certain circumstances it even includes Gulp, Grunt, and Bower. So, because those are driving it programmatically, they tend to be pushing everything a lot harder than when you’re just using npm on the command line. And we’ve run into a whole host of basically synchronization issues, race conditions, deadlocks, locking problems in general. And we put a lot of work into improving that story in npm 2. 

Again, that’s not a backwards-compatible change. The one change that is breaking backwards compatibility I will actually allow Ben to speak to because he’s the person who actually added that feature. Do you want to pick that up, Ben?

  Yeah, sure. So, we’ve been working a lot with npme trying to make it easier to just manage scripts with npm. So basically, we’ve written this little tool called ndm which generates upstart scripts for various operating systems from your package.json. And one of the main improvements we needed to make this easier was to be able to pass arguments into your script stanza inside npm. Before this, you could have a test stanza inside scripts, but there was no way to pass any arguments to it. So, the [awful] breaking change is that you can now, if you’re using something like lab, you can pass the grep flag into it and actually run a specific test. Unfortunately, due to the way our CLI parsing worked, this caused a breaking change. But it gives the script tag a lot more power in npm, which is cool. 

  Very cool.

  Awesome. Hey, just really quick. I think it might be useful for people out there that aren’t necessarily familiar with SemVer to have somebody explain SemVer, since that was the whole point behind making this 2.0, right?

  Controversy, controversy.

  Oh yeah, we would love to do that. Yes, this is controversial. [Laughs]

  I didn’t think it would be, but it is. [Laughs]

  You know, it’s any place humans and computers interact with other humans and computers, there’s bound to be controversy. 

So, basically SemVer is a distillation of some relatively general best practices and common expectations. The way that that ended up culminating in a specification which I think historically was one of the best things to happen to software versioning in a really long time and I appreciate this occurring, Tom Preston-Werner of GitHub wrote up more or less a manifesto. And then over time that got hammered, the raw stuff of that manifesto ended up getting hammered into an actual specification with musts and shoulds and mays and stuff. And so, npm was actually one of the first package systems, package ecosystems, that came to be around the time that SemVer was starting to be a popular thing. 

So, the basic idea of SemVer is every time that you have, your version is a three-tuple of numbers. So, it’s major version, minor version, and patch version, separated by periods. You bump the major version every time there’s a breaking change. You bump the minor version every time there’s an additive change. And you bump the patch version for changes that fix things without changing the API. So, the basic idea there is that you can guarantee. Okay, if I need something that was added in 1.4, then I should be able to work with 1.5. But 2.0 might break me. So, in order to upgrade from 1.4 to 2.0, I need to do some extra work. And if I’ve tested with 1.4.5, well 1.4.4 might have a bug. So, I want to require at least 1.4.5. That’s the one I’ve tested with. 

Now, on top of that, there are all kinds of other junk, like how do we handle 0.x.y versions? How do we handle 0.0.x versions? How do we handle versions that had what’s called a prerelease flag? So, you can do something like 2.1.2-alpha. And then that -alpha says, “Yeah, this is probably going to be what goes out in that version, but it’s not quite ready. So, play with it, give me feedback. But it might be broken.” And so, there’s been quite a bit of discussion and back and forth on that. I wrote the module node-semver, which is the main thing that people use, certainly the thing that npm uses, for parsing version numbers. 

And so, node-semver has been bumped up to, and node-semver adds the concept of ranges which are not actually specified in the SemVer.org specification but are really crucial to how people use npm and use versions and dependencies in npm. One of the points of controversy there is the caret operator, which is like Shift-6, the little hat on the US keyboard. And what that means is if I do ^1.2.3, that means go ahead and allow any additive or patch changes starting from version 1.2.3. According to, and then this gets confusing when it’s something like 0.1.2. Should I be allowed to pull in 0.2.3 or no? Or should I be allowed to pull in 0.1.3 or no? 

So, on one side of the spectrum you have people who say, like the spec says actually, any version that starts with a 0 is no-holds-barred, anything goes, non-semantic versioning. So, between 0.1.7 and 0.1.8 you could completely rewrite the API. On the other side of things, you have people who say 0 shouldn’t be magical. The major version and the minor version and the patch version mean what they mean and it should just always be the same. So, between 0.0.2 and 0.0.3, you should only be able to add patch changes. And then there’s this middle ground, which is the leftmost nonzero number is the major version. And then the next number is the minor version, and the next number is the patch version. So, 0.1.2 is like major version 1, minor version 2. So, between 0.1.2 and 0.2.0 you can have breaking changes. And between 0.1.2 and 0.1.3 you can have additive changes. 

So, we introduced this caret operator to behave according to that middle ground. And in semver 3.0, so it was brought to our attention, your docs say that the caret operator is, follow the spec but it actually doesn’t follow the spec because 0.x versions are not tied to a particular release. They’re allowed to upgrade. So, we said okay, that’s kind of interesting. That’s a fair complaint. So, it’s a breaking change in node-semver to make that change. So, let’s bump it from semver 2 to semver 3 and try and land that in npm. 

What we found almost immediately is that that just is terrible. [Chuckles] There are a huge number of people who are using, for example in Gulp or sorry in Grunt, though Gulp is at a pretty high major version by now I believe. But in Grunt, there are a lot of people who use the 0.4.4 and 0.4.5

and are also using peer dependencies quite a bit. So, what you have is a situation where if one Grunt plugin depends on ^0.4.5 and another Grunt plugin depends on ^0.4.4, then because they’re using a peer dependency on Grunt they would both collide with one another. And then this caused all kinds of havoc. So, we rolled that back in semver 4. 

Another thing that had been a longstanding point of contention and confusion is that if I publish a prerelease version of something, like for example 2.0.0-alpha.1 or something, then that will be used, according to the SemVer spec that is less than 2.0.0 because it’s a prerelease. So, it’s earlier. But it’s also greater than 1.0.0. So, if I had a version range like greater than or equal to 1.0.0, less than 2.0.0, let’s say that 2.0 is going to break me. Well, that would actually pull in the prerelease because it’s less than 2.0.0. And there’s a bunch of other cases where even a prerelease of a version that I might be able to do, the prerelease itself might break me. So, I want to make sure I don’t pull in those betas without having some understanding of it. 

And so, that’s why… anyway, long story short, in semver 4.0, prerelease versions don’t match ranges unless you explicitly say greater than or equal to 1.2.3-alpha. Then we’ll say, okay we allow prerelease versions on that one particular tuple. But 1.5.0-alpha won’t match that ever. Long story short, it’s all documented in the node-semver package. And so, that’s where people should go to see how SemVer works in npm.

  So, which version of SemVer does node-semver use?

  That’s so meta. 

  I know. [Laughter]

  When it comes to the specification versions, it’s going based on the SemVer 2.0.0 specification, which is the current one on SemVer.org. 

  How many SemVer specifications are there?

  Well, there was a SemVer 1.0 spec and then a SemVer 2 spec. 

  There are actually a few more versions than that if you go to SemVer.org. But they were mostly, yeah that’s right, they were release candidates for 2.0.0, which means that according to nodesemver 4, those would not have showed up as acceptable matches for anything trying to match on 2.0.0. [Laughter]

  I’ve talked to Tom myself about this. And it was always meant by him just to be fairly loose guidelines. So, I think the fact that people are arguing so much about the specifics is interesting. [Laughter]

  That is a good way to put it.

  Yeah.

  I think it’s very clear, if you look at the history of this, how it ended up coming to be a spec and how much people have gotten very extremely opinionated about it, myself included. I think that Tom had, I don’t want to say ridiculously unrealistic but mildly fantastical ideas about how easily people would get along with one another. This is the point where it’s the point of interface. It’s the friction point. And so, it’s going to draw a lot of that conflict and controversy.

  So, is the takeaway to never have a 0.whatever of any product you make?

  Yes. That is absolutely the takeaway.

  Well, alright. So, that’s a glib answer. The real answer is that you are free to use 0.x.y versions for projects that you feel aren’t quite ready to go yet. I will say that the npm community’s understanding of what that means is probably not exactly in sync with the reality portrayed by the SemVer standard. We had a long conversation about this because we actually did make an effort to change the spec to match our understanding of it. And really, I think where that ended up was npm looked at the world and we looked at how other communities are doing this. And all the communities using SemVer deviate from it in ways large and small. 

And as a person who is ultimately answerable to the community of users to npm, I was just like, “I have to be pragmatic about this.” I need to make sure that I am not breaking the community and causing everybody who’s actually using peer dependencies for plugin ecosystems like Grunt to have to chase down all the developers who’ve ever written plugins that use the caret operator and get them to change that so that people can still follow their workflows. I don’t think it’s reasonable to put the cost for that on the end users of the product. 

So, the argument to pragmatism was just like, “Well, let’s try as much as possible to make nodesemver work the way that our community expects it to and be more descriptive rather than prescriptive or normative.” Rather than say, “You have to do this if you want to be playing in the npm pool,” we instead say, “Let’s just redraw the lines of what the pool is to make things keep working while at the same time gently shepherding people onto the path.” 

And really, semver 4 still has an issue which is that all those restrictions around prerelease versions mean that people who are trying to couple version numbers, we’ve seen this already with PhantomJS and a couple of other packages where it’s like they’ve got one version number for the package, dash, and then the version number for the upstream package. They may not actually have a non-prerelease version in the npm registry. So, we still have to do some evangelization to people who are maintaining some of these packages and be like, “Hey, people use your package. That’s awesome. But the new version of npm is out and it’s going to cause all of this stuff to break for people who are depending on your package. Can we get you to publish a version of it that’s not prerelease?” 

So, pretty much everything that we do with an ecosystem as large and complex and braided with historical baggage as npm, there’s just no way to do anything without moving somebody’s cheese. But really, the goal of all this was to try and square up existing practice with what we consider to be a fairly rough idea of best practices.

  So, that gets at a larger issue which is when you have, this applies to everyone not just npm, when you have an existing user base using your service, how do you balance breaking things for some people versus making things better for everyone?

  I believe, and I think Isaac will back me up on this, that that is a judgment call. You have to have a fairly good idea of what the community’s practices are when possible. This is a tenet of product management. You have to drive things to numbers. You have to be able to say, “Well, when we look at all of the packages on the registry and we look at all the version numbers they’re using and all the version numbers that they’re depending on, we can see that this is the predominant pattern.” And then that ultimately feeds into what are going to be subjective judgment calls, right? You basically have to be able to justify what you’re doing to your user community. 

And if we work on npm – and I think this is probably true for most of the really popular package managers, I’m sure it’s true for Bundler, I’m sure it’s true for PyPI as well – you have immediate and dramatic feedback from your user base when you do something that is either surprising or unpleasant to them. My bête noire is the npm issue tracker. Last night, I was totally excited because a bunch of the reliability fixes in npm 2 allow me to go and just close out a huge swath of bugs. And I was just so excited when I went to bed last night because I’d managed to get the number of open issues on the issue tracker under 1100. I got to 1099 and I’m like, “This has been a good day. I’m going to bed.”

  [Chuckles]

  So, that is, you get voluminous, detailed, extremely finicky feedback from people. So, you have to have solid answers. And so, you have to be able to say, “Yes. This is worth the pain that I am causing you as an end user of my product,” and your dependents and the people who are in your community. And because we are working on this as an open source product, because we are enmeshed within this very active community which is the larger Node ecosystem, and because people are completely unafraid to just reach out and grab us on IRC and on Twitter and on GitHub itself, that means that we get a level and a volume of product feedback that I think a lot of closed source companies would kill to get. 

And occasionally, we are going to have to do things that are going to just unavoidably are just going to completely cause really major pain for people. And at that point, it’s just like, “We got to do it.” Sometimes we have to make these changes to make for a better user experience for everybody. And in my experience, while people will grumble about it sometimes for years afterwards, for the most part people are just like, “Okay. That makes sense.” As long as you have an explanation that’s not just, “Oh, because I thought it should work that way.”

  That makes sense.

  Also, that sounds like how most politicians work.

  [Laughs]

  I spend a lot of time when I’m talking to people who are getting involved in open source politics, not politics, but just open source in general, telling [inaudible]

  [Laughs]

  Well, that’s the thing. It is political. Ultimately, when you look at how a large and popular project goes, when I’m talking to people and they’re coming to me with something that they want to see added to npm, I’m like, “Okay, so I’m not going to say yes or no to that. But you have to make a case to me for this. And can we pilot this outside of npm itself or do we need to do it in the codebase?” And basically, this is not an idea that I came up with on my own. 

This is actually something that I learned when I was reading about the history of the New Deal. FDR was, he was elected under the promise that he was going to do all of these progressive things. And then he got into office and he was suddenly the chief executive for the United States. And it didn’t matter that he was the president if he couldn’t get stuff through Congress. So basically, he had to say to his constituency, “Hey, you want to see these things happen? You need to convince your representatives. You need to convince me. You need to make a case for it.” And that is very true also of open source.

We can wield power as maintainers of these projects. As the leaders of npm, we have a lot of power, as the leaders of Node. But that power only exists as long as we have the faith and trust of our user base. And so, really what we need to be doing is to be acting transparently and accountably enough that people continue to believe that we have their best interest at heart. They can go fork it. It’s open source. You can build your own version. There are probably five or six forks of npm out there and there are several different registry codebases out there as well. Our power results ultimately from the fact that we have the best product. We have the best product because ultimately we have the faith and trust of the user community. And we have that because we have earned it. And it’s easily lost and it’s hard to build. But it ultimately is a political process.

  Alright. I want to change topics a little bit. You mentioned briefly some of the new features that were coming in npm. Can we talk a little bit more about that? For example, scoped packages is one that was on the list.

  Yeah, sure. So, when we started this company actually, this goes back quite a while, part of the reason for deciding to do a startup around npm was that there were a bunch of features that companies and enterprise users of npm were desperately asking for. And when you have a situation where somebody’s pounding on your door with fistfuls of money, you want to figure out how to let them in, right? 

So, the thing that most people really want is they want to be able to use, and then what they want to pay for which is also important, is that they want to be able to use npm to manage private code. And so, if we want to have the npm registry be sustainable, it has to pay its own way. And one way to do that is to figure out how to deliver on a solution for managing my company’s private code using the same techniques and systems that developers are using for public, open source npm code.

So, the way that we thought through how to do this, so if you just have a private registry that’s running a local registry, or a private registry that’s running in your data center, or not, or elsewhere, you’re inevitably going to run into situations where you have a package name that conflicts with something in the upstream public registry. And it’s not that somebody did this to try and screw you over. They didn’t know that you published something private. That’s how private works. 

But you can get into a situation where just to pick one random example, let’s say I have something privately that’s called connect. And then I decide later, “Oh I want to start using Express on this new web app.” Well, Express depends on a module called connect. But instead of pulling down the public registry’s connect module, it’s going to pull down my private connect thing, which is something completely different. And then that’s not going to work, obviously. So, they typical way to work around this is to invent a bunch of crazy namespace stuff and put hyphens in front of everything. And it’s clumsy and a little bit weird. 

So, what we thought through how to do and if we’re going to have eventually private modules that are hosted by us, that are in the public registry, then we need to have a way to keep them somewhat hidden and also keep them from colliding with anything else. So, we came up with this idea of namespacing based on a scope. A scope is basically an @ sign and then a name and a forward slash. What that does is by having @mycompany/foo as the module name, I still can type npm install @myco/foo and it will work on the command line. I can put it in a package.json, so I depend on @myco/foo version 1.2.3, semver see previous drama. And then we can also have that as the URL on the registry. And then because the @ sign and the forward slash are not currently allowed, there’s no way that that’s going to collide with anything in a backwards-incompatible way. And that gives us a hook to start saying, “Okay everything that’s under this namespace is owned by this group. And they have control over it.” loves:

registry.npmjs.org. And then the other one would be registry.mycompany.local. And so, I could type on the command line npm login --registry=mycompany.com –scope=@myco. And now from that point forward, I supply my login credentials for my enterprise registry and from that point forward, everything that I’m doing on that scope is going to the internal registry. And everything that I’m doing that’s not on that scope is going to the public registry. 

So, I can be developing on open source code at my company. For example, there’s a lot of, PayPal has Kraken and Walmart has Happy. There’s a bunch of these companies which are getting involved in open source in a real way. But they still have private stuff that they want to keep behind the firewall and behind that curtain. And so, developers can very seamlessly just switch from one project to another without having to juggle our C files or do anything clumsy and crazy. It’s just going to automatically send the right things to the right places. 

And if you try to publish something with @myco/foo and let’s say you haven’t done the login, if you try to publish that to the public registry, it’s actually going to fail because we can detect, “Hey, this is not allowed. Get out.” And so, you don’t have the risk of accidentally publishing something which is private, which has happened a few times. We usually get a somewhat panicked email from somebody at some company saying, “Hey, can you please delete all copies of this?” So yeah, that’s it in a nutshell. You stand it up and you could keep using open source code and you could have all your private code internally. And so, the namespace-scoped modules is the way that we’ve been able to actually deliver on this. 

In order to justify paying for npm Enterprise, there are a couple of other things. You can actually have it synchronize down a whitelist of modules. So, your build script doesn’t have to go across the internet in order to pull down code. And you can also verify that things have been vetted, that legal and InfoSec have signed off on it or whatever the requirements are at different companies. Another big request we were getting from people who are using npm at their companies and their enterprise apps but they want to have a little bit more control over how it works.

  I think it’s worth mentioning too that this is out the door. You can try this out right now. Anyone can go to npmjs.org/enterprise and get a free trial to just play around with scoped modules.

  One thing that I noticed, Ben copied and pasted the npm login deal, if you have your own registry or you’re going to use another registry, is there any way that you can set that up so that it just automatically does the login stuff? Or automatically uses your own registry maybe without having to have the scope?

  You can be logged into multiple registries simultaneously now. The implementation of that basically maps your credentials to the actual registry URL. So, once you’ve logged into a given registry, instead of what you had to do with old versions of npm where you would log in to a registry, do stuff within and log in to another, so if you’re moving between the main registry and your private registry you’d have to keep switching off. And there are tools like npmrc to manage that for you. Instead now, you just go ahead and login to all the different registries that you have credentials on and then go about your business. The extra step that’s added on top of that is that you can also, when you’re logging in, provide a scope. And then any attempts to install packages that are part of that scope will go to that registry by default and will also use that registry’s credentials.

  So, does this persist past a reboot?

  Yeah.

  Yes.

  It ends up in your npmrc. And so, you only have to do the login step once. And from that point forward, your npmrc will have that registry associated with the private npm you’re running.

  And one little interesting behind the scenes detail on that is that with npm Enterprise, we’re introducing the notion of token-based authentication. Previously, you passed your username and your password with every authenticated call like publishing or whatever. And now, we’re just using a token-based auth for all of that stuff, which when we get it rolled out on the primary registry will be much simpler, much more secure than the old way of just having your credentials’ base64 encoded in your configuration file.

  Yeah, I was going to ask about that. If somebody steals your npmrc can they get in?

  Well, that has always been true and that will continue to be true until such a point as we add the notion of token revocation for token-based auth. 

  Okay.

  Well having said that, with the way npm Enterprise works right now, you authenticate through your GitHub Enterprise or your GitHub account. And it actually assigns you a token through the GitHub appliance. So, you can if you say had your npmrc stolen, you could go to your settings in GitHub and revoke the token from the application settings. So, if you think your npmrc’s been stolen, there is a way to revoke it, but you do it through GitHub, not through npm. 

  We’re working on adding other auth systems that will enable people using things other than GitHub in the future. That’s just the most common one that we heard from most of the people we were talking about. 

  Okay.

  So, I would like to talk more about the future, because even though a lot of these things are new to people, they’re, I don’t know, I think to us they’re feeling like they’re already done and in the past.

So…

  Rebecca, do you want to talk about the changes coming in npm or the problems that we’re trying to address?

  Sure. The core of this is the multistage install project, which you can see on the issue tracker it’s one of our milestones. And while that’s what we’ve been calling it, it’s really a change to anything that mutates the Node modules folder. And so, this is going to be npm install and npm remove and dedupe. And this is, currently it goes through, as it goes it’s doing it in the most obvious way right now. And the multistage install project is going to make it so that it does all the dependency resolution first. And then it does all of the downloads, and then it does all of the preinstalls. 

And by separating these pieces out, it allows us to, well it removes a lot of the race conditions that we had previously in the codebase. And this is what we were fighting with to get 2.0 out, was to fix a lot of the race conditions that were frankly hard to track down because the old code flow was so twisty. So, at the same time this is going to address, we’re hoping to address a lot of the issues that people have been having with peer dependencies. And npm-shrinkwrap is another major piece of this.

  Can you talk about both of those and maybe explain what they are? I’ve heard a lot about peer dependencies and I’ve heard a lot of people complain about them.

  Sure.

  But I haven’t used them very much.

  So, peer dependencies are a little weird, because what you say is, “Anything that requires me also requires this other thing.” So, it’s almost a way of injecting a dependency into the thing that uses your module. And so, on the surface…

  So, why wouldn’t you just require, why wouldn’t you have that as a regular dependency?

  So, on the surface that sounds a little strange. But this is really important if you’re writing, it’s used in plugin ecosystems. So, if you’re using, the most obvious place would be things like Grunt and Gulp. They have large plugin ecosystems. So, their plugins go, “I don’t personally use this module in any way, but in order to use my module it needs to be a part of this other thing.” And this is why they don’t include that module as a dependency. It’s because they don’t actually directly use that module, but they’re only useful in the context of that module. And so, they say, “Well, I designed this plugin to work with the following versions of the core parent module.” And so, the peer dependency is a way of specifying that. 

And it’s caused problems because of course if you’re very specific about the versions you work with, it means, “Oh, well I’ve got two plugins on my project and they want different versions of the parent module. And now everything’s going to break.” And it’s just a confusing form of dependency compared to all the other dependencies or child dependencies. 

  Well, yeah. One of the main goals of npm is to avoid dependency hell, which is finding yourself in a situation where you’ve depended on things to the extent that the dependency constraints are just not satisfiable and you can’t understand why things are broken. And one of the main problems of peer dependency is that it makes it very easy for you to put yourself back into dependency hell. So, above and beyond the work that Rebecca is doing to create the multistage install process, we also have another item on our longer term roadmap to basically nerf the way peer dependencies work. We’re going to make them do less stuff. And we’re going to make them be more explicit about what they are doing. Anyway, back to you, Rebecca.

  Yeah. So, peer dependency is the one piece. Shrinkwrap is another sticking point that we’ve had. And shrinkwrap is a way of specifying, the modules all have their version ranges that they allow, shrinkwrap is a way of saying, “Well, we have locally tested with specific versions of each of these modules. So, we only want to use those versions.” And it’s essentially a way of snapshotting your Node modules folder without actually taking a copy of it. It just copies down the versions of everything that you have installed. 

And the way it works right now is sometimes confusing. There are some corner cases. If I type in npm install and a module name that happens to be in shrinkwrap, I’ll only get the version that’s in shrinkwrap and it’s not always clear. That can be confusing because normally you type npm install and a module name and you get the most recent version of that module. Some of this is just going to be better messaging. So, improving npm to give more feedback when it does things that might be unexpected like, “I’m installing this older version because you told me to.” 

One of the intents of shrinkwrap is to make it so that, well in my module I can say I want specific versions of my direct dependencies but I can’t require specific versions of all of their dependencies. And npm-shrinkwrap gives you a way of doing that, because you couldn’t say, “If you see this module, this is the only version allowed.” So, our hope is that by centralizing the version of the module resolution and dependency resolution into a single piece that happens separate from everything else, we’ll be able to make the interactions between shrinkwrap and the other pieces more clear. 

And finally, there’s dedupe which just looks for modules that could be removed, because if you’ve got two dependencies that both include another module of the same version, why would you want two copies of it installed? They could be moved up to the top level. And this can sometimes produce non-intuitive results, like you can end up with older versions than you might expect because well, one module requires this older version and the other module can use the older version but ordinarily would get a newer one. And so, now it’s working with an older one than you would originally consider. But if everyone’s following SemVer, this works great.

  One other thing I want to point out about that is that once we have this stuff more straightened out, that allows us to, we have been recommending that people use shrinkwrap for making reproducible builds so that when you deploy, you shrinkwrap your application before you deploy. And then you know exactly what versions of things are going to be installed. That’s in theory. In practice, it actually hasn’t worked out that way in large part because the install algorithm has been non-deterministic because it’s been following the state machine all over the place. 

The work that Rebecca’s doing by consolidating all that stuff, separating it into separate phases, one of the main major goals for this is to make sure that we have completely repeatable builds. And by that, I mean when I start from one version of my application with one set of dependencies installed, and then run npm install and get a bunch of stuff installed, and create a shrinkwrap file, check that in and then you check it out and run npm install based on that shrinkwrap file, you’ll end up with the exact same versions of everything installed in your application that the first person did when they started. That’s been the hope for it all along, but it actually doesn’t work out that way in practice for a bunch of reasons that are basically just related to how the actual install algorithm works today.

  Right, right. And there are a host of corner cases that people run into where they’re surprised by the outcome of, dedupe doesn’t include development dependencies when it’s deduping. And there are arguments for this and against this, but we need better messaging around these things. And so, there’s other pieces we hope to include there. 

  That’s very cool. I’m really excited to see some of these go into effect. The dedupe is definitely handy and just some of these other cool features. Are there any other things that we can talk about? We’re getting toward the end of our hour here.

  Just to add with dedupe, the plan there is to actually fold that functionality into install. And so, that will no longer have to be something that’s done separately.

  Oh, nice.

  One of the issues that people have is there are a large number of people out there who would really like npm basically to work exactly the same as Bundler. They want to be able to check in their… they basically want to be able to go and install something, do things, and then they’re going to get an artifact, like npm-shrinkwrap. [Inaudible] But in the case of Bundler that would be the gem file, that lock that gets updated automatically when they do everything. And then the [ground truth] becomes that lock file that’s always updated whenever you do anything to change the status of your application. That’s pretty different than how npm has grown and how the ecosystem, a large chunk of the community around it thinks about it. But we do want to make that a more explicitly supported and smoother use case. 

So, one of the major outcomes of this if we get everything right is that you can make it, just generally make it a lot easier to support this as a primary way for you to package and distribute your applications. And to develop in such a way that you don’t actually have to be as mindful of the fact that you’re interacting with a shrinkwrap file or deduping things or whatever. And then also, the dedupe piece will pay dividends for one of our chronic support problems which is Windows, which we can probably spend a whole hour talking about all on its own. 

  [Chuckles]

  But if we are deduping by default, and we’re always trying to create its dependency tree as we can, it does lessen the thorny edges of working with Windows to a certain degree.

  Before we run out of time, currently two people who are also npm employees are traveling across Europe for conferences and vacations and such. But we are renovating the website pretty dramatically. This is going to include some pretty handy additions for finding modules and putting modules into groups called collections and ecosystems. And so, there’s actually a pretty interesting set of ecosystems that are already gathering modules into groups today. 

There’s stuff like Browserify, or you can look at all of the ecosystem of Grunt plugins or all the things that work on Windows or all the things that work on Azure, and so on. One of the first of these is a collection of all the modules that are designed to work with Salesforce APIs. And so, they’re actually doing, have a blog that’s going to be going live relatively soon on developer.salesforce.com/blogs and that’ll be talking about npm and how to connect it with Salesforce and stuff. So, there’s some pretty interesting stuff that’s going to be happening on the website. 

And there’s also going to be some ways for individual developers to say, “Here’s a collection of stuff that I use to do a login flow.” So, something that’s outside the scope of a single module which would be more like a blogpost or a collection of things. They could say, “I’m using these two Express plugins and here’s how I connect them. And then I’m talking to Redis and here’s how I’m doing that.” But anyway, yeah, so the exciting thing about that is being able to tie a package page back to an actual usage example of this thing and have people be able to share how they’re using modules together and which modules they’re using together and which ones complement one another well.

  Cool. That sounds really cool. And I like how it’s tied to a specific use case. The Salesforce example is very I think appropriate to that. 

  Yeah, for sure. They have a bunch of web APIs and a lot of people are using Node modules with them. But it’s not very clear, what’s the one I use for doing this one particular task? You’re going to find very small modules that address very discrete, individual subtasks of that, which is good. But it requires a lot of thoughtfulness on the part of the user. So, just jumping in and getting started for new people is very difficult still.

  Yeah.

  One thing I’d like to reiterate: if you want to get a feel for scoped modules before they’re in the public registry which is going to be a little ways out, just npm install npme on any CentOS or Ubuntu Trusty machine and you can start playing with scoped modules. 

  Cool.

  That’s super rad.

  Yeah. Alright, should we do some picks?

  Absolutely.

  Alright. Dave, do you want to start us with the picks?

  Sure. So, is it appropriate to pick other podcasts?

  Yes.

  So, one I’ve been really enjoying lately, if you’re a bit of a history geek, is Dan Carlin’s Hardcore History. He goes into some really cool detail about some human conflicts over the course of history, really personal touch. And I really enjoy it. 

  Nice. Jamison, what are your picks?

  I have three picks. The first pick is a talk by Brad Bouse. I don’t know how you pronounce his last name. He spoke at JSConf EU a week or two ago about creating art and experimenting with things that aren’t trying to build useful products for people. And that’s something that really appeals to me, just the joy of creating for creation’s sake. His slides are one of my picks.

The next pick is a little npm module called boids which is for doing a flocking simulation. So, it’s when you have this group of entities and they follow each other around via a few really simple rules. It’s some really, really simple artificial intelligence or I don’t know what the other word for it is. Anyway, it was cool that there was an npm module for that. It was very helpful. 

And then my last one is a site called 5013.es which is a bunch of side projects by someone. I think her name is supersole on Twitter. I don’t know. But it’s a bunch of art and music and WebGL code experiments. And they’re all pretty fun to play with and inspiring. Those are my picks.

  Cool. Joe, what are your picks?

  Alright, so I got three picks today. My first one is a really cool. I guess you call it a video game. It’s called JAXI – The Robot. But it’s actually an educational game used to teach kids how to program in JavaScript. And it’s done with a game where you control this robot. And in order to get the robot, it’s a platformer type game. If you want to get him to move around in the world and get farther in the game, you have to write JavaScript in order to get him to do that sort of stuff. And it teaches kids how to code in JavaScript using a game, which I thought was incredibly, incredibly clever. And you can support them on Kickstarter. And it’s just at JaxiTheRobot.com. 

Then my second pick is going to be a CSS approach that was developed by a friend of mine, Dave Geddes. It’s called csstyle where the second S in CSS is the first S in style. It’s just at csstyle.io. And it’s clean and simple styling for the web. He did a presentation on it in a previous user group and it was really, really cool and a great way to clean up your CSS. And it’s aimed at developers, people that aren’t necessarily experts in CSS. So, if you’re a developer but you don’t consider yourself to be awesome at CSS, it’s a great framework to look at.

And then lastly, I’m going to pick Maroon 5’s new album. I’m not sure if it’s called V or five. It’s the letter V. I’m not sure if you call it V or five. But their new album is actually pretty cool, a lot of good music. So, that’ll be my final pick. 

  Awesome. I just have one pick this week and that is Boomerang. It’s a plugin for Gmail. And I’ve just found it to be really handy. If you don’t know, I’m a freelancer. Hire me. Thanks. So, for follow ups I’ve just not been super good at follow ups. And my secret is Boomerang now. And basically what it is, is it’s a way of having conversations in Gmail brought back to your inbox. So for example, I’ll email somebody and say, “Hey, do you want to talk on Skype or grab lunch?” And they’ll email me back and then say, “Well, I’m busy for the next two weeks but can you check with me then?” And I say, “Yes.” And then I tell Boomerang to bring it back to my inbox in two weeks. 

Or some of the folks I email sometimes get busy. And so, I’ll send them an email and I’m pretty sure they got it but they don’t respond And so, I’ll just set it to bring it back to my inbox in a week if I don’t hear from them. And then I can email them back and say, “Hey, I bet you got busy. I was hoping we could talk about whatever.” And it works out really nicely. So, those are my picks. Isaac, what are your picks?

  I’m actually curious about Boomerang. How does that compare with FollowUpThen? 

  Like FollowUp.cc?

  No. 

  So, there’s a service out there, it’s at FollowUp.cc, that I’ve looked at before. And that one, you actually forward the email to them two days from now or whatever. 

  Yeah, yeah. FollowUpThen is similar, FollowUpThen.com.

  Yeah. So, it’s the same kind of thing. If you email it to them and then they send it back.

  Yeah, yeah.

  Boomerang’s actually a plugin in Gmail. 

  Oh.

  And since I use the web interface, I just as part of the form that I’m filling in, I just tell it when I want to be reminded to check up on it. 

  I see. That’s great.

  It’s right there in the interface and then it just puts a tag on it for Boomerang. And then yeah, it keeps track and tells it to come back. If you’ve set it to Boomerang 2, it also has a little part of the UI that if you open up an email that’s set to be scheduled to come back, it’ll tell you when it’s scheduled to come back. So, if you’re going through your email and stuff, then it’ll show you that, too. So anyway, I love it.

  Okay, cool. I just fixed my mic and I’m not going to move and I’m going to try and do these picks. So, I have three picks. The first is a book called Ancillary Justice by Ann Leckie which is fantastic and everybody needs to read.

The second is another podcast that I’ve been continuing to enjoy. I don’t know if I, I don’t think I mentioned it last time I was on JavaScript Jabber. But it’s called Roderick on the Line. It’s helping a lot of people. It’s one of those podcasts you really need to start from the beginning. And if you don’t like it by the fifth or sixth episode, you should probably reevaluate your life. 

And the third pick is actually an anti-pick. I’m going to unpick U2’s new album.

  [Laughs]

  And provide a link that will actually remove it from your account, which is fantastic. [Laughter]

  Awesome.

  Literally the only part of that of that that came through was the end. 

  Oh. I heard the whole thing.

  Yeah, me too. 

  Yeah. Forrest, maybe the problem is your speakers, not my microphone. [Chuckles]

  Your disease is spreading.

  Dang, getting gas-lighted by a CEO. [Laughter]

  You know, I got pretty used, when I was an employee I got pretty used to tuning out my boss, too. 

  Fair enough.

  [Chuckles] U2 Removal Tool. That’s funny. So, it’s not even worth what you paid for it. Is that what you’re saying, Isaac?

  I’m saying anything free is a rip-off. 

  Ah.

  [Chuckles]

  Nice.

  That’s a good life tip.

  Rebecca, what are your picks?

  I’m going to pick two parser projects. There’s nearley on npm which is an implementation of the early parsing algorithm in JavaScript. And this allows you to parse anything BNF can express. And it’s super-fast. And it provides much better diagnostics than other more popular parsing algorithms. And similarly, there is Marpa which is probably the most refined implementation of early parsing. This is something that came out of the Perl community. 

And nearley has been in the process of being updated to include the refinements that Marpa has done. But for people who are interested in parsers, even if you’re not interested in Perl, Marpa is well worth reading into, because they’ve done a lot of really cool stuff. It’s unbelievably fast. And the diagnostics it gives you are, it can actually tell you what’s wrong, which if you’ve ever written a yak-based grammar, that’s just unheard of. Those are my picks.

  Very cool. Ben, what are your picks?

  So first off, Forrest, Mikael Rogers, and myself and a few other Oakland Node people are putting together an event called DanceJS, which is going to be a bunch of tech talks about hacking music and also followed up by a dance party. Information’s at DanceJS.io. And it’s going to be on October 4th. So, I just wanted to put a plug in for that.

My next pick is a library I’ve been using a ton lately with all the CLI stuff I’ve been writing. It is called inquirer and it’s just a great tool for reading in command line arguments in Node.

I guess the third thing I’ll plug will be my own library, ndm which we’ve been using for Enterprise a ton, which is just a way to create run scripts on various operating systems for long-lived processes. 

  Cool. So, is it a daemon library or something?

  Yeah. It’s similar to Foreman but has more features than Foreman and has been built up around npm Enterprise. 

  That’s really cool. Forrest, what are your pics?

  I have two. The first is Edna Piranha’s revisit link. If you don’t know who Edna Piranha is, well first you’ve got to get on my level here. But second, she is the woman who is responsible along with Soledad Penades who you already mentioned, that’s supersole, for making Meatspace, the animated GIF based chat network. 

And revisit is her next evolution of a lot of the ideas behind Meatspace. It’s a set of simple API restrictions for building API services that basically let you glitch images and audio and other things where you can write some stream transform and produce soothing. And it’s a way to get people who are maybe more creatively inclined than hardcore coders into writing simple web services in a modern style. And it has produced just an endless stream of completely bizarre animated GIFs and vaporwave designs and all kinds of crazy culture. Everything that is weird and awesome on the internet seems to be tied to Edna Piranha in some way. 

My other pick is I have been completely obsessed with Ta-Nehisi Coates’s blogging about the Civil War. He’s a blogger about a lot of contemporary issues for The Atlantic and wrote a pretty highly regarded article making the case for reparations at the beginning of this summer. But he’s been very much completely obsessed with the history of the Civil War, making the history of the Civil War more relevant and direct for African American audiences, trying to look at it as an actual phenomenon, look at how it ties to reconstruction. Just a ton of stuff. 

It’s got me reading huge stacks of Ulysses Grant’s memoires and James McPherson’s ‘Battle Cry of Freedom’. And the guy has gone through and compiled a list of all of his writing about the Civil War on his own blog. I dropped the link to it in the notes. And it’s more than you can read in a week. And it’s just fantastically interesting. And there’s a really lively commentary community about it, this thing called the Effete Liberal Book Club grew up around it. And that’s been actually absorbing pretty much all of the time that I am not spending obsessing about npm.

  Very cool. Well, thank again for coming guys, especially on the short notice. But it’s just really interesting stuff coming through. 

  Thank you. Thanks for having us. 

  Alright.

  Yeah, this was great.

  Well, we will catch up with you all later.

  Alright. 

  Thank you.

  Thank you.

  Thanks to our listeners. We’ll catch you all next week. 

[Hosting and bandwidth provided by the Blue Box Group. Check them out at Bluebox.net.]