There’s always more to learn about security, especially nowadays. In this episode, the Jabberers continue their conversation with Feross Aboukhadijeh about supply chain security. You can never be too careful! (Well…maybe.)

In This Episode

1) How the BEST way to keep your security tight is NOT done on the computer
 2) Why we’re seeing a trend toward THESE kinds of packages in 2022
 3) What you NEED to know about dependencies and their expiration dates
On YouTube
Supply Chain Security - Part 2 - JSJ 525

Sponsors

Picks

Special Guest: Feross Aboukhadijeh.

Sponsored By:

CHARLES MAX_WOOD: Hey everybody and welcome back to another episode of JavaScript Jabber. This week on our panel, we have Steve Edwards. 

STEVE_EDWARDS: Hello from Portland where it snowed a few inches this morning in all the schools and everybody freaked out. 

CHARLES MAX_WOOD: AJ O'Neill. 

AJ_ONEAL: Yo, yo, yo. Coming to show live from Provo where it's been snowing the past couple days somewhat unexpectedly and I'm out of Mountain Dew today. 

CHARLES MAX_WOOD: Oh, that's tragic. I'm Charles Maxwood from Top End Devs. We also have Dan Shapir. 

DAN_SHAPPIR: Yeah. From nighttime in Tel Aviv, hopefully. not going to get disconnected this time around. 

CHARLES MAX_WOOD: We can hear you loud and clear this time, so we also have for us, Bukka DJ. 

FEROSS_ABOUKHADIJEH: You said my name perfectly, coming at you from sunny San Francisco. It's a beautiful day here. 

STEVE_EDWARDS: Well, I just keep thinking about what you told us last time, it was like you're going to go Bukka DJ and hey, easy to remember. 

FEROSS_ABOUKHADIJEH: Yeah, it works. I use it for a reason. 

CHARLES MAX_WOOD: Yeah.

 

Hey folks, this is Charles Max Wood from Top End Devs. And lately I've been working on actually building out Top End Devs. If you're interested, you can go to topendevs.com slash podcast and you can actually hear a little bit more about my story about why I'm doing what I'm doing with Top End Devs, why I changed it from devchat.tv to Top End Devs. But what I really want to get into is that I have decided that I'm going to build the platform that I always wished I had with devchat.tv and I renamed it to Top End Devs because I want to give you the resources that are gonna help you to build the career that you want. So whether you wanna be an influencer in tech, whether you want to go and just max out your salary and then go live a lifestyle with your family, your friends, or just traveling the world or whatever, I wanna give you the resources that are gonna help you do that. We're gonna have career and leadership resources in there and we're gonna be giving you content on a regular basis to help you level up and max out your career. So go check it out at topendevs.com. If you sign up before my birthday, December 14th. If you sign up before my birthday, you can get 50% off the lifetime of your subscription. Once again, that's topendevs.com. 

 

CHARLES MAX_WOOD: So we had you on for part one. This is part two, talking about some of the security stuff. To kind of get us started, I had a handful of questions I still wanted to ask at the end of the last one. And so I'm just gonna jump in and hit one of those real quick. So you were talking about using automated tools like...Socket, right? Or we were talking before the show about some of the other ones out there that, you know, scanned for known vulnerabilities and things like that. I'm imagining that there are other things that you ought to be doing beyond just kind of plugging a tool into your code base that will help you, you know, avoid security issues and things like that. And so I was wondering, let's say somebody comes along and says, oh, I'm going to use Socket. What else do you recommend that they do in order to make sure that their app is secure?

FEROSS_ABOUKHADIJEH: Yeah, that's a great question. I mean, I think probably the most important thing that you can do is to have a mindset shift around how you think about dependencies. So if you ship code to production, you are responsible for it, whether it's code you wrote or whether it's code that comes from your dependencies. Because when that code actually runs, it's running in the same, you know, all the code, whether it's your code or open source code, it's going to run in the same node process it's going to run in the same browser web page. It's all going to get bundled together. And it doesn't matter. It doesn't matter whether you tap those keys out on your keyboard or it came from someone else. It's all going to, it's all part of your app. And so therefore you're kind of ultimately responsible for the bugs in it and for the security issues that may be there. So I think that's like one kind of just general like mindset shift, I think that's really helpful. And once you start to think about it that way, then the way you pick dependencies sort of changes. So you no longer just check for does it get the job done? Does it have an open source license that lets me use it? Does it have good docs? That kind of stuff, and how many get up stars? You kind of go beyond that and start to say, okay, what is the philosophy of the team that makes this project? Are they judicious in the dependencies that they select? You wonder, you want to know, do they respond to security researchers that find problems in their package in a timely manner? Do they take this stuff seriously? or not? Like what is their policy for adding co-maintainers? Do they have 50 other people who have access to the project and any of them could publish a new version at any time they get hacked? Or are they very careful with who can do releases and do they have a good process around that? So it's a little bit more work and that's kind of where socket is like trying to help so you don't need to be an expert in all this stuff and do like an hour of research every time you're trying to select a package. Like we're hoping that you can just go to socket type in the package name and then will flag the most egregious issues if there are any. And then you can quickly find out that same information without having to go on like a hunt through, trawling through all these package JSON files of all the dependencies of the dependencies and so on and so forth. But yeah, that's probably like the main thing I'd say is just thinking differently about dependencies. I don't know, there's probably, I mean, I can mention other things too if you want, but that's the biggest one. Ye

CHARLES MAX_WOOD: ah, I mean, yeah, I'm just, I'm thinking through it and thinking, you know, that seems like a lot of work, but I think you could get adept at finding that information, right? And so it's like a five or 10 minute kind of gut check on the project, right? Where you go look at the project.json. Yeah. You look and see, yeah, all those things that you can look up all that stuff. And so then it's, okay, I've got a good feel for what may or may not actually be the case here. 

AJ_ONEAL: And I don't think it even takes five minutes. It's, it's really quick to do a quick survey, but, but the problem is transitive dependencies that That could become hours. And I think that's where socket.dev sounds like it's gonna provide a lot of value because it's looking at the whole chain of being able to give you some heuristics or some metrics about the dependencies, right? 

FEROSS_ABOUKHADIJEH: Yeah, I mean, like, you know, take install scripts as an example, right? Say you wanna know if when you install this, it's gonna run code on your computer and you're kind of curious to know that before you run the command npm install such and such package. Like you're not gonna be able to easily go through every dependency transitively all the way through and find out if any of them have an install script. I mean, you could theoretically do that, but like that's just a lot of work. And so it's much more helpful if you can just go to socket and see, like it says at the top, like there's gonna be an install script that runs and it's gonna come from this transitive dependency. Click here to read the script that it's gonna run and you can click it and read it. And it's not, again, and we don't link you to GitHub and show you what's on GitHub, because there's no guarantee that that's actually what's gonna run. We show you the actual source of truth, which is what code is in the NPM tarball that's gonna get downloaded from the NPM registry, and that's what's actually gonna run on your computer. So we show you the actual truth there. 

AJ_ONEAL: Yeah, that's a long-standing issue with NPM. I don't know why. I mean, I'm guessing it's just for user acquisition and you being a venture-backed company that they would rather pay lots of money to host stuff themselves than have greater security by getting it from the source through a decentralized protocol. 

CHARLES MAX_WOOD: Yeah, but if I'm hosting it on GitHub, I may have things in any number of states. People don't always manage their projects well. I think most open source, especially if people are contributing to them, are managed well, but... 

AJ_ONEAL: If you had to run npm version and npm publish, that's no different than running git tag. So if you've got the discipline to be able to run npm publish, then you have the discipline to be able to run get push. 

CHARLES MAX_WOOD: You know, it'd be interesting just to have that built in, right? To npm, right? This is a git repo. Do you want me to tag this too? 

AJ_ONEAL: Well, it does. You just run npm version. That's what npm version does. 

FEROSS_ABOUKHADIJEH: I think the problem is that... So npm's publish command is so flexible. Like, it lets you publish a folder of files. There's no requirement that you use git or github or anything like that. So you can publish like a random folder as long as there's a package JSON with like a version field in it, it'll publish it. And so that's one problem. And then the other one is like GitHub, you can rewrite the history in Git, right? So you can like change a Git tag to, so like if you tried to say like, why don't we just install stuff from GitHub? There's a lot of advantages to that idea. But I think one downside or one problem is when you publish something to NPM, it needs to be immutable so that people can't take the code down. And that's, they kind of went really hard into that direction after the left pad incident. Left pad was removed. So they need to be kind of an extra thing that they would add on that would say, like, once you tag something in an NPM, in a GitHub repo that's for NPM, then it would need to like never be able to be changed or something like that. 

DAN_SHAPPIR: Yeah, but now that- This is already solved. Yeah, especially now that GitHub basically owns NPM, you would have expected these types of integration issues to be solved, no? 

FEROSS_ABOUKHADIJEH: Yeah, I don't know, yeah. It doesn't seem like they're prioritizing it very much. Well, they explicitly did not want to-

AJ_ONEAL: to have good support for Git. There were issues open where they explicitly said, no, we don't want to support features of Git. We don't want proper branch and subdirectory support. But as far as the worrying about tags, that's not something you really gotta worry about. If somebody rewrites their Git history, then that should produce an error because you have a package lock. And then in the case of Go, for example, whenever you install something with Go...Unless you turn it off, it's going to check against the teeny tiny, teeny tiny package registry that Go maintains, where it just keeps a history of all Git tags and commit hashes. So that's not really a problem, because you've got it both locally and via the remote host. You can know that if a Git hash is changed, that's a very tiny amount of information. But then again, I mean, this is talking about the world that could be rather than the world that is.

DAN_SHAPPIR: Yeah, my feeling is more or less that they've got a system that works so they're not touching it kind of feels like to me. 

CHARLES MAX_WOOD: Well, and you guys kind of implied that NPM got folded up under GitHub and I don't, I don't know that I ever actually heard that. I know Microsoft acquired NPM, but I don't know if they said you guys are part of them guys. 

AJ_ONEAL: GitHub acquired NPM, Microsoft acquired GitHub, GitHub acquired NPM. 

CHARLES MAX_WOOD: Okay. 

AJ_ONEAL: So Microsoft owns all code and all AI related to code.They have done the ultimate embrace extend. 

CHARLES MAX_WOOD: Now they just need to give it all to China. Anyway, I do have another thing that I want to kind of hark back to because what you were talking about there made me think, and this is another question that I had last time that I wanted to ask, but I feel like it deserves a little more context and conversation than that is as the supply chain attacks become more common, AJ mentioned trans transitive transitory. I don't remember which words you used.

AJ_ONEAL: Transitive dependencies, meaning that you follow the tree all the way down. Oh, by the way, there was just another attack announced this morning, and it was almost exclusively typo attacks, where modules were just off by one letter or had JS dependent to the end of them or something like that. 

CHARLES MAX_WOOD: Right. So, I guess my question is, as these things become more common and this becomes a more common vector of attack, do you see a general push for...people to start using packages that have fewer dependencies? Like, do you see that as a movement that will happen within JavaScript? Or do you think people are just lazy and they're just gonna let it slide until it bites them too? 

FEROSS_ABOUKHADIJEH: I think you've already seen a trend toward that somewhat. There's certain maintainers I've already seen like advertising the fact that their package is zero dependencies, it has no dependencies, that kind of thing is like starting to become part of some packages marketing. So maybe we'll see that trend increase. I don't know. I think there's just so many advantages to using dependencies that there's so many advantages to this way of writing software that people are not going to be willing to give up. So I don't think we're going to see a massive change in the culture of the JavaScript community overnight in response to this. But who knows? 

DAN_SHAPPIR: It also seems a bit contradictory to me. I mean, it's like saying...Trust me as a dependency because I don't trust anybody as a dependency. 

AJ_ONEAL: It's kind of odd. Well, no, it's about not being careless. It's about not doing it. It's about not requiring dependencies for trivial crap. 

DAN_SHAPPIR: Oh, yeah. 

CHARLES MAX_WOOD: I've reduced your exposure. 

DAN_SHAPPIR: Yeah, no, for sure. I would. 

AJ_ONEAL: And don't get having just junk code that you haven't looked at. So for most things, you don't need 1900 dependencies.

DAN_SHAPPIR: You know, I totally, I totally agree. I would prefer a dependency that has fewer dependencies. I'm just saying that advertising a dependency as having no dependencies is kind of self-contradictory. 

AJ_ONEAL: I get what you're saying, but also I think that when I look at a module and I see it's got no dependencies, that means the module does what it says it's going to do. And it doesn't, my risk vector is really low. So I like seeing that on packages. Because if I need something to say integrate with postmark and it's zero dependencies, that makes sense. Why would you need any dependencies to make an HTTP request and set some environment variables? Right? Well, 

DAN_SHAPPIR: yeah, again, 

FEROSS_ABOUKHADIJEH: I guess it's sorry, 

DAN_SHAPPIR: go for it. 

FEROSS_ABOUKHADIJEH: No, you go for it. 

DAN_SHAPPIR: No, all I was saying that it I guess it really depends on what type of a dependency it is. If it's a really specific dependency that you need for really well defined tasks that you can kind of understand what its scope needs to be, then I totally agree. But if it's a dependency that provides a greater breadth of functionality, then I can understand why it might have its own sub-dependencies. For exactly the same reason that my project has multiple dependencies. 

AJ_ONEAL: Yeah, you look at something like Express, even if you wanted to clean it up, you're still gonna have some dependencies. Same thing with Create React app. I think they took it down from a thousand to a hundred or something in that categorical range, you're still going to need some, but you can be conscientious and not just say, Oh, every time I need a function, I'm going to Google Stack Overflow and npm install. I mean, I recently used a library that has three different request libraries in it. It's using Axios, request and request promise. That's Stack Overflow code right there. 

FEROSS_ABOUKHADIJEH: Yeah, no, it's like all things in software. There's, there's, there's no absolute rule that you can apply across the board. Like if you had the rule that I never use anything, I don't install anything that has dependencies, right? Like then you're gonna, that's overly restrictive and that's also not gonna always be the right decision. And then if you have the opposite opinion that it doesn't matter, that's also not right. I think it's like about weighing costs and benefits. So like, if you're writing an app that needs to do like code highlighting, for example, you wanna colorize the different tokens in the code and put it on a webpage, like you should probably use a dependency for that. Like you should probably not write that in house. And if there's a function in there that has to do with escaping the tokens safely to put into the HTML code, again, that might be a good instance where you wanna use a dependency because that code is tricky to get right and there may be changes that happen in the browser and you may just wanna have someone else outsource that to someone who's like focused on the security and then you can just call this function that says sanitize my tokens to be safely added to HTML. And so it's like, you know, those are good examples of having three HTTP libraries. That's like obviously a bad idea. So I just think it's a case by case thing and you need judgment and you need, yeah, you just need to kind of have some taste, unfortunately, it's not a thing that can be fully automated. 

DAN_SHAPPIR: So I have to make two specific comments that hopefully don't take us too deep down a rabbit hole. One is that I really don't get why people still use Axios now. Now that we've got native fetch both in the browser and in Node, or coming to Node. And the second is... Oh, well that's the thing. 

AJ_ONEAL: We don't have it in Node yet. 

DAN_SHAPPIR: Well, almost. And the other, AJ, remember that I gave you that sanitation code that was like two lines? 

AJ_ONEAL: I, on Twitter, for the DOM API, I don't think it actually was... There's different types of sanitization. I think that one stripped out rather than escaped, but yeah. 

DAN_SHAPPIR: No, it actually escaped.

FEROSS_ABOUKHADIJEH: Yeah, there's all kinds of... Because I teach the web security class at Stanford, and one of the first lectures we do on this is like, we talk about the different contexts inside the HTML, like where in the HTML you're inserting this user content, and the rules for escaping are very different depending on the context. So putting something like in the middle of it, like in the body of a tag, there's like one set of rules, putting something in an attribute. So between the quotes, between the double quotes or the single quotes in an attribute is like a different set of rules putting something in like a string in a JavaScript script tag is like much harder to do, which you probably shouldn't do. So it's all, it's actually, depending on what you're trying to do, it gets complicated. 

DAN_SHAPPIR: Oh yeah, yeah, for sure. It's just, I was amused because a while back, AJ, I think tweeted something like, why do we keep getting functions in JavaScript in the DOM that we don't need and don't get the functions that we do need, like escaping or something. And I responded with an escape function in a tweet, which basically created a div, put the string to be escaped using content text, and then got it back out using inner HTML. So that was the escaping hack that I used, but I'm really taking us off on a tangent. 

AJ_ONEAL: Well, I think it's good to know about those, these types of things, because if people, one good thing that has happened, although much of it's been lost in the noise, is that there are good tools that have become native to the platforms, but not enough people know that we, some of the most critical things have become native. There's still a ton that aren't, but there are many that are. And it'd be nice if we had a list somewhere of, you might not need X, you know? Probably somebody does have that. 

FEROSS_ABOUKHADIJEH: That's actually something that Socket can help with, by the way, so like, one of the things we're considering is having an opinionated, like one of the things we check for, it could be...And one of the rules, because if you think about what we're doing, it's sort of like we have these lint rules, right? And they're they're checking for different things in the package or different behaviors in the metadata, like is a new maintainer added, that kind of thing. And one of the rules could be literally like, do we does the community in general think that this package is should not be used? It's past its expiration date. So like one great reason to put something in that rule would be, well, there's a native version of this now. 

AJ_ONEAL: And like, you know, I don't care what the community thinks. I care what you think as an expert.not just the average Stack Overflow person. 

FEROSS_ABOUKHADIJEH: Yeah, yeah, yeah. Okay, that's fair, yeah. So it'd need to be a little bit more, it's not just like a voting, we take the voice of the community, it's more like, like object.assign, right? That's the thing that you can probably just rely on now that you can call object.assign. You don't need to polyfill that anymore. So if we see that you're about to go install that or that that's down the tree in your dependencies, we can kind of like. That's a relatively minor issue, but we can at least flag that so that the maintainer sees that and that, you know, okay, this thing has unnecessary polyfills in it. And like, that's a small signal that the quality of this package might be a little lower because someone hasn't gone in and like clean that up yet. Cause we're way past the point where we need that polyfill. 

DAN_SHAPPIR: The big one, it's not, 

CHARLES MAX_WOOD: it doesn't seem as simple to me though, as, as what you're saying, I mean, to a certain degree it is right. Hey, this is out there now, but I could also see my way to, Hey, you need to upgrade node to get this functionality natively. This is native now, but you need to upgrade. Or if you're supporting these browsers and not those browsers, depending on how quickly they're implementing the standards, then you can write. And so it's, it's not just a, Hey, this is out there now because I may be on an old version of stuff that doesn't play nice with it or doesn't have that built in. But you can tell me that it's there now if I upgrade.

DAN_SHAPPIR: My experience, by the way, is the reverse, is that most people use libraries way, way, way beyond their expiration date, way after the platform already supports whatever functionality they think that they might need. 

CHARLES MAX_WOOD: Oh, yeah. 

DAN_SHAPPIR: And by the way, the new one for me, the new, the new hotness that I have to mention is the internationalization object that's now built into JavaScript, the INTL. 

AJ_ONEAL: If you abuse it sufficiently, you can get time zone support in just a few kilobytes. 

CHARLES MAX_WOOD: Does that feel like bloatware to anyone else? Like, does it need to be in JavaScript natively? 

AJ_ONEAL: We need what INTL provides natively, yes. We need it in a much better way than the way it provides it, because it is basically unusable. And it's really slow, which usually doesn't matter because usually you're only doing things one at a time. But it's making...the operating system calls to go parse the, for example, the time zone database file every single time you call it to get a time zone thing. 

FEROSS_ABOUKHADIJEH: The main benefit of ITL is that you don't need to ship this, this big data set in user space. But yeah, we're going off on another tangent. 

CHARLES MAX_WOOD: I was going to say, and tangent. 

STEVE_EDWARDS: Tangents. We never go on tangents. 

AJ_ONEAL: But that's, that's what dependencies are. They are tangents. 

CHARLES MAX_WOOD: That's right. We follow these tangents way past their expiration date. Thanks Steve. But yeah, I mean, it is interesting, right? And Dan is, Dan's right. I mean, it may be the state of the art when I build the app, but yeah, I don't go back and audit my dependencies hardly ever. 

AJ_ONEAL: And there's something to be said for, don't break things that do work just to be cool. So I hate it when I see that a library's been completely rewritten in TypeScript and all of the bugs that were there when it was in JavaScript that are the logic bugs or whatever. They're still open issues for they still haven't solved it. So they took the, they took the week to go rewrite the whole dag on thing and TypeScript, but they didn't actually improve the quality of code at all. They just made it now incompatible with a whole bunch of other stuff and required transpilers and blah, blah, blah. Without any real substantial improvement to what the thing actually did. And that's something that I think I wouldn't, I don't want us to go breaking. If something's 10 years old, maybe we just need a new one. Right? Maybe we need a new one with a compatible API. Maybe it's fine that Jest is 10 years old and it carries all the baggage of trying to be compatible all the way back to node 0.4. Maybe we shouldn't just rewrite Jest in TypeScript, but maybe we should have a Jest 2 that uses the native APIs and that doesn't carry that baggage. And that's, again, that's kind of a balance thing because on the one side, yeah, it'd be nice if they just removed all of that baggage history and it was just plain vanilla JavaScript that didn't require transpiling, a hundred thousand dependencies to do just basically throwing errors and, and putting them in arrays and doing a little colorizing. But at the same time, if you did that, then there's stuff that's dirt old that when they went to do an NPM install and they don't have a package lock anymore because, or never even got a package lock because they're on node V six or eight or whatever it was. You know what I'm saying? 

STEVE_EDWARDS: Surely you just, 

CHARLES MAX_WOOD: I'm going to circle the year tangent back around to the main point. So does socket work on TypeScript or any other languages that compile either through WebAssembly or transpiled to JavaScript? Or do you have to check transpiled code? 

FEROSS_ABOUKHADIJEH: So socket analyzes whatever ends up in the tarball on NPM. So whatever gets published to NPM when the maintainer runs NPM publish. And I don't know that many people that are publishing their raw TypeScript sources to NPM packages. Usually the way that it's done is, pre-published step that compiles the TypeScript sources to a compiled output, and then that's what gets published, and then the TypeScript stuff's usually ignored and not included in the final package. And so, yeah, we parse the AST of the JavaScript file, and then we do our static analysis rules directly on the JavaScript sources. So anything that compiles the JavaScript will be included and covered by socket. And one of the interesting things about TypeScript specifically that we want to do a better job of is when it gets back to this problem we talked about before about the code on GitHub not matching the code on NPM, which is something that a lot of the supply chain attacks that have happened so far actually does, because it's such an easy thing to do and it makes it so much harder to realize that this bad code has been added, is that you go to GitHub and the code on GitHub doesn't match. So we could try to flag that and say, hey, we noticed the code seems different on GitHub than it does on NPM. But the thing that makes that hard to do is, obviously, like we just said, you're not publishing all the code that's in the GitHub repo to NPM. If it's TypeScript on GitHub, and you publish the compiled JavaScript to NPM, it's not gonna match. And so we need to get more sophisticated than just doing a naive comparison of the two and then reporting if they don't match. And so one of the things that we wanna do is get to a place where we can detect when the build is reproducible. This idea of reproducible builds has gotten really big in certain parts of the Linux ecosystem, where the idea is you want to be able to prove that the source code for a program has, rather the other way around. You're trying to prove that the binary code, the compiled code is produced from the source that everyone can look at and audit, because it's a lot easier to audit source code than it is to audit a compiled executable file there's a lot of challenges in getting like native code like that to become reproducible so that you can say we have reproducible builds because a lot of randomness and times and random crap can get embedded into the binary so that like if you compile it on your computer and I compile it on my computer, we might not get this exact same bit for bit output. And so then it's hard to then prove that these binaries are downloading actually came from the source code and weren't modified. The source code wasn't modified in some way before it was compiled. And so hopefully I'm making sense here, but basically JavaScript has a little bit of an easier problem here because what we can do is we can say, look, we're going to download every NPM package. We're going to run NPM install, NPM build, and then NPM pack, which is the command that produces the tarball before, you know, the step before you publish. So we're going to run NPM install, NPM build, NPM pack, and then we're going to take that result and we're going to compare it to what you actually published NPM. And if it's not exactly the same then we're going to treat that as a bad sign. And for everyone who does have that produce exactly the same thing, we're going to give them a big green check that says, this package has a reproducible build. And so even though the code that's on NPM might be this compiled mash, mishmash of, you know, whatever they use web assembly or they used TypeScript or they used something that compiled to some nasty output, and you can't really audit that as easily, well, we can say, look, we know that the version on GitHub, the source code there that you want to go look at actually matches what is on NPM. And so we can give them a little green check that says they do reproducible builds. And I think most packages shouldn't have a hard time passing this because they don't have like a build step, but then the ones that do really should try hard to get the screen check to increase the trust in packages. Hopefully that makes sense. 

CHARLES MAX_WOOD: So you're, it would avoid supply chain attacks on your supply chain attacks, right? Because if your JavaScript or if your TypeScript transpiler is compromised, right, then. 

FEROSS_ABOUKHADIJEH: Yeah.

CHARLES MAX_WOOD:then your final artifact won't match. 

FEROSS_ABOUKHADIJEH: Right, right, exactly, yeah, that's a great point. So if the person publishing the package has a compromised TypeScript compiler that's inserting extra stuff, then this would catch it because we'd be able to see, okay, our TypeScript compiler, it's producing a different output, so that's a problem. And so some projects, they might actually be, they'd be pretty close to having reproducible builds, but they may have some mistake, like they may not have fully specified their dev dependencies correctly or they may be relying on some globally installed libraries on their, on that developer's machine. And that's not specified in the package JSON. And so if they have stuff like that, that will affect whether we can show that they have a reproducible build or not, but that's something they should fix and they should, they should specify all of the actual requirements on the machine so that anyone can come along later and build it themselves and prove that it produces the same result. So that's not something we do now, but I think that's something that's like, we're really interested in getting to a point where that would be one of the rules we check for because it's such a useful thing to have. So it's just, we just need to build. We have a bunch of rules that are blocked right now on us being able to safely run code in a sandbox. So we can do more, we can almost like run the code. Right now we don't actually run the code to see what it does. We use static analysis to figure out what it does, without actually needing to execute it. So we're working on setting up a system where we can actually execute the code and observe it. And so once we have a safe environment in which to do that, we can start to do these other analyses that are even fancier and give us more data. 

DAN_SHAPPIR: So two comments, really. One is in the context of a safe environment, you guys should be looking at something like StackBlitz. Their web container sounds really interesting in the context of a safe environment to run stuff in. And the other thing that I wanted to mention I think in the context of tests, look, we're kind of sport for choice with packages on NPM. So if something is potentially flagged as being dangerous, even though it actually isn't, just because the developer or maintainer isn't properly configuring their system, I mean, we have a choice, we can use something else, and that will push them towards fixing it. 

FEROSS_ABOUKHADIJEH: Yeah, exactly. Exactly. 

AJ_ONEAL: I don't know push people towards fixing it. Cause if you've just got something that, cause all it really is is a vanity play and how much of your time are you going to extend once something's become popular. If you're not making money from it and it's just a pain in your side and people want to use something else, I think you just let them, let it die. 

FEROSS_ABOUKHADIJEH: I mean, that's a fine, that's a fine choice. Like we're not trying to make more work for maintainers. I'm really sensitive to this because as a maintainer, I know like it's just insane amount of issues that are coming in and there's, it's a never ending. It seems like it's a never ending list of requests from users. And so the goal isn't to, isn't to create more work for people, but it is, the goal is for it to make it clear to the user of the open source. Like what, what are the facts about this package? And if one of the facts is that you can't run NPM install and NPM build and get the same output, then we want to tell people that that's fine. I mean, the maintainer could say, we don't care. That's, that's up to them. But I think it's not that hard to do. I think for a lot of a lot of packages don't even have a build step. And so they're just automatically going to be passing this. And so, you know, I, and then it's a best practice anyway. If the, if the project is actually being maintained by a team of people that are working on it all the time, that's, I mean, they want, they want to get their, their builds set up so that any, any person contributing to the project can run it by just running those commands. Actually, that's one of the best parts about node. I have to say is like, you can just run NPM install and then like NPM test, you know, and you can, or like, you can run NPM install and NPM start. and a project will just start. Unlike a lot of other ecosystems, we have these really nice simple commands that most people in the community actually care about making them work. And it's not like run this obscure, random make command in this series of steps and do like five steps to be able to build the thing. We've all kind of unified on these very simple, small set of commands, install, test, build, start. I like that a lot. 

AJ_ONEAL: Yeah, those are nice conventions that are easy to get right if something's named server.js. It'll start when you do NPM run start, or I guess the alternative is whatever's in main if that doesn't exist. Yeah, that is nice. 

CHARLES MAX_WOOD: So I kind of want to push us in another direction and I'm just curious for us. You do have something that runs, right? And you're running it against real code. I mean, what kinds of things are you actually finding that are red flags or that people are doing? I mean, are you finding malicious stuff? Are you just finding? Hey, this is irresponsible stuff. You know, are there some real doozies out there? It's a funny stuff I mean, I don't know what what are you finding with socket right now? 

FEROSS_ABOUKHADIJEH: Yeah, so we found malware already Gone it taking down 

CHARLES MAX_WOOD: say it ain't so Yeah, 

FEROSS_ABOUKHADIJEH: unfortunately NPM is basic. I mean I shouldn't say unfortunately it's the blessing and the curse of NPM It's that anyone can publish and it's really really easy to publish and that's why we have such an explosion of people Contributing and why NPM is the fastest? growing ecosystem by far. It's a big part of it is that NPM made it like made no gatekeepers and made it really accessible to people. But the kind of flip side of that is that you should think of it kind of like a Wiki where anyone can edit it at any time. And, and if you, if you're downloading a package that has, let's say less than a hundred downloads a week, right? You should basically assume that you're, that that code isn't like, no one's looked at that code because as soon as, as soon as you publish an NPM package, I don't know if people know this, but as soon as you publish an NPM package, it automatically just gets like around 300 downloads without any real human actually installing it just because there's all these bots. And now Socket's one of those that downloads everything and analyzes it or archives it or does stuff with it. And so the download counts are a little inflated by all these bots that are downloading stuff automatically. So if you have a package and it doesn't have, I'd say, let's say a thousand downloads a week, right? Then it might not have any real users potentially. And so-you know, you need to be, and you should basically think of this as like, this is a random code file that someone uploaded on a Wiki page and you know, anyone can edit it and no one's looked at it. There's no, you know, there's no one at NPM that signs off and says, you can be a maintainer, you know, or you can be a publisher, right? It's completely open. 

AJ_ONEAL: Get your maintainer's license. 

FEROSS_ABOUKHADIJEH: Yeah. I mean, some, some ecosystems do that, right? I think before NPM, the norm used to be that you had to kind of ask to be like allowed to publish to, to the open source registries. I think. I mean, maybe that's still the case in some of these older ecosystems. I don't know, does anyone else, has anyone else had experience? I kind of came onto the scene and started in JavaScript land, so I have a lot less experience with like how it works in other ecosystems. 

AJ_ONEAL: Back before that, before you came on the scene, you emailed people with get diffs to ask them to fix things. Okay, so that's, it's a very, very short time. 

FEROSS_ABOUKHADIJEH: That sounds so fun.

AJ_ONEAL: It's a very, very short time in between when GitHub, yeah, yeah, I think you're right. But the window is so small and the exponential curve is so hockey stick that that little tiny window where open source was at a point where people were really publishing a lot of stuff. And then NPM is just, it just, that was, that was maybe a two year gap, right? 

FEROSS_ABOUKHADIJEH: Yeah, 

CHARLES MAX_WOOD: yeah. 

FEROSS_ABOUKHADIJEH: Remind me of the question again.

STEVE_EDWARDS: Chuck asked the question. Chuck asked the question. 

CHARLES MAX_WOOD: I did? 

DAN_SHAPPIR: Chuck basically asked whether, you know, to give us some interesting examples of stuff that you found. 

FEROSS_ABOUKHADIJEH: Oh yeah, yeah, yeah. So we found malware. It was really stupid. And it looked like it was... The thing is, a lot of this malware is really... You look at it and it's like, take process.env and put it into a string and then...send it via an HTTP request to random IP address. So it's like stealing all the tokens in the environment, right, it's like stuff like that. So yeah, so that's the kind of stuff we've found so far. But we also found, I mean, the kind of detections we're including also can find fancier stuff, like trickier stuff. So like, if you look at something like, because we went and looked at all the supply chain attacks that have happened so far in the history of NPM and wanted to make sure there was rules that could have caught all of them. And so like the one that I, the barometer that like, I think would be like kind of a gold standard if we could do it, if we could get to this point is to catch something like event stream. So the event stream attack in 2018 was, I think one of the most interesting attacks because it was obfuscated and it was, you know, it wasn't clearly doing like some of these things we've talked about, it wasn't clearly talking to the network or, you know, doing other things that we would flag, but what it was doing was it, They added a big blob of code at the bottom of one of the files. Do you guys remember this incident, Event Stream? Or should I quickly review it for everybody? 

CHARLES MAX_WOOD: We did a whole episode on it, but yeah. Give the two-minute version. 

FEROSS_ABOUKHADIJEH: Yeah. Yeah, so this guy, Dominic Tarr, who's actually a friend of mine, he's a really awesome maintainer, very creative guy, very generative, kind of has made, I think, like probably over 500 NPM packages at this point, but obviously can't maintain all of those. And when it came to Event Stream, He was, this was actually one of the earliest, it was a very, very early node package. It was sort of back before we had the standard Node.js streaming interface. And so people were still figuring out like, what should the API be for something that is a stream of data? And so one of those attempts was event stream. And then Dominic actually ended up moving on to, to another idea that he had for streams called pull streams. And then later on, he even moved on from that to another thing called push streams that he was working on. So he's, you know, he's just moving on to all these, you know, generating all these ideas. Meanwhile, Event Stream is sitting there and companies are using it now and a lot of popular packages depend on it, but no one's working on it anymore. And so Event Stream, despite being basically abandoned, is still being downloaded by a lot of people and a lot of companies are relying on it. And so somebody reached out to Dominic and told him, hey, we're using Event Stream at our company, and I noticed no one's maintaining it. I would be happy to volunteer to be a maintainer. And, you know, this is totally normal in open source land. Like people reach out all the time and say, hey, I can help out. You know, it seems like...You need help. And so yeah, he did what a lot of maintainers would do, which is they, you know, he said, sure, here you go, and provided access to this person. And then this person, I think it took them about a month, but they eventually added a dependency to Event Stream. And then it did something real at first, but then later on they went back in and updated it to include a backdoor. Basically added a blob of really like kind of tricky obfuscated code that that you couldn't tell it by looking at it, what it did. But obviously it's kind of sketchy to have this chunk of random code doesn't seem to be doing anything. We're clearly doing something important in the package just stuck there at the bottom of the file, right? And then what happened was for a while, no one noticed it because it didn't do anything in most environments. So if you ran it on your laptop, it wouldn't even run. It would just sort of like, it was a no op. It was pointless code. But what it was actually doing was it would look at the package JSON file of the root project. So like whatever app you're building, it would go up to that package JSON, it would go and look at it. It would look at the description field, which is gonna be some random string that's describing what your app does. And it would take that and use that as a key to decrypt the obfuscated code that included in the module. So they had this, they basically encrypted their attack code and they would attempt to decrypt it with this random string, you know, not random string, but the string that's from your package JSON. And so what that means is most people's package JSONs are going to contain, you know, whatever the description of their app is, and then it's not going to successfully decrypt. But they were clearly hoping, you know, that a particular company would would run this code and in that particular company's project, they knew what the description field was going to be. And that would successfully decrypt the malware and then would run. And so it only ran in that one company's system and really tricky, right? And so this meant that basically you'd only catch this if you were looking at the code and you saw this there and you were like, what the hell is this? And started digging deeper to figure it out. But most people aren't doing that. And so for a while, this just went unnoticed. And actually the community only caught it by a very, very random accident that is, to this day, I wonder actually how long it would have taken if we didn't have this really lucky break that helped people find it within the first week of it being published. But what ended up happening was the the attacker used a method in Node.js that was deprecated in the next version of Node, like a couple days later. So a new version of Node came out that deprecated the method that they used. And that meant that anyone running the new version of Node who was using Event Stream would see a deprecation warning and they would say, what is this coming from? And they would trace it back to this blob of really sketchy looking code. And they said, okay, I don't know what this is, but they started asking questions about it. And people kind of eventually, you know, kind of. Unminified it and kind of figured out roughly what it was doing and then they realized wait a minute Like what description field is this like what is the intended effect of this? And so then they realized what if we what if we what if we look at every description field on NPM? Because they're all public. You know a lot anything published NPM has it has a package JSON published with it What if we take all those strings all those description strings and we just try decrypting this with all of those like all 1.8 million strings and obviously if they were targeting some something that wasn't on NPM, then this wouldn't work because the the true decryption key would be some private repo somewhere. But they were lucky and it turns out the thing the attacker was trying to attack was also open source on NPM. And so when they got to that description field, it successfully decrypted and then they figured out what the code was doing and then also who was targeting. And it was targeting this crypto wallet that was written in Electron and used a bunch of NPM dependencies. And so this ended up making it into the build process of this of this crypto wallet and in the released version that their customers downloaded. And it stole Bitcoin from them, I believe. I don't remember exactly. I'm pretty sure people lost Bitcoin and it was really bad for the company. So obviously a really sneaky attack and really, really clever method of delivering it. And this was back in 2018. And I don't think we've seen anything as crazy as this since then, which makes me think like either we've been really lucky or this is happening and we don't know that it's happening. And I hope that's not the reason that I don't want to scare people, but I just, I don't know. I think this is, it's like such a, it's such a, I don't know. I just think it's a thing that, I would be shocked if this hasn't happened at least once or twice more since incident stream and we just don't know. 

DAN_SHAPPIR: And now the question that, you know, the obvious question would socket have caught this? 

FEROSS_ABOUKHADIJEH: Well, we would have caught the obfuscated code at the bottom of the file and that would have been flagged as an issue. And right now, that would have been on you as the user of Socket to go and click on that alert and dig deeper. So if we saw that, say you're using Event Stream in your project and this new version comes out of Event Stream with a new maintainer publishing it, we would leave a comment on the pull request where you are attempting to update to this new malicious version, right? And we would leave a comment that says, this now contains obfuscated code click here to go to the line where it's included. And so we wouldn't be able to guarantee, to tell you that this is 100% for sure bad, but we would be able to raise the alarm at least so that you could then decide, maybe we shouldn't just blindly click merge on this, we should kind of wait a little bit and see what's going on here. Now I think one thing that's interesting is if we could create a feed of these suspicious events and then provide that to the community and allow security researchers and just interested people in the community to look at this feed of potentially malicious events, but which we're not 100% sure are malicious, we could probably speed up the detection rate of these things. You could imagine saying, what if a package is over a certain number of downloads and it had a new maintainer added and it had obfuscated code added, right? And all that happened in a really quick succession? Well, that should probably make it into a feed of suspicious new versions that human reviewers could look at. And if a human reviewer looks at that and for sure definitively marks it as malware, well then we can then take our tool to the next step and instead of just merely alerting you in your GitHub comments that there's obfuscated code in here, go figure it out on your own, we could upgrade that to an actual block. We could actually just block it and say, this version, we're gonna fail this PR and we're gonna tell you in big, big warning letters, we think this is malware, do not merge this, right? And then we'll obviously in parallel go and try to get it taken down from NPM at the same time, because we want to protect everybody, even the people who don't have the socket bot installed on their GitHub account. So that's kind of where we want to get to with it. So right now, it's just going to leave you that comment, and it's going to be kind of on you to dig into it. But I think we need to get some more community features and more of this kind of like, maybe curated set of researchers to do this analysis. And that's where I'd love to get to.

 

So I'm here with JD from Raygun. JD, why did you start Raygun? You know, I started Raygun. It was actually our 11th product that we built. So you know, if you're a fellow software engineer thinking you want to build something and build a business, this was the 11th try, right? And we built it because way back when I was writing more code for customers, I used to instrument my code to send an email to myself when something went wrong. And it would let me kind of get in front of the issue before the customer complained. And so we built a whole product called Raygun for crash reporting initially. I'd expanded out into other areas, but it was really just building a full solution to what I'd been doing years earlier to try and build better software. I love that. Just scratching your own itch. It makes a ton of sense. And I do that too, with some of the stuff that I'm doing either with podcasting or programming. Yeah, absolutely. The most awkward thing was when we actually instrumented some of those prior 11 products, and that's when we realized that about 1% of users will ever actually report an issue. And you go, oh, we might've been a lot more successful earlier if we'd known that. So that's kind of the whole value prop of Raygun. Yep, absolutely. And it makes sense just to put it in there. So folks, if you're looking to try something like this, that'll tell you what your problems are, go check out raygun.com and get a free trial. 

 

DAN_SHAPPIR: Yeah, AJ, but do you really know? I mean, with sub dependencies and sub-subdependencies and so forth. Even if you think you know, you don't necessarily know. That you're saying I'm willing to pay for higher quality secure code. And I'm saying that how can I, as a maintainer, I would gladly take your money, but how can I guarantee that that's the situation? 

CHARLES MAX_WOOD: So I'm gonna untangent this. I think this is another discussion for another day. Tidelift does though offer some level of support for open source packages and they pay the maintainers to be responsive to what people need and do some of the things that AJ's talking about. It's not exactly what he's talking about, but anyway, um, if you're looking for something like that, check them out. And yeah, I think, I think there's a conversation to be had here, but I think it's for another day. So do we have any other questions? Is this going to expand beyond JavaScript and NPM? 

FEROSS_ABOUKHADIJEH: Yeah, definitely. I mean, I want to start with JavaScript and NPM because that's, that's my home. You know, that's my, the language that I spend all my time in. But I also, obviously people don't just write one language and companies have multiple languages that they use. And so a lot of the techniques that we're working on can be applied to other languages without too much work. So especially stuff around the metadata of packages, like that stuff is pretty similar across different package managers. Obviously all the source code analysis that we're doing is gonna need to be customized for each language. And...that'll take a bit of work. But yeah, I think we want to get to this sometime this year, probably like later in the year and make it so that you could install socket across your GitHub organization and just have all the, all the different repos, whatever their language, get the same level of protection. 

CHARLES MAX_WOOD: So the other question I have, well, I guess I have two more because I just thought of another one, but one of them is, so let's say that I run socket against my org and it flagged something and I go look at it and it's like, Oh, well, this is like six levels deep on express. It's not something where I can just go and say, well, just use the newer version because it may not be compatible, right? Or there may not be a fix for it yet. So what do you do in that case? 

FEROSS_ABOUKHADIJEH: Yeah, it's, it's a good point. Like it's one of the things that is really important with, with security tools is that they give you actionable like feedback, something you can actually do something with, right? And that's, that's one of the most annoying things about NPM audit today. And that's why people are always complaining about NPM audit is that it'll just tell you that you have like 15 critical vulnerabilities, and then you'll try to ask it to fix them and it'll won't be able to fix all of them. And so then you're just like, I guess I have 15 critical vulnerabilities. What am I supposed to do? So it's something that I think the best thing to do is to think of socket as like, we're trying to catch the stuff that is just like these anomalous events that hopefully don't happen too often. Like a package got hijacked by a bad guy and now it's doing something crazy. So Hopefully, when we alert, it's like a limited number. I hope that packages aren't hijacked on a frequent enough basis that we're gonna be spamming people with a lot of these alerts. It should be something that is relatively rare. And the action is basically do not update to this bad version if it's an update. Or if it's a new package you're adding, the action is like don't add this because, and don't add this until you've answered the question, why is this package doing X, Y, or Z behavior? And those behaviors are all things like, you know, this package talks to the network, it talks to these domains, these.ru domains or whatever, like you should be able to answer why it's doing that before you go ahead and proceed to include that dependency. So I think the action is, unfortunately, it's just like, you know, it is sort of like a, either use it or don't use it, or merge it or don't merge it at this point, but it's focused on things that are so bad that you really wouldn't want to merge. There's not really like a middle ground. Like if a bad guy took over this package, like,I mean, don't update to that version. That's all we can really tell you right now. 

CHARLES MAX_WOOD: The other question I have is, again, back to the kind of this framework idea or something. So let's say I'm just gonna throw Express out there, right? And so I'm going, you know, I'd really like Socket to run against Express, right? And just see what it comes back with. But Express isn't part of my org, and I don't have admin permissions on that org, right? So I can't just add the bot. So do I have to fork it in order to run socket against it or is there any way for me to just say hey that one's a public repo you can see it anyway just go do your thing 

FEROSS_ABOUKHADIJEH: oh yeah no so socket proactively runs all of the analysis on every open source package on npm so it's already the results are already there for every single package 

CHARLES MAX_WOOD: what about just public github repos that aren't npm 

FEROSS_ABOUKHADIJEH: we don't support that yet but that's actually a great idea you should be able to pop in a URL and you just get the results back yeah that's a really smart idea. No, so right now, I mean, the thing is like, so there's different, so static analysis tools generally, you apply them to your own code base and you want to find like problems in your own code. But the cool thing about Socket is it's static analysis for your dependencies. And so it's designed so that you don't need to like, you don't need to like run it on, it's not like it needs to be your repository to see the results. You can just look up package and get the results back for it without having to like, without it being your code, if that makes sense. 

CHARLES MAX_WOOD: Makes sense.

DAN_SHAPPIR: The beauty of open source is that it's open source. 

FEROSS_ABOUKHADIJEH: Yeah, but nobody looks at the code even though it's open source. That's what gets me. Like no one actually uses the advantages of open source, which is that you can look at the source and you can put your hands on it, you know, and you can modify it and you can look at it and inspect it and all that. They just sort of like treat it like it's a consumer product, you know, like it's, it's like the users of open source have abstracted themselves away from the truth of what the code is doing so much. I mean, that's back to the NPM website, but that's a big part of the problem is that you go to the open source package and you can't even see the source. So you just think of it as like, oh, I guess I just read the docs and it's an API and I consume it and I'm not a participant. You know, it doesn't feel like you can see the source as easily as you should. 

DAN_SHAPPIR: It reminds me that- 

CHARLES MAX_WOOD: Yeah, I think we're trained into that some. 

DAN_SHAPPIR: Yeah, it reminds me that how for years, the documentation for socket.io, and here's another thing that's called socket was basically we don't have documentation, read the source code. That was the documentation for a long time. 

CHARLES MAX_WOOD: Yeah. I was going to say, you know, I think we're trained into that some. I mean, we just get in our cars and just expect them to go. But if you open the hood, you can see all the parts and you can see where they go. And if somebody is willing to explain it to you or if you're trained as a mechanic, you know what they do. I mean, I enjoy working on my car. But before I really got into that, it was. It was a black box and it never really occurred to me that I could look and see what it was doing in a lot of cases. And I think it's the same thing here. We get that with so many other things where we're just expected to consume it, right, without ever asking what's in it. And so once you kind of break through that with your mindset, then you start realizing, oh, no, I can go see what this is doing, right? I can go pick this apart because all the pieces are laid out for me.

DAN_SHAPPIR: Yeah, but Chuck, realistically, one of the reasons that we use NPM or the main reason that we use something like NPM is that we don't want to muck about with all that code, with all that functionality. If I'm going to start reading through all the code, I might as well just write it myself. You know? 

CHARLES MAX_WOOD: That's true. That is true. Yeah. 

FEROSS_ABOUKHADIJEH: And that's part of the reason why the type of analyses that socket does don't focus on things like Oh, there might be a bug in this line of the dependency. Cause like, what are you going to do with that as a user of the dependency? You're not going to, first of all, you're not going to have the context of like, what is this internal function supposed to be doing? And like, you know, what, like, what do I, it's pointless information to a user of the open source package. So the types of things we focus on are actionable things, like high level descriptions of the behavior of the package that you can actually understand. Right. It's not like we're just taking some, some, some static analysis tool off the shelf like ESLint or something and running it on the package and telling you, hey, there might be an error that isn't handled on this line, because that's not something you can do anything about as the user of Express, right? You're not gonna, I mean, it does indicate maybe the quality of the package is low or something like that, but otherwise, you know, you don't want to see a big page of all these potential issues that are all about random, random lines inside the internals of a project. That's just not what people are gonna, no one's gonna find that useful. So we have to focus on the things that are like a simple description of the behavior of the package that anyone could understand without knowing about the internals. Right. Yeah, it's such a trade off. I mean, it's like the car example is so interesting because if we ever had supply chain problem and had to like start repairing our cars because we can't just, I don't know, buy a new one when it breaks or whatever, or if civilization collapsed and we needed to like survive, like we'd all be screwed. That's true. The knowledge is getting lost and it's getting put into the mind. It's just the specialists understand these things but it's also so powerful to have abstractions and to not need to know how your car works. So it's a double-edged sword. 

CHARLES MAX_WOOD: Yeah, well, even if you understood, you know, I mean, if my car blows a hose or has something seize up, I mean, a lot of times your only recourse is to get a new part. Yeah, if there's no supply for that part, I mean, I don't have the capability to make the new part in my backyard. There's just no way, right? So. Yeah. 

AJ_ONEAL: Yeah, why would you get a little bolt fixed on your electric battery that costs $15 when you could just buy a new Tesla for $80,000. 

FEROSS_ABOUKHADIJEH: Well, Apple computers aren't too far away from that. I mean, you go in to get your, you know, your like your key fixed on your keyboard and they tell you, oh, the keyboard is soldered to the motherboard or what do they call it? They call it the logic board, right? And the logic board also has all the RAM and the hard drive and all that soldered together. And so to get your key fixed, it costs $1,000 because they have to replace all the guts of the entire machine. 

AJ_ONEAL: Oh, I don't think it's that cheap. 

STEVE_EDWARDS: Well.Oh man, you talk about hitting the spot. I had, oh geez. Well, my MacBook Pro, my 2019, just one night, it was fine, woke up the next morning, my display is totally toast. You know, it looks like liquids got spilled inside the screen and I went to a shop and it's like, you know, sitting on my desk. It's like 750 bucks just to replace the whole display because the guy was explaining to me, you know, you can't just take out this piece in and replace it. You got to replace the whole display. It takes a week and, oh, sorry.

CHARLES MAX_WOOD: Yeah, but, but yeah, I mean, even the modular stuff where so highly dependent on other things, you know, you know, talking about a car where it is designed for you to be able to pull that part out. Right. Yeah. Anyway, 

DAN_SHAPPIR: yeah, just before we finish at the end of the day, it does seem like, you know, especially for us, when you talked about exposing this type of information that you discover out to the community, that's essentially almost all you can do it's like literally either decide not to use a package or make enough of a stink that somebody actually fixes it or something like that. Because if you're dependent on something and that's dependent on some other thing and that thing suddenly has a problem in it, like what are you going to do? You're like, basically you're stuck. I mean, I guess you could, 

FEROSS_ABOUKHADIJEH: :you can either not update or. 

DAN_SHAPPIR: Yeah. That's more like, like, yeah, the most you could, well, no, forking it is not that realistic at the end of the day. Mostly it's about either being able to not update.Hopefully that's something that you can do or alternatively just not release until it gets fixed. I don't know something along these lines. 

AJ_ONEAL: Forking is completely doable. It is so easy to click the fork button, change the package name, fix the bug, and be in publish. Put it scope it under your own organization. People are getting old. Yeah, the issue though is that any any open up, open up a PR, they'll never address it. 

CHARLES MAX_WOOD: Yeah, but the issue becomes that any future enhancements to the original you don't get, right? You have to keep merging it anyway. 

AJ_ONEAL: But at that point, the original isn't getting enhancements. 

CHARLES MAX_WOOD: Maybe. 

FEROSS_ABOUKHADIJEH: I think that it's both of you are right. I think I'll also add another challenge too, is that sometimes when you fork a package, I mean, if the bug is like three or four levels deep, you actually have to fork. The dependency. Yeah, if you fork the top level one, it's not easy to... Now there's stuff coming to NPM. I don't know if it works well. I've never tried it myself, but there's this overrides thing now, or I forget what they call it. I think it's called overrides, where you can say deep down in the tree, you wanna replace. 

AJ_ONEAL: I couldn't get it to work. 

FEROSS_ABOUKHADIJEH: Yeah. The other thing I saw that's interesting in this same space is Dino now has a feature as of last week that lets you vendor in a particular dependency. So you can...basically say, vendor this dependency, and then it will make a vendor folder with that dependency and all of its dependencies nested in this folder. And so, then it will prefer that over the remote one. And so, you can then check that into your Git repo. You can make whatever changes you need to make to it and then check it in. And I mean, it still has the same problems that Chuck, the same problem Chuck said, where you're not getting the improvements in the future changes, but it does, it-Oh, explain. 

AJ_ONEAL: Sorry, something's wrong with the internet and the audio is way out of sync. So I, but when, so that's the way people used to do it is they used to copy files into their local code. But that, that gets really messy, really fast in terms of mismatch of histories and, and not having the documentation that this was copied in because it's being modified, not having that diff. much prefer the idea of forking and publishing or being really clear about vendoring, that this is modified vendoring, not vendoring for the sake of reproducible builds. 

FEROSS_ABOUKHADIJEH: Yeah, interesting. Yeah, I think that's valid. Yeah, I don't know how clear the tool makes it that this came from this particular source and that it's been modified in this or that way, but presumably the tool would support that, but I haven't played with it yet. I'm glad that Dino is thinking about this type of thing and making that. Making it a little bit easier to to like quickly tweak something that you're depending on. 

CHARLES MAX_WOOD: Yep. All right I'm gonna push us to pick 

AJ_ONEAL: people have thought about this too 

CHARLES MAX_WOOD: the goat people I couldn't help it. Sorry. All right, let's do picks 

 

Hi, this is Charles Max wood from top-end devs and lately I've been coaching some people on Starting some podcasts and in some cases just taking their career to the next level You know whether you're beginner going to intermediate and immediate going to advanced whether you're trying to get noticed in the community or go freelance, I've been helping these folks figure out how to get in front of people, how to build relationships and how to build their careers and max out and just go to the next level. So if you're interested in talking to me and having me help you go to the next level, go to topendevs.com slash coaching. I will give you a one hour free session where we can figure out what you're trying to do, where you're trying to go and figure out what the next steps are. And then from there we can figure out how to get you to the place you want to go. So once again, that's topendevs.com slash coaching. S

 

CHARLES MAX_WOOD: Steve, why don't you start us off with pics? 

STEVE_EDWARDS: Certainly. So before I get to the high point of all podcast episodes of my dad jokes, there's this video that I was hearing about today and I think I've seen it a while ago. And it is an ad for a new thing out called the squatty potty and it's called the unicorn change the way I poop.So funny, so funny. First time I saw it, it reminded me of the classic commercials for that product, Poopery. One called Girls Don't Poop that had me laughing out loud when I first saw them. So I will post the link in the chat. I guess it's a product that was on Shark Tank. We got some funding through Shark Tank as a startup, but it's one of those things that takes the gross and makes it very funny because it makes it all finesse. It's great. 

CHARLES MAX_WOOD: The unicorn commercial is hilarious. 

STEVE_EDWARDS: Oh, it's so good. They even have a video on how they made it and everything that went into it. And it's great. It's really good. So, you know, I've had, over the past couple of years, I was, I happened to get laid off right at the beginning of COVID in 2020. And it didn't have anything to do with COVID. It had to do with idiocy. But, so I was doing a lot of interviews and I went into this one interview and we were talking about salary and stuff. And so the interview says, you're really asking a pretty high salary for someone with no experience. I said, well, the job's gonna be super hard since I don't know what I'm doing. Sorry, come on. Where's my drum joke? 

FEROSS_ABOUKHADIJEH: I laughed even though I was told not to encourage you. 

STEVE_EDWARDS: Yes, I know that. And my drum joke's not working, I'm so bummed. Okay, I wonder if somebody turned it off on me. And then, you know, there's always the classic cow jokes about, you know, what do you call a cow with two legs? 

CHARLES MAX_WOOD: Leaned beef. 

STEVE_EDWARDS: Leaned beef, right, or I lean. And then what do you call a cow with no legs? 

CHARLES MAX_WOOD: Ground beef. 

FEROSS_ABOUKHADIJEH: Ground beef. 

STEVE_EDWARDS: Ground beef, right. So where do you find the cow with no legs? 

DAN_SHAPPIR: Wherever you left him.

STEVE_EDWARDS: Yeah, exactly. Thank you. So just imagine the rim. 

DAN_SHAPPIR: Are those that jokes or are those that jokes or are those the second grade jokes? I don't know. 

STEVE_EDWARDS: I think they're one in the same really. I prefer to think of them jokes that really, oh, there we go. 

CHARLES MAX_WOOD: It's working for me. 

STEVE_EDWARDS: Oh, okay. I prefer to think of them jokes that really move you, but that's it for me. 

CHARLES MAX_WOOD: All right. AJ, what are your picks? 

AJ_ONEAL: Not whatever's going on with my internet connection. That's for sure. 

CHARLES MAX_WOOD: It's.It says there's an issue with your local storage, but anyway, it asks if you're in an incognito window. 

AJ_ONEAL: Yeah, I don't know what that's all about, but everything's been jittery for the last 20 minutes. Get a real browser. Anyway, let me try to get to my picks here. I've got them open. Okay, so first of all, I'm going to pick Go with versions because the Go people... So there's a conference talk, Go with versions, and the Go people have solved all these problems that package managers have. They have solved all...solvable problems. They have empirically solved every solvable problem of a package manager. And in the talk, they explain exactly how the Go package management system works and why it solves all solvable problems and kind of provable scenarios of saying, well, here's all the problems that have these are categorically the problems that have been encountered. And this is how if you take this approach, you get this trade off and this approach, you get this trade off. And these are the approaches that have basically no trade offs that are additional to anything else and has the fewest trade offs overall. So there's no other trade off you can make to get a better system. And the only trade offs that there are the ones that must be made no matter which system you pick. So they have created the perfect package management system and it's just really worth watching that talk and understanding how much care they've taken into that. And then I just kind of a hope for the future that as people continue to create package managers, rather than reinventing bugs, they will build upon the solution that the Go authors have come up with. And if you know, the Go authors are wrong, and there is something that can be solved that hasn't been solved by their solution, which I think it pretty much has, other than maybe some semantics of how you put a keyword, but from overrides to minimum version C dependency, etc, etc, locking validation of git tags. You know, it's just it's all there. The other thing that I will pick this makes me so sad. But it relates to part of the reason that we have these dependency problems. There is a TC 39 proposal for a regex.escape. And it is not accepted on the basis that if you don't escape, if you don't use it, it doesn't work. Which that it makes no sense to me. I've read the comments on here multiple times. I've gone through, there's about 50 comments on the thread trying to figuring out why it wasn't accepted. And it seriously appears to be that if you do not use the function to escape the strings, then the strings will not be escaped because the only example they give as to why it wasn't accepted is that concatenating multiple strings together where some of the strings are using the escape function and others aren't. And to me, that just seems silly. Because if you're going to escape unsafe strings, you should escape all unsafe strings, not half of them. Because obviously, if half of the strings are escaped and half of them aren't, then the half that aren't escaped could have vulnerable expressions of them. But it's interesting to look at if somebody else can shed light on it. I'd love it if you'd reach out and let me know what you understand. But one person said basically, no matter what API you create, the collective ability for people to find a way to use it wrong, will find a way to use it wrong. And this is no reason to not solve a problem for millions of developers to prevent a dozen developers who want to use it in a wrong way from being able to find a way to do so. But it's just sometimes it's these types of attitudes that these are why we have such crazy weird dependencies is because there's an obvious solution that's well thought out and understood. And for some reason it's, it's just rejected. And I did, yeah, again, somebody can, can understand the rationale and it is anything other than we're not including it because it doesn't work if you don't use it, uh, I please like me. The other thing that I'm going to pick because we're talking about it was XTZJS, which is a, it's as close as you can get to native time zone management in that works in browsers and node today. As I was talking about before, the international object gives you times in a very weird format, it gives them to you in arrays according to a locale. And so basically you just have to pick a locale that you know to exist. And then you'll get an array of numbers that are in the order that is specified for that locale. And so assuming that you, you basically can always assume that the, the Maybe the European locale or the American locale is going to be included on the computer even if it's a you know, maybe a Brazilian computer or whatever. I don't know exactly how they I'm assuming all operating systems probably include all locales but as long as you pick a popular one, then you should be able to rely on the order in which the numbers appear in the array. And if you do that, you can actually translate between time zones. But this is It ends up being about 50 lines of code or so. It's very tedious. And so xtzjs is something that I wrote that basically figures out what to do with the numbers in the array by choosing, I think I just choose the EN US time zone, and then am able to translate to any other time zone given that at least one return object has things in a predictable order. It's a silly thing, but it's...It's the simplest solution. It's just not useful if you need to do tens of thousands per second because it makes the operating system call every time through JavaScript. But yeah, those are the things that kind of came to my mind while we were on the show today. So, and then other than that, if you're interested in watching the live streams, check me out on YouTube, CoolAJ86 or Beyond Code Bootcamp for the more structured lesson style content. And I'll see you all around. 

STEVE_EDWARDS: So AJ, was that first pick? Could you say that was a new addition to string theory?

AJ_ONEAL: I think we could, no, the first one was version. 

STEVE_EDWARDS: The second one, sorry. I lost track of all of them. Sorry. 

AJ_ONEAL: No, yes, it definitely was a new addition to string theory, but, 

CHARLES MAX_WOOD: but only if you use it. All right, Dan, what are your picks? 

DAN_SHAPPIR: Okay. So my first pick is a, actually a pretty sad one. Uh, the day that we were recording this is the very same day in which Russia invaded the Ukraine and my previous employer Wix has several offices in Ukraine. So I actually got to visit there a couple of times and I know quite a number of people who live there and work there. And I'm really worried for their safety. These are really nice people who really celebrate their freedoms and the fact that they won them not so long ago. And all of a sudden it seems that these freedoms could be taken away from them. In fact, their lives could be taken away from them. And that's...really scary and sad. By the way, in this context, something that is really nice is the fact that Wix actually offered all their almost 1000 employees in Ukraine the option to fly them out with their families and and lodge them all expenses paid by Wix, which I think is pretty amazing thing to do and you know, for as long as it takes. So I hope that as many of Wix employees that we're offered this option actually took it for them and their families and that they are indeed safe right now. In any event, it's really sad and I really hope that by the time this episode actually airs, this will all be behind us. I don't know exactly what will happen though. So that would be my first pick. My second one is much, much more upbeat. I'm reading a book that I actually find to be fairly enjoyable. It's called Uprooted by Naomi Novik. It's a standalone fantasy book. We don't see so many of those anymore. And it's like a breath of fresh air to know that you read a book and it's a good story and it will be done by the time when you finish the book. And like I said, so far I'm enjoying it and I really highly recommend it. So it's Uprooted by Naomi Novik and those would be my two picks. 

CHARLES MAX_WOOD: Awesome. All right. I'm going to throw in a few picks. I just want to remind people, go check out topendevs.com. Check out the workshops, meetups, conferences, the whole shebang. I did a workshop yesterday about kind of long-term planning your career, figuring out what you want. Going to be doing one on staying current, uh, this next week. So anyway, keep, keep an eye out for those. The replays will be part of the top Endevs membership. I'm going to pick a board game. I'm actually going to pick, so my wife and her friends, they play pandemic is kind of their game. And at one point I had bought her a bunch of the expansions. And so I said something like, will you have all of the pandemics? And she said, no, not even close. And so for her next birthday, I got her all the rest of the board games. This one it's, it's kind of a different spin on it. And it's, it's funny because we lose way more often than we win, but it's still fun. It's the pandemic thulu game and basically have cultists and monsters that are rampaging through this town and you're trying to fight them off while you seal the gates before the old ones come back. Anyway, it's super fun. So I'm going to go ahead and pick that. And then, man, I got so many other things that I'm thinking about right now and going through. One of the things that's just been kind of top of mind lately is, so I'm getting ready to actually run for the school board for my local area. If AJ has kids in school and they go to the public school, they'll be in the school district for the, that I'm going to be running for the board on.

FEROSS_ABOUKHADIJEH: Good for you. We need more people to do that. 

CHARLES MAX_WOOD: But yeah. So my point, I guess, and my pick is if you care about what's going on in your area, often running for local office gives you more ability to affect outcomes, right? So if you're concerned about what they're teaching in the schools, run for school board. If you're concerned about what they're doing in your town and how they're zoning your building or whatever, you know, go. The zoning boards for your city are usually volunteer. You just start showing up and you can get appointed to the zoning commission without actually running. You can run for city council, you can run for mayor, you can run for your county government if you're in the US. Or if you're not in the US, then I guess it depends on however they demarcate their regions, but right, or states or whatever, right? Here in the US, most education stuff runs through the state legislature or the state school board. And so if you want to be involved, if you're worried about this stuff, And I'm just gonna encourage people, and I've done this before on the shows, go run for local office. I'm not gonna pontificate on what the particular issues are that I care about or whether somebody's right or wrong for agreeing or disagreeing with me. I feel like every local area probably has a different mix of people and you're probably gonna care about different issues depending on where you're at. I think some of them will be the same more or less everywhere, right? If you're running for school board, you clearly care that the kids are getting a quality education, but you may differ on curriculum or what things they can and can't use or what things they can and can't talk about with kids and Where you empower parents and things like that, but at the end of the day I mean, that's that's ultimately what got me going and what has me running So I just encourage y'all to get involved, right? I think the more people we have involved in paying attention the the better off we are in a lot of ways Just because at that point, I think a lot of times the politicians especially at the national level They get away with voting on things or doing things that their electorate wouldn't really care for because it's just so far away and so far removed from what they're doing, but the local stuff isn't. And if you're involved and you're paying attention at the national level, then you can make a difference there too. I've talked to enough politicians to know that if their phones start ringing and keep ringing, then they start really thinking about their position because they know people are watching what they're doing. And that's...often serves well enough to keep them honest as far as, well, if I vote, even if I want to vote this way for, you know, whatever reason, you know, selfish reasons or other reasons, if I vote that way, they're going to vote me out at the next round and I want to keep doing what I'm doing. And so anyway, I'm going to quit rambling about it, but I think, I think it's important. The other thing is, is that they kind of create this idea that you're at least in the US, like your, uh, congressman or your senator is so far away. And the reality is that they're not. Sometimes they make it hard for you to get to them. One of my senators is much better here in Utah about taking input from people than the other, but there are ways to get their attention. So anyway, so get involved locally is my other pick. For us, what are your picks? 

FEROSS_ABOUKHADIJEH: So I'll pick socket.dev. Obviously you gotta shout that out. You can check out socket if you go there. You can install the GitHub app, it's free, and you can get your repositories protected. But I'll also shout out If you go to the footer on Socket Dev, we have this really cool thing called removed packages. And you can check out packages that have been taken down from NPM for security reasons. And it's really cool. What we do is we basically watch NPM, and any time a package is removed, we add it to this page. And so you can actually go and see examples of malware that have been taken down. And it's really fascinating to look at. So we'll shout that out. And yeah, you can poke around and see the kind of stuff that's on there. You'll even notice sometimes there's packages that include company names. Like I'm looking right here and I see there's one. It's like Lyft, Lyft service, something or other. There's something about Yandex search engine UPS. I mean, so there's, there's like packages where people have tried to name them so that they get, they do what's called a dependency confusion attack where a company uses a package name that's an internal package name that is not supposed to be like as a private package, but then somebody will go publish a bad guy will publish something on the public registry that has the same name and then hopefully they hope that the internal tools get confused and install the public version instead of the private version. So you can find all these examples of that on there. Here's one called Netlify dash build, which is like not from Netlify. It's like some random. Yeah, it's deals. I'm looking at it here. It literally just steals data and the data that it's sending is like all the environment variables, the package JSON contents, the username of the machine that you're on, the host name of the machine, that kind of stuff. So just like who is running this or sort of stealing that info and sending it off. Anyway, so it's just interesting to see what is actually in these malware packages and you can get your hands dirty. I mean, obviously don't...Don't run any of these things. Don't copy the code and run it. But you can view it on our website safe and it's pretty fascinating. I'll also shout out Wormhole, which is a website to let you share files with end-to-end encryption. It's basically a successor to Firefox Send, which was this free service from Mozilla that did the same thing before they shut it down. So Wormhole is made by the same company that makes Socket. So, you know, I worked on Wormhole. In fact, this is what we worked on before we built Socket. And it's actually the inspiration for Socket because we wanted to make it possible to send files, you know, securely and without the service seeing anything about the file content. And like the server should see literally nothing about your files. That was our design goal. And I think we did a pretty good job of it. I mean, everything is end-to-end encrypted directly in the browser before it's sent anywhere and the data all expires within 24 hours. So the links just go away, so your data doesn't stay online forever. But one of the things we ran into when we were building Wormhole was, how do we trust our dependencies? Do we know that these dependencies aren't backdoored or aren't gonna have some kind of an attack that affects the security guarantees we're trying to give our users? And so that's kind of when we started looking around for solutions to the problem and ended up realizing there wasn't anything good. And then we went and built Socket. Well, we built something to protect Wormhole and then we realized, could help a lot of other people. So we released it as socket. But anyway, Wormhole is still online. It still has a lot of users and it's pretty handy to send files quickly. We tried making the user experience really, really good. So you can actually, there's no account needed and you can just drop a file on the page and you immediately get a link that shows up that you can copy and paste to somebody even before the files are done being uploaded. So we give you the link right away. And then if the downloader person clicks the link, even before it's finished uploading, it'll actually start to download on their computer because we stream the data from the uploader's computer to the downloader's computer using WebRTC, peer-to-peer connections. So it's like really, really cool, fancy stuff we did to make it literally the fastest way to send files. Anyway, so check it out. It's something that we hopefully, we intend to keep running and providing as a service to the community. It's totally free and no account needed. And the last thing I'll shout out is Chakra UI. So Chakra UI, if you haven't heard of it, it's a web component, kind of a, like a UI component library. It's for React right now. I think they also have a Vue.js version as well, but I use the React version. And it's this really nice set of polished React components. They're really flexible and accessible by default, and they look pretty good. And what's nice is they don't look like the material, you know, kind of, if you use Material UI, like your site kind of looks like a Google product in a way and everything starts to kind of look the same too. You start, you see material everywhere. So I like Chakra because it's just, it's different and it's really well designed components that don't have such a distinctive look that you can immediately spot what component library they come from. But the, and the team that work is working on them is really good. It was started by somebody named Sigoon Adabayo from Nigeria and he was a design, he worked in like a design shop. He was constantly making components for these client projects and he got tired of reinventing the wheel. So he kind of put all of his learnings into this Chakra UI component library. Anyway, it's really cool. Check it out. I recommend it. We use it on Socket and on Wormhole and it's working great for us. So those are my picks. 

CHARLES MAX_WOOD: Thanks for coming for us. This was awesome. If people want to connect with you online, what are the best places for them to do that? 

FEROSS_ABOUKHADIJEH: So my Twitter is just for us. My name is F-E-R-O-S-S. My website's frost.org. My email is just myname at myname.org. So yeah, people can feel free to email me if you have security questions or want to chat about open source or supply chain security anytime I'm down to, I like to meet people. So just feel free to email me or reach out on Twitter. My DMs are open. And then of course you can check out our work and what we've been talking about on this episode at socket.dev. 

CHARLES MAX_WOOD: Very cool. Well, thanks again for coming. And until next time folks, Max out.

DAN_SHAPPIR: Bye. 

FEROSS_ABOUKHADIJEH: Thanks so much for having me. Appreciate it. 

AJ_ONEAL: Adios. 

 

Bandwidth for this segment is provided by Cashfly, the world's fastest CDN. Deliver your content fast with Cashfly. Visit C-A-C-H-E-F-L-Y dot com to learn more.