The Rogues discuss security with Rein Henrichs.

Special Guest: Rein Henrichs.

 

Alright guys, I think I got to go be productive today. You have this job thing, I can’t figure out how to get rid of.

Hey everybody! And welcome to Episode 59 of the Ruby Rogues Podcast. This week on our panel we have Avdi Grimm.

Hello from Pennsylvania!

James Edward Grey.

Hello from outside of Pennsylvania!

I’m Charles Max Wood from teachmetocode.com. And we also have a special guest, and that’s Rein Henrichs.

Hello from Portland!

Portland? Awesome.

Not bad.

So what do you do in Portland?

I work for Living Social.

Oh! Like everybody else?

Like everybody else.

Except us. Alright, well do you want to introduce yourself really quick? And then we’ll get into our topic.

Sure! My name is Rein Henrichs. I am a Ruby developer and web developer. I’m a distributed systems builder. I’m somewhat of a security nuts, mostly because I got hacked a couple of years ago and that really sucks. And so I’ve been doing my best to make sure that doesn’t happen to me or other people ever since.

So that’s kind of cool. Can you tell us like some circumstances about that?

No. Yeah.

Story time!

Yey! Story time! So I was working at the company PHP Fog, which is now AppFog, and we were building out a pass, sort of like Heroku for PHP. And we were in that early startup; let’s just get something shipped out the door that people can use. And we weren’t really getting security that much. And what you might expect happened, which is it that someone else too it more seriously than us and hacked us. And it just looked really, really bad. They got root access to some our servers. They got access to our Twitter account. I mean it was bad. And so I was actually, I already scheduled to give a Ruby on Ales talk about how awesome our pass was. And I thought maybe that talk wouldn’t go over so well right after we got hacked. So then that talk became about how shitty it is to get hacked and what you could do when it happens. Those are some the lessons that we learned.

I was going to say, “Nice!”, but it’s not so nice.

Also, I got really hung over that day, so I think that helped after that error, sort of pitiful humbleness that is appropriate.

Cool. What kind of mistakes do you people usually make?

I think I can separate that into 3 basic areas: there’s preparedness, then there’s responding to the actual incident when you get hacked and then there’s how you disclose that to your users and to the public. And I think they’re all important. The way I look at it is, the longer your site stays on the internet… the longer it stays on the internet… let me rephrase. As the site is available on the internet for longer, the probability that it will be hacked approaches 1. So it’s sort of a Godden’s Law for Hacking because the internet is a scary, dangerous place. And you’re putting you app out there into the wild and it’s a hostile environment where people are going to be trying to hack you. And that’s just how it is.

Yeah. I think that makes sense. So you talked about preparedness and stuff and you’ve already said that when it happened to you, you feel like you guys weren’t taking it seriously and stuff, and then I know I’ve seen you say on Twitter, security is job zero or stuff like that. But you know, at what point, seriously, if I’m trying to put something together and stuff, obviously I have a ton of concerns about just getting something out there and things like that. At what point do I really start taking security seriously?

Well, what I’d say like to say obviously is right now or yesterday. But in the real world, it’s about risk management. It’s about assessing risk and then figuring out what the potential costs of certain actions are, or inaction. And I think one of the problems is that we tend to greatly underestimate the downside risks associated with security breaches. And it’s not just immediate loss of revenue from decreased sales or things like that. It’s also potential legal liability if you’re storing credit card data form quality or things like that. And it’s also loss of goodwill and good faith in the community and with your users. There’s lots of downside risks that I don’t think people are adding to the equation that if try to decide how much emphasis they should put some security. And the worst thing of all is to just not have that conversation, not think about it, too just not take that, you know, one day while you’re launching this product and go. What are we going to do about security and coming up with a good plan.

So how do you know you have a good plan as oppose to, maybe just being ignorant, and saying, “We’ll just put SSL on there and that’s good enough.”

I feel like you need to start out by thinking like a hacker. Think like I wish that I could come up with a better term than “hacker” because that’s become…It used to be the hackers were people that wrote code and now it’s hackers who are people that break code. And I think that’s bad but that’s the language the people are using. So I think you start up by thinking like somebody who’s trying to attack your system; you look at… if I wanted to break this system, some sort of you become a pentest-er, right? If I want to penetrate this system, how would I go about doing it and you analyze those risk areas, those vulnerabilities and figure out a plan to start improving.

I thought that was one of the cool things. Rein sent us a bunch of links and stuff to kind of look over about security and stuff. Well, there’s a lot of stuff in there. We’ll put some of the cool links in the show notes. But one of them I thought is good, it was just talking about how it’s really always a process and you’re at whatever point you’re at, but when you’re improving all the time, then security’s getting better.

Right. One of my favorite sayings in the security world is Drew Schneider’s saying, which is that “Security is a practice, not a product.” What that means is you can’t buy security off the rack. You can’t go to Intel and then say, “Make me secure.” What you can say is: I want a process for improving my security overtime. I want to know how I can discover the vulnerabilities. How I can respond to them and I want to know that today, my site is more secure than it was yesterday.

Did you guys see the RailsCast episode? It was either this week or last on Sidejacking. Did you guys not?

Yeah. I saw it.

I thought that was pretty interesting I mean it’s about how we take a HTTPS for a log in. And then when you’re logged in, we drop it back to normal connection and by doing that, it would make it easy to grab the cookie or plain text. It’s transmitted in plain text, grab the cookie and then we have your session. And…

Sorry. Go ahead.

Well, I was just saying that like it seems like that’s a mistake like I’ve seen every website on the internet. It’s incredibly common.

And the list of vulnerabilities keeps going everyday and attackers are getting more and more sophisticated on how they’re approaching things. What’s the name… what’s that… there is a Firefox extension that you can install that will let you, basically, steal other people’s session.

Are you talking about FireSheep?

Yeah. FireSheep.

I mean that’s the thing where I can go and log in to your Facebook because you forgot to use HTTPS.

And if you go to any place like co-working facility or something like that, you know, people could grab tons of stuff there unless you’re going to do like SSH tunneling or something.

Right. Or you have a VPN, which I think is a pretty good idea for everyone that wants to connect securely to be able to find data.

Right.

So we could spend literally, probably, 5 podcasts just talking about all of the things you should be doing. And I don’t want to do that. What I do want to do is I want to stress on people how important it is to start asking yourselves these questions: How am I storing passwords? How am I SSH-ing into these boxes? Do I really need 5 different SS… 5 different boxes all open to the world on SSH? Or Can I have the Bastion hosts in front of that? All of these questions are possibly you could be asking yourself about your systems and coming up with good answers.

So is it safe to say if you feel like you don’t know, like you know you need security now but you don’t know exactly what questions you should be asking, that it’s time to bring in an expert? Just for like a consultation to figure out like which areas you should be looking at.

I feel like once you’ve done what you think is your due diligence and said, “I don’t know what else to do.” I think that’s a great time to have some audit your system and give you some more good ideas. But do your own due diligence for as long and as much as you can. There are a lot of good resources out there. It’s like any, you know, broad deep subject, it’s going to take a while for them to be expert on it.

Right. So did you want to get in to some of the best practices on some of these scenes like, how am I storing passwords? or, how to manage SSH-ing into your machines.

So we can talk about BCrypt and SCrypt and we can talk about bastion hosts. We talked about a lot of the tactics of security but I’m really more interested on a strategic level. I think most web developers and good CEOs, if they set up priority for security; they can go find these examples themselves. So I don’t want to talk about the tactics as much as given, LinkedIn and eHarmony and Last.fm and League of Legends and Sony 20 times. I get this feeling that…

Sony 20 times. I love that.

That’s so hilarious. It’s a PS3 user both laughing and crying inside.

20-20 times what we have. I gave the feeling that we keep talking about security and then it would keep not getting secure. So I just want to keep hammering in how important it is to actually go look at your site right now and say, “Is it secure?” Someone could be hacking your site right now and many of you have upgraded your Rails to 3.2.5 whatever it is to prevent if we said, we disclose the SQL injection at that.

Right. That’s a good question. Let me ask you kind of a strategy question along those lines. You’re talking about Rails and it grows and improves over time. Yehuda has one point in the past that even if he’s developing an API, he would rather use Rails than anything else like, say, Sinatra, which may be common for that. Because he knows all the things that Rails does for you and that they have, like, they do have a place where people submit security issues and they do look at those and they do check into those. Do you think that the framework you choose is part of that strategy? Is it irresponsible, security-wise, to choose Sinatra which probably reads the details to you or not?

Like you totally set him up. I want to hear this.

If you don’t the resources to bet it yourself, I think go with something that someone else can bet for you and that means probably Rails. Because a lot of people have done a lot of work to make Rails secure and the list of security vulnerabilities that are known for Rails is actually a lot smaller than most similarly popular frameworks. How many of you actually looked at the security vulnerabilities list before I sent it?

I didn’t look at it before you sent at it to me. But I did look at it from your list.

Were you aware that that was a thing?

Yeah. Actually, I knew about it because of Yehuda’s talks. You mentioned that they do have that and they always have someone look through on all those stuff.

So right now, the total number of vulnerabilities is 25 and if I switch to something like WordPress, which I could be able to do… I have a lot of things and a lot of lists. When I’ve looked at other things like Django and WordPress, I found it on the whole Rails tends to have fewer vulnerabilities than most of these frameworks. But with Sinatra, what you’re getting is something with a lot of unknowns. Which means you have to do more due diligence to feel comfortable shipping that product, I think.

And to be fair, I think that Sinatra, like security is not really very much in the scope in what Sinatra does.

Right. It doesn’t do authentication. It doesn’t do database out of the box. So all of those things, if there’s an SQL injection in vulnerabilities in Sinatra, it’s because you add it.

Right. Yeah. But I think they consider that your domain.

Which kind of sucks though because it’s a really hard domain and putting that on to developers who really just want to ship something, like what you’re talking about, it’s not passing the buck but it isn’t their responsibility but it does make for a lot of extra work for developers who care about security when they want to deploy out something like in Sinatra.

So to bring this like a little bit more concrete, can you think of an example of something that, they might not think of that Rails is handling for you, security-wise, that’s something like Sinatra is leaving up to you.

Well, I bet a lot of them gets pissed off by the CSL wrap opens they’re getting buried into their forms like Ajax number. But that actually prevents one of the more common exploits and vulnerabilities in those websites. And it does it magically for free for you and only sometimes gets in the way.

Yeah and Rails is kind of the, any unique position to do that, right? Because it’s only kind of because it’s a full solution…

Right. They control so much of the stuff.

Right. But it’s practical for them to actually have that feature because they have to put something in the header of your pages. And then, they make you go to the form helpers, so that they can modify your form along your way, you know, things like that. Yehuda has also talked about that particular exploitance and said that he’s surprised when he goes to other communities and looks around for that particular solution that h’s all, “I usually find some snippety code. You’re supposed to paste in and it’s right about half of the time”, you know, or something like that.

I would generally not trust random snippets of code in the internet to protect me from serious security vulnerabilities. [laughter]

What!?! What you say it isn’t so?

So I’m a little curious. I mean, what areas should you be considering if you’re trying to audit your site for security? Obviously, there are things that you can do in your code but are there other areas that people generally miss? Or are there areas that you need to make sure you thoroughly cover?

So the general way that you go about this is that, you make a list of all the imports into your system. That can be SSH, that can be you URL bar, that can be all of your APIs; anything where someone can get data into your system somehow. Those are all vectors for a path. And you have to catalogue them and you have to decide what are the potential vulnerabilities for each and how do we mitigate them. It’s a lot of work. And the larger your system is, the more moving pieces it has, the more exposed it is to the internet, the more work it is. But again, if you leave one of those vulnerabilities unchecked and you forget to mitigate it and someone breaks in there, your entire system could be compromised. Sometimes it’s just the one that we could link in the chain which brings me to containment which is very important. Containment is the idea that when you do get a breach, because I really do feel that for most sites in the internet is just a matter of time or attention because most sites in the internet are probably insecure. What happened with LinkedIn could probably be happening to almost every site in the internet right now. In some fashion, if someone tried hard enough.

So why are you picking on websites if there’s so insecure. I just don’t know if they can take that.

Because it’s fun. No, because someone needs to.

There you go, playground boy, huh?

So I saw you talking about containment after the LinkedIn incident and one of the things you said that really resonated with me was, they have identified that they have this problem, and they’re trying to work it all out and they’re site is STILL UP, and that was an obvious mistake because they that they have a problem.

Yes, this gets into incident response and also the way they disclosed. One really bad thing they did was that they kept a compromised system running. And that’s really bad because they don’t know the extent of the compromise, during the compromise. They could have just gotten the database but what if they got more? What if they’re rooting your system? What if they are on that system as some non-privileged user trying to escalate privilege into your system as we speak?

Well the other problem that I see there is that if you go out and you say, “Look, we have a vulnerability.” Even if you don’t give the specifics, you basically are telling people that the system’s still up, you can come and find a way in.

Right. And you’re also telling all of the users of that system that these passwords may have been stolen. Yeah, you can reset your password, but then the hacker could just take it again. So good luck with that.

That’s why I like to keep all my sites on a box that sits on my desk because if I just need to compromise it, I just pull that work as well.

Yeah. You’re just going to turn it off. So that’s sort of the first step in incident responses like you’re saying, “Take any affected systems down. Pull the plug.” And one thing that’s really nice when you’re designing these systems is to have that big red button. That’s something I talked with building distributed systems is have a big red button that you can push that turns that system off. There lots of good reasons that you could need that button. Maybe you’re actually just trying to test the reliability of your system to partition-tolerant, maybe your trying to… maybe you just need some fallback solution for DDoS attacks. Maybe that system got hacked and you can shut it down quickly. Because the longer that system is up, the more likely that they can jump off from that system to another system. And the next thing there is, if you’re wondering if the system affected and you don’t know the answer, the answer is “yes”. You always have to assume the worst case scenario, not the best case scenario, which is something LinkedIn did really poorly for their disclosure. What they did is they said, “We have discovered the users who are at most risk and we are going to reset their passwords.” What metric could they possibly use for that?

Right.

How do they know which users were… they didn’t… as far as I know there are some passwords they never reset because I never got an email from like that. So they must think that my password was never compromised. But what information are they basing this on? The hacker posted the list of pass… of shots. Is it really that inconceivable that the hacker might have not posted half of his list? [laughter]

Can they possibly know that?

They may have not been utterly straightforward. What?!

Yeah. Well, you know, we know that they didn’t hack Avdi. So that’s all that really matters.

What kind of crazy topsy-turvy world do we live in, where system crackers don’t just come completely clean.

So if you look at what Last.fm and what League of Legends did, they actually did the right thing, which is they said their passwords have been compromised. We don’t know the extent so we’re going to reset everyone’s password. And we know this is inconvenient but you think it’s better than better. That’s the right thing. That’s the right thing to do when your database is compromised and people have and some has their shots. You assume that they’re all at risk because they are.

My gosh. There was a post I saw from using the LinkedIn SHAs, they figured out just some common passwords. I saw there was of like the Top 30, you know, its like, “Oh my gosh!”

1, 2, 3, 4, 5 and things like that?

Yeah. They’re so bad.

That’s the password of my luggage.

But you as someone building it’s like you can’t depend on your users to not be stupid. You don’t have to fall over that. You have to protect them sometimes from themselves.

Right.

If someone wants to choose the password: 1-2-3-4-5. Yeah. You can try to make that an invalid password. Or you could just use a password hashing system that won’t be broken in two hours when someone buys a quad-GPU and the soft box and rose and a lot there on it.

Oh geez.

That’s so awesome.

And you can do that. You can use BCrypt which is computationally hard, which means you can dial and how much computation it takes to generate a SHA. So the thing about most pictograph caches like MD5 and SHA is that they are designed to be really, really fast, which is great if you want to hash something but not great if you want to hash something in such a way that it can’t later be brute-forced.

Right.

Because that it really, really, really fastest is now a liability. So things like BCrypt, they’re computationally hard then what you dial in something else and said rather than taking 10 milliseconds, let’s make it to take a second, which at the point where you’re generating that SHA, is usually okay and maybe not a full second but even 10 or a hundred times faster or slower is better. And then there are things like SCrypt which are memory hard which means they take a lot of memory to generate the hash. So the hash, that’s 128byte may take 297 second to generate. And both of those things may brute forcing basically impossible with the maximum. I think with the maximum computational hardness in BCrypt, it takes on average until the heat death of the universe to brute-force a password.

Plenty of time.

You just need a better machine.

I like to spin up a bunch of universes in my personal black hole.

It’s nice that you can do that.

Alright. The cloud universe service. So let’s ask a question about that though, like you know, preventing things from being brute-forced is great and all but I’d seriously in that example of LinkedIn hacked passwords, I mean in the Top 30, there were so many that would fall to a dictionary attack or, you know, things like that. I was so shocked at how it extremely bad they were. Should we concern with like user education as well? Since that’s also…

We could try, but it won’t happen.

Yeah.

We have to accept the situation on the grand because the users are always going to be generating shitty passwords.

I like that more and more sites have the little like password strength meter now.

Anything we can do to educate users to generate better passwords or to use 1Password or LastPass or things like that is great. But we can’t expect that to happen. We can’t expect…

So many systems I think users end into bad habit over the years. Like particularly, the ones that force you… you don’t usually find us on the open web but like in in-house systems, internet systems, you would have the ones that force you to change your password every few months. And that’s terrible because all its doing is it saying, use the dumbest password algorithm that you can think of.

It’s theoretically great because it means that those passwords to constantly being reused and it prevents the human factor of what if you accidentally share your passwords. Also, forgets about those a human factor how people would respond to this…

Increment by one. And then if you just allow it, then its increment by one and add a character.

And stick the new one to your monitor.

Yeah.

Yeah! Exactly because that’s secure.

So I always take the approach that you have to be secure enough that you are at least secure your users will be safe. So that guy, his password is PASSWORD. You want to try to make them safe.

And yes, he didn’t just make that one up. It was one of the Top 30.

The password is PASSWORD. REIN: Yeah. You hear that. No one does that. People do that. A lot of them do that.

Did it trip it up then that they have a username field that they couldn’t fill in with USERNAME because it was already taken?

Where’s the any key?

I think that’s in the lock of the ESC key.

So then, when you have this instant, you get hacked… you’re preparedness, you talked about containment and stuff like that then what’s the ending? You’ve solved it. You’ve taken care of it.

What are your responsibilities to your users and so?

So let me step back to response for just a second and give you the list of the things you’re supposed to do. It’s a short list.

Okay.

Take the affected systems offline but don’t destroy them yet. Keep them available for forensic analysis. After you figure out how you got hacked and you’ve created and tested effects, rebuild any of the affected systems from scratch, nuke them and then nuke the old ones from space. You don’t want any of that old code infecting your new systems because if those systems are compromised, you don’t know what could still be on those systems. There could be backdoors. There could be Trojans, there could be all kinds of things that could make your system… a system once compromised is always compromised is how I look at it. And if you think you’re smart enough to fix that system in place and just to leave it standing, then you’re too stupid to fix it.

So let me get this straight, just to make sure I understand, let’s put that in practical terms. If I’m running some side up on EC2 and I get hacked and realized that it’s compromised, then I take it offline and list from the public and I started poking around in it and see what happened, figure that out, and then at that point I get my fix and all that: what you’re saying is really I need to take that server all the way down to shut it down, bring up another one…

Yes, disconnect it and the very least, disconnect it from the internet and from all of your other systems. Isolate it. Put it in a little glass cage. Make sure it can’t talk to anything.

Right.

No talking.

And then when I have it figured out, I need to bring up like a new instance and redeploy my app on that.

Exactly. And that’s a good reason to have configuration management in place. So you’re using puppet for this or you’re using SHA for this or something like that. When PHP Fog got hacked, we brought down 2,000 servers in about 15 minutes and then brought them all back up over the post in 24 hours.

Wow!

Not because that was the easy thing to do, but because it was the right thing to do to protect all of our users.

So one of the things I’m wondering about is, let’s say you get hacked, what’s the right way to tell your users, “By the way, your credit card might have been taken?”

Yeah. I’m glad that we’re talking disclosure because…

Oh! We are talking about Sony again? [laughter]

This is the thing that… I think this is the thing that is most damaging when you get it wrong because this is the actual public face of the incident. This is what people are responding to. And aside from potential liability issues with passwords stolen and identities stolen and things like that in my experience enlarging what has materialized. The loss of reputation and trust with your users could be a big problem. So you want to make sure that you disclose properly so that you don’t come across looking like a complete idiot and they would not trust you again. So the first step is disclose as soon as you know that you are hacked and you’re going to have to disclose because the system is already down and people are wondering why the site’s up. So you don’t really have a choice. You have to tell them something because chances are you’ve already affected quality of service in some point that requires some sort of disclosure any way.

The unit that broke, it’s not our fault.

Right. But the thing about disclosure is inevitably, the information comes to light of how stupid you really were, right? So start there. Don’t try to hide things. It will just look bad for you later when they learn it form someone else. And also recognize that when this happens to you as a founder, as lead engineer, as off sky who has a lot of ego invested in that product in it not being hacked and not looking bad, you’re going to be hurt, and you’re going to be angry and you’re going to be in a really bad say that might have to talk to users right now. But that’s too bad because you still have to do it. So you have to try to fight that sort of instinctual fight or flight response that’s going to make you avoid the situation or lash out at people. You have to push that down. Step aside. Step outside of yourself and try to crack a rational message for your users. And if you’re having trouble, talk to someone you trust. Have them look at what you’re saying. Analyze it for tone and how you’re pointing yourself out there in front of people. Make sure you’re saying what you want to say, how you want to say it. I think the human factor here is really important and it’s something that often overlooked.

I guess, hopefully, you’ve always been seeing yourself as on the side of your users, as an advocate for your users and so. I guess the trick is to continue to see yourself as being, first and foremost, an advocate for the people that use your system as oppose to sort of on the other side.

Right. I think it’s important to remember here that we’ve got hacked; you’re actually not the victim. It would be your users are the victims and they are the ones that you need to pet.

So I’ve also noticed like, for me just reading disclosure reports that like… it seems like the more well done the disclosure report is, the more detailed and showing exactly how we went wrong, the more I’m like, “Wow! I’m really glad they wrote this. That was nice. Now I understand what happened.” One of the great ones I remember was Amazon a while back had one of their really big outages where they screwed up having right router their network traffic and…

I remember that because it took us down for a weekend.

Right. And it was really bad, right? But the write-up on that was FANTASTIC.

Yeah. And you got the sense that they didn’t realize that that was a potential problem but now they are more aware of that problem than anyone else in the universe. Because they’ve taken the time to really analyze the hell out of it. You trust them more now than you did before.

Yeah. It was also… I felt like they said… like it helps you be aware that something like this could go wrong. Basically, they broke routing and that ended up taking down a massive portion of the internet. It was sad.

It’s because everybody uses Amazon, right?

In a lot of ways. You’d be surprised how many things were affected by it. Like people uploading to S3, uploading images to S3, those would start failing. That because of the way certain extensions handled those failures, they would get exceptions and then we have like email us when an exception occurs but because that started happening in such a massive volume due to all the outages, then that caused other problems. I mean it’s a really trickledown effect.

Interesting. Do you guys know where we can find that write-up? So people can go and look at it.

Alright. I’ll look for on Google. It was on that blog, I think, after the whole...

I think it was their US West Availability Zone, I think. They’re EBS logins, it was a cascading failure

of having… it’s like what you are saying. And essentially if they started trying to route data to new boxes but they’re bouncing algorithms were such that those boxes then became flooded and they went down that just cascaded from there.

Right. So one thing that just occurred to me that I’m wondering about and I’m going to change the topic a little bit here is, you mentioned that you should be protecting your users from themselves, in the sense, let’s say they’re using a stupid password. It should still be hard for somebody to hack them. How do you protect them against somebody just guessing their password?

You don’t.

Yeah, you can’t.

But you use something like BCrypt or something that makes it so good that they can brute-force it by guessing all the numbers.

The thing about that though is that brute-forcing is a numbers game, where if you have a list of 6 million passwords, probably some of them are going to loan up that you can crack. If you’re trying to guess one individual’s password, that’s going to take a long time because you have no off the fairly way of knowing if their password should be turned off.

Right.

So you would need to do enough human engineering to figure out if they have a weak password or in the first place, for that be able to make sense. It’s not as viable a threat because it requires on too many unknowns for the attacker. It’s a bad use of their time.

Yeah, but sometimes wasn’t Twitter had a while back? I think it was a fairly simple dictionary–like attack where they ended up getting an admin account with a very password. I think it was ‘HAPPY’ or something.

Yeah, there a few of super things that let your users log in a million times without either restricting their access or throttling that request. They might figure that out and go, “Oh! Well I can just hack their login.” That’s why it’s easier than trying to get the database.

Right.

But usually, a lot of systems, either it takes the request itself takes long enough or there’s some sort of exponential decay in response time where you’re trying to login for the hundredth time it takes 10 seconds or something. Or they just cut you off after 5. There are things like that can make that inevitable.

I like the time-based ones. The cut-offs occasionally annoy me, the small cutoffs. I think there’s really no point, correct me if I’m wrong, but there’s really no point to have like a small cut off like 3 or 5 or something like that.

No, there really isn’t.

I mean unless somebody has a really, really, really good idea of approximately what your password is and only needs a few tries. If they brute-forcing it, they’re going to be doing thousands of tries, millions in tries. And it’s so frustrating when you’re having just having a brain fart and these things are wrong. An email address or something dumb like that and it locks you out after 5 try.

Even, you know what, the weakest of passwords of a 6-digit alphanumeric password is going to take an average of 5 tries for someone to get it.

Right.

Some sites are pretty serious about that. I used my card at a site the other day the credit card and I have no idea why I guess, because it was some weird vendor or something that was one of those instances where the bank flagged that there’s possible fraud and didn’t send it through. And so I called them and talked to them like, “Okay, you can open it up or whatever”. And then I went back to the site and they were like, “You cannot use that card here for 24 hours.” and something else like. Hey! That’s aggressive. I’m not necessarily against it but still kind of aggressive.

You little sneak thief. We know all about your little tricks, James.

Yeah, I was generating credit card numbers with a RubyScript.

[laughs] So I have another question. Generally when the sites get hacked and let’s say that they still list out passwords, are they usually getting into the server itself and then attaching to the database and doing the queries or do they trick the website into doing it through SQL injection or something?

The former is the worst case scenario where they’ve actually compromised either the database server or the app server. It’s usually worse if they compromise the app server because then they can actually get access to your code and see what you’re doing to generate your password hashes. And that can give them much more information how they go about brute-forcing them. Sometimes it’s just SQL injection that gets some sort of dumped. Ideally, they are not actually your box because then if they are actually on your box while they are dumping the database, it could be trying to privilege escalate and they could do other nasty things. You have another good reason to shut your system down.

Right. Because ultimately, you may not know how they got the information.

Did I mention that you should shut the systems down? And not just keep them running. [laughter]

Wait. So we have to turn it off?

REIN: because I think it’s important to stress.

What do you think about frameworks like Metasploit?

I think they’re awesome. I think they’re great for PenTesters. I think they’re less great for malicious users but it’s sort of a free speech thing, right? We cannot get control who gets access to that code. But it’s great because it shows you how truly vulnerable a lot of systems are.

Yeah, I liked them at the talk, you know, where they show Metasploit and they like brute the box.

I’m going to rig Wi-Fi now.

Yup. It takes a couple of seconds. Okay, that’s depressing.

Oh! You’re running Windows NT Service Pack 2. I have a thing for that.

It’s interesting but it is good that it helps us find, you know, problems.

Yeah. I mean have a PenTester armed with things like Metasploit is your best friend. They’re going to tell you all the things that you never knew about that are vulnerabilities in new systems.

Someone malicious armed with Metasploit is your enemy.

Do you have any other resources that are useful for learning to think like a, I’m sorry, I have to say cracker?

As a matter of fact I believe that I have…

Can’t bring myself to call them hackers.

Fair enough.

You may well be both. I mean…

What are you talking about cracker?

He used that hacker and he used that…

Racist.

He doesn’t mean the racial slur.

No. No. it’s just…

Let’s see. I’m trying find, because there’s some good books on this, I’m trying to find the one that I’ve seen.

Yeah. There is even a book, I think, pretty recently focused on with web applications security in general, ain’t that right? I mean I’m sure there’s probably multiple.

So there’s some great book son PenTesting, like “Professional Penetration Testing”. And then there are… I know I saw those. Honestly. So no, I don’t have any resources for you right now but I’ll look around.

And we’ll put them in the show notes. Black Hat. That’s a good term.

Yes, black hat is a good term. If you’re thinking about putting your app out there in the wild and I keep saying that it’s a hostile environment, I want to stress that you can’t control how people access you site. You can request that if someone compromises your site, that they themselves follow proper a disclosure policy. But you can’t force them to put their white hat on and make it out of bed. Some people are just going to wake up, put that Black Hat on and try to break things. And you can’t control that and you’re not entitled to as much as you like to think that you are and as much as it may be actionable if someone doesn’t follow disclosure. People break the law all the time. You can’t control that. You’re not entitled to someone disclosing your vulnerability problems. So I see a lot of disclosure after action reports that really take on this sort of victim and how they like, “How could they do this to us?” or, “They should have disclosed properly and it’s their fault.” No. it’s still your fault. It’s still ultimately your fault and that hacker, if it wasn’t that one person, it would have been a different person. The hacker is never in my mind that caused the hack. They’re the proximate cause, yes, but the real cause is that your site is shitty and vulnerable.

One disclosure that comes to mind is somebody giving himself the commit-fit on a fairly popular web frame work on website that hosts the code.

So GitHub did an interesting thing with that, which is that they started out with their gut like fight or flight which was they banned him and call him out on his poor disclosure practices and all that stuff, right? And there was some…

To be clear though in the instant, maybe we should say what his intention actually was.

What he did is he found a vulnerability in GitHub’s handling of public keys that allowed him to inject his key into any project. He used that key to create a commit saying, “Look, I can’t commit into your project.” And that’s basically all he did and then GitHub put the band hammer down on him and trying to call him out on his black hat ways. Honestly, he could have done a lot worse. He could have made a lot of people very unhappy and all he did what’s essentially, a novelty commit into a site just as proof like, “Look. Yeah, I really did hack your public keys.” And then there are later more recent response was the one that people actually looked at that’s a nice way to handle that where they reinstated his account and they determined that he did not maliciously know these things. And if that was their first response, it would have been perfect. But they went through a couple of rounds of listening to that emotional reaction and going with it that they shouldn’t have done.

I confess I was sort of… man, I could totally understand where they were coming from. The tone…

Yeah. You are... I know I’ve been hacked. You are so pissed.

The way that was handled, I mean yes, the person could have done much worse, but they could have been a lot more, I mean, I don’t know. It was very, it’s kind of snotty. How can you allow this horrible vulnerability?

Yeah. He did act kind of like a dick but again you can’t control him.

Even GitHub’s actions though may not be all that unreasonable, like if you know a user has figured out how to compromise the system, then banning him as the first action. Maybe reasonable it’s the same thing as turning the system offline.

Yeah. I agree with you there but it wasn’t just the ban. It was the context with the ban happened which was a dialogue with GitHub where they frankly came across like children who would have their sandcastle thing.

Right. They could have come out and said, “Look. We shut down his account. It’s precautionary. Once we determine…”

Well we shut down his account to prevent, you know, as a precautionary measure, while were investigating and determining what to do with his account. But what they did is, nope we’re taking your toys away.

Never underestimate the ability of childish behavior to bring out childish behavior in them. [laughter]

You’re right. That’s for sure.

Well I thought it was a little bit funny too because you can setup a GitHub account in a matter of minutes.

And honestly, if he really wanted to do some damage, he could have done some damage. And it would have been before they banned him.

I thought they did a good job in the end though. Their later writer write-ups were really good and they talked about how their changing those security policies as it is. I felt like they learned something.

The point that they eventually got through there with how they disclose, I think really good. They set a great tone. They really came across as recognizing the magnitude of what had happened and realizing that they needed to come up with some policies and telling people enough information that they could start regaining some of that trust. I think they did a good job and I just wish that they had sniffed out a lot of this stuff that came in between where they sort of spoke what they felt in the moment, which turned out to not be all of that great of an idea.

Right. Alright. We’re about to where we need to do the picks. Is there anything we need to go over within the next few minutes

I have two more things to say about disclosure. One is that, if you’ve been hacked and you’re disclosing this incident, don’t try to sell me something. [laughter] Don’t try to promote your product.

That is not appropriate.

Is that called the “GoDaddy Disclosure”?

Yeah. And LinkedIn sort of did this too and a lot of companies see that they used it. It’s not the right time.

We just got hacked, we wanted to tell you about the details and if you would like some Viagra pills… [laughter]

And the second one is, if you’ve just gotten hacked and you’re disclosing how you just got hacked, now is not the right time to try to set yourself up as an expert in security. [laughter] And you keep telling your users how we started hashing, insulting your passwords because we all know these things about security now and “Hey! You should probably go change your password.”, or we recommend a strong password for security. Shut up about security! You’re the one that put us in this position in the first place. Be humble. Accept your responsibility and don’t try to use this as a marketing stuff.

Right.

It’s like Sony man, when they get hacked, I mean, you know, every other way. When they get hacked they say, “We’re bringing in this huge security expert… blah… blah…blah” and then you find them in the news again next week and it is like. “Well that helped.”

Yeah. You should have brought in the security expert before this happened.

You think Sony could afford it.

Well, I’ve been hit by a car 5 times now and I have really figured out how to avoid being hit by cars. So the next time we’re walking in the middle of a highway, I know how not to get hit.

So that actually reminds me of something that I think is really important as well which are these security incidents, they’re not the fault of the engineers. They’re not the fault of the engineers of LinkedIn or any of these places. They’re the fault of management. Probably all the way up to the executive level because security failures, I think, are a priority issue. They’re ultimately about not setting in the right priorities for your organization and making security long effect.

Absolutely.

Engineers just follow your lead. No one wants to build a site that’s vulnerable and watch it get hacked. But what we do is bend to pressures to maybe not worry about security right now because we’ve got to ship this feature. And those are management bills.

Yeah. It’s an easy trap to fall into the though.

That’s like the… I’m sorry. Go ahead.

I was just going to say it’s not an excuse but it is an easy trap falling to.

Yes, it is, I don’t want people to think that I‘m blaming the LinkedIn engineers for this or eHarmony engineers. They are good people doing their best to get their jobs done as professionals. This is always a management bill. Someone at the top now needs to get fired.

You know one of those links you gave us, it did talk about the eHarmony cracking and they got a bunch of passwords and that article was complaining that what eHarmony did so badly there is it didn’t go back to the people and say, “Hey! Don’t forget we changed your password on our side but you should go do in any other site where you use that password.” I thought that was an important tip that you may just not be talking about your vulnerabilities. It’s actually the internet as a whole.

When you get hacked, your job, your primary job when you’re disclosing is to protect your users. And to give them the information they need to protect themselves and you’re not just thinking about yourself. Here’s what happened in eHarmony. Here’s what we’re going to do about our thing that happened to us. It’s about putting yourself in the position of your users thinking, “What have I just exposed them to?”, “How are they not vulnerable in a way that they weren’t before?”, “What do I need to tell them to help mitigate that?” and that does include if you share your eHarmony password on other sites, those passwords need to be changed as well. That’s just the responsible thing to do. That’s thinking about your users first.

So somebody just trying to crash somebody’s blind dates? Or I kind of missed the point in hacking eHarmony.

Sometimes it’s just because it’s there.

Because people love getting into the system.

Some people just want to watch the world burn. [laughter]

That’s right.

I was waiting all episode to blow that one out.

Avdi, you were trying to say something before?

I was just saying, or Rein was saying, it’s really the fault of management and reminded me of the Space Shuttle Discovery, or not the Discovery, the Space Shuttle Columbia disaster findings where they said that ultimately, it was not an engineering failure. It was a culture failure. It was a failure of the bureaucratic system that they had in place. No one engineer could have prevented it from happening because it was the…

If it wasn’t this one thing, it would have been this other thing.

Yeah. There would be attitude that was being reflected down.

Yeah. That’s how it is with security too. If it’s not your SQL injection, it’s your SSH. If it’s not your SSH, it’s this and the other and it’s like any systemic problem, it needs a systemic solution.

Alright. Well, let’s get in to the picks. Avid, why don’t you start us off?

Okay. My pick today is something I actually, I brought up in the pre-show a few episodes back and then I forgot about it and picked something else. And the guys got off my case for leaving them hanging. So Rein has been talking a lot about the humble pie that you eat when you do something, when you screw up on security or something like this. But what about the time when your competitor was just been talking up all about how great their security is, gets cracked? That is the time when you eat Schadenfreude pie. And for that, my pick today is John Scalzi’s Classic Schadenfreude Pie Recipe which will be in the show notes.

It’s awesome.

[laughs] Love it.

More says in is the Pronunciation Manual in YouTube would say ‘Scoo-den-pree’. [laughter] If you guys see this, I guess this could be one of my picks. There is a YouTube account called Pronunciation Book. And it’s this weird attempt that’s just say words for English is a foreign language students. And it’s weird and it gets a lot of them wrong so this other accounts. Like your one job is to say a word properly and you’d think they could get that way, So this other account called Pronunciation Manual just picks a word and says it as improperly as he can think of. It’s hilarious. So your word becomes “Scoodenrudy’. And Sinectakey becomes “sina-kina-doty-chody”.

[laughter] So that wasn’t going to be one of my picks today, but I think now it is.

What’s it called again?

Pronunciation Manual on the YouTube.

Awesome.

Oh! That’s awesome. I think you just lost me a few hours there Rein.

Yeah. Yeah. I know.

Alright, did you have any other picks Avdi?

No. I think that’s it for me today.

Okay. James, what are your picks?

First of all for kind of relative pick, I have finally gotten into 1Password, which I think I’ll probably be the last person on Earth to discover but it is really cool. I mean how you can just have it generate good passwords for you and keep track on stuff like that. You don’t have to be the guy that has password for a password. Or when you’re forced to change it, you had one to it. It’s pretty cool stuff.

Yeah. I think that… available products because we should also mention that LastPass is a free alternative. I think it has the best integration.

LastPass is pretty awesome as well.

You could also share LastPass passwords, which is nice-ish but it does, I think, it’s a bad habit.

LastPass is much more cloud-oriented but it should be made clear that they are zero knowledge. So they’re basically, they never see your password. They actually download the entire… if you’re not… if you’re using, like not using one of the integrations, you’re just logging into the site, they actually download the entire decryption algorithm as JavaScript and then they download your blob of data and you put your password again and it decrypts it locally. They’re not actually… they can’t disclose your stuff.

I also think that one of the worst things ever is when a site’s password restrictions make the passwords that 1Password generate is invalid.

Yes.

It drives me up the wall.

I’m pretty sure they know more about passwords than you do.

Right.

And it’s always banks. You could only use lower case letters, period.

It’s like, that does help?

And you could only have 8 of them.

Well probably to round this little sidetrack off here a little, we should probably all say that using any kind of password generator, be it just that random site that just spits out passwords, or LastPass or 1Password. That’s probably superior to you making your password: 1-2-3-4-5. It’s totally okay if you’re in a space false movie but not so cool otherwise.

Yeah. I have to say I used LastPass. The way that it works with sharing passwords isn’t completely secure. I know that there are ways around it. If somebody shared a password to you, for you to actually get the password, but I really haven’t had too many problems with it and I generally and I’m only sharing with people that I trust anyway.

That’s also a feature that you totally need to use it.

Okay, so I do have one more pick and that’s open it and go ahead and pick Emacs today. I’ve been playing with Emacs quite a bit lately and it’s just hit version 24.1, which is nice because version 24 of Emacs has been under development for a long time and it’s kind of like the, in my opinion, a modern version of Emacs. It has a lot of modern features like being able to run small packages and stuff like that. So if you haven’t ever played with Emacs and stuff, you definitely want to start on this version because it’s pretty low on the pain scale for, you know, some things which is good for Emacs which is typically high. But it’s really neat. And you can see the list of all the different changes. I’ll put it in the show notes. I mean it’s a massive amount of just changes all over Emacs and stuff. If you haven’t checked it out in a while, or you’ve never checked it out, it’s probably a good time. It’s a new stable version, a lot of modern features and it’s pretty nice. Those are my picks.

Soon young Gray, you’re journey to the dark side will be complete.

[laughs] Alright. So I guess I’ll go next. I talked a lot about getting things done last week. I don’t know if I talked about the… I found this cool spacers for the wooden inbox, what do you call them? “Paper Inbox Tray”, whatever you call them. And I think they’re really cool. So I actually got it setup so that I have like this stack instead of having like 3 different trays sitting next to each other. It’s been pretty nice so I’m going to go ahead and pick those. I’ll put a link to them. I got them of Amazon for just a couple of bucks. I think they’re pretty handy. One other thing that I bought, I bought a MyBook. It’s an external hard drive for backups for my computer. And I’ve been running time machine against it and just another really nice alternative, I actually got a 1.5 terabyte drive and so I’ve set it up so that half of the drive is backups and the other half is just stuff that I want to move off of my primary drive. It’s been really handy for me that way. So those are my picks. Just some things that I’ve been playing with lately. I had a lot of time to dig in to some of the stuff or to dig in anything cooler. But anyway, that’s what I’ve got. Rein, what are your your picks?

I have one that is tech and one that is totally not tech. The first one is, Twitter recently released a tracing system called Zipkin. Have you guys seen this?

I saw it but I haven’t really looked into it. Tell us what it is.

It’s a really, really interesting if you’re building systems that you use Resque or have a lot of building pieces to talk to each other because what it does is that… it’s simply sticks a tracer on some job or requests that is moving through your system. And it watches it as it go to machine to machine, from endpoint to endpoint. It tracks the time it spends at each one. It can tell you, for instance, that when your user makes a request on your website, you actually talk to your database for this long. Then your other web service for this long, but then you also create the delayed job and here’s what happens to that. It’s far more… it tracks times and things like that that. It’s sort of similar of what you can get from…

New Relic?

Yeah, New Relic. But it’s more built around distributed systems and tracking complex interactions in those systems. And one of the things that’s difficult in distributed systems, is a lot of things happen sort of silently, or you don’t always know exactly what the date is doing. And sometimes tracking problems to their source can require a lot of going through logs and following it from one log to another log and that can be really time-consuming. So this is a great way to get some insight to have the data that is flowing through your distributed system. It’s beautiful.

What’s it called again?

Zipkin. Z-I-P-K-I-N. And there’s a blog post on their engineering blog about it. And then there’s the GitHub’s repository.

That sounds really cool.

And that I will look into those things.

Alright, anything else?

Yeah. And then I have one that’s totally, totally not tech related. On Netflix is a series that I have had only seen on VHS maybe 15 years ago back when I was still doing theater. It’s called “Playing Shakespeare”. It’s directed by John Barton who founded the Royal Shakespeare Company 1960 then directed it for 40 years. He is absolutely legendary and it talks about interpreting the verse of Shakespeare and deriving meaning and context from how Shakespeare uses meter and line and phrasing and all of those things. It’s fascinating and I think, for me, there is a very small amount of crossover and then it makes me more aware of language and that makes me more aware of the language that I use in other places like in writing a book. But it has, essentially, he took his Royal Shakespeare Company stable of some actors of all time and sat them in the television studio and have them go over different parts and really dig in on how to interpret those parts based on what the text tells them with the idea that Shakespeare included a lot of directing and instructions in how he phrases the text. So it has Judi Dench, Ben Kingsley, Linda Carron, and Patrick Stewart. All of these amazing actors reading Shakespeare and talking about Shakespeare with the best Shakespeare director of all time. So I think even if you don’t like Shakespeare but you are just interested in language, it’s still fascinating.

That’s really cool.

Nice. Sounds interesting. Alright. Well, if there’s nothing else then we’ll go ahead and remind you that we are doing a book club. We’re reading “Growing Object-Oriented Software Guided by Tests”. I have emailed the authors. I’ve gotten a response from one, but not the other. So we’re working on getting that lined up but we’ll probably be reading the book some time or be reviewing the book sometime in August. You can also sign up for Ruby Rogues Parlay by going to the Ruby Rogues website and in the right hand column, you can just click on the little PayPal button and we’ll get you on the mailing list. A quick reminder, it’s still a manual process. I’m working on finding ways of automating but there aren’t APIs for Google Groups. Anyway, it may take me up to a day to get you into the list, so just making you aware of that. Is there anything else that I need to announce before we wrap this out?

Josh is back next week.

Yey!

Yey!

I’m sorry I missed him.

He was sorry that he missed you too.

Makes me sad.

Alright. Well, I guess we are done.