The Rogues talk to Danielle Sucher about debugging.

Special Guest: Danielle Sucher.

 
DAVID:

I bought a microscope yesterday. And there was a splotch on it and I couldn’t figure out what it is and I did the scientific method trying to figure out where in the microscope the splotch was coming from. Turns out, I was seeing a reflection of my optic nerve.

Nice. [Chuckles]

Yeah, you can look in the microscope a really long time and you won’t find that.

Yeah.

So, when you gaze into the microscope, the microscope gazes back into you. [Laughter]

Also gazes back to me, yeah.

[inaudible] Are you saying that what you see inside Dave’s eyes is the abyss?

Yes.

Yeah, yeah.

I just want to know how he proved that hypothesis false. Did he gouge one of his eyes out? [Laughter]

Actually, and this is the part that I was very, very proud of, I finally switched eyes. And the splotch moved and changed shape.

Oh, that’s brilliant.

Wow.

Does this microscope make my optic nerve look fat?

Hey everybody and welcome to episode 150 of the Ruby Rogues Podcast. This week on our panel, we have Josh Susser.

Hey, good morning from sunny California.

James Edward Gray.

Good morning, everyone.

Avdi Grimm.

Good morning from snowy Pennsylvania. CHUCK: David Brady.

Good morning from Utah which is maintaining the conservation of fog with San Francisco.

I’m Charles Max Wood from DevChat.TV. And we have a special guest this week, and that is Danielle Sucher.

Good morning from New York City.

Can we get an introduction from you?

Sure, hi. I’m Danielle, as you now know. I live here in New York. I work at a non-profit called Case Commons where we do case managing software for social workers. I spend most of my time outside of work reading, hacking at open source, and petting my cat, who is very pretty.

Awesome.

Very nice. Alright well, we brought you on this week to talk about debugging. You gave a talk recently about the debugging mindset and deduction.

Yeah. I gave a talk at just a local meetup, New York Ruby Women. Most of it was trying to explain the idea of approaching debugging in terms of just verifying the facts and evaluating the evidence.

Half my slides were screenshots of Sherlock.

That was what sold me on the talk, actually. It’s all pictures of Sherlock taken from the BBC series, which is just awesome.

I found that actually really disorienting. Every time I saw a picture of Sherlock there, I was like, “Oh, that was such a great episode. Oh wait, what was this slide about?” [Laughter]

But the show is awesome because it’s all about science.

Yes.

I know. It’s really good.

I can totally see why James liked it. Science, Sherlock, and coding.

Yeah. It’s my kind of stuff, huh? So, you have tons of good tips in here. But I think one of my favorite is the one that you actually just start off with it about what to do when you get an error message. Do you want to talk about that? Because I’m sure I’m guilty of this all the time.

I have to teach this regularly. Yes, please.

I can sum it all up in one word: blame.

No, I think that’s, blame is useful and we should absolutely talk about it. And I mean that seriously. But I actually also have to teach people to read error messages all the time, which is why I put it in there. It seemed weird telling people, “Have you tried reading the error message?” But I forget to, too. I get all excited. I think, “Oh, I know what’s going on here. I don’t need to read this.” And then I have to flip back and look at it again.

I wonder if it’s almost gray page syndrome where, when Ruby throws an error message, it really throws an error message. You get the stack dump that’s 79 lines long and have of it’s in Active Record if it’s a Rails project. And so, you just assume…

You just assume that you’ve core dumped, right? So, “Eh, whatever. Gray page. I’m just going to guess what went wrong.” And like, “No, no, no, no. Read. Read. Look for the lines that aren’t Rails. It’s right there.”

You know, I believe one of the secrets of pair programming is that the other person, because they’re not typing, is actually reading the error messages.

That’s actually a big part of it. [Chuckles]

Yeah, having someone to just say, “Hey, can we just flip back to the terminal, please. I didn’t catch the number.”

Right. [Chuckles]

Yeah. What was that line number? Yeah. The thing about the stack traces as a starting point though is it seems like half the time, the lines that I really want to be looking at in the stack trace have been alighted.

Yeah.

It’s just like Ruby or Rails or somebody tries to omit. It’s that gray wall or the gray screen syndrome Davis was talking about.

Yeah.

Oh, we don’t want to show you all the stuff that you probably don’t care about. So, you’re like, I’m walking down through the stack trace and then see ten lines omitted. Damn it. [Chuckles] That’s right where it was.

Yeah well, and on that error page that you get from Rails, the default one, it always shows you. It’s application trace, and then it’s the framework trace, and then there’s the fault trace. And it always shows me the one that doesn’t have the right information in it. [Chuckles]

Yeah.

Yeah. But this is Ruby. If you say, put a debugger and then call caller, you can get more information.

Yup.

Yeah.

Awesome tip. Yeah, that’s actually on one of your slides. I think people forget about that. If you ever want to see the trace at any point in the program, there’s a method called caller that will give it to you. So, puts caller and you can see the full trace, not [omitted] and all.

Actually wrote a module called where-am-i because the module introduces another layer to the caller. Basically, all whereami does is it returns the caller, but up two levels. So, basically you can now say whereami in your code and it will tell you who called you.

I remember that trick. [Chuckles]

Yeah.

That’s great. I like that a lot.

So, I want to ask this. In the calendar, it said the debugging mindset. And we’re talking about in the trenches techniques for doing this stuff. Is there an actual mindset to this?

Yeah, I think there is. I think that a lot of people’s focus is in on, “Okay, I know what’s going on and I’m going to go through my whole application, try to figure out where I am.” But it’s really simple. It’s you want to figure out what the facts are. Stop assuming. What are the facts? What is the state of the world? What did I change recently that I can look at? We just can’t hold that many things in our head, so anything that lets us simplify, simplify, simplify, and focus in is really what’s critical.

Yeah. I worked on a system where every time we got a bug I would do a git log and I would look for a particular developer’s name.

Ouch.

It was brutal, but 80% of the time, that took me to where the error was.

I’m more thinking, so in Coders at Work which is this collection of interviews that Peter Seibel did, there was one with Joe Armstrong who’s one of the guys who were Erlang. And he had this great line where he talked about how he thinks all errors are pretty much within three statements of the place where you last changed your codes.

Yeah.

His story is about he said he debugged a hardware problem that way actually, without knowing anything about it, just by saying, “What did you touch last?”

So, part of the mindset then is just to gather information as opposed to just saying, “Oh, I know where things are?”

I always think of it this way. If you get too deep into studying the brain and how the brain works, one of the big realizations everybody has at some point is that your reality is not recorded, it’s constructed. So, you like to think that the brain is just a big video camera in our head and that whatever we see, it records all that and that’s the way it is. But the truth is that your brain is constructing reality, your reality around you, and that means that you can be fooled because your brain misinterprets some signals or whatever. And so, the construction of reality it has is not actually accurate to what’s going on. And then you can’t be dissuaded of that because you experienced it. You know that’s the way it is. That’s the problem.

Yeah. It’s like, “I never touched line of code in that file. I swear.”

Right, right. Yeah, that, right?

Right.

Yeah, it’s so easy to convince ourselves, yeah.

Yeah.

Well, the truth is everybody else’s code is total crap.

Did I tell you guys the argle bargle story? This is a really short story, but have I told it already?

I don’t think so.

No. Yeah?

So, I dump cache in my brain every two weeks. I do not remember what I’ve worked on two weeks ago. There’s a lot programmers who can remember what they worked on for six months, a year ago, and I feel that’s just a complete waste of time. I’d rather just get really good at reading code. So, I stop remembering what I’ve written. And a coworker pointed at a piece of code at the top of his screen, very top of his screen. He said, “Dave, what the hell were you doing here?” And I looked at it and I said, “Why are you asking me?” Or he said, “What the hell is this code doing?” And I said, “I have no idea. Why the hell are you asking me?” And he goes, “Because you wrote it.” And I said, “I’ve never seen that piece of code before in my life.” And he scrolled up one line and the comment was “argle bargle,” which is Dave Brady. Whenever I just have a kludgy hack that I just have to make it work, I write “argle bargle” on top of it.

And it basically means what’s about to follow is my best attempt at slinging crap on the wall to see what sticks. And I saw the “argle bargle” and I’m just busted. I have never seen that code. I have no memory of this code. But like James is saying, my construction of reality has no recording of that code.

So then, how did you figure out what it did?

I sat down with him and I read the code.

See, that sounds to me like a really good argument for really thorough commit messages.

Yup.

Yeah.

Yeah. But what you’re essentially doing is, you put “argle bargle” in there and that’s evidence that you wrote it and that it’s a hack. And so, what you’re doing is you’re setting yourself up so that you can go back later and look at that evidence and go, “Oh, this is a kludgy hack.”

Exactly.

Or the commit messages…

“Argle bargle” says this is worth less than its stated face value. Yeah, exactly. When you read this, read it with a squint.

But the trick there, the whole point of this discussion we’re having right now, is that if you can accept that what you think you know at any given time is a construction and it’s subject to validation, you could do a lot better. That’s why reading the error message is the best place to start. You may have an idea, you may think, but if the error message is pointing to the other side of the app, you need to just accept that your construction is probably wrong this time.

Right.

And you’ll save yourself hours if you can make that step.

That step is so hard.

It is hard.

To get yourself and to get other people to do it. Have you ever had the experience of sitting down with somebody and saying, “Let’s start you doing unit tests,” and they get offended that you want to spend all this time writing code that questions their ability to write a program?

Geez. I just try to start these things with, “Okay, this is what I think is going wrong. How can I prove that I’m mistaken about that?”

Right, right.

Yeah, yeah.

So what, admitting your ignorance is the beginning of wisdom.

Yeah.

Look, the goal is to prove that I’m wrong. That means I win. I’ve proved that I was stupid about something so I can move on to being stupid about something more interesting.

Yeah.

Right. If you can prove that it’s not there, if you have this instinct, your brain may or may not let it go. So, if you can just jump in there, throw in one line of code, rerun it, and you’re like, “Well, that’s never being called. Guess that’s out,” then you can let that go and move on.

Yeah, that’s another one. I really feel like it’s important to just clear things out of your head. So yeah, I want to follow the errors and go to what seems relevant, but I also want to just disprove the things I can disprove fast first. Get them out of the way.

Danielle, in your slides, you have one slide that talks about searching in the light. Can you tell that story?

That is so awesome.

Because it’s a great story.

Oh, sure. It actually goes two different ways. So, there’s this old joke where some drunk guys looking for their keys in the lights. And someone goes, “Oh hey, where’d you lose your keys? Let me help.” “Oh, well I lost them over there on the other side of the parking lot.” “Well, then why are you looking over here?” “Well, this is where the light is.”

[Chuckles] Yes.

Well, that’s really stupid. You should really look where you dropped the keys. That’s where you’re going to find them. But I went to a Peter Norvig talk actually, a few months back, where he talked about looking where light is as the actual goal, because you start where you could find information and work out from there. So, when I put that in my debugging talk, the idea was look where you can actually find information. See what you can gather from that, then move on to hunting in places that perhaps are trickier.

So, google.

So, google. That’s right.

That is one of the slides. Go with it, yeah.

And if your unit test suite passes and you still have this bug, it’s okay to have run the unit test suite, because that’s where the light is.

[Laughs] I don’t think you know where the light is.

Well yeah, but it’s like when your keys are missing, go quickly to the places where the light is. You might find your keys there. Who knows? But you can very quickly go through it and then it’s out of your head. Everything that we’ve tested in the unit test suite, I’ve got green dots across the screen.

I know it’s not in anything that we’ve covered in the test suite. I have to look outside that now.

The other thing is that if you go look where the light is, sometimes you could find someone else’s keys or a flashlight.

Yeah.

Nice.

Or a wallet. But admittedly, I’m looking in some pretty rough streets. [Laughter]

Or another car that somebody left the keys in.

Yeah, yeah. [Laughter]

Once you get them out of the car… Anyway.

I like the whole mindset conversation we’re getting into here. This is basically we’re talking about the scientific method and disproving hypotheses to come to the conclusion. And a lot of this is talking about how you get the data to either formulate a hypothesis or to reach a conclusion. And I love the whole walking through that process. The other fun detective show that I think has a great reference here is True Detective. And in the first episode he says, “Do any of your books have anything about jumping to conclusions?” where you prejudice your own ability to collect data by forming a potential conclusion too early. And I think you mentioned that in here in one of your Sherlock slides.

I’m sure I do.

Yeah.

I think it’s such a big thing. Speaking about pairing, one of the things I think I learned from pairing is I jump to a conclusion and say, “Okay, what are the three other possibilities that I can think of?” just to force myself to slow down and actually think.

I think that, one of the things that I’ll do when I’m really stuck trying to debug something is I will make sure that I prove that the obviously boneheaded ideas are boneheaded. If I look at something and say, “Oh, it could never be that.”

[Chuckles] Yeah, that’s a red flag.

Yeah.

Yeah, exactly. JOSH: As soon as I find, if I can catch myself saying that, if I can notice that I just dismissed something without validating it, I will force myself to go back and verify that my assumption that it was stupid was actually stupid.

I have a canned phrase that I use. I love it when my pair says that, “Oh, it can’t be this,” and just my judo sentence is that I say, “Fantastic. Show me.”

Yeah. For me it’s, “How do you know?”

Yeah.

Yeah, how do you know?

Oh, nice.

Well, we’ve been dancing really close to the discussion of actually experimentation. Experimentation can be actually going through and reproducing the bug. You know, they say, “Well, we sign in as this user. We click on this thing. And it does something wrong.” But we can also write tests that test the hypothesis. We can open up an IRB or Rails console or something like that, or Pry, and really go in and experiment. And that’s also a really good way to collect data. It’s a little more work-intensive than just looking through a stack trace and stuff. But it’s a terrific way to just verify certain conclusions. And if you validate that there is actually a problem there, then you can write a test around it, because you just did the experiment.

I think one of the great things about having a good test suite is that I can often use that to narrow down large portions of the system. If I run this test spec file, then doesn’t that narrow it down to being in these couple of objects? Probably, or something. It allows me to quickly search through the system in slices and narrow it down. I’m sorry. I cut you off, Avdi. What were you going to say?

I was just going to say I find myself naturally biased towards thinking through a solution, figuring it out by sheer force of will, rather than making an experiment. Making an experiment always feels like work. And I always want to just stare at the code long enough to just see it. And that’s not always a good attitude.

The trick that I do where like I said where I’ve poured my energy into getting good at reading code rather than remembering what I’ve written goes hand in hand with making experiments. There’s no latency. There’s no impedance mismatch of, “Oh, this doesn’t work the way I think it’s doing. I’m just going to break this expression completely and see if the code starts working or stops working in a different way.” And the reason why is because basically I’ve deliberately made myself as stupid as possible. That doesn’t sound as awesome outside my head as it did in.

No, it does. It does. [Chuckles]

But I will look at the code and I’ll just go, “This code doesn’t work. And I don’t want to think about why. What can I do to prove why it doesn’t work?” And so yeah, I tend to come at it very outside in and I treat even my own code like a hostile territory.

This seems like an excellent point to refer to the episode we did with Elisabeth Hendrickson on her book, “Explore It!” which is about…

I was thinking that as well, yes.

Yeah, it’s all about exploratory testing.

Yeah.

We did a whole, talk about a whole book about that subject.

Awesome.

Yeah, I was just thinking this reminded me of the thing where I’ll write a test that I know is going to fail just so the way it fails can tell me what the hell is going on.

Right? Yeah, exactly. And then the look of shock on your face is great when it passes. [Laughter]

Oh, that’s the worst, the worst.

Yeah.

Okay. So, I need to point to the words that every programmer has uttered at some point in their career, which is…

This is family-friendly.

“How did this ever work?”

Yes.

Yes.

Yes.

Yes.

I was thinking of different words that a programmer says. [Laughter]

Well, there are those, too.

D, did this ever work?

I love getting to the, “How did this ever work?”

So Danielle, what do you do when you hit that point? How did this ever work?

Try to go back in history and see if it ever worked in the first place. There’s no point in figuring out how it worked if it didn’t work.

Often you can find code that was put in but never actually put into use.

Yes.

Yeah. What I often find is I look at the code and I’ll go, “How did this ever work?” and I will find just upstream of it, one or two hops, a fold or a wrinkle in the code that causes the obviously broken code to never enter the broken state.

Right.

Yeah.

There’s this guard hidden upstream. It’s coupled, but there’s nothing to express that coupling.

So, in programming, two wrongs do make a right.

Yeah.

Absolutely.

Yeah.

That’s awesome.

Sometimes.

And a lot of times, there are two pieces of code that are superficially similar and one of them fell into disuse but never got deleted.

Yeah.

And that actually can be misleading when you’re debugging because you can be sure that you found the issue.

Yeah, look at this code. It’s so broken.

This clearly can’t work. And you think you found the issue and then you change it and it still doesn’t work.

Yeah.

Yeah.

That’s always the one that I wind up trying to fix.

That’s when you just delete it and see what happens.

Right. [Chuckles]

That’s actually not a terrible idea.

It’s not.

Oh, I meant it seriously. [Chuckles]

No, it is. Programmers have this attachment that we place such high value on code. And actually, I think I finally got over this watching, I think it might have been Zed Shaw’s PeepCode or something where he is just really ruthless with git reset. He will figure the whole thing out. He’s ready to save the world, git reset, start over. It’s that he loaded it into his head and that was the key, that he loaded it into his head. He doesn’t need the code anymore. And he will just do that all the time. And I think we have a natural tendency to be afraid of that. But I’ve finally gotten over that. I realized that, “Eh, if I just deleted it, it’s in Git somewhere. I can get it back if I have to.”

Yeah.

Whatever.

Dan Mayer has been doing some interesting stuff with production code coverage testing, using tools to see which code is actually being used in production and which code is never touched. Then you’re able to look at that report and go in and delete lots of code that actually is never being used. And I guess that relates to debugging, because that’s a few hundred lines less that you’ll be misled by.

Right.

That sounds awesome.

I guess you just have to be careful you don’t take out all your error-handling code. [Laughter]

Yeah. The big thing, especially with people coming from big static languages, I’ll find people that will comment out a hundred lines of code and then leave it. They’ll commit it. They’ll leave it in the codebase. And I delete that.

Don’t do that.

I delete that ruthlessly. And they’re like, “No, no, no!” And I’m like, “Dude, dude, dude. You can get it back.”

Yeah, that’s what Git’s for.

Yeah.

And we never will. [Laughs]

That’s right. That’s right. That’s awesome. Can we circle back to this google thing? Because I learned this lesson again yesterday. So yeah, can we talk about the googling problem?

Oh, sure. That’s just another one that just feels so silly to bother to say out loud. Have you tried googling the error message?

Right.

Because if you [haven’t, go] do it.

Yesterday at work, we had this problem. I knew I’d seen it a million times. I’m like, “Ah yeah, I think I know the algorithm.” I google it. I’m sitting there reading the algorithm page. I’m halfway down and I’m talking through the algorithm and how you do it. Meanwhile, my pair has intelligently gone to Stack Overflow, found the chunk of code, pulled it off. It’s in the app. It’s mostly working.

The best way to sound like an infallible guru while pairing, particularly while remote pairing so they can’t see your screen, is just to keep google things as they’re debugging.

Yes. [Laughter]

I’d be like, “Well, have you tried this?” and they’re like, “How did you know that?”

Dare I admit that I do that on my coaching calls all the time? [Laughter]

Google knows all.

And the thing is that a lot of times, it’s pretty seamless because I know where to find it is what it really boils down to, not necessarily that I knew it. And that’s as much the advantage in coaching as actually knowing it off the top of my head.

And that’s actually a skill that you can learn and develop and get better at.

Yeah.

The whole googling for error stuff.

And reading google results is a skill that [inaudible] better at.

Yes.

True.

I have had google fail me once in a hilarious way. And I had set up some PHP framework, like MediaWiki. It was a big one. And I went to go to the home page of the app and I got this, “Oops, something’s wrong in your setup,” and then it just dumped all of the state variables like “You’re running Apache version. You’re running PHP version this. Dadadadadada.” So, I took the message and I googled it. And instead of getting back the solution and the fix, I got back a list of over 10,000 websites that were up on the web with that framework installed wrong. And when google indexed them, it got that full error page. So, instead of getting the solution, I just realized I am not alone.

Everyone’s breaking this.

Nice. That is awesome.

And then apparently just deploying to production that way, which is awesome.

[Chuckles] And those of you who are doing it sneakily, so what you’re saying is you’re not teaching the people you’re pairing with how to google, huh?

Oh no, no, no. [Chuckles]

That is mean.

I want to add a proviso to that, which is that I always how I’m coming up with the answers. I always

try to point out the way I knew that was I just went to google and you should too.

Yeah. It’s so important. We just have this, I guess it’s hubris, that, “Oh, I know what that error message means.” [Chuckles]

Yeah. I think a lot of this is fighting our own egos.

Yes. So, I liked your zombie Richard Feynman reference. [Chuckles]

Yes.

The “Ideas are tested by experiment. That is the core of science.” Yeah, awesome. The other thing, I love Feynman stories of course, having gone to Cal Tech. One of the things, I’m sure I’ve mentioned this on an episode before at least once, is one of Feynman’s stories is when he shows up at the Manhattan Project. And it’s his first day and he knows nothing about what they’re doing. And somebody rolls out a huge blueprint of the…

Of the Oakridge gas diffusion setup. And he’s like, okay, I have to show my ignorance here, because I have no idea what they’re talking about. So, he just randomly puts his finger down at some point on the blueprint and says, “What’s this?”

And he says, “What happens if this breaks?”

Yeah, yeah, just something like this, “What about this? What if this breaks?” And the guys look at it and say, “Oh my god!” and they run out of the room. [Laughter]

And he’s like, “Oh crap. Now they think I know how to read blueprints.” [Laughter]

But the point of that story is he just sucked it up and said, okay I’m going to show how little I know here and ask a stupid question. But it turned out that there was a lot of value in asking that stupid question.

Yeah.

And if he had been defensive about his ego and not showing his ignorance, the war could have turned out very differently, right?

Yeah.

Yeah. I think it’s really helpful to be like, “I don’t know what the hell’s going on here.” That’s really cool. Okay, let’s talk about that.

Yeah. The war history would include the Oakridge gas plant disaster story.

Yes, right.

So, that’s like, don’t be afraid to ask dumb questions. But this I think goes well with the, you were talking about use a branch. And I loved the use a branch to create an easy resettable environment where you can do these kinds of experiments. And you have complete freedom to make as many changes as you want, because all you have to do is git reset --hard HEAD and you’re back to where you were.

I have to admit, I’m too much of a wimp to use reset that freely. I just end up with a lot of stashes.

Okay.

Stash is cool. Stash is cool.

Stash is actually kind of another form of reset, right?

Yeah.

It’s just pulled off that it makes us feel a little better because we know that we can stash apply and get it back, right?

Yeah, exactly. It’s just comforting.

So, you had a slide on this. Do you have any particular tips about using branches that you could share?

Not so much branches, but there are definitely a few git things that I think are super helpful.

Okay.

We were talking about blame earlier. That’s a really big one. I really shied away from using blame for a long time because of the name. And I thought, “Well I don’t want to be mean and think about this person as wrong and I’m blaming them. This is not the attitude I want to come from.”

[Whispers] But they’re really wrong.

[Whispers] [And now I’ve proved] them wrong. Or me, I was wrong.

You can alias it to git appreciate. [Chuckles]

Oh, there you go.

Subversion had an alias for praise and nobody used it, so they took it out.

[Chuckles] I think I should alias it as git what-were-they-thinking.

Yeah, git wtf.

Yeah, because I really need to look at something and I think, “I have no idea why they did this. What on earth is going on? What were they thinking?” If only they’d left a time capsule that told me exactly what the context was and what was in their heads. Oh gosh, they did.

Well, one nice thing about git blame is that if you look at it and it has their name and it says that they did it yesterday, usually, their cache hasn’t reset.

Still in their head? [Chuckles]

So, you can go over and you can go, “I’m having some trouble with this code,” and that way you’re not saying, “You screwed it up.”

Yeah.

And then you can gather more information. You can validate certain parts of your hypothesis because they can at least walk you through the thought process behind that code.

Yeah. Or you could do it and you see that this wasn’t changed for three years and this thing only broke yesterday. So, this probably isn’t the cause of the problem.

Right.

I’ve actually done the go talk to a coworker who pushed yesterday. And I got everything I needed to know in one sentence, which was, “Oh crap, did that get pushed?” [Laughter]

Nice.

Oops.

Yup.

Know that the git blame, I think Danielle’s right, we have this tendency, “Oh, then we’ll know who’s wrong,” but that can help in a lot ways to not just, it’s not about always putting the name with it. The way more important piece of information that it gives you is the git SHA for that line of code. Then you feed that to git show and then you get the commit message, the rest of the changes that were involved with that line. It gives you a more complete picture. It gives you the context, which is what matters.

Exactly. What changed at the same time and why did they claim they were changing it?

Right.

I also want to point out related to that, is that I find that in GitHub it’s a lot easier to read the commit diff than on my command line.

That’s a fair point, yeah.

And so, a lot of times I’ll take the SHA and I’ll feed it to GitHub instead and then just go look at the changes there.

That’s cool.

I use Gblame in vim, which makes it really nice.

There’s a thing I’d like to shout out to all of our listeners and get them to rebroadcast to everyone in the sound of their hearing. And that is we’ve been talking about scientific method and skepticism and good thinking process. But there’s something that scientists do and that’s they keep good notes. So, please, please, please, please, please get in the habit of writing disciplined, get in the discipline of writing good commit messages. Those are your notes that you’re going to come back to later, or somebody else is going to have to come back and read. I hate when I find a 75 file commit and the commit message is “fix.”

Yeah.

Although when I’m git blaming something to figure out what went on and the commit message is, “I’m a horrible person,” I think I pretty much have a good [inaudible]. [Laughter]

I’ve done that.

Yeah, yeah. I remember that comment, “This is a crime against humanity.” [Laughter]

Yes, yes.

I think I wrote that comment. Okay, so I have a question about how you deal with bugs that are resistant to reproduction in development.

Yes.

But they show up in production.

Yeah. Question one: what is different about your production system? How can you make your development system more similar? Anything, anything to get closer. Or, think about what are the differences and what could those differences be causing?

Yeah. The worst case I ever had of that was that we had a production environment and then a staging environment that we always just deployed to, to check things out before pushing to production. And production was having this terrible bug and staging was just fine. And it turned out that we were using a different hardware load balancer for staging than we were in production. And it was terminating the SSL stuff differently. And it was completely hidden form development that we had these two different pieces of hardware.

SSL is an excellent example of something that’s often different in production and development.

This brings up and interesting question, because I wonder if it’s always the best use of time and resources to try and make your development environment more like production versus trying to find some way of debugging on production. Because I think those are great examples of how there are always going to be differences between production and development and even between production and staging. And it can be both very difficult to simulate those in development and it can also be very difficult just to realize what those differences are. And you can sink a lot of time into that and I wonder if it’s worthwhile.

Yeah.

That’s a good question.

Yeah, that’s actually really fair because I’ve had problems where the differences were environment variables. You can’t really solve that by making your systems more similar.

Yeah.

One of the common examples that bites me in that is Ruby’s default encodings pull commonly from environment variables, like LCC typing. So, the system can be assuming UTF-8. Then you get it somewhere where the server’s different. It doesn’t have LCC type set to what you expect, so it defaults to US-ASCII. And there you go, there’s your bug.

I had one on the staging server where someone had accidentally switched things around with rbenv and suddenly our server was using a different version of Ruby.

Wow.

Wow.

So, it’s a little scary.

You just said a magic thing though. Staging server.

Yes.

So, the staging server is, in theory, all the real hardware and it’s got the full systems. It’s going to go out and talk to Verisign and Stripe and whatnot. And it’s probably using testing account stuff. It’s not running real credit cards. But it’s as close as you can get. You’re clearly not running… AVDI:

Something [that has a] load balancer.

Yeah, exactly.

[Chuckles] Yeah, same number of machines, same amount of data and pretty similar data that’s just been obfuscated. Yeah, exactly, as close as you can get.

And I had a load balancer bug and we solved it by putting a load balancer in the staging. We didn’t have a staging load balancer. So, I’m like, “No. This has to go into staging.”

But I think Avdi has a good point there about how it’s a big rabbit hole. There are still going to be differences even between staging and production. How powerful is it to be able to SSH into production, fire up a console, and start typing some lines?

Oh, yeah.

James and Danielle, you both cited bugs that you solved that were production only. I’m curious. How did you find those?

Looking at the error message in the log? I don’t know. I swear, I don’t remember what it was, but it was something that I remembered had changed in a more recent version of Ruby. And that’s what triggered me to thing to check for that particular problem.

In our case, after I’d figured out it was an encoding bug and that I couldn’t see anything wrong on my system, I just started thinking, okay I guess we’re talking about different encoding. And I don’t even have access to that production box. And I ended up asking our sys admin, can you just run this line of Ruby and tell me what it outputs? And that basically just spit out encoding default external or something, and solved it, it was different, and assumed that was it. AVDI: So, you did in effect debug on the production server.

Yes.

I did, yup.

Now, the way that you approach that kind of stuff has to be really different. I feel like when I’m doing something on my dev box locally, that it’s more like doing a chemistry experiment. But if I’m doing something on the server, it’s more like I’m being an astronomer and I’m looking at something that’s happening far away that I can’t really have control over what it’s doing. All I can do is change how I’m looking at it.

Yeah, it’s not like you’re going to bundle open on your server and start messing around with things.

It’s not like you should.

Right. [Laughter]

So, Ruby has historically sucked for this as well. There are other languages and frameworks which put a real premium on making it easy to debug in production. There are ones that go as far, I think Seaside goes as far as basically a failed request effectively just gets frozen and you can get in there and basically restart the code under debug on production where it failed. And we just don’t have that kind of tooling.

Right. Since we don’t have that tooling, what tooling do we have? Danielle, you mentioned logs.

People also use error reporting services or collecting metrics.

Yeah, lots of error reporting and metric services. Things like New Relic people use for Rails applications. So, you can see what kinds of errors are being thrown and stuff like that.

Or Rollbar.

Yeah.

Any of those services.

Yeah.

Okay. As there any other tools that people commonly use for collection that kind of information?

Yeah, all sorts of monitoring tools. I’ve been messing around a little with Clojure lately and contributing to an open source library called Riemann which does monitoring. It doesn’t really matter what kind of system the events are coming in from. But it does all sorts of interesting aggregate functions on streams. It can give you alerts any way you want.

What about instrumenting your application to make it easier to debug production faults? So, just a little bit of prudence so that you have the right data when you need it.

I think there’s an interesting balance you want to strike, because you want enough data to debug, but not so much information coming through that it’s hard to pay attention to the stuff that matters.

That’s a really good point right there. The tendency is grab everything. But then if you bury yourself in data, you didn’t make anything better. [Chuckles]

So, there’s an important distinction. There was a fantastic talk at MountainWest last year about the Air France flight that went down in the ocean off from Malaysia. And he talked about information overload. And he showed this is what the cockpit of a 747 looks like. And if you’re not a pilot, it’s terrifying, right? But then he said it’s not the amount of information that’s the problem. It’s what’s called alert fatigue where in addition to all this information, you’ve got this blaring hundred-decibel clack sound that’s going [makes alarm sounds] telling you something’s very, very wrong. And so, yeah you do want to be able to get everything, but you also need to be able to filter it so that you can turn things off.

Yes.

Right.

Even if it’s after the fact with just grep.

Yeah, that’s so, so critically important. Exactly.

I think one of the key things I try to use to make a decision about information I’m going to need later, when I’m programming it and I hit that interesting point where I find myself asking a question like, “Hmm, would it be better to do it this way or that way?” It actually doesn’t matter which one I choose. It just matters that I realized that’s a critical junction and so I probably better tell myself what happened there. If it’s interesting enough for me to be posing questions like that, it’s probably worth logging the metric and saying, “Oh, in this case we chose that.” And then at least I’ll be able to look at it and see what happened.

I wonder. Do you guys ever use log levels? So, you just change the log level that your production app is running at for a given period of time. And then put things in the guard clauses that check the log level?

I don’t. [Chuckles]

I haven’t.

Honestly, no.

Yeah.

I think the problem with that approach, I get it and I like it, and I think the problem with it is that it’s too hard to do. So, you think, “Okay, I’ll put in these extra debugging calls so most of the time they won’t be in there. And then I’ll turn them on.” So then, you’re debugging and you’re working with the app. You go like, “There, that’s it. That’s the bug.” And you’re like, “Oh, I don’t know what happened. Okay. I’ll go back, kill the server, change the logging level, restart it, try to go through that process again, and reproduce it.” It’s too much work to get back to that.

Yeah.

For me, the problem is that your entire codebase is at log level 0 always, when you read the source code.

Yeah, that’s true. Yeah, yeah, yeah, right. You have all these extra calls in there to log debug and they don’t ever go away.

Yeah. Right.

Yeah. I think that point earlier about filtering through the logs is actually more helpful to me, making sure that anything I’m logging has a nice easy greppable way of finding it so I can just filter through and find the relevant stuff.

Yeah, I have to admit it’s an idea that I’ve thought about for a while, but I’ve never actually gotten around to implementing in any of my projects. So, it’s some interesting feedback. And if anyone’s done that, I’d love to hear how it actually worked out or didn’t.

We’ll get lots of emails about how we’re doing it wrong. [Laughter] CHUCK: Hopefully, we’re doing it right. It wouldn’t be throwing exceptions, right?

That’s right.

Okay Danielle, can you explain rubber duck debugging.

Yeah. I love this.

What is that? Quack.

So, the idea if you explain clearly to someone what the problem is, that will force you to actually pause and think about all of your assumptions and everything you think you know and what your knowledge is. And by the time you’re finished explaining it, you’ll probably figure it out by yourself anyway. And you don’t have to explain to a person. You could explain it to your cat. You could explain to a rubber duck. It doesn’t really matter. When I actually first gave the talk on this, I brought a bunch of tiny rubber ducks and gave them out to everyone in the room just to make sure they had someone to explain their problems to.

Everybody needs a friend.

Yeah.

The rubber duck works because it just sits there and bobs up and down. It listens to you, right? [Chuckles]

Dogs are great. I do this with my wife all the time. And I bet it drives her crazy, because halfway through explaining the problem, you get the Aha moment and then you just stop talking.

Yes.

Because you’re like, “I’ve got it. Now I just need to figure out how to,” and then you run off. And she’s like, “Whatever.”

[Chuckles] Yeah, I do that to my husband. He’s like, “Wait, no. This was interesting. I wanted to hear the rest of the story.”

Right. [Chuckles]

I just get that blank look from my wife. And then I get the smile and, “Is everything alright?” when I just stop talking and run back upstairs.

Yeah.

You’re going to be okay, right? [Chuckles]

Great. Danielle, I have one more thing to ask you about and that’s being right by accident. [Chuckles]

Oh, boy.

Can you talk about that?

Yeah. You know when you change something and suddenly all your tests are green and everything works and you have no idea what just happened?

I hate that.

That’s the worst.

Yeah.

Yes.

There’s nothing worse than a test that’s passing and I don’t know why.

The nice thing is that you can still check it into another branch and then you can do a git diff on it. And then you can start to explore what it really means.

Yeah. The scary thing is that if I don’t know why it’s working, I don’t know why it didn’t work. And I don’t know when it might stop working again. [Inaudible]

I know people and have worked with people that once they have it passing, they’re very satisfied with that and don’t need to look into it. And I almost envy them that, because I will rip the whole damn thing apart to figure it out. [Laughter]

I cannot let it go.

So, there’s a refinement on TDD that Kent Beck talked about recently, maybe a couple of years ago, where he phrased it as “calling your shot.” So, when you’re playing pool and you’re taking a shot you say, “Okay, I’m going to put the 12 ball in the corner pocket.” And then if you accidentally hit the 7 ball in the side pocket, your shot’s no good. Yeah, so in TDD, when you write a test that you expect to fail, you want it to be red, what you do is before you hit run you say, “Oh, this is going to fail with the record not found error.”

I actually do that. And I had no idea it was a Kent beck thing.

It’s just I spent a lot of time pairing with people, so I like to say this is what I expect to keep myself honest, really.

Yeah.

Yeah. Well, it’s also an important part of the scientific method, is you want to make sure that you’re testing the right thing. Your experiment is actually set up right.

Yeah.

Yeah.

Right.

That’s falsifiability, right?

Yes.

[inaudible] one?

Yeah, or is it the null hypothesis? Oh, I forget all that. Never mind. [Laughter]

I just like being right. [Chuckles]

I like to be wrong. Wrong is much more interesting.

Yeah, you learn more when you’re wrong.

Basically what we’re saying is the saying, “I’d rather be lucky than smart,” is wrong because luck is not reproducible.

Not according to Katrina. [Chuckles]

That’s true.

Katrina is smart enough to look lucky. [Laughter]

What’s that lyric? It’s smarter to be lucky than it’s lucky to be smart?

Yeah.

Nice. You have a great slide in here that goes back to the earlier mindset thing. And I want to talk about it because it took me a while to figure this out, I think, for myself. But you have the great quote in there on Sherlock, “I need to go to my mind palace.” And I love that one. It’s like when you’re totally hung up on the problem and you’re just stuck. And it’s like Avdi says, you’ve got the blinders on and you’re so deep in it and you think, “I’m just going to dig deeper,” and you can’t let it go. There has to be some way to break that chain. And going to the “mind palace” is a great way to do it, to have something that just resets your thinking.

Yeah. Mind palace is actually a memory trick that helps you see, what are the associations between the things I know? We all have a lot of knowledge. Anyone listening to this has a lot of knowledge. Those are people who go out and try to learn things. But it’s weirdly hard to make use of that knowledge.

I find for me, the secret trick is leave the computer.

Yeah.

Once I do that, it changes the way I think about things. And then that’s all I need for the reset.

My way of dealing with it is very similar, except I will actually head over to a local restaurant, coffee shop, or café. And again, that little break and then just being in a different environment, I don’t know. It makes me think about things a little bit differently.

Yeah, very effective.

Yeah. I also like to just list out [inaudible] the things I know. What do those things imply? And just the physical act of writing out a list is actually very helpful. I don’t like to type it. I like to write it by hand on a piece of paper just to force myself to go through that process.

There’s something, there’s a little tiny debugging trick that I thought I’d bring up at some point here. It’s not exactly related to anything we’re talking about right now. But I think it was in the book ‘The Practice of Programming’ that introduced the idea of debugging numerology, which basically says pay careful attention to numbers that appear in your bugs. Often a number can give you a clue to where the bug comes from. And I remember I put this to use once a long time ago when I was trying to track down a memory leak. And I had managed to put some diagnostics in, some memory leak diagnostics in using a tool. And I managed to find that there were exactly a hundred allocations that weren’t matched by de-allocations. And I hadn’t tracked it down. Either I hadn’t tracked it down to the line of code or it was a line of code where there were also lots of legitimate allocations going on. But the hundred turned out to be the clue, because I was able to think about it for a little while and realize what was the one thing that happened exactly 100 times in the test scenario? And so, that’s just another tip that I’ve run across, is pay very close attention to numbers.

So, that reminds me of the 500-mile bug, which is one of my favorite…

I love this.

Right. You google for it, it’ll be the first result. And it’s this wonderful story about this guy who gets a bug report, which is we can’t send emails that travel more than 500 miles.

Yup. Insufficient postage.

It’s so insane. Yeah, exactly.

They didn’t have enough postage. [Laughs]

So insane.

Yeah, they get really tired.

And it turned out to be an issue with the speed of light.

That’s amazing. Definitely along the number lines, you can run into bugs where it works the first time but it won’t work the second time or things like that. So, if you can see how many times you’re trying it or something, it can help.

Have any of you guys worked through the Deitel and Deitel books? C++? I think they’re called the explained books. But they’re these big blue softbound textbooks. They’re really heavy-duty and they go through the whole language. And they’ve got them for Java and C++ and whatnot. And they do a wonderful thing early on. They basically give you a for loop that says for 1 to 10, srand time, and then print out a random number. And you run the program and it outputs the same number ten times in a row. And then you put a debug break statement in it and you step through the code and you get a different number each time. And they’re like, “Why do you think that is?” And they let you sit with it to really think about it. And then they go on to say, time only updates once a second. And when you’re stepping through it in a debugger, it takes you a second to get through the for loop. But when you run it as a program, it’s all over instantly. And they do it deliberately as a lesson of stuff can come at you from way outside your box of things, of all that you know. And that’s where you’re going to learn something new.

I always think of bugs like that whenever I put in a puts statement or something to see what’s going on and that changes the results.

Yeah.

Because I slowed it down with some I/O.

Right.

Like, “Oh, race condition.” That’s the first thing I think of as soon as I see it.

Heisenbugs, where observing changes the observed.

Yeah.

Right.

I had a code break that I put a puts statement in it at the end of the method. Well, puts returns nil. And the previous line was returning something useful. [Chuckles]

Right.

I’ve done that one. [Laughs]

So Danielle, you used one of my favorite words, which is heisenbug. Can you give us a definition?

Well, I think of that as the sort of bug where the very act of observing it changes what’s happening.

Yes.

Okay. So, the observer affects the outcome of the observation. I have a similar word that I also love, which is hindenbug.

Alright, what’s that one?

Oh, that’s just a bug that just comes out of the blue and explodes and destroys all your data.

That’s awesome.

So Danielle, speaking of heisenbugs, how do you feel about debuggers?

Yeah. Let’s do the debugger question.

Sure. They’re useful. I don’t like turning to them first. That’s the short answer. I think they’re absolutely useful when you need them. But I have to admit, I like print statements and actually reading my code first, because I like forcing myself to try to understand it. I find that helpful.

Interesting.

Are there particular techniques that you use for doing source code inspection when you’re trying to debug?

I use Pry a lot in the Ruby context for the most parts. And I like being able to see where things are defined and what the actual source of the actual thing I’m calling is.

Okay. So, you like an interactive tool that lets you explore the code?

Yeah. Once I get to the point where I feel like I need that, yeah, exactly.

What about, was it pragdave or Jim Weirich who was talking about zooming your source code really small so you can see the shape of it?

Pragdave.

That’s Dave Thomas.

That’s in ‘Pragmatic Programming’, the original book, yeah.

Yeah. JOSH: So, it’s you look at the shape without getting lost in the details.

It’s an interesting trick. You see the big methods bunch up and stick out.

I do that. I did that. And I’m such a visually oriented, spatially oriented thinker that I will do that will full-sized source code on the screen where I will go in and I’ll do an extract method. And I will read over the code real quickly to make sure yeah, I’ve got all the variables I’m extracting. And then I do the rest of the edit based on shape. I literally will extract the method and not read what I’ve written. I’ll just go, yeah, it’s shaped right. And I’ll run the unit test to see if it worked. And if the test passes, I often don’t go back and read the code. Am I committing career suicide here. [Laughter]

Just keep talking. I’m sure it’s going to work out fine.

Yeah, just keep spooling rope, dude. We just need a few more coils in the noose. No, the point is that the shape is so valuable, the visual shape of your code, is so valuable that you can often extract and move code without stopping to read every jot and tittle to make sure it’s right. You go, yup the shape looks right. Or you can look at some code and go, the shape on that is wrong.

Yeah, I’ve only done that that second way, that looks wrong. I’ve never tried to use it in terms of this looks right, I trust myself. And that sounds really…

No. The shape looks right and I trust my unit test.

Well, yes. That one I do. I certainly do the, I’ve copied this. I’m too lazy to read it. Let’s see what breaks when it fails.

Yeah. Yeah, that’s closer to it, yeah.

Okay. Are there other kinds of tricks that we use to navigate the code and find interesting places to look at for problems? What about churn? I think we talked, Dave you said to look for within three lines of the last change?

Yeah, yeah. I think I was quoting Joe Armstrong with that one. Oh, someone just linked to a gitchurn script that gives you the most recently changed files, or files in the order which they were changed. I’ll have to dig up the link for you guys. It’s really wonderful.

Metric_fu includes the churn gem as well.

Yeah. There’s definitely ways to look for it. You have those things on the project where every time there’s a major bug, it’s in there, or whatever, those things. The usual suspects, that’s how I think of those.

Yeah. There’s a strong…

Which for me, by the way, the usual suspects is whatever you introduced in the project plus the Rails autoloader.

Oh god, yeah. [Laughter]

Yeah. There’s an increasingly strong argument. We were talking about production versus debugging, where I’m really getting on board with dependency injection to decouple things. Or specifically to decouple things, so that I something is breaking, I would love for it to break on its own merits and not because it’s got an implicit dependency on something three levels away. And if you use dependency injection, you can’t get away with an implicit dependency three levels away. You have to have an explicit dependency. And you can actually see the unit tests setting up and tearing down the, yeah we’re actually depending on this thing three things away. And that’s a code smell that you can come back and fix.

Yeah. But sometimes, you need to fix something and you don’t have the time to completely refactor it first.

Oh yeah, yeah.

Mission critical thing, you have to do it now. That’s why I start commenting out bits of it and changing it to, not injecting dependencies, and just changing other pieces of this method to simplify it.

Yeah. Yeah, sorry. By dependency injection, I’m talking about the way I write new code.

Yeah. Yeah, that’s certainly absolutely ideal.

Yeah. I try to avoid things like in mocking frameworks methods like any_instance or something like that.

Yeah.

That’s a good sign that I’ve just totally lost control of what’s creating what. [Chuckles]

Yeah, I like how in the RSpec documentation, they put any_instance all the way at the end and preface it with a whole bunch of reasons why you should really never be using this. We only include this grudgingly.

Right. [Laughter]

You know, I remember back in 2008 when mocking frameworks were starting to become a big deal in the Ruby world. And it was all RSpec mocking versus Mocha versus RR. And the existence of the any_instance feature was the make or break feature for a mocking library. And now it’s like, “Stay away.” [Chuckles] Oh, how practices change.

Yeah.

This feature is sometimes useful when working with legacy code. Though in general, we discourage its use for a number of reasons. And then there’s a series of bullet points.

Exactly.

Sandi nailed down the rule that causes that code smell. And it’s a file that contains a class should not refer to a hard-coded class of any other class. And that’s the injection. But like we just said, sometimes you have to debug the code you have, not the code you want.

[Chuckles] Right.

Yeah. [Chuckles]

Sometimes.

Yeah, sometimes.

Is that like the code that Gotham deserves, rather than the code that… [Laughter]

Yes.

Oh, I heard that as you go to war with the army you have, yeah.

Nice. Danielle has great tips on that though. Treating the rest of the system as black boxes, commenting out as much as you possibly can, to verify that it’s not in this thing then just stop using that thing and replace it with some one-line stub or whatever. Just to narrow in on the real problem.

Narrowing in is so important. So, if you’re a scientist you talk about this as isolating variables. And if you’re constructing an experiment, the most valuable thing you can do is test changing one thing at a time. Because if you’re changing two or three or four things at a time and you get the result you wanted, you don’t know which of those things produced that result.

I only sort of agree with that. If you have a lot of things, if you change half of them, then at least you can do a binary search through the things that you want to try changing.

Okay, that’s a great point. Trying to narrow down from a large set to that small thing, there are a lot of techniques for doing that. That’s a great example there. But in the end, you want to narrow down to finding one thing.

Yeah. Absolutely yes, I agree.

You don’t want to commit those large set of changes. You want to commit the one that’s right.

No, this is when the reset thing is critical, yeah.

Yeah.

Yeah, exactly. But you had a point about that in your slides, Danielle, talking about Miller’s law. You want to bring that up?

Oh, sure. Miller’s law is that the average can only hold seven-ish items in their working memory at a time. So, we have our puny little human brains and we can’t actually keep track of anything. And we just have to accept that and to get rid of things.

So, by commenting stuff out or whatever, you’re saying, “Okay, I won’t have to keep track of that.”

I don’t have to understand that to fix this. I don’t have to keep track of that. You have to lessen the cognitive overhead in order to be able to focus on the few things that matter.

You only have seven registers in your processor.

Right.

Right.

Exactly, exactly. And if you’re going to go in and try to fix something in a huge system or a system you’re new to, there’s’ no way you’re going to actually be able to learn or understand all of it, or think about it at the same time.

I actually love it when I can fix code and not understand what it did.

[Chuckles] Yeah.

That always makes me feel great. [Laughter]

You know, I’ve just narrowed it down. It’s got to be happening in here. The tests tell me everything’s okay. When I flip this one thing over, everything goes green. We’re good. I have no idea what it’s doing, but that works. I think that’s cool.

Oh, yeah. To me, it’s fixing bugs in code where I don’t even know the language.

Right, yeah. [Laughter]

I love that. But I want to hark back to a few minutes ago when we said getting lucky doesn’t count. The talk that I reference about the Air France flight, he includes a definition of root cause. And it’s a beautiful definition. Root cause is the point we were at when we quit looking.

[Laughs] Nice.

Yeah. On the topic of narrowing down, another technique that might be worth bringing up is if it’s code that I don’t know and don’t understand as well, one thing I like to do is just start going through and putting assertions in, just basically validating my assumptions. Line by line put in an assertion, run the test or run the scenario, or whatever, and then put in another assertion. Because a lot of times, we look at a piece of code and we convince ourselves that we know exactly what it’s doing, and we’re completely wrong. And particularly what I see a lot is either we miss a conditional, like if somebody has stuck a conditional way at the end of the line or something like that, or we just assume that the circumstances can’t possibly be such that a particular conditional branch is taken. And so, just going through and putting in assertions that the code will get here or that this value will not be nil or that this value will be nil, before switches that are made on that value and stuff like that, they can really validate your assumptions and narrow down the problem space.

I will do that with the unit tests sometimes. I feel weird doing this, but I call them PNE assertions. PNE is a chemistry term. It stands for “poof, no eyebrows.” [Laughter]

I love it.

And what it means is…

You mean that was in your chemistry books in school?

No, I mean I did an experiment and I expected some outcome. But I did something heinously wrong and my chemistry set exploded. And now I have no eyebrows.

Dave, that’s the hindenbug.

That’s the hindenbug. [Laughter]

So, you get this weird conditional at the end that’s supposed to catch something that only happens once in a blue moon. And it turns out that we don’t actually have a conditional for it. We don’t think this can ever happen, but we don’t guard against it because we don’t think it can ever happen. And so, you go down to the end of the thing and you basically say look, we don’t have time to write all of the guard clause and the catch and to set up the framework to have the external service return the weird thing.” So, I’m just going to write an assertion right now that if this method receives this illegal data, it should PNE. It should hindenbug. It should blow up. So, you read the test suite and you’re like, wait a minute. Why are you writing unit tests that say this should blow up? I say, well I’m documenting that we have not covered this. And yeah, if you drop a stick of dynamite down this particular exhaust port, I’m mixing metaphors here, but the Death Star should blow up if you shoot a torpedo down this exhaust port. We’re not going to fix this design flaw. So, the correct behavior is it should all blow up. And if that turns out to be a real problem in production, we should come back and fix it.

Yeah, the external version of scattering assertions around is the characterization test, where you just write some tests to validate your assumptions about some code that you didn’t write or don’t remember. I’m actually a bigger fan of putting the assertions straight in the code and making sure that the code is exercised rather than writing lots of characterization tests.

Yeah.

Very cool.

It sounds like we’re winding down on the topic. Is there anything left to cover here?

It’s been a fanta-

Danielle, you mentioned something that I like in your slides. You mentioned OpenGem or bundle open?

Yeah.

Yeah, just as a way to open up the version of the gem that you’re actually using and say, put in a debugger or check your assumptions about what’s going on in the depths.

I’m waiting for James to…

I’m over here steaming, but I’m trying not to say anything. [Laughter]

Say it, say it.

I promise I pristine when I’m done.

That’s actually a key point that not everyone knows. Do you want to explain that?

Oh yeah. After you’ve been messing around in your gems and you’ve made a big mess, if you run bundle pristine, that will actually return them to their nice pristine state so you don’t screw it up for yourself later.

Nice.

Right. That’s my complaint, is that I see people do it and I don’t see people follow it with pristine.

Oh yeah, that’s a problem.

And that would be a difference between development and production when you’re done. [Chuckles]

And again, a lot of people don’t know about gem pristine. But I really do like the technique of just opening the gem using bundle open or the OpenGem gem, there are a couple of those, because on numerous occasion I have convinced myself that I am looking at the right gem, the right source on my disk, and in fact I am looking at the wrong one. And the one the one that my code is actually using is a different version sourced from a different place on the disk. And so, I just filled my head with lies by looking at the wrong thing. So yeah, if you can actually tell Ruby to open up the thing that your program is using rather than making assumptions about the thing your program is using, much, much better.

So, I love that technique used with caution. I ran into a wall doing that once. And I had to do the second order technique of that, which is I was trying to figure out what was going on in one of these authentications gems. I forgot which one it was. Maybe it was authlogic. It was so heavily refactored inside that it was really hard to understand the control flow within the code. And I was trying to find where it was actually doing the database query to fetch the user record that it was going to do the authentication test on and I couldn’t. I looked through the code for almost an hour and I couldn’t find it. And so, I just dropped down into Active Record and raised an exception in the code that did Active Record find.

Yeah, and then looked at the backtrace?

Yeah. And then I looked in the stack trace and I saw where in authlogic it was doing its thing.

Nice.

Yeah, that is a technique that I use sometimes, just a really cheap way of finding the backtrace.

Yeah.

I’m hesitant to use that one because I feel like sometimes I do that and there’s a rescue in the way and I never actually see it anyway.

There is that.

That’s actually a really good test though. If you raise and exception and nothing happens at the top level, somebody’s rescuing that. And you need to go find out who’s rescuing the exception and stab them in the face.

Yeah, but if you do that in Rails, your odds of it happening are 60% or something.

And this is also the case where after a little bit of trying and failing with that technique, that’s where I’m going to haul out the debugger.

Yeah.

Yeah.

I do find the debugger useful for stepping through other libraries and figuring out what the heck is going on in them.

Right.

When you say the debugger, are you talking about something like the debugger in RubyMine or are you talking about the debugger gem?

So, I’m usually talking about something like the debugger gem. And this is [chuckles] this is one of my points of major angst with Ruby, because as Danielle you pointed out in one of your slides, there are three different debugger gems for three different versions of Ruby.

Yes. [Laughter]

And it is a Ruby tradition that a new version of Ruby will be released many months before somebody comes up with a fixed debugger gem that actually works.

Yes.

Yeah. [Chuckles]

And this is one of the many reasons that I say never ever trust the debugger, because for one thing on the day you need it most, which is the day that you upgraded to a new version of Ruby, you will suddenly discover that it doesn’t work anymore. Mind you, this is a bigger problem in Ruby than in some other language ecosystems where they actually care about debuggers. But yeah, I do think that people should know how to use just the basic command line debuggers. Also Chuck, as you were saying, RubyMine puts a frontend on the various debugger gems that can make it a lot nicer. You can have watches and you can have really handy breakpoints and all kinds of good stuff that you would expect in a full IDE. The only downside is that because it’s trying to find out all sorts of stuff for you, it winds up calling methods on your behalf that you don’t know about. And so, the IDE is doing things like something.inspect in the background or something.class in the background. And if the problem is actually, if it’s a tough one because of some kooky monkey patch or something where .class breaks, you can actually wind up with weird second-order errors as a result of the debugger trying to find out information for you without asking you and things break. You think you’re getting to the root of the problem because you’re seeing these new errors. And in fact, they’re unrelated. They’re just other crap getting in the way. So, you got to be careful with that stuff. And I think it is a good idea to be familiar with just the raw command line debugger.

Yeah. The less tactful your debugger is, the more likely you are to generate heisenbugs.

Right.

So, I have another question that’s related to figuring out were bugs are and that is fixing bugs. Is there certain etiquette to that or do you just go in and just ram your code in there and make sure all the tests still pass?

It depends. How down is production right now?

Right.

Are you not serving requests at all? Are you unable to take credit cards? What’s the current status?

So, let’s assume, if it’s down the business needs it to be back up so they’re not losing money they would otherwise make. But just in general, if it’s something that’s not as critical I guess, or it’s something that’s just going to be pushed in the next release.

Well, what about one of these two-wrongs-make-a-right situations where you have a bug that’s not biting anyone too badly, but fixing that could make things worse by making another bug bite you more often?

Yeah. Like surfacing the bug, yeah.

Right, yeah.

Yeah.

That’s the time when you need to take a step back and do some design work.

That’s the time when I feel like changing careers. [Laughter]

That’s the time when I think about changing religions to one that lets me drink.

Oh man, just makes me think of poetry. There’s this Kenneth Koch poem, One Train May Hide Another. And I always think of it, one bug may hide another.

Yeah. Good point.

So, that’s actually I think something that happens to us fairly often when we’re debugging, is if you’re not aware that there are perhaps multiple bugs going on, you fix the bug that’s in front of you and things still don’t work.

Right.

Right. The reality is things are actually improved, because what you had was two layers of ignorance that happened to be lucky and you’ve removed one layer of ignorance. And now, you’re a lot less lucky. I don’t know if two wrongs make a right, but two nots do make an equals. And that’s kind of profound. Anyway, you’ve removed that first layer of ignorance and now you’ve got a real problem. And yeah, do you deploy that to production or do you maybe push that off into a story branch and say, guys, we got to take care of this.

Making that choice is I think in the situation where you have all the information, is a lot easier than us talking about it abstractly.

Yeah.

But the hard thing I think is being able to understand that there are multiple bugs interacting here. And that you actually have made things better, even though it seems like things are failing more.

Right.

Although, for some definition, that’s making things worse. [Chuckles]

Failing doesn’t have to be a bad thing. There have been times where the right thing to do in code was very difficult. But the chance that I would fail was very small. And this depends obviously on your use case a lot. For example, Twitter at the rate they take in tweets, something that fails on in a million times is a matter of seconds. So, obviously, they need a failure rate much higher than that. But it can be okay to let something fail, especially if you can detect it and communicate correctly with the user or something. A lot of times, it doesn’t matter. And sometimes, it’s worth throwing the failure in there, recording the metric. If you can recover correctly, record the metric, and move one. And then wait and see. Are we talking about something that’s happening once a month or once an hour?

So, going back to what I think Avdi was saying about using the command line debugger tools, a friend of mine has been doing some really fun stuff lately where she’s decided that, “Sometimes, reading Ruby is too annoying. I’m just going to use strace instead to understand what’s really happening and what the system calls are rather than parsing through the Ruby.”

Wow.

I think I know who you’re talking about.

You probably do. I’ve got a link here. It’s Julia Evans.

That’s a little hardcore.

I love these blog posts so very, very much.

Oh yeah, oh yeah.

This is going to hurt my head.

Have I picked these? I think I’ve picked these.

I don’t know.

I would have, but I can’t figure out which ones to pick because I love them all so much.

She does these things where she’s like, “I wrote a kernel and it turns the screen red and it hangs. And that was really cool.”

[Chuckles] That’s amazing.

But yeah, this idea of just seeing, what are the system calls? I don’t need to think about the highlevel language. I just want to understand what’s happening on this low-level and sometimes simpler level is really nifty and really interesting.

That is very interesting. It’s another way of changing your view. You’re looking at it in a different way than you were before.

It also gets over the fact that, strace gets over the fact that one of the big problems with debugging in Ruby is that in a regular debugger we don’t really have a view into the C code.

Right.

That’s a point where we had no debugger. We were using a version of Ruby that didn’t have a debugger. And we managed to cobble enough of it together using set_trace_func which is a Ruby call that is exposed to your Ruby code. And so, we wrote a debugger with it. And if you haven’t played with set_trace_func, that’s fun. It’s worth looking at.

Actually, if you haven’t played with it, don’t. In the newer versions, play with the object-oriented API that they have now.

Yeah.

It’s tracer I think it’s called or something. Way cooler. You can turn it on and off for given sections. You can do really cool stuff with it.

Tracer or TracePoint?

TracePoint, that’s it. Thank you.

Okay.

Yes. Thank you. Alright, should we do some picks? I think we’ve pretty much covered it.

I think so.

It’s been a great chat.

It really has.

Yes, absolutely.

Yes. It’s been a lot of fun. Thank you for having me on.

This is one of those episodes where I’m going to have to go back through and make a list. [Chuckles]

I think it’s pretty high on the brain-breakage meter.

Yeah.

That’s a good sign.

Yeah.

That I can be proud of.

Stuff I got to look into.

Awesome.

Alright, let’s do the picks. Josh, do you want to start us with picks?

Sure, I’d love to. I have two fairly relevant picks. So, my first one is going way back to 2009. So, Nick Kallen gave a presentation at GoGaRuCo in 2009 called Magic Scaling Sprinkles where he talks about a technique for tagging all of the log entries from the various servers in their system with a unique identifier that propagates through the system. So, if you’ve got a request that comes in the frontend system, you put a token on it. And then as that flows through the different layers of the system, all the logging gets associated with that. So, if you have an issue, you can unify the view of that request across all the different pieces of the system. And that was a pretty cool trick. So, I think that that’s relevant to how we debug this stuff. So, that’s one. And then Mislav Marohni_ did a post last month about every line of your source code is documented, where he talks about techniques for using Git to investigate the history of your source code and understand why changes were made and in what order. So, that’s a very good post on a powerful debugging technique. And that’s it for me this week.

Alright, James, what are your picks?

So, I just have one. It’s not totally debugging related, but the thing I’ve been debugging lately is I finally built a system that uses the Russian doll caching technique in Rails. So, I’ve had to figure out, how does all of that work? And especially with Rails’ invalidation of templates and all of that, which is kind of neat. And there’s this blog post from Kevin Faustino that is really concise and explains it very well I think. So, this blog post has been very helpful to me in getting my head around how all the caching stuff works in Rails 4. So, if you want to know, I recommend this. And that’s all I’ve got.

Alright, Avdi, what are your picks?

Pretty much all non-programming related picks this time. First of all, I have recently restarted watching the show Community. And it’s a lot of fun. I can’t remember. I might have picked it before.

But it’s a nice replacement for 30 Rock.

Avdi, where do you watch that?

Hulu.

Hulu. Okay.

Yeah. I have to put up with the awful ads. But [inaudible].

It’s also on Amazon Instant Video, if you don’t mind paying for it.

Oh right, okay.

Okay. But it’s not in the Prime ones?

No, I don’t think it is.

Okay, cool. Thanks.

And since I was a little kid, one of my favorite places in the world to go is the Air and Space Museum down in the National Mall in D.C. And for years, I have been wanting to go to their adjunct facility out by Dulles Airport, the Steven F. Udvar Hazy Center. And I finally got to go there. Me and a friend went out there the other day. And it was just as cool as I had always imagined. They have a gigantic hangar there. And they have just an amazing collection of aircraft and spacecraft. They have an SR71 Blackbird and they have the space shuttle, one of the space shuttles. And I got to get a picture of a launch vehicle that my dad worked on.

Cool.

Just an incredible collection of aircraft all through history. They have there all the stuff they couldn’t fit in the main Air and Space Museum. So, I highly recommend checking it out if you’re ever in the D.C. area. And one other thing, I saw that Neo, the company that Jim Weirich was involved with, has created a fund in his memory, the Weirich Fund. And it looks like it’s going to be funding a scholarship for somebody to learn about computer programming. And that seems like a pretty cool cause in Jim’s memory. So, that’s something to look into. Alright, that’s it.

Cool. David, what are your picks?

Okay. Dorothy Parker once wrote the cure for boredom is curiosity. There is no cure for curiosity. And I have two picks today that are all about curiosity. My two favorite ways to explore curiosity are looking at things and learning about them. So, my first pick is a microscope. It’s the My First Lab Duo-Scope Microscope. You can get it on Amazon for $60. It’s really cheap. It’s basically the highest end kids’ toy version of a microscope you can get. And by highest end kids’ toy, I mean it’s got actual real glass optics. So, the functioning parts of the microscope are real. And all of the other kit stuff is kind of chintzy and plastic. But it works like a real microscope. And you can actually look at cells and stuff like that. It’s very, very cool. The second way I love to explore curiosity is building things. And I was going to pick MAKE Magazine, but I see that James picked it a while back. Been a while enough though, so MAKE Magazine could probably be a second pick. But they’re running an ad right now for an iPad game called Mho’s Resistance. And one of the hardest parts of getting into electronics is learning the stupid resistor color codes. And Adafruit has put out a little $1 game for the iPad where you can spin the dials on the resistors to try and get the right value and drop it into the circuit without burning up the circuit. It’s stupid and silly and fun. And it teaches you to remember your resistor color codes. And it’s a lot of fun. So, those are my picks. CHUCK: Awesome. I’ll jump in with a couple of picks. First off, I’ve been looking at doing some iOS development with iBeacon stuff. And most of the iBeacon hardware out there is you can get a unit anywhere from $40 to $100. But I’ve had this Raspberry Pi sitting on my desk forever. And so, I was wondering if you could do it with that and it turns out you can. So, all you need is a Raspberry Pi, a power supply, you can get the SD card with the operating system just off of Amazon for $8. And then you get a little USB dongle that you plug into your laptop. And the Raspberry Pi has a, what do you call it, USB ports on it, so you can just plug it in there. So, the whole thing costs about $30 or $40 I guess. But the nice thing is that then you’ve got this modular system that you can totally hack on and play with. And I found an article that tells you how to make an iBeacon out of a Raspberry Pi. So anyway, I’m looking at playing with that. And then the other thing that I’m going to, it’s a little bit of a shameless self-promotion, but I’m starting to work on a SAS product for podcasters. And I figure that most of our listeners listen to podcasts. And so, I really just want to ask if I can. If you have a podcast, if you could go to FeedWrench.com, it’s just a little survey. And really all it is, is I’ll have a list of the features that FeedBurner provides and you can uncheck any you don’t really care about. And then I’ll have about five features in there of which you can choose three that you really do want that FeedBurner doesn’t provide. And that’s just to give me an idea of what’s important to people. And in the meantime, I’ll be doing the development on the critical parts of the app that FeedBurner does. So, it’s sort of a FeedBurner replacement except it’ll do things like work. [Snorts] [Laughter]

And it won’t have the 512 kilobyte limit on the feed. And I’ve had a few other issues with FeedBurner. So, that’s why I’m working on it. And every podcaster that I talk to about it gets really excited. So, I figure it’s a good idea. So anyway, I would really appreciate it if you’re a podcaster, if you go fill it out. And if you listen to podcasts, tell the host of the podcast you listen to, to go fill it out. And that’ll help me out a ton. And that’s all I got. Danielle, what are your picks?

Sure. So, let’s see. A few friends of mine are running a conference in New York called !!Con, which is going to be two days of ten-minute talks, which looks really exciting. And I haven’t actually submitted a proposal yet, but I should. And everyone should. And that should be really awesome. First thing I’m super psyched about right now. Let’s see. Like I said, lately I’ve been playing around with some Clojure. And that’s because this guy Kyle who goes by aphyr on Twitter, which I’m probably mangling the pronunciation of, has been writing some wonderful Clojure tutorials called ‘Clojure from the ground up’. And he’s really [sweet about it]. I [inaudible] him. I said, “Look, do you have a good idea for open source projects where I could just submit some tiny patches while learning to pretend that I actually know Clojure?” And he said, “Oh yeah. I do this Riemann thing. You should totally submit to it.” That’s that monitoring library that I was mentioning that deals with monitoring events from distributed systems. And I’ve just been having a really fun time messing around the code there and contributing to it. And it’s a really interesting library. I haven’t actually used it. I’ve only written bits of it, which feels a little bit backwards, but has been really fun for me. Beyond that, mostly I just want to list a few books that do feel relevant and useful to me. I really like Daniel Kahneman’s ‘Thinking, Fast and Slow’, which came out a couple of years ago. And it’s all about how we have these two ways of thinking through things. We have this fast system and the slow system. And the fast system is where we use our heuristics and we jump to conclusions and we do all those things we spent the past hour or so talking about trying to avoid doing. And we have our slow system where we pause and we actually go through things and try to understand them and catch ourselves. And it talks about how to force ourselves into using the slow system instead and different kinds of cognitive vices you can learn to recognize and avoid. I also really like ‘How to Solve It’ by George Pólya, which is about solving math problems in theory. But it feels like it’s full of useful problem solving tricks generally. Mark [inaudible] recommends it that I read a book called ‘How Doctors Think’ by Jerome Groopman, which I thought was wonderful. It just talked about a lot of tricks to make sure that you’re diagnosing health problems, how doctors figure out how to diagnose what’s going wrong based on the symptoms that people tell them about. And a lot of ideas there felt really, really generally applicable and useful to me.

These are great picks. They’re actually relevant, even.

Yeah.

They are great.

Okay. Two more relevant books, then. Fine. I can list books forever. ‘Working Effectively With Legacy Code’ by Michael Feathers.

Yes.

Woo!

That’s where I first, first learned about that trick where I don’t know what’s happening, let me write a test just to understand what’s happening and document the existing behavior, not to prove that it’s working correctly, not to prove that it’s doing what it should be doing, just to prove, to document what it is doing.

Yes.

Just perfectly obvious in retrospect, but didn’t know it until I read that book. And I can’t remember anything else I learned from that book, but it was worth that one idea.

I’m sure there were other useful things in there. I should probably reread it. And the other book that comes to mind immediately is, I think I mentioned this earlier, ‘Coders at Work’ which is just a bunch of interviews that Peter Seibel did. And he asked a bunch of really interesting people things like, how do you approach debugging? And, do you read other people’s code? And full of wonderful, wonderful stories.

Cool.

That’s very cool. Alright, well I guess we’ll go ahead and wrap up. Thanks for coming, Danielle. It was a very, very interesting conversation.

Thank you. I had a lot of fun. I really appreciate it.

Thank you, Danielle. We appreciate it.

Alright. Well, we are doing our next book club book. It’s going to be in about a month, if I remember right.

May 9th.

May 9th, so a month and a half. And that will be ‘Object Design’ and we’re going to have one of the authors on the show, Rebecca Wirfs-Brock. So, go get it. Apparently it is on Safari Online, which seems to be the best way to get it. And you can also find it on Amazon, though it’s out of print. So, it’s a little bit pricey to get there. But anyway, pick it up, read it, and we’ll talk about it then. And besides that, thanks for listening.