The Rogues discuss rescue projects.

 

For our 100th episode, we will record as the ‘Ruby Rouges’.

Hey everybody and welcome to Episode 89 of the Ruby Rogues podcast. This week on our panel, we have Katrina Owen.

Hello.

Josh Susser.

Good morning, everyone!

James Edward Gray.

I’m recording this episode from the Hot System.

Avdi Grimm.

I’m recording this episode from [Inaudible].

And I’m Charles Max Wood from devchat.tv and I just got back from New Media Expo and CES.

Where was that?

They were both in Vegas.

So can you actually tell us about what happened there or does everything that happens there, et cetera?

[Laughs] Well, so New Media Expo, I was actually there speaking on Podcasting. They also had the Podcast Awards and we were nominated as one of the Technology Podcasts.

Thank you, everyone.

And being nominated is kind of a big deal. So yeah, thank you everybody who nominated us.

I think we’re the only specific language Podcast in the entire Tech category. So, that was kind of crazy.

I think we were sort of the underdog. We definitely stood out as being pretty specialized, I think.

For sure.

Yeah, but it comes down to votes over the 15 days that they have the voting open and Todd Cochrane was talking about maybe shortening that. So, it might run a little bit differently next Fall. We’ll see about that. But anyway, Leo LaPorte was actually emceeing the event. And we lost out to ‘The Audacity to Podcast’ show and that’s done by Daniel J. Lewis. He’s actually made a pretty good name for himself in the podcast and technology space. So, I just want to congratulate him. But yeah, we were in the running. So, thanks everybody who voted for us. And you know, we’ll look forward to being nominated again next year.

Next year, we’re in it to win it!

No kidding.

Yeah. So, we have another announcement and I’ll let James take that one away.

Yeah, we did the voting recently. We asked all of you to go and vote about what was the best episode of 2012 and we had some great runners up. The Angela Harms episode did very well. Also, Gary Bernhardt is always great when we have him on the show and his episode where we talked about his testing style that was also very high up there. But the winning episode was Therapeutic Refactoring from Katrina when we had her on the show back before she was Rogue. So, I thought that was really cool. It kind of validated our choice to invite her to join us. So, that was awesome and thanks for letting us know what you enjoyed.

Yeah, we’ll have to get Katrina back on the show.

That’s a good point. We should ask her if she’d come back.

Anybody know where to find her?

She’s stuck in Norway. I don’t think she’ll come. [Laughter]

Next time we see her, we’ll have a foot tall golden statue of a programmer. [Laughter]

Dang!

Very cool.

So hey, what are we talking about this week? We have our amazing guest star, Katrina Owen. What are we talking about?

Gee! We’re talking about Rescue Projects today.

What’s a Rescue Project?

I don’t know. You’re the definition guy, tell me.

Let’s see, Rescue Projects, Britney Spears. [Laughter]

No, no, no! We need rescue projects that have some hope of being rescued.

Oh, no! She’s doing great now. A couple years ago, she was a total wreck. She was shaving her head on video and her life had become a cheap reality show. Now, she’s back on her game. She’s dancing, she’s singing, she’s recording albums.

Maybe, we should have her on the show. [Laughter]

What the heck!

I would die. [Laughter]

What did I call into this morning?

Die of excitement or die of shame. I’m kind of curious.

A Rescue Project is a project that you started with the best of intentions but somehow, things went wrong. And you need a little help from a trained professional to get you out of trouble. I have an example rescue project that I can mention, and that was Twitter. So, when I worked at Pivotal Labs, it had made news all over the Internet that Twitter had hired Pivotal Labs to come in and help rescue -- because they were getting fail wells all over the place and features weren’t working and Twitter was in a terrible state. Their whole software development process was so messed up that it literally took months from the time that people completed a feature and pushed it into Git, before it got pushed out the door and users got to use that new code. It was just a mess. They hired Pivotal. Pivotal went in. I never worked on that project but Pivotal went in and worked on tons of stuff and really helped them with their process and they turned it around and they really have an excellent engineering process and department.

So it sounds like that rescue was actually more about the process than the code quality.

And I think that’s a really important lesson because in most rescue projects, the code isn’t the big problem.

That’s a really interesting point.

It’s like Glenn Vanderburg was saying that, what is his quote? That he never saw a project fail for technical reasons, and the corollaryof that is he’s never seen one that succeeded for technical reasons.

Yeah. But at the same time, I mean, in most of these cases, you have a ton of technical debt. And so, I think the trick is, is to try to get them to quit digging in that hole as opposed to -- so, the code is kind of a hole but you got to get them to quit digging so you can fill it back in, and build the building with the right kind of foundation.

When Chuck and I were in Hawaii for Aloha Ruby Conf, we sat down with Corey Haines for a little

bit at dinner one night. And Corey Haines kind of made the joke and we all laughed. But he thought someday Rails projects would become synonymous with rescue projects. Obviously, we do see a lot of rescue projects in Rails but at the same time, I’m wondering if that’s even a bad thing. But Rails does give us the means to get something up and get it validated. And then, if you’re successful and maybe you didn’t have the best of systems in the beginning, the best processes and that didn’t result in the best of code or whatever the reasons, then you’re validated and you know that you have the means to get it fixed. So, it ends up becoming a rescue project. Is that necessarily a bad thing?

Well, I think you can generalize that to pretty much any software projects. So like, Java projects all become rescue projects eventually. I think it’s just something that happens in the ecosystem that as the average lifetime of existing projects gets longer and longer, more of those things will be at a state where they’re in need of rescuing.

Well, I think the other thing is, is that it’s basically -- I’m trying to think a consequence of the fact that in a lot of cases, we’re solving problems we haven’t solved before. And so, in some cases, we wind up solving them the wrong way. We don’t have the experience to know that we’re kind of walking down the wrong path. And so, we get way down there. We get real deep into the hole and then we realize that there’s a better way. So in some cases, we wind up rescuing our own projects when we have the time to go in and actually get it done right. When the pain of the decisions we’ve made gets to the point where it’s painful enough to force us over into that place where we refactor out the decisions we’ve made in the past.

Okay. So, we’ve talked about a few ways to look at what a rescue project is. Maybe we can spend just a few more minutes trying to come up with a compendium of the kinds of things that need rescuing because we’ve danced around it a little bit here. Does someone have one to start with?

Sure. The thing I see a lot of is performance that the initial design when something gets moderately popular, then it begins to have performance concerns and can’t keep up. And I think that’s totally normal and doesn’t even necessarily speak to the skill used to build it in a lot of ways because, for example, we often don’t know what the stress point is going to be until something does get a little popular and has the pressure put on it, so to speak. Then that makes those kinds of things more obvious and we know what to fix, right?

Yep, performance is one. I’ve seen ones where it was a one person development team. And you know, he’d been working on this code base for a couple years all on his lonesome. And you can imagine what that code base looked like. The typical thing is you have a very small team or a one person team that have created a code base and they realize that it needs some serious refactoring to get to the point where other people can work on it.

A lot of times, it was their first project or close to their first project, or their first project in Ruby or something like that. I mean, I’ve talked to people that were that one developer and they’re like, “Yeah, when I started this, I had no idea what I was doing. I had no idea how to test properly and so, none of the stuff that I wrote in the first couple years has any tests around it and stuff like that.” JOSH:

Stability, that’s a rescue project thing.

There’s a lot of other team patterns that can lead to it. I’ve seen the high turnover project. We had this one person, then we had this other person, then we had this other person. And you know, there’s the low communication team that just didn’t talk to each other that well. Maybe they were one of the more dysfunctional distributed teams where there was an onsite team and then there was like one or two satellite people that didn’t really communicate with the inside team as well.

I’ve seen also the outsourced project.

Oh yes, I’ve definitely seen that one. [Laughter]

How about the big port?

Actually, I was just going to bring up one that was pretty much that. I worked on one where they gave them the existing .NET app that they wanted rewritten in Rails. And it was like, “Here’s the code you have.” It was a very short number of months, I can’t remember. And we have to launch it in that time or we lose money, you know blah…blah…blah. And so, it’s basically almost a direct port, yeah.

I have another one. It’s the prototype that went into production. [Laughter]

Oh yeah.

She’s talking about all my programs now.

Which is highly related to the one that’s probably one of the most common I’ve seen which is the startup. [Laughter]

Then, you have the one where, “Oh, but we’re only going to use it once.” The $5 hair cut. [Laughter]

Yes.

Yeah. I think this is all very related to Michael Feathers calls any code. Well actually, I think Michael Feathers calls any code that doesn’t have tests around it, ‘Legacy Code’. But you know, there are other people that basically call any code you wrote yesterday Legacy Code. And it gets into sort of the gray area where in some ways almost every project is a rescue project, in some ways, in some areas. And not every “rescue project” I’ve been on has really been like, “It’s going to die tomorrow if we don’t completely turn it around.” But many, many projects find themselves going the wrong way on the entropy scale, put it that way. [Laughter]

Okay. So, we have this big list now of different kinds of projects that need rescuing. But I think the thing that they all have in common is the panic factor.

Very good point.

The people managing the project feel like they’re in trouble and that they are not capable of solving their problems on their own. So, rescue problems are not always consultants parachuting in and saving the day and then riding off into the sunset.

That’s what it’s like for me. You must be doing it wrong. [Laughter]

We all aspire to live up to that model you set for us, James. But I think that sometimes teams rescue themselves. You don’t need to call an outside help for rescue projects and I’ve rescued my own projects before. And I’ve seen other people do it too. So you can be your own hero.

Very true.

I think the term ‘Legacy Code’ is pretty interesting. Chad Fowler did a talk on that where a Legacy in a lot of other industries is a good thing and in our industry, it’s turned into this terrible thing that you absolutely do not want. And he coined a new term for it, and he calls it ‘Aftermath Code’. [Laughter]

I like that.

So, this is one of the things I have noted to talk about on here. And I’ve been caught saying this several times before is that existing code is not an asset, it’s a liability. I’ve had this conversation with startups a lot where they’ll come and they’ll say, “What are your assets?” “Well, I have this two year old code base and it does a lot of great stuff.” And, “That’s not an asset, that’s a liability. That’s two years of technical debt that you need to manage.”

It’s also two years of learning, though.

Sure, but that’s the asset.

Right.

The asset is the value that it provides for the company and how much your users are paying you for it, how much you’re learning from your code, all of the knowledge that’s included in it. But the actual implementation of it, all of those lines of Ruby or Java or JavaScript or whatever you wrote it in, that’s generally just a maintenance cost.

Yeah. That’s one thing that I think a lot of people don’t realize is that it could be well written code, it could be well organized, etcetera, etcetera. But time moves on. And so, as people find exploits on the framework or they find exploits on the Ruby VM or even just different things performance-wise or other that may or may not cause you pain, your code is backsliding just for the fact that people are still innovating and solving problems.

There’s an interesting question. In a way, I totally understand what Josh is saying about the existing code being a liability and such. But at the same time, as Katrina mentioned, that Chad Fowler talk was really good. And it’s about basically if your code’s been around years or something like that, or even two, that means hopefully ideally, it wasn’t just some company pumping, burning money or something. But hopefully, that means that people were using it and like it and that’s why it’s here and that’s why it needs improving and getting better. So, that’s kind of what I was trying to say earlier about the fact that a project needs rescuing is kind of nice. That we can get some prototype or whatever out there, some system in some form, test it out and then see if we need to take it to the next level.

What are the super powers that a hero needs to come in and rescue a project?

Git is the highest superpower.

And a good testing framework.

A good attitude.

Yeah. Boy, that’s so key. Isn’t it?

I think that’s probably the key if you want to rescue your own project is find a way to be the one that brings a fresh attitude to the rest of the team and that can be really hard because the cycle of entropy really, much as anything else, is a psychological cycle.

Can you talk about that cycle a little bit? We’ve mentioned that a little bit but I think that’s really important.

Yeah. It’s typically a lot of little things stacking up. You have a deadline to get a feature out and you maybe cut some corners to get it out on time. And those corners you cut, the tests you didn’t write make that bit of code harder to start testing the next time or adding tests for the next round of features. And it starts a little snowball where it spreads outwards from there as you write more code and you look at what it would take to get that code under tests. Like, “Ah, I don’t have time for that.” And so, you just go ahead and hack it in and then figure out what to do about the bugs as they crop up. And it spreads outwards to the team. It’s an attitude towards the code that gradually infects the whole team. And you just get depressed about the code. It bums you out. You get up in the morning and you bring it up and it just bums you out. And then, that leads to all kinds of psychological stuff where you start practicing avoidance, hitting Reddit or Twitter instead of touching the code because it bums you out so much to look at it. Then there are sort of tangential issues that crop up where maybe the tests you have start taking forever and so that gives you another excuse to get distracted. And then, you start having conversations with your team that are less about, “What awesome thing are we going to do next?” And are more about, “Man doesn’t this code suck,” which is more about commiseration sessions. It’s good to commiserate but at the same time, now the young impressionable members of the team that just joined start getting infected with the attitude of, “We all know that the code sucks and we just sort of deal with it,” rather than the attitude of ‘zero tolerance for suckage’. I think that one of the reasons that people ask me to do pair programming sessions is just because they’re kind of bummed out by their code. And they want to have somebody who’s not bummed out about their code and who can bring in a fresh technical perspective but also a fresh psychological perspective.

I totally agree. A fresh perspective is, I don’t know, sometimes the entire reason to bring somebody else in.

That’s a great point. Everything you just said, like I was just sitting over here nodding the whole time. I hope everybody can see that. But like when we look at code, especially when we’re brought in on a rescue project, I don’t know about you guys but my immediate instinct when I look at somebody else’s code and maybe it’s also when they tell me that this code needs some help or whatever. My immediate instinct, I can open it up and I swear I barely see it. And I just immediately have that reaction, it sucks. And you have to actually fight against that instinct.

You know, it’s funny and I’m sure you’ve had the experience of opening a piece of code, thinking that, and then realizing it’s a code you wrote. [Laughter]

It’s often mine.

It’s also worse though when it’s code that somebody you admire wrote. [Laughter] AVDI:

It’s funny though because I know exactly what you’re talking about and I used to experience that all the time. Open up any random piece of code, “Oh, this is crap.” And it’s actually I’m not feeling that as much anymore.

I hope I get over it soon.

You know, I’m looking at that and I'm realizing, it seems to correlate with the fact that I’m a consulting pair programmer now instead of somebody who’s on contracts long term. And I guess, it’s all about that fresh attitude again. I feel like I’ve got a little distance from, not just my projects, but all projects which obviously could easily be labeled as a hit and run artist. But it’s really improved my perspective on people’s code because nowadays, I’ll start a session and somebody will bring up their code and almost invariably the first thing they say is, “This code is crap. I’m so sorry about this code. This is the worst code ever.” Number one, it’s never the worst code I’ve seen because I’ve seen some really bad code. But number two, they are always like, “Where do I start? What is the worst thing about this code?” And I’m like, “I don’t know. It looks fine to me. What about it that is actually causing pain right now?” Once you can get a little bit of equanimity and sort of Zen distance from code in general, I think it starts bugging you a little less that lots of little details of the code are wrong because the methods are too long or the names are one letters. It becomes more about, “Well okay, this is code. Code is often like this.” But it may or may not be causing anyone any pain right now. So if that particular aspect of it isn’t causing anyone any pain, who the hell cares? Sandi Metz talks about the Omega Mess where you got this horrible, horrible, horrible ball of code. But if it works and if you can wall it off behind a nice interface that is basically just a little slot in the cell door where you pass food in, then who cares what it looks like on the inside?

The object-oriented world lets us build some scar tissue around that and move on.

Gary Bernhardt has been talking about that recently in a thread on Parley. That whole thread is just gold. It’s really worth reading. He has typical great Gary Bernhardt comments about how the code is evil and he might decide to don a hockey mask and go killing people and stuff. And as a programmer, he’s not prepared to handle that. But he talks about like what you said about how he likes to keep the evil together and kind of put that somewhere. And he’s not saying that he can’t be rehabilitated. But he feels like the neighbors should be warned when it moves in. [Laughter]

Yeah.

That whole thing. Just everything he says about code is just great.

So, we’ve talked a lot about the different types of rescue projects and some of the symptoms that we see in the teams and things like that. But I have to wonder, the one thing you keep bringing up Chad Fowler, I remember when he was talking about ‘The Big Rewrite’ and I’m wondering if and when that’s an appropriate response to a rescue project.

That’s a great question. I have it written down right here for us to talk about. When do you declare bankruptcy and go for the rewrite?

I don’t know if I completely agree with the term bankruptcy because ultimately, what you’re doing is you’re just saying there’s less technical debt to rebuild it than there is to rehabilitate it.

Sure. I mean, we talked about that when Sandi was on. Sometimes, you just need to throw in the towel and start over. That doesn’t mean that there’s nothing worth value harvesting in the existing code base. There’s often a lot of learning encoded there. But at some point, you need to -declaring bankruptcy doesn’t mean you’re going out of business either. It means that you’re restructuring the company and you’re just being protected from the creditors for a while, in many cases.

That’s a good point. I think it’s important to realize that it doesn’t have to be an either or choice, it doesn’t have to be commit all the way to rehabilitation or the complete rewrite. Like in the Songkick blog post which I know I’ve talked about in the past where they talk about how they move to a SOA architecture and stuff like that. They rewrote at least large portions of their system. I’m sure some of it’s still the same. But they get that really carefully. First, they would go through the code and wall off part of it behind some service object. They were still doing the exact same thing. They just inserted an object between the caller and the callee, right? And then that object had a method and then inside it was the same code that was always there. But once they had made that wall, then they can work on the other side of that wall safely. So then, they section off that part and turn it into an SOA and end up communicating with it remotely or whatever. And so, piece by piece, they kind of did a rewrite through rehabilitation.

Yeah. That’s one of my favorite things about Michael Feathers’ book on Legacy Code is just the thing that really liberated me, I read that code on my first Rails job when I was dealing with a really, really big app that had been accumulating since well before pre Rails 1.0 days, and that book was so timely for me. One of the biggest things was just the idea of sprouting, finding a seam somewhere and sprouting off a class. So, you’re not adding anymore code to one of the existing legacy classes. You’re adding code to a new class that you can drive test first from day one. And it just really turned my perspective around.

Coming at it from the other side, if you do choose the rewrite as opposed to the rehabilitation, then it also doesn’t mean you’re beginning back at ground zero. It’s like what Josh said, it’s usually more what was learned from the initial experiment or whatever that actually ends up being the asset that you take forward and sometimes that’s huge. The project I’m working on right now, we’re basically doing a rewrite from a system that was used primarily as a prototype and got successful in traction and now, we’re pretty much doing a rewrite but there’s a rule, ‘never start with SOA’ because you would need to see the system built out first. Well, they did that. They built the system out. And now, we’re building its replacement and we know it needs some SOA in certain areas. So, we’ve already gained that benefit. And how we’re building it out now is pretty cool because we build these services, and then we hook that service into the live system to some amount. So that we can see how traffic goes through it, monitor how well that’s going, what needs fixing, does it do the job we need it to do. So piece by piece, we’re actually connecting them into the main system but then someday, we’ll end up swapping that main frontend out for another frontend which we’re working on that does even more.

I worked on a project that was highly coupled and that was really our problem. So you couldn’t maintain one class without maintaining the whole system. And yeah, so we opted to rewrite it. But ultimately as we segregated things out, we did build an SOA architecture. But as we segregated things out, the funny thing was all the logic we needed was still in those classes. And so for the most part, we could just move the classes over, and then we could start working on the coupling that it no longer could count on because it was in its own component and that worked out. And so, it was a rewrite but it wasn’t a complete rewrite because we were stealing whole chunks of code out of the original piece mainly because the logic was sound and it did what we needed it to do. So yeah, I don’t think that you wind up going all the way one way or the other when you do this.

I think that’s very much dependent on the situation and the specific project. But I agree that most of the time, you don’t need to just jettison all the old code and start over and that the transition is valuable. I have a little four-step plan for how to do a rescue project that I will toss out here.

That’s interesting because I think I have a three-step plan I was thinking of earlier. So, I want to hear it.

I was hoping for a 12-step plan to get off of your terrible code. [Laughter]

So, the first step is admitting you have a problem. [Laughter]

I admit it, okay.

I don’t want to make fun of anybody doing a recovery. But I think that first step is really important. Okay. So, my little four-step plan is, step one, as Avdi said, when you’re stuck in a hole, stop digging. So, the first thing you do is you stop the bleeding. You stop making things worse. And usually, depending on the project but usually, what that means is stop writing code. Just stop all of your stories in tracker. Just don’t write new stuff.

That is so counter-intuitive though for a lot of people because they think they’re digging their way back out by writing more code.

If they were making things better by writing more code, they wouldn’t need to rescue.

That’s right.

So, the first thing you do is you stop writing code. Then the second thing you do is you take some time to figure out what the actual problems are. So, you have to do some research, you have to look at the code, you have to look at the thing in production, look at the error logs, just collect the information. And a big part of that is going to be looking at the team and how the team works.

A huge part of that. What you don’t want to do is come up with a description of the problem which is actually like your, ‘Pie in the sky’ re-architecting of the whole system.

Right.

So you do that. Collect information and analyze it, figure out what the real pain points are and what are the things that need to be fixed. And then three is, you make a plan for fixing that. What’s your approach? And okay, so this is that step two. And then, a miracle happens or something. Step three is profit. Step one, collect underpants. Step two, what? So to make a plan, that’s the magical, “What are we going to do?” That’s the insight. And then, step four is you just incrementally work towards digging yourself out of the hole.

It’s really important to realize that this is something that takes time. Day one is going to be painful and ugly and it’s still going to be painful and ugly a month later. It’s a very gradual process.

The other thing I want to point out too with this process is a lot of times, you wind up circling back to one or two because you start addressing some of these problems and you realize that there’s a deeper problem somewhere else, in the team, in another part of the code. And so, you have to constantly be looking at it and going, “Okay, what is the most effective? What is the right level of solving the problem?” Because it’s not always obvious when you start doing it, but at the same time you have to start somewhere. And even if you can’t identify the core issues at step two, you can identify some issues at step two. And then, you gain information as you work through steps three and four. Then you can come back and say, “Okay. Now that we’ve solved some of these issues that we figured out were more symptoms than this other thing, then we can start to address this other thing and make a plan to get that taken care of.” And eventually, you get to the point where your core issues are being handled and your code is better for it.

You know, what Katrina just said reminded me of something that I was talking to someone about in a session really recently where I was working with a guy where some refactoring, some pulling stuff out of a controller into some other object. And I was explaining how to write the tests as to drive that extraction. And I was saying, “You’re probably going to want to just set up the database in a state that would exercise this code and then move the code as it is over into its new home. Try to set things up so that you can just copy and paste the code that’s currently in this controller back into its new home.” And he was saying, “But that’s going to be a database tied test. So, that’s going to be a slow test.” And what I had to explain was, “Yeah it’s going to be, because the code that you’re moving over from the controller sort of digs deeply into the database structure, in order to easily extract it out, to just move it over as a copy and paste, the tests are going to be tied to that structure as well. And so they’re not going to be the most optimal tests in the world. But they are going to be a step better than what you have.” And I think that’s what a lot of people miss when they’re trying to get out of a hole, is that you can’t just jump to the top of a hole. Sometimes, you have to build little platforms along the way that are still kind of dirty and it’s not going to -- like Katrina said, it’s still going to be messy a month later and the steps along the way might not be steps all the way up to the imaginary ideal where everything’s super fast, isolated tests and stuff like that. You have to be able to accept incremental steps.

I think you also have to accept adding warps in order to get yourself out. You have to build scaffolding and these shims and these structures that will help you get out that you can then remove. And then, you can improve the code and remove scar tissue. But it’s messy, it’s surgery, it’s blood and guts.

I wound up explaining the possible path to the future that would involve actually, eventually throwing those tests away that we had just written, as the code got factored out more nicely.

I do like the imagery of the scaffold and really just real quick, I just want to point out that the scaffold makes it so that you can navigate your code again, so you can move up and down. It’s not the end product. But it makes it so that you can move through the code and get the stuff done that you need to like you guys were saying. And then eventually, you’ll remove it because you don’t need it. But in the meantime, I think it’s interesting that it’s an intermediary step that makes it so that you can get stuff done.

So, one of the things that I’ve noticed on every rescue project that I’ve done and I’ve done quite a few of them. I think anybody who’s been around and consulted for a of couple years has done a bunch of rescue projects. But I’ve noticed that on every project, there is one really huge issue in the code that is sort of just the worst thing possible in the code, and breaking that thing apart is often like clearing the log jam. After that, it’s much, much easier to make progress. There was one, my first really big rescue project. They were having terrible scaling and performance issues. And it ended up being their entire business was built around one query in their application essentially. Everything in their web application was feeding information into the database, so that the user could come to the website and they would refresh the page and they would see a whole bunch of information there. And that was an incredibly difficult query. That’s like going to your Twitter page and seeing the feed there. Or you go to Facebook and you look at your Timeline. It was that equivalent for them. It was the 98% activity for their users and it was a terrible query. And they didn’t realize that it was that one query that was breaking everything because that was putting such a ridiculous load on the database, that the database server was so overwhelmed, its entire performance was hosed. And it took us a while to figure out what was going on. And so, we had to do that step two research and analyze a lot. But once we figured that out, we came up with a reasonable strategy for solving that which involved changing the scheme out and de-normalizing things a lot, and then partitioning data so that there wasn’t as much of it to deal with all at once. It was a tough nut to crack. But after we fixed that one query to be able to run, I believe we improved it. It ran about 7,000 times faster when we were done. And the entire database layer got much more performant. And then, the entire app got much more performant and then we could actually see what all the small issues were. So, we got visibility into seeing what was going on in the application at large and things became much easier to deal with. And I’ve seen that pattern recur in many projects where there is one really tough nut. And if you can make a good go at that, then that frees up everything else in the application and suddenly it’s like a breath of fresh air. Somebody opened a window and everyone can breathe again and you can make progress. Does that send familiar?

Yes.

Yes it does, absolutely.

And that definitely comes back to the idea of a lot of times it’s the things that you can do for the psychological help of the team that really make a big difference.

Oh, yeah. Big win is a huge psychological boost for the team.

And I like the idea of getting the one big thing out of your face so that you can see all of the other issues.

I just want to talk about the psychology a little because I liked your steps a lot, Josh. And about the only thing I would add to them, in that second step where you’re trying to figure out the problems and what it needs, I think one of the key things is we get trapped in like a blame/guilt cycle there. We have that natural tendency to just move to that, and you’ve so got to get away from that. I talked earlier about how when I looked at somebody else’s code, my initial reaction is, “That sucks.” And what I actually mean is that ‘I don’t understand it because I haven’t spent the time to figure it out’. And then I’ve mentioned several times, I’m sure, that I think, “Oh, I’ll rewrite it in ten lines.” And then three weeks later, I have a really good appreciation for what that code was actually doing. And in that whole cycle of how things go bad, of not knowing where the stress was going to be on the system and that one query would end up being the horrible sticking point that kept the app from going forward, that’s all totally normal. That happens to everybody. And there’s no reason to blame your self. It’s really that you need to treat it as that asset of, you’ve learned what the problem is and how you need to fix it and what to avoid in this particular application that you have. That’s the one thing I wanted to add about step two. And then step three, also from kind of a psychological event, is you can fix it. Anything can be improved. And you won’t hit a point where you’re absolutely required to throw it away and rewrite it. Now that may be advantageous because you’re going to change a whole bunch of things at once, then maybe you can rebuild system in place and you can build on the side on something else and you can go much faster because of that. And it’s okay to redo and then transition over or something like that. There may be reasons but you can fix something. You can build the scaffolding. You can redirect the flow of water a little bit so that you can move some items and shift things around. It will be a hard long process, in some cases, but everything is fixable. You can change it.

James, I agree with that. But I will throw one caveat in there and that sometimes, the problem can’t be fixed with the team the way it is. I’m going to steal one of my picks and mention it now and that is Sarah Mei’s keynote from Ruby Conf last year. I may have actually picked that before in a previous episode but Sarah’s talk was really great. She talked about Conway’s Law where he said that the structure of your software follows the structure of your team. And I think Grady Booch’s example of that was that if you have a four-team project writing a compiler, you’ll end up with a four-pass compiler. And that the structure of your team or the processes that your team follow, are often the thing that you can get the most value from modifying. A lot of times just changing the way that the team communicates. And I’m going to plug extreme programming now because a lot of the practices in extreme programming were built to physically work against the natural concludity of developer versus product people versus managers, to all basically silo themselves and set up divisions in the team and point fingers, et cetera. Those are all very natural things to happen when you have certain kinds of team structures. And XP is very much about making feedback loops tight and communication responsive and it’s very much a process for clear communication. And if your team is in trouble, if your project’s in trouble, you can often do far worse then just going to XP explained, read through it and start applying stuff.

Just to focus in on one aspect of that, one of the lightweight changes you can make in your teams dynamics is just to start pairing. I think that’s really, really important on like Legacy projects, if for no other reason than moral support. If you know you’re in the rut where you’re going to go and look and that code and then you’re going to go and hit Twitter because, “Oh my God.” It can be so helpful to just have somebody by your side, either to commiserate or just to sort of help you keep focused because you’re talking about the code, or whatever. But just not to work on it alone. And hey, if that happens to be somebody that you’ve been playing the blame game with, then so much the better because if you actually sit down with hem, whether remotely or co-located, if you actually sit down with them and talk about the problems, you’ll probably find you have a lot of common ground and a lot less recrimination.

The other thing that I think a lot of rescue projects have is that all of this depression and frustration has caused broken trust. Like people don’t trust anymore, the clients don’t trust that they’ll deliver. And one of the really important things that I think, going back to what Josh said, I think that Agile or XP helps with, is getting that feedback loop as tight as you can. And the communication sort of improved so that the communication happens more quickly as well, will help build back that trust.

Absolutely.

That’s a great point.

I think teams also though, one you’re talking about Josh, especially that some things can’t be fixed with their certain teams. But I guess I feel too that then you start refactoring the team, and then you move to refactoring the code problem. There is a path there. It’s maybe a longer path and a trickier path but there is a path.

Another team refactoring is, just get some rest.

Absolutely.

The thing about the cycle, the death spiral, or the death march really is that you’re more and more stressed because things are getting behind. And so, you work longer and longer hours and that just makes it worse.

I always forget the power of walking away. That’s like one of my big crutches. I’ll get into it with some bug I had. This happened like two days ago, where I’m chasing something down, I’ve rewritten the code like three times to run into the exact same wall every time. And you just, you get stuck and you get those blinders on and you can't think outside the box anymore. And I ended up leaving, not by choice, but because I had some where else I had to be, and leaving. And sure enough, I’d been away from it for like an hour and the answer just pops into my head and not thinking about it at all. It’s just that you cannot underestimate the power of walking away.

Yeah. David and I used to do that, as well, when we were working at public engines. We’d, one of us would be facing a hairy problem and one of us looked at the other guy and go, “Hey, you want to go for a walk?” And so, we’d go walk around the parking lot and then come back in. And then usually, the solution would present itself during the walk, even though we were talking about life or whatever.

Also, what we were talking about earlier, that often, it’s the fresh perspective that has the most value, from an outside consultant. And so, if you can change your perspective, you can do it yourself.

One of my favorite tricks in all of programming that has saved me more times than I can possibly count, whenever I’m doing something and I’m really stuck, I just stop and do it backwards. Like whatever I’m trying to do, for example, if I’m trying to put commas in a number, every three digits or whatever, and it’s hard from the front, hit string reverse and try again. And the reason that almost always works is it immediately changes your perspective. It forces a perspective change. So, just reverse it and try again, works often.

Okay. So, I didn’t have a chance to just drop this quip in naturally but when we were talking about rescue projects. My life is a rescue project. [Laughter]

I think there are days we all feel that way.

When I wake up two minutes before the Rogues podcast begins, I definitely feel that way. [Laughter]

That’s why you need pairs. Right, Avdi?

[Laughs]Yeah. Actually, that’s kind of really true for me right now. And I guess, maybe I’ll talk about it a little bit just because I feel like there is a lot of crossover for like how this applies to the projects. But you know, I spent the last few weeks in a pretty depressed state. And looking, introspecting about it, I realized it’s the result of accumulated stress over months and months. And I think what my psyche was desperately trying to do was to force me to take a break despite my protestations that now of all times, I cannot possibly take a break. And I quite honestly, I spent the last week or so doing pretty much nothing but watching ‘Buffy, the Vampire Slayer’.

Except for Season 4, that’s incredibly therapeutic.

Yes, it is.

And I finally just said, “Okay fine. I accept it.” I’m at the point where I was basically too depressed to do anything else. I started dropping my commitments and goofing off instead. And I’m feeling a little better now. And speaking of having a pair, my wife gave me some really good counsel and support during this period. But I almost think that for some teams, the most counter-intuitive but effective thing might just be to take a break. You know, goof off for a while. And it’s going to feel really weird because now of all times, you can’t possibly afford to do it. But chances are, it may be the only thing you can really afford to do.

Well, that’s about the moment when Doctor Crusher insists that Captain Picard go on vacation.

Right. And he’s like, “I can’t do that now.”

And he always winds up on a planet with beautiful women and lots of problems.

And lots of problems, yeah. I saw that episode.

Yeah, really. His vacations don’t seem that relaxing to me. [Laughter]

Well, he has the disadvantage of being the main character.

That’s true. There is more expected of you when you’re the main character. [Laughter]

Especially when you are bald in such a way that all the women can’t seem to keep their eyes or hands off you. [Laughter]

I always thought that was funny. He’s like this older guy with a bald head and whatever he had all the women are like, “Oooh…”

In the future, that’s hot. [Laughter]

Anyway, let’s get to the picks. I think we’re devolving. [Laughter]

We’re devolved.

Avdi, you want to start us off?

I guess I’ll just pick this video that I think somebody on the Parley list recommended. And it’s not a new video but I hadn’t seen it. Michael Feathers did a talk called ‘The Deep Synergy between Testability and Good Design’. And it’s a really good video. I enjoyed it, worth seeing.

Alright. Josh, what are your picks?

As I mentioned before, Sarah Mei’s keynote from Ruby Conf last year. The title, ‘An Insufficiency of Good Design’, was a really great talk about how to navigate the waters of your process and your team structure and what impact that has on the product that you build and how you build it. So, I’m going to pick that. And then, I don’t think this has been picked before but I saw it mentioned on Parley and I actually finally tried it. It’s been on my radar for a bit. So Harvard has some online tests that they’ve made available that people can do to help uncover their implicit biases or their unconscious biases. So, they have these little five to ten minute tests that you can do where you can find out your unconscious biases against people based on race or based on gender and professionalism or various other things that people often have biases about. And I walked through a couple of these last night. And it was really fascinating. A lot of it lined up with my expectations and some of it didn’t. So I think it’s really good. There has been a lot of talk about diversity in the Ruby community in the last maybe couple of years even. But it’s heated up a lot in the last six months, I think. And it’s good for us to start at home and look at our own biases and see how that can affect our thinking. So, that was really interesting. And there is something I just noticed this morning and that’s those White House petitions. The ‘We, the People’ site at the White House. There was a blog post today talking about their raising the level of signatures from 25K because that now gets -- they reached that threshold usually in a couple days rather than a couple of weeks. And so, they are dropping the threshold. But as a part of that announcement, they announced that they are open sourcing their signature platform. And that they are hoping for contributions from people. But they are also making the platform basically freely available for anyone to adopt and use for their own efforts. I thought that was something worth mentioning. I think it’s really great to see the government involved in open source and contributing to open source.

But they did refuse to build a Death Star, Josh.

Josh, did the bias detector know that you’re biased against functional programming? [Laughter]

No. Nobody has built that. I was hoping somebody would build that.

We need the programming equipment but Josh has biases against functional programming, the callable interface.

The callable protocol.

Yeah.

I have no bias against functional programming. I just like to see it kept in its place. [Laughter]

Yeah. You see something written in a functional language and he says, “This code is crap.” [Laughter]

I’ve never said that about functional programming. I’ve only said that about functional programming masquerading as object-oriented programming.

Nice.

Alright. Katrina, what are your picks?

My first pick is an article that was published in the PragPub Magazine back in 2010 called the ‘Mikado Method’. And this is really helpful if you’re trying to break dependencies or discover dependencies in a complicated code base. And basically, you figure out what you need to do. And you do it as naively as you can until you get stuck. And then you draw a diagram saying what the dependency is, you scratch your whole refactoring and you start working on the dependency. And you build up this whole graph of dependencies which you then can do one by one. It’s really interesting. The second is a gem by Tim Pease called ‘Servolux’. So, if you’re writing a lot of daemons, this is very, very helpful.

That’s fun.

Yeah. I’ve been using it for a few months.

Yeah, it’s awesome.

So, that’s available on Github. And then the third pick is an iPad game that somebody told me about a few days ago called ‘The Room’. And it’s just this beautifully wrought puzzle game. It’s a long sort of puzzle box experience and it’s just gorgeous. So those are my picks.

Hey, Mark. Did nobody get that?

I didn’t get it.

Some listener is going to get that. [Laughter]

We hope.

Nobody here has seen ‘The Room’?

I haven’t.

No. [Laughter]

You guys are missing out. Alright. New extra pick, the movie ‘The Room’ which is the greatest film ever made. I particularly recommend watching it with the RiffTrax commentary. [Laughter]

I love those. James, what are your picks?

I actually had like a couple of totally goof off picks picked and then, the conversation was so good. I’m like, “I have picks that are actually related to this.” So, I switched up. So, I’ll be all fun and games next week but I’m all business this week. If you are doing rescue projects and refactoring and trying to get performance or busting people, eventually you’re going to find SOA. And we’ve talked a ton about that and how valuable it is. But I think it’s only common sense. What does Sandi say? 90% of the time, the answer in programming is go back, take smaller steps. Break it up into smaller pieces. So, SOA is all about that for whole applications. And there’s a really good talk from Ruby Conf from Chris Hunt, one of the Square guys, who we’ve had on the show. I mean, he works for Square, not that he is square. And it is a great talk about how they do service-oriented architecture at Square, really pay attention, they decide things like, “This is important to us.” And then they re-architect their process around that thing that is important to them. And I’m not even making judgments on whether or not it’s even a good thing to say is important or not, but it’s interesting how they change their process to put that thing in the front and it’s very awesome. And then, there is kind of a follow-up blog post written by one of the Collective Idea people, Steve Richert. Anyway, it’s about how you can use SSL and authentication with SOA to do some kind of neat tricks that I hadn’t thought about before. So, those two are kind of a nice one-two punch. And then for non-technical but totally mind-expanding for me, at least, it came up in Angela Harms’ episode about how valuable nonviolent communication is. And I finally got around to actually reading a book about nonviolent communication called ‘Nonviolent Communication’ and it’s really great. It’s one of those books where I’m pretty sure I don’t agree with absolutely everything in it, which means it was the perfect book to read, of course. I wouldn’t say this about very many books but I’m pretty darn certain that just reading this actually made me a better person. And it applies to almost everything we’ve talked about today, with the psychology, the blaming, the figuring out the needs, all of that is in there; super on topic and relevant. Those are my picks.

Does this mean you’re going to stop beating me up every time we have a conversation?

Shut up, Avdi. [Laughter]

Yeah, stop using your razor whip and cut us into shreds.

Yeah. That’s for sure. I’ve actually been going around. I don’t think -- I knew empathy was important and I think it was one of those things I always thought that I had. But I didn’t even really understand the definition of empathy until I read this book. And then, I’ve actually been making real effort to get better at it lately since I started reading it. And the other day, I said something to my wife and she actually turns around and she’s, “Are you practicing your empathy on me?” That was great. [Laughter]

Nice.

Awesome.

Well, I guess I’m the last one. I have two picks. They are both gadgets. And they’re things that have been real nice to have especially going down to CES. The first one is my video camera. It’s a Canon, not Canon. It’s a Kodak Zi8 camera. It’s just a little hand-held camera, does video at 1080p and 720p. And you know, I got some of the videos of some of the cool stuff they had at CES. And it just fits in my pocket which is real nice. So, I don’t have to carry around anything that’s real bulky or anything. And I can capture video of whatever it is that I’m looking at. The other one is something that I’ve kind of been using since I got back from CES. It’s something that my wife got me for Christmas. It’s a Powermat. And it uses inductance, magnetic inductance to charge devices. And it’s kind of cool because it will actually detect when your device is fully charged and then it will just stop putting power into it. And so, it doesn’t run all the time. And she got me one with three charging stations on it. So, I had to go buy a couple more of what they call ‘doors’, which are just little magnetic squares that just sit on there. And you can also get cases for your phones and stuff too, that have the squares built in, or the doors built in. But anyway, the Powermat is really kind of a cool deal. And I put a link to both of those in the show notes. So, go check those out. And yeah, we will wrap this up. Also, you can go sign up for Ruby Rogues Parley. I wasn’t on the show last week and I haven’t listened to it yet. So, I don’t know if this was brought up but you can go to parley.rubyrogues.com. You can spell ‘Parley’ with an E or an A, and it will take you to the right place.

But just for the record, the correct spelling is P-A-R-L-E-Y.

Yes. But anyway, just go to parley.rubyrogues.com and sign up. And we have the Stripe signup now. So, if you want to use a credit card instead of Paypal. I know a lot of people were asking for that. It’s available and you can go sign up. With that, we’ll wrap up. We’ll catch you all next week.

And just thanks to the panelists, again, for coming.

Anytime.

Thanks.