Check out JS Remote Conf and submit a Ruby Remote Conf CFP!

 

02:26 - Refactoring (Definition) and Where People Fail

03:55 - Tests and Refactoring

10:59 - Managing Scope

11:42 - Why We Refactor; The Value of Refactoring

17:13 - Refactoring Tools 

20:40 - When Refactoring Gets Put Off; Establishing a Code Culture

26:23 - Refactoring Strategies

37:38 - Performance Tradeoffs

41:42 - Generative Testing

50:33 - Measurement

53:08 - Examples and Resources

Picks

Longmire (Avdi)
Clash of Clans (Chuck)
Star Wars Commander (Chuck)
Cleaning your office (Chuck)
Hsing-Hui Hsu: Time flies like an arrow; Fruit flies like a banana: Parsers for Great Good @ RubyConf 2015 (Coraline)
Betsy Haibel: s/regex/DSLs/: What Regex Teaches Us About DSL Design @ RubyConf 2015 (Coraline)
Velocity 2016 Call for speakers (Saron)
RailsConf 2016 (Saron)

 
[This episode is sponsored by Hired.com. Every week on Hired, they run an auction where over a thousand tech companies in San Francisco, New York, and L.A. bid on Ruby developers, providing them with salary and equity upfront. The average Ruby developer gets an average of 5 to 15 introductory offers and an average salary offer of $130,000 a year. Users can either accept an offer and go right into interviewing with the company or deny them without any continuing obligations. It’s totally free for users. And when you’re hired, they give you a $2,000 signing bonus as a thank you for using them. But if you use the Ruby Rogues link, you’ll get a $4,000 bonus instead. Finally, if you’re not looking for a job but know someone who is, you can refer them to Hired and get a $1,337 bonus if they accept the job. Go sign up at Hired.com/RubyRogues.]

Hey everybody and welcome to episode 238 of the Ruby Rogues Podcast. This week on our panel we have Avdi Grimm.

Hello.

Saron Yitbarek.

Hey everybody.

Coraline Ada Ehmke.

Oh, I didn't see you there.

I'm Charles Max Wood from DevChat.TV. Quick shout-out, two things. JS Remote Conf is coming up, so sign up. Also, I've opened up the call for proposals for Ruby Remote Conf in March. So, if you want to speak or if you want to buy tickets, you can do that too, go to RubyRemoteConf.com and click on the 'Be a Speaker' link. This week we're going to… It's just us. We're just talking about refactoring.

Woohoo!

Before we get started, I just want to point out that this is my one year anniversary on Ruby Rogues. My first episode was 12/10/2014, episode 185 with Noel Rappin. And I am so happy to have spent the last year with everyone and learned so much and talked with so many amazing people. So, thank you, Chuck.

You made it.

Congratulations!

Woohoo!

I didn't do it. You did it. [Chuckles]

Alright. So, refactoring. This was something that was suggested by Coraline. And I know that you've done a talk on refactoring. Do you want to get us started with, what I'm curious about is where people fail in their refactoring.

Well, I think that we should start by defining our terms. There's the dictionary definition of refactoring where you are changing the internals for example of a method without changing its interface. I take a broader view of refactoring. This is rewriting [shit], basically.

[Chuckles] That's kind of where I come down.

Yeah. So...

Rewriting it to make it better.

Exactly. And sometimes that means changing the API, but doing so in a really reasonable manner and a really careful manner. So, I think one of the places where people fail in refactoring is not having a strong enough test suite to support the refactoring effort. I think it was Michael Feathers who, well Corey Haines attributed this quote to Michael Feathers, “If you're refactoring without testing you're just changing shit.”

So, having a test suite thus supporting your refactoring efforts is really, really critical. And really, it's not fun but I think that before you start on a refactoring project, you need to take a look at your test suite and flesh it out fully before you even get started, which takes away from some of the excitement of getting to rewrite some bad code. But it's really a necessary step. And if you don't do it, you're just going to mess yourself up later.

Now, I know somebody's out there and they're going, “Alright. So, I need tests before I refactor and I have this really, really hairy code that I want to go refactor but it is impossible to get tests around. What do you tell people there?”

Well, there are different kinds of tests that are appropriate to the situation. Like, maybe if you have a really fat method in a controller and you can't really tell what it's doing, then maybe outside-in testing is a better way to approach that and check the output of the function instead of worrying about all the internals. Because if something is untestable, it's untestable and that's probably why you're there in the first place.

Yeah, I'm with you on that. And that's the approach that I take with those kinds of things. And I've run into that before where essentially, something was really hard to test or really hard to put your head around. And so, by just writing some test to characterize, “I pass this in, I get this back out, I see these side effects,” it at least is somewhere to start. Because then I know if I change something along the lines of those assumptions anyway, then I know about it.

Yeah. And that's almost like attribute testing where you're not worried so much about the internals or what the code is doing but what is the output of it. Does the object that results from a call have certain qualities or not have certain qualities?

Yeah.

Another thing that's important about tests as they pertain to refactoring is that through the refactoring process you're going to probably be rewriting the kinds of tests that you don't want to be long-lived in your codebase. And that's fine. You can write tests that you're eventually going to throw away. As developers we like deleting code, but we're terrified of deleting tests. And I think that's something you really have to overcome through a refactoring effort because you're going to need tests to guide you along the way. But those are not things that are necessarily going to be useful in the long term.

I love deleting tests.

Tests are code, right? So, deleting code and deleting tests...

We're afraid of what we might lose. Like, “Oh, what if there's an edge case that that test is accommodating,” or I've just seen test suites grow and grow and grow because people either don't understand what's being tested or they're just afraid to lose some sort of key indicator of things going wrong, some sort of regression or something along those lines.

Well, the other thing is that we think about tests as a thing that we're supposed to do and supposed to have. And test coverage is this pound-your-chest awesome number that you want to have. And so, when you're deleting that it's almost like deleting verses out of the bible, right? You know that that's important. [Chuckles]

You know that there's something there that you should probably know about. And yeah, instead of actually evaluating and saying, “I wrote this and it may not have enough value to keep.”

You're talking about the Jeffersonian refactoring the bible then?

[Laughs] I've never heard of that.

The Jefferson bible, he cut out all of the references to anything mystical and just kept the practical wisdom.

Oh, that's interesting.

Oh. So Coraline, at some point earlier in one of your answers, you talked about how your codebase just might be too hairy in a place where it's untestable. I'm wondering, how do you decide when your codebase is untestable and when you're being maybe just a little bit lazy?

One thing you can try is try changing something and see if your test suite breaks. And if it doesn't, that means your tests are not as good as you thought they were.

Yeah, one thing that I've seen when I've had to refactor some hairy code that was hard to test was I just started trying to write the test and I find pretty quickly there's a lot going on here for me to keep in my head and codify into one test, or into a suite of tests that are simple enough so that you can look at them and understand what's going on. And so, at that point then I actually move into the, “Okay, I'm just going to characterize this.” The thing is that I feel like I'm heading down the path that Katrina gave in her Therapeutic Refactoring where she starts then writing the replacement for it in a nice and testable way before she starts deleting things and stuff like that. But ultimately then yeah, then you can start testing what you're going to refactor, too.

I think the key is to make sure that you just keep the refactoring and the refurbishment or whatever you want to call it, separate. Like plenty of times I run into situations where I want to refactor. Refactoring is defined as the tests stay the same. But then I make a change and I realize it breaks the tests and I realize I don't want to undo that change. It's a good change. And at that point, I need to switch hats. I need to explicitly switch hats and say, “Okay, I'm not refactoring anymore, because clearly I have come into a collision with some tests that do not reflect my thinking anymore. This is the test that is trying to force my code to stay in an old way that I don't want it to be in anymore.”

Mmhmm.

So, it's time to switch hats. I'm going to back out of what I was doing, where I thought I was refactoring, and I'm going to kill off this test. And maybe I'm going to write some new tests, I don't know, and create some new functionality maybe in parallel or something, based on those new tests. But yeah, it's a matter of separation to me.

Mmhmm.

What's fundamentally different in your approach Avdi, when you take off the refactoring hat? Because I guess I consider that rewrite as part of a refactoring effort, a lowercase R refactoring if you will.

Yeah, well I mean I guess I've been sufficiently burned by lowercase R refactoring over the years. The definition gets expanded to the point where all of the code being broken for two weeks is still referred to as a refactoring session. And that's just, it's just, I don't think it's a useful definition at that point.

Right.

So, these days I try to be very explicit, I try to stay very explicitly with the original Fowler definition of refactoring, which is I'm refactoring when I'm altering behavior without altering functionality, whether that functionality is hopefully tested or maybe I have to manually test it if it's not automated tested. But yeah, I'm breaking tests, then that's the point where I have to think, “Okay, these tests don't reflect what I want anymore and I need to think about that before I talk about refactoring.”

Right.

You talked about the two week long refactoring where crap is broken, too. And the thing is that if you are making that kind of sweeping change to your code, you're rewriting it. And even if you aren't completely....

Yeah, absolutely. That's a rewrite.

If you aren't completely rewriting the entire system, that is a lot of change to keep track of, instead of making small, targeted changes that make the code better and easier to reason about. And sure, there may be some grand refactoring or grand rewriting that is a series of small refactorings.

But if you're making that kind of sweeping change, you're in for a world of hurt.

It was a case of lying to ourselves, because nobody wanted to say, “We're going to do a rewrite,” because rewrites are scary. And so you say...

“Well, we just need some time to do a refactoring,” where this refactoring is something where all new development comes to a screeching halt and everything's broken for a couple of weeks. And honestly, a couple of weeks is like the optimistic scenario. But yeah, I think we're just scared of saying 'rewrite'.

I think some of it too comes down to developers being really bad at managing scope. We complain about scope creep if we're given an assignment, we're given a task or a feature to implement or a bug to fix and then the product stakeholder just keeps changing their mind about what they want. But we do that to ourselves through these large refactorings or rewrites as well, where we might set out to change something that we find really annoying or change an implementation that's really inefficient and we end up just expanding our own scope to the point where it's no longer manageable.

Yeah. It's like the days when I decide I want to clean my bathroom counter and an hour later I find myself at a Home Depot shopping for a leaf blower.

Scope creep. It happens.

Yeah. So, there are some, I think it might be helpful to take a step back and talk about why we like to refactor. What gives us the motivation to start refactoring in the first place?

For me, I think one of the biggest problems is when it starts getting hard to test.

That's definitely a sign that things can improve, right?

Yeah. Well, it's when things get… something gets big enough that I feel like I'm sort of awkwardly reaching into a big complicated mess in order to test it. And at that point it's like, “Hey, I think it's time to pull something out of here.”

I...

[That's] a logical, practical answer. Mine is more, I'm trying to avoid emotional pain. I'm at a point in my coding career, in my coding journey, where I'm not always sure what the next step is or what the best tool is to use but I can sense when I'm going in a terrible direction. And so, when I'm refactoring I feel like I'm soothing myself. I'm releasing the pain that I feel when I'm looking at a fat method and it's very chunky and just terribly done. I'm getting rid of that.

I think that in a certain sense, refactoring appeals to our aesthetic sensibilities. Like, we can be offended looking at a...

Yes.

Piece of gnarly code and want to make it something much more beautiful, something that's more elegant. And I think that's a natural urge that we have as developers. We want our code to be beautiful.

I love that so much. I feel like I hear all the time that coding is creative and artistic. And I don't really know how I feel about that in general. But the moment that I see it the most is in the refactoring process. And that's when I go, “Man, that's really pretty.”

Well, and the other thing is that a lot of times I feel like I'm looking at the code and I'm going, “That is really ugly.”

[Laughs] Yes.

Right? And it comes down to those aesthetics. And it's like I can feel or sense the problem there.

Yes.

This leans a little bit more toward code smell, but I can almost smell that the code isn't quite right, and so I want to fix it.

That can be really hard to sell to management though. Just like...

Yes.

The code is ugly and it needs to be made more elegant.

That was actually going to be my next question, is as developers we get the reward and the benefits. But for a lot of people listening who I'm sure are not in the best, may not be in the best engineering teams and may not have a boss or a management or a lead that gets it, how do we explain the value of refactoring?

One of the things that I like to talk about, and I talk about this in my data-driven refactoring talk, is that you do have to make the case to management. And one of the ways you can do that is by starting to track how much time in a given sprint is spent on bug fixes versus developing new features.

[That is a good one].

Because looking at code creates all the friction around change. So, if you can demonstrate that you are... inelegant code tends to be a source of a lot of bugs. So, if you can demonstrate that you can be more productive and deliver more feature points in a given sprint, in future sprints rather, by addressing code that is difficult to work with now, that can be one way to try and sell that to management.

I love that. And I feel like there's also an emotional cost to that, too. Because when you're fixing bugs, it's not exactly happy. You don't feel productive and like you're making progress. You're not moving forward, right? So, there's just the time which is more objective and there's the team spirit in a way of having to constantly fix things instead of moving forward and growing.

Yeah, definitely. And I think that's an easy to collect metric because if you're doing Scrum and you have story points assigned to things, you should be able to assign points to bugs and chores. And it's really simple math to figure out how much of our effort is being spent fixing code that is not optimal.

I think it's absolutely necessary to make that case sometimes. And everyone should feel empowered to make that case. And I even think that it should be, I think that programmers should be empowered to simply say, “Look. This is making me miserable and I shouldn't need any more justification than that.” If management is somebody who has one of those impeccable desks, you say, “Look, would you feel comfortable working if your desk was just covered in crap?” And that's basically the same case. That said, I'm really a big advocate of, if where you can, when you can, keeping refactoring sufficiently part of what you do that you don't need to make that case. When you look at the way Martin Fowler writes about it, the way Kent Beck writes about it, refactoring is very much, it's not something you have to justify because it's something you're always doing. It's just part of the work. It's part of the rhythmic nature of evolving code.

Mm.

So, I'm not saying that you shouldn't make that case sometimes, because sometimes you have to. It's just the way things develop. But I think ideally, we should be aiming for a situation where we never have to make that case because it's just a part of every moment, well every third moment of programming.

I think that unfortunately, I'm not a big advocate for Scrum actually. And I think that one of the reasons is that we get these artificially imposed deadlines and all these pressures that we put on ourselves by, first of all by estimating in the first place and secondly by making a commitment to get a feature done within a certain point range by the end of a sprint. But I think some of that pressure, when you're under that kind of pressure and you feel like, “I'm going to be measured by whether or not I can deliver this feature,” that might provide some motivation to not do the refactoring as you go that in your heart of hearts you know you should be doing.

I think we should talk about impediments, other than management, to refactoring.

Sure, like what?

For me, one of the things that I've noticed as I've tried to be more mindful about my coding practice is that there are a lot of little things that subtly raise the opportunity cost of refactoring. And a lot of people are shocked when they discover that I actually spend part of my Ruby coding time in Emacs and part of my Ruby coding time in RubyMine. And one of the reasons that I do that is that I realized in this practice of being mindful about coding, that using a traditional text editor was actually raising my opportunity cost for refactoring a little bit, relative to some of the other tools that are now available. So, I could see that extracting a method, I could feel in my hands the amount of effort that it would take to extract a method in my editor. And that would raise my resistance just a touch, just a degree. And that raised resistance here or there and everywhere it adds up.

And you can say, “Oh, you can write a macro,” or something like that. But actually some of the modern refactoring tools are a bit more advanced than that. And something like RubyMine, they actually have made some major steps forward where if I pull a method up into another class or if I extract a method out, it's going to do a really good job of finding all of the local variables that are referenced and turning those into parameters and asking me what I want to name them, et cetera. It's not perfect because Ruby is a dynamic language. And so, static analysis is limited. But it really does lower that opportunity cost to a significant degree, to the point where I can watch myself and I

can see I program in a different way and I make different choices when I'm using one tool versus the other.

So, I think that's something to be really carefully aware of, is we like to believe that our choices in coding are just rationally and consciously made. But I think that a lot of the choices we make are affected by unconscious resistances that build up over time.

I love that. I haven't thought of it that way before. I'm wondering Avdi, was there a moment that you realized this was happening and you changed the tools that you were using as a result of that?

I don't know if it was one moment. I've always had an awareness. I can be typing something out and I can think, “You know, I'm doing this in a very character by character way and I ought to be able to do this in a structural way.” And I think that a lot of programmers have probably felt that pain at some point or another. But I think where the awareness really steps up is when you force yourself to use a new tool that handles something differently. And you do that procedure, you make the effort to say, “Oh wait, is there an automated way to do this? Oh yes, there is. Let's try it out. Oh my gosh. That worked way better than I expected.” And you feel this sense of ease, this sense of, “Ah.”

SARON: Hmm.

That's what you have to expose yourself to, I think, to really see that resistance. Because you don't really see the resistance until you have something to measure it against.

Yeah. Yeah, and that's one thing that I really need to get better at. I have a pretty high pain tolerance for my tools. [Chuckles] So, I can use a crappy tool and still not feel motivated to change and to do something better. But I didn't think about the connection between using that tool and actually thinking about how it affects how I build in code. So, that's something I'm going to be more aware of, too.

Yeah. Well, there is a certain level of comfort that comes from familiarity. And so...

Yes, definitely.

Switching or learning a new tool sometimes is hard. One other impediment that I've seen to refactoring is just the fact that usually the code that you need to refactor the most is the code that nobody wants to touch. So, what happens is everybody puts it off and puts it off and puts it off until it becomes basically an emergency, until that's where the critical bug is. Or until that's what's holding up the release, or that's what slowing things down to the point where we just can't ignore it anymore.

I think that ties into what Avdi was saying about making refactoring a part of your regular process, too.

I agree.

Because by putting it off you're just accumulating, you're giving it more power over you.

Yeah. And I guess the big thing that stands in the way of that, it's easy enough for me to say that as a solo coder. I can develop that practice on my own. But then you find yourself on a team and it feels like you are developing that rhythmic practice of refactoring, but then when you have to deal with the code that was written by somebody else, it seems like they don't quite have that rhythm down yet. And addressing that is trickier and something that I don't have as good a handle on.

So, for those teams or those team members where other people may not have that rhythm down, do you feel like it's because they don't know, they never thought about it, they just don't agree with that rhythm? Where do you think that comes from?

A combination of the above. For some people it's just experience. But the thing about programming is that it involves humans. And humans have such a wide degree of variance in what we perceive as difficult and what we perceive as easy, and what we'll put up with. We talked about that pain tolerance. Some people have higher pain tolerance for tools. Some people have higher pain tolerance for code. Or just different things are painful to them than are painful to me. I find it... I hate long lines in code. I cannot mentally comprehend long lines. Like, I don't care about 80 columns because everything needs to fit on a circa 1970 terminal. I care about 80 columns because honestly, past 60 columns usually I can't keep track of what's being said on that line anymore. But there are other people whose brains are apparently wired differently.

So yeah, that's one of the toughies, is sometimes it's just because they're not seeing the pain. And it's hard because sometimes it's a situation where it's clear that they're not being honest with themselves about the pain. But other times, it really just isn't painful for them. Maybe they can hold more things in their head at once than you can.

Gives a certain degree of Stockholm syndrome around certain pieces of code, too.

And you know, yeah, exactly. And I think that might be where we get into the attitude in programming where we should be catering to the 1%. Like we should be catering to the so-called smart developers, the ones that can hold all those things in their heads, which honestly aren't always the most organized, et cetera, ones. But I think that's a long-standing attitude. But I think there's a lot to be said for catering to almost the lowest common denominator, catering to the level that everyone can comprehend easily.

And not making the sorts of choices when the code was initially written. Might be part of the reason why it's hairy now, if the code is overly clever and it made sense to somebody or they were showing off or they were able to hold more in their heads and they didn't have to be explicit about certain things. That might be the code that's causing problems for other people now.

Yeah. Although [sighs], so it's easy to say stuff like that and then get into the situation of saying, “Well, obviously the lowest common denominator is short methods, that anybody can understand.” Well, it's not that simple, because for some people it becomes very difficult to understand the big picture when there are lots of little methods. I love lots of little methods. I love the Smalltalk way of doing things. But I think it's completely legitimate that for some people, they're just, that means they're lost in a maze-y of twisty methods all alike. So, I don't know.

Might be about establishing a code culture around your individual developer team too, and figuring out what works for the most people in the team and doing your work accordingly.

Yeah, [absolutely].

One thing I think is interesting about this whole conversation is that we've gone from something semi-technical, writing code so that it's better, easier to make changes to, and we've gone completely over to the people issues. And...

Oh, all software problems are human problems.

It's so true. And we're totally seeing that here, right? Where it's, we're trying to make the code easier to reason about, which is a person problem. And then we're also talking about, “How do we get our team involved in refactoring?” which is also a people problem.

You know, it's funny we talk about software problems and people problems and then we say, “Oh, all software problems are actually people problems.” It occurred to me recently that that's a tautology.

Well, I mean software is entirely a human creation. There is no software without humans. Until we get to the point where AI is writing the software itself, that statement will be a tautology.

I'll enter the Matrix with you.

If I could just keep track of which pill is which.

Yeah [chuckles].

Just call me Neo.

But those human characteristics, I think we like to discount them. We like to think that we are perfectly rational and logical and just the pinnacle of intellectual beings when we're projecting what it means to be a software developer. And it's really easy to ignore those people skills. It's really easy to ignore the team dynamics and live in this removed from reality perfect world where everything is rational. But yeah, the real world is messy. So, should we talk about some specific refactoring strategies maybe?

Yeah. What's your favorite, Coraline?

I like the Extract Method a lot, because especially if you're doing a Rails application where you have a tendency toward fat controllers, I like to extract into service objects and just take that... I guess that goes beyond extract method actually, but I like extracting the service objects quite a bit to skinny up those controllers and make classes that are verbs and classes that represent processes and just clarify the intention of really [inaudible] code. It's a very satisfying sort of thing for me to do. A lot of developers are afraid of new files, but I love making new files.

That's another area where I feel like our tools are raising the resistance level more than is really warranted. I don't understand why at this point I have to explicitly create a new file in order to put a class in it. I realize the answer is, because I haven't automated it in Emacs yet, so it's my own stupid fault. [Laughter]

But you know it's amazing to me that that isn't standard part of the editor tools that everybody uses at this point, this idea of, I don't say, “Okay, create a new file. Now call it this. Now save it. Now, class Foo, blah, blah, blah, blah, blah.” It should just be, “I want to create a new class in this namespace named Foo. You figure it out.”

I think different people have different. Have different approaches to the file layouts though, Avdi. I am a very visual thinker. So, one of the reasons I don't use vim very often is that I don't like NERD tree. But I need to see that hierarchy of files. I need to see where my code fits into that overall structure. So, I'm fine with the manual step of creating a file because that makes me think about, where does this really belong?

That makes sense to me, but it seems like, I'm usually thinking of it as like, where does this class belong in the, I guess module hierarchy? And I feel like mostly in Ruby land we tend to just directly map module hierarchies to file hierarchies. These days we just replace camel case with underscores. I could be wrong.

Yeah, exactly. Those two things are very related for me when I'm doing that.

Yeah. I just, I feel like I'm reiterating, you know?

I can see the point there, but I don't find it to be a source of friction for me. I think it comes down to a matter of mapping identity and that identity is reflected in the file system layout as well as the module hierarchy.

Mmhmm, yeah.

Yeah, I'm kind of in the same place you are, Coraline, on that one.

Chuck, what's your favorite refactoring technique?

I have to say that the one that I probably use the most is Extract Method or Extract Class. And I really like Extract Class kind of for the same reasons you articulated with Extract Method. It's just fun for me to look at things and go, “Okay, what in here is related to each other but not related to the core principle or the core responsibility for this class?” And so, I'm applying Single Responsibility Principle and using it to Extract Class. And I really, really like tiny classes. I found that just having something that does just one thing and is really simple about the way that it approaches it as much as possible, and you can't always do that, I really like that. Now, I don't always go all the way to value objects. But for the most part, I am willing to extract a class just to get a couple of methods out of the way so that I can see what the main responsibility is of another class.

I get so annoyed by Rails controllers because they just tend to...

Accumulate so much business logic. And as soon as I see lots of private methods being added to the controller I'm like, “No, no, no, no, no, stop.” And it's usually… we're building APIs increasingly these days. And I think we're stuck on REST even though our APIs are calling for something different now. So, we end up doing a lot of logic, a lot of transformation of data, and a lot of process work in controllers, because we don't, so we simply don't know where else to put it. I'm hoping to talk about what comes after REST in some talks in 2016 because I have some [definite] ideas there. But I think that what we see in Rails controllers is a result of a shift in our thinking that our tools have not yet caught up with.

Ooh, can you give us a little hint of that? That sounds interesting.

It's actually hearkening back to RPC, the days of remote procedure calls. I think that REST works really well in a CRUD application. But increasingly our applications are going way beyond CRUD. And we had the engineer from Netflix on the show a few months back talking about how they moved to orchestration layers for their API and left REST behind. And I think that's a really interesting thing that more people need to be made aware of as an opportunity, as a possibility. And that sort of change in your thinking about what it is that the web server is doing for you, what it is that it's doing in response to your request, can end up having a really cleansing effect on the code that you're writing as well. Hopefully someone will invite me to speak about that topic.

Is that a hint?

Maybe.

I want to talk about one of my favorite refactorings.

Do it.

I like Introduce Explaining Variable a lot.

Ooh, that's a good one.

And I feel like it might be a little underused. That's where you take some expression in your code, like if you have a deeply nested expression where you've got some dots after a piece of data, and then maybe all that is inside parentheses being fed into another method call and you're doing something with the output of that method call, and it's all very neat and tight and elegant and functional. It's where you take some of those expressions and you pull them out as intermediate values that are assigned to named variables. And the variables explain the data. They explain the intermediate step that that data represents at that point, or that object represents at that point. And you can turn some gnarly nested logic into a series of intermediate steps that way. And I really like it because often it does a better job of explaining what a method is up to.

And I also like it because once you do that, it is a mechanical translation to turn that explaining variable into an extracted method if you decide it warrants it. Like, if you realize...

Exactly.

If you realize that you also have that same step in another method and you want to share it. And the third reason I like it is that a lot of times I find that once I extract several explaining variables out, the algorithm that I'm left with after the extractions suddenly looks exactly the same as an algorithm that I find in several other methods. And then the algorithm that I extracted those variances out of, like I extracted out the things that were different between those methods, the inputs that were different, now I have some core logic which is actually the same across all those methods. And there's an opportunity for extracting some kind of shared method or object. So, it's very clarifying in that way, showing, “These are the things that are different and then this is the thing that is the same.”

I'm kind of torn about explanatory variables except in the case that you just described where it's a step in the process of extracting them into small methods. I think that I don't like the look of what I call boxcars where you have just this long method chain transforming an array of hashes or something along those lines, and doing lots of maps and reduces. I think those things are really obtuse. So, I would rather see explaining variables in as situation like that. But I'm most satisfied when I can extract those into individual methods.

So, I actually didn't know that's what that was called. But I love that one. [Chuckles] That's one of my favorite ones.

I'm so glad that I know that it has a name now. But yeah, and that's exactly how I've been using it as well, as the step to almost clarify my thinking and my process and the things that I care about on my way to extracting out an entire method or a class. So, thank you, Avdi. Thank you for that.

You're welcome. And I'll also speak up for leaving it be and not necessarily extracting the method. I'm not necessarily, at least these days, I'm not necessarily someone who will extract a method just for the sake of it, particularly if I don't see it being the same logic being needed in multiple places. I'm okay with leaving it in place and just having it in that extracted, that explaining variable where it's very easy to extract it out when I need to. But in place, it's sitting there explaining the series of steps quite nicely.

And I think it's pretty situational. So, I can definitely see that. Avdi, have you ever had a refactoring go terribly, terribly wrong?

[Chuckles] Yes. I'm sure I have.

I don't know if I have like a specific example that springs to mind. I guess we all have.

I had trouble at a previous job where I was new on a team and I was asked to add a feature to an export class. And the class was so, it had so many responsibilities. It was like a parent. And it just made no sense whatsoever. And I was like, “I cannot in good conscience add this feature and just contribute to this chaos.” So, I started refactoring the class. And it was like unraveling a sweater. Once I started pulling the thread, just everything started coming apart. And what should have been like a half-sprint task spilled over into two sprints. And I just, I just could not find the seams. I could not figure out where to stop the refactoring process, because it was so deeply entangled with the code that surrounded it that there was just no clean endpoint. And I ended up giving up and giving the feature to someone else because I'm like, “I can't write this.”

Yeah, that's like, I guess that's what I was thinking of when I was thinking about projects in my past where a refactoring really turned out to be this endless rewrite slog. I had one that, I don't know, it feels like it took a summer or something. It was a couple of us working on it. And it probably didn't take quite that long, but it was just beastly. And it's definitely one of those experiences that I look back and think, “I'm not sure if I'm proud of that.” I wonder if coming at it now I would have handled it differently.

I feel like that'd be a great show topic one day of just doing horror stories of code, coding horror stories.

Oh, that's amazing. Yes, we should probably do that.

We should totally do that.

We totally...

Our Halloween episode.

It came from beyond the editor.

Yeah. [Laughs] I love that.

Saron, what's your favorite refactoring.

Well, I think yours, because now I know it's a thing. [Chuckles]

I'm really excited about that. But it was one of those things that I always, when I did it I would feel kind of bad because I wasn't using it besides, I wasn't using it outside of just being more clear to myself. It was like a selfish thing to do, it felt like. And I didn't feel like it was... I don't know, I guess that goes back to our point of coding problems are people problems, right? Because if it makes me feel better and if it gives me clarity, then I guess it is also good for my code. But I don't know. It felt not right, but I also found it very helpful. So, I ended up doing it anyway. So, now I feel very validated.

Ruby in particular, the language is supposed to make us happy. So, code that is more clear makes us happier than code that's obfuscated. So, any performance trade-offs, I don't know if there are performance... there must be performance trade-offs in creating a lot of local variables. But if you're getting clarity from it that makes the whole thing worthwhile.

Yup, very true.

There's this paper that I read recently called 'Artifacts in Software Design' by Reinhard Keil-Slawik. And it basically talks about the fact that in learning theory, people don't manipulate values and ideas totally in their minds. It's really a back and forth process between what goes on in people's minds and the artifacts that they manipulate while they're thinking about it. And a lot of our mental progress over the ages has come from introducing artifacts with higher and higher levels of abstraction. So, like counting on fingers and then tokens, a shell or a rock or something that can stand in for numbers, and then abacuses is a higher-level artifact that we can manipulate.

And I think that we look at the code as an end product. This is a big thing, a big part of what this paper is about, is that we often look at the code as an end product. And you know, sometimes we do regard it as something that exists as much to communicate to other programmers as it does to communicate to the machine which is good. It's a good step. But we still fail to look at it as a thinking tool. We're still thinking, okay this is a thing that I'm building. This is a thing that I'm making, whereas really the way humans manipulate ideas, artifacts are inextricably linked to that process. And so, the actual, that system manipulation, that process of manipulation is part of your thinking through the problem. And so, I think we need to elevate that. We need to respect that it's okay to do manipulations on your code that are just part of your manipulation of that artifact, the equivalent of doodling on the ground with a stick that helps you think through something.

And the code you leave behind can be a physical artifact of that thought process. Is that what you're saying?

Exactly, yeah, yeah. But the process itself is essential. It's not like the code that you're left with gives you everything, can transfer the entire process to somebody else's mind. The process that you went through of manipulating it and moving things around and changing things that you went through was an essential part of thinking through that problem.

That reminds me of some tweets that Sarah Mei has been making lately where she was questioning the concept or core metaphor of software development as a manufacturing process where the thing that matters is the end product and it has a point in time at which it's complete, which is really counter to the way we actually do software development. So, I wonder if changing our thinking about the end result of programming being idealized code as opposed to a work in progress that has these artifacts, that has visible processes, visible side effects from thought processes, visible documentation, those sorts of things... I wonder if we change our thinking about what it is that we're actually doing, if we'd be more comfortable with that sort of thing.

Yeah, you know, I move this, I extract this variable out, and that exerts pressure. Seeing that exerts pressure back on my mind and I see that and I think, “Oh yes, that feels better,” or, “That gives me a new idea.” It's this back and forth that doesn't exist if we don't actually do the moving things around and the different organizations.

If that is not already a talk, Coraline, I think that's another great talk idea that you have.

Well, I can't take credit for that, but Sarah Mei.

Yeah, I should say that Sarah Mei's tweets are my favoring software engineering blog right now.

Yes, definitely. [Laughter]

Nice.

I'd like to know more about what she's reacting to. I'd like some more context around her ideas. So, she needs to be encouraged to do some more long form blogging.

Sara, you are officially encouraged.

Yes.

Coraline, do you have any more questions or directions that you've been thinking about?

My focus around refactoring, I'm just looking at my notes from one of my talks and I talk a lot about the relationship between refactoring and testing and some of the different testing techniques that are appropriate in a refactoring context that maybe you would not be comfortable using in different contexts. Like in particular, generative testing, which is like really a hot button issue for a lot of people.

Okay. Do you want me to give you a good prompt for that?

Sure.

So Coraline, you've been talking about some of the testing techniques that you use in combination with refactoring in some of your talks. Do you want to talk about that a little bit?

Sure. There are certain types of tests that I'm comfortable using in a refactoring situation that I would not want to leave in the codebase long term. One of those testing techniques is doing generative testing. I think especially if you're looking at a really hairy method and you're not sure exactly what it's doing, you're not sure what the ideal values for it are and how it handles the edge cases, then doing generative testing where you're just programmatically generating a number of inputs and testing what the output of it looks like can be really helpful. It can tell you more than just a cursory look at the code can do in terms of what it's actually doing on the inside and what the end result of the process is.

And especially looking at, I think in generative testing the most important values are zero and infinity, and nil. So, try the edge cases. See exactly what the boundaries of what the function is doing are, and just really exercise the hell out of it using everything you can think of to throw at it. And then as you refactor you might want to thinking about, do I want the method to behave in exactly the same way even with these edge cases, even with something that throws an exception? Or do I want it to continue throwing an exception? Do I want it to handle it differently? But knowing where those edge cases are I think can be really, really handle.

One of the things I've done as well, and this is a weird testing technique, from changing the implementation of an algorithm, I'll actually capture the old algorithm inside of a lambda in my test suite. I will throw values at the old algorithm by invoking the lambda and capturing the result, and then call the new function and compare the results to make sure that I haven't inadvertently changed something about the implementation that's changing the output characteristics of a method. And again, that's not something I would want to leave in place for very long but it's just a way of double-checking that you haven't changed something unexpectedly.

That's cool. That's like the thing at a much coarser scale, larger scale, where you have two separate systems and you apply the same requests to both of them and log the responses and make sure that they match.

Yeah, it can be a handy technique. It's pretty ugly to implement but when you get the pattern down it gets easier.

I'm curious. In the context of Ruby code, are there any tools you're using to help with the generative testing?

No. I'm writing everything myself. I know there are some gems out there that are starting to get into delivering some functionality that people from other languages have come to expect. But up to now, I've written everything myself and it's all been throw-away.

Can you describe just at a broad level what it looks like to handwrite a generative test?

I want to make sure that my understanding of generative testing is the same as everyone else's. When I use the term, I'm thinking about generating a prescribed and randomized set of inputs for a method. So, I don't know if that fits everyone's description of generative testing. And I'm comfortable with code that is writing tests as a result of those ranges of inputs. So, it's really easy in RSpec. Most people don't realize this but you can actually create it blocks inside of an iterator and substitute, use string interpolation to drop values into the text of the it block. So, I like to think about, “What are expected inputs for a function?” and I'll make an array of those. And I'll do, “What are the unexpected inputs?” like nil or an empty array or an empty hash, those sorts of things. Or an integer if I know it's expecting a string. I'll add those to the array as well. And then I'll just iterate through the array and throw those values into the it block and throw those values into what the it block is testing and see what happens.

And do you use some broad assertions in something like that, like just the output should be a string or something like that?

Yeah, and I also check to make sure that it's not throwing an exception. If it does throw an exception I want to know about it. In some cases, I expect it to throw an exception and I want to know where the algorithm as it stands today fails, because as I'm rewriting it, something else might be expecting that exception to be raised. So, as I'm rewriting it I need to know that this exception is raised under this circumstance so I can preserve that behavior.

I love the pragmatism of your approach because I mean... so generating tests using iteration is something I've certainly done in RSpec. But it's also something that's kind of frowned on in the RSpec community as something that's a little confusing to read after the fact. But you're using it in a context where you're probably going to be throwing it away anyway. And so, it's a very pragmatic use of that capability without necessarily encoding it into your forever suite.

Yeah, exactly. And that's something you have to be comfortable with doing, is writing a throw-away test. Especially when you're trying to wrap your head around something, test can be a great way to capture your expectations or your assumptions about what code is doing. And then your tests should be able to support you in validating those assumptions.

Absolutely.

Another cool tool that I like a lot is a gem that Katrina Owen wrote called Approvals. It's for golden master testing. And what you do with Approvals is you capture the golden master of an object that's serialized, so just a [marshal] dump of an object. And then when you change your code and you run your test, it will actually compare the [marshal] dump of your object before and after and ask if your changes were intentional or if they're an unintentional side effect of some other change that you made. And if you like the change, you can say, “Yes, approve this as the new gold master,” otherwise you have an indication that you have changed that object signature in some way that maybe was unexpected. So, that comes in really handy, too.

I love gold master testing. I have sort of a not yet extracted gem that does the same thing. And I think it's a terrific technique.

Yeah, it's really cool. And it can be an easy way too, to do view testing.

Yeah, yeah.

Instead of writing all your Capybaras to expect this, you can just say, “Hey, when I perform this series of inputs I expect that the output HTML is going to be exactly the same as it was before.”

Right, right, yeah. And just the ability to say, this is one of the very first testing techniques I learned when I was getting started as a software engineer. And this was in a context that absolutely did not have unit tests the way we think of them. But one of the first things that one of the senior developers showed me was this idea of, you write a script that will cause the system to output a whole lot of data, and you just save that data, that output, as the golden master and periodically you run the script and you diff the results. And you go through it with a fine-toothed comb and you decide whether the new version makes sense. And it's tedious in some ways, but in some ways it's very efficient.

We actually have an application at my current job at healthfinch that we call flight simulator, which takes a common set of data and you basically say, “I want one VM set up using the master branch and I want another VM set up using my local branch.” You run the data through it and it gives you the diff of what happens when you process that data, which is like a really great tool to have.

Yeah. And it also, doing that makes me think a lot about, I wish more of my objects had a nice textual representation that was easily diffed. It makes me think about that a lot more.

'To string' is your friend.

Yes.

[Laughs] I override that on a lot of objects.

Or inspect, one of those two.

Yeah. I get really frustrated when I'm using an API and I think, like just yesterday I was using Net HTTP and I'm like, I want to know what my request actually looks like. I want to know what the instance variables on it are. And when you do a 'request dot inspect' all it gives you is the object id and I'm like, so frustrated that I have to go and look up the API. Gem authors, make your objects inspectable.

Yeah, yeah. It's so nice. It's one of those little niceties that you don't have to do it but it sure is friendly when you do. Also, probably a really easy pull request for somebody who wants to start contributing to a project, or relatively easy, is just going in and making some pretty output for API objects 'to s' or inspect methods.

That's a great idea. So, one of the other things I wanted to bring up in the context of refactoring is measurement. Because we have a satisfaction at the end of a refactoring that we've made the code better than it was when we got here. But sometimes we're refactoring for performance or to make buggy code less buggy. So, I think it's really important to establish a baseline at the beginning of your refactoring efforts, whether you're doing benchmarking or you're looking at churn and cross-referencing churn with the number of times that a particular piece of code is [churning] errors in your bug-tracking system. And keeping those sorts of metrics in mind can confirm or deny your suspicion that you made things better and can also be a really valuable tool for in the future convincing management that refactoring is worthwhile.

So, you can do a tool like Code Climate although I think the letter grades are too abstract. But you can use code quality tools and benchmarking tools to record what the state was at the beginning of the world before you touched the code and then what it looks like when you're done.

I have nothing to add to that. Have you applied that and gotten good results out of it?

Yeah. Actually, at an old job we built a system. It was sort of a Code Climate replacement where it went into a lot more details in terms of the code metrics. And it tracks changes to code quality with every merged branch. So, you could definitely see if the complexity was churning up or down, if the coupling was churning up or down. We didn't do performance characteristics. But that would have been something that would be really interesting to put in place, too.

And it's not hard to generate those sorts of metrics. If you wanted to, you could just have a static output that you check into a Git repo. It's a little hard to extract change over time from that kind of data. But it's not that hard to capture it and use that as part of your process as a [gut] check.

I wish we had a metric where it would measure programmer's blood pressure as they looked at a particular file.

Because I think that would be really valuable data.

Fitbit for programmers.

Yeah. Oh my god. Could you correlate Fitbit stats to what file you're looking at?

I bet you could with a tool that would look at what's your active window.

Yeah. And just like...

Just do a [inaudible].

Record heart rate or something.

Yeah.

See if the fight or flight response was kicking in.

Someone out there, a listener in Silicon Valley, just got an idea for a startup. [Laughter]

And it's all our fault.

CodeStress.org.

Quantified Coder.

Yup. So, one other thing that I'm sure people are wondering is, are there good places where people can find examples of these kinds of refactorings? We talked about our favorite refactorings. But is there a definitive list that's easily accessible?

I guess you're setting me up because I'm the one with the library.

I was going to say I'm really fond of 'Refactoring: Ruby Edition'.

Yeah. I actually have a link here for Martin Fowler. It's Refactoring.com/catalog. And it lists all the refactorings from his refactoring book and the Ruby edition.

Cool.

Yeah, definitely...

'Refactoring: Ruby Edition' was really an eye-opening book for me. It codified a lot of the refactoring instincts that I already had in place. And learning, like Saron did earlier like, “Oh, there's a name for that. That's really cool. That's an established pattern. That's kind of reassuring.”

Yeah. So, definitely that. Either of those books. I've read through the 'Refactoring: Ruby Edition'. Totally worth reading all the way through, by the way. Yeah, you can poke around. If nothing else, read the intro sections where it explains the smells.

Mmhmm.

That you should be aware of that you can then apply some of these refactorings to. But totally worth going through the whole thing. There are some great talks out there as well that step through the process of refactoring really well. So obviously, Coraline has been speaking about this. There's Katrina Owen's classic talk, the 'Therapeutic Refactoring' talk, and I think she's done more where she just shows refactoring step by step. And they're extraordinarily satisfying talks to watch. Sam Livingston-Gray did a terrific refactoring talk a few years back and I'll hunt that down and make sure it gets into the show notes.

Very nice. Alright, well should we get to picks? That is a yes.

Sure. [Chuckles]

Avdi, do you want to start us with picks?

Sure. I can only think of one thing today. I've been watching a new show on Netflix. Well, it's not a new show but it's new to me, called Longmire. And it's a crime drama and it's a western and it's pretty and it's enjoyable to watch and it generally doesn't suck. So, there you go. Longmire. Look it up on Netflix. That's it for me.

Alright. I'll go ahead and throw a couple of picks out there. The first one is, I might have picked this last week, I don't remember, but I've been playing a couple of games that are very similar to each other. One is called Clash of Clans and the other one is called Star Wars: Commander. So, one is Star Wars themed and the other one's more fantasy themed, I guess. And so, you build your camp or base and then you can actually go attack other people's camps or bases. And you can collect gold or elixir I think is what the other resource is in Clash of Clans. And you can join clans. I keep getting kicked out of clans because I'm too new. But anyway, that's a lot of fun.

Commander, and turned out that he was in that squad. So, just random happenstance. That's the way it worked out.

I do have another pick and this one's more of a general... it's not a resource online. And that is cleaning your office. So, I had a pretty big mess here in my office and I can see desktop again. I still have a couple of piles of things to go through, but it is much more approachable. So, I'm playing with that as well and trying to get everything else done. So, I've got a pile of cords and a pile of papers, is basically where I'm at, at this point. And then I've got a few things I want to hang up around the office.

So anyway, those are my picks. Coraline, what are your picks?

Sure. I got to go to RubyConf in San Antonio this year and there were a couple of talks. It was full of great talks. There were so many really interesting topics that people covered. But a couple of them really stood out to me that I wanted to highlight. The first is a talk by Hsing-Hui Hsu. Hopefully I haven't messed up her name. She's SoManyHs on Twitter. Her talk was 'Time Flies like an Arrow; Fruit Flies like a Banana: Parsers for Great Good'. So, she talks about parsers. She first starts out by talking about parsing spoken language grammatically and then compares that to the way that programming languages parse. And her theory is that by exploring the way our brains construct grammars to parse sentences, we can better understand how parsers work with code. So, it was technical, it was fun, it was grammatical. I'm an English nerd, so that was kind of cool. And just a great talk overall.

The second talk I wanted to highlight is a talk by Betsy Haible entitled 'What Regex Teaches Us About DSL Design'. DSLs, they have a bad rap in Ruby. I think that DSLs can be very handy and that we use DSLs more often than we realize. And Betsy's example was regex, which is a DSL that's been around for absolutely forever. And it's ugly and it's persistent as a cockroach but it has an internal beauty to it as well. So, she talked about when a DSL needs to go for beauty over usability and how that affects how you actually use them. She gets into breaking down the design of regexes and extracts them, extracts the regex DSL into principles that you can apply when you're writing your own DSLs. And how that applies to API design, making boundary decisions. So, 'What Regex Teaches Us About DSL Design' is a second pick. And Confreaks has videos of both these talks that will be available in the show notes.

Awesome.

So, I have two picks. They're both conference open, what are they called, CFPs. So, one is RailsConf. So, RailsConf talks are due January 15. It's happening in Kansas City which should be exciting. I was looking up Kansas City and apparently it's like a booming tech hub now and there's lot of cool things going down. So, I'm excited to go. And I hope you all submit talks.

And the other one is the Velocity Conference which is happening in Santa Clara. Santa Clara, how you do say that?

Santa Clara.

Santa Clara, okay, California. And that is due on January 11th 2016. But I think that one is more like DevOps focused. So, if you are interested in submitting talks or thinking about it, I'm more than happy to give you feedback on your talks. I've done that before for people. So, feel free to reach out to me and I hope you get your awesome ideas and your thoughts out there to share with the dev community.

Perfect.

Alright. Well, thanks for coming and hanging out. We'll go ahead and wrap the show up and we'll talk to you all next week.