The Rogues talk about practical metaprogramming with Steven Harms.

Special Guest: Steven Harms.

 

So folks, this is Episode 80 -- Practical Marijuana with Steven Harms.

Hey everybody and welcome to Episode 80 of the Ruby Rogues Podcast! This week on our panel, we have James Edward Gray.

Good morning everybody.

We also have Josh Susser.

Hey from overcast San Francisco!

Avdi Grimm.

Hello from sunny but cold, Pennsylvania!

I'm Charles Max Wood from devchat.tv and we have a special guest – Steven harms.

Hi there from… it’s actually sunny and warm, and beach house and bikinis in my district in San Francisco.

You are in the high rank district right?

[chuckles] No, no I'm just two blocks away.

So I have to ask – I know people are going to wonder -- are you related in any way to Angela Harms?

No. I'm in fact no way related to any Harmses that are tech famous -- and this Steven Harms is certainly not tech famous -- so there are no famous tech Harmses that I am aware of.

I believe Angel Harms was one of our most popular episodes ever Steve, so it’s kind of suck when you got to live up to. No pressure.

Oh. Well, my apologies to Angela and my apologies to Rogues for not having listened to every single podcasts to have known what a great guest Angela was.

But this guy is like really super silliest. It’s great.

Josh, stop using those fancy Latin words.

[chuckles] OK. Sorry.

Some of us are just super silly.

But David is not on the show today.

I know. He is not feeling well, so we'll think positive thoughts towards Dave. So we are going to be talking about “Practical Metaprogramming” -- because apparently, we are doing it wrong.

But before we do that, can we talk about Parley?

OK.

OK. So, the best thread on Ruby Rogues Parley this week is titled “Functional Programming for the Object-Oriented Programmer”. But if I were to title it, I would title it “Josh Susser Versus the World”. No, actually it’s really interesting thread. It’s somebody suggested that we read Brian Marick‘s book by that title and do it as an episode. And Josh came back with, “Functional programing hasn’t been that big for me.” And there was a good back and forth, and Josh kind of lay down a whole bunch of reasons why he didn’t think it was valuable and other people are laying down reasons why they do think it is valuable. And it’s actually a very interesting conversation. So, that's what we are up to on Parley this week.

James do you wanna mentioned what Parley is for someone who might just be tuning in first time?

Yeah good point. It is Ruby Rogues mailing list where you can chat with all of us Rogues and anybody we've ever had as a guest on the show. So, all kinds of great Ruby minds sharing great ideas.

Nice. Yeah we love Parley and we love all Parleyists and yeah—

Is that a word?

What?

Is that a word?

Sure it is. I just made it up.

Oh yeah. Good point.

So we also have all of our guests on there -- so if people have been a guest on the show, they are on that list. So we have really just a terrific list of people that discuss stuff. Anyway, now let’s get to mactical--

[laughs] “Mactical”? What?

That is not a word. [laughs] But I did not make it up. Anyway “Practical Metaprogramming”. So we talked about metaprogramming--

Episode 12. That was a long time ago.

Right. And so, have things changed since then? Or did we get stuff wrong? Or is there more to talk about there? Or---

Actually that's hilarious because the reason I asked Steve to come on the show is he gave a talk at Ruby Conf -- 2011 I think -- called “Practical Metaprogramming” where he starts off talking about Ruby Rogues Episode 12 and pretty much how bad we butchered the definition of metaprogramming. [chuckles] Here's a great line in his thing; he goes through all the things we’ve talked about and says, “And so by this, they’ve basically spent 10 minutes to *not* define what metaprogramming is.”

So Steve, here is your chance; give us the definition.

No hold on, I want a butchered definition from Josh first! [laughter]

I’d say I'm looking at the documentation here and about 10 minutes in, it turns out that the answer to “what metaprogramming wasn’t” was it was a Lisp or it was --- code, closures and state or it was ASTs versus Byte Code, or it was code generation versus runtime behavior changes, which basically at that point, that kind of implied in my head and I was like, “These guys aren’t telling us what it is.” [James and Chuck laughs] That was kind of… I was hoping to basically just leech material out of your podcast and write my slides from it. But regrettably, at least after the first 10 minutes, I was not helped with my plagiarism in that regard.

So Steven I have to agree with you that that part of the conversation was pretty messed up. I think from what I remember of it, I think we were all trying to make fine points about distinguishing what metaprogramming was compared to what a lot of people thought it was. And that was a big mistake because we never really started with the fundamentals.

And I think now it’s kind of nice -- like James just said -- I kind of gave this talk and I was doing a lot of thinking about it around Ruby Conf 2011 -- which was September -- so a little over a year ago. But, I had basically a year to kind of let those ideas marinate. And as I exhaled the last breath on the talk, I kind of felt like “I wrote this talk wrong entirely.” And I think this will be something that will kind of hit home with kind of what the gestalt is at the moment which is I really should have focused on message passing and all of that just sort of sit there and let dance in joy. But, if you really think about where programming actually starts -- in any regard – be that metaprogramming or just regular programming -- our focus should be on the messages that we need to send and it sort of seems that Josh and Sandi Metz and as Steven Ragnarök‘s talk at GoGaRuCo, is it seems like this notion about focusing on the messages that we send is now kind of becoming the starting point of where we think about programming. And I think that is kind of an interesting way to start thinking about how to program.

You know, there was a really good Ruby Tapas episode -- I think it was some time last week --where Avdi showed kind of that very concept. He started off by doing a symbol in metaprogramming trick but he very intentionally bound it tightly to the methods and then went forward and showed how that caused him problems. And then, he basically went back and rethought it in terms of messages as opposed to methods -- and showed how that basically fixed it.

That was a really cool episode. I can’t remember what it was called though.

Like Methods and Messages or something like that?

Yeah. That was the one that you had a cute trick in there for binding to a specific method on an object.

Yeah. Pulling a method reference.

So that you could call the method rather than sending a message? And Avdi, I actually used that technique as an anti-pattern example in the talk I gave at Ruby Conf.

So, can I stop us for a minute because well for one, he took three months to call us out on what we did wrong and it took us a year to respond. But we still have to find what metaprogramming is.

In our defense, it takes Steve about 10 minutes to talk about how bad we did talking about metaprogramming. [laughter] Just saying.

So it took us 10 minutes to not define metaprogramming and it took him 10 minutes to define that we haven’t defined metaprogramming?

Right. JOSH: And already about 10 minutes into the call now can we--

How about this. I will throw a definition out there -- and I will prepare myself to be savaged for this horrible definition -- but I think that it’s writing code that redirects a sent message at runtime. So I'm not sure I've got all the right concepts there but that's my working kind of definition.

OK so let’s break that down; writing code that redirects a sent message at runtime. What do you mean by redirects though? Let’s talk about that.

So, the redirection thing, if you watch the presentation I gave at RubyConf -- I'll kind of stealing my own thunder here -- which is that at the end of the day, all programming particularly in Ruby is sending messages. And the question is how do you catch those messages? And I've had some success explaining this to JavaScript programmers lately by basically just talking about how events bubble up a line. In Ruby, they bubble up in the ancestor chain. And so, the question is where you wanna put that catcher. And I will give great credit to James Gray in episode 12--

I know that guy.

Indeed. He puts in that -- what I affectionately call as Gary’s mandate -- is he says that the Ruby method call system is really important and you have to get your head around it at some point about how it does these look-ups. The method call system is a straight line with stops along the way and once you get the hang of it, you realize you can put a module at any point on that line. So what my -- is about redirection is that, you send the message out on to effectively a bus and you just cross your fingers and hope somebody catches it. Now the simple way of defining someone who will catch it is to use the def statement. So as long as you have someone on the bus -- and by that I mean that kind of computer science notion of bus – as long as you have someone on the bus that can catch that message, you will not receive… I guess that no method error. And you can proceed to handle it. And that gets us to the point of sort of saying, “Well, isn’t that just programming?” I

think that's where the kind of dictum where metaprogramming is just programming kind of comes from is the realization that the art of what we effectively do is designing captures for messages that we are going to throw under that bus.

OK. So I think there is a lot that. So that's like what is the messages thing, but I think the other half of what you say is at runtime. I think we can’t finish this definition till we talk about what do you mean by runtime.

Right. And I obviously in an interpreted language, that's a little bit funny -- but maybe this is sort of my own personal, limited, cognitive faculty -- but I conceive of that when I'm typing things into my editor and I proceed to run Ruby on that stuff, I perceive that kind of being like back in the old days of compiling and linking. So there is something that I typed in that's being run and then being effectively being auto evaled. I think that's kind of nice. But I also recognized that there is something that happens as a sort of secondary effect -- where I can take a message, break it apart, reinterpret it and then redirect it -- which is kind of the basic trick of a method_missing call. So I conceive that kind of activity -- say you are catching something at method_missing or updating a symbol or dynamically loading in a module -- I conceive that as kind of being at a separate phase at the runtime than in the sort of classical sense of “OK I type this on to my editor and I proceeded to run the Ruby binary against the contents of that buffer.”

So I really like where you are going with this -- and I think it’s important. So just to reiterate, basically what you are saying is we have to think of it in terms of, “I'm sending this message. I'm putting this thing on the line. It’s going to start cruising down the line, but I don’t really know exactly what it’s running to at the other end.” A lot of times it tends to run in to def with that same name -but maybe it runs into something else. But here’s my question; given your definition, is something like a simple Decorator where just pass a class inside of another class and then do a little something extra when I intercept that message, does that qualify metaprogramming? It seems like your definition.

I suppose it does. I suppose if I'm thinking about the decorator pattern, I feel a little bit like that's more in the sense of that it was composed, you are choosing to compose the class by bringing this decorator in. But [inaudible] the way that that's usually in Ruby or [inaudible] in Ruby would be to dynamically pull in that module to handle declaration. You don’t have to go to a class. So [inaudible] in both ways and that's kind of what is beautiful about Ruby is that you can see things from multiple facets. But, I would argue that the heart of metaprogramming – in at least the definition that I take in the talk – is well, metaprogramming is just programming – be it invoking a decorator pattern or just using sort of a mix in -- that's all very standard. But I do recognize that there is a style of programming that people sort of classically think it was metaprogramming -which usually hinges on some sort of message not being defined and handling something either via method_missing or a callback. So for the purposes of didactic purposes, I frequently understand that the person with the question is asking something focused on or something around method_missing or the dynamic construction of messages or the dynamic construction of modules.

So, I may have [inaudible] defining it.

It’s a hard thing to define. I mean it’s kind of--

I think there are like two categories of metaprogramming (this may be just rephrasing what you have just been taking around) and that's there is the stuff that we commonly do in Ruby, and simply can just be something like a feature toggle kind of like a thing around a method definition in a class or even like based on environment like if Rails equals production, then define the method one way, else define another or even just insert the method. So that sort of programmatically coupling your program together is one level of metaprogramming and then there's the level of metaprogramming that you see in Lisp-like languages that are iconic where you can manipulate the program structures really simply as if they are just the rest of your data. And that kind of metaprogramming we don’t usually do in Ruby. So it’s not the bread and butter of Ruby metaprogramming.

I'm kind of coming, you know one year later, I've with a lot of effort on your part as well as with Sandi Metz’s book is I think in some cases, I've seen people abuse the programming to cover weaknesses and object orientation, is you frequently kind of reach for that as a crouch and say, “Oh well I'm going for this.” And it turns out that maybe something along the lines of more sub classes or having classes that adhere more closely to the single responsibility principle might not have necessary [inaudible] some sort of method_missing -- metaprogramming solution.

Yeah. I think that's very good example what you said right there. For example, the example we had earlier with the decorator, if you just really wanna add a little something to one method it’s like you say you could define module and then you could extend that particular module with that particular object with that module or you can just build a class that takes new and saves the object reference and defines the same method and calls it normally. And you know what; the traditional ways is at least as composable, if not more so. And to give another example like I think we have a gross tendency to reach into a class and alias a methods so we can define a new version of it, so that we can call the old one or whatever whereas if we were a little bit better at designing our classes and we left the hooks you know, things like observers – so you could register yourself to being notified of an event when that happens you know, it’s often far superior in many ways right?

Well Yehuda definitely thought so. He changed a whole bunch of the Rails internals for Rails 3 to have it be able to work that way with hooks rather than having the Rails method change everything.

Yeah that's very true but then, if you read in Sandi Metz’ book, basically what Yehuda did in Rails 3 was to make everything a large inheritance of modules -- and you super everywhere -- which is definitely better than what it was, but if you read Sandi’s book she actually talks about how super is kind of another form of coupling. And you know, while better maybe not best. And that maybe it would be the better if you know, Active Record gave us more hopes then we did have to insert module instead we just define like method that it was expecting or stuff like that. So, kind of interesting.

So here's a little side note -- and maybe we can just spend two minutes talking about it -- Ruby 2.0 has this feature called Module#prepend. Are you guys familiar with that?

Yes.

No.

OK prepend is another way to avoid having the delay list method chain. And so in Ruby previous to 2.0, the only way you can pull in modules is you include them or you can extend them -- but let’s just talk about including them. The other class and you include it and that makes that module an ancestor of the class. If you are in a class and if you say module#prepend, it sticks the modules methods into the class, but it does it below the class’ own method and the inheritance hierarchy. And while that is not very useful if you are writing the class itself, you can use metaprogramming to say some class send prepend with the module name and that essentially what ups the class’ methods with the methods in the module. So you can do the sort of alias method change type behavior where you want more specialized behavior wrapping the behavior of the class. And then those things can cause super to get to the classes methods.

Fascinating.

Yeah. It’s great for being able to build stuff like you have to do in if you are trying to compose behavior and the author of the class didn’t provide you hooks.

I think it’s very interesting particularly with the… I can say when I read a blog past about this Josh. This thing about trying to extend say, my personal pet project, trying to extend a subtype of say, ---let’s say you'd want to inject that behavior as it fell to say one of several categories. And you realize that a lot I had to do internally to make my own project work, I would be just so much cleaner with the ability to pick where in that ancestor chain you inject. And I think that's going to be very powerful thing and I think it’s going to lead to definitely some interesting code coming up -- as people integrate their metaprogramming designs take advantage of module prepend.

I think prepend was Yehuda’s idea. And he's done a lot of these stuff structuring modules in Rails, so I'm hoping that it will simplify stuff in that code base.

And I could tell you as --- that I run the Yehuda Katz stack which is Rails plus ember [inaudible] patterns and the Ember framework is that when you create a new class, you have to explicitly call super. And so, I can definitely see some opinions if that's the case it’s also on Rails 3. I’ll be able to see consistency of that opinion appearing both in Ember as well as in Rails.

So we talked a lot about defining metaprogramming -- which is obviously challenging. But really in your talk, one of the things I loved was you basically just took a bunch of kind of simple principles and tried to use them to show what metaprogramming is kind of by example, sort of. So can you tell us like I think you have four things that you thought were key in understanding it and can you kind of give us and idea what those are?

Sure. I’d say that at the highest level, it is absolutely imperative -- and James I'm not trying to suck up to the show too much -- it is absolutely critical that you do understand the inheritance chain. What Josh is talking just now about Module#prepend, I immediately said, “To imagine what would happen if I call ancestors on a class of an instance. I have a working model of where mixed in modules would appear in that chain -- and so Josh is talking about where you prepend work in that I was like, “Oh I can see that. I’d basically insert something early on the bus vs. later. That's neat.” And it’s absolutely critical really if you want to understand metaprogramming that you eventually evolve that notion of the ancestor’s chain as your conceptual model. So again, I’ve give props to out to ----, his book was the one that inspired me to do this talk. And he goes in a great depth about how to build that mental model of the ancestor chain. So, I’d say as a meta tip to metaprogramming, you really wanna have a worthwhile understanding of that early on. But then the second point I’d say is that, once you understand that, then you want to understand this notion that you redirect messages by defining things. And as I kind of tried to start levelling people up, I kind of tried to start somewhere that was familiar, is that you can basically use def and def is great. Def is a great way of intercept a message and respond to it. What’s kind of interesting is that at RubyConf when I asked people and I say, “How would you redirect a message that’s sent to the ancestor chain?” Form the audience; I got, you know, “Oh, method_missing!” or some sort of alias method chain. But everyone went out of their way to not come up with just using definitely. And I

just want everybody to show proper respect -- def is your bread and butter for handling messages that’s in on that message bus. So you know, represent def. You know, it’s a good.

And def with class eval is one of my favorite ways to add methods to things at runtime.

Right. So the only case where you often find yourself in with metaprogramming where def won’t work is def is in a closure -- so it doesn’t have access to its surrounding environment. So if you are doing something dynamic like saying looping over set variables and defining a method for each one or something, you typically need access to that outside variable -- and that's where define method comes in because the body of the method that gets defined by define method is a block, therefore is a closure, therefore has access to the surrounding--

Right. I think it’s really important to understand that difference. And so with great power comes great cost. (Sorry Steven) and [chuckles] so the define method approach to adding to methods to classes at runtime, that's great and that gives you the closure but because it has that closure there, it has extra cost at runtime. And there is another layer of—

Sure.

So if you don’t need that closure aspect of it, then your program is going to be more efficient if you don't use define method if you just use def.

Oh absolutely. Yeah absolutely. Def should be your first choice. But if you do need that surrounding scope, that's why you see people using define method is they have to get to that variable or whatever on the outside -- which is something that's not easy to do with def. You'd have to define somewhere that would be accessible to def.

So one thing that's interesting about this that I've run into is that, I was writing some integration tests for Sinatra and the only way that I could find to basically mock out or stub out the helpers was to actually do this where you do a class eval and then you go in and effectively redefine the methods. But one thing I realized was that it was really difficult to back out when I was done running the tests. Is there a good kind of way to do a reverse doing class eval plus def or define method?

You can do undefined method.

Great.

Undef method. There's remove method or undef method and they are different and I have to look up--

So if I undef it or remove it then the old method will come back?

It depends. If you put one in front of it, if you said something on the line in front of it and you only remove that one on front -- which is I believe the effective remove method -- it just hits the immediate method.

Yeah. On def method actually it’s almost like it actually puts a marker in that says, “This method doesn’t exist -- period.”

That's right.

It will actually cover over something inherited whereas remove method will remove the one from the current class, but will still reveal. And that will just reveal any ones that are defined by parent classes. Chuck, in testing, one technique that's been used -- I don’t know if this has sort of been usable in your context -- but one technique that I have seen used particularly in other languages is rather than testing the class in question, you inherent from the class in question and you override the stuff that you need to override in the inherited class.

Oh, there's an idea. I like that.

Right. And in the case of Sinatra, that's super easy to do if you use the Sinatra-based style instead of just normal application style, where you defining everything at the top level. The base basically wraps Sinatra inside a name space.

So you just have a… it’s basically a test specific--

Right.

And you can do that up to a point, but if you have tight coupling to class names, sometimes you can’t get away with sub classes.

Right. If a library is in question, use explicit class names everywhere, then you are usually up the creek. And this is yet another reason why you using explicit class names everywhere isn’t such a great idea.

You know, that's a tip I actually picked up from Sandi was she said that at one point she said something like, “The number one best thing you can do is never just hardcode the name of another class inside of a class.” That if you are going to use it, fine, but stick it in the variable somewhere that gets passed so that and that variable can default what you are going to hardcode or whatever, but just so its somewhere where you can get at it. And I've been trying to religiously follow that in the last several months. And it’s amazing. Like, you can mock anything without having to actually say, “Hey when he goes for this class name or --- “you know, you can just pass in to the method you know, or whatever.

Yeah. So I wanna kind of get into -- I have a question regarding these techniques -- so if you pass in a class name or an object that you are effectively taking advantage of duck typing. It just has to respond the right way in order to be the right sort of object to do whatever needs to be done in the method. Is that metaprogramming? I mean, how far does the definition go? Because in that case, you are actually telling it, “This is the thing that needs to receive the method call”, as opposed to being explicit and saying, “--- exact class or --- exact object method call.”

I would argue not. I think I've said it earlier -- and I don’t mean to pejorative about it -- but understanding what's going on there -- which is part of Sandi’s excellent coverage of dependency injection -- I think that's just things I wish I had known more about with respect to object orientation when I started. I think that technique that she’s advocating is just good design. I wouldn’t go so far as to call that metaprogramming personally. I don’t know, maybe some of us has a contention on that.

We talked this a moment ago about -- don’t explicitly name classes, put the class in a variable and deal with it indirectly -- as one way to make code more metaprogrammable. So can we look at this from other ways to make code more possible to metaprogram? What are the techniques that set your code upright to have it be more metaprogrammable? And I guess from the perspective of OK it’s just the design, how much overlap is that with just making your code generally have less coupling and be easier to refactor and just overall better designed?

I think you nailed it right there. It’s that the less coupled objects are to each other, the easier it is to go in there and insert something in a different layer or mock around with it. So to me, that does make it more metaprogrammable -- but ironically, it also means that I find myself using heck of a lot less metaprogramming.

Exactly.

Because if the objects are just loosely coupled to each other, then I don’t need some clever metaprogramming trick. I just pass a different object in there that wraps the one I was messing with or whatever.

OK. So this is great because Steven started by saying that metaprogramming is redirecting where messages go and if you are using good object oriented design techniques to manipulate objects, you are doing that instead of manipulating messages.

In case it’s isn’t obvious, I'm completely in love with the Sandi, so you know, I’m just in awe--

Get in line buddy!

I know, I know. [laughter] I finished her book on actually a week ago – Tuesday – so that would have been election night and I went to bed thinking I'm in awe of two human beings; one of them is Nate Silver and the second of them is Sandi Metz and I just... I mean--

We've considered changing our name to The Sandi Metz Fan Club.

[chuckles] I can see why. One of the things in her videos is that she shows I think she basically takes… she gets a string and basically needs to constantize on it and you know, that's a very sort of bread and butter kind of metaprogramming thing to do. But what basically happened is that she basically had implemented a contract across several classes and then basically because of their duck type nature, she’s just basically like, “Well instead of having to type something to invoke or back in the name of the class” or whatever it is, she basically used constantize and I was like, “That was just a perfect use of Ruby’s awareness and introspective capabilities.” And so, I think when you have really good design, it turns out that the metaprogramming is there to help you type less -which is definitely what Sandi demonstrated. And I guess my kind of thought about this now is maybe metaprogramming is best when the domain has a lot of inherent instability, or inherent ambiguity. And that’s kind of now where I'm siting metaprogramming reached its strong point, is that if you are to take… if you have solid object oriented design principles in your application, when you hit the place where the ambiguity is such that maybe object orientation isn’t going to be as easy as one might think, maybe that's the time that you can reach for that metaprogramming hammer on your toolbox.

I like that. I think that if we look at object oriented design as a way to decrease coupling -- and one way to look at coupling is it give you more flexibility around the assumptions that you are making -that you don’t have to make everything in stone right up front -- is that if you can’t if you are not achieving that decoupling by basically structuring how the object move around your program, then you can do it by structuring how the messages flow.

Absolutely.

And… OK so where was I going with that… [chuckles]

I was about to ask you for an example where one will be more appropriate than the other.

So here's something that I think… Well, let’s see if this works. So, the dynamic finders in Active Record (which have been around for a long time), when you can say Users.findByEmail and that findByEmail method doesn’t exist anywhere in the code base until you try and use it. So Active Record will… it uses a method_missing handler. You have a user object you send it findByEmail and pass in an email. There's no method there, so it used to work by just having a method_missing handler that would figure out what to do. And then somewhere I think around Rails 2.0 or early Rails 3 they changed that -- I think it is definitely 2.x series of Rails -- They change that so that it didn’t just do the work, it figured out what it had to do to do the work and then generated the findByEmail method in that class. So the next time you called it, it was already defined and you don’t have to go through method_missing.

OK but I love the example you just used because it’s like you said you can do find by email or you can even do like find by email and name right?

Yes.

And then you can pass two parameters and stuff. So it’s all these clever metaprogramming and stuff to do all these stuff. So in Rails 4.0, they’ve replaced this mechanism with one stupid simple method called find by. And you do find by and you pass the hash and the key in the hash in the field and value of hash is the value that you wanna match -- so email James.’whatever’. But then of course it’s a hash, so you can have multiple keys, now you can do email. And then you kind of see like it was some clever bit of metaprogramming, but then they just went back and like, “Well we could probably just define a method that works just as well, right?”

In other words, they did what they always do and they finally got around to doing it the DataMapper way. [laughter]

But let me finish where I was going -- because I think that this is a great example, what James brought up -- is that, you have the metaprogramming dealing with the find by email method in your user class and that's great. And then you may decide as you work with the user class that find by email needs to be a little smarter than the default implementation that Active Record provides. So you can go in your user class and you can override the method definition. You can do whatever you want and find by email. So you know, maybe maps domains or some in weird way or what have you. I think that's a great way to reduce coupling where in your code, you have something that is smart enough to handle the default case but it provides a path where you can make the behavior more specific to what you need over time. And what James just mentioned how in Rails 4.0, there's this unfind by method that you just pass in the data -- that's great -- but it’s hard to override that in the same sort of specific way without all over everything else.

So I totally agree with that, except for the one problem with find by etc. which is that, find by email and name (as stupid example) is the same as find by name and email—

Yeah. I know. [chuckles]

[laughs] So once you get into the ends you could override that find by email and name you know, and then somebody switches it around on you and they miss your optimization. So I think a better hook mechanism… I think you would have to structure the hook mechanism a little bit differently to really make that work well.

It’s the fact that it’s difficult to customize is really a fall. I mean, should the class be better designed where you could hook in to how Rails does that. If they left you a hook method in there, then it would be easy to with.

That is absolutely true.

One think I think—

Hold on. Before you go there Chuck, I just wanna get back to the point I was trying to make (so it doesn’t get lost) which is that, you can use that technique of metaprogramming as a way of when you are exploring in your code base and trying to figure out what you need. You can do something simple like that and say, “Oh, there's probably going to be these methods that I need, but I don’t really know how to write all of them yet. So I'll just write this method_missing type hanlder and use some other metaprogramming technique.” And then once you've sorted things out, you can go back and hard wire those methods and create that functionality in a way that is maybe less flexible, but more appropriate with your mature code base.

One other thing that I think is interesting about the approach that you outlined there Josh is that, they do the work kind of in the magical land where it figures out what it supposed to do and then does it. But then it also creates a mechanism so that the next time you do it, it does just follow the regular object model -- and makes it work that way.

Yeah. There's a reason the Rails team did that. It’s a big performance penalty -- this dynamic thing. So like in Ruby the way method_missing works (if you are not familiar), if you have a call stack in hierarchy basically and you send the call, it walks all the way up to the tree looking for that method and then if it gets to the top, it goes all the way back down to the bottom looking for method_missing and it walks the tree again. So it’s got to at least by definition be about twice as slow right? Because--

Right and its actually like an order of magnitude or two or more slow because in the vm when you are doing method lookup, you can cache the result and then the vm can very quickly get to the right method -- and that's pretty stable in those situations. But when you are doing the method_missing handler, you actually have to execute some Ruby code in there. So you have to like break out in the vm level of operation move up to Ruby for that level of operation and that's much slower.

So, it walks that line again. So the way Rails went to doing it where the first time it paid that penalty to hit the method_missing, but after it had everything figured out, it went ahead and defined that method. So then the second time was probably quite a bit faster because it was a normal method lookup and the third, fourth and blah, blah, blah times were probably much faster because by then, the cache probably had it, you know, so—

So I guess my question is, are there any other times that we are going to want to use metaprogramming -- bedsides maybe we have a whole possibly infinite set of methods that we don’t know whether or not they are going to be called so we are just going to handle it through metaprogramming? Are there other examples, other good places where you use these kinds of things?

Well funny you should ask. [chuckles]

Tell us Steven.

Yeah. If you look through the slides that I gave at RubyConf, there’s a lovely one where the late

Whitney Houston and it says “How will I know?” [laughter]

By the way, just seeing the pictures in Steve’s slides is like half the reason to watch his talk.

Thanks. And if you actually watch me, I am wearing picture of dinosaurs extracting their revenge on meteors from his Saturday Morning Breakfast Cereal Comic. I would say that for me now, the thing that I'm kind of using to think about when I need to really reach for a lot of metaprogramming is when you have a domain which is inherently ambiguous. And I think Josh has a really relevant use case which is that the dynamic finders and ActiveRecord is that, find by name and email and height and lollipop preference you know, I mean that's really some amazing power that Rails gives you. For me, I have a certain passion around linguistics and particularly deadline particularly Latin and if you think about the way that say, a grammatical vector is specified for a verb in romance languages, you might say active voice and present tense first person singular number, that's the full specification of how you specify one sort of state of this verb. But what happens if you wanna chop off say one of the specifiers? Will then the result get a little bit more ambiguous? Say for example in Latin number has two possible values; singular or plural. So what if you wanted get active voice, negative mood, present tense, first person you leave off the number, so now the correct answer has two results. Let’s say you have to chop off yet another specifier in that vector, well now the correct answer has 6 possible values. And so what I kind of found in that what Ruby allowed me to express is that, this domain -- depending on how precise specification gets -- the correct answer set either grows or shrinks. And so I view that as a very legitimate use of metaprogramming because the specifiers are flexible as is the set of correct answers. And I think Josh’s example around dynamic finders is the same thing; the correct answer changes on the specifiers that you provide at the time the message is passed. So for me, I think that is probably my golden on metaprogramming -- is use a programmatic structure that tolerates ambiguity well when your domain is inherently ambiguous. That is about as concrete as I can get about ambiguity.

And I like that example because it… so James’ response about the find by method and “Oh you can put a hook in there to be able to override it for a particular set of attributes that you are doing the find by on” you could create that hook but it might be kind of tricky to work out what the things are you actually need to build that hook around and what the mechanism is for making that work well. The find by x dynamic finder style of metaprogramming is pretty straightforward to do that. And you don’t have to think about creating this whole other mechanism. So I think that it actually is a case where metaprogramming simplifies your programming task.

Yeah I agree. I was just pushing for more examples because it seems like yours was a pretty specific example of maybe a broader case and I think Steven explained that pretty well.

Yeah. Sure to give a simple, broad example that probably most of our audience is familiar with along the lines of Steve’s example – Active Record itself, when it makes an object it has several methods that are backed by fields in the database, right? And the interesting question in both that example and Steve’s is could those methods be defined upfront? Could you run a loop that sets all those methods out but what are the downsides of doing something like that? You know, so it’s an interesting question I think.

Yea. I think it depend on the trade-offs and how you are using whatever it is. But yeah it’s definitely something that is worth exploring. We are running out of time, are there any other metaprogramming things that we should cover?

I'm going to mock out the time object so that we can allocate a few more minutes here in the call.

OK.

You can do that?

[laughs] Yeah.

As long as he doesn't have to touch time zones.

Thank you.

Oh, way to rub lemon juice into that one. [laughter]

Well, that was my crowning achievement of Twitter.

Yeah how is your Twitter client working out? [laughter]

I discovered recently that having a very popular tweet can break Twitter -- in that everybody in the world thought that it was good to reply to my tweet and then I couldn't find every replies to me that I actually cared about. [chuckles] Now I know what it’s like to be a celebrity for five minutes.

Yeah you don’t care about what anyone is saying to you.

[chuckles] Well, a lot of people reminded me that in addition to hating time zones, I should also hate daylight saving time and character encodings. [laughs] OK. Anyway enough about that. OK so what were we talking about with metaprogramming?

I don’t know but you were mocking the time objects.

Oh wait mocking the time object we are running out of time. Mocking time, yeah. I guess that's all I have to say about it. Moving right along. So it happens when I'm not drinking coffee, right?

Steven, were you in the middle of a list of things that characterize—

Yeah that's true. I think we cut him off with his list.

I don’t think I was characterizing. I think James—

Not characterizing but like four things--

A process of moving on is -- well one, you can look at the slides and so, they are actually very digestible. But if we were to try and run through them what I kind of called you know, -- how to level up in metaprogramming -- you are going to start with understanding things like the open class in particular, the kernel method and the monkey patch. You need to understand how the adder methods work, and you need to understand how alias works -- how you can alias a method. This, plus the understanding of the ancestors chain, that's what I call ‘tier one’. And that's enough to let you paint yourself into a corner. And I think that's where podcast like this and where similar exposure will help you decide whether or not metaprogramming is warranted -- choosing between define method and just, you know using plain old definitely. And then I definitely say that the route to making sure that you have a good understanding of metaprogramming is understanding what the Singleton Class is and understanding the ancestors call. And as you move on, I’d say that the second tier is basically when you start integrating those… first your techniques and also where you start rolling objects send in more and more often is where you are more comfortable with catching message and then sending it to the object; as well as the sort of glittering diamond in the crown of metaprogramming which is making use of the method_missing call. And I think that’s probably… I’d say those are all the techniques.

Steven, I wanna add one more technique in there because I think it’s really important and I want Avdi to tell us about it.

Oh OK.

[laughs] Avdi, this is your rule that when you are adding methods to objects, you should always put them in a module whereas.

I figured Steven was probably going to get to that one next -- at another level.

I though Steven was done. [chuckles]

No.

Sorry.

I actually totally stole that technique from Avdi, so I do think he should definitely talk about it. I do recommend that as you are kind of moving on to the stratosphere of metaprogramming, you need to start thinking about basically the no holds barred technique -- which would be class eval and instance eval. And then from there on, you are pretty much free. I’d say you probably have the techniques in your belt to implement the metaprogramming style of programming. But at this point, you really need to think about how to do it responsibly. Because if you are writing this code in some other human, if only your future self is going to see it, you need to think about responsibility. I think that's where you need to think about understanding respond_to and respond_to_missing -- which are such beautiful methods. Respond_to_missing, I did a refactor on one of my classes and just cut out so much code that I was trying to handle in respond_to. Thanks to respond_to_missing, it was just amazing how much code that can save you [inaudible] programmatic idiom. And with that I would like to segue with Avdi's excellent, excellent observation about why you should put your… it’s why you should go… what is it… I guess you should go receive messages by putting it in to module or---

Something like that, yeah. Well, let me first say that using respond_to_missing, I think I haven’t finished the talk where they talk about this -- or finished watching the talk where the talked about this -- but I think that's one of the things that Jim Weirich refers to as polite metaprogramming, where you not only respond… you not only implement missing method to respond to some random method, but you also let other folks know that you are going to be… that you can respond to that by implementing respond_to_missing. And the cool thing about respond_to_missing is that you can reuse that in your method_missing and so you can dry up your test to see whether you can handle a given message.

Another cool thing about respond_to_missing -- in case we haven’t sold it yet -- if you implement respond_to_missing, then it makes more of Ruby’s object system work with those imaginary methods that you are handling through your method_missing. For example you can now use things like method to get a reference to them -- which you can’t normally do. But Ruby knows it’s there because it calls respond_to_missing to see if you are going to handle it and because you are it’s like a fake proxy method on top of it and does better.

Yeah. So as far as modules go, if you are going to write your own version of attr_reader or attr_writer, maybe you are like writing attr_reader that also has a default value or something like that. You know, the first thing you might turn to is like a define method or class eval or something like that that evals in a new method. And of course what's that going to do is that's going to put the method right there in the class where you set attr_reader or you know my attr_reader. And that means that if somebody wants to… in their class where they use that, if they want to slightly modify -- maybe they wanna put like a little bit of extra layer -- maybe like a caching layer or some kind of validation layer on top of that attr_accessor or something -- they are going to basically going to have to rewrite your code because as soon as they define the same method, your method is lost. So they are either going to have to do some fancy alias footwork or some fancy save that method object and then reuse it footwork or something like that, whereas, if you as the metaprogrammer implement that special attr_reader or special attr_accessor to insert into generate a module at runtime and put that generated method into that module and then insert the module into the class that's being extended, now you can inserted it into that wonderful ancestor chain. And so if somebody wants to do a customized version of your method, they can just write their own and call super to call back to your definition of it. They can reuse your definition very easily with super. So that's why modules are good.

So just explain that another way, if we have been talking about the class lookup calls a line and there's various points on that line where your messages can stop, then if you do something like class eval and --- in to the same space, you are trying to squeeze multiple things in to just one of those points on the line right? Whereas it’s better to take two points on the line, cut the distance in half and then put something better.

Exactly, yeah. I should get you to start like drawing pictures for my Ruby Tapas episodes. [laughter]

Nice. So we had this conversation a while ago on Rogues about putting methods in modules to metaprogram them in. And then I went back and added that to Active Record associations so that you can do the super trick with them.

Thank you. I actually used that in Active Records now so--

Yeah I do too -- amazingly enough.

Much more useful than I expected.

All right. Well, I think we are out of time. Really appreciate you coming on the show Steven and this has been an excellent episode. This is one of the ones where I’m probably have to go listen again just to make sure I understood everything that was talked about.

[chuckles] I know, I will.

CHUCK: So—

And from now on, we get stuck on a definition we'll just make our call out to Steven.

There we go -- steven.define.

Ooh, a delegator. I like it.

All right. Well let’s get into the picks. James, what are your picks?

OK. I just have two quick picks this time. The first one tech related pick is this cool library I found recently called hamster. I don’t know if you guys have seen this but it’s basically like various Ruby data structures but they are immutable and thread safe and all of that. So, very interesting. It has things like hash, set, list, stack, queue, vector -- and so like I said, all of them immutable. So that's pretty interesting stuff if you enjoy that style of programming. I’ve been looking at that more since we read GOOS and it talks about keeping your low level data structures immutable -- which is not something typically see with like hashes and things like that. So this library is kind of interesting for that. Then for a non-programming pick, I have been using iTunes Match recently -- that's Apple’s new music service. It’s pretty cool. I like it. It’s you basically pay them a fee – it’s like $25/year -- its ridiculously cheap. And they just match your music library based on what they have in the iTunes Music Store. So they have a ton of music so that just means most of your music is just basically instantly there. And then anything that doesn’t match, they just upload. There is an upload limit, but I was way under it. I think it is like 25,000 songs but they are only like a thousand of my songs that didn’t just match right off the store. So anyways, then your entire library is on the cloud either because it was matched or because it was uploaded. And anything that Apple match, they upgraded to their best they have. So even if you had an old mp3 ripped off a CD or whatever, you now have the best that the iTunes store has to offer of the same song. And then you can set your advices where they just stream music from the cloud. So it’s kind of awesome that you have like the computer and iPhone and the iPad and all that, because you can share the same music everywhere. And if you need not to stream for some reason, there's download buttons, so you can download an entire artist or an entire playlist or whatever. You know you are going to be off the grid for a while or something. Anyway it’s a neat service and I've been enjoying it, so you might wanna check that out. Those are my picks.

So does anyone remember -- like a decade ago -- when mp3.com got sued out of existence for offering the exact same service?

Well, did mp3.com work all the licensing deals with the music labels?

No because you couldn't do that back then because the music labels just thought that mp3 was another word for piracy back then.

Yeah, I know, I know. It’s just not how they don’t figure out that the—

Yeah I think they called it like backpack or something like that. You know, it would do the same thing -- go through your library and it would match the ones that they already had in their server farm and then you could stream your whole library.

Yeah it’s like the future was barrelling down on the music industry like a runaway train and they are like standing there with flags trying to get it to slow down.

Yeah. They’ve been seriously --- behind the times.

It’s like the quote by William Gibson: “The future is already here -- it’s just not evenly distributed.” Has been steadily refusing the distribution of the feature for… my gosh you know Napster was around when I was in college so you know, it’s hard to believe that it’s taken a number of years for them to actually get the message that evolve or die.

I wonder how many of those record labels Apple negotiated with the 2x4 to the face because… and honestly, the ones that they had to just be like, “Dude, hit him harder. Knock some sense into him.” [laughter]

Apparently they used Scott Forstall for those jobs. [laughter]

All right Avdi, what are your picks?

Well lest see… a book pick Rebuilding Rails by Noah Gibbs. And if you've ever wanted to understand the internals of Rails better, this is a good place to start because basically you go through and rewrite your own version of Rails -- and you know, it is a simplified version of Rails -but you will learn basically how it works and how the router works and how controllers work together with view and stuff like that and you learn it by rewriting it yourself. So I recommend that. And for non-programing pick, Aardman Animations Ltd. otherwise known as Aardman Studios, you may know these folks as the creators of Wallace and Gromit. They have done some movies as well and they’ve got a couple of shows of their wonderful Claymation on Netflix. Now there is Shaun the Sheep and there's a spin off now for I think even for younger kids called Timmy Time. And the thing that I love about these people, they do this Claymation shows and unlike so many other children’s programs, they have resisted the trend of converting a perfectly good Claymation show to like soulless CGI -- they are still doing stop motion. And they just have these great styles to them and their children’s programs are the kind of programs that you can put on and not wanna claw your eyes out if you are in the same room with it. So, yeah our household has benefited from their stuff because it’s like, sure put on some Shaun the Sheep -- nobody minds Shaun the sheep.

Yup. All right Josh, what are your picks?

OK. Let’s see… I have two video picks from the recent RubyConf and one of them is a gamble [chuckles] because Sarah Mei's keynote video has not been published. But Sarah Mei did a keynote and I believe she is the first woman ever had a keynote at a Ruby Conference -- only took 11 years.

Woo hoo!

[chuckles] Yeah.

Oh wait -- Sorry. That was depressing.

Yeah. [chuckles] Progress. Yay! So she did a talk called The Insufficiency of Good Design that really got people in the conference talking. And it was really about the interaction between how you structure your teams of people who work on software and what that does to the nature of the software itself. So that was a really great talk. And then other talk that I really liked… there were a lot of good talks in Ruby Conf but I think maybe the most important talk of the entire conference was Brian Ford gave a talk called Toward a Design for Ruby, where he talks mainly about the political situation of the all the different Ruby implementations and the kind of process by which the Ruby language design moves forward. And it’s just a half hour or so talk I guess 45 minutes or something like that, but it’s really worth watching if you feel like you have the stake in the future of Ruby.

Only if you have a stake in the future of Ruby.

If you don’t care about the future of Ruby, don’t bother watching it. [chuckles] [laughter]

Geez! I'm hanging up on this call right now. [laughter]

OK so, that's it for video picks. And then I have a fun pick. So Rick Olson turned me on to a comic book a couple of years ago called Powers -- it’s by Brian Michael Bendis and Michael Avon Oeming. These guys used to be comic book artists -- I think it was at Marvel -- and they have done some fairly high profile work there. But they wanted to write hardboiled detective murder mysteries involving superheroes and you can’t do that in Marvel because you can’t really run around killing off Spiderman (people will get upset). So they created a whole new comic universe where they can kill of people and then have murder mystery detective stories about it. It’s like the best superhero comic book that I have read since the Authority came out. And the reason that I am picking it now is because they have like 15 graphic novel compilations of the series as it existed a couple of years ago, and I’ve just read that they are going to be restarting the series. They are going to be rebooting it and continuing on the story line. So I'm going back and rereading the old stories so that I can get myself caught up again. And I just love them. They are really good. They are very mature, they use strong language -- it’s definitely it’s an R-rated comic, so it’s not the kind of thing you should share with your kids but if you are an adult connoisseur of comic books, I highly recommend it.

Awesome.

And that's it for me.

Steven what are your picks?

Well, as David is feeling ill, I was told I could get some extra picks, so I will toss a few of those in there. The first one I like to say is in my company – if I can slightly ever so slightly – I work at shopittome.com, so A. we are hiring, that's why I’m doing this and B. we have a lot of interesting data about people and retail habits and we were a Rails and EmberJS shop. And I'm told by Yehuda and Tom that we are one of the largest Implementations of Ember. So, it’s been a lot of fun to work on and I think there's a lot of great design direction behind Ember so, if we are looking for a JavaScript framework for thing about how to build applications, not how to build a single page app or an app --- or you know some tiny app, but you have to think about building a JavaScript applications that work online and offline, I highly recommend EmberJS and its associated data. That's my number one pick. I really do like Ember quite a lot. For some fun sort of programming tools that I have seen, my boss – so here I am sucking up again – I was actually working on a thing called fastrAck, which uses Go and Go routines to basically consistently index and create something like a C tags repository so that you can basically browse your code very, very quickly -and I’ll provide a link for that. I also recommend a tool that I wrote called GIFL, which is Google I'm Feeling Lucky. So if I'm writing markdown and you need to get a link from I'm Feeling Lucky – this is basically something I stole from Textmate -- look at GIFL. It is a simple little plugin that will allow you to go out and feel lucky and get those links and inject them in markdown and just makes life a lot easier. And then lastly, I’d love to maybe pitch a thing at that's kind of also fun, but I also think has a moral cause on it, which is social engagement. As programmers, we do all of things alone in dark rooms and I would encourage everyone listening to just try doing something social. I've been reading a lot about urban design and theories of community and they are used to this notion called a third place, which was not home but which is not work, where people went and they had a rich social life. So here, thankfully in San Francisco in the Greater Bay Area, you can actually pursue swing dancing -- that's just my personal third place. I go out usually twice a week and so see live music and meet my friends out in this third place. I think that's a really valuable thing. So get out and meet people and have social interaction. And kind of was associated with that -- I'm going to preach here -- which is that I believe that rationalist and seculars have a more obligation to donate and/or be involved and civic or intellectual institutions. So go see an opera, go to a museum or find somebody. You don’t have to be at that Bruce Wayne level of being socially involved to have an impact, but if you think about in the golden age of America -- say the early 20th century – we built these wonderful social institutions which were meant to spread the profusion of learning and rational thought, knowledge and appreciation of the sublime. And that we are not doing that as much as we used to anymore. And part is because it’s harder to concentrate wealth like you used to in the era of Carnegies and what not -- but just on a personal level, you should be engaged with your arts -- because people who believe in irrational behavior are funding irrational institutions, you know, al-Qaeda being a one. So I highly recommend that you be engaged with rational, tolerant, accepting and enlightened social institutions. So that's just my pitch on that.

All right. Well, I guess I'm going last. So I have two picks; the first one is something that I picked up when I was getting started with Ruby and it really helped me understand a lot of the things that we talked about here today and that is Dave Thomas’s videos -- The Ruby Object Model and Metaprogramming. He did this quite a while ago – I think it’s based on like Ruby 1.8. something, but the concepts are still relevant and it really helps you get a good idea of what's going on under the hood with a lot of the stuff that we talked about. The other pick that I have is I'm going to be speaking at New Media Expo. I'm going to be on the podcasting track and if you are interested in blogging, podcasting or anything like that, I highly encourage you to come out. I'm going to put an affiliate link on the website, so that if you wanna go, you can just click on it and you know, get a little commission for that. But anyway, either way just you know, if you are trying to build that audience or figure out some of the social media stuff, then this is definitely a great way to go. So those are my picks. And we’ll go ahead and wrap up the show. Are there any other announcements we wanna go before we end the show?

Thanks Steve for coming on.

Yeah thanks a lot.

Yeah absolutely.

It’s been a dream. I really enjoyed talking with you fine folks. So hopefully all the listeners will find this enlightening and feel free to reach out to me – GitHub, Twitter, whatever it is – If you want to take me the task for my definition metaprogramming.

Yeah it should be an interesting conversation because I think everybody comes up a little differently and so understanding some of the magic will I think everybody will be better at that kind of design in their code. All right well, let’s call it a show and get back to work I guess. [chuckles]

Goodbye Internet!

That's all folks!