The Rogues talk about ActiveRecord with Ernie Miller.

Special Guest: Ernie Miller.

 

Katrina, while you were gone, they voted me off the island for the terrible introduction I gave.

[Laughs] Should we try this again?

They say we’re going to try it one more time.

Okay.

Or I’ll just be summarily executed or something like that.

Okay.

It’s all part of James’s evil genius plan to take over the Internet.

Good morning everybody and welcome to Episode 110 of the Ruby Rogues podcast. Chuck is out today, so I’ll be your host, James Edward Gray II. And with me today are, Josh Susser.

Hey, good morning from bright and sunny San Francisco.

Katrina Owen.

Good morning from the phone booth.

And we have special guest, Ernie Miller, with us today.

Good morning from considerably less sunny than San Francisco but still awesome Louisville, Kentucky.

[Chuckles] Ernie, this is your first time on the show. Would you mind introducing your self?

Yeah, sure. I'm just this guy, you know. [Chuckles]

Okay, let’s move on. [Chuckles]

I've spent a lot of time working with ActiveRecord. I write a lot of Ruby gems that interacted with it, levels that are frankly deeper than any man should go. And I actually recently gave a talk, ‘An Intervention for ActiveRecord’ where I talked about some of those restorations which was something of a contrast from the last talk which was all about ‘Optimizing for Happiness’ and things that make me happy. So, that’s been an interesting kind of line to walk.

I think we kind of know you as the ‘Happiness Guy’. That talk has kind of gotten around and whenever people bring up Ernie Miller, people think, “Oh, happiness.” But once we saw you get angry, that was when you became really interesting.

Oh, well. Yes, I'm a many-faceted man. I have all sorts of personalities. I'm really excited to be on the Ruby Rogues and talk to you all. And it sounds like we should have a great time talking about all the various warts of ActiveRecord.

Oh, I'm looking forward to it. [Chuckles]

Why don’t you break it down for us, like what made you decide to give a talk called ‘An Intervention for ActiveRecord’?

It’s interesting. A long time ago, and by that, I mean, in Internet terms maybe five or six months ago, I had actually posted something to the Parley List about this. And I had been complaining about certain things regarding ActiveRecord. And at the time, I had said, “I think this is going to have to turn into a blog post or a talk on why I hate ActiveRecord.” And I decided ‘Why I Hate ActiveRecord’ is probably not the best way to title your talk especially if you're going to submit to RailsConf.

So, we went with ‘An Intervention for ActiveRecord’. But I've spent a lot of time, like I mentioned before, writing gems to interact with ActiveRecord to modify its behavior in certain ways, to try to simplify certain things. The more I dug into ActiveRecord’s cuts, I started to realize there was kind of an underbelly that a lot of us, we would rather not look at, we would rather just read the docs and use them and trust that everything works the way the docs say they do. And it got very frustrating. I don’t really know. I mean, I wish that I had, at the end of the talk, not to be a spoiler or anything. But at the end of the talk, there's a call to action and it gives some suggestions as to where to start. But really, I don’t know how much we can do to fix it now. But at least, it bears mentioning that there are a lot of gotchas that people just don’t realize until they're neck deep in ActiveRecord code.

I remember hearing a few years ago about an application or a tool that was written for, I guess it was on the UNIX platform and it’s probably in the 70’s. And I can't remember what it was. I think it might have been VI. And there was some problem with the interface, something that the creator thought, “You know, I think maybe I made a mistake here. But it’s too late to do anything about because there are already seven people using this.”

And I kind of think that this might be the same thing with ActiveRecord is it’s in use in so many places that if you do decide to, for example, get rid of default scopes, what is the process for doing that in a way that doesn’t frustrate and make people’s lives very, very difficult for a long, long time.

Right. And Katrina, I'm glad you mentioned that because it reminds me of something that I had to cut from the talk for time which is essentially, our biggest problem now. And when I mentioned that ActiveRecord in certain ways is like a sea and that all the good and bad that that entails, a lot of that means that because it’s in use so many places, people are going to be super afraid that changes are going to break their code. For a lot of people, ActiveRecord is, and I kind of have to chuckle when I say this, but it’s as close to the metal as a lot of Rails developers get. They consider that as super low-level stuff, “Oh, I'm hacking away my ActiveRecord.” And so, because of that, it’s like you're pulling the entire foundation off from under their building if you change ActiveRecord behavior.

Yeah, exactly. It was hard enough to go from like Rails Version 2 syntax to Rails Version 3 syntax.

It’s a great example of the time when ActiveRecord did go through heavy changes.

I feel like we’re getting a little ahead of ourselves here, talking about how to fix things.

[Chuckles] We’re engineers. Isn't that what we do?

Maybe we can talk about the problems first and then perhaps, that will guide us towards thinking about solutions.

I have an intro to that.

Okay.

Ernie, during your talk, you said something that really piqued my interest. You said, “Believe it or not, not all opinions are equal. You have to make trade-offs whenever you exercise an opinion.”

Right.

Could you talk about that a little bit?

I think that a lot of the trade-offs that were made and I'm speaking as a third party here and not having been involved in the original design of the ActiveRecord gem, at least. And so, I think that a lot of the trade-offs that were made were toward ease of ramp-up. And certainly, the ActiveRecord pattern in and of itself is designed for that. One of the things that I think is really interesting about that is that, I'm sure this has been mentioned before, but when Martin Fowler wrote about the ActiveRecord pattern, he said, “It’s a good choice for domain logic that isn't too complex.” And he goes on to say things like, “If it’s going to be used on something that doesn’t directly correspond to database tables, then you're going to have a bad time.” That’s essentially what Martin Fowler is saying. And if your business logic’s complex, you want to use direct relationships, collections, inheritance, and so forth. And all of these things are things that got grafted on to the ActiveRecord pattern using -- essentially everything that you're talking about there has been grafted on using Class Macros. Class Macros can be very handy. But a lot of times, their stand-ins and especially whenever you look at some of the decisions that were made towards saying, let’ say, ‘has many’ whatever, and a bunch of options to that ‘has many’ associations. A bunch of configuration for one object which really, if you were to approach something like, let’s say, how CarrierWave works where it mounts an object on a particular attribute of a model. If they would do something like that, then you can use all the things you already know about Ruby. So, I'm going to subclass the ‘has many’ association and make my behavior do whatever I need to do. And now, I can see it and can configure this mega object that association’s done. But I guess, to go back to answer your question, Katrina. Sorry, I do tangents. The thing about that I think is there was a very strong preference toward Class Macros even when plain old Ruby would have probably made more sense.

So, why do you think that is? Why do you think that Class Macros were chosen? Is it easier to understand the Class Macro? You said that part of some of the trade-offs that were made were part of simplifying the ramp-up time.

Right. I think that Class Macros are superficially easier to understand. They only become hairy once you start to look at how they're really implemented and need a behavior that’s not actually provided to be a configuration option.

One thing I think is useful to pull into the conversation here is when people talk about ORMs, Object-Relational Mapping systems, the phrase that always gets used is impedance mismatch, even though almost no one understands what impedance actually is in Electrical Engineering. [Laughter]

We all sort of understand the problem that you have, one model and a second model and you try and fit them together and it’s like a square peg into a round hole. And it never quite fits right. I think that a lot of the terrible stuff that’s in the ActiveRecord library is because of that impedance mismatch. And so, you have this object-oriented API on one side, or mostly object-oriented, and then you have this relational database model on the other side, and you're trying to fit them together. It’s hard to fit them together. So, at one level, it’s easier to paint a square around the round hole and pretend like the peg fits. I think that you can gloss over a lot of the problems of trying to get the two layers to couple together effectively by pretending that it’s not a problem.

You just dress it up and say, “This is just Ruby. Everything’s right, everything looks fine.” But you still have the reality of that mismatch that is never going to go away.

We were kind of talking about this a little bit in the pre-call. But I mentioned that I've been playing around with Sequel, the database library, yesterday at work. And it has this kind of interesting thing where the manipulations you do to either kind of building up these queries and it works on these datasets and stuff. And it’s more of a low-level than ActiveRecord. It’s like a step above using the PG gem directly but a step below using something like ActiveRecord when you're using this kind of query syntax, the non-ORM version of it. But then, it has an ORM on top of that. And what’s interesting about its ORM is how it doesn’t feel as much like an impedance mismatch. The ORM is just kind of another one of these datasets that it’s building up and is kind of a specialized form of that. And it seems to flow more naturally, I think, than the divide between ActiveRecord and Sequel because there's not that step in the middle.

I think that it lets you resolve that mismatch in your brain rather than…

Yeah, maybe that’s it.

Yeah. It doesn’t hide the relational aspects of the system. It brings them closer to the surface so that you can actually get your hands on them. But I think the problem with that is that superficially, it looks simpler. But all the value that the ActiveRecord pattern provides for you about mapping all that stuff into objects, you don’t have that anymore.

Someone recently and I think it might have been Glenn Vanderburg, was talking about how abstractions are always going to be leaky. What you want is that they'd be leaky in expected ways. You don’t you’re your abstractions to leak in unexpected and surprising ways. I think this is a lot of what this is all about is that the surprises and the gotchas are what make this so inherently difficult to understand.

Sure. And what I refer to that in my talk as is douchebag magic. You got the good magic and in effect, in the ActiveRecord documentation, it actually says, “Magic is not inherently a bad word. It’s one of the guiding philosophies.” So, I can agree with that. But there’s the good magic; the guy that pulls the rabbit out of the hat. And then, there’s the David Blaine Street Magic guy that goes around and just ruins people’s lives.

If you haven't seen the video, maybe we can link them in the show notes. I really wish that I had time to show one of those videos and license to show one of those videos at RailsConf actually. They're pretty hilarious. [Laughter]

I think one of the things that both Josh and James, you both touched on is that this kind of magic that we do, and Katrina, you mentioned the leaky abstractions, that leakiness is inherent. Glenn was definitely right, and I think it was Glenn because I've had that talk with him before. The thing about it is I’d even gone the wrong way and I've had something of a shift in my way of thinking lately. A lot of the gems and a lot of the software that I've written has tried to further speckle the cracks that are in the face of ActiveRecord. And in particular, I think that maybe the problem is not that we need to make the magic go further but rather that we should strip more of the magical way. With Squeel, Avdi was really kind to give me some kind words about Squeel when he made it one of his picks actually one week. But the thing with Squeel is I tried to make the magic go further. I did things like made association names more consistently more reusable. So, if you join an association by a particular name, you can also reference it in your wares and you don’t have to worry about how it’s aliased when it reached joined. That’s cool and all but it’s going to break down sometime. And so I think, stepping back a little bit that I went completely in the wrong direction with Squeel to be very honest. You're not going to make the magic last long enough. So, in all honesty, more magic needs to be eliminated from ActiveRecord.

I think that’s an interesting point but -- [expression] let’s see. I can't decide how much likely. [Laughter]

I can.

I mean, I definitely agree with your sentiment. Katrina brought up the leaky abstraction thing. And one of the lines that drives me crazy in ActiveRecord is like when you're doing a ‘where’ clause and you want to go from this value equals to this, to this value is in this list, and that’s ridiculously weird. You have to switch to the SQL version and put a question mark inside the parentheses or something like that.

Well, no.

Well, no.

You can give it an array.

Can you do the hash with an array?

Yeah.

Yeah.

It’s only if you do…

Not equal.

It’s only if you're using the SQL version, then you have to put parentheses and put the question mark inside the parentheses. Is that right?

That’s true, yeah, through your [inaudible] SQL interpolation.

I mean, why would I put a string? I’d just put a question mark and it knows to put the quotes for my string. So, why wouldn’t it put the parentheses for my array, it can detect that I passed an array. You know what I mean? Like, why do I need to put those parentheses there?

Oh, consistency. [Laughter]

I don’t know. It’s just weird, like it’s the place where the abstraction leaks. In that case, I would prefer more magic. I would rather just detect that I pass an array and put the parentheses around it.

You keep using that word ‘magic’. I do not think it means what you think it means.

Oh, good. Let’s get a definition of magic.

[Laughs] In Rails, magic is ‘we’re just going to do the thing that you probably want to do’ because we assume that you're doing the same thing that everyone else is doing. Mapping the name of the model to the name of the table is magic, right? But it’s really just following a convention. And most of what ActiveRecord calls magic is it seems like magic because it’s doing your work for you because you're doing the things the ActiveRecord way.

The definition that I found of magic is the power of apparently influencing the course of events by using mysterious or supernatural forces. [Laughter]

I think that what it refers to is supernatural record. [Laughter]

I think, Josh, you bring up a good point. You said that magic, in that definition, is kind of like how ActiveRecord intuits that the user model is connected with the user’s cable and then loads columns and stuff like that. There is another case where I would actually argue that I think the magic is great.

I like that magic.

Yeah, me too.

I think a slightly more complicated example of magic is Counter Caches. Counter Caches, I think, are amazingly magical because there's a couple of pieces that happen and it’s so awesome what it gives you. All you have to do is add a little option to your ‘belongs to’ association declaration, saying ‘counter cache true’. And suddenly, you’ve de-normalized the counting of the child records. So, if you have a post and it has a bunch of comment, it has many comments, and a comment belongs to a post. And you say ‘comment belongs to post counter cache true’, and then you put the comments count field into the schema through the post, suddenly every post knows how many comments have been made on it and you don’t have to do a query to go count up all of the comments. And you can query posts by which have the most comments. Query post can bring back the one that has the most comments. That’s really magical. But once you look at the code for it, it’s actually really simple what it’s doing and how it’s keeping the Counter Cache updated and in sync and all of that. And also, you see all of the places where it falls over and it can't keep up with what you're doing.

Right.

So, you’ve got us a positive thing or a negative thing?

Oh, I love it. I think it’s great.

Ernie has an example that I just love in his talk, that’s my favorite, of magic gone horribly wrong. Ernie, can you tell us about Callbacks?

The entire call will have to be about all the craziness in Callbacks.

Did we already call someone back? [Chuckles]

[Chuckles] Yeah. Katrina was disconnected from the phone booth.

Right. In ActiveRecord, there is a concept of basically attaching to any number of methods that you might call on your object and overriding their behavior. Of course, in ActiveRecord, we call that ‘Callbacks’. And actually, it’s mostly from Active Support. Most of the behavior is actually part of the Active Support Callbacks module. But really, whenever you look at it in plain Ruby, this is the kind of thing that we would do with inheritance and super all the time. Like for instance, it is very nice. And actually, I had a pretty good talk with DHH over dinner the night before I gave my talk. That was a fun conversation to have actually, “Hey, my talk is the only one about ActiveRecord at the conference.” He’s like, “Oh, really? What's it about?” I'm like, “It’s all the things that make me crazy about ActiveRecord. It’s kind of like a roast.” [Laughter]

I think DHH had Ernie thrown out of the restaurant.

[Laughs] But he took it really pretty well. Gosh, I don’t even know where to start with Callbacks. The thing about it is when David described what he liked about them, he liked that the behavior was declared at the very top of a file. So, you're saying, “After save, do this thing.” And that’s great as long as the behavior is super simple. But once you have Callbacks being added by, let’s say, gems, once you have Callbacks being added from multiple locations or you have Callbacks on an association that may or may not be getting called whenever you save associations with their parent, things get really messy really quickly, and you end up with kind of Callback spaghetti. The funny thing about it is if you want it, if you look at the way that ActiveRecord actually implements Callbacks for most of the primary ways that we use them - save and update and I think commit is in that file. But if you look at the ActiveRecord Callbacks module, it actually shows that it is calling super eventually. It’s getting the behavior that comes from the rest of ActiveRecord. But it wraps everything in this run_callbacks block. Run_callbacks is actually running code that’s generated from all of the various macros that you’ve called. So, as you're calling afterSave, it’s appending your afterSave value to a Callback chain. And that Callback chain can be -- and I’ll use the term ‘loosely’ compiled. It gets compiled to a giant string of Ruby. The thing gets evalled into a method that gets defined so that it’s basically cached. And if you'll look at the actual output, you can do this for fun some time. Log into a server somewhere and pull out whatever your craziest code is and actually grab the Callback. I think I have it in my slides actually. I’ll give you the exact thing you can run. If you want your SaveCallbacks, for instance, instantiate a user or whatever your object is and call _save_callbacks. And then, call compile on that. And you'll get back the string that actually gets generated by ActiveRecord. It’s just a huge number of If-Thens essentially that execute and maybe call some auto-magically define methods and such on your class as well. And although this is essentially to say ‘do this few things before this other thing and do these other things after this thing that you did’. Inheritance and super is great for that and it just works. I actually had people, and this is the problem. I think that people have gotten somewhat brainwashed by the Rails way of doing these things because I had people come up to me after my talk and I had people Email me after my talk saying, “Those demos you showed where you were just calling super in your create or update method, can I do those right now or do I need to have an update to Rails to be able to do that?” I'm like, “Yes, you can do that. It’s Ruby, it just works.”

Ernie, you showed that example and I thought it was so awesome the way you just like, “This is what a ‘beforeSave’ looks like.” And you override save and you do the before thing and then, you call super.

Right.

I actually took that example in a talk I gave at Scottish Ruby Conf and I took it one step further. And I just defined in a simple ActiveRecord-like thing, I just defined an empty hook method called beforeSave. So, to create the subclass, all I had to do was override beforeSave and I didn’t even need to worry.

James, you're going to love this one. You reinvented something that existed back in the two/three days.

[Laughs] Nice.

So, we got rid of that at some point. And I thought to myself, “Oh, gee. We could have actually had a template method that did all that stuff and call before…” But no, we don’t do that anymore.

In The Gang of Four Design Patterns book, there is a Pattern Chain of Responsibility. And I think that the ActiveRecord implementation of Callbacks is a weird variation of Chain of Responsibility. It does a lot of the stuff that Chain of Responsibility is supposed to do. But the prescription for using Chain of Responsibility is you use it when multiple objects can handle a request and you don’t know what they are explicitly and you want to dynamically specify those objects. With the ActiveRecord Callbacks, you're doing all of these work to compose up the responsibility chain but the thing that’s always handling the Callback is the object itself or you have an observer that’s hooked in. It’s not really a job for Chain of Responsibility.

Right. That is if you want some level of dynamism to it. In this case, the thing that I liked about the examples that I was showing on the slides and certainly, the example that James re-implemented, is that you can just read the code. You want to find out what happens before save, look at the beforeSave method, let’s say. Or if you want to find out what happens -- I think that it probably requires some creative renaming of certain methods inside ActiveRecord, for instance. It’s not the most intuitive thing to think that you need to define your own create or update method, not create method/update method but a method called create_ or _update method. That’s not the thing that I would think I would need to redefine if I wanted to do a beforeSave for instance.

This is a great example of sort of application of patterns gone wrong.

If you look in the Callback code, it’s this massive queue system, things built in there. And I guess, the argument for it, if I could try to make one, is that it’s this kind of inheritable queue the thing can be manipulated by subclasses and stuff. That only comes into play, I guess, if you're doing like STI or something like that because you need the inheritance.

Right.

I don’t know. It’s just like it is the most unbelievably complicated thing you could possibly imagine. For some things, they're fairly simple concept and so, you have to say, “Is this level of complexity needed so that in STI, I can remove a Callback that my pairing added,” or something. Like, that’s the edge case that this thing ends up making almost central or something.

But simplifying that to Standard OOP Inheritance would also solve that problem and even more simply.

Right, because you could override the method to do nothing? [Chuckles]

Right.

What's the trade-off here, though?

This is back to what I was talking about before and we sort of wandered off away from that. So, maybe I can try and make that point more stressfully or forcefully, is that you have this inherent impedance mismatch. You have a relational model in your database and you have an objectoriented model up in Ruby. It’s just the laws of physics. You can't make those two things work together perfectly. So, the art of building an ORM here is like James is talking about how you want your abstraction to leak in the right ways. If you look at the ActiveRecord library, it leaks in all sorts of really weird ways. Like, “Give me a string of SQL, or a fragment of SQL that I can compose the other with some other strings.” Yeah, isn't that great? That’s way too leaky. I think that a lot of the oddness that’s in the ActiveRecord API and all of these weird warts and strange applications of OOP patterns, I think sometimes that decision about which side of the model you want to favor. Sometimes it’s better to favor exposing the SQL and sometimes, it’s better to pretend like everything is object-oriented. And then sometimes, they don’t fit together. Okay, I'm talking in circles now. [Chuckles]

Right.

Somebody save me. [Chuckles]

That’s actually, and I think we’ve been veered off from the topic of magic from that because I was basically saying that I think that ActiveRecord maybe needs some of its magic scaled back. Not the magic that you were referring to, for instance, with things like knowing what the table name is. That’s great magic, I like that, at least as a convention. But things that it tries to do when it gets too smart for its own good and chew-chew in the foot. It’s one of those things, like for instance, count. The count method, especially in the world where you're doing chainable relations, this was not a big deal back in the 2X days whenever you would call count and give it conditions. But when you have a relation chain, so you're like ‘where this’ and group by that, if you throw a group by and then you call count, you don’t get an integer back anymore like you would expect. You get a hash. And even more magic is if you happen to count by a symbol or a string actually, that happens to be the name of a belongs to association on the thing that you're counting, you'll actually get ActiveRecord instances for the keys and the value will the number of that ActiveRecord.

I didn’t know that.

Yeah. It’s in my talk. You need to re-watch it. [Chuckles]

I do need to re-watch it. I've been too busy laughing over the hash thing. I did know about the hash one. Let’s take it to real here. So, this is a great example. What Ernie described here is that you would expect when you call count, you get a number. And if you bank on that in your application, you may be introducing bugs because if somebody throws group or call the group anywhere in the chain, before you get to that count, then you will not get a number. What we’re complaining about here is not that ActiveRecord has a way to count and group out the column based on by what you get. That’s not a bad thing, that’s great thing. It gives you another capability. The problem is the way you handle that is you make a different method that does that. But then, you would expect to get a hash back from instead of magically modifying count’s behavior underneath the hood.

Yeah.

That’s a place where the SQL abstraction is leaking out into the Ruby API. Polymorphism, in object-oriented terms, doesn’t mean that you can break the contract of the superclasses method, the Liskov Substitution.

Yeah, the Liskov Substitution Principle.

The return type of the subclasses method has to be substitutable for the return type of the superclasses method that you're overriding.

Right. And Ernie has a great point about this being a much bigger deal now in 3 because now, ActiveRecord is built and we are encouraged to use it in such a way that most queries are like half queries in progress. There’s this relation concept. And then, you tack on a couple of more things on it and fire it off. And if you get one of these half queries in progress, how can you safely call that count on it? You can't. I mean, you don’t know. Was group called? Am I going to get a hash here? Am I going to get an integer? Your only choice is to fire it off and then type-check against the result, I guess.

To be fair, there is another work around. You can say ‘except group’ and that’s actually chainable as well. The same way I think Active Support adds on a hash where you can say #except and you give it the number of keys you don’t actually want. You can do that on a relation. You can give it the specific things you don’t want. So, you could say ‘except group count’ but then, you're changing the meaning of the query because that group may have caused certain constraints to be observed that aren’t going to be observed anymore based on the group not being there. So, you're really changing the meaning of the query beyond just what you think you're doing. You're thinking the quick solution is, “Oh, group is throwing my count off. So, I’ll do an except...” No, something else is wrong.

I think that’s kind of the whole point. These cases like the Callbacks with the incredibly complicated implementation and the count when they could have just defined another method, it gives you some kind of a group cache kind of thing, whatever. There are simple ways to do these things that maybe they -- in the case of count, they definitely handle 100% of the cases. And in the Callbacks, I don’t know. I’d like to see a couple of the edge cases that the crazy complicated Callback queue handles. But I’d be hard presses on those. No matter what, they're definitely easier to reason about in the way and that’s got to be worth something over the ridiculous magic.

If you look at the execute group calculation method in ActiveRecord, even just the length of that method alone should tell you that there’s a lot of magic going on there that probably shouldn’t be happening. I think it took about two slides worth to scroll through whenever I talked about it into my talk and it’s just like that there should be the first sign that something hasn’t been fully thought out. I think going really back to what Josh was talking about. There has to be a choice in how the abstraction leaks. And I just think a lot of the times, users have said, “I want to contribute to ActiveRecord and I use it in this way. Therefore, I think ActiveRecord should share my opinions.” And I don’t think we have always done a great job of pushing back whenever some of those opinions -- for instance, here’s one I didn’t cover in the talk. Can we talk for a minute about accepts_nested_attributes_for?

Sure.

[Chuckles] So, here’s one of the many things about accepts_nested_attributes_for. It has side effects like enabling auto save on any associations that you use it on which by the way, if you go back and look, auto save was added explicitly to allow for accepts_nested_attributes_for. They came in on the same commit. It adds handling of virtual attributes like _destroy on these associations. And all that stuff happens at the model layer and really, it’s for essentially convenience at the view layer. Between the auto save association and nested attributes code, you're looking at around 350 lines of code. If you think about it, I kind of understand why it was implemented the way it did, what it’s doing is it’s trying to make you think about your associations as though they are actually attributes on the parent model that they're on. And so, that if you save the parent, that all of its associations because they are really just attributes of the parent anyway. So, it’s at least consistent in that sense. But it’s misdirection and it’s guaranteed to screw you over at some point. And it comes specifically from that impedance mismatch that Josh was talking about.

Oh, yeah. Ernie, I don’t know how, if you were around paying attention to the discussion that resulted in that feature and in that commit. Were you around?

I was around but not paying attention at that point.

I was actually involved in that feature development a little bit. It was Ryan Bates was one of the people who got that started. I forget everyone who was involved. It was a number of years ago. But the solution that came up that got committed was one that I wasn’t entirely happy with. I don’t think anyone was entirely happy with it so I guess you’ll call it a successful compromise. But I ended up never really using that feature and I was one of the people who was very excited about it because we had built something that did essentially the same thing in our application and it seemed to work really well. But looking back on what we have built for our application, our application was a pretty thin CRUD layer on top of the database. So, being able to do accepts_nested_attributes in our parent models, what was in the child models hat was the nested attributes was all very simple and there was very little of the domain layer that was exposed there. It was all just persistence. And when you're dealing with things at the persistence level which is what ActiveRecord is really meant to do, then that was really simple and straightforward. But when you start getting into, “Oh, we have a more sophisticated domain model,” and it doesn’t necessarily always map to persistence in a very straightforward way, that’s when you start getting in trouble.

Right. And whenever you look at the way that validations, for instance, factor into that auto save, you get a failure because one of the many things that’s associated failed to validate. Anytime you start tacking much logic at all on those other objects, you're going to run into pain. And I think that now, we all look back and go, “Ha! We should have just used form objects.” But we weren’t thinking that way a lot of times in the real world at that ecosystem at that point. I think that we have a ton now of patterns that we’ve all developed to work around some of ActiveRecord’s issues. And a lot of what you see is that we really only need, really need a fraction of what ActiveRecord is really trying to do. And so, we essentially -- I mean, I've seen some of these patterns. People are implementing a repository pattern where they're using ActiveRecord on the backend as just a way to retrieve the data. We end up relegating our ActiveRecord objects to almost essentially a row data gateway.

That’s interesting.

Yeah. I agree with that analysis. That’s kind of sad because ActiveRecord is more powerful than row data gateway. It can actually do a lot for you in that pattern. It feels like the ActiveRecord library is over reaching the pattern. What does it say? Its reach exceeds its grasp? And that means that it actually falls down a lot in doing things that it should be able to do pretty well.

I'm kind of hopeful that the Ruby Object Mapper which is Data Mapper 2, it got renamed to Ruby Object Mapper. I'm kind of hopeful that it will address some of that because in the Patterns book, we’ve been discussing here, it talks about how when you're out of what ActiveRecord can handle gracefully, then you need Data Mapper and that’s what the Ruby Object Mapper is finally can be the first pure Data Mapper implementation.

There goes my pick. [Laughs]

Oh, sorry. [Laughs]

No, it’s okay.

What I was thinking is its kind of point of view seems to be like, do the ActiveRecord-ish stuff when it makes it sense; but then, when it doesn’t give you a generic interface to control the mapping however you need. And to me, something accepts_nested_attributes_for is exactly that case where you bring in a bunch of objects, an album, and all the songs on it or whatever and you need to say how that is mapped to the database. It seems like it’s targeted at specifically that kind of problem.

What's the intervention? What can we do about all of these things?

The first thing that I hope that people are doing and the fact that you all invited me to talk with you is a great first step. I want people to talk about this stuff. It’s crazy that I can post something to the Rails Core list, talking about some of the -- I think in particular, not too long ago, I was a little disappointed. Let me first also state that I really like all the people that contribute to Rails. So, I don’t want to come off as critical of some person’s feature that they’ve added. But when we added to Rails 4, the chainable not, like where.not. First off, are you guys familiar with this feature in Rails

4?

No, I haven't seen it yet.

Okay.

Can you just say it again?

Sure, the where.not functionality in particular the where chain implementation. There's a feature now in Rails 4 where if you want to negate the criteria, like you want to use the hash syntax that you would pass in the where. So, you would say like, where id:3. And then, you want to instead find everything except the thing that has id:3. You can say where.not, and then to not, you would pass the hash of the various chunks. And you have to say not first, and then you actually add the hash after that. And the way it’s implemented is, let’s see. So, there's a where chain that actually is an object that kind of acts a little bit like a normal relation but has a very tiny subset of the methods that would be needed to do that. I'm actually going to find it real quick because it’s an interesting implementation.

So now, where sometimes will take no arguments, right?

Correct.

[Laughs] And it defaults in that case. Basically, if you look at the implementation of where, it used to say where ops and then rest, it would capture like other arguments because sometimes, people would pass in essentially an array of things but not use an array wrapper. And so, now ops defaults to a symbol named chain and if ops=chain, then it instantiates a new where chain. And the ‘where chain’ is somewhat duck typed to act like a relation but not so much because really it has just one method on it, not. And then, that not takes whatever values you pass through it, actually calls -and this is when it gets really crazy, it sends to a private method on the scope that actually called the not to begin with. And this is where you can follow along in relation query methods, a line 37 in the current version of the Rails source. And it actually sends to Buildware with same options that you had just passed. So, it gets back then the new where value that was being built by that Buildware. And then it goes through and looks at that relation and says, “Okay. So, if it’s an Arel::Nodes in, it changes it into a not in. If it’s an Arel::Nodes quality, it changes it to a not equal. If it’s a string, it wraps it into a not,” and so forth. And so, it does its best to try to negate the things that you had passed in. But by using the functionality that was on the -- and the whole thing is just really kind of strange to me the way that it was implemented. And so, I raised the question because when t I had first written MetaWhere and then later superseceded that with Squeel, it all came from a discussion I had had with lifo of lifo/docrails. And in particular, this was three years ago, I guess at this point. He had actually been working on something that was a little like MetaWhere, just a proof of concept and I think he called it super condition. And the whole general idea was like he was monkey patching symbol to do the stuff. When I was talking to him, he said, “What we really want to do is kind of let a few things flourish as plug-ins and see which one really gets the more traction seems to fit. And then we’ll go ahead and roll that into core.” When I first embarched on this path, I thought, “I won't have to be the sole maintainer of this thing forever. Sooner or later, something will get pulled into the Rails Core that will be awesome.” And this will seem like the 3.0 days when 3.0 beta was in play when I was working on this. And fast forward a number of years at this point, and we still haven't really exposed that much functionality, that much access to the Arel underpinnings of ActiveRecord. And so, I was asking, “Look, obviously, I have skin in the game. I don’t necessarily care what we choose. But wouldn’t it make sense to pick something at this point? Wouldn’t it make sense to do something that covers more than just not and doesn’t have this kind of strange implementation?” I had submitted a pull request to remove this extra class out of the mix because it was only going to be used for not anyway. So I just said, “Why not just have a where_not method if that’s the case?” And basically, it got shut down. I don’t understand. As far as I can tell, if there was a plug-in for it, it didn’t really receive much testing, it didn’t really receive much day to day use. Meanwhile, the stuff that I've written is getting used by thousands of people. It’s got lots of watchers, it’s got 250,000 installs or something on Ruby Gems. I'm not saying it’s ‘the way’ to do it. In fact, like I said, I think I'm going the wrong way in certain ways with Squeel at this point, too much magic. But I certainly would have expected that we wouldn’t just roll something out that nobody really had talked about and it shows up one day. And it still doesn’t fill that gap that allows you to access more of the Arel awesomeness that’s underneath ActiveRecord.

And yet, we have Array.10th. [Laughter]

Right.

We have Array.42nd.

No, no, no. It’s 42, I'm sorry.

Oh sorry, 42.

Which actually returns the 41st element. [Chuckles]

Right. [Laughs]

So, I raised this stuff only because I want people to talk. I don’t think that we should go 3+ years and not really have significant conversations around this. And when I mentioned it on Rails Core list, nobody really chimes in, nobody has an interest in talking about this. That’s how we intervene.

So, the intervention, if you want to get a little clear about this is like the start of your 12 step process, for some people. Like the first step is admitting you have a problem.

We’re good. We can move on.

Yeah. Well, we’ve admitted it. [Laughter]

The question is when do you get the Rails Core to admit there is a problem?

[Chuckles] I think there are a number of people on Rails Core that share concerns like that. I talked to one member who I don’t feel comfortable naming without his permission. At one point, at Ruby Conf actually, I said, “The more I hack on ActiveRecord internals, the more I try to hack around things and so forth, the more discouraged I get about ActiveRecord.” And he just said, “Yeah. I hear what you're saying.” [Laughs] So, I think a lot of people feel that way. But it’s sort of again, one of those things where the user base at this point is so huge that the amount of pain that you're going to cause people by changing some things or really removing things that should have never gone in, to begin with, is so great that I think people are just kind of telling the line at this point.

What's that saying? The best time to plants trees is 20 years ago or right now.

[Chuckles] Right. So, that’s kind of where I am with it. It’s going to pull the Band-Aid off, it’s going to hurt. But let’s just do it. I don’t know that you're going to see that. Ruby Object Mapper may start to provide some impetus for this.

Here’s the thing that I've been thinking about this. And that’s ActiveRecord was the only game in town for a long time. That was just the Object-Relational Mapping system that people used in Rails and it was the most mature one in Ruby, so everybody used it and everybody contributed to it. And like you said, all opinions are not created equal but a lot of people’s opinions got tacked in there. And it became this one-size-fits-all solution or at least, one-size-tries-to-fit-all solution. But if you look at Martin Fowler’s book, Patterns of Enterprise Application Architecture, if I said that right, he has set a bunch of different patterns to choose from for doing Object-Relational Mapping. ActiveRecord is just one of them. Data Mapper is another. And what I was trying to say earlier is that when you try and make one of these patterns into a one-size-fits-all solution, you actually hamper it from doing the thing that it’s good at well.

I agree.

And that was actually my biggest concern about the Data Mapper 1 project, was that rather than just trying to be a really good Data Mapper, they were also trying to have this compatibility feature where you could use it very much like ActiveRecord.

To be clear here, Data Mapper 1 was not an implementation of the Data Mapper pattern.

Right.

It’s an implementation of the ActiveRecord pattern with some kind of thing with Data Mapper thrown in and they have since realized that was foolish and didn’t do it right. And now, they're basically reversing that. In ROM, it is an implementation of the Data Mapper pattern with the niceties from ActiveRecord drawing in where it makes sense.

That’s good. But the thing that I would suggest as the most profitable way to pursue this stuff is, let things be what they're intended to be and not try to be one-size-fits-all and have more targeted solutions. Okay, ActiveRecord is really about ActiveRecord. It’s not about doing all the Data Mapper stuff of figuring out how to map your object on the multiple tables at the same time. But the problem that you got on the other side of that is that the transition from one pattern to another is often really hard.

Right. Where you get so far and you're like, “Ohhh…” It turns out ActiveRecord doesn’t meet our needs anymore so then your only option is to rip out 100% of the database code… [Laughter]

Yes.

Can I say too that I think this is probably a great illustration of why it’s not a great idea to name your gems after a design pattern directly? You lock yourself into one perception of what it is that you're building that may or may not continue to be accurate. It should be more about the API you're looking to expose rather than the underlying implementation. And I think sometimes we muddy the waters by naming a gem after the design pattern. I actually ran this talk, the Intervention for ActiveRecord talk by both Dan and Piotr from the Data Mapper, now ROM, team. And one of the things that they're doing that’s great, not just renaming it, [chuckles] but also they're building it up from a bunch of smaller primitive pieces. And Dan said at the time that he hopes to be able to extend the life longer than a single monolithic thing like ActiveRecord because it’s almost like a framework in and of itself to build the ORM that you want.

I think it’s great. We’ve even had some of the smaller pieces have already been picks by us and stuff because the smaller pieces are actually useful in their own right. So, it’s very interesting. Alright. So, we’ve been talking about this stuff for about an hour. Anything else before we move on to picks?

I've talked plenty. [Laughs]

[Laughs] Thanks, Ernie. It was very interesting and informative and fun for us to spend tons of time being bitten by ActiveRecord [inaudible]. JOSH: We should do one of these on Active Support sometime.

Yes.

Definitely. I actually want to do a talk about Active Support next in the vein of this one, I think.

That’s great.

Have you submitted to GoGaRuCo yet?

I haven't yet.

Just saying. [Laughter]

And Josh strums up some business. Alright Josh, give us your picks.

I really like that ROM-RB project that James already started to…[Chuckles] ROM-RB is what they turned Data Mapper into, right? Or Data Mapper 2?

Yes. It’s Data Mapper 2 renamed.

So, GitHub.com/rom-rb. It has all the stuff in there. I'm actually pretty excited about where that’s going now. Let me see what else I have here. Rubinius, I know we’ve picked it before but I just think it’s awesome that so much of the mysteries of what goes on inside MRI, the C-Ruby implementation, are more easily understood if you look at the Ruby source code for Rubinius. Just this morning, I woke up and I took a look at a little chat between Eric Hodel and Aaron Patterson and James talking about some stuff in Struct. And the Struct.C source file in MRI is over a thousand lines long, I think, something like that. Yeah, it’s around a thousand lines long. And then, if you look at the equivalent file in Rubinius, it’s about 200 lines long.

See, you can read it.

Yeah. [Chuckles] It’s amazing. I just want to shout out to Rubinius and say, if you're ever just trying to figure out what the heck is going on in C-Ruby, take a look at the Rubinius source code and it’s oftentimes a much easier way to go if you're not a real C programmer. I didn’t get my picks into my list here so I think that’s it for me this week.

Alright. Katrina, what have you got?

Oh, wait! Actually, I do have another pick. It’s a non-programming pick. Here in San Francisco, we have a place called The Exploratorium. I like to call it Nerdvana. [Chuckles] It’s an interactive science museum, is the best way to put it. If you live in San Francisco, go visit it. If you're visiting San Francisco, it’s now much easier to visit. They just moved from their location that was sort of near Golden Gate Bridge and now, it’s sort of near the Bay Bridge. It’s down on the Embarcadero. So, it’s right by all of the easy-to-get-to-tourists parts of town. It’s a great place to spend an afternoon especially if you have kids. They have a lot of really hands-on interactive stuff that’s amazingly cool.

That’s so cool that you mentioned that because I've actually been reading a travel book about San Francisco since I'm coming out for the first time ever. And I read over it and I was like, “Wow! That sounds kind of cool.”

Yes. And if you're grown-up and you don’t like kids, the first Thursday evening of the month, they have an adult swim night. So, that’s it for me.

Katrina, what have you got for us?

When I last upgraded my operating system on a Mac, they changed the way Spaces works. So, you just get the sort of endless list that you have to scroll sideways through. And it threw off my workflow pretty significantly. A few days ago, I discovered an app called Total Spaces by a company named BinaryAge. I was actually talking to the founder of this company in Sweden. He actually injects code into the OS to make Spaces right again. And it’s really, really nice. So, I've been using it for, I guess, three days. And I feel like I'm getting my workflow back to where it used to be which is very helpful. Total Spaces by BinaryAge. My second pick is a person. I've been working on a talk called Hacking Passion for months. And I presented it in Sweden and I have had a ton of help editing and proofing this talk by Sara Blackthorne. Sara is a writer, an editor, and also a budding developer. She’s doing a lot of Raspberry Pi stuff and some Ruby stuff and tutorials for helping kids get into programming on the Raspberry Pi. If you're doing anything - talks, slides, blog posts - talk to Sara Blackthorne about maybe having her help you get your thoughts in order and get the flow of whatever it is you're presenting. She’s very, very good. That’s all I've got.

I totally agree with you on the Spaces thing. I used to have four Spaces because then, it was only one keystroke to get to any Space no matter what Space I was in. You could either go left or right or up or down.

Exactly.

Yeah, because they were spatial. But once they line them all up in a fine line, [expression]. Okay. My own picks. For technology pick, Neal Ford had this great blog post about technology radars and what they are and why you should have one, your company and both like you. So, it’s a really interesting read and actually something I hadn’t considered. And he gets the idea, of course, from ThoughtWorks which does maintain a technology radar and that’s linked to in his post and that was great to go read too to see what kind of things are currently on ThoughtWorks technology radar and what they're trying to aim for and why right now. I enjoyed that and it’s worth a look. Also, I saw some positive tweets about my pick of the Civil Wars recently. The music my brother turned me onto. Emboldened by that, I will give you some other music he’s got me listening to. This one is Chris Thile. He’s a really good instrumental/singer/songwriter kind of guy. He’s primarily Mandolin, I think, but look through his stuff. When you do, go to the older stuff and it’s easy to tell because a lot of his CDs have pictures of him on the front and he started way when he was really young. You can literally just go back through the CDs as he ages. But if you go to some of the older stuff, especially ‘Not All Who Wander Are Lost’ is a great instrumental album. The newer stuff tends to be more sing-y and lyrical. But I tend to enjoy the other stuff. The other to not miss is his album called ‘The Goat Rodeo Sessions’ which he did with three other artists including Yo-Yo Ma. And that’s probably some of the best instrumental music -- mostly instrumental, just two songs were sing-y, that I've ever heard as really great stuff. So, check out Chris Thile. Those are my picks. Ernie, what have you got for us?

I have two picks. First one is a book that honestly, I went back through the picks list and was shocked it hadn’t been picked yet because it was just so much fun. And it’s also by an Ernie, Ernie Cline’s Ready Player One. If you haven't read it, it’s a really quick read. It’s full of a lot of 80’s nostalgia that frankly, as a nerd who grew up then, a lot of references to music, arcades. It’s about basically a future where there's Oasis which is basically a giant MMO that everybody plugs into. And the guy who developed it was a huge fan of the 80’s. And so, he created this kind of game within a game that would supposedly turn over some fabulous wealth or ownership of his company. I forgot specific details. If somebody found his treasure, then he basically made it to the end of the game. And there was like a global leader board and everything else. Anyway, what it ended up resulting in is a future where everyone was obsessed with the 80’s because it was what would set you up for life if you manage to win this game. It’s just a lot of fun, tons of things that will just have you kind of nodding along. As I understand, it’s his first book. And as a first effort particularly, it’s a really good effort. It’s a lot of fun. The second pick is a tool called Sqwiggle that I've had a lot of fun playing around with lately. And if you haven't tried it out, I’d recommend it. It’s actually a tool that’s designed to kind of create an office environment for people that aren’t in an office. For me, as a remote worker, I've actually had my whole team using it the past couple of weeks. Sqwiggle is really cool because it actually gives you this every ten seconds kind of update that somebody’s in front of their computer. You can see a black and white photo of them that just updates. And essentially, whenever you click on that photo, you're immediately connected to them in a one-onone video chat. So, it gives you that kind of somebody popping into your office feeling, kind of removes some of the barriers to communication that can happen on a distributed team. Certainly, that requires discipline because you need to know that you might be disrupting flow. But there's also an aspect of you can kind of tell whether or not somebody is super busy or not just by glancing at the look on their face or whatever.

“Stop playing with that toy on your desk.”

Yeah. It’s totally awesome. It’s definitely got some things to work out, certainly for a larger company like Living Social, my employer, which I should have plugged. And now, I am. We probably wouldn’t be able to use it as is. But for smaller teams, it is phenomenal and I wish those guys the best of luck because it’s a great start to that product.

Awesome. Ernie, thanks again for coming to talk to us and telling us about all the craziness inside of ActiveRecord. It was very enlightening.

It was a lot of fun.

Quick question, Ernie. I know I bugged you about submitting to GoGaRuCo. Do you have any other speaking gigs coming up?

I do. I'm actually going to give an updated version, at least this is what the current plan is. I'm going to give an updated version of the ActiveRecord talk. I'm sure things are going to change by the time I give it at Rails Club in Moscow in September.

Oh, wow!

They invited me about a week after RailsConf was over. I was like, “Yes, I would love to go to Moscow. You want to pay? Sure! Phenomenal!” [Laughs] So, I'm really excited about that. I haven't planned anything else yet this year. So, we’ll see what happens.

Okay, great.

Don’t forget that all of us Rogues are going out to Lone Star in July. So, come down to Austin and see us there, say ‘hello’ and catch our live show down there. And I think that’s going to wrap it up for this week. Thanks everyone for listening and we will see you all next week.

Don’t forget, Book Club Book next week.

That’s right. Book Club next week, it’s ‘Explore It!’

Elisabeth Hendrickson.

Yeah. Thank you. ‘Explore It!’ from Elisabeth Hendrickson. It’s a Prag Programmer title. So, check it out. It’s a quick read. I'm about a third way through it and I haven't spent very much time on it. It’s really good. I'm enjoying it. So, we’ll talk about it next week.