02:27 - Coding House Scholarship Winners with AJ and Aimee

 

09:48 - Ben Weintraub Introduction

10:40 - Wraithan Introduction

11:01 - Why Care About Monitoring?

13:08 - Mixedpanel

13:57 - How it Works on the Backend

17:26 - New Relic’s CEO: Lew Cirne

18:37 - How the Node Agent Works

23:27 - Deciding Which Databases to Support

26:41 - Browser Monitoring

32:54 - Using Zombie.js?

34:11 - Tree of Causality 

39:37 - Monetizing Aspect, Viewable Source/Source Available Code

47:28 - Performance

01:00:53 - New Relic

Picks

mraleph Blog (Wraithan)
v8-perf (Wraithan)
The Dear Hunter: A Night on the Town (Jamison)
React Rally (Jamison)
caddy (AJ)
Windows 10: Setup your Raspberry Pi 2 (AJ)
Remote debugging protocol (Ben)
Chrome Dev Tools Filmstrip View (Ben)

Special Guests: Ben Weintraub and Wraithan .

 

Here's what I'm going to do. I'm going to say the title of the show and I'm going to go around and introduce people!!

Hello and welcome to JavaScript Jabber. I'm your temporary host, AJ O'Neal. This is episode 166. We're discussing New Relic with Wraithan and Ben Weintraub. Also, we have Dave.

Hello.

Jamison.

Hello.

Of course, I'm coming at you live. And we have some exciting news today. Coding House, which is a code camp located a little south of San Francisco, reached out to ask if the JavaScript Jabber panelists would judge scholarship submissions for their upcoming cohort and announce the three winners. So, we've discussed it a little bit. And now Aimee and I are announcing the winning 100% scholarship and the two winning 50% scholarships.

So, interesting enough over the weekend as I was reading over this stuff and then AJ was spending some time, we converged on Thursday. And oddly enough my top three were almost the same as AJ's top three. So, this made our decision really, really easy.

Alright. So, there's a lot of different ways that you can rate these submissions, right? Because you can look at technical skill, what a person's already done, or what their ambitions are. And for me, what was really important was to see that it's somebody who is going to synergize with this opportunity, that they're going to produce more than just for themselves. And I saw that we had a number of candidates that were that way. So, things like people already having the blog, already being in the pattern of sharing knowledge, that ranked very highly for me. People who had dug in and created something even that was ugly but just that they were like, “Ah, I'm going to learn. I'm going to get in the grits of this and I'm going to learn what it is that I'm getting into.” And so, that ranked highly for me.

But the biggest thing was probably just seeing what kind of impact can I envision this person making? Is this person going to be able to reach out and teach others and bring more women into the community in a friendly and uplifting way? Or is this person going to be able to bring a unique perspective to their team and be able to problem solve in a way that might be different and might have new and interesting outcomes? And that's what went into my thought process on this. What went into yours?

So, my criteria, much the same, I wanted to see people who not only had just the basics of coding with some HTML and CSS but I wanted to see people that had already dug into some programming. So, whether it was Python or JavaScript, Ruby, it didn't matter. I wanted to see people that had done a little bit with GitHub because that showed that they had some practice with the command line. I think overall too, I was just judging the submissions in general because I think the more effort that you put into the submission showed the drive and ambition you had. And as a fellow bootcamper, I know that that's really what it takes to go the long haul. And then even after the bootcamp you're going to need that tenacity to keep going, because it's a long uphill battle.

Yeah. You're going to get discouraged. And you're going to have to just cry it out, sleep it out... [Laughter]

And go full bore the next day. It's hard.

My saying is, you have to get comfortable being uncomfortable because you're really going to be uncomfortable a lot. [Chuckles]

Alright. So with that little bit of background, maybe we'll move on. I'll give us a drum roll. [Drum roll]

So, we have two positions for people that are going to get a 50% scholarship. So, if you're in that position, one of these two people or even someone else who's unfortunately not going to win, remember there are sites like GoFundMe and Indiegogo and whatnot. You can get your friends and family involved to raise the rest of the money if that's a problem for you. So with that, we have Emily Dreisbach.

And I don't know if that's really how you say your name, but it is now.

Woohoo!

I liked Emily because of her way of explaining her thought process and problem solving, and then also seeing the work that she has done in preparation for this. And do you have anything to add to that?

Ah, that was the same thing too, just that she had done a little bit of programming. And then too, she just seemed to have a little bit of a different thought process. So, I like that.

Alright. And then our next 50% scholarship... [Drum roll]

Take it, Aimee.

Blake Gilmore. And I like Blake. Obviously, if she's already started digging into D3 then she already has her hands full and has I feel like a decent trajectory to go forward.

I really like that she's already out there in the community teaching. She wants to become a mentor. She's got what looks like a blog with lots of articles on it. So, she's definitely got some outreach. And I really thought all of those things were really great marks for someone to get this opportunity. And then our winner of the full scholarship... [Drum roll]

Berlin Sohn.

Woohoo! [Chuckles] I like Berlin obviously. Very, very cool that there is another figure skater out there. So Berlin, whenever you hear this, I want you to try to find me on Twitter or LinkedIn or something, because I think it's really, really cool that there's now another figure skater out there. But besides that, so AJ and I both really liked you. And for me, coming from skating I know the kind of passion you have to have to keep going and keep trying. You obviously put a ton of work into your submission. So, that showed that you know how to work hard. And then two, you have some technical skills already, which is going to be really important.

So, I have to say there was one little issue that almost tipped the scale in the other way. I played the mini game. And it didn't believe that green was a primary color.

Aww. [Laughs]

Now I know R, G, and B are the primary colors. I don't know what all this yellow is about.

But no, I love the little mini game in the submission.

Yeah.

And the note that I made was all caps “AWESOMESAUCE”. So, I can't even remember exactly what I was. I just remember after looking at the video and the blog post and the GitHub page that my impression was just 'awesomesauce'. And I didn't make any other note. [Chuckles] So looking back over it now to see what other things there were, but I like that Berlin put up a GitHub page. Obviously a little bit raw but has character to it. And I hope that you get to learn how to make it beautiful, too.

Yeah. I'm excited for everybody.

So, to those of you that weren't the winners in this particular way, go be winners in other ways. Go use resources that are at your disposal to raise some money so that you can help pay your way. Obviously there were a number of people that had really good applications and they'd put work into it. And we totally respect that and hope that you continue with your passion to grow and to learn, and hopefully that you'll be able to find your way to Code House as well.

Yeah, definitely keep going.

Alright. Well, thank you very much for joining us for this historic and epic event.

And now [go about your days]. Ben, Wraithan, do you guys want to introduce yourselves? Ben, go first.

Sure. So, my name's Ben Weintraub, although I like your pronunciation as well.

[Chuckles] That was from the made up country of Scandi-swedia. [Laughter]

Yeah, where I hail from.

Yes, yes.

So, we're both in Portland, Oregon. And we work at New Relic out here. I recently joined the browser monitoring team at New Relic. So, we make a little bit of JavaScript that you can plug into your pages and record JavaScript errors and information about Ajax requests and all kinds of other stuff about how your JavaScript code is running in the browser. Before that I was working on the Ruby agent here, which was the original thing that New Relic did, Rails application monitoring. And then before all of that I was working at Apple on iOS performance stuff. So, coming into this dynamic language world is a relatively new thing for me.

And I'm Wraithan. I'm an engineer on the Node.js agent. We are a module that you can install off of npm and then just require us at the top of your initializing file. And we spider out, wrap up everything that's [cool] and asynchronous, time it all, and send it back to New Relic, which is the gist of what we do. There's much more fun inside of that, though.

That sounds amazing. So, I have a really basic question. Let's say I have made my app. It does CRUD stuff. I don't know. It's a shopping cart specifically for...

Buying crud?

Expired coffee grounds.

Crud cart. [Chuckles]

Which have now become super cool again for some reason. Anyway, I have my application out there and it doesn't have any monitoring. I just check if it's up and it's up most of the time and that's fine. Why do I care about the stuff that you do? It's not super high traffic. I guess the shopping cart is a bad example because you probably care a lot about that. But maybe it's [chuckles], maybe it's something not super high traffic. It's just an app. I don't know.

A lot of our users use New Relic at a free tier because they're doing low-traffic stuff and they only really need to see a day's worth of data and just learn whether their site is up or not. And so, really New Relic can help you do the ops-y side of monitoring where you're trying to make sure your application's up and things are going well. And then it can also help you do the more of the performance side of things where you're actually digging in and trying to make your application faster. So, it caters to both crowds.

The other thing I would add to that is that even though it's not the original use case for New Relic, we have a lot of features that will allow you to get pretty interesting non-performance data out of your site as well. So, some of the stuff that you might use Google Analytics or a similar tool for, you can get out of New Relic as well. That's one of the things that the browser agent that I work on does, is track an individual event for every page view on your site. And you can record custom parameters with that event. And then you can go and use a tool called Insights that we have to filter through all of those events and slice and dice them based on the values of those parameters. So, you can use it for more than just performance analysis and knowing when your site is down. You can also do some analytics type stuff with it.

And for me, the answer to your question though is just I find all of this stuff really interesting. Even if I don't really care that much about when my site goes down, I'm still interested in what my response times look like and how many database queries am I making per average on a page and things like that.

That makes sense.

When you describe the events, how you can slice and dice data with your Insights product and stuff, this sounds a little bit to me like Mixpanel. Are they comparable?

I'm not super familiar with Mixpanel's product. But I think that yeah, some of the use cases are pretty similar.

Yeah. The browser agent gets compared against Mixpanel and it gets compared against Splunk and it gets compared against Google Analytics and a bunch of other tools like that, because it's gathering a lot of interesting session data. And then you combine that with insights and you can start to do funneling and other stuff to see how people are moving through your site. You can get some of that similar information off the server side as well. So, you can use Insights with both server and client-side stuff. And so, one of our biggest efforts in over the last year has been to add more types of events and better data on those events so that you could better use our Insights product.

So, I have an unrelated question about the backend of all this stuff. It seems like when you're doing event monitoring, that's when you need to get into all the big distributed systems stuff. Is that true? I know lots of people implement this kind of monitoring system themselves. There's stuff like StatsD and Graphite. And there are a trillion tools that you can use to roll your own stack. Can you talk a little bit about how it works on the backend for you guys?

Originally the answer to that question was very simple. [Chuckles]

And it was basically MySQL. We have... we still use MySQL for a lot of stuff. So, we have all of our time series data stored in MySQL using a ton of, I think we have... I don't know how many shards. But we have many database shards sharded by account. And we dynamically generate tables and stuff like that. So, that's for our oldest kind of data, which is time series data. And so, that's similar to what you would get out of something like StatsD where you have a count, a min, a max, a total, time. And then from that you can derive a mean. And you can derive a standard deviation, because we keep track of the sum of squares as well. Although then you have to assume that your data is normally distributed, which most things that we're interested in are not. So yeah, that's the answer for the time series data.

As of now we have a lot of other kinds of data that we collect as well. So, I guess I could break that down into roughly two other categories. One is traces. So, these are events that are... I guess you can think of them as event data but they're not as high volume as the kind of data that Insights feeds on. So, these would be things like transaction traces in the New Relic UI, which are a basically like a detailed view of a single web request. And all the database queries that happened during it. And the template rendering and everything like that. So, those are... each individual piece of data is larger. It's basically a blob storage thing. So, we're talking maybe on the order of a couple hundred K.

Upwards of a Meg or so in some of the cases with like Java agent and whatnot.

Yeah. I guess they can get...

And .NET, I think PHP.

They can get quite large because they're very detailed. Another example that would be what we call session traces in the browser product. So, this is basically a combination of... it looks very much like what you get out of Chrome Dev Tools or something like that. So, you see all the resource timing information. And then you see user interaction events like, “Oh there was a click here. And then a click handler ran for this long.” You see JavaScript errors on there. But it's over one particular browser session up to I think 10 seconds. So, that trace data is stored in a variety of different ways. Some of it's stored on S3. Some of it's stored in Cassandra. But that all depends on the individual use case and what volume of data we're talking about. Because our different products, we have a lot of different products and they have different requirements I guess.

And then the last category of data is stuff that's thrown in Insights, which is I guess is probably what you were originally asking about. [Chuckles] And so, that uses a custom database backend that was... our CEO still writes code and two and a half years ago I think, he went off and decided he was going to write a database. And so, he did an initial implementation of that. And we now have a team of people who work on that database. And yeah, it's... I don't know how many machines it’s distributed across but it's quite large.

It's colossal.

Yeah.

I don't mean to be offensive, but is your CEO a crazy person?

In a good way and a bad way. It just seems so insane to me to think, “I'm going to write a database and I'm going to use it in production for my company.”

Yeah.

Yeah. Well, and I've sat down and coded with him on Node projects and a bunch of other stuff. We have our engineering [off-site] each year. And last year we had a few drinks and then implemented some stuff to dump a ton of streaming data into that database and see how it handled it and stuff like that. So, we started pulling in the stuff you can get from the Twitter public API and all this other stuff to see what we can do. We came up with some cool demos off of that. And that was just me and the CEO hanging out. And I'm like, a leaf node in the graph of New Relic.

So, he is kind of crazy but not in a bad way.

Exactly.

Good.

And like I said, there's a whole team of people who now work on that. It's not like he's continuing to maintain that on his own while being the CEO. [Chuckles]

I don't think he really writes any code on it anymore.

He just gets real bored in a board meeting and holes up in his laptop.

He's working on [inaudible] recently.

Oh, okay.

Yeah.

So, he does do some stuff with it.

That's cool. So, I have a couple of questions about how the Node agent works. Does it just hook into the HTTP stuff built into Node?

We can start at the beginning a little bit there. The first thing it hooks into is the require. So, it basically replaces the underlying load with its own monkey-patched function. And so, it can get access to every module before it actually gets handed to your application. In this way, it's able to reach in and wrap up every method that's interesting including the core HTTP stuff as well as any database drivers that you use and frameworks that you use as well. So, it really spiders out and grabs all the data it can just by hooking into the require hook and then doing a little bit of monkeypatching here and there. But yeah, we do [inaudible] the core HTTP server and that's the only HTTP server that we support in it. There are some that are written in C out there that people use. But they're not very common. So for the most part, everyone sticks with the core stuff. All the frameworks build on top of the core stuff as well.

Yeah, I was going to say, basically if you're using a popular Node web framework it's probably using the built-in HTTP stuff.

Yeah.

Like Express or Sails or something.

Yeah, yeah. And actually, Sails is built on top of Express. So, we only instrument Express, Hapi, and Restify as far as web frameworks go. But because ActionHero, Sails, Kraken, and a number of other ones are built on top of Express, we get those ones kind of for free. They've also caused us as little bit of headache because they reached into Express in interesting ways that users don't do.

But it's just part of the fun of the job.

Can you talk a little bit more about the application-specific stuff that you do? You said you do specific things for Express. And that seems like one of the interesting differences between the... like Rails, you have Rails and you support it. And Node has a lot wider range of things that get used, for better or for worse.

Yeah.

But you do end up doing framework-specific stuff for several different frameworks, is sounds like.

Yeah. We use the core HTTP server to note when a transaction starts. And this is just a piece of WebWork. And a transaction is really just a request and response cycle in this world. And then we need to name that transaction. And in Rails, you would have used the controller name. That's the same with Django. In Java, you probably have a controller name as well. Almost everyone else has controller names. We pass anonymous functions to getters and assign them to a URL. That's the common case. And so, we have to instrument the framework in order to get the name of the route that's being used. And so, however you bind to that route is what we use, because those are unique to the route but not unique per request. So, if you have an ID parameter on your URL we use blog/:id not blog/37 for your transaction name. This is so it groups them all up by the controller action or whatever you want to call that, view, so that we can get good data for those things.

Because those are usually where the variation between requests show up.

And then does it do database monitoring stuff as well?

Mmhmm, yeah.

So, how do you that? Is it the same thing? You just hope they're using one of these supported database libraries?

Yeah, so we support almost every data store that people use in Node. We're only missing a couple of them. Like there are a few folks who are deploying on Azure and they use MS SQL. And we don't support that. And we don't support RabbitMQ. But the most common casein our world is folks who are using Mongo with Express. And we support Mongo across all the versions of the Mongo driver, starting with 0.9. And then we support Postgres and MySQL and Redis. And those are all really common. We've seen a little bit of uptake in more enterprise-y databases like Cassandra. And this is really showing that Node is being adopted in the enterprise and is becoming something that people are starting to trust on their more critical infrastructure.

And then we've also gotten a number of requests for things like Oracle support, and this is showing that a lot of these long-time Java shops are starting to pick up Node and using it with their Oracle backend that they've been using for probably 20 years at this point. So, it's cool because I can watch the ecosystem evolve and Node become more popular in a bunch of different areas, where when I started over a year ago, Node in the enterprise was like Walmart and PayPal experimenting with it. And that was it. Maybe a little bit of Yahoo switching out some of their widgets. So now, it's so many other big enterprises.

That's really interesting. It seems like it's a bit of a... who's that guy that always had to push the boulder back up the hill? Sisyphus?

Sisyphus.

Yeah. It seems kind of like a Sisyphean effort to support. There are new data stores all the time and there are new libraries for them all the time. How do you decide which ones to support? Is it just whoever bugs you the most on GitHub or something?

Yeah. It's the loudest screamers. No, we...

That is a good way to get louder screamers.

It is. It's really good. No, we internally use some tracking tools for our incoming requests. And we assign those things to accounts. And we know approximately what an account with worth. We use that sort of metric along with number of requests, because we may get a lot of requests from lower, small businesses that aren't going to add up to even one big, large customer. But having a large number of those is good for us as well, because we get more word of mouth. So, it's a combination of whether it's valuable to the business as a big money thing or valuable to the business for reputation. And also, just knowing the Node community. Mongo is extremely important even if no enterprise customer is using it at all. That is what the Node community uses on average. And we should be supporting the Node community on average.

The other thing I would say that feeds into it at least, or did when I worked on the Ruby agent is we're in a unique position of having information about what gems or npm modules large numbers of people have in their applications. And so, we can do some kind of analysis and see how popular, at least among our customers, are particular modules. And so, that can help feed into it as well. Although just because something is popular doesn't necessarily mean that people care about instrumenting it. So, you have to be a little careful there.

Yeah. But that's how we ended up instrumenting Postgres actually, is we didn't have a ton of requests for it. And we would get occasional requests from our support saying that someone's transaction was going through Postgres and some data wasn't being captured that they expected. But it was more that we just had a lot of users with Postgres in their stack. And they hadn't actually asked us for support. But we added it. And then a whole lot of people were very happy. So yeah, the data analysis on our users is really useful for us moving forward.

That's really interesting. So, you have a whole stack of stuff just by virtue of what you collect to parse through and figure out what to build next. That's cool.

Yeah, yeah. We collect every module that you have installed. This is partially or mostly for support purposes because knowing exactly what version people are on we can deduce whether a problem is in one part of the code or another [inaudible], in a lot of cases honestly. But a side effect is we have that data that we can analyze to help our customers more.

It's also presented in the UI which is actually surprisingly useful. It seems like a very basic thing of, “Oh, what versions of all these modules am I using?” But there are an unbelievable number of problems that arise because you thought you were on version X but you're actually on version Y in production. And so, it's nice to be able to see that presented in the UI as well.

Or one server out of your 35 server cluster...

Yeah.

Didn't update its packages for whatever reason and it's the one crashing all the time. But it's hard to detect that kind of thing where in the environment tab you can see that you have some version numbers that you don't expect in there.

That's cool.

Yeah.

So, we've been talking about Node and backend, database and server, HTTP request monitoring. I would love talk about your browser monitoring. So, I've been using New Relic for server-side monitoring for about three years on my app at work. It's a Python Django app. And just freaking love it. But several years ago we turned off the browser monitoring because we moved to Angular which made us pretty much a single-page app. And at the time New Relic was only doing monitoring of page load. And so, we got really weird data out of New Relic at the time because we didn't really have any more than one page load. Every user would come in to get a single page load and then they would navigate using client-side routing. And that's still the case for our app today. What does New Relic give you for that kind of single-page application?

That's actually the biggest thing that I'm going to be focusing on in the next several months.

We literally had a conversation about this, this morning or yesterday, that you're working on it.

Yeah, yeah. So, just for context here, I'm relatively new to the browser monitoring product. I started I think three weeks ago. So, I don't have all the answers for you. But yeah, I think the situation probably since you turned off the browser monitoring stuff has improved in terms of what we offer. So, the two biggest that you would get are we can track every Ajax request if you're on the browser pro. So, that can be helpful. It's sometimes hard to form a coherent narrative about what is happening, which Ajax requests are being triggered from which page loads. And you don't have anything really right now to tie it together, all of the Ajax requests for a given client-side navigation event. So, that's the problem that we're hoping to solve. But at least you can see all the Ajax requests that are happening and group them essentially based on the URL and get aggregate statistics about them.

And then the other thing that we offer is the session trace feature that I've mentioned. So, this is not aggregate data. So, you're talking about one particular browser's interaction with the page over time. But the session traces can span multiple client-side navigation events. And so, you can see a really detailed view of all the resource loads that happen when a client-side navigation event is triggered. And you can also annotate those traces with custom information. So, as an example if you wanted to be able to see session traces every time that a client-side navigation happened, there's an API that you could call in JavaScript that would say, “Hey, record an event starting at this time and ending at this time.” And that'll show up on your session trace so that you can be able to map that back and see, “Oh, okay. Client-side navigation was initiated here. And then all these resource loads happened and they took this long.” So, that's what we offer today.

As far as where we're going for the future, it is definitely our highest priority for the browser product right now, is better support for single-page apps. It turns out it's a really difficult... [Laughter]

Question to answer like, “When is a page load done?” especially if that page load is a client-side transition.

Yeah.

I'm curious.

Oh, man.

Whether you have thoughts about what you would like to see as the [inaudible].

Yeah, I have a really good thought on that. It just needs to be right. [Laughter]

Yeah, that's kind of what everyone wants.

If someone would define 'right', that would be really helpful.

Yeah. No, no, no, no, that should be easy.

We've left it as an exercise for you.

Yeah. [Chuckles] We wouldn't want to deprive you of the privilege of working through that problem.

Yeah.

But see, and I know exactly what you're talking about because we have an Angular app. And so, the page route loads really pretty quick. But then Ajax events start happening. And data comes in. And then we kill the digest and spend all this time loading tables and stuff. So, the page really isn't ready for the user. And that as near as I can tell, there's no cross-application, even within a single framework there's no way to really know when a page route, a page is fully loaded in a single-page app.

But one of the things I would love to see, and having used New Relic on the server side for so long, we are addicted to the New Relic graphs. Every time we do a deploy, we watch New Relic for a while afterward and we see that our response times go up, go down, what happened. And with Angular, what I would love to see is, did our digest times go up or down? If nothing else, that would be so cool to have New Relic chart that and have it charted per browser and per URL. It'd be really slick.

Sure, yeah. So, in the browser agent thus far we've avoided having any framework-specific instrumentation. So, we've avoided this Sisyphean task that we were talking about earlier. But there may come a time when we need to cross that bridge and start doing framework-specific stuff to provide features like that, that you're talking about.

I hope that time comes soon, because we're just going to build it. [Laughter]

Do it ourselves.

Yeah. The other thing I would say is, for the long-term the approach that we want to take to this kind of stuff is we want to build great APIs first. And then the framework-specific instrumentation, once those great APIs are in place should be really easy. It's just a matter of hooking into the right spots within the framework. I think this is a mistake that we've made in the past.

Yeah.

As a company, is we've focused too much on instrumenting individual frameworks and prioritized just getting that out the door. Which, that's important but frameworks change over time. And so, what's ultimately important is having a really solid platform that you can build upon that enables you to pick up the next great framework that comes along and just have it, be able to build support for it in a very short amount of time.

And exposing those APIs for our users as well is a big thing. Almost all of our platform, or our language agents started out with auto-instrumentation. And honestly as a company it probably made a lot more sense back then, because we really needed to be turnkey to make a name for ourselves. But now we made a name for ourselves and we can get people to use our products. It's just a matter of making them super solid. And so yeah, a lot of us on the language agent teams are reevaluating our custom instrumentation APIs to try and come up with better user-facing ones that we can then use to build our own instrumentation on top of to give better examples. And we can have a better conversation with our users because they can do stuff themselves if we don't support a framework, as well as support new frameworks quickly.

So, I got two things. I can confirm that it is a trust-buster, one who breaks up trusts. While you were talking about ways to find out if a page load is done, I don't know if this is helpful at all but there's something called Zombie.js. It uses JSDom. So, it kind of emulates the browser. But I think they have really good support for when stuff is done because they mock their own event loop. So, they just check if the event loop is empty. And if it's not empty, then there's more stuff that's going to happen and it isn't loaded yet. I have no idea if that's even possible at all in a real browser.

You wouldn't be able to get access to the event loop in a real browser but you can simulate access to the event loop by setting short timers to see if they execute really quickly or not. But that doesn't let you know that there is any work dispatched. That just lets you know if there's any work queued to be done. I don't know that... I know for a fact that V8 doesn't expose any of these things externally. It may expose it through their debugger API or their Chrome Debugger API. But I know it doesn't expose it to JS just by itself. Because otherwise I'd be using stuff like that in the Node agent as well.

Yeah, all you need to do is get all of your customers' customers to install this custom build of Chrome.

Yup.

Yeah, exactly.

Easy peasy.

That seems fine.

So, one of the approaches that we're experimenting with right now is trying to essentially maintain a tree of causality that crosses async boundaries within the browser agent. Because we already instrument all XHRs and all timers and things like that, we're sort of close to being able to do this already.

On the browser, you don't have any of the native module stuff that we have to worry about.

Right. Yeah, exactly.

Yeah. That's nice.

So, if were able to have that tree of causality, then we could do interesting things. Like say, okay, when we see for example a push state happen to do a client-side route change, let's trace back up to the user event that initiated that, say the click or the key press handler. And then let's wait until all of the subtrees under there are resolved in some way, or maybe not all of the subtrees but all of the Ajax requests say that are spawned as a result of that user event. Wait until they're all resolved and then call that the end of the page load. There are some problems with that approach. And there are a lot of tricky things to work through. But that's one of the things that we're experimenting with right now.

Yeah, the growing use of server-side events in WebSockets and other things like that make the tree of causality a little bit harder to deal with, especially in the Node world because everyone's starting to use those kinds of things now. Which is rad, it's great. Server-side events are one of my favorite things to work with when I'm working on frontend stuff. Normally, I avoid it like the plague. But if I can just create a page that can upload, update on a regular basis with server-side events, it makes me really happy.

When you talk about the tree of causality it reminds me of Todd Gardner's project or company called Track.js. You guys familiar with that?

I'm not, no.

No, no.

They do a lot of that instrumentation you're describing where they track every XHR and every timer. And they do it for the purpose of error-tracking. So, if an error happens they try to root cause the error and track every event that could have been related to that error so that it all gets bundled up into one unit. So, pretty similar but you guys are looking at it probably from a... you're probably interested in error-tracking. But that's just the tip of the iceberg, right?

Yeah, exactly. If we were able to do is, then we would definitely want to apply it to error-tracking as well, because it is really useful to be able to see. Right now we can say, “Well, an error happened and here's the backtrace that was provided by the browser for it.” But if that backtrace doesn't cross an async boundary then it's oftentimes not particularly helpful for diagnosis. But yeah.

And tracking things across asynchronous boundaries are one... it's the thing that the Node agent does. A lot of the other agents in Node land rely on native modules and hook into the JS engine to try and get some of this data where we're pure JavaScript. And so, we have to wrap up every asynchronous boundary and then use some interesting like shoving variables into closures to store state and then restore it when the callback gets called. And I know that the browser is doing a little bit of this similar stuff but they're using event emitters to do the same thing to restore state and everything. And so, it's a lot of that wrapping up all the asynchronous boundaries so we can find the cause.

It reminds me of that Google Dapper thing but on the client-side. Like where they build in tracking stuff to their RPC libraries and into all their servers so you can tell, if you make one call it caused these three other calls to these other services, then bundle them all back together.

Oh, the microservice.

Yeah.

We have stuff like that called cross-application tracing. And so, all of our platform and browser agents support it as far as I know. I'm pretty sure all of them do. Anyway...

I think so.

Yeah, and so it can...

That's neat.

It'll inject extra stuff into your headers for your external-bound requests as well as... not for Node because there are no really standard queuing libraries. But for a bunch of our other ones. Like as a Python engineer you're probably familiar with, or a Python person at least, you're probably familiar with Celery. And Celery is something we have [inaudible] MQ for which lets us pass messages through the message queue protocol and everything, and actually tie together every service that you have. And we've been building up some really cool visualizations around that.

Yeah, so you can see like, this web request spawned this message on Celery.

Yeah.

And that spawned three other web requests to some backend services. You see that whole tree of things.

This is important a lot for our bigger Node customers, because they're building out these frontends. And where they're using the browser agent. And then they want, they're doing a single-page web app so they need to track a request from the browser to Node, which is probably going to call out to a Java service because it's going to multiplex out the call so that you don't have to do seven calls which are really expensive over a user's network. And then you do all the stuff. You get it all back. You collate the data. And so, you see those and then from the Java stuff you see that you hit a bunch of different databases. And then this creates a whole web that you can actually analyze. And we're building that stuff out to be really helpful to our ops engineers that use our products. This way you can see broken, things that are getting slowed down at certain areas and everything. So, you can more quickly identify what the problem is when requests are getting slower, especially across a bigger microservice distributed architecture.

I have a total change of subject question. So, you have this npm module that's up on GitHub, right? How do you charge money for that? Is it all just in the servers? Could someone use your code to do something useful for themselves without implementing your backend?

There are really interesting things in our license that say that you can't use the Node agent to basically power another APM product. Our code is up on GitHub, but it is not OSI. It's not open source as far as the definition most people would use. But the source is out there that people can see. And you can just install this and run it. But unless you have a license key it's not going to work. And if you do have a license key then you're either on a free level or you're paying us to get the data. You can't actually use this to push into your own service.

Sure.

Basically what we charge for is the service of storing and presenting your data, rather than the agent itself.

Sure. So, what's the role of having it on GitHub then, if technically you can't use it unless you're paying New Relic? Is it just to build trust? Like, “Hey, look see. We're not doing anything weird”?

Trust is a big component of it. One of the original developers of the Node agent, Forrest Norvell who's now an npm dev and a friend of mine, he's the person who's doing a lot of the work on the npm CLI, he pushed forward to be open sourced. And it falls in the same vein as the Ruby agent was open sourced as well. And replace open source with sources viewable on GitHub, because...

Source available.

Yeah, source available. And a lot of it is because Node users and Ruby users are very, very used to be able to dive into the source of their gems or packages that they're using and see what's going on if something goes bad. Or, just read it to trust it or a bunch of other reasons. There's a bunch of stuff in the Node agent that isn't super specific to tracing in Node. And we'll hopefully be breaking more of those things out into their own modules. And then those will be truly open source. But even still, if you're not using this to push to an APM setup, you can use our code in various ways. You can't use it to push to like our competitor or to build a competitor.

So, I want to say I think that that is really cool, because I totally recognize that not everything needs to be Apache licensed and open source. There's work that you've done that you're proud of that you want to reap the benefits of. And it's really cool to take that compromise stance of, “Hey, we want to have trust. We want you to be able to tinker. We want you to be able to find bugs and fix them and help us out. And we want that respect of this is a product that has a lot of our intellectual property in it.” So, that's cool.

Yeah. I'm an ex-Mozilla dev. I'm very used to the super hyper open source world of Mozilla where literally almost nothing is closed source. The stuff that I had access to that was closed source were interview questions and ops stuff, and that was it. And ops stuff was actually mostly just the passwords and were other things like that, keys that had to be distributed. The rest of it, like a lot of our Puppet scripts and other things like that, or Chef (I forget what we use there), were all open source as well. Yeah, so I'm coming from a world of very, very open source. So, this kind of open source stuff, it itches me in the wrong way a little bit. But at the same time, it's much better than hyper closed source, right?

Being able to at least share my source with others and I'm able to actually spin up open source projects and get them approved, that's a thing we can do on our team or at our company which is really nice. We have a strong relationship with open source, as well as contributing back. I contribute to the Node project. I help out by running tests. And I've actually written a number of unit tests for the Node project itself and contributed to other libraries that we instrument. Because as an instrumentation engineer we find bugs in people's code. And we have to be almost as knowledgeable about their code as they are.

With what you said, it kind of irks you a little bit. I'm just curious. You recognize the value in getting paid, right?

Yes.

So, this is more of a philosophical can of worms to open.

It's a very philosophical can of worms. [Chuckles]

What would be the ideal solution? Is it possible to have a world where you open source all of your tools and you still have a business model?

There are people who do it. It is hard and it requires turning a lot of things on the head. New Relic is really good at what it does in the current state of the world, because we host a good number of servers and we keep them all up. And this is what we do. It's the platform as a service part that is really important to our users. The code becomes less important in that case. But at the same time, I understand that it can be hard to monetize stuff when you're open sourcing too much of your gear. I don't have a solution to it. I am just an open source hippie who would rather have everything open source if I could. I recognize that monetizing that can be difficult.

Yeah, I'm definitely an open source hippie a little bit, too. But I also sometimes would like to make more money off of some of the work I've done. So, I see it. I like the way that New Relic is doing it because it validates an idea that I think is still a good wholesome idea, too.

Yeah.

The other thing that's a more pragmatic thing, especially with the Ruby and Python and Node agents, because code in those languages is distributed primarily as source code, even if we wanted to go to great lengths to make sure that people couldn't read our source code, there's not really much we could do. We could re-implement a bunch of stuff in native code. But that's expensive and has a lot of drawbacks in its own. And so, on some level it's like yeah, we could go through effort to hide this but it's almost even easier to put it up on GitHub.

Because then, the thing that I really like about it is that if you're having a conversation with a customer about a problem that they're having, it's like now you have a shared space where you can reference particular chunks of code. And you can point them that, “Oh, looked. We fixed the issue that you were having.” Some customers you tell them, “We fixed the issues you were having,” and they're like, “Great. I'll go update.” Other customers really, really want to see, “Show me the code. Show me the line where you fixed it so that I can be sure that it's fixed before I deploy it.” and now we can easily do that. We can send them a link to the specific line in our code where we made some change. And they can see the Git commit where that change was introduced. So, I think it's really valuable just from the point of view of having a shared reference point that we can use when talking to customers.

And to the start of Ben's point, I spent the first number of years as software developer hacking games. Like literally just decompiling stuff and watching memory. So, if you can do that to games where they're really trying to hide that kind of stuff, you can do this to folks who aren't actually trying to hide it that hard. So yeah, open source or at least viewable source makes a lot more sense.

Yeah, I can tell you there's been a couple of times where it's been a nuisance to do what I wanted to do because it was like a Caesar cipher type crap in the code where it's like there's a secret in there. And somewhere there's a function that's minified that just puts random bits in between. It's annoying but it doesn't stop you from being able to do what you need to do.

Exactly, yeah. And when you're doing that kind of stuff, you're actually slowing down the performance of your code. And we're hyper sensitive to performance. I care a lot about how V8 JITs my code and how hidden classes are generated and making sure... we have a ton of extra things in our initializers to make sure that our hidden classes stay completely stable throughout the life of the application and stuff like that. Trying to obfuscate all that stuff and then maintain our high amount of performance would just be a pain. It's not worth it. So, we didn't really too much about performance necessarily. But it's one of my biggest passions in JavaScript. And really, all the languages I've picked up over the years. I was a Python engineer for six years before I came into JavaScript. And I spent most of my time optimizing people's codes and queries. So yeah, my...

Well, I think that performance would be a good think to talk about.

For most people's code in most cases, the performance implications of what the JIT's going to do with your code and whatnot doesn't matter. Just-in-time compiler, V8 goes through two different phases. It goes through Codegen and then Crankshaft. Codegen makes your code... it generates code as fast as possible but it'll do the stupidest crap while it's doing it just to get it to the point where your code is executable. Because this is, I'm working in Node. And so, I'm thinking about this stuff as a server-side engineer. But you have to remember that V8 was completely built around browser interactions. So, people load up a page and have a couple of dozen JavaScript sources. And then they have to... then it has to compile them all and get it to execute as fast as possible.

And then the next phase is your code has been executed for a little bit. And Crankshaft manages to start picking out the pieces of your code that are being called most often. And then it tries to compile it down to something like removing the checks or taking redundant checks and moving them up into a higher block. Or inline caching property lookups and stuff like that. And so, my first pick is Mraleph's blog. His tag line is 'crazy Russian compiler engineer'. He worked on HotSpot in the JVM. And then worked on Crankshaft in V8. And then worked on the JIT in Dart. So, he's the authority on that kind of stuff. Yeah, it's M-R-A-L-E dot P-H, is his blog. And I recommend reading literally everything he's ever written and watching all of his talks.

You'll be good friends with him after that, virtual friends.

I've only talked with him via Twitter. So, that'd be cool. But yeah, and then my other pick is my friend Thorsten does a lot of performance analysis in V8 as well. And so, he put together a whole repo and he's looking for more contributors and trying to find more people who are really interested in performance in Node. So, his setup is thlorenz.com/v8-perf. And 'Thorsten v8-perf', if you just search that you'll be able to find his stuff. It's just a whole repo of all sorts of really cool stuff he's found out about V8 in Node. And he's not restricting himself just to Node V8 but also doing the browser stuff as well. And it goes into crazy depth at a couple of different layers. But it's all from the practical point of view. Mraleph's blog is very much about from the engine implementer's point of view. And Thorsten's stuff is more about the user's point of view. And those are my two picks, because performance in Node is really rad.

I don't know how you say the first guy's name. Is it really Mr. Aleph? I guess I've never said it out loud. Okay.

Yeah. Mr. Aleph.

I feel like it's so formal.

Right? Well, try and pronounce his... I can't pronounce his name. I'm not even going to try and pronounce his real name.

Yeah. He has a really great series on benchmarks and how those benchmarks people put up on, what's that benchmarks site? I forgot the name of it now.

JsPerf?

Yeah, JsPerf, as just so bogus.

Yup.

Those are some of my favorite things.

And then he branches off of that and goes into the hidden class discussion and how JsPerf can either exasperate the problem or make it so that hidden classes are working even better than they should, and a bunch of other things. It's really cool.

Yeah, it's kind of fun to look at. But I don't believe that microbenchmarks are a good idea. I think that they're more dangerous. 2015 news alert! Benchmarks considered harmful. [Chuckles]

Yeah, yeah, exactly.

Because people start to bicker and argue about things that just don't matter. If doesn't matter if you plus-plus before or after. In fact, you should just do plus-equals one so it's more readable to newbies. Because that's 90% of the JavaScript community, is newbies. So, just write code that newbies can read. And if you're New Relic or if you're on the Node core team, when you're that cool, then go back and be like, “Oh, it turns out due to some weird thing V8, if I do minus two and then plus three, this is faster than adding five directly.” [Chuckles]

“And then subtracting two from it,” or whatever it is.

Yeah no, I totally agree. Microbenchmarks are really good if you're trying to do exactly what a microbenchmark does: test what something does when it is the innermost loop of a very, very hot function, like hot as in accessed very often. If you have something core to your application that is going to iterate over 100,000 records for Mongo and is going to do three operations to it, toss that into JsPerf. That's an ideal case to use JsPerf, to analyze what's going to happen. But if you're just serving up requests for your shopping cart where you're selling crud and whatnot, it's just [chuckles] not, it doesn't matter. None of those stuff matters. Your database queries are going to blow everything else out of the water. So, just optimize those.

And image manipulation, too. That's a place in the browser where you actually do a million things, literally a million things.

Yeah, exactly.

Yeah.

And if you're in the browser and you're trying to do things like you're playing with Canvas or WebGL, then use the profiler built into Chrome, right? Or Firefox. The profilers there are great. There's no reason to go use JSBench or JsPerf or one of the other benchmarking tools to microbenchmark a piece of code. Because you can just run the code and see what the hottest thing is. And you can even access that stuff from Node by, you can either use perf in Linux land. What his face, Thorsten, in v8-perf actually talked about how to use perf, which is just a Linux command line tool to generate flame graphs and it's fantastic. As well as you can hook into using the Chrome Debugging API from Node using Node Inspector. And now you can actually see what's going on and be able to see what the slowest part of your code is without having to run silly microbenchmarks.

Alright. And with that rant, let's move on into the other picks. Jamison, what are you picking this week?

My second favorite band ever, The Dear Hunter, released a track from their new upcoming album. It's called 'A Night on the Town'. And it's amazing. It's so good. They do melodramatic prog rock, light, and yeah, it's really good. So, check it out.

React Rally. If you use React.js or are interested in using it, or React Native or GraphQL or any of the cool things associated with it, I'm helping organize a conference. August 24th to the 25th. If you go to ReactRally.com you'll be able to buy tickets by the time this podcast comes out. And we'd love to see you there. Yeah, that's all I got.

Well, I got a couple of things to pick. One is Caddy. It is a web server written in Go. And it's an

HTTP 2.0 web server. So, it's supports server push. Well, it doesn't support server push yet but it will. [Laughter]

And it's going to support Let's Encrypt. They're working on that right now. It supports the compression in the encryption. And it's just meant to be, I don't know. Imagine that you had all the... because Node is not a good frontend, right? I think we can kind of agree on that somewhat. It's just not as good at serving files and at doing some of the proxy type tasks. Whereas a compiled language is generally going to be better at that. And Go is compiled. And it's also pretty much as easy to use as JavaScript. So, it's something that I like because I see... I'm going to be writing Go so that I can write plugins for this web server so that I can do stuff that doesn't work as well if I were to do it in Node, especially on a Raspberry Pi. Because on a Raspberry Pi, Node runs a little bit slow. Go still runs pretty dang fast.

And then also speaking of Raspberry Pi, I'm going to kind of sort of pick Windows10 in that I don't actually... I haven't used it and I don't really know anything about it. But if you have a Raspberry Pi, you can go on this form, and I think I gave the right link to it, and you can get Windows 10 for Raspberry Pi. And I was totally going to go through this installation process because I even found somebody that made a tool that turns their weird image into a normal image that you can just DD onto a flash drive. And then I realized that if I had installed it I wouldn't know how to get into it because I don't think Windows has SSH. So, I didn't get quite that far. I had it all downloaded and figured out. And then I was like, “Oh. I'm not hooking this thing up to a monitor.” But...

Well, you could hook it up to a monitor for 20 minutes and put PuTTY SSH on there. And then you can have an SSH thing.

Well, I realized I had no idea what I would do with it. Because even if I did SSH into it, I would still have to install all the Unix tools to be able to do anything with it. And then yeah...

PowerShell. [Laughs]

Sure yeah, PowerShell.

[Inaudible] Bash?

I remember seeing it at...

Old DOS games on it. [Chuckles]

Yeah.

Yeah, DOS games. Use a nice cutting edge 64-bit operating system to simulate an old 8-bit CPU. [Laughs]

Well, if you're into that sort of thing, there's also Tiny... I forget what the name of the company is, but it's tiny-something. And they just came out with TinyScreen. And so now, you can create your own Apple Watch and you can run Nintendo games on it. And maybe even Super Nintendo games.

Oh, and I also wanted to mention there's a bunch of competitors to Raspberry Pi that I didn't realize were out there. There's Orange Pi and Banana Pi. And particularly one that looks pretty good is ODROID. There's an ODROID-U3 and an ODROID-C1. So, if you can't get your hands on the $35 Raspberry Pi 2, check into some of these other things. Particularly, I think it was the ODROID-C1, actually has more memory and is more powerful than the Raspberry Pi 2. And it's at the same price. Now, I don't know if user community and documentation and all that's up to snuff.

So, it might be worth paying $20 more to get your Raspberry Pi now on Amazon. But it's an option.

Well, in that vein, so the ODROID-U3 is really nice. It's a quad core ARM testing processor. So, I use one at home to test my builds on ARM to make sure that my native stuff still works there. And then also, there's the Beaglebone Black which is a competitor as well, which has a ton of GPIO and comes by default with a Node API installed on it. So, you can use Node to interact with your GPIO.

That's cool.

Yeah.

The two picks that I have are both related to Chrome. One of them, I'm hoping they haven't been mentioned before. I didn't' see them on the picks page. So, one of them is the Remote Debugger protocol for Chrome, which I guess is how Chrome web driver is implemented and tools like that. But basically, the short version is you can launch Chrome with a particular command line flag and then open a WebSocket and talk to it over a WebSocket connection and send it commands that give you essentially the full power of everything you can do in Chrome Dev Tools. But you can easily write a program to script Chrome Dev Tools this way, basically. So, I just found out about that recently. I know it's been around for a while. But that's pretty cool. And the barrier to entry for it is pretty low because it's just like, well you write some JSON into a WebSocket and then you get back some JSON and you're good to go.

And Firefox has a similar remote debugging protocol as well. We used it when I was working on Firefox OS to see what's going on and everything. So, it's actually becoming more of a common pattern to do remote debugging protocols in the browsers. I'm hoping to see more of that kind of stuff out of IE and Safari in the future, too.

And then the other one is an experimental tool in Chrome Dev Tools called Filmstrip which will let you, it'll basically capture screen shots of... I'm not sure if it's every time but it'll capture screen shots as the page is going through the rendering process. So, I think this is a very cool idea. You have to do some weird magic to turn it on. And I posted a link that you can follow to read about said weird magic. But it's really helpful because it lets you see visually... you have all this stuff that Dev Tools provides you about what state the page is in, what resources have loaded and such. But it's hard to tell based on that information alone what the user was actually seeing at that point in time.

So, I've been messing around with capturing screen video as it's loading. And then also exporting

all the stuff from Dev Tools. But that's a huge pain. And this Filmstrip thing I think has the potential to be a lot more useful than that, because it just captures stills every time there's a change to how the page looks. And then you can correlate those back with your Timeline in Dev Tools. So, I'm hoping that that becomes a non-experimental feature at some point in the future, because I think it would be really awesome.

That sounds super cool.

So, how do people learn more? How do they get in touch with you? Who should they follow on Twitter?

So, I guess if you want to learn more about New Relic, you can just go to our website, NewRelic.com. Yeah, there's a discussion forum that's at discuss.NewRelic.com for just openended discussion questions. If you are a customer, if you're not yet a customer, it doesn't matter. Anybody can post there. And then as far as who to follow on Twitter, I'm on Twitter as @benweint, B-E-N-W-E-I-N-T. I'm not on there a huge amount but I do try to respond when people ask me questions.

And I'm on Twitter a bit but my account is private these days. So, @wraithan. So, W-R-A-I-T-H-AN. You can also follow me on my blog as well as the New Relic blog. I haven't posted to the New Relic blog in a little bit. So, I'm hoping to get more time to start diving into some of these topics that we talked about here. And then yeah, my blog is at Wraithan.net. So, just my name over and over again. That's how you can find me everywhere on the internet. I've used the same name since I was 13.

Sweet beans. Well, it was good to have you guys on the show. And I wish you well. Sayonara.

Thanks for coming.

It was super fun [inaudible].

Yeah, it was really fun. Thank you guys.

That's great.