Ian Johnson is a former Google UX engineer and data visualization engineer with ObservableHQ building data visualizations with JavaScript. He works on both the tools and the visualizations built with D3 on the web. He discusses how to use tools like D3 to tell a story using your data.

Panel

Guest

Sponsors

Links

Picks

Contact Dan:

Contact Steve:

Special Guest: Ian Johnson.

Sponsored By:

STEVE_EDWARDS: Hello everybody and welcome to another exciting episode of JavaScript Jabber. I am Steve Edwards, your host, the host with the face for radio and the voice for being a mime, but I am still your host. Today with me on our panel, we have Dan Chapier. Hi Dan, how's it going? 

DAN_SHAPPIR: Hi. Doing great here in sunny Tel Aviv. Well, it's actually night outside, so it's not so sunny anymore, but it was very nice and sunny today. So yeah. 

STEVE_EDWARDS: Yeah, we finally got, been inundated with rain over the past few days here in Portland after a very long, dry summer. So it's been nice to get the rain and watch the fire danger levels decrease for a change. And with us today, our very special guest is Mr. Ian Johnson from Observable HQ. How are you doing, Ian? 

IAN_JOHNSON: I'm pretty good. Weather's nice here in Atlanta. 

STEVE_EDWARDS: Good, good. Not too much humidity right now, or how's the humidity? 

IAN_JOHNSON: It's finally cooled down. It's finally calmed down with the humidity. 

STEVE_EDWARDS: Less than under 100%. 

IAN_JOHNSON: Yeah.

STEVE_EDWARDS: Yeah. So we had Ian, I recruited Ian to come on the show because we wanted to talk about D3, a graphing tool, plot visualization tool, however we want to call it. We'll get into that here in a bit. But first, Ian, why don't you give us a little intro about who you are, what you do, why you're famous, all those relative important things. 

IAN_JOHNSON: Sure. Yeah. I mean, I've been using D3 for almost 10 years now. My current role is a data developer at Observable. I do a lot of developer application as well. So working with the community, helping people understand how to use D3 and our platform as well. And yeah, I mean, I've always loved the sort of combination of artistic expression and kind of mathematical or programming skills that are required to come together for data visualization. And I've been working in this space since I originally moved out to Silicon Valley 10 years ago, working at different startups, spent a few years at Google as a UX engineer doing visualization. And yeah, now, well, I moved back to the South, but working at Observable and spending almost every day doing some kind of D3 or we have this plot library, which I'd love to talk about as well, another open source visualization tool, but definitely using JavaScript day in, day out. So I'm excited to be here and geek out about it.

 

Have you ever really been happy with your project management tool?Most are either too simple for a growing engineering team to manage everything or too complex for anyone to want to use them without constant prodding. Shortcut is project management built specifically for software teams and their fast, intuitive, flexible, powerful, and so many other nice positive adjectives. Let's look at some of their highlights. Team-based workflows. Individual teams can use Shortcut's default workflows or customize them to match the way they work. Org-wide goals and roadmaps. The work in these workflows is automatically tied into larger company goals. It takes one click to move from a roadmap to a team's work to individual updates and vice versa. Tight VCS integrations, whether you use GitHub, GitLab, or Bitbucket, Shortcut ties directly into them so you can update progress from the command line. Keyboard-friendly interface. The rest of Shortcut is just as keyboard-friendly as their power bar, allowing you to do virtually anything without touching your mouse. Throw that thing in the trash. Iterations planning. Set weekly priorities and then let Shortcut run the schedule for you with accompanying burn-down charts and other reporting. Give it a try at shortcut.com slash dev chat and get two months free. Again, that's shortcut.com slash dev chat shortcut, because you shouldn't have to project manage your project management. 

 

DAN_SHAPPIR: So it's all about the visualization for you. 

IAN_JOHNSON: Yeah. 

STEVE_EDWARDS: I remember my first experience with visualization was, man, I think it's been 10 years ago when I was at an agency doing, I did some work for a very, very large corporate customer, not sure if I can mention it, so I won't, but it's been a long time and it was a website and we were using Drupal back then, but it was all about data visualizations and allowing people to create and upload their own data visualizations for whatever. So yeah, to me, that was always pretty fascinating. And I guess the reason that I had asked Ian to come on is that in our application at GovTribe, we do a lot of data visualization and charting and of contract data and a lot of financial and date oriented type of visualizations. And we had been using a number of things, some core D3, some simple stuff like chart JS, and I had been in the, in a very long process to take something that had been in a chart JS and move it to something more full featured for things like annotations and some of the things that some of the higher level libraries don't necessarily handle. And so we'll get into maybe some of those details later about some of the things we ran into, but for setting the groundwork, I guess, if I'm mutilating some euphemisms there, laying the groundwork. There we go. So why don't we talk about D3, what it is at its core, how it's used, how types of use cases that you see for it, and so on. 

IAN_JOHNSON: Sure. Yeah. I mean, I'll say that your story is one that I think a lot of people have, right? that does sort of give you out of the box visualizations, you plug in your data, you get some stuff. And those are often very useful, but there's always some limit, right? Especially if you're trying to either customize something to your business logic, or you're trying to fit it into a product, you need a little, you always need to sort of dive in and have more control. And invariably, it comes back to D3. And I think, so, you know, what D3 is, it's part of the challenge with answering that is that it's a lot of things. There's actually 30 JavaScript modules that make it up. You can use them a la carte. You don't have to use the entire library. Depending on what you do. I like to think of it as kind of a toolbox for data visualization. So like, you know, some days you need a wrench, some days you need pliers. Some days you need a saw. Like depending on what you're doing, many of the common problems or tasks that you run into have some sort of nice tool in the toolbox for you. And that from there though, when most people come in contact with D3 for the first time, they probably see, if you go to the homepage, you'll see just like this montage of examples, like really beautiful things people have made. You might've seen some like really spectacular interactive news article and wondered how it was made, found out it's made in D3, almost all of them are. And so you see these like, powerful examples and oftentimes, you know, I know, I think like I want that, like I want to be able to do that. I want to see my data in that way. But then it turns out that like behind those examples, there are, you know, the usage of these various tools, like little APIs, stuff like that. But there's also a bunch of design work, a bunch of, you know, maybe some storytelling, like writing, communication work, as well as like programming JavaScript and data work. And then the level of abstraction that D3 is at is pretty close to like the DOM APIs. So like if you're using SVG or Canvas, like it gives you some nice ways to use those things, but you, you kind of have to know what those are in order to make use of D3. So you find like a lot of people, I mean, I've been, so I've been part of this community, I ran the meetups when in San Francisco for eight years, like for, you know, pandemic, for moving out here. We have a big Slack community. You see a lot of people come with projects and questions and stuff. The thing is that you get into it and all of a sudden it's not that you're learning how to make charts. You're learning how browser APIs work. You're learning about network security stuff like course and those kinds of things. And you're learning JavaScript functionality. So maybe a lot of people come from a Python background. They wanted to make interactive charts and they heard D3 was the way. And now all of a sudden they have to learn a lot of JavaScript and it's all doable, but it is kind of a lot. So, you know, when I say it's a lot of things, it is. But at the same time, it seems to fit this level of abstraction that like, if that's what you need to do, right? If you need to build custom interactive data visualizations for your product or for your communication, like it ends up being the toolbox that you want to have. 

STEVE_EDWARDS: Yeah. My experience has been... And I... Before actually starting to work with it quite as extensively as I was here recently. One of the things I'd always heard is that D3 is typical of a powerful tool in that you can do a lot, but it takes a lot to learn how to do everything. You really have to know, you have to understand SVG and all the different parts of an SVG, the different elements and how they all fit together and how to group them and to do that. So it's with great power comes great learning. I think it's my version of the phrase for that. And so one of the questions I wanted to ask is that there are a couple different ways that a couple different base elements that visualizations are based on in the HTML world. In my limited understanding, there's SVG and there's Canvas. And so D3 is based on SVG. Now you mentioned Canvas. Does it also work with Canvas elements or is it solely SVG? And then number two, why SVG for D3? Are there benefits of SVG over Canvas in using that as your base HTML element? 

IAN_JOHNSON: Sure, yeah, no, it's a really important question. So, well, I mean, but I would say that D3, I wouldn't say necessarily that it's based on SVG, but that there has a very, you know, a big chunk of the toolbox is dedicated to making working with SVG easier. So when you talk about using Canvas, there's a big chunk of more than half the toolbox, though, is things that could be used with any rendering engine. So you could actually use WebGL as well, which is technically an extension of Canvas in the browser. So there's all kinds of data manipulation tools. A pretty core data visualization concept is scales, right? So mapping from your financial numbers to like, the number of pixels on the screen or the colors that you use, right? Those tools are agnostic of the rendering. So you can use those no matter what you and a lot of times, like, if you want to talk about people building stuff in, in JavaScript frameworks, like React or Angular, that kind of stuff, they'll often end up like importing those tools and then handling the rendering of the DOM elements themselves, whether it's with SVG or HTML, and there's a couple of good like React libraries that that do that. So they still depend on D3, but D3 doesn't do the DOM stuff. That being said, why SVG? I think there's probably a couple of good reasons. One is that it's a vector graphics, right? So you can make a graphic in SVG and then you could export it and put it in a PDF. You could print it at extremely high resolution. You could zoom in. It'll never get pixelated. So for a lot of common applications of visualization SVG works really well for making crisp graphics. The other thing is that with, I just had, you can think of SVG or even like HTML elements as like a scene graph. So from like, this kind of comes from more like 3D graphics, but like if you make a rectangle in SVG, there's like a rect tag, just like you would have a div in an HTML, right? And then the way you often think about like if I want to change how that div looks, if I want to move it somewhere else on the page, absolutely position it or something, where I want to get with the rectangle, the thing is the browser is keeping track of that div or that rectangle. And so D3 makes it easy to say, you know what, I've got new data, I want that rectangle to move over here and change its appearance, but I don't need to keep track of an array. If I'm using Canvas it's sort of lower level. Like if I want to draw a rectangle and then I want to change something, I have to draw that rectangle again somewhere else. So I have to keep track of everything about that rectangle. So yeah, I think of SVG as this kind of, you know, similar to HTML is like a document that I can then lay things out and it will stay that way. And when someone else looks at it, they'll see the same thing. Canvas is much lower level drawing API where I'm actually turning pixels on and off. And that, but that has some advantages that it, it takes less memory. It can be faster. So you could plot more. Like if you, if you do want to plot a lot of things, like if you have more than 10,000 elements that you somehow want to show, then, or move them around, then canvas or WebGL will be where you would start looking because there's some overhead to, to having that many DOM elements. 

DAN_SHAPPIR: If I can pull us back a little bit. So these D3 is the stands for data driven documents, if I understand it correctly. Maybe it's worthwhile a little bit, at least for some of our listeners, to talk about what it actually means to be data driven. How is being data driven different than the normal approach to domonipulation that I would be using if I just use vanilla JS or some framework like React or something like that? 

IAN_JOHNSON: Sure.Yeah, I think that the, I mean, in my experience using React and building apps that aren't focused on data visualization, oftentimes they are pretty data driven, right? Like you're gonna, you have, let's say, a user page, and you're showing a user profile to somebody. Let's say you go to someone's Twitter homepage, right? Like all the things, the avatar, the name, all those things are driven by some data, right? Like the user data is probably a JSON object andyou're using the framework to say, like how to display it in HTML and CSS. I think with D3, the focus is often on this idea that you have a bunch of ways that you could encode data into visual elements and the choices you make in there are more, you know, technically you could do the same thing. You could build the same app with D3 or React, whether it's a data visualization app or a non data visualization focused app. It's just that the the conventions, the abstractions are much more geared towards helping you do visual encoding in D3, right? So if I want to make something simple like a bar chart, but I want to have the colors represent some scaled numbers or something like that, there's a function in D3 that makes that easy. Whereas if React has like a component lifecycle thing because you're much more often doing some kind of data entry as well, right? Like you're not just showing data, updating a database and keeping track of the UI and as data changes. So like there's a bunch more functions for that kind of behavior, I guess you could say. So I think the name data-driven documents, in the context of visual documents, is where that kind of comes from. 

DAN_SHAPPIR: So if I'm trying to understand what you're explaining, it seems to me that what you're saying is that in D3 I'm taking a more of a sort of a declarative approach where I'm specifying, let's say, as a function or some sort of a mapping, a relationship between data, collection of values, and the visual representation of that's associated with that data. So if the data changes, I've declared, I could say that within this range, it should be green. Within this range, it should be, I don't know, and within this range it should be red. So I don't have to write like the logic that handles this imperatively, rather it does it declaratively for me, the data just quote unquote flows through and comes out the other side as the visual representation based on those rules that I've specified. Is this more or less correct the way that I'm presenting it? And basically, a certain layer of those transformation is kind of, if I go back to your previous explanation, is independent of the mechanism that I use for the actual rendering. So if it's like the colors, I could use those colors with SVGs or with Canvas or even with regular DOM and the CSS. And then at the lower level, let's say, or somewhere down the pipe, I specify which actual visual representation I would like to use for that data. So the data kind of flows down the pipe, undergoes these various transformations or collects attributes along the way. And then finally, those attributes actually become that visual representation. 

IAN_JOHNSON: That's right. Yeah. 

DAN_SHAPPIR: OK, cool. So it seems to me kind of similar and maybe that goes to the name of your company to a sort of an observer, like a functional, how's it called, FRP sort of thing, like with observables and stuff like that. Is that kind of an accurate module to use, mental model to use? 

IAN_JOHNSON: Yeah, I think pretty close. I mean, so with D3 in particular, a lot of what you're doing is sort of actually as a user of D3, you're writing these callbacks or like these functions that maybe they're accessor functions or there's some way of saying like, yeah, this is the part I want you to use. And you're declaring to D3 what you want. And then it's gonna go, and a lot of times you don't actually, like something you'll notice in D3 is that you almost never write for loops, right? And that's sort of the more imperative, like I'm gonna loop through all the data and determine, you know, keep track of all the state and figure out what to do. Yeah, so D3 is is certainly more of a functional flavor to it. I think with observable, so we're a platform for these interactive JavaScript notebooks. Part of the way they work is it is based on a reactive model. So like you have these cells where you can write JavaScript in them. And then whenever you update one of those cells, it will figure out how to like the data will flow through and any cells that depend on that one will also update themselves. So and it would, you know, Mike Bostock, the creator of D3 also found it observable in part as, you know, learning kind of the way that you often code this kind of stuff. That environment, like having this environment is very helpful where you do, yes, exactly. You do want to write these small functions. Oftentimes, I mean, in my experience, I want to see what happens when I do those functions, kind of check that it makes sense. There's a lot of, when you really get into visualization, there can be a lot of math or like visual parameters, like things that's like, oh, if I, changing one number a little bit could make big differences in how things turn out. So being able to see those instead of imagining them is really powerful, yeah.

STEVE_EDWARDS: So yeah, you mentioned the notebooks. And when I was starting to get into learning about D3 in the core, I know that was one thing I noticed. I mean, you've got whole training videos just basically on how to use notebooks. And a lot of, if you search for examples, that's usually where you'll be directed is to your notebook. So can you talk real quick about what those are and how they work? Because as I see them, they're basically a self-contained environment, sort of like a, I guess, a code pin or a code sandbox type of place where you can code up a whole, you know, visualization graph of, of that you want to demonstrate or play with or allow other people to fork type of thing. 

IAN_JOHNSON: Yeah, absolutely. Yeah, I mean, I think that's a pretty good characterization. Like it's a notebook is somewhat of a sandbox. Like you write JavaScript. There's a few extra things that you can depend on from our platform, but you could easily write things that are totally portable. But it does mean that you don't need to set up a development environment. You can just require node modules. So as long as a node module works in the browser, it will work in the notebook. You can import from other notebooks. So that creates that sort of community effect of, like if I make a nice function or a nice visualization even, you could import that cell from my notebook and then replace the data and get, you know, same visualization. Like you mentioned the forking, right? Sort of, we're all familiar with that from GitHub. Like you see something interesting, a nice notebook, you want to try it, try to tweak it or replace your data in there. You can just fork it and go wild. But yeah, and like most of the D3, all the official D3 examples, and there's over 400 of them, are being managed on the platform now. It's just, it turns out to be a better way to deal with that many examples, to keep them updated and to showcase how, you know, what's going on in them. And that, you know, that's been interesting because that's sort of part of a journey, right? Like Mike made this blocks platform early on to showcase examples. Like one of the core, like sort of more philosophical principles behind D3 and observable is this idea of making examples to make better code, right? So like, and Mike has a talk, he gave a while ago on this, but like when you make an example to teach somebody or to showcase some functionality you've made or to answer a question or just to answer your own curiosity, kind of isolate what you're working on, there's a lot of additional benefits to doing it that way that end up paying off. Like, so even if your only goal was to sort of, let's say you wanted to do like a, some kind of creative gradient, CSS gradient on a div or something, right? So you could build that into your app and figure it out, right? Like read the documentation. But if you isolate an example that really sort of highlights exactly what you're trying to do and then shows how to do it, you get these second order benefits of, well, someone else might learn from it. Someone else might comment on it and give you a better tip. You might be able to share it with your colleague who's a designer and say, hey, this is what I was doing. And they point out something that you didn't think about. So there's this whole like, yeah, push to make examples. And in my personal opinion, I believe that the sheer number of examples available from D3 has made it, has played a huge role in why it's so popular, because you can see what's possible and then you can learn how to get there. And so yeah, what observable is kind of doubling down on that, the idea that not everything, like you certainly want to build, you know, if you're building software, you're building product. You're going to have a whole development environment, a whole software engineering process. But so much of what we do is understanding the available APIs is agreeing with other team members and other people on what exactly we're trying to show and how to show it. So there's so much process that can be improved by working in, you know, by example. 

DAN_SHAPPIR: So given, if I can just comment that given the breadth of functionality that D3 includes, which I think is also a major selling point of D3, is that for data manipulation and for visualization, it's highly probable that whatever function you want already exists within D3 and you don't need to implement it yourself. So certainly when you have such a wealth of functionality and just sheer number of functions having discrete examples for all of them, or at least most of them, is definitely really desirable and highly effective. I guess it just boils down to how easy it is for you to find the examples that you're interested in. It becomes a search problem. But yeah, I definitely agree with what you're saying. 

IAN_JOHNSON: Yeah, and I mean, that's a great point. Actually, years ago, I did an open source project called Block Builder that it was a D3-focused code pen. It wasn't this whole notebook model, but it was, can I make it easier to do in-browser editing of D3 examples? And a lot of people that use that made a lot more examples. Because it's not just the discrete functions that you're showing up, but it's also the combinations. So it could be a combination of powerful DOM API, D3 data processing, and a creative application to a data set. So you get this combinatorial explosion. And we did, we made a search engine and we at some point indexed like 40,000 examples that people have made across the community and you could filter it by API call that kind of stuff. And now with observable, it's, it's taking that to another level. Like we definitely have search in the app and we're constantly trying to, to make it easier to, to find what you need as you're in the process, because you're right, there is so much that you could do so much out there, yeah, does start getting really getting into how do I find the thing I'm looking for. 

STEVE_EDWARDS: So we've talked about, to tack here just a little bit, we've talked about Observable HQ and that's who you work for and it's a company that Mike Bobstock founded. So I was wondering if you could expand a little bit about the company itself. I know similar to many other software tools, libraries, what have you, it came out of a need, right? Somebody needed something and he started building it and eventually it took off. So I was just curious to see one, if you could expound on the history, you know, where Mike started out and how that came into observable. And then I'm also always curious about the business model observable itself. What did obviously D3 is an open source project that anybody can download and what's the business model for observable and how, how is it integrated with D3? 

IAN_JOHNSON: Sure. Yeah. So, I mean, starting with the history, Mike announced the company in a talk at this open vizconf, I believe it was 2018, and then he called it D3 Express. And yeah, the idea is a better way to code, right? Like I think the underlying, one of the underlying philosophies is that code is the ultimate tool of expression, right? Like if you want to express anything, if you can code it, like you can get there, right? It may take a lot of work, may take a lot of help, but if ultimately, if you want to get it done, like, you can get into code and make it work, right? So there's a lot of tools and things out there to not code with, but they all end up being like, you have to make decisions that limit expression in order to make it so that you don't have to code with it. Like there's some limits being set. So the idea with D3 Express and now observable is that you'll always be able to code, but we wanna make it easier to express the kinds of things you wanna express and we're really focused on thinking with data. So data visualization is going to play a big part in that. Certainly with all the experience with D3 and all the visualizations that people at the company have made in the past and are continuing to work on, we believe in the power of using visualization to help you think with data. But it's not just visualization. A lot of the work actually that goes into making good visualization is reshaping, cleaning up, wrangling the data. It's sort of the unseen, unthanks work that any, but anybody who professionally does data visualization will tell you that 80% of the job is actually wrestling with the data and then you get to find the right visual form for it to really express the pattern you're trying to highlight or the insight you found. So, so yeah, observable is, is about helping you do that. Like, think with data collaboratively, right? So we have teams where you can edit the same notebook together of course, things like commenting and other collaboration features. And that kind of gets into the business model, where the personal accounts, kind of like GitHub, it's like your personal account's free. But when you want to work as a team together, then you can pay for a kind of enterprise deal where you add seats and that kind of thing. So yeah, I mean, with D3 totally open source, it's actually not owned by Observable, but we use a lot. We continue to invest in it contribute to it. Right now in the middle of actually revamping a lot of the examples, we're on version D3 7.0. I believe when Observable started, it was on 5.0. So, you know, done a couple improvements. And then we're actually just released earlier this year Observable Plot, which is another open source visualization tool. Again, totally free. You can download the code on GitHub. But it has a different design from D3, and it does work really well in our platform but you can totally use it anywhere. And again, that gets into sort of, when we talk about thinking with data, when you're making data visualizations, a lot of the work there is the decisions you make, right? Like, how do I want to represent this data? It can be something as trivial as like which two colors, I mean, it can sound trivial, it's actually can be really important with like accessibility, colorblind, that kind of thing. But like, you can get down to decisions like which color should this be. Or it could be like what type of visual layout, what kind of chart do I want to have. Plot is designed to let you sort of stay in the design space. Like the decisions you make are all about which pieces of the data and how do you want to represent them. And they try to abstract away thinking about what DOM API am I going to use? And like, how am I going to process this data? Like, how am I going to do the rendering? Right? So it kind of tries to get you away from having like with D3, you want, you have to do both. You have to say, I want to represent this in this way. And in order to do that, I need to use these DOM APIs and these D3 functions. And then it all comes together in a visualization. But that's a really, it's a lot to think about. That's why it takes a lot of work to make a really good data visualization. And something that takes a lot of work means it takes a lot of time. And when you can cut out a lot of that, you can work faster. You can kind of visualize things as you think of them. And so far, my experience with that and the folks who've been trying it out has been really exciting to see like how differently you can look at data, how quickly you can get to new ideas because you just, I mean, you're still coding, but it's just like the abstraction is so much closer to the problem you're working on, which is how do I express this data, not how do I like render this data. 

 

Are you ready for Core Web Vitals? Fortunately, Raygun can help. These modern performance metrics play an important role in determining the health of your website, which is why Raygun has baked them directly into their real user monitoring tools. Now you can see your core web vital scores are trending across your entire website in real time and drill into individual pages to focus your efforts on the biggest performance gains. Unlike traditional tools, Raygun surfaces real user data, not synthetic, giving you greater insights and control. Filter your score by timeframe, browser, device geolocation, whatever matters to you most. And what makes RayGun truly unique is the level of detail they provide so you can take action. Quickly identify and resolve front end performance issues with full waterfall breakdowns, user session data, instance level diagnostics of every page request, and a whole lot more. Visit raygon.com today and take control of your core web vitals. Plan start from as little as $8 per month. That's raygun.com for your free 14 day trial. 

 

DAN_SHAPPIR: So it seems to me based on your description that what you're aiming for is kind of like sort of coding by example, as it were, that you're using a notepad or something to kind of, I don't know, interactively model the output that you like to get and then you kind of automagically generate the code that actually implements it, something like that. 

IAN_JOHNSON: Close. So yeah, I mean, the notebook environment, I think one of the interesting things about notebooks, right, because there's a lot of Python, like there's Jupyter notebooks and Python notebooks have been really successful in the data science community and R and R Studio, they have their notebooks there. I think the thing about notebooks is that when you use them a lot of times when you don't know what you want to get, like you may have an idea of where you're going, but you don't know exactly how to get there, right? So this happens all the time in data science. Like you have a big data set and you're trying to understand, like you know that there's something in there that you're, you know, maybe it's millions of images and you're trying to extract some features from it or something. But you don't know exactly how you're going to get there, so you have to try a bunch of stuff. And as you try those things, you know, you go down dead ends, you find surprising things, and you get curious. And so a notebook format allows you to kind of go on that exploration and without worrying too much about breaking the thing, right? Because if you're in just your text editor and you're rerunning your code, you have one thing that all has to work. And if you try something weird and it doesn't work, there's no way to sort of kind of step out of it and try it, right? So the notebook environment really supports this kind of thing where, yeah, you have a good starting point, you know you want to get somewhere, and then you got to figure out how. And the data space, data analysis, data science, like visualization, they all kind of fit that. But I think in my experience, also, like web programming fits that a lot, because there's so many APIs and so many techniques that you could use. Often I see something interesting and I say, how I want to get there, let me figure it out, right? Then on top of that, what we're doing is trying to make it easier to get to where you're going for, and so we're focused on thinking with data. So if we believe that seeing your data will help you think with it and help you get there faster, like understand some pattern or, right, then observable plot is our most recent attempt to make that journey easier. And I wouldn't say it generates code necessarily. It's implemented in D3. The way I like to think about plot is that it uses like really well-earned wisdom from 10 years of D3 development by Mike and others that like are then captured in heuristics and defaults that you can type just one line of plot and get something pretty good. And then you can start tweaking it and refining it and all the while seeing what you're getting and never having to drop down to the the next level abstraction of like getting the browser to render something correctly or joining DOM elements or updating them, that kind of stuff. 

STEVE_EDWARDS: So am I understanding correctly that this is sort of, can you say it sort of sits on top of D3 and supposed to and tries to handle all the legwork for you and just allow you to do the more fine tune, do more fine tuning, I guess, for your visualization? Is that a fair description? 

IAN_JOHNSON: Yeah. Yeah, I think that's a pretty fair description. 

STEVE_EDWARDS: So I'm- Yeah, I haven't played with that yet today. You know, talking to you, this is the first I've heard about it. And the reason I'm, I guess, coming from you guys, the people who created D3, this could be really good. But for a couple of months, I fought with another library a lot that tried to do the same thing, D3 plus, and there's other, there's very other, various other versions, D3 FC. And what I've seen that they did was, uh, Mike had a blog post about how you could just use JavaScript code you know, chain everything together, making different calls and build a graph from that by passing in configuration data. And what we continually found out, no matter which tool we used was by the end, we'd wish we'd just stuck with base D3 because of the limitations that these libraries continually run into. There's there's they just can't handle everything or it was very difficult. Poor documentation or where you just, every method has a stock description. Well, this, you know, inherit something from parent class and that's it. Well. Thank you, that doesn't tell me very much. And so the problem, like I said, we continually ran into was we would run into limitations and trying to integrate the base D3 with this tool that we were trying to use proved to be very time consuming and very difficult. So I guess I'm curious how observable plot would differ from some of those other libraries. And I don't know if you've looked at any of those tools or seen some of the various libraries that purport to sit on top of v3 and make things easier. This sounds like a lot of the same. So now that I'm rambling, what's your thoughts on what I just said? 

IAN_JOHNSON: As not rambling at all, that is like a recurring concern there. I mean, you know, in the last year, I've seen so many libraries come and go on top. I think that, you know, there's a couple, a couple of important things to keep in mind. One is like the goal, depending on the goal of what you're doing with the visualization, right? It sounds like if you're building a visualization into a product where you're trying to build something that has to fit a lot of design specifications, then you're gonna want, yeah, I think, my opinion is that you're gonna wanna build it in D3 and then build up from there because you'll have control over every decision that goes into that, whether it's a design decision of how and when to display things or whether it's business logic or just something like totally different that you want to do. It's going to allow you to make it at every level from like how you render up to like how, how it looks and how it works on mobile and all that kind of stuff. You get complete control. And like we've talked about that, that does take like having all that control means you have to make all those decisions, which takes a lot of work and. In a lot of cases learning so that you know how to make the best decision, but in, you know, invariably, if you're building something to, to like high standards, right? Then you're gonna want that control. Plot is designed more for the exploratory phase. So when you, I mean, it's particular. The thing is that without plot, you could use D3 for the exploratory phase as well. It's just that you had to do, in order to explore with it, you had to be really well versed in it, right? And like someone like myself, who's just, I've written bar charts from scratch, like hundreds of times, just because, right? Like I can memorize all these APIs. I can move fast, but that's a lot to ask of people who that's not their passion. It's like, yes, you want to get into visualization because you care about this data or because you care about this product or this story doesn't mean you want to become an expert in everything behind it. So, plot is designed more for getting you quickly to see the data, to understand the important things about it, to help you make the choices that will be the most impactful in your understanding, but it won't, you can tweak quite a bit of that. You can really quickly make some really good looking plots. And then the other thing is that if you do that and you decide I want to make this custom built into my product, plot may be good enough, but if you still find you need to go to D3, you can very easily take what you did in plot and then do it again in D3, because you have the scales, which is one of the, you know, some of the biggest sets of decisions you'll make will already be, excuse me, figured out, or a lot of the layout concerns that you might have, like, I don't know how much time I've spent filling with margins and insets and axes and all that stuff, right? I have nightmares about all the time I spent dealing with those. So the thing is, I would definitely recommend, especially if you're trying to design the thing or prototype your chart, do it first in plot, like it makes that kind of stuff so much easier to just like fiddle with and get something. And you can really easily like try to push it the boundaries, see what it does. And you don't have to worry about like all the D3 code to do that. And then if you're like, okay, now that this pretty much is what I want, you can sort of take what plot did for you and then look at it and say, okay, now I can implement it in D3 and then add my custom tweaks. And there's actually ways to extend plot with D3 without leaving plot as well. We're still working on the plugin architecture so that that's a bit easier to do. And plot is still very early in its developments, 0.2. So we're not even at a 1.0 yet. But yeah, I think it's really important to keep in mind the process, the design and development process for visualization that it's like, there's a whole spectrum there. And the different tools get you to certain parts of that. If you want to cover the entire one, start to finish. You know, D3 is, I've never seen anything other than D3 cover that whole thing. I think plot is designed to get you further, and that may be good enough depending on your task, right? If you're doing data science exploration, then like you may be able to just get a chart, save that, put it in your presentation and move on. Yeah, so that's how I kind of look at those things.

DAN_SHAPPIR: I think that the point that you made about the fact that given a certain amount of data that you've got from wherever. There's this whole exploration, first about how to best represent the data or in order to get across the point that you're trying to make. And then all the tweaking that has to go with actually making it visually attractive and usable is a ton of work that very often developers that are not so involved with data representation are not aware of all this effort that has to go in. We just get whatever UI specification we need to implement from the UX expert, and then we just go ahead and implement it in whatever. But this whole exploratory stage often is not something that we even consider. 

IAN_JOHNSON: Yeah. So we have a community leadership, you know, people have been part of the D3 community for a long time. And we, we definitely have been over time trying to make it easier because a lot of people do come to the homepage of D3, right? They heard, oh, we need to chart D3 as a thing. And then, you know, you come to an example, you try to copy, you're like, Hey, this is pretty much what I want. And then you try to like modify it a little bit. And you're like, what is going on? Like, why, why is there all this stuff? You, you kind of probably imagine that, oh, there should just be a component and I can tweak some options, but there's actually like this whole stack of knowledge and decisions behind that one example that you weren't aware went into that, right? And so that can be a shock off in. So, I mean, yeah, there's a lot of ways we're trying to, to soften that, make it easier for people to, to come in and express, but also, yeah, I think some amount of it is like understanding and respecting that there's a process and you don't have to necessarily invest in the whole thing, but there's now, like more and more tools that help you through that process. 

STEVE_EDWARDS: So I'm still trying to get my head around the observable plot that you were talking about. If I understood you correctly, it sounds like it's more of a prototyping type tool. Is that, is that correct? I was obviously pretty new at, you know, 0.2. So you haven't, it's not fully fleshed out, but it's not designed, is it designed, is the goal to have it be able to handle most everything that D3 itself can, or where does it fit into the ecosystem?

IAN_JOHNSON: Yeah, I think that the goal, I mean, prototyping is one way, but it may be like a more focused goal is to help you do data exploration, which is maybe more like as a common workflow for data scientists, data analysts, you know, business intelligence, that kind of stuff. So like it may be like in the JavaScript development space that the equivalent would be the prototyping. So plot kind of makes a lot of things easier and quicker in for understanding your data or like quickly making visualizations because basically every time you get a different perspective on your data, it may point out something that you weren't aware of. And I mean, I've found this really valuable as in my role as a developer when I'm like analyzing logs or trying to understand performance metrics. You know, there's a lot of like you could use like a default out of the box chart, but like if you can quickly see it, you know, aggregate by something, some other thing that you track or bend the time by different intervals, and then you find outliers or you find strange patterns, that flexibility will, will give you an understanding of how things are working that will inform the way that you develop, right? Like the way that you engineer, because now, you know, you have a clearer picture of what, how the system works and the things you need to accommodate. So yeah, I mean, I would say I wouldn't advocate that people say, oh, I don't need like, I want to build visualization into my app or I want to build, you know, a custom visualization that's been like designed thoughtfully to go use plot instead of D3. That's not really the target use case. It's not, it's not intended to replace D3. :

STEVE_EDWARDS: So, so let me put it this way. I'm coming from you. You mentioned this being useful for like a data scientist or someone who's trying to visualize their data. So if I'm coming at this from front end development position, I have an application or a web site that I maintain or that I'm building and I want to display some data. So observable plot isn't necessarily something I would plop into my site, use it and go in and build a graph that I'm going to display for in my application or my website. Is that true or am I misunderstanding? 

IAN_JOHNSON: Yeah. I mean, I would say that given how quickly, you can make plots with plot. I would say start there as a prototype. Figure out, run your data through it. If you're thinking, you know what, I was gonna make a stacked bar chart or something, right? Because I know I have these categories over time. And building that in D3, you could go look at a D3 example, you could do that. But it's like two lines of code and plot or something, right? So you could and then you can like add configurations and see, see what you might get. I would say like, it's always better to see, see your data on screen as fast as you can, because that will, it will invariably change the decisions you make about what you build. So yeah, use plot, try it real quick. See, see if it like makes sense. First of all, what you thought you wanted to see before you go and build it all up. And because it's so, will take so little time to do that. Then you're like, okay, that, that is what I wanted to see now I'll do the D3 equivalent, but you have the added sort of benefit that some of the important design decisions have already been like worked out in plot, right? So it's like, I'm using this, this property of the data to define the scale. I'm using this kind of scale. I'm using, you know, like this domain array, like all those things can be basically confirmed for you before you start actually building it out. Because then, and then you know, when you're building the D3, it becomes much easier to say like, Oh yeah, I copied the example. I'm going to switch out these scales here with the things that I chose. But I'm pretty sure that you wouldn't, I mean, it depends on what you're doing, but it probably not going to like build, like make a plot and then say, ship it to production, especially if you're in this sort of boat that you've described where you run into the limitations of exactly how you want to display things and building something to specification. Yeah. 

STEVE_EDWARDS: Okay. So we need to start wrapping up here for times sake, but I guess one thing real quick, and this just gets down to the nuts and bolts of data visualization and D3, and we've mentioned these a couple of times, and there's something that it took me a while to get my head around, is scale, domain, and range. And so those are like the nuts and bolts because you've got to figure, okay, how am I going to get my data laid out? What are my ticks, which is what they're called on the bottom and the side going to be? What's the you know, what are the values are going to be shown there. So could you give us a description of those three things, what they are and how they all sort of tie together? Cause they all, you basically have to nest one within the other within the other when you're sort of calculating things if I remember correctly. 

IAN_JOHNSON: Yeah. So, and I, I have a link I can share, like a, a tutorial that focuses on, on this part. That was actually a few good resources that I'm happy to link folks to, but the way that I think about this, that keeps, keeps my head straight about it is that your scale is a function, right? It's a function that helps you, its input is a piece of data and its output is some kind of visual encoding. So maybe the easiest example would be an X axis of a time series graph where it starts on January 1st, 2021, goes to December 31st, 2021. And so your data, the domain of that data is that time range, right? Is January 1st like beginning of the year to the end of the year. So you know that like all the data points are gonna be a date, all the X axis data points are gonna be some date between those two. So that's the domain. And the domain is often defined by the minimum and the maximum, right? So like the earliest date and the latest date. And then the range is the pixels. So like, I know I want my graph to be 800 pixels wide, but I want it to start, you know, with some margin on the left. So it's gonna start at 100 pixels and go to 700 pixels. I have some margin on the right, so it's going to go to 600 pixels. So my range is those pixel numbers, right? So there's a minimum and maximum pixel. So the scale then, like, once you declare that I declare my domain of my scale to be the beginning of the year, end of the year, I declare the range to be 100 to 700 pixels, and then I get a function that I can pass at any, any date in my domain, right? Like, so I have an array of data and each data point has a date associated with it. Maybe it's like an event sign up, a user sign up or something, right? And so then I can say, Oh, I want to draw a circle for every, every sign up. Like where do I place them on the X axis? I call my scale function and I pass the date of the sign up and it gives me back a pixel number and that's where I put the center of my circle on the X, you know, in the X dimension. So, yeah. 

STEVE_EDWARDS: Yeah. That makes sense. Which is one of those things that you sort of have to start playing with to get your head around when you figure out how to code it. But anyway, just wanted to pull out those three things because when it comes to visualizations, especially things with your axes, those are sort of the core fundamentals that you need to understand. So we'll make sure and put your blog post that details these into the show notes for sure. Yeah. So before we wrap up, is there anything else you wanted to cover or talk about that we haven't discussed yet? 

IAN_JOHNSON: You know, I've really enjoyed talking about this. I think like we've said, there's a whole depth you can get into. I think one of the interesting challenges is, yeah, is keeping these high level understandings and then being able to dive in only when you need to. I think that's what we're often trying to do with our notebooks on observable is help people understand the capabilities. And then the nice thing though is that the code is there. So like the link I'll share for the skills tutorials are are interactive, that you can play with the parameters to get a feel for what this function is doing for you. And I think that that, I mean, the way I learn is essential to be able to like touch what the concept, right, to be able to tinker with it and say, if I do this, what happens? And what if I do that? And then once I'm kind of satisfied that I get the pattern, I feel like I can then go and use that, that function or that technique in my work later. 

STEVE_EDWARDS: Right on. So with that, we will move into picks. 

 

Did you work your tail off to get that senior developer gig just to realize that senior dev doesn't actually mean dream job? I've been there too. My first senior developer job was at a place where all of our triumphs were the bosses and all the failures were ours. The second one was a great place to continue to learn and grow only for it to go under due to poor management. And now I get job offers from great places to work all the time. Not only that, but the last job interview I actually sat in was a discussion about how much my podcast had helped the people interviewing me. If you're looking for a way to get into your dream job, then join our Dev Heroes Accelerator. Not only will we help you get the kind of exposure that makes you attractive to your dream employer, but you'll be able to ask them for top dollar as well. Check it out at devheroesaccelerator.com. 

 

STEVE_EDWARDS: So let's start today with Dan. 

DAN_SHAPPIR: I'm not used to being first. It's usually you or Ajay. So today I'm going to have just one single pic but it's an important one. It's such an important topic that I probably will bring guests to talk specifically about this topic in the, you know, for a whole episode, but it's one that's worthwhile bringing up again and again, and that's Apple's, or actually more accurately, Apple's iOS behavior with regards to browsers. The fact that, you know, you might think that you're installing various different browsers on IOS but they're all WebKit, WebView on the inside. So it's, if you're using Chrome on iOS, it's a Chrome wrapper with around what is actually like a limited version of Safari inside of it. And if you want to understand why this is actually important, then there's a, and you don't want to get into all the technicalities uh... then there is uh... an article about this that uh... recently came out by uh... Richard uh... mcmahon s on uh... the new stack and he talks about apples browser engine ban and in fact i'm one of those people's trying to push the hash tag apple browser ban on on twitter in order to maybe somehow with regard to different browsers on iOS, and in general also increase their investment into WebKit in Safari. So I'm going to link to that. I think it's really an important topic. I think that Apple either intentionally or as a byproduct is effectively, I would even say destroying the web. And I'm really hopeful that somehow we can get them to change their attitude and their policy on this topic. So that would be my pick for today.VE: 

STEVE_EDWARDS: All right, thank you, Dan. So my picks will be the usual, I know the high point of the podcast for many people is my presentation of a couple of dad jokes. And those of you who listen will know that I have very high standards when it comes to my dad jokes. Anyway, so for today, did you know that the T-shirt is actually an abbreviated version of the Tyrannosaurus shirt because of the short arms. 

DAN_SHAPPIR: Okay. 

STEVE_EDWARDS: And then second of all, a simple question is one of those, those common sense things that you never really realized you're thinking about until somebody asks you, why should you never brush your teeth with your left hand? And this is coming from someone who's lefty and I found it's because a toothbrush works better. So anyway, those are my, my picks for the week. Ian had a delayed response there. It took him a couple of seconds to get that one. 

IAN_JOHNSON: And as a new father, I'm still sharpening his sword. 

STEVE_EDWARDS: Getting into the world of the dad joke. 

IAN_JOHNSON: Yeah, but I appreciate this lesson, this masterful lesson. 

STEVE_EDWARDS: Yes, I'm always willing to share a good job joke, as my family will tell you. Okay, Ian, what do you have for us today for picks? 

IAN_JOHNSON: I think I'd like to pick, it's an old talk by Brett Victor. It's called Inventing On Principle. And I I think it's still super relevant today. It's, it's kind of what set me on this journey of like browser-based, interactive data visualization code in a lot of ways. That sort of core thesis is that ideas are fragile. And I think that the demos that Brett put in that video, there's still some of them that we need to implement as a community. So I put a link that you can, can share with folks, but it's, yeah, it's just super inspiring talk and I recommend anybody who uses the web and builds for it to check it out. 

STEVE_EDWARDS: All right. That's awesome. Well, thank you very much for coming in. It's been a lot of fun hearing about D3, something only a programming geek could say fun in D3. But so if people want to get ahold of you, find out more about what you're working on, maybe contact you, where can they do that? 

IAN_JOHNSON: Twitter's probably the best way. Angelo on Twitter. Yeah. DM or mention. And I'm always happy to to talk D3 applied data viz, that kind of stuff. 

STEVE_EDWARDS: Right, and Jalat, it's E-N-J-A-L-O-T, and we'll put those links in the show notes as well. All right, well thank you again for coming in, and we will talk to everybody next week. 

IAN_JOHNSON: Thanks for having me. 

DAN_SHAPPIR: Bye everybody. 

 

Bandwidth for this segment is provided by Cashfly, the world's fastest CDN. To deliver your content fast with Cashfly, visit C-A-C-H-E-F-L-Y.com to learn more.