Iterators and generators were introduced into JavaScript way back in 2015, yet they remain an underused and often misunderstood features of the language. In this episode Dan describes the purpose of iterators, how they're implemented in JavaScript, and why you're using them even if you aren't aware that you are, via the spread operator for example. The panel then discusses the pros and cons of iterators in JavaScript, and why most devs don't explicitly use them.

Panel

Sponsors

Links

Picks

Sponsored By:

AIMEE_KNIGHT: Hello everybody and welcome to another episode of JavaScript Jabber. This week we have the panelists. So myself, Amy Knight, coming at you from Nashville. We have Dan Shapir. 

DAN_SHAPPIR: Hey, hey from Tel Aviv. 

AIMEE_KNIGHT: AJ.O. Neal. 

AJ_O’NEAL: Yo, yo, yo, these are registered guns. Flexing my arms for those of you that can't see telepathically into the present.

AIMEE_KNIGHT: Steve Edwards. 

STEVE_EDWARDS: Yo, yo, yo, coming at you from Portland, Oregon. 

AIMEE_KNIGHT: And this week we are going to kind of give the mic over to Dan because I think he specializes in some of the stuff we're gonna talk about iterators and generators. And then if we have time left over, probably get into modules. So with that, Dan, do you want to get us started? 

DAN_SHAPPIR: Yeah, sure. 

 

This episode is brought to you by Dexecure a company that helps developers make websites load faster automatically. With Dexecure, you no longer need to constantly chase new compression techniques. Let them do the work for you and focus on what you love doing, building products and features. Not only is Dexecure easy to integrate, it makes your website 40% faster, increases website traffic, and better yet, your website running faster than your competitors. Visit dexecure.com slash JSJabber to learn more about how their products work.

 

DAN_SHAPPIR: So for those of you who've been listening to the past couple of episodes, you may have noticed a recurring theme in which I kind of argue in favor of, let's call it modern JavaScript or ECMAScript. And AJ to an extent argues against it. Kind of sort of, and I'll probably misrepresent you AJ, but kind of arguing that ES5 was the proper JavaScript. And we've kind of lost our way ever since. Am I presenting your views correctly? 

AJ_O’NEAL: Yeah, I'd say that like if, as long as we can consider ECMAScript a separate language from JavaScript as colloquially termed, then I feel that is just. 

DAN_SHAPPIR: And I, like I said, I kind of take an exception with that because I, well, I'm not a fan of everything that was added to JavaScript via ECMAScript since ES5. And I do think that there is an issue around the surface area of the language becoming ever larger. I do think on the other hand that a lot of the stuff or features that have been added actually have merit and in particular the one that I wanted to argue with kind of argue with AJ and perhaps other members of the panel about iterators and also generators. I know that they often are presented together, but I'll kind of present them separately, at least to begin with, because I know for a fact that, AJ, I know that you feel that iterators definitely and generators definitely should not have been part of JavaScript. 

AJ_O’NEAL: As constituted. 

DAN_SHAPPIR: And I feel that especially iterators and also kind of generators are actually a good idea, have been fairly well integrated into JavaScript. And if anything, I consider them to be a big missed opportunity. And I'll explain why as we get along. But Amy, I think you have a question, right? 

AIMEE_KNIGHT: I did. You touched on what my question was. And I was going to say, maybe if you want, I would like to have AJ go first as to why. Like, what's his why behind the argument? I know Daniel probably expand on that in a second. But I'm curious, AJ, why should we have maybe stopped where we didn't? 

AJ_O’NEAL: So it's a whole wide field there. And there are things, we have to differentiate between JavaScript and a programming language in general. Okay. So let's just put that on the table. So we had a language that existed that had certain features and certain patterns that worked well and certain patterns that did not work well. So, you know, when you start out a path, you kind of have to pick an opinion and you have to go down that path, let's say with integrity And so JavaScript was a language that had, you know, it had advantages and disadvantages and, and just in general, like if you try to please everyone, you know, if you, if you stand for nothing, you'll fall for anything. And I feel like that's what JavaScript became it. I got into JavaScript and you have to understand this, you know, for, you know, the, the context of the story. I came into JavaScript when there was strong leadership and that's something I appreciate in a community. I appreciate when there's a leader who is strong and willing to make decisions, even if those decisions are sometimes wrong or don't please everyone. I value when someone's willing to make decisions so that progress can be made over a leader that doesn't make decisions or wants to turn everything into a committee vote or things like that, because I feel like that's a weaker form of leadership in many cases. I know that there are lots of good positive benefits about that and what I said could be taken way out of context, but as a general rule, That's good. So when Douglas Crockford was the figurehead of JavaScript, that's when I latched onto it. And I was like, this is a cool language. And this guy who in my eyes is the leader of JavaScript has opinions that I align with that were very similar to the opinions that you see in Python and go, which is, you know, smaller language is a better language. There should be one way to do things that should be clear and simple to understand, et cetera, et cetera. So if you read the Zen of Python, or if you look at the go proverbs. You know, that's kind of where I personally fall into alignment. So there's a whole section here that can be completely disregarded because it's just my personal story of how I came into the language, what my expectations were. And then they got a, a big reset. So there's definitely stuff in there. Like I've come around now that async await is actually implemented. It seems like everywhere. And this is one of my things is like, it's okay if we have new language features, if they actually get implemented. I don't want to be stuck in a world where we never can write JavaScript ever again. And we're forever relying on transpilers. So as some of the features have actually made it into the browser, some of them, I test them out and I use them sometimes begrudgingly and I walk away and I say, you know what? This is useful. I can see how this is helpful to create more correct programs in a simpler way that is easier to read. Maybe not easier to write all the, well, you get trade-offs sometimes. You can create programs that are easier to read, but they're a little more difficult to write. Sometimes you can create programs that are easier to write, but they're a little bit more difficult to read, but you have, you know, some of that trail. So with that, that I'll just shut up there for a second and, and, and let some comments come in. So I don't. 

AIMEE_KNIGHT: That's good. I can see a lot of, uh, I can see a lot of your points there. I guess Dan, turn it over to you. Let's talk about generators. Maybe start with like, what are generators? What are iterators?

DAN_SHAPPIR: Before that, I would like to kind of make a short response to AJ in the sense that you can look at the additions as a whole and you can look at them individually. That means you can say, you can make a point that JavaScript is good enough as it is at a certain point in time and regardless of the merit of a specific feature, I'm not going to be adding anything more because it's done. It's good enough as it is and it's done and should not be extended anymore. And there's an alternative option that looks at each feature individually. Now, if we look at specific features individually, that I can definitely point out features that I consider to be useless, like the exponent operator that for some reason was added into the language is totally useless. Nobody asked for it. I don't know why we got it. And I've never seen it in use. But other things that have been added specifically, I think have merit. Big ints are actually a good example. I think that big ints have merit. I think that the numeric system in JavaScript had certain limitations that made it really difficult, almost impossible, to build certain types of applications properly. Those dealing with currency, for example, were really problematic in JavaScript. And big ints specifically, address this while not dramatically expanding the surface area of the language. 

AJ_O’NEAL: And they could have done it without expanding the surface area of the language at all. I am a little bit disappointed that they decided to expand the surface area, but here's the thing. Like you said, this was something that was essential. This was something that was a real problem that if you did not have a built-in language solution for it, your solution was crappy. You could not create a good solution for big ints outside of the specification of the language itself. And so for that, I say this was fixing a bug in the language. It's not syntax sugar. It's not something superfluous or extraneous. It's literally if you have the type of problem that big ints solve, which is a real problem in today's world, whether you're in the browser or a node big ints meaning for those of you that aren't aware big ints are arbitrary sized integers. So if you go into your console and you do 0.1 plus 0.2, you do not get 0.3. And that is a result of floats. Big Ents allow you to have higher precision mathematics that can solve problems that are either financial problems, cryptographic problems, etc. So Big Ents were something that without them in the language, cryptography is basically useless. But anyway, so Big Ents is one where it's like that had to be there because you cannot solve the problems without it.

DAN_SHAPPIR: Now I'm forced to make another comment. You know, a lot of people like to write blog posts or make videos or give conference talks in which they bash JavaScripts. And one common recurring pattern is exactly the one that you mentioned about the 0.1 plus 0.2. And that's actually not the limitation of JavaScript, the language so much as it's a limitation of the numeric system that JavaScript chose to implement. And the fact that JavaScript built-in just had a floating type and did not have a built-in integer type, certainly not a big int type. 

AJ_O’NEAL: Well, 0.1 plus 0.2 is a problem that was in every language except for like Ruby 1.9, and they reverted it because it created problems between systems. Because if you do a mathematical operation on one system and you do the identical mathematical operation on another system, if both systems don't give back the...I triple E representation of the value, then you cannot verify that it was correct. So languages that do not, except for where you're using high precision numbers as a, as a specification of your program, it is expected that you get back the wrong answer because that is how the, yeah, that problem is multi. If you go in Python, it's there. If you go in Ruby present day, it's there. If you go and go, if you, any language you go in 0.1 plus 0.2 does not equal to 0.3. It's 0.3, a bunch of zeros, and then a four. 

AIMEE_KNIGHT: That's a fair point. And I think, like, I know when I was, like, learning JavaScript when I first saw people kind of make fun of the language with this example, I was like, well, wait, let me understand what's actually happening. And yes, it's the same thing in all languages. 

AJ_O’NEAL: Yeah, and it's that way on purpose. But when you define in your API, this is a high precision number, it is okay to give back 0.3 as an answer to something like that, but it has to be defined as, this program uses big integers or high precision numbers, it does not use IEEE floating point. That just has to be known. 

DAN_SHAPPIR: And with that, let's move over to iterators and generators. And I intentionally want to start with iterators as a distinct concept from generators, and you'll understand why hopefully soon. I do want to say that it's going to be a bit challenging doing this using just an audio medium because usually I actually gave conference talks about this and in those, obviously I had slides demonstrating how the APIs look like and so forth. 

AJ_O’NEAL: Well, for the, for the sake of us understanding each other, would you mind putting up the, doesn't zoom have like a whiteboard feature and that will help us at least talk more accurately about us and have shared understanding as we speak. 

DAN_SHAPPIR: Maybe, or alternatively I'll just use the chat, but let's see if I actually even need to go there. Okay.

AJ_O’NEAL: And I want to know what does iterator mean in your heart to you? Okay. So we might both agree on that, but the way it's implemented in JavaScript. 

DAN_SHAPPIR: So, yeah, you're absolutely correct because inter iterators, a concept transcends JavaScript it's actually represented in almost all modern high level programming languages in one form or another. So iterator it's worthwhile to actually start with iterator design which is an object oriented design pattern. For those of you who are familiar with something like the Gang of Four book on design patterns, the concept of a design pattern is a recurring pattern in code that is assigned a meaningful name so that when we just bring up this name in a technical conversation, everybody automatically understands what everybody else is talking about. Or for example, if you're looking through a piece of code and there's a comment in there saying, here I implement such and such pattern, then you have an initial understanding of what this code is going to be about and how it's going to operate without having to go and wait through the details of the actual implementation and understand what's going on. In this context, a very well known, at least in the JavaScript front end world, design pattern is the MVC, the Model View Controller Design Pattern which is a rather high level pattern that kind of encapsulates the structure of an entire application. The iterator pattern or the iterator design pattern is a much smaller design pattern dealing with subsections of an application or code. So what does it actually mean? Well, in most high level programming languages, you have the concept of containers or sometimes even sequences. A container lets say that's an object which contains, I don't know if a better term, other objects. The most well-known one, the simplest one probably, is the simple array type container. But there are others like dictionaries or maps or sets or whatever. There are a whole bunch of container types out there. And there are algorithms. Algorithms are operations that you can perform on to do various computations, but in particular, you have algorithms that are applicable to containers. An example might be a sort algorithm. You take a container of elements and then you sort them according to some attribute or key that they have. Another one might be to find a particular item in a container or to reverse a container or to some a container of numbers or to find the maximum value in a container and so on and so forth. There are a lot of such algorithms, most of them fairly simple, some of them more complex. And the point is that a lot of these algorithms are actually applicable to multiple types of containers. For example, if I'm going to, let's say, find the maximum value in a container, then any container that I can traverse sequentially is relevant for such an algorithm. There is no real, from the algorithm's perspective, theoretically at least, there shouldn't really be a difference between one type of a container and another type of a container. And in this context, what do I mean by type of a container? So one type of a container might be an array where you move between the elements by incrementing an index value. Another type of a container that enables sequential processing would be, let's say, a linked list where you move from one element to the next by, you know, each element has the reference to the next element and then you go along these references. So far so good? 

AJ_O’NEAL: I think so. So if we take this more concretely, if I were to define, let's say that, you know, I have an array and I have some other type of object, if I could arbitrarily define a method that is called next iterable item. Anything that has this method should be able to go into an iterator and by some means, it should be able to produce a result that comes after the one I'm currently on. Is that the broad picture of what you're saying? 

DAN_SHAPPIR: Yes, you're kind of jumping the gun here. So what I was- 

AJ_O’NEAL: That's because these bad boys are so big and strong. 

DAN_SHAPPIR: Ha ha ha ha. So the problem is that initially, let's say I'm not decoupling between the algorithms and the containers, then I would need to implement the sum algorithm, for example, for an array. And then if I have a linked list, I would need to re-implement the sum algorithm for the linked list. An iterator decouples between the container and the algorithm. An iterator presents a common interface that you can move from one element to the next in that container, for example. And then the algorithm just works on the iterator using that common interface and doesn't really care how the container that it's iterating over is implemented. So you're able to have just a single implementation of that algorithm that can work over a bunch of containers. 

AJ_O’NEAL: So this is pedantic, but I feel like it's important from the computer science perspective at least, and perhaps from practical implementation perspective as well. Is it important as part of the definition of an iterator that you can guarantee that if you go over the same iterator twice, you get the same sequence? So, for example, the numbers are 1, 3, 5, 7, 13, 42. The next time you go over it, you get that sequence and not 42, 5, 3, 1, 13 or whatever it was I said.

DAN_SHAPPIR: That's not necessarily in the part of the definition of an iterator, but usually that would be a good practice in terms of the implementation. And in most cases, this will be how the implementation works because it would actually be potentially be challenging to have it not work this way. 

AJ_O’NEAL: I ask that because some languages like JavaScript doesn't have many of the container types that other languages do, or at least not implemented in a way like the others do, but typically a hash map, for example, every time you create a hash map, you're actually going to get a different sequence of keys in the map because something is different about the way they're inserted. Like in Go, for example, they, because this is a problem, they actually define as part of the language spec that the keys are randomized on purpose so that an implementation of the language doesn't choose a different algorithm that produces un-predicted results.

DAN_SHAPPIR: As I recall, JavaScript, though the ES standard was recently enhanced to specify that certain containers will always return their elements in a consistent order, where previously it wasn't necessarily the case. 

AJ_O’NEAL: Well, it's in all practical... 

DAN_SHAPPIR: We're slightly deviating from where I wanted to go though. Okay. All right. So anyway, but you actually touched on a good point of JavaScript and the fact that JavaScript was able to make do without iterators for a really long time. Because if I looked at other programming languages like C++, like Java, they had iterators almost from the get-go. So, you had something like an I iterator type interface that an object would implement in order to be considered an iterator, and then it could be fed into one of these algorithms. And the reason is that those programming languages had multiple types of containers from the get-go. JavaScript, on the other hand, at least initially, seemed to have more or less just almost one or one and a half container types, certainly from the built-in types perspective. 

AJ_O’NEAL: Just an object. 

DAN_SHAPPIR: So yeah, well, actually, most people thought, didn't think about, I think that most, certainly novice. JavaScript developers didn't really think about object as containers, which was an issue because in JavaScript they are, but they actually thought about arrays as containers. In JavaScript, you wanted an array, you used an array. You wanted a stack, you used an array. You wanted a queue, you used an array. You wanted, you know, almost any, you wanted even, you even had sparse arrays. So almost anything that you wanted to do in terms of containers, you just may do with arrays. And consequently, it kind of made sense to have all these algorithms just implemented directly on top of arrays. So you had array.prototype.map and array.prototype.filter and dot reduce and dot index of and search and so on and so forth. Fine. Sorry, not search and sort and reverse. So all of these algorithms were just available for arrays and no other type of container. And almost immediately, even though it seemed as if JavaScript had that just one type of a container, they started running into problems. Because first of all, even built-in, they had more than one kind. Because like you said, objects in JavaScript are actually containers because they're kind of directories that map strings to values. But not even just that, because you had the DOM. And DOM had node lists. And node lists looked like arrays. They kind of emulated arrays. You could access them via an index, like you would an array. Looked almost the same as an array. But they, because they weren't really arrays, they didn't have the prototype of an array. And as a result, they didn't have all these mapping, all these algorithms implemented on top of them. And instead you either, had to copy everything into an array, which felt stupid, or you just had to write straight on, you know, for loops. And what's the whole point of having algorithms if you can't use them? 

AJ_O’NEAL: Yeah, and then buffer is the same way. ArrayBuffer also looks like an array, but then behaves differently because reasons. 

DAN_SHAPPIR: And the ultimate one, and the ultimate one in JavaScript, by the way, is actually a built-in thing, which was the arguments object which totally looked like an array, but totally wasn't. 

AJ_O’NEAL: I love the arguments object because it's the only pointer type that we have in JavaScript. It creates real pointers. If you overwrite the value, the value gets overwritten in the source object. 

DAN_SHAPPIR: Actually that kind of changed. I think when you use strict mode that it doesn't work this way. 

AJ_O’NEAL: Interesting. 

AIMEE_KNIGHT: I'm going to nerd nerd out for a minute and Dan ask you or AJ a question. Do either of you know, like I know the node list like trick you turn into an array, but is the reason that they didn't do that from the start just because they didn't want, are there certain things that certain methods that exist on an array or properties, I guess that just won't work on a node list? 

DAN_SHAPPIR: I actually think it's the reverse. 

AIMEE_KNIGHT: Okay. 

DAN_SHAPPIR: Cause with node lists, for example, they're like two types of node lists as I recall. Those are the static ones which are calculated and then you just have them. But there are also dynamic node lists where you can actually perform operations in the DOM and stuff gets automatically inserted or removed from the node list behind the scenes. And- 

AIMEE_KNIGHT: That makes more sense, yeah, cause it would break the other way around that I was just saying, I'm assuming. 

DAN_SHAPPIR: And in any case, the DOM is distinct from JavaScript. Just the DOM objects may look like JavaScript objects, but as AJ likes to remind us, they're actually C++ objects that are compatible with JavaScript interface. 

AJ_O’NEAL: And then there was a point at which Google was playing around with replacing the DOM with JavaScript for performance reasons, but that turned out to fail there, like how much of the web does it break test? And I wish I could find the blog post about this, because it was fascinating, but 

AIMEE_KNIGHT: that sounds really interesting. 

AJ_O’NEAL: Whenever the Google developers deploy a new feature, they do, they do a, like, how does this break the web test? And they kind of like just deployed on a couple of websites and see if a monitor within Chrome, if it starts throwing exceptions that it didn't throw before and things like that, or I don't know exactly how they monitor it, but they have some way of monitoring whether or not the sites that they randomly target break that day or, or the users they randomly target have a bad experience that day. And one of the problems there was that. People so heavily rely on the buggy. Well, I'll call it buggy. It is spec'd. It's not buggy. It's just a bad design, but people so heavily rely on those faults of the node list and such that it, they, they decided not to fix it. 

DAN_SHAPPIR: The funny thing is that now we do have a JavaScript sort of implementation of the DOM in the form of the VDOM that you get in react and other frameworks that use a virtual DOM. 

AJ_O’NEAL: And jQuery? 

DAN_SHAPPIR: Uh, kind of, I think in jQuery it's more of a JavaScript. So yeah, it's wrapping it or enhancing it to an extent, but in the virtual DOM, it's literally JavaScript objects. And that's by the way, one of the selling points of the virtual DOM, but it's faster because it's just JavaScript object. And so again, moving back to actual iterators. So for, so at least it seemed that you don't actually need an iterator because it seemed that you could just implement the algorithms directly on top of arrays and be done with it. And that's in fact, what was kind of done, but it turned out that even from the get-go, this was not really the case because of node lists in the DOM, because of various structures in, in, in node, because of stuff like argue, the arguments object in JavaScript itself, even strings can be considered to be a sort of arrays of characters. Yet you couldn't really apply the same algorithms and instead various algorithms had to be re-implemented on strings kind of like the way that they were also implemented on arrays. And what by the way the funny thing is that we got this really horrible pattern where you would transform a not array or an array lookalike into an array by calling slice on it. So you would do array.prototype.slice.call on the, let's say arguments, passing the arguments object in as the this or a node list in as the this as a way of transforming a not array into an actual real array by kind of hacking the system to invoke the slice algorithm that was initially designed for arrays but was implemented in such a way that it could actually work on objects which weren't arrays assuming they look sufficiently like arrays. And in AJ, I see by the expressions on your face that... 

AJ_O’NEAL: No, I'm recounting... It's not even about what you said. It's recounting my struggles with the array buffer because the array buffer actually doesn't use numbers. You know, like in JavaScript, we just have a number type. But when you use an array buffer, JavaScript is a...Strangely typed language because it absolutely has types that are types in the way that we think of types in a language like rust or go It's just that they're there Unexpectedly you expect there to never be types in JavaScript in a number to be a number. When you work with an array buffer if you're working with a UN8 array and you want to say reduce it. You assume that when you add things together that you're going to be able to get a number that's greater than 255 and it turns out that you're not. That sort of thing with array buffers was what I was about. 

DAN_SHAPPIR: Yeah. The funny thing though, it's a funny thing with array buffers, like, you know, they're, they're arrays, but not quite. So for example, you will find that a lot of these algorithms that were implemented on regular arrays are also available on, on the array buffers. And an example of an array buffer would be, let's say a new int eight array or you in 16 array or whatever. But if you try and you check, if you do like you compare the map functions from one object from a regular array to an array buffer, you'll find that it's a different function. So, 

AJ_O’NEAL: and they didn't use to be there. They didn't use to be there. So it was easier to, and before they were there, if you did the hacky do conversion, if you'd get what you expect, but the one that's actually built in, you don't

DAN_SHAPPIR: Yeah, it's, it's funny or, or sad, depending on your point of view. But in any event, the thing is that from the get-go, the fact that they coupled the algorithms to the built-in arrays was problematic. That resulted in things like jQuery hacking this sort of thing and introducing its own each algorithm as being complementary to the for each that you had in JavaScript. And then and then also supporting for each by wrapping everything in real arrays. And then you got stuff like Lodash where it kind of created its own versions of all these algorithms and made them into standalone functions that could be applied to containers other than arrays. And that's more or less where we were. So JavaScript did not have iterators. And instead, if you wanted to have algorithms on top of other types of containers, well, what you usually did was you either implemented them yourself or what was more common, you just didn't use other types of containers, which made JavaScript poor language or a more crippled language to an extent. And because there are cases where you want to have more sophisticated containers, and then some of these containers were actually added into the language. So we got map and set and weak map and weak set as additional types of containers in JavaScript itself. For those of you who may not remember, map is kind of similar to objects in the sense that it's a dictionary, but the big difference is instead of having the keys just being either strings or symbols, they can be any object, they can be really any object reference map a DOM element to something. And I won't even get into weak maps and weak sets because I think those go beyond the scope of this discussion. 

AJ_O’NEAL: Well, so the, just to reiterate, a normal object in JavaScript, whatever the value you use as a key becomes stringified. So whether it's a number or it's an object or it's a function, two stringets called on it when it becomes a key. Right. And you're saying these other ones, it basically uses the reference, which is something that in JavaScript, we typically don't even know it exists, but it's using the reference of the object as the key. 

DAN_SHAPPIR: Exactly. Which means, like I said, that you can use any object as a key. And the common example is using, let's say Dom nodes. So in the past, if I wanted to tack on additional functionality to a Dom element, I would usually modify the Dom element itself. Now I can use a map or a weak map to map that DOM element to some other object that contains the additional functionality that I want to bestow on that DOM element as a way of kind of decoupling or not modifying, not touching the DOM element itself. 

Hey, you know how your ops team keeps a pile of scripts and wiki documents explaining how to perform those routine and emergency tasks that keep your applications running? They might call them runbooks or playbooks. Our friends at Octopus Deploy were thinking DevOps is about collaboration. So doesn't it make sense for runbooks to be automated from the same place as deployments? Well, Octopus Deploy is now the first deployment tool with native runbook support. And the best thing is your runbooks can share configuration settings and automation steps with your deployments. So go find out more at octopus.com. 

 

AJ_O’NEAL: I've never had a use case myself where that was necessary. I've had somewhere it might've been slightly more convenient. But I could perhaps be persuaded that like big ends, this is something that is inconvenient enough that it makes certain implementations non-practical. 

DAN_SHAPPIR: It's a problem. It's a too big restriction in the language that the only way to effectively map things to other things is to convert the first things into strings. 

AJ_O’NEAL: Well, I don't disagree with that. But again, it is JavaScript. And generally in scripting languages, we don't deal with those types of things because scripting languages are meant to be lightweight and simple and kind of like, you can learn this in a weekend, which I would say we have surpassed that point with JavaScript. 

DAN_SHAPPIR: Oh yeah. Long, a while ago. I mean, it's no longer websites, it's web apps. I mean, you know, I I'll take an extreme example, like the Wix editor. I work at Wix and we've got the Wix editor, which is an online tool for building websites, it's got millions of lines of code. It's huge. It's not a built-in development environment and everything. It's, it's just a monster. And so, and it's all done in JavaScript or TypeScript. And, you know, you need sophisticated data structures for some of the things that we do there. So yes, we do use maps and sets and weak maps for certain things. Anyway, getting back, so additional data, additional containers were added into the language, which just highlighted the fact that you now would need to either re-implement all of these algorithms on top of them or maybe come up with a cleaner mechanism for decoupling the containers from the algorithms. And unfortunately, so the funny thing is, is they introduced iterators as a means to do the decoupling, but then didn't really use them. So they ended up both adding iterators into the language, but also re-implementing the various relevant algorithms on the prototypes of the various data structures. So instead of creating, let's say, a search or an index of algorithm that's totally independent of the container, they just kept on adding implementations of that function on the prototypes of the various containers, even though they introduced iterators which do enable this decoupling.

AJ_O’NEAL: So when I think of an iterator in JavaScript, correct me if I'm wrong here, my thought was that an iterator is what 4x of collection operates on. That that was the new syntax. 

DAN_SHAPPIR: Oh, that's exactly what it, so yeah, so let's finally get to how iterators are actually implemented in JavaScript because we've been talking for a long time and we didn't even get to that point. So an iterator in JavaScript is a really simple thing. An iterator in JavaScript is obviously an object because everything in JavaScript is an object. And this object just has this supposed to have this one method. It can have additional methods, but it must have this one method to be considered an iterator. And that method is the next method. So it's just an object that has a next method. So kind of like you have then on promises or things that are like promises and you have the concepts of thenables in JavaScript for objects that have the then method. I like to think about iterators as nextables. objects that have a next method because in JavaScript we don't have interfaces. So you can't have an I iterator interface. So instead we use duck typing and you want to be considered an iterator, just be an object and have a next method. That's good enough. And when that next method is invoked, it does several things. It returns the current value advances you to the next item and also indicates if you reach the end. Some programming languages separate these operations into two or three distinct methods in the on the interface. In JavaScript, they're all done that one next method. 

AJ_O’NEAL: Does JavaScript have a cursor type of idiom? 

DAN_SHAPPIR: It's implemented internally in the iterator. It's not exposed outside. So it's assumed that the iterator knows its place. So the next method also returns an object. And that object just has two fields on it. So it's more of a structure than an object, really. It's got a value field, which indicates the current value, and a done field, which is a Boolean or a truthy field, actually, that indicates whether this is the last one or there are more to come. 

AJ_O’NEAL: Well, that's a missed opportunity on returning an error of type done. 

DAN_SHAPPIR: Throwing an exception, for sure. So that's the way that it works. So you just, let's say you're on an array with three elements and you've got an iterator over it. And let's say the values in it are one, two, three, you'll get the first invocation of the next, you'll get the value one and the done will be false. Then you'll get the value two and done will be false. And then you'll get the value three and done will be false. And then you'll get value whatever and done will be true. So that's the way that, and from that point on, if you keep calling next, it will just return that done true and that's it. It won't crash or explode or anything like that. 

AJ_O’NEAL: I truly am displeased that they chose to nest it in a new object that has to be garbage collected rather than having you return or throw a specific error type. Like that's the way it would have been. been done in many other languages or I, you know, there's other features in other languages, but JavaScript has this feature. It could be done that way. 

DAN_SHAPPIR: Well, on the plus side in JavaScript, objects are really cheap. They, they worked hard to make object allocation really, really cheap. So there's that at least anyway. So those are all the iterators really are. And, oh, one more thing in order to get the iterator from the, for a collection, the collection needs to implement a method that is named not via key, but rather via symbol. So there's the global symbol dot iterator. And so you invoke the method whose name is symbol dot iterator, and you get back an iterator for that collection. 

AJ_O’NEAL: So this is a, I'm a little confused on that. Did they enhance objects, normal objects, not maps, but normal objects can not just have string values, but can also have symbol values?

DAN_SHAPPIR: Yes, symbol keys. 

AJ_O’NEAL: Symbol keys, that's what I meant to say, yes. 

DAN_SHAPPIR: Yeah, that was what I was saying before, that in JavaScript, keys on regular objects could be either strings or symbols. I think we actually discussed this in the previous episode. It's usually done when you want to have a name that's either private or guaranteed to not conflict with everything. 

AJ_O’NEAL: Yeah, yeah, yeah. 

DAN_SHAPPIR: Now, why they chose the way to obtain the iterator to be a symbol but the next to be a string is beyond, you know, you should ask a member of the committee, but that's what they did. So yeah, so if you want to get an iterator for a collection type object, you invoke the method that's whose name is symbol.iterator. The way that you will write it, say X is an array, you would type X open square parentheses, symbol with a capital S dot iterator, close brackets and then open parentheses, close parentheses, you know, because it's a function call and you get back the iterator over that collection. AJ_O’NEAL: But we don't really worry about this because that nastiness is being handled by the syntax sugar of the 4x of collection. Are there other things that use, in the language itself, are there other constructs that use iterators than 4x of collection? 

DAN_SHAPPIR: Yes, the spread operator when it's applied on to an object that has an iterator. So when you apply a spread operator, it actually looks to see whether the thing that it's being applied to has this method. And if so, it obtains the iterator and then the way that it spreads the values, that gets the values in order to spread them is by using that iterator to step through that collection. 

AJ_O’NEAL: So you're telling me if I want to get the stupid query object out of the browser as its list of keys and values, instead of having a like for each over it and all that mess, I could just do a dot dot dot and assign it to an array and it would become an array magically? 

DAN_SHAPPIR: Yes. So anything that you can do with a for that you can apply a for off to, you can also spread into an array. Or specifically, you can spread an array into an array. 

AJ_O’NEAL: Oh, gosh. I'm glad that you told me that, because every time I come across that my... I'll get into this later. My biggest argument against the iterators is the way that they're implemented, particularly in things like the URL objects. 

DAN_SHAPPIR: Yeah, but spread is really cool. The funny thing, though, with spread is that initially spread just worked with iterable objects. That is, objects that had iterators implemented for them. By the way, a common way of, let's say, transforming a map into an array of values or the area of the keys would be to exactly use the spread operator. And instead of using concat, you could spread two arrays into a single array, for example. 

AJ_O’NEAL: Just use the normal way that everybody understands. It's easy when you read it, you know what it's doing. 

DAN_SHAPPIR: Well, you get used to this. Yeah, but like, like you know, idioms in the language, you get used to them. You're, you're familiar with concat and concat has its weirdness because concat behaves differently if the parameter that you give to it is an array or if it's a regular object. 

AJ_O’NEAL: Which is lamentable. Before we move on to that, you said that you can use the spread operator on an object to turn it into an array. Does that mean that if I just have an object and I put dot dot dot and I assign it, I'm going to get the values as an array? 

DAN_SHAPPIR: Okay. So this is the funny thing that they did which on the one hand I totally understand why they did it, but on the other hand it's kind of sad. So initially you could only apply spread operator to objects that were actually iterable, like I said, that implemented that method to obtain an iterator. If you tried to apply the spread operator to anything that was not iterable, you would get an error. But then along came React, and React said, hey, we pass params into components and very often we want to pass the parameters that we got, but just add one property. So they implemented a spread on simple objects. So if you wanted to like add a, let's say you got the params object and you wanted to add a single property to that object before passing it on, but you didn't want to modify the original object because it's not yours to modify, you know, the whole immutable concept then you would create a new object, spread the original object into it, and then also add that additional property. 

AJ_O’NEAL: Can you type in the chat what this looks like? Because I'm doing what I think this should be in the console, and it is not matching up with what I think we're talking about. So if you could take two seconds to type that out for me. I'd appreciate it. 

DAN_SHAPPIR: So it's, let's say, x equals open curly brackets dot dot dot y. comma a colon 42 close curly brackets for anybody who's typing alone. And AJ, now you're on mute. 

AJ_O’NEAL: Whoops. So this is basically like object out of sign. To me, this looks like a shortcut. Cause if I did object out of sign, empty objects, comma the thing, comma another thing that would be the same as what the spread operator does. 

DAN_SHAPPIR: Practically. Yes. Technically no, because it actually you're doing this as part of the object creation. Whereas object.assign, you're creating an empty object and then copying onto it. 

AJ_O’NEAL: And the practical difference in that is that the position of the keys would change because... 

DAN_SHAPPIR: Not necessarily. Let's not necessarily any practical difference, but you know, I'm being pedantic. 

AJ_O’NEAL: To me, it is important if the keys change, if the way the keys would iterate changes, which it would with this, the two examples we just gave, if you were to do object.keys, you would not get identical. Arrays back. And so I, but I can say that a practical, 

DAN_SHAPPIR: so the point is that you can actually apply spread on an object into another object and it looks like the same spread operator, but it actually works differently because in this case, no iterators are involved. 

AJ_O’NEAL: And, and it's not an array. It's, it's purely an object. 

DAN_SHAPPIR: It's purely, yeah. Taking stuff out of an object. It's more akin to, to. I don't know, like a for in rather than a for all. Yeah. 

AJ_O’NEAL: This is, this is definitely the kind of thing where I would say this does not belong in the language because it is doesn't fall, you're abusing one syntax and in a situation in which it doesn't make sense and then saying, well, it should make sense because it was convenient for me, like we had object on a sign, this is more confusing. It's less clear and it is doing a behavior that is not reproducible 

DAN_SHAPPIR: Basically introduced this concept React as a preprocessor anyway for JSX and people just loved it and pushed on the committee to include it in the language as well. 

AJ_O’NEAL: So it's a strong leadership that can say no you're still a child and if you do it again I'll spank you. 

DAN_SHAPPIR: Well I think things are what they are. So like jQuery influenced the DOM in this case React influenced the ECMAScript itself. Again so...We're starting to run out of time and I didn't even get to generators. I don't know even if I'll get to them, but anyway, so like I said, iterators are just an object that has that next method returns an object that has the value and done property. You can actually play shortcut with it because, you know, so for example, done doesn't need to be Boolean. It needs to be truthy. So for example, if you just don't put done, it's as if you have done undefined, which is like having done false. And likewise, if you reach the last one and done is true, then you don't even need the value because nobody will use it. So, you know, you can even play fast and loose with that. And the coolest thing is, is that in order to implement iterators this way, you don't need generators at all. I mean, creating an object that has, you know, it's a fairly simple interface. It's not that difficult to implement. And for that, you don't even need generators. And the concept of decoupling is great. And even if they didn't introduce generators at all and just had this concept of iterators and then actually implemented a sort of a standard library of algorithms that would accept these iterators and slowly wean people off of using, you know, the map and reduce and whatever that you have on the array prototype in favor of a standard library that has these algorithms that just accept iterable objects. I think that would have been a good thing. Unfortunately, they didn't do that. 

AJ_O’NEAL: But here's my question on that. So from the computer science-y perspective in the theoretical world, it's like, oh yeah, iterators are cool, right? I get what you're saying about the whole re-implementing something slightly different ways. And I think, yeah, that could be handled better. But I don't think we need iterators to do that. 

DAN_SHAPPIR: So...Yeah, but I'll give you, I'll give you a case in point where iterators would have been really cool in this context. So first, actually two. So first of all, it would have been rather easy to add iterators to, let's say, node lists. And all of a sudden it would have been much easier to just work with these array like objects and you won't, and you could get rid of all this code that copies them into arrays with all the associated overhead and nastiness. So that's one advantage. The other advantage is that you can actually use iterators with sequences rather than collections. And a sequence is something that kind of looked like a collection, but instead of having all these values pre-filled as it were, it computes them on demand. So now you can loop over something, but that something is generating values as they become needed.

AJ_O’NEAL: So I only know of two theoretical use cases for this. One is the Fibonacci sequence. Like say you want to, you know, you want to get the next, because the Fibonacci sequence is infinite. Or, I mean, you could even say the counting sequence is infinite, but. A sequence that's infinite. You might want to advance through, but I understand that theoretically. I understand it in the classroom. I don't understand how it applies to any program that I've ever heard of anyone writing. And then the other one would be something where it's like you're returning a random number. And, but you could just, instead of calling an iterator, you could just call a function that returns a random number. So is there, do you have, you come across a practical use case where an iterator is actually valuable to the average Joe? 

DAN_SHAPPIR: Well, okay. So, or more specifically, have I come up with a use case where sequences are relevant to the average Joe, because if you have sequences, then iterators can be a really cool way to work with them. And the answer is, for example, RxJS. we've got these whole case of, you know, reactive programming libraries that are all built around the concept of call them sequences, call them streams. Yes, they also have, uh, they add the synchronicity into the, into the mix. But, but the point being that it's, it is useful to be able to look at a sequence of values either immediately computed or computed over time as a sequence or as a string. And in your case, for example, and yes, you can, you don't need to use this paradigm in order to work with these. So for example, let's say you're looking at sequence of information coming over some sort of a network connection, let's say a website, you don't need, you don't have to use iterators for something like that. You can be kind of like a event-based have like callback function that is involved whenever more data comes in but being able to write something like this as A loop once you get used to the concept actually makes a lot of sense 

AJ_O’NEAL: So I want to get Amy's thoughts because she's been way too quiet and I'm sure you've got something intelligent to say 

AIMEE_KNIGHT: I Mean I've been trying to think and like chime in when I have thoughts but Well, that's what I'm saying. If I have thoughts, I've been trying to chime in, but I'm just like undecided. I, I mean, in a perfect world, I think people are, you know, developers would understand the wise and be able to make a decision on the trade-offs when they're using new features, but I fear sometimes people reach for something because it's new and exciting and it's not always the best solution, which can make the maintenance of the code harder for other people. I don't know. Those are my only thoughts, really. 

DAN_SHAPPIR: What I would say about iterators in this context, and again, it kind of saddens me that it didn't happen to this extent, is that I think that this is one of those language features that are mostly applicable to people who build libraries, frameworks, or other parts of the infrastructure. I don't expect to see a whole lot of iterators in user-land. But I think that iterators could be really useful in system-land or whatever, because the way that you use them, which is either just for all, or the spread operator is actually fairly straightforward. You know, there's nothing overly complicated about doing a for all or doing a spread once you get used to them. And you don't even need to know exactly how it works underneath. So being able to implement new types of collections and sequences in the language, I think is a really powerful thing. And unfortunately it's not really happening. And also unfortunately we're not getting that standard library of algorithms that could have been implemented on top of it. And I kind of implemented such an example of how such a library might look like. I actually have a blog post that I'll probably link to at least in the picks, where I show how you could create such a library and how useful it could be. And it has one additional advantage, which I'll quickly touch off because I know that we're starting to run out of time and I didn't even get to generators and I didn't even get to asynchronous iterators. So we probably deserve a follow on episode at one point or another. But another advantage is that let's say I'm chaining a whole sequence of algorithms. And that does make a whole lot of sense when you are using algorithms. So I might map and then I might filter and then I might reduce for an example. So I do like a.map something.filter something.reduce. And the problem with the way that it's currently implemented in JavaScript is that you generate intermediate arrays for each one of these steps. So you go to the array and map all of it. Then you filter all of that, and then you reduce all of that. If you're just passing iterators through, then you don't create those intermediate arrays. And it gets complicated, but if you read my blog post, you'll actually see how it works. And you can also just stop. So if I don't need certain values, I don't even compute them. So it also ends up in the code being a whole lot more efficient. 

AJ_O’NEAL: So I, this was actually a little bit confusing to me in Rust at first, because I, if I remember correctly, a lot of the functional functions operate on iterators and they don't create intermediary values. And so it is like one object makes a pass through in some cases before the next thing does or something I might be misremembering, but it was something like that where it was a little bit surprising before I understood it because I was thinking in this JavaScript context like you're describing. 

DAN_SHAPPIR: So I'll put a link to my blog post in the show notes. And then if anybody wants to see how it works, and it's actually a surprisingly small amount of code. So it's like you said, it's a concept that you need to wrap your head around for sure. You know my son, for example, is doing the computer science 101. He's just started studying computer science in the university, and he's trying to wrap his head around recursion, which can be really simple once you get it. And certainly, implementations can be really simple, but it's a complex concept to potentially wrap your hand around. And it's the same in this case, but it's like I said, it's actually a surprisingly small amount of code. So I think with that, we will probably need to end. 

AJ_O’NEAL: I've got, I've got two comments I have to bring in. I have to bring in before we end. 

AIMEE_KNIGHT: Okay. 

AJ_O’NEAL: So with the case of iterators, that the, the case where I think that they would be practical is where you have some sort of set or operation that tend towards unbounded memory or CPU, meaning like, you know, the list, it cannot be well defined in terms of how long it is. Like. For example, my list of favorite people, that's a bounded set by virtue of I can only have so many favorite people. I can't favorite seven billion people. That would diminish what the word favorite means, right? 

DAN_SHAPPIR: So if you have seven billion friends on Facebook? 

AJ_O’NEAL: Dan, you know me, right? I don't have seven billion friends. 

DAN_SHAPPIR: No, but Facebook friends, Facebook friends, they have the same thing. 

AJ_O’NEAL: No, I have a small, close circle of friends. I do not have a wide net of friends, but anyway. Where the collection tend towards unbounded is where an iterator makes sense because you can't fit all of it in memory at once. So, for example, if I have in other languages where we're interfacing with a database or a node we interface with a database, an iterator almost makes sense. In practice, it doesn't work because the database has trouble keeping track of the cursor while records are being written to it. And so sometimes you might get things out of order or not what you expect anyway. But You know, if you have a five gigabyte database, you want an iterator to go through rows of the database because you can't fit all of the rows in memory at the same time. So if you could not JSON dot stringify something reasonably to me, that is a case for an iterator. It's something that you cannot fit into memory or that doesn't make sense to try to represent as a whole. My beef with JavaScript iterators, aside from some of the things that we've discussed semantically, like is the way that they've been implemented. They've been implemented on stupid things that are very well bounded, that do not become large or that things that you do want to JSON.stringify. And my primary example of this is the URL object and like the headers object and things like that. Some of them you can JSON.stringify or like you can substitute an object for the weird iterator thing, but some of them you can't. But that's my beef is like a list of headers is not an unbounded set. It's like 10 in the extreme case and a hundred in the mega Uber unfathomably extreme case. 

DAN_SHAPPIR: Yeah. But that's, that's exactly the point that I was alluding to, or one of the points that you got so used to the fact that in JavaScript, everything is just an array that you're, you're automatically saying, well, why shouldn't the headers just be an array and be done with it? And And my concept is, just to finish, and my concept is that in a world where you have iterators, you can just do, you have any type of a collection that you want, and then you don't need to concern yourself with how it's implemented on the inside, because you just have an iterator as a means to go over all its values. And doing a for-of or a spread operator on them is a small price to pay.

AJ_O’NEAL: If it worked with JSON.stringify, then maybe I could come onto your side a little bit. Because the truth of the matter is, we need to be able to send things over a network. You can't send an iterator over a network. Yes, you can, but then you have to have the language on the other side implementing the same patterns and you have to be really in agreement. And that is not what most people want in a web API, I don't think. 

DAN_SHAPPIR: So just...Just to say that before we move to picks, in a future episode, we'll probably need to discuss asynchronous iterators, which also kind of touch on the stuff that you've been now talking about, and also generators as a means to create both iterators and asynchronous iterators. But beyond the scope of the discussion today. 

 

Do you want to increase your level of contribution to your company or team? Move into an architect or expert role? Do you want people at work to see you as that go-to person for technical knowledge? Well, let me help. I'm starting a program to help developers move up in their careers using proven techniques and by starting a podcast in order to advance. Right now I'm only scheduling calls to see where you're at, where you want to go, and how we can get you there. There's no sales pitch involved. You can schedule a call with me at devchat.tv slash next level. 

 

AIMEE_KNIGHT: Okay. And with that, let's see, Dan, you want to go first? 

DAN_SHAPPIR: Okay. Why not? So I actually have two picks for today. Pick number one is that I'm surprised and excited that we got the COVID vaccines so quickly. I did not expect us to get vaccines in 2020, especially given everything else that happened this year. I expected us to only get them sometime during the middle of next year. And what do you know? They have been, they exist, they've been delivered, they've been approved by the FDA, and people are starting to get them. And hopefully they actually work and don't have a horrendous side effect. And they, I know that they started giving them out in the UK first, then now in the U S and also in Canada, they've arrived in Israel. They will be starting to vaccinate people sometimes next week. And now it's only a question of whether or not you want to get vaccinated. And if so, when interesting times. 

AJ_O’NEAL: You personally, Dan, are you comfortable with the idea of mRNA that is editing your cells? 

DAN_SHAPPIR: Well, given that the alternative is an RNA by the virus that's editing my cells, it's an interesting question. So it's whether it's, you know, obviously in a world where, where coronavirus did not exist, I wouldn't be taking this vaccine. But when the alternative is to get the modification by the virus or by the vaccine, that