Kevin Newton works at Shopify. He is specifically assigned to the Ruby and Rails Infrastructure Team. He returns to the show to talk about building Parsers in Ruby since he is working full-time on a new Ruby parser called "Prettier Ruby Plugin”. Prettier works with the majority of editors and support different languages.  He shares his journey on how he was able to create it.  He also talks about "exreg", which is a Ruby regular expression engine that he created. 

Hey everyone, welcome to another episode of Ruby Rogues. I'm David Kamura and today on our panel we have Valentino.

 

You know.

 

and Luke Stutters.

 

Hello.

 

And today we have a special guest, Kevin Newton.

 

Hey there, how's it going?

 

Good. Kevin, would you mind giving us a introduction to yourself, who you are, who you work for, and kind of just all those reasons why you're famous?

 

I definitely don't know about Famo.us, but yeah, my name is Kevin Newton. I work for Shopify on the Ruby and Rails infrastructure team. I, for the last year and a half, almost two years, I guess, I had been working on YJIT with Maxime Chevalet-Pauver, who is the team lead and a couple other folks. And recently I just switched teams to kind of my own little sub team, rewriting the Ruby parser. You may have seen me around on the internet talking about various parsing things or various formatting things all to do with the front end of the language, but that's what I've been doing.

 

Awesome. So where do you even get started with working on the parser?

 

Yeah, it's been a bit

 

Ha

 

of

 

ha

 

a long

 

ha.

 

road. Yeah, it's a good question. I think I would say about 2017, I was working at a small startup called Culture HQ. There were only three of us. And I really wanted consistent formatting, but I really didn't want to spend any time dealing with it. And Prettier had just come out, I think, in like 2016 or 2015, something like that. And it was great. It was really good for the JavaScript ecosystem. assets, made it really easy, hooked into the editor and all that. And I really wanted this for Ruby. And at the time, the only Ruby formatter that really existed was RuboCop, which is really not a Ruby formatter. It's a linter. And it has formatting rules, but that's not really the same thing. So pretty rare ships with a plugin system. So I started working on that. And I got to tell you, I knew nothing. I knew absolutely nothing. I was trying to write a parser for Ruby using regex. And it was really painful. And so finally I like asked around, did I like learn some things and eventually I learned about Ripper and abstract syntax trees and like how the tree can be represented. And I started going down the path of serializing the AST with Ripper and then serializing that to JSON, passing it back to prettier and then formatting from there. And over time that had eventually worked And that was great. A lot of people started using it, but it's a huge pain because you have to have node and Ruby working in harmony and you have to communicate over either a socket or like a TCP server and that's terrible. Eventually I came to Shopify and I talked to Rafael Franca about like, Hey, can you use prettier? He's like, no, I'm like, why not? He's like, uh, yeah, we're not going to introduce like something that requires node into our like dev tools. And I was like, yeah, that makes a lot of sense. And so I rewrote the thing in Ruby. I submitted to the Ruby Association and got like a grant for that work. And that resulted in a package called Syntex Tree. And it was all well and good, but river is a huge pain to work with because it really doesn't give you a lot of things on the tree. There's a lot of extra information that you gotta keep track of yourself. And for some things, you have to go all the way back to the source So there's a lot of information you don't have. And not only that, but the parser itself can't recover from any syntax errors. So as you're typing and as you're in your IDE and you're typing away and you haven't finished your thought yet, it's always gonna be in a state where there is a syntax error. And so Ripper will just choke. It'll just give you nil back for the tree. You get nothing. And that's the way that the Ruby parser works. And over time, a lot of different tools discovered this problem. So Sorbet, for instance, uses a fork of a parser that's called typedRuby that was the 2.4 parser, which was a fork of the C Ruby parser. And that parser didn't have any error tolerance. So when you were working with Sorbet, every time that you had a syntax error, the entire file lost all of its types, then it had to retype everything again, and it was massively painful. At the same time, and all of the other rubies out there are all trying to keep up with all the syntax changes in the upstream. And they're always behind because necessarily that's what it means to maintain a separate fork. And so anyway, we saw all of these problems at Shopify. And the Truffle Ruby and Sorbet and C Ruby teams are all at Shopify. I'm sorry, small part people that work on those teams are all at Shopify. manager, Ufuk, I'm not gonna try to pronounce his last name, who he, I pitched him, I said, we need a new parser really badly. And he didn't believe me at first. And then he watched the Sorbet and Truffle Ruby teams really struggle and he was like, oh wait, we need a new parser. And so anyway, this is an incredibly long way to answer and a long way of saying like, it has been a very, very long journey, but there have been a lot of different problems and issues that we've been trying to solve over time that have come together and culminated into this one project that's like, we need a parser that everyone can use that is error tolerant, that has an easy way to access it that you can build linters, formatters, static analysis tools on top of it. And so, yeah, after a couple months I, KokoBun joined the widget team, Jay Miller joined the widget team. enough support that I was able to move off and start working on this new parser. And the hope is that I can deliver this thing and then everyone can kind of, you know, much like in the darkness bind them, everyone can start to use this new parser and benefit from from its maintenance.

 

I have a question.

 

Yes.

 

What is a parser?

 

That's a great question. So in the course of running a programming language, a programming language is just a program in and of itself. So CRuby is a C program. It takes files. If you run the Ruby executable and you pass in a path name to a file, like foo.rb, it's just going to receive a file. And it's just going to be a plain text file. And a parser's responsibility is taking that plain text it into a tree-like structure, which is just, say, a bunch of objects that have pointers to other objects, that it's easily, you can work with it in order to convert it into something that you can understand and then execute.

 

one of those abstract syntax trees we've been hearing about.

 

Yeah, there's a lot of steps in the transformation. So when you first get it, in the simplest example, you can imagine just a calculator that you have to parse 1 plus 1 times 2 plus 3. And you can almost see the tree in your head of the nodes are where the operators are and the leaves are where the integers are. And you build up this big tree. And then when you want to go execute it, down the tree and execute one node at a time. Operator precedence and associativity comes into play, but that's not necessary for this metaphor. But yeah, that is an abstract syntax tree. There's also a concrete syntax tree, which has information about where all those nodes are in the source. But yeah, generally just working with trees is like how a parser takes an input and then converts it into a tree. And then most languages will then and convert it into a bytecode, which is just a linear list of instructions to operate on, which turns out is more efficient to execute. And that was what happened with Ruby 1.9 with YARV.

 

So what do you use it for? I mean, I

 

Yeah.

 

know you'd use it for prettier, like auto-formatting, which is kind of funny to auto-format the code using the language itself.

 

Yeah.

 

So what are people using it for? And what could people use it for that maybe don't now?

 

Yeah, for sure. So, kind of the most, so there's a couple of things. There's any Ruby implementation, like CRuby, MRuby, JRuby, TruffleRuby, they all have to have a parser. They have to be able to understand how plaintext can transform into a Ruby tree that can be used to execute it. So you can use it to build your own runtime, basically. That is kind of the main function of parsers. So that's that. Accessing the tree, though, can be used for a lot of other contexts as well. So formatters, definitely one where you have to have access to the tree and then understand like some context about how to format it. For instance, you have to know if something is an if statement. There are two different forms of if statements in Ruby. It can be in the modifier form or it can be the block form. And so you have to have that kind of like semantic information to know how to format it. Any kind of linter. RuboCop is going to need access to the syntax tree. RuboCop is going to see things like you have an if statement and in the predicate, you have an assignment. And RuboCop is going to say, hey, I want parentheses around that assignment. You didn't have parentheses, I want them now. So linters, type checkers. Every kind of type checker is going to need access to the tree. It's going to take the tree and usually it's going to convert it into something called a control flow graph through the program, but the first step is going to be parsing it. And then really any kind of more advanced static analysis tooling. I saw a really fascinating one come out about Rust recently that was like wherever your cursor was in your editor, only the code that was relevant to that line was visible and everything else was grayed out. So if you have like a hundred line method and you were the stuff that had touched foo would be highlighted. Super interesting.

 

That's really cool.

 

Yeah. So yeah, so you need a parser for really any of those tools, anything that wants to understand Ruby code, the first step is gonna be to parse it into a tree.

 

You mentioned earlier that the parsers currently available weren't very error tolerant.

 

Yeah.

 

So that when you were processing your file and it had a problem, it would go crazy. Surely, surely that's a good thing. Isn't a good thing that stuff blows up when you've made a syntax error? Why could it possibly be useful to parse code that's not correct?

 

Yeah, that's a great question. That's a great question. There are two, well really there's just one main reason. When you're in your editor, well actually I can think of a couple of reasons. If you wanna run your code and you have 17 syntax errors, do you wanna run it once and then get notified about a syntax error, fix it, run it again, get notified about the next syntax error, fix it, run it again, do that 17 times. Or do you wanna run it and get notified That's kind of like the main thing. So you want to be able to recover from those errors and understand where all of the errors in the file are at once. The other reason is if you are introducing a syntax error into your code while you're typing, the majority of time the rest of the file is going to be fine. So if you have code that is doing type checking or formatting or linting, and it can lint or format or type check the rest of the file one section until it's valid and cache the result of the rest of the file. That way, your editor tools become much more resilient as you're typing. But yeah, you're right. If you have a runtime, if this is in production, then it doesn't matter. You definitely want to choke on the first error. So it's a little bit of two different modes.

 

I don't know if we're doing iOS development, then I think they'd be accustomed to getting an error each time they try to compile. Like, oh no, you got another one. Oh no, you got another one.

 

Yeah.

 

Yeah, that's most

 

Yeah.

 

of my working day you've just eliminated there.

 

Well, you got to remember too that there's like a there's syntax errors and then there's runtime errors

 

Mm-hmm.

 

and And then in a language like I like a Objective-C or Swift for iOS like a lot of that's gonna be like linker or compiler errors So like I'm not solving any of those problems. I am exclusively solving syntax errors. That is it

 

That's

 

And

 

very

 

so

 

cool.

 

how does that translate to our day-to-day development?

 

Thanks for watching!

 

So is this going to be available, its plug-ins, for VS Code or similar editors?

 

Yeah, yeah. So it's still being discussed. It's hard because there's like 17 different language servers for Ruby right now. I think the future of it, I mean, I'm entirely biased, but I think the future is going to be through the Shopify's Ruby IDE, which is like the main Shopify plugin. All of the Shopify developers have it in their system. It provides everything that the current default Ruby one does, but it provides also a lot more. So if you have it shows RuboCop errors in your editor as you're typing. If you have Sorbet, it'll enable like jump to definition and stuff like that. And I'm working with that team to once the parser is more, you know, available and ready to go, it's going to start introducing, you know, error messages for syntax errors as you're typing. So in your day to day, it should be faster because you shouldn't have to go to the command line to determine if you have an error. And so all of that is just like making a better experience for tooling. If you've worked in VS code with TypeScript, you know like how good it can be or Rust or even C. Like there's like a lot of niceties out there that the Ruby community doesn't really take advantage of yet. So yeah, editor tooling is really the answer. And you know, not just VS code obviously, like hooking it into Vim or Emacs or whatever the hell else everyone else likes to use these days. That is also the plan.

 

You know, it's funny, I'm semi torn on the whole idea of like auto fixing things as you're typing. I've done, you know, a little bit of Go and Rust, and I like that about it in those languages. And I think it's just because it's not Ruby. And I know how unexpressive it is,

 

Yeah,

 

for the most

 

yeah,

 

part.

 

yeah, yeah.

 

But like thinking about doing it in Ruby, like I feel like it's like a creative

 

Yeah, yeah, yeah.

 

But like I want it to be able to I want to be able to make mistakes and kind of just like, you know, let it let it error.

 

Yeah, yeah, yeah. I get it, yeah.

 

But at the same time, I don't, you know, but I like that freedom, you know.

 

I will say that I have maybe a controversial take, but I want to build these formatters and these tools for other people's code. I don't want them for mine. I don't turn, if the project is just me, I don't turn the formatter on. Because I like, you know, everything is like artistically indented and like beautiful for me. But like, if I'm working

 

Thanks for watching!

 

on a code base with multiple people, it needs to be consistent. I don't want to deal with it. So there's probably like, there's probably something to be said for like ego in there. But yeah, we're not delving into my psychosis just yet. What else can I answer about parsers? or not. I have other projects that I also work on.

 

I got a question about working on the parsers and working with Ruby language tooling too, because

 

Yes.

 

that will kind of automatically puts you at the top of a tree, doesn't it really? You're now the alpha developer because your code is coding the other people's code. So,

 

Thanks for watching!

 

you know, you're kind of, there is no, there's nothing more upstream than Ruby core. Let's face it, that's the kind of A game, right? God knows if I could code, I'd be doing that, but I can't and it's tragic. I mean, there's nowhere else to go, is what I'm telling you, is kind of what I'm saying, right? If you're making the thing that makes the language work, there's nowhere else, what could you possibly do? If you're working on the thing that makes the language go.

 

Well, I will say there's always another person. I'm always, you're always at the whims of someone else. In my case, GCC, Clang, any number of the, uh, Windows compilers, you're at the mercy of the compilers, you're at the mercy of C, you're at the mercy of whatever libraries are, are shipped as, as a part of C Ruby, uh, you know, it's like, there's always one more level. Uh huh.

 

I don't know, you've seen that thing that Aaron Patterson was doing when he was kind of like trying to generate machine code from Ruby, you've seen that?

 

Yeah.

 

He's kind of compiler bypass. I think, you know, you've got to watch him, he's dangerous.

 

He is, yeah. Actually funny enough, I was pairing with him yesterday on the parser. Yeah, he is dangerous. It's funny. Yeah, I mean, like that's kind of the thing we do with YJIT is we have our own like in-memory assembler that assembles X86 and RM64 code. And yeah, I mean, there's just always, there's always another level. I will just say that. I was, okay, this is a complete tangent, but like I was working on the RM64 code And I was writing out the assembly for a particular instruction. And Alan Wu, who is a little bit of a genius on my team, he was telling me that we should use a different encoding for this instruction because the micro ops were better for this other encoding. And I was like, what it micro what? He's like, oh yeah, it's what the assembly gets compiled down to.

 

Sainsbury 80s.

 

I thought that assembly was the bottom.

 

That's what I thought. Ha ha ha.

 

I didn't know there was another level. It's like we cut the atom and all these quirks came out. It's like there's another level to that?

 

Hehehehe

 

Are you kidding me? I really thought I had reached the bottom, but no. There's always another level.

 

I couldn't tell Aaron or we'd get down there.

 

Yeah It's quite a different world. I mean, I'm programming every day in C now, like exclusively. And I don't know, I gotta say, I love it. It's a lot of fun, but I know that's, it's not everyone's cup of tea.

 

What do you love about it?

 

It's very predictable. Like if you screw up, it's because you screwed up. It's not because somebody else screwed up. It's been around forever. Like you're not the, like you're not, somebody else isn't causing your issue. You are causing your issue. So C has like a ton of pitfalls. Like there's a ton of things you need to be aware of, but you know where every one of those pitfalls is. Like it's a little different from a language like Rust where it's like, oh wait, behaves in that way, but only in this case. And sometimes in this other cases, it does this other thing and it depends on how many traits you've implemented. Like complete, I don't know. Rust always surprises me with his behavior. C never surprises me. There's not enough things in C to be surprising. It's effectively like a hammer versus like an extremely powerful electric tool. Like it's a hammer. It's not going to surprise you. You just have like only so many ways to use it.

 

So speaking of C, why rewrite the parser?

 

Yeah,

 

Is it not written in C?

 

it is written in C. That's a great question. There are a lot of different parser technologies out there. So you can write your own by hand, which is common. There is also parser generators. Parser generators typically will take some kind of abstract syntax, not saying abstract syntax, some kind of higher level syntax. There is a BNF, EBNF, something like that. Maybe I'm thinking of something else. There is a syntax, I can't remember the academic name for it, but it looks a lot like how you would describe trees in a language, and it parses that itself and then generates a C file for you. The most common of these is called bison, or at least it was the most common since like the 70s. It was originally called yak and then it became bison, which is a whole other thing. Yak was yet another C compiler, or compiler compiler and then right yet another compiler compiler that was what yak was and then bison was a play on words on the fact that it's a yak anyway all of this is to say there are parser generators there's also like antler and Hackerad is a thing that gets used in a whole bunch of different kind of parsing generators. Anyway, all these are from parser generators. Parser generators are all great. Problem is, parser generators are meant to parse context-free languages. Ruby is not a context-free language. Ruby is a context-sensitive language. I may have just gotten those two flipped. So this may require editing offered. I'm not sure. Either way, there's levels of the complexity is from Nobis hierarchy, right? And Ruby is a more complex language than parser generators generally work for. So you can shoehorn a parser generator into working for this. You just have to track a bunch of state on your own, change stuff as you go, which is possible. It's totally fine. But it's difficult. And if you look at parse.y, which is the grammar file used in CRuby, it's like tens of thousands of lines of C code. incredibly difficult to understand. Only really like two or three people actually commit to it. Mostly Nobu, who does like 95% of the commits to it because he's been doing it forever and he's definitely a mad scientist. But like you just have to have a massive amount of context in your head. It's very hard to really reason about because a lot of it is just like abstracted away by the parser generator. Error tolerance in a parser generator is very, very hard to use. to create these things called error productions. It goes into something called panic mode. There's all of this rigmarole that goes into adding error tolerance. On the other hand, if you're just writing a parser from scratch, you're just writing a program. And you have complete control over everything. So it's a lot more obvious what's going on, because you're writing the code. So all of this is to say, yes, the existing CRuby parser is written in C. It's generated by a grammar file that then goes into C. But you're kind of at the whims that generate the Lexar, the tools that generate the parser. It also was written originally in 1993 and hasn't changed all that much. It's just shoehorned in new and new and new and new and new stuff. And it's never really come around as being rewritten. A lot of stuff in CRV has been rewritten over the years, like the hash tables, encoding, the JIT compiler. All these things have been rewritten. The parser has really never been touched quite in that way. So it's a little bit of spring cleaning also.

 

horrible question as a follow up to that.

 

Sure, let's

 

So

 

hear it.

 

it's really great that Shopify is investing very heavily in Ruby itself and associated tooling. But if this is a really serious rewrite of the kind of fundamentals of Ruby, what steps are you taking to make sure that people like Nobu are on board with this? Because that's a really powerful community. And we've seen in other, I mentioned, and Oracle is the obvious example that comes to mind where big companies come in and say, hey, we're going to make everything better.

 

Yeah.

 

And well, my sequel eight happens. So what do you know? I mean, can you talk about that? Is that too political?

 

No, it's not political. Yeah, I mean, it is political, but I can talk about it, it's fine. Yeah, no, we had a meeting with Nobuo Matsukoichi and Mame before any work was done talking about, is this something that you would be interested in? Is this something you would support? They've been updated relatively regularly about the progress. It's still very early, so I don't actually have that much to show, to be honest. But yeah, the plan is to, as soon as I have something really to show, very, very keenly involved. The other thing I will say is that the entire goal of this project is 100% compatibility with the existing C-Ruby parser and ease of maintenance going forward. So if either of those two goals aren't accomplished, we're gonna scrap this. Now, I don't want that to happen and I fully intend on making it work, but the goal is to make life easier on the C-Ruby maintainers. So this isn't like Shopify coming in and saying, we're gonna make it better. This is more like Shopify investing in me and the people that are gonna work on the parser and we are being a part of the Ruby community. So yeah, that was a bit of a long answer, but yes, they are informed and hopefully everyone will be very on board when they see the code that doesn't yet exist.

 

I think that's a great answer. I'm just being mean by asking it.

 

It's an important question because you don't want to come in and just be like, this is the future like that's not How it should ever work and especially not in a community like Ruby where people are generally actually a little more kind So yeah, we're definitely not trying to to dominate or dictate Much more about collaborating and making sure everyone is happy about the direction while going in

 

So I wanted to talk a little bit about YJIT, and,

 

Yeah.

 

cause I'm curious, like, a parser seems like a perfect fit to like work in collaboration with a lot of other things in Ruby.

 

Yeah.

 

Like, like YJIT, I imagine. Can, do you have any insight on, on that, on what maybe is planned to kind of fit in with the parser work?

 

Um, so, so they're kind of at different ends of the, um, the lifetime of the interpreter. So the parser is kind of the entry point into the, the, the funk, the, the program, right? It like, it gets file paths and then it just parses everything and then hands it off to, uh, the file in Ruby is called compile that C, but it turns it into a byte code and, uh, then why did it sits on the complete other end? while the bytecode is running, it sees where the hotspots are and compiles individual bytecodes or multiple bytecodes into machine instructions. So. Yeah, I don't think there's gonna be too much overlap because if we do this right, then YJIT should not have to change at all. YJIT should just be able to continue to look at the existing instructions because when we integrate the new parser, it should create the same YARV instructions as it goes on.

 

So is it reading from the grammar's bytecode? Like the grammar gets boiled down? So, I'm gonna go ahead and do that. So, I'm

 

Yeah, so

 

gonna go ahead and do that.

 

it'll convert a tree into a bytecode first. That's what Ruby does right now. So the way that it works, if you imagine we go back to our calculator example, we do one plus two plus three. That's gonna be, it's gonna be, well, I'm gonna simplify. It's gonna be like op push, which is gonna push one onto a stack. And then it's gonna be op push, push two onto the stack. which is gonna pop those two values off and push three onto the stack. Then it's gonna do up push, push three onto the stack, up add, and then it's gonna pop three and three and Adam gets six. So like those kinds of instructions are compiled from the tree. The tree looks like a tree structure that has like one plus two plus three, right? That's like your overall arching tree. Then Ruby will like walk that tree and make those linear instructions. Um, so right, there's that like level of abstraction that exists between the, the parse tree and the, the byte code. Um, so, so hopefully that byte code should, uh, should keep it from infecting the rest of the code base. So to s- so to speak, like as soon as that compilation is done, everything else should remain the same. if that makes any sense.

 

I see.

 

I'm not sure if that tracked, but...

 

Yeah, I'm trying to think about it from the runtime portion of it. Because doesn't the JIT do some optimizations there as well during runtime? Or is it currently

 

Um,

 

not?

 

it doesn't do any optimizations on the parsing end. It does optimizations over the bytecode, but, um, See Ruby does its own optimizations on while it's compiling. It does things called people optimizations. It does some things like specialized instructions. So if you didn't have any specialized instructions, then one plus two would always result in like op send. Send is just the generic like send a method. We send the plus method to the one. But in Ruby, there is a opt plus, which sees that there are only two operands does some special stuff. So that is all part of the compilation phase, all part of the compilation pass. So by the time it gets to YJIT, it's already OPT+. It's like already been done.

 

I gotcha.

 

Yeah, so those optimizations are done in the interpreter, not in the JIT compiler.

 

So is there anything from YJ that you've been able to apply to your parser style or is it just completely two different beasts at this point?

 

Two different beasts. I learned a ton about systems programming from Wejit. And that really helped. Just having the background honestly was very helpful. C Ruby is also like an incredibly complex code base with a ton of lines of code. So having the general knowledge of like being in that code base really helped. Also just like meeting people through Wejit work really, really helped. Like the human factor is massive. the people on my own team, Maxime, Alan, Aaron, Noah, Jimmy, Koko, Ben, they're all great people to work with and I've learned a ton from working with them. But then just also through WaiJIT, I've worked a lot with Ruby Core and meeting those folks has also been incredibly helpful. And it's funny because I've met people through WaiJIT, like Benoit and Ilya, who are both people who are helping me now with the new parser. Benoit works on the Trump for Ruby team. Ilya works on the parser gem. So like getting people from around the community to help out has been, you know, big for the success of this project so far.

 

I know that the YJIT design is based on, I think, part of Maxime's PhD work,

 

Yes.

 

which is kind of really, really cutting edge. I can't pretend to understand it, but I'm just going to shout that out, that there's some very special stuff going on.

 

Yeah, yeah, it's kind of amazing work. Once you like internalize lazy basic block versioning which is the technology that she invented, it becomes a question of like, why were JITs ever not written this way? It's pretty genius. In very brief, I'm gonna see if I can do this. I don't know if I can do this, but in very brief, when you are writing a function, an opcode in Ruby, like opt plus, it's gonna check, are both of these operands integers? And if they are, it's gonna do a fast path for integer addition. Are both of these operands strings? Because you know, strings have plus, it'll do a fast path for string concatenation. Are both of these operands floats? And traditional method-based JITs will encode that type check into the machine code, because they have to, because it doesn't know in time, so it writes all that out to assembly. And that's how MJIT works by using the compiler to compile that function down into an assembly version. YJIT instead waits until that instruction is about to be executed, the opcode. And it says, what is the value on the stack? Does the stack have an integer? OK, then I'm only going to compile the code that's going to do integer addition. And I'm only going to do that. and Ruby are what are called monomorphic. You don't usually have like foo.bar, foo doesn't change type a lot in your code. Like the promise of duck typing has not been achieved, right? Most people, when they're calling a method, are calling it on a single class. So by having this kind of lazy basic bug versioning that only compiles specialized code for the type at runtime, you actually end up reducing the amount of code that you need to generate, quicker to generate code and less code is generated as a whole. I don't know if that helped at all, but that is the very, very general idea. There's also all kinds of deoptimization and stuff that get put into effect. But yeah, that's the top level layer of it.

 

to ask about regular expressions now

 

Let's do

 

and

 

it. Ha

 

I hate them

 

ha!

 

and when I first started programming Ruby I was working in a code base where someone someone had used regular expressions for everything that they could possibly use them for

 

Ugh.

 

and yeah and I swore a solemn oath that when I was coding nothing and nothing would be a regular expression everything

 

Yeah

 

would be using dot split and that's good

 

I love it.

 

As you get older, as you get older, then the promises you make yourself, it's like parenting, you know, all the things you said you'd never do come back, and suddenly you find yourself in the situation. And you remember those words you said to yourself that you've never used regular expressions yet here you are validating an email address at 2am.

 

Hahaha

 

You've got a project called xregge,

 

Yes.

 

expression library and I thought that's an amazing idea. A library of regular expressions, a place I can go and just

 

Ha ha ha  

 

borrow an email validator, right? Somewhere I could kind of just quickly get an American post zip code out of a random string. It's not that at all, is

 

No,

 

it?

 

it's not.

 

It's not a literal kind of smorgasbord of regular expressions. What is it?

 

It is an implementation of a regular expression library written in Ruby. So like regex, when you look at regex as a tool, it is effectively a state machine. Like if you have A, B, C, it's gonna say, you're gonna start your initial state and it's gonna say is the next character A. And if it is, is the next character B. If it is, is the next character C. And that is the state machine. You can model those out with like circles and lines, right? I have implemented this in Ruby because I want, so, so O'Nigbo is the regular expression library that Ruby uses right now. I'd like to have a pure Ruby regular expression library that we could apply some things that I've learned from YJIT to compile it down to, to JIT compile it down to assembly when necessary. The reason this is interesting is because to my knowledge, no pure Ruby rejects engine. There are a lot written in C, there's one written in Python, there's one written in Rust, most major languages have their own. But, you know, Enigmo is a separate library maintained outside of Ruby, that is Ruby's regular expression library. And I think that as YJIT gets better, having more things written in Ruby written in C is going to actually allow war optimizations to happen in the JIT compiler. So anyway, all of this is to say it is a regular expression engine. You can write your own regex using it and match them against input strings.

 

Kevin, I don't want to understand them. I just want to copy paste them in from

 

Yeah,

 

a stack overflow.

 

right, so, so, I mean, if you don't understand them at all, I can't really help you, but if you wanna understand just a little bit, if you wanna get a better, slightly better understanding, xreg actually has some interesting things where it can dump out any given regular expression to a diagram that will show you how it's

 

A

 

going

 

diagram,

 

to execute it,

 

but

 

yes.

 

like an actual like SVG?

 

Yes,

 

What?

 

literally an SVG.

 

Oh my God. All right. I'm on board. I'm back on board. Here we go.

 

The,

 

Wow.

 

the,

 

You buried the lead there. It doesn't even

 

yeah,

 

say it on the readme.

 

no, it doesn't. Yeah. I'm, I've still got a lot of work to do on it, but one of the things that I want to do is to make it so that if you're, uh, to have a language server that runs next to your editor where you can click on a regex and it will all, all of a sudden open up with a diagram that you can like see how it will execute. And then there's actually some, There's one for JavaScript where you can write in test strings, and it will show you what gets matched. So a better understanding of how regex works and a better explanation for the operators that exist in there, like if you have a dot star question mark that changes your greedy matcher into a lazy matcher. So I want to be able to highlight over the star question mark, and it'll show up documentation for what is this operator doing. Um, you know, regex is really, really obtuse and, uh, has, it's one of those things. It's just like, it's like a. Passed down with oral tradition, like hearing about like how this thing even works. So like having documentation built into your editor and, and, and explainer for like how this regex is going to function is, uh, is the overall goal. The other thing I will say is that the onigmo, okay, this gets a little technical, but onigmo is a regular expression library that uses non-deterministic finite automata to match against input strings. What that means is that for a given input string, the way that it models its state machine is that for any given input, it could actually end up in multiple states at once because something could, like a question mark, like a question mark, if A is there and if A is not there. And over time, it will have to do something called backtracking if it finds that, oh, actually, I didn't match. I need to go and try something else. That kind of backtracking can lead to something called catastrophic backtracking, which for a given input, you're going to make this regex work for like a full minute. And that can result in something called redos, which is regular expression of denial of service with a carefully crafted input string. If you have a bad regex that is vulnerable, to read us, you can crash that web worker because it's just sitting there working on it forever. There are multiple techniques in the industry and straight up academic papers to convert a non-deterministic finite automata into a deterministic finite automata or a lazy non-deterministic finite automata. There's like a bunch of academic pieces to this. You can't accomplish everything that you do with the same operators because there are and mashing negative and positive lookaheads and lookbehinds, those don't work quite as well in a DFA, but for the majority of Regex, you can convert them into something else that is more deterministic. And when you look at my Regex engines performance versus Enigma, which is the CRuby one, and you graph it out on a logarithmic scale for certain bad inputs, you see theirs just goes stays flat. So for some security reasons, this is actually, in my opinion, a better way to go because it mitigates those kind of security issues that come from a redos attack. Anyway,

 

Well, I think...

 

I don't know if I just lost everyone there, but...

 

how, listen, we've been putting the Ruby parser into panic mode. I still wanna know how to make the Ruby, I've never seen panic mode. I wanna hear more on that. Now we've got kind of DDoSing people with RegExes. Any excuse to not use RegEx is a good excuse.

 

Hahaha!

 

So yeah, that's good. I'll use that one in the next code review. Thank you very much.

 

Yeah, you're welcome. Yeah, it's only specially crafted regex, but it's surprising how quickly it happens. And it can happen really accidentally. That's the thing is like, regex is so poorly understood by the general community, including me. There's a lot of stuff in there that I can't remember. But we use it. We use it all the time. It's a very, very, like the rails, it's a very sharp knife, sharp tool use. So yeah, in Ruby 3.2, they introduced a timeout to every given regex. So you can like make it timeout after a certain number of seconds in case it doesn't match. But you know, that's kind of a that's kind of a blunt instrument for something that is a security hole.

 

You know, for me personally, I think RegEx is powerful hands down. What you can do with it. It's really cool. I like it. However, it's one of those things where I'm not familiar with it enough because I don't use it every day. And when I do use it, I don't go very in depth. I just do what I needed to do. But it's one of those things where I can come back to it a few weeks later and have no idea what it's doing.

 

Yeah.

 

You know, I have to copy that regex find some online Matcher that I can then paste in some examples to figure out what it's doing

 

Yeah.

 

You know, so that's my issue with regex is that I think that ultimately it's not very maintainable for the average developer

 

Right. Yeah, I think that there are ways to mitigate that though by, well, first of all, providing a lot more documentation than we currently have, but also by providing tools like a example generator, like take a regex, pass it into this thing, get out a set of a hundred strings that all match this pattern. And you can see what kind of thing is actually being matched here.

 

Yeah.

 

So like there are ways in editor, you know, that we can try to illuminate that which is currently very very unmaintainable as he said. I think we can we can get there

 

Yeah.

 

but it's definitely tough.

 

And with VS code, they had something similar to that for CSS. A lot of the issues I had with CSS was just the visibility of what is this actually going to affect.

 

Yeah.

 

And so when you scroll over, hover over a CSS thing, it'll now in VS code, it tells you an example of what is this actually going to match.

 

Yes.

 

So it's really cool. And I would love to see that for RegEx and other things in Ruby as well.

 

Yeah, for sure. Yeah, the more we can move toward editor tools that tell us what is going on and explain this without you having to leave the editor, the better off we're gonna be. I didn't realize I actually was super interested in this for a while until I realized that all the all the things that I had as side projects revolved around making this stuff kind of more accessible and making it you know more approachable for the average person.

 

You know, it's kind of funny because regular expressions are often like, I need to match this. Oh wait, no, I need to match this with this exception. And then this exception. And then eventually somebody introduces a regular expression that's instead of just string matching is, you know, okay, match these patterns. Uh,

 

Yeah.

 

but it's almost like we're, we're training ourselves. Like we're AI and we're, you know, we're, we're feeding ourselves. Like, you know, the information we need to auto correct all of these, like, I would like a regular expression that could just learn itself to match whatever it's given. Like,

 

Yeah.

 

hey, watch this string for the next three months and then match anything that matches that.

 

Yeah, for sure. I mean, honestly, there is a paper I saw that generates a regular expression from a list of inputs. And I think we could

 

Oh

 

100%

 

nice.

 

do that. You give xreg a text file. It gives you out the reg ex that will match it.

 

Yeah, totally.

 

It's possible. I mean, it might end up with dot star. But it should be possible to match that. And yeah. I mean, like, I'm obviously not gonna work on X-Reg all that much for the next, like, six months as I continue to crank on the parser and all the other stuff I'm doing. But yeah, at some point I'm gonna come back and I'm gonna start pushing really hard on it and I think that that is, like, top of mind is, like, making these things, like, not only easy to read and understand but also easy to write.

 

So I wanted to briefly ask you about, because I saw you were going to be talking at RubyConf Mini,

 

Yes.

 

and I'm super interested in these mini conferences

 

Yeah.

 

that may potentially spree into many other mini conferences.

 

Yeah.

 

How did you get into wanting to do that? And how has the experience been? What are your thoughts initially on it?

 

Yeah, I mean, I've talked to RubyConf before, but yeah, when they announced RubyConf Mini, in response to, I believe at least the response was in response to some social conditions created by the environment of Texas, of where the contract was. You know, I want to be supportive of that, absolutely. I want to make sure people are and heard and like they have a place to go. Part of it is also that I'm in Boston and it's a 45 minute drive for me and it is far easier for me to get to Providence. So I'm excited about that too. But yeah, I'm excited to see how it works. I like smaller conferences. The smallest conference, the first one I really went to was Ruby on Ice in Germany in January. I don't know if they're day that was the best conference I've ever been to. It was incredible. They served non-alcoholic hefeweizen for breakfast. It was amazing. It was like a single track conference. Everyone in the same room, seeing the same talks, everyone discussing the same things. I freaking loved it. So like the smaller conference feel is really, really nice. And so I'm very excited for Ruby and she's amazing and she's doing great work. I'm really excited to see everything that they've put together. But yeah, I'm just generally very excited about it. I think it's going to be really good and small conferences are awesome.

 

Today I learned that they made a non-alcoholic hevavison.

 

Yeah. Yeah.

 

never knew that.

 

Neither did I. And so when they first served it, I was like, what? Are you

 

Ha

 

all

 

ha ha.

 

serious? Like, it's not am. And they're like, no, no, no, non-alcoholic. Yeah. Yeah.

 

That's really cool. Because

 

Yeah.

 

I mean, I grew up in Bavaria. So I mean, you want a beer, you just drank beer.

 

Yeah, right.

 

Never knew that was non-alcoholic. Didn't know that was an option.

 

Yeah, it was an amazing conference. It was in Bavaria, actually. It was in Togernsee. And it's gorgeous there in January. And my wife came and she immediately rented a car and drove south to Switzerland to go skiing.

 

That's awesome.

 

And yeah, it was really fun.

 

I've always wanted to make it to keep Ruby weird.

 

Yeah.

 

I love the content that comes out of that every time. Ha ha ha.

 

Yeah, Keeper Ruby Word is definitely on my list. Brighton Ruby as well, definitely on the list. I know that, I think they're single track, but I also just saw Rails Sass, which I hadn't, somehow I had missed it, but it looks like an incredible conference. All these amazing things I saw on Twitter.

 

Yeah.

 

Yeah.

 

How do these people get to every one of these?

 

I don't, they don't have kids, I think is the answer.

 

That

 

They can't

 

was...

 

possibly.

 

Hehehe

 

Yeah, it's not that I couldn't go to the conference. I don't want to do that to my wife

 

Yes.

 

going to a conference. Because I mean, I have four kids that are under 10 years old. So it's like, it's just mean.

 

I just have the

 

So.

 

one two-year-old and even I feel that. So

 

Hehehehehehe

 

yeah.

 

So I think Rails Conference is going to be in Atlanta next year.

 

Nice.

 

At least if my memory serves correct. And I live in Atlanta, so that will be my first conference that I go to. Ruby-related

 

Nice!

 

one. So.

 

Nice, yeah. RailsConf is great. RailsConf is, you know, there's like a, because RailsConf is web, there's like a whole massive wide variety of talks. There's the, you know, the social stuff of like scaling teams and managing. And then there's also like the, there's JavaScript, there's CSS, there's actual Ruby, there's backend, frontend databases. Like it's a wide swath. It's really, it's really fun.

 

Yeah, I'm excited to drop in on the M. Ruby Conf.

 

Oh, are you going to that?

 

Well, I'm not going, I'm gonna watch it remotely, but

 

Yeah.

 

participate, if you will.

 

Yeah.

 

But I love how it's getting more concentrated. Like, you know, you have all these shoot-offs, you know, specialist-style conferences that focus on a very specific thing.

 

Yeah.

 

I like that a lot.

 

Yeah. Emory B is a wild other world that I have yet to delve into, but it seems awesome.

 

Is the parser going to be planned to make its way to mruby eventually?

 

Yeah, yeah, actually, it is. Matt's mentioned he's like, Yeah, I want to replace mRuby's parser with this one.

 

Oh, awesome.

 

Which would mean because I think mRuby is so like pretty behind in terms of syntax. So I think it would it would I mean, like it would bring pattern matching, for instance, to mRuby.

 

Oh cool.

 

Yeah, among a lot of other stuff.

 

I have a few circuit boards here I'm trying to get working with mRuby.

 

Nice, yeah. I also saw this project called Pico Ruby, which

 

Oh yeah,

 

I

 

the

 

hadn't

 

keyboard!

 

heard of before. Yeah, yeah, yeah, that's the tiny little stuff. Yeah, it seems awesome. So yeah, a lot of fun development stuff going on there. You know, it just the community is, it's a big umbrella. So it's fun to see all those different things being developed.

 

Yeah, I'm excited to see how this parser will kind of explode a lot of other stuff, you know.

 

Yeah, I'm very hopeful. It's, you know, I don't want to promise too much and I don't want to promise too early.

 

We're depending on you, Kevin.

 

Yeah, great. I will mitigate everything and say, it's not coming this year. It's probably not coming next year either. Like, this is going to be a long road, but I will say, like spring of next year, you're going to start hearing about this a lot more. I'm going to start.

 

That's not good enough, Kevin. You need to start

 

Ha!

 

stepping up.

 

I'll work on that. I'll work on that. But you know, spring of next year, that's the plan is I'm going to start publicizing it more, talking about it more. And I, you know, truffle Ruby will start experimenting with it. CRuby will start experimenting with it. Sorbet will probably start working with it. myriad parsers and instead are able to just use this one shared library because it doesn't have bindings to CRuby internals. It's just like its own like lib Ruby parser. You will just install this one package and it will just work in theory.

 

Well, I'm looking forward to using it with my new Lunar Vim. So

 

There you go.

 

I could finally get true language autocompletion and documentation in line.

 

Yeah, it should

 

It's

 

be

 

hard

 

fun.

 

otherwise. You've got to ad hoc a bunch of stuff.

 

Well, if you start using Syntext Tree today, Syntext Tree is the evolution of Prettier, it's a formatter, but it also has a whole object layer. If you start using it today, I can guarantee that by the time the new parser comes out, the day the new parser comes out, I will release a new version of Syntext Tree that takes advantage of it. So start using it now because it's not going anywhere. It is going to be the official version. my official version at least, and is going to be using the new parser the first day it's ready. So yeah.

 

Awesome, good to know.

 

Cool. Well, I'm going to get us pushed over to PIX. Kevin, if people want to find out or just follow you online, where should they go?

 

Yeah, I'm on Twitter, KDD Newton. Um, and, uh, you can see me on GitHub also KDD Newton, but yeah, mostly Twitter. Um, I publish on my blog, KDD Newton.com. Um, but I'll post all the links on Twitter.

 

Awesome. All right, well, let's get over to PIX. Valentino, do you have any?

 

Yeah. Let's see. My first one I've been revisiting some. I actually just got a time hop notification with this thing IDEO came out with. It's called the Design Kit. It's like a field guide. They call it human-centered design. It's easy to get lost working with computers all day. checking out to at least see how you can design for people again if you're not or forgot I Love it. It's pretty great. They have a hardcover book that you can get that's kind of cool looking and the other one I got is I may have used this before, but Mame has this incredible Ruby program that generates itself through 128 languages in a quine loop. He calls it the quine relay. Absolutely incredible. I recommend you check it out. It's the most absurd looking thing, but it's a lot of fun to watch and run it in on your console.

 

Awesome.

 

I saw that thing. It is wild. Absolutely

 

Hehehe

 

wild.

 

And Luke, do you have any picks?

 

so many picks. I have so many picks. Usually I steal my picks from the guest Twitter feed, but I'm not gonna I'm not gonna do that this week because uh because Kevin's been so amazing. So my first pick is Starlink. I've been operating last week in middle of nowhere in Canada on Starlink and that thing is amazing. You got to fill it with it, but my word was that deliver. It was much much Starlink is my first pick. My second pick is something which I know you've all seen before but it's new to me and it's grugbrain.dev. I've never seen this before, we've got a new guy in to do some rail stuff and grugbrain.dev is a marvellous, marvellous website, how to software from the mind of Grug. My third pick which I alluded to is a site called never write another regular expression ever again, just copy paste them in from there. Don't think about it, don't think about it, don't test it, just drop it in, it'll be fine. And my other new technology I found is a thing called Emmet. And I know you've all heard of it before, but I'd never seen it before. Emmet.io is some kind of magic plugin in VS code that just writes the code for you, But no check it out if you don't believe it is kind of like a kind of weird shorthand plug-in auto generate stuff I've been playing with Doing some ERB stuff with it But I just it just types in for you. It's amazing and My last pick because I want to pick something from your blog Kevin is an article you've written on bit mask intermediates on the new M1 chips something which is dear to my heart because While on Starlink I had to port a fairly hefty amount of C++ over to my M1 Mac because we had an equipment failure on site. So I had to get everything running. And of course this coded only ever seen x86 before. It had never verged onto the ARM world. So that was a fun Sunday evening. So those are my picks.

 

Awesome. And because I haven't been on for a bit, I have a few picks. One is the Blackmagic Design 810, which is a live production switcher. So if you have a nice SLR camera that you use for your web camera, and typically you would go into an Elgato Cam Link or something like that. But doing the live broadcasting, you're doing it all from your computer. you're using CPU resources for all of that. The ATEM basically absolves you from that, and you can do everything from the ATEM with the live streaming. It takes in your HDMI ports. It's basically like a fancy HDMI switcher. So I've been playing around with that, and it's really, really cool. The second thing is I like bonsai trees, and so I found a Lego bonsai tree. 800 piece bonsai tree kit and you can have a Lego bonsai tree that will never die on you and the last one

 

What? Dude, is there a high mortality rate among your bonsai trees?

 

Well, see, the thing is I bought initially this nice little cactus plant, little planner arrangement from Home Depot, and it thrived in my office. But I usually keep the lights off in my office and I just have a little desk lamp. And I don't know, I thought cactuses like hot weather and

 

Right?

 

light, you

 

Right?

 

know, because it's a plant. But no, so it thrived in my office. So I thought, you know, I'm going to put this little guy outside, It died. It was hot and it died out there. I guess it just got

 

You

 

accustomed to the darkness. So, so I ended up getting a succulent Lego set for my desk to replace the poor little lifeguides. And so I don't think that this is going to die out. I can know for those on the video. I can show you a picture of it with in the background. So you see back here I have my little succulents. I had baby Groot tending to them and then the Lego bonsai tree so I love Legos my kids love Legos. So we get to play with it a lot and So me switching the cameras like this. This is with the atem. It's super cool And then finally the last pick this is something that I was playing with last night was WebAssembly and And I'll kind of key up why I was looking into this. And specifically, the library was FFM Peg Wasm. And the idea that I had was you upload videos to YouTube. You have to wait for YouTube to do all of the transcoding to multiple formats with the bit rate streaming. And the number of servers that they have with that is insane. So what if you were to leverage Individuals computers that is doing the upload to do all the transcoding So actually built a tiny rails app where you upload a video before he even uploads On the browser with web assembly. It does all the transcoding right there on the browser Using temporary storage and then once it's done with some stimulus controllers. It attaches it to some file inputs Uploads it to the active storage then you have your uploads. So WebAssembly is really cool, and I'm really excited to see the future of it. Right now, it's actually real crappy and slow with the current FFmpeg implementation. So I don't think it's in for prime time yet, but it is still very promising. So Kevin, do you have any picks?

 

Yeah, a couple things. The first one I would say is, speaking of Mame's coins, he just he creates the most amazing things. I'll link to this, but he created a coin from like 2013 or something that is a program that models fluid dynamics using ASCII art. And the input to the program is the program itself. And you can watch as the characters descend and flow around and you can like to watch it and it's absolutely wild. I mean, it's a different way of thinking about programming, but it's beautiful, honestly. So that's the first one. The second one I will say is justforfunnoreally.dev. This was written by a guy named Tim Morgan who maintains a Ruby implementation called Natalie. And Natalie is a C++ ahead of time compiled Ruby implementation. had ability with CRuby, but it's getting there. And he's maintaining it for fun. And he creates this thing for fun because it's fun. And every time this thing shows up on Hacker News, everyone's all like, why would you do this? Like, it's not gonna be fast enough. It's not like the best. It's not gonna beat anything. And he always links to this website that he created because he got tired of answering that question of like, people can do things for fun and it can be okay. And that resonates with me a lot because I have a lot of side projects I don't really intend on replacing everything I write these things with. A lot of the time it's just for fun and a lot of the time it's for education. And if we lose that, if everything has to be better than something else, then it's gonna take a lot of the joy for me out of programming. So anyway, that is that. Check out Natalie for sure. It's a great project. You can learn a lot about Ruby by implementing Ruby, which seems to be my MO of learning about parsers by implementing parsers and learning about regex by implementing regex. So yeah, those are my picks.

 

Awesome. Well, I really appreciate you coming on and talking to us.

 

Yeah, thanks

 

Was.

 

for having me.

 

Was a lot of fun. Right. Well, that's all for this. Thanks for listening. me