They dive deep into the world of advanced development techniques and tools. In today's episode, Valentino and Scott explore the challenges and breakthroughs in the realm of AI and code generation with a particular focus on Ruby.

Valentino vents about the complexities of prompt engineering, while Scott shares insights on prompt design, function calling for structured output, and the importance of breaking down code for reliability and testing. They highlight the declining effectiveness of prompt tricks and the growing significance of using large language models over fine-tuning.

The conversation delves into the unique advantages of Ruby for rapid prototyping, with Scott Werner, CEO and founder of Sublayer, shedding light on his self-assembling AI agent framework built in Ruby. He explains how their prompt architecture fosters easy expansion and customization.

Listeners will also hear about Scott's experiences with ramping up on new teams, operationalizing experience, and designing classes and interfaces to align with desired coding styles. The episode concludes with discussions on experimenting with AI tools, the value of community events like the Ruby AI Builders Discord and meetups, and the insights gained from fine-tuning language models.

Get ready to explore the cutting-edge of AI in development and the potential of Ruby in creating powerful, automated solutions. Tune in and join the conversation!

Socials

Valentino Stoll [00:00:04]:
Welcome to another episode of the Ruby Rose Podcast. I'm your host today, Valentino Stohl, and we're joined by a very special guest, Scott Werner. Scott, do you wanna introduce yourself? Tell us why you're Ruby famous here.

Scott Werner [00:00:17]:
Yes. Ruby famous. I wish. I hope someday. Thanks for thanks for having me, Valentino. It's, it's it's super exciting to to be here and be a part of this. Yeah. So I'm here.

Scott Werner [00:00:28]:
I'm the, CEO and founder and cofounder of, of Sublayer where we're building, a self assembling AI agent framework in Ruby, and kind of supporting tools and infrastructure programming with AI and Ruby for and we were just talking, programming with AI and Ruby for and we were just talking, earlier. Basically, exclusively trying to get AI to write all my code for me and not me actually write the code, and primarily Ruby.

Valentino Stoll [00:01:06]:
That's awesome. So, just so we're clear, you're talking about sublayer is the project?

Scott Werner [00:01:13]:
Sorry. Yes. Sublayer is the gem. Sublayer is the gem. And so yeah. It's, you said, you know, we've been describing it as a a self assembling AI agent framework. And, kind of what I mean by that is that each of the each of the components that we've built are designed so that it's easy for an LLM to generate new additional components that you might need. So the the, the framework is very minimal, and it's really about kind of defining the, the conventions, the interfaces, and then kind of letting you lose to to expand on that for whatever you want in your application.

Valentino Stoll [00:01:54]:
I'm I'm super curious here. Like, where do you where do you start something like that? Like, how did you get involved in this? Right? Like, how do you just stop coding?

Scott Werner [00:02:07]:
It was tough. You know, I think I think a lot of us were really inspired when we saw chat gpt 4 chat or we saw GPT 4 come out and people using chat gpt and, you know, posting videos of I use this through this prompt in the chat gpt, I copy and pasted the code into x code, and I have a running app. And I've never coded before. And, you know, saw that and immediately was like, oh, well, those people are going to wanna wait to add functionality and modify that software over time. And, kind of realized that, a lot of the stuff that we do in XP, around like TDD and and the different refactorings we do, lend themselves really well to work with LOM. So breaking things out into small pieces. The smaller request the simpler request you have for the LLM that you the more likely it gives you something back. That's that's good.

Scott Werner [00:03:14]:
You know, and also kind of had this had this conversation with people of, like, you know, you would be mad at me if I gave you a 1, 000 line PR, but at least you could come over to my desk and yell at me. Right? If you got a 1, 000 line PR from ChatGPT, that would be a big problem. And so kind of with that line of thinking is where we we kind of started our journey.

Valentino Stoll [00:03:35]:
That's really funny. And I I just think of the cost associated with that too, for something that you don't want.

Scott Werner [00:03:40]:
Right. Right.

Valentino Stoll [00:03:44]:
That's really cool. So I've I've definitely checked out sublayer. I've used it to to generate specs because I don't like write writing specs.

Scott Werner [00:03:52]:
Oh, yes.

Valentino Stoll [00:03:53]:
I I'm not a test driven person, but I do drive tests. So I'm more of a backwards test driven developer, whatever you would call that. And so I use it. Yeah. It was a little script that I run, sublayer, and I say, hey. Generate a, you know, Rspec test for this Ruby file.

Scott Werner [00:04:13]:
And

Valentino Stoll [00:04:13]:
for the most part, it does pretty good. You know? That's fantastic. Yeah. So I I love the, the kind of syntax, for just like, hey. Here's a generator. Use this prompt to do it. It's super cool.

Scott Werner [00:04:29]:
Thanks.

Valentino Stoll [00:04:30]:
So why don't you give us, like, the the high level, like, how does this all work? Right?

Scott Werner [00:04:37]:
Yeah. A little, you know, a little bit of background like how and why it works is, you know, we we started calling it promptable architecture, but the the core idea was that there were going to be patterns, styles, ways that you build your application that lend itself well to LOMs taking a short request and giving you something back. And, you know, some of the things we started out with things we kind of already know, you know, clear interfaces, single responsibility principle, loose coupling, and kind of came about those through, you know, reasoning of, you know, it's going to be much easier to add code to your app if you can just drop that new file in rather than have to update a whole bunch of other different files. And, you know, other things like, having these conventions that you can throw into the prompt and the LLM will just believe you that that's your convention and follow it. And so that's that's been a lot of kind of our trial and error figuring out what are these what are these patterns? How do we how are we going to structure our applications differently to take advantage of this? And so, you know, the 1 of the main things around the the sublayer gem is, you know, every single component is very loosely coupled that, you know, you could have a 100 generators, you could have a 1000 generators, and they won't affect each other. And so, you can you know, 1 of the 1 of the things that we've been toying with is, like, maybe, you know, don't repeat yourself or duplication in your code base isn't that bad. If it takes, you know in the the docs, you can see that you can generate a generator in, like, you know, 10 seconds. Maybe it's not that bad if, you know, you and I both work on the same story if, you know, it's us just describing it and getting the code, and, it doesn't affect anything else in the code base.

Scott Werner [00:06:31]:
So so there's there's promptable architecture and so, like, that's been kind of our drive for this gem. And 1 of the the things that we found works really well with that is that you really only need to give the LLM 1 example to generate something new and so generate a variation off of that, which is where our our kind of main tool Blueprints comes in. And basically what it does, which I think is, you know, has been pretty common experience from a lot of the developers that I've talked to, 9 times out of 10 when you're working on a new user story, there's somewhere else in the code base that you do something similar or there's a pattern you probably wanna reuse. So you go, you look at that file, you repurpose some of the code, maybe copy and paste, and, you know, keep the code base very similar to what you're doing, which we kind of realized is a manual human version of Rag, which, you know, I I made a joke to somebody that, like, my development style is retrieval augmented generation. Very good at googling and then looking around the code base. And, you know, it was kind of mostly joking, but then kinda took a step back and was like, actually, maybe there's something to that.

Valentino Stoll [00:07:49]:
That's really funny. I I never thought of it like that, but, yeah, we are just, we're just doing rag.

Scott Werner [00:07:58]:
Right? And so realize that you can actually kind of operationalize that. And the you know, 1 of the things that I think for you you join a new team and what takes so long to ramp up is that experience of, oh, yeah. When when we when we run into that problem, we do this pattern over here that's in, you know, this file on, you know, this line. But like you only get that from months, years of working on that team, that intuition. Whereas you index these things, you put them in a vector database, and you query them, you can find them. And, you know, I we built that. We originally, actually, on the sublayer gem, thought we were going to build a whole bunch of different, you know, components and generators and stuff. But, on our product planning meeting on Monday, we're like you know, we basically thought we're gonna do about this much work this week and had the idea of maybe we'll just generate 1 of these generators.

Scott Werner [00:08:58]:
And then a whole work weeks worth of work was done in a day. And it was like, take a step back. What did we just kind of, like, discover? So, like, you know, building we found, like, building in that method building this method are, like, very loosely coupled components, small components, can be very powerful, and quick.

Valentino Stoll [00:09:21]:
I didn't realize that you have a a list of of predefined blueprints, on the site. That's really cool. Thanks, Adrian. Go ahead. Yeah. That's super cool. I was gonna say, the you know, the this mentality is makes so much sense. Right? Like, I feel like I've been at places before where they've had just, like, generators in in general, like, rails generators, right, where, okay, you just, like, wanna start up a new Rails app.

Valentino Stoll [00:09:47]:
Like, you use a generator. Like, you wanna create a new model, that has all of the conventions. You use a Rails generator for that. Like, it definitely has helped, like, in a consultancy, for example, like, you know, where you have all this boilerplate stuff that you just want to build off of. It's like, well, you use the preexisting stuff, and then at least you have the shell there and it's all hooked up and wired correctly, and then you can build off of that. This is, like, the next step after that. Right? Like because then you can then start filling out the stuff in that boilerplate, right, to adapt to whatever it may be

Scott Werner [00:10:23]:
Right.

Valentino Stoll [00:10:23]:
Which is really cool. And so, like, how does it how well does it work? Right? Like, I think that's, like, maybe, what a lot of people hesitate to jump into all this with. Right? Like, you know okay. So you have all this source material for it to use. Like, isn't it just making mistakes, though?

Scott Werner [00:10:45]:
So it can. Actually, 1 of the things that I've kind of realized and been trying to figure out how to how to describe. Right? But it's almost, like, by intentionally designing your classes and interfaces a certain way, you can almost, like, target the hallucinations to where you want them, which, you know, it makes mistakes, but they a lot of times are mistakes in the style that you would probably want anyway, which I know doesn't really make sense. But, like, so in the actually, what I think Wait. What do

Valentino Stoll [00:11:21]:
you what do you mean by that?

Scott Werner [00:11:25]:
So, like, well, you know, 1 of the things if you, you know, go ahead and generate something on the doc site, you'll see that they we have kind of this idea of an LLM output adapter. And right now it says, you know, there's a symbol for single string. So we're basically the the simplest generator you can make is here's a string, I'm gonna send a prompt to an LLM, and I want this specific string back. And like you said, like, your your example is the string could be our spec tests for, you know, some Ruby code that I've got. But if you ask it to build a generator that maybe gives you a list of, a list of refactors to make in that Ruby file, it might hallucinate that it needs a an LML output generator that generates a list. And while that might not exist, it'll still be in the conventions and the style of that of the library, which might exist, but also might point to, okay, well, here's the next feature we need to build. We need to flush out what it means to get a list back. And so having those having those spots where there's there's nothing different from the base example to the new example you're trying to make.

Scott Werner [00:12:42]:
But the places where it's different, you can basically design your code so that those are the pieces that change. Output adapter might be a little more complex. The the thing that I think you run into a lot is it just generates a new prompt for what you're trying to do and doesn't touch anything else in the rest of the code. And so I'm thinking about it like I'm targeting the hallucination to that prompt or to the to the initializer rather than anything else in the in the code base or in the in the file.

Valentino Stoll [00:13:12]:
Gotcha. Yeah. That makes sense. I I have so many questions around the Blueprints aspect of it. And, I mean, I I guess, like, where does where does somebody even get started? Like, is is Blueprints a good place to get started with the gen cogeneration aspects of it, or is that, like, a first test the waters with sublayer, and get used to the generators aspect of it? Like, where do you suggest somebody you like that's, hey. I have got a team of people. We have a conventions in place. We wanna start adopting this process.

Valentino Stoll [00:13:45]:
Like, where do you start?

Scott Werner [00:13:49]:
So yeah. I guess 1 1 place you can start, we just, you you saw on the the site, we just rolled out some new functionality where you can sign up, you can start to put some code examples in privately and start to generate from them to see how it how it acts. And that that could be a great place to kind of start to start to play with, promptability, which we've got a whole bunch of features we're gonna roll out, kind of being able to score or point out areas where, you know, where it might make the code more promptable. And so so that's 1 aspect if you if you're curious about playing with with your own code. And then on the sublayer docs, we actually and I'm sorry. This is very meta. But, we've actually created this this interactive docs product which we're we're going to be rolling out as well where, live in our docs where you can come and go to any of the the components. We have generators and actions right now and see it in action just right in what you're trying to do and it will give it give you reasonable code.

Scott Werner [00:15:00]:
There might be some things you depending on what you're trying to do, might not be implemented yet. But 1 of the things that we're doing, if you go to the running there's a there's a guide on the left about running local running a local LOM with LAMA file and what Andrew on our team does is actually in the demo of setting it up, go to the doc site, generate the code for the demo, runs it on the video. So I'm sorry. We're doing a little bit of a little

Valentino Stoll [00:15:33]:
bit of a

Scott Werner [00:15:33]:
little bit of a little bit of a cursive version.

Valentino Stoll [00:15:36]:
I was on earlier because I I had looked at Blueprints a while ago, and I I hadn't seen Sublayer's latest stuff. And this is so cool. Yeah. You just go on to the site, and you can, like, yeah, just generate this stuff. It's so cool. Sorry if we're inflating your bill here, but it's it's maybe why it's there. Yeah.

Scott Werner [00:15:57]:
Right.

Valentino Stoll [00:16:00]:
Yeah. So I'm I'm curious, like, you know, definitely local inference is, you know, a hot topic, and people are trying to you know, it's getting faster, and MacBooks are running it quicker. Is that something that you recommend people start with, though? Like, is it fast enough to, like, make make, like, the development flow, smooth?

Scott Werner [00:16:24]:
It's it's not yet. I would probably recommend starting with something like, you know, I think for for some some of the stuff that we're doing, you know, $5 in, in OpenAI API credits, can kind of help you get started. I think Google's has a free tier for even Gemini 1.5 Flash which is super fast. You know, we're we have we're using, you know, some of the, some of the models on our site And, you know, you can also just go and play with way play with stuff in with private blueprints. For the local models, I kinda do think that's where where it's probably going for for at least a lot of stuff. The problem I think the the the prohibitive thing about that is that you need a very powerful laptop to really be able to run the local models fast enough and powerful enough to actually get good results back. So, like, 4, 000, $5, 000 MacBook Pro or $5 in the open API or open AI API credits.

Valentino Stoll [00:17:48]:
I remember I I first saw the, you know, the first open models, and I I go and I, you know, install on my Raspberry Pi. So I'm thinking, oh, yeah. You know, like, I'll I'll just throw everything out a chain of Raspberry Pis, and it turns out you can't really, like, spread inference like that. It's not it's not it's not really there yet. And it's also super slow on anything, like, even on a, like, a nice m 1. Right? So, like, it it's kinda funny. Like, you sit there and you're, like, watching. It reminds me of the old days of, like, waiting for an image to load on dial up or something.

Valentino Stoll [00:18:24]:
Right? Like, you're waiting for the sentence to complete. Yeah. So I think I

Scott Werner [00:18:30]:
think it also you know, the risk you run is that it gives you the wrong impression. Like, if you're if you're just getting started, it's definitely good to throw $5 at some of the, like, frontier models like Claude 3.5, GPT 4 o, or or Gemini just to see what's possible. Because you can get the wrong impression if you're, like like, this llama thing just, like, went off the rails and started talking to talking about something unrelated. Right?

Valentino Stoll [00:18:58]:
Right. Yeah. So I'm interested in, like, have you found, like, certain models perform better for, like, certain tasks, like, related to cogeneration? Or are there ones that are specifically better at at generating Ruby code or, like, generating certain kinds of of Ruby code? Or or even just, like, are there certain tasks with cogeneration, like generating the specs for x or or something like that where some models just do a better job than others? Or is it kind of like just use any 1 of the top ones?

Scott Werner [00:19:33]:
I my opinion and my approach is kind of really just use the top ones because they're really those are going to be the slowest, most expensive, worst models we're ever going to have again. You know, it's just getting better and it's just scaling up. So, like, getting, you know, getting used to what's what's possible, and you, you know, even if you run to the wrong run into an issue, like, it's more about the UX and, like, what is this, you know, what does this imply if you can do this? I was telling talking to somebody about, you know, remember remember when everybody's like, oh, AI image generation is never gonna work because it keeps getting people 7 fingers. And, like, I can't remember the last time that was, you know, part of the conversation. But it was all about, like, here's a great way to generate an image. Ignore the hands for for now.

Valentino Stoll [00:20:23]:
Oh, that reminds me. I I saw recently, some some university has the the 3rd thumb. Where you're gonna actually, like, you you use your feet to, like, you like, to get, like, another thumb, like, prosthetic on your hand so that you can, like, you know, hold stuff while you're, like, doing something else with 1 hand. Mhmm. They show peeling a banana with 1 hand. You know?

Scott Werner [00:20:51]:
That guy's that guy's clearly AI generated.

Valentino Stoll [00:20:53]:
Oh my gosh. It's so funny.

Scott Werner [00:20:57]:
But I guess to more to your point, it's actually 1 of the things we've been trying to figure out the right way to present this and maybe it'll come out as something interactive on the blueprint site. But, we found that different attributes or different, like, different different attributes of code quality that we've talked about for a long time, are actually kind of quantifiable now where, 1 of the experiments we were running was, you know, having having really descriptive good names and then generating off of that blueprint that, you know, GPT 3.5 did really well. When you made those names worse or less descriptive, it had a harder time figuring it out, but GPT 4 could. And so, like, you know, that kind of goes that follows a lot of the scaling for the other models as well, where if you give it an easier problem to solve, it can solve it. But as soon as that problem gets harder, you need a more powerful model, which I've been testing out with with some people. Like, we might be able to put a dollar value on technical debt because it costs you a dollar per 1, 000 tokens versus, you know, 10¢ per 1, 000 tokens.

Valentino Stoll [00:22:15]:
That's really funny. Outsourcing the jobs we don't wanna do Right. To AI. It it's all a a cost per token at this point.

Scott Werner [00:22:27]:
It's it's always been.

Valentino Stoll [00:22:30]:
That's that's, it's an interesting idea. Like, you know, definitely refactoring makes a lot of sense. I I have a, a special GPT I made, like, that, you know, has a bunch of resources that I've collected over the years at its disposal to, like, help me refactor stuff specific to Ruby. And, it works great. And having that extra knowledge base, like, is awesome. And I'd use it all the time just for, like, rubber duck sessions. Right? Which I know the next step would be to then, you know, have it actually change stuff, which I haven't spent the time to do. But, like, now that, you know, you've made it so easy with sublayer, I'm definitely gonna revisit it because, that that's definitely a task where, like, you could just say, hey, like, generate a pull request for me, like, you know and all the GitHub APIs are there.

Valentino Stoll [00:23:19]:
Like, it could use at its disposal. Like, I I have you played, like, you with tool usage in that way yet, as far as, like, the sublayer, connection?

Scott Werner [00:23:31]:
Yeah. And actually, I guess, even to that that task is is kind of the where we see things going that, we we're kind of looking at it like, you know, the those kind of automations. Once you have that LLM and you have a, you know, an easy way to take get some data, send it to an LLM in a specific way, and then get something structured back, you can build things with that. And, and you can build things with that very quickly like that, you know, take this take this refactoring or these, you know, these suggestions make a pull request or, or, you know, kind of anything you can think of. We've been we've been toying with, with 1 idea where it kinda sits in our, sits in our video chat for our planning meeting in the beginning of the week that, watches the video, check the manages the transcript, analyze the transcript. And then anytime you know, you when somebody asks a question about a story and then you get an answer and then you forget to put it in the story and then, like, a week, you know, a few days later, what do we decide here? We can just build an automation to kinda pull that out and put that in the story for you. And, like, that's a super quick task that doesn't take that much work anymore. Right? And and so very long winded way of saying we've explored the the tool usage but I think we're looking at it a little differently than how the providers are offering it.

Scott Werner [00:25:06]:
You know, we we absolutely rely on function calling but only for only for like the structured output aspects of it. And relying on, you know, things that we're good at and things that we can do deterministically and reliably, which is, you know, call APIs and, you know, pull call make make requests and make you know, make get requests and get make post request, edit files locally, that kind of thing. But it matters to, like, get the the parameters that you're going to send to those APIs. And then we can do that deterministically. But, like, have the have the LLM do what it's good at and have us do what we're good at.

Valentino Stoll [00:25:55]:
Yeah. That makes a lot of sense. Do you see it more of, like, moving in the way that kind of GitHub is pushing with, like, configuration steps? Or is it, like I'm curious if you've played with, like, the the opposite of, like, the object oriented aspect of, like, okay. Here's a prompt and, you know, it can have all these extra functions and stuff wrapped into the the code aspect versus, like, you know, the configuration outline where, right, where it seems GitHub is going that way to fit better into their CI structure, to be honest, right, where you just have this YAML file and then it has prompts and can perform actions and stuff in it in that way. Like, where is that flexibility more advantageous in, like, putting it in your code versus, like, in something like a configuration file?

Scott Werner [00:26:53]:
That's a good question. You know, I we were definitely, coming at it from a a different perspective. You know, I think a lot of the the multifunction calling, the the the GitHub workspace, are very kind of conversation driven and so you have those functions that are structured ways for it to get in more information to do the next step in the conversation, which can be powerful. It can be very powerful. And, I mean, we've seen a great a bunch of examples of the power that that is possible. But I think from maybe this is just, you know, the thinking wishful thinking about software engine the future of software engineering that, you know, a lot of the principles that we found while building, while not exactly right when working with an LLM, still matter. And so, you know, 1 1 thing that I'm nervous about and haven't why I haven't really gone the route of like multiple different functions for it to choose from is, kind of the same reason why, you know, it's it's bad to have multiple it's bad to have a dozen parameters in your function. Right? There was a there was a quote that's something like, if you have if your if your function has 10 parameters, you probably forgot 1.

Scott Werner [00:28:25]:
That, like, a lot of those things while folksy and kind of in intuition and not, like, backed by any science, really, I don't know, still feel right to me. I still feel like you want to know that, like, this is gonna happen, this is gonna happen, and this is going to happen even if there's some, you know, fuzzy nondeterministic thing happening here.

Valentino Stoll [00:28:54]:
Yeah. I agree with that. I'm personally against the step by steps with with with LLM usage in general. Mostly because, like, you kind of want observability into that, and and kind of like a way to intercept those steps. Like, as an example, I see, like, the usage over time starts to become more, like, reactive than it does, like, input output. So, like, especially once you start using agents more or things that are, like can communicate with each other, it becomes like an observability thing. Right? Like, you end up having things that, like, are watching, you know, what is happening and can help redirect, right, if things that start going wrong or interrupt. And it when you have that step by step process, it starts to make it harder, to to change that.

Valentino Stoll [00:29:50]:
Right? Because it's it's a it's got the code flow. I'm like, okay. Well, you tell it to go to a different point in the code flow. Like, then it's like instruction jumping. Like, what are we back to see again? Right? Like, assembly.

Scott Werner [00:30:01]:
Like Right.

Valentino Stoll [00:30:03]:
I I use Ruby so that I don't have to know assembly. Right? Like, I think, like, Aaron Patterson is known notoriously known for his joke, like, you know, I do assembly so you don't have to. Right? Like and so I I see, you know, the the prompt usage kind of going in that direction too. Right? Like, people write natural language, and it does stuff for them. Right? Like, that's the whole driving force of all of this. It's like, you know, we had, like, a product manager come in and make some changes to get better outputs. Right? And, like, then our end users see the result of that and, like, very specific, you know, tweaks, like, general people can make. Right? Like, I've I work at Doximity, and it's like a social network for physicians.

Valentino Stoll [00:30:49]:
And they go on the ChatGPT, and they use ChatGPT to, like, solve problems for themselves. Right? Like, they'll go and they'll, you know, generate, like, a a denial appeal letter for some insurance provider, and they just, like then they use it. Right? And so they use natural language to make that happen. And, like, obviously, like, they have some issues. Like, they have there's a lot of handholding there. Right?

Scott Werner [00:31:11]:
Mhmm.

Valentino Stoll [00:31:12]:
And and so, like, the whole point is, like, to help drive people to use the natural language, and then us as developers are, like, to introduce, like, the advances of, like, extending it. Right? Like, okay. Chat, JP, all these models, like, yeah, they give you text. Like, well, because of that, you can have it generate any number of things because we're coders. Right? Like, we can start pipelining and and obfuscating that, you know, input output that the LMs give us to mutate in different ways. Right? And let the users don't have to see that. Right? But they could still give it the same inputs that us as developers would even give it. Right? Right.

Valentino Stoll [00:31:51]:
It's like that's where I see, like, alright. We gotta get away from this, like, alright, step by step instruction. Like, that's a programmer, like, mentality. Right? Like, we're we're starting to let the design evolution is gonna be really interesting to watch for developers, I think. I'm curious I'm curious, like, what what your thoughts are on that aspect. Like, where do you see, like, like, the design driving force of a lot of this stuff moving?

Scott Werner [00:32:20]:
You know, so my previous startup was actually a, a prototyping tool for voice apps. And so this was, like, right as Google Google Home and Alexa were were coming out. We built a tool for, you know, prototyping out different conversation flows and that same thing of step by step back and forth kind of conversation. And, you know, 1 of the things that we kind of realized is that conversation is kind of just like 1 1 modality. And we got acquired by Adobe, built that into XD because we, you know, firmly believe that, you know, you you want multiple modalities for inputting information to a machine. Sometimes it's voice because you're in a, you know, your your hands are busy and you can't actually press a button or, you know, you're cooking or something. But other times, you're on the subway and, like, asking asking your phone for your bank account information. It's probably not a great idea, but, like, pressing buttons on your phone is probably fine.

Scott Werner [00:33:28]:
And so all that being said, I think I think where things are going I don't think we're gonna throw out, like, the last however many decades of UX research that we have into, like, interfacing with machines, that there are going to be a lot of times where you have buttons and forms to, you know, that people can understand and and intuitively grasp. Right? Like, can you can you imagine going to Facebook and having to, like, write a sequel query or ask, like, the Facebook bot to add somebody as a friend versus, you know, just clicking the add friend button. And so I think it's going to be I think it's going to be a mix even if behind the scenes, there is going to be, you know, this nondeterministic natural language API that that we're that we've been using. But, like, from a user standpoint, not not requiring them to understand the right way to prompt. That makes sense.

Valentino Stoll [00:34:25]:
Yeah. That's interesting. I mean, prompt I I hate to say prompt engineering, but trying to get get the right thing to do what you want is still frustrating to me. And it is like it's so it reminds me of, like, when I first started to learn how to code, to be honest, which is a little funny. Like, I don't know.

Scott Werner [00:34:48]:
It's so exciting. It's

Valentino Stoll [00:34:49]:
so much fun

Scott Werner [00:34:49]:
to bring that, like, excitement back. Right?

Valentino Stoll [00:34:52]:
It it really is. Yeah. And, I don't know. Are there other tools that you use for that, like, you know, prompt design aspect that you'd recommend to people? I it's kind of

Scott Werner [00:35:08]:
radical. I'm I'm definitely kind of out out there on, the prompt design stuff. I you know, and you we've got an open source server for blueprints. We've got, sublayer is open source. You can see the prompts that we're using. We're very minimal. Most of the prompt design for us which I don't know if I do think guess maybe it's prompt design, but it's all about, like, what code are we going to give it and, like, making the problem easy. And so we haven't really used many tools because the there's maybe 10 lines of prompt that we rely on.

Scott Werner [00:35:49]:
But I think I think thinking about it like programmers is helpful. That, you know, if you have for, you know, we've been we're talking a little bit about this of just, like, what what make makes code promptable that if you have to say, like, I need this input, this input, this input, and do this other thing and then, you know, do that, it's probably bad code. It's also probably not very promptable code. And, like, taking that further, if you're asking the LLM to do this and this and this and that, it's probably gonna screw up somewhere. And so, like, break those apart the same way that you you would, you know, you would much rather not have a a 1, 000 line method. You break it up into individual pieces that you can individually test, are individually reliable, and then piece those together.

Valentino Stoll [00:36:38]:
Yeah. That's kinda my approach too. I I do use, like, sometimes we'll use, like, you know, the the models against each other to help build, right, and iterate on it, which is helpful. Like, how would you improve this? And then go to the other 1, and say, well, what do you think about it? Like, how would you improve this from? It's like the the they often will, like, you know, it do better, if you use the, like, a competing, like, model against, the other, which is a little funny. And especially if you tell it, like, oh, like, you're in a competition to, like, get the best prompt. Right?

Scott Werner [00:37:15]:
I was gonna say, do you, like, get them against each other? Like

Valentino Stoll [00:37:17]:
Yeah. I mean, I haven't tried to, like, take the the deep dive to be, like, this is a matter of life or death. Right? Like, and you can also win a $1, 000, 000. Right? Like, just shove all of the, you know, prompt tricks, which is kinda funny. Like, those are going away. Like, it's getting harder and harder to make those improvements work.

Scott Werner [00:37:38]:
But Yeah.

Valentino Stoll [00:37:39]:
I don't know if they're they're doing that training or spot. But Yeah.

Scott Werner [00:37:43]:
The training funny. Yeah.

Valentino Stoll [00:37:46]:
Yeah. That that brings me to my next point. Like, you know, Blueprint seems like it's, like, ripe for fine tuning. Right? Like, you have these preexisting conditions and preexisting formatting. You know, it seems like this would make a great flow to fine tune the models to give the outputs in more desirable formats. Like, do you even see that as, like, being a meaningful improvement, or, is it are are the better like, just you pay to use better models of this 1?

Scott Werner [00:38:15]:
I mean, the my my thinking right now is the is, you know, until we until we get to the, you know, the end of this curve of, you know, the cost dropping. I think they they drop by, like, mid quarter every 18 months or something like that, where, you know, it keeps getting cheaper and new models come out, which I think until until that stops or slows down, to really just just keep going. And, you know, I I wrote a post a little while ago, titled waste inferences, which basically kind of that call or just like, you know, we can actually build very simple applications and do very simple things even if, yeah, it costs a little bit more to, you know, use GPT 4 0 or Cloud 3.5 today. But 6 months from now, it's gonna cost half as much. And then 6 months from that, it's gonna cost half as much as that. And so, like, if you if you over optimize today, that's time you could be spent you you time you could be spending finding new patterns for when when these things cost nothing. I don't know if you remember. I mean, I remember early on, you used to get charged.

Scott Werner [00:39:39]:
Some people still get charged for bandwidth, but, like, the the, like, entry tier for any kind of web hosting was also a bandwidth charge and, like, if somebody linked to an image on your site, you'd go bankrupt. But, like, I haven't seen anybody, you know, really talk about s 3. But, like, the the costs are so minuscule now that it's not even, like, called out.

Valentino Stoll [00:39:59]:
Yeah. It's funny you mentioned that. I I remember being on a call. There was, like, some kind of, like, I don't know, like, presentation or something that a lot of the OpenAI engineers that were giving to a bunch of companies. I was, like, enough to join in. And, you know, a lot of people were asking, like, you know, is fine tuning, like, worth it at for these models, like and and they're, you know, how do you get even, like, your preexisting machine learning teams on board to start using this stuff. Right? And and what they were basically saying bluntly was, like, you know, the models are getting so good that fine tuning is really not gonna be worth it over the long run-in general. And that, like, basically, it would be better for your, like, machine learning team even to get used to using the large language models than it would be to have them, like, you know, do, like, machine learning to, like for a recommendation system or something like that.

Valentino Stoll [00:41:05]:
Like, over time, like, you would you would be more advantageous to, like, basically get your foot in the door to the models, and work through prompting than it would be to, like, get trained data on some kind of, like, you know, PyTorch or something like that for whatever you're trying to do. Like, Like, because over the long run, you still have to maintain all that. Like, you have to worry about the same things. Like, is it being effective doing its job? Right? And then also improving those aspects of it. Whereas the the models themselves, like, they they already are improving themselves. And you get and you can made it like you are. Right? Like, where you can get it to improve itself, like, iteratively. Yeah.

Valentino Stoll [00:41:50]:
And and so it's interesting to see, you know, it's I'm kind of torn on it, though, because I do I have worked with some fine tuning aspects to, like, get it to conform to a very specific outputs as an example. And it and it does, like, perform much better, for those very like, the you know, Obi's book is great, like, describing the narrow path, which is, like, the perfect example of, like, how, like, all of this works, really. Right? It's like the smaller and narrower the scope you can make the task, like, the better that it performs. And, like, you know, the problem with it now is, like, GBD 4 is, like, such a vast, like, knowledge base, right, that, like, it starts to perform poorly the greater the tasks that you ask of it or the more that you ask of it. So, sure, it can, like, start to distill and do a pretty good great job most of the time. But, like, the more you start to, like, ask for it, like, as an example, like, you know, what what which of these categories does this content fall into? Right? It's gonna make a lot of mistakes because you're asking it to, like, match up too many things at once. Right? But if you're just like, well, is it this category? Yes or no? It's, like, gonna get almost a 100% of the time. Right? And so, like, it's more think thinking like those aspects of it, which is kinda funny because, like, the pricing model is, like, perfect for them.

Valentino Stoll [00:43:15]:
Right? Like, the more that you use it, you know, the more money they make. But also, like, you know, the the more that you need to use it, the times wise, right, starts to increase, just by the nature of the design, which is, kind of funny.

Scott Werner [00:43:32]:
I just realized I thought that we should put a disclaimer on that waste and for some post. So I'm not being paid by any of you. Any of these API company.

Valentino Stoll [00:43:43]:
That's funny.

Scott Werner [00:43:46]:
But I guess the 1 thing I would say on that and, like, even to to your question and, you know, maybe startups could have started the should've said this a lot earlier. But, like, a lot of these these these things are so new. Right? That, like, a lot of what we're doing a lot of what we're doing there's a lot of, you know, papers coming out around a lot of these techniques. But I think everybody's just really, you know they have they have theories of why these things work. They can get it to work, and that might work for them. It might work for this. There might be pieces of what people are doing that are right or wrong. I don't think anybody knows for sure.

Scott Werner [00:44:27]:
And, like, especially how uncertain the future is that I think I think it's, you know, these opinions are mostly mine from the way that we're approaching it. I could be completely wrong. I've seen successes and I have seen, like, evidence that we're right about some things, but that doesn't mean that there are a whole bunch of other techniques and patterns and ways to to do it. And so, like, that's that's, you know, at this also kinda terrifying. I mean, like, no. No. He does. But it's also super exciting.

Scott Werner [00:45:01]:
Right? Like, it's so fun to see you know, to, like, expand a your understanding of these things. So if if something you know, for anybody that's listening, like, if something interests you and, like, you get super easy to try out. So, like, don't let don't let somebody say saying, like, this doesn't work or this does work. Stop you from just trying it because it's usually, like, a 2¢ API call to to find out if your idea, there's something there.

Valentino Stoll [00:45:38]:
Yeah. For sure. Now has never been a greater time to experiment, because you can just prove people wrong so easily or or prove yourself right. You know? Like, it's it's very easy to test that out, which is honestly great, with all the new tools that are out there. You know, perfect time to to plug the Ruby AI Builders Discord. You know? People just, like, drop in the wildest stuff, and you're like, oh, I could do that. Like, you know, definitely come join us.

Scott Werner [00:46:10]:
Yeah. It's we had a it was a couple weeks ago, and somebody found, that it could you could send it base 64 encoded information, and it would be on. And then so that was, like, a really fun thing we had on a on a Friday night. And then I was just like, what? How does how is it able to do this?

Valentino Stoll [00:46:30]:
That's awesome. Yeah. I I wonder if you can, like, skirt around some of the token issues. Yeah. Compression. I don't know. Yeah. I I wonder if it makes, yeah, it any slower at responding.

Valentino Stoll [00:46:46]:
I have a nervous. So I wanted to to briefly touch on, you know, for those that don't know, you've started hosting a, what what seems to be a repeat, event in New York City. Yes. The Ruby AI meetup. How how did you get started with that? Like, what what prompted it? Like, you know, I I attended the last 1. It I thought it was really, really great to to meet new Ruby as, like, in the AI space. But even non Ruby as were there, which was pretty cool to see. Yeah.

Valentino Stoll [00:47:19]:
How did you get involved in that?

Scott Werner [00:47:23]:
I've got a, you know, I've got a backstory, of course. So, you know, I 1 of the we we got asked a lot, you know, as we were building the start up, why Ruby? And I'm as you can maybe if you can see some of the books here, a little bit of a a software historian and really love looking and digging deep into, like, the reason for, why we do what we do. And kind of realized that, you know, every time there's a new platform shift or new kind of thing we can do with computers or as things change, there's, like, there's kind of a pendulum swing. And you you know, you kind of see, like, object oriented programming and the GUI kind of coming out of small talk. And then, you know, that's more of the the informal kind of, like, fuzzy reasoning kind of, approach, which then, you know, you got it got more formalized through, like, the eighties nineties when with, like, c plus plus and Java. And then, you know, database backed web applications became a thing. And we found that the dynamic languages, Ruby, Python, JavaScript made it possible to test out a whole bunch of ideas and find the pattern and find what we how what it means to have, like, these dynamic web applications. And then through the 20 tens or so, it's become more formalized with TypeScript and Rust.

Scott Werner [00:48:44]:
And we we're not really trying all these different variations. We've got we know what we wanna build, and we want that to be stable and scalable. But now with LLMs, nobody knows. Right? And so my thinking was, you know, I'm I need a dynamic language where it's gonna get out of the way and I can test test out an interface, a DSL, an idea super quickly. And started thinking back to, like, when I was getting into Ruby, how there's just like this explosion of different gems, people trying things out, like, why the lucky stiff was putting things out, like rapid fire. And, you know, something like camping comes about which inspires Sinatra. I don't know if this is direct, but in my mind, this is this is my head canon that camping inspires Sinatra, which inspires Flask and Express, which is, like, kind of what a lot of things are built on now. And we kind of are in that period now where, like, we wanna get bring people together and kind of like what I said before, just like I don't know.

Scott Werner [00:49:48]:
You know, maybe I'm right about some things and wrong about some things, but, like, we need to get together and share those ideas, bounce those ideas off each other, get into arguments, and, like, show show each other cool stuff that we're building. And so that's where the the happy hour came about. It was like, I feel like I feel like Ruby is right for this. I feel like there is, there is you know, from talking to people 1 on 1, there is definitely that feeling of, like, where did that magic go? So I was like, oh, let's see if we can let's see if we can bring some Ruby's together in New York and, you know, test that test that theory out. And so we did and it was it was a huge I feel like it was a huge success. Like, you know, got a chance to meet you. We had people coming from all over the East Coast to come come to it. And, yeah, I got, you know, we've sponsors for the next 1 coming up in July July 24th, test double, infield, fire hydrant.

Scott Werner [00:50:49]:
Saw the, you know, saw the excitement, saw the energy, and, yeah, we're we're trying to make the next 1 even bigger, bringing more people together. I think the another headcanon. I don't know if this is exactly right. The story you hear about, like, GitHub starting was, you know, the it, you know, happened at a a booth at a sports bar after after a meet up where they were like, you know, we should try this. And I don't know. I wasn't there. I don't know the conversation, but, like, that's the story you hear. And I, you know, wanted to try to, you know, make that.

Scott Werner [00:51:23]:
See if we can recreate some of that magic.

Valentino Stoll [00:51:26]:
Yeah. That's awesome. You know, I appreciate it. Like, I know a lot of people appreciate it. Like, yeah, that was the first, Ruby event I've been to, and I quit in quite a while, and just because it was close. And and, you know, like, there there wasn't you know, it it reminded me of the, the unconference, tracks. Right? You know, they're just like, come hang out at a conference, like, and not go to any talk. And it, like, removes all of the barriers to, like, meet people.

Valentino Stoll [00:51:53]:
Right?

Scott Werner [00:51:53]:
Like Right.

Valentino Stoll [00:51:53]:
Which is kinda funny, because you go to the pay for the conference so that you can listen to the talks. Good. But then they end up having, like, you know, somebody's just like it ends up being like a, lightning talks, but, like, side sidetracks, right, which is kind of funny. But it has that kind of, like, vibe to it, right, where, like, people are just, like, talking about, like, all of their excitement, which is, like, all this stuff is so exciting. And where do you start? And, like, what are people working on? Like, you know, you don't know until and, like, the the meetup has was, like, definitely the culmination of everybody just being, like, here's what we're working on. Like, this is exciting, you know, and, it was super cool to see. So I'm looking forward to going again. Awesome.

Valentino Stoll [00:52:37]:
Great. Yeah.

Scott Werner [00:52:39]:
Yeah, we can actually I guess I can oh, I don't have the the access to make comments, but I'll send this to you, the link to the link to the event if anybody

Valentino Stoll [00:52:50]:
Oh, awesome. Yep. We'll drop it in here. Yeah. Come join us. It was so much fun. And, yeah, like you said, people traveled for it. You know? It was really cool to see.

Scott Werner [00:53:04]:
Yeah. It's funny. You know? You don't realize, like, how, you know, how tough the last few years have been being, you know, remote and and hybrid that, like, you don't really get a chance to, like, as much, I guess, anymore. Just like meet and hang out. Yeah. Got got a lot of feedback of just like like that same, you know, your same comment of like, I haven't been to a Ruby event in a long time. I haven't, you know, haven't seen this many people at a Ruby event for years. But, like, there's definitely, you know yeah, something we've lost by, like, going mostly remote and hybrid, but being in person kinda brings back, for me anyway.

Valentino Stoll [00:53:49]:
Yep. Yeah. I remember my first, RailsConf. Where was it? In Maryland. And, yeah, it's definitely even intimidating in a larger setting. And having smaller groups, you know, it definitely is easier to to chat chat with people and and, socialize, I think.

Scott Werner [00:54:10]:
Yeah. But, you know, 1 thing, I think we we had this moment when we first met. It was like, I I know your name, but, like, I know you're from the avatar in Discord.

Valentino Stoll [00:54:19]:
Right. Right.

Scott Werner [00:54:20]:
I'm trying to find a way to, bridge that gap a little bit.

Valentino Stoll [00:54:26]:
Yeah. I mean, to be honest, if X just had, like, you know, print your avatar, like, I feel like a lot it would make, like, you know, these social connections easier to see. Mhmm. Slap that on. That's funny. So where where are you going next with this? Like, where where is, sublayer heading? Where is blueprints? Like, where do you where do you see, like, your next phase, in all of this?

Scott Werner [00:54:55]:
I think the next thing and, you know, with the the the topic of this is Ruby is a sleeping giant for AI application development. Like, I think it's, you know, we've kind of laid the foundation here with the the framework, new version. I'm trying to get the new version out last night but I had a couple of things I had to do. Lay the foundation with the framework. Lay the foundation with blueprints. And then really kind of show how easy it is to show and, like, spread the word of how easy it is to build these LN powered applications that changes a lot of changes the way, whether we think about what's possible. Right? I think there's still a lot of it's hard to it's hard to go from a place where, you know, a project in 2018 even, you're like, okay, that'll be about a team of like 15 people. It'll take, you know, a year or 2 to do turning into like a Tencent API call.

Scott Werner [00:55:58]:
Like, that mindset is like it's it's wild. Right? I'm sure you've you've experienced it with just like some of the things these models can do. And so really, like, our next steps are bringing more and more of that to, you know, to the forefront of just like what all what are all these things that you can do that take, you know, half a day to automate where, like, previously, probably just ignored it because it was so costly and so boring and mind numbing. And so building out the framework more, bring bringing more use cases to the forefront and then, you know, expanding Blueprints to make that even faster so that more and more of these examples that you have, the more things it can do and more things it can generate. And then I'm also trying to experiment with the the interactive docs piece, getting ready to roll out kind of making it possible for anybody with API docs to, have what we have in our docs, and see if see if there's interest there. So even if you're not even if you're not on the forefront of, like, trying to turn $1, 000, 000 projects into 10 cent API calls, you can still see some benefit today.

Valentino Stoll [00:57:17]:
Hey. You you know what I would love to see is, just being able to give, like, a GitHub link to, like, lines or something like that and say, like, do something with this. I think that can make real a really cool, like, example of, hey. Yeah. Like, here are the lines to some code. Like, you know, change this or something or make it say something silly. I don't know.

Scott Werner [00:57:40]:
Yeah.

Valentino Stoll [00:57:41]:
I don't know. Yeah. That, like, case but there's so much stuff you could do like that. Like, it's so easy. Yeah. I would love to see more, like, more ease of use. Right? Like, it's it's already easy, but, like, still people don't, you know, know where to start, which is which is a little bizarre at the same time. Right? Like, it is as easy as just typing.

Valentino Stoll [00:58:02]:
Right? I hate to, like, let to distill it down to just that, but, like, you know, at this point, you could just go to, like, you know, microsoft.com and, like, use chat gpt for free, you know. Like, it it is that easy to get started, which unfortunately, you gotta buy into that.

Scott Werner [00:58:24]:
Right. Right. Yeah. I mean, I think that's that's the thing. And I think there's you know, 1 of the things that I've been trying to figure out is how to I've been trying with that that example of, like, you know, you're already doing that. Rag is just you googling for, like, the docs, finding the docs and using it and using it in your task. That, like, I feel like there are a lot of, like, very big scary, like, technical terms that, you know, when you're getting ready to do a side project, it's like, I'm gonna have to, like, read all these thoughts to figure out what these things mean when, like, what they really mean is, like, put an example in the prompt. Or, like, in context learning, it's just, like, give the the the give some examples in the prompt.

Scott Werner [00:59:08]:
Right? And then and it figures out how to do it. Like, that I think, like, that simplification, I I really wanna try to find or or work with people to help find. Because, been talking about this with with, like, Rails where there used to be, like, oh, you have to know all of this stuff about, like, database normalization and third normal form and and and this and that, but it, like, it really boils down to, oh, you mean, like, customers have many purchases? Got it. Right? So I think I think there are that, I think, is 1 of the things that's going to happen need to happen next. So just, like, the, like, easier description, like, distill this down. There's, you know, all there's very deep deep explanations to take it further and make it more and more powerful. But to, like, get started, she's like, oh, you mean you, like, just copy and paste that into the

Valentino Stoll [01:00:05]:
Right. Yeah. The the dummies guide to prompting. You know? But to be honest, like, I never understood why, like, they came up with the phrases they did, like, you know, 0 shot or few shot. Like, what does all that mean? Like, there was it's almost, like, nonsensical to, like, say that when you just mean examples. Right? Or not examples. Like

Scott Werner [01:00:25]:
I I think that's, yeah. I think that's that's 1 of the things that that, you know, been talking about with, you know, Andre and I have been talking Andre from LangChain RB, have been talking about is that, you know, a lot of this stuff is coming out of research and coming out of academia. And, like, those things those those terms do matter for, like, those those domains. But from, you know, an application engineering side of things that are more, like, you know, applied, we we can have our own descriptions of these and why they work and that are separate from, you know, the the science and the theory and more of the the practice. Right? Like, what's the you know, in in theory, there's no difference between theory and practice, but in practice, there is.

Valentino Stoll [01:01:13]:
Well, is there anything else you wanted to to cover today before we, jump in the picks here?

Scott Werner [01:01:21]:
Yeah. Well, I you know, if, if anybody listening is going to Madison Ruby, I'm gonna be giving a talk going more into using LLMs in little bit of the the philosophy on, you know, how to how to deal with l m generated code. The talks called going postal. You know, won't give any spoilers away, but, you know, leaning very heavily, which Obi talks about Postel's Law in in his book. But, you know, being building your systems so that they can be liberal in what they expect liberal in what they accept and conservative in what they send. Just a little bit more liberal in what you what you accept than than what you're used to. And then, you know, there's a lot of things that Ruby can do easily. Any language can do anything.

Scott Werner [01:02:18]:
Right? But, like, the metaprogramming things that you can do in Ruby to change your application at run time, I think, are underexplored. So, be going into going into that a lot at the talk.

Valentino Stoll [01:02:35]:
That's awesome. Yeah. III share your, your sentiment there. I I've I've definitely tried to have, the LLM generate Ruby methods and run them and then reuse them. It's it's a lot of fun. A lot very dangerous, but a

Scott Werner [01:02:51]:
lot of fun.

Valentino Stoll [01:02:53]:
But, yeah, definitely underexplored, for sure. So that's cool. Yeah. People go check out Madison Ruby if you're not going. I'm I'm excited to to watch that talk, remotely at least. Yeah. Thanks. Hear what you got.

Valentino Stoll [01:03:10]:
So, if if people wanna reach out to you or find you on the web, you know, where can they do that?

Scott Werner [01:03:18]:
Sublayer.com, primary. Scott@sublayer.com. If you'd like to email me, atscottwernerd.or@twitter. And let's see. We've all you know, I'm also like, we talked about in the the Ruby AI Builders Discord, you know, sharing, you know, the discussion the big the big discussions right now are on that that Arc AGI challenge, which we've all been kind of racking our brains about. And then, you know, the let's see if I have a link here. On the Sublayer site, we have a we have an access to our Discord, which is a little bit more, you know, about our releases and and maybe some less Ruby specific stuff and more kind of the, you know, thinking on what is how do we how do we think about these these AI tools and, use them for building applications just in general.

Valentino Stoll [01:04:23]:
Yeah. I've been enjoying that channel as well. It's super interesting. The the stuff gets that gets posted, and you would think it would get shared in more places.

Scott Werner [01:04:33]:
To be

Valentino Stoll [01:04:33]:
honest, a lot of the, yeah, a lot of the papers that get shared, maybe people just aren't reading, as much as I I am. I don't know. I don't think so. But

Scott Werner [01:04:42]:
It feels like it feels like there's an unlimited amount of stuff to read.

Valentino Stoll [01:04:45]:
There's so much to read.

Scott Werner [01:04:47]:
I'd wake up every morning, like, where did I fall behind?

Valentino Stoll [01:04:52]:
Yeah. I feel like the paper I read last week is just, like, already, like, you know, dated. Like, it's like, how how is that possible? I know. Alright. Well, let's jump into pics. We've been talking about so much great stuff. Picks are segment where we just pick literally anything. It could be code if you want to.

Valentino Stoll [01:05:13]:
It doesn't have to be, but, it's typically what I pick. Yeah. And if you need a minute, I can go first. So, I've decided to, to fine tune a large language model for Ruby, mostly just for, the experiment, than anything. But, I'm calling it rubylang.ai. I'm gonna start building it in open and just, like, learning how the all this stuff works and how it might be used, to source all of the awesome open source Ruby that's out there that is designed incredibly, and make use of that and take advantage of it, and make a a new language model that is very Ruby centric. So I'm gonna see how it goes. I'm hoping that I'm successful in it, but I may not be.

Valentino Stoll [01:06:09]:
There may just be a bunch of lessons learned, but we'll see. You can follow my progress, Ruby lang dotai.

Scott Werner [01:06:19]:
Yeah. I mean, I saw that this you posted about it either yesterday or the day before, and, yeah, excited to see where that goes.

Valentino Stoll [01:06:26]:
Yeah. You know, I I got this stupid, massive GPU server at home, and I was just using it for inference. And I'm like, you know, I'm just just wasting away over here of, like, running inference on it. And so I thought, like, I should fine tune something. And Good. This seems like a perfect use case. So here we go.

Scott Werner [01:06:48]:
Oh, that's great. Yeah. So I guess I've got I got 2 things. 1, you know, the I talked a little bit around, like, the the formal, informal kind of pendulum swing. You know, 1 of the places where I got that from was an optigrim talk, the soul of software from, this was a long it was a long time ago at this point. But it's, there are couple versions of it up on YouTube. He goes into more of the the kind of the informal mindset and how there are those 2 2 splits of software and how neither 1 is particularly absolutely right. But more, it's each each 1 gives you different different things depending on what you need.

Scott Werner [01:07:41]:
So, like, that was that was very impactful for me, especially now given the, you know, the approach we're taking. And, the other thing, I read a lot of substacks and there's 1 called Strange Loop Cannon, which he had a post a little while back about what LLMs can't do, which, you know, is also a a trap a lot of times because as soon as you say it's not possible to do this, everybody on Twitter tries to prove you wrong. And in his post, he he did have 1 about, it can't do Conway's Game of Life. And I think, like, within a week, somebody had proven him wrong. He had tried fine tuning all these different things, couldn't figure it out, and, the collective wisdom of Twitter came up with a solution, I think. But then he has he has 1 recent That's awesome. It's it's very good. Highly recommend just the the substack in general.

Scott Werner [01:08:35]:
That 1 is very good. And then this latest 1, seeing like a network, is very, was very, very good, as well.

Valentino Stoll [01:08:46]:
That's awesome. Well, I I appreciate you coming on, Scott, and talking about all this, cogeneration stuff and the the love of Ruby AI. And, you know, it definitely is a sleeping giant, and I think we're just gonna see more and more of why that is. And, you know, we'll have to have you on again after, you know, you you start creating many companies meta meta meta lies all of this Ruby code generation. You know, I I could definitely foresee a, future where you're just like, oh, you know, create a real set that does this, and it just does it. Seems seems easy enough.

Scott Werner [01:09:26]:
It's simple. Yeah. No. Thank you for having me. This has been a lot of fun. It's been awesome, you know, chatting and catching up and excited to see you, in person a month.

Valentino Stoll [01:09:37]:
Yeah. Totally. Alright. Well, until next time, folks. I'm out of here, and, come come visit us next time.