Today we look at an applied use case for ML: developing intelligent meeting notes. Expect to learn about LLMs, AI assistants, and how to develop an AI startup.

 

Hello everyone, welcome back to another episode

 

of Adventures in Machine Learning. I'm one of your hosts, Michael Burke, and I do data engineering and ML at Databricks, and I'm joined by my co-host

Ben Wilson. I write code at Databricks.

 

Each one of these is going to change every episode.

 

Yes they are.

 

So today we are speaking with Artem Korn, and he is currently the Chief Product Officer and co-founder at Assembly AI. And their main product is ML that transcribes meetings and generates insights. So think next time you're on Zoom, if you have Assembly AI turned on, it'll automatically generate to-do lists, meeting summaries, potential project lists, that type of thing. So Artem, what inspired you to found

 

Oh guys, I'm really excited to be on this podcast with you. So my background is in technology management, product management and management consulting, a lot of it. And when I was doing management consulting, there was technology that was making its way into every facet of the workflow, but it stopped short of actually participating or understanding the conversation in meetings. And when you're doing management consulting, The progress is what drives the results. So your meetings are all day long. And when I met with my co-founder in 2019, and we were thinking about potential products and applications, new technologies that are coming up that could solve some problems, it was a very obvious thing that what the team discusses during meetings is very opaque. And so even though you had technology around how you conduct meetings, allowing remote work, the Zooms, the Google Meets and so on, what actually transpires in the meeting the technology never knows. And humans have to then go after the meeting and update their tools, update their teams, write up meeting minutes, all that great stuff. And we created assembly AI to address that. We thought that an AI participant in the meeting could be very very valuable.

 

God, was there a triggering meeting that made you say, hey, we need this right now? Cause I might've just had one. I'm sorry, I'm sorry.

 

Ha.

 

I don't know. I don't think there was one. I think in many ways, like a lot of these, it's interesting because without having this as an option, it's hard to be mad at it. As in, when, you know, let's say when I was growing up, all we had were beepers and chains. This was before, you know, the touch phones. But no one was upset that they couldn't Instagram their food, right? Like I wasn't like, I gosh darn, I like, I wish I could. I could photograph my breakfast. That was never a thought that crossed my mind, even though now that's what we do every day. So I think it's very similar. I think before the advent of AI participants in meetings, you don't sit there going, you know what I really need in this meeting? What would really help my life as an AI participant? Never occurs to you. But once you know it's an option, all kinds of, it's like, yeah, great. So maybe I don't have to come to this meeting. Maybe the AI can come instead of me. me and send me the notes, which is something our product does.

 

Yeah, that makes sense. Question, though. So you have, let's say, an AI that comes in and replaces me because I had a meeting conflict. How do you balance having thorough notes with having concise notes?

 

That's a really great question. So I think there's phases of evolution in this environment happening. And initially when other companies in our area said they had meeting notes, they actually had no meeting notes. They had a transcript, but they called it meeting notes because why not? Then AI approaches came along learning approaches came along and good summarizations became possible but then it's a question of what is a good summarization. So to your point, like how detailed, how lightweight. And so when we build our meeting notes feature set, which we call GLANs, it was actually based on our experience in consulting. It was what are... What I find useful. insightful in meeting notes that I can actually look at, be fairly well informed as to what happened in the meeting, and then pass things along to other people if I need to. So it's actually pretty static and it's based on our expertise from this background. But moving forward, and this is where this technology is evolving today, you can understand. So there's a dials. One dial is what type of a meeting is it? Because depending on the type of meeting, the summary of the meeting and the blocks of the summary of the meeting could be different. Whether it's a stand up with a team or it's some kind of a client meeting or it's a town hall, very different dials in terms of the type of a meeting. The other dial is the details, so in terms of summary and precision, like how much do you want to include? And that often is a personal preference. So our products are evolving into that category as well, where we can personalize and adjust. And then there's another step, which is personalized summary. So if I know who you are, meaning I know your role in the meeting, I know what your inputs in the meeting were, not only were, I know what, based on your role, I know what your inputs in the meeting would tend to be or are expected to be, let's say. And I could also assess your takeaways and then based on your role, I can give you a personalized summary for that meeting that really is impactful for you. So you can think, for instance, like in a stand-up again, you know, there's a, let's say there's a scrum master. If I know you're a scrum master, there's like certain kinds of participation you have in the meeting. If I know you're a team lead, it's something else. If I know you're a dev, it's things that are very specific to the things you're working on that you're interested in. So that level of customization is something that you can expect to see. over the next year or so come into the fore.

 

So with the summarization that's in place, I mean, sorry, I have a lot of questions for you, but first one, sort of a philosophical discussion with your clients that have, perhaps not the healthiest working environment. Maybe they work for a really big company that there's some sort of existential threat looming over everybody's heads of, hey, if I say the wrong things in meetings and then there's this transcript of it people going to read that and be like, why did we hire this idiot? Or, hey, this person isn't being constructive. Do people change their behavior? Have you heard that from people? Knowing that there's this permanent record of what is said in meetings. And do they adapt? Do they sort of get better at being in meetings because they know this is there? And so, I think that's a good point. I think that's a good point. I think that's a good point.

 

I think initially there is a, it's a bit jarring. And when you first start to have AI participants in your meetings, I think you start to be very self aware, happen to our team, like it's normal. You very quickly get used to it.

 

Thank you.

 

Like

 

Thank you.

 

today, if I'm on a meeting without assembly agent on the call, it feels a little bit off because I'm also a little bit nervous. Like I don't have this to refer back to. So I actually now am more comfortable on a call with an AI participant. And in terms of behavior, long term, so there's a short term behavior change that very quickly I think even outs. It's kind of like an initial spike that even out. Long term, I think you, I don't think there's a major behavior change. I think it kind of cuts off, like, you know, I don't, I definitely wouldn't like go crazy or go off in the meeting because I know the agent is there. Like, not that I, not that I do generally, for the audience out there. But it's something that I am aware of like in terms of like extremes, but other than that, it never really crosses my mind. I just have my meetings as I go along. So I think that's kind of like the curve of behavior changes. Like there's a spike initially, the spike draws out and then maybe like where you land is just a little bit of that kind of cutting off at the edges of behavior. Long term people just forget and get super used to it.

 

Interesting. I would

 

Thank you.

 

think that the first shift that a team would have if we're talking about, you know, customer, like a client service provider sort of discussion, which you were in consulting, you know, that's what Michael does full time. And I used to do that as well, where you have those type of clients where it's just brass tacks. The only thing that they want to that is, what are the blockers? Let's get through this meeting as quickly as possible because we all hate doing these meetings and we just wanna get through it. And then you can have one of those in the morning and you're like, all right, oh geez, I got a two hour block meeting and this is gonna eat up so much of my morning. And then after 27 minutes, you're like, wow, that was so efficient, that was awesome. We just got through everything. And the poor people that are taking notes are just furiously typing the entire time and they miss probably 30, 40% of the critical details You can only type and listen so much. But then we've all been in those meetings where it's like, all right, it's a 45 minute meeting, you know, scheduled meeting, it's been four and a half hours. And we've heard all about this guy's dog that had to go to the vet last week and, you know, somebody asking completely irrelevant questions about some other project that we're not even meeting about. So it seems like a tool like this would focus people. in that sort of relationship in a meeting. We're like, hey, we have an agenda, and we can see it in real time. Are we hitting all of the points? Are we getting summarizations? So that's pretty, pretty fascinating.

 

Does it filter out small talk?

 

So we've had a small talk filter in the product from like many years ago. Yeah, it's one of the earliest things we built because, yeah, as you're analyzing with ML, the initial versions of these algorithms were very extractive and categorization based. So for that, it was important to get rid of the garbage, not the garbage, but the non-productive conversation. In fact, we had, you know, our system worked is that there's a pre-process including things like small talk removal. So we had models trained on small talk and then remove them. But then we also had a part of the system that actually partisan still alive. It's called Kodama, which is a, it identifies areas of conversation that are most ripe for producing insightful key items. And you use things like, you know, different factors of conversation, much as this conversation about something that's happening in the future. So you're basically before, this is like, you know, there's world before GPT and like after GPT. So before GPT, you had to, let's say, help the models along. Basically you had to first reverse engineer certain contextual things in the conversation and then only apply like the, you know, the, the, the discernment, like, oh, this is an action item. This is, this is changing very rapidly. language models of today because the large language models are very good at reading comprehension. So they understand when it's small talk and they understand when it's productive conversation all on their own. They don't need help. So they're actually very contextually rich when they analyze. And so they need fewer hints. Some hints still help, but they need much fewer hints to do to get the job done. So yes, we used to remove small talk. But these days it's let. of a concern. Ben, to your point, so first of all, it sounds like very dry consultants of like, they just chop, chop, chop, right? Like there's, it's important to have some human, like elements in your meetings that creates, you know, that teaming and that kind of, that you enjoy working with the other people. So like just having it, like to the point, I think would be probably bad. Like there's a little bit of, you need like a little bit of fat on your steak. I don't know if that's a great analogy,

 

Yeah.

 

but, but I think, I think it, certainly you'll be able to see much better on how productive different meetings are in terms of next steps generated or productive next steps generated and you can make some conclusions based on that. That that that is already available. I don't think it's being tapped into really by organizations, but I think over the next couple of years that's going to start to be a factor in assessing the like operational effectiveness.

 

So a tech question for you. And if you don't want to give away the goods of how stuff is implemented, it's fine. But out of just sheer curiosity. So you have this large language model that's running through, well, I'm sure you have a suite, a host of many models that are doing all of this stuff. I counted at least 11 so far and just what you've said, of like, it's kind of how I think they would do it. Yeah, it's definitely way more than that. But you're processing audio, you know. flack encoded audio or something, you convert that to an array which gets converted into a tensor, it goes into this model, and you're getting basically audio to text of the raw, like here's everything that I heard. And then you're going to run that through summarization models, which are saying take this raw big body of text and just say, what are the key important things? And as you said, GPT models are GPT2 is arguably okay at that, I think, if you've really trained it well. So GPT-3 is just leaps and bounds ahead of what that was. 3.5 is even better. And now four, it's approaching magical territory, in my opinion. And then I know they're working on five, they're probably working on GPT-6 right now. But when we talk about these large language models and some of the deep implementations that can happen where you have, hey, I have this pre-trained, large model, it could be generative in nature. All of these are built on the transformer's architecture. So the concept of, hey, I can create this processor unit that's going to take in this massive body of text that is sort of the raw data or the raw summarization, and then you can create a prompt that has stateful context associated with the user ID. Do you currently have that where, hey, I want to listen. to ask questions of this model system and ask this bot, like, hey, what did it, did people talk about this in this meeting and just be able to get this quick summarization, single paragraph of text? Is that currently available or is that something that you're thinking about right now?

 

So fun you should ask. It's a feature called Semblian. And this feature is going live, believe it or not, tomorrow morning.

 

Nice.

 

It's been something we've been working on for a while. So

 

Saturday?

 

well, the team has to deploy in the weekend to make sure everything is hunky dory.

 

Got

 

And

 

it.

 

Monday is when the customers come on. So yeah, we love our Saturday Deployees. Saturday Morning Cartoons, Slash Deployees. Yeah, so it's exactly what you described. We call it chat GPT for meetings. You can go in and you could ask it basically any question about the meeting that you have. You can ask like what did this person say to that person? You could have it make a poem out of a meeting. Everything you can imagine with chat GPT, this thing will do. We have a lot of... secret sauce around making it effective and appropriate for the meeting context. There's some both conceptual challenges and also still technical challenges around that. Because you can just feed a meeting to GPT and get a response back. It's like there's all kinds of different weird limitations around. But yeah, this is exactly it. So you can talk to your assembly and about anything in the meeting and respond, everything from summaries. The craziest thing, this is the thing that blew me away and I've seen it blown other people away too, is that it's multilingual. So you can ask it a question in any language. You can have it, not, quote, unquote, and like almost any, right? All the popular ones. And you can ask it for a response in any language. No matter what language the meeting was in. Yeah, it's fascinating.

 

So what about stuff that... For our previous work and Michael's current work, when we're talking about, hey, we're providing advice and consulting to a customer. Even outside of the business consulting or SaaS industry, visualizations are a key part of summarizing a meeting. And GPT-4, one of the things that it's just released, it was two weeks ago. last week whenever it dropped, I was like, hey, you can pass, you know, you can use mixed modal approaches with this. We're like, hey, you can pass it an image and tell it to create something new out of that or describe things in it. So these multimodal models, is that something that's on the horizon too? And how far off do you think that is where you can ask it a question and say, hey, can you just generate some architecture diagrams for me? described or, hey, I need to do an executive report on this next week. Can you create a bunch of PowerPoint sides about this five hour long meeting?

 

Yeah, I think so there's this term going around this Cambrian explosion of startups in the environment. And I think it's very accurate because what these large language models now provide. So, you know, GPT-4 is the darling of the hour, of course, but Bard is coming around and there's going to be others. There's some open source ones have even seen starting to come together. So it's very exciting. It's sort of maybe in some way like the mature OS space or something like that.

 

Thank you. Thank you.

 

But these large language models provide this kind of crude, like crude oil effectively. But you can't just pour crude oil into your Toyota, right? So somebody needs to refine it to a certain kind of a fuel, whether it's like gasoline or diesel or kerosene is used. So these large language models are this crude oil and then there are refiner, so assembly AI is a kind of a refiner for large language models in addition to a bunch of other things, that provides a certain business value outcome. And so you're gonna see all, you're definitely going to see within the next, probably year, definitely within two years, what you just described, like based on this meeting, like generate architectural diagrams, But it'll probably be like things that are more specialized in that because you do need to Put some guardrails and put some kind of common sense structures around what it produces. You can just it's The problem is always context and so as brilliant as large language Models are like they don't know You know like they only know what they see in the immediate But they don't know the background like they don't know kind of your personality or who I am or like things about the background of the company, they know what they can see and then kind of leverage that against all their training. So you need to usually kind of give them the additional context and then guidance to get to an accurate business result. I think the accurate business result is the hard thing because it's easy to dolly yourself into a new image, right? It's like, oh, okay, draw me a robot. Okay, that looks like a robot success. identify a task that is an actual real test that needs to be done and assigned to a person by a particular date and figure out what that date should be. That's actually really, really hard to

 

Hmm

 

get it narrowed into a business value point. But that's happening and that's going to happen both in visualization and mixed modal and in text and in all sorts of other areas. We focus right now. We don't do anything visual today. We don't have that on the roadmap. we're very focused on understanding the conversation and then providing value out of the conversation. But I don't exclude it. I think maybe in the future that that could be a turn that we take because we find it so valuable.

 

I think your last paragraph of speech just now is probably the most relevant advice that anybody could give to people that are listening to this hype. Because you guys have been doing this for years. And anybody that's been, you know, if we go back in time, eight, nine years ago, when the initial, well, it's not initial, I think it was the fourth generation of hype around deep learning, like neural networks. You know, the first one was back in the 60s. got super exciting in the 1980s and people were like, wait a minute, these are really hard. But back, you know, pre, I remember what version of TensorFlow, I think it was pre 1.0 when it came out, everybody was like, this is gonna be revolutionary and it's gonna do all the things. And they put some image models out there, pre-trained ones, you can start using CNNs and everybody got really excited. And a lot of people sunk a lot of time and money into trying to get those things, work. But then everybody realized that, hey, they don't do everything. They do a lot of things. Like, hey, the CNN can classify what it sees in bounding boxes that it detects as important characteristics and images. But how do you make a product out of that? And that what you're talking about that focus on this one thing and realize that the model is a component, definitely it's important component, but most of the work is all of that ancillary work. And that's what really makes the product is like, how are you controlling the input data? How are you massaging the outputs? How are you doing safe fallbacks? How are you doing filtering when you talk about language outputs? Because the raw output of some of these things, I was a couple weeks ago, we're working with an old GPT two pre trained if I start, you know, if I train it on some fine

 

Thanks.

 

training on some potentially bogus data, I just downloaded Twitter data and had it learn on that and then started asking it, you know, some sequence classification of passing in just random stuff that I was typing. It's like, ooh, it's reporting stuff that there's no way you would send this to a customer. If you had this thing, the raw output out there, you know, as something that people could interact with, this thing is bad, like potentially super offensive about what it inferred from this sentence, or like the emotional classification ones, where training that on some, you know, overly antagonistic Twitter data, it starts classifying everything as like stuff that most people would find pretty offensive. It's like classifying it as happy. It's like, what? So yeah, that output, like how you handle the output coming out of that is the most critical thing. And I think you summarized it perfectly. Like, hey, we focus on doing this one thing really well. And, you know, based on your demo that I looked at and your website, it seems like you do an exceptional job at this.

 

Thank you. Yeah.

 

But yeah, so listeners, if you want to know how to monetize this awesome stuff, like that's the secret sauce right there. focus on that problem and pay attention to the details.

 

Absolutely.

 

Yeah, it's interesting. We just had, I guess, a month or two ago, we had a company kickoff for Databricks. And for those who don't know, Ben and I worked there. And Ben was smart enough to not show up. And it was really interesting to see how the company prioritizes. So if you go back to Facebook in 2008, the only metric that they cared, or pre-2008, I guess, the only metric that they cared about was the number of users. They didn't care about pretension. They didn't care about if users were interacting with the platform or anything like that. It was just the growth numbers. And Databricks has a sort of a similar prioritization strategy where it's not about growth, but it's about very specific initiatives. And it lets other seemingly important initiatives just not fall apart, but there's no active development. And I think that's one of the hardest things as a business leader is to figure out what to prioritize Because you can't put all of your eggs in every single basket. You have to pick baskets. So I was wondering, Artsum, what baskets are you putting your eggs in for assembly?

 

Yeah, so we had this classical startup challenge where you kind of, you grow this bouquet of features in your product and then you find out that that's not what brings the users to the platform. So we're a lot more selective these days. Our roadmap is focused right now on a few things. So one thing is that we do want to bring users, and we want to create attention. One of the challenges that GPT brought is that it lowered the differentiation barrier by a lot. So we used to have a very differentiated technology. It was very difficult for others to compete in the analytics aspect of our product. And today, it's a lot more accessible. So anyone can leverage. the large language model and generate a summary. So we want to make sure that we're maximizing what we get out of large language models and are introducing things like Semblian to really kind of maximize that value in the product and create interest in a customer. The other thing that we're doing is we're actually paying a lot of attention to retention and churn. And we want to make the product very understood, like kind of intuitively understood, and also nudge the user towards different areas without complicating the interface. So we're actually investing a lot in simplicity and also guiding the user into how things work, because remember this is such a new space, this is a category that effectively didn't exist until just a few years ago, and realistically didn't exist until maybe a year ago, because people just weren't thinking in these a big accelerator for this space. And now the AI technology is another big accelerator for this space. So helping the user along in terms of what this product does and how to maximize its use is also very, very important. Then there's, we're looking at some long-term roadmaps in terms of hitting functionality that you just can't get Like it's that's you know, so what are large like models are good at you know generating creative content They can be good at categorizing certain things But there are certain business problems that require a lot of finesse that just these large like new models cannot do That required some additional metadata from around the environment requires some understanding of like how roles and things are structured and so we're working on on features that would really impact working teams that are not, that you can't just get out of the box with something like GPT. So like tasks is one of those things. We're aiming to have the best system for understanding

 

Thank you.

 

team tasks and being able to plop them directly into your task management environment. So you've finished the meeting and your, you know, your Trello boards or whatever you use to manage tasks with the right content. So it's actually a really, really hard problem to solve. Just think about, you know, the team meets and says, okay, like, yeah, I want you to do this next week. Like, what does this mean? Right, so, and that's just one little element of the complexity of this problem. So, so those are, so near term, we're throwing in exciting things like Semblian, which are really, really cool and have all sorts of business value. We're softening that with functionality users use these new kinds of AI and empower tools because they're so new. And long term, we're looking at some of these like very difficult, technically and conceptually difficult landings like tasks.

 

So that's almost similar to what GitHub Co-Pilot Plus, I guess just released based on chat TPD4. I guess they did that news release yesterday or something. Where now you can do with GitHub Co-Pilot, not only is it using the advanced, much larger GPT-4 framework, but they turned on the ability to do speech to text for code development. And you can even explain it an abstract concept. Like, hey, can you, I need to basically write a function that opens and converts an image into JPEG format. Okay, it writes the function for that. Like, hey, can you add four unit tests? I want to see if you can load in, you know, byte array data, NumPy array, you know, byte encoded string and a bitmap image those four unit tests for you in the appropriate test structure. And then you say, okay, I need to interface with, you know, this Transformers model that is of this architecture, can you write a function that fetches that? It just starts writing the code. It's not perfect. You know, senior experienced developers are going to look at the code generated and be like, that's not how And that seems like that naturally fits into what you've already been doing. Like, hey, listen to this, these meeting minutes. is a natural extension of that developer productivity tools or PM productivity tools where you can interface with certain partnered ecosystems where you're like, hey, I want to plug in for JIRA so that when I say, hey, create a story that does these things, and it just starts transcribing the summarization in a particular style that you've trained it in. Stuff like that, that seems like it's in your wheelhouse big time.

 

Yeah, that's kind of the problem because a lot of things are in our wheelhouse. That was a big, so maybe, you know, last year at some point, we were thinking about different directions and that was actually the major problem. There was like so many cool things we can do from, you know, specializing, customizing, specializing, like summarizations around roles, to creating specialized applications for like a, like what would a scrum master want to get out of a meeting? and maybe even tool or some coaching, blah, blah, blah, some automation, like around a specific role. Yeah, I think we have a very cool voice AI platform which allows us to do these things, which just means that we can engage with your meeting environment very fluidly. Like we can participate in all of your different conference call systems. We sync with your calendar. We understand schedule complexity. We can also listen in the microphones. So we have a very effective way to participate in the environment and also then turn the audio into really good text, diarized by who said what. And that's also a special kind of a thing, like we have a partnership with Phillips that currently have one of their product lines is Smart Meeting, which is like a series of these conference call microphones. And we are one of the special things about Semblies, we support hybrid environments very well. So we're not just online. Like if some of the people are in microphone in a room online like we'll support that environment as well. So yeah there's just a lot that you can do. And I think the biggest challenge is like figuring out what not to get involved. And I think next year, you know this year I think we're pretty much spoken for like we know where we want to go, but next year there could be more platformed aspects for our technology where we could maybe for more specialized kind of functions for these different kinds of use cases. But we're trying very hard not to jump on all of those really cool like candies of business propositions. It is pretty cool. I did want to say that like for the co-pilot thing, exactly what you mentioned, like I think it's like a very powerful, magical, like staff that can do like wheeled, but in the hands of a junior,

 

Oh yeah. Dangerous.

 

right? And this also raises a question, okay, so the juniors will use this tool to write a lot of code, and then what happens? It goes up for code review to the lead. Does this mean now the lead is just gonna be swamped with garbage code? So we have to think about these kinds of things, right? Because as good as these tools are, they still make a lot of mistakes, like writing actual programming, they can give you good things, like good directional things, almost like kind

 

Thank

 

of

 

you.

 

templatey, but very often it will be either not working code or buggy in some weird way, unless you know what the thing is supposed to look like, I think you might cause more trouble than it's worth and not learn. Like that's another thing, right? Like how will you learn be a lead from that type of technology. So it brings all sorts of like new and interesting questions and dynamics and like in roles like what are roles now are going to do like, you know, the leads now have like all this work on top of them because they're getting much more like issue prone code because of GPT.

 

Yeah, one of my favorite, because I'm doing, I'm writing an interface for, you know, these sorts of things right now at Databricks. And there's a lot of testing that I have to do. And just keeping up to speed with all of the, you know, these massive developments that keep on happening, you know, the six a.m. the morning that after GPT-4 announced, I was just on there, not doing what other people were probably doing. I mean, there's, of course, other people were doing what I was doing. But I'm not having it, you know, write a blog post for me or tell me a joke or, you know, write a song for me. I'm more like, hey, can you give me the most efficient implementation of the Fibonacci sequence in these five languages? And then going to each one of them and saying, tell me what the computational complexity is of that. And I want you to make it, you know, from O log N, I want you to make an O N squared implementation for me. And then seeing how, if it could do that, if it could write bad code effectively, and then saying, hey, what's the worst computational complexity that you can create for solving this problem, and seeing what it was generating, and then having it regenerate other iterations of it. And it's a little bit, it's not really reverse engineering how the generative model works, because good luck with that. It's more just seeing how would people actually use this perspective. And then other times when exactly as you said, like a senior person, these tools are very useful because sometimes you're trying to move really quick and you're not really thinking about all of the aspects of different ways to approach this one function or this one, I don't really use it for function generation, but unit tests, I love using it for unit tests. Like, hey, here's a function. Can you write seven different unit tests? for me. And I just want to see what it, what it's analyzing about my code and what are the seven different things that it, it's trying to, it's basically pen testing my code. So then I say, Oh, these four are really good. These three really suck. But out of those four, one of those things I didn't even think of to test. That's really clever. It doesn't work. Like it doesn't like execute correctly, but it gives me that idea of like, Oh, I need to test this part of it because it saw this. the code. So that aspect of it in the hands of a senior, that could really senior developers super powerful. But yeah, on all the testing that I've done, even for the new iterations, I'd say it's like 40% of the time, marginally correct, ungenerating something that I wouldn't be upset with doing a PR over. The other 60% is like, yeah, this doesn't work, or It's so bad that, either performance-wise or security-wise, that it would be a danger to put into any sort of library. People are going to do it though.

 

Yeah, exactly. But you see it and you know, OK, this one is interesting. These three are crap. A more junior dev is going to be like, here's my seven unit tests. You know? So yeah,

 

Mm-hmm.

 

I think programming is probably one of the more difficult problems that is going to be solved by generative AI just because there's, until AI starts to understand enough project scope. Well, it's not just like point, like a write me this, you know, like a function that does the sort in like whatever order and squared or whatever you want, right? Like a function that you can write, but like write something in context of the project. That's really hard. So, yeah, I think that's a good point. I think that's a good point. I think that's a good point. I think that's a good point. I think that's a good point.

 

Yeah, you look at some of the open source packages that are out there that are on GitHub that this thing these things were trained on for GPT 3.5 and 4. If you ask it like, hey, can you I want a feature that that exposes this new rest API endpoint for this open source package. There's no way it's going to know what what files to change how to look through and and understand the levels of abstraction. If it's a professional open source package that's really well done by professional engineers, sometimes even senior engineers are like, I have no idea how I'm going to solve this. I have to reverse engineer how this was implemented. It's going to take me a day reading through the code to figure out like how do I want to implement this while fitting within the subtraction model. And I have tested it. like taking a couple of open source packages that are written in, you know, mostly a functional paradigm and then some in pretty much an exclusively object-oriented paradigm. And it does a little bit better on the functional provided that you give it all of the required functions in that dependency chain but still doesn't really quite grok what it needs to do when you have too much complexity in there. But the object-oriented stuff with extreme levels of abstraction, it just generates garbage. Comical garbage sometimes. You're like, why, why would you think that that would work? And then you tell it like, Hey, you're really far off base here. And then the fun thing for me to do is to see how far down the hallucination chain it can go before it's like, I give up, I don't know how to answer you. It's like basically writing a whole new programming language by the time I'm, I'm breaking it in that way. But yeah, to your point, junior dev wouldn't know that. It's like, Oh, I ran this, it didn't work. It's going to generate some even deeper crazy garbage on the next iteration. They're like, Oh, here's the exception I just got. Okay. Go even further down the crazy hole.

 

Yeah, there's a lot of interesting parallels between the timelines for assembly and chat GPT. So if you think about meeting notes in general, we had transcription 10, 15 years ago. It might have not been amazing, but it would work. You would have every single word from your recorded audio in a Google Doc or wherever it is. What assembly is currently doing is they're trying to extract sort of action items and sort of the low hanging fruit. I think the holy grail of artificial intelligence in general is it looks to identify things that humans can. So for instance, business risks and things of that nature like business opportunities is another great example. If we don't implement this feature, we will lose this amount of money. And I think that there are two components that lead into that feature. which is you have to have organizational context and train on how the organization runs industry Competitors that type of thing but you also need an algorithm that can think critically and Distill information and then make logical and maybe even creative jumps So question for both Ben and Artem How would we go about making that step from where we currently are in both AI and in meeting? Summarization, how do we go from this current step into the holy grail of it can what a thought leader in a business does.

 

Ben, you have ideas? I mean, I'm happy to say what we were thinking, because it's something we thought about before. I think it's a phase of evolution.

 

Yeah.

 

It's something we called enterprise awareness. There's other names for it. But effectively, one way to think about what's going on is, so now that you have AI participants in the environment that really know what's and they can go they can know what's going on with all of the teams. What's happening is you're creating this like digital trail of all of the content that and all of the discussions that are happening. So you can imagine that you know each kind of work group of teams has kind of a little brain that that knows what that team what those teams are up to in that work. Right, so like think of it like a chat GPT, but like in the work group that really can answer questions really well about the activity across those teams. Now that little guy can talk to another agent in another work group. And they can communicate. And so you can have this like this tiered hierarchical structure where the entire enterprise. becomes like this neural net type of a sensory organ that is plugged into everything that's going on in a very digital way. And so I think that is a holy grail for enterprise AI where there's AI involved across all team activities everywhere that can and creates these nodes of awareness in those activities and those nodes can communicate to each other, it's very brainy in the way, right? They can communicate each other. So for instance, for instance, let's say you have the marketing team and like, or my marketing department and they have like this little node and the node knows that, you know, there's a project that the marketing team is about, like, needs to start working on and creating like a PR, whatever, rollout. Well, it knows that that PR rollout depends on like a certain feature set that's coming out. So we can actually talk to the node in development and say, hey, how's that? project done? Like is it likely that it's coming on time? Like how confident are you? Is the dates good? And have that up to date information and then help the marketing team do their planning for that? Or like, you know, is there a spec change? Like has, you know, has everything approved? Right? So these nodes can actually like interact among themselves. And then if they're missing information, like that node in dev can talk to the team and say, hey, marketing, you know, like need some more info, like how are you guys? this very futuristic kind of a system and all of that can integrate into this sort of AI that can support enterprise level decision making. So it really creates these digital tentacles all through the organization that in not in real time necessarily but in like near real time can aggregate the data up and help the company understand like where things are moving. in a very like, in the very, you know, like not to overuse it, like in a very intelligent way, right? Because that aggregate node can then ask the higher level questions. Like, of all the activity that's happening across the, our operations, what metrics, what key metrics are we impacting this month, this quarter, this half a year, this year? Like, what is the impact, like predictive impact to all of the activities that are going on? management takes it okay like I see so we're not you know we're not doing enough to reduce churn or we're not doing enough to capture this market share and then that can flow back down into into the organization.

 

Wait, so you're going to be digitizing organizational structure where each node corresponds to a role in an organization. So you have the product manager role, the software engineer role, and then the CEO node that listens to each of them.

 

It's not a one for one. Think about the fact that we already have AI participants in meetings and that AI participant knows like for that team, basically everything that they're doing. Right? So it's already like following you. It's following Michael along to your meetings. It like, it hears what you have to say, what your coworkers have to say, like it knows what you're trying to accomplish. You know, in not too long, not into to this in future, it's also going to aggregate your collaboration material. like what you're doing in Slack or Microsoft Teams. So it's pretty well up to speed on everything that's going on, and it can assess it, and it can also be predictive about the outcomes. So now you have this node that has awareness of what the local team does. You can expand that to a local work group. Those can interact, and then you can expand that to, or raise that to a division, then raise that to the corporate. global, whatever have you. So it's not a, you're not like putting an AI next to each one of the humans. You just have AI as team members. But because they are AI team members, they don't forget. They can analyze constantly. They don't need to sleep. They can instantly exchange information. So they're a bit super human in a way. And if they get smart enough, they can start to be very contributing

 

That's fascinating.

 

Yeah, I was gonna say something similar to that, but I don't know, maybe I'm an optimist thinking in the future of like maybe 30 years after that is sort of a suite of AI that would be customized to each individual employee as well, that would be conversational in the effect that it can actually pose and notify want to theoretical sort of questions that somebody would have. Because we've all been assigned projects in whatever industry that we work in that we're like, is this really how I should be spending my time? Is this the best thing for the company? And nobody's gonna, I mean, some people do, at all hands meetings, raise their hand, ask some E-staff member, CEO or COO, or like, hey, is this really the direction company to be going and usually the response is yeah that's why we made the decision you don't have all of the context here but to have that sort of that sort of buddy along with you, whether wherever it's embedded in your day-to-day operations as something that you can actually speak to or type questions to or whatever. So like, hey, can you make sure that I'm not overextended on my project work? Like, hey, I have everything that's assigned to me. I scoped it, I guessed that this was gonna take me a week to do this and two weeks to do that and have that AI assistant say, hey Ben, Based on your previous execution on similar things, yeah, that's not happening, dude. Did you forget that you need to do this level of testing for this? This implementation doesn't just, it's not self-contained code. You need to do integration testing with these other different systems within the platform. And last time that you guys messed with that, it took three weeks to do. So I'm gonna go ahead and update for you if you accept this, that I'm gonna tack on an additional of work, I'm going to notify people in Slack for you. I'm going to notify their bots. And I'm also going to update all of our tracking software that's going to say, hey, this is when we're going to be done with this, and I'll notify this team and this other team that this is going to be delayed by three weeks. And having that sort of like bolting on to everything that you said, I think that's the next evolution definitely is that sort of team focused interaction. But I also think that having a confidential personal assistant that's not gonna be phoning home, that's just gonna like got your back, you can ask it anything. Like, hey, am I gonna like totally screw up here if I take this project on and the AI says, yeah, maybe wait for something else or it's listening in on the conversation. Like, hey, here's our quarterly projects that we got. Everybody let us know what you want to work on. And if you're like, oh, these three things sound really cool. And then have the AI bought be like, bro, no,

 

Thank you. Bye.

 

just take two of them. Let somebody else take that other one and you'll, you'll have a good quarter. Uh, yeah, I think something like that would be awesome.

 

I think that's a really interesting idea, like the split between like what's a corporate AI versus a personalized AI. And I think very, very relevant. It's a tough challenge because how do you expose a personal AI to the corporate information without risking it talking to things? And also like, can you trust it? Can you trust your personal AI? So there's a lot of questions like that. I think a really interesting example Yeah, you know, why are we working on this project? Now that like literally happened to me this week. You know, one of my leads connected. He's like, Hey, like the team, we have been doing so many rearchitectures so quickly. Well, obviously GPT is here. And like, you know, are we doing the right thing? Why are we so would be really, really helpful to have something like that that can answer those questions? I think we're pretty far away from that because it's very sensitive that information. But

 

Yes.

 

in terms of like, what should I be working on? I think I think that's like a lot. closer than you think. So like we have this feature in assembly that listens for your, what we call commitments, like when you say you were going to do something, effectively to do, but like we don't want to, like part of the reason we didn't want to call it a to do is because effectively what we're doing there is creating a list of what you're supposed to be working on. And this is very, very close to an AI telling you what to do, very, very close. In our case, we hope it's fairly innocuous because it's things you said yourself and you've committed to and we're not saying it's your to-dos, like they're

 

Thank you.

 

your commitments.

 

Thank you.

 

Whether you wanna make them your to-dos is your choice, but you can actually already connect that up to your to-do app. And so you have your meetings during the week and Monday you come in, your Microsoft to-do app is pre-filled with to-do activities based on what you committed to during the week. That's possible today. And this is like level, you know, this is level one.

 

Thank you. Bye.

 

Now, like imagine what level 50 looks like, right? Like because there's all these things now, you can like make sense of these tasks, you can prioritize them, you can have the corporate AI, I mean, it gets crazy. So we're getting into that territory and it's a little bit of a, you know, interesting and contentious territory of how much, you know, how much of ToDo's AI can give you in fact, effectively, There's like a really good word for this, marshaling

 

Mm-hmm.

 

the resource towards a goal without human input more or less.

 

next generation after that is being able to detect which of those tasks another AI bot could just autonomously do. You look at a devs task list for a particular sprint, maybe 30% of that is developed this new feature that doesn't exist. There's no reference for it anywhere. It's never been built before. So maybe a generative AI could, many decades from now, figure that out from abstract explanation of something. We're definitely not there right now, but another 20% might be, hey, you need to write some examples for code that you already wrote, then it did go on the website, or hey, we need just docs written for, remember all that code you wrote last sprint? Yeah, 40% of the methods weren't documented properly. So could you build out all of the, doc of that. Right now, JetGPT 3.5 for four, they're fantastic at writing, you know, docs. And I've tested it in seven different languages. I'm like, wow, it got the syntax is easy to do, you know, just look up a reference for that. But the stylistic guidelines that adhere to generally accepted standards for different languages, it nails it. And it's, I haven't like ticket in a sprint, but I can't say I won't in the future because it was pretty good. And that next generation of saying, hey, do you want me to do this for you? Because you have better things to do, like go figure out this complex feature. That's the part of these, you know, the application of this for my own personal, you know, selfishness. I really want to see these things get, you know, even better at sort of that, that team assistant that can raise its hand, its virtual hand, and say, hey, I got this, can I do it? Or can I at least make an attempt and then you can check my work? And I think that sort of activity is, it's the same as if you're a new person coming into a team. How do you win the hearts and minds of your team members? You volunteer for stuff that they either don't have time to do or don't wanna do. You're super humble. and feedback, it's employee 101 right there. It's like how you do it, how do you get accepted by a new team. And if AI models are behaving in a way that a really great new team member behaves as a human would, I think people will start anthropomorphizing them a bit and saying, we can't work without this bot. seeing people with emotional connections to this thing, is like, hey, this thing had my back and it did all this work for me. It was great. Yeah, I'm excited for that day.

 

For sure. Yeah. And it's not just code, right? Like a marketing meeting, um, or an HR meeting, like, Hey, we're talking about launching these new website. By the end of the meeting, you have five versions of the website waiting for you.

 

Listen.

 

You know, I, I, you know, I, like the bodice, you know, we'll say like, okay, I heard you're trying to do this new website. Like here's, you know, here's in the stuff here's in this style. Here's in that style. Here's a style based on the current site, blah, blah, blah. Like which style do you like? So you can actually pre-generate a lot of things based on the discussion. the team.

 

Yeah, we're coming up on time and we didn't even get through one third of my notes that I had. But unfortunately, we don't have unlimited, unlimited podcast time. So I'm going to summarize and we're going to wrap for our time as well. So some of the cool things that I heard in this podcast is the system that assembly AI is leveraging. They have a bunch of different models and they used to have a small talk removal model, but that's less of a concern now. And one of the core features that they leverage is sort of getting clever with extraction to classify types of conversation. So thinking about tenses of verbs, for instance, if you're in the future tense, it's probably something that requires it to do list or is theoretical. And if it's in the present tense, probably not. And then they're also working on getting chat, GBT based interactions to be fully integrated with this knowledge graph that is created through all the meetings. Some startup tenants that they follow are simplicity and intuitiveness. I think that's generally a good practice. No one likes complex anything. And then they're also focusing specifically on team tasks. So how to plan a project and how to get that project plan into your favorite Jira software or whatever it may be. And then from a sort of a Holy Grail perspective, it seems like one path forward at least is to develop a organizational knowledge graph. And this can be referenced by LLMs that use logic or any other service. And those can then be leveraged to make smart business decisions that are specific to your use case. And then finally, Ben had some advice. When you're joining a company, do all the stuff that people don't want to do. They'll like you for it. So, Artem, if people want to learn more about assembly AI or your work, where should they go?

 

www.sembly.ai, that's S-E-M-B-L-Y. On the website, there's a free plan, there's also a free trial for a professional version. Give it a whirl, by the time this podcast airs, Semblian will be live. And it's really fascinating, the kind of value you can get from everything from writing a next steps email for you based on the meeting to doing something fun with it, like, you know, extoll the virtues participants in the meeting as heroes in a storybook. So that's on the product. For me, I'm on LinkedIn, RTEM Corrin, A-R-T-E-M-K-O-R-E-N. You can find me there. And it was really great talking to you guys. Really interesting conversation.

 

Yeah,

 

Likewise.

 

well until until next time it's been Michael Burke and my co-host

 

I'm Chad Wilson.

 

and have a good day everyone.

 

See you next time.