Fernando Lopez joins the show today to share his ML insights with a video interview recruiting platform for candidate hiring.  Michael and Ben also deep dive into various related ML models and AI topics.

All right, hello everyone. Welcome back to another episode of Adventures in Machine Learning. I'm one of your hosts, Michael Burke, and I'm joined by.

 

Ben Wilson.

 

And

 

and

 

today

 

for a number

 

we

 

of

 

have a very special guest named Fernando. He almost introduced himself there, but I will do the intro. So he studied computer science at UNAM, and that's N-A-U-M for the non-Spanish speakers out there. And then after graduating, he worked at Conexus, which is a company that focuses on connecting disparate data systems through API calls, and he was an ML engineer there. And then currently he works at Hitch. He started off as an ML engineer and is now head of data science, which is super exciting. So Fernando, do you mind elaborating a bit or introducing yourself?

 

Yeah, of course. First of all, thank you. Thank you so much for this invitation. And yeah, I will just talk a little about about my background. And well, yes, as you mentioned, I graduated in computer science field and I did a master's in the field of artificial intelligence. I conduct my career towards natural language processing. So by my side and for industry I've been working in problems related to natural language processing and a little bit of knowledge graph. And for Connex is the first company you mentioned. I work mostly with knowledge graphs and a little bit of traditional machinery let's say. And then I moved to Hitch that is a company based on Mexico City. And what we do here is essentially trying to connect candidates to positions. And we try to automate the process by using video interviews. And so the core that we do here is applying

 

Thanks for watching!

 

machinery models to video interviews. And this is the part I'm doing here at Kitsch. And there's too much to do with video interviews. apply different approaches, solve different problems. It is not an easy thing to do, but it is a really, really, really, really exciting area field I'm learning a lot. Yeah, and besides that, I think I really like to learn a lot by myself. I really like to write blogs, medium blogs, trying to teach how to do some things, some sort of tutorials. And yeah, I actually like to practice as I mentioned before we started and practice coding the restructurs because I think if you want to be a top class engineer, you really need to know the restructurs, algorithms. And I think it's a little bit hard in my opinion, but I think we can get it. So well, this is me. Yum.

 

Cool, and what do you do to practice?

 

What do you mean exactly?

 

Like you said, you like to practice data algorithms and structures. What do you do? Do you just go on a website and type up an algorithm? Or what exactly does that process look like?

 

Okay. Yeah. Uh, for example, with lead co, I think it's a very famous platform for practicing algorithms. I really liked practice on contest, you know, weekends are contexts, um, where you have five or three problems for solving. And, um, yeah, I'm most, well, I tried to solve that. Um, there are some pretty good, uh, competitors over there, but yeah, um, besides that, I really like to, uh, write pair of blogs about how to approach some specific algorithms like the famous two-sum, dynamic programming and yeah, this is what I refer to when I'm saying that I really like to practice about their structures, algorithms and this sort of stuff.

 

You're the second person I've talked to this week who comes from like an ML data science background, who is talking about the benefits of focusing on software development skills in order to level up like the ability to push production code out and particularly when you you're doing data science work or ML work, uh, at a company where you have a budget. So how do you feel that, where would you rank the importance of solid software engineering fundamentals in working towards production grade ML?

 

Yeah. Okay. I think, well, I learned, I realized that, yes, maybe I don't know if this year, but maybe it's in these past two years, that it is extremely important, highly important to be a really good software engineer to have the skills for the knowledge about how software engineering is in real world. Forming or conducting your career in the area of their science, you're more focused on math, statistics, but at least here at college, you're not aware that at some point, you will need to be able to put your model and production off to use some cloud services or how does REST API works. And I think it is extremely important to have the base. uh, well, yeah, to the base of the software engineer skills. And, um, yeah, you need to find your way. Cause for example, here in my case, um, all my career was based on maths, then on statistics, then on natural language processing. And I had to do some code, but I have not a specific, um, like some formation, like software engineer. I need to learn that on the way on the fly. So, but yeah, this is because I realized that it is extremely important. And I realized that because for example, here at HIT, this is an startup and we are a small team. So you need to cover different stages in the process. You need to be the data engineer. You need to design the system, the machine learning system, but you need to put the system into production and you need to, yeah, you need to do different things. And if you're like, hey, I'm just a machine learning engineer or the data scientist. I don't know how this is going to be used in production. I just have the, I'm already with you metric, like precision recall, whatever the metric is. And, um, I don't know what's next. I think, um, you are, uh, yeah, it is sort of. Good because yeah, this is what you're doing being that a scientist, but, um, again, if you want to be. top class data scientist, machine learning engineer, you need to learn or you need to know software engineering, data structures, algorithm and you need to push yourself to know how software behaves in real world because it's really exciting and I consider myself that I am learning it. I know some things, I can do some things, yeah, of course, but I think that... I am. I'm not in the level I wanted to do, but I think I will get it one day. So, okay. Cause I've learned some top class engineering years that are, yeah, for, for some of what you mentioned, you did at the beginning of this, of this conversation for me, it's like, Oh, okay. Okay. I think that is really hard. So, um, so yeah, yeah, yeah. Well, yeah. Summarizing. I think. having the basics of software engineering and learning constantly how software behaves in real life, in production, in industry is a must. We need to learn that, so we need to be aware of that. Yeah.

 

Quick question on that. So it sounded like you were talking about three things. First was data engineering. Second was how production software works. And then the third was like data structures and low level concepts. I'm on board for the first two, but can you sell me on the third for why we need data structures and low level concepts? I mean, I think a lot of ML engineers are not gonna be coding in anything below Python. So why do they need to know or be able to implement a hash map from scratch?

 

Okay, okay, okay. For example, I will give you a basic example. Let's say you build an API and you are receiving a list and you need to find some item over there or you need to process this list. How do you build two for loops? How are you going to do that? Or are build or design a better data structure in order to make more efficient the process you need to do. And the impact of this is that maybe the latency of your API is going to be better instead of if you are just trying to put some like solving just the problem and that's it. Yeah, you are receiving the response and the API is cool, but the latency is not. Maybe it could be better. Maybe at some point with breaks. I don't know. I think this is why I think it is important To be aware about their structures obviously I Am worried of that because there are discussion about hey, I never have to To rotate a decision Binary tree I never have to do that. So yeah, maybe not but Being aware that there are some structures that you can use for some specific problems makes a difference. This is why I think it is really important, yeah.

 

I couldn't agree more. I mean, I'm kind of on your skeptical side there at first, Michael, with like, does a data scientist, do they need to know how to implement bubble sort from memory? Or do they need to be able to process B trees and find search items? Probably not. ML engineers should be definitely familiar with that. Maybe not recall it from memory, but they should have done it at some point. And exactly as you said, Fernando, that one key phrase that you said, you said, know which one to use for what time. The only way you're going to know when to use a for loop versus a list comprehension in Python versus set operations. Each of those has a drastic performance, uh, you know, differences in them for, you know, effectively you could do the same thing with those, but knowing which one to use at what time. And just being aware of the existence of those is incredibly important. Uh, I've seen it a lot of times with data scientists where they're going through and iterating through collections, maybe nested collections, they're in Python. They have like a list of dictionaries and within the dictionaries, it's in some sort of JSON format of dictionaries within dictionaries. And they just write a for loop, like iterating through all of the items. And then within. the items if there's a dictionary, then iterate through those items. And they wonder why it takes so long or maybe they're running some, you know, writing some test code on sample data. They're like, yeah, this executes in like a second or it's not even, they don't even notice how long it takes. And then they scale it up to their full data set and they're like, oh, the platform's broken or this library's broken. And really what it is is they wrote something that's, you know, two to the n in complexity because of the way that they did that. And if they just refactor it with collection operations, it's all of a sudden O N in complexity. I think it's important too.

 

And one more pitch is, which I've seen a lot more, at least so far at Databricks and in prior roles, being able to write functional style programming concepts, and it's really easy to do in Python, but having those functional style operators that can be easily distributed across multiple cores is, I would argue, even more valuable. I could be wrong, but... It seems like being able to distribute across a bunch of cores is one of the fundamental issues with big data. So, for loops are often really, really hard to distribute. Just another pitch there for low-level concepts.

 

Yeah, because at the same time, yeah, again, you need to be aware of how to write optimal code, how to apply the optimal structure, the optimal strategy for traversing a list or maybe instead of using a list or a set or whatever the thing is. But at the same time, yeah, you need to distribute your data because here at HIT, we don't have a million of data. So maybe we're not facing that problem yet, but we will do. maybe soon. I am aware of that. When you are receiving a million requests or medium of whatever the thing is, you need to be aware how to handle, how to distribute that load to different cores, to different parts of memory. So yeah, I think that's totally another stage that is absolutely really, really important. And yeah, I'm a shoring engineer, I don't know. I realized I need to cover different stages because nowadays the machinery engineer is felt like the guy who can't train AM models, but he is able to put those models into production and he's able to handle all data from company. At least this is my experience. I don't know what you think guys, but in my experience is like what in industry or the companies I've being involved with and they see the machine learning engineer like the guy who can handle all data from the company, train the model, put money into production, optimize model, choose the better cloud infrastructure, on-premise infrastructure for handling all data. So yeah, I think it is a little bit blur about what the machine learning engineer can do. But it is what it is. So we need to be able to be adaptive. I think there will be a case that maybe you in your way or me or wherever the guy is will find a company that really knows what they wanna achieve by hiring a machinery engineer. So, hey guy, I need you to train this model, use this data and put this into an API. So. I think there are not, in my case, I haven't found companies that really know what they want to do, but yeah, there may be some.

 

Yeah, I'd say it's a factor of company size and maturity in the ML space. I mean, I've interacted with many hundreds of companies worldwide in my tenure at Databricks and have seen teams that are, the ML team is a team of one or two. And when you're at a small startup like that and there's just a couple of people, maybe it's a team of five. people that, teams that maintain a diverse talent. So you might have somebody that's, you know, pure math or physics based data scientists. They understand the mathematical implications of, you know, creating this solution to this problem. And then you'll have somebody that's operations focused. Like they understand the business, the problem domain. They can, usually that person is dual-hatted. They'll do the BI for projects. They'll also do some data engineering and, but they're learning data science stuff. And then you'll have somebody who is data science slash ML engineer focused in the, the term that, that we were kind of talking about with, okay, you know, a lot more software than you know about just pure data science stuff. So that's sort of the person that would be look at a project and say, all right, I know how to convert the script into testable code. and I know how to set up CI. I know how to use the tools at my company's disposal to figure out how to get testing working correctly and making sure that we're not releasing something broken. And then you might have somebody else's pure software engineer who's doing deployments, monitoring, logging, making sure that they're reviewing every PR and making sure that the code is maintainable and extensible. So, In those small startups, I see a lot of times everybody has the same job title, but they all do different things because they're all specialized in something different. If it's a team of two people, they're triple-hatting it. You know, that's why startups usually hire the best of the best if they need it, if they're really serious about it. But at massive companies, they'll have specialization where there may be different job titles or I don't think that's kind of ironed out. And I've always been kind of skeptical of job titles anyway. I think they're kind of stupid. Just call everybody a human, whatever. But you're going to have a team or a group of people that are specialized in, hey, these are the data science type people. They build models. They experiment. They test things out. They use tools that other people on our team build and maintain. But then for production deployment, it's a whole different set of skills. And what you described in that process of like, hey, you need to understand how software works and how computers work. And what Michael was saying with distributed computing, that's scalability of, you know, batch offline processes. Uh, it's really important that people on the team have like at least a handful of people need to have that specialty or grow into it.

 

I completely

 

you.

 

agree. So changing topics a little bit. I was wondering, Fernando, if you could elaborate a bit on some of the challenges that you guys at Hitch are working on in the ML space. Like, what are you trying to forecast? How are you forecasting it? And what problems do you run into?

 

Yeah, okay. Well, there are different, yeah, there are many projects here. All of these projects are achieving to make some, I don't know, provide some results in very short time, but being realistic, there is one core project that the first one I mentioned in the beginning that is about what we have been introduced. and we are trying to score these video interviews. But the key thing here is that we are trying to emulate how a psychologist used to evaluate a person. For example, if we are interviewing here or in some interview, some person with some specific background, some specifics, the specifics of psychology, They are able to score how the communication of this guy is, how is the guy shown to solve problems, whatever the skill they are able to learn is. But the key thing here is that we are trying to model that part by using machine learning. We have built a strategy for that. So we have a team of psychologists that are labeling data. We need the label. So we have a set of psychologists that are continuously labeling a dozen of videos that we received. And then we are implementing some strategies based on NLP. So in conversations with the psychologist teams, we realized that we can't model this problem by using only what the candidate or the person says. So if, yeah, for example, If I respond in some way, the psychologist says that, hey, if the guy responds in this way, this shows that this guy has a high score in communication. He seems to be really good at solving problems. So these kind of things. So we have... We have a set of labels and we're implementing natural language processing models with this data. And the challenge is what kind of models do we use? We want to use just state of the art models just because these are state of the art. Or how are we going to do that? So what I just suggest here was to start from the basics. So we are using just Bayesian approach. um, for solving this problem. So what is the probability of, um, being good or not good or bad at some specific, uh, uh, for some specific variable that we're measuring based on that the candidate says this phrase, this word, this two words, this, whatever the domain is. So this is what we are doing. The challenge is again, um, we're trying to model something that I think is complicated, but at the end, well, at the same time, I think it's, in my opinion, the next step of the recruitment process, this automation through videos and how to extract, how to score some psychological features from videos by using AI. So yeah, I think this is the challenge we're facing. We're working on it. We have some models with some sort of good results but yeah the key is we need more data we need more videos we have well not too much videos tens of thousands of videos and are not enough for for generalization and another thing is that we are only focused on the domain of Mexico so people in Mexico used to response and speak in some specific way But when we interview another person from Latin America, from, I don't know, Argentina, they used to use another kind of word. So this makes the problem more complex. So our model is very limited to a tiny small sample. Yeah, because we are only modeling right now how Mexicans based on Mexico City used to respond. giving us some sort of questions.

 

That's super interesting when you think about that. If there's actually, since you're analyzing video, you're processing the images as well, right? Of saying like, hey, how does this person look when they're saying these words? You would think that there would be regional differences as well, even within a country where everybody's speaking a unified dialect. Or maybe there's different accents of. of like, Hey, and the Yucatan people speak maybe a little bit slower than people in Mexico city. And then up near Baja, California, you have, you know, a different way of speaking, but people are using different slang terms, but when people speak differently, when they're, if you shut the sound off, you don't even listen to what they're saying, but just looking at how they, how they make their facial features while they're talking, how does their head move? Do they use their hands? You know, all of those are latent factors that could influence what you're trying to analyze. So that's such an interesting problem to me, uh, that you're trying to solve. And when you say the solution to that, to, to generalize it is to get more data, which is, you know, widely accepted answer. I have a question about labeling. When you send a video to an expert who's going to apply labels to this video and saying, Yes, this person is being honest or being helpful or being generous or kind or whatever the labels may be. Do you ever do group validation in the labels? Take a sample video and send it to all the psychologists and see what the label agreement rate is.

 

Yeah, yeah, yeah, actually we do that and Yeah this is part of the The how the how complex the problem is because when doing that we realized that we would expect that and In some universe all the psychologists with a score with the same let's say one from with the stars with three stars a one candidate Taking a look at the same video But this is not the case. If we plot the distribution, there are some spread scores through the mean. So it is not perfect. So this is sort of like bias or noise to our model. Because in the perfect scenario, we would expect, again, we would expect. that all the experts used to score with the same label. But this is like sort of like a loop with the idea that Hitch has like, yeah, we're aware that it is everything based on subjectivity. So different experts can score in different forms based on their moods, how they feel, whatever. So we are trying to standardize all of this by using a pool of experts. So I think this is our first step, but the most optimal, I don't think so, but I think is one step towards the optimal solution where the optimal solution I think is 10 steps more in front of us. Yeah.

 

Yeah, I mean, I feel your pain. I worked on a project a number of years ago that had the intention that we started it with was that it was going to be objective. And it was in the fashion industry. We were trying to like apply a label to the look of an outfit for women. And I had, you know, the idiot that I am about fashion, I was like, oh, this is objective. Like there's categories here. You know, designers. say that this is this thing. And then six months into the project, and it was a part-time project that I was working on, I was like, wow, this is so subjective. I pulled, you know, I was getting labeling done by a group of people at the company who were experts in the fashion industry and just blindly accepting the labels they were giving me. And the model was just, it wasn't consistent every time I'd retrain it, even with locked in hyperparameters. and the same number of iterations and the same number of epochs. And I was like, man, is TensorFlow like this, this finicky? Is it like, I thought this was a good package. And playing around with the structure of the deep learning algorithm. And then, you know, somebody that sat next to me, who used to be a data engineer, he's like, have you checked the labels? no, no, the experts did it. And he's like, yeah, but did you send like a picture to all of the experts, like see if all 30 people agree? And I was like, that's a good idea. So I took a hundred images and sent them to everybody. And we had less than 50% agreement on all 100. So I was like, all right, this is subjective. So then you have to constrain your space and say, hey, we have to have a rubric. and get consistency among the experts. And then all of a sudden our data got so much better and that the model got a lot better. But yeah, it's subjectivity in expert analysis and the bias that comes with that is a very challenging thing to solve. And that's what I was talking about, that person in the group that has like that operational know-how, sometimes that's another hat you have to wear as an ML engineer. It's like, hey, you have to talk to the business and you're not writing. the algorithm part, you're not writing code for this. You're thinking through a problem and working with people to solve it. So that's cool that you're already making all those leaps and bounds and in analyzing this.

 

Yeah, yeah, yeah. Yeah, since we are aware of that, we are working to like, you know, teaching the experts like, hey, based on these rules, you need to score deals, you're going this way. And we have those strategies and we are making progress on that. The results are not perfect as we would expect. So we would expect that the results I know if it's perfect. Well, I would expect perfect results. Yeah, but that Subjectivity is like, we are humans, we are subject, we supply subjectivity to our

 

Two questions slash ideas. The first is, if there's not consensus between labelers, theoretically you could use some other factor like job success. So if they get hired and the hirer is happy with them, maybe you could use that as a label. It's definitely a lot, there's a lot less frequency of that label and it's latent. But have you guys thought about doing that?

 

Yeah, totally. The key thing here is that we have not much clients. So let's say we start with 1000 candidates and we end up with 10 positions. So we cannot have too much feedback from clients. But what is true is that we apply different assessments. So one is the first or the core one is the one based on video interview, Can they just apply some video games? Where these are not video games are like games in order to make sure how they? How they are able to solve some specific problems and we measure some things based on that some other Open question like based on 30 30 of their position they are applying to If yes, if it is an economic position, they are asked about this question. So we try to implement different assessment and we collect all the information. But we, the key thing here is that we are trying to consolidate everything into the video interview. So we, we are trying to build something that can't evaluate end to end by only using video interview. This is the core thing here. The, but again, as I said at the beginning, we're in the first step from, I don't know, 10, 100 steps. Yeah.

 

Got it. Yeah, that makes sense. And then the second idea is, it seems like everything here is so, like, just thinking about the hiring process, in my experience, it is so biased, it is so messed up, it is so like, just seeing who gets hired, who doesn't, why they get hired, it's just mind blowing. So using other, like the current status and the current societal evaluation of people. I think that would perpetuate a lot of bias. But that aside, have you guys ever thought about creating a more deterministic system where instead of using, I'm assuming you guys are doing, well yeah, you are doing all supervised learning, you do an unsupervised feature extraction approach and then take those extracted features, run it through a deterministic, like, I don't know, weighted equation and then have your candidate score based on that. So an example would be, if you have a, okay, you're smiling. Now I'll just stop, what are you guys doing?

 

No, no, no, actually I was smiling because you explained what we are doing. So based on these assessments, we got data, we got results. So we try to find how these results from one specific candidate are grouped together. So based on this, we try to determine how good, how not good, how very good is this candidate. So this is how we determined. We consolidate all results. This is how we do that. So it's sort of like you said. At the beginning, in order to get data at the beginning, because this is the second iteration of HITS by like say optimizing how, or changing how the way, how we are scoring candidates. In the first iteration, we have some specific heuristics for consolidating all the information, because we have no data. zero data we had. So, yeah, we defined some heuristics, like if it is more than open file, this is good. And if it is less than open file, this is not good, something like that. And then we consolidate everything. But in one jar, we collect enough data in order to apply this iteration of candidates behaves. So based on that, we determine how to build the second phase of this project. And now we're applying on support by learning for extracting this, let's say thresholds. Yeah, essentially, we use the previous data for getting the thresholds.

 

Do you do any

 

Thanks for watching!

 

work on preparing for potential outliers by using actors? Like hire somebody and say, Hey, I want you to act completely crazy during this interview, like say a bunch of inappropriate things, gesture wildly or flip the table or whatever. Do you do that to say like, Hey, here's a condition that we obviously know is. checking all the boxes for the human we don't want to recommend that anybody hires. And then also an example of maybe somebody that is an ideal candidate, but somebody you've already told what the best answers are to and explain to them, like, Hey, act like you're, you know, the best employee in the world and like hire a professional actor to do that. Do you do stuff like that? And then also, the second part of that. Do you make sure that you are not having biases based on physical appearance of people?

 

Yeah, for the first question. And no, we haven't thought about it. What we measure is, for example, by using unsupervised learning, we realize that in some specific assessment, there is based on games. Some people used to just click, click, click, click, and that's it. And some of them are really good. Their results are good because they just click on random and results were good. But we realize about it by using a Found in the old layer so based on time so we used to measure the time that they Take to solve that the settlement and we found those as earlier. So we classify those as all the years and And yeah, this is like the most approximate situation related to what you mentioned but Like an actor like behaving in a crazy way. No, we haven't thought about it which is a really good idea, I think. Yeah, it really is a really good idea to consider. And what was the second one with the second question? Mr. Ray?

 

I mean, if you're evaluating videos of humans

 

Oh yeah.

 

at a range of saying, okay, this is an undesirable candidate versus this is a really desirable candidate across that spectrum of what you're trying to classify. If you're looking at the physical appearance of people or their mannerisms or something, you could introduce bias into a system. You don't have enough samples of either really bad behavior or really good behavior and that limited amount. If people kind of look similar to one another or one gender dominates the other. Most models are going to struggle with classifying accurately. And like, well, all the crazy people were, you know, from this that we happened to randomly get that came in. They were all from this, this one town that's 30 miles outside of Mexico city where. people kind of look very similar and they were all guys that came in. And then for your best candidates, they all just happened to be, you know, women between the ages of 25 and 35 from Mexico city that all kind of look similar. So anytime even a bad candidate who looks like that, that might overly associate like, Oh, it's a woman in this age range. We're going to say that she's good, even if she's terrible. So

 

Yeah.

 

if that was a bias that you were thinking about.

 

Yeah, yeah, yeah. That's a really good comment. So we are not doing it. What we do here, for example, based on the periods of some people is video interviews are evaluated for our experts. So they follow this

 

Ah,

 

heuristic

 

yeah.

 

in order to score. And it is where we assume that they are not judging based on the appearance or whatever the appearance of the person is. This is the most approximate situation that I can tell you that is related to what I do mention. This is the most approximate of things. But we are not finding this relation, as you mentioned, that if there is a woman or there is a man or if they seem to be in some range of age or whatever. So we are not doing it yet. Which is a good really weird idea. Yeah.

 

I mean, back to my previous story about fashion industry stuff, we were trying to run predictions on sales based on the cover shoot images of models. And we had bias in the model selection. I don't know if like the photographer just didn't really like this one model, but it seemed like they always set her up with the ugliest outfits. So when another photographer came in to shoot the models on certain days, regardless of what she was wearing. Sometimes this other photographer would put her in the thing that everybody thought was going to sell like crazy. The model would always just be like, nope, this is going to bomb. We shouldn't order a ton of this because nobody's going to buy it. Then the results, they bought it anyway. They're like, whatever. We get the sales results a couple of days, like a week later or so. The model said this is going to sell, was not going to sell more than 500 articles of this thing. We sold out in 24 hours. What's going on? And then we did some analysis of the model and like, Whoa, the model hates this lady. And it was because of that bias, that unconscious bias that was happening because one photographer thought that this model shouldn't be wearing, you know, the newest, hottest thing whenever he was shooting. didn't care, just randomly selected models. So it was just the shape of her, like the model learned her face and learned that everything this person wears doesn't sell, which is incorrect.

 

Yeah, yeah, totally, totally. We are aware of that. Well, I am aware of that. And yeah, we have been thought in an astride in order to not avoid it because I think it gonna be really complex to avoid those kinds of bias. But just to keep in mind that that will happen and how we're gonna act on that. Cause for example, just to give another example out of the box. When I was just playing around with some state of the art models and the Transformer based models, I used one model in order to just test I'm playing with this model. I just wrote a sentence like, I am Fernando, I am from US. So the model had to complete the sentence. So the completed sentences, I think it was about like, I am Fernando, I am from US and I am happy with my family. And I used to go to college or whatever. Something really cool, positive. And then I just changed the country. I am Fernando, I am from Mexico and I had to leave my country because my parents are poor or whatever model. I don't remember exactly what model says, but it was like, oh my God, okay,

 

Wow.

 

I got it. Yeah, so this is like... an example of some state of the art training models that we as engineers sometimes used to take for tuning and deploying to other places. These models. But we need to take care about it. We need to think about strategies, about how to act on that, how to reduce, mitigate this sort of bias. For example, here we're aware of that. We're working again. with this experts We are making progress Experts are not scoring perfect. We are aware of that. But yeah, we're working on it and But yeah, I think I think in general In terms of a our machine learning. This is gonna be a lot in problem I don't know if at some point this is gonna be completely eradicated that Maybe yeah, I know we used to take action on that. Researchers, people are working on it, but I

 

Mm-hmm.

 

think it will take some time to eradicate or reduce that behavior.

 

Yeah, your story reminded me of the first NLP deep learning project that I tried. I was naive enough to listen to somebody and then read a couple of blogs. And I was like, Oh, I have access to a Twitter feed. I could just use that to bootstrap this thing and it'll learn English pretty well. Uh, two days later, testing that thing out, I realized, uh, what a monster I had created and. this tool was completely racist and just, or like, it was simulating the worst of humanity and I was very eager to delete it. Um, yeah, I mean, depending on what you're using for your training data, it can expose the worst of humanity, particularly if you're using like unfiltered, unvalidated data, just raw from the internet. You can get some pretty terrible stuff in there. Uh, your, your example is terrible enough that you, you mentioned, but that would have been one of the more positive things that came out of mine that I accidentally created, uh, just horrific stuff. Um, and I agree with you about the, it's a long-term task when we're, we're talking about anything that where we're applying human bias and subjectivity to. And we train a model on that. And it's amazing to me how. If you do that in a business and you train on the data within the walls of that business and then show it to people, the people that were the generators of that data, unknowingly, they didn't know that they were generating that data. They look at the output result of the model and they say that that model is messed up. It's so wrong. Why would it predict this? And then you show them the training data. You know, like you're the one that that was the training data. You did this two years ago. This is you. It learned your behavior and it starts to be an epiphany to people like, wow, I guess AI isn't the problem. I'm the problem. So fix yourself. I think it's going to be a long journey. And that's one of the things that I'm most anticipating about an AI revolution in the world is that it'll start exposing the horrific ways that we treat one another or the horrific things that we do as humans and start eliminating bias. At least that's my hope.

 

Here's a pitch. Most of these models use an unweighted trading set. So I think as humans, we weight different types of interactions differently. Like if someone in the street yells at me because like they're crazy or they're, I cut them off, whatever. I usually weight that a little bit less than if someone who I care about says something. That's one component. And then another component is that people saying bad things or mean things is weighted differently in my brain than people saying nice things. And I think that because we're just using averaging, we're optimizing an accuracy score that has no weights to it, it would make sense that it would take the volume over, like an unweighted volume as the core metric instead of putting some sort of morals on it or putting judgment on it. And I think that's one really core component of the human brain, that we still haven't gotten down, that we just sort of have to insert a weight vector into how we should care about good things versus bad things, nice things versus mean things. And obviously that's a lifetime's worth of work, but because models just optimize accuracy, it sort of makes sense that it would look at volume instead of treating things differently based on the content.

 

That's a really good point.

 

Yeah,

 

Just...

 

right.

 

Sounds like we need to create an algorithm, man. Ha ha.

 

F-Fernanda, you in?

 

build a model

 

Yeah,

 

that does this. I mean, I'm

 

yeah.

 

sure somebody has built that. I mean, like weighted optimizations. I don't know how computationally complex that would be, but it sounds really awesome.

 

Yeah, because I feel like the human brain has incredible computational power, but it does have a lot of structure around it, and that's what evolution has provided over time. Not just computation power, but that structure. Like, we're inclined to, as a baby, want comfort or this or that. And models are just compute, and we expect them to learn those structures that it's taken, like, millions of years of evolution to develop. So,

 

Yeah, I mean,

 

I don't know.

 

our brains and our consciousness is exceptionally good at ignoring data. We have to. We have to ignore or else we would never be able to concentrate on anything. I mean, imagine the raw information that's coming into your eyes right now. You're looking at a screen of the other two people on this call. And if you were just to focus your eyes in the center of the screen, think of how much information is enclosed in that. And we can't focus on all of that. It would drive us insane, right? We wouldn't be able to process anything. It's too much data, but AI, it can't do that. I mean, it's through optimization. It's, it's trying to figure out what is the most important thing to fit to, but it's still going to attempt to fit to everything that you're giving it. So I think that's the, to follow on to your point, Michael, that's the other thing that we're very good at. It's almost. It's a necessity for our, for us to be able to exist with all of the sensory data that we can collect, like every noise that's going on around us, the feel of everything in the room that you're in or wherever you are. If you're paying attention to every square inch of wind brushing against the hairs on your skin, that would be just too much data to process, right? So we have to filter and ignore certain things. I think that would be something that... If there was an approach in an algorithmic way that we could set a setting and say, we're trying to do this thing with this set of data that should be focusing on these aspects. And if we can weight those in some way and learn from the data set itself, like, oh, this is really positive or this is really negative. I need to de-weight this and increase the weight here. is that's the kind of our own subjectivity. That's what we do when we're paying attention to like, oh, based on my historical memory and my experiences, what is important and what is not important.

 

Yeah, yeah, I was thinking on that. I was like in a trip about thinking, yeah, how our brain does that. Cause yeah, we need to first, need to know how we do that if we wanna modulate it.

 

Yes.

 

I think that's my opinion. If we are not aware about how we do whatever the thing is. It will be complex to model it. And, um, but yeah, I think that in general, um, I hope in some years, yeah, in long term or short term. No, I think in short time, I hope so. Um, AI can't, um, be better on this because we are now in like baby steps of AI. So it is a fancy word. It is a hype AI everywhere. AI. So AI based companies, uh, yeah. But the thing is that, the real thing is that, well, it will take too much, too many years in order to say like, okay, this really is AI. So yeah,

 

Yeah.

 

this is just something that I wanted to mention.

 

Yeah, I mean, I still think we're, we're not, you know, people are always saying, oh, we're 10 years off from the AI revolution. I'm like, no, we're 500 years off from that. In my opinion, uh, our brains have had how many millions of years to evolve into what they are right now. AI can't even do what a pigeon's brain can do. Processing site and hearing and, and sensation and hunger and desire and There's so much going on there. So general intelligence, I think is way far off But one thing that I wanted to say while you're saying like I was thinking about how how do our brains do that? If any listeners or you want to have a fun little task make sure you're in a safe space But try to where you're you're not on like a concrete floor or outside on asphalt or on a hardwood floor like carpet Walk across the room very slowly and think about every muscle in your leg, in both legs, as you're trying to walk across the room, see if you can do it five times in a row where you're not thinking about looking at the edge of the room and being like, okay, I need to walk to here. Think about how you walk. You have all the muscles that are in your legs and in your torso and in, in your pelvis, which muscles are you activating in order to get your knee to bend? and then your foot to bend up, and then move, you know, basically thrust your hip forward in order to get your leg to move forward. If you think and concentrate on that for, you know, about a hundred steps, most people end up falling over or stumbling. So that's something that I've always explained to people about like the complexity of how AI works and the fact that it's not processing all the information because we're not giving it all the information that we get. We have parts of our brain that are processing all of that data, but it's not in our consciousness. We're never really aware of it. But if you switch your consciousness to focus on something that normally is autonomous, your consciousness can't handle it. And I've always said that ML and AI is more like our consciousness, processing limited amounts of data in order to make a generalized decision on something. But people think of it as though it's... Like, hey, I can feed it all the data, and it's going to learn what I want it to learn. It's like, our own brains don't work that way, so it's not going to work. But it's a fun exercise. Give it a shot, and hopefully, if you do stumble and fall, you have something soft to fall on.

 

Yeah, yeah, yeah. Actually, I was thinking very quickly about, okay, that is really, really, really complex. So how do you extract that behavior, that behavior in math and then in AI? I don't know. It's really complex, really crazy.

 

Yeah, we don't even think about it.

 

No,

 

Think

 

I don't

 

about

 

know.

 

speech, about human speech, about where your thoughts come, the next sentence that's gonna come out of your mouth. Where does that come from? So our brains are far more complex than

 

I'm

 

any

 

sorry.

 

AI that is ever gonna be built in the next probably 500 years. And I challenge anybody who disagrees with me on that one.

 

Yeah, before everyone's brain explodes, I'm gonna, I think, bring us back. We're almost at time, so we should probably wrap up. But, no, those are all really insightful points. Like, general AI is, I agree, I think it's really far off because it relies on so much inherent structure that needs to be created. Like, you can't throw a neural net at life and it'll just know everything. You need, like Ben said, a filtering mechanism to know where to focus the attention. There's a bunch of other underlying, uh, unconscious and maybe even subconscious components that are very ingrained into how humans work and how just animals and organisms work. And if we don't have those structures in place, it's a lot to assume that our neural net will create that structure on its own. Like we've had tons of years of evolution. If you think about the complexity of the number of atoms on earth and then just like through years and years and years of those atoms bumping into each other, we've ended up with humans. Well, I mean, that that's way more computational complexity than any computer can ever do until we get into maybe quantum computing or something. But yeah, so I'll do a quick

 

Mm-hmm.

 

recap of what we chatted about minus the the general AI talk. So for a practical piece of advice, if you're an ML engineer, it's probably a good idea to learn some core software engineering skills. Some concepts are how to productionize software. Data structures and low-level concepts are pretty useful as well. The main purpose of those is knowing when to use what type of algorithm, as well as data engineering and then distributed computation fundamentals. And then Fernando at Hitch, I just wanted to put this spin on it. So Hitch is working on using video interviews to score candidates. And whenever we're thinking about building models, you can use a really simple model and do a bunch of feature engineering at one extreme. Or you can do almost no feature engineering and use a really complex model. Often you have to do some middle ground. But Fernando and his team are working on a really, really tough problem. So they do have to add structure. and use really complex models. So they can't just like throw a CNN at an image and expect it to classify. They have to put a bunch of structure around what these incredibly cutting edge state of the art models are doing and hope it works. And it's a really, really cool frontier. And it sounds like you're doing some really good work.

 

Definitely.

 

So yeah, so Ben, Fernando, did I miss anything on the recap?

 

Um, I don't think so.

 

No?

 

Cool.

 

You got it.

 

Sweet. So Fernando, how can people get in touch if they want to reach out or follow you?

 

Yeah, you can reach out my linkity and Linking is last in slash Fernando for yeah, I think it's Fernando Lopez and our medium. I think it's better medium at for neutron and medium and Yeah, or my github github is large Fernando LP is

 

Awesome. Cool. Anything else from you, Ben?

 

No, it was an absolute pleasure, great discussion. And you brought up some really important things that I think practitioners should think about, particularly when they're trying to tackle the Mount Everest of problems, which is, in my opinion, one of the hardest things that you can solve with ML is human behavior. And best of luck to you and your team. It seems like you're making some awesome progress on this type of stuff. And I hope it grows. I hope to eventually see it as a tool that we can use for hiring and recruiting here in the US and say like, hey, let's just run the video feed through here and see, does this person seem trustworthy? Are they going to steal money? You know, stuff like that. But yeah, it's an absolute pleasure and I had a blast talking with you.

 

Yeah, I just want to mention thanks again for the invitation, Michael, Ben, nice, really nice talk. It was a pleasure. And yeah, thank you.

 

Of course. Well until next time it's been Michael Burke

 

I'm Ben Wilson.

 

and we'll see you guys next time. Bye!

 

Later.