Enjoy this intellectually stimulating conversation with Michael Berk and guest on the show, Aliaksei Mikhailiuk, ML/AI engineer at Snapchat as they discuss everything AI computer graphics to techniques on striking the efficiency-accuracy trade-off for deep neural networks on constrained devices.

Hello, everyone. Welcome back to another episode of Adventures in Machine Learning. I am your host, Michael Burke. Ben is still out. He'll be back at some point in the future. And today, we are joined by a special guest who's actually been on before, Alexei. Do you mind introducing yourself and explaining why you're famous?

 

Hi Michael, thank you very much for having me today. As you've mentioned, I've been here before and last time I've been found by Adventures in Machine Learning through my Medium account and one of my blog posts on the tools that I wish I'd learned while I was doing my PhD in machine learning that I currently use in industry. That might be useful for everyone listening to this podcast as well. So yeah, Adventures in Machine Learning found me through the blog post that I have on Medium, so check it out as well. Yeah, a few words about myself. I'm currently a machine learning engineer at Snap in the UK in London, and I have just recently joined the company. Before that, I was an AI team lead at Huawei Cambridge, and before that I was doing a PhD in applied machine learning at Cambridge.

 

Amazing. So both of those are pretty reputable positions. Do you mind walking us through what each one is like and how they're different potentially?

 

So one at Huawei and Snapchat, right?

 

Yes.

 

Yeah. So I've joined Huawei. Actually, it was a very nice coincidence when I was finishing up my PhD, a friend of mine who is Chinese had some friends who were working for Huawei and were looking for an AI engineer. And I happened to be looking for a job as well. So a friend of mine got us connected and... research department in Cambridge that I was working at is essentially a core machine learning on mobile research division that is looking to help out headquarters in China to build solutions specifically for mobile devices. And that would include computer vision applications, including enhancing their camera processing software or adding some new capabilities to the editing software. For example, adding new in-painting mechanism for videos or photos. This was one of the projects that we're working on. Additional branch of it that was recently been developed and where I was leading a team of of a few engineers was looking into applications of machine learnings to the graphics pipeline. So one particular interest that many mobile phone developers or producers now have is making the hardware capabilities and also at the same time enhancing user experience. And gaming market is a huge chunk of the work that they are addressing, where a lot of computations are happening on GPUs and those GPUs obviously being on mobile they consume a lot of energy if they are used at full scale. And one of the goals of using machine learning within Huawei was to... speed up some of the computations, reducing the power consumption of GPU, and ideally also making the user experience even better. So that was a very interesting blend of, again, computer graphics and machine learning for me, because originally in my PhD I was coming from an applied machine learning group that also had some interest in graphics, so I knew what it is and how the pipeline works. But that was also a very interesting challenge of leading a team that was coming both from computer graphics and machine learning backgrounds. And there was some very interesting challenges in making the work flow among the team members. And currently I've transitioned to Snapchat. The position is very much like Huawei. So it's also applied research in machine learning, but addressing the problem of... generative models for human faces. So you know that huge chunk of snapchats businesses producing filters or lenses for the human faces altering them in some way and the challenges are pretty much the same that I had at Huawei where The goal is to port the solution on mobile which means that we have very constrained resources at hand So yeah two months in the new role. So far so good.

 

Good to hear. What percent of your time would you say do you spend reading academic papers or getting into the literature versus getting into the data versus being in boring meetings versus doing other things?

 

That's a very good question. Actually, in my current position, I'd say 50% would be reading papers, simply because I need to get to know the area. And I think it's very much like that in every new role that you'd be starting, you first need to catch up with the major papers and technologies. 30% of the time would be working on the actual code and 20% in the meetings. But this is in the new role. Back at Huawei, it was a slightly different distribution where I'd say 70% would be reading papers because you need to pick the direction for the team and think a few steps ahead, and then 30% would be in the meetings. But yeah, it's a very interesting question

 

Got

 

because...

 

it.

 

I was expecting that after transitioning from academia, papers would be something that I would be spending very little time on. And luckily enough, I'm still catching up with the new advancements in the industry.

 

Are there good resources outside of papers, or do you need those cutting edge topics that are only discussed in white papers?

 

I like very much like the... well, it's also related to papers as well, but Kilture's YouTube channel, check it out. It's a very helpful resource for distilling the papers. I was actually watching one of the videos just before the podcast just now. Other than that, I do like Distill. This is a website that was launched by the... top AI researchers, I think that were Benjio and Godfellow that are backing it. I might be mistaken, but distil.pub.pub. So check it out as well. And Medium is quite a good resource for, again, just general topics in machine learning.

 

Got it. And for the listeners who aren't aware, Alexi does very technical and cutting edge research. So a lot of people tend to just be working with more simple models, but if you are on the cutting edge, these resources might be helpful for you. And we were, so before we started recording, we were actually chatting about the shift and why you made it. Could you elaborate a little bit about why you moved from Huawei to Snapchat?

 

Yeah, absolutely. So part of it was for personal reasons, but part of it was also, as I mentioned, part of the work that Huawei was doing has been drifting towards AI in computer graphics, which is a very, very interesting mixture of the two. I'd say that that would be one of the very hot topics in the nearest future for applications of machine learning overall. One of the things that I've learned at Huawei is that at the moment, if you want to stay up to date with the field and be the top in your field, you don't have the luxury of just doing machine learning. You need to know a lot about... other areas where it would be applied. So you become quite multidisciplinary in this way. So you would need to know very well how hardware works. You'd need to know if you're applying it to some problem, you need to know the ins and outs of the problem. So unfortunately we're gradually transitioning from the problems where you could just train a network on a bunch of data and have have it working straight away. We are now reaching the point where there would be many more challenges if we want to cross this line and do research on the next level. And as I've mentioned, this is essentially what Huawei is currently doing in applying machine learning to computer graphics. I just felt that at this stage for me it would be much more interesting to continue on the journey into the generative models. keeping on with the computer graphics applications.

 

But yeah, I can certainly echo that it's very important to be multidisciplinary when you're implementing stuff in the real world. Often if you have intuition about the product or the use case, that is extremely, extremely helpful and can very much guide what models you choose in the process that you take. So glad to hear that that is echoed as well. I was just wondering if you had any high level notes or thoughts about the transition. Like not the logistics of sending in your laptop and getting a new one, but overall in your career or anything of that nature.

 

Very good point, very good question. I think that one of the things that I've learned from, again, working for quite a few companies is that you need to take your role seriously even after you submit your resignation. I've seen quite a few examples of people leaving the job and submitting their papers and then just not caring at all about... colleagues and the part of the work that they were responsible for during the time that they were with the company in the last months of their employment. So definitely that. So make sure that when you are transitioning you're leaving in very good relationships with somebody with whom you worked with. That includes wrapping up, so making sure that you can document everything and I guess when you are joining a new company, one thing is obviously being eager to work and contribute, but at the same time not exhausting yourself. And this is again one of the mistakes that I had in the past where you burn out in the first several months and then lose interest in what you're doing. So make sure to take your time and... Remember that obviously when you're joining something new, there is a learning period, and take it gracefully. Those would be perhaps more soft skill oriented transition advices rather than something technical.

 

Yeah, I'm also making a bit of a career change, or at least changing jobs, and I can add sort of another story that echoes that. Oftentimes people, as soon as they realize that there's no stakes and they've submitted their resume, or submitted their papers, as you said, and are ready to go, they tune out. And I find that very disrespectful to the team. And it also tends to burn bridges and not preserve relationships. So from a logistics perspective, it's important to maintain those relationships so you never know when you're gonna need them, but also from a team respect and just like being a not an asshole, you should try to contribute to the team and continue to be a good teammate until the very end. And then, yeah, as you transition, it's important to be agile and you don't really know what the role is gonna entail, especially in the first week versus first month versus first three months. After about three months, you might have some sense, or maybe even a little sooner. But staying agile and learning while also knowing your limits is super, super important. So I can definitely echo that as someone has made a couple of job changes as well.

 

How long has it been since you started

 

I now...

 

in your position?

 

This is day five.

 

Day 5, okay.

 

So I'm a veteran already. Yeah Cool, so I was reading through the posts that were some of your most recent towards data science blog posts, and I was wondering if we could chat a little bit about machine learning on mobile devices specifically. Does that sound all right with you?

 

Yeah, absolutely.

 

Cool. So what are some of the major limitations when moving from giant GPU or NPU processing clusters to a mobile device with much more limited resources? What are some of the limitations that you think of?

 

So, obviously one of the limitations that you've just mentioned is the capacity of the hardware that we have at hand. But the problem doesn't start here, the problem actually starts much earlier, when we are trying to move a model that was trained in one framework. to a framework that would work on mobile. So one of the challenges in actually deploying things on mobile is that many of the libraries that we are using to train the models on large GPUs like TensorFlow and PyTorch, they would not necessarily work on mobile. Many of the mobile vendors have their own libraries that they expect the models to be in. So one of the first challenges that you would be thinking about is actually how to port the whole model to a device. Luckily there we have things like on the next, which is a format for the models that is allowing for an easy transition from one library to another. So the first problem starts here. So how do we actually move the model to the piece of code that would be capable of running on... Next problem is what Michael has mentioned, so limited resources that we have at hand. That includes both the memory and also the computational resources that we have at hand. And even though there is a lot of research into how to make models small enough and restrained enough to be compatible with mobile. Unfortunately not all of it is valid, as I've learned through the difficult errors, very painful errors, when you spend hundreds of hours in pruning a model and then it... I find out that actually it didn't reduce the memory consumption at all, because I have... reduced the number of weights, but instead of completely removing them, they were all zeroed, so I'm essentially using all the memory still on mobile. So those newbie mistakes, yeah, I've been through all of them. But yeah, one of the particular challenges, as I've mentioned, is resources. And when we're deploying a model on mobile, we need to think also about things like the Again, we have nowadays several processors which we can use for deep models. GPUs, for example, mobile GPUs or NPUs, neural processing units that are becoming more and more widespread in currently maybe top end mobiles, but they would, I think in a few years' time, they would be in every mobile phone that we would be using. So one of the... even though we have choice now, one of the problems now is that if we want, for example, to use one of the processors and at the same time we would want to communicate with another processor, we have a problem of data transfer from one processor to another. So there is quite a lot of resources, time resources, are wasted there. So I guess those four would be major limitations. So making the model actually ported on mobile by converting it to the form that would be acceptable for a mobile vendor that we are looking into. There is a problem of constrained memory, constrained compute power, and finally the communication bottleneck that is often present when we're deploying models on mobile, where we have to communicate between different parts of the ecosystem.

 

Do you think this is a problem that will become obsolete? And the reason that I ask that is, as we get more and more advanced hardware, we theoretically will have all the processing power we need to do these super complex models. We sent a man to the moon using less power than what's in a current iPhone. So as we get better and better hardware, will these problems become outdated? Or do you think we will still have relatively limited hardware or relatively complex models? that we will always need to be simplified.

 

Very, very good question. I think that we would be running into a problem where the applications that we would want to run on mobile, they would be catching up with the resources that we would have. So back in the day we probably were thinking that once we have a more powerful processor, I don't know, in computer, we would be able to perform any computation. But then once we have this processor, we start thinking about bigger and bigger applications. So once we have more available resources, we start thinking about the ways of how to utilize them. So unfortunately, I think that we would be always chasing the computational power. And that scares me a bit, because where is the limit? The limit would at some point be reached, because we have physical constraints of how small we can go in terms of the number of transistors that we can put on the chip. So what would happen next?

 

Yeah, that's a great question. If we're sticking with offline, well, first of all, do we have to stick with offline predictions or can we send something to the cloud, have it run on a giant server farm, then send it back? Or do we need that real time speed?

 

Well, depending on the application, but I am a much bigger fan of offline computations without the need to send data to the cloud, and there are several reasons for that. One of them is a big thing that we are hearing a lot about recently, is privacy. Sending any data away from your mobile. puts a big question mark of whether it would be leaked somewhere or it would be used appropriately. There are obviously many stories of when the data is not used for what it was intended to be used for. This is one problem. Another problem is, well, if we are sending so much data, then this data needs to somehow be transferred as well. So there would be a lot of cost in terms of the infrastructure that we would be needing to use. to transfer the data back and forth to the servers. Finally, the servers, the more data we send, the bigger the data that we send, the bigger service we need. And that puts a lot of cost on the companies that are running those servers. So obviously, that includes the compute power. So we need to pay for that memory, we need to pay for that. And it will be quite hard, at least in my view, will be quite hard to keep those applications. If we're using the applications that require online computations on the cloud, it will be quite hard to keep them free. So I would imagine that somehow the cost would be transferred to the user as well, so it's not preferable either. So those would be my points in why I think it's better to stick with mobile computations. And probably the biggest point from all, which we're quite often forgetting about, is the energy consumption that is there when we're using cloud computing. I've heard about this research into how much Google cluster centers are consuming. The energy consumption is equivalent of consumption of a small European country. So obviously this is not a very good thing for environment either.

 

Yeah, and one more thing to add, which was sort of hidden in a couple of the things that you said, which is user experience and latency. If we're sending a ton of data over the internet, it might not be as fast as, as an offline prediction. But yeah, a small European country. That's, that's kind of ridiculous. And Google has one of the, do you know the size differences between Google versus Amazon versus Microsoft clouds?

 

That I wouldn't know, to be honest. But I'd imagine that they

 

Yeah, me neither,

 

would

 

but.

 

be quite comparable.

 

Yeah, that's ridiculous. Cool, so moving on to how we solve these model complexity issues, you cited a couple solutions in your post. One is reducing the number of model parameters, for example, using pruning. Another is reducing the model size through quantization. So it's going from like a 64-bit to 32 to something smaller. knowledge distillation and a few others. Which of them have you found to be successful in your experience?

 

There are quite a few other options that we can use for, again, making the model portable on mobile. And typically there is a pipeline when we are developing deep learning applications for constrained devices. And the first thing that we would be looking into is... finding a good large model that would be performing the task well, and having that as a benchmark, we would be then looking into how to reduce the model size. And that might not necessarily come from the distillation first, it would be coming from maybe simplifying the architecture of the model, so removing some of the blocks, processing blocks that we have. Sometimes you read the paper and look at the architecture of the network and think, well, was this actually that necessary? Maybe if we remove this branch of the network, the solution would still work and quite often it happens that yes, it does work. So that would be the first step, looking into how we can reduce the model capacity somehow with the handcrafted logic-based decisions. Next step would be perhaps model distillation, so reduce the model size even further. But the question is what model architecture would use for a smaller learner model. So just to give a brief overview of what's happening in model distillation is we have a large model that makes really good predictions and unfortunately that model is not useful for us because it's too big and we want to somehow reduce the model. and we introduced a smaller model which is called a learner model. And what we're doing is we're transferring the knowledge from the bigger model to the smaller learner model by training it on the outputs of a larger model, but not only by penalizing misclassifications, but also by trying to mimic the distribution of the class labels in the bigger model. And in this way we can grasp this hidden knowledge that the larger model has learned about the relative probabilities, for example, for classes if we're looking into the classification problem. And one of the examples that I was giving in my blog is that, imagine that we have a classification problem where we are trying to classify a car, cat and a dog. And when there is an image of a cat coming and the large model classifies it as a cat, we would also ask the question what is the probability of that image being a dog or a car. And a sensible model would probably assign a slightly lower probability to a dog and even smaller probability to a car, because a cat and dog have much more in common than a cat in a car. So... leveraging this knowledge that is learned by a bigger model by distilling it into a smaller model is what is called model distillation, knowledge distillation. And this would be the next step in porting the models on mobile, so reducing the size of the model into a smaller one. But here we have the problem of what would be the network architecture for the smaller model. We can either use the model that we have already came up with, so throwing away some of the blocks. Or we could, for example, use something else, something that's called neural architecture search, where we would in an automatic way find an architecture of a model that would be performing the best for our task. So... Once we have the smaller model, we would then look into model quantization. I have looked into pruning quite a bit, but for my task that I was looking into, fortunately didn't work quite well, perhaps it would work for someone else. Certainly worse looking into structural pruning, where you are looking to reduce the model size by removing... computational blocks rather than individual weights. This is again from my experience. It's usually a bit more helpful. But typically the next step would be model quantization, so reducing the model size and I wouldn't say that I would prefer one particular technique of all that I've mentioned. Usually they work together the best.

 

Interesting. So if you had to pick two or three, it sounds like reducing the or mainly reducing the model complexity and then knowledge distillation, would you say? What's your

 

Absolut.

 

go-to toolkit? OK.

 

Perhaps knowledge distillation then quantization. So reducing the model size, distillation and then quantization.

 

Got it. Do you ever switch out of a neural network based algorithm, go to something simpler?

 

Very good question. One of the things that we've found when we were working on computer graphics solution at Huawei AI application to computer graphics was that conventional methods that do not include machine learning they actually or deep learning they work the best simply because it's quite hard to make deep learning models real-time on mobile. And those conventional techniques, they have provided us with sufficient quality while not consuming too much of the resources that we have at hand.

 

Got it. And now, do you ever try to find this middle ground between, let's say, a linear model and a super fancy LSTM? Do you ever try to use libraries such as PyTorch Lite that theoretically make the neural network structure a bit more lightweight and fast?

 

Well, that's a very interesting question. Obviously, when you'd be developing a solution and you want to deliver something, a final product, you would look into several branches. So that's exactly what we did when we were looking for a solution as well. We would make sure that we can get both solutions to a level that would allow us to say, okay, this deep learning approach works well, but at the same time it's not computationally feasible, even with all the tricks that we apply on the top. However, this linear regression solution, it works probably well enough in terms of the quality, but that would not consume as much resources as deep learning solution. So when it comes to striking a balance... at least from my experience, you'd need to look into several branches first before you'd converge to a solution. It obviously happens sometimes that you'd combine deep learning solution with something smaller and simpler, simply reducing the architecture of the deep learning model and substituting with something handcrafted, or using some pre-computed data in terms of having the internal computations in the deep model.

 

Got it. And you mentioned sort of this trade-off between user experience and computational complexity and thereby cost. How do you think about it and how does Snapchat?

 

Well, one thing that I like about both Huawei and Snapchat is that user experience always comes first. So, again, even though I haven't been in the role for too long at Snapchat, you can see how much goes under the hood, how much is put into actually looking for a solution that would add... in the first place, deliver very good user experience results. And only afterwards you'd be looking into reducing the model complexity. So when you see those masks, when you're using Snapchat, those have passed through very, very difficult rounds of selection in terms of how well they look. And obviously one of the challenges that Snapchat is facing is that it's very hard to automate the process of selection of those models for human faces, because there are numerous problems that are coming with that. First of all, you don't have an automatic quality metric that would tell how good the particular mask is, because the masks are so different that you'd need to have a proper user study. for whether the mask is good enough or not. Another problem is that, again, there are problems with the... related to making sure that the face mask is not only of very good quality but also delivers what is expected for all facial types and facial shapes and, I don't know, hairstyles and skin color. So those problems also exist, so there needs to be a very high quality control there as well. So from my experience, yes, Snapchat is definitely looking a lot into making sure that user experience is met. Whereas obviously computational resources that are used, well, they're just constrained by the real-time constraint. That said, you just need to fit any of your solutions into that, and I think that I'm a very big fan of this approach.

 

Yeah, and as a researcher that's a lot more fun, you're given a much more challenging problem. Instead of just saying, oh, make us the most money, it's give us the best product. So that makes a lot of sense. Before graduating those models, do you guys run A-B tests to see if they're better than the prior models?

 

Yeah, yeah, there is a whole team that is looking into how well the model performs and typically before the rollout there would be multiple trials in seeing whether any of the produced models works best or better than the benchmark. Obviously, I've seen the level of the quality control that is there and the levels that they are setting for a good model and it's incredibly challenging to meet those levels. So quite a lot of models are completely discarded because they are not meeting the expectation.

 

Got it. What are some of the easier use cases and some of the more challenging use cases you've run into in terms of offline mobile ML?

 

Um... I guess I would have to go back to my experience at Huawei here, where again the challenge was to speed up the computer graphics pipeline with machine learning and that was incredibly challenging because of several reasons. First of all, the performance that you would be expecting from computer graphics pipeline. When we are talking about computer graphics pipeline, we are talking about games, typically. So games in current mobile devices, they are not 30 FPS, they are not even 60 FPS, they are 120 FPS. To meet those speeds, you need incredibly fast models. So to give you just a... An example of how much time I had for the model when we were developing a solution for speeding up the computations on mobile and the computer graphics pipeline. The model had to run in less than 100s of a millisecond. So it's, sorry, less than 100s of a second. So slightly more than... milliseconds. And well, you probably worked with deep learning models and you can imagine how challenging it is to fit a deep learning solution into those kind of constraints. So, that was perhaps the challenge. So, we have really tight time constraints. And at the same time, we were looking into obviously solutions that would probably be... running on NPU rather than GPU, because GPU is already occupied by the computer graphics pipeline, so it's doing a lot of work and you don't want to take resources from there. And as I mentioned before, we are running into this data transfer bottleneck, where we're running the model on the NPU, but then to run the model on the NPU, you need to first transfer the data to the NPU, and then once the model has run there, you need to transfer it back. and we found that it actually ate most of the time that we had. So the time for the model was around one millisecond that we were left with, and this was a rather rather challenging task to perform. Another problem that we had there, probably the last one, when it comes to mobile applications, is that when we are talking about the computer graphics pipeline on mobile, quite a lot of things are happening at the driver level. meaning that we don't have explicit control of what's happening, what data is sent where, and extracting this data and working with quite limited amount of data was actually very very challenging as well.

 

Got it. That makes a lot of sense. Yeah, it's very important to consider that a lot of the mobile device resources will be used up by other processes on the phone. So you don't have, like you said, if there's something that's using the GPU, you don't have full access to a GPU. That's very insightful. And I was also wondering, what are some of the more exciting areas in this field? What are you excited about? What gets you up in the morning? What are some cool algorithms that are being developed? Could you share any of those?

 

specifically in mobile development or in general.

 

Let's start with mobile and then go to general.

 

So I guess, well, I think that there is a very cool branch of research that I find quite amazing that is also now starting to find its application on mobile, which is neural rendering. Again, this is something that comes from back of my experience with Huawei, where there was deep learning models that could aid computer graphics and neural net rendering is one of the applications that we are starting to look into. And this does not only apply to computer graphics when we're talking about games. Neural rendering is actually happening everywhere, in many applications that... we would be working with day-to-day in the nearest future. Think about, for example, Metaverse. There is a lot of neural rendering happening when we want to render someone's head in the augmented reality from just a few snapshots of their head. So this is probably one of the most exciting things that are, well, making me keep going from the morning. in terms of the machine learning research and applications.

 

Got it. And then as a more high level sort of taking a step back and looking at machine learning as a field from the past maybe five years and onward, what has been exciting to you and what will be exciting?

 

I guess one of the things that are particularly exciting are generative models. Again, the neural rendering is one of the branches of that, but generative models in general. So recent advancements in text to image, for example, Dali model by OpenAI, where you can type a line of text and it would generate an image for you. This was actually one of the things that's... A friend of mine has recently brought up with me whether I would want to try this new cool toolkit that they have released. So essentially OpenAI has let everyone try and use their model. It's somewhere on the web, so Dali too, you can look it up and you'd queue up and then they would let you use it for a bit and then you can see the results. So it's actually quite amazing what current models can do. and scary as well. So when maybe a few years ago I was looking at interviews with Bill Gates, who was saying that artificial intelligence is probably one of the biggest threats that we currently have, I was sort of disbelieving. But now that I've seen the scope of the applications and the quality of the results, I'm actually starting to think that, yeah, perhaps there is something in there that we should be aware of. So answering your question, generative models is something that excites me a lot from the previous research and ongoing research. But another thing that is also quite interesting and applies to generative models in particular is explainable AI. And this friend of mine who has actually recommended... looking into Dali is finishing up his PhD in explainable AI and it was very fascinating to learn from him how little we can actually extract from those deeper models, especially generative models, to understand how they work and how they come up with the solutions. If we want to make those models more practical than we have now and move them to a more life-critical or life-important applications, we definitely need to know how they come up with the solutions and here explainable AI comes at hand.

 

Yeah, explainability is a really tough area because you tend to only see the outputs and working from outputs back to the causes is really like a challenging concept in life. Like why is your friend such a nice person? Well, I don't know, maybe his parents were nice or maybe he like this or that. So working backwards from outputs to causes is such a challenging concept in general. But yeah. Thank you for those notes. Do you think that Terminator is on the horizon?

 

Definitely.

 

Yeah, everybody go get cans of soup and water. We're all done for. Yeah, one thing that I think would be really interesting is pairing those generative models into a metaverse slash internet space. Basically, we don't have any hardware capacities to support these learning types of models. But if we let them run free on the internet, they have an unlimited playground where they actually can do a lot of damage. So I'm curious to see how that plays out. But yeah, I think we're sorry, go ahead.

 

Well, we've already had this Twitter bot a few years ago that

 

Yeah.

 

turned into a rather nasty bot. I wanted to say person, but no. Turned out very, very nasty.

 

It is not. Yeah. Yeah, I mean, it's definitely an exciting time. And Europe is actually way ahead of the US in terms of ethics. Like the fact that you were talking about data privacy and sending it over the internet, that doesn't even compute for me as an American. And I think that the EU has done a lot, a lot better of a job to handle data security. And hopefully America follows suit, but we will see. Cool, so is there any closing thoughts or if people want to reach out, how can they get in touch?

 

Definitely look into my LinkedIn account. This is probably the easiest way of reaching out to me. Follow me on Medium. I try to regularly write about topics in machine learning, both technical and less technical, so more on the soft skill side that I'm learning on the way. you could use to reach out to me.

 

Well, thank you for bestowing so much knowledge about your career change, ML on mobile devices, and some other scintillating topics. Until next time, this has been Michael Burke and Alexey Mikhailuk, and thank you for tuning in.

 

Thank you very much.