Dori Exterman is the CTO at IncrediBuild. He joins the show to talk about the gaming industry and further explain their company, " IncrediBuild". He explains the prerequisites,  production process, and DevOps process for creating games. Moreover, they also tackle the factors that affect game development. 

What's going on everybody? Welcome to another episode of Adventures in DevOps. And we have quite the studio going on today. Joining me as always, Jonathan Hall.

 

Hey.

 

Welcome, Jonathan. And in addition to a live studio audience today, we have joining us the Chief Evangelist from Incredibild, Dory Eksterman. Welcome, Dory.

 

Thanks for having me.

 

Now it's a pleasure to have you. I'm actually looking forward to this topic. We are gonna be talking today about the gaming industry, which is something we haven't covered on our podcast before. But before we dive into that, Dory, do you wanna give us a little bit about your background and what you do?

 

Yeah, sure. So I'm in the software industry for almost 30 years now start, you know, always kind of PC, many different kinds of software, consulted to a lot of organizations in a variety of technological and architectural kind of new stuff. I'm doing a lot of talks in conferences, in podcasts such as this, working a lot with IncrediBuild marketing, all this kind of where technology meets business, meets go-to-market, that's where I find myself enjoying the most. Yeah, the software industry, it's really exciting. I really like what I'm doing. The customers that I'm speaking with will speak more about it, so I'll be able to share more. Yeah, that's more or less.

 

Bright on. Cool. And now we were just talking about this before we started recording the podcast that your role as chief evangelist has moved you from being oriented towards the technical teams to more customer oriented, right?

 

Yeah, I've been at IncrediBuild for many years as CTO, managing the R&D, also the product of IncrediBuild, taking it from a relatively small startup to the size that we are today, serving a very large portion of the expert pro kind of developers in the world today, leading the software development ecosystem. I think that I understood that what I like most is where technology meets business, meets go-to-market, being more close to the customer, to the way that we want to have IncrediBuild out there, bringing all the data back to the organization. That's the fun part. That's what I like to do most, and that's what I'm doing today.

 

right on. So I have actually no experience or even any idea of what the DevOps processes for publishing a game looks like but I can imagine it's pretty competitive and high pressured so walk us through what that what that kind of looks like.

 

Actually, it's very complex. You know the the pipeline is is much more complex than the regular software development pipeline because there are more More things, you know more things to a game than just code you have all the assets all the graphics You have multiple phases as part of your process You know, there are things that needs to be rendered as things that needs to be compressed all the softer kind of usual stack of compiling your code, running your unit test, integration test, API, sanity, whatever code analysis, but you have all these other stuff that are specifically related to games as well. It's a highly competitive market. So, you know, the need for speed is in the bone and marrow of this industry. It's also a very tech oriented industry. very adaptive to new technologies. I think that's one of the reasons that Incredibil grew so fast in this industry because people are keen to try out new technologies that bring them productivity. I think that when one of the things that categorize CI, CD, DevOps pipelines of games, game studios is that I think the DNA of these studios you know they must release on time if you want to meet your Christmas deadline then it has you know harsh implications on your game sales because you know people are waiting to buy that for Christmas yeah release one month after later and you know you're kind of stuck with projections that are not meeting expectations and this is one of the reasons that don't know if you're familiar with the crunch time kind of crunch time notorious culture in the gaming industry where you know before releases bring your sleeping bag to the office everyone's staying you know weekends etc so people are trying to avoid that and that's that's you know that puts a lot of pressure on automation and scalability on the ability to achieve more in terms of So that puts a lot of stress to their CI-CD DevOps pipelines, because they need to be super agile. While they're not really able to do that, they need to be able to do it in time. So that's the kind of thing that we need to do. And then, of course, we need to make sure that they're able to do it in time. So that's the kind of thing that we need to do. And then, of course, we need to make sure that they're able to do it in time. So that's the kind of thing that we need to do. As I said, there are strict release dates. Another thing, it's very competitive. When you look at games, they compete with other games for features, for graphics, et cetera. When you look, for example, at the Metaverse, which a lot of our customers are kind of still in this race for dominating the Metaverse, although it's still far, but the price is so big that you see a lot of investments still going on there. And when you look at the race, one of the main characteristics of race is in order to win it is to be fast. That's a major element. So I think that when you look at a lot of the, and I'm speaking about the AAA kind of studios, the one releasing the large games, I think that speed is embedded within everything that they try to do, especially in their agility and their CI-CD pipeline. And there is a good reason for that. They need a very high quality gamers are not forgiving you know bugs in games that

 

Wait,

 

much

 

what?

 

yeah yeah

 

Unless you play Fallout or Skyrim, then you expect bugs and you love them.

 

yeah

 

Yeah

 

sometimes sometimes it depends on which one you encounter

 

Right.

 

yeah so so i we see a lot of need for speed there so i think you can see you can see a lot of infrastructure tools inside their pipelines, whether things like IncrediBuild for accelerating the build and compile time and shader rendering, et cetera. You see a lot of... network accelerator and solutions for file transfer because once you have all the assets, it's not small assets. These are assets that have a lot of graphics with them. They need to be deployed to staging for QA, for testing. They need to then be deployed to production. But you're looking at a lot of data, a lot of assets. So that's a bit different from the regular software that you have. It's usually bigger than that. optimizations there as well and and automation is Critical, you know, they try to automate as much as they can

 

Thanks for watching!

 

putting a lot of effort in in self serve for developers to be able to have self serve kind of environments And and very techie. So, you know, they're their developers working Our super tech so they expect them to they they expect to have everything, you know in their fingertips characteristic that I see in their DevOps pipeline.

 

I'm really curious to hear what the state of the art is with regard to the sort of software development life cycle within games. You mentioned several things that, you mentioned like staging and QA and some of these things would sort of be yellow or red flags in the places I've worked because that usually means a slower release cycle. But then gaming is a different beast than I'm accustomed to. I mean, I don't know how you would write. I mean, I'm sure parts of... games can be written with TDD, for example, but

 

Yeah.

 

how do you ensure that that ending sequence happens in the right way without executing it? Even if it's automated, it takes 60 seconds to have your guy jump through the right hoops and do the right thing. So can you just talk to us, what does the state of the art look like and compare and contrast that to more like cloud-based web services that a lot of us are programming?

 

So I think that... This is a great question and I think I'm also coming from traditional software. So I think one of the things that you see which is very different than traditional software is when we test traditional software is kind of we know what's the expected result of the test would be, right? We have A, we want the result to be B and as long as the result is B, that's what we want to have, right? So it's very deterministic. and everything will be automated and all my tests, TDD, et cetera. And everything can be without any human touch in terms of quality. But when you're looking at the game, one of the most important things is the game experience. And game experience is not something that you can be deterministic about. This is something that you need someone to play experience that you want to convey to your customer base, right? Whether it has the kind of the feel of the game, the speeds that the game would like to offer to the gamers, whether it have kind of the excitement kind of stuff that they want. And this is not something that you can automate. This is something that you need some people that are expert, you know, that are gamers that can give you the human element of the feedback, that much or even almost at all in other software releases correct me if I'm wrong I don't know what your experience

 

Well,

 

is but I think it's it's

 

it

 

pretty

 

sounds

 

different

 

to me like you're describing

 

Thank you.

 

UX, or user experience, which of course is everything that a game is. I mean, traditional computer software has an input and produces an output. Games don't. They dig inputs, but the output technically is just whatever's on the screen or in your earphones or whatever, right? It's the UX is the whole point. But I mean, to play this devil's advocate here, let's say we're talking about, let's say we're talking about, social media website, right? There's that UX experience that still needs to be pleasant and

 

Yeah.

 

drive engagement however you want it to do that. So yes, you're gonna have your testers who do the scrolling and they click on the links and they see does the video pop up at the right place and is chatting pleasant, you know, these sorts of things.

 

Yeah.

 

But then I can still, I mean, I still think, I'm sure that... Facebook and LinkedIn and all those places still have tests around it. So once they decide what the chat box should look like, they put a test in place to make sure it does that. And I could imagine at least to some extent, you could do that in games, you know, all right. So

 

course.

 

let's just use a game that we all know and love, Mario. You know, we decided that when Mario jumps and hits the box, it should do the little ding-a-ling, whatever, and the coin should pop out just this way, right? So yes, the tester spent hours playing with it to see if it did the right thing. But then once they figured out what it should be like, they could put a test in place, I don't think they did with Mario written in assembly language on a 6502, but they could have put a test in place to ensure that the little coin appeared on the screen at the right time and did the right flips, whatever. And then they never have to test that aspect again. So I hear what you're saying, and I think that you're probably right that the sort of exploratory human testing is much heavier. in something that is essentially purely UX than it is on a social media site. But there's still elements of that in both.

 

You're right. So you have both. And that's I think that's one of the things that characterize gaming, you have both, you have

 

Mm-hmm.

 

both the traditional software development, because it's still software development. So you have unit testing, you have integration test, you have API test, because there are a lot of elements in games that are you know, not even UX, you know, you have the, the scorecard scorecards, you have the, you know, the user engagement, the gamer gamer engagement, you have, you have all this MMO, you know, players, kind of, and traditional players, et cetera. So there are a lot of stuff that are very traditional, and there are things that are very different. Because when you look at, so yes, the answer, Jonathan, is yes. They have also the automated test as much as they can for things that are relevant. I think that you'll be surprised to see less TDD, unit tests, and all these kind of tests in games that you would expect. kind of paste industry that's kind of producing more testing a bit less. I think there are things that, you know, when I'm speaking with the automotive market, for example, automotive industry, which is part of the industry that we are, that that's part of our customer base. There are things when I'm speaking with them, there are things that I'm saying that DevOps community in the automotive industry should learn from, from, from gaming. But there are things that they shouldn't, you know, testing is not

 

Hahaha

 

something I would recommend. the automotive industry to learn from gaming. But I think that there is a reason why, you need to kind of balance and as a CTO, I always needed to find the balance with the efforts that I'm putting. So I think that in gaming, you have some aspects of the test, but a lot is also about the experience. And when I take your example, when you look at UX in a website, for example, feel that you know that it feels fine you know to work with LinkedIn for example or Netflix and and it feels you know generally fine but there are limited things there are limited things that you can do in Netflix then you can do on on on on some kind of open world game

 

Right, right.

 

like whatever you know different things that you can encounter so many open-ended kind of scenarios that it's limitless, you know, and that's different than regular software. You know, when you're testing regular software, I can go and kind of tell you, you listen, you've tested 85% of the software functionality. With gaming, you know, I can speak about the elements like the coin in the Mario, the jumps or not, but

 

Mm-hmm.

 

whether, how difficult it is to reach this coin, how difficult it is after I finished the previous level, how difficult it is after, you know, try escape how many times do i fail to do that how does it uh how do i feel about it you know it's it's very different we're speaking

 

Yeah,

 

about

 

yeah.

 

uh psychology emotional stuff you know there are kind of the the the feel of the game etc so it's it's very it's very different but uh both things are true also for the devops you know you have the the regular devops stuff you have you know the DevOps, GitHub actions, whatever. But you also have some stuff in the pipeline that are unique for game development. You have repositories, okay, but like, you know, any repository that you're familiar with, but you have special repositories in the game at least for the AAA because they need to save as part of the game the assets. That's part

 

Mm-hmm.

 

of the repository. look at convention software, you have source code, you have a little more assets and things like that, and that's it. You can use something which can, you know, it's a regular repository. But when you start putting gigs of assets as part of versioning, you know, because it's part

 

Mm-hmm.

 

of versioning, you know, all these graphics are part of the game versioning, then you need to have a repository, at least for the huge, you know, and FIFAs and all this kind

 

Mm-hmm.

 

of stuff, you need to have repositories that can save assets which are huge and give you

 

Oh.

 

kind of the ability to download these assets, move them around. So again, you have the similar problems and also unique problems and that goes with the tools and practices that these studios developed. We've seen that a lot in work from home for example,

 

Oh sure.

 

expected to release games, you know, it's even more right. We all

 

Yeah,

 

sat

 

more

 

there

 

people want to play.

 

during COVID, you know, waiting for these games to be released. They actually made it, you know, we I think one of the things that we're seeing now is the games being postponed actually in 2022. And that's another topic of why they are postponed, et cetera. That's actually that's also part of DevOps and code complexity as a whole that we see. So that's another topic. we expected games to be even faster go to market. And they needed to cope with that. So I think more than other industries even. So I think that that's something that we see on a day-to-day basis. And it's actually very, it's very invigorating to work with this industry because they are always so fast paced. You know, I always, when I spoke with, speak with our product marketing, with our product managers, et cetera, I always tell them, you know, we just need to listen to our customers where they're going, especially in the gaming industry, we'll see the rest of the industries going as well and adopting these kinds of technologies. I remember, you know, first gaming company telling us that they are working with containers for their CICD, where container was just incredible. It's pretty, is there for a long time serving our customer base. So we've seen a lot of technology trends going up and down, et cetera. So we've been there when the first customer came up with say, listen, do you support containers as part of my CICD? This is something that I'm going to use. And once you see a large AAA studio containers you know that the rest of the market is going to follow its footsteps. So I think that's part of the things that help us identify the market trends as well being part of this industry.

 

I'm curious also to maybe finish up this topic before I move on. So you're building a game, you're probably building it for the latest generation of the Xbox, maybe the PlayStation, probably PC or some subset. But generally, most games aren't built for a single platform. They're built for more than one platform these days. What does that look like in your build pipeline? Are there physical Xboxes there that you're running tests on? And

 

Yep.

 

how does that work?

 

So yeah, so that's another complexity. That's an example of another complexity. So it really depends on the game, right? There are games that are targeting multiple platforms. There are games that target just a specific platform. And according to that, the pipeline and the way that developers are working with the game, it kind of, it affects that. For example, if usually when you see a game that targets multiple platforms that it targets, then you'll see most of the developers working on the PC version, the desktop version of the game, testing on the desktop version of the game, running all the CI, CD pipeline mostly and all the kind of automation stuff on the desktop game, because it's much easier to create a pipeline that's on a desktop, testing a desktop application and working as a developer on a desktop application, then every time you know that I need to test I need to build the entire pipeline for for my consoles for example so usually they will try to do as much things as possible on the desktop side and only when they need to test either it's something you know for example either either when they are before releases or before a major version then they have someone also testing the game etc. Another aspect is that performance testing, you know, performance tuning, that needs to be done on each platform separately. So you need to have people who are expert in performance testing and tuning for PlayStation or for Xbox or for Nintendo Switch etc. So these are things that you need to do explicitly. But it also affects the pipeline as you said Jonathan because I build for multiple multiple target platforms. Each target platform many times require their own kind of formats whether it is our asset format so you need to compress your assets for PlayStation you need to compress your assets for Xbox. Another thing is architecture you know to try and architect your game in a way that will allow you to have to be as much cross-platform as possible in order to for you to maintain as lowest code as possible, specifically dedicated for a single platform. And part of the things that obstruct these complexities are our build engine, sorry, our game engines. For example, Unreal Engine, which is, you know, the most popular game engine for AAA that's commercially available. They kind of obstruct a lot of the complexities for you. They allow you to use one game engine but still, you know, you still need to do your own pipeline, etc. But also, you know, one of the things in the gaming industry is that not everyone wants to work on the same game engine because they kind of feel that the game will look a bit similar to other games. So

 

Mm-hmm.

 

a lot of studios, they have their own game engines. And, you know, supporting and building a game engine, that's a huge investment. of their development investment, their pipeline investment. So it's not only that they need these large, for example, I'm just taking a specific example. EA have, they're part of the studios that are working with their own custom frostbite engine. It's a game engine, very complex one, serving some of the most sophisticated games and give them this unique look of these specific games. And they need to both maintain the game engine and all the complexities that come with it and the physics engines and the way that you bake textures and lighting and everything into your games. And in parallel, they need to develop the game itself. So it's kind of, it's more complex than just building software here, it's kind of you build the engine on which this with which the software is being built, but also the software itself, both of them, for example, most of the time in gaming, specifically our C plus plus in order to give you as much speed in terms of real time performance as possible. And, you know. get the best memory and resource utilization, et cetera. So these are part of the complexities that this ecosystem deals with.

 

Good answer. Thank you. Thank you for going into detail.

 

So you mentioned that talking about the. when talking to the automotive industry, you know, you wouldn't recommend that they take testing practices from the gaming industry. And gaming is one of a very small handful of industries that do kind of lead the way for the rest of technology. So what things are you seeing now that you, like if you could talk to like every other more traditional DevOps shop. What things would you tell them that they should be looking at or investigating to implement in their own processes?

 

I think one... is not being afraid from technology innovation. I think that one of the things that I see there is they are very keen to try new stuff and there are lots of new stuff. I think we see that software is the goal for having better quality products. Everyone understands that and software are being continuously developed. And I think that that's part of the things that traditional software industries, especially around embedded, to try out new stuff and that's something that I think that gaming is doing a lot, the game ecosystem. Another thing is that they really do everything that they can in order to be as fast as possible and it means using distributed computing tools like IncrediBuild, ToraxOrid, Build, with the cloud, whoever can afford it in order to gain maximum scalability. It means a self-serve infrastructure for the developers to get the fastest tools that they can. I think that these, these are the things that we see a lot. We see a huge amount of, of investment in, in, in, in efficiency of the pipeline. You will not, I saw a lot of customers coming from. engineering or things like that that are still in the nightly build kind of paradigm you know

 

Right.

 

you will not see a lot of AAA game studios doing nightly builds and that's it you will see a lot of build per commit you will see you know you will not see that many integration health you see a lot of working with branch features you know because they need to maintain this kind of fast-paced iteration frequency. So I think these are the things that a lot of the industries can learn from this industry particularly. But I think it's also important to go the other side. There was once a saying, it's not relevant that much today, but I don't know exactly who was it from the other side, but was someone telling the Bill Gates said to someone from the telling him you know if the automotive industry would progress as fast as the computer industry we would have flying cars right now and back then this this automotive CEO told Bill Gates that yes but we need to restore them once in a while while we are in the air so

 

Yeah

 

I think that's yeah that was during the blue screens you know Windows we don't see that many blue screens today but still more frequently than I need to restart my car for malfunctioning so I think that that's

 

Right.

 

And that's something that the automotive industry, embedded industry are excellent at. They're having the best quality products, following procedures and strict guidelines and architectural rules, et cetera, in order to make sure that they have zero tolerance for quality, for critical issues, and a lot of emphasis on security and zero trust security. which we see a lot now with our customers coming to us, when they kind of get in the automotive, embedded industry, defense, banks, et cetera, when they come and they introduce a new tool into their tool chain, like IncrediBuild for accelerating their build times, they wanna make sure that IncrediBuild doesn't add anything to their artifacts, right? Do you remember SolarWind was an amazing example of a man in the middle that's part of your tool chain and can really, really make a lot of damage to your release pipeline and further down the line. So they have a lot of them adopted zero trust security. If I'm putting something as part of the tool chain, I need to make sure that when I'm working with or without it, I'm getting the same artifact. So essentially with Incredibil, you can do that, but you cannot do that everywhere. But they put a lot of efforts on that. I think that in other industries, you'll see less effort. It really depends also with what you're expected to get as an artifact. You know, we have seen some bugs in games that if I would have seen them in my car, that would have been catastrophic.

 

Yeah.

 

Yeah, for sure. I like the idea of zero trust, zero trust security. Can you elaborate on that just a little bit about how you

 

Yeah.

 

accomplish that at IncrediBuild?

 

Yeah, so zero trust security is something that comes from the customer, you know. When you're looking at the C++ ecosystem, you can find it with the largest organizations today, and it's kind of characterized ecosystem. You can see it a lot in gaming and software and CAD, CAM simulation, et cetera, but you can see it in also industries that are very keen to security, like finance and embedded automotive, as cryptocurrency, a blockchain, et cetera. And there you wanna make sure that nothing is being injected to your final artifacts. So how can you do that? If I'm using something like IncrediBuild to accelerate my compilation, just for us to understand how we can do that. Essentially what IncrediBuild does is it takes your C++ compilation and instead of just running it on your laptop or your CI-CD node, we allow you to use all your idle CPU cycles within the organization in order to 16 cores on your laptop, you can tap into hundreds of cores. So if your build time can finish much faster, having 300 cores instead of 16 cores, that's exactly what IncrediBuild provides, both on-prem and scaling in the cloud or to the cloud, and also using smart caching. So it's always faster not to build anything and reuse a previous artifact than building something fast. So essentially, it's that. lab ranking on that is because when you introduce something like that as part of your DevOps pipeline and you have another tool that kind of touch your your compile and build process then this tool can potentially add additional stuff into your final artifact that you won't be aware of right so you want to make sure that this is not something that you can do that that software you This is the trust. When, and trust is relatively easier, right? Because if someone wants to trust IncrediBuild, then I can have a lot of things that I can help him trust IncrediBuild. Our customer base, more than thousand companies, the, our integration within Visual Studio, the fact that we are used within, this and that industries, et cetera. But zero trust means that these customers are coming and they say, it doesn't matter what you're going to say. It doesn't matter even if the best security firm will have a triple A kind of vote for you, it doesn't matter. We have zero trust, meaning that we need to make sure from ourselves that you are really not doing anything. So they need to have the means to make sure that they don't need to trust us. They can check for themselves. When you're looking at these kinds of tools, in order to work with IncrediBuild, they couldn't have tested their pipeline without me, right? Because they rely on IncrediBuild because they need to change their build script. So they cannot test it without. But with IncrediBuild, we do not need them to do that. They keep their own build scripts, their own tool chain. We just instrument that. So essentially they can run their same pipeline with IncrediBuild, which will run, you know, 20 minutes and they can run their entire pipeline without IncrediBuild, let's say it will take three hours,

 

Mm-hmm.

 

but it is still the same pipeline, one with IncrediBuild, one without IncrediBuild. And then they can compare the final artifacts. And if they see that with IncrediBuild and without IncrediBuild, they get the same final artifacts, then they can test by themselves that there is zero trust security in the way that IncrediBuild is working within their pipeline. So this is essentially, way that they can verify for themselves and not trust us saying that we are secured enough and in order to have that you need to have something that allow you to use your toolchain your processes because if I needed you to change it and use if I if IncrediBuild would have been a build system then you couldn't have tested your builds without us right because you would rely on us so that's that's part of the things that we need to have and kind of approach to allow customers to have zero trust you know with with products like and credit.

 

Right on. That's super cool.

 

And that's growing, you know, zero trust is growing a lot, you know, especially in these important areas like finance and embedded, et cetera, and are very prone to security. And I think that we will have more than that. The IoT industry is really booming. We are expected to have billions of IoT devices all over the world. You know, everyone's going to interact with a handful of IoT devices in his day-to-day life. devices are really you know vulnerable to hacks we've seen that right with you know using all these cameras etc to do ddos attacks everything that's connected to wi-fi so i think that we see a lot of security and devsec ops coming to the customers that we are serving

 

I'm curious to hear a little bit more about, and I don't know how much this really falls under your expertise since you're really doing building. Part of DevOps is not just building and delivering software, but monitoring and responding to that. How does that, I've been playing No Man's Sky again lately and it crashes pretty frequently and I always hit

 

Yeah

 

that send report button on my PlayStation. I don't know if anybody ever sees that. As far as I know,

 

they

 

it's

 

don't

 

just going into a dumpster somewhere. Well.

 

Yeah!

 

yeah they have it part of their pipeline is when someone presses this button it just go to there you know the junk and that's

 

It

 

it

 

goes to

 

yeah

 

Dev

 

because

 

Null.

 

you know

 

It just makes us poor losers feel better when our system crashes. Maybe they'll fix it. But what can you tell me about that aspect, observability and monitoring of games? Since it's not on their server, they don't get to just poke and prod it. They're depending on whatever data they might get out in the world.

 

yeah yeah so i think that that that's that's a soft area you know that's a that's a pain uh and and you know as as you saw yourself you can you can send the report uh did you did someone send you a patch jonathan

 

So,

 

after

 

I mean,

 

you press this button

 

not

 

not

 

to

 

to

 

me, but they have

 

you

 

been releasing

 

I'm sorry.

 

patches that have caused

 

yeah

 

it to crash

 

yeah yeah

 

less frequently. Ha

 

Yeah,

 

ha.

 

and that's actually that's what they're doing. I think that's, that's, and the successful game always have sequels and they have patches and they have upgrades and they need to continuously introduce new aspects to the game, you know, the successful games. So that's, that's another thing that's really, you know, putting a lot of pressure into their DevOps cycles, because it's not just, you know, release one product we will go, we, if this product is successful, we are going to release a lot and very because that's what generates money. It generates more money than it generates with usual software where you release patches, but you don't necessarily get revenue from them. But in the gaming, when you release a new update patch, et cetera, that means to achieve more revenue. So the entire DevOps pipeline, et cetera, and also kind of the monitoring rely on that in order for them to be able to continue to improve. And it's not only improving. And I think that's another aspect of gaming. And you specifically speak about the quality, but another aspect of gaming is the ability to generate more revenue out of the games. So another thing that in games, which is a bit different, although you have it in other software, but not to this extent, is to understand how can I get more revenue out of each player, right? What do I need to introduce? What kind of challenges do I need to introduce? my market, what kind of stuff I can sell as in-game, accessories, et cetera, and people will buy. So you need to have continuous analytics, not only on the quality of the game, but also in all the areas that really affect the revenue of the game. That's part of the art of continuously monitoring games, which is a bit different than regular software. tracking your bugs, et cetera. It's continuously being able to understand what will bring more revenue. If I'm introducing new stuff, whether people bought them, whether people visited them, whether I'm introducing new levels, how many people went to this level, how many people stayed in these levels, what was interesting, what wasn't interesting, it's a bit different than a regular software. So they're monitoring analytics tools are more sophisticated in a way bugs and things like that but also all the funnel flow and pipeline of the revenue creation and the revenue generation and revenue optimization that's part of making many not all games you know some of the games you just buy for whatever cost it is and that's it but many of the games rely heavily in terms of revenue on continuous improvement and in

 

Mm-hmm.

 

order to have continuous improvement you need to have continuous monitoring continuous and the ability to continuously try and offer new stuff and track how your game has responded to that. So these tools are very, they are out there. There are third-party tools that help you gather all these analytics, try new stuff, both for backtracking and et cetera, but also for revenue tracking and understanding how you can further optimize your profit from the game.

 

Pretty cool.

 

Thanks for watching!

 

Yeah, it's really cool. There is so much to learn from in this industry. It's really part of the software industry, but it's really also different. It has its own methodologies, pace. It's very, very interesting.

 

So in like traditional web development, if we're, if I have the landing page, you know, I'll release a couple landing pages and then AB test to see which one converts more customers. Is there a similar approach in gaming? Like will AB test this particular section of the game to see which one causes the user to buy more stuff?

 

Yep. Yeah, sure. But it really depends on the game, right? You have these mobile games where you can, we can change more frequently and you have these games that you release them once people downloaded into their consoles or they bought the CD, right? And the occasional patch. Of course, you have A B testing A B testing, you know, that's the way to test the things are actually working because before deploying it to the masses. So but it really depends, you know, on the game, A-B testing can be used. You know when you're looking at casual games for example Mobile games there is a lot more A-B testing than in console games, which is more complex To to kind of get this fast loop of A-B testing, but they are used everywhere But if you look at casual mobile you'll see a lot a lot of that there and and part of the Flow is to actually be able to introduce all these kind of new scenarios very, very frequently. When you're looking at these large mobile studios, one of the main things that they invest in is that their ability to iterate fast and introduce new revenue streams through their products by testing them and seeing whether they work or not. That's actually part of their DevOps. And there is a person who is responsible new stuff in order to generate more revenues that's that is part of the DevOps process and then analytics and is using A-B testing etc in order to make sure that more revenues can be generated and the way to efficiently generate them so the answer is definitely yes.

 

That's super cool. That's actually one of the things that I like about DevOps a lot is where you can tightly align with the business and like be closely integrated with the the wheels that actually move the business forward.

 

I think one of the things that is changing in the game industry is in favor towards that trend. You know, one of the things, one of the dramatic changes in gaming that's coming up is actually not the metaverse, which is, which will probably take more time until we, we have, you know, what we foresee as a metaverse. But one of the trends in gaming is the, is the cloud streaming games, right? It's all the ability that instead of installing the game on my PC, I can have service like PS Now or Xbox Live or whatever and I can stream the game and play it without downloading and installing it on my computer. There was you know I don't know if you're familiar with with Google Stadia. Google Stadia is an engine from Google that was supposed to be the new kind of thing in the game streaming and they shut down the project, unfortunately, but I think that the trends still continue. And once we'll see this trend continue more and more, we will have more the ability to, the students will have the ability to act more as, in the SaaS software than as they have today because they will be able to have more control over the game that is being streamed to the developer a game that you purchase on a store, on a CD, and install to your PlayStation or Xbox, right?

 

Thanks for

 

So

 

watching!

 

this entire industry is going towards kind of service, game as a service, games as a service kind of thing or model. And that is going to make it more, come more close to the kind of SaaS, web apps, things, model that we know of and more flexibility in that perspective. And the main reason that cloud streaming is a game, cloud streaming is that trend and we're going to see the entire industry moving towards that direction is because it brings the AAA games. Currently, you need to have either a very strong desktop or console in order to play these AAA games. And

 

Thanks

 

that

 

for

 

reaches

 

watching!

 

only 300 million players hardware for to allow them to play these AAA games but when I'm streaming these games I can play the same game on my tv on my mobile on my on my tablet etc so instead of having only 300 million players gamers potential now these studios will have 2.7 billion gamers as potential players that means nine times more gamers that they had before so the entire industry is kind of tapping into this amazing market and we see this not only from the gaming production side but also from the electronic consumer electronic side if you've seen the previous consumer electronic shows you see a lot of these coming to tvs and remote controls and kind of handheld devices all these things the investment in in cloud streaming that's going to be something that we'll see a lot of in this in this upcoming decade

 

That's just wild. That's just taking like cloud computing to a whole new level, you know, because we've tried it in a few different instances with, um, like, uh, like Amazon has their cloud nine web based development environment and, and VS code has some cloud type environment. We've seen desktop apps like Microsoft office. Now, I don't think you can even get it on a download, but you'll use Google docs, which is all web based. And now it's just going to to gaming which is sort of similar but like the compute and the performance characteristics are completely different.

 

Yeah, yeah. And again, you know, as all the previous examples, again, you have this similar and different in how game streaming look like versus other kind of streaming. For example, if we compare game streaming to Netflix, right? When you're when I'm streaming video, I'm just streaming video one way, right?

 

Mm-hmm.

 

I know, you know, that's the start of the movie. That's the end of the movie. Nothing's going to change. It's going to be the same movie. playing that you play that will or Jonathan play that right it's going to be the same movie But if the three of us are going to play a game, it's going to be a different game for each of us, right? So the streaming of a game is different, right? It needs to, the streaming is related to the way that I interact with it, which is different than when I'm streaming and Netflix movie and Netflix movie, I can forward rewind and pause, that's it, right? With a game, I can move right, I can move left, I can jump, I can do a lot of stuff. And then the game needs to, so it's a lot of about upload to because one of the things that you see in streaming is that they stream the next five minutes of the movie ahead of time right

 

Mm-hmm.

 

but in gaming you cannot do that as you know as you can do that because i don't know what you're going to do in the next five minutes so the complexities there are much more meaningful than anything we are seeing in other industries again again you see you know the the similarities and the differences you know that this industry and complexities offer so game streaming is is there it has a lot of complexities not everything is That's where we're heading. And Jonathan mentioned the SDKs, the ability to work and test things. And another thing that comes up, until now, developers worked on SDKs. They worked on their console. Everything needed to be developed. If you developed on PlayStation, you needed to have a PlayStation in order to test things and develop near the PlayStation, copy your game to the PlayStation and test it there, etc. And one of the things that as part of the cloud transition that we're seeing, is that these game SDKs are going to be offered as part of a cloud native development experience. That's something that we'll see this decade are coming up as well. And this, the gaming ecosystem development is actually waiting for that because it's going to remove a lot of the bottlenecks of moving files back and forth from my desktop, compiling to my console and waiting it for there. the game there. It's a lot of complexities and these kind of people don't have time to waste. So I think that once we see cloud as the case for gaming, that's another kind of, the next kind of phase that we'll see of game development transitioning to the cloud.

 

So I hate to burst your bubble, but I've been streaming games for decades. I used to be a mud player back on my Commodore 64

 

Hahaha!

 

with a 300 bod modem.

 

Hahahaha Ahem.

 

I'm sure this is just like that.

 

It's exactly the same, you know it is. Ha ha ha ha ha.

 

Yeah, yeah, yeah, a bit more complex today.

 

Yeah.

 

Actually, it brings us back to one of the things that we've seen, we spoke about COVID, one of the things that we've seen in 2022, and that's also related to DevOps, is that a lot of games have been postponed, a lot. Release dates, and that was the year with the most postponed games that we ever seen in the industry. And one of the reasons for that is that games became more and more complex. It's really complex. the game engine became more complex, the game itself, the demand from the customers, the quality, the cross platform, all these kinds of stuff caused a lot of delays in gaming. It was also related to COVID and the time it took the game industry to cope with work from home, with all this network transfer, et cetera, which makes it more complex. But I think that that's where DevOps is, in order to allow us all this automation, DevOps practices, agile practices, They are here in order to allow us not to miss our deadlines. Incredibil is here in order to help our customer the risk from not meeting their deadlines. And still, these are very, very capable studios with the best people, best software developers out there. But still, many of them are missing their deadlines. And that means a lot of the amount of journey that we still have in DevOps, in agility, in order to make sure that when we think that we are going to release something, we are actually going to meet the deadline that we gave, you know? And that's part of, you know, I currently, I'm not managing the R&D anymore. So lucky for me, I don't need to meet these deadlines. But when

 

Hehehehehe

 

I did, you know, you don't want to be there. You don't want to be and come to get, tell your CEO, listen, we need to postpone the game by six months. That's not something you want to. I don't want to be in the shoes of someone that needs to do that, you know, with an investor, etc. So, the riskifying investing in DevOps, investing in productivity tools, in acceleration tools, etc. That's part of the ways to avoid missing your deadlines. And I think we see that. We see the understanding into that as well.

 

right on. Well, is there anything else we should cover? coming up on 55 minutes here.

 

I need to get going soon. My wife informed me late that she's not going to be around to pick up the kid from daycare. So I guess I need to go do that. Yeah.

 

Eh, they'll hang on to the kid.

 

Probably they might not like me when I come pick them

 

Or

 

up

 

not. Or

 

Right?

 

not, Will.

 

I love him sitting on top of a garbage can, little bungee strap across him so he doesn't fall over. Cool, let's move on to pics then. Jonathan, getting pics?

 

Yeah, I've got a couple picks for this week. I just started listening to an audio book that I'm really enjoying. It's depressing though, so take that. It's a grain of salt, I guess. It's called The Man Who Broke Capitalism, How Jack Welch Gutted the Heartland and Crushed the Soul of Corporate America, and How to Undo His Legacy. So

 

Right on.

 

I remember reading

 

one.

 

a book that he wrote, I think it was called Straight From the Gut, several years ago when I had a boss who just worshiped this man. So I read the book to kind of try to understand my boss better or understand, understand thy enemy as they, as they say, I guess. Um,

 

Right.

 

Hahahaha

 

anyway, I read the book and I thought this is kind of interesting, but you know, I don't know, I didn't really agree with a lot of it. And this, this book is really opening my eyes to a lot of the stuff that, uh, that, that, that, that book talked about and just other, you know, quote unquote it's kind of history lesson about corporate America. in the corporate world in general, but specifically in America, but also worldwide. So it's a good book so far. I don't yet know how to undo his legacy or I would be doing that, but maybe at the end of the book I'll know the answer.

 

Did you try command Z?

 

I did.

 

Okay.

 

I just got a blue screen.

 

Got you.

 

My second pick is going to be a tool that a friend slash colleague of mine wrote. So he's the co-organizer of the Go Amsterdam meetup with me and a few others. So does that make us colleagues? We don't work together, but we're colleagues, we're volunteer colleagues in this meetup group, right? So he recently released a tool called Mox, M-O-X. And it's a sort of, it's kind of a cool project. It's an all-in-one mail server. It does SMTP, it does IMAP, it does SPF, DKIM, DMARC, TLS, Acme, slash, let's encrypt keys, all this stuff in a single binary. So you can easily host your own mail server. And he's asking

 

Thanks

 

for people

 

for

 

to

 

watching!

 

try it out and play with it. I thought that's great for this audience. If you haven't ever hosted your own mail server, you probably should just so you'd know what it's like and you know how terrible it is.

 

Right.

 

Yeah.

 

And if you do that and you're like, man, this is fun. I like this pain. but I wish it was easier, then Mox is probably for you. So go check that out. It's at github.com slash mjl dash slash mox. And I had him on my YouTube channel, Boldly Go, to talk, we did a sort of a code review of that. So if you're interested, you can find that on YouTube on the Boldly Go channel. But yeah, check out Mox, it's a cool project.

 

Right on. And that was a mail server, M-A-I-L-N-I-T-M-A-L-E.

 

Yeah, yeah, right. I know that the confusion, there's a lot of confusion there, but yeah, M-A-I-L, S-M-T-P, that sort of male.

 

That's why I'm here, is for the bad jokes.

 

Yeah.

 

Hehehehe

 

Dory, what about you? You got any picks for us this week?

 

Uhhhhhh Yeah, actually, just after trying to assemble my first mailbox, mail server, I think that my pick is a book. Besides software development, I'm practicing and teaching Zen Buddhism. So

 

Oh, right

 

for

 

on.

 

all of you, yeah, so, and I'm really into that. I think that's an amazing thing for you to do in order to get a lot of joy and full... fulfillment into your life so I will I will pick something from this you know maybe it will do someone good so I would I would consider Alan Watts the way of Zen Alan Watts is a is a writer it was a writer you know introduced Zen Buddhism to the US back there in the 60s 70s and he's a great teacher I think he he has he was a theologist and and kind of has a very nice way to introduce Zen to so I really recommend Zen Buddhism in general and Alan Watts writing in particular to go into this topic and I think that as software developers you know I'm really you know I was always looking into things that I can try myself you know I'm not into I don't consider it spirituality so I needed something that I can really taste and you know understand etc so I think that for target audience we are speaking with. So I really recommend, you know, go and check this out. That's really working.

 

Right on, yeah, agreed 100%. So my pick, since we're talking about gaming today, I'm gonna go back to one of my favorite games, and it's a few years old now, but the Half-Life series, the latest game of Half-Life, Alex is available on the Oculus VR. So it's a VR game, and it's just an amazing game. If you haven't played it, I highly recommend it. It's Alex A-L-Y-X, because the... Like we were talking about, the user experience is just so well done in this game. You put on the headset and you completely lose track of the fact that you're not in this world. You know, like I'll be playing and the dog will come in and touch my leg or something. It's like, oh

 

Ha

 

shit,

 

ha ha.

 

what the? Oh wait, it's a real life dog. Never mind. It's just that well done. So

 

It's not a smirk.

 

no. But it has you thinking for a few seconds.

 

Ha ha!

 

So that's my pick for the week. Dory, if people want to get in touch with you to learn more about this or follow you on social media, how can they do that?

 

yeah my LinkedIn always welcome Dory Eksterman text me and and I'll answer I'll be more than happy to chat

 

All right, cool. Well, thank you so much for coming on the

 

Thank

 

show

 

you, it

 

today.

 

was a pleasure.

 

This was a cool conversation, very insightful.

 

Thanks for coming.

 

Yeah, right on. Thanks

 

Thank you.

 

and we'll see y'all next week.