Star Citizen Live | Tech Talk w/ Benoit Beausejour
Transcript — every row opens the video at that moment
- 00:00:00
Let's get it. >> Hi everybody. >> Let Benwa do that thing in the background. >> The show's actually started while so this will be on the YouTube version. Hi everybody. Welcome to Star Citizen Live. I'm your host Jared Huckabe. If you've never seen Star Citizen live before, it's where we take about an hour, although not in this case. It's going to be a lot longer, out of the end of our our week sometimes, and uh talk to you, the Star Citizen community. Uh we put our developers to the test. We we we we have them answer questions. We have them talk about various aspects of their work. Uh sometimes we show off really unfortunate plushies that never made it into production. We do all manner of
- 00:00:47
things in this show. On the show this week, we are doing the prophesized sequel uh to a special ed to a special edition show that we did back in the beginning of February this year. Uh now, for those of you who might not remember or you've joined the project since then, welcome aboard. uh back in February this year uh after uh uh Chris Roberts announced in his letter from the chairman uh a fundamental change in the manner and focus of our development for this year uh Ben and I uh took to seal and spent two or three hours I don't remember how long >> about three yeah >> would have been good to look at that uh before I came here >> um to just shoot on every shoot on everything uh
- 00:01:34
talk talk about everything under the sun figure out what these changes really meant and stuff like this. And we said at the end of that show that we'd be back in roughly about uh six months. So six plus two is eight. It's nine. Would you expect anything less? >> That's right. >> Uh so we're back. We're we're we're back today. Do a quick recap. All right. First the Earth cooled, then the dinosaurs came. Um now we're back to talk about uh a whole variety of topics. This is not your traditional Q&A show where we take questions from the uh community. Although if we see some good questions in the Twitch chat, we might throw them in from time to time. This is our interview format. So, we have worked out a series of topics that we want to cover. Uh based on our previous conversations, we expect this show to go
- 00:02:22
a bit long. So, you know, if you got someplace to go, you know, you know, be sure to check out the YouTube replay later. Uh, if you'd rather uh listen to or read an AI summary or something on on on Reddit, I mean, I can't stop you. I can't stop you. Um, but yeah, you're going to do what you're going to do. We're going to do what we're going to do. Let's jump into it. Let me start by introducing uh our guest, the man who needs no introduction, but we're going to do it anyway. Chief Technology Officer, Mr. Benois Boseer. Ben Wisa, how you doing, man? >> It's getting better every time. your pronunciation. It's always going up. It's great. I'm feel good. I'm feel good. >> I feel good. >> Well, I appreciate that. Well, who did who did we uh uh Oh, I we just hired just added somebody uh to my to my team
- 00:03:11
and I've never like she's worked for me for about six, seven weeks now, and I've never once tried to say her last name out loud because I'm afraid. >> It's okay. You can do it. >> She's not watching the show, so she doesn't know I'm talking about her. >> Good. Um, Benwell, who are you and what do you do for Star Citizen? >> Uh, well, I am a developer. Uh, and I am the chief technology officer for CI. That means that I oversee everything technical. So, gameplay engineers, engineers, graphic engineers, AI engineers, everybody who would touches a computer almost, you know, that's basically my job. >> And how much of everything that's wrong is your fault? >> Well, Uh people on spectrum would say everything but uh uh yeah I mean you know basically
- 00:04:02
uh I empower all the teams to build the game and so technically everything kind of is my fault in a way you know >> fault fault fault fault. >> All right so like I said we we we we have a lot to talk about today. Who knows how long it's going to take because it's not like we have any time to rehearse or hash any of this stuff out ahead of time. So we're just going to jump into it. Um, we'll do we'll do it live. >> We'll do it live. So, part one >> of this show that I'm inventing right now in my head. >> Yeah. >> I want to I want to call part one the year of playability. So, the year of playability is not necessarily a statement that we ever made. We never we didn't say it in the in the in the February show or whatnot, but it come it comes from that that letter from the chairman that went out at the end of last year with a I think it was like 2025 a focus on playability.
- 00:04:51
It's a it's a fair assessment of of the mission that we set out for ourselves uh at the at the end of last year. Um just uh have we during during the February show, remember guys, we're figuring this out as we go along. It's been a busy week. So during that show, we talked a lot about how Star Citizen had been going in one direction for about 10 years, you know, since since the launch of the PU at the end of 2015 and up until the end of 2024 and what not. Roughly about 10 years there, you know, we we were we were on one kind of mentality. We were on one it it it changed a bit with three when 3.0 came out after that big long period where we didn't ship for eight months and my beard went crazy and whatnot. But we went to this very feature driven, we
- 00:05:39
were on this very feature- driven building the foundation of the game uh uh kind of attitude for so many years where we didn't we never we didn't take a whole lot of opportunities to stop and look at what we look at what was out there and and and spend time on fixing that and hardening that and so it was always about we got to get the next thing out. We got to get the next thing out. We got to get the next thing out because we have many of the same thoughts our backers do. It's like it's like you got to get this stuff out. Uh we decided that in 2025 because of the impending release of Squadron 42 in 2026 and the and the and the need to let those people focus on that and getting that closed out whatnot. We were going to allow the engineers that remained on their focuses remained on squ on Star
- 00:06:27
Citizen to take a step back from new feature work and look at hardening and fixing and addressing a lot of the long-standing issues. Now, I said a lot. I didn't say every. It's as many as we can within the time. You know, it's it's a 10-year journey towards one direction doesn't get turned around and fixed in 3 months, four months, 5 months. Uh, and we said this at the time. Nonetheless, there's still people who, you know, we we see the same comments, guys. We see the same things like, "Oh, so much for the year of playability." Um, that was literally my brother. I'm not I'm not making fun of anybody else. I'm literally quoting my brother. And if you've ever met my little brother, that's exactly how he sounds. Uh, because he plays the game. He plays the game. And I get crap for my little brother. It's not just the backers. It's It's you, Clark.
- 00:07:17
Welcome to the show. Um, so I want to talk about that. I want I want I want to talk about that. That's how little all all little brothers sound. At least in our heads. At least in our heads. I just want wanted y'all all to make I want to make sure everybody knows I was making fun of one specific individual uh that I show. >> I also sound like that sometimes when I talk to John Crew like what about my ship? You know, >> so so let so let's talk about this year of playability and and how it's gone so far. We said we'd check in uh six six months later. We're roughly here. Uh how how's this year of playability going? What what what have we done? What have we achieved? Uh where are we at in our process? Are we happy? You know, Ben, while you do the Benois thing. >> Yeah. Well, I mean, I want to go a bit further out than just the beginning of the year, right? Because let's let's say
- 00:08:07
let's take same date last year. So, let's go full one year back. Um, you know, the state of the game one year in the past was still single server uh DGS, right? So, we're talking about maximum 100 players per game shard, one server simulating all of the universe. No second star system, right? So, only one star system. Server performance, not good. >> Four, five frames. >> Yeah. I mean, if you're lucky, right? Like when you load it in >> like the Yeah. the the the the parameters for us to launch a patch was basically sevenish, which is horrible if you think about it, but and so that was last year, right? So we start from far, right? >> We were still >> we were still in the clutches of 30k. >> We were still in the clutches of 30k,
- 00:08:55
correct? So we we did have some some level of server recovery, but a lot of issues related to it that could have side effects. Uh some of them are still present. We'll talk about some of those. Uh but uh and so you know fast forward to now like the assessment is a bit is a lot better I think in my mind right like obviously we've had the release of 4.0 like it's been a bit of a roller coaster I'm not saying it's been just a straight line like we've we've we've taken some dips and then uh you know imp improved the stability and of the quality of the game overall but obviously we have the 4.0 final release which came with uh static server meshing which was the element that we needed to free up server performance right so on average now when we look at server perf um I mean other than some egregious problems which we've
- 00:09:45
identified the per we get per territory stats now so basically we don't just get like server FPS is X right we're going to know Stanton server FPS is X or you know Hurstston server FPS is Y and so we can compare depending on which territory what performance we're getting and FPS is just one of those parameter right there's a lot more >> uh but let's stick to FPS just because it's a simple mental model right because it does have an effect on your gameplay so nowadays server on average server FPS is up 20 easily right uh obviously some zones like let's say when we launched I think Stormageddon uh there were some issues affecting those zones that made those servers like dip below that quite quite a bit and so they drag the average down, but overall the the game server performance is is
- 00:10:34
way way way way better and I think having 6 700 we've pushed up to 800 players per shard uh which is something we're pushing at right now. We'll talk about this in if we talk about server meshing a bit more. Um, and so we're going to be pushing this really hard uh moving forward. But the I think that if you look at compared last year to this year uh for in my mind it's not the same game. Completely not the same game. Like now our if we look at even the stats uh from our players like uh session length is up like 10x. It's not even comparable. Yeah. uh you know so that means people play for longer periods of time and they stay online. I think server recovery is a big uh is a big help for that though the stability of this of the game server is
- 00:11:21
also improved dramatically and so um you know the this is all because of the efforts of the changes that we've done not only to the priority calls like you just explained right which is taking you know where we're going and trying to slowly shift it to change direction because a team like this of this size doesn't switch on a dime it takes it takes a long time for, you know, these these changes to trickle in. But now that we've got kind of the big organizational changes in that we're starting to see and reap the reap the rewards of it, which is I I hope it's showing in the game anyway. It's showing for me. So, I'm seeing it. So, that's you know, so compare one year back to now, I think it's a it's night and day. >> There's still a long ways to go. I'm not saying we're happy with this. I'm just saying,
- 00:12:08
>> yeah, >> you know, >> there's there's folks in the chat uh specifically talking about, okay, well, let's go do Hatheror and let's go do this and and you you you alluded to that there. It's you're talking about a macro sense throughout the the entire persistent universe, every single aspect of the game. It's to call out specific, oh well, this specific area is bad and this specific area is. That's not what we're talking about. We acknowledge that and those are areas that still need improvements at you know as we do this Russia nesting doll kind of conversation and go down we'll start drilling down to individual aspects later in the show at at its at at its broadest though at its broadest though overall the things that would just prevent people from being able to log in the things that would prevent people from being able to stay in the you know th those those larger
- 00:12:57
scale issues that it doesn't matter matter if you fix the other things. If you're constantly getting a 30k, if you're constantly getting kicked out, if you constantly can't get back in, it doesn't matter if we've improved Hatheror performance if you can't get to Hatheror. Those big those bigger things are what we're talking about here. And those have seen some measurable marked improvements. I mean, an improvement from four to 5 FPS to 20 FPS is huge. Nobody even talks about 30ks stuff anymore. Um, server recovery is I mean, it's we it's hard to make a trailer or anything. I we sat there we back when we thought about how how do we make a a trailer uh for server recovery and stuff and then it's like we we didn't for two reasons and I'm going to talk a little bit here about why marketing pushes some things and doesn't push some other things. one is there's no there there's no real interesting way to show it
- 00:13:47
without you know you you've all seen the server recovery screen you're sitting there and doing but also we I'm just going to be real we're only what we're only 13 minutes into it let's get to a real moment of truth here right now uh over the course of the 11 years I've been here we sometimes toot the horn like we've invented something that honestly should be standard it it it should it should be standard standard should it's it's if we were doing things different if we had had different priorities at this point or whatever that would have just been a thing we would have had before. So so we couldn't have server mesh we couldn't have server recovery before because we had to it was part of our the the whole u nesting architecture within meshing stuff we had to build an infrastructure before you could have a recovery that would bring that stuff back. So the way that we chose to do things meant that
- 00:14:35
some things couldn't come to later. I mean there's people again we were we we'll talk a little bit about org stuff and stuff. You know we still have orgs in our game and stuff. Those those are things that are foundational that you would find and you know day one of other MMOs and we're here you know 10 years into the PU and we still don't have it. So, I want to be real clear that when we talk about this and we and we celebrate our achievements, and I'm really proud of our achievements this year, the stuff that Ben Watson are real, this year is this year is a night and day difference from last year. I I think anybody being real candid and honest with themselves would would admit that. At the same time, I don't I want to be candid and honest and be like I don't also don't want to pretend like this was some we invented the wheel and whatnot. This this was this was this was the change
- 00:15:22
that Chris set out for us last year. It's hey, let's slow down. Let's refocus and let's find those things that we lost along the way and and and catch back up. Does that mean a slowdown on new features? Yeah, it does. And it has all year as we told said it would in February. But just little reality check for everybody. I want you to talk about how we did it though. We're going to spend some time. We're going to talk about freight elevators and all that stuff right now. But up here at the top though, how did we do this change? What were the organizational changes and the operational changes that we did that made this possible and that are going to continue for the foreseeable future? >> Well, um, so there's there's a series of things like companywide that have had to change for this to happen. Um the first
- 00:16:12
the first one we talked a little bit about this in January which was uh the idea of having hero cards or or joker cards I call them uh on specific issues right and so we've uh and then I it was funny because people were like give the joker card to this other bug right and it's like yeah that's basically what we do uh but we've extended that concept a bit further and I hate that we changed the name but we did change the name and we call those hero squads. Now, so what we've done is that we've we've taken a subject, for example, transit, right, which we've extensively talked about in January, and we've made a hero squad around it. So, that means we've dedicated resources just to the to all the bugs related to one system, right? And we've changed the methodology. So
- 00:17:01
the the the way that we used to deal with bug fixes in the past uh is basically engineers get the bug report, they put it in their task list, at some point they emit the fix and they and they they ship it in the patch. Now as the patch release comes closer and closer, those those bugs get verified and or need to be accelerated. And so it's always a bit of a the fixes are are are valid, they're correct, but they're we don't have time to test them fully. we've had a lot of regressions, right? And so regression was becoming a problem because we were going too fast to try to solve some issues. And so the some systems like transit are complicated enough that you you can't just emit a fix. >> I want to pause you there. What is regression? >> Regression is when I say I fixed bug X
- 00:17:50
and I've created bug Y. >> Okay? Yeah, you know, so it's it's basically when we're introducing new bugs or breaking other things because we try to fix >> and so regression was a big problem on the project in general. That's it was in part because of the speed at which things were going uh the code review process uh things like this. So we've these hero squads basically we flipped a bit the relationship. So we decided that fixing issues, let's say I'm going to keep transit as my hero squad example because it's an easy one to reason, but we've put resources dedicated to it and then we got the team the QA teams to basically derive the priorities within the squad. So instead of saying, hey, production management is going to, you know, drive the priorities, we got the QA leads to do that. And so now these
- 00:18:39
guys were completely driven by player impact. And so when they look at issues, they're like, is this impacting players or is it a fringe issue, right? And they decided on the priority. And it turned out that that accur that priority accuracy was a lot better. And we we asked those teams to be relentless into what they want to reach. And so classify the issues in order. And then the moment we get a green light from these guys, it's a celebration because they are hardcore. I've seen a I've seen a lot more Nathaniel Fischer this year than I ever >> the Nathaniel Fischer seal of approval is a real thing. You know, >> I want to inject Bronx 76 as Benois is wrong. Regression is when old issues resurface. >> You know, yes, that's also a definition that works. But the you know, we we use
- 00:19:27
the same word for both at CIG, but the you're you're kind of correct. The proper definition is that's an old issue that comes back. But in this case, the thing is the issues are so close by and and typically it is resurfacing an original issue. Uh that that's actually one of the issues with those big systems is that bug reports are so similar uh but are sometimes completely different circumstances. But >> yeah, two two things, three things, four things that all cause the same outcome. >> Same symptom. Yeah, exactly. So, hero push squads, we got that uh the embedded green light is what you just got done talking about, you know, let letting let letting QA sort of manage their priorities and greenlight things. >> No, that one's a bit of a separate one actually. So, so the the hero squads are is really about getting the our QA teams
- 00:20:16
to drive the priority on a specific subject and drive down, right? >> One of the change in methodology for the devs on those team on those team is like a bit of a change in approach. We in code review for these in these hero squad, we no longer just accept the fix based on code change. We ask the developer to explain what the problem is. Do can you reason at why this is happening? Second, proposed a change, right? And then the leads can validate this change before it even we start even writing any code for it. And then we verify the change with QA and and then that's when we actually allow the change to be to come into the stream at all. Right? So we do a lot more work before the work is actually started, a lot more work before it uh validation before it actually gets
- 00:21:03
submitted. And so um uh on those teams like developers working on those hero push squads have a much more uh methodology to apply uh to solve that because basically in the past we would the regression rate was was too high, >> right? And so the the big advantage of that setup is that those teams are faster to react at because they they're dedicated on one subject. And so if there is a regression because they still happen, they're faster to pick it up. But then the fixes are slower to put in, but you know that still is has been very effective. So uh transit is one of our hero push squads. Uh which to me has been kind of successful since the beginning of the year. like uh the I think the transit system we're going to talk a bit bit more about it later you know it's in a much better shape the
- 00:21:51
embedded green light is what you just talked about which is that was a major organizational change and so in the past the way that uh game devs at Cig on Star Citizen would work is that they would submit and tests later I know that sound crazy but it is what it is and it was what it was and so uh at the beginning of the year we set a completely changed this and so we we added this embedded green light process. So what is an embedded? So embedded are QA testers that are embedded in a dev team. So it's not you know everybody has a view of like there's a floor of QA develop QA people and then they receive a build and they test the build and stuff like that right but there's another class of of of testers that are embedded in teams and they become
- 00:22:38
specialists and what that team is doing right so you know if you have an embedded tester in a in the FPS weapon team that those people know all about FPS weapons how to test them what change effect what and so they can judge what they have to So the embedded green light is we gave that team the power to red light any change before it's submitted. And so that's a a big culture shift because it used to be hot and fast. Get it in, we'll test and fix later, right? If there's issues and we trust devs to we don't trust devs, we need to verify, >> we need to check it, right? And the whole point of this was to hit on uh regression or side effects, right? because our rate of regression and side effects was extremely high. Right.
- 00:23:26
>> I want to just I'm picturing the the comments. Somebody can screenshot Benwa and say Benwa by 2025. We don't trust devs. Go ahead. Right there. Do it right there. >> Right there. >> Um no, I mean we we we've had embedded QA for for years. This was about empowering them, you know, in in order to to be a bull work against letting letting this stuff through. >> Um Another new term I heard this year, uh, uh, hygiene initiatives. Now, when I heard this was coming up, I thought it was something else. >> You know, anybody anybody that's worked in a game studio knows that, you know, every once in a while there's that desk you walk by and you're like, I thought I thought I knew what this was about. It wasn't what I thought it was about. What are hygiene initiatives? >> So, this is not about BO if that's what
- 00:24:15
you're referring to. Uh but hygiene is a collection of initiatives related to cleaning up errors, logs, uh and in general just reports we get from the game engine that can signify errors or not. So if anybody's ever opened a console in Star Citizen, you probably whoa, what the hell's going on, right? Because there's tons of messages going through all kinds of stuff. Well, that's what is meant to clean up in a way, right? Uh now these things will take us a long time to to clean up algorith but we do we do have the same problem of signal to noise ratio in other areas. So like when when environment artists are working in the in the editor, uh they get level warnings, right? Well, there's a ton of those warnings that are either
- 00:25:03
irrelevant >> asserts. Well, yeah, asserts, warnings, right? And errors. And so the issue is uh level artist is going to get some warnings like, can I do anything about this? Right? Oh, there's 50,000 of them. Probably I don't I don't have to worry about this, right? The reality is one of those is actually an issue, right? And so the hygiene initiatives is like a collection of of things that we need to change and how we build the game uh to clean up these things so that the errors are meaningful. Uh we have the same uh thing on the data core side. So for those who don't know the the game data so all the records we use to configure you know uh the components or all the game data right is stored in a database called data core. And uh there's a ton of errors in data core that are reported schema violation, you know, things like that that they're not fatal errors, but
- 00:25:53
they're still there and we allow them to be there. And that's is historical in the past. And so the hygiene initiative is a is a means for us to reach zero errors in those areas so that we can then prevent them from even being submitted in the first place. So hygiene is basically presubmit verification on things and concerted efforts to solve these things so that we can be in a clean house. It's like the we want to clean the house, right? So that we can do better. >> Somebody in the chat was saying I missed it the name it said errors that should be warnings, warnings that should be infos. >> Yeah. Yeah. Very common. >> Bad classifications of things. Uh so we even have now uh systems put in place in the assert dialogues and uh that popup so that you can report a bad error. report a bad error. >> Literally now we get reports for bad errors so the devs can go in and fix the
- 00:26:40
wording >> or or change the severity or things like this. >> And there was something we did so that certain teams if the if the popup was directly related to them, they couldn't just dismiss it and continue with their work. >> Yeah. So the that's exactly right. So the I feel like an AI when I say that now you're absolutely right. Uh but the uh yeah, we changed the the way the SR dialogues work. asserts for those you who don't know is like when I'm a programmer and I'm asserting in an area of the code that this value must be set to one that means that if the value comes in and it's not one then we'll have an error right that will show up that's the assert error and typically you need to action asserts right and so or you know or something else is going to go wrong and so historically uh the assert dialogue was just a global thing and then we still have those get assigned to dev teams so
- 00:27:28
they they they are classified by code owners by dev team and so you can know if like it's an audio assert or if it's a you know programming assert or whatever which is great but we allowed people to ignore those either completely or just skip them right uh and on the other hand when you're actually testing the game or playing the game you get these asserts so you want to ignore them because you're doing something else you're not in in assert fixing mode and so what we we've changed the relationship towards assert inside the company so that now you can ignore error errors for any other team but yours. So if you get an assert error, you need to fix it. That's it, right? There's just no cuz the the thing that we were seeing in the company was like basically teams going like, "Oh, when are these I'm going to pick on audio, but >> sorry Graham and team, I'm not picking
- 00:28:15
on you guys. It's just random, but like when are the audio guys going to fix their right when as you know the reality is like we all have our own sensors to fix and so we change that relationship as well." >> Yeah. It's again over years and years you get this you get this mentality. So, it's like I I just need to do my work and the game is telling me there's an there's there's an error, there's an assert for some unrelated thing. Like, what I'm working on doesn't have anything to do with that. Doesn't have anything to do with that. So, you you you you smash it. You smash you smash. It's just like selecting all cookies because GDRP made the rule that now everybody's got to announce their cookies on your website. Now, I just accept all of them. All All you've done is you've made me hit a button. you get in this habit of hitting button and people and this is one of those things like people we're going to we're admitting this here guys you know tell where's open development I'm like well we're
- 00:29:02
sitting here saying that you know too many people got in the habit of clicking the button what it's >> it's >> we even found some some I'm not going to blame anybody but we've we've we've got analytics inside the company so we know who's putting asserts off right so we found some we found some [ __ ] >> a Paul Rendell I miss you just kidding That's certainly not fair. >> Paul's a good guy. He's a good guy. He's the one putting the [ __ ] in. >> I didn't know I didn't know he was here either. I ran into him today. He gave me a hug. I'm like, "Oh, I didn't know I was on hugging terms with Paul Randall." Nice. >> Um, all right. So, we've talked hero push squads. We've talked uh embedded green light. We've talked these hygiene initiatives uh that all help the way in which we make the game and make the patch and and make our content and
- 00:29:49
stuff. Um, we also started an initiative. I guess I want to keep this under organizational stuff, but tell me if I'm wrong. The stability score. >> Things. Talk to me about our internal stability scores and how we've determining them and how we're using them. >> Yeah. So, stability score is I don't think we've ever discussed this publicly. >> No, I don't think so. Uh so we we have this uh measure which is an indication that we use you know and the the the team that actually comes up with this would already tell you it's alchemy right like it's not an exact science but it does give us a pretty good idea so we try to establish a stability score for every build right we actually autocalculated based on the data that we get >> not patch every build >> every build yeah every build so like we send a build to PTU you know or QA or staging environments or whatever And we
- 00:30:38
always have a stability score that's calculated over time. And so what the stability score is without going I don't want to go into the alchemy of it. But we're trying to figure out basically the stability of the environment. And so if you're connected to a game server and you get disconnected, we're going to count that right now. The reason for your disconnection will be used as part of that. How long the build has been running is also part of that equation. And the concurrency of it is also part of it. And so with this blackmagic numbers, right, we basically get a score out of it. And the score is one thing. Now we've put labels on. So we used to use the score internally as just a number like, oh, it's a 0.0 it's a 008 build, you know, and then at some point I got a bit fed up with that because I'm like, what does that mean? And so the
- 00:31:26
values for this are optimal, excellent, stable, unstable, volatile, crippled, or critically unstable. So that's the range, right? So we go from optimal, which is great, uh, stability score to critically unstable, right? And all of the thing in between, right? Uh, and this is based on, you know, all the stats that we get. And we kind of autocalculated. Now, we still have Jeffrey Peace, our internal alchem alchemist, always, you know, >> playing around with these numbers or adjusting the score, telling us how bad we're doing it. But the it it is actually pretty accurate in terms of like what the perceived stability will be uh from a player's perspective. And so, uh if I compare what was September 11 last year, was it
- 00:32:14
like 323 maybe four at that point, right? 323 or four, I forget which one. Uh, I would say like the stability score for 323x was probably in the kind of unstable range, right? >> Okay, >> that that would be a normal range for the 3 3.2x uh you know uh this kind of thing, right? So the the stability score server score for uh 430 for example is between optimal and excellent. And so that means the the the rate of server crashes and disconnections overall is really good. Now the this score is not just server crashes, right? It's also client crashes or client disconnection, right? So if
- 00:33:02
everybody gets disconnected because one of our cloud routers dies, then the stability score of the build will be terrible even though the servers are up and they're running real fine, right? So it kind of encompasses everything. We're trying to get one number for all of it. >> What a what what what was the stability score for a 318? >> That was critically unstable. Yeah, I mean it's funny because the actually the server if you just took the server score for for 318 it wasn't that bad but the it's everything before that's why it was so critical in the 318 areas. It's because it was really a a massive backend you know problem at the time. Uh so yeah let's not talk about that. We're past that now. But uh >> the two things we don't talk about that damn holiday live stream
- 00:33:51
>> and alpha 318. >> That's right. At least alpha 318 had to happen. At least that had to happen in order to get everything that came after it. There was nothing about the holiday live stream that needed to happen. So >> you don't want the the makeup. Let's go back with the makeup. Okay. Um so yeah, so that's the stability score, right? So that's an interesting measure and I mean that proves that we've kind of really improved the the crash rate on game server. So the server errors that you get. Now there are we need to do more on the stability score because with static server meshing even the stability score is sometimes skewed. Uh we have run into area local uh crashes that would crash loop a server right non-stop and so that really contributes badly to the overall
- 00:34:39
score of the build but that's because it's one area. So uh you know we we're we're discussing how we can do more territory based uh scoring as well uh so that we can get a better idea. So basically what I'm saying is we can't get an actual just single number for all of the the stability stuff. There's always some factors that you know we've had AMD drivers or NVT drivers issues that affected stability scores some builds and so on. Yeah. >> So how do we use that to determine whether a build goes live? Like is there is there this it's it's I see these I I I see an aspect. We're just gonna let's just pull the curtain way way way back. Why not? >> Uh I I I I see the question >> is it worse than what's on live now? >> I see this question I see this question come up a lot in internal conversations
- 00:35:27
when let's just be real. If we want if we tr tried to wait for everything to always be optimal we release very few things. Correct. >> That's just >> that's the nature of shipping a game that's in development. >> I'm I'm not going to sit here try to hide or pretend otherwise. But so we know that not everything is going to be optimal. But where do we draw that line? How do we draw that line? And is it more than just is it worse than what's currently on live? >> No, we always try to shoot for better than live, right? uh when when we can. But there's a lot of things that we look at to be able to ship a patch, right? The now obviously for this year it was a very big change for the development teams because moving to a four-week
- 00:36:14
cycle means that we are always shipping. Like you know how we have a ship date, right, that we give ourselves and a schedule and then we've got our gate reviews and everything is like chained up to get there. Well, the closer you get to this this date, right, when you guys from from the players perspective, when you guys see like daily PTUs go go go go go and it's always update update update update, that's because we're getting close to the release, right? And so we're trying to get to the date and we're trying to validate all the things we need to validate as fast as possible. The focus is really on on shipping. So even minor issues become top priority, right? And so it's been really difficult for the teams this year to keep up with this because they we we finished a patch like yay okay we're late for the gate review like literally you know it's going like this and obviously we balance
- 00:37:02
this across multiple teams but there are teams that unfortunately are you know they're they're they're on everything and so they're always they're always on it. >> Yeah. I think it I think it gets lost sometimes. We we in of of the many things we changed this year, we've shipped eight or nine patches this year. Yeah. It depends on whether you count what happened in April as a patch or not, but it it's it's we've shipped at least eight patches this year. And in years past, you know, from from from the 3.0 era on, it was four a year. You know, if we were like some some one or two years, we only hit three and whatever. And now we've done eight this year. The the the the velocity change has been tremendous. uh tremendous. We've and it's helped, if I may, if I may, it's helped expose aspects of this. When I say this, I mean
- 00:37:52
making all of this. It's helped expose aspects of this that we're pretty good at and it's helped expose aspects of this that we still struggle with and still need to uh improve. Um and just what you were saying just just just really hit home with that. I we're we're in the progress we're in the process now of talk discussing adjusting the patch hands for 2026. It's way too early to to give any details on that. Nothing's decided. >> But but yes, this has been a pretty relentless pace and there's a patch coming out in September. There'll be a patch coming out in October. There'll be a patch coming out in November. And then there'll be a patch coming out in December. So it's it's a it's certainly a year unlike any other. >> Yeah. Now, one of the things we didn't discuss about for this in order to reach the stabilities, there's also internal
- 00:38:40
stability, right? That comes into play with this. >> I'm not familiar with internal stability. I've never had any myself. >> Well, so this one is a bit more subjective because we don't specifically have a score for it. But if I go back to last year, right, uh the way we would develop the game, and I don't want to go too deep in the discussion about source control and streams and all that stuff cuz that's just a rabbit hole, but in general when we work, we have a mainstream where we work in, right? In the past, this was a melting pot of everything. So core technology, editor, toolings, audio teams, content teams, gameplay teams, art, everything, right, in one stream, right? And um the fallout from that is that you go a bit faster
- 00:39:28
because you're building things, you're improving the tools, as you're improving the gameplay, as you're improving everything. But the reality is we also had major internal stability issues, which is imagine that you're an environment artist. You show up at work, you make your little coffee, you pull the the morning build. You're like, "All right, I'm ready to work on my on my ASD mission today. It's going to be super cool. Editor crash, right?" And now, oh yeah, yeah, yeah. The team's already on it. We're fixing it. But you just lost two hours. >> Yep. >> And so we had major issues with this, right? And so when we started the year last year when actually when we did the show just after the show I think we went and finished this new uh branching strategy where uh all of the patches this year have come out of the same stream uh that we call SC patch and this patch receives almost no technology
- 00:40:17
update. Yeah, >> it's basically us saying we can do this content with no new engine tech, no no editor update, no tooling update, nothing. Right now, it's not to say we are not touching the engine. We've we've fixed bugs, we've improved stability, like we but we bring these things embedded green light, full verification, tooling check before they go in. And so it's really about moving all these checks beforehand so that we don't destabilize the content teams who are on the they're under the gun all the time to produce the the stuff that they have to and we don't want to slow them down. And so that's another reason for why the stability is better. It's because we're also not introducing massively breaking changes in the engine that we then need to debug at patch time, right? And so that's been a big a big helper on stability at the cost of features, right? That was the whole point of the
- 00:41:05
year really so far. And now we're changing some of that with tech preview. I I want to hit one more thing before we go into tech preview and stuff. You know, features are going to begin. Yes, Virginia is true. Features are going to begin appearing uh uh again uh uh in I won't say the word soon. Uh but they're coming. Uh before before I get to tech preview because that was a really good transition into tech preview, but I want to do touch on one more thing. You mentioned the rise in session link. >> Yep. So, this is this is one of those metrics that we've talked a lot about. We're we're 40 minutes in. We've mentioned a lot of things that are probably never been said on a single development uh show before or podcast or whatever the heck you want to call this. Um, we don't often talk about session length, the the the am the
- 00:41:55
the the the amount of people who are logging in. You hear about concurrency. Uh, Steam has concurrency numbers and stuff, but but session length is a super important metric for how for stability and for uh enjoyment of the product and stuff like this. Talk to me about uh session length, how it's determined other than just hey, that's how long the person's in there. Why it's important and uh what's happened to that over the last nine months? >> Well, so you know, let's say in a the session length is calculated by you joining a game server. >> Hold on. I'm sorry. Spicy SEO7 says 40 minutes in said virtually nothing. >> Okay. >> Yeah, I stopped the whole show. >> Sure.
- 00:42:41
>> Stopped the whole show. >> Continue. Ben, >> I was saying the session length is is calculated when you connect a game shard, right? So sitting on the main menu doesn't count, right? It's really when you're actually actively connected uh to a game server. And obviously it ends when you disconnect. So that sounds like a simple metric. Uh and it it kind of is, but it's it's very important because it gives you an idea of like you can have very high concurrency and a very short session length. And that means that's a terrible patch. It means people are going in they let let's let's go let's go let's talk about the patch we should not talk about when you would get in game if you if you could get in game and then can't get out of bed out at four end of session you know super high concurrency but very
- 00:43:31
low session length and so the session length is a bit of an indicator of the quality of the of the game but also of the desire of the player to keep playing the game and so the longer the session length Uh it typically means that hey the content is being engaged with players are playing the game blah blah blah. Now I'm sure you know some people go out in deep space and they mine and that's it and they can mine for hours and they pull the average up. Other players you know go PvP for a couple minutes you know and then they they they exit and then you know they drive the average down. But overall the session length is like a it's always a metric that we look at as a secondary metric to look at the quality of of this. And so the the session length has more than doubled uh since the 3.x era uh of the game easily easily right and obviously in the case
- 00:44:21
of server recovery your session continues and so that helps you know quite a bit because before it would cut the session in half so there's other factors that contribute to it but that's why it's an important one. Okay so now I want to talk about tech preview. So we we we used tech preview when we launched 4.0 0 uh technically 4.0 didn't launch until Jan end of January this year. That's when it went to live. It was on tech preview at the end of last year. And we we made these statements that we were going to use tech preview in order to bring these new things in and maintain the stability of live. Uh it took eight months. I I correct me if I'm wrong. I don't think we did a tech preview between January and and and late late August, early September. and now we're we're using them for the the the
- 00:45:09
Vulcan stuff and the and the graphics stuff. Talk to me about talk to me about Tech Preview, why we're doing it, how it's working. Uh uh uh and uh let's just start there. I'll follow up when when you're done. >> So, I mean, let's start with the WID and half techch previews in let's say April of last year, right? We started this this idea of having tech previews last year, right? So, it's a new kind of tool that we have. Um, and obviously we use tech previews in the server meshing leadup, right? To validate key assumptions to actually isolate the testing. If you remember when we did the the major technical previews on the 3x branch for server meshing, it was taking the actual previous build with tremor meshing technology. So uninfluenced by
- 00:45:57
new code, uninfluenced by the current patch, something we know is kind of in a stable range that we can use to test the technology alone, right? Uh it's also a way to derisk just again bringing that in the stream and then going Cortez maneuver. Now we're stuck with it. We're moving forward. But you know uh and so the tech preview has has been invaluable for that and uh and especially for the server meshing project that's been a super important tool because we just don't have always the ability to replicate what players are going to do. And to be honest, even with bots and virtual, you know, players and whatever, it's never the same. and you know 4,000 players filling ships with hot dogs, you know, >> and right now we're using it for graphics stuff. How's that going? >> Yeah. So, uh this uh tech preview that we're bringing in is more than that
- 00:46:46
actually like behind the scenes, right? The there's uh this is literally I told you about SC patch. So, all the patches this year have been coming from SC patch. The tech preview contains all changes that we've brought to our mainstream. This is where the engine development happens. This is also where we bring back some technology from Squadron 42 back into Star Citizen and uh we are moving all of this together up to SC patch. So this will now become when we're happy with all the testing that we we got through on this and it checks all the green lights. This will become the new baseline at from which we build patches. And so uh right now this includes yeah about eight to nine months of graphics changes right which have not seen the public yet. And so a lot of Vulcan changes obviously uh to the great
- 00:47:35
pleasure of our great Ali Brown who uh you know really wants us to finalize this merge so that we can bring this in to players and uh there's some great stuff in there. So, a lot of Vulcan optimization, Vulcan changes, but also um in a way a way for us to test all of the these changes that we've been doing to the core engine rendering outside of the lab because we have a graphics lab internally that we test on, but it never again there's never anything like shipping it to players and having all the different kinds of hardwares and configs and settings and driver versions that we need. And so the majority of the stuff in this tech preview is playerf facing, but there's a ton of back-end tooling that comes with that. A major update to the editor with some new uh, you know, material edition systems and light group systems that are, you know,
- 00:48:23
in dire need of coming in and a major upgrades to our scripting system that power missions. Uh, which the the mission teams are, >> you know, Elliot and his team are like, when is it going to be for that? Yeah. No. >> Yes. that that that's I'm glad you brought that up because I had actually forgotten that that underneath that it looks like it's just for the graphics everything but yes no the the mission we call it mission 2.0 0 we call it a couple different things internally but they are chomping at the you already said that I'm using I'm reusing the same sorry I saw I saw I I saw a comment from Salty and Mike that said talk about session girth and I just I laughed and wasn't listening to anything you were saying for like 30 seconds >> okay >> it's it's a good joke >> it's a good joke it was a it was a good joke >> now to be clear on the mission tooling there right this is part of mission 2.0 it's the a prerequisite but it's it's
- 00:49:12
not the mission 2.0 system. That's a a much larger endeavor that we're still on. But the this time it's tooling, right? So if if if you know some players are techsavvy here, uh it's basically a visual scripting system that we're upgrading massively. Uh which was I think we've demoed this years ago at this point, but it's called the subsumption at the time and it's now called star script. And so the the star script system is a is basically taking, you know, the tooling that the the teams are using now and and and making it much more reliable, robust with syntax checking, error checking, validation, auto layout, stuff like that that the teams really need uh to uh build uh more quality missions. >> I'm sorry, let me get this straight. You you you took a new feature set and you you you took the base word for it and then added star in front of it and
- 00:50:01
called it that. No, we renamed Subsumption, which was the original name, and then we put star in front of it. Yeah, exactly. That's right. Hey, we have a theme. All right, we're going with it. I'm rolling with it. >> As long as as long as you're willing to get called out by me every time you do it. Um, all right. Uh, >> this tech preview is obviously important. Uh it it it's it'll be in tech preview for as long as it needs to be in order to uh to to shore up and then be integrated and become the new base from which we create our patches from what what is the next tech preview? Go ahead. Go ahead. Say >> well. So what's behind this right? This enables us to bring in I'm going to go back to streaming just to build the
- 00:50:49
excitement. No, I'm joking. the so off of our main technology stream is another feature that's been built there that needs all of this to work and that's the engineering gameplay uh tech preview that's right behind it. So the moment we can bring in the base tech preview changes then we'll be able to do uh to run the next testing on engineering gameplay uh straight off the of that patch. So Z is going to be super interesting. >> Yeah. So I want to be real clear with that. This this is the next set of tech previews. This is this is I want to go ahead and say and I'm I'm going to go ahead and say it so that if >> people can get angry at me and not you. Uh right now we're using tech preview to test Vulcan and these and and and and all these graphical stuff and all these mission changes and everything underneath and stuff like this. When this is done and get fully integrated, go to live and whatnot. The next thing
- 00:51:36
is engineering. We >> desperately I'm going to use the word desperately. desperately want this tech preview with engineering out before the end of the year. Now, this is the middle of September. It's still October, November, December. That's still a lot of time. It's not anytime, but anybody would say soon, but and that's no guarantee that we're even going to hit it. It's We've all been here. Somebody was joking, hey, drink it next time Jared says how long he's been here. Well, we've all been here, lots of us. I I recognize I've recognized some of these names for the whole time I've been here. We've all been here long enough to know that me saying it's our intention to get this before the end of the year is not the same as saying it will come before the end of the year. Just being honest with you here. But it is it is everybody's desperate intent in intention to get this thing into tech preview at the end of the year. We want it. We need it. We're tired. I did we
- 00:52:24
did five six stories in engineering. I did like two or three Citizen Cons with engineering. Like we it's time 300. Kick that mother out onto tech preview please. Uh, so we're trying. Uh, but I just want to make sure that everybody's super clear on what that means. >> Hey, but well on this, I got a little segue for you. >> All right. >> So, there's one thing that uh kind of proves the uh improvements, right, on the live service is that there's a lot less players on PTU and there's a lot less players on tech previews than uh before, which is a good thing, right? That means players want to play the real game. And I I'm really happy about that. But it does mean uh if you want to help, join OC previews, right? the more the more people we get, even if it's for a short period of time, it's really helpful. Uh because we get more a more larger uh set of people that can try it
- 00:53:14
out and that gives us more data. >> Yeah. I'm going to I want to talk about something. I got to show you you you you just you sparked something that's actually a bugaboo for me and I got a platform and I'm going to use it. >> So, we we did this change. We did this change this year. We went we went to what will be 10 or 11 patches throughout throughout the entire year. this almost monthly cadence of of live stuff. So, you've got the live stuff, a new live thing going every month. People like, you know, makes put something new in the game. We've had a thing new every month. We've had a we've had a PTU every month that runs along al along alongside it. Now, we've got tech preview. This pace that we're at this year has forced us to split our audience and stuff like this. It's like, well, do you want to be playing the live game or do you want
- 00:54:01
rather would you rather be playing the the PTU version or would or now with tech preview, would you rather be playing tech preview? And the reality is we need everybody everywhere. We we the the live version is the version of the game that we're making. That's the version we want. We want everybody to experience it, but we also need them for testing. So, we want them to be in PTU. We also need them for the specialized testing. So, we want them to be in in tech preview. But it seems, and this is one of the reasons why I talked earlier that we're re-evaluating the 2026 schedule, release schedule for this reason. It's we've created a situation where we want our players in too many places at once if I'm allowed to speak honestly, and I'll find out when I check my team's messages afterwards. Um, well, it's it's it's a thing that needs
- 00:54:47
addressing. So, and it's it's hard. It's it's it's hard. Um, I don't know short of short of altering the schedule which we're talking about, what more what more can we do to encourage people to participate in PTU and and tech preview other than us sitting here on the show and begging them. Is there something? >> Yeah, we've got we've got some ideas to be honest. like we're toying with a few ideas like incentivization programs or things like that but this is slippery slope you know >> it's a very slippery slope and I I for me I I kind of like the you know the kind of volunteer approach for this but you're right like it we do have an issue where >> you know the the we're testing this and we have a you know the next patch is
- 00:55:35
already cooking and so it's like it's always rolling. I mean, at the same time, it's a testament to our ops team that I want to salute one more time because these guys are always on the >> they're always on the clock. You know, there's always a build going out and so these guys are kicking ass. But, uh, >> it's hard because we don't we certainly don't want everybody playing nothing but PTU. You know, you second a patch goes out there, the next week there's another PTU. And it's like, oh, only play the worst, most broken version of Star Citizen. But we need your help testing it. But we it's it's this it's this weird catch 22 that I don't know that we have a a real great answer for other than adjusting the schedule which we're looking at um uh ne next year. Um and then it gets exasperated when we do uh uh timed events like like like Race for Stanton or whatnot when it's like okay
- 00:56:23
well if you want these rewards that are only available during this period of time then you have to be in live in order to get them. But then it's like well we got a PTU over here. wouldn't you like to be in a PTU over here? It's like, well, no, I want to get the rewards that are over there. And, you know, we we we extended Race for Stanton and that helped with that a little bit, but it's just that there is no there is no obvious answer to this. I'm I'm just I'm pontificating on one of the interesting challenges of our development model uh that that exists that I don't think anybody really ever talks about publicly. And today, we decided we were going to talk about a whole bunch of stuff. So, >> um >> still we're doing it live. We're doing it live. Uh it's one of those things we still need to figure out and address. Uh Bionuks had a question. Hey, Bioni. Talk about one of those people that's been
- 00:57:09
here as long as I have. Um it was a while back. I saved it. Bio. Uh why not have a set time for the servers to in the back end to just reset every day? >> Why not just flip the switch every day? Turn it off and turn it on again every day. >> Well, wait. That's one thing I forgot to put on the list of things we've changed this year, right? which is that we take maintenance windows. We used to never do that. Uh now why not do this every day, right? The issue that you get is that it's not just resetting the server. So you know rebooting the servers, reloading the game shard from persistence and continuing. You're going to get right back in the same uh situation. So let's imagine we're going to talk about cleanup
- 00:57:55
density a bit later, but let's go back to the 3.x X days where we persist all the data in the shard and we have deep intersections. So a ship entered another ship, it's an intersection is costing on cost on the server. The server FPS is now [ __ ] Uh super low physics threads, you know, using all the resources. Uh and then you're like, okay, cool. Let's do a daily reset, right? It's going to fix that, right? So you you go through it, you reboot it. Cool. Now server reloads from persistence and those ships are still intersecting and so you're right back where you were. So if you you know you would look at the graphs and you see FPS super low zero during the reset 30 five you know instantly and so uh you know what we've done in in in the past
- 00:58:44
for this is now we have way more systems to deal with like this to reclaim um to reclaim performance right for these cases. So we first of all we detect them. Um but also uh we do emit fresh shards uh periodically. So we kind of decide when it happens. Uh but uh you know when we do massive uh maintenances, weekly maintenances, we don't take every week but we do it when we have to make configuration changes or things like that. We'll take a downtime and we're probably going to emit new shards and or run the repair on some of those shards and then that will help. So we kind of do that. just don't do it systematically because the reality is that's not completely how it works uh for the for the backend persistence side of this and it's it's not something that we can solve easily because the conditions are
- 00:59:32
too kind of too great. You need the chart to be live to solve these things >> to be able to do it. >> What about a what about a traditional like every Tuesday scheduled maintenance period like like every every Tuesday at at at 4:00 a.m. Eastern time whatever things go down for two hours and come back. We're we're going towards that now like uh where we basically right now you can almost consider that every it's uh every Tuesday we try to do a maintenance typically early in the morning UK time is the is a good time in general uh just because of that's where the least players are playing typically but um that said the uh we don't take it every week because we don't always have to do it and so sometimes it's not necessary but we're going I I'm an advocate to always take it so that sometimes We use it to accelerate hot fixes. So, in case
- 01:00:20
you haven't noticed, we do hot fixes all the time, right? So, we have this new deployment systems now and we're able to roll hot fixes way faster. But the when we roll a game server hot fix, um it's going to take time to what's the word that Ahmed uses? Percolate. Percate. It's going to take a while before it percolates. Right. >> Percolate. >> Yeah. And so sometimes you can have a game shard that has, you know, 10 servers. Two of them have. fix. The others don't have it. So, depending on where you're at in the shard, you you have the fix or not. And it takes a while before all the game servers actually uh run into it. Now, obviously, if it's if it's an unstable build, it's going to be much faster because then server crash back new instance with the new build. Uh but these maintenance windows give us the opportunity to also synchronize that with a client fix when it's necessary. And so, we can take
- 01:01:08
everything down, patch the client, patch the all the game servers in one go, emit fresh shards if we have to, and then start again. And so that's why typically there's big changes that can come about in this in in maintenances and we want to use them more. Yeah, definitely. >> Right. Well, uh, we've completed our first hour and >> that was part one organizational changes. >> That was part one, organizational changes. Uh, let's take an interlude here. I'll ask you just a side question here. Let you let you get a drink or whatnot. >> Um, with our last patch, uh, with um, what was the last patch? the August patch uh 43 43. Uh >> there was a big curiosity with the wolf >> uh on on on PTU right right before we crossed the finish line uh to release that uh we discovered something that
- 01:01:55
didn't show up in any testing and whatnot. Uh walk me through what happened with the wolf. >> This is a beautiful one. That was fun. So yeah, like you mentioned, we're we're walking up to 43. We're at the point where, you know, pretty much everything has been, you know, validated and tested. And, you know, there's there's something with the wolf that, uh, you know, we're going to try to do more in the future is that we try to respect what we call the art of surprise. Is that typically beforehand, we used to put everything in a patch, go go. Uh, now for for certain things, we want to try to respect some level of surprise. We find that it's it's cooler for you guys when you discover things when it goes live and not necessarily on PTU two weeks before. Right. And so we're we're trying >> or shows every week. >> Orange shows every week. Exactly. Right.
- 01:02:43
So and and and so the the wolf is part of that. Uh and so the the wolf was going on PTU for validation because it was part of 43 release at the very last minute, right? So like I think 24 hours before just to all right, make sure everything's fine. Well, it turns out everything was not fine. And so, uh, we look at the at the reports. There was one crash that we had to solve, which was, uh, solve rapidly. But then we looked at the at some very curious reports. And so players would basically report, I forget the IC exactly, but the something like, I landed with my wolf, exited the ship, and then was walking around. My, then my friend took my wolf and flew away, and then the world despawned around me. So, initially it sounds like a fluke
- 01:03:30
report. >> Mhm. >> But then a bunch of people reproduced it and was like, I have this, I have this, I have this. And so, like, what is going on? And so, typically then that's where you our player relations team goes like something's fishy on here. So, we started to look into it. And um so we assembled a quick crew to check this out with and now this is kind of skeleton crew because we're we're moving too early to typically it's late very late here in the I'm in the UK right now. was very late here in the UK. US teams, it's night for them, you know. So, um, just get this crew on together, start looking at this issue. And so, basically, the issue that we're seeing is that sometimes, now when a bug starts with sometimes, you know, it's going to be a fun one. So, sometimes when you exit the wolf, you desync between the client and the
- 01:04:18
server. And what actually happened is if you looked at the so we have this tool we uh we call the server renderer which allows us to basically view what the server sees, right? And so think of it like a a c a side camera that we have when we have access to the game server and then >> our our game plant and you can see what the server sees. >> And so you'd see my character get out of the wolf, you know, with that super nice animation exit and you know he jumps at the end. >> Jumps off the ship. This is like super cool. And then I'm walking around on my client. Everything's fine. Game server. I'm literally in EVA animation floating over the planet. And the thing is I can't exit this condition, right? No matter what I'll do on my client now, the server will always see me attached to the ship in EVA anim. So we're like, okay, what's going on
- 01:05:10
right now? We start looking at this. So, at this point, John Crew and I are, you know, losing our minds trying to get a proper reproduction. But then we realize, oh, they we put new animations for this ship. So, the the Wolf was one of the one of the new ships that have enter exit animations, right, that are brand new, like just captured, important the game, all that stuff. We don't do this all the time. And, you know, putting animations in the game is not just, oh, put the anime in. It's like, it's timing. There's events to link them. And there's linking them to the actor system, how they get triggered, you know, inserting them into the ship pipeline, stuff like that. And so we start debugging this. First step was, oh yeah, they probably forgot to put some of the seat events on it. So, we do a first patch. We try it out. Now, the QA guys come back like, no, still
- 01:05:58
happens. So, like, okay, they now I think the timing is wrong on these events. So, let's change that. No, I think we went through like seven or eight iterations of fixes to try to figure out what was actually happening about that exact animation and why the ship system was disconnecting it. We went into full debugging on the actor. So, the actor is your character and all the states can go through and it was we could just see the two differ, right? But it made no sense. Why is one working and the other working? So at this point we're kind of running out of ideas and so that's when I called upon I used a button I called the bug smasher himself Mark Aen on the show and so Mark came in and was as flabbergasted as we were about what the hell is causing this and
- 01:06:48
then so we went into a bit more debugging sessions uh way at this point we're deep engine side physics and collisions to try to figure out what's going And then what in the end Mark just goes within, you know, in the typical buck [ __ ] smasher fashion just goes into a what about gravity? And so we take a look at how the the ships have fig physics grids. These grids have sizes and mesh sizes and the animations is based on the geometry. And there was no gravity in this ship. 000, right? And so it's a basically it was a combination of the physics grid. The fact that the actor on the server as runs on a at a lesser v uh um resolution. So you know your client when you're in game is at full resolution.
- 01:07:36
The server doesn't run at full resolution when he's simulating your character. And so combination of this would make that the server would actually reach the EVA state. So now you're floating in EVA, right? But with no gravity and no other movement, it will never solve itself. And he's also overlapping over the grid, which is too big. And so that was a bit of a combination of like blunders from geometry to wrong physics grid to out of uh um no gravity. And it made the bug also not 100%. It wasn't like I exit the ship and it happens. it would h need to happen like over it took about 10 to 15 times you would get it one time and so all right so we fixed the we fixed the bug we we get this in there internal Q validates it and then uh to be able to
- 01:08:25
properly validated as you know we need players and so that's where we mobilized you guys on PTU to come and try this out and this was one of the best moments I've had testing in this game where roughly 100 people showed up with wolves at boondoggle on Hurston and then 200 people are entering and exiting the wolf, right? To see if they get detached from the animation. It was fantastic moment. And so, yeah, that was a fun that was a fun one to troubleshoot. >> And and that was and that was your free uh audio only episode of Bug Smashers as a nice little interlude there. Uh I want to shout out Dennis Daniel who's in chat right now. >> What's up, Dennis? >> Hi, Dennis. Miss you, man. >> My man. >> All right, part one here of playability. Our little Bug Smashers interlude. Uh folks watching, we've got another hour or two
- 01:09:15
left, so we're going to be here for a bit. In case you're like, "Why would they talk about that?" Whatever. We're getting there. We're getting there. Uh part two, server meshing. Server meshing, past, present, future, static, dynamic, 3D. I'm just going to go get a get a get a cocon. I'll be back. Talk to me about server mission, man. Ah well you know so obviously we launched server meshing end of last year um and we've been running a server mesh environment for now seven months eight months right nine months we're n month n already uh and so that's the static server mesh and so that means we have territories just a quick recap for everybody and so that means that we have
- 01:10:02
a configuration for these territories and we can pick uh different top level object containers from the tree of the universe and and we basically distribute them on different uh you know resource servers. So at the moment we run at a 10 game server per shard setup and so that means that we basically distribute the key um areas you know on these 10 game servers and uh that that's how when you transition from one zone to another it's seamless but you're moving from one server to the other. So, we've been kind of taking the year to uh get used to being a server mesh environment. Uh, and so that means we've discovered a bunch of gameplay bugs related to this. I think we talked about some of those uh in January, but that have been solved, but the uh for example, the quantum drive systems, you know, didn't properly
- 01:10:51
handle the transition. Sometimes we'd lose power in transition and then that would cause the thing to reset, pull you out of granum, things like that. That's one example. But there's a there's a series of other issues that were related to that. Um but uh more importantly, we've been also learning about how to properly distribute the load of the simulation. So now we have this tool, we have this map. It's not the end of it. Right now, every patch this year, uh we've changed the configuration. I don't think we've had a single static configuration since the beginning of the year. Every patch has been different. And so uh obviously we've tried to make sure that we dedicate uh resources to where there's an event right so if it's SD location or hat locations or stormageddon locations we've kind of reconfigured the mesh at each release uh to basically guarantee some server power
- 01:11:39
at each places because we know that people congregate in these areas. We've also learned a couple of things like because uh you know the teams that make content don't necessarily always talk to the team that make the tech right and so how it's you know it's a you know so the where you put those locations or how many there are is important right uh and so uh that we kind of learned that through the hot moments where like they were a bit too close and we couldn't distribute them properly uh because of other limitations which I'll get into in a bit but and so we've change that config every time. And so it's been it's been really a good learning experience for the team. We've also solved a bunch of issues in the static mesh uh related to performance. Uh and so we've talked about this before, but the the hybrid, we talked about stability of game
- 01:12:28
servers, right, earlier, but the hybrid is at is at the center of this. This is the networking component that you're connected to when you're playing the game. He's doing the replication, he's doing the binding and all that stuff, right? And for this guy, the stability score is zero. Meaning if there's one crash, we don't ship the patch, right? That's it. Right. It that thing needs to run perfectly at all times. >> Yes. >> Um and and the the kind of quality metrics that we expect from the from the hybrid are are very very intense like the the the speed at which they transfer packets or accumulate packets and transfer it and between between uh areas and and elements has to be very very very fast at all times or else you get rubber banding. you get desync things like that. Uh and so there's been a lot of performance improvements behind the scene. Uh especially I think just before
- 01:13:17
Stormageddon or is it after? At this point it's a bit of a blur but I know that we found a couple of key optimizations that have really helped the hybrid maintain uh stability across some of the big binds that we were seeing. Again binding is the is this process that uh when your client act you know needs to bind to an area it means he's receiving updates for an area. this whole binding up uh uh uh process is is a bit uh costly on the hybrid especially when there's big areas being bound and so uh optimizations have gone into that uh and that has helped desync quite a bit. I mean we're still there's always room there to to improve for improvement and the team are still actively working on this. It's never going to be the end of it. Um but uh it's a it's much improved since the uh release of 4.0 you know, like the the I hope the rubber
- 01:14:06
banding is getting a lot better every patch. Like we're hoping to, you know, always cut some some of those big bottlenecks across as we go. >> Um, so that's static server mesh. We also learned a couple things on how to distribute those locations. So the cost of a location, right, and the amount of players like is not necessarily the same weight. >> It funny enough, a lot of players in the same place. Let's say you have an empty room, right, with nothing in it >> and I can pack 300 players there, no problem. Server is good, right? No problem. Uh, if I add AI in there, we start getting into issues, right? If I add ships in there, bigger issues. So that if you want the cost of the location and multiplied not necessarily multiplied but factored in by the number
- 01:14:54
of players there is what kind of is the cost of an area and how we should distribute the how should we should distribute the computing power that you know we pay for to run the game. And so we've been learning a lot about this in in in all these experiments we've been doing and that's informing dynamic server measuring right meaning how what the algorithm needs to be in order to simulation to the different uh components in the game that you know can do the replication. >> This weird this this weird equation this is in all the the factors of all right obviously how many players are there? Sure. uh uh what is the resource footprint of the location itself obviously how many ships are present because you know it it's not all ships are created equal the our process of
- 01:15:42
making ships have changed over the years some ships are more efficient than others so it's okay what what's what's the what's the resource footprint of the ships in the area what's the resource footprint of the location itself what's the resource footprint of the people in the area and then how where's the next closest area like cuz you know with with onx facilities. We we distributed a hundred of them and went up. But but like you mentioned with um a line in mine, our uh one of our big learnings, we'll we'll say learnings. It's a it's a nicer word than mistake was positioning was where we put them and whatever. We weren't able to separate them into into the shards that we wanted to. So being at one and being at the other one just doubled up doubled up the uh the performance hit. Uh, somebody in chat said, "It seems like it seems like uh
- 01:16:32
everything CI does is a test." Yes, I think that was Fly Mo Ninja if I remember. Fly Mo ninja. Hey, man. Yes. Everything we do is a test. We're We're We're learning. Don't There's a reason I used to end these shows with what did we learn this week. Don't Don't ever don't ever think that no aspect of game development isn't intricately tied up with the process of learning. Uh, not for us, for you, for everybody. It's like this. So yeah, we are we are learning as we do this and that's what's been talking about it. So dynamic server messing tomorrow, next week, what you got? So no well I'm not going to advance the date but I can tell you that the comp there's multiple things that compose dynamic server meshing right obviously the team is working on on those aspects right now right the key aspects that dynamic server meshing will
- 01:17:19
bring is um so a kind of arbitrary subdivision right meaning that we could assign a server to anything that's a streaming group so uh inside this big tree right you've seen the pees discussions where we show how kind of It's the data is laid out and it's a one big giant scene tree. Any of those uh nodes that are uh deemed a streaming group can actually have a server actually having authority over that entire area. And so DSM will definitely bring us the ability to kind of arbitrarily subdivide where we're no longer tied to these territories or this territory map. That's the first big thing we're going to uh we're going to gain from dynamic server meshing and we're working on that part at the moment, right? which is basically how we can um move these streaming groups from one server to the other. It's already
- 01:18:07
transferred, who's managing it, what component is taking care of that. Uh so that's under development right now. There's also a lot of simil I want to say parallels or similarities uh between dynamic server meshing and instancing, right? And so uh we've talked about that we're working on instancing. That's also part of this work, right? And so, uh, we technically already have the mechanics of instancing in the game in the form of the, um, instance hangers, right? Which basically allows you to have multiple stacked uh, areas on top of each other, but you get to see only one and only one of them is bound to your client and replicated to your client. uh historically if we go back in the 3x days when we put this in uh initially the the the instancing was actually not working from the networking side and so
- 01:18:55
you would replicate all the the hangers that were at that location even though you couldn't see it which was even worse. Now you actually get assigned the proper tag on your character and so uh when you actually take the elevator uh it's bringing you to an actual instance uh which is basically yours. So the instance tech that we're building on top of that will um will basically leverage some of the concepts we're putting into dynamic server meshing because we want to be able to distribute those instances across the mesh, right? And so uh right now they're location bound, right? But we want to not be able to just have them location bound but have any of the servers take those off uh so that we can share the load. again optimizing the distribution of the simulation uh of the computing workload that we have uh across what players are doing so that when you uh enter an instance we can
- 01:19:44
guarantee a certain level of of performance without needing to have a little you know dude at the back going like oh we need to change the territory map for this right basically and so we're actively working on all this stuff uh right now to bring us to the dynamic part of of server meshing and obviously we'll with that uh even though This is not our focus right now, right? Uh we will we are pl we are still planning to go to higher player counts but this is totally not our priority right now. Our priority is really on uh getting the functionality and the computing workload properly distributed automatically. Uh exactly. Um I'm glad you say we're not that's not our focus. Um, I talked earlier in the show about us
- 01:20:32
sometimes acting like we invented the wheel when it's one of those things that that that other people, you know, other games have had and stuff like this. Um, lots of games have some version of server meshing. >> Uh, you know, famously, what's the what's the other one? The the the Dune the Dune game whose name escapes me at the moment. You know, they they did a big thing on their version. >> Awakening. They they did a big thing on their version of of server meshing and stuff. what what how is ours different than what other things I'm going give you a hint 3D. How is ours different from what you're seeing in every other game? Just just cuz because I I see this come up. I I I I see you know I watch the the the I watch the YouTubes and the Twitches
- 01:21:20
sometimes uh and I see people like you know they keep acting like server meshing. you [ __ ] invented it and whatever. There's my one swear. Ah, it came. Sorry, Tyler. Um, why is our server meshing different? >> Um, well, so the You're right. Like it's not like uh server meshing isn't already like a thing, right? Like in in the past, like even you know, Optima Online had a server like transitions, right? >> Right. Now, they were not necessarily seamless, but you still had server transitions and gridding, right? which is a typical approach which is you take the the the world you split into a grid and then you have servers between them. The you know and then and then the rest of the different strategies come from how the servers communicate to each other right and how the state is distributed and how you transition from one to the
- 01:22:07
other and uh I think that the areas where our approach differs is that we've um we've extracted the simulation from the game and the replication of the game. So replication is the is how game server and clients stay in sync, right? And so when you're in an area and somebody opens the door, the door opens on your client and that's because it's been opened on the server and it's also because somebody else sent that had authority to send that message to open the door. Uh so this is all replication, right? Well, normally you have one game server and he does both replication and simulation, right? So we've taken the star engine, we've extracted this the the the networking from it. So the replication into the hybrid. Now we call him the hybrid now because it's it's a first stage of of server meshing, right?
- 01:22:54
But >> the the hybrid has all the networking components. He has the state in his memory. He holds all of the information uh for the shard and uh he he does the replication. So e you're connected as a client to the hybrid. So you're not connected to the different servers. You're actually on the the game server. You're actually that's simulating your area is actually you're not even on it. You're not even in active connection that server. You're active actively connected to the hybrid. And and the hybr the the game servers are actually clients with authority if you want, right? If we simplify. And so that's one area that where it differs. The second one is how we separate space, right? Right. And you alluded to it which is that our territory mappings or ser or our mappings to how the authority is is is managed is in 3D space. So it's not a
- 01:23:43
grid. Uh it's fully made in 3D space. And so that means that we can have uh it's 3D space and a tree, right? Which is the other bit. And so what what that means is that by taking the I'm going to try to make that simple. Uh let's say I'm on the yellow server, right? and suddenly the yellow server there's not enough players there and the location is a bit low cost then we might want to despawn that server. Well well we can transfer the authority from everything the yellow server is doing to its parent. there is a parent right whereas when uh you're doing griding that's a bit of a different model uh in the past I've seen solutions for meshing which include servers like adjacent servers talking to each other for example things like that the client actually disconnecting from one server connecting to the next right
- 01:24:31
which in all of these cases these were things that created side effects that we didn't want so in the past we did try these things like if I go way back when we started the server meshing projects that was the first approach was was like a bit more classical but we realized to properly offer for what we've shown in for example the meshing demo Paul did a couple years ago or what you're actually playing now on live you know to be able to have like bullets completely traverse >> right >> the transition and hit on the other side of it like we needed to have a solution that was native to all all how we were doing it the bet behind it was that the hybrid was going to be fast enough and so far it seems like it's working out though it is a lot of work to get it you know to keep it at that speed uh with all the needs that we have so that's the that's I would say was what's the differences in the approach and every game is going to have its own approach, right? Tailor made to its its gameplay,
- 01:25:19
right? So, it's in our case, you know, we're trying to make the biggest universe ever and so we need the most scalable uh way to approach it. Also, we've taken the most difficult route, I feel, but >> I think you need to I think you need to adopt CR hands a bit more. You were doing the thing and I kept like, no, little server inside a bigger server. I'm like, you you use your CR hands. It it help you know this this this server inside this server. So when this server disappears, the server above it can can get the authority. I'm like I'm just over here like >> use the CR hands. >> A [ __ ] Yeah, that's that was a missed opportunity. >> If only there was a time if ever there was a time for CR hands. >> Yeah, that's true. >> All right. any any any so so 3D server meshing uh and and and separating everything so connecting people to the hybrid instead
- 01:26:07
of instead of the server so that things can connect and carry through between from one server to another server seamlessly and stuff um anything else you want to say on server mission before we move on >> no I think that's pretty much the state of it like we're advancing on the dev for it uh obviously we don't you know we we don't want to bring in stabilized changes to the game right and so we're making sure that these are going to be tested first. We will go to a tech preview when we feel it's uh it's the right time. Right now it's not the right time and so uh we're still working on these aspects but that's our objective is to you know we're we're not going to do a big oh dynamic server meshing is up right that's not going to be how we're going to roll this out. We will most likely be bringing elements of meshing into play right like the ability to despawn servers in areas that are not being uh simulated. uh the ability to
- 01:26:56
scale up the mesh dynamically for from the territory map and then later on possibly go into you know uh then the instancing support that goes through it and then after that you know the final uh the final jewel of like dynamic resubdivision and reallocation. >> Sorry I didn't mean to laugh in the middle of what you're saying don't please in the chat goes I'm really glad I was here to see Disco's last show [Laughter] I've survived this long man. All right. part three. >> Big chickits. >> Oh, stretch. Stretch. Part three. What is my hair doing? Just looking fabulous from here. So weird. All right. Um, this phrase, this section I call uh big
- 01:27:46
ticket items. The the these are these are various issues that at some point in the year have generated some amount of angst, aggression, frustration, or any other word you can think of to describe Tyler Wicken. Um he's watching. Uh let's start with the let's let's start with the big one. Freight elevators. >> Yeah. Now, I already did a big five minute thing explaining Oh, there's a delay. He just popped up. Um Fre um where are we at now? Where are we at now? What's happening? What happened? What do you want to tell us about Freight Elevation? >> Let me pull up my notes. >> Okay. Uh so, Fred Elevators is one of our hero uh squads, right? Now, the you
- 01:28:39
know to the your previous interventions that came way too late in our planning, right? Right. And so that's one thing that, you know, I think that if we could go back in time, that squad would have started way way way earlier. And so I've got a bunch of, uh, >> to be I'm sorry. To be clear, work could start earlier. We hadn't had the hero squad on it. >> Correct. Exactly. >> Uh, good clarification. And so, uh, let me go through some of the stuff that we've been going through on freight elevators in general. So, um, to start with, uh, obstruction issues, right? And so a lot of the issues that we ran into were related to uh how the elevator actually detects obstructions and or uh contents. And so uh for example blocking the doorway
- 01:29:28
item being positions on the roofs of the planetary freight elevators. Now a lot of this by the way like I'm going to talk more about the planetary uh versions of it which are 10 times worse for some reason. Um but the in terms of obstructions like you could leave items on top of the elevator. Uh and then it was detected as an obstruction, right? And so these obstructions first of all the checks were super rough. Uh and so they would go there's something I'm not going to move which is what a bit like come on, you know? >> So somebody would put a a smold bottle on top of the freight elevator. >> Correct. >> And the whole damn thing would stop working. >> Exact. Well, yeah. Exactly. That would break the whole elevator until that was removed. Um, but >> incidentally med pen med pen. Oh my god, it was med pen so small. >> No. Yeah, but see that's even worse
- 01:30:16
because uh there's uh there's a problem with how the items are are set up, right? And so when we create a prop or we create an item, there's a bunch of things that go into it. You know, physics proxy, weapon proxy, you know, all kinds of things that go on top of it. One of them is is the bounds of the item, right? which is not there's multiple bounds in case you you thought it was just a sphere or like a shape or rectangle. There's there's more than one bounds. But long story short to say that a lot of items have incorrect bounds. And so what that does is that when you leave them on a flight elevator, they might clip into the elevator right itself. And so couple that with the bad uh obstruction checks uh that could cause some really crazy exploits like for example leaving I think a med pen on the on the freight elevator. it would
- 01:31:03
clip through the platform. You couldn't even see it. So, you'd be like, "There's nothing on it." And then it would still block it, right? Um I think that actually is not completely fixed. So, we we still have to the the squad still has to get to all of these items. And so, for the bounds issue, what we're doing is we're adding a lot more um a lot more checks on uh how the things are submitted again beforehand. So, we're we're revisiting all bounds for items, right? and how their setup is made to make sure that none of them can be in that situation. So, we're kind of doing a systemwide check on this uh in order to make sure that when we get into these situations, it you know, it's going to work the way we think it should. Um there's also in like this is all you'll see it's all
- 01:31:51
super interrelated, right? um items that would get destroyed but would no longer have geometry but would still be considered obstructions. And so that was a very egregious one I think that we saw very early which is there was a way for you to destroy an item. It geometry would disappear but it would leave uh the entity present. And so now that's a that elevator is gone. Like you know you need you need tools to see the the entity, right? I mean, at least with the med pen or something, people could zoom in and try to scan the pad, try to scan the thing and try to find it. People in chat people in chat are already like, "Oh, great. You've just told everybody how to do it." It's like this. It's It's We're also telling folks how to fix it. We're We're trying here. It's a It's a It's a delicate needle, the thread here. But
- 01:32:39
>> that that at least there was something people could do if they if they knew about it, which is why we're talking about it to try to find it, remove it. This next one though, the geo's gone. Yeah, that's it. This this one was the worst one, right? And so the the thing too with the with these the freight elevator bugs related here like some of them were set up of the actual freight elevator geometry, right? Actually that was a regression. So it was a bug that got fixed and was reintroduced uh which is the shape because we have multiple types of freight elevators, right? We have like the horizontal one. We've got ones that are uh vertical uh some with doors, some without. Uh and so the there was some of those best setup the planetary ones were especially egregious in that case. Uh and so there's still some bounce check to be
- 01:33:26
added but you know this is all you know when you got all these messages about you know obstructions and that was all related to this. Uh one of the other issues we had with with freight elevators uh was actually one of the first ones we had to fix uh which is that the so the the freight elevator is set up into two things. There's the kiosk and then there's the platform, right? And so the kiosk hosts the manager component. He's the one that's managing the stuff, right? And the platform is kind of slave to the manager. So there was an issue where depending on how an area would stream in or stream out, the platform or kiosk would stream in first or stream out last, right? And this caused these two components to now
- 01:34:16
be disconnected. So they would not find each other anymore. And so in those cases, that entire location was dead. Like that was it. That that location game over, right? Because they would never find each other. there was no model in place or process in place to go and try to fix that. And so this is you know in part of the what we do on the hero squads uh we've done the same with the transit system which is we introduce self-healing uh approach right and so in this case um you know the the system would actually retry to connect. So if a manager doesn't have its connection to its uh to its platform then it would get retrieded right over time until it actually connects. And so this way even in a case where we do run into an issue where they disconnect, they could still
- 01:35:03
reconnect. And so that solved the the issue that was actually rolled out in a hot fix. Um >> Sino says, "Are elevators CI kryptonite?" Elevators are the kryptonite of every game development team in the world. Elevators and doors. >> Yeah, doors and corners. >> Doors and elevators, man. >> Doors and corners, man. Um, let's continue on our list of bugs with freight elevators, right? the UI and platforms could get out of sync and so this is where the platform will have content. So there's a clearly a box on the on the thing and the UI is showing that it's empty, right? And so that prevents you from you know uh basically makes the whole thing inoperable because now you can't ship your your box at all. And so, um, in this case, I think that the the reconnection, the streaming
- 01:35:53
reconnection actually solves this by self-healing, but also the UI will query multiple times to be able to make sure it's always updated. This is a side effect of us trying to of the game teams trying to cache a bit more data than they should and so now it cached an empty elevator. So that's a weird state. so a bit of background on how the platform actually works, right? So the you've got your inventory system, right? Your warehouse, that's where your things are stored. The inventory system has that, right? So you can query for items in it. You get the list of what's in it. You can unsto items from it, which means bring them into the game world and physicalize them, or you can stow them into it. Uh, the freight elevator and the ship cargo platforms are actually using a an abstraction on top of that called the inventory container. And so
- 01:36:42
the inventory container can be open or closed, right? And so we had an issue where so what that means is that when the when the inventory container of the platform is closed that means that all of the items that are physically on it are actually stowed. It's opened it means that all of these the items that were in this inventory container are in the game world and being acted upon. So when you unsto things on your platform that's what happens. There was a weird case where in certain cases if there was a delay on the back end, right? So the backend systems for all these operations have have cues of operations that run and so they can handle the delay, right? But in this specific case, there was a a weird state where the what's actually unto the state of the container would be desynchronized. And so
- 01:37:30
>> we've added another repair flow for this that would actually detect the state of it and actually recynchronize it in the cases where that would happen. Uh this was because we didn't properly handle failures and so in the case where an unsto would fail the state would now be you know in a weird uh situation and so uh now we do the proper error handling which is uh another thing that we've changed in our processes for development is that error handling is is now part of code review. So we really need to make sure every single I mean that sounds like basic right but I mean I'm just saying the way it is right some cases were not handled and so you need to handle those failures >> you wanted an open development >> yeah that's it all warts and and issues right um and uh same thing for door states right
- 01:38:18
so whether the door is uh open or closed right after stream in and out or server crash right would sometimes clash and so the kiosk can think the door is closed but it actually is open and so we do the same now we can actually properly uh set the flow back properly so one of the lessons we've learned with uh freight elevators in general and all the issues I just went through and the reason why uh we ran into them is uh need way more multiplayer testing. I think that the the major key behind it is that you know we the the teams have spent time on the inhanger elevator right so we've we ran into some issues with with inh hanger elevator for sure and there's still issues there but the big difference with the resource drive event was that we were driving players to the freight elevators and there's
- 01:39:06
more than one player interacting with it right and so that's one area where we have improved our our our testing and our testing approaches so that they're properly early tested multiplayer and not just as a you know when we reach PTU or that in focus test now it's like part of the general testing for these open areas >> you you've mentioned something called self-healing >> oh wow what's self-healing >> well so self-healing is basically when we add functions to a system to detect that it's in a bad state and attempt to fix it right and so um this is not this is you could call it a cludge in some areas, right? Sometimes it's not the right thing. But I think that in this case for a lot of the systems that we have uh where we are working on a replacement for example and we're trying
- 01:39:55
to keep this one running uh self-healing will come in and basically rectify the situations, right? We have there the you know the the way our game is made with this our streaming system makes some trivial things in a normal game very complex and sometimes we run into these issues where the game teams can't really identify what's specifically causing an issue. they just know it happens, right? And so, uh, in the past, we used to wait for the key engineers that can actually decipher these conditions to come in, but these guys are obviously everybody's swamped with with work. And so, sometimes the delay was so long that you guys as players would have to deal with the condition for long periods of time. And so self-healing is basically an approach where we're we're telling the game developers, well, while you're
- 01:40:43
investigating the proper root cause for this, provide a self-healing alternative, right? Which keeps the system running properly, you know, self-repair. Basically, >> there's an interesting I don't know, maybe not maybe in maybe interesting is not the right word. There is a forever thing that happens uh in chat. It's happened during the during the course of this show. happened in every YouTube comment, everything. So, this idea that every developer can work on every single thing that that that if if this person is over here making a loading ramp for the Zeus, they could be they they could be better spending that time over here fixing network code and stuff. It's like, why are you working on that when you should be working on this? And the answer is almost always because that's what that person does,
- 01:41:31
>> right? they they that they can't do anything else. That's not what they're, you know, it's it's like it's like looking at a looking at a cook and asking them if they can fish. It's like it's sort of kind of related. It's gamedev, but it's not the same thing. Uh but even within the disciplines, that's a common thing. Even within the disciplines, even within engineers, there are still focuses and affinities and stuff. And there's still even within a field of engineers there there are still many things where and it's no slight against the person or whatnot. It's just this person has that specialty and that person has that specialty and that person has that specialy and you often come up to a thing where it's like well we can't make any progress on this until we get it in front of this person and that person has already got this other thing in front of them and that
- 01:42:18
other thing in front of them and stuff. So it's just it's another it's another way. It's not it's it's just it's it's it's another way I want to it's I don't know what I want to say what what I'm trying to say here. It's not it's I'm trying to illustrate that it's like anything like priorities and like anything else there are usually if you're asking the question why aren't they fixing this or why aren't they working that on that? The answer is almost never. It's because we don't want to. It's because it's in line. >> It's in line. It's just that simple. It's in line. >> Yeah, we do have a lot of hyper specialization too, right? And so that's the you're right like like even game programmers are not necessarily interchangeable for everything, right? That said, our approach by the way for our game teams is very multiddisipline,
- 01:43:08
very fluid, right? Which does enable a lot of this, right? Like a we we have engineers who are able to go from one system to another in in the game code, right? But when it and when you have to do cross discipline things that's where now we need you need a concerted effort and that's when it's get it gets queued up. What's the before we move off of freight elevators? What's the thing that's happening on in like you go to Wiccolo, you'll put all of your your recipe stuff on the thing. It goes okay. It takes it. It goes down. It completes the mission and then doesn't give you your thing back up. and now you're just out those items because the mission completed but it hasn't given you your reward and people have just wasted their time on a wickeloop. What is that? >> So I think that most of the of those issues are are a lot of those are still
- 01:43:57
being investigated actually right now right uh in order to really figure out what's happening. It is you know these systems are are intertwined right the reward system for example is part of the mission flow right whereas the freight elevator is just the vessel right to deliver this and so there's many reasons and timing what this could happen right and so I don't have a proper answer for this I didn't dive on this one specifically uh but I can tell you that I know the team is actively working on this one because it's it's a key I mean it's especially Especially for Wiklo rewards like internally we're inter Wiklo rewards is like the hey guys you know like come on let's get a priority on this one you know >> chat starting a campaign to send Wiccolo to Claire
- 01:44:47
>> um that'd be a nice Christmas gift >> so at the beginning of freight elers we're going from freight elers we'll move on to our next thing because there is a a kind of an obvious sounding solution to this when we're talking about putting med pens and smold bottles on things and and entities that where the physicality has disappeared but the the entity is still there and it seems like an obstruction cleanup. >> Entity cleanup seems like it'd be the obvious thing. Just crank that dial up to 500 and clear that [ __ ] off of there every every two minutes and it seems like it fix. Talk to me about entity cleanup. We're we're seeing ships left everywhere. We're seeing trash left everywhere. The the the Raptor's not here yet. What can you talk to us about? Sure. Entity clean. >> Well, so one thing we I think we touched on this a bit in January, right? That
- 01:45:36
the one of the things that we put into 4.0 was the beginning of um of some physics uh cleanup, right? Uh now since then, right since the beginning of the year, we've we've unified a lot of the cleanup systems, right? So uh uh namely we had an abandoned vehicle tracking system. We have the physics uh intersection cleanup system. We have the actual density manager. And so now all of this is brought has been kind of unified into one one platform. And so what we're able to do with this now is uh we have density classes. So depending on the type of item, we can define the density class which basically allows us to say no more than x in an in a volume of y for this kind of item. Right? And obviously there's way more parameters.
- 01:46:23
I'm just trying to keep it simple. So we can understand what it does. Uh we've recently added the ability for uh designers to basically change those density class basically vary these density classes by region. And so now they can say in this location make this super aggressive, make this less aggressive, right? And so the density manager part of this is really about just items in the game world that are dynamic that we want to make sure we keep at a manageable level, right? And so if you start putting water bottles in one place, at some point it'll start cleaning up because they're in the same volume and the density class of the water bottle will inform how many we want to keep and how uh we want to do that. There's also lifetime uh cleanup. So things we just want to we want those to live but we want them to disappear
- 01:47:10
after a certain amount of time. So we've added that functionality in. So that's a lot better. And then vehicles come into play. So, abandoned vehicles. Uh, we've been tuning this since the beginning of the year behind the scenes. This in the in the in the physics cleanup. So, the abandoned vehicles is basically when you leave a vehicle completely abandoned. So, no passengers, nobody's left on it. It's left on a planet. You leave it there already. It's put on a timer, right? And so, at some point, we'll have to clean this up, right? the uh if it gets interacted with, if passengers come into play on it, whatever that revives it, resets the timer, right? Uh I think if I remember right, that's set to something like two hours right now. Something like this for abandoned vehicles, regardless of class for now. So that is something we want to
- 01:47:59
add, which is basically uh change the the density class for larger ships so that maybe they go faster. You know, this is the kind of thing we need to keep tuning. Um uh and then so the abandoned vehicles will actually clean up those areas right now with the the new system we have now coming in, we'll be able to override these values per region. And so for planet side locations where we know there's going to be traffic, we can make that a lot more aggressive. Uh and the last part of the cleanup system is the uh the pink slip. Uh I call it the pink slip, which is hey, my ship turned pink and bam, it's gone. Right? which is basically when uh uh areas of uh of the game where your ship is entering into a deep intersection with something else, right? And this is true of all
- 01:48:48
areas, right? So if something enters deep intersection for a number of frames based on the size of the entities, size of the area, we will clean it up, right? We're going to first turn it pink, full pink, for a certain amount of time and then oof, it it disappears. The reason for that is because the servers are still very much physics bound. And there is no currently, I'm say currently, I'm not saying it's impossible, but currently there is no elegant way to kind of break those collisions automatically uh and cleanly. It typically creates these uh mastrom of of physics problems when you try to fix it. and now you've backed it out into another physics element that is now colliding. And so, you know, uh you end up into these massive unstable systems. And so, uh the the pink slip is there for the moment,
- 01:49:37
right? We're trying to improve this. Uh we've been tuning these values quite a bit since the beginning of the year. Every patch we adjust the density classes, the abandoned vehicle timers, and the the physics intersection costs a lot. I feel like uh we're seeing a lot less reports now. It's kind of uh you know, getting a lot better. I think we're getting into the sweet spot, but it's unfortunately not a magic, you know, just turn on density uh tracking, right? We already have density tracking. It's really about tuning the numbers so that it feels good, right? And that's the hardest part of this. >> Folks in chat are talking about, you know, just just make it salvage gameplay. I want to I want to preface that Ben W is not a game designer and what so you know those are game design decisions but but yeah I mean an obvious kind of thing would be you know have an abandoned ship generate
- 01:50:24
a salvage beacon you know you know you know ship to come and and grab and and and clean up and stuff but I think a big part of that is salvage needs a bit more attention too we need we need to make those materials worth more because it's not just enough to generate the missions we need to make the we need to make the the materials that you get from those missions more valuable and mean more and you know there are there's there are things coming around the corner that's not true there are things coming down the line let's just be down the line around the corner that's not accurate that's not remotely true things come down the line like you know crafting and whatnot they're going to need lots of materials from lots of different places and stuff like that so you know mining salvage you know lots of opportunities there to to link it with other gameplay systems and make it more make it more make people want to go out
- 01:51:14
salvage things. >> I mean, the amount of abandoned ship, for example, like let's say we did that, right? Tomorrow morning, we're like, "Okay, abandoned ship detected, salvage beacon goes up, there would be an overwhelmingly amount of of of salvage beacons because the amount of abandoned vehicles is extremely high, right? And so balancing those contracts after that becomes kind of impossible in this current setup. That's not to say we're not going to do that, right? But yeah, >> we need to put attention on salvage to be able to do that properly. >> Yeah. No, Jared's voice Jar's mic isn't going out. I'm was just forgetting where I needed to put my mouth in order to use I'm a professional something. All right. So, entity cleanup and freight elevators. Uh, another big bugaboo. Uh, this this one's a
- 01:52:04
relatively Okay, recent is not it's been exasperated recently. we'll say that >> um uh uh too many >> too many people too many no too many NPCs spawn management whatever the heck you want to call it uh personal pet people go people go to people go to an ASD or whatnot and it's like that you know they're they're trying to they're trying to whittle the numbers down and they get down to like 10 12 or 12 10 six and then it bump back up and stuff because things are responding too quickly. Uh what's going on? What can you tell us? This is this is one of the ones that completely infuriates me because it's it touches on many things. So, spawning in the game is all over, right? There's multiple different types of spawners. Uh some of them are like um item port spawners, for example. You
- 01:52:53
know, I hate using the bartender as an example, but uh apologies, but the, you know, bartender picks up a glass from an item port that's on the counter, and then we at some point going to respawn a glass on the on this so we can do the animation again or things like that, right? Same thing for like AA turrets. Um, in 4.0, we've had many, many, many, many issues related to over spawning from those spawners, and we fixed a lot of them. mainly conditions where designers will set those up as infinite spawn and no limits which is great right so that's one thing uh now since the we've been putting out content this year like the different locations all have like a population of AI NPCs right in the place uh and so it's been kind of we've had overspawn I think with every
- 01:53:41
content patch this year you know where every every patch there's a at least one report that comes up it's like infinite PC is at location. Is it meant to be this hard? Right? And I was like, all right, what's the cause? Now, this is one of the an example of like the symptom is the same. The cause has been different every time. Um, this is something I really want to get time to solve properly. Uh, but the replacement system uh is in the works, but it's not close to being there. But, so the way this works, like the I'll give you the most recent one, for example. I think it was in in 4.3 uh zero. The mission module is the system that actually activates the NPC population for a mission in a given area. Right? So when you enter the action area of a specific part of the
- 01:54:29
mission and the mission code is going to go and initialize subm modules and then one of those subm modules will say hey put put some AI in there and then it's going to activate the spawn closets in the area and that's going to start running. The issue is there was a a bug with the way the mission flow was working for these things that would cause these modules to doubly initialize in certain cases and when that started then it would actually do it again and again and again and so that's the latest source of these over responds. Um the the the proper fix for this is is is the population manager. This is something that our R&D team have been working on which is a replacement to all uh to both spawn the spawn closets are still going to be around but the systems that drives the spawn closets to make these AIs appear and spawn on top of like the um uh what's the word that they
- 01:55:21
use for this? the base population in an area like let's say walk around New Babage all the social NPCs currently all these social NPCs are placed there right and some of them like again the bartenders are driven by mission modules or area modules and those always get this overspawn so we just need this replacement system at this point like we're we're we keep repeating this error from different angles right and so the population management system is his approach is different he's going to know what's in the game world it's like I need five eyes. Do I have five guys? We're good, right? Like instead of going when I start I spawn five guys which is a complete you know problem. And so yeah over spawning will be solved by this for the for the mean in the meantime. uh you know we're
- 01:56:09
putting in more verifications for the cases where when you stream out and you stream in and the module resets or you bend on a mission or you start the mission during uh while you're already at the action area that the actual uh logic doesn't run twice and that there's actual limits in those areas. >> Sorry, I've just been thinking about a cheeseburger and fries from Five Guys. You said it. >> I just need those Five Guys. all right. So, so obviously, you know, that that that that that's a bit of a whack-a-ole thing until until we get the new system online. >> Yeah. One thing that I'm hopeful about though for this is that we've the the teams have actually built some good uh additional debugging tools that allows us to kind of view the state of this. It
- 01:56:56
it sounds a bit weird, but we it was difficult to even know how many NPCs are there at this location, but that that was actually a difficult information to find. So now we've got super good tools to be able to tell that. So that means if if I'm able to just open a tool and view that information, it means that game designers can do the same. And so that means they can debug those problems in advance without us having to do it post launch, which is the big thing. >> No, Chad, I don't have Five Guys money. Um, all right. Uh, freight elevators, uh, energy cleanup, spawn management. Um when server meshing came online with 4.0 uh it brought with it all this new stuff and all these great things that we could do now. We got to be able to to to increase the server cap uh to 600. We
- 01:57:46
got to improve that FPS performance on the server from 45 to 20 and stuff like this. It also had some knock-ons to some old systems specifically in beacons. So uh uh uh we resolved most of the interdiction stuff in July I feel like but talk to me real quick about why that happened and what the solution was >> little story behind this. So the the if we go back to the 3.x X days, right? Like the game server has everything, right? He's he's simulating everything. All the entities are on one server. And so there's a I don't want to say shortcuts because this was just the way you would build things at the time. Uh where for example doing interdiction was was really like the server has all the
- 01:58:34
entities. So he's able to do like cross zones check, he's able to raycast across zones, he's able to do all that stuff, right? Uh now with server meshing you could be on a quantum path that brings you across game servers right and if you've ever looked at the some of our brackers have done out outstanding tools to calculate uh interdiction positions and stuff like that. If you look at this you'll see that the edges and the ends of this is where actually matters that you get positioned. And so the that means that since we currently use the territory map around planets and moons, right, then typically that will interrupt the quantum jump of the interdiction that is the most effective or close to the most effective. And so um what that needed is a solution that allows us to have an a view of the
- 01:59:23
interdiction system that's basically cross server, right, for the game shark. And so the the replication layer has that information, right? We do have it in the hybrid, but we are very reluctant at allowing uh what I'll call game code here with no uh I'm trying to be porative here. We don't want to let game code query the replication layer uh for multiple reasons, but we want to keep them isolated. And so for this, we needed to build a solution. So the team have actually built a the online team built a new a new service uh that serves every shard but is is able to uh provide our game devs with uh basically a way to build a small representation a small data representation of the game world for things that matter. So, if you can picture this, you're in your ship, you
- 02:00:11
start your quantum drive. What actually happens is now the game server is going to register in this new service called the entity cluster service some piece of data that says we have a quantum drive at this location, right? And he's going to keep that updated, And on the other hand, I've got someone who starts a QED device, quantum interdiction device, QID device, QED, whatever the Q the introduction device inction device uh that will also connect to this and then inform this kind of uh uh matrix view of the quantum interdiction network that's doesn't matter where which server you're on. This is all kept the data in semi-real time as you're quantuming and moving and as the interdiction uh devices are activated or deactivated. And so what that allows the game server to do now is
- 02:00:58
to he's receiving these updates. He's got this this let's call it symbolic view of the of the game world to do it instant. So now he's able to do proper raycasting uh math to see if the interdiction is going to hit uh a cross server. Doesn't matter if you're you know on the same server or not. You can actually trigger the interdiction across. And then that basically gives us this this this ability to have cross-server coordination for things. uh without having to push all this to the replication layer or query it from it. So this is mostly a gameplay system, but we're we can use it we are using it for uh beacons. So we're going to be using it even more for the full beacon implementation, but uh it's going to be the same kind of thing where now you could be in New Babage and someone has a beacon
- 02:01:46
open in Pyro and if the game play wants to allow it, you might be able to get that beacon or see that beacon or bind to it, right? uh and beacons. I mean, this is right now. I mean, right now, we've got the Apollo, which is going to PTU like right now, if it isn't already. It It's It's >> What is it? I haven't checked out. >> Yeah, exactly. We've been doing this for a while. I'm not I'm not looking at the chats, but whenever tonight's PTU goes, uh the Apollo will be in for for for PTU folks who already own it. And that's why you saw the injections of the modules that were that were given to everybody. uh at no charge uh who owned the Apollo. So, you'll be able to test that this weekend um before we attempt to go to live uh later. Um where was I going with this? But oh, but medical beacons but
- 02:02:37
but beacons and specifically medical beacons here have had have had lots of compounded trouble since server meshing came on. First it was first it was the notification. And they used to get this there used to be this big toast notification in the center. And I'm going to remember this wrong and somebody in chat's going to be like, "That's not how it happened, Disco." I'm like, I'm trying. There's a big toast notification that used to happen in the middle. And then when we bumped the server not which was great for people who wanted to play first responders because they could go and they could do stuff and then they had this nice big notification that somebody needed a medical beacon and they could stop what they were doing and then go and run off to the rescue. And that was great. when we bumped the player caps up to 600, that became absolutely interminable and we and we just had to shut that down. It was like, okay, we can't have that. And so they lost that
- 02:03:24
functionality to save everybody else and that sucked. But then there was the issue with one with the second with the second solar system. Uh the beacon if if the beacon originated in Pyro and you were in Stanton or vice versa, you didn't get the waypoint. you didn't get the marker because the marker existed in another star system that wasn't loaded in to where you were. So, it just looked like, hey, I'm trying to pick up this the the this this rescue beacon and it looked like nothing was happening because it couldn't give you the marker. There was no intermediate marker. And then we had some there's some problem that started last month that meant we had to disable the entire thing uh while they work to fix that. this system's had a lot of problems and while not all of them are going to be fixed >> in time for for this stuff talk to me about the scope of the issue and and
- 02:04:13
what what we need to do to to make it better. >> Well, so out of all of this all the su the stuff you mentioned now the I know that our intent is to reactivate medical beacons. Absolutely. Right. I think that's a core part of the of the game we're trying to do and so we we really want those reactivated. the ser the entity cluster service I just told you about is what can enable us to do crossstar system beacons I don't think we're there yet but we definitely want to get there right where we could actually pick those up the so right now all of the kind of technology pieces that you need to fix all the issues you mentioned about beacons are there right we have the ability to have controlled notifications localized detection of beacons cross server crossstar system notifications whatever
- 02:05:01
It's really about, you know, can we put a a team on it to basically add all these things? But at least all of the hurdles are kind of solved with this this introduction work we did and this new service uh the cluster service is the thing that we need to be able to do that to reactivate all that. So that's already live. So >> and uh the bug that prevents people from being able to quantum to the beacon, >> well that definitely needs to be fixed. Uh we I think that even for this uh release, we're aiming for that to be solved because the beacons have to work. Even if even if you don't get crossstar system beacons just yet uh and it's not fully integrated with the cluster service, I think localized beacon should still be functional, you know, like you should be able to quantum to them and you should receive them properly. And uh yeah.
- 02:05:49
>> Yeah. I got a I got a I got a soft spot for first responder gameplay. Like like everything else everything else you do in the verse is is in some way, shape, or form self- serving. And there's nothing wrong with that. There's nothing wrong with going out and making your fortune uh be being a pirate or being a a minor or a salvager or or a racer. Although if you could make your fortune out being a racer, you know, you I don't know how much the day rally pays out. Um, but but it it's it's like like first responders, the people who choose to play as medics and stuff, they're they're choosing a loop that's just there to help other people. And I think that's cool. And I don't think we've done a super great job in the past of of really supporting that as much as we should. So, um, that's not a I don't
- 02:06:38
have the power to do anything about that. It's it's just I I just wanted to say that, you know, I recognize that and we can do better. >> Yeah. Um, >> let's go to a big one. Let's go to a big one where, you know, I'm trying to do like, you know, start with big ones. You know, freight elevators and energy cleanups and little guys like in addition. Um, transit system. >> I I don't hear you. There's something on the line. >> Shitty transit system. I'm in this I'm in the third hour. I'm in the third hour. I get I still That's my one for the hour. Oh, what are we doing? What are we doing? What are we >> Transit. So, well, we do have a hero
- 02:07:26
squad on this, right? And so the So, for everybody's sake, just to make sure we're all on the same page, transit system is the system that drives elevators, not freight elevators, elevators, trains, uh, and some areas uh, uh, base transit for these things. So the transit system has been uh the bane of our existence. I can't frame it to something other. And so at the beginning of the year, we put some significant resources in that hero squad and brought in some very very experienced developers to the team to help push these uh these fixes along. on top of what I discussed earlier about the methodology that changed for these issues um and the introdu introduction of self-healing uh techniques right to
- 02:08:15
try to get it moving and so um we are accepting the current state of transit right I feel that right now the uh we no longer lose carriages or when we do it gets autofixed right the destinations mostly work double doors rarely happen. Um the uh the system mostly responds well. There's still issues with uh I I've seen I I'm still seeing in my playthroughs like uh you know the the train timers of insane values you know which is yeah >> you know a bit of a drag or or um the transit still has this u unfortunate propension of in the event of a server crash and you are uh in movement on the train uh your client will keep
- 02:09:06
interpolating that >> which might end which might get you to space Right. >> We still get the occasional missing elevator and whatnot. There's still a few trains that trap people. >> Yes, that can still happen, right? But right now, the state at which this system is this the approach of that system is, we're kind of accepting it state, right? We're we're still tracking the big egregious issues that we have like the ones that we just mentioned, those are still being tracked and worked on. But overall from a leadership standpoint, we're kind of in a space where now we want to talk about transport, >> right? >> Yes. I've been waiting like you've been sitting here like, "Oh, we're going to accept it. We're going to accept it." I'm like, "Give them the drop, Ben." Don't just be like, "Yeah, we're going to accept it. We're just we're just fine." No, we think it's fine. We think it's going to be We don't think it's fine. We just we it's just we got
- 02:09:53
self-healing in and it can carry us until the new thing comes in. And that's transport system. >> All right. So transport is uh a complete rewrite of transit right which has a much different approach to how it manages peripherals things you click on to call up things gateways which are the areas at which the carriages stop and carriages of course and so the transport system is a lot more resilient in its approach to streaming problems which has been a thing in the past right for the transit uh it's also the way it's developed is a lot more robust uh and so the state of transport. Let's talk about that. So the the team have currently have the approach we've taken for it is the system is is kind of built at this point and we have a single transport link that we use internally. So not yet
- 02:10:42
on a playerf facing build but we have an internal build of this where new Babage habs are completely driven by the transport system. uh which is pretty big uh milestone for the team uh because it means like the system is basically kind of uh fully featured in terms of it's able to drive one of the major major habs uh drop uh we are currently working at instancing those transit transit links so just to reiterate what I said a little bit back in January uh the introduction of instance hangers which means dynamic gateways in the transit system is really what like made it a big issue, right? And so now the team are working at adding that functionality into transport to make sure that we support properly dynamic gateways which will allow you to have a transport link
- 02:11:31
to and out of an instance. Now this could be an instance hanger but in the future might be an area location that has some stuff in it, right? Uh and so that's the state of transport right now is we're working on the instance transit link and trains. Uh so we've done elevators and now we're putting trains in with the full feature which the trains have an additional peripheral which is the the timer uh the added timer on top of it. >> This is one of those things where we talked about it's this is now it's a it in the finished product it shouldn't look any different >> than what they've got. It should just work. It there should just be a time where it's like oh look all these things work and they've stopped killing you underneath. It's an entirely new system that was built after the imp the the the implica implementation of of server
- 02:12:19
meshing which is really the big thing. It's the old system was built long before server meshing. It's it's I I know we we reached out to the original developer who built it like he he's not with the project like hey you know you want to help us you want to help us reconfigure it for for the server mission world then there's like no you just need to build a new thing and what so you put together your team we're building the we're building the new system. I don't understand why it's not called Star Transport, though. I don't understand why it's not called Star Transport, but we're building the new uh >> Starport. >> Why aren't we calling it the Starport system? >> Hey, we could call it the Starport system. We're going to call it the Starport system. Now, um here's what's going to happen. Anytime you try to talk about it publicly, I'm
- 02:13:06
just going to call it the Starport system now. And it really doesn't matter what you call it internally. >> Uh why do we do these shows? >> But uh you know but but but yeah that that's like I said you've already got the first version. It's already working internally in New Babage. We're we're adapting it for instancing. Um and there's some some high hopes that we'll be able to get that in uh some point in >> we are absolutely tech previewing that system in place where >> our our goal is uh going to be to basically we want to try to do one full location with it. >> Yeah. So that all that entire location, elevators, trains, all that stuff is completely trans starport driven. >> And then uh tech preview that and make sure that you know the same methodology and QA checks that we've added for transit are going to have to fully pass
- 02:13:54
before we actually accept uh to bring it in. >> So tech preview for Vulcan and all these graphics things and the mission system, tech preview for engineering, and then tech preview for just use the elevator, please. >> just use the elevator. >> Pretty much. Exactly. Okay. >> Um, you said you said the transit system was your most despised system. I challenge you, sir, with the ATC. >> Oh, man. >> Well, that's one of the casualties of this, right? Uh, so at I hate ATC like I you know it's it's its functionality is really cool but it's it's one of those systems a bit like transit. >> Wait, it has functionality? >> Well, if yeah it does. So uh on ATC we haven't done a
- 02:14:46
lot of work on it mostly focused on metrics and so uh ATC right is this is a system that you call when you want landing or when you want to leave right it is unfortunately responsible for hang your cues uh and is also responsible to assign you a landing area and coordinate the uh your instancing hanger. So basically create the instance manager for your hanger that creates the hand and then manage the pad on your hanger. And so for ATC while we're completely not happy with its stability overall or the Q times which is the one thing that we're actually looking at at the moment is to make sure that the system works the way we want it. That's why we've put all these metrics in about at this location
- 02:15:35
the time the time to exit is this at this location is this. Do we have enough gateway? the system the system performed the way it should. So we're still in an info gathering phase here to have the proper plan. But we do know one thing is that we are going to change the way the interaction between ATC and hangers actually work. Currently ATC manages everything. Like I said it it gives you your landing area. It says hey I need an instance at this gateway. Path spawns your hanger instance created all that stuff. Adds that instance tag to your player. Then the doors open. Now you can get into your hanger, but now ATC sees the pad in your hanger as his landing area. He's managing this, right? And so he's taking care of all that stuff, which is a bit problematic because it's an old system. We know of its flaws.
- 02:16:25
>> And so, uh, we we do want to split these responsibilities so that there's a hanger manager. >> So that the moment that you're you're you're done going through those doors, that's it. Done. ATC's out of the picture. Now you're completely in your hanger and the hanger manager takes care of this. And so that we've been doing tech design on this, but unfortunately uh because of the different focuses we have this year, this is one casualty that I was not able to push forward on. >> Yeah, there's some of these stations that have like 10, 15, 20 doors, but you'd swear the ATC only ever uses one or two of them. >> Yeah, that's the one thing that we want to solve even with the old system before we even start anger management. Now, now when you do the new T system, you know, ATC, air traffic controller, Star Traffic Controller, you call it StarTC. StarTC to go with to go with Starport.
- 02:17:18
Somebody suggested >> or maybe or maybe Starport does ATC, too. >> Star traffic controller. Yeah, there you go. You're with me. The Star Troller. Um, all right. How you feeling, Ben? Well, >> I'm good. I'm good. >> All right. So, we've gone through we've gone through we've gone through our our year of playability. We've gone through our little bug smashers aside. We've gone through our our our what's this? Uh >> Freight Elevators. >> No, not before Freight Elevators. We did server meshing. Why isn't it called star meshing? >> We we went we went through star meshing. Uh we've gone through our list of of of of big ticket items. It's like this. Uh we've been doing this
- 02:18:06
for at 215 now. Um I want to go from big ticket items to >> bigger ticket items. >> The biggest I want to go to this is if I'm allowed Hold on. If I'm allowed to to to to to I'm gonna put my CI hat on. >> Go for it. >> There you go. And take my CI hat off. If I'm allowed to have an opinion that's just mine and mine alone. always >> the most egregious thing for from my perspective. The thing the thing that is, you know, after dealing with getting this getting the server frame rates up and getting the stability scores up and and and killing 30k, which is amazing. And I'm going to shout you that thing I'm going to shout you out through the till the cows come home on that one. That thing plagued us for so long. Um, work on work on transit to
- 02:18:55
transport, work on the freight elevators is still ongoing. all that stuff, great. To me, the most demoralizing, the most heartbreaking, the most aggravating thing is when your stuff disappears. It just it's just that it's just that simple. You spend you spend hours in there. You you you work your butt off. There are people who get together with members of their org to work towards earning a a Polaris from Wiccolo and if they manage to put all their stuff on the elevator and the elevator doesn't eat it, they then get their Polaris, they didn't get their Atlas, they then get their, you know, F8C, whatnot, and they have it. And then we release a patch, which we used to do
- 02:19:45
every 3 or four months, and now we've been doing every every month. So, as an unfortunate side effect of our change in our our our our our philosophy, we exasperated this problem and now it's it's a it's it's it's a it's a gamble. It's a crapshoot. Every time we release a patch, it's just who gets to keep it, who gets to lose it? Bing bing bing bing bing bing bing bing. What the heck is happening? >> What is happening? And what the hell are we doing about it? >> All right, cool. LTP, right, is the famous system talking about >> the main thing in the world. No, sorry. Go ahead. >> Well, let's go a bit into what LTP actually does, right? Because, uh, there's some misconception around it and also a bit of misconception to the
- 02:20:33
amount of effort actually put on it. But, uh, so historically, if we go back to the 3.x era, I guess, I think it's in the 3.x era. >> Yeah. uh we developed the first LTP system right which was came a bit as a side request right which was we patch four times a year was a bit like oh there's some stuff players want to be able to persist cross patch we're still in development so it's okay we're wiping all the time and we want the wipe because blah blah blah there's other considerations so let's build something so we built we built a very small uh system initially to persist stuff and the way that worked is that the game server would basically submit these commands to say this guy picked up a gun, this guy dropped a gun, this guy has this. And so that was a solution that kind of worked at this at the scale
- 02:21:21
of a single server kind of system. And we had a lot less items at the time. >> That solution was in no way scalable, right? Right. >> But it kind of did work better overall than the current one to be to be honest. Um, now come PES 318, come 4.0 I know dynamic server meshing. Now we've got we persist everything else, right? So we need to kind of change how we we deal with uh long-term persistence. And so let me describe a bit how this works currently. So when you acquire a ship from let's say the web platform, right? That that ownership record lives on the on the web platform. It's in it's in there, right? And every time you log in to the game, we check, oh, this guy is
- 02:22:09
this thing. Does he have that ship in the game? In is entitlements, right? >> It's called an entitlement. That's an important term. >> That's your entitlements, right? >> That's your entitlement. >> Uh, so the entitlement records, I know people are going to make the joke about, "Oh, yeah, we're all entitled." Yeah. Yeah. So, the entitlement >> I wasn't going to make that joke. >> I respect these people. >> I'm preempting you. So the so you get we make sure you have an entitlement record. When you actually open up your ASOP terminal, right, and you view your list of ships, what you're actually viewing is your list of entitlements, right? These are not your ships. These are actually your entitlement records. Attached to your entitlement records is the insurance document that goes with it, which describe the loadout of the
- 02:22:57
ship. We call this an entity class loadout which describes what is what is in each port right and if there's no entity class loadout or insurance document with it then it's the default loadout um so that is your your entitlement records that's what you see in ASOP then we also query for is there an entity for this entitlement record at the moment right and that will look at the game database in this current moment let's call it the the inventory database >> uh for now right So we cross link the two. So you'll in your for each row in your sub terminal you'll see your entitlement record. You'll see the state of the entitlement record will allow you to see if you need to deliver it first. That means I have no entity in the game. So when you deliver it now we create the entity in the game and then you if it's stowed in an inventory then you can
- 02:23:45
retrieve it right it starts from the entitlement record. Uh and then if the ship is stowed then you'll see its location is set right? If it's left in space at a location, you'll see its location is set. If it's lost or destroyed or or you know, uh I think there's a couple different states, but one of them is destroyed. Then you can claim the ship, right? And then that will use the insurance document to destroy whatever matches this entitlement. So, we have a way to find entities that were created from a given entitlement gets destroyed. And then we use the insurance document to give you a brand new ship right at the location where you're at. And so that's in game. That's what you have in game. That's the way this the in-game part works, right? So, uh, let's do a clear example. I'm in New Babage. I I have a a hawk in my list
- 02:24:36
and I retrie I I deliver it. It gets created at that location. I retrieve it. It gets untoed onto the pad. I use it around blah blah blah. I fly out to another place. I lose the ship. I respawn and cler if I could do my time. Then I go out to Grimx or whatever I respawn at. And now I'm at a different location. I see the hawk in my entitlement list, but the ship is nowhere to be found. I claim it. A new ship is created. It's put in there. >> Y. Okay. So before we go to the the ones you get in game, which is just to >> reiterate reiterate this so everybody understands this before we move on to the next thing >> because people are always wondering why do I always get the stuff I pay money for, >> you know, real money for, but the stuff that I earn in games is different. That's what we're in the process of explaining right now. Yes. the stuff that you get that you that you pay
- 02:25:24
actual money for or that you get uh awarded through like a a subscription or or a gift like we gave the armor to concierge a couple months ago and stuff like this. Those sit on the the web platform that every time you log into the game, the game goes, "Hey, what am I entitled to?" The platform says, "These are the things you're entitled to." And it gives you back. It's why remember the the the uh character reset back in the day and what not? That's all it did. It just refreshed those entitlements and that's why you got those those things those things back. That system is not simple. Denoa just what just what Beno described there was not simple in the least. But it's it's a pretty hefty system. We it works. We know it works. It's why you are never in any risk of losing the things that you pledge for. Um, and it's why sometimes certain items
- 02:26:12
like the um like the uh race for Stanton stuff, you know, trigger a trigger a thing that gives you a web platform entitlement and those go into your my hanger and you see them in your my hanger and that's why you get those and you have that that stuff is for the most part it's very stable. It's very stable. You can always rely on getting that stuff back. >> That's not how the in-game stuff works. >> Correct. spend one. >> So, so let's we went through the web platform stuff and how ASOP works, right? Let's go into now I go to New Deal and I buy a ship with UC, right? That will actually give you an entitlement, right? Because you you can claim those ships, right? You can transfer those ships. So, at that point up to now, these will work identically, right? But when you log into the game, we don't reconcile from an outsour outsource an outside source your list of
- 02:27:02
purchased in-game ships. There are they are just entitlements in your account, right? And that's it. We keep them. Okay, let's stay on ships for now, right? Before we get it, let's stick with ships. >> Now, the thing that's important to know here is that the actual ship you're playing with, the amount of possible states for that ship are almost infinite, right? You could leave the ship in space. You could leave it in in your friend's hanger for some reason. People are able to do that. You can store it in inventory. Like it's like it can be inside an address that's stowed in someone else's thing. Like, you know, all kinds of stuff can happen. And so that there's multiple paths to update that specific state of that ship. Whereas your entitlement once you have it, you have it. That's why your terminal always has it. Now, let's talk
- 02:27:51
about how LTP works. Now, now this is within a game patch. When we release a new patch, inventory database, entitlement database, they're gone, right? We start from scratch and now we're rebuilt. So, you reconnect, you come in, your web purchases are being reconciled, we create new entitlements for them, and then when you get in-game, you create new ships and then you play with those ships and everything works. What is LTP? LTP is when you acquire things in game and you have ships that you own, right? the way LTP was designed and maybe this is a the part that you know there's a mistake there right but still I'll explain how it works when you stow something in an inventory that you own
- 02:28:39
the systems will create what we call LTP records these LTP records basically say they're close to what an entitlement is right but they say Jared owns a hawk Right? And those are stored in in another database, right? That is outside the game environment and lives cross patches, right? And so now when we wipe the entitlement database and we wipe the game database and we start fresh from a new patch, when we reconcile your account the first time, we're going to give you your web purchases for sure and we're going to give you back your LTP records. And so that will create those entitlements. That's the intent. Now, the problem is, right, we need to make sure that we always stow the stuff back into inventories that we
- 02:29:26
own properly and that those records actually save properly, right? And it turns out that there's a million goddamn ways in which stuff gets stowed in this game. And so, for example, claiming a ship by insurance, then you get the ship, it gets stowed from another path. And in that path, the LTP wasn't correct. We've had cases where we tried to fix bugs related to the F7C turrets, for example, that got disconnected that were a default loadout item that we didn't want to wipe when you claim insurance on it. And that messed up the LTP records for it. And so this whole thing basically is very fragile. And so what we realize now is that the list of LTP records and the way we update them is not always a slam dunk. In fact, there are a bunch of
- 02:30:15
cases where the record's not updated, right? But we even had a case in 420 very recently where we realized two days after launching the patch that the way the recording was going for the ships, there was an issue where instead of giving you let's I'm going to take your Wickelop Polaris as my my example just to make it super egregious. your Wiccolo Polaris that you stowed in your inventory by going at your hanger and and storing it back in would actually persist as one of its components >> in LTP. So you would get like a cooler instead of the chip like you'd get and so that was obviously a very big issue and we had a we we ran corrective programs to solve that and so that's why if you notice in 421 >> you didn't have much reports of people
- 02:31:02
losing ships in 421 but now we're back at 430 and we've had reports of lost ships between patches and so we're pretty certain at this point that we know that there's still cases where LTP records are incorrect uh either through certain game flows there's also Unfortunately, a lot of uh signal to noise ratio issues like people claiming they've lost ships. We investigate their accounts. They never had those ships. We get a lot of that, right? And so at this point, we have we're we have to be very prudent. But >> see, I want I want I want to I want to jump in here make sure to make sure that this doesn't get lost in some of the stuff I'm I'm checking. So there there's there's two issues. We we need to we need to go we need to step back and explain how the checkout and the sto works because that's causing some confusion. We'll do that in a second. But this this bug where uh first bug is
- 02:31:52
things aren't getting stowed right. Things aren't getting stowed right. And so >> and and and so and so when it comes time to entitle at the beginning of a new patch cycle, you don't get the thing. Sometimes it's not getting stowed at all. Sometimes it's getting stowed and being logged as a component part or something else. And that's why it's like, oh, I'm supposed to have a ship. I don't have my ship, but I have this cooler >> instead. I don't I don't think I own this cooler and whatnot. We were on we were on a call, you and I, until well, it was it was like 4:00 in the morning uh my time when we were investigating this and and just giving you ex examples and we're trying to figure out what's going you know you you were >> the man was literally inside the the the raw LT LTP database. who were checking individual items like talking talking to backers on Twitter and Discord and like
- 02:32:42
okay when did you do this and what did you this like like working down at the most granular level in the middle of the night trying to trying to figure this out back and I can't even remember what patch it was um to figure out that they were being stored as other items. >> And so that you can write the skew to to fix that stuff. But all of this happens because of something that's that we haven't explained before and that's the checkout and stow system. So talk to talk to me about and talk to me be candid. It's okay about how the checkout and stow on this LTP system works. And I'm going to I'm going to preface you people. I'm going to preface this with something. We've admitted to a lot of peculiarities in the course of this show. Now we're we're about two and a half hours in. Um we've been very candid. We've been very forthright. Um
- 02:33:30
we've admitted to Benois said the words can't trust dev with this or whatnot. It's like this a lot of things are being said in justest and whatnot. What what I can't control what you take away from this conversation that we've had. My hope would be a a recognition that we're human and that we're constantly trying and growing and striving to do better. Uh if you choose to take other stuff from that, I I can't do much to help that. But explain the checkout and stow system because that's what's going to light the bulb for a whole lot of people of why what the struggle is here. >> So you have so you have the inventory system, right? where that stuff goes and gets stored in there and the records for
- 02:34:19
these. Let's say you have a ship that you store at you're at your hanger. There's a ship on the pad. That ship is unstoed. It actually lives in the game shard database and is actively being simulated on. Meaning we're getting updates from the game server on this ship. Its position, its location, the state of doors, trusters, all that stuff, right? It's, you know, m multiple messages per second, right? On these things. So, it's unstored. Now, you press the store button on your ASUP, right? Platform goes down, doors close. Now, the system will stow that ship. What that means is it deletes the ship from the shard, removes it from the game world, and then moves it to the inventory system. So, it gets stored into your inventory location, your location inventory, right? Gets stored
- 02:35:08
in there. Uh at that point, if you own the inventory where the ship is getting stored, we create an LTP record for it. Meaning that if we wipe the patch right at the second and you come back in the new patch, we will give you back that ship for sure. Right now, for the through the hanger, 100%. Right? But there's other areas where we create ships when they're already stored, right? There are areas where we do repair and we modify stuff like that. uh those are the areas where now the LTP record might not get created or we might interpret it as a separate thing. The other thing is when we stow things into an inventory that you own. It's not just a ship, right? This is the bit that uh sometimes you know people oversimplify. It's the tree of the ship, right? So, if
- 02:35:57
the ship has a box container that has 32C SCU of boxes that have handguns in them, I'm going to give you the ship plus the handguns. Anything in that tree that is flagged for the long-term processing. So, this is where designers look at items in the game and they say this, we want to store crossbatch, right? There are things we do not want to store crossbatch, ammo, consumables, stuff like this. We don't want to store this cross batch, right? That's a design decision that we've taken in the past. And so there's this white list that comes into play that decides what items will create LTP records for and when those get processed. And so the the the process of of of taking something that you're storing
- 02:36:45
and then creating LTP records from it is really about processing the whole tree. when you store an address, we might be creating hundreds of LTP records for stuff that's in it, right? Uh and so that gets to be an issue. Also, like if you have a you can start thinking about the edge cases now, right? You have a a constellation, I don't even know if it fits actually. Let me take a smaller ship. You have a gladius in the bay of an Idris and you store the address in your inventory. Technically, you might get the LTP record for the Gladius in this case. Right now, it depends on ownership and how that's actually trickling. And so, these are all the cases we have to deal with this. And so, when we told players in the past, oh, you need to store your stuff, right? What does that mean?
- 02:37:33
Right? I'll give you a concrete example of where there would be a lost ship, right? Based on these parameters, let's say there's no defect on the system, which we're not we're not completely sure yet. I'm pretty certain there is a defect, but let's start with the there's no defect. You play the game with your Gladius and you uh leave it in space somewhere and now you respawn at uh your at your station and your ASOP says Gladius destroyed, right? Or whatever. And now you have the claim button, right? And you're like, "Oh, cool. I'll do that later." And you leave the game and then we patch. You don't have that ship because when you unsto the ship to the game shard to play with it, we remove the LTP record for it, right? That's the way the system is designed. Now,
- 02:38:26
we're going to change that because that's, you know, there's too many hats in which stuff gets stowed that could lead to problems or that don't respect the rules properly. And that's what we've come to realize. And so without going into too much details about how we do this, right? But this is a bit part of our item imprint epic, right? So very large uh approach there because we always said item recovery tier zero is just tier zero. So there's more item input work there. But part of this is a revamp of how we deal with long-term persistence. And we're going to take a very I'm going to say simple approach to this. we're gonna persist your entitlements across patches. And so instead of trying to always reconcile all the data from the game state, which
- 02:39:15
I just gave a couple of examples, but I could go on, right, of examples where you know the turduckan example of like stuff and stuff and stuff and stuff, right? That and all these things can uh cause either the records to go to the wrong place or the wrong things being persisted. Uh so instead of that we're going to process your entitlements. So that means that if you see a ship in your ASOP terminal you're going to come back to the next patch you're guaranteed to have the same list. There's no way if send or butt because we're keeping that database cross patch completely. And now the other questions people haveve I've seen some comments people are like why don't you just persist the whole thing across patches right we cannot do that for um for reasons related to the data format. let's say between 4.2 and 4.3
- 02:40:06
the gladius to play with is not the same gladius between patches. It's completely changed. Its shape has changed. It's entity hierarchy has changed. The parameters on them have changed. The components that drive it have changed. And so the we don't yet have the tools or the uh the proper approach to version every single part of the game properly so that we could forward migrate the data. Right? So that the gladius that you've stored in 4.2, it's all trees, parameters, it's snapshots, it's variables, all that. So you come back in 43 and we're like, we migrated to the next one. That's something we want to do, but we're completely not there yet. And so we need this LTP layer. So your entitlement records actually serves that purpose right now because when you claim a ship from insurance, we will basically
- 02:40:55
do the same exact thing, which is we persist the fact that you own this ship and we allow you to obtain a recent version of that ship on the current patch. It'll be exactly the same mechanic. So let me let me let me attempt to do Chris hands here and say it another way so that between your way and my way people will have understand and you tell me if I ever mess up this >> So you buy something on the website it gets stored in a website version of of of this database where all these things are are done. You log you log into the game. The when you log into the game it says hey I need my entitlements. What are the things that I that that this person owns from this other thing? It queries this other database that's on the platform on the web, whatever. Gets a list of things, pulls down, and that's why you
- 02:41:42
have it. That system works. It's great. It's pretty robust. It's it's it's people don't have any trouble getting the stuff that that they buy. That's why that's always there. That's great. All right. Everything else you buy in the game is super insanely plentiful. From giant spaceships made of made of all these components down to individual bullets and everything, it's all physicalized and whatever. my maintaining all of that for every single player, every single version that gets left over and stuff is simply not possible. It's simply too big a thing. So, we have to decide what to save and what not to save. LTP is the is is this is a separate database, not the entitlement database over here, a separate database of the things that you have purchased, found, earned in game that gets locked that gets locked to that. We make choices on what that is.
- 02:42:30
Bullets, no. Med pens, no. A spaceship, yes. Armor, yes. And there lots of areas in between, but we'll we'll go with the extremes here. Med pens and spaceships. We'll use spaceships. We'll keep with spaceships as the example because it's the easiest way to explain this. So, you've bought the spaceship, you've earned the spaceship, whatever. You fly, you land it in a hanger, you stow it, boom, it creates an LTP record. Doesn't create that LTP record until you've got it home, you stowed it, and boom, you've got it. The next time you try to fly it, you pull it up from the hanger. That coming up from the hanger elevator is more than just a nice visual. It it it is a representation of what's happening to your LTP record. It is pulling that out of LTP and moving it into the live
- 02:43:18
server game database, the the the universe of the world, so that you can fly it around. It needs to do this in order to uh work for things like piracy and theft and prevent item duplication. has to come out of there to go here. So that's how we prevent duplication. This is all in a perfectly working scenario. There are plenty of bugs that creep in here. People are like, "Wait, I didn't stow this and I came back to two versions of the ship." Those are bugs. We're talking about how it was designed to work. Take it out. You're flying. It's no longer in LTP. You're flying around. It's it it's out there until you bring it back to a hanger. Stow it. make it go down and what at which point it recreates the LTP record, puts it back in. If your ship is destroyed while you're out flying it, it doesn't create
- 02:44:09
your new LTP record until you hit the claim button. You hit the claim button and then it cleans that up, puts it into the LTP record. So, that's not something we've ever been super great at communicating to to people. And why sometimes it's like, well, why have I lost the ship? What why have not lost the ship? We're explaining these things to help you better fight that fire until these new changes come in. That system has lots of bugs, lots of flaws, lots of issues here and there. We're if if you took anything from Ben Wis's conversation, his attitude, his demeanor, it's we fully recognize the limitations of it. It like many things in Star Citizen, it was a product of its time and it needs to be revised and and and and helped and bolstered as as the game continues to grow and get more
- 02:44:55
complex. Those are things you can do to help assuage that. Obviously, everybody knows the thing about landing and getting it back to the hanger. Don't forget to claim your things in the in the ASOP. It's not going to fix everything. Still all the outlying bugs and issues and stuff, but it's another thing that you can do to help you help yourself in the meantime. That said, we have a new solution that we're going to put into effect. And I'm not going to say anything when Ben when I throw throw it back to Ben Woah he can say what he wants to say about when we are when you earn a ship when you buy a ship and whatnot it's yours you've gone you've gone through the mission reward or the wicowo thing or the or the buying not when you steal it from somebody else but when you when you do one of the legitimate ways we are going to create an entitlement record not an LTP record an entitlement record up in another
- 02:45:44
database that's just like the one that we use for when you buy things and And then from that point on when now that we've recreated that record over there in that database not inside LTP that means that no matter what happens the next time you log in you're going to have access to your ship. The next time we do a patch you're going to have access to your ship with the same stability and the same reliability as the system we use to sell things and stuff like that. So that is what they're working on right now. So, we don't have to wait for the changes to LTP in order to resolve this terrible issue of people losing the ships that they've bought with in-game money or earned through Wiccolo rewards and stuff like that. How'd I do, Benois? >> That was actually pretty good. I mean, couple of things I'd correct, but the
- 02:46:33
not even worth it. I think that's totally fine. >> All right. Uh, you you want you want to tell anybody about when we might >> Well, we're actively working on this right now. So the big parts of the online team are are dedicated to this. There's a few um there's a few things that we need to solve in order to put this into full production like uh I mean release it. Um obviously since those are backend systems they have to be super robust and very well tested because you know you know what happens when they're not. Uh and uh but that's the approach. In the meantime though uh we're discussing mitigating uh options right. So, I'm not going to promise those here, but we've we did do a mitigation in 421 that actually kind of worked for ships, uh, specifically ships, not all items,
- 02:47:22
but for ships. And so, we might actually use that again to ease up on some of the pressure before all the systems come in because the reality is your entitlement is what you're you're actually driving at from the ASOP terminal. And so, we can just ensure that you have all those entitlements again, right? And so that that's the mitigation. And so we're probably going to be putting that into play uh just so that in the meantime while we're developing the feature, there's not as much uh you know, ship loss and things like that. Somebody in the chat was like, Jared, starplaining again. Starplaining again. Yeah. Yeah. It it's it's it's it's by my my my my grandfather used to used to serve on the USS Tingi and his his job
- 02:48:10
he served on it during World War II in Korea and his job was to be the person that sat next to the captain and when the captain gave the order to to repeat the order in the in in the radio and whatnot. So, so you heard the same thing from two different voices in two different ways so that you unders so that there was it reduced the amount of misunderstanding and what and what what happens inevitably because I've been I guess we've been doing this for a while drink everybody uh is is you let one person say and you only have one person say and then half the people get it and that's great and half the people don't get it because it's not delivered quite the way. We've found over the years, much to the annoyance to some people who call it starplaining, that when you say it two ways, two different ways, it helps both halves of the people in the community understand. There's there's usually somebody that's that's that gets it from one way and somebody that gets
- 02:48:58
it from the other way. And we like to do both because that's our job. So, you can call it Starplaining if you want, but >> I think you did a pretty good job to be honest. Uh what else we got left? Benois, we uh that that was pretty much all of our our baked ticket things. Um I know folks uh uh before we let you go, I mean we h we got 10 minutes left before we hit 3 hours. Let's go ahead and try. Um recently uh there was some press and stuff like that said Turbulent or some part part of Turbulent is becoming uh what is it called? Cloud Imperium Games Montreal stuff. How how's how's that going? >> Very good actually. I mean, this is we're coming up on the 2-year anniversary of the of the merger, right, of Turbulent and CIG. And I mean, it's going really well. I think overall the,
- 02:49:46
you know, the so we we did want to do some adjustments though because like, you know, the the the turbulent teams even though they were fully working on on Star Citizen, we're still, you know, under the turbulent name, right? which uh you know causes a bunch of grief to be honest in like having to explain you know like uh when we're hiring people and we're bringing into the team it's like so what is there you know oh Star Citizen but you know it's turbulent but really we're the same like you know it's it's okay like that doesn't help right so we we all wanted to make sure that this was kind of one good unified message and so for uh for the game teams over there uh basically we kind of split the two businesses because turbidance still exists as a as a web uh company, right? And the the web teams are fully inside CIG now. But the and so changing
- 02:50:35
the name of the studio to cloud impairment games Montreal made sense, right? Uh to be able to leverage the you know the outstanding talent in the city and you know build stable teams over there and the team has grown quite a bit. And so this was really about like you know clarifying stuff like this you know and I think that's it's going well. I mean the the the integration has been fantastic so far. I mean the the the the way the teams operate across the board at C are improving. I think that the you know even though there's transatlantic you know time zones uh you can see a lot of the fruit of the collaborations between the teams show up right in the stuff that we've produced this year. uh you know the the the teams travel back and forth, the key leads are exchanging and there's a really good gel between this and so no it's it's been to be honest really really positive so far so
- 02:51:24
it's good. >> Well that about does it for what I have for you Ben uh chats been like stands for Star Citizen long sometimes at least when Ben Juan and I get together uh and and people are like Star Citizen long seal longer than citizen gone uh direct that's crazy. Yeah, it will be it will be longer than Siggon Direct. Um uh let's let's talk about that for a second. Still got some time. >> Uh yeah, so in case folks uh uh don't know, Citizen Direct is coming. It's on October 11th. Uh it's why I'm here in my office and not on the spaceship. We've set the spaceship up as a permanent filming location for for Citizen Gone. Um why is SCCL longer than Citizen Direct? Because this just takes me and Benois. This just takes me and Benois
- 02:52:14
doing absolutely no prep, absolutely no prior discussion. Just like, hey, you want to chat for like 3 hours about all these things? I'm just going to ask you about a bunch of crap and you're going to, you know, tell the truth. Like, yeah, we can do that. And he that's a this is a relatively easy thing. Don't equate length with a difficulty. uh Citizen Con, even a Citizen Con direct still involves 20 30 different people all doing their part and we got to work with all their schedules and and like then there's editorial, there's editing it all together because it's like five different cameras at at at 4K each. And then there there there's all the B-roll creation, which is the same uh a wonderful dedicated sixperson team that makes all the trailers and all that all stuff like that. So, they got to do that in between the the win and then, you know, reviews and re-shoots and all that kind of stuff. So, uh, don't equate length with quality. Never said those words before. Um,
- 02:53:05
that was for him. He's he's laughing over there. You can't see him. Um, but I do want to say if you are a sub, uh, some more things about Syscon Direct. If you're a subscriber to the Squadron 42 newsletter, uh, that went out, uh, uh, that that we just that just went out with a with a map of the Odin system. Uh, there was a note at the end there about, you know, the Squadron 42 team still being heads down and working on that. Uh, I want to I want to second the information there and I want I want to star I'm going to star the information that was there. Um, there will be no Squadron 42 presence at Citizen Condirect this year. So, just in case there was any kind of kind of people were like, I didn't quite understand the message or maybe the message wasn't, you know, fully clear and what not and if it wasn't, you know, maybe that was on us. Uh, that team is heads down. We we we we
- 02:53:54
drew a we drew a line in the sand when we said 2026. I don't know if we're going to make it. I just know that we're going to do every single thing possible to make it. And part of that is not taking time for the distraction of citizen. It's why we're not doing a larger event this year. Uh it's why we're doing a smaller, more focused thing. Even even when we did digital citizen gone, those were still these big 6, 8, 10 hour things. We called that citizen gone. This is not sizzing on. It's sizzing on direct. It's a much smaller, more focused thing uh specifically on the work of the Star Citizen team uh and what they plan for the next year of development. uh we're not going to look at 1.0 stuff and so like this. It's just it's just we're going to take a look at what the next uh year of our intention is for that. So it's a much smaller more dedicated
- 02:54:43
thing. Uh I think you're still going to enjoy it. Uh people who are working on it are doing a lot of really hard work to make it uh as entertaining and uh uh informative as possible. But again, just want to be real clear with the expectations management. No Squadron 42 presence at all. That's not a That's not a release date. That's not any of that stuff. We're letting those people stay focused. They're letting those people stay focused. Uh we're on track. Uh there's nothing to There's nothing to report. There's no mystery. There's no anything we're hiding or whatever. It's just like we said when we announced Star Citizen, like we said when we announced Star Citizen Direct, we're trying to stay out of everybody's way and let them focus and let them do the work that they're here to do, which is make the
- 02:55:31
game. It's >> also a misconception that the all the resources are in Squadron 42, right? Like the the reality is there's two teams, right? And we're we got to push both games, right? And it's super important for that. Now, obviously, Squadron, unfortunately, it's got to be under a bit more a bit more secrecy, right? But, uh, >> No, it's it's it's not a So, I can already see the chat. The chat's already like, "Oh, it means this. It means that." Folks, it means that when it's time for the game to come out, you're going to hear a lot more about it, which just it's just it's always been different. We've always been all be big yappy on Star Citizen and less yappy on on Squadron. So, it's just an extension of that. It's just as as we get closer to our intended release date in 2026, we will start opening up the doors, opening up the communication. You'll start
- 02:56:17
seeing more of all that kind of stuff that you would expect in a proper go to market campaign. It's just, you know, we're just doing this. We're staying out of everybody. That's the same thing. It's why that's why there's fewer ISC's this year. It's why there's fewer this year. It's we just want the people as part of this change that we talked about for the first 30 minutes of this show, this reorganizational change. We want people's focus being on making the game. That's it. It's really no more complicated than that. So, uh, when the time is right, you'll see plenty of Squadron 42 stuff. I I promise. So, with that said, that's it. That's our show. Three hours. Thank you, of course, to Ben Wah, who flew halfway around the world to sit in a room >> like 100 meters that way. Uh uh and and
- 02:57:08
take the time uh to sit on the show with us. Um >> I even put my TDK shirt just for you. >> I know cuz you know you know I'm a astute collector of of >> high quality audio cassette tapes. Uh so thanks for watching. Uh thanks thanks for uh hanging out with us. Um thanks for being uh respectful in chat to your fellow chatters. saw a lot of folks with a lot of good conversation there. Uh, and thank you as always again. I I've already, but I'm going to do it Benoa for agreeing to come. I don't know. I've worked a lot of places in my life and I've I've been in a lot of studios and I've never known a CEO to be this willing to come on and share this level of canandor and insight with people. So, thank you. >> I'm glad for it. It's the DNA of our project, man. Got to do it. >> All right. Take care everybody. Love you
- 02:57:55
guys. >> Bye bye. [Music]
Transcript from YouTube’s automatic captions — punctuation and Star Citizen jargon are approximate. The video itself is hosted by CIG on the official channel.
