Star Citizen Live: Tech Talk

Transcript — every row opens the video at that moment

1. 00:00:05

   So, I had this plan and we're not going to look I had this plan. I had this idea of what this show was going to be and what not. You know, you and I, we've been doing this [gasps] every quarter for the last, you know, since the big change the beginning of last year. You know, we start, you know, the February 2025 and stuff like this. We've still got that show and we're still going to do a bunch of that show, but we just put 481 481 482. We just put 482 out yesterday. >> I've got you. We're live. >> Yeah. >> What happened? Um well it started with an attempt at a bug fix a month ago. So we with this

2. 00:00:57

   with this uh 4.8 release and the the way we've been doing patches this year. We have two streams when we work off the from the the working set. We have a base stream that we use for the base release of 4 Z and we have a hot fix stream where we bring all the continuous changes along. Um so there was an attempt at bug fixing something on the address that was actually creating some performance issues somewhere else uh that was checked into that stream and nobody saw it and went it went through. Uh so we've been on the hot fix stream now for a couple for 481 now going to 482. In 42, we went back to the base stream which are supposed to be at par at this point except additional content because we've we've added additional content in

3. 00:01:45

   the patch. That change was included. And so what's supposed to be a routine update became a massive width sprawling issue um because basically the Idris was no longer compatible with its persistent copy. And so what that would do is players have in their ASOP terminal, >> they have a version of the address that's from 481. >> This change, which was not supposed to be in the stream at all, gets in there. And so now the game server is expecting another shape for this. The game server has a series of checks and balances for this, but he's the last banker in the chain. Mhm. >> And so for him, if the schema is

4. 00:02:33

   incompatible, he's like, "There's data corruption risk. I'm out." >> And so >> who's he in this analogy? >> He's well, he's the game server. Okay. Basically, right? So the game server is trying to spawn the address wherever it is, right? >> Uh so that's called the async spawn uh chain. >> And so as he's trying to resolve the object containers for this and to materialize the ship in game, he comes around an incompatibility. And then there's a there's a hard check for this because if we let those through, if we disarm this check for example, then the game server might corrupt chips or might be twisting data and then we would go wider than that and so the game server would crash. So so there's there's all this so there's all this stuff that we have

5. 00:03:23

   to work on and fix. There's all this stuff that we are working on >> in fixing and >> yesterday in a patch that was just supposed to uh just supposed to add you know the two new you know gatac ships and some hot fixes to some of the more >> bring back the oddfixes from the oddfix branch and move forward >> completely unrelated checkin to the Idris. Yes, >> somehow made it through validation or missed validation. >> Correct. >> It's like this got in and now the server which is still built the persistence records which are still built on 481 and expecting a 481 version of the ship >> also 480 because 480 >> they're they're compatible >> really. Okay. So, so was expecting a 480 version of the ship, got a 482 version

6. 00:04:11

   of the ship without knowing that there was supposed to be a difference and just went, >> you know, all right, I'm a head out. >> That's right. >> You did the the the Spongebob meme. >> Yeah. And so every time an address would be attempted to spawn, that server in that region would crash and then restart and then try to load the address again and then crash again. And so as players were untowing more addresses and more shards, more servers got affected. Um, until you reach a point where now the ops team is in the background and they're like lighting all the fires like the Rohan uh calling the Rohan. I >> I saw you online at like 3:30 this morning. >> Yeah. Yeah. Yeah. And uh and so we're looking at this and basically we start cording off shards because at the beginning we're not sure what the issue is. Uh so we start cing off shards and

7. 00:05:00

   uh what happens in this case is like the more shards are getting offline the more uh the less capacity we have for players that are either entering the game or shard hopping. Uh we have a lot of people shard hopping. And so uh what happens is they hit the dreaded 60,12 error which means hey there's no capacity in your zone. There is capacity but it's we've we've had to restrict it in the meantime. And so we took a hard decision later in the night to uh prevent players from spawning addresses temporarily just to [snorts] kind of stop breaking other shards. We brought more shards online to keep that playing. And then we basically uh went through and and manually uh rectified uh that's what we call percussive maintenance. When you see that in the status, it

8. 00:05:48

   means we went in and we did a drastic measure to basically fix the addresses that were either spawned in shards or still left in inventories to make sure that they had the proper schema. And so now we know that all the addresses are are fine. We've lifted that temporary uh stoppage for that ship. And so now we're back. That's what happened completely out of left field. absolutely unrelated to all the stuff that we working on, the stuff that the patch was supposed to be >> and whatnot. This is >> this is a cabra trays being spawned by uh some uh some subcomponent in the address cargo room. >> And it's not even anything specific in this case. It's it's the address, but it's not because it's a capital ship or

9. 00:06:35

   anything. It's just it's >> No, it could have been any in that case. So normally there's a check-in discipline that we apply especially this year because we have a different uh publishing model this year and so ships that have been distributed already in a given major minor patch cannot be modified. Now for some reason this checkin passed all the validation check and ended up in the build. This is we're gonna we're gonna get to the the the show proper, guys. With what happened last night and Ben Wo up till all hours of the night, I to say this is frustrating is >> understatement. >> An understatement. It's I'm gonna I'm I'm I'm going to give you

10. 00:07:23

    some time. We're we're going to get into the show proper and then I'm going to come back to I'm gonna come back at some point about why is this happening and whatnot. It's it's it's if if I've got you here, I've got you I've got you guys here. >> I'm gonna >> I I have I have we have to get into it. We we have we have to get into it. Absolutely. It's it's >> But yeah, let's go ahead and let's go ahead and get started. Let's go ahead and get started with the show. Hi everybody. >> It's a very weird start the show. I'm Jared. um the uh content director for Cloud Imperium Games, Robert Space Industries, and the host of Star Citizen Live, which is the show that you're watching right now. Uh every approximately every quarter for the last year and a half, uh we have a show where we we have a schedule show where we have our chief techn uh tech techn technology tech technical

11. 00:08:12

    >> your pick. >> Well, I'm way off the whole script today. the chief technical officer Ben Wisa on the show uh to discuss what is essentially the current technical state of the game. Now, I want to be real clear. This is about the technical state of the game. Uh design issues or content issues or whatnot, not the realm of of the people that we have on uh today. Uh joining us on the show today uh uh because we kept threatening to do it and we we wanted him last time and whatnot and it's like he he made the mistake of promising, oh, I'll be on the next one. Yeah. >> Is our uh uh director of programming, >> gameplay programming, the gameplay engineering, gameplay programming, uh Yens, >> both of them. both of both at the moment. Uh and we're here to talk about 481 482 like this the current state of

12. 00:09:00

    the current technical state of the game. Now 48 at this point went out about four weeks ago. Yeah, >> about four weeks ago. >> Um there's there was a lot of really great stuff before we we just we just had a couple unscripted minutes about some bad stuff and we're going to get to more challenging stuff in a minute. Let's talk a little bit about how 48 did. I saw a lot of there was a lot of commentary online that 48 was as bad as 318 and whatnot. We don't need to rehash 318 whatnot. Anybody most people watching the show knows 318 at this point. Was it really as bad as 318 >> or or talk to me about how what what's your your um

13. 00:09:49

    >> your take on how 48 went? Um well for me if I take from the machine room side of things yeah like the we've been firefighting since we launched 48 I mean from a whole slew of different issues >> some of them are to be expected when we launch a page that has this many amount of core changes like 48 has um we've touched on systems that uh are really central to this reversal again so we're talking about the entitlement system which has not changed in years at this point and we've completely overhauled the entitlement system in 4.8 to power in part the item imprint system but more importantly to pull the entitlements out of the patch cycle and persist them across patches. So what we call the LTP layer is not just what you have in your

14. 00:10:39

    inventory, it's also what you own. Your ships and all of the items that you purchase in game now have an entitlement record that holds the insurance loadouts for it. uh the uh all of the customizations that you make out of it and that's hosted outside the game not outside the game but outside the game patch which means that when we come back from a patch there's no risk of oh I'm not getting my ship back right what you see in ASOP in the first patch will be exactly the same data source in the second patch and so that risk of like losing ships across is completely gone so we're very happy in fact that bit actually went very well in 4.8 right? Even though up to the launch it was, you know, there's always last minute things to look at. Uh [snorts] but we have been firefighting across a multitude of different issues that were tangential to

15. 00:11:26

    that. Um some of them we can touch on a little bit because I've made some notes to talk about gameplay systems here and why they were affected by some minor infrastructure issues that turned out to be massively contributing to the perception of the patch. Uh, so I'll talk a bit about that in a bit, but basically we've been firefighting since 480. Um, I want I want to touch on to one thing because we we just had a one-off cuff moment about the Idris last night, right? But, uh, two weeks prior to that, we had another downtime patch that, you know, needed a lot of rectification. So, the the people that uh, the teams that run the online group were, you know, hard at work on on this maintenance. Um, [snorts] and unfortunately, another one

16. 00:12:15

    of those uh unwanted check-ins came in during that 481 cycle where in an attempt to fix the VGL flight suits that was missing hands and feet. >> Mhm. Um, another another check-in came in to that removed subcomponents of this flight suits, but not just remove them from the flight suit, deleted them from existence uh from the game. And that's not something that you can do in a a minor patch. And so clearly that's two instances in this patch where changes and check-ins should have been validated and prevented and went in and then after that became an infrastructure problem because same issue here if somebody was wearing the VGL flight suit and referred

17. 00:13:05

    to these uh entity classes that were actually representing I think the nozzle of and the back the front pack of the suit then the game server would crush. And so we had again to do percussive maintenance to remove those items uh and fix that. And so that highlights to me a massive issue with how we validate check-ins and you know how we get changes in the game uh across persistence until we have full uh forward porting compatibility on the on the persistence layer which we don't have at the moment >> but will be game changer. >> Yes. How what's your perspective? [snorts] I'm keen. You know, it's >> Well, it's a it's you know, it's uh live servers. You

18. 00:13:53

    always you're always running, right? >> Always you're always running. Like we're talking about forward patching. You know, if we'd been behind closed doors, you would have done that, right? No one ever seen the game until you've you've done that piece of technology. But that's not who we are. We're we're always running. And that's what makes it really really exciting to work here as well, right? Because you know it's very we're putting oursel out there all the time. But if I look at this specific case, you you're saying like, but we have we have checks and we have contracts on what you can and can't do in these um substreams that we've already released where the data management is super super strict and obviously now we invalidating them twice in a row. It's

19. 00:14:42

    time to to um look at like hide how how do we >> how do we really >> you know add that extra layer of protection. >> Why is this happening? Um why look I it's it's I know I have to be I'm allowed a lot of latitude here. I'm allowed I'm allowed a lot of latitude here and I appreciate it from from the people that I work for, >> but this is this is really frustrating. >> This is how is it I I I understand that you know Rich Tyer, our senior game director, went on stage at Citizen Con, you know, two years ago and laid out the

20. 00:15:32

    road map to 1.0. It it's it's it's and I understand that through 25 and through the first half of 2026 we have made some legitimately ridiculous strides towards making this game actually the MMO it was always trying to be you know if people who followed this project 2015 2016 a lot of a lot of you know the years before you know it's it's very easy it was very easy to have questions like where are they going like like where are they going why why is why is this like that why is that like that? It's like it's it's we would often see questions. Has anybody ever played an MMO before? Do they know how to do this? It's like we and there's a reason there's a reason we made a major pivot. We made a major pivot at the end of 2024. In 2025, we went into this content mode and and we

21. 00:16:21

    said no new features. No new features. We're going to focus on content while we shore up all these things. And when I was you and me sitting here more than a year ago, like this. And in the last year and a half, we have tripled the amount of content that we've put out compared to 2024, compared to 2023, 2022, all combined. Like the amount of content we have put out in the last 18 months is amazing. And the people here are working their asses off to make it as good as possible. I don't doubt the intentions of any human being here. I don't doubt the dedication of anybody here. We're not We're not here today to apologize for anything. It's a I know that we're all working as hard except Steve. We're all working. We're all working as hard as possible.

22. 00:17:12

    But why does this happen? Why is this happening? It's it's I can't I can't I can't sit here and do this job and feel good about my job without asking you outright. Why is this happening? If everybody is working as hard as I know they are and everybody is working as dedicated as they are and as everybody is as talented as I as I know they are, why does something like this still happen? Why is it happened twice in the last two weeks? >> Not how you pointed to me when talented, [laughter] not you. >> I know, right? >> I I think the there there's a part of that which is that we're still paying we're transforming the game. We're still transforming the game into an MMO, >> right? >> We're still in the process of taking a game that used to run on a single server in a ser and now obviously a server

23. 00:18:02

    mesh, which that's working, but the game systems need to expand as well. And if you add the content pressure on top of that, right, of getting content into the game, releasing it at the same time, stabilizing it, hardening it, hardening it, and features that come in and do their inverse, new features come in and they destabilize the systems. Um, without stronger guardrails that also need to evolve with this, that's where we ended up. And so I think that to be honest I think at this right now we're we're outpacing our ability at of verifying everything that's been being added to the game. And this is where we need to put a lot of our energy at the moment is to put in those guard rails so that you know the the check-in that would cause this

24. 00:18:49

    crash uh issue, right? Those are identified at this point. And so it's all about adding these validation frameworks to prevent that from happening and keeping the data stable and fully validated. Now that sounds >> well why didn't you do that before? Right. Uh >> you're not you're not saying that there were no guard rails in place. >> No, >> there are guard rails in place but >> they're not keeping up with the pace. >> Correct. Yes. Exactly. like we create more we create more content and more features than we have time to add guardrails in to prevent these uh these these cases right uh you know an example I could give on something unrelated to persistence right but you know you you have mo we're at the point now we used

25. 00:19:38

    to like we make a weapon we make a ship that's it right now we do we have a team making weapons they balance the weapons they you know they design the weapons They work in that realm. They have guard rails in there, right? They they they they know what to check for outliers, data sets, you know, how the how power is distributed, etc. You go to the next team. The next team is making ship components. They have their guardrails, they have their approaches, they have their things, right? They balance it out. You have another team that's making recipes and dismantling, right? They have their own guardrails, their their own systems. You have another team that's making economy. They add those things to shops. They add pricing. They they make sure that that's balanced. And then they move on to another team that's making adding this content to loot boxes. Right? Now, we didn't have that uh a while ago, right? And then mission rewards added in there. All of these

26. 00:20:26

    have validation, but there's no holistic validation across, right? Where we we're at the point now where all these systems communicate with each other all the time, right? And with crafting has changed the game quite a bit on that on that area, right? Uh, and so that translates into more weapons being created, more content being created, more missions giving more rewards, and all these things need to be validated together, not in silos. >> Is this why is this why in 48 there was the um there was the thing where there was there was a weapon, I think it was a ship weapon. I I can't remember specifically, but you know, you you paid 20,000. I'm pulling a number out of my butt here. You paid 20,000 to buy the ship weapon, but then you dismantled it and you got like $80,000 worth of materials and

27. 00:21:13

    >> that you could sell in one area >> that you can send. So people were just buying the vendor thing and doing and you're saying that's because the the weapon and the pri and the econ the pricing and everything for the weapon was validated in one by one team in one silo, if you will. >> And then the >> the validation of the crafting and the breakdown and stuff was done in a different one. >> Sure. And there wasn't something that >> transversal to check all of that and and also like there's we're get we're definitely now post47 in a realm where just human validation is not enough right and so a lot of of game design data is typically you know managed by the team uh but I think that we're way past that and we need machine validation for everything because the game we're the rate at which we build it and the

28. 00:22:02

    the amount of technology we're putting in is just too fast for for validation to keep up. >> Okay. >> And we can't just rely on endpoint validation because in this case it was machine validation that caught the error. >> And stopped it from corruption. >> Yes. Yeah. Exactly. Like the machine validation had >> correct. Yes. Exactly. >> So have to come earlier. >> So is so for validation we're we're gonna let's just stay on this topic here for a bit. It's people are going to [snorts] as talented as everybody is as skilled as everybody is and stuff like this you know mistakes are going to be made and the wrong things going to be checked in and stuff like this and and the game is the game is growing at a scale like I' I

29. 00:22:50

    I I I've been here for more than a decade and stuff I know it's a Jared meme drink anytime I talk about how long I've been here you can just see it in my hair now weird Um, I am so incredibly proud of the game right now. It It's the last 18 months have hands down been the best 18 months that this game has had in since the PU went online in 2015. And anybody that says otherwise is being disingenuous. They they they just are. You know, I'm being hard with you guys here. I'm going to be a little harder. I'm going to be I'm being hard with you guys as well. It's like the last 18 months have been phenomenal for the progression of Star Citizen. It's our our our our daily average users are up. Our concurrency is

30. 00:23:38

    up. The the the content is the best content we've we we've ever put out. TSG is finally real, you know, we finally have, you know, you know, coordinated space combat in our, you know, space combat game. [sighs] Is this just is but it sounds like we're moving we're moving fast. We're moving fast and we're doing a lot. You said the guardrails aren't catching up. How do we I I get machine validation. I I get you got to build build some stuff. Is is is that all we can do? Is is does is that make me feel [gasps] make me feel better about how we're going to fix this is what I'm asking. Because if I feel better about how we're gonna fix this, maybe there's a reason for people watching to feel better about how we're gonna fix it. Sure. It's just

31. 00:24:26

    just the validation issue. >> Sure. >> So the it's it's a problem we've known we've known about, right, and we've been trying to solve, right? And so the more recently there's more changes that have been coming from the engine side to give us a framework for validation. When I say framework for validation is that game designers can participate in this, right? It's not like oh they need to hand it off to an engineer and engineer is busy and he can't do the work and now the validation is not added which is the mode that we've been in in the past right and so we've been in the in the process of building a validation framework for all game systems and all u pieces of data stored in the game that can do queries across all data whether it's hey I have a mission in this location that requires this that pays

32. 00:25:15

    out this is that an outlier compared to you And so being able to do cross data queries across the entire game, which is not something we've had before, which allows us then to just express these checks, find outliers into like mission payouts, find out liars into uh how uh you know, if we go back to the example of the gun is for sale at at this location, at this price, it's dismantleable into these components that can be sold at this location. That exchange value is too high, right? And so creating these validations will become part of the design work completely with this framework which will allow us to have all these guard rails. On top of that we've got the technical the more technical validation right. So the two issues we got in 48 uh

33. 00:26:05

    related to modifying ships that were already in the distributed in the main patch and removing entity classes for example. Um both of those things already have lock checks right now. That's the first thing we did when we discovered them. Uh but we're going to keep adding more of those compatibility check. Now 48 also has the beginning of database patching. So uh in 47 we didn't have the full ability to persist all the crafting properties on weapons. So if you crafted a weapons, all the modifiers were not necessarily going to persist through LTP, right? In 48, we have introduced in the back end patchable serialized variables. We didn't talk about it much because it's kind of mixed into the crafting system,

34. 00:26:54

    >> basically we now in the schema of a component which stores the data for something. Let's say it's a weapon, right? And I have the modifiers for it. Let's say elongated nozzle, faster damage rate, whatever. uh these values uh we now know which one of those values is patchable, right? And then we're able this way to persist them and patch them in the next release. We also determine which ones we store when you stow it. We don't stow all variables which used to be the case. And that's the foundation for us to keep working at what we really want to achieve, which is a forward patchable data set. So that now these even if if you know the ship team goes in changes all the serialized variables for a ship and rejigs the whole object container layout for a ship that will be able to live migrate your ship to the

35. 00:27:41

    new version without crashing the game servers in the meantime. So basically embracing this movable data set, but that's tricky to do. Obviously, we've been at this for a long time, but 48 has the beginning of that uh for just for crafting properties, >> right? That that's great for allowing us to to better manage when somebody checks something in that they didn't do. But but I'm talking about what are we doing to stop >> the the the check-ins that aren't supposed to happen. Sure. Like through validation. It's I want to >> Well, it's pre-validation, right? When I call this validation framework I'm talking about is pre-checkin, >> okay? >> Right. It's something that I I'm a >> I'm working on the game. I'm like, I'm going to add a new gun and I'm going to price it this and I'm going to make it super powerful. Right. Well, that would

36. 00:28:28

    flag I could not Yeah, I could not check that in, right? The system would go, wait, you need some you need some authorization before you can check this in because of these reasons, which currently we don't have something like this. That's cross-checking, right? Obviously in each silo, yes, but not within. >> Is this something Is this something I'm gonna have I'm going to have to I have to ask you the one question here. Is this something that we're talking about that's that's a 2027 thing? Is this Are there parts of this at least that can go that can go in? Is is is there is there hope for this >> sooner rather than later? >> We need this yesterday. Like I'm aware we needed yesterday, but but >> No, we're working on this right now. like uh to get all these checks and balances in place like I don't know if

37. 00:29:16

    it will be like a 49 kind of thing but it's it's happening now >> okay >> because we everybody around the table knows now like it's clear from the for experience that machine validation is required across the board so so why isn't this why isn't this just let's let's tackle this from another from another angle why isn't just throwing more QA way at this the answer. Why? Why? Why? Why isn't this just hire 500 more QA people and we can and this gets fixed? It's it's I suspect you probably have a an answer for this. Um you know I mean in this case it took the situation of like the data how the database on 4 48481 was and with the new

38. 00:30:08

    data from 482 to create the situation and that's not something that the QA environment has. So whatever we do, we as a technology group would have to come in and at least give them that and then if you put your 500 QA on it then then they would probably one of them would have spotted an address but without that environment you could put as many as you like on it. No one would ever have discovered this situation. this is just too far a move. And I think that's that's that's so come back to the the large data set, right? As you have a large data set, this you just can't scale in manpower alone. You need something that scales way beyond that. I suppose if you just hired if you just hired 500 QA people that that doesn't

39. 00:30:56

    mean there's 500 requisite programmers that can all that can all take the the stuff and fix it either. >> No, I think they would meet some frustration with yeah their their their output and our our ability to tackle it [laughter] would be would certainly uh create a se separate set of challenges. But also what's frustrating with this situation is that, you know, we've done multiple patches where this just worked fine. >> Oh, yeah. We we've been on this we we've been putting out a new piece of content every two weeks since the beginning of this year. You know, last year we move, you know, in for 2024 and before we were on a schedule of one patch every 3 months and we didn't I think we actually hit every 3 months once in the five years that we were on that cadence from 3.0 know to to four eight I I can't remember which which one

40. 00:31:46

    was January last year. We were trying to do a patch every quarter and we actually hit the quarter date maybe one or two times in all that time. >> But so it was four patches a year. Yeah. At most basically is what we were doing. And then in 2025 we as part of this major initiative we switched and we went 11 patches. >> We went 11 patches in 2024. Uh, again I it's it's again I I don't want to sound like a broke broken wet record. The content that was that was in those patches is is some of the stuff I'm most proud of that that these team that their team here has ever put out, but we we were able to do that because we pulled back on features. We we we pulled back on features and it was it was content team. There were a couple features here and there that would sneak in and whatnot because oh we

41. 00:32:33

    need this for that and the other thing, but for the most part it was all content. In 2026, we now switched to and any any that's been really paying attention and following the development would see this already. We've switched to a bimonthly patch where we put out a major patch every two months and we do these two week content drops in between. So, every two weeks there's there there there's there's there's a certain thing gets turned on. people who have been in PT who have been testing uh they they tested this aspect and then it wasn't there in the patch and it goes oh because it gets turned on two weeks it gets turned on two weeks after that gets turned and that was to to help with engagement throughout you know throughout the year to give people more things to do to keep people more interested. It's one of those things that you do when you're running a MMO and that was that's been great up until the last

42. 00:33:21

    >> four weeks that's worked. >> Yeah. And and so it's I don't I don't want to har I don't want to harp on this. I don't know how much of this is my own personal I'm stuck on it or whether this is what I'm supposed to be doing with with with with with with the with with the show. But it's it seems like it seems like we know what we have to do. >> We have to solve the process for sure. >> We have to solve the process. We have we have to solve the process. We have to build more tools. We have to scale our validation to match the rate of content. Yes. >> That we're doing >> and the complexity of content too, >> right? Because this year we also turned on features. We turned on the feature spot this this year after pulling back

43. 00:34:10

    on it in 2025. >> I think I've been having I can't tell if I've been messing up my years this whole time. Um, all right. How confident how confident do you feel? How confident do you feel that you've got a handle on this? >> I think on the I think we know what we have to do. I think it's going to take a while to get all of these guard rails in place, right? But in the meantime, I mean, we just have to respect the rules that we've set forward in the meantime, right? So that while we can't machine check everything this right this second and to be honest I'm not talking about months of work here on on these validation frameworks right but it's about putting in the ability for every team to contribute

44. 00:34:59

    guard rails right which right now it's a bit splintered and so we need to bring this all back together so I'm confident that for the especially the errors we just talked about right like releasing a piece of content that crashes every server on launch or twice, right? Um or evident misconfiguration of items or costs or recipes, right? I'm confident that that will solve the issue that those issues will no longer occur as long as we respect the validation. Nobody tripped them off, which you know the next battle >> the validation issues. >> Yeah, exactly. I think that we can eliminate that. >> All right. So for [snorts] the last

45. 00:35:47

    35 40 minutes, we've been off the program here a little bit. Uh the issue that went out issue that happened with 482 yesterday uh because of a validation issue. Um I didn't feel like we could go into the program that we had originally planned without addressing it. It it was just I was very frustrated and sometimes I abuse my position here to do this. So you're you're you're laughing. I'm the one that's going to get back and see if you know my bosses are still happy with me or not. Um we do have a lot we do have a lot to cover. A lot that we wanted to cover and stuff and we're going to start getting into that stuff now. I hope watching you'll forgive the diversion that you'll agree that it was worth it. Um, I think there was some

46. 00:36:37

    very useful stuff there. Um, and I do want to I I do want to say I have to say this. This is this is this is me personally this goes it was it was an issue with a ship yesterday but the ship team the ship teams around the world there there there's a team here there's a team in Austin. There's a team in Montreal have been some of the consistently quality deliverable delivering teams of this project for years and years and years and years and years. You know, it's it's easy. It's it's the last couple weeks have been a bit rough, but man, it's it's like where would we be? It's it's where would we be without the ship team. So, it's like unless this seems like we're coming down on the ship team today. That is not my intention.

47. 00:37:26

    They they happen to be they happen to be in the hot seat, you know, yesterday. It's like this, but this could literally happen to anybody. And it did. It happened. It It was It was It was a whole another team two weeks ago, and if we don't get these guard rails in place, it'll be a whole another team two weeks from now. >> So So, uh I just I >> For me, the takeaway is human guard rails are not enough. Like, yeah, that's the thing. >> Yeah. Just It's Yeah, I just didn't want it. It's it's unfair to make it seem it's unfair to make it seem like the ship team hasn't been absolutely killing it for years and years and they still are. So, um desync >> let's just let's just [gasps and sighs]

48. 00:38:16

    Nobody is ever going to say yes to being on this show ever again. Um, [snorts] Ben, while you uh I put you on the hook to write a big old write up on on DSYNC. And I know it's being worked on. I've seen the draft and what it's being worked on between every other initiative. You know, time spent writing is time spent not actually fixing things and making processes better. >> Also, we got a couple guest stars in it that I'm hoping will make a difference, too. >> So, that's still that's still underway. That's still underway. And it's a it's it's more like a it it's it's when we're done with it, it's going to be more like a one of those it's not a design brief. That's that's not the right thing. But kind of like what you saw with the gameplay guide. It's it's we're we're going to break it down and I promise you it'll be the most educated thing that

49. 00:39:04

    you'll ever see from a game from a game studio on on what it is and what what it isn't and what causes it and what doesn't cause it and stuff like that >> and the particularities of our game architecture. Yeah. But that said, when 4.8 went out, I I started with 4.0. 4.7 was was was great overall, but but there's been that there's a reason I had to call you and ask it. There's been this mounting desync for the last several patches. For the last several patches, it starts out small and it it gets bigger. It affects some people more disproportionately than others. There are there are people there are people that I have talked to whether at a bar citizen or just reading the chat here today or in any other show where they're having a virtually unplayable experience. And then there

50. 00:39:53

    are other people who are like that's not my experience. I'm having you know it's it's I'm having a wonderful time and whatnot. >> What can without just reading the entire dock that we're doing I I can't get past this show without talking about it at least somewhat. >> Yeah. What do we know is causing it at this point? What are we doing to resolve it? >> Just a simple question. >> Well, so um what we know about right now is that there are some meshes, when I say meshes, I mean a dedicated game server inside a shard that are running way below uh FPS spec that we want, right? And so we don't have a handle just yet about what is actually causing it. We do have performance captures and

51. 00:40:42

    we're tracking at least six to eight very specific optimizations that need to happen. One of them is within the area component which is used all over the engine. uh area and zone component they kind of go in in hand but uh where be again because we're pushing more content right the hierarchy trees of the game world is much bigger so there's more areas more zones more subdivision more more more uh there are some areas of that where if you add a lot of entities in those that enter or exit these areas there's you know the engine provides listeners to see track entities entering areas or leaving areas or zones changing, right? [snorts] And uh obviously there's concurrency

52. 00:41:30

    uh constructs around this to prevent one thread from crash accessing another thread, right? And so just keeping it simple without diving too much into the low level, which we will probably in some of our desync areas, but [snorts] there's lock contention in the area component at the moment in some places. >> There's lock contention, >> okay, >> in some areas of the game. uh when I literally mean area like you know if there's many entities in this spot because of the way it's configured uh we're seeing like uh basically FPS drops in those areas uh and so we're tracking about like I said 6 to8 major optimizations that we're trying to get in and hot in our hot fix channel right to try to get this FPS back up to the spec that we want which is above 20 you know uh across

53. 00:42:17

    >> server FPS >> server FPS correct >> somebody's going to somebody's going to take that clip and say, "Oh, no. Star Citizen is only looking for 20 FPS in gameplay and what?" >> No, no, no. And it's important to note that like when a server is running low FPS, it doesn't mean that it will create desync, but especially when you're interacting with entities that are simulated on that DGS AI, AI characters, NPCs, uh then the you're going to perceive these things because the source is slow at creating its its output. Um and you know the the traffic goes from your game client to the replication node >> replication node to game server. Now if game server is running slow at the beginning of the frame he's taking the state from the game uh from the network to the game and applying it at the end

54. 00:43:05

    of it. He's applying the changes back to the network. Right? If that loop is slow, then your client's going to predict something and then that prediction will get corrected the next frame and that will create a rubber band, right? And so that's the gist of why there's desync in general. >> Why does the client have to predict? Uh because if he doesn't predict then you're going to be at the tick of the server and you'll see like >> even if it was running perfect you'd see it in an FPS like because your >> because your need for it to be you know so so responsive is too high >> is this so all right so the so so the so the so the g the meshes inside the shard are c certain ones are slowing down >> correct it's not widespread in all

55. 00:43:55

    areas, >> which is why some people are in there having a great >> You're gonna tell you I'm not going to tell you which areas if that's what you're asking. [laughter] >> Uh [snorts] I got I got to ask the obvious question then. At least it's obvious. It it seems obvious to me. >> Is this is this related to the ship cleanup or lack lack thereof? That's one one of those things that we're looking at is how the what how aggressive or not the ship cleanup is actually happening. Uh that's one of the things we're looking at. Yes. I mean, because because you you can you you can you can watch a a Twitch stream or or look at a council report and you can you you'll see you'll see you'll see I was I want to say dozens and dozens, but I feel like it's even more, you know, brricked ships everywhere, you

56. 00:44:44

    know, with with with the with the with the insurance changes and whatnot, you know, breaking these ships. >> And whatnot. Is is this >> it's Is this viably part of what what it is? Is this is this >> in part? Yeah, it does for sure doesn't help because the more entities are replicated, the more the server needs to do every frame and so that contributes to this. So yes, >> you can usually find it when you look at a server that is performing great usually has a a larger a larger vehicle population. Certainly >> the server that is performing great has a larger >> no a not performing so great tends to have a slightly larger vehicle population or a very large active population. That can also happen in general just a lot of everything. We're we're a we're quite a heavy game entity wise.

57. 00:45:31

    >> We want to avoid getting to the sort of size that we were in Stanton before we went into meshing, right? Because that was consistently >> difficult to manage. Yeah. But if we're Okay. So So if we're So if we're already Stezzig is like fix it. Super helpful, dude. Um, >> oh man, it's so easy. We're done with this subject. >> You know what, Stezic? I honestly none of us had thought about that. That's >> all right. We don't even need to do the rest of the show, do we? >> Yeah, I know that there's been there's been some changes related to how we perform entity cleanup. So, an unattended vehicle cleanup used to be a separate system, right, that ran on its own. That's been merged into what we

58. 00:46:19

    call the density manager which is basically an engine functionality that will look at overpopulation in in in clusters of areas that obviously designers have control over. Uh you know for example we want to make sure you know like the exit routes for hangers are cleaned up and so these are you know have active cleanup zones configured right that's designer controlled. Now the engine will then after that apply the logic on the volume determine oh there's too many of X in this area and will start popping stuff out right he's going to do the cleanup now for ships the unattended vehicle clean ve sorry unattended vehicle cleanup system used to be separate now it's been brought into the density manager I think end of last year something like this so it's been a while

59. 00:47:06

    uh and this one tracks whether there's passengers on a player passengers on uh on the vehicle and we'll uh based on a certain timer configuration and the values are different depending on the areas will uh uh either stow back the ship into the originating inventory or destroy it depending on if it's a player owned ship or if it's an AI ship depending right now it does look like maybe there's a misconfiguration in key areas that are leaving ships there. So we don't think the system is incorrectly functioning. We think it might be inaccurately configured, >> in some areas. And so that would allow a very popular area to just accumulate ships like crazy, right? And so that's what we're looking at right now. >> Are you able to are you able to clean up are you able to clean are you able are

60. 00:47:55

    you able to differentiate between you obviously mentioned a a ship that has a person on it? That's great. But a ship that doesn't have a person on it, but is still a viable ship versus a versus a destroyed hulk versus a, you know, a brricked ship. From the inventory, you're able to distinguish between and set timers for >> uh I'm not sure if the density manager h will have a modifier specifically for brick or not, >> but it can it pens it. >> It it does or it will. >> It will. It will. it will eventually. If it doesn't, it will eventually. >> And it's less about timers. It's more about density. >> It's sort of trying to say you have overpop populated this area. So, it's less about like that ship has been there

61. 00:48:43

    for for a long period of time. It's more like there's a lot of ships here. >> Let's remove one. Let's figure out which one is best to remove for an vehicle does have a a timer as well. So, you kind of have these like it's sort of like we talked about validation. You start with a timer, then you start adding a density, then you start sort of building up these um views of the world to kind of understand which areas are problematic. Like, oh yeah, the the entrance to a hanger, >> we have to be super strict. >> Yeah, there's there's a place as we're talking about this something that pops in my mind that I think is also, >> you know, in our process now, not even development process, but let's say let's call it live operations of the game. We're very lean in terms of our live operations at CIG because we're in development, right? And so even though

62. 00:49:33

    we do run a live service, the we try to run a really lean ship, right, on that on that side. And so we don't have an army of site reliability engineers that are looking at like every aspect of like oh shark 25 is an overpopulation of ships at Jackson swap in you know between this time frame this time frame like we >> we don't go in that depth right and so we track over populations across all shards right but we can't go granular into this but it that is an area where we could improve this and detect those things to better tune these values. Typically right now we will tune the density manager values every patch or every hot fix but that is a slow cadence uh compared to how you would react if

63. 00:50:24

    you see oh my god you know on every shard this location is loaded you know let's affect the density manager change right now right now we don't have the ability to do that live and that's a problem that would that would improve the game quite a bit >> there's folks in the it would I don't want to I don't want just missed that. Um there's folks in the chat that are like just make the make the brick ship salvageable or or give towing mission stuff. Those are all >> they are salvageable as far as I know. >> Yeah. And towing missions, but it's those are all game design and and and mission design stuff. That that's not anything that the tech and even then a solution that requires entirely player that it relies on the players entirely to solve is not going to get us where we need. I I don't I don't disagree and I

64. 00:51:14

    know it's it's the intention. You they're supposed to be salvageable. If they're not, they will. Yes, they will be. It's just, you know, having towing missions to to take them away and dump them somewhere is is a great idea and stuff like that, but that can't be the only thing. There still needs to be a technical, >> you know, solution to clean this up. It's the reason I asked the reason I asked if it could separate everything is because and and uh I don't think there I don't think all of them are salvageable. It's it's there's there there's I'm I'm trying to remember an old conversation right now where there were there were rules about why this one could be salvageable and this one >> I know there's been changes related to this as well recently. Um, but like it's in my head, in my head, I'm just like, why don't we just disappear every brick ship right now until we get a handle on this? Like like

65. 00:52:01

    like like like just crank the crank the cleanup on bricked ships that we know don't belong to anybody. Just like they get to exist for one and a half seconds and then they're gone. It's like it's it's it's like it's there. >> I'm just imagining the Reddit post after that. My ship disappeared one second after >> a brick ship. No, but the ship only get No, a brricked ship. Bricked a brric ship is they're they're they're back at they're back at their ASOP terminal and they've they've reclaimed it. So why why can't it just >> vanish now? Now I'm I'm certain whenever I go off like this and whatnot and it sounds so simple and it sounds so obvious and then I get back to my desk and I've got I've got an engineer or something goes, it's not that simple, Jared. There's this and that. there's that there there's that dependency and blah blah and I'm like yeah I'm just a

66. 00:52:50

    layman who had a very simple idea but it just it's hard because it's it's I'm no different than half the people watching the the show. It's just it's just there are times when something seems so easy and so obvious and it's like just do that because that's what my head was screaming that whole time. That's where I was going. just just until we get a handle on it, destroy the brick ships right away. Um >> I always say if it was easy, it would have been done. So by that you have to assume there is reasons that makes it not easy and it's usually like the complexity in the interactions like Ben talked about you sell you sell a gun and then you deconstruct it and then the material can

67. 00:53:37

    be sold somewhere else for something else. These kind of complex interactions between lots and lots of different systems are usually what stops us for doing something that from a tech point of view is literally a flag in the database that says that's bricked. I can treat it as garbage. >> Yeah, there's there's this thing. It's what was the example? We had the example I have to look back in our our chat. There was something in the 4.1. It was it was the one of the old features for the for the hangar module for for the >> iter it's it's in our chat. It's in it's in our chat. Yeah. You know it's it's one of the issues and I said dude we need to remember that it's in it's in our chat >> when I when I when I said I said we need to bring that up. It's I can't I can't

68. 00:54:27

    remember. Um, but recently one of the things that that that that Flammix 481 was an old system was an old system from like six or seven years ago that hadn't been touched as some new piece of content touched for the first time and and it and it and it didn't work. I can't remember. It's just it's just it's just one of those if making games were easy, they'd be done. >> [laughter] >> you know, piv pivot your there wouldn't there wouldn't be a studio that closes every every week in the last two years. so what did so 4A comes out? >> Yeah, there's been other issues. We talked about the catastrophic ones, but there's also other tangential issues. you want you want you want you want to

69. 00:55:16

    you want to I we we've we've got a section coming up where we've we've pulled we've pulled the dashboard and what folks are working on get to that in a second if that's where you were going >> I want I want to stay a little bit more into 4.8 Right. So, um, [sighs] think how I want to do this We put up a thread. We put up a thread. We collected a bunch of questions. Uh, we pulled some of that. Some of that's affected uh, we've used that to inform certain of the of the topics here and stuff. One of the when you go through Okay, [snorts] if you had to put I I I understand that

70. 00:56:06

    nothing is simple. I and I understand that there that there there's probably 14 15 16 underlying causes that all contribute this. But if I ask the simple question, what changed in 48 that affected server performance so much? I'm just on like raw server performance. Sure. Whatever. >> What's the thing that comes to your head? >> Um I might get both of you on. Sure. So don't think you get out of this. It it's it's try try to make it >> for me it's behaviors. So you know there's one thing about running a game server is that the it's kind of an organic behavioral system, right? And so you can have a game server with a version of the

71. 00:56:54

    code, a version of the data running a certain way and then players start doing something different and that completely changes the performance profile. And so I feel like for in 48 most likely my hunch for these six to eight like performance issues that we're looking at, it's related to some imprint changes. So the way people use ships, leaving break ships behind. Are there more claims, you know, than before? Are there other egregious issues that showed up that are creating more claims that would affect, you know, the amount of entities on the server? Are is there a malfunction on, for example, the the density manager cleanup that is leaving items in place? uh or is there just people are going to a location that's

72. 00:57:43

    configured to be very loose on its density and it's somehow the the location that people have decided to go to in this patch. So sometimes the performance profile of the game servers changes just based on behaviors not on like oh we changed something. >> You're blaming the backers. >> No [snorts] I'm [clears throat] blaming the features that cause behavioral changes. [laughter] Uh, and so that for me that's what screams that's that's what comes out of it for me for you're talking raw server performance here. >> Yeah, I'm talking Yeah, I'm I'm not talking just the the raw server performance here. What what what are you what are you thinking, Yans? >> I mean, it's hard not to just parrot what you just said. I mean, it does feel like when I think of 48, there's nothing

73. 00:58:33

    there I can go. Yeah, that those gameplay changes we made could have [clears throat] caused it. The only thing I can think of is we did performance improvements, but there is a large change in how we handle vehicles and the lifetime of vehicles. >> You mean you mean the changes with the imprinting and the the insurance system? So, so, so, so the way the way that the the life of a vehicle in the universe changed it lifespan with with with the breaking and stuff like this and the way that players handle them because of the changes that we instituted. I want I I want to be clear it it's it's

74. 00:59:20

    >> so you're >> you're bringing it back you're bringing it back a big chunk of it, not everything back to the number of ships. I mean, that would be based on what I know of that patch went out. I just don't see us I don't I don't recall there being any larger changes on the gameplay side I feel like could explain it. >> Yeah, we're not we're not talking about there. I want to be real clear. It's it's you folks watching, you have a lot of experience with me and you have a lot of experience with Ben W. you know, less experienced with with with with yen's here. He's not saying that there are no bugs. There are no bugs and we we're talking about raw server FPS things. A bug does not impact server FPS. Uh you

75. 01:00:08

    you you know the freight elevator eating your your your your cargo does not affect server FPS and stuff. The question I asked was what is affecting the raw server FPS? And that answer is he's they're trying to find an answer specifically to that. Um, if it sounds like we don't know the answer, quite honestly, we're still working on it. It it's it's it's it's somebody in the chat somebody in the chat said, "It sounds like Jared's frustrated." Of course, I'm frustrated. These two guys at the table are frustrated. Quite a lot of people are frustrated. You It it's I I I I say it over and over again. We're having the same conversations you guys are having just, you know, three weeks earlier.

76. 01:00:56

    It's like this it's there's this there's this concept that game development is is game construction. You're just supposed to have all the answers and and you're supposed to know everything and you're and everything's supposed to be figured out before you do and stuff. And just the reality is that's not how it works. We're not talk I'm not talking about CI here. I'm talking about the entire industry. I'm talking about how games come together. I'm talking about programming quite honestly. You're It's discovery. It's discovery as you go through it. We always figure it out. We we always figure it out. Like the the the I could sit here I could sit here and just gush about the quality of the people that we have here. It's like you know when I go to a bar citizen, people always ask me like like is it hard to go out and do this? You know what I'm doing right now?

77. 01:01:46

    It's it's, you know, less le less le less than less than 12 hours after a patch went out and and was totally borked and whatever and you've got to go out live and and put on a face uh for the project and whatnot. Is it is it hard to do that? You know, after 318 I had to do the thing. After the freight elevators with Race for Stand, I had to do the thing and whatever, you know. Is it hard to do that? And the answer is no. It's not hard because of the people here. I I I believe in the people here more than anything. It's it's when you get when you get I I spend more time with the people who work here than you know across teams of most people. Everybody's working on their work. My work is the people. So So it's when I lose faith when I lose faith in the people here, that's when you'll stop seeing me, [snorts] but it hasn't happened in 12 years. It's not going to

78. 01:02:34

    happen anytime soon. I love the people here. Uh and they're working on if they haven't figured it out yet, they will figure it out. So you you've already got some tremendous >> for sure. And I think the six to be honest like on server performance the six to eight issues we're looking at now are definitely the source of the problems. >> You do you want to just enumerate what the six to8 issues are? >> Uh I just you mentioned it you mentioned it a couple times >> and it's just why don't we why don't we just use this opportunity? >> Of course I can. Well, I I don't want to dive deep dive into each of them because >> just give me top level >> that's a bit intense if we go down that route. But there's >> uh navigation system disp dispatching of tile configuration changes on planets which is affects our AI use the nav mesh on planets. So there's a performance call out on that that is costing us a

79. 01:03:21

    chunk of frame in the servers like this is purely based on the servers we know are affected by this problem. Um there's >> a lot of digging. >> Yeah. start digging through a lot of data. >> To find >> this is Coleman culprit that you know are going to make a meaningful impact. >> Yeah, I know. Exactly. This is like cross-linking a lot of things. There's um this is a pet peeve of a pet peeve of mine that I keep pushing on and we're gonna definitely resolve that from a software point of view, but there's uh an entity spawner in some of the areas that are affected that are coming in and out of streaming and respawning their spawn set every time. >> This is the NPCs that just go it just spouts out. >> And one example, I have multiple others uh related to this. Uh that's one the

80. 01:04:10

    area component we talked about. So the reg it's a what we call register with external systems is a phase when the components are a component or an entity is coming online when it's streaming in it needs to register with external systems. That's a notorious area that is an issue on multiple components. This one is from the area component is in that place a problem. >> So the first two are NPC related. >> Correct. This one is >> this this one is more engine and uh uh let's call it area configuration related. >> Uh this one bleeds into the zone lock that we talked about earlier. So those are two area and zone lock. So this is really tied to the amount of content in the game and the complexity of that content in some areas because we're also making more and more complex scenes,

81. 01:05:00

    right? you know, like space station and you know, the content we've put out recently, whether it's TSG or rockcracker or like there's a lot more complex layouts in those elements that contribute a lot to this. And so those are part of that. The ship [snorts] cleanup is another big one that we're looking at at the moment. uh which is the source probably of the other two which is the amount of registered entities that have either MFDs uh and or uh the flight system active >> entities that specifically have MFDs. >> So it it's it's it's an excessive number of MFDs. >> Why why isn't MFDs so >> Well, we don't know specifically. I bet that if I went into that area and I look, there would be a bunch of MFDs

82. 01:05:49

    [laughter] that are somehow visible. So, that's been an these components have a visibility aspect to it where if you look at it, it's going to replicate for you, but if you don't look at it, it doesn't. Uh, might be a malfunction with that. We're not sure, >> right? So, the ship is breaking, but the MFDs are still visible. And >> yeah, could be like broken ship MFDs, you know, like we've seen weird things like, you know, everybody knows about the graveyard. Mhm. >> There's been other graveyards in other areas, right? So, there might be a new one we haven't found yet where there's a somehow MFDs end up there. Right. >> So, you have a couple AI NPC related things. Uh NPCs and the way they interact with the surface of of of worlds, planets, and moons. uh NPC spawners, you know, these these things that that that that that load out, load

83. 01:06:37

    in, then respawn their entire their their entire a lotment of a of of AI when they're already out there. So, you got you got some NPC stuff, you got some location stuff. The locations are getting more getting more complex and the area system is not properly calling them. >> Yeah, it's it's an optimization at this point. The functionality is correct. It's just it's slow. A lot of >> A lot of it. Yeah. And and then and then there's ships ship specific one the cleanup stuff and then there's some raw entity stuff that we also suspect yes is related to the ships because the ships have the most >> entities and one final one that uh is uh fire. So there's been some some ships catch fire >> and depending on the size of the voxal grid we've seen the area physics

84. 01:07:26

    properties of how the temperature system interacts with fire is actually needs a big optimization and it's costing us frame time where that's a combination thing because we internally we can't reproduce it but we see the the cost on on some of the game servers that are running slow. Okay. So these these are the these are the top eight >> yes >> is kind of kind of culprits to just raw >> service server performance. We can optimize these eight things we should see >> big gains. I wasn't going to say big but I'll let you say it. You know see big you know big gains in server performance. >> Yeah. Now the thing with everything with optimization right it's it's a fog of war right when you >> you fix that. That's what was slowing it

85. 01:08:14

    down. Then you find the next thing ends up >> until you you know you you get rid of the elephant and behind it the is another one. >> It's it's it's the sevenup can that was slowing us down. You know the Pepsi can would have been slowing us down the whole time, but the Seven Up was doing it first. So the Pepsi never got a chance to slow us down. You you optimized Seven Up and now Pepsi's like my time is here. I can screw you over. >> I can now be the one on top. >> The choice of a new generation. >> Yeah. [laughter] That's how old I am. Did you guys have that campaign here? A bunch of children. There's only one nodding head over there. >> Uh, you know, the same properties apply to crashes, right? Typically, when we do like stability checks on on game servers, which by the way, other than the two big issues that we've had, the baseline stability for 4.8, 8 outside of

86. 01:09:03

    these two events of the missing entity classes and the and the and the Idris problem overall stability has been very uh has been in a really good range overall and so we've been happy about this other than now the stats are terrible because of this. So we have to take a look but the same properties apply to crashes. So when we look at top crashes, you're like, "Oh, we just have to fix this one. It's super high frequency, then we fix it, and then oh my god, there's three more, right?" And then so it's a fog of war. Optimization and and crash stability are like fog of war activity where yeah, you've got you've got seven now. You know what those are, we fix them. It might be the end of it, it might not be. >> You can get clusters as well. And that's the same on the performance side. We might be looking very very closely at the the servers performing poorly and we

87. 01:09:53

    might be fixing things that help those but that doesn't help at all to the other servers because they might be running into completely different different issues like it there might be something that's happening specifically on those servers that's unique to them that brings them to that low rate whereas the rest of them don't benefit as much from that. So you always have to sort of look look closely at the details and react to those but proactively try to look at the big picture and sort of take you take you sort of where are we going overall once we fix that cluster there how do we get all of it to sort of benefit from a smooth smooth ride here >> just a ray of sunshine yes >> yes that's it I'm just >> we want bringing butter >> yeah it's going to be all plain sailing. >> Um, I wanted to touch on in my notes

88. 01:10:43

    I've got because there was other issues in 4.8 that you know come from other optimizations that we've done and or uh other improvements that also contributed to some perception on the patch. >> Um, so there's one thing that we're battling on on the game side is that, you know, we mentioned how we're we're transforming the game from one model to the MMO model, right? And that means scaling systems. When we say that, we really mean transforming those systems to be service powered, right? So instead of being all in the game server and it's all good, right? We transfer these things to be service powered, right? >> So the server no longer has to run transit. The server no longer has to run >> in part, right? It could be transit, it could be other things, right? uh like

89. 01:11:31

    for example missions had to a part of the mission system needed to be pulled out of there to be able to have server meshed missions for example right so there's a lot of work behind that to basically dissect some services and move them to some game systems and move them to services now interacting from the game client with services that are in in in on cloud infrastructure is kind of a different discipline than being a gameplay programmer doing a gameplay feature right let's say I'm doing I'm working on the the you know the main character the actor in the actor team like Yens for example and I'm doing a feature around that that my world is kind of self-contained to the actor >> if I'm also doing the replication side of this which typically we tend to our

90. 01:12:19

    gameplay programmers do both but I know about the replication side of things >> but that's the bubble it doesn't exit that bubble right >> when you [snorts] talk about cloud cloud infrastructure, cloud services that you need to invoke, you know, service APIs to do something, whether it's inventory service, right? Whether it's reputation systems, whether it's friend service, social systems, whatever. Um, that's a different type of programming that tends to always be based in what you know for the layman's is asynchronous programming. So uh not all C++ gameplay programmers are well-versed in the asynchronous aspects. Uh uh no more and more today, but still it's still always a little bit of a a jump. And so we've

91. 01:13:07

    realized that there's a little bit of fragility >> in how some of our game systems interact with cloud services. So the one of the particularities with this is that you need to design for things that can fail. That sounds horrible, but that's the way it is, right? Uh when you visit a website and it's loading assets and one asset doesn't load, your browser is going to retry and you're not even going to see the difference. Right now, the game in some areas, and I'm going to focus on the one I wanted to talk about today, which is inventory, uh is very fragile in its interactions with the backend systems. And so, we had an issue in 4.8 That was from a backend perspective extremely minor. We're talking about, you know, a hundred requests

92. 01:13:56

    out of millions of requests every two minutes would have a problem, right? But the net effect because that was affecting the inventory is that people's inventory would get stuck. And so the effect on the game was completely magnified from what the backend team >> because you can't do anything without your inventory really. >> Well, yeah. The so the inventory has this issue. This I say it's not an issue. It's a property where >> all of the moves that you do in your inventory need to be ordered. They need to be performed in order. And that requirement is extreme because you can't go, you know, put this helmet in the in the on the head and then put this other helmet on the head and take this one back and stow it and then move the core armor. And then if I if the backend systems were completely asynchronous and do this in parallel, you would end up

93. 01:14:44

    with like mixed results, you know, and so [snorts] we could not have prediction, which would be the other option. But then obviously we run into the problem that >> performance [snorts] >> I put the helmet on my head >> and I'm waiting. >> Yeah. Yeah, >> you're waiting for the server to catch up. >> I'm waiting. >> You're waiting. I'm waiting. You're waiting for the server to >> While I'm doing that, I can't do anything else. >> And so the >> anytime, you know, I'm waiting for my helmet to show up. >> And so the the the this minor, like I said, from a backend perspective, 100, you know, >> 100 times out of out of two million is what what you said. >> 100 out of two million >> every two minutes. >> Every two minutes, right? Which was a heartbeat problem. Anyway, we it's been fixed, but the net effect was that you're making moves on your inventory and then one of the moves stays there to the possible maximum of the timeout, which is 8 minutes in this case, which

94. 01:15:34

    that's the problem, [laughter] but >> it's a long time. >> And so now, if you looked at I mean backers at this point that have been with us for a long time all run with some display info and so they see the pending moves debug on the UI and so they can see, oh, there's a move that's stuck, right? Uh, [snorts] and so that's how these things were stuck. If you were unlucky and you did a move in one of those two minute window, these I say every two minutes, but it was really a 3-second small window. If you did a move in those 3 seconds, then your inventory was stuck for 8 minutes, which is crazy in eternity, you know, and so that I use that example first for to explain it because a lot of people ask the question, what's happening when this happens? That's what happened. Uh, now we fixed that issue obviously. So that no longer is the the the backend issue is no longer occurring, but we still

95. 01:16:22

    have that weakness on the client. So that's one thing we need to solve now is we're we've realized Yensz and I have had multiple conversations about this, but we've realized that there's a lot of areas in the game, whether it's freight elevators, whether it's inventory management handling, and I'm not talking about the UI here. I'm talking about what's behind it when you do your moves. Like >> the UI is where you get to see it, but it's it's actually something underneath. Okay. Yeah. Uh, you know, these systems there's a lot of there's we've identified a series of weaknesses in how either legacy systems were built because they were built before these services existed and then asynchronous calls were tacked on in the middle to, you know, get something working. Uh and so we we we've started to develop a um a a charter that we want to we've started to

96. 01:17:14

    analyze every game system with this charter, right? To uh it's basically a very simple uh red amber green kind of tracking, right? Where we can look at one system. Let's say we look at inventory management. We can ask all these questions, right? And we'll with this we will identify all the pitfalls. uh that could that this system needs to resolve. So without having a bug report I can just look at the code and go you know what it's this breaks this principle I need to fix this principle right. One example of principle is player items are sacred, right? So if I have actions to do, for example, I need to turn in stuff for a mission, I will not delete the items then progress the mission. I

97. 01:18:03

    will progress the mission and on success and confirmation, I will then delete and then if the delete fails, I need to have compensating actions to move forward. Right? So that's an example of like the analysis that we're doing. um asynchronous programming we talked about is needs to be explicit for all paths. So we're evaluating all of that. There's multiple other areas in the charter, but that's one of the things we're putting in place to augment the interactions between game client, server, and services so that when things go wrong, and they do, right? That's the nature of of internet packets not being delivered, services not being available, cloud services being there or not. uh that there's no like irreparable breakage >> and that the experience can still

98. 01:18:51

    continue and be resilient. >> Flip says Jared looks exhausted. I'm not the one that was up till 4:00 in the morning fixing [laughter] this thing. You know, it's it's it's somebody what was it ear earlier earlier was like it's like this is what happens. Wait, where is it? Uh, one of our moderators, uh, nuked it, but Wrangle McDangle nuked comment says, "This is what the community looks like when they start to realize the company they give their money to money to doesn't really care about them." Sad. This was this is the CTO who was up until four o'clock this morning, you know, bug fixing, you know, not not not not just not just the directors and the and and and the programmers and he had plenty of people with them and whatnot, but they're up to all hours of the night trying to get

99. 01:19:38

    this stuff going and whatnot. Then they come here and they do this and whatnot. This is not people who don't care. But anyway, that was a comment about how tired I look. This is just how I look. >> I just It's the gray. Yeah. Well, there's more gray now for some reason, but [laughter] um um so this is this is this is interesting. I I want to touch on the charter. I want to I want to I want to touch this this thing that you said while remember it real quick. >> Very quick >> building. So So the system was built for kind of perfection. It expects everything to work the way that the way the way that it wants to. And because the system was built without the expectation of some level of failure, >> it ends up when it gets some level of failure, it fails completely. So you

100. 01:20:27

     actually get >> So in a weird way, you if you build something with the expectation of no failure, you end up with a lot of failure. And if you build something with the expectation of a little bit of failure, you end up with less failure than you otherwise would have because it because it's it's more nimble and it can it can adjust and it can it's what it's a self-healing >> you know self-healing stuff. Yeah, it's there's this there's this idea of, you know, we don't plan for failure or whatever. And every time I hear something like that, we don't plan for failure or whatever. I'm just remembered of NASA NASA in the 60s like these are people who plan for every kind of failure and that's part of what they why they succeeded. So, you know, what you're talking about just kind of reminded me of this this this interesting paradigm shift or or philosophical shift of

101. 01:21:17

     creating for some level of failure so that so that we can be nimble and and self-correcting and then we end up having fewer things like what what what we're experiencing where the whole thing just craps out for eight minutes and stuff. It's a it's an interesting philosophical >> I think yeah Beno is absolutely right as well with challenges from a from a gameplay side certainly if you come from a background of of of of running thing only on your local machine is so responsive right whatever you do is just comes back immediately and suddenly >> yeah [laughter] exactly suddenly you have to work um work on these premises where where your timeout might be 8 minutes later it's a it's a >> you know from a scale point of >> extreme example a very extreme example and you and and you go

102. 01:22:04

     >> uh how how do you and and as well as as much as you all you know from my point of view as a as as from the gameplay side is obviously I still want to always be super super even if it's self-healing under the hood you should still see see everything you do have an immediate reaction in game so that's why we always predict and that's why sometimes things go wrong but we accept that things will go wrong and we roll back but we try to sort of be right more than we're wrong. So like you said, you know, >> be only be wrong a tiny bit of a time, not all the time. So I mean I wanted to touch on the charter just I mean without going into details but just because people are asking you know what are you doing about it right and I think that this kind of audit where we can kind of leverage our programmers not just to

103. 01:22:52

     build the features but all because you know gameplay programmers are going to be in their world when they're doing something and so they're making the feature they're trying to make it feel good you know and then they're like it works >> that's the happy path right now let me unplug the internet right let me you you know, unplug the internet for 2 seconds and plug it back in, right? Or it's like it's basically, you know, on in on the online side, we call it chaos engineering, which is more like let's kill processes and see what happens so that we can recover. uh on the game side uh I think that we kind of need to bring this perspective to the table to every developer so that they can evaluate their own work against you know basically a rag sheet of like explain the longest async chain in this component and does it have every path handled and can it be reached right

104. 01:23:40

     right well just with that I solved half the bugs that we can see and some of the the systems that interact with cloud services specifically are are the areas that that are there now there's way more princip principles and stuff to look at in the area. But that's what we've been doing is to to basically like bring this so that it's not just the leaders that bring it the the sort of the hammer down of like you need to do this this way and this is what we do. It's more like bring it as a culture uh throughout the gameplay teams and the engineering teams. >> There's a we're we're going to get into our endgame here uh and talk about some we've been talking about a bunch of generalized stuff. We're actually we're actually going to get into literal bugs here in a bit and and do and do a little uh uh uh pseudo bug smashers here for a

105. 01:24:28

     little bit. Um before we get to that because that that's our last part and then and then we got to go and whatnot. I want to the conversation that you're seeing here today is the kind of thing that makes it makes me still be here. It's other it's there's this idea there's this idea that happens often that any video game company is a monolith that that that that everybody that works for a video for a video company all thinks the same thing all works the same way all has the same level of experience and and and and and philosophies and stuff like that. And if there's anything, if you've watched the what was we just looked at it the other day, it's like like 1,600

106. 01:25:16

     videos that I've made for this, you know, place in this time. If you watched all of those, you're going to the first thing you should know is it's not a monolith. It's, you know, we're not a monolith. No studio is a monolith. There are are people with different ideas and different philosophies and different aptitudes all working together to try to make this creative thing. You'll see a hundred quotes online about how the the creation of a video of a video game is more miracle than anything else. That we take this time and show that we're not perfect. That we take this time and show that even after 14 years, 15 years, we're still learning. We're still we're still developing. We're still changing the way that we work. We're still we're still getting better at stuff. It's easy to weaponize that against us. Oh, well,

107. 01:26:07

     they don't have any idea what they're doing or I can't believe they're still figuring this out. You know, this like this because I see the comments. I know there's those of you out here that will do that. And there's nothing that I'm going to say or do here today that will stop you from doing that. So, do what you're going to do. I love and I think it's more honest and I think it's more in line with what this project set out to do back in 2012 to sit here to have these shows and have these moments and and be like, "Yeah, we're working on it. We're figuring out we're changing the way we work where it's it's it's we fell behind in this area. We we have we have to we have to change the the way that we program build

108. 01:26:56

     things for uh for for for for for an MMO as opposed to you know what we were doing before and stuff like this. It's I just I I just always appreciate anytime somebody comes on this show and is willing to roll those dice because there will be people there are already people in the chat now. There will be people on YouTube. There will be people on Reddit and like this who will take the things said here today and weaponize them. And that's fine. It happens. It It's always going to happen. It'll never stop happening. But it doesn't stop you from coming here and being as candid as you are. And I I I I really appreciate that. We we really did meet at like 9:00 this morning and for 3 hours talked about, you know, everything like where are we still going to do this show? we it's how do we do this without being as honest and as candid as we are

109. 01:27:46

     and stuff like this. It's it's just like we just kind of do it >> when you know when you don't know what to do. Usually I find the truth is the safest bet. >> Um so with that said unless you have something you want to add. I want to get to the the next the last kind of phase. >> I got one more thing. You got one more thing >> that wasn't my list that I just wanted to touch on which is uh Shardlock. [laughter and snorts] Now >> fact that I'm bringing it up >> by all means. What's shardlock been? >> Well, so I I want to say first that shardlock is a you know it's kind of a community word. We didn't come up with this actually. Uh, but the way we've architected Star Citizen, even though it's an MMO, we operate it from a from an ops side of things like a

110. 01:28:37

     matchmake, like a matchmade game, right? When you come in and you pick a shard or you shard up hop to one of your friends, uh, you're just getting one of the shards that's currently running, right? and the process of bringing your character and untowing it into the shard and then allowing you to unsto items from your inventory down into the shard and putting it back up there and getting out of there, right? is kind of our attempt at not applying a classic MMO setup server setup to Star Citizen where you would create your character on a server which then needs to run at all times that when it's low population we need to consolidate that we need to character transfer or character rename and you know this typical kind of MMO

111. 01:29:25

     grid system right um It's it's it's a company desire to not do that pattern and to instead keep it fluid like it is now. So shard lock is a little bit of a price to pay for the rapid movement between shards. Your character will need to stay in the shard until the hybrid decides that you've you're properly disconnected that you've been gone or that you're coming back or not and then move you back up. Now in forite we did run into some issues for some players. So there's a subset of players that ended up being shard lock. In this case, it was because of a of an infrastructure failure. Uh a small knock on from a change that we made to as part of imprint. But basically these characters were we track where you are with the

112. 01:30:17

     what we call the sto context. And so for the entities that we care about because we don't care about every entity. Your player entity is definitely one we care about. We track where it is in the game world using your stow context. So are you stowed in the global inventory and in what inventory are you stowed or are you unto in a shard and if so in which shard? Um the problem that some characters ran into is that um some players are not very patient with the system when they go back to the main menu. And so they would go back to main menu and say, "Oh, your character is still in the shard, right? Or they would kill the client." Sometimes they do that, kill the client, come back in, and then oh, your character is still in the shard. I'm shard lock. Character repair. Well, the character repair in that case is the one that would cause the problem where now the player would never get

113. 01:31:04

     stowed because the repair would try to repair your character and the anyway, there's a bug there that made it so that now your character cannot be stowed. Uh, and so these players were shardlocked. Others, some others ran into another condition. It was a fairly small subset of players overall that got in that situation, but uh it took us a while to identify the the root cause that that fix been deployed. But I just wanted to mention that shard luck in and of itself isn't a bad thing. It's what enables us to have the flexibility to bring you into another shard kind of transparently, but we need to wait for the process to finish on your current shard uh before we can move you up. Right now, we we've put in place We're I should say we are in the process of putting in place active monitoring

114. 01:31:52

     for players that get untoed in shard and stay there for a long period of time so that we can attempt self corrective actions if we run into that condition again so that this way we don't need to chase the actual source we can actually get some self-healing in place uh but I just wanted to touch on what shardlock is and why it's kind of normal and that it's better to be a little bit patient with it than to jump to character to repair right away. >> Yeah. I don't have anything to add to that. That was >> Me neither. [laughter] Sounds really good. [snorts] It's not short lock unless we have to. >> And if you short, just wait for a bit and you'll be good. >> Yeah. So much of what we've talked about today is essentially growing pains of this at the

115. 01:32:41

     end of 2024. You know, the leadership of the project had a good strong look at everything that we were doing and the direction that we had been going and stuff like that and said, "Is this going to get us where we want to go?" And it was decided, "No, that's we're we're not going where we want to go." Uh, so we switched things up in January 2025 and you know again after Rich went on stage and said this is the road to 1.0. We've been on this road all of these major content drops storm Stormbreaker to ASD to you know rock and these are all locations in the 1.0 no story. They they they have content that you're playing now and but these are all part of this larger road map of the ent of the of the major story of 1.0 that will

116. 01:33:30

     come online with that and the gameplay that's designed to take you through the five star systems as like that. It's we are actively building this 1.0 out in front of everybody and stuff and we made more progress in the last 18 months than we did probably in the four years prior to that. I see if I get in trouble for saying that but it's what I believe. Um, >> well, you always want to accelerate. So, >> yeah. Well, you know, it it it's it's it's just and the results the results have spoken for themselves. There's there's this thing there's this thing they don't really talk about most much. It's uh concurrency and daily active users. Concurrency is how many people you have online at any given time. time and day daily average users is how

117. 01:34:20

     many you know individual people come in you know and interact with your game during the day. The normally these numbers are not revealed publicly. It's it's it's just it's just if if you go and you search for for World of Warcraft DAU and whatnot or or whatever you're you're not going to find that. You're going to find a bunch of you're going to find a bunch of um you're going to find a bunch of uh estimates and stuff. Uh, one of the we're discussing internally right now is actually exposing that because even with we spent a lot of time talking about the problems in 4.8 and whatnot and 4.8 had its share of stuff. Obviously, we don't need to we literally just sat here and

118. 01:35:07

     and talked about it, but it also had some tremendous successes. the daily average users for instance in I'm waiting for the numbers. I I just I just sent the request. I just sent the request. Thank you. Thank you. Uh Michael Smith and whatnot. Oh, I just made him an accomplice. That's it. All right. If I get in trouble for doing this and whatever. So versus 2024 numbers. Let's just go by month here to to to to to show what this progress because it's it's one thing for me to say that it's been the great 18 months and whatever, but if you don't have data to back it up, I'm just bullshitting. I'm just saying words. So, here's here's the in January

119. 01:35:56

     2024 versus 20 uh versus 2026 saw a 30.1% increase in daily average users. 30.1% between what we were what the number of users who were logging in and playing Star Citizen before and after this last and and as part of this last 18 months of stuff in February it in 2024 to 2026.8% increase in daily average users and stuff like that. Uh March 50.2% 2%. April, 59.8%. May May's always a big month for us. It

120. 01:36:44

     was a modest increase of 14%. You because May, you know, May and November always draws everybody back and whatever. Never had a problem getting people in in May and November. And then in June, you know, which which is when which which is when 48 and like this, a I'm sorry, 55.4% increase over o over over 2024. A 25.1% over 2025 and whatnot. So, so it's been 25 and 25 over the la over the last two Junes and whatever. The game is unequivocably going in the right direction. It's the amount of users we have, the amount of people that are engaged. This isn't new signups. These are people who are actively playing the game is increas increasing every day on average by 50%.

121. 01:37:37

     Which is a growth that Star Citizen has not had at any point like this. And I imagine some aspect of that has to affect all of these issues. You know, issues that were this big before become this big when all these people go in. But the reason those numbers are improving is because of the work that you're doing and you're doing and and and and Rich and Rick and Ian and all of those guys are doing to make Star Citizen into the thing that it was always supposed to be and we weren't going in the right direction like this for the last 18 months there and it it's proving out in the numbers and whatever. So, it's my heart's I I don't know if you can you can hear my heartbeat right now. I'm way out over the ledge at this moment. I

122. 01:38:24

     don't know if we're going to sit here and we're going to we're going to talk about everything that was wrong and whatnot and stuff like this. I I don't think it's fair without also saying that these guys out there, yeah, they've got some work to do. There's some bugs to fixes. There's some performance optimizations that are needed, but they are killing it. You are killing it. And so the numbers speak for themselves. So, thank you. And we'll we'll see if I back next week for another show. Um, >> I mean, there are a lot more players. That's for sure. >> There are a lot more players. There are so many more players now than there was a year ago. Um, and it's just I I feel like that has to play that that that has

123. 01:39:13

     to say something about how successful we've been even in spite of all this stuff. Even in spite of my very clear frustrations. >> Poor poor uh Seb and and and and and uh Lauren and whatnot. They they they got a message from me about an hour and a half before a show and says, "Strike the set. We're doing something different. We're not doing the show." So it's like I I just and THEY'RE JUST LIKE WHAT? So thank you for for for allowing me [laughter] that said before we go before we go we have I I have a letter I have a letter I want to read and then we're going to get into a thing. This is from uh the lead producer on Star Citizen. So this is this is the

124. 01:40:01

     producers are at all levels. You know you've seen directors. You don't see a lot of producers. You don't see a lot of producers on on the shows and whatnot usually because they're too busy running around making sure everybody else is doing their work and like I don't got time for you. Uh we tried to get Phil here today for this and like and again too busy too busy going through and making sure everything but >> uh he'll say about me. >> Wait [laughter] a minute. I'm also really busy. >> You're almost never here. So you're clearly very busy. Honestly, it's been a while that doesn't seem to have anything going on. [laughter] That's why he's up in the middle of the night. >> I know. So, I'm going to I'm going to read this letter from film that's going to go we're going to go into our last section here. Okay. So, after the release of 480, we reassessed our bug prioritization process. We're into the bug phase of the show here. Instead of

125. 01:40:48

     having priority determined by the develop by the development team and validated by QA, we handed over the selection uh some elements of the selection and prioritization in certain categories of bugs to the player experience team. The player experience team is internally always pushing and advocating for issues that players are most impacted by. So, this should in theory address a perceived disconnect between us as developers fixing a lot of issues and players feeling like little has improved. Now, that's not to say that the priorities in the past were detached from what players reported on issue council, Spectrum, or Reddit, but clearly things weren't working as well as they could have been. With 4.9, we will work we will work through a newly refined list of issues affecting instance hangers, freight elevators,

126. 01:41:36

     quantum travel, star map, inventory, docking, exploits, login flow, and issues that falsely destroy your ships and more. On top of that, we're tracking a longer list of quality of life issues with new player that new players new players report regularly. Many veterans know a lot of convoluted workarounds to some of these issues, but really they shouldn't have to deal with those to begin with. And because we want to ensure Star Citizen is as new player friendly as it can be, we're tackling a wider range of common problems that impact this first hour or so of any session. So, in addition to the things we're going to go on, there's a there's a new effort about things that specifically affect the very first hour. There are about 80 80 issues remaining on our priority list. And while we tackle a lot of them, while we will

127. 01:42:25

     tackle a lot of them for 4.9, we want to avoid just playing whack-a-ole with some of those bugs, fixing a lot of bugs is not the same thing as fixing the root causes behind the symptoms that appear as bugs. For instance, freight elevators have seen around 70 or 80 bug fixes uh this summer. But due to the reactive nature of that endeavor, players don't feel like it's nearly as stable or reliable as it should be. Now, freight elevators will have a lot of interactions with other freight elevators that have a lot of interactions with other systems will need to be made reliable, but they're a core gameplay mechanic and must be stable at all times. I misread that that sentence, but I'm going to keep going. On the flip side, we had a similar situation with Transit in 2024 and early 2025 where we fixed a lot of bugs that didn't seem to make a difference. We then invested that time into building better debugging tools and self-repair

128. 01:43:12

     flows. That's the self-healing I mentioned before. and yielded a much more stable system even though on paper way fewer bugs were actually fixed. We want to repeat this methodical approach with other systems like freight elevators, hangers and more. For that reason, we don't expect everything to be magically solved for 490 and I wouldn't read it if he did. Uh we will we will continue this initiative beyond that into 410 and beyond. But we're more confident that in this new trans the same approach that we use for transit to bring these systems in line with the scale and complexity requirements that Star Citizen has in 2026. Now what we're about to see or what we're about to talk about see there's nothing on the thing is what people are working on today. Uh do you have your dashboard up? I do. You want you want to hold your dashboard up for just hold it up for this camera

129. 01:43:59

     right here. Stay on this camera. >> No, no. Stay on this camera. There you go. Hold Can you hold that? Hold your computer. No. No. That camera right there. It's closer. Little bit. >> Can you see it? >> Forward. Forward. >> You see it? Oh, that doesn't really look like anything. It's exposed. Wrong. Okay. Switch that one. >> Switch to Beno. Ben's closer camera. >> Ben's camera. >> Ben. >> No, it's still it's still overexposed. Okay. Anyway, this is a dashboard of what people are working on today. So, this this this was this these particular bugs were prioritized in line with the player experience team. This is a new initiative inside the basically like as Philip said not just what issue council says and what dev thinks but we're actually adding a third kind of pillar in here and that's hey the people who

130. 01:44:48

     the people player experience team like the community time like the community team they're both involved in this it's you are the ones who are seeing what they're anecdotally saying on spectrum on Reddit on Twitch on YouTube and stuff come in with that uh with with that um um atypical information, the anecdotal information, and so we can apply the issue council stuff with the developer stuff, with the anecdotal stuff, and find a new way to prioritize which bugs get fixed. So, we're going to try this. We're going to see how it works and to do more than just say, "Hey, we're going to do it in a way." We're going to tell you, we're going to go through and tell you what people are actively working on today, and then you can tell us whether we're on the right track. Benoa, what do you got?

131. 01:45:34

     >> Well, so yeah, I thought we would take a look at a couple of those. So, I've got all the uh categories that you listed and uh >> Yeah, give me the category first so I can prime my >> Right. Right. Prime prime your mind to the right side. Okay, good. So, uh hangers. Going to start with hangers. >> Oh, that's a tough [clears throat] start. Okay. Yeah, I'm ready. >> Uh so, now I'm going to I'm going to sort of pick the ones that I think are interesting in the list, right? uh in each segment. So we're going to take a look at this. So >> these aren't in any particular order. This is just this just some of they're being worked on today. Right now exactly. So um PU multilocations instance hangers exterior hanger doors remain closed while interior hanger door opens. >> So that's when your hanger interior doors open but the gateway outside is

132. 01:46:21

     still locked out. >> Yeah. So they belong to two different parts, right? You have the exterior and you have the interior. The interior is yours. you you can control that in your instance. >> The exterior needs to be synchronized with your interior door. So, what most likely happens there is for some reason your interior thinks you've been given the right to this gateway or you could have been given a right to a different gateway and there's an exterior door opening somewhere to a nothing. >> And and and your door is opening to a closed door. >> Right. So there's a few things that could happen there and and and normally it's that sort of handshake between the exterior and the interior which >> via the notorious ATC. >> Notorious ATC. >> I see. I see Jockin in chat says that's

133. 01:47:09

     a rare issue to happen, but W if you fix it. It's rare for Jockin, but we have the luxury of having logging for for every single player in the game. And while you've been lucky enough that it's rare for you, it is not for many others. >> But this one goes with his little brother. um overlapping hangers or player reposition when landing in hangar ATC given but seen as trespassing in someone else's hanger results in being repositioned which I think is uh sibling >> mostly. >> Yeah. Yeah, definitely. Because you're looking at a situation where you think you you think your interior is being given a certain gateway. You're heading there. >> But it's actually someone else's >> castle. So you're like >> Yeah. Not good. correctly trespassing but obviously incorrectly been told >> the trespassing system rock solid

134. 01:47:58

     [laughter] case. >> All right, >> the trespassing system it it's like Johnny on the spot. Hey, you're in the wrong hanger, dude. >> But you're not supposed to be in the wrong hanger. >> And then and then the third trifecta in this segment, which if you put them all together, uh is a players can get stuck at repositioning player independently after getting imprison impounded, sorry. And so if you get impounded, there's a chance that the server will never uh complete the repositioning that's supposed to take you out of your hanger. Most likely in this case, one of our charter elements, which is that there's an asynchronous call, waiting for a call back that never happens. There should be a timeout so that it can be retrried and complete. In this case, get stuck there. >> Well, it could also be validation. for some reason someone has not given us the or the reposition position that were

135. 01:48:48

     created are no longer in a valid position because I think one of the rules is you will not be repositioned until we know there's floor underneath you >> right >> so until this floor >> and and that's a learning because I'm sure that we reposition before without checking that and >> that led to some interesting ones corrosion on Twitch says I entered my hanger last night and the entire hanger despawned and someone else came into the same anger as and I was EVA in space on the internal geometry of the station. >> Sweet. That is definitely related to this. >> Yeah, it sounds like his reposition positioning position wasn't amazing in that case. Also does does disprove my floor. >> Cuz he clearly did not have floor, >> right? No. Okay. Um this other refresh is interesting. Okay.

136. 01:49:40

     Uh maybe there's been progress. Okay, so uh next one is uh the one my personal hanger uh pet peeve is retrieving vehicle while another occupies the hanger platform will cause the retrieved vehicle to instantly spawn and clip through the vehicle occupying the platform. >> All right. So So Fs in chat if that's ever happened to you. You you can keep doing what you're what [laughter] you're doing. >> Okay. And so this is basically typically >> that's what I thought. >> Thank you, Chad. This typically >> happened to anyone. >> Interesting. Good to know. >> So typically this >> happens when you use the ASOP terminal outside of your hanger and for some reason your hanger was stowed with a ship on the pad,

137. 01:50:28

     uh, which normally shouldn't happen, but in this case happens. And so there's a lot of missing checks here where basically we don't check if there's a vehicle there. And so it gets spawned into a direct intersection, which is also very bad for the server because it cannot resolve that. And so one of those two ships is either going to go uh the dreaded pink uh destruction if you remember that. >> Well, there's still got Fs in chat, so I'm going to say that was a high that that's been correctly identified as a high priority. >> All right. What else you got? Uh again, in case you've nodded off after two hours here, uh we're going through the bugs that are actively being worked on [snorts] today, not like today ethereally, like on June 18th right now. They're literally people on these things

138. 01:51:15

     right now. >> Uh player loses access to persistent hanger at primary residence is a [snorts] classic one. My character is actually in this state right now. Uh >> where is your hanger? Have you found >> area 18? >> Where have you as is it actually still there? Yes. >> Or is it gone? >> No, the the hanger is actually there, but they didn't respond in it in my case. >> But I think that's some >> in the case of these for most players I've discussed with the the hanger is actually either stowed in another inventory somewhere else for some reason. So that's the hunch so far. >> We'll see what the team actually comes up with, but that's that's one that we definitely want to solve. Now, I've got more on the hanger side, but I I figure in in the interest of time of time, I would keep going with the other categories. Uh, we've got login flow related issues. So, we talked a bit

139. 01:52:03

     about shard lock. That's one of them. So, basically being a bit more communicative about shard lock and no longer having shardlock at all for players that stay in the shard. Um, 64,08 spawn resolution error. uh and his little brother 64,0006 disconnection again resolved player location. Um I can speak to this one a little bit. This is typically a symptom. So, we're if we fix this issue specifically where the display will change, but what actually happens is that when you connect to a game shard, uh the replication layers or the hybrid that's in front of it is going to coordinate bring your character into the game world, but it doesn't know to start which of the 10 servers he's running, your character should be on. And so he

140. 01:52:52

     needs one of those servers to help resolve your location so that we can unsto you in the right area in the tree and then end off authority to the server that's managing this this region. The way this is done is that the the replication layer will hand this work off to any of the game servers on the shard. Now, when we have addresses blowing up servers left and right, there's a very high chance that the server this will be ended up to is dead or is in the process of dying and is rebooting. And so what happens in this case is that you the player will get to the end of the context establisher task, the CCTS, and he will get a 64,08 or 12 error. And so we need to make this a bit more robust on that side so that the

141. 01:53:41

     either we know which servers are alive completely and so that if a server's doing a server reboot we don't assign them a resolution task that's would be one thing uh but also trying to streamline that right so that it's a bit more efficient >> if [snorts] a server is dead we shouldn't assign to >> uh small one I wanted to touch on here was friends list showing zero out of one right instead of the player counts per shard. Uh so we're we're rejigging this bit. It's going to come back. Uh this is a temporary thing. >> Is that a is that a priority fix though? I mean what what is what what is that >> in the login flow section? It is >> all right. Well, >> it's been reported. >> Sure. But but it it's part of this initi part of this initiative is is to make sure that people are on the most, you know, correct important stuff like this. >> Uh give me one for tell me somebody's working on docking.

142. 01:54:31

     >> Correct. Let's talk about docking. Uh the fir very first one. Okay. >> Unable to dock using ducking colors. >> That is problematic, isn't it? >> Could argue that. >> Yeah. Uh we looked at this this morning and it was actually resolved when we were talking. It is now reopened. So we're going to have to dive into that. Uh but uh basically docking is is a is a subject that's always coming back. Uh and this patch has been also contributing to the overpopulation of ships because players would just you know basically land on pads and leave the ships there or you know uh where there's no impound and so that's a big issue. So first one is unable to dock using docking callers. >> So I can quickly just go on. So when you

143. 01:55:19

     say it's resolved and then reopen, what's what happens there is the developers to [snorts] the best of their abilities found the issue, >> the fixed issue, they've tested the issue locally, then gone. >> It's is solid. >> Mhm. >> I know how to cause the error. I put a fix in. My fix has been validated. I've now resolved it solve the issue. But obviously >> in this case, what sounds like the future is not working at all. Clearly there is a situation that either they didn't predict or that doesn't happen on a local machine. Again we talked about >> you know you don't have >> that works on my machine but the second it got up to network validation it fell down. >> That said though I mean I mean I will say that we have a very very impressive

144. 01:56:07

     setup for locally running what is >> a replica of what we play in live which is amazing. It's an amazing system. Oh, you've got you got there there's there's an entire pseudo live environment right here in the in the office with all the CI dev kits. >> Yeah. Yeah. It's pretty cool. >> Um give me >> but yeah unfortunately in this case clearly >> it is there one or two more for docking. >> Yeah. After manually docking a vehicle there is a chance for the docking bridge to lose collision causing the actor to start the EVA. most likely close to another issue we've seen on the command module about managing physics skip list which is how we you know elements we want to skip from collision in certain cases from a given

145. 01:56:55

     perspective uh yeah probably related >> is this when you you're trying to dock and it ends up just eating you out into space. >> Yeah. Um, so someone you don't want the docking collar to collide with your ship and cause lots of problems there, right? So there's there's a period in time when the two are trying to get together and >> just people just need to get good and you know stop >> um >> they have to dock perfectly. >> Uh cargo services will not respond to hails making players unable to request cargo loading zone related to ATCN docking in this case. That's another one we're looking at. Um >> uh freight elevators. >> Freight elevators. Let's skip to that. Um so hauling missions have no cargo to pick up at the collect location. So this

146. 01:57:46

     is basically when you get to your your mission location and you know it's like hey pick up this addonite from this location and you get there and there's nothing to pick up. So >> there's somebody on there right now. >> Uh yes. >> Again everything we're talking about in this section has somebody actively assigned to it working on it right now. >> Yeah. Uh that's a bridge between mission and freight elevator which is interesting which is you know requires specific people in this case. Um freight elevator is not working and displaying an overload error preventing cargo pickup. Do you want to speak about this one because I think we talked about this to death. >> Well I think we we talked about the overload error is is one of those that's a little bit too generic that can imply many different things. uh similar to the ATC or the ASOP saying your ship is unknown, >> You know, it becomes like becomes a bit

147. 01:58:34

     of a catch all and and >> people pull up and it's it's like they pull up and it looks like it's like a small box or whatever, but they're getting an overload error. The the system thinks the small box is a much bigger box kind of thing. And like Philip said in the in the letter you you've so eloquently read out obviously we fixed tons and tons of bugs on freightators as part of of of a ongoing initiative. >> Um and we probably fixed lots of overload issues as part of that but because overload is a catch all you know >> unless you unless you get more granularity. >> This will be affected by people throwing the water bottles. I was going to say, yeah, because that was that thing that was that thing a year ago when people would put the bottle on top and just kill the entire >> Yeah. Well, it's a variation of that really because the the freight elevator

148. 01:59:22

     looks at the volume, right? And we'll make sure that it's got stuff that you own in it and there's no overload on it. So, that's one of the things. Um, >> yeah, it does have good good cleanup to the point where it was cleaning up layers recently. See, that's that's an example where like there's another bug further down, which is you have a chance to lose your cargo by activating the freight elevator kiosk, which is basically a malfunction of that cleanup to prevent the overload. So, you know, that needs to be solved. >> That I like you just you you just casually mentioned the freight elevator eating eating cargo. >> That's the worst. that that's >> well for me that would be near the top because that goes against >> I mean I I I I I don't need to be pick Wiccolo's PR agent here but the reason there's the reason there's an a reason for a lot of the bad sentiment about

149. 02:00:11

     Wiccolo is because the freight elevator is eating the cargo. Yes. >> And you know and it's easy to say well this is a Wiccolo problem. It's not a Wiccolo problem at all. Wiccolo's Wiccolo is a disembodied head and torso behind a So you don't think this slate of hand that he's going like I'll have something that >> and whatnot. It's it it's this freight ele it's this freight elevator issue is this is this is >> or maybe Wiko runs the 0000 graveyard and we didn't know about it. >> Well secretly my boss and he's the one that's threatening to fire me every time. >> No [laughter] is except for frustrating. >> Yes. This this violates the first principle in the charter we're putting in place which is player items are sacred. >> But it does also highlights the in the same in the inventory. there's this fragility in the um uh transition between the physical world and the

150. 02:01:00

     virtual kind of UI world that that a lot of the sort of inventory lives in if it's sort of back end visualization of the back end. It's a virtual item in a way and the physical item. I I don't want to spend we're we're we're we're over our a lot of time and stuff and you you literally have an airplane to catch uh to go back, but it >> I sorry the the the freight elevator disappearing cargo thing is is my >> major major thing. And again, when when I do my you know simpleton's hat >> on it, it seems like it would just it's like okay like like you got an order of operations. You've got to delete. Shouldn't the delete just be the absolute last? Shouldn't you Shouldn't you just complete every other action that you're supposed to do? Your mission

151. 02:01:48

     checking blah blah and then delete is at the end. It sounds like deletes just happening too early. >> Yeah. Well, in this case, it's a function of the cleanup. Yeah. >> Yeah. It could be the cleanup, right? It's something added >> to prevent another issue now causing an issue that's worse than what we were trying to prevent in the first place. Now, we come back to That makes more sense. Okay. >> Again, what Philip said in his letter, this is this is one of the places where we want to go back and kind of go like, right? So, it might mean that in short term, you're not seeing like bing bing bing bing bing, lots of like individual fixes, but we're going to go like what's actually at the very bottom? Where is >> So, it looks like it's an issue with with Wiccolo, but it's actually an issue with the freight elevator. It's actually an issue with the cleanup system. >> Exactly. So, can we make the cleanup system? So, the cleanup system is giving

152. 02:02:36

     the freight elevator a bad rep and the freight elevator is giving Wiccolo a bad rep. And the issue is Okay. >> Yeah. Well, maybe we can find a way that we don't need the cleanup at all. >> All right. Uh, just one last one on freight elevators before moving out is uh freight elevators being inoperable, >> going inoperable [clears throat] completely. >> Uh, which is still a thing, which you know, we've had multiple fixes for, but you know, that's one that we definitely need to lose out. >> Should I jump into QT and star map? >> I was going to say, let's do QT and star map. >> Mhm. So, uh, quantum travel route resets after setting route. That's, uh, >> it's annoying when the satnav does that. >> That's, yeah, not good. >> It's so, it's like, it's not exactly, okay, I was about to say it's not gamebreaking. It is gamebreaking because

153. 02:03:24

     you if you can't get to where you're going, that's pretty darn game-breaking, but it ends up working after a second or a third or a fourth attempt and whatever. That's which is what I mean by by by by the thought was it's not game but it's so it's just one of those things it just makes things look so bad. It's like it's like >> yeah it needs to be flawless >> the the because it's that's again a traversal system right the star map becomes like the routing system. But >> yeah, and I think I think you know again from from gameplay point of view doesn't want to want to sound super apologetic or anything but that routing is is one of those that >> it's is written for a few technology iterations ago now. So um >> we got some interesting changing coming with the future. under was really I think

154. 02:04:10

     >> yeah little seemingly simple things like small things not simple small things like that just really erode confidence >> in anything like if I can't even plot where I want to go and you can't get me there it's like it's just so well so the first one is it resets um next is doesn't plot around obstacles Uh next is while in station destination on opposite side won't set route. those are the main ones. I think there's there's more obviously being worked on but the those are the main ones I wanted to call out. The uh on QT tunnel exit can be in the same system as entrance where you basically go through the

155. 02:05:00

     quantum jump successfully. Right? You're not you're you're piloting it right? You're doing all the things right. you get to the end and then you're back in standard. Uh, obviously that uh that's not great. So, we're looking into that at the moment. I mean, hasn't, forgive me for being blunt, hasn't some version of that bug existed since we added jump points in 4.0. >> Uh different different source cause though, I believe. >> Yeah. It's one of those things. It's one of those things where you fix that cause and then you fix Seven Up and then Pepsi shows up and you fix Pepsi and then you >> Yeah. So, >> well, that's why that's why, you know, coupled with this effort of of prioritizing these issues, there's also the robustness evaluation we're doing across the board, right? Because it's not just about getting the bugs fixed

156. 02:05:48

     and then they break again. It's about trying to change the the culture behind how we fix them and then sort of what we check in the periphery of the fix, right? So make sure it's root cause analysis and that it's not just a patch on top of another one, right? Uh so that's quantum in general. I got more if you want to keep going. >> Uh star map. Oh, I guess that was star >> That was star map. Quantum. >> Um let's we haven't we haven't said one word about exploits. We haven't said one word. >> Okay. Well, what we did te technically the part where you could buy the the weapon or whatever item it was and then you could break it down and then you could sell the broken down materials for four times what >> that's a that's worse than an exploit. That's a that's legal and legit in this case because we put those prices in, right? So, it's not an exploit in this case, but we do we are tracking several

157. 02:06:40

     potential exploits that you know that range. We basically we're not going to go into details into exploits because that's kind of an area that we want to too public about. You don't want people to be using those exploits if they are confirmed or >> Uh but uh >> well we benefit [snorts] we benefit from those people not knowing we know what we know. It lets us track it lets us learn more about it. It lets us find not just that cause of it but that cause of it and that cause of it. You know, it's it's very easy >> it's very easy to be frustrated when you see an exploit go and continue. It's like, why the hell isn't CI doing something about it? When it's like, well, no, we are. >> We just we need we're we're baiting the hook. It's like if we smack this cause of it now,

158. 02:07:27

     >> then we we never get to see this one and we never get to see this one. So, so, so there are there are reasons why we leave these things going for perhaps longer than some people would prefer because it's how we find out how it's happening. It's how we find out who is doing it and and like this. So, it's just it's it's one of those ones where I I I could see why it's frustrating, but it's actually that's actually the process working. Uh so we like we're tracking exploits across different range of types of exploits whether they're environment exploits or related to the economy exploits or duping or or or adjacent duping or you know mission bypass things like that. So we're we're tracing a bunch of those. We're tracking we have a great team doing this work by the way like that support the developers across

159. 02:08:16

     with all our crazy questions about what who and when and how and so these guys are doing great supporting us. We did not talk about inventory. We kind of skimmed over it. Um uh >> I know we could do a whole hour on inventory go right ahead. >> Yeah. But uh so items that are being worked on today like you mentioned. So inventory becomes unavailable and contents of backpack station loot inventory is empty. Uh so this player reporting I've lost everything. Well the so from our findings so far you haven't lost everything. It's really uh the inventory is not loading it for some reason and it will come back. Uh some of the backend modifications we've done recently might are going to help on that front. Um when double clicking an armor item in the inventory and the adjacent armor item is also equipped. That's a Russian [clears throat] and 48. Uh and

160. 02:09:04

     so that's a bit of an issue because uh you know click a core armor and installs the helmet that's above. Uh I know the team's working on that. Uh the undersuit icon slot in the player inventory will be missing after being sent to Kletcher. I think that's by design. No, I'm joking. Uh but yeah, that's an issue. So if you go to prison, >> he's saying criminals don't deserve undersuits. >> There's no undersuits for criminals. But no, I'm joking. So uh we got to fix that. Um what else are we looking at here? When opening the inventory and the player has a lot of items, it can be delayed opening it with no items available or missing icons. So icon rendering in the inventory has had a lot of work in 4.8 to speed up the rendering. We're still not where we want to be. Like we've optimized the way the

161. 02:09:52

     UI is actually rendering them, the way the actual PES is constructed. Uh but now we need to really look at the at the RT graphics side of this to make that super snappy. The goal is really you press I, it's all there. There's no no delay at all. And so we're not there yet, but we do want to get there. Now, this is even more pronounced on very large inventories at the moment. So >> yeah, and if if the delay for whatever reason is because we we haven't yet been able to populate your snapshot of your inventory, there should be an indicator. >> Yeah, >> I think they just got lost in transits. I know the guys were talking about that. >> spinning icons. >> Spinning icons. Yeah, because the state needs to be displayed as well. That's the That's the key. >> ideally,

162. 02:10:42

     of course. >> Yeah. Well, I mean, the icons have been kind of >> questionable for years, but you're talking about the the recent >> the, you know, the recent it's it's a different problem with the icons than the issues we've had with the icons before, >> It's not about the look. It's about the timings and getting them on screen. >> Ben wants them >> like that. Um, there's one more that I want to call out on inventory, which is like there is a the way it's written is item lost when equipped by mouse click, which is a bit crazy, but there is a there's a case where items don't end up where you think they end up. And so we we people think they're lost. We haven't we don't have evidence that they're lost yet, but they

163. 02:11:29

     go they don't go where you're supposed to go. Yeah, exactly. >> I'm pretty sure that is actually the definition of lost. No, >> lost. >> I expect it to be in this place, but it's actually in some place. >> Normally, it can be found again by doing it again because it goes where it goes is the same. So, your next item will go to the same place, which will then replace that item and either send it back or put it into your hands when you realize it. >> This is again about the going from the sort of virtual item to the physical item. We we we we put it on you but like hidden and then you play out the performance of putting it into your hand from that state. >> Rather than just popping it into your hand. >> It's that fragility in the the sort of more complex

164. 02:12:16

     >> scenario >> when you have multiple a sync systems they have to go through lots and lots of steps. Uh I didn't go through most of the first hour hardening that we're doing as well, but there's a lot of issues that are being tracked from the first hour of gameplay. Like basically you're a new player, you come in, what's your first hour of gameplay? So, uh this goes from ship combat behaviors, uh rubber banding, we talked about server performance here, that definitely is related. Um uh uh turning off the power and back on the power on the selected starter ships doesn't also turn on the radar. >> Uh you know these are things these are a bit more less critical but critical to the first hour of gameplay. Right. So that gives you a bit of an idea.

165. 02:13:06

     all right. Somebody somebody in the chat somebody somebody in the chat as you know we were talking about exploits. It's like it'd be easier if you told us what an exploit is and isn't whatever. And I' I've been I've been thinking about that. It happened like five minutes ago and whatnot. It's it's it's sticking in my head cuz it's like we can't give a list of exploits because that just draws a bunch of people to those exploits and makes the problem worse. Like that's just that's just not something any online game does. And there there there there's a long established history predates CIG why you can't do that. And I'm like so so and then sometimes sometimes I'm known for being a little

166. 02:13:54

     snarky >> really >> surprise. And and so my brain was like, it's like if you get that if you get that little excited feeling in your in in your stomach, like you're like you're like you're getting away with something that that somebody would be upset with if they found out about it, that's probably a good indicator that it's an exploit. And if you don't have that, then I'm like, it's like my brother never had that. So I'm like, that's what I wanted to say. And like, "No, my little brother never possessed that." So, I'm like, "Well, that's not going to work either." So, I don't know, guys. I I I don't I don't know. I don't know an effective way of of explaining what is an exploit and what isn't an exploit. Just it's it's just

167. 02:14:43

     it's just try to utilize whatever judgment you have. If you if you don't know, ask a friend maybe. You know, it's it's like I said, it's I I was I was already with that snarky answer and then I remembered my own brother would never maybe is my brother a bad person? I don't think so. >> I maybe >> Anyway, I'm way off the reservation there. [laughter] Um so yeah, everybody, that's our show. Uh uh uh uh I hope I think about how I want to say this. We threw the old show away. We did a little bit of it here at the end. We did what you just saw was a little bit of what we had in the in the old show and whatnot. Uh so I apol it it's it's I hope you

168. 02:15:33

     don't mind us taking the opportunity to to have this conversation with you again. This is part of our schedule stuff. You were you were you were here. You're literally, it's not no coincidence that you're over here and right now to do this. It's part of our our schedule stuff that we set out to do um back at the beginning of 2025 and stuff. Um I really appreciate that we get to do this. I really appreciate that you guys take the time to do this. Uh I appreciate that you all will sit through and listen to this. And I mean I already said everything I need to say before. for those of you who choose not to weaponize it and choose to recognize that we are human beings who are who who will make mistakes but you know are on the whole are doing are are doing work that I'm incredibly proud of. Um I appreciate you. Um that's it for our

169. 02:16:22

     show. Uh I think we're back. I have no idea if we're back next week or whatn not, but uh uh after that um I will say all of this talk all of this talk right now is meaningless without action. It is uh we've been on this cadence. We've been on this cadence. Uh if if you follow the cadence of patches this this year, uh you you you you should know that 49 is scheduled for the middle of July. It was the end of January and the end of March and then in May we switched to the to the middle. The next one is June. It follows the same cadence as last year. We just had ones in the middle. So the

170. 02:17:10

     cadence has been in place for 18 months now. The next patch is scheduled for the middle of July. Uh, I don't have specific answers for you yet, but I know there are conversations going on about whether that should happen or not, whether whether 49 goes ahead as planned, but we remove content from it, whether we keep the content in and we push that release back further to give it more time to address. The only thing I can confidently comment on right now is that the people in this room and the other big decision makers that are having this conversation right now do not want a repeat of what just happened. So, we're going to So, through one method or another,

171. 02:17:58

     whether it's removing content from 49 that we don't think is ready or whether it's pushing 49 back until the content is, I don't know which direction we're going to take, but none of what we've talked about today is worth anything without meaningful action. So, uh stay tuned to find out which direction that goes. Check in with your roadmap roundup or I'm sure there'll be an announcement from the community team or whatever about which path uh we go. Uh, but that's it. That's our show. Thanks for watching. I'm Jared. That was Ben Wang. That was Yens. Uh, uh, you have a I think your mom was in chat, Yens. There was some, there was at least one person here who was like totally in love with you and was like, "They should be on the show every week and whatnot." So, so I agree on the show. >> So, so, so, so, so hi, Yenz's mom, I'm

172. 02:18:46

     assuming. Uh, thanks for watching. Uh, we'll see you next time, everybody. >> Thank you. [laughter]