Star Citizen Live SPECIAL EDITION: A More Stable Universe

Transcript — every row opens the video at that moment

1. 00:00:05

   hi there we're back yes take a moment and accept that this is my actual face I apologize to any children and small animals that it may frighten it's my parents fault so I've got a little bit of a speech here before we begin let's see if I can get through without messing up probably not it's a new year and those that have followed the project for a while can tell you that the first couple months of any year at cig is a period of examination of planning and of taking those triumphs and lessons like for the previous year and using them to chart an improved and refined way forward for the continuing development of Star Citizen and Squadron 42 so far so good and this year we've set the focus of those efforts early with December's letter from the chairman where CEO and founder

2. 00:00:54

   of the company Chris Roberts stated his intention to evolve both the existing patch relief Cadence and the content within for Star Citizen away from the previous new feature Laden model and into a more content-driven and stability focused Endeavor so sitting here right now what the hell does that really mean does it mean that we wave a magic wand and it all gets better in a single monthly patch like last week's 40 401 no of course not does it mean holding patches back for three four five months at a time with no new content until there's a perfect release not quite because we still need to make new stuff to feed the beast and as many of you already know while PTU and Tech preview have their valuable and essential uses there they

3. 00:01:42

   will still never reach the concurrency numbers of live Alpha and it is all too often the stresses of that high concurrency live Alpha that ultimately reveal the true underlying sources of problems beneath for a project like this so if patches are going to keep coming no matter what then what actually changing this year so like everything else in Star Citizen I'm certain this to will continue to expand and mature as we move throughout the year but in its simplest terms what this means is pulling engineers and programmers off of a lot of the new fancy Feature work they've always been on and putting them on less sexy but more quality of life driven efforts like rebuilding elevators and transit to better work with the realities of server meshing fixing issues where your hanger eats your ship and your cargo disappears

4. 00:02:30

   finally introducing some version of unique item recovery as quickly and as efficiently as we can and much much more it means a greater focus on refining the gameplay that's already there in the persistent universe and less time spent on continuously introducing the instability of a multitude of brand new features every 3 months or so it also means a new content and gameplay delivery schedule intricately tied to the introduction of new ongoing story narratives for the persistent universe that allow you as players to reshape the Galaxy around you beginning with 401 with then the faction Warfare that will change pyro by the end of this year so what no more new features no it just means that we're going to continue to work on new gameplay like Resource

5. 00:03:17

   Network crafting and Base building in the background and do it slower more deliberately and with more time to cook it means these new features not coming out as quickly as originally planned because the same super talented engineers and programmers that used to be working on those will now be focused on fixing what's essentially already there first and when these new features are ready to do so they'll go into expanded dedicated Tech preview testing instead of the Pu to ensure their functionality and stability ultimately this year is a recommitment to our desire to make start citizen a place where everyone can live out their sci-fi dreams whether they're industrial commercial civil impartial or controversial simply about traversal or other people's

6. 00:04:05

   dispersal but always Universal and hopefully with little to no reversal and to do so with an ambitious monthly patch Cadence focused on content not features that we hope brings with it and new places to go new things to do and new fixes to the issues that have often hampered the experience in the past I got cute there for a minute so I'm just going to explain it again just to be clear monthly patches with little to no brand new featur features causing brand new instabilities instead full of progressive bug fixes and refactors that will build over time and with new story driven content primarily utilizing game systems already in place that will continue developing the stories of Stanton and Pyro and Beyond of factions and sandworms and science run a muuk and more including new game in-game adventures and discoveries along the way

7. 00:04:53

   and that last part is super relevant to you and me in our relationship and why I'm talking so much right Now Quiet that's not true I've never needed a reason to talk this much with this next evolution of star citizens development meaning a more content-driven in Universe experience and with big Feature work taking a back seat to fixing what's already there and then moving into the public Tech preview development environment once it's ready farther down the line so too must our weekly video offerings change and to preserve that discovery of new story to evade spoiling the outcome of new content and to avoid doing 5 weeks of consecutive shows that all basically say every engineer is working on elevators in transit right now we're making changes to the weekly ISC and seal content and Cadence so like

8. 00:05:42

   everything else that continues to evolve in change as the year goes on we'll be bringing inside Star Citizen and Star Citizen live together on Thursdays we're going to be adding a new behind the ship series and alternating between them as the stories we want to tell each week demands for instance last week was ETS fortune on Tuesday this week we're having Chief technology officer Ben W boser who's sitting patiently to my right here onal for an indepth chat about 4.0 preview uh why it was done and how it helped the project how we used to work how we're going to work going forward answer questions about this new focus on stability going forward and I've even pulled an extensive list of jeras that we're going to go over to find out what the heck is really going on with some of these persistent bugs and then next week you'll get another BTS show on the upcoming Super Hornet mark two on Tuesday and then an ISC on

9. 00:06:31

   Thursday where we'll dive deeper into what the content driven side of this new initiative really means we're going to recap the story so far and look at how the story lines of the persisent universe will begin to take shape later this year we haven't even started yet I almost missed my mouth so so like the recent cancellation of the Lunar New Year free flight was done to avoid further destabilizing the current Star Citizen experience for our players this changed to our programming will provide a multitude of benefits for our players and developers allowing us to preserve the full experience of the content work for proper in-game Discovery by you but just as importantly letting the devs keep much of their focus frankly where we all think it should be on making Star Citizen in Squadron 42 everything we all want it to be and with fewer distractions by me the

10. 00:07:19

    guy they've all come to fear and dread whenever I walk over to their desk except you you don't get to escape you're mine now there will still be vide videos dedicated to the making of Star Citizen just fewer than before that will still be new ships added to the verse most will just use existing gameplay systems to avoid creating instability wherever possible there will be more patches than never before approximately once a month we're currently Counting about 10 over the course of the year full of bug fixes and new story-driven content and there's going to be fewer Big Ticket features you were used to in the past uh uh being introduced until things are shorted up in short in short too late it's going to be a very interesting year to be certain and yeah this is a lot of talk to which

11. 00:08:09

    I'm just going to say yeah this is a talk show it's what we do on SEO the ultimate proof the ultimate proof of these efforts will be seen little by little month after month over the course of this entire year and we hope you'll join us and continue to support us on this new adventure towards a more stable universe so with that I'm going take a drink and we'll get started with the show proper roll the Q card the T I thought about pre-recording it but I didn't we're doing it live oh one take Huckabee hi everybody Welcome to Star Citizen live I'm your host Jared Huckabee and if you've never seen Star Citizen live before well this is a

12. 00:08:56

    really interesting one to start on uh this is our special edition a more stable Universe episode with me and chief technology officer Ben W bour Ben W I've been talking too much say something hi everybody how you doing Beno I'm good I'm good so for anyone who doesn't know you let's start us off explain who you are and why you're the person that's here talking to us today on this show so I'm uh the chief technology officer or as the acronym means chaos team organizer and so my job is to um now lead the engineers in their development efforts but also in the the development of the star engine and also both games right to make sure that Engineers know what they have to do that they have the tools to do it and that they have the right priorities overall

13. 00:09:43

    based on the two projects so this show is going to go a bit longer than normal we're going to break it into a couple different sections we're going to talk about a few different aspects of development and then for the the latter half of the show we are literally going to bring up jir on my screen and we're going to talk to him about all the most most pressing bugs well not all of them but many of the most pressing bugs and find out what the heck's going on but before we get to that we I want to talk about how we used to work then talk about this big change uh that's going into effect this year so on soth so let's start with the old ways uh let's start back at the end of last year with 4.0 and the 4.0 preview over the last two months we've seen a lot of opinions online about how that went and why we did things the way that we did so you tell us why did we choose uh what

14. 00:10:34

    did we learn from 4on 4on was in testing for 3 months before it went to preview and then we left it on for another month and a half um how did this process go well so during the development of 4.0 the there was a lot of new technology coming on right so that's it's a major patch that changes a lot of the fundamentals uh obviously throughout the year we've been developing server meshing we've done Tech previews on server meshing uh and so getting to the end of the of that process basically exiting the tech preview streams coming back with the 4.0 content was a major Endeavor to synchronize all that make sure the new content that was produced fit with the technology and that it was working even results that we got with the 324 release for example were completely you know uh obscured by the

15. 00:11:23

    new content right and so uh at the end of the year we had we took a decision to split the two releases right so leave 324 up correct put 4.0 in a preview correct because as we were getting closer to the end of the year right and it's time's running out what we knew from the Integrations is that the patch was not quite ready because we needed more concurrency on it uh to discover more issues and we were kind of at the maximum of what our PTU concurrency could actually deliver and so but we knew it wasn't ready for a live release uh because uh you know we want to be fair with our assessment of where the patch was at uh we knew there was some egregious issues that we have yet to discover I'll talk a bit later about the

16. 00:12:12

    fog of War of of crashes um and so we decided to go with the the lesser of known of the two problems which was leave 324 Al for players to enjoyed in a stable release right uh but at the same time give them access to a 4.0 preview now since we had taken the decisions months ago that the 4.0 patch was going to be a full wipe that gave us the ability to basically preview the patch but allowing you guys to keep your persistent progress uh in the preview as as we move up to at some point bring 4.0 on the live environment so overall I think that was a that was a very difficult decision to to do this option for us but it I think it was the best for the project because it allowed us to De risk a little bit the do her initial release

17. 00:13:00

    which had a lot of issues even during the preview uh a lot of the Skeleton Crew teams worked throughout the break to actually document hfix some of those issues as well and help players that got into weird situations uh and start the ball rolling for the year with already like important fixes to the underlying technology two follow questions to what you just said you said that the Clock Was ticking yeah what what clock did did 4.0 have to go out before December um that that itself is a decision that was question that was questioned by many people internally and externally what is this clock you were talking about well in this case it's like the the our project is different from other game projects in that we need to release the code at some point to be able to move on to bigger projects and the PTU is a is a great

18. 00:13:50

    process for us but it does have its limits at some point we gain nothing from additional PTU builds because things function well at that level of concurrency and we need way more concurrency to go up even when we plan patches we know at which moments the in the PTU will have concurrency and we try to maximize those for 4.0 we kind of ran out of gas of like attention and so we needed to get this additional hump over the over the The Edge to get the technology out bigger concurrency and the ability to fix it uh in order to then plan for the release so you said things run well not everything was running well but it's it's what I it's what we've said before it's what I said the begin of the show sometimes you can't find those underlying problems that they're causing it to run less than

19. 00:14:39

    well until you get that concurrency up so to that end wouldn't it have made more sense to just take 324 down and force everybody into 4.0 like we've done every other December in the past right which was that was the our typical decision typically was to just go for it right in this case I think we we it's a shows a change of mentality which is that we're trying to respect our live environment a bit more and so the decision was taken to just allow players to keep keep playing 324 as a even a fallback or as their primary play environment if they preferred uh while you could jump on the preview experiment and then just see you know try the new content out but then always have your stable experience to go back to but realistically sure and for the next for the next couple hours

20. 00:15:28

    you're going to get a lot of question I know sure who in their right mind was going to log into 324 with 4.0 sitting right there like like like like who drives up to Disneyland and and and look what Disneyland right there goes to California Adventure instead like did we did we actually was anything gained by doing this really yeah like overall like throughout the break right the during the 4.0 preview there was actually a a fairly large contingent the player still playing 324 uh now it might have been because of some of the issues at launch right so they fall back on the 324 batch right which was a bit the kind of the the safety net that we wanted to provide uh but you're right like of course like we were banking on the fact that people would go on the 4.0 preview to give us that concurrency but at the same time we want to give them an escape hatch sounds a little bit like we were

21. 00:16:16

    trying to have our cake and eat it too a little bit but I think it was it's it's a like previously we would have just gone with the 4.0 live at the end of December which would have been a major issue and a lot of players wouldn't have been been able to play during the break okay so we made this decision what did we gain from it over the holidays over the second half of December and the and all through January until 401 what did we gain from from running these two separate environments well we've been able to actually do a lot of live comparisons of the two workloads which was very interesting in fact to to obviously the 4.0 preview workload was bigger than the 324 one but we were kind of able to comp compare both we did gain the uh the ability to fix early some of the most egregious

22. 00:17:04

    issues that would have been a real complete like disaster otherwise right that showed up with the concurrency uh I'll talk a bit more about those actually we're going to dive into that quite a bit uh and so what we've gained is is is the ability to provide a test environment that was bigger while allowing players to keep an escape PCH and then compare both both builds so that's the preview versus 324 and I wanted to get that out of the way but let's go back to to just 4.0 itself 4.0 started testing I think I think it was in evocati in October uh went to wave 1 in November we left it in Tech preview over December in January that's about four months that's about four months of testing did we fix anything oh yeah oh

23. 00:17:55

    yeah fix a lot here's the thing I I know what the answer is I'm being a little cheeky with you but it's hard not to look at the community and look at the sentiment look at YouTube and twitch and spectrum and Reddit and and everyone else and if if you just if you took a real cursory first glance it looked like what the f did we were we doing for four months sure what were we what the f were we doing for four months been all right well so September October November December right so since September we started the real voad in October cor and so the for these four months the majority of the development was aimed at finishing the features to make sure that they were server masching compatible so we still had a uh server maching play test at the ear the beginning of October

24. 00:18:44

    right that was the final one before we actually reintegrated the streams and so then came some realizations uh we were very happy with some of the tests that we've done we've identified all the issues with our server machine tests uh we knew what we had to do and then we came back to the main line from the 324 content and we started running server meshing in pyro and major issues started to show up when we started the PTU um at uh beginning of October the first PTU was a absolute disaster the all of the metrics that we had captured in 324 and we were like yeah these look good we're completely off the charts in with the 4.0 content same technology same codebase but completely new content and so

25. 00:19:33

    uh the we realized this that the size of the levels were different the way the content was was created was a bit different and that had major implications on the streaming and replication systems and so uh if some of you guys were part of the first couple of evocati builds uh or large ptus in October the network uh performance was absolutely horrendous because um and I can jump into an anecdote I do have my notes cuz when we've got a long show to go to so I'm going to be referring to some of those but uh basically there was a a new change in the engine where we tried to optimize some of the the way we deal with streaming radius and streaming radius is a is a bit of a problem in every game right or it's basically the radius of objects and you decide what

26. 00:20:19

    you're going to load based on this based on your Viewpoint um which is a typical issue in games right for performance reasons in in Star Citizen it is is a a much bigger uh impact because the streaming radius actually affect what your client is going to load in what the hybrid which is the replication server in the middle needs to replicate and what the game server is going to also have to provide right and simulate and so there was an issue resulting from one of the engine optimizations where now I'm going to say the final solution not the how we got there we had to investigate this for several several several days and uh to find it but basically the the way we compound an aggregate objects would actually inflate the bounds of things and so the new locations in um in pyro

27. 00:21:09

    would actually have a radius of several hundred, kilometers and so basically what the result was was every game client in pyro was basically binding a very large part of the environment and that created the work on the hybrid service which needs to replicate this data upwards of 15 to 20 megabits per client across megabits megabits yeah which is uh in the case of our messaging is is substantial and so obviously in those cases interaction lag up the wazu uh nothing was functional and so we lost several amounts of times there but we learned a couple of lessons which are going to inform some of the changes we'll be doing this year but uh so that was the beginning of the 4.0 uh Journey right the first build where we were

28. 00:21:57

    coming in confidence from our tests first PTU what's going on right and so another thing that couldn't be discovered until scaled up correct correct um and then well in in this case it's a combination of scale and content right with the content scaled up yeah well in this Cas it's literally scaled up which caused the streaming problem right uh so that's a quick anecdote um what else we've been doing on on uh starting from there was we've established our player accounts for launch right so from the the testing uh we we decided to shoot for a 500 player Shard with a basically a 500 what we call soft cap and 600 player cap which uh gives us some flexibility in matchmaking players on and off that number felt good for several reason performance but also uh the uh and head

29. 00:22:47

    room but also the kind of the the ability for the game itself to support that amount of players so the amount I've talked about this before in some of my server meshing reports but the amount of gate ways we have for hangers uh you know how many hanger holes are there around stations right for you to be able to get out without the big queue things like that uh with more players right now the Game Wouldn't support it properly so uh that was a good we've established those numbers and then the team set up to finalize the latest elements that we've identified during the test to launch so uh I've talked about Network bind optimizations before so that had to be completed so this was a full solid month of work to keep optimizing the network bind so that we stopped having these quick stalls that was a very large one another piece of this was the teleport

30. 00:23:36

    API now I know that sounds what is this magic right uh in a normal game server teleporting objects is very simple because the server has all the entities in memory and you're like just move there our teleport API needs to support cross server teleportation of an object right so an object that's in Stanton technically can be teleported through the apis on another uh Shard uh not Shard another server in The Shard and so the teleport API had to be kind of finished there are several systems that use this uh for example uh common example is if you die and we need to respawn your character we need to move your player uh from where he died to where he needs to respond and that can be completely out anywhere right so we don't know what game server will have it and so in this case the teleport API had

31. 00:24:24

    to be kind of finished uh so that's part of the work that was done in those 4 months for launch uh the other one was the territory configuration so uh if you followed the development during the the last couple months we went at some point from the typical small Shard of like two servers to and 400 players and at some point we exploded the size of the shards that's because our new territory configuration came in into play and that's how we were able to experiment and come to the final level of the the shards that we have now which is basically we have five servers on each side of the star system so uh each each star system has its own space server we call a space server for now it's just cool name because they treat the open space and the jump Gates uh but then after that every major uh planet has its

32. 00:25:13

    own game server that's dedicated to its its Zone uh and so that really like really opened the shards up and was we were truly now in full server meshing mode obviously a lot of issues came from that not from the configur configuration but from having more servers would has Ripple effects on how the entity and in the game would actually behave so we have to touch that uh we also identified something that we were not able to finish for 4.0 which was uh the coling of instan and Terriers so and I'll talk a lot more about hangers later but basically when you hear someone in another hanger that's Network instance calling right so we'll talk about that in a bit uh now the one of the big fears that we had

33. 00:26:01

    for 4.0 was that you know of course it's the debut of server mhing it's the it's a brand new you know deployment for the game it's a lot of Technology change and but you know there's always a subtext that performance going to get better if we go server missing right and the reality is the initial test showed that it did not now not because of our assumptions were incorrect but because as always when you don't completely drive on the performance it tends to get worse and so we mobilized everybody like the entire company basically all the engineers right where we started a series of uh I called them Joker card initiatives right so several teams during the last push received the Joker

34. 00:26:49

    card and what that allows them to do is to cut the red tape so for example the network team had the The Joker card they could request anything from any team and get priority and that allowed them to really move forward on their changes and their fixes and so we started a bunch of those Joker card initiatives uh on performance specifically and so every single day we would take captures we would review hundreds and hundreds of captures identify code parts that needed fixing and then literally without a big process right so we've cut the process down you talked about Jas earlier right this this part of the process no jira right and we just called out here's a performance hotspot and then directors from the company our amazing Tech teams

35. 00:27:38

    and our Engineers just jump on those things and go I know what to do optimize next patch right and we kept grinding like this to really make sure that we could deliver above 15 to 20 FPS per server at least right to get started and we want to go even further but that was kind of our Target was to deliver something where we're no longer dipping under 10 at all ever right and so um for example the the uh AI team uh discovered through this process that there was a very large navigation mesh generation that was directly on the main thread that cost us 15 to 20 sometimes milliseconds which is the frame budget's 33 right and so but the response from the teams were amazing they rapidly refactored and optimized really key pieces of code to shave on

36. 00:28:26

    these things that made a very very big difference uh out of these Joker card initiatives I'm going to jump into the important ones uh physics is very big issue for our servers I've said that before not because our physics code is terrible and in fact it's great but we demand a lot from it and we started to look into well what's going on here why why is it costing us so much and so we really profiled on that side quite a bit together as a team uh so so one of the things to remember here is that our bugs persist right if I think forget they persist bugs persist and so but what I mean by that in this case is if I spawn an object if I spawn an object

37. 00:29:16

    intersecting with this desk right in any other game when the map reloads these objects are no longer colliding and we're good right in Star Citizen those things will stay in complete intersection in the database and in The Shard and in its persistence forever right and uh the physics code tries to solve this right and that's what's costing us performance in big in big parts right uh and so we started this Joker card initiative that we call the physics expensive event resolution and so it's about the physics engine detecting these things which it already did but now raising it for resolution saying I need a solve for this and so a ship that intersects with a wall or that

38. 00:30:05

    spawn in a wall uh you know a weapon that falls through the floor and that's stuck in there right debris has been a big one right and so um we needed a way to solve this so what the physics expensive event system does is it raises an event for the game and the game now receives these object and it says uh you know debris and players ships well I'm going to do something with the debris I'll either delete it stow it transform it move it right and so as we start doing this it's actually cleaning up all these intersections Across The Shard and basically giving us back this performance step by step and so one of the very large things that was very egregious in this was actually a bug on the ships where on explosion some of the components of the ship could stay static

39. 00:30:54

    in the world throughout the explosion but you know behind the scenes when the ship explodes we do a whole bunch of things we replace some meshes with damage meshes uh some of the elements are physically deform and move uh and we create debris we literally spawn debris now if those components stay static and we spawn debris we're we're creating intersections debris straight intersection into these components and so that was a very big part of the problem we fixed that bug but we also added a system to basically raise the issues with the debris so that the be cleaned up automatically um another one was the AA turrets so you know as you guys play you destroy these AA turrets at the key locations right and there was a problem with the setup of these AA turrets that

40. 00:31:44

    when they died the same in the same way as a ship when it dies and the components detached right the a turet had an issue in in its setup where it would go like this but overlapping each of each other what basically every time someone killed an AA turret we add it to the server load forever and so obviously now that's being evicted completely from the system somebody in the chat is like you can destroy the turrets sometimes sometimes so yeah they might destroy you first but I want to I want to jump a little ahead here just I don't want to get too too far in L obviously all this stuff is super important I've seen some numbers floating around in some various chats about how many bugs and how many fixes were actually you know uh resolved over

41. 00:32:33

    the course of over the course of those four months uh I asked you to look into the numbers what were the actual numbers so for 4.0 between 3243 and 41 with the team have fixed 2,900 bugs in total but that's over 3,300 tasks so a bug can require a couple tasks to F to fix so in General that's the that's the number of fixes overall I'm going to ask a very candid question because 2900 bugs is a very impressive number but uhhuh did we fix the right ones did did you fix the right ones did had it's it's it's obviously those ones were all many of those obviously are important we we have a tremendous uh codery of directors

42. 00:33:23

    you know a lot of students around the world who are prioritizing these things and I'm sure I'm sure they get us you know a couple frames here are they solve an issue that's going to affect those 10 people over there and like this but there are many of our long-term bugs the the things that have just been an albatross around our neck for long time a long time and the second half of this program we're going to get into those one by one so I don't want uh so without going into each one of those because we're going to do that in a few minutes here why these 2900 other bugs and not just two or three of the one that would change the lives of hundreds of people thousands of people well so I don't I don't do prioritization but right what I can tell you is that most of those issues are are filed they're called out right and they're raised up but the

43. 00:34:14

    complexity of some of them require a specific handling so you know if uh like I've read some stuff over the break where we we've hotfixed server certain issues like there was was a if I remember a certain exploit uh that came out for some duping that was and some people thought it was a beneficial change even though it was a bug and you know and we fixed it and people are like well why why did you fix it when you could fix something else right it's all about capacity to fix so do we have the right person available for this problem some problems require the experts the our Manpower is extremely specialized and uh for some of the older systems it's even worse uh I call those the the sandworm kind of feature where basically I've called them out before but ATC asop

44. 00:35:05

    terminal hangers and Transit are four of the systems that are always oh I'm just going to do a quick change never happens never a quick change in those systems right yeah and so uh yeah I want to jump in here yeah uh uh unomas in twitch chat is yelling in all capital letters you are the experts true one of the thing one of the things that if you watch any of our programs over the years I've been doing this for a decade here now this is is that not everybody works on everything sure not everything not not you know you know Kian is great at this doesn't know crap about that you know you Paul is wonderful at this you know would be as lost as I would be tackling this other thing there there are a finite number of experts that are available on any one thing and like we

45. 00:35:54

    talked the last time you here the folks who made ATC aren't here anymore so ATC has been one of those features that had rotted and it was it was there was always this there's this terrible Sophie's choice and we have a section about about how these choices are made here where it's like okay we can take we can take someone like Ken or someone like Paul or whatever and put them on this but they're going to be on this for five months because they've got to start from scratch whereas they could fix 15 or 20 other issues in that same time and you're like well and this this constant calculus of is that one worth the weight of that of those 20s and it's not a situation I I envy you or anybody else you said you don't make the the prioritizations you're in these

46. 00:36:41

    conversations along with everybody else you mean you don't make them alone you're you're not the sole person who gets to decide and so like this it's it's like many things are done in a committee because everybody's got their perspective everybody's got their priorities everybody's got their idea of what they think is the most important been in chats where Chris will say this is the most important and Aaron will say this is the most important and Yen will say this is the most important you're like well no this is the most important and we have the same conversation that the backers have just three weeks earlier is usually what you know what I say so not this P it's prioritization and also having the right resources and having that right resource available because sometimes the right person for that is also the right person for this and they've been in this for 3 months and to pull them off of this right now

47. 00:37:30

    would be devastating to that because then you get Tech debt and that begins to rot and it's like we just and we sit there there are so many times where I've been in meetings or sit in chats and just ground my teeth because we all know what needs to happen we all know that that thing needs to St but there isn't the resource at the moment to go and do it and that's as F at least as frustrating for us as it is for the players probably more so for the players if I'm being completely honest but sure um no but you're right it's the it's all about having the right people for the right issues at the right time in the right historically you know if we're being frank we've been focusing more on the features we've focusing more on the new stuff correct and I think that you know the the you know what Chris outlined in his in his letter for the

48. 00:38:19

    chairman is really the direction we want to go which is to to bring back those experts into helping us fix the things that affect the players day in and day out so let's start talking about the letter for the chairman now sure so in December Chris published a letter from the chairman set that that would set the direction for our entire 2025 I did a big thing at the beginning about this and what I'm going to ask you for your version of this I want to read specifically a few words that were from that letter as we jump into the new year our primary focus across our Star Citizen teams is clear improving playability our drive for playability surrounds three critical areas performance stability and content no features in there we understand that the live environment must be far more reliable and consistent and we are

49. 00:39:07

    determined to deliver on this the sentiment many of you have shared and one we wholeheartedly agree with is that if the current game as it stands today ran smoothly with fewer obstacles and bugs it would provide an unparalleled experience over the years we've experimented with different approaches to improve quality of life and tackling Tech Deck uh Tech debt however the ongoing challenge has been balancing these efforts alongside the introduction of new features and technology so in testing new features within the same pipeline as content it's led to a bunch of intended Ripple effects creating instability hindering performance and impacting overall gameplay in some cases new features that are not fully mature have delayed the release of content or cause blockers that impact our progress this has prompted us to re-evaluate our approach

50. 00:39:56

    to feature and content roll out focusing on steps that we can take to bring the game to a better State while maintaining steady delivery and that's the end that's the last I'll read from this Stu so sure I just got done at the top of the show explaining what that means in my words right I want to hear what that means in your words H what are we actually doing to put those words into effect this year what are we actually changing stage one is we're committing more resources to fixing issues that affect the live environment but also that are reported by players and so in a typical setup we we have to take those resources from features right so that means that normally if we're on 50 different features all these Engineers are working on new new content uh to build new

51. 00:40:46

    components or new features new uis for these new gameplay elements and this can go on for a while because there's prototyping there's iteration there's you know merging QA and this can take a while to bring it in and as Chris is pointing out creates Ripple effects and so we have to slow down on features to get those resources to jump into what I call these Camary issues like we're going to look at a couple later but issues that require oh I need a physics expert oh I need an audio expert oh I need a graphics expert oh I need a gameplay expert I need a guy who wrote this system I need a guy world this system an issue that just keeps growing until it's solved in order to do that we need to give a joker card to those issues and so that's what we're going to do this year is to slow down on features

52. 00:41:35

    we still need to build features but we're going to pick the ones we can do while still investing in correcting the problems that we know are have been in the game either forever or they're newly and very active and even if they're super complicated to solve and require a refactor right so we'll talk about refactors I'm sure in the next list but uh it also means um uh you know what I put it in my notes cuz you know we kind of prepared a list so that I can just organize my thoughts but uh increasing bug fixing quotas for teams so in general you know teams have a not necessarily a quota to fit but they have a lot of time for bug fixing we're going to try to jump that by a lot so that we can actually jump on more bug fixes that matter that's the thing that's key right

53. 00:42:23

    even though we fixed 3,000 issues you know there are maybe you know 15 or 20 or 30 issues that are really affecting players day in and day out that are bugs or defects that must be resolved and so we've put that in a list called the director's list so every director has a chance to basically flag an issue in their domain and say I'm going to need a joker card for this issue and it needs to be solved right and so that list is also going to transform itself into uh a release criteria so you know typically when we do a PT were going through these waves right uh and uh you know the the process at which we go to the next wave is always a bit you know oh it looks good we look at metrics we look at you know disconnects and stability in general but we don't necessarily look at

54. 00:43:11

    quality of the game specifically because we're in development and so it's always been a bit different for us but I think that we need to this to change this process for us to move away from being an alpha game we need to start moving away from the we constant development in order to do that we need to professionalize our criteria for launch and that means that director's list need to go down it needs to get the solves that it needs and those become criterias to move forward now we're going to have to make decisions because we cannot fix everything out at the same time again it's a question of resources but the the goal is to make a significant dent in that list even if it keeps uh you know going up there you said you said take increase the number of Engineers and and programmers you're not talking about going out and hiring a hundred new

55. 00:43:59

    people you're you're talking about you know the people who were working on Base building the people who were working on Crafting the people who were working on resource networks like that it's about he's about taking those those seniors those people with with with with the experience the the the one that uh Uno said the experts about taking those experts and pulling them off the new feature development corre and putting them on the less sexy you know stuff like hey fix it elevator that's an example it's okay all right so I uh I want to talk about the three areas that were covered in the in the letter from the CH we'll start with stability yeah so I know this is going to sound like a simplistic question and there are probably lots of star C and I can see them in the chat right now uh uh JJ land 614 says no one cares about bug

56. 00:44:49

    fixing uh is is everybody right is JJ land 614 right everybody um sorry that there was no reason for me to do that sure um so lots of people who have followed these shows and and they know parts of this I've seen the people in chat I already know this because I watch everything this is a show for everybody right um what does working on stability really mean what is it's we've had a lot of platitudes here and platitudes are necessary they're helpful to explain things but translate this into real world is [ __ ] for me right I use my one swear what is working on stability actually translate into as far as action goes well it it translates into aggressively monitoring uh what we

57. 00:45:38

    call internally stability covers like some of the very simple metrics like do servers crash right or do hybrids crash uh do clients crash and together do these cause client disconnects right and so all these metrics go into what we call basically stability but it goes further than that those are just basically the indicators like the big level indicators what stability is is like things like barebone things like are you able to successfully connect to the game can you log in correct uh can you do successful interactions like can you go to your ship and press F to join the your cockpit and then not it take 35 seconds uh is it consistent and reliable mean

58. 00:46:26

    meaning like is it like this all the time or is it there only sometimes right so stability is about consistent and reliable experience interactions it's also when we work on stability we're talking about improving the observability observability so teams that when you guys are playing the game you have a very immersive experience and issues that affect you during your game play are part of your bubble right we need to be able to see those issues from the outside so from the server side from the cloud environment side and so improving observability gives us that because there are many systems in the game that have not been instrumented to be observed right and so sometimes you know some some older systems I'm not going to name ATC uh you know are going to have like

59. 00:47:15

    some hiccups that we can't even see but players report to us and then we have to investigate but we need more observability so that's working on stability is part of that changing systems that typically are problematics into selfhealing systems so that they can just repair their their uh themselves so when they get into a case that we know we just modify the system to make sure it tends towards a state that's functional uh and the observability comes into play to make sure that it does function uh working on stability also means establishing budgets so budgets for errors uh budgets for performance and so that affects everybody in the company but if we we talk purely on stability it's all about a is the content that we've produced here or the

60. 00:48:02

    feature we've produced here using too much resources whether it's CPU GPU server calls bandwidth storage networking uh so reinforcing those those budgets going about finding the offenders right that go above these budgets and chasing them down solving them so that's stability work uh reducing error rates obviously that's part of it that's the resulting and uh the the challenge with working stability is that uh especially in light of server crashes is that it's a it's a fog of War so uh you can launch a game and you're like a version of the game and there's one crash that's going around we can identify DD crashes right so we we know what they are uh so we're looking into one and then we're like oh

61. 00:48:51

    if we fix this one it's going to be very stable right but then when you fix that one that allows the program to continue and then more bugs show up no more crashes show up so it's a fog of War it needs a constant care and attention uh one thing we've done this year to help that is that we have a much more robust hot fix process uh that frankly we use a bit too much but uh it does does give us the ability to get some fixes out when we see big problems right away but it's it's to contribute to that fog of War so that's what working on stability for me is it's all of these issues together do the same thing for performance sure uh performance is uh it's a larger one and so performance touches on both the client the hybrid and the server the

62. 00:49:39

    hybrid is the replicant that takes care of the networking uh and so performance is about obviously frame rate to begin with on the client and on the server uh we talked earlier about server optimizations uh it's about loading times so make sure making sure that the load times are acceptable uh in all kinds of areas like I'm not just talking about the loading screen I'm talking about loading an assets streaming in an area uh uh you know fully initializing a a full location right uh that's the load times I'm talking about uh memory utilization so making sure that we use efficient memory that we don't go above our memory budget uh same thing here the keyword is budgeting so we need to have way stricter budget we need to work on that area and then finally resource

63. 00:50:28

    utilization of your machine in general but also network connections so latency and uh bandwidth connectivity and stability of the connection in general uh that need to be improved so that's what working on performance is so stability is one area performance is another one all right so who works on these things because this isn't so we're talking about this major change here we're we're taking people off of features and putting them on New features and putting them on existing uh uh existing features existing work this isn't something that vehicle artists do they get to keep doing their their thing for the most part this isn't something that somebody who makes a distribution center for the environment team does this isn't something that somebody who makes um a really unfortunate

64. 00:51:15

    heart-shaped bed for the subscriber program does it was performant [Laughter] who's the person who's actually do who are the people that are actually doing this work who who Who's Who who's changing to do this stuff so it's again engineering right that has a large focus on stability and performance but it's also um you know the uh we have a Ser a team an online team that takes care of what we call site reliability so these guys are definitely the first Watchers of all these metrics and these guys will call out issues that we need to solve right and then they will Loop in engineers and so that's the core of the people working on this but I I kind of want to push back on what you just said

65. 00:52:04

    about like oh the guys on the ship team don't have to do performance right which I think that it you know sometimes the way things are made in the game have a very big impact on performance right uh you know uh I can give examples but like areas of levels that are created with hundreds of entities right instead of using like a bare minimum or there's techniques to use where group merging certain objects uh need to help reduce the actual effective size of things so the teams on content are also uh kind of on the hook for performance less so on stability but they're on the hook for performance and so they're not totally free but they are less in the line of fire because typically an engineer is needed to identify a problem first and

66. 00:52:51

    then it might reach a Content team depending on what the issue is okay um what's the difference between an engineer and a programmer it's basically the same thing almost why do we use both terms I don't know I think it's when you use programmer when you like you talk about a coder and you talk about an engineer when he's like he's got an architectural approach to it I guess but they're s they're very similar in my mind sounds like full of [ __ ] yeah a little bit but uh you're not going to count that one where's the where's the jar uh you need a you need a drink you're talking quite a lot all right so let's put things in motion now yeah it's 20 it's 2025 we we we we we we've got our mandate we've got our directive

67. 00:53:38

    set for us by Chris Roberts um we touched a little bit about prioritization and delegating responsibility so I want I want to talk about this it's it's no secret that anybody who's watched my my my stuff for years knows that that I I I like to have the conversation that no other game company would ever dream of having and and there's nobody from another gamean that would sit here for for two hours and answer these kind of questions so thank you um I want to talk about delegating responsibility let me think about how I want to put this obviously we have a lot of talented engineers and programmers across all of our Studios uh but some are Juniors and some are seniors who have all their own

68. 00:54:25

    Specialties and aptitudes how do we decide who works on what because because when I look at if I were sitting on the outside and I used to sure if I were sitting on the outside I would look at some of the decisions that have been made over the last couple years uh elevators have been a plague for forever a a transit system has had a refactor or two in the past and what not those were all before a server me which re was now which will now require if I were sitting on the outside I would look at this and wonder what the hell are they thinking from time to time sure why so so so I want I would like you to talk about how this process is done I know you said

69. 00:55:14

    that you're not the one solely responsible but you can speak to this process and and and and how this responsibility is delegated not just to what bugs get worked on but who right works on this stuff talk to me for a bit about this well so it's it's it's an interesting uh problem right because fundamentally all these talented programmers and Engineers are are human beings right behind it and so they even Paul even Paul I mean he's he does have a beast mode but uh he still human and uh I think that the one of the challenges that the teams face regarding this prior not necessar prioritization let's talk assignment first right is uh there's a certain level of ownership right if I wrote a system then I own it right in a way and then further iterations or bug

70. 00:56:05

    fixes on this thing I will be the best qualified to do it because I know the code anybody else that needs to go in this code is going to have a you know a little ramp up right and sometimes we can't pay that ramp up especially at the pace at which you know I keep saying that in our project it's different typically we don't release a game before it's in it's in a better State than most games don't run live game environment in the middle of their develop exactly and and that I call it the cost of shipping is what hurts us the most because now any priority issue for the live patch needs to become absolute right but so to back to assignment right so one programmer writes a system e owns it right so we're going to try to go back to this person when there's bug fixes or items to go to they're have the least on-ramp time yeah exactly but now

71. 00:56:53

    sometimes uh it can it can happen that's a junior developer it can happen that a senior developer it really depends right uh some systems also end up with no owner when you know there's churn and people leave and you know there's Replacements or people move on there's a lot of internal movement at cig where people want to change their specialy or and so you want to keep that distribution fair for the people in there like you don't want to sign someone on on pure bug fixes for five years uh because they're not going to stay with the company right you need you need to balance this out uh in terms of the of how you assign the work uh now again I'm not talking prioritization assignment right uh typically we're going to try to pair Juniors with seniors to encourage growth so that

72. 00:57:41

    these Juniors can become seniors and then we can depend even more on them right and so that's another constraint that sometimes you need to spend time from a senior on a spec specific issue to help a junior on another issue sometimes you'll have two people on an issue not because they can make the baby as fast but because that's how you train that Junior to become and that's one less person that could be working on something else that's right because you need this is part of their training this is their growth okay correct um there's also so we talked Junior seniors right we make it simple but there's also leadership right so uh lead developers like I don't know you know most people know this but the lead developer is not necessarily the best developer of the team wow name names no but the Le's responsibility is is

73. 00:58:30

    management right it's making sure that the issues move and that they that our plan of action is sound that it's vetted that's it's you know code reviews make sure the quality is there but also just orchestrating the team right so typically a lead should spend roughly 40% of his time doing Management on the team right so leadership is also a key element in that even if you pair people and you have a million things going together what is the drum beat is the leadership right and so you need to have all those combinations to be able to have a healthy engineer team and so uh you want to avoid overburdening as well sometimes uh you have some developers that come into the company and they're really good right but they're juniors right so they they're learning fast they

74. 00:59:18

    have a lot of energy they're going forward uh and so PA of leas resistance we're going to go like oh well look can you look at this can you look at this can you look at this and at some point that person's overburdened and we have to release our patches so time constraints come in and so cold quality goes down right and and that's an issue we have to battle every single day now we had to release our patches with those features in in the old system you know you're referring back to how we how we did it before in this new system in this new way it's it's it's a patch every month but it's what is ready to go into those patches we've said some version of this before sure and let's just be super honest we haven't been super good at sticking to the philosophy you know it was it was after it was

75. 01:00:05

    after three it was I think it was actually it was after 3 it was after a really long wait for 3 where you know some of us had to grow a really stupid beard yes that that it was like okay we're no longer going to wait for features anymore we're going to do this quarterly patch and then if a feature isn't ready we're just not we're not going to wait for it it gets pulled out we weren't so the famous go process yeah we weren't we weren't really good at that which we internally called the go go go process yes which we we moved away from I'm very H I'm very happy to hear you admit that the go no go process was a bit of a joke it was the go go go process and it's because we there was this mentality here where we thought that we were a featur driven project every patches people expect X number of features uh uh to be in there we have to

76. 01:00:55

    make these Targets in order to keep people engaged and stuff and it was just this Oro boros of [ __ ] oh God that's three um where it just we're constantly chasing our our tail and not not getting where we needed to go now that's not to throw the pastes out we're super proud of what we've accomplished here but there's a point where there's a point we're doing the same thing year after year the same way the same way and we always made changes here and there and expecting a different result is the definition of insanity so year and this whole thing that we're talking about today and we'll talk more about on ISC next week is we have it's time to try something different now this is incredibly scary for us I'm I'm going to go on the record and say we have been

77. 01:01:44

    a featur driven project since the PE was first introduced in 2015 uh we doubled down on that after 3 uh 3 at the end of 2017 to stop St that and to and to turn all of our Focus internally to turn it all Inward and say and to say to this incredibly talented programmer you don't get to work on the super sexy hot new feature you have to work on elevators now that is going to create problems for us keeping people engaged keeping people interested and stuff like this uh that's when patches come out and when we get to April or May or whatever and people are like where's my new feature we haven't had a new feature in so many months that's going to be create problems for

78. 01:02:32

    us on the marketing side and stuff like this it's this is this this this endeavor is going to create struggles for us in how we and how we position the game and how we Market the game and how we attract players uh uh to to come in and play but we've tried the other thing for so long and it's it's it's collectively what I'm seeing from everybody from from Chris to Aaron to to to Rich tyer to everybody down the line is this is something I've never seen in 10 years here is this everybody's hit a point where we've all we finally reached a consensus that we've done it this way enough we have to try this now and we have to make this experience better you know uh uh famously we were watching Big streamer jumped in the game on Friday had like 177,000 viewers and we and

79. 01:03:21

    177,000 people viewers were just watching him fall into an elevator over there and it was just like we had already made this decision like this decision has been in place since December but we were all just watching going man this can't come fast enough so this is scary but it has to be done I want to ask you another question about this about this assignment and the priority stuff I I just touched on it say you have something unsexy but super critical right and it's a and the the fix is a long and tedious process rewriting the transit a system and it's going to take tie someone up for the next two or three months while they work maybe it's longer is this something you put on a senior because it's super critical and then you end up tying that person up for

80. 01:04:09

    an entire quarter of the year or is it something that you stick a junior on because you need that senior doing three other things instead H is this not the ultimate developer Sophie's choice and is this not what what we've struggled yes with for a decade the time I've been here how do we how are we fixing this now well so previously the answer would have been Junior with some support right that that's what we would have done before it would have been no we can't compromise the feature you know this work is important it's going to be transformational for the Pu this fix or this rewrite you know we're going to do like we're going to give a support network around someone that's in a team that's not necessarily one of our Superstars you know and I mean I want a team where everybody can grow into a

81. 01:04:56

    superstar you know but sometimes you know you have to be in the middle right and so the previous setup would have been something like this right where we would give a support network but the person in charge would have been you know not necessarily like the leads and the people that really make push the needle because we those would be on transformational features for the Pu uh the answer this year will be no we're using our transformational players to come and solve these problems for us because they have to be solved right like we keep talking about Transit this not you know Transit has touch a million people at cig and we've tried to you know keep it running for a while but uh now it's time to put you know a real effort behind it right which is to basically take resources from the from the transformational elements and and go

82. 01:05:45

    and fix the foundation so we've now called ourselves out on our own BS and and we're we're now putting everybody where you know where they where they need to be how do you decide whether that old feature is fixable or whether it's so far be gone that it needs a refactor sure well so that's that's a good question right because sometimes the wor the worst cases like this is when a system or an issue is like almost fixed right where like a developer comes in is like oh I have a fix for this right but what we need to do now is also look at these features to answer that question which is is the way the feature or the

83. 01:06:34

    system was constructed compatible with what the Pu is going to be or what the technology behind the Pu will be which is a combination of the streaming technology that we've built before that has been a very complicated and complex system even on its own without server mhing and then the systems under streaming and mhing together and so there are systems in the game that are not adapted to that approach right and so when we look at a at a one of the key problems that we see in the game we're going to look at the underlay system and if we need to rewrite it'll be because yeah like it needs to be Rewritten to take advantage of streaming and server meshing properly right versus trying to patch and kind of

84. 01:07:23

    Cobble it together right to make it work so that's how we make the decision is really we're going to have to look deep in all these things to decide which ones need we've already done some of it like we we're calling out some stuff already I was lit I Lally saw a meeting earlier today where where you you're literally presenting every single engineer in the company and and and and and like okay it's like and it was a collective effort it was okay so it's where do we put this person where do we put this person I've never seen in the in the time that I've been here such a granular kind of over arching okay no we need to re-evaluate everything all prior assumptions are done let let's look at all let's look at the entire field of players we have and decide who's supposed to go where best and like stop everything fresh start that leads me to my next question here so we're we're a

85. 01:08:11

    company of near or over a thousand people I can't keep track across the stud in the world but there's still a limited number of these engineers and programmers in the company delegating them between these tasks is hard enough yeah but you've also got this thing called Squadron 42 that's right over here and in in in in October we flashed up a big 2026 on that yeah um I was there when I was I saw the 2016 way back when and and Chad zamza spoke for all of us with like what so we we we we stayed away from from from pointing at a year for so long because we grow mostly underestimated in last time but now we've done this so

86. 01:08:59

    you've got this gold po gold poost I was going to say Hangman's news but gopost is much more positive sitting here yeah that's got to get done too yeah so what is this split like is is this just an even 5050 split this many of them are on Star Citizen this many of them were on Squadron 42 is it 6040 is it hard is it flexible is it is is it there is is are there Joker cards that bounce between the the projects uh does does Chris have a veto every 3 months where he gets to rearrange the uh the people how does this split work so the this this that's an interesting question because it it is you know we are making two games uh there's some good news right because a lot of the gameplay systems that power the actor you know and that power the

87. 01:09:49

    core of the game are the same between the two games and so improvements on one end result in improvements on the other one minus plus or minus the online versions of these features right uh in the in the let's say we take one team right which is the most important in this one like the the gameplay teams right that's where we have to be more fine grain right we there's this thing that we have to be careful about which is Contex switching right obviously building a single player cinematic game and building a multiplayer MMO like game are similar but the context switching between the two projects costs time right so you want to find a scheme where you have the resources on what you're trying to achieve at each Milestone like squ 42 has its internal Milestone and this

88. 01:10:37

    Cadence needs to respect this year which is extremely aggressive in order to get a game completely uh complete by the end of the year and for ready for 26 release uh and so there are resources that are you know more suited to the type of things that s42 kind of needs right now uh in in its polish phase and its uh you know finalization phase for the for all the levels and all the the the chapters and so obviously these people will be favored uh to work on squand 42 so there's a a a series of people that are on there I don't have an exact percentage but you know it's not 50/50 uh because there are also things that are built on the PE that are beneficial on that side and so the thing that are if if Yen was here with me he would say the thing we need is flex right and so Flex

89. 01:11:27

    is when we need reinforcements we can get them right and so we have a a model now where there are a a a series of Engineers that will work you know dedicated to Squadron uh but we can Flex on both sides depending right on the milestones and the two projects have now a a a a Cadence that's adjacent and so we kind of know at every Milestone what Star Citizen needs and what Squadron needs and so we can Flex accordingly and these Fe features are kind of uh a lot of those features are similar in both ends and some of the bug fixes that we need on SC are also needed right on on both sides especially actor related bugs and so that's a tricky balance right I I I hear what you're saying and and and it makes sense and I I was in that meeting where Chris and and yans were talking

90. 01:12:14

    about the need for flex and stuff and I and and and I I agree my first show in two months maybe it'll be my last one how did there there's a phrase that that comes to my head while I'm listening to you say it's it's this jack of all trades and master of none thing now that's not precisely analogous to this but if people are bouncing how does any one thing get done like like like that that that that's the first question I have I I have there and I know there's no easy answer to this but but you agreed to come on the show and you agreed to take the uneasy question so I'm I'm taking advantage of our of our decade long relationship and and putting you on the spot here it's yes it's absolutely important if if if if this

91. 01:13:05

    thing in Squadron has to be done because it will unblock 50 other people so yeah okay we'll take those two people from Star Citi for two weeks and get it done and that's a worthwhile use of flex and I understand why that exists and why that's important but how do you keep that from becoming just this and and and we just left with this morph two amorphous blobs that never get finished yeah the the key is in a specialization right so uh we have teams like I can give an example like the mission system team for example are highly specialized in the online mission system right and so oh and Elliot malie don't leave him out and Elliot right the Kid Rock of missions but the uh the this assignment

92. 01:13:55

    right uh you take the resources that are good at one element that one project needs and you can dedicate it then you're not losing anything from the other side right the flex obviously is there in your extra kind of chunk of resources right every team has like core members and extra members that can flex and so the the idea with this is has been to make a plan where we can actually focus on the things that matter and we pick what we think matters right so our player experience team on the SC side helps us like for the bug fixes that we're going to be looking at in priority uh we know what those are so we make sure those are covered so it's about not just prioritization and assignment and just blatantly going SC Squadron you know it's really about

93. 01:14:44

    looking at what are we going to what do we want to achieve and on SC we want to achieve a better playability and so we're going to commit resources to that the flex will come only when we ship right that's the moment where Flex actually happens so end of Milestones uh and so the impact at at this will kind of be minimal because we minimize the context switch on the key resources like leadership is a key part of this the right leads in the right places make a big difference okay those ones need to not move that's the yeah no it it makes sense it's not a question with an easy it's not a question with an easy answer and it's not a question most marketing shows would probably ask their exacts live on yeah the internet but like I we're going for it we're

94. 01:15:34

    going we're going for it um I want to talk about one more topic before we get to we before we get to a topic that I've listed in my notes as persistent buggy verse right uh and that's Alpha 41 so last week was was just last week right we released Alpha 401 um I saw people online people that if if if I'm being as Frank with those people as I am with you probably should have some people expected the Silver Bullet patch like like like there was a letter in December and they thought in January there would be a silver bullet where everything was fixed and and and and and and I personally find that that

95. 01:16:22

    expectation a bit silly on the flip side of that that though these are people who have been waiting yeah they've been waiting a long time for us [Music] to figure this out for for for for for us to have for for for us to to to to to come in from the cold and and and and and realize that so many years are doing it this way it's time to switch to do it this way so I understand the the frustration and everything is Justified if if I'm being completely honest it was not a silver bullet patch sure this 402 that's that's it might even go to PTU today or tomorrow I we're here so I can't see the chats that they're trying to put on P you know for the end of for the end of this month uh will not be the next Silver Bullet either right

96. 01:17:11

    uh as I said at the beginning of the show this will be a thing where month after month we uh we we we we patch we patch we patch without new features but just new content built on existing features and Bug fixes bug and slowly we start changing the temperature of the water you you know of our persistent Universe what was my question it sounded good anyway but I'm sure I was going somewhere is this is this the is this the approach is is is this the when can we start so I see what I read and what I read is what I wrote for myself is done dumb I don't I don't I don't want to ask that question I I

97. 01:17:59

    want 401 did not solve all the problems no is it a legitimate improvement over 40 in many areas it's a drastic change yes uh especially for the issues that uh affected a lot of players during the break if you played during the break on 4.0 preview like a lot of very dramatic problems have been solved in for a or hot fix right because the 401 also includes all the hot fix from 40 so it is fundamentally you know a better patch but it regressed in key areas that are very problematic yes I'm looking at a chat right now drastic with with with with a lot of emoji from from the

98. 01:18:46

    observability standpoint right from not from the I think that you know in terms of the forward- facing elements of the patch that we have regressed in 401 but from the infrastructure level uh we have actually made a step forward and so it's it's a 5051 it's a two steps forward one step back kind of thing yes I would say so yeah and the one step back just happens to be the part that the players see correct yeah okay uh uh I know there's a there there's we're we're about to jump into this the buggy verse just I want I want to I'm on the verge of something here and I I'm trying to figure out what it is in my head what I what I what I want to say here um I guess what I really want to ask

99. 01:19:48

    is okay it wasn't going to be it's not going to be in one patch it was never going to be in one patch the the any expectations that would be in one patch or two patch frankly a bit silly but we're all when I say we're all I mean I mean not just the people who play the game I mean the people who make the game too we're all frustrated yeah we're all tired we're all exhausted of logging in and seeing people stuck at one in position seeing and seeing elevators not showing up and and people falling through trams and stuff like this it's not going to be one patch and it's not going to be two patches maybe it's three maybe it's five maybe it's seven I guess the question I really want to ask you before we move into the J section here is is this plan

100. 01:20:38

     going to work I think so but what it needs right is and I think Chris's letter kind of sets the commitment that's what we needed right uh going from the previous way of making the game to this new way it like like a company of our size takes a takes time to shift gears and change direction and so even though you're like okay guys we're this is our Focus now reorder priorities assignments you know uh it takes a while before it actually takes form right and you need you need the people in the teams to really grasp what we're after right in terms of increased playability what it means for them in the day-to-day right and I think that we have the commitment from the leadership to change

101. 01:21:27

     this and we have the means to do it so uh in order to support this effort right it's not just we're talking about priorities and orders and uh that's one part of the puzzle but what the other part of the puzzle is to make sure these teams are properly supported to do this right um it's in um paying the tech depth that's been playing the project for a long time that we've been kind of dealing with putting in the corner right tackling some of it but then you know letting it build up and I think that we need to attack that head on and to support our teams it's about helping them improve the output through better process better pre-submission checks better validation building the tools that will allow them to have confidence

102. 01:22:16

     that when they put a fix that it's fixed or that when we say a bug is fixed that it is fixed which is you know let's just say that been an issue right no yes yes no many have been the times where I've seen everybody from you to Chris to yen in the chat saying you know they they they read they read a report on Reddit or they see something on Twitch or YouTube and and they find the they find the jir and the jir says it's fixed yeah it's like I thought this says it was fixed and you know that happens for a lot of reasons we've talked a lot about how there are multiple causes for the same outcome and yeah that cause of that outcome was fixed but now we've just exposed a second different cause for the same outcome um it's it's it's frustrating we're doing a lot of we're doing a lot

103. 01:23:05

     of the problem is jir something in the chat um we're doing a lot of talk here we're doing a lot of talk here and we still got some more talk to do the proof will ultimately be in the action in the action in in in the in the month after month P patch after patch there won't be a patch every single month like I said we're we're we're currently looking at about 9 or 10 on the schedule we actually have a meeting tomorrow where we're going to kind of hammer that down and have a better idea um certain things certain things affect the patch Cadence you but um but yeah but but but more patches this year than any year prior uh but with less of the feature workor far less of the feature work and more of the content and and the and the bug fixes the content that doesn't require new

104. 01:23:55

     features like somebody somebody in the chat was just saying do less events it's not the events that are the events that are built on new features that events built on existing feature workor and stuff can can can still be fine that's part of what we're looking at uh so like um the proof is going to be the proof it's going to it's going to take it's going to take this year ultimately it's just the product right that's the thing that counts it's what's the output right does the output get better are we still dealing with the same problems that we've seen before did we actually solve any of those issues or are we still going to look at you know yeah uh you know interrupted missions or you know any of these bugs that are plaguing us right and uh I think that that's the key right and that's how we should be measured it's in does this new method work does is the product

105. 01:24:42

     improving right and in its quality yeah uh kid Casper first time chatter says did a Dev ever played Siege of orison yeah sieg of orison is a great example of of of an event that that took substantial feat feat work and the feature work that was underneath it was not was the stuff that was not ready for prime time you know do you know it's it's again it's not exclusively the idea of an event that's bad it's how you build the event whether you build the event on current existing proven Technologies or whether you try to do something crazy and we're still going to try to let's be real I've been here for too long to to sit here in front of everybody and say we're going to stop trying things new that's not who we are we're going to dram atically reduce the risk we're going to dramatically reduce how far out over the edge with new

106. 01:25:30

     things that we do there are going to be moments this year where we still try something new here and there and hopefully it works better than than in the past because of our new tech preview situation because of of our of of the new methodology uh there may be one or two moments this year where we try something new and it still bites Us in the butt that's just I've been here too long to even remotely Hint it saying otherwise but on the whole what I've seen from people at every level this year is something I've not seen in the 10 years I've been here and that is this Collective realization that we have to change and stuff so I I'm I'm more hopeful for this year than I've been for other years uh it's not going to be a year full of a bunch of exciting new features it's going to be a dramatically

107. 01:26:17

     different year with different content and stuff it'll be interesting to see how it works it's scary because we've never yeah it's it's completely different for the project for sure all right persistent buggy verse yeah um okay so it's not all going to be fixed in one or two two or even three patches but if I'm allowed I think I can speak for everybody in the community when I say what the f about some of these longterm issues so so so the these long-term existing issues that have plagued the game since 324 or longer so uh I got together with folks from the community team uh Tyler and and and Lenny and and and and and and Freya and all those folks I got together with

108. 01:27:05

     people from the player relation team uh uh Bayer and and and and and and his folks and and a got a few of these from Chris Roberts himself there you go I got a list of issues that I want to ask you about and get some status updates on sure uh this is basically the part of the show uh where I hold your where we Dive In but to the fire sure you ready yeah so in no particular order no there is an order hang on before we start right so I'll be the messenger here so we did a little prep right we looked at a couple of these because I needed to prep I don't unfortunately I don't know everything you don't know the current status of every single major bug in the game Ben no no I do not I wish did I mean that'd be great but I did do the Scavenging hunt to make sure that we got at least an update on most of those and so that's you know we'll we'll get

109. 01:27:52

     out of it all right go for it I want to start with server andur and stability yeah and I'm just going to read my notes here we've got frequent login issues including being stuck at position one in the queue for extended periods Shard loocks the servers becoming unresponsive or dead with players extensively discussing strategies for finding working servers often relying on joining friends in instances so let's tackle these one by one y login issues and being stuck in position one so this one we're actually deeply investigating right now it looks like so we have a and I'll go deep with technical details here to to probably explain not just the status report so uh matchmaking right so when you click on that button join the universe you're talking to the matchmaking service or the matchmaking

110. 01:28:40

     system he will find one of The Shard instances that's available right all these shards report their health uh to make sure that they're you know doing good that if one of their servers is not good or their FPS is not in a good condition and it always reports a score uh and we will always try to match you to either something that has Shard Affinity to you or uh Shard Affinity well it's either your player is already in that Shard right that's the ultimate Affinity or you've been in that Shard before so we try to have a level of of Shard Affinity um and um now the problem with stuck position one is that we've matchmate You On A Shard and now the matchmaking service is basically telling The Shard this guy's coming that's called a process called authorization

111. 01:29:28

     and for some reason now the latest update that I've got this morning on this is that the the authorization call sometimes will take multiple minutes to close out and so we don't know what the cause is on this one yet it's kind of a a new identified condition but that's one that we need to hot fix absolutely on live in order to get that back in there so that's one that when we figure it out is fixable by hot fix some things require a patch some things require a hot fix that can be done under so that's a thing that can be fixed hot fix any day of I can speak a bit about the hotfix uh how it works right now I don't like to use the word hot fix for everything but that's the word we have for now right because not all fixes are necessarily hot fixes there some of them are just they need to

112. 01:30:15

     go out um but so the uh what we cannot fix right now is a server change so a change that go on the game servers that's the one of the easiest things to roll out so we can actually make a code change and then we start deploying it and then your game servers will slowly start updating right and that is the easiest one to roll out uh but there's some requirements for this there must be no data change no asset change and the network compatibility of the build needs to be maintained and so that means that there are some things we cannot hot fix uh because it would cause a client compatibility uh and so there are some bugs that we're like we have a fix we can't roll it out right we have we have to wait for the next patch window well

113. 01:31:03

     yeah exactly right which uh we are talking about in in interest of this to have now like maintenance Windows where we can actually roll out uh data fixes right where we now need to force players to upgrade their client but that would allow us to actually you know take the environment down also do some maintenance we do rarely do live maintenance on the environment just to you know change things we kind of try to keep them the way they are but this will allow us to actually perform Life Changes more aggressively uh so we're trying to nail the process now uh for that but and so the second kind of of thing we can change is a hybrid fix now those are the are very hard to do because a hybrid restart will disconnect all players on a Shard and so we can't upgrade A Shard that's already running

114. 01:31:51

     and so we have much fewer game shards now than before uh and so that means that you know disrupting them will have a drastic effect on the on the live environment and so those fixes are are a bit of problem so typically uh it reflects in the number and the name of your Shard when the hot fixes roll out chat is just spamming do it just do it one of the things that we don't do that a lot of MMOs do is have that regular Tuesday downtime or something like that and that are the times where those things are necessary for other games is this something that's something we've considered yes that's what we're trying to prepare for now is to have an official maintenance window that we we we're not thinking about systematically doing a downtime but we're thinking

115. 01:32:39

     about scheduling it and then we'll let players know before that happens and then we'll be using those uh those moments to do infrastructure changes or upgrades that normally we would have to wait for um I that would be I'm just looking at chat right now there's nothing you've said in the last hour and a half that's getting the reaction they're just like yes give us a maintenance window it's your [ __ ] four no that's five is that five I think five um load times went way up yeah in 4.0 yes way up that's yeah especially for pyro pyro side players would actually time out of their load yeah what gives so uh interestingly

116. 01:33:30

     there uh now that's still being investigated but the current hypothesis on this is that the way the Pyro system was built uh so first of all the the requirement when you connect right what the what the hell is the game doing right in that loading screen and then you you open the console and his stuff flying around what is it doing right it's going through this context establisher set of tasks uh which are like all the phases for the connection right uh currently when players load into pyro uh and have a load a long load time it's seems to be related to the way the system was built where there's an entity in the Pyro system that holds the space scape for the system right now we're peing behind the currens of how

117. 01:34:18

     the levels are made but uh there's a requirement for the C connection right right that requires that all the entities that the client needs to load need to have an authority assigned right now so this is directly a server mesh change and so if the if an entity that you're loading doesn't have an an authority yet which could be any server in the mesh right uh if one of them doesn't have an authority yet then the load time will have to wait for this entity to gain an authority now how can this entity not have an authority so our current hypothesis is that if the server that is hosting that pyro spacecap entity which has a giant radius so every client loading into pyro needs

118. 01:35:09

     that entity if the server that hosts that entity is currently rebooting because it has a crash on it then all players will have to wait for that server to do the full provisioning loop boot up acquire load the data Acquire The Entity and then players will load in for some players that overrides the that bust the time it needs for the timeout for the log the login so that's our current lead and we think we have a fix but uh you know we're obviously that's another one that we we would roll out I think it's a code fix on the hybrids that we can actually roll out for that uh while you were giving that answer Chris Roberts just messaged and says Beno are we having login issues again and just and Yom is like uh I'm poking Ahmed because beno's on talking about

119. 01:35:59

     this right now so get off beno's back Chris delete do not hit enter um no okay so um I know I see people who are stuck in this thing who were waiting they shut their client down and they log back in they shut their client down like is that is that helping their case or is that hurting their case in in this I have a lot to say about that so I'm head out so un un you know everybody kind of has a recipe for fixing login issues right A lot of them involve disconnection Shard hopping loading into AC rebooting twice

120. 01:36:49

     you know printing a document I don't know right um that is actually one of the things that we've identified as missing in our testing process so we we do some load testing before we launch a patch uh and so we test with clients Lo logging in logging out and everything's great what we're not testing is Lune client persistent Universe ala 4 lunch client persistent univer AC right altta four client persent and so a lot of the issues that we've seen in the four release when we put in the persist hanger flow for example that allows you to log into your hanger the system for this needs to move your your player we call it unstow your player Into The Shard it also needs to

121. 01:37:37

     unstow your hanger now when players are constantly lugging in and out of shards now you've got multiple shards doing this process for you and they need to complete that process uh We've identified a ton of concurrency issues with this in 4.0 uh elements that we didn't test for because we're you know you test a you test the load not the repeated concurrency and so uh that's one thing we've identified that we absolutely need to add to our testing process because it's it's one of the other things I've talked to this about this before the difference between PTU and live is like a 20x increase yeah uh that's also one big difference is that on PTU players don't Shard hop that much no one one of the forever things that it's if if there's a double-edged

122. 01:38:28

     sword of PTU and there are many but but this is like the one that's always amused me people are more patient on PTU than they are on live because they recognize that they're they're helping us they're they're helping the development of the game they're doing a job and that that patience is necessary and wonderful but it doesn't make for an effective test of live and it's yet another reason we have we have to go to these live environments sometimes and we don't discover these things to live environments because live is where is where hide comes out right you know so many people in PTU are Dr je are Dr Jackal and they're they're all nice and and and and and knowledgeable anded and then and then live environment is where yeah it's yeah it's it's interesting

123. 01:39:17

     it's right well that's the one of the key differences right and so a lot of the issues that you know are plaguing us right the second right are still remnants of that that we we are tracing a lot of them I spent I mean a lot of time on Spectrum during the break during the work the the vacation workation uh you know actually digging into a lot of those to be able to make sure that when the team would come back that we'd have you know some traction on some of those uh last question on servers because then we're going to get to the sto character stowing and repair here yeah what are Shard loocks and why do servers become unresponsive well so that second part's why little little little question here all right so Shard loock I mean we we never really call this Shard loock this is like a

124. 01:40:06

     community term uh you're never really locked into a Shard but your character can only be into one and so uh if you uh get match made into uh one of let's say us e1b 010 that everybody body's favorite Shard which it is true that Shard is cursed I mean it's right now it's always 01 zero I don't know why but that one is always the one um take it out to the shed and shoot it it doesn't matter doesn't matter which which patch the zero us e 010 Shard for some reason is the is the bane of our existence anyway uh so you go play there your character will get unstowed so hybrid will unsto your character moves it from the global database that allows you to go anywhere to that sh now you

125. 01:40:54

     are according to community Shard locked right because you're playing in that Shard right now what's supposed to happen is that if you quit the client or you get disconnected or your client crashes when you come back you're going to make match made on this Shard again right now if that Shard is no longer available during the time where you decided to come back or whatever uh then that's going to give you a nice little 60k error right which will say hey your character is there uh that Shard is no longer available now what that means is that behind the scenes there's a process that's taking the if The Shard is crashed there's a process that's currently processing that Shard and will stole back your character so it's time time will fix it right in this case um during the 4.0 preview some players got into a state where the

126. 01:41:42

     character could not get stoed back that was a back in uh a bug that prevented that from happening and so their character were was kind of seemingly corrupted and so we had to go in and fix a lot of those that costus a permanent 60,3 I think is the one uh where you basically some players were stuck in that mode for a long time and so we had to actually do manual actions to to solve that uh but otherwise in most of the time time will make your character go back to the sto position and then once you're stoed back you're able to go back in there so that's what we call or I think that's what the community calls Shard loocking yeah it was definitely the 630 it's it's building reputation you know it's coming for 30k is Crown slowly um let's talk about elevators and

127. 01:42:32

     tramps all right talk about elevators and trams uh we didn't talk about repair you don't want to talk about repair well actually that question was mostly about the 6030 and stone you you basically covered all that stuff that's kind of true so elevators and tramps yeah broken or missing elevators in various locations preventing access to hangers in other areas or simply trapping players inside trams that either let you fall through the floor or Yeet you into lower orbit my question to you what what the [ __ ] that's six um so brief history right of Transit this a very old system that's been in the game for a long time and

128. 01:43:21

     uh Transit in the context of streaming is already a non-trivial problem to solve because the and I need to dive into the why so to explain a bit you know the complexities behind it but so the transit system is composed of multiple components right you've got peripherals so that's for example the button you press to call an elevator or a tram uh it's the indicators that say Hey the tram's coming in you know this amount of time those are peripherals you have gateways which are locations where a carriage can stop at so for example if you're in lville uh you know and you go to the train where the train stops that's it's like a station that's called a

129. 01:44:07

     Gateway uh you have carriages which are either elevators trains trams you know all these things that move along a path uh and then they go from Gateway to Gateway and they respond to orders that are submitted from peripherals all sounds nice and D what's the problem right so in in a in any game any other game that doesn't have full on level streaming uh all of these things are always loaded so the manager the piece of code that actually is managing the the fulfilling the orders and moving the carriages in general or has the logic of the carriages is always all this network is streamed in it's it's all loaded so it's always available it knows where the carriages are it knows knows what's happening everything is there in our

130. 01:44:55

     environment because of our socs technology that allows us to have more performance on the servers because we unload the rest of the zone that means we unload those carriages so think about your Lowville train that you see stop at you and then leave on the server if nobody's on the track it doesn't have a carriage there it's unloaded right and so the current iteration of the system actually destroys that carriage and will recreate carriages when it needs it right right and there's been a ton of issues with this so that's the first part right is the streaming part uh that makes it a bit more complicated the second part is that the way the system was built initially uh it would fulfill these orders to those uh send and orders basically I need to you

131. 01:45:43

     call the elevator from your hanger I need to send a lift there so the system is going to tell the lift to move the lift move moves on its own right so you will jump and then do the actual path we want those to be physicalized so they're actually following certain paths uh now the issue is that combin streaming with these orders and state is lost in certain conditions and so the the system just needs a whole lot of changes on its communication layer to be reliable and it needs a fundamental change in how it fulfills these orders instead of just sending a message to a carriage and hoping for the best the transit manager needs to be actually uh always tending towards the state that it

132. 01:46:33

     should be in so that it can self-heal for any kind of issues right uh whether creating an the The Carriage failed or streaming out and losing a piece of it the system needs to change to be able to properly do that so that's what the new system that we're consider that we're writing is going to do is going to solve these communication issues with streaming and it will make the system self-healed but we also need to do fixes to the current system immediately right to get that working so now we we do have a crack team on this right now where we've significantly increased the size of the group that is looking at these at the current elevators issues which we have absolutely regressed on and they're looking at adding the self-healing component I just talked about directly

133. 01:47:20

     in the system so I'm I was happy to hear that we have now proper reproductions for at least a couple of cases where the gateways uh are basically becoming defunct and so you call the elevator and it never shows up right that's the number one case that we see at the moment or the other one is a lift shows up you press a button and it doesn't move right so that's when the the transit manager lost this carriage or or the Gateway is no longer usable and so now we have those reproductions uh and they're not just happening at uh to players now we have an actual case where we're able to actually go down and we've seen the what actually is going on so we can put the self-healing things I just talked about to fix it so in the past when we've

134. 01:48:09

     talked about some of these issues and why they haven't been fixed or whatever it's because we've had this aversion to doing double work yeah we don't want to we don't want to work on a Band-Aid when we know the long-term solution is ripping the whole damn thing out and replacing with something else try to do minimal change on the existing and said what you just described though is doing both though so so at present I just want to confirm this here at present some amount of persons is work are working on writing a brand new transit system built for built for the server meshing reality correct and all this stuff yes and but at the same that's going to take X number of months to do not wanting to wait for that because we need to fix we need to fix now there's also another

135. 01:48:56

     group of there's also another group of people focused on a short-term self-healing fix to the current system yes that may not fix everything but will dramatically improve it is our hope yes well we want it to function the way it should function which is all the time like reliable right now it might mean because we are still going to have the issue of the stream streaming out and some error handling that it might take longer for a lift to arrive in certain cases but it will arrive arrive and it will because it'll have that error checking it'll have that that self healing so the idea is that it won't just not show up it won't just won't you won't just fall through it it'll just you'll you may wait twice as much as you were used to in those cases but then it will show up correct correct well that would be something at least at least for the short term yes that would be something okay and that that helps me

136. 01:49:43

     explain why we're doing both right no we need doing short FES and long okay that makes more sense to me um somebody asked uh uh do the do the elevators in Squadron 42 work they use the same system but a lot the streaming bubbles in Squadron 42 a lot higher so they will get stay streamed single player y uh and then many people are like just build stairs like who knows maybe maybe maybe maybe we can convince Ian and Nick will be all stairs or ladders maybe I don't know um hangers yeah hangers all right failing to spawn ships correctly platforms getting stuck ships being destroyed while they're stored

137. 01:50:35

     yeah I saw that one I'm like what like uh or hangers being assigned incorrectly one by what where do you want to start all right I got to come clean first on the ships being destroyed when stowed that was 100% me so uh we were looking at at some hold on look no look right in the camera yes it was you Beno yes uh so the that was a backend change that we did uh to try to alleviate some of the load that we were seeing on some areas and and one of those side effects that was unplanned was that the location ID of ships weren't being updated uh when they

138. 01:51:24

     changed and so the way the asop terminal again you know moving Sands right uh does its reconciliation which shows you uh what's available for you it actually uses the location as the key element to know if a ship is actually you know available uh not only is it St but it needs to be in the right location uh now this change that I had made uh for optimization did cause for a couple days the issue where the locations weren't updated and so all your ships that were stolen locations would show as destroyed but they were there but they would show as destroyed in asop and so yeah I'm going to take the blame for that one that was me is is is this like is this like if a branch falls in a forest and nobody can hear it does it really you know the ship's not technically destroyed but if it reports destroyed and you can't call it because

139. 01:52:12

     it reports destroyed isn't it destroyed even if it's not destroyed yeah uh well actually claiming it would actually destroy it too so technically it's destroyed right but you resolve that one yeah that one's resolved okay so so let's get back so let's talk about uh uh failing to spawn ships correctly platforms getting stuck right so let's talk about hangers first right okay so um I mean people that play more on the PTU know how their system works but it's actually a bit special in that um first of all the hangers are instance so uh they're instance on your Shard you have access to your hanger based on you know your ownership of it uh but all the hangers on a given ATC Gateway so these are the actual hanger doors on the

140. 01:53:02

     stations right uh are all stacked on top of each other right so that's why that there's been bugs in the past uh during the PTU cycles for 4.0 where canar oh yeah that that was that and that bug took a while to fix uh because it required technology change uh but that was also crazy like the sound was just you know incredible and so the what that means is that when you're in the station and you want to go to your hanger you have two options one of them is use the lift now we just talk about the lift right when you call the lift it will actually unstow your hanger create your hanger and then give you permission to go there right uh right the moment you hit that button so there's a lot of things that need to happen in right

141. 01:53:49

     suggestions then you hand through your hanger uh there's render cing that comes into play that will make sure that you only see your hanger even though everybody's on top of each other that physics only affect you right uh so they're all stacked uh now one of the big problems that we're dealing with right now and we've we're we've been diving it into it for for a while is that the ship elevator you know has a couple of issues right I've seen I've seen you know an Avenger rising from the dve through it I've seen you know uh ships spawning on top of each other um and so there's there's two problems behind it the first one is that the platform itself is uh it's moved right it's not a physical

142. 01:54:38

     movement so it's not physics driving the movement the ship is physics enabled that is a recipe for disaster correct when you see flapping like that it's like the combination of things right and so uh uh so we're we're currently like discussing a solution because there are limitations that the way it's made is the way that they were limited to make it but we need to fix it so it's either both elements are statically moved or they're both physically driven you can't have this combination that's where that's what causes all these things so that's why sometimes server FPS is good performance is good all your ships are going to go up that platform of things great and some other times you know half the ships in the is in the is in the platform form and that's an issue So currently we're rejigging reang the layout and the setup

143. 01:55:26

     of the hangers to to set this up properly so that it never happens again uh the unstow issue is actually we think at the moment a a bug where in the timing of how you when you leave your hanger we need to Stow back this hanger right because if you go to another Shard we won't you to have access to your hanger the timing of this if there's a ship on the pad when you leave there's a possibility in the timing where the ship will be stowed with the hanger instead of being stowed on its own and then the hanger is clean and that condition is not properly detected when you unsto the hanger again and so in the cases where for you as a player you're not going to see a difference in that you leave with the hanger on the ship on the pad you

144. 01:56:16

     leave your hanger you come back on another Shard uh and you just go to your hanger your ship's going to be on it but if you call a ship from the station and the ship was carried with your hanger up and down then the ATC is not going to know that there's a ship on that pad and he will attempt to spawn it there and that's where crazy happens and So currently we're looking at that complete cycle to make sure that the timing is correct and there's also a couple of other issues related to this some Corruption of data can happen in that case and so that we have a an entire team dedicated to the hangers right now just making sure these issues get solved because they're affecting everybody I'm happy to report though that one other issue that we had with the hangers uh got solved related to how you exit the hanger so um usually in a

145. 01:57:07

     ball of flamy wreckage well or or after 25 minutes of queuing where the system tells you hey in 100 seconds you'll be able to leave and so uh so we did hot fix uh a a change to the way that what I call the hanger cues uh are managed right so the again the ATC manages the cues it's managing how you exit your hanger which doors open stuff like that and at concurrency so when a lot of players are in the same location leaving and entering this queue and those timers would get completely out of whack and so it would either Grant too much time to someone to leave or enter we're talking 10x the time or give you literally three seconds to leave and then paper players

146. 01:57:57

     would get stuck in the in the doors where the doors would go and then would try to squeeze in at the doors would squeeze them out and then you know the rest is a little bit of a party so we hopefully have something on live already alleviating this but we're still tracking to confirm it completely that it doesn't happen again uh for the folks that are still in the chat saying please just give us staircases uh the hangers are instanced the stairs would just still go to the elevator is a portal between the the the the the real world and your instance world even if you spent five minutes going up and downstairs you would still have to hit a portal where where you have to queue and push a button and it makes a portal for your thing that it's the same thing an elevator is just the mechanism uh stairs don't solve it unfortunately uh hard

147. 01:58:48

     work of developers and Engineers solve it um what's the obstruction tuning and and stuff it's it's I saw um I I I've I've seen images internally of uh elevator won't go down and then and then Brett or somebody on on player experience whatever will jump in and go under the thing and you they'll see a dragonfly stuck under the elevator that stops IT from going down and so the problem we talked about about physical physicalization of the platform is Def definitely related because then the obstru code that will a you can't you know bring down the ship elevator because there's obstruction initially was like if there's like a speckle of dust under that platform it's not going to move right which was let's just say a bit too aggressive way too aggressive and so I

148. 01:59:39

     think at this point we're doing cleanup to just try to Stow back if there's if one of your ship goes on the platform we'll deal with it and stow it back but it still needs work to make sure that you know even if I mean I remember one of the early 4.0 preview bugs was like a water bottle would clip through the platform and then would prevent The Hanger from functioning completely and so now we're trying to again in the interest of self-healing for these systems right it's about let's not just work at preventing the condition let's also work at what if it happens anyway right we need to do aggressive cleanup to keep the system working like the hangers the elevators uh The Hanger doors the ATC systems are all part in my mind of what called the traversal of the game right and bugs that affect the traversal of the game that prevent you

149. 02:00:27

     from going to your ship and leaving or entering in and refueling uh they are of the utmost priority right they have to be solved because if that doesn't work then the game doesn't work they're critical and and lastly I'm looking at the this jur for the hangers being assigned in correctly uh location ID it's got your name on it well I've been investigating that one oh okay yeah you're not responsible I'm not responsible for it yet until proven otherwise but uh in this case the um it seems that there's a condition where you can have more than one hanger in a location of the same size which should not happen and when the system gets in this state it might pick one or the other which is which can lead to you not having a hang a hanger at all so that's the thing

150. 02:01:17

     that we're looking trying to find out how it actually happens okay uh second batch of hanger issues sure there's a lot yeah hanger lifts that clip player ships and Destroy them doors not opening and closing correctly losing your personal hanger completely and and I I the overlapping with the physical audio we've covered that one so so hanger lifts that just clipped a little tiny corner of my spaceship and make it go explode sure um what's physicalized there Ben well so on the I was going to touch a bit on ship cleanup and not NE ship cleanup but density Cleanup in general uh we talked about the physics events right uh in the past uh earlier in the show right so the physics events are a godsend for the

151. 02:02:07

     server performance because we evict all these debris that are causing intersections slowing down physics uh and so that's one of the reasons why we've been able to get performance of the servers up in general now one of components well the same system is actually responsible for a pink glow that shows up on a ship and will just pop it right and so uh that's part of the of how we try to resolve the physics intersection for ships right and the way it works right now is that it has a smoothing number of frames where if the physics engine is like heavy heavy heavy heavy intersection at some point it will go like okay like something needs to happen uh the the way we handle that event is still a bit too aggressive

152. 02:02:54

     right now and so in some cases it triggers a heavy intersection and will pop out the ship but if the we don't want this to happen to attended vehicles ever uh and so we're working right now at adjusting that flow so that only attended unattended vehicles in full intersection for a very long number of frames are cleaned up and so right now it's a bit too aggressive for this this part but it the distinct glow uh pink glow is is is we added that to make sure we could identify those problems which we expected would kind of happen uh until we tuned the system properly um we are at two hours wow I want to I want to keep going yeah let's go Pete Dave Lauren are you okay honestly I'm not even joking if you

153. 02:03:45

     need to go just leave the camera switched on Beno nobody nobody needs to see this face anymore just leave the camera switched on Beno we're going to keep going are you cool okay but seriously if you do have to go do it just wave and be like it's on Beno I can see it um I want to keep going yeah inventory and item persistence issues right items are disappearing from inventories ships and storage containers this includes bought items looted items Mission cargo what's causing this what are we doing to resolve it well so the on the inventory system right there's uh or I should say on the item system so the way the entities are tracked there entities have entity classes that Define

154. 02:04:32

     what they are right and there are I want to say tags to or configurations we can attach to entities uh for things that we want to automatically clean up right again we're back on clean up right because it's important uh so for example AI uh you know characters will have an what we call an auto sto or Auto Berry policy Auto sto will bring something back to an inventory after certain conditions are met Auto Berry will destroy an entity if certain conditions are met right automatically so that's our ability this gives us our ability to go this thing is good let's say for 12 hours and after that you know we can get rid of it this is true for like debris or things we spawn on the side or things

155. 02:05:20

     like that and so a lot of people have reported losing items for example from uh czs that were acquired in czs um that's you know at that point it wasn't even a system issue it was really those items were flagged Auto destroy basically and so some players looted armor and were like you know started using those armors and then when the conditions met for the auto bu those items disappeared right I almost lost this one and so that was a bad set up right and so now I know that a lot of those items have been uh retagged right and so the auto policy will not apply to them or not if they're attended again it's it's all about you know making sure that the right tags are on them so that explains some of the inventory things uh

156. 02:06:08

     there was one uh that was I was looking into with with a player actually recently where um there when you read your inventory there's caching going on right and when your inventory changes that cash is supposed to be expired at in certain locations for some reason the that cache like especially in czs would causing some problems so opening your inventory in the czs would be empty and then you open it again and then you store something in it and now other items show up that's a Telltale issue of the cash so we're actually revamping the query cache right now uh for these elements to make sure it's always consistently updated H your stuff is there it's just it just can't show it's cash as nothing yeah exactly um with long-term persistence it seems like there's this is this what it seems like there's always some level of

157. 02:06:59

     something disappearing yeah uh uh every patch yeah what's going on here so ltp is a fairly complicated system right uh but if I can simplify it as much as I can uh when you Stow something in an inventory and there's and and we take that item and we process it let's take a easy example let's say a single gun P4 you have a P4 you put an inventory that you own a record is created for your it's an ownership record that says this this player owns a P4 right now these records we write to all the time if you unstow that gun meaning you you put it in your hands right we destroy that record CU now it's in play

158. 02:07:49

     so you play with the gun and then you stow it back that recreates the the record so these records should always represent what you've got in your inventory that you own now where it gets complicated is that these things are can be hierarchies that contain items that then need to be processed and so we don't persist everything to ltp only a select set of entity classes that designers control will go into ltp for example ships or is a good example things like that now uh when we St a ship we look at its entire hierarchy and there might be things attached or Loosely left in the ship that are ltp the list correct and now they have to be added in so the uh also with ltp there's also sometimes perception issues that

159. 02:08:37

     things should persist right like for examp a classic case is a ship left in The Shard between patches what you're flying your ship in the in 324 and we release 4.0 if you leave the ship there in The Shard right you you don't have an noip record for it right the ship needs to be stoed uh in order to to be there uh when it we recover it but we do think that there are still issues with some of the recording events in in cases where ships are claimed because at that point when when you join it in the next patch you're let's say you have a ship you you bought with u and you store it at a location that ship actually it's recorded ID comes from your your long-term persistent record of that ship

160. 02:09:26

     and in the case where you claim that ship again we think that there might be some cases where the inventory the the record data doesn't follow and so the ltp record is not recreated and that could be a source of some of the Lost ships in the ltp uh data but overall we're not happy at all with that solution in general and uh a bit like you were talking about trans that we want to do a two-prong approach on this one of them is we want to find the cases that are left in the current system that don't behave the way we want them but we're also putting significant effort at a different system for this which is that we don't want to have to wipe the database at all so we want to be able to keep the our your inventories intact between patches and have a

161. 02:10:15

     forward uh compatible uh model where we can actually keep all this data there the way it is and then like check a file from perforce it's not you're not destroying it you're just corre out nobody else can write to it nobody else can mess with it it's yours and then when you stop it it goes back it goes back in so it's not a non-destructive yeah and check say and so the the well our goal with this is is there's that's a a fairly large Endeavor like we're not going to fix it in a second right we're going to still Hot Patch the current system to make sure that the cases that we've seen are are fixed but the the longer term fix is to make sure we only process the data that's going to be compatible so differentiate between the serialized variables on on entities that are uh important to a a ship from those

162. 02:11:05

     that are runtime only that's a big difference so those will be stowed in the global database and between patches our goal is to just keep that database like this so there's no oh we're getting records back from a thing that was been recording for 3 months it's like no it's the exact same that you have on the live patch and the new patch so there's no loss at all right so that's our our Target for this new ltp solution those are obvious those those are bugs and and inefficiency and stuff there's still always been this there's been this existing problem I think it I mean it's been here since I've been here but I think it's really started to to Really gain steam in like 2017 2018 and it's been forever it's this it's this unique item recovery sure I'm pretty sure there was even a thread

163. 02:11:52

     today that reached the top of one of those channels uh where it's just like you know we we get all these cool things uh you get cool things packaged in with your pledge purchases or whatnot and then you're too afraid to wear them you're too afraid to use them because you fall through one planet fall through one lousy planet and you lose it forever um we're not going to go into it here because there's another show dedicated to it but that is that is absolutely prioritized this year it's prioritized in a way that I've never seen it prioritized uh the talk is STO and the work is starting so we've got a signed off design uh and and people are on it right now and uh I mean the thing the thing that speaks that speaks a little

164. 02:12:40

     bit to also the change of mentality we're trying to affect which is tackling things like this right are not just bugs and quality issues but also playability ISS isues yeah right that need to be addressed as well it's going to be a three- tiered solution again we we'll have a show where we go through it all filming with Dean next week on like this there'll be a shortterm uh that'll fix that'll help a lot there'll be a longterm that'll that'll help even more and then the the super long term is crafting and Base building you you just being a you get blueprint for everything you can just make your own and Stu like that so you never ever have to worry about so it's it's a multi-stage thing but this first stage just got signed off and it's inactive development and you excited well it's just I I I joke not I joke I I Say it With all sincerity

165. 02:13:27

     that the same conversations the backers have we have internally just three weeks earlier 100% been I mean sometimes rinding our gears on sometimes it's scary for everything sometimes it's scary so so that to see actual movement on this right now is exciting y um one last hanger call out before we go the aggressive ship despawning mechanic uh sometimes ships are despawned while people are still inside them yeah that's that should not happen ever in fact nice to hear it said out loud no the system is made is supposed to be made that it we have a passenger count on each of those ship entities and if there's a passenger uh even if he's not piloting and even if it's not his ship we should not be despawning those ships at all and so that's that's definitely on the lookout right now as as to why it

166. 02:14:14

     happens I mean there's a couple root causes unfortunately for it just looks the same right again if you don't have the Ping glow right the Ping glow is the tell tell tells it's physics right it's the physics detection which we're going to ease up if the ship just despawns out of midair right I've seen this happen with ATC sometimes has with the hang the the timer problem that we fixed be before had an issue where it could despawn your ship mid-flight when you left which was egregious uh I think that one has been solved but jur is still out I want a proper confirmation on it um the one where your ship is considered and attended and you're still in it and it despawns is completely like that needs to be solved right like odd fix right away

167. 02:15:03

     okay Mission bugs yeah now we went through a refactor with 40 that reduced that reduced the mission options hopefully temporarily but we're still seeing a lot of the same issues that we saw before arriving at a location and having no AI spawn mission objectives failing to register and cargo disappearing upon completion uh the fight for pyro event going on right now is particularly affected uh with many PE players unable to complete it or various aspects of it due to bugs um save stand in chapter one was like walking through a Minefield trying to figure out which parts worked and which parts didn't uh the the 402 has more Mission stuff in the content it's like talk talk to me about missions missions sure

168. 02:15:53

     please Elliot this one's for you I gave you crap on the other one but I'm doing this for you man okay up out so uh let's start at the beginning right what what did we do in 404 missions right uh so the way missions worked previously is that every we had first one server with all the entities all the the 100 players playing on it when you played missions there was an invisible entity created on the server somewhere the location doesn't matter right uh that would manage that mission right and the way the missions work if you can picture it in your mind is that there's a mission management module and then when you reach certain locations right the when you stream in that location a

169. 02:16:41

     mission module at that location is going to actually kick in and has its own Logic for like if you're picking up a box for examp let's say a box deliver fore Mission you get to that location there's a mission module that pops up does the a pickup logic when you UNP pickup it's like oh you need to go to this other one then there's another mission module at that location now that's the way and then if I add Global events on top of that there's a global events manager on top of that that was like a global thing on the server now I've heard I've said Global multiple times here every single one I said every every single time I said Global is not compatible with server mching because we don't have a global entity setup right and so what needed to happen is a is a fairly large engineering task I I hate that we just said refactor that that we

170. 02:17:32

     branded this as refactor because it's really a new core mission system behind the scenes which is completely outside of the game servers and does the orchestration of what a mission is right and the global flow of emission hey you have something to complete there are phases objectives there's State tracking there's requirements and rewards right and so that Ser it's the mission system what we call it uh there's no longer a mission entity that's hold on the server that just manages the whole thing they all communicate to the mission system that then ticks the logic of the mission forward even though all these Mission modules written by the Kid Rock of mission uh can do all kinds of uh fun things uh you know uh at the each of

171. 02:18:21

     different locations so the fundamental work was to make sure that the mission system was functional in server meshing that was really the part of the gameplay element that was completely incompatible with server meshing uh another piece that we needed to solve uh in the build for for and as part of the mission system is how markers work and so markers are always a big issue have always been a big issue in Star citizens since the very very beginning uh and with server meshing became even more problematic so uh you know let me State the problem you're on a pyro game server and I'm on a Stanton game server but we I add you to my party now I want a party marker on you right my game server is doesn't even have your star system loaded like it

172. 02:19:10

     doesn't even know what's going on right and so for that we needed a piece of technology that would allow the streaming system to stream your entity marker on my server so I could actually see your position changing and so we've built all of this in place uh but that's a fundamental change to how markers work and now I'm going to dive into the why do we have that many bugs behind it it's almost like we need a a replication layer for for the mission system itself almost right like for for things that are tracking objectives across systems things like that right and so think the way we call those things like the markers are in dependently streamable entities uh which basically allows us to have a a marker on an entity that's you

173. 02:19:58

     know anywhere in the in the server mesh now that system works right it's it's in place it's been Fairly reliable but that touches on all the markers in the game and so a lot of issues came up for that and one of the things that we have not tackled with the refactor is how missions are built and to Elliot's credit right I think that uh we're using Elliott's name but there I'm using there's James there you know you know Nick all these guys you know but so the credit to all these guys right uh we have not provided this team with the right tools to build compelling missions they they they are dealing with an archaic scription system that dates

174. 02:20:45

     back to a previous era uh that we are just about to finish modernizing by replacing uh to give these guys modern tools and modern approaches and so going back to markers uh the markers are completely managed by game designers and set and removed and displaced and whatever and that makes for very complicated systems and logic that's not very robust and so that's why even with a robust marker system because the markers are manageable with hat hoc and Mission Scripts it makes it really complicated to follow like I remember distinctly when we did the golden ticket event which the that's that was the whole mission was like there's a t a marker on the ticket right well now oh the the holder puts the ticket on his core armor

175. 02:21:35

     now am I tracking the core armor and then you know what you know how does that work and so and the mission logic had to manage all these cases and so now that we've kind of finished the foundation for making the mission system function in server mesh and we're going to obviously keep it ating on making it robust in this setting and fixing the missions to make sure that they also leverage the system properly bringing back the missions that are missing but also improving the ones that are there so that they're even more robust right uh we're we we have started the engineering work with the new Star script Foundation uh to rethink how missions are made from the GetGo to give you know Elliott Nick James and all these mission designers the right tools so that they're not bogged down with

176. 02:22:23

     details if you saw the way they make missions now it's like involved right and we want to make sure that these guys focus on the flow and not on the functionality they have to build so that programmers can support that content so when we we're talking about this change of focus you know we're saying oh we're not we're to deprioritize Features we're also going to prioritize features for Content as well right so that they can make content that's fun and reliable so that missions that are have been notoriously problematic or events that have been notoriously problematic become fun and reliable right and that's what we really want to do for the Mission Part all right that's going to be incredibly important yes uh the mission stuff especially for this year this this idea that of a narratively driven content driven year you know if we're

177. 02:23:13

     not providing you know 8 to 10 sexy new features every single patch we're now providing you know place to go and things to do instead you know connected to this narrative again we'll talk more about the narrative on next week's ISC having that mission system seems like one of the larger if I were in charge of the priorities it seems like one of the larger ones but we actually have some conversation about how you can affect that priority thing at the end of the show don't let me forget um Quantum Drive yeah Quantum travel Quantum travel the bug that keeps on giving there's a lot of starting and stopping with Quantum travel a lot of repeating the same actions over and over to get something to work and and a lingering issue of fuel States not sinking between meshes or simply Vanishing Al together

178. 02:24:01

     where's my Quantum fuel Ben and of course some ships themselves disappear during Quantum travel right a rarer case but possible do we have our own Bermuda Triangle what's going on well yeah Yen did add some notes here and said oh if you if you find the Bermuda Triangle please report it on I see uh but uh so Quantum has been since the very beginning we put the first mhing test in place that was the first thing right it's like ah Quantum stops when I cross a server boundary that's what happens right still so the same symptom has been again triggered by multiple you know issues you find five causes the sixth one shows up correct exactly and uh the latest ones are related to the resource container that so again all the I don't

179. 02:24:50

     want to dive too deep on this but like entities have classes that Define what they are and they have components on them that Define their functionality so like you know if I have a a box with a health component then the health component manages its Health its damage and its destruction right so we the entire game is made like this with components that Define things and so the the Quantum Drive is made up of multiple components and these components need to deal with the change of authority when we cross a server boundary and so there's a lot of entities intertwined for Quantum Drive to function fuel resource container and Quantum Drive and we've been fixing issues with this transition on these entities non-stop since the beginning

180. 02:25:38

     because the there would be a fix then it would work for two patches then a change to how we deal with Chum points would come in and then rebre it right and so are in deep regression on a lot of things right and so to solve that we need a lot more focus on validating the issues it's like yeah we we fixed this okay let me try transition one two 3 four like more uh I want to say more QA but more QA we need a lot more QA to validate the fixes uh we're in the home stretch we're in the home stretch how you guys doing if you need to leave just you know Pete I'll drive the microphones back to you tomorrow if if you guys need to take off

181. 02:26:28

     um so so a little behind the scenes here I I write a dock and you all the things I want to hit and Stu like this and I make myself little notes about things I want to remember that I that I hit put some red this one just says prisons I'm tired what's going on it's all I wrote for this prisons I'm tired what's going on so uh prisons is and I've learned about this uh in inquiring a bit about the mission the prison bug right uh is actually a mission and it's a very old Mission yeah right so it's a mission module that runs the prison logic and so for those I mean I mean I'm sure everybody's familiar but for those who are not for some reason there's a a bug that lasted through preview and the

182. 02:27:17

     beginning of 40 I think or it was fixed just before that where um if you were a criminal and you got caught and you ended up in clutcher you got what you deserved well I don't know but uh you uh could try the prison escape which didn't work because the mission logic that drives the prison escape is tied to this uh and so the codes weren't outputting correctly the logic was basically made for the old mission system it was not part that's part of the refractor and so uh it it basically depended on when you get to that location to give you a new Mission instance for your prison experience uh which the new one the new system requires that these pers these modules are persistent and stay there which accumulate the logic change and so the logic becames desynchronized with the

183. 02:28:06

     state of the mission so what that did is that you would literally do your time either you smuggled in ad anite from somebody else or mine it pay your due and then the elevator look look forward to the front to the The Prisoner who paid us blah blah blah and then you would press the elevator again elevators are back press the elevator to leave and then it would never detect that the action area that you're leaving is actually triggering which triggered the uh you know leave prison logic right and so um the the guys have done a hot fix for this in in preview actually at the end of the preview I think where we've they've moved the logic for this leave prison flow uh to the actual law component that's on your character and

184. 02:28:54

     so now we're guaranteed that no matter how weird this flow would get it's actually going to be reliable you just said we guarantee it's going to be reliable we we are 2 hours and 30 minutes in this you just use the phrase it's guaranteed to be reliable let me rewind it should be a lot more reliable because your players always streamed in so I have a I have a fairly good confidence on this fix as long as the elevator works that's a big caveat benoa as long as the elevator we could sit here for another two and a half hours and say all kinds of grand things in just long as the elevator Works maybe we'll put some stairs in

185. 02:29:43

     prison in fact there are but you need the prison escape code to get them uh all right uh got like four four more topics we're done uhhuh mining yeah sometimes Mina are taking up to an hour to fully load in making them rather impossible to mine yeah mining has been actually a dreaded gameplay Loop in 40 that's one of the reasons when when we did the evaluation whether 324 or 40 preview whatever uh we did take a look at mining like is mining functional is mining in the prospector functional is mining in the mole functional is mining on the Rock functional is hand mining functional mining The Rock has NE never been functional it actually was intermittent in the first report but I hate the rock DS

186. 02:30:31

     uh the so it's we've identified what we think is causing this delay it's basically the conversion from the render node object because the mining and and harvestables are always in large quantities right they could be a very large performance hug and so they're highly virtualized and so there there's a conversion from render node to entity that is for some reason very slow and so we're diving into try to figure out what's going on but that's what's happening so that's why they it takes time for them to show up properly there's nothing wrong with The Rock DS people just with the people who like it all right I'm officially tired um the views and opinions of Jared Huck not represent those of Cl Imperium games Robert Space Industries or its

187. 02:31:18

     subsidiaries and didn't talk about the mission uh reduce strict rep Gates the the what the reduce strict rep the sorry Mission sharing has also been reduced with new strict rep Gates is one stop one I just thought that sucked I didn't think you had an answer for it no I got I an answer for it I went to the people you screwed up Mission sharing and the payouts and it's miserable and it's terrible now well so there was a bug that caused everybody in a given mission to get the reward for the missions right so that means if a mission payout was 10K then everybody would get 10K everybody participating it was amazing well so loved it when that so the funny bit is I talked to Rich tyer about it and he's like I C I couldn't catch it before they fixed it that's

188. 02:32:07

     what I want and so ultimately it wasn't exactly what we wanted but that's the direction we want to go in we want to encourage you guys to play together and so having those Rewards divided is kind of a a a Counterpoint right to get people to play together like why would you I group with you if I'm going to get half the payout right and so the thing is like the system doesn't work the way we want it to do so the fix did go through right because it was an easy fix and we had the right persons in place to do it but that's not where we want this to be we we do want to improve that that part we're we're we're we're we're at two and a half hours yeah and talking about all the bugs that still need to be fixed correct and the one bug we did fix

189. 02:32:55

     was one we didn't want to fix I know it's crazy right it's a train you know I challenge you whatever you you think about the project or me or Ben ju the show you've never seen you've never seen an episode of any any kind of Dev communication like this from any other uh game developer nobody's ever been willing to sit in a chair for 2 and a half hours plus and and be grilled about all this so so all right uh just just two more or three more three more three more subjects three more subjects we got mining done three more subject what's the next one Vulcan yeah and Nvidia driver issues yeah so players who are using Vulcan

190. 02:33:42

     with the latest NVIDIA drivers are reporting graphical artifacts and crashes I know some of these crashes uh were seen from third party s Wes what was that one that we found one guy who had like 40 in of the same crash and it was all linked to this third party thing he was using oh was that Reade on wasn't reh it was called something else I can't remember um overall what can you tell us about what's happening with Vulcan and the NVIDIA drivers well so the we're still investigating the cause but for now right we think it's related to a specific driver version from Nvidia uh related to the upgrades for the 5090 cards so we're not quite sure exactly but it is definitely there is when you update your NVIDIA drivers there are two drivers one for direct 3D and one for Vulcan it doesn't look like it but there are two uh and so we think

191. 02:34:31

     that there's an issue related to the laser version of the driver for Vulcan specifically and so you could downgrade your drivers and it probably solved the issue in the meantime but we're still investigating to make sure it's completely not you know related to this but other we've been investigating other games are also having the same issue and so we're yeah there there was one one or two drivers that go where somebody I saw somebody they had glowing heads all the character heads were glowing in St they were on both the you know the we don't think it's a third party app in this case though yeah know that was just one instance we were chasing down I was watching we do you do need to be careful with third party apps that inject into the 3D pipeline for multiple reasons one of them being that the ESC sandbox can catch it but also because they can cause

192. 02:35:20

     crashes so all right the last two mhm are about how we make these things better issue counsil so issue Council has bugs issue Council has some issues with it players are talking about uh the effectiveness and usability of the issue Council there I've seen them expressing frustration with the dupe system and suggesting improvements to the reporting process there is signicant a discussion about the difficulty of reporting recurring bugs that have previously been marked as fixed why is this happening and how do we address this going forward these are the folks who are well so the the trying to help us the first thing I would say is that the issue council at cig is of extreme importance right even though for

193. 02:36:11

     players if you don't feel like your issues being actioned or that there's retroaction from the team you can be certain that we have especially Bayer's group is at the Forefront of this but issue Council reports are significantly of high quality in general when they do pass the filters of course right but and so we use that data every day right to confirm issues uh a lot of the issues we have on our famous director's list that I just talked about are not internal Jas they're IC reports right and so that process is super key for us it's a very difficult uh signal to noise ratio process though because the level of effort put into reporting is not necessarily of of smooth uh qu like the quality range is

194. 02:37:02

     is large is what I'm trying to say some reports are like game no work others are like extremely filled with evidence and and you know and those are the ones we want obviously uh we do want to improve the dupe uh system so i' I've I've seen several uh backers on the ETF and elsewhere that are very active at calling out dupes and for now I think that their voice are kind of modeled by people saying oh I can reproduce right instead of saying oh I confirmed this dup right so we're actively looking at how we can improve that flow because that's a major like when when we identify a duplicate it's a major win right so when you contribute to the issue Council a new report is depending on the issue right is takes a lot of effort for it to reach us because you

195. 02:37:51

     need to get reproductions on it reproducing another issue is a great contribution but identifying a dupe is probably the best contribution you can make because that helps us a lot and for now that's not leveled enough so we we want to change the the way isue Council Works to make that a lot more at the Forefront for sure but the prioritization and the upvotes have historically been used but in this new process we're also trying to lean into this a lot more and so the police fix Hammer button right to upvote an issue we're going to try to give it a voice and actually make uh some of the internal discussions we're having I can share some thoughts if you guys are interested uh or even to have Community identified issues directly in the

196. 02:38:40

     director's list like make the community a director in this list there's also some amount of things being marked dupes that aren't dupes sure you you know it's it's you know people are moving fast some people think they've seen that before or whatever or they think this is the same thing as that but it's not it's just in the same ballpark or whatever so there's also this there's also this thing that happens where things are getting marked duped and they're not duped and that's I don't know how you battle that no that's a balancing act right it's uh that's why for now I I feel like the dup markers uh are kind of more experienced uh you know testers that are helping us and uh but duping right will happen typically when we actually inake a bug right from the

197. 02:39:28

     community and we're going to try to Repro it internally that's where we're like oh it's been Mark as a do but it actually is a new symptom or it's the same case as another one or not do you do you think there's some amount of is it possible that there's some amount of semi malicious intent there PE people who are far fing dupes or using Bots or whatever because as I know there are incentives for participation in the IC and stuff like this uh obviously we incentivized the use of of the issue Council because it was in incredibly important to our uh to the way that we make the game but any incentive sure tends to be weaponized in some way shape or form so have have you have you seen evidence of this is is this no uh not so far right not so far as to say like oh

198. 02:40:19

     yeah there's this going on right the the worst things that we've seen so far on IC would be things like brigading a specific issue right um and that's why also we've been kind of careful with rewards to the issue conso it's not because we don't want to reward players that are doing stuff there like we we are doing some of them but we're always trying to stay asynchronous so that you can't go oh they're going to reward for this so I'm going to go and create reports or I'm going to create dupes or I'm going to create whatever it it it it it's obviously you're you're the person who would know and it's not that I don't I I I don't I don't believe it what I'm see what I what I've seen and and and what and what I'm reading in chat right now um would you commit to maybe taking a look at that and maybe seeing if if if maybe it is being absolutely manipulated

199. 02:41:06

     yeah I'm absolutely look at it okay especially if there's if there's a sense that that's happening yeah y uh yes Apple Tango it's the best part about living in the UK um I've never had that I last obviously we obviously we're not covering every bug every every topic in the thing it's well you knows we've done more than any human being in the world David is I've never seen this look on David's face before like David was a kind and jovial spirit and now he's like down and now he's like like like like I'm gonna pay for this yeah um the error codes themselves mhm famously yeah 30k could mean 40 different things 50 different things 60 things we have made progress we've

200. 02:41:53

     absolutely made progress but even that 630 is is that's not just one thing there there's multiple causes for even the 630 and St like this is there is there any are there any initiatives in place to to to to help this drive down anybody who worked in it I worked in it for 20 years before I came here I in everybody knows that the worst errors are the ones where you can't rely on the error code and you there's no you're never going to meet get 100% you're never going to get to a point where every single eror code is just going to spell out to you everything that's exactly wrong with thing but I feel like there's still PL there's still improvements we can make absolutely we're not at that point where there's nothing more we can do no so what are we doing about well so on ericos is the ericos all going to go and merge pop-ups into that right because I

201. 02:42:42

     mean the error popups date back to the very Original start of the project like nobody's touched these things in forever right uh so the you know the nicely misaligned cancel button is my favorite pet peeve right why haven't we fixed that but well we're fixing it so I've we're we actually have a a CL right now to change all these pop-ups to make them at least readable you know and aligned right we start with that but um no you start with the elevators then you fix up so the the um the air codes themselves right so there's already good categorization for issues uh so every kind of uh like 30,000 errors 40,000 errors 60,000 errors are all like

202. 02:43:29

     families of Errors unfortunately over the years we've added more systems we've grown the infrastructure and there's a lot more causes for exit or fatal errors than there are actual code assigned in those categories and so what we're doing now is kind of seeing if we can Surface more information uh you know 30,9 is typically related to your character uh during login but there's a million causes that could cause a 30,000 n whether it's that Shard can't outo your player or your player is missing a component or you know and we're trying to surface more of that so that there's more of these cases so that you can as a player know a bit more about it also the wording for some of those errors are being revamped because you know they're not always very clear uh and we're

203. 02:44:19

     trying to I want to do I want to do that that that shocked bird GIF right now continue correct and so adding more details and more granularity to our erodes changing the wording making sure they're more effective at representing what they are and eliminating a bunch of the there's a couple cases where we fall to a default error code for kind of unknown conditions that's what we need to flesh out is like the unknown what is the unknown condition you know try to surface more a codes from the other systems to get more of that information in place so we need we need more clarity on that side so that players like oh again a 19k error what does that mean right like I go in in Spectrum a lot uh in the SC testing chat and I see players

204. 02:45:08

     going like what's the fix the 19k errors right and a 19k raw error is like your client can't talk to the backend service at all so it's typically a client side problem firewall machine connected something like that but the the aor message itself is really unclear right it's like oh we failed to log you in right well what can I do as a player to fix that right so we're trying to get a lot more clarity on these error messages that's part of like the improvements of these things that I've been plaguing us forever these goddamn pop-ups are one of them I've had a dear with these goddamn pop-ups hey you want you want a funny one like here's a a funny one on these popups right cuz we're all complaining about them but we lost those popups in during the 4.0 right everybody was having a black screen everybody's complaining about a black screen like I get a black screen

205. 02:45:55

     right well what actually happened is that someone that I shalln't name right but a developer fixed another issue where when you die we need to move the camera and move your character and it looks like crap right so we didn't want to see that and so the way to solve that was to uh Fade Out the game youi including the popups that was it you know and so in this case it was if you got disconnected then the popup would go fat to Black great Serenity Now All right so we're here y we're at end this was a raw Game

206. 02:46:45

     Dev going forward I touched on this about halfway through but going forward with a focus on stability performance and content this year this means less time available fewer resources spent and no patches dedicated specifically to new features it's not that new features won't come out or they're not being worked in the in the on the background just that instead of 10 15 people working on them there's now a much smaller group and the the time we're not going to time lock it into specific things we're going to make good on the promise we made back with 3.0 quite honestly start citizen has been this heavily feature-driven project in the past every every three months or so you know a patch with a bunch of new

207. 02:47:34

     features comes out that brings people back they come in they see the new features uh they they they it's it's it's the cycle that's existed for so many years and we're we're stopping that now uh uh and we're focusing on fixes and quality of life and all no no more it's just from a marketing standpoint I'm you know these trailers where it's where it's like this feature this feature this feature this feature this feature like those days are are not here this year that's not what this year is going to be about so this thing is it's I'm excited and I'm scared at the same at the same moment if I'm being completely honest because we've never seen what a Content driven stability focused year of Star Citizen is like it's going to be different and like anything else with

208. 02:48:22

     Star Citizen it's going to change and evolve over the year because somebody will have an idea in April and we'll tweak this and somebody will have another idea in June and we'll tweak so all we know is that it's going to be different it's going to be different hopefully better but it's not what we've done before do you think this is scary and and and and and more importantly why is now the time for the change well I think the first of all it is very scary right because it's a change of how we've been doing things forever right which is like a focus on transformational change to the game you know and constant Evolution whereas now we're focusing on hardening and playability and looking at you know what players are doing in the game and

209. 02:49:10

     going like everybody's hitting this problem like you know and then it's not just the voice and the side is you know hey we should fix this right it's like no no we're fixing fixing this like every every director in the company is going to be on top top Duty for this right which but what's scary is that um how did the Yen frame this for us he said that the stability and performance heightens hunger for new content and new features right and that's the part that's scary that when the if we do achieve what we're set to achieve we're going to need features as well right and so you get to that point then everybody's going to try like 's nothing to do now it's it's it's a sushi restaurant right yeah but it's just has to be done and the reason why it's time

210. 02:49:57

     to do this is because we all like Chris said it in his letter that's the part that resonated the most to me is like we all acknowledge that if the game that we have now functioned the way it's supposed to it'd be a great game right that's what we want we want a great game and we can that we can build on you know and pass a road to get us out of alpha right right and that's that's what that's why it's time now that's why we're committing to this and if there's one thing you want you've obviously spent a lot of time we've obviously spent a lot of time here sorry Pete took my hand off my face audio guys just like stop talking into your F I'm tired if there's one thing you want to make sure people take away from this today the the letter in in in December the all of the refocus focusing and

211. 02:50:49

     restructuring that's going on now uh and from this admittedly a lot of talk the proof will be in the pting over the next year yeah what do you want people to take away from today I mean I I think I want players to know that we know about the state of the game we're not oblivious to it we really care about making the work of our teams shine and we're going to work our ass off to get this game stable and solid and to be the game that we want to be I think it's it's about time we do that change well spirit wolf uh 655 in chat says pure BS all right trying guys uh benw thank you so much for taking your time to be

212. 02:51:40

     here uh folks don't know this there was a the the entire ctg team is here for the for for the annual Summit there was a big celebratory party for launching server meshing and finally getting out the door it's happening right now it started right as we went on the air and benois you know chose to be here uh to do to do this instead he chose to be here for three hours and and face the tough conversations yeah we're just about three hours thank you so much I got to thank David Pete and Lauren for sticking it out this is this this is not what I sold them when I said come help me make the show um uh uh the community team and the player experience team are going to put put up a thread on Spectrum where uh they discuss uh many of the the pressing issues that we talked about today and they're going to have you vote on which ones you think are most pressing

213. 02:52:27

     obviously we have our own internal uh uh prioritization we talked about the director's list and stuff like this but it was it was a it was an initiative that was really important to the top leadership we wanted to know what you guys what you guys are thinking and what you guys are doing so look for that I'm not sure when it's going to go live or in what subsection just check it out on Spectrum uh it'll be up um thank again um and hopefully I don't get flamed by the game devs for my details we'll see yeah let's let's see if I've still got a job back when I go to the thing let's say um I want to say one more thing uh before we go this give me a a minute here um this is this is going to be a real

214. 02:53:15

     change of topic here in November uh for IE Show episode 3 uh we released this this this this puppet song that that that John crew and I did uh that I wrote and this thing that I've been dreaming about cat our fantastic creative director for the project made it all happen and stuff like this and it was one of the things I was most proud of and about an hour after that episode went live uh on November 28th uh I called my dad uh those those uh those who have followed the project for a while know my dad is a Star Citizen uh uh ever since that random Day in 2017 when I got a text message hey son when is the cutless red get this or whatever I'm like what the hell are you talking about my family is not supposed to be

215. 02:54:03

     involved in this thing he's been a Star Citizen for a long time uh I called him up because it was one of those things that I knew would make him proud and it would make him laugh and it did he answered the phone in a hospital bed uh there was nothing to worry about at that point he just felt a little weird they his age they would take take precautions and he assured me everything was fine we watched it together and we laughed and everything was great and we said goodbye and we talked about how excited we were to see each other uh at Christmas about an hour later he had the first of a series of heart attacks and on December 9th as after a week of convincing me that I did not need to rush out there because he was

216. 02:54:52

     going to be okay and everything uh he passed away and when I went back and I'm doing all the things that a son has to do when their father passes away uh I'm going through his computer I'm going through his records and making sure everything what I discovered was I knew he liked Star Citizen I did not understand the breath of involvement that he had he retired he retired from from from the Army and from uh the correction service and from Homeland Security and all the things he did over his life and he became a professional Star Citizen his Discord was full of over a hundred unread DMS and as I went through them they were all talking about Star Citizen

217. 02:55:40

     he was in at least 14 different Star Citizen discourse he was in a number of organizations and he was a he was an officer in several and he was able to do that because of you I didn't talk about Star Citizen with him we had a standing rule that when I came home for holidays and whatnot he was not allowed to ask me a question about Star Citizen until after dinner was done and the the plates were and the plates were were done it was really important to me to have one aspect of my life that was not completely consumed with Star Citizen and I made that my family his engagement with this project was all because of you there were star citizens at his funeral there were star citizens at as

218. 02:56:29

     wake and not the worst farar citizen I've ever been to but what I came away with is just a profound appreciation for many of your roles in his life his retirement was playing with you so many of you names I see in the chat I saw in the chat today which is why I'm I'm saying this now so for me personally thank you for being there for him when I wasn't able to be and you made the last

219. 02:57:19

     six seven years of his life uh some of his happiest so that's all I wanted to say thank you for watching thank you for playing Star Citizen thank you for being the community uh that you've always been even the ones that call out total BS uh you're all appreciated uh that's it for this show uh tune in next week uh there's a BTS episode and an ISC episode about something I can't even think about what they are anymore thank thank you and uh we'll see you next week everybody take care bye thanks