Star Citizen: Reverse the Verse LIVE - Object Container Streaming

7 September 201801:02:341,705 linesWatch on YouTube ↗

Transcript — every row opens the video at that moment

00:00:06
[Music] Hi everybody. Welcome to a special edition of Reverse the Verse Live here in uh the Tagert Conference Room. The beautiful, stylish uh Keith is shaking his head over there. Tagert conference room in in Wimslow, United Kingdom. Uh I'm your host, content manager for Global Video Production, Jared Huckabe. And today we are joined by an esteemed panel of engineers and QA and programmers here to discuss a topic very close to my heart. Uh I was going to say something funny like basket weaving but no it's object container streaming. Uh now object container streaming is something we've been talking about for a long time. uh we we it's we when we started the conversation about uh serialized variables once upon a time
00:00:54
for buying calling object container streaming network performance improvements you know all these all these different topics that all kind of roll into an overall performance uh uh improvement for Star Citizen. Uh we wanted as we approach 33 and the uh the inclusion of Hursten and and object container streaming, we wanted to take one of these last opportunities to really dive into what this system is and explore the work that's being done and uh maybe dispel a couple myths about it because we see a lot of misinformation on on the forums and stuff like that. Uh people thinking it's a silver bullet that's going to solve every problem in the world and stuff like that. So, I really want to take some time to talk through exactly what object container streaming is uh and what it means for Star Citizen. Before we do that, we do the same thing every week. We assume that nobody has seen any other episode
00:01:41
of Reverse the Verse before. So, let's take a few moments and introduce ourselves. And we'll start over here with you, Gordon. Who are you and what do you do for Star Citizen? I am Gordon Mlan. I'm a senior gameplay programmer um on the FPS team here in Wimsilo. So, um my main rema is the actor. So I own the actual player and any sort of human type person running around. Gotcha. Yeah. So I've been here for like three and a half years now. Say you've been here just about as long as I am. Yeah, I've been here for quite a bit now. So yeah. And uh you said uh most of your focus is on FPS stuff, but how how what what have you been doing with regards to object container streaming? Yeah. So I was sort of drafted in as support gameplay engineering support for object container streaming. Um so it's a lot of sort of um mostly bug fixing and sort of fixing
00:02:29
peripheral issues around OCS. Um sort of probably the main biggest one was the likes of the mission markers for instance. So making them work with OCS tracking tracking a mission when you're out of range of such like um yeah and then general performance stuff as well. Tracking all the stuff that he brings up. Yeah, pretty much. So Chris, who are you and what do you do for Star Citizen? Uh I'm Chris Eckersley. I'm lead technical QA in the UK. And what do you do for start? So what what does that do? Um so we're part of the main QA department, but uh we tend to handle more of the tools and techy side of things. Uh so we tend to be a bit more hands-on with the servers than the rest of the department, which is why uh OCS and Bulling stuff tends to fall our way. Gotcha. So less of the less of the
00:03:18
gameplay stuff. A big misconception about QA is that it's all just playing the game all day. Uh you're you're you're you're sitting there beating on the tools and the servers and the back end and stuff like that. That's right. Yeah. Gotcha. And last but certainly not least, Clive. Who are you and what do you do for Star Citizen? I'm Clive Johnson. I'm the lead network programmer. Um what do I do? Uh yeah, you're Mr. Happy Funtime. Yeah, program a little bit of networking. Uh try and avoid going into meetings. Um, so yeah, my my main focus is the the client server network rather than the back end side. Uh, so it's it's getting the game plan to talk to the server and get the information back uh as efficiently and um with as little bandwidth as possible. Gotcha. Now we've featured Clive on other shows about networking and about
00:04:06
buying calling and stuff like that. Um, we as always with Reverse the Verse, we are going to be taking questions live from the Twitch chat where you can submit your questions with the word question in capital letters surrounded by brackets. You can also do so in Spectrum, our live uh communication platform on robertspace industries.com. Uh, the question period is going to come a little bit later because I I want to I want to I want to start with a with an oral journey, if you would. I want to shake things up a little bit a little different. I don't I only get one week a year in these other studios. Uh, so now I've got you here all in a room. You can't fake, oh, the Skype call went down or whatever because I'm here. Um, let's just talk about OCS in general. Uh, let's start at the beginning. What were we trying to achieve with OCS? It's it's a what what what was you know we we started working on OCS I think when I
00:04:54
was talking to Luke Hail who's a a the associate producer that that handles a lot of your work he said the the earliest works on this you started about three years ago uh when we started work on OCS 3 years ago what were we trying to achieve the the main concerns always been trying to fit Star Citizen onto um a a player's computer Um obviously we servers tend to be a lot more powerful than the domestic users computer. So having all this expansive solar system uh all the planets all the locations uh all the ships loaded in all the time is just not possible without masses and masses of memory which is much more than
00:05:42
your typical user is going to have. Um, so the primary concern is just making the game fit, breaking it up into smaller chunks that can fit on each player's PC. Gotcha. when we when we started moving towards 360°ree fully realized moons and planets and wanting to do that without loading screens, you know, you leave up from one and you don't want to like you set course for from Hurst into microte and you don't want to go, you know, quantum jump and then it goes, you know, loading screen, you know, and then and change that. Um, object container streaming was was the tech that we wanted to be able to seamlessly move like when you turn away from Hurston, you move away from Hurston to pull Hursten out of out of out of a out of a client's memory and then feed in microte that and then vice versa. And this and it's not just the
00:06:31
juggling of the client side, it's also of the server side. I understand it's doing the same thing with the server because even once we start putting in all these space stations and and uh uh planets and moons and everything uh servers still have a finite amount of memory finite amount of processing power. So there's there's client side OCS and then there's still a serverside component. Am I right with that? That's that's right. That's not something we're working on for 3 further down the line. Um but yeah, because the solar system is going to get even more densely populated and more planets and so on, uh even the servers with a massive amount of memory that they have won't be able to keep everything in place at the same time. So they'll need to stream in and out locations based on demand. So when a player goes to a location or as he's heading towards a location rather, the server will detect that go okay, I need to stream in the
00:07:19
the entities that are there that the all the planet and directs and whatever around. Um, so by the time the player reaches that location, it's all in memory on the server, it'll the server has to coordinate that with the client itself and tell the client what it needs to stream in. So the entities are present on the client as well. Uh, and then when the player leaves that location, the server's free to unload that information, free the memory for anything else. We were talking just a few minutes before we went live and uh this isn't a technology that we're inventing for Star Citizen that this this this this juggling act of moving things in and out of of of memory in video games. This exists for many years. Yes. Yeah. What's the difference with Star Citizen?
00:08:07
Why why why is this such a big thing for Star Citizen then? Um well, map streaming is generally more common in single player games. Um, a lot, not all, but a lot of, um, multiplayer games will confine the maps to smaller sizes so that it can all fit in memory all at once. Uh, so once you've loaded in, all the memory is resident on your client. Sorry, all the data is resident on your client. No streamings required and that cuts down uh, the chance of any stalls uh, or any frame rate hits uh, and makes for a smoother online experience. said that the challenge is to do streaming during a multiplayer game. Mhm. Not have any stalls or um performance hits. And it's also, as I say, it's about the scale that we're doing this
00:08:54
on. Um obviously our our maps are millions of times bigger than than most maps of of virtually any other game. So yeah, it's it's a question of scale and trying to do it while you're online as well. Gotcha. So, it's it's not that we're necessarily reinventing the wheel here. It's that we're adapting the wheel to a scale that's previously unheard of. Yeah, it Yeah, we are we're probably one of the bigger games ever envisioned at this point. Um, and we've kind of we've we've done quite a lot of stuff over the last couple years where we've had like snippets of game like Star Commander where it's been like a small contained bubble of the game and we're now going right, it's now that, but it's in this big massive universe. It's making everything bigger and by going well if we only need to stream in this one bit
00:09:43
where the FPS is happening the performance should be the same as star reading probably on the client. So um now so that's why we want to do it what's this look like when when when we get there now now obviously 33 and you you mentioned it earlier and I'm glad you brought up the the server side object container streaming is not a is not a part of the work that we're doing right now. Uh we're doing we're working on the client side uh object container streaming. Uh but once this is all done, let let's let's jump ahead. You know how much I love jumping ahead to the far future, but I'm going to do it this time. Once we jump ahead, once once we've got binding and once we've got uh server side uh object container streaming and then in the future once we have server side, what did I say that right? Client once we have client side
00:10:31
object data streaming and then later on down the road once we have server side object container streaming. What's what's that going to look like as as far as performance? Do does everybody suddenly just get 90 frames a second on everything? It it's it's what what what's this picture look like? Um, okay. It it's not it's not a silver bullet as you kind of mentioned before. So, we can't make any guarantees on how performance is going to improve. Um, partly because there's so many variations of CPUs and motherboards and RAM and everything else and GPUs that you can't predict how it's going to affect all the different combinations. So, it's a best guess uh what we think is actually going to help um what it's going to look like. So effectively the
00:11:19
the difference to what you have now is you start off in your bed at port also um and yellow is loaded on your clients um and there are entities on your client in memory hopefully idle and not doing anything if the um the in range um update calling is is working correctly. Uh the difference will be with OCS you'll load into Port Alisar and basically Port Alisar and a few entities for the the uh the what's it called the uh the star map will be loaded and that's it. Uh any players and ships that are around your immediate vicinity will be loaded in. Um anyone who quantum travels to your current location will dynamically be loaded in.
00:12:07
Mhm. um anyone who comes and travels away will be loaded out and your client won't have any awareness of them anymore. Um so it cuts down the amount of memory that your client needs. Uh it should cut down some of the performance overhead uh for maintaining all these um entities of memory as well. Um I suppose as you move around the solar system, yeah, it will unload areas that are uh it that it doesn't need anymore unloading the new ones. So you head off to to Lefki, it will stream Leking for the first time and that's the first time your your client is aware of those entities in that location. Okay. Uh what criteria do we use for that? Is is it just proximity? Is that the only criteria? It's we we we we set
00:12:56
a we set a bubble or or a streaming window around you and anything within this area is visible to you and anything that's outside is is not loaded into memory pretty much. Yeah, it's it's not exactly a hard bubble. Uh it's not a a defining distance. It's actually the um kind of the the size the object will appear to you if you were to look at it. So an object that's twice as big, you will see it the same size as an object that's half as big, twice as far away. See what I mean? Okay. So a a planet will stream out Well, no, that's a bad example. Yeah. Leki will stream out very far away, but a small spaceship like um and a Gladius will stream out much closer.
00:13:45
Gotcha. So there's there's there's a there's a like a tagging system involved in like it it's that the system's going to know that if if you're approaching Levky to bring Levky in at this far away. Yeah. But if it if it's a Starfare to bring the Starfare in at this far away and if it's a Hornet to bring the Hornet in at this far away like that. So So it's so it's an intelligent kind of switching. Yes. Between So like that. And and is that is that um and that's something ult at the end at the when all is said and done. Is that is that something handled by the servers or by the clients or that's handled entirely by the servers? Okay. Yeah. Interesting. All right. So, uh the work on when we we're doing we we've got notes because this is a pretty dense topic here. Um when when the work started on this uh it was originally envisioned is kind of
00:14:33
three steps. Uh bind calling which we've talked about quite a bit. Uh then client side uh OCS which is what we're working on right now. Uh and then uh later down the road server side OCS. Yeah. Um let's talk real quick about bindulling and what it is and what its role here is. If you can just who wants to give me just a nickel version. What is bindulling? Anybody? One, two, three. Don't want to jump in when there's a problem. Go for it. Yeah. So um this is a test score. This this a I should have revised further. This is Yeah. Um yeah. So it's this is the act of unbinding the entities on the client when they go out of range. Um so if you're too far away
00:15:21
from stuff, it basically streams it out is the idea. Um there's quite a lot of implications for that for gameplay code. Um which I'm obviously slightly concerned with being a gameplay programmer. Um, so it's a type of thing of if I'm on one side of the universe and I've got to go and kill someone on the other side of the universe, I would normally in the game just now you see a little objective marker, but that's not there because he doesn't exist on my client because he has been unbound. Um, or similarly if you've got have a comm's call with someone who's too far away, he'll be unbound. Um, or sort of similar things. Um, and there's from a more technical standpoint, we have um the likes of all our entity pointers themselves will as soon as something goes out range, they'll all become invalid. You can't talk to the entity anymore because he doesn't exist for you. You can't um like you can't get his status. You can't get the amount of
00:16:09
health he's got because it doesn't it no longer exists on your client. Um, and you've got to be able to handle um him obviously leaving and then him coming back again later on. So if you've got to fix stuff up and if you've then got to um reworked your gameplay system to make it work with someone that's just came back. Um so what are the answers to those things? Um so for the likes of the the markers uh they were solved by having an independently streamable object. So that's basically saying we've got this little marker which a really small lightweight thing. Um it's really far away from you but the server is going to tell you that it has to exist in the client because you need to know some stuff about it. um it's removed from the normal hierarchy. So it's not um if you've got a marker, for instance, on a person that's inside a ship that's inside another ship, normally you would need to have all these zones all the way
00:16:56
down to get him. Um in this case, it's just that marker and it just gives you the zone position of the biggest zone that's still streamed in. So the planet or the root zone if he's just out in the space somewhere. I don't know, maybe I'm wrong on this. I'm picturing like when you're playing golf and you you you you pick up your ball and you put a quarter down in its place. It's like so so that you know where that thing's supposed to be. The ball isn't actually there anymore, but but you know where it's going to be and when it comes time for you to actually need to interact with it, it's going to know where kind of um and it's also got the advantage of we can do things like the because it's so far away, the position doesn't need to update constantly because it's so far away, you can't tell if it's 10 cm out. So it can update every 30 frames or every 60 frames or whatever, which also obviously increases performance ever so slightly.
00:17:43
That's going to be a big one for server FPS. Yeah. Um and it's more um we don't need to get the messages from the client from the server as often. So it's not it's not affected. And then once you get in range, once it gets fully bound, everything else has been unbound. So the network performance is then the stuff. Um yeah, and then obviously there's stuff I'm not as involved in. the legs of the radar scanning stuff has got to be able to work when the stuff it's scanning doesn't exist on your client. And so a lot a lot of the game code was changed from everything happens on the client to the client has to ask the server for some results or it has to push requests to the server for things to happen. We've seen that. It's a big uh it's a big uh concern of the community is is uh we're seeing it coming in the questions right now about how scan how long range scanning is supposed to work. If this
00:18:30
thing isn't loaded in for if in your client universe it doesn't exist, how are you supposed to scan for it and discover it? Well, I mean even the likes of when Clive was mentioning what how we decide if something streams out is based on how big it is on your screen. So if there's a Gladius that's 20 km away, they're going it's going to be one pixel on your screen. it can probably stream out or it's going to be less than a pixel it can stream out. Um there's some adjustments made to that for the likes of the radar. So the Gladius will be made to think it's bigger because it needs to stay around because although you can't see it, your radar can see it. So there needs to be some kind of um adjustment there. So you're still able to still able to get the up to date and the high end the high um frequency updates to see where it is. And and how does how does serialized variables play into buying calling? It's
00:19:19
a serialized variables that that in my terrible analogy and forgive me if I keep using it, but the quarter that you put down is is is is that what a serializ I think one of the complexities of OCS is there's a whole interconnected bunch of technologies that will need to work. So there's serialized variables need to be there. There's actual bindcom itself. There's the likes of the um the um I've lost my train of thought now where I was going with this. other technologies as well that only need to exist for it to work. Um the serialized variables are kind of like the thing that sort of almost I don't want to say glues it all together because that's a bit of a cliche phrase but um uh for the likes of the marker you'll have serialized variables that have all the data you need in it. Um the server will be able to tell you the data you the data you need to have in those variables how often you need to get it. Um and it
00:20:06
will kind of do it seamless almost seamlessly from a game gameplay point of view. So we've just got the data we need to use. And and uh for again just assuming that this is somebody's first show uh what is serialized what are serialized variables? So serialized variables are the um the mechanism that we expose to the gameplay programmers to allow them to communicate over the network. So um the game's built out of entities. Entities are built out of components. components have these serialized variables which are values that can change over time as the gameplay programs assign new values to it. Those change values will get replicated over the network to the equivalent entity on your clients. It will uh change the value of the serialized variable there as though it's being changed exactly by the server. So
00:20:55
it's it's a remote control over um the state of entities in the game. Not when when we've I've had a number of conversations about this over the last year. Um I always and please correct me if I'm wrong. I always kind of saw them as like save games like like like it it was it's you know here's something actively doing thing and then when it gets converted into a serialized variable it's kind of like that's like the save state file for it. It's yeah it's exactly it's almost exactly like that. We we use serialized variables understanding what's going on. We use serialized rules not just for uh networking but because serializing to a network it it doesn't necessarily need to be a network. You've got this serialization process that converts from the um I suppose the native format that
00:21:45
the program's running with uh into another format and it could be the network it could be a save game file it could be the persistent database. So we use serialized variables for those three things. Okay. again and forgive me but these are the engineers and the programmers so I I am the layman in many things it also over the conversations it's almost like uh uh like I worked in a professional baseball for years and I I imagine like when the catcher is giving signs to the to the pitcher short form information that means long form stuff there's so kind of compression reducing the um not so much on the on the data itself the the the the way serialized variables save us um oh how to explain this. Okay,
00:22:37
so one way of um updating the state of entities in a game over the network is to periodically every so often just send the state over to all the clients. Um and that can be inefficient because you're sending the same values over and over again and nothing's changed. So serialized variables track when something has changed and go, okay, this one now needs to be sent. Okay, these ones that haven't changed don't need to be sent. So that saves us a lot of bandwidth. It's kind It's kind It's kind of like when my mom keeps calling me and says, "How are you doing?" I'm like, "I'm fine. Anything new?" "No, nothing new." It's like now the serialized variable is she would only call me when something's actually happened. Sorry, my mom doesn't watch the show. Uh I I love you mom in case you suddenly
00:23:24
did dad watches the show. So all right so that's fine callulling that's serialized variables we've talen kind of high level let's get into the work that we're doing right now which is essentially full client streaming that's the section of of OCS that that we've been focusing on right now. Um, one of the big uh uh misconceptions uh that that that I wanted to address here is is the notion that when 3 comes out, when Hurstston comes out, that that's the that's all of OCS that it's done. That that it's there and we're done and suddenly everything is great. You can see you like he's just like no that's not I can I can feel the heat radi of them right now. Um, we did bind calling. Uh, we were we got bind calling to a to to a state where it's feature complete
00:24:12
at least. Um, now we're working on full client streaming uh for Hurst and everything. uh when that's done and that's into your hands and after you guys have beat on it through EvoTotti and and and PTU and then even with our live testing environment which is still a testing environment uh once we've got that to a place where we're happy with it I guess or content with it then we move on to server uh side client streaming but let's so I just want to that's just to help paint a picture there for full client streaming the stuff we're working on right now what is full client streaming let's just start with a simple thing. We'll go we'll go from there. Um, it's kind of what we said already. I'm not sure what else we can say about it. So, it's yeah, it's the server dictating what your uh client can see,
00:25:03
what entities it's aware of. Um, and yeah, as you move around controlling uh changing which entities those are present on your machine. Okay. So, so buying calling is the is the tech of dropping things out and everything. Yeah. So, client side. Okay. So, the the the difference between the the two and how they fit together is um object container streaming um is something that we need for Squadron 42 as well. So, a single player um single player game. Um, so it's very much the kind of um traditional level streaming that you that we see in uh in many games. So that controls um that's a big So that controls the uh the
00:25:54
figuring out the distances of entities, what needs to be present uh and what doesn't. uh at the minute on the server uh everything's present all the time. So we've just got this distance calculation based on entities that exist uh and then it can do the um the streaming in and out. The other part of OCS is the asynchronous entity loading. Mhm. So, um, up until now, uh, anytime we spawn an entity, it happens on the main thread. Um, and the main thread will block for how long it takes to spawn that entity. And that includes the time it takes to load any files off the disk. So, textures and, uh, physics, geometry, and anything else. Um part of OCS is making that um spawning process
00:26:42
asynchronous. And by that we mean we tell it to start and then when it finishes on a background thread we get the results back and the entity is finished and we can put it into the game. Kind of right. Right. Yeah. So, um, those are the kind of the two parts of OCS that that determining what a client needs to do and being able to, um, asynchronously, uh, spawn entities and get rid of them. Um, binding culling comes in by replicating those server side decisions over to um, over the network, over to clients so that the streaming happens for them as well. So it's server server side control of bind calling and server side control of OCS on your clients.
00:27:32
All right. Right. Yeah. Right. So what's Forgive me for for asking question Tyler our lead community manager he takes the questions from the community and he feeds me up. He's sitting like people in chat are saying I don't have any questions. You're talking about all the things we want to and I've been doing this for a while. Uh um the question that comes to my mind is what's if we've got buying calling already if if we've got blind calling as a feature complete what what takes so long what what's the time sync here? What what's what's this it's if you've got the if you've got the tech if you've got the technology you've got the code written whatever uh where where's where are the hours going right now? Okay. Um so yeah we've got the async entity loading working we've got um the
00:28:23
rangebased decision making working we've got buying column working it's now fitting those together because what what we currently have without asynchronous spawning is binding will say okay these or the what have we got the streaming manager will say these entities are out of range for this client you don't need them anymore buying will relay those messages, the client will drop them. Um, and that's fine. But when you come into range of some entities, and the client needs to spawn them, it will do a blocking spawn. Puts them all on the main thread. It puts your game launches to a halt. Yeah. So the changes that we the part that we're left with now is changing the the network messaging that handles
00:29:11
spawning so that it can work with this asynchronous process because the network messages as they currently stand um assume this blocking synchronous spawn this as soon as I've said spawn it will be spawned um and I can start sending messages to that entity. But because it's an asynchronous process we've got to load the files from the disk. It can take depending on the the number of entities that we're trying to trying to spawn. So, you know, say a whole planet. Say a whole planet. It's going to take a few seconds to do. Um, so we need the client to start spawning, do all its work. When it's finished, go back to the server and say, I'm done now. Can you send me some updates? And then the server can start uh communicating information to those clients uh to those new entities put them in their final
00:29:58
state because things have probably changed during the time it's taken that from the time it's started from the time it finished something has changed. Yeah. And then when all that process is finished the client can finally go right I've got the I can put these entities into my local simulation and they can start interacting. All right, Chris, you you've gotten off light here, but you're you're uh working with the QA team that's that's testing this. You you've been working with what what what's involved in testing uh object container streaming. What kind of things are you are you working with? What are you looking at? Um, so with it involving entities dropping in and out, it's a lot of traveling around, but it's also finding all the the different entities, how we can spawn them, move them, change the state for one player without changing it for another, bringing the players back together, like
00:30:47
uh the states of everything match up. and we're like looking for all the edge cases as well at the moment. Um because it's everything's fine apart from stalling when you start loading up. Uh like these flights to Levki. Uh there's a stall as it brings Levki in. Um sorry, I'm losing my train of thought. Um sorry, it's all gone. He you me you mentioned uh when two two players in there and and and the calling if if he and I are are are playing you say we're in a dog fight or whatever we're we're a threeperson furball here and I get so far away that the system goes okay it's
00:31:39
going to callull it for me but it's got to leave it for him because he's still in in range. Uh, that's not as easy as I'm saying as as I as I as I'm making it, is it? And and we'll be looking for things like as you're dog fighting, um, you've got a mate in your cargo bay throwing a cargo crate out the back and you're blowing up Gordon at the same time. As another player who when he left, you were just dog fighting happily. He comes back, everything's different. Does his does his client show everything as it should be? Is the cargo crate there in space? Is your cargo bay open? Is Gordon alive or dead? Um, so it's all these different states, a lot of different states. And one of the things that, um, I've been looking at myself this week is we need not just two people doing it.
00:32:29
Um, two people seem to be quite easy to keep track of. Um what happens with the server when there is people or a group of people not just out in space dog fighting but you've got someone at Allosar Grim Hex Levki people out in space uh people off in Hurston. So we've been using headless clients which are just sort of uh lowweight clients don't actually render the game but we can give them basic commands. So, we're filling the server. Um, the zombies Yeah. putting zombies, they they do basic things. They'll they'll get out of their bed at Allar. They'll use console commands to zip around the solar system. Um, so we're seeing what's happening with the server now. What's the load like when there's someone everywhere
00:33:18
or nearly everywhere because the player limit is uh 50 and it's a big solar system, so you can't get everywhere with 50 people right now. It's uh I I've I've no doubt and I've heard I've heard in off-hand conversation uh how anxious many people are to get this on into testing into into wider testing. You know, one of the advantages Star Citizen has in our development is that we run a live testing environment in the middle of our game development and even from Evocati uh to the larger PTU to even our live alpha environment. Even the live alpha environment is a testing environment. uh guys like you and your team, you know, QA here in Manchester and in Frankfurt and Austin and LA, you're knocking on pipes as much as you can. Mhm. But I don't think there's I think it's safe to say that when when this does go out, we're going to see a a
00:34:09
plethora, as Elapo would say, of all new kind of issues, things that we could never that things that we've never seen, interactions with this with this system, the streaming, pulling it out, pulling it out of four people's thing, but not pulling it out of two people's things and and vice versa. It's like that. That that will be a a very interesting challenge for you guys to to track down when that happens. not just going to be hitting the object container streaming new issues that have been introduced since the last time we did a live release um other systems other content has been updated as well. So we're going to be dealing with uh the bind calling object container stuff uh working out what's going on there and all the new fun issues that get introduced just from the general development elsewhere. You hinted at a a good point there. Object streaming hits just about every
00:34:57
single aspect of Star Citizen. I mean the everything's got to be converted or OSR has got to be converted. The asteroids around Yila have to be converted. I mean everything that's in the game now has some kind of fundamental change to its underlying structure. So it's not just the new things like Hursten and its four moons that that that we're adding. It's also everything that's already in 32 gets a fundamental change. So, so new bugs, bugs that you're finding out and bugs that you're chasing, uh, can affect not just the new the the the new content, but every single aspect of Star Citizen today. Like maybe only Arena Commander is safe. Are we doing anything with her? Yeah, I don't think I think OCS is touching Arena Commander yet. So, uh, is is that uh is that daunting?
00:35:46
Um, yes, definitely. I personally haven't worked on on that side of things. I think it's kind of Yeah, I think the amount of teams even it touches. So like sort of um one of the changes we've been having to make for the uh this the async spawning stuff, everything all the components have had to be made thread safe. All the components of the game have had to be made thread safe so they can be spawned off the main thread. Um for the actor that means all the actors components first need to be made thread safe. So you've got a bunch of components that need to be done by the audio team, a bunch that need to be done by AI, a bunch that need to be done on all these different people. Um, and some components depend on other ones being done first. So I couldn't convert, for instance, the local player component until the particle stuff was converted. So we get this big dependency graph that sometimes goes several levels deep uh
00:36:34
before you can actually get um there. And there's a yeah, there's just an awful lot of stuff. There's 100 odd components on the actor now. Um, yeah, when you say components, you're not talk uh backers know components as the things in your ship that you're talking about all the raw components in the most base vocabulary sense that make up the entire game. Yeah. So, it's our game runs off an entity component system. So, the entity is like an actor. It has an actor component. It has a local player component. It's got an audio component. It's got all these various variety of different things. So it's there'll be ones for animation and ones for effects and all these plethora of different things um which will be shared between different entities as well. So there'll be other entities that also have animation components and stuff. Um but yeah everything had to be made thread
00:37:22
everything had to be able to be spawned for say sorry not so when I so when I was asking before about you know where the time sync was going through and every existing component of of the game had to be just about every existing component. And we've already talked about how this has been an ongoing thing. Most of my last quarter, so quote Q2 was spent doing this for the actor. And a big big chunk of that was we had a there's a scripting language called Lua um that we use extensively in the game. Um and it basically had to go um because it's not particularly thread safe or it's not it wasn't there was no good way to make it safe for spawning things as a asynchronously. So um it was a case of going through for the actor there was an awful lot of Louis and there was an awful lot of stuff that just had been around since this engine was created.
00:38:13
Everyone going has evolved over the course of the engine. Um and yeah it was just a lot of tiding up and cleaning up and moving things around. Um and again it was by multiple departments as well. Now, as we're in, we we talked a bit about how even once the the the client side streaming is done, the stuff that we're working on for Hurst and everything, uh there's still the whole server side that needs to come after that. But as we're approaching the finish line of of our client side streaming here, are uh what are you seeing in the game when when when things work? What what's your gameplay experience like? Um I uh overheard someone today use the phrase truly amazing. I don't want to oversell it. Um but um
00:39:02
yeah, we I know the um the backers have complained about performance like quite legitimately. It's not at the state anyone wants it to be. Um but we're seeing uh really raw builds each day. Like typically our performance can be pretty terrible. Um, but now everyone's reporting improvement. Some people reporting drastic improvements. It's it's just it's nicer to test when you're hitting a really nice smooth frame rate. Now, he makes a good point. I I I set Chris up there. I I wanted to know what his experience was like. At the same time, uh, part of this part of OCS, this is where I get to deflate the balloon a little bit is, you know, improving the performance on everything that we have right now, but part of the reason we do
00:39:51
that is so that we can add so much more to the game. So, it it's it's it's one of those things. It's like in in my terrible analogies again, you know, it's it's it's I'm carrying a a 50 lb backpack and OCS is me going to the gym and working out for for 10 year for 5 years or whatever and now that 50 lb backpack feels like a 10 lb backpack and I'm awesome. But the next step after that is to add 500 more pounds to my backpack, you know? So, so it's it's that's one of those things where we when we say it's not the silver bullet that fixes everything. There will still be a continuing network improvements. There will still be continuing optimization of of every aspect of the game. You know, it is a it is a OCS is not the answer to everything. It's it's a it's a foundational technology that allows us
00:40:38
to continue building and expanding the Star Citizen universe to uh well soon Hursten and then Arc Corp and Microte and Crusader and then the entire system and then more systems beyond that. So it's uh so it's one of So it it's interesting. I I I don't think we can predict and I was asking around. I don't think we can predict you know what kind of performance players will see with 33 yet or when Hursten is is added yet because well we're still working on it first of all. We're still putting it together but you know we we we make all these improvements to what we have now and then we add more onto it. So it's it's it's a constant juggling. I think all of OCS is juggling. It's the server juggling things. It's the client juggling things and then with development it's OCS is like adding 50 more arms but then we're going to throw
00:41:26
75 more balls at it and go juggle it. What we're seeing now is like you're um when you're in Port Alisar you're not you know you're not getting updates from Levki Grimhex or anything like that. So performance is great. You go to Lorville your your frame rate is still dropping very low because you're still loading Lville and everything that's in it. Lurville's big. It's it's very it's getting bigger. No no no spoiler there. Lurville is big. All right. So, I think I think that's our that's our deep background. Now, now we're going to take questions. We we Tyler's been collecting some questions uh from the community. We also put up a thread earlier this week in Spectrum where folks who maybe couldn't join us live could submit questions. It also allowed people to vote on which question they wanted to see most. I'm going to start with the single most voted on question. Uh nearly 500 votes the last time I looked at it. It says, oh,
00:42:14
actually I already just answered this. Sorry. Uh, can we expect performance improvements in the game because of OCS in its first iteration or will it or will it only work in making a larger verse viable while maybe allowing more players per instance? Um, so yeah, I just kind of answered that, didn't I? Is there anything else we want to add to that or is that Sorry, I got carried away. I went ahead and answered. Uh, they said they talk about the allowing more players per instance. That's is that linked to OCS in any way? No, that's that's that's not um so the performance benefits, the memory savings that you're going to get, if any hedge bets, your mileage may vary. Um is going to be purely client side. Uh server side, all those entities still have to exist on the server. Uh they
00:43:03
need to be updated when anyone's near to them. Um, so the server load is going to be heavier than it has been in the past because we're increasing the scale of um of the the solar system massively, right? Massive amounts of new content. So, um I don't think it's likely we're going to increase server count uh the player counts. You make you you just made a really good point there. The these improvements uh uh when when when this system comes online is strictly client side. Yes. So the next step is the server side improvements and until that happens we are still adding a planet and four moons to these servers. So the the load on our server side is going to increase. Pretty dramatically uh when this comes online. So it'll be interesting to see to it's I think we're all going to be looking at
00:43:51
what the what the improvements are and how it all works. Yeah. When that time comes. All right. uh from let's see um why are client and server side OCS split from the live chat like why why aren't we just working on both at the same time? so yeah okay server side streaming requires um some extra technologies that we haven't got yet. So we're by working purely on client side streaming uh we're getting some of the benefit uh onto our clients um and we're not having to do all the technology at once. So the kind of extra pieces that we need are uh on server side stream once um a server decides to
00:44:40
unload some entities what happens to uh they're going to have to be persisted out by current state um and then we're going to have to reload that that state back in. So, it's it's not clear if we need the the full persistence to work, but it's going to have to be something pretty close uh to full persistence to get the server side stream at work. Um how will does OCS affect uh uh our view or render distance? It shouldn't do. Um the as I say as Gordon said as well that the the range at which entities get streamed out and streamed in uh should be pixel size or thereabouts. So you shouldn't notice anything. Uh but because of the time it takes to load
00:45:28
things in there may be a little bit of popping but it shouldn't affect the draw distance really. Okay. Um, what are the uh what are the current improvements that OCS has already you talked a little bit about that already. Are there any that any anything else that you guys can can think of right now? Current improvements that you've already seen uh from a development point of view, it takes me less time to load the game. Um, which is nice. Um, so if I'm wanting to test very simple stuff and I only need all the star, I just need to the PU takes less time for me to load because it doesn't have to load all the stuff that's out of range. So you can iterate between builds faster. Yeah. So it's it's less of that. It actually makes it possible to load the game as well because the map's so much bigger now. Not everyone has quite
00:46:17
large not everyone here has quite large uh memory machines. So some of them can't actually load the full map. It's that big. So with uh they need um bind calling on uh well we've got two maps. We've got a cut down map and the full map. They have to work on the cut down map or if they want uh want to load the format they have to turn bind colon on to be able to get it actually in working on their machine. It's that much bigger. The um the reduced memory is helping us as well with um I mentioned earlier about filling a server up um which is effectively using one tester's time across multiple machines. The less memory the game's using, um, the more game clients we can put on a single machine, it kind of the headless zombies. The headless. Yeah. Um, how deep does the container tree extend? In other words, is there a limit
00:47:05
to how many containers you can put in other containers? That is deep. Yeah. About that deep. Yeah. So, there's no limit to how many containers could be nested inside each other. Um, a container. Well, the the solar system is built out of a root container. So, we've got the Stanton system container. Uh, and then inside that, um, there's a container for each of the planets, I believe. Uh, and the planets themselves contain object containers for the moons and the space stations and the planet itself. And then on those, there'll be like for Grim Hex, there'll be the Grimhex object container. And then inside GrimHex, there'll be object containers for the different shops. So I think we're probably getting on to about seven levels deep there. Well, there's in theory there's no limit.
00:47:53
Okay. Uh what's been the biggest challenge you you've been you've been fa you faced in the development of OCS? I I personally I think you're going to get a different answer depending who you ask. I I'm extremely tempted to say networking, but I'm not. I'm gonna say that it's probably the stuff Gordon was talking about the conversion of um entity spawning over to uh to to be thread safe thread safe and and asynchronous. It's been a massive effort that and also kind of the production side of things because it's touched so many teams. Yeah, it's to manage all that. uh he mentioned that I was sitting with uh Luke Hail who's the associate producer who's who's uh handling a lot of this um who wanted to be here but he's actually in a production meeting
00:48:41
right now. uh he was he was saying that some of the uh dependencies you like like I need to work on this but in order to work on this this person needs to work on this and for this person to work on this this some of the dependencies uh uh uh right now are four deep you know- which is which is which is you know just to draw the thing it's he's need he's needing to work on something but he can't work on something until he finishes his work on something and he can't finish it until I'm done working my thing and I can't finish until you finish working on the Why are you slowing us all up? I apologize. Yeah, but uh dependencies are one are one of the the major things and that's what uh that's what producers bring to this and I know we wanted to we wanted to have the producer on to to have that final uh uh perspective on here just juggling all those dependencies and and
00:49:29
you know and prioritizing. It's like it's like your work is holding up two people's work, but his work is holding up three people's work, you know, and and figuring out that prioritization of where to put resources. Uh that's that's been a major challenge. How about you with at the end of this the end of this chain at QA? What's what's your biggest struggle? Um I think the fact that it touches everything means we need to test everything and it's not just testing everything with bind calling because uh when you find an issue on our say the bind calling server um is it a bind calling issue you sometimes you can't tell just from the symptoms of the issue alone you go back to a regular server you need to reproduce it there so you're not just testing everything you're testing everything twice uh with it um affecting things coming and going um these aren't tests you can usually do by yourself.
00:50:17
You need um two testers, preferably as many testers as you can get together. And uh we only have a finite amount with regardless of how many zombies I can necromanance. Um so it's it's been a massive a massive strain on just our resources of getting everybody we need to be where we need to be covering as many things as we can. One of the questions in the live chat just said, uh, while while testing OCS, have there been, what's the funniest or most interesting bug you've encountered? see, now you've said funny, all I can think of is crashes. Um, there's been a fun ones where you're trying to catch an AI who's decided to run away from you. Um, I think this was
00:51:05
earlier on. So he's in range flying away and you had an issue where he would reach um sort of like outside of uh your streaming window and he'd stop. So then you you'd catch up and eventually you'd get the updates from this AI and he'd fly a bit further because actually he's even further. So you're constantly having this little stop start race against the guy you're kind of catch up trying to catch up and you don't even know where he really is. That doesn't sound very fun, dude. Um, how will different hardware units like CPU or GPU or RAM or VRAMm uh benefit from OCS? Is this is this something that can conceivably help all aspects of computer? You squeeze a little more CPU out, squeeze a little
00:51:52
more memory out, or is it just or only certain aspects of performance? But I think mostly it's going to affect RAM. That's where you're going to get the biggest save. Um, it should I'm being a bit more cautious than than Chris. Uh, it should have improved things on CPU. It probably won't uh have much effect on GPU. Yeah, I I think from my exper Well, my experience of in the last week or two, um, there will be CPU improvements probably, but we're also ramming a lot more stuff in. Um, so although while there might be some gains, um, we've also got a significantly more large amount of EI and such. That's uh most of the questions that that had came in on both the uh spectrum thread and the live chat have to do with people wanting to know how big the improvement is and the
00:52:39
and the difficult thing about that is that like I said we are adding every day we are adding more things into it uh uh aspects of the world are becoming art complete. You know the ship ships are are finishing up uh you know the we're adding the hammerhead which like the biggest ship we've we've we've added in the persistent universe at this point. So, we're making all these improvements, but at the same time, we're adding all of these new things. So, it's it's hard. It's a it's kind of like a shell game at the moment. Forgive me, I'm on an analogy trend. J Keith is just Oh my god. You don't have to look at me like that. I know. But, uh, but yeah, it's a we will be finding out the the degree of improvement together is basically I think this the safest way to to to uh to answer that question. Does OCS have any drawback?
00:53:28
Um, apart from the time it takes for us to test it, that's the only one like on our end. You mentioned earlier some of the challenges that in involved like like the scanning things like like h having to put having to leave the markers when something is called and what so that people can still see it and stuff. Yeah, I think um I don't want to generalize but most gameplay programmers are quite happy to just assume everything exists all the time and sort of do things a little bit. I'd be I'd be very happy if I just did every single entity in the client at the same time because it'd make my job a lot easier. Whereas now there's a lot of I've got to make sure every entity I'm accessing exists and if it stops existing while I'm talking to it, what do I do? Um and lots and lots of yeah lots more technical challenges and more um
00:54:19
it just it just requires you to think about things. Not not that we don't think about things, but you've got to be very aware of the implications of a lot of stuff might stream out or I might I might if I'm designing this feature, will it work when not everyone's here? Do I need to move it to exist on the server rather than the client? Do I need to um tweak things to make them work? Um, you're good. One of the questions that just came in the chat says, uh, after this hits the community, you know, when OCS hits the community, uh, how are they going to be able to identify bugs that are related to OCS and not related to something else? It it's because you know many of our members of our community participate in the issue council which is our fantastic tool that you know lets the community you know assist in the bug tracking and the bug
00:55:06
discovery process like how how how do we know when something is OCS related or or it's not just one of the other thousands of types of bugs that um I'd say any bug that you spot report it to the issue council anyway and let us sort it out. But probably the the kind of things you're going to see are um perhaps missing entities or uh stalls as you approach planets or derelict or whatever. Um what else might you see? Some kind of popping as things appear. Yeah. Um the likes of teleporting actors. Um that's one that's kind of floated around a little bit. So just see guys just jump around the place um occasionally sort of issues with
00:55:58
uh one of the questions that came in was the difference between asynchronous background spawning and normal background spawning. I think we we covered that a little bit. Is the only difference just whether it's on the main thread or whether it's on the the secondary threads or Yeah, the the background part of that says that it's it's off the main thread. the a you can do asynchronous spawning on the main thread but it's a a bit of a it's a different way of doing things. So it's for us it's a very precise way of describing the process. let's see we actually we actually we did cover quite a quite a few of these. Uh you me you mentioned that uh OCS uh is still necessary for Squadron 42. Uh uh how does it play into Squadron 42 just in the size of the the the the the full systems and everything when you're
00:56:47
Yes. But basically, again, it's down to the size of the maps. Gotcha. All right. Um Yeah. Well, let's see. Uh that's I'm looking for other questions that have come in. We've actually hit just about everything here. Uh is there anything else you wanted to if you if you could tell the community one thing about OCS? If you wanted if you wanted to make sure that people understood one thing about it, what do you want to make sure people understand? that it's it's been a as you said earlier like three years since since we kind of started work on on various parts of it and although we've not been working not everyone's been working on it flat out for those three years it's it's taken yeah it's taken a massive amount of time a lot of changes tons of people it's it's not down to to just us in this room or or our teams in
00:57:37
particular it's it's a companywide effort it's been Yeah, been a long road and it's hopefully worth it when we finally get there. What'd you get? Yeah, I think just sort of to reiterate that it's it's touched every single element of the game. Um, and it is still currently it's still currently touching every single element in the game. Um, and it is it's I've not it like I said it's a collection of features, but I've never there's never been a feature that's really there's not been as many emails that have come out to absolutely everyone going, "Is this going to break anything for you?" Um, it's needed everyone to pitch in. You see a whole lot more BCC everyone. Yeah. On OCS threads than you do just about anything else. And I think as well, as Cloud said, it's not
00:58:25
been a it's not been a short road. I mean even well before even object container streaming there had to be object containers themselves and that started as I was starting. So that's three and a half years ago. Um so it's been that was kind of the the very beginning of it and it's um this has obviously been on the plan since then. It's just taken three and a half years of tech and improvements and it doesn't end with this next release in Hursten. It's no well I mean this is the kind of thing it's it's going to be ongoing forever probably. Um there's always going to be things to improve and things to more stuff that he's added to it. Um sort of the server streaming is obviously the next major step and then there's there's stuff beyond that as well. What about you Chris? Um just to reiterate that it touches everything and from the QA standpoint
00:59:14
there's show me on the game where OCS touch everywhere. Um yeah it touches everything. There's going to be a lot of edge cases that with our department like as cool as everyone is there's only so many of us there's only so many evocart there's only so many PTU testers as well even though there are a lot things that are going to things are going to break I think um please submit to the issue council and always submit your crashes yes uh that that'll that that'll be my point that that I want to add uh this is a massive undertaking that touches every asp aspect of the game. It's why I wanted to do this show. It's why when they gave me one week a year to come over here, I wanted to use it for this, you know, it's it's this this is every department from from UI to to to every
01:00:04
system design, mining, you know, FPS, locomotion. I mean, every every game system touches OCS in some way because of all the conversions to thread safe and stuff like that. um when this does get into your hands uh for for testing whether you're on evokati or whether you're in PTU or whether you're still helping us test once it reaches the live environment um your help on this is going to be appreciated uh that this this is one of the most the biggest fundamental changes to to Star Citizen that I can think of in the times that we're being here and it's not just on the feature side not just that we're adding a planet to this which is a pretty massive change contentwise but under the hood we're making this massive change to how everything works under the hood. So, it will be a uh it will be a uh it'll be an
01:00:52
adventure and I think uh for folks who have followed our development and continue to follow our development as it goes on, this will be one of those watershed moments when we look back and be like, "Oh, yeah. I remember when they added OCS and and everything was upside down and stuff like that." You know, it'll it'll be it'll be an interesting thing to get through. And uh I'm I'm excited. It's it's one of those building blocks and one of those growing moments for the game that that we've been working towards for a long time. and I'm excited to to finally finally approach the the finish line, whatever that may be. So, yeah, that's it. I got no other housekeeping. This was all I had to do today. So, uh thank you so much, Clive and Chris and Gordon, for taking time at the end of your week to be here and to agree to this. I know it's uh not everybody's fun most fun thing to do. when I got into video games
01:01:41
and I want to make video games, I did it so I could show up on live streams with Jared. So uh thank you so much uh Clive Gordon and Chris uh for Reverse the Verse Live. I'm Jared Huckabe. I've got nothing else to say. Uh tune in next week, but I'll be back in LA after my World 3E tour and we'll find out what we're doing then. Uh take it easy everybody. See you later. Thanks for watching. For the latest and greatest in Star Citizen Squadron 42, you can subscribe to our channel or you can check out some of the other shows and you can also head to our website at www.robertspaceindustries.com. Thank you very much for watching.

Transcript from YouTube’s automatic captions — punctuation and Star Citizen jargon are approximate. The video itself is hosted by CIG on the official channel.