Star Citizen - Calling All Devs: Core Tech Updates

Transcript — every row opens the video at that moment

1. 00:00:08

   Hey everybody, welcome to another episode of Calling All Devs, our question and answer series where we take questions from you, the Star Citizens themselves, and pose them directly to our developers over video conferencing. On this week's show, we are tackling some of the most voted up and requested topics from the community. We are talking core tech. We're going to be talking iCache, we're going to be talking global persistence, and then we're going to be talking about server meshing in the end. So, to get us started, we are here with Chad McKinney. Chad, how you doing, man? Doing great. Thanks for having me. Thank you for being on the show. Now, we are we are coming to you for this because you are you are part of a team of folks who work across the studios across the world who are all heavily invested in this core tech that that will allow us to expand Star

2. 00:00:56

   Citizen to the size and the scope that we that we all want it to be. Um so, basically, let's just start the conversation. Talk to us about iCache. Yeah, iCache is effectively kind of shorthand for a collection of services that are interfacing with the game servers and back-end databases to allow us to both write out and query data efficiently and in a fault-tolerant way. So, this is really important because right now, the way the the servers are working, they're using an in-memory kind of database for this for this global data. Ultimately, that's not feasible. We if we need to build it out so that things can work in a mesh. In order for that to happen, we need to actually write this out into a place that all the servers can look at it and agree on what the

3. 00:01:45

   state of the world is. So, the iCache is our solution to make that access scalable. And right now, the iCache infrastructure on the back end is actually largely done, and it's it's um now at the point where game side, we're hooking up our game code in order to correctly write out the data uh efficiently, query the data efficiently, as well as fix up some of the game code systems that had previously maybe had some assumptions that don't work correctly within the new context. Uh so, that's kind of where we're at at the moment. Effectively, what we're trying to do is cache certain amounts of certain data that we know that we need. And this prevents us from hitting the database, which is a more expensive access. So, and by caching it, we're optimizing our access. And we're also

4. 00:02:36

   able to build in a certain amount of fault tolerance in the system through various techniques like duplication and regeneration. So, this gives us fast access for the information that we need when we need it. Um and then, at the same time on the game side, okay, I just lost my train of thought. I'm sorry. Take two. So, so yeah, it's effectively, like like you mentioned, like the difference between having a hard disk where the data is stored for long-term storage, that's like more permanent, and your RAM that it kind of effectively gives you faster access that you're happier to write to, but you know that it's temporary. So, the iCache storage is storage that's not meant to actually capture and define the state of the world, but rather is an intermediary between the final state of

5. 00:03:23

   the world, which is the actual database. So, it's a yeah, that that's a good analogy. Uh so, you did say earlier that we're pretty much finished. I want to get a little more little more clarity on that. That that's that doesn't mean uh what does finished finished mean? Sure. So, I think to to clarify here, what I'm talking about is that the particular services that compose the the infrastructure on the back end for the iCache framework is largely written. So, the back-end engineers have written it, they've tested it, um the APIs have all been negotiated. So, we we have to define not only the how the service works, but also like how it communicates with the game server. We've done all that work. We've built out the structure of the data, how the database is laid out, all that's done. And so, now it's

6. 00:04:12

   taking this this system that we have and actually connecting it to the game itself. Yeah, and to be clear, like we have known for a long time that we're going in this direction. So, on the game code side the changes that are happening right now aren't normally transformative or or refactors or anything like this. They're smaller sets of changes just just to make sure that we have followed all the best practices Awesome. all over the place and that things are persisting correctly and that the the querying mechanisms that we're using use the new API. So, there is some small changes as far as like the API that you use to query data. There's some new concepts that we've introduced specifically to help leverage um uh in order that and specifically to allow us to support things like physicalized inventory and the new kind of cargo

7. 00:05:02

   changes that we're going to support that are also going to be working with physicalized inventory. So, we we have been going in this direction, but we need to do some final work on the game code in order to make it operate correctly, but also we need to make the fundamental querying mechanisms and physicalized inventory infrastructure on the game side to support the game systems on top of them. So, it sounds like because we've known we were working towards this, we've we've we've already worked on a lot of precursors. We've already uh uh put our proverbial ducks in a row. Yeah, we've we've been heading in this direction for actually years. So, whenever we started on client-side streaming, we know at that point, actually before that that we would be ending up here, and then the ultimate destination is spoiler alert later, server meshing. So, I think we've made

8. 00:05:52

   tremendous progress, and we've done some rather transformative changes in some of the game like underlying game engine code that I think um is is amazing. So, I I think that we've made a lot of progress. I want to say that we're um closer to the end than the beginning is how I would put it. I think I think that's fair and safe to say. I appreciate that. Um and when iCache comes online, I know it's I know it's a a stepping stone towards these other things we're going to talk about, but what kind of immediate impact would it have for the players? Like how how will the players recognize when iCache is in and working? What what what How does their experience change? Sure. So, I would say that some of the the most apparent changes that will come

9. 00:06:38

   about is that we're going to have both a combination of better response on on server and service queries for this kind of persistent data as well as better fault tolerance and and scalability. What that will like directly manifest as all the different times in the game where you're end up interacting with these databases, which is constantly. It'll be things like when you spawn a ship, whenever you change your your load outs, or your helmet, or your armor, whenever you change something about your vehicles, whenever you purchase something, whenever you deal with cargo. There's all these places in the game where you're interacting with the persistent state of the game. And in all those places right now, we're struggling both with scaling to support the number of players that we want, especially whenever we have like a big uh event or, you know,

10. 00:07:28

    new release, we're really pushing on the current infrastructure that we have. Um but on top of it, it'll make it more fault-tolerant so that whenever inevitably something goes wrong the game just doesn't fall over. With the current what we call a monolithic pCache that we have, if that service goes down, that's just it for all persistence queries for the entire, not just server, but all the servers. So, this is clearly, you know, not the right place to be. Um but this new set of services addresses that by not, you know, putting all your eggs in one basket, as it were, and allowing for a multiplicity of handlers to be able to react to any request. So, iCache. iCache. iCache. Okay. So, the next step beyond iCache in these pillars is

11. 00:08:17

    is it's kind of server meshing and global persistence, but for the sake of this conversation, we'll move one in front of the other here. Let's Let's go to Let's talk global persistence. Once we have iCache, how does that take us to global persistence? Right. So, global persistence is something that is supported now that we have this iCache framework. What it means is that we're going to start writing out data about the state of the world that does not include player account data. Previously, we would only store data that was relevant to a particular player's account, things like their ships, their weapons, their item port attachments. Perhaps you stole something from somebody, we'll store that. But that's where it stops. We want to do more though. The game that we envision is

12. 00:09:03

    this incredibly high-fidelity, high-persistent world where your actions matter, you have an impact on the world. Well, the only way to make that work is is to actually store out the state of those changes and to store out the state of the world itself. So, that's what I I've been calling global persistence. So, the reason why you haven't done this to date though is that the amount of data that we're talking about is like a magnitude or two more. It's It's much much more data. And if we tried to write that out with the current system, it would just, you know, cough over and die. It would not go well. So, iCache gives us the scalability to actually deal with that kind of that scale of data. Um but on top of that, it's a requirement to support server meshing. But before we get to server meshing, you

13. 00:09:53

    know, global persistence really is, in my opinion, a feature in and of itself, and it's part of the vision of of the ultimate version of Star Citizen. So, to me it was very important that we make sure that we execute on this, that we're finding all the places in the game that we want to support this kind of persistent state, and make sure that we update all the game systems to work accordingly. So, we actually have a version of server or sorry, we actually have a version of global persistence right now. It's just not apparent to players because when they log out of the server and log back in, they'll go into a different server instance often, and they'll see something different. And if the server that they're on is crashes and recovers, the history that it had is gone. So, we actually are maintaining much of this data, but it's just not in a way that

14. 00:10:41

    players will really ever tangibly see. And where we're going is making it more permanent and on top of that, recoverable. And so that as you get into the game, we're going to be able to expand out using server meshing, the number of players within effectively an instance or network we'll start calling a shard. And as that expands, you'll be playing more and more in effectively the same kind of world line or or set of history. And your experience will be that you will see you and your play groups, you know, your effects on the world become more permanent. And the idea is that for all of the players in the game, they end up seeing a version of the game that is effectively permanent for them. It is from the time they start playing the game to the time that they end, everything is there, right? And to be

15. 00:11:30

    clear though, there are going to be sets of gameplay around, you know, maybe dynamic events in the game that happen, or there will be places in the game where maybe there's more maintenance that will recover some of the the the broken and messiness that accrues from all this player behavior. Uh you can imagine that if you have maybe a military base or or maybe a high traffic like high high cost neighborhood in like area 18 or something like that, they're going to have more facilities in place to maintain it, to keep it clean and fixed up. Whereas maybe a seedier location perhaps in pirate space is going to you know, all the mess that's left around is going to stick around for a while. At that point, we need to talk to Clive. We need to get the skinny on server meshing. So, Chad, I'm going to let you go. Thank you so much for taking the time to be here on the show. You're always a a pleasure to have on.

16. 00:12:18

    Thank you for having me. All right, take care, man. All right. So, uh costume change. It's the next day, and my next call is for as promised, Clive Johnson. Clive, how you doing, man? Hey Jared, I'm good. How are you? I'm doing well. Thank you for taking the call. Uh we are talking core tech updates for Star Citizen. Uh we just got done talking to Chad McKinney about uh service streaming and iCache and global persistence. Uh and we are now coming to you for a little talk about uh something we like to call server meshing. So, um because every show is somebody's first show, and there might be folks who've joined the project who don't know what it is what we're talking about. Um let's just start by introducing uh the idea of server meshing, what it is, and why we're pursuing it. And then we'll go for an update after that. Okay, well, currently um

17. 00:13:07

    um players play the game, they connect to a server, and it's a single server uh holding up to 50 clients running the entire simulation on that one server. that's um going to be a a a severe limitation um because obviously there's a finite amount of processing available on my computer. Uh so, there's a finite number of clients that it can support. Mhm. So, as we want to expand the the number of concurrent users, the number of active users in the same simulation at the same time, we need to split that simulation off over multiple servers. So, the original idea is to have um multiple servers running different parts of the simulation. So, you might have a server running

18. 00:13:56

    uh the game at Port Olisar, you might have a different server running the game at say, Levski. And that's great, and they're completely independent. But the problem then becomes how you get from Port Olisar to Levski or back again if the um simulations are running on different servers. So, we need to connect the servers together. And what that basically means is we'll have all these servers running different parts of the simulation, and they will um selectively choose which other servers to connect to, so that the players can transition back and forth, or that they can share parts of the simulations together. and then they'll drop that connection when it's no longer needed, and they can make new connections with other servers. All right. So, now that we've uh

19. 00:14:44

    now that we've we've reiterated for some, we've introduced others to the idea of server meshing, uh where are we at right now in the process? I know I know we've got teams working on iCache, and uh global persistence is on the way, and both of those are absolutely necessary and crucial to server meshing as we talked with Chad McKinney. But uh your team's not sitting there just waiting for them to finish their work. You guys are You guys are in progress now. Uh tell us tell us what you're working on now. Uh well, right now um we're working on a couple of things. We're working on um getting the time synchronization uh correct, which probably doesn't sound very server meshing related. But we've been aware for a while um that there's been um lag um rubber banding and kind of desync

20. 00:15:32

    issues present in the main game um obviously with all these servers connected to each other in the server mesh, that's going to make the problem worse. That's the thinking, so we kind of need to address it a bit now. Um we can't promise that we're going to completely solve it first go, but we're we're investigating the situation. Uh and trying to make as many improvements as we can. And a lot of the issues seem to come back to time synchronization between the clients and the server. And in server meshing, just to remind everyone, servers will be connected to each other, so one server will be acting as a client of another and receiving updates from it. So, they could too experience um rubber banding and desync and so on. So, we're going to try and make sure that clock synchronization is right. Um so, it's it's not just about getting

21. 00:16:22

    the um the clocks actually in sync. It's more fundamental than that. Some of the problems we found are changing how the engine actually handles the concept of time, which sounds quite grand, maybe. But yeah, it's it's basically it's just fixing a load of bugs that have crept in over the years. It's uh it's quite tricky, and it's quite um detailed work, but yeah, we're getting through it. We're getting seeing some good results now. Hopefully that will be done soon. And then yeah, we're also looking at getting two servers talking to each other, which may surprise some people that that's not the first thing that we did. But it kind of was cuz you know, obviously the first thing you try when you're trying to build server meshing is you

22. 00:17:09

    just try and connect two servers together, and you see what happens. And we weren't really expecting it to work, and we weren't too surprised when they didn't. But it was quite a useful exercise, and it showed kind of what didn't work. Um and what we would have to change to make it work. So, we kind of built those tests out, and people are working through them right now. Uh and we're getting pretty close to where we can give it another go and have a a reasonable shot at the two servers actually being able to communicate. So, the the the fundamental problem there is that the engine's based on the assumption um a server thinks it owns the game session, which with server meshing obviously they're sharing the game session between them. So, we've got to

23. 00:17:57

    make one server set up, and it kind of has the game session, and then another server wants to join it, and it goes, "Okay, yeah, you can share into it. You you you now have as much authority and ownership of this game session as I do." It's a bit of a shift for how the game code and the the network code thinks about um game sessions and how everything relates to each other. So, throughout the game code, well, throughout the engine, there's plenty of checks going, "If this is a server, do something like this. If it's a client, do something like this." Um which doesn't fit with kind of the server meshing model where you've got an authoritative server, you've got a non-authoritative server, and you've got regular game clients. So, all these kind of if statements were based on the assumption that you could only ever have one authoritative

24. 00:18:44

    server. So, they all needed to change. We're talking like thousands of locations in the code. So, we kind of did an audit. Um we split that up into hundreds of tasks uh for people to look at. That got spread out across basically every team has an engineer in it. And they've been working through it. Um nearly at the end. There's a few stragglers left to do, but we're nearly done now with the change. It was this new Excuse me. This new API that basically says, "Am I the machine that has authority over this entity? If I am, I can do it this way. If not, I'm supposed to do this instead." Um so, that was a massive change, and that in itself is getting put out into the new builds, and you can't see any changes. And that's a good thing because it shows that we're not

25. 00:19:31

    breaking stuff by making these changes. So, it's it's an early test. I I think we're doing okay. So, the the idea was um to try and get as many of the the kind of um the tasks as much of the work done early where it might have a knock-on to other teams, so like changing how other people write code to decide whether something should be handled authoritatively or not. Um how they might handle time, any any of those kind of changes we want to do as early as possible. So from this point on, people are writing code the correct way and we've got less to go back and kind of revisit. Got you. Um and because there's such a a huge amount of code that's already been written, we're kind of expecting it to take a while for those changes to get um to get worked through.

26. 00:20:18

    So while that's going on, we the network team are kind of working on getting the servers connecting to each other. Um also done some other changes to it like headless clients and all things good to try and allow um them to be used for automated testing. So that when we get these server meshes up and running, we can automatically test them rather than have to UI try and sort these things out. Um yeah, so there's there's a lot of different tracks all kind of progressing in parallel. Um and we're we're making progress. It's not easy. It's a lot to cover. I think overall, it sounds like between the conversations with you and and Chad that we're making some pretty good progress. Uh we're generally happy with where we're at. yeah, anything else you want to add

27. 00:21:07

    before I let you go? Um no, I don't think so. Um have I forgotten something? No. No, that's all good. All right, well that about does it for this episode of Calling All Devs. I want to give a very special thanks to Chad McKinney and to Clive Johnson for for stepping outside their comfort zones a little bit and and speaking with us very candidly about the progress of their work. It's it's not always the easiest thing to do. Uh but we always appreciate it when they can. Um and then just one more word about iCache, server meshing, global persistence, all those things. Those are major signposts in the road of Star Citizen's development, but the work does not end there. Uh while we're feeling very good about our progress and as I think Chad probably put it best, we're closer to the end than we were to the beginning. Uh we will we look forward to continuing

28. 00:21:54

    to update you on the progress of not the just these features, but all the features of Star Citizen. Uh these ones and the ones beyond. So for Calling All Devs, I'm Jared Huckaby. Until next time.