Star Citizen: Bugsmashers! - Server Bugs

Transcript — every row opens the video at that moment

1. 00:00:06

   [Music] Hey everyone, welcome to Bug Smashers. I'm your host, Mark Ament. I'm a senior gameplay programmer here at Ciga and I'm here to take you behind the scenes on today's hottest bugs. Bashes. Hey everyone, we are here in Crusader and I have two clients. Uh, client three or he's number two and then client one. The reason why it says two, three is because the dedicated server is one then two and three. So, I have my two clients and I'm going to have the third guy spawn a debug ship. Normally, I would

2. 00:00:56

   run up to a terminal and spawn a ship, but because I am lazy and I don't feel like running, I am going to spawn a uh let's spawn a gladius. Woo. So, I spawned a Gladius, and he'll automatically get teleported into there. Oh, love debug commands. And I have the other client. Uh, let's see. We got to go find where that ship is. Let's go to light speed. There he is. So, so we have the guy in the seat. Pretty pretty standard stuff. And what we're now going to do is go back over to the guy in the seat. Guy's running around. Run. And we are going to disconnect.

3. 00:01:45

   Goodbye. Let me remove this break point. All right, let's disconnect. And of course I will trigger break points everywhere else because of other debugging. But um so as this guy processes its exit, let me explain uh the bug and why we're here. Um there the bug is basically um when you hit two people or any number and one guy is in a ship and he leaves when he comes back on his machine. getting up out of the bed, spawning into the normal crusader area. But on everyone else's machine, including um um some guy watching it, like this guy right here, he will spawn back into the ship. And unfortunately,

4. 00:02:34

   everyone um thinks he's in the ship except for the server and that local player. Um according to both of them, he's running somewhere else around um Crusader doing whatever he wanted to do. And now we have a desync between all what the remote players to see and what those two the server and the local client sees. And what that also means is you get some weird oddities that allow you to kill the player um in the ship when he's not even there. He could be running around and then he dies all a sudden because well it's reporting the wrong spot. So once this guy has fully disconnected, we'll see exactly what's going on. Actually, it takes a little bit to

5. 00:03:22

   disconnect. So what we could look at is in here in the player code, um, when he gets deleted or destroyed, before that happens, we have to do a bunch of cleanup. And this particular section of the code is when the player cleans up. And as you can see here, when we clean up, we're telling um uh the vehicle, this is we call it the seat host because shenanigans. Uh we basically tell the vehicles uh we get all of his seats and basically say, hey, evict everybody and particularly evict this guy cuz he's getting deleted. And the server is supposed to tell everyone else, hey, this guy's getting removed. Um so it comes into this bit of code. on on the server and it will run exit. But if you notice, it's running this thing

6. 00:04:12

   called exit init. And exit initial uh implementation to remove him from the seat. This will play an animation or teleport him to an exit position, whatever it needs to do. It's the implementation um that all clients will run. However, it's not the code path that set makes the server sell everyone to run it. It's like, hey, this is the stuff that they would be running. So it's responsible for running the exit code the actual implementation but not actually dispatching it to letting everybody know that hey I should exit instead of running exit int our implementation we should be running nope exit. Ah come on see exit there if I could spell. There we go. We

7. 00:05:04

   should be running exit. Uh exit will run the implementation. Um it execute exit will do an animation then eventually do execute exit but it will also send the state to all the remote machines. And since the server is not going through this particular path, it's just skipping it and just directly removing him. All the remote machines aren't knowing that this guy shouldn't be in the seat anymore. Uh so he's still processing his exit. Um and we shall go to reset. So ideally what we'd want to do in here instead is change this to exit so that it runs the same sequence of events except it makes

8. 00:05:52

   sure it dispatch it to all the remote clients. All right. So now this guy's back. So, this guy is still looking at the ship with nobody in it. And we're going to reconnect this guy in to see the bug. And as we see the bug, um, what we want to do to fix it, let's check if we're the server. Make sure we exit. And we want to skip transitions because we don't want to play an exit animation. And we want to force it. Now the other thing we could have done

9. 00:06:46

   is instead of in here where we uh dispatch the uh the event uh we have all the clients do it. So that way they could also exit instead of having the server tell them which is true except we also want to make sure that the server is authorative over everything especially because of persistence and I mean when we have the same entity IDs we need to make sure that everyone's up to date and not just because some guy thinks a certain state should be in. So having the server dictate these things is always a a bonus. Now, there is code elsewhere that handles um removing the player properly when he's in the seat, but it won't correctly reset his um entity ID. And that's why we need the server to tell everyone that

10. 00:07:36

    this is the case, especially since um before we delete the entities, it's up to the server to say which stuff was deleted. So, if the server says, "Hey, remove this guy from the seat." followed by delete him. Then we ensure um all the clients have the same uh logic. Let's load the level. All right. So, this guy spawned in. He thinks he's in the bed. Server thinks he's in the bed. However, that other client thinks he's attached. And he even has broken animations. So from the perspective of um the the remote clients, this would be everyone

11. 00:08:26

    and this guy, they're out of sync. So let's give it a try with our fix and see what happens. Compile. Compile. Compile. See, this is the part they don't show you all the time. Yeah, we might fix a bug, but it takes a few bit of time to compile everything, restart all the servers, verify the fix, then give it to QA, make sure they verified it, and then hopefully a day or two later it gets in or a new bug arises because we just uncovered another thing that this thing was hiding. Ah, game design, art development. All right, we are back in business. So,

12. 00:09:17

    let's fast forward. Spawn the gladius all over. And the other guy use magical cheats to fast forward. Let's see. Where did he spawn? There he is. All right. So, there he is. Now we're going to exit. Bam. Sorry, I walked out. Now we are processing the exit. So the server will tell all the remote clients, hey, this guy has left. As you can see, he left. The other guy is slowly leaving. And I

13. 00:10:12

    just realized um that true actually enables uh transitions where we don't want transitions on. All right. And just to be extra safe um before committing to be available, we will remove that. And what this will do is um allow the local client to perform this as well. However, the server will still dictate the exit so that um the local client will make sure he's in that state and just to reinforce it, the server will say, "Hey, flush it." And with those two things, the bug will be good to go. And once this guy Oh, no. client two

14. 00:11:01

    um disconnects. We'll reconnect them and we'll see that he's no longer in the seat when he reconnects and he's in fact in his little cubby hole on all machines. All right, as you guys see, he connected fully and he's no longer in the ship. So, bugs fixed, problem solved. Hope you guys enjoyed. until next time's exploit. So, as you guys saw, we had a little uh bit of a bug on this the server where when a guy was leaving a server and he is inside of a seat, uh the server would remove him from the seat and wouldn't tell anyone else about it. Um we have everything being server authorative. So if server doesn't tell the clients or remote clients that something happened,

15. 00:11:49

    uh things get a little bit wonky. And because that server didn't tell everyone else that this guy left the seat, um everyone else thought he was still there. And once he rejoined, since we have persistent um numbers for what our players are, our entity IDs, um everyone else thought he was still in the seat when he reconnected. So all the other remote machines thought he was still in those um uh previous seat. And even though he was like maybe miles away from re when he rejoined and from a perspective he's still in the vehicle but he's not now the server correctly informs everyone that he has left and now everyone sees that he has successfully left the seat. So when he reconnects he won't be in there and then that can't be used as an exploit. Well, hope you guys enjoyed. Until next time.

16. 00:12:37

    Thank you for watching. So, if you want to keep up with the latest and greatest in Star Citizen and Squadron 42's development, please follow us on our social media channels. See you soon.