Star Citizen: Around the Verse - Performance and Optimization

8 March 201800:58:271,357 linesWatch on YouTube ↗

Transcript — every row opens the video at that moment

00:00:15
hello and welcome to another episode of around the verse I'm Shawn Tracy technical director of content tech art tech animation general tools and engine tools and I'm sandy sandy Gardner so this week we bring you a very special episode we'll get to our monthly star citizen project update followed by a look at the ongoing optimization processes and Jessie Spano will learn about the dangers of too much caffeine what we're joking of course but this episode is special as it is our first to be filmed in front of a live studio audience [Applause] that's right we're very excited to welcome some of our subscribers joining
00:01:02
us here today in our LA studio they're getting an up-close look at our studio as well as the absolutely thrilling world of a TV hosting segments and that's not sarcasm folks we've already had to pee a storm-off set and just minutes ago jarred flipped the craft services table you have craft services all joking aside we do have a lot to cover in our star citizen project update for March let's check in with Ricky Jetley and see what's been keeping the devs busy since our last update taken away Ricky hi and welcome to this one's update on progress for the persistent universe the UI visual team have been working on the visor vehicle status Holograms which have been extensively rewritten to utilize the new RTC technology developed by the render team previously vehicles displayed on the HUD
00:01:51
visor were constructed using the same hierarchy as a separate entity from the vehicle you are viewing with the introduction of RTT the vehicle scene and the visor or MFT is now actually a rear ender of the full vehicle this means that rather than being an imitation of the target it's now a true reflection of what's in Weld or space and accurately reflects all the real parts and attachments like the thrusters and weapons this was accomplished by building a list of the render nodes that were selectively rendered to a render target using a custom material shader we can apply the color tinting to visualize the damage status of the vehicle parts whilst avoiding some of the more expensive features of doing a full physically-based forward render this render target vehicle can then be rendered whenever
00:02:39
the player selects the own or target view on display this was hardwired to exist on the helmet visor in three Oh but in three one you'll be able to enable it on the MFD displays as well the visor and MFD displays all have their own render targets which the to DUI gets displayed on plus an optional camera that can show a vehicle to yes we have a render target objects rendering other render target objects it's all very inception and can be quite confusing working out who is rendering what sometimes we also use the same RTT technology when rendering the communication system often referred to its cons instead of selectively rendering the vehicle we actually rendered the full scene camera of the pilot to a render target so the player can see who they are communicating with this comms render target similar to the vehicle via status
00:03:28
Holograms can then be displayed on any given displays render target whether that is a vehicle's MFD screen the mobi glass or the visor one thing we're introducing for the 3-1 release is the ability to dictate the camera angle when viewing your vehicle or the focus vehicle this allows the player to view them from a series of fixed camera angles such as top bottom left side right side etc this is the addition to the live view which shows the vehicle's orientation from the pilots perspective another piece of new tech we added 43.0 and are improving 43.1 is how the vehicle displays it described initially we started with a fixed list of displays which were available per dashboard describing each of the displays available for example the Gladius pilot see has one heads-up display and six MFD
00:04:16
display screens we have now added the ability to connect additional auxilary displays we currently utilize this by connecting the visor display and the 3d radar display this feature will eventually allow the player to customize their cockpit by allowing them to add or remove auxiliary displays as they desire when the visor connects to the dashboard it plugs in has multiple additional screens for the UI to utilize meaning we can pipe views from the various controllers to these optional visor screens some examples would be the target vehicle hologram the energy view and the weapon to you the procedural planetary team have continued to make improvements to the planet tech we added a new way to support multi channels while improving the color blending and breakup all of which operate at the same runtime performance costs as the current
00:05:05
tech transitioning from the ground to orbit is now a much smoother experience thanks to the strides made in color blending we also separate the ecosystem procedural distribution setup from the rendering channels I setup which means a large number of these changes will be visible on yella in the 3-1 release finally we also improve the existing tools to reposition all planetary outposts automatically the environment team are finalizing the last few pieces of the truckstop exterior and are incorporating the new common element hangars with work on lighting polish and advertising to follow for interiors the procedural layer tool is now automatically generating lots of successful layouts for us overnight this is using only the base library of rooms available but we are already seeing some unusual and visually diverse layouts we
00:05:54
estimate we've crunched through around 1000 layouts so far during the initial period of development not all have been suitable for use but it's a good indicator of the amount of variety we can achieve using the tool we have a number of ships and vehicles approaching the flight ready phase and the VFX Department have been a busy adding effects to them including the anvil terrapin the misc razor and the tumbrill cyclone as well as the aegis reclaimer [Music] they've also been focusing on impact
00:07:43
reworks for weapons such as the Gemini r97 the PRA our distortion Skagen and scourge railgun during their recent sprint EU gameplay for each team one completed all their new tasks work for the PM AVMA and are now in the process of bug fixing and polishing for inclusions in 3.1 another big push for the next release was to replace the bespoke customization screen we currently have for star marine to bring it in line with the codebase already shared between arena commander and the loader apps in MOBA glass this will allow us to only update one single piece of code that is shared across all loadout customization screens it's a big saving for code as we only need to fix a bug once rather than multiple times this means that each interface for the players only need to learn a single
00:08:31
system so any user experience improvements will be felt on all loadout screens for the future the appearance is admittedly still very rough as we've been focusing our attention on making sure the code works correctly and that the functionality is not lost in fact switching to the new interface means their players can now mix and match light medium and heavy armor pieces individually with the weight of armor playing a more important role the stability for star marine players to customize various parts adds an extremely good level of strategy to the game the code work for this functionality is now in place so the UI team will begin updating the visuals and bringing the quality over the next week for 3/1 at the same time the UI team is also hard at work on wider visual improvements to the MOBA class up itself
00:09:19
along with prototyping the new layout we showed a few weeks ago an ATV we're also looking at ways to increase the visibility of MOBA glass in general one of the big improvements we're hoping to achieve is for the mobi glass to occupy more of the screen space in an effort to increase the legibility and clearly display the many visual elements finally the teams looking to bring up the quality of the apps by adding additional improvements to the rendering rules and shaders applied to the 3d models we should create a very good noticeable improvement to inventory management for three 1-1 the tech art team in LA has been continuing to work on the character customizer a new looping idle animation was chosen to provide some variation in body movement instead of a single deadpan facial animation a selection of
00:10:08
more expressive facial Alan's were implemented and randomized in mannequin to bring more life to the character model in addition all selected options now have work-in-progress icons with the technology needed to display preview icons for selectable items like skin tones and hair colors which will now be complete the final polished icons for each selection types are planned for the near future next the non head body parts had a specular highlight passed by the character team after the team decided that the heads needed a polish path to match the rest of the body parts as well as a line pass to add fidelity to the head detail another area of focus is a transition between the head and body textures previously a visible scene was noticeable between the head and the torso so the team worked on making the
00:10:56
transition much smoother thus removing the scene we have also had significant improvements to the camera system the field of view camera position and the depth of field parameters were adjusted to create a more dramatic presentation of the selection character in the environment and finally for this team various issues with persistence specifically saving and entering the pu have been addressed some bug's popped up while working on the character customizer like selecting items saving and then entering the pu other issues such as quickly skipping through selections or cancelling were also discovered and addressed teams in Austin Los Angeles have been working on the service beacon which marks the beginning of the player generated content although this is only the beginning 43.1 were
00:11:46
planning on allowing players to pay each other for services such as personal transport or combat assistance once a contract has accepted quantum travel markers will be created on the contract initiator so the service provider can easily get to them both parties will be able to see where the other person is while the contract is active either party involved in the contract can can anytime but be warned you'll be able to raid the other person where contract completion is not easily determined for instance when does combat systems end we are close to finishing our second sprint on this feature we're still trying to get the backend services for the system online and we've been running into a few issues where the system is not yet communicating properly with the diffusion services that keep us multi server and multi thread safe the
00:12:33
character team spent some time fixing a few bugs that were preventing helmet lights from operating as expected such as one example where lights persisted when you take off your helmet the team has also been working on an annoying bug where players were able to wear their hats and helmets at the same time that's it for this month's update for the work being done on the persistent universe as always thanks to our subscribers for sponsoring this and all of our shows and of course thank you to all our backers for continuing to support the development of both Star says and squadron 42 we'll see you next month thanks Ricky as you can see from ship hood improvements to planet tech - Moby glass enhancements so the devs remain busy with the push towards alpha 3.1 yep there are many features and upgrades like these planned for all four of our
00:13:21
quarterly releases throughout the year one item that will be particularly focused on now is performance optimization yes these ongoing improvements to frame rate will continue to be developed throughout the alpha phase and beyond let's hear more about the entire optimization process from the team that's been working on it in this week's feature optimization one of the big big big big things that we're trying to focus on this year the biggest problem is we have a wear space game right we have a lot and I mean a lot of entities too many to count and I'm sure someone will put a number somewhere the more entities you have sometimes they need to do some complex logic and they may have to update to add a timer check a timer do something to
00:14:12
continue the game going forward and if you have a lot of these it could be a very problematic thing if everyone has to update these every frame if there's 5,000 things that's five tiles and things someone has to do before the next frame could occur from a book ID few a video game as a visualization of an simulated virtual bird constructed out of a basic update loop consisting out of a few simple steps sample input external inputs I accept at the beginning of the frame so that game can react to those such input can be keyboard mouse controller input but other network data in case of a multiplayer game update words dead the virtual world has a state such a state is a multitude of things like the kind of objects which exist
00:15:00
it's a position their color player heads values and so on this state can change over time for example an egg and walk from point A to point B doing the steps that changes over time are computed the SD wendice in the last step during each frame so when knowing here it is data mined what is visible to allow the play I look into the simulated virtual world objects which are determined to be visible as and worn on the display we will focus on the update simulation and when the scene steps those are the performance critical ones without going into too much detail it can be said that each of those steps requires a fixed amount of operations mathematical or logical to update the word state and refresh the displayed image if all those operations can be done within one
00:15:47
sixtieth of a second as within 60 milliseconds we redo the rule update loop 60 times each second resulting in 60 fps frames per second but the source operation take 50 millisecond the game will refresh a scene only 20 times a second resulting in 20 FPS the real point of performance optimization is to find ways to reduce the fixed amount of operations to be able to execute them within the target firmware on the target hardware to better explain what performance optimisation consists of in modern games the previous mental model of the basic update loop must be extended by some more details by the upcoming explanation will still omit many parts it forgives a more accurate explanation of interaction of various hardware components like CPU and GPU to go back to the initial example a single frame looks like this
00:16:37
here we use an imaginary execution unit which does all the processing for the word update and to see my name the input sampling is omitted as it is generally doesn't affect performance but nowadays practically each computer has a chip specifically developed for rendering the GPU those two workers normally split between the CPU and GPU it is immediately clear that such a setup is not very resource efficient the GPU and CPU are running in lockstep and only one of the two chips actively doing computations at a time you can improve the situation by letting the GPU work on one film by the CPU update the simulation of the next film in parallel this pattern is called pipelining and appears everywhere and working with computers for example and watching online videos the next few frames are
00:17:26
downloaded by the already downloaded films a processed and displayed in parallel but pipelining doesn't come without implications and the CPU produces the data faster than the GPU can process it the GPU will act more and more behind the flaking behind will be perceived as additional latency on top we need to store the CPU computed state some main memory by the GPUs and relative processor and there's no device as infinite memory we need to stop the CPU at some point and love the GPU to catch up as we cannot store more state forever here we speak of CPU being blocked waiting for the GPU a starter doesn't be generally up for better latency in only love's GPU to lag behind up to 1 the same problem can happen the different valve on if the consumer is processing the data faster doesn't the
00:18:14
producer which is called starvation in general one component will always be slower than all other components which implies that the game will always be bound by something here CPU or GPU does the first goal of performance optimization is to identify which components perform to spawn if the framerate won't be affected if you optimize GPU performance by the GPU as already started this approach is also called critical path analysis as it helps to identify the fast which terminates performance a similar complex situation emerged on multi sweating CPUs the HDP you know is a multiplex acute in units which we refer to as hard passwords to cope versus strategies you need to be found to distribute separations over the different Hardware sweats the obvious choice is to use the same pipe any approach as with GPUs but
00:19:04
that has two large 12x and CPUs work balancing a Shawn and GPU CPU examples the system will always be bound by the slowest component to parented CPU work we would need to find chunks of work which alert chico together and always takes the same time this in itself was very hard to impossible in a dynamic game and with the increasing number of hardness wet CPU cores we would soon run out of systems to put beside each other it would either need a specific set up for each different CPU core count residing in a maintains nightmare which is why our chosen approach is to batch the privations and to utilize inherent parallelization found in video games for example there's fairly only once in your character to update so we can put each different character onto a different CPU core the theme happiness was any kind of
00:19:54
objects which we have this pattern allows us to scale more flexible with the CPU cores the process to change all the game logic is ongoing and we are moving more and more parts of the game logic over to this approach to atlast more CPU cores more frequently and improve our performance in port Latin see the roots 30 with 60fps discussion in the context of games it's not only about our curriculum refreshes and it visual quality this other about input latency input latency is normally measured in firms if each firm needs a fixed amount of operations regardless of the underlying hardware so if we need three frames from user input say we display an effect the input will have an input latency of three times 60 milliseconds resulting in their 48
00:20:41
milliseconds even running with 60fps if our games run to 30 FPS which means a firm duration of Seleucia milliseconds it will have an input latency of 99 milliseconds nearly doubling the time between input and a visual reaction without being less fluent impression of the game adding more part banding here would only increase the perceived latency which is why we try to not depend on pipelining for performance on the CPU even if it would result in a better frame rate if it will cost us a verse and not really acceptable input like one of the questions we get asked quite a lot is with our server having a target frame rate of 30 frames a second and clients having their target frame rate of 60 frames a second how's that possible
00:21:28
why aren't they tied together and the reason for that is that we're not operating in something called lockstep and lockstep would be where the server sends an update and the client has to wait until it receives an update before it goes okay this is where they sense the should be and then has to wait for another update and goes right ends these moves to here and so on what we actually do is the several sanity's updates 30 frames a second the clients running at 60 frames a second in theory the client should receive an update for entities every two frames so it has to kind of guess what's happened in the middle frame so basically what it's doing is its locally simulating all the entities so an entity moving across the flat plane fixed speed and a fixed direction
00:22:16
it can guess where the entity should be at each point and it'll get the yet the next update from the server and go mm-hmm that's slightly out from where it should be uh needs a nudge that one a little bit over to the left a bit layer down on all the entities turned a bit so I need some logic a little bit over the other way so that allows that the client the service have different frame rates and we don't have to worry about doing something like operating in lockstep or any other language which can cause big issues because of latency on the set on the internet and how that varies quite randomly all the time it would be quite a bad experience it's alright for things like ten based games and but something like you know a live-action game like we're trying to David it's just not
00:23:04
going to work reducing the amount of operations can be achieved in multiple ways which one is used mostly depending on the current situation time to implement it and so on most often multiple approaches are as a combined detecting objects too far away to absorb as a player here we need to decide after which distance you want to disable objects what to do it's a gray area of an object's are barely visible how to make it look like objects moved by the player wasn't looking at it in various other corner cases let us assume a system where we need to keep track of 1000 objects in a single list if we delete such an object you must remove it from our list the straightforward approach is to check each entry in our list for our object and we found it we will move the object afterwards we copy each following object in the slot before
00:23:53
it to fill up the hole from removing the object this will always require 1000 operations such an approach can work well for a small list but if the list is large or if you remove objects very frequently it will take a good chunk of our precious operation budget just we need a smarter solution inside each object we will remember the position inside our list let me now remove an object we immediately know it's position then we can take the last element in the list place it into the hole and update its reference to this moved object missus we're down to four operations from 1000 which is a nice performance game but the other change to behavior if you know we are the objects in our list such ririn can be okay or not depending on how the objects in the list are used this was only one example of an optimization of a
00:24:42
very small part of Cebu game in general the wool optimization processes iteratively finding where the game is bound CPU GPU of each CPU sweat batch and come up with a better solution to use less operations for the most expensive sections there are many moving pieces to this puzzle generally the tech content or tech art people will be focused on the GPU side of the thing the the number of draw calls we have about a budget of about 2500 draw calls for all ships on screen at one time and a ship can be anywhere between 500 draw calls to 800 draw calls at its highest resolution so we try to optimize that as best we can we have a few tricks in our
00:25:33
bag to do that one of the things is LOD s we have artists creating a level of detail meshes which reduce the number of polygons and reduce the materials as you go into the distance these are being hand done and also generated by our simplygon tool we're also reducing the damage we're doing or reworking some damage on old legacy model ships say that like the Mustang when you blow up a ship you have all these little pieces of debris that used to be hand modeled now we actually just break the ship hull in half and and change the shader so you see some little called UV to damage this reduces the number of debris items and therefore reduces the number of draw calls quite a bit a draw calls are liked
00:26:21
and material passes on a mesh so we're really one of the other ways we can reduce draw calls is by combining meshes together and what as a process called skinning and as I work on landing gear I can take many pieces of the landing gear Pistons and gears and and the feet and combine them into one mesh and I drive that all through a skeleton so instead of 10 pieces with 10 draw calls each you get one mesh with 10 draw calls so really good significant savings there another trick that we do is call with our viz areas and portal culling so when we're on the outside of the ship we don't want to draw the interior and even when you're another room in the ship we
00:27:09
don't want to draw the next room you can think of portals as as like as I get door when it's open it draws the inside of the room when it's closed it calls the room in the geometry out we use our visitor portals to connect to the doors which animate open and close and then drive the portal to turn on and off and like say the caterpillar those doors are skinned so they're optimized and they have portals on them you will have noticed there are some issues with the portals recently and we are working on about four or five major portal issues but we will get them and it will be you know a faster and cleaner and you can walk outside again and then see the exterior in space by standing on your
00:27:59
deck another thing that's being worked on right now is called signed distance fields and this is a technology that our Chris Rayne is working on it's a different way of describing a volume by recording these distances so it's stores distances and what we've done to this point is use a volumetric approach where we have these box holes and we fill the ships with voxels and this gives us our local physics grid but when we move to a signed distance field it'll be much faster we can describe shapes much more a much higher resolution and we'll know more immediately if you're inside the ship or outside the ship colliding with
00:28:47
the ship or not this is gonna open the door for new shield tech a new visual on the shields so instead of a bubble that's kind of you know loosely shaped around the ship you'll see a very tight shield that that pushes just out from the plates of the ship and also one of the things we're excited about is of course when we break the ship's in half the sighing distance feels gonna is like a first step in getting us our multi physics grits so we'll be able to keep the interiors around and then allow you to reclaim and walk around in the end the ships after a battle so you'll find new items and things that way Network code was a big performance bottleneck on the server and the reason being the that
00:29:37
24 clients that we were supporting and PU at the time to gather all the information from the simulation had to write it out 24 times to each different Channel and the way it would do this is if you think about different threads each CPU doing a different amount of work you kind of have the the main thread running along this way coming right there's a frame there's a frame and each point in the frame towards the end of the frame won't go right now that Wako go off do you think right some store folks and that work go it on the phone go okay right and start doing stuff and as long as it is finished before the main thread started the next frame everything was fine but if the network code is running
00:30:25
a bit slow then it would stall the next frame on the main thread it causes that some wait until the network started finished by the time 3.0 came out we kind of well came out with quite a few the network changes that we've been planning so without the introduction of serialize variables that's been him for quite a while by the time 3.0 came out pretty much all of the game code had been conveyed from using the old aspects and our mind systems to the replacements which is serialize variables and remote methods and what that meant for us is that we could focus our efforts on optimizing one particular code path and we just leave the legacy system not performing
00:31:15
as well as we'd like but because not so much stuff who's using it it's not really an issue the big kind of difference we made the serialize variables is that the fat from the game saying okay what's the state of everything but the things that have changed and we need to update watch the current state because I need to send that across the network that was paralyzed so instead of the network thread doing all the work at a particular sing point with the main thread it would just go okay how many threads have we got available okay all of you go and group grab some data and send out some that way and then the next part would be okay now we need to take that data and we need to send it off to each of the different channels so that they know that they need to do some work and the channel being a the network codes representation of a climb so then
00:32:04
that was paralyzed as well and then the final part before our lives was the okay now we've actually all the channels we've got the day that they need to send we need to actually go and send it so that was paralyzed as well so we went from one thread doing all the work just splitting it over as many threads as we had available so on the clients we've kind of got the opposite situation where they're receiving all these updates on the server it'll come in through the socket they'll get processed by the network thread and then there'll be a sync point with the the main thread it'll go okay right no it's ready I mean photos ready here's all your data this is what's being received from the server right go off and do you next you know be a simulation that part has always been pretty pretty
00:32:54
quick to be honest and Hong Kong clients were talking about roughly a millisecond to process all that they had even though it takes proportionally quite a lot longer to write the day erect and package it all up on the server the unpacking of it's out always been quite quick so the network code itself has never been a performance issue and for clients because they're mostly just receiving data from the server and what they actually sent is so much smaller than what the server sends that it's never really been an issue for them at all so buying calling supposed to address this say this mismatch between client and server the share process and power that you have on servers and their ability to
00:33:44
process everything and the much smaller processing power than most clients will have and the idea behind buying Colin is that a clients only really interested in what's immediately around it anything that's on a different planet or a couple of hundred kilometers away and even out in spaces no no way you could reasonably say it just shouldn't be there and by that I literally mean that your client should not know about it at all that would save all the processing of those entities on the client it would save a bit of processing work on the server because it doesn't have to communicate information about those entities to your client so it's definitely something we want to do and it's just it's not
00:34:36
particularly easy thing to do you need a way of controlling the updates on the server so that it's only updating not all the entities it knows about but it's only updating the ones that it knows are in range of any player at all so the ones that no player can see just doing little date and it saves processing time on the server and we've had to change assumptions on the networking code side where previously it would say okay I've got an entity it's a network density every client needs to know about this and be assumptions all the way through the network code that relied on that not being true so we've got to change all
00:35:23
those and so it's a bit by bit process that we've kind of unraveled stuff and add the new stuff and optimize stuff that will allow Baron Collins to be possible and we're almost there that and the steps that are left are really if you imagine a situation that you've got buying calling in and entities that aren't immediately around your clients don't exist what happens when you fly to a different location so you quantum travel to a different planet you need all those entities to be there so they have to be spawned so say I was good to tell you about those entities information comes over the network and then your client has to go and create them up until now all the entities all the entities spawning as being synchronous or called blocking which means that the point in the code
00:36:13
where it says why I need this entity it'll basically stop whatever it's doing it'll go and load the data in for that entity create the entity itself register it with physics and whatever else it needs to do and then go okay this function can exit let's carry on and do the rest of the game so we need a synchronous fantasy spawn which is also known as object container streaming the idea being that a client will get a message from server and go okay I need this entity he'll go off and go okay well what files does a sense they need to load what data doesn't need to load so I'll go off and do that on the background thread okay I finish loading that okay now I can create the entity and it's not stalled the processing of
00:37:02
the game it's not stalled your clients at all while that's going on you can continue to play so obviously that takes a bit longer so the trick is to try and make sure that you finish spawning entities by the time they actually appear so you've got to kind of preemptively decide to sport and stuff so that we need this object container streaming that's that's the technology that's gonna allow us to do the streamed events these in in order to be able to make fine calling work so what I really want to do is develop the by uncle in technology finish it off as early as possible so the in-house we can be testing it and making sure that it works finding all the problems that it's gonna cause perhaps an example of that would be say you've got a mission from a mission giver and miles a car and you
00:37:56
need to forego on five miles four miles of sat in this bar in Levski and you're on port olisar so we need a little marker rendering on your client to show where miles is so you know where its head but with barn calling the miles I heart and see won't be there so what to be draw the marker on so they'll kind of problems we need to try and figure out ways to deal with that there's a couple of different ways we could do with that situation but until we do we can't really put bond calling in because it could break the gameplay probably going to see the full implementation of buying calling in until so we get object containing a stream in it and put frozen internally we need to get it as soon as possible so that we can we can move on with some of
00:38:43
the other stuff that were planning so what we've kind of done which is another thing so I've mentioned buying colon of mentioned object containing extreme then there's also serialized variable culling and which is because we can't do barn calling just yet because gone get out into the hands of the backers and serialize variable culling is what we can do so the way that works is same sort of distance checks that we're doing before and if an entity is too far away clients the server will just say you know what I'm not actually going to send you any updates for that entity and what that will mean is that because the entities not receiving any updates on
00:39:31
the client the client can go okay well I can put this entity to sleep I don't need to do any CPU processing on it you can just stay where it is so you can get a lot of the performance gain that we'd hope to see from behind calling from this serialize variable to the size of our games are generally not very optimized during development it's very clear that the faster version is more complex and as more state even the small example and more complex code means more places where something can go wrong resulting in a bug because offices you always have to decide on the trade-off between production speed and performance if we focus everything on performance feature development will be slowed down a lot including future features as I need to be develop against more complex code base to make
00:40:19
it even worse we could optimize all this code just to later realize that the specific part was not really needed or we need to implement something else entirely as a feature was simply not fun there's a famous quote premature optimization is a root of all evil in other words we need the rules final system to correctly understand and analyze it to make meaningful performance optimizations but so much for the theory the critic reality our game can already be played and playing with 15fps it's no fun to help the code leads and production to decide which features optimization should be focused on we perform a performance analyzation sometimes with features need optimizations to work our features needed to implement an optimization resulting in dependencies between developers making the group progress
00:41:07
more complex other times code changes are trickier than expected we're going to do too much risk shortly before release reciting the new trickle-down effect of delays for other optimizations or features we use a set of different tools for performance analyzation the very basic tool we are using with a classic sampling profiler such a profiler works by stopping the CPU at a fixed time interval and recording what the CPU was doing at that time if a function needs a lot of operations to live a higher chance to be executed when the CPU is sampled resulting in a higher sample count which indicates a high cost function won't work of assembling profiler so it only shows a function where the time spent but no context of either function was executed on the CPU in relation to what other code or high-level state of the
00:41:56
game like the number of active vehicles for this we use mostly use an instrumented profiler which shows executed function antelope when Alera t but shows what CPU cores executing when this profiler can either show us plots of certain performance values we are interested in those truths are very useful to analyze a specific situation but star citizen is a different beast we have up to 50 players and many more planned on a single shirt server which as execute the physics simulation this means what any player is doing can affect other players is about making sure as law caused me to run my operations which can be observed indirectly as jumping movements on the clients Weber bending and so on all effects of the server not sending updates frequently enough when they need to replicate the
00:42:44
server behavior and we also been looking at some of the telemetry frameworks that we use to actually help us visualize and track what the actual performance issues within the game inside the codebase actually are some of the improvements we've done to the telemetry for example allow us to capture data automatically during our play tests the day it would automatically be captured when the framerate drops below a certain threshold so for example we can use this kind of auto capturing to make sure we grab moments where we see these nasty
00:43:37
bugs occurring which cause big performance drops we would then take this data onboard analyze it as a team and then look to see where the problems were lying the various problems could then be either addressed by some of the engineers on the performance team or in certain cases we might actually find that the ownership of the the issues in question would will be from engineers on other teams so in that case we'd have to coordinate with them to look at fixing off certain issues that that might have cropped up in the play tests it's currently quite early stages for the auto capturing and with the new telemetry system but the initial tests
00:44:26
seem quite positive so we're hoping for free one that we see in a lot of these nasty issues a lot sooner and we're able to fix them up in plenty of time for the actual release some of the other things that we've looked at adding to trap the perform are some auto tests the auto tests would be run by a test build machine once per internal bill but we do these these internal tests would be something like having a automated player walk around a busy area the games such as Levski or it could be have the player spawn in into a ship alongside 30 AI ships and these
00:45:20
auto tests would track various framerate statistics allowing us to see as we go from build build is our performance improving have we accidentally pushed the performance back a bit in which case these tests allow us to catch issues as they arise and whilst they don't necessarily give a perfect measure of the performance that we're likely to see in the live environment they can at least give us a good indication that we're moving in the right direction rather than us putting in a lot of optimization code changes and not actually knowing whether the changes having the desired effect so for three zero one there's really two good
00:46:12
ways at the moment for us to gather data from which we can decide what needs optimizing code wise two ways that we have at the moment are taking the auto captured data from the internal play tests we do with up to 50 internal QA and a handful of engineer's participating and the other way we would look to gather data to analyze to see what we can optimize is just by doing local tests around machines when we run those we can spin up a server connecting a client we can potentially can in headless clients should we want we
00:47:00
can play the game fairly normally but with the additional their debug commands that are available to us we can do things like like jump outside olisar instantly put ourselves in a ship we can spawn 50a eyes and so we quite quickly can put ourselves in it in a scenario that in the real releases would see the framerate stressed more than just running around as a single player in all us are playing on the wise empty server with those different situations I'd also tend to do things like spawn 50a I then move the player away to another location in the game that's far
00:47:51
far away from olisar because then in that situation we'd sort of expect really that the performance should should be better because there's now a bunch of vehicles on the server but those vehicles are now so far away from the players vehicle that we shouldn't need to update them on the client with the same level of detail that we previously did when we were sat right next to him in and olisar so we'd be running these various different scenarios locally taking a look at a data and by analyzing it we to look out for instances like that where we could get some clear gains by not updating things that didn't need to be
00:48:41
updated we could also take that data and look into things like what are the most expensive functions in the game what what are the functions that are being called most frequently which functions take the longest to run which functions do a lot of allocation and de-allocation of memory we could also look at functions that were coming from code that we didn't actually necessarily even need to run for various game modes or code that had been there since we inherited the engine so we would kind of look in it looking at ways to improve pre-existing code that we did need but
00:49:33
also often some of the biggest wins will be actually removing or refactoring older systems that we didn't need or didn't need to use in the ways that we were currently using them for PE for example one of the issues that we've we've always had is being able to to test with tester savable and a full load of clients so particularly now we've gone to 50 players per P Yuma and you can imagine the amount of manpower it takes and you know the logistical nightmare of trying to organize 50 people in QA or you know volunteer developers to try and play test the game for a couple of hours and fill the service and get back performance data and then it's it's equally it's if you
00:50:25
ask you know so you observe a slow down while you doing this test if you want to ask someone what happened you have to ask 50 people and you get 50 different answers does this kind of yeah it's quite difficult to sell the tests and again the quality of information back that you require so we're working on and these things we've got called headless clients which are basically clients with our players little they're all dumb robots still press the keys in random ways there's no key Bo's just a program that runs by its virtual keyboard it's just mashing the joystick and mushing the keyboard wiggling the mouse and it's smart enough that it can get out of bed and get into a ship and then just fly around and firing in random directions and that generates enough load that we
00:51:15
can kind of simulate roughly what we can expect a player might do and previously the headless clients have only been gone on development developers machines and to get enough headless clients to help felt server they're being migrated on to virtual machines running in the cloud so they have to be spun up on demand and shut down the other thing I mentioned was getting back a better quality of information so we've had telemetry in the game servers for a while now we've had some telemetry from clients as well it's been a bit of a fragmented approach into how that's being done so we're working with the engine team and DevOps again to try and get a more unified approach get richer information about what the performance is where the
00:52:06
bottlenecks are on each of the clients under server try and get more contextual information about what everyone was doing there each point in the game so we could go okay those performance drops here what was everyone doing all right that guy catapillar blow it off lots of cargo okay that's wrong really yeah with a traditional game you probably tend to find that the performance for the majority of the lifecycle the development would be actually pretty low probably fifteen frames a second or less whereas the issue we have is a early access title if you like is that we've we've got to get our features in write
00:52:54
our new code write our new systems but yeah we have to get those systems performing in an optimal manner that allows people to play the game at a reasonable frame rate and enjoy it rather than having the luxury of only testing the game internally for the majority or the development cycle and then knowing exactly what content we want in the game knowing what works and then being able to optimize that in the last few months of the project so really we do have an extra overhead of sort of continual perform optimizations that we need to look at but that's all part of the great
00:53:42
challenge that we've got here right in the code foot and what we all see is a massive groundbreaking title Joran development there's never going to be a line that we're in the cross and performance is something going to be fantastic it's always going to be this incremental process of getting better and occasion we're gonna get worse so basically what's what's gonna happen is as new features come online the first thing they're probably going to do besides work fingers crossed is make performance a bit worse and I'm long as we've measured the impact we're going to try and optimize it back up again to the levels that everybody's kind of happy with but development being the key thing it's never going to get to
00:54:29
that kind of 60 frames per second for everybody because it's more important that we spend the time on actually developing the game getting everything in getting everything working so the kind of optimizations that we're dealing with now you know there's a few small kind of micro optimizations making things go particularly fast but it's kind of system level optimizations so yeah we know we're gonna need buy and call and we know we're gonna need server mashing we know we're gonna need the the entity component update scheduler well you know we're gonna need object container streamit so these kind of system level things these pieces have to go in so that we can build the rest of the game but that's the primary focus important to me it's it's to give us an attack to build the rest of the game yeah I guess the biggest takeaway out of this is there's never one answer an optimization there's billions I mean
00:55:17
that's why we have those profile markers in there we need to see what's going on and work towards generating a solution to beef up that part of the thread that's going slow maybe we were to turn off some updates in the you know that are out of range used in sterilized coaling or maybe we have to optimize the way that weird handling physics on the planet because this particular way takes a little bit longer it's analyzing the data figuring out where 'no thread that we or having issues in and then finding a solution that works for that and doesn't break the game that's probably the biggest takeaway in the biggest challenge optimisation affects everyone from devs to players we all consider performance implications as we develop
00:56:04
star citizen and we even have an optimization strike team to keep up with our own development that's right and with 3.1 around the corner we have a few ships coming online the reclaimer the cyclone the Terrapin and the razor and on the concept front the AG is Vulcan remains available through the end of the month add one of these versatile support ships to your fleet and become a beacon of hope for distressed pilots everywhere we also have a ship package that features the Vulcan along with some other ages vessels and a standalone skin pack available to everyone that's right so even those of you waiting to pick up a Vulcan in-game can secure these eye-catching paint jobs now make sure you tune in to this week's new episode of calling all devs and lore makers Guide to the galaxy up on the site now and on this week's reverse the verse Clive Johnson and Rob Johnson two sides of the performance optimization coin
00:56:52
will be stopping by to answer your question following today's show that airs live tomorrow at 8 a.m. PST a huge thanks to all of our subscribers for sponsoring all of our shows and a special thanks to the subscribers live with us in the studio thanks for being part of this very special episode and thanks to everyone for tuning in thanks also of course to all of our backers you make the development of star citizen and squadron 42 possible that's all for now until next week we'll see you around you
00:58:07
thanks for watching for the latest and greatest in star citizen squadron 42 you can subscribe to our Channel or you can check out some of the other shows and you can also head to our website at

Transcript from YouTube’s automatic captions — punctuation and Star Citizen jargon are approximate. The video itself is hosted by CIG on the official channel.