Star Citizen Live: Builds, Platforms and Publishes

18 September 202000:55:201,652 linesWatch on YouTube ↗

Transcript — every row opens the video at that moment

00:00:02
hi everybody uh welcome to another episode of star citizen live builds publishes and platforms or what's it platforms and publishers who knows i'm your host jared huckabee and joining us on the show this week are a round table of incredibly technical director type folks who are involved in making bringing all the aspects of star citizen that all of our wonderful developers create over the course of the days and the months and weeks and bring them together into builds and publishes and and make them accessible to you the players so we are stepping back behind the scenes here and we are joined by well a couple people let's just go around the room and uh see who we're talking to miles i think it's been the you've been the longest since you've been on one of our shows we're going to start with you first uh who are you what do you do for star citizen
00:00:50
i'm the director of build operations i manage the team which uh deals with the version control mostly and build system creating the builds that we deliver out to the players okay uh ahmed we'll go to you next who are you and what do you do for star uh am ahmad checker i've been with cig for five years and a half my team and i were publishing technology we received the build once it comes out from the build system and we try to deliver it to every single player whether internally externally leveraging a lot of cloud technologies automation and some good old-school also system management and system engineering about how to keep all these moving parts together to deliver like a good player experience at the same time to harvest as much debug information and
00:01:39
because we're in active development so it's kind of very unique kind of role we're not just serving a live product we're also yeah yeah all right uh and then there's benoit so on the show today benoit who are you what do you do uh you need to say my last name first [Laughter] bo ah so good actually you i know you did cool um uh i'm ben i'm the cto at turbulent uh so i manage and oversee all of the teams that make platform services and so that includes website launcher and a couple of in-game
00:02:27
services as well so so yeah so that means taking miles is built that goes through ahmed systems and uh getting that into your heads not just so i imagine the three of you uh work pretty closely throughout throughout the the month week day i don't know well yeah how many bills a day i'm at sometimes a lot a lot i mean that's that's the good thing we we we deploy a lot of builds behind the scenes that we get to decide what they can make here or not so not all the builds that we deploy you will get to see right sometimes there's so many yeah yeah not every build even makes it to the point where it can deploy so yeah some of the some of the builds are uh yeah so let's talk about that before we
00:03:14
get into the question and answer period uh which is you know what most of the show is this is a bit of a uh esoteric show it's different than we normally do um most most of the time uh when people look at the stuff that you know other games put out you know they're buying this and stuff we're never talking about build systems and publishing and platform stuff so so we're in an area that that a lot of folks may not be familiar with so let's talk about uh how a how a build becomes a law that was school ass rock nevermind uh let's begin by to do an overview how does the the build and publish process work so i mean it obviously starts with the developer committing a change once that goes through that process we get uh we have tools which allow the
00:04:03
developers to kick builds from our release streams uh once the build is finished we go through a manifest process and we create what we call the build manifest and basically it's just the list of files that dictates what the launcher should patch down and we upload that and kind of hand off the comment from there yeah that's that's where the custody kind of switches from a team to another but also we do not really really have silos like all these teams actually work together for so many years so it can be that we can actually be following up on the building and build system and trying to inspire what could be wrong and you could be also looking at deployment and see what could have gone wrong so from from that moment that we have a build we have a manifest it depends on the size of the deployment every deployment is going to go to in a single region a small number of audience
00:04:51
a key or a staging deployment this is fully automated you go through a web interface you click deploy everything gets created from scratch share nothing like vms get created a kubernetes deployment for a new generation of services name space gets created a helm deployment chart gets initiated that that chart as a global chart has other charts underneath it a lot of movement pieces so many movements gets deployed so you can have that fleet of services and the simulators that indicate the game server is deployed in the cloud accessible whether it's an internal group of people or xml if it's external we so far due to how rapid things are changing we'll have to have a captain to have someone who's really running the deployment so every deployment in that case is a get branch you go take our automation repo and nikki a branch and that branch would
00:05:39
have some of the facts that are specific to this deployment these facts are always changing we're allowing very last-minute change because we are a development company we are really developing as we go so now we're trying to take away any any ability to to to change things very fast so that deployment gets created it's a blue green deployment and all the resources get allocated around the world and it's hidden at this time our internal qa start hitting it and depending depending every deployment has a goal too like every deployment has a very specific goal whether we're delivering a feature we're testing a feature we're collecting feedback and kiwi hits that has that part hard and there is a another checklist with other departments everyone has to know this is a new gamer experience where we are exporting or publishing a full gamer experience it's not a bunch of
00:06:25
virtual machines just being able in the cloud and everyone has to get an agreement and all the work gets done and then the players see the light comes on on the launcher and i think that's that's where you get to see more i mean platform is involved in all of that like all these systems are super super involved that platform you get to see more like okay we're ready to publish right now and i think benoit can explain that better than me yeah so that's where uh usually my team comes in where uh we've built tools to allow the uh the deployment to go out so that includes securing the build uh from you know uh who needs to be to have access to it as ahmed was pointing out there's a lot of builds that players don't see that are used for internal testing that are used for you know queueing a specific feature or doing ad hoc testing on on a specific refactor or something like that so
00:07:14
that is internal but the moment it needs to go external so either it goes on ptu or on production uh that's where the launcher release tools come in and so this allows us to take the build manifest that miles was speaking about and basically make it available to players to patch and so that's where the delta patching technology comes in that's able to do rapidly look at the build that you have and the build that's now available and make compute the difference and then just give you the objects that you need to run the latest version of the game obviously for that there's a lot of user facing services and every time there's a user facing service that means a lot of responsibility related to security scalability etc etc so that's mostly the platform teams that take care of of the maintenance of those systems but
00:08:01
the operation of it is fully in the teams and so that's the great part so that's it's pretty much the build process in a nutshell i think in a nutshell that's the that's the the the nickel version the knuckle version of the process at the very least uh you know we only got an hour long show here sure so we do so uh as usual uh we're taking questions live from the chat if you have a question that's related to the build publish or platform process you can submit it in the chat these are all things that exist external of the game so as a general rule of thumb if your question has anything to do with what's inside the game this is not the show for that so those questions won't be asked additionally we also collected questions throughout the week as we usually do
00:08:48
from spectrum we pulled out the ones that were appropriate to our guests here so let's start in with those while some of the live ones come in uh miles how does the process of creating and maintaining a build from a dev branch work uh for example how is specific functionality selected from the dev branch put into a build such as 310 while others are left behind in like game dev and then once it's created how are these changes maintained between each build sure so we actually maintain dozens of dev streams where individual features live and are housed and as those features get completed they're they're merged up into the more collective streams that get tested by qa as well so we can kind of see how they play together before creating the
00:09:36
release branch eventually um usually once everything's ready for like a 310 or 311 we'll branch that out and start cutting builds from there we have a pretty robust inclusions process that allows us to strip it down and make sure things aren't going out that shouldn't be yet and then as far as i think i already touched on that we kind of have a nice user interface that developers anybody production can go and request a build so production a lot of times is the ones responsible for kicking the release builds and maintaining that pipeline of who does what when the builds where um as far as um how to get selected things so obviously bug fixes go in our release streams we have a bunch of internal automated tools
00:10:24
um that as soon as changes are submitted to those streams we start bringing those changes back down to the lower level developers and distributing them out so that everybody has access to those changes as quick as possible um yeah i think it's i think it's one of those things that maybe isn't always apparent that you know everybody's working in these individual feature streams you know so that everybody so that the active feature team can work in their stream and the ship team can work in their stream the ship art team can work in there you know the featured team is this one our team is this one and they're all working in these different streams so they don't step on each other's you know toes and overwrite each other's work while they're working and then that process of integration is where where bugs are born it's huge um we actually manage quite a few micro services that kind of back the
00:11:12
whole build process and uh that's probably the most heavily used microservices that integration process at the moment it it's thousands a day easily yeah that's uh the the the number you know you know every every check-in generates a number when the and the builds that go live the people often see like we call it patch 311 but then there's that you know secondary number the the change log number you know that is how many different uh check-ins there have been each one and then when people people who follow evocati or follow pt or whatever when they see that number rise like that is how many have made that particular bill not enough check-ins are done just the ones that actually were included in that particular build and put in it's a it's a staggering number
00:12:00
right it does do a couple traffic jams on those integration days for sure yes absolutely um then what what database technology are we using and why uh are we using like what sql nosql what are we using well so the the the answer is complicated but we do use a plethora of databases uh so uh platform systems tend to rely on on structured data and so they're gonna leverage technologies like mysql or mariadb which is the same kind of engine but we since a lot of the game systems are also uh made by micro services then they can each service can actually use its own
00:12:50
data store and so we have the ability to use the right data store for the right use so we don't have it's not like nosql for everything or sql for everything right it's more like this use case needs a structured data set and so we're going to use this engine right so in general uh the technologies that are in use in the game today and in platform both both of them share a lot of these technologies but uh mysql is is used to persist a lot of the data on both ends there's a mongodb is also a big player in a lot of the dynamic variable systems that power the game and redis is a as a real workhole workhorse for a lot of the really real-time stuff like you know spectrum red counts or distributed logs we're going to use redis for a lot of this which is it's really our our swiss army knife of
00:13:38
databases there so the right database for the right job correct correct anything using like filemaker pro no no uh no though though some people want those features i was just about to say we advocated this a week ago we had it but we had to duplicate it yeah um ahmed uh what type of aws servers uh do the platform and pu currently run on okay yeah what can you tell us yeah yeah i mean we we can tell everything because it's public offering everyone knows about it we use a large large large variety of vm sizes and families but the major common factor between all of them is the aws nitro which is a kvm lightweight
00:14:28
technology that aws introduced with the next generation of new generation and when it comes to the game servers the game servers run in c5 if we have some backers watching the live stream from the days before 3.0 we were doing some comparisons between the c4 and the c5 these were changes in the hypervisor changes in the in the cpu powering the vm underneath when it comes to the cloud you're usually trying to to hit a very specific sensitive part of your workload so for the game servers we have to meet in two main areas we need to maintain a decent ratio between the memory and the compute and we need to make sure that we have very fast compute so c5s fit that very well we have other services that are single threaded that we need a very very fast single core clock speed so you would run on z1d also another nitro
00:15:16
instance we have other services that require moderate cpu clock speed but they have larger memory usage you'll be running an r5 so it it's it's all in our availability actually uh a lot of our workload we try not to be stuck to one single size because we need to make sure if there is any issues and we cannot allocate vms from the size we always have a backup size we have backup regions behind our regions but sometimes you have to choose your battle whether you're willing to go to a second region to allocate resources or you're willing to go for the second most optimal machine size i hope this is not so boring but we use a lot of them and we try to use the new ones i think that's interesting the thing that's interesting about that is that you realize that sometimes you try to get a vm from the cloud and you can't because they're out of vms and so yeah
00:16:04
and you end up playing the world's most advanced game of whack-a-mole i mean it's what keeps the job like interesting and entertaining it's it just it just got the image what is it there that the the ahmed was there was spinning plates um a follow-up here uh are our game servers dynamically deployed depending on users the user requesting to play and can game servers dynamically scale by load yes we are very proud of our road scaling system we have a buffer based dodge scaling system game servers are a slot based so if the game instance will take specific number of slots you cannot overload that so when you auto scale a system like that you are bound to use what we call a buffer system where you always have a
00:16:51
step ahead of you of vms or instances both in the logical and the like in the physical the already server players and that buffer that buffer size how much you maintain ahead of you has to do with how fast it takes an innocence from scratch to load the map and be ready for players so these are adjusted this is what scaling system works on the regional level so the guys in sydney could be it could be a rush hour for them so we start spawning vms over there the us and the you are not affected and yeah it's it's uh it's a very essential system for us because game game and behavior like gamers behavior is always up and down and we cannot have resources either around and these these are quite large vm so we cannot just have them hanging out for no use not with that attitude all right let's see what we got from the
00:17:43
chat here uh miles what's changed hardware wise uh since the last time uh we we talked about this uh uh and what effect has it had on build times and publishes sure uh well i think it's been about four years since we last talked about this topic so um we've pretty much upgraded across the board be that storage ramps adding more cpus um we've also spent a lot of time utilizing all the hardware a lot better with better build distribution across the machines and a lot of that has helped greatly over the last four years so we used to at least six hours per build back then we could only have about two running at any given time um nowadays we're sitting at about three to
00:18:33
four we can have up to four builds running simultaneous simultaneous simultaneously and um we probably put out 12 to 15 builds a day now um it's pretty regularly especially with all the feature stream support that we've added over the last year or two um the builds are getting kicked very regularly and it's almost a constant queue and stream of builds coming through where's that phrase come from kicking builds like what what when did builds become a thing that were kicked well i mean it's like kicking a can down the road right you just keep it going until it gets through the trash that's a very appropriate analogy that's probably it exactly miles um is merging feature branches difficult um it can be
00:19:24
um fortunately for me the developers typically own a lot of the processes because nobody can solve their resolution conflicts better than they can um there is a somewhat heavy build process to that where um we always try to test the changes upon merging and make sure that they're not going to break anything in the stream that they're going to so that that is time consuming i'd say it was a better better answer yeah you you mentioned earlier that that developers have the ability to kick off a build is is is that a is that a good thing i go back and forth on that every day i think typically it's a good thing the less barriers that we can put in front of developers and allow them to iterate more quickly without need from
00:20:12
intervention from us is it helps both them and us because it frees us up to continue to work on more tools that make their workflow more efficient and and allows them to not be stuck waiting on us for things um how do we test and benchmark our systems for stability that's interesting so the there's been a lot of work there for in the past um to try to get the uh get that up to speed um nowadays uh there are multiple ways where stability is being tested so uh on the platform side for example we have a a set of tools built on locust which is a load testing framework written in python which is super nice uh and it allows us to basically simulate you know thousands and
00:21:00
thousands of users uh so we can write scenarios about what they do for example you know uh melt the ship like what if you know x amount of players melt the ship right what how does the system react so we can we can test that and we've used the same principle in the same framework to build game uh game facing locust system that we call pestilence which is basically yes i know the naming is nice but basically what pestilence is able to do is mimic what you guys do in game so he's able to log into the game connect to the servers and then do specific actions that we want to test so if let's say you know we have a service that has an issue and we're trying to see what's causing issues and we can actually write a scenario that does that and then we can spawn thousands and thousands of locusts doing that
00:21:48
in game to simulate what happens when we actually launch a build because you know that's usually what happens is like there's a big storm of people doing the same action and so uh so we can test the ability this way now it's getting more and more integrated into the process where we write scenarios for different services as we go recently we've added the ability to even you know provision account data from it and so we're able to you know pre-warm some of the databases uh to prevent some services from getting hit too hard when the patch is launched things like that so that's one thing that that's the major thing that we do for service and platform specifically for load testing on the game side i'll go forward more so these are headless clients obviously exactly those are even more than headless clients they're not even running the game itself they're
00:22:37
literally just you know running the cloud side of the game if you want okay okay but we do have endless clients as well that's where i was going right where there are automated systems that can do which is called a cloud test launcher we can which can actually do that running the game client as well but the testing there is more focused around testing the game server performance than you know the cloud the the mmo side of the game basically yeah that's martin senior and his team working on that cloud test launcher we'll actually be talking to uh his team about it in isc next quarter hopefully cool awesome in case people wonder how often how far ahead we playing these things um let's see ahmed is there hardware parity between all the different regions uh it often seems like well let's just
00:23:26
let's just go there let's just it's just let's just get that one yeah yeah 100 we do not discriminate they all they all have the same spec uh what really happens is player behavior affects a lot what's happening in the game server and also your latency like the the game server the dedicated game server is a very very sensitive workload it's very sensitive to any kind of overheads instead of any kind of latency issues so no everything is apparently we do not discriminate actually the only workload that we have the always running the same exact platform no change no matter what the environment is is a dedicated game server there are other services that we can give them less resources in smaller environments because we know they're not going to get hard when it comes to that monologue simulator cool dedicated game server we always make sure that it has the same exact spec after where it goes okay i i know the the second half of
00:24:15
that question was uh what was this person was from the australian i'm guessing this person's from the australian region and they've never seen that it seems a little more sluggish but as far as the actual hardware behind the servers they are at parrot it is the same yeah and we actually care a lot for our our customers in sydney like one time we had issues in the sydney abc and we had to locate some uh some vans in mumbai like in india which is it's the far but still it was the closest area in dws and we got really upset we're like super fast we're gonna make decision go back to sydney so if they're having any issues like i i guarantee them it's not related to the hardware uh how how much storage space is needed for one game instance it's the same way that like when it
00:25:03
comes to how much so that the windows built into linux build a difference in size right but the different size is very like not that big of a deal that the data file the the the b4k file that everyone has i have to have it on my servers too so we can stream all these all these static files in and out and make the game server run so it's very close to how much you download to run your client okay um you you you said you said linux build and i know there are people i've already seen it show up uh you want to go into detail server builds linux server build yes i don't talk politics i like technology just wanna just wanna clear that up there so that people don't take that uh where should it be let's see what else we got from the chat here
00:25:50
uh when a new build is required for the ptu how long does it take to compile on average three to four hours roughly is there is there something in particular that could spike that one way or the other um well there's always a lot of load on the system so that could always cause times to fluctuate quite a bit um there's a lot of uploading that takes place during that so sometimes the network can be bogged down it's not just compilation time there's a lot of fan out and fan and there's a lot of steps down we go do a lot of things in battle and then we can look them back and and these are systems sometimes sometimes they are under development themselves sometimes we make
00:26:37
changes that make things take longer or take shorter but yeah if we were talking in general like might definitely said between three to four hours i would average and you're right i'm in that when a build is finished cooking it's not ready to be distributed and it gets low on it for a while now it's too hot exactly you know let it let it sit a bit right but no it's because it needs to be uploaded right so that it can be distributed and that step is fairly long as well that's why the total time is true for us the client compiles themselves or relatively quick as the the data is typically what takes the longest and that can go for an hour or two for sure a big game it is a big game and only getting bigger uh what methods practices do you utilize to organize
00:27:27
branch management between teams um that's base that's our um kind of our internal feature stream setup where we have um teams as they are getting ready to start work on new features they will come to us and request a feature branch and devops will go create that feature branch set up their integration flows using the automated tools that we mentioned earlier and they'll move into that and and once it's feature complete we help them where needed to get back into our dev branches and kind of repeat the process with teams as we go is it annoying trying to find a file that was supposed to integrate and didn't occasionally um
00:28:19
hey benoit let's talk issue council here's a good one uh are systems like the issue council directly linked uh to the internal jiras or whatever internal jarrahs and stuff yeah yeah that's that's been the idea since the beginning right is that the uh report that's filed and confirmed on the issue council gets uh if you see the cigi on it then it's been imported to jira basically and so at that point it can be tracked internally it can be assigned to because you know product you know developers need a jira ticket to work or check in right and so that's the way we legitimize the changes that they have to do and so yeah issue consoles connect the the same same way to feed the status status back from issue cancel um you know there are there's we're aiming to do improvements to that flow to give more
00:29:06
visibility but you know we still have a long ways to go before we can hit that the roadmap is the same thing the current roadmap so the cars that you see on the roadmap also comes from jira so the the this is a live representation of the production as as it stands the moment it gets it gets put up there uh so yeah it is connected and that's the key it's the data source and the data destination uh what level of automation do we have when in our build and deployment process is it highly manual is it highly automated how would you characterize it ahmed i'll talk about the flaming slides so when it comes to deployment depending on that i kind of touched on that earlier under the deployment size if it's a smaller deployment that locates like reasonable amount of cloud resources this is fully automated anyone
00:29:55
can go through a go interface and click and these deployments you are not allowed to change so much of them you can only change a few specific facts and it's based based on a system that runs off get off of like a source version control system and uh and against the automatically everything gets built and everything can get destroyed also by by goi or by a user interface when it comes to any large public facing environment these we're talking about a lot of resources so in the current phase of the project because we also still have systems or areas in automation that imperative where we tell it how to do things not what we want that's that's a very important topic a lot of people i see and the questions are from the devops community that's a big thing we're trying to shift a lot of our stuff from telling the automation how to get there to what we want and let the automation business system
00:30:42
like frameworks obtain that state but to go back to the point if you have a larger environment at the initial part of it you have to have a human decision you have to have a review another human decision making sure that these are the correct values and then it goes from there and it becomes automated the node scaling kicks in and manages the environment from here yeah i think it's uh it's comparable to a lot to the way that we use like procedural generation in our our planets and stuff you can use these automation tools to help you and to assist and to you know get you started or maybe even finish a job but that's there's always got to be these check posts where humans are involved to validate the decisions to to determine what's going on it's just you you'll never there's there's no replacement for the human element and just add something really quick as
00:31:31
the project matures as parts of the project material as we know we're not going to be moving a lot of knobs here we can offset these parts and make it like super self maintained self healing is a big topic in our industry and we want to talk about it if a reactive at the active environment or the environment that reacts to itself and we might i get to talk about a little bit about how we're trying to enhance uh the metrics or the traces or the logs or the insights inside our services so we can let our infrastructure make decisions based on decent sites yeah yeah i mean one one key part that's still a human intervention i think both on platform and game is the the flip of the environments right so the the game as ahmed was mentioning is is deployed using a green blue deployment that means that you while you guys are playing a game online on prod for example
00:32:19
and we're preparing a new build there's the you know that's the blue one and then the green one's being built or vice versa right and so at some point when the bill's validated there's the the flip happens and a human says okay we're flipping and that's where the the bill goes live it gets available and launcher servers are flipping over and rotating and so that's that's still a human doing that and it actually requires a couple people saying yes yes yes before it actually happens oh oh no i i remember i learned that specifically with the uh the launch of 3.0 when we're sitting we're sitting i'm sitting in a bathroom with a razor ready to destroy that giant gremlin that was on my face and there's and everybody's going through the go no-go checks and everybody's like oh everybody's right everybody and then it comes down to it it's got to be chris or aaron to give the final go before
00:33:06
they actually flip the environment over and and uh and and chris goes it's just everybody's quiet goes well maybe we should do a couple more checks i'm sure jared doesn't want to get rid of the beard just yet and boy did i i got i'm i'm in that call i'm supposed to keep it muted i'm not supposed to contribute i'm just there monitoring but you can bet i unmuted my mic and i'll let some people have some colorful words and no one will forget this 2.0 it's very similar to the build system as well which is highly automated but somebody always has to make the decision to request a build even if that's a release bill right from the live chat is there a secret dev build a secret dev build secret dev there's
00:33:55
always a secret to building secret they've all got uh ahmed what uh what deployment strategy are we using are we doing uh rolling canary what are we using so we do them all like we discussed in the beginning about how we do blue green blue green is usually when we are delivering uh a lot of breaking changes a lot of things are like okay we cannot have half and half these things are not going to play nice together and this is an in development like a product currently in development so we need to get you guys in the new stuff so we can see what's going on from the moment that we publish a big build to a large environment like live we can do well we call them hot fixes we can do between five and twenty half eggs and these are either canary released or
00:34:45
rolled like in three 310 we carried six hot fixes for the hub server and we rolled the last one uh there and that happens in every single aspect there are areas of the game that can allow for some toggling where we can toggle features on and off or hot loading or hot changing data and there are other areas that because the challenge in running a game in the cloud is the statefulness the experience between a very immersive game like ours a game client like ours and the game server is super stateful this is a very fragile experience if you did anything here you lose that that connection and and we we are actually always discussing how we can make it make that less fragile so people like us are not do not have so many precious parts of our environments that we cannot
00:35:33
touch the players are off but we do both we do canary we do ruling and we also initially do a blue green hopefully in the future as the project matures we start dealing with the concept of a deployment it's an environment that's always going and there's always pieces moving all the way around that we we still have have to do a lot more things before we get there it's fair uh miles what's the uh what's the average duration of a full production rebuild on the main star citizen branch thankfully thanks to our caching we haven't had to do one of these in a very i don't think we've had to do one of those in a while um i think the last time that we did do one it ran for about 24 hours thankfully that almost never happens
00:36:20
nowadays remember i'm sorry just i'm i'm trying to remember which chris roberts game it was where somebody basically deleted the entire uh it basically deleted all of game dev and they basically savage salvaged the game by having somebody had taken a copy home to the ball game to work at home and they had to like they had to like carry this person's computer from from home into the office with you know uh until this i try to remember which game that was i think it was strike commander ben would ben would know ben would know i think it was strike commander um how many uh oh we touched on this a little bit but
00:37:06
we'll do it again how many human or manual interactions are there between commit and deploy to get a build there's only two it'd be developer making the commit and then the developer kicking the build depending on what environment you're wanting to deploy to it could be potentially one more human interaction which uses the same interface we kick our builds through to deploy to our internal staging environments i think ahmed could address outside of that for me yeah yeah if it's a larger environment it goes through like people talking it goes through people deciding that this is the ability we want to go with and then from then the humans just just do the decisions we were talking about earlier everything is driven by version control everything is driven by get but there's a human saying yes and we
00:37:54
did not abstract that into a goo yet we don't we didn't see the need for that this is this is um we call ourselves devops i know this is some kind of haiti topic for a lot of people devops is a philosophy not a title but that's what the market goes but we try to to really hold the values of devops close to us we share the ownership of that product this gaming experience is ours i can spend more than 50 of my time investigating an issue about how a player isn't able to connect something because i i have the access to all these tools and i can try to go and investigate it so to go back when it comes to deployment to a larger environment that's usually where human decisions are being implemented but all all the actual work of provisioning configuring securing is dominated but it would take days it would take weeks to deploy the whole environment manually like with how it
00:38:41
spreads around the world uh i was trying to remember what was trying to remember what the devops uh what that devops chart was the the figure what what what are the points on that one trying to remember what they are it's it's a lot but see that's the thing these these terms are coined and and they create some some issues sometimes um my take on this not to i can take the whole house out there be close to your problems know what kind of problems you're trying to solve and we are in an amazing time where there are a lot of open source projects that you can go and and see what everyone else is doing
00:39:28
and get it done every every organization taking every project they can devops is kind of unique and we we are one of the very unique ones that's a safe bet uh in the last year what this says in the last year but if you want to go beyond the last year you go beyond last year uh what patch was the most challenging for you on the back end if you think of a nightmare patch for you guys what what what what's what's the one that you don't like to think about you want to go last almond i'm i'm thinking i think 3.0 for me because uh because the the patch before that was so far right and and to you know to echo what uh almond was talking about in terms of
00:40:17
devops philosophy if you don't deploy for a long time then you forget how to deploy right and so 3.0 was the the longest running in development patch of star citizen i think that we've had right and and uh launching that touch on everything everything was new there was like it was the first first for everything so it's like a brand new game and so for me that was by far the the most difficult to get across the finish line uh and that's you know then going to quarterly made it a whole lot better like as now we can deploy you know more often which is which is how you get this rhythm going right and keeping keeping it going it also means you know to be fair the more you deploy the more you break stuff as well right so there there is a risk there yeah don't touch it right exactly don't
00:41:08
touch it it's good but uh yeah for 3.0 for me by far i don't know for you guys how about you miles uh my my problems are very different usually uh i'm i'm more heavily involved at the beginning of the cycle less than the end so um 3.9 was actually pretty challenging to transition to as far as getting the new stream set up and making sure all the changes were integrated to where they needed to be we definitely had a good time with that one there's a lot of workflow changes at 3.9 for sure yeah yeah for me it's a difficult take between 3.0 and 3.9 uh 3.0 definitely 3.0 we rolled out a new hypervisor a new generation of virtual machines and
00:41:57
a lot of new things and it was in a very critical time i think we have an episode of one of our shows that i'm not very good at keeping up with where we talk about what happened with 3.0 3.0 there there was there was chats between cr and tony and uh still very like i i still remember very very very dear message to me when tony said i gotta go put some some gifts under the tree and come back real quick that that was 3.0 and it was a lot of fun it was a lot of fun uh 3-9 was a lot of fun too with like fleet week wow we we we are spoiled by our player base our blair berries are actually developing this game with us and i'm i'm not saying that because like it's if you really see that and 3.9 was a lot of people doing a lot we were learning a lot we cannot stop
00:42:44
learning from what happened during three nights and flee week and uh one of the major also ideas in devops is measure everything and uh has been when i was talking earlier testing a game and attributing this to a game is challenging it's it's it's relatively doable to test rbcs or api calls but the immersion like how the game runs and everything that is difficult and three nine we we had that we had we had that okay everyone is testing everything everywhere they're doing everything and we were receiving all this data and uh digesting a react into it i i do not have an accurate number but the number of half fixes that went in three and a half day week was really really high yeah i was i was wondering if one of you guys were going to mention the conversion to the delta patcher if not the that conversion but actually
00:43:34
this actually went pretty well that was 3-0 yeah it was three i wasn't [Music] on ptu for like a month and a half i just remember my biggest memory about launching the delta patch system was the the moment we had the ability to push a build a day to players we did that for 48 days straight it was all right i i do have very vivid memories of converting the build system to support that that was yeah that was definitely a challenging time as well yeah i i have a sad memory from 2015 by coming to the office at 1am so we can prepare
00:44:21
the old school bed so we released at 6am and that was voluntarily like like we would go and work on it because it used to take hours to prepare these facts and make sure you have all the diffs between all the old versions yeah you actually bring good memories about the delta vector and right now sometimes we see we look at the cdn graph absent we see these beta bytes coming out like can you imagine if the delta bachelor was not here how much data you're pushing it's yeah it's one of these things that we take for granted now but it was an amazing big exchange for us critical technology for sure it's a seemingly simple thing but it's like absolutely critical to the ongoing life of any mmo uh let's see what else we got um will i cash and global persistence change how
00:45:09
long term persistence is handled oh well that's interesting so um not really in fact the the long-term persistence was engineered to basically uh persist your data that you earn in game between patches right so it saves your money and the history of all of the uec money that you've got so every transaction up and down current values of all the your ledgers um the inventory systems that's part of ltp is also there to store hey you own a hornet right or a you own a constellation right um the state of these objects so you know oh this hornet's missing is like left wing was or was abandoned at grim hex or so that's stored in this in the persistent system so icash will basically bring more state tracking more
00:45:58
state persistence but we're still going to persist the same amount or the same kind of records between patches to be able to rebuild the state when we do a data wipe on the icash site so the the reason and the reason why that's still an issue is because the schemas of those objects is still moving right and so we we still need to do that but um the only change that will come from it is that um there are some parts of the icash api that allow better handling of stacking objects so let's say you buy 50 magazines right the way that's handled in the akash is is much better and so the ltp systems will track that better and so that's not the only change but the functionality for players should function the same way and for anybody that hasn't checked it out uh check out the last the previous
00:46:46
episode of calling all debs where we talked with chad mckinney about uh icash and its effect and lead into global persistence uh let's see we got 10 minutes left we're almost done are there any improvements to publishing technology you're excited about in the near future yes so as we take some of our legacy systems through imperative uh described deployment where we're describing to automation system how to do things to into more declarative systems we don't want to throw in a lot of jargons but people who work in devos technology most probably know what we're talking about here that is very exciting for us we're also looking at how we can leverage some of the bare metal resources that allowed a lot of cloud providers now we've always been trying to squeeze as much power as we can get
00:47:34
and we want to see what will happen when we are going to have the hypervisor anymore maybe maybe we'll get some some more power out of it uh again we we are not until we measure till we measure things and we say benchmarks but yeah i'm excited for these two things one of the most important things i'm excited for we have two new engineers that will join the team on monday so that is that is a big thing hopefully more hands on the keyboard more minds more more people designing all these challenging solutions with us so yeah a lot of a lot of good stuff happening this has definitely been a uh uh a year of growth actually last year was here we've been growing for a while so i'm thinking about it uh over 600 and somewhat people knew yeah let's see all right we've been throwing yeah we've been
00:48:21
throwing these softballs let me look for some hard balls here when a server crashes here we go when a server crashes why when a server crashes uh how long is it until that server is ready to go again and is this process automated yes so the server doesn't have to come again if if we are actually in a timer we're shrinking down we are actually have a lot the servers can go away and never get used again there's another pool of servers if we are in a time actually we need resources the server will crash and it will come back in a different new incarnation we'll have new logical name to it have new id to it and i'll become part of the pools of innocences that can host games but nothing nothing has like a long-lasting
00:49:11
name or a long-lost existence everything is that the common word that we use in our field is a thermal so these instances are just instances they come and go all the time no one touches them they decide to question their own they get sad ants not cattle cattle not pets right yeah yeah absolutely absolutely ephemeral good word use ahmed uh what benchmarks do you use to determine how a particular build progresses from evocati to ptu to live now before you answer that though i want to clarify that it's nobody in this call that determines whether something goes from one stage to another stage those are those those are calls made by uh you know you know the the directors and above and stuff like that but as far
00:49:58
as the benchmarks just speaking to the type of information that we use to make those calls it's a very collective democratic call the way it works whenever we are in the mode of releasing we have the live sync meeting that like one of my favorite meetings we get to have it early in our day kind of like towards the end of the day in europe and we bring up a lot of data whenever we go to btu there are several types of reports one of them is the stability report the stability report tries to model that how much disruption we are causing to the player experience and then we have a player feedback report that comes from the player experience team and it takes an account all the issue council issues the feel of the game if there are specific features in the gameplay qa also goes and takes that
00:50:47
that report and goes through and try to reproduce by the morning by the next morning all these like data got collected we have performance data and then then everyone has a case right and it's it's a lot of conflicting factors together that decide we want to progress faster between waves because we need the concurrency we need more players to see things how how things are going and you get to decide what the messaging is going to be like what's the trade-off okay it's going to be amazing in all these areas and this feature is going to be really bad i think our players our backers will understand that we can just explain avoid this area it will be all right and yeah it's a lot of moving factors when it comes to benchmark it's it's a quantified collected crash information stability information
00:51:35
performance information resource usage information all collected together we'll abstract the number in a model we call the stability score and the same side also we have we have hardware working teams that try to collect and summarize the game experience and both of them get together and then we have different goals and different groups and a very democratic way of deciding how things would go and yeah hopefully we make the good decisions every time that's a that's a that's a great answer there's there's there's a ton of hard data but then there's also that um that experiential stuff that we get from from our qa and stuff from the from the rep we use we go to the player you know teams you know and get their feedback from the players just like that and all of that gets put into a a gumbo of sorts where we it looks and
00:52:24
it creates a picture that's unique for every single uh release candidate and it's it's like from this candidate to this candidate to this candidate uh the the exact criteria that determines whether this moves forward and the next step may be different for each one uh so it's a there's no easy a rubber stamp answer for it but uh we use as many tools and uh equipment and people as possible to make the best possibilities even even development teams like some of the game service guys are going to be watching live ptu builds when they're you know to see performance metrics and get a sense of it right and so that's also in the mix you're right it is a hodgepodge of different you're full of you're always there just like kick it just kick it out
00:53:11
just push it go go when was in these things he's like i'm on the east coast it's already late just push it time bills uh that's it guys you've done it congratulations uh thank for taking the time out of your busy days at the end of your week uh to be here on the show with us um it's it's a i i it's not it's not always the uh it's a relatively dry subject uh for for folks and people often ask me why uh why are you doing this i'm like because this shit's important and that's twice i've sworn the last five minutes that's right i'm gonna get an email i'm gonna get an email sorry guys uh we are going to uh we are going to do a raid who are we raiding today uh we are rating a similia
00:54:01
uh similia is a streamer in sweden if i remember correctly uh so so check them out um and uh if you if you jump into the raid uh there that when it kicks off here uh tell samilia uh to push it live benoit said push it live ship it push it benoit said what did i say sweden i think i meant switzerland switzerland switzerland so check that tell me where's america it's cool called the outer world i'm just joking i'm just joking uh please send all letters to here uh james austin texas for ahmed for miles for benoit i'm jared
00:54:50
take care everybody and we'll see you next week

Transcript from YouTube’s automatic captions — punctuation and Star Citizen jargon are approximate. The video itself is hosted by CIG on the official channel.