Star Citizen: Around the Verse - Austin Studio Update

11 January 201800:56:471,481 linesWatch on YouTube ↗

Transcript — every row opens the video at that moment

00:00:14
hello and welcome to another episode of around the verse our weekly look at star citizen's development I'm sandy Gardner and I'm Chris Robbins to give you a more in-depth look into what we're up to we have a new format for around the verbs focusing on one topic per episode each month we'll cycle through a studio update from one of our offices or an update on star citizen's persistent universe eight squadron 42 update and a installment of shipshape yeah I'm looking forward to having it will allow us to dive a little deeper than we managed to last year on some of our ATVs will also be premiering a couple of new shows this year calling all devs a new Q&A series hearkening back to the days of 10 for the chairman and 10 for the producers etc and reverse the verse which is coming back from the dead which will be a live weekly talk show where
00:01:02
our devs will discuss the most recent around the verse and other star student news and I think that'll be kind of fun and interesting yes it will be an Erin Roberts is our first guest on the first reverse the verse which airs tomorrow at noon PST in addition to the new shows we've been refining our subscriber experience based on feedback from the community yes we have and we're kicking off a new year of perks today with a series of two ketsu action figures to collect and display in your hangar January's figure is a Vandal warrior with Imperator level members receiving a special variant as well yeah very cool and this week's studio update we head to Austin Texas where the team recently finished up publishing efforts for star citizen alpha 3.0 let's take a look at what went into getting the latest build live in the PU hey everyone tyler witkin lead community
00:01:59
manager in the austin texas studio and i am joined by director of persistent universe Tony Zurich Tony how you doing it's good so now we concluded 2017 with releasing star citizen alpha 3.0 to the live servers there's a lot that goes on behind curtains to get something like that done so let's talk a little bit about the run-up to 3.0 probably a bit more challenging than what we would have liked we would have all liked to have finished it you know a few months in advance and been able to you know coast right up to the end of the year in the end it came down to there was actually a lot of debate as to whether or not we should cut some of the critical foundational features from 3oh and pushed them into three one but we really wanted to ensure that we got the basic skeleton for what we're gonna need for all of the evolutionary gameplay
00:02:48
enhancements that we're gonna deliver in all of the you know the three point x series 3.1 3.2 3.3 on a quarterly basis next year and so we really you know despite the fact that it really pushed just right up to the very edge we were determined to get all of the critical features not just the you know the the big obvious ones like you know the procedural planetary stuff and everything like that but i'm talking about things like the dynamic shop prices i'm talking about things like interdiction which is really necessary for things like cargo transport to have any sort of difficulty associated with them you had to have that you know functionality within the game and the fact that we got that in there and we were able to work through most of the big you know the big complex issues puts us into a very good position going
00:03:39
forward such that now most of the uncertainty most of the you know the the research and development most of the the more serious challenge of getting these pieces of completely nicked a new technology integrated and functional within the game and talking to the other systems without out of the way it should make all of these three point x iterations a lot more predictable doesn't mean that everything that we aim for because we're always aiming fairly aggressively in terms of what we want to deliver because we want to see this just like the community in the game as soon as possible and so we always try to push ourselves regardless of whether we think it's gonna be you know do you know difficult or easy you know to make it we want to go ahead and slot it in there and try and sometimes sometimes we go ahead and reveal those more
00:04:28
aggressive dates to to the community I think that we're being a bit more conservative certainly in our planning and stuff next year and the whole concept of going date based as opposed to feature based is going to dramatically change I think everything next year all of a sudden we're not going to hold up the entire version for one or two features that aren't quite there we'll push out what we actually have which is going to make the game you know each one of these iterations is basically going to push you know what what makes up the game significantly forward and so we're gonna be able to get you know the latest and greatest out to the community as quickly as possible and then if something does slide from say yeah three one and two three two they'll see it a few months down the road yeah and the community is actually really excited about that us us too I mean this is it's it's painful from a
00:05:17
development perspective to have so many things in there like I said it's like for three Oh was a bit of an anomaly because we really wanted to get the you know the you know like I said earlier the structural foundation into the game and we didn't want to leave pieces out and then we deliver something and yet they're still big missing critical pieces of infrastructure that we were gonna have to tackle you know on three one or three two and because they're big critical pieces of infrastructure if they slide from three one then all of a sudden it's not just one or two or you know things that might slip you may have an entire branch of the of what you're hoping to deliver to the community isn't possible without that actually being in there there are a lot of you know knock-on effects from things like the dynamic pricing in terms of the economy that if you don't have that piece of the
00:06:05
puzzle working all of that gets pushed and so now that we've got the basic structure in there's certainly the potential for little individual pieces you know here and there to slide you know from one three-point X release to another topic but it won't stop the content flow and the impact is likely to be much more localized than it would be if we were missing a really really big piece of the plumbing absolutely so you talked a little bit about some of the releasing 3.0 and how it was tough let's say at the end of December what were some of the challenges you guys encountered well I mentioned a couple of them there the you know the dynamic prices were right up until literally at the very end we had testing some rudimentary testing that was happening a few weeks before we
00:06:54
ran out the door but we were probably within about two weeks the timeline before it all finally started to click together same thing with the interdiction stuff and you kind of saw the outcome of this which is we were tweaking some of this stuff right up until you know the very end and we finally got it in at work there were a few minor things which you know which we paid for with some late nights you know right on you know Christmas Eve Christmas Day etc to where we have you know a very small number of people that just really wanted to make sure that what players were able to experience and enjoy over the holidays didn't have any really egregious you know problems a few of us basically just put in the time to go ahead and push out some you know some some temporary patches to basically make
00:07:44
those things function considerably better than where it was a few days earlier yeah never ceases to amaze me I mean all the time I've been here it I've seen consistently people almost saying like begging can I can I stay well let's get this out the door it's really testament that we really want this just oh yeah yeah yeah and it's actually interesting because near the end you have everybody wants to put as good a face everybody wants to make sure that what they're working on is going to be perceived in the best you know possible light by the community but as a company and you're trying to close so many things down you really do have to be taking entire portions of the project and saying okay this is you know we can't risk tweaking any more of this because we may destabilize something and we're basically doing these you know these release builds and we're trying to
00:08:32
get the number of you know of significant issues down to the bare minimum and so a lot of this stuff near the end you're looking at it you see a problem and you're saying well it's a problem it's not very big and you know the impact is gonna be fairly limited you know this is what its gonna do to gameplay and so you're weighing all of these and making the decision of whether or not you actually have to fix it for three out or whether you can go ahead and push it till it's you know to where we'll deal with it with till three one on some of these things you're able to just you know close close something downs such that you don't necessarily fix the problem entirely but you're able to patch it such that with very little effort so that it works well enough for three oh and that basically buys you to me the hard dates what their quarterly releases it's really gonna help that because it's really gonna organize the entire process yeah the the dates the
00:09:20
dates will change the dates will change everything not least of which is 303 o was always gonna be a gargantuan release there's no doubt you know there was never any question about that the end so it's very difficult when you're talking about integrating that much new technology and a lot of that technology hadn't yet been finalized so there were still a lot of implementation issues and certainly you saw some of the than you know the knock-on effects in terms of well once you get it all glued together what's the performance like oh the performance is not optimal okay well we've got a limited amount of time to basically make that run as well as it should it's not that we don't you know see the issues it's not that we don't want to fix the issues it's just that by the time you actually finally get all these pieces interconnected and working you're able to test you
00:10:09
you know then time does start getting short obviously you know on the performance side three point one one of the primary focuses of three point one is going to be to deal with speeded game basically finding the source of some of you know the biggest problems in terms of you know server and client framerate and bringing those up substantially to improve the player experience the the single biggest feature I would say by far is of course the the whole procedural planetary this dramatically changes the entire game because all of a sudden we've got such a larger play field to you know on which we can paint all of this content in addition to that what I personally like is the fact that the types of gameplay that you'll get in that environment all you on that
00:10:56
environment is going to be significantly and dramatically different than what you're gonna get when you're flying around in the spaceship all of a sudden you know when you're talking about things like you know first you know first person you know activity down on a planet you're talking about things like taking cover you're talking about you know you're much more exposed you can't take nearly as much damage as say you know your ship potentially good you're talking about you may be dramatically out gun not because the other guys got a bigger or fancier ship but because he's got a ship or a buggy and you're on foot so all of a sudden the magnitude of the differential between what different players or you know a player and the NPC's could be bringing to a given encounter eating it increases exponentially and what that's going to
00:11:43
do is open up a lot of interesting possibilities for the clever player to basically to take to take this and figure out - you know to use their brain to basically figure out how best to exploit an opportunity to turn it around to where they're at a disadvantage but because they come up with a more clever strategy they're able to you know overcome in a way that another player that's not as experienced wouldn't necessarily be able to do and so there's gonna be this whole aspect of getting to know more about the game and and as you start to learn more about you know how things work of course you will be able to excel at whatever you're trying to do within the game if you if you tailor this to something like mining there's
00:12:31
going to be which is now and I'm happy to say that we're now turning our attention to this in significant fashion because we're we were gonna try to get this out you have four three two the initial iteration of mining and results and resolve it to a level such that players are actually kind of like the cargo transport to where you're actually able to go down on the surface you're actually able to scout find areas where there are you know worthwhile concentrations of valuable or the value of which will be determined by what the price is at a given Trading Post at that moment in time sometimes the price of a particular commodity may be higher sometimes it may be lower and therefore the you know the justification for you
00:13:19
investing time and money and effort and enduring risk to actually extract these natural resources from an area and basically card it back and sell it it 11 flow along with the economy and so you will finally start to see some of this emergent gameplay that we've been talking about you know so long actually coming to fruition which is awesome now a lot of folks noticed in the community that we had some bug fixes and changes come in over the holiday break while the team was you know supposedly out of office which begs the question how do those changes take place because sometimes fixes and changes require a patch sometimes they don't so how does that work yeah the reason why we were able to effect some dramatic changes to the game for a couple of specific issues was because those particular issues were
00:14:10
controlled via these back-end services as opposed to the game per se the game basically takes care of all of the I'd say the the high fidelity simulation but it gets a lot of information from these back-end services and you're gonna see that like for example in terms of shopping the game server itself doesn't actually know what the price of something is it doesn't know where certain come out of these you know should which shops you know should actually have them doesn't know what the quantity should be it doesn't know what the relative demand is it doesn't know about any of that stuff that's all controlled via you know this this holes you know back inside and so what the game does is it understands when it needs to request certain chunks of that information it knows how to display it to the players to allow them to make
00:14:58
choices and then it knows how to take the outcome of those choices and plug it back into you know it back into the service infrastructure so that it can have you know the desired effect and when you're talking about things like what we were working on you know Christmas Eve Christmas Day there were things along the lines of the environmental mission scenarios which are controlled the probability volume service and the actual problem that we were running into was that some of those environmental scenarios weren't being properly cleaned up and so they were residing too long on the server and so all we wind up effectively doing is basically tweaking the probabilities of some of those things so that we made them a little bit less likely to occur you know tamp down you know the number of ships and some of the scenarios and
00:15:46
what that in essence did is it gave the cleanup functionality that was running on the server a little bit more time to clean things up before the next yes scenarios you know on average would be getting created so basically it just evened out the you know the you know the the demand load at any given time a little bit and so the interesting part about this is that as we push more functionality of this sort from the server back to these back-end services it makes the iteration much much easier all of a sudden we don't need to basically do a big you know gargantuan you know build of the entire game run a tqa do a much more massive test all the sudden we can be much more you know focused and specific in terms of what
00:16:33
we're testing it's like well it's the same game version we're just gonna tweak the servers or we're just going to tweak the data that the service actually utilizes and you'll see a lot more of this you know we'll have you know for 3.2 for example there will be a commodity service will have there's gonna be a service beacon service there's going to be will be changing you know fairly soon so that we have the air traffic control system we'll go to a service and so in general it's just taking certain key select pieces of functionality from the game and putting them into the back end where they logically belong and there are a number of reasons for this you get some nice benefits you know like I'm talking about the ability to more rapidly iterate and tweak this stuff but one of the big reasons why we have to do this is this is necessary speak
00:17:21
we're getting closer to the point to where we're gonna try to get this whole you know server mesh functionality going to where the game is not exclusively run on one individual server that knows everything about everything where we're headed in the not-too-distant future is you know essentially going to be there's a solar system and there's hundreds and hundreds of different servers and each one of those servers knows about the key points they know where the Sun is they know where the planets are they know where you know some major pieces are that they could actually see or that the player on them could interface to at any point but in general most of their attention will be focused upon just a little localized area where they're at and this will give us all sorts of benefits in terms of you know performance it'll give us all sorts of benefits you know in terms of a whole
00:18:10
number of other things and you can start to understand why you need this when you think about so imagine you've got a server and it just knows about some key a few key elements of the solar system and but most of its knowledge is fairly local to the area that it represents but now all of a sudden a player that's going to go from that server to the far into the solar system he needs to be able to query for example all of the probability volumes that lie across this enormous expanse the problem is that server doesn't have knowledge of all of those probability volumes and so what you really need is a higher level system that does have that does have knowledge of those particular data structures to the level of fidelity that you need it and like I said it's the problems that we're usually concerned with on the back end are usually quite a bit different in
00:18:58
concept from the ones that we're worried about over on the game side the game side is really where you know like I said you take care of all the high fidelity simulation all that type of stuff the back end is usually concerned you know I would say in general it's more macro cosmos weeks yes okay cool now you keep mentioning probability volumes I've heard that term thrown around a lot what is a probability volume a probability volume is really just an area of space that dictates what you could potentially see within it and by that I mean are there a lot of pirates this area should you be seeing a lot of cargo transports going back and forth from where should they be going from where should they be coming into where should they be going should they you know if they're headed in one direction
00:19:46
so that should they already be loaded up if they're going in the other direction should they be empty if they're going somewhere and they actually have cargo what type of cargo should they have and this is all dictated by these probability volumes and the way the workflow you know generally works is that the designers set up these so-called environmental mission stars we refer to them as missions they're not missions as you would normally think of in terms of well you have to explicitly accept something and there's some sort of you know definitive you know reward that you're gonna get when we say environmental mission scenarios what we're really talking about is just stuff it's just the world happening as you're driving by and so the first step of this is that the designers create all of these environmental mission scenarios and then for each one of those scenarios they go and they tag it in various ways
00:20:35
to identify what type of content it contains they can also associate various rules you know and requirements with you know with each one of these scenarios and these scenarios can now be linked to each one of these areas and what we're able to do is basically say how the probability of running into this particular type of scenario and these can be combined you can have you know you can have an area that's tagged for pirates and it's tagged for Frazer makes an attractive pirates in this area yeah and and then you can logically combine them so that oh well if I for example you can combine these things explicitly and implicitly and by that I mean if you have pirates and you have freighters then it's only natural that you would also see a lot of debrief because some pirates have actually blown up
00:21:22
freighters and if you have security Ayoo that's actually in that system then sometimes if you think about if you think about how big space is the odds of you coming upon a freighter Rises it's being assaulted by a pirate you know is actually really really loyalist it's it's not it's like the vast majority of the time you would cross the area of space where they were well or those two came into contact or well after and that's not for each other and and that's in general not that interesting and so we kind of tilt you know or we're able to control the likelihood of you seeing more or less of something so the player sees a little bit more of the interesting stuff and not quite as much of you know the the after-effects except insofar as we actually want that to you know establish
00:22:09
the general you know atmosphere of a given area in other words if you're in a dangerous party and you see lots of you know lots of debris you know that can actually be used to make that area you know to basically broadcast the fact that you're in a dangerous zone but that remote salvaging profession oh yeah that's yeah that's that's there's a whole discussion we can have about the salvage so in general you're able to take these probability volumes the designers right now and they can create these regions of space and they can link them to these environmental mission scenarios which have been specially which have been which have basically had tags and various other rules and such associated with them so that we're able to instantiate them as appropriate based upon you know distance from the source
00:22:58
point of you know of these volumes and really the what the the the piece of the puzzle that we don't yet have operational which we're going to be aiming to implement later this year is gonna be all of the systemic connections so that what an area basically knows about is dictated by what's actually going on via you know just players and NPCs in general in other words right now the designers are having to do a little bit more hard coding of these volumes than what we're gonna be doing and you know six or nine months time at that point what we'll wind up happening is you'll see the the macro you know economy will basically be providing
00:23:48
inputs to the probability volumes which will then be querying and all the environmental mission scenarios to figure out what you know what you should see given time and place very cool and so why do we decide to go with this technology why probability volumes the big benefit to going this route is really one of optimization clock cycles in general on servers a zero-sum game and so we want to be able to put the vast majority of the clock cycles to work where there are actual players but we also need to have the simulation aspects in terms of well if you've got a lot of pirates in an area and they're basically picking on shipping well then you would expect that the price of security you know goes up to a certain level and if players aren't responding or even if they are there's also going
00:24:35
to be NPCs they're responding and so how should that gradual rise in additional security forces taper off the frequency of piracy you know within that sector and so what the probability volumes give us in essence is really the ability to more effectively simulate a a big complicated world where a lot of things would theoretically be happening but we're controlling it via dials and sliders as opposed to having to explicitly instantiate individual pirate ships and physicalize them and actually have them do the whole you know a flight control systems are basically you know being you know tracking other you know players that are other NPCs etc to get this effect we're able to control a lot
00:25:24
of that stuff at a much more algorithmic level and to the player if you know if we get this all right they should see no difference what you should see is basically you're going from point A to point B through all of these areas that you know dictate you know certain probabilities of certain things and it very easy you know like I said we know with distance such that you may be going through you know a low probability pirate area that it's high then it's low Julius when you're when you're going from planet to planet there may be 20 different areas to where you know the likelihood of encounters kicks up and then you know in between those areas for whatever reason maybe because there are more security forces maybe because there's nothing of value in there whatever the case may be you're you're not likely to run into any
00:26:12
encounters at all and so really you can think of them as a type of optimization so that we're able to get all of the benefits of explicitly simulating you know millions and millions and millions of NPC you know characters within the game without actually having to create millions and millions of ships and actually run that fairly computationally expensive logic on the servers which in the end would take away from how many clock cycles we could devote to the experience that immediately surrounds any you know any particular blood kind of like the lod technology but for gameplay encounters that's that's exactly it that's that's really cool and so the last question I have for you is let's take a look forward specifically with probability volumes what's next I imagine we're gonna iterate upon this technology yeah the the next big step
00:27:03
will be like I mentioned hooking this up to the back end economy so that you start to get you know more you know systemic linkage of all of these different you know behavioral characteristics right now for example you have that you know the price of items can basically you know it you know they can rise and fall based upon the inventory and stuff like that but they're you know we don't have the macro system in places oh well once the price of you know titanium yeah you know or once the quantity of titanium in this area reaches a certain point then I really don't need any other more quite you know any more titanium regardless of what my inventory level is until I basically got in a certain quantity of copper because it's copper and titanium together that will be used to formulate ship holes and I need ship hulls because
00:27:50
players are demanding lots of repair services and they're demanding lots of repair services because there's lots of combat activity in this particular area and so what you what you should see ya later this year will be more more linkage of these probability volumes into you know into the into this you know this this whole systemic you know economic backend so that you start to get a lot more a lot more intelligent flow of of prices in quant tee's and what npcs in what types of you know scenarios you should see within anything in a given area according to these higher level rule says well thanks Tony so as you can tell getting three point out the door was a complex and ambitious process so let's take a moment in here from some of the others on the team the large part of our focus is on publishing
00:28:40
in the backend systems we we do all the deployments for PTU and live releases and so all the code fixes all everything that gets done by the developers in the development studios comes here for the builds and then that goes to QA and eventually becomes a publish it goes out to the players and our run up to 3.0 I think things really got intense for most of the support teams that we have here in Austin know that the QA team was working pretty much around the clock keeping up with the amount of build candidates that we had flowing through the system and every time they find a bug the developers were really quick to send in fixes and so we were really pushing a lot of builds a lot of build candidates as you know we went through a lot of BTU publishes multiple publishers
00:29:29
per week all of this is driving us toward the final 3.0 release and we would hope to put it out a little bit earlier but every time we'd get a build we'd say just a little bit more a little bit a little bit more fixes a little bit more improvements and so toward the end we were really stressing all of our systems our critical systems were at four X capacity and we were really you know I think it was there was a buzz here everybody was really having a good time we were working long hours but everybody had that energy and we were happy but of course when you when you push the systems this hard often times you end up kind of over overdoing it a little bit and so we definitely had some system failures toward the end in fact right at the very end it was our
00:30:18
last candidate we had over drived our storage systems to the point where some of the drives overheated and we lost one of our flash arrays and this was a pretty pretty critical time because that was our that was our live release and we were suddenly faced with this fact of what if we can't publish before Christmas so but fortunately you know all the all the guys in this team are really used to this kind of pressure and so everybody jumped to action and we were able to do some quick maneuvering within the build system and IT jumped in we had like six or seven drives to replace it didn't take long before we were back up and running it made a late night for us but we finally got it out so I felt really good that we actually
00:31:06
accomplished it in spite of the challenges that we had our mad he's he's like a one-man army for us once we once we had released the 3.0 immediately we saw a tremendous response and so many people came in it was really exciting to see them on the Delta patcher and and all of our graphs were lighting up with the amount of bandwidth were consuming and of course people start jumping into the to the servers and we expect a shion's we thought we had planned pretty well but Ahmed was very quick to grow the server space and the server capacity and the regions that we needed it and I don't think he really stopped over the holidays everybody said goodbye a great
00:31:54
job job well done and Ahmed just kept on kept on going I think I think he I think he really enjoys doing that I think that was probably the best holiday gift we could have given him was to be able to keep working on the servers we did have a number of issues pop up during the holidays and it's just the stuff we kind of expected some that we didn't but fortunately now we've spent enough time working on our back-end systems and our back-end procedures that we're able to do quite a bit without having to do an actual patch which is important because I think the players noticed I noticed some of the live streamers they were having a good time and they could tell that systems were improving even as they were playing and that's Ahmed and his team doing their work on the backend
00:32:42
over the Christmas break that everybody Angele 3.0 was such was such a big release that has a lot of inspiration in our end that I'm sure like on the backers end as well that we had to blend very very well for it whenever you are about to make such a large release to a larger audience across regional locations in the cloud what are you gonna look at servers in the US and the EU in Australia you have to coordinate with your club provider to make sure that there is enough capacity for what you are about to do and right before the listing 3.0 AWS services announced a new generation of high UVM's called c5 and we will write on like once we receive that the new that they have in now generally available we have already adopted them in our pipeline so
00:33:30
it was an important thing for us to make sure that the game is gonna run on the latest VMs was the latest hypervisor the latest CPUs that we can get an AWS but at the same time we are trying to allocate a large number of resources at the newest hottest generation and all in essences so we had to file out a lot of like let me increase requests with them making sure that there is gonna be enough capacity we had to have backup plans so if we're gonna have region an island I can only rely that I'm gonna get all my instances from there I could have another backup region in Paris same thing like here in the US we would have a main region in Northern Virginia and a backup region Oregon even within the region itself there are ways to allocate your requests and and your networking in a way that you can
00:34:21
maximize your availability and work around any quirks that you can see during deployment it was definitely not an easy release but I think it was the most cared for release like I think I've witnessed a lot of releases I've been being dealing with live releases I think since 1.2 or 1.3 I believe 1.2 and every single public release I was major part of it that at least was the most cared for that at least took the most amount of of half excessive changes that were done on the fly where players can really like we were also so graceful with the way we applying how fixes and a lot of situations that had to incur extra effort on us to make sure that the fakes can exist without kicking all players out would maintaining the highest stability so yeah the release started we allocated large numbers of
00:35:10
VMs because we were anticipating a lot of players now what they built a batch sure a lot of players already within the BTO from last night and we knew the moment we'll open the servers or have like two or three thousand five thousand these gonna be easy numbers because a lot of players have been playing in the btu and although we acceptable drive we would just we just keep allocating more resources building more instances and everything was going relatively good we were trying to look at the performance because the behavior that the players am oblique is completely definitely BTU when the players are never Katia or meet you for a while you start getting used to that daily thing where you walk in the game look around see what's going on that's a completely different behavior from a player or haven't seen 3.0 at all I just want to see it for the first time and then you have 50 of them in the same innocence trying to see what they will do so yeah a few hours and we started
00:35:58
realizing that we have some sort of a deadlock at the GS with the ticketing game server would get in a situation where it spins over something and players end up like disconnecting and the game server crashes and going a lot through the logs that an environment that the size of live generates is just insane like you're imagining 60 70 thousand messages every 30 seconds that's the amount of logs gets processed and indexed every 30 seconds so you have to sit through these locks super quick to figure out what's going on this must be a common a common factor a common medium why all of these are crashing or is it all crashing in a specific bet that we haven't seen before and then we realized as one of the missions causes some sort of issue with the physics that caused these years to get deadlocked and
00:36:45
in no time we were talking 27th of December but a lot of people just went home gonna enjoy their holiday but it doesn't matter like in no time I had I had the back-end service team with me I had Tony Zurich with me and we made some tweaks and that was the first half X we mitigated this crash with ultra dead and the game started and it just like Jeffrey Pease always say that's like an onion you're peeling it off and as you uncover a layer of unstability or it you get to see another one we started realizing that introduction is really really really intense and it's affecting the gameplay experience players getting through the day to the law the mana VI in the instance and we had to make changes in they get in the matchmaker and the way it matches players into innocence is we had to tweak more different aspects and the service probability volume service just like
00:37:33
responsible for doing this logic and managing it and they actually was a was a good thing when it comes to like relying on service as a model rather than keeping everything in the game server itself having a lot of logic outside the game server allowed us to make a lot of ha fixes so very long story short that took I think three or four days like I believe it 20 30 25 26 these are just one thing there's just one thing of watching a change after change and trying to see the before when certain see that the difference and everyone was there like we had QA with us we had Tony Zurich working with us chris Roberts himself was like almost everyone our making sure what's going on making different decisions changing our strategy how we try to deal with it how we look at it it was just a great thing we enjoyed we enjoyed the vacation with
00:38:22
the players watching them watching these hours long YouTube videos of gameplay watching we were really really happy with being able to play 3.0 before 2017 ends so yeah it would it was not an easy release but I don't think anything will top 3.0 for quite a while it's a lot of kind of judging different priorities and so one of the challenges was in Austin we handled the publishers so we've been when we started publishing at the beginning of October we started publishing nearly every day and so we had to balance that with other testing priorities and so I would I would think that would be one of our biggest challenges and so that entailed us to work really closely with stakeholders in
00:39:10
production to figure out the right priority and so we can make sure we can get tested what what needed to be tested on that case-by-case basis so when we do find an issue we deem to be rather serious what we do is we'll we'll write it up in ER and our project management software we use JIRA and then we'll email it out to the leadership of the team and then they can it gives the issue visibility and then people can make make decisions on whether that would either stop the publish or if it wasn't a published situation it would get everybody together to solve solve the issue we have a test request system developed by our own Chris Eckersley the
00:40:01
our QA lead and in the UK and that was as we approached the deadline for 3.0 development started utilizing that process more and more because as we approach the deadline build stability started to be more and more important and and we couldn't risk the 33.0 becoming unstable right at this root at this critical juncture so the development adopted the QA test request system or more vigorously in every single feature related check in towards the end had had to be tested properly and so that increased our workload immensely at the same time we had to continue these publishes nearly daily one of the things that we're now doing is we we actually have monitoring
00:40:51
systems now that can see what players are doing you know how many players on our server what their performance is and if we sense if the if the monitoring system sends any degradation or if there's any kind of a problem one of the things we can do now is mark that server for downgrade so that without having a serious drop you know like dumping people off of their game session that we can easily let them fade off so that when they log out and come back in they'll be automatically switched to a new server what we do is we use this as a method to apply hot fixes so we don't have to do a immediate downing of all the services and then bring them back up which would be very disruptive to all our game
00:41:36
play especially now that people have missions and you know they're really a lot more to do we wanted to do it this way so that it would really be behind the scenes you can get these fixes in well this is not all fixes but a lot of the fixes that we saw during the holiday break weird things that we could do on the back end through hot fixing in this in this manner one of the things we have seen in this release that was really really interesting is layer attention how players want to keep playing so every now and then we would face an issue with one of the services that Mike x6 700 players in and in most in most industries you know that your client is such a like in the web industry where I came if the bait started loading like half a second more or a second more that really affects like you user time people just go do something else go to eat something else that was not the case we
00:42:24
would have a crash it wipes out like 400 players and then we get bad like 600 it was just players in knew that the game is out there to be enjoyed and also at the same time to figure out bugs that we cannot figure out without these insane large green currencies that we were having thanks to the new Delta batcher we were able to get the players like and super fast time whoever fetched the BT today before we ever lose an avocado was able to get a new batch super fast actually a lot of them were trying to figure out how the trade system works building trade apps a lot of them trying to correlate bugs to each other's whenever I was watching that the servers during during the the holiday I will be watching server stats looking at different metrics and at the same time I'm watching forums watching chat watch all kind of social media and a lot of times you would get tip about something
00:43:12
happening because they manage to like do you know do you know it's not just the game here just to spend time to play the game to the Domitian didn't know the art part of the game what you're doing is the games you're building the game with us so the D start testing every single possibility and just letting us know telling oh if I if I did this and I did that I really think that this kind of chef is that the reason behind a bad performance or the reason behind this this glitch we are seeing we spent a lot of time watching the PTU and just looking for our major crashes and things that were you know players were encountering that was making the game unplayable and worked really hard to figure those out and on Saturday we felt like we were already and we went ahead and published 3.0 so in Saturday evening Ahmed and I were setting up various servers for arena commander in star marine and
00:44:01
additional PU instances and we were looking at the crashes and we saw a lot of people are having issues getting in and a lot of servers were crashing so we had to work with Tony Z to figure out what was happening and we found out that players were able to get asteroids over a tolis are still and that was causing a lot of crashes so we had to log into the QA servers and duplicate the issue and then test the fixes at one point we did have to do some major restarts but we did probably three or four hot fixes throughout the period between Saturday and the end of our holiday break just in myself and a few others we're logging in from home and checking the QA servers to see if any new the new fixes were gonna cause any issues and then we would hop
00:44:50
fix the game servers on the fly you know with Ahmed it just a different experience how close we are as usually people running doing my job would not have a lot of a lot of contact order or even knowing their end user like if I'm running a web server that serviced millions of users per day no one will ever know me you know my name have to deal with me and in star citizen it's a different experience because you feel like each one of these guys is dedicating hours of his day to come play the game this the game and try to have fun completely different like intimate relationship between you and your end user you get to know like it's a different feel it's a different field with 30.0 exactly it was much much more because we knew that that's that's a holidays players are hanging out with their families by the same time we want to see their concurrency every single
00:45:37
time we released you want to see how high can get how many concurrent players we can have what kind of issues we'll see at that scale and we haven't seen concurrency but like before people definitely are not having Christmas dinner now they are playing the game it was a lot of fun under any other project I've been on you would have a team in the in the dozens if not in the scores of people working to support this amount of service but we really only have a handful of guys but they're just so good at what they do I know they're watching every little aspect every frame rate every piece of memory they're keeping an eye on it and watching the traffic patterns of the users so one of the things that our man really enjoys doing is bringing servers up when certain time zones happen so that we have enough capacity or taking them down to save a little money all this is
00:46:24
critical work and eventually we'll have a whole separate team to do just that but for now it's pretty much all meds fun project that initial published to the evocati was our first step into getting 300 out the door and that first night when we were deploying we actually had a six-hour Skype called with a deployment team just to make sure that everything was going the way it should the reason it took so long was that we really hadn't deployed with this new launcher and with a bunch of the back-end services changes that we had so we wanted to make sure that we got everything right and we updated our checklist to make sure that you know everything went smoothly and we can monitor the server's as a player started to come in well it was just so many changes being introduced with three oh right the basically the entire game had
00:47:12
changed all the backend services had been rewritten it was a brand-new launcher so we wanted to make sure that while we tested it for many many many weeks with QA prepping for the evocati says it's you just can never predict you know you'll give it to one backer who has some weird setup and it just takes the whole system down and or encounter some weird issue with the release channels so you know we just want to make sure things went as smoothly as possible after the first evocati push we had just dozens and dozens of deployments and you know now that we had the Delta patch sure we could put out builds easily every day and just iterating with the feedback from the evocati and then ultimately from the the public PTU feedback we were getting and just pushing out bills as fast as we could with as many fixes and feature deployments as we could once we hit the the public group things changed a little
00:48:01
because the deployment process from evocati is fairly simple we do have a much faster checklist just to get the build out to them once the build is going to the public audience the QA checklist actually ends up taking a little bit longer to make sure things are tested a little better and then as we moved into late December and started heading towards live we know we started doing more and more checks and having to recheck some of the older systems that maybe had been neglected for a little bit cuz we were just pushing out so many fixes and so many you know new features into into the build that you know like we had to double check like hey arena commander was working a couple weeks ago let's do another you know deep dive into it to make sure that nothing broke in the meantime you know and then that actually ended up running into that Saturday and you know December 23rd
00:48:50
where we a small team came in on Austin to deploy on that Saturday when we deployed around about 2:30 ish Austin time on on December 23rd that's Saturday and then immediately we were monitoring all the the core services and the game servers talking to players make sure you know things were working okay there were a couple issues with some of the the missions or interdiction that we found immediately and the great thing about some of the work Tony Z has done is that all of those back-end services run you know on our side to where he can make changes to the code the missions you know interdiction timings or how often it happens and we can deploy those hot fixes on our end and then gracefully restart all of the servers so that
00:49:37
players don't even really know anything's changed but things get much better so we actually ended up deploying ha fix is almost every single day of the Christmas break starting from the 24th all the way through I think January 2nd Christmas night was pretty interesting because for the most part the instances were fine all day and then all of a sudden things started to happen and we were you know logging in remote and a bunch of us were yeah we didn't really do any calls but we were all in Skype you know just you know trying to figure out what was happening and I think it was like 3:00 a.m. and some people realized that they needed to put Christmas presents no to the tree and but you know we were all watching the servers I was providing a you know FPS information to Chris and Tony providing you know other graphs to show them what the fixes had how the fixes had improved
00:50:26
the gameplay there are things the players can do that will sometimes cause the instance again a bad state that we can see that in graphs and we can kind of predict that and fix that before we can make an instance set so that it nobody will go into it and it'll slow it once everybody's left it'll restart itself and it's you know we do a lot of things to make sure that the players aren't affected as much as possible so what would happen is we'd identify an issue so for instance the that first day on the 23rd there were asteroids popping that was actually an environment mission that was spawning in places it shouldn't be spawning at and causing all kinds of physics grid nonsense that would that I was just causing massive FPS drops right and so what Tony ended up doing is just removing that mission from the actual
00:51:15
service and we just rebooted everything and we have a way of we can ungracefully you know force everyone out of the servers and shut everything down and restart it but we also have the option to gracefully restart services so what happens is Jim will look at what servers are having a population declining and once the population of a server reaches zero it'll just reboot that one so we'll actually have players experiencing a slightly different game at different times because not every instance has been restarted but then over the course of the next you know 24 hours all the servers will just naturally restart and then they'll be running on the better code I would say that right now one of the the biggest struggles we're having is that the connection to like our database and our persistence cache kind of can-can fall over and when that happens there's this short window where
00:52:03
if anything if a player does anything while that service is down it just causes all kinds of wild inconsistencies with the data and this will often end up with the player accounts becoming broken and we'll have to go in as game support and say okay I can reset your data here or you know we can fix this unfortunately it means like a lot of players lose out on missions or their cargo gets lost and it's unfortunate but we already have already identified most of those issues and are working on fixes we're doing something that nor the game I've worked on is doing its you know this combination of MMO simulator FPS all in one giant seamless universe our players defective it's like you know that's that's crazy how are they gonna do that then our QA experience it's how are they gonna do that I want to understand this and so it's been a good learning experience
00:52:51
a lot of fun the team has just been doing such an incredible job I'm just so proud of them from the I had such an accomplished and capable leadership team and from the QA managers all the way down to the QA testers everybody on the team in their own right is a rockstar and together they just successfully support the development of the team and I'm just really proud of that I think there's a lot of people that realized in in at least with this particular publish cycle because it was such an important one for us that all of our process kicks in when that final build comes for that final publish but what people don't remember is that this team is doing that constantly every time we do a publish to the PTU it's pretty much the same
00:53:39
process it's just two different servers with different security protocols so for the most part the team is doing this every day and you may only see two publishes three publishes a week but we're actually running the process for a published candidate every day it's not until the very end when QA tells us whether or not we can actually release that version or if there's some critical fix that they're waiting for but the by then the work has already been done so we're constantly publishing constantly exercising the systems and this helps us to really clean out any of the rough edges and smooth things over so that when we do have a major 3.0 publish it really goes down pretty much like clockwork in fact I remember one of the things that it wasn't really frustrating but I just noticed that it happened is
00:54:27
we have a very rigid checklist that we follow to make sure that everybody does every step in a proper order the team was publishing so quickly that night that it was hard for the production team to even keep up with the checklist just because things were happening so rapidly well that's it for this time hope you enjoyed the update and we'll see you next time so the Austin team is going to be busy this year because we're planning on delivering content enhancements and refinements in a quarterly release schedule which I think will be pretty cool for an in-depth look at what's been happening at all our studios be sure to head to the RSA website this weekend where you can read our monthly report for December yes and that's all for today's show remember to check out the squadron 42 web page as
00:55:15
well and enlist to receive the newsletter else you won't receive it yes so definitely a list and thank you to all our subscribers your support and feedback makes us and able to be able to do things like this show around the verse as well as the series that we're gonna do like Christmas remember to tune into the premiere tomorrow at noon PST and also the tumbrel nova concept remains on sale through January 15th so there's still time to add this bruiser to your fleet yes there is and of course thanks to all our backers who make the development of star citizen and squadron 42 possible we won't be able to do this right you sir thank you very much we would not and until next week we will see you around the base [Music] thank you for watching so if you want to
00:56:26
keep it with the latest and greatest in star citizen a squadron 42 development please follow us on our social media channels see you soon

Transcript from YouTube’s automatic captions — punctuation and Star Citizen jargon are approximate. The video itself is hosted by CIG on the official channel.