Star Citizen: June Subscriber's Town Hall feat. Engineering Team

29 June 201600:53:151,419 linesWatch on YouTube ↗

Transcript — every row opens the video at that moment

00:00:00
e welcome everybody welcome to another
00:00:57
special subscriber edition of reverse the ver reverse the verse reverse the verse I'm your community manager Jared hucke just back from vacation welcome everybody oh I'm going outside my box here hold on what is that all right uh with us today are members of the Austin engineering team uh we've got with us Mr Ahmed Ahmed say hello to the to the nice fans hi everybody ahed uh we have Mr Jason say hi Jason hey and Mr Tom hello Mr Tom I'm still saying Mr first names because I'm terrible with last names but is it it's it's it's Tom Sawyer it's Jason Eli and El see that's why mad with things and Ahmed why don't you just say your last name it's Sher Sher yes Shaker there y see that's why I don't even
00:01:47
bother trying I don't even know why I tried anymore so uh if you if You' never watched one of our special subscriber D Shar TVs uh once a month we do a special broadcast anybody can watch but we take questions exclusively from our subscribers the subscribers are the subset of our backers who subsidize all of our broadcasts like around the verse reverse the verse uh Buck Smashers the monthly jump machine itself I'm sorry oh we we are having an issue hearing we can't hear you you can't hear me at all all right yeah go ahead and cut it um sorry Tom was asking about the air conditioner uh basically they they the subscribers uh subsidize all of the output that we put out to the community um and this is one of those things so if you're
00:02:35
interested in asking questions to the engineering team we are taking questions exclusively from our subscriber chat you can go to robertsspaceindustries.com community/ chat and open up the subscriber tab uh of course you have to be a subscriber to have access to the subscriber tab uh for any reason you're not a subscriber and you'd like to become one you can check the uh the main menu at the top of the website for information about how to become a subscriber so why don't we start uh why don't we start talking a little bit about who we are and what you guys do so folks can uh can uh tailor their questions specifically to you guys because there's there's a lot of misconceptions about what an engineering team does uh a lot of folks were were calling you the devops team earlier you know in the week and stuff like that so let's let's start the broadcast by clearing up a lot of misconceptions uh before we get started so Ahmed why don't
00:03:23
you start us off and tell us a little bit about what it is that you actually do for cloud Imperium games so yeah um principal Dev Ops engineer mainly I take care of day day-to-day operation of all game Cloud infrastructure and uh I take care of the automation configuration with a large team uh we have I I think most of the subscribers know a bunch of the names we have on a team we have miles Kagan Andy Nate Jey P Gerard that's the def Ops Team I'm here to presen my team not just myself and yeah we we are the people that run all the cloud infrastructure and take care of the builds that you guys get to see and play all okay and where did you come to where did you come to us from you where were you I came from the web industry this is my first gaming job I I had gaming CL like game Studio clients before but they were
00:04:11
mainly doing like web uh games uh I came from operation and a startup uh in Egypt called spir systems we were mainly uh supporting uh startups struggling underload our our big biggest bigger segment or like the biggest segment we had of clients were mainly people who have very high traffic in a very short period of times you know how to manage their scaling they want to move from being in bare metal to virtualization to Cloud move between Cloud providers so yeah my background was mainly in in the cloud and the scaling area automation all of that and when I Saw the J posting a very high fidelity MMO that needs to run in the cloud that was very interesting the statefulness in the game all of that is completely different from anything in the web so um I I thought and I was correct that it will bring of
00:04:59
like some crazy challenges that I've never seen before and here I am have been here for uh a year and 3 months almost gotcha we'll come back to that how uh M an MMO that runs in the cloud is is is different and an interesting challenge want to get to the other introductions here Jason uh tell us what you do for for cloud Imperium games um well I'm in charge of the architecture of the backend Services um pretty much on a day-to- day um you know we identify what the you know what game features need as far as back and services and um you know help design the proper services and interaction between those in the game um you know pretty much just make the world work on the in the back end right the things that people don't see we're kind of underappreciated in that sense but when something happens we're the first ones to fair enough uh can you
00:05:49
give us an example of what a backend service is uh yes like a friend service um we have the we just you know we have persistence now which basically is in charge of of uh bringing uh items from the game servers into the database and um you know that sort of thing uh we also have uh the general instance manager which is um essentially in charge of spooling up and shutting down game instances and managing game sessions okay and that's different from net code yeah uh Network code is more between the clients and the actual game server the game server would be you know like you're Arena commander game sessions or when you're actually flying around in Crusader um that's the more real time um you know twitch type of play um the backend Services is more um you know it's it's um the stuff between the servers as opposed to yeah command
00:06:38
and control yeah command and control gotcha all right and uh Tom what do you do for cloud and premium games uh yeah I work with uh you know Jason here I'm a senior server programmer um as well and uh yeah you know just uh working on the back end as well just uh with all the services that we have in the back end dealing with a lot of the command and control tools that you know we give the QA guys and live Ops Team you know so that we have you know some idea of what's going on in real time both in the public test realm and the live and things like that and of course anytime there's an issue uh you know I log in and uh you know I'm you know doing heart surgery on live servers uh you know in production and stuff uh behind the scenes you know uh sometimes you know there are issues uh when we do a fresh deployment something that's been you know built up uh between like a 24 and a
00:07:26
241 and things like that just uh making sure everything works so yeah got yeah the three of us here are pretty much the primary fire fighters we're the first line of defense in most cases yes something goes wrong it is very common in the uh 24 to 48 hours after a new patch goes out to to see you guys in Skype at all hours of the night yeah yeah so do doing some uh very risky stuff but you know the The Show Must Go On you we must uh keep these servers up and running the best we can so absolutely and Jason where where were you before Cloud Imperium games um a number of places let's see I started in the game industry at um origin working on Ultima and Crusaders um and then I actually have my started my own company Asylum software I started making my own little MMOs uh Elder lands and lothar online and these little they small boutique uh type of
00:08:15
games retro style things so nothing on the scale of this um and that's where I got into actually you know building you know massive online worlds and such so this is really attractive coming over here to this it's on such a grand scale um I also worked at Qualcomm as a a server engineer over there um building um you know high performance Network applications all right and how about you Tom where were you before this um I was I was with the Sony PlayStation for almost a decade in uh San Diego um I was with the online Technology Group and uh we built up uh you know network library libraries and server architecture for first party and second party Studios and you know by the time I left uh you we shipped about 160 uh Playstation titles using the framework um I mean I have to admit you know I'm more of an FPS guy
00:09:03
before I came to you know Cloud Imperium games you know you kind of do things a little bit differently and now that you know uh we're in MMO there's a whole different set of uh problems to solve and uh that's where Jason is you know really really helpful you know just having his MMO experience and uh persistence you know itemization itemization is you know in my mind the most technical complex part an MMO just you know managing just all these items that can be traded with other players you can purchase them on a website and it syns up and um you know trade yeah tradeable in mailboxes and shops and stuff so and this is a completely different Beast as far as itemization in an MMO I mean we're on like orders of magnitude more yeah than other MMOs so
00:09:51
you know this uh this uh you know 24 release is huge uh there's a huge investment uh to get ionization done right with persistence and then of course you know when we're talking about persistence we're talking about account based persistence you know your player your your stuff your ships and things like that but we're going to be growing over time as soon as we have like uh you know instance persistence and you know more of a worldwide persist yeah there'll be there's so much more persistent data coming but the most important right now is to get an itemization for players so when they they're in the game and they acquire items when they change the load out on their ship if that stuff persists between sessions between um build now persistence is something that it's fundamental for every MMO we we talk about it a lot like it's like it's a major thing for us and and and it is but
00:10:40
I it is a it's like tires on a car you can't have an MMO without resistance so one of the reasons it's such a big deal for us that we talk about as much is because no other MMO is out at this point in development no other MMO is is publishing you know an alpha prior to the creation of their persistent you know database and and the cash and all these backend systems and what so I mean we hear we hear a lot of chat from the community about you know why is persistence such a big deal and why not it's it's it's a big deal because it is a it is a huge milestone in our development it's something we always knew had to come it's it's something that's been on the road map since the beginning but because we're building this game out in front of people we're building we're allowing
00:11:28
them to play it as we build it and whatnot they're experiencing this Milestone that every MMO goes through at some point but they're experiencing it as it happens instead of months and and in most cases years after this Milestone happens in development uh what can you tell us about building persistence while everybody is sitting there watching um well I guess I'll say that it's um it's something that's definitely needed people don't don't realize you know the importance of it it's it's not a sexy feature to promote you know for pr- wise because it's an expected feature it's not like hey we got this great new feature over here it's got a lot of blinging you all are going to love it it's it's more of an expected thing but it takes a huge amount of work and um you know um we had to be very careful on how we brought it into the
00:12:17
universe we had to bring it in at the right time we had to make sure a lot of the uh game design was you know the itemization and such was was kind of hammered out and you know it's never 100% you know like we still have a lot of data that's yet to be designed before we can persist it but we had to get the players inventory their ships or entitlements uh you know their pledges and such we had to make sure that there's a lot of things locked down with that before we could even begin a strategy on how how to persist it because there's a lot more that goes on behind the scenes especially with this game considering that we have you know your average players going to have thousands upon thousands of items rather than hundreds has NE MMOs um we had to take uh very specific approaches and tailor the data in such a way to where we can't we don't get into situations where we're bogged down we have throughput issues and such like that so
00:13:04
it's a it's definitely an all Hands-On de feature because uh you know not only do the server programmers have the backend uh to to manage all this data um you know for you know potentially 50 to 250 you know thousand current players and just managing all this data and stuff and with our caching Services um but you know you got the gameplay guys you got the UI team you know that have to you know have visual display plays inside the game so that you can manipulate uh the different items and move them around and stuff and uh get those uh you know changes you know taking place pushing back up to the persistent cach service and then we got these old Delta timers actually right to the database you know we've got uh you know these these middle tiers business tiers you know to to manage you know the items uh so that we have complete control of when rides to the database if
00:13:54
we didn't have those middle tiers and you know oh my God we would we would blow the database driver out water with all these you know transactions coming through right we have to have you know these different tiers because um one particular game server with may have like 30 people or 20 people on it whatever and a lot of items are changing and being sent to the caching layer um which holds the data for certain amount of time and then we'll write to the database in a controlled in a controlled manner but when we have a 100 game servers you know that that becomes more problem so we have to have multiple layers of throttling and caching in order to um um have complete control over the rate at which data comes into the system and into the backend system and and into the database in a way to where we don't kill the database and plus the auditing uh tools you know so that customer uh support and uh you know
00:14:42
QA and all that stuff you know for every item in the game you know we can get a history of where it was at any given time when it got traded when it got sold to a vendor when it was purchased or you know deleted from the website and got resync you know with the universe you know cluster things like that so there's a lot of command and control Tools in place is to you know manage items so now Ahmed you talked a bit about uh building MMO in the cloud and how that's different than uh talk talk to us about what what makes Star Citizen in the cloud such a unique thing so not not to get too much back when when when the cloud movement started people said it's not for everyone you cannot just take your app and put in the cloud in a VM because it doesn't matter whether you have it in a very have it in
00:15:29
a VM in a small kind of hosal Provider to be in one of the big public clouds you have to use the other abstraction layers that the cloud provides to you the cloud provides to you object storage block storage that can move between VMS VMS can be migrated different concepts of firewall different concept of vbn and everything is offered to you as a service that's a lot of good work happening for you saving you and your team from doing all this tedious work that's technically inventing the wheel however you are relying on uh resources shared among other people so every single public CL provider tells you you got to design for failure expect me to go down at any moment expect this to happen expect this to change and the statelessness so in web websites most of the time people
00:16:20
will try to obtain what we call like there's no State get within the firal resource that might diet any moment and I have the state saved outside in redundant form and can be accessed in different ways which is the the basic concept of distribute systems having this in the web is like being a first class citizen in a country that talks your language everyone does this and you look at everyone other Playbook and steal a game from him and just do it and people go thre him towards that games are the complete opposite they are very very stateful an MMO is extremely stateful it's not even match based it's not session based so you expect that you leave your machine running for 24 hours if you can hold standing out for 24 hours playing and you want this to
00:17:07
happen and also you're not going to tolerate having a 404 or 5002 that you see in the website in the middle of that huge epic game where you get like your amazing Lo whatever you are looking for so given this sort of a stability in the cloud is not really easy and dealing with the cloud quirks because yeah the cloud does a lot of good work for me but also takes a lot of control out of my hand I could oh I could have made this thing worked that thing in this way and stopped the password here and only replicated this half and I can't I'm using whatever the struction layer the club provider provided me and the tools I'm using just like anyone else so the challenge is MMO games are an alien in the cloud while all the web technology are the first class citizen so be being this kind of alien that no one else is
00:17:55
doing what we so we have been watching a lot of companies when Z go and talk with with other tools other conferences talking about their Cloud adaptation not a lot of people really manage to have the simulator which is the dedicated game server in the cloud efficiently having everything working fine and that's the challenge there are people that did what I mentioned in the beginning they took their original data center bare metal infrastructure and carried it and put it in the cloud on DMS and it's the same exact way that's not what we're trying to do and it's a challenge because as I mentioned and uh to get a tiny bit technical if if you want to use a new tool too late too late if you want to use a new tool in the cloud or a new service in the cloud and you you C in a modern language and your use case is just create like a CR
00:18:44
something that you create to remove records and all of that you would find an example within five minutes and you'll look at it and you just saved your yourself a lot of time and then move from there using a C++ using a game that's a different like thing to deal way now you're talking about challenges uh we have some plans in place to address these challenges uh some some of them we're we're not quite ready to talk about yet um certain reveals of certain things we like to leave to Chris and what to to talk about when when the time comes but uh in the in the broadest sense in the stuff that's not going to get us in trouble tonight uh what can you what can you tell us about uh how we're facing those challenges yes so the way we look at it we want to implement what's currently being known as the
00:19:33
event driven data center how to have so the more I provide what we call sometimes fluid resources the games that they can move I deployment time like they call these Main buzzwords in our business devops the time to Market how fast would it be for me to scale a game server for you how how easy would me to retain resources I not need how can I repurpose resource es uh because the cloud is mostly a utility computing you pay for what you use M uh and that's a completely different approach then you own your data center and built it and now these stuff are yours and then he use it you not use it that's different story so we back to my original idea I was talking about we we want to implement the event driven data center that every piece of of of software that
00:20:22
goes through a major event that the infrastructure should know about would find an event through a a a wide channel of info buses that have reactors reacting to it and acting upon it that's that's the big picture so and we can orchestrate this the way we want because orchestration is is the is the word here that's that's what you need to know you need to make all the all the logic that all that Jason and Tom and the gameplay guys are writing react or like report what it needs and my infrastructure reacts to it whether by expanding by replacing by oh that's that's that's a bad uh member of our cluster we should take should take that oh no this is not going to get Auto healed or self healed we got to walk Amad up THS up that's the plan uh move to that
00:21:12
and uh the way we usually do it in div Ops and even in backend Services shops you cannot really sit in a room and get this amazing big white paper and draw everything it's impossible you will definitely have bad designs so we go through the MVB like the minimum value product okay this this will deliver my idea I'm going to scale right now or I'm going to make the players get in the game and watch because maybe there might be a new feature I'm not really putting in my mind or maybe it's it's the same concept that all the successful startups followed like how Facebook believe that you should move fast and break things however you until you go out the market the market correct you that's also goes really correct your point in the beginning about persistence if we laid out persistence before seeing the game itself and came the game with a get going we'll be just going breaking walls
00:22:01
and moving around them while right now we are more mature we can start from a point where we understand this this is what the game looks like the game designers had their time this discussed some ideas bounce it back and forth it the implementation is going to be a lot easier when you have something to test way at one point we didn't even have any of the game data design we nobody knew what we had to persist for example so we had actually go through several meetings and to figure out what are we going to persist and how are we going to do it y i i remember those meetings sitting there like no he go the players have this and the players have this and and this needs to sit up with turbulent and this needs this this needs to be P pulled down and and live in the universe so oh yeah and there's a lot of transitioning on ownership and communication of all that data yeah I mean that's still evolving today absolutely I mean there's still game
00:22:48
systems Yet to Come online that will have to be integrated into persistence as it goes um we're going to get to the we're GNA get to your subscriber questions in just a minute I want to ask uh Jason and Tom one more question uh follow up on you were talking about the scale of the game and how the scale of Star Citizen dwarfs things that you'd worked on before uh talk to us a little bit about the scale of Star Citizen we can start with itemization um like a game like World of Warcraft for example you can have on the you can have hundreds of items on the character and that seems like a lot yeah a lot of items to keep track of and so on and store in your inventory or in your bank and so on where in Star Citizen just with your ships you know the things that you buy on the website right now you are already in the thousands and you can easily have players with uh you know we're going to
00:23:36
have north of 50,000 items and associated with one player um that's a scale that I've never had to deal with before but it's um you know it's something that we're you know we've been able to put a lot of thought into and figure out the best way to deal with it and you know and it's still going to evolve because you know at some point if we you know have a million online at one time that's a 50,000 per player that's a lot of data to move around so so you know before persistence you know our our architecture was very firstperson shooter like um you know coming from Sony uh the initial server cluster kind of looked like you know uh some of the models that we had over at Sony um you know where we had 250,000 concurrent uh with the simulators um you know with that type of framework uh we know where to scale horizontal if we
00:24:23
break from 250 to a million uh we know what we need to Federate next so um taking those principles uh for from selling and applying to cig um you know we can we can most likely hit those same numbers but once you add persistence then we have to you know go back to the drawing board and you know figure figure out where the bottlenecks are going to be what services are need to get Federated you know horizontally and things like that so yeah yeah like we've uh we we're writing all of our um services and C+ plus just for the performance and um we can you know any any given service can handle a large number of players and a large amount of data passing through it um so one of the ways we're going to scale like Thomas saying horizontally is we have to there's two things to consider there's you know we have to uh load balancing and then um availability so we
00:25:13
have to make sure that if any service goes down that you know there's something else there another one there to pick it up or whatever that there that the service that that that this the service is always available no matter no matter what so we'll have multiple instances of a particular service up at any given time and if one goes down there's another one available and so on the load balancing aspect would be um just to distribute Pro nope across multiple instances of a particular service definitely want to apply the uh the the mega server um you know methodologies you know where you have a single login and just like a single universe that you um are in where you can play with anybody around the world rather than fragmenting up uh you know the player base on a ser server basis that once you lock into a server
00:26:00
that's it you know you can only interact with those people so we're trying to you know design that upfront um so with the mega server uh architecture in mind for both uh you know persistence and you know the regular matchmaking and uh persistent Universe uh Concepts yeah and the the services themselves right now like players and objects have a location so we already view them as a location in one universe and there's you know other technical issues that have to be addressed with the game servers and and interaction in the instances and stuff like that to um to really make the players feel like they are in that one large universe but we've already been we've already been fighting that battle on on on the back end and uh you used a buzz word just a few minutes ago and I can already tell the questions are coming in what's a mega server um one giant world it's essentially a collection of servers of
00:26:49
services working together to give the perception of one world one server right rather than like World of Warcraft you know we got a list of like 250 servers that you know once you join you know you're locked into that yeah and this is pretty interesting it's an interesting challenge because we have to not only deal with us players we have to uh you know consider people in Europe and in Asia and South America and so on so we Australia don't leave Australia out I'll I said don't leave Australia out I'll get post oh Australia yeah it's it's a global Mega server so we have to make sure that the players feel that you know they are there and that they that they can play with their us friends they can play with their European friends whomever you know then it feels like your next door you know you're not transitioning into another realm type of thing gotta all right guys
00:27:38
so we'll start taking questions from our subscribers uh again subscribers uh if you wish to ask a question of uh Ahmed Tom or Jason you can type your question into the subscriber chat on the RSI website uh please preface your question with the word question surrounded by brackets uh it'll help me pick it out through the uh through the chat we are not taking question from either twitch chat or the regular General chat on our website strictly through the subscriber chat uh first question up is from China shop Rodeo he wants to know have you ever had a typo break everything uh similar to inst instances where DNS configurations have brought down the cloud uh to Chrome pointing entire workflow workflows to some other website I did have one of these none of us never had one of these yeah one time
00:28:26
so the the way we deploy our environment is very well known in the cloud and the devops industry by the word green blue deployments that we would have a full environment running whether it's BTU or Live players can see it and then we St we stage a shadow environment complete full and then at the moment we just switched the little balance from the BL it moved so um I can't remember was it 22 I can't remember it was and I had to edit very very large file with a lot of different I mean we are an alpha project so sometimes we broke our own rules and you would I'm gonna test it right before I do it yeah yeah we have no time um the thing that we have in they Ops and I think we have an Austin we deliver whatever the company needs to be delivered as quick as possible and when never it needed to be delivered so yeah
00:29:14
it was one of these things where you hack something really quick and I have an extra space in a URL that goes for a platform Avis uh in a very gigantic dictionary that will generate a lot of configuration that will go to the services and this space led to issue with authentication so QA was testing trying to get in and nothing is working and it it took a while that the edor message wasn't very very clear so yeah the these things always happen but the good thing is we did not do it in production production we had it in the staging one so we get to fix it and see it and I'm sure like every person running a live production servers all the time get to see some of these human errors yeah all right uh jiru has a question he wants to know if you use common framework like Springs or struts on the
00:30:02
back end uh if I remember right spring and struts uh you know was a you know Java jav J2 framework uh our our universe cluster is primarily C++ uh with some thirdparty libraries uh that we customize but uh the web platform on the otherand do use a lot of web- based you know Technologies and stuff so okay uh catastrophe disastrophe a combination of disaster and catastrophe disastrophe good name uh what configuration management tools does the team use to administer configure servers in the cloud also what kind of log analysis management tools are used yeah um I'm not a huge fan of of of focusing on the brands but like yeah we uh we
00:30:49
currently use chef and we use bunch of our uh home rou scripts that mainly does the provisioning so I'm gonna try to talk like in a we can bring up we can bring up a graph that would explain a lot of my work and a lot of what we do what you mean you had a graph ready SC and I picked this question just for this what we didn't rehearse this and pull up this all right yeah that's so if we start all the way in the right you guys can see Aus that's where everything is being built current build system is based on a build system called buildbot uh it's highly modified uh and customized to our own needs the way it
00:31:37
schedules uh and and manage the Fanning in fanning out all dependency everything in the in the build system is is custom made for us so the outcome from the build system will be a build which is all the binaries needed to run the client to run the server under the back end services and a bunch of other files they get uploaded to object storage in the cloud as you can see still on eyy and from there now we have the build we want to deliver it as fast as possible our current cloud provider gives us uh a good Advantage here that we can create an AAL box at this moment and we pull the bill down and we do whatever we want to do in the O at this moment like install packages make sure that we change your limits make make sure that everything is optimized the way you want
00:32:25
it and then we create snapshots from both the this prodct takes a few minutes usually 20 minutes 25 minutes it high depends on the when link content I mean how who's fighting with me over uploading something it could be me too uploading somewhere else so yeah this is mainly like internet speed kind of limit and then we get the snapshot the snapshot can create desks and these desks can be shared across the whole cluster so if we scroll back up again now you understand that the build existed in the snapshot you as a player sitting here that's your game client you start your launcher Batcher and your launcher Bacher would go grab the build for you from the CDN the CDN or the contact delivery network is bunch of uh endpoints uh globally distributed trying to deliver you as
00:33:12
fast as possible the files that you need and you pull them down in your game client now you have the Bild the Bild needs to connect so you you would have some connection between you and the Hub low balancer that takes you to the hub servers all of them these so you have bunch of different Hub servers that handle your connection you go there over TCP The Hub servers go to the platform make sure that you are the guy who you claim you are and you have a connection with the platform from here the Hub service will take your traffic to most of the services that Tom and Jason did here in the core server they live in a single box and and we can say why they live in a single box for now and one of them is the persistence Services we have two main Services persistence cach and persistance service itself as they
00:33:59
connect to our mySQL database clusters that has Masters and slaves and back having all these schol stuff and here you set all you dedicated Game servers and different game servers all these boxes I drawn uh represent an environment any Cloud environment right now in Cloud andum games has to be three different VMS with three different types of rules one of them for the Hub one of them for all the core services and one of them for the game game server this is not any way near final that's what we need now to get everything running and see how we want to scale and change of stuff uh the difference between environment between QA BTU on live QA is one one one BTU is multiple of these one of these multiple of these live is much
00:34:48
larger uh the question that he asked also relates to these bunch of servers we have our infrastructure servers we have log aggregation systems we have monitoring systems tools for diff support and tools for analytics uh we use Chef here we use bunch of home RO scripts here and over here we have a Splunk we have uh a graphite graph statsd uh we have a stack driver uh we have bunch of uh uh core dump like analysis to look like we have bunch of servers have GDB server for developers to look at we have some stuff related to Google break I I I want not like I know that we have a lot of backers who came from technical background and as H posts so I'm trying to keep imbalance where like I use a lot of technical jar and
00:35:35
then I go back to something high level so yeah this is not any weird final actually right now if I mentioned that we are working on a new deployment by line that would be the way that been driven the center I was talking about or the scaling in and out or like out yeah in and out would would be like easy and and usable under heavy load all right turn this off very cool let's get you back on screen wrong buttons all right so thank you for having that graph all all set up on uh our next question is a is a very uh popular question about patch reduction sizes uh what can uh Ean Dragon wants to know if you can Pro uh
00:36:22
what Insight you can provide as to the progress of uh patch reduction the the uh the system that checks and and sends you only the files that you need instead of everything else I want what can you tell us about that system yeah that this project is running lots of caffeine lots of people across the world this is one of the biggest projects that we have and it involves people from so many different uh departments we have people from Devas we have people from it people from engine programmers people in the platform everyone is working together to have our own Bacher and I also like the launcher system or the new Bach delivery system is not going to only affect the players it's going to affect even our internal developers the way that they deliver receive their own builds actually our internal developer has kind
00:37:11
of an educ case the real to the player outside in the world because you pull a lot of builds all the time so yeah work work work is running I see I see all that the that change list coming in and out I see all the commits happening we get to have meetings every now and then to make sure that we are all still have the same features that we require from from the from the whole system it it's going to be awesome I mean our our so we feel really bad that when we deliver three builds for you guys in one single day you can only play the last one because you by the time you finish the first one and we want to give you this seamless kind like I'm a devops guy and I live for the world that that has my TSA on all what I care about is giving you the content that ready for you as fast as possible so having a smaller batch is a key for all of us MH yeah it's been working yeah it's one of
00:37:59
those challenges again because we're operating a live environment at such an early point in our development you know any other game would have you know persistence would be in the the this this low-level patching system would be in all this stuff would be in before they even started their first Alpha so um and and uh it's it's Mike is it Mike picket who's working on that you yes Mike picket Mike picket was one of the very first people that was talking about this issue having his own design on it at I like it extends it extends to a lot of people because that that the way we are looking at it to do everything right you have to have this system native and all the end points that need to use it so every part should be aware of it to deliver this kind of experience we like I'm downloading now the game is playable then it can be a complete all these fun
00:38:49
amazing good stuff need to be done and the scale always the scale is the key that you need to do this right you need to do this that that's not an NG game I mean we we are yeah it's a startup but we understand how big our community is so when we build a foundation technology like this we need to look at all edge cases make sure everything will work even if we have 1 million people downloading right now because we don't not really want to be the ones that have our own success our our game is very successful by it is right now and we are here just to serve the success gotta yeah it's it's I wanted to show throw a shout out to Mike you know we have a very large team uh with our video content Community only in one place and so now in two places now we have Austin uh we only get to Showcase a select few number of faces you know week
00:39:36
in week out but we have we have a a huge team of people working on just about every aspect of this game so we like to like to give shout outs wherever possible so Mike good work I've never met you I've never even talked to you but I know your name and I know your work I know your working your butt off y so uh so thanks Mike all right what else do we got here um we have a we put up a thread over the weekend to collect some questions early um there was one that we have to ask where was it uh something about Ahmed being awesome I can't find it yeah there it goes C droma asked this question specifically for Amed why are you so amazing um it's the glasses I don't know uh Laura turnis asks what color network
00:40:26
cable is is your Nemesis he says his his he hates the light purple ones he hates the light purple Network caes he brings up good memories like dealing with with stuff that I can touch with my hands like stopped a while ago like a few years ago when we had the to wire cables and remember which wiring and what colors we hit yeah that that's how we start nowadays we forget about all these stuff and that's the beauty of the cloud we tell us stuff to happen and they happen you don't all always happen the right way and that's our job to make sure always come out the right way uh but yeah I don't know I can't remember I didn't I didn't wire a network cable for a while I didn't touch an rg55 this for a yes no I I I I ran my own IT company for a number of years and and absolutely whenever possible I got I got some intern or somebody to wire my
00:41:16
cables cuz I I just getting down into the nitty-gritty and then remember try to try you you do a on one end and then B on the other one you're like what the heck was you know you start questioning you start thinking moment happened after 20 actually this this bringed up that that talk about the cable a few years ago none of us would have imagined that was one single person that you'll be running 1500 cores that's the beauty of the cloud because we're talking about the cloud a lot when two went up I'm not going to talk about concurrency but there was a lot of cours running under Star Citizen name at our club provider and having this over seen by one or two people is is how cool it is that we don't really need to worry about the cables the TRS all these stuff at this phase uh Daz asks a question that we're not going to answer but I want to ask it
00:42:06
uh to to explain why we're not going to answer it because this type of question comes up quite often Dez asks hey guys in some M MMOs bought armies that set out to grind currency become an issue what is being done to prevent exploits farming from being an issue um whe we get asked a lot what we're going to do to prevent cheating and stuff like that uh we don't answer those questions because we don't want the bad guys to know what we're doing so I it's I I I understand why the good guys would want to know uh what what the things that we're doing the details and how we're going to go about it it's just one of those uh catch 22s where when we the more info on those things that we put out the more info the bad guys have to circumvent so just know that Jason's got a lot of tricks up his seve if you know based on his experience and so the best thing to do you don't answer the
00:42:54
question but you make them you know you make sure that they know we're watching we know what they're going to do and make them paranoid so we will catch them yep we know who you are Robert oh yeah command and control command and control um uh we hear a lot about uh client frame rate and server frame rate yeah is that something you can speak about what what what dictates a server frame rate and how that can affect I can so there there's not a clear luck point I mean that that's that's not my area of ownership of work that's the area I'm like very first close neighbor to I do believe that we have a very very specific lock step but this is what locks the client right right like the client frame rate is usually the rate at
00:43:42
which you're refreshing your your your the graphics and so on so um on the server side on the dgs we call it the dgs um that's the simulation rate and they're they're not locked together so okay all we can yeah absolutely and I know a lot of players are concerned with the FPS and uh I I was planning I first of all I want to ask every single person who plays BTU and all the amazing people who play avocati and the great people that I woke up during the night is like hey I have a new combination I wanted to stay and people calling each other and coming online this is amazing I haven't seen anything I can imagine this even possible in my interview a year and a few months ago I was asking ah two you guys teaching me about games how do you guys just test it's like us people it's like what do you mean it's like our our Gamers our players are awesome I can't
00:44:31
believe that we'll be able like I have a new case I want to test it so like you will see it will happen in a few minutes you just say you want to test that you'll find people coming from nowhere so uh uh all what you guys we try to do at BTU is defining pain points that should be addressed uh optimization is is a deep rabbit hole that you need always to be worried about reatur optimization so but also we have a product out in the market that you guys are playing want to make it blable and fun so all what I'm trying to Aid myself and my team with the guys working in Frankfurt and guys work in the UK and over here in LA to try to point them as much as we can to that main pinpoint like oh if you spend sometime in this we might get some Advantage yeah but what we have right now is nowhere near to what we would deliver we have so many
00:45:19
checks in our bags that we know it's not the right time for us to like battle test them but we really need to care about like yeah you guys were suffering that much and we enhanced it 10% 5% 7% yeah it's still livable and we still having an eye on what's going on and also sometimes we are not willing to enhance anything right now but we want to see did did did our new cuder logic or or complexity taxed us a lot did it became very expensive or not so thank you so much guys for all the amazing work you do testing BTU yeah there's a rad Telemetry profiles being generated all the time and that gets uh you know disseminated too you the teams appropriately and stuff so people will have on their radar what is most expensive on on the game servers as well as the client stuff and you know just you know tackle it at the right time but
00:46:08
there's just so many features going into the game it's you know like I say pre premature optimization and stuff once you start doing uh optimizing code you know it looks a little different than normal and you don't want to come over and say hi right and there's a wide range of things that we can optimize I mean uh from the client side you're going to have you know things from render ing uh you're going to have a client side simulation and also you're going to have Network latency then on the back end there's you know there's all the physics and all the other things that happen there uh then there's also things that um where the game servers May rely on responses from the the the the services and so there's a a wide number so we don't want to pre- optimize now but when we start you know and we do and we we optimize certain things over time the low hanging fruit and such like that but um you know there's so many different departments that are going to you know focus on their little areas but
00:46:56
we have to focus on them at the right time like you know we pre-optimized now we could end up paying ourselves in the corner later exactly there's probably I mean there's probably nothing more important in game development than prioritization I mean we talk about the size of our team and our team is massive but it's still finite there's only so many things they can work on at any given time and and when you know when when you know something is temporary when you know something is is is meant to hold us over to get us to the new system that's coming down you know in a later patch one you don't want to spend spend you know too much time optimizing or or working on the system that you know is temporary and you're going to replace so it's a constant Balancing Act between all the I I it's always that the picture we the thing we always use was the guy that was spinning plates you know all different poles when's spinning a bunch of plates on top of top of the
00:47:44
10 different poles so when whenever we were spinning up servers we used to call them spinning spinning up plates SP disc and stuff like that uh let's see um the jabber walk has a question he says says will the website be hosted served from the same servers as the game or will they just access the same back-end database um the back the website and the backend services are about to have a very close relationshipi prior to this um it's pretty much a they it's always been us you know the the services and platform um but we're now coming together so um we can we would the services will look at them as another service and they look at us as part of their own system so we're kind of we're getting married we're using a rful apis right now which works on the internet but we're
00:48:31
co-located now so we're going to have dedicated channel so we get B directional communication yeah we'll have a set of services that have uh you know stateful high-speed communication mechanisms to the platform and um you know we've been working with them on on how to architect both sides to uh um make that as streamless as possible yeah make the data the the response and the data and so because and I'll use an example right now when you buy an entitlement or you you you purchase a ship on on on the website um it there's no way for platform to tell us that you purchase the ship so when you log in we grab your package which contains all the items that you all the ships that you purchase and so on and then we know about it and then we can you know create the actual and gain items and persist them um but with the highp speed the the new this new highspeed Communication System um the moment you buy a ship they
00:49:20
will notify us we can expand the ship immediately and have it ready for you by the time you log in the game and when it comes to Community Based Fe features you know like statistics and achievements and and things like that a lot of those a lot of those uh features really are best imple implemented with like a web- based technology um and then just if we have a a direct line communication we can just you know trigger those things immediately so and we'll be able to call functions from them directly as of you know like as a service and so on and there's an API we provided for them as well so they can uh request any data from us and generate events we can generate events back and forth to each other it's one of that that's what's one of the things that's so awesome about working with a partner like turbulent so are they're awesome yeah it's the first time I uh you know got on a project where the website was done before the game yes and then so much we all think
00:50:12
about we all think about solving the problems in the same way so it's it's a very good match and so much of the functionality like ship ownership and and all the stuff it all exists on the website and now we're pulling that down into the game yeah know it's uh it it's great to have that kind of Buy in from a partner because you know when ever you're working with a third party vendor you never quite know exactly what the level of buying is you know this is on any project not just video games and whatever so we're we're continuously very lucky to have Partners like turbulent with the with the level of buying they have so uh Beno I love you I love you all right so uh Captain Steve we we're we're wrapping up here fast run out of time he wants to know what hypervisor do you use in the cloud and also what core operating system do you use to run the game yeah currently we are in GCE Google Cloud engine the hypervisor used over
00:51:00
there is their own custom KVM uh I don't believe that the hypervisor that we would use would be I mean yeah there are different way of elation how we abstract the CBU instructions from from the bare mle to the VMS but I don't think we would be having an issue whether we running on Zen and on KV [Music] welcome back twitch uh if you're not a streamer you're probably watching a an ad right now uh basically we had a computer crash here we're at the end of our broadcast anyway so we just came back to say thanks for everybody to for watching and for supporting us and for submitting questions um Ahmed Jason Tom anything you want to say before we let
00:51:48
you go we are very thankful for ear your time guys in BTU we're going to have a stress test at 300 p.m. a little bit that would be great to have you guys testing G Cod making sure it looks solid and nice H yeah we are really grateful for you guys helping us with all this staying oldest work that's my end Y and we're happy to be here you know putting together something and you know amazing and giving you all a great gaming experience got nothing Tom uh I'm just hoping the PTU goes well so I can fall asleep tonight so fair enough fair enough stress test is started starting in three minutes you can get with Will and Discord y so so if you're in the PTU and you have access to Discord jump in there now we'll be doing our big stress test for 241 in just a few minutes uh
00:52:37
Ahmed uh Jason Tom thanks so much for taking the time I know we tried to get you in last uh last month and you know work work you know prevailed but uh than thanks for taking an hour out of your time to to sit with us and chat with us and and talk about a little bit about your work and take some questions from the fans so guys uh thanks a lot uh regular RTV is back on Friday uh be sure to check out uh around the verse episode 100 tomorrow uh it's part one of our of our multi-part episode 100 uh celebration so check that out and uh I'll see you again on Friday so thanks guys thank you all right have a good

Transcript from YouTube’s automatic captions — punctuation and Star Citizen jargon are approximate. The video itself is hosted by CIG on the official channel.