Star Citizen Live Gamedev: Life of QA

13 August 202101:02:481,720 linesWatch on YouTube ↗

Transcript — every row opens the video at that moment

00:00:00
and then we'll we'll do this hi everybody welcome to star citizen live game dev life of qa i was about to say life of qua and i'm like no i'm not gonna do that and then i talked about not doing it so i just defacto did it i'm your host jared huckabee and uh if you've never seen star citizen live before so where we take about an hour at the end of our week when we hang out with a with a collection of our finest uh developers and george stadler and talk about uh our work our experience our life uh in development as it were uh on today's show we are jumping into an often misunderstood
00:00:48
misrepresented aspect of game development quality assurance so before we get started let's take a few minutes and get to know who our players are we will start with someone who probably needs no introduction but i'm going to do it anyway she is the she is the versus very first original jump point pilot uh you might remember from the citizencon 2949 demo uh let's give a big round of applause for samantha haley thank you guys it's really nice to see everybody again virtually um yes so i'm here uh representing qa today um and through qa i got to do something really amazing at that citizencon get to fly the carrick and in front of
00:01:35
everybody and it was such a wonderful experience to get to do that so it's great to be back and and to kind of engage with the community a bit more again now what is it that you actually do for star citizen so in terms of my my actual job other than you know flying the carrick sometimes um i am the lead of the qa resource team and basically what that means is i deal with everything in terms of training up our new hires including our experienced staff as well i go through uh training with those guys too we deal with all kinds of guides and information and making sure that everybody is you know at the tip top of their craft for qa basically
00:02:22
all right now joining us on the show we have several members of the qatar you guys are all from the uk studio am i not mistaken all right so we have members from our our uk studio here uh let's start with a a new new and improved george george who are you and what do you do for star citizen hello everyone i'm george h i am currently the um qa embedded on the network team i've been with uh cloud now for probably about a year worked in a few departments in qa so i've kind of got like a varied experience different departments um i was the senior for sam a while back and then moved on to the embedded team or the embedded network team probably around about a month ago so i am you know here to help the network do the networking things basically
00:03:11
so so all the uh all the 30ks are your fault harsh um moving on to jack jack who are you and what do you do for star citizen uh i'm jack so i've been dragged along by my good friend sam to talk a little bit about the qa i do i'm part of the core team on qa so our kind of bread and butter is we sort of look at the core gameplay loops so making sure that the player works making sure that when you're loading the game new beverage is still there and we haven't accidentally deleted it um and the missions that sort of stuff really all the basics and last but certainly not least uh an oh an old hand here it starts in this
00:04:02
this gentleman and i i i gotta give him crap for this he he he he was here at cig around the same time i started and then he got out and then recently he came back he came back george george zero georgia original who are you and what do you do for star citizen i'm i'm george s i am a well i'm a new qa tester on the gameplay team so testing new gameplay features as they're coming in i've just come out of sam's amazing training as well a week ago so just had my first week back in the real world of q of qa which is interesting to see how things have developed and changed obviously before three years ago almost the day i was in qa and before that i was in player relations so some of you may
00:04:51
recognize the name from there as well yeah i think i'm still waiting for you to answer one of my tickets actually [Laughter] just probably all right so on today's show we're doing things a little different like now on star citizen live we usually have uh designers artists engineers stuff like this uh as we spoke of uh at the beginning of last week's uh inside star citizen because we are in our quarter three of the year and because that's the citizencon into iae corridor uh everybo everybody is heads down working on their presentations and stuff for for that stuff uh it gives us the opportunity to stretch our our proverbial and metaphysical legs a little bit and explore aspects of star citizen's
00:05:40
development that maybe we otherwise wouldn't throughout the rest of the year so uh just like on isc where we took the opportunity to delve into the production life of of steph bedford and we're going to jump into some other lives here in the coming weeks there i wanted to take this opportunity to jump behind the scenes into a an aspect of game development that's near and dear to my heart which is many of our finest folks get their careers uh start their careers rather in quality assurance it's it's a it's a terrific way to learn uh almost every aspect of game development because there's there's very little that goes into star citizen or any video game that quality assurance does not touch in some way shape or form names you might recognize like ricky jelly got their start in qa
00:06:29
phil from our audio team got a start in qa tyler wicken our director of community got to start in qa and i'm not going to go and list the list but it is a terrific place to learn the industry to learn almost every aspect of game development so with that i want to turn things over to sam here for for a little while uh sam you are the resource lead yeah it is your job to to train these train these these these people up and make sure that they can jump into what is what is a very dense and very a thick game with a lot of tribal knowledge uh uh where do we start with this process yeah definitely so i mean when it comes to star citizen it is absolutely massive
00:07:18
and when you know you're looking for people to hire you obviously want to try and take on people who have a little bit of qa knowledge to begin with but recently we have been um like reaching out beyond people who have maybe that games experience already um into people who have you know a passion for the project um for example quite a few of our new starters did actually start off as players of star citizen for example i know uh george plays star citizen as well um george stabler that is um so he started off from that um and it's just really kind of inspiring to take people through that process of knowing nothing about games to
00:08:06
kind of warming them up i guess and getting them out of the the myths and the kind of preconceptions they might have around qa to the point where you know they're fully ready to get into the job and test our massive kind of universe basically yeah when it comes to those kinds of um i guess myths and stuff um a lot of the people who like i say don't have that game's background may kind of have some conceptions about um qa testers and what they do and that kind of thing so i guess we'll kind of touch on some of those points um so for starters yeah so let's job let's start with the biggest myth buster
00:08:53
in the world qa just plays the game i mean i would love to get paid to just play the game that would be absolutely wonderful um but sadly it's not maybe quite as appealing as that i mean there's the video that goes around in the game circles where there's the um girl who comes in and kind of says hey guys are you done testing that game i need another one uh designed and they go yeah just tighten up the graphics on level three man i can't believe they pay us to play games and i guess that is kind of the the opinion that a lot of people get uh because of that um but yes when it comes to actual qa i think it's more to do with being kind of like an information broker for a developer
00:09:42
and just kind of making sure we know exactly everything about the game uh keeping a pulse on it as it goes from the beginning uh for example you know jack starts off with the most broken kind of new interesting stuff on the core team those guys deal with all that kind of thing and then as it goes on to the more finalized products um you might see it through you know george would kind of uh see that through to the released product basically so as well as getting to play the game which we we do um repetitively but we do get to play the game um we we kind of are making sure we're providing good detail to the developers on on how to reproduce bugs uh debug information for anything that
00:10:31
comes up um prioritizing certain checks for example if new features are coming in we need to make sure that we are rinsing those and maybe de-prioritizing some of the older stuff so yeah that's kind of a little bit more about the qa role do any of you guys have any points on just playing the game what what kind of things you do outside take us through it take us through a typical day if there is such a thing as a typical day describe the daily routine here do you want to one of you guys want to take that one i mean yeah oh no go oh okay i've landed on this one and so like sam said i work on the uh the core team which is kind of like we
00:11:18
check like the core um mechanics of the game to make sure that when all the new and exciting stuff comes in uh we haven't accidentally introduced a new variable that like completely ruins the game somehow so i'm kind of there on the front lines almost of just making sure that the game in its base form is still playable so in core team we usually sort of have like a series of checks and we'll like cycle through them so we'll spend a couple of weeks making sure the players working then we'll spend a couple of weeks making sure that all the different landing zones like corus and um new babbage area 18 they're all still working properly we might check missions and we'll kind of go through all of those and once we finish we just kind of loop back around to the start and start so a usual day for qas um i'll come in uh the rest of my team have usually
00:12:06
already started because i enjoy a line um and we'll sort of spin up our own sort of like microcosm a little tiny server that we just test with these six or seven people we've got and we have a big long list of hundreds and hundreds of checks that are all individually very basic so it might be like you go into port olisar and it's like are all the textures correct is the atmosphere correct and we'll just kind of systematically go through all those checks for whatever section of the game we're looking at for that week um and anything that's not wrong we'll sit down write it out and then pass that information up to the developers and then it's kind of their job from there to sort of fix it now i want to follow up there you say with the six or seven people that's
00:12:55
not the sum total of everybody in the q a no very specifically seven uh eight people on my team just looking at the uh core features again there's lots of different teams of people looking at other areas of the game yeah and it's often important to have multiple people because when you because over the course of days weeks months you're looking at the same assets these same locations over and over again and it's it's very easy to get um i don't know if there's there's probably a word for it but but just just uh not colorblind but just yeah snowblind i guess uh just you've looked at the same thing so many times that even if something's drastically different this time it's not always incredibly apparent because you just
00:13:43
you start to see the previous versions every time you look at something yeah okay uh anybody else want to add to the typical day before we move on to the next um i'll add a little bit from the network side as well so kind of similar to jack as well but obviously because i'm on the network team i'm just testing you know network related things and like persistent tech and stuff and one of the caveats for me doing this was i have to pick up the network team and how great the network team are which you guys are fantastic so hello if you're watching clive johnson yeah hello to clive if you're watching um so my typical day you know i i'm usually an early bird so i'm in it like seven o'clock um if we're continuing with a task that you know we've been set beforehand we'll just crack on with that um you know check things like emails and messages like normal
00:14:31
and then usually around about 11 30 we'll have like a stand-up where the team will get together and have like a video meeting and say you know this is what we're doing today um or what we've done for this morning and this is what we're now doing for the rest of the day and if anything like major comes up you know we can point that in a meeting saying no we're testing this build but we're having this blocker issue which is preventing us from doing it so because we talk to the devs directly you know it gives us that more of a like a frontline contact to say you know this is the issue that's blocking us and then you know we can go away get any evidence that we need any videos logs all that sort of stuff pass it on and then we can basically continue with our day um and then usually about four o'clock i'm heading off for the day to address some of the comments that are in the chat right now uh as as we've mentioned several times
00:15:18
before in other shows 30ks are a kind of a blanket error it's for anything that causes a disconnect between you know the client and the server and it is the same result but it can be caused by dozens and dozens of different of different sources so when when when the very frustrating question bubbles to the surface why haven't they fixed the queue in the 30k it's actually we fixed nearly a hundred of them it's just that there are more it is a catch-all it is a when the system cannot be more specific this is the error that it gives so it often has many many many different causes um all right so first myth busted
00:16:05
qa players just play the game the next one uh when when qa folks find bugs they immediately get fixed who wants to tackle this oh i wish oh wow and that touches on a bit of the things that people are saying about in terms of the 30ks and stuff as you know we probably uh have felt your pain um thoroughly every day you know while we're trying to test stuff as well it is definitely a struggle but we like i say like i mentioned for the last point we are information brokers um we deal the information but we don't get to decide when a developer has time to you know get in and actually fix the problem so unless it's something that is really severe and
00:16:55
a lot of people are seeing quite consistently um it might be something that people have to de-prioritize qa do try and raise flags about severe issues or things that we find particularly frustrating gameplay issues or you know particularly ugly art issues all that kind of thing but it is at the end of the day down to the developers to find time to fix that kind of stuff um they also have to find time to create new content and jam all of that in there and somehow get that released out to you guys honestly they do incredible work to manage to get that stuff done so yes unfortunately some of the you know more pertinent smaller bugs um may not get fixed for quite a long time uh but we commiserate and do please keep
00:17:43
sending them in to us we love to you know try and deal with the issues that the community call out particularly um and i mean george will have dealt with some aspect of that with your old job as well um didn't you with the player relations stuff yeah yeah so one of my bits and player relations was i believe it's become become more of an actual team now it was one of my jobs uh was with chris danks back in the day we used to whether there was a big patch we'd collate all the major information from the community like go into chat look at the issue council everything and find out what was annoying people most and send the top 10 over to qa to have a look through and get bugged up or linked to bugs which were already in and make sure
00:18:31
every the developers were actually aware of the the community's pet peeves they're the worst ones affecting the people on the ptu you you you used a phrase uh earlier show the information brokers that i think is is a fantastic assumation of of one of the core tenets of being qa it's it's a lot of folks will a lot of the assumptions that we see publicly are that if a bug makes it to live and this is in any game no you know we're talking in general here anything from cyberpunk to you know to full release to us who are in alpha here the idea that there's the assumption that if a bug makes it the live it must not have been seen or it must not have been noticed or found and whatnot when
00:19:19
the reality is often that these things have to be prioritized it's like anything else like we say quite often even with 700 staff working in five studios around the world there are only so many things you can be working on at any given time and it is often a triage it is often a triage of figuring out what the most uh aggressive bugs what the most invasive bugs are and then just the ones that you can have solutions for not everyone has not everything that you encounter has a solution that becomes readily apparent and some things can't be fixed until other systems come online or other tools come online and it's you know it's important you know it's it it's it's it's it's super invasive and drives you and everybody else crazy but it's like
00:20:08
the the tool that you need in order to be able to address it won't be ready in time so it's it's a it's a terrible terrible game of triage uh but one of the things i want to make sure that we that we uh uh that we demystify here is that they do not go out because they are not discovered i'm sure that happens you know sometimes occasionally but oh yeah but uh it is it is it is just usually a matter of triage it is it is i met you you guys are you guys are having the same reactions that the players do which is like yeah yeah definitely we do like my heart goes out to you guys because we we feel it too but yeah like jared says there are occasions where there are things that slip through the net um there are things that you know us as a team
00:20:56
we have about a hundred um qa staff across all of our studios um and it's just it's not enough for a game of the scale when you've got thousands of people out there playing it so please do you know keep telling us about your stuff um you never know it might be something that we've missed along the way and we would love to kind of hear about your problems or enough reports can help change the priority the prior disease exactly exactly that cannot understate the importance of the issue council and folks participation in it yes absolutely and the new issue council is absolutely fantastic um so yeah definitely make the most of that and use it because those uh player experience guys are doing wonderful work with it all right so uh we're gonna we're gonna
00:21:43
move into uh our next section which is about qa structure and how it works but i want to do one more myth before we before we move on to that are the enemies of developers the the the the antagonistic uh a relationship between developers in qa what do you have to say about that not at all yeah definitely not when it comes to um to devs and mqa obviously we are there to kind of somewhat ruin their day and we do come in and you know sometimes they'll just be something that they have not expected that crops up um but the vast majority of the time i i think we have a pretty good connection with the devs i none of them have called me names to my face um but yeah we work very closely
00:22:31
um pretty much from the beginning of when a feature is being designed so for example at the moment in the run up to 315 we've got the um healing and inventory changes that are coming in um we've been working really closely with the active future team on that uh guys like chris parry um doing absolutely wonderful work and they help guide us and we help guide them so you know we we're all kind of striving towards that same goal of having a good quality product uh developers also test their own work so you know it's not just us that are coming around and saying hey it's broken if they go through and launch it up and something's broken they also don't want that so yes as much as it is kind of like a tom and jerry cat and mouse kind of
00:23:18
a comedic relationship in theory um in reality we do have quite a good bond with those guys which is really fun um and honestly qa gets to touch on so many different areas of gameplay that we do have like a close connection with all of the devs say for example you know artists may never get the chance to work with a particular area um just because of the nature of their job um so it's really great to kind of have our tendrils in there and george being like uh directly in the network team obviously if you didn't have a good relationship with those guys that wouldn't work so well i mean i'm hoping this stuff's good yeah and i started going very also we kind of know
00:24:09
potentially because we touch on everything we potentially know a bit more about other features than the devs because they'll be very focused on what they're doing there one little thing i mean for example back when i was doing ui stuff uh before when i was here before there was an occasion where literally they they did something and it it it impacted another system they hadn't even considered but the second we saw it we were aware and we obviously fed that back so it gives it so having that relationship with the teams it gives you via you've got the ability to give them them knowledge sometimes they don't necessarily have access to because they're so focused on their one little area that's a good point it can be very easy for developers to silo and and and and tunnel vision down into the thing that they're working and because the the qa
00:24:58
the qa is embedded and and kind of this this viscous membrane that sits between all the different departments and whatever the the they're usually the first the many times the first time somebody goes you might want to check with this this team over here because what you're doing is going to affect uh what they're doing and stuff it it's really great and i also when you were telling the story just i often i often feel like the grim reaper because whenever whenever i approach this goes back more to when we were all in the office anything but whenever i would approach somebody's desk or whatever that you know there would always be this oh no jared's coming to ask me to be in a video or something jerry's asking me to be in his so so you know very often i would approach and then you'd see people like oh this is the time to get up and use the bathroom you know and go away and it just just this is the first time i
00:25:47
realized that qa can sometimes uh probably have the same effect like when a qa person is going to say oh which one of my systems is broken and not working right now so i can relate [Music] yeah the embedded thing is i think is a good transition into our next section which is about cig structures so many qa teams in let's just put it this way there there's not one way to make a video game there's not one way to make a video game every video game developer out there every studio around the world does things in their own way and slightly different from one another but many many folks many companies will sequester their cue their qa they'll be in a different building altogether or on a different floor and whatnot we take a different approach here at cag and we
00:26:37
take the word embedded uh very seriously we put our qa like right there in with the teams at the same at the same row of desks with the designers and the programmers and everything that are working on them uh talk to us about the qa structure yeah absolutely so like you say we do have those main kind of qa teams like jack's team like george's team that are the kind of main gameplay uh focused qa guys and and they run through together as like a team of qa granted they still do kind of contact um developers and work with them on that way but they're more of a kind of qa team we've got those main qa teams um like those in the uh uk and also in austin texas we have a few like um
00:27:27
testers who work as part of like main teams like that as well as the core and the gameplay teams we also have a vehicles and fps team which i led for a short amount of time it was really fun to to kind of delve into the ships and the shooters kind of stuff basically um and then we also have the tech team who deal with everything editor-based so all of the kind of tools that our developers use that also need to work um so not just the game playing aspect of things also the game making aspect of testing too um but yes like george here we have multiple um embedded testers who are part of their teams directly so george is part of the
00:28:16
network team we have people in the ui team audio team and basically they get to be the person who is the point of contact who knows everything about that particular area when it comes to qa we are generalists we try and do our best to provide info across the entire game uh but obviously you know we can't remember everything there's a lot of stuff um in terms of logging debug um i can't even get started on crashes there's so much with crashes we have a crash guide that is literally an hour read on its own um just all different sections so yeah it can get quite complex and having those embedded guys who are literally there to know the exact ins and outs of literally
00:29:04
how to develop everything to do with that particular team is so so useful for general qa for the developers themselves uh it is a really really good kind of symbiotic relationship i guess that we have going on i mean george do you want to talk a little bit about your transition over to embedded yeah so it was kind of like a weird culture shock in a way and so before joining games i've went to like retail for almost 10 years and did my like degree and then started in qa and moved for like a few different departments so before i was on like live release so what game play is now so we you know good test like the features before they go live um then i moved on to the core team which jack is on now and then i moved over to tech and then then i moved over to the resource team with sam for a good like couple of months and then i ended up moving on to
00:29:51
the embedded team um so it's quite weird of a culture shock because the first thing that i thought would go into an embedded team was like you know i need to make sure everything's good so you know if i see a texture bug i need to like bug it up which you know that is the promise but usually when you're in embedded thing you just focus on the stuff that your team does because that's why you're there um you know you're running around trying to fix all of these things like bug all of these things up and it's like no just take a minute calm down fix your network stuff bug your network stuff up and then relay that to the devs um and that's the kind of the way the team works i think the biggest thing is that when you're like a generalist qa is that you know you're not really talking to debs like on a day-to-day basis really because you're more like relating to your seniors or your leads um whereas you know going on to an embedded team you know you're talking to you know your lead who's obviously an engineer developer and plus like your
00:30:38
seniors and the junior programmers and all that like on a day-to-day basis um and when sam was saying before you know it's like a catch-22 so when people say you know oh you know qa or dennis must hate qa because you know they're always finding these bugs you know can't they just stop so we can do other things but at the end of the day it goes hand in hand so you know i'll go to like my lead or my senior and be like oh you know i'm not too sure how to use this specific tool or command or something and it'll come back at me but oh you know it does this it means xyz um and kind of the same vice versa as well so you know if dev is in game and they're not too short like a specific thing because you know a lot of the things in game do change quite rapidly as well because because the game is huge and then it comes to somebody you know i'm not sure how i do xyz so then we'd go and help them do what they need to do in order to check their fixes have taken basically yeah anybody else
00:31:32
you want to add anything but i think we kind of somewhat touched on the the aspects of it from the uh the day-to-day stuff but yeah like george says it's kind of the different dynamics of we have like qa teams and then we have our embedded guys and then we have like the squadron team which is kind of a hybrid between embedded and also being its own kind of small team of qa which uh those guys did great work as well right so so the structure and the format is malleable depending on the needs of that particular aspect of game development definitely uh let's talk a little bit about the the relationship to the rest of the developers uh you guys are up you guys are sometimes you guys are sometimes asked to do more than just
00:32:20
test like like we talked about that at the top of the show sam you must you most famously flew the the character in the uh the jump point demo at citizencon two years ago uh what what are these what are these other aspects of your job besides just testing and writing up bugs so yeah obviously the the most obvious one i can point to is the stuff like where we get to be involved in the citizencon uh playthroughs that kind of thing is super fun i know george did some work on the um testing for the citizencon stuff as well um but yeah being able to like perform that live and kind of do that in front of a crowd is a little bit of a different skill than my day-to-day kind just me and the two monitors kind of testing um
00:33:08
very nerve-wracking uh but super fun um so yeah we we do things like that we do uh game playing for marketing footage as well so say for example if they've got a particular feature they want to highlight um sometimes we'll get key way to do particular maneuvers or capture particular footage and game but yeah live playthroughs and stuff like i said on the stage or we sometimes do recorded playthroughs of things as well uh four citizencon we play sometimes during review meetings so say for example they're reviewing a future for the directors kind of want to see how a particular aspect of gameplay is going um in terms in-client how is it actually looking um will sometimes get involved with those
00:33:55
so that can be a bit nerve-wracking as well um but probably doesn't quite match up to you know sitting on stage with cr there and todd just there it cannot be overstated how much you don't want developers playing their own game uh it's just it's we we've showcased it on a couple the friday live streams in the past and there's a reason we don't do it anymore it's a let's just say the skill set that is required to make a game and the skill set that is required to play a game are are our if it were a venn diagram there would be two separate circles which always surprises people that when they jump into game with the people who make the game or whatever like why are they so bad at their own game i'm like different skill sets folks yeah
00:34:44
difference fully especially the art kind of based stuff uh those guys barely got just where they were good at playing the game that transfer over um yeah let's talk about working alongside the community i think the last pillar of cig structure here you've got how you work with the devs and and and stuff like this but let's talk about working alongside the community uh we famously have our issue council we we we we take the extra step of deputizing uh a a a giant chunk of our community in order to uh help us and inform us and uh help with regression and stuff like this uh talk to me about that what's that like does that make your job easier does it make your job harder please say it's easy it definitely doesn't make it harder now
00:35:33
i mean it maybe it gives us a little bit more work to do but there's always plenty of work to do with star citizen so that's totally fine now uh we work really closely with the player experience guys um to kind of investigate those issues that come in from issue council like i said those top 10 issues do come to us we do look into them we do make sure they're updated on the database so you can you know rest assured we are definitely touching your bugs like we do see them and we do appreciate all of the kind of info that you can give us about those particular issues especially if it's something that's on and off um sometimes having that stuff from the issue council is the key to us finding out exactly where to go and how to you know provide the right debug um you know if it's a particular
00:36:21
thing that we didn't think along the way a step that we were missing um that's super useful to have um we also do outside of the issue account stuff we do take things from the community from the forums if there are like feedback things that we think are probably bugs we do pick up those as well um videos um you know for example on youtube all of our content creators there and streamers we do get a lot of that content sent through to us and kind of say hey uh have you seen this happen is there a bug for it help um and we'll usually trawl through our database for that um so yeah it's really kind of an interesting position here at cig especially in comparison to other qa roles i've done in the industry where we are fully
00:37:10
tight-knit with the community um because we are releasing constantly and it's it's an experience i've just never had before when you're working on a project that is going to be released and kind of shipped off and you know nobody else is looking at it it gives you a whole other perspective um and you know you're able to do those big load testing of thousands of people or you know 50 people to a server every single time um which we don't often get to do in qa so that's really really um kind of you know an experience i didn't expect to have when i came to this job i guess and has been really interesting it is like having another crew of testers that could basically help us sweep everything that we need to do i mean how have you guys found
00:37:57
interacting with the community um i mean i know we've kind of touched on it with george but it's it's really really useful because i know back uh i keep saying back in the day back in the day there um there was an issue we had uh levski where occasionally everything would go psychedelic colors and we saw it three four times in testing and we couldn't repro it there's nothing we could do it just happened every so often and so we couldn't work out what was causing it at all and obviously it then went to the pt went to the ptu to to be tested and and actually because it gives you those extra people throwing it and we could see what was causing it and things and passed that on past that on
00:38:44
and it was it's so useful to have that information to be able to feedback and certainly in situations like that with things which are ultra low reproduction some chance of us reproducing them in gen in in standard testing are it's just luck whether it's we get it and we can reproduce the what caused it or even work it out at the time so we're talking about you're right now we're talking about you know the players the communities and how they can help with the testing is there any i'm looking at the chat here are there any uh guidances advices that you can give the community for when they're filling out their issue council reports thanks to to look out for things to avoid so i mean for starters things like video
00:39:34
footage and um screenshots obviously if you can provide that kind of physical evidence it makes it a lot easier for us to kind of maybe see things that perhaps you don't know the particular word for um externally so having those videos are really useful particularly mentioning the types of ships and the like variant of ship that you are encountering an issue in can be really useful to us actually um a lot of bugs end up being you know specific to certain manufacturers or certain types of ships because of the way that they're set up so if it is a ship's book maybe going into detail about the ship that you have whether you've changed the loadout um for example if you've changed any of your
00:40:22
settings so keybinds or um graphics settings that kind of thing really honestly the more info you can provide the better basically do you guys have any other suggestions for things that people could provide not really i don't think i've ever seen an issue council bug and gone oh no they've given me too much information please add less tr bugs like the more you can give me the happier i am because it gives me like a much better starting point to try and reproduce what you're seeing in game if you've given me loads of information i can just jump in game get to it straight away and go like that it i can still do that if you've given me less information but it might take me a bit longer yeah so yeah the more information the better
00:41:09
really don't don't think don't think oh that's been there forever they will they'll know about it because we might not it might be something which has been missed and missed and missed and if you raise it we'll then go look and check and we might even go oh that's gotta be and it's it's it's been like that for that's something which has been there for ages and then we'll go search and we won't find it so even if you think we must know let us know about it or it could be something that was de-prioritized and every time it happens and that can help to up the prioritization of it so you know so so just because you see it already just because you think oh they must already know about it already does not stop submit that submit that submit that it helps in a number of ways exactly
00:41:57
let's go to our our our our third pillar today's show here we've talked about you know general myth busting of q and a we've talked about the structure of q and q a q and a of qa and how they work within cig sam you are the resource lead yeah your your unique uh attendants are are maybe best served for training stuff we've actually had several people in the chat to today already talk about you know can you take us through some of the training uh some of the resources now obviously training for qa we've only got 20 minutes left in the show we're not going to be able to do the full thing but but but give us a high level what is what is what is training for qa involved because it is often an entry-level uh position for game development yeah absolutely and
00:42:45
like i say we've been taking on a few people that are coming from the community who don't have any experience or education in games it can help obviously if you have that kind of background but even i coming from a place where i had qa experience learning star citizen was kind of its own uh mountain to climb so it is kind of something that we we thought about in the qa department and we wanted to kind of prioritize making sure that the people coming through knew the game knew what they were doing and were able to kind of get into work and test the project quickly i guess so yeah when it comes to a project like ours with the longevity of the project it is super worth it to invest a good amount
00:43:34
of time into fully training our people from scratch and yeah for just from our guys i mean um jack did kind of you had like a bit of university games and indie background before coming into qa didn't you yeah so i did like um some game stuff at university and i was in like a much smaller company for people uh before i got to star citizen um and i think that does help um but i would say it's like in no way required uh i don't think during training there was any point where i was like oh i did it at university so i'm so much better than everybody else who hasn't done it everybody was kind of just there like oh wow this is a lot of stuff to learn thrown in the deep end and yeah you know george h coming from retail uh george
00:44:24
hess was literally in the navy before you came to uh cig so you know it really you can come to this career from pretty much anywhere as long as you've got the passion for it and you show us that you are willing to learn then i i love to take new people who are just you know passionate about star citizen and i want to test it and make sure it's good so when it comes to the actuality of what we do with the training uh we take all of our new starters here on an 18-day crash course through everything um star citizen um for starters that first uh week is kind of just game familiarization obviously we do need to be able to play the game so for those of us who maybe haven't played star citizen before but even those who have um it's good to get
00:45:13
some time to dedicately sit down with each area kind of go through bit by bit we start off with some game basics being able to get your ship fly around go from place to place then we go through missions so all of the kind of contracts that we have in game through contract manager mining which now is quite an in-depth thing um in terms of you know we have the ship mining ground vehicle mining fps mining all that kind of different stuff so we have a full day dedicated to just kind of getting those down um then we kind of touch on our other areas as well like um arena commander and star marine and theaters war i mean uh you know that thing that we are working on hehe um so yes we do touch on the
00:46:02
many game areas as well to make sure that everybody's kind of up to date with all of those after we do that if you were at the citizencon that um i kind of was on stage for you might have got the chance to play theaters of war there actually they had a really cool setup room for that yeah it's it's really good stuff and i'm hoping that you guys can kind of play it soon but yes after we've done all of the game playing stuff we move on to the technical aspects of things so that is all of our test commands and what we call kind of c-vars so allowing us to kind of get all of that debug information go from place to place much faster um say for example we have go-to commands that
00:46:52
allow us to zip from planet to planet in an instant uh which saves us a lot of time while testing as i'm sure you guys are aware it takes a while to get from planets planet naturally how to run our internal builds like jack was talking about before we do spin up our own uh separate servers and kind of run basically what we call development builds which allow us to get a lot more debug information development information that you can't get on the live servers our internal editor as well we do touch over some basics for that even though most of our testers don't usually um dabble too much in editor we do kind of give them a little bit of basics around how to actually set that up um and then we start to move on to things
00:47:42
like say for example qa basics uh writing books uh what kind of evidence that you need to provide for certain books and what kind of rules you need to follow while writing books searching for books uh testing like existing books that kind of thing and then lastly uh we do crashes which yeah it does take a little while to kind of go through all of those um yes while you guys are well acquainted with the 30k they do come in many different flavors um internally and because of that we kind of have to know how to deal with each one um which can take a little while but once you've kind of got wrapped your head around the basics um i would say that's probably one of the most technical and difficult things that we have to do in
00:48:31
qa so once you've got that down you're pretty much on your way how has work from home changed this process yeah so i mean since we have started working from home literally all of these guys have kind of come in during the working from home um time i i literally have never seen any of these guys in office which is a crazy thought to think about now um it's strange my brain kind of places people in the office as if they were there the whole time everything's very flat i have no idea what height they are i just kind of guess um yeah it's it's been a huge challenge to adapt everything to working from home
00:49:18
but honestly through doing voice chats each day um kind of just while we're working we'll kind of use voice chats to each other it kind of almost feels like being in the office these days and it's really useful for me um to be able to like screen share while i'm walking through particular things i do miss the aspect of being able to kind of lean over and help kind of guide someone directly but you know we we do our best and you know the wonders of technology have kind of allowed us to do a lot i mean how was it for you guys starting a new job while working from home it was actually it's really good it was really really well done it was a bit weird on the first day but it was so it was so it so smooth so so just basically straight into it it
00:50:07
didn't really feel any different from starting a job anywhere else it sounds daft it which is good that's good how about you jack um it's it's been an interesting experience um it's it's been very different that there are times in training um and even like after you finish your training because 18 days sounds like a lifetime of training you've barely scratched the surface um there are times when you're new and you're unsure about things that it is a little a little bit anxious a little bit stressful not having a person in the room with you to kind of like if it all goes horribly wrong just like jump in and like fix it for you do have to be a little bit independent and kind of
00:50:56
work with people kind of talking in your ear like okay i know it sounds terrible but it's not actually that bad we've seen this before it's fine what you need to do is this this and this i think during the training period that is a consideration but you kind of you work through it um i hope it's all worked out no one's fired me yet so i must be doing something okay he's doing pretty okay jijin in the chat says it weirdly doesn't feel live to me is the show live no i pre-recorded this it's not live yeah we pre-recorded this in the second three weeks here on the live chat now just means that you have no free will and that you were pre-destined to be here and chat today and ask this question
00:51:48
with that we've got a couple minutes left uh let's uh let's let's jump into some uh of the chat questions um how much automated regression testing are you able to implement uh versus inefficient and messy manual testing ooh this person doesn't like the manual testing so my biases show yes your biases are showing that that's totally fine i understand yeah so actually i mean automating uh testing for a project like this is is quite challenging to say the least as it stands we are moving slowly towards being able to kind of um automate certain test cases we do have a guy who is embedded automation actually and he is working slowly towards making it more available to all of us but we do
00:52:36
have a couple of things that we use internally already i know george on your time on the tech team you might have touched on some of this stuff with test runner yeah that's correct so test runner would you know it would grab a build and you know run through like basic sanity checks it doesn't go too in-depth because obviously when you be obviously making an internal development building and run it through test run it can take a few hours to do that so it goes through the basic checks and and obviously when we try and grab a build it'll give us all the information so you know this was fine this was fine this was fine but this was broken but sometimes we can get around it because it's not like a core thing that you know would stop us from testing but there have been cases you know we tried to grab builds and you know it would just break continuously um but that's where test runner kind of picks that things up so we can kind of identify that before we go into game to say you know we need to do this task but this build is broken what do we do kind of thing and then
00:53:25
obviously it's usually just the devs will go and fix it you know asap kick a new build or we just like usually revert back and then carry on the testing yeah exactly it's mostly kind of launch sanity checks and that kind of thing yeah basically yeah very basic pass fails at the moment but we are working our way towards it and it is something that we are very interested in so if you are you know an authority on automated testing you know please do come in we would love to hear your opinions what is your most valuable virtue in qa and why is it patience there's a lot of leading questions i think jack can answer this the best to be honest you deal with the most crashes on your team the stream i work on is notorious for
00:54:14
being uh potentially the least stable one in the game um i think the patience is obviously a big part of it but i think the thing i've er used the most since i moved on to the core team is my memory remembering things a lot of issues tend to maybe disappear and then come back and after a while you just kind of like remember like the really bad ones so something might appear that's been gone for like three months and you can just look and go oh i know exactly what that is i've seen that before and you'll go and you'll find and you'll link it back up and you'll escalate with the devs and go this has started happening again what did you do please please help the other one i'd say is curiosity quite a bit because you find which isn't quite right and you pull the string and then
00:55:03
you find something else and then you find something else and then you find something else yes i think that is a lot of what a lot of people don't really understand about qa sometimes is that you know there's one thing and you think oh you know that's fine it's an easy fix but then that will branch off into 100 thousand different things that you know that needs to be fixed so does that so does that so does that before this main thing can be fixed it branches off so much game dev is so complex it's crazy it's a i really like that i don't think that can be understated it's the one thing we can't teach is that curiosity that inquisitiveness that that ability to you know not just see something and go oh i know what that is that's that and then leave it at that move on it's looking under the hood and pulling on that string the proverbial
00:55:52
string as you said and just watching the entire sweater come coming down and then knowing that you know the developer that you're sitting next to the programmer is you're just you're creating more and more work for this person but it's like you have to keep pulling you're just kind of looking at them like i'm like i'm sorry it doesn't work so sorry i got paid for this i guess uh all right uh we are just about done uh the last question i see here from the from the chat says uh what is your what is your favorite aspect of working in qa uh i don't want to be too it's a little too uh i i don't know it's a saccharine for me so i want to ask what's your least favorite aspect of working in qa and then we'll do the favorite because there's kind of everybody's got
00:56:40
something they wish were different crashes are the main bug bear especially you know you're working on a test you get right to the end and the server's gone you're like absolute pain in my life you're just trying to get something all set up you've been with somebody else multiple people you just set up to do something get something tested and nope yeah you you're managing to like kind of hook up your thing you're about to like catch the bug that you've been hunting all day and then the pop-up a completely unrelated crash yep yeah completely something else and you're like no i don't have time for this that's not the one you wanted you're not the crash i'm looking for i'm looking for this one not you
00:57:28
oh yeah every time yeah i guess that would probably be my least favorite aspect when when bugs kind of pop up in in the search for other books and you just kind of have to go not now shh the things i'll have to look into later the good old sticky note of unloved books yep now we can now we can do what's your favorite oh i would have to say the amount of things that we learn we learn so much in qa whether it's like a kind of like a parts of like a dev kind of thing and just the game and just like general how you know how games are put together i mean i mean i love learning things so that's what i love about qa it's just you know being a sponge of knowledge for me it's it's there's a lot of
00:58:18
different things but the thing i think i enjoy the most is stuff it's the people it's such a a good vibrant group of people who even when you're having the horrible crashes you'll be you'll be able to have a joke and have a laugh and it's if it wasn't for a good group of people i don't think you i don't think you could i could do it so but it's such a good group people you're lucky we let you back in what about you jack well should have gone before george because i was to say the same sort of thing i think um especially on the game dev stream which does tend to have some minor issues with stability you do form a real sense of like camaraderie with like the people around you because it's not just you it's having like a horrible time and you've
00:59:06
been trying to get into game for four hours and you've not made it past the main menu yet it's you and seven other people who are also having a horrible time and haven't made it past the main menu yet kind of oh friends for life forged in that fire together i feel bad about knowledge and everyone's like about people i do love everyone i work with by the way that's too late josh you don't like that please i i understand george i am and how about you sam um yeah i guess as as well as being able to kind of do do the same things as those guys i love learning stuff i love kind of picking the brains of all the different developers especially you know the big brain people like the directors who just so much about how to make everything cutting edge um
00:59:55
it's difficult um i guess just generally being able to work to support games is something that i really kind of enjoy when i came out of university i did an animation degree and i hated animation i came out of it i was like never again never again in my life but i knew i still wanted to kind of support developers and i knew i wanted to support games development and do that kind of stuff because that is something i'm so passionate about i didn't have a clue what i wanted to do and just kind of fell into qa by pure chance um just because it was something that was entry level and you know everybody could kind of have a go at and i ended up staying here um because i love it so much so yeah i think it is just something that really is an interesting career and
01:00:44
really kind of is is a way to get your fingerprints on a game um and kind of put your influence on a game's development um you know without having to necessarily be a creative or something along those lines and for those of you who are watching either live on twitch or on youtube on the replay if you're interested in pursuing a career in quality assurance you can check out the cloudemperiumgames.com jobs website we got studio we have six studios around the world now we got la we got austin we got manchester we got darby or derby depending on how you pronounce it we got frankfurt and we've got our newest studio in montreal so lots of opportunities there to join the team and help make star citizen uh the best damn i was going to say space sun but it's so
01:01:32
much more at this point aiming to be so much more at this point best damn video game of all time i'm gonna get crap for that afterwards all right sam jack george george i'm jared thank you so much for joining us here on star citizen live uh tune in for inside star citizen next week we are taking a look at hospitals the new hospital and a new babbage in oars in general the grim hex one which is surprisingly cool and of course the clinics that are coming along and well as well as exploring behind the scenes of the gold standard work going on for the saber the gladius and the retaliator first so stay tuned to that and then we'll be back here next friday with uh something i don't know yet we'll figure it out uh take care everybody godspeed hi guys uh see you in the verse
01:02:21
thanks for watching everyone i've only been doing this for like seven years i should figure out how to end a show sometimes i should figure out how to end the show you

Transcript from YouTube’s automatic captions — punctuation and Star Citizen jargon are approximate. The video itself is hosted by CIG on the official channel.