Star Citizen Live: Meet the Devs - Publishing and Corporate Technology

9 February 202401:05:571,802 linesWatch on YouTube ↗

Transcript — every row opens the video at that moment

00:00:04
Hi everybody. Uh Chris Roberts is messaging me at wondering where his blender from the Battle of the Bricks in 2021 went. And I'm happy to tell him that we broke it. It's literally happening live. Sorry for the timing. I probably have a job. Hi, welcome to Star Citizen Live, our weekly chat behind the scenes with the developers who make Star Citizen and Squadron 42 possible. On this week's show, uh we have a special treat. We are uh reverting to our original Meet the Devs format because we have three esteemed CIG luminaries who are visiting from the fabled land of Austin, Texas. That wasn't how I intended to say that.
00:00:52
In my head, I was like, let's put a fun spin on on on on on saying it like it's a far-off place and out. You can stop recording. We won't be posting this one to YouTube after all. Uh let's just get to our guests. Uh we've got a John, uh Mike, and Ahmed. John, Mike, Ahmed, how are you doing, guys? Hey there. Good. Good. How are you? Uh thanks for having us. You're visiting from from Austin. We don't often get you guys here live and person. Um I know you had some like summit stuff and going on whatever and while you were here, you know, I did I did my normal, you know, walk of the video Grim Reaper past where you guys were sitting. Hey, you want to be on video? Are you here? Uh so, I appreciate you taking the time to be on the show. Um we're going to We don't often have uh folks at the executive level on the show. Uh so,
00:01:41
we're going to take this next hour and we're going to get to know you, uh get to know a little bit about what you do and who you are and why you made the terrible terrible decision to get into video games. So, uh John, we'll start with you. Um tell us who you are and what you do for Star Citizen. Cool. I'm John Erskine. I'm the head of the publishing organization for CIG. I've been with the project since essentially the very beginning of the project. I joined in the beginning of 2013. Uh at CIG, publishing encompasses everything technical that's not related to the game engine itself. So, essentially, after a developer checks their code into our source control, Perforce, then it goes into our build system and the build system is the start of the
00:02:31
publishing pipeline. So, when a build gets generated, it goes through a series of, you know, automated tests and it eventually goes to a QA department. It eventually gets published to some kind of environment, whether it's an internal development environment, a staging environment, or maybe a test environment. And that's all part of the publishing technology group. Uh then we have also our live QA team and ultimately our customer support team that are supporting the game once it's either on PTU or on live, people have questions about the game afterwards, or, you know, problems with their purchases or all that sort of stuff. So, all of those variety of things at CIG, we call publishing in one form or another. So,
00:03:19
that's my group. There's a whole team of us. We have people all over the world. principally, we're based in Austin, where I'm based, or at least the most of the leadership team is based there. We've also had a really long fruitful relationship with the platform team at Turbulent, which is based in Montreal. So, I head up a lot of the functions there related to the e-commerce and the sort of technology behind the platform there. So, we're dispersed all over the place and do like a dozen different things and it's pretty exciting. It's really sort of your answer is better than the one I had in my head, but it's I always saw publishing, at least for CIG and as everything that was outside, underneath, and around the games itself.
00:04:07
That's a good way to say it, too. It is. It it's it's it's basically, you know, you know, we've got our developers and and our five studios around the world working hard and and you and your team and those who work for you and everything uh basically make it possible for for them to do their stuff. It's whether that's providing the IT infrastructure for them to, you know, do their work Yep. uh to the publishing infrastructure to, okay, now you've done your work, let's get it into the hands of of of of the backers, uh from the platform stuff like you said to to track uh pledges and and and and entitlements and stuff. It's really it's you you run this entire cloud that just sits around, you know, the game itself. And I always thought that was interesting. We We I don't think as I'm talking about this, I'm trying to remember the last time you and I ever sat here and and talked about this and I'm not sure that we ever It's been a
00:04:56
while, if ever. How about that? Yeah, yeah, yeah. Um That's a Yeah, that's a good description because it's sort of like the game We have entire technology teams devoted to the game itself. A lot of the services that run the game, the game engine, and everything that happens there, but I like your description that we are sort of surround all of that maybe in a warm fuzzy blanket and make it possible for people outside the company ultimately to play the game. Okay. Um I'm going to go to Ahmed next just because, you know, we go publishing to publishing technology. Ahmed, who are you and what do you do? Um Try to keep it under 35 minutes. Yep. I'm Ahmed Shaker. I'm senior director of publishing technology. Uh my team and I are responsible for all the effort and technology that goes from the moment the
00:05:44
build is done running till anybody gets to play the game. So, we're responsible for delivering the game internally, externally, to all of our backers. I've been with CIG for almost 9 years. I had the honor, my team and I, of shipping every single build from 1.1.2 till hopefully 3.23. You mean hopefully? I mean, you have to be hopeful with every single publisher going that it will make it and it will go. You have to be hopeful, says the man who's wearing little rose-colored colored Um yeah, you're there's always this there's this perception internally sometimes that your Okay, I was just going to tell you how how your you and your job appear to to me and other people uh because we
00:06:32
don't get to see you. You're you're you're away in on Austin. We just kind of hear you. The the visual that I have in my head is you in a dark room with a bunch of spinning plates just just just just pulling levers like like like Willy Wonka in some kind of thing to try to keep everything going. I know I know that's not the literal truth. It's you you and you have your team, but but am I close to that? I mean, it is It is pretty close. That that the fact is, we do not just own the infrastructure that the game runs on. We take custody of the game from the moment it comes to us. So, we proud ourselves of being experts in how the game runs. And sometimes the developers who work in the code themselves come sit with us and get amazed by how we get to know where
00:07:20
might be the issue before even we get to look at logs or look at a look at graphs or look at any kind of data to figure out what's going on. Uh the game is a living creature. Uh it's In in its own high fidelity and the dedication that we see from the community, we get also to see that our community does not deal with it as a rigid system. They they they try to always interact with it in a very very unique way. We saw a lot of that, especially after persistent entity streaming was released. And uh we like the challenge. We like that they bring into us. They try to break our systems. They try to break the game. So, there's always a lot of fun going on and and again, it's it's not like you're hosting any kind of very established, static piece of software, making sure it has connectivity and IO and compute. No, this this is an ongoing thing that's
00:08:10
always evolving and a lot of the times we learn about the software we're running as it age. So, as as a lot of the backers remember, we have we have publishers that could live for a few days. We have publishers that could live for 90 days. And every single day there's a new a new thing to learn. So, yeah, there's a lot of holes to to fill and a lot of plates to spin. And I'm glad you bring that up, the learning. It it's it's I'm often I often say this is video game development, not video game construction. You know, construction is this you know, is this world where you figure everything out ahead of time. You've got the blueprints. You've got the engineering. It's it's like there should not be any variation if you're doing your your construction right from the final product to what was laid out in the plans. Development is not that. Development is
00:08:57
All right, let's try this and see what happens. Now let's try this and see what happens. And why the hell is that happening? Mhm. And stuff like that and it's uh it's it's what makes I used to work in construction. It's why I prefer video game development to construction quite a lot. It's it's it's far more interesting and you're learning constantly every single day. Mike Jones. Tell everybody who you are, what you do. I'm vice president of corporate technology. And this encompasses uh what we call corporate technology. That's infrastructure and networking and basically anything that runs that is computer related or service related. We also have IT.
00:09:46
And IT's function is similar, but IT's focus is largely directed toward the employees and their needs. So, is there do they need a computer upgrade? Do they need a new monitor? Uh is their software working correctly? this is a huge task, but it's also really fun and really rewarding. I also have a team that does setup and architecture and design for the computer or network aspects of our live events or shows. So, this this is a really fun piece when we get to do them. Uh so, it's a
00:10:33
it's probably for me the best possible job I could ever have. To be able to do this, it's my hobby, so I love it. Let's dive a little bit into that difference a little bit more. I think we have a very technologically savvy crowd, you know, audience I think. You know, a lot of them can easily understand what IT is and stuff like that. I mean, I worked IT for 20 years and stuff. Corporate technology though, that like we say when you talk about infrastructure, talk to me talk to me a little bit more about what that is. The people might not understand the you know the the the the workings with the you know the the infrastructure workings the the the connections with our you know third-party vendors and providers and stuff. Talk to me about this world
00:11:21
of corporate technology. What is this? Okay, so uh when we say infrastructure, we mean the things that it takes for overall operations. So, that could be anything from the network that we use to connect to the internet or the secure network that we use to connect to studios to each other so that people can collaborate. Uh the servers that uh we have our software running on internally to help uh developers do their work, storage systems, all the security that goes around that. We also in corporate technology, we also have a pretty large operation uh that is our global licensing. This this is a pretty massive
00:12:09
undertaking considering we have over 10,000 unique licenses that we need to maintain for every different development operation, every accounting function, any any number of things, and my team is the one that not only orchestrates all of that, works those deals, makes sure that everybody has what they need, and also monitors for compliance so that we're not ever out of compliance on a license or behind on a firmware update. Infrastructure means a lot of things that are behind the scenes. So, it's the stuff that makes it work, if that helps. Yeah, the the the foundation and and and and IT is more of the you know it's
00:12:57
I I think I always saw I'm going to I'm going to attempt another analogy and we'll see if I get it right. You know, it's I always found IT in general computer stuff can all can be very analog analogous to the medical profession. You know, you know most IT folks are like the general practitioners. Those are the people that are there it's like when you say, "Oh, I'm feeling tired or I or my head hurts or whatever." It's like, "Okay, they're they're dealing with the you know the the the general collection of ills that can plague a an individual user." And the and and the corporate tech folks and technology stuff that's more that's more like the you know the research scientists and this the stuff that people are looking for cures for cancer and stuff like that you know and build and building the and finding the answers that
00:13:43
will then trickle to to IT and stuff later and stuff like this. It's a it's it's a really unique it's really unique world. I only ever existed in the IT world. So, when I whenever I cross paths with a corporate technology person, I was always I was always trying desperately to pretend I understood what they were talking about. I'm like, "I'm pretty good and I don't know what the hell you're talking about, you know, level two you know internet and stuff like this." Okay, I do know what that is, but I wasn't a good example. You also mentioned the the the event stuff. Um I think that's where you and I have most of our interactions and stuff because I mean I we've been I've been tied to just about every event we've ever run uh in the in the 9 10 years that I've been here. Um There's always that some of my some of my best memories of these events are either those times before
00:14:32
where we're there a week before and just setting up computers and running cable and and stuff like that or the time after when Let's think think about how diplomatically I want to say this. Everybody else has finished. Everybody else has left. Everybody else has breathing their sigh of relief and leaving and then there's still this team that has to all right, let's pack it all up and and and and break it down and stuff like this. And uh just I don't know I have this I have this very specific memory of riding around in your truck in Austin. Uh very impressive truck uh and just being like or or or riding with Hassan in you know at at during Gamescoms and stuff. I always I always managed to find myself hanging
00:15:21
with the with with with with your people whenever I could because it there was just I always liked that they were some of the very hardest working and dedicated people to making these things happen, so I just don't know if I ever shared that with you, but I You cannot say enough about these teams. In the earliest days, we just had a couple of us that would go to a show. I think my first one was in 2013. And I think we had four demo machines and a server. Not much. We even had to string our own cabling for just to get internet. Um but it's really grown since then. And so now it's lots of people. But basically what we do these days, we learned over time that the needs of this game and and really the vision that
00:16:10
we're trying to present at these shows usually exceeds the expectation of any venue. So, we have had to invent ways of getting around that. Uh a good example of this is last year's CitizenCon where we had certain constraints that meant we couldn't hit the vision that we wanted to achieve. So, we basically well, I say we, my team of super expert specialists built out pretty much a whole studio, condensed it down into a traveling package so that we could take with us everything we needed so that we didn't have to lean on anybody else. And
00:16:58
by the time we were done, it was a pretty I felt really proud of what the team put together with 150 PCs, a stack of servers. We had a lot of help though. You you're you're you're you're talking about and I I don't want this to get lost in the in the conversation. You're talking about you and I and all your team and you built a microcosm of the entire Star Citizen development environment and brought it on site to LACC so that we could download you know right from the data store. So, you so you know you check-ins are replicated and so and so you know all those 150 server computers didn't have to hit the internet you know to go you know a venue convention center internet. They could they could all run you know locally on that. It's just you you It it it's the thinking about is like let's the entire Star Citizen
00:17:47
environment everything that is Star Citizen as far as software, as far as technology, as far as communication, and building a duplicate microcosm of it that fits into a room was very very impressive. We essentially yeah, we you could think about a CitizenCon as the like you're saying in in a week, we build a sixth studio on site with you know, networking, security, build system, distribution. This year we had private cloud for game hosting, Wi-Fi, you know, everything you would expect a studio to have. And then like you said at the end of the show, we got 8 hours to tear it down and get it in a box and get it out of the building. So, that's essentially what the week looks like for us. Yeah, it's the worst part of those things. It's it's the teardown. It's the it's the and and how fast you have to
00:18:35
get out because these venues have another client That's right. All right, so we've dabbled into talk of the old days. I I want I want I want to turn the whole show into it now. Um I want to I want to talk a little bit about where you guys come came from and and and your your journeys here. Um Uh I'm going to we're going to start with you just cuz I picked random here. Where did your journey to CIG start? Did you always know this is this is what you wanted to do? Did you Talk to me, how did you get to CIG? Uh interesting. So, moving to the US, I'm from Egypt, and during my stay in Egypt, a group of friends and I had our own little shop where we used to support Star Citizen around the world with all the technology needs when it comes to infrastructure work, DevOps work, even
00:19:23
pre-dating the word DevOps, pre-dating cloud. And one of my specialty was building private clouds before the word cloud also was pretty cool. I've been in this field for quite some time. And moving to a new place, I wanted to find a cool story to to attach to. And job hunting, I'm out in Texas trying to find something really unique, and I stumbled upon the one of the DevOps positions in in in Star Citizen. I still remember I met Francesca a few days ago and I think I was remembering when I first sent the cover letter. Um back when I was in Egypt, I had one of my clients that was a gaming studio, not in anywhere the size of CIG, but it was one of my favorite clients to visit because you get to see so many aspects when it comes to computer science. Uh and also you get to see the artistic aspect. That is not really something you
00:20:10
find in a lot of other clients when it comes to people that would use infrastructure. So I I I applied for many jobs and I hope that this will be the one. And I think a few hours later I got a phone call. I think the cover letter made made an effect. And I think a week later I was meeting with the team with our first interview and the journey started from there. In between I got to consume a lot of video content. I mean, at that time and we still till now we like to share a lot about our vision and it was fascinating to see that there is a vision to create an MMO with high fidelity, cloud-based, with latest technology. And um I like being in the bleeding edge. I like challenges. So it kept going from there. I love hearing that because, you know, there's this there's this assumption that the shows that we make and stuff like this are
00:20:58
just for the backers, just for you guys who are watching and stuff like this. but as has been made very apparent over the decade I've been here, it's also a heck of a recruiting tool. It's also a heck of a learning tool. And once you've been recruited and catching up it's there there is literally not a week that goes by where I don't get some message from somebody who's joined the team and said, you know, you know, "Hey, I learned about CIG. I learned about the project by watching this or that or whatever." Or or "Hey, I'm here. They told me to ask you are are there some things that you can send me that so that I can like jam and and really learn who this thing is?" And I've now got just a I've got a I've got a copy-pasta where I'm like, "Watch these videos. You'll learn you you you you know, you'll be able to audit CIG 101 by the end of training program.
00:21:47
had you uh had you worked in another game studio before this? No, I just had one of my clients back back in Egypt that was a gaming studio and it was like one of about 35 clients. But again, there has been that always fascination with with gaming and how games are being made. I still remember being very young and sitting after the end of every game to read the credits. And there has always been the aspect of that you do not see as many guys with a name like mine in the credits. So it was also another interesting bit to how can I navigate the world? How can I make it there? And it's still till now one of the most rewarding areas of working, especially now that I was here attending a summit, is you get to work with brilliant minds. You were you were relating our field to the medical field. I think it's a very very good analogy. So we have brain surgeons, we have heart
00:22:35
surgeons, we have cancer research guys and it's We got David Donkerly, too. Absolutely. So it's it's a great privilege to to work in in a shop that really cares about computer science and get to use the magic of that science in so many areas and then you get to have the art in top of it. And in our case, the lore of the game is is fascinating to see how it relates to human civilization in different in different areas, how it relates to some some World War II style, how it relates to old civilizations that clashed. So I I I don't I like I don't want to sound cheesy, but it's very hard not to sell our project to anybody else about it. It's one of a kind. So it was not hard for me that once I saw the job posting, I said this is the one that I'm going to put all my effort in the cover
00:23:21
letter and here we are. I'm glad that I I don't think I don't think I've ever met anybody that wasn't super glad you were here. Um John, I'll move to you. Uh where did your journey start? You I know I I know a little bit I know the barest minimum of your journey, but you went through quite a quite a few places before you got here. Talk to us about that. Yeah. So my journey in the game industry started uh I was living in Austin and I was managing a retail store that specialized in outdoor gear and travel and the sort of exciting stuff like that. And my roommate was is one of was one of my lifelong friends who is also a CIG employee now. His name is Paul Vaden. He's in our CorpTech department with uh Mike and
00:24:10
he's a critical member of our team, but uh anyway, Paul and I've been friends since we were in almost kindergarten. We have birthday party pictures together from when we were kids and stuff. To make a long story short, he worked at Origin Systems in Austin and was uh working on an early version of Ultima Online. And I he was we were talking one day at the apartment, you know, and and he was telling me like uh "Hey, I guess it started before that because I would go hang out there at Origin with the guys and play like, you know, Quake on the LAN cuz they had great connection and stuff. And so at some point, you know, I was probably spending too much time up
00:24:57
there after hours playing games and goofing off and they sort of said, "Well, hey, you know, we have a open position working on Ultima Online." And I sort of said, "Well, what's that?" You know. And they gave me a beta key and I went home and I just played the beta version of UO like nonstop for a few days and I said, "Well, yeah, sign me up." So I went through an interview and got a job, you know, as a game master on the beta version of Ultima Online. And um that was the start of it really is working at Origin, getting to know the team. I was particularly fascinated with online games as a way to, you know, as a vehicle for connecting people. This
00:25:45
these were like the really early days of you know, what started to become massively multiplayer online games. You know, there's a long history before this starting with, you know, MUDs and bulletin board games and I'd played those games since I was a little kid, you know, early MUDs and stuff like that. But that was really cool. And so I ended up working at Origin and Electronic Arts for a few years, advanced my, you know, career internally. It was sort of funny because one of the things that was attractive to me about going to work there was that I I felt like I had a lot of responsibility when I was managing this store and I felt like I had less responsibility just basically being a customer service
00:26:34
person. And in about 6 months I was managing the department again and had a lot of responsibility and at sometimes wished I was back at the store, right? To be to be fair. Um but fast forward, uh I left there in 2001 to work with Richard Garriott and a small team to start a new company called Destination Games and uh pretty soon after we started that company, we actually joined forces with NCsoft and became NCsoft's western division uh brilliantly called NCsoft West and um we built a publishing organization for NCsoft to service their rest of world uh venues and I ended up staying with that project for 9 years and left with
00:27:23
Richard and a small team to form a company called Portalarium and we focused on social and mobile games and we had built some really really cool tech, but never really had a product that, you know, hit the market super strong strongly. And so uh in 2012 we had been talking with Chris about some of his ideas about getting back into the game industry and um ultimately uh when he launched the initial crowdfunding at the Game Developer Conference in Austin in 2012, he was hanging out in our office space in downtown Austin and using that as kind of a base of operations. And so we sort of chatted a little bit about some
00:28:12
of his ideas and um what he wanted to do with Star Citizen and the crowdfunding campaign and uh few months later uh I was working with Chris instead of Richard. And then here I am. It's been about 11 years since then and we've come a long ways since those early days. Game Master to Chief Publishing Officer. Who would have thought, right? Yeah. Yeah. Yeah. You know, to be to be honest and maybe this is I mean, I don't want to make too big of a thing out of it, but I remember I worked with this guy as a game master in the like in the very beginning and he sort of had this attitude of like, "This is stupid. We're doing grunt work.
00:29:00
You know, if they fix these bugs, you wouldn't need us." And whatever. And I distinctly remember having this conversation with him saying, "You know, this is the future of video games that we're doing here today. Like this is really going to mean something and you should take it seriously." And so I sort of had that attitude from the beginning of like, "This is really where this is going." And I think it's interesting to think about this the aspects of those early games that we thought were really revolutionary. Like we had stories of, you know, people who had military family members that were deployed and they would use Ultima Online as a way to talk to one another after hours and stuff. And now we take that for granted as a super mainstream activity, whereas 20 years ago that was kind of fringe, you
00:29:49
know, and so I think a lot of the things that we consider mainstream today, social networking and all this kind of stuff really has its roots in these kinds of games. So, I thought that was exciting even when I was first getting into the game. Not that I've got perfect future vision, but I was at least and smart enough to realize like this is really meaningful when it comes to interact how people interact virtually and it's going to mean something in the future. I get that. I love that. Uh I was my start in the industry was a GM at Blizzard for the launch of Wildstar back in 2004. to start. Yeah. Mhm. Wow. We weren't We weren't quite prepared for the for the deluge. That was our story, too, in New World. Yeah. We had at at Blizzard we had we had you know everybody shared information in those
00:30:36
days. Like there was there was none of the tribalism that exist now. So, we had the concurrency numbers for every other MMO in existence and you're like they built an infrastructure that was like, "Okay, Blizzard was like, "We expect to get everybody." Like they were operating on the side that we were just going to take everybody from every game. So, they built a they built a platform and an infrastructure that supposedly could handle that. And then we exceeded that very quickly. And there were a lot of uh yeah, there were a lot of nights sleeping under my desk so that I could be there at the next shift in the morning not lose the time going home and stuff, yeah. Yep. But you know but you know we're we're going to we're going to we're going to talk about horror stories and and how we've evolved since then. Mike, your journey, Mr. Lagman. No. Tell tell me that. your funny story. Uh So, first of all, I I had just stumbled
00:31:27
into this. I had no aspirations. In fact, I didn't even realize that games were made. It didn't occur to me. I played them, but I never thought about what it takes to make a game and all of that. And uh had You're talking about a long time ago. This is really a long time ago. You said where did the journey start, right? Okay. So, yeah, the I got an opportunity to tour a video game development company, Sierra Online. All right. That's way back. Yeah. at the end of the tour I just it's really kind of blowing me away how cool this was, right? And I got this meet this guy and it was a job offer and and I he said uh I have two openings. It's
00:32:14
customer service and quality assurance. And I remember thinking, well, I don't even know is this for real? Wait, am I actually getting a job here? I didn't even come to get one, right? But I guess they were desperate. And so I asked which one pays better. And he said, "Well, customer service starts out better, but it's pretty hard. then I then I remembered, here's what you're supposed to do. At least that's what they taught us. How do I get your job?" And and he said, "Well, if that's the if that's what you want to do, if you want to be in leadership, you need to you need to start out in QA." So, I did. it you know it went from there. Uh I ran QA teams and then I found that I had this real strong talent and interest in technology and so I just
00:33:03
kept focusing on that and learning as much as I could and always going to the right like conventions and shows to learn more about tech and get ahead of the curve. and then, you know, I I I always enjoyed MMOs. So, that was part of the the journey. But what really brought me to CIG, cuz I I had known about Wing Commander. I knew about Chris, but I was I hadn't really, you know, being on the West Coast I didn't wasn't as connected. And so this video came up on YouTube and it was Chris at a at like I think the first CitizenCon. And he was to you know introducing his team and talking about what they were
00:33:51
doing and I was just blown away. How is so transparent, so different than everything that I have learned in my career leading up to this. But the but the live stream started cutting out. They were having bandwidth problems. No. And I thought, "Good lord, who whoever's in charge of IT there is probably having a really bad day." And you know what? I think it was 2 days later John called me up and said, "Hey, you want to come down here? We may have something we want to talk to you about." And so that was the real beginning of my journey and I couldn't be happier because of anything that I've done in technology or IT, it's it felt like it was always on rails. You didn't have a lot of
00:34:41
creativity. Here, we're doing something no one else would try. And that means for me I get to design solutions and work on things that we're actually encouraged to solve problems that other companies don't have. That's candy land for us guys. There's there's this there's this uh there's this phenomenon and I'm and I I and it affects me quite a lot, so I know it's going to affect you. This the I'm just going to go ahead and call it what it is. The crazy asks. That that that that that you're just minding your business one day. You're having a good day. Things are going the way, you know, this is this is how life should be and then a message on Teams shows up or an email comes up and there's just this outrageous ask and your first reaction
00:35:29
is like, "No. Nobody does this. There's a reason nobody does this." And then but then you start to think about it and you start to wonder, is the reason nobody does this because nobody's figured it out yet or is the reason they don't don't because it can't be done? And we I generally find that the people who assume the second one, they move on to other they move on to other projects. The the people that that that that succeed here and the other ones they go, "Well, no, let's see if we can figure out what nobody else has figured out." And sometimes, you know, we succeed and sometimes we learn the hard way. Hi Holiday live stream 2016. Memories. I'm I'm I'm hearing Vietnam helicopters and
00:36:18
I I got I can do the same thing for Ahmed. Watch. 3:18. I was about So, talking about what John was talking about about how games push boundaries. That is so true and it's one of the areas that we're always chatting about. Always comes up. Like right now everybody's obsessed with AI. When you think about it, AI is powered by GPUs. GPUs came to exist because of video games. So, kind of like a similar analogy. When 3.18 published from PTU to live and a lot of players did not know that the typical ratio between a PTU and live is usually 25x. Uh the problem with 3.18 though is we decided You want all the problems. That is true, but like it it starts from the fact that our universe with all of our moons and planets and players and
00:37:07
ships, we decided to persist everything we can as often as we can. And that is a a recurring theme in Star Citizen. A lot of players might not notice that we challenge ourselves as early as possible. And instead of waiting till we reach the million or the 2 million or the 3 million concurrent players. So, we try to amplify the challenge all the time by avoiding any kind of hacks or shortcuts. And you see in every single aspect, but I get to see it myself as well in my field. So, I still remember when we first published our public cloud providers reached out to us and said, "What is wrong with you Like what what is wrong with us?" And
00:37:55
they said that we're moving an insane amount of traffic between zones. Some of our backers are technologists and you know, when you deploy resources in the cloud, you have to split your resources across zones. You cannot sit in a single zone because zones can go down and they do go down uh quite often. So, we at at the first few days and the first few days by the way we were not peaking concurrency at all. A lot of backers chose not to play the first few days. They like to wait cuz a lot of them learned that we hotfix a lot of issues behind the scenes. Yeah. We were pushing 1.2 petabytes of hot traffic every single day and that's between zones. That's not between every single server. We still had traffic inside the zone. And that was the part 1.2 petabytes. Petabyte every day. Between zones.
00:38:42
Between zones every single day in one region, not in all of our regions. We exist in four regions. And that and that's just unheard of. Yeah, the Amazon guys were like, "You're we've never seen this before. Not like you're you've gone up from what you're used to." They were like, "We don't have any other customers that are pushing this kind of traffic." A day. And then when you think about it, like social media, a human being has to create content. That content has to be uploaded and it has to be interacted with and most of that is happening in eventual consistency or through compression. Uh the fact that every single simulator that we have and right now while we have instances with a reasonable amount of players, we have copies of our universe. All these copies are producing data as fast as the system can handle. And we
00:39:31
are a convicted group of people, so none of us is telling everybody else, "Hey hey, please slow down a little bit." No, everybody's saying, "Give me more." And that goes all the way down to the wire, to the database. And I I I know that a lot of players are whenever they hear the word database, they get really upset. That that the bit that I would like for some of them to understand is we didn't want to hold an our innovation behind the scenes for too long. They get to see right now we're testing server meshing and and uh uh server crash recovery and and tick preview. Each and every one of us cannot wait for that to go to live. So, 3.18.0 was one of these stories where we could have stayed longer till we figure out every single little problem that could happen, but they were hungry and thirsty and and really very eager to play that version of the game.
00:40:22
thank you thanks to all of them for for staying with us till we figured out some of the kinks out. We still still have a lot more to go, but we will keep pushing because that's what we do. That's why we're all here. I yeah, that's that's good. I appreciate that, Ahmed. Let's talk about We sort of we've we've we've moved into we've moved into the horror story phase of of of of of our of our show. Uh so, let's go back chronologically here. So, now you've all started at CIG. Now, I was just by I I think we almost started almost the exact same time. I believe so. What was your first What was your first month? What would What would your I think I was April 2015 or March I'm April 2015. Yeah, we started we So, I I was a contractor for a year before But I don't HR doesn't count that time, though. Mhm. You don't get benefit for that year?
00:41:09
Okay. No, my official start was April. So, so we've been the same uh you guys have predated us, though. Um I want to hear cuz I don't often get a chance to ask these things. Uh I mean we we see us now, you know, five studios around the world, you know, you know, 1,100 plus you you you you people, uh the infrastructure that we've got going on stuff, but I want to hear about those first days when when you started, you know, the basement in Austin because I never get to ask anybody about this stuff. Tell tell me what it was like. You get these opportunities, you come into this company, and it's a basement under a house in in Austin. This is not unusual for a startup, but but but talk to me about these days. You've got to have some stories. Yeah.
00:41:57
Well, you know, my first firefight came uh about 5 minutes after Chris announced the crowdfunding project and the website went down. Uh I got uh involved peripherally with that project because like I mentioned Chris's team was camping out in our office, and they we were comparing notes and so forth. And then at that time, we didn't have an office. Chris was uh you know, still getting things off the ground. There were a few members of our team that had already been working with Chris on the prototype, and I think everybody was working remotely or, you know, from around the world. So, in January of 2013 was when we got our first office, and it was in uh
00:42:47
in In Austin, there are these historic homes that are like three or four stories tall, and it's funny because uh this place where CIG had our first office, uh my wife had actually worked in that building before in an in an an environmental firm that had offices there on the first and second and third floor. And when I went to uh meet the team and really talk about switching from Portalarium to early CIG, uh they gave me directions, and I said, "Well, I know right where that is. That's perfect." And so, I go up, and I knock on the door of the first floor, and the the nice lady that met me there said, "Oh, no, you got to go downstairs." And she directed me to the basement. So,
00:43:35
it was funny when I told my wife, I was like, earlier I told her, "Hey, they're officing in your old building." And I had to go home and say, "Actually, they're in the basement of the building." And she said, "Oh, that's where the archaeology department was, you know." And that was kind of how it was for us in the early days. Um we had about When I joined, I think there were about 12 people that worked there. Uh and there were some people in Los Angeles, Chris, you know, and a few other people were in Los Angeles at the time. In the early early days, we had an out art outsource team that was in Monterey, Mexico. CIG bot. CIG bot with Sergio's team. We had another outsource team that was in Montreal at Behavior. And that's actually how we met Benoit
00:44:25
and the team at Turbulent was by a referral from Behavior to help us work out these website, you know, issues that we had because we overwhelmed our initial platform. So, anyway, that first office was a hoot. It was perfectly functional office. We did have a few windows even though it was mostly subterranean. Uh you know, for those you know, veteran backers, you can see I think there's a couple of walk around studio tours of that office that may be still hanging out on YouTube, and we used to broadcast the original Wingman's Hangar uh weekly show from there with the green screen and all kinds of stuff. You've come a million miles from that here. Congratulations. though.
00:45:11
Yeah, and so um that was an interesting experience. Uh we stayed there for about a year. We grew to the point where I think we had about 35, you know, maybe almost 40 people in that space, and it was very crowded. We literally had, you know, almost to the point of people sitting on each other's laps and so forth. So, it it got out of hand before we moved, but we moved into a nice new office in Austin uh in January of 2014, and in fact, we're still in that same office in Austin to this day. So, yeah. That's the office you started. Did Did you go into the current office, or were you in the basement? Oh, I started in the basement, and it wasn't a hoot. It was actually Maybe perceptions differ? Are you saying
00:45:58
Yeah, because by the time when I started, it was closer to the end of the of the basement. And so, it was overcrowded. And it was if I had to deploy a PC, it was you put it up like over, you know, to cuz otherwise you don't have enough room. You can't carry things around in there. Oh, the stern looks I'd get from certain devs if I bumped their chair leg or something. Um I I I came to the company right around the time that we launched the Hangar module for the very first time. And we didn't really have much, I mean, in the way of infrastructure. We had, I think, two tower servers. That was it. That's what ran the company. And uh there was no server room. It was I was introduced to my office, which was really a Harry
00:46:47
Potter style uh under the staircase office or closet. And this is also the server room. And I have to tell you this funny part is when I I cuz I, you know, I'm you're fresh in a new job, you want to make a good impression, but it's not making a very good impression on me because they have this black contractor tarp over the top of the of the servers. And I wondered, well, why is this? Well, it They pointed out that that's because the sewer pipes go over your head, and if they leak, we don't want that getting on the servers. But I didn't get a tarp. Anyway, that that office, we got out of there pretty quick after of there as fast as we could. I think I'd like to say that I was instrumental in moving the company to the new That's true. to the new office.
00:47:34
That's the truth. Wait Wait, what What is this? Is there a story? Uh we used a truck for one thing. Yeah. You mean you were instrumental in literally the physical act of Literally moving. Yeah, so the the uh one of the one of one of my first hires, uh a man named Chris, um I had worked with him before, and I uh brought him down from Canada to help me with this move. And he did such a great job, the company asked for him to stay, and he moved his whole family down, but he and I carried the core of the company, that like all of our intellectual property, source code, everything. And I I'll never forget cuz he would always say, "Slow down, we're coming up on a curve. You don't want to lose per force. You don't want to
00:48:22
That's the company back there. Be careful." Pivot. And Ahmed, when you started, we they were in the new location cuz you you started in them. Yes. What was your What was your first impression of of of the office and and and CIG at in April 2015 cuz I'm curious what your your impression was in August. My impression was in April. I I want to see if our impressions line up. Uh the first one was that when I went to my interview, I wore a suit. And I'm standing outside trying to get in, and one of our coworkers comes out in baggy shorts uh that have paint on them, and just runs right by me. And right behind him, I see one of our coworkers in tank tops. And I was like, I made an awful mistake. I'm wearing a full suit. And then I got in, and um
00:49:10
it just you see a dark space with a lot of artists and a lot of cool-looking people, and you just walking around with a leather notebook and wearing a suit. And uh I get to have my interview in that what we call the war room. And at that time, that that EC used to have a very interesting vibration, and I have a very sensitive ear. So, yeah, that was my very first impression. And I came first day to work, and I'm not sure if Mike will like the story or not, and I I a Windows PC. And I have not been a Windows user for quite some time. It's quite common in my field that people use Linux and it's like, "What is this?" You're like, "It's your computer." And I think that's how Mike and I got to be friends and know each other. He came to my office and knocked on my desk and he said, "My name is Mike Jones and that's all what you need to know." And walked away from me. And next night we were hanging out assembling my first Linux machine and
00:49:58
that's how things went. But if you're talking about memorable moment, I still remember our concurrency. We had times when we used to cheer because we have 200 players online and we have big magnitude above that right now. I still remember when we first launched it Area 18 and the whole QA team had their buggies with them by the elevator for every player to come in and ram into them. Okay. I have a confession on that. Right. Yeah. So we're about So okay. So we're about to put Area 18 in for the first time. And in my very one of my very first conversations with uh uh Jake Ross. Now I'm going to I'm going to I'm I'm going to I'm going to preface this by saying I didn't know
00:50:46
what my limits were or or or or or or or what what I should be involved in and what I shouldn't be involved in. Uh I was kind of all over the place in 2015. Anytime somebody needed something, we didn't have a person to do it. I'm like, "I'm going to do it." I was the events manager for a while. I was the I was I was I was the platform guy before Eric Green. Eventually like, "Wait, Jared's doing the platform? I think we're going to have to hire somebody." It's like, "Keep Jared away from the platform." Uh it's like but we're a day away from launching Area 18 and I go to Jake Ross and I say, "You know what would be fun? Put some Greycat buggies in there." It's you. And he goes "Oh, yeah, we should put some Greycat buggies and let people drive around." And it was just this innocent little suggestion and it took 24 hours to turn around and then when
00:51:34
three Area 18 launched for the first time, it was carnage. It was absolute carnage. It was peo- people were just mowing everybody down with the Greycats. The Greycats, you could look at them intently and they would explode. Mhm. And whatnot but uh yeah, it was I I Jake Ross did not throw me under the bus. When Chris goes "Where did the Greycats come from?" Whatever. And and he was just like it was it was a decision made internally. "We'll uh we'll take them out in the next build." But he he had a chance to throw me under the bus that day and he didn't. Yeah, so many so many great moments. Jake Ross. 2.0, I still remember I think it's one of the longest time I stayed awake. Not because I was able to do much. It was still like we were crashing but then watching the streamers just counting the
00:52:21
crashes every single time and she'd keep going back in again. Uh 3.0, I think one of our coworkers, I'm not going to name, was about to propose. And he was literally on his laptop right in front of their in-laws saying, "Hey guys, let's launch the game so I can go inside and meet these people for the first time prove that I'm a responsible man." So we have a lot of amazing moments and I think uh I I like right now we're talking a lot about the future because we're here to to have a summit laying out the plans for the next 2 years and we just keep dreaming of what 1.0 will look like and and how how we get to see the the game getting the shape that we all dreamed of. So yeah, it's very difficult to to to pick a certain moment but there's a lot of memories, a lot of a lot of sleeping under the desk. Think you and I one time on a call at 2:00 a.m. in the morning and I was leaning
00:53:08
back in my desk and I flipped backward. I'm not sure if you remember or not. And you were like, "That's it. Ahmed's gone." AHMED'S DOWN. AHMED'S DOWN. SOMEBODY, we need a well check. Yeah. I'm going to check on Ahmed now. The the the the the the the the proposal thing reminds me of 3.0. When you know, I made the stupid stupid bet with the community. Like I'm I'm on this show, the version of this show then Chris is on there. And and and and Chris is saying, you know, it's it's this is April or May. Chris is saying that 3.0 will be out by August. So 3.0 I'm like, you're you're going to say that? You're going to say 3.0's out in by August? He goes, "3.0 will be out by August." And the community goes, "No shave till 3.0." Oh, no. And Chris looks over looks over at my screen and reads and reads that off and and and and says
00:53:57
"You should do that. You no shave till 3.0." I'm like, "Are you serious that this is going to go to you you go go August?" He says, "Yes." And then it's December of 20 whatever. Yeah. And I'm in a bathroom, you know, with with razor just waiting in this long line and you know, when we launch the patches, there's there's this you know, go no go. You know, everybody's checking in, you know, the departments. It's very much like a shuttle launch. Mhm. And then it comes down to Aaron and Chris at the end. And Aaron's like, "So Chris, you ready to launch it?" And and and Chris is like "Uh let's wait a few minutes cuz I know Jared's in a bathroom or somewhere waiting to shave." And I'm only on this call for visibility. I'm not supposed to unmute in there. I unmuted that and I was like, "You like this And and they did and that was great. Um I also want to just because again, we've been doing this long enough and I I can see people like uh Ahmed mentioned 1.0 guys. We are always
00:54:47
talking about the far-flung future of of I of I was just I see of Star Citizen. Please don't read anything into that and whatnot. What's 3 323 next and then something after that and then something after that and then something after that. So just it's just I know you know, every y'all like to to grab on to little morsels and whatever. Please don't read any anything into We're we're always we're always talking about that far-flung future stuff and you have to know where you're going to you know, so. Don't do it to yourselves. Enjoy where we're at. We need them in black flash. I'll just flash it in front of the camera. Uh so yeah, so um gosh, we've only got a a few minutes left. Um I don't want to I know you've all got
00:55:34
you're here representing the work of dozens and dozens and dozens of people and whatever and I'm not going to put you on the spot uh and ask you to remember everybody's names right now because inevitably you'll forget somebody and somebody will But is there some kind of maybe just shout out the the teams maybe the individual teams that that all work under you uh and what the responsibilities are and stuff like this and just give them a few moments recognition. John, you're at the Actually, we'll start with Ahmed. He's He'll have less people. Ahmed, who who are who who You want to shout out some teams that Absolutely. We're We're a lean team so I can actually mention all the names. A lot of people put a lot of time and a lot of effort to to ship and maintain and keep Star Citizen going. Bryce, Jeffrey, Anthony, Loic, Haitham, Scott Justin, Jared. Like it's it People get
00:56:24
to see my name sometimes. Some of the old backers know me. It's not just me, guys. There's a lot of folks and without them I would have done nothing. There's so many hands in the keyboard. There's so many eyes in the screens and it takes a lot of very sincere effort to keep the game going and being published the way it does and just keep hoping the best for us, guys, and we'll keep serving you as good as we can. Uh Mike. I do have quite a few people and I don't want to put the whole list. Yeah, I don't say I would need some time for that. But yeah, the way we're set up, um we have we do have key functions and the teams that support them. So there's IT managers at each of our studio locations. Our largest location has two. each of those are supported by a really
00:57:14
cool team. And these are people that are really passionate about what they do. And you can tell when you interact with them. Mhm. Uh and then in Austin, we have additional specialists. So you know, from network engineering to systems engineering, administration security, cybersecurity even specialists. And so rather than naming them all out, I'd like to kind of call them out in that way. The people who manage all the licenses and stuff. Well, yeah. I already said his name. That's Chris. He's a one-man army. Yep. Yep. And I'll I'll give a special shout out to Chris. He's one He's He's one of the people anytime I message him, he is right there, Johnny-on-the-spot with a with a with a response. And whenever he doesn't respond in just a few minutes, I'm like,
00:58:02
"Are you okay?" He He He's like, "Oh, I just had to go to sleep." I'm like, "Okay. No, you're you're you're you're entitled to sleep, man." You know, because he's usually he's he's really good. It's like that. We like that. I was just thinking you just all the software You mentioned the licenses earlier and my mind would just started to think about, you know, cuz we're so we're so enabling of people. It's It's like when people come and say, "Well, what software do you use to do Oh, you're you're better in this than that." We build tools to let these softwares talk to each other and everything. But I just I I imagine what that must be like when someone's like, "Hey, we need to add another piece of software." And just Yeah. You just wait till Chris Roberts has like voodoo dolls of all of us. You know, it it we're a really interesting company even in the game space because we are equal parts game company, technology company and uh
00:58:53
production facility. I mean, we are in a beautiful set here that that you operate in. You have very specialized needs that are very different from Ahmed's team or uh average game designer or the finance department or whatever. So as a as a corp tech publishing organization, we're supporting you know, everything from mocap to legal document needs to cybersecurity to all kinds of stuff. It Like we said, it makes it really interesting because no two days are ever the same. No. Yeah. Yeah. No. And I still never know what day it is and you can Yeah. of the week. Uh and John, you've got so many teams under you. Yeah, well, we've talked about quite a few things, but I would like to, you know, I can't think of everybody, but I'd like to call out for sure, you know,
00:59:42
Eric Green and our publishing operations team. They handle all a lot of our web platform, everything that you see on the site when there's a major sale or a major event, their team has had a hand in coordinating and preparing that for public consumption. Our marketing teams and art teams and stuff. They handle the actual they come up with the ideas and the content and so forth, but Eric's team really makes it ready to consume, you know. That his team's also include our release management team, which is critical for the day-to-day operations of Star Citizen. We have, you know, in a given day that team is probably considering three to five different builds that may go out to the PTU
01:00:30
environment or to one of our special testing channels. And so, you know, Jeff and Marissa and Scott are, you know, they're up early talking with the team here, Francis and Matt who are, you know, been coordinating the development activities throughout the day here. They wake up in Austin, we have a handoff, they take the baton and then by the time the evening rolls around, you know, we've got probably a new build that's out there for people to try or or or not as the case may be. We have Will Leverett who heads up our player experience and our live QA teams. So, when we do have a build candidate that we're thinking about putting out on PTU, the live QA team with Bayor and Waka and some of these names that the community will know very well,
01:01:17
they're instrumental. We have a whole team of people that will test this build internally, make sure that, you know, hopefully there's no spoiler content in there, that it runs as expected, that there's not problems with, you know, all the different back-end systems that have to be checked. And then Will Scott's other leadership team, John and John, there's two different people named John. We have a joke amongst us, we've got a chat room called the three Johns. So, we all need to come up with snappy nicknames, I guess, but anyway, they will handle customer support and e-commerce support and all the different pieces that go along with that. We've got our whole team in Montreal that's handling all sorts of different
01:02:05
platform technology development and web development and e-commerce development and that's headed up by Doreen. So, we've got our build department that handles a variety of things. It's headed up by Miles. They're they're constantly, you our our development process is very rapidly evolving and so they're always providing build system support and making sure developers can have access to the latest builds. I was just looking at a report in the fourth quarter, you know, we generated, you know, like thousands of builds of various flavors for, you know, every team has their own custom stream that they're working in for the animation team and the audio
01:02:54
team and whatever and so his his department really keeps that all running smoothly or as smoothly as possible. We have our data solutions team and the list goes on and on. So, it's exciting to see how much it takes to make it all work. Shout out to Miles. When when I I I went off the reservation at some point last year. We had ended the show early and whatever and I got out a whiteboard and I drew the entire build system and how that stuff works. And as I'm doing it, I'm like, I really should have talked to Miles before. And then I'm I'm I hope I don't mess this up. And and and I did it and I the show stopped and like the first thing I did was go back to the computer and look at look at teams and just just expected to get
01:03:42
shouted out by Miles. He goes, "You did pretty good." Yeah. Pretty good. I was like I was like That's a good Miles impression. Congratulations. Only because I learned from the best, Miles. That's pretty good. Yeah, that's All right, so yeah, so that's that does it, guys. You did it. You survived an hour long Star Citizen Live. How you feel? Okay. I feel pretty good. Let's do it again. All right. We'll we'll we'll do it again next time you come over. Thanks for hanging out with us. Thanks for watching. If you haven't checked it out, this week's Inside Star Citizen was on the branding team out of Montreal. Lots of cool looks at their work and their contributions to maintain the overall immersive sense of the persistent universe. Next week, you can return back to Inside Star Citizen. We're doing our what's probably going to be our last look at distribution
01:04:29
centers. The big major microcosms of the entire persistent universe that are scheduled to come out in Alpha 3.23. We'll be looking at that and the variety of mission content stuff that happens around them, within them and through them and stuff like that. And then we'll be back here for Star Citizen Live next week. We're hoping to have Team Kean and the folks behind the EVA and the personal interaction system and the deployable action episode from a few weeks ago. We'll have them back on. We'll collect questions from you and you guys can send your questions to them and put them on the hot seat. So, for Star Citizen Live, that was Mike, that was John, that was Ahmed. Don't forget that Coramore Lunar New Year and a free flyer probably going on right now. You can check all that stuff out on the robertsspaceindustries.com website. And I think we have a we have we have a raid going on.
01:05:18
It was somebody and then they went offline and then it was somebody and they went offline and now I've lost track of it. So, we'll start the raid. Thanks for watching everybody. Thanks for supporting the project. Thanks for letting us chat with folks like this and we'll see you next week. Take care.

Transcript from YouTube’s automatic captions — punctuation and Star Citizen jargon are approximate. The video itself is hosted by CIG on the official channel.