PCG Article QA's importance in the gaming industry

[DXCHASE]

https://www.pcgamer.com/quantic-QA-tester-report/

I think articles like this are highly important for any avid video gamer. Understanding the industry itself is crucial to me, I believe its important to understand what it takes (or supposed to) when making a game. QA testers are seemingly considered to be the 'lower-class' of game development, even though, say Cyberpunk 2077 for example (one of the examples used in the article) couldve used a lot more TLC in this department before the game released.

The part of this article that got me the most was the wages. They are just awful. I can imagine living on 800 a month sucks and feel that QA testers should recieve more pay and maybe proper training from these companies that hire them, amongst many other things that could be different.

Plenty of games ive played that i wished they wouldve invested more time into QA testing (OUTRIDERS).

Thoughts on QA testing ?

 

[Zloth]

Well, one big thing: we don't know what bugs have been reported. When QA says there's a bug, that doesn't mean the folks that need to fix it drop everything to eradicate the imperfection. From our vantage point, we can't really tell the difference between a bug that QA found but didn't get fixed and a bug that surprised everyone. You'll have to be part of development to have any hope of telling a bad QA department/contractor from a good one.

QA can be relatively simple or really hard. If all you ask of QA is that they give a quick report of what was happening at the time along with a dump of the machine state and a video of the last X minutes, that's not going to require a lot of training. If you want them to gather more information about a bug, though, having a lot of experience and understanding of how computers do what they do becomes necessary. So, if you ask them to go from "I finished the dialog with Joe Schmoe and it said he took my sword, but it was still in my inventory" over to "Sword doesn't vanish from inventory after Joe Schmoe's quest if the third dialog option is used AND the sword still has a buff/debuff spell on it - but it will vanish once the spell expires and the character is reloaded either via loading a save or moving to another zone" then more expertise is going to be important.

 

[mainer]

I think that article gave some interesting insights into what can actually happen in the QA quality assurance/tester stage of a game. It's something that we, as game players, don't really see and probably take for granted. I'm sure the process varies widely depending upon the game or developer, but the CP2077 example was particularly insightful.

I look at RPGs, especially open world RPGs like Skyrim, and the tens of thousands (or more) of possibilities that any given player might encounter based on his/her choices, and there's no way any group of QA testers will catch every single bug.

Which is also why I like to see games go into Early Access on Steam, thousands of different players all playing a bit differently are going to catch & report more bugs than any QA team will ever see. I see a lot of complaints about Early Access games, especially on Steam Forums, such as "I'm not going to pay full price for a game just to be a beta tester". That's fine, everybody has an opinion, I just think in the long run it makes for a better game.

Larian has used this model of Early Access for the Divinity OS games, as well as their current game Baldur's Gate 3, and I think they've had great success with it.

 

[WoodenSaucer]

It's obvious from the state of games when they launch that the quality control/assurance isn't nearly as robust as it needs to be. Game companies have gotten in the habit of putting out unfinished games, knowing they can just patch them later. I don't blame the QA teams for that, though. Nor do I blame the dev teams. I blame game companies' corporate management that wants to push games out before they're ready so they can start lining their pockets.

A couple of thoughts about the actual QA jobs. I agree that playing games to try to find bugs isn't a skilled trade on the same level as programming or developing the games. But can you imagine your job being to play upcoming games all the time? That would be awesome. But in the first post, DXCHASE said that he can imagine living on $800/month sucks. You wouldn't live on that. Those people either live in their parents' basement, or they have a second job.

 

[Brian Boru]

can you imagine your job being to play upcoming games all the time? That would be awesome

But that's not a QA job—that's a content creator like journo or YouTuber etc job. QA is usually performing a detailed and specified series of operations, and cataloging the results—almost definitely a real slog after any initial gloss has worn off.

tens of thousands (or more) of possibilities that any given player might encounter based on his/her choices, and there's no way any group of QA testers will catch every single bug

Absolutely right.

Now add in the billions/trillions of possible states for the machines that game will be played on. What hardware parts & versions & drivers? What OS version and build? What software is active while the game is in play?

Very few people realize how impossible it is to check for breakages which will occur among the general public.

 

[Foxhood]

I'm more in embedded systems development, rather than full-blown software.

But i do know that a good report is worth its weight in gold. Sure end-users would find and file reports too, but you often end up with a flood of them which you gotta sift through to check for duplicates and/or search if any have a lead to what exactly went wrong. It costs a LOT of time and resources as you gotta check, fix, try to pry extra info loose from those suffering (like logs) and deal with the PR fallout for having released such a "buggy mess".

Good quality Quality Assurance is vital on beating that kind of stuff to the punch as ideally you would get a single good report per bug nicely filed with all resources needed to rapidly track down and fix. Saving time and effort that quickly add up over time.

But alas. It seems that the higher-ups often are blind to how skimping on QA in practice tends to lead to greater expenses and problems in the long run. They'd rather assume it "won't be that bad" and brag about how cost-effective they are now, rather than later...

 

[Brian Boru]

higher-ups often are blind to how skimping on QA in practice tends to lead to greater expenses and problems in the long run

Finance used to have functions like QA and Customer Service as cost centers rather than profit drivers. And Finance always has a big say, especially in public companies where money is the main metric.

 

[WoodenSaucer]

But that's not a QA job—that's a content creator like journo or YouTuber etc job. QA is usually performing a detailed and specified series of operations, and cataloging the results—almost definitely a real slog after any initial gloss has worn off.

I know they have a list of things to do and check for. But part of that is going to entail playing the game. But I agree that the methods they have to use to do it would probably make it a major chore instead of just having fun with it, like we would.

I used to work with a guy whose brother was QA for movie companies. His job was to watch DVDs/Blu rays of movies that hadn't come out yet and test the discs out. It's a job, but the guy got to watch every movie before anyone else had a chance to.

 

[Zloth]

But alas. It seems that the higher-ups often are blind to how skimping on QA in practice tends to lead to greater expenses and problems in the long run. They'd rather assume it "won't be that bad" and brag about how cost-effective they are now, rather than later...

They don't seem to want to fix the bugs, either. The software industry as a whole seems far more interested in adding new features than making existing features work better. Games seem to be a little BETTER than most about fixing non-critical issues!

I don't know what the reasoning is. I'm sure some is just a workflow thing. Features are typically things somebody asks for, then people figure out how it can be done, then it gets put on a schedule, and it goes out when expected. (More or less.) (Mostly less.) Bugs, on the other hand, just crop up. Nobody has planned when they should be done. Putting fixes on the schedule means pushing one of the planned things off the schedule. All in all, management would rather they just didn't exist - so they tend to ignore them.

Programmers can be guilty of it, too. They've got their own schedules of things they want to get done. When you get one done, you want it to be DONE! But no - you've got to go back in, remember what the frak you were doing (hint: COMMENTS!), and figure out a way to fix it. It's like hearing that the game you finished two months back has a boss you somehow skipped - now you need to install the game again, find a save kinda near the boss, remember what you're doing in the game, and win again.

P.S. Assuming you don't hate game bosses. It's not as bad as what @Brian Boru is thinking right now, thank goodness.

 

[Brian Boru]

Features are typically things somebody asks for

Yes, they will drive profits thru new sales.

Bugs … management would rather they just didn't exist

Yes, they're seen as a cost, an expense—which is the opposite of what managers are typically measured on.

With so many functions being outsourced these days—finance, HR, legal, etc—there should be scope for a bug fixing industry. C'mon, you young go-getters! There's already a talent pool just sitting there in the modding communities.

 

[Pifanjr]

I don't blame the QA teams for that, though. Nor do I blame the dev teams. I blame game companies' corporate management that wants to push games out before they're ready so they can start lining their pockets.

I'm past the point where I blame management, I now blame the consumers. Companies have one surefire way of measuring what the public wants: whether they'll pay money for it. The fact that games keep being released broken only makes sense if it makes more money than spending more time polishing the game.

And it's not like there's a shortage of good video games available. With physical goods it might happen that good quality products are too expensive or too hard to get, but that's not something gamers need to worry about. People choose to buy broken games, at this point I don't think they can claim ignorance any more. Game companies are just providing what the people are asking for.

 

[WoodenSaucer]

I'm past the point where I blame management, I now blame the consumers. Companies have one surefire way of measuring what the public wants: whether they'll pay money for it. The fact that games keep being released broken only makes sense if it makes more money than spending more time polishing the game.

Yeah, I hear ya. But I've already moved past that stage, and now I'm at the next point where you're back to blaming game company management, because they're the ones who created us and turned us into the addicts we are.

 

[Brian Boru]

I'm past the point where I blame management, I now blame the consumers

Hmm, I dunno, I think companies have access to too many tricks to manipulate consumers. They did create consumers after all ~100 years ago when modern marketing was born to artificially create demand where none existed at the time.

Not black & white of course, consumers have done well too.

 

[Pifanjr]

Yeah, I hear ya. But I've already moved past that stage, and now I'm at the next point where you're back to blaming game company management, because they're the ones who created us and turned us into the addicts we are.

Hmm, I dunno, I think companies have access to too many tricks to manipulate consumers. They did create consumers after all ~100 years ago when modern marketing was born to artificially create demand where none existed at the time.

Not black & white of course, consumers have done well too.

That's true too, marketing is still hugely influential and often bends the truth or straight up lies without any repercussions.

 

[Shodan_]

I quit a position as a corporate trainer in public security and 911 dispatch to do QA in a mobile app company. I have nothing to prove anymore and standing in front of a class of police officers after sleeping 3 hours in a shite hotel room gets old after 7 years.

I wanted to get back to gritty tech and that's what I got. Only thing I miss from the old job is the money, although in my position it is much more than 800 bucks a month. The gaming industry is straight up shafting their QAs.

My main project is a sports app with live betting, live results, live video players, fantasy leagues and such. All these live things have to me mocked from the ground up in order to be verified without actual games or auto races being on.

Baseball is cool because they play in the afternoon so I know when I'm working on that.

All this to say, I spend more time in back end DBs and debugging proxies analyzing API call/responses and mocking Json states and data than I spend navigating the app itself. I'm in direct and constant contact with 8-9 devs and have my own branch for pull requests. I'm much more meticulous than the person before me who is also very good, just different type of QA attitude (I would say more efficient in high level things while I'm more detail oriented and can forget the big picture sometimes, still a noob at this), we still miss a bunch of things. Like there are so many effing screen ratios in mobile, devs have to account for massive variation in everything.

QA as domain is huge. It goes from user layer testing to deep involvement in the very structure of things. I have friends who also do QA who have never heard of Charles Proxy while it's my main tool. I don't think I would enjoy play testing much as I want to be messing about with bits of code and things like that.

Games and the expectation of games in modern times are impossible to reconcile with bug free products. It was actually never possible but the scope has increased so much it is now more apparent than ever. The reality is for every bug fixed there are 20 more reports of other bugs that won't get fixed, because fixing them all means the product would never come out. When I read posts from people far removed from dev work who will blame QA for train wrecks like CP2077 it makes me laugh.

As if armies of playtesters would've somehow completely missed the absolutely insane amount of problems that you're now experiencing.

 

[Zloth]

Arg, you're making me salivate, @Shodan_. All we ever get for testing is some spare time from a help desk person that only knows the basics of how our applications work. Users rarely tell us anything. The main way I find bugs is by having my apps email me!

 

[Shodan_]

Arg, you're making me salivate, @Shodan_. All we ever get for testing is some spare time from a help desk person that only knows the basics of how our applications work. Users rarely tell us anything. The main way I find bugs is by having my apps email me!

That's one way to do it.

You might want to look into services like Firebase Crashlytics. It will log and graph out automated crash reports over time by version and platform and spit out a detailed stack trace for every one of them so even if it's something you can't reproduce you can tell what's going on.

Also if I'm testing something in the QA or staging branch and I get crashes I can gather up that info and forward it to the dev team.

One of the most useful thing is sometimes people don't have their automatic updates on and ignore app store notifications, so you get spikes in crashes or complaints after a release and you can tell these crashes are from users who are 2-3 versions behind and this new release is the one that breaks everything but you won't scramble around to figure it out because at this point it's not your problem if 2% if your userbase isn't up to date, even though that 2% still amounts to a few thousand people so it looks like a nuke went off. You can then post that kermit tea meme in the slack channel and resume your good work.

 

[Zloth]

Nah, no crashing, just buggies. These are desktop apps, for the most part, and back ends for web pages. Even the desktop apps use ClickOnce, so I can pop up a message box to tell folks when an update is available so restart or there will be.... trouble. (I would put the Robocop clip on the warning, but SOME PEOPLE will make comments about professionalism. Mostly folks who control my paycheck. Harrumph!)

 

[Zimbaly]

Over the years I've played with a few game testers. One in particular was part of our Left 4 Dead crew.

I told him when I was younger I wanted to do what he did, and I just didn't pursue it cause others told me just to avoid that job. Well he said they were saving me a headache. He's like picture this, you get to work, you get your coffee and turn on your pc. You sit there and load up the game and it crashes. So from there you get the log and send it in to the programmers and wait for them to fix that issue, but while you do you have keep trying to get the game to run. Everytime you had an issue it would be logged and sent.

Finally they fix the issue and the game boots up and going to the main menu. You finally have a game that loads and click to go to the next menu or start the game...... it crashes. So again you get the log and send it away waiting for it to be fixed, but while you do you have to keep loading the game and try to get it going, over and over and over.

Now finally the game gets past the main menu and loads into the game. Finally happy you push your controller to walk forward and it crashes. It crashes when ya move left, right, back, hit any button, but maybe it won't crash when ya hit start.

This goes on all day, everyday till the game starts to get ironed out. Over and over and over, for months on end and as time goes on it drives ya bonkers. They fix the crashing that happens when ya move all around, push most buttons, but now the start button push that worked before now causes the game to crash. Guess what happens next. This all could go on for months. Load your pc for it to crash, all day, everyday. It's not a job for everyone.

He loved and hated his job, not just because of how much that part of the job could get more then annoying, but also cause of the content he'd see not make it into games. He said bad decisions would be made to remove something great in a game for something that wasn't even close to being as good as what was removed or they had to cut it for reasons of time. I still remember him saying for how much you loved Warhammer Space Marine, you would hate it if you had seen what was cut. It always made him bummed knowing how much better that game could of been.

He also told me about Mr. Toots, one of the greatest weapons in video game history.

If anyone is looking at getting into this field, don't let this post scare ya off, you may love the job, just be aware it's not all fun and games.

 

[Brian Boru]

This all could go on for months. Load your pc for it to crash, all day, everyday

That sounds like the devs or their coordinators did no checking at all. That's most unusual in the dev situations I've heard of.

In the multi-dev or multi-studio situation, there are usually senior devs at a coordinator level, whose job includes taking the various components from the different sources and making sure they work together at a basic level on their equipment. If not, review with dev/studio and get fixed.

QA is for testing the non-basic levels, different configs, different user choices etc.

There's a basic principle for saving time and money, which is to discover flaws at the earliest possible stage—dev tests his bit of code, coordinator tests bits work together, QA test a tiny fraction of possible user configs, pre-order customers test a much larger fraction of configs.

 

[Zloth]

I wonder how automated testing fits in to game QA? When it comes to a program that can't even start, there's no point in having people work on it. The build server should be able to figure that out.

 

[DXCHASE]

I'm past the point where I blame management, I now blame the consumers. Companies have one surefire way of measuring what the public wants: whether they'll pay money for it.

This is exactly how i feel with Nvidia and their GPU's. The prices are the way they are because of the consumer i mean the 4090 is already reaching 3k and im 100% certain someone will buy it.

In relation to QA testing, its a good thing Nvidia has QA after reading that the 4090ti was cancelled because it was melting psus.

 

[Brian Boru]

I wonder how automated testing fits in to game QA?

Should be a huge component, probably over 95% by now. Maybe that's part of the current problem, companies relying on it too much before it's ready for prime time.

I had fun with Sikuli X a while back, made a couple of scripts with it to automate going thru an app's work process by using visual clues in the app's UI—not cos it was best way, but cos Sikuli is cool Add the current major advancements in image processing AI and a very complete auto-testing process can't be far away, where humans are only needed for exception management.

However, auto-testing will only be able to make a small dent in bugs discovered in the field once they figure out a way to iterate thru all the possible configs that users could have. That's likely a long way off—maybe needs quantum computing?—since with current tech, the time required to cycle thru trillions of combos would take too long.

 

[Rolfil]

I wonder how automated testing fits in to game QA? When it comes to a program that can't even start, there's no point in having people work on it. The build server should be able to figure that out.

I think you really have to separate out testing and QA. When I was doing Software Engineering then small dev teams would test their own parts. This always used 100% automated tests. You could even make a small change, submit it to your own personal code repository and these tests would automatically run.

Above that level you would have some level of automatic Integrated tests.
To me QA was the very last level of testing. My experience with them was terrible. You really didn't want to give them stuff with a load of bugs in it. They were rewarded by basically how many issues they raised and the relationship between QA and devs could be really toxic.