Game testing
From Wikipedia, the free encyclopedia
 |
It has been suggested that Game tester be merged into this article or section. (Discuss) |
Game testing refers to the process of software testing for video games. The formal process, performed by professional game testers, begins well into game development, depending on the game's allotted budget for testing. High-budget flagship titles (called "AAA", pronounced "triple-A") may begin testing with the first playable alpha, while low-budget and casual games might not enter testing until a release candidate (RC) is ready. Professional game testing may involve general playtesting but should not be confused with "public betas" or "stress tests" in which players simply do as they please and report bugs that are found incidentally. Public releases such as these are almost certainly late betas or RCs.
Contents |
[edit] Areas of testing
[edit] Functionality
This is the type of testing that is most commonly associated with the phrase "game testing", as it usually entails playing the game in some form. Functionality testing does not require much in the way of technical knowledge, though understanding basic concepts about programming can give testers insight about the possible cause of a problem, which both reduces the amount of work they have to do, and generally makes their reports easier to understand by the developer. Functionality testers look for general problems within the game itself or its user interface. Major issues include stability (does the game crash?), correctness of game mechanics (does the sword deal damage?), and integrity of game assets (are textures corrupted?).
[edit] Compliance
In large part, compliance testing is the raison d'etre for the existence of game testing labs. First-party licensors for console platforms (e.g. Nintendo, Sony, Microsoft) have strict technical requirements for each title that is licensed for their platforms. Sony publishes a Technical Requirements Checklist (TRC), while Microsoft publishes Technical Certification Requirements (TCR), and Nintendo publishes a set of "guidelines" that serve a similar purpose. Some of these requirements are highly technical in nature and fall outside the scope of the game tester; however, other parts, most notably the formatting of standard error messages, handling of memory card data, and handling of legally trademarked and copyrighted material, do fall within the responsibility of the game testers to test. If even a single violation is found by the first party during submission for license approval, the game can be rejected (kicked back) to the publisher, possibly costing tens of thousands of dollars in additional testing and resubmission fees. In addition, the delay may cause the title to miss an important launch window, potentially costing the publisher even larger sums of money. The title cannot be published for the system without first-party approval, so it is in the publisher's financial interest to ensure that the title is tested for compliance as rigorously and as frequently as possible.
The requirements themselves are proprietary documents released to developers and publishers under confidentiality agreements. They are not available for the general public to review, although familiarity with these standards is considered a valuable skill to have as a tester.
Compliance may also refer to regulatory bodies such as the ESRB, if the game targets a particular content rating. Testers must flag "objectionable content" in the game that may be inappropriate for the desired rating. As with licensing, games that do not receive the desired rating must be re-edited, retested, and resubmitted at additional cost.
[edit] Compatibility
Compatibility testing is normally required only for PC titles, and testers assigned to this task are expected to have at least basic knowledge of PC hardware. An individual who can assemble their own PCs and setup an operating system is minimally qualified for this job. Compatibility testing differs from functionality testing in that the tester does not really care whether the game itself is correct, but simply that it runs on different configurations of hardware. The tester is usually provided with multiple PCs and/or extra parts. This collection of hardware encompasses various brands of CPUs, video cards, sound cards, and input peripherals such as gamepads and joysticks, and possibly other components as well. The tester is also expected to cross-test different hardware configurations with different operating systems. In addition to a basic pass/fail test, the tester may also be asked to evaluate performance. The results of these performance tests will be used to determine the game's advertised minimum system requirements. In some cases, compatibility or performance issues may be fixed by the developer, especially if a popular brand and model of a certain component is affected. In other cases, especially with legacy hardware or software, supported will simply be dropped, and the game's system requirements will reflect this.
[edit] Localization
While general text issues are a minor part of functionality testing, some QA departments use dedicated localization testers. This is especially true with Japanese publishers. Early Japanese game translations were rife with Engrish, and in recent years this has become a more serious concern with Japanese developers as games have taken on a more cinematic presentation. To maintain the impact of the drama, localization testers are employed to review the game's script (which may be thousands of pages long for RPGs or other story-driven experiences), looking to make technical corrections as well as reviewing the translation work itself to ensure that it matches natural speech patterns. Thus, a localization tester acts as an editor. Though localization testing is currently a mainstay of Japanese game development, it will likely expand as the Chinese, Korean, and Russian game industries gain global traction. Those seeking work in this area do not necessarily need to speak foreign languages, but being able to understand the original texts is a helpful and desirable skill. Unlike other forms of testing, however, localization may involve very little actual game interaction. In some cases, the only times that the tester needs to play the game is to understand the context of dialog or interface messages that are not immediately obvious. The game's "script", which is not a script in the literary sense, but rather a collection of all the text strings referenced in the game, is typically cataloged in a spreadsheet that the tester is given to work with.
[edit] Soaking
Soaking, in the context of video games, typically involves leaving the game running for a period of hours or days in various modes of operation, such as idling, paused, or at the title screen. This test requires no user interaction beyond initial setup, and is therefore usually managed by lead testers. The purpose of soaking is to detect the presence of memory leaks or rounding errors that manifest over time, resulting in system instability or odd game behavior. Soak tests are one of the checked requirements for first-party compliance and must be performed on every title. Because it requires no user interaction, soaking is also a euphemism for employees shirking their duties (i.e. not playing the game).
[edit] Methodology
There is no "gold standard" method for game testing, and most methodologies are developed in-house, by individual video game developers and publishers. And, methodologies are refined from time to time and may differ for different types of games (for example, the methodology to test a MMORPG will be different from the testing for a casual game). But outlined below is a typical methodology of game testing.
- Identification: Incorrect program behavior is analyzed and identified as a bug.
- Reporting: The bug is reported to the developers using a defect tracking system. At minimum, the circumstances of the bug and steps to reproduce are included in the report. Developers sometimes request additional documentation such as real-time video of the bug's manifestation.
- Analysis: This step of the process involves the developer responsible for the bug, such as an artist, programmer or game designer. It is outside the scope of the game tester's duties, though if there are inconsistencies in the bug report, the tester may be approached to provide more information or evidence.
- Verification: After the developer claims the bug to be fixed, it is the responsibility of the tester to verify that the bug no longer occurs. Not all bugs are addressed by the developer. Reported bugs may be claimed as features (often expressed as "NAB" or "not a bug"), and may also be waived (given permission to be ignored) by producers, game designers, or even lead testers, according to company policy.
- Regression testing: As game development progresses, closed bugs are checked again to ensure that they have not regressed, or re-manifested. For example, a programmer might undo a fix, having forgotten why the change was made in the first place. Though tedious, this type of testing is important since unexpected changes to code can cause old problems to resurface in the same or different ways.
[edit] As a career
Within the game industry, testing usually falls under a title such as quality assurance (QA). However, game QA is far less technical than general software QA. Many game testers have only a high school diploma and no technical expertise. Game testing is normally a full-time job with expectation of regular overtime, but employees are usually hired as temps and the length of employment varies. In some cases, if the tester is working for a publisher, the tester may be sent off to work at the developer's site rather than in his employer's own offices. The most aggressive recruiting season is late summer/early autumn, as this is the start of the crunch period for games to be finished and ship in time for the holiday season.
A common misconception is that professional game testing is akin to a public beta test or stress test, where players are expected to enjoy the game and report any bugs they happen to find. Testers who do this in QA are fired in short order. In reality, game testing is highly focused on finding bugs, often using tedious methodologies. Even if one could play the game freely, there is no guarantee that the game is stable or fun enough to be enjoyable. A tester may be required to play the same portion of a game repeatedly for hours at a time. Understandably, burn-out is common in this field.
Despite the demanding and risky nature of the job, game testing doesn't pay a great deal and is usually paid hourly, with wages ranging from USD$8 to $15 per hour. As temps, testers typically receive no benefits or holidays and simply take unpaid vacation days when desired. Many game testers take this job because they enjoy gaming as a hobby; unfortunately this "love of the game" is exploited to its fullest by employers. Some testers use the job as a stepping stone in the game industry, but the success of this strategy is unproven, and depends on which part of the game industry the tester desires to work in. QA resumés, which display non-technical skill sets, tend towards management, then to marketing or production. Those wishing to land a job in programming, art, or design usually need to demonstrate their skills in these areas, either by taking jobs outside the industry and/or working on mods.
[edit] Hierarchy
The organization of any test group differs between organizations, but a typical one may include the individuals below.
- Game producer: responsible for the game as a whole. Producers are typically assigned on a per-game basis. They look at the game from a broad perspective, and rarely on a daily basis. In addition to testing, they are responsible for setting deadlines in coordination with marketing. Their approval is required for final submission or "gold" status.
- Manager: responsible for all games assigned to his testing department. The manager rarely tests games himself, but instead coordinates all of the different game projects and ensures that they are on track. They also communicate with (or act as) the human resources department, making decisions for hiring, firing, or settling other issues related to employees, though not on a daily basis. The manager gives final test approval for any project, possibly overriding the recommendations of those beneath him. Managers in turn can be overridden by a producer, though this is rare due to liability concerns. However, if a project is extremely late and/or over-budget, the producer may be forced to override QA's recommendation.
- Supervisor: the most visible authority in the office, supervisors are typically long-term and experienced employees who interact directly with testers and serve as a middleman between them and upper management. Supervisors are usually assigned to multiple titles and may test each to a limited degree. They also handle menial management tasks such as collecting employee timecards. Like managers, supervisors sign off on projects, but mostly in an advisory role. The QA manager will usually base his approval on the supervisor's, since the manager typically does not have intimate knowledge of the game's issues.
- Lead Tester: assigned on a per-game basis. They are ultimately responsible for the testing that occurs on that title, and may communicate directly with the supervisor, manager, or producer where the game is concerned. They are responsible for filing daily reports on bugs and progress. Senior leads are also often the ones responsible for writing test plans, which are then approved by the supervisor, manager, and producer. Junior leads may receive assistance from the supervisor or other leads in creating test plans. Leads are expected to test the game whenever they are not performing their lead duties, and they are responsible for approving bugs and maintaining the bug database for their project. Lead testers do sign off on projects, but have almost no influence over the final decision to ship; their recommendations are usually redundant with the supervisor's since both are mostly aware of remaining issues. Lead testers are also typically responsible for compiling representative samples of game footage for submission to regulatory bodies such as the ESRB.
- Assistant Lead Tester: Though fundamentally not much different than a regular tester, an assistant lead is generally recognized as having more experience and/or being of higher caliber, and may be groomed into a full lead position. The assistant lead may handle the more menial tasks of leadership, such as training new team members or dealing with trivial issues. He may also be asked to fill lead duties if the lead tester is absent. Assistant leads may be given higher permissions in the bug database and help with maintenance, though they usually lack full permissions. Assistant leads may or may not be asked to sign off on projects, but in any case their opinion is largely unimportant and is usually identical to the lead's.
- Testers: more formally called "Analysts", these are the rank-and-file workers of varying commitment and skill who are expected to test whatever game they are assigned to. Testers may be assigned to a single game during its entire production, or they may be shuffled around or temporarily brought onto other projects as demanded by the department's schedule and specific needs. They report directly to the lead or assistant lead for any game-related issues, but still go to their supervisors for more general issues concerning employment. Regular testers usually have minimal database permissions, with the ability to add bug reports that must be approved by a lead.
- Lead Tester: assigned on a per-game basis. They are ultimately responsible for the testing that occurs on that title, and may communicate directly with the supervisor, manager, or producer where the game is concerned. They are responsible for filing daily reports on bugs and progress. Senior leads are also often the ones responsible for writing test plans, which are then approved by the supervisor, manager, and producer. Junior leads may receive assistance from the supervisor or other leads in creating test plans. Leads are expected to test the game whenever they are not performing their lead duties, and they are responsible for approving bugs and maintaining the bug database for their project. Lead testers do sign off on projects, but have almost no influence over the final decision to ship; their recommendations are usually redundant with the supervisor's since both are mostly aware of remaining issues. Lead testers are also typically responsible for compiling representative samples of game footage for submission to regulatory bodies such as the ESRB.
- Supervisor: the most visible authority in the office, supervisors are typically long-term and experienced employees who interact directly with testers and serve as a middleman between them and upper management. Supervisors are usually assigned to multiple titles and may test each to a limited degree. They also handle menial management tasks such as collecting employee timecards. Like managers, supervisors sign off on projects, but mostly in an advisory role. The QA manager will usually base his approval on the supervisor's, since the manager typically does not have intimate knowledge of the game's issues.
[edit] In the media
The main characters in the 2006 film Grandma's Boy worked as video game testers. The main game used in the movie, called Demonik, was in reality developed by Terminal Reality, but has been cancelled by the publisher.
"Testers", a 2007 webseries, depicts the weekly adventures of a video game tester. The series itself is touted as a viral video game, where the viewer votes and chooses what should happen next.
[edit] See also
[edit] External links
- Sloperama: "Working as a Tester" from game industry veteran, Tom Sloper
- IGDA's Quality Assurance Special Interest Group
