Get testers playing your indie game NOW! feature - Sequence Storm

Everything I Learned Playtesting SEQUENCE STORM

Hello! My name is Daniel and I’m the developer of the new rhythm action game SEQUENCE STORM.

➡️ But FIRST, check out SEQUENCE STORM on Steam!

I must apologize, because this isn’t going to contain everything that there is to know about playtesting! Instead it is everything that i have gleaned from my own personal experience developing and playtesting a game as a team of one.

Most of the testing that I did for SEQUENCE STORM was with remote testers, as I never really had the opportunity or resources to bring testers in. I think this worked out really well in the end, however, because I got to test with a very diverse set of people and hardware.

Watching testers fail over and over can be painful, but in the end it is so valuable!

Pretend to be Somebody That You Aren’t

Imagine that you are meeting a friend’s baby nephew for the first time. You are going to feel compelled to ooh and aah, without regard to how cute or ugly this kid might be. When it comes to your game, however, this is the bottom line: you need to know just how ugly this baby is.

You will get better, more honest results when a player doesn’t know that the game they are testing is your precious baby that you have personally spent time and energy working on. Instead, it can be helpful to act in a role of a representative for your company. In every communication with a tester, pretend that you are just a lowly QA intern. It sounds silly, but people are more willing to provide negative feedback when you create the illusion of an emotional detachment between yourself and the game.

For the same reasons, it can be unhelpful to have friends and family test your game, aside from younger kids who have less of a filter.

Personalized Builds

It isn't a great idea to send a full release build of your game to strangers on the internet. Cut out parts of the game that aren't being tested. Also, create custom builds for each tester and put their name on it. If somebody happens to leak the build, you will know who did it!

Diverse Testing

Diversity is the most important in game testing, because you want your game to appeal to as many customers as possible. You might have a mindset that your game should only appeal to the most hardcore gamers, or to superfans of the genre. Instead, think about how you can appeal to a wider audience. Maybe your game can be the conduit for new players to become those hardcore, superfan players.

Non-Gamers

You might be taking a lot for granted if you are only testing with established gamers. There are plenty of people that are technically inclined, but who would not necessarily define themselves as “gamers.” These testers will expose any assumptions that you have made about your players, and any gameplay tropes that you might be relying upon.

For SEQUENCE STORM, testing with some non-gamers made it clear that the game needed to have several options for timing difficulty, because pressing buttons with accurate timing is something that a lot of people have trouble with.

Kids

This is probably the only case where testing with friends and family can be beneficial. Kids don’t have as much as a filter, and they will let you know when they are bored or losing interest. Also, you will learn a lot about your game’s tutorials and onboarding when you test them with a younger audience. Your tutorials are guaranteed great when a ten-year-old can understand them, but it can be very difficult to get to that point. Testing with kids is one case where you really want to be in the room so you can get the full reaction in the moment.

Fans of Other Genres

For SEQUENCE STORM, I found that many players were not particularly interested in rhythm games. So, the challenge became to find ways to make the game more interesting to those players that are not necessarily hardcore rhythm game players.

This is why SEQUENCE STORM has two different modes of gameplay: racing, which is more casual, and score mode, which is a pure rhythm game experience.

Testing Experts and Non-Experts

Testers with QA experience are going to have a very different eye for testing than somebody that has never tested professionally before.

Language Diversity

You may not have the budget to support many languages, and not every potential customer will be fluent in your game’s languages. However, your game can still be made playable even to those customers that aren’t fluent.

Hardware Diversity

It is important to test on as many configurations as possible, especially with PC games.

Pay to Playtest

If you are wondering why nobody on Reddit or Twitter is playtesting your game, you may as well wonder why nobody is offering to program it for you as well. Playtesting is important work, and you will have the best results if you are willing to pay. So, where can you get this diverse set of testers from around the world? It’s easy when you speak the universal language: money.

There are many people looking for small jobs on websites like Upwork. Posting a job where you are “looking for playtesters” can result in dozens of applications for you to choose from within a day.

Technical Requirements

When posting a job, be specific about technical requirements. Whether your game is for PC or tablets, be sure that potential testers have the right equipment. If your game requires a gamepad or any other hardware, you may want to be very picky about exactly what kind of gamepad a tester has. For instance, an official XBox controller is going to be much less problematic than a generic brand, or a gamepad that requires some kind of conversion software. However, you may also want to actually test on a diverse set of input hardware, if that is something that you want to support. Also, be clear about any requirements for recording or transmitting video. Not every tester will have the hardware or bandwidth for it.

Be There For Your Testers

Sometimes a tester might run into a technical problem and struggle with it for a while without generating any useful feedback. Let testers know that they should contact you immediately if there are any technical problems or questions.

Playtesting is Networking

When you are in contact with a large number of people, you will also build up contacts. For SEQUENCE STORM, I found a very talented game designer through a testing job, which I later hired again to help with some game design decisions.

When working worldwide, you will also find a large number of people that speak other languages, and this is a great resource if you want to translate your game. You will already have an idea about potential translators that have a good work ethic.

Here’s the Sensitive Part

One aspect of diversity that is often overlooked is economic diversity. You might feel tempted to pay less when you have a large pool of applicants, especially when many of those people come from different economic backgrounds. Why pay $6 per hour when you could pay $4, or even $2? It is up to you to decide what you are comfortable with. If you are careful about selecting testers, preparing your test build, and being specific with your requirements, you can get a very good bang for your buck, without exploiting anybody.

Make the Point Of Your Playtest Clear

Make it clear what you DON’T need tested. This helps to avoid wasting your tester’s time on unimportant details. If the game is early in development, and the graphics aren’t finished, then it isn’t useful for a tester to spend time screenshotting minor graphical issues.

Because we are looking for diversity in testers, not all testers will be used to playing games that are incomplete. Explain the current development state to playtesters.

Do it On Video

I thought I gave players enough time to react and dodge this barrier. Not even close!

Having video of a playtesting session is vital in some situations.

• When game development is early and you have anxiety that the game is even going to work properly on the user’s machine.
• When testing gameplay tutorials or other critical sections.
• When trying to understand the players’ very first impressions of the game.

As the developer, it can be hard to tell if the game is going to successfully lead players through a tutorial or a specific, critical set of actions. Getting these critical moments on video will make it very clear where you may need to do more work to guide the player. These critical moments can happen anywhere from launching the game, menus, and gameplay.

If your intention is to test the player’s very first impression of the game, then make it absolutely clear that testers should NOT start up the game and poke around before they start doing any recording. This is something that some people will naturally want to do with a new game. However, it can be upsetting to find that the player had already spent 10 minutes playing before they started recording. Be sure that you get those critical first minutes!

I had an issue with some players not understanding the mission menu. However, I wouldn't have caught this if I didn't test, or if I only cared about what happened during gameplay. I added a yellow arrow to guide the player to the next mission after they complete the first one, and I haven't had any problems since.

When NOT to Get it On Video

However, it isn’t always going to be appropriate to get video! We are looking for testers from all over the world. For some testers, it might be asking a lot for them to find the storage space and bandwidth to transmit an hour or more of gameplay video. Also, sometimes you may want to do a long-term test that lasts for a dozen hours or more. That is where logging comes in.

Just Log It

Logging has been just as useful as recording video, and I think it can even be better than video in some cases. The goal is that you can scan through the log and get a mental picture of exactly what the player is doing.

Here are some examples of things to log:

• Timestamps for every line in the log
• The players’ hardware configuration
• Player feedback
• Average frame rates
• Load times
• Which UI elements the player is interacting with
• Mission/quest success or failure
• Items or equipment that the player is using
• OpenGL errors

Don’t create a new log for each session. Instead, just keep appending to one log file, so that the player only needs to send you one file. It might wind up being enormous for extended testing jobs, but it is still much smaller and faster to parse than hours of video.

Here's a snippet of a log for a player's performance at the end of a mission:

[297844] ######################### PLAY LOG START #########################
[297844] 	timing mode: Normal
[297844] 	Mission: Mission2-2 - Daemon Hustle
[297844] 	Mission difficulty: standard
[297844] 	Attempts: 2 Successes: 1 Failures: 1
[297844] 	Track: Amada beginner
[297844] 	time: 2'03"006
[297844] 	timepc: 0.830426
[297844] 	top speed: 125.479%
[297844] 	score: 0.779747%
[297844] 	sync: 66
[297844] 	perf: 48
[297844] 	good: 28
[297844] 	bad:  9
[297844] 	miss: 8
[297844] 	misspc: 0.107595
[297844] 	holds: 0/0
[297844] 	boosts:         0
[297844] 	hypers:         0/0
[297844] 	Health: 0 Cornering: 0 TopSpeed: 0 Accel: 0
[297844] 	Checkpoint resets:    0
[297844] 	Mission success - medal: 1
[297844] 	Dead: 0
[297844] 	Song Progress: 1.00136
[297844] 	Frame time avg: 11.5491
[297844] 	Leeway time avg: 1.58484
[297844] 	Update time avg: 1.50861
[297844] 	Total play time minutes: 7.51196
[297844] ######################### PLAY LOG END #########################

Enable Cheats

Be sure that testers can use a cheat to get past something in the case that they get stuck, but don’t tell them about it right away. Instead, let them know that they can contact you if they get stuck, and then offer the cheat if you think it will help with the test.

Pop Quiz

When you are testing remotely, it can be hard to gauge how the player is feeling about the game. So, the game can periodically ask the player how they are feeling.

You might present a list of options, and the tester can select any number of them, for example:

• OK
• Fun
• Too Easy
• Too Difficult
• Confusing
• Boring
• Annoying

As well being able to type in any other input. This information all goes into your logs, of course.

Questionnaires

I found that it wasn’t useful to simply ask each tester the same set of questions after each playtest. Instead, it was more helpful to review each players’ logs and videos, and then formulate questions specific to their experience with the game.

Playtest Early and Playtest Often

Make playtesting a part of your regular development routine. Whenever you add a new feature to the game, or a new piece of UI, or a new tutorial, do some playtesting! Get into the habit of always having somebody, somewhere testing the game for you. Even if things look rough and aren't finished, or if you aren't sure that you've got the right solution, getting another pair of eyeballs on a design can clear things up in a hurry.

Wrap It Up

There it is folks, everything that I know about playtesting! I hope you found this information useful. Please let me know what you think!

➡️ Check out SEQUENCE STORM on Steam!

➡️ Follow me on Twitter!