Early beta testing round - Post Mortem news - Empires in Ruins

So, our first closed beta testing round just closed. A few feedback are still slowly coming, but here's why we are satisfied. Not just satisfied about positive feedback, but satisfied by the way we setup the round, and satisfied by the amount and quality of constructive criticism received as well. First of all, the players received a link to our repository containing the EiR_v0840_05 version of the game, that had been previously tested internally in the team for a couple of weeks to make it as stable as possible. Indeed crashes were almost completely absent from the testing, and the only few bugs encountered were almost never blocking.

The tested game version still lacked some of the towers (some artilleries and temples) that are now in game, some of the units and their features and was all in all a testing version, i.e. stable enough to be played for a while, providing enough gameplay to see where the overall game is going (including a small part of plot and dialogs too), but far from polished. Together with the build we provided a small strategy guide of hints and tips, a list of the shortcuts and commands, and a link to an anonymous google feedback form. I'd like to point out at the anonymous detail, because we asked all the testers (many of which we didn't know or barely know, and we found through our facebook and twitter pages, IndieDB and reddit/TowerDefense) to be absolutely merciless. And we allowed the forms to be anonymous to try and make people being as nasty as possible ;) And this is how the form was structured. 10 questions, each with a score of 1(worst) to 10 (best) and an optional comment, 5 final open questions, and the space to report bugs, strange behaviors, and endcomments.

And these where in detail the questions we asked. 15 out of 23 anonymous testers did answer them, and here you find reported the grades, the peeks in the distribution, the min and max values for each and a short comment on their comments.

Main Map

1) Tutorial - Usability, usefulness, appropriateness. Does it help? Is it easy to follow? Right length? Grades: Avg 7.2 (Min 1, Peeks 6/7/8(3), Max 10)
Comment : The testers found the actual tutorial useful but filled up with too much information. Overall they asked to split the tutorial in more smaller tutorials, with less "passive" and more "active" steps.

2) UX/UI - Usability, usefulness, appropriateness. Are they intuitive to use?
Grades: Avg 7.13 (Min 3, Peek 8(7), Max 9)
Comment : Overall thanks to the tutorial, the players had no major issues in the UX/UI (the feedback we've got before we added the tutorial was quite the opposite, but had us thinking indeed that UX only needed a first hint to work). Players ask for more polish and better visuals (and our Alban is right now working on that, a complete UI remake) but pretty good overall here too.

3) Visuals - Quality and appropriateness of the ingame visuals to the game style
Grades: Avg 7.86 (Min 6, Peek 9(5), Max 10)
Comment: Not much to say, people are loving George's (3d) and Konrad's (Illustration) work on the main map since the beginning.

4) Art/visuals coherence - Did everything look as if belonging properly? Map/characters/UIs, etc?
Grades: Avg 8.86 (Min 7, Peek 9(6), Max 10)
Comment: This is a question we asked specificically because a GameDev expert we had feedback from for an older version, complained a lot specifically on this. After his feedback we changed quite some things, but we wanted to make sure to know how users felt about it. So far so good!

Battle Map

1) UX/UI - Is interaction smooth and intuitive? Does it hinder or help the gameplay?
Grades: Avg 7.53 (Min 5, Peek 7(6), Max 10)
Comment: Testers ask for a better polish in the UI and more readable Icons (and as said above we are making that), for some features to minimize the number of click (for operations like repairing towers), but overall good feedback. Most of the people found it smooth to use and easy to remember.

2) Gameplay - Is gameplay interesting? Is the game too frenetic or does it feel too passive? Does it satify TD players needs or is it too complex and more towards RTS?
Grades: Avg 7.26 (Min 1, Peek 7/8/9(4), Max 9)
Comment: This was one of the point we were more interested in, due to the fact that we are trying to have TD and RTS meeting somewhere (nearer the TD side). Grades and comments were quite spread out as expected, some people asking for more interactive stuff, some people asking for less. Overall I think it's heavily influenced by balance as well (see question below), but we can't complain!

3) Balance - Is the game balanced enough in the current stage (that is still very primitive, but already helps us balancing out the overall difficulties). Did you win too easily or did you fail too many times the battle?
Grades: Avg 6.57 (Min 2, Peek 8(3), Max 10)
Comment : A premise : the game right now only features a single map, the one used for development and that will not be in the real game, and that map has EVERYTHING in, while in game most of the stuff and enemies will unlock progressively through research and map. So out of a map like that, balance is hard and onboarding as well. Moreover, with a lot of game features missing, performing a fine tuning of the balance right now would be almost useless for the future, we just wanted to see if the balance didn't throw people off completely. Again, some said too easy, some said too hard, some said totally fine ;) Grades average and distribution seem to confirm.

4) Art use and Coherence - The terrain is still an old placeholder we will change soon, but how does the rest of the art fit the game and with each other? Enemies/units/towers, etc? Are the game elements readable well enough at all the zoom levels?
Grades: Avg 8.06 (Min 4, Peek 8/10(4), Max 10)
Comment : We pre-warned the players about the terrain, we expect the final ones to be a real game changer in this, but still, people like visuals and animations in general, we're happy about that cause it was a big focus for us.

Overall

1) Music - How well do you think the music fits the game spirit? Does it work well enough for you in battle with the slow/fast tracks? Does the difference between main map and battle map music work for you?
Grades: Avg 8.4 (Min 5, Peek 10(6), Max 10)
Comment : Music is quite a special thing for us on the game. Our friends from the Celtic band Tribauta provided us some of their songs, and the lovely guys from Red Dew Hellpipes (lovely is a joke, they are amazing hairy and nasty and mean bastards, we love them though :D ) wrote some folk metal just for the game (bagpipes and distorted guitars). And we are absolutely happy that people liked it. Some mentioned it being a bit repetitive, but we have 2 more songs coming, and that shall help. (post release, if we'll have the budget, we might get some more in as well)

2) Overall art coherence - Considering that some things are still temporary, how does in general the overall artwork feel when thinking about menus, main map, battle maps, dialogs, etc? Is it coherence or not?
Grades: Avg 8.6 (Min 6, Peek 10(5), Max 10)
Comment : As I said above, we wanted to make sure there was an overall artistic coherence in the whole game, across main map and battle maps. Apparently according to the players now there is.

3) Overall game/settings characterization - Does the game have enough of a given personality at this stage? does it feel "immersive" and easy to remember (think about characters, music, style, etc etc)(Of course we know that having no missions nor plot implemented yet doesn't help)
Grades: Avg 7.71 (Min 5, Peek 6/8/10(4), Max 10)
Comments : This was another big focus of us, because we think that to be memorable a game needs not just excellent gameplay, but also a strong personality, strong characters and something that makes it easy to remember. Some people found our low brow humour a bit too cheap, and some found the dialogs a bit too crass ( :D ), a lot of other people liked the humour, the dialogs and the characters. We will have our Yannis soon go through them to smoothen up a bit some edges and polish the language a bit, but overall it seems like our general message is getting to the players more or less as we wanted it.

Open questions (taken from this Gamasutra's article )

1. What was your favorite moment or interaction?
Comment : Hard to summarize them all, but people liked a lot the dialogs and characters of the plot, and the moment in which the managed to beat the game.

2. What was your least favorite moment or interaction?
Comment : To try and find some sort of common points, the tutorial (too much information and/or too long) and repairing the towers.

3. When did you feel the most clever?
Comment : Here definitely when they managed to beat some hard part of the battle map for most of them.

4. Was there anything you wanted to do that the game wouldn't let you do?
Comment : Pointless to summarize, every single tester said something different :D

5. If you had a magic wand and could change any aspect of the game or your experience, what would it be? Unlimited budget and time.
Comment : Pointless to summarize, every single tester said something different :D (e.g. someone asked for less management in the main map, someone asked for more, someone asked for more towers, someone for less, etc etc)

Woah, that was a lot that I wrote! But well, we got a lot of information out of this early testing round. We are absolutely happy about positive and negative feedback. I would say overall, 1 person didn't like the game almost at all ( it can happen! :P) but the overall crowd seemed to quite appreciate, and provided plenty of ideas, comments, etc.
This is actually quite amazing because while the positive feedback is super motivating for our team, the negative one (and as you saw from the grades, they had no problems in saying that something didn't work for them when it didn't) is instead amazing to know where to focus future efforts. Ah, plus we built the game in development mode, so that it provided an output log that the tester could send us by email in case of manifest bugs. That's it guys, if 10% of you managed to get here, know that I am proud of you, even though I think you're madmen! I hope you found it useful, i go back coding magic auras for the temples now, cheeers!

Emiliano, H&R