I've learnt a few surprising lessons while developing music game Substream. Linking music and visuals can be powerful, but the illusion can also be easy to break...
Timing is everything.
When we see and hear things happening concurrently we try to draw an association. So when it comes to syncing up music and game events, timing is extremely important. But triggering visuals at the beat isn't always best. If an animation's prominence builds over a few frames, triggering it before the beat can provide a better sense of synchronicity, so that the two events "hit" together.I synced up my events by hand, and I've learnt to be careful and get it right on the first pass. If I misplace the position of a note by even a quarter beat it creates an interesting kind of problem; I can usually sense that an animation amongst a string of notes was mistimed quite clearly, but not spot where. The illusion that music and animations are one is easily broken, even when you can't point to it.
Pitch is nothing.
When making movements that appear to be linked to the pitch of the music, you can lie a lot. As long as things generally go one way when the pitch goes up, and usually the other way when it falls, it's all good.Rock Band is a great example of this. You have five buttons on the guitar but many notes in the music. If a note is roughly higher than the previous one the player will usually be asked to press a button that's somewhere higher on the fretboard. Non-guitar players will quickly get a feel for which end makes higher notes and which makes lower notes and fully accept this system, even though it's used so liberally by the game.
You can alter perceptions!
Although animations need to be timed well, it's possible to get away with missing notes completely. And this is where things start to get spooky.If ten notes play and nine have an animation, the one without can appear quieter, or not seem to count as part of the melody. Alternatively, adding an animation usually linked to a snare at a time when no snare is playing can lead you to believe it was there, very quiet - you're not sure if you heard it. A bigger than average animation can make a note appear louder.
But it's not synaesthesia.
When a colorful musically animated game comes along marketing departments sometimes like to bring out this fancy word. I used it once or twice, but after doing some research I decided against it.Synaesthesia is a condition of mind where the experience of one sense is associated with a completely different sense or idea. This can mean that hearing a musical note comes with a certain colour, but it can also mean that numbers have a physical location in space or that certain words form a taste. This is something people experience their whole lives, but it's rarely considered a negative by them. Although someone might try to communicate their experiences in a game, you can't really induce Synaesthesia.
Some people will never see it.
Sad news. Some people just don't "get it". Maybe all their attention is on the gameplay so they see only see the audio connections subconsciously. Or perhaps they are not musically minded. But some people have stepped away from my game at conventions and been completely surprised when I've told them "it was all synchronised to the soundtrack". At least they still think the game looks and sounds nice as separate factors. But unfortunately that special link will always pass a small percentage of people by completely.
