TUNING IN

As the name implies, much of Grand Theft Auto’s gameplay is spent hopping from one hot car to the next. But cruising downtown in a boosted four-door coupe isn’t quite the same without some East Coast rap on the stereo. One of GTA’s most popular audio features is the virtual in-car entertainment. There are 17 radio stations players can flick between, so they can bob along with their homies to N.W.A’s Appetite for Destruction, or wind the windows down and have a gang sing-along to Queen’s Radio Ga Ga. There’s even talkback radio, spoof DJs, advertisements, jingles and phone-in guests. Some of the stations are curated by artists, and Flying Lotus is so into it, he’s updating his FlyLo FM station with more exclusive game-only tracks for the reissue.

It’s a rich musical environment in itself, but there’s also an adaptive non-linear, interactive score — equivalent to about 12 albums’ worth of music — created by German electronic music group, Tangerine Dream, that underpins the entire game.

Conner: “We’ve always been fans of the band and having licensed a few of their album tracks on previous games, we knew their sound would complement our vision for GTA V. When we approached them to work with us on an interactive score, we gave Edgar Froese a demo of the game and he loved it. Both he and wife and artist, Bianca Acquaye, visited us at the Rockstar North studio a few times to discuss the scoring process — it was so important they had a true experience of the game, and got a real feel for it, before starting work. Edgar heard a prototype interactive score track we had working on a few GTA missions and knew exactly what to do from there. The man is a legend and bringing his sound into this new interactive technology was very exciting. It helped that Edgar has already scored countless films, so he knew exactly how to convey the emotion for each mission.”

With a mind-boggling amount of both licensed and original music in play, mastering the music was always set to be a significant challenge, as Conner explains: “Levelling off so many different styles of music with hundreds of tracks that can literally flick from one to another anytime the player decides, is no easy feat. I’ve been mastering music for over 20 years so I’ve learned a trick or two, but GTA V did pose a challenge for a number of reasons.

“Every licensed track pre-1999 was easy; everything after 2008 was a volume-war. I had to de-master tracks as they were already crushed to the limit. I deliberately processed the radio content a lot more than on previous games in the series so players could hear a clear difference between the score and the radio.

“As far as consistency on the radio goes, this time round the DJ dialogue turned out to be the biggest job. We recorded them in so many different locations; we even recorded one actor in their own home. To complicate things even more, a number of different mics were used, which meant each recording sounded different.

“I really wanted the DJs to sound mega-bassy, like real radio, but I had to pull back on that in the mix because it was taking over. I couldn’t allow the DJs to have more punch than an explosion, so it was all about giving the radio and score its own place in the mix. As GTA V has multiple sounds in a huge, living world, I had to pull out some frequencies in the music across the board, in order to give the player a good overall sound and eliminate ear fatigue.”

COUNTING VOICES

Talking about talking, recording GTA V’s dialogue is a mammoth production in itself. With not only 150,000 lines of script to record, but 700 actors voicing them, it’s probably the longest line of credits you’ll ever see.

Morton: “The process starts very early in development and continues right to the end. Really, you could write a book about it, but in summary, the game city demographics are looked at and the ‘virtual’ population decided on — which types of people are needed, the areas they live in, and so on. Models are made for the ‘peds’ (pedestrians) and then we set about planning voices. Dan Houser, co-founder of Rockstar will come up with a mini-character brief and backstory for each one, which then goes to the ped writing team who create a bespoke script for each one — about 200 lines of dialogue on average. The actors are cast, and recording time booked. Dealing with about 700 actors means recording peds has to begin months before the game is complete. And there are often revisions as game features continually evolve during development.

“We ran several large recording sessions throughout production, often recording 100 peds in a week (most only take an hour). We had three studios working simultaneously; each one with a director from Rockstar present plus an audio person like me to ensure what’s recorded is exactly what’s needed. As well as getting actors in and out on time, the producers make sure everything runs smoothly. After recording comes ‘take’ selection, and then the lines are edited, each voice mastered and the lines added to the game. It’s kinda manic sometimes, but Rockstar has been building huge scale games for years and the New York production team is expert at this stuff.” 

Central to the believability of GTA’s dialogue is its street cred. Not many script writers can grasp the slang of one gang, let alone an entire population as diverse as Los Santos, San Andreas — so Rockstar just hired the real deal.

Morton: “Dan Houser, Michael Unsworth and Rupert Humphries wrote GTA V. There’s always a huge amount of detail put into the characters from the main players through to the pedestrians that walk the street. Dan has always gone to great lengths to make sure that populations are as varied as they are in real life, and that even the smallest character amounts to more than just a ‘random guy on the street’. When we recorded the gang peds, some of the actors would actually be gang members in real life and would let us know when lines in their script were things they wouldn’t say. We always ended up with fantastic results when guys like that would take the script and make it their own.
“We’ve found, in general, whether it’s for a game or for a movie, you’ll get far better results when you treat your actors like humans — they are assets rather than tools. Actors can often bring things to roles you haven’t anticipated, and while as director you have the final say, you’ll often get nuggets of ‘performance gold’ by working with actors rather than just commanding them. After all, they want to do their job well.”

Conner: “For the main player characters in GTA V, we used a great deal of motion capture — the actors would physically play out the scene together and their movements, facial expressions, and dialogue would all be recorded together. When you can capture an ensemble performance with the right actors you get incredible results — far more detailed than recording everything separately. It’s obviously a lot more resource-heavy than the old way where we didn’t capture that data, but definitely worth it. Ned Luke, Steven Ogg and Shawn Fonteno’s performances really brought their characters to life.”

ALL THE RAGE

In a game like Grand Theft Auto, most everything you see makes a sound — people on the street, birds in the sky, cars in the distance… the coffee cup hitting the floor as a man flees a gunfight. The mix and 3D positions of these sounds relative to the player’s orientation must be continually calculated frame by frame to determine a correct and immersive aural experience for the player. Add that to the dialogue and music handling and you have quite a technical headache. Playback priorities must be carefully managed to create a viable run-time mix balance — you can’t replay everything, nor should you.

Enter RAGE, the audio engine governing the lot: it provides replay logic controls, culling of sounds according to distance and priority weighting, and providing the team a virtual digital surround mixing system.

Conner: “RAGE Audio was designed and built by our in-house team, meaning it’s perfectly suited to these sorts of games. It calculates and tracks about a thousand virtual sounds or voices at any one time, 96 of which will be rendered in the mix. When sounds are added, the volume of each file is calculated. The engine then uses that information in conjunction with what else is happening in-game to work out which sounds/voices should be heard. For instance, a loud sound happening miles away may not be heard over a quieter sound happening close up. It’s a great system for prioritising sounds automatically and working out what is, and isn’t important to the player.

“Many sounds are created in real-time from tiny components rather than using a single pre-built sound effect. The benefits are zero memory-wastage, and it also gives the sound designers flexibility because the sound variation is created as it’s triggered.

“We included some real-time synthesis in GTA V by using AMP — a modular ‘connect-the-cables’-type interface, fully integrated with RAGE Audio. The benefit of synthesis over samples is creating infinitely variable sounds. Sounds that would normally be quite static, if represented by recorded effects, suddenly have life breathed into them, and their behaviour appears more realistic. It’s also very memory efficient. You can do a lot with noise and filters, for instance, creating the sound of air conditioning units. “AMP is also used for DSP. In GTA V you can switch between three characters during missions. They’re often talking to each other via headsets so the voices heard through an earpiece need to be changed on-the-fly as the player switches roles. We designed a headset effect chain to process the voices at run-time as required.”

Morton: “Vehicle sounds are obviously a big deal — from the huge array of cars and trucks through to aeroplanes. The sounds of the cars are again made of tiny components recorded with a multiple-microphone setup. The vehicles sound different close-up than at distance — as they would in real life, and there are other details such as the sound of an engine cooling down after you’ve been thrashing your vehicle and so on. It also meant we could use the components of one vehicle as part of other vehicles. The engine recordings are processed with a granular playback engine so that as the car speeds up or slows down, the sound is perfectly in sync with the vehicle and follows the behaviour of the player. It was a massive step up in quality and realism from the old-school game method of pitching loops or just using static recordings.”

LEAVING THE ROCKSTAR LIFE

As they enthuse about Grand Theft Auto, it’s abundantly clear that working on such a notable world-class game series has been an amazing adventure for the duo. But they’ve since left Rockstar, and now they’re bringing all that expertise to bear in a new venture, Solid Audioworks — the production company they recently co-founded to provide a full audio service; including dialogue, sound effects and score, as well as movie post-production. But surely leaving Rockstar-dom can’t have been an easy decision.

Conner: “For me, after almost 20 years working on the GTA series, seeing the process travel from generation to generation, post-‘V’ seemed like the perfect moment to depart, go out on a high, re-charge the creative batteries and try something completely new. There are so many other games/films I want to work on and tick off the bucket list.

Morton: “Leaving certainly wasn’t a decision made lightly. For me, leaving the Rockstar family was about being able to work from my own studio, closer to my own family, and expand my horizons project-wise. There are lots of interesting games being developed — both indie and AAA — I really wanted to try something different. First of all, I fell into doing film work, which led to bumping into Craig at a pre-shoot meeting. We started discussing working more together, and that subsequently led to us setting up Solid Audioworks. As to what’s next, we will be busy on some film post production until the end of the year, after that we’re very much looking forward to getting back into what we love — games and game audio. Watch this space.”

Story: John Broomhall

And for composers like Jason Graves, the creative largesse is truly liberating: “I seriously doubt a film or TV project would allow me multiple days on the recording stage with a full orchestra recording experimental effects without any idea what the end result will sound like (all three Dead Space scores). Or commissioning an original steel and glass instrument for a soundtrack without any idea of what it looks like until the camera crew arrives to film it for a behind-the-scenes television special (Tomb Raider). Those are two of the many creative experiments I’ve enjoyed throughout the years in video games.”

The industry used to talk about triple-A quality games — now it’s ‘quad-A’. These are titles whose production values are sky high, driving a steady increase in the use of not only the world’s leading creative talent but also its most prestigious recording and post facilities.

Paul Lipson: “We are working across the industry in partnerships that are helping us achieve class-leading results for our releases. On the bespoke content side, we work with musicians throughout the US and the UK to record our scores — and enjoy a relationship with the AFM (the American Federation of Musicians, the largest union in the USA/Canada), and facilities like Skywalker Sound, Abbey Road, etc. We commission scores from some of the best composers working today, and encourage innovation and new approaches to original music.”

Giles Farley: “At Pinewood and Shepperton Studios we bring to video games the exact same creativity, expertise and passion we bring to major motion picture releases. For instance, recording foley for Fable: The Journey or mixing cut scenes for Rome: Total War, the level of talent involved is world-class and we’re excited about our future game projects.”

ROAD TO FREEDOM

With hard disk, memory and streaming limitations largely removed on next-gen consoles, artists can more easily realise their audio visions.

Jason Graves: “Now it simply comes down to budget — how many minutes of music and how much time the developer can spend implementing everything. I think from an interactive music standpoint, we’ve kind of ‘arrived’ with the new generation of consoles. As far as final playback within the game is concerned, you can really do anything now.”

Nick Laviers concurs: “I think in their own way, games have followed a similar course to the movie industry. In the early days it was more about what technology and practices film companies developed to maintain their competitive edge, then as the years went by and cameras, audio equipment, etc., became standardised and ubiquitous, the pushes forward with technology were smaller and came about more slowly. Here in 2014, video games have become a mature entertainment industry and in game audio, our challenges are now more in the area of content creation. This now drives the technical side, whereas I think in the past we often dreamt up the technology first!”

As the power of ideas increasingly trumps the power of technology in game audio we are witnessing a paradigm shift in creativity.

Nick Arundel: “Over the last 15 years I have seen the move from games struggling to fit creative ideas within the available technology, to games that really understand what makes an engaging experience, regardless of the technology.

“Historically, the technical questions in the way of achieving our aims were the foremost issues facing developers. As technology improved the language used by developers shifted towards more ‘creative’ questions, such as what do we want the player to experience and how do we want them to feel? In some ways the technology has become transparent.

“A huge part of this shift is the wide adoption of middleware tools, and for sound design this means Wwise and Fmod. These tools are the video game industry’s ‘ProTools’; the de facto standard. There is now a pool of talented sound designers who think within the framework of these tools. These tools have affected every aspect of how we approach sound within games, from how we record, to editing, implementation and mastering.

“It’s exciting to see the growing number of sound designers who want to embrace the challenge of video game sound and face the issue of ‘permanence’, a unique problem/opportunity in video game sound. We face the task of creating complete ‘worlds’ of sound, not sound for just a single shot. These worlds can be explored at the player’s leisure without the linear time-line of film. They have a ‘permanence’. We hand a large degree of the movie director’s control to the player, and we must make sure it sounds great, not in one case (the shot), but in every case. The skills needed to face this challenge are not directly transferable from film. With the help of middleware, a ‘tool box’ of techniques has emerged and become the working language of the industry.

“We can now finally start grappling with the issues around what makes a great player experience. Books can give a fantastic insight into someone’s psychology, a film can show you great spectacle, but what are video games good at? What stories suit video games? How do we manipulate or guide the player? These problems can now be tackled without first considering daunting technological hurdles.”

INDIE EXPLOSION

Quad-A is of course here to stay, but it’s not the whole story. Artistically brilliant titles are being created at a fraction of the Quad-A production price tag by a burgeoning indie sector thriving in a commercial world where barriers to entry just keep falling. Meanwhile the proliferation of smartphones and tablet devices is creating yet more opportunity for innovative and potentially lucrative gaming-to-go.

Nick Laviers: “Not everybody will be (or need to be) at the ‘bleeding edge’, just like not every film is released in 3D. There are more timeless things which sell games, like storytelling and game-play design. Perhaps more so in games than films, the visual and sonic quality play second fiddle to the core facets that make games compelling. There’s room for more of a range of production values in games these days to illuminate these facets. Even ignoring ‘old’ games, there’s mobile games, social network games, a massive gamut of different styles of indie PC/Console games and then of course AAA PC/Console games. Right now, my kids are loving a low-res looking platform game called Starbound. While I’m simultaneously being stunned by the incredible artistic achievement in Ryse. Two very different aesthetic experiences but both enjoyable because of the story and game-play.”

Ben Minto: “The variety of delivery platforms for today’s games has democratised the market. 15 years ago it was mainly PC, console and handheld. Today the starting point, in terms of cost, are the free2play, micro-transaction titles, and at the top end, the $60 titles which cost double that when you factor in downloadable content/membership programs. Today, in theory, anyone can make a game. An indie title made by a handful of people can compete on the same stage as a huge AAA title involving hundreds of people, across multiple teams and studios. Big isn’t always the ‘best’, especially with game audio, which has seen a range of awesome smaller titles take various crowns for best audio — Journey, Limbo, Runner 2, Bastion, Papa Sangre II, Device 6, etc.”

THE NEXT LEVEL

Right now, all the signs are that game music, sound and dialogue is set to enjoy a period of creative excellence at all budget levels — just as with movies.

Nick Laviers: I was chatting to a movie sound designer friend of mine recently who was telling me about the big-budget blockbuster he’s working on; he also mentioned he was working on a smaller indie project just to be more playful and inventive, not expecting to make as much money from it. The smaller project lets him develop his craft in new directions which will eventually benefit the bigger projects. That feels kind of healthy to me. We have the same opportunities in games now and I’m hoping my future contains a diversity of different sized projects!”

Meanwhile, Steve Schnur is certainly optimistic about game music: “The traditional record business may be dead, yet the music industry is stronger than ever. Within the next 10 years, our ability to create, expose and discover new music of every genre will be beyond anything the industry or the consumer has known before.”

Perhaps the most exciting facet of all is the lack of technical limits hampering creativity. Those old party poopers — time and budget — will realistically always be lurking in the wings, but at least those precious implementation tools and middleware systems now allow sound designers and composers to take more ownership of the integration of their work into the game, a big leap towards fully realising their creative ambitions.

That said, we’re not altogether out of the technical woods, so where should audio types focus their energies?

Bruce Swanson: “We need creative solutions to problems unique to game audio, like how to achieve a controlled and focused mix when the camera can be anywhere and the composition of sound emitting objects is variable. But ultimately, as the roles of mediums like games and film blur and merge, I suppose the most important place to focus your energies is on understanding and evolving the language of audio as it pertains to storytelling — it’s still about, and always will be — what to play, what not to play, and why?”

Text: John Broomhall

ON THE MARK

The result of a long, sunny day’s firing of an impressive array of gun hardware in Hollywood was a huge ProTools session with everything synced up and mix perspectives carefully laid out. The close gun recording was used for a range of 10 metres and then two further perspectives were chosen from the total of five available. These perspective choices then figure in game-play helping the player readily discern enemy positions. Arundel: “You have the sense of distant warring factions in the city with the sound telling the player, ‘these folk firing at each other don’t concern you right now’. But then we very quickly need to swap to ‘look out, they’re firing at you’. The courtyard recordings all went into that layer of ‘thugs in the distance fighting’, while the recordings from the street and the end of the street went into the thugs firing at you.

“The day in the steel mill was amazing – it’s quite a dangerous place and it took some organising to get permission but yielded so many incredible sounds. At one end the ‘cooked’ girders came out and a crane picked them up, dropping them off somewhere far off. Every now and again, from about half a mile away, you felt a massive thud and this distant metal clang would echo through all the warehouses. In part of the game, The Joker has set himself up in a steel mill but he’s half turned it into a messed-up funfair, hence also recording at the funfair museum in Devon. We were able to collect myriad ‘musical’ sound effects from old carousels and waltzers, as well as lots of mechanical sounds running on their own without the musical elements.”

When it came to the sound of Batman himself, Arundel and his team strived to make the sound as believable as possible. “He’s got gadgets for everything and for me, they’ve got to sound totally believable – even when you know full well, in real life, the device couldn’t actually exist – it couldn’t do that thing – but it’s got to sound like it really could. John Roesch who did all the foley for Dark Knight and Batman Begins took care of a lot of those elements. Not that we wanted a copy of those movies – it’s a game and the way you approach it is different. We don’t have a Batman ‘audio bible’ as such but I think you just know when you’re doing it wrong. A good example is the cape opening – what we call the ‘wa-chonk’ – because that’s how it literally sounds as the cape flaps open and goes rigid. We spent ages on it this time out as you’re now outside for 60-70% of the time – high up, in tall areas. We really wanted the ‘wa-chonk’ to be a big pay-off sound-wise. As soon as you say that to a sound designer, you’ll get all these whooshy growls from animals – interesting, subjective content. But I really wanted it to be believable as a real world sound. It’s one of the sounds that people really care about.”

The audio team’s involvement in dialogue recording can be summarised as technical management to determine consistency of microphone usage and signal paths, and ensuring everything is recorded faithfully and correctly from three different perspectives – using close mic, Lavalier (usually mounted somewhere on the actor), and shotgun. Arundel: “Probably 90% of the dialogue in the game is from a shotgun – more of a production sound. Recording this way means the actors can move around, waggle their hands, and do the things actors do. They are less bothered about mic position. However, with three feeds, you’ve tripled your decision making and file handling…”

STEADY IN THE MIX

The final mix of music, sound, and dialogue was undertaken internally at Rocksteady’s two fully calibrated studios on a combination of Genelec and PMC monitors. Arundel did a first-pass pre-mix in 5.1. Then colleague Lee Banyard conducted a second pass, listening and tweaking more in stereo. Lastly, a group review took place and the resultant mix was pushed out into the wider development studio where most of the art, coding, and design team listened on headphones or flat-screen TVs.

The mechanics behind the mix process were down to Audiokinetic’s Wwise, the underlying audio middleware ‘engine’: “There are a tremendous amount of mix states – obviously one per room and then per type of game-play, be it combat, predator, conversation – hundreds of them. There was some debate about my initial mix. We’re always keeping an eye on loudness and I’m quite keen for the game not to be that loud that often. It frees us up to create some big surprises. I reviewed my initial mix at home and took feedback from other people, concluding we weren’t quite loud enough. I reduced the dynamic range at that stage and because we were using the McDSP limiter on the final output we didn’t have to remix the whole thing – just change the mastering limiter settings. Each mix iteration got easier but I think next time, we might bring someone in to do the final tweaking with completely fresh ears.

“The hardest kind of aural perception test I’ve ever done is having to write down four simultaneous music parts by ear. If you’re really good at listening, you can clearly hear four elements, but you’ve still got a hierarchy – you probably don’t listen that much to the bass, you pick up key notes and work it out. I think most people can hear two or three things at once before they start aggregating sound – lumping sounds together as say, ‘all the environment’ or ‘all the walla [background crowd murmur]’. When we’re mixing, we’re asking, ‘what’s the thing that leads this?’ A conversation? Music? Is it the sound design?’ This informs our mix. You have believable environmental sounds that signal to you the type of room you’re in with realistic reverb, but we also have other sets of sounds that are meant to push the player towards a certain type of feeling and these are not at all literal. We spend nearly as much time thinking about how the room should feel as to its sonic believability.

“We use a fair amount of real-time DSP including a very realistic reverb and delay in the city areas which works well on the dialogue. You get some really nice moments as you run away from certain characters, the speech reflecting off the walls down the street. One game feature is a surveillance scanner. When deployed, it surveils the thugs and tunes into their conversations. There’s McDSP FutzBox processing on that dialogue. As you approach the thugs, that cross fades into its furthest reflected and delayed reverb sound and then crossfades into a drier sound as you come up close – all mixed in real-time. Add to this what, for me, are totally believable city population sounds, and you really feel like you’re Batman roaming around Arkham City.”

SUPERHUMAN EFFORT

Amazingly, and very unusually in video games, the same man who stands at the helm of this gargantuan effort, steering a global audio team towards their creative goals, also composed much of the music score, which weighs in at over two hours’ worth, recorded by Nick Woolage at Air Studios, no less.

It seems that perhaps, something of the Caped Crusader’s superhero multi-tasking skills may have rubbed off on Arundel himself.