The Growing Field of Immersive, Multimedia Events

Not long after the sun set on February 3, visitors to the Bellforge Arts Center in Medfield, Massachusetts, braved a frosty wind and ventured outdoors to play with blocks.

These were no ordinary toys, however. Rather, they were part of Sound Sculpture, an interactive musical artwork conceived by composer, percussionist, and interdisciplinary artist Ryan Edwards B.M. '11, cofounder of Boston-based MASARY Studios. The installation, which tours nationally, allows users to create musical compositions by moving around oversized plastic blocks that measure 17 inches on a side and that look for all the world like enormous glowing ice cubes.

Each block contains a package of electronics that includes a location tag, a microcontroller, an LED light array, and a Wi-Fi transponder. The blocks communicate with a computer that uses custom software to track their location in space, generating signals that cause the blocks to change color while triggering notes of varying pitches and durations across a multichannel surround-sound speaker array. The resulting system comprises a patented, location-aware instrument that a group of individuals can play together regardless of musical training.

While the technology behind Sound Sculpture is impressive, it is also essentially invisible to users, allowing even the least technically inclined to experience the act of composition in a direct, hands-on way. At Bellforge, children used the blocks to build towers, forts, and stairs, collectively improvising a multimedia artwork as they experimented with different configurations.

“The point isn’t that we made an instrument that is location-aware,” Edwards says. “The point is that it’s a social environment for creativity that often surprises people with what they can do.”

Sound Sculpture is illustrative of the evolving field of immersive, interactive art—a field that has experienced rapid growth over the past decade or so due to technological and societal changes—that often combines sound and light to create unique, multisensory experiences.

“We try to create a synergy between the sonic and the visual—not to merely represent one alongside the other, but to blend the two worlds into a singular presentation,” says Assistant Professor Peter Zebbler Berdovsky, founder of Zebbler Studios in Boston.

Interactive art that incorporates audio and visual elements is not new. Examples of such work can be found in the early 20th century, although the trend toward immersion and interactivity really took off during the ’60s and ’70s. Over the past several decades, advances in spatial audio technology and projection mapping—the projection of video content onto 3D surfaces such as stages, sculptures, and building facades—have significantly expanded the ability of practitioners to create novel experiences. So too has the availability of sophisticated, affordable software, like Touch Designer and Unreal Engine, that enables users to create content in real-time based on participant input: digital sensors and cameras can track audience members’ motions, for example, and this data can be used to trigger or even generate musical accompaniment and visual displays.

Consumers, meanwhile, have become increasingly accustomed to interactive experiences mediated by technology—partly thanks to the ubiquity of smartphones and similar devices, and partly thanks to the pandemic, which forced many people to live through their screens.

As a result, there’s been a proliferation of immersive, interactive work in recent years, some of it produced by studios like MASARY and Zebbler for museums, festivals, and public events, and some of it produced by groups like Meow Wolf, Wonderspaces, and ARTECHOUSE that mount installations in dedicated locations across the country. Major corporations have also picked up on the trend, with companies from Nike to Intel commissioning installations to promote their brands. And live events are being similarly transformed by the adoption of spatial audio, programmable LED lighting, and projection mapping.

“The immersive industry is a huge industry. And it’s growing rapidly,” says Daniel Pembroke, program director of the Live Music Production and Design graduate program at Berklee NYC. Pembroke, who runs the New York-based installation house Sluice Labs, points to the 2023 opening of the Sphere—a $2.3 billion, 366-foot-tall spherical theater in Las Vegas that hosts immersive and interactive events—as evidence of just how large the sector has become.

It is also extremely varied. Javier Cruz P.D. '11, who works for the Brooklyn-based immersive media design studio Volvox Labs, has composed music, written code, and created visuals for clients from IBM to the contemporary multimedia artist Roy Nachum. Recently, Cruz contributed to a custom-built theater experience that Cadillac commissioned to introduce its new line of electric vehicles. A short film about the history of the carmaker was projected in high definition on the theater’s walls while a score Cruz composed filled the space in multichannel surround sound. At the same time, software tracked participants’ movements with digital sensors and projected video content based on those movements onto the theater floor. Even the exterior of the theater became part of the installation, as shifting 3d images played across wraparound LED lighting arrays on the building’s facade, turning the entire structure into an enormous trompe-l’oeil. 

“This industry, and this artform, can create magic,” Cruz says.

Much of that magic comes from the powerful combination of sound with light. “We’re visual creatures,” says Edwards. “So the more we accompany a sound work with visuals, the more we can connect with our audience.”

Connecting with audiences is top of mind for Danny Scheer M.A. '23, who helps create immersive audio and visual environments for live shows in New York City. Scheer studied songwriting along with creative media and technology at Berklee NYC, and he sees the work that he does now—namely, manipulating sound and light to communicate and reinforce an artist’s vision—as a form of “prosody in space.”

At the most recent iteration of the Sound and Vision of David Bowie, an annual mini-festival at the Cutting Room in New York City that celebrates the late singer’s birthday with performances of his songs, Scheer and his colleagues enhanced the emotional contrast suggested by the evening’s set list with a series of coordinated audio-visual maneuvers.

The finale to “Lazarus,” an emotionally intense work in which Bowie addressed his own mortality, ended with blindingly bright movers (mechanized, computer-controlled lights) silhouetting the performers from behind, creating a sense of separation between the artists onstage and their audience. Then the theater plunged into darkness, with only a single spotlight illuminating a lone performer holding a guitar and singing the opening lines of “Space Oddity” (“Ground control to Major Tom…”). By using spatial audio techniques to manipulate the sound in the room, Scheer and his colleagues were able to collapse the distance between artist and spectator, drawing both into the same intimate sonic and visual space.

“Spatial audio in itself is great,” Scheer says. “But if you have visual input that confirms that auditory sensation—you see it there, you hear it there, it is there—you can really paint this whole world.”

Painting, or perhaps building, a whole world is an apt description of what Zebbler Studios achieved at the Envision Festival in Costa Rica this past March.

Zebbler and his team, which included the Providence-based musician and video artist Andrew Hlynsky M.M. '16, were charged with designing and fabricating one of several live stages at the weeklong event. Using local materials such as bamboo, corrugated metal roofing, and hand-cut plywood, they constructed a physical framework with a sculptural facade and recessed platforms for dancers. To that they added computer-controlled LEDs as well as conventional lights, and then projected video imagery onto the entirety of the stage’s 3D surface. Working from a single control table, the team was able to synchronize these different lighting systems with each other and with the live music—rap and reggae, rock and world electronica—emanating from the stage.

“Our stage was around 120 feet wide and 40 feet tall, so for people standing in the middle of the crowd, it must have felt like a pulsating, psychedelic, rhythmic IMAX screen—fully surrounding their field of vision, yet orchestrating the visuals to fully match the music in real time,” Zebbler says.

Hlynsky, who develops technology for interactive and immersive experiences, even hooked up a PlayStation controller to a custom software patch so that he could control the onstage visuals while mingling with the crowd, altering what attendees saw in response to the music.

In addition to his work for Zebbler Studios, Hlynsky works on a wide range of projects with corporate clients and with the Rhode Island artist collective known as the Reliquarium. He particularly enjoys crafting inclusive environments that invite participation, blurring the distinction between artists and audiences. “Creating situations where there are no spectators is great,” he says.

Hlynsky created just such a situation at the Berklee Abu Dhabi Center in 2022. His instructions were simple: build something musical and interactive; draw inspiration from the location of the site; and provide people with a memento of their experience. He settled on a touchscreen-based experience that drew on the rich tradition of geometrically complex tile patterns found throughout the Islamic world. Participants began by collectively creating tile patterns on a large touchscreen display. The tiles were then mapped to musical notes, transforming the display into a giant visual keyboard that participants could play by donning motion sensors and moving their hands across the digital tiles. Afterward, they received a keepsake video clip.

The project took six months to complete and involved a series of daunting technical challenges. But for Hlynsky, who delights in using interactive art to break down social divisions and level the experiential playing field, the end result was worth it. “These interactive spaces are unpretentious in a way,” he says. “Anybody can go in and enjoy it, and it doesn’t matter if you’re Black or White or gay or rich or poor or anything.”

Although Hlynsky tackled the Abu Dhabi project alone, the range of skills required to create such installations—Pembroke says that the Berklee NYC program covers everything from sound and lighting design to coding and project management—means that most of this work is collaborative and team-based.

The live events that Scheer works on, for instance, may require a director, a lighting designer, and a camera technician, along with several audio specialists. Studios like MASARY and Zebbler, meanwhile, typically employ multiple people with different areas of expertise. MASARY, for example, has 10 people on staff with backgrounds in composition and performance, architecture, and artificial intelligence.

“There’s an emerging, studio-based practice of collaborative creatives that are wildly combining sound, light, tech, and other media,” says Edwards, whose interest in multimedia work grew out of his experience playing West African drum music for dancers. “They rely on one another, and they’re better for it.”

Collaboration was essential for the design and fabrication of solstice, a project that MASARY undertook to celebrate the winter solstice at Mount Auburn Cemetery in Cambridge, Massachusetts. The solstice marks the longest night of the year as well as the point in the Earth’s orbit when the days begin to lengthen once again; solstice included components inspired by the celestial underpinnings of the event and by the various rituals that mark the annual transition from darkness to light.

Over the course of several days last December, visitors experienced cinematic, projection-mapped displays tailored to the cemetery chapel’s Gothic facade and accompanied by original electronic music; witnessed an artist’s rendering of a solar eclipse; and wandered through Phase Garden, an installation comprising 12 towers—each equipped with a loudspeaker and color-changing lights—that were arranged along a pedestrian path circumscribing a circular garden. The towers were controlled by custom software that generated a constantly shifting musical and visual landscape based on the ever-changing relationships among the Earth, the sun, and the stars.

“All of the pacing, all of the lighting, and all of the sound incidents were truly connected to celestial time,” says Edwards, who adds that the mathematics of the relationships guaranteed that the work wouldn’t repeat itself unless it played for 30,000 more years.

As was the case with Sound Sculpture, however, the technical complexities of solstice’s Phase Garden were invisible to the people who encountered it. Instead, they simply perceived the totality of the installation as a fully integrated piece of sonic and visual art—a singular experience that could not be achieved with either sound or light alone.