Beyond the Visual: ‘Maelstrom’ Initiates a New Dialogue between Senses and Information

How do we embrace the overwhelming forces of data today to bridge the gap between evolution and technology, technology and humanity? ‘Maelstrom,’ a collaborative masterpiece by Yufan Xie and Mel Lewis, along with sound maestro Ziqian Yin and visual artist Wei Wu, presents an answer through a groundbreaking real-time audio-visual interactive interface. Premiered at SUSAS (Shanghai Urban Space Art Season), this audio-visual installation, developed in Unreal Engine and utilizing Dolby Atmos spatial audio technology, uses 7.1 surround sound and generative projection, empowering the audience with the means of creating their own unique responsive spatial auditory experience. It explores the concept of “information ecology” and perceptive memories. The installation plunges visitors into a sensory vortex, providing an innovative guided experience.

Cognition in the Information Ecosystem

Data is shaping the future of humanity, along with our capacity to harness it. In contrast to general AI-generated art, Maelstrom places the audience as the driver of the experience, enabling them to shape and unfold the experience through time and space. Confronted with the influx of visual information and the inability to process it, it poses the question of embracing a multi-sensory approach as a means of navigation. Amid the cyclical flow of information, the audience, drawing upon individual memories and existing cognitive habits, gradually learns and adapts the inheritance and evolution of environmental features through spatially mapped messages, revealing a mutual influence and organic symbiosis with the environment. The installation embodies an interconnected ecological system where human actions and outcomes reshape each other.

This piece reflects on the prevailing overflow of data and technology that shapes our information ecosystem and the sensory overload imposed upon humanity. It delves into the impact of “immersive” information environments and generative technologies on our cognition: Within today’s data ecosystem, how do immersive experiences and generative technologies influence our capacity to process information? How do we filter pertinent information, discern the operational patterns of our surroundings, and find a way amidst the tumult of information flows?

“Maelstrom” manifests the flow and transformation of information through spatial audio installations and auditory representations, guiding us through constantly shifting environments. Amid the cyclical ebb and flow of information, the audience, drawing upon individual memories and existing cognitive habits, gradually comprehends the inheritance and evolution of environmental features across scales, actualizing a mutual molding and organic symbiosis with the environment. How can we perceive patterns in otherwise seemingly chaotic and unpredictable streams of activity as a means of not only surviving but also embracing the inevitable interdependence of an unpredictable future? Marshall McLuhan recognized the parallels of today to Edgar Allan Poe’s novel “A Descent into the Maelstrom.” In this short story, a sailor gets so absorbed in thinking that he gets caught in a maelstrom. Recognizing that he cannot row his boat out of the situation, he must observe the objects within the maelstrom and grab onto the recurring objects as a means of escape. The lesson: only through pattern recognition can we cope with information overload.

This vortex embodies the challenges of our information age today, and the exhibit replicates the process of “detachment” through information overload and yet possesses cyclical patterns in time. It epitomizes the process of being repeatedly “sucked into” and “extricated from” the vortex through information overload and filtration.

From the perspective of Information Ecology, the “Maelstrom” team attempts to respond and supplement the theme of this year’s Shanghai Urban Art Season, “Metro-biosis”: “Metro-biosis, or co-existence, as an everlasting topic between humans and their environment, encompasses not just our relationship with the ecological and urban surroundings, but also our dynamic relationship with the surrounding information environment from an individual standpoint. How we process the information within our perceptual range and utilize new tools to harmonize our relationship with information flows is an emotional and cognitive challenge, a dual test posed by personal experiences and environmental scales.”

Crafted with the Unreal Engine and Dolby Spatial Audio, Maelstrom envelopes audiences in an unparalleled spatial-visual and auditory experience through 7.1 surround sound and generative video projections. Guided by the console, visitors can manipulate the projections, navigating through the created environment as if meandering amidst it. Following auditory cues from beyond their visible range, they traverse a network reminiscent of satellite imagery and biological colonies. As participants move outward from the vortex’s core, the turbulence and noise wane, and direction becomes more discernible, ushering in the subsequent scene. These scenes, while strikingly analogous, bear their unique distinctions: visually, they resemble the shifting terrains of maps from different epochs, a portrayal of inherited, evolving, and rejuvenating images. Aurally, they resonate with electronic music and live sound performances composed of recursive sonic landscapes—within these cyclical transitions, participants navigate the space swiftly, drawing upon their past habits and memories. The audience and “Maelstrom” adapt and coexist, embarking on a journey about memory and adaptability amidst an environment teeming with intricate information.

Wei Wu commented on the installation, stating, “This multisensory experience offers the audience a multidimensional perception. We aim to engage their visual, auditory, and spatial senses, drawing them deeply into the artwork and encouraging personal interpretation.”

The Fusion of Art and Technology: Generative Imagery and Spatial Audio

The visual of “Maelstrom” employs generative imagery. Compared to conventional AI art, “Maelstrom” places the audience at the epicenter of the experience, expanding perceptions and experiences and offering fresh perspectives on the organization of the information landscape. The installation employs artificial intelligence generation models to evolve urban satellite images, mirroring geological shifts and generational evolutions. Characteristics and memories passed down through generations enable adaptation to new surroundings based on past experiences. The similarity between preceding and subsequent scenes, along with sensory memories of these scene features, accumulate as experiences, guiding the audience’s movement within the space. Concurrently, the audience’s actions determine the evolutionary direction of the next scene, forging a reciprocal molding relationship with the environment.

From an auditory standpoint, “Maelstrom” is an interactive sound installation, guiding its audience to understand and respond to environmental shifts through an evolving sonic narrative. As an extension of its predecessors, “Acoustic Garden” and “GAP+,” this work transcends traditional audio-visual experiences, morphing into an interactive musical instrument and space where audiences can learn and engage. It challenges the conventions of spatial music and non-linear narratives, moving beyond mere audibility, visualization, and material translation of information. Guided by spatial audio, the installation allows all audience members, including the visually impaired, to participate as independent performers.

Yufan Xie emphasizes: “I’ve always believed that performance and experience don’t solely belong to professional performers but to everyone. Much like the celebratory rituals held by our ancestors in tribes, art, and interactive experiences are collective activities that everyone can partake in, playing a decisive role in the narrative’s direction. Currently, ‘multi-sensory’ is widely used, yet it often appears non-interactive, merging several elements. We strive to introduce a fresh perspective, integrating the musical system with spatial design, resulting in a structural shift in narrative and experience.”

“Maelstrom” is not just an audio-visual feast but also a profound exploration of interactivity in the information age. This piece will be displayed at the Shanghai Urban Space Art Season 2023 until November 20th. Audiences are warmly invited to experience this melding of art and technology.

About the Maelstrom Team

The entire creative team behind “Maelstrom” boasts an educational background in architecture while venturing into various fields, including interactive design, mixed reality, sound art, and media art.

Yufan Xie is an interdisciplinary designer and artist dedicated to the holistic practice of architectural design, computational design, and audio-visual performances. He pioneers interactive narratives and cognitive support systems under overlapping concepts of sonification and visualization, investigating the relationship between the body and the environment within multisensory narratives and the underlying behavioral mechanisms. His works often utilize 3D printing, spatial audio, and mixed reality technologies to materialize audience-led installations. Since 2019, Yufan has concentrated on designing music and performance systems rooted in spatial contexts, venturing into novel avenues of spatial audio performance scenes and interactive music forms.

Mel Lewis is a multi-disciplinary designer who specializes in speculative design and world-building to create aspirational futures through immersive storytelling as they pertain to the challenges of the built environment of today. Real-time installations invite active engagement through multi-sensory participatory experiences employing game development mechanics, architectural design, sound editorial, and musical composition.

Ziqian Yin is a distinguished sound artist and urban planner from Beijing, China. As a sound artist, he passionately engages in various artistic endeavors through electronic music, soundscape design, and scoring. Serving as an urban planner with expertise in geospatial analysis, his research extensively delves into geographical information theory, urban spatial concepts, and environmental studies. While his sonic creations vary, they consistently revolve around crafting spatial order using nuanced layers and textures of sound, forging immersive auditory experiences. His creative methodology predominantly stems from sampling the natural and built environments, as well as digital media, followed by the deconstruction and reconstruction of these auditory elements, seamlessly integrating electronic synthesizers and effect processors.

Wei Wu is a multidisciplinary artist and creative technologist harnessing the intersection of speculative design and emergent technologies. Her interactive work focused on computational expression and narrative performance, utilizing robotic fabrication, 3D scanning, and extended reality, using physical and virtual media to create immersive experiences. Wei holds a masters in Art, Design, and the Public Domain from Harvard Graduate School of Design and cross-registered at MIT Media Lab.

About Shanghai Urban Space Art Season 2023

This exhibition proudly features over 140 participants, teams, and institutions from 14 countries and regions, including China, France, Italy, Spain, Denmark, the Netherlands, Greece, Israel, the United States, Australia, New Zealand, South Korea, Argentina, and Lithuania. Renowned contributors include artists like Qiu Anxiong, Tango, Claude & Petitpierre, Xu Ge, Liu Yi, Cao Minghao & Chen Jianjun; architects such as Zhang Yonghe, Liu Yichun, Zhang Bin, Liu Yuyang, Zhang Jiajing, etc., Esteemed institutions like the Mies van der Rohe Foundation, Rome’s 21st Century Art Museum, Central Academy of Fine Arts Design School, Shanghai Natural History Museum, and Suzhou Garden Museum have all come together to present over 30 outdoor exhibits and more than 70 indoor displays.

The thematic exhibition area has assembled a formidable joint curatorial team convened by the Shanghai Urban Planning & Design Research Institute. The organizer invited Dean Li Xiangning of Tongji University’s School of Architecture and Urban Planning to lead as the chief curator, with the young curator Gao Changjun serving as the executive curator. Collaborating with UCCA Ullens Center for Contemporary Art and the Nature Conservancy, the team also boasts the esteemed architect Professor Yuan Feng, who takes on the role of the main exhibition hall’s renovation architect. This collective, encompassing expertise in planning, architecture, art, and ecology, collaboratively delves into urban strategies emphasizing a harmonious coexistence between humans and nature.

Main Exhibition Area: Xuhui West Bank Southern Section, Longyao Road to Dianpu River 5km Riverside Area.

Main Exhibition Hall Address: 100 meters south of the Gangkou Ferry Station, Xuhui District, Shanghai (Formerly the White Cat Warehouse).

Opening Dates: September 25, 2023 – November 20, 2023

Open Tuesday – Sunday, 10:00 AM – 5:00 PM. Free admission to the public. Closed on Mondays (Open as usual on September 25 and October 2).

Organizers: Shanghai Planning&Nat Resources Bur, Shanghai Municipal Administration of Culture and Tourism, Xuhui District People’s Government, Yangtze River Delta Ecological Green Integrated Development Demonstration Zone Executive Committee, Jiading District People’s Government, Qingpu District People’s Government, Songjiang District People’s Government, Fengxian District People’s Government, China (Shanghai) Pilot Free Trade Zone Lin-gang Special Area Administration.

Hosted by: Shanghai Urban Planning & Design Research Institute, Tongji University, Shanghai West Bank Development (Group) Co., Ltd.