[中][ENG] 動捕實驗：敏捷地表演虛擬舞蹈

Mocap Experiment: An Agile Way to Explore Virtual Dance

文：譚之卓

關鍵詞：動作捕捉、⾝體、去軀體化、實驗、感知、虛擬現實、虛擬舞蹈、實時表演

實驗的動機

我的動作捕捉（mocap，下稱：動捕）實驗是測驗物理和虛擬現實中的動覺意識及本體感覺，在實踐中使環境的無形维度變得可感知。主要透過編創虛擬舞蹈，探索「身體與技術之間的互動」將如何為創作者和觀眾產生對現實的感知。

我從好奇甚麼是「虛擬身體的在場」（the presence of virtual bodies）開始。對舞蹈x技術領域的濃厚興趣促使我於2020年2⽉參與了由城市當代舞蹈團組織的網路駐留計劃，與國際藝術家在線合作。隨後我編排與跳《渡》探究真實與虛擬的邊界，運用頭戴式顯示設備來帶觀眾及舞伴進入想像的世界，曾以PCVR版本及劇場形式呈現。2020年12月我加入Eugenia Kim組織的Leonardo21藝術研究駐地，發起一系列實踐包括mocap-ing、設計虛擬現實世界、與跨學科學者進行討論等。

在技術的硬件越來越先進、軟件及內容⾼速發展的環境，影響表演藝術的範式轉移。例如第一視角的可及性轉變了本體感覺。我意識到研究虛擬舞蹈，需要花精力去大膽探索甚麼是身體。首先得先了解動捕，此技術可記錄物體或人的運動的過程，被應用在醫學、遊戲和教育等領域。動捕包括光學系統、使⽤附著在舞者⾝體上的標記或無標記的深度動捕相機等。我使用的是⼀套無線傳感器套裝Rokoko Smartsuit Pro。

我做實驗的目標之一，是尋找身體和表演的基本關係。我基於專業舞者的經驗和外部反饋，在動態中更新創意概念。也許心理壓力和種種磨練需要時間轉化為可練習的想法，但這個過程有助於我更好地吸收知識。

動捕實驗過程

動捕的初步實驗／照片由譚之卓提供

我的動捕實驗中有一個練習是想像傳感器（sensors）透過皮膚包裹著我的關節和骨頭，塑造著我的肌肉和神經系統。我使用的設備有19個傳感器收集運動數據，數據輸入電腦後可創建新的骨骼架構。由關節驅動的化⾝（avatar），既是阻力也共享能量，讓我將想像的舞蹈在不同空間實現。因此在這個可循環的練習中，我能突破物理身體的限制，不依賴2D影像去紀錄，將有體積的運動軌跡變成數據。當我感知全身捕捉的意象，可以在虛擬環境中轉化有意義的行為。

遇到去軀體化（disembodiment）的多種故障（glitches）時，我理解到設備的特性，如膝蓋的⽅向與頭部不⼀致時會使電腦⽣成的化⾝有無意識滑動的情況等。為了讓⾃⼰在虛擬空間裡有主動表演的權⼒，⽽不⽤後期花太多時間在清洗數據，我需在動捕前清楚表演的邏輯，讓表演能更精確地被渲染。反向的流程是我即興地玩任務，然後將動捕的數據組合為一系列，輸入遊戲引擎，再詮釋虛擬⾝體的執⾏內容。我發現實驗中動捕設備的特性不會直接影響創作的意圖，因為我使用技術性較低的方式，它並没有為我提供深刻的感官反饋。雖然作為一個對象的動捕服裝可以被编排並用於不同的目的，去探索表演夥伴提供的策略、與技術藝術家交流。實際上還有很多未知數有待發現。

動捕延伸的活動：2021年7至9⽉我被聘為 ROXY - TanzLabor 遠程數字舞蹈創客，在⼀個劇場的裝置中進⾏AR舞蹈表演。21年9⽉我在柏林數字⼈⽂藝術的研討會分享經驗，視頻稍後會公開。

虛擬舞蹈的實時表演

2021年4⽉，我以客座藝術家身份與瑞士XR先鋒藝術家Gilles Jobin及其團隊合作，擴展想像力的邊界。我參與的La Comédie Virtuelle是多用戶社交VR體驗，前期與三位舞蹈藝術家合作排練。公眾可通過虛擬現實設備（Oculus和HTC Vive）或電腦參與。他／她們會變成不同顏⾊的火柴棍，與舞者和其他虛擬化⾝相遇，任意走動並聽到不同區域裡的音樂。

數字實時舞蹈涉及很多設備的組合，每⼀個表演者需要與技術⼈員討論虛擬⾝體的變化。我會以形容為「隱形編舞」的⽅式來增強意識去編程內容，從⽽在跨領域合作中建立信任，感知到不同⼈在混合（hybrid）現實中的體驗。

我在家實現實時演出／照片由譚之卓提供

在作品中，我的化身在日內瓦最古老的劇院外表演，真身卻全程在家裡完成實時表演及傳輸動捕數據。場景中也會播放實境視頻，讓公眾了解舞蹈從何⽽來。因此我在家設定了多個屏幕，確定效驗（calibration）的位置可以被現場看見。通過即時反觀捕捉的數據和聽到舞伴的聲音，來改進虛擬和現實世界中的誤差去達到較生動的感應。

在演出中，化身近距離接觸會出現穿透身體的情況，有時甚⾄⾶出虛擬空間。有趣的是「效驗」作為淡入或淡出虛擬環境的選擇，是與技術舞伴精心設計的互動。換句話說，動態數據在引擎中的偏離（displacement）是真實的共舞經驗。

總結

我還在觀察隱形編舞中虛擬⾝體的變化，來幫助挖掘我和世界的關係。數字軀體化（digital embodiment）將⽤新的材料和技術代理（agency）。也許讓參與者在現實的地⾯上跨越障礙去冒險、體驗不同的⼈⽣、交換虛擬的⾝份等，他／她們可對虛擬⾃然進⾏反饋（feedback），想像多維度的⽣命。

技術的潛⼒和危機在持續改變⼈的⽣活，而個體和群體正在嘗試⾯對這些挑戰。我認為在理解了領域中的第一原理（first principle）、⽅法和個⼈興趣後，將更⼤膽地感知「甚麼是現實？甚麼是⾝體？甚麼是舞蹈？」我期待藝術家將未來的視野和研究轉化為⼤眾可以接觸的形式，與社群分享經驗。

==

譚之卓

年輕的亞洲跨界電影和技術的舞蹈創作者。專注於身體的目的、形式和方法，探索藝術和世界的真實。現任城市當代舞蹈團專業舞者。導演的舞蹈短片作品《喺度？》、《月之悖論 I》入選15個國際舞蹈影像節。

---

Mocap Experiment: An Agile Way to Explore Virtual Dance

Original text: Zelia ZZ Tan

Translator: Penelope Zhou

Keywords: motion capture, body, disembodiment, experiments, senses, virtual reality, virtual dance, real-time performance

Motivation Behind the Experiment

Remote collaboration／Photo provided by Zelia ZZ Tan

My motion capture (mocap) experiment is designed to test kinaesthetic awareness and proprioception in both physical and virtual realities. In practice, the invisible dimension of the environment becomes perceivable. Through the choreography and performance of virtual dance, this mocap experiment sets out to explore how the interaction between the body and technologies can help creators and audiences form a perception of reality.

My exploration of the subject started with the presence of virtual bodies. I have always been interested in the possibilities of the cross-over between dance and technologies, and it was this curiosity that inspired me to take part in the City Contemporary Dance Company’s Online Artist in Residence Programme in February 2020, through which I collaborated with a number of international artists using the power of the internet. Later, I choreographed and performed my work trans-, a contemplation on the boundaries between physical and virtual realities. The piece, which invited the audience and dance partners into an imaginary realm using VR headsets, was presented both as a PCVR experience and a theatre performance. In December 2020 I joined the Leonardo21 artistic research residency organized by Eugenia Kim and engaged in a series of mocap projects, virtual reality designs and interdisciplinary discussions with professionals from multidisciplinary backgrounds.

Today, cutting-edge technologies—advancing at an unprecedented speed both in hardware and software—are creating a paradigm shift in the nature of the performing arts. For example, the accessibility of the first perspective transforms proprioception. This made me realize that in order to explore virtual dance, I must first understand what constitutes a “body”. For that, I enlisted the help of mocap, a technology that records the movement of objects or people and has been employed in a wide range of applications in areas such as medicine, gaming and education. The gear involved includes optical systems, various types of markers and mocap cameras, among others. I opted for the Rokoko Smartsuit Pro, a mocap suit with built-in wireless sensors.

The main goal of this experiment is to search for the fundamental relationship between the body and the act of performing. I revised and improved my creative concept based on my experience as a dancer and external feedback. It took some time for me to turn my mental stress and the challenges I faced into practicable ideas, but this process was instrumental in helping me digest all the new knowledge and experience.

The Mocap Process

When I danced wearing the mocap suit, I imagined that the sensors were attached to my joints and bones and were moving with my muscles and nervous system. The suit has 19 sensors and the movement data that they collect is processed by the computer to construct a brand new bone structure. This avatar allows me to present the dances of my imagination in another space, and to break through the limitations of the physical body, turning volumetric movement trajectories into digital data without having to rely on 2D imagery. As I practiced more, I developed a better awareness of how my entire body was being tracked and was therefore able to dance in a way that could translate into more meaningful behaviour in the virtual environment.

Mocap data application／Photo provided by Zelia ZZ Tan

Meanwhile, various glitches involving disembodiment helped me gain a better understanding of the features of the equipment—for example, when my knee and my head moved in different directions simultaneously, my virtual avatar started sliding. In order to have maximum control of my avatar’s movement rather than having to spend a lot of time on data clean-up, I needed to have a clear understanding of how to perform my dance so it could be accurately rendered digitally. I also tried reversing this process, by first improvising various tasks, then combining and inputting the mocap data into the game engine, and interpreting the actions of the avatar in the following steps. What I discovered was that the features of the mocap equipment do not have a direct impact on my creative intentions, because I used a less technical approach and it did not provide me with profound sensory feedback. Although the mocap suit as an object can be choreographed and used for different purposes, to explore the strategies provided by performance partners and communicate with technical artists, in fact many unknowns remain to be discovered.

Related mocap activities: From July to September 2021 I was hired by ROXY – TanzLabor as a Distant Digital Dance Maker to perform AR dance in a theatre setting. In September 2021, I shared my experience in Digital Research at the Humanities and Arts conference in Berlin, of which the video clip will be made public later.

Real-time Virtual Dance Performances

In April 2021, I started a collaboration with Swiss artist Gilles Jobin—known for his cutting-edge work in XR (extended reality)—and his team as a guest artist. The project, La Comédie Virtuelle, was designed to be a multi-user social VR experience whose goal was to expand the boundaries of our imagination. Prior to my involvement, three other dancers had participated in it. The public could access the experience using VR headsets（Oculus and HTC Vive）or their computers. As they enter the virtual reality, viewers are transformed into matchsticks of varying colours and have encounters with the dancer and other avatars; they can walk around freely and hear the music in different areas.

Real-time virtual dance involves a combination of various equipment. Every performer needs to discuss with their technical team how they want their digital body to move or transform. Through a method that I describe as “invisible choreography”, I was able to build trust in this interdisciplinary collaboration and to get a sense of what this hybrid reality experience was like for different people.

In the work, my avatar danced outside the oldest theatre in Geneva, Switzerland, while I was in fact in my own home, performing and generating mocap data in real-time, the footage of which was played alongside the virtual performance to provide the audience with an understanding of how the dance came to be. I set up a series of screens in my home and made sure that my mocap systems were calibrated correctly, so that the viewers could see me. Through observing real-time data and listening to my dance partners, I was able to reduce the discrepancy of movement between the physical and virtual realities, thus achieving more vivid telepathy.

During the performance, when avatars came in close proximity with one another, sometimes they passed through each other’s bodies, or even disappeared from the virtual space altogether. What is interesting is that "calibration", as an option for fade-in or fade-out of the virtual environment, is the choreographed interaction with the technical partner. Namely, the displacement of dynamic data in the game engine is part of the authentic experience of a collaborative virtual dance.

Conclusion

Via my observations on the changes of the virtual body when performing the invisible choreography, I was able to further explore the relationship between myself and the world around me. With new materials and technological agency, digital embodiment can enable participants to take risks, experience a different life, exchange identities, and interact with the virtual environment, giving feedback to the virtual environment, and imagining life with many dimensions.

With the potential and pitfalls of technology continuously altering every facet of life, we are working hard to address the myriad challenges posed by technological advancement, both individually and collectively. After obtaining a deeper understanding of the first principles, the methods and my own interest in this area, I feel empowered to look further into the questions of “What is reality?”, “What is the body?” and “What is dance?”. I look forward to seeing more artists turning their creative vision and research into multimedia, interdisciplinary artwork that is accessible to the public, and sharing their experiences with local and international communities.

==

Zelia ZZ Tan

A young Asian dance creator crossing over to film and technology. Focused on the purpose, form and method of the body, to explore the truth of art and the world. A professional dancer at the City Contemporary Dance Company. Among the five short films she has directed, Over Here? and Moon Paradox I have been presented at 15 international dance film festivals.