Humanoid Drone Swarm × Real-time Motion Capture: "Giants Dancing in the Sky"

Our company supports April Dream, which aims to make April 1st a day to express dreams. This press release is a dream of "NILL Co., Ltd.".

NILL Co., Ltd. (Headquarters: Tokyo, hereinafter "NILL") has launched the "Swarm Shadow Synchronization Project," which involves thousands of drones flying in a human formation and synchronizing in real-time with the movements of a performer on the ground.

Giants Dance on the Night Sky Stage

In recent years, drone shows have seen tens of thousands of drones flying in formation around the world, greatly expanding the possibilities of the night sky as a canvas. However, the dominant method of production has been to replay pre-programmed trajectories. "Drone swarms that respond to human movements in real-time" have not yet been realized globally due to challenges in collision avoidance and latency control.

The Swarm Shadow Synchronization Project steps into this "never-before-done" territory.

The full-body motion of a single ground performer is captured in real-time, and that data is immediately mapped to a swarm of thousands of drones in the air. The drones form a formation along the human skeleton; if the performer raises an arm, the drones' "arm" rises, and if they dance, the drones' "body" dances—a glowing giant tens of meters tall appears in the night sky, replicating the movements of a human on the ground in real-time.

NILL's Motion Capture Technology Connects Sky and Ground

In this project, NILL is leading the motion capture direction and the development of real-time motion transmission technology.

As a creative development studio specializing in VR/MR content development and virtual live event development, NILL has extensive experience with real-time 3D technology utilizing Unity and Unreal Engine. From motion capture direction for music videos to technical supervision for interactive content, the technology to "convert human movements into digital data and reflect them in real-time visuals" is a core strength of NILL.

In the Swarm Shadow Synchronization Project, this technology is being extended from "reflection in visuals" to "physical control of drone swarms." We are developing a system to transmit performer movements to drone swarms with low latency, allowing thousands of drones to safely and smoothly trace human movements.

Drone Swarm = 3D Point Cloud Data to End the "One-Night-Only" Performance

By its nature, a drone show is a "one-night-only" performance. The moment a giant of light dances in sync with a human in the night sky has, until now, been exclusive to the audience present at the venue.

In this project, the thousands of drones flying in the night sky are treated as 3D point cloud data. The position coordinates, emission color, and trajectory of each drone are recorded in real-time, and the formation data itself, generated from the performer's movements, is saved as a 3D asset of 3D point cloud data.

The light trails, the three-dimensional formation of the drone swarm—the entire spatial experience, which cannot be fully conveyed by 2D video, is completely recorded in 3D, and the performance itself is assetized as IP (intellectual property).

By treating the drone swarm projected in the sky as "point cloud data" in 3D space and archiving the "formation trajectory generated in real-time from the performer's movements" as 3D data (IP creation)—we are challenging a new way to preserve drone shows that goes beyond the scope of video recording.

Currently, technical verification is underway, and for 3D archiving of performances, we are adopting "3Dasset.io" (https://3dasset.io/), a platform in Japan that supports various 3D data formats including 3D Gaussian Splatting (3DGS).

Collaboration with World-Renowned Artists and Special Stages

We plan to debut the Swarm Shadow Synchronization Project with a world-renowned dancer and choreographer as the performer, at a special location symbolizing Japan.

The moment the performer quietly raises an arm, the giant in the night sky mimics the movement, and thousands of light points simultaneously arc. As the performance intensifies, the giant of light also dynamically moves, and the ground and sky respond as one living entity—we will make this vision a reality.

About Us

NILL Co., Ltd. is a creative company that plans and produces highly experiential content centered on virtual live events, XR, and 3D expression technologies. Utilizing motion capture and real-time expression, we provide end-to-end solutions from strategic design to implementation for diverse needs in entertainment and corporate promotion. By combining cutting-edge technology with unique directorial skills, we continue to create "new experiences where reality and virtuality merge."

NILL Co., Ltd.
1-3-12 Shinjuku, Shinjuku-ku, Tokyo 160-0022 Ichome Sambankan 601

HP：https://nill.jp/