Squarepusher BTS

abstract

Music video “Terminal Slam” is a subversive video where all the ads around the city is deleted or replaced with the ones related to Squarepusher. Inspiration behind this is Daito Manabe’s thought that the scenery in the city will become rewritable with AR/MR glasses In the near future, as seen in the video.
With machine learning technologies,

  • - Object Detection (to recognize the position of objects in the image)
  • - Semantic Segmentation (to classify the segment of the image into categories based on meaning of each pixels by labeling every pixels)
  • - Image Inpaint (image restoration technique)

are applied to create the video.
Also, masks are generated from those data to add optical camouflage effects in post-processing such as glitch effect only on the areas where humans are.

Since such detection to extract only ads was not possible yet with current machine learning technologies, some operations had to be done by hands. But it is expected that those operations also can be automated within the near future.

Below are footage used in the video, footage added with the effects, and some open source code that was actually used.

Original video

optical camouflage effects to people for privacy reason.

Object Detection

Ads by human

Semantic Segmentation

Ads by human

Mask (Semantic Segmentation)

Ads by human

Depth Estimation

Inpaint

Glitch Composer

Screen capture images of a Max patch to synchronize glitch effects with audio and compose visuals with machine learning data.

Sound Interaction

The quality of motion of audio-reactive graphics improves by extracting each of drums and bass separately from 2-ch sound with Spleeter.
https://github.com/deezer/spleeter

Fake Ads

We created fake ads of Squarepusher to replace the ones in the city with. Below are some exmaples.

Credit

Director, Glitch Effects and Interaction Designer
Daito Manabe (Rhizomatiks)
Film and Editing director
Kenichiro Shimizu (PELE)
Machine Learning Engineer
Yuta Asai (Rhizomatiks)
Video Export Tool Developer
2bit
Effects Artist
Aya Takamatsu (Rhizomatiks)
Ad Graphic Designer
Kaori Fujii (Rhizomatiks)
CG Director
Junichi Ebe (Freelance)
Effects Supervisors
Kenta Katsuno (+Ring)
Takeshi Ozaki (+Ring)
Effects Artists
Mikita Arai (Freelance)
Masaki Takahashi (Freelance)
Digital Artists
Yuki Hirakawa (+Ring)
Yu Onishi (+Ring)
Kenta Hasegawa (+Ring)
Ayaka Yamaguchi (+Ring)
Takeya Kamimura (+Ring)
Ryuichi Ono (Freelance)
CG Producer
Toshihiko Sakata (+Ring)
VFX Airtist
Yoshinobu Okino (Nomad)
Colorlist
Felipe Szulc (Nomad)
Cinematographer
Kazuki Takano (TRIVAL)
Cinematographer / Operator
Takuya Higa (Cyanworksandfilms)
MIDI Data designer
Kyoko Koyama
Cast
SARA (ELEVENPLAY)
Film Producer
Chikako Nagai (PELE)
Producer
Takao Inoue (Rhizomatiks)