Rhythm of Motion Extraction and Rhythm-Based Cross-Media Alignment for Dance Videos

Wei-Ta Chu and Shang-Ying Tsai

Multimedia Computing Laboratory
Dept. of Computer Science and Information Engineering
National Chung Cheng University

1. Introduction

We present how to extract rhythm information in dance videos and music signals, and accordingly correlate two media based on this novel representation. From dancer’s movement, we construct motion trajectories, detect turnings and stops of trajectories, and then estimate rhythm of motion (ROM). For a music signal, music beats are extracted based on integration of audio dynamics in different frequency bands. Two modalities are therefore represented by sequences of rhythm information to facilitate determination of cross-media correspondence. Two applications are developed to show the feasibility of utilizing cross-media correspondence – background music replacement and music video generation. In the experiments, we evaluate performance of ROM extraction, and conduct subjective/objective evaluation to show that rich experience can be obtained by the proposed applications. We also suggest that rhythm information may be used in more multimedia content applications, such as rhythm-based multimedia retrieval.

2. Evaluation Dataset

Note that the data provided here may be used freely for research purposes but it cannot be used for commercial purposes.

We provide videos with Audio Video Interleave (AVI) Format.
Encoding settings -
Video: MPEG-4 Video 320x240 (4:3) 30.00fps 232~247Kbps
Audio: MPEG Audio 48000Hz stereo 320Kbps
Recommended player: Cyberlink PowerDVD or media player classic (download)

• Dataset 1: Single dancer with clear background but with heavy shadow -- Download (14.3 MB)

• Dataset 2: Single dancer with relatively complex background, with slight camera motion. Dancers may move arbitrarily in the field of view -- Download (95.8 MB)

• Dataset 3: Multiple dancers with complex background. Videos in this set are longer -- Download (140 MB)

3. Background Music Replacement (Dataset2)

Sample Results:

Original Video (Click to download)	Video After Background Music Replacement (Click to download)

All Results:

Original Video	Video After Background Music Replacement	Background Music
1_1 1_2 1_3 1_4 1_5 1_6 1_7 1_8 1_9 1_10	1_1 1_2 1_3 1_4 1_5 1_6 1_7 1_8 1_9 1_10	"All Night Long" by Mary Jane Girls
2_1 2_2 2_3 2_4 2_5 2_6 2_7 2_8 2_9 2_10	2_1 2_2 2_3 2_4 2_5 2_6 2_7 2_8 2_9 2_10	"Sax A Go Go" by Candy Dulfer.
3_1 3_2 3_3 3_4 3_5 3_6 3_7 3_8 3_9 3_10	3_1 3_2 3_3 3_4 3_5 3_6 3_7 3_8 3_9 3_10	"Funk You Up" by Erykah Badu
4_1 4_2 4_3 4_4 4_5 4_6 4_7 4_8 4_9 4_10	4_1 4_2 4_3 4_4 4_5 4_6 4_7 4_8 4_9 4_10	" Hip Hop is Dead" by Nas

 

4. Music Video Generation

Music Video of Dataset 2 (download)
Background music: "Smack That" by Akon ft. Eminem



Music Video of Daraset 1 & 2 (download)
Background music: "Funk You Up" by Erykah Badu.

5. Citation

W.-T. Chu and S.-Y. Tsai, "Rhythm of Motion Extraction and Rhythm-Based Cross-Media Alignment for Dance Videos," IEEE Transactions on Multimedia, vol. 14, no. 1, pp. 129-141, 2012.

Any problem please contact .