CAREER: Human-Computer Collaborative Music Making

职业：人机协作音乐制作

• 批准号: 1846184
• 负责人: Zhiyao Duan
• 金额: $ 49.92万
• 依托单位: University of Rochester
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2019
• 资助国家: 美国
• 起止时间: 2019-06-01 至 2024-05-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1846184&HistoricalAwards=false
• 关键词: CAREER; Human; Computer; Collaborative; Music

Music is part of every culture on earth, and the enjoyment of music is nearly universal. Music performance is often highly collaborative; musicians harmonize their pitch, coordinate their timing, and reinforce their expressiveness to make music that strikes the hearts of the audience. This research envisions a human-computer collaborative music making system that allows people to collaborate with machines in a manner similar to that in which we collaborate with each other. This is of great significance, as we live in a world where the interaction between humans and machines is becoming deeper and broader, so developing systems that allow us to collaborate with machines is a primary goal of research into cyber-human systems, robotics, and artificial intelligence. Project outcomes will advance the state of the art in automated accompaniment systems by empowering machines with much stronger music perception skills (audio-visual attending to individual parts in ensemble performances vs. monophonic listening), much more expressive music performance skills (expressive audio-visual rendering vs. timing adaptation of audio only), and much deeper understanding of music theory and composition rules (composition and improvisation skills vs. music theory novice). This project will showcase the powerful connection between music and technology, which has inspired generations of great multidisciplinary thinkers such as Pythagoras, Galilei, Da Vinci, and Franklin. The techniques developed in this project will be applied to augmented concert experiences through collaborations with the Eastman School of Music and the Chinese Choral Society of Rochester. Outreach to pre-college and college students will be accomplished through a variety of activities, including lab visits, a summer mini-course on "music and math" and teaching and advising in the unique and interdisciplinary Audio and Music Engineering program at the University of Rochester. The project has four research thrusts with the following expected outcomes: 1) Attending to Human Performances: algorithms for machine listening and visual analysis of multi-instrument polyphonic music performances; 2) Rendering Expressive Machine Performances: computational models for expressiveness and audio-visual rendering techniques for expressive performances; 3) Modeling Music Language for Improvisation: computational models for compositional rules, and algorithms for music generation, harmonization, and improvisation; 4) System Integration: a human-computer collaborative music making system, and a set of design principles backed by subjective evaluations. The research will advance existing interaction mechanisms toward human-computer collaboration. It will also advance the current static-object-displaying type of augmented reality to more intelligent, dynamic and collaborative augmented reality in music performances. The research on audio-visual analysis will advanes both machine listening and visual understanding of audio-visual scenes in the music context. The research on visual rendering of expressive performances will open a new field of computational modeling of visual expressiveness in musical performances. And the research on computational music language models is fundamental for many tasks in music informatics, including transcription, composition, and retrieval. The integration of analysis, performance and music language modeling towards a real-time collaborative system represents a new level of intelligent real-time computing.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

音乐是地球上每种文化的一部分，音乐的享受几乎是普遍的。音乐表演通常是高度协作的。音乐家协调他们的音调，协调他们的时机，并加强表达能力，使音乐吸引观众的心。这项研究设想了人类计算机的协作音乐制作系统，该系统使人们可以与机器协作的方式类似于我们彼此合作的方式。这具有重要意义，因为我们生活在一个人类与机器之间的相互作用变得越来越深，更广泛的世界中，因此开发使我们与机器合作的系统是研究网络人类系统，机器人和人工智能的主要目标。项目成果将通过具有更强的音乐感知能力（在整体表演中与单个零件与单声道听力与单声道听力的单个部分相处）的能力来提高自动化伴奏系统的艺术状态）新手）。该项目将展示音乐和技术之间的强大联系，这激发了众多伟大的多学科思想家（例如毕达哥拉斯，加利利，da Vinci和Franklin）的几代人。该项目开发的技术将通过与伊士曼音乐学院和罗切斯特中国合唱学会的合作来应用于增强音乐会的体验。将通过各种活动，包括实验室访问，关于“音乐和数学”的夏季迷你课程以及在罗切斯特大学独特，跨学科的音频和音乐工程计划中的教学和建议来实现与大学和大学生的宣传。该项目具有四个研究作用，具有以下预期结果：1）参加人类表演：用于机器聆听的算法和对多仪器多形音乐表演的视觉分析； 2）渲染表达机器的性能：用于表达性能的表达性和视听渲染技术的计算模型； 3）为即兴创作的音乐语言建模：构图规则的计算模型，以及用于音乐生成，协调和即兴创作的算法； 4）系统集成：人力计算机协作音乐制作系统，以及一套由主观评估支持的设计原则。这项研究将促进现有的互动机制朝着人类计算机的协作发展。它还将把当前的静态对象播放类型的增强现实类型推向音乐表演中更聪明，动态和协作的增强现实。关于视听分析的研究将对音乐环境中的音频场景进行机器聆听和视觉理解。关于表达性能的视觉渲染的研究将为音乐表演中视觉表达的计算建模开辟新的领域。关于计算音乐语言模型的研究对于音乐信息学的许多任务，包括转录，构图和检索至关重要。分析，性能和音乐语言建模与实时协作系统的整合代表了一个新的智能实时计算水平。该奖项反映了NSF的法定任务，并被认为是值得通过基金会的知识分子和更广泛影响的评估评估标准来通过评估来支持的。

项目成果

BeatNet: A real-time music integrated beat and downbeat tracker

BeatNet：实时音乐集成节拍和强拍跟踪器

• 发表时间: 2021
• 期刊: International Society for Music Information Retrieval
• 作者: Heydari, Mojtaba;Cwitkowitz, Frank;Duan, Zhiyao
• 通讯作者: Duan, Zhiyao

SingNet: a real-time Singing Voice beat and Downbeat Tracking System

SingNet：实时歌声节拍和强拍跟踪系统

• DOI: 10.1109/icassp49357.2023.10096580
• 发表时间: 2023
• 期刊: Speech and Signal Processing (ICASSP
• 作者: Heydari, Mojtaba;Wang, Ju-Chiang;Duan, Zhiyao
• 通讯作者: Duan, Zhiyao

BachDuet: A deep learning system for human-machine counterpoint improvisation

BachDuet：人机对位即兴创作的深度学习系统

• 发表时间: 2020
• 期刊: Proceedings of the International Conference on New Interfaces for Musical Expression
• 作者: Benetatos, Christodoulos;VanderStel, Joseph;Duan, Zhiyao
• 通讯作者: Duan, Zhiyao

Draw and Listen! A Sketch-Based System for Music Inpainting

画和听！

• DOI: 10.5334/tismir.128
• 发表时间: 2022
• 期刊: Transactions of the International Society for Music Information Retrieval
• 作者: Benetatos, Christodoulos;Duan, Zhiyao
• 通讯作者: Duan, Zhiyao

Skipping the Frame-Level: Event-Based Piano Transcription With Neural Semi-CRFs

• 发表时间: 2021
• 作者: Yujia Yan;Frank Cwitkowitz;Z. Duan