General Profile:

Research Project Summary:

Generative models can be used to learn the latent structure of musical sequences and to generate novel sequences by sampling the learned latent space. There have been recent successes doing this directly in the audio domain, but such models take a huge amount of time and resources to train and to sample, which prohibits interactive use. Such systems should not be designed to replace musicians but rather be used as tools to aid the creative process. To this end, symbolic representations such as scores or piano rolls are often used to create more interactive models. Users can input musical sequences by clicking on a grid or musical staff. These sequences can then be extended or harmonized by a generative model. Users can constrain the generated music via interactive interfaces with buttons, sliders, and menus. These can control musical features such as switching between major and minor keys, displaying alternate generated sequences, and allowing users to generate further sequences that are similar to or different from current options. However, there is room to develop methods that allow even more interaction and require less expertise in musical notation. One way of doing this would be to allow users to draw shapes that represent different aspects of music, such as the melodic contour, note density or rubato. A generative model can then choose a sequence of note pitches and durations that best match such shapes. Such a system could be useful for people without much musical experience, for professional composers wishing to quickly iterate ideas or in music/art therapy sessions for people to express themselves more freely. This project uses a variational autoencoder (VAE) with recurrent encoder and decoder networks to generate novel sequences of virtuosic piano music. Curves are generated to match the overall shape of musical sequences by taking a moving average of note pitches. These are then compared to predefined curve classes to find their closest class. These curve classes are used to condition the VAE to generate sequences similar to other sequences with those shapes. An interactive web-based interface allows users to draw their own curves, which are matched to one of the predefined curve classes. These are then used to generate novel musical sequences based on the shape of the user’s curves. Users can view, play back, and download generated sequences via a MIDI piano roll embedded in the interface.