I was sufficiently pleased with the results of my previous experiments to create 2D images by having Microsoft Copilot write code for Processing. Accordingly, this week, I wanted to move on to slightly more advanced experiments, namely using Microsoft Copilot and Processing to create simple animations. One quick notes is that Processing does not typically output movie files, so instead I used QuickTime to make screen recordings in order to capture these animations.
I built the first experiment off of the spirograph visualization of the ten brightest stars in the Milky Way. I simply asked Copilot to add some randomness to the spirograph code to create an animation effect. Notice how the various spirographs each moves at its own rate, which I think is a nice feature.
I then had processing create code that would create a video analog of waves in the ocean.
Then, I had Copilot write code that would generate an animation of a constantly morphing plaid pattern.
Regardess of the specifics, these experiments indicate the potential for Processing to create animation that could be used to demonstrate basic scientific principles (such as the wave animation above), or to create animations that could be used in social media posts, or be used as part of a music video.
I am in a working group of faculty tasked with finding ways to incorporate generative AI, specifically Microsoft Copilot, into their workflow. Given my background as a music maker, my intent is explore creative applications of Microsoft Copilot in music, as well as the arts in general.
One use for generative AI is to use it to write computer code. In addition to standard programming languages, such as C++, JavaScript, and Python, Microsoft Copilot can generate code in numerous other languages, including languages that are orient to creating music and graphics. In terms of graphics creation one such programming language is Processing. Amongst other things, Processing will create 2D images and video, as well as interactive images and video using computer keyboard, mouse, and MIDI data as means of input. Clearly this language offers a lot of cool potential for image and video creation, but let’s start out fairly simply to begin with. Let’s start with using data visualization processes to generate graphic scores for music.
Graphic scores are an alternate means of music notation that are commonly aligned with improvisation or aleatoric (chance) music. While generally associated with modernism and post modernism, graphic scores have a long history. One of the earliest surviving examples is a 15th century composition called Belle, bonne, sage by Baude Cordier. The score for this piece is written such that the music looks like a heart.
Data visualization is a more established more conventional process. The core of the practice is to map data on to visual elements to aid in the analysis of a large data set. A successful visualization can help a researcher identify trends or identify anomalies. Some data visualizations can really be visually compelling. Are data visualizations art? I think many may disagree on that. However, can data visualizations be art? I think the answer to that is a definite yes. If we alter the goal of a visualization from being able to reveal patterns and anolamies to researches to the goal of creating a stunning or compelling image, such an image would count, as far as I am concerned, as visual art.
The visual nature of graphic scores often create a curious situation. Namely, many graphic scores are sufficiently compelling enough to count as visual art. However, in use, these visual works would only be seen by the performers, and not the audience, unless the instructions tell the performer to display the score in such a manner that the audience will see it. Accordingly do graphic scores have to be visually compelling? I would say that they do not. They simply have to convey the situation the composer wants to communicate to the performer in an effective way. However I think it does matter if a graphic score is visually compelling. Such a compelling image can potentially inspire a perform, and it may also encourage the performer to take the composition more seriously.
In terms of this experiment that’s where generative AI and Processing come into play. For this experiment, rather than asking generative AI to come up with an image for me to use, I ask it (in this case Microsoft Copilot) to generate Processing code that will generate a visualization of a set of data. As is the case with any use of generative AI, it is possible that that AI may hallucinate and create data that doesn’t exist. So, the results would have to be verified if we were going to use it for research, but we aren’t. In this case we are using it to create a compelling image for use as a graphic score.
I started by asking Microsoft Copilot to generate Processing code that would display spiral graphs. I then asked it to add color, to have the center points spiral out from the center, and to allow the use to save images. From this starting point, I asked it to generate code that would visualize astronomy data. Specifically I asked it to use the ten brightest stars in the Milky Way to map declination onto the x axis, ascension onto the y axis, and brightness onto Spiral size. At this point I also asked it to include a degree of randomness not related to these three parameters so it would generate different outputs each time.
Occasionally the code it generated would not work, so I would tell Microsoft Copilot what error was returned, and it would regenerate the code such that the error was fixed. Regardless, I found the ten brightest stars experiment to be a good proof of concept. Being a fan of transportation, especially commercial aviation, trains, and subways, I decided to move onto a series of experiments involving Boston’s MBTA. Initially I asked Copilot to generate processing code that displayed 50 spiral graphs, each of which relates to the 50 most used subway stops for the MBTA, mapping usage of the station onto size, and using the color of the spiral graph to denote which subway line it relates to.
I found this image to be sufficiently compelling to imagine a graphic score for four or more players, where each player was assigned a route: the green, red, blue, or orange lines. The east / west axis could be mapped onto time, while the north / south axis could be mapped onto pitch. The size of each spiral could be used by the performer to choose the number of notes a given event would consist of, and / or to determine the volume of the event. After several experiments I decided it was necessary to include every single subway station, in order to avoid too much silence, and to give each performer ample opportunity to be involved. Note that the green line offers (by far) the most events with 70 stations.
The red line follows with 22 stations.
The orange offers 20 stations.
The blue station comes up last with only 12 events.
Note that each subway line offers its own distinct contour. The composite score of these stations offers a sparse solo by the green line for the first half of the piece, a busy ensemble for the middle of the work, and a directional, but increasing sparse duo between the red and blue lines at the end.
The ethics of this are certainly disputable. Could I have done this? That is learn yet another computer programming language and write a program that would translate a data set into an image? Yes, undoubtedly I could have. It is in my still set, but I imagine this has saved me tens of hours of work. As time goes on I could learn Processing code well enough to hack the code myself and eventually write my own code. Also, there’s the small matter of learning the mathematics behind spiral graphs. This is also I could have learned on my own, but again, this saved me tens of hours of work.
How does this make me feel as a creator? I’m personally alright with it, as I could have created this piece on my own by doing 15 minutes to an hour of research online to get the data, and then create a score by plotting it by hand using graph paper and colored pencils. This would have likely taken me an hour or two to accomplish, but it would have looked crude and unpolished. Also, I would not have been able to do the spiral graphs. I would have just made dots of various sizes. So, I would easily have been able to create this piece without AI, but it would have looked sloppier, less professional, and cruder. Since I am primarily a musician / composer, and not a visual artist, I don’t feel that the essential part for me (the music) would be substantially different one way or the other.
Another useful comparison I would make in this scenario would be the idea of hiring a professional music copyist to clean up my notation for a performance or publication. When you hire a professional copyist, that individual is not making any creative decisions, they are simply making the composer, arranger, or orchestrator’s creative decisions more legible. However, I would argue that in the sake of honesty it would be worth declaring the use of AI in the score itself for this particular experiment.
As a music technology educator I feel that it is incumbent upon me to have a decent amount of knowledge about the current state of Music AI, as well as have at least a beginning level of expertise in the use of the technology. That being said, I’ve been waiting for an opportunity where it made pedagogical sense to get started with Music AI. I recently revised my songwriting course so that it is open to open to students who don’t read music.
For each project I give students a template in LogicPro. This template features three session player tracks (keyboard, bass, and drums). Students can then edit the tempo, the instrumentation, the chord track, as well as the details of each of the three session player tracks. This allows students to customize the template to conform to the mood of the song they wish to write.
Ultimately, students will overdub vocals of them singing their song. Since I expect that several of the students will be uncomfortable with sharing their vocal tracks with the rest of the students I wanted to give them the option of using vocal replacement technology for their final draft. Currently Suno AI is widely considered to be the best Music AI available.
Accordingly, I recorded the vocals for my first sample song project for the course, and submitted the track, including the three aforementioned session player tracks to my Suno AI account. In order to do vocal replacement and style changes in Suno, you have to have a Pro account, which is ironically their middle tier account, which is currently priced at $10 a month. If you attempt to upgrade an account from a free account, it will offer you a deal of $8 a month, but require you to pre-pay for an entire year. If you are like me, and intend on using Suno for a brief time, for instance a few months, you can select monthly, and price will rise to $10 a month, and you’ll be able to add your payment options.
Figure 1: Upgrading
Go to account, but make sure to select Monthly, instead of Yearly.
Long story short, here’s my original track. I wrote the lyrics, the melody, and the chord progression. The backing track is a super basic backing track created in Logic Pro using three session player tracks (keyboards, bass, and drum machine). I recorded the vocals, and processed them using a noise gate, compression, EQ, pitch correction, vocal transformer, and reverb.
Notice that my original song is pretty odd. It uses the meters 4/4, 5/4, and 6/4. Likewise, the introduction uses an odd progression of a Csus chord followed by an F augmented chord.
I brought this track into Suno AI, and used a Pro account to use the Cover function to replace the vocals, and change the style. I used the advanced feature to have the generated versions follow the original with a 90% accuracy setting. Every time you do this it actually generates two versions. In some cases, the AI generated a version where the lyrics were moved around, so I threw those out. Thus, some of the styles represented below only have one example.
You’ll also hear that several of the styles are not accurate. That being said, what was generated is still fairly appealing (at least from a commercial perspective), and (at least as an artifact) interesting. The melody and chords for the verse and chorus also is usually very similar to the original. In most cases rhythm (meters) seems to be fairly similar in each of the version to the original. Often the generated cover does not use the odd chord progression used in the introduction of the original version, but replaces it with the verse melody and chord progression.
There are some very clear and just criticisms of AI generated art. For instance there are the copyright issues (see https://www.theverge.com/23444685/generative-ai-copyright-infringement-legal-fair-use-training-data for example), the race issues (see https://futurism.com/dall-e-mini-racist or https://www.vox.com/recode/23405149/ai-art-dall-e-colonialism-artificial-intelligence), as well as others. There are also alarmist positions that AI art generators will take away jobs from working artists. While it is easy to to be sympathetic with such a concern I have not seen any actual data or evidence that points to such a trend in any measurable way. I don’t want to be sidetracked in this post, but perhaps in a future post I might explore the question of have drum machines impacted job opportunities for percussionists, which right now may be the closest comparison for which we have a decades long track record.
Regardless of such concerns, I believe that these technological tools are here with us to stay, and that any effort to put the AI genie back in the bottle will likely be unsuccessful. Even if large groups of nations ban or at least significantly curtail the legality of AI art generators, the internet has no boundaries, and any nation that lacks such regulation would likely become the base for any such continued activity. Therefore, for me personally, I think the best course of action is to engage with the technology, discover what it is good at, what it is not good at, and determine whether there may be useful applications for AI generated art. While it will likely take decades to fully digest the ability of AI art generators to transform (both good and bad) culture, I believe there’s no better time to start than the present. As one might imagine, my route into the topic of AI art is through the lens of what could this mean to working musicians.
We live in a visual society. Most individuals are more adapt at processing visual stimuli than they are at processing sound. This presents challenges for musicians.
Social media sites such as Instagram are nearly a necessary component of a working musician’s self promotion strategy. That being said, as a tool, it is really oriented to sharing visual images more than audio. When you do want to share audio in Instagram or TikTok, they typically require a visual component as part of a video file. There are certainly tools for adding images to audio for social media, and there are simple tools for adding titles to audio for use on social media. However, such tools can make one’s videos look very similar to thousands of others on a platform as such tools are often widely adopted.
Furthermore, most social media experts advise having one post per day, and often three or more in order to engage with an audience. This labor is also multiplied by the number of social media sites you use. Even if one eschews social media, musicians frequently have the need for visual content for album covers and concert posters. While there are musicians who enjoy creating visual content, time spent creating such content can often be at the expense of writing, recording, or performing.
There are many time effective solutions for content creation for musicians, including leaving a camera setup in your studio at all times, leaving a corner of your house or apartment ready to go (clean, well lit, uncluttered) for quick video posts, or putting a phone mount on your dashboard so you can post from your car with little effort. That being said, AI art generators can be another tool that musicians can use to help self manage their visual image and social media sites.
When thinking of visual content for social media feeds it is useful to think of an artist’s brand. Most often it is convenient for an artist’s brand to match their personality, so they don’t have to think very hard about whether a given image is consistent with their brand. Fortunately, my artist brand and personality pretty much both reduce to “eccentric weirdo.” AI art generators can be used to create images that are consistent with many, but perhaps not all, visual brands. Those who are interested in experimenting with using such tools to generate content for musicians may want to spend some time in a Facebook or Reddit group devoted to generated AI art to see what sorts of things AI art generators do well, and what they do not do well. Suffice it to say the abilities of AI generators increase whenever someone generates an image, so if the quality and abilities seem to be insufficient to your needs or tastes, you may want to wait a few weeks or a few months, and check back to see how the abilities have progressed.
In future posts I may offer up some of my personal experiences engaging with these tools, but for now, I leave this as an introduction to vein of exploration.
#/media/File:CordierColor.jpg){kind=link}