Postmortem

For this prototype, I was interested in exploring ways to utilize Unity's ML Agents in a sound toy. I actually have a bit of an aversion to machine learning in music after reading one too many sensational articles about AIs that write Bach (a promise that is never really fulfilled and a goal I don't really understand), but having recently starting learning about ML Agents I decided to give them a try.  I specifically avoided, however, compositional goals for the machine learning side of things and instead tried for a spatial and entity-based system. very much inspired by Electroplankton, that would hopefully produced some interesting emergent behaviors.

My feelings on the result are mixed. I like it a as a simple sound toy, but had hoped for more patterned behavior from the ML Agents. Furthermore, only later in the process did I realize that dspTime scheduled behavior and the ML Agents training process aren't especially compatible. I'm not totally sure on the specifics, but I think when the agents run at increased rate for training, that rate is not reflected in the audio scheduling. Aside from some general bugs in training, this also means that the relative timing of scheduled events and ML actions (and therefore the agent behavior generally) is different  in training than when running a trained model. Ultimately this isn't a fatal problem for this sort of project - I wanted expressive behavior as opposed to strict  goal-oriented behavior, so it's okay if the model is a bit broken - but it certainly made training to my preference much harder. 

This challenge also dragged the rest of the prototype down a bit. While I like the theming so far (water bugs!), it would've been nice to work more on the visuals. The musical design also feels underdone. While a pentatonic scale is easy to work with and doesn't feel out of place with the theming, it's a bit boring and I would have at least liked to implement some longer-term variation. I'd have liked to do something with collisions as well, and there is currently a bug with pitch mapping and scheduling where it plays the pitch for where a bug last moved instead of where it is now. 

Nevertheless, I learned a lot with this prototype both about ML Agents and dspTime scheduling, and I at least think it sounds nice.