CodeSounding is an open-sourcesonificationframework which makes possible to hear how any existing Java program “sounds like”, by assigning instruments and pitches to code statements (if, for, etc) and playing them as they are executed at runtime. In this way, the flowing of execution is played as a flow of music and its rhythm changes depending on user interaction.
The major challenges are 1) keeping realtime synchronization between instructions executed and sounds played, 2) how to “compress” the CPU speed into a meaningful rhythm, that is how to going from something like 2GHz to 44kHz.
We propose different approaches, with increasing degrees of realtime synchronization / “code fidelity”:
- every instruction can append one note into anABCfile, which is then converted in a MIDI and played in a separated thread. In this approach we tipically pick one note every thousandth instructions, nevertheless it allows interesting polyphonic pieces.
- we can play MIDI notes directly, via jMusic. We tried a sort of dynamic pitch modulation, depending on the logarithm of the instructions triggered and on how quickly changes its slope. It sounds “dark” and has a poor realtime synchronization:see a live.
- every MIDI-based approach does not allow a real sound mixing; it is possible to play two instruments on the same time, but you cannot change a timbre on realtime (unless by using pitch bend messages, although this affects just the frequency). Therefore a different approach could be to write directly into an audio buffer – in our case, we write into aJACKbuffer. This allows a finer wave control and a good code synchronization, but requires to write a custom synthesis algorithm by hand.
- finally, to allow a greater control on realtime sound synthesis we exploit someunit generator-based solutions:
- withJSynprogramming-oriented people can trigger sinusoids, filters, etc. without leaving the Java environment. Note that JSyn is not an open source software.
- musicians and GUI-oriented people can use eitherMax/MSP(commercial) orPure Data(open source) in order to make visual patches and exploiting the existing objects. The interactions with CodeSounding are based on UDP packets, sent to different ports, so the sonification can live on a different machine. In any case it stayes on a separated process.
 |  |  | |  ABC |
| jMusic | ||||
| JACK |
