CodeSounding is an open-source sonification framework which makes possible to hear how any existing Java program “sounds like”, by assigning instruments and pitches to code statements (if, for, etc) and playing them as they are executed at runtime. In this way, the flowing of execution is played as a flow of music and its rhythm changes depending on user interaction.
The major challenges are 1) keeping realtime synchronization between instructions executed and sounds played, 2) how to “compress” the CPU speed into a meaningful rhythm, that is how to going from something like 2GHz to 44kHz.
We propose different approaches, with increasing degrees of realtime synchronization / “code fidelity”:
- every instruction can append one note into an ABC file, which is then converted in a MIDI and played in a separated thread. In this approach we tipically pick one note every thousandth instructions, nevertheless it allows interesting polyphonic pieces.
- we can play MIDI notes directly, via jMusic. We tried a sort of dynamic pitch modulation, depending on the logarithm of the instructions triggered and on how quickly changes its slope. It sounds “dark” and has a poor realtime synchronization: see a live.
- every MIDI-based approach does not allow a real sound mixing; it is possible to play two instruments on the same time, but you cannot change a timbre on realtime (unless by using pitch bend messages, although this affects just the frequency). Therefore a different approach could be to write directly into an audio buffer – in our case, we write into a JACK buffer. This allows a finer wave control and a good code synchronization, but requires to write a custom synthesis algorithm by hand.
- finally, to allow a greater control on realtime sound synthesis we exploit some unit generator-based solutions:
- with JSyn programming-oriented people can trigger sinusoids, filters, etc. without leaving the Java environment. Note that JSyn is not an open source software.
- musicians and GUI-oriented people can use either Max/MSP (commercial) or Pure Data (open source) in order to make visual patches and exploiting the existing objects. The interactions with CodeSounding are based on UDP packets, sent to different ports, so the sonification can live on a different machine. In any case it stayes on a separated process.
 |  |  | |  ABC |
| jMusic | ||||
| JACK |
Example
Kidal D. is author at LeraBlog. The author's views are entirely their own and may not reflect the views and opinions of LeraBlog staff.
You must log in to post a comment.