Coupled neural attractors

Towards the end of my PhD I made a second audiovisual rendering of feeding circuit data (the first is quite different and is available here)


For those of you looking to do something similar, this blog post explains how it's done (mainly Matlab). Code and data are available here: neuralAttractor.m, nineSpikeRates.txt. Suggested improvements are welcome too.


Data

The input data for the video is 1000 seconds of spike time data from nine neurons that has been converted into smooth normalized spike rates. The data was recorded from two bilateral triplets of motorneurons and one triplet of 'extra-CPG' neurons that adapt the feeding behavior of the six motorneurons to the availability of food (see Harris et al. 2012 for details).

data = load('nineSpikeRates.txt');

Video

The video is made entirely in Matlab. Use this line to initialize and set the length of the movie

videoFrames = moviein(length(data)-10); 

Then, for t = 1:length(data)-10, run

dataSegment = data(t:t+9, :);
figure(1)
plot3(dataSegment(:,1), dataSegment(:,2), dataSegment(:,3), 'o', 'Color', 'r', 'MarkerSize', 30);
hold on
plot3(dataSegment(:,4), dataSegment(:,5), dataSegment(:,6), 'o', 'Color', 'r', 'MarkerSize', 30);
plot3(dataSegment(:,7), dataSegment(:,8), dataSegment(:,9), 'o', 'Color', 'b', 'MarkerSize', 30);
axis([0 1 0 1 0 1])
grid on
videoFrames(t) = getframe(1);
close(1)

This creates and saves multiple 3D plots showing successive 10 second segments of neuron triplet activity.

Finally, to save the video, use this line

movie2avi(videoFrames, 'videoOut.avi', 'compression', 'none', 'FPS', 10);

Audio

To play music in Matlab I use the Java library JFugue. Download the library jfugue-4.0.3.jar, or whatever is the latest version, from jfugue.org. Use the following lines to make the library available to Matlab. Note that the javaaddpath command doesn't like to be executed as part of a script - you may get an IMPORT error. To get around this you can run the javaaddpath line manually before you run the script (this is possibly a newbie error on my part).

javaaddpath('jfugue-4.0.3.jar')
import org.jfugue.Player
import java.io.File
player = Player() 

The JFugue player takes a single long string as input. Each note is represented by an integer between 0 and 127 enclosed in square brackets (JFugue in fact supports a very wide range of musical commands, like 'Cmaj'). For example, try:

notes = '[48] [50] [52] [53] [55] [57] [59] Cmaj'
player.play(notes)

JFugue can play several notes simultaneously. Each channel or 'voice' is indicated by V1, V2 etc. You can specify an instrument for each channel, e.g. I0 for piano, I24 for guitar etc. You'll also want to specify a tempo for the entire string, using T[beats per minute]. For example:

notes = 'T[120] V1 I0 [48] [50] [52] [53] [55] [57] [59] [60] V2 I24 [60] [59] [57] [55] [53] [52] [50] [48]'
player.play(notes)

To generate the audio for this video I initialized the JFugue string notes = 'T[600] ' and ran the following for n = 1:3

notes = [notes horzcat('V', num2str(n), ' I11 ')];
for t = 10:length(data)
   note = round(mean(data(t,[1 2 3]*n)*30+30));
   if note > 30
      notes = [notes horzcat('[', num2str(note), '] ')];
   else
      notes = [notes 'R '];
   end
end

This takes the mean spike rate of each neuron triplet at each time step (t) and converts it to a bracketed integer between 30 and 60 (a mid-pitch range) unless the spike rate is zero, in which case 'R ' is added, which JFugue interprets as a pause. t starts from 10 here so that the sound played at each time step will correspond to the front of the 10 second segments of activity used to generate the video frames.

Use this to save your audio as a midi file

filePath = File('audioOut.mid')
player.saveMidi(notes, filePath)

Finalize

Now you need to merge video and audio. As far as I know there's no way to do this in Matlab. JFugue only outputs midi files and unfortunately Windows Movie Maker doesn't accept midi, so if that's what you're planning to use to merge you'll need to convert the midi to wav first. I use Switch Sound File Converter for this. You may need to play around with the number of beats per minute at the head of the JFugue string to get audio that's the same duration as the video, especially if you're using many data points per second. If you play the YouTube movie above to the end you'll see that I didn't get this completely right.

An example of the video file I get after merging videoOut.avi with audioOut.wav (this is audioOut.mid converted to wav) in Windows Movie Maker is available here. Good luck.