Real-time visualisations with audio and video, which direction should I look?

Question

I have a little pet project with a friend where we're looking to create a program for real-time visualisations, mainly utilising audio/video material, controlled by MIDI. Now, the MIDI part is not a problem, you seem to find decent solutions for almost any language, but I find myself terribly unsure of which way I should look regarding handling the video in a smart way. I'm looking for both fast seeking and additional visualisations (multiple superimposed pictures for example).

I have already experimented with a couple of options that I found were extremely easy to play with and seemed to offer at least something for the task, but with each of them I felt I might run into dead ends or low performance later on when looking to add features etc. So far, I tried Pure Data, Max and Processing.

What I'm mostly asking advice for is to direct me onto an optimal or at least a decent path regarding dealing with the videos. The biggest problem is I find myself using all my time only trying to find out what programming language or library I should use. If I only got that much guidance I could finally start really working on it and advance.

I suppose I'm most comfortable with python but any suggestions are welcome. I have read a little about gstreamer and I'm thinking there might be something there, but now we're talking about a relatively low level library that would take at least some time to produce any results with, as opposed to Processing or Pure Data/Max, for instance.

In addition to the language/library I'm curious about the importance of the video format. It goes a little beyond me when we start talking about codecs, I-, P-, B-frames and whatnot. Who knows, there could even be a solution where we'd use an optimal video format, cram that baby into RAMdisk or something and get satisfactory seek speed with only that.

Answer 1

personally i would suggest using a cross-platform (at least w32, linux, OSX) FLOSS solution. if you ever want to change your OS, you can pretty much rule out VVVV and max/jitter.

i would suggest to either use:

• Pure Data / Gem for a graphical patching system

• Processing if you prefer a traditional, text-based approach to programming (Processing is really just a front-end to java)

note, that personally i am highly biased towards Pd.

Answer 2

my guess is that the problem is not MIDI input. most languages do support midi devices even home made ones. i guess its better to check out more specific software that deal with real time video/manipulation/projection

one which is kind of easy is VVVV http://vvvv.org/

another one is TouchDesigner used in Amon Tobin's ISAM shows (check them out) http://createdigitalmotion.com/2011/08/amon-tobins-isam-behind-the-scenes-ask-your-questions/

Answer 3

I recommend having a look at creative toolkits and for practical reasons choose what language your prefer:

• MaxMSP with Jitter/PureData
• for c++ try OpenFrameworks with ofxMixi or Cinder (and there should be MIDI block/addon for it)
• for Java try Processing with proMIDI

If language isn't an issue, maybe go ahead with MaxMSP and Jitter. Max has objects for handling midi input and parsing it. Jitter deals with fast matrix operations(images/video/gl/etc.) I think the equivalent for PureData is Gem

From experience, Java's garbage collector sometimes has the nasty habit of getting in the way when dealing with realtime audio, but that's mostly true for synthesis. You should be fine with MIDI. Depending on the resolution/amount of content the video part might get a bit slow.

Which is why I recommend either going straight for a c++ creative toolkit or Max. If you're not sure about video content, need many tweaks/iterations and fast changes, maybe you should go straight for Max which is easy to prototype in/pretty fast and you can pretty much 'live code' adapt you patcher while the code is executing and update/adapt/improvize based on your needs.