[SYNTHETIC WARBLING]
Previously we have looked at different types
of advanced motion capture systems,
like the optical marker based system,
and also they inertial systems.
Now we're going to look at how we can use regular video
cameras for analysing music related movement.
And when I say regular video camera, that
can be anything from the simplest
type of web camera like the one we
have here, which is a consumer camera
that you can easily attach to your computer.
And it's also the same thing as is typically
built in to your computer.
It can also be somewhat more advanced, industrial type
of cameras that we also use in a lab setting.
These typically have better lighting system, and also
a larger chip, so you get better images
from this type of camera.
Up to more professional type of cameras, where you have more
advanced controls, and better optics,
so that you can change, for example, the lenses
and get good visual results.
Additionally, another very interesting type
of camera that has become much more affordable
now in recent years is the one that we
have in the Xbox Kinect, and other type of cameras,
which is a depth camera.
So that can sense also the depth in image,
not only the plain, two dimensional image
that we typically have.
So these we also use a lot in music research these days.
So now we're going to look at how
it's possible to work with video analysis on the computer.
And here I have the programme Maximus P open,
which is a graphical programming environment that we often use
in music research.
And I have different modules here
for importing video from the web camera
that we have standing here now.
And then I could see myself in the image.
One important thing to remember when
we're working with video analysis
is that a video image is in fact just set of numbers,
and we can look at this here by seeing
these numbers change as I move.
So these are numbers between 0 and 255.
When working with video analysis,
one of the most common ways of starting out
is to look at what we call the motion image.
And the motion image is in fact just the calculation
where we're subtracting the frames following
each other in a video sequence.
What that means is that whenever there is a change between two
frames, that will be visible.
If there's no change it, it will not
be visible, as we see here now.
So I have my original image here,
and now I do the calculation in grayscale,
because that's a little bit easier for the computer to do.
Then as we see here in the motion image,
it's only my motion that is shown and is visible.
Then I can philtre this to make it a little bit clearer.
And I can also add a noise removal philtre,
so that we can see the motion more easily.
And plus, we also-- I could add some edge detection.
Although in this case I have a very noisy background,
so it doesn't really help here.
From the motion image itself.
It's possible to calculate various types of features.
And one of the features that we work a lot with here in our lab
is to look at what we call the quantity of motion.
Which is, in a sense, the sum of all the active pixels
in the motion image.
So we have the motion image up here, and when I move
you can see the white pixels.
If I sit still, there is nothing.
And by adding up all these pixels,
we get a number called the quantity
of motion, which we can see here in this blue plot here.
So if I move a lot, we can see that.
If I sit still, there's not very much happening.
So in many ways, the quantity of motion
is similar to also the activity that we
can get from inertial sensors, such as the accelerometers.
And it's a good way to look at the general movement
in the sequence.
We can also calculate the centroid on motion
to look at where exactly in the image we are.
And also, for example, a bounding box, as we see here.
Thats the square wrapped around the image.
That shows the extent of the motion.
These numbers, we can use for analytical purposes
to plot, for example, with score material,
or with some sonic features.
But since we also do this in real time
it's also possible to use this to create music.
For example, in this case here where
I touch the horizontal motion to a piano sound,
so you can play piano by moving here.
Or, another example here with drums,
where it's possible to play drums
by trying to hit different parts of the image,
so that you can control this in real time.
So in a way, for a lot of the things we're doing here,
we can work between analysis on one side,
and synthesis on the other very easily, because it's really
the same thing going on.
Now if you want to work more specifically
on the analysis of, for example, larger
stretches of musical movement, we
need to have some tools for doing that as well.
And I developed a tool, or a method
I called motiongrams, which is a technique for visualising
movement over time.
And you could say that well, is it
really necessary to visualise movements?
- because movements after all, is visual to start with.
But the thing is that just by looking at,
for example, the video stream, it's
difficult to get the grasp of the entire longer
sequences of movement material.
So for this motiongrams as we see here,
we get a plot over time of how motion evolves over time.
So this is done by simply calculating the average
for each of the lines in the motion image,
and then plotting them over time.
And this can be done either horizontally
or it can be done vertically, as we see here.
So for this plot here, we can see
the movement going up and down.
While for this plot here, we can see the sideways movement.
And together, they represent the musical movement
over longer stretches of time, for example, several minutes,
or even hours.
So if I sit still, there will be nothing.
And if I move, it will be displayed in these images.
Coming back to the idea of combining analysis
and synthesis, it's even possible
to create sound from such motion reps.
As we talked about earlier, we are often
working between analysis and synthesis, where
it's possible to either look at things from a more
scientific perspective, or from an artistic,
or sometimes even in between.
And again, from this motiongrams that we have looked at,
it's interesting that they can also be used to create sound.
Because a motiongram is in many ways
similar to a spectrogram of sound.
So by taking a motiongram and interpreting it
as a spectrogram, we can actually
create sound from the video.
And here's an example of this, where
I'm turning this motiongram into sound through an inverse FFT
technique.
And even more interesting is that we
can add, for example, a video effect,
and it will turn out to be an audio effect,
like this blurring we're doing now in the image here.
We'll end up with some kind of reverb,
with some filtering on top of it in the sound.
So these are just some examples of how it's possible to work
with video material on the computer, both for analyzing,
but also analysing relationships between movement and sound,
but also to create sound from movement.