Displaying Kinect video stream with MonoGame

I’ve been playing around with MonoGame and the Kinect sensor lately, and gotten some pretty cool results. I thought I’d some some tips & tricks about how to get a game developed in MonoGame to use the Kinect sensor. Today we are just going to grab the color stream from the Kinect, and display it to the screen as a 2d image.

This blog post assumes that you already have the Kinect sensor & API installed (I’m using the old Kinect 1.8 API, I don’t have the new sensor yet) and you already understand the basics of kinect developement.  The source code I’m going to share will use things like Nuget and git submodules, so you should have a basic understanding of those concepts.

If you are just beginning Kinect development, I’d highly recommend Kinect for Windows SDK Programming Guide, it’s currently on sale for only $5 on the packt site. This book covers the entire Kinect API and is really well written and easy to understand.

First, you are going to need to define the variables to hold the Kinect data when it comes in, and the texture we are going to render to.

/// <summary>
/// the texture to write to
/// </summary>
Texture2D pixels;

/// <summary>
/// temp buffer to hold convert kinect data to color objects
/// </summary>
Color[] pixelData_clear;

/// <summary>
/// The horizontal size of the texture we want to display
/// </summary>
private const int ScreenX = 1024;

/// <summary>
/// the vertical size of the texture we want to display
/// </summary>
private const int ScreenY = 768;

Now that those variables are declared, we need to initialize a few of them in the LoadContent method of our game.


/// <summary>
/// LoadContent will be called once per game and is the place to load
/// all of your content.
/// </summary>
protected override void LoadContent()
{
//Create the texture that will be displayed on screen
pixels = new Texture2D(graphics.GraphicsDevice, ScreenX, ScreenY, false, SurfaceFormat.Color);

//Create the temp buffer that will be used to store Kinect data
pixelData_clear = new Color[ScreenX * ScreenY];

//Iniitalize the temp data to black (this is probably unecessary, but if no kinect sensor it's better to be sure)
for (int i = 0; i < pixelData_clear.Length; ++i)
{
pixelData_clear[i] = Color.Black;
}

//Initialize the kinect sensor, etc...
}

Now that everything is setup, we are ready to parse Kinect data. This is the method called everytime the kinect has finished created a frame of data from the color stream:


/// <summary>
/// Event handler for Kinect sensor's ColorFrameReady event
/// </summary>
/// <param name="sender">object sending the event</param>
/// <param name="e">event arguments</param>
private void SensorColorFrameReady(object sender, ColorImageFrameReadyEventArgs e)
{
using (ColorImageFrame colorFrame = e.OpenColorImageFrame())
{
if (colorFrame != null)
{
// Copy the pixel data from the image to a temporary array
colorFrame.CopyPixelDataTo(this.colorPixels);

//get the width of the image
int imageWidth = colorFrame.Width;

//get the height of the image
int imageHeight = colorFrame.Height;

// Convert the depth to RGB
for (int pixelIndex = 0; pixelIndex < pixelData_clear.Length; pixelIndex++)
{
//get the pixel column
int x = pixelIndex % ScreenX;

//get the pixel row
int y = pixelIndex / ScreenX;

//convert the image x to cell x
int x2 = (x * imageWidth) / ScreenX;

//convert the image y to cell y
int y2 = (y * imageHeight) / ScreenY;

//get the index of the cell
int cellIndex = ((y2 * imageWidth) + x2) * 4;

//Create a new color
pixelData_clear[pixelIndex] = new Color(colorPixels[cellIndex + 2], colorPixels[cellIndex + 1], colorPixels[cellIndex + 0]);
}
}
}
}

Now the last thing to do is to render that texture to the screen during our Draw method that is called every frame:

/// <summary>
/// This is called when the game should draw itself.
/// </summary>
/// <param name="gameTime">Provides a snapshot of timing values.</param>
protected override void Draw(GameTime gameTime)
{
GraphicsDevice.Clear(Color.CornflowerBlue);

//copy the temp buffer to the 2d texture
pixels.SetData<Color>(pixelData_clear);

//calculate proper viewport according to aspect ratio
Resolution.ResetViewport();

//setup the spritebatch object for rendering
spriteBatch.Begin(SpriteSortMode.Immediate,
BlendState.AlphaBlend,
null, null, null, null,
Resolution.TransformationMatrix());

//Render the texture to the screen
spriteBatch.Draw(pixels, new Vector2(0, 0), null, Color.White);

spriteBatch.End();

base.Draw(gameTime);
}

That’s it! To summarize:

  • we setup the MonoGame variables to display the kinect stream
  • initialized all those variables
  • consumed Kinect data and stored it in a temp buffer
  • rendered that temp buffer to a texture and displayed on screen

If you’d like to see the full source code in a working example, it can be viewed on my GitHub at ColorBasicsFullScreen_KinectMonogame

Assuming your github is setup, you can pull that code and run it using the following git commands:

git clone git@github.com:dmanning23/ColorBasicsFullScreen_KinectMonogame.git;
cd ColorBasicsFullScreen_KinectMonogame;
git submodule init;
git submodule update;

Cheers!