Displaying Kinect video stream with MonoGame

I’ve been playing around with MonoGame and the Kinect sensor lately, and gotten some pretty cool results. I thought I’d some some tips & tricks about how to get a game developed in MonoGame to use the Kinect sensor. Today we are just going to grab the color stream from the Kinect, and display it to the screen as a 2d image.

This blog post assumes that you already have the Kinect sensor & API installed (I’m using the old Kinect 1.8 API, I don’t have the new sensor yet) and you already understand the basics of kinect developement.  The source code I’m going to share will use things like Nuget and git submodules, so you should have a basic understanding of those concepts.

If you are just beginning Kinect development, I’d highly recommend Kinect for Windows SDK Programming Guide, it’s currently on sale for only $5 on the packt site. This book covers the entire Kinect API and is really well written and easy to understand.

First, you are going to need to define the variables to hold the Kinect data when it comes in, and the texture we are going to render to.

/// <summary>
/// the texture to write to
/// </summary>
Texture2D pixels;

/// <summary>
/// temp buffer to hold convert kinect data to color objects
/// </summary>
Color[] pixelData_clear;

/// <summary>
/// The horizontal size of the texture we want to display
/// </summary>
private const int ScreenX = 1024;

/// <summary>
/// the vertical size of the texture we want to display
/// </summary>
private const int ScreenY = 768;

Now that those variables are declared, we need to initialize a few of them in the LoadContent method of our game.


/// <summary>
/// LoadContent will be called once per game and is the place to load
/// all of your content.
/// </summary>
protected override void LoadContent()
{
//Create the texture that will be displayed on screen
pixels = new Texture2D(graphics.GraphicsDevice, ScreenX, ScreenY, false, SurfaceFormat.Color);

//Create the temp buffer that will be used to store Kinect data
pixelData_clear = new Color[ScreenX * ScreenY];

//Iniitalize the temp data to black (this is probably unecessary, but if no kinect sensor it's better to be sure)
for (int i = 0; i < pixelData_clear.Length; ++i)
{
pixelData_clear[i] = Color.Black;
}

//Initialize the kinect sensor, etc...
}

Now that everything is setup, we are ready to parse Kinect data. This is the method called everytime the kinect has finished created a frame of data from the color stream:


/// <summary>
/// Event handler for Kinect sensor's ColorFrameReady event
/// </summary>
/// <param name="sender">object sending the event</param>
/// <param name="e">event arguments</param>
private void SensorColorFrameReady(object sender, ColorImageFrameReadyEventArgs e)
{
using (ColorImageFrame colorFrame = e.OpenColorImageFrame())
{
if (colorFrame != null)
{
// Copy the pixel data from the image to a temporary array
colorFrame.CopyPixelDataTo(this.colorPixels);

//get the width of the image
int imageWidth = colorFrame.Width;

//get the height of the image
int imageHeight = colorFrame.Height;

// Convert the depth to RGB
for (int pixelIndex = 0; pixelIndex < pixelData_clear.Length; pixelIndex++)
{
//get the pixel column
int x = pixelIndex % ScreenX;

//get the pixel row
int y = pixelIndex / ScreenX;

//convert the image x to cell x
int x2 = (x * imageWidth) / ScreenX;

//convert the image y to cell y
int y2 = (y * imageHeight) / ScreenY;

//get the index of the cell
int cellIndex = ((y2 * imageWidth) + x2) * 4;

//Create a new color
pixelData_clear[pixelIndex] = new Color(colorPixels[cellIndex + 2], colorPixels[cellIndex + 1], colorPixels[cellIndex + 0]);
}
}
}
}

Now the last thing to do is to render that texture to the screen during our Draw method that is called every frame:

/// <summary>
/// This is called when the game should draw itself.
/// </summary>
/// <param name="gameTime">Provides a snapshot of timing values.</param>
protected override void Draw(GameTime gameTime)
{
GraphicsDevice.Clear(Color.CornflowerBlue);

//copy the temp buffer to the 2d texture
pixels.SetData<Color>(pixelData_clear);

//calculate proper viewport according to aspect ratio
Resolution.ResetViewport();

//setup the spritebatch object for rendering
spriteBatch.Begin(SpriteSortMode.Immediate,
BlendState.AlphaBlend,
null, null, null, null,
Resolution.TransformationMatrix());

//Render the texture to the screen
spriteBatch.Draw(pixels, new Vector2(0, 0), null, Color.White);

spriteBatch.End();

base.Draw(gameTime);
}

That’s it! To summarize:

  • we setup the MonoGame variables to display the kinect stream
  • initialized all those variables
  • consumed Kinect data and stored it in a temp buffer
  • rendered that temp buffer to a texture and displayed on screen

If you’d like to see the full source code in a working example, it can be viewed on my GitHub at ColorBasicsFullScreen_KinectMonogame

Assuming your github is setup, you can pull that code and run it using the following git commands:

git clone git@github.com:dmanning23/ColorBasicsFullScreen_KinectMonogame.git;
cd ColorBasicsFullScreen_KinectMonogame;
git submodule init;
git submodule update;

Cheers!

RoboJets update

Been slowly but surely making some progress on RoboJets…

Done a lot of work on firing missiles and counter-measures. It’s pretty fun now! Just need to get some bugs worked out in their flocking algorithms…

Got rid of that placeholder background and added a cool scrolling starfield effect. It looks awesome!
Check out the

Most importantly, got the game flow up and running. It’s not just a tech demo anymore, it’s an actual game that you can win or lose 🙂

Some new screenshots:

RoboJets 2014-06-28 17-03-35-38

RoboJets 2014-06-28 17-03-10-84

RoboJets 2014-06-28 17-02-03-17

Cheers!

Opposites available on Ouya!

Opposites is available up on the Ouya storefront!  Go download it and try it out, if you like the full version is only three bucks.  This is a really good version, the controls are really tight and the scoring and game play has been cleaned up a lot.  Thanks yall!

I’m an old school XNA programmer, so I did it in MonoGame, which is an open source version of XNA.  It was pretty painless, but there were a few quirks to watch out for.  I’ve got a bunch of MonoGame wrappers up on my GitHub account: github.com/dmanning23

The whole Ouya community has been great too.  I even talked to some of the Ouya folks, who helped me out with a problem.  Coming from Xbox Live Indie Games and Microsoft’s super “hands off” approach to that whole scene, actually talking to someone from Ouya was really awesome.  The players & other devs have been really great too, so far it’s been a really positive experience.  The numbers aren’t great, but a lot of that is my fault.  I did like a “stealth launch” and just threw Opposites up there without sending out press releases or anything, so it’s been in the tail end of the Sandbox not really getting a lot of downloads.  That’ll probably pick up once it gets out in the real storefront though.  People that have downloaded it though, conversation rate has been right around 5% too, I couldn’t be happier.  For a three dollar downloadable title, that’s amazingly high.

Anyway, I’m just glad that it’s out there, and people are having fun with it.  If you haven’t tried it yet, you totally should.  Cheers!

BulletML for C# and Monogame

This is a thing I’ve been workin on for awhile… Kenta Cho from ABA Games made a scripting language for bullet hell shooters a few years ago called BulletML. It’s pretty cool stuff. You can just write up an XML doc with the BulletML grammar and load it up in a parser and start shooting bullets all over the place. Even better, there are hundreds of examples out on teh internet that you can use.

There hasn’t been a very good C# port of BulletML… I found a pretty crummy one up on Github, tore it apart and put it back together. It’s quite a bit cleaner, more comments, better error checking, uses more of the features of C# and less of a straight port from the original C implementation ect etc

The sample program is up on github: BulletMLQuickStart

If you just want the BulletML parser it can also be found on github: BulletML

note that all the solution & project files are for Monogame & Xamarin Studio… eventually there will be Linux and Android ports. Way in the future might be Mac & IOS ports, but I don’t have access to a Mac for that 😛

Porting to Visual Studio and XNA will be left as an exercise for the reader 😉

It still needs a bit of work… the homing missile bits don’t seem to work quite right in all the examples, and it shoots more bullets than it should. It could also benefit from some better exception handling & error reporting.

Cheers!

FontBuddySample

Hey I added a new project to github for testing the FontBuddy:

FontBuddySample on GitHub

It just does some examples of each of the different font effects. Each is drawn with left, center, and right justification so you can see how they all behave:

FontBuddySample Screenshot

Note that since this is a different project, it uses the FontBuddy as an external project, located in ./externals/FontBuddy. You might have to pull this down separately or something, idunno I’m a bit new to using externals in git.

Cheers!

very handy GameTimer library

I put all the game timer code I use up on GitHub.  It’s super useful for doing stuff like speeding up gameplay, slowing down gameplay, hit pause, etc.

I’ve found that Timers in games are very hierarchical.  You’ve got the main clock, which is basically run off the cpu clock or whatever and never stops.  You’ve got a game clock which can be paused, or if the game is kinda sluggish or frantic you can tweak the speed this clock runs at.  Then you’ve got all the clocks that run off that, including character clocks, particle engine, physics clocks, etc.  If these are run hierarchically, when you pause the main game clock all the ones underneath it pause too.

Also when playing a network game, can send those clocks from the server to all clients, and all the players will instantly be synched up. Very neat trick!

Anyway, this is the way I do it. From readme:

A bunch of timer objects that can be used for time effects like hiererchical time, slowdown, speedup, hitpause, etc.
The most important thing is that these timers can be used hierarchically. You can pass one gameclock into the update function of another one, which updates based on that dude's time. So the timer chain might look something like this:
MainTime
|
GameTime
/ | \
Character1Time ParticleEngineTime Character2Time
So like the MainTime is a GameTimer.GameClock and is updated first from the XNA GameTime object. This one always runs at realtime, so you don't want to pause or slow it down. Can problably run HUD or menu transitions off this clock.
The GameTime is also a GameTimer.GameClock and would be used to run the gameplay itself. This one can be paused for the Pause menu, or slowed down/sped up if the game play speed needs to be changed. Since all the clocks below this one are updated off it, they will also pause/slowdown/speedup.
Under that, you could have the ParticleEngine running off it's own clock separate form the player characters. That way when the players are punching the snot out of each other, you can add some hit pause to add crunch and the particles will still be flying around.
Idunno, that's just example how I use it in my fighting games. I also send these timers over the network to sync up the server & clients.
Cheers!