Computer aided board-gaming

Summary

I've played a lot of games in my time. So when I first saw augmented reality my brain started to wrap itself around how it could be used in a gaming context. Having never worked with any sort of pose estimation techniques before, I decided to (perhaps foolishly) embark on turning my masters project in to a simple augmented reality board-game. One of my inspirations for this project came from the (back then) contemporary Playstation 3 title, Eye of Judgement. Despite it's modest 330,000 unit sales, it was still one of the first commercial games to heavily integrate augmented reality, as a type of game mechanic.

Augmented reality can help properly mesh video and board games. It enables us to retain the hidden information - something one player should know, but another should not - which is crucial when several people play together. Additionally, the computer can handle tedious tasks that the players would prefer to avoid, like tedious math or convoluted rules. The cherry on top is the fact that the computer can also manage non-player elements in a much more interesting manner. For example, the computer could control an additional Ai player for the other players to play against.

Quick Facts

Master Thesis Project, completed over six months in 2008
C++ Game Engine written from Scratch
Ogre3D used as graphics engine
3D Pose Estimation written from Scratch
Quantitative study comparing Pose estimation to OpenCV counter-part. Comparable performance
Informal comparison to commercial counter-part (Eye of Judgement)
Qualitative user-study with 10 participants

Resources

Master Thesis

Aspirations and Goals

This project was 6 month project that I ever undertook. Although most of the projects I worked on during my time at the University of Copenhagen involved a group of 2-3 people, this one ended up as a solo project. After presenting my initial ideas to my supervisor, he seemed cautiously optimistic. Although my education had covered a lot of 2D/3D graphics, as well as general computer vision, I had never specifically worked with pose estimation before.

The few available open-source projects (like the ARToolKit) as well as the supposed bible for pose estimation (Multiple View Geometry) made me hopeful that I would not be biting off more than I could chew.

Breaking Point

Approximately 3 months into the project, my feelings were divided. On the one hand, I felt I had a significant grasp on the 3D engine (Ogre3D) I had decided to use for the project. I also felt the the basic game engine I was building was beginning to take shape and my thesis was even coming along nicely. However, the single biggest unknown in my project was still constantly looming at the back of my mind, unsolved. Specifically, the problem of proper pose estimation.

I was worried. Although the supposed bible for all things "estimatory" in 3D had been on my desk for the past 3 months, and I still felt like I only understood half of the problem I was trying to solve: Estimating a real-world camera position from 4 known, projected points. I'd like to pretend that there are two sides to the story, and that the book probably explained what I needed in terms I simply didn't understand. But I can't. For this particular case, the book remains, in my humble opinion, incredibly vague and downright lacking. Even after I had the good fortune of conversing with Daniel Wagner over a period of several weeks, helping me tune into what exactly wasn't working as expected and helping me with the final pieces of the puzzle, I still could not find this solution in the book. Actually, that's somewhat incorrect. The book does address this case, but fails to mention key details which I - to this day - still do not believe are 'trivial'.

But enough about my personal struggle with this particular book. If you're ever in need of understanding how to solve this particular pose estimation problem, I have gone to great lengths to detail every single step of the process in my master thesis which I encourage you to peruse at your leisure. Having finally gotten all the pieces into place, I could move onto creating a use case scenario.

The Game

Originally, I had envisioned designing a simple tower-defence game using augmented reality, but my initial prototypes (see videos below), made me realize two important facts. First of all, a good tower-defence game is anything but simple. Even one that only has to hold the players attention for about 10-15 minutes would needed to have been properly play-tested and balanced to ensure an interesting progression. Second, the playing field simply wasn't grand enough for my visions. At most, I could reliably fit approximately 10 by 6 tags on the playing area.

With time quickly running through my fingers, I bit the bullet and made the decision to instead implement a very simple train-track game. The new game was overly simplistic compared to my initial aspirations, but still served as a useful case study of the users experience of interacting with augmented reality elements.

Gallery

The complete testing setup. Computer, game pieces, camera! Admittedly somewhat awkward to items right in front of you, yet being forced to view a recording of your own hands.

Tag detection algorithm. 1) (Top Left) The original unmodified image as recorded by the camera. 2) (Top Right) Radial distortion compensation. 3) Greyscale conversion. 4) Thresholding. 5) Edge detection. 6) Square/Tag detection.

Gameplay screenshots. 1) (Top Left) The game begins with all tags present, yet scattered. 2) (Top Right) Three tags help define the play area. 3) The train departure area spawns giving the player a while to orient track tags. 4) The player constructs a simple track path. 5) If the train reaches the end of the track, the game is over.

Various lighting scenarios to determine robustness of tag detection algorithm. For each scenario, three pictures are shown. Camera image (with radial distortion compensation), Edge detected image, and the final tag detection result. 1) (Top image) Regular lighting conditions, with backboard. 2) Evening (heavy shadow) lighting, with backboard. 3) Table lamp lighting, with backboard. 4) Regular lighting conditions, no backboard. 5) Evening (heavy shadow) lighting, no backboard. Table lamp lighting, no backboard.

Various lighting scenarios to determine robustness of tag detection algorithm. For each scenario, three pictures are shown. Camera image (with radial distortion compensation), Edge detected image, and the final tag detection result. 1) (Top image) Regular lighting conditions, black backboard. 2) Regular lighting conditions, black backboard with hand. 3) Dark lighting conditions. 4) Spot-light conditions. 5) Half shadowed lighting conditions. 6) Very poor lighting conditions.

Early Prototypes

These prototype videos show the gradual progress towards the final working prototype, showcased in the video at the very top of this page. As previously noted, originally I intended the game to a simple tower-defence clone. However, lack-of-time and technical constraints forced me to change the game into a simpler 'pipe-dream' clone.

Lasse Laursen