View on GitHub

Guanxiong Chen

just a CPEN dude

Jack in a Box

Summary

This is a one-month long project I completed in a team of five Computer Engineering students. We built a kiosk that allows users to play blackjack against it. Major components of the system include a DE1-SoC board that runs the game’s mechanics, a Raspberry Pi board that recognizes cards’ face values, a touchscreen that allows user inputs and visual outputs, and a optical-mechanical system to recognize card values and to deal cards.

Here is what the entire kiosk look like:

Major technologies involved in this project are embedded software and hardware and machine learning (ML).



System Architecture

Hardware Architecture

The diagram below illustrates the hardware components in our kiosk, and the jobs that each delegates:

The image is taken from our group report

Software Architecture

And here is the diagram that shows the logic of our software components, and how they communicate:

The image is taken from our group report

My Roles in the Project

I completed the following tasks with my teammates:

Highlights

Implementing Mainframe - the Game Mechanics

Together with Jaskirat Singh, I implemented the blackjack’s mechanics as a bare-metal C program running on the ARM Cortex-A9 processor on my DE1 board. We call the program Mainframe. The program can respond to touchscreen inputs, issue commands to deal cards, and compute scores for the (human) player and the (kiosk) dealer. Aside from game mechanics, we also implemented a GUI in the C program.

Below are some pictures of an operating instance of Mainframe:

The welcome screen

Dealing a card to the dealer

The screen that shows game result

Implementing Card Value Recognition

Requirements

One of the requirements for our kiosk is that, it should be able to deal cards automatically, and recognize the face values of the cards it deals, in order to compute scores for the player or the kiosk.

High-Level Design

To this end, I implemented a image processing pipeline in Python on a Raspberry Pi board, with the help from my friend Bole Ma (Prior to the project I did not know jack about CV/CG, so he baby-sitted me throughout this part). The major steps within the pipeline are:

  1. Take a picture of a card
  2. Crop the picture and leave the region with face value only
  3. Add padding and resize the left region
  4. Extract features from the region
  5. Predict face value using a KNN classifer model
Hardware Design - Step 1

Here is the hardware setup of the pipeline. We used a Pi camera firmly mounted on the card holder to take pictures:

Software Design - Step 2 to 5

Step 2 to 5 are the “ML steps” within the entire pipeline. The diagram below more specifically explains our ML pipeline for face value recognition:

Also we trained the model by ourselves, as opposed to grabbing it from somewhere on the internet. First of all, We built our training set consisting of 9,830 images by manually taking pictures of each face value and conducting data augmentation. Then I trained with two classification algorithms - SVM and KNN. Once training is done, I tested each model under various lightning conditions. Eventually I found that the two models yielded the same performances (100% accuracy with 25 test images), so I just selected KNN as my final choice.

Hurdles

CV - Traditional or ML-based?

A major challenge we had was choosing between traditional image recognition methods such as template matching and ML-based methods for recognizing card values. And in fact I tested a template matching algorithm and it somehow worked. Eventually we did not use it for its terribly low speed - while our KNN-based algorithm can recognize a digit in less than a second, the template matching algorithm takes roughly 30 seconds! That is intolerable if we want our players to have any fun with our kiosk.

Image Quality

Another challege was ensuring card images we got from our camera are clear. We noticed that if our camera is not well-focused, or some shadows projected on the card, our ML pipeline would perform horribly bad. So we had to build a case to cover our camera: