An Android app that uses Augmented Reality for the study of chess.
Veient-hi més enllà: la realitat augmentada aplicada a la pràctica dels escacs (Catalan for Seeing beyond: augmented reality applied to the practice of chess) is the title for my Bacceaulerate final project (Treball de Recerca), developed during my last year (September to June).
Using the Vuforia Augmented Reality (AR) library for the Unity engine, I made an Android application with several AR-based features related to chess. The goal was in showcasing the many possibilities that this technology offered rather than delivering a full fleshed-out product ready for commercial use.
All the features implemented are marker-based and thus require some sort of physical object for the camera to track and put digital information on top. In the previous image, the chess leaflet lying on the table is the marker and the app puts a digital, playable chess board on top.
The camera can the be moved (including rotation, zooming, etc.) at will to see the chess board from any perspective. Alternatively, the marker can be moved and all the digital information will be moved accordingly.
The app is available for Android 4.1 or later (any mobile device from around 2013 or later should work).
It offers several features, including Aprèn (Learn), with basic information about the game rules; Practica (Practice), with exercises to improve your level; and Juga (Play) with a playable chess board.
The first Learn feature is a description of each piece, that shows basic information about its name, movement, relative value with respect to the others and some tips about how to play with it.
The app tracks the user position so all the text panels are always facing the camera, allowing them to be read from any angle.
In order to test the knowledge learned from the previous feature, I implemented a mini-game where the user is asked to tap on the squares where the piece can move to.
In this case, a knight (shown as a 3D model of a horse, as a joke :) is placed on a random square; if the user taps on a square that the knight can move to, it turns into grass, otherwise a flame appears on that square. Note that all this is animated when seen through the camera.
The two main Practice features are:
A map of Barcelona (limited only to my district) showing the location of the nearest chess clubs, along with some basic information about them and the distance to the user’s location.
I also 3D-modeled some of the most iconic buildings in the area to serve as reference to the user (I tried not to do too many and keep everything low-poly because I wanted it to be able to run on low-end devices, like the old phone that I was using for testing).
An overlay for chess books showing post-it notes with tips and user-reviews and a star rating on the bottom. Again, notice that this time it is the book cover itself that acts as a marker!
Plus a coffee cup, ‘cause why not! At some point during development I also implemented a paper airplane that was flying around the room, but I ended up removing it because it was too distracting.
Finally, the main feature of the app is the ability to play a game on a virtual chess board. I designed a custom low-poly piece set to allow it to run smoothly on any device.
The user can move pieces by tapping on the starting and destination squares. To help users that are not very familiar with the chess rules, an overlay indicating the possible moves is shown once a piece is tapped on.
Note that, while basic chess rules are easy to define (e.g. one move per turn, the movement of each piece, captures, etc.) there are also a lot of intricacies and special cases: promotion, castling, en passant, draw for reptition, etc. This would require a lot of programming for the little visibility it would have in the final product. Since the focus of this project was in showcasing the possibilities of AR rather than making a fully usable chess app, I simply ommited these rules and stayed with the most basic set.
Finally, I feel like I should mention a few features that I considered or tried to implement but didn’t make it into the final program:
Piece recognition: the current system that shows piece information still uses a printed marker and shows a 3D model of the piece. I thought that it would be much cooler and useful in a real life scenario if the user could just point the camera to a physical piece and have the information displayed around it (essentially using the piece as a marker).
But it turned out to be very difficult to track and recognize physical pieces, much more than flat paper markers; my intuition is that it’s due to pieces having very little detail, both in shape and texture. I tried painting some tracking dots on a plastic piece, but still wasn’t enough.
Object recognition: other than pieces, I considered using other physical objects as markers, such as a chess clock that would have instructions displayed around it. But it still offered a large array of challenges that made it very difficult to implement; most of those were related to lightning and required very strict conditions that made it impractical to use in real life.
Online multiplayer: just, no. I did consider it at the beginning, since it would be really cool to have an online multiplayer system, where two people could play from different devices. But it was just too complicated and out of my area of knowledge to be feasible to implement in this project. Maybe in a future one…
Voice recognition: since my clumsy hands sometimes struggled to tap on the correct square when moving pieces on the chess board, especially when playing from far away so it was very small on the screen, I considered adding a voice recognition feature where the user could say aloud the moves instead of tapping on the phone screen.
I managed to get it working to an extent, but I encountered two important difficulties:
- Since board squares are described using coordinates, and the difference in pronunciation between some letters is very small (such as d and e, and even more in Catalan!), the system often misunderstood the squares. Alternatives could be to use old chess notation or some kind of military alphabet, but it started to get impractical.
- The system I managed to get working on my laptop relied on Windows libraries for voice recognition that were not available in Android devices, which used their own Google ones. I could not really get it working to the same level of accuracy as I had on my laptop, so I eventually gave it up.
If you want to give this project a try, all the files are hosted in Google Drive. It includes the written thesis, app installer with instructions (in Catalan) and markers to be used.