Making (actual use) of augmented reality
This is the background story of the Elokuutio application project made for the metal group Stam1na. The application uses AR to enable fans to unlock and view bonus material through interaction with the album cover art.
From the developer’s perspective the digital industry can sometimes be quite frustrating. You are surrounded by awesome new technology but you are not able to use it just yet. To make profit out of it, you may have to wait until it reaches the plateau of productivity. In our case with augmented reality, or more friendly AR, the wait took five years.
Augmented reality is the integration of digital information with the users environment in real time.
When we started working for the upcoming Stam1na release – Elokuutio (a theme album about the digital dystopia of tomorrow), it was obvious that the album should have some kind of a digital dimension. After many wild ideas we ended up creating a concept of a “mobile app that does something”. That something should be something more for those who had bought the actual record.
Teemu Leinonen, who was responsible of the visual design came up with the idea of symbols, one for each song in the album. After Mats Holmberg was done 3d modelling and rendering, we had an excellent set of assets to work with. It was very clear at this point, that we had to take everything out of these 3d-models. And, as we needed to bind the app and the album cover for exclusivity… my 5 year wait was suddenly over.
Unity and Vuforia
I’d been working with Unity for couple of projects and I found it a very comfortable for this one as well. If you need to publish on multiple platforms (we did iOS and Android) and have 3D graphics to deal with, Unity is the way to go.
When it comes to AR, you have couple of options when working with Unity. After a two week version-support-mismatch-waiting-for-the-patch-nightmare we still ended up using the Unity add-on called Vuforia. It takes care of the AR tech part and the workflow is solid. I’ve been playing around with several AR platforms once in a while, but I have to say I was amazed with the quality and performance of Vuforia. Of course modern smartphones play their part when it comes to performance.
Five star tracking
After you get it all set up, working with Vuforia is painless. First you upload images you want to track to the Vuforia developer web service. In our case it was about 10 pages of album cover art. After uploading you get an instant “trackability” star rating of the image. After you are done you can download the tracker data in one package which you’ll then import into your Unity project for use as a tracker database.
Back in Unity the Vuforia does it’s magic comparing the camera image against the database. In other words, it is constantly looking for images you added from whatever you see through your camera. It’s also worth noticing, that all this is done several times per second. After a successful recognition, the application keeps track of the image position and rotation in 3d space as the camera (mobile device) moves or rotates. It then uses this information to draw objects on top of the camera image. As we chose to display the same 3d objects on top of their printed instances, we managed to create an illusion of the symbols coming alive. Only by touching those virtual symbols, you’ll get access to the bonus material hidden inside the application.
The end result, where the real life image meets the virtual one, is some kind of modern magic.
TIP: We had a couple of zero star trackers, (of course the CD covers were already printed before testing!) but managed to get them working by boosting the image contrast before uploading to Vuforia target manager.
Slope of Enlightenment
Our goal was to give something exclusive to the record owners: video interviews and a bonus track. The problem we faced was how could an app know that the user had bought the record? Basically, we needed only the image recognition software to recognise the cover art images, and this is what AR is really good at. It turned out we got some wow-effects as a bonus in the process.
Antti Hyyrynen (Stam1na), Juha (JCO) and Tuomo Saikkonen (Sakara Records) demonstrating the Elokuutio app (in Finnish).