Panopticon captures and examines video data

What did the team learn from recently capturing and examining video data of people in the lab playing Hotline Miami and Tekken 2?

We noted that Players exhibited a number of different behaviours based on what was happening in the game, including leaning forward when concentrating, and leaning back when tension was released. They tended to smile frequently – particularly when key game events occurred.

When playing the two-player fighting game Tekken 2, they displayed more facial expressions, and also interesting so when the single player recognised that someone was standing behind them!

These findings have led the team to consider what can be learnt about the experience of a game from a sequence of facial expressions pulled.

The Panopticon project has deployed a data capture set up at the National Videogame Arcade (NVA), aiming to capture sufficient video data of people playing, so that eMax can be trained to better recognise player behaviours and expressions automatically.

Photo by Element5 Digital from Pexels

If you’re visiting the NVA this week and want to participate in the Panopticon research project, or want to learn more, look out for us in the foyer!

 

Panopticon

The Panopticon project is developing a system for measuring how visitors to the National Videogame Arcade in Nottingham engage with the various wild and wonderful games that they exhibit in that space. We’ve taken the name from the infamous prison design where occupants were watched at all times by a single guard, however the modern twist is to involve the visitors to the NVA in a meaningful conversation and decision about what’s done with the data they generate.

We’ve spent the first couple of months thinking about and beginning to develop our engagement tracking system. This is a computer vision based system that, using cameras mounted on various games in the NVA, will watch and measure how engaged players are. That’s more complicated than it sounds. Firstly, the NVA has lots of different kinds of game and exhibits that players and visitors can explore. We did an initial site visit with Exhibition Manager Alex, and came to the conclusion that we want to experiment with tracking engagement with three different kinds of exhibit:

  • Arcade games with one or two people stood in front of a display playing the game
  • Room scale games of which the NVA has a few that involve players physically moving and leaping around
  • Traditional exhibits kept in glass cases that visitors can peruse

The variety here leads to some challenges for how we can measure engagement. The technology we’re developing is built into a small form-factor PC and camera that can be attached to the different exhibits. The tech will watch and interpret the body poses of the people using the exhibit and how they’re standing or moving, but also what kind of facial expressions they’re pulling, for example are they laughing, focused, or excited.

To be able to make sense of what the camera sees we first need to train the technology, or rather for the technology to learn. We’ve built a temporary gaming booth fitted out with cameras in the Mixed Reality Lab, and the next step will be to invite people to visit and spend half an hour playing on an Xbox to capture the initial data that we need. Using this data we can build a model of  measurable engagement that we can use to subsequently measure the engagement of visitors to the NVA.

But this is only half of the picture. The other work we’re doing is to figure out how to have a conversation with the player about the data that we’re capturing rather than just wholesale capturing everything, which most people would quite rightly think was overly intrusive. Each player is given a token that they can use to explicitly signal that they are engaging with the game, and which also acts as an access token to the data that is being captured. This token, and whether the player chooses to give it to someone else, or even gift it to the NVA, drives the conversation about data ownership. We’re exploring some lightweight NFC tags that can be easily integrated into a variety of form factors.

 

We’re having some interesting conversations with the NVA about what the servoken should look like, and prototyping possible ideas using additive manufacturing. It could look like a coin, inspired by the old coin operated arcade games, but then we don’t actually want players to lose their tokens inside a machine. Or it could look like Portal’s Companion Cube, where the material emancipation grill erases unwanted data about you when you leave the exhibition.

By Martin Flintham