Performance Video//Designing the User Interface

Performance Video//Designing the User Interface

Unit Handbook


Title:  Designing the User Interface (Interactive Media) and Performance Video (Digital Media Production) 

Outline Context/Brief 

The brief is to make a musical instrument and perform with it.  Creating a piece of dynamic improvised performance on the subject of ‘belonging’.

(what does ‘belonging’ mean)

For those specialising in Designing the User Interface (Interactive Media) you will concentrate on the making of the instrument as a user interface, on usability, the nature of tactile and haptic feedback, the semiotics of the interface etc. 

For those specialising in Performance Video (Digital Media Production) you will focus creating emotive expression through effecting video and sound in a performance and on real time effects programming effects, video mapping, etc.

The brief poses a number of questions that we would like you to discuss, comment on and perhaps even pose solutions to: In this society what constitutes a performance, how close does a performer have to be to the audience, is something streamed over the web live, is something that is generative a performance etc, what is an interface, what happens when work becomes play etc?

We will look at range of subject including general usability heuristics (rules of thumb), semiotics, GUI’s, notations and language.

Approaches might be as diverse as physically making instruments or using the web as audience and performers,


Real-time video and audio processing (output video FX) using Ableton and  MAXMSP feedback from sensors from Phidgets or Arduino and MAXMSP or Wire or Processing

Synopsis of study

Learning Agreement


Name: Mathew Brewer


Course: Digital Media Production            Level: 5

Unit:  Designing the User Interface (Interactive Media Pathway)

Reference number:  IMD510

Credit points: 22.5

Study hours: 225 hours

Tutor(s): Liam Birtles, Phil Beards, Amanda Stephenson and Bob Cotton

Synopsis of study

For the Designing the User Interface Unit I will be working on producing a “musical instrument”, which can be played by anyone (as in someone with no musical ability). The main theme which is to be investigated, as part of this project, is that of the “Remembrance of Childhood” specifically my own.

One overriding theme from my child hood memories is that of playing video games, mainly the Super Mario Bros. Games on the Super Nintendo Entertainment System, this in combination with the idea that the best children’s toys require little prior knowledge of how they function have lead to the desire to attempt to produce a prototype of an instrument which can literally played through child like play.

 In order to achieve the production of a technical prototype of this style of play based instrumentation I will research methods of which I will be able to produce a Physical user interface (PUI) that literally anyone will be able to pick up and play.

 As I will be “specialising in Designing the User Interface (Interactive Media) I will concentrate on the making of the instrument as a user interface, on usability, the nature of tactile and haptic feedback, the semiotics of the interface etc.” 

 In all, my main attempt will be to discover, through experimentation, the best and worst methods of producing a fun an entertaining prototype for a musical instrument that represents my most cherished childhood memories whilst allowing for “fun” and “pick up and play” ideals to be at the heart of the final product.

 I intend to at the least research into the use of games controllers, Phidget’s RFID tags, OSCulator, Ableton Live and maxMSP in order to create the best method of control in the production of this instrumental prototype.

A1 to develop an understanding of the user interface within contemporary design
A2 to review a range of user interface designs
A3 to develop skills that demonstrate an ability to design and produce an
interactive user interface
A4 to review the use of visual and audio feedback
A5 to investigate alternative user interface methods
Learning outcomes
On completion of this unit you will be able to:
LO1 Identify, use and analyse an interactive system
LO2 Investigate and use audio feedback
LO3 Investigate and use visual cues
LO4 Design and produce an interactive interface

Initital Notes: 

Notes page 1notes page 2notes page 3Key things from the above hand written pages which could be of use:

Make a Musical Instrument + Perfrom with it

Subject of Belonging

Growing Up

Remembering Childhood

Cultural References

What Is  a Perfromance?

Minimalist//Simplifying the approach

PechaKucha 20 slides in 20 seconds (method of quick fire presentation)

Memories from my childhood:

mario power ups

Mario Sprites (Property of Nintendo) 

Youtube video of Level 1-1 Super Mario Bros.

After Speaking with Felix (my partner in crime so to speak) we came to the conclusion that we both had strong memories of playing video games from the Super Nintendo and Sega Mega Drive era when we were younger.

From this discussion we concluded that many people have had similar experiences/attachments to games/mainstream pop culture characters and therefore this would be the basis of our project.

Guitar Hero Controller: 

As part of our idea for the project to represent our combined fondness for our childhood memories of gaming, specifically the Super Mario Bros. game we intitally decided to use a games controller, and having read about an application called OSCulator for mac which can be used in conjunction with Abeton Live amongst other things so that a whole range of games controllers and devices can be used as midi controls.


Tesla Coil Mario Theme (youtube) 


Guitar Hero Midi Controller (youtube) 

After seeing Channel 4 dot com’s Twist out Words campaign

Considered producing a similar application which built up video of people playing instrumental parts of the mario theme which the user could build up by clicking on links to the video and play it back at there own leisure. for an idea of how this works view the link below:

Channel 4 Twist our words 

And now for some idea’s of instruments to be played/filmed for the Mario Orchestra: 

Possible Instrument sounds to use in

Mario Orchestra

Instruments for idea:
Mario theme
Multiple Instruments
Video Clips
(trigger via guitar hero controller/iphone)
Pepople wear mario/luigi costume
3.steel drums
9.8-bit noise
10.pots and pans?

Split into groups of Vocal, Wind instrument, Strings and Percussion. (performers wear colour coded t shirts for video)

Idea Development (again) 

After consideration Of multiple avenues we came to the conclusion that the main reocurring thread in all of our idea generation was that of an Orchestra style build up of loops.

Also another important factor was the “Rememberance of Childhood” and a theme closely assosciated with childhood is the ability to play, for this reason we feel that that production of an instrument needed to have a very childlike pick up and play approach on the end user level.

maxMSP(vimeo explanations by me) 

look mah no buttons.//RFID//explained (vimeo)

Fade in Fade Out//explained (vimeo) 

One RFID fade in Fade Out//explained (vimeo) 

Unable to show Prototype at its absolute fully working best:


That’s right, as it stands due purely to a technical hiccup in the form of a lack of the correct number of RFID tags and RFID readers, the Prototype won’t be fully functional in time for the crit, but I will be able to show a semi working version and evidence why during the presentation. (this information is correct at time of writing: 17.35 GMT Sun 9th Jan 2011) more explanation in the video link below.

One RFID Reader Two cups//why it wont work with multiple tags on one reader//explained (vimeo) 

Need To Use .aif files:

I discovered by reading the maxMSP help files (of which there are copious ammounts to wade through) that maxMSP’s audio format of choice is the .aif extension. So once I found a midi format of the Super Mario Bros. theme on the internet I imported it to Garageband and set about spliting the midi into four seperate tracks/instruments and saving each instrument as a seperate .aif file.

Mario Midi 

Convert to .aif files using Garageband 

Mario .aif clean electric guitar 

Mario .aif elctric tremelo guitar 

Mario .aif piano 

Mario .aif Rock Drumkit 


 Initial maxMSP Patch (right click//view full image) 

final Patch Screen shot 

Final Patch: (right click//view full image) 

Final Idea (clarification)

For this project what myself and Felix have tried to create, is something which both reflects our childhood memories of playing Mario games, and also with some more time to get beyond the Prototype stage, would allow us to create something that even a child could be introduced to playing with sound. We felt that in order to achieve this, we would create a play set/stage of the most iconic (to us anyway) level of the original Super Mario Bros. and trigger music from the game as you passed the figure (which in this case should be Mario) through the stage. I think that it fits with the brief because we have questioned who is the performer and who is the audience, and have come to the conclusion that the performer can be there own audience, whilst at the same time hopefully attracting others over to see what is going on, and then in turn they could get involved with playing too.

We also decided quite early on that it was really important for this kind of musical play, to be embodied in the real world, with a Physical User Interface as this is something that can encourage all age groups to at least have a go.


Evaluation:(before the crit) At this final stage of the project I feel that I am unsure of how strong our final outcome is in relation to the theme/idea which we envisioned for the over all piece. It was our intention to create a “musical instrument” for a “performance” which conveyed an essence of our childhood memories. To do so we have built a recreation of the first level of what we consider to be collectively one of the key (gaming) moments of our formative years; Super Mario Bros from the Super Nintendo era. The Intent being that Someone/ a user with no musical knowledge what so ever will be able to pick up and play with the physical user interface we have designed and through play be able to build up the original Super Mario Bros. theme by placing objects (with attached RFID tags) onto the stage. and remove parts of the track simply by removing the objects. I feel that RFID tags/Readers may have not been the perfect medium for this kind of project and given more time feel that this kind of project could be realised far more effectively by using other more suitable technology. However as a learning process I have gained knowledge of maxMSP, and have become aware of other applications such as Ableton Live and OSCulator, each use a mixture of MIDI and can be used in conjunction with maxMSP for more extensive creative endeavours, which could possibly be beneficial in further projects.

Post Presentation/Crit thoughts:

After the Presentation/crit, it was clear to me that even though I had put in a huge amount of work simply trying to get RFID tags to do the Job required of the project, they were almost certainly not the best way of solving the issue. (as they caused more problems than they solved). 

The two main issues with RFID specifically, are that the Readers only accept on real input of any use at a time, and although they do provide maxMSP with a second bang on removal from the RFID reader device, it’s less useful because the first bang contains data which in this case is the ID number of the RFID tag, where as on removal no data is given, the RFID reader simply recognises that there is no longer a tag within its proximity. (As you will see in our final prototype video, this can cause a jittery effect when trying to trigger audio).

I personally feel that the final prototype is still a work in progress, but that the idea behind it is strong enough to warrant further development, although it could benefit from being executed slightly differently. Such as approaching it with different software, and Hardware that is slightly more reliable/up to the task. What I would like to do next, is create a range of much larger Blocks and Shells and other things from the Mario universe which trigger there own individual sounds when placed onto a surface. Similar to the Reactable, but still utilising the theme of belonging and remembering childhood gaming experiences.

Final Prototype: