Research Proposal – SonicViz

 

“ The medium is the message.”

- Marshal McLuhan

 

“ The message is the message.”

- Walter Bender, M.I.T. Media Lab

 Aims, Expected outcomes and Significance

The aim of this project is to investigate new computer interface metaphors for the

real-time and simultaneous performance of dynamic imagery and sound.

 

In terms of outcomes, the project will provide the groundwork needed to develop a new generation of commercial interactive audiovisual products including, but not limited to:

 

Outputs will be a set of reusable software components that can be utilised in a number of different visual-music products and usage scenarios.

 

The course is being undertaken as a Masters of Music by Project. Due to the compressed course timeframe culminating in the REV festival for showcasing project results (an immovable project milestone in April 2001) a COTS (Commercial Off The Shelf) development strategy has been adopted for selecting enabling technologies such as:

·        Interactive audio and music engine

·        Controllers and Sensors

·        Target computer platforms (Web, Mobile, Fixed location)

·        Software Development Kits

 

The focus of the project is thus on rapid software prototyping to enable the elicitation of the key Human-Computer interaction issues which will contribute to the success of the aforementioned products in the emerging marketplace.


Background and Significance

“The quest for an audiovisual “unity of the senses” is an ancient one, which extends at least as far back as the Classical Greek philosophers who developed tables of correspondence between colors and musical pitches. Although the recorded history of actual audiovisual instruments is only four centuries long, we may surmise that the roots of the idea are as old as music, shadow puppetry and painting themselves. A proliferation of audiovisual expression systems designed over the last hundred years—made possible by the technological affordances precipitated by the scientific, industrial and information revolutions—has dramatically expanded the set of expressive languages available to humankind…The computer is a natural choice for such an exploration, as its fundamental material is pure information, essentially unfettered by the constraints of the physical world.” [Levin, Golan 2000]

 

Moore’s law continues unabated in terms of CPU speed (~doubled every 18 months) and has seen unprecedented power delivered to the desktop, and is still projected to do so for the near term. This trend has also migrated to the mobile and emerging wearable computer market. Coupled with advances in ancillary technologies such as display, real-time graphics, sound/DSP, power, sensors, and communications we can envision a range of innovative consumer products previously restricted to academic music and sound research labs. In terms of Human-Computer interaction, new types of applications need to be developed to exploit these technologies, and low-cost, non-invasive, multimodal interface technologies will form the primary basis of user interaction [Paradiso, Joseph A 1998].

 

Interrelationships among HCI Topics

Figure 1 Human-Computer Interaction [ACM SIGCHI 2001]

 

 

 

Control-interfaces for music require a balance between functionality and adaptability to be successful. This project will not only construct unique specialised 2D/3D visual-music instruments, but will also aid in our understanding of control interfaces and usage scenarios of such instruments.

 

Possible visual-music interfaces will not only have more degrees of freedom, but may be ergonomically integrated with the entire body,  capturing expressive nuances from the mouth and its breath, electromagnetic brain waves and other body signals, eye-tracking, posture, and four-limbed interactions [Levin, Golan 2000].

DesignGoal

Figure 2 Synaestetic musical bonding of sound and Image: Adapted from [Marrin, Teresa 2000 #101]

 

Recent releases of visual-music products using existing game controller devices points to increasing interest in this emerging area: [ModernGroove 2001] [Codemasters 2001] [Artworks.co.uk 2001] [Matsura, Masaya 1999] [SSEYO 2001] [3DMaxMedia 2000] [Creative Technology 2001] [Analog Devices 2001] [Beatnik 2001].

 

Patent analysis also indicates commercial progress in the visual-music area, and is an area that any potential product developer must be aware of so as to avoid inadvertent patent infringement. Yamaha Corporation’s recently awarded patent #6225545 “Musical image display apparatus and method storage medium therefore” [Suziki, et al (Yamaha Corp) 2001] indicates their intention to try and stake a claim in this emerging area. Fortunately, close analysis reveals a number of potential patent infringement avoidance strategies. Another patent of interest is “Method and apparatus for using visual images to mix sound “ [Gibson, David 1998].

 

Recent academic research findings into useful human-music computer control are also available for mining, including; [Rowe, Robert 2001] [CHI2001 2001] [Cook, Perry 2001] [IRCAM 2000] [Wessel, David 2001] [Levin, Golan 2000]. The computer music software industry and its extensive and varied user base are also a fertile research area, as are the music hardware manufacturers.

Research Plan, Methods, and Techniques.

All research will be carried out at the REV project Lab and the chief investigators home lab.

Hardware and Software requirements

  1. Augment current Lab systems with 2 x high end PC’s (1 PC + 1 Laptop - min 733mhz PIII 128MB Ram). High end graphics card. High end Audio card with multiple outputs. Multi boot OS with Win2k/Linux.
  2. Wireless base station for network testing and client cards.
  3. 2 x Compaq iPaq PDA’s or similar (Sony/Palm).
  4. 2 x Data projectors
  5. Controller(s): Selection from: Roland HPD-15, GK-2A + VG88/GR-33, Ztar, iCube , Sensorlab http://www.steim.nl/sensor.html or other microcontrollers
  6. Sensors: Selection from http://www.infusionsystems.com/fsd.html , Gyration SDK http://www.gyration.com/kore/catalog/home/content.html

 

Software: Academic versions available

  1. Software Applications: SSEYO KoanPro/KoanX, Wavelab, Cubase VST/32, FruityLoops, Flash, Dreamweaver, Premiere, Tao Intent SDK
  2. Software development: Java (free), IDE: NetBeans, Visual Café + Structurebuilder (Academic License) or Together/J (Academic license).

Project Outputs

The core project concept is to develop a set of reusable software components that can be utilised in a number of different visual-music products and usage scenarios. These products are currently identified as:

  1. Interactive Music and Audio Games [Creative Technology 2001] …
  2. Musical aLife [Berry, Rodney 2001] [Dahlstedt, Palle 2000]
  3. Audio Visual Musical Instrument (SonicViz) [Ars Electronica, 2001]

 

These components will be embodied in a number of “proof-of-concept” products and instruments.

Design Methodology

Figure 3 illustrates an iterative Virtual Environments design process easily adapted to the visual-music development domain, while figure 4 details some of the finer grained aspects. These can also be augmented with valuable lessons learned from recent computer music HCI [Matthews, Max 2001] [Cook, Perry 2001] [Orio, Nicola 2001] [Wessel, David 2001] [CHI2001 2001] [Levin, Golan 2000].

Process

Figure 3

Narrative design in this case would refer to visual and musical composition elements.

 

Interface Design

Figure 4 Interface design and Usability Engineering [Greenberg, Saul 2001]

 

 

Visual-music Product Evaluation.

Qualitative and quantitative evaluation will be continuously performed against the following performance criteria (not product specific):

 

Formative and summative usability testing will also be conducted [Greenberg, Saul 2001]. 

 

Development and Rehearsal of Concert Performance Repertoire.

One of the final aims of this project is to also produce repertoire in the form of notated scores and improvisations specifically composed for the instrument(s). This serves as a final evaluation of the visual-music products through performance, demonstration, and/or user participation to a live audience at the REV festival.

 

Project Management Methodology

[Booch, Grady 1998]

 

A commercial software development process has been adopted so as to reflect the industry-focused nature of the project under consideration. Terms such as “Business Modeling (Use case/scenario elicitation)”, “Business need (game, toy, installation,  etc)” and “Customer representative (musician, consumer, etc)” reflect this market focus.

The project is managed around a minimal implementation of the Rational Unified Process (RUP) called dX. RUP is a best practice software project framework that describes a class of processes that are iterative and incremental, as figures 3 and 4 illustrate. The dX process will itself be adapted to fit the unique requirements of a real-time visual-music system development process, within the obvious constraints of time, budget and resource. These constraints and requirements will be defined more clearly when the REV equipment possibilities become clearer.

Dev Process

Figure 5 Core development process workflows and milestones

 

Projected Outcomes.

The project will create software with significant potential commercial outcomes for adoption on current and emerging hardware platforms. The artistic outcomes of the project address an important and emerging area of music and contribute significantly to the Australian and International culture. There will be several areas of the project that will be well suited to Masters and Doctorate projects, developing and expanding visual-music research methodologies and performance research practices.

 

Chief Investigator Background

Educational Qualifications:

 

Commcercial Experience

 

Creative Experience

References.

 

SIGCHI, A. (2001). ACM SIGCHI : Special Interest Group on Human Computer Interaction.. http://www.acm.org/sigchi/cdg/cdg2.html

 

Aigrain, P. (1999). New Applications of Content Processing of Music. Journal of New Music Research, 28(4), 274-280.

 

Devices, A. (2001). SoundMAX. http://www.staccatosys.com/audioforgames/#spx

 

Ars Electronica (2001). Active Score Music: Scribble and Small Fish Tale. Ars Electronica. www.aec.at

 

Artworks.co.uk (2001). Organic Sound to Light. http://www.artworks.co.uk/home.htm

 

Beatnik (2001). Beatnik. http://www.beatnik.com/general/press/01-09-05-xmf.html

 

Berry, R. (2001). Listening Sky.. http://www.mic.atr.co.jp/~rodney/

 

Booch, G., Martin, R. C., & Newkirk, J. (1998). Object Oriented Analysis and design with Applications. Addison Wesley Longman.

 

CHI2001 (2001). New Interfaces for Musical Expression. http://www.csl.sony.co.jp/~poup/research/chi2000wshp/

 

Codemasters (2001). MTV Musicgenerator2. http://www.codemasters.com/mtvmg2/

 

Cook, P. (2001). Principles for Designing Computer Music Controllers. http://www.csl.sony.co.jp/~poup/research/chi2000wshp/#

 

Technology, Creative. (2001). Oozic™ Reactor. http://www.soundblaster.com/oozic/

 

Dahlstedt, P. & Nordhal, M. (2000). Living Melodies: Coevolution of Sonic Communication. Chalmers University of Technology, Sweden.

 

Downie, M. (2001). Behaviour, animation, music: the music and movment of synthetic characters [MA]. MASSACHUSETTS INSTITUTE OF TECHNOLOGY.

 

Gibson, D. (1998). Method and apparatus for using visual images to mix sound. http://www.virtualmixer.com/vm_test_djl/flash.html

 

Goudeseune, Camille  http://zx81.ncsa.uiuc.edu/camilleg/

 

Greenberg, S. (2001). Human-Computer Interaction & Computer Supported Cooperative Work.. www.cpsc.ucalgary.ca/~saul/481/index.html

 

IRCAM (Ed.). (2000). Trends in Gestural Control of Music. IRCAM.

 

Jorda, S. (2001). New Musical Interfaces and New Music Making Paradigms. http://www.csl.sony.co.jp/~poup/research/chi2000wshp/#

 

Levin, G. (2000). Audio Visual Environment Suite.. http://acg.media.mit.edu/people/golan/aves/

 

Levin, G. (2000). Painterly Interfaces for AudioVisual Performance [MA]. MASSACHUSETTS INSTITUTE OF TECHNOLOGY.

 

Marrin, T. (2000). Inside the Conductor’s Jacket:Analysis, Interpretation and Musical Synthesis of Expressive Gesture [PhD]. MASSACHUSETTS INSTITUTE OF TECHNOLOGY. http://www.media.mit.edu/~marrin/Thesis.htm

 

Matsura, M. (1999). Vib Ribbon. Nana-On sha for Playstation. http://www.thegia.com/psx/vib/vib.html

 

Matthews, M., Verplabnk, B., & Sapp, C. (2001). A Course on Controllers. http://www.csl.sony.co.jp/~poup/research/chi2000wshp/#

 

ModernGroove (2001). ModernGroove (2001). http://mgi.moderngroove.com/titles/default.asp

 

Mulder, A. (1998). Design of Virtual Three-dimensional Instruments for Sound Control [PhD]. Simon Fraser University.

 

Orio, N., Schnell, N., & Wanderley, M. (2001). Input Devices for Musical Expression:Borrowing Tools from HCI. http://www.csl.sony.co.jp/~poup/research/chi2000wshp/#

 

Paradiso, J. A. (1998). Getting the Picture. IEEE Computer Graphics and Applications(May/June 1998).

 

Rowe, R. (2001). Machine Musicianship. Masachusetts Institute of Technology.

 

SSEYO (2001). Koan. www.sseyo.com

 

Suziki, e. a. (. C. (2001). Musical image display apparatus and method storage medium therefor. http://patft.uspto.gov/netacgi/nph-Parser?Sect1=PTO2&Sect2=HITOFF&p=1&u=/netahtml/search-adv.htm&r=1&f=G&l=50&d=CR98&S1=5,812,688.UREF.&OS=ref/5,812,688&RS=REF/5,812,688

 

Wessel, D. & Wright, M. (2001). Problems and Prospects for Intimate Musical Control of Computers. http://www.csl.sony.co.jp/~poup/research/chi2000wshp/#