BioSonics – Interactive Growth System

Daniel Bisig

Artificial Intelligence Laboratory, University of Zurich, Switzerland..

e-mail: dbisig@ifi.unizh.ch

Abstract

BioSonics is an interactive art installation that explores the transformation of human interaction into spatial and temporal patterns by a dynamical and self-organized system. It both looks at aesthetic qualities of pattern formation processes and explores means of intuitive interaction with complex systems. In BioSonics, users interact with the growth processes of an organism. These processes are controlled by a chemical reaction network whose underlying dynamics eventually leads to changes in the organism’s appearance and behavior. BioSonics produces both visual and acoustic feedback. Each chemical in the system controls its own sound synthesis engine whose acoustic output depends on the characteristics of the chemical, its concentration and its spatial position. Users interact with the organism by means of several microphones. The sounds picked up by the microphones are converted back into chemicals, therefore leading to changes in the dynamics of the chemical reactions and the growth processes they control. By using an acoustic interface, the user’s influence on the system is embedded within the acoustic properties of the environment of which the interactive system as well as other users are a part. Interaction therefore becomes a collaborative endeavor to which the growing organism continually responds by changing its appearance as well as by modifying its acoustic feedback and sensitivity.

1. Introduction

Complexity theory has become one of the most influential scientific disciplines since the end of the 20th century [2]. As a result, complex system research nowadays plays an important role in many scientific disciplines. More recently, application oriented areas such as engineering, entertainment and art are becoming influenced by this research. Of particular interest to art and entertainment are the capabilities of complex systems to create patterns at a variety of spatial and temporal scales that constantly change, adapt and evolve [8]. The highly dynamic aesthetics of these patterns form an important motivation for artistic explorations. Complex systems respond to external influences in a variety of ways ranging from an immediate return to the previous state to a drift into a qualitatively different regime. In an interactive setup, complex systems therefore tend to respond in non-trivial and surprising ways to user input. This fact renders interactive complex systems both attractive and problematic at the same time. Interaction can quickly become frustrating and boring for an inexperienced user if no causality between input and feedback seems to exists. Consequently, one of the major challenges in applying complex systems to interactive art and entertainment consists in the development of a suitable combination of feedback and interaction. The system BioSonics which we present in this paper tries to address these issues.

2. Concept

BioSonics explores both the aesthetics of a particular complex system as well as means of intuitive interaction with such a system. It draws inspiration from biological principles and concepts in Human Computer Interaction (HCI).

Biological systems represent particularly interesting complex systems with regard to their flexibility, robustness, adaptivity and wide range of morphological and behavioral patterns. All biological organism arise through a process of growth which in itself is a combination of several complex systems acting in parallel on different organizational levels (molecular, cellular, tissue). Some of these processes are sensitive to environmental influences. This sensitivity results in the formation of a complex adult morphology which is adapted to the properties of its habitat. BioSonics implements an abstract form of a growth process by combining an artificial chemistry [3] with a simple physical simulation of a body morphology. Growth results from the mutual influence between these two organizational levels. The chemical system is sensitive to user input. Therefore, the user activity forms part of the dynamics which ultimatively controls the appearance of shape.

In HCI research a body of literature exists that looks at issues of interaction with artificial systems. The exploration of adequate forms of sensory feedback constitutes an important aspect of HCI research. The concepts of direct manipulation [7] and multimodal interaction [10] try to transform abstract representations into direct sensations. In many situations the interactive experience benefits from the combination of several sensory modalities that complement each other [1]. Dynamic visualization and computational steering concepts [5] address issues of real time interaction with computer programs that generate a large number of temporally changing data. BioSonics tries to combine some of these HCI concepts. It does so by providing simultaneously visual and acoustic feedback about the ongoing growth process. The artificial chemical system is transformed into sound whereas the organism’s morphology is visualized using 2D graphics. These two modalities complement each other: vision excels at discriminating spatial details while audition is particularly good in the detection of temporal patterns. Interaction with the system happens in real time and relies on input via several microphones. This input directly affects the chemical system. By this way acoustics is used as the same modality for both interaction and feedback.

3. Implementation

The implementation of BioSonics involves the following aspects: installation hardware, morphological model, chemical model, growth , visual and acoustic feedback, and interaction.

3.1. Installation Hardware

BioSonics was developed as an interactive art installation. As can be seen on figure 1 the installation consists of a horizontal plexiglas plate held in position by four aluminum rods and a black cube. The plate is positioned at about 1.3 meters above ground and has a size of one square-meter. The cube contains all electronic components such as a computer, a beamer and

Figure 1: BioSonics Installation. Depicted is the installation which was shown at the exhibition “Abstraction Now” in Vienna.

two acoustic mixers. One set of speakers is built into each vertical face of the cube. These four speakers constitute a quadraphonic sound system (see 3.6). The computer generated image is back-projected through a circular hole in the cube’s top face onto the plexiglas plate. Since the plexiglas plate is slightly transparent the displayed image seems to float in midair. This gives spectators the impression of looking down into a pond containing a growing organism. A microphone protrudes from the upper end of each aluminum rod. These four microphones serve as acoustic input devices for interaction with the growing organism (see 3.7).

Figure 2: Extended Mass Spring System. Mass-Points are indicated by filled circles. Lines between these circles represent springs. The following forces are indicated in the figure: 1) spring force 2) angular force 3) Brownian force 4) chemotaxis force.

3.2. Morphological Model

The structure of the artificial organism is implemented as an extended mass-spring system (see figure 2). In addition to the spring and damping forces modeled in standard mass-spring systems, BioSonics implements an angular, Brownian and chemotaxis force. The angular force causes two successive springs to assume a preferred relative orientation. The Brownian force contributes a random component to the overall force vector. The chemotaxis force points into the direction of a chemical gradient (see 3.3). The simulation of the morphological model proceeds by using a simple Eulerian integration scheme. The paper should be completed (if possible) using Microsoft Word . The paper size should be A4 (210 mm x 297 mm). Line spacing should be 1.

3.3. Chemical Model

BioSonics implements a very simple artificial chemistry consisting of a total of six different types of chemicals. These types differ with regard to their diffusion coefficients, their initial concentration at the beginning of the simulation, the current concentration and the reactions they participate in. Reactions are always of the following type:

Reactions are unidirectional. The direction of the reaction depends on the sign of the reaction rate. The three reaction partners can be the same chemical, a different chemical or a null chemical (e.g. no chemical). Therefore, any of the following reactions can be represented:

Each reaction is specified by the three chemicals involved, its rate, yield, threshold and saturation. Chemical concentrations and reaction parameters are in arbitrary units and always range either between 0 and 1 (concentration, yield, threshold, saturation) or between -1 and 1 (rate). Chemical concentration changes are calculated as follows:

Each structural element contains either one (spring) or three (mass-point) chemical compartments. Every compartment stores its own set of chemicals. Chemicals can be exchanged between neighboring compartments by diffusion (see figure 3). Reactions between chemicals occur within these compartments. This chemical system represents a sophisticated variant of reaction diffusion systems [9] in which the organization of the organisms morphology determines the shape and neighborhood relationships of the chemical dish grid.

Figure 3: Chemical System. Part A depicts reactions among chemicals. Circles represent the different types of chemicals. Positive reaction rates are represented by filled arrows, negative rates by outlined arrows. In Part B chemical compartments are indicated as small outlined circles containing a set of six chemicals each. Arrows between chemical compartments indicate diffusion of chemicals. Part C hints at how the arrangement of chemical compartments in Part B forms a small subset of all the chemical compartments which are embedded in the entire structure.

Figure 4: Reaction Network. Each compartment contains the same reaction network. Circles represent the different types of chemicals. Structural elements are depicted as rounded rectangles. Arrows represent reactions: filled arrows indicate positive effects, empty arrows negative effects.

3.4. Growth

Growth of the simulated organism results from the interactions between its artificial chemical system and its body morphology. These interactions are implemented as a special kind of reactions in which both chemicals and structural components form the reaction partners.Essentially, any chemical parameter can control any structural parameter and vice versa. By this way the structure and the chemicals form a reaction network (see figure 4). Possible reactions include: creation and deletion of structural elements, consumption of chemicals in order to sustain structural elements, modification of structural parameters, and fusion of structural elements due to proximity. Growth always starts from an single mass-point containing a set of initial chemical concentrations. In the absence of any user input the internal dynamics of these chemicals lead to a mostly deterministic growth process (see figure 5).

For the moment all reactions are hard-coded and somewhat arbitrary. The intention in designing the reactions was to create a system that shows both a fairly interesting behavior when left on its own but at the same time is very responsive to user input.

Figure 5: Growth Process. This time sequence depicts the graphical representation of the structure (top) and the spatially averaged chemical concentrations (bottom). Time runs from left to right. Numbers indicate corresponding positions in simulation time. For information concerning the visualization of the structure refer to 3.5.

3.5. Visual Feedback

BioSoncis displays the structure of the growing organism graphically as a collection of simple 2D shapes such as rectangles for mass-points and lines for springs. Each mass-point and spring possesses in addition to its physical parameters a set of values controlling its display characteristics (color hue, color saturation, color alpha, diameter and line width). These characteristics are subject to change based on the reactions that take place between chemicals and structural parameters (see figure 4). To convey the spatial dynamics of the growing structure, corresponding mass-points between consecutive time frames are connected by lines. The length and direction of these lines visualize the direction and magnitude of the structure’s motions. At the same time successive images representing consecutive structural states are displayed on top of each other with older images gradually fading into the background. . Depending on the amount of fading applied BioSonics displays a more or less densely entangled meshwork of points and lines. This mesh-work creates the illusion of a highly complex morphology despite the fact that due to computational performance issues the actual morphology is never larger than about 100 mass-points and springs. During the visualization the simulation automatically switches between different fading values. These fading values in combination with the changing display characteristics of the structure create a wide variety of graphical patterns (see figure 6).

Figure 6: Visual Representations of the Growing Organism. During the growth process it is not only the body structure of the organism that changes but its graphical representation as well.

3.6. Acoustic Feedback

BioSonics relies on acoustics in order to render the dynamics of the simulated chemical processes perceivable. The acoustic output is generated by means of additive synthesis [6]. The frequencies and amplitudes of the combined sine waves are characteristic for each type of chemical. Each visible chemical compartment continuously plays all six chemical sounds at an amplitude which is proportional to the concentration of these chemicals within the compartment. In the current setup (see 3.7) a quadraphonic sound system is built into the installation. Chemical sounds are positioned within the quadraphonic sound space depending on the relative screen position of the corresponding compartments. Compartments at the center of the screen play their chemical sounds at equal loudness on all four speakers. Shifting a compartment away from the screen center results in an equally asymmetric spatial audio output. By this way the temporally and spatially changing sound patterns directly reflect changes in the chemical system.

3.7. Interactivity

Interaction with a complex system constitutes one of the key aspects of BioSonics. Since BioSonics relies on sound both as a means of providing feedback and modality of interaction, it implements a unique form of the direct manipulation concept [7]. In the current setup, four microphones pick up all the sounds provided by users and the environment. The frequency spectra of the recorded sounds are compared to the sounds associated with the various chemicals. If the similarity is sufficiently hight a certain amount of the corresponding chemical in fed into the organism thereby changing the state of the chemical system (see figure 7). The dynamics of the chemical system is therefore both the result of its internal reaction network and the activity of the users and the environment. The amount of chemical which is infused in a particular compartment depends sound similarity, amplitude and distance between compartment and sound source. Each microphone acts as a chemical source possessing its own position in screen coordinates. The distance is calculated according to the following formula:

and clipped to positive values. In this equation, CP is the position of the chemical compartment, CM is the center of mass of the organism and S is the position of the chemical source. The chemical concentration infused into the chemical compartment is multiplied by the squared inverse of this distance. By this way the effect of interaction on the chemical processes is strongest within those compartments that are closest to the interacting user. By choosing a particular microphone users decide which parts of the organism are exposed most strongly to interactivity related changes. When interacting with BioSonics, users not only infuse chemicals into the growing organism but also create chemical gradients within the simulated environment of the organism. These gradients result from the differing amounts of chemicals which are produced by each chemical source. The organism exhibits chemotactic behavior within these gradients. Each type of chemical possesses a certain attractiveness for the organism. Chemicals which lead to an increase in the growth rate of the organism have a positive attractivity value. On the other hand, chemicals that delay growth or cause a reduction in the size of the organism have negative attractivity values. Depending on this attractiveness a force vector pointing towards or away from the steepest gradient slope is applied to all mass-points of the organism’s structure (see figure 8). The sum of all these force vectors is multiplied by the squared distance of the organism center of mass to the edge of the screen. The resulting chemotaxis force causes the organism to move towards users which cause the production of attractive chemicals.

Apart from exerting influence on the dynamics of the chemical system interactivity also affects the acoustic properties of the chemicals. Whenever the system evaluates the spectral similarity between the recorded sounds and the chemical sounds it tries to improve the best match by slightly changing the additive synthesis parameters of the corresponding chemical sound. Over an extended period of time this mechanism causes the acoustic feedback of BioSonics to match the acoustic properties of its environment. As a result, the growth process becomes increasingly sensitive to frequent environmental sounds.

Figure 7: Interaction. This time sequence depicts the graphical representation of the structure (top graph), the spatially averaged chemical concentrations (middle graph), and the interactively produced chemicals (bottom graph). Time runs from left to right. Numbers indicate corresponding positions in simulation time.

4. Results and Discussion

The project BioSonics has been presented to various people both at the Artificial Intelligence Lab of the University of Zurich and during an exhibition entitled “Abstraction Now” which took place from August to September 2003 at the Künstlerhaus in Vienna, Austria. During these presentations we were mainly interested in informal feedback concerning the aesthetics and interactivity of the system.

The aesthetics of the visual feedback has provoked very positive feedback. Both the somewhat unorthodox display system as well as the abstract nature of the visuals contributed to this positive response. In a newspaper article [4] the visual representation is described as almost pictorial in appearance. The smootsh motions of the continuously rearranging structure were appreciated as well. It was mainly this constant metamorphosis which reminded spectators of living systems. Surprisingly, nobody considered the combination of abstract graphical elements with smooth live-like motions to be of contradicting aesthetical value.

The acoustic qualities of the system have received more critical feedback. Some people appreciated the slowly changing timbre of the sound. For other people its was exactly this sound quality which they considered to be monotonous and boring. Regardless of musical

Figure 8: Chemotaxis. This figure illustrates the concept of chemotaxis implemented in BioSonics. The four microphones which act as chemical sources are indicated by large circles. In the extreme situation depicted here each microphone produces exactly one chemical at maximum concentration and none of the others. The organism is highly attracted to the chemical produced by the top left microphone, it dislikes the chemical produced by the bottom right microphone, and it is indifferent to the chemicals produced by the other two microphones. For this reason the overall motion of the organism is towards the top left microphone.

taste we think that in the current implementation the acoustic output is flawed for two reasons. Firstly, by having hundreds of chemical compartments produce chemical sounds of identical timbre but slightly different dynamics at the same time leads to a blurring of the acoustic output. This effect becomes more pronounced as the organism gains in size. Secondly, the aesthetics of temporal change is not necessarily the same for visual and acoustic feedback. In BioSonics, the temporal characteristics of acoustics and visuals are very similar because both the chemical system and the structure of the organism are tightly linked. In order to produce a slowly changing visual appearance the accompanying acoustic output is necessarily slowly changing as well. By using a more indirect and complicated relationship between chemical processes and structural changes this problem could possibly be solved. On the other hand, highly indirect chemical effects will impair the interactivity of the system since both the system’s response and predictability is lowered. It remains a challenge to find a proper balance between these two conflicting goals.

Our evaluation of interactivity comprised the following aspects: the ability of the system to catch a user’s interest, whether this interest is maintained by providing an engaging interaction, and the amount of intuitive understanding the users acquired by simply interacting with the system. For obvious reasons, feedback concerning these issues was restricted to users who didn’t know the system in advance and did not possess any background in the field of complex systems.

Initial interest was generated through the changing feedback of the system and because of the fact that these changes were not immediate. This interest was further supported by the perceived degree of synchronization between acoustic and visual output. The fact that synchronization was neither total nor totally absent, mediated the impression that both outputs are coupled by a non-trivial algorithm. Interactivity played an important role in generating continued interest. It became clear that users had to overcome two obstacles to engage in interaction. First of all, users are not accustomed to interact with an installation by using acoustics, in particular in a museum setup. The default form of exploring an art installation is to stare at it in silent astonishment. Secondly, despite pictographic labeling most users didn’t recognize the microphones as such. For these two reasons we explicitly had to tell users to produce sounds in order to have them realize that the system responds to acoustic input. It quickly became obvious that different users interacted in very different ways. Some users were happy to expose the entire organism to very noisy sounds which usually resulted in large bursts of growth. In order to cause smaller and more diverse changes in the growth process users had to produce strongly pitched sounds. In this case, the sound input matched only one or two chemicals which were consequently infused into the system. Users that interacted in this way quickly realized, that they could recreate structural changes they had previously observed by mimicking the acoustic feedback the system provided. This type of interaction was therefore a prerequisite to create a longer lasting interest in the system and to promote an intuitive understanding of the system’s behavior.

5. Conclusion and Outlook

BioSonics has been conceived and designed as an interactive art installation that explores the aesthetics of pattern formation by complex systems as well as issues of intuitive interaction with such systems. In its current implementation, the system allows users to interact in a playful and exploratory way with an artificial growth system that provides both acoustic and visual feedback. The informal user feedback concerning the aesthetics of the system’s feedback and its interactivity has been mostly positive.

For this reason we believe that the current approach to the creation of an interactive complex system for art and entertainment is sufficiently promising to justify further research. This research will mainly concentrate on improvements in the systems acoustic feedback and its interactivity.

The mostly monotonous characteristics of the acoustic feedback results from a possibly inadequate acoustic synthesis technique and a too simple relationship between chemical dynamics and structural effects. By increasing the complexity of the underlying chemical system a wider variety of temporal patterns could be produced. The relationship between the chemical system and the morphology needs to be carefully redesigned in order to maintain the slow paced life-like metamorphosis of the organism. In an improved version of BioSonics the number of chemical compartment whose chemicals produce sound should be reduced. Such a subset of sound producing compartments could either be interactively selected by the user or be specified automatically based on the proximity of the compartments to the microphones. In this way the compartments which are most likely to be affected by the users interaction are the ones that provide acoustic feedback. Finally, the acoustic output could be rendered more interesting by employing a different method of sound synthesis. In such a setup not every chemical would necessarily produce its own sound but could rather act in combination with other chemicals to control sound synthesis.

Interactivity could be improved by giving users means of navigating within the morphological structure. Instead of drawing the entire structure at a constant magnification users could choose to zoom in on particular details of the growing organism. This sort of control could be achieved by tracking the users hand on top of the display.

Finally, we would like to move to 3D graphics for representing the structure of the organism. The added dimension allows the display of a wider variety of structures and improves the immersiveness of the interactive experience, in particular when combined with 3D spatial sound.

5. Acknowledgement

The interesting discusssions with Dale Thomas on technical and conceptual topics pertaining to this work are highly appreciated. This research is part of a project entitled “Embodied Artificial Intelligence” that is funded by the Swiss National Science Foundation.

References

[1] Lynne E. Bernstein and Christian Benoît. For speech perception by humans or machines, three senses are better than one. In Proceedings of the International Conference on Spoken Language Processing, pages 1477–1480, 1996.

[2] Peter Coveney and Roger Highfield. Frontiers of Complexity. Faber and Faber, 1995.

[3] Peter Dittrich, Jens Ziegler, and Wolfgang Banzhaf. Artificial chemistries - a review. Artificial Life, 7(3):225–275, 2001.

[4] Malewitschs erben im digitalzeitalter: «abstraction now» in wien. Westdeutsche Zeitung, 2003.

[5] Jurriaan D. Mulder, Jarke J. van Wijk, and Robert van Liere. A survey of computational steering environments. Future Generation Computer Systems, 15(1):119– 129, 1999.

[6] Curtis Roads. The Computer Music Tutorial. MIT Press, 1996.

[7] Ben Shneiderman. Direct manipulation for comprehensible, predictable and controllable user interfaces. In Intelligent User Interfaces, pages 33–39, 1997.

[8] Christa Sommerer and Laurent Mignonneau. Modeling complex systems for interactive art. In Applied Complexity - From Neural Nets to Managed Landscapes, pages 25–38. New Zealand Institute for Crop and Food Research, 2000.

[9] Alan Turing. The chemical basis of morphogenesis. Philosophical Transactions of the Royal Society, (237):37–72, 1952.

[10] Alex Waibel, Minh Tue Vo, Paul Duchnowski, and Stefan Manke. Multimodal interfaces. Artificial Intelligence Review, 10(3-4):299–319, 1996.