ArTbitrating: an evolutionary arbitrary mapping from visual to sound domain

Artemis Sanchez Moroni, Dr.

Division of Robotics and Computer Vision, CenPRA, Campinas, Brazil.

e-mail: Artemis.Moroni@cenpra.gov.br

Abstract

Here, we present ArTbitrating, and evolutionary computational system for visual and sound composition. ArTbitrating emerged from attempts of modelling some kind of creativity in visual and sound domains. Two previous interactive evolutionary systems were developed in visual and sound domain. In ArTbitrating, abstract visual compositions, as well as image files, are transformed in trajectories for production in sound domain.

1. Introduction

Kandinsky is painting music. That is to say, he has broken down the barrier between music and painting, and has isolated the pure emotion which, for want a better name, we call the artistic emotion. The effect of music is too subtle for words, and the same with Kandinsky´s paintings. Presumably the lines and colors have the same effect as harmony and rhythm in music have on the truly musical.

Color-music is no new idea [1], attempts have been made to play compositions in color, by flashes and harmonies; also music has been interpreted in color. Kandinsky refers to attempts to paint in color-counterpoint. Picasso admirers hailed him as a visual musician.

Nowadays, some visual artists are using evolutionary programs to help them produce images that they could not have imagined otherwise. Work on genetic algorithms suggests that unconscious, nondeliberative psychological processes might enable largely random (but useful) combinations and sensible selections to be made in human minds [2]. Karl Sims´s computer graphics program, for instance, uses genetic algorithms to generate new images from preexisting images [3, 4]. These systems typically operate by presenting the user with a collection of images (initially random), shown next to each other simultaneously on-screen, from which the user chooses the parent or parents of the next generation of images. The new generation is created by some set of genetic operators, the corresponding new images are computed, and then these images are again displayed for further choice. With only a few such generations of viewing and selection, users can follow promising visual avenues to create quite striking final images.

Moreover, in sound domain, a new generation of composition researchers is discovering that by using simulated evolution techniques it is relatively easy to obtain novelty – often complex novelty – despite of it is correspondingly difficult to rein in the direction that novelty takes [5 - 7].

And since we have evolutionary systems for image generation and sound generation, why not to devise a system applied to composition in visual and sound domains? This is what is being proposed next.

2. ArTbitrating: an evolutionary environment for visual and sound composition

ArTbitrating environment emerged from two other evolutionary environments, VOX POPULI, an interactive environment for sound production, and Art Lab, applied to visual domain. VOX POPULI can be described as an interactive evolutionary system that can be used as a new form of musical instrument [6, 7].

At Art Lab, an interactive genetic algorithm (IGA) is used for the generation and evolution of geometric abstract compostions [16]. Art Lab´s interface permitts the user to generate sets of four frames, each time, of the most common graphic primitives available in any programming environment: lines, boxes, arc, circles, ellipses, miscellaneous. Art Lab´s interface promotes the presentation of the generated abstract compositions for a "human mentor" to evaluate them.

At ArTbitrating environment such compositions result in sound trajectories, or curves that guide the sound production. Thus, each visual composition is the matrix of a sound production. And since that we are using visual compositions to generate sound trajectories, why not to use image files?

Be the visual composition a rendered composition or a file image, both are treated as attractors in a dynamic system, which results in the sound composition. The problem is the same: how to map visual attributes in sound attributes. Very simple mappings were applied and even with those very simple mappings some interesting material was produced. Figure 1 presents the evolutionary environment ArTbitrating while realizing a sound performance using Kandinsky´s composition On White II (1929).

To define a mapping from visual to sound domain is not a simple problem. In both domains, there is a lot of controversy about “values”. Kandinsky offers some clues in his book Concerning the Spiritual in Art, which we intend to apply to this work [8]. According to him, “keen colors are well suited by sharp forms (e.g, a yellow triangle), and soft, deep colors by round forms (e.g., a blue circle). But it must be remembered that an unsuitable combination of form and color is not necessary discordant, but may, with manipulation, show the way to fresh possibilities of harmony.” Further, Kandinsky places: “In music a light blue is like a flute, a darker blue a cello; a still darker a thunderous double bass; and the darkest of all – an organ.”

All these problems are congregated in arTbitrariness, which refers to the initiative of upgrading the aesthetical judgment through evolutionary computation and others population based techniques for exploratory search, and is interpreted as an iterative interactive optimization process [9]. The main goal of arTbitrariness is to avoid to leave to the artist what can (already) be optimized and to avoid to leave to the machine what can’t be optimized (yet), looking for an arbitrary point among subjectivity and objectivity, with its associated automation capability.

Figure 1. On the left, the ArTbitrating environment is showed presenting a performance using a mapping for sound generation that was created from Kandinsky´s visual composition On White II, on the right.

3. Conclusion

At ArTbitrating, the user can experience to create an abstract visual composition and “hear” it. He can also generate a sound production from a image file. If there is a lot to do with relation to the mapping of the attributes of the visual domain to the sound domain, each domain, by itself, already contains a universe of challenges.

Two main challenges are foreseen in ArTbitrating environment: the first is relative to the visual domain. It is necessary to add more sophisticated graphical resources, as well as criteria for the generation of visual compositions. The other is relative to the mapping itself of the attributes of the objects of the visual domain to the sound domain. Both refer to the question of the subjectivity, running into the context of the arTbitrariness. If computational creativity is still in its early days, the attempts to emulate some kind of creativity on computers already present impressive results.

4. Acknowledgements

I am very thankful to the students Rafael Boccaletto Maiolla, Leonardo Laface de Almeida and Daniel Gurian Domingues for their Java support. I am also thankful to Dr. Fernando Von Zuben, who stimulated who stimulated me in theis work. I am much obliged to my co-advisor, Dr. Jônatas Manzolli, co-author of VOX POPULI, for all his strong support in music computing and to NICS/Unicamp for allowing me to use the tools developed there. Finally, I would like to thank to PIBIC/CNPq program and to CenPRA, for the possibility of developing this work.

5. References

1. Peacock, K. Instruments to Perform Color-Music. Leonardo Vol. 21, 4 (1988) 397 – 406

2. Boden, M.: Dimensions of Creativity. 1st edn. The MIT Press, New York (1996)

3. Sims, K.: Interactive Evolution of Equations for Procedural Models. The Visual Computer, Vol. 9, 8 (1993) 466 – 476

4. Todd, S., Latham, W.: The Mutation and Growth of Art by Computers. In: Bentley, P. (ed.): Evolutionary Design by Computers. Morgan Kaufmann, San Francisco (1999) 221 – 250

5. Todd, P. M., Werner, G. M.: Frankensteinian Methods for Evolutionary Music Composition. In Griffith, N., Todd, P. M. (eds): Musical Networks: Parallel Distributed Perception and Performance. The MIT Press, Cambridge (1999) 313 – 339

6. Moroni, A., Manzolli, J., Von Zuben, F.J. & Gudwin, R.: Vox Populi: An Interactive Evolutionary System for Algorithmic Music Composition. Leonardo Music Journal Vol. 10 (2000) 49 – 54

7. Moroni, A., Manzolli, J., Von Zuben, F.J. & Gudwin, R.: Vox Populi: Evolutionary Computation for Music Evolution. In Bentley, P. (ed.): Creative Evolutionary Systems. Morgan Kaufmann, San Francisco (2002) 205 – 221

8. Kandinsky, W.: Concerning the Spiritual in Art. Dover, New York (1977)

9. Moroni, A., Von Zuben, F.J., Manzolli, J.: ArTbitration: Human-Machine Interaction in Artistic Domains. Leonardo Vol. 35, 2 (2002) 185 – 188