Producing cinematographic narration by making use of Artificial Life techniques.
Faculty of Communication and Media Studies
National & Kapodistrian University of Athens
In the case of an interactive artwork, the spectator/user finds herself interacting with a "living" technological artifact, in the sense that the form and narration of this artifact evolves according to the actions of the user. The notion of a "living" entity is used, in the context of the science of artificial life (AL), in an effort to design and produce systems that simulate natural living systems. An AL approach, however, is not necessarily applicable only to AL applications.
This paper documents the theoretical aspect of a research project that deals with the application of an artificial life approach to creating a system of cinematographic narration. This project investigates the possibility of producing an autonomous cinematographic narration system, in which meaning results from a kind of hypermontage, conditioned by genetic algorithms. Such a system is currently under study and will eventually be developed in cooperation with the Dept. of Informatics of the University of Athens.
It is often impossible for the author of an interactive artwork to predict all possible forms of narration that the user may experience. In the case of genetic art, the process of experiencing and “reading” the work is replaced by a process of observing the development and maturing of a "living" system. Instead of content formed in a predetermined manner, the user experiences an unanticipated, autonomously and spontaneously formulated kind of content.
A traditional work of art may be read in multiple ways, which depend on the subjectivity (and arbitrariness) of the spectator-reader. In the case of AL, the multiplicity of the artwork mainly depends on the manner in which the artifact is being written. Having the same structural elements as a starting point may result in different readings. Having the same data as a starting point, may lead to different versions of the same work of art.
The roles of the author-producer as well as the spectator-reader’s are being reconsidered in the context of AL technology. Even if certain elements of the content have been pre-recorded, the "new artifact" is actually being created in real time, in the presence of the public. Coding and decoding, "reading" in the technical sense and "interpretation" coincide in real time: writing and reading are intertwined.
In this case, signifier and signified correspond to the genotype and phenotype of DNA. Following the rules imposed by genetic algorithms, the resulting signifier elements alter what is signified by the cinematographic text in an unanticipated way.
“Form is the end, death. Form-giving is movement, action. Form-giving is life.” 
Each and every form of narrative is inextricably linked to the medium via which it is communicated. Both oral and written speech adhere to the rules of linguistics as well as to the potential acquired by the physical “writing field”. Oral speech takes into account the physical abilities of the speaker as well as the auditory interface system, at work between her and the environment, while written speech takes advantage of the ability to lay down pictures and schemes on paper or on any other surface. The listener or reader should be able to decode the communicated message. In the case of oral interaction they both have to speak the language used and their sound comprehension system or lip reading ability should be at work, whereas in order to read a written message they have not only to speak the language in question but also to read it. Digital technology supports the digitization of static and animated image, audio, text as well as non-linear access to any type of digitized media.
Narratives supported by hypermedia systems adhere to a series of conventions that render them readable. However, we cannot suggest that a particular writing system, entailing grammar and syntax, that could totally condition the creation of digital hypermedia systems, has so far been established. For the time being it would be quite hard to create an analytic and strict writing code in compliance with a linguistics model.
It is suggested that a language of narrative that stems from representation through moving images and is supported by computers can be formed via the combining existing theories of cinematic language with theories that propose reconsideration or even change-over of conventional forms of narrative . Becoming familiar with the grammar and syntax of cinematic language constitutes the first stage of reading, which is essential for conceiving and further comprehending the message communicated through new media. A person may watch the input and output of information between her and the computer, through a monitor or a projector, which constitutes the visualization field of moving images, accompanied by sound. The projected image follows, to a great extent, the visual and semiotic conventions already known to us, originally from cinema and later from television.
With reference to interactive narratives and more specifically in the case of interactive cinema, there exists a “live” spectacle, the narrative and duration of which are activated, controlled and affected by the viewer. The latter does not remain a mere observer: she is simultaneously assigned the role of director, editor and often the lead actor. The computer provides the potential for an interaction process. Apart from new possibilities to create cinematic narrative afforded by these technologies, sensory and multi-sensory freedom to perceive is also provided.
Does the “death of the author”  necessarily lead to the birth of the reader?  Not literally. The role of the author has become more discrete. The author is there for instructing the reader on how to write her own script, based on rules predetermined by the creator herself, which render the story more or less flexible.
An older concept, widely common nowadays, is “hypertextuality” . Hypertext, as well as hyperimage are the key component parts of new media pluralism. They are the features that provide the user with the right to choose. In correspondence with the aforementioned terms, the concept of hypermontage  has been introduced, with reference to interactive cinema.
Apart from interaction itself, the source of other elements of an interactive artifact, namely the shots and the montage, relate to a certain extent to conventional cinema. Interactive artifacts are hybrid systems constructed by directors as far as their contents and mechanisms are concerned.
New media are still using the cinematic language that recognizes the séquence as the structuring element of the audiovisual transmission system. Cinema differs from other narrative methods in that linear narrative evolves within time and space. The introduction of the time parameter in narrative has imposed a new writing method, able to establish a correlation amongst the protagonists, their surroundings, the story plot and time, via their representation through images and text. That form of expression is exemplified on paper by comics and storyboards, the nature of which is far more abstract than the rich and often realistic images that prevailed in European culture, mainly after the Renaissance period, where a whole narrative was included within a single image.
This paper documents the theoretical aspect of a research project that studies the production of cinematic narrative via the use of Artificial Life (AL) techniques. It discusses the issue of whether a database is able to produce cinematic narrative, without human intervention in direction and montage, and if this is so, in what manner does this production occur and evolve.
A system in which the process of creation follows an evolutionary path is currently under study and will eventually be developed in cooperation with the Dept. of Informatics of the University of Athens. The would-be product is to be non-predictable by the system designer. Upon such a process for producing meaning, the computer holds a key role. Composition is removed from the hands of a human and is mainly conducted by a machine.
A database is an information set with a regular structure. Databases, as sets of classified information, have existed since the ancient years: inventories and filings of merchandise, telephone directories even traditional libraries may all be deemed databases. The elements a database comprises are classified according to their special features.
In the case of a database that comprises elements of written narrative, these may be classified with respect to their length (the number of words they consist of), the language they are written in, the date they were written, the time in which the story is set, the type of the story (drama, comedy, etc.), the writing style/genre (poem, prose, fiction) etc. When a user is searching a film database, she has to deal with a list .
Manovich  maintains that the fragmentary organization of databases relates to the anti-narrative logic of the media that use them. He claims that if new elements are added from time to time, the outcome is a collection rather than a story. Indeed, how can a coherent narrative or any other thought process remain intact if it keeps changing? However, he also points out that computer games are actually experienced by players as narratives. He attributes their narrative character to the fact that computer games always have a specific aim.
As a cultural form, databases represent the world as a list of non-orderly elements. Conversely, a narrative creates a cause and effect plot amongst sometimes seemingly random elements. Therefore, database and narrative are natural rivals. They both struggle to dominate the same field of human culture and they both claim the right for the exclusive production of meaning .
Manovich also suggests that the database lies at the heart of creation of a cultural product in the computer era. He claims that database and narrative do not hold equal positions within the computer culture. More specifically, a database may support a narrative but there is nothing in the logic of the medium itself that would necessarily lead to its creation.
Interactive art works are characterized by the potential for multiple choice during navigation. The user is faced with hypertextual structures and is asked to act upon the interface and the system rules that have been laid down and determined by the system creator. The form and narrative of the play’s plot are affected by the courses of action followed by the viewer, who also plays the role of the user, through the feedback process. The user acts on the system and the system reacts, thereby establishing a communicational relation, a human – machine dialogue, that is structured and conveyed as an audiovisual art work.
The various potential forms of an interactive play are finite. The creator of the system is in position to forecast in advance the potential forms that the play may exhibit, as a result of interaction with the user. Even in cases when the system has been programmed to pick up an element over a group of elements at random, through the “random” command, it is easy to find all possible combinations that may be applied by the computing system, by means of probability theory. The number of options for interaction and navigation, as well as the consequent results are predetermined by the system creator.
The research project presented here, investigates the possibility of constructing a narrative, that cannot be predicted by the system creator and in which the creation process follows an evolutionary path. In that case the role of the computer is vital: the process of composition is to be conducted by the machine instead of the human.
The computation system that is adequate for exploring evolution as a creative process, entailing any random and indefinite elements of nature and culture, shall be more effective if it operates upon a mechanism simulating natural evolution stages. The discipline that attempts to simulate nature and living organisms in order to study and comprehend their mechanisms is Artificial Life (AL). Artificial Life is often depicted as an attempt to comprehend complex behaviors through simple rules . The term AL was coined in 1989 by Christopher Langton, who defined it as “the study of man made systems exhibiting behaviors typical of natural living systems” .
Genetic algorithms constitute the core method applied to simulate biological genetics through digital computation. They are based on Darwin's theory of evolution and are being used for the solution of problems, following an evolutionary process in order to achieve the best (fittest) solution (survivor), which derives from an initial set of solutions (hardly optimum) called population. A genetic algorithm contains the “genotype” that is a string code specifying a “phenotype”. The phenotype may as well be any digital apparatus: artificial organism, three-dimensional form or software part. Via simulation of genetic alterations caused by sexual reproduction and mutation, the genetic algorithm alters the genotype and the phenotype. Since the whole process is computed and does not involve biological processes, fertilization is rapid and productive. Wide ranges of phenotypes may come out, which are often automatically evaluated with respect to their “fitness”, in accordance with special criteria. As to operational applications, a rapid process of artificial evolution is applied so as to solve a complex problem through the consideration of a wide range of potential results.
Johnson explains: “Generally speaking, genetic algorithms (GA) exemplify a computation technique working on the principles of physical genetics with a view to working out solutions through problem scrutiny. A traditional genetic algorithm maintains a number of possible solutions to the problem, where each individual consists of chromosomes determined by the DNA string (genotype) that is mapped into a specific possible solution (phenotype). The seemliness of every possible solution determines each individual’s fitness……individuals whose chromosomes decode the best solutions have higher fitness” .
In most cases artificial life systems comprise two levels of application: computation and emergence. This bipolar state can be understood in a sense that broadens the concept of emergence and its application to AL related art practices. The computation level may be conceived as a more general technological layer, a designed framework of software and hardware. On the other hand, the global emergence level may be conceived as a phenomenal and behavioral product of the above technological layer. Such a distinction can be easily made in a common computation system between the software and the material “machine” and its phenomenal  products, the screen image and the sound produced: each level supports and generates the other. The key distinction lies in the relation between those two levels: within the daily operational use of the computer, causality is precise and instant: I type and, with some luck, letters appear on the screen. In artificial life systems there does not exist such simple correspondence between the essential and the phenomenal (cause and effect) but rather a complex tangled causality generating facts and events that appear to be introducing a novelty, something more .
The existence of causality that is not unilaterally linked to its cause is not a new concept. Julia Kristeva (1974) coined a multilateral definition of the text, borrowing from the terminology of biology:
“...According to the new restrictive principle, the text is redefined as any language practice such that the functions of the geno-text are projected on the pheno-text, asking the reader to create the meaning anew” .
The relation between the reader and their subject of study becomes unrestricted. The text is dealt with as a living organism interacting with the reader and evolving. The theoretical approach proposed by Kristeva is being put to practice today as a result of scientific and technological progress, that enables theories originally appearing as utopian to become feasible in practice.
Nicholas Schöffer, famous for kinetic and cybernetic sculptures he created during the 1950s and 1960s, maintains that this is metacreation: “We no longer create work, we create creation… We are able to generate… results… exceeding the intentions of their creators, in indefinite numbers” .
Creation as a process is not limited to a single product. It entails an entire system, whose component parts convert into others that further evolve in turn. In his book Beyond Modern Sculpture, Burnham  places cybernetic art within a more expanded theoretical framework focusing on art’s ultimate purpose, which is to reproduce life.
Tenhaaf  makes the connection between artificial life and art according to their common feature: interpretation. Both activities build up associative stories based upon their “chaotic environment”. Interpretation and representation are features that both art and artificial life have in common. Tenhaaf supports that both artificial life and art project metaphorical notions. Artists often recognize artificial life techniques as innately representative and use them as such. The notion of representation includes and entails the notion of metaphor and therefore that of interpretation. Is there a correlation between the interpretation of an artwork and scientific results? Is it possible to set objective criteria for the evaluation of an artwork or subjective and arbitrary criteria for the assessment of scientific-purpose simulation?
For the creation of a cinematic work both the creator and the viewer make use of cinematic language. Encoding and decoding processes are both governed by the same rules. As to the planning and making of a film, the director uses a group of cultural (daily habits, social conduct, etc.) or purely cinematic (montage and rhythm) codes, with the aim of producing meaning.
Being experienced in reading such a format, the viewer watches, understands and interprets, according to her own personal subjective criteria , the work displayed . The same language is used in both writing and reading, irrespective of the “author’s” or the “reader’s” ability to manage and process them.
With reference to the generative narrative production system that this project is developing, the output of such a system will also make use of cinematic language. The viewer is once more faced with a linear audiovisual narrative, which she decodes according to the terms already known to her from cinema. However, the composition of the play follows entirely different paths. The role of the director lies precisely in the organization and the plan of the composition process. Hence she disregards the cinematic codes of montage and rhythm and attempts to attribute objective qualities to the properties of isolated footage, in order to divide them into groups/categories.
The creator organizes the database including the audiovisual materials that are to be used for the construction of the final product and also builds up the software mechanism, which will process the composition of the database constituent parts. Therefore, the language used is a programming language. The rules governing the software in question take into account certain restrictions linked to the way we conceptualize cinema and related to montage theories. However, the language used to phrase the specific instructions of the software operation is purely a programming language. Human-machine communication through a language “understood” by both entities, is translated into a cinematic narrative that presupposes the ability of linear audiovisual narrative reading so as to be read by the viewer.
In a cinematic play the director is actively involved throughout the construction process: he works himself and he also plays the role of coordinator between his associates so that the final outcome be as close as possible to the concept she initially had in mind with respect to aesthetics, narrative and emotions. The director’s job ends at the point where the film is screened. (Figure 1)
Figure 1: Director’s work comparing to the audiovisual product.
In a “cinematic” work produced through genetic algorithms, the director’s role is restricted to the shooting and organization of the footage as well as the creation and/or adaptation of the software mechanism of production. Editing is automatically conducted by the system. The role of the director is limited to the “organization” of material so that the computation system can begin and complete the process of narrative composition, without the need of any further assistance by the creator. More precisely, the role of the “director” should be appointed to the system, for the creator/organizer provides it with the resources required for the execution of this task.
The process of a cinematic film production, usually results in the creation of a single product. No matter how many times it is screened, the film remains the same. In the system under study, the narrative products that could be built, as a result of the same work functioning, may be countless. Every time the viewer commands the system to start screening, it starts editing the database elements anew. Given the fact that the process is being conducted by genetic algorithms, which constantly alter the produced outcome, the composition of shot is unforeseeable and so is the number of the potential results. Due to the high complexity of the whole process, there is practically no chance of two outputs being the same. (Figure 2)
Figure 2: The number of outcoming results.
In an evolutionary cinematic system, the creator functions as a driving force, a stimulus of a process that goes beyond the scope of the creator’s imagination and may acquire unpredictable forms, in compliance with strict and specific rules. The designer of such a system examines the potential, the limitations and the power of the rules she establishes. She also monitors the formation of the rules she has set.
Systems that “imitate” natural selection processes for the evolution of an entity, set strict rules for the control of the evolutionary process. The results after each stage of evolution may be unpredictable for the constructor but the rules remain unaltered. Computations applied to the management of the genome by transforming it, altering the genotype or handling user interaction remain unaffected by the evolutionary processes .
After the creation of the generative narrative production system, an experimental process will be conducted, which will involve the viewing of the system’s functionality and cinematic production by a considerable number of viewers for assessing the process and its outcome, by qualitative and quantitative methods of research. The results of such an evaluation will influence the theoretical part of this research project.
1. Klee P., “Lecture of Tuesday, 9 January 1924,” in The Nature of Nature, 269 in Whitelaw (2004), p. 15.
2. Theories that illustrate such an approach were coined in the 1960s, such as Barthes’ and Kristeva’s, concerning the relationships between author – writing and reader – reading.
3. Barthes R. (1984), «La mort de l'auteur », in Le Bruissement de la langue, Paris, Le Seuil, first published in 1968, p. 69.
4. In the sense that the reader has acquired an active role in the reading process.
5. Kristeva J. (1970) «Le texte du roman », La Haye/Paris: Mouton.
6. Hakola M.,“Hypermontage”
7. Within a film database, the user inputs key elements that characterize the film she wishes to watch and is then asked to choose from a list of films that share the feature or features inserted as information in the classification system.
8. Manovich L. (2001), “Database as a Symbolic Form”
9. Adami C. (1998), “Introduction to Artificial Life”, Berlin: Springer-Verlag.
10. Langton C., (1989), <http://www.vieartificielle.com>.
11. Johnson C. G. (2003), «Exploring Sound-Space with Interactive Genetic Algorithms», in Leonardo, Volume 36, Issue 1, MIT Press.
12. In this case the term “phenomenal” stands for apparent, visible.
13. Whitelaw M. (2004), “Metacreation Art and Artificial Life”, The MIT Press, p. 215.
14. Kristeva J. (1974), «La Révolution du langage poétique : l'avant-garde à la fin du XIXe siècle», Lautréamont et Mallarmé, Paris : Éditions du Seuil, (2 éd.1985).
15. Schöffer N., “Sonic and Visual Structures: Theory and Experiment.”
16. Burnham J. (1968), “Beyond Modern Sculpture: The Effects of Science and Technology on the Sculpture of This Century”, London: Penguin.
17. Tenhaaf N. (1998), “As Art Is Lifelike: Evolution, Art and the Readymade,” Leonardo 31, no. 5, in Whitelaw, M. (2004), p. 184.
18. Metz C. (1969), “Coeur et ame de l’image”, in Media, INRP.
19. Christian Metz strictly points out that “the image is not polysemic but the viewer is”.
20. Whitelaw M. (2004), “Metacreation Art and Artificial Life”, The MIT Press, p. 220.
 For example: faux raccords must be avoided.