Producing
cinematographic narration by making use of Artificial Life techniques.
Iro Laskari
Faculty of Communication and Media Studies
National & Kapodistrian University of Athens
e-mail:
ilaskar@media.uoa.gr
Abstract
In the case of
an interactive artwork, the spectator/user finds herself interacting with a
"living" technological artifact, in the sense that the form and
narration of this artifact evolves according to the actions of the user. The
notion of a "living" entity is used, in the context of the science of
artificial life (AL), in an effort to design and produce systems that
simulate natural living systems. An AL approach, however, is not necessarily
applicable only to AL applications.
This paper
documents the theoretical aspect of a research project that deals with the
application of an artificial life approach to creating a system of
cinematographic narration. This project investigates the possibility of
producing an autonomous cinematographic narration system, in which meaning
results from a kind of hypermontage, conditioned by genetic algorithms.
Such a system is currently under study and will eventually be developed in
cooperation with the Dept. of Informatics of the University of Athens.
It is often
impossible for the author of an interactive artwork to predict all possible
forms of narration that the user may experience. In the case of genetic art,
the process of experiencing and “reading” the work is replaced by a process of
observing the development and maturing of a "living" system. Instead
of content formed in a predetermined manner, the user experiences an
unanticipated, autonomously and spontaneously formulated kind of content.
A traditional
work of art may be read in multiple ways, which depend on the subjectivity (and
arbitrariness) of the spectator-reader. In the case of AL, the multiplicity of
the artwork mainly depends on the manner in which the artifact is being
written. Having the same structural elements as a starting point may result in
different readings. Having the same data as a starting point, may lead to
different versions of the same work of art.
The roles of
the author-producer as well as the spectator-reader’s are being reconsidered in
the context of AL technology. Even if certain elements of the content have been
pre-recorded, the "new artifact" is actually being created in real
time, in the presence of the public. Coding and decoding, "reading" in
the technical sense and "interpretation" coincide in real time: writing
and reading are intertwined.
In this case,
signifier and signified correspond to the genotype and phenotype of DNA.
Following the rules imposed by genetic algorithms, the resulting signifier
elements alter what is signified by the cinematographic text in an
unanticipated way.
“Form is the
end, death. Form-giving is movement, action. Form-giving is life.” [1]
1. Introduction
Each and every
form of narrative is inextricably linked to the medium via which it is
communicated. Both oral and written speech adhere to the rules of linguistics
as well as to the potential acquired by the physical “writing field”. Oral
speech takes into account the physical abilities of the speaker as well as the
auditory interface system, at work between her and the environment, while
written speech takes advantage of the ability to lay down pictures and schemes
on paper or on any other surface. The listener or reader should be able to
decode the communicated message. In the case of oral interaction they both have
to speak the language used and their sound comprehension system or lip reading
ability should be at work, whereas in order to read a written message they have
not only to speak the language in question but also to read it. Digital
technology supports the digitization of static and animated image, audio, text
as well as non-linear access to any type of digitized media.
Narratives
supported by hypermedia systems adhere to a series of conventions that render
them readable. However, we cannot suggest that a particular writing system,
entailing grammar and syntax, that could totally condition the creation of
digital hypermedia systems, has so far been established. For the time being it
would be quite hard to create an analytic and strict writing code in compliance
with a linguistics model.
It is
suggested that a language of narrative that stems from representation through
moving images and is supported by computers can be formed via the combining
existing theories of cinematic language with theories that propose
reconsideration or even change-over of conventional forms of narrative [2].
Becoming familiar with the grammar and syntax of cinematic language constitutes
the first stage of reading, which is essential for conceiving and further
comprehending the message communicated through new media. A person may watch
the input and output of information between her and the computer, through a
monitor or a projector, which constitutes the visualization field of moving
images, accompanied by sound. The projected image follows, to a great extent,
the visual and semiotic conventions already known to us, originally from cinema
and later from television.
With reference
to interactive narratives and more specifically in the case of interactive
cinema, there exists a “live” spectacle, the narrative and duration of which
are activated, controlled and affected by the viewer. The latter does not
remain a mere observer: she is simultaneously assigned the role of director,
editor and often the lead actor. The computer provides the potential for an
interaction process. Apart from new possibilities to create cinematic narrative
afforded by these technologies, sensory and multi-sensory freedom to perceive
is also provided.
Does the
“death of the author” [3] necessarily lead to the birth of the reader? [4] Not
literally. The role of the author has become more discrete. The author is there
for instructing the reader on how to write her own script, based on rules
predetermined by the creator herself, which render the story more or less
flexible.
An older
concept, widely common nowadays, is “hypertextuality” [5]. Hypertext, as well
as hyperimage are the key component parts of new media pluralism. They are the
features that provide the user with the right to choose. In correspondence with
the aforementioned terms, the concept of hypermontage [6] has been introduced,
with reference to interactive cinema.
Apart from
interaction itself, the source of other elements of an interactive artifact,
namely the shots and the montage, relate to a certain extent to conventional
cinema. Interactive artifacts are hybrid systems constructed by directors as
far as their contents and mechanisms are concerned.
New media are
still using the cinematic language that recognizes the séquence as the
structuring element of the audiovisual transmission system. Cinema differs from
other narrative methods in that linear narrative evolves within time and space.
The introduction of the time parameter in narrative has imposed a new writing method,
able to establish a correlation amongst the protagonists, their surroundings,
the story plot and time, via their representation through images and text. That
form of expression is exemplified on paper by comics and storyboards, the
nature of which is far more abstract than the rich and often realistic images
that prevailed in European culture, mainly after the Renaissance period, where
a whole narrative was included within a single image.
This paper
documents the theoretical aspect of a research project that studies
the production of cinematic narrative via the use of Artificial Life (AL)
techniques. It discusses the issue of whether a database is able to produce
cinematic narrative, without human intervention in direction and montage, and
if this is so, in what manner does this production occur and evolve.
A system in
which the process of creation follows an evolutionary path is currently under
study and will eventually be developed in cooperation with the Dept. of
Informatics of the University of Athens. The would-be product is to be
non-predictable by the system designer. Upon such a process for producing
meaning, the computer holds a key role. Composition is removed from the hands
of a human and is mainly conducted by a machine.
2. Databases as the locus
of information collection
A database
is an information
set with a regular structure. Databases, as sets of classified information,
have existed since the ancient years: inventories and filings of merchandise,
telephone directories even traditional libraries may all be deemed databases.
The elements a database comprises are classified according to their special
features.
In the case of
a database that comprises elements of written narrative, these may be
classified with respect to their length (the number of words they consist of),
the language they are written in, the date they were written, the time in which
the story is set, the type of the story (drama, comedy, etc.), the writing
style/genre (poem, prose, fiction) etc. When a user is searching a film
database, she has to deal with a list [7].
Manovich [8]
maintains that the fragmentary organization of databases relates to the
anti-narrative logic of the media that use them. He claims that if new elements
are added from time to time, the outcome is a collection rather than a story.
Indeed, how can a coherent narrative or any other thought process remain intact
if it keeps changing? However, he also points out that computer games are
actually experienced by players as narratives. He attributes their narrative
character to the fact that computer games always have a specific aim.
As a cultural
form, databases represent the world as a list of non-orderly elements.
Conversely, a narrative creates a cause and effect plot amongst sometimes
seemingly random elements. Therefore, database and narrative are natural
rivals. They both struggle to dominate the same field of human culture and they
both claim the right for the exclusive production of meaning [8].
Manovich also
suggests that the database lies at the heart of creation of a cultural product
in the computer era. He claims that database and narrative do not hold equal
positions within the computer culture. More specifically, a database may
support a narrative but there is nothing in the logic of the medium itself that
would necessarily lead to its creation.
3. Interactive narrative
Interactive
art works are characterized by the potential for multiple choice during
navigation. The user is faced with hypertextual structures and is asked to act
upon the interface and the system rules that have been laid down and determined
by the system creator. The form and narrative of the play’s plot are affected
by the courses of action followed by the viewer, who also plays the role of the
user, through the feedback process. The user acts on the system and the system
reacts, thereby establishing a communicational relation, a human – machine
dialogue, that is structured and conveyed as an audiovisual art work.
The various
potential forms of an interactive play are finite. The creator of the system is
in position to forecast in advance the potential forms that the play may
exhibit, as a result of interaction with the user. Even in cases when the system
has been programmed to pick up an element over a group of elements at random,
through the “random” command, it is easy to find all possible combinations that
may be applied by the computing system, by means of probability theory. The
number of options for interaction and navigation, as well as the consequent
results are predetermined by the system creator.
The research
project presented here, investigates the possibility of constructing a
narrative, that cannot be predicted by the system creator and in which the
creation process follows an evolutionary path. In that case the role of the
computer is vital: the process of composition is to be conducted by the machine
instead of the human.
4. Generative narrative
The
computation system that is adequate for exploring evolution as a creative
process, entailing any random and indefinite elements of nature and culture,
shall be more effective if it operates upon a mechanism simulating natural
evolution stages. The discipline that attempts to simulate nature and living
organisms in order to study and comprehend their mechanisms is Artificial Life
(AL). Artificial Life is often depicted as an attempt to comprehend complex
behaviors through simple rules [9]. The
term AL was coined in 1989 by Christopher Langton, who defined it as “the study
of man made systems exhibiting behaviors typical of natural living systems”
[10].
Genetic
algorithms constitute the core method applied to simulate biological genetics
through digital computation. They are based on Darwin's theory of evolution and
are being used for the solution of problems, following an evolutionary process
in order to achieve the best (fittest) solution (survivor), which derives from
an initial set of solutions (hardly optimum) called population. A genetic algorithm
contains the “genotype” that is a string code specifying a “phenotype”. The
phenotype may as well be any digital apparatus: artificial organism,
three-dimensional form or software part. Via simulation of genetic alterations
caused by sexual reproduction and mutation, the genetic algorithm alters the
genotype and the phenotype. Since the whole process is computed and does not
involve biological processes, fertilization is rapid and productive. Wide
ranges of phenotypes may come out, which are often automatically evaluated with
respect to their “fitness”, in accordance with special criteria. As to
operational applications, a rapid process of artificial evolution is applied so
as to solve a complex problem through the consideration of a wide range of potential
results.
Johnson
explains: “Generally speaking, genetic algorithms (GA) exemplify a computation
technique working on the principles of physical genetics with a view to working
out solutions through problem scrutiny. A traditional genetic algorithm
maintains a number of possible solutions to the problem, where each individual
consists of chromosomes determined by the DNA string (genotype) that is mapped
into a specific possible solution (phenotype). The seemliness of every possible
solution determines each individual’s fitness……individuals whose chromosomes
decode the best solutions have higher fitness” [11].
In most cases
artificial life systems comprise two levels of application: computation and
emergence. This bipolar state can be understood in a sense that broadens the
concept of emergence and its application to AL related art practices. The
computation level may be conceived as a more general technological layer, a
designed framework of software and hardware. On the other hand, the global
emergence level may be conceived as a phenomenal and behavioral product of the
above technological layer. Such a distinction can be easily made in a common
computation system between the software and the material “machine” and its
phenomenal [12] products, the screen image and the sound produced: each level
supports and generates the other. The key distinction lies in the relation
between those two levels: within the daily operational use of the computer,
causality is precise and instant: I type and, with some luck, letters appear on
the screen. In artificial life systems there does not exist such simple
correspondence between the essential and the phenomenal (cause and effect) but
rather a complex tangled causality generating facts and events that appear to
be introducing a novelty, something more [13].
The existence
of causality that is not unilaterally linked to its cause is not a new concept.
Julia Kristeva (1974) coined a multilateral definition of the text, borrowing
from the terminology of biology:
“...According
to the new restrictive principle, the text is redefined as any language
practice such that the functions of the geno-text are projected on the
pheno-text, asking the reader to create the meaning anew” [14].
The relation
between the reader and their subject of study becomes unrestricted. The text is
dealt with as a living organism interacting with the reader and evolving. The
theoretical approach proposed by Kristeva is being put to practice today as a
result of scientific and technological progress, that enables theories
originally appearing as utopian to become feasible in practice.
Nicholas
Schöffer, famous for kinetic and cybernetic sculptures he created during the
1950s and 1960s, maintains that this is metacreation: “We no longer create
work, we create creation… We are able to generate… results… exceeding the
intentions of their creators, in indefinite numbers” [15].
Creation as a
process is not limited to a single product. It entails an entire system, whose
component parts convert into others that further evolve in turn. In his book
Beyond Modern Sculpture, Burnham [16] places cybernetic art within a more
expanded theoretical framework focusing on art’s ultimate purpose, which is to
reproduce life.
Tenhaaf [17]
makes the connection between artificial life and art according to their common
feature: interpretation. Both activities build up associative stories based
upon their “chaotic environment”. Interpretation and representation are
features that both art and artificial life have in common. Tenhaaf supports
that both artificial life and art project metaphorical notions. Artists often
recognize artificial life techniques as innately representative and use them as
such. The notion of representation includes and entails the notion of metaphor
and therefore that of interpretation. Is there a correlation between the
interpretation of an artwork and scientific results? Is it possible to set
objective criteria for the evaluation of an artwork or subjective and arbitrary
criteria for the assessment of scientific-purpose simulation?
5. Creating generative
narrative
5.1
About cinematic language
For the
creation of a cinematic work both the creator and the viewer make use of
cinematic language. Encoding and decoding processes are both governed by the
same rules. As to the planning and making of a film, the director uses a group
of cultural (daily habits, social conduct, etc.) or purely cinematic (montage
and rhythm) codes, with the aim of producing
meaning.
Being
experienced in reading such a format, the viewer watches, understands and
interprets, according to her own personal subjective criteria [18], the work
displayed [19]. The same language is used in both writing and reading,
irrespective of the “author’s” or the “reader’s” ability to manage and process them.
With reference
to the generative narrative production system that this project is developing,
the output of such a system will also make use of cinematic language. The
viewer is once more faced with a linear audiovisual narrative, which she
decodes according to the terms already known to her from cinema. However, the
composition of the play follows entirely different paths. The role of the
director lies precisely in the organization and the plan of the composition
process. Hence she disregards the cinematic codes of montage and rhythm and
attempts to attribute objective qualities to the properties of isolated
footage, in order to divide them into groups/categories.
The creator
organizes the database including the audiovisual materials that are to be used
for the construction of the final product and also builds up the software
mechanism, which will process the composition of the database constituent
parts. Therefore, the language used is a programming language. The rules
governing the software in question take into account certain restrictions
linked to the way we conceptualize
cinema and related to montage theories[1].
However, the language used to phrase the specific instructions of the software
operation is purely a programming language. Human-machine communication through
a language “understood” by both entities, is translated into a cinematic
narrative that presupposes the ability of linear audiovisual narrative reading
so as to be read by the viewer.
5.2
Director involvement in the process
In a cinematic
play the director is actively involved throughout the construction process: he
works himself and he also plays the role of coordinator between his associates
so that the final outcome be as close as possible to the concept she initially
had in mind with respect to aesthetics, narrative and emotions. The director’s
job ends at the point where the film is screened. (Figure 1)
Figure 1: Director’s work comparing to the audiovisual
product.
In a
“cinematic” work produced through genetic algorithms, the director’s role is
restricted to the shooting and organization of the footage as well as the
creation and/or adaptation of the software mechanism of production. Editing is
automatically conducted by the system. The role of the director is limited to
the “organization” of material so that the computation system can begin and
complete the process of narrative composition, without the need of any further
assistance by the creator. More precisely, the role of the “director” should be
appointed to the system, for the creator/organizer provides it with the
resources required for the execution of this task.
5.3
The number of outcoming results
The process of
a cinematic film production, usually results in the creation of a single
product. No matter how many times it is screened, the film remains the same. In
the system under study, the narrative products that could be built, as a result
of the same work functioning, may be countless. Every time the viewer commands
the system to start screening, it starts editing the database elements anew.
Given the fact that the process is being conducted by genetic algorithms, which
constantly alter the produced outcome, the composition of shot is unforeseeable
and so is the number of the potential results. Due to the high complexity of
the whole process, there is practically no chance of two outputs being the
same. (Figure 2)
Figure 2: The number of outcoming results.
In an
evolutionary cinematic system, the creator functions as a driving force, a
stimulus of a process that goes beyond the scope of the creator’s imagination
and may acquire unpredictable forms, in compliance with strict and specific
rules. The designer of such a system examines the potential, the limitations
and the power of the rules she establishes. She also monitors the formation of
the rules she has set.
Systems that
“imitate” natural selection processes for the evolution of an entity, set
strict rules for the control of the evolutionary process. The results after
each stage of evolution may be unpredictable for the constructor but the rules
remain unaltered. Computations applied to the management of the genome by
transforming it, altering the genotype or handling user interaction remain
unaffected by the evolutionary processes [20].
6. Future work
After the
creation of the generative narrative production system, an experimental process
will be conducted, which will involve the viewing of the system’s functionality
and cinematic production by a considerable number of viewers for assessing the
process and its outcome, by qualitative and quantitative methods of research.
The results of such an evaluation will influence the theoretical part of this
research project.
References
1.
Klee P., “Lecture of Tuesday, 9 January 1924,” in The Nature of
Nature, 269 in Whitelaw (2004), p. 15.
2.
Theories that illustrate such an
approach were coined in the 1960s, such as Barthes’ and Kristeva’s, concerning
the relationships between author – writing and reader – reading.
3.
Barthes R. (1984), «La mort de
l'auteur », in Le Bruissement de la langue, Paris, Le Seuil,
first published in 1968, p. 69.
4.
In the sense that the reader has
acquired an active role in the reading process.
5.
Kristeva J. (1970) «Le
texte du roman », La Haye/Paris: Mouton.
6.
Hakola
M.,“Hypermontage”
<http://www.kromaproductions.net/HYPERMONTAGE.htm>.
7.
Within a film database, the user
inputs key elements that characterize the film she wishes to watch and is then
asked to choose from a list of films that share the feature or features
inserted as information in the classification system.
8.
Manovich L. (2001), “Database as a
Symbolic Form”
<http://transcriptions.english.ucsb.edu/archive/courses/warner/english197/Schedule_files/
Manovich/Database_as_symbolic_form.htm>
9.
Adami C. (1998), “Introduction to
Artificial Life”, Berlin: Springer-Verlag.
10.
Langton C., (1989),
<http://www.vieartificielle.com>.
11.
Johnson C. G. (2003), «Exploring
Sound-Space with Interactive Genetic Algorithms», in Leonardo,
Volume 36, Issue 1, MIT
Press.
12.
In this case the term “phenomenal”
stands for apparent, visible.
13.
Whitelaw
M. (2004), “Metacreation Art and Artificial Life”, The MIT Press, p. 215.
14.
Kristeva J. (1974), «La Révolution du
langage poétique : l'avant-garde à la fin du XIXe siècle», Lautréamont et
Mallarmé, Paris : Éditions du Seuil, (2 éd.1985).
15.
Schöffer N., “Sonic and Visual
Structures: Theory and Experiment.”
<http://www.olats.org/schoffer/savs2.htm>.
16.
Burnham J. (1968), “Beyond Modern
Sculpture: The Effects of Science and Technology on the Sculpture of This
Century”, London: Penguin.
17.
Tenhaaf N. (1998), “As Art Is
Lifelike: Evolution, Art and the Readymade,” Leonardo 31, no. 5, in Whitelaw,
M. (2004), p. 184.
18.
Metz C. (1969), “Coeur et ame de
l’image”, in Media, INRP.
19.
Christian Metz strictly points out
that “the image is not polysemic but the viewer is”.
20.
Whitelaw
M. (2004), “Metacreation Art and Artificial Life”, The MIT Press, p. 220.