Selective Extraction of Point Sets from Photographs as Starting Events of Generative-Art Line Drawings
Kunsthochschule Kassel, Generative Design Lab, University of Kassel, Germany.
Advanced Computing Center for the Arts and Design (ACCAD), The Ohio State University, Columbus, USA
Assuming a reductionistic position, a drawing is essentially an arrangement of lines and strokes. Whether placed on the paper by the hand of an artist or by a generative algorithm, a drawing emerges in both instances. Nevertheless, it seems reasonable to clearly distinguish between the two production processes: “drawn by the hand of an artist”, and “drawn with an algorithmic generative process”. In each one of these distinct universes, the “Universe of hand drawing” and the “Universe of machine drawing”, infinitely many very unique drawings exist. For the execution of a line, a certain (though not necessary) logic prevails: The line starts at a defined point on the plane and it ends somewhere on the plane. The decision of where to start has to be made first, then the line may “develop”.
In this paper we concentrate on the question of starting points with a focus to generate those from existing photographic images. A procedure is proposed to extract and collect point sets as starting events for generative-art line drawings. A number of strategies are suggested, one of which is implemented using “Max” and “Jitter”, commercially available software used primarily for real-time sound and video processing. The extracted point-sets are then used as inputs to a generative programme. Experimental drawings, following the rule system of this programme have been generated and executed on a pen plotter. The results are encouraging and they suggest interesting aspects for further research.
This project also forms the test bed of our investigations into the use of collaborative software, facilitating joint creation of generative artwork. A number of emerging technologies will be discussed which are enabling the interactive process of rapid generation and evaluation to be globally distributed.
We here consider algorithmic generated artwork based on lines only. Such artwork is very close to what we usually refer to as being a drawing. Our frame of reference for art drawings has quite naturally a focus on drawings made by the hand of an artist. The “universe of hand drawings” is accumulated over the centuries, a very richly populated universe indeed, and its roots reach way back to the early beginnings of art as a manifestation of mankind. The drawing is based on lines and strokes, which, viewed geometrically, start at a particular location in the plane and end somewhere. Between starting point and end point the line displays “character” which may be thought of as a set of features such as soft, curling, thick-thin, straight, light, poetic, violent and many others. The hand of an artist is capable of producing a fantastic large number of such characteristics that we admire by looking at drawings from the “universe of hand drawings”. The (art) drawings we are concerned with here are of a very difficult nature. They all reside in the “universe of machine drawings”, an equally rich and fantastic but only marginally explored universe of its own [1,2].
Artists and programmers have studied a multitude of methods for representing, searching, and culling the space of possible machine drawings. From some of the earliest machine drawing work by artists like Csuri and Cohen [3,4] to more recent approaches by Bhatnagar, Orellana, Moura, and many others , exploring methods of transferring the human ability to create drawings to machines continues to be investigated. Whether any of these approaches (or any possible approach) involves creativity remains an open question [6,7].
Although we consider only some very specific aspects, we do this with the expressive intention of describing a complete generative production process for algorithmically generated artwork, based on lines. This process draws on existing software “Max” and “Jitter”  commercially available and primarily used for real-time sound and video processing, and a generative programme which has evolved over the years (from the early eighties) and has been used since to produce algorithmic generative drawings. In this programme, a particular emphasis is placed on the starting points of lines, the rational being: the starting point of a line has to be placed with great consciousness, because it is from this point where the line “develops” into the plane. The starting points for the line in a drawing have an overriding importance. As with decisions in art, such a statement is based on some “personal logic” and it is contaminated by ideological preconceptions – but nevertheless, it is working as a conceptual frame and, important in this context, it is easily cast into the code of a programme.
Digital photographs are everywhere nowadays – we take them with our mobile phones, with web cams, with small and large digital cameras, and they are all over the web. For this work we filter such photographs to extract point sets of a certain magnitude n (2000) at the moment, which are then used as the starting points for lines in the generative programme. The lines are polygons and are generated with a feature driver, heavily relying on random processes. The generative pipeline for a drawing includes these steps:
(1) Select photograph
(2) Manipulate photograph with Jitter
(3) Decide on filter strategy
(4) Extract point set
(5) Submit point set to generative programme, generate drawing
(6) Plot result on pen plotter
The programme and the development of the cooperative environment (steps 2 – 4) – which we regarded as an issue of highest priority for this project – have been developed in Ohio. The generation of the drawings and the plotting (steps 5 and 6) has been done in Kassel.
For the decision on which filtering method to use, quite a number of strategies, including the following, are available:
(1) Use darkest area of photograph
(2) Use “light” areas only
(3) Use second level criteria
(4) “Thinning out” of pixels
(5) Pick in a range of p%
(6) Use Boolean operations
(7) Focus on a selected area, thin out rest
(8) Find contours, areas of changing density
We have initially focused on the strategy using the darkest areas of a photograph, and we will investigate some of the other strategies in further research efforts. Figure 1 shows the main steps of the process. Figure 2 is showing a detail of the generated plotter drawing.
Fig. 1 Photograph of model of Duomo in Milan (left). Jitter programme and visualization
of the resulting point set (middle). Plotter drawing generated from derived point set (right).
Fig. 2 Detail of the generated plotter drawing
Cycling `74’s Max and Jitter software products are examples of a number of programmes, which historically have allowed data filtering algorithms to be rapidly developed graphically by the interactive creation of data-flow networks. These networks consist of boxes of text and parameters, wired together to represent algorithms (i.e., ordered processing functions, logic, control, etc.) Order, placement, and position of the nodes and lines in these graphs are critical to the execution of the programme. The interface for creating them is not unlike early computer drawing programmes, but the data-flow “drawings” are functional. There is of course something very attractive and elegant about drawing images which can be executed to create drawings. The visual representation of algorithms can provide a more intuitive interface for artists and designers than would a text editor and compiler.
Fig. 3: Data-flow example: a simple “queryImage” sub patch
In the network in the left of Figure 2, a top-level network (or patch) can be functionally decomposed into a number of sub patches with inputs and outputs. These sub patches are encapsulated in the boxes beginning with a p and given a unique name. An example of a simple queryImage sub patch responsible for sampling a picture at a given coordinate is shown in Figure 3. For those unfamiliar, a brief walkthrough of this patch should illuminate the means by which data-flow programming may be conducted.
In the patch in Figure 3, an image is first received and stored as a matrix at the top (jit.matrix). A stream of sample coordinates arrive via the receive object named coord (r coord). Each (x,y) coordinate pair received is prepended with the message getcell and passed to the matrix object. In response to each of these, the matrix object emits a new message consisting of the word cell, followed by a set of seven arguments. When a cell message is detected it is routed to the unpack object, which splits up the list into individual arguments, only the fifth of which is used: the value at the specified matrix coordinate location. This integer is stored in place of the 128 in the > Boolean operator object. A random number in [0,255] is then generated and compared to the stored value. This particular sub patch example can be used to probabilistically create points at dark or light locations. The bottom of the patch can be easily modified to obtain any number of alternative results.
Returning to the primary algorithm, the patch in Figure 2 when initialized loads and displays a user specified image. Users can adjust the image’s brightness and contrast before initiating the point selection process. When the GO button is pressed, points are selected at random and tested for whatever traits are specified by the queryImage sub patch discussed previously. Since the plotting software has a limit of two thousand points per run, this was used as an upper bound for point set generation. Once the required number of satisfactory points has been found, the programme stops. If the exportPoints sub patch has been activated, then each coordinate that “passes” is normalized, scaled, and translated into plotting coordinates, and exported to a text file in the format required by the plotting software.
PU;PU -1565, 4871;
PD -1565, 4871;
PD -1216, 4133;
PD -850, 3712;
PD -2095, 3376;
PD -2292, 2880;
PD -2531, 2113;
PU;PU -5446, 5808;
PD -5446, 5808;
PD -5639, 5987;
PD -5787, 6499;
PD -6627, 7653;
PD -6884, 8029;
PD -7159, 8395;
PU;PU 7000, 187;
PD 7000, 187;
PD 7203, -132;
PD 8189, -422;
PD 6907, -1744;
PD 6859, -2558;
PD 6849, -3490;
PU;PU 7000, -1499;
PD 7000, -1499;
PD 7161, -1324;
PD 7531, -1306;
PD 7854, -916;
PD 8395, -557;
PD 9386, 74;
PU;PU -4509, 3372;
PD -4509, 3372;
PD -5387, 3594;
PD -6098, 1692;
PD -6987, 4549;
PD -7429, 4770;
PD -7804, 4985;
Fig. 4 Data point processing.
A number of transformational steps have to be carried through which end with the plotter drawing. In Fig. 4, the main steps are illustrated by using the respective data format, which result from the transformations. The left column of Fig. 4 shows the (x,y) coordinate-pairs for starting points of polygons extracted from a photograph with Jitter. The column in the middle shows the resulting (x,y) coordinate-pairs for polygon generation (note the separation flags between polygon data), which come from the generative programme. The entries on the right are an example of the HPGL code needed for plotting. For clarity, the first three entries on the left are arranged to correspond with the starting points of the generated polygons. As the process continues, the content and size of the respective files changes. The simple (x,y) coordinate pairs are expanded to represent polygons, starting in the provided locations. This code in turn is expanded by adding the commands required for plotting. Mostly these are Pen Up (PU) or Pen Down (PD) commands for directing the pen, and a few other commands.
As an analog to the prior discussion of starting points and plotting, the genesis of the authors’ collaborative process bears examination. While initially meeting together at a Generative Art conference in Milan (GA2000), it wasn’t until later meetings in Portugal and Los Angeles that a way to work together was agreed upon. In addition to an interest in pursuing the primary generative work described above, there was a strong desire to attempt to explore the practical use of the many emerging technologies becoming commonly available for collaborative work. The results were mixed but illuminating and promising.
Generative art has traditionally been created either by single artist/programmers or occasionally by an artist working with a programmer at the same site. While the tools have existed for some time at a corporate level for remote collaborative work, and at times small groups of artists have been able to briefly make use of such high-end equipment, it is only recently that advances in commonly available, inexpensive software and hardware have made it possible for individual artists to practically investigate tools for working together on a global level, as if they were next door from one another.
Software like SubEthaEdit , Wikis , and iChat AV  are quickly approaching the point at which groups of individuals can set up remote collaborations without requiring technical support teams. High quality full screen international video conferencing is now available to anyone with a computer and a broadband connection. The authors used Apple’s iChat AV with iSight Firewire cameras on Powerbooks to communicate at will between Germany and the United States with great success. Work was conducted for arbitrary lengths of time, with cameras being manually redirected to show work in progress, interfaces, explain new software, show and diagnose problems, etc. The next generation of this video conferencing software to be released in a few months will support shared connections between up to four users, which should further increase collaborative abilities.
Collaborative document editing tools are slowly becoming available. SubEthaEdit is one of the more popular ones of the past couple years. It is commonly used among co-located groups to jointly take notes, collaboratively authoring text documents in real time. On the same network there is zero configuration required. More recent versions support internet-wide document editing.
Wikis have the advantages of being plain text, accessible with any web browser, and being open to anyone with no user configuration. This advantage can be a disadvantage of course if security is required, but malicious changes are trivial to undo by the first person that spots them. A Recent Changes log typically makes this simple. There is a bit of a learning curve for creating new pages, links, images, etc., but it is significantly simpler than learning HTML and web page authoring (or a new word processor, for that matter.)
Finally, by using tools like Max that support drag and drop networking, assembling arbitrary networked application construction has become possible for non-programmers. Dancers, musicians, and artists have for years now been able to build interfaces in which button and slider events are transmitted globally.
Our goal in using these tools is the joint creation of generative artwork. We are working to interactively generate and evaluate results at multiple locations, globally distributing the creative effort as well as artistic and technical expertise. Currently we are designing and developing a system architecture in which remote and local viewers can control the display of remote views of collaborators, their desktops, and their environments.
The most significant problem facing remote collaboration is related to security, but not in the usual way. As levels of spam, worms, spyware and viruses increase, system administrators are increasingly locking down their networks. Perhaps paradoxically, while a relatively insecure site will have little problem setting up and using remote collaborative tools, a site with better technical support will require more technical support. Negotiations and justification of which network ports, of what type, in whose firewalls, need to be opened to whom, can greatly complicate remote cooperation. SubEthaEdit, iChat AV, and Max remote networking each gave us problems of this nature to varying degrees. While video conferencing was easily resolved, two-way document editing via SubEthaEdit and Max-based collaborative image editing were not. Using a wiki provided the least amount of difficulties in this vein.
An open question also remains of how the initiation of such collaborations might be encouraged using collaborative emerging technologies. This current effort was born the “old fashioned” way: via dinners, talks, and meetings at international conferences.
It is not an uncommon position among Computer Scientists to believe software is “rotting” and becoming obsolete very fast. The mix of “old” and “new” programmes, hardware, systems and programming languages used in the approach discussed here is strange enough to warrant short reflection. Some of the “older” code used, like HPGL 1, draws its power from the simplicity of the concepts behind it. With a few basic commands, code to produce line-oriented plots can be written easily. It is so simple that we can produce a running programme just by applying the “find and replace” function of a text editor to a file of x,y coordinate-pairs. Programmes generating HPGL code can be written in many programming languages. To solve some of the generative issues in the development of the work reported here, the programming language MaxSPITBOL, based on concepts dating back to the late 60th of last century, has been used. It is still running perfectly on new Macintoshes, and it is one of the cases where the “rotting” in fact does not exist at all. Instead it turns into one of the fabulous history stories in Computer Science of consistency and longevity. In the beginning of the procedural chain we use Jitter, a quite modern tool, and at the very end the plotting code. The final generative aspects are running entirely on an old Apollo and the programme has been written in Fortran IV. And this constellation has been producing drawings for many years .
Assuming, the starting points of lines are of particular importance when a drawing is generated, and assuming the drawing generator simply accepts supplied starting points, not producing them itself, it makes sense to develop strategies for generating such sets of starting points. It is suggested to use digital images (photographs) and extract sets of data points (according to criteria) from them. A procedural chain has been designed, programmed and applied, which will filter and transform those data for input to the generative programme which will produce a pen plotter drawing as a piece of art, standing in its own right. The results from these experiments are encouraging. The drawings generated using this approach have a strong, specific and consistent “character”.
The results of this effort have been produced in a computer supported collaborative work process. We plan to continue exploring collaborative software including shared desktop components, remote shared control and viewing of software and hardware, and practical solutions to networking challenges. Within generative art, the central topic of this paper, a high demand for instrumental knowledge in quite disparate fields like Computer Science, Art, Media and many others is common. Collaborative work makes a lot of sense in this area. High-end collaboration solutions exist but we desire artist accessibility for global collaborative tools without requiring advanced technical support. The creation of generative art can be collaborative across time, mixing old and new technology, and across space, bringing together the different backgrounds and expertise of partners living distant from each other.
 Dehlinger, H.; Qi, Dongxu "Art Experiments and Mathematical Explorations into the Universe of Machine Generated Drawings", Proceedings of the International Conference on Electronic Arts, Chicago, ISEA 1997.
 "Bildnerische Experimente mit der geplotteten Linie" in: Bauer, Dehlinger, Mathias; "Design, Kunst, Computer: Computer in den künstlerischen Bereichen", Verlag Jenior und Preßler, Kassel, Okt. 1991, ISBN 3-928172-07-7
 Csurivision Ltd. “The Art and Ideas of Charles Csuri,” http://www.csurivision.com/csuri/, 2004.
 Harold Cohen. http://crca.ucsd.edu/~hcohen/, 2004.
 Artbots: The Robot Talent Show. http://artbots.org, 2004.
 Margaret Boden. The Creative Mind: Myths and Mechanisms. Basic Books, 1991.
 Douglas Hofstadter. Fluid Concepts and Creative Analogies. HarperCollins, 1996.
 Cycling `74. Max/MSP/Jitter. http://www.cycling74.com, 2004.
 TheCodingMonkeys. SubEthaEdit. http://www.codingmonkeys.de/subethaedit/, 2004.
 “What is Wiki.” http://wiki.org/wiki.cgi?WhatIsWiki, 2004.
 Apple Computer, Inc. iChat AV. http://www.apple.com/ichat/, 2004.
 Hans Dehlinger Zeichnungen, kassel university press GmbH, ISBN 3-89958-534-8, Kassel 2004, www.upress.uni-kassel.de
 MaxSPITBOL was designed and implemented by Mark B. Emmer in the late 80th and is based on the Macro SPITBOL system of Robert B.K. Dewar. SPITBOL, Speedy ImplemenTation of SnoBOL. Snobol4 is a programming language from Bell Laboratories designed for pattern matching and all non-numeric computation.