Investigating Composers’ Perceptions and Style through Musical Composition Games

M. Nyssim Lefford, PhD, Research Scientist

Interactive Institute, Sonic Studio, Acusticum 4, 941 28  Piteå, Sweden

http://www.tii.se/sonic

nyssim.lefford@tii.se

Abstract

Our models of the generative process rely heavily on the analysis and deconstruction of artifacts, on completed musical compositions. However, creators’ and non-creators’ perspectives of musical structure differ. Non-creators do not generally have access to the constraints and intentions that shape the composition process. Relying on experimental observation, we have conducted a study that compares the generative decisions, perceptions and preferences of creators. These experiments focus on creators’ in situ definitions of rhythmic, musical structure, and suggest new ways that the generative process may be modelled.

Our data is collected through musical composition “games”. Through games, non-creators can observe generative strategy and share a context with creators. In the music composition games, subject-composers faced similar constraints and degrees of freedom in the composing task making their responses comparable. The game format allows us to search for correlations between the responses of each subject and across experiments with varied constraints. Comparisons across creators and across generative contexts help us identify aspects of the generative processes that distinguish or unify composers.

Through the music composition games, we investigate the nature of generative strategizing, and observed different “styles” of perceiving similarity in musical structure among creators. The patterns produced in these games and the findings derived from observing how the games are played elucidate the roles of metric inference, preference and the perception of similarity in the generative process; lead us to a representation of generative decision tied to a creator’s perception of structure; and suggest new ways to model the generative process and the styles of individual creators through measuring perceptions.

1. Perception and Structure

Creators’ and non-creators’ perspectives of musical structure differ. The patterns creators generate are intended to have particular features and particular relationships to other patterns. Patterns and intentions do not emerge from an infinite set of possibilities, but through constraints. Regardless of what was intended, the resulting musical structures can be perceived many different ways by non-creators. Non-creators do not generally have access to the constraints and intentions that shape creators’ perceptions and the generative process of composing, and our models of the generative process suffer from the resulting distorted view. We have conducted a study to explore composers’ in situ perceptions of musical structure. [1] The following discussion begins with a generalized investigation into perception, intention and style in artistic artifacts, and then moves into music specific explorations.

Artistic intention is revealed through non-causal, non-incidental features. A work of art contains many causal features that are simply by-products of working in that medium. Photography, for example, uses real objects as subjects. Regardless of subsequent manipulation, real objects provide the basis for representation in this medium, and real objects bring certain non-intentional features to the work as do the camera’s technical constraints like focus or field of view. By contrast, these causal properties do not exist in painting and drawing. Distinguishing causal features from intentions are essential for understanding the generative process and the mannerisms and styles of individual creators. [2]

As non-creators, our analysis of artistic intention rests in part on our ability to understand the context in which the creator creates - the constraints, variables, and degrees of freedom available in the moment of production. Any instance of perception, any view of structure within a medium, is filtered through a particular context. The context is all the interrelated conditions under which an object, or in this case an artistic gesture, exists. While non-creators may appreciate similar contexts, we cannot be certain we are sharing a context with the creator. Our understanding of the creators’ intentions is at best skewed. As non-creators observing completed works of art we can only infer the artist intention. We do not know if we define structure within those works similarly.

Music and Art theory provide a basis for defining structure, but theorists examine artistic artifacts not artistic processes. The conventions of Western tonal harmony or formal features attributed to artistic style by art theorists are ascribed a posteriori to the act of creation. In the visual arts, architects Habraken and Gross suggest, “how we use ‘designing’ points not to an object of design, but to a process.” [3] When we describe the generative process by relying on theoretical definitions of structure we squash the artists’ intention or ignore it completely. We fail to recognize what subsets of features are significant to a creator at the moment of creation, and we do not fully understand the intended function of each pattern or structural relationship. These pertinent points of reference, the filters through which creators perceive, leave an indelible mark on creators’ mannerisms and personal styles of expression.

2. Intention, Strategy and Style

Is it possible to recognize and characterize an artist’s intention? Intention is executed through some artistic strategy that produces desired features or structures within a medium. As non-creators we cannot with certainty identify the intended features; but we can observe creators’ actions, and therefore, strategy in action. When a draughtsman draws a line he intends to draw the line in a particular way and perceives it relative to other lines he has drawn or intends to draw. The craft of art forgery, that is the generation of works in the styles of established artists, provides a highly instructive inlet into intention and the generative process. The able forger possesses great insight into artistic strategy. Notorious forger Eric Hebborn (1934-1996) observed the distinction between intended marks and copies in the preparatory drawings of engravers. Although engravers’ drawings are sometimes mistaken for preparatory studies by the original artists, Hebborn notes,

“All but the very best of these engravers’ drawings can be distinguished from the original productions by a certain lifelessness in the line. Every line is meticulously copied, but in the process something of the spontaneous touch of the creative draughtsman is lost. The reproductive engraver does not as a rule really know how to draw, and can therefore only produce the outward appearance of the lines…” [4]

Hebborn attributed this difference to the speed with which a line was drawn. The engraver, by necessity, works much more slowly than the original artist. The spontaneity to which Hebborn responds is the product of more than the speed, technique and trajectory of the charcoal pencil as a line is drawn by the artist. An artist draws each line with particular speed, technique and trajectory because he intends something by drawing the line in this way. There is a strategy and execution. The speed of the engraver’s stylus betrays the authenticity of the work in two ways. Not only are the traces overtly methodical leaving telltale signs of studied pressure variations, but they also divulge an unnatural languidness and a misappropriated attention in the aesthetic decision it attempts to imitate. The engravers and the original artists do not share a similar context. Furthermore, they do not perceive or define structure in either the original or the copy similarly. Successful forgers such as Hebborn can intuit the original artists’ context and perceptions. Additionally, given similar constraints Hebborn could strategize similarly to produce comparable features and results without laboriously copying existing patterns. Still, Hebborn worked by intuition, and what is needed to better understand and model the generative processes is a more empirical approach.

3. Style and Structure

Forgery is not a craft generally associated with music. In music research, a more conventional way to study generative process is to analyze and deconstruct compositions, and subsequently synthesize via artificial or computational means compositions that resemble those analyzed. Unlike forgery, this approach is somewhat removed from artistic practice and has been primarily the province of music theorists. It has provided a sophisticated understanding of similarity in structure across works, and enumerated the variety of ways structure can be perceived in these works. Validation for modelling and synthesis technique comes from comparing features between real and artificially synthesized artifacts, for example, in the work of composer David Cope. [5] Based on an exhaustive analysis of a catalog of compositions, Cope created several music systems that generate pieces in the styles of famous composers. Cope’s work provides us with ample structural analysis of the original artifacts, and a tremendous body of research about the computational modelling of musical structure and style. Nevertheless, it leaves us with unsatisfying answers about the human processes behind the generation of these artworks.

Detailed statistical analysis of artifact features such as intervallic relationships in the melodies of a particular composer are far more involved than any observations made by a human composer for the purposes of his or her own generative process, and lack the “spontaneity” Hebborn described. Additionally, the deconstruction process on part of the non-creator (and model builder) unavoidably emphasizes his or her own preferences for structural analysis. Such choices may make for compelling generative art, but limit our perspective on the generative process and how to model it. Our study attempts to move away from theoretical definitions of structures. But the further away we move from common knowledge, the less knowledge we share with the creator. As a result, we must construct experimental scenarios in which creators can reveal information about their perceptions and the saliencies of features within the patterns they construct to non-creators.

How can the non-creator gain access to the perceptions of the creator? The primary objective of our study was to tie a generative decision - in this case the creation of a musical pattern - to some measure of the creator’s perceptions. Since we as non-creators are not privy to all the perceptions and constraints influencing a composer, this coupling of perceptions to generated pattern can help us to understand the influence of perception and constraint on the generative process. In particular, we will look at composers’ strategies for conceiving of structure, their preferences for particular combinations of patterns, and their perceptions of similarity between patterns. Are there styles of perceiving, and do the intuitions of individual creators yield a distinct style that can be characterized? We will attempt to probe these questions through a series of perceptual and cognitive experiments that use musical games to collect information about the generative process in music.

4. Games

The process of generating music may be compared to playing a game with goals, constraints, rules and strategies. Games can serve as a model for the interrelated mechanisms of music creation to some extent, but also provide a format for an experimental technique. Games offer one way for creators and non-creators to share, at least in part, a context. Games are relatively well-bounded in that a set of constraints and objectives are pre-defined while outcomes remain variable. We have devised musical composition games in which the context, constraints and degrees of freedom available to creators is fixed and well-defined. In these well-bounded composition games, creators generate musical patterns.

Just as observers can analyze the actions of game players, we will attempt to analyze the actions of creators within composition games building on the knowledge of the constraints and goals that we share with the creator. The outcomes of games can be compared by looking at the outcomes of different responses to the same game and responses to games with varying constraints or goals. Patterns produced through composition games can be compared because the circumstances through which they were generated are comparable. This approach for analyzing the generative process has been used previously in Visual Design research. [6, 7] We have adapted this technique for musical settings, and also for isolating specific aspects of perception.

Our composition games are simple, sparse and rhythmic. We hope that simplicity will bring more clarity to some of the fundamental aspects of the generative process in music. In the next sections, we survey three different types of musical composition game, and report preliminary findings in each case. All the composition games are played on a computer using a graphical interface, and all the subjects in this study play an instrument or sing. They varied in age, gender, musical education and experience, instrument played, and genre preference.

5. Strategy

The objective of the first of the three game-experiments was to see if similar constraints would yield similar patterns, or similar strategies for generating patterns. Twenty-one subject-composers made looping, rhythmic patterns for us with simplistic sound samples that we provided. Samples consisted of a single onset of one timbre pitched to a C or G. In all the experiments, to create the patterns, subjects dragged and dropped sound samples on a graphical interface. The samples were seen by subjects as numbered, blocks, with no musical notation or waveform. Subjects could not modify the samples, only the times at which they occurred, and there were some constraints on the intervals of silence between samples. Some subjects were given via text target features to incorporate into their newly, generated patterns. Subjects were also given a questionnaire and asked to describe in writing their strategies for structuring the patterns the way they did.

An interesting thing appeared in the results. Although subjects generated very different patterns, we observed some similar strategies among creators for generating and describing structure in these patterns. Subjects often utilized objects external to the pattern that they were generating as structural templates. Subjects would then conceive of and perceive an overlap of features between this object and the pattern’s features. These objects were often things like events, objects from the physical world and occasionally linguistic structures. As common, was a strategy that involved defining some internal relationship between sound objects contained within the pattern, for example, transformations, spatial or temporal relationships, or occasionally hierarchical relationships. These structural templates appeared to be the basis for generating structure. The next two game experiments take a far more bottom-up approach, and look at preference and the perception of similarity. Do musical creators share similar ways of perceiving structure?

6. Preference and Similarity

To collect responses that can be measured more consistently across subjects, for the next two experiments we used a set of nine rhythms (figure 1) as our set of musical samples. This further reduced the degrees of freedom available and the variation between patterns produced. Subjects added to or organized these short patterns. Henceforth, these will be referred to as the “nine.” The nine, rhythmic samples are all two seconds. Intervals of silence at the end of the samples are preserved when they are concatenated together. Each sample has four attacks or sound events of one percussive, instrument timbre. There are no accents or alterations in pitch within each sample.

Because we are looking for similar perceptions across creators, and subsequently to better map their individual perceptions to the patterns they generate, we used a preference and a similarity test using these nine rhythm samples to establish a baseline for each subject and look for trends in the population. These were perceptual tests rather than composition games. We will use the results of the baseline in the analysis of the patterns produced in the games. Fourteen subjects participated in the baseline.

1	·	·	-	-	·	·	-	-
2	·	·	-	-	·	-	·	-
3	·	·	-	-	·	-	-	·
4	·	-	·	-	·	·	-	-
5	·	-	·	-	·	-	·	-
6	·	-	·	-	·	-	-	·
7	·	-	-	·	·	·	-	-
8	·	-	-	·	·	-	·	-
9	·	-	-	·	·	-	-	·

Figure 1: The nine, rhythmic samples used in experiments 2 and 3.

7. Preference Baseline Test

The first baseline test is for preference. The nine rhythm samples were combined into eighty-one pairs. Subjects were told to think of the pairs as antecedent-consequence or call and response phrases. The antecedents and consequences were pitched a fourth apart. They were asked to perform a total ranking of all eighty-one pairs. Subjects ranked the pairs for preference by clicking and dragging samples into preference order on the computer screen.

Figure 2: Preference Baseline Test

The results revealed a tremendous amount of variation in preferences both within the set of responses for each subject and between subjects. However, some trends did emerge. There were some antecedents for which a significant number of subjects chose the same consequence as the best sounding or most preferred. There were a few instances where subjects who agreed on good antecedent-consequence pair also agreed on poor pairings for the same antecedent. Age, gender, musical experience and education, instruments, and genre preferences were not good predictors of preference ranking in this case. These findings suggest preference and context are changing from antecedent to antecedent.

8. Similarity Baseline Test

In the similarity baseline test, subjects listened to pairs of the nine rhythm samples and were asked to rate each pair for similarity on a scale of zero to ten. Zero indicated that the pairs are not in anyway similar, and ten indicated that they are perceived as identical. These responses were then plotted using multidimensional scaling (MDS). This provides a good visual representation for similarity between patterns and a way to compare subjects’ responses. The temporal position of the first two attacks within the rhythm appeared to be a crucial determining factor for the majority of subjects. The nine can be divided into three groups in which each of the three rhythms have the first two attacks in common. The most basic rhythmic inference subjects can make is to perceive metric subdivisions grouping rhythmic attacks in twos or threes. It might also be assumed that some subjects responded more generally to the features of regularity and density.

As a result, in most of the MDS plots we see the nine rhythms segregated into clusters in which the first two attacks are the same in the all within cluster patterns. But also, three different proximity patterns, or styles of perceiving similarity appeared the results. In type I, clusters of patterns 1,2,3; 4,5,6 and 7,8,9 are distributed in relative isolation from one another. This was the largest group of responses. In type II, two of the three clusters are proximal. This was the smallest category. In type III, the clusters are less distinct and all the patterns are grouped close to each other.

Figure 3: Styles of perceiving similarity. Examples of types I, II and III from left to right (clusters grouped to illustrate proximity)

Again, age, gender, musical experience and education, instruments and genre preferences varied within each category. We used the baseline data to analyze the patterns collected in the next two game experiments.

9. Make An Antecedent Game, Experiment Two

Experiment Two explores the connection between preference and context. In this experiment, the fourteen subjects created an antecedent pattern with four attacks and then ranked the nine, given rhythm samples in order of preference as potential consequences. To make an antecedent, subjects clicked and dragged samples into place (figure 4). These samples consisted of one onset of the same percussive sound used in the given nine consequences. Antecedents and consequences were pitched a fourth apart. There were some constraints on temporal subdivisions of the antecedent pattern, but more than nine rhythmic configurations are possible. The downbeat was required. After creating the antecedent, the nine consequences were shuffled into preference order. Subjects were asked to notate which consequence they thought sounded worst, in other words, which was least preferred as a consequence.

Subjects then made the least preferred consequence pattern the most preferred. On new workspaces, subjects generated new antecedents that made that previously least preferred consequence now sound best or most preferred. All subjects could make the worst better, but some could not make it best.

Figure 4: Make an Antecedent Game

As a result, we were able to plot out complete transitions from the preferred pattern combination in the first part of the game through configurations where the least preferred improves (figure 5).

Figure 5: Tracking changes in preference from worst to best contexts

Interestingly, some subjects created patterns that work well for both the original and new antecedents. For example, in the chart above (figure 5), the second and third most preferred consequences are the same for both antecedents. However, there did not seem to be a correlation between the kinds of antecedents subjects created and their preferences for antecedent-consequence pairings and/or their perceptions of similarity reflected in the baseline tests. To address that problem, we created one last game experiment.

10. Chain Four Patterns Game, Experiment Three

This game used only the nine rhythm samples from the baseline tests. Ten subjects generated new, rhythmic phrases by chaining four of the nine together. Again, new patterns were generated by clicking and dragging samples into position. Subjects were free to use any of the nine samples more than once. Each subject created four pairs of chains. Chains within a pair were to be similar to each other, but had to begin with a different one of the nine provided rhythm samples. The first pair was to sound regular (whatever that meant to the composer), the second irregular, the third dense and the last sparse. After composing the chains, subjects were asked how many phrases they heard in each chain - in other words, whether they heard these chains as having four phrases, or had some other structure emerged.

Figure 6: Chain Four Patterns Game

We then mapped on each subject’s MDS plot the transitions between the nine given rhythm samples in each chain. What results is an interesting representation of a generative decision superimposed onto similarity space. The newly generated pattern can be directly compared to each subject’s perception of structure.

Figure 7: Plotting transitions across similarity space

These new chains can be divided into three categories: circular chains in which chains begin and end on the same rhythm; repetitive chains in which there is a repeated combination of patterns; and asymmetrical chains in which the first and the last rhythms are different. The majority of chains produced were asymmetrical. Also, new phrase structures emerged even though subjects had been familiarized with the individual, nine rhythms in the baseline. The majority of chains were heard as containing two phrases. Interestingly, subjects with different MDS plots, or different styles of perceiving similarity, followed similar strategies for constructing chains. Therefore, it is likely we are witnessing the influence of some higher-level criteria. In future, by further refining the constraints and limiting degrees of freedom, we can further refine the connections between perceptions and generated patterns.

11. Conclusions

Through the game experiments, we were able to compare creators’ perceptions of structure and the patterns they generated. These preliminary results revealed three different styles of perceiving structure. Furthermore, we were able to track changes in individual creators’ preferences as they generated patterns. We offered a representation of a generative decision mapped onto similarity space. Obviously, much more research is necessary to understand strategy, preference and similarity in even this sparse generative context. However, there are other form-bearing dimensions to consider, and we need to expand beyond preference and similarity and develop more complicated games.

Nonetheless, these preliminary findings hold implications for future simulations and syntheses of musical patterns with the mannerisms of particular creators. We offer not only new ways that the generative process may be observed, but also new types of representations for creators’ perceptions. These maybe used to model the personal style and mannerisms of individual artists. The game-experiments provide insight into characteristics that correspond to the idiosyncrasies that distinguish creators as well as to the similarities they share. Also, elaboration on the analogy between games and the generative process and generative strategizing presents further potential for new models. Thus we, the modeller, are better equipped to face the challenges of synthesizing patterns with stylistically salient features.

References

[1] Lefford, N. (2005) The Structure, Perception and Generation of Musical Patterns. PhD thesis. Cambridge, MA: Massachusetts Institute of Technology.

[2] Mitchell, W.J. (1992) The Reconfigured Eye: Visual Truth in the Post-Photographic Era. Cambridge, Massachusetts: MIT Press. In particular Roger Scruton’s argument about intentional and causal components in photography. p. 29-30

[3] Habraken, N.J. and Gross, M., [et al.] (1987) Concept Design Games: A Report Submitted to The National Science Foundation Engineering Directorate, Design Methodology Program. Cambridge, MA: Dept. of Architecture, Massachusetts Institute of Technology. p. 2-10

[4] Hebborn, E. (1991) Drawn to Trouble: The Forging of an Artist: An Autobiography. Edinburgh : Mainstream. p. 213

[5] Cope, D. (1991) Computers and Musical Style. Madison, WI: A-R Editions. Also Cope, D. (2001) Virtual Music: Computer Synthesis of Musical Style. Cambridge, MA: MIT Press.

[6] Habraken, N.J. and Gross, M. (1987)

[7] Iversen, O. and Buur, J. Design is a Game: Design Competence in a Game Setting. Preprint. Habraken and Gross and Iversen and Buur are not the only ones to utilize this design game technique, but they had a notable influence on this study.