Details

This talk was presented at Community Research Seminar event on AI and music at the Mellon Sawyer Seminar “Histories of AI: A Genealogy of Power” at the University of Cambridge (virtually) on 20 November 2020.

Slides

A PDF file with the slides for this presentation are available for download here (20.5 MB).

Abstract

Around 1960, Walter Reitman of the Complex Information Processing group at the Carnegie Institute of Technology (now Carnegie Mellon University) made tape recordings with his co-investigator Marta Sánchez ‘thinking aloud’, as an unnamed experimetnal subject composed a fugue at the piano keyboard. Reitman used protocol analysis to mine the 150-page transcript of this recording, seeking design inspiration for a new computer model of ‘human information-processing’—Argus—which was intended to complement the then-recent work of his colleagues Herbert Simon and Allen Newell on the General Problem Solver. I relate and contextualise this unusual historical case, which shows how Western art music composition was used in the experimental systems of early 1960s AI research as a proxy for so-called ‘ill-defined problems’ and as an apodeictic demonstration of supposed algorithmic creativity. With the release of the Google ‘Bach doodle’ in March 2019, little appears to have changed in how high culture is mobilised in the rhetoric that surrounds AI systems.

Talk outline

In this talk, I’m going to speak about some early 1960s AI-adjacent research that was carried out by Walter Reitman and Marta Sánchez, two researchers at the Carnegie Institute of Technology in Pittsburgh, PA. Together, Reitman and Sánchez explored how the particular case of music composition might yield insights into how human beings solved problems in general. Because Reitman conducted his research in the context of the Cognitive Information Processing Group at Carnegie Tech between 1957 and about 1964, he used contemporary computational and information-processing techniques to both articulate his research question and to express his results. Very coarsely stated, these researchers believed that human cognition could be usefully understood by designing, implementing, and analyzing computer simulations of intelligent behavior. On Reitman’s view, his collaborative work with Sánchez revealed the limitations of a then-state-of-the-art problem-solving software tool: Herbert Simon and Allen Newell’s General Problem Solver. Their cognitive-psychological study of music composition led to the creation of a new problem-solving system named Argus, which differed from Simon and Newell’s General Problem Solver in significant ways. In Mind as Machine, Boden briefly describes the Argus system but has little to say about either its musical connections or Sánchez’s intellectual contribution to the project.¹

The question I’ll explore today is: if, as Reitman claims, new computer models of human cognition informed by study of music composition could be lodged as an internal critique of prior results within the field of information-processing research, then precisely what kinds of musical experiences and concepts figured in this critique? Both musical high culture in general, and, as I will describe, particular cultural forms that are geographically and historically specific to 17th-century European art music, were mobilised by these early cognitive information-processing researchers. And, as I will briefly indicate at the end, very similar cultural forms continue to do so today. Both the immanent vocabulary native to these music-cultural forms—in particular, fugues and chorales—as well as their relationships to specific pedagogical regimes, and attendant notions of spontaneity, genius, and creativity, informed the research questions that Sanchez and Reitman investigated in their collaborative work and, indeed, the solutions that they found thinkable in that context.

---

My interest in Sánchez and Reitman’s collaboration was stoked by an unusual document in the Herbert Simon Collection at the Carnegie Mellon University Libraries. The Simon collection contains draft and final copies of many papers in the working paper series of the “Cognitive Information Processing” Group at Carnegie Tech, where Simon, Newell and others who had been previously affiliated with the same RAND-sponsored worked. CIP Working Paper #37 dates from November 1960 and is entitled “The Composition of a Fugue: Protocol and Comments”. A little about the authors. Marta Sánchez was a Chilean pianist, born in 1923. She was raised in Santiago and received her first degree in piano performance and music education there at the University of Chile. Sánchez was then trained in Dalcroze eurythmics, a school of music pedagogy developed in the late 19th century, in which bodily movement in time as a technique for inculcating good musical habits—“eu-rhythmics”—in students, often from an early age. Its founder, Émile Jacques-Dalcroze emphasized the value of kinesthetic sensibility to music, and as such, worked to de-emphasize written, theoretical representations of music in the pursuit of musical sensitivity. After moving to the United States she began teaching eurythmics at the Carnegie Institute in 1957, where she remained for over 40 years. Sánchez would go on to complete a doctoral degree in musicology at the University of Pittsburgh in 1978.

Reitman, who completed his undergraduate degree at Harvard and did graduate research at Wesleyan and the University of Michigan, joined the faculty as an assistant professor of Industrial Administration and Pyschology at Carnegie Tech in 1957. There Reitman taught courses in psychology there and developed a course notes into the text-book “Cognition and Thought: An Information-Processing Approach”, published by Wiley in 1965.² In this book, Reitman provided an erudite summary of the nascent field of cognitive science and introduced the basic concepts of an information-processing account of human cognition to a non-programming audience, effectively making a case for computer programming as a tool for psychological research into human behaviors such as planning and problem-solving. In the introduction to this book, whose final chapters introduce state-of-the-art models of human cognition, Reitman acknowledged the influence of his working relationship with Sánchez, “a rare and patient friend to whom I owe much more than I can acknowledge.”³ Reitman’s most extensive discussion of the fugue-composition protocol comes in a chapter of this book, which is entitled “Creative Problem Solving: Notes from an Autobiography of a Fugue”, and I’ll draw on this chapter later. Later, Reitman would join the faculty at Rennslaer Polytechnic Institute and, notably, founded the quarterly academic journal Cognitive Pyschology in 1970.⁴

Together, Sánchez and Reitman supervised the preparation of the 150-page transcript that I now describe. This working paper documents the composition of a fugue in an atonal style. One musical-technical point that has to be made before I do: a fugue is a particular kind of musical composition dating from at least the 17th-century that must meet certain music-structural criteria. The basic compositional operation that animates the fugue is imitation: the simultaneous statement of recognizably similar musical statements that explore the emergent qualities of carefully structured melodies when superimposed upon each other. The specific recurring musical idea that becomes the central topic of a particular fugue is called the fugue’s “subject”. Fugue has a long history as a genre of composition that demands a particular kind of musical literacy, and has been used for centuries in pedagogical contexts to train and assess students in music composition.⁵ The fugue-composing sessions took place over at least five distinct recording sessions, with at least one gap lasting a month or longer between them. The fugue-composition protocol, then, is a partial record of a conversation between a piano-playing composer (probably named “Cliff”) and an experimenter, which has been tape-recorded and transcribed more or less verbatim. The vast majority of the words recorded are attributed to the composer, though a number of comments appear to be attributed to the experimenter, who may be either “Nelson”, Reitman, or Sánchez herself. It is probable that more than one person played the role of experimenter.

The page excerpted here (from page 10 of the transcript) is representative. The document is especially striking because it freely mixes verbal commentary and musical notation, using Western music-notational conventions to record the actions of the performer-composer as they approached the compositional task. Staff notation examples illustrate each of the musical options explored by the composer, as they build towards the final version of the fugue, which is reproduced in the document as an appendix. In its dense mixture of notation and transcribed speech, the various inscriptions that make up protocol study document lend it the character of the classical “boundary object”, as it was originally formulated by Star and Grisemer.⁶ The preparation of such documents is not without precedent in the field of music, especially in small-cohort studies of musical improvisation since the 1980s; usually, however, raw transcripts remain unpublished and are not discussed at length.⁷ In this excerpt, the composer is near the start of the composition and is trying to write what he calls a “counter melody”, a musical line that will complement the fugue’s subject according to the rules of counterpoint but should nevertheless retain for itself a distinct musical identity. He tries “[c]arrying out the idea of enlarged syncopation” and notes immediately, “that wont work, because it’s repetitive of the theme itself.”⁸

In the chapter in which he commented on the transcript, Reitman singled out the same passage: he explained how the intermediate candidate solution for a subpart of the problem is rejected by the solver because it is too similar to the theme as it has been decided upon by the composer earlier in the compositional process. Reitman would attend to this phenomenon—whereby the current state of the composition leads the experimental subject to announce further—as an example of “constraint proliferation.” The problem of fugue composition, as Reitman operationalized it, seems to pose but one major, admittedly open, constraint: “that the end product be a fugue.”⁹ For Reitman, the value of the fugue-composition protocol study was that it how many different types of constraints could emerge in the solution of a problem, a feature of so called “ill-defined problems” as loosely specified as the composition of a fugue, as we will later see.

Sometimes, longer discussions between the composer and the experimenter are recorded in the transcript. One of a handful of such tangential discussions is especially revealing, as it discloses how the multiple meanings of the same word—all instances of a musical “movement” metaphor—cause obvious problems for the logocentric experimental setup. It could hardly have escaped Sánchez’s attention, trained as she was in Dalcroze eurythmics, that any record of the composer-pianists bodily movements or gestures was liquidated by the combination of the tape-recording apparatus and the musical and verbal transcription processes.¹⁰ And, from time to time, the experimental subject reflects on the experimental set up itself, complaining that “this is where it becomes hard to work with someone around, or without being able to think freely, having to speak.”¹¹ The composer has clearly been asked to describe their thought process out loud as they make progress with the fugue. So doing would allow Sánchez and Reitman to feed back their experimental subject’s intuitions into their own conceptual model of human problem solving. As Reitman notes, the composer-pianist made several references to writing “pianistically”, which led Reitman to better understand the music-compositional notion of “idiomatic” writing: the idea that certain musical elements are better suited to the affordances of a particular musical instrument. In his commentary on the fugue protocol, Reitman noted that “idiomatic” writing was yet another kind of problem constraint, a feature of the fugue composition task left unstated that only emerges when it is realised in a particular solving context.¹²

This document, then, is a laboratory artifact of the “think aloud” experimental paradigm, an established psychological technique for eliciting problem solving strategies from human subjects with a view to later analysis and decomposition into cognitive subroutines, sometimes to be implemented computationally. Famously, “think aloud” studies had conducted with college sophomores by Newell and Simon to determine the problem-solving strategies for the General Problem Solver. Referring collectively to these “introspective” techniques for gaining insight into human-problem solving strategies, Dick has written—in an aside—that “all introspection is a narrativization—a story telling based on the esoteric experience of one’s self.”¹³ This passing observation does much to capture the attraction of protocol study transcripts as an object of research, since the information they contain is not only “raw data” for the scientific process, but also a window onto the sense-making strategies of particular individuals within the historical context of the early 1960s cognitive science laboratory. Biography, training, expertise, mood and tenacity all converge in such transcripts. Once again their status as boundary objects looms: as lab artifacts, their interpretation is constrained by the norms and conventions of contemporary scientific research; as historical documents, we are freer to weave them into new and unexpected relationships with the benefit of hindsight.

---

The use of “think aloud” data to which the transcript bears witness was only one debt of many that Sánchez and Reitman’s approach to the study of human problem solving had to Allen Newell, J.C. Shaw and Herbert Simon’s work on the Logic Theorist and the later General Problem Solver during the late 1950s and into the early 1960s. Stephanie Dick’s recent work has carefully explicated the key work of of these three men; almost all of what I know of their research program is due to her exemplary research, with the rest filled in by Reitman’s own gloss on their work for the reader of his textbook.¹⁴ To borrow Reitman’s words, “one of the major achievements of [their] […] work was to provide more concrete ways of thinking about how we might represent human problem-solving strategies and problem-solving behavior.”¹⁵ But, as Dick argues, the fact that computational cognitive models of human information-processing of this school were destined to be run on actual—and thus resource-limited—digital computers means that this was no free play of ideas.¹⁶ Not only was the behavior of idealized rational subjects “bounded” by the process of implementation, so too was the explanatory remit of theories of human cognition so materialised.

Thus, despite his acknowledged debt to Newell and Simon’s pioneering work, Reitman’s research output during this period between 1960 and 1965 can be viewed as a gentle if confident critique of later versions of the General Problem Solver. Reitman’s computational modeling efforts in this vein focused on a new information-processing model of problem solving that he called Argus, the first published description of which appears in the journal Behavioral Science in an 1964 article with co-authors Richard Grove and Richard Shoup, all of Carnegie Tech.¹⁷ Unfortunately, I don’t have to time to get into the weeds of precisely what Reitman thought Argus added to the story of information-processing modelling of cognitive structure. However, I’ll mention just one of the key features that Argus had, which the General Problem Solver lacked: its ability to work on “problems” out of sequence. Reitman designed Argus so that several cognitive constructs could be updated at once, passing “activation” values around a semantic network that stored concept representations, acknowledging as inspiration both Hebb’s work on neural assemblies and Oliver Selfridge’s Pandemondium model.¹⁸ Reitman beared down quite heavily on the General Problem Solver’s strict serial processing of problem representations. Reitman noted, by contrast, that “Argus is not at all single-minded, is much less in control of what it remembers and forgets, and also is much more prone to conflict and serendipity. In fact, the assumption that work on a problem may and will be interrupted by the occurrence of interesting and unexpected ideas about other matters is built into the basic structure of the program.”¹⁹

In the 1965 textbook, Reitman summarises neatly the course of events which, in his view, brought him to these aspects of Argus’ design:

Our original aim was to see whether and how [fugue composition] might be brought within the framework of the General Problem Solver [(GPS)]. Our surrogate question was: How might we reproduce and account for the composer’s behavior using GPS as a model of the thinking processes involved? In other words, we used GPS as a tool for studying the behavior we observed in the protocol. We found that there appeared to be important aspects of the behavior we could not encompass within the framework of the early GPS versions with which we worked. The result was the evolution of our own framework [Argus], in many ways derived from the GPS model, but differing from it in some respects. In particular, the effort gave rise to the problem framework presented in Chapter 5 [“Problems”] and the active memory model of problem solving to be presented in Chapter 8 [“Argus: An Information-Processing Model of Thought”].²⁰

The crux of the matter is this: Sanchez and Reitman used the fugue-composition task to mount a critique of a well-established, non-hypothetical information-processing model of human cognition: GPS. This episode shows how certain music-cultural activities, within a particular experimental system, redirect the course of technoscientific research. Most interestingly, it shows an example where early cognitive science was the client discipline of music… and not the other way round, which is so often the case.

It’s important to point out that, according to the actors themselves, it was a feature of the problem itself that precipitated this change in research direction. The task of composing a fugue, writes Reitman, is an “ill-defined problem.”²¹ As Reitman himself points out, the class “ill-defined” problems had its own definitional problems. In 1961, Marvin Minsky had designated “well-defined” problems as those problems for which “we are given some systematic way to decide when a proposed solution is acceptable.”²² He neglected, however, to provide a positive definition of what an “ill-defined” problem was, leaving Reitman to step in to the breach. According to a historical survey of the notion of “ill-defined” problems by the twenty-first century practitioners Lynch et al., Reitman’s book was the first attempt at a systematic definition of this class of problems.²³ Much more could be said about how Reitman’s penchant for illustrative examples from the fugue-composition exercise makes it difficult to disaggregate music-theoretical and cognitive-pyschological terminology in the book. This is particularly true of his discussion of ideas like “ill-defined” problems, problem constraints, and, indeed, his exposition of the notion of “problem” most generally.²⁴

Instead, I will finish up by making a connection with a topic within computational cognitive accounts of human behavior that remains important today. Whereas the problems discussed by Reitman in the earlier chapters of his text lent themselves more readily to modeling as information-processing activities, Reitman claimed that the class of ill-defined problems demanded what “creativity” on the part of the problem solver.²⁵ Again, there is considerably more that could be said about this claim since creativity, as Jamie Cohen-Cole discusses, is a concept that bore a distinctive political charge during the Cold War that wends its way from psychological research with human subjects into artificial intelligence through the work of Newell, Shaw and Simon, among others.²⁶

For now, it must suffice to say that the spectrum of “well-definedness” of particular problems in human information-processing is closely related to informal assessments of how creative a computational agents needs to be in order to solve them. If, as Reitman claims explicitly, fugue composition represented a problem at the most “ill-defined” end of such a spectrum, then a problem-solving model that could successfully compose such a piece would have evidenced a particularly sophisticated grasp of computational creativity.²⁷ The point is, of course, that this was of little concern to Reitman, who explained that the fact that a fugue-composing program was some years off “does not make the model builder’s surrogate question any the less powerful.”²⁸ Nevertheless, by yoking his now-clarified concepts of “ill-defined” problems to the need for “creative” computational machines, Reitman set the stage for a complex and ongoing interplay between terms of art native to cognitive psychologists (such as “well-defined” and “ill-defined” problems) and more widely used (and broadly contested) terms like “creativity”, which come freighted with meanings that both predate and post-date the cognitive-scientific moment. Perhaps surprisingly, it turned out that music composition—and specifically the composition of a fugue—played no small part in this alliance.

Works cited

---

1. Margaret A. Boden, Mind as Machine: A History of Cognitive Science, 2 vols. (Oxford : New York: Clarendon Press ; Oxford University Press, 2006), 373–375.↩︎

2. Sections of this book appear as a CIP Working Paper.↩︎

3. Walter R. Reitman, Cognition and Thought: An Information-Processing Approach (New York: Wiley, 1965), viii.↩︎

4. Boden, Mind as Machine, 363.↩︎

5. Prix de Rome; Tripos; Today↩︎

6. Susan Leigh Star and James R. Griesemer, “Institutional Ecology, ‘Translations’ and Boundary Objects: Amateurs and Professionals in Berkeley’s Museum of Vertebrate Zoology, 1907-39:” Social Studies of Science, August 1989, 387–420, https://doi.org/10.1177/030631289019003001. See also, Susan Leigh Star, “This Is Not a Boundary Object: Reflections on the Origin of a Concept,” Science, Technology, & Human Values 35, no. 5 (September 2010): 601–17, https://doi.org/10.1177/0162243910377624.↩︎

7. Ron Roozendaal, “Psychological Analysis of Musical Composition: Composition as Design,” Contemporary Music Review 9, nos. 1-2 (January 1993): 311–24, https://doi.org/10.1080/07494469300640531, 315. See also, John A Sloboda, The Musical Mind: The Cognitive Psychology of Music (Oxford: Clarendon Press, 1985).↩︎

8. M. Sanchez and W. R. Reitman, “The Composition of a Fugue: Protocol and Comments,” CIP Working Paper, November 1960, 10.↩︎

9. Reitman, Cognition and Thought, 169.↩︎

10. See “Analysis of Protocols” paper for discussion of process.↩︎

11. Sanchez and Reitman, “The Composition of a Fugue.”, 10.↩︎

12. Reitman, Cognition and Thought, 170.↩︎

13. Stephanie Aleen Dick, “After Math: (Re)Configuring Minds, Proof, and Computing in the Postwar United States” (2015), https://dash.harvard.edu/handle/1/14226096, 74.↩︎

14. Dick; Stephanie Dick, “Of Models and Machines: Implementing Bounded Rationality,” Isis 106, no. 3 (September 2015): 623–34, https://doi.org/10.1086/683527↩︎

15. Reitman, Cognition and Thought, 124.↩︎

16. Dick, “After Math.”.↩︎

17. Walter R. Reitman, Richard B. Grove, and Richard G. Shoup, “Argus: An Information-Processing Model of Thinking,” Behavioral Science 9, no. 3 (1964): 270–81, https://doi.org/10.1002/bs.3830090312.↩︎

18. Reitman, Cognition and Thought, 203. D. O. Hebb, The Organization of Behavior (New York: Wiley, 1949). O. G. Selfridge, “Pandemonium: A Paradigm for Learning,” in Proceedings of the Symposium Mechanisation of Throught Processes, ed. D. V. Blake and A. M. Uttley (London: H.M. Stationery Office, 1959).↩︎

19. Reitman, Cognition and Thought, 22.↩︎

20. Reitman, 180.↩︎

21. Reitman, 167.↩︎

22. M. Minsky, “Steps Toward Artificial Intelligence,” Proceedings of the IRE 49, no. 1 (January 1961): 8–30, https://doi.org/10.1109/JRPROC.1961.287775. Cited in Reitman, Cognition and Thought, 148.↩︎

23. Collin Lynch et al., “Concepts, Structures, and Goals: Redefining Ill-Definedness,” International Journal of Artificial Intelligence in Education 19, no. 3 (2009): 253–66.↩︎

24. Reitman, Cognition and Thought, 142–146ff.↩︎

25. Reitman, 167.↩︎

26. Jamie Nace Cohen-Cole, The Open Mind: Cold War Politics and the Sciences of Human Nature (Chicago, Ill.: University of Chicago, 2014).↩︎

27. Reitman, Cognition and Thought, 151.↩︎

28. Reitman, 180.↩︎