Given the degree to which and the ease with which multimodal texts and strategies tend to be misunderstood, subject to
ridicule, if not roundly dismissed, it is crucial that in addition to broadening notions of concepts like writing, reading, text,
technology, and composing we make a concerted effort to resist…the tendency to identify as “childlike,” “merely creative,”
“expressivist,” “artistic,” “nonacademic,” or “experimental” texts that explore the meaning potential of other modes.
— Jody Shipka
Struggling with how to write this introduction and the deadline approaching, I woke up early one morning to two vivid fragments of a dream, something about The Seven Questions Initiative (no idea what that might refer to) and three words, “Words fail me.” I realized the latter was a way to open this introduction, that it captured one of my key responses to Provocations. So let’s begin.
Words fail me. I am watching/hearing the video of Trisha Campbell embodying and envoicing Josephine Miles. Reading the transcript, I realize that some of the words are represented as quotations, inviting me to take the other words as Campbell (or Campbell and other collaborators not named) channeling Miles, filling in the narrative gaps that appear even in a life as voluminously written and archived as Miles’s has been. I see Campbell (I suppose), talking through a partial, quite uneven screen that is an image of Miles. Then Campbell-as-Miles begins to describe how Miles approached a literary analysis of Wordsworth by counting key words Wordsworth wrote. The video cuts to a hand (I suppose Campbell’s) holding a pencil over a piece of paper and making tally marks | | | |, but then after the second group of five just appears, narrative voiceover continuing along with another sound track of someone raspily counting, the video runs in reverse a bit, the tally goes backward, some marks disappear, then it jumps forward again (they’re back). When the second voice whispers “nine,” the paper tally is moving from 10 to 11.
Campbell’s video closes with an image of two pages from Paul Miller’s Rhythm Science, but it doesn’t identify the source as such; instead, the camera focuses in tightly on what looks like a notebook page, green lines that seem to be a drawing of something on paper that is tinted in pale shades of green yellow red, a circular hole drilled in the page and the edges of other multi-colored pages visible in the hole as well as something red at the bottom. The words read: “today, the voice you speak with may not be your own.” Miller (a.k.a. D. J. Spooky) scratches and remixes that fact in his art.
Okay, we’re fairly used to thinking about voices, to pondering the issue of who is really speaking, to considering how digital media might further complicate such questions, but then how do we talk about who is writing the tally and who is editing the video of the tally and adding sound tracks? We have argued (Prior, Hengst, Roozen, and Shipka; Prior and Hengst) that our language for dialogic semiotics is quite limited. What is often still presented as “reported speech” (quotation and paraphrase) has gotten more fine-grained attention, for example, from corpus linguists Semino and Short, who begin by separating out modes (represented speech, writing, and thought) and go on to identify 21 types, which can be further modified by six codes (e.g., hypothethical, inferred). And this nuanced profusion of categories only takes into account the mode that is represented. They were analyzing represented speech, writing, and thought only in a written corpus. So, we could add another set of modal codes for the means of what Goffman called animation: represented speech, writing, and thought that is written, spoken, signed (see the end of Shipka’s first video), filmed, painted, coded. And how many manners and styles might that add to the basic categories?
When Shipka reads a quote and the video zooms into and pans unevenly around the page of the book where the quote came from, what do we call that filmed visual quotation/citation combined with the spoken (and if you have the transcript perhaps also read) quotation? In short, there is much to be done yet in teasing out the dialogic possibilities in our words. But what do we have for other semiotics (for represented, remixed, and replayed gestures, voice qualities, bodily movements and postures, interaction orders, architectures, fashions, musics, mathematics, designs in all kinds of media)?
Practically nothing. When it comes to the other semiotics, we aren’t even close. Semiotic remediation (Prior, Hengst, Roozen, and Shipka; Prior and Hengst) was a call for a dialogic semiotics and some close attention to cases that illustrate the need; it was not the Semino and Short map of the categories, though to be fair we weren’t aiming for such a map, taking to heart Judith Irvine’s point that making sense in this here-and-now requires a sense of shadow conversations (or, we argued, shadow acts) that happened in the past, are happening elsewhere now, and that might happen in the future. The force of shadow acts points to the chaotic rhizomatic character of human lives and acts in the world. And accounting for that, as Irvine argues persuasively, calls for a dialogics that does not fit inside the boxes of even the finest, most nuanced categorizations. So, when I try to describe the dialogic dynamics of these multimedia texts, words fail me.
Words fail us. The four multimedia texts, these four provocations, make that point at many levels. They illustrate the surprising exuberance of others’ semiotic lives, the infathomability of loss, the fog of others’ motives, the power of signs to evoke experience and yet their failure to illuminate it clearly or finalize it (shout-out to Bakhtin for noticing unfinalizability). However, as Shipka warns, digital texts and digital scholarship are still provocative. We don’t always know how to read (words fail us) them, can miss their signifying practices, can easily see them either as creative-expressive (“What a moving narrative,” I say, as I listen to the The Moth Radio Hour on my local public radio station as I drive around town) or as falling somewhere on that scale that goes from children’s art to avant-garde art, as an artifact that disrupts our typical semiotic expectations, making what Wysocki has identified as the basic move of new media—calling attention to the materiality of the text, to its contexts of production, to the affordances of this text-in-context.
Words fail us, but let’s begin again. I need to circle back to say something introductory, an “oh, by the way” introduction in midstream. Provocations is the title of the new CCDP series that the four multimedia texts of this collection, Reconstructing the Archive, launch. Provocations is a provocative choice. The primary senses of the word involve either challenging someone to a fight or doing something that causes strong emotions (anger, irritation) in others. So, the question perhaps is what or whom these texts are meant to provoke. At one level, all four are just exploring different senses of the notion of the archive, of oral history, of how to understand and transform the literate and material artifacts of an archive. I’ll just say a bit about each text (and each author has also offered her own framing of her work and the work of the collection).
Jody Shipka’s “On Estate Sales, Archives, and the Matter of Making Things” explores the use of unconventional archives, of found texts picked up at estate sales, garage sales, and flea markets. The first video she offers, “The Things They Left Behind: Estate Sale as Archive,” explores one example, boxes of material from a couple (Dorothy and Fred, who died in the early 1980s), boxes that included scrapbooks, trip diaries, and various other documents and objects. Dorothy was the main collector/producer/curator (words fail us) of these materials, which Shipka argues offer an archive of everyday life, of people who wouldn’t imagine their papers would be important enough to go to the local historical society or a university library. She reads the ordinary literate and semiotic practices of Dorothy’s scrapbooks as rather extraordinary, as presenting, for example, a quite interesting ideology of what is reportable. Dorothy emerges as an canny representer of lived experience, someone who documents multiple views from the windows of her house, who arranges a wide variety of multiple media in the scrapbook, who documents many mundane and even problematic moments of the couple’s lives (e.g., including medical reports and images from hospital stays). Shipka’s interests here are less those of a historian than of a scholar interested in semiotic remediation/multimodality.
As Shipka works with the archive, she also considers how to remediate scholarship itself. For example, when she is quoting an academic source, the video often zooms in on fragments of the pages of the article or book, just as it zooms in on fragments of the scrapbooks. The video script is also made partly visible, converted into microfilm (color, reverse, and regular) and then filmed moving unevenly through a zoomed-in microfilm reader.
The final section of the video presents the Inhabiting Dorothy project, where Shipka shares some of Dorothy’s texts and images online and invites other scholars to remediate them. In one form of remediation, Jody and her partner follow and document the trip itineraries of two of Dorothy and Fred’s trips. This orientation to using material and digital resources to perform the archive runs across all of these texts, but Shipka’s is the only one to crowdsource the performances (to ask others, as she does in her pedagogy, to show her what is possible). She identifies seven categories of repurposing/remediation in the 40 responses to her invitation and displays examples of each type. Finally, underscoring her deep interest in both material and digital forms of composing, Shipka provides a second, composing-process video, “On the Making of ‘The Things They Left Behind,” which details (voiceover and video) decisions, events, challenges, social networks, and technologies that composed the first video.
Words fail me and it will take me a long time and repeated watching to get all that Shipka has embedded in these videos, much as I still read and re-read Voloshinov and continue to find new layers and arguments that inform and surprise. One small example, in the videos, transitions are marked with a repeated grainy image/sound of Shipka holding a classic wooden ventriloquist’s dummy (birds are singing and there are sounds, “tap tap tap,” like checking a microphone, and her voice saying “Is this thing on? Are you recording?”). This repeated and layered visual-aural trope evokes ventriloquation as material and digital practice, a matter of complex interactional footings. Who’s talking? Who’s looking up? Who’s tapping? What is the word for ventriloquation across other semiotics? Words fail us.
Alexandra Hidalgo’s “Family Archives and the Rhetoric of Loss” tackles another family archive, in this case an archive of her family, part of which she says she absconded with on a visit to her paternal grandmother’s house. The video at the center of this text considers especially the question of positioning in relation to an archive with strong personal resonances, and this example involves quite strong resonances as Hidalgo delves into materials that represent her father, Miguel Hidalgo Briceño. An economist, author, inventor and adventurer, Briceño was divorced from her mother when he disappeared in the Venezuelan Amazon. Hidalgo was six years old at the time. As she tells of her quest to understand the mystery of her father’s life and disappearance, Hidalgo shows scenes of family who have been her informants along the way as well as images of a variety of materials (texts, letters, photos, sculptures, homes, vistas) that she possessed or has secured from others in the family or has visited and taken photographic and video recordings of. Many of the images show Hidalgo positioned in interaction with her family and friends and with the environments and objects she finds or collects. The video here is silent, visual backdrop to the voice-over exposition and narrative (again available as a transcript to be read). We see Hidalgo interacting with family, artifacts, and texts, see scenes of family life, actions and nonverbal expressions, see talking, but we don’t hear it. Hidalgo’s voice is the main channel in this text.
Hidalgo does not find her father, either in the sense of uncovering what happened to him in the Amazon or of coming to a clear and settled picture of him. Worse, she finds some unsettling insights into her paternal grandmother. Nevertheless, she realizes what she has discovered along the way is how these stories, texts, objects and images trace the absence of her father and the ambiguities of life.
Here words fail us again. Hidalgo tells a compelling tale of loss, of the search for an ancestor, but in the end her father remains elusive. His life, his experiences, his disappearance, the sense of it all is expanded by multiple media and autoethnographic work, but never nailed down and finalized.
Erin Anderson’s “’What Hadn’t Happened’: A Multimedia Memoir” takes up a thread of The Olive Project (Anderson, 2011) about her paternal grandmother, a multimedia exploration of an oral history as narrative composition. Through a blend of text, voiceover, audio recordings of interviews, and sharp photographic images with (pasted on them á la Post-Secret or ransom notes) typewritten words or phrases (which turn out to be represented speech from interviews or represented writing from newspapers), Anderson slowly unwraps another family mystery, the death of Olive’s younger sister, Gloria, who drowned in a wash boiler on their Minnesota farm in 1934, when Olive was three years old. As the tale unfolds, we learn how Olive’s mother initially blamed Olive publicly (it was in the newspaper) for leaving the cover off the tub, but ultimately (years later) confessed to Olive that it had been her , not Olive, who had made the error. Anderson also talks to Olive’s younger brother and sister and to a cousin who was on the farm that day. She presents an audio collage (with transcript) from the interviews that narrate and evaluate these events. Erin and her father also take a road trip to Minnesota, find and photograph the old farm and its now dilapidated buildings, and buy a copper wash boiler (an image of which is presented) from the era of the death at an antique store.
The voices and images convey the emotional weight of those events, but here again words fail us. How can we understand that tragic moment from 80 years ago, a pregnant mother finding her daughter drowned, apparently blaming her 3-year-old for the death to cover up her culpability, the dialogic and deeply affective reverberations of those events in the lives of the family?
And back now to Trisha Campbell’s “I am Josephine Miles: A Digital Reprocessing,” which takes up the more conventional archival and public records of Josephine Miles, a poet, critic, and compositionist. Here the focus is on narrating Campbell’s own sense-making about Miles’s life, the embodied performance of Campbell-as-Miles. Campbell frames the short video (just over eight minutes) around taking historiography as haunting (Baliff), and the video plays with this trope at every turn as the image of Miles floats ghostlike in front of Campbell’s face, as Campbell channels Miles (the medium is the messenger), as the film style and sound track play with conventions of postmodern horror films (spooky sounds, disturbing visual effects), as Campbell closes with a quote from D. J. Spooky. It is a bit eerie watching Campbell’s face behind the cracked and shifting image of Miles’s face, envoicing and channeling Miles, performing the archive. As I said at the beginning, words fail us.
These four multimedia texts call on us to read differently. At one level, all four follow a general generic framework. Their digital design is unified. (Anderson and Campbell did the overall site design.) Each text begins with an introduction of the work, and each intro includes links to one or two multimedia objects. Shipka, Hidalgo, and Campbell each link to videos (two for Shipka). Anderson links to a page with images and audio files. At another level, all four multimedia texts trouble the laminated lines between performance and analysis, between disciplinary scholarship and everyday narrative, between multimedia artifacts and processes of production, between story and critical engagement with a story. All four texts call new media attention to complex dialogic semiotics. “Today the voice you speak with may not be your own,” says Miller.
In 2014 we (Prior and Hengst) became grandparents, and we regularly get notifications from our cloud photostream that images and sometimes videos have been uploaded of our granddaughter and her parents and friends. Recently, I have turned again and again to one of the videos from this summer. It shows Ardea (about 16 months old) on a swing at a playground in France, her first time. Her dad is taking the video from directly in front of her and pushing the wooden bucket swing pretty gently. I see Ardea smiling and enjoying the movement and then there is this moment: She begins to sway in the seat, looks at her hands holding the rope cables of the swing, and I see her realizing that she can pump the swing, that she is pumping the swing. I see her face light up (her eyebrows raise; she smiles with rapturous joy). This image plays over and over in my mind: the utter joy of learning and being in that moment. The fleeting frames of that video have become emblematic for me of the amazing human potential that comes with each birth, but they also trigger an unutterable sadness. I know that we so routinely squander (as well as selectively tap into) human potential, that the capacity for sheer joy is often overwhelmed by pain, suffering, oppression, tragedy, and violence.
I could in this digital medium share a still from the video or even the video itself (if I asked for and got permission from her parents), but I haven’t asked. If you saw it, you might see some version of what I see, or you might not. For that matter, what I see and feel watching that video might capture some part of what Ardea experienced and felt in those moments, or I could be overreading (words fail us) the image.
In the haunting mystery that is our semiotic being in the world, it’s not just words that fail us. Images fail. Film fails. Digital representations fail. All signs fail. Our capacity to express our being never catches up to the totality of our ever-evolving being-in-the-world. What is provocative in Provocations is the way these pieces edge into the disorienting borderlands of dialogic semiotics, into the dense unending mix of illumination and insight with obscurity, confusion and silence. Our goal then cannot be to succeed in some hegemonic dream of transcendent representation, but just to multiply and amplify, disrupt and critique, our knowledge, perceptions, assumptions, feelings, and being in the short span we have on this planet. In this latter task, I hope you find, as I have, that these four texts are illuminating in human as well as scholarly terms. And I look forward to future provocations in this series so well begun.
Anderson, Erin R. “The Olive Project: An Oral History Composition in Multiple Modes.” Kairos: A Journal of Rhetoric, Technology, and Pedagogy 15.2 (2011):
http://kairos.technorhetoric.net/15.2/topoi/anderson/index.html
Bakhtin, Mikhail. The Dialogic Imagination: Four Essays. Trans. Caryl Emerson and Michael Holquist. Ed. Michael Holquist. Austin: U of Texas P, 1981. Print.
Baliff, Michelle. “Historiography as Hauntology: Paranormal Investigations of the History of Rhetoric.” Theorizing Histories of Rhetoric. Ed. Michele Baliff. Carbondale: Southern Illinois UP, 2013. 139-53. Print.
Goffman, Erving. Forms of Talk. Philadelphia: U of Pennsylvania P, 1981. Print.
Irvine, Judith. “Shadow Conversations.” Natural Histories of Discourse. Eds. Michael Silverstein and Greg Urban. Chicago: Chicago UP, 1996. 131-59. Print.
Miller, Paul. Rhythm Science. Boston: MIT Press, 2004. Print.
Prior, Paul, Julie Hengst, Kevin Roozen, and Jody Shipka. “‘I’ll Be the Sun’: From Reported Speech to Semiotic Remediation Practices.” Text and Talk 26.6 (2006): 733–66.
Semino, Elena, and Mick Short. Corpus Stylistics: Speech, Writing, and Thought Presentation in a Corpus of English Writing. London: Routledge, 2004. Print.
Shipka, Jody. Toward a Composition Made Whole. Pittsburgh: U of Pittsburgh P, 2011. Print.
Voloshinov, Valentin. Marxism and the Philosophy of Language. Trans. Ladislav Matejka and Irwin Titunik. Cambridge: Harvard UP, 1973. Print.
Wysocki, Anne. “Openings and Justifications.” Writing New Media: Theory and Applications for Expanding the Teaching of Composition. Eds. Anne Frances Wysocki, Johndan Johnson-Eiola, Cynthia Selfe, and Geoffrey Sirc. Logan: Utah State UP. 2004. 1-41. Print.