The interplay between text, images, audio, and video are at the heart of transmedia storytelling. It is this interplay that gives transmedia narratives the potential to create narrative experiences that engage audiences across multiple dimensions.
Working with multiple media to ensure that they don’t “step” on each other and the overall story is one of the biggest challenges for transmedia storytellers.
We can draw some lessons on how to work with multiple media by looking at works that incorporate text and images. Graphic novels (a.k.a. comic books) and photo essays are beginning to be used in fiction and non-ficition transmedia projects and we are seeing the development of interactive picturebooks.
Over the next few blog posts, I’m going to take a look at what we can learn from picturebooks in particular and how we can apply those lessons to the design of transmedia narratives. The principles for integrating text and images in a picturebook can be readily applied to transmedia narratives that include audio, images, and other forms of communication.
The traditional approach to using text and images is for one to be dominent and the other to play a supporting role. In their book Children’s Picturebooks: The Art of Visual Storytelling authors Martin Salisbury and Morag Styles note:
In most contexts, illustration provides a visual accompaniment to words, a prompt or aid to the imagination that aims to augment the overall experience of the book. (pg. 89)
This tendency is common in today’s transmedia project. At the heart of the narrative might be a television series, a feature film, or a novel with web-based media serving a secondary role. Many of the projects cited as pioneers in transmedia storytelling are prime examples of this, with the transmedia elements being nothing more than marketing material to drive the audience to a particular product.
In order for transmedia storytelling to grow as an artform, it is essential that creators move beyond the marketing paradigm. Salisbury and Styles state:
…in the case of the picturebook, words and pictures combine to deliver the overall meaning of the book; neither of them necessarily makes much sense on its own but they work in unison. And the most satisfying picturebooks create a dynamic relationship between words and pictures. Often this duality can be in the form of a playful dance, where images and words can appear to flirt with and contradict each other. (pg. 89)
In a transmedia narrative, a dynamic relationship needs to be created between the various media elements. The part of the story told in a series of YouTube clips must engage in the “playful dance” with the textual story elements in an e-book. Text, images, audio, and video need to work together to enrich, expand, and enhance each other, creating an integrated whole.
Part of this integration involves the technical elements of a transmedia narrative’s interaction – effective navigation design, functioning links, calls-to-action that move the reader through the narrative, and so on.
Equally important is the integration of various media from a storytelling perspective. With picturebooks, Salisbury and Styles said:
…the relationship between word and image already lies at the heart of what makes a picturebook good, bad, or indifferent. Fabulous artwork can be admired, but if the words don’t interact with the pictures in interesting ways the book as a whole will not be a success. On the other hand, the written text may be superb but if the pictures are bland the overall effect will be mediocre. The very best illustrators…create memorable picturebooks with the words and pictures connect brilliantly. (pg. 89)
The best transmedia storytellers will figure out how to connect the words, pictures, audio, video, and other forms of media into relationships that will create memorable stories. Transmedia storytelling as an artform is only just beginning to emerge. The body of knowledge on how to create these types of stories is in its infancy and it will take time for it to be developed.
However, we can look to other forms of storytelling, learn from them, and begin to apply those lessons to our own transmedia projects.
In my next post, I will take a closer look at some of the design principles of effective picturebooks and talk about how they can be applied to transmedia narratives.