Multimodal Writing is Like Baking a Cake

by

Frank DiGirolamo


Featured Image: “Katie’s birthday cake” by Conall, CC BY 2.0

I’m posting a bit later than usual. Lots of tests and projects due this week I had to prioritize those first but I made sure to set time for this. This week I’m looking at Thinking across Modes and Media (and Baking Cake): Two Techniques for Writing with Video, Audio, and Images by Crystal VanKooten.

In this essay VanKooten analyzes how digital writers create multimodal text through combining various elements and compares it to baking a cake. First, VanKooten explains what “multimodal” is. Multimodal means when you combine various modes to get a message across. VanKooten describes as a mode as “a way of communicating”. A basic example of a multimodal is text is a video. When you make a video, you combine images, sounds, and written words, which are separate modes on their own. Other examples she cites are movies, memes, posters, articles, and websites.

Although this essay focuses on digital multimodal texts, multimodal texts can also be non-digital. VanKooten compares this to baking a cake as when you bake, you combine various ingredients. When the cake is done, it doesn’t look like a mixture of the ingredients, but rather something entirely new. And like a baker who has access to various types of ingredients, each mode can blend together in various ways.

VanKooten looks at two ways that modes can blend together. The first of which is “Integration”. There are actually various types of integration like substitution, disambiguation, labeling, example, reinforcement, completion, chunking, clustering, and framing. VanKooten focuses on reinforcement because students tend to use this one the most. She uses a term call “cross-modal reinforcement”, which is when multiple modes appear on screen at the same time to reinforce an author’s point. An example she uses is when a student writes out words in a video while they’re speaking over those same words. Another example is when a student writes out facts on a Powerpoint slide and has a graph that shows that same information on that same slide.

The second way is “Juxtaposition”. VanKooten describes it as when different modes appear next to one another in a single space or moment to create a new meaning or idea to the audience. She explains two different types. The first is “sequentially”, where you put sounds and images next to or after one another. An example she gives is when her students present different images in progressive time to tell a story.

The second is “simultaneously”, when sounds and images are layered on top of each other at the same time to create contrast, tell a story, or to compose a transition. Examples she gives is when students use a split screen or picture-in-picture. VanKooten compares juxtaposition to a layer cake as while there are multiple layers on top of one another, they are not fully mixed. The only connections are the filling in the middle and the frosting on the top and outsides.

A major example that VanKooten to showcase both integration and juxtaposition is “A College Collage: Not Going Back”. It was a video by Evan Kennedy, one of her old students as part of a video composition assignment. In it, Kennedy uses different modes at the same time to reinforce his themes about mental health in college. In one instance, there a female voice that says “Fighting yourself to wake up, get up, take a shower, try and breathe, try and smile, try and act like you believe you have something to live for.” At the same time, an image sequence of a shadowed, female figure walks down a hall, and slowly towards us.

For Juxtaposition, Kennedy had the Demi Levato song “Old Ways” play throughout the video. At various points, Kennedy uses images and words in a sequence that matches the beat, lyrics, and changes in pace. He carefully juxtaposes each word and image alongside one another. For instance, at one point in the song, Levato sings “I’m down again / I turn the page / The story’s mine / No more watchin’ the world from my doorstep / Passin’ me by / And I just keep changing these colors….”. While that’s happening, Kennedy juxtaposes the lyrics with a sequence of images starting with a stack of books growing on a shelf. Following this, a growing pyramid of empty pill bottles and then the year “2009” on appear the screen . Each image changes with the percussive beat of the song. We eventually anticipate the image to change everytime we hear this beat.

My Thoughts:

So unlike the last few essays, which were full of loaded and complicated language that most people wouldn’t say to each other in a conversation, this one felt more like a breather to read. I guess the cake metaphor really worked for me. Although I think it’s more likely because I’ve actually been already doing this kind of stuff without realizing it.

For instance, I think this post on its own is “integration”. I explained the metaphor between multimodal text and cake baking and to reinforce it, I put a cake as the featured image.

One medium I think where multimodal text applies in both ways are comic books/manga. I’m gonna sick to comic books for my explanation as I haven’t read much manga. I’m probably one of the few people in the world who reads more comics than manga.

Comic books are different traditional books. In additional to the words you’ll except, there are pictures on every page that are referred to as panels. Each panel shows an illustration of what’s going on in the story. The words in the story will be usually be in the in the form of various text boxes and balloons. The balloons represent what a character is saying. The boxes are usually narration to explain to the reader something that characters wouldn’t naturally say.

In the case of “integration”, the text and the panels compliment each other. The text balloon will have a character say something along the lines of “I’m mad”. At the same time, the panel where that balloon is will show an illustration of that same character punching something to reinforce that they’re mad.

In the case of “juxtaposition”, there are multiple panels on most pages. They’re meant to be read in a sequence of left-to-right, top-to-bottom to tell a story. Occasionally, you’ll see one big panel take up in an entire page that’s referred to as a “splash page”. But as I said a few times, there are pages. Most comic issues vary from 20 to 30 pages. Each page is to be read in a sequence as you turn book left-to-right.

When it comes to simultaneously, that’s a bit more tricky. Comics are meant to be seen as a sequence happening one after another. If there are two or more panels/pages that meant to be happening at the same time, the reader will likely not know at first. They’ll assume that the first one happened and then the second shortly after. Maybe you could add a text box that says that the panels happening at the same type, but it would be awkward and break the flow of the story. Occasionally, there are pages that seem to convoy simultaneous panels but there far and few between.

As I said before, I can’t speak for manga but from the little I’ve read, manga writers and artists seem to do a similar thing to their western counterparts. The main difference is that the pages and panels are right-to-left instead of left-to-right.

Another medium I’ve done over the years is video making. I haven’t been making any videos recently, mostly because of my classes (like this one) but I have some experience that I can talk about. So on it’s own, video making is more-or-less sequential multimodal text. Historically, a video is a sequence of images that when played very quickly, it would look like one image was moving. So basically, whenever I recorded a video, I was already making a multimodal text without knowing it.

While integration is not necessarily when making a video, viewers tend to want more than just the video. Even in the age of “silent” film, they weren’t truly silent. Silent in this case usually means that you can’t hear the characters talk. However, there frequently be a music track to accompany the footage. Whether it’d be pre-recorded track, or a live band playing alongside the film. In the case of my videos, many of them are me playing a video game, so there’s already the audio of that playing. In addition to that, I frequently record my voice giving my thoughts and reactions to the game. It’s not a lot but it counts (if only barely).

And I think that’s all I have to say that’s relevant to this post. I could go on longer but two things. 1. I would just go on tangents about comic books that no one but me cares back. 2. I have to also put my attention on my other assignments this semester, which even includes this very class. Thank you all for reading and I’ll see you all again next week. No later than Monday.


Comments

Leave a Reply

Your email address will not be published. Required fields are marked *