Leen Breure, July 12, 2013
It is a truism to say that we live in a visual culture. Producing visual material has become quite easy. Smartphones with good quality camera's are widely used. All-in-one printers have compact scanners built in, which deliver images good enough for most forms of digital publication. Displaying visuals everywhere has become part of our lifestyle thanks to tablets and other mobile devices. The large choice of social media makes sharing visual material to a trivial activity, even as simple as pushing a button on a modern web-based camera.
We tend to become 'digital natives'. In October 2011 a video was posted on YouTube, showing a one-year old girl sweeping her fingers across an iPad's touchscreen, and, in the following scenes, trying to do the same on the pages of paper magazines as though they too were screens. The experience that nothing happens, has given the title to this clip: A Magazine Is an iPad That Does Not Work. Perhaps, the title is too suggestive, as Jabr argues in his article in the Scientific American; psychological research does not support the idea that tablets are superior to printed journals [Jabr].
A digital native: technology codes our minds, changes our OS.
Video by Jean-Louis Constanza.
We should be aware that all visual representations are constructed and have their codes, conventions, and biases. The creators had certain goals in mind have and the visual product comes with features which should be subject to scrutiny in the same we as we have been trained in critical reading [Kellner]. Therefore, reading images requires a visual literacy, which encompasses, among other things, the multimodal, interactive and emotional aspects of visualization:
"21st century literacy is the set of abilities and skills where aural, visual and digital literacy overlap. These include the ability to understand the power of images and sounds, to recognize and use that power, to manipulate and transform digital media, to distribute them pervasively, and to easily adapt them to new forms." [A Global Imperative]
However, the focus on current digital technology may be misleading. We may adopt a broader definition of visual culture, which connects us to the past and puts our current visualization in a long-term historical perspective, as Paul Jay remarks in Picture This: Literary Theory and the Study of Visual Culture:
"Even if we were to limit the concept of visual culture to pictorial representation, visual culture seems to me to have a long history that makes the idea of a clear break between the modern-as-textual and the postmodern-as-visual quite problematical.
Pictorial representation in the West begins in the flat, inert surfaces of Medieval religious art, shifts in the early Renaissance with the depiction of perspective, emotion, and the body, develops further around a fascination with the increasingly intricate depiction of built spaces that first competes with, and then nearly overwhelms the focus on religious drama, and then begins a kind of inexorable march toward realism and the secular depiction of everyday life, first in painting, photography, film, and then in video and digital technologies. From this point of view, it is hard to see how a fascination with the visual per se can be isolated as a postmodern phenomenon."
Images do not exist in a vacuum; their usage is in some way connected with text, genre conventions and views on reality, which have their roots deep in the past. Illustration and visualization are also part of the rhetorical devices in scientific discourse. This leads to the question, what influence current trends in visualization influence have on the various forms of scholarly communication, varying from (digital) journals to more informal presentations on the web.
Research has — for very good reasons — privileged the written text as the highest form of intellectual expression and considered visual representations as second-rate illustrations of results. It is the discourse that should convince the critical reader and which is supported by data which may be visualized to ease understanding. For a long time publication in highly ranked (printed) journals have been considered as the gold standard in research, which left a mark on the digital versions as well.
What is the impact of the rise of visual culture on scholarly presentations? Is their any influence that is strong enough to match the powerful traditions of textual genres? Is their any continuation in visualization from before the digital age?
The progress in science has been strongly linked up with visualization techniques, for example X-ray photography in in medical sciences, MRI scans and computed tomography [Smelik]. Lisa Cartwright comes to an interesting thesis in her book Screening the Body: Tracing Medicine's Visual Culture. She claims that the cinematic apparatus can be considered as a cultural technology for the discipline and management of the human body, and that the long history of bodily analysis and surveillance in medicine and science is critically tied to the history of the development of the cinema as a popular cultural institution and a technological apparatus [Cartwright, p. 3].
_250x378.png)
Scientific visualization has slowly developed from the mixture of empirical fact, magical beliefs and hearsay prevalent in early modern times to representation of the "truth", being never able to escape from a certain cultural bias [Pauwels]. Visualization in science is not primarily an effort to create a replica of reality, but, since science is about revealing mechanisms, structures and laws, it serves purposes of resolving a problem and facilitating knowledge transfer. Michael Lynch has argued in favor of an approach that situates scientific visualization in the broader context of research activities.
Current research projects produce a lot of data in digital format, which can be directly captured and transformed in visual representations, mostly graphs and charts. They form the core of the illustrations of most papers, together with photo's and video's. Multimedia and data sets belong to the non-printable content, and, depending on the scientific fields and facilities of journals, are often offered in a special section of the publisher's website.
Recently Elsevier has introduced a novel publication concept: the Article of the Future, which includes interactive graphs, interactive maps and 3D animated models in the digital versions of many of their journals. Some charts come with two versions: a static overall view and interactive one, which allows selection of information and precise reading of value on a curve. Simple schematic maps have got a more information rich interface using Google maps, with clickable points of interest. Audio and video fragments may be part of the article itself or included the appendix.
A different way of scientific visualization is explored by the publishing house Brill, in close cooperation with the Dutch Huygens Institute, the Dutch data archive DANS (both part of the Royal Netherlands Academy of Arts and Sciences), the Museum Boerhave (Leyden) and Wild Card, a game company. In this project 3D models of historical scientific devices and processes are created and directly linked to printed publications. A QR-code in the printed article provides the reader with a quick link to the related presentation on the publisher's companion website, where multimedia material will be published.
The tough ties of genre conventions have prevented, that the rise of visual culture has produced a revolution in scientific journals, even in the online-only ones. The claims in the YouTube clip about the Article of the Future as the next generation in research publishing and as true immersion in author's world of discovery are highly exaggerated. True, this new format and all look-alikes will considerably extend the written message with valuable and sometimes captivating representations, but a truly immersive experience requires a change in the communication pattern itself.
Such a multimedia presentation implies a more evenly balanced and integrated use of the different media. Images are no longer illustrations that could be left out without breaking up the discourse. The usual textual presentation may be intertwined with a authoring style in which visual content is used as a "hook" for an argument. Some experiments in multimedia scholarship go into that direction, but, of course, don't easily fit in existing scholarly genres [Burgess; Ball].
The Web has given rise to new forms of scientific discourse in the form of social media and blogs. Science blogging may arise from different motives: to share content and to express opinions, to organize thoughts and ideas and to create relationships inside and outside the author's own scholarly community. It is a way to create visibility and establish an online reputation [Shema; Puschmann].
The Science Photo Blog of Voice of America is a long list of images with short captions, mostly linked to more detailed web documents. Tumblr has a science section with small posts consisting of a photo plus a large caption. Browsing through the blogs of Nature, the Scientific American, Discover and the Science Blogs one finds considerably much more visuals than in the journals, although the majority is limited to texts and photos (typically one image that illustrates the subject very well). Including YouTube videos is not uncommon. It also seems to depend on the taste and preferences of the author and the kind of topic discussed.
From Tumblr:
Current Biology (July 6, 2013).
Some disciplines have more visual material than others and then visualization is obvious. For example, there many physical things to talk about in the Blogoshere, a blogging site of the American Geophysical Union. Interaction between visuals and text is nicely practiced in the large blog Religion in American History: some images are only illustrations, but many of them are part of the discourse. The influence of climate change on Tanzanian bird distributions requires pictures of birds and a map. Ethan Siegel's blog Does dark matter affect the motion of the solar system? (July 3, 2013) is an interesting mixture of text and captivating images of the universe. His weekend diversion Spider Webs... on drugs? (July 6, 2013) could hardly be written without the fascinating images of spider webs, which clearly show the effects of psychoactive drugs, such as amphetamine, mescaline, strychnine and LSD. But these specimens of multimedia communication are exceptions.
The genre of blogs may mix with other modes of communication; the Blog Picture Science is a companion blog site to the Big Picture Science radio show, one-hour science program produced at the SETI Institute’s radio studio in Mountain View, California. Therefore, it contains audio fragments as well. However, for a real change in genre you have to look elsewhere.
Full multimedia literacy the underlying principle in Multimedia Scholarship, most well-known from the courses in the core program of the University of South California [Barish] and from the journal Vectors, which doesn't seek to replace text, but encourages a fusion of old and new media in order to foster ways of knowing and seeing that expand the rigid text-based paradigms of traditional scholarship.
The concept of multimedia scholarship is too complex to be discussed in this context and deserves a blog of its own. The contribution of scholarly multimedia is usually medium dependent and comes in radical physical forms, which creates confusion over how to categorize and evaluate them. Most academic institutions adhere to the traditional print standards when it comes to evaluation of scholarly output (peer-review, highly ranked journals), which makes, that "we continue to privilege content over form, intellectual labor over physical labor, and print over digital media" [Burgess].
Innovative experiments are also made in the rhetorically related field of multimedia journalism. In December last year the New York Times published the Snow Fall - The Avalanche at Tunnel Creek, a feature article by John Branch and an interactive prize winning multimedia experience with images, video, maps, moving infographics and image slideshows. The main story is presented in chapter-style stages. Users are free to explore the content further by clicking on video and slideshows.
A sensational proof of multimedia journalism,
New York Times, December 2012.
It has been labeled as "a watershed in multimedia journalism". In the newsroom, people are saying: "Can we snowfall this story?" [Rue]. It uses Javascript (jquery) and novel curtain effects, which reveal or cover images as the user scrolls. The design enhances the narrative by triggering emotional involvement.
A similar integrated alternation of text and multimedia in the popular genre is to be found in ibooks, the proprietary epub3 format of Apple. A good example is Al Gore's Our Choice, an iPad app, which is a follow-up to his call-to-action of An Inconvenient Truth. It has the same style as an ibook, with georeferenced photo's (clicking on as mall globe icon displays a map with a pin on the location where it was taken and some of them have audio fragments), video's and interactive graphs (e.g. in chapter 11 the growth of the world population and CO2-emission from 1 AD to 2300 in the form of an interactive timeline-graph). "What resulted combines the expressiveness of the traditional form with the educational simplicity of a documentary, all tied together with an utterly smooth, remarkably beautiful interface that does justice to the material it presents, and the format it's presented within", as one of the reviewers wrote.
_400x305.png)
The app Our Choice by Al Gore and publised by
Push Pop Press.
The answer on the question at the beginning, what influence current trends in visualization have on the various forms of scholarly communication, cannot be simple and one-dimensional. The innovative concept of 'Article of the Future' will offer more than many researchers care for. Quick publication in a highly ranked journal with illustrations as far as necessary will be the bottom-line. Period. This has much to do with the primary purpose, i.e. timely sharing of the data-based results of scientific discovery, and also with the system of credits and prevailing genre conventions.
In particular genre and credits are able to break the rushing waves of visualization. Going more to the periphery of scholarly communication, to the blogs, social media and popular presentations of research, where rules are few and credits come through visibility, we see a natural link with the multimedia trend in today's society. However, the connection is also dependent on individual preferences and capabilities in this respect. Visual literacy is indispensable to be productive in multimodal communication and, unfortunately, still a missing link in general academic training.
Leave a Reply(2005) A Global Imperative. The Report of the 21st Century Literacy Summit. The New media Consortium.
Ball, C.E. (2012) Assessing Scholarly Multimedia: A Rhetorical Genre Studies Approach. Technical Communication Quarterly , 21: 61–77.
Barish, S. & Daley, E. (2009) Multimedia Scholarship for the Twenty-First Century (PDF).
Burgess, H.J. & Hamming, J. (2011) New Media in the Academy: Labor and the Production of Knowledge in Scholarly Multimedia. Digital Humanities Quarterly 5.3.
Cartwright, L. (1995) Screening the Body: Tracing Medicine's Visual Culture. University of Minnesota Press.
Jabr, F. (2013) The Reading Brain in the Digital Age: The Science of Paper versus Screens, Scientific American April 7, 2013.
Jay, P. (2000) Picture This: Literary Theory and the Study of Visual Culture. An address delivered at La Sapienza, Rome, Italy, March 2000.
Jorink, E. (2011) Beyond the lines of Apelles, Johannes Swammerdam, Dutch scientific culture and the representation of insect anatomy, in: Eric Jorink, Bart Ramakers (eds.), Art and science in the early modern Netherlands/Kunst en wetenschap in de vroegmoderne Nederlanden, Zwolle, 2011, p. 148-183
Kellner, D. (2008) Critical Perspectives on Visual Imagery in Media and Cyberculture.
Pauwels, L (ed.) (2006) Visual Cultures of Science: Rethinking Representational Practices in Knowledge Building And Science Communication. Dartmouth College Press.
Puschmann, C. & Mahrt, M. (2012) Scholarly Blogging: A New Form of Publishing or Science Journalism 2.0? in: Tokar, A., Beurskens, M., Keuneke, S., Mahrt, M., Peters, I., Puschmann, C., van Treeck, T., & Weller, K. (Eds.). (2012). Science and the Internet, Düsseldorf. Düsseldorf University Press, p. 171- 81.
Rue, J. (2013) The ‘Snow Fall’ effect and dissecting the multimedia longform narrative. Multimediashooter, April 21.
Shema H., Bar-Ilan J. & Thelwall, M. (2012) Research Blogs and the Discussion of Scholarly Information. PLoS ONE 7(5): e35869. doi:10.1371/journal.pone.0035869.
Smelik, A. (2010) The Scientific Imaginary in Visual Culture. V&R unipress GmbH.
