Computational Media Aesthetics (CMA) is "the algorithmic study of a
number of image and aural elements in media and the computational analysis of
the principles that have emerged underlying their use and
manipulation, individually or jointly, in the creative art of clarifying,
intensifying, and interpreting some event for the audience." (Computational
Media Aesthetics: Finding meaning beautiful. IEEE Multimedia 8, 4 (2001), pp. 10--12)
CMA is an approach to crossing the semantic gap--"the simplicity of available
visual features and the richness of user semantics"--by using knowledge of
production grammars, the aesthetic elements they manipulate and their intended
effects upon the audience, in order to infer about media content. CMA is an
attempt to see media with its makers eye, and requires an equal grasp of
computational frameworks and production domain issues. While for analytic
purposes CMA is primarily applicable to 'produced' media, such as film or tv,
its usefulness extends also to 'unproduced media' by virtue of the insight it
gives into fundamental physiological responses upon which grammars are built
and leverage. For the application to synthesis of media, CMA has an important
role in building tools and techniques for giving raw media 'production'
qualities.
The following projects all have a core CMA component:
Indexing
The current global digital landscape is characterized by a growing flood of
data coupled with an inability to automatically create indices that allow us to
cross the semantic gap. CMA has been used to address this by providing
semantically rich indices for a number of production domains. Previous work has
formulated techniques to extract film tempo, rhythm and high order dramatic
structures, along with genre-based sound and colour atmosphere patterns, and
even underlying scene 'take' structure detection. These indices allow, for
example, formulation of queries in terms understood by the user, such as scene
breaks, turning points and
climaxes, presence and absence of actors, and other scene and sequence-level
motifs, such as dialogs and chases. Other work has focused on discovering
didactic structures in educational and documentary film.
Browsing and Summarization
An interesting application of CMA is to the task of summarizing existing media.
Summarization can be thought of as re-production, i.e. transforming existing
media to achieve a different goal, in this case reduced viewing time or space.
CMA is applicable in this case both for
identifying salient portions of media (to retain) and for finding
representations of that media that will support the new goal in the altered
context. Such a process could be termed aesthetic-perceptual compression.
IUI 2008 Short Paper submission Demonstration Video[Quicktime][AVI]
New Demo Coming Soon...
**Disclaimer: This demonstration includes a very small portion (< 5%) of the movie The Truman Show ((c) Paramount Pictures) for research purposes under the fair use provision. If you are the copyright holder and believe your rights are being infringed, please contact the webmaster and it will be promptly removed.
Story Weaving
Conceptually, if algorithms to detect aesthetic elements are turned 180 degrees
we obtain the ability to embed aesthetic qualities in media. In that case raw,
unproduced media can be given production qualities. This is an exciting
application of CMA given that the vast majority of digital capture devices
today are used by amateurs. Previous work has looked to use the tight coupling
of computation with digital capture devices (digital cameras, smartphones,
PDAs) to provide help to the amateur at capture time. We began with an
authoring paradigm that required the creation of a template of events prior to
filming an occasion. The system has evolved to a fully mobile and dynamic
solution in the form of a smart camera called eMediaTE (embedded Media To
Everyone). The camera takes input from the user about their situation
and goal, and provides a suggestive interface during filming, and ultimately
edits footage into a coherent video automatically, complete with audio and
transition effects, ready to be archived or shared wirelessly. Footage
suggestions and the editing process are driven by underlying algorithms
targeting diverse media aesthetics. The resulting artifact requires no desktop
editing, and comes with rich indices by virtue of the filming process that are
otherwise difficult to obtain (such as subjects), which make the material
easier to find again amid the volume of personal archives.
