In this chapter, we will discuss how our understanding of narratives and the experiences communicated in novels, poems, essays and folk tales is formed and shaped by the material objects we interact with and how the basic media types of speech, text and images are integrated differently according to the conventionsof different qualified media types. We can approach the question of themediality of literature from different directions. Different material technologies that are used for writing and reading define and transform literature. We will consider the activities that define literature, such as the reading and writing of text and how text and images can be integrated. Finally,we will also consider how social and cultural settings, norms and institutions construct notions of what is and what is not (considered to be) literature. All three aspects are subject to more change and variation than the prototypical image of ashelf filled with books may suggest

The main purpose of this work is to provide a semiotic ontology for redescription of active cognitive externalism, recently developed by the paradigm 4E (embodied, embedded, enactive, extended cognition). In our approach, distributed cognitive systems (DCSs) are described as semiosis, signs in action. We explored the relationship between semiosis and cognition, as conceived by C. S. Peirce, in association with the notion of distributed cognitive system (DCS). We introduce Peircean externalist approach with an emphasis on the notion of temporal distribution of semiosis, and describe DCSs, and their elements, as "sign in action". To develop this argument, we describe an example of DCS -- verbal-musical improvisation of repente, repentismo, or viola singing. It is a phenomenon of verbal-musical improvisation that takes the form of a challenge in versified oral poetry. We describe the phenomenon as the embodiment of the formal structure of a cognitive task and an inferential process. This embodiment corresponds to a semiotization of the repente performances as DCSs. The temporally distributed tendency of repentismo organizes the DCS as a system that performs metasemiotic experiments on the action of signs.

Intersemiotic translation can be described as a “cognitive pump”, a cognitive artefact, or a thinking tool that is designed to scaffold and distribute artistic creativity. Thinking tools (physical and virtual tools employed by cognitive systems) are part of the material and cultural niches of human cognition. We describe intersemiotic translation as such a thinking tool. How does it work? As an anticipatory augmented intelligence technique, intersemiotic translation works as an anticipatory and predictive tool; it anticipates new, unexpected events and patterns of semiotic behaviour and keeps the emergence of new patterns under control. From this perspective, intersemiotic translation decreases the cost of choice for an agent (artist) operating in a cognitive niche by increasing the predictability of the emergence of patterns of semiotic behaviour in that niche. At the same time, it works as a generative model, providing new, unexpected, surprising data in the target system and affording competing results that allow the system to generate candidate instances. From this perspective, intersemiotic translation increases the complexity of the cognitive niche. We explore these ideas here, taking advantage of an example of an intersemiotic translation of Gertrude Stein’s experimental prose to theatrical dance.

The central idea of this thesis is that the relationship between cognition,media and environments is regulated by surprise. The relationship between cognition, media and environments is a foundational problem for studies of cognition, culture and/or communication. This thesis introduces a view on this relationship, based on the concept of surprise within the framework of Peircean process semiotics. This framework consists of an original interpretation and application of Peirce's semiotics oriented by premisesfrom process philosophy. Peirce's semiotics conceives of communication,cognition and one's relation to the environment in terms of triadic signaction, or semiosis. “Triadic” here expresses a skepticism of any dualism (mind/matter, subject/object, representation/represented object...). Process philosophy is a worldview and a mode of thinking based on the notion of dynamic and emergent processes. Peircean process semiotics entails two positions: semiotic processualism and active semiotic externalism. In semiotic processualism, semiosis is a dynamic and temporally distributedprocess. Semiosis interweaves present, past, and future according to ageneral and recursive temporality. In active semiotic externalism, agency in semiosis is external to the agents themselves, distributed in cognitive niches of organisms, artifacts, and environmental conditions.

Semiosis depends on and responds to an underlying logical situation of surprise. Surprise underlies and regulates temporal distribution of semiosisand the externalist agency of semiosis. I illustrate these notions withexamples from Brazilian poetry. From the viewpoint of semioticprocessualism, surprises are crises in the temporal distribution of semiosis,when the recursive time of habit and anticipation is disturbed. I illustrate this view with concrete poetry, a form of avant-garde poetry. Against versified poetry, Brazilian concrete poets pursued a crisis in the time of the poem and in the historical time of poetry. From the viewpoint of activesemiotic externalism, surprises are shifts in agency, when the roles of agent and patient change within a cognitive niche. I illustrate this view with Repente, a tradition of improvisational and competitive poetic dialogues. Repente produces surprise for the poets and for the audience through thejoint dynamic action of versification procedures, guitar playing and singing.

The notions of "multilevel", "level of description and organization" are quite familiar for the studies of complex systems. Examples include physico-chemical, biological computational, socio-economic, historical, and cultural systems. Poems are complex phenomena and can be described as multilevel hierarchical systems. Different levels of description of a poem correspond to different sets of components, structures and processes (e.g., context, lexicon, syntax, prosody, rhythm, typography) that asymmetrically restrict each others. We suggest the notion of "multilevel translation" to characterize an operation whose focus is a solution of a situated problem. The operation involves selection and re-creation of a hierarchical multilevel system of constraints. This perspective allows us to approach the translation of a poem as the solution of a multilevel, situated and ill-defined problem. We describe the steps of this operation, and we exemplify it by the translation of the poem "The Expiration", by Augusto de Campos.

We explore Peirce's pragmatic conception of sign action, as a distributed and emergent view of cognition and exemplify with the emergence of classical ballet. In our approach, semiosis is a temporally distributed process in which a regular tendency towards certain future outcomes emerges out of a history of sign actions. Semiosis self-organizes in time, in a process that continuously entails the production of more signs. Emergence is a ubiquitous condition in this process: the translation of signs into signs cannot be inferred from the properties of the components of a semiotic triad alone, but has to take into account a complex interaction between a micro-semiotic and macro-semiotic level of description. This interaction can be understood as an interplay of potentialities and tendencies, or upward constitutive determinative relations and downward selective determinative relations. According to this view, emergence is a central defining condition of processes of meaning. Ballet is a sign in action. The emergence of classical ballet is a self-regulatory process, in which a system of different kinds of cognitive artifacts (musical, bodily/motor, spatial/architectonic) and agents obtained a stable semiotic relation throughout many phases of development between the 16th and the 19th Century. One case is the development of the verticality of dance in classical ballet as a semiotic relation connecting proscenium arch stages, dancing bodies, and audiences. This development is micro-semiotically determined by the spatial constraints of the proscenium arch stage, and macro-semiotically determined by a historical construction of the dancing body as a sign within a network of semiotic chains, such as the intersemiotic regulation of body of the dancer by principles coming from painting. This is not only the emergence of actual meaning, but also the emergence of an open-ended field of potential and general meanings, an autonomous tendency of development. To say that ballet, as sign action, emerges, is to say that cognitive artifacts such as dancer's bodies, stages and audience's point of view, musical compositions, costumes, all sorts of supporting institutions, etc, constitute a niche for sign action, interacting according to tendencies of development that didn't exist before.

Intersemiotic translation (IT) can be described as a cognitive artifact, designed to distribute artistic creativity. Cognitive artifacts are part of material and cultural niches of human cognition. They have different forms and can be used in many different activities. Their varied morphology includes “material and mental” structures (Norman 1993), “designed for and opportunistic” entities (Hutchins 1999), and “transparent and opaque” processes (Clark 2004). For several authors, cognition is full of cognitive artifacts; even more radically, cognition is a network of artifacts. For many artists, intersemiotic translation is one of these tools. But what is its ontological nature? And how does intersemiotic translation work? As an augmented intelligence technique, intersemiotic translation works as a generative model, providing new, unexpected, surprising data in the target system and affording competing results that allow the system to generate candidate instances. To describe this process, we introduce a model of intersemiotic translation based on Peirce’s mature semeiotic. At the end of the chapter, we speculate about the role that abductive inference can have in the process of generating new ideas in an artistic domain. What we have done here must be considered a preliminary tentative model of intersemiotic translation as a cognitive artifact to externalize creativity.

Intersemiotic translation (IT) can be described as a cognitive artefact designed as a predictive, generative, and metasemiotic tool that distributes artistic creativity. Cognitive artefacts have a huge variety of forms and are manipulated in many different ways and domains. As a projective augmented intelligence technique, IT works as a predictive tool, anticipating new, and surprising patterns of semiotic events and processes, keeping under control the emergence of new patterns. At the same time, it works as a generative model, providing new, unexpected, surprising data in the target-system,​​ and affording competing results​ ​which allow the system to generate candidate instances. As a metasemiotic tool, IT creates a metalevel semiotic process, a sign-action which stands for the action of a sign. It creates an ‘experimental laboratory’ for performing semiotic experiments. IT submits semiotic systems to unusual conditions and provides a scenario for observing the emergence of new and surprising semiotic behaviour as a result. We explore these ideas taking advantage of two examples of ITs to theatrical dance: (1) from one-point visual perspective to classical ballet and (2) from John Cage’s protocols of music indeterminacy to Merce Cunningham’s choreographic composition.

Here we describe Peircean post-1903 semiosis as a processualist conception of meaning, and relate it to contemporary active externalism in Philosophy of Cognitive Science, especially through the notion of cognitive niche construction. In particular, we shall consider the possibility of integrating (a) the understanding of “semiosis as process” within Peirce’s mature semiotics with (b) an elaboration of the concept of cognitive niche from the point of view of niche construction theory and process biology research.

The influence exerted by Peirce’s semeiotic on Translation Studies ‘has been close to nil’. Nothing has yet happened that looks like a ‘semiotic turn in translation studies’. This is surprising. Peirce’s pragmatic model of meaning as the ‘action of signs’ (semiosis) has had a deep impact on philosophy, psychology, theoretical biology, linguistics, and cognitive sciences, besides all branches of semiotics. Why is such an influence not observed in a field of studies so strongly impregnated by semiotic notions, like translation studies? How could such an influence be exerted? Douglas Robinson’s book is about these questions. His book is not a review or analysis of the reasons why a ‘semiotic turn in translation studies’ has never happened. In fact, it is mainly about Dinda Gorlée’s works. Gorlée has forged the most systematic inter-theoretical relationship between Peirce’s semeiotic and Translation Studies. Her papers and books tentatively build an initial step of a Peircean transformation in Translation Studies’ research agenda. In our opinion, if this project has not succeeded yet, Robinson’s book will not accomplish it either. Why? Because it does not explore the implications resulting from a rigorous mapping between fundamental premises, problems, methods and models delineating the research domains. Even so, Douglas Robinson’s book is an important and necessary work to understand the difficulties involved in this project. His main ideas regarding the possibility of an inter-theoretical relationship are found in chapters 1 and 4 (other chapters are presented as case studies and empirical descriptions.) In these chapters, we find (non-systematically) many of Peirce’s ideas on semiosis, phenomenological categories, abductive inference, etc. It is a good supposition that the exploration of these ideas should produce a remarkable set of unprecedented consequences (otherwise it is a useless academic cost).