In this presentation I will discuss recent insights into both the time course of pragmatic processing and the key neural infrastructure for inferring speaker meaning from coded meaning. I will show why mirror neurons are not able to handle pragmatic information. In addition, I will present evidence for the role of the Theory of Mind (ToM) network in processing of pragmatic information.

Empathy is a complex cognitive ability based on the reasoning of others' affective states. In order to better understand others and express stronger empathy in dialogues, we argue that two issues must be tackled at the same time: (i) identifying which word is the cause for the other's emotion and (ii) reflecting those specific words in the response generation. However, existing approaches for recognizing emotion cause words in text require sub-utterance level annotations, which is demanding. Taking inspiration from social cognition, we leverage a generative estimator to infer emotion cause words from utterances with only emotion labels. We show our approach improves best performing dialogue agents on generating more focused empathetic responses in terms of both automatic and human evaluation.

We inspect complement coercion sentences (she finished the coffee or he started a book) as a case study for modeling open-ended pragmatic interpretation. Existing computational work treats complement coercion interpretation as a task of choosing a single best-fit verb (she finished drinking the coffee). We instead present crowdsourcing and modeling data that supports broadening the predicted classes to better capture naturalistic interpretation.

Progress on language generation has experienced a huge boost with the advent of large models trained on huge amounts of text. However, this kind of language modelling will only take us that far. Most natural language use is driven by communicative goals and is often grounded both in the conversational context and in extralinguistic information. Can we take inspiration from human production strategies in situated environments to drive forward natural language generation models? I will argue that yes, we can, and present a few examples of recent and ongoing research carried out within my group that follow this research programme.

In this talk, I'll argue that human-like language use in a variable and non-stationary social environment requires a more radical shift in our models of meaning. People not only rely on pragmatic reasoning to enrich static literal meanings, but flexibly create new literal meanings together to suit the task at hand. In other words, the central computational problem of communication is not simply transmission in context, as in classical formulations, but continual learning within and across social contexts. As a case study, I'll present a physical assembly task where pairs of human participants worked together to reconstruct block towers. We found that human participants rapidly coordinated on new, more abstract language that captured each scene’s underlying structure. Motivated by these findings, we extend recent hierarchical models of convention formation with a Bayesian program learning module. This model suggests a path toward more adaptive language models that are able to 'find the right words for the job' and collaborate with human partners in a wider variety of novel contexts.

Exploiting a loophole, taking advantage of the ambiguity of language to do what someone says but not what they want, is a familiar facet of fable, law, and everyday life. Engaging with loopholes requires a nuanced understanding of goals, social ambiguity, and value alignment. Scientifically, the development of loopholes can help us better understand human communication, and design better human-AI interactions. However, cognitive research on this behavior remains scarce. A survey of parents reveals that loophole behavior is prevalent, frequent, and diverse in daily parent-child interactions, emerging around ages five to six. A further experiment shows that adults consider loophole behavior as less costly than non-compliance, and children increasingly differentiate loophole behavior from non-compliance from ages four to ten. We discuss the implications and limitations of the current work, together with a proposal for a formal framework for loophole behavior.

Consumers of image descriptions want them to be context-sensitive, but previous crowdsourced efforts to create text from images have presented the images in isolation. We tested whether untrained crowdworkers naturally take context into account when writing image descriptions by asking them to write descriptions for images that we embedded in the first paragraph of a Wikipedia article. Our analysis shows that the produced descriptions were statistically significantly more likely to reflect contents of the article they were presented with than those of mismatched articles. These findings have implications on the extent and usefulness of training crowdworkers when developing large scale context-sensitive description corpora, as well as the development of deep learning models for automatic description generation.

Everyday conversation comes with an important affordance: interaction. Amongst other forms of metacommunication, interaction allows for the use of other-initiated repair: where a receiver signals trouble in understanding a producer's utterance, thereby prompting the producer to repeat or clarify. This phenomenon is ubiquitous in everyday conversation, but its affordance has largely been ignored in computational models of language use and language evolution. In this talk, I explore what happens when we add other-initiated repair to (i) a model of disambiguation in language use, and (ii) a model of the cultural evolution of compositional structure in language. In the first case study, we show that interactive repair may help outsource some of the computational resource demands of pragmatic reasoning to interaction (where disambiguation takes place across multiple turns). In the second case study, we show that interactive repair may play a role in 'locking in' compositional structure over generations in the cultural evolution of language.

We will take a whirlwind, mile high, tour of the literature on the moment-to-moment processing of two simple quantity implicatures: scalar implicatures (avoidance of underinformative statements) and the inference that adjectives will be used contrastively (avoidance on overinformativity). On the basis of the scalars, I will propose that there are two routes by which implicatures are calculated: a slow bottom-up route and top-down route that leads the appearance of instantaneous implicature. This top-down route relies on the speaker’s conceptualization of the context in linguistically relevant terms. This analysis makes some novel predictions about the role of speaker modelling in the adjective inference. I’ll present unpublished data that support these new predictions.

Providing examples is one of the most common way for end-users to interact with program synthesizers. However, program synthesis systems assume that examples consistent with the program are chosen at random, and do not exploit the fact that users choose examples pragmatically. Prior work modeled program synthesis as pragmatic communication, but required an inefficient enumeration of the entire program space. In this paper, we show that it is possible to build a program synthesizer that is both pragmatic and efficient by approximating the joint distribution of programs with a product of independent factors, and performing pragmatic inference on each factor separately. This naive factored distribution approximates the exact joint distribution well when the examples are given pragmatically, and is compatible with a very simple neuro-symbolic synthesis algorithm.

Verbal communication is an ubiquitous aspect of human interaction occurring in many contexts; however, it is primarily studied in the limited context of two people communicating information. Understanding communication in complex, multi-party interactions is both a scientific challenge for psycholinguistics and an engineering challenge for creating artificial agents who can participate in these richer contexts. We adapted the reference game paradigm to an online 3-player game where players refer to objects in order to coordinate selections based on the available utilities. We ran games with shared or individual payoffs and with or without access to language. Our paradigm can also be used for artificial agents; we trained reinforcement learning-based agents on the same task as a comparison. Our dataset shows the same patterns found in simpler reference games and contains rich language of reference and negotiation.

Task-oriented dialog is inherently about contextual action: users address the system from a specific context and the system must decide what to do in response. This talk will present some of the core principles of the Semantic Machines team's approach to conversational AI: program synthesis for action prediction, compositionality for handling complex tasks, metacomputation for reference and revision, error handling for dialog management, and dynamic generation for truthful output. I will also mention ways in which real-world constraints can help to inform the design of conversational systems.