Emotion Recognition Using Graph Convolutional Networks

Classifying Conversations using Graphs

Recently, deep learning has made much progress in natural language processing (NLP). With many new inventions such as Attention and Transformers leading to state of the art models such as BERT and XLNet, many tasks such as textual emotion recognition have become easier. This article will introduce a new method to conduct emotion recognition from conversations using graphs.

What is Emotion Recognition?

Simply put, emotion recognition (ERC) is the task of classifying the emotion behind a piece of written task. Given a piece of text can you tell if the speaker is angry, happy, sad, or perhaps confused? It has many far-ranging applications in healthcare, education, sales, and human resources. At the highest level, the task of ERC is useful because many are convinced that it is a stepping stone to building a conversationally intelligent AI that is able to talk with a human.

An example of using ERC to power a health assistant (image taken from [1])

Currently, the two major pieces of innovation that most ERC is built on are recurrent neural networks (RNN) and attention mechanisms. RNNs such as LSTMs and GRUs look at text sequentially. When the text is long, the model’s memory of the beginning parts is lost. Attention mechanisms solve this well by weighing different parts of the sentence differently.

However, RNNs+Attention still have trouble taking into context of personality, topic, and intent from neighboring sequences and also the speaker (basically all the very important parts of any conversation). Couple this with the lack of labeled benchmark datasets for personality/emotion it becomes really difficult not just to implement but also to measure the result of new models. This article will summarize a recent paper that solves much of this by using a relatively new innovation called graph convolution networks: DialogueGCN: A Graph Convolutional Neural Network for Emotion Recognition in Conversation [1].

Context Matters

In a conversation, context matters. A simple “Okay” can mean “Okay?”, “Okay!” or “Okay….” depending on what you and the other said before, how you are feeling, how the other person is feeling, the level of tension and many other things. There are two types of context that matter:

1. Sequential Context: The meaning of a sentence in a sequence. This context deals with how past words impact future words, the relationship of words, and semantic/syntactic features. Sequential context is taken into account in RNN models with attention.
2. Speaker Level Context: Inter and intra-speaker dependency. This context deals with the relation between the speakers as well as self-dependency: the notion that your own personality change and affect how you speak during the course of the conversation.

As you can probably guess, it’s speaker level context that most models have difficulty taking into account. It turns out you can actually model speaker level context very well using graph convolutional neural networks and this is exactly the approach DialogueGCN takes.

Representing Conversations as Graphs

In a conversation there are M speakers/parties represented as p[1] , p[2] , . . . , p[M] . Each utterance (a piece of text someone sends) is represented as u[1] , u[2] , . . . , u[N] . The final goal in ERC is to accurately predict each utterance as one of happy, sad, neutral, angry, excited, frustrated, disgust, or fear.

The entire conversation can be built as a directed graph:

A conversation graph with 2 speakers and 5 utterances

G = (V, E, R, W)

• Utterances are the nodes ( V ). Edges are paths/connections between the nodes (E) . Relations are the different types/labels of the edges (R) . Weights represent the importance of an edge (W) .
• Every edge between two nodes v[i] and v[j] has two properties: the relation ( r ) and weight ( w ). We’ll talk more about this in just a bit.
• The graph is directed . Therefore, all edges are one way. The edge from v[i] to v[j] is different from the edge that goes from v[j] to v[i] .

One thing to notice from the figure is that each utterance has an edge connected to itself. This represents the relation of the utterance to itself. In more practical terms, this is how an utterance impacts the mind of the utterance’s speaker.

Context Window

One major problem of a graph representation is that if a conversation is really long, there can be many edges for a single node. And since each node is connected to every other node, it scales quadratically as the size of the graph increases. This is very computationally expensive.

To resolve this problem, in DialogueGCN, the graph’s edges are constructed based on a context window with a specific size . So for a node/utterance i , only the past size and future size utterances are connected in a graph (any node in the range of i- size to i+ size ).

A context window of size 1 around the third utterance

Edge Weights

Edge weight are calculated using an attention function. The attention function is set up so that for each node/utterance, the incoming edge weights all sum up to 1. Edge weights are constant and do not change in the learning process.

In simplified terms, the edge weight represents the importance of the connection between two nodes.

Relations

The relation of an edge depends on two things:

• Speaker dependency: Who spoke u[i] ? Who spoke v[j] ?
• Temporal dependency: Was u[i] uttered before u[j] , or the other way around?

In a conversation, if there are M different speakers, there will be a maximum of M (speaker of u[j] ) * M (speaker of u[j] ) * 2 (whether u[i] occurs before u[j] , or the reverse) = 2 M ² relations.

We can list all of the relations from our example graph above:

A table of all the possible relations in our example

Here’s the same graph with the edges’ relation labelled according to the table:

Edges are labelled with their respective relation (refer to table above)

In our example, we have 8 different relation. At a high level, relation is an important property for the edge because who spoke what and when matters a lot in a conversation. If Peter asks a question and Jenny responds, this is different from Jenny first saying the answer and then Peter asking the question (temporal dependency). Likewise, if Peter asks the same question to Jenny and Bob, they may respond differently (speaker dependency).

Think of the relation as defining the type of the connection, and the edge weight representing the importance of the connection.

Think of the relation as defining the type of the connection, and the edge weight representing the importance of the connection.

The Model

The DialogueGCN model uses a type of graph neural network known as a graph convolutional network (GCN).

Just like above, the example shown is for a 2 speaker 5 utterance graph.