Media Bias Detection using Deep Learning Libraries in Python

All data was acquired from All the News dataset created by Andrew Thomson. It is freely available and you can download it anytime. It is separated into three large CSV files, all containing a table that looks like this:

p = 0.4
df = pd.read_csv('articles.csv',header=None,skiprows=lambda i: 1>0 and random.random() > p)

n_s = list(n_s_b.iloc[:,9].values) + list(n_s_p.iloc[:,9].values) \
 + list(n_s_a.iloc[:,9].values) + list(n_s_n.iloc[:,9].values)

n_s = [word.replace('New York Post','') for word in n_s]
n_s = [word.replace('Breitbart','') for word in n_s]
n_s = [word.replace('New York Times','') for word in n_s]
n_s = [word.replace('Atlantic','') for word in n_s]

e_All = similarity_matrix(n_s)

I don’t want to take too much time explaining how to build a Neural Network, there are many hands-on articles published in Towards Data Science and many other sources where you can read and follow very good tutorials to do so³ ⁴ ⁵. In here I will simply present an architecture that yielded good results. This architecture is one of many previous iterations and one that I personally found to work. With that said, let’s dig into it! The classifier I built looks like this:

It has an input layer of 512 neurons (one per embedding value) and two hidden layers of 40 neurons, each with a dropout layer with a fixed dropout fraction of 0.25. It also has an output layer using a Softmax activation function with four neurons to classify Media Outlets and two neurons to classify Media Bias (not show here). In terms of code it looks like this:

For both scenarios (outlets and bias) all parameters are the same except for the optimizer’s learning rate and of course the amount of neurons in the output layer (either four or two). Intuitively it makes scene that simpler solutions (binary in the case of bias) should be optimized faster. For this reason I decided to use a learning rate of 0.00015 for Outlet classification and 0.0005 for Bias classification.

Finally and before checking results, we need to split data into training and test set. For this we will used a Stratified Shuffle Splitter , so this looks like this:

With this done, we only have to train the network. We can do this with:

And now we are ready to check the results!

Let’s look at Outlet classification first. After training (about 15 min on a 8-core CPU machine, no GPU needed), the validation and training accuracy curves look like this (code for visualizing is provided later in the text):

What does it mean? Well, first of, Accuracy of the validation set is almost 0.80, which can be considered good. Remember, this is the accuracy from four Media Outlets! Meaning that this Neural Network can identify with relatively good performance the source of the News being reported based only in written semantic content , very cool! Let’s now look at the Loss to validate if our model is properly fitted:

Since both curves “look” the same (the difference between them is minimal), we can say that our Neural Network has learned rules that are general enough to avoid overfitting, but specific enough to yield good performance on a validation set, a quality of Deep Learning that is always desirable⁶. With this in mind, we can now look at the confusion matrix to go deeper into the results:

It is clear that News stories from Breitbart and The New Work Post were classified with higher accuracy, but this could be due to the fact that there is simply a higher representation of these two classes. We could try resampling methods to balance the classes, but for now I would leave it this way and get back to resampling for our binary classification task described in the next paragraphs.

Moving forward, we can explore the results for Media Bias classification. Remember, this is a binary classifier (either inclined to the Left or to the Right). First, accuracy and loss curves:

Very impressive if you ask me! We got an Accuracy of almost 0.9, which means that effectively, we are identifying with a very good performance if a News Story is biased towards the Left or to the Right. What about the confusion matrix?

Looks good, but it is clear that Left-leaning reports are underrepresented and hence the accuracy is not fully reflecting what is going on here. Accuracy alone can be misleading. Another metric that can be helpful for uneven class distributions is the F1 score⁷. We can very easily compute it with sklearn’s built-in function, so simply doing this does the trick:

from sklearn.metrics import f1_score
f1_score(y_test,news_DNN.predict_classes(X_test))

Which throws:

>>> 0.78

Good, but not perfect. We can try balancing the classes in the training set only⁸ with an oversampler to get better results. I will use SMOTE from the amazing Python library imblearn , so simply add this line just before fitting:

After training with balanced data, the confusion matrix now looks like this:

More balanced. What about F1 and Accuracy? 0.80 and 0.88 respectively. Very good!

NOTE: All visualization was done using seaborn . Here’s the code I wrote to create all the figures:

In this text, I walked you through the steps I took to build, train and visualize a News Outlet/Bias classifier, which yielded very good results. We were able to get fair accuracy scores for Outlet classification, which suggests that the nature of text alone might contain information about where a given Story was written. It is difficult to draw precise conclusions with a sample this size, but the idea is there, we can use Keras, TensorFlow and visualization libraries in Python to classify News Stories. What was more remarkable is the fact that Bias can be identified with very good performance, meaning that to some extent, semantic rules in written Media implicitly contain a political Bias. In a way we knew this from the beginning, but it is nonetheless interesting to note that this information is there and that with the proper tools can be automatically identified. What other problems you think this approach could solve?

Now that you have the tools you can copy the code and use it in your own project.

Thanks for reading!