Exploratory Data Analysis in R for beginners

A Step-by-step Approach From Data Importing to Cleaning and Visualization

Photo from: NicoElNino, Getty Images/iStockphoto

Exploratory Data Analysis( EDA ) is the process of analyzing and visualizing the data to get a better understanding of the data and glean insight from it. There are various steps involved when doing EDA but the following are the common steps that a data analyst can take when performing EDA:

1. Import the data
2. Clean the data
3. Process the data
4. Visualize the data

What would you expect to find in this article?

This article focuses on EDA of a dataset, which means that it would involve all the steps mentioned above. Therefore, this article will walk you through all the steps required and the tools used in each step. Therefore, you would expect to find the followings in this article:

1. Tidyverse package for tidying up the data set
2. ggplot2 package for visualizations
3. corrplot package for correlation plot
4. Some other basic functions to manipulate data like strsplit (), cbind (), matrix () and so on.

For beginners to EDA, if you do not have a lot of time and do not know where to start , I would recommend you to start with Tidyverse and ggplot2 . You can do almost everything about EDA with these 2 packages. There are various resources online like DataCamp , Setscholars , and books like Introduction to Data Science and so on.

In this article, I would walk you through the process of EDA through the analysis of the PISA score dataset which is available here .

Let’s get started, folks!!

Importing the data

Before importing the data into R for analysis, let’s look at how the data looks like:

When importing this data into R, we want the last column to be ‘numeric’ and the rest to be ‘factor’. With this in mind, let’s look at the following 3 scenarios:

df.raw <- read.csv(file ='Pisa scores 2013 - 2015 Data.csv', fileEncoding="UTF-8-BOM", na.strings = '..')df.raw1 <- read.csv(file ='Pisa scores 2013 - 2015 Data.csv')df.raw2 <- read.csv(file ='Pisa scores 2013 - 2015 Data.csv',na.strings = '..')

These are 3 ways of importing the data into R. Usually, one with go for the df.raw1 because it seems to be the most convenient way of importing the data. Let’s see the structure of the imported data:

df.raw1 <- read.csv(file ='Pisa scores 2013 - 2015 Data.csv')str(df.raw1)

There are 2 problems that we can spot immediately. The last column is ‘factor’ and not ‘numeric’ like what we desire. Secondly, the first column ‘Country name’ is encoded differently from the raw dataset.

Now let’s try the second case scenario

df.raw2 <- read.csv(file ='Pisa scores 2013 - 2015 Data.csv',na.strings = '..')str(df.raw2)

The last column is now ‘numeric’. However, the name of the first column is not imported correctly.

What about the last scenario?

df.raw <- read.csv(file ='Pisa scores 2013 - 2015 Data.csv', fileEncoding="UTF-8-BOM", na.strings = '..')
str(df.raw)

As you can see, the first column is now named properly and the last column is ‘numeric’.

na.strings = '..'

This allows R to replace those blanks in the dataset with NA. This will be useful and convenient later when we want to remove all the ‘NA’s.

fileEncoding="UTF-8-BOM"

This allows R, in the laymen term, to read the characters as correctly as they would appear on the raw dataset.

Cleaning and Processing the data

install.packages("tidyverse")
library(tidyverse)

We want to do a few things to clean the dataset:

1. Make sure that each row in the dataset corresponds to ONLY one country: Use spread() function in tidyverse package
2. Make sure that only useful columns and rows are kept: Use drop_na() and data subsetting
3. Rename the Series Code column for meaningful interpretation: Use rename()

df <- df.raw[1:1161, c(1, 4, 7)] %>%      # Keep useful columns
  spread(key=Series.Code, value=X2015..YR2015.) %>%  
  rename(Maths = LO.PISA.MAT,                        
         Maths.F = LO.PISA.MAT.FE,
         Maths.M = LO.PISA.MAT.MA,
         Reading = LO.PISA.REA,
         Reading.F = LO.PISA.REA.FE,
         Reading.M = LO.PISA.REA.MA,
         Science = LO.PISA.SCI,
         Science.F = LO.PISA.SCI.FE,
         Science.M = LO.PISA.SCI.MA
         ) %>%
  drop_na()

Now let’s see how the clean data looks like:

view(df)

Visualizing the data

1. Barplot

install.packages("ggplot2")
library(ggplot2)#Ranking of Maths Score by Countriesggplot(data=df,aes(x=reorder(Country.Name,Maths),y=Maths)) + 
  geom_bar(stat ='identity',aes(fill=Maths))+
  coord_flip() + 
  theme_grey() + 
  scale_fill_gradient(name="Maths Score Level")+
  labs(title = 'Ranking of Countries by Maths Score',
       y='Score',x='Countries')+ 
  geom_hline(yintercept = mean(df$Maths),size = 1, color = 'blue')

Similarly, we can rank by science score and reading score too, just change the name accordingly.

2. Boxplot

If we use the dataset above, we will not be able to draw a boxplot. This is because boxplot needs only 2 variables x and y but in the cleaned data that we have, there are so many variables. So we need to combine those into 2 variables. We name this as df2

df2 = df[,c(1,3,4,6,7,9,10)] %>%   # select relevant columns 
  pivot_longer(c(2,3,4,5,6,7),names_to = 'Score')view(df2) 

Great! Now we can make boxplots

ggplot(data = df2, aes(x=Score,y=value, color=Score)) + 
  geom_boxplot()+
  scale_color_brewer(palette="Dark2") + 
  geom_jitter(shape=16, position=position_jitter(0.2))+
  labs(title = 'Did males perform better than females?',
       y='Scores',x='Test Type')

geom_jitter() allows you to plot the data points on the plot.

You can play around with the code above to get various plots. For example, I can change from ‘color = Score’ to ‘fill=Score’:

ggplot(data = df2, aes(x=Score,y=value, fill=Score)) + 
  geom_boxplot()+
  scale_fill_brewer(palette="Green") + 
  geom_jitter(shape=16, position=position_jitter(0.2))+
  labs(title = 'Did males perform better than females?',
       y='Scores',x='Test Type')

The plot looks a bit messy. A better visualisation would be to separate Subjects and Genders and plot them side by side.

How can we do that?

Since we want to separate Subjects and Genders from a column containing ‘Subject.Gender’ (e.g. Maths.F), we need to use strsplit () to do this job for us

S = numeric(408)     # create an empty vector
for (i in 1:length(df2$Score)) {
  S[i] = strsplit(df2$Score[i],".",fixed = TRUE)
}

Now S is a list of 408 components, each of which has 2 sub-components ‘Subject’ and ‘Gender’. We need to transform S into a data frame with 1 column of Subject and 1 column of Gender . We will name this data frame as df3

df3 = S%>%unlist() %>% matrix(ncol = 2, byrow = TRUE)%>% as.data.frame()view(df3)

We now need to combine this df3 with df2 we created earlier, and name the result as df4

df4 = cbind(df2,df3) 
colnames(df4) = c('Country','Score','Value','Test','Gender')
df4$Score = NULL # since the 'Score' column is redundant
view(df4)

Awesome! Now the data looks clean and neat. Let’s create multiple plots with the use of facet_wrap() function in ggplot2

ggplot(data = df4, aes(x=Test,y=Value, fill=Test)) + 
  geom_boxplot()+
  scale_fill_brewer(palette="Green") + 
  geom_jitter(shape=16, position=position_jitter(0.2))+
  labs(title = 'Did males perform better than females?',
       y='Scores',x='Test')+
facet_wrap(~Gender,nrow = 1)

Here, we categorized the plot by Gender, hence facet_wrap(~Gender,nrow = 1)

We can also categorize the plot by Test by changing facet_wrap(~Test,nrow = 1)

ggplot(data = df4, aes(x=Gender,y=Value, fill=Gender)) + 
  geom_boxplot()+
  scale_fill_brewer(palette="Green") + 
  geom_jitter(shape=16, position=position_jitter(0.2))+
  labs(title = 'Did males perform better than females?',
       y='Scores',x='')+
facet_wrap(~Test,nrow = 1)

By looking at these plots, we can make some insights about the performance of males and females. Generally, males performed better in Science and Maths, but females performed better in Reading. However, it would be naive for us to make a conclusion only after looking at the boxplot. Let’s dive deeper into the data and see any other insights we can get after manipulating the dataset.

Since I want to compare the performance of Males and Females in each subject across all participating countries, I would need to calculate the % difference in terms of the score for each subject between males and females and then plot it out to visualize.

How would I do this in R?Use mutate() function

Let’s look at the original clean data set that we have, df

We will now use the mutate() function to calculate, for each country, the % difference between Males and Females for each subject.

df = df %>% mutate(Maths.Dif = ((Maths.M - Maths.F)/Maths.F)*100,
         Reading.Dif = ((Reading.M - Reading.F)/Reading.F)*100,
         Science.Dif = ((Science.M - Science.F)/Science.F)*100,
         Total.Score = Maths + Reading + Science,
         Avg.Dif = (Maths.Dif+Reading.Dif+Science.Dif)/3
         )view(df)

Now let’s plot this out to visualize the data better

##### MATHS SCORE #####
ggplot(data=df, aes(x=reorder(Country.Name, Maths.Dif), y=Maths.Dif)) +
  geom_bar(stat = "identity", aes(fill=Maths.Dif)) +
  coord_flip() +
  theme_light() +
  geom_hline(yintercept = mean(df$Maths.Dif), size=1, color="black") +
  scale_fill_gradient(name="% Difference Level") +
  labs(title="Are Males better at math?", x="", y="% difference from female")

This plot represents the % difference in scores using Females as reference . A positive difference means males scored higher, while a negative difference means males scored lower . The black line represents the mean difference across all countries.

Some interesting insights one can draw from here:

• In general, Males performed better than Females in Maths, especially in most of the DCs, males scored generally higher than females
• Interestingly, in Singapore and Hongkong, females and males performed equally well, with the difference in score is around 0 in each of these countries. This is good insight maybe for policy-makers because we do not want a huge gap between males and females performance in education.

We can do the same thing for Reading and Science score.

3. Correlation Plot

df = df[,c(1,3,4,6,7,9,10)] #select relevant columns

To create correlation plot, simply use cor():

res = cor(df[,-1]) # -1 here means we look at all columns except the first columnres

We can calculate p-value to see whether the correlation is significant

install.packages("Hmisc")
library("Hmisc")res2 <- rcorr(as.matrix(df[,-1))

The smaller the p-value, the more significant the correlation.

The purpose of this section about correlation plot is to introduce to you how to calculate correlation between variables in R. For this dataset, it is obvious that all the variables are correlated

To visualize

install.packages("corrplot")

library(corrplot)
corrplot(res, type = "upper", order = "hclust", 
         tl.col = "black", tl.srt = 45)

The stronger the color and the bigger the size, the higher the correlation. The result is similar to the one we got earlier: All the variables are intercorrelated.

That is it! Hope you guys enjoyed and picked up something from this article. While this guide is not exhaustive, it will more or less give you some ideas of how to do some basic EDA with R.

If you have any questions, feel free to put them down in the comment section below. Thank you for your read. Have a great day and happy programming!!!