Exploratory Data Analysis in R for beginners
A Stepbystep Approach From Data Importing to Cleaning and Visualization
Photo from: NicoElNino, Getty Images/iStockphoto
Exploratory Data Analysis( EDA ) is the process of analyzing and visualizing the data to get a better understanding of the data and glean insight from it. There are various steps involved when doing EDA but the following are the common steps that a data analyst can take when performing EDA:
 Import the data
 Clean the data
 Process the data
 Visualize the data
What would you expect to find in this article?
This article focuses on EDA of a dataset, which means that it would involve all the steps mentioned above. Therefore, this article will walk you through all the steps required and the tools used in each step. Therefore, you would expect to find the followings in this article:
 Tidyverse package for tidying up the data set
 ggplot2 package for visualizations
 corrplot package for correlation plot
 Some other basic functions to manipulate data like strsplit (), cbind (), matrix () and so on.
For beginners to EDA, if you do not have a lot of time and do not know where to start , I would recommend you to start with Tidyverse and ggplot2 . You can do almost everything about EDA with these 2 packages. There are various resources online like DataCamp , Setscholars , and books like Introduction to Data Science and so on.
In this article, I would walk you through the process of EDA through the analysis of the PISA score dataset which is available here .
Let’s get started, folks!!
Importing the data
Before importing the data into R for analysis, let’s look at how the data looks like:
When importing this data into R, we want the last column to be ‘numeric’ and the rest to be ‘factor’. With this in mind, let’s look at the following 3 scenarios:
df.raw < read.csv(file ='Pisa scores 2013  2015 Data.csv', fileEncoding="UTF8BOM", na.strings = '..')df.raw1 < read.csv(file ='Pisa scores 2013  2015 Data.csv')df.raw2 < read.csv(file ='Pisa scores 2013  2015 Data.csv',na.strings = '..')
These are 3 ways of importing the data into R. Usually, one with go for the df.raw1 because it seems to be the most convenient way of importing the data. Let’s see the structure of the imported data:
df.raw1 < read.csv(file ='Pisa scores 2013  2015 Data.csv')str(df.raw1)
There are 2 problems that we can spot immediately. The last column is ‘factor’ and not ‘numeric’ like what we desire. Secondly, the first column ‘Country name’ is encoded differently from the raw dataset.
Now let’s try the second case scenario
df.raw2 < read.csv(file ='Pisa scores 2013  2015 Data.csv',na.strings = '..')str(df.raw2)
The last column is now ‘numeric’. However, the name of the first column is not imported correctly.
What about the last scenario?
df.raw < read.csv(file ='Pisa scores 2013  2015 Data.csv', fileEncoding="UTF8BOM", na.strings = '..') str(df.raw)
As you can see, the first column is now named properly and the last column is ‘numeric’.
na.strings = '..'
This allows R to replace those blanks in the dataset with NA. This will be useful and convenient later when we want to remove all the ‘NA’s.
fileEncoding="UTF8BOM"
This allows R, in the laymen term, to read the characters as correctly as they would appear on the raw dataset.
Cleaning and Processing the data
install.packages("tidyverse") library(tidyverse)
We want to do a few things to clean the dataset:
 Make sure that each row in the dataset corresponds to ONLY one country: Use spread() function in tidyverse package
 Make sure that only useful columns and rows are kept: Use drop_na() and data subsetting
 Rename the Series Code column for meaningful interpretation: Use rename()
df < df.raw[1:1161, c(1, 4, 7)] %>% # Keep useful columns spread(key=Series.Code, value=X2015..YR2015.) %>% rename(Maths = LO.PISA.MAT, Maths.F = LO.PISA.MAT.FE, Maths.M = LO.PISA.MAT.MA, Reading = LO.PISA.REA, Reading.F = LO.PISA.REA.FE, Reading.M = LO.PISA.REA.MA, Science = LO.PISA.SCI, Science.F = LO.PISA.SCI.FE, Science.M = LO.PISA.SCI.MA ) %>% drop_na()
Now let’s see how the clean data looks like:
view(df)
Visualizing the data
 Barplot
install.packages("ggplot2") library(ggplot2)#Ranking of Maths Score by Countriesggplot(data=df,aes(x=reorder(Country.Name,Maths),y=Maths)) + geom_bar(stat ='identity',aes(fill=Maths))+ coord_flip() + theme_grey() + scale_fill_gradient(name="Maths Score Level")+ labs(title = 'Ranking of Countries by Maths Score', y='Score',x='Countries')+ geom_hline(yintercept = mean(df$Maths),size = 1, color = 'blue')
Similarly, we can rank by science score and reading score too, just change the name accordingly.
2. Boxplot
If we use the dataset above, we will not be able to draw a boxplot. This is because boxplot needs only 2 variables x and y but in the cleaned data that we have, there are so many variables. So we need to combine those into 2 variables. We name this as df2
df2 = df[,c(1,3,4,6,7,9,10)] %>% # select relevant columns pivot_longer(c(2,3,4,5,6,7),names_to = 'Score')view(df2)
Great! Now we can make boxplots
ggplot(data = df2, aes(x=Score,y=value, color=Score)) +
geom_boxplot()+
scale_color_brewer(palette="Dark2") +
geom_jitter(shape=16, position=position_jitter(0.2))+
labs(title = 'Did males perform better than females?',
y='Scores',x='Test Type')
geom_jitter() allows you to plot the data points on the plot.
You can play around with the code above to get various plots. For example, I can change from ‘color = Score’ to ‘fill=Score’:
ggplot(data = df2, aes(x=Score,y=value, fill=Score)) +
geom_boxplot()+
scale_fill_brewer(palette="Green") +
geom_jitter(shape=16, position=position_jitter(0.2))+
labs(title = 'Did males perform better than females?',
y='Scores',x='Test Type')
The plot looks a bit messy. A better visualisation would be to separate Subjects and Genders and plot them side by side.
How can we do that?
Since we want to separate Subjects and Genders from a column containing ‘Subject.Gender’ (e.g. Maths.F), we need to use strsplit () to do this job for us
S = numeric(408) # create an empty vector for (i in 1:length(df2$Score)) { S[i] = strsplit(df2$Score[i],".",fixed = TRUE) }
Now S is a list of 408 components, each of which has 2 subcomponents ‘Subject’ and ‘Gender’. We need to transform S into a data frame with 1 column of Subject and 1 column of Gender . We will name this data frame as df3
df3 = S%>%unlist() %>% matrix(ncol = 2, byrow = TRUE)%>% as.data.frame()view(df3)
We now need to combine this df3 with df2 we created earlier, and name the result as df4
df4 = cbind(df2,df3) colnames(df4) = c('Country','Score','Value','Test','Gender') df4$Score = NULL # since the 'Score' column is redundant view(df4)
Awesome! Now the data looks clean and neat. Let’s create multiple plots with the use of facet_wrap() function in ggplot2
ggplot(data = df4, aes(x=Test,y=Value, fill=Test)) +
geom_boxplot()+
scale_fill_brewer(palette="Green") +
geom_jitter(shape=16, position=position_jitter(0.2))+
labs(title = 'Did males perform better than females?',
y='Scores',x='Test')+
facet_wrap(~Gender,nrow = 1)
Here, we categorized the plot by Gender, hence facet_wrap(~Gender,nrow = 1)
We can also categorize the plot by Test by changing facet_wrap(~Test,nrow = 1)
ggplot(data = df4, aes(x=Gender,y=Value, fill=Gender)) +
geom_boxplot()+
scale_fill_brewer(palette="Green") +
geom_jitter(shape=16, position=position_jitter(0.2))+
labs(title = 'Did males perform better than females?',
y='Scores',x='')+
facet_wrap(~Test,nrow = 1)
By looking at these plots, we can make some insights about the performance of males and females. Generally, males performed better in Science and Maths, but females performed better in Reading. However, it would be naive for us to make a conclusion only after looking at the boxplot. Let’s dive deeper into the data and see any other insights we can get after manipulating the dataset.
Since I want to compare the performance of Males and Females in each subject across all participating countries, I would need to calculate the % difference in terms of the score for each subject between males and females and then plot it out to visualize.
How would I do this in R?Use mutate() function
Let’s look at the original clean data set that we have, df
We will now use the mutate() function to calculate, for each country, the % difference between Males and Females for each subject.
df = df %>% mutate(Maths.Dif = ((Maths.M  Maths.F)/Maths.F)*100, Reading.Dif = ((Reading.M  Reading.F)/Reading.F)*100, Science.Dif = ((Science.M  Science.F)/Science.F)*100, Total.Score = Maths + Reading + Science, Avg.Dif = (Maths.Dif+Reading.Dif+Science.Dif)/3 )view(df)
Now let’s plot this out to visualize the data better
##### MATHS SCORE ##### ggplot(data=df, aes(x=reorder(Country.Name, Maths.Dif), y=Maths.Dif)) + geom_bar(stat = "identity", aes(fill=Maths.Dif)) + coord_flip() + theme_light() + geom_hline(yintercept = mean(df$Maths.Dif), size=1, color="black") + scale_fill_gradient(name="% Difference Level") + labs(title="Are Males better at math?", x="", y="% difference from female")
This plot represents the % difference in scores using Females as reference . A positive difference means males scored higher, while a negative difference means males scored lower . The black line represents the mean difference across all countries.
Some interesting insights one can draw from here:
 In general, Males performed better than Females in Maths, especially in most of the DCs, males scored generally higher than females
 Interestingly, in Singapore and Hongkong, females and males performed equally well, with the difference in score is around 0 in each of these countries. This is good insight maybe for policymakers because we do not want a huge gap between males and females performance in education.
We can do the same thing for Reading and Science score.
3. Correlation Plot
df = df[,c(1,3,4,6,7,9,10)] #select relevant columns
To create correlation plot, simply use cor():
res = cor(df[,1]) # 1 here means we look at all columns except the first columnres
We can calculate pvalue to see whether the correlation is significant
install.packages("Hmisc") library("Hmisc")res2 < rcorr(as.matrix(df[,1))
The smaller the pvalue, the more significant the correlation.
The purpose of this section about correlation plot is to introduce to you how to calculate correlation between variables in R. For this dataset, it is obvious that all the variables are correlated
To visualize
install.packages("corrplot") library(corrplot) corrplot(res, type = "upper", order = "hclust", tl.col = "black", tl.srt = 45)
The stronger the color and the bigger the size, the higher the correlation. The result is similar to the one we got earlier: All the variables are intercorrelated.
That is it! Hope you guys enjoyed and picked up something from this article. While this guide is not exhaustive, it will more or less give you some ideas of how to do some basic EDA with R.
If you have any questions, feel free to put them down in the comment section below. Thank you for your read. Have a great day and happy programming!!!
猜你喜欢

7
A more advanced method of doing EDA with ‘ggplot2’ and ‘tidyverse’

15
Visuals, graphs and more visuals! Continui...

4
Fluid Annotation: An Exploratory Machine Learning–Powered Interface for Faster Image Annotation

15
In recent weeks, news of the devastating wildfires sweeping parts of the US state of California have featured prominently in the

18
There are of Microsoft Excel is going to be over, so would you like to learn the next generation one of the most powerful data processing tool and in deman d sk...

17
Teaching Each Other New Coding Skills Every...

21
Exciting news! We’re officially launching the beta version of our new Data Analyst in R path . If you’re looking to learn R for data analysis or data science, w...

125
TensorFlowExamples  Concise TensorFlow Examples for Beginners with Latest APIs

82
ES6 for beginners. The post covers let, const, arrow functions, for..of, map, sets and spread attributes.

65
ResourcesFrontEndBeginner  The most essential list of resources for FrontEnd beginners (?? ?? & ??)
 上一页
 下一页
关于极客头条
聚合每日国内外有价值，有趣的链接。