Mathematics and Statistics for Data Science
- Descriptive Data Analysis:
- Rather than find hidden information in the data, descriptive data analysis looks to summarize the dataset.
- They are commonly implemented measures included in the descriptive data analysis:
- Central tendency (Mean, Mode, Median)
- Variability (Standard deviation, Min/Max)
Central tendency
https://statistics.laerd.com/statistical-guides/measures-central-tendency-mean-mode-median.php
A central tendency (or measure of central tendency) is a single value that attempts to describe a set of data by identifying the central position within that set of data.
The mean (often called the average) is most likely the measure of central tendency that you are most familiar with, but there are others, such as the median and the mode.
The mean, median and mode are all valid measures of central tendency, but under different conditions, some measures of central tendency become more appropriate to use than others. In the following sections, we will look at the mean, mode and median, and learn how to calculate them and under what conditions they are most appropriate to be used.
Mean
Mean (Arithmetic)
The mean (or average) is the most popular and well known measure of central tendency.
The mean is equal to the sum of all the values in the data set divided by the number of values in the data set.
So, if we have values in a data set and they have values the sample mean, usually denoted by (pronounced x bar), is:
The mean is essentially a model of your data set. It is the value that is most common. You will notice, however, that the mean is not often one of the actual values that you have observed in your data set. However, one of its important properties is that it minimises error in the prediction of any one value in your data set. That is, it is the value that produces the lowest amount of error from all other values in the data set.
An important property of the mean is that it includes every value in your data set as part of the calculation. In addition, the mean is the only measure of central tendency where the sum of the deviations of each value from the mean is always zero.
When not to use the mean
The mean has one main disadvantage: it is particularly susceptible to the influence of outliers. These are values that are unusual compared to the rest of the data set by being especially small or large in numerical value. For example, consider the wages of staff at a factory below:
Staff | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 |
---|---|---|---|---|---|---|---|---|---|---|
Salary |
The mean salary for these ten staff is $30.7k. However, inspecting the raw data suggests that this mean value might not be the best way to accurately reflect the typical salary of a worker, as most workers have salaries in the $12k to 18k range. The mean is being skewed by the two large salaries. Therefore, in this situation, we would like to have a better measure of central tendency. As we will find out later, taking the median would be a better measure of central tendency in this situation.
Another time when we usually prefer the median over the mean (or mode) is when our data is skewed (i.e., the frequency distribution for our data is skewed). If we consider the normal distribution - as this is the most frequently assessed in statistics - when the data is perfectly normal, the mean, median and mode are identical. Moreover, they all represent the most typical value in the data set. However, as the data becomes skewed the mean loses its ability to provide the best central location for the data because the skewed data is dragging it away from the typical value. However, the median best retains this position and is not as strongly influenced by the skewed values. This is explained in more detail in the skewed distribution section later in this guide.
Mean in R
mean(iris$Sepal.Width)
Median
The median is the middle score for a set of data that has been arranged in order of magnitude. The median is less affected by outliers and skewed data. In order to calculate the median, suppose we have the data below:
65 | 55 | 89 | 56 | 35 | 14 | 56 | 55 | 87 | 45 | 92 |
---|
We first need to rearrange that data into order of magnitude (smallest first):
14 | 35 | 45 | 55 | 55 | 56 | 56 | 65 | 87 | 89 | 92 |
---|
Our median mark is the middle mark - in this case, 56. It is the middle mark because there are 5 scores before it and 5 scores after it. This works fine when you have an odd number of scores, but what happens when you have an even number of scores? What if you had only 10 scores? Well, you simply have to take the middle two scores and average the result. So, if we look at the example below:
65 | 55 | 89 | 56 | 35 | 14 | 56 | 55 | 87 | 45 |
---|
We again rearrange that data into order of magnitude (smallest first):
14 | 35 | 45 | 55 | 55 | 56 | 56 | 65 | 87 | 89 |
---|
Only now we have to take the 5th and 6th score in our data set and average them to get a median of 55.5.
Median in R
median(iris$Sepal.Length)
Mode
The mode is the most frequent score in our data set. On a histogram it represents the highest bar in a bar chart or histogram. You can, therefore, sometimes consider the mode as being the most popular option. An example of a mode is presented below:
Normally, the mode is used for categorical data where we wish to know which is the most common category, as illustrated below:
We can see above that the most common form of transport, in this particular data set, is the bus. However, one of the problems with the mode is that it is not unique, so it leaves us with problems when we have two or more values that share the highest frequency, such as below:
We are now stuck as to which mode best describes the central tendency of the data. This is particularly problematic when we have continuous data because we are more likely not to have any one value that is more frequent than the other. For example, consider measuring 30 peoples' weight (to the nearest 0.1 kg). How likely is it that we will find two or more people with exactly the same weight (e.g., 67.4 kg)? The answer, is probably very unlikely - many people might be close, but with such a small sample (30 people) and a large range of possible weights, you are unlikely to find two people with exactly the same weight; that is, to the nearest 0.1 kg. This is why the mode is very rarely used with continuous data.
Another problem with the mode is that it will not provide us with a very good measure of central tendency when the most common mark is far away from the rest of the data in the data set, as depicted in the diagram below:
In the above diagram the mode has a value of 2. We can clearly see, however, that the mode is not representative of the data, which is mostly concentrated around the 20 to 30 value range. To use the mode to describe the central tendency of this data set would be misleading.
To get the Mode in R
install.packages("modeest") library(modeest)
> mfv(iris$Sepal.Width, method = "mfv")
Skewed Distributions and the Mean and Median
We often test whether our data is normally distributed because this is a common assumption underlying many statistical tests. An example of a normally distributed set of data is presented below:
When you have a normally distributed sample you can legitimately use both the mean or the median as your measure of central tendency. In fact, in any symmetrical distribution the mean, median and mode are equal. However, in this situation, the mean is widely preferred as the best measure of central tendency because it is the measure that includes all the values in the data set for its calculation, and any change in any of the scores will affect the value of the mean. This is not the case with the median or mode.
However, when our data is skewed, for example, as with the right-skewed data set below:
we find that the mean is being dragged in the direct of the skew. In these situations, the median is generally considered to be the best representative of the central location of the data. The more skewed the distribution, the greater the difference between the median and mean, and the greater emphasis should be placed on using the median as opposed to the mean. A classic example of the above right-skewed distribution is income (salary), where higher-earners provide a false representation of the typical income if expressed as a mean and not a median.
If dealing with a normal distribution, and tests of normality show that the data is non-normal, it is customary to use the median instead of the mean. However, this is more a rule of thumb than a strict guideline. Sometimes, researchers wish to report the mean of a skewed distribution if the median and mean are not appreciably different (a subjective assessment), and if it allows easier comparisons to previous research to be made.
Summary of when to use the mean, median and mode
Please use the following summary table to know what the best measure of central tendency is with respect to the different types of variable:
Type of Variable | Best measure of central tendency |
---|---|
Nominal | Mode |
Ordinal | Median |
Interval/Ratio (not skewed) | Mean |
Interval/Ratio (skewed) | Median |
For answers to frequently asked questions about measures of central tendency, please go to: https://statistics.laerd.com/statistical-guides/measures-central-tendency-mean-mode-median-faqs.php
Measures of Variation
Range
The Range just simply shows the min and max value of a variable.
In R:
> min(iris$Sepal.Width) > max(iris$Sepal.Width) > range(iris$Sepal.Width)
Range can be used on Ordinal, Ratio and Interval scales
Quartile
https://statistics.laerd.com/statistical-guides/measures-of-spread-range-quartiles.php
Quartiles tell us about the spread of a data set by breaking the data set into quarters, just like the median breaks it in half.
For example, consider the marks of the 100 students, which have been ordered from the lowest to the highest scores.
- The first quartile (Q1): Lies between the 25th and 26th student's marks.
- So, if the 25th and 26th student's marks are 45 and 45, respectively:
- (Q1) = (45 + 45) ÷ 2 = 45
- So, if the 25th and 26th student's marks are 45 and 45, respectively:
- The second quartile (Q2): Lies between the 50th and 51st student's marks.
- If the 50th and 51th student's marks are 58 and 59, respectively:
- (Q2) = (58 + 59) ÷ 2 = 58.5
- If the 50th and 51th student's marks are 58 and 59, respectively:
- The third quartile (Q3): Lies between the 75th and 76th student's marks.
- If the 75th and 76th student's marks are 71 and 71, respectively:
- (Q3) = (71 + 71) ÷ 2 = 71
- If the 75th and 76th student's marks are 71 and 71, respectively:
In the above example, we have an even number of scores (100 students, rather than an odd number, such as 99 students). This means that when we calculate the quartiles, we take the sum of the two scores around each quartile and then half them (hence Q1= (45 + 45) ÷ 2 = 45) . However, if we had an odd number of scores (say, 99 students), we would only need to take one score for each quartile (that is, the 25th, 50th and 75th scores). You should recognize that the second quartile is also the median.
Quartiles are a useful measure of spread because they are much less affected by outliers or a skewed data set than the equivalent measures of mean and standard deviation. For this reason, quartiles are often reported along with the median as the best choice of measure of spread and central tendency, respectively, when dealing with skewed and/or data with outliers. A common way of expressing quartiles is as an interquartile range. The interquartile range describes the difference between the third quartile (Q3) and the first quartile (Q1), telling us about the range of the middle half of the scores in the distribution. Hence, for our 100 students:
However, it should be noted that in journals and other publications you will usually see the interquartile range reported as 45 to 71, rather than the calculated
A slight variation on this is the which is half the Hence, for our 100 students:
Quartile in R
quantile(iris$Sepal.Length)
Result 0% 25% 50% 75% 100% 4.3 5.1 5.8 6.4 7.9
0% and 100% are equivalent to min max values.
Box Plots
boxplot(iris$Sepal.Length, col = "blue", main="iris dataset", ylab = "Sepal Length")
Variance
https://statistics.laerd.com/statistical-guides/measures-of-spread-absolute-deviation-variance.php
Another method for calculating the deviation of a group of scores from the mean, such as the 100 students we used earlier, is to use the variance. Unlike the absolute deviation, which uses the absolute value of the deviation in order to "rid itself" of the negative values, the variance achieves positive values by squaring each of the deviations instead. Adding up these squared deviations gives us the sum of squares, which we can then divide by the total number of scores in our group of data (in other words, 100 because there are 100 students) to find the variance (see below). Therefore, for our 100 students, the variance is 211.89, as shown below:
- Variance describes the spread of the data.
- It is a measure of deviation of a variable from the arithmetic mean.
- The technical definition is the average of the squared differences from the mean.
- A value of zero means that there is no variability; All the numbers in the data set are the same.
- A higher number would indicate a large variety of numbers.
Variance in R
var(iris$Sepal.Length)
Standard Deviation
https://statistics.laerd.com/statistical-guides/measures-of-spread-standard-deviation.php
The standard deviation is a measure of the spread of scores within a set of data. Usually, we are interested in the standard deviation of a population. However, as we are often presented with data from a sample only, we can estimate the population standard deviation from a sample standard deviation. These two standard deviations - sample and population standard deviations - are calculated differently. In statistics, we are usually presented with having to calculate sample standard deviations, and so this is what this article will focus on, although the formula for a population standard deviation will also be shown.
The sample standard deviation formula is:
The population standard deviation formula is:
- The Standard Deviation is the square root of the variance.
- This measure is the most widely used to express deviation from the mean in a variable.
- The higher the value the more widely distributed are the variable data values around the mean.
- Assuming the frequency distributions approximately normal, about 68% of all observations are within +/- 1 standard deviation.
- Approximately 95% of all observations fall within two standard deviations of the mean (if data is normally distributed).
Standard Deviation in R
sd(iris$Sepal.Length)
Z Score
- z-score represents how far from the mean a particular value is based on the number of standard deviations.
- z-scores are also known as standardized residuals
- Note: mean and standard deviation are sensitive to outliers
> x <-((iris$Sepal.Width) - mean(iris$Sepal.Width))/sd(iris$Sepal.Width) > x > x[77] #choose a single row # or this > x <-((iris$Sepal.Width[77]) - mean(iris$Sepal.Width))/sd(iris$Sepal.Width) > x
Shape of Distribution
Skewness
- Skewness is a method for quantifying the lack of symmetry in the distribution of a variable.
- Skewness value of zero indicates that the variable is distributed symmetrically. Positive number indicate asymmetry to the left, negative number indicates asymmetry to the right.
Skewness in R
> install.packages("moments") and library(moments) > skewness(iris$Sepal.Width)
Histograms in R
> hist(iris$Petal.Width)
Kurtosis
- Kurtosis is a measure that gives indication in terms of the peak of the distribution.
- Variables with a pronounced peak toward the mean have a high Kurtosis score and variables with a flat peak have a low Kurtosis score.
Kurtosis in R
> kurtosis(iris$Sepal.Length)