Supervised Machine Learning for Fake News Detection

Acknowledgement

Thanks for Muhammad, Graham and Mark

Chapter 2 - Training a Supervised Machine Learning Model for fake news detection

Supervised text Classification for fake news detection Using Machine Learning Models

Procedure

• The Dataset

• Splitting the data into Train and Test data

• Cleaning the data

• Building the Document-Term Matrix

• Model Building

• Cross validation

• Making predictions from the model created and displaying a Confusion matrix

Results

Summary of Results

Algorithms	Author	Package	Keyword	Accuracy						Running time for the entire data
				Kaggle fake news dataset		Fake news challenge dataset		Fake news Detector dataset		Using a GoogleCloud VM: 64 vCPUs, 240 GB memory (Ubuntu 18.04)
				1000 rows	The entire data: 20,800 rows	1000 rows	The entire data: 49,972 rows	1000 rows	The entire data:	Kaggle fake news dataset	Fake news challenge dataset	Fake news Detector dataset
Naive Bayes	Bayes, Thomas	We used RTextTools, which depends on e1071	NB*	Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://en.wikipedia.org/api/rest_v1/":): {\displaystyle {\color{blue}69,03%}}	Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://en.wikipedia.org/api/rest_v1/":): {\displaystyle {\color{blue}76.41%}}	Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://en.wikipedia.org/api/rest_v1/":): {\displaystyle {\color{blue}73.91%}}	Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://en.wikipedia.org/api/rest_v1/":): {\displaystyle {\color{blue}93.95%}}			Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://en.wikipedia.org/api/rest_v1/":): {\displaystyle 43 secs}	Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://en.wikipedia.org/api/rest_v1/":): {\displaystyle 36 secs}
Support vector machine	Meyer et al., 2012	We used RTextTools, which depends on e1071	SVM*	Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://en.wikipedia.org/api/rest_v1/":): {\displaystyle {\color{blue}81%}}	Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://en.wikipedia.org/api/rest_v1/":): {\displaystyle {\color{blue}95.42%}}					Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://en.wikipedia.org/api/rest_v1/":): {\displaystyle 59 mins}
Random forest	Liawand Wiener, 2002	We used RTextTools, which depends on randomForest	RF	Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://en.wikipedia.org/api/rest_v1/":): {\displaystyle {\color{blue}84%}}						Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://en.wikipedia.org/api/rest_v1/":): {\displaystyle 0 secs}
General linearized models	Friedman et al., 2010	We used RTextTools, which depends on wglmnet	GLMNET*	${\color {blue}91\%}$	${\color {blue}94.58\%}$					$15secs$
Maximum entropy	Jurka, 2012	We used RTextTools, which depends on maxent	MAXENT*	${\color {blue}87.33\%}$	${\color {blue}96.09\%}$					$54secs$
Extreme Gradient Boosting	Chen & Guestrin, 2016	xgboost	XGBOOST*	${\color {red}93\%}$	Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://en.wikipedia.org/api/rest_v1/":): {\displaystyle {\color{red}97.42%}}		Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://en.wikipedia.org/api/rest_v1/":): {\displaystyle {\color{red}95.36%}}			Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://en.wikipedia.org/api/rest_v1/":): {\displaystyle 9 mins}	Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://en.wikipedia.org/api/rest_v1/":): {\displaystyle 21 mins}
Classification or regression tree	Ripley., 2012	We used RTextTools, which depends on tree	TREE	Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://en.wikipedia.org/api/rest_v1/":): {\displaystyle {\color{blue}91.33%}}						Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://en.wikipedia.org/api/rest_v1/":): {\displaystyle 0 secs}
Boosting	Tuszynski, 2012	We used RTextTools, which depends on caTools	BOOSTING	Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://en.wikipedia.org/api/rest_v1/":): {\displaystyle {\color{blue}92.67%}}						Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://en.wikipedia.org/api/rest_v1/":): {\displaystyle 0 secs}
Neural networks	Venables and Ripley, 2002	We used RTextTools, which depends on nnet	NNET	Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://en.wikipedia.org/api/rest_v1/":): {\displaystyle {\color{blue}00.00%}}						Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://en.wikipedia.org/api/rest_v1/":): {\displaystyle 0 secs}
Bagging	Peters and Hothorn, 2012	We used RTextTools, which depends on ipred	BAGGING**
Scaled linear discriminant analysis	Peters and Hothorn, 2012	We used RTextTools, which depends on ipred	SLDA**
* Low-memory algorithm ** Very high-memory algorithm

Evaluation of Results

We evaluate our approach in different settings. First, weperform cross-validation on our noisy training set; second,and more importantly, we train models on the training setand validate them against a manually created gold standard.17Moreover, we evaluate two variants, i.e., including and exclud-ing user features. [smb:home/adelo/1-system/1-disco_local/1-mis_archivos/1-pe/1-ciencia/1-computacion/2-data_analysis-machine_learning/gofaaaz-machine_learning/5-References/7-Weakly_supervised_searning_for_fake_news_detection_on_twitter.pdf]

The Gofaaas-Fake News Detector R Package

Installation

Functions

Datasets used

Kaggle Fake News Dataset

https://www.kaggle.com/c/fake-news/data

Distribution of the data:

The distribution of Stance classes in train_stances.csv is as follows:

rows	unrelated	discuss	agree	disagree
49972	0.73131	0.17828	0.0736012	0.0168094

Fake News Detector Dataset

Gofaaas Fake News Dataset

Algorithms

Naive Bayes

Naïve Bayes is based on the Bayesian theorem, there in order to understand Naïve Bayes it is important to first understand the Bayesian theorem.

Bayesian theorem is a mathematical formula for determining conditional probability which is the probability of something, happening given that something else has already occurred.

• P(c|x) is the posterior probability of class (target) given predictor (attribute).
• P(c) is the prior probability of class.
• P(x|c) is the likelihood which is the probability of predictor given class.
• P(x) is the prior probability of predictor.

Prior probability, in Bayesian statistical inference, is the probability of an event before new data is collected.

Posterior probability is the revised probability of an event occurring after taking into consideration new information.

In statistical terms, the posterior probability is the probability of event A occurring given that event B has occurred.

Support vector machine

Random forest

Extreme Gradient Boosting

The XGBoost R package

XGBoost - Extreme Gradient Boosting:

• https://xgboost.readthedocs.io/en/latest/
• https://cran.r-project.org/web/packages/xgboost/index.html

The RTextTools package

RTextTools - A Supervised Learning Package for Text Classification:

• https://journal.r-project.org/archive/2013/RJ-2013-001/RJ-2013-001.pdf
• http://www.rtexttools.com/
• https://cran.r-project.org/web/packages/RTextTools/index.html

Chapter 3 - Gofaas Web App

A way to interact, test and display the model results