Overfitting in Supervised Learning

Supervised Learning

In Lecture 4, the idea of supervised classification was introduced.

Supervised classification is a way to solve the classification problem by first training the classifier with data of known classification. Then, the classifier can do classification based on the known data set. For example, in assignment we trained the classifier with two pieces of data, namely "health.txt" and "business.txt". Then it can classify the new input words with them.

(Figure from http://www.nltk.org/book/ch06.html)

Overfitting

During the training, we often focus on the accuracy of the classifier. But being too accurate with the training data may introduce another problem - overfitting.

Overfitting means "a statistical model describes random error or noise instead of the underlying relationship" (source). A set of training data likely contains some noise and errors that are not related to the classification. For example, in our assignment 2, both health.txt and business.txt contains words that are neither health related nor business related, such as "slow", "system". If we train a classifier by attempting to perfectly fit the training data, those unrelated words may be classified wrongly. Overfitting occurs.

Bootstrap aggregating

In order to solve overfitting, a method called boostrap aggregating, or bagging.

Using bagging, a number of new training sets can be generated from the original training set. It is done by sampling the original training set with replacement - i.e. each word can be selected more than one time. Usually hundreds or thousands of boostrap samples are used.

After getting the samples, predictions will be by voting among all the samples. The result which gets the majority vote will be the prediction. By considering a large number of boostrap sample, bagging reduced the effort of overfitting.

An improvement over bootstrap aggregating is Random Forest. It can improve the prediction result if one or a few features are a strong predictor.