Lecture Three: Content Analysis and Natural Language Processing II

Lecture Summary

1. Document comparisons
1. Represent a document by "bag-of-words"
2. ignore grammar, dependencies, order of words etc.
3. Similarity measurements by Jaccard’s Index
1. Cannot reflect importance of words
2. Term weighting by TF-IDF
4. Similarity measurements by Term Frequency & Inverse Document Frequency
1. tf-idf(t,d) = tf(t,d) x idf(t)
2. tf(t,d) = number of times t appears in d
3. df(t) = number of documents that contains t
4. idf(t) = log(N/df(t)) where N = number of documents
5. Similarity of measurements by Vector Space Model
2. Text Classification
1. Two steps of classification:
1. Train a classifier by using labeled data (‘supervised’ learning)
2. Classify new data using the classifier
2. Naïve Bayes Text Classification
1. Idea: Probability of a document belonging to a class depending on the probability of the words in the document belonging to the class
2. 
   

3. Text Clustering


1. Unsupervised learning
1. Hard:each item can only belong to one cluster
2. Soft: each item can belong to different clusters at to different extents
2. K-mean Clustering
1. Goal: to find K points (centroids) in the item space that represent the K clusters of items
2. Represent documents using the vector space model
3. Standard algorithm for K-Means clustering
1. Randomly come up with K centroids.
2. Determine the cluster membership of each data point.
3. Update the k-th centroids with the new membership assignments (Z is the number of points in that cluster)
4. 
3. Model-based Text Clustering

1. Generalize K-mean clustering to a soft text clustering model

Notes

This lecture discussed some methods of comparing and classifying documents. Some mathematics tools including probability and vectors were used in these methods. After undergraduate, I haven't used these tools for years. I really haven't to practice them again.

The idea of document comparison is very interesting to me. I am thinking whether this technique can be used in identifying and classifying phishing websites. In my job, I need to handle some phishing website cases. We can only identify them manually. If we can represent a website using bag-of-words, we can compare the suspected phishing websites and the legitimate websites by algorithms.