Lecture Two: Content Analysis and Natural Language Processing I

Lecture Summary

1. Content analysis
1. Analytical technique that can be used to extract secondary meaning from information
2. Vocabulary Analysis - examine documents purely as a collection of words and look at patterns
3. Readability Indexes - the ease with which a native speaker of average intelligence can read and understand a passage of text
4. Normative Comparison - compare documents to some standard of normal text
2. Natural Language Processing
1. Make the computer understand natural language input and generate natural language output
2. How is NLP done?
• Pre-Processing - remove useless parts of the documents
• Tokenization - break the raw text into meaningful units
• Normalization - e.g. cases
• Stemming - reduce words of different forms to the "root"
• Parts of Speech (POS) Tags
• Language Model - probabilitic models of sequences of words; enables word predictions
• unigram, bigram, trigram...
• p(wi) = c(wi)/N
• p(wi|wi-1) = c(wi-1wi)/C(wi-1)
• p(wi|wi-2wi-1) = c(wi-2wi-2wi)/C(wi-2wi-1)
3. Google Book’s N-Gram Viewer

Notes

The main dish of this lecture is the steps of doing NLP. As a developer, most of the works can be done by simply calling libraries. However, we do need to understand the theories behind them. Among the introduced steps, I paid more attention on the N-Gram models. It is interesting to have computer predicting something that it doesn't really understand. I hope I can have chance to use those models in real life cases.
N-Gram Viewer is an interesting source of research. I will keep an eye on it.