Information Cascade and Election

What is Information Cascade?

Information cascade is referring to the effect that people's decisions being affected by the choices of a large group of people, particular when people make decisions sequentially.

For example, when people decide which products to buy, the best selling product often attracts more customer. The agent followed the decisions of other customers.

Information cascade occurs when:

1. Agents make decisions sequentially
2. Agents make decisions rationally based on the information they have
3. Agents do not have access to the private information of others
4. A limited action space exists (e.g. an adopt/reject decision)

Information Cascade in Social Media

From the lecture, an experiment in Facebook was introduced to test the effect of information cascade in social media. The result showed the effect did happen.

Actually not only did it happen in experiments, it also affected the voting rate of 2012 US president election.

In US president election 2002, Facebook launch a new feature called "I'm Voting". When a user pressed the button, his/her friends will see the "I'm voting" button appeared on the user's top newsfeed.

Election fulfilled the four condition required for information cascade.

1. Voters vote sequentially
2. Voters make their decisions (vote or not? who to vote?) rationally
3. Voters do not have the full picture of the decisions made by others
4. The action space is limited

Thus, in theory, information cascade can happen. In fact, study found out it boosted the turnout rate by 2.2%, proving information cascade did happen.

So what?

The experiment and the real life case showed the power of information cascade in social media. Its influential power far exceeds traditional media. I can predict that the different parties will shift more and more attentions from traditional media to social media. A lot of opportunities are awaiting to be dug out. 

Overfitting in Supervised Learning

Supervised Learning

In Lecture 4, the idea of supervised classification was introduced.

Supervised classification is a way to solve the classification problem by first training the classifier with data of known classification. Then, the classifier can do classification based on the known data set. For example, in assignment we trained the classifier with two pieces of data, namely "health.txt" and "business.txt". Then it can classify the new input words with them.

(Figure from http://www.nltk.org/book/ch06.html)

Overfitting

During the training, we often focus on the accuracy of the classifier. But being too accurate with the training data may introduce another problem - overfitting.

Overfitting means "a statistical model describes random error or noise instead of the underlying relationship" (source). A set of training data likely contains some noise and errors that are not related to the classification. For example, in our assignment 2, both health.txt and business.txt contains words that are neither health related nor business related, such as "slow", "system". If we train a classifier by attempting to perfectly fit the training data, those unrelated words may be classified wrongly. Overfitting occurs.

Bootstrap aggregating

In order to solve overfitting, a method called boostrap aggregating, or bagging.

Using bagging, a number of new training sets can be generated from the original training set. It is done by sampling the original training set with replacement - i.e. each word can be selected more than one time. Usually hundreds or thousands of boostrap samples are used.

After getting the samples, predictions will be by voting among all the samples. The result which gets the majority vote will be the prediction. By considering a large number of boostrap sample, bagging reduced the effort of overfitting.

An improvement over bootstrap aggregating is Random Forest. It can improve the prediction result if one or a few features are a strong predictor.

Lecture Three: Content Analysis and Natural Language Processing II

Lecture Summary

1. Document comparisons
1. Represent a document by "bag-of-words"
2. ignore grammar, dependencies, order of words etc.
3. Similarity measurements by Jaccard’s Index
1. Cannot reflect importance of words
2. Term weighting by TF-IDF
4. Similarity measurements by Term Frequency & Inverse Document Frequency
1. tf-idf(t,d) = tf(t,d) x idf(t)
2. tf(t,d) = number of times t appears in d
3. df(t) = number of documents that contains t
4. idf(t) = log(N/df(t)) where N = number of documents
5. Similarity of measurements by Vector Space Model
2. Text Classification
1. Two steps of classification:
1. Train a classifier by using labeled data (‘supervised’ learning)
2. Classify new data using the classifier
2. Naïve Bayes Text Classification
1. Idea: Probability of a document belonging to a class depending on the probability of the words in the document belonging to the class
2. 
   

3. Text Clustering


1. Unsupervised learning
1. Hard:each item can only belong to one cluster
2. Soft: each item can belong to different clusters at to different extents
2. K-mean Clustering
1. Goal: to find K points (centroids) in the item space that represent the K clusters of items
2. Represent documents using the vector space model
3. Standard algorithm for K-Means clustering
1. Randomly come up with K centroids.
2. Determine the cluster membership of each data point.
3. Update the k-th centroids with the new membership assignments (Z is the number of points in that cluster)
4. 
3. Model-based Text Clustering

1. Generalize K-mean clustering to a soft text clustering model

Notes

This lecture discussed some methods of comparing and classifying documents. Some mathematics tools including probability and vectors were used in these methods. After undergraduate, I haven't used these tools for years. I really haven't to practice them again.

The idea of document comparison is very interesting to me. I am thinking whether this technique can be used in identifying and classifying phishing websites. In my job, I need to handle some phishing website cases. We can only identify them manually. If we can represent a website using bag-of-words, we can compare the suspected phishing websites and the legitimate websites by algorithms.

Lecture Two: Content Analysis and Natural Language Processing I

Lecture Summary

1. Content analysis
1. Analytical technique that can be used to extract secondary meaning from information
2. Vocabulary Analysis - examine documents purely as a collection of words and look at patterns
3. Readability Indexes - the ease with which a native speaker of average intelligence can read and understand a passage of text
4. Normative Comparison - compare documents to some standard of normal text
2. Natural Language Processing
1. Make the computer understand natural language input and generate natural language output
2. How is NLP done?
• Pre-Processing - remove useless parts of the documents
• Tokenization - break the raw text into meaningful units
• Normalization - e.g. cases
• Stemming - reduce words of different forms to the "root"
• Parts of Speech (POS) Tags
• Language Model - probabilitic models of sequences of words; enables word predictions
• unigram, bigram, trigram...
• p(wi) = c(wi)/N
• p(wi|wi-1) = c(wi-1wi)/C(wi-1)
• p(wi|wi-2wi-1) = c(wi-2wi-2wi)/C(wi-2wi-1)
3. Google Book’s N-Gram Viewer

Notes

The main dish of this lecture is the steps of doing NLP. As a developer, most of the works can be done by simply calling libraries. However, we do need to understand the theories behind them. Among the introduced steps, I paid more attention on the N-Gram models. It is interesting to have computer predicting something that it doesn't really understand. I hope I can have chance to use those models in real life cases.
N-Gram Viewer is an interesting source of research. I will keep an eye on it.

Lecture One: Fundamentals of Social Media and Social Media Analytics

In the first lecture, the fundamentals of Social Media and Social Media Analytic were introduced.

Lecture Summary

 This lecture included the following areas:

1. What are social networking and social media?
2. The key technology that empowers social media - Web 2.0.
3. The study of social media - science and engineering
4. Understanding the Users
1. Behaviorist Approach
2. Social Cognition Approach
5. Dokdo/Takeshima - An example of wikipedia
6. Social media analytic - concerned with developing and evaluating informatics tools and frameworks to extract, analyze, summarize, and visualize information
1. Not only data mining and statistical analysis
2. An interdisciplinary area involving BOTH humanities and technologies
7. Social media analysis - focus on the “analysis” component within social media analytics
1. Aims for conducting social media analysis
8. Work flow of social media analytic
1. Social media
2. Data collection
3. Analysis and development
4. Interpretation and presentation of results
5. Applications / Decision-making / Prediction / Research Findings
9. Some case studies

Notes

This lecture was essentially a high level overview of whole course.

Of the whole lecture, to me, the most difficult idea is social cognition. It is a new term that I have never met before so I spent some time to understand its meaning.

Social Cognitive Theory (SCT) - People continue to influence and be influenced by the social environment surrounding them

 I wonder if I misunderstood this theory. It looks like common sense. I cannot think of how to apply this theory in the analysis. 

My First Blog Post

Hello everyone! Welcome to my first blog!

Blog is an excellent place for consolidating newly learnt knowledge. Not only can the writer have a chance to revise the new knowledge, the others can also exchange ideas with the writers easily. It is beneficial to both the writer and the readers.

I would like to use this place to summarize what I have learnt in the Social Media Analytic course.