Assignment 5¶
In this assignment, you'll scrape text from The California Aggie and then analyze the text.
The Aggie is organized by category into article lists. For example, there's a Campus News list, Arts & Culture list, and Sports list. Notice that each list has multiple pages, with a maximum of 15 articles per page.
The goal of exercises 1.1 - 1.3 is to scrape articles from the Aggie for analysis in exercise 1.4.
Exercise 1.1. Write a function that extracts all of the links to articles in an Aggie article list. The function should:
Have a parameter
urlfor the URL of the article list.Have a parameter
pagefor the number of pages to fetch links from. The default should be1.Return a list of aricle URLs (each URL should be a string).
Test your function on 2-3 different categories to make sure it works.
Hints:
Be polite to The Aggie and save time by setting up requests_cache before you write your function.
Start by getting your function to work for just 1 page. Once that works, have your function call itself to get additional pages.
You can use lxml.html or BeautifulSoup to scrape HTML. Choose one and use it throughout the entire assignment.
import requests
import requests_cache
requests_cache.install_cache('demo_cache')
from bs4 import BeautifulSoup
import pandas as pd
def extract_link(url, num_page):
"""
Extracts all of the links to articles in an Aggie article list.
Argument: url, num_page
Return url_page link
"""
url_page = url + 'page/' + str(num_page) + '/'
news = requests.get(url_page)
news_soup = BeautifulSoup(news.content, "lxml")
links = news_soup.findAll("a", {"class": "more-link"})
url_list = [link.get('href') for link in links]
return url_list
url = "https://theaggie.org/campus/"
num_page = 6
extract_link(url, num_page)
Exercise 1.2. Write a function that extracts the title, text, and author of an Aggie article. The function should:
Have a parameter
urlfor the URL of the article.For the author, extract the "Written By" line that appears at the end of most articles. You don't have to extract the author's name from this line.
Return a dictionary with keys "url", "title", "text", and "author". The values for these should be the article url, title, text, and author, respectively.
For example, for this article your function should return something similar to this:
{
'author': u'Written By: Bianca Antunez \xa0\u2014\xa0city@theaggie.org',
'text': u'Davis residents create financial model to make city\'s financial state more transparent To increase transparency between the city\'s financial situation and the community, three residents created a model called Project Toto which aims to improve how the city communicates its finances in an easily accessible design. Jeff Miller and Matt Williams, who are members of Davis\' Finance and Budget Commission, joined together with Davis entrepreneur Bob Fung to create the model plan to bring the project to the Finance and Budget Commission in February, according to Kelly Stachowicz, assistant city manager. "City staff appreciate the efforts that have gone into this, and the interest in trying to look at the city\'s potential financial position over the long term," Stachowicz said in an email interview. "We all have a shared goal to plan for a sound fiscal future with few surprises. We believe the Project Toto effort will mesh well with our other efforts as we build the budget for the next fiscal year and beyond." Project Toto complements the city\'s effort to amplify the transparency of city decisions to community members. The aim is to increase the understanding about the city\'s financial situation and make the information more accessible and easier to understand. The project is mostly a tool for public education, but can also make predictions about potential decisions regarding the city\'s financial future. Once completed, the program will allow residents to manipulate variables to see their eventual consequences, such as tax increases or extensions and proposed developments "This really isn\'t a budget, it is a forecast to see the intervention of these decisions," Williams said in an interview with The Davis Enterprise. "What happens if we extend the sales tax? What does it do given the other numbers that are in?" Project Toto enables users, whether it be a curious Davis resident, a concerned community member or a city leader, with the ability to project city finances with differing variables. The online program consists of the 400-page city budget for the 2016-2017 fiscal year, the previous budget, staff reports and consultant analyses. All of the documents are cited and accessible to the public within Project Toto. "It\'s a model that very easily lends itself to visual representation," Mayor Robb Davis said. "You can see the impacts of decisions the council makes on the fiscal health of the city." Complementary to this program, there is also a more advanced version of the model with more in-depth analyses of the city\'s finances. However, for an easy-to-understand, simplistic overview, Project Toto should be enough to help residents comprehend Davis finances. There is still more to do on the project, but its creators are hard at work trying to finalize it before the 2017-2018 fiscal year budget. "It\'s something I have been very much supportive of," Davis said. "Transparency is not just something that I have been supportive of but something we have stated as a city council objective [ ] this fits very well with our attempt to inform the public of our challenges with our fiscal situation." ',
'title': 'Project Toto aims to address questions regarding city finances',
'url': 'https://theaggie.org/2017/02/14/project-toto-aims-to-address-questions-regarding-city-finances/'
}Hints:
The author line is always the last line of the last paragraph.
Python 2 displays some Unicode characters as
\uXXXX. For instance,\u201cis a left-facing quotation mark. You can convert most of these to ASCII characters with the method call (on a string).translate({ 0x2018:0x27, 0x2019:0x27, 0x201C:0x22, 0x201D:0x22, 0x2026:0x20 })If you're curious about these characters, you can look them up on this page, or read more about what Unicode is.
def extract(url):
"""
extract information from the url
Argument: url
Return: a dictionary with necessary information
"""
article = requests.get(url)
news_soup = BeautifulSoup(article.content)
contentnews = news_soup.find("div", attrs = {"itemprop": "articleBody"})
content = contentnews.find_all("p")
newscontent = "\n".join([con.text for con in content])
news_split = newscontent.split("\n")
author = news_split[-1]
news = news_split[0:-1]
news = "".join(news)
news = news.translate({ 0x2018:0x27, 0x2019:0x27, 0x201C:0x22, 0x201D:0x22, 0x2026:0x20 })
tit = news_soup.findAll("h1", {"class": "entry-title"})
title = "".join([t.text for t in tit])
article_dict = {"author": author, "text": news, "title": title, "url": url}
return article_dict
url = "https://theaggie.org/2017/02/14/project-toto-aims-to-address-questions-regarding-city-finances/"
extract(url)
Exercise 1.3. Use your functions from exercises 1.1 and 1.2 to get a data frame of 60 Campus News articles and a data frame of 60 City News articles. Add a column to each that indicates the category, then combine them into one big data frame.
The "text" column of this data frame will be your corpus for natural language processing in exercise 1.4.
# get the first 4 pages to get the 60 articles
page = [i for i in range(1, 5, 1)]
# extract all 60 articles from the campus news page
article_campus = [extract_link("https://theaggie.org/campus/", num_page) for num_page in page]
url_output = [[extract(url) for url in link] for link in article_campus]
pd_campus = [pd.DataFrame(output) for output in url_output]
df_campus = pd.concat(pd_campus)
df_campus["category"] = "campus news"
df_campus.index = range(60)
# extract all 60 articles from the city news page
article_campus = [extract_link("https://theaggie.org/city/", num_page) for num_page in page]
url_output = [[extract(url) for url in link] for link in article_campus]
pd_city = [pd.DataFrame(output) for output in url_output]
df_city = pd.concat(pd_city)
df_city["category"] = "city news"
df_city.index = range(60)
# combine two dataframes
df = pd.concat([df_campus, df_city], ignore_index = True)
df.head()
df.tail()
Exercise 1.4. Use the Aggie corpus to answer the following questions. Use plots to support your analysis.
What topics does the Aggie cover the most? Do city articles typically cover different topics than campus articles?
What are the titles of the top 3 pairs of most similar articles? Examine each pair of articles. What words do they have in common?
Do you think this corpus is representative of the Aggie? Why or why not? What kinds of inference can this corpus support? Explain your reasoning.
Hints:
The nltk book and scikit-learn documentation may be helpful here.
You can determine whether city articles are "near" campus articles from the similarity matrix or with k-nearest neighbors.
If you want, you can use the wordcloud package to plot a word cloud. To install the package, run
conda install -c https://conda.anaconda.org/amueller wordcloudin a terminal. Word clouds look nice and are easy to read, but are less precise than bar plots.
import nltk
import string
import random
import itertools
from nltk.corpus import stopwords
from nltk.tokenize import word_tokenize
from nltk.stem import PorterStemmer
from nltk.stem.wordnet import WordNetLemmatizer
from nltk.probability import FreqDist
%matplotlib inline
from wordcloud import WordCloud
import matplotlib.pyplot as plt
from sklearn.feature_extraction.text import TfidfVectorizer
import pandas as pd
import numpy as np
# Use sources from the Youtube Channel
def nlp_test(text):
"""
use natural language processing to remove and lemmatize text
Argument: text
Return: stemed text
"""
stop_words = set(stopwords.words("english"))
common = ["said", "also", "one", "would", "like", "new", "many", "make", "really", "think"]
common_words = [x.decode('UTF8') for x in common]
words = nltk.word_tokenize(text)
# remove stop words and common words
words = [wor.lower() for wor in words]
filtered = [wor for wor in words if not wor in stop_words]
filtered = [wor for wor in filtered if not wor in common_words]
# word lemmatization for English words only
lmtzr = WordNetLemmatizer()
lem = [lmtzr.lemmatize(fil) for fil in filtered if fil.isalpha()]
return lem
def graph_wordCloud(text):
wordcloud = WordCloud(background_color='white', width=1200, height=1000, max_words=40).generate(text)
return plt.imshow(wordcloud)
def topic_df(dataframe):
# apply natural language processing to each row of the article dataframe
manipulate_text = dataframe.apply(lambda row: nlp_test(row["text"]), axis=1)
all_words = [w for w in manipulate_text]
merged_all_words = list(itertools.chain.from_iterable(all_words))
# Text Classification
fd = nltk.FreqDist(merged_all_words)
output = fd.most_common(25)
pd_article = pd.DataFrame(output)
pd_article.rename(columns = {list(pd_article)[0]: 'Topics', list(pd_article)[1]: 'Counts'}, inplace = True)
return pd_article
def topic_graph(dataframe):
# apply natural language processing to each row of the article dataframe
manipulate_text = dataframe.apply(lambda row: nlp_test(row["text"]), axis=1)
all_words = [w for w in manipulate_text]
merged_all_words = list(itertools.chain.from_iterable(all_words))
# Text Classification
fd = nltk.FreqDist(merged_all_words)
# Graph the text distribution
fd.plot(40,cumulative=False)
text_final = ' '.join(merged_all_words)
graph_wordCloud(text_final)
def bargraph(df_topic):
ax = df_topic.plot.bar(x='Topics', rot=0, title='Most Popular Topics', figsize=(15,10), fontsize=9)
ax.set_ylabel("Counts")
ax.set_xlabel("Topics")
patches, labels = ax.get_legend_handles_labels()
ax.legend(patches, labels, loc='best')
for p in ax.patches:
ax.annotate(str(p.get_height()), (p.get_x() * 1.005, p.get_height() * 1.005))
Below are the topics that the Aggie covers the most for the dataframe that contains city and campus articles¶
Here are the top 25 popular topics:
topic_graph(df)
df_topic = topic_df(df)
bargraph(df_topic)
Below are the topics that the Aggie covers the most for the dataframe that contains campus articles¶
topic_graph(df_campus)
campus_topic = topic_df(df_campus)
bargraph(campus_topic)
Below are the topics that the Aggie covers the most for the dataframe that contains city articles¶
topic_graph(df_city)
city_topic = topic_df(df_city)
bargraph(city_topic)
As we can see from the analysis above, the topics that Aggie covers the most are davis, student, uc, campus and people. If we separate the news based on campus and city, the campus news cover most about student, uc, davis, campus and university. The city news cover most about davis, city, community, people and year. Therefore, city articles typically cover different topics than campus articles.
Top 3 pairs of most similar articles Analysis¶
The rows and columns in the similarity matrix corresponds to the articles. Therefore, the top three entries that have the largest values/similarity scores correspond to the top 3 pairs of most similar articles.
vectorizer = TfidfVectorizer(tokenizer=nlp_test, stop_words='english',smooth_idf = True, norm = None)
tfs = vectorizer.fit_transform(df['text'])
sim_matrix = tfs.dot(tfs.T)
# convert a sparse matrix to a (2-d) NumPy matrix
array_text = sim_matrix.toarray()
array_text
# convert the array into a list
array_list = array_text.tolist()
a_list = list(itertools.chain.from_iterable(array_list))
# return 3 maximum numbers in the list
top = sorted(a_list, reverse = True)[:3]
def return_title(df):
"""
print out the top three similar articles for each dataframe
Argument: dataframe
Return: print out the title name
"""
for num in top:
var = np.nonzero(sim_matrix == num)
print df["title"][var[1][0]]
Top 3 pairs of most similar articles¶
return_title(df_campus)
return_title(df_city)
import pandas as pd
def combine_text(df_campus, df_city, num):
"""
combine the content from the campus news and the city news
Argument: df_campus, df_city, num
Return: the combined text
"""
var = np.nonzero(sim_matrix == top[num])
campus_text = df_campus["text"][var[1][0]]
city_text = df_city["text"][var[1][0]]
total_text = campus_text + city_text
return total_text
text_output = [combine_text(df_campus, df_city, num) for num in range(0, len(top))]
Below are the graphs that illustrate the common words that are shared in the top 3 pairs of most similar articles Analysis¶
The bigger the fonts are, the more they appear in both articles
graph_wordCloud(text_output[0])
Common words are Student, Campus, Davis, year, muslim, community
graph_wordCloud(text_output[1])
Common words are Davis, housing, student, field
graph_wordCloud(text_output[2])
Common words are student, UC, Abroad, Davis
Analysis¶
Yes. I think this corpus is a good representative of the Aggie because the Aggie is the newspaper that serves the UC Davis campus and community. Therefore, the topics that the authors cover should be mostly related to the Davis campus and community, which is consistent with my analysis above. We can infer that the content that the Aggie covers is mostly related to the current news. However, the focus on the campus news is different from the city news.