Truncated SVD Demo¶
In [1]:
from __future__ import print_function
import numpy as np
import json
First, we will load the data in...¶
If you have your own dataset, this cell simply makes a list of text files.
In [2]:
with open("kickstarter.jsonlist") as f:
documents = [(x['name'], x['category'], x['text'])
for x in json.loads(f.readlines()[0])
if len(x['text'].split()) > 50]
#To prove I'm not cheating with the magic trick...
np.random.shuffle(documents)
In [3]:
print("Loaded {} documents".format(len(documents)))
print("Here is one of them:")
print(documents[0])
Lets make the term-by-document matrix, and do some decompositions!¶
In [4]:
from sklearn.feature_extraction.text import TfidfVectorizer
help(TfidfVectorizer)
In [5]:
vectorizer = TfidfVectorizer(stop_words = 'english', max_df = .7,
min_df = 75)
my_matrix = vectorizer.fit_transform([x[2] for x in documents]).transpose()
In [6]:
print(type(my_matrix))
print(my_matrix.shape)
How many dimensions does our data live in?¶
In [7]:
from scipy.sparse.linalg import svds
u, s, v_trans = svds(my_matrix, k=100)
In [8]:
print(u.shape)
print(s.shape)
print(v_trans.shape)
In [9]:
import matplotlib
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
plt.plot(s[::-1])
plt.xlabel("Singular value number")
plt.ylabel("Singular value")
plt.show()
So it looks like most of our data lives in as few as 10 dimensions. Lets take 40 dimensions to be safe.¶
In [10]:
words_compressed, _, docs_compressed = svds(my_matrix, k=40)
docs_compressed = docs_compressed.transpose()
In [11]:
print(words_compressed.shape)
print(docs_compressed.shape)
It's time for some magic!¶
In [12]:
word_to_index = vectorizer.vocabulary_
index_to_word = {i:t for t,i in word_to_index.iteritems()}
print(words_compressed.shape)
In [13]:
#row normalize
from sklearn.preprocessing import normalize
words_compressed = normalize(words_compressed, axis = 1)
In [14]:
def closest_words(word_in, k = 10):
if word_in not in word_to_index: return "Not in vocab."
sims = words_compressed.dot(words_compressed[word_to_index[word_in],:])
asort = np.argsort(-sims)[:k+1]
return [(index_to_word[i],sims[i]/sims[asort[0]]) for i in asort[1:]]
Magic trick time! You pick the words.¶
Sugguestions:
- technology
- children
- vinyl
- community
- school
- food
- nuclear
- america
- fabric
In [25]:
closest_words("nuclear")
Out[25]:
In [16]:
print(word_to_index.keys()[:200])
In [17]:
from sklearn.manifold import TSNE
tsne = TSNE(verbose=1)
In [18]:
print(docs_compressed.shape)
#we'll just take the first 5K documents, because TSNE is memory intensive!
subset = docs_compressed[:5000,:]
projected_docs = tsne.fit_transform(subset)
print(projected_docs.shape)
In [19]:
plt.figure(figsize=(15,15))
plt.scatter(projected_docs[:,0],projected_docs[:,1])
plt.show()
Does this space mean anything, though? Lets color by some of the categories¶
In [20]:
from collections import Counter
cats = Counter([x[1] for x in documents])
print(cats)
In [21]:
from collections import defaultdict
cat_to_color = defaultdict(lambda: 'k')
cat_to_color.update({"Photography":'g',
"Music":'c',
"Food":'r',
"Comics": "b"})
color_to_project = defaultdict(list)
for i in range(projected_docs.shape[0]):
color_to_project[cat_to_color[documents[i][1]]].append(i)
In [22]:
plt.figure(figsize=(15,15))
for color, indices in color_to_project.iteritems():
indices = np.array(indices)
plt.scatter(projected_docs[indices,0], projected_docs[indices,1],
color = color)
plt.show()
Lets search for nearest kickstarter projects!¶
Note that we don't really need an inverted index because our document vectors are only 40 dimensional, rather than |V| dimensional.
In [23]:
docs_compressed = normalize(docs_compressed, axis = 1)
def closest_projects(project_index_in, k = 5):
sims = docs_compressed.dot(docs_compressed[project_index_in,:])
asort = np.argsort(-sims)[:k+1]
return [(documents[i][0],sims[i]/sims[asort[0]]) for i in asort[1:]]
In [24]:
for i in range(10):
print(documents[i][0])
for title, score in closest_projects(i):
print("{}:{:.3f}".format(title[:40], score))
print()