-
Notifications
You must be signed in to change notification settings - Fork 120
/
Copy pathbuzzm.html
239 lines (224 loc) · 23.6 KB
/
buzzm.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
<h1 id="buzz-model">Buzz model</h1>
<h1 id="markdown-version-of-buzz-data-analysis">Markdown version of Buzz data analysis</h1>
<p>by: Nina Zumel and John Mount Win-Vector LLC</p>
<p>To run this example you need a system with R installed (see <a href="http://cran.r-project.org" class="uri">http://cran.r-project.org</a>), and data from <a href="https://github.com/WinVector/PDSwR2/tree/master/Buzz" class="uri">https://github.com/WinVector/PDSwR2/tree/master/Buzz</a>.</p>
<p>We are not performing any new analysis here, just supplying a direct application of Random Forests on the data.</p>
<p>Data from: <a href="http://ama.liglab.fr/datasets/buzz/" class="uri">http://ama.liglab.fr/datasets/buzz/</a> Using: <a href="http://ama.liglab.fr/datasets/buzz/classification/TomsHardware/Relative_labeling/sigma=500/TomsHardware-Relative-Sigma-500.data">TomsHardware-Relative-Sigma-500.data</a></p>
<p>(described in <a href="http://ama.liglab.fr/datasets/buzz/classification/TomsHardware/Relative_labeling/sigma=500/TomsHardware-Relative-Sigma-500.names">TomsHardware-Relative-Sigma-500.names</a> )</p>
<p>Crypto hashes: shasum TomsHardware-<em>.txt </em> 5a1cc7863a9da8d6e8380e1446f25eec2032bd91 TomsHardware-Absolute-Sigma-500.data.txt * 86f2c0f4fba4fb42fe4ee45b48078ab51dba227e TomsHardware-Absolute-Sigma-500.names.txt * c239182c786baf678b55f559b3d0223da91e869c TomsHardware-Relative-Sigma-500.data.txt * ec890723f91ae1dc87371e32943517bcfcd9e16a TomsHardware-Relative-Sigma-500.names.txt</p>
<p>To run this example you need a system with R installed (see <a href="http://cran.r-project.org">cran</a>), Latex (see <a href="http://tug.org">tug</a>) and data from <a href="https://github.com/WinVector/PDSwR2/tree/master/Buzz">PDSwR2</a>.</p>
<p>To run this example: * Download buzzm.Rmd and TomsHardware-Relative-Sigma-500.data.txt from the github URL. * Start a copy of R, use setwd() to move to the directory you have stored the files. * Make sure knitr is loaded into R ( install.packages('knitr') and library(knitr) ). * In R run: (produces <a href="https://github.com/WinVector/PDSwR2/blob/master/Buzz/buzzm.md">buzzm.md</a> from buzzm.Rmd).</p>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r"><span class="kw">knit</span>(<span class="st">'buzzm.Rmd'</span>)</code></pre></div>
<p>Now you can run the following data prep steps:</p>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r">infile <-<span class="st"> "TomsHardware-Relative-Sigma-500.data.txt"</span>
<span class="kw">paste</span>(<span class="st">'checked at'</span>, <span class="kw">date</span>())</code></pre></div>
<pre><code>## [1] "checked at Thu Apr 18 09:30:23 2019"</code></pre>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r"><span class="kw">system</span>(<span class="kw">paste</span>(<span class="st">'shasum'</span>, infile), <span class="dt">intern=</span>T) <span class="co"># write down file hash</span></code></pre></div>
<pre><code>## [1] "c239182c786baf678b55f559b3d0223da91e869c TomsHardware-Relative-Sigma-500.data.txt"</code></pre>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r">buzzdata <-<span class="st"> </span><span class="kw">read.table</span>(infile, <span class="dt">header =</span> <span class="ot">FALSE</span>, <span class="dt">sep =</span> <span class="st">","</span>)
makevars <-<span class="st"> </span>function(colname, <span class="dt">ndays =</span> <span class="dv">7</span>) {
<span class="kw">sprintf</span>(<span class="st">"%s_%02g"</span>, colname, <span class="dv">0</span>:ndays)
}
varnames <-<span class="st"> </span><span class="kw">c</span>(
<span class="st">"num.new.disc"</span>,
<span class="st">"burstiness"</span>,
<span class="st">"number.total.disc"</span>,
<span class="st">"auth.increase"</span>,
<span class="st">"atomic.containers"</span>, <span class="co"># not documented</span>
<span class="st">"num.displays"</span>, <span class="co"># number of times topic displayed to user (measure of interest)</span>
<span class="st">"contribution.sparseness"</span>, <span class="co"># not documented</span>
<span class="st">"avg.auths.per.disc"</span>,
<span class="st">"num.authors.topic"</span>, <span class="co"># total authors on the topic</span>
<span class="st">"avg.disc.length"</span>,
<span class="st">"attention.level.author"</span>,
<span class="st">"attention.level.contrib"</span>
)
colnames <-<span class="st"> </span><span class="kw">unlist</span>(<span class="kw">lapply</span>(varnames, <span class="dt">FUN=</span>makevars))
colnames <-<span class="st"> </span><span class="kw">c</span>(colnames, <span class="st">"buzz"</span>)
<span class="kw">colnames</span>(buzzdata) <-<span class="st"> </span>colnames
<span class="co"># Split into training and test</span>
<span class="kw">set.seed</span>(2362690L)
rgroup <-<span class="st"> </span><span class="kw">runif</span>(<span class="kw">dim</span>(buzzdata)[<span class="dv">1</span>])
buzztrain <-<span class="st"> </span>buzzdata[rgroup ><span class="st"> </span><span class="fl">0.1</span>,]
buzztest <-<span class="st"> </span>buzzdata[rgroup <=<span class="fl">0.1</span>,]</code></pre></div>
<p>This currently returns a training set with 7114 rows and a test set with 791 rows, which is the same as when this document was prepared.</p>
<p>Notice we have exploded the basic column names into the following:</p>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r"><span class="kw">print</span>(colnames)</code></pre></div>
<pre><code>## [1] "num.new.disc_00" "num.new.disc_01"
## [3] "num.new.disc_02" "num.new.disc_03"
## [5] "num.new.disc_04" "num.new.disc_05"
## [7] "num.new.disc_06" "num.new.disc_07"
## [9] "burstiness_00" "burstiness_01"
## [11] "burstiness_02" "burstiness_03"
## [13] "burstiness_04" "burstiness_05"
## [15] "burstiness_06" "burstiness_07"
## [17] "number.total.disc_00" "number.total.disc_01"
## [19] "number.total.disc_02" "number.total.disc_03"
## [21] "number.total.disc_04" "number.total.disc_05"
## [23] "number.total.disc_06" "number.total.disc_07"
## [25] "auth.increase_00" "auth.increase_01"
## [27] "auth.increase_02" "auth.increase_03"
## [29] "auth.increase_04" "auth.increase_05"
## [31] "auth.increase_06" "auth.increase_07"
## [33] "atomic.containers_00" "atomic.containers_01"
## [35] "atomic.containers_02" "atomic.containers_03"
## [37] "atomic.containers_04" "atomic.containers_05"
## [39] "atomic.containers_06" "atomic.containers_07"
## [41] "num.displays_00" "num.displays_01"
## [43] "num.displays_02" "num.displays_03"
## [45] "num.displays_04" "num.displays_05"
## [47] "num.displays_06" "num.displays_07"
## [49] "contribution.sparseness_00" "contribution.sparseness_01"
## [51] "contribution.sparseness_02" "contribution.sparseness_03"
## [53] "contribution.sparseness_04" "contribution.sparseness_05"
## [55] "contribution.sparseness_06" "contribution.sparseness_07"
## [57] "avg.auths.per.disc_00" "avg.auths.per.disc_01"
## [59] "avg.auths.per.disc_02" "avg.auths.per.disc_03"
## [61] "avg.auths.per.disc_04" "avg.auths.per.disc_05"
## [63] "avg.auths.per.disc_06" "avg.auths.per.disc_07"
## [65] "num.authors.topic_00" "num.authors.topic_01"
## [67] "num.authors.topic_02" "num.authors.topic_03"
## [69] "num.authors.topic_04" "num.authors.topic_05"
## [71] "num.authors.topic_06" "num.authors.topic_07"
## [73] "avg.disc.length_00" "avg.disc.length_01"
## [75] "avg.disc.length_02" "avg.disc.length_03"
## [77] "avg.disc.length_04" "avg.disc.length_05"
## [79] "avg.disc.length_06" "avg.disc.length_07"
## [81] "attention.level.author_00" "attention.level.author_01"
## [83] "attention.level.author_02" "attention.level.author_03"
## [85] "attention.level.author_04" "attention.level.author_05"
## [87] "attention.level.author_06" "attention.level.author_07"
## [89] "attention.level.contrib_00" "attention.level.contrib_01"
## [91] "attention.level.contrib_02" "attention.level.contrib_03"
## [93] "attention.level.contrib_04" "attention.level.contrib_05"
## [95] "attention.level.contrib_06" "attention.level.contrib_07"
## [97] "buzz"</code></pre>
<p>We are now ready to create a simple model predicting "buzz" as function of the other columns.</p>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r"><span class="co"># build a model</span>
<span class="co"># let's use all the input variables</span>
nlist =<span class="st"> </span>varnames
varslist =<span class="st"> </span><span class="kw">as.vector</span>(<span class="kw">sapply</span>(nlist, <span class="dt">FUN=</span>makevars))
<span class="co"># these were defined previously in Practical Data Science with R</span>
loglikelihood <-<span class="st"> </span>function(y, py) {
pysmooth <-<span class="st"> </span><span class="kw">ifelse</span>(py ==<span class="st"> </span><span class="dv">0</span>, <span class="fl">1e-12</span>,
<span class="kw">ifelse</span>(py ==<span class="st"> </span><span class="dv">1</span>, <span class="dv">1</span><span class="fl">-1e-12</span>, py))
<span class="kw">sum</span>(y *<span class="st"> </span><span class="kw">log</span>(pysmooth) +<span class="st"> </span>(<span class="dv">1</span> -<span class="st"> </span>y) *<span class="st"> </span><span class="kw">log</span>(<span class="dv">1</span> -<span class="st"> </span>pysmooth))
}
accuracyMeasures <-<span class="st"> </span>function(pred, truth, <span class="dt">threshold=</span><span class="fl">0.5</span>, <span class="dt">name=</span><span class="st">"model"</span>) {
dev.norm <-<span class="st"> </span>-<span class="dv">2</span> *<span class="st"> </span><span class="kw">loglikelihood</span>(<span class="kw">as.numeric</span>(truth), pred) /<span class="st"> </span><span class="kw">length</span>(pred)
ctable =<span class="st"> </span><span class="kw">table</span>(<span class="dt">truth =</span> truth,
<span class="dt">pred =</span> pred ><span class="st"> </span>threshold)
accuracy <-<span class="st"> </span><span class="kw">sum</span>(<span class="kw">diag</span>(ctable)) /<span class="st"> </span><span class="kw">sum</span>(ctable)
precision <-<span class="st"> </span>ctable[<span class="dv">2</span>, <span class="dv">2</span>] /<span class="st"> </span><span class="kw">sum</span>(ctable[, <span class="dv">2</span>])
recall <-<span class="st"> </span>ctable[<span class="dv">2</span>, <span class="dv">2</span>] /<span class="st"> </span><span class="kw">sum</span>(ctable[<span class="dv">2</span>, ])
f1 <-<span class="st"> </span><span class="dv">2</span> *<span class="st"> </span>precision *<span class="st"> </span>recall /<span class="st"> </span>(precision +<span class="st"> </span>recall)
<span class="kw">print</span>(<span class="kw">paste</span>(<span class="st">"precision="</span>, precision, <span class="st">"; recall="</span> , recall))
<span class="kw">print</span>(ctable)
<span class="kw">data.frame</span>(<span class="dt">model =</span> name,
<span class="dt">accuracy =</span> accuracy,
<span class="dt">f1 =</span> f1,
<span class="dt">dev.norm =</span> dev.norm,
<span class="dt">AUC =</span> sigr::<span class="kw">calcAUC</span>(pred, truth))
}
<span class="kw">library</span>(<span class="st">"randomForest"</span>)</code></pre></div>
<pre><code>## randomForest 4.6-14
## Type rfNews() to see new features/changes/bug fixes.</code></pre>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r">bzFormula <-<span class="st"> </span><span class="kw">paste</span>(<span class="st">'as.factor(buzz) ~ '</span>, <span class="kw">paste</span>(varslist, <span class="dt">collapse =</span> <span class="st">' + '</span>))
fmodel <-<span class="st"> </span><span class="kw">randomForest</span>(<span class="kw">as.formula</span>(bzFormula),
<span class="dt">data =</span> buzztrain,
<span class="dt">importance =</span> <span class="ot">TRUE</span>)
<span class="kw">print</span>(<span class="st">'training'</span>)</code></pre></div>
<pre><code>## [1] "training"</code></pre>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r">rtrain <-<span class="st"> </span><span class="kw">data.frame</span>(<span class="dt">truth =</span> buzztrain$buzz,
<span class="dt">pred =</span> <span class="kw">predict</span>(fmodel, <span class="dt">newdata =</span> buzztrain, <span class="dt">type=</span><span class="st">"prob"</span>)[, <span class="dv">2</span>, <span class="dt">drop =</span> <span class="ot">TRUE</span>])
<span class="kw">print</span>(<span class="kw">accuracyMeasures</span>(rtrain$pred, rtrain$truth))</code></pre></div>
<pre><code>## [1] "precision= 1 ; recall= 0.999360613810742"
## pred
## truth FALSE TRUE
## 0 5550 0
## 1 1 1563
## model accuracy f1 dev.norm AUC
## 1 model 0.9998594 0.9996802 0.1049166 1</code></pre>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r">WVPlots::<span class="kw">ROCPlot</span>(rtrain, <span class="st">"pred"</span>, <span class="st">"truth"</span>, <span class="ot">TRUE</span>, <span class="st">"RF train performance, large model"</span>)</code></pre></div>
<div class="figure">
<img src="buzzm_files/figure-markdown_github/model-1.png" alt="" />
</div>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r"><span class="kw">print</span>(<span class="st">'test'</span>)</code></pre></div>
<pre><code>## [1] "test"</code></pre>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r">rtest <-<span class="st"> </span><span class="kw">data.frame</span>(<span class="dt">truth =</span> buzztest$buzz,
<span class="dt">pred =</span> <span class="kw">predict</span>(fmodel, <span class="dt">newdata=</span>buzztest, <span class="dt">type=</span><span class="st">"prob"</span>)[, <span class="dv">2</span>, <span class="dt">drop =</span> <span class="ot">TRUE</span>])
<span class="kw">print</span>(<span class="kw">accuracyMeasures</span>(rtest$pred, rtest$truth))</code></pre></div>
<pre><code>## [1] "precision= 0.832402234636871 ; recall= 0.84180790960452"
## pred
## truth FALSE TRUE
## 0 584 30
## 1 28 149
## model accuracy f1 dev.norm AUC
## 1 model 0.9266751 0.8370787 0.42056 0.9702102</code></pre>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r">WVPlots::<span class="kw">ROCPlot</span>(rtest, <span class="st">"pred"</span>, <span class="st">"truth"</span>, <span class="ot">TRUE</span>, <span class="st">"RF train performance, large model"</span>)</code></pre></div>
<div class="figure">
<img src="buzzm_files/figure-markdown_github/model-2.png" alt="" />
</div>
<p>Notice the extreme fall-off from training to test performance, the random forest over fit on training. We see good accuracy on test (around 92%), but not the perfect fit seen on training.</p>
<p>To try and control the over-fitting we build a new model with the tree complexity limited to 50 nodes and the node size to at least 100. This is not necessarily a better model (in fact it scores slightly poorer on test), but it is one where the training procedure didn't have enough freedom to memorize the training data (and therefore maybe had better visibility into some trade-offs.</p>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r">fmodel <-<span class="st"> </span><span class="kw">randomForest</span>(<span class="kw">as.formula</span>(bzFormula),
<span class="dt">data =</span> buzztrain,
<span class="dt">maxnodes =</span> <span class="dv">50</span>,
<span class="dt">nodesize =</span> <span class="dv">100</span>,
<span class="dt">importance =</span> <span class="ot">TRUE</span>)
<span class="kw">print</span>(<span class="st">'training'</span>)</code></pre></div>
<pre><code>## [1] "training"</code></pre>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r">rtrain <-<span class="st"> </span><span class="kw">data.frame</span>(<span class="dt">truth =</span> buzztrain$buzz,
<span class="dt">pred =</span> <span class="kw">predict</span>(fmodel, <span class="dt">newdata=</span>buzztrain, <span class="dt">type=</span><span class="st">"prob"</span>)[, <span class="dv">2</span>, <span class="dt">drop =</span> <span class="ot">TRUE</span>])
<span class="kw">print</span>(<span class="kw">accuracyMeasures</span>(rtrain$pred, rtrain$truth))</code></pre></div>
<pre><code>## [1] "precision= 0.809937888198758 ; recall= 0.833759590792839"
## pred
## truth FALSE TRUE
## 0 5244 306
## 1 260 1304
## model accuracy f1 dev.norm AUC
## 1 model 0.9204386 0.8216761 0.3412138 0.9763961</code></pre>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r"><span class="kw">print</span>(<span class="st">'test'</span>)</code></pre></div>
<pre><code>## [1] "test"</code></pre>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r">rtest <-<span class="st"> </span><span class="kw">data.frame</span>(<span class="dt">truth =</span> buzztest$buzz,
<span class="dt">pred =</span> <span class="kw">predict</span>(fmodel, <span class="dt">newdata=</span>buzztest, <span class="dt">type=</span><span class="st">"prob"</span>)[, <span class="dv">2</span>, <span class="dt">drop =</span> <span class="ot">TRUE</span>])
<span class="kw">print</span>(<span class="kw">accuracyMeasures</span>(rtest$pred, rtest$truth))</code></pre></div>
<pre><code>## [1] "precision= 0.816666666666667 ; recall= 0.830508474576271"
## pred
## truth FALSE TRUE
## 0 581 33
## 1 30 147
## model accuracy f1 dev.norm AUC
## 1 model 0.920354 0.8235294 0.5751291 0.9624625</code></pre>
<p>And we can also make plots.</p>
<p>Training performance:</p>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r">WVPlots::<span class="kw">ROCPlot</span>(rtrain, <span class="st">"pred"</span>, <span class="st">"truth"</span>, <span class="ot">TRUE</span>, <span class="st">"RF train performance, simpler model"</span>)</code></pre></div>
<div class="figure">
<img src="buzzm_files/figure-markdown_github/plottrain-1.png" alt="" />
</div>
<p>Test performance:</p>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r">WVPlots::<span class="kw">ROCPlot</span>(rtest, <span class="st">"pred"</span>, <span class="st">"truth"</span>, <span class="ot">TRUE</span>, <span class="st">"RF test performance, simpler model"</span>)</code></pre></div>
<div class="figure">
<img src="buzzm_files/figure-markdown_github/plottest-1.png" alt="" />
</div>
<p>Notice this has similar test performance as the first model. This is typical of random forests: the degree of over-fit in the training data is not a good predictor of good or bad performance on test data.</p>
<p>So we are now left with a choice, unfortunately without clear guidance: which model do we use? In this case we are going to say: use the simpler model fit on all the data. The idea is the simpler model didn't test much worse and more data lets helps the model reduce over-fitting. We will return the simpler model fit only on training data, so we have some control and some disjoint test data to evaluate the model on.</p>
<p>Save a sample of the test data.</p>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r">sample_test <-<span class="st"> </span>buzztest[<span class="kw">sample.int</span>(<span class="kw">nrow</span>(buzztest), <span class="dv">100</span>), , drop =<span class="st"> </span><span class="ot">FALSE</span>]
<span class="kw">write.csv</span>(sample_test, <span class="st">"buzz_sample.csv"</span>,
<span class="dt">row.names =</span> <span class="ot">FALSE</span>,
<span class="dt">quote =</span> <span class="ot">FALSE</span>)</code></pre></div>
<p>Save variable names, model, and test data.</p>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r">fname <-<span class="st"> 'thRS500.RDS'</span>
items <-<span class="st"> </span><span class="kw">c</span>(<span class="st">"varslist"</span>, <span class="st">"fmodel"</span>, <span class="st">"buzztest"</span>)
<span class="kw">saveRDS</span>(<span class="dt">object =</span> <span class="kw">list</span>(<span class="dt">varslist =</span> varslist,
<span class="dt">fmodel =</span> fmodel,
<span class="dt">buzztest =</span> buzztest),
<span class="dt">file =</span> fname)
<span class="kw">message</span>(<span class="kw">paste</span>(<span class="st">'saved'</span>, fname)) <span class="co"># message to running R console</span>
<span class="kw">print</span>(<span class="kw">paste</span>(<span class="st">'saved'</span>, fname)) <span class="co"># print to document</span></code></pre></div>
<pre><code>## [1] "saved thRS500.RDS"</code></pre>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r"><span class="kw">paste</span>(<span class="st">'finished at'</span>, <span class="kw">date</span>())</code></pre></div>
<pre><code>## [1] "finished at Thu Apr 18 09:33:05 2019"</code></pre>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r"><span class="kw">system</span>(<span class="kw">paste</span>(<span class="st">'shasum'</span>, fname), <span class="dt">intern =</span> <span class="ot">TRUE</span>) <span class="co"># write down file hash</span></code></pre></div>
<pre><code>## [1] "f2b3b80bc6c5a72079b39308a5758a282bcdd5bf thRS500.RDS"</code></pre>