cacheing possible for cr_cn?

[HeidiSeibold]

I am downloading citation infos for quite some DOIs. Would be nice to have the ones I already downloaded in cache. Would that be possible?

Session Info

> sessionInfo()
R version 3.5.1 (2018-07-02)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Debian GNU/Linux buster/sid

Matrix products: default
BLAS: /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.8.0
LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.8.0

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C               LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
 [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8    LC_PAPER=en_US.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C             LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] rcrossref_0.9.0    kableExtra_1.0.1   plyr_1.8.4         googlesheets_0.3.0 DiagrammeRsvg_0.1 
[6] DiagrammeR_1.0.0   knitr_1.20        

loaded via a namespace (and not attached):
 [1] Rcpp_1.0.0         tidyr_0.8.2        visNetwork_2.0.5   assertthat_0.2.0   rprojroot_1.3-2   
 [6] digest_0.6.18      V8_2.0             mime_0.6           R6_2.3.0           cellranger_1.1.0  
[11] backports_1.1.2    evaluate_0.11      httr_1.4.0         ggplot2_3.1.0      pillar_1.3.1      
[16] rlang_0.3.1        lazyeval_0.2.1     curl_3.3           rstudioapi_0.9.0   miniUI_0.1.1.1    
[21] DT_0.5             rmarkdown_1.10     urltools_1.7.1     webshot_0.5.1      downloader_0.4    
[26] readr_1.3.1        stringr_1.3.1      htmlwidgets_1.3    triebeard_0.3.0    igraph_1.2.2      
[31] munsell_0.5.0      shiny_1.1.0        compiler_3.5.1     influenceR_0.1.0   rgexf_0.15.3      
[36] httpuv_1.4.5.1     pkgconfig_2.0.2    htmltools_0.3.6    tidyselect_0.2.5   httpcode_0.2.0    
[41] tibble_2.0.1       gridExtra_2.3      XML_3.98-1.16      viridisLite_0.3.0  crayon_1.3.4      
[46] dplyr_0.7.8        later_0.7.5        crul_0.7.0         grid_3.5.1         xtable_1.8-3      
[51] jsonlite_1.6       gtable_0.2.0       magrittr_1.5       scales_1.0.0       bibtex_0.4.2      
[56] stringi_1.2.4      promises_1.0.1     viridis_0.5.1      bindrcpp_0.2.2     xml2_1.2.0        
[61] brew_1.0-6         RColorBrewer_1.1-2 tools_3.5.1        glue_1.3.0         purrr_0.3.0       
[66] hms_0.4.2          Rook_1.1-1         yaml_2.2.0         colorspace_1.3-2   rvest_0.3.2       
[71] bindr_0.1.1   

[sckott]

thanks for the issue @HeidiSeibold Its definitely possible, but will have to think more about it.

So I can get a very clear look at the use case:

• do you imagine only wanting this for cr_cn, or other fxns, maybe that's the only one you use?
• do you care what the cache is? do you want to cache to a file path that you set? or let the pkg take care of the caching behind the scenes and you just get to pull from the cache when you've made the request before?

[HeidiSeibold]

Hi @sckott, thanks for the quick answer.

I've been using only cr_cn() so far to get paper infos from DOIs and to me it does not really matter what happens behind the scenes as long as it is faster than downloading the information (sometimes my connection is a bit slow, which is why I am asking for the cacheing in the first place 😉). Does that answer your questions?

Previously I'd been using https://www.doi2bib.org/, but cr_cn() allows me more flexibility and automation which I like a lot :)

[sckott]

thanks for the further details @HeidiSeibold

slow connection is a good reason, and even if you have a fast one, it's good to not have to wait for a query you've made recently already

I'm thinking this through now. Hoping to have something to try soon ...

[sckott]

@HeidiSeibold sorry for the delay on this, could add a temporary solution, but wanted to get a good solution that is general purpose and robust, etc. - working on it here https://github.com/ropenscilabs/webmiddens/

[HeidiSeibold]

No worries, it's not something urgent. Thanks for letting me know ;)

[sckott]

okay, thanks

[sckott]

@HeidiSeibold if you have time, i have something ready for you to try now:

remotes::install_github("ropensci/rcrossref@webmiddens")
library(rcrossref)
library(webmiddens)
wm_enable()
wm_configuration("foostuff")

# first request is a real http request
cr_cn(dois="10.1126/science.169.3946.635", verbose=TRUE)
# second request uses cached data
cr_cn(dois="10.1126/science.169.3946.635", verbose=TRUE)

# to delete cached files you can get the current "midden"
x <- webmiddens::midden_current()
# then "destroy" it
x$destroy()

[sckott]

@maelle if you have a chance, also curious to hear your input on this - see above comment ^^

[maelle]

comments in #202

[maelle]

I installed using your instructions but it doesn't seem anything is cached?

library(rcrossref)
webmiddens::wm_enable()
webmiddens::wm_configuration("foo2")

# first request is a real http request
cr_cn(dois="10.1126/science.169.3946.635", verbose=TRUE)
#> [1] "@article{Frank_1970,\n\tdoi = {10.1126/science.169.3946.635},\n\turl = {https://doi.org/10.1126%2Fscience.169.3946.635},\n\tyear = 1970,\n\tmonth = {aug},\n\tpublisher = {American Association for the Advancement of Science ({AAAS})},\n\tvolume = {169},\n\tnumber = {3946},\n\tpages = {635--641},\n\tauthor = {H. S. Frank},\n\ttitle = {The Structure of Ordinary Water: New data and interpretations are yielding new insights into this fascinating substance},\n\tjournal = {Science}\n}"
webmiddens::midden_current() -> midden
midden$cache$files()
#> NULL
# second request uses cached data
cr_cn(dois="10.1002/app.27716", verbose=TRUE)
#> [1] "@article{Tang_2008,\n\tdoi = {10.1002/app.27716},\n\turl = {https://doi.org/10.1002%2Fapp.27716},\n\tyear = 2008,\n\tpublisher = {Wiley},\n\tvolume = {108},\n\tnumber = {3},\n\tpages = {1490--1495},\n\tauthor = {Qunwei Tang and Jianming Lin and Jihuai Wu},\n\ttitle = {The preparation and electrical conductivity of polyacrylamide/graphite conducting hydrogel},\n\tjournal = {Journal of Applied Polymer Science}\n}"

^{Created on 2020-03-28 by the reprex package (v0.3.0)}

Session info

devtools::session_info()
#> ─ Session info ───────────────────────────────────────────────────────────────
#>  setting  value                       
#>  version  R version 3.6.3 (2020-02-29)
#>  os       Ubuntu 18.04.4 LTS          
#>  system   x86_64, linux-gnu           
#>  ui       X11                         
#>  language en_US                       
#>  collate  en_US.UTF-8                 
#>  ctype    en_US.UTF-8                 
#>  tz       Europe/Paris                
#>  date     2020-03-28                  
#> 
#> ─ Packages ───────────────────────────────────────────────────────────────────
#>  package     * version    date       lib
#>  assertthat    0.2.1      2019-03-21 [1]
#>  backports     1.1.5      2019-10-02 [1]
#>  bibtex        0.4.2.2    2020-01-02 [1]
#>  callr         3.4.2      2020-02-12 [1]
#>  cli           2.0.2      2020-02-28 [1]
#>  crayon        1.3.4      2017-09-16 [1]
#>  crul          0.9.2.91   2020-03-28 [1]
#>  curl          4.3        2019-12-02 [1]
#>  desc          1.2.0      2019-11-04 [1]
#>  devtools      2.2.2      2020-02-17 [1]
#>  digest        0.6.25     2020-02-23 [1]
#>  dplyr         0.8.5      2020-03-07 [1]
#>  DT            0.13       2020-03-23 [1]
#>  ellipsis      0.3.0      2019-09-20 [1]
#>  evaluate      0.14       2019-05-28 [1]
#>  fansi         0.4.1      2020-01-08 [1]
#>  fastmap       1.0.1      2019-10-08 [1]
#>  fauxpas       0.2.0      2018-03-01 [1]
#>  fs            1.3.1      2019-05-06 [1]
#>  glue          1.3.2      2020-03-12 [1]
#>  highr         0.8        2019-03-20 [1]
#>  hoardr        0.5.2      2018-12-02 [1]
#>  htmltools     0.4.0      2019-10-04 [1]
#>  htmlwidgets   1.5.1      2019-10-08 [1]
#>  httpcode      0.2.0      2016-11-14 [1]
#>  httpuv        1.5.2      2019-09-11 [1]
#>  httr          1.4.1      2019-08-05 [1]
#>  jsonlite      1.6.1      2020-02-02 [1]
#>  knitr         1.28       2020-02-06 [1]
#>  later         1.0.0.9002 2020-03-05 [1]
#>  magrittr      1.5        2014-11-22 [1]
#>  memoise       1.1.0.9000 2020-01-06 [1]
#>  mime          0.9        2020-02-04 [1]
#>  miniUI        0.1.1.1    2018-05-18 [1]
#>  pillar        1.4.3      2019-12-20 [1]
#>  pkgbuild      1.0.6      2019-10-09 [1]
#>  pkgconfig     2.0.3      2019-09-22 [1]
#>  pkgload       1.0.2      2018-10-29 [1]
#>  plyr          1.8.6      2020-03-03 [1]
#>  prettyunits   1.1.1      2020-01-24 [1]
#>  processx      3.4.2      2020-02-09 [1]
#>  promises      1.1.0.9000 2020-03-05 [1]
#>  ps            1.3.2      2020-02-13 [1]
#>  purrr         0.3.3      2019-10-18 [1]
#>  R6            2.4.1      2019-11-12 [1]
#>  rappdirs      0.3.1      2016-03-28 [1]
#>  Rcpp          1.0.4      2020-03-17 [1]
#>  rcrossref   * 1.0.0.93   2020-03-28 [1]
#>  remotes       2.1.1      2020-02-15 [1]
#>  rlang         0.4.5      2020-03-01 [1]
#>  rmarkdown     2.1        2020-01-20 [1]
#>  rprojroot     1.3-2      2018-01-03 [1]
#>  sessioninfo   1.1.1      2018-11-05 [1]
#>  shiny         1.4.0.2    2020-03-13 [1]
#>  stringi       1.4.6      2020-02-17 [1]
#>  stringr       1.4.0      2019-02-10 [1]
#>  testthat      2.3.2      2020-03-02 [1]
#>  tibble        2.1.3      2019-06-06 [1]
#>  tidyselect    1.0.0      2020-01-27 [1]
#>  triebeard     0.3.0      2016-08-04 [1]
#>  urltools      1.7.3      2019-04-14 [1]
#>  usethis       1.5.1.9000 2020-02-26 [1]
#>  webmiddens    0.0.3.91   2020-03-28 [1]
#>  webmockr      0.6.2.91   2020-03-28 [1]
#>  withr         2.1.2      2018-03-15 [1]
#>  xfun          0.12       2020-01-13 [1]
#>  xml2          1.2.5      2020-03-11 [1]
#>  xtable        1.8-4      2019-04-21 [1]
#>  yaml          2.2.1      2020-02-01 [1]
#>  source                                  
#>  CRAN (R 3.6.0)                          
#>  CRAN (R 3.6.1)                          
#>  CRAN (R 3.6.3)                          
#>  CRAN (R 3.6.1)                          
#>  CRAN (R 3.6.1)                          
#>  CRAN (R 3.6.0)                          
#>  Github (ropensci/crul@0de245e)          
#>  CRAN (R 3.6.1)                          
#>  Github (r-lib/desc@61205f6)             
#>  CRAN (R 3.6.1)                          
#>  CRAN (R 3.6.1)                          
#>  CRAN (R 3.6.1)                          
#>  CRAN (R 3.6.3)                          
#>  CRAN (R 3.6.1)                          
#>  CRAN (R 3.6.0)                          
#>  CRAN (R 3.6.1)                          
#>  CRAN (R 3.6.1)                          
#>  CRAN (R 3.6.1)                          
#>  CRAN (R 3.6.0)                          
#>  CRAN (R 3.6.1)                          
#>  CRAN (R 3.6.0)                          
#>  CRAN (R 3.6.3)                          
#>  CRAN (R 3.6.0)                          
#>  CRAN (R 3.6.1)                          
#>  CRAN (R 3.6.0)                          
#>  CRAN (R 3.6.0)                          
#>  CRAN (R 3.6.0)                          
#>  CRAN (R 3.6.1)                          
#>  CRAN (R 3.6.1)                          
#>  Github (r-lib/later@0fb877a)            
#>  CRAN (R 3.6.0)                          
#>  Github (hadley/memoise@58d3972)         
#>  CRAN (R 3.6.1)                          
#>  CRAN (R 3.6.0)                          
#>  CRAN (R 3.6.1)                          
#>  CRAN (R 3.6.1)                          
#>  CRAN (R 3.6.1)                          
#>  CRAN (R 3.6.0)                          
#>  CRAN (R 3.6.3)                          
#>  CRAN (R 3.6.1)                          
#>  CRAN (R 3.6.1)                          
#>  Github (rstudio/promises@627dfc6)       
#>  CRAN (R 3.6.1)                          
#>  CRAN (R 3.6.1)                          
#>  CRAN (R 3.6.1)                          
#>  CRAN (R 3.6.0)                          
#>  CRAN (R 3.6.1)                          
#>  Github (ropensci/rcrossref@50673d4)     
#>  CRAN (R 3.6.1)                          
#>  CRAN (R 3.6.1)                          
#>  CRAN (R 3.6.1)                          
#>  CRAN (R 3.6.0)                          
#>  CRAN (R 3.6.0)                          
#>  CRAN (R 3.6.3)                          
#>  CRAN (R 3.6.1)                          
#>  CRAN (R 3.6.0)                          
#>  CRAN (R 3.6.1)                          
#>  CRAN (R 3.6.0)                          
#>  CRAN (R 3.6.1)                          
#>  CRAN (R 3.6.0)                          
#>  CRAN (R 3.6.0)                          
#>  Github (r-lib/usethis@2a3d134)          
#>  Github (ropenscilabs/webmiddens@1d84f8e)
#>  Github (ropensci/webmockr@c7772eb)      
#>  CRAN (R 3.6.0)                          
#>  CRAN (R 3.6.1)                          
#>  CRAN (R 3.6.3)                          
#>  CRAN (R 3.6.0)                          
#>  CRAN (R 3.6.1)                          
#> 
#> [1] /home/maelle/R/x86_64-pc-linux-gnu-library/3.6
#> [2] /usr/local/lib/R/site-library
#> [3] /usr/lib/R/site-library
#> [4] /usr/lib/R/library

[sckott]

woopsy, for now need to explicitly load webmiddens with library(webmiddens), see changed instructions above. I also meant to have the same cr_cn request, so fixed that. I'll have to figure out if I can make it work without having to load webmiddens

[maelle]

aaah ok now it works. As written in my comments I'd expect the enabling/configuration to be specific to rcrossref (i.e. if I write footstuff the cache would be in /home/maelle/.cache/R/rcrossref/foostuff not /home/maelle/.cache/R/foostuff

library(rcrossref)
library(webmiddens)
wm_enable()
wm_configuration("foostuff")

# first request is a real http request
cr_cn(dois="10.1126/science.169.3946.635", verbose=TRUE)
#> [1] "@article{Frank_1970,\n\tdoi = {10.1126/science.169.3946.635},\n\turl = {https://doi.org/10.1126%2Fscience.169.3946.635},\n\tyear = 1970,\n\tmonth = {aug},\n\tpublisher = {American Association for the Advancement of Science ({AAAS})},\n\tvolume = {169},\n\tnumber = {3946},\n\tpages = {635--641},\n\tauthor = {H. S. Frank},\n\ttitle = {The Structure of Ordinary Water: New data and interpretations are yielding new insights into this fascinating substance},\n\tjournal = {Science}\n}"
webmiddens::midden_current() -> midden
midden$cache$files()
#> [[1]]
#> <hoard file> 
#>   path: /home/maelle/.cache/R/foostuff/_middens4d455d66b7f2
#> 
#> [[2]]
#> <hoard file> 
#>   path: /home/maelle/.cache/R/foostuff/_middens4d459685e26
# second request uses cached data
cr_cn(dois="10.1126/science.169.3946.635", verbose=TRUE)
#> [1] "@article{Frank_1970,\n\tdoi = {10.1126/science.169.3946.635},\n\turl = {https://doi.org/10.1126%2Fscience.169.3946.635},\n\tyear = 1970,\n\tmonth = {aug},\n\tpublisher = {American Association for the Advancement of Science ({AAAS})},\n\tvolume = {169},\n\tnumber = {3946},\n\tpages = {635--641},\n\tauthor = {H. S. Frank},\n\ttitle = {The Structure of Ordinary Water: New data and interpretations are yielding new insights into this fascinating substance},\n\tjournal = {Science}\n}"

# to delete cached files you can get the current "midden"
x <- webmiddens::midden_current()

^{Created on 2020-03-28 by the reprex package (v0.3.0)}

[sckott]

glad it works now!

that's a good idea to have the cache folder be specific to the package its being used in.