Skip to content

BrowserCacheFeature

Jump to bottom Edit New page
Jim Miller edited this page Nov 17, 2023 · 22 revisions

Browser Cache Feature

Updated as of FanFicFare v4.20. Earlier versions, refer to this version.

FanFicFare has the ability to read your Firefox, Chrome or Chromium based browser cache instead of (or in addition to) going out to the servers for them. Primary focus is on fanfiction.net, fictionpress.com or ficbook.net sites, but it may work on some additional sites now. Sites that require HTTP POST for login or don't allow caching of pages will not work with this feature.

As of March 2023, ficbook.net isn't allowing caching of story pages anymore.

The reason to do this is for when FFF is failing due to CloudFlare, but you can still see the story pages manually in your browser.

This feature has been tested on Firefox and Chrome browsers on Windows, Mac and Linux. Users report it works with Chromium-based browsers Vivaldi and Microsoft Edge and Firefox-based browser Waterfox.

As of v4.20, this features does not work with the FanFictionDownloader program.

To use this (with fanfiction.net):

  1. Figure out where your browser cache directory is and in personal.ini (see INI File page for location) set browser_cache_path under [defaults]. Many browsers will tell you your cache location if you enter about:cache into the address bar. There are additional suggestions listed below.

  2. Set use_browser_cache:true under [www.fanfiction.net].

  3. Navigate in your browser to chapter 1 of the the story you want to download.

  4. Click the cover image thumbnail to see the bigger version--if you care about covers.

  5. Click the author link once if you care about not getting the author image as cover.

  6. Click through all the chapters of the story. Or, if you are updating an existing story, click the first and last chapters (or all new chapters).

    Note: If you see a Cloudflare page before you see a story chapter page (usually when you first go to the site), that page is not loaded in cache correctly. You need to load it again by clicking in the address bar and hitting Enter (remove ?__cf... you may see), or by clicking to a different chapter/page and clicking a link back (not your browser back button). You may also need to hit Ctrl-R or click the reload button--if you see a 'Resend data' dialog on reload, you aren't on the direct page yet.

    Also see the section below about WebToEpub.

  7. Download story in FanFicFare as normal.

Alternatively, starting with v4.20, you can skip steps 3-6 and instead set open_pages_in_browser:true in personal.ini. FFF will open pages it doesn't already find in browser cache in your default browser to populate the browser cache. This requires that you use your default browser with browser_cache_path.

Notes:

  • It has been common lately (writing in Feb 2021) for fanfiction.net update emails to arrive before the update has been processed on all fanfiction.net's servers. I recommend waiting an hour or so after receiving an update email before trying to update the story in FFF, even with Browser Cache.
  • As of v4.20, Browser Cache looks up and reads URLs in your browser's cache real time instead of scanning all cache entries. This is fast, but requires matching the exact URL.
  • Still uses browser_cache_age_limit to determine when a page is too old (default: 4 hours)--not the cache max-age given by the server (typically 15 minutes). If you viewed a story recently, but before an update, your cache may well contain an out-dated version.
  • Most browsers allow you to 'force reload' a page by holding the shift or control key while clicking or pressing reload(aka refresh). Which exactly varies by browser--Google will be able to tell you. This will update the browser cache for that page.
  • You can use this with multiple stories at a time, just remember that browser_cache_age_limit applies.
  • If you are a fan of the check_next_chapter feature, you can view a chapter one number higher by manually editing the address in the address bar. But if it isn't found and you have open_pages_in_browser:true, FFF may pop open that page several times as ffnet doesn't allow Not Found pages to cache.

Browser Cache Settings

defaults.ini settings:

Copy what you need from defaults.ini to personal.ini.

[defaults]
## As a work around for certain sites blocking automated downloads,
## FFF now offers the ability to look for pages in your Chrome(or
## Chromium-derived) or Firefox browser's cache.  Requires both
## use_browser_cache:true and browser_cache_path to be set.
##
## browser_cache_path needs to be set to the location of YOUR browser
## cache.  Here are a few examples for different OS.  Note that all
## have YOUR user name in them somewhere as well as a Profile name,
## frequently Default.  Make sure you have personalized (and
## uncommented) ONE browser_cache_path setting.  Also note that Chrome
## browsers added an additional directory level, 'Cache_Data', in
## early 2022.

## Note also that browser_cache_path goes under [defaults] because
## browser cache is read once and shared between all downloads for
## performance.

## Further note that there is a time cost to first reading the browser
## cache on each download session (not Calibre session)

## windows:
### Chrome:
#browser_cache_path:C:\Users\YourUser\AppData\Local\Google\Chrome\User Data\Default\Cache\Cache_Data
#browser_cache_path:C:\Users\YourUser\AppData\Local\Google\Chrome\User Data\Profile 1\Cache\Cache_Data
### Firefox
#browser_cache_path:C:\Users\YourUser\AppData\Local\Mozilla\Firefox\Profiles\ZjwI7Fo4.default\cache2

## mac:
### Chrome:
#browser_cache_path:/Users/your.user/Library/Caches/Google/Chrome/Default/Cache/Cache_Data
#browser_cache_path:/Users/your.user/Library/Caches/Google/Chrome/Profile 2/Cache/Cache_Data
### Firefox
#browser_cache_path:/Users/your.user/Library/Caches/Firefox/Profiles/43fkezvc.default-release/cache2

## linux:
### Chrome:
#browser_cache_path:/home/youruser/.cache/google-chrome/Default/Cache/Cache_Data
#browser_cache_path:/home/youruser/.cache/google-chrome/Profile 1/Cache/Cache_Data
### Firefox
#browser_cache_path:/home/youruser/.cache/mozilla/firefox/dk4o1y83.default-release/cache2

## It's common for browser cached files to be kept for several hours
## or more.  You can limit the age of cached files FFF will use from
## browser_cache_path with browser_cache_age_limit.  Only cached files
## that were downloaded within 'browser_cache_age_limit' hours will be
## used.  If set to -1, all cached files will be used.
browser_cache_age_limit:4.0

Further Settings:

[www.fanfiction.net]

## browser cache will only be used if use_browser_cache:true and ONLY
## for fanfiction.net right now.  Requires a browser_cache_path set in
## [defaults].
use_browser_cache:true

## use_browser_cache_only:true prevents FFF from falling through to
## request from the server when a page isn't in the browser cache.
## You can turn this on to be sure you're only using the browser
## cache.
#use_browser_cache_only:false

## If browser_cache_path is set *and* use_browser_cache:true *and*
## use_browser_cache_only:true, then you can also set
## open_pages_in_browser:true then FFF to attempt to open each page it
## can't already find in browser cache in your default browser, then
## check for it in the cache again.  Note that your browser_cache_path
## setting *must* use your default browser for this to work.
#open_pages_in_browser:false

## Pre-existing ffnet INI settings you might want to think about
## changing with use_browser_cache:

## If you want covers to work correctly with use_browser_cache, you'll
## need to click the cover image at least once for each story.  Or
## turn cover off.
never_make_cover:true

## If you want skip_author_cover to work correctly with
## use_browser_cache, you'll need to click the author page at least
## once for each story.  Or turn skip_author_cover off.
##
## From defaults.ini:
## April 2015, ffnet changed their story and user pictures urls to be
## the same.  Now the only way to know if the story image is really
## the author image is to go get the author image and check.
skip_author_cover:false

## If you turned this on before, you probably want it off while using
## browser cache.
##
## From defaults.ini:
## ffnet used to have a tendency to send out update notices in email
## before all their servers were showing the update on the first
## chapter.  It generates another server request and doesn't seem to
## be needed lately, so now default it to off.
check_next_chapter:false

## ffnet sometimes doesn't update the metadata on the first chapter
## immediately.  meta_from_last_chapter:true will instruct the adapter
## to take the metadata from the last chapter instead of chapter 1.
## It can generate another server request and usually isn't necessary,
## so it defaults to off.
meta_from_last_chapter:false

WebToEpub

The WebToEpub (Chrome | FireFox) browser add-on is an excellent tool and can be used to download epubs from several sites, albeit without FanFicFare's focus on parsing metadata and Calibre integration.

WebToEpub can also be used to spin through the chapters for you to load them into the browser can while it generates it's epub (the button is 'Pack EPUB').

If WebToEpub gets 'stuck' while downloading chapters, you can manually click through few fanfiction.net pages in another tab (likely seeing a Captcha) and that will frequently allow it to continue.

Note that WebToEpub is known to crash the Vivaldi browser. If this happens to you, consider using an alternative browser.

FanFictionDownloader

FanFictionDownloader is a long standing tool for downloading fanfiction stories for different sites, including fanfiction.net and fictionpress.com, that runs on Windows, Mac and Linux (I couldn't get 0.9.4 to run on Ubuntu 21.04 or 20.04, but I honestly didn't try very hard).

As of FanFicFare v4.20, the Browser Cache feature does not work with FanFictionDownloader and fanfiction.net anymore. FanFictionDownloader downloads chapters by .../99999/1/, .../99999/2/ etc. FanFicFare now downloads ffnet chapters by .../99999/1/Story-Title, .../99999/2/Story-Title which is how ffnet themselves link to chapters.

FYI, FanFicFare used to also be named FanFictionDownLoader or fanficdownloader before being renamed. FanFictionDownloader.net had that name first. There's never been any relation but similarity of purpose between the two.

VPN

Some VPNs prevent FFF's Browser Cache Feature from working. Rather than operating at the network level, they operate as a proxy, changing all the cached URLs unrecognizably. Try with no VPN running.

Websites known not to work

If If there is a no-cache or max-age=0 entry in the cache-control of the HTTP headers, caching is disabled for that website. This can be checked either through Inspect Element > Network in the browser or by using a third-party tool such as this one provided by DNSChecker.

Here are websites known to not work with the browser cache:

  • archiveofourown.org (and possibly other OTW archives such as Squidgeworld)
    • Note that after Archive of Our Own implemented Cloudflare in July 2023, there have been reports of it working. See #975 for more information.
  • ScribbleHub
  • ficbook.net (as of March 2023)
Add a custom footer
Add a custom sidebar
Clone this wiki locally