Android AsyncStorage: "Error restoring state" for large datasets

[jamesisaac]

This library has been working perfectly for me with smaller size state trees, but trying to use it on bigger ones I'm running into these errors when relaunching the app:

10:39:16 redux-persist/getStoredState: Error restoring data for key: entities {}

10:39:18 Possible Unhandled Promise Rejection (id: 0):
Couldn't read row 0, col 0 from CursorWindow.  Make sure the Cursor is initialized correctly before accessing data from it.

Specifically, I'm trying to store a lot of data from a server locally. To give an idea of the size, running JSON.stringify(payload).length gives 2368916, so it looks like it should be within the 6MB limit.

I'm not having any of the performance issues described in #185 either - the app runs fairly smoothly (perhaps because writes are used sparingly). It's just a case of closing the app and reopening it leading to this error when it tries to rehydrate.

[rt2zz]

There is some additional discussion around android limits here: #199 (comment)

I do not have any solves off the top of my head, but I am open to ways we can be more resilient to storage limits.

[jamesisaac]

Hmm, I thought it might not be due to the size limit, as from my (limited) understanding, if JSON.stringify(payload).length gives 2368916, then the size being stored is about 2.4MB (1 byte per char?), so under Android's limit of 6MB?

But if this sounds exactly like the storage limit problem, no worries, I will look into some of the solutions specific to that limitation.

[jamesisaac]

Ahhh, I've just realised that it might be to do with the development platform I'm using (Exponent) running multiple apps from the same host app. So I guess that 6MB storage limit might be shared among all of their saved data. Will investigate further.

[rt2zz]

@jamesisaac interesting, that could definitely be related. Do you have control over expanding the limit? Or possibly using other native modules like https://github.com/sriraman/react-native-shared-preferences

[jorge627]

Hi
I have exactly the same problem, I'm trying to store a lot of data from a server locally in device.

redux-persist/getStoredState: Error restoring data for key: pos Error: Couldn't read row 0, col 0 from CursorWindow. Make sure the Cursor is initialized correctly before accessing data from it.

Couldn't read row 0, col 0 from CursorWindow. Make sure the Cursor is initialized correctly before accessing data from it.
Error: Couldn't read row 0, col 0 from CursorWindow. Make sure the Cursor is initialized correctly before accessing data from it.

I've tried things like this in MainApplication.java - onCreate method:
long size = 50L * 1024L * 1024L; // 50 MB
com.facebook.react.modules.storage.ReactDatabaseSupplier.getInstance(getApplicationContext()).setMaximumSize(size)

but it doesn't work for me

@jamesisaac could you solve it ??

thanks in advance

[kennethpdev]

I have the same problem like @jorge627 anyone?

[johnwayner]

I'm having a similar issue. I suspect we are hitting the max sqlite cursor window size on Android which is 2MB: http://stackoverflow.com/questions/21432556/android-java-lang-illegalstateexception-couldnt-read-row-0-col-0-from-cursorw

[kennethpdev]

@johnwayner so no way around this rather than reducing the size/data of a key or the reducer?

[johnwayner]

@kenma9123 That's the assumption I'm working under. I have a data dump from a user that clearly demonstrates that the limit ( at least on their device) is 2MB.

I'll be adding code to delete old data. If that doesn't keep key data under the limit, then I'll be splitting up my larger keys. It will result in more cross talk between reducers, which is unfortunate.

[robwalkerco]

@kenma9123 I've worked around this issue by creating my own storage for Android that uses the filesystem rather that AsyncStorage.

Hopefully that will help others with this same issue.
Perhaps there would be a way to integrate the storage implementation back into redux-persist so as to provide a more robust React Native Android storage solution?

Expand to see the code

/**
* @flow
*/

import RNFetchBlob from 'react-native-fetch-blob'

const DocumentDir = RNFetchBlob.fs.dirs.DocumentDir
const storagePath = `${DocumentDir}/persistStore`
const encoding = 'utf8'

const toFileName = (name: string) => name.split(':').join('-')
const fromFileName = (name: string) => name.split('-').join(':')

const pathForKey = (key: string) => `${storagePath}/${toFileName(key)}`

const AndroidFileStorage = {
  setItem: (
    key: string,
    value: string,
    callback?: ?(error: ?Error) => void,
  ) =>
    new Promise((resolve, reject) =>
      RNFetchBlob.fs.writeFile(pathForKey(key), value, encoding)
        .then(() => {
          if (callback) {
            callback()
          }
          resolve()
        })
        .catch(error => {
          if (callback) {
            callback(error && error)
          }
          reject(error)
        })
  ),
  getItem: (
    key: string,
    callback?: ?(error: ?Error, result: ?string) => void
  ) =>
    new Promise((resolve, reject) =>
      RNFetchBlob.fs.readFile(pathForKey(toFileName(key)), encoding)
        .then(data => {
          if (callback) {
            callback(null, data)
          }
          resolve(data)
        })
        .catch(error => {
          if (callback) {
            callback(error)
          }
          reject(error)
        })
  ),
  removeItem: (
    key: string,
    callback?: ?(error: ?Error) => void,
  ) =>
    new Promise((resolve, reject) =>
      RNFetchBlob.fs.unlink(pathForKey(toFileName(key)))
        .then(() => {
          if (callback) {
            callback()
          }
          resolve()
        })
        .catch(error => {
          if (callback) {
            callback(error)
          }
          reject(error)
        })
  ),
  getAllKeys: (
    callback?: ?(error: ?Error, keys: ?Array<string>) => void,
  ) =>
    new Promise((resolve, reject) =>
      RNFetchBlob.fs.exists(storagePath)
      .then(exists =>
        exists ? Promise.resolve() : RNFetchBlob.fs.mkdir(storagePath)
      )
      .then(() =>
        RNFetchBlob.fs.ls(storagePath)
          .then(files => files.map(file => fromFileName(file)))
          .then(files => {
            if (callback) {
              callback(null, files)
            }
            resolve(files)
          })
      )
      .catch(error => {
        if (callback) {
          callback(error)
        }
        reject(error)
      })
  ),
}

export default AndroidFileStorage

[kennethpdev]

@robwalkerco wow great idea. I tried it and it works fine. You should create a repo for this and let it grow.

[robwalkerco]

@kenma9123 I've created a project on npm with a cleaned up version of the above code. Check out https://www.npmjs.com/package/redux-persist-filesystem-storage

[kelset]

I just had the same issue and @robwalkerco 's lib fixed it - thanks a lot man!

(btw I think this issue can be closed..?)

[rahulbhankar786]

@robwalkerco Thanks you saved my day.

[shehzadosama]

For me, this solution worked.
Add this piece of code at the end of onCreate() in MainApplication.java

try {
  Field field = CursorWindow.class.getDeclaredField("sCursorWindowSize");
  field.setAccessible(true);
  field.set(null, 100 * 1024 * 1024); //100MB
  } catch (Exception e) {
 if (BuildConfig.DEBUG) {
  e.printStackTrace();
  }
  }

Also import these on top of the MainApplication.java

import android.database.CursorWindow;
import java.lang.reflect.Field;

[infy11]

For me, this solution worked. Add this piece of code at the end of onCreate() in MainApplication.java

try {
  Field field = CursorWindow.class.getDeclaredField("sCursorWindowSize");
  field.setAccessible(true);
  field.set(null, 100 * 1024 * 1024); //100MB
  } catch (Exception e) {
 if (BuildConfig.DEBUG) {
  e.printStackTrace();
  }
  }

Also import these on top of the MainApplication.java

import android.database.CursorWindow;
import java.lang.reflect.Field;

@shehzadosama thanks for the solution, however this will lead to out of memory crashes since everytime you try to read something from sqlite, it will try to allocate the cursor of size 100MB