[Feature] Support IndexedDB for shared auth use cases

[vthommeret]

Some services (notably Firebase Auth) use IndexedDB to store authentication state. The current API's for reusing authentication state only support cookies and local storage: https://playwright.dev/docs/auth#reuse-authentication-state

While it is possible to write IndexedDB code to manually read authentication state, serialize it, store it in an environment variable, and then add it back to a page, it requires around 150 lines of code due to the verbosity of IndexedDB's API. See https://github.com/microsoft/playwright/discussions/10715#discussioncomment-1904812.

Some possibly nice capabilities:

1. Save and restore all IndexedDB databases (similar to this suggestion What's the best way to run tests in parallel using same persistent context #7146 (comment))
2. Ability to specify a specific database and/or object store and/or object name to save and restore since IndexedDB's can be quite big.

This is also similar to the request to add support for session storage with a few people also expressing interested in IndexedDB support — #8874

cc @pavelfeldman

[awallace10]

Running into this issue now with a PWA app that stores its JWT token in indexedDB. Having to go way out of the way to extract it from the API response header, then send it on each call (not the best at all)

[Bessonov]

@awallace10 don't store jwt in any browser's local storage like localStorage, sessionStorage and even indexedDB or sql. Not only they are insecure, they aren't needed with authorization code flow.

[awallace10]

@awallace10 don't store jwt in any browser's local storage like localStorage, sessionStorage and even indexedDB or sql. Not only they are insecure, they aren't needed with authorization code flow.

I am just the QA Engineer and not the dev for the app. We are working on a different solution since this has come up, but we still need to access the indexeddb for other data that is being stored.

[MatthewSH]

@awallace10 don't store jwt in any browser's local storage like localStorage, sessionStorage and even indexedDB or sql. Not only they are insecure, they aren't needed with authorization code flow.

I can understand your hesitance to use IndexedDB for storing this kind of information. It could very well be used for XSS attacks and it's recommended to store it with an httpOnly cookie. However, use cases are different. PWA can throw a wrench in there since, AFAIK, SWs and WWs don't have access to those specific cookies (I could be wrong on this one, the documentation I've found is spotty with httpOnly cookies). Some people may need this and some use cases don't have access to a database to store these sessions in and memory storage isn't an option due to load balancing (or a multitude of other factors) and require the use of some form of stateless token. Every use case is different and every development team should find the pros and cons of it while also finding the best solution for them.

These points have been argued for a long time and there are so many different ways to do the same thing. However, all of this is irrelevant and just derailing the core issue which is that Playwright seemingly does not have the ability to easily access the IndexedDB and since you have to be on the page to access that domain's IndexedDB it can become challenging.

[Bessonov]

@MatthewSH It's off-topic. But this is exactly the reason behind so much insecure solutions and data breaches in the wild. Just because devs don't understand security, don't understand how it works and don't understand the specs like RFC6749, RFC7636, OIDC or even HttpOnly, SameSite, CSRF and so on, doesn't mean that there are "different use cases" or that it's reasonable (from security point of view) to have "so many" different use cases at all. Often devs don't even understand the implications of so called "stateless auth" you mentioned. Even worse, they dream that "stateless auth" is superior and more scalable than cookie/session-based solutions... Even usage of ready-to-go solutions like auth0 or keycloak doesn't necessary leads to secure solutions, if devs doesn't understand the whole picture. Because of that you can already see that OAuth2.1 has removed some insecure use cases, even they are still possible with mentioned solutions. I've conducted many interviews in the last 6 years and accompanied multiple security audits in the same time frame. Probably, only 10-20% of devs are aware of the problems at all and less than 3-5% I would trust implementing security related staff.

I agree that it isn't relevant to this feature request, but I can literaly see the next data breach.

[lubomirqa]

@awallace10 having the same issue. Were you able to perform UI testing with just a token in your case?

[odinho]

With the new setup feature of Playwright (different from global-setup), one which is really nice, the above Firebase login workaround won't work. Since it uses an environment variable to pass the login data around.

Did anyone already find a nicer way to do the shared auth with indexeddb using the 'depenency-based' setup?

[jenvareto]

I would really like to see support for IndexedDB. The application I work on uses Firebase authentication, and Firebase uses IndexedDB. Firebase is a popular platform used by a number of enterprise companies with logos on the front page at https://firebase.google.com/. Whether or not this is a good idea is irrelevant to Playwright; I and other test automation engineers needs to support this popular platform in our tests.

Support for Firebase was the # 1 concern I had for being able to switch from another UI automation platform to Playwright. I was able to write a solution for it, but if I hadn't then as much as I like this project it would not have been viable. I would really like to remove the parts of my code that handle injecting data into IndexedDB and let Playwright handle it through context instead.

How can the community help? I'm willing to document up use cases if you need additional data.

[jenvareto]

@odinho : I implemented it as follows:

In a project that's basically global setup:

1. Load the app's login page so that the Firebase IndexedDB database is created (preferred to initializing it in test code).
2. Get a token with username/password auth (https://identitytoolkit.googleapis.com/v1/accounts:signInWithPassword
3. Call /v1/accounts:lookup to get user info.
4. Construct the user info object that gets stored in IndexedDB. (you can see it in the dev tools)
5. Store that in a json file.

If you need to use Firebase service credentials to get the token you can do that instead of steps 1 and 2. You get the same info. You can do it with raw HTTP requests or use the Firebase admin plugin.

In a beforeEach hook:

1. Load the app's login page.
2. Load the user info object from file.
3. Inject the user info object as a record in the Firebase IndexedDB's object store using a JS function run in the browser with page.evaluate.

From here, use whatever URL you normally use as your entry point into the application.

I have the above defined in a project called 'setup' and set that as a dependency of my e2e tests.

I'll need to tweak this a bit to support multiple users, but so far so good.

[akratofil]

@odinho : I implemented it as follows:

In a project that's basically global setup:

1. Load the app's login page so that the Firebase IndexedDB database is created (preferred to initializing it in test code).

2. Get a token with username/password auth (https://identitytoolkit.googleapis.com/v1/accounts:signInWithPassword

3. Call /v1/accounts:lookup to get user info.

4. Construct the user info object that gets stored in IndexedDB. (you can see it in the dev tools)

5. Store that in a json file.

If you need to use Firebase service credentials to get the token you can do that instead of steps 1 and 2. You get the same info. You can do it with raw HTTP requests or use the Firebase admin plugin.

In a beforeEach hook:

1. Load the app's login page.

2. Load the user info object from file.

3. Inject the user info object as a record in the Firebase IndexedDB's object store using a JS function run in the browser with page.evaluate.

From here, use whatever URL you normally use as your entry point into the application.

I have the above defined in a project called 'setup' and set that as a dependency of my e2e tests.

I'll need to tweak this a bit to support multiple users, but so far so good.

Hi, could you please provide how exact did you add value to IndexedDB using page.evaluate ? I'm trying but it is not working for me. Thanks

[jenvareto]

Hi, could you please provide how exact did you add value to IndexedDB using page.evaluate ? I'm trying but it is not working for me. Thanks

@akratofil Here you go.

This is the method that handles IndexedDB. It's run in the browser using page.evaluate. The parameter userinfo is an object with the JSON blob that will be written as the key/value pair in the firebaseLocalStorage object store. This is the object that I cache to disk after the initial authentication.

/**
   * Sets the auth info in the browser's IndexedDB storage. Call this using page.evaluate.
   */
  async setAuthInBrowser({ userInfo }) {
    function insertUser(db, user) {
      const txn = db.transaction('firebaseLocalStorage', 'readwrite');
      const store = txn.objectStore('firebaseLocalStorage');
      const query = store.add(user);

      query.onsuccess = function (event) {
        console.log(event);
      };

      query.onerror = function (event) {
        console.log(event.target.errorCode);
      };

      txn.oncomplete = function () {
        db.close();
      };
    }

    const request = window.indexedDB.open('firebaseLocalStorageDb');
    request.onerror = (event) => {
      console.error(`Database error}`);
    };

    request.onsuccess = (event) => {
      const db = request.result;

      insertUser(db, userInfo);
    };
  }

This is the method that calls it. It pulls the cached user credentials from disk, goes to the login page, then executes page.evaluate.

/**
   * Logs in the user with cached storage credentials
   */
  async loginUser(email = process.env.USER_EMAIL) {
    await test.step('Log in with cached credentials', async () => {
      const firebase = new FirebaseAuth();
      const userInfo = await firebase.getUserInfo(email);

      await this.goto();
      await this.page.evaluate(firebase.setAuthInBrowser, { userInfo });
    });
  }

[tonystrawberry]

Hello!
@akratofil @odinho @lubomirqa
In case you haven't solved your problem yet, I managed to make use of shared authentication (Firebase) for my test cases using the following steps:

• Create a auth.setup.ts that will be run before all the tests.
  • It will login the user (in my case I login via email/password combination provided in the URL).
  • After login, Firebase saves the data in IndexedDB so I copy all the data to local storage and finally call page.context().storageState to save the authentication data in localStorage into playwright/.auth/user.json.

import { test as setup } from '@playwright/test';

const authFile = 'playwright/.auth/user.json';

setup('authenticate', async ({ page }) => {
  // Perform authentication steps
  // Start from the index page with the e2eToken query parameter
  // That will automatically log in the user
  await page.goto(`/[email protected]&password=${process.env.E2E_FIREBASE_USER_PASSWORD}`);

  // Wait until the page redirects to the cards page and stores the authentication data in the browser
  await page.waitForURL('/cards');

  // Copy the data in indexedDB to the local storage
  await page.evaluate(() => {
    // Open the IndexedDB database
    const indexedDB = window.indexedDB;
    const request = indexedDB.open('firebaseLocalStorageDb');

    request.onsuccess = function (event: any) {
      const db = event.target.result;

      // Open a transaction to access the firebaseLocalStorage object store
      const transaction = db.transaction(['firebaseLocalStorage'], 'readonly');
      const objectStore = transaction.objectStore('firebaseLocalStorage');

      // Get all keys and values from the object store
      const getAllKeysRequest = objectStore.getAllKeys();
      const getAllValuesRequest = objectStore.getAll();

      getAllKeysRequest.onsuccess = function (event: any) {
        const keys = event.target.result;

        getAllValuesRequest.onsuccess = function (event: any) {
          const values = event.target.result;

          // Copy keys and values to localStorage
          for (let i = 0; i < keys.length; i++) {
            const key = keys[i];
            const value = values[i];
            localStorage.setItem(key, JSON.stringify(value));
          }
        }
      }
    }

    request.onerror = function (event: any) {
      console.error('Error opening IndexedDB database:', event.target.error);
    }
  });

  await page.context().storageState({ path: authFile });
});

• In my playwright.config.ts, I add setup as a dependency inside projects.

const config: PlaywrightTestConfig = {
  ...

  projects: [
    {
      name: 'setup',
      testMatch: /.*\.setup\.ts/,
    },
    {
      name: 'Desktop Chrome',
      use: {
        ...devices['Desktop Chrome'],
      },
      dependencies: ['setup'],
    }
  ],
}

• I then create a utility function that will get the contents of playwright/.auth/user.json and copy it into IndexedDB again so that Firebase can authenticate the user in my tests.

import { Page } from "@playwright/test";

export const authenticate = async (page: Page) => {
  // Start from the index page
  await page.goto(`/`);

  // Get the authentication data from the `playwright/.auth/user.json` file (using readFileSync)
  const auth = JSON.parse(require('fs').readFileSync('playwright/.auth/user.json', 'utf8'));

  // Set the authentication data in the indexedDB of the page to authenticate the user
  await page.evaluate(auth => {
    // Open the IndexedDB database
    const indexedDB = window.indexedDB;
    const request = indexedDB.open('firebaseLocalStorageDb');

    request.onsuccess = function (event: any) {
      const db = event.target.result;

      // Start a transaction to access the object store (firebaseLocalStorage)
      const transaction = db.transaction(['firebaseLocalStorage'], 'readwrite');
      const objectStore = transaction.objectStore('firebaseLocalStorage', { keyPath: 'fbase_key' });

      // Loop through the localStorage data inside the `playwright/.auth/user.json` and add it to the object store
      const localStorage = auth.origins[0].localStorage;

      for (const element of localStorage) {
        const value = element.value;

        objectStore.put(JSON.parse(value));
      }
    }
  }, auth)
}

• Finally, I call the authenticate() function in the beforeEach of my tests.

import { authenticate } from "./utils/authenticate";

test.beforeEach(async ({ page }) => {
  await authenticate(page);
});

It works well for me here: https://github.com/tonystrawberry/matomeishi-next.jp
If you have any improvements suggestions, I am all ears (just started playing around with Playwright two days ago so my code may be weird or unnecessary complicated)!

[cduran-seeri]

I hope Playwright support Firebase Auth out of the box soon. By the moment I'm using @tonystrawberry 's solution, thank you.

[Fredx87]

I found an easier way to use firebase auth with Playwright, which is just to instruct firebase to use local storage when running in playwright:

import React from "react";
import ReactDOM from "react-dom/client";
import App from "./App";
import { auth } from "./auth";
import { browserLocalPersistence, setPersistence } from "firebase/auth";

const setupAuthPromise = () =>
  import.meta.env.MODE === "e2e"
    ? setPersistence(auth, browserLocalPersistence)
    : Promise.resolve();

setupAuthPromise().then(() => {
  ReactDOM.createRoot(document.getElementById("root") as HTMLElement).render(
    <React.StrictMode>
      <App />
    </React.StrictMode>,
  );
});

I am using Vite and running the app with --mode e2e for using it with Playwright, but it is possible to use any other method to detect if the page is running inside Playwright.

[jenvareto]

Unfortunately, Google deprecated using local storage to hold auth information. Just something to keep in mind in case unexpected problems occur.

[dora-gt]

We really need this.

[rahulrakesh16]

By when this feature will be released?

[GorvGoyl]

In my project, the Firebase login process takes more than a minute when using playwright. I tried using @tonystrawberry's approach to use IndexedDB instead, but it still takes the same amount of time. Is anyone else facing this issue? It would be great if someone knows a solution to this.

[OVO-Josh]

I've been using a hybrid of @jenvareto and @tonystrawberry 's answers to get things working for my tests with:

• auth.utils.ts

export const authenticate = async (page: Page) => {
  // Start from the index page
  await page.goto('http://localhost:3000/');
  const userInfo = JSON.parse(
    fs.readFileSync('playwright/.auth/firebase.json', 'utf8')
  );
  // Set the authentication data in the indexedDB of the page to authenticate the user
  await page.evaluate((userInfo) => {
    function insertUser(db, user) {
      const txn = db.transaction('firebaseLocalStorage', 'readwrite');
      const store = txn.objectStore('firebaseLocalStorage');
      store.add(user);
      txn.oncomplete = function () {
        db.close();
      };
    }
    const request = window.indexedDB.open('firebaseLocalStorageDb');
    request.onsuccess = () => {
      const db = request.result;
      insertUser(db, userInfo);
    };
  }, userInfo);
};

• auth.setup.ts

const firebaseConfig = {
 YOUR CONFIG HERE
};

initializeApp(firebaseConfig);
const auth = getAuth();

const URL_LOCAL = 'http://localhost:3000';

//see here: https://github.com/microsoft/playwright/issues/11164
//for why we have to do this weird hack to get auth state
setup('authenticate', async ({ page }) => {
  await page.goto(URL_LOCAL);

  const config = JSON.parse(
    fs.readFileSync('config/firebaseSecrets.json', 'utf-8')
  );

  const response = await page.request.post(
    'https://www.googleapis.com/oauth2/v4/token',
    {
      data: {
        grant_type: 'refresh_token',
        client_id: config['google_client_id'],
        client_secret: config['google_client_secret'],
        refresh_token: config['google_refresh_token'],
      },
    }
  );

  const body = await response.json();

  const { id_token } = body;
  const credential = GoogleAuthProvider.credential(id_token);

  const userCredential = await signInWithCredential(auth, credential);

  const user = userCredential.user;

  const firebaseDB = {
    fbase_key: `firebase:authUser:${auth.config.apiKey}:${auth.app.name}`,
    value: {
      apiKey: auth.config.apiKey,
      appName: auth.app.name,
      createdAt: user.metadata.creationTime,
      displayName: user.displayName,
      email: user.email,
      emailVerified: user.emailVerified,
      isAnonymous: user.isAnonymous,
      lastLoginAt: user.metadata.lastSignInTime,
      phoneNumber: user.phoneNumber,
      photoURL: user.photoURL,
      providerData: user.providerData,
      stsTokenManager: user.toJSON()['stsTokenManager'],
      tenantId: user.tenantId,
      uid: user.uid,
      refreshToken: user.refreshToken,
    },
  };

  const data = JSON.stringify(firebaseDB);

  fs.writeFile('playwright/.auth/firebase.json', data, (err) => {
    if (err) throw err;
  });
});

• in my spec files

test.beforeEach(async ({ page }) => {
  await authenticate(page);
});

• in playwright.config.ts

projects: [
    { name: 'setup', testMatch: /.*\.setup\.ts/ },
    {
      name: 'chromium',
      use: {
        ...devices['Desktop Chrome'],
        storageState: 'playwright/.auth/user.json',
      },
      dependencies: ['setup'],
    },
]

This has been working great for my UI tests, but it unfortunately seems to fail when trying to run them with the CLI, with the page.evaluate() seemingly being skipped. Anyone else had this? I can't seem to find a reason for why this would happen, and it seems like a pretty good workaround otherwise

[OVO-Josh]

OK I figured it out so leaving the answer here in case it helps anyone in the future. The issue was due to indexedDB operations being asynchronous, but them not being waited for in the above answer. Changing the page.evaluate to the below has done the trick:

await page.evaluate(async (userInfo) => {
    function insertUser(db, user, resolve, reject) {
      const txn = db.transaction('firebaseLocalStorage', 'readwrite');
      const store = txn.objectStore('firebaseLocalStorage');
      store.add(user);

      txn.oncomplete = function () {
        db.close();
        resolve();
      };

      txn.onerror = function () {
        db.close();
        reject();
      };
    }

    return new Promise(function (resolve, reject) {
      const request = window.indexedDB.open('firebaseLocalStorageDb');
      request.onsuccess = () => {
        const db = request.result;
        insertUser(db, userInfo, resolve, reject);
      };
      request.onerror = () => {
        reject(request);
      };
    });
  }, userInfo);

[jslegers]

1. Use case

I'm trying to extract all content I produced @ https://legacy.mage.space/u/johnslegers before disappears for good in less than 10 days.

Since downloading 14K images with corresponding prompts is quite insane, I figured I'd write a crawler for it instead.

---

2. Remarks related to this issue

It took me a while to get this right, since (1) I'm using the Python version of Playwright (2) in combination with Crawlee & (3) I first needed to remove an existing key from the Firebase DB before I could replace it with a new key.

I'll probably create a demo repo of my finished project in the very near future after I cleaned up my code, but for the time being here's some snippets with code that allowed me to get me to correctly log in on Chromium.

---

3. Snippets

3.1 Dump Firebase use to Json

First, I extracted my Firebase user by copy-pasting the following script in the console of the website I'm trying to scrape :

// Source https://gist.github.com/Matt-Jensen/d7c52c51b2a2ac7af7e0f7f1c31ef31d

(() => {
    const asyncForEach = (array, callback, done) => {
        const runAndWait = i => {
            if (i === array.length) return done();
            return callback(array[i], () => runAndWait(i + 1));
        };
        return runAndWait(0);
    };

    const dump = {};
    const dbRequest = window.indexedDB.open("firebaseLocalStorageDb");
    dbRequest.onsuccess = () => {
        const db = dbRequest.result;
        const stores = ['firebaseLocalStorage'];

        const tx = db.transaction(stores);
        asyncForEach(
            stores,
            (store, next) => {
                const req = tx.objectStore(store).getAll();
                req.onsuccess = () => {
                    dump[store] = req.result;
                    next();
                };
            },
            () => {
                console.log(JSON.stringify(dump));
            }
        );
    };
})();

3.2 Adding the correct user info after first removing the wrong info

This is the Javascript code that's injected when the page is loaded in my request handler in __main__.py . For the time being it's just a test string in __main__.py, but it will be moved into a separate .js file with the code & a .json file with the login data.

It's an adapation of the code from the previous comment by from OVO-Josh.

// Adaptation of the code by OVO-Josh

(function adduser() {
    function insertUser(db, user) {
        const txn = db.transaction('firebaseLocalStorage', 'readwrite');
        const store = txn.objectStore('firebaseLocalStorage');
        store.delete(user.fbase_key);
        store.add(user);
        txn.oncomplete = function(ev) {
            db.close();
        };
        txn.onerror = function(ev) {
            console.error(ev.target.error.message)
            db.close();
        };
        return txn;
    }
    const request = window.indexedDB.open('firebaseLocalStorageDb');
    request.onfailure = function(ev) {
        console.error(ev.target.error.message);
    };
    request.onsuccess = function(ev) {
        const db = request.result;
        return insertUser(db, {
            "fbase_key": "firebase:authUser:_____:[DEFAULT]",
            "value": {
                "uid": _____,
                "email": _____,
                "emailVerified": true,
                "displayName": _____,
                "isAnonymous": false,
                "photoURL": _____,
                "providerData": [{
                        "providerId": "google.com",
                        "uid": _____,
                        "displayName": _____,
                        "email": _____,
                        "phoneNumber": null,
                        "photoURL": _____
                    },
                    {
                        "providerId": "password",
                        "uid": _____,
                        "displayName": _____,
                        "email": _____,
                        "phoneNumber": null,
                        "photoURL": _____
                    }
                ],
                "stsTokenManager": {
                    "refreshToken": _____,
                    "accessToken": _____,
                    "expirationTime": _____
                },
                "createdAt": _____,
                "lastLoginAt": _____,
                "apiKey": _____,
                "appName": "[DEFAULT]"
            }
        });
    };
    return request;
})()

3.2 Load page from Python

Here's my Python request handler in __main__.py, where I actually add the JS inject, after first waiting until the DOM has loaded :

async def request_handler(context: PlaywrightCrawlingContext) -> None:
    context.log.info(f"Processing {context.request.url} ...")
    page = context.page

    # Wait until the content I'm interested in is loaded
    await page.wait_for_selector(selector)
    # Update the users in the Firebase DB with the correct value
    # add_user is the above JS code that's injected
    await page.evaluate(add_user)

        _____

I created a request for this same feature @ #32300

[manodupont]

@tonystrawberry I followed your suggestion. I can see everything works fine. But sometimes it happens that the state.json file that is output, doesn't contain the firebase key, like the firebaseDB is not yet initialized.

Do you have any recommendations into making sure that when the page.evaluate happens, the firebaseDB is really there and containing stuff ?

Thank you

[vrknetha]

I create a library to use indexeddb operations.

https://github.com/vrknetha/playwright-indexeddb