Skip to content

Latest commit

 

History

History
94 lines (67 loc) · 6.24 KB

DEVELOPMENT.org

File metadata and controls

94 lines (67 loc) · 6.24 KB
Raw

Use Github actions

The main source of truth is the Github Actions CI pipeline. There is also a CircleCI pipeline, but it only tests systemd integration.

If you fork the repository, go to “Actions” tab on your fork and click “I understand my workflows, go ahead and enable them” (like on this screen). This will run CI for your commits/branches even before you open the PR.

If you have any issues, looking at what github actions do might be the quickest way to investigate them. In particular, you can try uncommenting the tmate SSH helper and pushing this change (as a separate commit). This will allow you to SSH directly onto the CI server and run any commands to investigate/reproduce tests. See tmate docs for more info.

If you open a PR and some checks fail, please try to fix them if possible, but if it’s unclear to you why a partucular check failing, don’t hesitate to ask!

Working on backend (promnesia indexer/server)

use editable install

  • during development, the easiest way to run promnesia as scripts/promnesia. Alternatively, you can use pip install -e ..
  • similarly, if you’re working on a module which uses HPI, it’s much easier to install it in editable mode Also see HPI development docs.

running tests & mypy checks

First, check out all submodules: git submodule update --init --recursive. This will fetch some input data only used by tests.

To actually run checks there are several options. Some examples:

  • tox: would just run all of them (takes a while) (tox is a tool to run tests in isolated Python environments, google it for moe info if you’re curious)
  • tox -e mypy-core: will run only mypy-core checks
  • tox -e tests -- -k test_search: only run tests that contain test_search substring (everything after -- passed down to pytest)
  • pytest -s tests/config_tests.py: run just one file via pytest
  • mypy -p promnesia.sources.hypothesis: only checks this module via mypy

You might want to consult tox.ini, pytest.ini and mypy.ini.

When you run tests, some tests will be marked as yellow and as SKIPPED. There is no need to worry about it – by default some tests aren’t running because they are using GUI (so only run manually), or work in progress, or something else. As long as github actions are happy it generally should be fine.

structure of tests

  • tests/indexer_test.py – usually tests sources and how they extract visits (without creating the database)
  • tests/integration_test.py – usually tests indexing, and does checks against actual database
  • tests/server_test.py – usually tests server responses
  • tests/cli.py – tests everything, starting from CLI interation
  • tests/end2end_test.py – tests everything, including the extension! These don’t run by default because they are using GUI and the setup is quite elaborate..
  • tests/demos.py – extra end-to-end tests, also used for recording Promnesia feature demo videos.

I say ‘usually’ because it’s a little messy and the separation is not 100% clear at times. For example, most tests in server_test also run indexing and extraction, so they aren’t very ‘unit’ tests. I’m not against it, but with limited time resources full-scale tests give better coverage (at the expense of more confusion when they fail).

Build extension

cd extension
./build --firefox # you can also use --chrome
        --lint    # [optional], run webext linter
        --release # [optional], build in the release mode (with optimizations)
        --publish # [optional], release to the Chrome Web Store/Mozilla addons

You’ll find the result in dist/. After that, you can load it in your browser and develop.

  • on Firefox, the temporary loaded extensions only persist until the browser restart. Chrome doesn’t have that issue.
  • on Firefox for Android, web extensions on Android are mostly broken at the moment, see here (unless you’re using Firefox Nightly)

Run end-to-end tests

End-to-end test go thought the full cycle from indexing & backend to automated extension tests, and are very helpful to make sure you don’t break things.

You need:

  • firefox + geckodriver, or chrome + chromedriver
  • pip3 install pyautogui for keyboard automation

Example of running a test:

PYTHONPATH=src python3 -m pytest -s 'tests/end2end_test.py::test_add_to_blacklist[chrome]'

Releasing

AMO (addons.mozilla.org)

cd extension
./build --firefox --lint --release --publish

Note that this will only upload the extension (see web-ext sign command reference and web-ext-submit for more info). You need go to https://addons.mozilla.org/en-GB/developers/addon/promnesia/versions, there you should see your extension as ‘pending approval’. Usually it’s approved within minutes.

Sometimes it may be flagged for a review, so you’ll need to upload source code. You can use git archive master --output=promnesia-source.zip if you’re asked for it.

CWS (chrome web store)

cd extension
./build --chrome  --lint --release --publish

This command should work in theory and upload automatically… however getting the api keys is super annoying (see this).

Also it might still fail anyway, and require you to justify permissions etc.. If you only release the extension once in a while maybe simpler to upload the zip manually 🤷 Usually review in CWS takes a couple of days.