Getting Started

Ready to contribute? Here's how to set up scrapy-redis for local development.

Setup environment

1. Fork the scrapy-redis repo on GitHub.

2. Clone your fork locally:

git clone [email protected]:your_name_here/scrapy-redis.git

3. Install your local copy into a virtualenv. Assuming you have virtualenvwrapper installed, this is how you set up your fork for local development:

pip install virtualenv>=20.23 # tox require version >=20.23
virtualenv --python=/usr/bin/python3 ~/scrapy_redis
source ~/scrapy_redis/bin/activate
cd scrapy-redis/
pip install -r requirements-tests.txt
pip install .

4. Create a branch for local development:

git checkout -b name-of-your-bugfix-or-feature

Manage python version with pyenv

1. Install dependencies

sudo apt update && sudo install -y make build-essential libssl-dev zlib1g-dev libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm

2. Download pyenv from git to local

git clone https://github.com/pyenv/pyenv.git ~/.pyenv

3. Add PYENV_ROOT to path and initialized pyenv

echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.bashrc
echo 'export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.bashrc
echo -e 'if command -v pyenv 1>/dev/null 2>&1; then\n eval "$(pyenv init -)"\nfi' >> ~/.bashrc

4. Re-activate shell

exec "$SHELL"

5. Verify the installation

pyenv versions

And you're ready to go! Now you can make your changes locally.

Setup testing environment

1. When you're done making changes, check that your changes pass flake8, bandit, pylint and pytest, which can be tested using tox:

pip install -U tox
tox

2. Note that if the error of No module named scrapy_redis shows, please check the install scrapy-redis of your branch by:

pip install .

3. Or change the import lines:

from scrapy_redis import xxx # from this
from src.scrapy_redis import xxx # to this

4. Commit your changes and push your branch to GitHub:

git add .
git commit -m "Your detailed description of your changes."
git push origin name-of-your-bugfix-or-feature

5. Submit a pull request through the GitHub website.