Flask server/client application with RQ (Redis Queue) for asynchronous polling of a Cloud pull-subscription
This repository contains source code for a Flask application that may be used to poll and consume messages, published with Google's Cloud Pub/Sub. To create a working example messages published from Cloud Build service have been used, but this is only for the proof-of-concept, as similar procedure could be used for other Pub/Sub topics and subscriptions. Heroku dynos have been used as a remote-staging environment, utilizing the free-tier, with one web dyno and one worker dyno, allowing hosting of the application together with a single rq
worker.
The application can listen for new messages on a subscription created in a GCP project. When a new message is received it is stored in the database as a JSON string and also displayed in the current user's view (client side). Bootstrap (Flask-Bootstrap), JavaScript, JQuery and plugins (Infinite-Scroll) have been used on the client-side to make the user experience more interesting. On the server-side, SQLite has been used for the local development, while PostgreSQL and Gunicorn for Heroku deployment. RQ was chosen instead of Celery as a lighter solution, but one could similarly use Celery instead of RQ. There is also another task-queue, psq
, which is a Cloud Pub/Sub Task Queue for Python, developed by Google for this purpose. This is not an official product and is still in development, so RQ was chosen instead of psq
.
The project has been developed in a Python virtualenv
created and managed using pipenv
. It also adds/removes packages from your Pipfile
as you install/uninstall packages and generates the Pipfile.lock, which is used to produce deterministic builds. You will need pipenv
to be able to quickly and reliably reproduce a working environment to run the app locally. If you are unfamiliar with pipenv
have a look at a few quick tips to get you started.
You need to have a working GCP project, setup with Pub/Sub API, and a service account to get your GOOGLE_APPLICATION_CREDENTIALS
key file. You then need to create a topic and a subscription.
Clone the repo and then cd
in the project's root directory (e.g. gcpubsub
) then create virtualenv
and install dependencies:
git clone https://github.com/chrisliatas/pubsub-monitor.git gcpubsub
cd gcpubsub
pipenv install
This will install all project dependencies. Please note that the project is built using Python 3.7. When installing of all requirements is finished you will have a new virtualenv
for the application that you can work-on by running:
pipenv shell
You will need to have your new virtualenv
activated. You then have to:
export FLASK_APP=gcpubsub.py
export GOOGLE_APPLICATION_CREDENTIALS=path_to_key_file.json
export GCP_PROJECT_ID=your_GCP_project_name
export PUBSUB_TOPIC=your_pubsub_topic_name
export SUBSCRIPTION_NAME=your_subscription_name
Start a separate terminal in the same directory, activate your virtualenv
(pipenv shell
) and run:
rq worker gcpubsub-tasks
# You will get an output similar to:
09:16:13 RQ worker 'rq:worker:host-01.30454' started, version 0.12.0
09:16:13 *** Listening on gcpubsub-tasks...
09:16:13 Cleaning registries for queue: gcpubsub-tasks
On your first terminal run:
flask deploy
flask run
You should be able now, to view the application at http://localhost:5000
. The application has a default user admin
with password cannotguess
, created by flask deploy
command, that you can use for your initial login. You only need to run flask deploy
for your first run. The default password can be changed after you login, from within the web interface.
Clicking on Start polling
will initiate the background task that listens for messages in your subscription. If you have any messages, these will be loaded and displayed as soon as they are received. You can use the publish.py
under the directory helpers
if you need to create and publish sample messages to test the app.
If you would like to know more about the app, usage considerations, Heroku deployment and alternatives, have a look at the post:
Flask + RQ: Async polling a Cloud subscription
This app has been developed utilizing the following open source projects and libraries:
Dependency | License |
---|---|
github.com/metafizzy/infinite-scroll | GNU GPL license v3 |
github.com/miguelgrinberg/microblog | MIT License |
This work is released under the Apache License, Version 2.0.