Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Feature: custom outputs #315

Open
wants to merge 82 commits into
base: master
Choose a base branch
from
Open

Feature: custom outputs #315

wants to merge 82 commits into from

Commits on Dec 4, 2017

  1. Temp version change

    @showerst
    showerst committed Dec 4, 2017
    Configuration menu
    Copy the full SHA
    4b7ae82 View commit details
    Browse the repository at this point in the history

  2. Add GCPS settings

    @showerst
    showerst committed Dec 4, 2017
    Configuration menu
    Copy the full SHA
    8d5a2d5 View commit details
    Browse the repository at this point in the history

  3. pluggable backends initial commit

    @showerst
    showerst committed Dec 4, 2017
    Configuration menu
    Copy the full SHA
    fd267ae View commit details
    Browse the repository at this point in the history

  4. move settings from cmd line to env

    @showerst
    showerst committed Dec 4, 2017
    Configuration menu
    Copy the full SHA
    7b38107 View commit details
    Browse the repository at this point in the history

Commits on Dec 5, 2017

  1. add google-cloud-pubsub install requirement

    Saxon Stiller committed Dec 5, 2017
    Configuration menu
    Copy the full SHA
    17d8a3a View commit details
    Browse the repository at this point in the history

  2. try setting up conditional import of module based on env var

    Saxon Stiller committed Dec 5, 2017
    Configuration menu
    Copy the full SHA
    8090d65 View commit details
    Browse the repository at this point in the history

  3. tiny var name change since not really dealing w/ class

    Saxon Stiller committed Dec 5, 2017
    Configuration menu
    Copy the full SHA
    5253ffc View commit details
    Browse the repository at this point in the history

  4. try to set up pub/sub for env var auth (since it will be difficult to… 

    … pass json key file into build)
    Saxon Stiller committed Dec 5, 2017
    Configuration menu
    Copy the full SHA
    6685f30 View commit details
    Browse the repository at this point in the history

  5. small style changes to abide by flake8

    Saxon Stiller committed Dec 5, 2017
    Configuration menu
    Copy the full SHA
    e2d2889 View commit details
    Browse the repository at this point in the history

  6. tiny styling change

    Saxon Stiller committed Dec 5, 2017
    Configuration menu
    Copy the full SHA
    c95963f View commit details
    Browse the repository at this point in the history

  7. add in obj pre_save method to match file writing save

    Saxon Stiller committed Dec 5, 2017
    Configuration menu
    Copy the full SHA
    48e215b View commit details
    Browse the repository at this point in the history

  8. change env var for output switch and remove from settings

    Saxon Stiller committed Dec 5, 2017
    Configuration menu
    Copy the full SHA
    cbf13f0 View commit details
    Browse the repository at this point in the history

  9. bunch o name changes

    Saxon Stiller committed Dec 5, 2017
    Configuration menu
    Copy the full SHA
    6d5e4a8 View commit details
    Browse the repository at this point in the history

  10. remove unused import_lib

    Saxon Stiller committed Dec 5, 2017
    Configuration menu
    Copy the full SHA
    8d5622b View commit details
    Browse the repository at this point in the history

  11. remove extra whitespace (causing diff change)

    Saxon Stiller committed Dec 5, 2017
    Configuration menu
    Copy the full SHA
    774a53b View commit details
    Browse the repository at this point in the history

  12. ditto

    Saxon Stiller committed Dec 5, 2017
    Configuration menu
    Copy the full SHA
    1a1a1da View commit details
    Browse the repository at this point in the history

  13. remove all just because hehe

    Saxon Stiller committed Dec 5, 2017
    Configuration menu
    Copy the full SHA
    9b8eca9 View commit details
    Browse the repository at this point in the history

  14. single quote consistency

    Saxon Stiller committed Dec 5, 2017
    Configuration menu
    Copy the full SHA
    2a6e749 View commit details
    Browse the repository at this point in the history

Commits on Dec 6, 2017

  1. add debuggering line

    Saxon Stiller committed Dec 6, 2017
    Configuration menu
    Copy the full SHA
    9e72ffb View commit details
    Browse the repository at this point in the history

  2. more debug printing

    Saxon Stiller committed Dec 6, 2017
    Configuration menu
    Copy the full SHA
    fd6eb6a View commit details
    Browse the repository at this point in the history

  3. remove debugging

    Saxon Stiller committed Dec 6, 2017
    Configuration menu
    Copy the full SHA
    9ca4c3d View commit details
    Browse the repository at this point in the history

  4. (try) add scopes and tiny flake8 patch

    Saxon Stiller committed Dec 6, 2017
    Configuration menu
    Copy the full SHA
    4a39f4a View commit details
    Browse the repository at this point in the history

  5. try another way for scopes

    Saxon Stiller committed Dec 6, 2017
    Configuration menu
    Copy the full SHA
    5c5c69a View commit details
    Browse the repository at this point in the history

  6. oops, forgot to specify keyword for arg

    Saxon Stiller committed Dec 6, 2017
    Configuration menu
    Copy the full SHA
    a5bc525 View commit details
    Browse the repository at this point in the history

  7. try a tuple instead?

    Saxon Stiller committed Dec 6, 2017
    Configuration menu
    Copy the full SHA
    0d26c22 View commit details
    Browse the repository at this point in the history

  8. meh, trying this, maybe i missed something

    Saxon Stiller committed Dec 6, 2017
    Configuration menu
    Copy the full SHA
    6e42604 View commit details
    Browse the repository at this point in the history

  9. try another way

    Saxon Stiller committed Dec 6, 2017
    Configuration menu
    Copy the full SHA
    ef688c3 View commit details
    Browse the repository at this point in the history

  10. patch validation to reference caller

    Saxon Stiller committed Dec 6, 2017
    Configuration menu
    Copy the full SHA
    dad6321 View commit details
    Browse the repository at this point in the history

  11. one last shot, for this nite my friend

    Saxon Stiller committed Dec 6, 2017
    Configuration menu
    Copy the full SHA
    5e02928 View commit details
    Browse the repository at this point in the history

  12. try again

    Saxon Stiller committed Dec 6, 2017
    Configuration menu
    Copy the full SHA
    d9362d1 View commit details
    Browse the repository at this point in the history

  13. trying json file loaded into single env var

    Saxon Stiller committed Dec 6, 2017
    Configuration menu
    Copy the full SHA
    5696342 View commit details
    Browse the repository at this point in the history

  14. debug step

    Saxon Stiller committed Dec 6, 2017
    Configuration menu
    Copy the full SHA
    951f92b View commit details
    Browse the repository at this point in the history

  15. undo debug step

    Saxon Stiller committed Dec 6, 2017
    Configuration menu
    Copy the full SHA
    ef28c48 View commit details
    Browse the repository at this point in the history

  16. add reference to issue on github that helped me so dearly

    Saxon Stiller committed Dec 6, 2017
    Configuration menu
    Copy the full SHA
    6564f03 View commit details
    Browse the repository at this point in the history

  17. remove kafka for now to make the diff/pr focused on googel cloude

    Saxon Stiller committed Dec 6, 2017
    Configuration menu
    Copy the full SHA
    4cbe0a3 View commit details
    Browse the repository at this point in the history

  18. small version change

    Saxon Stiller committed Dec 6, 2017
    Configuration menu
    Copy the full SHA
    68222b8 View commit details
    Browse the repository at this point in the history

Commits on Dec 7, 2017

  1. match env var used in google cloud documentation for the project id

    Saxon Stiller committed Dec 7, 2017
    Configuration menu
    Copy the full SHA
    3a4ef00 View commit details
    Browse the repository at this point in the history

  2. remove project_id var and just use env var during topic setup

    Saxon Stiller committed Dec 7, 2017
    Configuration menu
    Copy the full SHA
    8c5f9da View commit details
    Browse the repository at this point in the history

  3. ditto, remove topic var and just use env var during topic setup

    Saxon Stiller committed Dec 7, 2017
    Configuration menu
    Copy the full SHA
    db461d7 View commit details
    Browse the repository at this point in the history

  4. refactor google auth to allow for automatic detection of credentials,… 

    … in case user is hosting on gcp
    Saxon Stiller committed Dec 7, 2017
    Configuration menu
    Copy the full SHA
    4542caf View commit details
    Browse the repository at this point in the history

Commits on Dec 13, 2017

  1. add bill_identifier prop to vote_event for a more reliable way to ref… 

    …erence bill relationship
    Saxon Stiller committed Dec 13, 2017
    Configuration menu
    Copy the full SHA
    77d41ad View commit details
    Browse the repository at this point in the history

  2. nevermind re: new field; fallback to env var switch option

    Saxon Stiller committed Dec 13, 2017
    Configuration menu
    Copy the full SHA
    25b72ac View commit details
    Browse the repository at this point in the history

Commits on Dec 14, 2017

  1. remove some whitespace from stringified json sent to pub/sub and expl… 

    …icitly encode to utf-8
    Saxon Stiller committed Dec 14, 2017
    Configuration menu
    Copy the full SHA
    9a717bd View commit details
    Browse the repository at this point in the history

  2. use pub/sub topic_path publisher helper method to obtain topic path w… 

    …ithout hardcoding the topic string format
    Saxon Stiller committed Dec 14, 2017
    Configuration menu
    Copy the full SHA
    3509b2c View commit details
    Browse the repository at this point in the history

  3. remove automatic topic creation; assume user will create the topic pr… 

    …ior to using to avoid the extra request
    Saxon Stiller committed Dec 14, 2017
    Configuration menu
    Copy the full SHA
    7c5fe64 View commit details
    Browse the repository at this point in the history

  4. do not require ordering before publishing to pubsub

    Saxon Stiller committed Dec 14, 2017
    Configuration menu
    Copy the full SHA
    70ab25c View commit details
    Browse the repository at this point in the history

Commits on Dec 23, 2017

  1. update google pubsub dependency version

    doubleswirve committed Dec 23, 2017
    Configuration menu
    Copy the full SHA
    31a8b09 View commit details
    Browse the repository at this point in the history

Commits on Dec 27, 2017

  1. tiny update to service account env var name

    doubleswirve committed Dec 27, 2017
    Configuration menu
    Copy the full SHA
    8f30f76 View commit details
    Browse the repository at this point in the history

Commits on Jan 17, 2018

  1. merge upstream and settle conflicts

    doubleswirve committed Jan 17, 2018
    Configuration menu
    Copy the full SHA
    9276645 View commit details
    Browse the repository at this point in the history

Commits on Jan 18, 2018

  1. add base output class that wraps some before and after save handling

    doubleswirve committed Jan 18, 2018
    Configuration menu
    Copy the full SHA
    301dabe View commit details
    Browse the repository at this point in the history

  2. add local file output subclass to handle/replace default/existing scr… 

    …ape output to filesystem
    doubleswirve committed Jan 18, 2018
    Configuration menu
    Copy the full SHA
    7be4e6e View commit details
    Browse the repository at this point in the history

  3. remove save_object method from scraper class as it has been moved to … 

    …the base output class (w/ implementation left to subclasses) and add small helper method to get output target instance based on env vars
    doubleswirve committed Jan 18, 2018
    Configuration menu
    Copy the full SHA
    e7bf5fe View commit details
    Browse the repository at this point in the history

  4. update google cloud pubsub class to use new output base class

    doubleswirve committed Jan 18, 2018
    Configuration menu
    Copy the full SHA
    a77eb67 View commit details
    Browse the repository at this point in the history

Commits on Jan 19, 2018

  1. remove local file output init method

    doubleswirve committed Jan 19, 2018
    Configuration menu
    Copy the full SHA
    957a0bd View commit details
    Browse the repository at this point in the history

  2. Merge pull request opencivicdata#1 from doubleswirve/custom-export--d… 

    …ry-it-up

Custom export update
    Saxon Stiller authored Jan 19, 2018
    Configuration menu
    Copy the full SHA
    6f8f8a4 View commit details
    Browse the repository at this point in the history

  3. patch scraper reference prop in google cloud output

    doubleswirve committed Jan 19, 2018
    Configuration menu
    Copy the full SHA
    ce3456c View commit details
    Browse the repository at this point in the history

Commits on Jan 21, 2018

  1. add amazon sqs output

    doubleswirve committed Jan 21, 2018
    Configuration menu
    Copy the full SHA
    cd967b1 View commit details
    Browse the repository at this point in the history

  2. add amazon sqs output target to condition

    doubleswirve committed Jan 21, 2018
    Configuration menu
    Copy the full SHA
    757e907 View commit details
    Browse the repository at this point in the history

  3. add boto3 dep

    doubleswirve committed Jan 21, 2018
    Configuration menu
    Copy the full SHA
    99afa3e View commit details
    Browse the repository at this point in the history

  4. do not encode for aws sqs

    doubleswirve committed Jan 21, 2018
    Configuration menu
    Copy the full SHA
    337beab View commit details
    Browse the repository at this point in the history

  5. update pubsub filename

    doubleswirve committed Jan 21, 2018
    Configuration menu
    Copy the full SHA
    4165197 View commit details
    Browse the repository at this point in the history

  6. update initialization of publisher with new env var

    doubleswirve committed Jan 21, 2018
    Configuration menu
    Copy the full SHA
    f550a37 View commit details
    Browse the repository at this point in the history

Commits on Jan 22, 2018

  1. break some more helper methods out to the parent class and remove pub… 

    …date meta data from google pubsub as it is already part of the message object (during subscription)
    doubleswirve committed Jan 22, 2018
    Configuration menu
    Copy the full SHA
    c985014 View commit details
    Browse the repository at this point in the history

  2. try renewing the publisher client every publish...

    doubleswirve committed Jan 22, 2018
    Configuration menu
    Copy the full SHA
    4cf74ac View commit details
    Browse the repository at this point in the history

  3. patch order so var error does not occur

    doubleswirve committed Jan 22, 2018
    Configuration menu
    Copy the full SHA
    71dc7b7 View commit details
    Browse the repository at this point in the history

  4. smore patches

    doubleswirve committed Jan 22, 2018
    Configuration menu
    Copy the full SHA
    a01dc61 View commit details
    Browse the repository at this point in the history

  5. try to debug http version

    doubleswirve committed Jan 22, 2018
    Configuration menu
    Copy the full SHA
    2e55902 View commit details
    Browse the repository at this point in the history

  6. remove http version testing and revert to instantiating the publisher… 

    … client just once in __init__
    doubleswirve committed Jan 22, 2018
    Configuration menu
    Copy the full SHA
    1c832dd View commit details
    Browse the repository at this point in the history

  7. add debug helper method to parent class and update improts

    doubleswirve committed Jan 22, 2018
    Configuration menu
    Copy the full SHA
    289f9a0 View commit details
    Browse the repository at this point in the history

Commits on Jan 23, 2018

  1. try upgrading the grpc lib version

    doubleswirve committed Jan 23, 2018
    Configuration menu
    Copy the full SHA
    e2c950b View commit details
    Browse the repository at this point in the history

  2. remove grpc dep, no dice

    doubleswirve committed Jan 23, 2018
    Configuration menu
    Copy the full SHA
    c6e4d71 View commit details
    Browse the repository at this point in the history

  3. Merge pull request opencivicdata#2 from doubleswirve/custom-export--g… 

    …oogle-pubsub-env

Google Pub/Sub env var adjustments and helper methods
    Saxon Stiller authored Jan 23, 2018
    Configuration menu
    Copy the full SHA
    0adc9a5 View commit details
    Browse the repository at this point in the history

  4. Merge branch 'master' into custom-export

    doubleswirve committed Jan 23, 2018
    Configuration menu
    Copy the full SHA
    585dc90 View commit details
    Browse the repository at this point in the history

  5. Merge branch 'custom-export' into custom-export--amazon-sqs

    doubleswirve committed Jan 23, 2018
    Configuration menu
    Copy the full SHA
    0b5d707 View commit details
    Browse the repository at this point in the history

  6. update info logging

    doubleswirve committed Jan 23, 2018
    Configuration menu
    Copy the full SHA
    90bc055 View commit details
    Browse the repository at this point in the history

  7. port over some more helper to sqs

    doubleswirve committed Jan 23, 2018
    Configuration menu
    Copy the full SHA
    5ade2fe View commit details
    Browse the repository at this point in the history

  8. add s3 handling for messages that are too large

    doubleswirve committed Jan 23, 2018
    Configuration menu
    Copy the full SHA
    838b36e View commit details
    Browse the repository at this point in the history

  9. update s3 key format

    doubleswirve committed Jan 23, 2018
    Configuration menu
    Copy the full SHA
    9d5e9d2 View commit details
    Browse the repository at this point in the history

Commits on Jan 24, 2018

  1. update number formatting for py version

    doubleswirve committed Jan 24, 2018
    Configuration menu
    Copy the full SHA
    820d395 View commit details
    Browse the repository at this point in the history

Commits on Mar 27, 2018

  1. Merge pull request opencivicdata#3 from doubleswirve/custom-export--a… 

    …mazon-sqs

Custom export  amazon sqs
    Saxon Stiller authored Mar 27, 2018
    Configuration menu
    Copy the full SHA
    9db57ba View commit details
    Browse the repository at this point in the history

Commits on Aug 15, 2018

  1. resolve conflicts

    @doubleswirve
    doubleswirve committed Aug 15, 2018
    Configuration menu
    Copy the full SHA
    81bb820 View commit details
    Browse the repository at this point in the history

Commits on Dec 11, 2018

  1. Update opencivicdata min version

    @showerst
    showerst committed Dec 11, 2018
    Configuration menu
    Copy the full SHA
    2978f4c View commit details
    Browse the repository at this point in the history