Feature/concurrent downloads

[0verbyte]

Wanted to get some initial feedback on this.

Changes proposed in this pull request:

• Adds support for implementing task logic using ProcessPoolExecutor, related Create Downloader class #54
• Emoji downloading is now done concurrently using the new base class, fixes Download emoji command should run concurrently #53

This isn't quite ready to be merged since the save file after the downloads have finished isn't being generated. There is now weirdness with the logger, since things are running concurrently the main REPL will return immediately after calling execute_tasks, and logs will start showing up afterwards moving the Slacker > prompt out of context. Will need to decide how to best reset the prompt.

[0verbyte]

Travis is failing because __write_save_file is unused now, will fix.

[netromdk]

I like the approach.

When I run it as-is with emoji.list -p /tmp/ then I get a lot of these exceptions:

Traceback (most recent call last):
  File "/usr/local/Cellar/python/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/queues.py", line 234, in _feed
    obj = _ForkingPickler.dumps(obj)
  File "/usr/local/Cellar/python/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
TypeError: can't pickle _thread.RLock objects

[netromdk]

For private fields, it's customary to prefix with __.

[netromdk]

Regarding the log output, it might be best to buffer that until it either succeeds or fails, and then post to the actual logger? Then it won't mess with the stdout until ready/finished.

[0verbyte]

When I run it as-is with emoji.list -p /tmp/ then I get a lot of these exceptions

Interesting that you're getting those exceptions. It seems to be related to Python version 3.6.5, I'm running 3.7.0 and not seeing those exceptions. I'll investigate further.

Regarding the log output, it might be best to buffer that until it either succeeds or fails, and then post to the actual logger? Then it won't mess with the stdout until ready/finished.

The REPL is now blocked until the tasks have completed. Blocking should be fine in this case since a user shouldn't be able to execute multiple commands while others are running, that could get messy IMO.

Although it could be interesting to support optional multiple command execution as a command line multi-flag, such as --command [COMMAND] --command [COMMAND].

% ./slacker.py --command emoji.list -p /tmp/emojis/ --command channels.list

[0verbyte]

I was able to repro the issue with Python 3.6 running in a Docker container.

FROM python:3.6-alpine

WORKDIR /app/slacker/
COPY . .

RUN pip install -r requirements.txt

ENTRYPOINT ["python3", "slacker.py"]

Instead of passing the logger directly to the tasks, the logs are being buffered and the caller can choose to replay the logs or not. This seems to fix the issue where things weren't able to be locked, can you give it a try now?

[netromdk]

Now it works great! Thanks!