Not possible use input-file option

[Ruslan0Dev]

Describe the bug
Not possible make this:
aria2c -i for_aria.txt

To Reproduce

aria2 = aria2p.API(aria2p.Client(host='http://localhost', port=6800, secret=''))
aria2.add_uris([], options={'input-file': 'for_aria.txt'})

System (please complete the following information):

• aria2p version: 0.9.1
• Python version: 3.8.3
• OS: Windows 10

[pawamoy]

Hmm yeah I didn't plan this kind of use. Maybe you could simply open the file, read its lines, and feed them to aria2.add* methods?

[Ruslan0Dev]

I understand ... But if the options in file, like this:

https://example.com/file.txt
    out=data.txt
    referer=https://example.com/downloads
    dir=D:\loads
   checksum=sha-1=0192ba11326fe2298c8cb4de616f4d4140213838
https://...next url
    out=data2.txt

it requires an additional parser) is not difficult, but why should I write it if it is already in aria2c
Sorry, I checked the documentation for RPC API, and it turned out this feature can not exist

[pawamoy]

OK so there's no way to tell the JSON-RPC daemon to add downloads using an input file with options?

We could consider adding the functionality in aria2p itself then. Something like an API.add_input_file method.

[jonnieey]

@pawamoy Whats would be the approach on this issue. I was thinking that one could provide an input file, which is parsed to create objects {uri: URI , options : [OPTIONS]}, then pass them to the add method. Could this work or you had a different idea on this?

[pawamoy]

I think it's the right approach. I would see it like this:

• in API, add a add_input_file method that expects a file path as arguments, and contents that can be parsed (uris separated by newlines, with optional options between them, indented with at least one space)
• if there is a parsing error, raise an exception
• in API.add, when adding as a file, first try to add as an input file, using add_input_file, and if an exception is raised, fallback on current behavior

Something like this:

    def add(self, uri: str) -> List[Download]:
        ...

        if path_exists:
            if path.suffix == ".torrent":
                new_downloads.append(self.add_torrent(path))
            elif path.suffix == ".metalink":
                new_downloads.extend(self.add_metalink(path))
            else:
                try:
                    new_downloads.extend(self.add_input_file(path))
                except ValueError:
                    for line in read_lines(path):
                        if line:
                            new_downloads.extend(self.add(line))
        ...

[jonnieey]

I tried to implement this but had the following issues. 1. If a value error is raised should the parsing stop or should it continue to parse other uri with valid options and inform invalid options to the user. 2. According to your suggestion, if an error is raised it should fallback on current behaviour (ie to read as normal file with uris). if the file was an input file then options will be added as uris. I have a beta version that reads all files and adds download options incase of an aria2 input file. If it is a normal file it adds the downloads but without options. It parses tab delimited options and new line indented options. What is your take on this? jonnieey@f84bcd6

[pawamoy]

Hello! Thank you for your patience 😅

You make a good point. Then I think we should only read a file as an input file since it will also support normal files (one URI on each line). If there's an error during parsing, it should be recorded so we can warn the user.

Now I looked at your code, and it seems you're not doing exactly what you describe here:

uri, download_options = line.rstrip().split("\t")[0], line.split("\t")[1:]

It seems here that you are trying to read an URI and its options on the same line, which is not correct.

The format accepted by aria2 is tab-delimited URIs on the same line (multiple URIs for one download), and options indented with one space, on new lines.

I would write the add_input_method like so:

def input_downloads(lines):
    """Helper to split downloads in an input file."""
    block = []
    for line in lines:
        if not line.startswith(" "):
            if block:
                yield block
                block = []
        block.append(line.rstrip("\n"))
    if block:
        yield block


def add_input_file(input_file):
    downloads = []
    with Path(input_file).open() as fd:
        for download_lines in input_downloads(fd):
            uris = download_lines[0].split("\t")
            options = {}
            for opt_line in download_lines[1:]:
                opt_name, opt_value = opt_line.split("=")
                options[opt_name.lstrip() = opt_value
            downloads.append((uris, options))
    return downloads  # or do something else with downloads

This function can read both input file as described by aria2, as well as normal files with just regular URIs on each line.
It would still need changes to handle possible formatting errors 🙂

[jonnieey]

Yeah. No problem I'll check it out and try to add some error handling functionality. :thumbsup:

[KOLANICH]

I'd like Aria2p having a very simple and basic mode like in https://github.com/prebuilder/downloaders.py/blob/master/downloaders/backends/aria2c.py

And to have it in a separate package, so the most of deps are unneed. But I don't want to maintain such a package ...

And maybe a compatibility layer, implementing the same, but through libaria.

[pawamoy]

Hi @KOLANICH, not sure to understand. Such a "simple and basic mode" can be achieved by running aria2c directly I guess? Or you want to use it programmatically, in Python? Then the solution is to implement support for an input file option, as discussed in this issue.

And to have it in a separate package, so the most of deps are unneed. But I don't want to maintain such a package ...

Me neither 😅

[KOLANICH]

. Such a "simple and basic mode" can be achieved by running aria2c directly I guess? Or you want to use it programmatically, in Python?

Yes, programmatically. If I wanted to run aria2c from bash, I wouldn't have used python.

Then the solution is to implement support for an input file option, as discussed in this issue.

Not quite. My impl is very simple: one just enumerates URIs and files to which download, the code generates an aria2c input file and feeds it to aria2c via a pipe, that is accessed by aria2c via its file path. No http-RPC, all the files are added before start. Its main advantage is that it is very small and simple and has very little deps. But having multiple packages for the same thing just fragments the ecosystem, it is not good.

[pawamoy]

I see. Unfortunately it won't happen in aria2p, as it is really a JSON-RPC client. It wouldn't make sense to have methods using JSON-RPC and others spawning aria2c processes.

[pawamoy]

Hey @jonnieey, very promising code! If you send a PR I can review it today 🙂

[pawamoy]

Implemented in #91 🎉! Thanks again @jonnieey