Skipping posts that generate file names that are longer than the OS maximum.

[etui-null]

I have an extractor setup for FANBOX that I cannot get to go past a link that generates a filename that is longer than linux's maximum allowed file path size. I have tried manually editing the database file but even with all the entries present in the archive file it still attempts to write metadata and subsequently kills my program. Is there a way to get this link to be properly skipped?

Extractor config:

"fanbox":
{
    "archive": "archive-fanbox.sqlite3",
    "filename": "{num:>04}-{filename}.{extension}",
    "directory": ["{creatorId}-{user[name]}", "{id}-{title}"],
    "metadata": true,

    "cookies":
    {
        "FANBOXSESSID": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
    },

    "postprocessors":
    [{
        "name": "metadata",
        "event": "post",
        "filename": "{title}.txt",

        "#": "write text content",
        "format": [
            "{content:?//}",
            "{html:?//}",
            "{text:?//}",
            "{excerpt:?//}"
        ]
    }]
}

Error:

fanbox: Unable to download data:  OSError: [Errno 36] File name too long: './festa1130coro-ふぇすた＠coro/5237274-lz・hlest たくし上げ見抜き\u3000原寸＋髪型差 分２種(ポニテ、通常)＋眼鏡差分(ありなし)＋表情差分３種(照、煽、蔑)＋パンツなし差分３種(つるつる、陰毛黒、陰毛髪色)＋各ぶっかけ差分(計96枚？)/lz・hlest たくし上げ見抜き\u3000原寸＋髪型差分２種(ポニテ、通常)＋眼鏡差分(ありなし)＋表情差分３種(照、煽、蔑)＋パンツなし差分３種(つるつる、陰毛黒、陰毛髪色)＋各ぶっかけ差分(計96枚？).txt'

As you can see, the file path is massive and I have no idea how to skip past metadata writing. And no editing the config to remove the postprocessor isn't really an option as I need this data for links that don't throw errors.

[Hrxn]

You can limit the length of a metadata field like this:

<name>[:180]
i.e.
"filename": "{num:>04}-{filename[:180]}.{extension}"

[etui-null]

I see, 9 bytes too many. I tried with the limit at [:80] and that worked, would you happen to know what the maximum size I could make it would be?

[kattjevfel]

https://en.wikipedia.org/wiki/Comparison_of_file_systems#Limits 255 bytes on most file systems.

[mikf]

I know this is long overdue, but I've finally added a way to limit the number of bytes instead of characters in 69865dc.

                "filename": "{title[b:250]}.txt"

---

would you happen to know what the maximum size I could make it would be?

Assuming all Japanese characters need 3 bytes, 250 // 3 = 83 should be enough when using per-character slices [:83]

[Hrxn]

Sorry if this seems like a stupid question, but..
What is the benefit of slicing by number of bytes instead of characters? What am I missing here?

[mikf]

On file systems where you can only use 255 bytes for a filename and those names are encoded in UTF-8, the amount of possible characters depends on the size of Unicode code points, which can be up to 6 bytes per.
For example, a filename consisting entirely of Japanese characters, which need 3 bytes each, would only allow for 85 characters before surpassing the 255 byte limit, while it would be perfectly fine for this name to have 255 ASCII characters, which need only 1 byte each.
Basically, the number of possible characters depends on the "type" of characters, while the number of possible bytes is always 255.

This is not really a problem on Windows/NTFS, which stores characters in UTF-16 and allows for 255 UTF-16 encoded code points per name, which usually correspond 1-to-1 with the number of characters.