Postprocessor question

[Twi-Hard]

Is it possible to automatically sort jsons into directories based on the id of the image?
This works for the images but not jsons:

"directory": [
    "images",
    "\fE str(id // 5000 * 5000 + 5000)"
],

[mikf]

directory for metadata post processors can currently only be a static value and is interpreted as a path relative to the current download location (i.e. extractor.directory), so by default the jsons should be placed in the same directory as the images.

[Twi-Hard]

Pings are good. I have a huge amount of GitHub emails, and the titles are mostly truncated in the Gmail app so sometimes I miss ones I want to see. When I get pinged the GitHub app sends a push notification.

I don't actually know python so I asked Claude to add the print statements. This is the result:

❯ gdl https://derpibooru.org/0 --no-skip
/abc/archive/booru/derpibooru.org/images/5000/derpibooru_0.jpgDEBUG: Directory segments: ['json', '5000']
DEBUG: Cleaned directory path: json/5000/
DEBUG: Full directory path: /abc/archive/booru/derpibooru.org/images/5000/json/5000/
DEBUG: Absolute path: /abc/archive/booru/derpibooru.org/images/5000/json/5000
DEBUG: Base directory: /abc/archive/booru/derpibooru.org/images/5000/json/5000/
DEBUG: Full file path: /abc/archive/booru/derpibooru.org/images/5000/json/5000/derpibooru_0.jpg.json
DEBUG: Absolute file path: /abc/archive/booru/derpibooru.org/images/5000/json/5000/derpibooru_0.jpg.json

I also tried without using my wrapper script:

❯ gallery-dl --config /abc/environment/config/gdl_general.conf https://derpibooru.org/0 --no-skip
[gallery-dl][debug] Starting DownloadJob for 'https://derpibooru.org/0'
[derpibooru][debug] Using PhilomenaPostExtractor for 'https://derpibooru.org/0'
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): derpibooru.org:443
[urllib3.connectionpool][debug] https://derpibooru.org:443 "GET /api/v1/json/images/0 HTTP/11" 200 1299
[derpibooru][debug] Using download archive '/home/mane/tools/gallery-dl/archive/derpibooru.org.db'
[postprocessor.exec][debug] Using exec archive '/home/mane/tools/gallery-dl/archive/booru-svg-to-png.db'
[derpibooru][debug] Active postprocessor modules: [MtimePP, MetadataPP, MetadataPP, ExecPP]
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): derpicdn.net:443
[urllib3.connectionpool][debug] https://derpicdn.net:443 "GET /img/view/2012/1/2/0__artist+needed_safe_derpy+hooves_pegasus_pony_g4_adventure+in+the+comments_bag_building_chair_cigar_derpibooru+legacy_duo_duo+male+and+female_eyes_featu.jpg HTTP/11" 200 134712
/abc/archive/booru//derpibooru.org/images/5000/derpibooru_0.jpgDEBUG: Directory segments: ['json', '5000']
DEBUG: Cleaned directory path: json/5000/
DEBUG: Full directory path: /abc/archive/booru//derpibooru.org/images/5000/json/5000/
DEBUG: Absolute path: /abc/archive/booru/derpibooru.org/images/5000/json/5000
DEBUG: Base directory: /abc/archive/booru//derpibooru.org/images/5000/json/5000/
DEBUG: Full file path: /abc/archive/booru//derpibooru.org/images/5000/json/5000/derpibooru_0.jpg.json
DEBUG: Absolute file path: /abc/archive/booru/derpibooru.org/images/5000/json/5000/derpibooru_0.jpg.json
/abc/archive/booru//derpibooru.org/images/5000/derpibooru_0.jpg

After showing it that output it added different debugging stuff which actually fixed the problem. So maybe you can look at the changed code to see what's different?

the changed code:

    def _directory_format(self, pathfmt):
        formatters = pathfmt.directory_formatters
        conditions = pathfmt.directory_conditions
        try:
            pathfmt.directory_formatters = self._directory_formatters
            pathfmt.directory_conditions = ()
            
            print("DEBUG: Applying formatters:")
            segments = []
            for i, fmt in enumerate(self._directory_formatters):
                try:
                    segment = fmt(pathfmt.kwdict)
                    print(f"DEBUG: Formatter {i} output: {segment}")
                    segments.append(segment)
                except Exception as e:
                    print(f"DEBUG: Error in formatter {i}: {str(e)}")
            
            print("DEBUG: Final segments:", segments)
            directory = pathfmt.clean_path(os.sep.join(segments) + os.sep)
            print("DEBUG: Cleaned directory path:", directory)
            full_path = os.path.join(pathfmt.realdirectory, directory)
            print("DEBUG: Full directory path:", full_path)
            print("DEBUG: Absolute path:", os.path.abspath(full_path))
            return full_path
        finally:
            pathfmt.directory_conditions = conditions
            pathfmt.directory_formatters = formatters

gallery-dl output:

❯ gdl https://derpibooru.org/0 --no-skip
/abc/archive/booru/derpibooru.org/images/5000/derpibooru_0.jpgDEBUG: Applying formatters:
DEBUG: Formatter 0 output: ..
DEBUG: Formatter 1 output: ..
DEBUG: Formatter 2 output: json
DEBUG: Formatter 3 output: 5000
DEBUG: Final segments: ['..', '..', 'json', '5000']
DEBUG: Cleaned directory path: ../../json/5000/
DEBUG: Full directory path: /abc/archive/booru/derpibooru.org/images/5000/../../json/5000/
DEBUG: Absolute path: /abc/archive/booru/derpibooru.org/json/5000
DEBUG: Base directory: /abc/archive/booru/derpibooru.org/images/5000/../../json/5000/
DEBUG: Full file path: /abc/archive/booru/derpibooru.org/images/5000/../../json/5000/derpibooru_0.jpg.json
DEBUG: Absolute file path: /abc/archive/booru/derpibooru.org/json/5000/derpibooru_0.jpg.json
DEBUG: Directory not found, creating: /abc/archive/booru/derpibooru.org/images/5000/../../json/5000/

And the file is in the correct location:

❯ ls /abc/archive/booru/derpibooru.org/json/5000/
derpibooru_0.jpg.json

Do you think the issue could be related to python versions? I'm using Python 3.12.3.

[mikf]

@Twi-Hard thanks for this.

Your modified code does not call pathfmt.build_directory() but does the formatting step by itself, which seems to make all the difference.

diff --git a/gallery_dl/postprocessor/metadata.py b/gallery_dl/postprocessor/metadata.py
index a520a34c..dfeada7a 100644
--- a/gallery_dl/postprocessor/metadata.py
+++ b/gallery_dl/postprocessor/metadata.py
@@ -159,7 +159,12 @@ class MetadataPP(PostProcessor):
         try:
             pathfmt.directory_formatters = self._directory_formatters
             pathfmt.directory_conditions = ()
-            segments = pathfmt.build_directory(pathfmt.kwdict)
+
+            segments = []
+            for i, fmt in enumerate(self._directory_formatters):
+                segment = fmt(pathfmt.kwdict)
+                segments.append(segment)
+
             directory = pathfmt.clean_path(os.sep.join(segments) + os.sep)
             return os.path.join(pathfmt.realdirectory, directory)
         finally:

DEBUG: Directory segments: ['json', '5000']

DEBUG: Final segments: ['..', '..', 'json', '5000']

I'd guess your path-strip setting includes a period ., which is why '..' gets reduced to an empty string and ignored when pathfmt.build_directory() is involved.

[Twi-Hard]

If I comment out "path-strip" it does work :)
I use "path-strip": "windows", because I view and filter out stuff not related to what I want using a Windows VM, plus it's good to use in case I ever need to share stuff with a Windows user.
If you made a change, it still doesn't work with "path-strip" enabled for me yet. I tried again after re-updating the dev branch.

[mikf]

@Twi-Hard "fixed" in 1bdb0f7
This is not at all an optimal solution, but other options I could think of would either include ignoring certain path-… settings for metadata files or doing some major rewrites regarding path handling, so I went with this.

[Twi-Hard]

Thank you :)

[mikf]

There's now a base-directory option for metadata post processors.
Enabling it would make using .. in directory unnecessary.