Zurg Ingestion: ZurgEndpointSuffix configuration value ignored?

[Naralux]

I'm trying to get the Zurg ingestion to work, but I keep getting 404's. I get the idea that the value of ZurgEndpointSuffix as configured in the settings.json is not used.

Excerpt from my settings.json file:

    "Ingestion": {
      "ZurgInstances": [
        {
          "Url": "http://zurg:9999",
          "EndpointType": 1,
          "ApiKey": ""
        }
      ],
      "ZileanInstances": [],
      "EnableScraping": true,
      "Kubernetes": {
        "EnableServiceDiscovery": false,
        "KubernetesSelectors": [],
        "KubeConfigFile": "/$HOME/.kube/config",
        "AuthenticationType": 0
      },
      "ScrapeSchedule": "0 * * * *",
      "ZurgEndpointSuffix": "/http/torrents/",
      "ZileanEndpointSuffix": "/torrents/all",
      "RequestTimeout": 10000
    },

• Zilean (3.3.0) and Zurg (0.9.3) are running in the same Docker Compose network. http://zurg:9999 is reachable from within the Zilean container.
• The Example Configuration uses "ZurgEndpointSuffix": "/debug/torrents".
• http://zurg:9999/debug/torrents does not exist. http://zurg:9999/http/torrents/ does output a list of torrents.
• Class IngestionConfiguration sets the default value of ZurgEndpointSuffix to /debug/torrents:

  zilean/src/Zilean.Shared/Features/Configuration/IngestionConfiguration.cs

  Line 10 in f8af7f9

  public string ZurgEndpointSuffix { get; set; } = "/debug/torrents";

• Class StreamedEntryProcessor method ProduceEntriesAsync seems to be using hardcoded value of /debug/torrents instead of the value configured in settings.json:

  zilean/src/Zilean.Scraper/Features/Ingestion/Processing/StreamedEntryProcessor.cs

  Line 41 in f8af7f9

  GenericEndpointType.Zurg => $"{_currentEndpoint.Url}/debug/torrents",

1. Could this have something to do with the 404's?
2. Or is /debug/torrents the correct endpoint? If so, is that a sponsor-only feature?

[Naralux]

Logging excerpt:

zilean  | [22:04:49] | INFO | "Zilean.Scraper.Features.Ingestion.Endpoints.GenericIngestionScraping" | Starting ingestion scraping
zilean  | [22:04:49] | INFO | "Zilean.Scraper.Features.Ingestion.Endpoints.GenericIngestionScraping" | Adding Zurg instances to the list of URLs to process
zilean  | [22:04:49] | INFO | "Zilean.Scraper.Features.Ingestion.Processing.GenericProcessor" | Processing URL: GenericEndpoint { Url: "http://zurg:9999", EndpointType: Zurg, ApiKey: "" }
zilean  | [22:04:49] | EROR | "Zilean.Scraper.Features.Ingestion.Processing.GenericProcessor" | Error while fetching and producing entries for URL: "http://zurg:9999"
zilean  | System.Net.Http.HttpRequestException: Response status code does not indicate
zilean  | success: 404 (Not Found).
zilean  |   at HttpResponseMessage System.Net.Http.HttpResponseMessage.
zilean  |      EnsureSuccessStatusCode()
zilean  |   at async Task
zilean  |      Zilean.Scraper.Features.Ingestion.Processing.StreamedEntryProcessor.
zilean  |      ProduceEntriesAsync(ChannelWriter<Task<StreamedEntry>> writer,
zilean  |      CancellationToken cancellationToken) in
zilean  |      /build/src/Zilean.Scraper/Features/Ingestion/Processing/
zilean  |      StreamedEntryProcessor.cs:51
zilean  | ┌────────────────────┬───────┬───────────────────┐
zilean  | │ Description        │ Count │ Additional Info   │
zilean  | ├────────────────────┼───────┼───────────────────┤
zilean  | │ Processed torrents │ 0     │ Time Taken: 0.09s │
zilean  | └────────────────────┴───────┴───────────────────┘
zilean  | [22:04:50] | INFO | "Zilean.Scraper.Features.Ingestion.Endpoints.GenericIngestionScraping" | Ingestion scraping completed for 1 URLs
zilean  | [22:04:50] | INFO | "Zilean.ApiService.Features.Sync.GenericSyncJob" | Generic SyncJob completed

[titooo7]

Hi there, I'm having a similar issue. In any case, I got a question. I notice that you don't have any ApiKey for ZurgInstances.

Is that because you edited it out? Aren't we suppose to add there the api key found in the token line of one of the config.yml files?