Add support for extra HTTP headers in IRCAuthorization

[talatuyarer]

Adds support in DuckDB’s Iceberg extension to attach user-provided “extra headers” to all HTTP requests made to the REST catalog (including the initial /v1/config call and subsequent table/namespace operations).

This aligns DuckDB with how other Iceberg clients treat REST catalogs. Example usage:

CREATE SECRET iceberg_secret_single (
	TYPE ICEBERG,
	CLIENT_ID 'admin',
	CLIENT_SECRET 'password',
	OAUTH2_SERVER_URI 'http://127.0.0.1:8181/v1/oauth/tokens',
	EXTRA_HTTP_HEADERS MAP {'X-Custom-Header': 'custom-value'}
);

I also fixed a bug if catalog return prefix which has slashes in it. Duckdb encode those slashes which it should not encode.

[talatuyarer]

Hi @Tmonster Could you review my pr ?

[Tmonster]

Hi @talatuyarer,

yes, I will take a look. Just have been super busy with some things at the moment.

[Tmonster]

Looks good to me.
Can you add some tests to make sure they are sent in the request as well?

In a test like at test/sql/local/irc/test_table_information_requests.test you can see us check the request. To check the headers I think something like

select request.headers from duckdb_logs_parsed('http');

should work

[talatuyarer]

@Tmonster Thank you for pointer I added also tests too. and also run make format to fix format issue fyi

[talatuyarer]

@Tmonster @Tishj This is ready to merge :)

[Tmonster]

Thanks! Just set up a test against some other IRC catalogs. Will take another look when that finishes

[Tmonster]

Looks like some of the cloud tests are failing. You can see the run here. Seems like the s3tables attach is failing. They return an already encoded s3 prefix. The response to the /config endpoint looks something like this

{
  "defaults": {
    "write.object-storage.partitioned-paths": "false",
    "s3.delete-enabled": "false",
    "io-impl": "org.apache.iceberg.aws.s3.S3FileIO",
    "write.object-storage.enabled": "true",
    "prefix": "arn%3Aaws%3As3tables%3Aus-east-2%3A<account_id>%3Abucket%2Ficeberg-testing",
    "rest-metrics-reporting-enabled": "false"
  },
  "overrides": {}
}

And the url we hit now for namespaces is

/iceberg/v1/arn%3Aaws%3As3tables%3Aus-east-2%3A<account_id>%3Abucket%2Ficeberg-testing/namespaces"

The url we hit for table listing is

/iceberg/v1/arn%3Aaws%3As3tables%3Aus-east-2%3A<account_id>%3Abucket/iceberg-testing/namespaces/default/tables

Notice the difference in bucket%2Ficeberg-testing and bucket/iceberg-testing

It seems the URL builder in GetTables is building the following components in the GetTable request

  path_components = {
    std::__1::vector<std::__1::string, std::__1::allocator<std::__1::string> > = size=6 {
      [0] = "v1"
      [1] = "arn:aws:s3tables:us-east-2:<account_id>:bucket"
      [2] = "iceberg-testing"
      [3] = "namespaces"
      [4] = "default"
      [5] = "tables"
    }
  }

But in the GetSchemas the URL builder has the following components.

  path_components = {
  std::__1::vector<std::__1::string, std::__1::allocator<std::__1::string> > = size=3 {
    [0] = "v1"
    [1] = "arn:aws:s3tables:us-east-2:<account_id>:bucket/iceberg-testing"
    [2] = "namespaces"
  }
}
host = "s3tables.us-east-2.amazonaws.com/iceberg"
params = size=0 {}
}

It seems like here you missed AddPrefixComponent.

I think the fix here is to write some functionality to detect if the prefix is already encoded or not? If it is encoded, decode it and add it as the prefix. Also, S3Tables is super easy to set up here for debugging. It really is just a matter of creating a bucket in the S3Tables console.

I don't know what BigLake returns (encoded or decoded), would be nice if you could share that here

Also, can you add a test where you explicitly add an Authorization header? And then make sure it gets overridden?

[talatuyarer]

I think the fix here is to write some functionality to detect if the prefix is already encoded or not? If it is encoded, decode it and add it as the prefix.

AddPrefixComponent handles this by checking for /. If the prefix is URL-encoded, it won't contain /.

I've added a unit test to ensure the Authorization header isn't overridden by extra headers and also added the missing AddPrefixComponent.

[talatuyarer]

@Tmonster BigLake is very user-friendly; a public dataset simplifies Iceberg testing, and all you need is a Google Cloud Project. Refer to the instructions here: #665 (comment)

[Tmonster]

Hi @talatuyarer I forked and opened a new PR here. The problem is we already decode url prefixes when they are returned from the catalog.
We cannot leave the prefixes in a url-encoded state because we re-encode urls to the catalog. Leaving components encoded will cause them to be double encoded and requests will start to fail.

I've checked this also against the public Biglake bucket you mentioned, along with our cloud tests and everything seems to work fine. Feel free to leave comments on my PR if you have anything you'd like to add