Commit endpoint issues

[fhdk]

In attempts to deduce why I have not been able to fully use the commit endpoint, I have refactored the test cases to print out some debug information

For security reasons - the token is stripped to display the first 15 and the last 15 chars.

• the content of the request prior to sending
• the content of the response received

The content appear correct for a multipart/form-data request

From the last comment #92 (comment) and the example post #92 (comment)

• the only difference is the authentication method
• you are using cookie
• I am using bearer

I have also created testcases using cookie authentication to verify.

For these CookieAuth labelled test, I first used a cookie named token and when that failed I added a cookie named access_token.

Where the commit endpoint for uploading packages was previously accepting upload there seems to be an issue now.

These test cases results in

• bearer token posts result in 200 - but nothing happens
• cookie token posts result in 401 - token not found

From the debug information it appears like everything is OK.

I feel like I have exhausted my options of troubleshooting the endpoint - please let me know what you make of this

I could of course be a matter of letting the backend do the job - after it is async ...

Related issues

• Move request reports 200, but package not moved #84
• Copy package request reports 200 but no package is copied #92
• Delete packages request reports 200 but no package deleted #93
• Commit endpoint doesn't report permission issues #106

[fhdk]

bxt_upload_pkg : CookieAuth
cookiejar     : token: eyJhbGciOiJIUzI...f6PGSkhpdb-_e6A
cookiejar     : access_token: eyJhbGciOiJIUzI...f6PGSkhpdb-_e6A
req headers   : {'Accept': 'application/json', 'Content-Type': 'multipart/form-data'}
req url       : https://bxt.staging.manjaro.org/api/packages/commit
form data     : b'<binary content removed> - see below exerpt'
request begin    -->  2024-07-21 10:46:42
response recv    -->  2024-07-21 10:46:42
response headers -->  {'Alt-Svc': 'h3=":443"; ma=2592000', 'Content-Length': '45', 'Content-Type': 'application/json; charset=utf-8', 'Date': 'Sun, 21 Jul 2024 08:46:42 GMT', 'Server': 'Caddy, drogon/1.8.2'}
response status  -->  401
response content -->  b'{"message":"No token found","status":"error"}'

Textual representation of the form (with binary data removed)

--6410000b9b81b5e89b0b9fe35ea60f1b
Content-Disposition: form-data; name="package2.section"
  {"branch": "testing", "repository": "extra", "architecture": "x86_64"}
--6410000b9b81b5e89b0b9fe35ea60f1b
Content-Disposition: form-data; name="package1.section"
  {"branch": "testing", "repository": "extra", "architecture": "x86_64"}
--6410000b9b81b5e89b0b9fe35ea60f1b
Content-Disposition: form-data; name="package2.signature"; filename="a-dummy2-0-0-any.pkg.tar.zst.sig"
<binary data>
--6410000b9b81b5e89b0b9fe35ea60f1b
Content-Disposition: form-data; name="package1.signature"; filename="a-dummy1-0-0-any.pkg.tar.zst.sig"
<binary data>
--6410000b9b81b5e89b0b9fe35ea60f1b
Content-Disposition: form-data; name="package1.file"; filename="a-dummy1-0-0-any.pkg.tar.zst"
<binary data>
--6410000b9b81b5e89b0b9fe35ea60f1b
Content-Disposition: form-data; name="package2.file"; filename="a-dummy2-0-0-any.pkg.tar.zst"
<binary data>
--6410000b9b81b5e89b0b9fe35ea60f1b--

As can be seen it the form data appears to be in arbitrary order (created by the request library).

Is the parts is expected in lexical order?
Does it confuse the backend parser if it does not?

If the backend parser expect the content in lexical order - that can done by altering the object from Python dictionary to a tuple - giving a result providing in the same order as it was created - grouping the form parts together in logical order

--9d61aaf9eeda9018374fdf3d5212b340
Content-Disposition: form-data; name="package1.file"; filename="a-dummy1-0-0-any.pkg.tar.zst"
<binary data>
--9d61aaf9eeda9018374fdf3d5212b340
Content-Disposition: form-data; name="package1.signature"; filename="a-dummy1-0-0-any.pkg.tar.zst.sig"
<binary data>
--9d61aaf9eeda9018374fdf3d5212b340\
Content-Disposition: form-data; name="package1.section"
{"branch": "testing", "repository": "extra", "architecture": "x86_64"}
--9d61aaf9eeda9018374fdf3d5212b340
Content-Disposition: form-data; name="package2.file"; filename="a-dummy2-0-0-any.pkg.tar.zst"
<binary data>
--9d61aaf9eeda9018374fdf3d5212b340
Content-Disposition: form-data; name="package2.signature"; filename="a-dummy2-0-0-any.pkg.tar.zst.sig"
<binary data>
--9d61aaf9eeda9018374fdf3d5212b340\
Content-Disposition: form-data; name="package2.section"
{"branch": "testing", "repository": "extra", "architecture": "x86_64"}
--9d61aaf9eeda9018374fdf3d5212b340--

[fhdk]

I am speculating - and decoded the decodable part of the jwt

The cookie token has payload

{
  "exp": 1721626663,
  "iat": 1721625763,
  "iss": "bxt",
  "kind": "access_token",
  "storage": "cookie",
  "username": "**********"
}

And the bearer token has payload

{
  "exp": 1721626867,
  "iat": 1721625967,
  "iss": "bxt",
  "kind": "access_token",
  "storage": "bearer",
  "username": "**********"
}

Is the storage property used by the daemon, to reject a token issued for another storage ?

If that is so, it explains why my CookieAuth test-cases return 401 as I assumed they could be used for either authentication.

[fhdk]

STATUS

To recap - I have the upload functionality using bearer token back in place.

I found that requests_toolbelt MultipartEncoder did a better job than posting the files object.

It was your comment somewhere about the added filename property to the form content which gave me the clue I needed there.

The move, copy and delete function still returns 200 but nothing happens.

[fhdk]

STATUS

A simple move request for my test packages sinks into a deep black hole - I had to add a timeout in the http client.

As of right now - response is never received from the commit endpoint when POSTing a move form

bxt_move_pkg  : BearerAuth
req headers   : {'Authorization': 'Bearer eyJhbGciOiJIUzI...xM5hWgLKlFpJc9I', 'Accept': 'application/json', 'Content-Type': 'multipart/form-data; boundary=2d0704ea7dbd432c82a3306de1804098'}
req url       : https://bxt.staging.manjaro.org/api/packages/commit
form data     : <MultipartEncoder: {('to_move', '[{"name": "a-dummy1", "from_section": {"branch": "testing", "repository": "extra", "architecture": "x86_64"}, "to_section": {"branch": "testing", "repository": "extra", "architecture": "aarch64"}}, {"name": "a-dummy2", "from_section": {"branch": "testing", "repository": "extra", "architecture": "x86_64"}, "to_section": {"branch": "testing", "repository": "extra", "architecture": "aarch64"}}]')}>
multipart_data: b'--2d0704ea7dbd432c82a3306de1804098\r\nContent-Disposition: form-data; name="to_move"\r\n\r\n[{"name": "a-dummy1", "from_section": {"branch": "testing", "repository": "extra", "architecture": "x86_64"}, "to_section": {"branch": "testing", "repository": "extra", "architecture": "aarch64"}}, {"name": "a-dummy2", "from_section": {"branch": "testing", "repository": "extra", "architecture": "x86_64"}, "to_section": {"branch": "testing", "repository": "extra", "architecture": "aarch64"}}]\r\n--2d0704ea7dbd432c82a3306de1804098--\r\n'
request begin    -->  2024-07-26 09:02:25
RequestException -->  2024-07-26 09:02:55
HTTPSConnectionPool(host='bxt.staging.manjaro.org', port=443): Read timed out. (read timeout=30)

For a delete request, the result is the same

bxt_delete_pkg: BearerAuth
req headers   : {'Authorization': 'Bearer eyJhbGciOiJIUzI...GgnHY1WQXuW59io', 'Accept': 'application/json', 'Content-Type': 'multipart/form-data; boundary=5340feb22e2e444bbaa1fea2fdeec4b3'}
req url       : https://bxt.staging.manjaro.org/api/packages/commit
form data     : <MultipartEncoder: {('to_delete', '[{"name": "a-dummy1", "section": {"branch": "testing", "repository": "extra", "architecture": "x86_64"}}, {"name": "a-dummy2", "section": {"branch": "testing", "repository": "extra", "architecture": "x86_64"}}]')}>
multipart_data: b'--5340feb22e2e444bbaa1fea2fdeec4b3\r\nContent-Disposition: form-data; name="to_delete"\r\n\r\n[{"name": "a-dummy1", "section": {"branch": "testing", "repository": "extra", "architecture": "x86_64"}}, {"name": "a-dummy2", "section": {"branch": "testing", "repository": "extra", "architecture": "x86_64"}}]\r\n--5340feb22e2e444bbaa1fea2fdeec4b3--\r\n'
request begin    -->  2024-07-26 09:38:43
RequestException -->  2024-07-26 09:39:13
HTTPSConnectionPool(host='bxt.staging.manjaro.org', port=443): Read timed out. (read timeout=30)

And the copy request

bxt_copy_pkg  : BearerAuth
req headers   : {'Authorization': 'Bearer eyJhbGciOiJIUzI...GgnHY1WQXuW59io', 'Accept': 'application/json', 'Content-Type': 'multipart/form-data; boundary=71c42dfaa54a4a388dea52448208d205'}
req url       : https://bxt.staging.manjaro.org/api/packages/commit
form data     : <MultipartEncoder: {('to_copy', '[{"name": "a-dummy1", "from_section": {"branch": "testing", "repository": "extra", "architecture": "x86_64"}, "to_section": {"branch": "testing", "repository": "extra", "architecture": "aarch64"}}, {"name": "a-dummy2", "from_section": {"branch": "testing", "repository": "extra", "architecture": "x86_64"}, "to_section": {"branch": "testing", "repository": "extra", "architecture": "aarch64"}}]')}>
multipart_data: b'--71c42dfaa54a4a388dea52448208d205\r\nContent-Disposition: form-data; name="to_copy"\r\n\r\n[{"name": "a-dummy1", "from_section": {"branch": "testing", "repository": "extra", "architecture": "x86_64"}, "to_section": {"branch": "testing", "repository": "extra", "architecture": "aarch64"}}, {"name": "a-dummy2", "from_section": {"branch": "testing", "repository": "extra", "architecture": "x86_64"}, "to_section": {"branch": "testing", "repository": "extra", "architecture": "aarch64"}}]\r\n--71c42dfaa54a4a388dea52448208d205--\r\n'
request begin    -->  2024-07-26 09:41:01
RequestException -->  2024-07-26 09:41:31
HTTPSConnectionPool(host='bxt.staging.manjaro.org', port=443): Read timed out. (read timeout=30)

[fhdk]

The upload packages transaction now works as expected.

A combination of efforts I assume - thank you.

Issue #84 is not resolved as of now - so I close this and leave #84 open