API requests are getting intercepted by cloudflare DDOS protections, and also sort/filter doesn't work #1525

kernkraft235 · 2025-01-11T23:27:09Z

kernkraft235
Jan 11, 2025

What I did

This is the reference request, I tried many other variations

curl -X POST 'https://api.macosicons.com/api/search' \
  -H "Content-Type: application/json" \
  -H "x-api-key: $MACOSICONS_TOKEN" \
  -d '{
    "query": "WebBites",
    "searchOptions": {
      "hitsPerPage": 2,
      "offset": 0,
      "page": 1,
      "filters": ["downloads >= 20", "category = BO0gbTCPUK"]
    }
  }'

What I received

Cloudflare protection, I can't turn on javascript on a curl request, and I doubt this is how the API is meant to work.

<!DOCTYPE html>
<html lang="en-US">

<head>
    <title>Just a moment...</title>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
    <meta http-equiv="X-UA-Compatible" content="IE=Edge">
    <meta name="robots" content="noindex,nofollow">
    <meta name="viewport" content="width=device-width,initial-scale=1">
    <style>
        * {
            box-sizing: border-box;
            margin: 0;
            padding: 0
        }

        html {
            line-height: 1.15;
            -webkit-text-size-adjust: 100%;
            color: #313131;
            font-family: system-ui, -apple-system, BlinkMacSystemFont, Segoe UI, Roboto, Helvetica Neue, Arial, Noto Sans, sans-serif, Apple Color Emoji, Segoe UI Emoji, Segoe UI Symbol, Noto Color Emoji
        }

...

        #challenge-error-text {
            background-image: url(data:image/svg+xml;base64,PHN2ZyB4bWxuc.....);
            background-repeat: no-repeat;
            background-size: contain;
            padding-left: 34px
        }

        @media (prefers-color-scheme:dark) {
            body {
                background-color: #222;
                color: #d9d9d9
            }
        }
    </style>
    <meta http-equiv="refresh" content="390">
</head>

<body class="no-js">
    <div class="main-wrapper" role="main">
        <div class="main-content"><noscript>
                <div class="h2"><span id="challenge-error-text">Enable JavaScript and cookies to continue</span></div>
            </noscript></div>
    </div>
    <script>(function () { window._cf_chl_opt = { cvId: '3', cZone: "api.macosicons.com", cType: 'managed', cRay: '8f80f...', cH: 'vLKnZRp3gh2.....', cUPMDTk: "\/api\/search?__cf_chl_tk=im7Gud0nY...", cFPWv: 'b', cITimeS: '173...', cTTimeMs: '1000', cMTimeMs: '390000', cTplC: 0, cTplV: 5, cTplB: 'cf', cK: "", fa: "\/api\/search?__cf_chl_f_tk=im7Gud0nY3GJ92k......", mdrd: "q1nDaw8ecOL.........." }; var cpo = document.createElement('script'); cpo.src = '/cdn-cgi/challenge-platform/h/b/orchestrate/chl_page/v1?ray=8f80f5...'; window._cf_chl_opt.cOgUHash = location.hash === '' && location.href.indexOf('#') !== -1 ? '#' : location.hash; window._cf_chl_opt.cOgUQuery = location.search === '' && location.href.slice(0, location.href.length - window._cf_chl_opt.cOgUHash.length).indexOf('?') !== -1 ? '?' : location.search; if (window.history && window.history.replaceState) { var ogU = location.pathname + window._cf_chl_opt.cOgUQuery + window._cf_chl_opt.cOgUHash; history.replaceState(null, null, "\/api\/search?__cf_chl_rt_tk=im7Gud0nY3GJ9..." + window._cf_chl_opt.cOgUHash); cpo.onload = function () { history.replaceState(null, null, ogU); } } document.getElementsByTagName('head')[0].appendChild(cpo); }());</script>
</body>

</html>

Work-around

#!/usr/bin/env python3
import browser_cookie3
import requests
import sys
import json
import os
from requests_html import HTMLSession

api_key = os.getenv("MACOSICONS_API_KEY")
search_query = sys.argv[1]
search_url = 'https://api.macosicons.com/api/search'
cj = browser_cookie3.chrome()

# Define headers (mimicking a browser)
headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36',
    'Content-Type': 'application/json',
    'x-api-key': api_key,
}

# can sort on 'appName', 'downloads', 'timeStamp'
data = {
    "query": search_query,
    "searchOptions": {
      "hitsPerPage": 20,
      "page": 1,
      "offset": 0
    }
}
session = HTMLSession()
response = session.post(search_url, headers=headers, json=data)

if response.status_code == 200:
    # Load the response JSON
    json_data = response.json()

    # Print valid JSON (with double quotes) for `jq`
    print(json.dumps(json_data, ensure_ascii=False, indent=4))
else:
    print(f"Error: {response.status_code}")
    print(response.text)

Filters and Sort is broken

Yes, I did read the meilisearch documentation, and I even tried the provided reference search that uses those parameters.
The above script fails when I use this for data:

data = {
    "query": search_query,
    "searchOptions": {
      "hitsPerPage": 20,
      "page": 1,
      "offset": 0,
      "filters": ["downloads >= 20"],
      "sort": ["downloads:desc", "appName:asc"]
    }
}

I tried several variations, tried only doing one array value at a time, tried making it not an array, and in all cases I had the python script dump the raw json it was using in the request and it matched what the documentation said.

I even paid the $10 thinking it might be behind the paywall.

I'm not even trying to use this for commercial use or any other type of financial gain. My motivations are purely about 'ricing' my OS UI (wish there was a better word for it). I'm trying to make a Stable Diffusion LoRa without having to manually name/tag/caption every file. This is a "mission" I've been on for a long time and the only thing in my way has been this website. Initially it was because of the way the website names downloads, but then I saw an API was added but got bamboozled once again when sort/filter didn't work.

I'd be willing to pay like $100 to just get a dump of the icons (with metadata!!!) and a license agreement that prohibits commercial use or refers them back to macosicons.com where you can license it commercially (I'm keeping my $100 in that case).

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

API requests are getting intercepted by cloudflare DDOS protections, and also sort/filter doesn't work #1525

{{title}}

Replies: 0 comments

Select a reply

API requests are getting intercepted by cloudflare DDOS protections, and also sort/filter doesn't work #1525

kernkraft235 Jan 11, 2025

What I did

What I received

Work-around

Filters and Sort is broken

Replies: 0 comments

kernkraft235
Jan 11, 2025