403 Error while using the REST API for code search

[lgingerich]

Select Topic Area

Question

Body

Hello, I am attempting to do some code search for multiple keywords, but I keep getting a 403 error that I can't figure out. My understanding of the API is that it will return a max of 100 results per page and authenticated requests are limited to 30 per minute.

My search result set is larger than 100, so I have to page through the results. I have three query strings to do this for.

For query_string_0, I get 100 results and save them to a dataframe. Then I get the next 100 results and concatenate dataframes. I continue this in a while loop until no more results are returned. Finally, I follow this same process for query_string_1 and query_string_2, all while concatenating into a single dataframe.

Sometimes I get the 403 while on page=2 of query_string_0, sometimes I get it while on page=6 of query_string_0. I'm checking the rate limit endpoint each time and haven't hit my limits. Would greatly appreciate any help to figure out what could be causing this. Thanks!

I've included my full code below as well as a picture of my debugging output.

import requests
import json
import pandas as pd
import time
import logging

headers = {
    "Accept": "application/vnd.github.json",
    "Authorization": "Bearer <access_key>",
}

# Define GitHub search criteria
base_url = "https://api.github.com/search/code?"
keywords = ["hardhat-zksync-deploy", "hardhat-zksync-solc", "hardhat-zksync-vyper"]
languages = ["javascript", "json", "python", "typescript", "yaml"]

# Build GitHub query strings
query_string_0 = (
    base_url + "q=" + keywords[0] + "+language:" + languages[0] + "+language:" + languages[1] + "+language:"
    + languages[2] + "+language:" + languages[3] + "+language:" + languages[4] + "&per_page=100"
)
query_string_1 = (
    base_url + "q=" + keywords[1] + "+language:" + languages[0] + "+language:" + languages[1] + "+language:"
    + languages[2] + "+language:" + languages[3] + "+language:" + languages[4] + "&per_page=100"
)
query_string_2 = (
    base_url + "q=" + keywords[2] + "+language:" + languages[0] + "+language:" + languages[1] + "+language:"
    + languages[2] + "+language:" + languages[3] + "+language:" + languages[4] + "&per_page=100"
)
query_strings = [query_string_0, query_string_1, query_string_2]

# Initialize DataFrame and while loop condition
df = pd.DataFrame()
response = True

for i in range(len(query_strings)):
    page_num = 1

    while response:
        url = query_strings[i] + "&page=" + str(page_num)

        # Check API rate limits
        response_rate_limit = requests.get(
            "https://api.github.com/rate_limit",
            headers=headers,
        )
        print(response_rate_limit.json())

        # Code search GET request
        response = requests.get(
            url,
            headers=headers,
        )        

        response_json = response.json()

        if response.status_code == 200:
            print("query string:", url)
            print("record count:", response_json["total_count"])
            print("status code:", response.status_code)
            response_normalized = pd.json_normalize(response_json, record_path=["items"])
        elif response.status_code != 200:
            logging.error(response.status_code)
            logging.error(response.request.url)

        df_add = pd.DataFrame(response_normalized)
        df_add["keyword"] = keywords[i]
        df = pd.concat([df, df_add], axis=0, ignore_index=True)

        page_num += 1

        # Authenticated requests can make up to 30 requests per minute. Add 2 second delay to stay under this.
        time.sleep(2)

        print("Concatenated DataFrame length:", len(df))
        print("--------------------------------------------")

[github-actions[bot]]

🕒 Discussion Activity Reminder 🕒

This Discussion has been labeled as dormant by an automated system for having no activity in the last 60 days. Please consider one the following actions:

1️⃣ Close as Out of Date: If the topic is no longer relevant, close the Discussion as out of date at the bottom of the page.

2️⃣ Provide More Information: Share additional details or context — or let the community know if you've found a solution on your own.

3️⃣ Mark a Reply as Answer: If your question has been answered by a reply, mark the most helpful reply as the solution.

Note: This dormant notification will only apply to Discussions with the Question label. To learn more, see our recent announcement.

Thank you for helping bring this Discussion to a resolution! 💬