use 429_sleep_interval for subdomaincenter

[indeed404]

subdomaincenter should also use the config value and not a static value of 20 seconds

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93%. Comparing base (9b01860) to head (62b8813).
⚠️ Report is 10 commits behind head on dev.

Additional details and impacted files

@@          Coverage Diff          @@
##             dev   #2605   +/-   ##
=====================================
- Coverage     93%     93%   -0%     
=====================================
  Files        402     402           
  Lines      33276   33268    -8     
=====================================
- Hits       30764   30756    -8     
  Misses      2512    2512           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[indeed404]

@liquidsec is there an estimate when we can expect this to be merged? we are currently running off my fork and would be nice if we could be running off of official bbot releases again

[liquidsec]

@liquidsec is there an estimate when we can expect this to be merged? we are currently running off my fork and would be nice if we could be running off of official bbot releases again

Hi, thanks for the PR. Should go in dev today, but it could be some time before the next dev->stable push. That depends on a lot of other moving parts.

[TheTechromancer]

Thanks for the PR. Since this module was written, we added 429 logic into the base module. So this can be updated to use .api_request(), which respects the global settings. I'll make the update.

[TheTechromancer]

@indeed404 I don't have permission for your branch. Can you make this change?

diff --git a/bbot/modules/subdomaincenter.py b/bbot/modules/subdomaincenter.py
index 3d98a9587..6b84b4023 100644
--- a/bbot/modules/subdomaincenter.py
+++ b/bbot/modules/subdomaincenter.py
@@ -12,25 +12,11 @@ class subdomaincenter(subdomain_enum):
     }
 
     base_url = "https://api.subdomain.center"
-    retries = 2
-
-    async def sleep(self, time_to_wait):
-        self.info(f"Sleeping for {time_to_wait} seconds to avoid rate limit")
-        await self.helpers.sleep(time_to_wait)
 
     async def request_url(self, query):
         url = f"{self.base_url}/?domain={self.helpers.quote(query)}"
-        response = None
-        status_code = 0
-        for i, _ in enumerate(range(self.retries + 1)):
-            if i > 0:
-                self.verbose(f"Retry #{i} for {query} after response code {status_code}")
-            response = await self.helpers.request(url, timeout=self.http_timeout + 30)
-            status_code = getattr(response, "status_code", 0)
-            if status_code == 429:
-                await self.sleep(self.scan.web_config.get("429_sleep_interval", 20))
-            else:
-                break
+        # we use longer timeout since this API can be slow (it returns a lot of results)
+        response = await self.api_request(url, timeout=self.http_timeout + 30)
         return response
 
     async def parse_results(self, r, query):

Thanks

[indeed404]

@indeed404 I don't have permission for your branch. Can you make this change?

diff --git a/bbot/modules/subdomaincenter.py b/bbot/modules/subdomaincenter.py
index 3d98a9587..6b84b4023 100644
--- a/bbot/modules/subdomaincenter.py
+++ b/bbot/modules/subdomaincenter.py
@@ -12,25 +12,11 @@ class subdomaincenter(subdomain_enum):
     }
 
     base_url = "https://api.subdomain.center"
-    retries = 2
-
-    async def sleep(self, time_to_wait):
-        self.info(f"Sleeping for {time_to_wait} seconds to avoid rate limit")
-        await self.helpers.sleep(time_to_wait)
 
     async def request_url(self, query):
         url = f"{self.base_url}/?domain={self.helpers.quote(query)}"
-        response = None
-        status_code = 0
-        for i, _ in enumerate(range(self.retries + 1)):
-            if i > 0:
-                self.verbose(f"Retry #{i} for {query} after response code {status_code}")
-            response = await self.helpers.request(url, timeout=self.http_timeout + 30)
-            status_code = getattr(response, "status_code", 0)
-            if status_code == 429:
-                await self.sleep(self.scan.web_config.get("429_sleep_interval", 20))
-            else:
-                break
+        # we use longer timeout since this API can be slow (it returns a lot of results)
+        response = await self.api_request(url, timeout=self.http_timeout + 30)
         return response
 
     async def parse_results(self, r, query):

Thanks

Hey, unfortunately this solution won't work for us so I cannot make the change on my fork. In the past (before you made 429_sleep_interval a config value) we had this value set to 3 seconds and statically set the subdomaincenter timeout to 2 seconds in our fork (unfortunately github doesn't show the git history correctly before i synced so these changes aren't in the history anymore). our investigations showed, that bbot ran significantly faster with these settings which no negative impact on results. now we have both values set to 3 seconds (as they are now based on the same config value) and this works great for us. if we go back to letting subdomaincenter timeout for over 30 seconds, this will have a significant negative impact on the length of our bbot runs again.

may i ask if you have done extensive testing yourself which confirms a timeout of x+30 seconds is the better option?

[TheTechromancer]

Okay, yeah there are certain modules like subdomain center that tend to take longer than the default timeout to return, especially for big targets.

The reason we added 30 seconds there was because when the module was written, subdomaincenter was timing out. Raising the timeout allowed it to return successfully.

It's possible that was a temporary slowdown and that can now be removed.

EDIT: It seems the API response is much quicker now. We can remove the added timeout.

[indeed404]

cool, i applied the change as described, good to go from my side

[indeed404]

i also just realized i had a brain fart and the http_timeout has nothing to do with the 429_sleep_interval i was talking about, so the longer timeout here wouldn't even matter and my comment above doesn't make sense, sorry about that :D

[TheTechromancer]

Thanks and lastly we can remove the timeout altogether since it will respect the defaults. So the line becomes:

response = await self.api_request(url)

[indeed404]

done