Rate limiting Batch.libretexts.org

[ymdahi]

Problem

According to MindTouch, there is a 5000 calls per 5 minute threshold on their API service.
batch.libretexts.org (159.65.100.70) seems to be exceeding this threshold, and is therefore being blocked automatically.

This may/may not be impacting other services, such as transclusions (#153).

Context

Batch hosts nodePrint, a microservice that generates PDFs of books/pages within LibreTexts.

We were notified recently (Sept 28) by MindTouch devs that this server had been blocked for exceeding the limit.

Possible Solution(s)

• Limit the number of calls
• Use batch/queuing to better manage calls

Next Steps

• Document nodePrint as best as we can
• Find a way to limit calls

[ymdahi]

@Miniland1333 : any comments would be helpful.

[Miniland1333]

I am unsure what changed that started causing this issue since I haven't touched that code in a while and they haven't complained about the load being that high.

To do a baseline test, I would recommend turning off the concurrency settings here:

Libretext/nodePrint/nodePrint.js

Line 72 in 530a67e

const numThreads = kubernetesServiceHost ? 6 : 4;

Set numThreads and concurrentTexts to 1. Hopefully, this should cause a proportional decrease in Mindtouch's traffic and you should check with them to confirm whether or not this occurs. If this doesn't fix the issue, then we truly have a deep mystery on our hands.

[ymdahi]

TODO:

• check in with MT to see if batch.LT has been blocked since the changes in SSL issue with MindTouch Transclusion Script #153
• implement changes in line72 in nodePrint.js as per Henry's comments

[Miniland1333]

FYI it looks like batching for Chemistry has been broken for some time. I think you need to adjust schedule.js in order to get this working again with the lower rate.

https://chem.libretexts.org/Under_Construction/Development_Details/Admin_Tools/API_Dashboard

[Miniland1333]

Duplicate of #169