Make request async to hackerearth api so that could achieve better concurrency

[jinankjain]

No description provided.

[matthewwo]

Since sources are compiled file by file now, I don't think there is a need to use HackerEarth asynchronously. In my opinion, the current method (using sync ajax calls without callback url) is enough to handle the usage.

[sahildua2305]

@jinankjain I agree that making requests async will help in achieving better concurrency for the hackIDE.
Though I am yet to find a way to generate submission ID every time a request is made so as to keep track of responses from HackerEarth API. Please feel free to contribute if you find a way.

[jinankjain]

@sahildua2305 Every time when you do async function call to hackerearth api you will receive an code_id use that as submission ID and store that in DB to keep track of submission id.

[arijeetmkh]

@sahildua2305 Couldn't we use some kind of asynchronous task processing library out there (celery, rq) and use their task ids as the unique submission id. Celery's result backend will store this task id along with information about its run state. We could then poll and query if a running task is successful or not still pending.

[sahildua2305]

@arijeetmkh Yes, that can be done. I have never worked with Celery. I will have to look into it. Would you like to give it a shot?

[the-realtom]

@sahildua2305 what are the benefits of an async request to the hackerearth api? I'm assuming the only api calls are to the /compile and /run endpoints only. Is it the fact that multiple sync calls will cause noticeable blocking when the usage is high? I wouldn't mind taking this up btw.

[sahildua2305]

Hey @the-realtom, having async requests to the API will make the application more reliable as that will also reduce the load on HackerEarth's API. I talked to HackerEarth engineers and they suggested me to have async requests.

Would love to see your PR 👍

[PansulBhatt]

@sahildua2305 Your calls already look like they are asynchronous. Is there something that I am missing here. I was trying to contribute but while reviewing found that the call to compile (custom.js line:235) I found that this was already async.

[the-realtom]

@PansulBhatt You were looking at the client Javascript calls to the Django server. The requests from the server to the hackerearth API compile/run endpoints is what the issue is referring to.

[sahildua2305]

Exactly, @the-realtom 👍

[MasteMind]

It says i need the client token for the ide to show up, when running in local. How can i get that?

[sahildua2305]

@MasteMind You can get it here.

[vinibiso]

Hello! I'm looking to contibute, I think I can tackle this. @the-realtom are already on it? If so do you need any help? I'm here from Up For Grabs.

Also I have to say this would technically be my first PR to a Open Source project so if you don't want a first timer with the whole PR thing I understand. I have worked with django for 3 years now if that's worth anything 😛.

[sahildua2305]

@vinibiso absolutely welcome here! Thanks for showing interest in tackling this issue. Please feel free to open a PR once you have something.

[vinibiso]

Hey @sahildua2305 sorry to bother but, while trying to run collect static, or straight up runserver it gives me the following:

No module named whitenoise.django

That's after creating my virtualenv and installing all the requirements.
I also checked just to see and it seems to be installed. Have you run into this issue before?

collecting whitenoise==2.0.6
Using cached whitenoise-2.0.6-py2.py3-none-any.whl
Installing collected packages: whitenoise
Successfully installed whitenoise-2.0.6

[sahildua2305]

@vinibiso hi!

No, I have not faced this issue. Did you check about it online?

[vinibiso]

Never mind. It seems as tho my virtual env skip a couple of key packages from the requirements for some reason.

Now I just getting a Bad Request. But I'll figure it out. Ty!

[vinibiso]

Just to leave a little bit of a progress report here.

I manager to figure out how to make the request assync.
However, like mentioned before, we will need celery and a broker, in my case I'm using Redis.

I have it all setup and configured now I just have to implement the two request functions as tasks and I believe it will be done.

[vinibiso]

@sahildua2305 I think it's done. It's gonna be a little tricky to test however because of the new requirements and as well as the fact that it only get's client secret when debug is not on.

While doing it and taking a look at the API, do we really need django to do this? Couldn't we just use JS to request? Is it because of client_secret?

[sahildua2305]

@vinibiso You're right. Plus I am not 100% convinced that I want this feature to be there as the API has been pretty much stable for some time now.

[vinibiso]

Hey @sahil865gupta I think it's done, if you wanna take a look.

Right. It makes sense. Honestly looking at how celery works, I don't know if it's actually worth the whole redis, celery setup on the server if not much will change.

I made the changes more as a personal challenge because I had never worked with Celery before.

A good idea would be to try and make some tests and compare before and after the changes maybe by putting a bunch of code so that the API hangs a little bit more and try and run a bunch of another instances. In my case, when I put it to run and opened another instance it was instance no hanging.

[sahildua2305]

@vinibiso it's been 3 years and I am convinced that we should switch to async version of HackerEarth API. Are you still up for taking this on?