+
+ Frequently Asked Questions (FAQ)
+ Here is an always-in-WIP list of popular questions and answers that may help you resolve
+ your
+ issue without having to contact us. Please go through the list below before reaching out to
+ us.
+
Frequently Asked Questions (FAQ)
+ Here is an always-in-WIP list of popular questions and answers that may help you resolve
+ your
+ issue without having to contact us. Please go through the list below before reaching out to
+ us.
+ For all users:
+
+ - What information should I include in my message?
+ For a prompt resolution of your issue, please include the following details:
+
+ - Challenge URL
+ - Challenge title
+ - Participant team name
+ - Submission PK if relevant; Submission file URL also works.
+ - Any other relevant information that will help us understand the issue
+
+
+ - What happens if I do not provide enough detail?
+ Messages lacking sufficient detail may not be addressed, as we receive
+ numerous
+ messages each day.
+
+
+ For participants:
+
+ - I am having trouble with accessing the data for a challenge.
+ If you are having issues with getting data for a particular challenge, please reach
+ out
+ to the hosts directly.
+ - I am unable to submit because the challenge has a `.edu`
+ requirement.
+
+ Please reach out to the hosts directly for assistance.
+
+ - I see a "Verify your email to continue" message despite having
+ verified.
+ If you encounter this message, please try using incognito mode or clear your browser
+ cache.
+ - Submission is stuck at submitted or running.
+ Please provide the challenge URL and submission PK in the contact message below.
+
+ - My team for the CARLA Autonomous Driving Challenge 2.0 (or a similar
+ challenge)
+ has not been accepted yet.
+ What should I do?
+ Please reach out to the hosts. This approval is managed at the host end.
+ - I need to make changes (add/remove) to my participant team.
+ This is not possible at the moment. We are looking into implementing a feature where
+ you
+ can unparticipate in a
+ challenge if there are no submissions made for a challenge. At the moment, this is
+ not supported, however.
+
+ For challenge hosts:
+
+ - I need my challenge approved by the admin.
+ Please create the challenge and make successful submissions for each phase. Then
+ click
+ on "Request for Approval" button on the challenge page. Any request sent through
+ contact form will not be entertained.
Also, please do not spam us by
+ repeatedly submitting the request using the button. We have added that feature for
+ your
+ convenience, please use it wisely.
+ - I am unable to send approval request for my challenge. I get an error with
+ "Following challenge phases have missing submissions".
+ This is expected. If you are facing this error, you need to make a successful
+ submission (which reaches 'Finished' status) for each challenge phase.
+ This is to prevent our resources from being wasted with incorrect evaluation
+ scripts, and to ensure smoothness for participants as well.
+ - I am having trouble with downloading submissions from the "All Submissions"
+ page. What should I do?
+ This happens when there are too many submissions for the backend to compile. Please
+ reach out to us with you challenge PK and we can download multiple submission files
+ and
+ share. However, please refrain from re-using the same challenge every year as that
+ will
+ keep worsening the issue.
+ - I accidentally deleted my account. How can I restore it?
+ Please make sure to be careful when dealing with these things. Send us a contact
+ message
+ with your user asking us to reactivate your account.
+ - I found inconsistency with the documentation. What should I do?
+ We appreciate any reports with issues in our documentation, please reach out to us
+ and
+ explain the problem clearly. We highly appreciate open-source contributions, please open a PR if you can.
+ - Is it possible to access the participants list for my
+ challenge?
+ Yes, please use the Analytics Dashboard to
+ download the participant team details.
+ - I am a new challenge host wanting to create a challenge. I would like to
+ meet
+ with the team to discuss my requirements.
+ Thanks for using EvalAI. We have a detailed documentation on how to host challenges
+ here: Host a
+ challenge.
+ We also have a starter template here: EvalAI
+ Starters
+ Template.
+
+ Our recommendation is to go through the documentation and try to host a challenge
+ and
+ reach out to us with specific issues or questions.
+ Despite that, if you still feel the need, please reach out to us with your
+ requirements
+ and we will schedule a meeting
+ with you.
+ A few things to note:
+
+ - We are non-profit, i.e. we do not charge the hosts for hosting challenges,
+ although that may change in the future.
+ - We have a privacy policy, please read it here: Privacy
+ Policy.
+ - We are a small team, and while we do consider new features which may be
+ general
+ enough, we may not be able to accommodate all requests.
+ - If you are trying to host a docker-based challenge, then we request you to
+ please arrange your own AWS account.
+ - We request you to please use remote evaluation servers (your own) if
+ possible as
+ that helps us save costs and sustain.
+
+
+ - I am trying to create a challenge with remote evaluation. I need challenge
+ primary key and queue details.
+ Please enable `remote_evaluation: true` in your challenge configuration. Then, go to
+ Manage Tab of your challenge to access the details.
+ - I need to make all submissions for my challenge public.
+ Unfortunately, we do not have support for this as of now. We are working on it and
+ will
+ update here once it is available.
+ Also, please note to ask such questions before starting the challenge. We are
+ usually unable to change such settings once the challenge has started and
+ submissions
+ are in.
+ - I am hosting a docker-based (code-upload/static-code-upload) challenge. I
+ want to see the logs on my pods
+ which stay running/submitted.
+ To view logs on your pods, you may use the following commands. Please look into
+ kubectl/AWS EKS docs for further info:
+
+ 1. Export required variables on your AWS CloudShell:
+ export AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID}
+ export AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY}
+ export CHALLENGE_PK=${CHALLENGE_PK}
+ export CLUSTER_NAME=${CLUSTER_NAME}
+ export ACCOUNT_ID=${ACCOUNT_ID}
+ export AUTH_TOKEN=${AUTH_TOKEN}
+
+ 2. Set eks configuration on AWS:
+ aws eks --region $AWS_DEFAULT_REGION update-kubeconfig --name $CLUSTER_NAME
+ aws configure set aws_access_key_id $AWS_ACCESS_KEY_ID
+
+ 3. Commands to get different entities on EKS:
+ kubectl get jobs
+ kubectl get nodes
+ kubectl get pods
+ kubectl get cnp
+
+ 4. Commands to get logs for a specific pod:
+ kubectl logs submission-293486-8kfwt
+
+ 5. Commands to describe a particular pod:
+ kubectl describe pods submission-252973-4vx28
+
+ 6. Commands to delete a pod:
+ kubectl delete pods submission-295910-9lb69
+
+
+
+ For a prompt resolution of your issue, please include the following details: +
-
+
- Challenge URL +
- Challenge title +
- Participant team name +
- Submission PK if relevant; Submission file URL also works. +
- Any other relevant information that will help us understand the issue +
+ Messages lacking sufficient detail may not be addressed, as we receive + numerous + messages each day. +
+ If you are having issues with getting data for a particular challenge, please reach + out + to the hosts directly.
+ Please reach out to the hosts directly for assistance. +
+ If you encounter this message, please try using incognito mode or clear your browser + cache.
+ Please provide the challenge URL and submission PK in the contact message below. +
+ Please reach out to the hosts. This approval is managed at the host end.
+ This is not possible at the moment. We are looking into implementing a feature where + you + can unparticipate in a + challenge if there are no submissions made for a challenge. At the moment, this is + not supported, however.
+ Please create the challenge and make successful submissions for each phase. Then + click + on "Request for Approval" button on the challenge page. Any request sent through + contact form will not be entertained.
Also, please do not spam us by + repeatedly submitting the request using the button. We have added that feature for + your + convenience, please use it wisely.
+ This is expected. If you are facing this error, you need to make a successful + submission (which reaches 'Finished' status) for each challenge phase. + This is to prevent our resources from being wasted with incorrect evaluation + scripts, and to ensure smoothness for participants as well.
+ This happens when there are too many submissions for the backend to compile. Please + reach out to us with you challenge PK and we can download multiple submission files + and + share. However, please refrain from re-using the same challenge every year as that + will + keep worsening the issue.
+ Please make sure to be careful when dealing with these things. Send us a contact + message + with your user asking us to reactivate your account.
+ We appreciate any reports with issues in our documentation, please reach out to us + and + explain the problem clearly. We highly appreciate open-source contributions, please open a PR if you can.
+ Yes, please use the Analytics Dashboard to + download the participant team details.
+ Thanks for using EvalAI. We have a detailed documentation on how to host challenges + here: Host a + challenge. + We also have a starter template here: EvalAI + Starters + Template. +
+ Our recommendation is to go through the documentation and try to host a challenge + and + reach out to us with specific issues or questions.
+ Despite that, if you still feel the need, please reach out to us with your + requirements + and we will schedule a meeting + with you. + A few things to note: +
-
+
- We are non-profit, i.e. we do not charge the hosts for hosting challenges, + although that may change in the future. +
- We have a privacy policy, please read it here: Privacy + Policy. +
- We are a small team, and while we do consider new features which may be + general + enough, we may not be able to accommodate all requests. +
- If you are trying to host a docker-based challenge, then we request you to + please arrange your own AWS account. +
- We request you to please use remote evaluation servers (your own) if + possible as + that helps us save costs and sustain. +
+ Please enable `remote_evaluation: true` in your challenge configuration. Then, go to + Manage Tab of your challenge to access the details.
+ Unfortunately, we do not have support for this as of now. We are working on it and + will + update here once it is available. + Also, please note to ask such questions before starting the challenge. We are + usually unable to change such settings once the challenge has started and + submissions + are in.
+ To view logs on your pods, you may use the following commands. Please look into + kubectl/AWS EKS docs for further info: + +
1. Export required variables on your AWS CloudShell:
+export AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID}
+ export AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY}
+ export CHALLENGE_PK=${CHALLENGE_PK}
+ export CLUSTER_NAME=${CLUSTER_NAME}
+ export ACCOUNT_ID=${ACCOUNT_ID}
+ export AUTH_TOKEN=${AUTH_TOKEN}
+
+ 2. Set eks configuration on AWS:
+aws eks --region $AWS_DEFAULT_REGION update-kubeconfig --name $CLUSTER_NAME
+ aws configure set aws_access_key_id $AWS_ACCESS_KEY_ID
+
+ 3. Commands to get different entities on EKS:
+kubectl get jobs
+ kubectl get nodes
+ kubectl get pods
+ kubectl get cnp
+
+ 4. Commands to get logs for a specific pod:
+kubectl logs submission-293486-8kfwt
+
+ 5. Commands to describe a particular pod:
+kubectl describe pods submission-252973-4vx28
+
+ 6. Commands to delete a pod:
+kubectl delete pods submission-295910-9lb69
+