This is our first release along with our submission of the paper. We document a list of artifacts and TODOs here. Please see our Appendix B in the paper for details. Please read this before you use ReaSCAN. Here is a short summary of that section in bullet points:
- Non-comprehensive Linguistic Structures: Commands from ReaSCAN follow a specific linguistic template and are non-comprehensive in covering all linguistic structures.
- Non-comprehensive Distractors: ReaSCAN is not able to cover all possible distractors to make sure every part of the command is necessary to resolve the referring expression.
- Shapes and Relations Biases: The frequency distributions of shapes and relations may be biased due to the generation program.
- Self-exclusiveness: We assume every object mention in the command matches a unique object in the world.
- Other Induced Artifacts: We also discuss frequency distributions of verbs, adverbs, agent facing directions, agent-target relative directions, etc.
Future TODOs:
- More interesting splits: We plan to release more splits testing on different aspects of neural models.
- Abstract Reasoning Task: Currently, it is still in the proof-of-concept stage. We plan to release a new set containing abstract reasoning tasks alone. Potentially, instead of in grid worlds, we plan to release an image-based version that aligns well with the Abstract Reasoning Corpus.