Low Success Rate of Lookup Action in HotpotQA Reproduction with ReactReflectAgent

[ShawnMenz]

Hi,

I am currently reproducing the results of HotpotQA. However, I have a question regarding the behavior of the Lookup action on specific keywords.

Most of the time, the Lookup action returns "No Result." I have attached two examples to illustrate this behavior: the first shows a "No Result" for a short sentence, and the second for a single word.

---

"No Result" for a short sentence

---

"No Result" for a single word

---

Experiments

Experiment 1

• react_llm: GPT-3.5-Turbo-0301
• reflect_llm: GPT-3.5-Turbo-0301
• Data: 100 questions from hotpot-qa-distractor-sample.joblib
• Total trials: 5
• Reflexion strategy: REFLEXION
• Agent Type: ReactReflectAgent

During this experiment, there were 9 Lookup actions performed in total, but only 2 returned successful results: one for "Abheri" and another for "cryptocurrency."

Experiment 2

• react_llm: GPT-3.5-Turbo-0301
• reflect_llm: Llama-3.1-8B-instruct
• Data: 100 questions from hotpot-qa-distractor-sample.joblib
• Total trials: 5
• Reflexion strategy: REFLEXION
• Agent Type: ReactReflectAgent

In this experiment, no successful Lookup cases were observed.

[becklabs]

Hello,

The "Lookup" tool is imported from LangChain (see hotpotqa_runs/environment.py):

from langchain import Wikipedia
from langchain.agents.react.base import DocstoreExplorer
explorer = DocstoreExplorer(Wikipedia())
explorer.search(...)
explorer.lookup(...)

From my interpretation, explorer.lookup looks for instances of the query string in the last document searched by exporer.search.

In the examples you provide, the "Lookup" tool seems to be functioning correctly:

1. https://en.wikipedia.org/wiki/Greatest_Hits_Radio_South_Yorkshire has no instance of the string "broadcasting company based in".
2. https://en.wikipedia.org/wiki/Baba_Yaga_(film) has no instance of the string "sisters".

Further, see #43.

@ShawnMenz