Merge "Planning" and "Grounding" data

[jianguoz]

Thanks for the valuable work!

I noticed that there are distinct models for both planning and grounding tasks, specifically the lumos_unified_plan_iterative and lumos_unified_ground_iterative

I'm interested in knowing whether it's feasible to combine the datasets for planning and grounding and then use them to train a single model that can perform both functions. If this approach is viable, what specific adjustments would be necessary? And if it's not advisable, could you shed some light on why that might be the case?

[WadeYin9712]

Hi Jianguo,

Thanks so much for your interest in Lumos!

It's totally to doable to combine the planning and grounding datasets. Suppose one of the conversational planning module annotations is like (UP_1, AP_1), (UP_2, AP_2), ... (UP_n, AP_n). The corresponding grounding module annotation is (UG_1, AG_1), (UG_2, AG_2), ..., (UG_n, AG_n). You can easily combine them into the way like (UP_1, AP_1, UG_1, AG_1), (UP_2, AP_2, UG_2, AG_2), ... , (UP_n, AP_n, UG_n, AG_n). Each parenthesis pair (UP_m, AP_m, UG_m, AG_m) indicates a complete planning and grounding process for a certain subgoal. It doesn't need any more additional change to our annotations.

Please let me know if you have more questions about it. Thank you!