Conformal Temporal Logic Planning using Large Language Models: Knowing When to Do What and When to Ask for Help

Conformal Temporal Logic Planning using Large Language Models:
Knowing When to Do What and When to Ask for Help

Paper

Code (coming soon)

Demo (coming soon)

Abstract

This project addresses a new motion planning problem for mobile robots tasked with accomplishing multiple high-level sub-tasks, expressed using natural language (NL), in a temporal and logical order. To formally define such missions, we leverage LTL defined over NL-based atomic predicates modeling the considered NL-based sub-tasks. This is contrast to related planning approaches that define LTL tasks over atomic predicates capturing desired low-level system configurations. Our goal is to design robot plans that satisfy LTL tasks defined over NL-based atomic propositions. A novel technical challenge arising in this setup lies in reasoning about correctness of a robot plan with respect to such LTL-encoded tasks. To address this problem, we propose HERACLEs, a hierarchical conformal natural language planner, that relies on a novel integration of existing tools that include (i) automata theory to determine the NL-specified sub-task the robot should accomplish next to make mission progress; (ii) Large Language Models to design robot plans satisfying these sub-tasks; and (iii) conformal prediction to reason probabilistically about the correctness of the designed plans and mission satisfaction and to determine if external assistance is required. We provide extensive comparative experiments on mobile manipulation tasks. Paper can be found here.

Framework Structure

More details coming soon.

Experiment Video 1: Reacting to unknown environments

Deliver either Coke #1 or Coke #2 to X3

Experiment Video 2: Reacting to ambiguous NL instructions

Bring a drink to LC and bring another drink to LC

Experiment Video 3: Demonstration on complex mobile manipulation tasks

Eventually deliver Coke #1, Coke #2, and a water bottle to LC. Do not deliver either Coke until the bottle of water is delivered

Experiment Video 4: Baseline Demonstration

Comparison Model

Citation

[arxiv version]

@misc{wang2023conformal,
    title={Conformal Temporal Logic Planning using Large Language Models: Knowing When to Do What and When to Ask for Help}, 
    author={Jun Wang and Jiaming Tong and Kaiyuan Tan and Yevgeniy Vorobeychik and Yiannis Kantaros},
    year={2023},
    eprint={2309.10092},
    archivePrefix={arXiv},
    primaryClass={cs.RO}
}

Acknowledgements

We thank Samarth Kalluraya for useful feedback on the project. The website template is from Code as Policies.