GROOT: GPT-based human-RObOT interface

Abstract

Large Language Models (LLMs) have revolutionized the realm of Natural Language Processing (NLP). Their proficiency in planning and reasoning, combined with code generation capabilities, presents a novel avenue for robotics applications. This work introduces GROOT, a novel speech-based language-agnostic middleware that uses instructions and code examples as grounding principle to leverage Generative Pre-Trained Transformer’s ability to produce code for new and unseen tasks. Unlike methods based on language-conditioned robot policies, GROOT capitalises on auto-regressive code generation inspired by Code-as-Policy (CaP) and ProgPrompt. The aim is to create a human-robot interface using GROOT that enables the embodiment of an LLM to take user instructions like ’move in a square’, ’move 20 cm in front’, ’go to position ’X’ on the grid’ and return policy code based on robot API. In this paper, GROOT was used in a number of experiments to assess its performance to few-shot learning against spatial reasoning, logical reasoning and compound simulated tasks. This work reflects the potential of prompting code-based examples and API-based instructions as a grounding method to integrate large-language models with robotic platforms, envisioning seamless and intuitive human-robot interactions.