Novelty Adaptation Through Hybrid Large Language Model (LLM)-Symbolic Planning and LLM-guided Reinforcement Learning

Hong Lu¹, Pierrick Lorang^1,2, Timothy R. Duggan¹, Jivko Sinapov¹, Matthias Scheutz¹

¹Department of Computer Science, Tufts University, Medford, MA, USA ²Austrian Institute of Technology GmbH, Vienna, Austria

Abstract

In dynamic open-world environments, autonomous agents often encounter novelties that hinder their ability to find plans to achieve their goals. Specifically, traditional symbolic planners fail to generate plans when the robot's planning domain lacks the operators that enable it to interact appropriately with novel objects in the environment. We propose a neuro-symbolic architecture that integrates symbolic planning, reinforcement learning, and a large language model (LLM) to learn how to handle novel objects. In particular, we leverage the common sense reasoning capability of the LLM to identify missing operators, generate plans with the symbolic AI planner, and write reward functions to guide the reinforcement learning agent in learning control policies for newly identified operators. Our method outperforms the state-of-the-art methods in operator discovery as well as operator learning in continuous robotic domains.

Novelties Injected

Kitchen scene figure — **Kitchen.** A novel lid blocks access to the pot, requiring a new pick-up-lid-from-pot operator. This is the easiest discovery setting because random exploration can displace the lid and reach a state from which the existing planner can finish the task.

Coffee box figure — **Coffee-Box.** The coffee pod appears inside a novel box. Existing skills retrieve objects from the tabletop, so the planner must add a pick-up-from-open-box operator and the robot must learn to retrieve the pod from the container.

Nut assembly figure — **Nut Assembly.** A square nut is placed on a novel round peg. Tabletop pick-up operators cannot handle this starting state; the missing pick-up-nut-from-peg operator enables the robot to remove the nut before continuing the assembly task.

Coffee drawer figure — **Coffee-Drawer.** The pod is hidden inside a closed drawer. This is the hardest discovery setting: the robot needs both open-drawer and pick-up-from-open-drawer operators, and random exploration is unlikely to discover the required sequence.

Hybrid LLM Symbolic Planner vs Operator Discovery (OD)

Domain	Hybrid Successes	OD Successes	Hybrid Time	OD Time
Kitchen	10/10	10/10	75.19 sec	85.51 sec
Nut Assembly	10/10	0/10	120.76 sec	>7h
Coffee (box)	10/10	0/10	45.26 sec	>7h
Coffee (drawer)	7/10	0/10	681.83 sec	>7h

Identifying missing operators. Across ten runs per domain, the hybrid planner succeeds in every Kitchen, Nut Assembly, and Coffee-Box run and seven Coffee-Drawer runs. Operator Discovery (OD) succeeds only in Kitchen. OD searches through reinforcement learning, with each run capped at one million timesteps; the hybrid planner instead uses LLM suggestions and symbolic search to identify the missing operators.

LLM-Guided Subgoal Learning

Success and progress rate bar plots — **Success and progress across learned operators.** LLM-Guided (LG) sub-goal learning is compared with LEAGUE-Sparse (LS), which rewards completion of all operator effects, and Reward Machine (RM), which provides sub-goal bonuses. Success measures completion of the operator; progress is the percentage of sub-goals achieved. Results are averaged over ten seeds, with error bars showing the standard error of the mean. LLM-generated reward shaping helps guide exploration between sub-goals, producing higher average success and progress than the baselines across the operators shown.

Candidate reward function evaluation curves showing success rate over training timesteps — **Reward function candidates during training.** Each panel tracks sub-goal success rate over training timesteps. The LLM generates three reward shaping candidates, with semantically identical candidates filtered out, so some sub-goals have fewer alternatives. The worst-performing candidate is eliminated at intervals of 200,000 timesteps. At the end of each phase, the best candidate and its policy are retained; the next phase reuses the selected reward logic for earlier sub-goals while exploring new candidates for the current one.

BibTeX

@misc{lu2026noveltyadaptationhybridlarge, title={Novelty Adaptation Through Hybrid Large Language Model (LLM)-Symbolic Planning and LLM-guided Reinforcement Learning}, author={Hong Lu and Pierrick Lorang and Timothy R. Duggan and Jivko Sinapov and Matthias Scheutz}, year={2026}, eprint={2603.11351}, archivePrefix={arXiv}, primaryClass={cs.RO}, url={https://arxiv.org/abs/2603.11351}, }