AI RESEARCH PAPER ANALYSIS

The system assumes a known semantic layout (rooms and surfaces) from a semantic SLAM system, but only partial information about objects is available. The environment contains manipulable objects, rooms, and surfaces. For each task-relevant object, its room, surface, and pose are denoted, with categorical and continuous beliefs. Building on SS-Replan, PO-TAMP is modeled as a hybrid discrete-continuous belief-space stochastic shortest-path problem. CoCo-TAMP takes a TAMP specification with objects, predicates, initial literals, goal literals, and parameterized actions. Throughout planning and execution, CoCo-TAMP maintains beliefs for each task-relevant object and triggers replanning on execution failures. The underlying TAMP planner is PDDLStream, coupling symbolic actions with streams for continuous variables. The 'detect' action is parameterized by object, surface, room, pose, base configuration, head configuration, and head trajectory. Its cost is inversely proportional to the belief over the object's state, steering the planner toward informative views. The goal is to improve efficiency by using LLM-driven initializations and object co-location models, measured by cumulative planning and execution time and number of replanning iterations.

CoCo-TAMP is an integrated planning and execution framework. It uses LLMs to generate initial prior beliefs over rooms and surfaces by formulating the most likely room and surface selection as a multiple-choice question answering (MCQA) task. Object state estimation involves computing the posterior probability of an object's semantic location and pose using a recursive Bayesian filtering framework. The belief distribution is factored into three conditional terms for room, surface, and pose. Observation models include both continuous and categorical components, using visibility values (Ur,t, Us,t) to account for partial observability. The visibility-aware observation model is defined for cases where an object is placed or not placed in a location, with false negative (Pfn) and false positive (Pfp) rates. A key component is the co-location model, which leverages LLM embeddings to capture semantic similarity between objects (sim(j,k) ∈ [-1,1]). This model interpolates between uniform distribution and Kronecker delta based on similarity, increasing belief for similar objects found together and decreasing for dissimilar ones. CoCo-TAMP also uses LLMs to decide whether to enable the co-location model based on observed object semantics. Object pose is estimated using a particle filter with observation models for visible and non-visible cases, adjusting particle weights based on distance and semantic similarity for non-visible objects when similar ones are observed.

CoCo-TAMP was evaluated in household environments through simulation and real-world experiments, measuring cumulative planning and execution time and number of replanning iterations. Large-scale simulations used the Housekeep dataset for common-sense object placements across diverse layouts (4 rooms/8 surfaces to 8 rooms/32 surfaces), including occluding obstacles. Six variants were compared: Baseline, Co-Model, LLM generated belief update (LGBU), MCQA, MCQA with Co-Model, and MCQA with LGBU, differing in initial belief generation and co-location model use. GPT-4o consistently outperformed other LLMs (Llama-3.1-8B, Mistral-7B, Deepseek-llm-7b-chat) for MCQA. Real-world experiments used a Toyota HSR robot in a mock apartment (living room, kitchen) with three surfaces, for a task like relocating an apple. The setup included other objects (banana, screwdriver) and occlusions (cracker/cereal boxes) to test co-location and partial observability. Results showed notable reductions in planning/execution time and replanning iterations for MCQA with Co-Model, validating LLM-guided priors and co-location models. LGBU showed less robustness, failing in adversarial settings where Bayesian methods succeeded.

Belief-space planning and object search literature are closely related to CoCo-TAMP. Belief-space TAMP planners typically model uncertainty at task and motion levels and use approximate solutions for POMDPs, often requiring replanning. CoCo-TAMP enhances this efficiency by using LLMs to shape beliefs. Object search methods, while tackling POMDPs for object localization, do not fully address TAMP problems as they lack task-level symbolic components. LLMs have been increasingly integrated into TAMP, replacing engineered components for constraints, plan computation, or spatial relationships. Some approaches translate natural language to PDDL representations. CoCo-TAMP specifically leverages LLMs to query common-sense knowledge about object placement and co-location, distinguishing it from prior work focused on planning or constraint generation.

CoCo-TAMP is a belief-space planning and execution framework that integrates LLMs for common-sense reasoning into PO-TAMP. It encodes two key types of common-sense knowledge: (1) objects are more likely to be found in specific locations, and (2) semantically similar objects tend to be co-located. While not yet evaluated in non-household domains (e.g., factories, hospitals) or scenarios with unavailable environment layout information, these are future research directions. The framework holds promise for making belief-space planning practical under partial observability.