AI Research Analysis
Semantics in Robotics: Environmental Data Can't Yield Conventions of Human Behaviour
The word semantics, in robotics and AI, has no canonical definition. It usually serves to denote additional data provided to autonomous agents to aid HRI. Most researchers seem, implicitly, to understand that such data cannot simply be extracted from environmental data. I try to make explicit why this is so and argue that so-called semantics are best understood as data comprised of conventions of human behaviour. This includes labels, most obviously, but also places, ontologies, and affordances. Object affordances are especially problematic because they require not only semantics that are not in the environmental data (conventions of object use) but also an understanding of physics and object combinations that would, if achieved, constitute artificial superintelligence.
— JAMIE FREESTONE, The Australian National University, Canberra, ACT, Australia
Executive Impact & Key Metrics
Understanding the foundational limitations of semantics in robotics is crucial for pragmatic AI development, focusing resources on achievable goals and avoiding common pitfalls.
0
Total Citations
0
Total Downloads
0
Time to Publication
0
Published Year
Deep Analysis & Enterprise Applications
Select a topic to dive deeper, then explore the specific findings from the research, rebuilt as interactive, enterprise-focused modules.
Defining Semantics for HRI
It is difficult to find a canonical definition of semantics in robotics. The term is used to describe the kind of data or information that an autonomous system would utilise in human-robot interaction (HRI), namely high-level abstract information such as labels, categories, affordances, places and so on. This much is unremarkable, and there are countless teams working on advances in these areas. But some researchers go further. They envision systems that autonomously understand the larger meaning of objects and events in their environment. They do not need to have the semantics provided, for they will infer or extract it directly from raw data. This kind of semantics is always in the near future. It will be central to the promised future of humanoid robots, which attracted $1.6 billion in investment in 2024 alone. Yet-to-be-invented robots, deployed in homes and offices, will perceive their environments and, unsupervised, will know the possible affordances of objects, the uses of a room and the intended meaning of utterances from their human interlocutors. They will do so without being linked to pre-existing databases of high-level information and without exhaustively observing and then imitating humans.
Environmental Data Cannot Yield Semantics
Environmental data is defined as any data that could be extracted from the environment, largely overlapping with sensory data. However, semantics (high-level, abstract, human-oriented information like labels, places, affordances, and ontologies) cannot be solely extracted from raw environmental data. These semantic data map environmental observations to human actions and are dependent on conventions of human behaviour. While environmental data can be sensed, the underlying *history* or *established convention* of behavior is not inherent in the data itself. A system would need to observe multiple trials or be explicitly provided with this information. Therefore, human-free environmental data cannot, by itself, yield semantics.
Conventions as the Foundation of Semantics
Conventions provide a powerful framework for understanding how agents attribute meaning. This attribution does not require objects or events to possess intrinsic meaning, nor does it require agents' internal states to possess intrinsic meaning. Instead, it relies on behavioral regularities, not internal representations. The essence of communication and social interaction is basing one's behavior on the behavior of other agents. Conventions emerge when multiple agents interact with the same objects or events and derive benefit from coordinating their responses, leading to alignment of behaviors over time. Thus, semantics are essentially observable regularities in human behavior that must be learned or supplied, not extracted.
Affordances: A Convention-Based Challenge
In robotics, "affordance" is often used synonymously with "function" or "use," encompassing highly specific, multi-object actions. This goes beyond early robo-centric definitions like "liftability." The conventional understanding aligns with the paper's pragmatic view of affordances as an object's actual use by relevant humans, tied to human conventions. The challenge for robots is that these affordances are not latent in an object's physical properties. For a household robot, knowing a mug is used as an ornament, not for drinking, depends on local human custom. Autonomously inferring affordances, especially complex, multi-object combinations, implies solving problems akin to scientific discovery and invention, which are currently beyond AI capabilities.
Enterprise AI Adoption Flow
Observe Environmental Data
→
Identify Human Actions & Regularities
→
Establish Human Conventions (labels, uses, places)
→
Provide Semantic Data to AI System
→
Enable Contextual Human-Robot Interaction (HRI)
Advanced ROI Calculator
Estimate your potential annual savings and hours reclaimed by implementing smarter, convention-aware AI solutions.
Annual Savings Potential
$0
Hours Reclaimed Annually
0
Your AI Implementation Roadmap
A structured approach to integrating sophisticated AI systems that understand and leverage human conventions, avoiding semantic pitfalls.
Discovery & Strategy
Initial consultation, needs assessment, and goal alignment. Define core AI objectives based on real-world human interaction patterns.
Pilot Program Design
Select a high-impact use case, establish success metrics, and design a minimal viable AI solution focusing on explicit convention learning or provision.
Data & Infrastructure Prep
Audit existing data, establish pipelines for environmental *and* conventional semantic data, and ensure scalable infrastructure for HRI.
AI Model Development
Train, test, and refine AI models for the pilot, integrating explicit semantic data sources (e.g., LLMs, rule bases) alongside environmental perception.
Deployment & Optimization
Launch the pilot in a controlled environment, monitor performance in real HRI scenarios, and iteratively optimize based on observed human conventions.
Expansion & Integration
Scale successful pilots across the enterprise, integrate convention-aware AI into broader workflows, and foster internal expertise in managing semantic data for HRI.
Ready to Build Smarter AI?
Avoid the pitfalls of contentful semantics and build AI systems that truly understand human behavior. Schedule a consultation with our experts today.
Ready to Get Started?
Book Your Free Consultation.
Let's Discuss Your AI Strategy!
Lets Discuss Your Needs
Select a Date
Sun
Mon
Tue
Wed
Thu
Fri
Sat