iChores:
Intuitive
Collaboration
with
Household
Robots
in
Everyday
Settings
Antonio
Galiza
Cerdeira
Gonzalez
1
,
Peter
Hoenig
2
,
Matthias
Hirschmanner
2
,
Karolina
Zrobek
3
,
Pawel
Gajewski
3
,
Tessa
Pulli
2
,
Petr
Vanc
4
,
Simon
Dratva
4
,
Gabriela
Sejnova
4
,
Radoslav
Skoviera
4
,
Karla
Stepanova
4
,
Michal
Vavrecka
4
,
Markus
Vincze
2
,
Bipin
Indurkhya
1
Abstract
— We
present
iChores
,
an
open-source,
robot-
agnostic
interface
designed
to
facilitate
intuitive
human
in-
teraction
for
household
tasks,
enabling
users
to
direct
robots
through
natural
language
and
deictic
gestures
(pointing)
to
perform
complex
chores,
such
as
tidying
a
room.
The
system
utilizes
a
modular
microservice
architecture
that
integrates
state-of-the-art
vision,
natural
language
understanding
(NLU),
and
reasoning
modules.
We
detail
the
implementation
of
the
framework, provide a public repository for reproducibility, and
present
an
evaluation
of
30
scenarios
of
increasing
complexity.
The
general
architecture
of
iChores
and
its
various
modules
are
briefly
described.
The
implementation
is
publicly
available
through
our
GitHub
project
page,
where
all
modules
are
provided
as
their
own
repositories.
I.
I
NTRODUCTION
Recent
advances
in
object
recognition
and
manipulation
enabled
service
robots
to
perform
chores
such
as
tidying
up
household
environments.
However,
a
significant
barrier
remains:
the
lack
of
intuitive,
flexible
interfaces
that
allow
non-expert
users
to
communicate
intent
naturally.
Existing
human-robot
collaboration
(HRC)
systems
often
rely
on
single
modalities,
such
as
gesture-based
commands
[1]
or
restricted linguistic
templates
[2],
which
result
in
rigid inter-
action
patterns
and
a
narrow
range
of
supported
commands
[3].
Although
the
importance
of
multimodal
communication
is
well-recognized
[4],
[5],
current
research
lacks
holistic
approaches
that
consider
the
entire
robotics
pipeline
from
perception
to
execution.
Although
Chen
et
al.
[4]
demon-
strate
the
potential
of
joint
gesture
and
language
processing,
their
work
does
not
extend
to
downstream
planning
or
task
execution.
[5],
[6]
propose
entire
systems
for
interpreting
multimodal
input,
but
are
limited
to
specific
scenarios,
such
as
cooking
[5]
or
teleoperation
via
video
calls
[6].
Furthermore,
a
significant
branch
of
research
focuses
on
language-guided manipulation [7], [8], yet these methods fre-
quently
simplify
physical
interaction
by
using
basic
motion
primitives
or
targeting
the
geometric
center
of
an
object.
1
Center
for
Cognitive
Science,
Jagiellonian
University,
Krakow,
Poland
{
antonio.gonzalez,bipin.indurkhya
}
@uj.edu.pl
2
Automation
and
Control
Institute
(ACIN),
TU
Wien,
Austria
{
hoenig,hirschmanner,pulli,vincze
}
@acin.tuwien.ac.at
3
AGH
University
of
Krakow,
Poland
{
kzrobek@student,
pgajewski@
}
agh.edu.pl
4
Czech
Institute
of
Informatics,
Robotics,
and
Cyber-
netics,
CTU
in
Prague,
Czech
Republic
{
petr.vanc@,
dratvsim@fel,
radoslav.skoviera@,
karla.stepanova@,
michal.vavrecka@
}
cvut.cz
.
Fig.
1.
iChores
architecture
implemented
in
TIAGo
robot
during
testing.
Upper pictures stand for initial states in testing scenario #1 ( simple gesture
aligned
with
command)
and
scenario
#26
(multiple
gestures
and
compound
verbal
command).
Lower
pictures
depicts
goal
states
after
execution.
Such
simplifications
are
effective
for
simple
shapes
but
fail
when
faced
with
the
complex
geometries
common
in
household
environments.
Furthermore,
these
methods
are
limited
to
object
grasping,
as
pick-and-place
tasks
require
an
accurate
6D
object
pose
[9].
Recent
progress
in
Vision-Language
Models
(VLMs)
has
paved
the
way
for
more
flexible
interactions.
Although
early
context-sensitive
efforts
struggled
with
generalization
[10],
recent
work
by
Lai
et
al.
[11]
successfully
integrated
Large
Language
Models
(LLMs)
with
deictic
posture
for
elderly
care.
However,
such
systems
are
often
tightly
coupled
to
specific
hardware
or
demographics.
In
contrast,
our
pro-
posed
system,
iChores
,
stands
out
as
a
modular
framework
that
is
robot-agnostic,
requiring
only
MoveIt
Action
Server
compatibility.
Unlike
approaches
that
rely
on
specialized
user
interfaces
for
manipulation
[12]
or
precise
short-range
pointing
[13],
iChores
enables
natural
communication
for
a
diverse
range
of
tasks
through
an
open-source
architecture
designed
for
reproducibility.
Our
contributions
are
as
follows.
•
A
robot-agnostic
framework
with
a
highly
modular
microservice
architecture,
allowing
flexible
component
exchange.
•
An integrated Reasoning Module that fuses LLM-based
intent
extraction
with
real-time
3D
gesture
tracking
to
resolve
command
ambiguity.
•
A
fully
open-source
implementation
with
comprehen-
sive
benchmarking
to
promote
reproducibility
and
ex-
This article has been accepted for publication in IEEE Robotics and Automation Practice. This is the author's version which has not been fully edited and
content may change prior to final publication. Citation information: DOI 10.1109/RAP.2026.3676940
This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/
tensibility
in
the
HRC
community.
II.
A
RCHITECTURE
The
iChores
framework
is
designed
with
a
focus
on
high
modularity
and
immediate
deployability.
This
architecture
ensures
that
the
system
remains
flexible
enough
to
be
easily
maintained,
upgraded,
or
customized
for
specific
research
objectives and household tasks. To validate its robot-agnostic
nature,
we
tested
the
modules
across
three
distinct
robotic
platforms: PAL Robotics TIAGo and TIAGo++ (all modules)
and
Toyota’s
HSR
robot
(vision,
gesture
detection,
and
speech
recognition
modules).
As
illustrated
in
Fig.
2,
the
framework
is
divided
into
four modules: i) Vision, ii) Natural Language Understanding,
iii)
Reasoning,
and
iv)
Movement.
This
structured
approach
covers the core competencies required to effectively perform
general
service
tasks.
Human detection
Speech Manager
Human Distance
Gesture Detection
Mediapipe Processor
GDRNet pipeline
Contact-GraspNet /
Pre-annotated grasps
Modality Fusion
LangChain-Based
Reasoning
Speech-to-Text
Whisper
Sentence Instruct Transformer
Qwen2.5/UNCOM
Natural Language Understanding
Motion Module
Head Controller
Movement
Arm-Torso Controller
YCB-V object detection
YOLOv8
YCB-V 6DoF pose estimation
GDR-Net++
estimate_poses.srv
detectron2_service_server.srv
returnGrasps.srv
String
Transcription
Instruct
String
/motion/pick
Pick
/motion/prepare
Prepare
HRICommand
/reasoner/hricommand
estimate_pointing_gesture.srv
Point
Reposition
/human/needed_
head_rotation
Request
Tts
Point
shoulder
elbow
wrist
Vision
Fig. 2.
Overall system architecture including the four key modules: Vision,
Natural
Language
Understanding,
Reasoning,
and
Movement
The
modules
available
within
the
iChores
framework
are
designed
to
understand
and
execute
tasks
with
increasing
complexity
up
to
multiple
gestures
and
compound
language
instructions (see the appendix for details). This offers a solid
foundation
for
performing
open-set
household
tasks
(such
as
organizing,
trash
sorting,
grocery
unpacking,
and
food
preparation).
To
allow
for
flexible
exchange
of
components,
the
sys-
tem
is
structured
around
a
microservice
architecture.
Each
component
provides
specific
services
within
an
independent
containerized
environment,
ensuring
that
the
larger
task
is
performed
through
decoupled
interactions.
This
modularity
allows
the
framework
to
accommodate
modules
with
dif-
ferent
software
requirements
seamlessly,
with
orchestration
managed
via
Docker
Compose.
Communication
is
enabled
primarily
through
ROS
1
Noetic.
To
ensure
compatibility
across
the
diverse
robotic
platforms
and
software
versions
used
in
our
testing,
each
module
operates
within
its
own
isolated
environment.
Sections
III,
IV,
V,
and
VI
explain
the
components
of
the
iChores
framework
in
greater
detail.
III.
V
ISION
To utilize vision for scene understanding and command in-
terpretation,
iChores
addresses two primary challenges. First,
it
employs
MediaPipe’s
BlazePose
to
detect
and
interpret
pointing
gestures,
effectively
disambiguating
verbal
instruc-
tions
(see
section
III-A).
Second,
it
integrates
YOLOv8
[14]
for
2D
recognition
with
GDRNPP
[15]
for
estimating
the
6DoF
pose
and
identifying
the
grasp,
facilitating
complete
understanding
of
objects
(see
section
III-B).
Together,
these
capabilities
allow
the
robot
to
interpret
the
intent
of
the
user
and
perform
precise
physical
manipulations
accurately.
A.
Pointing
Detection
Gestures serve as a fundamental pillar of human communi-
cation,
particularly
in
goal-oriented
tasks.
In
scenarios
such
as
‘pick
up
this
object’,
deictic
gestures
such
as
pointing
play
a
key
role
in
identifying
the
intended
target
object
[16].
We
adopt
BlazePose
[17]
for
real-time
detection
and
tracking
of
body
poses
using
the
MediaPipe
[18]
framework
and
our
Gesture
toolbox
[1].
Because
standard
3D
pose
estimation via GHUM [19] often lacks the reliability required
for
precise
HRI,
we
convert
2D
joint
locations
into
3D
space
by
sampling
the
depth
channel
of
the
robot’s
RGB-D
camera. For smoothing and noise suppression, a median filter
is
applied
to
the
neighborhood
of
detected
joints
within
the
depth
image.
We
define
the
pointing
direction
as
the
vector
that
originates
at
the
shoulder
joint
and
passes
through
the
wrist. Empirical testing demonstrated that this shoulder-wrist
vector
provides
superior
directional
stability
and
accuracy
compared
to
the
traditional
elbow-wrist
vector.
B.
Object
Recognition
and
Grasp
Pose
Estimation
The
iChores
vision module employs a hierarchical pipeline
to facilitate object recognition and spatial understanding. Ob-
jects are detected in the 2D image space using YOLOv8 [14].
Specifically,
we
used
the
L-variant
of
YOLOv8,
trained
on
the
YCB-V
data
set
[20]
with
standard
data
augmentation
and
training
configurations.
To
estimate
the
pose
hypothesis
for
robotic
grasping,
we
implement
the
6D
pose
estimation
algorithm
GDRNPP
[15],
a
combination
of
GDRNet
[21]
with depth refinement. Viable grasp poses were pre-annotated
in
Blender
by
aligning
robot-specific
grippers
with
YCB-V
object
meshes.
At
runtime,
these
candidates
are
transformed
into
the
robot’s
base
frame
and
evaluated
for
kinematic
This article has been accepted for publication in IEEE Robotics and Automation Practice. This is the author's version which has not been fully edited and
content may change prior to final publication. Citation information: DOI 10.1109/RAP.2026.3676940
This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/
feasibility
via
MoveIt!.
Another
possibility
in
our
pipeline
is
the
use
of
Contact-GraspNet
[22],
which
can
be
used
to
expand
system
capabilities
beyond
YCB-V
dataset
objects,
albeit
at
a
higher
computation
cost.
IV.
N
ATURAL
L
ANGUAGE
U
NDERSTANDING
To
reflect
natural
human
communication,
iChores
enables
robots
to
interpret
speech
and
gestures
from
non-expert
users.
This
ensures
that
interaction
remains
intuitive
without
requiring
specialized
technical
knowledge.
A.
Automatic
Speech
Recognition
Speech
recognition
is
performed
using
OpenAI’s
Whisper
model [23]. Incoming audio is continuously monitored using
a
ring
buffer
paired
with
Silero
Voice
Activity
Detection
(VAD)
[24].
Upon
detecting
speech,
the
system
transfers
the
ring buffer content to a dedicated secondary buffer that stores
all
audio
data
from
this
point
forward.
Once
the
utterance
is
finished,
the
complete
buffer
is
processed
by
the
Whisper
model
and
the
resulting
transcript
is
published
as
a
ROS
message
for
downstream
tasks.
B.
Human
intention
estimator
To process unstructured, and often ambiguous, natural lan-
guage instructions, we utilize an NLP module that maps user
commands
into
a
structured
format.
This
module
leverages
the few-shot learning capabilities of Large Language Models
(LLMs)
to
perform
named
entity
recognition
and
intent
classification,
extracting
key
attributes
such
as
the
action,
target object, specific parameters, and visual properties (e.g.,
color).
The
model
is
guided
by
a
few-shot
prompt:
Input:
‘Pick
up
the
red
apple.’
Output: ‘action: pick, object: apple, action parameter: up,
color:
red’
The
module
is
designed
for
easy
switching
between
dif-
ferent LLMs based on available computational resources and
desired
performance.
The
tested
models
include
Qwen2.5-
0.5B-Instruct
[25],
SmolTulu-1.7b-Reinforced
[26],
and
DeepSeek-R1-Distill-Qwen-7B
[27].
V.
R
EASONING
The reasoning module, inspired by our previous Show and
tell
architecture
[28],
converts
the
output
of
the
vision
and
language
understanding
modules
into
a
unified
action
plan.
Since speech alone may not fully convey user intent, gesture
and scene information are also used to clarify commands and
generate
executable
pick-and-place
plans.
We
provide
two
distinct
implementations
of
reasoning
logic
within
the
framework:
one
merges
the
input
from
the
natural
language,
gesture,
and
vision
modules
into
a
unified
command
(Section
V-A);
the
other
uses
a
LLaMA3-based
LangChain
agent
to
generate
plans
from
multimodal
inputs
(Subsection
V-B.1).
Both
are
available
in
the
framework,
allowing
users
to
select
the
reasoning
strategy
that
is
best
suited
to
their
specific
task
requirements.
A.
Modality
fusion
The input processing modules vary in information density,
with
Language
Processing
typically
providing
the
richest
data.
We
use
it
as
the
base
and
enrich
it
with
output
from
other
modules
to
fill
gaps
and
resolve
ambiguities,
such
as
object
position
or
placement
location.
Since
all
inputs
are
probabilistic,
the
fusion
process
is
also
probabilistic.
Command
components
(e.g.,
object,
properties,
locations)
are
combined
using
weighted
fusion,
maximizing
overall
confidence while accounting for differences in each module’s
scoring
scale.
B.
Plan
generation
The
reasoner
module
combines
different
modalities
to
determine
the
action,
target
object,
and
location.
Language
can
either
directly
reference
the
target
(e.g.
‘the
red
bowl’)
or
use
a
more
general
reference
(e.g.
‘a
bowl’)
or
deictic
terms
(e.g.
‘this’
or
‘next
to
this’).
Deictic
references
re-
quire
additional
input
from
the
gesture
module,
where
the
likelihood
of
targets
is
evaluated
based
on
their
distance
from
the
pointing
vector.
In
both
cases,
the
language
often
lacks
precise
position
details,
which
are
provided
by
the
object
detection
module
(GDRNet).
Detected
objects
are
scored
based
on
their
alignment
with
the
language
or
deictic
reference,
with
scores
combined
multiplicatively
when
both
are
used.
This
approach
enables
the
identification
of
the
target using language alone, pointing alone, or a combination
of
both
modalities,
as
detailed
in
[29].
1)
LLM
reasoner:
An
alternative
planning
reasoner
for
high-level
command
understanding
was
implemented,
using
the
capabilities
of
the
llama3.2:70B
large
language
model
[30] within a LangChain agent framework [31]. LLM agents
can
autonomously
determine
and
execute
a
sequence
of
ac-
tions based on given objectives. This adaptability is achieved
through
the
agent’s
ability
to
interact
with
external
tools,
effectively
augmenting
its
knowledge
and
operational
scope.
A
ReAct
prompt
technique
[32]
was
used
to
retrieve
the
execution
plan.
A
separate
chat
with
the
LLM
was
used
to
place
and
release
an
object
where
spatial
reasoning
was
required.
This
sub-prompting
approach
is
designed
to
extract
precise
three-
dimensional
coordinates
for
object
placement,
based
on
the
available
object
list,
their
positions,
and
diameters.
This
process
employs
a
few-shot
learning
template
similar
to
the
one proposed by Wang et al. [33]. The primary agent remains
unaware
of
these
sub-prompts,
maintaining
a
modular
and
decoupled
architecture.
This
design
allows
complex
multi-
step
manipulation
tasks
to
be
decomposed
into
simpler,
manageable
subtasks,
with
the
LLM
acting
as
a
central
coordinator
and
spatial
reasoning
engine.
VI.
M
OVEMENT
Based
on
the
information
from
modality
merging,
the
reasoner
generates
the
most
likely
output
command,
which
is encapsulated in the HRICommand message. This message
includes
details
about
the
desired
actions
and
objects,
as
well
as
probabilistic
information
(a
list
of
alternative
actions
This article has been accepted for publication in IEEE Robotics and Automation Practice. This is the author's version which has not been fully edited and
content may change prior to final publication. Citation information: DOI 10.1109/RAP.2026.3676940
This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/
and objects with their associated probabilities). This message
is
then
received
in
the
Pick
&
Place
module,
where
an
ActionTemplate is created. It contains a predefined sequence
of
operations
(e.g.,
to
grasp
pose,
closing
gripper,
lifting
object).
To
move
the
arm
to
the
desired
pose,
we
use
the
MoveIt!
planner,
as
it
allows
for
collision
avoidance
and
can
be
used
with
different
robots.
1)
Pick:
When
the
Pick
&
Place
module
receives
the
command
to
pick
an
object,
it
chooses
the
object
with
the
highest
probability
of
the
HRICommand
to
be
picked.
Each
object
has
a
list
of
viable
grasp
poses
associated
with
it
(these
can
be
precomputed
as
we
work
with
known
objects).
The
pose
that
is
reachable
and
has
the
lowest
cost
is
selected
as
the
target
pose
and
passed
to
the
planner.
2)
Place:
When
the
place
action
is
required, the place pose is selected by the solution of pointing
gesture
recognition,
or
by
extracting
the
place
pose
from
the
language
instructions
directly.
This
pose
is
then
used
to
fill
out
the
‘Place’
ActionTemplate
and
send
it
to
the
planner.
A.
Evaluation
of
the
system
To
evaluate
iChores
system,
we
designed
scenarios
of
increasing complexity for each component (vision, language,
gestures,
reasoning,
motion),
with
accompanying
module-
specific
metrics.
Vision
scenarios
progress
from
single
to
multiple
YCB-V
objects
with
varying
features,
evaluated
by
object
detection
accuracy
and
6D
pose
error.
Gesture
recog-
nition
scenarios
range
from
single
to
consecutive
pointing
gestures
for
disambiguation,
measured
by
the
pointing
di-
rection
accuracy
and
disambiguation
success
rate.
Language
scenarios cover varying command complexity and ambiguity,
with
metrics
including
precision/recall
of
task
information
extraction
and
the
effectiveness
of
distilled
LLM
instruction
templates. Reasoning evaluation involves tasks fusing vision,
language,
and
gesture
information,
progressing
from
clear
commands
to
ambiguous
ones
requiring
disambiguation,
including multi-step tasks orchestrated by the LLM reasoner,
with
metrics
like
action
plan
generation
success
and
target
object
selection
accuracy.
Movement
scenarios
increase
the
complexity
of
motion
sequences
(move,
pick,
place),
evalu-
ated by the success rate of motion subsets, grasp success, and
placement
accuracy.
The
overall
performance
of
the
system
is
measured
by
the
success
rate
of
scenario
completion.
A
file
containing
the
complete
evaluation
scenarios
is
available
among
the
supplementary
files
for
this
paper.
VII.
E
VALUATION
AND
A
NALYSIS
We
deployed
the
system
on
a
humanoid
robot
(TIAGo++)
for testing. The test setup contained a table with five or more
known and detectable objects. The human operator then tells
the instruction and (if required) points to the target object(s).
The
input
modules
process
the
collected
data,
which
is
then
fused
in
the
reasoner
and
the
message
for
the
movement
module
is
created.
The
robot
then
executes
the
action
with
the
most
probable
target.
The
summarized
results
for
the
30
scenarios
with
increasing
complexity
are
in
Fig.3.
More
evaluation
trials
can
be
seen
in
the
following
videos:
(same
color), (same shape), (object to the left), (object to the right).
Fig.
3.
Success
rate
for
(planning
and
execution)
in
30
scenarios.
Each
scenario
was
repeated
10
times.
The
robot’s
evaluation
revealed
that
the
architecture
is
capable
of
processing
even
the
most
challenging
scenarios;
however,
execution
often
fails
due
to
the
insufficient
accu-
racy
of
the
6-DoF
vision
module
to
estimate
object
poses,
leading to grasp failures. Detailed success rates for individual
modules
are
provided
in
the
supplementary
file.
VIII.
C
ONCLUSIONS
& F
UTURE
W
ORK
We presented the
iChores
framework for household service
robotics
and
its
individual
components.
Our
modular
system
provides great flexibility and a strong foundation for building
state-of-the-art
robotics
solutions
in
the
future.
It
can
be
customized
to
different
robots,
requiring
only
small
changes
to
the
configuration
files.
The
empirical
results
in
complex
scenarios
with
multiple
gesture
and
complex
language
com-
mands
confirm
the
plausibility
of
the
system.
A.
Limitations
To
enhance
the
reproducibility
of
the
results
obtained
during
the
research
that
led
to
the
development
of
the
vision
modules, object
detection and
pose estimation are
fine-tuned
to YCB-V dataset objects. The present vision modules cannot
detect
other
objects
competently
yet.
This
work
focused
on
command
understanding
and
pick-
and-place operations in a tabletop scenario without requiring
the
robot
to
navigate.
This
scope
limitation
prevents
tasks
that need movement in larger environments. Additionally, the
reasoning
system
currently
supports
only
basic
operations
like
picking,
moving,
and
placing
objects.
However,
due
to
the
modular
software
design,
the
system
can
be
easily
extended
with
more
motion
schemes.
B.
Future
Work
Our
framework
is
functional
and
easily
extensible
for
researchers
and
practitioners,
but
we
have
yet
to
evaluate
its
usability
with
end
users.
Extensive
performance
evaluations
of
the
reasoner
and
all
modules
are
underway.
We
have
also
studied how humans teach both other humans and the virtual
TIAGo++
robot
using
gestures
and
language,
but
those
findings
have
yet
to
be
integrated
to
improve
performance.
Additionally, in cases where the system cannot resolve am-
biguous
commands
without
precise
pointing,
a
conversation
module is needed to allow the robot to request clarifications.
This article has been accepted for publication in IEEE Robotics and Automation Practice. This is the author's version which has not been fully edited and
content may change prior to final publication. Citation information: DOI 10.1109/RAP.2026.3676940
This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/
APPENDIX -
iChores
R
EPOSITORIES
AND
E
VALUATION
The
iChores
framework
is
made
publicly
available
through
our
GitHub
project
page:
https://github.
com/iChores-research
,
where
all
modules
are
pro-
vided.
Each
component
of
the
iChores
framework
is
acces-
sible
individually.
The main pipeline iChores
pipeline integrates all currently
available microservices. This pipeline serves as the backbone
of
the
iChores
framework,
orchestrating
the
interaction
be-
tween
various
microservices
using
Docker
compose
to
de-
liver
a
comprehensive
and
efficient
solution.
The
repository
can
be
accessed
in
this
repository.
Vision
Modules
a)
GDRNPP
pipeline:
The
backbone
of
the
vision
modules,
available
in
this
repository
and
present
in
the
iChores
pipeline.
b)
Human
detection
module:
The
human
detection
module searches users by moving the robot head and issuing
voice
instructions
until
the
person
is
at
a
position
in
which
MediaPipe
can
accurately
detect
gestures.
It
is
available
in
this
repository.
c)
Gesture
Detection:
Multiple
gesture
detection
mod-
ules
are
available,
depending
on
the
particular
application.
Currently,
a
pointing
gesture
detection
module
is
available
in
the
iChores
pipeline.
In
the
additional
material,
it
can
be
found
in
iChores
pipeline
skeleton
.
Additionally,
the
Modality
Fusion
reasoner
uses
our
gesture
detection
module
available
in
this
repository.
Natural
Language
Understanding
d)
Automatic
Speech
Recognition:
Automatic
Speech
Recognition
is
performed
using
the
code
available
in
this
repository.
e)
Natural
Language
Processing
Module:
The
natural
language
processing
module
is
responsible
for
transcribing
natural
speech
into
a
structured
command
template.
It
is
available
in
the
Modality
fusion
reasoner’s
repository
and
in
its
own
repository.
Reasoners
f)
Modality
fusion
reasoner:
The
modality
fusion
rea-
soner
generates
the
final
movement
plan
from
the
outputs
of
other
modules.
It
is
available
in
this
repository.
g)
LLM
LangChain
reasoner:
This
reasoner
is
an
al-
ternative
to
the
Modality
fusion
reasoner,
using
LLM
agents
to
determine
and
execute
a
sequence
of
actions
from
user
commands
and
output
of
other
modules.
It
is
available
in
the
branch
llm
reasoner
of
the
iChores
pipeline
repository,
in
this
repository.
Movement
h)
Pick
and
Place
Module:
This
module
is
responsible
for
pick-and-place
execution,
following
the
plans
generated
by
the
reasoner
module.
It
is
available
in
this
repository.
Deployment
Instructions
All
aforementioned
modules
can
be
deployed
individu-
ally,
having
README
files
detailing
how
their
utilization.
Regarding
the
main
iChores
pipeline,
it
is
necessary
to
first
clone
the
repository
using
the
command:
$
git
clone
https://github.com/iChores
-research/iChores_pipeline.git
And
then
move
into
the
repository
using
$
cd
iChores_pipeline
Or,
if
you
already
downloaded
it
from
the
additional
material,
just
cd
it
into
the
iChores
pipeline
folder.
Then
you
need
to
download
and
update
the
sub
modules
with:
$
git
submodule
init
$
git
submodule
update
Now,
it
is
necessary
to
create
a
configuration
file
for
your
system.
Create
a
copy
of
one
of
the
pre-exisiting
parameter
files, e.g.
config/params
realsense.yaml
, and change the cam-
era
intrinsics
to
match
those
of
your
camera,
and
the
names
of
the
color
and
depth
image
topics,
as
well
as
the
image
encoding
and
the
depth
scale
of
the
camera.
Currently,
the
image
size
must
be
640
×
480
.
To
start
the
pipeline,
use
the
command
below,
changing
IP
and
ports
as
necessary:
$
ROS_MASTER_URI
=
http://127.0.0.1:11311
ROS_IP
=127
.0.0.1
DATASET
=
ycbv
CONFIG
=
.yaml
docker-compose
up
C.
System
Evaluation
Scenarios
In
order
to
evaluate
the
capabilities
of
the
iChores
frame-
work,
we
have
proposed
scenarios
with
increasing
complex-
ity
for
vision,
language
understanding,
gesture
detection,
reasoning,
and
movement
modules.
The
list
of
all
scenarios
is
available
in
the
Table
language
gesture
scenarios.xlsx
file
of
the
supplementary
files,
which
describes
them
in
terms
of
objects,
classes,
colors,
language
used,
and
whether
gestures
are
used
in
the
user’s
request.
D.
Detailed
Evaluation
Results
and
Experiment
Videos
The
results
presented
in
Section
VII
only
highlight
the
success
rate
of
the
overall
planning
and
execution
phases,
without
delving
into
the
very
relevant
details
of
the
failure
rate
of
each
individual
module.
This
information
is
essential
to
identify
which
systems
are
current
performance
bottle-
necks
in
the
state
of
the
art.
Thus,
we
included
the
detailed
experiment
results
table
detail
results
evaluation.png
in
the
Additional
Files.
Moreover,
to
better
illustrate
the
performance
of
our
framework,
we
included
videos
Compi-
lationManipulationTiago.mp4
,
LLMAgentPick.mp4
,
Tiago4DvisionTest.mp4
,
Tiago
gesture
manipulation2.mp4
,
Tiago
gesture
navigation.mp4
,
which
show,
respectively,
several
evaluation
scenario
tests,
a
successful
test
with
the
LangChain-based
reasoner,
the
robot
vision
system
working,
a
test
of
the
pointing
gesture
detection
module,
and
gesture-based
robot
navigation
tests.
This article has been accepted for publication in IEEE Robotics and Automation Practice. This is the author's version which has not been fully edited and
content may change prior to final publication. Citation information: DOI 10.1109/RAP.2026.3676940
This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/
ACKNOWLEDGMENT
This research is supported by the National Science Center,
Poland,
under
the
OPUS
call
in
the
Weave
program
under
project
number
2021/43/I/ST6/02489;
by
the
Austrian
Sci-
ence
Fund
(FWF),
under
project
No.
I
6114,
iChores
;
and
by
the
Czech
Science
Foundation
(GA
ˇ
CR)
grant
no.
23-
04080L
and
the
Ministry
of
Education,
Youth
and
Sports
of
the
Czech
Republic
through
e-INFRA
CZ
(ID:90254).
R
EFERENCES
[1]
P. Vanc, J. K. Behrens, K. Stepanova, and V. Hlavac, “Communicating
human intent to a robotic companion by multi-type gesture sentences,”
in
2023
IEEE/RSJ
International
Conference
on
Intelligent
Robots
and
Systems
(IROS)
.
IEEE,
2023,
pp.
9839–9845.
[2]
S.
Stepputtis,
J.
Campbell,
M.
Phielipp,
S.
Lee,
C.
Baral,
and
H. Ben Amor, “Language-conditioned imitation learning for robot ma-
nipulation tasks,”
Advances in Neural Information Processing Systems
,
vol.
33,
pp.
13 139–13 150,
2020.
[3]
I. El Makrini, S. A. Elprama, J. Van den Bergh, B. Vanderborght, A.-J.
Knevels,
C.
I.
Jewell,
F.
Stals,
G.
De
Coppel,
I.
Ravyse,
J.
Potargent
et
al.
,
“Working
with
walt:
How
a
cobot
was
developed
and
inserted
on
an
auto
assembly
line,”
IEEE
Robotics
&
Automation
Magazine
,
vol.
25,
no.
2,
pp.
51–58,
2018.
[4]
Y.
Chen,
Q.
Li,
D.
Kong,
Y.
L.
Kei,
S.-C.
Zhu,
T.
Gao,
Y.
Zhu,
and
S.
Huang,
“Yourefit:
Embodied
reference
understanding
with
language
and
gesture,”
in
Proceedings
of
the
IEEE/CVF
International
Conference
on
Computer
Vision
,
2021,
pp.
1385–1395.
[5]
D. Whitney, M. Eldon, J. Oberlin, , and S. Tellex, “Interpreting multi-
modal
referring
expressions
in
real
time,”
in
2016
IEEE
International
Conference
on
Robotics
and
Automation
(ICRA)
.
IEEE,
2016,
pp.
3331–3338.
[6]
A.
Xiao,
N.
Janaka,
T.
Hu,
A.
Gupta,
K.
Li,
C.
Yu,
and
D.
Hsu,
“Robi
butler:
Multimodal
remote
interaction
with
a
household
robot
assistant,”
in
2025
IEEE
International
Conference
on
Robotics
and
Automation
(ICRA)
.
IEEE,
2025,
pp.
4337–4344.
[7]
R.-Z.
Qiu,
Y.
Hu,
G.
Yang,
Y.
Song,
Y.
Fu,
J.
Ye,
J.
Mu,
R.
Yang,
N.
Atanasov,
S.
Scherer
et
al.
,
“Learning
generalizable
feature
fields
for
mobile
manipulation,”
arXiv
preprint
arXiv:2403.07563
,
2024.
[8]
L.-H.
Lin,
Y.
Cui,
Y.
Hao,
F.
Xia,
and
D.
Sadigh,
“Gesture-informed
robot
assistance
via
foundation
models,”
in
7th
Annual
Conference
on
Robot
Learning
,
2023.
[9]
P.-C.
Huang
and
A.
K.
Mok,
“A
case
study
of
cyber-physical
system
design:
Autonomous
pick-and-place
robot,”
in
2018
IEEE
24th
Inter-
national Conference on Embedded and Real-Time Computing Systems
and
Applications
(RTCSA)
,
2018,
pp.
22–31.
[10]
S.
Constantin,
F.
I.
Eyiokur,
D.
Yaman,
L.
B¨armann,
and
A.
Waibel,
“Interactive
multimodal
robot
dialog
using
pointing
gesture
recogni-
tion,”
in
European
conference
on
computer
vision
.
Springer,
2022,
pp.
640–657.
[11]
Y.
Lai,
S.
Yuan,
Y.
Nassar,
M.
Fan,
A.
Gopal,
A.
Yorita,
N.
Kubota,
and
M.
R¨atsch,
“Natural
multimodal
fusion-based
human–robot
inter-
action:
Application
with
voice
and
deictic
posture
via
large
language
model,”
IEEE
Robotics
&
Automation
Magazine
,
2025.
[12]
K.
Wang,
Z.
Wang,
K.
Nakagaki,
and
K.
Perlin,
““push-that-there”:
Tabletop
multi-robot
object
manipulation
via
multimodal’object-level
instruction’,”
in
Proceedings
of
the
2024
ACM
Designing
Interactive
Systems
Conference
,
2024,
pp.
2497–2513.
[13]
C.
Matuszek,
L.
Bo,
L.
Zettlemoyer,
and
D.
Fox,
“Learning
from
unscripted deictic gesture and language for human-robot interactions,”
in
Proceedings
of
the
AAAI
Conference
on
Artificial
Intelligence
,
vol.
28,
no.
1,
2014.
[14]
G.
Jocher,
A.
Chaurasia,
and
J.
Qiu,
“Ultralytics
yolo,”
https://github.
com/ultralytics/ultralytics,
Jan.
2023.
[15]
X.
Liu,
R.
Zhang,
C.
Zhang,
B.
Fu,
J.
Tang,
X.
Liang,
J.
Tang,
X.
Cheng,
Y.
Zhang,
G.
Wang,
and
X.
Ji,
“Gdrnpp,”
https://github.
com/shanice-l/gdrnpp
bop2022,
2022.
[16]
A.
Saupp´e
and
B.
Mutlu,
“Robot
deictics:
How
gesture
and
con-
text
shape
referential
communication,”
in
Proceedings
of
the
2014
ACM/IEEE
international
conference
on
Human-robot
interaction
,
2014,
pp.
342–349.
[17]
V.
Bazarevsky,
I.
Grishchenko,
K.
Raveendran,
T.
Zhu,
F.
Zhang,
and
M. Grundmann, “Blazepose: On-device real-time body pose tracking,”
arXiv
preprint
arXiv:2006.10204
,
2020.
[18]
C. Lugaresi, J. Tang, H. Nash, C. McClanahan, E. Uboweja, M. Hays,
F.
Zhang,
C.-L.
Chang,
M.
G.
Yong,
J.
Lee
et
al.
,
“Mediapipe:
A
framework
for
building
perception
pipelines,”
arXiv
preprint
arXiv:1906.08172
,
2019.
[19]
H.
Xu,
E.
G.
Bazavan,
A.
Zanfir,
W.
T.
Freeman,
R.
Sukthankar,
and
C.
Sminchisescu,
“Ghum
&
ghuml:
Generative
3d
human
shape
and
articulated pose models,” in
Proceedings of the IEEE/CVF Conference
on
Computer
Vision
and
Pattern
Recognition
,
2020,
pp.
6184–6193.
[20]
Y.
Xiang,
T.
Schmidt,
V.
Narayanan,
and
D.
Fox,
“Posecnn:
A
convolutional neural network for 6d object pose estimation in cluttered
scenes,”
2018.
[21]
G.
Wang,
F.
Manhardt,
F.
Tombari,
and
X.
Ji,
“Gdr-net:
Geometry-
guided
direct
regression
network
for
monocular
6d
object
pose
esti-
mation,”
in
Proceedings
of
the
IEEE/CVF
Conference
on
Computer
Vision
and
Pattern
Recognition
(CVPR)
,
2021,
pp.
16 611–16 621.
[22]
M.
Sundermeyer,
A.
Mousavian,
R.
Triebel,
and
D.
Fox,
“Contact-
graspnet:
Efficient
6-dof
grasp
generation
in
cluttered
scenes,”
2021.
[23]
A.
Radford,
J.
W.
Kim,
T.
Xu,
G.
Brockman,
C.
McLeavey,
and
I.
Sutskever,
“Robust
speech
recognition
via
large-scale
weak
super-
vision,”
in
International
conference
on
machine
learning
.
PMLR,
2023,
pp.
28 492–28 518.
[24]
S.
Team,
“Silero
vad:
pre-trained
enterprise-grade
voice
activity
de-
tector
(vad),
number
detector
and
language
classifier,”
https://github.
com/snakers4/silero-vad,
2024.
[25]
A.
Yang,
B.
Yang,
B.
Zhang,
B.
Hui,
B.
Zheng,
B.
Yu,
C.
Li,
D.
Liu,
F.
Huang,
H.
Wei,
H.
Lin,
J.
Yang,
J.
Tu,
J.
Zhang,
J.
Yang,
J.
Yang,
J.
Zhou,
J.
Lin,
K.
Dang,
K.
Lu,
K.
Bao,
K.
Yang,
L.
Yu,
M.
Li,
M.
Xue,
P.
Zhang,
Q.
Zhu,
R.
Men,
R.
Lin,
T.
Li,
T.
Xia,
X.
Ren,
X. Ren, Y. Fan, Y. Su, Y. Zhang, Y. Wan, Y. Liu, Z. Cui, Z. Zhang, and
Z. Qiu, “Qwen2.5 technical report,”
arXiv preprint arXiv:2412.15115
,
2024.
[26]
S.
Alrashed,
“Smoltulu:
Higher
learning
rate
to
batch
size
ratios
can
lead
to
better
reasoning
in
slms,”
2024.
[Online].
Available:
https://arxiv.org/abs/2412.08347
[27]
DeepSeek-AI,
“Deepseek-r1:
Incentivizing
reasoning
capability
in
llms
via
reinforcement
learning,”
2025.
[Online].
Available:
https:
//arxiv.org/abs/2501.12948
[28]
P.
Vanc,
R.
Skoviera,
and
K.
Stepanova,
“Tell
and
show:
Combining
multiple
modalities
to
communicate
manipulation
tasks
to
a
robot,”
arXiv
preprint
arXiv:2404.01702
,
2024.
[29]
M.
Prochazka,
S.
Dratva
, P.
V anc, R.
Skoviera, K.
Stepanova
,
and M. Vavrecka, “Probabilistic reasoner integrating multiple gestures
and
language
instructions
to
control
a
sequential
action
execution
in
a
humanoid
robot,”
ICAR
2025
,
2025.
[30]
A.
Grattafiori,
A.
Dubey,
A.
Jauhri,
A.
Pandey,
A.
Kadian,
A.
Al-
Dahle,
A.
Letman,
A.
Mathur,
A.
Schelten,
A.
Vaughan
et
al.
,
“The
llama
3
herd
of
models,”
arXiv
preprint
arXiv:2407.21783
,
2024.
[31]
H.
Chase,
“LangChain,”
Oct.
2022.
[Online].
Available:
https:
//github.com/langchain-ai/langchain
[32]
S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao,
“React:
Synergizing
reasoning
and
acting
in
language
models,”
in
International
Conference
on
Learning
Representations
(ICLR)
,
2023.
[33]
Y.-J.
Wang,
B.
Zhang,
J.
Chen,
and
K.
Sreenath,
“Prompt
a
robot
to
walk
with
large
language
models,”
in
2024
IEEE
63rd
Conference
on
Decision
and
Control
(CDC)
.
IEEE,
2024,
pp.
1531–1538.
This article has been accepted for publication in IEEE Robotics and Automation Practice. This is the author's version which has not been fully edited and
content may change prior to final publication. Citation information: DOI 10.1109/RAP.2026.3676940
This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/