New Metrics for a New Generation of Robots
The framework combines classical robotics indicators with metrics that better capture
the behaviour of biologically inspired systems. Beyond task success, it tracks:
execution time and reaction time across behavioural levels (reflex
<10 ms, reactive <100 ms, adaptive <1 s); total power consumption
across sensors, actuation, computation and training; system complexity in
terms of sensor count, network parameters and model architecture; task
progression and forwards/backwards transfer across tasks; agility
in performing fast and efficient movements; and environmental disruption
during operation.
This unified approach provides a comprehensive baseline for evaluating systems that
aim to match or exceed the performance of classical systems, enabling multi-objective
optimisation tailored to specific application requirements.
The framework proposes four benchmark task categories of progressively increasing complexity,
inspired by the RoboSoft Challenge Competition. Each task challenges the system's ability
to coordinate sensing, computation and actuation across different levels of control,
highlighting the strengths of neuromorphic architectures in handling both internal
variations and external environmental dynamics.
1. Precision Manoeuvring — Navigation among static and dynamic obstacles
The robot navigates through progressively narrowing apertures, executes constrained
angular turns around rigid obstacles, and avoids unstable elements that fall if touched.
This tests precision motion control and real-time path planning. Real-world applications
include assistive robots in hospitals, search-and-rescue missions, and agricultural
robots in greenhouses.
2. Adaptive Terrain — Locomotion across varied terrain types
The robot traverses mixed surfaces including cardboard, sand, and inclined trapezoidal
slopes (7° ascent and descent), requiring flexible locomotion strategies to handle
changes in friction and surface compliance. Relevant to stair-climbing robots,
planetary exploration rovers, and quadruped robots used in construction sites.
3. Parameters Adaptation — Adaptive control under changing mechanical properties
The robot adjusts its internal control parameters in response to changes in body
stiffness (low to high and back), challenging the controller's robustness to
alterations in the physical system — analogous to exoskeletons adapting to
user gait, or robots adapting to terrain stiffness.
4. Grasping and Manipulation — Interaction with rigid and deformable objects
Three manipulation tasks evaluate dexterity and control precision: lifting and holding
a rigid object (500 ml water bottle) for 7 seconds without destabilising it; handling
a deformable object (empty flexible bottle) without excessive compression or slippage;
and solving the Tower of Hanoi with three rings in at most 11 moves. Applications
include human–machine interaction, surgical robotic systems, and pick-and-place systems.
"The aim is to offer the research community a shared, open, and reproducible foundation
on which different robotic control approaches can be fairly compared under conditions
closer to the real world."
— Giulia D'Angelo, CTU FEL
