IEEE Robotics and Automation Magazine

168 Pages·2022·37.14 MB·English

IE
E
E
R
O
B
O
T
IC
S
&
A
U
T
O
M
A
T
IO
N
M
A
G
A
Z
IN
E

B
IO
M
IM
E
T
IC
P
E
R
C
E
P
T
IO
N
, C
O
G
N
IT
IO
N
, A
N
D
C
O
N
T
R
O
L

V
O
L
U
M
E
2
9
• N
U
M
B
E
R
4
• D
E
C
E
M
B
E
R
2
0
2
2
Displaying the world in your hands. Inventing new ways to interact.

Force Dimension designs and manufactures the finest master haptic devices for leading-edge
aapppplliiccaattiioonnss iinn rreesseeaarrcchh,, mmeeddiiccaall,, iinndduussttrryy,, aanndd hhuummaann eexxpplloorraattiioonn..

The ScanTrainer from Intelligent Ultrasound is an educational tool that uses real patient
scans and curriculum-based teaching across obstetrics, gynecology, general medicine,
and emergency medicine. The system integrated a customized delta.3 haptic device.
Force Dimension
Switzerland
www.forcedimension.com
[e-mail]
Force
Vol. 29, No. 4 DECEMBER 2022
ISSN 1070-9932
http://www.ieee-ras.org/publications/ram
FEATURES
10 A Multimodal Fusion Model for Estimating
Human Hand Force
Comparing Surface Electromyography and Ultrasound Signals
By Yongxiang Zou, Long Cheng, and Zhengwei Li
25 A Hybrid Visual–Haptic Framework for Motion
Synchronization in Human–Robot Cotransporting
A Human Motion Prediction Method
By Xinbo Yu, Sisi Liu, Wei He, Yifan Wu,
Hui Zhang, and Yaonan Wang
36 Toward Holistic Scene Understanding
A Transfer of Human Scene Perception to Mobile Robots
By Florenz Graf, Jochen Lindermayr, Çag˘atay Odaba¸si,
and Marco F. Huber
50 Biomimetic Electric Sense-Based Localization
A Solution for Small Underwater Robots in a Large-Scale Environment
By Junzheng Zheng, Jingxian Wang, Xin Guo, Chayutpon Huntrakul,
Chen Wang, and Guangming Xie
66 Biomimetic Force and Impedance Adaptation
Based on Broad Learning System in Stable
and Unstable Tasks
Creating an Incremental and Explainable Neural
Network With Functional Linkage
By Zhenyu Lu and Ning Wang
78 Controlling Maneuverability of a Bio-Inspired Swimming
Robot Through Morphological Transformation
Morphology Driven Control of a Swimming Robot
By Kai Junge, Nana Obayashi, Francesco Stella,
Cosimo Della Santina, and Josie Hughes
92 Simulation to Real
Learning Energy-Efficient Slithering Gaits for a Snake-Like Robot
By Zhenshan Bing, Long Cheng, Kai Huang, and Alois Knoll
OONN TTHHEE CCOOVVEERR
This issue of IEEE Robotics and Automation 104 A Robust Visual Servoing Controller
Magazine focuses on the state of the art
for Anthropomorphic Manipulators With
in biomimetic perception, cognition,
and control research. Field-of-View Constraints and Swivel-Angle Motion
Overcoming System Uncertainty and Improving Control Performance
©SHUTTERSTOCK.COM/MONIKA WISNIEWSKA By Jiao Jiang, Yaonan Wang, Yiming Jiang, He Xie,
Haoran Tan, and Hui Zhang (Continued)
If you like an article, click this icon to
record your opinion. This capability is
available for online Web browsers and
offline PDF reading on a connected device.
Digital Object Identifier 10.1109/MRA.2022.3218353
DECEMBER 2022 • IEEE ROBOTICS & AUTOMATION MAGAZINE • 1
FEATURES
(Continued)
115 A Small-Scale, Rat-Inspired Whisker
Sensor for the Perception
A Publication of the IEEE ROBOTICS AND AUTOMATION SOCIETY
of a Biomimetic Robot Vol. 29, No. 4 December 2022 ISSN 1070-9932 http://www.ieee-ras.org/publications/ram
Design, Fabrication, Modeling, and Experimental
Characterization EDITORIAL BOARD Humanitarian Technology: Vacant
By Yulai Zhang, Shurui Yan, Zihou Wei, Xuechao Chen, Editor-in-Chief Standards: Craig Schlenoff (USA)
Yi Guo
Toshio Fukuda, and Qing Shi President’s Message: Frank Park (Korea)
([e-mail])
Regional Spotlight: Megan Emmons (USA)
Stevens Institute of Technology (USA)
127 Contact Shape and Pose Recognition Student’s Corner: Francesco Missiroli (Germany)
Editors
Utilizing a Multipole Magnetic Tactile Sensor Elena De Momi TC Spotlight: Yasuhisa Hirata (Japan)
With a Metalearning Model Politecnico di Milano (Italy) Women in Engineering: Karinne Ramirez
By Ziwei Xia, Bin Fang, Fuchun Sun, Huaping Liu, Jindong Tan Amaro (Sweden)
Weifeng Xu, Ling Fu, and Yiyong Yang University of Tennessee (USA) IEEE RAS Vice-President
Associate Editors of Publication Activities
Todd Murphey (USA)
Ming Cao
University of Groningen (The Netherlands) RAM home page:
http://www.ieee-ras.org/publications/ram
Feifei Chen
Jiao Tong University (China) IEEE Robotics and Automation Society
Executive Office
Carlos A. Cifuentes
Amy Reeder
Bristol Robotics Laboratory (UK)
Program Specialist
Kingsley Fregene [e-mail]
Lockheed Martin (USA)
Advertising Sales
Antonio Frisoli Mark David
Scuola Superiore Sant’Anna (Italy) Director, Business Development—Media &
Jonathan Kelly Advertising
University of Toronto (Canada) Tel: [phone]
Fax: [phone]
Ka-Wai Kwok
[e-mail]
The University of Hong Kong (Hong Kong)
IEEE Periodicals
Surya G. Nurzaman
Magazines Department
Monash University (Malaysia)
Kristin Falco LaFleur
Weihua Sheng Senior Journals Production Manager
COLUMNS & DEPARTMENTS Oklahoma State University (USA) Patrick Kempf
Yue Wang Senior Manager Journals Production
Clemson University (USA) Janet Dudar
4 FROM THE EDITOR’S DESK Enrica Zereik Senior Art Director
CNR-INM (Italy) Gail A. Schnitzer
6 PRESIDENT’S MESSAGE Houxiang Zhang Associate Art Director
Norwegian University of Science Theresa L. Smith
8 FROM THE GUEST EDITORS
and Technology (Norway) Production Coordinator
138 WOMEN IN ENGINEERING Past Editor-in-Chief Felicia Spagnoli
Bram Vanderborght Advertising Production Manager
141 INDUSTRY ACTIVITIES Vrije Universiteit Brussel (Belgium) Peter M. Tuohy
RAM Column Manager Production Director
145 TC SPOTLIGHT Amy Reeder (USA) Kevin Lisankie
Editorial Services Director
RAM Editorial Assistant
148 COMPETITIONS Joyce Arnold (USA) Dawn M. Melley
Staff Director,
158 SOCIETY NEWS COLUMNS Publishing Operations
Competitions: Hyungpil Moon (Korea) IEEE-RAS Membership
164 CALENDAR
Education: Andreas Müller (Austria) and Subscription Information:
+1 800 678 IEEE (4333)
From the Editor’s Desk: Yi Guo (USA)
Fax: [phone]
Industry News: Tamas Haidegger http://www.ieee.org/membership_services/
(Hungary) membership/societies/ras.html
IEEE prohibits discrimination, harassment, and bullying. For more information,
Digital Object Identifier 10.1109/MRA.2022.3218352 visit http://www.ieee.org/web/aboutus/whatis/policies/p9-26.html.
IEEE Robotics and Automation Magazine (ISSN 1070-9932) (IRAMEB) is published quarterly by photocopy beyond the limits of U.S. Copyright law for the private use of patrons 1) those post-1977
the Institute of Electrical and Electronics Engineers, Inc. Headquarters: 3 Park Avenue, 17th Floor, articles that carry a code at the bottom of the first page, provided the per-copy fee indicated in the
New York, NY 10016-5997 USA, Telephone: [phone]. Responsibility for the content rests code is paid through the Copyright Clearance Center, 222 Rosewood Drive, Danvers, MA 01923
upon the authors and not upon the IEEE, the Society or its members. IEEE Service Center USA; 2) pre-1978 articles without a fee. For other copying, reprint, or republication permission, write
(for orders, subscriptions, address changes): 445 Hoes Lane, P.O. Box 1331, Piscataway, NJ 08855 Copyrights and Permissions Department, IEEE Service Center, 445 Hoes Lane, Piscataway,
USA. Telephone: [phone]. Individual copies: IEEE Members US$20.00 (first copy only), NJ 08854. Copyright © 2022 by the Institute of Electrical and Electronics Engineers Inc. All rights
non-Members US$140 per copy. Subscription rates: Annual subscription rates included in IEEE reserved. Periodicals postage paid at New York and additional mailing offices. Postmaster: Send
Robotics and Automation Society member dues. Subscription rates available on request. Copyright address changes to IEEE Robotics and Automation Magazine, IEEE, 445 Hoes Lane, Piscataway, NJ
and reprint permission: Abstracting is permitted with credit to the source. Libraries are permitted to 08854 USA. Canadian GST #125634188 PRINTED IN THE U.S.A.
2 • IEEE ROBOTICS & AUTOMATION MAGAZINE • DECEMBER 2022
Say Hello_to the next
robotic innovator
Research teams from around the globe were asked
to submit their concepts for the “Open Platform
Challenge”. Three teams made it to the finals and will
be presenting their projects on automatica fair in
June 2023. The award comes with a 20,000-euro prize.
Meet our finalists.
KUKA Innovation
Award 2023
MeRLIn
Team SPIRIT, Team Fashion & Robotics, Team JARVIS,
German Aerospace Center, Germany University of Arts and Industrial Design Politecnico di Milano, Italy
A single large refinery often requires Linz, Austria The goal of the team from Politecnico
more than 50,000 maintenance and The fashion and textile industry are di Milano is to develop a complete
inspection routines. The team from the under great pressure to reduce the plug-and-play method for programming
German Aerospace Center is working on a environmental footprint by producing collaborative robotic applications
telepresence solution that enables safe and more durable, high-quality products and fully compatible with the new iiQKA
intuitive operation of aerial manipulators developing circular material flows. The ecosystem. This will facilitate their in-
for industrial applications in the oil and gas departments for “Creative Robotics” and troduction in small and medium-sized
industry. “Fashion and Technology” at UfG Linz are enterprises, enabling unskilled
working on creating a way for textile operators to program the robot for a
companies and designers to increase their new task and, thanks to the system
production by setting up micro factories AI-enhanced capabilities, generalize to
with collaborative robot systems. unknown situations, new tasks, and
product variants.
www.kuka.com/InnovationAward2023
FROM THE EDITOR’S DESK
Humans and the Environment
By Yi Guo
T
he COVID-19 pandemic has We need to nurture the envi- concepts involve principles
changed a lot of things, one ronment we live in and found in nature for creating
of which is human behavior. avoid environmental dam- robot locomotive capabili-
For me, I found a new hobby age that essentially endan- ties; multirobotic systems
of hiking during the first year of the gers human health. study how animal swarms
pandemic. I hiked in dozens of state The role nature plays in communicate and exchange
parks around me during the fall and human welfare and economic information and the hierarchical
winter sea- activity has been overlooked. Complex and territorial structure of animal societ-
sons, some of natural systems can flip from one equi- ies; and robotics learning draws inspira-
We need more and which I did librium to another when under pres- tion from human learning and neural
not even know sure. The demands humans currently network-based brain cognition capabil-
more innovative
existed before place on nature may not be sustainable ity. The field of soft robotics stems from
robots to join the pandemic. by Earth’s ecosystem. Natural capital bioinspired concepts and research on
I felt relieved was recently included in an analysis of deformable structures, soft materials,
humans’ effort
both physical- the sustainability of current rates of and morphological computation. While
to protect the ly and mental- economic growth. As we learn more bioinspired robotics is a very broad
ly after the through interaction with nature, it is research area, challenges exist in many
environment.
weekend hik- necessary to be aware of the sustain- frontiers, including the integration of sci-
ing trips, and ability of the ecosystem around us and ence and technology advances toward
it was helpful pay attention to the impact of human enhanced robotics capability in percep-
for me to reduce the Zoom fatigue behavior and what new technology will tion, cognition, learning, and control.
built up during work days. impose on the environment. This December issue is a special issue
Nature creates the environment that Robots can help protect the environ- (SI) on biomimetic perception, cogni-
we live in. Environmental damage can ment by adding monitoring capabilities tion, and control: “From Nature to
lead to new diseases. Nearly two-thirds in extreme conditions, such as robotic Robots.” We received more than 30 sub-
of the hundreds of diseases that emerged vehicles for supply transport, under-ice missions to this themed issue, which is
in the past century were transmitted exploration to support climate change the highest number of SI submission in
from animals to humans, including studies in the Arctic, and surface and the past two years. I’d like to thank the
HIV/AIDS and probably COVID-19. underwater robots to monitor chemical lead guest editor, Dr. Chenguang Yang,
Scientists also found a correlation and oil pollution. Robotics can also and his team, Dr. Shan Luo, Dr. Nathan
between the loss of forests in Africa and help fight climate change by reducing Lepora, Dr. Fanny Ficuciello, Dr. Dong-
the outbreaks of Ebola. Altering land use carbon emissions through renewable heui Lee, Dr. Weiwei Wan, and Dr.
threatens biodiversity, and deforestation energy resources, helping crops survive Chun-Yi Su, for their hard work manag-
and intensive farming are linked to out- droughts, and planting trees. We need ing the peer review process within the
breaks of transmitted diseases. Also, sci- more and more innovative robots to constrained time frame of this SI. I’d also
entists have studied how people have join humans’ effort to protect the envi- like to thank the assisting IEEE Robotics
altered living organisms and exerted an ronment. We have no choice. and Automation Magazine associate edi-
evolutionary pressure on other species. Bioinspired robotics is a field of tor, Surya Nurzaman, for his help on the
robotics that studies biological systems SI. I hope you find inspiration from the
to achieve engineering goals. Among feature articles. Enjoy reading!
Digital Object Identifier 10.1109/MRA.2022.3213198
Date of current version: 2 December 2022 the well-studied topics, locomotion
4 • IEEE ROBOTICS & AUTOMATION MAGAZINE • DECEMBER 2022
2
2
0
2
PRESIDENT’S MESSAGE
Rethinking the Research Paper
By Frank Park
L
ike many of you, I’ve spent ing as a technical field. Our details) and performed the
a good part of the summer work is more and more comparison experiments as
months harried by paper dead- embodied and validated by faithfully as we could. We’ve
lines. For each paper, the story code, data, and bench- not yet received feedback on
unfolds something like this: the tri- marks, and like the machine these latest set of experiments,
umph and euphoria that one experi- learning and data science but already I can anticipate
ences immediately after submitting the research communities, there is heated discussions on the experi-
paper quickly wears off, to be replaced growing acceptance within our com- mental setup, algorithm implementa-
by an encroaching regret that with munity on the need to make code and tion, and how truly meaningful these
more time, the data available. comparisons are in the absence of acces-
paper could What we still seem to lack in robot- sible, reproducible benchmarks.
While more evidence have been so ics, however, are benchmarks. To be fair, I have also noticed that several our
much better. benchmarks have been developed for conferences are now experimenting
needs to be collected
Then the re- grasping, pick-and-place, and various with double-anonymous (or double-
before any definitive views arrive: other robotics tasks, but these tend to be blind) reviews, open discussion phases
poring over the narrowly defined for specific hardware between reviewers and authors, and
conclusions can be
comments, de- and environment requirements and are several other new practices. All these
drawn, experimenting cisions are difficult to implement. Simulation efforts to try to reduce bias in our
quickly made benchmarks have been proposed as an reviews—be it gender, nationality,
with new review
on whether to alternative, but even the best simulators author or institutional reputation, or
practices is a very accept or reject today lack the ability to model friction, any number of factors— are highly wel-
a criticism— contact, deformations, and other com- come and in keeping with our commu-
welcome development.
should I hold plex physical interactions in a realistic nity’s spirit of innovation and fairness.
firm or show way. I think it’s fair to say that the Some members of our community are
some contrition for not explaining inherent challenges of developing currently studying the advantages and
more clearly?—and whether to comply benchmarks for robotics are much challenges of implementing double-
with all the requests (demands?) for ad- harder than, say, those for vision or nat- anonymous reviews, as well as the
ditional experiments and comparisons ural language. experiences of other research commu-
to existing work. The road to publica- Returning to the review of our paper, nities. While more evidence needs to be
tion is always a bumpy one, it seems. not surprisingly a reviewer had asked for collected before any definitive conclu-
Lately I’ve noticed that reviewers comparisons of our method against sions can be drawn, experimenting
seem to be asking for more and more another recently published method, with new review practices is a very wel-
elaborate experimental comparisons claiming this to be the state-of-the-art. come development.
against the state-of-the-art. At one Since no code was provided for this I encourage you to let your voices be
level, it’s a sign that robotics is matur- state-of-the-art method, we spent a great heard in the ongoing community dis-
deal of effort implementing the algo- cussion on how we measure research
rithm (whose description turned out to progress, and exploring ways to further
Digital Object Identifier 10.1109/MRA.2022.3214390
Date of current version: 2 December 2022 be lacking some small but crucial reduce bias in our review process.
6 • IEEE ROBOTICS & AUTOMATION MAGAZINE • DECEMBER 2022

FROM THE GUEST EDITORS
Biomimetic Perception, Cognition,
and Control: From Nature to Robots
By Chenguang Yang , Shan Luo, Nathan Lepora, Fanny Ficuciello,
Dongheui Lee, Weiwei Wan, and Chun-Yi Su
Awide range of technological surface electromyography signals and using the detachment of feathers while
developments are inspired by A-mode ultrasound signals of forearm maintaining the same low-level control-
biological individuals and ad- muscles. Yu et al. introduce a hybrid ler. Bing et al. apply reinforcement learn-
vanced synthetic materials, visual–haptic framework enabling a ing (RL) on snake-like robot control that
cognitive sensors, control algorithms, robot to achieve motion synchroniza- can fully exploit the hyper-redundant
artificial intelligence technology, and in- tion in human–robot co-transporting. bodies of the robot. Simulations and
telligent systems. One of the major chal- Graf et al. build a framework for experiments show that RL can generate
lenges is to create a comprehensive transferring ideas from human scene substantially more energy-efficient gaits
study by integrating different tech- perception to robot scene perception than those generated by conventional
niques into ro- to contribute toward robots’ holistic model-based controllers. Jiang et al. pro-
botic systems so scene understanding, based on a wide pose a humanoid control method based
On the topic of that the perfor- survey and comparison of robotic on visual servoing by utilizing a swivel
mance of ro- scene perception approaches with angle derived from the human arm to
robot system and
bots can be neuroscience theories and studies of realize the human-like behavior of
control, Stella et al. improved and human perception. anthropomorphic robot manipulators.
applied to more In the area of intelligent cognition The proposed method is substantiated
develop a feather
complex and and learning, Zheng et al. propose a based on a 7-degree-of-freedom Sawyer
star-like robot that diverse scenari- biomimetic electric sense-based local- robot and a constructed visual servo
os. The articles ization scheme, including an electric physical platform.
can actuate layers of
in this issue sense-based localization scheme and In the field of biomimetic sensors,
flexible feathers and focus on the three model-based perception meth- Zhang et al. design a rat-inspired
state of the art ods for large-scale underwater local- whisker sensor for a biomimetic ro-
detach them at will.
in biomimetic ization. Lu and Wang introduce a botic rat and demonstrate its superior
perception, novel biomimetic force and impedance tactile perception performance. Ex-
cognition, and control research and aim adaption framework based on Broad perimental results demonstrate its
to explore related technical avenues in Learning System for robot control in outstanding texture discrimination
the multimodal bioinformation percep- stable and unstable environments. The ability and excellent performance on
tion framework, intelligent cognition connections of created neural network contour reconstruction. Xia et al. de-
and learning, robotic systems and con- layers and the settings of the feature velop a soft magnetic tactile sensor
trol, and new biomimetic sensors. nodes are explainable by human motor and a multipole magnetization meth-
In the research of multimodal bio- control and learning principles. od for extracting contact surface fea-
information perception framework, On the topic of robot system and tures. We sincerely thank all the
Zou et al. propose the clinical readi- control, Stella et al. develop a feather authors for their contributions and
ness of a multimodal fusion model star-like robot that can actuate layers of the editors for their efforts and hope
to estimate hand force based on flexible feathers and detach them at will. that the contents of this issue can
Based on this optimized feather and bring information and inspiration to
theoretical framework, the new robotic researchers in related fields.
Digital Object Identifier 10.1109/MRA.2022.3213199
Date of current version: 2 December 2022 setup can change its motion path by
8 • IEEE ROBOTICS & AUTOMATION MAGAZINE • DECEMBER 2022