Biomedical engineer Dr Gough Lui is using electronics to give the next generation of surgeons “hands-on” training.
At Sydney’s Liverpool Hospital, trainees undergoing their surgical skills training were finding it challenging to put the feedback they received into practice. The reason: mentoring over the shoulder.
An experienced surgeon would watch what the resident was doing and provide feedback. However, this feedback was not specific enough to enable change or enhance the resident’s skills. The problem attracted the attention of Dr Gough Lui, a biomedical engineer at Western Sydney University and the “engineer-in-residence” at Liverpool Hospital.
Lui set about using his expertise to devise a solution that would augment traditional surgical training. He recognised that surgeons employ intricate hand movements and focused on ways to track and measure these manoeuvres.
“There are virtual reality systems for surgical training but they’re quite expensive and you’re lucky if you are working at a hospital that has even one of them,” Lui tells create. “While simulators do provide some kind of automated feedback, it’s not easy to understand and there is no graphical representation.”
The need for a simple-to-use system inspired Lui, supported by funding from the James N. Kirby Foundation, to work on a glove-based solution that focused on individual finger motions.
“We started by instrumenting the fingers with force-sensitive resistors so we could measure how they’re squeezing on to the tools that they’re using,” says Lui. “We were interested in seeing how they were gripping the tool, how hard they were pushing on it, and whether they’re going to snap the instruments.”
Iterating from a prototype
Lui initially placed the electronics on the back of the glove to detect acceleration and hand orientation, then added force-sensors to the fingertips. “But experienced surgeons reported that the gloves reduced their touch sensitivity and were too bulky, hindering movement,” he says. For the next iteration, Lui removed the force sensors, opting for inertial measurement unit (IMU) sensors on the back of the hand.
A commercially available Bluetooth wireless unit sitting on the back of the hand transmits the signals for capture and analysis. “We don’t measure the fingertip force now, but at least we have the posturing of the hand and the speed of rotation, of linear acceleration — that is quite useful,” Lui says.
However, much of a surgeon’s dexterity and skill lie in their finger manipulation, and capturing this accurately was the next hurdle. To do so, Lui relied on tiny nine-axis IMUs that combine three-axis accelerometers, gyroscopes and magnetometers. These are placed on thin flexible PCBs and embedded in the glove so they lie on the back of each finger segment, recording measurements without hindering the surgeon’s movement or feeling.
“There is a wide understanding within the surgical fraternity that basic skills are perhaps more important than anything else,” says Lui.
A surgeon might need to change the way they operate for each patient because their body layout is a little bit different. The patient might be a distinct size and weight, for instance, or could have a tumour in an uncommon spot. “Surgeons have to adapt to such unique situations for every surgery,” says Lui. “But for things like throwing a stitch, that is the same, and they will be doing it thousands of times — possibly even in a week.
“So, when you think of it over the full length of the surgical career, some of these more basic skills are much more vital, and they’ve also been shown to be good indicators of the overall acquisition of higher-level skills. So, if you’re good at the basics, generally you are good at the more difficult aspects.”
That’s not to say that dexterity is the only component. “We are acutely aware that surgical skills also involve decision-making, and that’s something that our system won’t be able to teach you,” says Lui. “But we feel that any contribution to this field is worthwhile, because a lot of trainees have complained that these skills are difficult to acquire, and they drop out.”
This becomes an issue as experienced surgeons approach retirement age.
“They’ve developed these skills over 30 or 40 years and some of them have approached us and said, ‘We would like to be recorded because we want our hands and our movements to inspire another generation of surgeons’,” Lui says.
The smart glove may make this possible and, in doing so, reduce the risk to and improve the outcomes of people operated on by younger surgeons.
Data collection
Since surgeons normally double-glove for infection control, Lui’s system adheres to the back of an inner glove that goes over the hand. PCB strips on the glove along the back of the thumb, index and middle finger lead to a motherboard resting on the back of the hand. The motherboard holds a rechargeable lithium-polymer pouch battery that powers an ESP32 microcontroller providing wireless connectivity. This flat assembly is then encapsulated in a second latex glove.
Each node in the electronics assembly has a nine-axis IMU and with three nodes on each finger, leading to 81 channels.
“That’s quite a lot of data but, basically, each IMU is providing three-axis acceleration, gyroscope, and magnetometer outputs,” says Lui. “Later on, we transform that data into fewer channels and may even transform them into quaternions.”
Quaternions are mathematical formulae used to represent an orientation in 3D space and are used in 3D game programming. “It’s a little bit complicated, but there has also been interest in using AI,” says Lui.
However, he is unconvinced about AI’s benefits. “I feel that there is a black box with AI and ‘explainability’ has always been a hurdle,” Lui says. “You can train AI on data, and it will tell you ‘yes’ or ‘no’, but why is it a ‘yes’ and why is it a ‘no’? If we can’t work that out and communicate it to our trainees, we’re no better than the person watching over the shoulder saying that was good and that wasn’t good.”
Building a database of surgeons’ movements
So Lui is pursuing a different approach, looking at template-matching data analysis. His solution is to enable trainees to compare their movements with pre-recorded actions of experienced surgeons.
“We also know that not every surgeon does things the same way, so we are building a library of experienced surgeons’ movements,” he says. “The trainee can then say, ‘Well, I want to be like that surgeon. I want to compare myself to them. How do they do it?’ And then see the difference in the way they are manipulating each part of their finger — and also their hand overall.”
Lui considers building the database of experienced surgeons as his most vital activity for now. The other challenge is providing feedback during the surgery. Audio and haptic feedback can be distracting and build dependence. Lui’s solution is to record the procedure and allow the surgeons to match their movement data from the IMUs, but also see visually what was happening to their hands.
“Sometimes it’s not intuitive looking at IMU data. What does a heading of 135 degrees or a rotation of 600 degrees per second actually mean?” asks Lui. “But we can provide some guidance by actually showing them the visual difference as well.”
Chip shortages have delayed production, but Lui aims to trial the device by the end of the year. “The close relationship between clinicians and engineering researchers has been quite vital in shaping how this innovation is moving, and also for us to have willing participants to trial our prototypes and be enthusiastic about it,” he says.