Robot Teleoperation Fatigue

Preliminary Notes

This project was not possible without the resources provided by Dr. Zhi Jane Li. Visit HiRo Lab for more information.

During my studies at WPI, I briefly contributed to HiRo lab, led by the amazing Dr.Jane Li. During my time there, I was part of a team that developed a preliminary study on Robotic Teleoperation Fatigue. Specifically, I implemented a simulation of a generalized surgical robot in order to measure a given user’s physical fatigue over time.

Some Definitions

I should note that this article is a gross summary of the materials produced during the project lifecycle. If you are inclined to learn more, feel free to contact me.

Clinical Problem

Hospitals, like most organizations, are adapting to technological advances in order to more efficiently perform tasks while saving coveted resources. Recently, we are seeing the dawn of teleoperated robots in order to perform surgeries and even day to day nursing tasks. The former is synonymous with the vastly popular da Vinci system developed by Intuitive Surgical. For day to day nursing tasks, HiRo labs has proposed a robotic solution called TRINA (Tele-Robotic Intelligent Nursing Assistant).

However, the robotic solution is most often bottlenecked by the learning curve a given teleoperation method has. For example, while it is always possible for a user to control a given robotic system using a keyboard and mouse, the time commitment to master the mapping between keystrokes and movement will be incredibly massive, not to mention the lack of flexibility in terms of fast, rapid, and long movements. For this reason, many companies have opted to use teleoperation interfaces such as a joypad, a “puppet”, and even the body of the teleoperator.

After research on these teleoperation methods, there is an understood cost of physical activity with respect to how intuitive the teleoperation method is. An example would show that a keyboard and mouse is not physically demanding of a teleoperator, but the lack of mobility is evident, and the learning curve is massive. Inversely, solutions such as a mock “puppet” of a robot or a system integrating the human body for movement is extremely intuitive. However, these solutions cost physical effort, which induces physical fatigue.

Although this fatigue is understood to occur, there is a significant lack of research on the degree of fatigue experienced during teleoperation. In both the surgical and nursing settings aforementioned, fatigue can cause the operator to make errors that he/she would otherwise not make, which could put the patient at risk.

The Clinical Problem: Physical Fatigue effects the performance of teleoperators, and can leave to mistakes and patient risk

The Study

Before mentioning the details, I am required to say that this study was approved by the WPI IRB. All participants were given the proper training to teleoperate the Baxter robotic system and 3D Systems Touch Haptic Devices.

The general plan of attack for this preliminary study is to use EMG signal analysis to determine muscle group use during Teleoperation. As mentioned before, there are two main types of fatigue that could occur in common robotic teleoperation when considering the medical space. Therefore, this study is split between general robotic teleoperation using a tele-nursing robot, and a generalized surgical robotic environment using Virtual Reality and two 3D Systems Touch Haptic Devices.

Surgeon Fatigue

Since the da Vinci is the most popular surgical robot currently, it was used as the main model for surgeon positions and maneuvers for controlling a general robot.

The ergonomics of the da Vinci system were fabricated considering multiple challenges associated with mapping movements made by a human arm into a robotic system. These include but are not limited to surgeon posture, ergonomics, interface commands & gestures, and stereo vision. Even though these variables were taken into consideration, the interface for the da Vinci system has significant ergonomic issues relating to the control interface. A summary of the literature shows multiple papers delivering conclusions of fatigue, although none provided metrics.

To model the fatigue over time that a surgeon would encounter while using the master of a da Vinci system, our study considered variables such as average surgery time using the model system, common surgical movements and gestures, and overall posture. The result from these considerations was a study that is comprised of qualitative and quantitative metrics.

The study is ordered in the following fashion (Where the tasks are repeated until the patient is exhausted):

1. Questionnaire
2. Practice
3. Tasks
   1. Cut
   2. Object Retrieval
   3. Suture
4. Questionnaire

The Simulation

Since obtaining a cadaver and a da Vinci system for our study was out of reach, a simulation was created using Unity 3D. Considering the proposed tasks of cutting, retrieving an object, and suturing, some simple objects were modeled in Solidworks in order to qualify as suitable replacements for the general task of manipulating soft tissue.

In tandem with the soft tissue surrogates would be two end effectors (Robotics nomenclature for the tip of a robot arm) and foreign objects beneath the soft tissue. The end effectors were controlled by the user using two 3D Magic Haptic Touch Devices. These 3D Magic Haptic Touch Devices give the teleoperator control of all objects in the environment, including manipulating the position and orientation of objects. However, certain limitations were implemented in order to isolate the fatigue to only induced by physical demands, and removing as much unnecessary mental barrier as possible to the average user:

1. All objects, unless under direct manipulation, are not subject to gravitational forces with the exception of the foreign objects
2. Several camera perspectives were allotted to the user in order to minimize the learning curve
3. As mentioned before, we opted to keep movements real time and not induce the realistic ~100ms of lag in order to isolate the physical fatigue

In terms of data collection, VICON and EMG data was being obtained in order to assess position, orientation, and muscle activity.

Results and Discussion

The data that was collected for the study, as a whole, is currently being used for a future conference paper. Due to this restriction, I will be showing the results for a single user, but not the total correlation between all users. The figures below represent the total data, after analysis, obtained via the EMG and VICON systems (Respectively) for one user:

Closing Thoughts

This project brought new and exciting technologies to my toolbox, including EMG and VICON data analysis, Haptic Devices, and a great amount of System Integration practice.