UNSW researchers are carrying out work with virtual reality technology that could allow scientists to walk around 3D versions of cells and collaborate with each other.
The University of New South Wales’s Art & Design’s Associate Professor John McGhee is working with Professor Maria Kavallaris from UNSW Medicine and Children’s Cancer Institute in the second phase of the Journey to the Centre of the Cell project. The initiative combines scientific data, microscopy images and animation to create a virtual reality (VR) world of cells and blood vessels.
One goal of the research is to be able to see a drug enter a cancer tumour to allow clinicians to highlight the target for chemotherapy or radiotherapy, allowing drugs to be deployed more accurately.
Tracking 3D cells in tumours could also show scientists what happens when cells move in real time to observe the spread of cancer. The team has been working at the Australian Research Council Centre for Excellence looking at new types of drug delivery systems using nanoparticles and different types of nanomaterials.
As part of that work, UNSW researchers have been trying to develop ways to visualise and utilise data, which is primarily two-dimensional cell structure data taken with a microscope.
To see where drugs go, clinicians often have to do a scan or look at them on a microscope to see where the nanoparticles – the drugs – are going.
Even if that data is visualised in 3D, it can be difficult to understand, even for scientists who work in that field.
“We’ve got a 3D model of a cell structure, but we want to allow the science community to actually walk around that cell structure, rather than just peer down at it from a bird’s eye view,” McGhee said.
The team has used HTC Vive headsets and Oculus Rift headsets and developed a technique where scientists can actually move around in the cell structures, giving users the illusion of being in space.
“So it sort of activates a different part of your brain where you feel like you’re on a virtual field trip, as opposed to just looking at a proxy two-dimensional image,” McGhee said.
Last year the team looked into creating a single-user experience, but more recently they have been aiming to create a multi-user VR experience. This means developing techniques so that a group of scientists could walk around a cellular structure and make notes, interact with the structure, and see and talk to each other in the virtual reality world.
It’s hoped that a fully immersive experience like this could potentially involve researchers from elsewhere, such as overseas or from another university.
“We’ve not proven that yet, but that’s one of our goals, to possibly demonstrate that by seeing cells in an immersive way in a headset, it could actually allow the science community to see the data in new ways,” McGhee said.
“And ultimately, if they can collaborate more efficiently and see it in a new way, there’s a chance that we could actually improve efficiency of discovery.”
The researchers have also been looking at using this virtual reality experience for educational purposes.
In the early stages of the project, the team knew they had a lot of data that they wanted to visualise and allow more access to, but they didn’t know whether it would work or whether VR headsets would be a good idea.
“The intent was very much about playing with the tech, trying out visualisation techniques and seeing where it landed,” McGhee said. “But along the way, we just started to realise that education was a big part of what VR could enhance.”
The team recently published a study looking at this area, finding a significant increase in the exam results of the students who wore the headset over the ones who didn’t.
Challenges
One of the main challenges of the project was dealing with the size of the data.
Another was how to interact with the data and draw meaning from it, with the team deciding to split up the data into different levels like a video game.
“So you can go from the surface of the cell, on the top of the cell, the cell membrane, into the nucleus and the mitochondria and all the structures inside the cell in maybe two separate levels,” McGhee said.
This means not all the data is displayed at the same time, which involved a lot of UX development and experience development.
As part of an ongoing project, the team is also working with St Vincent’s Hospital to look at ways to help stroke patients make sense of their clinical data and understand why they had a stroke to help motivate them to do physiotherapy. This includes allowing patients to walk around their own blood vessels and see where stroke occurs and potentially enabling them to see a virtual avatar of themselves after their scans and the proliferation of a drug that they’ve taken.
The next stage of the overall project is to define the platform for multi-user interaction, looking at measuring the effectiveness of the platform and whether the VR experience is impacting on the discovery process.
By the end of 2018, McGhee is hoping to have a multi-user system up and fully running, and in 2019, the team is hoping to carry out studies to assess the impact of using the system on the discovery process.