ISSN 1866-8836
Клеточная терапия и трансплантация

Experience in the use of a radio-oriented augmented reality navigation system for biopsy of a neoplasm of the lower jaw

Anna V. Lysenko1, Andrey I. Yaremenko2, Vladimir M. Ivanov3, Sergey V. Strelkov3, Elizaveta A. Ivanova2

1 Department of Maxillofacial Surgery, Research Institute of Dentistry and Maxillofacial Surgery, St. Petersburg, Russia
2 Department of Maxillofacial Surgery, Pavlov University, St. Petersburg, Russia
3 Peter the Great St. Petersburg Polytechnic University, St. Petersburg, Russia

Dr. Anna V. Lysenko, Department of Maxillofacial Surgery, Research Institute of Dentistry and Maxillofacial Surgery, Pavlov University, 44 Petrogradskaya Emb., 197101, St. Petersburg, Russia
Phone: +7 (812) 429-03-33

Citation: Lysenko AV, Yaremenko AI, Ivanov VM et al. Experience in the use of a radio-oriented augmented reality navigation system for biopsy of a neoplasm of the lower jaw. Cell Ther Transplant 2021; 10(3-4): 78-83.

doi 10.18620/ctt-1866-8836-2021-10-3-4-78-83


Radiomics is a quantitative approach to medical imaging that applies advanced mathematical analysis in order to improve the existing data available to clinicians. Radiomics quantifies texture information by mathematical extraction of spatial distribution of the signal intensities and pixel relationships. Quantitative evaluation of the texture 2-D information employs analytic techniques from the field of artificial intelligence. The data derived from radiographic images, when compared with clinical data, may potentially provide additional information aiming for support of decision-making in clinical medicine. In this study, a preliminary radiomic analysis of a lower jaw neoplasm was performed. Based on the data obtained, the optimal site for tissue biopsy was chosen. During diagnostic intervention, an augmented reality navigation system was used which took into account the results of the mentioned mathematical analysis.


Radiomics, augmented reality, dynamic navigation systems, jaw neoplasms.


Medical imaging is one of the main diagnostic methods that defines the treatment strategy. The main advantage of X-ray data examination is its ability for digitilization. The data obtained by the digitization provide multidimensional data which, due to appropriate standardization and analysis, enable additional information about specific features of the diseases [1].

Radiomics is a quantitative method for the analysis of digitized X-ray images, which is based on mathematical analysis [2]. Radiomics allows quantifying texture information by mathematically extracting the spatial distribution of signal intensities and pixel relationships [2, 3]. The concept of radiomics is based on the search for imaging biomarkers that are specific to certain pathological processes undetectable by standard visual inspection of the generated images. Visually noticeable differences in the intensity, shape and texture of the image can be quantified using radiomics, which allows the more objective process of interpreting X-ray images [3]. The key purpose of radiomics applied for medical image analysis is to reveal objective non-invasive prognostic biomarkers of the disease, presuming its further transition to personalized medicine [4].

Four main steps for extraction of radiomic data from medical images are as follows:
1. Acquisition of images using methods of radiological diagnostics;
2. Segmentation of the region of interest (ROI or volume of interest – VOI);
3. Extraction of radiomic features;
4. Obtaining biomarkers of imaging [5].

However, in order to improve the accuracy of diagnostic procedures, such as biopsy of the neoplasm, the obtained data require integration into the clinical picture. This can be done by directly overlaying the region of the location of the imaging biomarker on the region of neoplasm through augmented reality.

Augmented reality is the projection of any digital information (images, video, text, graphics, etc.) over a real image [6]. As a result, the real world is supplemented with artificial elements and new information. This technology is becoming increasingly popular in various fields of medicine, including maxillofacial surgery [6]. The following is required with respect to ensurance of the technology functioning:
1. Acquisition of images using methods of conventional radiology diagnostics (obtaining data on a three-dimensional object);
2. Segmentation of the region of interest (e.g., neoplasms);
3. Loading the region of interest into augmented reality glasses;
4. Conducting surgery in augmented reality, focusing on the location of the region of interest.

As seen from the below algorithms, working in radiomics and augmented reality has several identical stages. Moreover, integration of the latter technique contributes to creation of a radio-oriented navigation system (Fig. 1).


Figure 1. Algorithm for radiome analysis of Cone Beam Computed Tomography images and the algorithm for the operation of augmented reality technology (Original picture)

The purpose hereof is to present an algorithm for radiomic image analysis using the example of a neoplasm of the lower jaw and to demonstrate practical applicability of this technology, i.e., to conduct a radiomically targeted biopsy of a neoplasm directed by the augmented reality technology.

Materials and methods

The authors used a comprehensive open source platform PyRadiomics, built into 3D Slicer, the interface of the software was designed for analyzing and working with medical images, to handle the radiological data. The software is intended for processing and extraction of radiomic characteristics from medical images using a large panel of hard-coded function algorithms (Fig. 2).


Figure 2. Appearance of the software 3D Slicer

Stage 1. Acquisition of images using methods of radiology diagnostics


Figure 3. Appearance of the occlusal splint with X-ray contrast marks (A); Patient positioning at the cone-beam computed tomography with occlusal splint (B)

The first stage is to identify the clinical problem and obtain a digital image, excluding low quality studies. When planning to perform an intervention in augmented reality, it is necessary to perform an X-ray examination using X-ray contrast markers, according to which the augmented reality marker will be localized and calibrated for the prospective surgery.

Taking into account the mobility of lower jaw, an individual occlusal splint with X-ray contrast marks and an integrated holder for the augmented reality marker was pre-fabricated. Cone beam computed tomography (CBCT) was performed with a mouthguard containing X-ray contrast marks fixed in the patient's mouth (Fig. 3).

Stage 2. Segmentation of the region of interest

Upon receipt of the radiological data in DICOM format, one should define the region of interest (ROI) in two-dimensional projection (2D), or volume of interest (VOI) in three-dimensional (3D) projection begins. ROI/VOI determine the region in which the radiomic features are calculated. The image segmentation may be performed manually, semi-automatically or fully automatically (Fig. 4).

Stage 3. Extraction of radiomic features

After image segmentation and processing, the extraction of radiomic features can be performed, which is performed automatically after activation of the PyRadiomics module. The features extracted from images could be divided into morphological parameters (volume and shape), histogram features (description of gray tone intensities, texture analysis) (Fig. 5).

Stage 4. Obtaining imaging biomarkers

After receiving the radiomic data, selection or reduction of specific features is carried out. After that, the multivariate data analysis is started: a connection is established between the texture features of the gray level coincidence matrix and the morphological type of neoplasms. Isolated radiomic features that closely correlate with clinical findings may be assigned to the imaging biomarkers (Fig. 6).


Figure 4. Segmentation of the area of interest in 3D Slicer


Stage 5. Creation of a radiomally oriented augmented reality navigation system

To perform the surgery, two augmented reality markers were used. Marker 1 was attached to a holder embedded into the occlusal splint (as in cone-beam computer tomography?? CTCBCT). Marker 1 was associated with an image of a segmented neoplasm of the lower jaw with a highlighted visualization biomarker. Marker 2 was attached to the handle of the dental handpiece. Projection of the bone trephine was attached to marker 2 (Fig. 7 a, b).


Figure 7. Marker 1 fixed in the oral cavity (A); marker 2, at the dental handpiece (B)

Stage 6. Radiomic-guided biopsy of mandibular neoplasm in augmented reality

During the surgical intervention after fixation of the augmented reality markers, we checked calibration of the marker positions, projection of the lower jaw neoplasm, and the tip of the biopsy instrument. The surgery was performed under local anesthesia. Skeletonization of the outer plate of the lower jaw in the area 3.3-4.3 was performed. Bone tissue in the region under inspection had no visible abnormal changes (Fig. 8).


Figure 8. The body of the lower jaw in the projection of the neoplasm. Pathological changes are not visualized

The neoplasm imaging was made in augmented reality. Using a bone trephine, a fragment of the formation was sampled in projection of the imaging biomarker (Fig. 9).


Figure 9. Location of the neoplasm in virtual imaging (A), and actual location of the object (B)

The material was sent for histological examination. The postoperative period was uneventful.

Histological report was as follows: an area of fibrous dysplasia of the lower jaw.


In the presented clinical case, according to the results of the radiomic analysis of the segmented neoplasm of the lower jaw, its distinctive radiomic features were revealed - imaging biomarkers, that were transferred to the surgical field using augmented reality technology.

The object of a lower jaw neoplasm obtained as a result of segmentation with a highlighted area of region biomarkers is combined with real anatomical structures in the oral cavity at the time of surgery by means of augmented reality technology. The combined model of this neoplasm demonstrates its anatomical and topographic position and structure, thus making it possible to determine the optimal operative access and most informative biopsy area for histological examination. Usage of this approach to tracking the instrument tip (bone trephine) in mixed reality enables control of its position, immersion depth, and reduced risk of traumas to adjacent anatomical structures. Real volume and topography of the object and adjacent anatomical structures fully corresponded to the data obtained in the course of virtual planning.


Analysis of medical images allows non-invasive assessment of the characteristics of human tissues. However, the interpretation of research results is often quite subjective. Recent advances in acquisition and analysis of medical imaging allow high-precision digital data extraction to quantify the difference between healthy and diseased tissue.

Radiomics applies mathematical analysis and advanced computational methodologies to medical imaging data to provide quantitative descriptors of pathological tissues. This is especially true for oncology.

Radiomics analysis of X-ray images has the potential to promote development of personalized medicine.

Clinical application of radiomic data is of the greatest interest for targeted biopsy of neoplasms in maxillofacial area. In order to improve accuracy of the diagnostic procedure, we recommend usage of augmented reality technology, which makes it possible to accurately visualize the most informative biopsy area for histological examination.

On the basis of advances in modern digital technologies such as radiomics and augmented reality, one can personalize and adapt established methods of treating diseases in the maxillofacial region for each patient, to improve the results of diagnosis and treatment, as well as reducing the number of complications.


  1. Van Griethuysen JJM, Fedorov A, Parmar C, Hosny A, Aucoin N, Narayan V. Computational radiomics system to decode the radiographic phenotype. Cancer Research 2017; 77(21) :104-107. doi: 10.1158/0008-5472.CAN-17-0339
  2. Aerts HJWL, Velazquez ER, Leijenaar RT, Parmar C, Grossmann P, Carvalho S. Decoding tumour phenotype by noninvasive imaging using a quantitative radiomics approach. Nature Сommun. 2014; 5(1): 1-9. doi: 10.1038/ncomms5006
  3. Caudell JJ, Torres-Roca JF, Gillies RJ, Enderling H, Kim S, Rishi A, et al. The future of personalised radiotherapy for head and neck cancer. The Lancet Oncology 2017; 18( 5):266-273. doi: 10.1016/S1470-2045(17)30252-8
  4. van Timmeren JE, Cester D, Tanadini-Lang S, Alkadhi H, Baessler B. Radiomics in medical imaging – "How-to" guide and critical reflection. Insights Imag. 2020; 11(1): 1-16. doi: 10.1186/s13244-020-00887-2
  5. Yaremenko АI, Lysenko AV, Ivanova EA, Galibin OV. Augmented reality technology for auricular reconstruction in the treatment of microtia. Cell Ther Transplant. 2020; 9(2): 78-82. doi: 10.18620/ctt-1866-8836-2020-9-2-78-82
  6. Lysenko AV, Razumova AYa, Yaremenko AI, Mirzakhmedov RR. Augmented reality in the treatment of sialolithiasis. Stomatology. 2020; 99(4): 64-66. doi: 10.17116/stomat20209904164

Volume 10, Number 3-4

Download PDF version

doi 10.18620/ctt-1866-8836-2021-10-3-4-78-83

Back to the list