An artificial intelligence–based image recognition model using indocyanine green cholangiography to identify the hepatocystic triangle during minimally‑invasive cholecystectomy

Introduction

Minimally-invasive cholecystectomy, performed using laparoscopic or robotic methods, is one of the most frequently performed surgical procedures.^1-3 Owing to its advantages, including faster recovery and better cosmetic outcomes, approximately 750 000 minimally-invasive cholecystectomies are performed annually in the United States, accounting for roughly 90% of all cholecystectomies.⁴ However, this operation can lead to severe complications, such as intraoperative bile duct damage. Therefore, even experienced surgeons must be cautious while performing it.^5,6

Bile duct injury is a complication associated with significant perioperative morbidity and mortality (up to 3.5%), reduced long-term survival and quality of life, and high rates of subsequent litigation.⁷ Moreover, late complications arising after bile duct injury include liver fibrosis or even secondary biliary cirrhosis and portal hypertension, often resulting from prolonged biliary obstruction.^8-10 One of the major causes of bile duct damage is misjudgment of the hepatocystic triangle’ location, which may occur regardless of the surgeon’s experience.⁶ The bile duct serves as a conduit for bile flow and appears as a greenish structure; however, it is often embedded within adipose or fibrous tissue. In patients with obesity or severe cholecystitis, the bile duct is not clearly visible, and may be mistaken for another structure. Therefore, an objective localization method that can help prevent bile duct damage during surgery is required. Recently, artificial intelligence (AI) technology based on deep learning has emerged as a promising tool in this field.^11,12

Deep learning, a subset of machine learning, trains computers to perform human-like tasks, such as image recognition and prediction. Unlike conventional algorithms that rely on predefined rules, deep learning establishes basic parameters and allows computers to learn autonomously by recognizing patterns through multiple processing layers. This approach has been increasingly applied in diagnostic imaging, including radiology and endoscopy, and is now being implemented in surgery.^13-15 Several studies have explored AI models to track the hepatocystic triangle and prevent bile duct injury during laparoscopic cholecystectomy (LC).^11,16,17 However, a major limitation of these studies lies in the subjective identification of the hepatocystic triangle. Despite ongoing research, precise localization of this anatomical region remains challenging when it is not visible.

Aim

To overcome this limitation, this study developed a model to determine the exact location of the hepatocystic triangle using indocyanine green (ICG) fluorescent staining technology called ICG-based near-infrared cholangiography (NIRC).

Materials and methods

This study was approved by the Institutional Review Board of the Hallym Medical Center (2022-10-011). All research involving human participants was conducted in accordance with the Declaration of Helsinki. Written informed consent was obtained from all participants prior to inclusion. The sample size was determined after reviewing a study by Madani et al.¹¹ The area under the curve (AUC) for imaging-based recognition of the hepatocystic triangle in the aforementioned study was 0.79. Assuming a 3% difference in AUC, a 2-sided significance level of 0.05, and a statistical power of 90% (accounting for a dropout rate of 20%), the required total sample size was estimated at 200 patients. The calculation was performed using G-Power 3.1 software (Heinrich-Heine University, Düsseldorf, Germany).

Dataset and annotations

A total of 264 cholecystectomy videos (laparoscopic, n = 158 and robotic, n = 42) performed at the Hallym University Medical Center between January 2018 and December 2021 were retrospectively collected in an anonymized manner. Of them, 64 videos documenting any image corruption due to camera issues (n = 32) or blurriness due to moisture (n = 32) were excluded. The remaining 200 videos were analyzed. The inclusion criteria comprised: 1) a diagnosis of gallbladder polyps or symptomatic gallstones without evidence of common bile duct stones (grade 1–2 as per the Tokyo guidelines for the diagnosis and severity grading of acute cholecystitis)¹⁸; 2) age between 18 and 80 years; 3) ability to adhere to the protocol and provide written informed consent; 4) no previous upper abdominal surgery; and 5) no liver cirrhosis. The patients’ characteristics are summarized in Table 1. The cohort consisted of 114 men (57%) and 86 women (43%), at mean (SD) age of 43.1 (8.9) years and mean (SD) body mass index (BMI) of 26.7 (4.6) kg/m². Most patients were classified as American Society of Anesthesiology class II (74%), while 26% were class I.

Table 1. Patient characteristics (n = 200)

Parameter		Value
Sex	Women	86 (43)
	Men	114 (57)
Age, y		43.1 (8.9)
Body mass index, kg/m2		26.7 (4.6)
ASA classification	Class I	52 (26)
	Class II	148 (74)
Indication of surgery	Calculous cholecystitis	154 (77)
	Gallbladder polyp	46 (23)
Surgical type	Laparoscopic cholecystectomy	83 (42)
	Robotic cholecystectomy	117 (58)
Preoperative laboratory results	White blood cell count, × 103	7.3 (2.2)
	Total bilirubin, μmol/l	13.68 (6.84)
	Alanine aminotransferase, IU/l	65.2 (85.8)
	Alkaline phosphatase, IU/l	78.6 (37.9)
Data are presented as number (percentage) or mean (SD). Abbreviations: ASA, American Society of Anesthesiology

The primary indications for surgery were calculous cholecystitis (77%) and gallbladder polyps (23%). Regarding the surgical approach, 83 patients (42%) underwent LC, whereas 117 patients (58%) were subjected to robotic cholecystectomy.

The hepatocystic triangle was defined as a triangular space at the porta hepatis of surgical importance, as it is dissected during cholecystectomy. Its boundaries—the cystic duct (left border), common bile duct (right border), and inferior surface of the liver (superior border)—were clearly identified during NIRC. The images from the selected videos included frames from the point of hepatocystic triangle exposure to cystic duct ligation, confirmed in both the original surgical videos and ICG fluorescence images (RUBINA ICG platform; KARL STORZ SE Co., Tuttlingen, Germany; Firefly Fluorescence imaging; Intuitive Surgical Inc., Sunnyvale, California, United States). ICG-based NIRC was performed within 1 hour after intravenous administration of 0.25 mg/kg ICG to facilitate accurate bile duct identification.¹⁹ All images were anonymized and assigned serial numbers. The annotation process was performed iteratively to maximize model learning efficiency (Figure 1), and was as follows: 1) extraction and labeling of the original image files showing the hepatocystic triangle from surgical video images; 2) extraction and labeling of the ICG fluorescence image files depicting the exact triangle location; and 3) overlaying and matching of the original and ICG fluorescence images, and setting the hepatocystic triangle as the annotation target for model labeling.

Labeling accuracy was verified by 1 experienced surgeon (TY) and 1 medical imaging technician (J-UH). After matching ICG fluorescence images, both confirmed the final labeled location of the triangle. The annotation process was conducted using the MakeSense platform (http://makesense.ai).

Data analysis

A total of 3796 images were extracted from 200 cholecystectomy videos, and divided into training and validation datasets. The images and corresponding landmarks (hepatocystic triangle) were separated by patient to prevent data leakage between the datasets. The final dataset consisted of 2979 training and 817 validation images.

Detection of the landmark (hepatocystic triangle)

In computer vision, the task of locating and predicting an object’s position is referred to as object detection. Since the landmark’s location should be indicated in real time, both accuracy and calculation speed were critical considerations in model development. Therefore, we used the YOLO architecture (Ultralytics, London, United Kingdom) for real-time object detection,²⁰ specifically, the most recent YOLO model—the YOLOv5s—which we selected from the official Ultralytics implementation (version 7.0; https://github.com/ultralytics/yolov5) because of its excellent trade-off between speed and accuracy in surgical object-detection tasks.²¹

The YOLOv5s model follows a structured approach to object detection. First, it processes an input image through the feature extraction step, where it identifies important patterns, such as the brightness and shape of the hepatocystic triangle. Then, the model refines this information at the feature processing stage, which helps improve detection under different conditions, such as variations in patient anatomy or surgical lighting. Finally, at the detection stage, the model generates a bounding box around the hepatocystic triangle, helping surgeons visualize the area of interest. This structured approach ensures that the model can operate efficiently in real time without compromising accuracy. By dividing the detection process into these three steps, the model remains interpretable and clinically applicable for real-world surgical use.

Deep learning model training

Data augmentation was applied to increase the diversity of the training set and improve model generalization. The following on-the-fly augmentations were applied with default YOLOv5 probabilities: resizing to 640 pixels × 640 pixels, horizontal flip (P = 0.5), scale (±0.5), translation (±0.1), hue (±0.015), saturation (±0.7), and value (±0.4) mosaic augmentation (P >0.99), which combines multiple images to enhance learning efficiency. The training process was conducted for up to 600 epochs using a batch size of 16 on an Nvidia RTX 3090 graphical processing unit (Nvidia Corp., Santa Clara, California, United States). Early stopping was implemented to prevent overfitting by terminating training when no improvements were observed for 100 consecutive epochs. The model was trained using a stochastic gradient descent optimizer with a momentum method, with an initial learning rate of 0.01 that progressively decreased according to a λ learning rate scheduler. For real-time feasibility, the trained model was evaluated for computational efficiency. The inference speed was approximately 10–15 ms per image at a resolution of 640 pixels × 640 pixels, demonstrating near-instantaneous detection suitable for intraoperative use. This speed enables seamless integration into surgical workflows without introducing significant delays.

Statistical analysis

When predicting landmarks, the YOLOv5s model outlines bounding boxes on each landmark against the output image files. This model generates multiple bounding boxes overlapping around a target, which are then refined using the nonmaximum suppression (NMS) algorithm.²² The model’s performance varies depending on the intersection-over-union (IoU) threshold set for NMS. Bounding boxes predicted in duplication are removed if their IoU value exceeds the threshold specified by the hyperparameter. We systematically varied the NMS IoU threshold from 0.2 to 0.9 in 0.1 increments. The performance was evaluated using standard common objects in context object-detection metrics: mean average precision (AP) at IoU threshold 0.5, precision, recall, and F1-score, calculated at a confidence threshold of 0.25. To optimize the IoU threshold between the predicted bounding box and the ground truth, we evaluated performance using an AP score, a standard metric for assessing object detection accuracy.

Results

Observation of training with early stopping

In Figure 2, we illustreted the learning process of our model. During training (Figure 2A), the training loss gradually decreased as the number of epochs increased. However, the validation loss began to rise again after reaching its lowest point around 40 epochs. This phenomenon is generally regarded as overfitting, which occurs when a model becomes biased towards the training data, resulting in poor generalization to unseen datasets.

Since overfitting had already progressed, early stopping was applied to prevent inefficient learning.²³ Early stopping is used to prevent overfitting by halting training when the validation loss fails to decrease over a specified number of epochs. Therefore, although we initially set the training for 600 epochs, the process terminated earlier due to the application of early stopping (Figure 2B).

Optimal intersection over union threshold

Through multiple experiments, we determined that the optimal IoU threshold for our model was 0.6 (Figure 3; Table 2). At this optimal IoU threshold, the final model demonstrated the precision–recall curve and an AP score shown in Figure 4 (AP score = 0.859).

Table 2. Average precision score according to each nonmaximum suppression’s intersection over union threshold (0.2–0.9)

NMS’s IoU threshold	AP score for IoU threshold
0.2	0.829
0.3	0.836
0.4	0.84
0.5	0.844
0.6	0.859
0.7	0.837
0.8	0.82
0.9	0.735
Abbreviations: AP, average precision; IoU, intersection over union; NMS, nonmaximum suppression

Discussion

In our study, we developed a training model that detects the location of the hepatocystic triangle based on the YOLOv5 architecture. To contextualize our findings, we examined the characteristics of different object detection algorithms. YOLOv5 has been reported to outperform its predecessor, YOLOv3, with an approximate 10%–15% increase in AP while maintaining computational efficiency.^20,21 In comparison with alternative detection frameworks, such as EfficientDet-D0 (Google, Mountain View, California, United States) and Faster R-CNN (Microsoft, Redmont, Washington, United States), YOLOv5 offers a favorable trade-off, providing high detection accuracy without significant sacrifices in processing speed. Considering these factors, our selection of YOLOv5 represents a well-balanced choice, offering both accuracy and efficiency for real-time hepatocystic triangle detection in surgical procedures.

ICG fluorescence–guided laparoscopy provides several advantages, including enhanced tissue contrast, improved accuracy of bile duct identification, and better intraoperative lymph node retrieval.²⁴ This approach is minimally invasive, does not require ionizing radiation, and integrates seamlessly with current laparoscopic systems equipped with NIR modules. Currently, cholangiography using ICG is used in various hepatobiliary surgeries. The advantage of this technique is that it allows for accurate identification of the bile duct even when it cannot be distinguished from the surrounding soft tissue, and it enables visualization of anatomical structures and bile duct variations without dissection or tissue damage.¹⁹ Unlike conventional cholangiography, ICG-based NIRC during LC is especially convenient because it can check the bile duct damage in real time without requiring radiology equipment, specialized personnel, irradiation, and additional costs.²⁵ Therefore, this study applied the laparoscopic images of NIRC using ICG to develop a prediction model of the hepatocystic triangle’s localization.

LC is one of the most commonly performed surgical procedures worldwide. However, the incidence of iatrogenic bile duct damage has increased significantly from 0.2% to 3.5%, according to previous studies.^26,27 As uncontrolled bile leak after bile duct damage is a fatal risk, every surgeon must pay close attention to the hepatocystic triangle during LC.^28,29 Moreover, even experienced surgeons may accidentally cause bile duct damage due to misrecognition of its anatomy. In fact, in a survey study of 600 skilled surgeons, 72.3% experienced bile duct damage and 40.5% admitted that they had caused bile duct damage by misrecognizing its structure.³⁰ Therefore, to overcome this problem, image visualization research based on AI offers a promising alternative, as it may help identify accurate images using repeated learning models and minimize human recognition errors.

Currently, studies utilizing AI to detect the location of the bile duct in LC are investigating 3 different approaches. The first model identifies the hepatocystic triangle after bile duct dissection, and develops a detection model based on accurate images obtained in this process.^12,17 However, complete dissection of the hepatocystic triangle is not commonly performed in clinical practice; therefore, the obtained image data may be far from the actual surgical field. Furthermore, because these images are acquired postdissection, bias may be introduced depending on the extent of tissue manipulation, making them unsuitable for predicting the hepatocystic triangle’s location before dissection.

The second model detects the location of the hepatocystic triangle by recognizing the surrounding anatomical landmarks.^16,31 This allows for identification of specific landmarks without exfoliating the bile duct. Since multiple landmarks are used in this model, it can provide objective data on the hepatocystic triangle’s location. However, this model does not yield precise information about the hepatocystic triangle itself. Indeed, the surgeon can predict the location of the bile duct through the information on the surrounding landmarks, which are the operation’s critical points. However, it is not known whether the actual hepatocytic triangle is located in the surrounding landmarks. In particular, caution should be exercised when using this model during surgery, as deterioration of the hepatocystic triangle may occur in the cases of previous surgery or adhesions.

The third model distinguishes and predicts the “safe zone” (go zone) and “risk zone” (no-go zone) based on the location of the hepatocystic triangle.¹¹ This method effectively assists surgeons by color-coding safe and dangerous areas, facilitating safer LC performance. However, the approach is time-consuming, and requires extensive manual annotation by experienced surgeons across thousands of images.

In this study, we developed a model for the detection of the hepatocystic triangle’s location using ICG-based cholangiography. The annotation process was performed by confirming the actual localization of the hepatocytic triangle. Therefore, our image recognition model is expected to be more accurate than other existing models. In addition, as the bile duct was detected without dissection during LC, this model could help surgeons identify the duct’s location before tissue dissection. Moreover, as shown in Figure 1, we annotated the dataset by overlaying the ICG fluorescence regions from 2 corresponding images, which simplified the annotation process. We aimed to find the optimal predictive value using the IoU threshold and AP score. Among IoU thresholds ranging from 0.2 to 0.9, the 0.6 threshold showed the optimal AP score.

There are several limitations to be acknowledged. First, even though ICG cholangiography is useful for bile duct visualization, its tissue penetration depth is limited to 5–10 mm.^32,33 In addition, ICG metabolism varies depending on liver function.¹⁹ Therefore, to obtain high-quality images, a good view of the structure and adequate patient selection must be ensured. To minimize the limitations, our study enrolled patients with normal liver function, without a surgical history or severe adhesions, and no inflammation of the gallbladder. Second, this model was trained on still images from a limited number of patients rather than video sequences. In clinical practice, predictive modeling based on continuous video data would be more useful. We plan to develop a video-based predictive model using a larger NIRC dataset in future research. Finally, cross-validation testing has not yet been performed, which limits the generalizability of our findings. Moreover, quantitative comparisons with other studies remain challenging. To date, investigations have been conducted only in a few centers with limited cohorts,^11,12,16 and robust comparative analyses will require multiple studies with adequately powered sample sizes. As this study represents a preliminary step, we plan to evaluate the model’s predictive performance in future experiments under more clinically realistic conditions.

To further assess the model’s performance, we examined the cases where the YOLOv5 model failed to correctly detect the hepatocystic triangle. We identified 2 primary failure scenarios: 1) low-light conditions reducing contrast in ICG fluorescence (n = 2) and 2) anatomical variations leading to inconsistencies with the training data (n = 1). Low-light conditions have been attributed to impaired tissue penetration in the patients with thick peritoneal fat or peritoneal adhesions secondary to inflammation.¹⁹ Therefore, in individuals with severe cholecystitis or obesity, NIRC may fail to delineate the precise anatomy of the extrahepatic bile ducts buried within dense connective tissues prior to dissection of the hepatocystic triangle. Wang et al³² reported that BMI above 25 kg/m² reduced the visibility of biliary structures during NIRC, and similar findings were observed by Dip et al.³⁴ In the current study, only the patients with grade 2 or lower cholecystitis were included. However, 9 patients had BMI equal to or greater than 25 kg/m², among whom 2 cases of image-detection failure under low-light conditions were recorded. Additionally, 1 case of detection failure occurred due to biliary tract variation. Preoperative magnetic resonance cholangiopancreatography showed a low-lying hepatocystic triangle, characterized by early cystic duct origin within the pancreas. Addressing these limitations in future work could involve incorporating more diverse training datasets, refining the preprocessing pipeline to enhance contrast in low-light conditions, and developing adaptive algorithms that account for anatomical variations.

Conclusions

To the best of our knowledge, this is the first predictive model used for locating the hepatocytic triangle via intraoperative ICG cholangiography to prevent bile duct damage during LC. We expect that our model will be helpful in clinical practice, as it can predict the bile duct’s actual location before tissue dissection. We plan to develop a more advanced model that can be used in clinical practice by analyzing surgical fields.