Authors: Çağhan Köksal, Ghazal Ghazaei, Felix Holm, Azade Farshad, Nassir Navab
Abstract: Graph-based holistic scene representations facilitate surgical workflow
understanding and have recently demonstrated significant success. However, this
task is often hindered by the limited availability of densely annotated
surgical scene data. In this work, we introduce an end-to-end framework for the
generation and optimization of surgical scene graphs on a downstream task. Our
approach leverages the flexibility of graph-based spectral clustering and the
generalization capability of foundation models to generate unsupervised scene
graphs with learnable properties. We reinforce the initial spatial graph with
sparse temporal connections using local matches between consecutive frames to
predict temporally consistent clusters across a temporal neighborhood. By
jointly optimizing the spatiotemporal relations and node features of the
dynamic scene graph with the downstream task of phase segmentation, we address
the costly and annotation-burdensome task of semantic scene comprehension and
scene graph generation in surgical videos using only weak surgical phase
labels. Further, by incorporating effective intermediate scene representation
disentanglement steps within the pipeline, our solution outperforms the SOTA on
the CATARACTS dataset by 8% accuracy and 10% F1 score in surgical workflow
recognition
Source: http://arxiv.org/abs/2407.20214v1
