Authors: Mehrnoosh Mirtaheri, Nikhil Varghese, Chandra Khatri, Amol Kelkar
Abstract: Task-oriented dialogue systems rely on predefined conversation schemes
(dialogue flows) often represented as directed acyclic graphs. These flows can
be manually designed or automatically generated from previously recorded
conversations. Due to variations in domain expertise or reliance on different
sets of prior conversations, these dialogue flows can manifest in significantly
different graph structures. Despite their importance, there is no standard
method for evaluating the quality of dialogue flows. We introduce FuDGE (Fuzzy
Dialogue-Graph Edit Distance), a novel metric that evaluates dialogue flows by
assessing their structural complexity and representational coverage of the
conversation data. FuDGE measures how well individual conversations align with
a flow and, consequently, how well a set of conversations is represented by the
flow overall. Through extensive experiments on manually configured flows and
flows generated by automated techniques, we demonstrate the effectiveness of
FuDGE and its evaluation framework. By standardizing and optimizing dialogue
flows, FuDGE enables conversational designers and automated techniques to
achieve higher levels of efficiency and automation.
Source: http://arxiv.org/abs/2411.10416v1
