Computer Science > Computation and Language
[Submitted on 17 May 2024 (v1), revised 31 May 2024 (this version, v6), latest version 15 Jul 2024 (v8)]
Title:SPOR: A Comprehensive and Practical Evaluation Method for Compositional Generalization in Data-to-Text Generation
View PDF HTML (experimental)Abstract:Compositional generalization is an important ability of language models and has many different manifestations. For data-to-text generation, previous research on this ability is limited to a single manifestation called Systematicity and lacks consideration of large language models (LLMs), which cannot fully cover practical application scenarios. In this work, we propose SPOR, a comprehensive and practical evaluation method for compositional generalization in data-to-text generation. SPOR includes four aspects of manifestations (Systematicity, Productivity, Order invariance, and Rule learnability) and allows high-quality evaluation without additional manual annotations based on existing datasets. We demonstrate SPOR on two different datasets and evaluate some existing language models including LLMs. We find that the models are deficient in various aspects of the evaluation and need further improvement. Our work shows the necessity for comprehensive research on different manifestations of compositional generalization in data-to-text generation and provides a framework for evaluation.
Submission history
From: Ziyao Xu [view email][v1] Fri, 17 May 2024 09:25:30 UTC (265 KB)
[v2] Mon, 20 May 2024 07:56:47 UTC (265 KB)
[v3] Tue, 21 May 2024 08:31:05 UTC (265 KB)
[v4] Wed, 22 May 2024 18:22:23 UTC (265 KB)
[v5] Sat, 25 May 2024 19:37:00 UTC (266 KB)
[v6] Fri, 31 May 2024 07:22:42 UTC (271 KB)
[v7] Wed, 19 Jun 2024 18:09:32 UTC (271 KB)
[v8] Mon, 15 Jul 2024 11:47:29 UTC (272 KB)
References & Citations
Bibliographic and Citation Tools
Bibliographic Explorer (What is the Explorer?)
Litmaps (What is Litmaps?)
scite Smart Citations (What are Smart Citations?)
Code, Data and Media Associated with this Article
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub (What is DagsHub?)
Gotit.pub (What is GotitPub?)
Papers with Code (What is Papers with Code?)
ScienceCast (What is ScienceCast?)
Demos
Recommenders and Search Tools
Influence Flower (What are Influence Flowers?)
Connected Papers (What is Connected Papers?)
CORE Recommender (What is CORE?)
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.