The Fabrication of Reality and Fantasy: Scene Generation with LLM-Assisted Prompt Interpretation

Yi Yao; Chan Feng Hsu; Jhe Hao Lin; Hongxia Xie; Terence Lin; Yi Ning Huang; Hong Han Shuai; Wen-Huang Cheng

doi:10.1007/978-3-031-72670-5_24

The Fabrication of Reality and Fantasy: Scene Generation with LLM-Assisted Prompt Interpretation

Yi Yao^*, Chan Feng Hsu, Jhe Hao Lin, Hongxia Xie, Terence Lin, Yi Ning Huang, Hong Han Shuai, Wen-Huang Cheng

^*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

In spite of recent advancements in text-to-image generation, limitations persist in handling complex and imaginative prompts due to the restricted diversity and complexity of training data. This work explores how diffusion models can generate images from prompts requiring artistic creativity or specialized knowledge. We introduce the Realistic-Fantasy Benchmark (RFBench), a novel evaluation framework blending realistic and fantastical scenarios. To address these challenges, we propose the Realistic-Fantasy Network (RFNet), a training-free approach integrating diffusion models with LLMs. Extensive human evaluations and GPT-based compositional assessments demonstrate our approach’s superiority over state-of-the-art methods. Our code and dataset is available at https://leo81005.github.io/Reality-and-Fantasy/.

Original language	English
Title of host publication	Computer Vision – ECCV 2024 - 18th European Conference, Proceedings
Editors	Aleš Leonardis, Elisa Ricci, Stefan Roth, Olga Russakovsky, Torsten Sattler, Gül Varol
Publisher	Springer Science and Business Media Deutschland GmbH
Pages	422-438
Number of pages	17
ISBN (Print)	9783031726699
DOIs	https://doi.org/10.1007/978-3-031-72670-5_24
State	Published - 2025
Event	18th European Conference on Computer Vision, ECCV 2024 - Milan, Italy Duration: 29 Sep 2024 → 4 Oct 2024

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	15080 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	18th European Conference on Computer Vision, ECCV 2024
Country/Territory	Italy
City	Milan
Period	29/09/24 → 4/10/24

Keywords

Diffusion Model
Large Language Models (LLMs)
Realistic-Fantasy Benchmark
Text-to-image Generation

Access to Document

10.1007/978-3-031-72670-5_24

Cite this

Yao, Y., Hsu, C. F., Lin, J. H., Xie, H., Lin, T., Huang, Y. N., Shuai, H. H., & Cheng, W.-H. (2025). The Fabrication of Reality and Fantasy: Scene Generation with LLM-Assisted Prompt Interpretation. In A. Leonardis, E. Ricci, S. Roth, O. Russakovsky, T. Sattler, & G. Varol (Eds.), Computer Vision – ECCV 2024 - 18th European Conference, Proceedings (pp. 422-438). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 15080 LNCS). Springer Science and Business Media Deutschland GmbH. https://doi.org/10.1007/978-3-031-72670-5_24

Yao, Yi ; Hsu, Chan Feng ; Lin, Jhe Hao et al. / The Fabrication of Reality and Fantasy : Scene Generation with LLM-Assisted Prompt Interpretation. Computer Vision – ECCV 2024 - 18th European Conference, Proceedings. editor / Aleš Leonardis ; Elisa Ricci ; Stefan Roth ; Olga Russakovsky ; Torsten Sattler ; Gül Varol. Springer Science and Business Media Deutschland GmbH, 2025. pp. 422-438 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{9cd9693d99ee43388d0d89237b5c0922,

title = "The Fabrication of Reality and Fantasy: Scene Generation with LLM-Assisted Prompt Interpretation",

abstract = "In spite of recent advancements in text-to-image generation, limitations persist in handling complex and imaginative prompts due to the restricted diversity and complexity of training data. This work explores how diffusion models can generate images from prompts requiring artistic creativity or specialized knowledge. We introduce the Realistic-Fantasy Benchmark (RFBench), a novel evaluation framework blending realistic and fantastical scenarios. To address these challenges, we propose the Realistic-Fantasy Network (RFNet), a training-free approach integrating diffusion models with LLMs. Extensive human evaluations and GPT-based compositional assessments demonstrate our approach{\textquoteright}s superiority over state-of-the-art methods. Our code and dataset is available at https://leo81005.github.io/Reality-and-Fantasy/.",

keywords = "Diffusion Model, Large Language Models (LLMs), Realistic-Fantasy Benchmark, Text-to-image Generation",

author = "Yi Yao and Hsu, {Chan Feng} and Lin, {Jhe Hao} and Hongxia Xie and Terence Lin and Huang, {Yi Ning} and Shuai, {Hong Han} and Wen-Huang Cheng",

note = "Publisher Copyright: {\textcopyright} The Author(s), under exclusive license to Springer Nature Switzerland AG 2025.; 18th European Conference on Computer Vision, ECCV 2024 ; Conference date: 29-09-2024 Through 04-10-2024",

year = "2025",

doi = "10.1007/978-3-031-72670-5_24",

language = "English",

isbn = "9783031726699",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer Science and Business Media Deutschland GmbH",

pages = "422--438",

editor = "Ale{\v s} Leonardis and Elisa Ricci and Stefan Roth and Olga Russakovsky and Torsten Sattler and G{\"u}l Varol",

booktitle = "Computer Vision – ECCV 2024 - 18th European Conference, Proceedings",

address = "德國",

}

Yao, Y, Hsu, CF, Lin, JH, Xie, H, Lin, T, Huang, YN, Shuai, HH & Cheng, W-H 2025, The Fabrication of Reality and Fantasy: Scene Generation with LLM-Assisted Prompt Interpretation. in A Leonardis, E Ricci, S Roth, O Russakovsky, T Sattler & G Varol (eds), Computer Vision – ECCV 2024 - 18th European Conference, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 15080 LNCS, Springer Science and Business Media Deutschland GmbH, pp. 422-438, 18th European Conference on Computer Vision, ECCV 2024, Milan, Italy, 29/09/24. https://doi.org/10.1007/978-3-031-72670-5_24

The Fabrication of Reality and Fantasy: Scene Generation with LLM-Assisted Prompt Interpretation. / Yao, Yi; Hsu, Chan Feng; Lin, Jhe Hao et al.
Computer Vision – ECCV 2024 - 18th European Conference, Proceedings. ed. / Aleš Leonardis; Elisa Ricci; Stefan Roth; Olga Russakovsky; Torsten Sattler; Gül Varol. Springer Science and Business Media Deutschland GmbH, 2025. p. 422-438 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 15080 LNCS).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - The Fabrication of Reality and Fantasy

T2 - 18th European Conference on Computer Vision, ECCV 2024

AU - Yao, Yi

AU - Hsu, Chan Feng

AU - Lin, Jhe Hao

AU - Xie, Hongxia

AU - Lin, Terence

AU - Huang, Yi Ning

AU - Shuai, Hong Han

AU - Cheng, Wen-Huang

N1 - Publisher Copyright: © The Author(s), under exclusive license to Springer Nature Switzerland AG 2025.

PY - 2025

Y1 - 2025

N2 - In spite of recent advancements in text-to-image generation, limitations persist in handling complex and imaginative prompts due to the restricted diversity and complexity of training data. This work explores how diffusion models can generate images from prompts requiring artistic creativity or specialized knowledge. We introduce the Realistic-Fantasy Benchmark (RFBench), a novel evaluation framework blending realistic and fantastical scenarios. To address these challenges, we propose the Realistic-Fantasy Network (RFNet), a training-free approach integrating diffusion models with LLMs. Extensive human evaluations and GPT-based compositional assessments demonstrate our approach’s superiority over state-of-the-art methods. Our code and dataset is available at https://leo81005.github.io/Reality-and-Fantasy/.

AB - In spite of recent advancements in text-to-image generation, limitations persist in handling complex and imaginative prompts due to the restricted diversity and complexity of training data. This work explores how diffusion models can generate images from prompts requiring artistic creativity or specialized knowledge. We introduce the Realistic-Fantasy Benchmark (RFBench), a novel evaluation framework blending realistic and fantastical scenarios. To address these challenges, we propose the Realistic-Fantasy Network (RFNet), a training-free approach integrating diffusion models with LLMs. Extensive human evaluations and GPT-based compositional assessments demonstrate our approach’s superiority over state-of-the-art methods. Our code and dataset is available at https://leo81005.github.io/Reality-and-Fantasy/.

KW - Diffusion Model

KW - Large Language Models (LLMs)

KW - Realistic-Fantasy Benchmark

KW - Text-to-image Generation

UR - http://www.scopus.com/inward/record.url?scp=85206378656&partnerID=8YFLogxK

U2 - 10.1007/978-3-031-72670-5_24

DO - 10.1007/978-3-031-72670-5_24

M3 - Conference contribution

AN - SCOPUS:85206378656

SN - 9783031726699

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 422

EP - 438

BT - Computer Vision – ECCV 2024 - 18th European Conference, Proceedings

A2 - Leonardis, Aleš

A2 - Ricci, Elisa

A2 - Roth, Stefan

A2 - Russakovsky, Olga

A2 - Sattler, Torsten

A2 - Varol, Gül

PB - Springer Science and Business Media Deutschland GmbH

Y2 - 29 September 2024 through 4 October 2024

ER -

Yao Y, Hsu CF, Lin JH, Xie H, Lin T, Huang YN et al. The Fabrication of Reality and Fantasy: Scene Generation with LLM-Assisted Prompt Interpretation. In Leonardis A, Ricci E, Roth S, Russakovsky O, Sattler T, Varol G, editors, Computer Vision – ECCV 2024 - 18th European Conference, Proceedings. Springer Science and Business Media Deutschland GmbH. 2025. p. 422-438. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-031-72670-5_24