Personalizing Text-to-Image Diffusion Models by Fine-Tuning Classification for AI Applications

Rafael Hidalgo; Nesreen Salah; Rajiv Chandra Jetty; Anupama Jetty

doi:10.1007/978-3-031-47721-8_44

Personalizing Text-to-Image Diffusion Models by Fine-Tuning Classification for AI Applications

Rafael Hidalgo
, Nesreen Salah
, Rajiv Chandra Jetty
, Anupama Jetty

Computer Science

Montclair State University

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

Stable Diffusion is a captivating text-to-image model that generates images based on text input. However, a major challenge is that it is pretrained on a specific dataset, limiting its ability to generate images outside of the given data. In this paper, we propose to harness two models based on neural networks, Hypernetworks and DreamBooth, to allow the introduction of any image into Stable Diffusion, addressing versatility with minimal additional training data. This work targets AI applications such as augmenting next-generation multipurpose robots, enhancing human-robot collaboration, feeding intelligent tutoring systems, training autonomous cars, injecting subjects for photo personalization, producing high quality movie animations etc. It can contribute to AI in smart cities: facets such as smart living and smart mobility.

Original language	English
Title of host publication	Intelligent Systems and Applications - Proceedings of the 2023 Intelligent Systems Conference IntelliSys Volume 1
Editors	Kohei Arai
Publisher	Springer Science and Business Media Deutschland GmbH
Pages	642-658
Number of pages	17
ISBN (Print)	9783031477201
DOIs	https://doi.org/10.1007/978-3-031-47721-8_44
State	Published - 2024
Event	Intelligent Systems Conference, IntelliSys 2023 - Amsterdam, Netherlands Duration: Sep 7 2023 → Sep 8 2023

Publication series

Name	Lecture Notes in Networks and Systems
Volume	822

Conference

Conference	Intelligent Systems Conference, IntelliSys 2023
Country/Territory	Netherlands
City	Amsterdam
Period	9/7/23 → 9/8/23

UN SDGs

This output contributes to the following UN Sustainable Development Goals (SDGs)

SDG 11 Sustainable Cities and Communities

ASJC Scopus subject areas

Control and Systems Engineering
Signal Processing
Computer Networks and Communications

Keywords

ANN
Data mining
Image processing
Movie animations
Photo personalization
Stable diffusion
Text-to-image creation

Access to Document

10.1007/978-3-031-47721-8_44

Cite this

Hidalgo, R., Salah, N., Chandra Jetty, R., & Jetty, A. (2024). Personalizing Text-to-Image Diffusion Models by Fine-Tuning Classification for AI Applications. In K. Arai (Ed.), Intelligent Systems and Applications - Proceedings of the 2023 Intelligent Systems Conference IntelliSys Volume 1 (pp. 642-658). (Lecture Notes in Networks and Systems; Vol. 822). Springer Science and Business Media Deutschland GmbH. https://doi.org/10.1007/978-3-031-47721-8_44

Hidalgo, Rafael ; Salah, Nesreen ; Chandra Jetty, Rajiv et al. / Personalizing Text-to-Image Diffusion Models by Fine-Tuning Classification for AI Applications. Intelligent Systems and Applications - Proceedings of the 2023 Intelligent Systems Conference IntelliSys Volume 1. editor / Kohei Arai. Springer Science and Business Media Deutschland GmbH, 2024. pp. 642-658 (Lecture Notes in Networks and Systems).

@inproceedings{bdb25c988d384945a6f70dda311ff1fe,

title = "Personalizing Text-to-Image Diffusion Models by Fine-Tuning Classification for AI Applications",

abstract = "Stable Diffusion is a captivating text-to-image model that generates images based on text input. However, a major challenge is that it is pretrained on a specific dataset, limiting its ability to generate images outside of the given data. In this paper, we propose to harness two models based on neural networks, Hypernetworks and DreamBooth, to allow the introduction of any image into Stable Diffusion, addressing versatility with minimal additional training data. This work targets AI applications such as augmenting next-generation multipurpose robots, enhancing human-robot collaboration, feeding intelligent tutoring systems, training autonomous cars, injecting subjects for photo personalization, producing high quality movie animations etc. It can contribute to AI in smart cities: facets such as smart living and smart mobility.",

keywords = "ANN, Data mining, Image processing, Movie animations, Photo personalization, Stable diffusion, Text-to-image creation",

author = "Rafael Hidalgo and Nesreen Salah and \{Chandra Jetty\}, Rajiv and Anupama Jetty",

note = "Publisher Copyright: {\textcopyright} 2024, The Author(s), under exclusive license to Springer Nature Switzerland AG.; Intelligent Systems Conference, IntelliSys 2023 ; Conference date: 07-09-2023 Through 08-09-2023",

year = "2024",

doi = "10.1007/978-3-031-47721-8\_44",

language = "English",

isbn = "9783031477201",

series = "Lecture Notes in Networks and Systems",

publisher = "Springer Science and Business Media Deutschland GmbH",

pages = "642--658",

editor = "Kohei Arai",

booktitle = "Intelligent Systems and Applications - Proceedings of the 2023 Intelligent Systems Conference IntelliSys Volume 1",

}

Hidalgo, R, Salah, N, Chandra Jetty, R & Jetty, A 2024, Personalizing Text-to-Image Diffusion Models by Fine-Tuning Classification for AI Applications. in K Arai (ed.), Intelligent Systems and Applications - Proceedings of the 2023 Intelligent Systems Conference IntelliSys Volume 1. Lecture Notes in Networks and Systems, vol. 822, Springer Science and Business Media Deutschland GmbH, pp. 642-658, Intelligent Systems Conference, IntelliSys 2023, Amsterdam, Netherlands, 9/7/23. https://doi.org/10.1007/978-3-031-47721-8_44

Personalizing Text-to-Image Diffusion Models by Fine-Tuning Classification for AI Applications. / Hidalgo, Rafael; Salah, Nesreen; Chandra Jetty, Rajiv et al.
Intelligent Systems and Applications - Proceedings of the 2023 Intelligent Systems Conference IntelliSys Volume 1. ed. / Kohei Arai. Springer Science and Business Media Deutschland GmbH, 2024. p. 642-658 (Lecture Notes in Networks and Systems; Vol. 822).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Personalizing Text-to-Image Diffusion Models by Fine-Tuning Classification for AI Applications

AU - Hidalgo, Rafael

AU - Salah, Nesreen

AU - Chandra Jetty, Rajiv

AU - Jetty, Anupama

PY - 2024

Y1 - 2024

N2 - Stable Diffusion is a captivating text-to-image model that generates images based on text input. However, a major challenge is that it is pretrained on a specific dataset, limiting its ability to generate images outside of the given data. In this paper, we propose to harness two models based on neural networks, Hypernetworks and DreamBooth, to allow the introduction of any image into Stable Diffusion, addressing versatility with minimal additional training data. This work targets AI applications such as augmenting next-generation multipurpose robots, enhancing human-robot collaboration, feeding intelligent tutoring systems, training autonomous cars, injecting subjects for photo personalization, producing high quality movie animations etc. It can contribute to AI in smart cities: facets such as smart living and smart mobility.

AB - Stable Diffusion is a captivating text-to-image model that generates images based on text input. However, a major challenge is that it is pretrained on a specific dataset, limiting its ability to generate images outside of the given data. In this paper, we propose to harness two models based on neural networks, Hypernetworks and DreamBooth, to allow the introduction of any image into Stable Diffusion, addressing versatility with minimal additional training data. This work targets AI applications such as augmenting next-generation multipurpose robots, enhancing human-robot collaboration, feeding intelligent tutoring systems, training autonomous cars, injecting subjects for photo personalization, producing high quality movie animations etc. It can contribute to AI in smart cities: facets such as smart living and smart mobility.

KW - ANN

KW - Data mining

KW - Image processing

KW - Movie animations

KW - Photo personalization

KW - Stable diffusion

KW - Text-to-image creation

UR - https://www.scopus.com/pages/publications/85182514364

U2 - 10.1007/978-3-031-47721-8_44

DO - 10.1007/978-3-031-47721-8_44

M3 - Conference contribution

SN - 9783031477201

T3 - Lecture Notes in Networks and Systems

SP - 642

EP - 658

BT - Intelligent Systems and Applications - Proceedings of the 2023 Intelligent Systems Conference IntelliSys Volume 1

A2 - Arai, Kohei

PB - Springer Science and Business Media Deutschland GmbH

T2 - Intelligent Systems Conference, IntelliSys 2023

Y2 - 7 September 2023 through 8 September 2023

ER -

Hidalgo R, Salah N, Chandra Jetty R, Jetty A. Personalizing Text-to-Image Diffusion Models by Fine-Tuning Classification for AI Applications. In Arai K, editor, Intelligent Systems and Applications - Proceedings of the 2023 Intelligent Systems Conference IntelliSys Volume 1. Springer Science and Business Media Deutschland GmbH. 2024. p. 642-658. (Lecture Notes in Networks and Systems). doi: 10.1007/978-3-031-47721-8_44

Personalizing Text-to-Image Diffusion Models by Fine-Tuning Classification for AI Applications

Abstract

Publication series

Conference

UN SDGs

ASJC Scopus subject areas

Keywords

Access to Document

Other files and links

Fingerprint

Cite this