Conditional [MASK] Discrete Diffusion Language Model

Résumé

Although auto-regressive models excel in natural language processing, they often struggle to generate diverse text and provide limited controllability.Non-auto-regressive methods could be an alternative but often produce degenerate outputs and exhibit shortcomings in conditional generation.To address these challenges, we propose Diffusion-EAGS, a novel framework that integrates conditional masked language models into diffusion language models through the theoretical lens of a conditional Markov Random Field.In doing so, we propose entropy-adaptive Gibbs sampling and entropybased noise scheduling to counterbalance each model's shortcomings.Experimental results show that Diffusion-EAGS outperforms baselines and achieves the best quality-diversity tradeoff, demonstrating its effectiveness in nonautoregressive text generation.Context Jake was playing with his toys.He accidentally broke his favorite one.He cried a lot over it.His parents decided to replace it for him.Keyword not stop Jake just could not stop crying.Jake feel It made Jake feel So much better.would enjoy Jake said he would enjoy the new toy Context Neil was in Sofia, Bulgaria.He was enjoying a trip backpacking through Europe....He thought the food and culture in

Citer ce document

Accès au document

Texte intégral en lecture en ligne, réservé aux abonnés SPHAERO et aux membres de l'institution. Se connecter

Auteur(s)

Statistiques

Consultations : 1

Téléchargements : 0