This article is more than 1 year old

Microsoft's Chinese chatbot inspired by images to write poetry

Xiaoice's verse is 'disgusting' say rival human poets

Microsoft’s chatbot Xiaoice does a lot more than other bots. She has presented the weather on live TV and now even composed a book of poems.

Xiaoice has written 12 million of poems in fact. In a paper on arXiv, researchers from Microsoft, National Taiwan University, and the University of Montreal explained that all text Xiaoice wrote is inspired by images.

“Given an image, we first extract a few keywords representing objects and sentiments perceived from the image. These keywords are then expanded to related ones based on their associations in human written poems. Finally, verses are generated gradually from the keywords using recurrent neural networks trained on existing poems,” according to the paper’s abstract.

Using images to conjure up text isn’t new. It’s used in caption contests and has been explored for machine translation and interpretability - maybe researchers can understand neural networks better if they explained themselves in plain text.

Xiaoice’s poem generating system is made up of several neural networks. The first step uses two convolutional neural networks (CNNs) to analyse images and comes up with a list of nouns to describe the image, whilst the other neural network spits out adjectives used as sentiment analysis. The CNNs were trained on ImageNet.

For example, given a picture of different vehicles on a road with buildings in the background, the system spits out nouns like city, street and adjectives such as busy and scared. These words are then passed to a recurrent neural network (RNN) to compose poems from.


The keyword generation is done by two CNNs. The poem generation side is made up of a recurrent neural network. Image credit: Cheng et al.

Each sentence in the poem is constructed around the key words. There are two levels to the RNN; one concentrates on predicting the next word in the sentence, and the other focuses on the overall structure of the sentences.

The RNN was trained on 2,027 modern Chinese poems with a total of 45,729 sentences and a character vocabulary size of 4,547 scraped from, a database containing thousands of classical and modern poems.

AI is too cold and emotionless

Here’s one translated in English taken from the book published by Cheers Publishing. The image next to is a majestic-looking elephant covered in dust and is called The World’s Tragic Protagonist.

The World’s Tragic Protagonist

  • I have failed to love my own life

    Yet I touched the spirit in your eyes

    My heart lives these times I shall tell the stories of my tears

  • Boundless life

    Never saw the recovery of earth

    The world’s tragic protagonist

    The people of those tragic times

Xiaoice seems to be pretty dark. It seems quite impressive, is it a gloomy poem about climate change?

Her other ones are less lucid. Here’s another one called She married many colours in the world next to a picture of a living room.

She married many colours in the world

The stars are shining for days

The red sun

See Miles of Angels in the world I was like a dream

Look at the stars that Twinkle

The Sun on the West Hill

That frog is in the shallow water

She married many colours in the world.

“The quality of the system can certainly be much improved. While many poems sounds good, some seem strange, with unrelated sentences,” Jian-Yun Nie, co-author of the paper and a professor at the University of Montreal, explained to The Register.

"This has to do with the techniques used: the association between concepts and sentences are learned from a set of human poems, and the system tries to produce verses according to what is learned. This is not truly the writing process of many poets."

“In general, a poet would have some ideas or feelings to express, and try to find sentences and structures that best express them. For Xiaoice, there is not really an internal feeling or sentiment to express. She is given a few keywords or an image, then the learned model is used to generate sentences.”


As the singularity approaches, neural network pens black metal album


“In my opinion, a big improvement could be made by endowing the system with a feeling (or a model of it), so that a poem becomes an expression of feelings rather than of words. However, it is a debatable question whether an AI system could have feeling or sentiment.”

The lack of emotion has horrified some human poets, according to South China Morning Post.

“It disgusted me with its slippery tone and rhythm. The sentences were aimless and superficial, lacking the inner logic for emotional expression,” said Yu Jian, a poet based in the Yunnan a province in southwestern China.

Nie, however, believed that people did care for a chatbot’s poems even if they are hit or miss.

“Poetry is a very special area. The reader should make sense of what is read. So the unexpected verses may make the reader wonder how to interpret them. This sometimes leads to a very pleasant feeling and reading experience. I believe that what people like in Xiaobing's poems is the unexpected elements.” ®

More about


Send us news

Other stories you might like