Microsoft's Chinese chatbot inspired by images to write poetry

Xiaoice's verse is 'disgusting' say rival human poets

Microsoft’s chatbot Xiaoice does a lot more than other bots. She has presented the weather on live TV and now even composed a book of poems.

Xiaoice has written 12 million of poems in fact. In a paper on arXiv, researchers from Microsoft, National Taiwan University, and the University of Montreal explained that all text Xiaoice wrote is inspired by images.

“Given an image, we first extract a few keywords representing objects and sentiments perceived from the image. These keywords are then expanded to related ones based on their associations in human written poems. Finally, verses are generated gradually from the keywords using recurrent neural networks trained on existing poems,” according to the paper’s abstract.

Using images to conjure up text isn’t new. It’s used in caption contests and has been explored for machine translation and interpretability - maybe researchers can understand neural networks better if they explained themselves in plain text.

Xiaoice’s poem generating system is made up of several neural networks. The first step uses two convolutional neural networks (CNNs) to analyse images and comes up with a list of nouns to describe the image, whilst the other neural network spits out adjectives used as sentiment analysis. The CNNs were trained on ImageNet.

For example, given a picture of different vehicles on a road with buildings in the background, the system spits out nouns like city, street and adjectives such as busy and scared. These words are then passed to a recurrent neural network (RNN) to compose poems from.


The keyword generation is done by two CNNs. The poem generation side is made up of a recurrent neural network. Image credit: Cheng et al.

Each sentence in the poem is constructed around the key words. There are two levels to the RNN; one concentrates on predicting the next word in the sentence, and the other focuses on the overall structure of the sentences.

The RNN was trained on 2,027 modern Chinese poems with a total of 45,729 sentences and a character vocabulary size of 4,547 scraped from, a database containing thousands of classical and modern poems.

AI is too cold and emotionless

Here’s one translated in English taken from the book published by Cheers Publishing. The image next to is a majestic-looking elephant covered in dust and is called The World’s Tragic Protagonist.

The World’s Tragic Protagonist

  • I have failed to love my own life

    Yet I touched the spirit in your eyes

    My heart lives these times I shall tell the stories of my tears

  • Boundless life

    Never saw the recovery of earth

    The world’s tragic protagonist

    The people of those tragic times

Xiaoice seems to be pretty dark. It seems quite impressive, is it a gloomy poem about climate change?

Her other ones are less lucid. Here’s another one called She married many colours in the world next to a picture of a living room.

She married many colours in the world

The stars are shining for days

The red sun

See Miles of Angels in the world I was like a dream

Look at the stars that Twinkle

The Sun on the West Hill

That frog is in the shallow water

She married many colours in the world.

“The quality of the system can certainly be much improved. While many poems sounds good, some seem strange, with unrelated sentences,” Jian-Yun Nie, co-author of the paper and a professor at the University of Montreal, explained to The Register.

"This has to do with the techniques used: the association between concepts and sentences are learned from a set of human poems, and the system tries to produce verses according to what is learned. This is not truly the writing process of many poets."

“In general, a poet would have some ideas or feelings to express, and try to find sentences and structures that best express them. For Xiaoice, there is not really an internal feeling or sentiment to express. She is given a few keywords or an image, then the learned model is used to generate sentences.”


As the singularity approaches, neural network pens black metal album


“In my opinion, a big improvement could be made by endowing the system with a feeling (or a model of it), so that a poem becomes an expression of feelings rather than of words. However, it is a debatable question whether an AI system could have feeling or sentiment.”

The lack of emotion has horrified some human poets, according to South China Morning Post.

“It disgusted me with its slippery tone and rhythm. The sentences were aimless and superficial, lacking the inner logic for emotional expression,” said Yu Jian, a poet based in the Yunnan a province in southwestern China.

Nie, however, believed that people did care for a chatbot’s poems even if they are hit or miss.

“Poetry is a very special area. The reader should make sense of what is read. So the unexpected verses may make the reader wonder how to interpret them. This sometimes leads to a very pleasant feeling and reading experience. I believe that what people like in Xiaobing's poems is the unexpected elements.” ®

Other stories you might like

  • Stolen university credentials up for sale by Russian crooks, FBI warns
    Forget dark-web souks, thousands of these are already being traded on public bazaars

    Russian crooks are selling network credentials and virtual private network access for a "multitude" of US universities and colleges on criminal marketplaces, according to the FBI.

    According to a warning issued on Thursday, these stolen credentials sell for thousands of dollars on both dark web and public internet forums, and could lead to subsequent cyberattacks against individual employees or the schools themselves.

    "The exposure of usernames and passwords can lead to brute force credential stuffing computer network attacks, whereby attackers attempt logins across various internet sites or exploit them for subsequent cyber attacks as criminal actors take advantage of users recycling the same credentials across multiple accounts, internet sites, and services," the Feds' alert [PDF] said.

    Continue reading
  • Big Tech loves talking up privacy – while trying to kill privacy legislation
    Study claims Amazon, Apple, Google, Meta, Microsoft work to derail data rules

    Amazon, Apple, Google, Meta, and Microsoft often support privacy in public statements, but behind the scenes they've been working through some common organizations to weaken or kill privacy legislation in US states.

    That's according to a report this week from news non-profit The Markup, which said the corporations hire lobbyists from the same few groups and law firms to defang or drown state privacy bills.

    The report examined 31 states when state legislatures were considering privacy legislation and identified 445 lobbyists and lobbying firms working on behalf of Amazon, Apple, Google, Meta, and Microsoft, along with industry groups like TechNet and the State Privacy and Security Coalition.

    Continue reading
  • SEC probes Musk for not properly disclosing Twitter stake
    Meanwhile, social network's board rejects resignation of one its directors

    America's financial watchdog is investigating whether Elon Musk adequately disclosed his purchase of Twitter shares last month, just as his bid to take over the social media company hangs in the balance. 

    A letter [PDF] from the SEC addressed to the tech billionaire said he "[did] not appear" to have filed the proper form detailing his 9.2 percent stake in Twitter "required 10 days from the date of acquisition," and asked him to provide more information. Musk's shares made him one of Twitter's largest shareholders. The letter is dated April 4, and was shared this week by the regulator.

    Musk quickly moved to try and buy the whole company outright in a deal initially worth over $44 billion. Musk sold a chunk of his shares in Tesla worth $8.4 billion and bagged another $7.14 billion from investors to help finance the $21 billion he promised to put forward for the deal. The remaining $25.5 billion bill was secured via debt financing by Morgan Stanley, Bank of America, Barclays, and others. But the takeover is not going smoothly.

    Continue reading

Biting the hand that feeds IT © 1998–2022