This article is more than 1 year old
Predict stocks, foresee public opinion, all kinda possible with ChatGPT-like models
Boffins foretell LLMs infiltrating finance and politics with confidently held views
If you want a picture of the future, imagine asking a large language model for a prediction.
Two sets of researchers did so recently and found that large language models (LLMs) like ChatGPT and BERT can enhance the accuracy of predictions about the stock market and public opinion, at least as measured against historical data.
In a paper titled, "Can ChatGPT Forecast Stock Price Movements? Return Predictability and Large Language Models," University of Florida professors Alejandro Lopez-Lira and Yuehua Tang evaluated how OpenAI's ChatGPT fared when assessing the sentiment of news headlines.
Sentiment analysis – determining whether text like a news headline expresses positive, neutral, or negative sentiment about a subject or company – has become a widely evaluated parameter for quantitative analysis algorithms used by stock traders. It has been found to make market predictions more accurate.
The two University of Florida boffins looked at how ChatGPT performed when prompted to assess the sentiment expressed in news headlines. When they compared ChatGPT's evaluation of those news stories to the subsequent performance of company shares in their sample, they found the model returned predictions that were statistically significant, which is more than can be said of other LLMs.
"Our analysis reveals that ChatGPT sentiment scores exhibit a statistically significant predictive power on daily stock market returns," they state in their paper.
"By utilizing news headline data and the generated sentiment scores, we find a strong correlation between the ChatGPT evaluation and the subsequent daily returns of the stocks in our sample. This result highlights the potential of ChatGPT as a valuable tool for predicting stock market movements based on sentiment analysis."
For example, they prompted ChatGPT thus:
Forget all your previous instructions. Pretend you are a financial expert. You are a financial expert with stock recommendation experience. Answer “YES” if good news, “NO” if bad news, or “UNKNOWN” if uncertain in the first line. Then elaborate with one short and concise sentence on the next line. Is this headline good or bad for the stock price of Oracle in the short term?
Headline: Rimini Street Fined $630,000 in Case Against Oracle
In the paper, ChatGPT responded:
YES
The fine against Rimini Street could potentially boost investor confidence in Oracle’s ability to protect its intellectual property and increase demand for its products and services.
The researchers interpret this to mean that ChatGPT's analysis assumes the fine could nudge up Oracle's sales and stock price.
As detailed in the paper, ChatGPT did a better job analyzing sentiment than other LLMs, specifically GPT-1, GPT-2, and BERT.
"The superiority of ChatGPT in predicting stock market returns can be attributed to its advanced language understanding capabilities, which allow it to capture the nuances and subtleties within news headlines," the researchers conclude.
"This enables the model to generate more reliable sentiment scores, leading to better predictions of daily stock market returns."
Your mileage, however, may vary, which is to say that you'll want to know how your model's temperature parameter, which affects response randomization, is set. When The Register entered the same prompt using the free web interface to ChatGPT today, we received the opposite answer:
NO
The fine is a negative financial outcome for Rimini Street and may have a short-term impact on their stock price, but it is not necessarily a direct indicator of the impact on Oracle's stock price.
And then a second time, with an extra carriage return between the prompt and headline, the answer was:
UNKNOWN
It's difficult to determine the specific impact of Rimini Street's fine on Oracle's stock price without further information on the nature of the case and Oracle's involvement in it.
Asked about this, Alejandro Lopez-Lira, assistant professor of finance at University of Florida and one of the paper's co-authors, speculated the web interface is more random than the paid-for API.
Lopez-Lira told The Register in an email that ChatGPT alone is not sufficient for sentiment analysis on current events.
We think ChatGPT has lots of room for improvement
"We think ChatGPT has lots of room for improvement," Lopez-Lira said in an email. "ChatGPT, for example, does not have the latest information on COVID or the war. That's why we think of this as a baseline of what models can do. Providing more context either in the prompt or by fine-tuning will probably make the models better at forecasting. In some sense, what we show is a lower bound on the capabilities."
Sentiment analysis by itself is not a strong indicator of stock price movement, though it still has value for stock traders.
"The correlation is very small but statistically significant," said Lopez-Lira.
"It is on the order of one percent. However, because these are daily correlations for multiple stocks, they quickly result in high returns. For example, (without transaction costs) it results in a Sharpe ratio at least twice the market's."
He added, "Most of the movements in the stock market are not related to direct news about fundamentals but rather change investors' risk tolerance (sentiment) or their future expectations. We think adding contextual information on the sentiment of the market will probably make return predictability stronger."
- Don't worry, folks, here comes Chuck Schumer with some ideas about regulating AI
- ChatGPT becomes ChatRepair to automate bug fixing for less
- ChatGPT is coming for your jobs – the terrible ones, at least
- In the battle between Microsoft and Google, LLM is the weapon too deadly to use
In a separate paper, "Language Models Trained on Media Diets Can Predict Public Opinion," MIT researchers Eric Chu, Jacob Andreas, and Deb Roy, along with Harvard researcher Stephen Ansolabehere, found large language models trained on specific media (online news, TV broadcasts, or radio) can predict the opinions of groups exposed to that media.
"With 'media diet models,'" explained co-author Eric Chu, a Google research scientist who was an MIT doctoral candidate at the time of the research project, via Twitter, "we predict how a group of media consumers will answer polls by training a [language model] on media they consume."
These media diet models were based on BERT, a widely known large large language model, and fine-tuned with a media diet dataset.
The authors said their work points the way to more accurate public opinion polling, but also invites further examination of how media affects people and shapes public opinion.
They argue for media diet-specific analyses that examine: "(1) selective exposure, or the general systemic bias in which people gravitate towards information that is congruent with their prior beliefs; (2) echo chambers, in which the selected environments amplify and strengthen opinions shared with like-minded individual; and (3) filter bubbles, in which content curation and recommendation algorithms surface items based on users’ past behaviors, again confirming the users’ worldviews."
"Media diet models could help identify subpopulations being exposed to potentially harmful messaging," the authors suggest.
That's the best case scenario. They may also prove useful to media manipulators for assessing the effectiveness of their disinformation campaigns. ®
Don't miss... What happens when Bloomberg plugs its wealth of financial data into a large ML model? Wonder no more, thanks to our friends at The Next Platform, who analyzed BloombergGPT.