Comparing financial news sentiment with stock returns
I compared two ways to analyze 20,550 Wall Street Journal articles: FinBERT sentiment scores, and a neural network trained to predict returns from frozen BERT embeddings. I wanted to know whether either approach helped predict individual stock returns or the following month's market return.
FinBERT's direction accuracy was 49.3%; the return network's was 50.0%. The return network predicted almost every sample as an increase. Its F1 score of 0.67 therefore did not indicate useful prediction, and its return estimates were close to zero.
FinBERT produced more variation in sentiment from month to month. Its correlation with the following month's S&P 500 return was −0.32, with p = 0.31. This small test sample did not provide statistically significant evidence of return predictability. Distinguishing the tone of an article and predicting a stock return are different tasks.
FinBERT assigns positive, negative and neutral probabilities; the sentiment score is P(positive) − P(negative). In a separate experiment, frozen 768-dimensional BERT embeddings feed a neural network trained on daily returns. The BERT encoder itself is not fine-tuned. Monthly averages are then compared with the following month's S&P 500 return.
Training uses articles from 2017–2018, validation uses 2019, and testing uses 2020. Articles are truncated to 512 tokens. The test period includes COVID-19 and supplies only about twelve monthly observations, which limits the market-level comparison.