MIL-BERT: Classification of Arbitrarily Large Text with Performance and Explanatory Guarantees

Abstract

Objective/Background

Most artificial intelligence systems can analyze only a limited amount of text at once. Shortening a document can remove important evidence, while processing the entire document may require substantial computing resources and offer little insight into how a prediction was made. We introduce MIL-BERT, a method for efficiently classifying long documents and large text collections by finding the passages most relevant to a decision. It has both performance and explainability guarantees in this way.

Methods

MIL-BERT divides text into shorter, overlapping passages and uses a BERT-based language model to identify a small number of informative excerpts. Only these selected passages are used to train the final classifier, reducing memory requirements while still allowing the system to search the complete text. We evaluated MIL-BERT on seven datasets covering political bias in news, trigger warnings in stories, author characteristics, and established long-document classification benchmarks. The largest samples contained 882,343 tokens—nearly one million tokens of text.

Results

MIL-BERT produced new state-of-the-art results in three application areas. It achieved a macro-F1 score of 0.87 for classifying the political bias of news sources. For story-level trigger-warning detection, it achieved macro- and micro-F1 scores of 0.37 and 0.75, respectively. It also improved results for predicting authors’ gender and occupation from collections of social media posts. On broader long-document benchmarks, MIL-BERT remained competitive with leading approaches. Models trained using labels for entire documents or collections could also classify individual articles and passages without passage-level training labels such as with trigger-warning detection. Because the system identifies the excerpts that drive each prediction, its decisions can be reviewed and interpreted by people.

Conclusions

MIL-BERT provides an efficient and explainable approach to long-document text classification. It is particularly useful when the evidence needed for a decision appears in a small number of passages, offering an alternative to shortening documents or processing all text at once.

Keywords: long-document classification; MIL-BERT; BERT; artificial intelligence; natural language processing; explainable AI; interpretable machine learning; multiple-instance learning; text classification; weak supervision; political-bias detection; trigger-warning detection