A comparative study published on arXiv on June 2, 2026 examines the performance of the Transformer models BART, BERT, and RoBERTa for automatic text summarization. The central finding: BART achieves superior results on abstractive summarization tasks because the model is the only one of the three to utilize a sequence-to-sequence architecture for generation tasks. BERT and RoBERTa are primarily encoder-based models designed for text comprehension tasks.

Architectural Foundations of the Compared Models

BART (Bidirectional and Auto-Regressive Transformers) combines architectural components from BERT and GPT. The model features a bidirectional encoder like BERT and an autoregressive decoder like GPT. The original Transformer architecture is based on an encoder-decoder model in which the encoder learns high-dimensional representations of input sequences that the decoder then maps to output sequences.

The models are pretrained on large text corpora such as BookCorpus or Wikipedia before being fine-tuned for specific tasks. BART was introduced in the paper "BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension" and uses a denoising method for sequence-to-sequence pretraining.

Performance Analysis of BART-LARGE-CNN

A study published in October 2024 evaluated BART-LARGE-CNN after fine-tuning on the BBC NEWS DATASET. The researchers tested the model across six different article types and documented the following results:

  • Technical articles: Summaries were fluent, grammatically correct, accurate, concise, and coherent with no deviations from the source text
  • Articles with numerical facts: Summaries showed fluency and coherence but contained factual errors and inaccuracies
  • Sports articles: Summaries were fluent and accurate but omitted important information
  • Health articles: Summaries met all quality criteria (fluency, duplicate-free, accurate, concise, coherent)
  • Political articles: Summaries met all quality criteria

The fine-tuned model showed significant improvements over baseline variants in ROUGE-Score and BERTScore. During fine-tuning, training loss decreased while validation loss initially decreased and then increased again – an indicator of overfitting. The model performs well with newspaper articles but produces poor results with dialogue inputs.

BART Compared to Other Transformer Models

According to research findings from April 2025, BART achieves state-of-the-art results on various abstractive tasks such as conversation summarization, question answering, and general summarization tasks. With comparable training resources, BART achieves the performance of RoBERTa on the GLUE and SQuAD benchmarks.

A study published in 2025 compared BART, T5, and Fin-T5 for long document summarization using the Gutenberg Dataset. Additional comparative analyses include models such as LongT5, BigBird Pegasus, HAT-BART, and DYLE. A review published in December 2024 identifies BART, BERT, and PEGASUS as standard approaches for abstractive text summarization of individual documents.

Evaluation Metrics and Research Methodology

The research primarily uses ROUGE-Scores and BERTScore as evaluation metrics. ROUGE measures lexical overlap between generated and reference summaries, while BERTScore assesses semantic similarity. These metrics allow quantitative comparisons of model performance across different datasets.

Application Domains and Specializations

Transformer-based summarization models are deployed in several areas:

  • News article summaries (a research paper published in May 2025 focuses on German-language texts)
  • Long document summaries
  • Abstractive conversation summaries
  • Question-answering contexts with summarization components

The structural advantage of BART over BERT and RoBERTa for text summarization results from the sequence-to-sequence architecture optimized for generation tasks. BERT and RoBERTa are encoder-based models primarily designed for comprehension tasks and are therefore structurally less suited for abstractive summarization.