Pattern Recognition · 2026 · Vol. 179

KRCapVLM

Beam-guided knowledge replay for knowledge-rich image captioning using vision-language model

Reem Aljunaida,b · Qasim Umera · Sajjad Mahmooda · Mahmood Niazia · Muzammil Behzada,c,*

a King Fahd University of Petroleum and Minerals, Dhahran, Saudi Arabia
b Imam Abdulrahman Bin Faisal University, Dammam, Saudi Arabia
c SDAIA-KFUPM Joint Research Center on Artificial Intelligence, Dhahran, Saudi Arabia
* Corresponding author. Email: muzammil.behzad@kfupm.edu.sa

63.30%
KnowCap recognition accuracy with +Beam
92.6
KnowCap CIDEr with +Beam
57.69%
Recognition accuracy on unseen categories
32,000
Training samples in the combined setup
Abstract

Generating informative and knowledge-rich image captions remains a challenge because existing captioning models often produce generic descriptions lacking specificity and contextual depth. KRCapVLM is a knowledge replay-based image captioning framework built upon a vision-language model. It incorporates beam search decoding to encourage explicit knowledge expression, attention-based modules in the image encoder to improve visual feature representation, and training schedulers to improve optimization reliability and consistency of downstream performance.

On KnowCap, recognition accuracy improves from 50.40% to 63.30%, accompanied by a +2.3 CIDEr increase. The paper also evaluates generalization to previously unseen knowledge categories and reports stronger knowledge-aware captioning while maintaining robust generic captioning performance.

Method

A knowledge-replay framework with three targeted improvements

KRCapVLM framework from Figure 1 of the paper
Figure 1. KRCapVLM framework. Knowledge replay is performed in parallel with downstream task fine-tuning to reinforce the model's knowledge of concepts.
01

Beam Search

Beam search is used to generate higher-quality pseudo-captions for replay samples, allowing multiple caption possibilities to be explored rather than relying on greedy decoding.

02

Patch Self-Attention

An additional self-attention layer models interactions among image patches before visual features are fused with text.

03

Learning-Rate Scheduler

Cosine annealing regulates optimization when the training objective combines captioning, knowledge prediction, and distillation losses.

Custom OFA-Large model with Patch Self-Attention from Figure 2
Figure 2. Custom OFA-Large model with Patch Self-Attention for image captioning.

Training objective

Standard COCO image-caption samples use cross-entropy supervision. Replay samples from the CC12M subset use beam-generated pseudo-captions and jointly activate knowledge prediction and knowledge distillation objectives. The total loss is:

Ltotal = Lce + λkLkpred + λdLdistill

Datasets

Training and evaluation data

MS-COCO

Generic captioning

Karpathy split with 113,287 training images, 5,000 validation images, and 5,000 test images, each with five human-written captions.

CC12M

Replay subset

Over 20,000 image-text pairs filtered using 122 predefined keywords. The experiments randomly select 5,000 image-keyword pairs for replay.

KnowCap

Knowledge benchmark

1,424 image-caption pairs spanning 240 knowledge categories; 424 validation and 1,000 test samples. The unseen subset contains 520 samples across 120 unseen categories.

Training split used in the experiments

27,000 COCO image-caption pairs + 5,000 CC12M image-keyword pairs = 32,000 combined training samples.

Results

KnowCap and COCO performance

Values below are transcribed from Table 1 of the published paper.

ModelKnowCap B1B4MRCIDErRec.COCO CIDEr
OFA zero-shot32.89.411.524.639.239.80%22.1
OFA-Finetuned35.610.812.126.641.738.50%134.6
+ K-Replay58.822.720.243.090.350.40%130.6
+ Scheduler58.623.920.944.192.654.20%130.4
+ Beam57.721.920.442.592.663.30%130.8
+ Attention58.122.120.543.292.058.90%129.4
Knowledge token statistics
ModelAvg. tokensCaptions ≥1 tokenMax
+ K-Replay0.15415.4%1
+ Scheduler0.17116.9%2
+ Beam0.22322.1%2
+ Attention0.17116.9%2
Key observation

Beam search increases explicit knowledge expression

The +Beam variant produces the highest average number of explicit knowledge tokens per image and the largest proportion of captions containing at least one knowledge token.

The paper notes that richer entity-specific phrasing can slightly reduce CIDEr because CIDEr rewards n-gram consensus with reference captions.

Generalization to unseen categories

ModelB1B4MRCIDErRec.
OFA zero-shot31.79.811.324.136.939.20%
OFA-Finetuned35.011.512.326.542.039.00%
+ K-Replay57.923.219.642.683.945.80%
+ Scheduler58.724.320.844.291.055.60%
+ Beam56.221.719.541.882.657.69%
+ Attention57.522.520.143.084.455.00%

Comparison with state-of-the-art forgetting methods

ModelB1B4MRCIDErRec.
EWC56.921.819.142.073.630.40%
Recall and Learn53.720.118.139.370.637.20%
Child-Tuning55.821.718.841.574.733.80%
Adapter54.420.517.640.163.730.40%
Vanilla Fine-tuning35.610.812.126.641.738.50%
K-Replay58.822.720.243.090.350.40%
+ Scheduler58.623.920.944.192.654.20%
+ Beam57.721.920.442.592.663.30%
+ Attention58.122.120.543.292.058.90%
Qualitative Results

More specific, knowledge-rich captions

Figure 3 from the paper compares K-Replay and KRCapVLM across diverse images. Knowledge concepts are highlighted in green in the original figure.

Qualitative comparison of K-Replay and KRCapVLM captions
Figure 3. Comparison of captions generated by K-Replay and KRCapVLM across diverse image examples.

K-Replay

Captions can remain generic, such as “a group of people standing in front of a tower.”

KRCapVLM

Captions can express more specific real-world concepts, such as “Big Ben,” “Tesla,” “Shibuya,” “Starbucks,” “Hollywood,” “Walmart,” and “Space Needle.”

Experiment Setup

Implementation details

OFA-Large

Vision-language model

Official PyTorch OFA-Large checkpoint, with the proposed patch self-attention module.

A100

Hardware

Experiments conducted using an NVIDIA A100 GPU with batch size 8 for 10 epochs.

k = 5

Beam size

The paper adopts beam size 5 as a balance between recognition accuracy and computational efficiency.

Optimization: learning rate 7 × 10−6, label smoothing 0.1, distillation temperature T = 16, λk = 1.0, λd = 1.0.

Beam-search overhead: average per-step training time increased from 5.53 s with greedy decoding to 6.47 s with k = 5, a 17.1% increase. This overhead occurs during pseudo-caption generation for replay samples rather than standard downstream inference.

Citation

Cite KRCapVLM

@article{Aljunaid2026KRCapVLM,
  title   = {KRCapVLM: Beam-guided knowledge replay for knowledge-rich
             image captioning using vision-language model},
  author  = {Aljunaid, Reem and Umer, Qasim and Mahmood, Sajjad
             and Niazi, Mahmood and Behzad, Muzammil},
  journal = {Pattern Recognition},
  volume  = {179},
  pages   = {113568},
  year    = {2026},
  doi     = {10.1016/j.patcog.2026.113568}
}