Structure or shortcuts? Cross-domain transfer in scientific figure classification

Four vision architectures trained to classify scientific figures from one field, then tested on another, to see what they actually learned.

Co-author with Tirth Acharya. Experiment design, training and analysis.

Three hypotheses

H1: a plain CNN, biased toward texture, will lose the most across domains. H2: adding explicit spatial attention (CBAM) to the CNN will help it generalise. H3: Vision Transformers, with global self-attention, will transfer best.

Three figure types

Statistical plot, diagram, and equation or algorithm. Three classes on purpose: with classes this distinct, an error says something about transfer and not about a fuzzy label.

Four models

EfficientNet-B0 (4.0M parameters, the texture-biased baseline), EfficientNet-B0 with CBAM (4.2M, channel and spatial attention), ViT-Small (21.7M, 6 heads) and ViT-Base (85.8M, 12 heads).

Two-phase fine-tuning

The backbone is frozen for the first five epochs while the new head trains, then everything is unfrozen at a lower learning rate. Class weights correct for the smaller equation class.

Test in domain and across domains

SciCap held-out figures for in-domain, ACL-Fig for cross-domain, seeds 42, 123 and 456. Mean and standard deviation are reported for every number.

Look inside the models

Grad-CAM on the CNNs, attention rollout and head-diversity analysis on the transformers, and linear CKA between the penultimate layers of every pair of models.