The Wisdom of a Crowd of Brains:
A Universal Brain Encoder

Published in Proceedings of CCN 2026 • Contributed Talk in the conference

Roman Beliy*  ·  Navve Wasserman*  ·  Amit Zalcher  ·  Michal Irani

Weizmann Institute of Science  |  *Equal contribution

Correspondence: roman.beliy@weizmann.ac.il, navve.wasserman@weizmann.ac.il

Paper Code
Overview of the Universal Image-to-fMRI Brain-Encoder
Overview. The Universal Image-to-fMRI Brain-Encoder trains jointly on multiple subjects & datasets. It predicts fMRI activation of each brain-voxel on any image via cross-attention between learned brain-voxel embeddings and deep image features. Missing file: figures/Figure_Overview_shrank7.PNG

Abstract

Image-to-fMRI encoding is important for both neuroscience research and practical applications. However, such "Brain-Encoders" have been typically trained per-subject and per fMRI-dataset, thus restricted to very limited training data. In this paper we propose a Universal Brain-Encoder, which can be trained jointly on data from many different subjects/datasets/machines. What makes this possible is our new voxel-centric Encoder architecture, which learns a unique "voxel-embedding" per brain-voxel. Our Encoder trains to predict the response of each brain-voxel on every image, by directly computing the cross-attention between the brain-voxel embedding and multi-level deep image features. This voxel-centric architecture allows the functional role of each brain-voxel to naturally emerge from the voxel-image cross-attention. We show the power of this approach to: (i) combine data from multiple different subjects (a "Crowd of Brains") to improve each individual brain-encoding, (ii) quick & effective Transfer-Learning across subjects, datasets, and machines (e.g., 3-Tesla, 7-Tesla), with few training examples, and (iii) we show the potential power of the learned voxel-embeddings to explore brain functionality (e.g., what is encoded where in the brain).

Universal-Encoder Architecture

Universal Encoder architecture
Universal-Encoder Architecture. Input: an image & a brain-voxel index (a pointer to its Voxel-Embedding vector). Output: predicted fMRI activation of this brain-voxel on that image. The model has 3 main components: (a) Feature Extraction Block — extracts multi-scale (DINO-adapted) image features; (b) Learned Voxel-Embedding — captures the unique functionality of each voxel; (c) Cross-Attention Block — establishes the connection between voxel-functionality and relevant image features through spatial attention, a multi-layer perceptron (MLP), and functional attention. Missing file: figures/UniversalEncoder_shrank9.png

Results

Quantitative Comparison

Setting 1. Encoding accuracy for subjects who completed all recording sessions (NSD dataset).

Method S1 S2 S5 S7 Avg
Ridge Regression (CLIP) 0.510.480.470.380.46
Ridge Regression (DINOv2) 0.560.520.500.370.49
Gaziv et al. 0.500.520.480.420.48
Adeli et al. 0.650.620.590.450.58
Universal Encoder (Ours) 0.69 0.72 0.71 0.65 0.69

Setting 2. Following Bao et al. / Chen et al. preprocessing; metrics averaged on subjects 1, 2, 5, 7.

Method Pearson ↑ MSE ↓
BrainDIVE 0.3230.353
fWRF 0.3430.361
MindSimulator 0.3550.385
NRF 0.3580.345
Universal Encoder (Ours) 0.392 0.336

Qualitative Evaluation

Qualitative evaluation figure
Qualitative Evaluation. (a) Visual comparison of real vs. encoder-predicted fMRI. (b) Top 5 retrieved images for each "Query" test-fMRI. Missing file: figures/retrieval_brains_1.png

The Wisdom of a Crowd of Brains

Wisdom of a crowd of brains
The Wisdom of a Crowd of Brains. By aggregating data from multiple subjects, our Universal-Encoder improves encoding of any individual subject. We compared 3 models: (i) "Baseline" single-subject encoder of Gaziv et al., (ii) "Universal Encoder – single subject" — our architecture trained on each subject separately, (iii) "Universal Encoder – multiple subjects" — our model trained on data from 8 subjects. (a) Pearson Correlation (per voxel) between predicted & ground-truth fMRI. (b) Retrieval Accuracy (Top1 & Top5) of the GT image per "Query" fMRI. Missing file: figures/wisdom_of_crowds_3.png

The Wisdom of the Crowd of Datasets

Wisdom of the crowd of datasets
The Wisdom of the Crowd of Datasets. Using data from a high-quality 7T dataset (NSD) enhances encoding performance in lower-quality (3T & 4T) datasets. Missing file: figures/Figure_Wisdom_dataset_3.png

Transfer-Learning to New Subjects / Datasets

When a new subject or dataset is encountered, the Universal-Encoder does not need to be retrained from scratch. All weights of the pre-trained encoder are frozen, and only the 256-dimensional voxel-embeddings of the new subject are optimized — requiring very little new data. Remarkably, as few as 100 subject-specific examples suffice to outperform a dedicated model trained on the entire training set of 9,000 examples. Transfer also works across scanner resolutions, e.g., from 7T NSD to older 3T and 4T datasets with much smaller training sets and different visual stimuli.

Transfer learning to new subjects and datasets
Transfer-Learning to new subjects/datasets. Pre-trained on NSD, the Universal-Encoder adapts to new subjects with few data. Transfer-Learning (purple) significantly outperforms single-subject models (orange). Missing file: figures/Figure_transfer_3.png

Brain Exploration via Voxel-Embeddings

As part of training, the Universal-Encoder maps functionally-similar brain voxels — both within the same brain and across different brains — to nearby voxel-embeddings. Applying k-means clustering to these embeddings reveals shared functional regions across subjects, even though the subjects viewed completely different images and the embeddings were optimized independently. Discovered clusters are activated by semantically coherent categories: Food, Faces, Text, Outdoor Scenes. Notably, these shared functional regions are not anatomically aligned , showing the embeddings capture functional roles rather than anatomical identity. Finer-grained clustering within known regions such as PPA also reveals meaningful sub-regions, e.g., Indoor vs. Outdoor Scenes.

Brain exploration through voxel embedding clusters
Brain Exploration. Clustering voxel-embeddings in the Embedding-Space leads to discovery of functional brain regions. The functional role of each embedding cluster is unveiled through the images that give highest activation to each cluster. Missing file: figures/brain_clusters_rebuttal.png

BibTeX

@inproceedings{beliy2026wisdom,
  title     = {The Wisdom of a Crowd of Brains: A Universal Brain Encoder},
  author    = {Beliy, Roman and Wasserman, Navve and Zalcher, Amit and Irani, Michal},
  booktitle = {Proceedings of the 9th Annual Conference on Cognitive Computational Neuroscience},
  year      = {2026},
  doi       = {10.32470/oet3bk8},
  url       = {https://openreview.net/forum?id=ciC5UUZ2Fs}
}