Semantic Reconstruction

Semantic reconstruction is the dual task of visual image reconstruction: instead of rebuilding the image a user sees from neural activity, it rebuilds thoughts, concepts, and linguistic meaning. Tang et al.'s 2023 Nat Neuroscience paper was the first to bring this goal to a practical level.

1. What Is Semantic Reconstruction

Difference from image reconstruction

Image reconstruction (MindEye): from visual cortex → image
Semantic reconstruction: from language-related brain regions → text / meaning

Semantic regions

Superior temporal gyrus (STG): speech perception
Middle temporal gyrus (MTG): word meaning
Angular gyrus: semantic integration
Parietal (precuneus): episodic memory

These regions are not visual cortex — they process language, memory, and abstract concepts.

2. Tang 2023 Nat Neuroscience

Tang, LeBel, Jain & Huth (2023) is the pioneering work:

Experiment

7 subjects lying in fMRI while listening to 16 hours of podcasts
fMRI BOLD recording
Goal: reconstruct the semantics of what they heard from fMRI

Method

fMRI BOLD (15 s window)
  ↓
Encoder → GPT-2 input embedding
  ↓
Beam search (candidate sentences)
  ↓
Pick the one most consistent with fMRI
  ↓
Reconstructed "meaning"

Key points

Not word-level reconstruction — fMRI is slow (~1 s), word-by-word is infeasible
Reconstructs the gist of a sentence
Uses GPT's language prior to fill in the details

Results (examples)

Actually heard	Reconstructed
"I don't have a driver's license yet"	"she has not even started to learn to drive yet"
"I get up from the air mattress and press my face against the glass"	"I just continued to walk up to the window and opened the glass"

The meaning is right, the words differ — this is the signature of semantic-level reconstruction.

3. Key Techniques

Encoder design

The BOLD signal at each fMRI voxel predicts GPT-2's representation of the current text segment.

During training: - Input: fMRI 15 s window - Target: GPT-2 embedding of the text heard in the same window - Loss: regression MSE

Beam search decoding

At generation time: 1. Generate candidate sentences from GPT-2 (beam size ~200) 2. Each candidate → predict fMRI → compare with actual fMRI 3. Pick the highest-consistency one

This is the "brain signal as guidance, LM as generator" paradigm — aligned in spirit with RL from human feedback.

4. Semantic vs Word-Level Accuracy

What it can do

Sentence gist
Sentiment (positive/negative)
Topic (travel, work, characters)
Specific nouns (dog, car, house)

What it cannot do

Function words (the, is, a)
Specific word choice
Grammatical detail

Evaluation

BERTScore: semantic similarity
BLEU: word overlap (low)
Human judgment: comprehensibility rate

Tang 2023's BERTScore ~0.85 vs baseline 0.5 — semantics correct but words differ.

5. First Shock to Data Privacy

Tang 2023 triggered a major neurorights discussion:

Key experiment

They tested whether "a user can hide their thoughts": - Asked subjects to deliberately think of something else - fMRI reconstruction accuracy dropped significantly

Conclusion: the current system requires user cooperation and is hard to decode from an uncooperative subject.

Privacy design

Cooperation principle: the system should require active user participation
Passive scanning should be legally prohibited (Chile 2021 constitution, CO 2024 law)
fMRI + LLM combined is a "potential mind-reading technology" — legislation is urgent

This made semantic reconstruction a direct motivator for the Neurorights chapter.

6. Extensions and Variants

MindLLM (2024)

Longer stories
Cross-subject
Visual descriptions

Brain-to-Story (2024)

Continuous stories rather than isolated sentences
Long-context capabilities of LLMs come into play

Schölkopf group: episodic-memory reconstruction

fMRI records subjects recalling past events
Reconstruct the recalled event
First attempt at "recall decoding"

7. Contrast with Speech BCI

	Speech BCI (Willett 2023)	Semantic reconstruction (Tang 2023)
Signal	Spike	fMRI BOLD
Speed	62 WPM	Sentence-level
Accuracy	Word-level 9.1% WER	Semantic level
Brain region	vSMC (motor)	Semantic cortex
Scenario	Attempted speech	Listening to language

Speech BCI decodes "what you want to say"; semantic reconstruction decodes "what you mean" — fundamentally different tasks.

8. Clinical Potential of Semantic Reconstruction

Aphasia diagnosis

Healthy person listens to a story vs patient listens to the same story
Compare whether fMRI reconstruction can recover the "meaning that should be understood"
Quantitative assessment of language comprehension

Vegetative state

fMRI + story in vegetative / minimally conscious patients
If meaning can be reconstructed → demonstrates presence of consciousness
Related to post-2020 "cognitive motor dissociation" research

Communication aid

Fully locked-in patients who cannot speak or move
Hear a question → fMRI → semantic answer
Slower than typing, but may be the only option

9. Philosophical Implications

Readability of thought

Tang 2023 overturns the philosophical assumption that "thoughts are private."

What is decoding

A spectrum of literal → semantic → intentional reconstruction
Still far from "mind reading" (requires cooperation, low resolution)
But the direction is clear

Linguistic thought

Intriguingly, semantic reconstruction works precisely because most higher-level thinking is verbalized. Non-verbal thought (emotion, intuition) is still hard to decode.

10. The LLM-Accelerated Future

Tang 2023 used GPT-2. If replaced by GPT-4/Claude: - Stronger semantic priors - Better beam-search candidates - More natural reconstructions

2025+ expectation: GPT-4-class LLMs + more fMRI data → another leap in reconstruction quality.

11. Logical Chain

Semantic reconstruction decodes meaning, not words — fundamentally different from visual and speech BCI.
Tang 2023 first achieved practical semantic reconstruction with fMRI + GPT-2.
Method = neural activity as guidance for LLM generation, distinct from direct mapping.
Semantically correct ≠ word-level correct — high BERTScore, low BLEU.
Privacy experiments show current systems need cooperation — but legislation is still necessary.
Clinical, diagnostic, and communication aids are direct applications of semantic reconstruction.
LLM upgrades will keep raising reconstruction quality.

References

Tang et al. (2023). Semantic reconstruction of continuous language from non-invasive brain recordings. Nat Neuroscience. https://www.nature.com/articles/s41593-023-01304-9
Huth et al. (2016). Natural speech reveals the semantic maps that tile human cerebral cortex. Nature.
Jain et al. (2018). Incorporating context into language encoding models for fMRI. NeurIPS.
Chen et al. (2024). MindLLM: brain decoding via Large Language Models. arXiv.
Radford et al. (2019). Language models are unsupervised multitask learners. OpenAI. — GPT-2