Geneformer from scratch: a cross-disciplinary glossary

The nano-Geneformer series mixes two vocabularies: single-cell biology and deep learning. This page defines the terms from both, each with one concrete example from the posts, and maps the ones that turn out to be the same idea.

The two vocabularies

Rank encoding makes a cell a sentence, so many single-cell and language-model ideas are the same idea under two names.

Single-cell biology	Deep learning
a cell	a sentence
a gene	a word (token)
gene expression	term frequency (word count)
ranking a cell's genes	tokenizing a sentence
the 2,048-gene vocabulary	the token vocabulary
a housekeeping gene	a stopword
a marker gene	a keyword (content word)
gene co-expression	word co-occurrence
a cell atlas	a text corpus
cell type / identity	a document's topic
cell state	a document's sentiment
a cell embedding	a sentence embedding
a gene embedding	a word embedding
masked-gene prediction	masked language modeling
Geneformer	BERT
the median trick	TF-IDF weighting
deleting a gene (in-silico)	ablating an input feature

Single-cell biology

cell atlas: A single-cell reference dataset of many cells from one tissue, each profiled and labeled by cell type and disease state. (≈ a text corpus) In the series: the Smillie et al. 2019 ulcerative-colitis atlas, 365,492 cells from 30 donors, annotated into 51 cell types.
cell type vs. cell state: Cell type is a cell's stable identity (which gene program it runs); cell state is its transient condition like inflamed versus resting. (≈ a document's topic vs. its sentiment) In the series: an inflamed fibroblast and a resting fibroblast are the same cell type in different states; across patient folds the embedding reads type stably while disease macro-F1 swings from 0.33 to 0.65, averaging 0.51.
chemokine: A signaling protein whose gene, when highly ranked in a cell, marks inflammation and immune-cell recruitment. In the series: the inflammatory chemokines CXCL1 and CXCL3 sit at the very top of an inflamed fibroblast's ranked sentence (CXCL1 > MMP3 > CXCL3 > COL3A1 > …).
collagen & extracellular matrix (ECM): Structural proteins (collagens) and the scaffold they form outside cells; their genes co-express and mark fibroblasts. In the series: deleting COL3A1 from a fibroblast lowers the model's predicted score for the rest of its ECM program (COL6A3, COL1A1, POSTN, FN1, MMP2).
epithelium: The sheet of lining cells covering a tissue surface; in the colon it is metabolically dense and mitochondria-rich. In the series: colonic epithelium is mitochondria-rich, a median ~17% mitochondrial reads versus ~4% for immune and stromal cells, so a blanket mito filter would wrongly delete healthy epithelium.
fibroblast: A connective-tissue cell that produces a tissue's structural collagen proteins, identifiable by its collagen gene program. In the series: one inflammatory fibroblast (229 genes) ranks CXCL1 > MMP3 > CXCL3 > COL3A1 > RARRES2 > SOD2 > LUM > COL1A2, with collagens COL3A1, COL1A2 and LUM marking it.
gene expression: How strongly each gene is switched on in one cell, measured as transcript counts by scRNA-seq. (≈ term frequency, a word's count) In the series: each cell's gene counts are normalized to 10,000 total (CP10k), yet among the 2,048 vocabulary genes typical expression still ranges from about 0.55 to 61.
gene program: A set of genes a cell type switches on together; running the same program makes two cells look alike. (≈ a topic, a set of co-occurring words) In the series: a fibroblast runs the collagen program (COL1A1, COL3A1, LUM), a plasma cell the antibody program (immunoglobulins), a goblet cell the mucus program (MUC2, TFF3).
goblet cell: A mucus-secreting cell of the gut lining, recognized in the model by mucus genes like TFF3, FCGBP, ITLN1 and CLCA1. In the series: a held-out goblet cell reads FCGBP > ITLN1 > CLCA1 > [MASK], and the model fills the blank with the mucus gene TFF3 at 11%.
housekeeping gene: A gene expressed at steady levels in nearly every cell, so it carries no identity and is dropped from the 2,048-gene vocabulary. (≈ a stopword) In the series: housekeeping genes barely vary, so none make the 2,048 most-variable-gene vocabulary; the same variable-gene step also drops the stable ribosomal genes.
immunoglobulin: An antibody gene (for example IGHA1, IGLC3); a plasma cell's transcriptome is mostly immunoglobulin, so these genes mark plasma cells. In the series: hiding the top gene of a held-out plasma cell, the model recovers the immunoglobulin IGHA1 at 93%, with runners-up IGLC3, IGLC2, IGHA2 also immunoglobulins.
marker gene: A gene whose high expression identifies a cell type; the rank encoding pushes it to the top of that cell's sentence. (≈ a keyword) In the series: across 100,000 cells the top rank-1 genes are markers, not housekeeping: TFF3 marks goblet cells, CD74 antigen-presenting cells, CXCL14 fibroblasts.
mitochondrial genes (MT-): Genes from the mitochondrial genome (MT- prefix); a high fraction flags dying cells, but colon epithelium is naturally mitochondria-rich. (≈ a high-frequency word like "the") In the series: colonic epithelium runs a median ~17% mitochondrial reads versus ~4% for immune and stromal cells, so a blanket mito% < 10% filter would delete healthy epithelium.
plasma cell: An antibody-producing cell whose transcriptome is mostly immunoglobulin; it is the single most common cell type in this UC atlas. In the series: plasma cells are the commonest type in the atlas, so always-guess-Plasma is right 32.5% of held-out cells; one plasma cell expresses only 35 genes.
ulcerative colitis (UC) & IBD: Ulcerative colitis is a form of inflammatory bowel disease (IBD); the series tests whether the model reads UC severity off cells. In the series: the atlas has 12 healthy controls and 18 UC patients, each UC biopsy sampled at an inflamed and a non-inflamed site, giving three severity groups.

Deep learning

BERT: A transformer pretrained by masking about 15% of tokens and predicting them, with every token reading its full left and right context. In the series: nano-Geneformer uses BERT's 80/10/10 masking rule (80% become [MASK], 10% become a random gene, 10% left unchanged) on a cell's ranked genes.
cell embedding: One fixed 256-number vector summarizing a whole cell, the mean of its contextual gene vectors, computed with no labels. (≈ a sentence embedding) In the series: each cell becomes one row of a (100000, 256) embedding matrix; a linear classifier on these names 51 cell types on held-out patients 79.9% of the time.
cross-entropy loss: The training penalty equal to minus the log-probability the model put on the true masked gene, averaged over masked positions (a confident right call at probability 0.9 costs only $-\ln(0.9) \approx 0.1$ ). (≈ a language model's loss on the true word) In the series: blanking CXCL1 at probability 0.0036 costs $-\ln(0.0036) \approx 5.6$ ; averaged over masked genes the loss reaches 5.19, well under the $\ln(2050) \approx 7.63$ random-guess baseline.
embedding (token & gene embedding): A learned 256-number vector for one gene, looked up by its token id; similar-role genes should sit close together. (≈ a word embedding) In the series: after weight tying, COL1A1's gene embedding sits next to fibroblast and collagen genes PDGFRA and COL14A1, and gene-gene similarities roughly double.
epoch: One complete pass over every training cell; pretraining repeats it several times so the masked-gene loss keeps falling. In the series: python train.py --epochs 2 runs two full passes over the ~80k training cells in about 3 hours on an Apple Metal GPU.
fine-tuning vs. frozen model: Frozen means the pretrained weights stay fixed while only a classifier trains on top; fine-tuning would instead keep updating those weights. In the series: the model stays frozen (no fine-tuning) and the pretrained embedding is graded as-is, with only a logistic regression fitted on its 256-d output.
foundation model: One model pretrained once on many unlabeled cells, then reused as a fixed feature source for many downstream tasks without retraining. In the series: nano-Geneformer's frozen encoder produces one (100000, 256) cell-embedding table, computed once with no labels and reused to cluster, classify, and visualize.
gradient descent & backpropagation: The training update that lowers masked-gene loss: backpropagation computes each weight's gradient, gradient descent nudges every weight a small step to reduce error. In the series: backprop (loss.backward()) computes each weight's gradient, then the optimizer step applies it; over two epochs the masked-gene loss falls from ~7.76 to 5.19.
hidden state (contextual gene vector): A gene's 256-number vector after self-attention, describing that gene in the context of all the other genes in its cell. (≈ a contextual word vector) In the series: the encoder turns each cell's ranked genes into one 256-number contextual vector per gene, shape (64, 1024, 256) for a batch; our fibroblast contributes 229 such vectors.
in-silico perturbation: Deleting a gene from a cell inside the frozen model, recomputing its embedding, and measuring how the model's expectations shift. (≈ a wet-lab gene knockout) In the series: deleting COL3A1 from an inflamed fibroblast lowers the model's score for COL1A1 by 0.92 and turns down the whole collagen program (COL6A3, POSTN, FN1).
linear classifier: A logistic regression fitted on the frozen 256-d cell embedding to read a label, measuring what the embedding already presents in easy-to-read form. In the series: on the frozen embedding a logistic-regression classifier names the correct one of 51 cell types on held-out patients with 79.9% accuracy, against a 2.0% chance floor.
logit & softmax: Logits are the raw per-gene scores from a linear layer; softmax turns them into probabilities over the 2,050-gene vocabulary that sum to 1 (softmax of $[2, 1, 0]$ is about $[0.67, 0.24, 0.09]$ ). (≈ a language model's word probabilities at a blank) In the series: the MLM head produces 2,050 logits per gene; softmax over them gives a blanked CXCL1 in the fibroblast probability 0.0036, about 1 in 280.
masked-gene pretraining (MLM): Hiding about 15% of a cell's genes and training the model to predict them from the rest, with no labels. (≈ masked language modeling in NLP) In the series: hiding a held-out plasma cell's top gene, the model recovers the immunoglobulin IGHA1 at 93%; across 400 held-out cells top-1 recovery is 7.1%.
mean-pooling: Averaging a cell's contextual gene vectors, column by column, into one 256-number cell vector, skipping the PAD slots (one column of three genes averages as $(0.2 + 0.4 + 0.0)/3 = 0.2$ ). (≈ averaging word vectors into a sentence vector) In the series: the fibroblast's 229 gene vectors average down to one 256-number cell embedding; a plasma cell's 35 vectors average to the same 256-number size.
pretraining (self-supervised): Training the model on unlabeled cells by hiding genes and predicting them, so it learns reusable structure before any labeled task. In the series: masked-gene pretraining on the ~80k training cells for two epochs drove the loss from about 7.76 toward 5.19, with no labels used.
rank encoding: Turning a cell into a list of gene tokens ordered by how high each gene is relative to its own typical level. (≈ tokenization) In the series: cell 9 rank-encodes to CXCL1 > MMP3 > CXCL3 > COL3A1 > ..., 229 gene tokens then padded with PAD to length 1024.
self-attention (query, key, value; heads): Each gene rewrites its own 256-vector as a weighted blend of every other gene's value vector, weights set by query-key dot products. (≈ words attending to each other in a sentence) In the series: nano-Geneformer uses 4 heads across 4 layers; on 80 held-out fibroblasts a blanked COL1A1's attention lands most on collagen neighbors at layer 4 head 3, 0.56.
token & vocabulary: A token is one gene's id; the vocabulary is the fixed set of genes (plus PAD and MASK) the model can read. (≈ a word and its vocabulary) In the series: the vocabulary holds 2,050 tokens (2,048 variable genes plus PAD and MASK), and token id 494 maps to the gene CXCL1.
transformer: A neural network that reads a cell's ranked gene tokens and lets every gene mix in context from every other gene via self-attention. In the series: nano-Geneformer stacks four Transformer encoder layers (self-attention plus feed-forward) at d_model 256 with 4 heads, about 4.5M parameters.
weight tying: Reusing a single gene-embedding table for both the input lookup and the output prediction head, forcing the two representations to match. In the series: tying the tables put COL1A1 next to fibroblast genes PDGFRA and COL14A1, roughly doubling their similarities, but left the cell-level classifiers unchanged.

Methods and evaluation

accuracy (top-1 / top-5): The fraction of held-out cases the model labels exactly right on its first guess; top-5 counts the true answer among its five highest-scoring. In the series: recovering a masked gene out of 2,048, nano-Geneformer scores 7.1% top-1 and 16.1% top-5 accuracy on held-out cells.
baseline: The reference method you race the model against (here PCA-50) to see whether pretraining bought anything a simple linear method wouldn't find. (≈ a control condition) In the series: the 50-dim PCA baseline scores 82.3% cell-type accuracy on held-out patients, edging the frozen embedding's 79.9%.
cosine distance: A similarity measure comparing only the direction of two cell embeddings, ignoring their length (identical directions score 1, perpendicular ones 0), used to find cells that look alike. In the series: cells are ranked by cosine distance between their 256-dimensional embeddings to answer "what else looks like this one?".
counterfactual: A "what if this gene were absent" comparison: rerun the frozen model without the gene and compare embeddings, rather than assuming the other genes stay put. (≈ an ablation study) In the series: removing COL3A1 from a fibroblast shifts its embedding by $\lVert \Delta z \rVert = 1.69$ , and almost all of that comes from the other 121 genes changing context, not from dropping COL3A1's own vector.
dot product: Multiply two equal-length vectors entry by entry and sum to one number, larger when the two vectors point the same way (for example $[1,2,3] \cdot [4,5,6] = 4 + 10 + 18 = 32$ ). In the series: the MLM head scores gene $g$ by the dot product $W_g \cdot h_i$ of its weight row with a gene's 256-number context vector.
F1 (macro-averaged): A per-class score balancing precision and recall ( $\frac{2 \cdot 0.6 \cdot 0.4}{0.6 + 0.4} = 0.48$ for precision 0.6 and recall 0.4), then averaged evenly across all 51 cell types so rare types count as much as common ones. In the series: on the 51-way cell-type task the frozen embedding scores macro-F1 0.63, against PCA's 0.68.
generalization: Whether the classifier's accuracy holds on patients the model never saw, so it reflects real biology rather than memorized batch quirks. (≈ validation on an independent cohort) In the series: the 79.9% cell-type accuracy is scored on 19,791 cells from six patients held out of training, so it measures generalization.
healthy-inflamed axis: The unit vector in embedding space pointing from the average inflamed cell toward the average healthy cell. In the series: the healthy-inflamed axis spans a 4.5-unit gap; deleting IGHG3, the largest shift, moves a cell only 0.256 units along it.
median trick: Dividing each gene's count by its own median level across cells, so a gene ranks high only when unusually loud for itself. (≈ the IDF term of TF-IDF) In the series: a count of 6 over median 2 (score 3.0) outranks a count of 20 over median 10 (score 2.0), so g0 > g2.
patient split & held-out set: Assigning whole patients (not individual cells) to train or test, so the classifier can't cheat by recognizing which patient a cell came from. In the series: 30 patients total, every 5th held out (6 patients, 19,791 cells), giving 80,209 train and 19,791 held-out cells.
PCA (principal component analysis): A linear method that compresses each cell's ~2,048-gene expression into 50 principal components, used as the classical baseline for the model. In the series: a 50-dimensional PCA of the 2,048 variable genes scores 82.3% cell-type accuracy on held-out patients.
quality control (QC): Deciding which cells are real by removing each sample's extreme outliers on total counts and mitochondrial %, adapted to the tissue. (≈ data cleaning) In the series: the ±3-SD-per-sample rule drops 2.0% of UC cells (median 49% mitochondrial) while keeping the high-mito healthy colonic epithelium (~17% median).
ROC-AUC: A metric asking whether the classifier ranks a type's cells above all others, rather than whether it wins the single top guess (0.5 is random ranking, 1.0 perfect). In the series: the CD4+ PD1+ type the classifier names zero times still scores ROC-AUC 0.99; the embedding's macro AUC is 0.990.
Spearman correlation: A correlation from −1 to +1 measuring whether two quantities rise and fall together by rank, not by their exact values (+1 is identical ranking, 0 unrelated, -1 reversed). In the series: the perturbation ranking correlates with a simple inflamed-versus-healthy enrichment test at Spearman $+0.62$ ( $p \approx 10^{-27}$ ).
t-SNE & UMAP: Nonlinear methods that project the 256-d cell embedding down to 2-D, for an atlas-style map you can look at. In the series: a t-SNE of the frozen 256-d cell embeddings shows the seven lineages (epithelium, plasma, T/NK, myeloid, and the stromal trio) as clean islands.
vector magnitude (norm & unit vector): The length of a vector, the square root of its summed squared entries (the length of $[3, 4]$ is $\sqrt{9 + 16} = 5$ ); a unit vector is one rescaled to length 1. In the series: CXCL1's token and rank embeddings each have magnitude about 17, and their near-orthogonal sum has magnitude $\approx 24$ .