Embedding tables: from token IDs to vectors
An embedding table is a matrix with one row per token in the vocabulary and one column per dimension, so token ID 4217 maps to row 4217. Looking up an embedding is array indexing, the cheapest operation there is, and the table's size is vocabulary × dimensions × bytes per number. The values in each row are learned during training, which is how words used in similar contexts end up with similar vectors.
Lookup is indexing
The tokenizer gives you IDs, and the embedding layer turns each ID into a row of the table.
In a framework it looks like a layer, and underneath it's table[ids].
import numpy as np
rng = np.random.default_rng(0)
vocab = {"the": 0, "cat": 1, "kitten": 2, "sat": 3, "car": 4}
dim = 8
table = rng.normal(size=(len(vocab), dim)).astype(np.float32)
ids = [vocab[w] for w in "the cat sat".split()]
token_vectors = table[ids]
sentence_vector = token_vectors.mean(axis=0)
print("ids:", ids)
print("token vectors:", token_vectors.shape)
print("sentence vector:", sentence_vector.shape)
print("table size in bytes:", table.nbytes)
The table here is random, so "cat" and "kitten" aren't close yet. Training moves the rows so that tokens used in similar contexts end up near each other.
Size
| Vocabulary | Dimensions | Bytes per number | Table size |
|---|---|---|---|
| 32,000 | 4,096 | 2 (fp16) | about 262 MB |
| 128,000 | 4,096 | 2 (fp16) | about 1 GB |
| 50,000 | 384 | 4 (fp32) | about 77 MB |
The embedding table is a large part of a small model's parameters, which is one reason vocabulary size is a design decision.
Token embeddings vs sentence embeddings
The input embedding table in an LLM gives one vector per token, with no context. "Bank" gets the same row in "river bank" and "bank account". The transformer layers then mix in context, so the representation of "bank" changes as it passes through the model.
Sentence embeddings, the ones you store in a vector database, come from embedding models that run the whole text through a transformer and pool the outputs (mean pooling, or a special token) into one vector. For retrieval, you use an embedding model, not the LLM's input table. See embedding model selection.
Static embeddings
Word2Vec and GloVe were early methods that learned one fixed vector per word. They're still useful for learning, because you can load them and see that "king" sits near "queen", and that related words cluster.
Check your understanding
Compute the embedding table size for a model you use, from its vocabulary size and hidden dimension.
From the floor
In the AI data structures lecture, I loaded a pretrained word embedding dataset (GloVe-style vectors) and showed that the vector for a word is a long row of decimal numbers, that you can't read meaning from any single number, and that when you plot the vectors, related words cluster together, which is the property every semantic search system depends on.