DistMult: Bilinear Knowledge Graph Embeddings

Chapter 1: The Bilinear Model

The core idea of embedding-based KG completion is simple: map every entity and relation to a low-dimensional vector, then define a scoring function that assigns a high score to true triples and a low score to false ones.

The most general bilinear scoring function for a triple (h, r, t) is:

Here h and t are embedding vectors for the head and tail entities, and W_r is a matrix specific to relation r. This is the full bilinear model — "bilinear" because the score is linear in both h and t separately.

The problem with the full bilinear model is scale. If entity embeddings have dimension d, then W_r has d² parameters — one matrix per relation. With thousands of relations, this explodes. DistMult's insight is to restrict W_r to be a diagonal matrix.

When W_r = diag(r) for a vector r, the bilinear product simplifies dramatically:

This is the DistMult scoring function. Instead of d² parameters per relation, we now have just d — a 10,000x reduction for typical embedding sizes.

Chapter 2: The Scoring Function

The DistMult scoring function is deceptively simple:

Think of it as a three-way dot product: for each dimension i, multiply the head, relation, and tail values together, then sum. This is also called a Hadamard product followed by summation.

Intuitively: if h_i and t_i are both large (say, both entities are strongly "European"), and r_i is also large for dimension i (dimension i is relevant to this relation), the contribution to the score is high. The relation vector r acts as a selector — emphasizing the dimensions that matter for this relation type.

From score to prediction

For a query like (Marie Curie, won-prize, ?), we compute f_r(h, t) for every possible entity t, sort by score, and report the top-k. If the correct answer (Nobel Prize) appears in the top 10, that's a "hit@10."

Chapter 3: The Symmetry Property

Here's a key property hidden inside the DistMult formula. What happens if we swap h and t?

Swapping head and tail leaves the score unchanged. DistMult is symmetric by construction: for any relation r, f_r(h, t) = f_r(t, h). Always.

Relations in knowledge graphs come in three flavors:

Most real-world relations are not symmetric. This is the fundamental limitation of DistMult — and it's precisely what ComplEx (the next paper) is designed to fix.

Chapter 4: Training

DistMult is trained with negative sampling. For each true triple (h, r, t), we generate corrupted triples by replacing either h or t with a random entity. We then maximize the score of true triples and minimize the score of corrupted ones.

The loss function is a pairwise ranking loss (or equivalently, a softmax cross-entropy over all possible entities):

Where (h_j', t) are negative samples with random head entities. σ is the sigmoid function. The model learns to assign high scores to true triples and near-zero (post-sigmoid) scores to false ones.

Regularization

Embeddings are L2-regularized to prevent them from growing unboundedly (large embeddings could always produce high scores). The loss becomes:

python
import torch
import torch.nn as nn

class DistMult(nn.Module):
    def __init__(self, n_entities, n_relations, dim):
        super().__init__()
        self.entity_emb  = nn.Embedding(n_entities, dim)
        self.relation_emb = nn.Embedding(n_relations, dim)
        nn.init.xavier_uniform_(self.entity_emb.weight)
        nn.init.xavier_uniform_(self.relation_emb.weight)

    def score(self, h_idx, r_idx, t_idx):
        h = self.entity_emb(h_idx)   # (batch, dim)
        r = self.relation_emb(r_idx)  # (batch, dim)
        t = self.entity_emb(t_idx)   # (batch, dim)
        return (h * r * t).sum(dim=-1)  # (batch,)

    def forward(self, pos_h, pos_r, pos_t, neg_h, neg_r, neg_t):
        pos_score = self.score(pos_h, pos_r, pos_t)
        neg_score = self.score(neg_h, neg_r, neg_t)
        loss = -torch.log(torch.sigmoid(pos_score)).mean()
        loss += -torch.log(torch.sigmoid(-neg_score)).mean()
        return loss

Chapter 5: Results

Yang et al. evaluated DistMult on two standard benchmarks: WordNet (WN18) — a lexical database with 18 relation types like hypernym, hyponym, and member-of — and Freebase (FB15k) — a curated subset with 1,345 relation types across 14,951 entities.

The primary metric is Hits@10: for each test triple, rank all possible tail entities by score. What fraction of the time does the true tail appear in the top 10? Higher is better.

On FB15k, DistMult's 57.7% also beats TransE's 47.1%. But there's a catch: FB15k contains many inverse relations (e.g., both "capital-of" and "is-capital") that artificially inflate scores. Later work (Toutanova and Chen, 2015) showed DistMult's advantage partly comes from exploiting these inverse pairs.

Chapter 6: DistMult vs TransE

TransE and DistMult represent two fundamentally different philosophies for modeling knowledge graph relations, and understanding the difference is crucial for knowing when to use each.

TransE (Bordes et al., 2013) models relations as translations in embedding space. A true triple (h, r, t) should satisfy h + r ≈ t. The scoring function is the negative distance: f_r(h, t) = −||h + r − t||.

This complementarity is not accidental — it points toward a deeper truth. You need a model expressive enough to handle both symmetric and antisymmetric relations. That's exactly what ComplEx provides, by extending DistMult to complex vector spaces.

Chapter 7: Connections

DistMult is a foundational paper in knowledge graph embeddings — simple enough to analyze mathematically, strong enough to anchor years of subsequent research. Here's where it fits in the broader landscape.

Limitations and when NOT to use DistMult

• Antisymmetric relations: Use ComplEx or RotatE instead.
• Compositional patterns: (A parent-of B, B parent-of C → A grandparent-of C). DistMult cannot model reasoning chains. Use path-based models or R-GCN.
• Inductive setting: New entities at test time (not seen during training). DistMult has no mechanism — use GNN-based encoders.

Further reading:

• ComplEx: Trouillon et al. (2016) — extends DistMult to complex space to handle antisymmetric relations.
• RotatE: Sun et al. (2019) — models relations as rotations in complex space, handles symmetry, antisymmetry, inversion, and composition.
• TuckER: Balazevic et al. (2019) — Tucker decomposition, strictly more expressive than DistMult.