Perplexity

#nlp #deep-learning #interview #evaluation

Perplexity is an Intrinsic Evaluation
Using Extrinsic Evaluation or downstream tasks is slow and hard
So people use Perplexity to compare different models and with other research
Perplexity tells us how good the model is generating sentences
Less Perplexity is better

Perplexity from Likelihood

$\begin{aligned} P P L & = P (w_{1} w_{2} . . . w_{N})^{- \frac{1}{N}} \\ = \sqrt[N]{\frac{1}{P (w_{1} w_{2} . . . w_{N})}} \\ = \sqrt[N]{\frac{1}{\prod_{i} P (w_{i} | w_{1} w_{2} . . . w_{i - 1})}} \end{aligned}$

Perplexity from Cross Entropy

$P P L = e^{C E}$

Related Notes

GloVe Embedding
N-gram Method
Sub-word Tokenizer
GRU
Encoder Only Transformer