Perplexity

#nlp #deep-learning #interview #evaluation

Perplexity is an Intrinsic Evaluation
Using Extrinsic Evaluation or downstream tasks is slow and hard
So people use Perplexity to compare different models and with other research
Perplexity tells us how good the model is generating sentences, or how close the sentence is to the pre-training data distribution
Less Perplexity is better
- or it means that current text is highly similar to the pre-training data

Perplexity from Likelihood

$\begin{aligned} P P L & = P (w_{1} w_{2} . . . w_{N})^{- \frac{1}{N}} \\ = \sqrt[N]{\frac{1}{P (w_{1} w_{2} . . . w_{N})}} \\ = \sqrt[N]{\frac{1}{\prod_{i} P (w_{i} | w_{1} w_{2} . . . w_{i - 1})}} \end{aligned}$

Perplexity from Cross Entropy

$P P L = e^{C E}$

Related Notes

Root Mean Squared Logarithmic Error (RMSLE)
Reinforcement Learning from Human Feedback (RLHF)
Byte Pair Encoding (BPE)
Contrastive Loss
Skip Gram Model