Perplexity

#nlp #deep-learning #interview #evaluation

Perplexity is an Intrinsic Evaluation
Using Extrinsic Evaluation or downstream tasks is slow and hard
So people use Perplexity to compare different models and with other research
Perplexity tells us how good the model is generating sentences
Less Perplexity is better

Perplexity from Likelihood

$\begin{aligned} P P L & = P (w_{1} w_{2} . . . w_{N})^{- \frac{1}{N}} \\ = \sqrt[N]{\frac{1}{P (w_{1} w_{2} . . . w_{N})}} \\ = \sqrt[N]{\frac{1}{\prod_{i} P (w_{i} | w_{1} w_{2} . . . w_{i - 1})}} \end{aligned}$

Perplexity from Cross Entropy

$P P L = e^{C E}$

Related Notes

Learning Rate Scheduler
TF-IDF
Saddle Points
Forward Feature Selection
Causality vs. Correlation