Perplexity

#nlp #deep-learning #interview #evaluation

Perplexity is an Intrinsic Evaluation
Using Extrinsic Evaluation or downstream tasks is slow and hard
So people use Perplexity to compare different models and with other research
Perplexity tells us how good the model is generating sentences
Less Perplexity is better

Perplexity from Likelihood

$\begin{aligned} P P L & = P (w_{1} w_{2} . . . w_{N})^{- \frac{1}{N}} \\ = \sqrt[N]{\frac{1}{P (w_{1} w_{2} . . . w_{N})}} \\ = \sqrt[N]{\frac{1}{\prod_{i} P (w_{i} | w_{1} w_{2} . . . w_{i - 1})}} \end{aligned}$

Perplexity from Cross Entropy

$P P L = e^{C E}$

Related Notes

Tokenizer
AdaGrad
Huber Loss
Count based Word Embeddings
Logistic Regression vs. Neural Network