BLEU Score

BLEU = BiLingual Evaluation Understudy
BLEU tells us how good the generated sentence is compared to the reference ground truths
As BLEU tells us how good the prediction is, it is often compared with Precision
BLEU score is computed with the multiplication of Brevity Penalty and Geometric Mean of Precision
Heavily used in Machine Translation
Brevity penalty is used to see if the generated text length is smaller than the original text and gives penalty if it is, as the generation can get higher score by generating a smaller sequence than the original text

BLEU Score

$\begin{aligned} BLEU-N & = Brevity-Penalty * \exp ({Geometric Mean of Precision}_{1. . . N}) \end{aligned}$ $Brevity-Penalty = m i n (1, \exp (1 - \frac{reference-length}{generation-length})) if gen-length < ref-length else 1$ ${Geometric Mean of Precision}_{1. . . N} = \sum_{n = 1}^{N} w_{n} l o g p r e_{n}$ $p r e_{n} = \frac{# of n-gram mathched in both target & generation}{total # of n-grams in the generation}$

Problems with BLEU Score

Doesn't consider semantic meaning
Doesn't consider synonyms
Struggles with non-english language
Depends on the reference set
Hard to compare with different Tokenizer
1. as different tokenizer will break sentence in different parts; hence n-gram using tokenizer-A is not same as n-gram using tokenizer-B

Problems with BLEU Score

Related Notes