Character Tokenizer
In character tokenizer, sentence is split into character level.
Example
Sentence: [billy go to school]
Tokens: [b i l l y g o t o s c h o o l]
Cons:
- Characters has no syntactic or semantic meaning into them like words
In character tokenizer, sentence is split into character level.
Sentence: [billy go to school]
Tokens: [b i l l y g o t o s c h o o l]