Self-Supervised Learning
Self-supervised learning is a combination of Supervised Learning and Unsupervised Learning.
- Like an Unsupervised Learning, it doesn't have labeled data
- It creates targets from the data itself and makes the task Supervised Learning
- It helps as the number of data without label is in tons compared to labeled data
- Labeling data is expensive
Examples:
- BERT used Masked Language Modeling on a huge corpus of text data
- BERT used Next Sentence Prediction on the same corpus
- Another example can be use of Contrastive Learning on the text document when multiple encoding of same sentence can be positive language and the other sentences on the batch can be soft negatives.