Jump to content

Self-supervised learning: Difference between revisions

From Wikipedia, the free encyclopedia
Browse history interactively
← Previous editNext edit →
Content deleted Content added
VisualWikitext
No edit summary
No edit summary
Line 1: Line 1:
'''Self-supervised learning''' (SSL) refers to a [[machine learning]] paradigm, and corresponding learning methods, for processing unlabelled data to obtain useful representations that can help with downstream learning tasks. The most salient thing about SSL methods is that they do not need human-annotated labels, which means they are designed to take in datasets consisting entirely of unlabelled data samples. Then the typical SSL pipeline consists of learning supervisory signals (labels generated automatically) in a first stage, which are then used for some supervised learning task in a second stage. For this reason, SSL can be described as an intermediate form of [[unsupervised learning|unsupervised]] and [[supervised learning]].
'''Self-supervised learning''' (SSL) refers to a [[machine learning]] paradigm, and corresponding methods, for processing unlabelled data to obtain useful representations that can help with downstream learning tasks. The most salient thing about SSL methods is that they do not need human-annotated labels, which means they are designed to take in datasets consisting entirely of unlabelled data samples. Then the typical SSL pipeline consists of learning supervisory signals (labels generated automatically) in a first stage, which are then used for some supervised learning task in the second and later stages. For this reason, SSL can be described as an intermediate form of [[unsupervised learning|unsupervised]] and [[supervised learning]].


The typical SSL method is based on an [[artificial neural network]] or other model such as a [[decision list]].<ref>{{cite journal|last1=Yarowsky|first1=David|date=1995|title=Unsupervised Word Sense Disambiguation Rivaling Supervised Methods|url=https://aclanthology.org/P95-1026/|journal=Proceedings of the 33rd Annual Meeting of the Association for Computational Linguistics|pages=189–196|doi=10.3115/981658.981684|location=Cambridge, MA| publisher=Association for Computational Linguistics| access-date = 1 November 2022}}</ref> The model learns in two steps. First, the task is solved based on an auxiliary or pretext classification task using pseudo-labels which help to initialize the model parameters.<ref>{{Cite journal|last1=Doersch|first1=Carl|last2=Zisserman|first2=Andrew|date=October 2017|title=Multi-task Self-Supervised Visual Learning|url=http://dx.doi.org/10.1109/iccv.2017.226|journal=2017 IEEE International Conference on Computer Vision (ICCV)|pages=2070–2079|publisher=IEEE|doi=10.1109/iccv.2017.226|arxiv=1708.07860|isbn=978-1-5386-1032-9|s2cid=473729}}</ref><ref name="auto1">{{Cite journal|last1=Beyer|first1=Lucas|last2=Zhai|first2=Xiaohua|last3=Oliver|first3=Avital|last4=Kolesnikov|first4=Alexander|date=October 2019|title=S4L: Self-Supervised Semi-Supervised Learning|url=http://dx.doi.org/10.1109/iccv.2019.00156|journal=2019 IEEE/CVF International Conference on Computer Vision (ICCV)|pages=1476–1485|publisher=IEEE|doi=10.1109/iccv.2019.00156|arxiv=1905.03670|isbn=978-1-7281-4803-8|s2cid=167209887}}</ref> Second, the actual task is performed with supervised or unsupervised learning.<ref>{{Cite journal|last1=Doersch|first1=Carl|last2=Gupta|first2=Abhinav|last3=Efros|first3=Alexei A.|date=December 2015|title=Unsupervised Visual Representation Learning by Context Prediction|url=http://dx.doi.org/10.1109/iccv.2015.167|journal=2015 IEEE International Conference on Computer Vision (ICCV)|pages=1422–1430|publisher=IEEE|doi=10.1109/iccv.2015.167|arxiv=1505.05192|isbn=978-1-4673-8391-2|s2cid=9062671}}</ref><ref>{{Cite journal|last1=Zheng|first1=Xin|last2=Wang|first2=Yong|last3=Wang|first3=Guoyou|last4=Liu|first4=Jianguo|date=April 2018|title=Fast and robust segmentation of white blood cell images by self-supervised learning|url=http://dx.doi.org/10.1016/j.micron.2018.01.010|journal=Micron|volume=107|pages=55–71|doi=10.1016/j.micron.2018.01.010|pmid=29425969|issn=0968-4328}}</ref><ref>{{Cite journal|last1=Gidaris|first1=Spyros|last2=Bursuc|first2=Andrei|last3=Komodakis|first3=Nikos|last4=Perez|first4=Patrick Perez|last5=Cord|first5=Matthieu|date=October 2019|title=Boosting Few-Shot Visual Learning With Self-Supervision|url=http://dx.doi.org/10.1109/iccv.2019.00815|journal=2019 IEEE/CVF International Conference on Computer Vision (ICCV)|pages=8058–8067|publisher=IEEE|doi=10.1109/iccv.2019.00815|arxiv=1906.05186|isbn=978-1-7281-4803-8|s2cid=186206588}}</ref> Other auxiliary tasks involve pattern completion from masked input patterns (silent pauses in speech or image portions masked in black). Self-supervised learning has produced promising results in recent years and has found practical application in [[Audio signal processing|audio processing]] and is being used by [[Facebook]] and others for [[speech recognition]].<ref name="auto">{{Cite web|title=Wav2vec: State-of-the-art speech recognition through self-supervision|url=https://ai.facebook.com/blog/wav2vec-state-of-the-art-speech-recognition-through-self-supervision/|access-date=2021-06-09|website=ai.facebook.com|language=en}}</ref> The primary appeal of SSL is that training can occur with data of lower quality, rather than improving ultimate outcomes. Self-supervised learning more closely imitates the way humans learn to classify objects.<ref name="auto3">{{Cite web|last=Bouchard|first=Louis|date=2020-11-25|title=What is Self-Supervised Learning? {{!}} Will machines ever be able to learn like humans?|url=https://medium.com/what-is-artificial-intelligence/what-is-self-supervised-learning-will-machines-be-able-to-learn-like-humans-d9160f40cdd1|access-date=2021-06-09|website=Medium|language=en}}</ref>
The typical SSL method is based on an [[artificial neural network]] or other model such as a [[decision list]].<ref>{{cite journal|last1=Yarowsky|first1=David|date=1995|title=Unsupervised Word Sense Disambiguation Rivaling Supervised Methods|url=https://aclanthology.org/P95-1026/|journal=Proceedings of the 33rd Annual Meeting of the Association for Computational Linguistics|pages=189–196|doi=10.3115/981658.981684|location=Cambridge, MA| publisher=Association for Computational Linguistics| access-date = 1 November 2022}}</ref> The model learns in two steps. First, the task is solved based on an auxiliary or pretext classification task using pseudo-labels which help to initialize the model parameters.<ref>{{Cite journal|last1=Doersch|first1=Carl|last2=Zisserman|first2=Andrew|date=October 2017|title=Multi-task Self-Supervised Visual Learning|url=http://dx.doi.org/10.1109/iccv.2017.226|journal=2017 IEEE International Conference on Computer Vision (ICCV)|pages=2070–2079|publisher=IEEE|doi=10.1109/iccv.2017.226|arxiv=1708.07860|isbn=978-1-5386-1032-9|s2cid=473729}}</ref><ref name="auto1">{{Cite journal|last1=Beyer|first1=Lucas|last2=Zhai|first2=Xiaohua|last3=Oliver|first3=Avital|last4=Kolesnikov|first4=Alexander|date=October 2019|title=S4L: Self-Supervised Semi-Supervised Learning|url=http://dx.doi.org/10.1109/iccv.2019.00156|journal=2019 IEEE/CVF International Conference on Computer Vision (ICCV)|pages=1476–1485|publisher=IEEE|doi=10.1109/iccv.2019.00156|arxiv=1905.03670|isbn=978-1-7281-4803-8|s2cid=167209887}}</ref> Second, the actual task is performed with supervised or unsupervised learning.<ref>{{Cite journal|last1=Doersch|first1=Carl|last2=Gupta|first2=Abhinav|last3=Efros|first3=Alexei A.|date=December 2015|title=Unsupervised Visual Representation Learning by Context Prediction|url=http://dx.doi.org/10.1109/iccv.2015.167|journal=2015 IEEE International Conference on Computer Vision (ICCV)|pages=1422–1430|publisher=IEEE|doi=10.1109/iccv.2015.167|arxiv=1505.05192|isbn=978-1-4673-8391-2|s2cid=9062671}}</ref><ref>{{Cite journal|last1=Zheng|first1=Xin|last2=Wang|first2=Yong|last3=Wang|first3=Guoyou|last4=Liu|first4=Jianguo|date=April 2018|title=Fast and robust segmentation of white blood cell images by self-supervised learning|url=http://dx.doi.org/10.1016/j.micron.2018.01.010|journal=Micron|volume=107|pages=55–71|doi=10.1016/j.micron.2018.01.010|pmid=29425969|issn=0968-4328}}</ref><ref>{{Cite journal|last1=Gidaris|first1=Spyros|last2=Bursuc|first2=Andrei|last3=Komodakis|first3=Nikos|last4=Perez|first4=Patrick Perez|last5=Cord|first5=Matthieu|date=October 2019|title=Boosting Few-Shot Visual Learning With Self-Supervision|url=http://dx.doi.org/10.1109/iccv.2019.00815|journal=2019 IEEE/CVF International Conference on Computer Vision (ICCV)|pages=8058–8067|publisher=IEEE|doi=10.1109/iccv.2019.00815|arxiv=1906.05186|isbn=978-1-7281-4803-8|s2cid=186206588}}</ref> Other auxiliary tasks involve pattern completion from masked input patterns (silent pauses in speech or image portions masked in black). Self-supervised learning has produced promising results in recent years and has found practical application in [[Audio signal processing|audio processing]] and is being used by [[Facebook]] and others for [[speech recognition]].<ref name="auto">{{Cite web|title=Wav2vec: State-of-the-art speech recognition through self-supervision|url=https://ai.facebook.com/blog/wav2vec-state-of-the-art-speech-recognition-through-self-supervision/|access-date=2021-06-09|website=ai.facebook.com|language=en}}</ref> The primary appeal of SSL is that training can occur with data of lower quality, rather than improving ultimate outcomes. Self-supervised learning more closely imitates the way humans learn to classify objects.<ref name="auto3">{{Cite web|last=Bouchard|first=Louis|date=2020-11-25|title=What is Self-Supervised Learning? {{!}} Will machines ever be able to learn like humans?|url=https://medium.com/what-is-artificial-intelligence/what-is-self-supervised-learning-will-machines-be-able-to-learn-like-humans-d9160f40cdd1|access-date=2021-06-09|website=Medium|language=en}}</ref>

Revision as of 16:58, 17 December 2022

Self-supervised learning (SSL) refers to a machine learning paradigm, and corresponding methods, for processing unlabelled data to obtain useful representations that can help with downstream learning tasks. The most salient thing about SSL methods is that they do not need human-annotated labels, which means they are designed to take in datasets consisting entirely of unlabelled data samples. Then the typical SSL pipeline consists of learning supervisory signals (labels generated automatically) in a first stage, which are then used for some supervised learning task in the second and later stages. For this reason, SSL can be described as an intermediate form of unsupervised and supervised learning.

The typical SSL method is based on an artificial neural network or other model such as a decision list.[1] The model learns in two steps. First, the task is solved based on an auxiliary or pretext classification task using pseudo-labels which help to initialize the model parameters.[2][3] Second, the actual task is performed with supervised or unsupervised learning.[4][5][6] Other auxiliary tasks involve pattern completion from masked input patterns (silent pauses in speech or image portions masked in black). Self-supervised learning has produced promising results in recent years and has found practical application in audio processing and is being used by Facebook and others for speech recognition.[7] The primary appeal of SSL is that training can occur with data of lower quality, rather than improving ultimate outcomes. Self-supervised learning more closely imitates the way humans learn to classify objects.[8]

Types

For a binary classification task, training data can be divided into positive examples and negative examples. Positive examples are those that match the target. For example, if you're learning to identify birds, the positive training data are those pictures that contain birds. Negative examples are those that do not.[9]

Contrastive self-supervised learning

Contrastive self-supervised learning uses both positive and negative examples. Contrastive learning's loss function minimizes the distance between positive samples while maximizing the distance between negative samples.[9]

Non-contrastive self-supervised learning

Non-contrastive self-supervised learning (NCSSL) uses only positive examples. Counterintuitively, NCSSL converges on a useful local minimum rather than reaching a trivial solution, with zero loss. For the example of binary classification, it would trivially learn to classify each example as positive. Effective NCSSL requires an extra predictor on the online side that does not back-propagate on the target side.[9]

Comparison with other forms of machine learning

SSL belongs to supervised learning methods insofar as the goal is to generate a classified output from the input. At the same time, however, it does not require the explicit use of labeled input-output pairs. Instead, correlations, metadata embedded in the data, or domain knowledge present in the input are implicitly and autonomously extracted from the data.[10] These supervisory signals, generated from the data, can then be used for training.[8]

SSL is similar to unsupervised learning in that it does not require labels in the sample data. Unlike unsupervised learning, however, learning is not done using inherent data structures.[10]

Semi-supervised learning combines supervised and unsupervised learning, requiring only a small portion of the learning data be labeled.[3]

In transfer learning a model designed for one task is reused on a different task.[11]

Training an autoencoder intrinsically constitutes a self-supervised process, because the output pattern needs to become an optimal reconstruction of the input pattern itself. However, in current jargon, the term 'self-supervised' has become associated with classification tasks that are based on a pretext-task training setup. This involves the (human) design of such pretext task(s), unlike the case of fully self-contained autoencoder training[12].

Examples

Self-supervised learning is particularly suitable for speech recognition. For example, Facebook developed wav2vec, a self-supervised algorithm, to perform speech recognition using two deep convolutional neural networks that build on each other.[7]

Google's Bidirectional Encoder Representations from Transformers (BERT) model is used to better understand the context of search queries.[13]

OpenAI's GPT-3 is an autoregressive language model that can be used in language processing. It can be used to translate texts or answer questions, among other things.[14]

Bootstrap Your Own Latent is a NCSSL that produced excellent results on ImageNet and on transfer and semi-supervised benchmarks.[15]

The Yarowsky algorithm is an example of self-supervised learning in natural language processing. From a small number of labeled examples, it learns to predict which word sense of a polysemous word is being used at a given point in text.

DirectPred is a NCSSL that directly sets the predictor weights instead of learning it via gradient update.[9]

References

  1. ^ Yarowsky, David (1995). "Unsupervised Word Sense Disambiguation Rivaling Supervised Methods". Proceedings of the 33rd Annual Meeting of the Association for Computational Linguistics. Cambridge, MA: Association for Computational Linguistics: 189–196. doi:10.3115/981658.981684. Retrieved 1 November 2022.
  2. ^ Doersch, Carl; Zisserman, Andrew (October 2017). "Multi-task Self-Supervised Visual Learning". 2017 IEEE International Conference on Computer Vision (ICCV). IEEE: 2070–2079. arXiv:1708.07860. doi:10.1109/iccv.2017.226. ISBN 978-1-5386-1032-9. S2CID 473729.
  3. ^ a b Beyer, Lucas; Zhai, Xiaohua; Oliver, Avital; Kolesnikov, Alexander (October 2019). "S4L: Self-Supervised Semi-Supervised Learning". 2019 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE: 1476–1485. arXiv:1905.03670. doi:10.1109/iccv.2019.00156. ISBN 978-1-7281-4803-8. S2CID 167209887.
  4. ^ Doersch, Carl; Gupta, Abhinav; Efros, Alexei A. (December 2015). "Unsupervised Visual Representation Learning by Context Prediction". 2015 IEEE International Conference on Computer Vision (ICCV). IEEE: 1422–1430. arXiv:1505.05192. doi:10.1109/iccv.2015.167. ISBN 978-1-4673-8391-2. S2CID 9062671.
  5. ^ Zheng, Xin; Wang, Yong; Wang, Guoyou; Liu, Jianguo (April 2018). "Fast and robust segmentation of white blood cell images by self-supervised learning". Micron. 107: 55–71. doi:10.1016/j.micron.2018.01.010. ISSN 0968-4328. PMID 29425969.
  6. ^ Gidaris, Spyros; Bursuc, Andrei; Komodakis, Nikos; Perez, Patrick Perez; Cord, Matthieu (October 2019). "Boosting Few-Shot Visual Learning With Self-Supervision". 2019 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE: 8058–8067. arXiv:1906.05186. doi:10.1109/iccv.2019.00815. ISBN 978-1-7281-4803-8. S2CID 186206588.
  7. ^ a b "Wav2vec: State-of-the-art speech recognition through self-supervision". ai.facebook.com. Retrieved 2021-06-09.
  8. ^ a b Bouchard, Louis (2020-11-25). "What is Self-Supervised Learning? | Will machines ever be able to learn like humans?". Medium. Retrieved 2021-06-09.
  9. ^ a b c d "Demystifying a key self-supervised learning technique: Non-contrastive learning". ai.facebook.com. Retrieved 2021-10-05.
  10. ^ a b R., Poornima; L., Ashok (2017). "Problem Based Learning a Shift from Teaching Paradigm to the Learning Paradigm". Indian Journal of Dental Education. 10 (1): 47–51. doi:10.21088/ijde.0974.6099.10117.6. ISSN 0974-6099.
  11. ^ Littwin, Etai; Wolf, Lior (June 2016). "The Multiverse Loss for Robust Transfer Learning". 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE: 3957–3966. arXiv:1511.09033. doi:10.1109/cvpr.2016.429. ISBN 978-1-4673-8851-1. S2CID 6517610.
  12. ^ Kramer, Mark A. (1991). "Nonlinear principal component analysis using autoassociative neural networks" (PDF). AIChE Journal. 37 (2): 233–243. doi:10.1002/aic.690370209.
  13. ^ "Open Sourcing BERT: State-of-the-Art Pre-training for Natural Language Processing". Google AI Blog. Retrieved 2021-06-09.
  14. ^ Wilcox, Ethan; Qian, Peng; Futrell, Richard; Kohita, Ryosuke; Levy, Roger; Ballesteros, Miguel (2020). "Structural Supervision Improves Few-Shot Learning and Syntactic Generalization in Neural Language Models". Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Stroudsburg, PA, USA: Association for Computational Linguistics: 4640–4652. arXiv:2010.05725. doi:10.18653/v1/2020.emnlp-main.375. S2CID 222291675.
  15. ^ Grill, Jean-Bastien; Strub, Florian; Altché, Florent; Tallec, Corentin; Richemond, Pierre H.; Buchatskaya, Elena; Doersch, Carl; Pires, Bernardo Avila; Guo, Zhaohan Daniel; Azar, Mohammad Gheshlaghi; Piot, Bilal (2020-09-10). "Bootstrap your own latent: A new approach to self-supervised Learning". arXiv:2006.07733 [cs.LG].
Retrieved from "https://en.wikipedia.org/w/index.php?title=Self-supervised_learning&oldid=1127962122"