Oldest Papers

This paper shows that the masked-modelling principle driving the success of large foundational language models can be effectively applied to video by making predictions in latent space. We introduce V-JEPA, a method for self-supervised learning from video that predicts masked spatio-temporal regions in a learned representation space. Our latent video prediction strategy produces visual features that can be applied to various downstream image and video tasks without adaption of the model's parameters (using only frozen evaluation), achieving 82.1% on Kinetics-400 and 71.2% on Something-Something-v2, surpassing the previous best video models by +4 and +10 points respectively. We also demonstrate the benefit of video pretraining compared to image pretraining for tasks involving motion understanding, where V-JEPA outperforms the largest state-of-the-art image models, DINOv2 and OpenCLIP. Finally, V-JEPA trained only on video achieves 77.9% on ImageNet classification without any image fine-tuning, surpassing the previous best video model by +6 points top-1.

No ratings yet

View paper →

Improving language understanding by generative pre-training

A Radford, K Narasimhan, T Salimans, I Sutskever

Natural language understanding comprises a wide range of diverse tasks such as textual entailment, question answering, semantic similarity assessment, and document classification. Although large unlabeled text corpora are abundant, labeled data for learning these specific tasks is scarce, making it challenging for discriminatively trained models to perform adequately. We demonstrate that large gains on these tasks can be realized by generative pre-training of a language model on a diverse corpus of unlabeled text, followed by discriminative fine-tuning on each specific task. In contrast to previous approaches, we make use of task-aware input transformations during fine-tuning to achieve effective transfer while requiring minimal changes to the model architecture. We demonstrate the effectiveness of our approach on a wide range of benchmarks for natural language understanding. Our general task-agnostic model outperforms discriminatively trained models that use architectures specifically crafted for each task, significantly improving upon the state of the art in 9 out of the 12 tasks studied. For instance, we achieve absolute improvements of 8.9% on commonsense reasoning (Stories Cloze Test), 5.7% on question answering (RACE), and 1.5% on textual entailment (MultiNLI).

No ratings yet

View paper →

Discourse analysis as a new cross-discipline

Teun A. van Dijk

Slideshare

1. Discourse analysis has its origins in classical rhetoric but declined as an academic discipline. It reemerged in the 1960s through structuralist analyses of narratives, myths, and forms of communication across disciplines like anthropology, linguistics, and semiotics. 2. In the mid-1960s, structural analyses of discourse were published that applied structural linguistics to literature, film, and other cultural forms. This sparked interest in systematically studying language use, discourse, and communication across fields. 3. By the early 1970s, discourse analysis was emerging as a new interdisciplinary field drawing from functional linguistics, text linguistics, and approaches studying indigenous narratives, conversation, and other genres. - Download as a PDF or view online for free

★ 5.0 (1)

View paper →