Newest and Lowest Rated Papers
On the Relationship between Self-Attention and Convolutional Layers
Jean-Baptiste Cordonnier, Andreas Loukas, Martin Jaggi
arXiv·2019
Recent trends of incorporating attention mechanisms in vision have led researchers to reconsider the supremacy of convolutional layers as a primary building block. Beyond helping CNNs to handle long-range dependencies, Ramachandran et al. (2019) showed that attention can completely replace convolution and achieve state-of-the-art performance on vision tasks. This raises the question: do learned attention layers operate similarly to convolutional layers? This work provides evidence that attention layers can perform convolution and, indeed, they often learn to do so in practice. Specifically, we prove that a multi-head self-attention layer with sufficient number of heads is at least as expressive as any convolutional layer. Our numerical experiments then show that self-attention layers attend to pixel-grid patterns similarly to CNN layers, corroborating our analysis. Our code is publicly available.
No ratings yet
View paper →Do Vision Transformers See Like Convolutional Neural Networks?
Raghu, Maithra, Unterthiner, Thomas, Kornblith, Simon, Zhang, Chiyuan, Dosovitskiy, Alexey
proceedings.neurips.cc
No abstract available
No ratings yet
View paper →Does Object Binding Naturally Emerge in Large Pretrained Vision Transformers?
Yihao Li
openreview.net
Object binding, the brain’s ability to bind the many features that collectively
represent an object into a coherent whole, is central to human cognition. It groups
low-level perceptual features into high-level object representations, stores those
objects efficiently and compositionally in memory, and supports human reasoning
about individual object instances. While prior work often imposes object-centric
attention (e.g., Slot Attention) explicitly to probe these benefits, it remains unclear
whether this ability naturally emerges in pre-trained Vision Transformers (ViTs).
Intuitively, they could: recognizing which patches belong to the same object should
be useful for downstream prediction and thus guide attention. Motivated by the
quadratic nature of self-attention, we hypothesize that ViTs represent whether
two patches belong to the same object, a property we term IsSameObject. We
decode IsSameObject from patch embeddings across ViT layers using a similarity
probe, which reaches over 90% accuracy. Crucially, this object-binding capability
emerges reliably in self-supervised ViTs (DINO, MAE, CLIP), but markedly
weaker in ImageNet-supervised models, suggesting that binding is not a trivial
architectural artifact, but an ability acquired through specific pretraining objectives.
We further discover that IsSameObject is encoded in a low-dimensional subspace
on top of object features, and that this signal actively guides attention. AblatingIsSameObject from model activations degrades downstream performance and works
against the learning objective, implying that emergent object binding naturally
serves the pretraining objective. Our findings challenge the view that ViTs lack
object binding and highlight how symbolic knowledge of “which parts belong
together” emerges naturally in a connectionist system.
No ratings yet
View paper →Dynamic Model of Visual Recognition Predicts Neural Response Properties in the Visual Cortex
Rajesh P. N. Rao, Dana H. Ballard
Neural Computation·1997·194 citations
The responses of visual cortical neurons during fixation tasks can be significantly modulated by stimuli from beyond the classical receptive field. Modulatory effects in neural responses have also been recently reported in a task where a monkey freely views a natural scene. In this article, we describe a hierarchical network model of visual recognition that explains these experimental observations by using a form of the extended Kalman filter as given by the minimum description length (MDL) principle. The model dynamically combines input-driven bottom-up signals with expectation-driven top-down signals to predict current recognition state. Synaptic weights in the model are adapted in a Hebbian manner according to a learning rule also derived from the MDL principle. The resulting prediction-learning scheme can be viewed as implementing a form of the expectation-maximization (EM) algorithm. The architecture of the model posits an active computational role for the reciprocal connections between adjoining visual cortical areas in determining neural response properties. In particular, the model demonstrates the possible role of feedback from higher cortical areas in mediating neurophysiological effects due to stimuli from beyond the classical receptive field. Simulations of the model are provided that help explain the experimental observations regarding neural responses in both free viewing and fixating conditions.
No ratings yet
View paper →Convolutional Neural Networks as a Model of the Visual System: Past, Present, and Future
Grace W. Lindsay
Journal of Cognitive Neuroscience·2021·377 citations
Convolutional neural networks (CNNs) were inspired by early findings in the study of biological vision. They have since become successful tools in computer vision and state-of-the-art models of both neural activity and behavior on visual tasks. This review highlights what, in the context of CNNs, it means to be a good model in computational neuroscience and the various ways models can provide insight. Specifically, it covers the origins of CNNs and the methods by which we validate them as models of biological vision. It then goes on to elaborate on what we can learn about biological vision by understanding and experimenting on CNNs and discusses emerging opportunities for the use of CNNs in vision research beyond basic object recognition.
No ratings yet
View paper →Sergey Tsokolov: A Brief Life In Science
Unknown Author
Science 2.0·2014
A few days ago, I was working at home when the phone rang. I answered, and was surprised to hear a soft, accented voice asking for me. It was Lada Tsokolova, calling from Germany, with the sad news that her husband Sergey had just died of cancer. I was stunned. Sergey was young!
No ratings yet
View paper →Transformations of Lamarckism: From Subtle Fluids to Molecular Biology - Google Books
Unknown Author
Google Books
In 1809 -- the year of Charles Darwin's birth -- Jean-Baptiste Lamarck published Philosophie zoologique, the first comprehensive and systematic theory of biological evolution. The Lamarckian approach emphasizes the generation of developmental variations; Darwinism stresses selection. Lamarck's ideas were eventually eclipsed by Darwinian concepts, especially after the emergence of the Modern Synthesis in the twentieth century. The different approaches -- which can be seen as complementary rather than mutually exclusive -- have important implications for the kinds of questions biologists ask and for the type of research they conduct. Lamarckism has been evolving -- or, in Lamarckian terminology, transforming -- since Philosophie zoologique's description of biological processes mediated by "subtle fluids." Essays in this book focus on new developments in biology that make Lamarck's ideas relevant not only to modern empirical and theoretical research but also to problems in the philosophy of biology. Contributors discuss the historical transformations of Lamarckism from the 1820s to the 1940s, and the different understandings of Lamarck and Lamarckism; the Modern Synthesis and its emphasis on Mendelian genetics; theoretical and experimental research on such "Lamarckian" topics as plasticity, soft (epigenetic) inheritance, and individuality; and the importance of a developmental approach to evolution in the philosophy of biology. The book shows the advantages of a "Lamarckian" perspective on evolution. Indeed, the development-oriented approach it presents is becoming central to current evolutionary studies -- as can be seen in the burgeoning field of Evo-Devo. Transformations of Lamarckism makes a unique contribution to this research.
No ratings yet
View paper →Listening Otherwise, Music Miniaturized: A Different Type of Question about Revolution
Rey Chow
Discourse·1990
No abstract available
No ratings yet
View paper →Liberation Is Not Integration: On liberal Zionism, one-state fantasies, and what Palestinians actually want – Mondoweiss
Lara Kilani
Mondoweiss·2025
The left’s view of Palestine’s future often reduces to vague terms like “one-state solution” or “equal rights for all” but few address the tough questions they raise, especially: how can Palestinians live with those who carried out the Gaza genocide?
No ratings yet
View paper →The Polyketide Synthase Gene <i>pks4</i> of Trichoderma reesei Provides Pigmentation and Stress Resistance
Lea Atanasova, Benjamin P. Knox, Christian P. Kubicek, Irina S. Druzhinina, Scott E. Baker
Eukaryotic Cell·2013·82 citations
<jats:title>ABSTRACT</jats:title>
<jats:p>
Species of the fungal genus
<jats:named-content content-type="genus-species">Trichoderma</jats:named-content>
(Hypocreales, Ascomycota) are well-known for their production of various secondary metabolites. Nonribosomal peptides and polyketides represent a major portion of these products. In a recent phylogenomic investigation of
<jats:named-content content-type="genus-species">Trichoderma</jats:named-content>
polyketide synthase (PKS)-encoding genes, the
<jats:italic>pks4</jats:italic>
from
<jats:named-content content-type="genus-species">T. reesei</jats:named-content>
was shown to be an orthologue of pigment-forming PKSs involved in synthesis of aurofusarin and bikaverin in
<jats:named-content content-type="genus-species">Fusarium</jats:named-content>
spp. In this study, we show that deletion of this gene in
<jats:named-content content-type="genus-species">T. reesei</jats:named-content>
results in loss of green conidial pigmentation and in pigmentation alteration of teleomorph structures. It also has an impact on conidial cell wall stability and the antagonistic abilities of
<jats:named-content content-type="genus-species">T. reesei</jats:named-content>
against other fungi, including formation of inhibitory metabolites. In addition, deletion of
<jats:italic>pks4</jats:italic>
significantly influences the expression of other PKS-encoding genes of
<jats:named-content content-type="genus-species">T. reesei</jats:named-content>
. To our knowledge, this is the first indication that a low-molecular-weight pigment-forming PKS is involved in defense, mechanical stability, and stress resistance in fungi.
</jats:p>
No ratings yet
View paper →Outsider, Witch, and Theorist: a Character Analysis
Nick
Substack·2025
Or, "The Coco Essay." Contains spoilers up through chapter 90.
No ratings yet
View paper →The Scholar and the State | Fiction as Political Discourse in Late Imperial China
Liangyan Ge
uw.manifoldapp.org·2015
In imperial China, intellectuals devoted years of their lives to passing rigorous examinations in order to obtain a civil service position in the state bureaucracy. This traditional employment of the literati class conferred social power and moral legitimacy, but changing social and political circumstances in the Ming (1368–1644) and Qing (1644–1911) periods forced many to seek alternative careers. Politically engaged but excluded from their traditional bureaucratic roles, creative writers authored critiques of state power in the form of fiction written in the vernacular language.
In this study, Liangyan Ge examines the novels *Romance of the Three Kingdoms*, *The Scholars*, *Dream of the Red Chamber* (also known as *Story of the Stone*), and a number of erotic pieces, showing that as the literati class grappled with its own increasing marginalization, its fiction reassessed the assumption that intellectuals’ proper role was to serve state interests and began to imagine possibilities for a new political order.
No ratings yet
View paper →