Oldest Papers
Statistical Physics Analysis of Graph Neural Networks: Approaching Optimality in the Contextual Stochastic Block Model
O. Duranthon, L. Zdeborová
Physical Review X·2025
<jats:p>Graph neural networks (GNNs) are designed to process data associated with graphs. They are finding an increasing range of applications; however, as with other modern machine learning techniques, their theoretical understanding is limited. GNNs can encounter difficulties in gathering information from nodes that are far apart by iterated aggregation steps. This situation is partly caused by so-called oversmoothing; and overcoming it is one of the practically motivated challenges. We consider the situation where information is aggregated by multiple steps of convolution, leading to graph convolutional networks (GCNs). We analyze the generalization performance of a basic GCN, trained for node classification on data generated by the contextual stochastic block model. We predict its asymptotic performance by deriving the free energy of the problem, using the replica method, in the high-dimensional limit. Calling the number of convolutional steps, we show the importance of going to large depth to approach the Bayes optimality. We detail how the architecture of the GCN has to scale with the depth to avoid oversmoothing. The resulting large depth limit can be close to the Bayes optimality and leads to a continuous GCN. Technically, we tackle this continuous limit via an approach that resembles dynamical mean-field theory with constraints at the initial and final times. An expansion around large regularization allows us to solve the corresponding equations for the performance of the deep GCN. This promising tool may contribute to the analysis of further deep neural networks.</jats:p>
No ratings yet
View paper →SYNTH: the new data frontier
Pleias Authors
pleias.fr·2025
Since GPT-3, language models have been mostly trained on large collections of web archives. During the past year, Frontier AI labs have reconsidered this approach as they move toward reasoning and agentic models requiring a large amount of generally unwritten traces for thoughts, actions or tool calls.
We release SYNTH, a fully generalist synthetic dataset, that represents a fundamental break from the common pre-training paradigm: what if we trained for reasoning and focused on the assimilation of knowledge and skill that matters? Classic benchmarks are already operating under this assumption: MMLU, gsm8k, MATH, are all ultimately based on collections of high school exercises.
SYNTH stems from 50,000 vital Wikipedia articles, expanded into a wide collection of problems and resolution paths, from math exercises to creative writing, information extraction or sourced synthesis. While similar traces might be available in standard pretraining datasets, they are isolated and noisy which ultimately slows down the learning process and delays the acquisition of valuable reasoning skills.
We trained two SOTA small reasoners - Baguettotron and Monad - on less than 200 billion tokens from SYNTH, that is between 10 and 50 less data than models with similar or lower performance. Baguettotron is currently the best in class across major industry benchmarks (MMLU, gsm8k, HotPotQA). 56M parameters Monad is a contender for the smallest viable model, as it attains non-random performance on the same benchmark suite.
No ratings yet
View paper →