Your personalresearch library

Discover papers, track what you're reading, rate and review.Build a library that grows with your curiosity.

Get Started Explore Papers

Currently Reading

Want to Read

All Papers

Publication Date

Newest Oldest

Average Rating

Highest Lowest

Flash Multi-Head Feed-Forward Network

Minshen Zhang, Xiang Hu, Jianguo Li, Wei Wu, Kewei Tu

arXiv·2025

We explore Multi-Head FFN (MH-FFN) as a replacement of FFN in the Transformer architecture, motivated by the structural similarity between single-head attention and FFN. While multi-head mechanisms enhance expressivity in attention, naively applying them to FFNs faces two challenges: memory consumption scaling with the head count, and an imbalanced ratio between the growing intermediate size and the fixed head dimension as models scale, which degrades scalability and expressive power. To address these challenges, we propose Flash Multi-Head FFN (FlashMHF), with two key innovations: an I/O-aware fused kernel computing outputs online in SRAM akin to FlashAttention, and a design using dynamically weighted parallel sub-networks to maintain a balanced ratio between intermediate and head dimensions. Validated on models from 128M to 1.3B parameters, FlashMHF consistently improves perplexity and downstream task accuracy over SwiGLU FFNs, while reducing peak memory usage by 3-5x and accelerating inference by up to 1.08x. Our work establishes the multi-head design as a superior architectural principle for FFNs, presenting FlashMHF as a powerful, efficient, and scalable alternative to FFNs in Transformers.

★ 5.0 (1)

View paper →

Task adaptation of Vision-Language-Action model: 1st Place Solution for the 2025 BEHAVIOR Challenge

Ilia Larchenko, Gleb Zarin, Akash Karnatak

arXiv·2025

We present a vision-action policy that won 1st place in the 2025 BEHAVIOR Challenge - a large-scale benchmark featuring 50 diverse long-horizon household tasks in photo-realistic simulation, requiring bimanual manipulation, navigation, and context-aware decision making. Building on the Pi0.5 architecture, we introduce several innovations. Our primary contribution is correlated noise for flow matching, which improves training efficiency and enables correlation-aware inpainting for smooth action sequences. We also apply learnable mixed-layer attention and System 2 stage tracking for ambiguity resolution. Training employs multi-sample flow matching to reduce variance, while inference uses action compression and challenge-specific correction rules. Our approach achieves 26% q-score across all 50 tasks on both public and private leaderboards.

No ratings yet

View paper →

When Allah Gives, That Too Is a Test.

yumna 💌

Substack·2025

Every gift carries a quiet question: what will you do with what you’ve been given?

No ratings yet

View paper →

Shtetl-Optimized » Blog Archive » Theory and AI Alignment

Unknown Author

Shtetl-Optimized·2025

The following is based on a talk that I gave (remotely) at the UK AI Safety Institute Alignment Workshop on October 29, and which I then procrastinated for more than a month in writing up. Enjoy! Thanks for having me! I'm a theoretical computer scientist. I've spent most of my career for ~25 years studying…

No ratings yet

View paper →

PreviousPage 20 of 436Next

Eliezer's Unteachable Methods of Sanity

Eliezer Yudkowsky

LessWrong·2025

"How are you coping with the end of the world?" journalists sometimes ask me, and the true answer is something they have no hope of understanding and I have no hope of explaining in 30 seconds, so I usually answer something like, "By having a great distaste for drama, and remembering that it's not about me." The journalists don't understand that either, but at least I haven't wasted much time along the way. Actual LessWrong readers also sometimes ask me how I deal emotionally with the end of the world. I suspect a more precise answer may not help. But Raymond Arnold thinks I should say it, so I will say it. I say again, I don't actually think my answer is going to help. Wisely did Ozy write, "Other People Might Just Not Have Your Problems." Also I don't have a bunch of other people's problems, and other people can't make internal function calls that I've practiced to the point of hardly noticing them. I don't expect that my methods of sanity will be reproducible by nearly anyone. I feel pessimistic that they will help to hear about. Raymond Arnold asked me to speak them anyways, so I will. ---------------------------------------- Stay genre-savvy / be an intelligent character. The first and oldest reason I stay sane is that I am an author, and above tropes. Going mad in the face of the oncoming end of the world is a trope. I consciously see those culturally transmitted patterns that inhabit thought processes aka tropes, both in fiction, and in the narratives tha...

No ratings yet

View paper →

AI in 2025: gestalt

technicalities, Gavin Leech

LessWrong·2025

This is the editorial for this year’s "Shallow Review of AI Safety". (It got long enough to stand alone.) Epistemic status: subjective impressions plus one new graph plus 300 links. Huge thanks to Jaeho Lee, Jaime Sevilla, and Lexin Zhou for running lots of tests pro bono and so greatly improving the main analysis. tl;dr * Informed people disagree about the prospects for LLM AGI – or even just what exactly was achieved this year. But they at least agree that we’re 2-20 years off (if you allow for other paradigms arising). In this piece I stick to arguments rather than reporting who thinks what. * My view: compared to last year, AI is much more impressive but not proportionally more useful. They improved on some things they were explicitly optimised for (coding, vision, OCR, benchmarks), and did not hugely improve on everything else. Progress is thus (still!) consistent with current frontier training bringing more things in-distribution rather than generalising very far. * Pretraining (GPT-4.5, Grok 3/4, but also the counterfactual large runs which weren’t done) disappointed people this year. It’s probably not because it wouldn’t work; it was just ~30 times more efficient to do post-training instead, on the margin. This should change, yet again, soon, if RL scales even worse. * Edit: See this amazing comment for the hardware reasons behind this, and reasons to think that pretraining will struggle for years. * True frontier capabilities are likely obscured by systema...

★ 5.0 (6)

View paper →

On fine-tuning Boltz-2 for protein-protein affinity prediction

James King, Lewis Cornwall, Andrei Cristian Nica, James Day, Aaron Sim, Neil Dalchau, Lilly Wollman, Joshua Meyers

arXiv·2025

Accurate prediction of protein-protein binding affinity is vital for understanding molecular interactions and designing therapeutics. We adapt Boltz-2, a state-of-the-art structure-based protein-ligand affinity predictor, for protein-protein affinity regression and evaluate it on two datasets, TCR3d and PPB-affinity. Despite high structural accuracy, Boltz-2-PPI underperforms relative to sequence-based alternatives in both small- and larger-scale data regimes. Combining embeddings from Boltz-2-PPI with sequence-based embeddings yields complementary improvements, particularly for weaker sequence models, suggesting different signals are learned by sequence- and structure-based models. Our results echo known biases associated with training with structural data and suggest that current structure-based representations are not primed for performant affinity prediction.

★ 4.0 (1)

View paper →

mTORC1-Dependent Signaling in Layer 5b Neurons Is Required for Memory Consolidation

Schutte, M., Biermeier, A. J., Konkel, K., Nasr, A., Dayaram, A., Rosenmund, C., Sachdev, R., Larkum, M. E., Haucke, V., Mikhaylova, M.

bioRxiv·2025

Memory consolidation relies on activity-dependent neuronal plasticity, particularly in deep-layer cortical neurons that integrate long-range inputs from medial temporal lobe (MTL) structures. In the neocortex, apical tuft dendrites of layer 5b pyramidal neurons receive direct input from the MTL within cortical layer 1, and their activation is critical for associative learning. However, the molecular mechanisms that enable learning-induced plasticity in these neurons remain poorly understood. Here, we demonstrate that activity-dependent, mTORC1-regulated translation in layer 5b neurons contributes to learning and memory. Using in vivo paradigms such as exposure to an enriched environment (EE) and in vitro models of network activation by chemical long-term potentiation (cLTP), combined with cell type-specific translational profiling (RiboTag), we identify candidate mRNAs associated with experience-dependent plasticity. We further show that pharmacological inhibition of canonical mTORC1 signaling during memory-relevant tasks abrogated memory consolidation. Our findings identify the mTORC1 pathway as a central regulator of learning-induced protein translation and cortical plasticity.

★ 1.0 (1)

View paper →

Your personalresearch library

All Papers

Flash Multi-Head Feed-Forward Network

Task adaptation of Vision-Language-Action model: 1st Place Solution for the 2025 BEHAVIOR Challenge

When Allah Gives, That Too Is a Test.

Shtetl-Optimized » Blog Archive » Theory and AI Alignment

Eliezer's Unteachable Methods of Sanity

AI in 2025: gestalt

On fine-tuning Boltz-2 for protein-protein affinity prediction

Lesbian Space Princess - WLW Film Reviews

how to make money as a random dickhead

Mechanical Habits

Validation of the Parental Worries About Racial Profiling Scale Across Black and Latine Parents

mTORC1-Dependent Signaling in Layer 5b Neurons Is Required for Memory Consolidation

Your personalresearch library

All Papers

Flash Multi-Head Feed-Forward Network

Task adaptation of Vision-Language-Action model: 1st Place Solution for the 2025 BEHAVIOR Challenge

When Allah Gives, That Too Is a Test.

Shtetl-Optimized &raquo; Blog Archive &raquo; Theory and AI Alignment

Eliezer's Unteachable Methods of Sanity

AI in 2025: gestalt

On fine-tuning Boltz-2 for protein-protein affinity prediction

Lesbian Space Princess - WLW Film Reviews

how to make money as a random dickhead

Mechanical Habits

Validation of the Parental Worries About Racial Profiling Scale Across Black and Latine Parents

mTORC1-Dependent Signaling in Layer 5b Neurons Is Required for Memory Consolidation

Shtetl-Optimized » Blog Archive » Theory and AI Alignment