Your personalresearch library
Discover papers, track what you're reading, rate and review.Build a library that grows with your curiosity.
Currently Reading
Want to Read
All Papers
Advanced filtersTopic: AllExperimental
Topic
You Can’t AI-Proof the Classroom, Experts Say. Get Creative Instead.
Emma Whitford
Inside Higher Ed | Higher Education News, Events and Jobs·2025
Artificial intelligence is here to stay. Faculty members explain how to keep students engaged in their own learning and prevent them from relying on AI. Blue books made a comeback in 2025. In an effort to prevent students from feeding final essay prompts into ChatGPT, some professors asked their students to sit down and write in-person in the lined, sky-blue booklets that served as the college standard for written assessments in the pre-laptop era.
★ 4.3 (4)
View paper →SonicMoE: Accelerating MoE with IO and Tile-aware Optimizations
Wentao Guo, Mayank Mishra, Xinle Cheng, Ion Stoica, Tri Dao
arXiv·2025
Mixture of Experts (MoE) models have emerged as the de facto architecture for scaling up language models without significantly increasing the computational cost. Recent MoE models demonstrate a clear trend towards high expert granularity (smaller expert intermediate dimension) and higher sparsity (constant number of activated experts with higher number of total experts), which improve model quality per FLOP. However, fine-grained MoEs suffer from increased activation memory footprint and reduced hardware efficiency due to higher IO costs, while sparser MoEs suffer from wasted computations due to padding in Grouped GEMM kernels. In response, we propose a memory-efficient algorithm to compute the forward and backward passes of MoEs with minimal activation caching for the backward pass. We also design GPU kernels that overlap memory IO with computation benefiting all MoE architectures. Finally, we propose a novel "token rounding" method that minimizes the wasted compute due to padding in Grouped GEMM kernels. As a result, our method SonicMoE reduces activation memory by 45% and achieves a 1.86x compute throughput improvement on Hopper GPUs compared to ScatterMoE's BF16 MoE kernel for a fine-grained 7B MoE. Concretely, SonicMoE on 64 H100s achieves a training throughput of 213 billion tokens per day comparable to ScatterMoE's 225 billion tokens per day on 96 H100s for a 7B MoE model training with FSDP-2 using the lm-engine codebase. Under high MoE sparsity settings, our tile-aware token rounding algorithm yields an additional 1.16x speedup on kernel execution time compared to vanilla top-$K$ routing while maintaining similar downstream performance. We open-source all our kernels to enable faster MoE model training.
No ratings yet
View paper →From the Ruins of Utopia - Journal #159
Julieta Aranda
e-flux.com·2025
Julieta Aranda imagines a nonhierarchical, world-making masculinity.
No ratings yet
View paper →