Minsuk Chang

Can we develop a roadmap for how to build the next generation of large language models? Find out in our latest People of ACM profile as we interview Sewon Min, an Assistant Professor at University of California, Berkeley and a research scientist at the Allen Institute for AI. She is part of the Berkeley Artificial Intelligence Research Lab (BAIR) as well as the Berkeley NLP Group. Min was one of only two people who received an Honorable Mention for the ACM Doctoral Dissertation Award. In her interview, she explains what 'non-parametric' large language models are and how they are advancing the power and precision of AI applications. She also predicts that recent innovations will open up new forms of collaboration between data owners, model developers, and users, benefiting all parties involved. Read the full interview here: https://buff.ly/s6w2D5L

29

1 Comment

I'm excited to share a recent work "CoLA: Compute-Efficient Pre-Training of LLMs via Low-Rank Activation", led by my students Ziyue Liu, Ruijie Zhang and Zhengyang Wang. Paper link: https://lnkd.in/gGM_Pvgt (code link availabe at the bottom of Page 1). Key idea: it is widely observed that the activation values in neural networks have low-dim structures, so we replace each linear transform h=Wx [or followed by an activation function in an MLP layer] with a bottleneck structure [see Fig. (a)]: --replace the weight matrix W as a low-rank factorization W=BA; --insert nonlinear activations between B and A. Main benefits: CoLA and its memory-efficient impelmentation (CoLA-M) can acheive all 4 major goals simultaneously [see Fig. (b) and (c)]: --GPU memory reduction; --LLM model size reduction (by 50%); --Pre-training (1.86x) and inference (1.67x) throughput improvement; --Maintaining (or even improving) the performance (perplexity) of standard pre-training. This method is tested on the LlaMA architecture with 60M to 7B parameters. No pruning, quantization or GPU optimization is used. Collaborators: Franck Cappello, Bogdan Nicolae and Paul Hovland (Argonne National Labs), Zi Yang (SUNY at Albany).

171

3 Comments

This week in The Batch, Andrew Ng urges developers who want to stay competitive to pair strong computer science fundamentals with AI-assisted, agentic coding skills. Plus: 💼 AI-led interviews improved hiring and retention, study shows 🏙️ Hangzhou, China’s emerging AI hub 🌿 Google quantifies Gemini’s per-prompt footprint 🛡️ Meta introduces LlamaFirewall to secure agentic LLMs Read The Batch: https://hubs.la/Q03G_MPk0

51

2 Comments

🚀 Excited to share Compressive Light-Field Tokens (CLiFT) — a new scene representation for neural rendering developed by my students (Zhengqing Wang, Yuefan Wu, Jiacheng Chen, Fuyang Zhang). CLiFT is a Light-Field Token, compressed for compute-efficient and adaptive neural rendering, achieving (2x to 16x) reduction in storage and (66% to 100%) increase in FPS, while preserving high visual quality with only (0.54dB to 3dB) drop in PSNR. 🔗 Project: https://lnkd.in/gBqvMWi8 📦 Code coming 8/1!

166

3 Comments