Mark Saroufim

Some recent, exciting news on compressed tokenizers and how individual tokens modify images in encoder-decoder models. I hope this opens up a new avenue of research with smaller models across frameworks (perhaps JEPA?). https://lnkd.in/evFw8WBu

69

2 Comments

Ok, lets see just how cool blisteringly fast code generation can be. Here is very long prompt borrowed from Daniel Kim. Its important to see the prompt, so I put it below. Sit back and marvel at just how fast Cerebras Systems inference is. Prompt: a Python script that uses Pygame to simulate a single red ball bouncing inside a rotating regular hexagon, all in SI units with a conversion of 100 pixels = 1 meter. The window should be 800×600 pixels at 60 FPS, with a black background. The hexagon is centered on the window, has a radius of 250 pixels (2.5 m), and rotates clockwise at 60 degrees per second. The hexagon does not respond to gravity, but the ball does. Use a gravitational acceleration of 9.81 m/s², and give the ball a restitution of 0.92 so it bounces. The ball should be 10 pixels in radius (0.1 m), colored red, and start near the hexagon’s center but offset upwards by 1 meter. It should have an initial horizontal velocity of 1 m/s (i.e., 100 pixels/s). To prevent tunneling through the hexagon edges, implement multiple substeps (e.g., 5) each frame and carefully resolve collisions by pushing the ball out of the polygon when penetration occurs, then reflecting its velocity. Render the hexagon’s edges in white with a line width of 3, draw the red ball as a circle, and label the window caption as “Red Ball in a Rotating Hexagon (SI Units).” End the script cleanly when the user closes the window.

149

2 Comments

Stock vesting schedules have been morphing, as we all know. And Waymo just slipped a new experiment into the mix. Two recent Waymo offers shared on Levels.fyi tell the story: - An L3 software-engineer package that vests RSUs over two years, 50 percent each year. - An L5 package that keeps the classic four-year, evenly split design. That contrast hints at a deliberate split strategy: shorter horizons for early-career hires, longer runways for senior talent. The logic tracks. New grads often treat their first role as a springboard, so a two-year vest ensures most of their grant actually pays out before they move on. For seasoned engineers, who hold bigger grants and deeper domain knowledge, Waymo still wants the retention pull of a four-year schedule. Waymo isn’t alone. Across tech, equity timelines are turning into a tuning knob: - Lyft issues single-year new-hire grants, then refreshes annually. - Uber front-loads at 35 / 30 / 20 / 15. - NVIDIA moved to 40 / 30 / 20 / 10. - Pinterest compresses to roughly 50 / 33 / 17 over three years. - A handful of high-growth startups grant two-year blocks by default. What’s driving the shift? 1. Pay-for-performance. Shorter cycles let companies re-price grants and reward top performers without being locked into four-year promises. 2. Dilution math. Fewer unvested shares lower equity overhang, especially valuable when stock prices spike. 3. Flexibility. Refreshers become the main retention lever, not year-three and year-four vest cliffs. 4. Talent liquidity. For early-career hires, faster access to equity can feel more tangible than a promise that’s years away. If you’re negotiating, remember that earlier liquidity doesn’t always mean bigger upside. Always compare total grant size, ask how refreshers are determined, and run the tax math on concentrated vesting events. Have you seen a two-year or front-loaded schedule in your offer recently? Comment below with your thoughts and I’ll send over a vesting report we put together here at Levels.fyi. #vestingschedule

137

8 Comments

The next talk in our deep learning performance tutorial features Amir Gholami from UC Berkeley Electrical Engineering & Computer Sciences (EECS) presenting work to accelerate inference of LLMs with long context length

58

1 Comment

NVIDIA Research just published an intersting paper stating that SLMs aren't just "budget alternatives" to LLMs, but they're often superior for real-world applications. Their analysis of popular agents like MetaGPT, Open Operator, and Cradle shows that 40-70% of current LLM calls could be effectively replaced by specialized SLMs. Meanwhile, Andrej Karpathy envisions an even more radical future: a small "cognitive core" model that runs locally by default, with always-on capabilities, multimodal input/output, and the ability to dial up reasoning power when needed while delegating complex tasks to cloud-based oracles. The convergence is clear: we're moving from monolithic, cloud-dependent LLMs toward modular, efficient architectures. SLMs offer 10-30x better cost efficiency, lower latency, and can be fine-tuned faster and at a reasonable cost. For agentic systems performing repetitive and specialized task are better handled with careful context engineering and specialized SLMs. The future isn't about choosing between big and small models, but it's about intelligent orchestration of both. Paper: https://lnkd.in/gZAgH4Y6 Andrej Karpathy's X post: https://lnkd.in/g23777Mc

114

2 Comments

Faster & Smarter LLM Inference for Long Sequences with FlexPrefill Tackling the quadratic bottleneck in the LLM prefill phase is a critical challenge for deployed LLMs, especially as context windows grow. Fixed sparse attention patterns offer a partial solution but lack the adaptability for diverse, real-world inputs. FlexPrefill is a novel sparse pre-filling mechanism that dynamically adapts its attention strategy in real-time. Key innovations: - Query-Aware Pattern Determination: FlexPrefill analyzes the input (via Jensen-Shannon divergence) to intelligently switch between complex, query-specific attention patterns and efficient, predefined ones. - Cumulative-Attention Index Selection: It dynamically allocates a computational budget for each attention head, calculating just enough query-key pairs to meet a predefined attention score threshold Results: Up to 9.6x acceleration over FlashAttention on very long contexts. Superior throughput and accuracy compared to prior methods like MInference and StreamingLLM on benchmarks like RULER and InfiniteBench. For ML engineers working on LLM optimization and deployment, this means faster inference without the typical performance trade-offs. Check out the paper below ⬇️

37

4 Comments

As you start investigating performance of different algorithms/models on different GPUs having an intuition on how Memory is accessed by threads (and groups of threads) can help you to gain a deeper appreciation of lower-level optimizations. This becomes even more complicated as you do multi-gpus and multi-node (groups of multiple GPUs). Read more: https://lnkd.in/eN9nteYx #deeplearning #machinelearning

54

2 Comments

Doing more of what I love recently: timbre modelling of the Tonbak. The Tonbak (or Zarb) is an Iranian goblet drum, widely regarded as the principal percussion instrument of Persian music. I’ll spare you the real-time timbre transfer explanation, since you’re probably already familiar with the concept. If not, just google RAVE and DDSP, read a bit of the material, and try training something yourself. What interests me here is twofold. First, can we define certain input sound characteristics that would deliberately target a specific latent trajectory, ultimately producing a plausible and musically useful output? In other words, what if we could design, or even synthesize, input signals that, when triggered and use for inferring these models, consistently generate desirable sonic qualities? The other one is complements the former, how far could we take this algorithmically by employing complex pre/post processing DSPs. Anyways, I’ll leave you with this documentation snippet for your to hear some yourself. It feels nice to revisit this project. #computermusic #nueralsynthesis #aimusic

103

9 Comments