July Musings
Quick, raw thoughts (but in a slightly different format this month)
It's the last day of July, and I've been contemplating what to write.
Most of my July was spent dabbling in GPU financializing and a deep dive into data, the continued saturation of benchmarks, the appreciation for the sheer human efforts that go into improving a model by 100 basis points, and theories like Kolmogorov Complexity. Many thanks to folks like Jack Morris and Ryan Greene for seeding these thoughts.
RL Environments continue to be hot. But there's been a notable shift in the way startups are approaching this. Applied Compute has expanded into inference in an attempt to expand TAM. TML has released its own models with surprising performance from a smaller v2 model that benefited from additional RL and data mixing. My guess is that RL that is pushing the boundary of intelligence is becoming more and more difficult and expensive to provide due to the time required to craft datasets. The same effort and capital can go into servicing a wider variety of customers that require much less intelligence. Value is the game now, not performance.
More than anything, however, SSI has been top of mind for me. I've never met Ilya. But I have spoken to two people who are close to him. What I've noticed about him is a type of religious fervor towards machine learning. With quotes like "if you believe in deep learning, deep learning will believe in you!" and wording like "you feel [the pieces coming together]" and "it's kind of magical" across his lectures, he speaks more like a mystic than a scientist. I often picture an archbishop, with his colorful ceremonial garb, preaching to a room full of zealots. Only this time, it's Ilya wearing a bird t-shirt and the venue is the Simons Institute.
Ilya's emergence in the public light coincided with the release of Kimi-K3 (or at least that's the narrative I've built in my head). I see this as a Yin and Yang moment. When I spoke to the Moonshot team last year, shortly after the release of K2, what was palpable was their belief in AGI. Zhilin's ability to recruit talent was the direct result of selling his vision towards AGI. This was how they reached 300 employees last July (contrary to recent news that they're currently at 300), consisting largely of first authors of core techniques used across frontier labs like RoPE.
The West has Ilya, the East has Zhilin. Ilya is committed to a cathedral style of building where only SSI's employees can participate. Zhilin is building a bazaar.
Some ponderings on SSI's approach from April 2024
