Everyone's freaking out about the HBM shortage but SK Hynix and SanDisk just quietly launched a whole new type of memory

HBM is the expensive fast memory that's in short supply and driving the whole AI hardware trade right now. Last week SK Hynix and SanDisk dropped a standard for a new thing called High Bandwidth Flash that basically slots in underneath it, almost as fast but with way more capacity for cheaper. Google's already on board. It's built for AI inference, not training. training is the one-time cost of building the model; inference is every time someone actually uses it, which is forever. It feels like the whole industry is shifting from pay once to build it to pay forever to run it, and the memory everyone's crowded into might be the wrong one for where this is actually going. Please tell me I'm not overthinking this one.

9 comments held Reddit says 0 on reddit ↗

u/bizaromax 1 16 hours ago

At some point we will not be saying there is enough training done and tech will change again and again.
u/Novel-Lifeguard6491 OP 1 16 hours ago

I think it’s not so much that training stops, but that once something’s actually deployed the inference volume grows so much faster that the balance shifts
u/zzulus 1 15 hours ago

Well, yes, but there's also online and offline training
u/chanc2 1 15 hours ago

I think this industry is still rapidly evolving with many different compute and memory architectures in development. There is a bifurcation between training and inference for both compute and memory.
u/SaltsMoon 1 15 hours ago

I'd think of HBF less as an HBM replacement and more as another layer in the memory stack. If inference is mostly about reading weights, having cheaper, nonvolatile memory closer to the accelerator could take some pressure off expensive DRAM.
u/myothercarisayoshi 1 14 hours ago

Slop slop slop
u/hidetoshiko 1 14 hours ago

Bottom line up front for you non-techies: HBF is not a replacement for HBM. If HBF succeeds, in a thriving AI pervasive future, they will basically coexist with HBM with a different role to play within the ecosystem. It's NOT an either-or situation. Having said that, HBF is one obvious solution to address the memory wall problem, but it's not the only one. There are competing ideas as well, like CXL.
u/Friendly-Profit-8590 1 13 hours ago

So we don’t want to buy SNDK for HBM we want to buy SNDK for High Bandwidth Flash?
u/Probutkickerz 1 12 hours ago

For most people including myself memory probably goes in the too hard pile if you dont actually know the nuances of the technology. I get that HBM and DRAM are really hard to make but I wouldn’t be that surprised if another company made something that completely bypass them, but I guess its probably going to be the HBM makers themselves.