Open source , zoals Open Ai was. Ondertussen kunnen/zijn zowat alle andere, ook onderzoekscentra in België en bv Meta met Lama aan het bekijken / evalueren hoe ze dit kunnen implementeren in hun modellen en/of dit daadwerkelijk zo efficient is, zijn we nu beter af met de Amerikaanse overheid ?
Hieronder een bloemlezing wat het efficiënter zou maken :
Dit komt uit een artikel wat vorige vrijdag was gepubliceerd en vermoedelijk de trigger was voor de crash maandag bij o.a. Nvidia
https://youtubetranscript...5_the_short_case_for_nvda
A major innovation is their sophisticated mixed-precision training framework that lets them use 8-bit floating point numbers (FP8) throughout the entire training process. Most Western AI labs train using "full precision" 32-bit numbers (this basically specifies the number of gradations possible in describing the output of an artificial neuron; 8 bits in FP8 lets you store a much wider range of numbers than you might expect— it's not just limited to 256 different equal-sized magnitudes like you'd get with regular integers, but instead uses clever math tricks to store both very small and very large numbers— though naturally with less precision than you'd get with 32 bits.) The main tradeoff is that while FP32 can store numbers with incredible precision across an enormous range, FP8 sacrifices some of that precision to save memory and boost performance, while still maintaining enough accuracy for many AI workloads.
DeepSeek cracked this problem by developing a clever system that breaks numbers into small tiles for activations and blocks for weights, and strategically uses high-precision calculations at key points in the network. Unlike other labs that train in high precision and then compress later (losing some quality in the process), DeepSeek's native FP8 approach means they get the massive memory savings without compromising performance. When you're training across thousands of GPUs, this dramatic reduction in memory requirements per GPU translates into needing far fewer GPUs overall.
Another major breakthrough is their multi-token prediction system. Most Transformer based LLM models do inference by predicting the next token— one token at a time. DeepSeek figured out how to predict multiple tokens while maintaining the quality you'd get from single-token prediction. Their approach achieves about 85-90% accuracy on these additional token predictions, which effectively doubles inference speed without sacrificing much quality. The clever part is they maintain the complete causal chain of predictions, so the model isn't just guessing— it's making structured, contextual predictions.
One of their most innovative developments is what they call Multi-head Latent Attention (MLA). This is a breakthrough in how they handle what are called the Key-Value indices, which are basically how individual tokens are represented in the attention mechanism within the Transformer architecture. Although this is getting a bit too advanced in technical terms, suffice it to say that these KV indices are some of the major uses of VRAM during the training and inference process, and part of the reason why you need to use thousands of GPUs at the same time to train these models— each GPU has a maximum of 96 gb of VRAM, and these indices eat that memory up for breakfast.
Their MLA system finds a way to store a compressed version of these indices that captures the essential information while using far less memory. The brilliant part is this compression is built directly into how the model learns— it's not some separate step they need to do, it's built directly into the end-to-end training pipeline. This means that the entire mechanism is "differentiable" and able to be trained directly using the standard optimizers. All this stuff works because these models are ultimately finding much lower-dimensional representations of the underlying data than the so-called "ambient dimensions". So it's wasteful to store the full KV indices, even though that is basically what everyone else does.
Not only do you end up wasting tons of space by storing way more numbers than you need, which gives a massive boost to the training memory footprint and efficiency (again, slashing the number of GPUs you need to train a world class model), but it can actually end up improving model quality because it can act like a "regularizer," forcing the model to pay attention to the truly important stuff instead of using the wasted capacity to fit to noise in the training data. So not only do you save a ton of memory, but the model might even perform better. At the very least, you don't get a massive hit to performance in exchange for the huge memory savings, which is generally the kind of tradeoff you are faced with in AI training.
They also made major advances in GPU communication efficiency through their DualPipe algorithm and custom communication kernels. This system intelligently overlaps computation and communication, carefully balancing GPU resources between these tasks. They only need about 20 of their GPUs' streaming multiprocessors (SMs) for communication, leaving the rest free for computation. The result is much higher GPU utilization than typical training setups achieve.
"Another very smart thing they did is to use what is known as a Mixture-of-Experts (MOE) Transformer architecture, but with key innovations around load balancing. As you might know, the size or capacity of an AI model is often measured in terms of the number of parameters the model contains. A parameter is just a number that stores some attribute of the model; either the "weight" or importance a particular artificial neuron has relative to another one, or the importance of a particular token depending on its context (in the "attention mechanism"), etc."
:strip_exif()/i/2007212560.jpeg?f=imagegallery)
:strip_exif()/i/2007212562.jpeg?f=fpa)
/i/2004653106.png?f=fpa)
:strip_exif()/i/2005545080.jpeg?f=fpa)
/i/2004618396.png?f=fpa)
:strip_icc():strip_exif()/u/882025/crop5e234564386cc_cropped.jpeg?f=community){kind=small}
/u/263454/wie%2520dit%2520leest%2520is%2520gek.png?f=community){kind=small}
:strip_icc():strip_exif()/u/33880/crop5ec3d2d5f2541_cropped.jpeg?f=community){kind=small}
:strip_icc():strip_exif()/u/789283/crop65b0dfe4af024_cropped.jpg?f=community){kind=small}
:strip_exif()/u/32775/fokker-60x60.gif?f=community){kind=small}
/u/321091/crop5eb047be81d0c_cropped.png?f=community){kind=small}
/u/176086/crop5f0823fa5e8d6_cropped.png?f=community){kind=small}
/u/62384/crop61891f444d6e9.png?f=community){kind=small}
/u/99142/crop62758e978b3e3_cropped.png?f=community){kind=small}
:strip_icc():strip_exif()/u/25132/kaypro-mini.jpg?f=community){kind=small}
:strip_icc():strip_exif()/u/328790/crop5db579bad2203_cropped.jpeg?f=community){kind=small}
/u/111174/sachiel-small.png?f=community){kind=small}
/u/217510/crop660db19c1cf7b_cropped.png?f=community){kind=small}
:strip_icc():strip_exif()/u/141475/767.jpg?f=community){kind=small}
/u/5028/assassins-creed-II-iconwarped.png?f=community){kind=small}
/u/16450/icon06.png?f=community){kind=small}
:strip_icc():strip_exif()/u/475476/crop5ee378972b771_cropped.jpeg?f=community){kind=small}
:strip_icc():strip_exif()/u/235816/crop596740581e453_cropped.jpeg?f=community){kind=small}
/u/488564/crop6821228665cf4_cropped.png?f=community){kind=small}
:strip_icc():strip_exif()/u/489983/crop5db33928bbeea_cropped.jpeg?f=community){kind=small}
