That's completely different. That's like saying you want to compare the Nvidia 5090 GPU to the latest Call of Duty.
3 hours ago [-]
cubefox 19 hours ago [-]
You are right, people who downvoted you are just ignorant.
cubefox 1 days ago [-]
Mamba-3 is an architecture while diffusion is, I believe, a type of objective. So these are not mutually exclusive and therefore not comparable.
gyrovagueGeist 22 hours ago [-]
Not wrong, but I think it's more accurate to say:
Mamba is an architecture for the middle layers of the network (the trunk) which assumes decoding takes place through an autoregressive sequence (popping out tokens in order). This is the SSM they talk about.
Diffusion is an alternative to the autoregressive approach where decoding takes place through iterative refinement on a batch of tokens (instead of one at a time processing and locking each one in only looking forward). This can require different architectures for the trunk, the output heads, and modifications to the objective to make the whole thing trainable. Could mamba like ideas be useful in diffusion networks...maybe but it's a different problem setup.
joefourier 18 hours ago [-]
Mamba doesn't assume auto-regressive decoding, and you can use absolutely use it for diffusion, or pretty much any other common objective. Same with a conventional transformer. For a discrete diffusion language model, the output head is essentially the same as an autoregressive one. But yes, the training/objective/inference setup is different.
cubefox 19 hours ago [-]
Linear architectures are at least heavily used in image diffusion models. More so in fact than in language models.
nl 19 hours ago [-]
I mean I guess but the diffusion objective and the ability to do simultaneous decode both dictate pretty different architectures in practice.
I'm not sure that I buy their conclusion that more compute during inference is good.
Yes, batch=1 inference is mostly memory bandwidth bound, not GPU compute bound. But no provider does batch=1 inference. Everyone groups all the requests into a batch, and the GPU computes them together.
With a fused kernel, that means the GPU streams the tensors from VRAM, and does a bunch of compute on different conversations in the batch, at the same time.
If they increase the amount of compute required per token, that just reduces the maximum batch size a GPU can handle. In practice, yes this does mean each GPU can serve less users. Providers aren't leaving GPU cores idle normally during inference.
zozbot234 22 hours ago [-]
> Everyone groups all the requests into a batch, and the GPU computes them together.
You're only saving on fetching read-only parameters, and not even on that if you're using MoE models where each inference in the batch might require a different expert (unless you rearrange batches so that sharing experts becomes more likely, but that's difficult since experts change per-token or even per-layer). Everything else - KV-cache, activations - gets multiplied by your batch size. You scale both compute and memory pressure by largely the same amount. Yes, GPUs are great at hiding memory fetch latency, but that applies also to n=1 inference.
jychang 20 hours ago [-]
Well, the actual inference providers put each expert on its own single GPU. Deepseek explicitly does this.
Read-only parameters is also usually the majority of space. Deepseek is 700GB of params. Meanwhile kv cache is small (Deepseek is about 7GB at max context) and ssm/conv1d cache is even smaller- IIRC Qwen 3.5 is 146MB per token regardless of context size. Not sure about how Mamba-3 works, but I suspect read-only parameters are still a significant amount of memory bandwidth.
I guess the question isn't whether compute is 1:1 with memory, but rather if you run out of compute before you run out of vram adding more users.
zozbot234 18 hours ago [-]
> Well, the actual inference providers put each expert on its own single GPU.
Experts are usually chosen on a per-layer basis, not just by token, so I'd think this requires having lots of GPU's to make it worthwhile. You could do it with a single physical GPU by switching expert-layer mixes in a round-robin fashion after the batch for any single expert-layer mix is completed (essentially a refined version of expert offloading). But still, not easy.
jychang 2 hours ago [-]
Correct, but note that's exactly what inference providers do.
> The minimum deployment unit of the decoding stage consists of 40 nodes with 320 GPUs. The attention part employs TP4 with SP, combined with DP80, while the MoE part uses EP320.
EP320 means expert parallelism, each on 320 GPUs.
btown 19 hours ago [-]
Throughput is indeed king for the standard-tier mindshare-capture play. But there are many who would pay multiple times the current cost for agentic systems for engineers and executives, if it meant a meaningful reduction in latency. The economics could work extremely well.
Havoc 19 hours ago [-]
Why would execs need latency?
I can see it for engineering - coding with slow ai is painful
metanonsense 19 hours ago [-]
Impatient execs can also be painful. EDIT: Writing this while I am waiting for Codex to complete, so I may enjoy slow AI more than the usual developer ;-)
btown 13 hours ago [-]
The economic effect of latency is measured not by the incremental productivity itself, but by the combined economic downforce of thousands of resulting HN and Reddit comments :)
notnullorvoid 18 hours ago [-]
Focusing on needs of providers isn't a very good long term strategy if you believe compute will eventually move to self hosted and on premises solutions where large batch sizes aren't needed.
jychang 2 hours ago [-]
That's a foolish take.
That's like gamers thinking most of Nvidia's revenue coming from gaming GPUs, so Nvidia should prioritize gamers.
Inference is ruled by inference providers, not local. Local inference is a rounding error, and will remain as such unless there is economic incentive otherwise.
sroussey 17 hours ago [-]
Local has a batch size of 1. If you are already memory bound then you leave compute on the table. Why not use it?
Not sure they target local though…
yorwba 22 hours ago [-]
Their latency measurements comparing Mamba-2 and Mamba-3 are done with a batch size of 128. It doesn't seem like Mamba-2 was compute-bound even at that batch size.
jychang 20 hours ago [-]
Well, Deepseek batch sizes are something like 8192, so 128 isn't much.
Is there a reason we don’t switch halfway through? ie start with a classic LLM and switch to something linear like mamba as context grows
lambda 18 hours ago [-]
Because something linear like Mamba doesn't perform as well; so you'd have a performance cliff, where suddenly the model would get more dumb and forget a lot of what was going on.
Instead, you can get benefits from both by doing both in parallel. This can let you reduce the size of the O(n^2) attention mechanism, so while it's still quadratic, it reduces the constant quite a bit while still retaining a lot of performance, as the linear context mechanism can work for the tasks its well suited for while allowing attention to play to its strengths.
The recent Nemotron 3 Nano and Super models from NVIDIA are hybrid architectures this way, with most of their context layers as Mamba while retaining enough attention to continue to be competitive on the more complex tasks that require the quadratic attention.
They did do that, 2 years ago. The problems are that 1) mamba makes accuracy worse as context size grows, 2) Nvidia GPUs are designed for transformers, and 3) all the software out there is also designed for transformers. It's still useful in some applications but it doesn't beat regular transformers if you have the gear
energy123 18 hours ago [-]
Probably best achieved by model routing, either an indirection behind the chat UI or an API user does it themselves by calling a different API for long context queries.
mountainriver 18 hours ago [-]
We kinda do do this with hybrid mamba transformers
cubefox 18 hours ago [-]
Linear time complexity models are bad at in-context retrieval, which limits their performance on various tasks, so a pure linear model isn't currently feasible anyway, at least for language models. Instead they recommend mixing linear and attention layers. Presumably this mostly solves the performance problem (at least n benchmarks), but it also means the mixed architecture is no longer linear. It will still be faster and less RAM hungry in long context than a pure transformer though.
roger_ 14 hours ago [-]
Can anyone explain why Mamba models start with a continuous time SSM (and discretize) vs discrete time?
I know the step isn’t fixed, also not sure why that’s important. Is that the only reason? There also seems to be a parameterization advantage too with the continuous formulation.
jeffhwang 16 hours ago [-]
I'm glad I clicked through bc I thought the article was about Mamba, the package manager I associate with Python (similar to conda).
This is really promising. Are they now going to scale this up to hundreds of billions of parameters? Why stop at 1.5B if they found a potentially SOTA architecture?
snek_case 13 hours ago [-]
Probably constrained by training resources. It's much easier to experiment with a smaller architecture. You may need many training runs to figure out hyperparameters for example. If each run needs multiple GPUs for a week the cost adds up quickly. I think it makes a lot of sense to start small.
manlymuppet 14 hours ago [-]
I'm looking forward to the fifth iteration of this model.
diablevv 18 hours ago [-]
[dead]
daliliu 19 hours ago [-]
[dead]
robofanatic 1 days ago [-]
> Mamba-3 is a new state space model (SSM) designed with inference efficiency as the primary goal — a departure from Mamba-2, which optimized for training speed. The key upgrades are a more expressive recurrence formula, complex-valued state tracking, and a MIMO (multi-input, multi-output) variant that boosts accuracy without slowing down decoding.
Why can’t they simply say -
Mamba-3 focuses on being faster and more efficient when making predictions, rather than just being fast to train like Mamba-2.
esquire_900 1 days ago [-]
This is sort of what their first sentence states? Except your line implies that they are fast in training and inference, they imply they are focusing on inference and are dropping training speed for it.
It's a nice opening as it is imo
cubefox 24 hours ago [-]
They don't say anything about dropping training speed.
estearum 21 hours ago [-]
> a departure from Mamba-2, which optimized for training speed.
?
cubefox 19 hours ago [-]
Yes? Mamba-2 optimized for training speed compared to Mamba-1. Mamba-3 adds optimization for inference. These are pretty much version numbers.
i000 14 hours ago [-]
Agreed. What you wrote was probably the input, what we see is the LLM output with the directive to "make us sound smart, put gratuitous em-dash"
E-Reverance 1 days ago [-]
The first sentence basically does though, no?
robofanatic 1 days ago [-]
Of course my only objection was the language. LLMs are now old enough to leave the jargon behind and talk in simple easy to understand terms.
oersted 24 hours ago [-]
I’d argue the opposite, the terminology is fairly mainstream by now and “inference” has a much more specific sense than “making predictions”.
mufasachan 1 days ago [-]
The blog is technical, technical terms in the TL;DR seems relevant to me.
arendtio 1 days ago [-]
I don't get the downvotes, as I had trouble understanding the intro as well. It seems it was written for a very specific audience.
qeternity 24 hours ago [-]
Yes, it is written for a specific audience.
That is not a reason for snark.
As other commenters have noted, it’s well written.
magicalhippo 23 hours ago [-]
> I don't get the downvotes
Because the blog post is a technical one and the intro contains very common jargon, and the proposed alternative was wrong.
renewiltord 20 hours ago [-]
Found the guy who made the Windows error messages say “Your computer did an oopsie :(“ instead of including any useful information.
camillomiller 1 days ago [-]
I don’t know why you’re being downvoted. As a longtime editor your version is immensely better.
Looks like the original was probably not human-written.
stavros 18 hours ago [-]
Why would the simpler version be better for a technical audience?
Mamba is an architecture for the middle layers of the network (the trunk) which assumes decoding takes place through an autoregressive sequence (popping out tokens in order). This is the SSM they talk about.
Diffusion is an alternative to the autoregressive approach where decoding takes place through iterative refinement on a batch of tokens (instead of one at a time processing and locking each one in only looking forward). This can require different architectures for the trunk, the output heads, and modifications to the objective to make the whole thing trainable. Could mamba like ideas be useful in diffusion networks...maybe but it's a different problem setup.
Yes, batch=1 inference is mostly memory bandwidth bound, not GPU compute bound. But no provider does batch=1 inference. Everyone groups all the requests into a batch, and the GPU computes them together.
With a fused kernel, that means the GPU streams the tensors from VRAM, and does a bunch of compute on different conversations in the batch, at the same time.
If they increase the amount of compute required per token, that just reduces the maximum batch size a GPU can handle. In practice, yes this does mean each GPU can serve less users. Providers aren't leaving GPU cores idle normally during inference.
You're only saving on fetching read-only parameters, and not even on that if you're using MoE models where each inference in the batch might require a different expert (unless you rearrange batches so that sharing experts becomes more likely, but that's difficult since experts change per-token or even per-layer). Everything else - KV-cache, activations - gets multiplied by your batch size. You scale both compute and memory pressure by largely the same amount. Yes, GPUs are great at hiding memory fetch latency, but that applies also to n=1 inference.
Read-only parameters is also usually the majority of space. Deepseek is 700GB of params. Meanwhile kv cache is small (Deepseek is about 7GB at max context) and ssm/conv1d cache is even smaller- IIRC Qwen 3.5 is 146MB per token regardless of context size. Not sure about how Mamba-3 works, but I suspect read-only parameters are still a significant amount of memory bandwidth.
I guess the question isn't whether compute is 1:1 with memory, but rather if you run out of compute before you run out of vram adding more users.
Experts are usually chosen on a per-layer basis, not just by token, so I'd think this requires having lots of GPU's to make it worthwhile. You could do it with a single physical GPU by switching expert-layer mixes in a round-robin fashion after the batch for any single expert-layer mix is completed (essentially a refined version of expert offloading). But still, not easy.
https://arxiv.org/pdf/2412.19437
> The minimum deployment unit of the decoding stage consists of 40 nodes with 320 GPUs. The attention part employs TP4 with SP, combined with DP80, while the MoE part uses EP320.
EP320 means expert parallelism, each on 320 GPUs.
I can see it for engineering - coding with slow ai is painful
That's like gamers thinking most of Nvidia's revenue coming from gaming GPUs, so Nvidia should prioritize gamers.
Inference is ruled by inference providers, not local. Local inference is a rounding error, and will remain as such unless there is economic incentive otherwise.
Not sure they target local though…
https://arxiv.org/html/2412.19437v1 "the batch size per expert is relatively small (usually within 256 tokens)"
Instead, you can get benefits from both by doing both in parallel. This can let you reduce the size of the O(n^2) attention mechanism, so while it's still quadratic, it reduces the constant quite a bit while still retaining a lot of performance, as the linear context mechanism can work for the tasks its well suited for while allowing attention to play to its strengths.
The recent Nemotron 3 Nano and Super models from NVIDIA are hybrid architectures this way, with most of their context layers as Mamba while retaining enough attention to continue to be competitive on the more complex tasks that require the quadratic attention.
See https://magazine.sebastianraschka.com/i/168650848/18-nemotro... for some discussion on this architecture
I know the step isn’t fixed, also not sure why that’s important. Is that the only reason? There also seems to be a parameterization advantage too with the continuous formulation.
https://github.com/mamba-org/mamba
https://arxiv.org/abs/2603.15569
Why can’t they simply say -
Mamba-3 focuses on being faster and more efficient when making predictions, rather than just being fast to train like Mamba-2.
It's a nice opening as it is imo
?
That is not a reason for snark.
As other commenters have noted, it’s well written.
Because the blog post is a technical one and the intro contains very common jargon, and the proposed alternative was wrong.