Ask HN: Will something replace transformers and GPUs in the next 10 years?
I don't know why but I have the feeling that a huge amount of money (NVIDIA worth trillions, billions dumped into LLM-based companies) is supported by a single idea that might eventually be replaced by a completely different paradigm or by a new approach to LLMs that could make all of this worth nothing in a matter of months.
To be clear, I’m not saying this isn’t a super impactful idea and I actually think it's a revolution. Since Attention Is All You Need came out in 2017, everything has changed fast but it’s just that I also feel that another breakthrough from research could appear in a similar way.
I think what is more likely to happen is that how 'transformers' work may change. Or a new way of working might be introduced.
Transformer models are already reaching peak of their capabilities vs the cost to run them. All I see currently is basically the 'burn' phase, which is important to get to the 'sustain' phase.
With new stuff coming literally every week, I believe it is not a long time till either transformers get replaced, or some new version of them is introduced which is far more efficient.
I know nothing about this industry, but I imagine that with quantum bits (QuBits), the need for powerful GPUs will diminish, and power requirements will be significantly lower.
Alternatively, one can imagine that GPUs will integrate Qubits.