Hey, great read as always. Your point about robotics and the open question of model size for on-device vs. low-latency edge datacenter is super insightful. I'm curious, considering advancements like quantization and NPUs, how much further can we push on-device efficiency? This feels like a pivotal point for broad robot deployment, especialy for its energy implications.
1) Models will very likely get much smaller for the same capability level as we move forward. That's been happening very rapidly so there's reason for optimism there. Also, we know we can get human level intelligence using the ~20 W that the human brain uses. Those are strong reasons to think robots will run entirely on device in the long term.
2) However, it seems likely to me that in the short/medium term robots use a split brain approach. Using the human brain as an analogy, the robot's neocortex (the high level reasoning and planning part of the brain) will live in the cloud and the rest (motor control, perception, etc.) will live on device. That mix makes the robot more failsafe and power efficient, while also getting access to the best reasoning abilities as well.
Hey, great read as always. Your point about robotics and the open question of model size for on-device vs. low-latency edge datacenter is super insightful. I'm curious, considering advancements like quantization and NPUs, how much further can we push on-device efficiency? This feels like a pivotal point for broad robot deployment, especialy for its energy implications.
Thanks - two main thoughts on your question.
1) Models will very likely get much smaller for the same capability level as we move forward. That's been happening very rapidly so there's reason for optimism there. Also, we know we can get human level intelligence using the ~20 W that the human brain uses. Those are strong reasons to think robots will run entirely on device in the long term.
2) However, it seems likely to me that in the short/medium term robots use a split brain approach. Using the human brain as an analogy, the robot's neocortex (the high level reasoning and planning part of the brain) will live in the cloud and the rest (motor control, perception, etc.) will live on device. That mix makes the robot more failsafe and power efficient, while also getting access to the best reasoning abilities as well.