Rasmus Aagaard @rasgaard.com · Feb 20 LLM cloud inference dominates usage, but should it? Local models and accelerators have improved massively over recent years. Perfect routing to best local model "reduce energy consumption by 80.4%, compute by 77.3%, and cost by 73.8% versus cloud-only deployment" arxiv.org/pdf/2511.07885
9 likes 3 replies
Replies
It will be interesting to see whether local models overcome the historical bottleneck of DIY tech barrier to entry, or whether AI permeates the Internet of Things and cloud remains dominant because of consumer lethargy. I am hopeful, but not overly optimistic.
Incredible headline numbers but: “We study single-query inference (batch size = 1), to (1) isolate intrinsic model-accelerator efficiency from system-level serving optimizations…” at batch size 1, something like a B200 is burning power starving for work. Cloud inference isn't running like that!
Men det kræver stadig en temmelig kraftig enhed derhjemme. Jeg leger med at lave et transkriberingsprogram lokalt på en bærbar med en rtx studio 4070 8GB og den pust og stønner, når den skal processere først Whisper og Pyannotate og efterfølgende fx llama3.1 til renskrivning af transkriptionen.