If you were running a LLM locally on android through llama.cpp for use as a private personal assistant. What model would you use?
Thanks for any recommendations in advance.
You must log in or register to comment.
It very much depends on your phone hardware, RAM affects how big models can be and CPU affects how fast you’ll get the replies. I’ve successfully ran 4B models on my 8GB RAM phone, but since it’s the usual server and client setup which needs full internet access due to the lack of granular permissions on Android (Even AIO setups needs open ports to connect to itself) I prefer a proper home server. Which, with a cheap GFX card, is indescribably faster and more capable.
I was honestly impressed with the speed and accuracy I was getting with Deepseek, llama, and Gemma on my 1660ti.
$100 used and it was seconds to get responses.
Not sure if a mobile device have that type of processing power lol
maid + VPN to Ollama on your own computer.
Use an Onion service with client authorisation to avoid needing a domain or static IP.
I don’t recommend it. I ran local AI in my phone before (iPhone but same difference), and just asking it stuff makes it warm up to touch. Battery also takes a hit.
It also messes up multitasking features since it uses up most memory which kills background apps. Phones weren’t designed for this.
Best way is to host it in an actual dedicated machine that can be accessed remotely.
You want to run it on the phone itself? I don’t think any phone would be good enough for that. The issue with AI assistants is not just privacy. It’s also the resource consumption (and of course stolen content). It’s so high and only these big companies with huge server farms can do it.
If you just want a voice assistant for simple commands, I’ve heard of an open source local assistant called Dicio. But I don’t think you can talk to it like ChatGPT or something.
I’ve successfully ran small scale LLM’s on my phone, slow but very doable. I run my main AI system on an older, midrange gaming PC. No problems at all.
Dicio is a pre-programmed assistant, which one can talk to if one has speech recognition software installed. It has a preset of tasks it can do, in my experience it’s quite incomparable to how LLM’s work.
I have pocketpal setup on my pixel with graphene os and its pretty awesome. I agree that AI is inherently bad considering the environmental impact and the amount of data that is illegally needed to train AI.
That being said pocketpal is openaoirce and is great. https://github.com/a-ghorbani/pocketpal-ai