Question for the local AI nerds
Loading reactions…
Harley Richardson
Loading reactions…
greg
That's what I thought about the responses. and yes, lm studio is flawed in this way. I was actually thinking about a python script to read out the response upon completion for lm studio. I'm also considering Anything LLM or ollama. I could create my own gui for ollama, or use open web ui.
Loading reactions…
Harley Richardson
You wouldn't even need to create a GUI for ollama, because it also serves an open AI endpoint, so existing tools can plug straight into it. Ollama has its own API yes, but it also serves an open AI one at /v1. Lm studio offers things that ollama doesn't easily offer, such as quantising your KV cache which is useful for longer contexts.
Loading reactions…
hamidreza
Loading reactions…
Nick Stockton
I use VOLlama from https://github.com/chigkim/VOLlama with both local and remote Open AI compatible endpoints. It's an accessible frontend written in Python with WXPython for the GUI and Accessible Output2 for screenreader output. It works pretty well for me.
Loading reactions…