I'll stay on 20B for now. The difference to 7B is clear. So far I haven't experienced any strange endless loops. If I'm bored at some point, I can do a test with a 70B model and take several minutes for the response times. You still get an impression of the quality of the answers. At the moment I'm ...

I've been reading a lot lately to find out what's technically possible. Laptops always seem to have the GPU as a bottle neck. If the laptop has Thunderbolt, an eGPU can be considered: https://egpu.io/best-egpu-buyers-guide/ Unfortunately not cheap. I'm already thinking about running a second graphic...

I now have a Linux computer with a MSI GeForce RTX 4060 Ti Ventus 2X Black 16G installed. I can even load mxlewd-l2-20b.Q5_K_M.gguf now. The response times can be neglected. If I understand this correctly, only the memory of the graphics card is important. I have also written a new extension with wh...

I finished my first attempt at an extension:
https://github.com/H2SO4H2SO4/joi-extension
Maybe someone has use for it.

Thank you for your post. After a week I got the stuff running on my machine (8 i7 cores, 32GB RAM, GTX 1660 Ti). The response times for zephyr-7b-alpha.Q5_K_M.gguf are under 30 seconds. I use Silero TTS and Vosk STT to have both hands free :w00t: Unfortunately, converting numbers into words doesn't ...