I'll stay on 20B for now. The difference to 7B is clear. So far I haven't experienced any strange endless loops.
If I'm bored at some point, I can do a test with a 70B model and take several minutes for the response times.
You still get an impression of the quality of the answers.

At the moment I'm ...

I've been reading a lot lately to find out what's technically possible.
Laptops always seem to have the GPU as a bottle neck. If the laptop has Thunderbolt, an eGPU can be considered:
https://egpu.io/best-egpu-buyers-guide/
Unfortunately not cheap.
I'm already thinking about running a second ...

I now have a Linux computer with a
MSI GeForce RTX 4060 Ti Ventus 2X Black 16G installed.
I can even load mxlewd-l2-20b.Q5_K_M.gguf now.
The response times can be neglected.
If I understand this correctly, only the memory of the graphics card is important.

I have also written a new extension with ...

I finished my first attempt at an extension:
https://github.com/H2SO4H2SO4/joi-extension
Maybe someone has use for it.

Thank you for your post. After a week I got the stuff running on my machine (8 i7 cores, 32GB RAM, GTX 1660 Ti).
The response times for zephyr-7b-alpha.Q5_K_M.gguf are under 30 seconds. I use Silero TTS and Vosk STT to have both hands free :w00t:
Unfortunately, converting numbers into words doesn't ...