Does imatrix require some special support from the software running the LLM to shine over non-imatrix quants?
#2
by
seedmanc
- opened
I'm using Faraday (backyard) and GPT4ALL, do those support imatrixes? In my personal benchmark I couldn't notice the Q5 imatrix do any better than the reqular Q5.
Not really. Any GGUF backend will work. The benefits when you get close to Q6 might be very small, and are likely more noticeable in the Q4 and under range, although Q5 still benefits from it. In practice to notice a direct improvement it will depend in your own usage and needs.
Lewdiculous
changed discussion status to
closed