The main goal is to run the model using 4-bit quantization on a MacBook
Plain C/C++ implementation without dependencies
Apple silicon first-class citizen - optimized via ARM NEON and Accelerate framework
AVX2 support for x86 architectures
Mixed F16 / F32 precision
4-bit quantization support
Runs on the CPU
I have no idea if it works correctly. Please do not make conclusions about the models based on the results from this implementation. For all I know, it can be completely wrong. This project is for educational purposes.
[hide=guest]
https://github.com/ggerganov/llama.cpp
[/hide]
Plain C/C++ implementation without dependencies
Apple silicon first-class citizen - optimized via ARM NEON and Accelerate framework
AVX2 support for x86 architectures
Mixed F16 / F32 precision
4-bit quantization support
Runs on the CPU
I have no idea if it works correctly. Please do not make conclusions about the models based on the results from this implementation. For all I know, it can be completely wrong. This project is for educational purposes.
[hide=guest]
https://github.com/ggerganov/llama.cpp
[/hide]
