|
@@ -61,7 +61,7 @@ variety of hardware - locally and in the cloud.
|
|
|
- Plain C/C++ implementation without any dependencies
|
|
- Plain C/C++ implementation without any dependencies
|
|
|
- Apple silicon is a first-class citizen - optimized via ARM NEON, Accelerate and Metal frameworks
|
|
- Apple silicon is a first-class citizen - optimized via ARM NEON, Accelerate and Metal frameworks
|
|
|
- AVX, AVX2 and AVX512 support for x86 architectures
|
|
- AVX, AVX2 and AVX512 support for x86 architectures
|
|
|
-- 2-bit, 3-bit, 4-bit, 5-bit, 6-bit, and 8-bit integer quantization for faster inference and reduced memory use
|
|
|
|
|
|
|
+- 1.5-bit, 2-bit, 3-bit, 4-bit, 5-bit, 6-bit, and 8-bit integer quantization for faster inference and reduced memory use
|
|
|
- Custom CUDA kernels for running LLMs on NVIDIA GPUs (support for AMD GPUs via HIP)
|
|
- Custom CUDA kernels for running LLMs on NVIDIA GPUs (support for AMD GPUs via HIP)
|
|
|
- Vulkan, SYCL, and (partial) OpenCL backend support
|
|
- Vulkan, SYCL, and (partial) OpenCL backend support
|
|
|
- CPU+GPU hybrid inference to partially accelerate models larger than the total VRAM capacity
|
|
- CPU+GPU hybrid inference to partially accelerate models larger than the total VRAM capacity
|
Mirko185