|
|
@@ -3,7 +3,7 @@
|
|
|
- [Background](#background)
|
|
|
- [News](#news)
|
|
|
- [OS](#os)
|
|
|
-- [Supported Devices](#supported-devices)
|
|
|
+- [Hardware](#hardware)
|
|
|
- [Docker](#docker)
|
|
|
- [Linux](#linux)
|
|
|
- [Windows](#windows)
|
|
|
@@ -24,19 +24,20 @@
|
|
|
- **Nvidia & AMD Plugins**: These are plugins extending oneAPI's DPCPP support to SYCL on Nvidia and AMD GPU targets.
|
|
|
|
|
|
### Llama.cpp + SYCL
|
|
|
-This SYCL "backend" follows the same design found in other llama.cpp BLAS-based paths such as *OpenBLAS, cuBLAS, CLBlast etc..*. The oneAPI's [SYCLomatic](https://github.com/oneapi-src/SYCLomatic) open-source migration tool (Commercial release [Intel® DPC++ Compatibility Tool](https://www.intel.com/content/www/us/en/developer/tools/oneapi/dpc-compatibility-tool.html)) was used for this purpose.
|
|
|
|
|
|
-The llama.cpp SYCL backend supports:
|
|
|
-- Intel GPUs.
|
|
|
-- Nvidia GPUs.
|
|
|
+The llama.cpp SYCL backend is designed to support **Intel GPU** firstly. Based on the cross-platform feature of SYCL, it could support other vendor GPUs: Nvidia GPU (*AMD GPU coming*).
|
|
|
|
|
|
-*Upcoming support: AMD GPUs*.
|
|
|
+When targeting **Intel CPU**, it is recommended to use llama.cpp for [Intel oneMKL](README.md#intel-onemkl) backend.
|
|
|
|
|
|
-When targetting **Intel CPUs**, it is recommended to use llama.cpp for [x86_64](README.md#intel-onemkl) approach.
|
|
|
+It has the similar design of other llama.cpp BLAS-based paths such as *OpenBLAS, cuBLAS, CLBlast etc..*. In beginning work, the oneAPI's [SYCLomatic](https://github.com/oneapi-src/SYCLomatic) open-source migration tool (Commercial release [Intel® DPC++ Compatibility Tool](https://www.intel.com/content/www/us/en/developer/tools/oneapi/dpc-compatibility-tool.html)) was used for this purpose.
|
|
|
|
|
|
## News
|
|
|
|
|
|
+- 2024.4
|
|
|
+ - Support data types: GGML_TYPE_IQ4_NL, GGML_TYPE_IQ4_XS, GGML_TYPE_IQ3_XXS, GGML_TYPE_IQ3_S, GGML_TYPE_IQ2_XXS, GGML_TYPE_IQ2_XS, GGML_TYPE_IQ2_S, GGML_TYPE_IQ1_S, GGML_TYPE_IQ1_M.
|
|
|
+
|
|
|
- 2024.3
|
|
|
+ - Release binary files of Windows.
|
|
|
- A blog is published: **Run LLM on all Intel GPUs Using llama.cpp**: [intel.com](https://www.intel.com/content/www/us/en/developer/articles/technical/run-llm-on-all-gpus-using-llama-cpp-artical.html) or [medium.com](https://medium.com/@jianyu_neo/run-llm-on-all-intel-gpus-using-llama-cpp-fd2e2dcbd9bd).
|
|
|
- New base line is ready: [tag b2437](https://github.com/ggerganov/llama.cpp/tree/b2437).
|
|
|
- Support multiple cards: **--split-mode**: [none|layer]; not support [row], it's on developing.
|
|
|
@@ -59,16 +60,11 @@ When targetting **Intel CPUs**, it is recommended to use llama.cpp for [x86_64]
|
|
|
|Windows|Support|Windows 11|
|
|
|
|
|
|
|
|
|
-## Supported devices
|
|
|
-
|
|
|
-### Intel GPUs
|
|
|
+## Hardware
|
|
|
|
|
|
-The oneAPI Math Kernel Library, which the oneAPI base-toolkit includes, supports intel GPUs. In order to make it "visible", simply run the following:
|
|
|
-```sh
|
|
|
-source /opt/intel/oneapi/setvars.sh
|
|
|
-```
|
|
|
+### Intel GPU
|
|
|
|
|
|
-- **Tested devices**
|
|
|
+**Verified devices**
|
|
|
|
|
|
|Intel GPU| Status | Verified Model|
|
|
|
|-|-|-|
|
|
|
@@ -80,16 +76,18 @@ source /opt/intel/oneapi/setvars.sh
|
|
|
|
|
|
*Notes:*
|
|
|
|
|
|
-- Device memory can be a limitation when running a large model on an intel GPU. The loaded model size, *`llm_load_tensors: buffer_size`*, is displayed in the log when running `./bin/main`.
|
|
|
+- **Memory**
|
|
|
+ - The device memory is a limitation when running a large model. The loaded model size, *`llm_load_tensors: buffer_size`*, is displayed in the log when running `./bin/main`.
|
|
|
|
|
|
-- Please make sure the GPU shared memory from the host is large enough to account for the model's size. For e.g. the *llama-2-7b.Q4_0* requires at least 8.0GB for integrated GPUs and 4.0GB for discrete GPUs.
|
|
|
+ - Please make sure the GPU shared memory from the host is large enough to account for the model's size. For e.g. the *llama-2-7b.Q4_0* requires at least 8.0GB for integrated GPU and 4.0GB for discrete GPU.
|
|
|
|
|
|
-- If the iGPU has less than 80 EUs *(Execution Unit)*, the inference speed will likely be too slow for practical use.
|
|
|
+- **Execution Unit (EU)**
|
|
|
+ - If the iGPU has less than 80 EUs, the inference speed will likely be too slow for practical use.
|
|
|
|
|
|
-### Nvidia GPUs
|
|
|
-The BLAS acceleration on Nvidia GPUs through oneAPI can be obtained using the Nvidia plugins for oneAPI and the cuBLAS backend of the upstream oneMKL library. Details and instructions on how to setup the runtime and library can be found in [this section](#i-setup-environment)
|
|
|
+### Nvidia GPU
|
|
|
+The BLAS acceleration on Nvidia GPU through oneAPI can be obtained using the Nvidia plugins for oneAPI and the cuBLAS backend of the upstream oneMKL library. Details and instructions on how to setup the runtime and library can be found in [this section](#i-setup-environment)
|
|
|
|
|
|
-- **Tested devices**
|
|
|
+**Verified devices**
|
|
|
|
|
|
|Nvidia GPU| Status | Verified Model|
|
|
|
|-|-|-|
|
Neo Zhang Jianyu