ソースを参照

llama : skip output reordering for single token batches (#17466)

This commit adds a check to skip the output reordering logic when
n_outputs == 1. With a single output token, the data is trivially
sorted and the reordering code is currently doing unnecessary work
(resetting and rebuilding output_ids to the same values).

The motivation for this change is improved code clarity and avoiding
confusion when debugging. While the performance impact is probably
negligible, this unnecessary work happens on every decode call in
llama-server when processing batches with single-token outputs.
Daniel Bevenius 1 ヶ月 前
コミット
134e6940ca
1 ファイル変更1 行追加1 行削除
  1. 1 1
      src/llama-context.cpp

+ 1 - 1
src/llama-context.cpp

@@ -1248,7 +1248,7 @@ int llama_context::decode(const llama_batch & batch_inp) {
 
 
         // make the outputs have the same order they had in the user-provided batch
         // make the outputs have the same order they had in the user-provided batch
         // note: this is mostly relevant for recurrent models atm
         // note: this is mostly relevant for recurrent models atm
-        if (!sorted_output) {
+        if (!sorted_output && n_outputs > 1) {
             GGML_ASSERT((size_t) n_outputs == out_ids.size());
             GGML_ASSERT((size_t) n_outputs == out_ids.size());
 
 
             // TODO: is there something more efficient which also minimizes swaps?
             // TODO: is there something more efficient which also minimizes swaps?