 Georgi Gerganov
|
08f10f69c3
llama : remove notion of CLS token (#11064)
|
1 year ago |
|
Georgi Gerganov
|
afa8a9ec9b
llama : add `llama_vocab`, functions -> methods, naming (#11110)
|
1 year ago |
|
Georgi Gerganov
|
f66f582927
llama : refactor `src/llama.cpp` (#10902)
|
1 year ago |
|
Georgi Gerganov
|
30caac3a68
llama : the WPM vocabs use the CLS token as BOS (#10930)
|
1 year ago |
 wwoodsTM
|
ff252ea48e
llama : add DRY sampler (#9702)
|
1 year ago |
|
Georgi Gerganov
|
755a9b2bf0
llama : add infill sampler (#9896)
|
1 year ago |
|
Georgi Gerganov
|
11ac9800af
llama : improve infill support and special token detection (#9798)
|
1 year ago |
|
Georgi Gerganov
|
8c475b97b8
rerank : use [SEP] token instead of [BOS] (#9737)
|
1 year ago |
 Zhenwei Jin
|
6102037bbb
vocab : refactor tokenizer to reduce init overhead (#9449)
|
1 year ago |
|
Georgi Gerganov
|
31ac5834fe
llama : keep track of all EOG tokens in the vocab (#9609)
|
1 year ago |
|
Georgi Gerganov
|
df270ef745
llama : refactor sampling v2 (#9294)
|
1 year ago |
|
Zhenwei Jin
|
4af8420afb
common : remove duplicate function llama_should_add_bos_token (#8778)
|
1 year ago |
 fairydreaming
|
d3f0c7166a
Stop the generation when <|eom_id|> token is encountered - needed for Llama 3.1 tool call support (#8858)
|
1 year ago |
|
Georgi Gerganov
|
938943cdbf
llama : move vocab, grammar and sampling into separate files (#8508)
|
1 year ago |
