 Daniel Bevenius
|
70cd37dbbe
requirements : update transformers/torch for Embedding Gemma (#15828)
|
4 months ago |
 Georgi Gerganov
|
afa8a9ec9b
llama : add `llama_vocab`, functions -> methods, naming (#11110)
|
1 year ago |
 compilade
|
fa79495bb4
llama : fix pre-tokenization of non-special added tokens (#8228)
|
1 year ago |
|
compilade
|
3fd62a6b1c
py : type-check all Python scripts with Pyright (#8341)
|
1 year ago |
 jaime-m-p
|
213701b51a
Detokenizer fixes (#8039)
|
1 year ago |
|
jaime-m-p
|
37bef89433
tokenizer : BPE fixes (#7530)
|
1 year ago |
|
jaime-m-p
|
3b38d48609
Per token attributes (#7685)
|
1 year ago |
|
jaime-m-p
|
02c1ecad07
Tokenizer WPM fixes (#7500)
|
1 year ago |
|
jaime-m-p
|
d7e852c1bc
Tokenizer SPM fixes for phi-3 and llama-spm (bugfix) (#7425)
|
1 year ago |
|
jaime-m-p
|
917dc8cfa6
Tokenizer SPM fixes for phi-3 and llama-spm (#7375)
|
1 year ago |
|
jaime-m-p
|
b43272afa2
Unicode codepoint flags for custom regexs (#7245)
|
1 year ago |
|
jaime-m-p
|
43248e5594
llama3 custom regex split (#6965)
|
1 year ago |
