| .. |
|
.editorconfig
|
6381d4e110
gguf : new file format with flexible meta data (beta) (#2398)
|
2 years ago |
|
ggml-vocab-aquila.gguf
|
ff5a3f0c09
Work on the BPE tokenizer (#3252)
|
2 years ago |
|
ggml-vocab-baichuan.gguf
|
daab3d7f45
Add more tokenizer tests (#3742)
|
2 years ago |
|
ggml-vocab-bert-bge.gguf
|
f4ab2a4147
llama : fix BPE pre-tokenization (#6920)
|
1 year ago |
|
ggml-vocab-bert-bge.gguf.inp
|
92139b90af
tests : add test-tokenizer-0.sh + fix some tokenizers (#7036)
|
1 year ago |
|
ggml-vocab-bert-bge.gguf.out
|
92139b90af
tests : add test-tokenizer-0.sh + fix some tokenizers (#7036)
|
1 year ago |
|
ggml-vocab-command-r.gguf
|
889bdd7686
command-r : add BPE pre-tokenization (#7063)
|
1 year ago |
|
ggml-vocab-command-r.gguf.inp
|
889bdd7686
command-r : add BPE pre-tokenization (#7063)
|
1 year ago |
|
ggml-vocab-command-r.gguf.out
|
889bdd7686
command-r : add BPE pre-tokenization (#7063)
|
1 year ago |
|
ggml-vocab-deepseek-coder.gguf
|
f4ab2a4147
llama : fix BPE pre-tokenization (#6920)
|
1 year ago |
|
ggml-vocab-deepseek-coder.gguf.inp
|
92139b90af
tests : add test-tokenizer-0.sh + fix some tokenizers (#7036)
|
1 year ago |
|
ggml-vocab-deepseek-coder.gguf.out
|
92139b90af
tests : add test-tokenizer-0.sh + fix some tokenizers (#7036)
|
1 year ago |
|
ggml-vocab-deepseek-llm.gguf
|
f4ab2a4147
llama : fix BPE pre-tokenization (#6920)
|
1 year ago |
|
ggml-vocab-deepseek-llm.gguf.inp
|
92139b90af
tests : add test-tokenizer-0.sh + fix some tokenizers (#7036)
|
1 year ago |
|
ggml-vocab-deepseek-llm.gguf.out
|
92139b90af
tests : add test-tokenizer-0.sh + fix some tokenizers (#7036)
|
1 year ago |
|
ggml-vocab-falcon.gguf
|
f4ab2a4147
llama : fix BPE pre-tokenization (#6920)
|
1 year ago |
|
ggml-vocab-falcon.gguf.inp
|
92139b90af
tests : add test-tokenizer-0.sh + fix some tokenizers (#7036)
|
1 year ago |
|
ggml-vocab-falcon.gguf.out
|
92139b90af
tests : add test-tokenizer-0.sh + fix some tokenizers (#7036)
|
1 year ago |
|
ggml-vocab-gpt-2.gguf
|
f4ab2a4147
llama : fix BPE pre-tokenization (#6920)
|
1 year ago |
|
ggml-vocab-gpt-2.gguf.inp
|
92139b90af
tests : add test-tokenizer-0.sh + fix some tokenizers (#7036)
|
1 year ago |
|
ggml-vocab-gpt-2.gguf.out
|
92139b90af
tests : add test-tokenizer-0.sh + fix some tokenizers (#7036)
|
1 year ago |
|
ggml-vocab-gpt-neox.gguf
|
daab3d7f45
Add more tokenizer tests (#3742)
|
2 years ago |
|
ggml-vocab-gpt2.gguf
|
ea5497df5d
gpt2 : Add gpt2 architecture integration (#4555)
|
2 years ago |
|
ggml-vocab-llama-bpe.gguf
|
f4ab2a4147
llama : fix BPE pre-tokenization (#6920)
|
1 year ago |
|
ggml-vocab-llama-bpe.gguf.inp
|
f99e1e456e
llama : lookup word in vocab before doing BPE merges (#7193)
|
1 year ago |
|
ggml-vocab-llama-bpe.gguf.out
|
f99e1e456e
llama : lookup word in vocab before doing BPE merges (#7193)
|
1 year ago |
|
ggml-vocab-llama-spm.gguf
|
f4ab2a4147
llama : fix BPE pre-tokenization (#6920)
|
1 year ago |
|
ggml-vocab-llama-spm.gguf.inp
|
92139b90af
tests : add test-tokenizer-0.sh + fix some tokenizers (#7036)
|
1 year ago |
|
ggml-vocab-llama-spm.gguf.out
|
92139b90af
tests : add test-tokenizer-0.sh + fix some tokenizers (#7036)
|
1 year ago |
|
ggml-vocab-mpt.gguf
|
f4ab2a4147
llama : fix BPE pre-tokenization (#6920)
|
1 year ago |
|
ggml-vocab-mpt.gguf.inp
|
92139b90af
tests : add test-tokenizer-0.sh + fix some tokenizers (#7036)
|
1 year ago |
|
ggml-vocab-mpt.gguf.out
|
92139b90af
tests : add test-tokenizer-0.sh + fix some tokenizers (#7036)
|
1 year ago |
|
ggml-vocab-phi-3.gguf
|
3b38d48609
Per token attributes (#7685)
|
1 year ago |
|
ggml-vocab-phi-3.gguf.inp
|
92139b90af
tests : add test-tokenizer-0.sh + fix some tokenizers (#7036)
|
1 year ago |
|
ggml-vocab-phi-3.gguf.out
|
92139b90af
tests : add test-tokenizer-0.sh + fix some tokenizers (#7036)
|
1 year ago |
|
ggml-vocab-qwen2.gguf
|
229ffff872
llama : add BPE pre-tokenization for Qwen2 (#7114)
|
1 year ago |
|
ggml-vocab-qwen2.gguf.inp
|
229ffff872
llama : add BPE pre-tokenization for Qwen2 (#7114)
|
1 year ago |
|
ggml-vocab-qwen2.gguf.out
|
229ffff872
llama : add BPE pre-tokenization for Qwen2 (#7114)
|
1 year ago |
|
ggml-vocab-refact.gguf
|
92139b90af
tests : add test-tokenizer-0.sh + fix some tokenizers (#7036)
|
1 year ago |
|
ggml-vocab-refact.gguf.inp
|
92139b90af
tests : add test-tokenizer-0.sh + fix some tokenizers (#7036)
|
1 year ago |
|
ggml-vocab-refact.gguf.out
|
92139b90af
tests : add test-tokenizer-0.sh + fix some tokenizers (#7036)
|
1 year ago |
|
ggml-vocab-stablelm.gguf
|
f4ab2a4147
llama : fix BPE pre-tokenization (#6920)
|
1 year ago |
|
ggml-vocab-starcoder.gguf
|
f4ab2a4147
llama : fix BPE pre-tokenization (#6920)
|
1 year ago |
|
ggml-vocab-starcoder.gguf.inp
|
92139b90af
tests : add test-tokenizer-0.sh + fix some tokenizers (#7036)
|
1 year ago |
|
ggml-vocab-starcoder.gguf.out
|
92139b90af
tests : add test-tokenizer-0.sh + fix some tokenizers (#7036)
|
1 year ago |
jaime-m-p