瀏覽代碼

py : remove unnecessary hasattr (#4903)

Georgi Gerganov 2 年之前
父節點
當前提交
5c99960901
共有 1 個文件被更改,包括 4 次插入5 次删除
  1. 4 5
      convert-hf-to-gguf.py

+ 4 - 5
convert-hf-to-gguf.py

@@ -266,11 +266,10 @@ class Model:
                 toktypes.append(gguf.TokenType.USER_DEFINED)
             elif reverse_vocab[i] in added_vocab:
                 tokens.append(reverse_vocab[i])
-                if hasattr(tokenizer, "added_tokens_decoder"):
-                    if tokenizer.added_tokens_decoder[i].special:
-                        toktypes.append(gguf.TokenType.CONTROL)
-                    else:
-                        toktypes.append(gguf.TokenType.USER_DEFINED)
+                if tokenizer.added_tokens_decoder[i].special:
+                    toktypes.append(gguf.TokenType.CONTROL)
+                else:
+                    toktypes.append(gguf.TokenType.USER_DEFINED)
             else:
                 tokens.append(reverse_vocab[i])
                 toktypes.append(gguf.TokenType.NORMAL)