diff options
| author | Bjørn Christian Seime <bjorncs@yahooinc.com> | 2023-06-08 14:23:16 +0200 |
|---|---|---|
| committer | Bjørn Christian Seime <bjorncs@yahooinc.com> | 2023-06-08 14:23:16 +0200 |
| commit | c0652d7794a90e0afb593fc1a3db17c99606a808 (patch) | |
| tree | 17887acf2818107bbeb7355f5ee463f5fb02873d /configdefinitions | |
| parent | c3d8c532e0f5b1db896d8693409098e8c2980da1 (diff) | |
Disable padding and make it configurable
Diffstat (limited to 'configdefinitions')
| -rw-r--r-- | configdefinitions/src/vespa/hugging-face-tokenizer.def | 5 |
1 files changed, 3 insertions, 2 deletions
diff --git a/configdefinitions/src/vespa/hugging-face-tokenizer.def b/configdefinitions/src/vespa/hugging-face-tokenizer.def index 18b3631e494..bc0d5300de5 100644 --- a/configdefinitions/src/vespa/hugging-face-tokenizer.def +++ b/configdefinitions/src/vespa/hugging-face-tokenizer.def @@ -9,5 +9,6 @@ model[].language string model[].path model addSpecialTokens bool default=true -maxLength int default=-1 -truncation bool default=false
\ No newline at end of file +maxLength int default=512 +truncation bool default=true +padding bool default=false |