diff options
| author | Bjørn Christian Seime <bjorncs@yahooinc.com> | 2023-05-26 15:44:33 +0200 |
|---|---|---|
| committer | Bjørn Christian Seime <bjorncs@yahooinc.com> | 2023-05-26 15:44:33 +0200 |
| commit | da7eaaf9389b6b2d1bfd1bcff3770d61b150fd1f (patch) | |
| tree | 9562eb4764e15ea9452f8f0911bfe3737c5666b7 /linguistics-components/src/main/resources/configdefinitions | |
| parent | 17f53418cfaaf49b87f06123af0aaf73f3593fe8 (diff) | |
Make truncation and max length configurable
Diffstat (limited to 'linguistics-components/src/main/resources/configdefinitions')
| -rw-r--r-- | linguistics-components/src/main/resources/configdefinitions/language.huggingface.hugging-face-tokenizer.def | 4 |
1 files changed, 3 insertions, 1 deletions
diff --git a/linguistics-components/src/main/resources/configdefinitions/language.huggingface.hugging-face-tokenizer.def b/linguistics-components/src/main/resources/configdefinitions/language.huggingface.hugging-face-tokenizer.def index 5e58547879c..67b3b927f94 100644 --- a/linguistics-components/src/main/resources/configdefinitions/language.huggingface.hugging-face-tokenizer.def +++ b/linguistics-components/src/main/resources/configdefinitions/language.huggingface.hugging-face-tokenizer.def @@ -8,4 +8,6 @@ model[].language string # The path to the model relative to the application package root model[].path model -addSpecialTokens bool default=true
\ No newline at end of file +addSpecialTokens bool default=true +maxLength int default=-1 +truncation bool default=false
\ No newline at end of file |