aboutsummaryrefslogtreecommitdiffstats
path: root/model-integration
Commit message (Collapse)AuthorAgeFilesLines
* update onnx.protoArne Juul2023-06-234-80/+453
| | | | | * use latest version from https://github.com/onnx/onnx/blob/main/onnx/onnx.proto * track API changes (enum -> int32)
* Prefer truncation configuration from tokenizer modelBjørn Christian Seime2023-06-121-6/+19
| | | | | | | Only override truncation if not specified or max length exceeds max tokens accepted by model. Use JNI wrapper directly to determine existing truncation configuration (JSON format is not really documented). Simply configuration for pure tokenizer embedder. Disable DJL usage telemetry.
* Add missing wiring of pooling strategyBjørn Christian Seime2023-06-081-11/+1
|
* Disable padding and make it configurableBjørn Christian Seime2023-06-081-0/+1
|
* Merge pull request #27297 from vespa-engine/bjorncs/bert-embedder-services-xmlBjørn Christian Seime2023-06-064-49/+54
|\ | | | | Bjorncs/bert embedder services xml
| * Make pooling strategy configurable for Huggingface embedderBjørn Christian Seime2023-06-053-17/+54
| |
| * Move config definition to `configdefinitions`Bjørn Christian Seime2023-06-051-32/+0
| |
* | Add necessary options to use failOnWarningsgjoranv2023-06-051-0/+4
|/
* Introduce services.xml syntax for configuring HuggingFace embeddersBjørn Christian Seime2023-06-022-29/+6
|
* Properly ignore token type ids from tokenizer if disabledBjørn Christian Seime2023-05-301-2/+2
|
* Remove dead codeBjørn Christian Seime2023-05-262-43/+0
|
* Make truncation and max length configurableBjørn Christian Seime2023-05-261-12/+3
|
* Use GPU by default if availableBjørn Christian Seime2023-05-222-2/+4
|
* Revert "Revert "Bjorncs/huggingface tokenizer""Bjørn Christian Seime2023-05-126-210/+29
| | | | This reverts commit 2bb74878879b3acb1919fd658b8f2c476d8129d6.
* Revert "Bjorncs/huggingface tokenizer"Arnstein Ressem2023-05-126-29/+210
|
* Handle models requiring token type idsBjørn Christian Seime2023-05-112-13/+20
|
* Don't lower caseBjørn Christian Seime2023-05-111-1/+1
|
* Disable special tokens by defaultBjørn Christian Seime2023-05-111-0/+1
|
* Mark HF integration as betaBjørn Christian Seime2023-05-111-0/+2
|
* Make HF tokenizer a separate embedderBjørn Christian Seime2023-05-115-197/+6
|
* Don't specify both package and namespaceBjørn Christian Seime2023-05-112-1/+1
|
* Upgrade HF Tokenizer to 0.22.1Bjørn Christian Seime2023-05-081-1/+1
|
* Handle nullsBjørn Christian Seime2023-05-081-0/+4
|
* fixup! Require GPU when requested and available for Bert + HF embeddersBjørn Christian Seime2023-05-081-1/+1
|
* Require GPU when requested and available for Bert + HF embeddersBjørn Christian Seime2023-05-085-5/+6
|
* Require GPU when available for ONNX evaluation in global-phase and embeddersBjørn Christian Seime2023-05-083-5/+42
|
* Make thread pool size configurableBjørn Christian Seime2023-05-055-17/+24
|
* Make normalization optionalBjørn Christian Seime2023-05-052-2/+8
|
* Allow for manual configuration of GPUBjørn Christian Seime2023-05-052-1/+8
|
* Move config to same package as componentBjørn Christian Seime2023-05-052-1/+1
|
* Split out HF TokenizerBjørn Christian Seime2023-05-054-23/+174
|
* Put the openai client in a separate componentJon Bratseth2023-04-2515-482/+20
|
* Export APIJon Bratseth2023-04-204-1/+218
|
* Merge pull request #26777 from vespa-engine/bratseth/openai-clientJon Bratseth2023-04-1910-0/+312
|\ | | | | Llm completion abstraction and OpenAi implementation
| * Use record and use default record toStringJon Bratseth2023-04-191-9/+1
| |
| * Fix typoJon Bratseth2023-04-191-1/+1
| |
| * Avoid ignored directory nameJon Bratseth2023-04-192-2/+45
| |
| * Llm completion abstraction and OpenAi implementationJon Bratseth2023-04-199-0/+277
| |
* | Merge pull request #26753 from vespa-engine/bjorncs/global-phaseArne H Juul2023-04-191-5/+9
|\ \ | |/ |/| Use quarter vcpu by default if execution mode is parallel
| * Use quarter vcpu by default if execution mode is parallelBjørn Christian Seime2023-04-171-5/+9
| |
* | Merge pull request #26754 from vespa-engine/bratseth/jdk20Jon Bratseth2023-04-181-2/+2
|\ \ | | | | | | Build with jdk20
| * | Build with jdk20Jon Bratseth2023-04-171-2/+2
| | |
* | | Pull endtokenJon Bratseth2023-04-181-1/+1
| | |
* | | Revert "Merge pull request #26744 from ↵Jon Bratseth2023-04-182-8/+14
| |/ |/| | | | | | | | | | | vespa-engine/revert-26708-allow-start-end-sequence-tokens-as-args-bertbaseembedder" This reverts commit d025a93015e66efc0027d81a64e70530d6cb240e, reversing changes made to 4f2f29e1459b900d4b074f5cfc4c126837c54bfd.
* | Revert "Allow start end sequence tokens as args bertbaseembedder"Jon Bratseth2023-04-142-14/+8
|/
* Include createTokenTypeIdsconnell gough2023-04-131-0/+1
|
* Remove separator input and fix spelling errorconnell gough2023-04-131-3/+2
|
* Add default special tokensconnell gough2023-04-131-2/+2
|
* Add special tokens as arguments and allow tokenTypeIds to be nullconnell gough2023-04-132-7/+13
|
* Add lz4-java for xxhashBjørn Christian Seime2023-03-311-0/+5
|