summaryrefslogtreecommitdiffstats
path: root/linguistics/src/main
Commit message (Collapse)AuthorAgeFilesLines
* Require replacements to be applied during tokenizationJon Bratseth2021-06-153-12/+11
|
* Revert "Merge pull request #17754 from ↵Jon Bratseth2021-05-056-13/+245
| | | | | | | vespa-engine/revert-17747-bratseth/special-tokens-take-2" This reverts commit a2c9cd4bc04f1a3eaa31524b3970b96be5c2eda9, reversing changes made to 8c61a373af0066fbdf1cca354c24b197c7347321.
* Revert "Reapply "Bratseth/special tokens""Jon Bratseth2021-05-056-245/+13
|
* Revert "Merge pull request #17746 from ↵Jon Bratseth2021-05-056-13/+245
| | | | | | | vespa-engine/revert-17738-revert-17737-revert-17736-bratseth/special-tokens" This reverts commit 491856b396d003885e159345fe3f533f0fa35933, reversing changes made to 3720186303f4aef1d185525eaf61092097a64ec9.
* Revert "Revert "Revert "Bratseth/special tokens"""Jon Bratseth2021-05-056-245/+13
|
* Revert "Revert "Bratseth/special tokens""Jon Bratseth2021-05-046-13/+245
|
* Revert "Bratseth/special tokens"Jon Bratseth2021-05-046-245/+13
|
* Avoid config in simple tokenizerJon Bratseth2021-05-041-7/+4
|
* Expose tokens as mapJon Bratseth2021-05-042-6/+13
|
* Wire in (but don't use) SpecialTokensJon Bratseth2021-05-046-18/+40
|
* Move specialtokens to linguisticsJon Bratseth2021-05-042-0/+206
|
* No functional changesJon Bratseth2021-04-145-109/+68
|
* No functional changesJon Bratseth2021-02-031-1/+1
|
* Add a testJon Bratseth2020-11-121-3/+0
|
* Use full name in config definition file namesHarald Musum2020-09-101-0/+0
|
* handle plugin tokenizer returning tokens with empty original stringArne Juul2020-08-241-1/+4
|
* Surrogate aware gram splittingJon Bratseth2020-06-251-24/+85
|
* SpareCapacityMaintainer sketchJon Bratseth2020-06-126-66/+35
|
* variables in lambdas must be finalArne Juul2020-04-242-10/+16
|
* Apply suggestions from code reviewArne H Juul2020-04-243-7/+7
| | | Co-Authored-By: Jon Bratseth <bratseth@oath.com>
* add more tracing and debug logging of stemmingArne Juul2020-04-244-1/+25
|
* Add/corect copyright headersJon Bratseth2020-01-031-0/+1
|
* Build tensors purely with floatsJon Bratseth2019-04-261-1/+1
|
* Move bound builder double array into double subclassJon Bratseth2019-04-261-1/+1
|
* Allow destructive changes in manually deployed zonesJon Bratseth2019-04-011-1/+1
|
* Nonfunctional changes onlyJon Bratseth2019-01-241-1/+1
|
* Generate html5 javadocgjoranv2019-01-211-7/+7
|
* Remove deprecated method (again)Jon Bratseth2019-01-212-16/+0
|
* Make SimpleLinguistics simple againJon Bratseth2019-01-214-104/+32
| | | | | - Remove SimpleLinguistics config and optional use of Optimaize - Add Optimaize to OpennlpLinguistics; on by default and config to disable
* Remove deprecated apis in linguistics.gjoranv2019-01-213-41/+0
|
* Deprecated methods and add OptimaizeDetectorJon Bratseth2018-11-016-0/+125
|
* Prepare for removal of deprecated membersJon Bratseth2018-10-163-4/+8
|
* Reduce code duplicationHenning Baldersheim2018-10-052-15/+14
|
* Do not create huge optimaize structures when not necessary.Henning Baldersheim2018-10-052-1/+9
|
* Add copyright headerJon Bratseth2018-10-012-0/+2
|
* Defer loading the huge optimaize knowledgepool until you really need it. ↵Henning Baldersheim2018-09-101-20/+32
| | | | This cuts min memory footprint by 100MB+.
* Send global constantsJon Bratseth2018-09-063-1/+4
|
* Add missing newline at end of fileBjørn Christian Seime2018-07-261-1/+2
|
* Add config for simple-linguisticsBjørn Christian Seime2018-07-263-8/+45
| | | | Add a config parameter for enabling/disabling optimaize detector
* Merge pull request #6452 from jefimm/mastergjoranv2018-07-251-2/+52
|\ | | | | use com.optimaize.langdetect for lang detection
| * use com.optimaize.langdetect for lang detectionJefim Matskin2018-07-241-2/+52
| |
* | Export package com.yahoo.language.opennlp.gjoranv2018-07-241-0/+5
|/
* Merge pull request #6444 from vespa-engine/bratseth/java-model-inferenceJon Bratseth2018-07-231-1/+6
|\ | | | | Bratseth/java model inference
| * Java model evaluation WIPJon Bratseth2018-07-201-1/+6
| |
* | add opennlp stemmers - revert previous changesJefim Matskin2018-07-184-118/+153
| | | | | | | | https://github.com/vespa-engine/vespa/issues/6403
* | add lang detection and opennlp stemmersJefim Matskin2018-07-171-3/+7
| | | | | | | | https://github.com/vespa-engine/vespa/issues/6403
* | add lang detection and opennlp stemmersJefim Matskin2018-07-171-3/+2
| | | | | | | | https://github.com/vespa-engine/vespa/issues/6403
* | add lang detection and opennlp stemmersJefim Matskin2018-07-172-6/+114
| | | | | | | | https://github.com/vespa-engine/vespa/issues/6403
* | Fix author tag for SimonBjørn Christian Seime2018-07-058-8/+8
|/
* Merge pull request #6228 from vespa-engine/bratseth/nonfunctional-changesgjoranv2018-06-193-4/+4
|\ | | | | Nonfunctional changes only