aboutsummaryrefslogtreecommitdiffstats
path: root/linguistics/src/main/java/com/yahoo/language/simple/SimpleTokenizer.java
Commit message (Collapse)AuthorAgeFilesLines
* Update copyrightJon Bratseth2023-10-091-1/+1
|
* Don't remove indexable symbols when stemmingJon Bratseth2023-06-021-1/+7
|
* Use dollar and hour base unitsJon Bratseth2023-05-191-2/+2
|
* Determine token types considering all charactersJon Bratseth2022-08-161-5/+21
|
* Update 2017 copyright notices.gjoranv2021-10-071-1/+1
|
* Revert "Merge pull request #17754 from ↵Jon Bratseth2021-05-051-10/+17
| | | | | | | vespa-engine/revert-17747-bratseth/special-tokens-take-2" This reverts commit a2c9cd4bc04f1a3eaa31524b3970b96be5c2eda9, reversing changes made to 8c61a373af0066fbdf1cca354c24b197c7347321.
* Revert "Reapply "Bratseth/special tokens""Jon Bratseth2021-05-051-17/+10
|
* Revert "Merge pull request #17746 from ↵Jon Bratseth2021-05-051-10/+17
| | | | | | | vespa-engine/revert-17738-revert-17737-revert-17736-bratseth/special-tokens" This reverts commit 491856b396d003885e159345fe3f533f0fa35933, reversing changes made to 3720186303f4aef1d185525eaf61092097a64ec9.
* Revert "Revert "Revert "Bratseth/special tokens"""Jon Bratseth2021-05-051-17/+10
|
* Revert "Revert "Bratseth/special tokens""Jon Bratseth2021-05-041-10/+17
|
* Revert "Bratseth/special tokens"Jon Bratseth2021-05-041-17/+10
|
* Wire in (but don't use) SpecialTokensJon Bratseth2021-05-041-10/+17
|
* variables in lambdas must be finalArne Juul2020-04-241-5/+8
|
* Apply suggestions from code reviewArne H Juul2020-04-241-2/+2
| | | Co-Authored-By: Jon Bratseth <bratseth@oath.com>
* add more tracing and debug logging of stemmingArne Juul2020-04-241-1/+9
|
* Update copyright headersJon Bratseth2017-06-141-1/+1
|
* Revert "Update copyright headers"Jon Bratseth2017-06-141-1/+1
|
* Update copyright headersJon Bratseth2017-06-141-1/+1
|
* Revert "Copyright header"Jon Bratseth2017-06-131-1/+1
|
* Copyright headerJon Bratseth2017-06-131-1/+1
|
* Nonfunctional changes only#Jon Bratseth2017-05-191-1/+1
|
* PublishJon Bratseth2016-06-151-0/+76