aboutsummaryrefslogtreecommitdiffstats
path: root/linguistics/src/main/java/com/yahoo/language/simple/SimpleToken.java
Commit message (Expand)AuthorAgeFilesLines
* Revert "Merge pull request #29328 from vespa-engine/revert-29314-bratseth/cas...Jon Bratseth2023-11-141-10/+27
* Revert "Bratseth/casing take 2"Harald Musum2023-11-131-27/+10
* Prefer first stem to original if non equalJon Bratseth2023-11-101-10/+27
* Update copyrightJon Bratseth2023-10-091-1/+1
* No functional changesJon Bratseth2022-09-111-11/+9
* Determine token types considering all charactersJon Bratseth2022-08-161-33/+16
* Update 2017 copyright notices.gjoranv2021-10-071-1/+1
* Pure Java sentencepiece implementationJon Bratseth2021-09-131-2/+3
* Require replacements to be applied during tokenizationJon Bratseth2021-06-151-0/+4
* Nonfunctional changes onlyJon Bratseth2018-06-191-1/+1
* Update copyright headersJon Bratseth2017-06-141-1/+1
* Revert "Update copyright headers"Jon Bratseth2017-06-141-1/+1
* Update copyright headersJon Bratseth2017-06-141-1/+1
* Revert "Copyright header"Jon Bratseth2017-06-131-1/+1
* Copyright headerJon Bratseth2017-06-131-1/+1
* PublishJon Bratseth2016-06-151-0/+188