diff options
| author | Henning Baldersheim <balder@yahoo-inc.com> | 2024-01-11 16:02:16 +0000 |
|---|---|---|
| committer | Henning Baldersheim <balder@yahoo-inc.com> | 2024-01-11 16:02:16 +0000 |
| commit | e8c46d963b2f00b4b02de0783aa5c8a39cfd0c95 (patch) | |
| tree | 0161b150ea3ccf6ed3278a3eec5c5cf114891829 /streamingvisitors/src/vespa/vsm/searcher | |
| parent | 8a14af615bee86a178ea4838cc91d2079d9007aa (diff) | |
Add brief class documentation.
Diffstat (limited to 'streamingvisitors/src/vespa/vsm/searcher')
| -rw-r--r-- | streamingvisitors/src/vespa/vsm/searcher/tokenizereader.h | 4 |
1 files changed, 4 insertions, 0 deletions
diff --git a/streamingvisitors/src/vespa/vsm/searcher/tokenizereader.h b/streamingvisitors/src/vespa/vsm/searcher/tokenizereader.h index 76ca2e8d24b..f10c8910e82 100644 --- a/streamingvisitors/src/vespa/vsm/searcher/tokenizereader.h +++ b/streamingvisitors/src/vespa/vsm/searcher/tokenizereader.h @@ -7,6 +7,10 @@ namespace vsm { +/** + * Handles tokenization of utf8 input with on the fly normalization. + * It handles Normalizing::NONE, Normalizing::LOWERCASE, and Normalizing::LOWERCASE_AND_FOLD + */ class TokenizeReader { public: using byte = search::byte; |