Detect language after tokenization

This is a prerequisite to try to be smarter about what subset of the input text is used for language detection,
however it breaks functionality in one subtle way: If an application does not pass language explicitly (such that
it must be detected), and the input is CJK, and there are configured special tokens, those special tokens will
not be detected if they are surrounded by word characters (instead of e.g space).

1 files changed, 2 insertions, 2 deletions

diff --git a/container-search/src/main/java/com/yahoo/prelude/IndexFacts.java b/container-search/src/main/java/com/yahoo/prelude/IndexFacts.java
index 3631dedeffc..3f931c92489 100644
--- a/container-search/src/main/java/com/yahoo/prelude/IndexFacts.java
+++ b/container-search/src/main/java/com/yahoo/prelude/IndexFacts.java
@@ -18,11 +18,11 @@ import static com.yahoo.text.Lowercase.toLowerCase;
  * session.getIndex(indexName).[get index info]
  * </code></pre>
  *
- * @author <a href="mailto:steinar@yahoo-inc.com">Steinar Knutsen</a>
+ * @author Steinar Knutsen
  */
 // TODO: We should replace this with a better representation of search definitions
 //       which is immutable, models clusters and search definitions inside clusters properly,
-//       and uses better names.
+//       and uses better names. -bratseth
 public class IndexFacts {
 
     private Map<String, List<String>> clusterByDocument;