Merge branch 'master' into arnej/extend-intellij-plugin

6 files changed, 151 insertions, 83 deletions

diff --git a/sd-plugin/BACKLOG.md b/sd-plugin/BACKLOG.md
new file mode 100644
index 00000000000..19e185dcc6b
--- /dev/null
+++ b/sd-plugin/BACKLOG.md
@@ -0,0 +1,34 @@
+### Open Issues
+
+1. In some cases, the grammar prefers not to enforce bad syntax, because if the parser encounters bad syntax it stops 
+and can't build the PSI tree. That means none of the features will work for a file like that. For example, in cases 
+where an element supposes to have zero-to-one occurrences, the grammar will treat it as zero-to-many. 
+2. In order to enable the grammar recognize some keywords as identifiers (e.g. "filter" as a field's name), the 
+identifier rule (named "IdentifierVal") wraps the regex (ID_REG) and the KeywordOrIdentifier rule (which contains all 
+the keywords in the language). 
+3. The implementation of the GoTo Declaration feature is not exactly the same as IntelliJ. In IntelliJ if a reference
+has several declarations, after clicking "Goto Declaration" there is a little window with all the declarations to choose 
+from. It can be done by changing the method "multiResolve" in SdReference.java to return more than one declaration. The 
+problem with that is that it causes the "Find Usages" feature to not work. For now, I decided to make the plugin 
+"Goto Declaration" feature show only the most specific declaration by the right rank-profile scope.
+4. The "Find Usages" window can group usages only under rank-profiles and document-summaries. Other usages appear 
+directly under the .sd file. In order to create another group type of usages' group, you'll need to create 2 classes: 
+one for the extension "fileStructureGroupRuleProvider" (e.g. SdRankProfileGroupingRuleProvider.java), and one for the 
+grouping rule itself (e.g. SdRankProfileGroupingRule.java).
+Another open problem is that the navigation isn't working in the current grouping rules. It means that when clicking on 
+the group headline (e.g. some name of a rank-profile) the IDE doesn't "jump" to the matching declaration.
+5. Goto declaration doesn't work for document's inherits. e.g. if document A inherits from document B, B doesn't have a 
+reference to its declaration.
+6. There aren't any tests for the plugin.
+
+### Some useful links:
+1. JetBrains official tutorials: https://plugins.jetbrains.com/docs/intellij/custom-language-support.html and
+https://plugins.jetbrains.com/docs/intellij/custom-language-support-tutorial.html
+2. Grammar-Kit HOWTO: Helps to understand the BNF syntax.
+   https://github.com/JetBrains/Grammar-Kit/blob/master/HOWTO.md
+3. How to deal with left-recursion in the grammar (in SD for example it happens in expressions). Last answer here: 
+https://intellij-support.jetbrains.com/hc/en-us/community/posts/360001258300-What-s-the-alternative-to-left-recursion-in-GrammarKit-
+4. Great tutorial for a custom-language-plugin, but only for the basics (mainly the parser and lexer):
+   https://medium.com/@shan1024/custom-language-plugin-development-for-intellij-idea-part-01-d6a41ab96bc9
+5. Code of Dart (some custom language) plugin for IntelliJ: 
+https://github.com/JetBrains/intellij-plugins/tree/0f07ca63355d5530b441ca566c98f17c560e77f8/Dart
\ No newline at end of file
diff --git a/sd-plugin/README.md b/sd-plugin/README.md
index 74ca1781596..1cab356f78e 100644
--- a/sd-plugin/README.md
+++ b/sd-plugin/README.md
@@ -1,25 +1,42 @@
 <!-- Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. -->
-This directory holds the code for an IntteliJ plugin for reading SD files.
+This directory holds the code for an IntelliJ plugin for reading SD files.
 
-NOTE: This is the source code, not the plugin itself. In order to be able to use the plugin you'll need to download it from JetBrains Marketplace or create a zip file and load it to IntelliJ (details later).
+NOTE: This is the source code, not the plugin itself. In order to be able to use the plugin you'll need to download it 
+from JetBrains Marketplace or create a zip file and load it to IntelliJ by choosing "Install Plugin from Disk".
 
-Before cloning, you should download Gradle and create a Gradle project.
-You should also download Grammar-Kit plugin from the Marketplace.
+Before starting, you should:
+1. Download Gradle 7 (if you don't have it already). 
+2. Make sure that the bundled Plugin DevKit plugin is enabled (inside IntelliJ). 
+3. Optional- Download Grammar-Kit plugin from JetBrains Marketplace (inside IntelliJ). It helps with reading the .bnf file.
+4. Optional- Download PsiViewer plugin from JetBrains Marketplace (inside IntelliJ). It helps to test the grammar defined
+in the .bnf file.
 
+### Working Process
 The grammar is defined in 2 files:
 - sd.bnf
 - sd.flex
 
-After cloning, you should:
-1. Right-click the sd.bnf file and press "Generate Parser Code" 
-2. Right-click the sd.flex file and press "Run JFlex Generator"
- 
-Now you should have a "gen" folder next to the "java" folder, and it contains all the parser and lexer code.
+In order to generate the lexer and parser's code, you need to run in the command line:
 
-Important note! After any change in one of this 2 files (bnf, flex) you'll need to generate again. The proper way is to delete the "gen" folder and then do 1-2 again.
+    ./gradlew generateSdParser
+    ./gradlew generateSdLexer
 
-Now, you can run the gradle task "intellij/runIde", open a project with some sd file and see how the plugin works on it.
+You should now have a "gen" folder next to the "java" folder, and it contains all the parser and lexer code.
 
-In order to test the plugin locally (on you IDE, not by running the gradle task "runIde"), you can run the gradle task 
-"intellij/buildPlugin". It would create a zip file in the directory build\distributions. You can load it to IntelliJ by 
-clicking the "settings" in preferences/Plugins and click "Install Plugin from disk".
\ No newline at end of file
+NOTE- Running those tasks would reset the "gen" folder, and all the previous generated files would be deleted before the
+new ones would be generated.
+
+Now, you can run the gradle task "intellij/runIde" (or "./gradlew runIde" in the command line), open a project with some sd file and see how the plugin works on it.
+
+### Build the Plugin
+In order to build the plugin and create a zip file from it, you should run the command:
+    
+    ./gradlew buildPlugin
+
+Or since it's a default task you can just run:
+
+    ./gradlew
+
+This task also invokes the tasks generateSdParser and generateSdLexer as a part of the building process.
+
+Now, you'll have a zip file in the directory build\distributions.
diff --git a/sd-plugin/build.gradle b/sd-plugin/build.gradle
index 0a4798032d9..e5b39769b96 100644
--- a/sd-plugin/build.gradle
+++ b/sd-plugin/build.gradle
@@ -38,7 +38,7 @@ compileJava {
 
 
 group 'org.yahoo.native'
-version '1.0.1'
+version '1.0.2'
 
 sourceCompatibility = 11
 
diff --git a/sd-plugin/src/main/java/org/intellij/sdk/language/SdSyntaxHighlighter.java b/sd-plugin/src/main/java/org/intellij/sdk/language/SdSyntaxHighlighter.java
index 46cbffe0aa8..e6452685e3c 100644
--- a/sd-plugin/src/main/java/org/intellij/sdk/language/SdSyntaxHighlighter.java
+++ b/sd-plugin/src/main/java/org/intellij/sdk/language/SdSyntaxHighlighter.java
@@ -173,6 +173,7 @@ public class SdSyntaxHighlighter extends SyntaxHighlighterBase {
         constants.add(SdTypes.SUMMARY);
         constants.add(SdTypes.INDEX);
         constants.add(SdTypes.SET_LANGUAGE);
+        constants.add(SdTypes.LOWERCASE);
         constants.add(SdTypes.FAST_SEARCH);
         constants.add(SdTypes.FAST_ACCESS);
         constants.add(SdTypes.PAGED);
diff --git a/sd-plugin/src/main/java/org/intellij/sdk/language/lexer/sd.flex b/sd-plugin/src/main/java/org/intellij/sdk/language/lexer/sd.flex
index 5368551b81b..585a6672189 100644
--- a/sd-plugin/src/main/java/org/intellij/sdk/language/lexer/sd.flex
+++ b/sd-plugin/src/main/java/org/intellij/sdk/language/lexer/sd.flex
@@ -32,12 +32,12 @@ ID_WITH_DASH = [a-zA-Z_][a-zA-Z0-9_-]*
 WHITE_SPACE=[ \t\n\x0B\f\r]+
 
 COMMENT=#.*
-SYMBOL= [|:{}(),.\[\]]
+SYMBOL= [!$|:{}(),.\[\]]
 INTEGER = [0-9]+
 FLOAT = {INTEGER}[.][0-9]+[e]?
 COMPARISON_OPERATOR = [<>]|(==)|(<=)|(>=)|(\~=)
 ARITHMETIC_OPERATOR = [\-+*/]
-STRING = [\"][^\"\n]*[\"]
+STRING = \"([^\"\\]*(\\.[^\"\\]*)*)\"
 WORD = \w+
 
 
@@ -109,17 +109,21 @@ WORD = \w+
   "rank"                     { return RANK; }
   "filter"                   { return FILTER; }
   "normal"                   { return NORMAL; }
+  "literal"                  { return LITERAL; }
   "indexing-rewrite"         { return INDEXING_REWRITE; }
   "none"                     { return NONE; }
   "query-command"            { return QUERY_COMMAND; }
   "full"                     { return FULL; }
-  "dinamic"                  { return DYNAMIC; }
+  "static"                   { return STATIC; }
+  "dynamic"                  { return DYNAMIC; }
   "source"                   { return SOURCE; }
   "to"                       { return TO; }
   "matched-elements-only"    { return MATCHED_ELEMENTS_ONLY; }
 
   "input"                    { return INPUT; }
   "mutable"                  { return MUTABLE; }
+  "enable-bit-vectors"       { return ENABLE_BIT_VECTORS; }
+  "enable-only-bit-vector"   { return ENABLE_ONLY_BIT_VECTOR; }
   "document-summary"         { return DOCUMENT_SUMMARY; }
   "from-disk"                { return FROM_DISK; }
   "omit-summary-features"    { return OMIT_SUMMARY_FEATURES; }
@@ -127,10 +131,12 @@ WORD = \w+
   "as"                       { return AS; }
 
   "rank-profile"             { return RANK_PROFILE; }
+  "model"                    { return MODEL; }
   "match-phase"              { return MATCH_PHASE; }
   "order"                    { return ORDER; }
   "ascending"                { return ASCENDING; }
   "descending"               { return DESCENDING; }
+  "locale"                   { return LOCALE; }
   "max-hits"                 { return MAX_HITS; }
   "diversity"                { return DIVERSITY; }
   "min-groups"               { return MIN_GROUPS; }
@@ -154,6 +160,7 @@ WORD = \w+
   "constants"                { return CONSTANTS; }
   "second-phase"             { return SECOND_PHASE; }
   "rerank-count"             { return RERANK_COUNT; }
+  "rank-features"            { return RANK_FEATURES; }
 
   "weight"                   { return WEIGHT; }
   "index"                    { return INDEX; }
@@ -203,6 +210,10 @@ WORD = \w+
   
   "body"                     { return BODY; }
   "header"                   { return HEADER; }
+  "summary-to"               { return SUMMARY_TO; }
+      
+  "evaluation-point"         { return EVALUATION_POINT; }
+  "pre-post-filter-tipping-point" { return PRE_POST_FILTER_TIPPING_POINT; }
       
   // In here, we check for character sequences which matches regular expressions defined above.
   {ID}                       { return ID_REG; }
diff --git a/sd-plugin/src/main/java/org/intellij/sdk/language/parser/sd.bnf b/sd-plugin/src/main/java/org/intellij/sdk/language/parser/sd.bnf
index 243e177dbdd..67bdca7d2dc 100644
--- a/sd-plugin/src/main/java/org/intellij/sdk/language/parser/sd.bnf
+++ b/sd-plugin/src/main/java/org/intellij/sdk/language/parser/sd.bnf
@@ -28,21 +28,21 @@ NOTE: This grammar does not enforce zero-or-one occurrences of elements (treats
         ID_WITH_DASH_REG = 'regexp:[a-zA-Z_][a-zA-Z0-9_-]*'
         WHITE_SPACE = 'regexp:\s+'
         COMMENT = 'regexp:#.*'
-        SYMBOL = 'regexp:[|:{}(),.\[\]]'
+        SYMBOL = 'regexp:[!$|:{}(),.\[\]]'
         COMPARISON_OPERATOR = 'regexp:[<>]|(==)|(<=)|(>=)|(~=)'
         ARITHMETIC_OPERATOR = 'regexp:[\-+*/]'
         INTEGER_REG = 'regexp:[0-9]+'
         FLOAT_REG = 'regexp:[0-9]+[.][0-9]+[e]?'
-        STRING_REG = 'regexp:[\"][^\"]*[\"]'
+        STRING_REG = 'regexp:\"([^\"\\]*(\\.[^\"\\]*)*)\"'
         WORD_REG = 'regexp:\w+'
     ]
 }
 
-SdFile ::= SchemaDefinition
-SchemaDefinition ::= (search | schema) IdentifierVal? '{' SchemaBody '}'
+SdFile ::= SchemaDefinition | DocumentDefinition
+SchemaDefinition ::= (search | schema) IdentifierVal? (inherits IdentifierVal)? '{' SchemaBody '}'
 SchemaBody ::= SchemaBodyOptions* DocumentDefinition SchemaBodyOptions* // Does not support zero-or-one occurrences
 private SchemaBodyOptions ::= SchemaFieldDefinition | ImportFieldDefinition | DocumentSummaryDefinition | 
-                              RankProfileDefinition | 
+                              RankProfileDefinition | IndexDefinition |
                               FieldSetDefinition | ConstantDefinition | OnnxModelDefinition | StemmingDefinition |
                               raw-as-base64-in-summary | SchemaAnnotationDefinition
 
@@ -54,18 +54,13 @@ SchemaFieldDefinition ::= field IdentifierVal type FieldTypeName '{' SchemaField
                             
 FieldTypeName ::= ("array" '<' (FieldTypeName | IdentifierVal) '>') | ("weightedset" '<' SingleValueFieldTypeName '>') | 
                   ("map" '<' (FieldTypeName | IdentifierVal) ',' (FieldTypeName | IdentifierVal) '>') | TensorType |
-                  SingleValueFieldTypeName
+                  (SingleValueFieldTypeName '[' ']') | SingleValueFieldTypeName
 private SingleValueFieldTypeName ::= "string" | "int" | "long" | "bool" | "byte" | "float" | "double" | "position" | "predicate" | "raw" | "uri" | 
                              "reference" '<' IdentifierVal '>' | "annotationreference" '<' IdentifierVal '>' | IdentifierVal
-private TensorType ::= "tensor" '<' ("float" | "double" | "int8" | "bfloat16") '>' '(' TensorDimension (',' TensorDimension)* ')'
-private TensorDimension ::= WORD_REG ('{' '}') | ('[' INTEGER_REG ']')
+private TensorType ::= "tensor" ('<' ("float" | "double" | "int8" | "bfloat16") '>')? '(' TensorDimension (',' TensorDimension)* ')'
+private TensorDimension ::= WordWrapper (('{' '}') | ('[' INTEGER_REG ']'))
 
-SchemaFieldBody ::= SchemaFieldBodyOptions* // Does not support zero-or-one occurrences
-SchemaFieldBodyOptions ::= SchemaFieldIndexingDefinition | AttributeDefinition | RankDefinition | IndexingRewriteState |
-                           MatchDefinition | StructFieldDefinition | QueryCommandDefinition               
-                    
-SchemaFieldIndexingDefinition ::= indexing (':' SchemaFieldIndexingStatement) | ('{' SchemaFieldIndexingStatement+ '}')
-SchemaFieldIndexingStatement ::= (input IdentifierVal) | IndexingStatement 
+SchemaFieldBody ::= DocumentFieldBodyOptions* // Fields of schemas and documents defined the same way here
 
 DocumentSummaryDefinition ::= document-summary IdentifierWithDashVal (inherits IdentifierWithDashVal)? '{' DocumentSummaryBody '}' 
                               { mixin="org.intellij.sdk.language.psi.impl.SdNamedElementImpl"
@@ -86,7 +81,7 @@ private FieldSetBodyOptions ::= (fields ':' IdentifierVal (',' IdentifierVal)*)
 ConstantDefinition ::= constant IdentifierVal '{' ConstantBody '}'
 ConstantBody ::= ConstantBodyOptions*
 private ConstantBodyOptions ::= (file ':' FilePath) | (uri ':' UriPath) | (type ':' TensorType)
-private FilePath ::= (IdentifierVal | WORD_REG) ('.' | '/' | IdentifierWithDashVal | WORD_REG)+            
+private FilePath ::= WordWrapper (('.' | '/') WordWrapper)+   
 private UriPath ::= ('H'|'h') ('T'|'t') ('T'|'t') ('P'|'p') ('S'|'s')? ':' ('//')? (IdentifierWithDashVal | '.' | '/' | ':')+
 
 
@@ -100,7 +95,7 @@ SchemaAnnotationDefinition ::= AnnotationDefinition
                           implements=["org.intellij.sdk.language.psi.SdDeclaration" "org.intellij.sdk.language.psi.SdNamedElement"]
                          }
                           
-private AnnotationDefinition ::= annotation IdentifierVal '{' AnnotationFieldDefinition* '}'
+private AnnotationDefinition ::= annotation IdentifierVal (inherits IdentifierVal)? '{' AnnotationFieldDefinition* '}'
 AnnotationFieldDefinition ::= field IdentifierVal type FieldTypeName '{' '}'
                           { mixin="org.intellij.sdk.language.psi.impl.SdNamedElementImpl"
                             implements=["org.intellij.sdk.language.psi.SdDeclaration" "org.intellij.sdk.language.psi.SdNamedElement"]
@@ -109,11 +104,13 @@ AnnotationFieldDefinition ::= field IdentifierVal type FieldTypeName '{' '}'
 //-------------------------
 //--- Expressions rules ---
 //-------------------------
-RankingExpression ::= ParenthesisedExpr | BooleanExpr |ArithmeticExpr | IfFunctionExpr | 
-                      QueryDefinitionExpr | FunctionCallExpr | PrimitiveExpr
+RankingExpression ::= FilePathExpr | ParenthesisedExpr | BooleanExpr | ArithmeticExpr | IfFunctionExpr | 
+                      QueryDefinitionExpr | FunctionCallExpr | InListRankingExpr | PrimitiveExpr
 
-IfFunctionExpr ::= "if" '(' (InListRankingExpression | RankingExpression) ',' RankingExpression ',' RankingExpression ')'
-InListRankingExpression ::= RankingExpression "in" '[' RankingExpression (',' RankingExpression)* ']'
+FilePathExpr ::= file ':' (FilePath | WordWrapper)
+
+IfFunctionExpr ::= "if" '(' (InListRankingExpr | RankingExpression) ',' RankingExpression ',' RankingExpression ')'
+InListRankingExpr ::= RankingExpression "in" '[' RankingExpression (',' RankingExpression)* ']'
 
 BooleanExpr ::= RankingExpression COMPARISON_OPERATOR RankingExpression
                 
@@ -125,41 +122,42 @@ FunctionCallExpr ::= IdentifierWithDashVal '(' RankingExpression (',' RankingExp
  
 ParenthesisedExpr ::= '(' RankingExpression ')'
 
-PrimitiveExpr ::= (('-')? INTEGER_REG) | (('-')? FLOAT_REG) | IdentifierVal | RankFeature
+PrimitiveExpr ::= (('-')? INTEGER_REG) | (('-')? FLOAT_REG) | IdentifierVal | RankFeature | STRING_REG
 
 //-------------------------
 //-- Rank Profile rules ---
 //-------------------------
-RankProfileDefinition ::= rank-profile IdentifierWithDashVal (inherits IdentifierWithDashVal)? '{' RankProfileBody '}'
+RankProfileDefinition ::= (rank-profile | model) IdentifierWithDashVal (inherits IdentifierWithDashVal)? '{' RankProfileBody '}'
                           { mixin="org.intellij.sdk.language.psi.impl.SdNamedElementImpl"
                             implements=["org.intellij.sdk.language.psi.SdDeclaration"]
                            }
 private RankProfileBody ::= RankProfileBodyOptions* // Does not support zero-or-one occurrences
 private RankProfileBodyOptions ::= MatchPhaseDefinition | NumThreadsDefinition | FunctionDefinition | TermwiseLimitDefinition | 
                            ignore-default-rank-features | RankPropertiesDefinition | FirstPhaseDefinition | 
-                           SummaryFeaturesDefinition | MatchFeaturesDefinition | RankFeaturesDefinition |
-                           SecondPhaseDefinition | ConstantsDefinition |
-                           RankDefinition | RankTypeDefinition | MinHitsDefinition | NumSearchPartitionDefinition
+                           SummaryFeaturesDefinition | MatchFeaturesDefinition | RankFeaturesDefinition | 
+                           SecondPhaseDefinition | ConstantsDefinition | RankDefinition | RankTypeDefinition | 
+                           MinHitsDefinition | NumSearchPartitionDefinition | FieldWeightDefinition
 
 MatchPhaseDefinition ::= match-phase '{' MatchPhaseBody '}'
 MatchPhaseBody ::= MatchPhaseBodyOptions+
-MatchPhaseBodyOptions ::= (attribute ':' IdentifierVal (order ':' (ascending | descending))?) | (max-hits ':' INTEGER_REG)
-                          | DiversityDefinition // Does not support zero-or-one occurrences
+MatchPhaseBodyOptions ::= (attribute ':' IdentifierVal) | (order ':' (ascending | descending)) | (max-hits ':' ('-')? INTEGER_REG)
+                          | DiversityDefinition | (evaluation-point ':' ('-')? FLOAT_REG) |
+                          (pre-post-filter-tipping-point ':' ('-')? FLOAT_REG) // Does not support zero-or-one occurrences
 DiversityDefinition ::= diversity '{' DiversityBody '}'
 DiversityBody ::= DiversityBodyOptions*
-private DiversityBodyOptions ::= (attribute ':' IdentifierVal) | (min-groups ':' INTEGER_REG) | (cutoff-factor ':' FLOAT_REG) |
+private DiversityBodyOptions ::= (attribute ':' IdentifierVal) | (min-groups ':' ('-')? INTEGER_REG) | (cutoff-factor ':' ('-')? FLOAT_REG) |
                                  (cutoff-strategy ':' (strict | loose))
 
 private NumThreadsDefinition ::= num-threads-per-search ':' INTEGER_REG
-private TermwiseLimitDefinition ::= termwise-limit ':' (FLOAT_REG | INTEGER_REG)
-private MinHitsDefinition ::= min-hits-per-thread ':' INTEGER_REG
+private TermwiseLimitDefinition ::= termwise-limit ':' ('-')? (FLOAT_REG | INTEGER_REG)
+private MinHitsDefinition ::= min-hits-per-thread ':' ('-')? INTEGER_REG
 private NumSearchPartitionDefinition ::= num-search-partition ':' INTEGER_REG
+FieldWeightDefinition ::= weight IdentifierVal ':' INTEGER_REG
 FirstPhaseDefinition ::= first-phase '{' FirstPhaseBody '}'  { mixin="org.intellij.sdk.language.psi.impl.SdFirstPhaseDefinitionMixin" }
 FirstPhaseBody ::= FirstPhaseBodyOptions* // Does not support zero-or-one occurrences
-private FirstPhaseBodyOptions ::= (keep-rank-count ':' INTEGER_REG) | (rank-score-drop-limit ':' (FLOAT_REG | INTEGER_REG)) | ExpressionDefinition
+private FirstPhaseBodyOptions ::= (keep-rank-count ':' INTEGER_REG) | (rank-score-drop-limit ':' ('-')? (FLOAT_REG | INTEGER_REG)) | ExpressionDefinition
 
-ExpressionDefinition ::= expression ((':' RankingExpression) | ('{' RankingExpression* '}') |
-                                     (':' file ':' FilePath))
+ExpressionDefinition ::= expression ((':' RankingExpression) | ('{' RankingExpression* '}'))
 
 SecondPhaseDefinition ::= second-phase '{' SecondPhaseBody '}'
 SecondPhaseBody ::= SecondPhaseBodyOptions*
@@ -167,7 +165,7 @@ private SecondPhaseBodyOptions ::= (rerank-count ':' INTEGER_REG) | ExpressionDe
 
 RankPropertiesDefinition ::= rank-properties '{' RankPropertiesBody '}'
 RankPropertiesBody ::= (RankPropertiesKey ':' RankPropertiesValue)+
-RankPropertiesKey ::= (IdentifierWithDashVal | STRING_REG | '(' | ')' | '.' | ',')+ 
+RankPropertiesKey ::= (IdentifierWithDashVal | STRING_REG | '(' | ')' | '.' | ',' | '$' | INTEGER_REG)+ 
 RankPropertiesValue ::= (('-')? INTEGER_REG) | (('-')? FLOAT_REG) | WORD_REG | IdentifierVal | STRING_REG
 
 FunctionDefinition ::= (function | macro) inline? IdentifierVal '(' (ArgumentDefinition (',' ArgumentDefinition)*)? ')'
@@ -182,9 +180,9 @@ ArgumentDefinition ::= IdentifierVal
 
 SummaryFeaturesDefinition ::= summary-features ((':' RankFeature+) | ((inherits IdentifierVal)? '{' RankFeature* '}'))
 
-MatchFeaturesDefinition ::= match-features (':' RankFeature+) | ('{' RankFeature* '}')
+MatchFeaturesDefinition ::= match-features ((':' RankFeature+) | ('{' RankFeature* '}'))
 
-RankFeaturesDefinition ::= rank-features (':' RankFeature+) | ('{' RankFeature* '}')
+RankFeaturesDefinition ::= rank-features ((':' RankFeature+) | ('{' RankFeature* '}'))
 
 ConstantsDefinition ::= constants '{' (IdentifierVal ':' RankPropertiesValue)* '}'
 
@@ -234,14 +232,13 @@ DocumentFieldDefinition ::= field IdentifierVal type FieldTypeName '{' DocumentF
 
 DocumentFieldBody ::= DocumentFieldBodyOptions* // Does not support zero-or-one occurrences
 private DocumentFieldBodyOptions ::= StructFieldDefinition | MatchDefinition | IndexingDefinition | AttributeDefinition | 
-                             AliasDef | RankDefinition | IndexingRewriteState | QueryCommandDefinition | SummaryDefinition |
+                             AliasDefinition | RankDefinition | IndexingRewriteState | QueryCommandDefinition | SummaryDefinition |
                              BoldingDefinition | (id ':' INTEGER_REG) | IndexDefinition | (normalizing ':' IdentifierWithDashVal) | 
                              SortingDefinition | StemmingDefinition | (weight ':' INTEGER_REG) | WeightedSetDefinition |
-                             RankTypeDefinition | DictionaryDefinition
-
+                             RankTypeDefinition | DictionaryDefinition | SummaryToDefinition | body
 //***** Field's body elements ******//
 // Struct
-StructFieldDefinition ::= struct-field IdentifierVal '{' StructFieldBody '}'
+StructFieldDefinition ::= struct-field IdentifierVal ('.' IdentifierVal)? '{' StructFieldBody '}'
                           { mixin="org.intellij.sdk.language.psi.impl.SdNamedElementImpl"
                             implements=["org.intellij.sdk.language.psi.SdDeclaration" "org.intellij.sdk.language.psi.SdNamedElement"]
                           }
@@ -251,61 +248,66 @@ StructFieldBodyOptions ::= IndexingDefinition | AttributeDefinition | MatchDefin
                            StructFieldDefinition | SummaryDefinition            
 // Match
 MatchDefinition ::= match ((':' MatchProperty) | ('{' MatchProperty+ '}'))
-MatchProperty ::= text | exact | exact-terminator | word | prefix | cased | uncased | substring | suffix | max-length | 
-                  gram | gram-size
+MatchProperty ::= text | exact | (exact-terminator ':' STRING_REG) | word | prefix | cased | uncased | substring | 
+                  suffix | (max-length ':' INTEGER_REG) | gram | (gram-size ':' INTEGER_REG) | WordWrapper
 // Indexing
-IndexingDefinition ::= indexing (':' IndexingStatement) | ('{' IndexingStatement+ '}')
-IndexingStatement ::= IndexingStatementOptions (('|' IndexingStatementOptions)*) | ((';' IndexingStatementOptions)*)
+IndexingDefinition ::= indexing ((':' IndexingStatement) | ('{' IndexingStatement+ '}'))
+IndexingStatement ::= IndexingStatementOptions ((('|' | ';') IndexingStatementOptions)*)
                       // Does not support zero-or-one occurrences
-IndexingStatementOptions ::= summary | attribute | index | set_language
+IndexingStatementOptions ::= summary | attribute | index | set_language | lowercase | (input (IdentifierVal | IndexingStuff)+) | 
+                             ('{' IndexingStatementOptions '}') | IndexingStuff+
+private IndexingStuff ::= WordWrapper | INTEGER_REG | FLOAT_REG | STRING_REG | ('{' IndexingStatementOptions+ '}') |
+                          ':' | ('|' IndexingStatementOptions) | ';' | '.' | '(' | ')' | ARITHMETIC_OPERATOR | COMPARISON_OPERATOR
 // Attribute
-AttributeDefinition ::= attribute ((':' SimpleAttributeProperty) | ('{' (SimpleAttributeProperty | ComplexAttributeProperty)+ '}'))
-SimpleAttributeProperty ::= fast-search | fast-access | paged | mutable // Does not support zero-or-one occurrences
-ComplexAttributeProperty ::= AliasDef | SortingDefinition | DistanceMetricDef // Does not support zero-or-one occurrences
+AttributeDefinition ::= attribute ((':' SimpleAttributeProperty) | ('{' (ComplexAttributeProperty | SimpleAttributeProperty)+ '}'))
+SimpleAttributeProperty ::= fast-search | fast-access | paged | mutable | enable-bit-vectors | enable-only-bit-vector | WordWrapper // Does not support zero-or-one occurrences
+ComplexAttributeProperty ::= AliasDefinition | SortingDefinition | DistanceMetricDef // Does not support zero-or-one occurrences
 DistanceMetricDef ::= distance-metric ':' IdentifierWithDashVal
 // Alias
-AliasDef ::= alias IdentifierVal? ':' IdentifierWithDashVal
+AliasDefinition ::= alias (IdentifierWithDashVal ('.' IdentifierWithDashVal)*)? ':' IdentifierWithDashVal ('.' IdentifierWithDashVal)*
 // Stemming
 StemmingDefinition ::= stemming ':' IdentifierWithDashVal
 // Rank
 RankDefinition ::= rank ((IdentifierVal? ':' RankingSetting) | ('{' RankingSetting '}'))
-RankingSetting ::= filter | normal
+RankingSetting ::= filter | normal | literal | WordWrapper
 // Indexing Rewrite
 IndexingRewriteState ::= indexing-rewrite ':' none
 // Query Command
-QueryCommandDefinition ::= query-command ':' IdentifierVal | STRING_REG
+QueryCommandDefinition ::= query-command ':' (IdentifierVal | STRING_REG | WordWrapper)
 // Summary
-SummaryDefinition ::= summary ((':' SummaryBodyOptions) | (IdentifierWithDashVal? (type FieldTypeName)? '{' SummaryBody '}'))
+SummaryDefinition ::= summary IdentifierWithDashVal? (type FieldTypeName)? ((':' SummaryBodyOptions) | ( '{' SummaryBody '}'))
                       { mixin="org.intellij.sdk.language.psi.impl.SdSummaryDefinitionMixin" }
 SummaryBody ::= SummaryBodyOptions* // Does not support zero-or-one occurrences
-SummaryBodyOptions ::= full | dynamic | (source ':' IdentifierVal (',' IdentifierVal)*) |
-                       (to ':' IdentifierVal (',' IdentifierVal)*) | matched-elements-only
+SummaryBodyOptions ::= full | static | dynamic | (source ':' (IdentifierVal ('.' IdentifierVal)?) (',' IdentifierVal ('.' IdentifierVal)?)*) |
+                       (to ':' IdentifierVal (',' IdentifierVal)*) | matched-elements-only | BoldingDefinition
+// Summary To
+SummaryToDefinition ::= summary-to ':' WordWrapper (',' WordWrapper)*
 // Bolding                       
 BoldingDefinition ::= bolding ':' (on | off | true | false)
 // Index
-IndexDefinition ::= index IdentifierVal (':' IndexProperty) | ('{' IndexProperty* '}')
+IndexDefinition ::= index IdentifierVal? ((':' IndexProperty) | ('{' IndexProperty '}'))
 IndexProperty ::= IndexPropertyOptions*
 private IndexPropertyOptions ::= (alias ':' IdentifierWithDashVal) | StemmingDefinition | (arity ':' INTEGER_REG) | 
                                  (lower-bound ':' INTEGER_REG ('L')?) | (upper-bound ':' INTEGER_REG ('L')?) |
-                                 (dense-posting-list-threshold ':' FLOAT_REG) | enable-bm25 | HnswDefinition
+                                 (dense-posting-list-threshold ':' FLOAT_REG) | enable-bm25 | prefix | HnswDefinition
 HnswDefinition ::= hnsw '{' HnswBody '}'
 HnswBody ::= HnswBodyOptions*
 private HnswBodyOptions ::= (max-links-per-node ':' INTEGER_REG) | (neighbors-to-explore-at-insert ':' INTEGER_REG) |
                             (multi-threaded-indexing ':' (on | off | true | false))
 // Sorting
-SortingDefinition ::= sorting (':' SortingProperty) | ('{' SortingProperty* '}')
+SortingDefinition ::= sorting ((':' SortingProperty) | ('{' SortingProperty* '}'))
 SortingProperty ::= ascending | descending | (function ':' SortingFunction) | (strength ':' SortingStrength) | 
                     (locale ':' IdentifierWithDashVal)
 SortingFunction ::= uca | raw | lowercase
 SortingStrength ::= primary | secondary | tertiary | quaternary | identical
 // Rank Type
-RankTypeDefinition ::= rank-type IdentifierVal ':' IdentifierVal
+RankTypeDefinition ::= rank-type IdentifierVal? ':' IdentifierVal
 // Weighted Set
-WeightedSetDefinition ::= weightedset (':' WeightedSetProperty) | ('{' WeightedSetProperty* '}')  // Does not support 
+WeightedSetDefinition ::= weightedset ((':' WeightedSetProperty) | ('{' WeightedSetProperty* '}'))  // Does not support 
                                                                                                   // zero-or-one occurrences
 WeightedSetProperty ::=  create-if-nonexistent | remove-if-zero
 // Dictionary
-DictionaryDefinition ::= dictionary (':' DictionarySetting) | ('{' DictionarySetting* '}') 
+DictionaryDefinition ::= dictionary ((':' DictionarySetting) | ('{' DictionarySetting* '}')) 
 DictionarySetting ::= hash | btree | cased | uncased
 //***** End of Field's body elements ******//
 
@@ -313,6 +315,8 @@ DictionarySetting ::= hash | btree | cased | uncased
 //---- Util rules -----
 //---------------------
 
+private WordWrapper ::= KeywordOrIdentifier | KeywordNotIdentifier | ID_REG | ID_WITH_DASH_REG | WORD_REG
+
 IdentifierVal ::= KeywordOrIdentifier | ID_REG { mixin="org.intellij.sdk.language.psi.impl.SdIdentifierMixin"
                                                  implements=["org.intellij.sdk.language.psi.SdIdentifier"]
                                                }
@@ -328,12 +332,12 @@ KeywordOrIdentifier ::= schema | search | document | struct | field | type | ind
                         order | ascending | descending | diversity | constants | expression | weight | match |
                         function | macro | inline | text | exact | word | prefix | cased | uncased | substring | suffix | 
                         gram | paged | mutable | alias | sorting | strength | locale | uca | lowercase |
-                        primary | secondary | tertiary | quaternary | identical | rank | filter | normal | none | full | dynamic | 
-                        source | to | strict | loose |
+                        primary | secondary | tertiary | quaternary | identical | rank | filter | normal | literal | 
+                        none | full | dynamic | source | to | strict | loose |
                         bolding | on | off | true | false | id | normalizing | stemming | arity | hnsw | dictionary | hash | btree |
                         fieldset | fields | constant | annotation
-                        | attribute | body | header | index | 
-                        reference | summary | set_language
+                        | attribute | body | header | index | static |
+                        reference | summary | set_language | model
 
 // Note- in this form, those keywords can't be use as identifier-with-dash!
 KeywordNotIdentifier ::= struct-field | document-summary | omit-summary-features | from-disk | rank-profile | rank-type |
@@ -345,5 +349,6 @@ KeywordNotIdentifier ::= struct-field | document-summary | omit-summary-features
                          indexing-rewrite | query-command | matched-elements-only | lower-bound | upper-bound | 
                          dense-posting-list-threshold | enable-bm25 | max-links-per-node | neighbors-to-explore-at-insert | 
                          multi-threaded-indexing | create-if-nonexistent | remove-if-zero | raw-as-base64-in-summary |
-                         onnx-model | cutoff-factor | cutoff-strategy | on-match | on-rank | on-summary
+                         onnx-model | cutoff-factor | cutoff-strategy | on-match | on-rank | on-summary | enable-bit-vectors |
+                         enable-only-bit-vector | summary-to | evaluation-point | pre-post-filter-tipping-point
                          
\ No newline at end of file