diff options
author | Jon Bratseth <bratseth@oath.com> | 2021-11-04 19:01:44 +0100 |
---|---|---|
committer | GitHub <noreply@github.com> | 2021-11-04 19:01:44 +0100 |
commit | acd8d2349d11396160c2fb2cb502375e6c6429eb (patch) | |
tree | f2db1dda7f8b988b300ca908f83c4566aaf17e94 /sd-plugin | |
parent | e9bf7a444158624c88448459a407b4e1e4beb97e (diff) | |
parent | d2eebf3db9954dfba6cd593dcffbee55147af783 (diff) |
Merge branch 'master' into arnej/extend-intellij-plugin
Diffstat (limited to 'sd-plugin')
-rw-r--r-- | sd-plugin/BACKLOG.md | 34 | ||||
-rw-r--r-- | sd-plugin/README.md | 45 | ||||
-rw-r--r-- | sd-plugin/build.gradle | 2 | ||||
-rw-r--r-- | sd-plugin/src/main/java/org/intellij/sdk/language/SdSyntaxHighlighter.java | 1 | ||||
-rw-r--r-- | sd-plugin/src/main/java/org/intellij/sdk/language/lexer/sd.flex | 17 | ||||
-rw-r--r-- | sd-plugin/src/main/java/org/intellij/sdk/language/parser/sd.bnf | 135 |
6 files changed, 151 insertions, 83 deletions
diff --git a/sd-plugin/BACKLOG.md b/sd-plugin/BACKLOG.md new file mode 100644 index 00000000000..19e185dcc6b --- /dev/null +++ b/sd-plugin/BACKLOG.md @@ -0,0 +1,34 @@ +### Open Issues + +1. In some cases, the grammar prefers not to enforce bad syntax, because if the parser encounters bad syntax it stops +and can't build the PSI tree. That means none of the features will work for a file like that. For example, in cases +where an element supposes to have zero-to-one occurrences, the grammar will treat it as zero-to-many. +2. In order to enable the grammar recognize some keywords as identifiers (e.g. "filter" as a field's name), the +identifier rule (named "IdentifierVal") wraps the regex (ID_REG) and the KeywordOrIdentifier rule (which contains all +the keywords in the language). +3. The implementation of the GoTo Declaration feature is not exactly the same as IntelliJ. In IntelliJ if a reference +has several declarations, after clicking "Goto Declaration" there is a little window with all the declarations to choose +from. It can be done by changing the method "multiResolve" in SdReference.java to return more than one declaration. The +problem with that is that it causes the "Find Usages" feature to not work. For now, I decided to make the plugin +"Goto Declaration" feature show only the most specific declaration by the right rank-profile scope. +4. The "Find Usages" window can group usages only under rank-profiles and document-summaries. Other usages appear +directly under the .sd file. In order to create another group type of usages' group, you'll need to create 2 classes: +one for the extension "fileStructureGroupRuleProvider" (e.g. SdRankProfileGroupingRuleProvider.java), and one for the +grouping rule itself (e.g. SdRankProfileGroupingRule.java). +Another open problem is that the navigation isn't working in the current grouping rules. It means that when clicking on +the group headline (e.g. some name of a rank-profile) the IDE doesn't "jump" to the matching declaration. +5. Goto declaration doesn't work for document's inherits. e.g. if document A inherits from document B, B doesn't have a +reference to its declaration. +6. There aren't any tests for the plugin. + +### Some useful links: +1. JetBrains official tutorials: https://plugins.jetbrains.com/docs/intellij/custom-language-support.html and +https://plugins.jetbrains.com/docs/intellij/custom-language-support-tutorial.html +2. Grammar-Kit HOWTO: Helps to understand the BNF syntax. + https://github.com/JetBrains/Grammar-Kit/blob/master/HOWTO.md +3. How to deal with left-recursion in the grammar (in SD for example it happens in expressions). Last answer here: +https://intellij-support.jetbrains.com/hc/en-us/community/posts/360001258300-What-s-the-alternative-to-left-recursion-in-GrammarKit- +4. Great tutorial for a custom-language-plugin, but only for the basics (mainly the parser and lexer): + https://medium.com/@shan1024/custom-language-plugin-development-for-intellij-idea-part-01-d6a41ab96bc9 +5. Code of Dart (some custom language) plugin for IntelliJ: +https://github.com/JetBrains/intellij-plugins/tree/0f07ca63355d5530b441ca566c98f17c560e77f8/Dart
\ No newline at end of file diff --git a/sd-plugin/README.md b/sd-plugin/README.md index 74ca1781596..1cab356f78e 100644 --- a/sd-plugin/README.md +++ b/sd-plugin/README.md @@ -1,25 +1,42 @@ <!-- Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. --> -This directory holds the code for an IntteliJ plugin for reading SD files. +This directory holds the code for an IntelliJ plugin for reading SD files. -NOTE: This is the source code, not the plugin itself. In order to be able to use the plugin you'll need to download it from JetBrains Marketplace or create a zip file and load it to IntelliJ (details later). +NOTE: This is the source code, not the plugin itself. In order to be able to use the plugin you'll need to download it +from JetBrains Marketplace or create a zip file and load it to IntelliJ by choosing "Install Plugin from Disk". -Before cloning, you should download Gradle and create a Gradle project. -You should also download Grammar-Kit plugin from the Marketplace. +Before starting, you should: +1. Download Gradle 7 (if you don't have it already). +2. Make sure that the bundled Plugin DevKit plugin is enabled (inside IntelliJ). +3. Optional- Download Grammar-Kit plugin from JetBrains Marketplace (inside IntelliJ). It helps with reading the .bnf file. +4. Optional- Download PsiViewer plugin from JetBrains Marketplace (inside IntelliJ). It helps to test the grammar defined +in the .bnf file. +### Working Process The grammar is defined in 2 files: - sd.bnf - sd.flex -After cloning, you should: -1. Right-click the sd.bnf file and press "Generate Parser Code" -2. Right-click the sd.flex file and press "Run JFlex Generator" - -Now you should have a "gen" folder next to the "java" folder, and it contains all the parser and lexer code. +In order to generate the lexer and parser's code, you need to run in the command line: -Important note! After any change in one of this 2 files (bnf, flex) you'll need to generate again. The proper way is to delete the "gen" folder and then do 1-2 again. + ./gradlew generateSdParser + ./gradlew generateSdLexer -Now, you can run the gradle task "intellij/runIde", open a project with some sd file and see how the plugin works on it. +You should now have a "gen" folder next to the "java" folder, and it contains all the parser and lexer code. -In order to test the plugin locally (on you IDE, not by running the gradle task "runIde"), you can run the gradle task -"intellij/buildPlugin". It would create a zip file in the directory build\distributions. You can load it to IntelliJ by -clicking the "settings" in preferences/Plugins and click "Install Plugin from disk".
\ No newline at end of file +NOTE- Running those tasks would reset the "gen" folder, and all the previous generated files would be deleted before the +new ones would be generated. + +Now, you can run the gradle task "intellij/runIde" (or "./gradlew runIde" in the command line), open a project with some sd file and see how the plugin works on it. + +### Build the Plugin +In order to build the plugin and create a zip file from it, you should run the command: + + ./gradlew buildPlugin + +Or since it's a default task you can just run: + + ./gradlew + +This task also invokes the tasks generateSdParser and generateSdLexer as a part of the building process. + +Now, you'll have a zip file in the directory build\distributions. diff --git a/sd-plugin/build.gradle b/sd-plugin/build.gradle index 0a4798032d9..e5b39769b96 100644 --- a/sd-plugin/build.gradle +++ b/sd-plugin/build.gradle @@ -38,7 +38,7 @@ compileJava { group 'org.yahoo.native' -version '1.0.1' +version '1.0.2' sourceCompatibility = 11 diff --git a/sd-plugin/src/main/java/org/intellij/sdk/language/SdSyntaxHighlighter.java b/sd-plugin/src/main/java/org/intellij/sdk/language/SdSyntaxHighlighter.java index 46cbffe0aa8..e6452685e3c 100644 --- a/sd-plugin/src/main/java/org/intellij/sdk/language/SdSyntaxHighlighter.java +++ b/sd-plugin/src/main/java/org/intellij/sdk/language/SdSyntaxHighlighter.java @@ -173,6 +173,7 @@ public class SdSyntaxHighlighter extends SyntaxHighlighterBase { constants.add(SdTypes.SUMMARY); constants.add(SdTypes.INDEX); constants.add(SdTypes.SET_LANGUAGE); + constants.add(SdTypes.LOWERCASE); constants.add(SdTypes.FAST_SEARCH); constants.add(SdTypes.FAST_ACCESS); constants.add(SdTypes.PAGED); diff --git a/sd-plugin/src/main/java/org/intellij/sdk/language/lexer/sd.flex b/sd-plugin/src/main/java/org/intellij/sdk/language/lexer/sd.flex index 5368551b81b..585a6672189 100644 --- a/sd-plugin/src/main/java/org/intellij/sdk/language/lexer/sd.flex +++ b/sd-plugin/src/main/java/org/intellij/sdk/language/lexer/sd.flex @@ -32,12 +32,12 @@ ID_WITH_DASH = [a-zA-Z_][a-zA-Z0-9_-]* WHITE_SPACE=[ \t\n\x0B\f\r]+ COMMENT=#.* -SYMBOL= [|:{}(),.\[\]] +SYMBOL= [!$|:{}(),.\[\]] INTEGER = [0-9]+ FLOAT = {INTEGER}[.][0-9]+[e]? COMPARISON_OPERATOR = [<>]|(==)|(<=)|(>=)|(\~=) ARITHMETIC_OPERATOR = [\-+*/] -STRING = [\"][^\"\n]*[\"] +STRING = \"([^\"\\]*(\\.[^\"\\]*)*)\" WORD = \w+ @@ -109,17 +109,21 @@ WORD = \w+ "rank" { return RANK; } "filter" { return FILTER; } "normal" { return NORMAL; } + "literal" { return LITERAL; } "indexing-rewrite" { return INDEXING_REWRITE; } "none" { return NONE; } "query-command" { return QUERY_COMMAND; } "full" { return FULL; } - "dinamic" { return DYNAMIC; } + "static" { return STATIC; } + "dynamic" { return DYNAMIC; } "source" { return SOURCE; } "to" { return TO; } "matched-elements-only" { return MATCHED_ELEMENTS_ONLY; } "input" { return INPUT; } "mutable" { return MUTABLE; } + "enable-bit-vectors" { return ENABLE_BIT_VECTORS; } + "enable-only-bit-vector" { return ENABLE_ONLY_BIT_VECTOR; } "document-summary" { return DOCUMENT_SUMMARY; } "from-disk" { return FROM_DISK; } "omit-summary-features" { return OMIT_SUMMARY_FEATURES; } @@ -127,10 +131,12 @@ WORD = \w+ "as" { return AS; } "rank-profile" { return RANK_PROFILE; } + "model" { return MODEL; } "match-phase" { return MATCH_PHASE; } "order" { return ORDER; } "ascending" { return ASCENDING; } "descending" { return DESCENDING; } + "locale" { return LOCALE; } "max-hits" { return MAX_HITS; } "diversity" { return DIVERSITY; } "min-groups" { return MIN_GROUPS; } @@ -154,6 +160,7 @@ WORD = \w+ "constants" { return CONSTANTS; } "second-phase" { return SECOND_PHASE; } "rerank-count" { return RERANK_COUNT; } + "rank-features" { return RANK_FEATURES; } "weight" { return WEIGHT; } "index" { return INDEX; } @@ -203,6 +210,10 @@ WORD = \w+ "body" { return BODY; } "header" { return HEADER; } + "summary-to" { return SUMMARY_TO; } + + "evaluation-point" { return EVALUATION_POINT; } + "pre-post-filter-tipping-point" { return PRE_POST_FILTER_TIPPING_POINT; } // In here, we check for character sequences which matches regular expressions defined above. {ID} { return ID_REG; } diff --git a/sd-plugin/src/main/java/org/intellij/sdk/language/parser/sd.bnf b/sd-plugin/src/main/java/org/intellij/sdk/language/parser/sd.bnf index 243e177dbdd..67bdca7d2dc 100644 --- a/sd-plugin/src/main/java/org/intellij/sdk/language/parser/sd.bnf +++ b/sd-plugin/src/main/java/org/intellij/sdk/language/parser/sd.bnf @@ -28,21 +28,21 @@ NOTE: This grammar does not enforce zero-or-one occurrences of elements (treats ID_WITH_DASH_REG = 'regexp:[a-zA-Z_][a-zA-Z0-9_-]*' WHITE_SPACE = 'regexp:\s+' COMMENT = 'regexp:#.*' - SYMBOL = 'regexp:[|:{}(),.\[\]]' + SYMBOL = 'regexp:[!$|:{}(),.\[\]]' COMPARISON_OPERATOR = 'regexp:[<>]|(==)|(<=)|(>=)|(~=)' ARITHMETIC_OPERATOR = 'regexp:[\-+*/]' INTEGER_REG = 'regexp:[0-9]+' FLOAT_REG = 'regexp:[0-9]+[.][0-9]+[e]?' - STRING_REG = 'regexp:[\"][^\"]*[\"]' + STRING_REG = 'regexp:\"([^\"\\]*(\\.[^\"\\]*)*)\"' WORD_REG = 'regexp:\w+' ] } -SdFile ::= SchemaDefinition -SchemaDefinition ::= (search | schema) IdentifierVal? '{' SchemaBody '}' +SdFile ::= SchemaDefinition | DocumentDefinition +SchemaDefinition ::= (search | schema) IdentifierVal? (inherits IdentifierVal)? '{' SchemaBody '}' SchemaBody ::= SchemaBodyOptions* DocumentDefinition SchemaBodyOptions* // Does not support zero-or-one occurrences private SchemaBodyOptions ::= SchemaFieldDefinition | ImportFieldDefinition | DocumentSummaryDefinition | - RankProfileDefinition | + RankProfileDefinition | IndexDefinition | FieldSetDefinition | ConstantDefinition | OnnxModelDefinition | StemmingDefinition | raw-as-base64-in-summary | SchemaAnnotationDefinition @@ -54,18 +54,13 @@ SchemaFieldDefinition ::= field IdentifierVal type FieldTypeName '{' SchemaField FieldTypeName ::= ("array" '<' (FieldTypeName | IdentifierVal) '>') | ("weightedset" '<' SingleValueFieldTypeName '>') | ("map" '<' (FieldTypeName | IdentifierVal) ',' (FieldTypeName | IdentifierVal) '>') | TensorType | - SingleValueFieldTypeName + (SingleValueFieldTypeName '[' ']') | SingleValueFieldTypeName private SingleValueFieldTypeName ::= "string" | "int" | "long" | "bool" | "byte" | "float" | "double" | "position" | "predicate" | "raw" | "uri" | "reference" '<' IdentifierVal '>' | "annotationreference" '<' IdentifierVal '>' | IdentifierVal -private TensorType ::= "tensor" '<' ("float" | "double" | "int8" | "bfloat16") '>' '(' TensorDimension (',' TensorDimension)* ')' -private TensorDimension ::= WORD_REG ('{' '}') | ('[' INTEGER_REG ']') +private TensorType ::= "tensor" ('<' ("float" | "double" | "int8" | "bfloat16") '>')? '(' TensorDimension (',' TensorDimension)* ')' +private TensorDimension ::= WordWrapper (('{' '}') | ('[' INTEGER_REG ']')) -SchemaFieldBody ::= SchemaFieldBodyOptions* // Does not support zero-or-one occurrences -SchemaFieldBodyOptions ::= SchemaFieldIndexingDefinition | AttributeDefinition | RankDefinition | IndexingRewriteState | - MatchDefinition | StructFieldDefinition | QueryCommandDefinition - -SchemaFieldIndexingDefinition ::= indexing (':' SchemaFieldIndexingStatement) | ('{' SchemaFieldIndexingStatement+ '}') -SchemaFieldIndexingStatement ::= (input IdentifierVal) | IndexingStatement +SchemaFieldBody ::= DocumentFieldBodyOptions* // Fields of schemas and documents defined the same way here DocumentSummaryDefinition ::= document-summary IdentifierWithDashVal (inherits IdentifierWithDashVal)? '{' DocumentSummaryBody '}' { mixin="org.intellij.sdk.language.psi.impl.SdNamedElementImpl" @@ -86,7 +81,7 @@ private FieldSetBodyOptions ::= (fields ':' IdentifierVal (',' IdentifierVal)*) ConstantDefinition ::= constant IdentifierVal '{' ConstantBody '}' ConstantBody ::= ConstantBodyOptions* private ConstantBodyOptions ::= (file ':' FilePath) | (uri ':' UriPath) | (type ':' TensorType) -private FilePath ::= (IdentifierVal | WORD_REG) ('.' | '/' | IdentifierWithDashVal | WORD_REG)+ +private FilePath ::= WordWrapper (('.' | '/') WordWrapper)+ private UriPath ::= ('H'|'h') ('T'|'t') ('T'|'t') ('P'|'p') ('S'|'s')? ':' ('//')? (IdentifierWithDashVal | '.' | '/' | ':')+ @@ -100,7 +95,7 @@ SchemaAnnotationDefinition ::= AnnotationDefinition implements=["org.intellij.sdk.language.psi.SdDeclaration" "org.intellij.sdk.language.psi.SdNamedElement"] } -private AnnotationDefinition ::= annotation IdentifierVal '{' AnnotationFieldDefinition* '}' +private AnnotationDefinition ::= annotation IdentifierVal (inherits IdentifierVal)? '{' AnnotationFieldDefinition* '}' AnnotationFieldDefinition ::= field IdentifierVal type FieldTypeName '{' '}' { mixin="org.intellij.sdk.language.psi.impl.SdNamedElementImpl" implements=["org.intellij.sdk.language.psi.SdDeclaration" "org.intellij.sdk.language.psi.SdNamedElement"] @@ -109,11 +104,13 @@ AnnotationFieldDefinition ::= field IdentifierVal type FieldTypeName '{' '}' //------------------------- //--- Expressions rules --- //------------------------- -RankingExpression ::= ParenthesisedExpr | BooleanExpr |ArithmeticExpr | IfFunctionExpr | - QueryDefinitionExpr | FunctionCallExpr | PrimitiveExpr +RankingExpression ::= FilePathExpr | ParenthesisedExpr | BooleanExpr | ArithmeticExpr | IfFunctionExpr | + QueryDefinitionExpr | FunctionCallExpr | InListRankingExpr | PrimitiveExpr -IfFunctionExpr ::= "if" '(' (InListRankingExpression | RankingExpression) ',' RankingExpression ',' RankingExpression ')' -InListRankingExpression ::= RankingExpression "in" '[' RankingExpression (',' RankingExpression)* ']' +FilePathExpr ::= file ':' (FilePath | WordWrapper) + +IfFunctionExpr ::= "if" '(' (InListRankingExpr | RankingExpression) ',' RankingExpression ',' RankingExpression ')' +InListRankingExpr ::= RankingExpression "in" '[' RankingExpression (',' RankingExpression)* ']' BooleanExpr ::= RankingExpression COMPARISON_OPERATOR RankingExpression @@ -125,41 +122,42 @@ FunctionCallExpr ::= IdentifierWithDashVal '(' RankingExpression (',' RankingExp ParenthesisedExpr ::= '(' RankingExpression ')' -PrimitiveExpr ::= (('-')? INTEGER_REG) | (('-')? FLOAT_REG) | IdentifierVal | RankFeature +PrimitiveExpr ::= (('-')? INTEGER_REG) | (('-')? FLOAT_REG) | IdentifierVal | RankFeature | STRING_REG //------------------------- //-- Rank Profile rules --- //------------------------- -RankProfileDefinition ::= rank-profile IdentifierWithDashVal (inherits IdentifierWithDashVal)? '{' RankProfileBody '}' +RankProfileDefinition ::= (rank-profile | model) IdentifierWithDashVal (inherits IdentifierWithDashVal)? '{' RankProfileBody '}' { mixin="org.intellij.sdk.language.psi.impl.SdNamedElementImpl" implements=["org.intellij.sdk.language.psi.SdDeclaration"] } private RankProfileBody ::= RankProfileBodyOptions* // Does not support zero-or-one occurrences private RankProfileBodyOptions ::= MatchPhaseDefinition | NumThreadsDefinition | FunctionDefinition | TermwiseLimitDefinition | ignore-default-rank-features | RankPropertiesDefinition | FirstPhaseDefinition | - SummaryFeaturesDefinition | MatchFeaturesDefinition | RankFeaturesDefinition | - SecondPhaseDefinition | ConstantsDefinition | - RankDefinition | RankTypeDefinition | MinHitsDefinition | NumSearchPartitionDefinition + SummaryFeaturesDefinition | MatchFeaturesDefinition | RankFeaturesDefinition | + SecondPhaseDefinition | ConstantsDefinition | RankDefinition | RankTypeDefinition | + MinHitsDefinition | NumSearchPartitionDefinition | FieldWeightDefinition MatchPhaseDefinition ::= match-phase '{' MatchPhaseBody '}' MatchPhaseBody ::= MatchPhaseBodyOptions+ -MatchPhaseBodyOptions ::= (attribute ':' IdentifierVal (order ':' (ascending | descending))?) | (max-hits ':' INTEGER_REG) - | DiversityDefinition // Does not support zero-or-one occurrences +MatchPhaseBodyOptions ::= (attribute ':' IdentifierVal) | (order ':' (ascending | descending)) | (max-hits ':' ('-')? INTEGER_REG) + | DiversityDefinition | (evaluation-point ':' ('-')? FLOAT_REG) | + (pre-post-filter-tipping-point ':' ('-')? FLOAT_REG) // Does not support zero-or-one occurrences DiversityDefinition ::= diversity '{' DiversityBody '}' DiversityBody ::= DiversityBodyOptions* -private DiversityBodyOptions ::= (attribute ':' IdentifierVal) | (min-groups ':' INTEGER_REG) | (cutoff-factor ':' FLOAT_REG) | +private DiversityBodyOptions ::= (attribute ':' IdentifierVal) | (min-groups ':' ('-')? INTEGER_REG) | (cutoff-factor ':' ('-')? FLOAT_REG) | (cutoff-strategy ':' (strict | loose)) private NumThreadsDefinition ::= num-threads-per-search ':' INTEGER_REG -private TermwiseLimitDefinition ::= termwise-limit ':' (FLOAT_REG | INTEGER_REG) -private MinHitsDefinition ::= min-hits-per-thread ':' INTEGER_REG +private TermwiseLimitDefinition ::= termwise-limit ':' ('-')? (FLOAT_REG | INTEGER_REG) +private MinHitsDefinition ::= min-hits-per-thread ':' ('-')? INTEGER_REG private NumSearchPartitionDefinition ::= num-search-partition ':' INTEGER_REG +FieldWeightDefinition ::= weight IdentifierVal ':' INTEGER_REG FirstPhaseDefinition ::= first-phase '{' FirstPhaseBody '}' { mixin="org.intellij.sdk.language.psi.impl.SdFirstPhaseDefinitionMixin" } FirstPhaseBody ::= FirstPhaseBodyOptions* // Does not support zero-or-one occurrences -private FirstPhaseBodyOptions ::= (keep-rank-count ':' INTEGER_REG) | (rank-score-drop-limit ':' (FLOAT_REG | INTEGER_REG)) | ExpressionDefinition +private FirstPhaseBodyOptions ::= (keep-rank-count ':' INTEGER_REG) | (rank-score-drop-limit ':' ('-')? (FLOAT_REG | INTEGER_REG)) | ExpressionDefinition -ExpressionDefinition ::= expression ((':' RankingExpression) | ('{' RankingExpression* '}') | - (':' file ':' FilePath)) +ExpressionDefinition ::= expression ((':' RankingExpression) | ('{' RankingExpression* '}')) SecondPhaseDefinition ::= second-phase '{' SecondPhaseBody '}' SecondPhaseBody ::= SecondPhaseBodyOptions* @@ -167,7 +165,7 @@ private SecondPhaseBodyOptions ::= (rerank-count ':' INTEGER_REG) | ExpressionDe RankPropertiesDefinition ::= rank-properties '{' RankPropertiesBody '}' RankPropertiesBody ::= (RankPropertiesKey ':' RankPropertiesValue)+ -RankPropertiesKey ::= (IdentifierWithDashVal | STRING_REG | '(' | ')' | '.' | ',')+ +RankPropertiesKey ::= (IdentifierWithDashVal | STRING_REG | '(' | ')' | '.' | ',' | '$' | INTEGER_REG)+ RankPropertiesValue ::= (('-')? INTEGER_REG) | (('-')? FLOAT_REG) | WORD_REG | IdentifierVal | STRING_REG FunctionDefinition ::= (function | macro) inline? IdentifierVal '(' (ArgumentDefinition (',' ArgumentDefinition)*)? ')' @@ -182,9 +180,9 @@ ArgumentDefinition ::= IdentifierVal SummaryFeaturesDefinition ::= summary-features ((':' RankFeature+) | ((inherits IdentifierVal)? '{' RankFeature* '}')) -MatchFeaturesDefinition ::= match-features (':' RankFeature+) | ('{' RankFeature* '}') +MatchFeaturesDefinition ::= match-features ((':' RankFeature+) | ('{' RankFeature* '}')) -RankFeaturesDefinition ::= rank-features (':' RankFeature+) | ('{' RankFeature* '}') +RankFeaturesDefinition ::= rank-features ((':' RankFeature+) | ('{' RankFeature* '}')) ConstantsDefinition ::= constants '{' (IdentifierVal ':' RankPropertiesValue)* '}' @@ -234,14 +232,13 @@ DocumentFieldDefinition ::= field IdentifierVal type FieldTypeName '{' DocumentF DocumentFieldBody ::= DocumentFieldBodyOptions* // Does not support zero-or-one occurrences private DocumentFieldBodyOptions ::= StructFieldDefinition | MatchDefinition | IndexingDefinition | AttributeDefinition | - AliasDef | RankDefinition | IndexingRewriteState | QueryCommandDefinition | SummaryDefinition | + AliasDefinition | RankDefinition | IndexingRewriteState | QueryCommandDefinition | SummaryDefinition | BoldingDefinition | (id ':' INTEGER_REG) | IndexDefinition | (normalizing ':' IdentifierWithDashVal) | SortingDefinition | StemmingDefinition | (weight ':' INTEGER_REG) | WeightedSetDefinition | - RankTypeDefinition | DictionaryDefinition - + RankTypeDefinition | DictionaryDefinition | SummaryToDefinition | body //***** Field's body elements ******// // Struct -StructFieldDefinition ::= struct-field IdentifierVal '{' StructFieldBody '}' +StructFieldDefinition ::= struct-field IdentifierVal ('.' IdentifierVal)? '{' StructFieldBody '}' { mixin="org.intellij.sdk.language.psi.impl.SdNamedElementImpl" implements=["org.intellij.sdk.language.psi.SdDeclaration" "org.intellij.sdk.language.psi.SdNamedElement"] } @@ -251,61 +248,66 @@ StructFieldBodyOptions ::= IndexingDefinition | AttributeDefinition | MatchDefin StructFieldDefinition | SummaryDefinition // Match MatchDefinition ::= match ((':' MatchProperty) | ('{' MatchProperty+ '}')) -MatchProperty ::= text | exact | exact-terminator | word | prefix | cased | uncased | substring | suffix | max-length | - gram | gram-size +MatchProperty ::= text | exact | (exact-terminator ':' STRING_REG) | word | prefix | cased | uncased | substring | + suffix | (max-length ':' INTEGER_REG) | gram | (gram-size ':' INTEGER_REG) | WordWrapper // Indexing -IndexingDefinition ::= indexing (':' IndexingStatement) | ('{' IndexingStatement+ '}') -IndexingStatement ::= IndexingStatementOptions (('|' IndexingStatementOptions)*) | ((';' IndexingStatementOptions)*) +IndexingDefinition ::= indexing ((':' IndexingStatement) | ('{' IndexingStatement+ '}')) +IndexingStatement ::= IndexingStatementOptions ((('|' | ';') IndexingStatementOptions)*) // Does not support zero-or-one occurrences -IndexingStatementOptions ::= summary | attribute | index | set_language +IndexingStatementOptions ::= summary | attribute | index | set_language | lowercase | (input (IdentifierVal | IndexingStuff)+) | + ('{' IndexingStatementOptions '}') | IndexingStuff+ +private IndexingStuff ::= WordWrapper | INTEGER_REG | FLOAT_REG | STRING_REG | ('{' IndexingStatementOptions+ '}') | + ':' | ('|' IndexingStatementOptions) | ';' | '.' | '(' | ')' | ARITHMETIC_OPERATOR | COMPARISON_OPERATOR // Attribute -AttributeDefinition ::= attribute ((':' SimpleAttributeProperty) | ('{' (SimpleAttributeProperty | ComplexAttributeProperty)+ '}')) -SimpleAttributeProperty ::= fast-search | fast-access | paged | mutable // Does not support zero-or-one occurrences -ComplexAttributeProperty ::= AliasDef | SortingDefinition | DistanceMetricDef // Does not support zero-or-one occurrences +AttributeDefinition ::= attribute ((':' SimpleAttributeProperty) | ('{' (ComplexAttributeProperty | SimpleAttributeProperty)+ '}')) +SimpleAttributeProperty ::= fast-search | fast-access | paged | mutable | enable-bit-vectors | enable-only-bit-vector | WordWrapper // Does not support zero-or-one occurrences +ComplexAttributeProperty ::= AliasDefinition | SortingDefinition | DistanceMetricDef // Does not support zero-or-one occurrences DistanceMetricDef ::= distance-metric ':' IdentifierWithDashVal // Alias -AliasDef ::= alias IdentifierVal? ':' IdentifierWithDashVal +AliasDefinition ::= alias (IdentifierWithDashVal ('.' IdentifierWithDashVal)*)? ':' IdentifierWithDashVal ('.' IdentifierWithDashVal)* // Stemming StemmingDefinition ::= stemming ':' IdentifierWithDashVal // Rank RankDefinition ::= rank ((IdentifierVal? ':' RankingSetting) | ('{' RankingSetting '}')) -RankingSetting ::= filter | normal +RankingSetting ::= filter | normal | literal | WordWrapper // Indexing Rewrite IndexingRewriteState ::= indexing-rewrite ':' none // Query Command -QueryCommandDefinition ::= query-command ':' IdentifierVal | STRING_REG +QueryCommandDefinition ::= query-command ':' (IdentifierVal | STRING_REG | WordWrapper) // Summary -SummaryDefinition ::= summary ((':' SummaryBodyOptions) | (IdentifierWithDashVal? (type FieldTypeName)? '{' SummaryBody '}')) +SummaryDefinition ::= summary IdentifierWithDashVal? (type FieldTypeName)? ((':' SummaryBodyOptions) | ( '{' SummaryBody '}')) { mixin="org.intellij.sdk.language.psi.impl.SdSummaryDefinitionMixin" } SummaryBody ::= SummaryBodyOptions* // Does not support zero-or-one occurrences -SummaryBodyOptions ::= full | dynamic | (source ':' IdentifierVal (',' IdentifierVal)*) | - (to ':' IdentifierVal (',' IdentifierVal)*) | matched-elements-only +SummaryBodyOptions ::= full | static | dynamic | (source ':' (IdentifierVal ('.' IdentifierVal)?) (',' IdentifierVal ('.' IdentifierVal)?)*) | + (to ':' IdentifierVal (',' IdentifierVal)*) | matched-elements-only | BoldingDefinition +// Summary To +SummaryToDefinition ::= summary-to ':' WordWrapper (',' WordWrapper)* // Bolding BoldingDefinition ::= bolding ':' (on | off | true | false) // Index -IndexDefinition ::= index IdentifierVal (':' IndexProperty) | ('{' IndexProperty* '}') +IndexDefinition ::= index IdentifierVal? ((':' IndexProperty) | ('{' IndexProperty '}')) IndexProperty ::= IndexPropertyOptions* private IndexPropertyOptions ::= (alias ':' IdentifierWithDashVal) | StemmingDefinition | (arity ':' INTEGER_REG) | (lower-bound ':' INTEGER_REG ('L')?) | (upper-bound ':' INTEGER_REG ('L')?) | - (dense-posting-list-threshold ':' FLOAT_REG) | enable-bm25 | HnswDefinition + (dense-posting-list-threshold ':' FLOAT_REG) | enable-bm25 | prefix | HnswDefinition HnswDefinition ::= hnsw '{' HnswBody '}' HnswBody ::= HnswBodyOptions* private HnswBodyOptions ::= (max-links-per-node ':' INTEGER_REG) | (neighbors-to-explore-at-insert ':' INTEGER_REG) | (multi-threaded-indexing ':' (on | off | true | false)) // Sorting -SortingDefinition ::= sorting (':' SortingProperty) | ('{' SortingProperty* '}') +SortingDefinition ::= sorting ((':' SortingProperty) | ('{' SortingProperty* '}')) SortingProperty ::= ascending | descending | (function ':' SortingFunction) | (strength ':' SortingStrength) | (locale ':' IdentifierWithDashVal) SortingFunction ::= uca | raw | lowercase SortingStrength ::= primary | secondary | tertiary | quaternary | identical // Rank Type -RankTypeDefinition ::= rank-type IdentifierVal ':' IdentifierVal +RankTypeDefinition ::= rank-type IdentifierVal? ':' IdentifierVal // Weighted Set -WeightedSetDefinition ::= weightedset (':' WeightedSetProperty) | ('{' WeightedSetProperty* '}') // Does not support +WeightedSetDefinition ::= weightedset ((':' WeightedSetProperty) | ('{' WeightedSetProperty* '}')) // Does not support // zero-or-one occurrences WeightedSetProperty ::= create-if-nonexistent | remove-if-zero // Dictionary -DictionaryDefinition ::= dictionary (':' DictionarySetting) | ('{' DictionarySetting* '}') +DictionaryDefinition ::= dictionary ((':' DictionarySetting) | ('{' DictionarySetting* '}')) DictionarySetting ::= hash | btree | cased | uncased //***** End of Field's body elements ******// @@ -313,6 +315,8 @@ DictionarySetting ::= hash | btree | cased | uncased //---- Util rules ----- //--------------------- +private WordWrapper ::= KeywordOrIdentifier | KeywordNotIdentifier | ID_REG | ID_WITH_DASH_REG | WORD_REG + IdentifierVal ::= KeywordOrIdentifier | ID_REG { mixin="org.intellij.sdk.language.psi.impl.SdIdentifierMixin" implements=["org.intellij.sdk.language.psi.SdIdentifier"] } @@ -328,12 +332,12 @@ KeywordOrIdentifier ::= schema | search | document | struct | field | type | ind order | ascending | descending | diversity | constants | expression | weight | match | function | macro | inline | text | exact | word | prefix | cased | uncased | substring | suffix | gram | paged | mutable | alias | sorting | strength | locale | uca | lowercase | - primary | secondary | tertiary | quaternary | identical | rank | filter | normal | none | full | dynamic | - source | to | strict | loose | + primary | secondary | tertiary | quaternary | identical | rank | filter | normal | literal | + none | full | dynamic | source | to | strict | loose | bolding | on | off | true | false | id | normalizing | stemming | arity | hnsw | dictionary | hash | btree | fieldset | fields | constant | annotation - | attribute | body | header | index | - reference | summary | set_language + | attribute | body | header | index | static | + reference | summary | set_language | model // Note- in this form, those keywords can't be use as identifier-with-dash! KeywordNotIdentifier ::= struct-field | document-summary | omit-summary-features | from-disk | rank-profile | rank-type | @@ -345,5 +349,6 @@ KeywordNotIdentifier ::= struct-field | document-summary | omit-summary-features indexing-rewrite | query-command | matched-elements-only | lower-bound | upper-bound | dense-posting-list-threshold | enable-bm25 | max-links-per-node | neighbors-to-explore-at-insert | multi-threaded-indexing | create-if-nonexistent | remove-if-zero | raw-as-base64-in-summary | - onnx-model | cutoff-factor | cutoff-strategy | on-match | on-rank | on-summary + onnx-model | cutoff-factor | cutoff-strategy | on-match | on-rank | on-summary | enable-bit-vectors | + enable-only-bit-vector | summary-to | evaluation-point | pre-post-filter-tipping-point
\ No newline at end of file |