Merge pull request #20119 from vespa-engine/bratseth/intellij-vespa-plugin

Bratseth/intellij vespa plugin

12 files changed, 189 insertions, 54 deletions

diff --git a/config-model/src/main/javacc/SDParser.jj b/config-model/src/main/javacc/SDParser.jj
index c3851f2918a..d3d992c11f5 100644
--- a/config-model/src/main/javacc/SDParser.jj
+++ b/config-model/src/main/javacc/SDParser.jj
@@ -1,10 +1,9 @@
 // Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root.
-// --------------------------------------------------------------------------------
-//
-// JavaCC options. When this file is changed, run "mvn generate-sources" to rebuild
-// the parser classes.
+
+// Schema parser.
 //
-// --------------------------------------------------------------------------------
+// NOTE: When this is changed, also change integration/intellij/src/main/bnf/ai/vespa/intellij/schema/parser/sd.bnf
+
 options {
     UNICODE_INPUT = true;
     CACHE_TOKENS  = false;
@@ -15,11 +14,6 @@ options {
     USER_CHAR_STREAM = true;
 }
 
-// --------------------------------------------------------------------------------
-//
-// Parser body.
-//
-// --------------------------------------------------------------------------------
 PARSER_BEGIN(SDParser)
 
 package com.yahoo.searchdefinition.parser;
diff --git a/integration/intellij/.gitignore b/integration/intellij/.gitignore
index fa33885331d..96a25be6725 100644
--- a/integration/intellij/.gitignore
+++ b/integration/intellij/.gitignore
@@ -12,4 +12,9 @@ gradle-app.setting
 !gradle/wrapper/gradle-wrapper.jar
 
 # Cache of project
-.gradletasknamecache
\ No newline at end of file
+.gradletasknamecache
+
+# These are not needed, but IntelliJ automatically adds them
+gradlew
+gradlew.bat
+gradle/
diff --git a/integration/intellij/BACKLOG.md b/integration/intellij/BACKLOG.md
index 19e185dcc6b..aaef8ddd411 100644
--- a/integration/intellij/BACKLOG.md
+++ b/integration/intellij/BACKLOG.md
@@ -1,34 +1,41 @@
 ### Open Issues
 
-1. In some cases, the grammar prefers not to enforce bad syntax, because if the parser encounters bad syntax it stops 
-and can't build the PSI tree. That means none of the features will work for a file like that. For example, in cases 
-where an element supposes to have zero-to-one occurrences, the grammar will treat it as zero-to-many. 
-2. In order to enable the grammar recognize some keywords as identifiers (e.g. "filter" as a field's name), the 
-identifier rule (named "IdentifierVal") wraps the regex (ID_REG) and the KeywordOrIdentifier rule (which contains all 
-the keywords in the language). 
-3. The implementation of the GoTo Declaration feature is not exactly the same as IntelliJ. In IntelliJ if a reference
-has several declarations, after clicking "Goto Declaration" there is a little window with all the declarations to choose 
-from. It can be done by changing the method "multiResolve" in SdReference.java to return more than one declaration. The 
-problem with that is that it causes the "Find Usages" feature to not work. For now, I decided to make the plugin 
-"Goto Declaration" feature show only the most specific declaration by the right rank-profile scope.
-4. The "Find Usages" window can group usages only under rank-profiles and document-summaries. Other usages appear 
-directly under the .sd file. In order to create another group type of usages' group, you'll need to create 2 classes: 
-one for the extension "fileStructureGroupRuleProvider" (e.g. SdRankProfileGroupingRuleProvider.java), and one for the 
+In some cases, the parser stops on bad syntax and can't build the PSI tree. 
+That means no features will work in the file. 
+
+To enable the grammar recognize some keywords as identifiers (e.g. "filter" as a field's name), 
+the identifier rule (named "IdentifierVal") wraps the regex (ID_REG) and the KeywordOrIdentifier 
+rule (which contains all the keywords in the language). 
+
+The implementation of the GoTo Declaration feature is not exactly the same as IntelliJ. 
+In IntelliJ if a reference has several declarations, after clicking "Goto Declaration"
+there is a little window with all the declarations to choose from. 
+It can be done by changing the method "multiResolve" in SdReference.java to return 
+more than one declaration. The problem with that is that it causes the "Find Usages" 
+feature to not work. For now the plugin "Goto Declaration" feature shows only the 
+most specific declaration by the right rank-profile scope.
+
+The "Find Usages" window can group usages only under rank-profiles and document-summaries.
+Other usages appear directly under the .sd file. To create another group type of usages' group, 
+you'll need to create 2 classes: one for the extension "fileStructureGroupRuleProvider" 
+(e.g. SdRankProfileGroupingRuleProvider.java), and one for the 
 grouping rule itself (e.g. SdRankProfileGroupingRule.java).
-Another open problem is that the navigation isn't working in the current grouping rules. It means that when clicking on 
-the group headline (e.g. some name of a rank-profile) the IDE doesn't "jump" to the matching declaration.
-5. Goto declaration doesn't work for document's inherits. e.g. if document A inherits from document B, B doesn't have a 
-reference to its declaration.
-6. There aren't any tests for the plugin.
-
-### Some useful links:
-1. JetBrains official tutorials: https://plugins.jetbrains.com/docs/intellij/custom-language-support.html and
-https://plugins.jetbrains.com/docs/intellij/custom-language-support-tutorial.html
-2. Grammar-Kit HOWTO: Helps to understand the BNF syntax.
-   https://github.com/JetBrains/Grammar-Kit/blob/master/HOWTO.md
-3. How to deal with left-recursion in the grammar (in SD for example it happens in expressions). Last answer here: 
-https://intellij-support.jetbrains.com/hc/en-us/community/posts/360001258300-What-s-the-alternative-to-left-recursion-in-GrammarKit-
-4. Great tutorial for a custom-language-plugin, but only for the basics (mainly the parser and lexer):
-   https://medium.com/@shan1024/custom-language-plugin-development-for-intellij-idea-part-01-d6a41ab96bc9
-5. Code of Dart (some custom language) plugin for IntelliJ: 
-https://github.com/JetBrains/intellij-plugins/tree/0f07ca63355d5530b441ca566c98f17c560e77f8/Dart
\ No newline at end of file
+Another open problem is that the navigation isn't working in the current grouping rules. 
+It means that when clicking on the group headline (e.g. some name of a rank-profile) 
+the IDE doesn't "jump" to the matching declaration.
+
+Goto declaration doesn't work for document and schema inherits. E.g. if document A inherits from 
+document B, B doesn't have a reference to its declaration.
+
+There aren't any tests for the plugin.
+
+Semicolons in indexing statements are marked with red background for some reason. 
+They are not marked as errors.
+
+Type suggestions should include all primitive types, not just annotations
+
+Even if the parser continues, only the first error in a file is marked.
+
+Aliases with dot does not work.
+
+
diff --git a/integration/intellij/README.md b/integration/intellij/README.md
index 36cefc22477..fec547f8a06 100644
--- a/integration/intellij/README.md
+++ b/integration/intellij/README.md
@@ -32,3 +32,21 @@ However, gradle is configured with a maven directory layout.
 
 With the first (?), you can run the gradle task "intellij/runIde" (or "./gradlew runIde" in the command line), 
 open a project with some sd file and see how the plugin works on it.
+
+
+## Some useful links:
+
+1. JetBrains official tutorials: https://plugins.jetbrains.com/docs/intellij/custom-language-support.html and
+   https://plugins.jetbrains.com/docs/intellij/custom-language-support-tutorial.html
+
+2. Grammar-Kit HOWTO: Helps to understand the BNF syntax.
+   https://github.com/JetBrains/Grammar-Kit/blob/master/HOWTO.md
+
+3. How to deal with left-recursion in the grammar (in SD for example it happens in expressions). Last answer here:
+   https://intellij-support.jetbrains.com/hc/en-us/community/posts/360001258300-What-s-the-alternative-to-left-recursion-in-GrammarKit-
+
+4. Great tutorial for a custom-language-plugin, but only for the basics (mainly the parser and lexer):
+   https://medium.com/@shan1024/custom-language-plugin-development-for-intellij-idea-part-01-d6a41ab96bc9
+
+5. Code of Dart (some custom language) plugin for IntelliJ:
+   https://github.com/JetBrains/intellij-plugins/tree/0f07ca63355d5530b441ca566c98f17c560e77f8/Dart
\ No newline at end of file
diff --git a/integration/intellij/build.gradle b/integration/intellij/build.gradle
index 3c3f0a73e01..4801bc810b0 100644
--- a/integration/intellij/build.gradle
+++ b/integration/intellij/build.gradle
@@ -36,7 +36,7 @@ compileJava {
 }
 
 group 'ai.vespa'
-version '1.0.0'
+version '1.0.0' // Also update pom.xml version if this is changed
 
 sourceCompatibility = 11
 
diff --git a/integration/intellij/pom.xml b/integration/intellij/pom.xml
new file mode 100644
index 00000000000..84a24b788b8
--- /dev/null
+++ b/integration/intellij/pom.xml
@@ -0,0 +1,90 @@
+<?xml version="1.0"?>
+<!-- Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. -->
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
+    <modelVersion>4.0.0</modelVersion>
+    <parent>
+        <groupId>com.yahoo.vespa</groupId>
+        <artifactId>parent</artifactId>
+        <version>7-SNAPSHOT</version>
+        <relativePath>../parent/pom.xml</relativePath>
+    </parent>
+    <artifactId>vespa-intellij</artifactId> <!-- Not used - plugin is build by gradle -->
+    <version>1.0.0</version> <!-- See copy-zip below, which depends on this being the same as the v. in build.gradle -->
+    <description>
+        Maven wrapper for the gradle build of this IntelliJ plugin.
+    </description>
+
+    <dependencies>
+    </dependencies>
+
+    <build>
+        <plugins>
+            <!-- The Gradle plugin must be built by Gradle; therefore the compilation is skipped. -->
+            <plugin>
+                <groupId>org.apache.maven.plugins</groupId>
+                <artifactId>maven-compiler-plugin</artifactId>
+                <configuration>
+                    <skipMain>true</skipMain>
+                    <skip>true</skip>
+                </configuration>
+            </plugin>
+            <!-- Gradle is also responsible for creating javadoc such that this task is skipped here. -->
+            <plugin>
+                <groupId>org.apache.maven.plugins</groupId>
+                <artifactId>maven-javadoc-plugin</artifactId>
+                <configuration>
+                    <skip>true</skip>
+                </configuration>
+            </plugin>
+            <!-- Tie Maven executions into the Gradle life-cycle. -->
+            <plugin>
+                <groupId>org.codehaus.mojo</groupId>
+                <artifactId>exec-maven-plugin</artifactId>
+                <executions>
+                    <execution>
+                        <id>gradle-clean</id>
+                        <phase>clean</phase>
+                        <goals>
+                            <goal>exec</goal>
+                        </goals>
+                        <configuration>
+                            <executable>gradle</executable>
+                            <arguments>
+                                <argument>clean</argument>
+                            </arguments>
+                        </configuration>
+                    </execution>
+                    <execution>
+                        <id>gradle-build</id>
+                        <phase>compile</phase>
+                        <goals>
+                            <goal>exec</goal>
+                        </goals>
+                        <configuration>
+                            <executable>gradle</executable>
+                        </configuration>
+                    </execution>
+                </executions>
+            </plugin>
+            <!-- Copies the artifact created by Gradle back to the Maven target folder. -->
+            <plugin>
+                <groupId>org.apache.maven.plugins</groupId>
+                <artifactId>maven-antrun-plugin</artifactId>
+                <executions>
+                    <execution>
+                        <id>copy-zip</id>
+                        <phase>install</phase>
+                        <goals>
+                            <goal>run</goal>
+                        </goals>
+                        <configuration>
+                            <target>
+                                <copy file="build/distributions/vespa-${project.version}.zip" todir="target" overwrite="true" />
+                            </target>
+                        </configuration>
+                    </execution>
+                </executions>
+            </plugin>
+        </plugins>
+    </build>
+</project>
diff --git a/integration/intellij/src/main/bnf/ai/vespa/intellij/schema/parser/sd.bnf b/integration/intellij/src/main/bnf/ai/vespa/intellij/schema/parser/sd.bnf
index f01ce41dd64..6be7302a28c 100644
--- a/integration/intellij/src/main/bnf/ai/vespa/intellij/schema/parser/sd.bnf
+++ b/integration/intellij/src/main/bnf/ai/vespa/intellij/schema/parser/sd.bnf
@@ -32,7 +32,7 @@
         COMMENT = 'regexp:#.*'
         SYMBOL = 'regexp:[!$|:{}(),.\[\]]'
         COMPARISON_OPERATOR = 'regexp:[<>]|(==)|(<=)|(>=)|(~=)'
-        ARITHMETIC_OPERATOR = 'regexp:[\-+*/]'
+        ARITHMETIC_OPERATOR = 'regexp:[\-+*/%]'
         INTEGER_REG = 'regexp:[0-9]+'
         FLOAT_REG = 'regexp:[0-9]+[.][0-9]+[e]?'
         STRING_REG = 'regexp:\"([^\"\\]*(\\.[^\"\\]*)*)\"'
@@ -44,7 +44,7 @@ SdFile ::= SchemaDefinition | DocumentDefinition
 SchemaDefinition ::= (search | schema) IdentifierVal? (inherits IdentifierVal)? '{' SchemaBody '}'
 SchemaBody ::= SchemaBodyOptions* DocumentDefinition SchemaBodyOptions* // Does not support zero-or-one occurrences
 private SchemaBodyOptions ::= SchemaFieldDefinition | ImportFieldDefinition | DocumentSummaryDefinition | 
-                              RankProfileDefinition | IndexDefinition |
+                              RankProfileDefinition | IndexDefinition | DocumentStructDefinition |
                               FieldSetDefinition | ConstantDefinition | OnnxModelDefinition | StemmingDefinition |
                               raw-as-base64-in-summary | SchemaAnnotationDefinition
 
@@ -170,7 +170,7 @@ RankPropertiesBody ::= (RankPropertiesKey ':' RankPropertiesValue)+
 RankPropertiesKey ::= (IdentifierWithDashVal | STRING_REG | '(' | ')' | '.' | ',' | '$' | INTEGER_REG)+ 
 RankPropertiesValue ::= (('-')? INTEGER_REG) | (('-')? FLOAT_REG) | WORD_REG | IdentifierVal | STRING_REG
 
-FunctionDefinition ::= (function | macro) inline? IdentifierVal '(' (ArgumentDefinition (',' ArgumentDefinition)*)? ')'
+FunctionDefinition ::= (function | macro) inline? IdentifierVal ( '()' | '(' (ArgumentDefinition (',' ArgumentDefinition)*)? ')' )
                        '{' ExpressionDefinition '}'  
                        { mixin="ai.vespa.intellij.schema.psi.impl.SdNamedElementImpl"
                          implements=["ai.vespa.intellij.schema.psi.SdFunctionDefinitionInterface" "ai.vespa.intellij.schema.psi.SdNamedElement"]
@@ -182,7 +182,7 @@ ArgumentDefinition ::= IdentifierVal
 
 SummaryFeaturesDefinition ::= summary-features ((':' RankFeature+) | ((inherits IdentifierVal)? '{' RankFeature* '}'))
 
-MatchFeaturesDefinition ::= match-features ((':' RankFeature+) | ('{' RankFeature* '}'))
+MatchFeaturesDefinition ::= match-features ((':' RankFeature+) | ((inherits IdentifierVal)? '{' RankFeature* '}'))
 
 RankFeaturesDefinition ::= rank-features ((':' RankFeature+) | ('{' RankFeature* '}'))
 
@@ -237,7 +237,7 @@ private DocumentFieldBodyOptions ::= StructFieldDefinition | MatchDefinition | I
                              AliasDefinition | RankDefinition | IndexingRewriteState | QueryCommandDefinition | SummaryDefinition |
                              BoldingDefinition | (id ':' INTEGER_REG) | IndexDefinition | (normalizing ':' IdentifierWithDashVal) | 
                              SortingDefinition | StemmingDefinition | (weight ':' INTEGER_REG) | WeightedSetDefinition |
-                             RankTypeDefinition | DictionaryDefinition | SummaryToDefinition | body
+                             RankTypeDefinition | DictionaryDefinition | SummaryToDefinition | header | body
 //***** Field's body elements ******//
 // Struct
 StructFieldDefinition ::= struct-field IdentifierVal ('.' IdentifierVal)? '{' StructFieldBody '}'
@@ -250,15 +250,16 @@ StructFieldBodyOptions ::= IndexingDefinition | AttributeDefinition | MatchDefin
                            StructFieldDefinition | SummaryDefinition            
 // Match
 MatchDefinition ::= match ((':' MatchProperty) | ('{' MatchProperty+ '}'))
-MatchProperty ::= text | exact | (exact-terminator ':' STRING_REG) | word | prefix | cased | uncased | substring | 
+MatchProperty ::= text | token | exact | (exact-terminator ':' STRING_REG) | word | prefix | cased | uncased | substring |
                   suffix | (max-length ':' INTEGER_REG) | gram | (gram-size ':' INTEGER_REG) | WordWrapper
 // Indexing
 IndexingDefinition ::= indexing ((':' IndexingStatement) | ('{' IndexingStatement+ '}'))
-IndexingStatement ::= IndexingStatementOptions ((('|' | ';') IndexingStatementOptions)*)
+IndexingStatement ::= IndexingStatementOptions (('|' IndexingStatementOptions)*) (';')?
                       // Does not support zero-or-one occurrences
-IndexingStatementOptions ::= summary | attribute | index | set_language | lowercase | (input (IdentifierVal | IndexingStuff)+) | 
+IndexingStatementOptions ::= summary | attribute | index | set_language | lowercase |
+                             (input (IdentifierVal | IndexingStuff)+) |
                              ('{' IndexingStatementOptions '}') | IndexingStuff+
-private IndexingStuff ::= WordWrapper | INTEGER_REG | FLOAT_REG | STRING_REG | ('{' IndexingStatementOptions+ '}') |
+private IndexingStuff ::= WordWrapper | INTEGER_REG | FLOAT_REG | STRING_REG | ('{' IndexingStatement '}') |
                           ':' | ('|' IndexingStatementOptions) | ';' | '.' | '(' | ')' | ARITHMETIC_OPERATOR | COMPARISON_OPERATOR
 // Attribute
 AttributeDefinition ::= attribute ((':' SimpleAttributeProperty) | ('{' (ComplexAttributeProperty | SimpleAttributeProperty)+ '}'))
@@ -266,7 +267,7 @@ SimpleAttributeProperty ::= fast-search | fast-access | paged | mutable | enable
 ComplexAttributeProperty ::= AliasDefinition | SortingDefinition | DistanceMetricDef // Does not support zero-or-one occurrences
 DistanceMetricDef ::= distance-metric ':' IdentifierWithDashVal
 // Alias
-AliasDefinition ::= alias (IdentifierWithDashVal ('.' IdentifierWithDashVal)*)? ':' IdentifierWithDashVal ('.' IdentifierWithDashVal)*
+AliasDefinition ::= alias (IdentifierVal)? ':' IdentifierWithDashVal ('.' IdentifierWithDashVal)*
 // Stemming
 StemmingDefinition ::= stemming ':' IdentifierWithDashVal
 // Rank
diff --git a/integration/intellij/src/main/jflex/ai/vespa/intellij/schema/lexer/sd.flex b/integration/intellij/src/main/jflex/ai/vespa/intellij/schema/lexer/sd.flex
index b4491acc717..78334de1961 100644
--- a/integration/intellij/src/main/jflex/ai/vespa/intellij/schema/lexer/sd.flex
+++ b/integration/intellij/src/main/jflex/ai/vespa/intellij/schema/lexer/sd.flex
@@ -38,7 +38,7 @@ SYMBOL= [!$|:{}(),.\[\]]
 INTEGER = [0-9]+
 FLOAT = {INTEGER}[.][0-9]+[e]?
 COMPARISON_OPERATOR = [<>]|(==)|(<=)|(>=)|(\~=)
-ARITHMETIC_OPERATOR = [\-+*/]
+ARITHMETIC_OPERATOR = [\-+*/%]
 STRING = \"([^\"\\]*(\\.[^\"\\]*)*)\"
 WORD = \w+
 
diff --git a/integration/intellij/src/main/resources/META-INF/pluginIcon.png b/integration/intellij/src/main/resources/META-INF/pluginIcon.png
deleted file mode 100644
index c8dc4764e35..00000000000
--- a/integration/intellij/src/main/resources/META-INF/pluginIcon.png
+++ /dev/null
diff --git a/integration/intellij/src/main/resources/META-INF/pluginIcon.svg b/integration/intellij/src/main/resources/META-INF/pluginIcon.svg
new file mode 100644
index 00000000000..3a795976efd
--- /dev/null
+++ b/integration/intellij/src/main/resources/META-INF/pluginIcon.svg
@@ -0,0 +1,19 @@
+<?xml version="1.0" encoding="utf-8"?>
+<svg id="Layer_1" data-name="Layer 1" viewBox="0 0 40 40" preserveAspectRatio="xMidYMin slice" xmlns="http://www.w3.org/2000/svg">
+  <defs>
+    <style>.cls-1{fill:#ffc43c;}.cls-2{fill:url(#linear-gradient);}.cls-3{fill:#1a7db6;}.cls-4{fill:url(#linear-gradient-2);}.cls-5{fill:#303030;}</style>
+    <linearGradient id="linear-gradient" x1="964.75" y1="34.82" x2="919.68" y2="63.4" gradientTransform="matrix(-0.276923, 0, 0, 0.276923, 291.653198, 2)" gradientUnits="userSpaceOnUse">
+      <stop offset="0" stop-color="#da9728"/>
+      <stop offset="0.54" stop-color="#ffc43c"/>
+    </linearGradient>
+    <linearGradient id="linear-gradient-2" x1="46.38" y1="94.56" x2="-29.88" y2="53" gradientUnits="userSpaceOnUse" gradientTransform="matrix(0.276923, 0, 0, 0.276923, 3.212999, 2)">
+      <stop offset="0" stop-color="#005a8e"/>
+      <stop offset="0.54" stop-color="#1a7db6"/>
+    </linearGradient>
+  </defs>
+  <title>VespaLogo copy</title>
+  <polygon class="cls-1" points="22.736 2 8.685 11.543 8.685 27.971 22.736 18.064 36.787 27.971 36.787 11.543 22.736 2"/>
+  <polygon class="cls-2" points="22.736 18.064 22.736 2 36.787 11.543 36.787 27.97 22.736 18.064"/>
+  <polygon class="cls-3" points="17.264 21.936 3.213 12.029 3.213 28.457 17.264 38 31.315 28.457 31.315 12.029 17.264 21.936"/>
+  <polygon class="cls-4" points="17.264 21.936 17.264 38 3.213 28.457 3.213 12.03 17.264 21.936"/>
+</svg>
\ No newline at end of file
diff --git a/integration/intellij/src/main/resources/META-INF/pluginIcon_dark.png b/integration/intellij/src/main/resources/META-INF/pluginIcon_dark.png
deleted file mode 100644
index c8dc4764e35..00000000000
--- a/integration/intellij/src/main/resources/META-INF/pluginIcon_dark.png
+++ /dev/null
diff --git a/pom.xml b/pom.xml
index d43affb3565..9cf3630e463 100644
--- a/pom.xml
+++ b/pom.xml
@@ -85,6 +85,7 @@
         <module>hosted-zone-api</module>
         <module>http-utils</module>
         <module>indexinglanguage</module>
+        <!--<module>integration/intellij</module>-->
         <module>jaxrs_client_utils</module>
         <module>jaxrs_utils</module>
         <module>jdisc-cloud-aws</module>