diff options
author | Jon Bratseth <bratseth@oath.com> | 2021-11-22 09:46:31 +0100 |
---|---|---|
committer | GitHub <noreply@github.com> | 2021-11-22 09:46:31 +0100 |
commit | d24c2e73a266642da1ff8d3e92a41740be992647 (patch) | |
tree | ad4475020d6a02407b703d7c5c015eb1c3150209 | |
parent | 5320f639fa47dbf2e200476d572e8b82d0b74632 (diff) | |
parent | b23bc29ede35c32196f457c3363ae5ea92b9cbe5 (diff) |
Merge pull request #20119 from vespa-engine/bratseth/intellij-vespa-plugin
Bratseth/intellij vespa plugin
-rw-r--r-- | config-model/src/main/javacc/SDParser.jj | 14 | ||||
-rw-r--r-- | integration/intellij/.gitignore | 7 | ||||
-rw-r--r-- | integration/intellij/BACKLOG.md | 69 | ||||
-rw-r--r-- | integration/intellij/README.md | 18 | ||||
-rw-r--r-- | integration/intellij/build.gradle | 2 | ||||
-rw-r--r-- | integration/intellij/pom.xml | 90 | ||||
-rw-r--r-- | integration/intellij/src/main/bnf/ai/vespa/intellij/schema/parser/sd.bnf | 21 | ||||
-rw-r--r-- | integration/intellij/src/main/jflex/ai/vespa/intellij/schema/lexer/sd.flex | 2 | ||||
-rw-r--r-- | integration/intellij/src/main/resources/META-INF/pluginIcon.png | bin | 30317 -> 0 bytes | |||
-rw-r--r-- | integration/intellij/src/main/resources/META-INF/pluginIcon.svg | 19 | ||||
-rw-r--r-- | integration/intellij/src/main/resources/META-INF/pluginIcon_dark.png | bin | 30317 -> 0 bytes | |||
-rw-r--r-- | pom.xml | 1 |
12 files changed, 189 insertions, 54 deletions
diff --git a/config-model/src/main/javacc/SDParser.jj b/config-model/src/main/javacc/SDParser.jj index c3851f2918a..d3d992c11f5 100644 --- a/config-model/src/main/javacc/SDParser.jj +++ b/config-model/src/main/javacc/SDParser.jj @@ -1,10 +1,9 @@ // Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. -// -------------------------------------------------------------------------------- -// -// JavaCC options. When this file is changed, run "mvn generate-sources" to rebuild -// the parser classes. + +// Schema parser. // -// -------------------------------------------------------------------------------- +// NOTE: When this is changed, also change integration/intellij/src/main/bnf/ai/vespa/intellij/schema/parser/sd.bnf + options { UNICODE_INPUT = true; CACHE_TOKENS = false; @@ -15,11 +14,6 @@ options { USER_CHAR_STREAM = true; } -// -------------------------------------------------------------------------------- -// -// Parser body. -// -// -------------------------------------------------------------------------------- PARSER_BEGIN(SDParser) package com.yahoo.searchdefinition.parser; diff --git a/integration/intellij/.gitignore b/integration/intellij/.gitignore index fa33885331d..96a25be6725 100644 --- a/integration/intellij/.gitignore +++ b/integration/intellij/.gitignore @@ -12,4 +12,9 @@ gradle-app.setting !gradle/wrapper/gradle-wrapper.jar # Cache of project -.gradletasknamecache
\ No newline at end of file +.gradletasknamecache + +# These are not needed, but IntelliJ automatically adds them +gradlew +gradlew.bat +gradle/ diff --git a/integration/intellij/BACKLOG.md b/integration/intellij/BACKLOG.md index 19e185dcc6b..aaef8ddd411 100644 --- a/integration/intellij/BACKLOG.md +++ b/integration/intellij/BACKLOG.md @@ -1,34 +1,41 @@ ### Open Issues -1. In some cases, the grammar prefers not to enforce bad syntax, because if the parser encounters bad syntax it stops -and can't build the PSI tree. That means none of the features will work for a file like that. For example, in cases -where an element supposes to have zero-to-one occurrences, the grammar will treat it as zero-to-many. -2. In order to enable the grammar recognize some keywords as identifiers (e.g. "filter" as a field's name), the -identifier rule (named "IdentifierVal") wraps the regex (ID_REG) and the KeywordOrIdentifier rule (which contains all -the keywords in the language). -3. The implementation of the GoTo Declaration feature is not exactly the same as IntelliJ. In IntelliJ if a reference -has several declarations, after clicking "Goto Declaration" there is a little window with all the declarations to choose -from. It can be done by changing the method "multiResolve" in SdReference.java to return more than one declaration. The -problem with that is that it causes the "Find Usages" feature to not work. For now, I decided to make the plugin -"Goto Declaration" feature show only the most specific declaration by the right rank-profile scope. -4. The "Find Usages" window can group usages only under rank-profiles and document-summaries. Other usages appear -directly under the .sd file. In order to create another group type of usages' group, you'll need to create 2 classes: -one for the extension "fileStructureGroupRuleProvider" (e.g. SdRankProfileGroupingRuleProvider.java), and one for the +In some cases, the parser stops on bad syntax and can't build the PSI tree. +That means no features will work in the file. + +To enable the grammar recognize some keywords as identifiers (e.g. "filter" as a field's name), +the identifier rule (named "IdentifierVal") wraps the regex (ID_REG) and the KeywordOrIdentifier +rule (which contains all the keywords in the language). + +The implementation of the GoTo Declaration feature is not exactly the same as IntelliJ. +In IntelliJ if a reference has several declarations, after clicking "Goto Declaration" +there is a little window with all the declarations to choose from. +It can be done by changing the method "multiResolve" in SdReference.java to return +more than one declaration. The problem with that is that it causes the "Find Usages" +feature to not work. For now the plugin "Goto Declaration" feature shows only the +most specific declaration by the right rank-profile scope. + +The "Find Usages" window can group usages only under rank-profiles and document-summaries. +Other usages appear directly under the .sd file. To create another group type of usages' group, +you'll need to create 2 classes: one for the extension "fileStructureGroupRuleProvider" +(e.g. SdRankProfileGroupingRuleProvider.java), and one for the grouping rule itself (e.g. SdRankProfileGroupingRule.java). -Another open problem is that the navigation isn't working in the current grouping rules. It means that when clicking on -the group headline (e.g. some name of a rank-profile) the IDE doesn't "jump" to the matching declaration. -5. Goto declaration doesn't work for document's inherits. e.g. if document A inherits from document B, B doesn't have a -reference to its declaration. -6. There aren't any tests for the plugin. - -### Some useful links: -1. JetBrains official tutorials: https://plugins.jetbrains.com/docs/intellij/custom-language-support.html and -https://plugins.jetbrains.com/docs/intellij/custom-language-support-tutorial.html -2. Grammar-Kit HOWTO: Helps to understand the BNF syntax. - https://github.com/JetBrains/Grammar-Kit/blob/master/HOWTO.md -3. How to deal with left-recursion in the grammar (in SD for example it happens in expressions). Last answer here: -https://intellij-support.jetbrains.com/hc/en-us/community/posts/360001258300-What-s-the-alternative-to-left-recursion-in-GrammarKit- -4. Great tutorial for a custom-language-plugin, but only for the basics (mainly the parser and lexer): - https://medium.com/@shan1024/custom-language-plugin-development-for-intellij-idea-part-01-d6a41ab96bc9 -5. Code of Dart (some custom language) plugin for IntelliJ: -https://github.com/JetBrains/intellij-plugins/tree/0f07ca63355d5530b441ca566c98f17c560e77f8/Dart
\ No newline at end of file +Another open problem is that the navigation isn't working in the current grouping rules. +It means that when clicking on the group headline (e.g. some name of a rank-profile) +the IDE doesn't "jump" to the matching declaration. + +Goto declaration doesn't work for document and schema inherits. E.g. if document A inherits from +document B, B doesn't have a reference to its declaration. + +There aren't any tests for the plugin. + +Semicolons in indexing statements are marked with red background for some reason. +They are not marked as errors. + +Type suggestions should include all primitive types, not just annotations + +Even if the parser continues, only the first error in a file is marked. + +Aliases with dot does not work. + + diff --git a/integration/intellij/README.md b/integration/intellij/README.md index 36cefc22477..fec547f8a06 100644 --- a/integration/intellij/README.md +++ b/integration/intellij/README.md @@ -32,3 +32,21 @@ However, gradle is configured with a maven directory layout. With the first (?), you can run the gradle task "intellij/runIde" (or "./gradlew runIde" in the command line), open a project with some sd file and see how the plugin works on it. + + +## Some useful links: + +1. JetBrains official tutorials: https://plugins.jetbrains.com/docs/intellij/custom-language-support.html and + https://plugins.jetbrains.com/docs/intellij/custom-language-support-tutorial.html + +2. Grammar-Kit HOWTO: Helps to understand the BNF syntax. + https://github.com/JetBrains/Grammar-Kit/blob/master/HOWTO.md + +3. How to deal with left-recursion in the grammar (in SD for example it happens in expressions). Last answer here: + https://intellij-support.jetbrains.com/hc/en-us/community/posts/360001258300-What-s-the-alternative-to-left-recursion-in-GrammarKit- + +4. Great tutorial for a custom-language-plugin, but only for the basics (mainly the parser and lexer): + https://medium.com/@shan1024/custom-language-plugin-development-for-intellij-idea-part-01-d6a41ab96bc9 + +5. Code of Dart (some custom language) plugin for IntelliJ: + https://github.com/JetBrains/intellij-plugins/tree/0f07ca63355d5530b441ca566c98f17c560e77f8/Dart
\ No newline at end of file diff --git a/integration/intellij/build.gradle b/integration/intellij/build.gradle index 3c3f0a73e01..4801bc810b0 100644 --- a/integration/intellij/build.gradle +++ b/integration/intellij/build.gradle @@ -36,7 +36,7 @@ compileJava { } group 'ai.vespa' -version '1.0.0' +version '1.0.0' // Also update pom.xml version if this is changed sourceCompatibility = 11 diff --git a/integration/intellij/pom.xml b/integration/intellij/pom.xml new file mode 100644 index 00000000000..84a24b788b8 --- /dev/null +++ b/integration/intellij/pom.xml @@ -0,0 +1,90 @@ +<?xml version="1.0"?> +<!-- Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. --> +<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd"> + <modelVersion>4.0.0</modelVersion> + <parent> + <groupId>com.yahoo.vespa</groupId> + <artifactId>parent</artifactId> + <version>7-SNAPSHOT</version> + <relativePath>../parent/pom.xml</relativePath> + </parent> + <artifactId>vespa-intellij</artifactId> <!-- Not used - plugin is build by gradle --> + <version>1.0.0</version> <!-- See copy-zip below, which depends on this being the same as the v. in build.gradle --> + <description> + Maven wrapper for the gradle build of this IntelliJ plugin. + </description> + + <dependencies> + </dependencies> + + <build> + <plugins> + <!-- The Gradle plugin must be built by Gradle; therefore the compilation is skipped. --> + <plugin> + <groupId>org.apache.maven.plugins</groupId> + <artifactId>maven-compiler-plugin</artifactId> + <configuration> + <skipMain>true</skipMain> + <skip>true</skip> + </configuration> + </plugin> + <!-- Gradle is also responsible for creating javadoc such that this task is skipped here. --> + <plugin> + <groupId>org.apache.maven.plugins</groupId> + <artifactId>maven-javadoc-plugin</artifactId> + <configuration> + <skip>true</skip> + </configuration> + </plugin> + <!-- Tie Maven executions into the Gradle life-cycle. --> + <plugin> + <groupId>org.codehaus.mojo</groupId> + <artifactId>exec-maven-plugin</artifactId> + <executions> + <execution> + <id>gradle-clean</id> + <phase>clean</phase> + <goals> + <goal>exec</goal> + </goals> + <configuration> + <executable>gradle</executable> + <arguments> + <argument>clean</argument> + </arguments> + </configuration> + </execution> + <execution> + <id>gradle-build</id> + <phase>compile</phase> + <goals> + <goal>exec</goal> + </goals> + <configuration> + <executable>gradle</executable> + </configuration> + </execution> + </executions> + </plugin> + <!-- Copies the artifact created by Gradle back to the Maven target folder. --> + <plugin> + <groupId>org.apache.maven.plugins</groupId> + <artifactId>maven-antrun-plugin</artifactId> + <executions> + <execution> + <id>copy-zip</id> + <phase>install</phase> + <goals> + <goal>run</goal> + </goals> + <configuration> + <target> + <copy file="build/distributions/vespa-${project.version}.zip" todir="target" overwrite="true" /> + </target> + </configuration> + </execution> + </executions> + </plugin> + </plugins> + </build> +</project> diff --git a/integration/intellij/src/main/bnf/ai/vespa/intellij/schema/parser/sd.bnf b/integration/intellij/src/main/bnf/ai/vespa/intellij/schema/parser/sd.bnf index f01ce41dd64..6be7302a28c 100644 --- a/integration/intellij/src/main/bnf/ai/vespa/intellij/schema/parser/sd.bnf +++ b/integration/intellij/src/main/bnf/ai/vespa/intellij/schema/parser/sd.bnf @@ -32,7 +32,7 @@ COMMENT = 'regexp:#.*' SYMBOL = 'regexp:[!$|:{}(),.\[\]]' COMPARISON_OPERATOR = 'regexp:[<>]|(==)|(<=)|(>=)|(~=)' - ARITHMETIC_OPERATOR = 'regexp:[\-+*/]' + ARITHMETIC_OPERATOR = 'regexp:[\-+*/%]' INTEGER_REG = 'regexp:[0-9]+' FLOAT_REG = 'regexp:[0-9]+[.][0-9]+[e]?' STRING_REG = 'regexp:\"([^\"\\]*(\\.[^\"\\]*)*)\"' @@ -44,7 +44,7 @@ SdFile ::= SchemaDefinition | DocumentDefinition SchemaDefinition ::= (search | schema) IdentifierVal? (inherits IdentifierVal)? '{' SchemaBody '}' SchemaBody ::= SchemaBodyOptions* DocumentDefinition SchemaBodyOptions* // Does not support zero-or-one occurrences private SchemaBodyOptions ::= SchemaFieldDefinition | ImportFieldDefinition | DocumentSummaryDefinition | - RankProfileDefinition | IndexDefinition | + RankProfileDefinition | IndexDefinition | DocumentStructDefinition | FieldSetDefinition | ConstantDefinition | OnnxModelDefinition | StemmingDefinition | raw-as-base64-in-summary | SchemaAnnotationDefinition @@ -170,7 +170,7 @@ RankPropertiesBody ::= (RankPropertiesKey ':' RankPropertiesValue)+ RankPropertiesKey ::= (IdentifierWithDashVal | STRING_REG | '(' | ')' | '.' | ',' | '$' | INTEGER_REG)+ RankPropertiesValue ::= (('-')? INTEGER_REG) | (('-')? FLOAT_REG) | WORD_REG | IdentifierVal | STRING_REG -FunctionDefinition ::= (function | macro) inline? IdentifierVal '(' (ArgumentDefinition (',' ArgumentDefinition)*)? ')' +FunctionDefinition ::= (function | macro) inline? IdentifierVal ( '()' | '(' (ArgumentDefinition (',' ArgumentDefinition)*)? ')' ) '{' ExpressionDefinition '}' { mixin="ai.vespa.intellij.schema.psi.impl.SdNamedElementImpl" implements=["ai.vespa.intellij.schema.psi.SdFunctionDefinitionInterface" "ai.vespa.intellij.schema.psi.SdNamedElement"] @@ -182,7 +182,7 @@ ArgumentDefinition ::= IdentifierVal SummaryFeaturesDefinition ::= summary-features ((':' RankFeature+) | ((inherits IdentifierVal)? '{' RankFeature* '}')) -MatchFeaturesDefinition ::= match-features ((':' RankFeature+) | ('{' RankFeature* '}')) +MatchFeaturesDefinition ::= match-features ((':' RankFeature+) | ((inherits IdentifierVal)? '{' RankFeature* '}')) RankFeaturesDefinition ::= rank-features ((':' RankFeature+) | ('{' RankFeature* '}')) @@ -237,7 +237,7 @@ private DocumentFieldBodyOptions ::= StructFieldDefinition | MatchDefinition | I AliasDefinition | RankDefinition | IndexingRewriteState | QueryCommandDefinition | SummaryDefinition | BoldingDefinition | (id ':' INTEGER_REG) | IndexDefinition | (normalizing ':' IdentifierWithDashVal) | SortingDefinition | StemmingDefinition | (weight ':' INTEGER_REG) | WeightedSetDefinition | - RankTypeDefinition | DictionaryDefinition | SummaryToDefinition | body + RankTypeDefinition | DictionaryDefinition | SummaryToDefinition | header | body //***** Field's body elements ******// // Struct StructFieldDefinition ::= struct-field IdentifierVal ('.' IdentifierVal)? '{' StructFieldBody '}' @@ -250,15 +250,16 @@ StructFieldBodyOptions ::= IndexingDefinition | AttributeDefinition | MatchDefin StructFieldDefinition | SummaryDefinition // Match MatchDefinition ::= match ((':' MatchProperty) | ('{' MatchProperty+ '}')) -MatchProperty ::= text | exact | (exact-terminator ':' STRING_REG) | word | prefix | cased | uncased | substring | +MatchProperty ::= text | token | exact | (exact-terminator ':' STRING_REG) | word | prefix | cased | uncased | substring | suffix | (max-length ':' INTEGER_REG) | gram | (gram-size ':' INTEGER_REG) | WordWrapper // Indexing IndexingDefinition ::= indexing ((':' IndexingStatement) | ('{' IndexingStatement+ '}')) -IndexingStatement ::= IndexingStatementOptions ((('|' | ';') IndexingStatementOptions)*) +IndexingStatement ::= IndexingStatementOptions (('|' IndexingStatementOptions)*) (';')? // Does not support zero-or-one occurrences -IndexingStatementOptions ::= summary | attribute | index | set_language | lowercase | (input (IdentifierVal | IndexingStuff)+) | +IndexingStatementOptions ::= summary | attribute | index | set_language | lowercase | + (input (IdentifierVal | IndexingStuff)+) | ('{' IndexingStatementOptions '}') | IndexingStuff+ -private IndexingStuff ::= WordWrapper | INTEGER_REG | FLOAT_REG | STRING_REG | ('{' IndexingStatementOptions+ '}') | +private IndexingStuff ::= WordWrapper | INTEGER_REG | FLOAT_REG | STRING_REG | ('{' IndexingStatement '}') | ':' | ('|' IndexingStatementOptions) | ';' | '.' | '(' | ')' | ARITHMETIC_OPERATOR | COMPARISON_OPERATOR // Attribute AttributeDefinition ::= attribute ((':' SimpleAttributeProperty) | ('{' (ComplexAttributeProperty | SimpleAttributeProperty)+ '}')) @@ -266,7 +267,7 @@ SimpleAttributeProperty ::= fast-search | fast-access | paged | mutable | enable ComplexAttributeProperty ::= AliasDefinition | SortingDefinition | DistanceMetricDef // Does not support zero-or-one occurrences DistanceMetricDef ::= distance-metric ':' IdentifierWithDashVal // Alias -AliasDefinition ::= alias (IdentifierWithDashVal ('.' IdentifierWithDashVal)*)? ':' IdentifierWithDashVal ('.' IdentifierWithDashVal)* +AliasDefinition ::= alias (IdentifierVal)? ':' IdentifierWithDashVal ('.' IdentifierWithDashVal)* // Stemming StemmingDefinition ::= stemming ':' IdentifierWithDashVal // Rank diff --git a/integration/intellij/src/main/jflex/ai/vespa/intellij/schema/lexer/sd.flex b/integration/intellij/src/main/jflex/ai/vespa/intellij/schema/lexer/sd.flex index b4491acc717..78334de1961 100644 --- a/integration/intellij/src/main/jflex/ai/vespa/intellij/schema/lexer/sd.flex +++ b/integration/intellij/src/main/jflex/ai/vespa/intellij/schema/lexer/sd.flex @@ -38,7 +38,7 @@ SYMBOL= [!$|:{}(),.\[\]] INTEGER = [0-9]+ FLOAT = {INTEGER}[.][0-9]+[e]? COMPARISON_OPERATOR = [<>]|(==)|(<=)|(>=)|(\~=) -ARITHMETIC_OPERATOR = [\-+*/] +ARITHMETIC_OPERATOR = [\-+*/%] STRING = \"([^\"\\]*(\\.[^\"\\]*)*)\" WORD = \w+ diff --git a/integration/intellij/src/main/resources/META-INF/pluginIcon.png b/integration/intellij/src/main/resources/META-INF/pluginIcon.png Binary files differdeleted file mode 100644 index c8dc4764e35..00000000000 --- a/integration/intellij/src/main/resources/META-INF/pluginIcon.png +++ /dev/null diff --git a/integration/intellij/src/main/resources/META-INF/pluginIcon.svg b/integration/intellij/src/main/resources/META-INF/pluginIcon.svg new file mode 100644 index 00000000000..3a795976efd --- /dev/null +++ b/integration/intellij/src/main/resources/META-INF/pluginIcon.svg @@ -0,0 +1,19 @@ +<?xml version="1.0" encoding="utf-8"?> +<svg id="Layer_1" data-name="Layer 1" viewBox="0 0 40 40" preserveAspectRatio="xMidYMin slice" xmlns="http://www.w3.org/2000/svg"> + <defs> + <style>.cls-1{fill:#ffc43c;}.cls-2{fill:url(#linear-gradient);}.cls-3{fill:#1a7db6;}.cls-4{fill:url(#linear-gradient-2);}.cls-5{fill:#303030;}</style> + <linearGradient id="linear-gradient" x1="964.75" y1="34.82" x2="919.68" y2="63.4" gradientTransform="matrix(-0.276923, 0, 0, 0.276923, 291.653198, 2)" gradientUnits="userSpaceOnUse"> + <stop offset="0" stop-color="#da9728"/> + <stop offset="0.54" stop-color="#ffc43c"/> + </linearGradient> + <linearGradient id="linear-gradient-2" x1="46.38" y1="94.56" x2="-29.88" y2="53" gradientUnits="userSpaceOnUse" gradientTransform="matrix(0.276923, 0, 0, 0.276923, 3.212999, 2)"> + <stop offset="0" stop-color="#005a8e"/> + <stop offset="0.54" stop-color="#1a7db6"/> + </linearGradient> + </defs> + <title>VespaLogo copy</title> + <polygon class="cls-1" points="22.736 2 8.685 11.543 8.685 27.971 22.736 18.064 36.787 27.971 36.787 11.543 22.736 2"/> + <polygon class="cls-2" points="22.736 18.064 22.736 2 36.787 11.543 36.787 27.97 22.736 18.064"/> + <polygon class="cls-3" points="17.264 21.936 3.213 12.029 3.213 28.457 17.264 38 31.315 28.457 31.315 12.029 17.264 21.936"/> + <polygon class="cls-4" points="17.264 21.936 17.264 38 3.213 28.457 3.213 12.03 17.264 21.936"/> +</svg>
\ No newline at end of file diff --git a/integration/intellij/src/main/resources/META-INF/pluginIcon_dark.png b/integration/intellij/src/main/resources/META-INF/pluginIcon_dark.png Binary files differdeleted file mode 100644 index c8dc4764e35..00000000000 --- a/integration/intellij/src/main/resources/META-INF/pluginIcon_dark.png +++ /dev/null @@ -85,6 +85,7 @@ <module>hosted-zone-api</module> <module>http-utils</module> <module>indexinglanguage</module> + <!--<module>integration/intellij</module>--> <module>jaxrs_client_utils</module> <module>jaxrs_utils</module> <module>jdisc-cloud-aws</module> |