diff options
437 files changed, 4836 insertions, 3316 deletions
diff --git a/Code-of-Conduct.md b/Code-of-Conduct.md deleted file mode 100644 index 2b04099b2ca..00000000000 --- a/Code-of-Conduct.md +++ /dev/null @@ -1,54 +0,0 @@ -<!-- Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. --> - -# Verizon Media Open Source Code of Conduct - -## Summary -This Code of Conduct is our way to encourage good behavior and discourage bad behavior in our open source projects. We invite participation from many people to bring different perspectives to our projects. We will do our part to foster a welcoming and professional environment free of harassment. We expect participants to communicate professionally and thoughtfully during their involvement with this project. - -Participants may lose their good standing by engaging in misconduct. For example: insulting, threatening, or conveying unwelcome sexual content. We ask participants who observe conduct issues to report the incident directly to the project's Response Team at opensource-conduct@verizonmedia.com. Verizon Media will assign a respondent to address the issue. We may remove harassers from this project. - -This code does not replace the terms of service or acceptable use policies of the websites used to support this project. We acknowledge that participants may be subject to additional conduct terms based on their employment which may govern their online expressions. - -## Details -This Code of Conduct makes our expectations of participants in this community explicit. -* We forbid harassment and abusive speech within this community. -* We request participants to report misconduct to the project’s Response Team. -* We urge participants to refrain from using discussion forums to play out a fight. - -### Expected Behaviors -We expect participants in this community to conduct themselves professionally. Since our primary mode of communication is text on an online forum (e.g. issues, pull requests, comments, emails, or chats) devoid of vocal tone, gestures, or other context that is often vital to understanding, it is important that participants are attentive to their interaction style. - -* **Assume positive intent.** We ask community members to assume positive intent on the part of other people’s communications. We may disagree on details, but we expect all suggestions to be supportive of the community goals. -* **Respect participants.** We expect occasional disagreements. Open Source projects are learning experiences. Ask, explore, challenge, and then _respectfully_ state if you agree or disagree. If your idea is rejected, be more persuasive not bitter. -* **Welcoming to new members.** New members bring new perspectives. Some ask questions that have been addressed before. _Kindly_ point to existing discussions. Everyone is new to every project once. -* **Be kind to beginners.** Beginners use open source projects to get experience. They might not be talented coders yet, and projects should not accept poor quality code. But we were all beginners once, and we need to engage kindly. -* **Consider your impact on others.** Your work will be used by others, and you depend on the work of others. We expect community members to be considerate and establish a balance their self-interest with communal interest. -* **Use words carefully.** We may not understand intent when you say something ironic. Often, people will misinterpret sarcasm in online communications. We ask community members to communicate plainly. -* **Leave with class.** When you wish to resign from participating in this project for any reason, you are free to fork the code and create a competitive project. Open Source explicitly allows this. Your exit should not be dramatic or bitter. - -### Unacceptable Behaviors -Participants remain in good standing when they do not engage in misconduct or harassment (some examples follow). We do not list all forms of harassment, nor imply some forms of harassment are not worthy of action. Any participant who *feels* harassed or *observes* harassment, should report the incident to the Response Team. -* **Don't be a bigot.** Calling out project members by their identity or background in a negative or insulting manner. This includes, but is not limited to, slurs or insinuations related to protected or suspect classes e.g. race, color, citizenship, national origin, political belief, religion, sexual orientation, gender identity and expression, age, size, culture, ethnicity, genetic features, language, profession, national minority status, mental or physical ability. -* **Don't insult.** Insulting remarks about a person’s lifestyle practices. -* **Don't dox.** Revealing private information about other participants without explicit permission. -* **Don't intimidate.** Threats of violence or intimidation of any project member. -* **Don't creep.** Unwanted sexual attention or content unsuited for the subject of this project. -* **Don't inflame.** We ask that victim of harassment not address their grievances in the public forum, as this often intensifies the problem. Report it, and let us address it off-line. -* **Don't disrupt.** Sustained disruptions in a discussion. - -### Reporting Issues -If you experience or witness misconduct, or have any other concerns about the conduct of members of this project, please report it by contacting our Response Team at opensource-conduct@verizonmedia.com who will handle your report with discretion. Your report should include: -* Your preferred contact information. We cannot process anonymous reports. -* Names (real or usernames) of those involved in the incident. -* Your account of what occurred, and if the incident is ongoing. Please provide links to or transcripts of the publicly available records (e.g. a mailing list archive or a public IRC logger), so that we can review it. -* Any additional information that may be helpful to achieve resolution. - -After filing a report, a representative will contact you directly to review the incident and ask additional questions. If a member of the Verizon Media Response Team is named in an incident report, that member will be recused from handling your incident. If the complaint originates from a member of the Response Team, it will be addressed by a different member of the Response Team. We will consider reports to be confidential for the purpose of protecting victims of abuse. - -### Scope -Verizon Media will assign a Response Team member with admin rights on the project and legal rights on the project copyright. The Response Team is empowered to restrict some privileges to the project as needed. Since this project is governed by an open source license, any participant may fork the code under the terms of the project license. The Response Team’s goal is to preserve the project if possible, and will restrict or remove participation from those who disrupt the project. - -This code does not replace the terms of service or acceptable use policies that are provided by the websites used to support this community. Nor does this code apply to communications or actions that take place outside of the context of this community. Many participants in this project are also subject to codes of conduct based on their employment. This code is a social-contract that informs participants of our social expectations. It is not a terms of service or legal contract. - -## License and Acknowledgment. -This text is shared under the [CC-BY-4.0 license](https://creativecommons.org/licenses/by/4.0/). This code is based on a study conducted by the [TODO Group](https://todogroup.org/) of many codes used in the open source community. If you have feedback about this code, contact our Response Team at the address listed above. diff --git a/ann_benchmark/src/vespa/ann_benchmark/setup.py b/ann_benchmark/src/vespa/ann_benchmark/setup.py index 2991c225823..74f4e2d7307 100644 --- a/ann_benchmark/src/vespa/ann_benchmark/setup.py +++ b/ann_benchmark/src/vespa/ann_benchmark/setup.py @@ -18,7 +18,7 @@ setup( name="vespa_ann_benchmark", version="0.1.0", author="Tor Egge", - author_email="Tor.Egge@verizonmedia.com", + author_email="Tor.Egge@yahooinc.com", description="Python binding for the Vespa implementation of an HNSW index for nearest neighbor search", long_description="Python binding for the Vespa implementation of an HNSW index for nearest neighbor search used for low-level benchmarking", ext_modules=[Extension("vespa_ann_benchmark", sources=[])], diff --git a/application/src/test/java/com/yahoo/application/container/ContainerSearchTest.java b/application/src/test/java/com/yahoo/application/container/ContainerSchemaTest.java index f65cf4e2675..40bdf023c63 100644 --- a/application/src/test/java/com/yahoo/application/container/ContainerSearchTest.java +++ b/application/src/test/java/com/yahoo/application/container/ContainerSchemaTest.java @@ -18,7 +18,7 @@ import static org.junit.Assert.assertThat; * @author gjoranv * @author ollivir */ -public class ContainerSearchTest { +public class ContainerSchemaTest { @Test public void processing_and_rendering_works() throws Exception { diff --git a/client/go/README.md b/client/go/README.md index 7b5b222503c..d3803828b4c 100644 --- a/client/go/README.md +++ b/client/go/README.md @@ -1,6 +1,13 @@ +<!-- Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. --> The command-line tool for Vespa.ai. Use it on Vespa instances running locally, remotely or in the cloud. Prefer web service API's to this in production. Vespa documentation: https://docs.vespa.ai + +Developer notes: + + $ brew install go + $ make test + diff --git a/client/go/cmd/clone.go b/client/go/cmd/clone.go index e7d77c83ebb..6550e24e094 100644 --- a/client/go/cmd/clone.go +++ b/client/go/cmd/clone.go @@ -7,6 +7,7 @@ package cmd import ( "archive/zip" "errors" + "fmt" "io" "log" "net/http" @@ -96,7 +97,7 @@ func cloneApplication(source string, name string) { } } if !found { - fatalErr(nil, "Could not find source application '", color.Cyan(source), "'") + fatalErrHint(fmt.Errorf("Could not find source application '%s'", color.Cyan(source)), "Use -f to ignore the cache") } else { log.Print("Created ", color.Cyan(name)) } diff --git a/client/go/cmd/testdata/applications/withEmptyTarget/pom.xml b/client/go/cmd/testdata/applications/withEmptyTarget/pom.xml index 2b1875a5532..bb7a232ea2a 100644 --- a/client/go/cmd/testdata/applications/withEmptyTarget/pom.xml +++ b/client/go/cmd/testdata/applications/withEmptyTarget/pom.xml @@ -1 +1,2 @@ -<services/>
\ No newline at end of file +<!-- Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. --> +<services/> diff --git a/client/go/cmd/testdata/applications/withTarget/pom.xml b/client/go/cmd/testdata/applications/withTarget/pom.xml index 2b1875a5532..bb7a232ea2a 100644 --- a/client/go/cmd/testdata/applications/withTarget/pom.xml +++ b/client/go/cmd/testdata/applications/withTarget/pom.xml @@ -1 +1,2 @@ -<services/>
\ No newline at end of file +<!-- Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. --> +<services/> diff --git a/client/src/main/java/ai/vespa/client/dsl/GeoLocation.java b/client/src/main/java/ai/vespa/client/dsl/GeoLocation.java index c0d8fabc42f..f60dd6bed6b 100644 --- a/client/src/main/java/ai/vespa/client/dsl/GeoLocation.java +++ b/client/src/main/java/ai/vespa/client/dsl/GeoLocation.java @@ -1,3 +1,4 @@ +// Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. package ai.vespa.client.dsl; import org.apache.commons.text.StringEscapeUtils; diff --git a/client/src/main/java/ai/vespa/client/dsl/NearestNeighbor.java b/client/src/main/java/ai/vespa/client/dsl/NearestNeighbor.java index 6c95d2b6fd7..3c3f33ea8da 100644 --- a/client/src/main/java/ai/vespa/client/dsl/NearestNeighbor.java +++ b/client/src/main/java/ai/vespa/client/dsl/NearestNeighbor.java @@ -1,3 +1,4 @@ +// Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. package ai.vespa.client.dsl; import java.util.stream.Collectors; diff --git a/client/src/main/java/ai/vespa/client/dsl/Q.java b/client/src/main/java/ai/vespa/client/dsl/Q.java index 32b79c84458..a709ec826fc 100644 --- a/client/src/main/java/ai/vespa/client/dsl/Q.java +++ b/client/src/main/java/ai/vespa/client/dsl/Q.java @@ -162,12 +162,17 @@ public final class Q { * Weakand weak and. * https://docs.vespa.ai/en/reference/query-language-reference.html#weakand * - * @param field the field * @param query the query * @return the weak and query */ - public static WeakAnd weakand(String field, Query query) { - return new WeakAnd(field, query); + public static WeakAnd weakand(Query query) { + return new WeakAnd(query); + } + + /** @deprecated use weakand(query) */ + @Deprecated // Remove on Vespa 8 + public static WeakAnd weakand(String ignored, Query query) { + return new WeakAnd(query); } /** diff --git a/client/src/main/java/ai/vespa/client/dsl/Text.java b/client/src/main/java/ai/vespa/client/dsl/Text.java index aca43d40b6a..34931b78d2f 100644 --- a/client/src/main/java/ai/vespa/client/dsl/Text.java +++ b/client/src/main/java/ai/vespa/client/dsl/Text.java @@ -1,15 +1,17 @@ // Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. - package ai.vespa.client.dsl; import java.util.Locale; /** * Text utility functions. + * * @author arnej - **/ + */ final class Text { + public static String format(String format, Object... args) { return String.format(Locale.US, format, args); } + } diff --git a/client/src/main/java/ai/vespa/client/dsl/WeakAnd.java b/client/src/main/java/ai/vespa/client/dsl/WeakAnd.java index 3059e4adea1..cbd796a6ad5 100644 --- a/client/src/main/java/ai/vespa/client/dsl/WeakAnd.java +++ b/client/src/main/java/ai/vespa/client/dsl/WeakAnd.java @@ -5,13 +5,10 @@ import java.util.stream.Collectors; public class WeakAnd extends QueryChain { - private String fieldName; private Annotation annotation; - private Query value; + private final Query value; - - WeakAnd(String fieldName, Query value) { - this.fieldName = fieldName; + WeakAnd(Query value) { this.value = value; this.nonEmpty = true; } @@ -55,8 +52,8 @@ public class WeakAnd extends QueryChain { boolean hasAnnotation = A.hasAnnotation(annotation); String s = - Text.format("weakAnd(%s, %s)", fieldName, - value.queries.stream().map(Object::toString).collect(Collectors.joining(", "))); + Text.format("weakAnd(%s)", + value.queries.stream().map(Object::toString).collect(Collectors.joining(", "))); return hasAnnotation ? Text.format("([%s]%s)", annotation, s) : s; } } diff --git a/client/src/test/groovy/ai/vespa/client/dsl/QTest.groovy b/client/src/test/groovy/ai/vespa/client/dsl/QTest.groovy index 1bada4e8f59..1040202f761 100644 --- a/client/src/test/groovy/ai/vespa/client/dsl/QTest.groovy +++ b/client/src/test/groovy/ai/vespa/client/dsl/QTest.groovy @@ -233,15 +233,15 @@ class QTest extends Specification { given: def q = Q.select("*") .from("sd1") - .where(Q.weakand("f1", Q.p("f1").contains("v1").and("f2").contains("v2"))) - .and(Q.weakand("f3", Q.p("f1").contains("v1").and("f2").contains("v2")) + .where(Q.weakand(Q.p("f1").contains("v1").and("f2").contains("v2"))) + .and(Q.weakand(Q.p("f1").contains("v1").and("f2").contains("v2")) .annotate(A.a("scoreThreshold", 0.13)) ) .semicolon() .build() expect: - q == """yql=select * from sd1 where weakAnd(f1, f1 contains "v1", f2 contains "v2") and ([{"scoreThreshold":0.13}]weakAnd(f3, f1 contains "v1", f2 contains "v2"));""" + q == """yql=select * from sd1 where weakAnd(f1 contains "v1", f2 contains "v2") and ([{"scoreThreshold":0.13}]weakAnd(f1 contains "v1", f2 contains "v2"));""" } def "geo location"() { diff --git a/clustercontroller-apps/src/main/java/com/yahoo/vespa/clustercontroller/apps/clustercontroller/ClusterController.java b/clustercontroller-apps/src/main/java/com/yahoo/vespa/clustercontroller/apps/clustercontroller/ClusterController.java index 32ad2d7d779..1eca2d18493 100644 --- a/clustercontroller-apps/src/main/java/com/yahoo/vespa/clustercontroller/apps/clustercontroller/ClusterController.java +++ b/clustercontroller-apps/src/main/java/com/yahoo/vespa/clustercontroller/apps/clustercontroller/ClusterController.java @@ -53,7 +53,7 @@ public class ClusterController extends AbstractComponent controllers.put(options.clusterName, controller); status.put(options.clusterName, statusPageServer); } else { - controller.updateOptions(options, 0); + controller.updateOptions(options); } } } diff --git a/clustercontroller-core/src/main/java/com/yahoo/vespa/clustercontroller/core/FleetController.java b/clustercontroller-core/src/main/java/com/yahoo/vespa/clustercontroller/core/FleetController.java index 10ba907b5a2..ebde8f3a98c 100644 --- a/clustercontroller-core/src/main/java/com/yahoo/vespa/clustercontroller/core/FleetController.java +++ b/clustercontroller-core/src/main/java/com/yahoo/vespa/clustercontroller/core/FleetController.java @@ -83,8 +83,6 @@ public class FleetController implements NodeStateOrHostInfoChangeHandler, NodeAd private final StatusPageServer.PatternRequestRouter statusRequestRouter = new StatusPageServer.PatternRequestRouter(); private final List<ClusterStateBundle> newStates = new ArrayList<>(); private final List<ClusterStateBundle> convergedStates = new ArrayList<>(); - private long configGeneration = -1; - private long nextConfigGeneration = -1; private final Queue<RemoteClusterControllerTask> remoteTasks = new LinkedList<>(); private final MetricUpdater metricUpdater; @@ -107,8 +105,6 @@ public class FleetController implements NodeStateOrHostInfoChangeHandler, NodeAd @Override public FleetControllerOptions getOptions() { return options; } @Override - public long getConfigGeneration() { return configGeneration; } - @Override public ContentCluster getCluster() { return cluster; } }; @@ -153,7 +149,7 @@ public class FleetController implements NodeStateOrHostInfoChangeHandler, NodeAd new LegacyNodePageRequestHandler(timer, eventLog, cluster)); this.statusRequestRouter.addHandler( "^/state.*", - new NodeHealthRequestHandler(dataExtractor)); + new NodeHealthRequestHandler()); this.statusRequestRouter.addHandler( "^/clusterstate", new ClusterStateRequestHandler(stateVersionTracker)); @@ -302,13 +298,12 @@ public class FleetController implements NodeStateOrHostInfoChangeHandler, NodeAd nodeLookup.shutdown(); } - public void updateOptions(FleetControllerOptions options, long configGeneration) { + public void updateOptions(FleetControllerOptions options) { var newId = FleetControllerId.fromOptions(options); synchronized(monitor) { assert newId.equals(context.id()); context.log(logger, Level.INFO, "FleetController has new options"); nextOptions = options.clone(); - nextConfigGeneration = configGeneration; monitor.notifyAll(); } } @@ -551,8 +546,6 @@ public class FleetController implements NodeStateOrHostInfoChangeHandler, NodeAd long currentTime = timer.getCurrentTimeInMillis(); nextStateSendTime = Math.min(currentTime + options.minTimeBetweenNewSystemStates, nextStateSendTime); - configGeneration = nextConfigGeneration; - nextConfigGeneration = -1; } private void selfTerminateIfConfiguredNodeIndexHasChanged() { diff --git a/clustercontroller-core/src/main/java/com/yahoo/vespa/clustercontroller/core/status/NodeHealthRequestHandler.java b/clustercontroller-core/src/main/java/com/yahoo/vespa/clustercontroller/core/status/NodeHealthRequestHandler.java index 39fab92d02c..6a683ce6c04 100644 --- a/clustercontroller-core/src/main/java/com/yahoo/vespa/clustercontroller/core/status/NodeHealthRequestHandler.java +++ b/clustercontroller-core/src/main/java/com/yahoo/vespa/clustercontroller/core/status/NodeHealthRequestHandler.java @@ -9,26 +9,17 @@ import com.yahoo.vespa.clustercontroller.core.status.statuspage.StatusPageServer */ public class NodeHealthRequestHandler implements StatusPageServer.RequestHandler { - private final RunDataExtractor data; - - public NodeHealthRequestHandler(RunDataExtractor data) { - this.data = data; - } + public NodeHealthRequestHandler() {} @Override public StatusPageResponse handle(StatusPageServer.HttpRequest request) { StatusPageResponse response = new StatusPageResponse(); response.setContentType("application/json"); response.writeContent("{\n" + - " \"status\" : {\n" + - " \"code\" : \"up\"\n" + - " },\n" + - " \"config\" : {\n" + - " \"component\" : {\n" + - " \"generation\" : " + data.getConfigGeneration() + "\n" + - " }\n" + - " }\n" + - "}"); + " \"status\" : {\n" + + " \"code\" : \"up\"\n" + + " }\n" + + "}"); return response; } diff --git a/clustercontroller-core/src/main/java/com/yahoo/vespa/clustercontroller/core/status/RunDataExtractor.java b/clustercontroller-core/src/main/java/com/yahoo/vespa/clustercontroller/core/status/RunDataExtractor.java index ee2b74372ae..599931ba13c 100644 --- a/clustercontroller-core/src/main/java/com/yahoo/vespa/clustercontroller/core/status/RunDataExtractor.java +++ b/clustercontroller-core/src/main/java/com/yahoo/vespa/clustercontroller/core/status/RunDataExtractor.java @@ -10,7 +10,6 @@ import com.yahoo.vespa.clustercontroller.core.FleetControllerOptions; public interface RunDataExtractor { FleetControllerOptions getOptions(); - long getConfigGeneration(); ContentCluster getCluster(); } diff --git a/clustercontroller-core/src/test/java/com/yahoo/vespa/clustercontroller/core/ClusterFeedBlockTest.java b/clustercontroller-core/src/test/java/com/yahoo/vespa/clustercontroller/core/ClusterFeedBlockTest.java index 7c4e05aa0e9..a52370a0654 100644 --- a/clustercontroller-core/src/test/java/com/yahoo/vespa/clustercontroller/core/ClusterFeedBlockTest.java +++ b/clustercontroller-core/src/test/java/com/yahoo/vespa/clustercontroller/core/ClusterFeedBlockTest.java @@ -135,8 +135,7 @@ public class ClusterFeedBlockTest extends FleetControllerTest { assertTrue(ctrl.getClusterStateBundle().clusterFeedIsBlocked()); // Increase cheese allowance. Should now automatically unblock since reported usage is lower. - int dummyConfigGeneration = 2; - ctrl.updateOptions(createOptions(mapOf(usage("cheese", 0.9), usage("wine", 0.4))), dummyConfigGeneration); + ctrl.updateOptions(createOptions(mapOf(usage("cheese", 0.9), usage("wine", 0.4)))); ctrl.tick(); // Options propagation ctrl.tick(); // State recomputation assertFalse(ctrl.getClusterStateBundle().clusterFeedIsBlocked()); diff --git a/clustercontroller-core/src/test/java/com/yahoo/vespa/clustercontroller/core/DistributionBitCountTest.java b/clustercontroller-core/src/test/java/com/yahoo/vespa/clustercontroller/core/DistributionBitCountTest.java index e38e48375ed..e9d3a9cf83e 100644 --- a/clustercontroller-core/src/test/java/com/yahoo/vespa/clustercontroller/core/DistributionBitCountTest.java +++ b/clustercontroller-core/src/test/java/com/yahoo/vespa/clustercontroller/core/DistributionBitCountTest.java @@ -40,12 +40,12 @@ public class DistributionBitCountTest extends FleetControllerTest { public void testDistributionBitCountConfigIncrease() throws Exception { setUpSystem("DistributionBitCountTest::testDistributionBitCountConfigIncrease"); options.distributionBits = 20; - fleetController.updateOptions(options, 0); + fleetController.updateOptions(options); ClusterState currentState = waitForState("version:\\d+ bits:20 distributor:10 storage:10"); int version = currentState.getVersion(); options.distributionBits = 23; - fleetController.updateOptions(options, 0); + fleetController.updateOptions(options); assertEquals(version, currentState.getVersion()); } @@ -56,7 +56,7 @@ public class DistributionBitCountTest extends FleetControllerTest { public void testDistributionBitCountConfigDecrease() throws Exception { setUpSystem("DistributionBitCountTest::testDistributionBitCountConfigDecrease"); options.distributionBits = 12; - fleetController.updateOptions(options, 0); + fleetController.updateOptions(options); waitForState("version:\\d+ bits:12 distributor:10 storage:10"); } diff --git a/clustercontroller-core/src/test/java/com/yahoo/vespa/clustercontroller/core/GroupAutoTakedownLiveConfigTest.java b/clustercontroller-core/src/test/java/com/yahoo/vespa/clustercontroller/core/GroupAutoTakedownLiveConfigTest.java index 77bcff1e7d5..e6c5c31010f 100644 --- a/clustercontroller-core/src/test/java/com/yahoo/vespa/clustercontroller/core/GroupAutoTakedownLiveConfigTest.java +++ b/clustercontroller-core/src/test/java/com/yahoo/vespa/clustercontroller/core/GroupAutoTakedownLiveConfigTest.java @@ -9,8 +9,6 @@ import static org.junit.Assert.assertFalse; public class GroupAutoTakedownLiveConfigTest extends FleetControllerTest { - private long mockConfigGeneration = 1; - private static FleetControllerOptions createOptions(DistributionBuilder.GroupBuilder groupBuilder, double minNodeRatio) { FleetControllerOptions options = defaultOptions("mycluster"); options.setStorageDistribution(DistributionBuilder.forHierarchicCluster(groupBuilder)); @@ -21,8 +19,7 @@ public class GroupAutoTakedownLiveConfigTest extends FleetControllerTest { } private void updateConfigLive(FleetControllerOptions newOptions) { - ++mockConfigGeneration; - this.fleetController.updateOptions(newOptions, mockConfigGeneration); + this.fleetController.updateOptions(newOptions); } private void reconfigureWithMinNodeRatio(double minNodeRatio) { diff --git a/clustercontroller-core/src/test/java/com/yahoo/vespa/clustercontroller/core/MasterElectionTest.java b/clustercontroller-core/src/test/java/com/yahoo/vespa/clustercontroller/core/MasterElectionTest.java index 0599baf7c7a..bf3efb17ddf 100644 --- a/clustercontroller-core/src/test/java/com/yahoo/vespa/clustercontroller/core/MasterElectionTest.java +++ b/clustercontroller-core/src/test/java/com/yahoo/vespa/clustercontroller/core/MasterElectionTest.java @@ -296,7 +296,7 @@ public class MasterElectionTest extends FleetControllerTest { for (FleetController fc : fleetControllers) { FleetControllerOptions myoptions = fc.getOptions(); myoptions.zooKeeperServerAddress = zooKeeperServer.getAddress(); - fc.updateOptions(myoptions, 0); + fc.updateOptions(myoptions); log.log(Level.INFO, "Should now have sent out new zookeeper server address " + myoptions.zooKeeperServerAddress + " to fleetcontroller " + myoptions.fleetControllerIndex); } timer.advanceTime(10 * 1000); // Wait long enough for fleetcontroller wanting to retry zookeeper connection @@ -447,7 +447,7 @@ public class MasterElectionTest extends FleetControllerTest { FleetControllerOptions newOptions = options.clone(); for (int i=0; i<fleetControllers.size(); ++i) { FleetControllerOptions nodeOptions = adjustConfig(newOptions, i, fleetControllers.size()); - fleetControllers.get(i).updateOptions(nodeOptions, 2); + fleetControllers.get(i).updateOptions(nodeOptions); } waitForMaster(0); log.log(Level.INFO, "SHUTTING DOWN FLEET CONTROLLER 0"); diff --git a/clustercontroller-core/src/test/java/com/yahoo/vespa/clustercontroller/core/NodeSlobrokConfigurationMembershipTest.java b/clustercontroller-core/src/test/java/com/yahoo/vespa/clustercontroller/core/NodeSlobrokConfigurationMembershipTest.java index c1f1d0e1bc1..ea539cc13e0 100644 --- a/clustercontroller-core/src/test/java/com/yahoo/vespa/clustercontroller/core/NodeSlobrokConfigurationMembershipTest.java +++ b/clustercontroller-core/src/test/java/com/yahoo/vespa/clustercontroller/core/NodeSlobrokConfigurationMembershipTest.java @@ -46,7 +46,7 @@ public class NodeSlobrokConfigurationMembershipTest extends FleetControllerTest // cluster. If we do not re-fetch state from slobrok we risk racing nodeIndices.add(foreignNode); options.nodes = asConfiguredNodes(nodeIndices); - fleetController.updateOptions(options, 0); + fleetController.updateOptions(options); // Need to treat cluster as having 6 nodes due to ideal state algo semantics. // Note that we do not use subsetWaiter here since we want node 6 included. waitForState("version:\\d+ distributor:7 .4.s:d .5.s:d storage:7 .4.s:d .5.s:d"); @@ -65,13 +65,13 @@ public class NodeSlobrokConfigurationMembershipTest extends FleetControllerTest assertTrue(configuredNodes.remove(new ConfiguredNode(0, false))); configuredNodes.add(new ConfiguredNode(0, true)); options.nodes = configuredNodes; - fleetController.updateOptions(options, 0); + fleetController.updateOptions(options); waitForState("version:\\d+ distributor:4 storage:4 .0.s:r"); // Now remove the retired node entirely from config assertTrue(configuredNodes.remove(new ConfiguredNode(0, true))); - fleetController.updateOptions(options, 0); + fleetController.updateOptions(options); // The previously retired node should now be marked as down, as it no longer // exists from the point of view of the content cluster. We have to use a subset diff --git a/clustercontroller-core/src/test/java/com/yahoo/vespa/clustercontroller/core/RpcServerTest.java b/clustercontroller-core/src/test/java/com/yahoo/vespa/clustercontroller/core/RpcServerTest.java index 05ee96500d5..fee1d33725c 100644 --- a/clustercontroller-core/src/test/java/com/yahoo/vespa/clustercontroller/core/RpcServerTest.java +++ b/clustercontroller-core/src/test/java/com/yahoo/vespa/clustercontroller/core/RpcServerTest.java @@ -346,7 +346,7 @@ public class RpcServerTest extends FleetControllerTest { options.slobrokConnectionSpecs = this.options.slobrokConnectionSpecs; this.options.maxInitProgressTime = 30000; this.options.stableStateTimePeriod = 60000; - fleetController.updateOptions(options, 0); + fleetController.updateOptions(options); waitForState("version:\\d+ distributor:7 storage:7 .0.s:m .1.s:m .2.s:r .3.s:r .4.s:r"); } @@ -376,7 +376,7 @@ public class RpcServerTest extends FleetControllerTest { options.slobrokConnectionSpecs = this.options.slobrokConnectionSpecs; this.options.maxInitProgressTime = 30000; this.options.stableStateTimePeriod = 60000; - fleetController.updateOptions(options, 0); + fleetController.updateOptions(options); waitForState("version:\\d+ distributor:7 storage:7 .0.s:m .1.s:m"); } @@ -415,7 +415,7 @@ public class RpcServerTest extends FleetControllerTest { options.slobrokConnectionSpecs = this.options.slobrokConnectionSpecs; this.options.maxInitProgressTime = 30000; this.options.stableStateTimePeriod = 60000; - fleetController.updateOptions(options, 0); + fleetController.updateOptions(options); waitForState("version:\\d+ distributor:5 storage:5"); } @@ -430,7 +430,7 @@ public class RpcServerTest extends FleetControllerTest { options.slobrokConnectionSpecs = this.options.slobrokConnectionSpecs; this.options.maxInitProgressTime = 30000; this.options.stableStateTimePeriod = 60000; - fleetController.updateOptions(options, 0); + fleetController.updateOptions(options); waitForState("version:\\d+ distributor:7 storage:7 .0.s:r .1.s:r .2.s:r .3.s:r .4.s:r"); } @@ -444,7 +444,7 @@ public class RpcServerTest extends FleetControllerTest { options.slobrokConnectionSpecs = this.options.slobrokConnectionSpecs; this.options.maxInitProgressTime = 30000; this.options.stableStateTimePeriod = 60000; - fleetController.updateOptions(options, 0); + fleetController.updateOptions(options); waitForState("version:\\d+ distributor:7 storage:7 .0.s:r .1.s:r .2.s:r .3.s:r .4.s:r"); } diff --git a/config-model/src/main/java/com/yahoo/config/model/deploy/SearchDocumentModel.java b/config-model/src/main/java/com/yahoo/config/model/deploy/SearchDocumentModel.java index d4b7004a1b6..604e2b0a44b 100644 --- a/config-model/src/main/java/com/yahoo/config/model/deploy/SearchDocumentModel.java +++ b/config-model/src/main/java/com/yahoo/config/model/deploy/SearchDocumentModel.java @@ -1,6 +1,7 @@ // Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. package com.yahoo.config.model.deploy; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.SearchBuilder; import com.yahoo.vespa.documentmodel.DocumentModel; import com.yahoo.vespa.model.search.NamedSchema; @@ -35,16 +36,16 @@ public class SearchDocumentModel { public static SearchDocumentModel fromBuilderAndNames(SearchBuilder builder, Map<String, String> names) { List<NamedSchema> ret = new ArrayList<>(); - for (com.yahoo.searchdefinition.Search search : builder.getSearchList()) { - ret.add(new NamedSchema(names.get(search.getName()), search)); + for (Schema schema : builder.getSearchList()) { + ret.add(new NamedSchema(names.get(schema.getName()), schema)); } return new SearchDocumentModel(builder.getModel(), ret); } public static SearchDocumentModel fromBuilder(SearchBuilder builder) { List<NamedSchema> ret = new ArrayList<>(); - for (com.yahoo.searchdefinition.Search search : builder.getSearchList()) { - ret.add(new NamedSchema(search.getName(), search)); + for (Schema schema : builder.getSearchList()) { + ret.add(new NamedSchema(schema.getName(), schema)); } return new SearchDocumentModel(builder.getModel(), ret); } diff --git a/config-model/src/main/java/com/yahoo/config/model/test/MockApplicationPackage.java b/config-model/src/main/java/com/yahoo/config/model/test/MockApplicationPackage.java index b3da8ca14b9..1ef5dcef411 100644 --- a/config-model/src/main/java/com/yahoo/config/model/test/MockApplicationPackage.java +++ b/config-model/src/main/java/com/yahoo/config/model/test/MockApplicationPackage.java @@ -96,7 +96,7 @@ public class MockApplicationPackage implements ApplicationPackage { protected File root() { return root; } @Override - @SuppressWarnings("deprecation") + @SuppressWarnings("deprecation") // NOT redundant public String getApplicationName() { return "mock application"; } @@ -206,6 +206,15 @@ public class MockApplicationPackage implements ApplicationPackage { .withSchemaDir(dir).build(); } + // TODO: It might work to just merge this and the above + public static ApplicationPackage fromSearchDefinitionAndRootDirectory(String dir) { + return new MockApplicationPackage.Builder() + .withRoot(new File(dir)) + .withEmptyHosts() + .withEmptyServices() + .withSchemaDir(dir).build(); + } + public static class Builder { private File root = new File("nonexisting"); diff --git a/config-model/src/main/java/com/yahoo/documentmodel/NewDocumentType.java b/config-model/src/main/java/com/yahoo/documentmodel/NewDocumentType.java index b9e46989fa3..08a0f8b9882 100644 --- a/config-model/src/main/java/com/yahoo/documentmodel/NewDocumentType.java +++ b/config-model/src/main/java/com/yahoo/documentmodel/NewDocumentType.java @@ -10,7 +10,7 @@ import com.yahoo.document.annotation.AnnotationType; import com.yahoo.document.annotation.AnnotationTypeRegistry; import com.yahoo.document.datatypes.FieldValue; import com.yahoo.searchdefinition.FieldSets; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.document.FieldSet; import com.yahoo.searchdefinition.processing.BuiltInFieldSets; @@ -22,6 +22,7 @@ import java.util.LinkedHashMap; import java.util.LinkedHashSet; import java.util.LinkedList; import java.util.Map; +import java.util.Optional; import java.util.Set; import static java.util.Collections.emptySet; @@ -53,7 +54,7 @@ public final class NewDocumentType extends StructuredDataType implements DataTyp this( name, new StructDataType(name.getName() + ".header"), - new FieldSets(), + new FieldSets(Optional.empty()), documentReferences, importedFieldNames); } @@ -337,7 +338,7 @@ public final class NewDocumentType extends StructuredDataType implements DataTyp return this; } - /** The field sets defined for this type and its {@link Search} */ + /** The field sets defined for this type and its {@link Schema} */ public Set<FieldSet> getFieldSets() { return Collections.unmodifiableSet(fieldSets); } diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/Application.java b/config-model/src/main/java/com/yahoo/searchdefinition/Application.java new file mode 100644 index 00000000000..19a536a18f7 --- /dev/null +++ b/config-model/src/main/java/com/yahoo/searchdefinition/Application.java @@ -0,0 +1,63 @@ +// Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. +package com.yahoo.searchdefinition; + +import com.yahoo.config.application.api.ApplicationPackage; +import com.yahoo.config.application.api.DeployLogger; + +import java.util.Collections; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; + +/** + * A collection of objects representing the content of an application package. + * This is created, then added to, and lastly validated when there is no more content to add. + * At that point it is ready to use for deriving configuration. + * + * @author bratseth + */ +public class Application { + + private final ApplicationPackage applicationPackage; + private final RankProfileRegistry rankProfileRegistry; + private final Map<String, Schema> schemas = new LinkedHashMap<>(); + + public Application(ApplicationPackage applicationPackage) { + this(applicationPackage, new RankProfileRegistry()); + } + + // TODO: Almost sure the rank profile registry passed is always new RankProfileRegistry() (apart from in some tests), so remove the parameter + public Application(ApplicationPackage applicationPackage, RankProfileRegistry rankProfileRegistry) { + this.applicationPackage = applicationPackage; + this.rankProfileRegistry = rankProfileRegistry; + } + + public ApplicationPackage applicationPackage() { return applicationPackage; } + + public RankProfileRegistry rankProfileRegistry() { return rankProfileRegistry; } + + public void add(Schema schema) { + if (schemas.containsKey(schema.getName())) + throw new IllegalArgumentException("Duplicate schema '" + schema.getName() + "' in " + this); + schemas.put(schema.getName(), schema); + } + + /** Returns an unmodifiable list of the schemas of this application */ + public Map<String, Schema> schemas() { return Collections.unmodifiableMap(schemas); } + + /** Used by SearchBuilder, for now */ + void replaceSchemasBy(List<Schema> schemas) { + this.schemas.clear(); + for (var schema : schemas) + this.schemas.put(schema.getName(), schema); + } + + /** Validates this. Must be called after all content is added to it. */ + public void validate(DeployLogger logger) { + schemas.values().forEach(schema -> schema.validate(logger)); + } + + @Override + public String toString() { return "application " + applicationPackage.getApplicationId(); } + +} diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/DefaultRankProfile.java b/config-model/src/main/java/com/yahoo/searchdefinition/DefaultRankProfile.java index da611513570..56a739ced8b 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/DefaultRankProfile.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/DefaultRankProfile.java @@ -19,8 +19,8 @@ public class DefaultRankProfile extends RankProfile { * * @param rankProfileRegistry The {@link com.yahoo.searchdefinition.RankProfileRegistry} to use for storing and looking up rank profiles. */ - public DefaultRankProfile(Search search, RankProfileRegistry rankProfileRegistry, RankingConstants rankingConstants) { - super("default", search, rankProfileRegistry, rankingConstants); + public DefaultRankProfile(Schema schema, RankProfileRegistry rankProfileRegistry, RankingConstants rankingConstants) { + super("default", schema, rankProfileRegistry, rankingConstants); } /** diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/DistributableResource.java b/config-model/src/main/java/com/yahoo/searchdefinition/DistributableResource.java index 472bc9d5413..08194c578e7 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/DistributableResource.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/DistributableResource.java @@ -1,3 +1,4 @@ +// Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. package com.yahoo.searchdefinition; import com.yahoo.config.FileReference; diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/DocumentModelBuilder.java b/config-model/src/main/java/com/yahoo/searchdefinition/DocumentModelBuilder.java index e7d0b72d46b..14404528acc 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/DocumentModelBuilder.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/DocumentModelBuilder.java @@ -57,17 +57,17 @@ public class DocumentModelBuilder { return scratchInheritsMap.isEmpty(); } - public void addToModel(Collection<Search> searchList) { + public void addToModel(Collection<Schema> schemaList) { List<SDDocumentType> docList = new LinkedList<>(); - for (Search search : searchList) { - docList.add(search.getDocument()); + for (Schema schema : schemaList) { + docList.add(schema.getDocument()); } docList = sortDocumentTypes(docList); addDocumentTypes(docList); - for (Collection<Search> toAdd = tryAdd(searchList); - ! toAdd.isEmpty() && (toAdd.size() < searchList.size()); - toAdd = tryAdd(searchList)) { - searchList = toAdd; + for (Collection<Schema> toAdd = tryAdd(schemaList); + ! toAdd.isEmpty() && (toAdd.size() < schemaList.size()); + toAdd = tryAdd(schemaList)) { + schemaList = toAdd; } } @@ -116,26 +116,26 @@ public class DocumentModelBuilder { return out.toString(); } - private Collection<Search> tryAdd(Collection<Search> searchList) { - Collection<Search> left = new ArrayList<>(); - for (Search search : searchList) { + private Collection<Schema> tryAdd(Collection<Schema> schemaList) { + Collection<Schema> left = new ArrayList<>(); + for (Schema schema : schemaList) { try { - addToModel(search); + addToModel(schema); } catch (RetryLaterException e) { - left.add(search); + left.add(schema); } } return left; } - public void addToModel(Search search) { + public void addToModel(Schema schema) { // Then we add the search specific stuff - SearchDef searchDef = new SearchDef(search.getName()); - addSearchFields(search.extraFieldList(), searchDef); - for (Field f : search.getDocument().fieldSet()) { + SearchDef searchDef = new SearchDef(schema.getName()); + addSearchFields(schema.extraFieldList(), searchDef); + for (Field f : schema.getDocument().fieldSet()) { addSearchField((SDField) f, searchDef); } - for (SDField field : search.allConcreteFields()) { + for (SDField field : schema.allConcreteFields()) { for (Attribute attribute : field.getAttributes().values()) { if ( ! searchDef.getFields().containsKey(attribute.getName())) { searchDef.add(new SearchField(new Field(attribute.getName(), field), !field.getIndices().isEmpty(), true)); @@ -143,7 +143,7 @@ public class DocumentModelBuilder { } } - for (Field f : search.getDocument().fieldSet()) { + for (Field f : schema.getDocument().fieldSet()) { addAlias((SDField) f, searchDef); } model.getSearchManager().add(searchDef); @@ -354,10 +354,10 @@ public class DocumentModelBuilder { throw new IllegalArgumentException("Data type '" + sdoc.getName() + "' is not a struct => tostring='" + sdoc.toString() + "'."); } } - for (AnnotationType annotation : sdoc.getAnnotations()) { + for (AnnotationType annotation : sdoc.getAnnotations().values()) { dt.add(annotation); } - for (AnnotationType annotation : sdoc.getAnnotations()) { + for (AnnotationType annotation : sdoc.getAnnotations().values()) { SDAnnotationType sa = (SDAnnotationType) annotation; if (annotation.getInheritedTypes().isEmpty() && (sa.getInherits() != null) ) { annotationInheritance.put(annotation, sa.getInherits()); diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/DocumentOnlySearch.java b/config-model/src/main/java/com/yahoo/searchdefinition/DocumentOnlySchema.java index d10aa26f429..c672b662874 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/DocumentOnlySearch.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/DocumentOnlySchema.java @@ -1,7 +1,6 @@ // Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. package com.yahoo.searchdefinition; -import com.yahoo.config.application.api.ApplicationPackage; import com.yahoo.config.application.api.DeployLogger; import com.yahoo.config.application.api.FileRegistry; import com.yahoo.config.model.api.ModelContext; @@ -13,10 +12,10 @@ import com.yahoo.searchdefinition.document.SDDocumentType; * * @author vegardh */ -public class DocumentOnlySearch extends Search { +public class DocumentOnlySchema extends Schema { - public DocumentOnlySearch(ApplicationPackage applicationPackage, FileRegistry fileRegistry, DeployLogger deployLogger, ModelContext.Properties properties) { - super(applicationPackage, fileRegistry, deployLogger, properties); + public DocumentOnlySchema(Application application, FileRegistry fileRegistry, DeployLogger deployLogger, ModelContext.Properties properties) { + super(application, fileRegistry, deployLogger, properties); } @Override diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/DocumentReference.java b/config-model/src/main/java/com/yahoo/searchdefinition/DocumentReference.java index 2b0ade3797a..145effcdc9d 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/DocumentReference.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/DocumentReference.java @@ -11,18 +11,18 @@ import com.yahoo.document.Field; public class DocumentReference { private final Field referenceField; - private final Search targetSearch; + private final Schema targetSchema; - public DocumentReference(Field referenceField, Search targetSearch) { + public DocumentReference(Field referenceField, Schema targetSchema) { this.referenceField = referenceField; - this.targetSearch = targetSearch; + this.targetSchema = targetSchema; } public Field referenceField() { return referenceField; } - public Search targetSearch() { - return targetSearch; + public Schema targetSearch() { + return targetSchema; } } diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/DocumentReferenceResolver.java b/config-model/src/main/java/com/yahoo/searchdefinition/DocumentReferenceResolver.java index 95fdd5dc297..26c140e58fe 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/DocumentReferenceResolver.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/DocumentReferenceResolver.java @@ -7,7 +7,6 @@ import com.yahoo.searchdefinition.document.SDDocumentType; import com.yahoo.searchdefinition.document.SDField; import java.util.Collection; -import java.util.List; import java.util.Map; import java.util.stream.Stream; @@ -24,9 +23,9 @@ import static java.util.stream.Collectors.toMap; */ public class DocumentReferenceResolver { - private final Map<String, Search> searchMapping; + private final Map<String, Schema> searchMapping; - public DocumentReferenceResolver(List<Search> schemas) { + public DocumentReferenceResolver(Collection<Schema> schemas) { this.searchMapping = createDocumentNameToSearchMapping(schemas); } @@ -64,13 +63,13 @@ public class DocumentReferenceResolver { } ReferenceDataType reference = (ReferenceDataType) field.getDataType(); String targetDocumentName = getTargetDocumentName(reference); - Search search = searchMapping.get(targetDocumentName); - if (search == null) { + Schema schema = searchMapping.get(targetDocumentName); + if (schema == null) { throw new IllegalArgumentException( String.format("Invalid document reference '%s': " + "Could not find document type '%s'", field.getName(), targetDocumentName)); } - return new DocumentReference(field, search); + return new DocumentReference(field, schema); } private static boolean isAttribute(Field field) { @@ -78,10 +77,10 @@ public class DocumentReferenceResolver { return sdField.doesAttributing(); } - private static Map<String, Search> createDocumentNameToSearchMapping(List<Search> searchDefintions) { - return searchDefintions.stream() - .filter(search -> search.getDocument() != null) - .collect(toMap(search -> search.getDocument().getName(), identity())); + private static Map<String, Schema> createDocumentNameToSearchMapping(Collection<Schema> schemaDefintions) { + return schemaDefintions.stream() + .filter(search -> search.getDocument() != null) + .collect(toMap(search -> search.getDocument().getName(), identity())); } private static Stream<Field> fieldStream(SDDocumentType documentType) { diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/DocumentsOnlyRankProfile.java b/config-model/src/main/java/com/yahoo/searchdefinition/DocumentsOnlyRankProfile.java index a6694ed7bff..acab2b96772 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/DocumentsOnlyRankProfile.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/DocumentsOnlyRankProfile.java @@ -14,9 +14,9 @@ import java.util.List; */ public class DocumentsOnlyRankProfile extends RankProfile { - public DocumentsOnlyRankProfile(String name, Search search, RankProfileRegistry rankProfileRegistry, + public DocumentsOnlyRankProfile(String name, Schema schema, RankProfileRegistry rankProfileRegistry, RankingConstants rankingConstants) { - super(name, search, rankProfileRegistry, rankingConstants); + super(name, schema, rankProfileRegistry, rankingConstants); } @Override diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/FieldOperationApplierForSearch.java b/config-model/src/main/java/com/yahoo/searchdefinition/FieldOperationApplierForSearch.java index 6460fdcdbc7..52110e95ac7 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/FieldOperationApplierForSearch.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/FieldOperationApplierForSearch.java @@ -14,8 +14,8 @@ public class FieldOperationApplierForSearch extends FieldOperationApplier { //Do nothing } - public void process(Search search) { - for (Field field : search.extraFieldList()) { + public void process(Schema schema) { + for (Field field : schema.extraFieldList()) { apply(field); } } diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/FieldSets.java b/config-model/src/main/java/com/yahoo/searchdefinition/FieldSets.java index ba015ebaba6..065ade9b094 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/FieldSets.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/FieldSets.java @@ -4,19 +4,28 @@ package com.yahoo.searchdefinition; import java.util.Collections; import java.util.LinkedHashMap; import java.util.Map; +import java.util.Optional; +import java.util.stream.Collectors; import com.yahoo.searchdefinition.document.FieldSet; /** - * The field sets owned by a {@link Search} + * The field sets owned by a {@link Schema} * Both built in and user defined. * * @author vegardh */ public class FieldSets { - private final Map<String, FieldSet> userFieldSets = new LinkedHashMap<>(); - private final Map<String, FieldSet> builtInFieldSets = new LinkedHashMap<>(); + private final Optional<Schema> owner; + private final Map<String, FieldSet> userFieldSets; + private final Map<String, FieldSet> builtInFieldSets; + + public FieldSets(Optional<Schema> owner) { + this.owner = owner; + userFieldSets = new LinkedHashMap<>(); + builtInFieldSets = new LinkedHashMap<>(); + } /** * Adds an entry to user field sets, creating entries as needed @@ -48,12 +57,22 @@ public class FieldSets { /** Returns the built in field sets, unmodifiable */ public Map<String, FieldSet> builtInFieldSets() { - return Collections.unmodifiableMap(builtInFieldSets); + if (owner.isEmpty() || owner.get().inherited().isEmpty()) return Collections.unmodifiableMap(builtInFieldSets); + if (builtInFieldSets.isEmpty()) return owner.get().inherited().get().fieldSets().builtInFieldSets(); + + var fieldSets = new LinkedHashMap<>(owner.get().inherited().get().fieldSets().builtInFieldSets()); + fieldSets.putAll(builtInFieldSets); + return Collections.unmodifiableMap(fieldSets); } /** Returns the user defined field sets, unmodifiable */ public Map<String, FieldSet> userFieldSets() { - return Collections.unmodifiableMap(userFieldSets); + if (owner.isEmpty() || owner.get().inherited().isEmpty()) return Collections.unmodifiableMap(userFieldSets); + if (userFieldSets.isEmpty()) return owner.get().inherited().get().fieldSets().userFieldSets(); + + var fieldSets = new LinkedHashMap<>(owner.get().inherited().get().fieldSets().userFieldSets()); + fieldSets.putAll(userFieldSets); + return Collections.unmodifiableMap(fieldSets); } } diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/ImmutableSearch.java b/config-model/src/main/java/com/yahoo/searchdefinition/ImmutableSchema.java index 605176e0c07..d0f958741fe 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/ImmutableSearch.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/ImmutableSchema.java @@ -12,17 +12,19 @@ import com.yahoo.vespa.documentmodel.SummaryField; import java.io.Reader; import java.util.List; import java.util.Map; +import java.util.Optional; import java.util.stream.Stream; /** - * An interface containing the non-mutating methods of {@link Search}. - * For description of the methods see {@link Search}. + * An interface containing the non-mutating methods of {@link Schema}. + * For description of the methods see {@link Schema}. * * @author bjorncs */ -public interface ImmutableSearch { +public interface ImmutableSchema { String getName(); + Optional<? extends ImmutableSchema> inherited(); Index getIndex(String name); ImmutableSDField getConcreteField(String name); //TODO split in mutating/immutable by returning List<ImmutableSDField> @@ -45,4 +47,5 @@ public interface ImmutableSearch { List<ImmutableSDField> allFieldsList(); Map<String, SummaryField> getSummaryFields(ImmutableSDField field); + } diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/ImportedFieldsEnumerator.java b/config-model/src/main/java/com/yahoo/searchdefinition/ImportedFieldsEnumerator.java index 25cdd1e08cd..a1248409368 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/ImportedFieldsEnumerator.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/ImportedFieldsEnumerator.java @@ -3,7 +3,7 @@ package com.yahoo.searchdefinition; import com.yahoo.searchdefinition.document.SDDocumentType; -import java.util.List; +import java.util.Collection; /** * Enumerates and emplaces a set of all imported fields into a SDDocumentType from @@ -11,9 +11,9 @@ import java.util.List; */ public class ImportedFieldsEnumerator { - private final List<Search> schemas; + private final Collection<Schema> schemas; - public ImportedFieldsEnumerator(List<Search> schemas) { + public ImportedFieldsEnumerator(Collection<Schema> schemas) { this.schemas = schemas; } diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/Index.java b/config-model/src/main/java/com/yahoo/searchdefinition/Index.java index bc1800f7ad7..793fcc59f9d 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/Index.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/Index.java @@ -88,11 +88,11 @@ public class Index implements Cloneable, Serializable { * Whether this field should be stemmed in this search definition, * this is never null */ - public Stemming getStemming(Search search) { + public Stemming getStemming(Schema schema) { if (stemming != null) return stemming; else - return search.getStemming(); + return schema.getStemming(); } /** diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/LargeRankExpressions.java b/config-model/src/main/java/com/yahoo/searchdefinition/LargeRankExpressions.java index 76ad7e9a4ed..c33400fdcc1 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/LargeRankExpressions.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/LargeRankExpressions.java @@ -1,3 +1,4 @@ +// Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. package com.yahoo.searchdefinition; import com.yahoo.config.application.api.FileRegistry; diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/OnnxModels.java b/config-model/src/main/java/com/yahoo/searchdefinition/OnnxModels.java index 9c1ee2bb609..e249fe59f14 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/OnnxModels.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/OnnxModels.java @@ -8,6 +8,7 @@ import java.util.Collection; import java.util.Collections; import java.util.HashMap; import java.util.Map; +import java.util.Optional; /** * ONNX models tied to a search definition or global. @@ -16,11 +17,16 @@ import java.util.Map; */ public class OnnxModels { - private final Map<String, OnnxModel> models = new HashMap<>(); private final FileRegistry fileRegistry; - public OnnxModels(FileRegistry fileRegistry) { + /** The schema this belongs to, or empty if it is global */ + private final Optional<Schema> owner; + + private final Map<String, OnnxModel> models = new HashMap<>(); + + public OnnxModels(FileRegistry fileRegistry, Optional<Schema> owner) { this.fileRegistry = fileRegistry; + this.owner = owner; } public void add(OnnxModel model) { @@ -35,20 +41,34 @@ public class OnnxModels { } public OnnxModel get(String name) { - return models.get(name); + var model = models.get(name); + if (model != null) return model; + if (owner.isPresent() && owner.get().inherited().isPresent()) + return owner.get().inherited().get().onnxModels().get(name); + return null; } public boolean has(String name) { - return models.containsKey(name); + boolean has = models.containsKey(name); + if (has) return true; + if (owner.isPresent() && owner.get().inherited().isPresent()) + return owner.get().inherited().get().onnxModels().has(name); + return false; } public Map<String, OnnxModel> asMap() { - return Collections.unmodifiableMap(models); + // Shortcuts + if (owner.isEmpty() || owner.get().inherited().isEmpty()) return Collections.unmodifiableMap(models); + if (models.isEmpty()) return owner.get().inherited().get().onnxModels().asMap(); + + var allModels = new HashMap<>(owner.get().inherited().get().onnxModels().asMap()); + allModels.putAll(models); + return Collections.unmodifiableMap(allModels); } /** Initiate sending of these models to some services over file distribution */ public void sendTo(Collection<? extends AbstractService> services) { - models.values().forEach(model -> model.sendTo(services)); + asMap().values().forEach(model -> model.sendTo(services)); } } diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/RankExpressionBody.java b/config-model/src/main/java/com/yahoo/searchdefinition/RankExpressionBody.java index 8c6830de815..b3dec554cdc 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/RankExpressionBody.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/RankExpressionBody.java @@ -1,3 +1,4 @@ +// Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. package com.yahoo.searchdefinition; import java.nio.ByteBuffer; diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/RankProfile.java b/config-model/src/main/java/com/yahoo/searchdefinition/RankProfile.java index 68faa462fdd..7f3b018d569 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/RankProfile.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/RankProfile.java @@ -61,7 +61,7 @@ public class RankProfile implements Cloneable { private final String name; /** The search definition owning this profile, or null if global (owned by a model) */ - private final ImmutableSearch search; + private final ImmutableSchema search; /** The name of the rank profile inherited by this */ private String inheritedName = null; @@ -147,18 +147,18 @@ public class RankProfile implements Cloneable { * Creates a new rank profile for a particular search definition * * @param name the name of the new profile - * @param search the search definition owning this profile + * @param schema the search definition owning this profile * @param rankProfileRegistry the {@link com.yahoo.searchdefinition.RankProfileRegistry} to use for storing * and looking up rank profiles. */ - public RankProfile(String name, Search search, RankProfileRegistry rankProfileRegistry, RankingConstants rankingConstants) { + public RankProfile(String name, Schema schema, RankProfileRegistry rankProfileRegistry, RankingConstants rankingConstants) { this.name = Objects.requireNonNull(name, "name cannot be null"); - this.search = Objects.requireNonNull(search, "search cannot be null"); + this.search = Objects.requireNonNull(schema, "search cannot be null"); this.onnxModels = null; this.rankingConstants = rankingConstants; this.rankProfileRegistry = rankProfileRegistry; - this.applicationPackage = search.applicationPackage(); - this.deployLogger = search.getDeployLogger(); + this.applicationPackage = schema.applicationPackage(); + this.deployLogger = schema.getDeployLogger(); } /** @@ -166,7 +166,8 @@ public class RankProfile implements Cloneable { * * @param name the name of the new profile */ - public RankProfile(String name, ApplicationPackage applicationPackage, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, RankingConstants rankingConstants, OnnxModels onnxModels) { + public RankProfile(String name, ApplicationPackage applicationPackage, DeployLogger deployLogger, + RankProfileRegistry rankProfileRegistry, RankingConstants rankingConstants, OnnxModels onnxModels) { this.name = Objects.requireNonNull(name, "name cannot be null"); this.search = null; this.rankProfileRegistry = rankProfileRegistry; @@ -179,7 +180,7 @@ public class RankProfile implements Cloneable { public String getName() { return name; } /** Returns the search definition owning this, or null if it is global */ - public ImmutableSearch getSearch() { return search; } + public ImmutableSchema getSearch() { return search; } /** Returns the application this is part of */ public ApplicationPackage applicationPackage() { @@ -254,8 +255,8 @@ public class RankProfile implements Cloneable { } } - private RankProfile resolveInherited(ImmutableSearch search) { - SDDocumentType documentType = search.getDocument(); + private RankProfile resolveInherited(ImmutableSchema schema) { + SDDocumentType documentType = schema.getDocument(); if (documentType != null) { if (name.equals(inheritedName)) { // If you seemingly inherit yourself, you are actually referencing a rank-profile in one of your inherited schemas @@ -266,7 +267,7 @@ public class RankProfile implements Cloneable { } return rankProfileRegistry.resolve(documentType, inheritedName); } - return rankProfileRegistry.get(search.getName(), inheritedName); + return rankProfileRegistry.get(schema.getName(), inheritedName); } private RankProfile resolveInherited() { diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/RankProfileRegistry.java b/config-model/src/main/java/com/yahoo/searchdefinition/RankProfileRegistry.java index 52b9fba391b..4825da4087e 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/RankProfileRegistry.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/RankProfileRegistry.java @@ -14,7 +14,7 @@ import java.util.Map; import java.util.Set; /** - * Mapping from name to {@link RankProfile} as well as a reverse mapping of {@link RankProfile} to {@link Search}. + * Mapping from name to {@link RankProfile} as well as a reverse mapping of {@link RankProfile} to {@link Schema}. * Having both of these mappings consolidated here make it easier to remove dependencies on these mappings at * run time, since it is essentially only used when building rank profile config at deployment time. * @@ -30,14 +30,18 @@ public class RankProfileRegistry { /* These rank profiles can be overridden: 'default' rank profile, as that is documented to work. And 'unranked'. */ static final Set<String> overridableRankProfileNames = new HashSet<>(Arrays.asList("default", "unranked")); - public static RankProfileRegistry createRankProfileRegistryWithBuiltinRankProfiles(Search search) { + public RankProfileRegistry() { + + } + + public static RankProfileRegistry createRankProfileRegistryWithBuiltinRankProfiles(Schema schema) { RankProfileRegistry rankProfileRegistry = new RankProfileRegistry(); - rankProfileRegistry.add(new DefaultRankProfile(search, rankProfileRegistry, search.rankingConstants())); - rankProfileRegistry.add(new UnrankedRankProfile(search, rankProfileRegistry, search.rankingConstants())); + rankProfileRegistry.add(new DefaultRankProfile(schema, rankProfileRegistry, schema.rankingConstants())); + rankProfileRegistry.add(new UnrankedRankProfile(schema, rankProfileRegistry, schema.rankingConstants())); return rankProfileRegistry; } - private String extractName(ImmutableSearch search) { + private String extractName(ImmutableSchema search) { return search != null ? search.getName() : MAGIC_GLOBAL_RANKPROFILES; } @@ -69,17 +73,21 @@ public class RankProfileRegistry { /** * Returns a named rank profile, null if the search definition doesn't have one with the given name * - * @param search the {@link Search} that owns the rank profile. + * @param schema the {@link Schema} that owns the rank profile * @param name the name of the rank profile * @return the RankProfile to return. */ - public RankProfile get(String search, String name) { - Map<String, RankProfile> profiles = rankProfiles.get(search); + public RankProfile get(String schema, String name) { + Map<String, RankProfile> profiles = rankProfiles.get(schema); if (profiles == null) return null; return profiles.get(name); } - public RankProfile get(ImmutableSearch search, String name) { - return get(search.getName(), name); + + public RankProfile get(ImmutableSchema schema, String name) { + var profile = get(schema.getName(), name); + if (profile != null) return profile; + if (schema.inherited().isPresent()) return get(schema.inherited().get(), name); + return null; } public RankProfile getGlobal(String name) { @@ -129,7 +137,7 @@ public class RankProfileRegistry { * @param search search definition to fetch rank profiles for, or null for the global ones * @return Collection of RankProfiles */ - public Collection<RankProfile> rankProfilesOf(ImmutableSearch search) { + public Collection<RankProfile> rankProfilesOf(ImmutableSchema search) { return rankProfilesOf(extractName(search)); } diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/RankingConstants.java b/config-model/src/main/java/com/yahoo/searchdefinition/RankingConstants.java index 8df68b210ce..1e0eacaea16 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/RankingConstants.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/RankingConstants.java @@ -6,7 +6,9 @@ import com.yahoo.vespa.model.AbstractService; import java.util.Collection; import java.util.Collections; +import java.util.HashMap; import java.util.Map; +import java.util.Optional; import java.util.concurrent.ConcurrentHashMap; import java.util.function.Function; @@ -18,11 +20,16 @@ import java.util.function.Function; */ public class RankingConstants { - private final Map<String, RankingConstant> constants = new ConcurrentHashMap<>(); private final FileRegistry fileRegistry; - public RankingConstants(FileRegistry fileRegistry) { + /** The schema this belongs to, or empty if it is global */ + private final Optional<Schema> owner; + + private final Map<String, RankingConstant> constants = new ConcurrentHashMap<>(); + + public RankingConstants(FileRegistry fileRegistry, Optional<Schema> owner) { this.fileRegistry = fileRegistry; + this.owner = owner; } public void add(RankingConstant constant) { @@ -33,12 +40,14 @@ public class RankingConstants { if ( prev != null ) throw new IllegalArgumentException("Ranking constant '" + name + "' defined twice"); } + public void putIfAbsent(RankingConstant constant) { constant.validate(); constant.register(fileRegistry); String name = constant.getName(); constants.putIfAbsent(name, constant); } + public void computeIfAbsent(String name, Function<? super String, ? extends RankingConstant> createConstant) { constants.computeIfAbsent(name, key -> { RankingConstant constant = createConstant.apply(key); @@ -50,16 +59,26 @@ public class RankingConstants { /** Returns the ranking constant with the given name, or null if not present */ public RankingConstant get(String name) { - return constants.get(name); + var constant = constants.get(name); + if (constant != null) return constant; + if (owner.isPresent() && owner.get().inherited().isPresent()) + return owner.get().inherited().get().rankingConstants().get(name); + return null; } /** Returns a read-only map of the ranking constants in this indexed by name */ public Map<String, RankingConstant> asMap() { - return Collections.unmodifiableMap(constants); + // Shortcuts + if (owner.isEmpty() || owner.get().inherited().isEmpty()) return Collections.unmodifiableMap(constants); + if (constants.isEmpty()) return owner.get().inherited().get().rankingConstants().asMap(); + + var allConstants = new HashMap<>(owner.get().inherited().get().rankingConstants().asMap()); + allConstants.putAll(constants); + return Collections.unmodifiableMap(allConstants); } /** Initiate sending of these constants to some services over file distribution */ public void sendTo(Collection<? extends AbstractService> services) { - constants.values().forEach(constant -> constant.sendTo(services)); + asMap().values().forEach(constant -> constant.sendTo(services)); } } diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/Search.java b/config-model/src/main/java/com/yahoo/searchdefinition/Schema.java index 415fcd1d34e..b2d4f0592fe 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/Search.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/Schema.java @@ -7,6 +7,7 @@ import com.yahoo.config.application.api.FileRegistry; import com.yahoo.config.model.api.ModelContext; import com.yahoo.config.model.application.provider.BaseDeployLogger; import com.yahoo.config.model.deploy.TestProperties; +import com.yahoo.document.DataTypeName; import com.yahoo.document.Field; import com.yahoo.searchdefinition.derived.SummaryClass; import com.yahoo.searchdefinition.document.Attribute; @@ -26,105 +27,125 @@ import java.io.Reader; import java.util.ArrayList; import java.util.Collection; import java.util.Collections; +import java.util.HashSet; import java.util.Iterator; import java.util.LinkedHashMap; import java.util.List; import java.util.Map; +import java.util.Objects; import java.util.Optional; import java.util.TreeMap; import java.util.logging.Level; import java.util.stream.Stream; /** - * A search definition describes (or uses) some document types, defines how these are turned into a relevancy tuned - * index through indexing and how data from documents should be served at search time. The identity of this - * class is its name. + * A schema contains a document type, additional fields, rank profiles and document summaries. * * @author bratseth */ // TODO: Make a class owned by this, for each of these responsibilities: // Managing indexes, managing attributes, managing summary classes. // Ensure that after the processing step, all implicit instances of the above types are explicitly represented -public class Search implements ImmutableSearch { +public class Schema implements ImmutableSchema { private static final String SD_DOC_FIELD_NAME = "sddocname"; private static final List<String> RESERVED_NAMES = List.of( "index", "index_url", "summary", "attribute", "select_input", "host", SummaryClass.DOCUMENT_ID_FIELD, "position", "split_foreach", "tokenize", "if", "else", "switch", "case", SD_DOC_FIELD_NAME, "relevancy"); - /** Returns true if the given field name is a reserved name */ - public static boolean isReservedName(String name) { - return RESERVED_NAMES.contains(name); - } - - private final FieldSets fieldSets = new FieldSets(); - - /** The unique name of this search definition */ + /** The unique name of this schema */ private String name; + /** The name of the schema this should inherit all the content of, if any */ + private final Optional<String> inherited; + /** True if this doesn't define a search, just a document type */ private final boolean documentsOnly; - private boolean rawAsBase64 = false; + private Boolean rawAsBase64 = null; + + /** The stemming setting of this schema. Default is BEST. */ + private Stemming stemming = null; - /** The stemming setting of this search definition. Default is BEST. */ - private Stemming stemming = Stemming.BEST; + private final FieldSets fieldSets = new FieldSets(Optional.of(this)); - /** Documents contained in this definition */ - private SDDocumentType docType; + /** The document contained in this schema */ + private SDDocumentType documentType; - /** The extra fields of this search definition */ + /** The extra fields of this schema */ private final Map<String, SDField> fields = new LinkedHashMap<>(); - /** The explicitly defined indices of this search definition */ private final Map<String, Index> indices = new LinkedHashMap<>(); - /** The explicitly defined summaries of this search definition. _Must_ preserve order. */ + /** The explicitly defined summaries of this schema. _Must_ preserve order. */ private final Map<String, DocumentSummary> summaries = new LinkedHashMap<>(); /** External rank expression files of this */ private final LargeRankExpressions largeRankExpressions; - /** Ranking constants of this */ private final RankingConstants rankingConstants; - /** Onnx models of this */ private final OnnxModels onnxModels; - private Optional<TemporaryImportedFields> temporaryImportedFields = Optional.of(new TemporaryImportedFields()); + /** All imported fields of this (and parent schemas) */ + // TODO: Use empty, not optional + // TODO: Merge this and importedFields + private final Optional<TemporaryImportedFields> temporaryImportedFields = Optional.of(new TemporaryImportedFields(this)); + /** The resulting processed field */ private Optional<ImportedFields> importedFields = Optional.empty(); - private final ApplicationPackage applicationPackage; + private final Application owner; private final DeployLogger deployLogger; private final ModelContext.Properties properties; /** Testing only */ - public Search(String name) { - this(name, null, null, new BaseDeployLogger(), new TestProperties()); + public Schema(String name) { + this(name, Optional.empty(), null, null, new BaseDeployLogger(), new TestProperties()); } + + public Schema(String name, + Application application, + FileRegistry fileRegistry, + DeployLogger deployLogger, + ModelContext.Properties properties) { + this(name, Optional.empty(), application, fileRegistry, deployLogger, properties); + } + /** - * Creates a proper search definition + * Creates a schema * - * @param name of the the searchdefinition - * @param applicationPackage the application containing this + * @param name of the schema + * @param inherited the schema this inherits, if any + * @param application the application containing this */ - public Search(String name, ApplicationPackage applicationPackage, FileRegistry fileRegistry, DeployLogger deployLogger, ModelContext.Properties properties) { - this(applicationPackage, fileRegistry, deployLogger, properties, false); + public Schema(String name, + Optional<String> inherited, + Application application, + FileRegistry fileRegistry, + DeployLogger deployLogger, + ModelContext.Properties properties) { + this(inherited, application, fileRegistry, deployLogger, properties, false); this.name = name; } - protected Search(ApplicationPackage applicationPackage, FileRegistry fileRegistry, DeployLogger deployLogger, ModelContext.Properties properties) { - this(applicationPackage, fileRegistry, deployLogger, properties, true); + protected Schema(Application application, FileRegistry fileRegistry, DeployLogger deployLogger, ModelContext.Properties properties) { + this(Optional.empty(), application, fileRegistry, deployLogger, properties, true); } - private Search(ApplicationPackage applicationPackage, FileRegistry fileRegistry, DeployLogger deployLogger, ModelContext.Properties properties, boolean documentsOnly) { - this.applicationPackage = applicationPackage; + private Schema(Optional<String> inherited, + Application application, + FileRegistry fileRegistry, + DeployLogger deployLogger, + ModelContext.Properties properties, + boolean documentsOnly) { + this.inherited = inherited; + this.owner = application; this.deployLogger = deployLogger; this.properties = properties; this.documentsOnly = documentsOnly; largeRankExpressions = new LargeRankExpressions(fileRegistry); - rankingConstants = new RankingConstants(fileRegistry); - onnxModels = new OnnxModels(fileRegistry); + rankingConstants = new RankingConstants(fileRegistry, Optional.of(this)); + onnxModels = new OnnxModels(fileRegistry, Optional.of(this)); } protected void setName(String name) { @@ -136,22 +157,27 @@ public class Search implements ImmutableSearch { return name; } - /** - * Returns true if this doesn't define a search, just some documents - * - * @return if the searchdefinition only has documents - */ + /** Returns true if this only defines a document type, not a full schema */ public boolean isDocumentsOnly() { return documentsOnly; } + @Override + public Optional<Schema> inherited() { + return inherited.map(name -> owner.schemas().get(name)); + } + /** * Returns true if 'raw' fields shall be presented as base64 in summary - * Note that tis is temporary and will disappear on Vespa 8 as it will become default, and only option. + * Note that this is temporary and will disappear on Vespa 8 as it will become default, and only option. * * @return true if raw shall be encoded as base64 in summary */ - public boolean isRawAsBase64() { return rawAsBase64; } + public boolean isRawAsBase64() { + if (rawAsBase64 != null) return rawAsBase64; + if (inherited.isEmpty()) return false; + return requireInherited().isRawAsBase64(); + } public void enableRawAsBase64() { rawAsBase64 = true; } @@ -162,20 +188,14 @@ public class Search implements ImmutableSearch { * @throws NullPointerException if this is attempted set to null */ public void setStemming(Stemming stemming) { - if (stemming == null) { - throw new NullPointerException("The stemming setting of a search definition " + - "can not be null"); - } - this.stemming = stemming; + this.stemming = Objects.requireNonNull(stemming, "Stemming cannot be null"); } - /** - * Returns whether fields should be stemmed by default or not. Default is ALL. This is never null. - * - * @return the default stemming for this searchdefinition - */ + /** Returns whether fields should be stemmed by default or not. Default is BEST. This is never null. */ public Stemming getStemming() { - return stemming; + if (stemming != null) return stemming; + if (inherited.isEmpty()) return Stemming.BEST; + return requireInherited().getStemming(); } /** @@ -184,10 +204,10 @@ public class Search implements ImmutableSearch { * @param document the document type to add */ public void addDocument(SDDocumentType document) { - if (docType != null) { + if (documentType != null) { throw new IllegalArgumentException("Searchdefinition cannot have more than one document"); } - docType = document; + documentType = document; } @Override @@ -214,7 +234,6 @@ public class Search implements ImmutableSearch { } public void setImportedFields(ImportedFields importedFields) { - temporaryImportedFields = Optional.empty(); this.importedFields = Optional.of(importedFields); } @@ -240,7 +259,7 @@ public class Search implements ImmutableSearch { public List<ImmutableSDField> allFieldsList() { List<ImmutableSDField> all = new ArrayList<>(); all.addAll(extraFieldList()); - for (Field field : docType.fieldSet()) { + for (Field field : documentType.fieldSet()) { all.add((ImmutableSDField) field); } if (importedFields.isPresent()) { @@ -258,8 +277,8 @@ public class Search implements ImmutableSearch { * @return the contained or used document type, or null if there is no such document */ public SDDocumentType getDocument(String name) { - if (docType != null && name.equals(docType.getName())) { - return docType; + if (documentType != null && name.equals(documentType.getName())) { + return documentType; } return null; } @@ -268,7 +287,7 @@ public class Search implements ImmutableSearch { * @return true if the document has been added. */ public boolean hasDocument() { - return docType != null; + return documentType != null; } /** @@ -276,7 +295,7 @@ public class Search implements ImmutableSearch { */ @Override public SDDocumentType getDocument() { - return docType; + return documentType; } /** @@ -288,7 +307,7 @@ public class Search implements ImmutableSearch { public List<SDField> allConcreteFields() { List<SDField> allFields = new ArrayList<>(); allFields.addAll(extraFieldList()); - for (Field field : docType.fieldSet()) { + for (Field field : documentType.fieldSet()) { allFields.add((SDField)field); } return allFields; @@ -299,11 +318,16 @@ public class Search implements ImmutableSearch { */ @Override public Reader getRankingExpression(String fileName) { - return applicationPackage.getRankingExpression(fileName); + return owner.applicationPackage().getRankingExpression(fileName); } + public Application application() { return owner; } + @Override - public ApplicationPackage applicationPackage() { return applicationPackage; } + public ApplicationPackage applicationPackage() { + if (owner == null) return null; + return owner.applicationPackage(); + } @Override public DeployLogger getDeployLogger() { return deployLogger; } @@ -321,10 +345,9 @@ public class Search implements ImmutableSearch { @Override public SDField getConcreteField(String name) { SDField field = getExtraField(name); - if (field != null) { - return field; - } - return (SDField)docType.getField(name); + if (field != null) return field; + + return (SDField) documentType.getField(name); } /** @@ -335,7 +358,7 @@ public class Search implements ImmutableSearch { * @return The named field, or null if not found. */ public SDField getDocumentField(String name) { - return (SDField)docType.getField(name); + return (SDField) documentType.getField(name); } /** @@ -353,11 +376,17 @@ public class Search implements ImmutableSearch { } public Collection<SDField> extraFieldList() { - return fields.values(); + if (inherited.isEmpty()) return fields.values(); + var fields = new HashSet<>(requireInherited().extraFieldList()); + fields.addAll(this.fields.values()); + return fields; } + public Collection<SDField> allExtraFields() { Map<String, SDField> extraFields = new TreeMap<>(); - for (Field field : docType.fieldSet()) { + if (inherited.isPresent()) + requireInherited().allExtraFields().forEach(field -> extraFields.put(field.getName(), field)); + for (Field field : documentType.fieldSet()) { SDField sdField = (SDField) field; if (sdField.isExtraField()) { extraFields.put(sdField.getName(), sdField); @@ -376,7 +405,10 @@ public class Search implements ImmutableSearch { * @return the SDField of this name */ public SDField getExtraField(String fieldName) { - return fields.get(fieldName); + SDField field = fields.get(fieldName); + if (field != null) return field; + if (inherited.isEmpty()) return null; + return requireInherited().getExtraField(fieldName); } /** @@ -389,10 +421,9 @@ public class Search implements ImmutableSearch { } /** - * <p>Returns an index, or null if no index with this name has had some <b>explicit settings</b> applied. Even if - * this returns null, the index may be implicitly defined by an indexing statement.</p> - * <p>This will return the - * index whether it is defined on this search or on one of its fields</p> + * Returns an index, or null if no index with this name has had some <b>explicit settings</b> applied. Even if + * this returns null, the index may be implicitly defined by an indexing statement. This will return the + * index whether it is defined on this schema or on one of its fields. * * @param name the name of the index to get * @return the index requested @@ -400,34 +431,33 @@ public class Search implements ImmutableSearch { @Override public Index getIndex(String name) { List<Index> sameIndices = new ArrayList<>(1); - Index searchIndex = indices.get(name); - if (searchIndex != null) { - sameIndices.add(searchIndex); - } + + getSchemaIndex(name).ifPresent(sameIndices::add); for (ImmutableSDField field : allConcreteFields()) { - Index index = field.getIndex(name); - if (index != null) { - sameIndices.add(index); - } - } - if (sameIndices.size() == 0) { - return null; - } - if (sameIndices.size() == 1) { - return sameIndices.get(0); + if (field.getIndex(name) != null) + sameIndices.add(field.getIndex(name)); } + if (sameIndices.size() == 0) return null; + if (sameIndices.size() == 1) return sameIndices.get(0); return consolidateIndices(sameIndices); } + /** Returns the schema level index of this name, in this or any inherited schema, if any */ + Optional<Index> getSchemaIndex(String name) { + if (indices.containsKey(name)) return Optional.of(indices.get(name)); + if (inherited.isPresent()) return requireInherited().getSchemaIndex(name); + return Optional.empty(); + } + public boolean existsIndex(String name) { - if (indices.get(name) != null) { + if (indices.get(name) != null) + return true; + if (inherited.isPresent() && requireInherited().existsIndex(name)) return true; - } for (ImmutableSDField field : allConcreteFields()) { - if (field.existsIndex(name)) { + if (field.existsIndex(name)) return true; - } } return false; } @@ -435,7 +465,7 @@ public class Search implements ImmutableSearch { /** * Consolidates a set of index settings for the same index into one * - * @param indices The list of indexes to consolidate. + * @param indices The list of indexes to consolidate * @return the consolidated index */ private Index consolidateIndices(List<Index> indices) { @@ -458,8 +488,8 @@ public class Search implements ImmutableSearch { !consolidated.getRankType().equals(current.getRankType())) { deployLogger.logApplicationPackage(Level.WARNING, "Conflicting rank type settings for " + - first.getName() + " in " + this + ", using " + - consolidated.getRankType()); + first.getName() + " in " + this + ", using " + + consolidated.getRankType()); } } @@ -470,19 +500,24 @@ public class Search implements ImmutableSearch { return consolidated; } - /** - * All explicitly defined indices, both on this search definition itself (returned first) and all its fields - * - * @return The list of explicit defined indexes. - */ + /** All explicitly defined indices, both on this schema itself (returned first) and all its fields */ @Override public List<Index> getExplicitIndices() { List<Index> allIndices = new ArrayList<>(indices.values()); + + if (inherited.isPresent()) { + for (Index inheritedIndex : requireInherited().getExplicitIndices()) { + if ( ! indices.containsKey(inheritedIndex.getName())) // child redefinitions shadows parents + allIndices.add(inheritedIndex); + } + } + for (ImmutableSDField field : allConcreteFields()) { for (Index index : field.getIndices().values()) { allIndices.add(index); } } + return Collections.unmodifiableList(allIndices); } @@ -503,7 +538,10 @@ public class Search implements ImmutableSearch { * @return Summary found. */ public DocumentSummary getSummary(String name) { - return summaries.get(name); + var summary = summaries.get(name); + if (summary != null) return summary; + if (inherited.isEmpty()) return null; + return requireInherited().getSummary(name); } /** @@ -520,47 +558,56 @@ public class Search implements ImmutableSearch { return summaryField; } } - return null; + if (inherited.isEmpty()) return null; + return requireInherited().getSummaryField(name); } /** * Returns the first explicit instance found of a summary field with this name, or null if not present explicitly in * any summary class * - * @param name Thge name of the explicit summary field to get. - * @return The SummaryField found. + * @param name the name of the explicit summary field to get. + * @return the SummaryField found. */ public SummaryField getExplicitSummaryField(String name) { for (DocumentSummary summary : summaries.values()) { SummaryField summaryField = summary.getSummaryField(name); - if (summaryField != null && !summaryField.isImplicit()) { + if (summaryField != null && !summaryField.isImplicit()) return summaryField; - } } - return null; + if (inherited.isEmpty()) return null; + return requireInherited().getExplicitSummaryField(name); } /** * Summaries defined by fields of this search definition. The default summary, named "default", is always the first * one in the returned iterator. - * - * @return The map of document summaries. */ public Map<String, DocumentSummary> getSummaries() { - return summaries; + // Shortcuts + if (inherited.isEmpty()) return summaries; + if (summaries.isEmpty()) return requireInherited().getSummaries(); + + var allSummaries = new LinkedHashMap<>(requireInherited().getSummaries()); + allSummaries.putAll(summaries); + return allSummaries; } + /** Returns the summaries defines in this only, not any that are inherited. */ + public Map<String, DocumentSummary> getSummariesInThis() { return Collections.unmodifiableMap(summaries); } + /** - * <p>Returns all summary fields, of all document summaries, which has the given field as source. If there are + * Returns all summary fields, of all document summaries, which has the given field as source. If there are * multiple summary fields with the same name, the last one will be used (they should all have the same content, if - * this is a valid search definition).</p> <p>The map gets owned by the receiver.</p> + * this is a valid search definition).The map becomes owned by the receiver. * - * @param field The source field. - * @return The map of summary fields found. + * @param field the source field + * @return the map of summary fields found */ @Override public Map<String, SummaryField> getSummaryFields(ImmutableSDField field) { - Map<String, SummaryField> summaryFields = new java.util.LinkedHashMap<>(); + Map<String, SummaryField> summaryFields = inherited.isPresent() ? requireInherited().getSummaryFields(field) + : new java.util.LinkedHashMap<>(); for (DocumentSummary documentSummary : summaries.values()) { for (SummaryField summaryField : documentSummary.getSummaryFields()) { if (summaryField.hasSource(field.getName())) { @@ -572,15 +619,14 @@ public class Search implements ImmutableSearch { } /** - * <p>Returns one summary field for each summary field name. If there are multiple summary fields with the same + * Returns one summary field for each summary field name. If there are multiple summary fields with the same * name, the last one will be used. Multiple fields of the same name should all have the same content in a valid * search definition, except from the destination set. So this method can be used for all summary handling except - * processing the destination set.</p> <p>The map gets owned by the receiver.</p> - * - * @return Map of unique summary fields + * processing the destination set. The map becomes owned by the receiver. */ public Map<String, SummaryField> getUniqueNamedSummaryFields() { - Map<String, SummaryField> summaryFields = new java.util.LinkedHashMap<>(); + Map<String, SummaryField> summaryFields = inherited.isPresent() ? requireInherited().getUniqueNamedSummaryFields() + : new java.util.LinkedHashMap<>(); for (DocumentSummary documentSummary : summaries.values()) { for (SummaryField summaryField : documentSummary.getSummaryFields()) { summaryFields.put(summaryField.getName(), summaryField); @@ -612,11 +658,11 @@ public class Search implements ImmutableSearch { @Override public boolean equals(Object o) { - if (!(o instanceof Search)) { + if (!(o instanceof Schema)) { return false; } - Search other = (Search)o; + Schema other = (Schema)o; return getName().equals(other.getName()); } @@ -651,8 +697,10 @@ public class Search implements ImmutableSearch { return false; } - /** The field set settings for this search */ - public FieldSets fieldSets() { return fieldSets; } + public FieldSets fieldSets() { return fieldSets; } + + /** Returns the schema inherited by this, or throws if none */ + private Schema requireInherited() { return owner.schemas().get(inherited.get()); } /** * For adding structs defined in document scope @@ -660,19 +708,36 @@ public class Search implements ImmutableSearch { * @param dt the struct to add * @return self, for chaining */ - public Search addType(SDDocumentType dt) { - docType.addType(dt); // TODO This is a very very dirty thing. It must go + public Schema addType(SDDocumentType dt) { + documentType.addType(dt); // TODO This is a very very dirty thing. It must go return this; } - public Search addAnnotation(SDAnnotationType dt) { - docType.addAnnotation(dt); + public Schema addAnnotation(SDAnnotationType dt) { + documentType.addAnnotation(dt); return this; } public void validate(DeployLogger logger) { + if (inherited.isPresent()) { + if (! owner.schemas().containsKey(inherited.get())) + throw new IllegalArgumentException(this + " inherits '" + inherited.get() + + "', but this schema does not exist"); + + // Require schema and document type inheritance to be consistent to keep things simple + // And require it to be explicit so we have the option to support other possibilities later + var parentDocument = owner.schemas().get(inherited.get()).getDocument(); + if ( ! getDocument().inheritedTypes().containsKey(new DataTypeName(parentDocument.getName()))) + throw new IllegalArgumentException(this + " inherits '" + inherited.get() + + "', but its document type does not inherit the parent's document type"); + } for (var summary : summaries.values()) summary.validate(logger); } + /** Returns true if the given field name is a reserved name */ + public static boolean isReservedName(String name) { + return RESERVED_NAMES.contains(name); + } + } diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/SearchBuilder.java b/config-model/src/main/java/com/yahoo/searchdefinition/SearchBuilder.java index 0f7ead43868..dc5ab752f31 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/SearchBuilder.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/SearchBuilder.java @@ -21,6 +21,7 @@ import com.yahoo.searchdefinition.parser.SDParser; import com.yahoo.searchdefinition.parser.SimpleCharStream; import com.yahoo.searchdefinition.parser.TokenMgrException; import com.yahoo.searchdefinition.processing.Processing; +import com.yahoo.searchdefinition.processing.Processor; import com.yahoo.vespa.documentmodel.DocumentModel; import com.yahoo.vespa.model.container.search.QueryProfiles; import com.yahoo.yolean.Exceptions; @@ -32,22 +33,24 @@ import java.nio.file.Path; import java.util.ArrayList; import java.util.Collection; import java.util.Collections; +import java.util.HashSet; import java.util.Iterator; import java.util.LinkedList; import java.util.List; +import java.util.Set; /** - * Helper class for importing {@link Search} objects in an unambiguous way. The pattern for using this is to 1) Import + * Helper class for importing {@link Schema} objects in an unambiguous way. The pattern for using this is to 1) Import * all available search definitions, using the importXXX() methods, 2) provide the available rank types and rank * expressions, using the setRankXXX() methods, 3) invoke the {@link #build()} method, and 4) retrieve the built * search objects using the {@link #getSearch(String)} method. */ +// Since this was created we have added Application, and much of the content in this should probably migrate there. public class SearchBuilder { private final DocumentTypeManager docTypeMgr = new DocumentTypeManager(); private final DocumentModel model = new DocumentModel(); - private final ApplicationPackage app; - private final RankProfileRegistry rankProfileRegistry; + private final Application application; private final QueryProfileRegistry queryProfileRegistry; private final FileRegistry fileRegistry; private final DeployLogger deployLogger; @@ -55,9 +58,11 @@ public class SearchBuilder { /** True to build the document aspect only, skipping instantiation of rank profiles */ private final boolean documentsOnly; - private List<Search> searchList = new LinkedList<>(); + private List<Schema> schemaList = new LinkedList<>(); private boolean isBuilt = false; + private final Set<Class<? extends Processor>> processorsToSkip = new HashSet<>(); + /** For testing only */ public SearchBuilder() { this(new RankProfileRegistry(), new QueryProfileRegistry()); @@ -109,15 +114,14 @@ public class SearchBuilder { QueryProfileRegistry queryProfileRegistry) { this(app, fileRegistry, deployLogger, properties, rankProfileRegistry, queryProfileRegistry, false); } - private SearchBuilder(ApplicationPackage app, + private SearchBuilder(ApplicationPackage applicationPackage, FileRegistry fileRegistry, DeployLogger deployLogger, ModelContext.Properties properties, RankProfileRegistry rankProfileRegistry, QueryProfileRegistry queryProfileRegistry, boolean documentsOnly) { - this.app = app; - this.rankProfileRegistry = rankProfileRegistry; + this.application = new Application(applicationPackage, rankProfileRegistry); this.queryProfileRegistry = queryProfileRegistry; this.fileRegistry = fileRegistry; this.deployLogger = deployLogger; @@ -169,8 +173,9 @@ public class SearchBuilder { private String importString(String str, String searchDefDir) throws ParseException { SimpleCharStream stream = new SimpleCharStream(str); try { - return importRawSearch(new SDParser(stream, fileRegistry, deployLogger, properties, app, rankProfileRegistry, documentsOnly) - .search(docTypeMgr, searchDefDir)); + return importRawSearch(new SDParser(stream, fileRegistry, deployLogger, properties, application, + application.rankProfileRegistry(), documentsOnly) + .schema(docTypeMgr, searchDefDir)); } catch (TokenMgrException e) { throw new ParseException("Unknown symbol: " + e.getMessage()); } catch (ParseException pe) { @@ -180,24 +185,19 @@ public class SearchBuilder { /** * Registers the given search object to the internal list of objects to be processed during {@link #build()}. A - * {@link Search} object is considered to be "raw" if it has not already been processed. This is the case for most + * {@link Schema} object is considered to be "raw" if it has not already been processed. This is the case for most * programmatically constructed search objects used in unit tests. * - * @param rawSearch the object to import. + * @param schema the object to import. * @return the name of the imported object. * @throws IllegalArgumentException if the given search object has already been processed. */ - public String importRawSearch(Search rawSearch) { - if (rawSearch.getName() == null) - throw new IllegalArgumentException("Search has no name."); - String rawName = rawSearch.getName(); - for (Search search : searchList) { - if (rawName.equals(search.getName())) { - throw new IllegalArgumentException("A search definition with a search section called '" + rawName + - "' has already been added."); - } - } - searchList.add(rawSearch); + public String importRawSearch(Schema schema) { + if (schema.getName() == null) + throw new IllegalArgumentException("Schema has no name"); + String rawName = schema.getName(); + application.add(schema); + schemaList.add(schema); return rawName; } @@ -223,14 +223,14 @@ public class SearchBuilder { if (isBuilt) throw new IllegalStateException("Model already built"); if (validate) - searchList.forEach(search -> search.validate(deployLogger)); + application.validate(deployLogger); - List<Search> built = new ArrayList<>(); + List<Schema> built = new ArrayList<>(); List<SDDocumentType> sdocs = new ArrayList<>(); sdocs.add(SDDocumentType.VESPA_DOCUMENT); - for (Search search : searchList) { - if (search.hasDocument()) { - sdocs.add(search.getDocument()); + for (Schema schema : schemaList) { + if (schema.hasDocument()) { + sdocs.add(schema.getDocument()); } } @@ -241,51 +241,56 @@ public class SearchBuilder { new FieldOperationApplier().process(sdoc); } - var resolver = new DocumentReferenceResolver(searchList); + var resolver = new DocumentReferenceResolver(schemaList); sdocs.forEach(resolver::resolveReferences); sdocs.forEach(resolver::resolveInheritedReferences); - var importedFieldsEnumerator = new ImportedFieldsEnumerator(searchList); + var importedFieldsEnumerator = new ImportedFieldsEnumerator(schemaList); sdocs.forEach(importedFieldsEnumerator::enumerateImportedFields); if (validate) new DocumentGraphValidator().validateDocumentGraph(sdocs); var builder = new DocumentModelBuilder(model); - for (Search search : new SearchOrderer().order(searchList)) { - new FieldOperationApplierForSearch().process(search); // TODO: Why is this not in the regular list? - process(search, new QueryProfiles(queryProfileRegistry, deployLogger), validate); - built.add(search); + for (Schema schema : new SearchOrderer().order(schemaList)) { + new FieldOperationApplierForSearch().process(schema); // TODO: Why is this not in the regular list? + process(schema, new QueryProfiles(queryProfileRegistry, deployLogger), validate); + built.add(schema); } - builder.addToModel(searchList); + builder.addToModel(schemaList); if ( validate && ! builder.valid() ) throw new IllegalArgumentException("Impossible to build a correct model"); - searchList = built; + schemaList = built; isBuilt = true; } + /** Returns a modifiable set of processors we should skip for these schemas. Useful for testing. */ + public Set<Class<? extends Processor>> processorsToSkip() { return processorsToSkip; } + /** - * Processes and returns the given {@link Search} object. This method has been factored out of the {@link + * Processes and returns the given {@link Schema} object. This method has been factored out of the {@link * #build()} method so that subclasses can choose not to build anything. */ - private void process(Search search, QueryProfiles queryProfiles, boolean validate) { - new Processing().process(search, deployLogger, rankProfileRegistry, queryProfiles, validate, documentsOnly); + private void process(Schema schema, QueryProfiles queryProfiles, boolean validate) { + new Processing().process(schema, deployLogger, application.rankProfileRegistry(), queryProfiles, validate, + documentsOnly, processorsToSkip); } /** - * Convenience method to call {@link #getSearch(String)} when there is only a single {@link Search} object + * Convenience method to call {@link #getSearch(String)} when there is only a single {@link Schema} object * built. This method will never return null. * * @return the built object * @throws IllegalStateException if there is not exactly one search. */ - public Search getSearch() { + public Schema getSearch() { if ( ! isBuilt) throw new IllegalStateException("Searches not built."); - if (searchList.size() != 1) - throw new IllegalStateException("This call only works if we have 1 search definition. Search definitions: " + searchList); + if (application.schemas().size() != 1) + throw new IllegalStateException("This call only works if we have 1 schema. Schemas: " + + application.schemas().values()); - return searchList.get(0); + return application.schemas().values().stream().findAny().get(); } public DocumentModel getModel() { @@ -293,7 +298,7 @@ public class SearchBuilder { } /** - * Returns the built {@link Search} object that has the given name. If the name is unknown, this method will simply + * Returns the built {@link Schema} object that has the given name. If the name is unknown, this method will simply * return null. * * @param name the name of the search definition to return, @@ -301,26 +306,25 @@ public class SearchBuilder { * @return the built object, or null if none with this name * @throws IllegalStateException if {@link #build()} has not been called. */ - public Search getSearch(String name) { + public Schema getSearch(String name) { if ( ! isBuilt) throw new IllegalStateException("Searches not built."); if (name == null) return getSearch(); - - for (Search search : searchList) - if (search.getName().equals(name)) return search; - return null; + return application.schemas().get(name); } + public Application application() { return application; } + /** - * Convenience method to return a list of all built {@link Search} objects. + * Convenience method to return a list of all built {@link Schema} objects. * - * @return The list of built searches. + * @return the list of built searches */ - public List<Search> getSearchList() { - return new ArrayList<>(searchList); + public List<Schema> getSearchList() { + return new ArrayList<>(schemaList); } /** - * Convenience factory method to import and build a {@link Search} object from a string. + * Convenience factory method to import and build a {@link Schema} object from a string. * * @param sd The string to build from. * @return The built {@link SearchBuilder} object. @@ -346,7 +350,7 @@ public class SearchBuilder { } /** - * Convenience factory method to import and build a {@link Search} object from a file. Only for testing. + * Convenience factory method to import and build a {@link Schema} object from a file. Only for testing. * * @param fileName the file to build from * @return the built {@link SearchBuilder} object @@ -373,7 +377,7 @@ public class SearchBuilder { } /** - * Convenience factory method to import and build a {@link Search} object from a file. + * Convenience factory method to import and build a {@link Schema} object from a file. * * @param fileName the file to build from. * @param deployLogger logger for deploy messages. @@ -414,6 +418,7 @@ public class SearchBuilder { return builder; } + public static SearchBuilder createFromDirectory(String dir, FileRegistry fileRegistry, DeployLogger logger, ModelContext.Properties properties) throws IOException, ParseException { return createFromDirectory(dir, fileRegistry, logger, properties, new RankProfileRegistry()); } @@ -430,7 +435,8 @@ public class SearchBuilder { ModelContext.Properties properties, RankProfileRegistry rankProfileRegistry, QueryProfileRegistry queryProfileRegistry) throws IOException, ParseException { - return createFromDirectory(dir, MockApplicationPackage.fromSearchDefinitionDirectory(dir), fileRegistry, logger, properties, rankProfileRegistry, queryProfileRegistry); + return createFromDirectory(dir, MockApplicationPackage.fromSearchDefinitionAndRootDirectory(dir), fileRegistry, logger, properties, + rankProfileRegistry, queryProfileRegistry); } private static SearchBuilder createFromDirectory(String dir, @@ -462,27 +468,27 @@ public class SearchBuilder { // TODO: The build methods below just call the create methods above - remove /** - * Convenience factory method to import and build a {@link Search} object from a file. Only for testing. + * Convenience factory method to import and build a {@link Schema} object from a file. Only for testing. * * @param fileName The file to build from. - * @return The built {@link Search} object. + * @return The built {@link Schema} object. * @throws IOException Thrown if there was a problem reading the file. * @throws ParseException Thrown if there was a problem parsing the file content. */ - public static Search buildFromFile(String fileName) throws IOException, ParseException { + public static Schema buildFromFile(String fileName) throws IOException, ParseException { return buildFromFile(fileName, new BaseDeployLogger(), new RankProfileRegistry(), new QueryProfileRegistry()); } /** - * Convenience factory method to import and build a {@link Search} object from a file. + * Convenience factory method to import and build a {@link Schema} object from a file. * * @param fileName The file to build from. * @param rankProfileRegistry Registry for rank profiles. - * @return The built {@link Search} object. + * @return The built {@link Schema} object. * @throws IOException Thrown if there was a problem reading the file. * @throws ParseException Thrown if there was a problem parsing the file content. */ - public static Search buildFromFile(String fileName, + public static Schema buildFromFile(String fileName, RankProfileRegistry rankProfileRegistry, QueryProfileRegistry queryProfileRegistry) throws IOException, ParseException { @@ -490,16 +496,16 @@ public class SearchBuilder { } /** - * Convenience factory method to import and build a {@link Search} object from a file. + * Convenience factory method to import and build a {@link Schema} object from a file. * * @param fileName The file to build from. * @param deployLogger Logger for deploy messages. * @param rankProfileRegistry Registry for rank profiles. - * @return The built {@link Search} object. + * @return The built {@link Schema} object. * @throws IOException Thrown if there was a problem reading the file. * @throws ParseException Thrown if there was a problem parsing the file content. */ - public static Search buildFromFile(String fileName, + public static Schema buildFromFile(String fileName, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfileRegistry queryProfileRegistry) @@ -508,36 +514,36 @@ public class SearchBuilder { } /** - * Convenience factory method to import and build a {@link Search} object from a raw object. + * Convenience factory method to import and build a {@link Schema} object from a raw object. * - * @param rawSearch the raw object to build from. + * @param rawSchema the raw object to build from. * @return the built {@link SearchBuilder} object. - * @see #importRawSearch(Search) + * @see #importRawSearch(Schema) */ - public static SearchBuilder createFromRawSearch(Search rawSearch, + public static SearchBuilder createFromRawSearch(Schema rawSchema, RankProfileRegistry rankProfileRegistry, QueryProfileRegistry queryProfileRegistry) { SearchBuilder builder = new SearchBuilder(rankProfileRegistry, queryProfileRegistry); - builder.importRawSearch(rawSearch); + builder.importRawSearch(rawSchema); builder.build(); return builder; } /** - * Convenience factory method to import and build a {@link Search} object from a raw object. + * Convenience factory method to import and build a {@link Schema} object from a raw object. * - * @param rawSearch The raw object to build from. - * @return The built {@link Search} object. - * @see #importRawSearch(Search) + * @param rawSchema The raw object to build from. + * @return The built {@link Schema} object. + * @see #importRawSearch(Schema) */ - public static Search buildFromRawSearch(Search rawSearch, + public static Schema buildFromRawSearch(Schema rawSchema, RankProfileRegistry rankProfileRegistry, QueryProfileRegistry queryProfileRegistry) { - return createFromRawSearch(rawSearch, rankProfileRegistry, queryProfileRegistry).getSearch(); + return createFromRawSearch(rawSchema, rankProfileRegistry, queryProfileRegistry).getSearch(); } public RankProfileRegistry getRankProfileRegistry() { - return rankProfileRegistry; + return application.rankProfileRegistry(); } public QueryProfileRegistry getQueryProfileRegistry() { diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/UnrankedRankProfile.java b/config-model/src/main/java/com/yahoo/searchdefinition/UnrankedRankProfile.java index 6bd30538abd..acf034362ca 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/UnrankedRankProfile.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/UnrankedRankProfile.java @@ -11,8 +11,8 @@ import com.yahoo.searchlib.rankingexpression.parser.ParseException; */ public class UnrankedRankProfile extends RankProfile { - public UnrankedRankProfile(Search search, RankProfileRegistry rankProfileRegistry, RankingConstants rankingConstants) { - super("unranked", search, rankProfileRegistry, rankingConstants); + public UnrankedRankProfile(Schema schema, RankProfileRegistry rankProfileRegistry, RankingConstants rankingConstants) { + super("unranked", schema, rankProfileRegistry, rankingConstants); try { RankingExpression exp = new RankingExpression("value(0)"); this.setFirstPhaseRanking(exp); diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/derived/AttributeFields.java b/config-model/src/main/java/com/yahoo/searchdefinition/derived/AttributeFields.java index 24cc15dde21..67ed37800f9 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/derived/AttributeFields.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/derived/AttributeFields.java @@ -4,7 +4,7 @@ package com.yahoo.searchdefinition.derived; import com.yahoo.config.subscription.ConfigInstanceUtil; import com.yahoo.document.DataType; import com.yahoo.document.PositionDataType; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.document.Attribute; import com.yahoo.searchdefinition.document.Case; import com.yahoo.searchdefinition.document.Dictionary; @@ -44,14 +44,14 @@ public class AttributeFields extends Derived implements AttributesConfig.Produce public static final AttributeFields empty = new AttributeFields(null); - public AttributeFields(Search search) { - if (search != null) - derive(search); + public AttributeFields(Schema schema) { + if (schema != null) + derive(schema); } /** Derives everything from a field */ @Override - protected void derive(ImmutableSDField field, Search search) { + protected void derive(ImmutableSDField field, Schema schema) { if (unsupportedFieldType(field)) { return; // Ignore complex struct and map fields for indexed search (only supported for streaming search) } diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/derived/Derived.java b/config-model/src/main/java/com/yahoo/searchdefinition/derived/Derived.java index fded9502c49..30cb236984d 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/derived/Derived.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/derived/Derived.java @@ -6,7 +6,7 @@ import com.yahoo.config.ConfigInstance.Builder; import com.yahoo.document.Field; import com.yahoo.io.IOUtils; import com.yahoo.searchdefinition.Index; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.document.ImmutableSDField; import com.yahoo.searchdefinition.document.SDDocumentType; import com.yahoo.searchdefinition.document.SDField; @@ -36,14 +36,14 @@ public abstract class Derived implements Exportable { * and derive(SDField) for each search definition level field * AND sets the name of this to the name of the input search definition */ - protected void derive(Search search) { - setName(search.getName()); - derive(search.getDocument(), search); - for (Index index : search.getExplicitIndices()) - derive(index, search); - for (SDField field : search.allExtraFields()) - derive(field, search); - search.allImportedFields().forEach(importedField -> derive(importedField, search)); + protected void derive(Schema schema) { + setName(schema.getName()); + derive(schema.getDocument(), schema); + for (Index index : schema.getExplicitIndices()) + derive(index, schema); + for (SDField field : schema.allExtraFields()) + derive(field, schema); + schema.allImportedFields().forEach(importedField -> derive(importedField, schema)); } @@ -51,11 +51,11 @@ public abstract class Derived implements Exportable { * Derives the content of this configuration. This * default calls derive(SDField) for each document field */ - protected void derive(SDDocumentType document, Search search) { + protected void derive(SDDocumentType document, Schema schema) { for (Field field : document.fieldSet()) { SDField sdField = (SDField) field; if ( ! sdField.isExtraField()) { - derive(sdField, search); + derive(sdField, schema); } } } @@ -64,13 +64,13 @@ public abstract class Derived implements Exportable { * Derives the content of this configuration. This * default does nothing. */ - protected void derive(ImmutableSDField field, Search search) {} + protected void derive(ImmutableSDField field, Schema schema) {} /** * Derives the content of this configuration. This * default does nothing. */ - protected void derive(Index index, Search search) { + protected void derive(Index index, Schema schema) { } protected abstract String getDerivedName(); diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/derived/DerivedConfiguration.java b/config-model/src/main/java/com/yahoo/searchdefinition/derived/DerivedConfiguration.java index 65bda6eb576..54e9a4ccb61 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/derived/DerivedConfiguration.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/derived/DerivedConfiguration.java @@ -14,7 +14,7 @@ import com.yahoo.io.IOUtils; import com.yahoo.protect.Validator; import com.yahoo.search.query.profile.QueryProfileRegistry; import com.yahoo.searchdefinition.RankProfileRegistry; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.derived.validation.Validation; import com.yahoo.vespa.model.container.search.QueryProfiles; @@ -30,7 +30,7 @@ import java.util.concurrent.ExecutorService; */ public class DerivedConfiguration { - private final Search search; + private final Schema schema; private Summaries summaries; private SummaryMap summaryMap; private Juniperrc juniperrc; @@ -48,23 +48,23 @@ public class DerivedConfiguration { * Creates a complete derived configuration from a search definition. * Only used in tests. * - * @param search the search to derive a configuration from. Derived objects will be snapshots, but this argument is + * @param schema the search to derive a configuration from. Derived objects will be snapshots, but this argument is * live. Which means that this object will be inconsistent when the given search definition is later * modified. * @param rankProfileRegistry a {@link com.yahoo.searchdefinition.RankProfileRegistry} */ - public DerivedConfiguration(Search search, RankProfileRegistry rankProfileRegistry) { - this(search, rankProfileRegistry, new QueryProfileRegistry()); + public DerivedConfiguration(Schema schema, RankProfileRegistry rankProfileRegistry) { + this(schema, rankProfileRegistry, new QueryProfileRegistry()); } - DerivedConfiguration(Search search, RankProfileRegistry rankProfileRegistry, QueryProfileRegistry queryProfiles) { - this(search, new BaseDeployLogger(), new TestProperties(), rankProfileRegistry, queryProfiles, new ImportedMlModels(), new InThreadExecutorService()); + DerivedConfiguration(Schema schema, RankProfileRegistry rankProfileRegistry, QueryProfileRegistry queryProfiles) { + this(schema, new BaseDeployLogger(), new TestProperties(), rankProfileRegistry, queryProfiles, new ImportedMlModels(), new InThreadExecutorService()); } /** * Creates a complete derived configuration snapshot from a search definition. * - * @param search the search to derive a configuration from. Derived objects will be snapshots, but this + * @param schema the search to derive a configuration from. Derived objects will be snapshots, but this * argument is live. Which means that this object will be inconsistent when the given * search definition is later modified. * @param deployLogger a {@link DeployLogger} for logging when doing operations on this @@ -72,34 +72,34 @@ public class DerivedConfiguration { * @param rankProfileRegistry a {@link com.yahoo.searchdefinition.RankProfileRegistry} * @param queryProfiles the query profiles of this application */ - public DerivedConfiguration(Search search, + public DerivedConfiguration(Schema schema, DeployLogger deployLogger, ModelContext.Properties deployProperties, RankProfileRegistry rankProfileRegistry, QueryProfileRegistry queryProfiles, ImportedMlModels importedModels, ExecutorService executor) { - Validator.ensureNotNull("Search definition", search); - this.search = search; + Validator.ensureNotNull("Search definition", schema); + this.schema = schema; this.queryProfiles = queryProfiles; - if ( ! search.isDocumentsOnly()) { - streamingFields = new VsmFields(search); - streamingSummary = new VsmSummary(search); + if ( ! schema.isDocumentsOnly()) { + streamingFields = new VsmFields(schema); + streamingSummary = new VsmSummary(schema); } - if ( ! search.isDocumentsOnly()) { - attributeFields = new AttributeFields(search); - summaries = new Summaries(search, deployLogger); - summaryMap = new SummaryMap(search); - juniperrc = new Juniperrc(search); - rankProfileList = new RankProfileList(search, search.rankingConstants(), search.rankExpressionFiles(), - search.onnxModels(), attributeFields, rankProfileRegistry, + if ( ! schema.isDocumentsOnly()) { + attributeFields = new AttributeFields(schema); + summaries = new Summaries(schema, deployLogger); + summaryMap = new SummaryMap(schema); + juniperrc = new Juniperrc(schema); + rankProfileList = new RankProfileList(schema, schema.rankingConstants(), schema.rankExpressionFiles(), + schema.onnxModels(), attributeFields, rankProfileRegistry, queryProfiles, importedModels, deployProperties, executor); - indexingScript = new IndexingScript(search); - indexInfo = new IndexInfo(search); - indexSchema = new IndexSchema(search); - importedFields = new ImportedFields(search); + indexingScript = new IndexingScript(schema); + indexInfo = new IndexInfo(schema); + indexSchema = new IndexSchema(schema); + importedFields = new ImportedFields(schema); } - Validation.validate(this, search); + Validation.validate(this, schema); } /** @@ -109,7 +109,7 @@ public class DerivedConfiguration { * @throws IOException if exporting fails, some files may still be created */ public void export(String toDirectory) throws IOException { - if (!search.isDocumentsOnly()) { + if (!schema.isDocumentsOnly()) { summaries.export(toDirectory); summaryMap.export(toDirectory); juniperrc.export(toDirectory); @@ -171,8 +171,8 @@ public class DerivedConfiguration { this.indexingScript = script; } - public Search getSearch() { - return search; + public Schema getSearch() { + return schema; } public RankProfileList getRankProfileList() { diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/derived/ImportedFields.java b/config-model/src/main/java/com/yahoo/searchdefinition/derived/ImportedFields.java index 281ab6536bd..a63b88f9445 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/derived/ImportedFields.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/derived/ImportedFields.java @@ -3,7 +3,7 @@ package com.yahoo.searchdefinition.derived; import com.yahoo.document.DataType; import com.yahoo.document.PositionDataType; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.document.Attribute; import com.yahoo.searchdefinition.document.ImmutableSDField; import com.yahoo.searchdefinition.document.ImportedComplexField; @@ -25,13 +25,13 @@ public class ImportedFields extends Derived implements ImportedFieldsConfig.Prod private Optional<com.yahoo.searchdefinition.document.ImportedFields> importedFields = Optional.empty(); - public ImportedFields(Search search) { - derive(search); + public ImportedFields(Schema schema) { + derive(schema); } @Override - protected void derive(Search search) { - importedFields = search.importedFields(); + protected void derive(Schema schema) { + importedFields = schema.importedFields(); } @Override diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/derived/IndexInfo.java b/config-model/src/main/java/com/yahoo/searchdefinition/derived/IndexInfo.java index 7950cd758fd..879ad570c26 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/derived/IndexInfo.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/derived/IndexInfo.java @@ -10,7 +10,7 @@ import com.yahoo.document.PositionDataType; import com.yahoo.document.PrimitiveDataType; import com.yahoo.document.StructuredDataType; import com.yahoo.searchdefinition.Index; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.document.Attribute; import com.yahoo.searchdefinition.document.BooleanIndexDefinition; import com.yahoo.searchdefinition.document.Case; @@ -55,33 +55,33 @@ public class IndexInfo extends Derived implements IndexInfoConfig.Producer { private final Set<IndexCommand> commands = new java.util.LinkedHashSet<>(); private final Map<String, String> aliases = new java.util.LinkedHashMap<>(); private final Map<String, FieldSet> fieldSets; - private Search search; + private Schema schema; - public IndexInfo(Search search) { - this.fieldSets = search.fieldSets().userFieldSets(); + public IndexInfo(Schema schema) { + this.fieldSets = schema.fieldSets().userFieldSets(); addIndexCommand("sddocname", CMD_INDEX); addIndexCommand("sddocname", CMD_WORD); - derive(search); + derive(schema); } @Override - protected void derive(Search search) { - super.derive(search); // Derive per field - this.search = search; + protected void derive(Schema schema) { + super.derive(schema); // Derive per field + this.schema = schema; // Populate fieldsets with actual field objects, bit late to do that here but for (FieldSet fs : fieldSets.values()) { for (String fieldName : fs.getFieldNames()) { - fs.fields().add(search.getField(fieldName)); + fs.fields().add(schema.getField(fieldName)); } } // Must follow, because index settings overrides field settings - for (Index index : search.getExplicitIndices()) { - derive(index, search); + for (Index index : schema.getExplicitIndices()) { + derive(index, schema); } // Commands for summary fields // TODO: Move to fieldinfo and implement differently. This is not right - for (SummaryField summaryField : search.getUniqueNamedSummaryFields().values()) { + for (SummaryField summaryField : schema.getUniqueNamedSummaryFields().values()) { if (summaryField.getTransform().isTeaser()) { addIndexCommand(summaryField.getName(), CMD_DYNTEASER); } @@ -100,11 +100,11 @@ public class IndexInfo extends Derived implements IndexInfoConfig.Producer { } @Override - protected void derive(ImmutableSDField field, Search search) { - derive(field, search, false); + protected void derive(ImmutableSDField field, Schema schema) { + derive(field, schema, false); } - protected void derive(ImmutableSDField field, Search search, boolean inPosition) { + protected void derive(ImmutableSDField field, Schema schema, boolean inPosition) { if (field.getDataType().equals(DataType.PREDICATE)) { addIndexCommand(field, CMD_PREDICATE); Index index = field.getIndex(field.getName()); @@ -127,7 +127,7 @@ public class IndexInfo extends Derived implements IndexInfoConfig.Producer { boolean isPosition = isPositionField(field); if (field.usesStructOrMap()) { for (ImmutableSDField structField : field.getStructFields()) { - derive(structField, search, isPosition); // Recursion + derive(structField, schema, isPosition); // Recursion } } @@ -151,8 +151,8 @@ public class IndexInfo extends Derived implements IndexInfoConfig.Producer { if (attribute != null && attribute.isFastSearch()) addIndexCommand(field.getName(), CMD_FAST_SEARCH); } else if (field.doesIndexing()) { - if (stemSomehow(field, search)) { - addIndexCommand(field, stemCmd(field, search), new StemmingOverrider(this, search)); + if (stemSomehow(field, schema)) { + addIndexCommand(field, stemCmd(field, schema), new StemmingOverrider(this, schema)); } if (normalizeAccents(field)) { addIndexCommand(field, CMD_NORMALIZE); @@ -201,12 +201,12 @@ public class IndexInfo extends Derived implements IndexInfoConfig.Producer { && field.getMatching().getCase().equals(Case.UNCASED)); } - static String stemCmd(ImmutableSDField field, Search search) { - return CMD_STEM + ":" + field.getStemming(search).toStemMode(); + static String stemCmd(ImmutableSDField field, Schema schema) { + return CMD_STEM + ":" + field.getStemming(schema).toStemMode(); } - private boolean stemSomehow(ImmutableSDField field, Search search) { - if (field.getStemming(search).equals(Stemming.NONE)) return false; + private boolean stemSomehow(ImmutableSDField field, Schema schema) { + if (field.getStemming(schema).equals(Stemming.NONE)) return false; return isTypeOrNested(field, DataType.STRING); } @@ -456,7 +456,7 @@ public class IndexInfo extends Derived implements IndexInfoConfig.Producer { } private Stemming getEffectiveStemming(ImmutableSDField field) { - Stemming active = field.getStemming(search); + Stemming active = field.getStemming(schema); if (field.getIndex(field.getName()) != null) { if (field.getIndex(field.getName()).getStemming()!=null) { active = field.getIndex(field.getName()).getStemming(); @@ -472,7 +472,7 @@ public class IndexInfo extends Derived implements IndexInfoConfig.Producer { if (field.getStemming() != null) { return !field.getStemming().equals(Stemming.NONE); } - if (search.getStemming()==Stemming.NONE) return false; + if (schema.getStemming() == Stemming.NONE) return false; if (field.isImportedField()) return false; if (field.getIndex(field.getName())==null) return true; if (field.getIndex(field.getName()).getStemming()==null) return true; @@ -563,19 +563,19 @@ public class IndexInfo extends Derived implements IndexInfoConfig.Producer { private static class StemmingOverrider extends IndexOverrider { - private Search search; + private Schema schema; - public StemmingOverrider(IndexInfo owner, Search search) { + public StemmingOverrider(IndexInfo owner, Schema schema) { super(owner); - this.search = search; + this.schema = schema; } public boolean override(String indexName, String command, ImmutableSDField field) { - if (search == null) { + if (schema == null) { return false; } - Index index = search.getIndex(indexName); + Index index = schema.getIndex(indexName); if (index == null) { return false; } diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/derived/IndexSchema.java b/config-model/src/main/java/com/yahoo/searchdefinition/derived/IndexSchema.java index 8b5e9d4ca8e..ce5731bcdeb 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/derived/IndexSchema.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/derived/IndexSchema.java @@ -7,7 +7,7 @@ import com.yahoo.document.Field; import com.yahoo.document.StructuredDataType; import com.yahoo.document.TensorDataType; import com.yahoo.document.WeightedSetDataType; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.document.BooleanIndexDefinition; import com.yahoo.searchdefinition.document.FieldSet; import com.yahoo.searchdefinition.document.ImmutableSDField; @@ -31,9 +31,9 @@ public class IndexSchema extends Derived implements IndexschemaConfig.Producer { private final Map<String, FieldCollection> collections = new LinkedHashMap<>(); private final Map<String, FieldSet> fieldSets = new LinkedHashMap<>(); - public IndexSchema(Search search) { - fieldSets.putAll(search.fieldSets().userFieldSets()); - derive(search); + public IndexSchema(Schema schema) { + fieldSets.putAll(schema.fieldSets().userFieldSets()); + derive(schema); } public boolean containsField(String fieldName) { @@ -41,15 +41,15 @@ public class IndexSchema extends Derived implements IndexschemaConfig.Producer { } @Override - protected void derive(Search search) { - super.derive(search); + protected void derive(Schema schema) { + super.derive(schema); } private boolean isTensorField(ImmutableSDField field) { return field.getDataType() instanceof TensorDataType; } - private void deriveIndexFields(ImmutableSDField field, Search search) { + private void deriveIndexFields(ImmutableSDField field, Schema schema) { // Note: Indexes for tensor fields are NOT part of the index schema for text fields. if ((!field.doesIndexing() && !field.isIndexStructureField()) || isTensorField(field)) @@ -62,7 +62,7 @@ public class IndexSchema extends Derived implements IndexschemaConfig.Producer { } String fieldName = field.getName(); for (Field flatField : lst) { - deriveIndexFields(flatField, search); + deriveIndexFields(flatField, schema); } if (lst.size() > 1) { FieldSet fieldSet = new FieldSet(fieldName); @@ -73,9 +73,9 @@ public class IndexSchema extends Derived implements IndexschemaConfig.Producer { } } - private void deriveIndexFields(Field field, Search search) { + private void deriveIndexFields(Field field, Schema schema) { IndexField toAdd = new IndexField(field.getName(), Index.convertType(field.getDataType()), field.getDataType()); - com.yahoo.searchdefinition.Index definedIndex = search.getIndex(field.getName()); + com.yahoo.searchdefinition.Index definedIndex = schema.getIndex(field.getName()); if (definedIndex != null) { toAdd.setIndexSettings(definedIndex); } @@ -98,11 +98,11 @@ public class IndexSchema extends Derived implements IndexschemaConfig.Producer { } @Override - protected void derive(ImmutableSDField field, Search search) { + protected void derive(ImmutableSDField field, Schema schema) { if (field.usesStructOrMap()) { return; // unsupported } - deriveIndexFields(field, search); + deriveIndexFields(field, schema); } @Override diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/derived/IndexingScript.java b/config-model/src/main/java/com/yahoo/searchdefinition/derived/IndexingScript.java index 4bf4b21eb75..cabe8d001bd 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/derived/IndexingScript.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/derived/IndexingScript.java @@ -3,7 +3,7 @@ package com.yahoo.searchdefinition.derived; import com.yahoo.document.DataType; import com.yahoo.document.PositionDataType; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.document.ImmutableSDField; import com.yahoo.vespa.configdefinition.IlscriptsConfig; import com.yahoo.vespa.configdefinition.IlscriptsConfig.Ilscript.Builder; @@ -39,20 +39,20 @@ public final class IndexingScript extends Derived implements IlscriptsConfig.Pro private final List<Expression> expressions = new ArrayList<>(); private List<ImmutableSDField> fieldsSettingLanguage; - public IndexingScript(Search search) { - derive(search); + public IndexingScript(Schema schema) { + derive(schema); } @Override - protected void derive(Search search) { - fieldsSettingLanguage = fieldsSettingLanguage(search); + protected void derive(Schema schema) { + fieldsSettingLanguage = fieldsSettingLanguage(schema); if (fieldsSettingLanguage.size() == 1) // Assume this language should be used for all fields addExpression(fieldsSettingLanguage.get(0).getIndexingScript()); - super.derive(search); + super.derive(schema); } @Override - protected void derive(ImmutableSDField field, Search search) { + protected void derive(ImmutableSDField field, Schema schema) { if (field.isImportedField()) return; if (field.hasFullIndexingDocprocRights()) @@ -75,11 +75,11 @@ public final class IndexingScript extends Derived implements IlscriptsConfig.Pro expressions.add(new StatementExpression(new ClearStateExpression(), new GuardExpression(expression))); } - private List<ImmutableSDField> fieldsSettingLanguage(Search search) { - return search.allFieldsList().stream() - .filter(field -> ! field.isImportedField()) - .filter(field -> field.containsExpression(SetLanguageExpression.class)) - .collect(Collectors.toList()); + private List<ImmutableSDField> fieldsSettingLanguage(Schema schema) { + return schema.allFieldsList().stream() + .filter(field -> ! field.isImportedField()) + .filter(field -> field.containsExpression(SetLanguageExpression.class)) + .collect(Collectors.toList()); } public Iterable<Expression> expressions() { diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/derived/Juniperrc.java b/config-model/src/main/java/com/yahoo/searchdefinition/derived/Juniperrc.java index 8dad49f40d1..327d7df4d07 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/derived/Juniperrc.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/derived/Juniperrc.java @@ -1,7 +1,7 @@ // Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. package com.yahoo.searchdefinition.derived; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.vespa.documentmodel.SummaryField; import com.yahoo.vespa.documentmodel.SummaryTransform; import com.yahoo.vespa.config.search.summary.JuniperrcConfig; @@ -20,19 +20,19 @@ public class Juniperrc extends Derived implements JuniperrcConfig.Producer { /** * Constructs a new juniper rc instance for a given search object. This will derive the configuration automatically, - * so there is no need to call {@link #derive(com.yahoo.searchdefinition.Search)}. + * so there is no need to call {@link #derive(Schema)}. * - * @param search The search model to use for deriving. + * @param schema The search model to use for deriving. */ - public Juniperrc(Search search) { - derive(search); + public Juniperrc(Schema schema) { + derive(schema); } // Inherit doc from Derived. @Override - protected void derive(Search search) { - super.derive(search); - for (SummaryField summaryField : search.getUniqueNamedSummaryFields().values()) { + protected void derive(Schema schema) { + super.derive(schema); + for (SummaryField summaryField : schema.getUniqueNamedSummaryFields().values()) { if (summaryField.getTransform() == SummaryTransform.BOLDED) { boldingFields.add(summaryField.getName()); } diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/derived/RankProfileList.java b/config-model/src/main/java/com/yahoo/searchdefinition/derived/RankProfileList.java index dc84e1fdd8a..faee6d67dbf 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/derived/RankProfileList.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/derived/RankProfileList.java @@ -12,7 +12,7 @@ import com.yahoo.searchdefinition.RankingConstant; import com.yahoo.searchdefinition.RankingConstants; import com.yahoo.vespa.config.search.RankProfilesConfig; import com.yahoo.searchdefinition.RankProfile; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.vespa.config.search.core.OnnxModelsConfig; import com.yahoo.vespa.config.search.core.RankingConstantsConfig; import com.yahoo.vespa.config.search.core.RankingExpressionsConfig; @@ -23,13 +23,14 @@ import java.util.Collection; import java.util.LinkedHashMap; import java.util.List; import java.util.Map; +import java.util.Optional; import java.util.concurrent.ExecutionException; import java.util.concurrent.ExecutorService; import java.util.concurrent.Future; import java.util.logging.Logger; /** - * The derived rank profiles of a search definition + * The derived rank profiles of a schema * * @author bratseth */ @@ -45,18 +46,18 @@ public class RankProfileList extends Derived implements RankProfilesConfig.Produ public static RankProfileList empty = new RankProfileList(); private RankProfileList() { - rankingConstants = new RankingConstants(null); + rankingConstants = new RankingConstants(null, Optional.empty()); largeRankExpressions = new LargeRankExpressions(null); - onnxModels = new OnnxModels(null); + onnxModels = new OnnxModels(null, Optional.empty()); } /** * Creates a rank profile * - * @param search the search definition this is a rank profile from + * @param schema the schema this is a rank profile from * @param attributeFields the attribute fields to create a ranking for */ - public RankProfileList(Search search, + public RankProfileList(Schema schema, RankingConstants rankingConstants, LargeRankExpressions largeRankExpressions, OnnxModels onnxModels, @@ -66,11 +67,11 @@ public class RankProfileList extends Derived implements RankProfilesConfig.Produ ImportedMlModels importedModels, ModelContext.Properties deployProperties, ExecutorService executor) { - setName(search == null ? "default" : search.getName()); + setName(schema == null ? "default" : schema.getName()); this.rankingConstants = rankingConstants; this.largeRankExpressions = largeRankExpressions; this.onnxModels = onnxModels; // as ONNX models come from parsing rank expressions - deriveRankProfiles(rankProfileRegistry, queryProfiles, importedModels, search, attributeFields, deployProperties, executor); + deriveRankProfiles(rankProfileRegistry, queryProfiles, importedModels, schema, attributeFields, deployProperties, executor); } private boolean areDependenciesReady(RankProfile rank, RankProfileRegistry registry) { @@ -82,38 +83,38 @@ public class RankProfileList extends Derived implements RankProfilesConfig.Produ private void deriveRankProfiles(RankProfileRegistry rankProfileRegistry, QueryProfileRegistry queryProfiles, ImportedMlModels importedModels, - Search search, + Schema schema, AttributeFields attributeFields, ModelContext.Properties deployProperties, ExecutorService executor) { - if (search != null) { // profiles belonging to a search have a default profile - RawRankProfile rawRank = new RawRankProfile(rankProfileRegistry.get(search, "default"), + if (schema != null) { // profiles belonging to a search have a default profile + RawRankProfile rawRank = new RawRankProfile(rankProfileRegistry.get(schema, "default"), largeRankExpressions, queryProfiles, importedModels, attributeFields, deployProperties); rankProfiles.put(rawRank.getName(), rawRank); } Map<String, RankProfile> remaining = new LinkedHashMap<>(); - rankProfileRegistry.rankProfilesOf(search).forEach(rank -> remaining.put(rank.getName(), rank)); + rankProfileRegistry.rankProfilesOf(schema).forEach(rank -> remaining.put(rank.getName(), rank)); remaining.remove("default"); while (!remaining.isEmpty()) { List<RankProfile> ready = new ArrayList<>(); remaining.forEach((name, rank) -> { if (areDependenciesReady(rank, rankProfileRegistry)) ready.add(rank); }); - processRankProfiles(ready, queryProfiles, importedModels, search, attributeFields, deployProperties, executor); + processRankProfiles(ready, queryProfiles, importedModels, schema, attributeFields, deployProperties, executor); ready.forEach(rank -> remaining.remove(rank.getName())); } } private void processRankProfiles(List<RankProfile> ready, QueryProfileRegistry queryProfiles, ImportedMlModels importedModels, - Search search, + Schema schema, AttributeFields attributeFields, ModelContext.Properties deployProperties, ExecutorService executor) { Map<String, Future<RawRankProfile>> futureRawRankProfiles = new LinkedHashMap<>(); for (RankProfile rank : ready) { - if (search == null) { + if (schema == null) { onnxModels.add(rank.onnxModels()); } diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/derived/SearchOrderer.java b/config-model/src/main/java/com/yahoo/searchdefinition/derived/SearchOrderer.java index b55064cc49d..8703eb11433 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/derived/SearchOrderer.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/derived/SearchOrderer.java @@ -4,8 +4,8 @@ package com.yahoo.searchdefinition.derived; import com.yahoo.document.DataTypeName; import com.yahoo.searchdefinition.DocumentReference; import com.yahoo.searchdefinition.DocumentReferences; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.document.SDDocumentType; -import com.yahoo.searchdefinition.Search; import java.util.*; @@ -23,7 +23,7 @@ import java.util.*; public class SearchOrderer { /** A map from DataTypeName to the Search defining them */ - private final Map<DataTypeName, Search> documentNameToSearch = new HashMap<>(); + private final Map<DataTypeName, Schema> documentNameToSearch = new HashMap<>(); /** * Reorders the given list of search definitions such that any supertype @@ -32,21 +32,21 @@ public class SearchOrderer { * * @return a new list containing the same search instances in the right order */ - public List<Search> order(List<Search> unordered) { + public List<Schema> order(List<Schema> unordered) { // Description above state that the original order should be preserved, except for the dependency constraint. // Yet we botch that guarantee by sorting the list... - unordered.sort(Comparator.comparing(Search::getName)); + unordered.sort(Comparator.comparing(Schema::getName)); // No, this is not a fast algorithm... indexOnDocumentName(unordered); - List<Search> ordered = new ArrayList<>(unordered.size()); - List<Search> moveOutwards = new ArrayList<>(); - for (Search search : unordered) { - if (allDependenciesAlreadyEmitted(ordered, search)) { - addOrdered(ordered, search, moveOutwards); + List<Schema> ordered = new ArrayList<>(unordered.size()); + List<Schema> moveOutwards = new ArrayList<>(); + for (Schema schema : unordered) { + if (allDependenciesAlreadyEmitted(ordered, schema)) { + addOrdered(ordered, schema, moveOutwards); } else { - moveOutwards.add(search); + moveOutwards.add(schema); } } @@ -58,9 +58,9 @@ public class SearchOrderer { return ordered; } - private void addOrdered(List<Search> ordered, Search search, List<Search> moveOutwards) { - ordered.add(search); - Search eligibleMove; + private void addOrdered(List<Schema> ordered, Schema schema, List<Schema> moveOutwards) { + ordered.add(schema); + Schema eligibleMove; do { eligibleMove = removeFirstEntryWithFullyEmittedDependencies(moveOutwards, ordered); if (eligibleMove != null) { @@ -70,8 +70,8 @@ public class SearchOrderer { } /** Removes and returns the first search from the move list which can now be added, or null if none */ - private Search removeFirstEntryWithFullyEmittedDependencies(List<Search> moveOutwards, List<Search> ordered) { - for (Search move : moveOutwards) { + private Schema removeFirstEntryWithFullyEmittedDependencies(List<Schema> moveOutwards, List<Schema> ordered) { + for (Schema move : moveOutwards) { if (allDependenciesAlreadyEmitted(ordered, move)) { moveOutwards.remove(move); return move; @@ -80,29 +80,29 @@ public class SearchOrderer { return null; } - private boolean allDependenciesAlreadyEmitted(List<Search> alreadyOrdered, Search search) { - if (search.getDocument() == null) { + private boolean allDependenciesAlreadyEmitted(List<Schema> alreadyOrdered, Schema schema) { + if (schema.getDocument() == null) { return true; } - SDDocumentType document = search.getDocument(); + SDDocumentType document = schema.getDocument(); return allInheritedDependenciesEmitted(alreadyOrdered, document) && allReferenceDependenciesEmitted(alreadyOrdered, document); } - private boolean allInheritedDependenciesEmitted(List<Search> alreadyOrdered, SDDocumentType document) { + private boolean allInheritedDependenciesEmitted(List<Schema> alreadyOrdered, SDDocumentType document) { for (SDDocumentType sdoc : document.getInheritedTypes() ) { DataTypeName inheritedName = sdoc.getDocumentName(); if ("document".equals(inheritedName.getName())) { continue; } - Search inheritedSearch = documentNameToSearch.get(inheritedName); - if (!alreadyOrdered.contains(inheritedSearch)) { + Schema inheritedSchema = documentNameToSearch.get(inheritedName); + if (!alreadyOrdered.contains(inheritedSchema)) { return false; } } return true; } - private static boolean allReferenceDependenciesEmitted(List<Search> alreadyOrdered, SDDocumentType document) { + private static boolean allReferenceDependenciesEmitted(List<Schema> alreadyOrdered, SDDocumentType document) { DocumentReferences documentReferences = document.getDocumentReferences() .orElseThrow(() -> new IllegalStateException("Missing document references. Should have been processed by now.")); return documentReferences.stream() @@ -111,11 +111,11 @@ public class SearchOrderer { .allMatch(alreadyOrdered::contains); } - private void indexOnDocumentName(List<Search> searches) { + private void indexOnDocumentName(List<Schema> schemas) { documentNameToSearch.clear(); - for (Search search : searches) { - if (search.getDocument() != null) { - documentNameToSearch.put(search.getDocument().getDocumentName(),search); + for (Schema schema : schemas) { + if (schema.getDocument() != null) { + documentNameToSearch.put(schema.getDocument().getDocumentName(), schema); } } } diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/derived/Summaries.java b/config-model/src/main/java/com/yahoo/searchdefinition/derived/Summaries.java index a91bbeed9b2..5fdc51e8f5f 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/derived/Summaries.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/derived/Summaries.java @@ -2,7 +2,7 @@ package com.yahoo.searchdefinition.derived; import com.yahoo.config.application.api.DeployLogger; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.vespa.documentmodel.DocumentSummary; import com.yahoo.vespa.config.search.SummaryConfig; import java.util.List; @@ -16,12 +16,12 @@ public class Summaries extends Derived implements SummaryConfig.Producer { private List<SummaryClass> summaries=new java.util.ArrayList<>(1); - public Summaries(Search search, DeployLogger deployLogger) { + public Summaries(Schema schema, DeployLogger deployLogger) { // Make sure the default is first - summaries.add(new SummaryClass(search,search.getSummary("default"), deployLogger)); - for (DocumentSummary summary : search.getSummaries().values()) { + summaries.add(new SummaryClass(schema, schema.getSummary("default"), deployLogger)); + for (DocumentSummary summary : schema.getSummaries().values()) { if (!summary.getName().equals("default")) - summaries.add(new SummaryClass(search,summary, deployLogger)); + summaries.add(new SummaryClass(schema, summary, deployLogger)); } } diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/derived/SummaryClass.java b/config-model/src/main/java/com/yahoo/searchdefinition/derived/SummaryClass.java index 2b40822b23b..5f88ddba5f5 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/derived/SummaryClass.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/derived/SummaryClass.java @@ -4,7 +4,7 @@ package com.yahoo.searchdefinition.derived; import com.yahoo.config.application.api.DeployLogger; import com.yahoo.document.DataType; import com.yahoo.prelude.fastsearch.DocsumDefinitionSet; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.vespa.config.search.SummaryConfig; import com.yahoo.vespa.documentmodel.DocumentSummary; import com.yahoo.vespa.documentmodel.SummaryField; @@ -42,12 +42,12 @@ public class SummaryClass extends Derived { * * @param deployLogger a {@link DeployLogger} */ - public SummaryClass(Search search, DocumentSummary summary, DeployLogger deployLogger) { + public SummaryClass(Schema schema, DocumentSummary summary, DeployLogger deployLogger) { this.deployLogger = deployLogger; - this.rawAsBase64 = search.isRawAsBase64(); + this.rawAsBase64 = schema.isRawAsBase64(); this.omitSummaryFeatures = summary.omitSummaryFeatures(); deriveName(summary); - deriveFields(search,summary); + deriveFields(schema, summary); deriveImplicitFields(summary); } @@ -62,9 +62,9 @@ public class SummaryClass extends Derived { } } - private void deriveFields(Search search, DocumentSummary summary) { + private void deriveFields(Schema schema, DocumentSummary summary) { for (SummaryField summaryField : summary.getSummaryFields()) { - if (!accessingDiskSummary && search.isAccessingDiskSummary(summaryField)) { + if (!accessingDiskSummary && schema.isAccessingDiskSummary(summaryField)) { accessingDiskSummary = true; } addField(summaryField.getName(), summaryField.getDataType(), summaryField.getTransform()); diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/derived/SummaryMap.java b/config-model/src/main/java/com/yahoo/searchdefinition/derived/SummaryMap.java index c53dba0bab0..cf182a1afbc 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/derived/SummaryMap.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/derived/SummaryMap.java @@ -1,7 +1,7 @@ // Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. package com.yahoo.searchdefinition.derived; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.document.ImmutableSDField; import com.yahoo.vespa.config.search.SummarymapConfig; import com.yahoo.vespa.documentmodel.DocumentSummary; @@ -23,19 +23,19 @@ public class SummaryMap extends Derived implements SummarymapConfig.Producer { private Map<String,FieldResultTransform> resultTransforms = new java.util.LinkedHashMap<>(); /** Creates a summary map from a search definition */ - SummaryMap(Search search) { - derive(search); + SummaryMap(Schema schema) { + derive(schema); } - protected void derive(Search search) { - for (DocumentSummary documentSummary : search.getSummaries().values()) { + protected void derive(Schema schema) { + for (DocumentSummary documentSummary : schema.getSummaries().values()) { derive(documentSummary); } - super.derive(search); + super.derive(schema); } @Override - protected void derive(ImmutableSDField field, Search search) { + protected void derive(ImmutableSDField field, Schema schema) { } private void derive(DocumentSummary documentSummary) { diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/derived/VsmFields.java b/config-model/src/main/java/com/yahoo/searchdefinition/derived/VsmFields.java index a301f3983a6..2fd90d6c87e 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/derived/VsmFields.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/derived/VsmFields.java @@ -12,7 +12,7 @@ import com.yahoo.document.datatypes.Raw; import com.yahoo.document.datatypes.StringFieldValue; import com.yahoo.document.datatypes.TensorFieldValue; import com.yahoo.searchdefinition.FieldSets; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.document.FieldSet; import com.yahoo.searchdefinition.document.Matching; import com.yahoo.searchdefinition.document.SDDocumentType; @@ -30,20 +30,20 @@ public class VsmFields extends Derived implements VsmfieldsConfig.Producer { private final Map<String, StreamingField> fields=new LinkedHashMap<>(); private final Map<String, StreamingDocumentType> doctypes=new LinkedHashMap<>(); - public VsmFields(Search search) { - addSearchdefinition(search); + public VsmFields(Schema schema) { + addSearchdefinition(schema); } - private void addSearchdefinition(Search search) { - derive(search); + private void addSearchdefinition(Schema schema) { + derive(schema); } @Override - protected void derive(SDDocumentType document,Search search) { - super.derive(document, search); + protected void derive(SDDocumentType document, Schema schema) { + super.derive(document, schema); StreamingDocumentType docType=getDocumentType(document.getName()); if (docType == null) { - docType = new StreamingDocumentType(document.getName(), search.fieldSets()); + docType = new StreamingDocumentType(document.getName(), schema.fieldSets()); doctypes.put(document.getName(), docType); } for (Object o : document.fieldSet()) { diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/derived/VsmSummary.java b/config-model/src/main/java/com/yahoo/searchdefinition/derived/VsmSummary.java index 44ba9df8226..a2c90bc92f5 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/derived/VsmSummary.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/derived/VsmSummary.java @@ -2,9 +2,9 @@ package com.yahoo.searchdefinition.derived; import com.yahoo.document.PositionDataType; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.document.SDDocumentType; import com.yahoo.searchdefinition.document.SDField; -import com.yahoo.searchdefinition.Search; import com.yahoo.vespa.documentmodel.DocumentSummary; import com.yahoo.vespa.documentmodel.SummaryField; import com.yahoo.vespa.config.search.vsm.VsmsummaryConfig; @@ -20,23 +20,23 @@ public class VsmSummary extends Derived implements VsmsummaryConfig.Producer { private Map<SummaryField, List<String>> summaryMap = new java.util.LinkedHashMap<>(1); - public VsmSummary(Search search) { - derive(search); + public VsmSummary(Schema schema) { + derive(schema); } @Override - protected void derive(Search search) { + protected void derive(Schema schema) { // Use the default class, as it is the superset - derive(search, search.getSummary("default")); + derive(schema, schema.getSummary("default")); } - private void derive(Search search, DocumentSummary documentSummary) { + private void derive(Schema schema, DocumentSummary documentSummary) { if (documentSummary==null) return; for (SummaryField summaryField : documentSummary.getSummaryFields()) { List<String> from = toStringList(summaryField.sourceIterator()); - if (doMapField(search, summaryField)) { - SDField sdField = search.getConcreteField(summaryField.getName()); + if (doMapField(schema, summaryField)) { + SDField sdField = schema.getConcreteField(summaryField.getName()); if (sdField != null && PositionDataType.INSTANCE.equals(sdField.getDataType())) { summaryMap.put(summaryField, Collections.singletonList(summaryField.getName())); } else { @@ -52,9 +52,9 @@ public class VsmSummary extends Derived implements VsmsummaryConfig.Producer { * Don't map if not struct either. * @param summaryField a {@link SummaryField} */ - private boolean doMapField(Search search, SummaryField summaryField) { - SDField sdField = search.getConcreteField(summaryField.getName()); - SDDocumentType document = search.getDocument(); + private boolean doMapField(Schema schema, SummaryField summaryField) { + SDField sdField = schema.getConcreteField(summaryField.getName()); + SDDocumentType document = schema.getDocument(); if (sdField==null || ((document != null) && (document.getField(summaryField.getName()) == sdField))) { return true; } @@ -66,7 +66,7 @@ public class VsmSummary extends Derived implements VsmsummaryConfig.Producer { } if (summaryField.getSourceCount()==sdField.getStructFields().size()) { for (SummaryField.Source source : summaryField.getSources()) { - if (!sdField.getStructFields().contains(new SDField(search.getDocument(), source.getName(), sdField.getDataType()))) { // equals() uses just name + if (!sdField.getStructFields().contains(new SDField(schema.getDocument(), source.getName(), sdField.getDataType()))) { // equals() uses just name return true; } if (sdField.getStructField(source.getName())!=null && !sdField.getStructField(source.getName()).doesSummarying()) { diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/derived/validation/IndexStructureValidator.java b/config-model/src/main/java/com/yahoo/searchdefinition/derived/validation/IndexStructureValidator.java index c41437f2531..76f52bce2a9 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/derived/validation/IndexStructureValidator.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/derived/validation/IndexStructureValidator.java @@ -2,7 +2,7 @@ package com.yahoo.searchdefinition.derived.validation; import com.yahoo.searchdefinition.document.SDDocumentType; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.derived.DerivedConfiguration; import com.yahoo.searchdefinition.derived.IndexingScript; import com.yahoo.vespa.indexinglanguage.ExpressionVisitor; @@ -14,14 +14,14 @@ import com.yahoo.vespa.indexinglanguage.expressions.OutputExpression; */ public class IndexStructureValidator extends Validator { - public IndexStructureValidator(DerivedConfiguration config, Search search) { - super(config, search); + public IndexStructureValidator(DerivedConfiguration config, Schema schema) { + super(config, schema); } public void validate() { IndexingScript script = config.getIndexingScript(); for (Expression exp : script.expressions()) { - new OutputVisitor(search.getDocument(), exp).visit(exp); + new OutputVisitor(schema.getDocument(), exp).visit(exp); } } diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/derived/validation/Validation.java b/config-model/src/main/java/com/yahoo/searchdefinition/derived/validation/Validation.java index a32fe67312f..fd75fbb2f48 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/derived/validation/Validation.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/derived/validation/Validation.java @@ -1,12 +1,12 @@ // Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. package com.yahoo.searchdefinition.derived.validation; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.derived.DerivedConfiguration; public class Validation { - public static void validate(DerivedConfiguration config, Search search) { - new IndexStructureValidator(config, search).validate(); + public static void validate(DerivedConfiguration config, Schema schema) { + new IndexStructureValidator(config, schema).validate(); } } diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/derived/validation/Validator.java b/config-model/src/main/java/com/yahoo/searchdefinition/derived/validation/Validator.java index 170639f5deb..7d3e5b8e9ed 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/derived/validation/Validator.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/derived/validation/Validator.java @@ -1,7 +1,7 @@ // Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. package com.yahoo.searchdefinition.derived.validation; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.derived.DerivedConfiguration; /** @@ -10,11 +10,11 @@ import com.yahoo.searchdefinition.derived.DerivedConfiguration; public abstract class Validator { protected DerivedConfiguration config; - protected Search search; + protected Schema schema; - protected Validator(DerivedConfiguration config, Search search) { + protected Validator(DerivedConfiguration config, Schema schema) { this.config = config; - this.search = search; + this.schema = schema; } public abstract void validate(); diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/document/ImmutableImportedSDField.java b/config-model/src/main/java/com/yahoo/searchdefinition/document/ImmutableImportedSDField.java index 925b67014e5..71814c31087 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/document/ImmutableImportedSDField.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/document/ImmutableImportedSDField.java @@ -4,7 +4,7 @@ package com.yahoo.searchdefinition.document; import com.yahoo.document.DataType; import com.yahoo.document.Field; import com.yahoo.searchdefinition.Index; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.vespa.documentmodel.SummaryField; import com.yahoo.vespa.indexinglanguage.expressions.Expression; import com.yahoo.vespa.indexinglanguage.expressions.ScriptExpression; @@ -151,7 +151,7 @@ public class ImmutableImportedSDField implements ImmutableSDField { } @Override - public Stemming getStemming(Search search) { + public Stemming getStemming(Schema schema) { throw createUnsupportedException("stemming"); } diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/document/ImmutableSDField.java b/config-model/src/main/java/com/yahoo/searchdefinition/document/ImmutableSDField.java index d54f76614c0..6d1dee67641 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/document/ImmutableSDField.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/document/ImmutableSDField.java @@ -4,7 +4,7 @@ package com.yahoo.searchdefinition.document; import com.yahoo.document.DataType; import com.yahoo.document.Field; import com.yahoo.searchdefinition.Index; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.vespa.documentmodel.SummaryField; import com.yahoo.vespa.indexinglanguage.expressions.Expression; import com.yahoo.vespa.indexinglanguage.expressions.ScriptExpression; @@ -69,7 +69,7 @@ public interface ImmutableSDField { Stemming getStemming(); - Stemming getStemming(Search search); + Stemming getStemming(Schema schema); Ranking getRanking(); diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/document/ImportedFields.java b/config-model/src/main/java/com/yahoo/searchdefinition/document/ImportedFields.java index 92a55d40048..18bd3f43445 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/document/ImportedFields.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/document/ImportedFields.java @@ -1,6 +1,8 @@ // Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. package com.yahoo.searchdefinition.document; +import com.yahoo.searchdefinition.Schema; + import java.util.Collections; import java.util.Map; diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/document/SDDocumentType.java b/config-model/src/main/java/com/yahoo/searchdefinition/document/SDDocumentType.java index 5d50abec8bb..6424db1c2dd 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/document/SDDocumentType.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/document/SDDocumentType.java @@ -14,7 +14,7 @@ import com.yahoo.documentmodel.NewDocumentType; import com.yahoo.documentmodel.VespaDocumentType; import com.yahoo.searchdefinition.DocumentReferences; import com.yahoo.searchdefinition.FieldSets; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import java.io.Serializable; import java.util.ArrayList; @@ -66,17 +66,15 @@ public class SDDocumentType implements Cloneable, Serializable { /** * For adding structs defined in document scope * - * @param dt The struct to add. + * @param dt the struct to add * @return self, for chaining */ public SDDocumentType addType(SDDocumentType dt) { NewDocumentType.Name name = new NewDocumentType.Name(dt.getName()); - if (getType(name) != null) { - throw new IllegalArgumentException("Data type '" + name.toString() + "' has already been used."); - } - if (name.getName() == docType.getName()) { - throw new IllegalArgumentException("Data type '" + name.toString() + "' can not have same name as its defining document."); - } + if (getType(name) != null) + throw new IllegalArgumentException("Data type '" + name + "' has already been used."); + if (name.getName() == docType.getName()) + throw new IllegalArgumentException("Data type '" + name + "' can not have same name as its defining document."); ownedTypes.put(name, dt); return this; } @@ -113,12 +111,11 @@ public class SDDocumentType implements Cloneable, Serializable { return this; } - /** - * Access to all owned datatypes. - * @return all types - */ + /** Returns all owned datatypes. */ public Collection<SDDocumentType> getTypes() { return ownedTypes.values(); } - public Collection<AnnotationType> getAnnotations() { return annotationTypes.getTypes().values(); } + + // TODO: Include inherited + public Map<String, AnnotationType> getAnnotations() { return annotationTypes.getTypes(); } public AnnotationType findAnnotation(String name) { return annotationTypes.getType(name); } public Collection<SDDocumentType> getAllTypes() { @@ -149,13 +146,13 @@ public class SDDocumentType implements Cloneable, Serializable { * The document type id will be generated as a hash from the document type name. * * @param name The name of the new document type - * @param search check for type ID collisions in this search definition + * @param schema check for type ID collisions in this search definition */ @SuppressWarnings("deprecation") - public SDDocumentType(String name, Search search) { + public SDDocumentType(String name, Schema schema) { docType = new DocumentType(name); docType.contentStruct().setCompressionConfig(new CompressionConfig()); - validateId(search); + validateId(schema); inherit(VESPA_DOCUMENT); } @@ -196,10 +193,12 @@ public class SDDocumentType implements Cloneable, Serializable { public Collection<SDDocumentType> getInheritedTypes() { return inheritedTypes.values(); } - protected void validateId(Search search) { - if (search == null) return; - if (search.getDocument(getName()) == null) return; - SDDocumentType doc = search.getDocument(); + public Map<DataTypeName, SDDocumentType> inheritedTypes() { return inheritedTypes; } + + protected void validateId(Schema schema) { + if (schema == null) return; + if (schema.getDocument(getName()) == null) return; + SDDocumentType doc = schema.getDocument(); throw new IllegalArgumentException("Failed creating document type '" + getName() + "', " + "document type '" + doc.getName() + "' already uses ID '" + doc.getName() + "'"); } @@ -249,7 +248,7 @@ public class SDDocumentType implements Cloneable, Serializable { for (Field pField : parent.fieldSet()) { if (pField.getName().equals(field.getName())) { if (!pField.getDataType().equals(field.getDataType())) { - throw new IllegalArgumentException("For search '" + getName() + "', field '" + field.getName() + + throw new IllegalArgumentException("For " + this + ", field '" + field.getName() + "': Datatype can not be different from that of same field " + "in the supertype '" + parent.getName() + "'"); } @@ -300,7 +299,7 @@ public class SDDocumentType implements Cloneable, Serializable { @Override public String toString() { - return "SD document type '" + docType.getName() + "'"; + return "document type '" + docType.getName() + "'"; } private static SDDocumentType createSDDocumentType(StructDataType structType) { @@ -312,7 +311,7 @@ public class SDDocumentType implements Cloneable, Serializable { return docType; } - /** The field sets defined for this type and its {@link Search} */ + /** The field sets defined for this type and its {@link Schema} */ public FieldSets getFieldSets() { return fieldSets; } diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/document/SDField.java b/config-model/src/main/java/com/yahoo/searchdefinition/document/SDField.java index a66820a9bfe..566342403d4 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/document/SDField.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/document/SDField.java @@ -12,7 +12,7 @@ import com.yahoo.language.Linguistics; import com.yahoo.language.process.Embedder; import com.yahoo.language.simple.SimpleLinguistics; import com.yahoo.searchdefinition.Index; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.fieldoperation.FieldOperation; import com.yahoo.searchdefinition.fieldoperation.FieldOperationContainer; import com.yahoo.tensor.TensorType; @@ -672,11 +672,11 @@ public class SDField extends Field implements TypedKey, FieldOperationContainer, * Whether this field should be stemmed in this search definition */ @Override - public Stemming getStemming(Search search) { + public Stemming getStemming(Schema schema) { if (stemming != null) return stemming; else - return search.getStemming(); + return schema.getStemming(); } @Override diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/document/TemporaryImportedFields.java b/config-model/src/main/java/com/yahoo/searchdefinition/document/TemporaryImportedFields.java index 5733779ccc7..96f2f2f1d24 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/document/TemporaryImportedFields.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/document/TemporaryImportedFields.java @@ -1,7 +1,10 @@ // Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. package com.yahoo.searchdefinition.document; +import com.yahoo.searchdefinition.Schema; + import java.util.Collections; +import java.util.HashMap; import java.util.LinkedHashMap; import java.util.Map; @@ -14,15 +17,32 @@ import java.util.Map; */ public class TemporaryImportedFields { + private final Schema owner; private final Map<String, TemporaryImportedField> fields = new LinkedHashMap<>(); + public TemporaryImportedFields(Schema owner) { + this.owner = owner; + } + public void add(TemporaryImportedField importedField) { fields.put(importedField.fieldName(), importedField); } - public boolean hasField(String fieldName) { return fields.get(fieldName) != null; } + public boolean hasField(String fieldName) { + boolean has = fields.get(fieldName) != null; + if (has) return true; + if (owner.inherited().isEmpty()) return false; + if (owner.inherited().get().temporaryImportedFields().isEmpty()) return false; + return owner.inherited().get().temporaryImportedFields().get().hasField(fieldName); + } public Map<String, TemporaryImportedField> fields() { - return Collections.unmodifiableMap(fields); + if (owner.inherited().isEmpty()) return Collections.unmodifiableMap(fields); + if (owner.inherited().get().temporaryImportedFields().isEmpty()) return Collections.unmodifiableMap(fields); + + var allFields = new HashMap<>(owner.inherited().get().temporaryImportedFields().get().fields()); + allFields.putAll(fields); + return allFields; } + } diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/expressiontransforms/OnnxModelTransformer.java b/config-model/src/main/java/com/yahoo/searchdefinition/expressiontransforms/OnnxModelTransformer.java index abfb0c43d84..faf3c8085d8 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/expressiontransforms/OnnxModelTransformer.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/expressiontransforms/OnnxModelTransformer.java @@ -2,7 +2,7 @@ package com.yahoo.searchdefinition.expressiontransforms; import com.yahoo.path.Path; -import com.yahoo.searchdefinition.ImmutableSearch; +import com.yahoo.searchdefinition.ImmutableSchema; import com.yahoo.searchdefinition.OnnxModel; import com.yahoo.searchdefinition.RankProfile; import com.yahoo.searchlib.rankingexpression.Reference; @@ -54,7 +54,7 @@ public class OnnxModelTransformer extends ExpressionTransformer<RankProfileTrans } public static ExpressionNode transformFeature(ReferenceNode feature, RankProfile rankProfile) { - ImmutableSearch search = rankProfile.getSearch(); + ImmutableSchema search = rankProfile.getSearch(); final String featureName = feature.getName(); if ( ! featureName.equals("onnxModel") && ! featureName.equals("onnx")) return feature; diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/processing/AddAttributeTransformToSummaryOfImportedFields.java b/config-model/src/main/java/com/yahoo/searchdefinition/processing/AddAttributeTransformToSummaryOfImportedFields.java index 213928b3edb..ac4d51221ef 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/processing/AddAttributeTransformToSummaryOfImportedFields.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/processing/AddAttributeTransformToSummaryOfImportedFields.java @@ -3,7 +3,7 @@ package com.yahoo.searchdefinition.processing; import com.yahoo.config.application.api.DeployLogger; import com.yahoo.searchdefinition.RankProfileRegistry; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.document.ImmutableImportedComplexSDField; import com.yahoo.searchdefinition.document.ImmutableSDField; import com.yahoo.vespa.documentmodel.SummaryField; @@ -20,21 +20,21 @@ import java.util.stream.Stream; */ public class AddAttributeTransformToSummaryOfImportedFields extends Processor { - public AddAttributeTransformToSummaryOfImportedFields(Search search, + public AddAttributeTransformToSummaryOfImportedFields(Schema schema, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { - super(search, deployLogger, rankProfileRegistry, queryProfiles); + super(schema, deployLogger, rankProfileRegistry, queryProfiles); } @Override public void process(boolean validate, boolean documentsOnly) { - search.allImportedFields() - .forEach(field -> setTransform(field)); + schema.allImportedFields() + .forEach(field -> setTransform(field)); } private Stream<SummaryField> getSummaryFieldsForImportedField(ImmutableSDField importedField) { - return search.getSummaryFields(importedField).values().stream(); + return schema.getSummaryFields(importedField).values().stream(); } private void setTransform(ImmutableSDField field) { diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/processing/AddExtraFieldsToDocument.java b/config-model/src/main/java/com/yahoo/searchdefinition/processing/AddExtraFieldsToDocument.java index 152399337fd..9ade1786e37 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/processing/AddExtraFieldsToDocument.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/processing/AddExtraFieldsToDocument.java @@ -6,7 +6,7 @@ import com.yahoo.searchdefinition.RankProfileRegistry; import com.yahoo.document.DataType; import com.yahoo.document.Field; import com.yahoo.document.PositionDataType; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.document.Attribute; import com.yahoo.searchdefinition.document.ImmutableSDField; import com.yahoo.searchdefinition.document.SDDocumentType; @@ -15,14 +15,15 @@ import com.yahoo.vespa.documentmodel.SummaryField; import com.yahoo.vespa.model.container.search.QueryProfiles; /** - * This processor creates a {@link com.yahoo.searchdefinition.document.SDDocumentType} for each {@link Search} object which holds all the data that search + * This processor creates a {@link com.yahoo.searchdefinition.document.SDDocumentType} for each {@link Schema} + * object which holds all the data that search * associates with a document described in a search definition file. This includes all extra fields, summary fields and * implicit fields. All non-indexed and non-summary fields are discarded. */ public class AddExtraFieldsToDocument extends Processor { - AddExtraFieldsToDocument(Search search, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { - super(search, deployLogger, rankProfileRegistry, queryProfiles); + AddExtraFieldsToDocument(Schema schema, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { + super(schema, deployLogger, rankProfileRegistry, queryProfiles); } //TODO This is a tempoarry hack to avoid producing illegal code for fields not wanted anyway. @@ -32,44 +33,43 @@ public class AddExtraFieldsToDocument extends Processor { @Override public void process(boolean validate, boolean documentsOnly) { - SDDocumentType document = search.getDocument(); + SDDocumentType document = schema.getDocument(); if (document != null) { - for (Field field : search.extraFieldList()) { - addSdField(search, document, (SDField)field, validate); + for (SDField field : schema.extraFieldList()) { + addSdField(schema, document, field, validate); } //TODO Vespa 8 or sooner we should avoid the dirty addition of fields from dirty 'default' summary to document at all - for (SummaryField field : search.getSummary("default").getSummaryFields()) { + for (SummaryField field : schema.getSummary("default").getSummaryFields()) { if (dirtyLegalFieldNameCheck(field.getName())) { - addSummaryField(search, document, field, validate); + addSummaryField(schema, document, field, validate); } } } } - private void addSdField(Search search, SDDocumentType document, SDField field, boolean validate) { + private void addSdField(Schema schema, SDDocumentType document, SDField field, boolean validate) { if (! field.hasIndex() && field.getAttributes().isEmpty()) { return; } for (Attribute atr : field.getAttributes().values()) { - // TODO Vespa 8 or before: Check if this sould be removed or changed to _zcurve. + // TODO Vespa 8 or before: Check if this should be removed or changed to _zcurve. if (atr.getName().equals(field.getName() + "_position")) { DataType type = PositionDataType.INSTANCE; if (atr.getCollectionType().equals(Attribute.CollectionType.ARRAY)) { type = DataType.getArray(type); } - addField(search, document, new SDField(document, atr.getName(), type), validate); + addField(schema, document, new SDField(document, atr.getName(), type), validate); } else if (!atr.getName().equals(field.getName())) { - addField(search, document, new SDField(document, atr.getName(), atr.getDataType()), validate); + addField(schema, document, new SDField(document, atr.getName(), atr.getDataType()), validate); } } - addField(search, document, field, validate); + addField(schema, document, field, validate); } - @SuppressWarnings("deprecation") - private void addSummaryField(Search search, SDDocumentType document, SummaryField field, boolean validate) { + private void addSummaryField(Schema schema, SDDocumentType document, SummaryField field, boolean validate) { Field docField = document.getField(field.getName()); if (docField == null) { - ImmutableSDField existingField = search.getField(field.getName()); + ImmutableSDField existingField = schema.getField(field.getName()); if (existingField == null) { SDField newField = new SDField(document, field.getName(), field.getDataType(), true); newField.setIsExtraField(true); @@ -79,14 +79,14 @@ public class AddExtraFieldsToDocument extends Processor { } } else if (!docField.getDataType().equals(field.getDataType())) { if (validate) - throw newProcessException(search, field, "Summary field has conflicting type."); + throw newProcessException(schema, field, "Summary field has conflicting type."); } } - private void addField(Search search, SDDocumentType document, Field field, boolean validate) { - if (document.getField(field.getName()) != null) { + private void addField(Schema schema, SDDocumentType document, Field field, boolean validate) { + if (document.getField(field.getName()) != null && !(document.getField(field.getName()) == field)) { if (validate) - throw newProcessException(search, field, "Field shadows another."); + throw newProcessException(schema, field, "Field shadows another."); } document.addField(field); } diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/processing/AdjustPositionSummaryFields.java b/config-model/src/main/java/com/yahoo/searchdefinition/processing/AdjustPositionSummaryFields.java index fd103f3901c..176fcf2130f 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/processing/AdjustPositionSummaryFields.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/processing/AdjustPositionSummaryFields.java @@ -6,7 +6,7 @@ import com.yahoo.document.ArrayDataType; import com.yahoo.document.DataType; import com.yahoo.document.PositionDataType; import com.yahoo.searchdefinition.RankProfileRegistry; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.document.Attribute; import com.yahoo.searchdefinition.document.ImmutableSDField; import com.yahoo.vespa.documentmodel.DocumentSummary; @@ -21,13 +21,13 @@ import com.yahoo.vespa.model.container.search.QueryProfiles; */ public class AdjustPositionSummaryFields extends Processor { - public AdjustPositionSummaryFields(Search search, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { - super(search, deployLogger, rankProfileRegistry, queryProfiles); + public AdjustPositionSummaryFields(Schema schema, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { + super(schema, deployLogger, rankProfileRegistry, queryProfiles); } @Override public void process(boolean validate, boolean documentsOnly) { - for (DocumentSummary summary : search.getSummaries().values()) { + for (DocumentSummary summary : schema.getSummaries().values()) { scanSummary(summary); } } @@ -37,7 +37,7 @@ public class AdjustPositionSummaryFields extends Processor { if (isPositionDataType(summaryField.getDataType())) { String originalSource = summaryField.getSingleSource(); if (originalSource.indexOf('.') == -1) { // Eliminate summary fields with pos.x or pos.y as source - ImmutableSDField sourceField = search.getField(originalSource); + ImmutableSDField sourceField = schema.getField(originalSource); if (sourceField != null) { String zCurve = null; if (sourceField.getDataType().equals(summaryField.getDataType())) { @@ -77,7 +77,7 @@ public class AdjustPositionSummaryFields extends Processor { } private void ensureSummaryField(DocumentSummary summary, String fieldName, DataType dataType, Source source, SummaryTransform transform) { - SummaryField oldField = search.getSummaryField(fieldName); + SummaryField oldField = schema.getSummaryField(fieldName); if (oldField == null) { SummaryField newField = new SummaryField(fieldName, dataType, transform); newField.addSource(source); @@ -97,9 +97,9 @@ public class AdjustPositionSummaryFields extends Processor { } private boolean hasPositionAttribute(String name) { - Attribute attribute = search.getAttribute(name); + Attribute attribute = schema.getAttribute(name); if (attribute == null) { - ImmutableSDField field = search.getField(name); + ImmutableSDField field = schema.getField(name); if (field != null && field.isImportedField()) { attribute = field.getAttribute(); } @@ -121,7 +121,7 @@ public class AdjustPositionSummaryFields extends Processor { } private void fail(SummaryField summaryField, String msg) { - throw newProcessException(search.getName(), summaryField.getName(), msg); + throw newProcessException(schema.getName(), summaryField.getName(), msg); } } diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/processing/AttributeProperties.java b/config-model/src/main/java/com/yahoo/searchdefinition/processing/AttributeProperties.java index e7cc562b42d..42e14a2c1ee 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/processing/AttributeProperties.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/processing/AttributeProperties.java @@ -3,10 +3,10 @@ package com.yahoo.searchdefinition.processing; import com.yahoo.config.application.api.DeployLogger; import com.yahoo.searchdefinition.RankProfileRegistry; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.document.Attribute; import com.yahoo.searchdefinition.document.ImmutableSDField; import com.yahoo.searchdefinition.document.SDField; -import com.yahoo.searchdefinition.Search; import com.yahoo.vespa.model.container.search.QueryProfiles; /** @@ -16,13 +16,13 @@ import com.yahoo.vespa.model.container.search.QueryProfiles; */ public class AttributeProperties extends Processor { - public AttributeProperties(Search search, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { - super(search, deployLogger, rankProfileRegistry, queryProfiles); + public AttributeProperties(Schema schema, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { + super(schema, deployLogger, rankProfileRegistry, queryProfiles); } @Override public void process(boolean validate, boolean documentsOnly) { - for (ImmutableSDField field : search.allConcreteFields()) { + for (ImmutableSDField field : schema.allConcreteFields()) { String fieldName = field.getName(); // For each attribute, check if the attribute has been created @@ -33,7 +33,7 @@ public class AttributeProperties extends Processor { } // Check other fields or statements that may have created this attribute. boolean created = false; - for (SDField f : search.allConcreteFields()) { + for (SDField f : schema.allConcreteFields()) { // Checking against the field we are looking at if (!f.getName().equals(fieldName)) { if (attributeCreated(f, attribute.getName())) { diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/processing/AttributesImplicitWord.java b/config-model/src/main/java/com/yahoo/searchdefinition/processing/AttributesImplicitWord.java index 27e9d5a070f..d3fa3282f4f 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/processing/AttributesImplicitWord.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/processing/AttributesImplicitWord.java @@ -4,10 +4,10 @@ package com.yahoo.searchdefinition.processing; import com.yahoo.config.application.api.DeployLogger; import com.yahoo.searchdefinition.RankProfileRegistry; import com.yahoo.document.DataType; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.document.ImmutableSDField; import com.yahoo.searchdefinition.document.Matching; import com.yahoo.document.NumericDataType; -import com.yahoo.searchdefinition.Search; import com.yahoo.vespa.model.container.search.QueryProfiles; /** @@ -18,13 +18,13 @@ import com.yahoo.vespa.model.container.search.QueryProfiles; */ public class AttributesImplicitWord extends Processor { - public AttributesImplicitWord(Search search, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { - super(search, deployLogger, rankProfileRegistry, queryProfiles); + public AttributesImplicitWord(Schema schema, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { + super(schema, deployLogger, rankProfileRegistry, queryProfiles); } @Override public void process(boolean validate, boolean documentsOnly) { - for (ImmutableSDField field : search.allConcreteFields()) { + for (ImmutableSDField field : schema.allConcreteFields()) { processFieldRecursive(field); } } diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/processing/Bolding.java b/config-model/src/main/java/com/yahoo/searchdefinition/processing/Bolding.java index 8c62cdfaec6..575a037be91 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/processing/Bolding.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/processing/Bolding.java @@ -5,7 +5,7 @@ import com.yahoo.config.application.api.DeployLogger; import com.yahoo.searchdefinition.RankProfileRegistry; import com.yahoo.document.DataType; import com.yahoo.searchdefinition.document.ImmutableSDField; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.vespa.documentmodel.SummaryField; import com.yahoo.vespa.model.container.search.QueryProfiles; @@ -17,14 +17,14 @@ import com.yahoo.vespa.model.container.search.QueryProfiles; */ public class Bolding extends Processor { - public Bolding(Search search, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { - super(search, deployLogger, rankProfileRegistry, queryProfiles); + public Bolding(Schema schema, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { + super(schema, deployLogger, rankProfileRegistry, queryProfiles); } @Override public void process(boolean validate, boolean documentsOnly) { if ( ! validate) return; - for (ImmutableSDField field : search.allConcreteFields()) { + for (ImmutableSDField field : schema.allConcreteFields()) { for (SummaryField summary : field.getSummaryFields().values()) { if (summary.getTransform().isBolded() && !((summary.getDataType() == DataType.STRING) || (summary.getDataType() == DataType.URI))) diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/processing/BoolAttributeValidator.java b/config-model/src/main/java/com/yahoo/searchdefinition/processing/BoolAttributeValidator.java index f4bf4432aa0..6676e20c341 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/processing/BoolAttributeValidator.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/processing/BoolAttributeValidator.java @@ -3,7 +3,7 @@ package com.yahoo.searchdefinition.processing; import com.yahoo.config.application.api.DeployLogger; import com.yahoo.searchdefinition.RankProfileRegistry; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.document.Attribute; import com.yahoo.vespa.model.container.search.QueryProfiles; @@ -16,20 +16,20 @@ import com.yahoo.vespa.model.container.search.QueryProfiles; */ public class BoolAttributeValidator extends Processor { - public BoolAttributeValidator(Search search, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { - super(search, deployLogger, rankProfileRegistry, queryProfiles); + public BoolAttributeValidator(Schema schema, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { + super(schema, deployLogger, rankProfileRegistry, queryProfiles); } @Override public void process(boolean validate, boolean documentsOnly) { - for (var field : search.allConcreteFields()) { + for (var field : schema.allConcreteFields()) { var attribute = field.getAttribute(); if (attribute == null) { continue; } if (attribute.getType().equals(Attribute.Type.BOOL) && !attribute.getCollectionType().equals(Attribute.CollectionType.SINGLE)) { - fail(search, field, "Only single value bool attribute fields are supported"); + fail(schema, field, "Only single value bool attribute fields are supported"); } } } diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/processing/BuiltInFieldSets.java b/config-model/src/main/java/com/yahoo/searchdefinition/processing/BuiltInFieldSets.java index fc23042649b..ce93c1a5c3f 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/processing/BuiltInFieldSets.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/processing/BuiltInFieldSets.java @@ -5,7 +5,7 @@ import com.yahoo.config.application.api.DeployLogger; import com.yahoo.document.DocumentType; import com.yahoo.searchdefinition.RankProfileRegistry; import com.yahoo.document.Field; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.document.SDField; import com.yahoo.vespa.model.container.search.QueryProfiles; @@ -19,8 +19,8 @@ public class BuiltInFieldSets extends Processor { public static final String SEARCH_FIELDSET_NAME = "[search]"; // Public due to oddities in position handling. public static final String INTERNAL_FIELDSET_NAME = "[internal]"; // This one populated from misc places - public BuiltInFieldSets(Search search, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { - super(search, deployLogger, rankProfileRegistry, queryProfiles); + public BuiltInFieldSets(Schema schema, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { + super(schema, deployLogger, rankProfileRegistry, queryProfiles); } @Override @@ -29,21 +29,21 @@ public class BuiltInFieldSets extends Processor { addSearchFieldSet(); // "Hook" the field sets on search onto the document types, since we will include them // on the document configs - search.getDocument().setFieldSets(search.fieldSets()); + schema.getDocument().setFieldSets(schema.fieldSets()); } private void addSearchFieldSet() { - for (SDField searchField : search.extraFieldList()) { - search.fieldSets().addBuiltInFieldSetItem(SEARCH_FIELDSET_NAME, searchField.getName()); + for (SDField searchField : schema.extraFieldList()) { + schema.fieldSets().addBuiltInFieldSetItem(SEARCH_FIELDSET_NAME, searchField.getName()); } } private void addDocumentFieldSet() { - for (Field docField : search.getDocument().fieldSet()) { + for (Field docField : schema.getDocument().fieldSet()) { if (docField instanceof SDField && ((SDField) docField).isExtraField()) { continue; // skip } - search.fieldSets().addBuiltInFieldSetItem(DocumentType.DOCUMENT, docField.getName()); + schema.fieldSets().addBuiltInFieldSetItem(DocumentType.DOCUMENT, docField.getName()); } } diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/processing/CreatePositionZCurve.java b/config-model/src/main/java/com/yahoo/searchdefinition/processing/CreatePositionZCurve.java index 7b0ea68db5a..699abb1e792 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/processing/CreatePositionZCurve.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/processing/CreatePositionZCurve.java @@ -6,7 +6,7 @@ import com.yahoo.searchdefinition.RankProfileRegistry; import com.yahoo.document.ArrayDataType; import com.yahoo.document.DataType; import com.yahoo.document.PositionDataType; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.document.Attribute; import com.yahoo.searchdefinition.document.SDField; import com.yahoo.vespa.documentmodel.SummaryField; @@ -33,18 +33,18 @@ import java.util.logging.Level; */ public class CreatePositionZCurve extends Processor { - public CreatePositionZCurve(Search search, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { - super(search, deployLogger, rankProfileRegistry, queryProfiles); + public CreatePositionZCurve(Schema schema, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { + super(schema, deployLogger, rankProfileRegistry, queryProfiles); } @Override public void process(boolean validate, boolean documentsOnly) { - for (SDField field : search.allConcreteFields()) { + for (SDField field : schema.allConcreteFields()) { DataType fieldType = field.getDataType(); if ( ! isSupportedPositionType(fieldType)) continue; if (validate && field.doesIndexing()) { - fail(search, field, "Indexing of data type '" + fieldType.getName() + "' is not supported, " + + fail(schema, field, "Indexing of data type '" + fieldType.getName() + "' is not supported, " + "replace 'index' statement with 'attribute'."); } @@ -57,8 +57,8 @@ public class CreatePositionZCurve extends Processor { String zName = PositionDataType.getZCurveFieldName(fieldName); SDField zCurveField = createZCurveField(field, zName, validate); - search.addExtraField(zCurveField); - search.fieldSets().addBuiltInFieldSetItem(BuiltInFieldSets.INTERNAL_FIELDSET_NAME, zCurveField.getName()); + schema.addExtraField(zCurveField); + schema.fieldSets().addBuiltInFieldSetItem(BuiltInFieldSets.INTERNAL_FIELDSET_NAME, zCurveField.getName()); // configure summary Collection<String> summaryTo = removeSummaryTo(field); @@ -88,8 +88,8 @@ public class CreatePositionZCurve extends Processor { } private SDField createZCurveField(SDField inputField, String fieldName, boolean validate) { - if (validate && search.getConcreteField(fieldName) != null || search.getAttribute(fieldName) != null) { - throw newProcessException(search, null, "Incompatible position attribute '" + fieldName + + if (validate && schema.getConcreteField(fieldName) != null || schema.getAttribute(fieldName) != null) { + throw newProcessException(schema, null, "Incompatible position attribute '" + fieldName + "' already created."); } boolean isArray = inputField.getDataType() instanceof ArrayDataType; @@ -109,7 +109,7 @@ public class CreatePositionZCurve extends Processor { private void ensureCompatibleSummary(SDField field, String sourceName, String summaryName, DataType summaryType, SummaryTransform summaryTransform, Collection<String> summaryTo, boolean validate) { - SummaryField summary = search.getSummaryField(summaryName); + SummaryField summary = schema.getSummaryField(summaryName); if (summary == null) { summary = new SummaryField(summaryName, summaryType, summaryTransform); summary.addDestination("default"); @@ -117,7 +117,7 @@ public class CreatePositionZCurve extends Processor { field.addSummaryField(summary); } else if (!summary.getDataType().equals(summaryType)) { if (validate) - fail(search, field, "Incompatible summary field '" + summaryName + "' type "+summary.getDataType()+" already created."); + fail(schema, field, "Incompatible summary field '" + summaryName + "' type " + summary.getDataType() + " already created."); } else if (summary.getTransform() == SummaryTransform.NONE) { summary.setTransform(summaryTransform); summary.addDestination("default"); diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/processing/DictionaryProcessor.java b/config-model/src/main/java/com/yahoo/searchdefinition/processing/DictionaryProcessor.java index d8331746968..dc620e38e96 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/processing/DictionaryProcessor.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/processing/DictionaryProcessor.java @@ -5,11 +5,10 @@ import com.yahoo.config.application.api.DeployLogger; import com.yahoo.document.NumericDataType; import com.yahoo.document.PrimitiveDataType; import com.yahoo.searchdefinition.RankProfileRegistry; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.document.Attribute; import com.yahoo.searchdefinition.document.Case; import com.yahoo.searchdefinition.document.Dictionary; -import com.yahoo.searchdefinition.document.Matching; import com.yahoo.searchdefinition.document.SDField; import com.yahoo.vespa.model.container.search.QueryProfiles; @@ -20,12 +19,12 @@ import com.yahoo.vespa.model.container.search.QueryProfiles; * @author baldersheim */ public class DictionaryProcessor extends Processor { - public DictionaryProcessor(Search search, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { - super(search, deployLogger, rankProfileRegistry, queryProfiles); + public DictionaryProcessor(Schema schema, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { + super(schema, deployLogger, rankProfileRegistry, queryProfiles); } @Override public void process(boolean validate, boolean documentsOnly) { - for (SDField field : search.allConcreteFields()) { + for (SDField field : schema.allConcreteFields()) { Attribute attribute = field.getAttribute(); if (attribute == null) continue; attribute.setCase(field.getMatching().getCase()); @@ -35,20 +34,20 @@ public class DictionaryProcessor extends Processor { if (attribute.isFastSearch()) { attribute.setDictionary(dictionary); } else { - fail(search, field, "You must specify 'attribute:fast-search' to allow dictionary control"); + fail(schema, field, "You must specify 'attribute:fast-search' to allow dictionary control"); } } else if (attribute.getDataType().getPrimitiveType() == PrimitiveDataType.STRING) { attribute.setDictionary(dictionary); if (dictionary.getType() == Dictionary.Type.HASH) { if (dictionary.getMatch() != Case.CASED) { - fail(search, field, "hash dictionary require cased match"); + fail(schema, field, "hash dictionary require cased match"); } } if (! dictionary.getMatch().equals(attribute.getCase())) { - fail(search, field, "Dictionary casing '" + dictionary.getMatch() + "' does not match field match casing '" + attribute.getCase() + "'"); + fail(schema, field, "Dictionary casing '" + dictionary.getMatch() + "' does not match field match casing '" + attribute.getCase() + "'"); } } else { - fail(search, field, "You can only specify 'dictionary:' for numeric or string fields"); + fail(schema, field, "You can only specify 'dictionary:' for numeric or string fields"); } } } diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/processing/DisallowComplexMapAndWsetKeyTypes.java b/config-model/src/main/java/com/yahoo/searchdefinition/processing/DisallowComplexMapAndWsetKeyTypes.java index 74bc476f96b..6ffe00458a8 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/processing/DisallowComplexMapAndWsetKeyTypes.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/processing/DisallowComplexMapAndWsetKeyTypes.java @@ -8,9 +8,9 @@ import com.yahoo.searchdefinition.RankProfileRegistry; import com.yahoo.document.DataType; import com.yahoo.document.MapDataType; import com.yahoo.document.PrimitiveDataType; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.document.SDField; import com.yahoo.document.WeightedSetDataType; -import com.yahoo.searchdefinition.Search; import com.yahoo.vespa.model.container.search.QueryProfiles; /** @@ -20,8 +20,8 @@ import com.yahoo.vespa.model.container.search.QueryProfiles; */ public class DisallowComplexMapAndWsetKeyTypes extends Processor { - public DisallowComplexMapAndWsetKeyTypes(Search search, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { - super(search, deployLogger, rankProfileRegistry, queryProfiles); + public DisallowComplexMapAndWsetKeyTypes(Schema schema, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { + super(schema, deployLogger, rankProfileRegistry, queryProfiles); } @Override @@ -30,7 +30,7 @@ public class DisallowComplexMapAndWsetKeyTypes extends Processor { // TODO also traverse struct types to search for bad map or wset types. // Do this after document manager is fixed, do not start using the static stuff on SDDocumentTypes any more. - for (SDField field : search.allConcreteFields()) { + for (SDField field : schema.allConcreteFields()) { checkFieldType(field, field.getDataType()); } } @@ -42,12 +42,12 @@ public class DisallowComplexMapAndWsetKeyTypes extends Processor { } else if (dataType instanceof WeightedSetDataType) { DataType nestedType = ((WeightedSetDataType) dataType).getNestedType(); if ( ! (nestedType instanceof PrimitiveDataType)) { - fail(search, field, "Weighted set must have a primitive key type."); + fail(schema, field, "Weighted set must have a primitive key type."); } } else if (dataType instanceof MapDataType) { DataType keyType = ((MapDataType) dataType).getKeyType(); if ( ! (keyType instanceof PrimitiveDataType)) { - fail(search, field, "Map key type must be a primitive type."); + fail(schema, field, "Map key type must be a primitive type."); } checkFieldType(field, ((MapDataType) dataType).getValueType()); } diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/processing/DiversitySettingsValidator.java b/config-model/src/main/java/com/yahoo/searchdefinition/processing/DiversitySettingsValidator.java index 282ca6b3645..40de25dbc76 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/processing/DiversitySettingsValidator.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/processing/DiversitySettingsValidator.java @@ -4,7 +4,7 @@ package com.yahoo.searchdefinition.processing; import com.yahoo.config.application.api.DeployLogger; import com.yahoo.searchdefinition.RankProfile; import com.yahoo.searchdefinition.RankProfileRegistry; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.document.Attribute; import com.yahoo.vespa.model.container.search.QueryProfiles; @@ -13,8 +13,8 @@ import com.yahoo.vespa.model.container.search.QueryProfiles; */ public class DiversitySettingsValidator extends Processor { - public DiversitySettingsValidator(Search search, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { - super(search, deployLogger, rankProfileRegistry, queryProfiles); + public DiversitySettingsValidator(Schema schema, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { + super(schema, deployLogger, rankProfileRegistry, queryProfiles); } @Override @@ -22,7 +22,7 @@ public class DiversitySettingsValidator extends Processor { if ( ! validate) return; if (documentsOnly) return; - for (RankProfile rankProfile : rankProfileRegistry.rankProfilesOf(search.getName())) { + for (RankProfile rankProfile : rankProfileRegistry.rankProfilesOf(schema.getName())) { if (rankProfile.getMatchPhaseSettings() != null && rankProfile.getMatchPhaseSettings().getDiversity() != null) { validate(rankProfile, rankProfile.getMatchPhaseSettings().getDiversity()); } @@ -30,8 +30,8 @@ public class DiversitySettingsValidator extends Processor { } private void validate(RankProfile rankProfile, RankProfile.DiversitySettings settings) { String attributeName = settings.getAttribute(); - new AttributeValidator(search.getName(), rankProfile.getName(), - search.getAttribute(attributeName), attributeName).validate(); + new AttributeValidator(schema.getName(), rankProfile.getName(), + schema.getAttribute(attributeName), attributeName).validate(); } private static class AttributeValidator extends MatchPhaseSettingsValidator.AttributeValidator { diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/processing/ExactMatch.java b/config-model/src/main/java/com/yahoo/searchdefinition/processing/ExactMatch.java index deb37adfa94..a516667c524 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/processing/ExactMatch.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/processing/ExactMatch.java @@ -5,7 +5,7 @@ import com.yahoo.config.application.api.DeployLogger; import com.yahoo.document.CollectionDataType; import com.yahoo.document.DataType; import com.yahoo.searchdefinition.RankProfileRegistry; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.document.Matching; import com.yahoo.searchdefinition.document.SDField; import com.yahoo.searchdefinition.document.Stemming; @@ -27,30 +27,30 @@ public class ExactMatch extends Processor { public static final String DEFAULT_EXACT_TERMINATOR = "@@"; - ExactMatch(Search search, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { - super(search, deployLogger, rankProfileRegistry, queryProfiles); + ExactMatch(Schema schema, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { + super(schema, deployLogger, rankProfileRegistry, queryProfiles); } @Override public void process(boolean validate, boolean documentsOnly) { - for (SDField field : search.allConcreteFields()) { - processField(field, search); + for (SDField field : schema.allConcreteFields()) { + processField(field, schema); } } - private void processField(SDField field, Search search) { + private void processField(SDField field, Schema schema) { Matching.Type matching = field.getMatching().getType(); if (matching.equals(Matching.Type.EXACT) || matching.equals(Matching.Type.WORD)) { - implementExactMatch(field, search); + implementExactMatch(field, schema); } else if (field.getMatching().getExactMatchTerminator() != null) { - warn(search, field, "exact-terminator requires 'exact' matching to have any effect."); + warn(schema, field, "exact-terminator requires 'exact' matching to have any effect."); } for (var structField : field.getStructFields()) { - processField(structField, search); + processField(structField, schema); } } - private void implementExactMatch(SDField field, Search search) { + private void implementExactMatch(SDField field, Schema schema) { field.setStemming(Stemming.NONE); field.getNormalizing().inferLowercase(); @@ -62,7 +62,7 @@ public class ExactMatch extends Processor { && ! field.getMatching().getExactMatchTerminator().equals("")) { exactTerminator = field.getMatching().getExactMatchTerminator(); } else { - info(search, field, + info(schema, field, "With 'exact' matching, an exact-terminator is needed," + " using default value '" + exactTerminator +"' as terminator"); } @@ -75,7 +75,7 @@ public class ExactMatch extends Processor { } ScriptExpression script = field.getIndexingScript(); if (new ExpressionSearcher<>(IndexExpression.class).containedIn(script)) { - field.setIndexingScript((ScriptExpression)new MyProvider(search).convert(field.getIndexingScript())); + field.setIndexingScript((ScriptExpression)new MyProvider(schema).convert(field.getIndexingScript())); } } @@ -85,8 +85,8 @@ public class ExactMatch extends Processor { private static class MyProvider extends TypedTransformProvider { - MyProvider(Search search) { - super(ExactExpression.class, search); + MyProvider(Schema schema) { + super(ExactExpression.class, schema); } @Override diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/processing/FastAccessValidator.java b/config-model/src/main/java/com/yahoo/searchdefinition/processing/FastAccessValidator.java index a6b61e28bfc..60cc5c1cbb4 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/processing/FastAccessValidator.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/processing/FastAccessValidator.java @@ -3,7 +3,7 @@ package com.yahoo.searchdefinition.processing; import com.yahoo.config.application.api.DeployLogger; import com.yahoo.searchdefinition.RankProfileRegistry; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.document.Attribute; import com.yahoo.vespa.model.container.search.QueryProfiles; @@ -16,26 +16,23 @@ import java.util.stream.Collectors; */ public class FastAccessValidator extends Processor { - public FastAccessValidator(Search search, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { - super(search, deployLogger, rankProfileRegistry, queryProfiles); + public FastAccessValidator(Schema schema, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { + super(schema, deployLogger, rankProfileRegistry, queryProfiles); } @Override public void process(boolean validate, boolean documentsOnly) { if ( ! validate) return; - String invalidAttributes = search.allFields() - .flatMap(field -> field.getAttributes().values().stream()) - .filter(FastAccessValidator::isIncompatibleAttribute) - .map(Attribute::getName) - .collect(Collectors.joining(", ")); + String invalidAttributes = schema.allFields() + .flatMap(field -> field.getAttributes().values().stream()) + .filter(FastAccessValidator::isIncompatibleAttribute) + .map(Attribute::getName) + .collect(Collectors.joining(", ")); if ( ! invalidAttributes.isEmpty()) { throw new IllegalArgumentException( - String.format( - "For search '%s': The following attributes have a type that is incompatible with fast-access: %s. " + - "Predicate, tensor and reference attributes are incompatible with fast-access.", - search.getName(), - invalidAttributes)); + "For " + schema + ": The following attributes have a type that is incompatible with fast-access: " + + invalidAttributes + ". Predicate, tensor and reference attributes are incompatible with fast-access."); } } diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/processing/FieldSetSettings.java b/config-model/src/main/java/com/yahoo/searchdefinition/processing/FieldSetSettings.java index 291b84fa67b..1c50f78b539 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/processing/FieldSetSettings.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/processing/FieldSetSettings.java @@ -3,7 +3,7 @@ package com.yahoo.searchdefinition.processing; import com.yahoo.config.application.api.DeployLogger; import com.yahoo.searchdefinition.RankProfileRegistry; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.document.FieldSet; import com.yahoo.searchdefinition.document.ImmutableSDField; import com.yahoo.searchdefinition.document.Matching; @@ -22,42 +22,42 @@ import com.yahoo.vespa.model.container.search.QueryProfiles; // (this requires adding normalizing and stemming settings to FieldSet). public class FieldSetSettings extends Processor { - public FieldSetSettings(Search search, + public FieldSetSettings(Schema schema, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { - super(search, deployLogger, rankProfileRegistry, queryProfiles); + super(schema, deployLogger, rankProfileRegistry, queryProfiles); } @Override public void process(boolean validate, boolean documentsOnly) { - for (FieldSet fieldSet : search.fieldSets().userFieldSets().values()) { + for (FieldSet fieldSet : schema.fieldSets().userFieldSets().values()) { if (validate) - checkFieldNames(search, fieldSet); - checkMatching(search, fieldSet); - checkNormalization(search, fieldSet); - checkStemming(search, fieldSet); + checkFieldNames(schema, fieldSet); + checkMatching(schema, fieldSet); + checkNormalization(schema, fieldSet); + checkStemming(schema, fieldSet); } } - private void checkFieldNames(Search search, FieldSet fieldSet) { + private void checkFieldNames(Schema schema, FieldSet fieldSet) { for (String field : fieldSet.getFieldNames()) { - if (search.getField(field) == null) - throw new IllegalArgumentException("For search '" + search.getName() + - "': Field '"+ field + "' in " + fieldSet + " does not exist."); + if (schema.getField(field) == null) + throw new IllegalArgumentException("For " + schema + ": Field '" + field + "' in " + + fieldSet + " does not exist."); } } - private void checkMatching(Search search, FieldSet fieldSet) { + private void checkMatching(Schema schema, FieldSet fieldSet) { Matching matching = fieldSet.getMatching(); for (String fieldName : fieldSet.getFieldNames()) { - ImmutableSDField field = search.getField(fieldName); + ImmutableSDField field = schema.getField(fieldName); Matching fieldMatching = field.getMatching(); if (matching == null) { matching = fieldMatching; } else { if ( ! matching.equals(fieldMatching)) { - warn(search, field.asField(), + warn(schema, field.asField(), "The matching settings for the fields in " + fieldSet + " are inconsistent " + "(explicitly or because of field type). This may lead to recall and ranking issues."); return; @@ -67,16 +67,16 @@ public class FieldSetSettings extends Processor { fieldSet.setMatching(matching); // Assign the uniquely determined matching to the field set } - private void checkNormalization(Search search, FieldSet fieldSet) { + private void checkNormalization(Schema schema, FieldSet fieldSet) { NormalizeLevel.Level normalizing = null; for (String fieldName : fieldSet.getFieldNames()) { - ImmutableSDField field = search.getField(fieldName); + ImmutableSDField field = schema.getField(fieldName); NormalizeLevel.Level fieldNorm = field.getNormalizing().getLevel(); if (normalizing == null) { normalizing = fieldNorm; } else { if ( ! normalizing.equals(fieldNorm)) { - warn(search, field.asField(), + warn(schema, field.asField(), "The normalization settings for the fields in " + fieldSet + " are inconsistent " + "(explicitly or because of field type). This may lead to recall and ranking issues."); return; @@ -85,16 +85,16 @@ public class FieldSetSettings extends Processor { } } - private void checkStemming(Search search, FieldSet fieldSet) { + private void checkStemming(Schema schema, FieldSet fieldSet) { Stemming stemming = null; for (String fieldName : fieldSet.getFieldNames()) { - ImmutableSDField field = search.getField(fieldName); + ImmutableSDField field = schema.getField(fieldName); Stemming fieldStemming = field.getStemming(); if (stemming == null) { stemming = fieldStemming; } else { if ( ! stemming.equals(fieldStemming)) { - warn(search, field.asField(), + warn(schema, field.asField(), "The stemming settings for the fields in the fieldset '"+fieldSet.getName()+ "' are inconsistent (explicitly or because of field type). " + "This may lead to recall and ranking issues."); diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/processing/FilterFieldNames.java b/config-model/src/main/java/com/yahoo/searchdefinition/processing/FilterFieldNames.java index 201aaf46470..d89f83c333f 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/processing/FilterFieldNames.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/processing/FilterFieldNames.java @@ -3,10 +3,10 @@ package com.yahoo.searchdefinition.processing; import com.yahoo.config.application.api.DeployLogger; import com.yahoo.searchdefinition.RankProfileRegistry; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.document.RankType; import com.yahoo.searchdefinition.document.SDField; import com.yahoo.searchdefinition.RankProfile; -import com.yahoo.searchdefinition.Search; import com.yahoo.vespa.model.container.search.QueryProfiles; import java.util.Iterator; @@ -21,23 +21,23 @@ import java.util.logging.Level; */ public class FilterFieldNames extends Processor { - public FilterFieldNames(Search search, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { - super(search, deployLogger, rankProfileRegistry, queryProfiles); + public FilterFieldNames(Schema schema, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { + super(schema, deployLogger, rankProfileRegistry, queryProfiles); } @Override public void process(boolean validate, boolean documentsOnly) { if (documentsOnly) return; - for (SDField f : search.allConcreteFields()) { + for (SDField f : schema.allConcreteFields()) { if (f.getRanking().isFilter()) { filterField(f.getName()); } } - for (RankProfile profile : rankProfileRegistry.rankProfilesOf(search)) { + for (RankProfile profile : rankProfileRegistry.rankProfilesOf(schema)) { Set<String> filterFields = new LinkedHashSet<>(); - findFilterFields(search, profile, filterFields); + findFilterFields(schema, profile, filterFields); for (Iterator<String> itr = filterFields.iterator(); itr.hasNext(); ) { String fieldName = itr.next(); profile.filterFields().add(fieldName); @@ -47,18 +47,18 @@ public class FilterFieldNames extends Processor { } private void filterField(String f) { - for (RankProfile rp : rankProfileRegistry.rankProfilesOf(search)) { + for (RankProfile rp : rankProfileRegistry.rankProfilesOf(schema)) { rp.filterFields().add(f); } } - private void findFilterFields(Search search, RankProfile profile, Set<String> filterFields) { + private void findFilterFields(Schema schema, RankProfile profile, Set<String> filterFields) { for (Iterator<RankProfile.RankSetting> itr = profile.declaredRankSettingIterator(); itr.hasNext(); ) { RankProfile.RankSetting setting = itr.next(); if (setting.getType().equals(RankProfile.RankSetting.Type.PREFERBITVECTOR) && ((Boolean)setting.getValue())) { String fieldName = setting.getFieldName(); - if (search.getConcreteField(fieldName) != null) { + if (schema.getConcreteField(fieldName) != null) { if ( ! profile.filterFields().contains(fieldName)) { filterFields.add(fieldName); } diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/processing/ImplicitSummaries.java b/config-model/src/main/java/com/yahoo/searchdefinition/processing/ImplicitSummaries.java index edf9f744150..6b09234c469 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/processing/ImplicitSummaries.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/processing/ImplicitSummaries.java @@ -5,10 +5,10 @@ import java.util.logging.Level; import com.yahoo.config.application.api.DeployLogger; import com.yahoo.searchdefinition.RankProfileRegistry; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.document.Attribute; import com.yahoo.document.PositionDataType; import com.yahoo.searchdefinition.document.SDField; -import com.yahoo.searchdefinition.Search; import com.yahoo.vespa.documentmodel.DocumentSummary; import com.yahoo.vespa.documentmodel.SummaryField; import com.yahoo.vespa.documentmodel.SummaryTransform; @@ -23,24 +23,24 @@ import static com.yahoo.searchdefinition.document.ComplexAttributeFieldUtils.isC */ public class ImplicitSummaries extends Processor { - public ImplicitSummaries(Search search, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { - super(search, deployLogger, rankProfileRegistry, queryProfiles); + public ImplicitSummaries(Schema schema, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { + super(schema, deployLogger, rankProfileRegistry, queryProfiles); } @Override public void process(boolean validate, boolean documentsOnly) { - DocumentSummary defaultSummary = search.getSummary("default"); + DocumentSummary defaultSummary = schema.getSummary("default"); if (defaultSummary == null) { - defaultSummary = new DocumentSummary("default", search); + defaultSummary = new DocumentSummary("default", schema); defaultSummary.setFromDisk(true); - search.addSummary(defaultSummary); + schema.addSummary(defaultSummary); } - for (SDField field : search.allConcreteFields()) { - collectSummaries(field, search, validate); + for (SDField field : schema.allConcreteFields()) { + collectSummaries(field, schema, validate); } - for (DocumentSummary documentSummary : search.getSummaries().values()) { + for (DocumentSummary documentSummary : schema.getSummaries().values()) { documentSummary.purgeImplicits(); } } @@ -49,7 +49,7 @@ public class ImplicitSummaries extends Processor { sdField.addSummaryFieldSources(summaryField); } - private void collectSummaries(SDField field ,Search search, boolean validate) { + private void collectSummaries(SDField field , Schema schema, boolean validate) { SummaryField addedSummaryField=null; // Implicit @@ -65,7 +65,7 @@ public class ImplicitSummaries extends Processor { } if (fieldSummaryField != null) { for (String dest : fieldSummaryField.getDestinations()) { - DocumentSummary summary = search.getSummary(dest); + DocumentSummary summary = schema.getSummary(dest); if (summary != null) { summary.add(fieldSummaryField); } @@ -79,7 +79,7 @@ public class ImplicitSummaries extends Processor { addedSummaryField.setTransform(SummaryTransform.ATTRIBUTE); } if (attribute.isPrefetch()) { - addPrefetchAttribute(attribute, field, search); + addPrefetchAttribute(attribute, field, schema); } } } @@ -92,7 +92,7 @@ public class ImplicitSummaries extends Processor { if (field.doesSummarying()) { for (Attribute attribute : field.getAttributes().values()) { if ( ! attribute.isPosition()) continue; - DocumentSummary attributePrefetchSummary = getOrCreateAttributePrefetchSummary(search); + DocumentSummary attributePrefetchSummary = getOrCreateAttributePrefetchSummary(schema); attributePrefetchSummary.add(field.getSummaryField(PositionDataType.getDistanceSummaryFieldName(fieldName))); attributePrefetchSummary.add(field.getSummaryField(PositionDataType.getPositionSummaryFieldName(fieldName))); } @@ -106,35 +106,35 @@ public class ImplicitSummaries extends Processor { summaryField.setTransform(SummaryTransform.ATTRIBUTE); } - if (isValid(summaryField, search, validate)) { - addToDestinations(summaryField, search); + if (isValid(summaryField, schema, validate)) { + addToDestinations(summaryField, schema); } } } - private DocumentSummary getOrCreateAttributePrefetchSummary(Search search) { - DocumentSummary summary = search.getSummary("attributeprefetch"); + private DocumentSummary getOrCreateAttributePrefetchSummary(Schema schema) { + DocumentSummary summary = schema.getSummary("attributeprefetch"); if (summary == null) { - summary = new DocumentSummary("attributeprefetch", search); - search.addSummary(summary); + summary = new DocumentSummary("attributeprefetch", schema); + schema.addSummary(summary); } return summary; } - private void addPrefetchAttribute(Attribute attribute,SDField field,Search search) { + private void addPrefetchAttribute(Attribute attribute, SDField field, Schema schema) { if (attribute.getPrefetchValue() == null) { // Prefetch by default - unless any summary makes this dynamic // Check if there is an implicit dynamic definition SummaryField fieldSummaryField = field.getSummaryField(attribute.getName()); if (fieldSummaryField != null && fieldSummaryField.getTransform().isDynamic()) return; // Check if an explicit class makes it dynamic (first is enough, as all must be the same, checked later) - SummaryField explicitSummaryField = search.getExplicitSummaryField(attribute.getName()); + SummaryField explicitSummaryField = schema.getExplicitSummaryField(attribute.getName()); if (explicitSummaryField != null && explicitSummaryField.getTransform().isDynamic()) return; } - DocumentSummary summary = getOrCreateAttributePrefetchSummary(search); + DocumentSummary summary = getOrCreateAttributePrefetchSummary(schema); SummaryField attributeSummaryField = new SummaryField(attribute.getName(), attribute.getDataType()); attributeSummaryField.addSource(attribute.getName()); attributeSummaryField.addDestination("attributeprefetch"); @@ -143,26 +143,26 @@ public class ImplicitSummaries extends Processor { } // Returns whether this is valid. Warns if invalid and ignorable. Throws if not ignorable. - private boolean isValid(SummaryField summaryField, Search search, boolean validate) { + private boolean isValid(SummaryField summaryField, Schema schema, boolean validate) { if (summaryField.getTransform() == SummaryTransform.DISTANCE || summaryField.getTransform() == SummaryTransform.POSITIONS) { int sourceCount = summaryField.getSourceCount(); if (validate && sourceCount != 1) { - throw newProcessException(search.getName(), summaryField.getName(), + throw newProcessException(schema.getName(), summaryField.getName(), "Expected 1 source field, got " + sourceCount + "."); } String sourceName = summaryField.getSingleSource(); - if (validate && search.getAttribute(sourceName) == null) { - throw newProcessException(search.getName(), summaryField.getName(), + if (validate && schema.getAttribute(sourceName) == null) { + throw newProcessException(schema.getName(), summaryField.getName(), "Summary source attribute '" + sourceName + "' not found."); } return true; } String fieldName = summaryField.getSourceField(); - SDField sourceField = search.getConcreteField(fieldName); + SDField sourceField = schema.getConcreteField(fieldName); if (validate && sourceField == null) { - throw newProcessException(search, summaryField, "Source field '" + fieldName + "' does not exist."); + throw newProcessException(schema, summaryField, "Source field '" + fieldName + "' does not exist."); } if (! sourceField.doesSummarying() && ! summaryField.getTransform().equals(SummaryTransform.ATTRIBUTE) && @@ -199,21 +199,21 @@ public class ImplicitSummaries extends Processor { return true; } - private void addToDestinations(SummaryField summaryField,Search search) { + private void addToDestinations(SummaryField summaryField, Schema schema) { if (summaryField.getDestinations().size() == 0) { - addToDestination("default", summaryField, search); + addToDestination("default", summaryField, schema); } else { for (String destinationName : summaryField.getDestinations()) - addToDestination(destinationName, summaryField, search); + addToDestination(destinationName, summaryField, schema); } } - private void addToDestination(String destinationName, SummaryField summaryField, Search search) { - DocumentSummary destination = search.getSummary(destinationName); + private void addToDestination(String destinationName, SummaryField summaryField, Schema schema) { + DocumentSummary destination = schema.getSummary(destinationName); if (destination == null) { - destination = new DocumentSummary(destinationName, search); - search.addSummary(destination); + destination = new DocumentSummary(destinationName, schema); + schema.addSummary(destination); destination.add(summaryField); } else { diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/processing/ImplicitSummaryFields.java b/config-model/src/main/java/com/yahoo/searchdefinition/processing/ImplicitSummaryFields.java index 0e13f38e207..ff0a0d07102 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/processing/ImplicitSummaryFields.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/processing/ImplicitSummaryFields.java @@ -4,7 +4,7 @@ package com.yahoo.searchdefinition.processing; import com.yahoo.config.application.api.DeployLogger; import com.yahoo.searchdefinition.RankProfileRegistry; import com.yahoo.document.DataType; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.vespa.documentmodel.DocumentSummary; import com.yahoo.vespa.documentmodel.SummaryField; import com.yahoo.vespa.documentmodel.SummaryTransform; @@ -16,13 +16,13 @@ import com.yahoo.vespa.model.container.search.QueryProfiles; */ public class ImplicitSummaryFields extends Processor { - public ImplicitSummaryFields(Search search, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { - super(search, deployLogger, rankProfileRegistry, queryProfiles); + public ImplicitSummaryFields(Schema schema, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { + super(schema, deployLogger, rankProfileRegistry, queryProfiles); } @Override public void process(boolean validate, boolean documentsOnly) { - for (DocumentSummary docsum : search.getSummaries().values()) { + for (DocumentSummary docsum : schema.getSummariesInThis().values()) { if (docsum.getInherited() != null) continue; // Implicit fields are added to inheriting summaries through their parent addField(docsum, new SummaryField("rankfeatures", DataType.STRING, SummaryTransform.RANKFEATURES), validate); addField(docsum, new SummaryField("summaryfeatures", DataType.STRING, SummaryTransform.SUMMARYFEATURES), validate); @@ -36,4 +36,5 @@ public class ImplicitSummaryFields extends Processor { } docsum.add(field); } + } diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/processing/ImportedFieldsResolver.java b/config-model/src/main/java/com/yahoo/searchdefinition/processing/ImportedFieldsResolver.java index c737df5d44f..f8a28061897 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/processing/ImportedFieldsResolver.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/processing/ImportedFieldsResolver.java @@ -7,7 +7,7 @@ import com.yahoo.document.PositionDataType; import com.yahoo.searchdefinition.DocumentReference; import com.yahoo.searchdefinition.DocumentReferences; import com.yahoo.searchdefinition.RankProfileRegistry; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.document.Attribute; import com.yahoo.searchdefinition.document.ImmutableSDField; import com.yahoo.searchdefinition.document.ImportedComplexField; @@ -26,7 +26,7 @@ import static com.yahoo.searchdefinition.document.ComplexAttributeFieldUtils.isM import static com.yahoo.searchdefinition.document.ComplexAttributeFieldUtils.isMapOfSimpleStruct; /** - * Iterates all imported fields from SD-parsing and validates and resolves them into concrete fields from referenced document types. + * Iterates all imported fields from schema parsing and validates and resolves them into concrete fields from referenced document types. * * @author geirst */ @@ -35,15 +35,15 @@ public class ImportedFieldsResolver extends Processor { private final Map<String, ImportedField> importedFields = new LinkedHashMap<>(); private final Optional<DocumentReferences> references; - public ImportedFieldsResolver(Search search, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { - super(search, deployLogger, rankProfileRegistry, queryProfiles); - references = search.getDocument().getDocumentReferences(); + public ImportedFieldsResolver(Schema schema, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { + super(schema, deployLogger, rankProfileRegistry, queryProfiles); + references = schema.getDocument().getDocumentReferences(); } @Override public void process(boolean validate, boolean documentsOnly) { - search.temporaryImportedFields().get().fields().forEach((name, field) -> resolveImportedField(field, validate)); - search.setImportedFields(new ImportedFields(importedFields)); + schema.temporaryImportedFields().get().fields().forEach((name, field) -> resolveImportedField(field, validate)); + schema.setImportedFields(new ImportedFields(importedFields)); } private void resolveImportedField(TemporaryImportedField importedField, boolean validate) { @@ -166,8 +166,8 @@ public class ImportedFieldsResolver extends Processor { private ImmutableSDField getTargetField(TemporaryImportedField importedField, DocumentReference reference) { String targetFieldName = importedField.targetFieldName(); - Search targetSearch = reference.targetSearch(); - ImmutableSDField targetField = targetSearch.getField(targetFieldName); + Schema targetSchema = reference.targetSearch(); + ImmutableSDField targetField = targetSchema.getField(targetFieldName); if (targetField == null) { fail(importedField, targetFieldAsString(targetFieldName, reference) + ": Not found"); } @@ -193,14 +193,15 @@ public class ImportedFieldsResolver extends Processor { } private void fail(TemporaryImportedField importedField, String msg) { - throw new IllegalArgumentException("For search '" + search.getName() + "', import field '" + importedField.fieldName() + "': " + msg); + throw new IllegalArgumentException("For " + schema + ", import field '" + + importedField.fieldName() + "': " + msg); } private void fail(TemporaryImportedField importedField, String importedNestedFieldName, String msg) { if (importedField.fieldName().equals(importedNestedFieldName)) { fail(importedField, msg); } - throw new IllegalArgumentException("For search '" + search.getName() + "', import field '" + - importedField.fieldName() + "' (nested to '" + importedNestedFieldName + "'): " + msg); + throw new IllegalArgumentException("For " + schema + ", import field '" + + importedField.fieldName() + "' (nested to '" + importedNestedFieldName + "'): " + msg); } } diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/processing/IndexFieldNames.java b/config-model/src/main/java/com/yahoo/searchdefinition/processing/IndexFieldNames.java index f0f52c49875..76e97ff0f2e 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/processing/IndexFieldNames.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/processing/IndexFieldNames.java @@ -4,7 +4,7 @@ package com.yahoo.searchdefinition.processing; import com.yahoo.config.application.api.DeployLogger; import com.yahoo.searchdefinition.RankProfileRegistry; import com.yahoo.searchdefinition.document.SDField; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.vespa.model.container.search.QueryProfiles; /** @@ -18,17 +18,17 @@ public class IndexFieldNames extends Processor { private static final String FIELD_NAME_REGEXP = "[a-zA-Z]\\w*"; - public IndexFieldNames(Search search, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { - super(search, deployLogger, rankProfileRegistry, queryProfiles); + public IndexFieldNames(Schema schema, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { + super(schema, deployLogger, rankProfileRegistry, queryProfiles); } @Override public void process(boolean validate, boolean documentsOnly) { if ( ! validate) return; - for (SDField field : search.allConcreteFields()) { + for (SDField field : schema.allConcreteFields()) { if ( ! field.getName().matches(FIELD_NAME_REGEXP) && ! legalDottedPositionField(field)) { - fail(search, field, " Not a legal field name. Legal expression: " + FIELD_NAME_REGEXP); + fail(schema, field, " Not a legal field name. Legal expression: " + FIELD_NAME_REGEXP); } } } diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/processing/IndexingInputs.java b/config-model/src/main/java/com/yahoo/searchdefinition/processing/IndexingInputs.java index 6117f3097ab..e589bd8e91a 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/processing/IndexingInputs.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/processing/IndexingInputs.java @@ -3,7 +3,7 @@ package com.yahoo.searchdefinition.processing; import com.yahoo.config.application.api.DeployLogger; import com.yahoo.searchdefinition.RankProfileRegistry; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.document.SDField; import com.yahoo.vespa.indexinglanguage.ExpressionConverter; import com.yahoo.vespa.indexinglanguage.ExpressionVisitor; @@ -21,13 +21,13 @@ import com.yahoo.vespa.model.container.search.QueryProfiles; */ public class IndexingInputs extends Processor { - public IndexingInputs(Search search, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { - super(search, deployLogger, rankProfileRegistry, queryProfiles); + public IndexingInputs(Schema schema, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { + super(schema, deployLogger, rankProfileRegistry, queryProfiles); } @Override public void process(boolean validate, boolean documentsOnly) { - for (SDField field : search.allConcreteFields()) { + for (SDField field : schema.allConcreteFields()) { ScriptExpression script = field.getIndexingScript(); if (script == null) continue; @@ -35,7 +35,7 @@ public class IndexingInputs extends Processor { script = (ScriptExpression)new DefaultToCurrentField(fieldName).convert(script); script = (ScriptExpression)new EnsureInputExpression(fieldName).convert(script); if (validate) - new VerifyInputExpression(search, field).visit(script); + new VerifyInputExpression(schema, field).visit(script); field.setIndexingScript(script); } @@ -85,11 +85,11 @@ public class IndexingInputs extends Processor { private class VerifyInputExpression extends ExpressionVisitor { - private final Search search; + private final Schema schema; private final SDField field; - public VerifyInputExpression(Search search, SDField field) { - this.search = search; + public VerifyInputExpression(Schema schema, SDField field) { + this.schema = schema; this.field = field; } @@ -97,10 +97,10 @@ public class IndexingInputs extends Processor { protected void doVisit(Expression exp) { if ( ! (exp instanceof InputExpression)) return; String inputField = ((InputExpression)exp).getFieldName(); - if (search.getField(inputField).hasFullIndexingDocprocRights()) return; + if (schema.getField(inputField).hasFullIndexingDocprocRights()) return; - fail(search, field, "Indexing script refers to field '" + inputField + "' which does not exist " + - "in document type '" + search.getDocument().getName() + "', and is not a mutable attribute."); + fail(schema, field, "Indexing script refers to field '" + inputField + "' which does not exist " + + "in document type '" + schema.getDocument().getName() + "', and is not a mutable attribute."); } } } diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/processing/IndexingOutputs.java b/config-model/src/main/java/com/yahoo/searchdefinition/processing/IndexingOutputs.java index c1ed856bb11..1479f9d8f5a 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/processing/IndexingOutputs.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/processing/IndexingOutputs.java @@ -5,7 +5,7 @@ import com.yahoo.config.application.api.DeployLogger; import com.yahoo.searchdefinition.RankProfileRegistry; import com.yahoo.document.DataType; import com.yahoo.document.Field; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.document.SDField; import com.yahoo.vespa.documentmodel.SummaryField; import com.yahoo.vespa.documentmodel.SummaryTransform; @@ -24,40 +24,40 @@ import java.util.*; */ public class IndexingOutputs extends Processor { - public IndexingOutputs(Search search, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { - super(search, deployLogger, rankProfileRegistry, queryProfiles); + public IndexingOutputs(Schema schema, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { + super(schema, deployLogger, rankProfileRegistry, queryProfiles); } @Override public void process(boolean validate, boolean documentsOnly) { - for (SDField field : search.allConcreteFields()) { + for (SDField field : schema.allConcreteFields()) { ScriptExpression script = field.getIndexingScript(); if (script == null) continue; Set<String> summaryFields = new TreeSet<>(); - findSummaryTo(search, field, summaryFields, summaryFields); - MyConverter converter = new MyConverter(search, field, summaryFields, validate); + findSummaryTo(schema, field, summaryFields, summaryFields); + MyConverter converter = new MyConverter(schema, field, summaryFields, validate); field.setIndexingScript((ScriptExpression)converter.convert(script)); } } - public void findSummaryTo(Search search, SDField field, Set<String> dynamicSummary, Set<String> staticSummary) { - Map<String, SummaryField> summaryFields = search.getSummaryFields(field); + public void findSummaryTo(Schema schema, SDField field, Set<String> dynamicSummary, Set<String> staticSummary) { + Map<String, SummaryField> summaryFields = schema.getSummaryFields(field); if (summaryFields.isEmpty()) { fillSummaryToFromField(field, dynamicSummary, staticSummary); } else { - fillSummaryToFromSearch(search, field, summaryFields, dynamicSummary, staticSummary); + fillSummaryToFromSearch(schema, field, summaryFields, dynamicSummary, staticSummary); } } - private void fillSummaryToFromSearch(Search search, SDField field, Map<String, SummaryField> summaryFields, - Set<String> dynamicSummary, Set<String> staticSummary) { + private void fillSummaryToFromSearch(Schema schema, SDField field, Map<String, SummaryField> summaryFields, + Set<String> dynamicSummary, Set<String> staticSummary) { for (SummaryField summaryField : summaryFields.values()) { - fillSummaryToFromSummaryField(search, field, summaryField, dynamicSummary, staticSummary); + fillSummaryToFromSummaryField(schema, field, summaryField, dynamicSummary, staticSummary); } } - private void fillSummaryToFromSummaryField(Search search, SDField field, SummaryField summaryField, + private void fillSummaryToFromSummaryField(Schema schema, SDField field, SummaryField summaryField, Set<String> dynamicSummary, Set<String> staticSummary) { SummaryTransform summaryTransform = summaryField.getTransform(); String summaryName = summaryField.getName(); @@ -69,7 +69,7 @@ public class IndexingOutputs extends Processor { if (summaryTransform.isDynamic()) { DataType fieldType = field.getDataType(); if (fieldType != DataType.URI && fieldType != DataType.STRING) { - warn(search, field, "Dynamic summaries are only supported for fields of type " + + warn(schema, field, "Dynamic summaries are only supported for fields of type " + "string, ignoring summary field '" + summaryField.getName() + "' for sd field '" + field.getName() + "' of type " + fieldType.getName() + "."); @@ -94,13 +94,13 @@ public class IndexingOutputs extends Processor { private class MyConverter extends ExpressionConverter { - final Search search; + final Schema schema; final Field field; final Set<String> summaryFields; final boolean validate; - MyConverter(Search search, Field field, Set<String> summaryFields, boolean validate) { - this.search = search; + MyConverter(Schema schema, Field field, Set<String> summaryFields, boolean validate) { + this.schema = schema; this.field = field; this.summaryFields = summaryFields.isEmpty() ? Collections.singleton(field.getName()) : summaryFields; this.validate = validate; @@ -116,7 +116,7 @@ public class IndexingOutputs extends Processor { return true; // inject appropriate field name } if ( validate && ! fieldName.equals(field.getName())) { - fail(search, field, "Indexing expression '" + exp + "' attempts to write to a field other than '" + + fail(schema, field, "Indexing expression '" + exp + "' attempts to write to a field other than '" + field.getName() + "'."); } return false; diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/processing/IndexingValidation.java b/config-model/src/main/java/com/yahoo/searchdefinition/processing/IndexingValidation.java index 915c2040224..eb9b561da73 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/processing/IndexingValidation.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/processing/IndexingValidation.java @@ -8,7 +8,7 @@ import com.yahoo.document.MapDataType; import com.yahoo.document.PositionDataType; import com.yahoo.document.WeightedSetDataType; import com.yahoo.searchdefinition.RankProfileRegistry; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.document.Attribute; import com.yahoo.searchdefinition.document.SDField; import com.yahoo.vespa.documentmodel.SummaryField; @@ -33,16 +33,16 @@ import java.util.Set; */ public class IndexingValidation extends Processor { - IndexingValidation(Search search, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { - super(search, deployLogger, rankProfileRegistry, queryProfiles); + IndexingValidation(Schema schema, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { + super(schema, deployLogger, rankProfileRegistry, queryProfiles); } @Override public void process(boolean validate, boolean documentsOnly) { if ( ! validate) return; - VerificationContext context = new VerificationContext(new MyAdapter(search)); - for (SDField field : search.allConcreteFields()) { + VerificationContext context = new VerificationContext(new MyAdapter(schema)); + for (SDField field : schema.allConcreteFields()) { ScriptExpression script = field.getIndexingScript(); try { script.verify(context); @@ -51,7 +51,7 @@ public class IndexingValidation extends Processor { converter.convert(exp); // TODO: stop doing this explicitly when visiting a script does not branch } } catch (VerificationException e) { - fail(search, field, "For expression '" + e.getExpression() + "': " + e.getMessage()); + fail(schema, field, "For expression '" + e.getExpression() + "': " + e.getMessage()); } } } @@ -94,15 +94,15 @@ public class IndexingValidation extends Processor { private static class MyAdapter implements FieldTypeAdapter { - final Search search; + final Schema schema; - MyAdapter(Search search) { - this.search = search; + MyAdapter(Schema schema) { + this.schema = schema; } @Override public DataType getInputType(Expression exp, String fieldName) { - SDField field = search.getDocumentField(fieldName); + SDField field = schema.getDocumentField(fieldName); if (field == null) { throw new VerificationException(exp, "Input field '" + fieldName + "' not found."); } @@ -114,21 +114,21 @@ public class IndexingValidation extends Processor { String fieldDesc; DataType fieldType; if (exp instanceof AttributeExpression) { - Attribute attribute = search.getAttribute(fieldName); + Attribute attribute = schema.getAttribute(fieldName); if (attribute == null) { throw new VerificationException(exp, "Attribute '" + fieldName + "' not found."); } fieldDesc = "attribute"; fieldType = attribute.getDataType(); } else if (exp instanceof IndexExpression) { - SDField field = search.getConcreteField(fieldName); + SDField field = schema.getConcreteField(fieldName); if (field == null) { throw new VerificationException(exp, "Index field '" + fieldName + "' not found."); } fieldDesc = "index field"; fieldType = field.getDataType(); } else if (exp instanceof SummaryExpression) { - SummaryField field = search.getSummaryField(fieldName); + SummaryField field = schema.getSummaryField(fieldName); if (field == null) { throw new VerificationException(exp, "Summary field '" + fieldName + "' not found."); } diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/processing/IndexingValues.java b/config-model/src/main/java/com/yahoo/searchdefinition/processing/IndexingValues.java index 36a74147f05..815e5a42df4 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/processing/IndexingValues.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/processing/IndexingValues.java @@ -4,7 +4,7 @@ package com.yahoo.searchdefinition.processing; import com.yahoo.config.application.api.DeployLogger; import com.yahoo.searchdefinition.RankProfileRegistry; import com.yahoo.document.Field; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.document.SDField; import com.yahoo.vespa.indexinglanguage.ExpressionConverter; import com.yahoo.vespa.indexinglanguage.expressions.Expression; @@ -17,15 +17,15 @@ import com.yahoo.vespa.model.container.search.QueryProfiles; */ public class IndexingValues extends Processor { - public IndexingValues(Search search, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { - super(search, deployLogger, rankProfileRegistry, queryProfiles); + public IndexingValues(Schema schema, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { + super(schema, deployLogger, rankProfileRegistry, queryProfiles); } @Override public void process(boolean validate, boolean documentsOnly) { if ( ! validate) return; - for (Field field : search.getDocument().fieldSet()) { + for (Field field : schema.getDocument().fieldSet()) { SDField sdField = (SDField)field; if ( ! sdField.isExtraField()) { new RequireThatDocumentFieldsAreImmutable(field).convert(sdField.getIndexingScript()); @@ -50,7 +50,7 @@ public class IndexingValues extends Processor { @Override protected boolean shouldConvert(Expression exp) { if (exp instanceof OutputExpression && mutatedBy != null) { - throw newProcessException(search, field, + throw newProcessException(schema, field, "Indexing expression '" + mutatedBy + "' attempts to modify the value of the " + "document field '" + field.getName() + "'. Use a field outside the document " + "block instead."); diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/processing/IntegerIndex2Attribute.java b/config-model/src/main/java/com/yahoo/searchdefinition/processing/IntegerIndex2Attribute.java index de108d4b711..ff6224d8d19 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/processing/IntegerIndex2Attribute.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/processing/IntegerIndex2Attribute.java @@ -6,7 +6,7 @@ import com.yahoo.searchdefinition.RankProfileRegistry; import com.yahoo.document.NumericDataType; import com.yahoo.searchdefinition.document.SDField; import com.yahoo.searchdefinition.Index; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.vespa.indexinglanguage.ExpressionConverter; import com.yahoo.vespa.indexinglanguage.ExpressionVisitor; import com.yahoo.vespa.indexinglanguage.expressions.AttributeExpression; @@ -25,13 +25,13 @@ import java.util.Set; */ public class IntegerIndex2Attribute extends Processor { - public IntegerIndex2Attribute(Search search, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { - super(search, deployLogger, rankProfileRegistry, queryProfiles); + public IntegerIndex2Attribute(Schema schema, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { + super(schema, deployLogger, rankProfileRegistry, queryProfiles); } @Override public void process(boolean validate, boolean documentsOnly) { - for (SDField field : search.allConcreteFields()) { + for (SDField field : schema.allConcreteFields()) { if (field.doesIndexing() && field.getDataType().getPrimitiveType() instanceof NumericDataType) { if (field.getIndex(field.getName()) != null && ! (field.getIndex(field.getName()).getType().equals(Index.Type.VESPA))) continue; @@ -39,9 +39,9 @@ public class IntegerIndex2Attribute extends Processor { Set<String> attributeNames = new HashSet<>(); new MyVisitor(attributeNames).visit(script); field.setIndexingScript((ScriptExpression)new MyConverter(attributeNames).convert(script)); - warn(search, field, "Changed to attribute because numerical indexes (field has type " + + warn(schema, field, "Changed to attribute because numerical indexes (field has type " + field.getDataType().getName() + ") is not currently supported." + - " Index-only settings may fail. Ignore this warning for streaming search."); + " Index-only settings may fail. Ignore this warning for streaming search."); } } } diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/processing/LiteralBoost.java b/config-model/src/main/java/com/yahoo/searchdefinition/processing/LiteralBoost.java index 51ef1c312a8..afe004fd5f3 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/processing/LiteralBoost.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/processing/LiteralBoost.java @@ -3,9 +3,9 @@ package com.yahoo.searchdefinition.processing; import com.yahoo.config.application.api.DeployLogger; import com.yahoo.searchdefinition.RankProfileRegistry; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.document.SDField; import com.yahoo.searchdefinition.RankProfile; -import com.yahoo.searchdefinition.Search; import com.yahoo.vespa.model.container.search.QueryProfiles; import java.util.Iterator; @@ -25,21 +25,21 @@ import java.util.Iterator; */ public class LiteralBoost extends Processor { - public LiteralBoost(Search search, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { - super(search, deployLogger, rankProfileRegistry, queryProfiles); + public LiteralBoost(Schema schema, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { + super(schema, deployLogger, rankProfileRegistry, queryProfiles); } /** Adds extra search fields and indices to express literal boosts */ @Override public void process(boolean validate, boolean documentsOnly) { - checkRankModifierRankType(search); - addLiteralBoostsToFields(search); - reduceFieldLiteralBoosts(search); + checkRankModifierRankType(schema); + addLiteralBoostsToFields(schema); + reduceFieldLiteralBoosts(schema); } /** Checks if literal boost is given using rank: , and set the actual literal boost accordingly. */ - private void checkRankModifierRankType(Search search) { - for (SDField field : search.allConcreteFields()) { + private void checkRankModifierRankType(Schema schema) { + for (SDField field : schema.allConcreteFields()) { if (field.getLiteralBoost() > -1) continue; // Let explicit value take precedence if (field.getRanking().isLiteral()) field.setLiteralBoost(100); @@ -51,26 +51,26 @@ public class LiteralBoost extends Processor { * This is required because boost indices will only be generated by looking * at field boosts */ - private void addLiteralBoostsToFields(Search search) { - Iterator i = matchingRankSettingsIterator(search, RankProfile.RankSetting.Type.LITERALBOOST); + private void addLiteralBoostsToFields(Schema schema) { + Iterator i = matchingRankSettingsIterator(schema, RankProfile.RankSetting.Type.LITERALBOOST); while (i.hasNext()) { RankProfile.RankSetting setting = (RankProfile.RankSetting)i.next(); - SDField field = search.getConcreteField(setting.getFieldName()); + SDField field = schema.getConcreteField(setting.getFieldName()); if (field == null) continue; if (field.getLiteralBoost() < 0) field.setLiteralBoost(0); } } - private void reduceFieldLiteralBoosts(Search search) { - for (SDField field : search.allConcreteFields()) { + private void reduceFieldLiteralBoosts(Schema schema) { + for (SDField field : schema.allConcreteFields()) { if (field.getLiteralBoost() < 0) continue; - reduceFieldLiteralBoost(field,search); + reduceFieldLiteralBoost(field, schema); } } - private void reduceFieldLiteralBoost(SDField field,Search search) { - SDField literalField = addField(search, field, "literal", + private void reduceFieldLiteralBoost(SDField field, Schema schema) { + SDField literalField = addField(schema, field, "literal", "{ input " + field.getName() + " | tokenize | index " + field.getName() + "_literal; }", "literal-boost"); literalField.setWeight(field.getWeight() + field.getLiteralBoost()); diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/processing/MakeAliases.java b/config-model/src/main/java/com/yahoo/searchdefinition/processing/MakeAliases.java index 36371dd9295..14d75c6438e 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/processing/MakeAliases.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/processing/MakeAliases.java @@ -3,10 +3,10 @@ package com.yahoo.searchdefinition.processing; import com.yahoo.config.application.api.DeployLogger; import com.yahoo.searchdefinition.RankProfileRegistry; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.document.Attribute; import com.yahoo.searchdefinition.document.SDField; import com.yahoo.searchdefinition.Index; -import com.yahoo.searchdefinition.Search; import com.yahoo.vespa.model.container.search.QueryProfiles; import java.util.ArrayList; @@ -20,22 +20,22 @@ import java.util.Map; */ public class MakeAliases extends Processor { - public MakeAliases(Search search, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { - super(search, deployLogger, rankProfileRegistry, queryProfiles); + public MakeAliases(Schema schema, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { + super(schema, deployLogger, rankProfileRegistry, queryProfiles); } @Override public void process(boolean validate, boolean documentsOnly) { List<String> usedAliases = new ArrayList<>(); - for (SDField field : search.allConcreteFields()) { + for (SDField field : schema.allConcreteFields()) { for (Map.Entry<String, String> e : field.getAliasToName().entrySet()) { String alias = e.getKey(); String name = e.getValue(); - String errMsg = "For search '" + search.getName() + "': alias '" + alias + "' "; - if (validate && search.existsIndex(alias)) { + String errMsg = "For " + schema + ": alias '" + alias + "' "; + if (validate && schema.existsIndex(alias)) { throw new IllegalArgumentException(errMsg + "is illegal since it is the name of an index."); } - if (validate && search.getAttribute(alias) != null) { + if (validate && schema.getAttribute(alias) != null) { throw new IllegalArgumentException(errMsg + "is illegal since it is the name of an attribute."); } if (validate && usedAliases.contains(alias)) { diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/processing/MakeDefaultSummaryTheSuperSet.java b/config-model/src/main/java/com/yahoo/searchdefinition/processing/MakeDefaultSummaryTheSuperSet.java index 3524f5666b3..a82c8d0c6be 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/processing/MakeDefaultSummaryTheSuperSet.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/processing/MakeDefaultSummaryTheSuperSet.java @@ -3,7 +3,7 @@ package com.yahoo.searchdefinition.processing; import com.yahoo.config.application.api.DeployLogger; import com.yahoo.searchdefinition.RankProfileRegistry; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.vespa.documentmodel.DocumentSummary; import com.yahoo.vespa.documentmodel.SummaryField; import com.yahoo.vespa.documentmodel.SummaryTransform; @@ -29,14 +29,14 @@ import com.yahoo.vespa.model.container.search.QueryProfiles; */ public class MakeDefaultSummaryTheSuperSet extends Processor { - public MakeDefaultSummaryTheSuperSet(Search search, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { - super(search, deployLogger, rankProfileRegistry, queryProfiles); + public MakeDefaultSummaryTheSuperSet(Schema schema, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { + super(schema, deployLogger, rankProfileRegistry, queryProfiles); } @Override public void process(boolean validate, boolean documentsOnly) { - DocumentSummary defaultSummary=search.getSummary("default"); - for (SummaryField summaryField : search.getUniqueNamedSummaryFields().values() ) { + DocumentSummary defaultSummary= schema.getSummary("default"); + for (SummaryField summaryField : schema.getUniqueNamedSummaryFields().values() ) { if (defaultSummary.getSummaryField(summaryField.getName()) != null) continue; if (summaryField.getTransform() == SummaryTransform.ATTRIBUTE) continue; if (summaryField.getTransform() == SummaryTransform.ATTRIBUTECOMBINER) continue; diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/processing/MatchConsistency.java b/config-model/src/main/java/com/yahoo/searchdefinition/processing/MatchConsistency.java index d9c62967649..c302ef63cfd 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/processing/MatchConsistency.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/processing/MatchConsistency.java @@ -3,10 +3,10 @@ package com.yahoo.searchdefinition.processing; import com.yahoo.config.application.api.DeployLogger; import com.yahoo.searchdefinition.RankProfileRegistry; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.document.Matching; import com.yahoo.searchdefinition.document.Matching.Type; import com.yahoo.searchdefinition.document.SDField; -import com.yahoo.searchdefinition.Search; import com.yahoo.vespa.indexinglanguage.ExpressionVisitor; import com.yahoo.vespa.indexinglanguage.expressions.Expression; import com.yahoo.vespa.indexinglanguage.expressions.IndexExpression; @@ -22,11 +22,11 @@ import java.util.Map; */ public class MatchConsistency extends Processor { - public MatchConsistency(Search search, + public MatchConsistency(Schema schema, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { - super(search, deployLogger, rankProfileRegistry, queryProfiles); + super(schema, deployLogger, rankProfileRegistry, queryProfiles); } @Override @@ -34,17 +34,17 @@ public class MatchConsistency extends Processor { if ( ! validate) return; Map<String, Matching.Type> types = new HashMap<>(); - for (SDField field : search.allConcreteFields()) { - new MyVisitor(search, field, types).visit(field.getIndexingScript()); + for (SDField field : schema.allConcreteFields()) { + new MyVisitor(schema, field, types).visit(field.getIndexingScript()); } } - private void checkMatching(Search search, SDField field, Map<String, Type> types, String indexTo) { + private void checkMatching(Schema schema, SDField field, Map<String, Type> types, String indexTo) { Type prevType = types.get(indexTo); if (prevType == null) { types.put(indexTo, field.getMatching().getType()); } else if ( ! field.getMatching().getType().equals(prevType)) { - warn(search, field, "The matching type for index '" + indexTo + "' (got " + field.getMatching().getType() + + warn(schema, field, "The matching type for index '" + indexTo + "' (got " + field.getMatching().getType() + ") is inconsistent with that given for the same index in a previous field (had " + prevType + ")."); } @@ -52,12 +52,12 @@ public class MatchConsistency extends Processor { private class MyVisitor extends ExpressionVisitor { - final Search search; + final Schema schema; final SDField field; final Map<String, Type> types; - MyVisitor(Search search, SDField field, Map<String, Type> types) { - this.search = search; + MyVisitor(Schema schema, SDField field, Map<String, Type> types) { + this.schema = schema; this.field = field; this.types = types; } @@ -65,7 +65,7 @@ public class MatchConsistency extends Processor { @Override protected void doVisit(Expression exp) { if (exp instanceof IndexExpression) { - checkMatching(search, field, types, ((IndexExpression)exp).getFieldName()); + checkMatching(schema, field, types, ((IndexExpression)exp).getFieldName()); } } } diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/processing/MatchPhaseSettingsValidator.java b/config-model/src/main/java/com/yahoo/searchdefinition/processing/MatchPhaseSettingsValidator.java index 8afc85f2e1f..b697c584ece 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/processing/MatchPhaseSettingsValidator.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/processing/MatchPhaseSettingsValidator.java @@ -4,7 +4,7 @@ package com.yahoo.searchdefinition.processing; import com.yahoo.config.application.api.DeployLogger; import com.yahoo.searchdefinition.RankProfile; import com.yahoo.searchdefinition.RankProfileRegistry; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.document.Attribute; import com.yahoo.vespa.model.container.search.QueryProfiles; @@ -15,8 +15,8 @@ import com.yahoo.vespa.model.container.search.QueryProfiles; */ public class MatchPhaseSettingsValidator extends Processor { - public MatchPhaseSettingsValidator(Search search, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { - super(search, deployLogger, rankProfileRegistry, queryProfiles); + public MatchPhaseSettingsValidator(Schema schema, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { + super(schema, deployLogger, rankProfileRegistry, queryProfiles); } @Override @@ -24,7 +24,7 @@ public class MatchPhaseSettingsValidator extends Processor { if ( ! validate) return; if (documentsOnly) return; - for (RankProfile rankProfile : rankProfileRegistry.rankProfilesOf(search)) { + for (RankProfile rankProfile : rankProfileRegistry.rankProfilesOf(schema)) { RankProfile.MatchPhaseSettings settings = rankProfile.getMatchPhaseSettings(); if (settings != null) { validateMatchPhaseSettings(rankProfile, settings); @@ -34,9 +34,9 @@ public class MatchPhaseSettingsValidator extends Processor { private void validateMatchPhaseSettings(RankProfile rankProfile, RankProfile.MatchPhaseSettings settings) { String attributeName = settings.getAttribute(); - new AttributeValidator(search.getName(), + new AttributeValidator(schema.getName(), rankProfile.getName(), - search.getAttribute(attributeName), attributeName).validate(); + schema.getAttribute(attributeName), attributeName).validate(); } public static class AttributeValidator { diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/processing/MatchedElementsOnlyResolver.java b/config-model/src/main/java/com/yahoo/searchdefinition/processing/MatchedElementsOnlyResolver.java index 77064038053..709c79cd79b 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/processing/MatchedElementsOnlyResolver.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/processing/MatchedElementsOnlyResolver.java @@ -6,7 +6,7 @@ import com.yahoo.document.ArrayDataType; import com.yahoo.document.DataType; import com.yahoo.document.WeightedSetDataType; import com.yahoo.searchdefinition.RankProfileRegistry; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.document.ComplexAttributeFieldUtils; import com.yahoo.searchdefinition.document.ImmutableSDField; import com.yahoo.vespa.documentmodel.DocumentSummary; @@ -25,13 +25,13 @@ import static com.yahoo.searchdefinition.document.ComplexAttributeFieldUtils.isS */ public class MatchedElementsOnlyResolver extends Processor { - public MatchedElementsOnlyResolver(Search search, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { - super(search, deployLogger, rankProfileRegistry, queryProfiles); + public MatchedElementsOnlyResolver(Schema schema, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { + super(schema, deployLogger, rankProfileRegistry, queryProfiles); } @Override public void process(boolean validate, boolean documentsOnly) { - for (var entry : search.getSummaries().entrySet()) { + for (var entry : schema.getSummaries().entrySet()) { var summary = entry.getValue(); for (var field : summary.getSummaryFields()) { if (field.getTransform().equals(SummaryTransform.MATCHED_ELEMENTS_FILTER)) { @@ -42,7 +42,7 @@ public class MatchedElementsOnlyResolver extends Processor { } private void processSummaryField(DocumentSummary summary, SummaryField field, boolean validate) { - var sourceField = search.getField(field.getSingleSource()); + var sourceField = schema.getField(field.getSingleSource()); if (sourceField != null) { if (isSupportedComplexField(sourceField)) { if (isComplexFieldWithOnlyStructFieldAttributes(sourceField)) { @@ -84,12 +84,12 @@ public class MatchedElementsOnlyResolver extends Processor { } private void fail(DocumentSummary summary, SummaryField field, String msg) { - throw new IllegalArgumentException(formatError(search, summary, field, msg)); + throw new IllegalArgumentException(formatError(schema, summary, field, msg)); } - private String formatError(Search search, DocumentSummary summary, SummaryField field, String msg) { - return "For search '" + search.getName() + "', document summary '" + summary.getName() - + "', summary field '" + field.getName() + "': " + msg; + private String formatError(Schema schema, DocumentSummary summary, SummaryField field, String msg) { + return "For " + schema + ", document summary '" + summary.getName() + + "', summary field '" + field.getName() + "': " + msg; } } diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/processing/MultifieldIndexHarmonizer.java b/config-model/src/main/java/com/yahoo/searchdefinition/processing/MultifieldIndexHarmonizer.java index 265715e8eb9..39220b8cf8f 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/processing/MultifieldIndexHarmonizer.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/processing/MultifieldIndexHarmonizer.java @@ -3,8 +3,8 @@ package com.yahoo.searchdefinition.processing; import com.yahoo.config.application.api.DeployLogger; import com.yahoo.searchdefinition.RankProfileRegistry; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.document.SDField; -import com.yahoo.searchdefinition.Search; import com.yahoo.searchdefinition.processing.multifieldresolver.IndexCommandResolver; import com.yahoo.searchdefinition.processing.multifieldresolver.RankTypeResolver; import com.yahoo.searchdefinition.processing.multifieldresolver.StemmingResolver; @@ -25,18 +25,18 @@ public class MultifieldIndexHarmonizer extends Processor { /** A map from index names to a List of fields going to that index */ private Map<String,List<SDField>> indexToFields=new java.util.HashMap<>(); - public MultifieldIndexHarmonizer(Search search, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { - super(search, deployLogger, rankProfileRegistry, queryProfiles); + public MultifieldIndexHarmonizer(Schema schema, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { + super(schema, deployLogger, rankProfileRegistry, queryProfiles); } @Override public void process(boolean validate, boolean documentsOnly) { - populateIndexToFields(search); - resolveAllConflicts(search); + populateIndexToFields(schema); + resolveAllConflicts(schema); } - private void populateIndexToFields(Search search) { - for (SDField field : search.allConcreteFields() ) { + private void populateIndexToFields(Schema schema) { + for (SDField field : schema.allConcreteFields() ) { if ( ! field.doesIndexing()) continue; addIndexField(field.getName(), field); } @@ -51,12 +51,12 @@ public class MultifieldIndexHarmonizer extends Processor { fields.add(field); } - private void resolveAllConflicts(Search search) { + private void resolveAllConflicts(Schema schema) { for (Map.Entry<String, List<SDField>> entry : indexToFields.entrySet()) { String indexName = entry.getKey(); List<SDField> fields = entry.getValue(); if (fields.size() == 1) continue; // It takes two to make a conflict - resolveConflicts(indexName, fields, search); + resolveConflicts(indexName, fields, schema); } } @@ -65,12 +65,12 @@ public class MultifieldIndexHarmonizer extends Processor { * * @param indexName the name of the index in question * @param fields all the fields indexed to this index - * @param search the search definition having this + * @param schema the search definition having this */ - private void resolveConflicts(String indexName,List<SDField> fields,Search search) { - new StemmingResolver(indexName, fields, search, deployLogger).resolve(); - new IndexCommandResolver(indexName, fields, search, deployLogger).resolve(); - new RankTypeResolver(indexName, fields, search, deployLogger).resolve(); + private void resolveConflicts(String indexName, List<SDField> fields, Schema schema) { + new StemmingResolver(indexName, fields, schema, deployLogger).resolve(); + new IndexCommandResolver(indexName, fields, schema, deployLogger).resolve(); + new RankTypeResolver(indexName, fields, schema, deployLogger).resolve(); } } diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/processing/MutableAttributes.java b/config-model/src/main/java/com/yahoo/searchdefinition/processing/MutableAttributes.java index 35b17234a1d..198f89caddb 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/processing/MutableAttributes.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/processing/MutableAttributes.java @@ -3,24 +3,24 @@ package com.yahoo.searchdefinition.processing; import com.yahoo.config.application.api.DeployLogger; import com.yahoo.searchdefinition.RankProfileRegistry; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.document.SDField; import com.yahoo.vespa.model.container.search.QueryProfiles; public class MutableAttributes extends Processor { - public MutableAttributes(Search search, DeployLogger deployLogger, + public MutableAttributes(Schema schema, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { - super(search, deployLogger, rankProfileRegistry, queryProfiles); + super(schema, deployLogger, rankProfileRegistry, queryProfiles); } @Override public void process(boolean validate, boolean documentsOnly) { - for (SDField field : search.allConcreteFields()) { + for (SDField field : schema.allConcreteFields()) { if ( ! field.isExtraField() && field.getAttributes().containsKey(field.getName())) { if (field.getAttributes().get(field.getName()).isMutable()) { - throw new IllegalArgumentException("Field '" + field.getName() + "' in '" + search.getDocument().getName() + + throw new IllegalArgumentException("Field '" + field.getName() + "' in '" + schema.getDocument().getName() + "' can not be marked mutable as it is inside the document clause."); } } diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/processing/NGramMatch.java b/config-model/src/main/java/com/yahoo/searchdefinition/processing/NGramMatch.java index 0995762da6e..b5f573ecf71 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/processing/NGramMatch.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/processing/NGramMatch.java @@ -5,7 +5,7 @@ import com.yahoo.config.application.api.DeployLogger; import com.yahoo.document.CollectionDataType; import com.yahoo.document.DataType; import com.yahoo.searchdefinition.RankProfileRegistry; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.document.Matching; import com.yahoo.searchdefinition.document.SDField; import com.yahoo.searchdefinition.document.Stemming; @@ -22,21 +22,21 @@ public class NGramMatch extends Processor { public static final int DEFAULT_GRAM_SIZE = 2; - public NGramMatch(Search search, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { - super(search, deployLogger, rankProfileRegistry, queryProfiles); + public NGramMatch(Schema schema, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { + super(schema, deployLogger, rankProfileRegistry, queryProfiles); } @Override public void process(boolean validate, boolean documentsOnly) { - for (SDField field : search.allConcreteFields()) { + for (SDField field : schema.allConcreteFields()) { if (field.getMatching().getType().equals(Matching.Type.GRAM)) - implementGramMatch(search, field, validate); + implementGramMatch(schema, field, validate); else if (validate && field.getMatching().getGramSize() >= 0) throw new IllegalArgumentException("gram-size can only be set when the matching mode is 'gram'"); } } - private void implementGramMatch(Search search, SDField field, boolean validate) { + private void implementGramMatch(Schema schema, SDField field, boolean validate) { if (validate && field.doesAttributing() && ! field.doesIndexing()) throw new IllegalArgumentException("gram matching is not supported with attributes, use 'index' in indexing"); @@ -48,15 +48,15 @@ public class NGramMatch extends Processor { field.getNormalizing().inferCodepoint(); field.setStemming(Stemming.NONE); // not compatible with stemming and normalizing field.addQueryCommand("ngram " + n); - field.setIndexingScript((ScriptExpression)new MyProvider(search, n).convert(field.getIndexingScript())); + field.setIndexingScript((ScriptExpression)new MyProvider(schema, n).convert(field.getIndexingScript())); } private static class MyProvider extends TypedTransformProvider { final int ngram; - MyProvider(Search search, int ngram) { - super(NGramExpression.class, search); + MyProvider(Schema schema, int ngram) { + super(NGramExpression.class, schema); this.ngram = ngram; } diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/processing/OnnxModelConfigGenerator.java b/config-model/src/main/java/com/yahoo/searchdefinition/processing/OnnxModelConfigGenerator.java index a2eae585a25..c7c1db3b862 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/processing/OnnxModelConfigGenerator.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/processing/OnnxModelConfigGenerator.java @@ -7,7 +7,7 @@ import com.yahoo.config.application.api.DeployLogger; import com.yahoo.searchdefinition.OnnxModel; import com.yahoo.searchdefinition.RankProfile; import com.yahoo.searchdefinition.RankProfileRegistry; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.expressiontransforms.OnnxModelTransformer; import com.yahoo.searchlib.rankingexpression.rule.CompositeNode; import com.yahoo.searchlib.rankingexpression.rule.ConstantNode; @@ -36,14 +36,14 @@ import java.util.Map; */ public class OnnxModelConfigGenerator extends Processor { - public OnnxModelConfigGenerator(Search search, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { - super(search, deployLogger, rankProfileRegistry, queryProfiles); + public OnnxModelConfigGenerator(Schema schema, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { + super(schema, deployLogger, rankProfileRegistry, queryProfiles); } @Override public void process(boolean validate, boolean documentsOnly) { if (documentsOnly) return; - for (RankProfile profile : rankProfileRegistry.rankProfilesOf(search)) { + for (RankProfile profile : rankProfileRegistry.rankProfilesOf(schema)) { if (profile.getFirstPhaseRanking() != null) { process(profile.getFirstPhaseRanking().getRoot()); } @@ -78,17 +78,17 @@ public class OnnxModelConfigGenerator extends Processor { String modelConfigName = OnnxModelTransformer.asValidIdentifier(path); // Only add the configuration if the model can actually be found. - if ( ! OnnxModelInfo.modelExists(path, search.applicationPackage())) { + if ( ! OnnxModelInfo.modelExists(path, schema.applicationPackage())) { path = ApplicationPackage.MODELS_DIR.append(path).toString(); - if ( ! OnnxModelInfo.modelExists(path, search.applicationPackage())) { + if ( ! OnnxModelInfo.modelExists(path, schema.applicationPackage())) { return; } } - OnnxModel onnxModel = search.onnxModels().get(modelConfigName); + OnnxModel onnxModel = schema.onnxModels().get(modelConfigName); if (onnxModel == null) { onnxModel = new OnnxModel(modelConfigName, path); - search.onnxModels().add(onnxModel); + schema.onnxModels().add(onnxModel); } } } diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/processing/OnnxModelTypeResolver.java b/config-model/src/main/java/com/yahoo/searchdefinition/processing/OnnxModelTypeResolver.java index aa3c5db4e83..4153cca4b5b 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/processing/OnnxModelTypeResolver.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/processing/OnnxModelTypeResolver.java @@ -5,7 +5,7 @@ package com.yahoo.searchdefinition.processing; import com.yahoo.config.application.api.DeployLogger; import com.yahoo.searchdefinition.OnnxModel; import com.yahoo.searchdefinition.RankProfileRegistry; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.vespa.model.container.search.QueryProfiles; import com.yahoo.vespa.model.ml.OnnxModelInfo; @@ -21,15 +21,15 @@ import com.yahoo.vespa.model.ml.OnnxModelInfo; */ public class OnnxModelTypeResolver extends Processor { - public OnnxModelTypeResolver(Search search, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { - super(search, deployLogger, rankProfileRegistry, queryProfiles); + public OnnxModelTypeResolver(Schema schema, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { + super(schema, deployLogger, rankProfileRegistry, queryProfiles); } @Override public void process(boolean validate, boolean documentsOnly) { if (documentsOnly) return; - for (OnnxModel onnxModel : search.onnxModels().asMap().values()) { - OnnxModelInfo onnxModelInfo = OnnxModelInfo.load(onnxModel.getFileName(), search.applicationPackage()); + for (OnnxModel onnxModel : schema.onnxModels().asMap().values()) { + OnnxModelInfo onnxModelInfo = OnnxModelInfo.load(onnxModel.getFileName(), schema.applicationPackage()); onnxModel.setModelInfo(onnxModelInfo); } } diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/processing/OptimizeIlscript.java b/config-model/src/main/java/com/yahoo/searchdefinition/processing/OptimizeIlscript.java index 82bcff265c0..3175862e900 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/processing/OptimizeIlscript.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/processing/OptimizeIlscript.java @@ -3,7 +3,7 @@ package com.yahoo.searchdefinition.processing; import com.yahoo.config.application.api.DeployLogger; import com.yahoo.searchdefinition.RankProfileRegistry; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.document.SDField; import com.yahoo.vespa.indexinglanguage.ExpressionOptimizer; import com.yahoo.vespa.indexinglanguage.expressions.ScriptExpression; @@ -14,22 +14,22 @@ import com.yahoo.vespa.model.container.search.QueryProfiles; */ public class OptimizeIlscript extends Processor { - public OptimizeIlscript(Search search, + public OptimizeIlscript(Schema schema, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { - super(search, deployLogger, rankProfileRegistry, queryProfiles); + super(schema, deployLogger, rankProfileRegistry, queryProfiles); } @Override public void process(boolean validate, boolean documentsOnly) { - for (SDField field : search.allConcreteFields()) { + for (SDField field : schema.allConcreteFields()) { ScriptExpression script = field.getIndexingScript(); if (script == null) continue; field.setIndexingScript((ScriptExpression)new ExpressionOptimizer().convert(script)); if ( ! field.getIndexingScript().toString().equals(script.toString())) { - info(search, field, "Rewrote ilscript from:\n" + script.toString() + + info(schema, field, "Rewrote ilscript from:\n" + script.toString() + "\nto\n" + field.getIndexingScript().toString()); } } diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/processing/PagedAttributeValidator.java b/config-model/src/main/java/com/yahoo/searchdefinition/processing/PagedAttributeValidator.java index 5e344f5a824..d108a620fd9 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/processing/PagedAttributeValidator.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/processing/PagedAttributeValidator.java @@ -4,7 +4,7 @@ package com.yahoo.searchdefinition.processing; import com.yahoo.config.application.api.DeployLogger; import com.yahoo.document.Field; import com.yahoo.searchdefinition.RankProfileRegistry; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.document.Attribute; import com.yahoo.tensor.TensorType; import com.yahoo.vespa.model.container.search.QueryProfiles; @@ -17,11 +17,11 @@ import com.yahoo.vespa.model.container.search.QueryProfiles; public class PagedAttributeValidator extends Processor { - public PagedAttributeValidator(Search search, + public PagedAttributeValidator(Schema schema, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { - super(search, deployLogger, rankProfileRegistry, queryProfiles); + super(schema, deployLogger, rankProfileRegistry, queryProfiles); } @Override @@ -29,7 +29,7 @@ public class PagedAttributeValidator extends Processor { if (!validate) { return; } - for (var field : search.allConcreteFields()) { + for (var field : schema.allConcreteFields()) { for (var attribute : field.getAttributes().values()) { if (attribute.isPaged()) { validatePagedSetting(field, attribute); @@ -42,7 +42,7 @@ public class PagedAttributeValidator extends Processor { var tensorType = attribute.tensorType(); if (!tensorType.isPresent() || !isDenseTensorType(tensorType.get())) { - fail(search, field, "The 'paged' attribute setting is only supported for dense tensor types"); + fail(schema, field, "The 'paged' attribute setting is only supported for dense tensor types"); } } diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/processing/PredicateProcessor.java b/config-model/src/main/java/com/yahoo/searchdefinition/processing/PredicateProcessor.java index df357d6b704..73b75d6e23a 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/processing/PredicateProcessor.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/processing/PredicateProcessor.java @@ -7,7 +7,7 @@ import com.yahoo.document.DataType; import com.yahoo.document.datatypes.IntegerFieldValue; import com.yahoo.document.datatypes.LongFieldValue; import com.yahoo.searchdefinition.Index; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.document.Attribute; import com.yahoo.searchdefinition.document.BooleanIndexDefinition; import com.yahoo.searchdefinition.document.SDField; @@ -34,30 +34,30 @@ import java.util.List; */ public class PredicateProcessor extends Processor { - public PredicateProcessor(Search search, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { - super(search, deployLogger, rankProfileRegistry, queryProfiles); + public PredicateProcessor(Schema schema, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { + super(schema, deployLogger, rankProfileRegistry, queryProfiles); } @Override public void process(boolean validate, boolean documentsOnly) { - for (SDField field : search.allConcreteFields()) { + for (SDField field : schema.allConcreteFields()) { if (field.getDataType() == DataType.PREDICATE) { if (validate && field.doesIndexing()) { - fail(search, field, "Use 'attribute' instead of 'index'. This will require a refeed if you have upgraded."); + fail(schema, field, "Use 'attribute' instead of 'index'. This will require a refeed if you have upgraded."); } if (field.doesAttributing()) { Attribute attribute = field.getAttributes().get(field.getName()); for (Index index : field.getIndices().values()) { BooleanIndexDefinition booleanDefinition = index.getBooleanIndexDefiniton(); if (validate && (booleanDefinition == null || ! booleanDefinition.hasArity())) { - fail(search, field, "Missing arity value in predicate field."); + fail(schema, field, "Missing arity value in predicate field."); } if (validate && (booleanDefinition.getArity() < 2)) { - fail(search, field, "Invalid arity value in predicate field, must be greater than 1."); + fail(schema, field, "Invalid arity value in predicate field, must be greater than 1."); } double threshold = booleanDefinition.getDensePostingListThreshold(); if (validate && (threshold <= 0 || threshold > 1)) { - fail(search, field, "Invalid dense-posting-list-threshold value in predicate field. " + + fail(schema, field, "Invalid dense-posting-list-threshold value in predicate field. " + "Value must be in range (0..1]."); } @@ -68,29 +68,29 @@ public class PredicateProcessor extends Processor { attribute.setDensePostingListThreshold(threshold); addPredicateOptimizationIlScript(field, booleanDefinition); } - DocumentSummary summary = search.getSummary("attributeprefetch"); + DocumentSummary summary = schema.getSummary("attributeprefetch"); if (summary != null) { summary.remove(attribute.getName()); } - for (SummaryField summaryField : search.getSummaryFields(field).values()) { + for (SummaryField summaryField : schema.getSummaryFields(field).values()) { summaryField.setTransform(SummaryTransform.NONE); } } } else if (validate && field.getDataType().getPrimitiveType() == DataType.PREDICATE) { - fail(search, field, "Collections of predicates are not allowed."); + fail(schema, field, "Collections of predicates are not allowed."); } else if (validate && field.getDataType() == DataType.RAW && field.doesIndexing()) { - fail(search, field, "Indexing of RAW fields is not supported."); + fail(schema, field, "Indexing of RAW fields is not supported."); } else if (validate) { // if field is not a predicate, disallow predicate-related index parameters for (Index index : field.getIndices().values()) { if (index.getBooleanIndexDefiniton() != null) { BooleanIndexDefinition def = index.getBooleanIndexDefiniton(); if (def.hasArity()) { - fail(search, field, "Arity parameter is used only for predicate type fields."); + fail(schema, field, "Arity parameter is used only for predicate type fields."); } else if (def.hasLowerBound() || def.hasUpperBound()) { - fail(search, field, "Parameters lower-bound and upper-bound are used only for predicate type fields."); + fail(schema, field, "Parameters lower-bound and upper-bound are used only for predicate type fields."); } else if (def.hasDensePostingListThreshold()) { - fail(search, field, "Parameter dense-posting-list-threshold is used only for predicate type fields."); + fail(schema, field, "Parameter dense-posting-list-threshold is used only for predicate type fields."); } } } @@ -104,7 +104,7 @@ public class PredicateProcessor extends Processor { script = new StatementExpression(makeSetPredicateVariablesScript(booleanIndexDefiniton), script); - ExpressionConverter converter = new PredicateOutputTransformer(search); + ExpressionConverter converter = new PredicateOutputTransformer(schema); field.setIndexingScript(new ScriptExpression((StatementExpression)converter.convert(script))); } @@ -125,8 +125,8 @@ public class PredicateProcessor extends Processor { private static class PredicateOutputTransformer extends TypedTransformProvider { - PredicateOutputTransformer(Search search) { - super(OptimizePredicateExpression.class, search); + PredicateOutputTransformer(Schema schema) { + super(OptimizePredicateExpression.class, schema); } @Override diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/processing/Processing.java b/config-model/src/main/java/com/yahoo/searchdefinition/processing/Processing.java index ddb158a3d84..a484476e978 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/processing/Processing.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/processing/Processing.java @@ -3,12 +3,13 @@ package com.yahoo.searchdefinition.processing; import com.yahoo.config.application.api.DeployLogger; import com.yahoo.searchdefinition.RankProfileRegistry; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.processing.multifieldresolver.RankProfileTypeSettingsProcessor; import com.yahoo.vespa.model.container.search.QueryProfiles; import java.util.Arrays; import java.util.Collection; +import java.util.Set; /** * Executor of processors. This defines the right order of processor execution. @@ -94,31 +95,34 @@ public class Processing { } /** - * Runs all search processors on the given {@link Search} object. These will modify the search object, <b>possibly + * Runs all search processors on the given {@link Schema} object. These will modify the search object, <b>possibly * exchanging it with another</b>, as well as its document types. * - * @param search The search to process. - * @param deployLogger The log to log messages and warnings for application deployment to + * @param schema the search to process + * @param deployLogger the log to log messages and warnings for application deployment to * @param rankProfileRegistry a {@link com.yahoo.searchdefinition.RankProfileRegistry} - * @param queryProfiles The query profiles contained in the application this search is part of. + * @param queryProfiles the query profiles contained in the application this search is part of + * @param processorsToSkip a set of processor classes we should not invoke in this. Useful for testing. */ - public void process(Search search, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, - QueryProfiles queryProfiles, boolean validate, boolean documentsOnly) { + public void process(Schema schema, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, + QueryProfiles queryProfiles, boolean validate, boolean documentsOnly, + Set<Class<? extends Processor>> processorsToSkip) { Collection<ProcessorFactory> factories = processors(); factories.stream() - .map(factory -> factory.create(search, deployLogger, rankProfileRegistry, queryProfiles)) + .map(factory -> factory.create(schema, deployLogger, rankProfileRegistry, queryProfiles)) + .filter(processor -> ! processorsToSkip.contains(processor.getClass())) .forEach(processor -> processor.process(validate, documentsOnly)); } /** * Runs rank profiles processors only. * - * @param deployLogger The log to log messages and warnings for application deployment to + * @param deployLogger the log to log messages and warnings for application deployment to * @param rankProfileRegistry a {@link com.yahoo.searchdefinition.RankProfileRegistry} - * @param queryProfiles The query profiles contained in the application this search is part of. + * @param queryProfiles the query profiles contained in the application this search is part of */ public void processRankProfiles(DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, - QueryProfiles queryProfiles, boolean validate, boolean documentsOnly) { + QueryProfiles queryProfiles, boolean validate, boolean documentsOnly) { Collection<ProcessorFactory> factories = rankProfileProcessors(); factories.stream() .map(factory -> factory.create(null, deployLogger, rankProfileRegistry, queryProfiles)) @@ -127,7 +131,8 @@ public class Processing { @FunctionalInterface public interface ProcessorFactory { - Processor create(Search search, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles); + Processor create(Schema schema, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, + QueryProfiles queryProfiles); } } diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/processing/Processor.java b/config-model/src/main/java/com/yahoo/searchdefinition/processing/Processor.java index c4f15367831..e8f60335362 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/processing/Processor.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/processing/Processor.java @@ -7,7 +7,7 @@ import com.yahoo.document.Field; import com.yahoo.searchdefinition.Index; import com.yahoo.searchdefinition.RankProfile; import com.yahoo.searchdefinition.RankProfileRegistry; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.document.RankType; import com.yahoo.searchdefinition.document.SDField; import com.yahoo.searchdefinition.document.Stemming; @@ -24,7 +24,7 @@ import java.util.logging.Level; */ public abstract class Processor { - protected final Search search; + protected final Schema schema; protected final DeployLogger deployLogger; protected final RankProfileRegistry rankProfileRegistry; protected final QueryProfiles queryProfiles; @@ -32,16 +32,16 @@ public abstract class Processor { /** * Base constructor * - * @param search the search to process + * @param schema the search to process * @param deployLogger Logger du use when logging deploy output. * @param rankProfileRegistry Registry with all rank profiles, used for lookup and insertion. * @param queryProfiles The query profiles contained in the application this search is part of. */ - public Processor(Search search, + public Processor(Schema schema, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { - this.search = search; + this.schema = schema; this.deployLogger = deployLogger; this.rankProfileRegistry = rankProfileRegistry; this.queryProfiles = queryProfiles; @@ -62,19 +62,19 @@ public abstract class Processor { /** * Convenience method for adding a no-strings-attached implementation field for a regular field * - * @param search the search definition in question + * @param schema the search definition in question * @param field the field to add an implementation field for * @param suffix the suffix of the added implementation field (without the underscore) * @param indexing the indexing statement of the field * @param queryCommand the query command of the original field, or null if none * @return the implementation field which is added to the search */ - protected SDField addField(Search search, SDField field, String suffix, String indexing, String queryCommand) { - SDField implementationField = search.getConcreteField(field.getName() + "_" + suffix); + protected SDField addField(Schema schema, SDField field, String suffix, String indexing, String queryCommand) { + SDField implementationField = schema.getConcreteField(field.getName() + "_" + suffix); if (implementationField != null) { deployLogger.logApplicationPackage(Level.WARNING, "Implementation field " + implementationField + " added twice"); } else { - implementationField = new SDField(search.getDocument(), field.getName() + "_" + suffix, DataType.STRING); + implementationField = new SDField(schema.getDocument(), field.getName() + "_" + suffix, DataType.STRING); } implementationField.setRankType(RankType.EMPTY); implementationField.setStemming(Stemming.NONE); @@ -83,12 +83,12 @@ public abstract class Processor { String indexName = field.getName(); String implementationIndexName = indexName + "_" + suffix; Index implementationIndex = new Index(implementationIndexName); - search.addIndex(implementationIndex); + schema.addIndex(implementationIndex); if (queryCommand != null) { field.addQueryCommand(queryCommand); } - search.addExtraField(implementationField); - search.fieldSets().addBuiltInFieldSetItem(BuiltInFieldSets.INTERNAL_FIELDSET_NAME, implementationField.getName()); + schema.addExtraField(implementationField); + schema.fieldSets().addBuiltInFieldSetItem(BuiltInFieldSets.INTERNAL_FIELDSET_NAME, implementationField.getName()); return implementationField; } @@ -97,11 +97,11 @@ public abstract class Processor { * definition. */ protected Iterator<RankProfile.RankSetting> matchingRankSettingsIterator( - Search search, RankProfile.RankSetting.Type type) + Schema schema, RankProfile.RankSetting.Type type) { List<RankProfile.RankSetting> someRankSettings = new java.util.ArrayList<>(); - for (RankProfile profile : rankProfileRegistry.rankProfilesOf(search)) { + for (RankProfile profile : rankProfileRegistry.rankProfilesOf(schema)) { for (Iterator j = profile.declaredRankSettingIterator(); j.hasNext(); ) { RankProfile.RankSetting setting = (RankProfile.RankSetting)j.next(); if (setting.getType().equals(type)) { @@ -112,38 +112,38 @@ public abstract class Processor { return someRankSettings.iterator(); } - protected String formatError(String searchName, String fieldName, String msg) { - return "For search '" + searchName + "', field '" + fieldName + "': " + msg; + protected String formatError(String schemaName, String fieldName, String msg) { + return "For schema '" + schemaName + "', field '" + fieldName + "': " + msg; } - protected RuntimeException newProcessException(String searchName, String fieldName, String msg) { - return new IllegalArgumentException(formatError(searchName, fieldName, msg)); + protected RuntimeException newProcessException(String schemaName, String fieldName, String msg) { + return new IllegalArgumentException(formatError(schemaName, fieldName, msg)); } - protected RuntimeException newProcessException(Search search, Field field, String msg) { - return newProcessException(search.getName(), field.getName(), msg); + protected RuntimeException newProcessException(Schema schema, Field field, String msg) { + return newProcessException(schema.getName(), field.getName(), msg); } - public void fail(Search search, Field field, String msg) { - throw newProcessException(search, field, msg); + public void fail(Schema schema, Field field, String msg) { + throw newProcessException(schema, field, msg); } - protected void warn(String searchName, String fieldName, String message) { - String fullMsg = formatError(searchName, fieldName, message); + protected void warn(String schemaName, String fieldName, String message) { + String fullMsg = formatError(schemaName, fieldName, message); deployLogger.logApplicationPackage(Level.WARNING, fullMsg); } - protected void warn(Search search, Field field, String message) { - warn(search.getName(), field.getName(), message); + protected void warn(Schema schema, Field field, String message) { + warn(schema.getName(), field.getName(), message); } - protected void info(String searchName, String fieldName, String message) { - String fullMsg = formatError(searchName, fieldName, message); + protected void info(String schemaName, String fieldName, String message) { + String fullMsg = formatError(schemaName, fieldName, message); deployLogger.logApplicationPackage(Level.INFO, fullMsg); } - protected void info(Search search, Field field, String message) { - info(search.getName(), field.getName(), message); + protected void info(Schema schema, Field field, String message) { + info(schema.getName(), field.getName(), message); } } diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/processing/RankingExpressionTypeResolver.java b/config-model/src/main/java/com/yahoo/searchdefinition/processing/RankingExpressionTypeResolver.java index e6b58cb6c12..f5c3fa35e34 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/processing/RankingExpressionTypeResolver.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/processing/RankingExpressionTypeResolver.java @@ -6,7 +6,7 @@ import com.yahoo.search.query.profile.QueryProfileRegistry; import com.yahoo.searchdefinition.MapEvaluationTypeContext; import com.yahoo.searchdefinition.RankProfile; import com.yahoo.searchdefinition.RankProfileRegistry; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchlib.rankingexpression.ExpressionFunction; import com.yahoo.searchlib.rankingexpression.RankingExpression; import com.yahoo.searchlib.rankingexpression.Reference; @@ -15,7 +15,6 @@ import com.yahoo.tensor.TensorType; import com.yahoo.tensor.evaluation.TypeContext; import com.yahoo.vespa.model.container.search.QueryProfiles; -import java.util.HashMap; import java.util.HashSet; import java.util.Map; import java.util.Set; @@ -36,11 +35,11 @@ public class RankingExpressionTypeResolver extends Processor { private final QueryProfileRegistry queryProfiles; - public RankingExpressionTypeResolver(Search search, + public RankingExpressionTypeResolver(Schema schema, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { - super(search, deployLogger, rankProfileRegistry, queryProfiles); + super(schema, deployLogger, rankProfileRegistry, queryProfiles); this.queryProfiles = queryProfiles.getRegistry(); } @@ -49,12 +48,12 @@ public class RankingExpressionTypeResolver extends Processor { if (documentsOnly) return; Set<Reference> warnedAbout = new HashSet<>(); - for (RankProfile profile : rankProfileRegistry.rankProfilesOf(search)) { + for (RankProfile profile : rankProfileRegistry.rankProfilesOf(schema)) { try { resolveTypesIn(profile, validate, warnedAbout); } catch (IllegalArgumentException e) { - throw new IllegalArgumentException("In " + (search != null ? search + ", " : "") + profile, e); + throw new IllegalArgumentException("In " + (schema != null ? schema + ", " : "") + profile, e); } } } diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/processing/ReferenceFieldsProcessor.java b/config-model/src/main/java/com/yahoo/searchdefinition/processing/ReferenceFieldsProcessor.java index 698e654f1cb..ed23b1ca606 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/processing/ReferenceFieldsProcessor.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/processing/ReferenceFieldsProcessor.java @@ -4,7 +4,7 @@ package com.yahoo.searchdefinition.processing; import com.yahoo.config.application.api.DeployLogger; import com.yahoo.document.ReferenceDataType; import com.yahoo.searchdefinition.RankProfileRegistry; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.document.SDField; import com.yahoo.vespa.documentmodel.DocumentSummary; import com.yahoo.vespa.documentmodel.SummaryTransform; @@ -20,11 +20,11 @@ import com.yahoo.vespa.model.container.search.QueryProfiles; */ public class ReferenceFieldsProcessor extends Processor { - public ReferenceFieldsProcessor(Search search, + public ReferenceFieldsProcessor(Schema schema, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { - super(search, deployLogger, rankProfileRegistry, queryProfiles); + super(schema, deployLogger, rankProfileRegistry, queryProfiles); } @Override @@ -34,7 +34,7 @@ public class ReferenceFieldsProcessor extends Processor { } private void clearSummaryAttributeAspectForExplicitSummaryFields() { - for (DocumentSummary docSum : search.getSummaries().values()) { + for (DocumentSummary docSum : schema.getSummaries().values()) { docSum.getSummaryFields().stream() .filter(summaryField -> summaryField.getDataType() instanceof ReferenceDataType) .forEach(summaryField -> summaryField.setTransform(SummaryTransform.NONE)); @@ -42,7 +42,7 @@ public class ReferenceFieldsProcessor extends Processor { } private void clearSummaryAttributeAspectForConcreteFields() { - for (SDField field : search.allConcreteFields()) { + for (SDField field : schema.allConcreteFields()) { if (field.getDataType() instanceof ReferenceDataType) { removeFromAttributePrefetchSummaryClass(field); clearSummaryTransformOnSummaryFields(field); @@ -51,14 +51,14 @@ public class ReferenceFieldsProcessor extends Processor { } private void removeFromAttributePrefetchSummaryClass(SDField field) { - DocumentSummary summary = search.getSummary("attributeprefetch"); + DocumentSummary summary = schema.getSummary("attributeprefetch"); if (summary != null) { summary.remove(field.getName()); } } private void clearSummaryTransformOnSummaryFields(SDField field) { - search.getSummaryFields(field).values().forEach(summaryField -> summaryField.setTransform(SummaryTransform.NONE)); + schema.getSummaryFields(field).values().forEach(summaryField -> summaryField.setTransform(SummaryTransform.NONE)); } } diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/processing/ReservedDocumentNames.java b/config-model/src/main/java/com/yahoo/searchdefinition/processing/ReservedDocumentNames.java index e017726370f..4a39a52a005 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/processing/ReservedDocumentNames.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/processing/ReservedDocumentNames.java @@ -3,8 +3,8 @@ package com.yahoo.searchdefinition.processing; import com.yahoo.config.application.api.DeployLogger; import com.yahoo.searchdefinition.RankProfileRegistry; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.document.SDDocumentType; -import com.yahoo.searchdefinition.Search; import com.yahoo.vespa.model.container.search.QueryProfiles; import java.util.HashSet; @@ -23,18 +23,17 @@ public class ReservedDocumentNames extends Processor { } } - public ReservedDocumentNames(Search search, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { - super(search, deployLogger, rankProfileRegistry, queryProfiles); + public ReservedDocumentNames(Schema schema, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { + super(schema, deployLogger, rankProfileRegistry, queryProfiles); } @Override public void process(boolean validate, boolean documentsOnly) { if ( ! validate) return; - String docName = search.getDocument().getName(); - if (RESERVED_NAMES.contains(docName)) { - throw new IllegalArgumentException("For search '" + search.getName() + "': Document name '" + docName + - "' is reserved."); - } + String docName = schema.getDocument().getName(); + if (RESERVED_NAMES.contains(docName)) + throw new IllegalArgumentException("For " + schema + ": Document name '" + docName + "' is reserved."); } + } diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/processing/ReservedFunctionNames.java b/config-model/src/main/java/com/yahoo/searchdefinition/processing/ReservedFunctionNames.java index d997422c8f0..8194b9f9e06 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/processing/ReservedFunctionNames.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/processing/ReservedFunctionNames.java @@ -5,7 +5,7 @@ import com.google.common.collect.ImmutableSet; import com.yahoo.config.application.api.DeployLogger; import com.yahoo.searchdefinition.RankProfile; import com.yahoo.searchdefinition.RankProfileRegistry; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchlib.rankingexpression.parser.RankingExpressionParserConstants; import com.yahoo.vespa.model.container.search.QueryProfiles; @@ -22,8 +22,8 @@ public class ReservedFunctionNames extends Processor { private static Set<String> reservedNames = getReservedNames(); - public ReservedFunctionNames(Search search, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { - super(search, deployLogger, rankProfileRegistry, queryProfiles); + public ReservedFunctionNames(Schema schema, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { + super(schema, deployLogger, rankProfileRegistry, queryProfiles); } @Override diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/processing/SearchMustHaveDocument.java b/config-model/src/main/java/com/yahoo/searchdefinition/processing/SearchMustHaveDocument.java index b9ecc2e234d..68717f1f06b 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/processing/SearchMustHaveDocument.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/processing/SearchMustHaveDocument.java @@ -3,7 +3,7 @@ package com.yahoo.searchdefinition.processing; import com.yahoo.config.application.api.DeployLogger; import com.yahoo.searchdefinition.RankProfileRegistry; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.vespa.model.container.search.QueryProfiles; /** @@ -14,17 +14,17 @@ import com.yahoo.vespa.model.container.search.QueryProfiles; */ public class SearchMustHaveDocument extends Processor { - public SearchMustHaveDocument(Search search, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { - super(search, deployLogger, rankProfileRegistry, queryProfiles); + public SearchMustHaveDocument(Schema schema, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { + super(schema, deployLogger, rankProfileRegistry, queryProfiles); } @Override public void process(boolean validate, boolean documentsOnly) { if ( ! validate) return; - if (search.getDocument() == null) - throw new IllegalArgumentException("For search '" + search.getName() + - "': A search specification must have an equally named document inside of it."); + if (schema.getDocument() == null) + throw new IllegalArgumentException("For " + schema + + ": A search specification must have an equally named document inside of it."); } } diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/processing/SetRankTypeEmptyOnFilters.java b/config-model/src/main/java/com/yahoo/searchdefinition/processing/SetRankTypeEmptyOnFilters.java index eaab175a46e..6631258813a 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/processing/SetRankTypeEmptyOnFilters.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/processing/SetRankTypeEmptyOnFilters.java @@ -3,9 +3,9 @@ package com.yahoo.searchdefinition.processing; import com.yahoo.config.application.api.DeployLogger; import com.yahoo.searchdefinition.RankProfileRegistry; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.document.RankType; import com.yahoo.searchdefinition.document.SDField; -import com.yahoo.searchdefinition.Search; import com.yahoo.vespa.model.container.search.QueryProfiles; /** @@ -15,13 +15,13 @@ import com.yahoo.vespa.model.container.search.QueryProfiles; */ public class SetRankTypeEmptyOnFilters extends Processor { - public SetRankTypeEmptyOnFilters(Search search, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { - super(search, deployLogger, rankProfileRegistry, queryProfiles); + public SetRankTypeEmptyOnFilters(Schema schema, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { + super(schema, deployLogger, rankProfileRegistry, queryProfiles); } @Override public void process(boolean validate, boolean documentsOnly) { - for (SDField field : search.allConcreteFields()) { + for (SDField field : schema.allConcreteFields()) { if (field.getRanking().isFilter()) { field.setRankType(RankType.EMPTY); } diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/processing/SortingSettings.java b/config-model/src/main/java/com/yahoo/searchdefinition/processing/SortingSettings.java index f81ff184417..33e9634323b 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/processing/SortingSettings.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/processing/SortingSettings.java @@ -3,10 +3,10 @@ package com.yahoo.searchdefinition.processing; import com.yahoo.config.application.api.DeployLogger; import com.yahoo.searchdefinition.RankProfileRegistry; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.document.Attribute; import com.yahoo.searchdefinition.document.SDField; import com.yahoo.searchdefinition.document.Sorting; -import com.yahoo.searchdefinition.Search; import com.yahoo.vespa.model.container.search.QueryProfiles; /** @@ -16,23 +16,23 @@ import com.yahoo.vespa.model.container.search.QueryProfiles; */ public class SortingSettings extends Processor { - public SortingSettings(Search search, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { - super(search, deployLogger, rankProfileRegistry, queryProfiles); + public SortingSettings(Schema schema, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { + super(schema, deployLogger, rankProfileRegistry, queryProfiles); } @Override public void process(boolean validate, boolean documentsOnly) { if ( ! validate) return; - for (SDField field : search.allConcreteFields()) { + for (SDField field : schema.allConcreteFields()) { for (Attribute attribute : field.getAttributes().values()) { Sorting sorting = attribute.getSorting(); if (sorting.getFunction() != Sorting.Function.UCA) { if (sorting.getStrength()!=null && sorting.getStrength() != Sorting.Strength.PRIMARY) { - warn(search, field, "Sort strength only works for sort function 'uca'."); + warn(schema, field, "Sort strength only works for sort function 'uca'."); } if (sorting.getLocale() != null && ! "".equals(sorting.getLocale())) { - warn(search, field, "Sort locale only works for sort function 'uca'."); + warn(schema, field, "Sort locale only works for sort function 'uca'."); } } } diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/processing/StringSettingsOnNonStringFields.java b/config-model/src/main/java/com/yahoo/searchdefinition/processing/StringSettingsOnNonStringFields.java index ea034665ae1..8c6b07515aa 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/processing/StringSettingsOnNonStringFields.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/processing/StringSettingsOnNonStringFields.java @@ -5,28 +5,28 @@ import com.yahoo.config.application.api.DeployLogger; import com.yahoo.searchdefinition.RankProfileRegistry; import com.yahoo.document.CollectionDataType; import com.yahoo.document.NumericDataType; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.document.SDField; -import com.yahoo.searchdefinition.Search; import com.yahoo.vespa.model.container.search.QueryProfiles; public class StringSettingsOnNonStringFields extends Processor { - public StringSettingsOnNonStringFields(Search search, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { - super(search, deployLogger, rankProfileRegistry, queryProfiles); + public StringSettingsOnNonStringFields(Schema schema, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { + super(schema, deployLogger, rankProfileRegistry, queryProfiles); } @Override public void process(boolean validate, boolean documentsOnly) { if ( ! validate) return; - for (SDField field : search.allConcreteFields()) { + for (SDField field : schema.allConcreteFields()) { if ( ! doCheck(field)) continue; if (field.getMatching().isTypeUserSet()) { - warn(search, field, "Matching type " + field.getMatching().getType() + + warn(schema, field, "Matching type " + field.getMatching().getType() + " is only allowed for string fields."); } if (field.getRanking().isLiteral()) { - warn(search, field, "Rank type literal only applies to string fields"); + warn(schema, field, "Rank type literal only applies to string fields"); } } } diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/processing/SummaryConsistency.java b/config-model/src/main/java/com/yahoo/searchdefinition/processing/SummaryConsistency.java index 613674c5c6f..80f9a15f1e5 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/processing/SummaryConsistency.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/processing/SummaryConsistency.java @@ -5,9 +5,9 @@ import com.yahoo.config.application.api.DeployLogger; import com.yahoo.document.DataType; import com.yahoo.document.TensorDataType; import com.yahoo.searchdefinition.RankProfileRegistry; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.document.Attribute; import com.yahoo.document.WeightedSetDataType; -import com.yahoo.searchdefinition.Search; import com.yahoo.searchdefinition.document.ImmutableSDField; import com.yahoo.vespa.documentmodel.DocumentSummary; import com.yahoo.vespa.documentmodel.SummaryField; @@ -24,26 +24,26 @@ import static com.yahoo.searchdefinition.document.ComplexAttributeFieldUtils.isC */ public class SummaryConsistency extends Processor { - public SummaryConsistency(Search search, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { - super(search, deployLogger, rankProfileRegistry, queryProfiles); + public SummaryConsistency(Schema schema, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { + super(schema, deployLogger, rankProfileRegistry, queryProfiles); } @Override public void process(boolean validate, boolean documentsOnly) { - for (DocumentSummary summary : search.getSummaries().values()) { + for (DocumentSummary summary : schema.getSummaries().values()) { if (summary.getName().equals("default")) continue; for (SummaryField summaryField : summary.getSummaryFields() ) { - assertConsistency(summaryField, search, validate); - makeAttributeTransformIfAppropriate(summaryField, search); - makeAttributeCombinerTransformIfAppropriate(summaryField, search); + assertConsistency(summaryField, schema, validate); + makeAttributeTransformIfAppropriate(summaryField, schema); + makeAttributeCombinerTransformIfAppropriate(summaryField, schema); } } } - private void assertConsistency(SummaryField summaryField, Search search, boolean validate) { + private void assertConsistency(SummaryField summaryField, Schema schema, boolean validate) { // Compare to default: - SummaryField existingDefault = search.getSummary("default").getSummaryField(summaryField.getName()); + SummaryField existingDefault = schema.getSummary("default").getSummaryField(summaryField.getName()); if (existingDefault != null) { if (validate) assertConsistentTypes(existingDefault, summaryField); @@ -51,27 +51,27 @@ public class SummaryConsistency extends Processor { } else { // If no default, compare to whichever definition of the field - SummaryField existing = search.getExplicitSummaryField(summaryField.getName()); + SummaryField existing = schema.getExplicitSummaryField(summaryField.getName()); if (existing == null) return; if (validate) assertConsistentTypes(existing, summaryField); - makeConsistentOrThrow(existing, summaryField, search); + makeConsistentOrThrow(existing, summaryField, schema); } } /** If the source is an attribute, make this use the attribute transform */ - private void makeAttributeTransformIfAppropriate(SummaryField summaryField,Search search) { + private void makeAttributeTransformIfAppropriate(SummaryField summaryField, Schema schema) { if (summaryField.getTransform() != SummaryTransform.NONE) return; - Attribute attribute = search.getAttribute(summaryField.getSingleSource()); + Attribute attribute = schema.getAttribute(summaryField.getSingleSource()); if (attribute == null) return; summaryField.setTransform(SummaryTransform.ATTRIBUTE); } /** If the source is a complex field with only struct field attributes then make this use the attribute combiner transform */ - private void makeAttributeCombinerTransformIfAppropriate(SummaryField summaryField,Search search) { + private void makeAttributeCombinerTransformIfAppropriate(SummaryField summaryField, Schema schema) { if (summaryField.getTransform() == SummaryTransform.NONE) { String source_field_name = summaryField.getSingleSource(); - ImmutableSDField source = search.getField(source_field_name); + ImmutableSDField source = schema.getField(source_field_name); if (source != null && isComplexFieldWithOnlyStructFieldAttributes(source)) { summaryField.setTransform(SummaryTransform.ATTRIBUTECOMBINER); } @@ -94,9 +94,9 @@ public class SummaryConsistency extends Processor { return summaryType.equals(existingType); } - private void makeConsistentOrThrow(SummaryField field1, SummaryField field2, Search search) { + private void makeConsistentOrThrow(SummaryField field1, SummaryField field2, Schema schema) { if (field2.getTransform() == SummaryTransform.ATTRIBUTE && field1.getTransform() == SummaryTransform.NONE) { - Attribute attribute = search.getAttribute(field1.getName()); + Attribute attribute = schema.getAttribute(field1.getName()); if (attribute != null) { field1.setTransform(SummaryTransform.ATTRIBUTE); } diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/processing/SummaryDiskAccessValidator.java b/config-model/src/main/java/com/yahoo/searchdefinition/processing/SummaryDiskAccessValidator.java index 4ae624d2c35..4bc553b1669 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/processing/SummaryDiskAccessValidator.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/processing/SummaryDiskAccessValidator.java @@ -3,7 +3,7 @@ package com.yahoo.searchdefinition.processing; import com.yahoo.config.application.api.DeployLogger; import com.yahoo.searchdefinition.RankProfileRegistry; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.derived.SummaryClass; import com.yahoo.searchdefinition.document.ImmutableSDField; import com.yahoo.vespa.documentmodel.DocumentSummary; @@ -23,11 +23,11 @@ import static com.yahoo.searchdefinition.document.ComplexAttributeFieldUtils.isC */ public class SummaryDiskAccessValidator extends Processor { - public SummaryDiskAccessValidator(Search search, + public SummaryDiskAccessValidator(Schema schema, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { - super(search, deployLogger, rankProfileRegistry, queryProfiles); + super(schema, deployLogger, rankProfileRegistry, queryProfiles); } @@ -36,12 +36,12 @@ public class SummaryDiskAccessValidator extends Processor { if ( ! validate) return; if (documentsOnly) return; - for (DocumentSummary summary : search.getSummaries().values()) { + for (DocumentSummary summary : schema.getSummaries().values()) { for (SummaryField summaryField : summary.getSummaryFields()) { for (SummaryField.Source source : summaryField.getSources()) { - ImmutableSDField field = search.getField(source.getName()); + ImmutableSDField field = schema.getField(source.getName()); if (field == null) - field = findFieldProducingSummaryField(source.getName(), search).orElse(null); + field = findFieldProducingSummaryField(source.getName(), schema).orElse(null); if (field == null && ! source.getName().equals(SummaryClass.DOCUMENT_ID_FIELD)) throw new IllegalArgumentException(summaryField + " in " + summary + " references " + source + ", but this field does not exist"); @@ -66,8 +66,8 @@ public class SummaryDiskAccessValidator extends Processor { return field.doesAttributing(); } - private Optional<ImmutableSDField> findFieldProducingSummaryField(String name, Search search) { - return search.allFields().filter(field -> field.getSummaryFields().get(name) != null).findAny(); + private Optional<ImmutableSDField> findFieldProducingSummaryField(String name, Schema schema) { + return schema.allFields().filter(field -> field.getSummaryFields().get(name) != null).findAny(); } } diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/processing/SummaryDynamicStructsArrays.java b/config-model/src/main/java/com/yahoo/searchdefinition/processing/SummaryDynamicStructsArrays.java index a9bd57705f8..c350c6800ed 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/processing/SummaryDynamicStructsArrays.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/processing/SummaryDynamicStructsArrays.java @@ -5,7 +5,7 @@ package com.yahoo.searchdefinition.processing; import com.yahoo.config.application.api.DeployLogger; import com.yahoo.searchdefinition.RankProfileRegistry; import com.yahoo.document.*; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.document.SDField; import com.yahoo.vespa.documentmodel.SummaryField; import com.yahoo.vespa.documentmodel.SummaryTransform; @@ -21,15 +21,15 @@ import com.yahoo.vespa.model.container.search.QueryProfiles; */ public class SummaryDynamicStructsArrays extends Processor { - public SummaryDynamicStructsArrays(Search search, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { - super(search, deployLogger, rankProfileRegistry, queryProfiles); + public SummaryDynamicStructsArrays(Schema schema, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { + super(schema, deployLogger, rankProfileRegistry, queryProfiles); } @Override public void process(boolean validate, boolean documentsOnly) { if ( ! validate) return; - for (SDField field : search.allConcreteFields()) { + for (SDField field : schema.allConcreteFields()) { DataType type = field.getDataType(); if (type instanceof ArrayDataType || type instanceof WeightedSetDataType || type instanceof StructDataType) { for (SummaryField sField : field.getSummaryFields().values()) { diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/processing/SummaryFieldsMustHaveValidSource.java b/config-model/src/main/java/com/yahoo/searchdefinition/processing/SummaryFieldsMustHaveValidSource.java index 3ebfa629895..2e2144b819d 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/processing/SummaryFieldsMustHaveValidSource.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/processing/SummaryFieldsMustHaveValidSource.java @@ -3,7 +3,7 @@ package com.yahoo.searchdefinition.processing; import com.yahoo.config.application.api.DeployLogger; import com.yahoo.searchdefinition.RankProfileRegistry; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.derived.SummaryClass; import com.yahoo.vespa.documentmodel.DocumentSummary; import com.yahoo.vespa.documentmodel.SummaryField; @@ -17,15 +17,15 @@ import com.yahoo.vespa.model.container.search.QueryProfiles; */ public class SummaryFieldsMustHaveValidSource extends Processor { - SummaryFieldsMustHaveValidSource(Search search, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { - super(search, deployLogger, rankProfileRegistry, queryProfiles); + SummaryFieldsMustHaveValidSource(Schema schema, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { + super(schema, deployLogger, rankProfileRegistry, queryProfiles); } @Override public void process(boolean validate, boolean documentsOnly) { if ( ! validate) return; - for (DocumentSummary summary : search.getSummaries().values()) { + for (DocumentSummary summary : schema.getSummaries().values()) { for (SummaryField summaryField : summary.getSummaryFields()) { if (summaryField.getSources().isEmpty()) { if ((summaryField.getTransform() != SummaryTransform.RANKFEATURES) && @@ -56,7 +56,7 @@ public class SummaryFieldsMustHaveValidSource extends Processor { private void verifySource(String source, SummaryField summaryField, DocumentSummary summary) { if ( ! isValid(source, summaryField, summary) ) { - throw new IllegalArgumentException("For search '" + search.getName() + "', summary class '" + + throw new IllegalArgumentException("For " + schema + ", summary class '" + summary.getName() + "'," + " summary field '" + summaryField.getName() + "': there is no valid source '" + source + "'."); } @@ -71,11 +71,11 @@ public class SummaryFieldsMustHaveValidSource extends Processor { } private boolean isDocumentField(String name) { - return search.getField(name) != null; + return schema.getField(name) != null; } private boolean isSummaryField(String name) { - return search.getSummaryField(name) != null; + return schema.getSummaryField(name) != null; } } diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/processing/SummaryNamesFieldCollisions.java b/config-model/src/main/java/com/yahoo/searchdefinition/processing/SummaryNamesFieldCollisions.java index b7febdd73b1..1c6f8d54ea3 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/processing/SummaryNamesFieldCollisions.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/processing/SummaryNamesFieldCollisions.java @@ -7,7 +7,7 @@ import java.util.Map; import com.yahoo.collections.Pair; import com.yahoo.config.application.api.DeployLogger; import com.yahoo.searchdefinition.RankProfileRegistry; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.vespa.documentmodel.DocumentSummary; import com.yahoo.vespa.documentmodel.SummaryField; import com.yahoo.vespa.documentmodel.SummaryField.Source; @@ -21,8 +21,8 @@ import com.yahoo.vespa.model.container.search.QueryProfiles; */ public class SummaryNamesFieldCollisions extends Processor { - public SummaryNamesFieldCollisions(Search search, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { - super(search, deployLogger, rankProfileRegistry, queryProfiles); + public SummaryNamesFieldCollisions(Schema schema, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { + super(schema, deployLogger, rankProfileRegistry, queryProfiles); } @Override @@ -30,7 +30,7 @@ public class SummaryNamesFieldCollisions extends Processor { if ( ! validate) return; Map<String, Pair<String, String>> fieldToClassAndSource = new HashMap<>(); - for (DocumentSummary summary : search.getSummaries().values()) { + for (DocumentSummary summary : schema.getSummaries().values()) { if ("default".equals(summary.getName())) continue; for (SummaryField summaryField : summary.getSummaryFields() ) { if (summaryField.isImplicit()) continue; @@ -41,12 +41,12 @@ public class SummaryNamesFieldCollisions extends Processor { String prevSource = prevClassAndSource.getSecond(); if ( ! prevClass.equals(summary.getName())) { if ( ! prevSource.equals(source.getName())) { - throw new IllegalArgumentException("For search '"+ search.getName() + - "', summary class '" + summary.getName()+"'," + - " summary field '" + summaryField.getName() + "':" + - " Can not use source '" + source.getName() + + throw new IllegalArgumentException("For " + schema + + ", summary class '" + summary.getName() + "'," + + " summary field '" + summaryField.getName() + "':" + + " Can not use source '" + source.getName() + "' for this summary field, an equally named field in summary class '" + - prevClass + "' uses a different source: '"+prevSource+"'."); + prevClass + "' uses a different source: '" + prevSource + "'."); } } } else { diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/processing/TagType.java b/config-model/src/main/java/com/yahoo/searchdefinition/processing/TagType.java index 57deb82854a..8cebfc89458 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/processing/TagType.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/processing/TagType.java @@ -4,7 +4,7 @@ package com.yahoo.searchdefinition.processing; import com.yahoo.config.application.api.DeployLogger; import com.yahoo.searchdefinition.RankProfileRegistry; import com.yahoo.document.*; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.document.Matching; import com.yahoo.searchdefinition.document.RankType; import com.yahoo.searchdefinition.document.SDField; @@ -17,16 +17,16 @@ import com.yahoo.vespa.model.container.search.QueryProfiles; */ public class TagType extends Processor { - public TagType(Search search, + public TagType(Schema schema, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { - super(search, deployLogger, rankProfileRegistry, queryProfiles); + super(schema, deployLogger, rankProfileRegistry, queryProfiles); } @Override public void process(boolean validate, boolean documentsOnly) { - for (SDField field : search.allConcreteFields()) { + for (SDField field : schema.allConcreteFields()) { if (field.getDataType() instanceof WeightedSetDataType && ((WeightedSetDataType)field.getDataType()).isTag()) implementTagType(field); } diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/processing/TensorFieldProcessor.java b/config-model/src/main/java/com/yahoo/searchdefinition/processing/TensorFieldProcessor.java index 32fefc6d489..a253076f320 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/processing/TensorFieldProcessor.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/processing/TensorFieldProcessor.java @@ -5,7 +5,7 @@ import com.yahoo.config.application.api.DeployLogger; import com.yahoo.document.CollectionDataType; import com.yahoo.document.TensorDataType; import com.yahoo.searchdefinition.RankProfileRegistry; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.document.HnswIndexParams; import com.yahoo.searchdefinition.document.ImmutableSDField; import com.yahoo.searchdefinition.document.SDField; @@ -18,13 +18,13 @@ import com.yahoo.vespa.model.container.search.QueryProfiles; */ public class TensorFieldProcessor extends Processor { - public TensorFieldProcessor(Search search, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { - super(search, deployLogger, rankProfileRegistry, queryProfiles); + public TensorFieldProcessor(Schema schema, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { + super(schema, deployLogger, rankProfileRegistry, queryProfiles); } @Override public void process(boolean validate, boolean documentsOnly) { - for (var field : search.allConcreteFields()) { + for (var field : schema.allConcreteFields()) { if ( field.getDataType() instanceof TensorDataType ) { if (validate) { validateIndexingScripsForTensorField(field); @@ -43,8 +43,8 @@ public class TensorFieldProcessor extends Processor { private void validateIndexingScripsForTensorField(SDField field) { if (field.doesIndexing() && !isTensorTypeThatSupportsHnswIndex(field)) { - fail(search, field, "A tensor of type '" + tensorTypeToString(field) + "' does not support having an 'index'. " + - "Currently, only tensors with 1 indexed dimension supports that."); + fail(schema, field, "A tensor of type '" + tensorTypeToString(field) + "' does not support having an 'index'. " + + "Currently, only tensors with 1 indexed dimension supports that."); } } @@ -79,7 +79,7 @@ public class TensorFieldProcessor extends Processor { var attribute = field.getAttributes().get(field.getName()); if (attribute != null && attribute.isFastSearch()) { if (! isTensorTypeThatSupportsDirectStore(field)) { - fail(search, field, "An attribute of type 'tensor' cannot be 'fast-search'."); + fail(schema, field, "An attribute of type 'tensor' cannot be 'fast-search'."); } } } @@ -88,7 +88,7 @@ public class TensorFieldProcessor extends Processor { private void validateHnswIndexParametersRequiresIndexing(SDField field) { var index = field.getIndex(field.getName()); if (index != null && index.getHnswIndexParams().isPresent() && !field.doesIndexing()) { - fail(search, field, "A tensor that specifies hnsw index parameters must also specify 'index' in 'indexing'"); + fail(schema, field, "A tensor that specifies hnsw index parameters must also specify 'index' in 'indexing'"); } } @@ -98,7 +98,7 @@ public class TensorFieldProcessor extends Processor { } if (isTensorTypeThatSupportsHnswIndex(field)) { if (validate && !field.doesAttributing()) { - fail(search, field, "A tensor that has an index must also be an attribute."); + fail(schema, field, "A tensor that has an index must also be an attribute."); } var index = field.getIndex(field.getName()); // TODO: Calculate default params based on tensor dimension size @@ -112,7 +112,7 @@ public class TensorFieldProcessor extends Processor { private void validateDataTypeForCollectionField(SDField field) { if (((CollectionDataType)field.getDataType()).getNestedType() instanceof TensorDataType) - fail(search, field, "A field with collection type of tensor is not supported. Use simple type 'tensor' instead."); + fail(schema, field, "A field with collection type of tensor is not supported. Use simple type 'tensor' instead."); } } diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/processing/TextMatch.java b/config-model/src/main/java/com/yahoo/searchdefinition/processing/TextMatch.java index c1c18a552ab..a8065ec4bb5 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/processing/TextMatch.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/processing/TextMatch.java @@ -5,7 +5,7 @@ import com.yahoo.config.application.api.DeployLogger; import com.yahoo.searchdefinition.RankProfileRegistry; import com.yahoo.document.CollectionDataType; import com.yahoo.document.DataType; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.document.Matching; import com.yahoo.searchdefinition.document.SDField; import com.yahoo.searchdefinition.document.Stemming; @@ -29,13 +29,13 @@ import java.util.TreeSet; */ public class TextMatch extends Processor { - public TextMatch(Search search, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { - super(search, deployLogger, rankProfileRegistry, queryProfiles); + public TextMatch(Schema schema, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { + super(schema, deployLogger, rankProfileRegistry, queryProfiles); } @Override public void process(boolean validate, boolean documentsOnly) { - for (SDField field : search.allConcreteFields()) { + for (SDField field : schema.allConcreteFields()) { if (field.getMatching().getType() != Matching.Type.TEXT) continue; ScriptExpression script = field.getIndexingScript(); @@ -49,7 +49,7 @@ public class TextMatch extends Processor { Set<String> dynamicSummary = new TreeSet<>(); Set<String> staticSummary = new TreeSet<>(); - new IndexingOutputs(search, deployLogger, rankProfileRegistry, queryProfiles).findSummaryTo(search, + new IndexingOutputs(schema, deployLogger, rankProfileRegistry, queryProfiles).findSummaryTo(schema, field, dynamicSummary, staticSummary); @@ -57,16 +57,16 @@ public class TextMatch extends Processor { visitor.visit(script); if ( ! visitor.requiresTokenize) continue; - ExpressionConverter converter = new MyStringTokenizer(search, findAnnotatorConfig(search, field)); + ExpressionConverter converter = new MyStringTokenizer(schema, findAnnotatorConfig(schema, field)); field.setIndexingScript((ScriptExpression)converter.convert(script)); } } - private AnnotatorConfig findAnnotatorConfig(Search search, SDField field) { + private AnnotatorConfig findAnnotatorConfig(Schema schema, SDField field) { AnnotatorConfig ret = new AnnotatorConfig(); Stemming activeStemming = field.getStemming(); if (activeStemming == null) { - activeStemming = search.getStemming(); + activeStemming = schema.getStemming(); } ret.setStemMode(activeStemming.toStemMode()); ret.setRemoveAccents(field.getNormalizing().doRemoveAccents()); @@ -103,8 +103,8 @@ public class TextMatch extends Processor { final AnnotatorConfig annotatorCfg; - MyStringTokenizer(Search search, AnnotatorConfig annotatorCfg) { - super(TokenizeExpression.class, search); + MyStringTokenizer(Schema schema, AnnotatorConfig annotatorCfg) { + super(TokenizeExpression.class, schema); this.annotatorCfg = annotatorCfg; } diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/processing/TypedTransformProvider.java b/config-model/src/main/java/com/yahoo/searchdefinition/processing/TypedTransformProvider.java index e309a459a67..d6fdb901ab7 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/processing/TypedTransformProvider.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/processing/TypedTransformProvider.java @@ -3,7 +3,7 @@ package com.yahoo.searchdefinition.processing; import com.yahoo.document.DataType; import com.yahoo.document.Field; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.document.Attribute; import com.yahoo.vespa.indexinglanguage.ValueTransformProvider; import com.yahoo.vespa.indexinglanguage.expressions.*; @@ -13,12 +13,12 @@ import com.yahoo.vespa.indexinglanguage.expressions.*; */ public abstract class TypedTransformProvider extends ValueTransformProvider { - private final Search search; + private final Schema schema; private DataType fieldType; - TypedTransformProvider(Class<? extends Expression> transformClass, Search search) { + TypedTransformProvider(Class<? extends Expression> transformClass, Schema schema) { super(transformClass); - this.search = search; + this.schema = schema; } @Override @@ -26,19 +26,19 @@ public abstract class TypedTransformProvider extends ValueTransformProvider { if (exp instanceof OutputExpression) { String fieldName = ((OutputExpression)exp).getFieldName(); if (exp instanceof AttributeExpression) { - Attribute attribute = search.getAttribute(fieldName); + Attribute attribute = schema.getAttribute(fieldName); if (attribute == null) throw new IllegalArgumentException("Attribute '" + fieldName + "' not found."); fieldType = attribute.getDataType(); } else if (exp instanceof IndexExpression) { - Field field = search.getConcreteField(fieldName); + Field field = schema.getConcreteField(fieldName); if (field == null) throw new IllegalArgumentException("Index field '" + fieldName + "' not found."); fieldType = field.getDataType(); } else if (exp instanceof SummaryExpression) { - Field field = search.getSummaryField(fieldName); + Field field = schema.getSummaryField(fieldName); if (field == null) throw new IllegalArgumentException("Summary field '" + fieldName + "' not found."); fieldType = field.getDataType(); diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/processing/UriHack.java b/config-model/src/main/java/com/yahoo/searchdefinition/processing/UriHack.java index 224d7593631..84dc6d369fc 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/processing/UriHack.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/processing/UriHack.java @@ -7,7 +7,7 @@ import com.yahoo.document.CollectionDataType; import com.yahoo.document.DataType; import com.yahoo.document.WeightedSetDataType; import com.yahoo.searchdefinition.RankProfileRegistry; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.document.SDField; import com.yahoo.searchdefinition.document.Stemming; import com.yahoo.vespa.model.container.search.QueryProfiles; @@ -23,29 +23,29 @@ public class UriHack extends Processor { private static final List<String> URL_SUFFIX = Arrays.asList("scheme", "host", "port", "path", "query", "fragment", "hostname"); - UriHack(Search search, - DeployLogger deployLogger, - RankProfileRegistry rankProfileRegistry, - QueryProfiles queryProfiles) { - super(search, deployLogger, rankProfileRegistry, queryProfiles); + UriHack(Schema schema, + DeployLogger deployLogger, + RankProfileRegistry rankProfileRegistry, + QueryProfiles queryProfiles) { + super(schema, deployLogger, rankProfileRegistry, queryProfiles); } @Override public void process(boolean validate, boolean documentsOnly) { - for (SDField field : search.allConcreteFields()) { + for (SDField field : schema.allConcreteFields()) { if (field.doesIndexing()) { DataType fieldType = field.getDataType(); if (fieldType instanceof CollectionDataType) { fieldType = ((CollectionDataType)fieldType).getNestedType(); } if (fieldType == DataType.URI) { - processField(search, field); + processField(schema, field); } } } } - private void processField(Search search, SDField uriField) { + private void processField(Schema schema, SDField uriField) { String uriName = uriField.getName(); uriField.setStemming(Stemming.NONE); DataType generatedType = DataType.STRING; @@ -69,8 +69,8 @@ public class UriHack extends Processor { if (uriField.getIndex(suffix) != null) { partField.addIndex(uriField.getIndex(suffix)); } - search.addExtraField(partField); - search.fieldSets().addBuiltInFieldSetItem(BuiltInFieldSets.INTERNAL_FIELDSET_NAME, partField.getName()); + schema.addExtraField(partField); + schema.fieldSets().addBuiltInFieldSetItem(BuiltInFieldSets.INTERNAL_FIELDSET_NAME, partField.getName()); } } diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/processing/UrlFieldValidator.java b/config-model/src/main/java/com/yahoo/searchdefinition/processing/UrlFieldValidator.java index f059785057b..72f903f8365 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/processing/UrlFieldValidator.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/processing/UrlFieldValidator.java @@ -4,7 +4,7 @@ package com.yahoo.searchdefinition.processing; import com.yahoo.config.application.api.DeployLogger; import com.yahoo.document.DataType; import com.yahoo.searchdefinition.RankProfileRegistry; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.document.ImmutableSDField; import com.yahoo.vespa.model.container.search.QueryProfiles; @@ -13,19 +13,19 @@ import com.yahoo.vespa.model.container.search.QueryProfiles; */ public class UrlFieldValidator extends Processor { - public UrlFieldValidator(Search search, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { - super(search, deployLogger, rankProfileRegistry, queryProfiles); + public UrlFieldValidator(Schema schema, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { + super(schema, deployLogger, rankProfileRegistry, queryProfiles); } @Override public void process(boolean validate, boolean documentsOnly) { if ( ! validate) return; - for (ImmutableSDField field : search.allConcreteFields()) { + for (ImmutableSDField field : schema.allConcreteFields()) { if ( ! field.getDataType().equals(DataType.URI)) continue; if (field.doesAttributing()) - throw new IllegalArgumentException("Error in " + field + " in " + search + ": " + + throw new IllegalArgumentException("Error in " + field + " in " + schema + ": " + "uri type fields cannot be attributes"); } diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/processing/ValidateFieldTypes.java b/config-model/src/main/java/com/yahoo/searchdefinition/processing/ValidateFieldTypes.java index 923726b7696..0d4bfd09677 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/processing/ValidateFieldTypes.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/processing/ValidateFieldTypes.java @@ -5,7 +5,7 @@ import com.yahoo.config.application.api.DeployLogger; import com.yahoo.document.DataType; import com.yahoo.document.TensorDataType; import com.yahoo.searchdefinition.RankProfileRegistry; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.document.Attribute; import com.yahoo.vespa.documentmodel.DocumentSummary; import com.yahoo.vespa.documentmodel.SummaryField; @@ -23,22 +23,22 @@ import java.util.Map; */ public class ValidateFieldTypes extends Processor { - public ValidateFieldTypes(Search search, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { - super(search, deployLogger, rankProfileRegistry, queryProfiles); + public ValidateFieldTypes(Schema schema, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { + super(schema, deployLogger, rankProfileRegistry, queryProfiles); } @Override public void process(boolean validate, boolean documentsOnly) { if (!validate) return; - String searchName = search.getName(); + String searchName = schema.getName(); Map<String, DataType> seenFields = new HashMap<>(); verifySearchAndDocFields(searchName, seenFields); verifySummaryFields(searchName, seenFields); } final protected void verifySearchAndDocFields(String searchName, Map<String, DataType> seenFields) { - search.allFields().forEach(field -> { + schema.allFields().forEach(field -> { checkFieldType(searchName, "index field", field.getName(), field.getDataType(), seenFields); for (Map.Entry<String, Attribute> entry : field.getAttributes().entrySet()) { checkFieldType(searchName, "attribute", entry.getKey(), entry.getValue().getDataType(), seenFields); @@ -47,7 +47,7 @@ public class ValidateFieldTypes extends Processor { } final protected void verifySummaryFields(String searchName, Map<String, DataType> seenFields) { - for (DocumentSummary summary : search.getSummaries().values()) { + for (DocumentSummary summary : schema.getSummaries().values()) { for (SummaryField field : summary.getSummaryFields()) { checkFieldType(searchName, "summary field", field.getName(), field.getDataType(), seenFields); } diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/processing/ValidateFieldTypesDocumentsOnly.java b/config-model/src/main/java/com/yahoo/searchdefinition/processing/ValidateFieldTypesDocumentsOnly.java index 334e2f8c340..e8b218a80ad 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/processing/ValidateFieldTypesDocumentsOnly.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/processing/ValidateFieldTypesDocumentsOnly.java @@ -4,22 +4,22 @@ package com.yahoo.searchdefinition.processing; import com.yahoo.config.application.api.DeployLogger; import com.yahoo.document.DataType; import com.yahoo.searchdefinition.RankProfileRegistry; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.vespa.model.container.search.QueryProfiles; import java.util.HashMap; import java.util.Map; public class ValidateFieldTypesDocumentsOnly extends ValidateFieldTypes { - public ValidateFieldTypesDocumentsOnly(Search search, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { - super(search, deployLogger, rankProfileRegistry, queryProfiles); + public ValidateFieldTypesDocumentsOnly(Schema schema, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { + super(schema, deployLogger, rankProfileRegistry, queryProfiles); } @Override public void process(boolean validate, boolean documentsOnly) { if ( ! validate) return; - String searchName = search.getName(); + String searchName = schema.getName(); Map<String, DataType> seenFields = new HashMap<>(); verifySearchAndDocFields(searchName, seenFields); } diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/processing/ValidateFieldWithIndexSettingsCreatesIndex.java b/config-model/src/main/java/com/yahoo/searchdefinition/processing/ValidateFieldWithIndexSettingsCreatesIndex.java index 2a14616e5e7..268a9f27b3c 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/processing/ValidateFieldWithIndexSettingsCreatesIndex.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/processing/ValidateFieldWithIndexSettingsCreatesIndex.java @@ -3,11 +3,10 @@ package com.yahoo.searchdefinition.processing; import com.yahoo.config.application.api.DeployLogger; import com.yahoo.searchdefinition.RankProfileRegistry; -import com.yahoo.searchdefinition.document.ImmutableSDField; import com.yahoo.searchdefinition.document.Matching; import com.yahoo.searchdefinition.document.Ranking; import com.yahoo.searchdefinition.document.SDField; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.vespa.model.container.search.QueryProfiles; /** @@ -17,8 +16,8 @@ import com.yahoo.vespa.model.container.search.QueryProfiles; */ public class ValidateFieldWithIndexSettingsCreatesIndex extends Processor { - public ValidateFieldWithIndexSettingsCreatesIndex(Search search, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { - super(search, deployLogger, rankProfileRegistry, queryProfiles); + public ValidateFieldWithIndexSettingsCreatesIndex(Schema schema, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { + super(schema, deployLogger, rankProfileRegistry, queryProfiles); } @Override @@ -27,15 +26,15 @@ public class ValidateFieldWithIndexSettingsCreatesIndex extends Processor { Matching defaultMatching = new Matching(); Ranking defaultRanking = new Ranking(); - for (SDField field : search.allConcreteFields()) { + for (SDField field : schema.allConcreteFields()) { if (field.doesIndexing()) continue; if (field.doesAttributing()) continue; if ( ! field.getRanking().equals(defaultRanking)) - fail(search, field, + fail(schema, field, "Fields which are not creating an index or attribute can not contain rank settings."); if ( ! field.getMatching().equals(defaultMatching)) - fail(search, field, + fail(schema, field, "Fields which are not creating an index or attribute can not contain match settings."); } } diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/processing/WordMatch.java b/config-model/src/main/java/com/yahoo/searchdefinition/processing/WordMatch.java index 0d8045705b7..c4989604457 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/processing/WordMatch.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/processing/WordMatch.java @@ -6,7 +6,7 @@ import com.yahoo.searchdefinition.RankProfileRegistry; import com.yahoo.searchdefinition.document.Matching; import com.yahoo.searchdefinition.document.SDField; import com.yahoo.searchdefinition.document.Stemming; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.vespa.model.container.search.QueryProfiles; /** @@ -21,12 +21,12 @@ import com.yahoo.vespa.model.container.search.QueryProfiles; */ public class WordMatch extends Processor { - public WordMatch(Search search, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { - super(search, deployLogger, rankProfileRegistry, queryProfiles); + public WordMatch(Schema schema, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { + super(schema, deployLogger, rankProfileRegistry, queryProfiles); } public void process(boolean validate, boolean documentsOnly) { - for (SDField field : search.allConcreteFields()) { + for (SDField field : schema.allConcreteFields()) { processFieldRecursive(field); } } diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/processing/multifieldresolver/IndexCommandResolver.java b/config-model/src/main/java/com/yahoo/searchdefinition/processing/multifieldresolver/IndexCommandResolver.java index c9ee6425f97..56926436545 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/processing/multifieldresolver/IndexCommandResolver.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/processing/multifieldresolver/IndexCommandResolver.java @@ -3,7 +3,7 @@ package com.yahoo.searchdefinition.processing.multifieldresolver; import com.yahoo.config.application.api.DeployLogger; import com.yahoo.searchdefinition.document.SDField; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import java.util.ArrayList; import java.util.Arrays; @@ -28,8 +28,8 @@ public class IndexCommandResolver extends MultiFieldResolver { harmonizedCommands.addAll(Arrays.asList(harmonize)); } - public IndexCommandResolver(String indexName, List<SDField> fields, Search search, DeployLogger logger) { - super(indexName, fields, search, logger); + public IndexCommandResolver(String indexName, List<SDField> fields, Schema schema, DeployLogger logger) { + super(indexName, fields, schema, logger); } /** diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/processing/multifieldresolver/MultiFieldResolver.java b/config-model/src/main/java/com/yahoo/searchdefinition/processing/multifieldresolver/MultiFieldResolver.java index c27fe7d875d..80bb4faeaa5 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/processing/multifieldresolver/MultiFieldResolver.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/processing/multifieldresolver/MultiFieldResolver.java @@ -3,7 +3,7 @@ package com.yahoo.searchdefinition.processing.multifieldresolver; import com.yahoo.config.application.api.DeployLogger; import com.yahoo.searchdefinition.document.SDField; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import java.util.List; /** @@ -13,14 +13,14 @@ public abstract class MultiFieldResolver { protected String indexName; protected List<SDField> fields; - protected Search search; + protected Schema schema; protected DeployLogger deployLogger; - public MultiFieldResolver(String indexName, List<SDField> fields, Search search, DeployLogger logger) { + public MultiFieldResolver(String indexName, List<SDField> fields, Schema schema, DeployLogger logger) { this.indexName = indexName; this.fields = fields; - this.search = search; + this.schema = schema; this.deployLogger = logger; } diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/processing/multifieldresolver/RankProfileTypeSettingsProcessor.java b/config-model/src/main/java/com/yahoo/searchdefinition/processing/multifieldresolver/RankProfileTypeSettingsProcessor.java index 34f60116aa8..260d76c8ca3 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/processing/multifieldresolver/RankProfileTypeSettingsProcessor.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/processing/multifieldresolver/RankProfileTypeSettingsProcessor.java @@ -9,7 +9,7 @@ import com.yahoo.search.query.profile.types.TensorFieldType; import com.yahoo.searchdefinition.FeatureNames; import com.yahoo.searchdefinition.RankProfile; import com.yahoo.searchdefinition.RankProfileRegistry; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.document.Attribute; import com.yahoo.searchdefinition.document.ImmutableSDField; import com.yahoo.searchdefinition.document.ImportedField; @@ -29,8 +29,8 @@ import java.util.Optional; */ public class RankProfileTypeSettingsProcessor extends Processor { - public RankProfileTypeSettingsProcessor(Search search, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { - super(search, deployLogger, rankProfileRegistry, queryProfiles); + public RankProfileTypeSettingsProcessor(Schema schema, DeployLogger deployLogger, RankProfileRegistry rankProfileRegistry, QueryProfiles queryProfiles) { + super(schema, deployLogger, rankProfileRegistry, queryProfiles); } @Override @@ -43,8 +43,8 @@ public class RankProfileTypeSettingsProcessor extends Processor { } private void processAttributeFields() { - if (search == null) return; // we're processing global profiles - for (ImmutableSDField field : search.allConcreteFields()) { + if (schema == null) return; // we're processing global profiles + for (ImmutableSDField field : schema.allConcreteFields()) { Attribute attribute = field.getAttributes().get(field.getName()); if (attribute != null && attribute.tensorType().isPresent()) { addAttributeTypeToRankProfiles(attribute.getName(), attribute.tensorType().get().toString()); @@ -53,8 +53,8 @@ public class RankProfileTypeSettingsProcessor extends Processor { } private void processImportedFields() { - if (search == null) return; // we're processing global profiles - Optional<ImportedFields> importedFields = search.importedFields(); + if (schema == null) return; // we're processing global profiles + Optional<ImportedFields> importedFields = schema.importedFields(); if (importedFields.isPresent()) { importedFields.get().fields().forEach((fieldName, field) -> processImportedField(field)); } @@ -69,7 +69,7 @@ public class RankProfileTypeSettingsProcessor extends Processor { } private void addAttributeTypeToRankProfiles(String attributeName, String attributeType) { - for (RankProfile profile : rankProfileRegistry.rankProfilesOf(search)) { + for (RankProfile profile : rankProfileRegistry.rankProfilesOf(schema)) { profile.addAttributeType(attributeName, attributeType); } } diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/processing/multifieldresolver/RankTypeResolver.java b/config-model/src/main/java/com/yahoo/searchdefinition/processing/multifieldresolver/RankTypeResolver.java index 89acda469b7..231a97f30ec 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/processing/multifieldresolver/RankTypeResolver.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/processing/multifieldresolver/RankTypeResolver.java @@ -4,7 +4,7 @@ package com.yahoo.searchdefinition.processing.multifieldresolver; import com.yahoo.config.application.api.DeployLogger; import com.yahoo.searchdefinition.document.RankType; import com.yahoo.searchdefinition.document.SDField; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import java.util.List; import java.util.logging.Level; @@ -18,8 +18,8 @@ import java.util.logging.Level; */ public class RankTypeResolver extends MultiFieldResolver { - public RankTypeResolver(String indexName, List<SDField> fields, Search search, DeployLogger logger) { - super(indexName, fields, search, logger); + public RankTypeResolver(String indexName, List<SDField> fields, Schema schema, DeployLogger logger) { + super(indexName, fields, schema, logger); } public void resolve() { diff --git a/config-model/src/main/java/com/yahoo/searchdefinition/processing/multifieldresolver/StemmingResolver.java b/config-model/src/main/java/com/yahoo/searchdefinition/processing/multifieldresolver/StemmingResolver.java index 72942d651d0..1bcf646c8ec 100644 --- a/config-model/src/main/java/com/yahoo/searchdefinition/processing/multifieldresolver/StemmingResolver.java +++ b/config-model/src/main/java/com/yahoo/searchdefinition/processing/multifieldresolver/StemmingResolver.java @@ -2,7 +2,7 @@ package com.yahoo.searchdefinition.processing.multifieldresolver; import com.yahoo.config.application.api.DeployLogger; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.document.SDField; import com.yahoo.searchdefinition.document.Stemming; @@ -15,8 +15,8 @@ import java.util.logging.Level; */ public class StemmingResolver extends MultiFieldResolver { - public StemmingResolver(String indexName, List<SDField> fields, Search search, DeployLogger logger) { - super(indexName, fields, search, logger); + public StemmingResolver(String indexName, List<SDField> fields, Schema schema, DeployLogger logger) { + super(indexName, fields, schema, logger); } @Override @@ -29,13 +29,13 @@ public class StemmingResolver extends MultiFieldResolver { SDField stemmingField = null; for (SDField field : fields) { if (stemming == null && stemmingField==null) { - stemming = field.getStemming(search); + stemming = field.getStemming(schema); stemmingField = field; - } else if (stemming != field.getStemming(search)) { - deployLogger.logApplicationPackage(Level.WARNING, "Field '" + field.getName() + "' has " + field.getStemming(search) + - ", whereas field '" + stemmingField.getName() + "' has " + stemming + - ". All fields indexing to the index '" + indexName + "' must have the same stemming." + - " This should be corrected as it will make indexing fail in a few cases."); + } else if (stemming != field.getStemming(schema)) { + deployLogger.logApplicationPackage(Level.WARNING, "Field '" + field.getName() + "' has " + field.getStemming(schema) + + ", whereas field '" + stemmingField.getName() + "' has " + stemming + + ". All fields indexing to the index '" + indexName + "' must have the same stemming." + + " This should be corrected as it will make indexing fail in a few cases."); } } } diff --git a/config-model/src/main/java/com/yahoo/vespa/documentmodel/DocumentSummary.java b/config-model/src/main/java/com/yahoo/vespa/documentmodel/DocumentSummary.java index 23c1a48b6fe..967e662dc6b 100644 --- a/config-model/src/main/java/com/yahoo/vespa/documentmodel/DocumentSummary.java +++ b/config-model/src/main/java/com/yahoo/vespa/documentmodel/DocumentSummary.java @@ -2,7 +2,7 @@ package com.yahoo.vespa.documentmodel; import com.yahoo.config.application.api.DeployLogger; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import java.util.ArrayList; import java.util.Collection; @@ -21,10 +21,10 @@ public class DocumentSummary extends FieldView { private boolean omitSummaryFeatures = false; private String inherited; - private final Search owner; + private final Schema owner; /** Creates a DocumentSummary with the given name. */ - public DocumentSummary(String name, Search owner) { + public DocumentSummary(String name, Schema owner) { super(name); this.owner = owner; } @@ -55,13 +55,13 @@ public class DocumentSummary extends FieldView { } public SummaryField getSummaryField(String name) { - var parent = getInherited(); - if (parent != null) { - return parent.getSummaryField(name); - } - return (SummaryField) get(name); + var field = (SummaryField)get(name); + if (field != null) return field; + if (getInherited() == null) return null; + return getInherited().getSummaryField(name); } + // TODO: This does not handle overriding in child summaries correctly public Collection<SummaryField> getSummaryFields() { var fields = new ArrayList<SummaryField>(getFields().size()); var parent = getInherited(); @@ -109,11 +109,6 @@ public class DocumentSummary extends FieldView { return owner.getSummary(inherited); } - /** Returns the name of the summary this was declared to inherit, or null if not sett to inherit anything */ - public String getInheritedName() { - return inherited; - } - @Override public String toString() { return "document summary '" + getName() + "'"; diff --git a/config-model/src/main/java/com/yahoo/vespa/model/VespaModel.java b/config-model/src/main/java/com/yahoo/vespa/model/VespaModel.java index b5fc0e88540..92e1226ddf6 100644 --- a/config-model/src/main/java/com/yahoo/vespa/model/VespaModel.java +++ b/config-model/src/main/java/com/yahoo/vespa/model/VespaModel.java @@ -178,7 +178,7 @@ public final class VespaModel extends AbstractConfigProducerRoot implements Seri version = deployState.getVespaVersion(); fileRegistry = deployState.getFileRegistry(); largeRankExpressions = new LargeRankExpressions(deployState.getFileRegistry()); - rankingConstants = new RankingConstants(deployState.getFileRegistry()); + rankingConstants = new RankingConstants(deployState.getFileRegistry(), Optional.empty()); validationOverrides = deployState.validationOverrides(); applicationPackage = deployState.getApplicationPackage(); provisioned = deployState.provisioned(); @@ -189,7 +189,7 @@ public final class VespaModel extends AbstractConfigProducerRoot implements Seri rankProfileList = new RankProfileList(null, // null search -> global rankingConstants, largeRankExpressions, - new OnnxModels(deployState.getFileRegistry()), + new OnnxModels(deployState.getFileRegistry(), Optional.empty()), AttributeFields.empty, deployState.rankProfileRegistry(), deployState.getQueryProfiles().getRegistry(), @@ -336,7 +336,7 @@ public final class VespaModel extends AbstractConfigProducerRoot implements Seri } private OnnxModels onnxModelInfoFromSource(ImportedMlModel model) { - OnnxModels onnxModels = new OnnxModels(fileRegistry); + OnnxModels onnxModels = new OnnxModels(fileRegistry, Optional.empty()); if (model.modelType().equals(ImportedMlModel.ModelType.ONNX)) { String path = model.source(); String applicationPath = this.applicationPackage.getFileReference(Path.fromString("")).toString(); @@ -350,7 +350,7 @@ public final class VespaModel extends AbstractConfigProducerRoot implements Seri private OnnxModels onnxModelInfoFromStore(String modelName) { String path = ApplicationPackage.MODELS_DIR.append(modelName + ".onnx").toString(); - OnnxModels onnxModels = new OnnxModels(fileRegistry); + OnnxModels onnxModels = new OnnxModels(fileRegistry, Optional.empty()); loadOnnxModelInfo(onnxModels, modelName, path); return onnxModels; } diff --git a/config-model/src/main/java/com/yahoo/vespa/model/application/validation/ComplexAttributeFieldsValidator.java b/config-model/src/main/java/com/yahoo/vespa/model/application/validation/ComplexAttributeFieldsValidator.java index cb9472568e5..43bf8133c74 100644 --- a/config-model/src/main/java/com/yahoo/vespa/model/application/validation/ComplexAttributeFieldsValidator.java +++ b/config-model/src/main/java/com/yahoo/vespa/model/application/validation/ComplexAttributeFieldsValidator.java @@ -4,7 +4,7 @@ package com.yahoo.vespa.model.application.validation; import com.yahoo.config.model.deploy.DeployState; import com.yahoo.document.DataType; import com.yahoo.document.PositionDataType; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.document.ComplexAttributeFieldUtils; import com.yahoo.searchdefinition.document.ImmutableSDField; import com.yahoo.vespa.model.VespaModel; @@ -39,18 +39,18 @@ public class ComplexAttributeFieldsValidator extends Validator { } } - private static void validateComplexFields(String clusterName, Search search) { - String unsupportedFields = search.allFields() - .filter(field -> isUnsupportedComplexField(field)) - .map(ComplexAttributeFieldsValidator::toString) - .collect(Collectors.joining(", ")); + private static void validateComplexFields(String clusterName, Schema schema) { + String unsupportedFields = schema.allFields() + .filter(field -> isUnsupportedComplexField(field)) + .map(ComplexAttributeFieldsValidator::toString) + .collect(Collectors.joining(", ")); if (!unsupportedFields.isEmpty()) { throw new IllegalArgumentException( String.format("For cluster '%s', search '%s': The following complex fields do not support using struct field attributes: %s. " + "Only supported for the following complex field types: array or map of struct with primitive types, map of primitive types. " + "The supported primitive types are: byte, int, long, float, double and string", - clusterName, search.getName(), unsupportedFields)); + clusterName, schema.getName(), unsupportedFields)); } } diff --git a/config-model/src/main/java/com/yahoo/vespa/model/application/validation/NoPrefixForIndexes.java b/config-model/src/main/java/com/yahoo/vespa/model/application/validation/NoPrefixForIndexes.java index c8558b70b6e..67667674ea9 100644 --- a/config-model/src/main/java/com/yahoo/vespa/model/application/validation/NoPrefixForIndexes.java +++ b/config-model/src/main/java/com/yahoo/vespa/model/application/validation/NoPrefixForIndexes.java @@ -2,10 +2,10 @@ package com.yahoo.vespa.model.application.validation; import com.yahoo.config.model.deploy.DeployState; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.document.ImmutableSDField; import com.yahoo.searchdefinition.document.Matching; import com.yahoo.searchdefinition.Index; -import com.yahoo.searchdefinition.Search; import com.yahoo.searchdefinition.derived.DerivedConfiguration; import com.yahoo.vespa.model.VespaModel; import com.yahoo.vespa.model.search.AbstractSearchCluster; @@ -28,16 +28,16 @@ public class NoPrefixForIndexes extends Validator { IndexedSearchCluster sc = (IndexedSearchCluster) cluster; for (DocumentDatabase docDb : sc.getDocumentDbs()) { DerivedConfiguration sdConfig = docDb.getDerivedConfiguration(); - Search search = sdConfig.getSearch(); - for (ImmutableSDField field : search.allConcreteFields()) { + Schema schema = sdConfig.getSearch(); + for (ImmutableSDField field : schema.allConcreteFields()) { if (field.doesIndexing()) { //if (!field.getIndexTo().isEmpty() && !field.getIndexTo().contains(field.getName())) continue; if (field.getMatching().getAlgorithm().equals(Matching.Algorithm.PREFIX)) { - failField(search, field); + failField(schema, field); } for (Map.Entry<String, Index> e : field.getIndices().entrySet()) { if (e.getValue().isPrefix()) { - failField(search, field); + failField(schema, field); } } } @@ -47,8 +47,8 @@ public class NoPrefixForIndexes extends Validator { } } - private void failField(Search search, ImmutableSDField field) { - throw new IllegalArgumentException("For search '" + search.getName() + "', field '" + field.getName() + + private void failField(Schema schema, ImmutableSDField field) { + throw new IllegalArgumentException("For " + schema + ", field '" + field.getName() + "': match/index:prefix is not supported for indexes."); } } diff --git a/config-model/src/main/java/com/yahoo/vespa/model/application/validation/RoutingSelectorValidator.java b/config-model/src/main/java/com/yahoo/vespa/model/application/validation/RoutingSelectorValidator.java index 8f9a604f65f..7112c4b9a0a 100644 --- a/config-model/src/main/java/com/yahoo/vespa/model/application/validation/RoutingSelectorValidator.java +++ b/config-model/src/main/java/com/yahoo/vespa/model/application/validation/RoutingSelectorValidator.java @@ -10,7 +10,6 @@ import com.yahoo.vespa.model.search.IndexedSearchCluster; /** * Validates routing selector for search and content clusters - * */ public class RoutingSelectorValidator extends Validator { @@ -30,4 +29,5 @@ public class RoutingSelectorValidator extends Validator { } } } + } diff --git a/config-model/src/main/java/com/yahoo/vespa/model/application/validation/change/search/IndexingScriptChangeMessageBuilder.java b/config-model/src/main/java/com/yahoo/vespa/model/application/validation/change/search/IndexingScriptChangeMessageBuilder.java index ba776bb13a1..ce8347b66c1 100644 --- a/config-model/src/main/java/com/yahoo/vespa/model/application/validation/change/search/IndexingScriptChangeMessageBuilder.java +++ b/config-model/src/main/java/com/yahoo/vespa/model/application/validation/change/search/IndexingScriptChangeMessageBuilder.java @@ -1,7 +1,7 @@ // Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. package com.yahoo.vespa.model.application.validation.change.search; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.document.ImmutableSDField; import com.yahoo.searchdefinition.document.Matching; import com.yahoo.searchdefinition.document.NormalizeLevel; @@ -18,16 +18,16 @@ import com.yahoo.vespa.documentmodel.SummaryTransform; */ public class IndexingScriptChangeMessageBuilder { - private final Search currentSearch; + private final Schema currentSchema; private final ImmutableSDField currentField; - private final Search nextSearch; + private final Schema nextSchema; private final ImmutableSDField nextField; - public IndexingScriptChangeMessageBuilder(Search currentSearch, ImmutableSDField currentField, - Search nextSearch, ImmutableSDField nextField) { - this.currentSearch = currentSearch; + public IndexingScriptChangeMessageBuilder(Schema currentSchema, ImmutableSDField currentField, + Schema nextSchema, ImmutableSDField nextField) { + this.currentSchema = currentSchema; this.currentField = currentField; - this.nextSearch = nextSearch; + this.nextSchema = nextSchema; this.nextField = nextField; } @@ -55,8 +55,8 @@ public class IndexingScriptChangeMessageBuilder { } private void checkStemming(ChangeMessageBuilder builder) { - Stemming currentStemming = currentField.getStemming(currentSearch); - Stemming nextStemming = nextField.getStemming(nextSearch); + Stemming currentStemming = currentField.getStemming(currentSchema); + Stemming nextStemming = nextField.getStemming(nextSchema); if (!currentStemming.equals(nextStemming)) { builder.addChange("stemming", currentStemming.getName(), nextStemming.getName()); } diff --git a/config-model/src/main/java/com/yahoo/vespa/model/application/validation/change/search/IndexingScriptChangeValidator.java b/config-model/src/main/java/com/yahoo/vespa/model/application/validation/change/search/IndexingScriptChangeValidator.java index 699a63cdd9f..e64a3d44bba 100644 --- a/config-model/src/main/java/com/yahoo/vespa/model/application/validation/change/search/IndexingScriptChangeValidator.java +++ b/config-model/src/main/java/com/yahoo/vespa/model/application/validation/change/search/IndexingScriptChangeValidator.java @@ -3,7 +3,7 @@ package com.yahoo.vespa.model.application.validation.change.search; import com.yahoo.config.application.api.ValidationId; import com.yahoo.config.provision.ClusterSpec; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.document.ImmutableSDField; import com.yahoo.vespa.indexinglanguage.ExpressionConverter; import com.yahoo.vespa.indexinglanguage.expressions.Expression; @@ -25,20 +25,20 @@ import java.util.Optional; public class IndexingScriptChangeValidator { private final ClusterSpec.Id id; - private final Search currentSearch; - private final Search nextSearch; + private final Schema currentSchema; + private final Schema nextSchema; - public IndexingScriptChangeValidator(ClusterSpec.Id id, Search currentSearch, Search nextSearch) { + public IndexingScriptChangeValidator(ClusterSpec.Id id, Schema currentSchema, Schema nextSchema) { this.id = id; - this.currentSearch = currentSearch; - this.nextSearch = nextSearch; + this.currentSchema = currentSchema; + this.nextSchema = nextSchema; } public List<VespaConfigChangeAction> validate() { List<VespaConfigChangeAction> result = new ArrayList<>(); - for (ImmutableSDField nextField : new LinkedHashSet<>(nextSearch.allConcreteFields())) { + for (ImmutableSDField nextField : new LinkedHashSet<>(nextSchema.allConcreteFields())) { String fieldName = nextField.getName(); - ImmutableSDField currentField = currentSearch.getConcreteField(fieldName); + ImmutableSDField currentField = currentSchema.getConcreteField(fieldName); if (currentField != null) { validateScripts(currentField, nextField).ifPresent(r -> result.add(r)); } @@ -57,7 +57,7 @@ public class IndexingScriptChangeValidator { ScriptExpression nextScript = nextField.getIndexingScript(); if ( ! equalScripts(currentScript, nextScript)) { ChangeMessageBuilder messageBuilder = new ChangeMessageBuilder(nextField.getName()); - new IndexingScriptChangeMessageBuilder(currentSearch, currentField, nextSearch, nextField).populate(messageBuilder); + new IndexingScriptChangeMessageBuilder(currentSchema, currentField, nextSchema, nextField).populate(messageBuilder); messageBuilder.addChange("indexing script", currentScript.toString(), nextScript.toString()); return Optional.of(VespaReindexAction.of(id, ValidationId.indexingChange, messageBuilder.build())); } diff --git a/config-model/src/main/java/com/yahoo/vespa/model/ml/OnnxModelInfo.java b/config-model/src/main/java/com/yahoo/vespa/model/ml/OnnxModelInfo.java index f0309cc630b..88139de7888 100644 --- a/config-model/src/main/java/com/yahoo/vespa/model/ml/OnnxModelInfo.java +++ b/config-model/src/main/java/com/yahoo/vespa/model/ml/OnnxModelInfo.java @@ -107,7 +107,7 @@ public class OnnxModelInfo { unboundSizes.add(size); if (unboundSizes.size() > 1) { throw new IllegalArgumentException("Found conflicting sizes for unbound dimension " + - "for type '" + onnxType + "'"); + "for type '" + onnxType + "'"); } // Handle dimensions with symbolic names @@ -115,7 +115,7 @@ public class OnnxModelInfo { String symbolicName = onnxType.dimensions().get(i).getSymbolicName(); if (symbolicSizes.containsKey(symbolicName) && ! symbolicSizes.get(symbolicName).equals(size)) { throw new IllegalArgumentException("Found conflicting sizes for symbolic dimension '" + - symbolicName + "' for input '" + onnxName + "'"); + symbolicName + "' for input '" + onnxName + "'"); } symbolicSizes.put(symbolicName, size); } @@ -131,7 +131,7 @@ public class OnnxModelInfo { if (app.getFile(generatedModelInfoPath(pathInApplicationPackage)).exists()) { return loadFromGeneratedInfo(pathInApplicationPackage, app); } - throw new IllegalArgumentException("Unable to find ONNX model file or generated ONNX info file"); + throw new IllegalArgumentException("Unable to find ONNX model '" + path + "'"); } static public boolean modelExists(String path, ApplicationPackage app) { diff --git a/config-model/src/main/java/com/yahoo/vespa/model/search/IndexedSearchCluster.java b/config-model/src/main/java/com/yahoo/vespa/model/search/IndexedSearchCluster.java index 28682e92f37..a68993be543 100644 --- a/config-model/src/main/java/com/yahoo/vespa/model/search/IndexedSearchCluster.java +++ b/config-model/src/main/java/com/yahoo/vespa/model/search/IndexedSearchCluster.java @@ -5,7 +5,8 @@ import com.yahoo.config.model.deploy.DeployState; import com.yahoo.config.model.producer.AbstractConfigProducer; import com.yahoo.prelude.fastsearch.DocumentdbInfoConfig; import com.yahoo.search.config.IndexInfoConfig; -import com.yahoo.searchdefinition.DocumentOnlySearch; +import com.yahoo.searchdefinition.DocumentOnlySchema; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.derived.DerivedConfiguration; import com.yahoo.vespa.config.search.AttributesConfig; import com.yahoo.vespa.config.search.DispatchConfig; @@ -195,10 +196,10 @@ public class IndexedSearchCluster extends SearchCluster @Override protected void deriveAllSchemas(List<SchemaSpec> localSearches, DeployState deployState) { for (SchemaSpec spec : localSearches) { - com.yahoo.searchdefinition.Search search = spec.getSearchDefinition().getSearch(); - if ( ! (search instanceof DocumentOnlySearch)) { - DocumentDatabase db = new DocumentDatabase(this, search.getName(), - new DerivedConfiguration(search, + Schema schema = spec.getSearchDefinition().getSearch(); + if ( ! (schema instanceof DocumentOnlySchema)) { + DocumentDatabase db = new DocumentDatabase(this, schema.getName(), + new DerivedConfiguration(schema, deployState.getDeployLogger(), deployState.getProperties(), deployState.rankProfileRegistry(), diff --git a/config-model/src/main/java/com/yahoo/vespa/model/search/NamedSchema.java b/config-model/src/main/java/com/yahoo/vespa/model/search/NamedSchema.java index 06f8198151f..334493b4a92 100644 --- a/config-model/src/main/java/com/yahoo/vespa/model/search/NamedSchema.java +++ b/config-model/src/main/java/com/yahoo/vespa/model/search/NamedSchema.java @@ -1,7 +1,7 @@ // Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. package com.yahoo.vespa.model.search; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import java.util.Collection; @@ -11,22 +11,22 @@ import java.util.Collection; // TODO: This class is quite pointless public class NamedSchema { - private final Search search; + private final Schema schema; private final String name; public static final String fileNameSuffix = ".sd"; - public Search getSearch() { - return search; + public Schema getSearch() { + return schema; } public String getName() { return name; } - public NamedSchema(String name, Search search) { + public NamedSchema(String name, Schema schema) { this.name = name; - this.search = search; + this.schema = schema; } //Find search definition from a collection with the name specified diff --git a/config-model/src/main/java/com/yahoo/vespa/model/search/StreamingSearchCluster.java b/config-model/src/main/java/com/yahoo/vespa/model/search/StreamingSearchCluster.java index 8dc711f50f6..a9f9102c11a 100644 --- a/config-model/src/main/java/com/yahoo/vespa/model/search/StreamingSearchCluster.java +++ b/config-model/src/main/java/com/yahoo/vespa/model/search/StreamingSearchCluster.java @@ -4,6 +4,7 @@ package com.yahoo.vespa.model.search; import com.yahoo.config.model.deploy.DeployState; import com.yahoo.config.model.producer.AbstractConfigProducer; import com.yahoo.prelude.fastsearch.DocumentdbInfoConfig; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.derived.AttributeFields; import com.yahoo.searchdefinition.derived.DerivedConfiguration; import com.yahoo.vespa.config.search.AttributesConfig; @@ -89,11 +90,11 @@ public class StreamingSearchCluster extends SearchCluster implements throw new IllegalStateException("Logical indexes are not supported: Got " + local.size() + " search definitions, expected 1"); } } - private void deriveSingleSearchDefinition(com.yahoo.searchdefinition.Search localSearch, DeployState deployState) { - if (!localSearch.getName().equals(docTypeName)) { - throw new IllegalStateException("Mismatch between document type name (" + docTypeName + ") and name of search definition (" + localSearch.getName() + ")"); + private void deriveSingleSearchDefinition(Schema localSchema, DeployState deployState) { + if (!localSchema.getName().equals(docTypeName)) { + throw new IllegalStateException("Mismatch between document type name (" + docTypeName + ") and name of search definition (" + localSchema.getName() + ")"); } - this.sdConfig = new DerivedConfiguration(localSearch, deployState.getDeployLogger(), + this.sdConfig = new DerivedConfiguration(localSchema, deployState.getDeployLogger(), deployState.getProperties(), deployState.rankProfileRegistry(), deployState.getQueryProfiles().getRegistry(), diff --git a/config-model/src/main/javacc/SDParser.jj b/config-model/src/main/javacc/SDParser.jj index 19bea494347..0f60db40069 100644 --- a/config-model/src/main/javacc/SDParser.jj +++ b/config-model/src/main/javacc/SDParser.jj @@ -28,6 +28,7 @@ import com.yahoo.document.*; import com.yahoo.documentmodel.*; import com.yahoo.compress.Compressor; import com.yahoo.compress.CompressionType; +import com.yahoo.searchdefinition.Application; import com.yahoo.searchdefinition.document.*; import com.yahoo.searchdefinition.document.annotation.SDAnnotationType; import com.yahoo.searchdefinition.document.annotation.TemporaryAnnotationReferenceDataType; @@ -40,8 +41,8 @@ import com.yahoo.searchdefinition.DefaultRankProfile; import com.yahoo.searchdefinition.RankProfileRegistry; import com.yahoo.searchdefinition.RankProfile.MatchPhaseSettings; import com.yahoo.searchdefinition.RankProfile.DiversitySettings; -import com.yahoo.searchdefinition.Search; -import com.yahoo.searchdefinition.DocumentOnlySearch; +import com.yahoo.searchdefinition.Schema; +import com.yahoo.searchdefinition.DocumentOnlySchema; import com.yahoo.searchdefinition.UnrankedRankProfile; import com.yahoo.searchdefinition.fieldoperation.*; import com.yahoo.searchlib.rankingexpression.FeatureList; @@ -61,6 +62,7 @@ import com.yahoo.language.Linguistics; import com.yahoo.language.process.Embedder; import com.yahoo.language.simple.SimpleLinguistics; import com.yahoo.search.query.ranking.Diversity; +import java.util.Optional; import java.util.Map; import java.util.List; import java.util.ArrayList; @@ -68,15 +70,15 @@ import java.util.LinkedHashMap; import java.util.logging.Level; /** - * A search definition parser + * The schema parser * * @author bratseth */ - @SuppressWarnings("deprecation") +@SuppressWarnings("deprecation") public class SDParser { private DocumentTypeManager docMan = null; - private ApplicationPackage app; + private Application application; private FileRegistry fileRegistry; private DeployLogger deployLogger; private ModelContext.Properties properties; @@ -86,20 +88,20 @@ public class SDParser { /** * Creates a parser * - * @param documentsOnly true to only parse the document aspect of a search definition (e.g skip rank profiles) + * @param documentsOnly true to only parse the document aspect of a schema (e.g skip rank profiles) */ public SDParser(SimpleCharStream stream, FileRegistry fileRegistry, DeployLogger deployLogger, ModelContext.Properties properties, - ApplicationPackage applicationPackage, + Application application, RankProfileRegistry rankProfileRegistry, boolean documentsOnly) { this(stream); this.fileRegistry = fileRegistry; this.deployLogger = deployLogger; this.properties = properties; - this.app = applicationPackage; + this.application = application; this.rankProfileRegistry = rankProfileRegistry; this.documentsOnly = documentsOnly; } @@ -411,20 +413,20 @@ SPECIAL_TOKEN : // -------------------------------------------------------------------------------- /** - * The rule consumes any search definition and returns the corresponding object. This is the only production that should + * The rule consumes any schema and returns the corresponding object. This is the only production that should * ever consume leading newlines. * - * @param dir The directory containing the file being parsed. - * @return The search definition object. + * @param dir the directory containing the file being parsed + * @return the schema object */ -Search search(DocumentTypeManager docMan, String dir) : +Schema schema(DocumentTypeManager docMan, String dir) : { this.docMan = docMan; - Search search; + Schema schema; } { - (<NL>)* (search = rootSchema(dir) | search = rootDocument(dir)) - { return search; } + (<NL>)* (schema = rootSchema(dir) | schema = rootDocument(dir)) + { return schema; } } /** @@ -434,43 +436,44 @@ Search search(DocumentTypeManager docMan, String dir) : * @param dir the directory containing the file being parsed. * @return the schema definition object. */ -Search rootSchema(String dir) : +Schema rootSchema(String dir) : { String name; - Search search; + String inherited = null; + Schema schema; } { - ( ( <SCHEMA> | <SEARCH> ) name = identifier() { - search = new Search(name, app, fileRegistry,deployLogger, properties); - rankProfileRegistry.add(new DefaultRankProfile(search, rankProfileRegistry, search.rankingConstants())); - rankProfileRegistry.add(new UnrankedRankProfile(search, rankProfileRegistry, search.rankingConstants()));} - lbrace() (rootSchemaItem(search) (<NL>)*)* <RBRACE> (<NL>)* <EOF>) - { return search; } + ( ( <SCHEMA> | <SEARCH> ) name = identifier() (<INHERITS> inherited = identifier() )? { + schema = new Schema(name, Optional.ofNullable(inherited), application, fileRegistry, deployLogger, properties); + rankProfileRegistry.add(new DefaultRankProfile(schema, rankProfileRegistry, schema.rankingConstants())); + rankProfileRegistry.add(new UnrankedRankProfile(schema, rankProfileRegistry, schema.rankingConstants()));} + lbrace() (rootSchemaItem(schema) (<NL>)*)* <RBRACE> (<NL>)* <EOF>) + { return schema; } } /** * Consumes an element of a schema block. This and rootSearch() are the only rules that should ever consume * trailing newline tokens. * - * @param search The search object to modify. + * @param schema the schema object to modify. * @return Null. */ -Object rootSchemaItem(Search search) : { } -{ - ( document(search) - | rawAsBase64(search) - | documentSummary(search) - | field(null, search) - | index(search, null) - | rankingConstant(search) - | rankProfile(search) - | searchStemming(search) - | useDocument(search) - | structOutside(search) - | annotationOutside(search) - | fieldSet(search) - | importField(search) - | onnxModel(search) ) +Object rootSchemaItem(Schema schema) : { } +{ + ( document(schema) + | rawAsBase64(schema) + | documentSummary(schema) + | field(null, schema) + | index(schema, null) + | rankingConstant(schema) + | rankProfile(schema) + | searchStemming(schema) + | useDocument(schema) + | structOutside(schema) + | annotationOutside(schema) + | fieldSet(schema) + | importField(schema) + | onnxModel(schema) ) { return null; } } @@ -480,99 +483,99 @@ Object rootSchemaItem(Search search) : { } * @param dir the directory containing the file being parsed. * @return the schema definition object. */ -Search rootDocument(String dir) : +Schema rootDocument(String dir) : { - Search search = new DocumentOnlySearch(app, fileRegistry, deployLogger, properties); + Schema schema = new DocumentOnlySchema(application, fileRegistry, deployLogger, properties); } { - ( (rootDocumentItem(search) (<NL>)*)*<EOF> ) - { return search; } + ( (rootDocumentItem(schema) (<NL>)*)*<EOF> ) + { return schema; } } /** * Consumes a single item from within a root document node. * - * @param search The search object to modify. + * @param schema the schema object to modify. * @return Null. */ -Object rootDocumentItem(Search search) : { } +Object rootDocumentItem(Schema schema) : { } { - ( namedDocument(search) ) + ( namedDocument(schema) ) { return null; } } /** * Consumes a use-document statement. This currently does nothing. * - * @param search the search object to modify. + * @param schema the schema object to modify. */ -void useDocument(Search search) : { } +void useDocument(Schema schema) : { } { <USEDOCUMENT> <COLON> identifier() } /** - * Consumes a document element. The name defaults to the search's name, but may be set. + * Consumes a document element. The name defaults to the schema's name, but may be set. * - * @param search the search object to add content to. + * @param schema the schema object to add content to. */ -void document(Search search) : +void document(Schema schema) : { - String name=search.getName(); + String name = schema.getName(); SDDocumentType document; } { - ( <DOCUMENT> (name = identifier())? (<NL>)* { document = new SDDocumentType(name, search); } + ( <DOCUMENT> (name = identifier())? (<NL>)* { document = new SDDocumentType(name, schema); } [ inheritsDocument(document) (<NL>)* ] - <LBRACE> (<NL>)* (documentBody(document, search) (<NL>)*)* <RBRACE> ) + <LBRACE> (<NL>)* (documentBody(document, schema) (<NL>)*)* <RBRACE> ) { - search.addDocument(document); + schema.addDocument(document); } } /** * Consumes a document element, explicitly named * - * @param search the search object to add content to. + * @param schema the schema object to add content to */ -void namedDocument(Search search) : +void namedDocument(Schema schema) : { String name; SDDocumentType document; } { - ( <DOCUMENT> name = identifier() (<NL>)* { document = new SDDocumentType(name, search); } + ( <DOCUMENT> name = identifier() (<NL>)* { document = new SDDocumentType(name, schema); } [ inheritsDocument(document) (<NL>)* ] - <LBRACE> (<NL>)* (documentBody(document, search) (<NL>)*)* <RBRACE> ) + <LBRACE> (<NL>)* (documentBody(document, schema) (<NL>)*)* <RBRACE> ) { - search.addDocument(document); + schema.addDocument(document); } } /** * Consumes a document body block * - * @param document The document type to modify. - * @param search The search object to add content to. + * @param document the document type to modify. + * @param schema the schema object to add content to * @return Null. */ -Object documentBody(SDDocumentType document, Search search) : +Object documentBody(SDDocumentType document, Schema schema) : { } { - ( annotation(search, document) + ( annotation(schema, document) | compression(document, null) | headercfg(document) | bodycfg(document) - | structInside(document, search) - | field(document, search) ) + | structInside(document, schema) + | field(document, schema) ) { return null; } } -void rawAsBase64(Search search) : +void rawAsBase64(Schema schema) : {} { - <RAW_AS_BASE64_IN_SUMMARY> { search.enableRawAsBase64(); } + <RAW_AS_BASE64_IN_SUMMARY> { schema.enableRawAsBase64(); } } /** @@ -651,10 +654,10 @@ void inheritsDocument(SDDocumentType document) : /** * Consumes a field block from within a document element. * - * @param document The document type to modify. - * @param search The search object to add content to. + * @param document the document type to modify + * @param schema the schema object to add content to */ -void field(SDDocumentType document, Search search) : +void field(SDDocumentType document, Schema schema) : { String name; SDField field; @@ -663,22 +666,22 @@ void field(SDDocumentType document, Search search) : { <FIELD> name = identifier() <TYPE> type = dataType() { - if (name != null && com.yahoo.searchdefinition.Search.isReservedName(name.toLowerCase())) { + if (name != null && Schema.isReservedName(name.toLowerCase())) { throw new IllegalArgumentException("Reserved name '" + name + "' can not be used as a field name."); } field = new TemporarySDField(name, type, document); } - lbrace() (fieldBody(field, search, document) (<NL>)*)* <RBRACE> + lbrace() (fieldBody(field, schema, document) (<NL>)*)* <RBRACE> { if (document != null) { document.addField(field); } else { - search.addExtraField(field); + schema.addExtraField(field); } } } -void fieldSet(Search search) : +void fieldSet(Schema schema) : { String setName; String field; @@ -690,8 +693,8 @@ void fieldSet(Search search) : { <FIELDSET> setName = identifier() lbrace() (( - ( <FIELDS><COLON> field = identifier() { search.fieldSets().addUserFieldSetItem(setName, field); } - ( <COMMA> field = identifier() { search.fieldSets().addUserFieldSetItem(setName, field); } )* ) + ( <FIELDS><COLON> field = identifier() { schema.fieldSets().addUserFieldSetItem(setName, field); } + ( <COMMA> field = identifier() { schema.fieldSets().addUserFieldSetItem(setName, field); } )* ) | ( <QUERYCOMMAND> <COLON> (queryCommand = identifierWithDash() | queryCommand = quotedString())) { queryCommands.add(queryCommand); } | @@ -702,21 +705,21 @@ void fieldSet(Search search) : // Apply settings after parsing since all user field items must be set first for (Object command : queryCommands) - search.fieldSets().userFieldSets().get(setName).queryCommands().add((String)command); + schema.fieldSets().userFieldSets().get(setName).queryCommands().add((String)command); for (Object setting : matchSettings) { ((SDField)setting).applyOperations(); - search.fieldSets().userFieldSets().get(setName).setMatching(((SDField)setting).getMatching()); + schema.fieldSets().userFieldSets().get(setName).setMatching(((SDField)setting).getMatching()); } } } /** - * This rule consumes a annotation block from within either a document element or a search element. + * This rule consumes a annotation block from within either a document element or a schema element. - * @param search the search object to add content to. + * @param schema the schema object to add content to */ -void annotationOutside(Search search) : +void annotationOutside(Schema schema) : { String name; SDAnnotationType type; @@ -727,19 +730,19 @@ void annotationOutside(Search search) : type = new SDAnnotationType(name.trim()); } [ inheritsAnnotation(type) (<NL>)* ] - lbrace() (type = annotationBody(search, type)) <RBRACE> + lbrace() (type = annotationBody(schema, type)) <RBRACE> { - if (search.getDocument()==null) throw new IllegalArgumentException("Can't add annotation '"+name+"' to a document type, define a document type first or declare the annotation inside of one."); - search.addAnnotation(type); + if (schema.getDocument()==null) throw new IllegalArgumentException("Can't add annotation '"+name+"' to a document type, define a document type first or declare the annotation inside of one."); + schema.addAnnotation(type); } } /** - * This rule consumes a annotation block from within either a document element. + * Consumes a annotation block from within either a document element. * - * @param document The document object to add content to. + * @param document the document object to add content to */ -void annotation(Search search, SDDocumentType document) : +void annotation(Schema schema, SDDocumentType document) : { String name; SDAnnotationType type; @@ -750,7 +753,7 @@ void annotation(Search search, SDDocumentType document) : type = new SDAnnotationType(name.trim()); } [ inheritsAnnotation(type) (<NL>)* ] - lbrace() (type = annotationBody(search, type)) <RBRACE> + lbrace() (type = annotationBody(schema, type)) <RBRACE> { document.addAnnotation(type); } @@ -758,15 +761,15 @@ void annotation(Search search, SDDocumentType document) : /** - * This rule consumes a single element of an annotation body block. + * Consumes a single element of an annotation body block. * - * @param search The search object to add content to. - * @param type The type being built. + * @param schema the schema object to add content to + * @param type the type being built * @return a modified or new AnnotationType instance */ -SDAnnotationType annotationBody(Search search, SDAnnotationType type) : +SDAnnotationType annotationBody(Schema schema, SDAnnotationType type) : { - SDDocumentType struct = new SDDocumentType("annotation." + type.getName(), search); + SDDocumentType struct = new SDDocumentType("annotation." + type.getName(), schema); } { (structFieldDefinition(struct) (<NL>)*)* @@ -791,15 +794,15 @@ void inheritsAnnotation(SDAnnotationType annotation) : /** * This rule consumes a struct block from within a document element. * - * @param search The search object to add content to. + * @param schema the schema object to add content to */ -void structInside(SDDocumentType document, Search search) : +void structInside(SDDocumentType document, Schema schema) : { SDDocumentType struct; } { ( - struct = structDefinition(search, document) + struct = structDefinition(schema, document) ) { document.addType(struct); @@ -809,27 +812,27 @@ void structInside(SDDocumentType document, Search search) : /** * This rule consumes a struct block from within a document element. * - * @param search The search object to add content to. + * @param schema the schema object to add content to */ -void structOutside(Search search) : +void structOutside(Schema schema) : { SDDocumentType struct; } { ( - struct = structDefinition(search, search.getDocument()) + struct = structDefinition(schema, schema.getDocument()) ) { - search.addType(struct); + schema.addType(struct); } } /** * This rule consumes a struct block from within a document element. * - * @param search The search object to add content to. + * @param schema the schema object to add content to */ -SDDocumentType structDefinition(Search search, SDDocumentType repo) : +SDDocumentType structDefinition(Schema schema, SDDocumentType repo) : { String name; SDDocumentType struct; @@ -837,7 +840,7 @@ SDDocumentType structDefinition(Search search, SDDocumentType repo) : { <STRUCT> name = identifier() { - struct = new SDDocumentType(name, search); + struct = new SDDocumentType(name, schema); } lbrace() (structFieldDefinition(struct) (<NL>)*)* <RBRACE> { @@ -860,7 +863,7 @@ SDDocumentType structDefinition(Search search, SDDocumentType repo) : /** * This rule consumes a data type block from within a field element. * - * @return The consumed data type. + * @return the consumed data type */ DataType dataType() : { @@ -957,7 +960,7 @@ void structFieldDefinition(SDDocumentType struct) : } { <FIELD> name = identifier() <TYPE> type = dataType() { - if (name != null && com.yahoo.searchdefinition.Search.isReservedName(name.toLowerCase())) { + if (name != null && Schema.isReservedName(name.toLowerCase())) { throw new IllegalArgumentException("Reserved name '" + name + "' can not be used as a field name."); } field = new TemporarySDField(name, type, struct); @@ -971,36 +974,36 @@ void structFieldDefinition(SDDocumentType struct) : * This rule consumes a struct subfield from a document field body. This is not to be confused with a document * struct's fields, but rather this is a subfield of a document field of type struct. * - * @param field The field to modify. - * @param search The search object to add content to. - * @param document The document type to modify. + * @param field the field to modify + * @param schema the schema object to add content to + * @param document the document type to modify */ -void structField(FieldOperationContainer field, Search search,SDDocumentType document) : +void structField(FieldOperationContainer field, Schema schema, SDDocumentType document) : { String name; SDField structField; } { <STRUCTFIELD> name = identifier() { - if (name != null && com.yahoo.searchdefinition.Search.isReservedName(name.toLowerCase())) { + if (name != null && Schema.isReservedName(name.toLowerCase())) { throw new IllegalArgumentException("Reserved name '" + name + "' can not be used as a field name."); } FieldOperationContainer structFieldOp = new StructFieldOperation(name); field.addOperation((StructFieldOperation) structFieldOp); } - lbrace() (structFieldBody(structFieldOp, search, document) (<NL>)*)* <RBRACE> + lbrace() (structFieldBody(structFieldOp, schema, document) (<NL>)*)* <RBRACE> } /** * This rule consumes a single element of a field body block. * - * @param field The field being built. - * @param search The search object to add content to. - * @param document The owning document, or null if this is a search field. - * @return Null. + * @param field the field being built + * @param schema the schema object to add content to + * @param document the owning document, or null if this is a search field + * @return Null */ -String fieldBody(SDField field, Search search, SDDocumentType document) : { } +String fieldBody(SDField field, Schema schema, SDDocumentType document) : { } { ( alias(field) | attribute(field) | @@ -1011,7 +1014,7 @@ String fieldBody(SDField field, Search search, SDDocumentType document) : { } header(field) | id(field, document) | summaryInField(field) | - index(search, field) | + index(schema, field) | indexing(field) | indexingRewrite(field) | match(field) | @@ -1020,7 +1023,7 @@ String fieldBody(SDField field, Search search, SDDocumentType document) : { } rank(field) | rankType(field) | sorting(field, field.getName()) | - structField(field, search, document) | + structField(field, schema, document) | summaryTo(field) | weight(field) | weightedset(field) ) @@ -1029,21 +1032,21 @@ String fieldBody(SDField field, Search search, SDDocumentType document) : { } /** * This rule consumes a single element of a struct subfield body block. - * Only elements that are supported in streaming search and indexed search (with complex attributes) are allowed. + * Only elements that are supported in streaming schema and indexed schema (with complex attributes) are allowed. * - * @param field The field being built. - * @param search The search object to add content to. - * @param document The owning document, or null if this is a search field. - * @return Null. + * @param field the field being built + * @param schema the schema object to add content to + * @param document the owning document, or null if this is a schema field + * @return Null */ -String structFieldBody(FieldOperationContainer field, Search search, SDDocumentType document) : { } +String structFieldBody(FieldOperationContainer field, Schema schema, SDDocumentType document) : { } { ( summaryInField(field) | indexing(field) | attribute(field) | match(field) | queryCommand(field) | - structField(field, search, document) | + structField(field, schema, document) | summaryTo(field) ) { return null; } } @@ -1494,17 +1497,17 @@ void fieldStemming(FieldOperationContainer field) : } /** - * This rule consumes a stemming statement for a search element. + * This rule consumes a stemming statement for a schema element. * - * @param search The search to modify. + * @param schema the schema to modify */ -void searchStemming(Search search) : +void searchStemming(Schema schema) : { String setting; } { <STEMMING> <COLON> setting = identifierWithDash() - { search.setStemming(Stemming.get(setting)); } + { schema.setStemming(Stemming.get(setting)); } } /** @@ -1764,20 +1767,20 @@ void indexingRewrite(FieldOperationContainer field) : { } } /** - * Consumes a document-summary block from within a search block. + * Consumes a document-summary block from within a schema block. * - * @param search The search object to add content to. - * @return Null. + * @param schema the schema object to add content to + * @return Null */ -Object documentSummary(Search search) : +Object documentSummary(Schema schema) : { String name; DocumentSummary summary; } { ( <DOCUMENTSUMMARY> - name = identifierWithDash() { search.addSummary(summary = new DocumentSummary(name, search)); } - [inheritsDocumentSummary(summary, search)] + name = identifierWithDash() { schema.addSummary(summary = new DocumentSummary(name, schema)); } + [inheritsDocumentSummary(summary, schema)] lbrace() ( <FROMDISK> { summary.setFromDisk(true); } | @@ -1793,10 +1796,10 @@ Object documentSummary(Search search) : /** * This rule consumes an inherits statement of a document summary. * - * @param documentSummary The document summary to modify. - * @param search The search object documentSummary is being added to. + * @param documentSummary the document summary to modify + * @param schema the schema object documentSummary is being added to */ -void inheritsDocumentSummary(DocumentSummary documentSummary, Search search) : +void inheritsDocumentSummary(DocumentSummary documentSummary, Schema schema) : { String name; } @@ -1822,11 +1825,11 @@ Object documentSummaryItem(DocumentSummary summary) : { } /** * Consumes an index block for a field element. * - * @param search The search object to add content to. - * @param field The field to modify. - * @return Null. + * @param schema the schema object to add content to + * @param field the field to modify + * @return Null */ -Object index(Search search, FieldOperationContainer field) : +Object index(Schema schema, FieldOperationContainer field) : { IndexOperation op = new IndexOperation(); String indexName = (field != null) ? field.getName() : null; @@ -1846,7 +1849,7 @@ Object index(Search search, FieldOperationContainer field) : Index index = new Index(indexName); op.applyToIndex(index); - search.addIndex(index); + schema.addIndex(index); } else { field.addOperation(op); } @@ -1906,11 +1909,11 @@ void hnswIndexBody(HnswIndexParams.Builder params) : } /** - * Consumes a onnx-model block of a search element. + * Consumes a onnx-model block of a schema element. * - * @param search The search object to add content to. + * @param schema the schema object to add content to. */ -void onnxModel(Search search) : +void onnxModel(Schema schema) : { String name; OnnxModel onnxModel; @@ -1923,7 +1926,7 @@ void onnxModel(Search search) : lbrace() (onnxModelItem(onnxModel) (<NL>)*)+ <RBRACE> ) { if (documentsOnly) return; - search.onnxModels().add(onnxModel); + schema.onnxModels().add(onnxModel); } } @@ -1960,11 +1963,11 @@ Object onnxModelItem(OnnxModel onnxModel) : } /** - * Consumes a constant block of a search element. + * Consumes a constant block of a schema element. * - * @param search The search object to add content to. + * @param schema the schema object to add content to */ -void rankingConstant(Search search) : +void rankingConstant(Schema schema) : { String name; RankingConstant constant; @@ -1977,7 +1980,7 @@ void rankingConstant(Search search) : lbrace() (rankingConstantItem(constant) (<NL>)*)+ <RBRACE> ) { if (documentsOnly) return; - search.rankingConstants().add(constant); + schema.rankingConstants().add(constant); } } @@ -2020,11 +2023,11 @@ String uriPath() : { } } /** - * Consumes a rank-profile block of a search element. + * Consumes a rank-profile block of a schema element. * - * @param search The search object to add content to. + * @param schema the schema object to add content to */ -void rankProfile(Search search) : +void rankProfile(Schema schema) : { String name; RankProfile profile; @@ -2033,12 +2036,12 @@ void rankProfile(Search search) : ( ( <MODEL> | <RANKPROFILE> ) name = identifierWithDash() { if (documentsOnly) { - profile = new DocumentsOnlyRankProfile(name, search, rankProfileRegistry, search.rankingConstants()); + profile = new DocumentsOnlyRankProfile(name, schema, rankProfileRegistry, schema.rankingConstants()); } else if ("default".equals(name)) { - profile = rankProfileRegistry.get(search, "default"); + profile = rankProfileRegistry.get(schema, "default"); } else { - profile = new RankProfile(name, search, rankProfileRegistry, search.rankingConstants()); + profile = new RankProfile(name, schema, rankProfileRegistry, schema.rankingConstants()); } } [inheritsRankProfile(profile)] @@ -2368,7 +2371,7 @@ void minHitsPerThread(RankProfile profile) : /** * This rule consumes a num-search-partitions statement for a rank profile. * - * @param profile The rank profile to modify. + * @param profile the rank profile to modify */ void numSearchPartitions(RankProfile profile) : { @@ -2381,7 +2384,7 @@ void numSearchPartitions(RankProfile profile) : /** * This rule consumes a num-threads-per-search statement for a rank profile. * - * @param profile The rank profile to modify. + * @param profile the rank profile to modify */ void termwiseLimit(RankProfile profile) : { @@ -2394,7 +2397,7 @@ void termwiseLimit(RankProfile profile) : * This rule consumes a rank-properties block of a rank profile. There is a little trick within this rule to allow the * final rank property to skip the terminating newline token. * - * @param profile The rank profile to modify. + * @param profile the rank profile to modify */ void rankProperties(RankProfile profile) : { } { @@ -2405,7 +2408,7 @@ void rankProperties(RankProfile profile) : { } /** * This rule consumes a single rank property pair for a rank profile. * - * @param profile The rank profile to modify. + * @param profile the rank profile to modify */ void rankProperty(RankProfile profile) : { @@ -2630,7 +2633,7 @@ TensorType tensorType(String errorMessage) : } } -void importField(Search search) : +void importField(Schema schema) : { String fieldRefSpec; String aliasFieldName; @@ -2646,9 +2649,9 @@ void importField(Search search) : int indexOfDot = fieldRefSpec.indexOf('.'); String documentReferenceFieldName = fieldRefSpec.substring(0, indexOfDot); String foreignFieldName = fieldRefSpec.substring(indexOfDot + 1); - TemporaryImportedFields importedFields = search.temporaryImportedFields().get(); + TemporaryImportedFields importedFields = schema.temporaryImportedFields().get(); if (importedFields.hasField(aliasFieldName)) { - throw new IllegalArgumentException("For search '" + search.getName() + "', import field as '" + aliasFieldName + "': Field already imported"); + throw new IllegalArgumentException("For schema '" + schema.getName() + "', import field as '" + aliasFieldName + "': Field already imported"); } importedFields.add(new TemporaryImportedField(aliasFieldName, documentReferenceFieldName, foreignFieldName)); } diff --git a/config-model/src/test/derived/annotationsreference/annotationsreference.sd b/config-model/src/test/derived/annotationsreference/annotationsreference.sd index 11972c13c39..a0e2af3c6f5 100755 --- a/config-model/src/test/derived/annotationsreference/annotationsreference.sd +++ b/config-model/src/test/derived/annotationsreference/annotationsreference.sd @@ -3,24 +3,25 @@ search annotationsreference { document annotationsreference { - annotation banana { - field brand type string { } - } + annotation banana { + field brand type string { } + } - annotation food { - field what type annotationreference<banana> { } - } + annotation food { + field what type annotationreference<banana> { } + } - annotation cyclic { - field blah type annotationreference<cyclic> { } - } + annotation cyclic { + field blah type annotationreference<cyclic> { } + } - annotation a { - field foo type annotationreference<b> { } - } + annotation a { + field foo type annotationreference<b> { } + } - annotation b { - } + annotation b { + } } + } diff --git a/config-model/src/test/derived/lowercase/lowercase.sd b/config-model/src/test/derived/lowercase/lowercase.sd index de8c7ad729e..80edc837830 100644 --- a/config-model/src/test/derived/lowercase/lowercase.sd +++ b/config-model/src/test/derived/lowercase/lowercase.sd @@ -1,3 +1,4 @@ +# Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. schema lowercase { document lowercase { @@ -18,4 +19,4 @@ schema lowercase { indexing: input array_field_source | for_each { lowercase } | summary | index } -}
\ No newline at end of file +} diff --git a/config-model/src/test/derived/schemainheritance/attributes.cfg b/config-model/src/test/derived/schemainheritance/attributes.cfg new file mode 100644 index 00000000000..ad48742a20f --- /dev/null +++ b/config-model/src/test/derived/schemainheritance/attributes.cfg @@ -0,0 +1,155 @@ +attribute[].name "importedschema_ref" +attribute[].datatype REFERENCE +attribute[].collectiontype SINGLE +attribute[].dictionary.type BTREE +attribute[].dictionary.match UNCASED +attribute[].match UNCASED +attribute[].removeifzero false +attribute[].createifnonexistent false +attribute[].fastsearch false +attribute[].huge false +attribute[].paged false +attribute[].ismutable false +attribute[].sortascending true +attribute[].sortfunction UCA +attribute[].sortstrength PRIMARY +attribute[].sortlocale "" +attribute[].enablebitvectors false +attribute[].enableonlybitvector false +attribute[].fastaccess false +attribute[].arity 8 +attribute[].lowerbound -9223372036854775808 +attribute[].upperbound 9223372036854775807 +attribute[].densepostinglistthreshold 0.4 +attribute[].tensortype "" +attribute[].imported false +attribute[].distancemetric EUCLIDEAN +attribute[].index.hnsw.enabled false +attribute[].index.hnsw.maxlinkspernode 16 +attribute[].index.hnsw.neighborstoexploreatinsert 200 +attribute[].index.hnsw.distancemetric EUCLIDEAN +attribute[].index.hnsw.multithreadedindexing true +attribute[].name "child_field" +attribute[].datatype STRING +attribute[].collectiontype SINGLE +attribute[].dictionary.type BTREE +attribute[].dictionary.match UNCASED +attribute[].match UNCASED +attribute[].removeifzero false +attribute[].createifnonexistent false +attribute[].fastsearch false +attribute[].huge false +attribute[].paged false +attribute[].ismutable false +attribute[].sortascending true +attribute[].sortfunction UCA +attribute[].sortstrength PRIMARY +attribute[].sortlocale "" +attribute[].enablebitvectors false +attribute[].enableonlybitvector false +attribute[].fastaccess false +attribute[].arity 8 +attribute[].lowerbound -9223372036854775808 +attribute[].upperbound 9223372036854775807 +attribute[].densepostinglistthreshold 0.4 +attribute[].tensortype "" +attribute[].imported false +attribute[].distancemetric EUCLIDEAN +attribute[].index.hnsw.enabled false +attribute[].index.hnsw.maxlinkspernode 16 +attribute[].index.hnsw.neighborstoexploreatinsert 200 +attribute[].index.hnsw.distancemetric EUCLIDEAN +attribute[].index.hnsw.multithreadedindexing true +attribute[].name "parent_field" +attribute[].datatype STRING +attribute[].collectiontype SINGLE +attribute[].dictionary.type BTREE +attribute[].dictionary.match UNCASED +attribute[].match UNCASED +attribute[].removeifzero false +attribute[].createifnonexistent false +attribute[].fastsearch false +attribute[].huge false +attribute[].paged false +attribute[].ismutable false +attribute[].sortascending true +attribute[].sortfunction UCA +attribute[].sortstrength PRIMARY +attribute[].sortlocale "" +attribute[].enablebitvectors false +attribute[].enableonlybitvector false +attribute[].fastaccess false +attribute[].arity 8 +attribute[].lowerbound -9223372036854775808 +attribute[].upperbound 9223372036854775807 +attribute[].densepostinglistthreshold 0.4 +attribute[].tensortype "" +attribute[].imported false +attribute[].distancemetric EUCLIDEAN +attribute[].index.hnsw.enabled false +attribute[].index.hnsw.maxlinkspernode 16 +attribute[].index.hnsw.neighborstoexploreatinsert 200 +attribute[].index.hnsw.distancemetric EUCLIDEAN +attribute[].index.hnsw.multithreadedindexing true +attribute[].name "parent_imported" +attribute[].datatype STRING +attribute[].collectiontype SINGLE +attribute[].dictionary.type BTREE +attribute[].dictionary.match UNCASED +attribute[].match UNCASED +attribute[].removeifzero false +attribute[].createifnonexistent false +attribute[].fastsearch false +attribute[].huge false +attribute[].paged false +attribute[].ismutable false +attribute[].sortascending true +attribute[].sortfunction UCA +attribute[].sortstrength PRIMARY +attribute[].sortlocale "" +attribute[].enablebitvectors false +attribute[].enableonlybitvector false +attribute[].fastaccess false +attribute[].arity 8 +attribute[].lowerbound -9223372036854775808 +attribute[].upperbound 9223372036854775807 +attribute[].densepostinglistthreshold 0.4 +attribute[].tensortype "" +attribute[].imported true +attribute[].distancemetric EUCLIDEAN +attribute[].index.hnsw.enabled false +attribute[].index.hnsw.maxlinkspernode 16 +attribute[].index.hnsw.neighborstoexploreatinsert 200 +attribute[].index.hnsw.distancemetric EUCLIDEAN +attribute[].index.hnsw.multithreadedindexing true +attribute[].name "child_imported" +attribute[].datatype STRING +attribute[].collectiontype SINGLE +attribute[].dictionary.type BTREE +attribute[].dictionary.match UNCASED +attribute[].match UNCASED +attribute[].removeifzero false +attribute[].createifnonexistent false +attribute[].fastsearch false +attribute[].huge false +attribute[].paged false +attribute[].ismutable false +attribute[].sortascending true +attribute[].sortfunction UCA +attribute[].sortstrength PRIMARY +attribute[].sortlocale "" +attribute[].enablebitvectors false +attribute[].enableonlybitvector false +attribute[].fastaccess false +attribute[].arity 8 +attribute[].lowerbound -9223372036854775808 +attribute[].upperbound 9223372036854775807 +attribute[].densepostinglistthreshold 0.4 +attribute[].tensortype "" +attribute[].imported true +attribute[].distancemetric EUCLIDEAN +attribute[].index.hnsw.enabled false +attribute[].index.hnsw.maxlinkspernode 16 +attribute[].index.hnsw.neighborstoexploreatinsert 200 +attribute[].index.hnsw.distancemetric EUCLIDEAN +attribute[].index.hnsw.multithreadedindexing true diff --git a/config-model/src/test/derived/schemainheritance/child.sd b/config-model/src/test/derived/schemainheritance/child.sd new file mode 100644 index 00000000000..a209919763d --- /dev/null +++ b/config-model/src/test/derived/schemainheritance/child.sd @@ -0,0 +1,43 @@ +schema child inherits parent { + + document child inherits parent { + + field cf1 type string { + indexing: summary + } + + } + + fieldset child_set { + fields: cf1, pf1 + } + + stemming: shortest + + index child_index { + stemming: shortest + } + + field child_field type string { + indexing: input pf1 | lowercase | index | attribute | summary + } + + rank-profile child_profile inherits parent_profile { + } + + constant child_constant { + file: constants/my_constant_tensor_file.json + type: tensor<float>(x{},y{}) + } + + onnx-model child_model { + file: small_constants_and_functions.onnx + } + + document-summary child_summary inherits parent_summary { + summary cf1 type string {} + } + + import field importedschema_ref.importedfield2 as child_imported {} + +} diff --git a/config-model/src/test/derived/schemainheritance/documentmanager.cfg b/config-model/src/test/derived/schemainheritance/documentmanager.cfg new file mode 100644 index 00000000000..0dd87029f32 --- /dev/null +++ b/config-model/src/test/derived/schemainheritance/documentmanager.cfg @@ -0,0 +1,98 @@ +enablecompression false +datatype[].id 1381038251 +datatype[].structtype[].name "position" +datatype[].structtype[].version 0 +datatype[].structtype[].compresstype NONE +datatype[].structtype[].compresslevel 0 +datatype[].structtype[].compressthreshold 95 +datatype[].structtype[].compressminsize 800 +datatype[].structtype[].field[].name "x" +datatype[].structtype[].field[].datatype 0 +datatype[].structtype[].field[].detailedtype "" +datatype[].structtype[].field[].name "y" +datatype[].structtype[].field[].datatype 0 +datatype[].structtype[].field[].detailedtype "" +datatype[].id 990242616 +datatype[].structtype[].name "importedschema.header" +datatype[].structtype[].version 0 +datatype[].structtype[].compresstype NONE +datatype[].structtype[].compresslevel 0 +datatype[].structtype[].compressthreshold 95 +datatype[].structtype[].compressminsize 800 +datatype[].structtype[].field[].name "importedfield1" +datatype[].structtype[].field[].datatype 2 +datatype[].structtype[].field[].detailedtype "" +datatype[].structtype[].field[].name "importedfield2" +datatype[].structtype[].field[].datatype 2 +datatype[].structtype[].field[].detailedtype "" +datatype[].id 167001639 +datatype[].documenttype[].name "importedschema" +datatype[].documenttype[].version 0 +datatype[].documenttype[].inherits[].name "document" +datatype[].documenttype[].inherits[].version 0 +datatype[].documenttype[].headerstruct 990242616 +datatype[].documenttype[].bodystruct 0 +datatype[].documenttype[].fieldsets{[]}.fields[] "importedfield1" +datatype[].documenttype[].fieldsets{[]}.fields[] "importedfield2" +datatype[].id -1266134872 +datatype[].referencetype[].target_type_id 167001639 +datatype[].id 836075987 +datatype[].structtype[].name "parent.header" +datatype[].structtype[].version 0 +datatype[].structtype[].compresstype NONE +datatype[].structtype[].compresslevel 0 +datatype[].structtype[].compressthreshold 95 +datatype[].structtype[].compressminsize 800 +datatype[].structtype[].field[].name "pf1" +datatype[].structtype[].field[].datatype 2 +datatype[].structtype[].field[].detailedtype "" +datatype[].structtype[].field[].name "importedschema_ref" +datatype[].structtype[].field[].datatype -1266134872 +datatype[].structtype[].field[].detailedtype "" +datatype[].structtype[].field[].name "parent_field" +datatype[].structtype[].field[].datatype 2 +datatype[].structtype[].field[].detailedtype "" +datatype[].id 1175161836 +datatype[].documenttype[].name "parent" +datatype[].documenttype[].version 0 +datatype[].documenttype[].inherits[].name "document" +datatype[].documenttype[].inherits[].version 0 +datatype[].documenttype[].headerstruct 836075987 +datatype[].documenttype[].bodystruct 0 +datatype[].documenttype[].fieldsets{parent_set}.fields[] "pf1" +datatype[].documenttype[].fieldsets{[]}.fields[] "importedschema_ref" +datatype[].documenttype[].fieldsets{[]}.fields[] "pf1" +datatype[].documenttype[].importedfield[].name "parent_imported" +datatype[].id 81425825 +datatype[].structtype[].name "child.header" +datatype[].structtype[].version 0 +datatype[].structtype[].compresstype NONE +datatype[].structtype[].compresslevel 0 +datatype[].structtype[].compressthreshold 95 +datatype[].structtype[].compressminsize 800 +datatype[].structtype[].field[].name "cf1" +datatype[].structtype[].field[].datatype 2 +datatype[].structtype[].field[].detailedtype "" +datatype[].structtype[].field[].name "parent_field" +datatype[].structtype[].field[].datatype 2 +datatype[].structtype[].field[].detailedtype "" +datatype[].structtype[].field[].name "child_field" +datatype[].structtype[].field[].datatype 2 +datatype[].structtype[].field[].detailedtype "" +datatype[].id 746267614 +datatype[].documenttype[].name "child" +datatype[].documenttype[].version 0 +datatype[].documenttype[].inherits[].name "document" +datatype[].documenttype[].inherits[].version 0 +datatype[].documenttype[].inherits[].name "parent" +datatype[].documenttype[].inherits[].version 0 +datatype[].documenttype[].headerstruct 81425825 +datatype[].documenttype[].bodystruct 0 +datatype[].documenttype[].fieldsets{parent_set}.fields[] "pf1" +datatype[].documenttype[].fieldsets{child_set}.fields[] "cf1" +datatype[].documenttype[].fieldsets{child_set}.fields[] "pf1" +datatype[].documenttype[].fieldsets{[]}.fields[] "cf1" +datatype[].documenttype[].fieldsets{[]}.fields[] "importedschema_ref" +datatype[].documenttype[].fieldsets{[]}.fields[] "pf1" +datatype[].documenttype[].importedfield[].name "parent_imported" +datatype[].documenttype[].importedfield[].name "child_imported" diff --git a/config-model/src/test/derived/schemainheritance/ilscripts.cfg b/config-model/src/test/derived/schemainheritance/ilscripts.cfg new file mode 100644 index 00000000000..0a8e10859a0 --- /dev/null +++ b/config-model/src/test/derived/schemainheritance/ilscripts.cfg @@ -0,0 +1,11 @@ +maxtermoccurrences 100 +fieldmatchmaxlength 1000000 +ilscript[].doctype "child" +ilscript[].docfield[] "pf1" +ilscript[].docfield[] "importedschema_ref" +ilscript[].docfield[] "cf1" +ilscript[].content[] "clear_state | guard { input pf1 | lowercase | tokenize normalize stem:\"SHORTEST\" | index child_field | attribute child_field | summary child_field; }" +ilscript[].content[] "clear_state | guard { input pf1 | lowercase | tokenize normalize | index parent_field | attribute parent_field | summary parent_field; }" +ilscript[].content[] "clear_state | guard { input pf1 | summary pf1; }" +ilscript[].content[] "clear_state | guard { input importedschema_ref | attribute importedschema_ref; }" +ilscript[].content[] "clear_state | guard { input cf1 | summary cf1; }" diff --git a/config-model/src/test/derived/schemainheritance/importedschema.sd b/config-model/src/test/derived/schemainheritance/importedschema.sd new file mode 100644 index 00000000000..1b5acff8a26 --- /dev/null +++ b/config-model/src/test/derived/schemainheritance/importedschema.sd @@ -0,0 +1,15 @@ +schema importedschema { + + document importedschema { + + field importedfield1 type string { + indexing: attribute + } + + field importedfield2 type string { + indexing: attribute + } + + } + +}
\ No newline at end of file diff --git a/config-model/src/test/derived/schemainheritance/index-info.cfg b/config-model/src/test/derived/schemainheritance/index-info.cfg new file mode 100644 index 00000000000..3febd5c5226 --- /dev/null +++ b/config-model/src/test/derived/schemainheritance/index-info.cfg @@ -0,0 +1,73 @@ +indexinfo[].name "child" +indexinfo[].command[].indexname "sddocname" +indexinfo[].command[].command "index" +indexinfo[].command[].indexname "sddocname" +indexinfo[].command[].command "word" +indexinfo[].command[].indexname "pf1" +indexinfo[].command[].command "index" +indexinfo[].command[].indexname "pf1" +indexinfo[].command[].command "type string" +indexinfo[].command[].indexname "importedschema_ref" +indexinfo[].command[].command "index" +indexinfo[].command[].indexname "importedschema_ref" +indexinfo[].command[].command "attribute" +indexinfo[].command[].indexname "importedschema_ref" +indexinfo[].command[].command "type Reference<importedschema>" +indexinfo[].command[].indexname "importedschema_ref" +indexinfo[].command[].command "word" +indexinfo[].command[].indexname "cf1" +indexinfo[].command[].command "index" +indexinfo[].command[].indexname "cf1" +indexinfo[].command[].command "type string" +indexinfo[].command[].indexname "child_field" +indexinfo[].command[].command "index" +indexinfo[].command[].indexname "child_field" +indexinfo[].command[].command "lowercase" +indexinfo[].command[].indexname "child_field" +indexinfo[].command[].command "stem:SHORTEST" +indexinfo[].command[].indexname "child_field" +indexinfo[].command[].command "normalize" +indexinfo[].command[].indexname "child_field" +indexinfo[].command[].command "plain-tokens" +indexinfo[].command[].indexname "child_field" +indexinfo[].command[].command "type string" +indexinfo[].command[].indexname "parent_field" +indexinfo[].command[].command "index" +indexinfo[].command[].indexname "parent_field" +indexinfo[].command[].command "lowercase" +indexinfo[].command[].indexname "parent_field" +indexinfo[].command[].command "stem:SHORTEST" +indexinfo[].command[].indexname "parent_field" +indexinfo[].command[].command "normalize" +indexinfo[].command[].indexname "parent_field" +indexinfo[].command[].command "plain-tokens" +indexinfo[].command[].indexname "parent_field" +indexinfo[].command[].command "type string" +indexinfo[].command[].indexname "parent_imported" +indexinfo[].command[].command "index" +indexinfo[].command[].indexname "parent_imported" +indexinfo[].command[].command "lowercase" +indexinfo[].command[].indexname "parent_imported" +indexinfo[].command[].command "attribute" +indexinfo[].command[].indexname "parent_imported" +indexinfo[].command[].command "type string" +indexinfo[].command[].indexname "parent_imported" +indexinfo[].command[].command "word" +indexinfo[].command[].indexname "child_imported" +indexinfo[].command[].command "index" +indexinfo[].command[].indexname "child_imported" +indexinfo[].command[].command "lowercase" +indexinfo[].command[].indexname "child_imported" +indexinfo[].command[].command "attribute" +indexinfo[].command[].indexname "child_imported" +indexinfo[].command[].command "type string" +indexinfo[].command[].indexname "child_imported" +indexinfo[].command[].command "word" +indexinfo[].command[].indexname "parent_set" +indexinfo[].command[].command "attribute" +indexinfo[].command[].indexname "parent_set" +indexinfo[].command[].command "index" +indexinfo[].command[].indexname "child_set" +indexinfo[].command[].command "attribute" +indexinfo[].command[].indexname "child_set" +indexinfo[].command[].command "index" diff --git a/config-model/src/test/derived/schemainheritance/parent.sd b/config-model/src/test/derived/schemainheritance/parent.sd new file mode 100644 index 00000000000..51b11dad444 --- /dev/null +++ b/config-model/src/test/derived/schemainheritance/parent.sd @@ -0,0 +1,41 @@ +schema parent { + + document parent { + + field pf1 type string { + indexing: summary + } + + field importedschema_ref type reference<importedschema> { + indexing: attribute + } + + } + + fieldset parent_set { + fields: pf1 + } + stemming: none + index parent_index { + stemming: best + } + field parent_field type string { + indexing: input pf1 | lowercase | index | attribute | summary + } + rank-profile parent_profile { + } + constant parent_constant { + file: constants/my_constant_tensor_file.json + type: tensor<float>(x{},y{}) + } + onnx-model parent_model { + file: small_constants_and_functions.onnx + } + document-summary parent_summary { + summary pf1 type string { + } + } + import field importedschema_ref.importedfield1 as parent_imported { + } + raw-as-base64-in-summary +} diff --git a/config-model/src/test/derived/schemainheritance/rank-profiles.cfg b/config-model/src/test/derived/schemainheritance/rank-profiles.cfg new file mode 100644 index 00000000000..9e68045fab0 --- /dev/null +++ b/config-model/src/test/derived/schemainheritance/rank-profiles.cfg @@ -0,0 +1,11 @@ +rankprofile[].name "default" +rankprofile[].name "unranked" +rankprofile[].fef.property[].name "vespa.rank.firstphase" +rankprofile[].fef.property[].value "value(0)" +rankprofile[].fef.property[].name "vespa.hitcollector.heapsize" +rankprofile[].fef.property[].value "0" +rankprofile[].fef.property[].name "vespa.hitcollector.arraysize" +rankprofile[].fef.property[].value "0" +rankprofile[].fef.property[].name "vespa.dump.ignoredefaultfeatures" +rankprofile[].fef.property[].value "true" +rankprofile[].name "child_profile" diff --git a/config-model/src/test/derived/schemainheritance/small_constants_and_functions.onnx b/config-model/src/test/derived/schemainheritance/small_constants_and_functions.onnx Binary files differnew file mode 100644 index 00000000000..0d4bffa5b57 --- /dev/null +++ b/config-model/src/test/derived/schemainheritance/small_constants_and_functions.onnx diff --git a/config-model/src/test/derived/schemainheritance/summary.cfg b/config-model/src/test/derived/schemainheritance/summary.cfg new file mode 100644 index 00000000000..451894ec2c1 --- /dev/null +++ b/config-model/src/test/derived/schemainheritance/summary.cfg @@ -0,0 +1,49 @@ +defaultsummaryid 1044042981 +classes[].id 1044042981 +classes[].name "default" +classes[].omitsummaryfeatures false +classes[].fields[].name "parent_field" +classes[].fields[].type "longstring" +classes[].fields[].name "pf1" +classes[].fields[].type "longstring" +classes[].fields[].name "rankfeatures" +classes[].fields[].type "featuredata" +classes[].fields[].name "summaryfeatures" +classes[].fields[].type "featuredata" +classes[].fields[].name "child_field" +classes[].fields[].type "longstring" +classes[].fields[].name "cf1" +classes[].fields[].type "longstring" +classes[].fields[].name "documentid" +classes[].fields[].type "longstring" +classes[].id 2134223620 +classes[].name "parent_summary" +classes[].omitsummaryfeatures false +classes[].fields[].name "pf1" +classes[].fields[].type "longstring" +classes[].fields[].name "rankfeatures" +classes[].fields[].type "featuredata" +classes[].fields[].name "summaryfeatures" +classes[].fields[].type "featuredata" +classes[].id 178044032 +classes[].name "attributeprefetch" +classes[].omitsummaryfeatures false +classes[].fields[].name "parent_field" +classes[].fields[].type "longstring" +classes[].fields[].name "rankfeatures" +classes[].fields[].type "featuredata" +classes[].fields[].name "summaryfeatures" +classes[].fields[].type "featuredata" +classes[].fields[].name "child_field" +classes[].fields[].type "longstring" +classes[].id 1486475170 +classes[].name "child_summary" +classes[].omitsummaryfeatures false +classes[].fields[].name "pf1" +classes[].fields[].type "longstring" +classes[].fields[].name "rankfeatures" +classes[].fields[].type "featuredata" +classes[].fields[].name "summaryfeatures" +classes[].fields[].type "featuredata" +classes[].fields[].name "cf1" +classes[].fields[].type "longstring" diff --git a/config-model/src/test/derived/schemainheritance/summarymap.cfg b/config-model/src/test/derived/schemainheritance/summarymap.cfg new file mode 100644 index 00000000000..ad98cbe76e7 --- /dev/null +++ b/config-model/src/test/derived/schemainheritance/summarymap.cfg @@ -0,0 +1,13 @@ +defaultoutputclass -1 +override[].field "rankfeatures" +override[].command "rankfeatures" +override[].arguments "" +override[].field "summaryfeatures" +override[].command "summaryfeatures" +override[].arguments "" +override[].field "child_field" +override[].command "attribute" +override[].arguments "child_field" +override[].field "parent_field" +override[].command "attribute" +override[].arguments "parent_field" diff --git a/config-model/src/test/derived/schemainheritance/vsmsummary.cfg b/config-model/src/test/derived/schemainheritance/vsmsummary.cfg new file mode 100644 index 00000000000..91ca5363a25 --- /dev/null +++ b/config-model/src/test/derived/schemainheritance/vsmsummary.cfg @@ -0,0 +1,17 @@ +outputclass "" +fieldmap[].summary "child_field" +fieldmap[].document[].field "child_field" +fieldmap[].command NONE +fieldmap[].summary "pf1" +fieldmap[].document[].field "pf1" +fieldmap[].command NONE +fieldmap[].summary "parent_field" +fieldmap[].document[].field "parent_field" +fieldmap[].command NONE +fieldmap[].summary "cf1" +fieldmap[].document[].field "cf1" +fieldmap[].command NONE +fieldmap[].summary "rankfeatures" +fieldmap[].command NONE +fieldmap[].summary "summaryfeatures" +fieldmap[].command NONE diff --git a/config-model/src/test/derived/tokenization/tokenization.sd b/config-model/src/test/derived/tokenization/tokenization.sd index 4510a574d60..a74c52e4d9d 100644 --- a/config-model/src/test/derived/tokenization/tokenization.sd +++ b/config-model/src/test/derived/tokenization/tokenization.sd @@ -1,3 +1,4 @@ +# Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. schema tokenization { document tokenization { @@ -20,4 +21,4 @@ schema tokenization { indexing: input text_array | for_each { lowercase } | for_each { normalize } | index | summary } -}
\ No newline at end of file +} diff --git a/config-model/src/test/java/com/yahoo/config/model/ApplicationDeployTest.java b/config-model/src/test/java/com/yahoo/config/model/ApplicationDeployTest.java index adb6e423a39..0ccbec529ed 100644 --- a/config-model/src/test/java/com/yahoo/config/model/ApplicationDeployTest.java +++ b/config-model/src/test/java/com/yahoo/config/model/ApplicationDeployTest.java @@ -13,8 +13,8 @@ import com.yahoo.config.provision.ApplicationId; import com.yahoo.document.DataType; import com.yahoo.io.IOUtils; import com.yahoo.path.Path; -import com.yahoo.searchdefinition.DocumentOnlySearch; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.DocumentOnlySchema; +import com.yahoo.searchdefinition.Schema; import com.yahoo.vespa.config.ConfigDefinition; import com.yahoo.vespa.config.ConfigDefinitionKey; import com.yahoo.vespa.model.VespaModel; @@ -58,7 +58,7 @@ public class ApplicationDeployTest { List<NamedSchema> schemas = tester.getSchemas(); assertEquals(schemas.size(), 5); for (NamedSchema searchDefinition : schemas) { - Search s = searchDefinition.getSearch(); + Schema s = searchDefinition.getSearch(); switch (s.getName()) { case "music": case "laptop": @@ -66,7 +66,7 @@ public class ApplicationDeployTest { case "sock": break; case "product": - assertTrue(s instanceof DocumentOnlySearch); + assertTrue(s instanceof DocumentOnlySchema); assertEquals(DataType.STRING, s.getDocument().getField("title").getDataType()); break; default: diff --git a/config-model/src/test/java/com/yahoo/document/test/SDDocumentTypeTestCase.java b/config-model/src/test/java/com/yahoo/document/test/SDDocumentTypeTestCase.java index c878ea9951a..4714f4396dd 100644 --- a/config-model/src/test/java/com/yahoo/document/test/SDDocumentTypeTestCase.java +++ b/config-model/src/test/java/com/yahoo/document/test/SDDocumentTypeTestCase.java @@ -4,7 +4,7 @@ package com.yahoo.document.test; import com.yahoo.document.DataType; import com.yahoo.document.DataTypeName; import com.yahoo.documentmodel.VespaDocumentType; -import com.yahoo.searchdefinition.SchemaTestCase; +import com.yahoo.searchdefinition.AbstractSchemaTestCase; import com.yahoo.searchdefinition.document.SDDocumentType; import com.yahoo.searchdefinition.document.SDField; import org.junit.Test; @@ -17,7 +17,7 @@ import static org.junit.Assert.*; * @author Thomas Gundersen * @author bratseth */ -public class SDDocumentTypeTestCase extends SchemaTestCase { +public class SDDocumentTypeTestCase extends AbstractSchemaTestCase { // Verify that we can register and retrieve fields. @Test diff --git a/config-model/src/test/java/com/yahoo/document/test/SDFieldTestCase.java b/config-model/src/test/java/com/yahoo/document/test/SDFieldTestCase.java index 223cd6a6e85..36a4c7bdafc 100644 --- a/config-model/src/test/java/com/yahoo/document/test/SDFieldTestCase.java +++ b/config-model/src/test/java/com/yahoo/document/test/SDFieldTestCase.java @@ -2,7 +2,7 @@ package com.yahoo.document.test; import com.yahoo.document.DataType; -import com.yahoo.searchdefinition.SchemaTestCase; +import com.yahoo.searchdefinition.AbstractSchemaTestCase; import com.yahoo.searchdefinition.document.SDDocumentType; import org.junit.Test; @@ -11,7 +11,7 @@ import static org.junit.Assert.fail; /** * @author Thomas Gundersen */ -public class SDFieldTestCase extends SchemaTestCase { +public class SDFieldTestCase extends AbstractSchemaTestCase { @Test public void testIdSettingConflict() { diff --git a/config-model/src/test/java/com/yahoo/searchdefinition/AbstractSchemaTestCase.java b/config-model/src/test/java/com/yahoo/searchdefinition/AbstractSchemaTestCase.java new file mode 100644 index 00000000000..a08ec110219 --- /dev/null +++ b/config-model/src/test/java/com/yahoo/searchdefinition/AbstractSchemaTestCase.java @@ -0,0 +1,78 @@ +// Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. +package com.yahoo.searchdefinition; + +import com.yahoo.io.IOUtils; + +import java.io.BufferedReader; +import java.io.BufferedWriter; +import java.io.IOException; + +import static helpers.CompareConfigTestHelper.assertSerializedConfigEquals; +import static helpers.CompareConfigTestHelper.assertSerializedConfigFileEquals; + +public abstract class AbstractSchemaTestCase { + + protected static void assertConfigFile(String filename, String cfg) throws IOException { + assertSerializedConfigFileEquals(filename, cfg); + } + + protected static void assertConfigFiles(String expectedFile, + String cfgFile, + boolean orderMatters, + boolean updateOnAssert) throws IOException { + try { + assertSerializedConfigEquals(readAndCensorIndexes(expectedFile), readAndCensorIndexes(cfgFile), orderMatters); + } catch (AssertionError e) { + if (updateOnAssert) { + BufferedWriter writer = IOUtils.createWriter(expectedFile, false); + writer.write(readAndCensorIndexes(cfgFile)); + writer.newLine(); + writer.flush(); + writer.close(); + System.err.println(e.getMessage() + " [not equal files: >>>"+expectedFile+"<<< and >>>"+cfgFile+"<<< in assertConfigFiles]"); + return; + } + throw new AssertionError(e.getMessage() + " [not equal files: >>>"+expectedFile+"<<< and >>>"+cfgFile+"<<< in assertConfigFiles]", e); + } + } + /** + * This is to avoid having to keep those pesky array index numbers in the config format up to date + * as new entries are added and removed. + */ + private static String readAndCensorIndexes(String file) throws IOException { + StringBuilder b = new StringBuilder(); + try (BufferedReader r = IOUtils.createReader(file)) { + int character; + boolean lastWasNewline = false; + boolean inBrackets = false; + while (-1 != (character = r.read())) { + // skip empty lines + if (character == '\n') { + if (lastWasNewline) continue; + lastWasNewline = true; + } + else { + lastWasNewline = false; + } + + // skip quoted strings + if (character == '"') { + b.appendCodePoint(character); + while (-1 != (character = r.read()) && character != '"') { + b.appendCodePoint(character); + } + } + + // skip bracket content + if (character == ']') + inBrackets = false; + if (! inBrackets) + b.appendCodePoint(character); + if (character == '[') + inBrackets = true; + } + } + return b.toString(); + } + +} diff --git a/config-model/src/test/java/com/yahoo/searchdefinition/ArraysTestCase.java b/config-model/src/test/java/com/yahoo/searchdefinition/ArraysTestCase.java index 801459693de..131c6e19c68 100644 --- a/config-model/src/test/java/com/yahoo/searchdefinition/ArraysTestCase.java +++ b/config-model/src/test/java/com/yahoo/searchdefinition/ArraysTestCase.java @@ -18,16 +18,16 @@ import static org.junit.Assert.assertTrue; * * @author bratseth */ -public class ArraysTestCase extends SchemaTestCase { +public class ArraysTestCase extends AbstractSchemaTestCase { @Test public void testArrayImporting() throws IOException, ParseException { - Search search = SearchBuilder.buildFromFile("src/test/examples/arrays.sd"); + Schema schema = SearchBuilder.buildFromFile("src/test/examples/arrays.sd"); - SDField tags = (SDField)search.getDocument().getField("tags"); + SDField tags = (SDField) schema.getDocument().getField("tags"); assertEquals(DataType.STRING, ((CollectionDataType)tags.getDataType()).getNestedType()); - SDField ratings = (SDField)search.getDocument().getField("ratings"); + SDField ratings = (SDField) schema.getDocument().getField("ratings"); assertTrue(ratings.getDataType() instanceof ArrayDataType); assertEquals(DataType.INT, ((ArrayDataType)ratings.getDataType()).getNestedType()); } diff --git a/config-model/src/test/java/com/yahoo/searchdefinition/ArraysWeightedSetsTestCase.java b/config-model/src/test/java/com/yahoo/searchdefinition/ArraysWeightedSetsTestCase.java index 45ec5176bc7..0d5a4b357b1 100644 --- a/config-model/src/test/java/com/yahoo/searchdefinition/ArraysWeightedSetsTestCase.java +++ b/config-model/src/test/java/com/yahoo/searchdefinition/ArraysWeightedSetsTestCase.java @@ -16,24 +16,24 @@ import static org.junit.Assert.assertTrue; * * @author Einar M R Rosenvinge */ -public class ArraysWeightedSetsTestCase extends SchemaTestCase { +public class ArraysWeightedSetsTestCase extends AbstractSchemaTestCase { @Test public void testArrayWeightedSetsImporting() throws java.io.IOException, com.yahoo.searchdefinition.parser.ParseException { - Search search = SearchBuilder.buildFromFile("src/test/examples/arraysweightedsets.sd"); + Schema schema = SearchBuilder.buildFromFile("src/test/examples/arraysweightedsets.sd"); - SDField tags = (SDField) search.getDocument().getField("tags"); + SDField tags = (SDField) schema.getDocument().getField("tags"); assertTrue(tags.getDataType() instanceof ArrayDataType); assertEquals(DataType.STRING, ((CollectionDataType)tags.getDataType()).getNestedType()); - SDField ratings = (SDField) search.getDocument().getField("ratings"); + SDField ratings = (SDField) schema.getDocument().getField("ratings"); assertTrue(ratings.getDataType() instanceof ArrayDataType); assertEquals(DataType.INT, ((CollectionDataType)ratings.getDataType()).getNestedType()); - SDField flags = (SDField) search.getDocument().getField("flags"); + SDField flags = (SDField) schema.getDocument().getField("flags"); assertTrue(flags.getDataType() instanceof WeightedSetDataType); assertEquals(DataType.STRING, ((CollectionDataType)flags.getDataType()).getNestedType()); - SDField banners = (SDField) search.getDocument().getField("banners"); + SDField banners = (SDField) schema.getDocument().getField("banners"); assertTrue(banners.getDataType() instanceof WeightedSetDataType); assertEquals(DataType.INT, ((CollectionDataType)banners.getDataType()).getNestedType()); } diff --git a/config-model/src/test/java/com/yahoo/searchdefinition/AttributeSettingsTestCase.java b/config-model/src/test/java/com/yahoo/searchdefinition/AttributeSettingsTestCase.java index 869cc2678b2..22ae06d929a 100644 --- a/config-model/src/test/java/com/yahoo/searchdefinition/AttributeSettingsTestCase.java +++ b/config-model/src/test/java/com/yahoo/searchdefinition/AttributeSettingsTestCase.java @@ -25,16 +25,16 @@ import static org.junit.Assert.*; * * @author bratseth */ -public class AttributeSettingsTestCase extends SchemaTestCase { +public class AttributeSettingsTestCase extends AbstractSchemaTestCase { @Rule public final ExpectedException exceptionRule = ExpectedException.none(); @Test public void testAttributeSettings() throws IOException, ParseException { - Search search = SearchBuilder.buildFromFile("src/test/examples/attributesettings.sd"); + Schema schema = SearchBuilder.buildFromFile("src/test/examples/attributesettings.sd"); - SDField f1=(SDField) search.getDocument().getField("f1"); + SDField f1=(SDField) schema.getDocument().getField("f1"); assertEquals(1, f1.getAttributes().size()); Attribute a1 = f1.getAttributes().get(f1.getName()); assertThat(a1.getType(), is(Attribute.Type.LONG)); @@ -45,7 +45,7 @@ public class AttributeSettingsTestCase extends SchemaTestCase { assertFalse(a1.isRemoveIfZero()); assertFalse(a1.isCreateIfNonExistent()); - SDField f2=(SDField) search.getDocument().getField("f2"); + SDField f2=(SDField) schema.getDocument().getField("f2"); assertEquals(1, f2.getAttributes().size()); Attribute a2 = f2.getAttributes().get(f2.getName()); assertThat(a2.getType(), is(Attribute.Type.LONG)); @@ -56,7 +56,7 @@ public class AttributeSettingsTestCase extends SchemaTestCase { assertFalse(a2.isRemoveIfZero()); assertFalse(a2.isCreateIfNonExistent()); assertThat(f2.getAliasToName().get("f2alias"), is("f2")); - SDField f3=(SDField) search.getDocument().getField("f3"); + SDField f3=(SDField) schema.getDocument().getField("f3"); assertEquals(1, f3.getAttributes().size()); assertThat(f3.getAliasToName().get("f3alias"), is("f3")); @@ -69,17 +69,17 @@ public class AttributeSettingsTestCase extends SchemaTestCase { assertFalse(a3.isRemoveIfZero()); assertFalse(a3.isCreateIfNonExistent()); - assertWeightedSet(search,"f4",true,true); - assertWeightedSet(search,"f5",true,true); - assertWeightedSet(search,"f6",true,true); - assertWeightedSet(search,"f7",true,false); - assertWeightedSet(search,"f8",true,false); - assertWeightedSet(search,"f9",false,true); - assertWeightedSet(search,"f10",false,true); + assertWeightedSet(schema, "f4", true, true); + assertWeightedSet(schema, "f5", true, true); + assertWeightedSet(schema, "f6", true, true); + assertWeightedSet(schema, "f7", true, false); + assertWeightedSet(schema, "f8", true, false); + assertWeightedSet(schema, "f9", false, true); + assertWeightedSet(schema, "f10", false, true); } - private void assertWeightedSet(Search search, String name, boolean createIfNonExistent, boolean removeIfZero) { - SDField f4 = (SDField) search.getDocument().getField(name); + private void assertWeightedSet(Schema schema, String name, boolean createIfNonExistent, boolean removeIfZero) { + SDField f4 = (SDField) schema.getDocument().getField(name); assertEquals(1, f4.getAttributes().size()); Attribute a4 = f4.getAttributes().get(f4.getName()); assertThat(a4.getType(), is(Attribute.Type.STRING)); @@ -93,14 +93,14 @@ public class AttributeSettingsTestCase extends SchemaTestCase { @Test public void requireThatFastAccessCanBeSet() throws IOException, ParseException { - Search search = SearchBuilder.buildFromFile("src/test/examples/attributesettings.sd"); - SDField field = (SDField) search.getDocument().getField("fast_access"); + Schema schema = SearchBuilder.buildFromFile("src/test/examples/attributesettings.sd"); + SDField field = (SDField) schema.getDocument().getField("fast_access"); assertEquals(1, field.getAttributes().size()); Attribute attr = field.getAttributes().get(field.getName()); assertTrue(attr.isFastAccess()); } - private Search getSearch(String sd) throws ParseException { + private Schema getSearch(String sd) throws ParseException { SearchBuilder builder = new SearchBuilder(); builder.importString(sd); builder.build(); @@ -108,8 +108,8 @@ public class AttributeSettingsTestCase extends SchemaTestCase { } private Attribute getAttributeF(String sd) throws ParseException { - Search search = getSearch(sd); - SDField field = (SDField) search.getDocument().getField("f"); + Schema schema = getSearch(sd); + SDField field = (SDField) schema.getDocument().getField("f"); return field.getAttributes().get(field.getName()); } @@ -183,7 +183,7 @@ public class AttributeSettingsTestCase extends SchemaTestCase { assertTrue(attr.isMutable()); } - private Search getSearchWithMutables() throws ParseException { + private Schema getSearchWithMutables() throws ParseException { return getSearch( "search test {\n" + " document test { \n" + diff --git a/config-model/src/test/java/com/yahoo/searchdefinition/CommentTestCase.java b/config-model/src/test/java/com/yahoo/searchdefinition/CommentTestCase.java index 050d5299c7d..35eccb19151 100644 --- a/config-model/src/test/java/com/yahoo/searchdefinition/CommentTestCase.java +++ b/config-model/src/test/java/com/yahoo/searchdefinition/CommentTestCase.java @@ -14,12 +14,12 @@ import static org.junit.Assert.assertEquals; * * @author bratseth */ -public class CommentTestCase extends SchemaTestCase { +public class CommentTestCase extends AbstractSchemaTestCase { @Test public void testComments() throws IOException, ParseException { - Search search = SearchBuilder.buildFromFile("src/test/examples/comment.sd"); - SDField field = search.getConcreteField("a"); + Schema schema = SearchBuilder.buildFromFile("src/test/examples/comment.sd"); + SDField field = schema.getConcreteField("a"); assertEquals("{ input a | tokenize normalize stem:\"BEST\" | summary a | index a; }", field.getIndexingScript().toString()); } diff --git a/config-model/src/test/java/com/yahoo/searchdefinition/DiversityTestCase.java b/config-model/src/test/java/com/yahoo/searchdefinition/DiversityTestCase.java index 6c4eb4748c8..e6191772904 100644 --- a/config-model/src/test/java/com/yahoo/searchdefinition/DiversityTestCase.java +++ b/config-model/src/test/java/com/yahoo/searchdefinition/DiversityTestCase.java @@ -43,7 +43,7 @@ public class DiversityTestCase { " }\n" + "}\n"); builder.build(); - Search s = builder.getSearch(); + Schema s = builder.getSearch(); RankProfile.MatchPhaseSettings matchPhase = rankProfileRegistry.get(s, "parent").getMatchPhaseSettings(); RankProfile.DiversitySettings diversity = matchPhase.getDiversity(); assertEquals("b", diversity.getAttribute()); diff --git a/config-model/src/test/java/com/yahoo/searchdefinition/DocumentGraphValidatorTest.java b/config-model/src/test/java/com/yahoo/searchdefinition/DocumentGraphValidatorTest.java index 4fae4d92cb0..f57c63b440f 100644 --- a/config-model/src/test/java/com/yahoo/searchdefinition/DocumentGraphValidatorTest.java +++ b/config-model/src/test/java/com/yahoo/searchdefinition/DocumentGraphValidatorTest.java @@ -30,21 +30,21 @@ public class DocumentGraphValidatorTest { @Test public void simple_ref_dag_is_allowed() { - Search advertiserSearch = createSearchWithName("advertiser"); - Search campaignSearch = createSearchWithName("campaign"); - Search adSearch = createSearchWithName("ad"); - createDocumentReference(adSearch, advertiserSearch, "advertiser_ref"); - createDocumentReference(adSearch, campaignSearch, "campaign_ref"); + Schema advertiserSchema = createSearchWithName("advertiser"); + Schema campaignSchema = createSearchWithName("campaign"); + Schema adSchema = createSearchWithName("ad"); + createDocumentReference(adSchema, advertiserSchema, "advertiser_ref"); + createDocumentReference(adSchema, campaignSchema, "campaign_ref"); DocumentGraphValidator validator = new DocumentGraphValidator(); - validator.validateDocumentGraph(documentListOf(advertiserSearch, campaignSearch, adSearch)); + validator.validateDocumentGraph(documentListOf(advertiserSchema, campaignSchema, adSchema)); } @Test public void simple_inheritance_dag_is_allowed() { - Search grandfather = createSearchWithName("grandfather"); - Search father = createSearchWithName("father", grandfather); - Search son = createSearchWithName("son", father); + Schema grandfather = createSearchWithName("grandfather"); + Schema father = createSearchWithName("father", grandfather); + Schema son = createSearchWithName("son", father); DocumentGraphValidator validator = new DocumentGraphValidator(); validator.validateDocumentGraph(documentListOf(son, father, grandfather)); @@ -52,16 +52,16 @@ public class DocumentGraphValidatorTest { @Test public void complex_dag_is_allowed() { - Search grandfather = createSearchWithName("grandfather"); - Search father = createSearchWithName("father", grandfather); - Search mother = createSearchWithName("mother", grandfather); + Schema grandfather = createSearchWithName("grandfather"); + Schema father = createSearchWithName("father", grandfather); + Schema mother = createSearchWithName("mother", grandfather); createDocumentReference(father, mother, "wife_ref"); - Search son = createSearchWithName("son", father, mother); - Search daughter = createSearchWithName("daughter", father, mother); + Schema son = createSearchWithName("son", father, mother); + Schema daughter = createSearchWithName("daughter", father, mother); createDocumentReference(daughter, son, "brother_ref"); - Search randomGuy1 = createSearchWithName("randomguy1"); - Search randomGuy2 = createSearchWithName("randomguy2"); + Schema randomGuy1 = createSearchWithName("randomguy1"); + Schema randomGuy2 = createSearchWithName("randomguy2"); createDocumentReference(randomGuy1, mother, "secret_ref"); DocumentGraphValidator validator = new DocumentGraphValidator(); @@ -70,54 +70,54 @@ public class DocumentGraphValidatorTest { @Test public void ref_cycle_is_forbidden() { - Search search1 = createSearchWithName("doc1"); - Search search2 = createSearchWithName("doc2"); - Search search3 = createSearchWithName("doc3"); - createDocumentReference(search1, search2, "ref_2"); - createDocumentReference(search2, search3, "ref_3"); - createDocumentReference(search3, search1, "ref_1"); + Schema schema1 = createSearchWithName("doc1"); + Schema schema2 = createSearchWithName("doc2"); + Schema schema3 = createSearchWithName("doc3"); + createDocumentReference(schema1, schema2, "ref_2"); + createDocumentReference(schema2, schema3, "ref_3"); + createDocumentReference(schema3, schema1, "ref_1"); DocumentGraphValidator validator = new DocumentGraphValidator(); exceptionRule.expect(DocumentGraphValidator.DocumentGraphException.class); exceptionRule.expectMessage("Document dependency cycle detected: doc1->doc2->doc3->doc1."); - validator.validateDocumentGraph(documentListOf(search1, search2, search3)); + validator.validateDocumentGraph(documentListOf(schema1, schema2, schema3)); } @Test public void inherit_cycle_is_forbidden() { - Search search1 = createSearchWithName("doc1"); - Search search2 = createSearchWithName("doc2", search1); - Search search3 = createSearchWithName("doc3", search2); - search1.getDocument().inherit(search3.getDocument()); + Schema schema1 = createSearchWithName("doc1"); + Schema schema2 = createSearchWithName("doc2", schema1); + Schema schema3 = createSearchWithName("doc3", schema2); + schema1.getDocument().inherit(schema3.getDocument()); DocumentGraphValidator validator = new DocumentGraphValidator(); exceptionRule.expect(DocumentGraphValidator.DocumentGraphException.class); exceptionRule.expectMessage("Document dependency cycle detected: doc1->doc3->doc2->doc1."); - validator.validateDocumentGraph(documentListOf(search1, search2, search3)); + validator.validateDocumentGraph(documentListOf(schema1, schema2, schema3)); } @Test public void combined_inherit_and_ref_cycle_is_forbidden() { - Search search1 = createSearchWithName("doc1"); - Search search2 = createSearchWithName("doc2", search1); - Search search3 = createSearchWithName("doc3", search2); - createDocumentReference(search1, search3, "ref_1"); + Schema schema1 = createSearchWithName("doc1"); + Schema schema2 = createSearchWithName("doc2", schema1); + Schema schema3 = createSearchWithName("doc3", schema2); + createDocumentReference(schema1, schema3, "ref_1"); DocumentGraphValidator validator = new DocumentGraphValidator(); exceptionRule.expect(DocumentGraphValidator.DocumentGraphException.class); exceptionRule.expectMessage("Document dependency cycle detected: doc1->doc3->doc2->doc1."); - validator.validateDocumentGraph(documentListOf(search1, search2, search3)); + validator.validateDocumentGraph(documentListOf(schema1, schema2, schema3)); } @Test public void self_reference_is_forbidden() { - Search adSearch = createSearchWithName("ad"); - createDocumentReference(adSearch, adSearch, "ad_ref"); + Schema adSchema = createSearchWithName("ad"); + createDocumentReference(adSchema, adSchema, "ad_ref"); DocumentGraphValidator validator = new DocumentGraphValidator(); exceptionRule.expect(DocumentGraphValidator.DocumentGraphException.class); exceptionRule.expectMessage("Document dependency cycle detected: ad->ad."); - validator.validateDocumentGraph(documentListOf(adSearch)); + validator.validateDocumentGraph(documentListOf(adSchema)); } /** @@ -127,8 +127,8 @@ public class DocumentGraphValidatorTest { @Test public void self_inheritance_forbidden() { try { - Search adSearch = createSearchWithName("ad"); - SDDocumentType document = adSearch.getDocument(); + Schema adSchema = createSearchWithName("ad"); + SDDocumentType document = adSchema.getDocument(); document.inherit(document); fail("Expected exception"); } @@ -137,22 +137,22 @@ public class DocumentGraphValidatorTest { } } - private static List<SDDocumentType> documentListOf(Search... searches) { - return Arrays.stream(searches).map(Search::getDocument).collect(toList()); + private static List<SDDocumentType> documentListOf(Schema... schemas) { + return Arrays.stream(schemas).map(Schema::getDocument).collect(toList()); } - private static Search createSearchWithName(String name, Search... parents) { - Search campaignSearch = new Search(name); + private static Schema createSearchWithName(String name, Schema... parents) { + Schema campaignSchema = new Schema(name); SDDocumentType document = new SDDocumentType(name); - campaignSearch.addDocument(document); + campaignSchema.addDocument(document); document.setDocumentReferences(new DocumentReferences(Collections.emptyMap())); Arrays.stream(parents) - .map(Search::getDocument) + .map(Schema::getDocument) .forEach(document::inherit); - return campaignSearch; + return campaignSchema; } - private static void createDocumentReference(Search from, Search to, String refFieldName) { + private static void createDocumentReference(Schema from, Schema to, String refFieldName) { SDField refField = new TemporarySDField(refFieldName, ReferenceDataType.createWithInferredId(TemporaryStructuredDataType.create(to.getName()))); SDDocumentType fromDocument = from.getDocument(); fromDocument.addField(refField); diff --git a/config-model/src/test/java/com/yahoo/searchdefinition/DocumentReferenceResolverTest.java b/config-model/src/test/java/com/yahoo/searchdefinition/DocumentReferenceResolverTest.java index 28c96f39556..260b7a98fe7 100644 --- a/config-model/src/test/java/com/yahoo/searchdefinition/DocumentReferenceResolverTest.java +++ b/config-model/src/test/java/com/yahoo/searchdefinition/DocumentReferenceResolverTest.java @@ -31,28 +31,28 @@ public class DocumentReferenceResolverTest { @Test public void reference_from_one_document_to_another_is_resolved() { // Create bar document with no fields - Search barSearch = new Search(BAR); - SDDocumentType barDocument = new SDDocumentType(BAR, barSearch); - barSearch.addDocument(barDocument); + Schema barSchema = new Schema(BAR); + SDDocumentType barDocument = new SDDocumentType(BAR, barSchema); + barSchema.addDocument(barDocument); // Create foo document with document reference to bar and add another field SDField fooRefToBarField = new SDField ("bar_ref", ReferenceDataType.createWithInferredId(barDocument.getDocumentType())); AttributeUtils.addAttributeAspect(fooRefToBarField); SDField irrelevantField = new SDField("irrelevant_stuff", DataType.INT); - Search fooSearch = new Search(FOO); - SDDocumentType fooDocument = new SDDocumentType("foo", fooSearch); + Schema fooSchema = new Schema(FOO); + SDDocumentType fooDocument = new SDDocumentType("foo", fooSchema); fooDocument.addField(fooRefToBarField); fooDocument.addField(irrelevantField); - fooSearch.addDocument(fooDocument); + fooSchema.addDocument(fooDocument); - DocumentReferenceResolver resolver = new DocumentReferenceResolver(asList(fooSearch, barSearch)); + DocumentReferenceResolver resolver = new DocumentReferenceResolver(asList(fooSchema, barSchema)); resolver.resolveReferences(fooDocument); assertTrue(fooDocument.getDocumentReferences().isPresent()); Map<String, DocumentReference> fooReferenceMap = fooDocument.getDocumentReferences().get().referenceMap(); assertEquals(1, fooReferenceMap.size()); - assertSame(barSearch, fooReferenceMap.get("bar_ref").targetSearch()); + assertSame(barSchema, fooReferenceMap.get("bar_ref").targetSearch()); assertSame(fooRefToBarField, fooReferenceMap.get("bar_ref").referenceField()); } @@ -62,12 +62,12 @@ public class DocumentReferenceResolverTest { SDField fooRefToBarField = new SDField( "bar_ref", ReferenceDataType.createWithInferredId(TemporaryStructuredDataType.create("bar"))); AttributeUtils.addAttributeAspect(fooRefToBarField); - Search fooSearch = new Search(FOO); - SDDocumentType fooDocument = new SDDocumentType("foo", fooSearch); + Schema fooSchema = new Schema(FOO); + SDDocumentType fooDocument = new SDDocumentType("foo", fooSchema); fooDocument.addField(fooRefToBarField); - fooSearch.addDocument(fooDocument); + fooSchema.addDocument(fooDocument); - DocumentReferenceResolver resolver = new DocumentReferenceResolver(singletonList(fooSearch)); + DocumentReferenceResolver resolver = new DocumentReferenceResolver(singletonList(fooSchema)); exceptionRule.expect(IllegalArgumentException.class); exceptionRule.expectMessage( @@ -78,19 +78,19 @@ public class DocumentReferenceResolverTest { @Test public void throws_exception_if_reference_is_not_an_attribute() { // Create bar document with no fields - Search barSearch = new Search(BAR); - SDDocumentType barDocument = new SDDocumentType("bar", barSearch); - barSearch.addDocument(barDocument); + Schema barSchema = new Schema(BAR); + SDDocumentType barDocument = new SDDocumentType("bar", barSchema); + barSchema.addDocument(barDocument); // Create foo document with document reference to bar SDField fooRefToBarField = new SDField ("bar_ref", ReferenceDataType.createWithInferredId(barDocument.getDocumentType())); - Search fooSearch = new Search(FOO); - SDDocumentType fooDocument = new SDDocumentType("foo", fooSearch); + Schema fooSchema = new Schema(FOO); + SDDocumentType fooDocument = new SDDocumentType("foo", fooSchema); fooDocument.addField(fooRefToBarField); - fooSearch.addDocument(fooDocument); + fooSchema.addDocument(fooDocument); - DocumentReferenceResolver resolver = new DocumentReferenceResolver(asList(fooSearch, barSearch)); + DocumentReferenceResolver resolver = new DocumentReferenceResolver(asList(fooSchema, barSchema)); exceptionRule.expect(IllegalArgumentException.class); exceptionRule.expectMessage( "The field 'bar_ref' is an invalid document reference. The field must be an attribute."); diff --git a/config-model/src/test/java/com/yahoo/searchdefinition/FieldOfTypeDocumentTestCase.java b/config-model/src/test/java/com/yahoo/searchdefinition/FieldOfTypeDocumentTestCase.java index d4557549b14..fdd7fe95c45 100644 --- a/config-model/src/test/java/com/yahoo/searchdefinition/FieldOfTypeDocumentTestCase.java +++ b/config-model/src/test/java/com/yahoo/searchdefinition/FieldOfTypeDocumentTestCase.java @@ -19,7 +19,7 @@ import static org.junit.Assert.assertSame; /** * @author Einar M R Rosenvinge */ -public class FieldOfTypeDocumentTestCase extends SchemaTestCase { +public class FieldOfTypeDocumentTestCase extends AbstractSchemaTestCase { @Test public void testDocument() throws IOException { diff --git a/config-model/src/test/java/com/yahoo/searchdefinition/ImportedFieldsEnumeratorTest.java b/config-model/src/test/java/com/yahoo/searchdefinition/ImportedFieldsEnumeratorTest.java index f12129155b1..5f507a1d5d6 100644 --- a/config-model/src/test/java/com/yahoo/searchdefinition/ImportedFieldsEnumeratorTest.java +++ b/config-model/src/test/java/com/yahoo/searchdefinition/ImportedFieldsEnumeratorTest.java @@ -19,35 +19,35 @@ public class ImportedFieldsEnumeratorTest { @Test public void imported_fields_are_enumerated_and_copied_from_correct_search_instance() { String PARENT = "parent"; - Search parentSearch = new Search(PARENT); - SDDocumentType parentDocument = new SDDocumentType(PARENT, parentSearch); + Schema parentSchema = new Schema(PARENT); + SDDocumentType parentDocument = new SDDocumentType(PARENT, parentSchema); var parentField = new SDField("their_field", DataType.INT); AttributeUtils.addAttributeAspect(parentField); parentDocument.addField(parentField); - parentSearch.addDocument(parentDocument); + parentSchema.addDocument(parentDocument); String FOO = "foo"; - Search fooSearch = new Search(FOO); + Schema fooSchema = new Schema(FOO); SDField fooRefToParent = new SDField( "foo_ref", ReferenceDataType.createWithInferredId(parentDocument.getDocumentType())); AttributeUtils.addAttributeAspect(fooRefToParent); - var fooImports = fooSearch.temporaryImportedFields().get(); + var fooImports = fooSchema.temporaryImportedFields().get(); fooImports.add(new TemporaryImportedField("my_first_import", "foo_ref", "their_field")); fooImports.add(new TemporaryImportedField("my_second_import", "foo_ref", "their_field")); - SDDocumentType fooDocument = new SDDocumentType(FOO, fooSearch); - fooSearch.addDocument(fooDocument); + SDDocumentType fooDocument = new SDDocumentType(FOO, fooSchema); + fooSchema.addDocument(fooDocument); String BAR = "bar"; - Search barSearch = new Search(BAR); + Schema barSchema = new Schema(BAR); SDField barRefToParent = new SDField( "bar_ref", ReferenceDataType.createWithInferredId(parentDocument.getDocumentType())); AttributeUtils.addAttributeAspect(barRefToParent); - var barImports = barSearch.temporaryImportedFields().get(); + var barImports = barSchema.temporaryImportedFields().get(); barImports.add(new TemporaryImportedField("my_cool_import", "my_ref", "their_field")); - SDDocumentType barDocument = new SDDocumentType(BAR, barSearch); - barSearch.addDocument(barDocument); + SDDocumentType barDocument = new SDDocumentType(BAR, barSchema); + barSchema.addDocument(barDocument); - var enumerator = new ImportedFieldsEnumerator(List.of(parentSearch, fooSearch, barSearch)); + var enumerator = new ImportedFieldsEnumerator(List.of(parentSchema, fooSchema, barSchema)); enumerator.enumerateImportedFields(parentDocument); assertImportedFieldsAre(parentDocument, List.of()); // No imported fields in parent diff --git a/config-model/src/test/java/com/yahoo/searchdefinition/IncorrectRankingExpressionFileRefTestCase.java b/config-model/src/test/java/com/yahoo/searchdefinition/IncorrectRankingExpressionFileRefTestCase.java index 5dccd0afd76..aab8c9939d2 100644 --- a/config-model/src/test/java/com/yahoo/searchdefinition/IncorrectRankingExpressionFileRefTestCase.java +++ b/config-model/src/test/java/com/yahoo/searchdefinition/IncorrectRankingExpressionFileRefTestCase.java @@ -15,16 +15,16 @@ import static org.junit.Assert.fail; /** * @author bratseth */ -public class IncorrectRankingExpressionFileRefTestCase extends SchemaTestCase { +public class IncorrectRankingExpressionFileRefTestCase extends AbstractSchemaTestCase { @Test public void testIncorrectRef() throws IOException, ParseException { try { RankProfileRegistry registry = new RankProfileRegistry(); - Search search = SearchBuilder.buildFromFile("src/test/examples/incorrectrankingexpressionfileref.sd", + Schema schema = SearchBuilder.buildFromFile("src/test/examples/incorrectrankingexpressionfileref.sd", registry, new QueryProfileRegistry()); - new DerivedConfiguration(search, registry); // cause rank profile parsing + new DerivedConfiguration(schema, registry); // cause rank profile parsing fail("parsing should have failed"); } catch (IllegalArgumentException e) { String message = Exceptions.toMessageString(e); diff --git a/config-model/src/test/java/com/yahoo/searchdefinition/IncorrectSummaryTypesTestCase.java b/config-model/src/test/java/com/yahoo/searchdefinition/IncorrectSummaryTypesTestCase.java index 4e579fa62aa..76f8c05c1c8 100644 --- a/config-model/src/test/java/com/yahoo/searchdefinition/IncorrectSummaryTypesTestCase.java +++ b/config-model/src/test/java/com/yahoo/searchdefinition/IncorrectSummaryTypesTestCase.java @@ -11,7 +11,7 @@ import static org.junit.Assert.fail; * * @author bratseth */ -public class IncorrectSummaryTypesTestCase extends SchemaTestCase { +public class IncorrectSummaryTypesTestCase extends AbstractSchemaTestCase { @Test public void testImportingIncorrect() throws ParseException { try { diff --git a/config-model/src/test/java/com/yahoo/searchdefinition/IndexSettingsTestCase.java b/config-model/src/test/java/com/yahoo/searchdefinition/IndexSettingsTestCase.java index 12b9a7a9937..0e25798fbe2 100644 --- a/config-model/src/test/java/com/yahoo/searchdefinition/IndexSettingsTestCase.java +++ b/config-model/src/test/java/com/yahoo/searchdefinition/IndexSettingsTestCase.java @@ -17,23 +17,23 @@ import static org.junit.Assert.assertTrue; * * @author bratseth */ -public class IndexSettingsTestCase extends SchemaTestCase { +public class IndexSettingsTestCase extends AbstractSchemaTestCase { @Test public void testStemmingSettings() throws IOException, ParseException { - Search search = SearchBuilder.buildFromFile("src/test/examples/indexsettings.sd"); + Schema schema = SearchBuilder.buildFromFile("src/test/examples/indexsettings.sd"); - SDField usingDefault=(SDField) search.getDocument().getField("usingdefault"); - assertEquals(Stemming.SHORTEST,usingDefault.getStemming(search)); + SDField usingDefault=(SDField) schema.getDocument().getField("usingdefault"); + assertEquals(Stemming.SHORTEST,usingDefault.getStemming(schema)); - SDField notStemmed=(SDField) search.getDocument().getField("notstemmed"); - assertEquals(Stemming.NONE,notStemmed.getStemming(search)); + SDField notStemmed=(SDField) schema.getDocument().getField("notstemmed"); + assertEquals(Stemming.NONE,notStemmed.getStemming(schema)); - SDField allStemmed=(SDField) search.getDocument().getField("allstemmed"); - assertEquals(Stemming.SHORTEST,allStemmed.getStemming(search)); + SDField allStemmed=(SDField) schema.getDocument().getField("allstemmed"); + assertEquals(Stemming.SHORTEST,allStemmed.getStemming(schema)); - SDField multiStemmed=(SDField) search.getDocument().getField("multiplestems"); - assertEquals(Stemming.MULTIPLE, multiStemmed.getStemming(search)); + SDField multiStemmed=(SDField) schema.getDocument().getField("multiplestems"); + assertEquals(Stemming.MULTIPLE, multiStemmed.getStemming(schema)); } @Test @@ -52,10 +52,10 @@ public class IndexSettingsTestCase extends SchemaTestCase { " }", "}" )); - Search search = builder.getSearch(); - Index contentIndex = search.getIndex("content"); + Schema schema = builder.getSearch(); + Index contentIndex = schema.getIndex("content"); assertTrue(contentIndex.useInterleavedFeatures()); - Index extraIndex = search.getIndex("extra"); + Index extraIndex = schema.getIndex("extra"); assertTrue(extraIndex.useInterleavedFeatures()); } diff --git a/config-model/src/test/java/com/yahoo/searchdefinition/IndexingParsingTestCase.java b/config-model/src/test/java/com/yahoo/searchdefinition/IndexingParsingTestCase.java index 8deaddf9749..428114f47e5 100644 --- a/config-model/src/test/java/com/yahoo/searchdefinition/IndexingParsingTestCase.java +++ b/config-model/src/test/java/com/yahoo/searchdefinition/IndexingParsingTestCase.java @@ -11,7 +11,7 @@ import static org.junit.Assert.assertNotNull; * * @author frodelu */ -public class IndexingParsingTestCase extends SchemaTestCase { +public class IndexingParsingTestCase extends AbstractSchemaTestCase { @Test public void requireThatIndexingExpressionsCanBeParsed() throws Exception { diff --git a/config-model/src/test/java/com/yahoo/searchdefinition/MultipleSummariesTestCase.java b/config-model/src/test/java/com/yahoo/searchdefinition/MultipleSummariesTestCase.java index cbdb11f49fd..181f12083c4 100644 --- a/config-model/src/test/java/com/yahoo/searchdefinition/MultipleSummariesTestCase.java +++ b/config-model/src/test/java/com/yahoo/searchdefinition/MultipleSummariesTestCase.java @@ -11,7 +11,7 @@ import java.io.IOException; * * @author bratseth */ -public class MultipleSummariesTestCase extends SchemaTestCase { +public class MultipleSummariesTestCase extends AbstractSchemaTestCase { @Test public void testArrayImporting() throws IOException, ParseException { SearchBuilder.buildFromFile("src/test/examples/multiplesummaries.sd"); diff --git a/config-model/src/test/java/com/yahoo/searchdefinition/NameFieldCheckTestCase.java b/config-model/src/test/java/com/yahoo/searchdefinition/NameFieldCheckTestCase.java index 67fd54142f7..46da671d860 100644 --- a/config-model/src/test/java/com/yahoo/searchdefinition/NameFieldCheckTestCase.java +++ b/config-model/src/test/java/com/yahoo/searchdefinition/NameFieldCheckTestCase.java @@ -14,7 +14,7 @@ import static org.junit.Assert.fail; * * @author Lars Christian Jensen */ -public class NameFieldCheckTestCase extends SchemaTestCase { +public class NameFieldCheckTestCase extends AbstractSchemaTestCase { @Test public void testNameField() { @@ -52,7 +52,8 @@ public class NameFieldCheckTestCase extends SchemaTestCase { "}"); fail("Should throw exception."); } catch (Exception e) { - assertEquals("For search 'duplicatenamesinsearch', field 'grpphotoids64': Incompatible types. Expected Array<long> for index field 'grpphotoids64', got string.", e.getMessage()); + assertEquals("For schema 'duplicatenamesinsearch', field 'grpphotoids64': " + + "Incompatible types. Expected Array<long> for index field 'grpphotoids64', got string.", e.getMessage()); } } diff --git a/config-model/src/test/java/com/yahoo/searchdefinition/OutsideTestCase.java b/config-model/src/test/java/com/yahoo/searchdefinition/OutsideTestCase.java index 4f24f8a50d5..fda1f184a62 100644 --- a/config-model/src/test/java/com/yahoo/searchdefinition/OutsideTestCase.java +++ b/config-model/src/test/java/com/yahoo/searchdefinition/OutsideTestCase.java @@ -13,13 +13,13 @@ import static org.junit.Assert.assertTrue; * * @author bratseth */ -public class OutsideTestCase extends SchemaTestCase { +public class OutsideTestCase extends AbstractSchemaTestCase { @Test public void testOutsideIndex() throws IOException, ParseException { - Search search = SearchBuilder.buildFromFile("src/test/examples/outsidedoc.sd"); + Schema schema = SearchBuilder.buildFromFile("src/test/examples/outsidedoc.sd"); - Index defaultIndex=search.getIndex("default"); + Index defaultIndex= schema.getIndex("default"); assertTrue(defaultIndex.isPrefix()); assertEquals("default.default",defaultIndex.aliasIterator().next()); } diff --git a/config-model/src/test/java/com/yahoo/searchdefinition/PredicateDataTypeTestCase.java b/config-model/src/test/java/com/yahoo/searchdefinition/PredicateDataTypeTestCase.java index e1b0edb0d0c..e5afb481b80 100644 --- a/config-model/src/test/java/com/yahoo/searchdefinition/PredicateDataTypeTestCase.java +++ b/config-model/src/test/java/com/yahoo/searchdefinition/PredicateDataTypeTestCase.java @@ -136,7 +136,7 @@ public class PredicateDataTypeTestCase { String sd = searchSd(predicateFieldSd("indexing: summary | index | attribute\nindex { arity: 2 }")); exception.expect(IllegalArgumentException.class); - exception.expectMessage("For search 'p', field 'pf': Use 'attribute' instead of 'index'. This will require a refeed if you have upgraded."); + exception.expectMessage("For schema 'p', field 'pf': Use 'attribute' instead of 'index'. This will require a refeed if you have upgraded."); SearchBuilder.createFromString(sd); } @@ -145,7 +145,7 @@ public class PredicateDataTypeTestCase { String sd = searchSd(predicateFieldSd("indexing: summary | index \nindex { arity: 2 }")); exception.expect(IllegalArgumentException.class); - exception.expectMessage("For search 'p', field 'pf': Use 'attribute' instead of 'index'. This will require a refeed if you have upgraded."); + exception.expectMessage("For schema 'p', field 'pf': Use 'attribute' instead of 'index'. This will require a refeed if you have upgraded."); SearchBuilder.createFromString(sd); } diff --git a/config-model/src/test/java/com/yahoo/searchdefinition/RankProfileRegistryTest.java b/config-model/src/test/java/com/yahoo/searchdefinition/RankProfileRegistryTest.java index 21ab7a75f46..d2486395340 100644 --- a/config-model/src/test/java/com/yahoo/searchdefinition/RankProfileRegistryTest.java +++ b/config-model/src/test/java/com/yahoo/searchdefinition/RankProfileRegistryTest.java @@ -33,24 +33,24 @@ public class RankProfileRegistryTest { @Test(expected = IllegalArgumentException.class) public void testRankProfileDuplicateNameIsIllegal() { - Search search = new Search("foo"); - RankProfileRegistry rankProfileRegistry = RankProfileRegistry.createRankProfileRegistryWithBuiltinRankProfiles(search); - RankProfile barRankProfile = new RankProfile("bar", search, rankProfileRegistry, search.rankingConstants()); + Schema schema = new Schema("foo"); + RankProfileRegistry rankProfileRegistry = RankProfileRegistry.createRankProfileRegistryWithBuiltinRankProfiles(schema); + RankProfile barRankProfile = new RankProfile("bar", schema, rankProfileRegistry, schema.rankingConstants()); rankProfileRegistry.add(barRankProfile); rankProfileRegistry.add(barRankProfile); } @Test public void testRankProfileDuplicateNameLegalForOverridableRankProfiles() { - Search search = new Search("foo"); - RankProfileRegistry rankProfileRegistry = RankProfileRegistry.createRankProfileRegistryWithBuiltinRankProfiles(search); + Schema schema = new Schema("foo"); + RankProfileRegistry rankProfileRegistry = RankProfileRegistry.createRankProfileRegistryWithBuiltinRankProfiles(schema); for (String rankProfileName : RankProfileRegistry.overridableRankProfileNames) { - assertNull(rankProfileRegistry.get(search, rankProfileName).getFunctions().get("foo")); - RankProfile rankProfileWithAddedFunction = new RankProfile(rankProfileName, search, rankProfileRegistry, search.rankingConstants()); + assertNull(rankProfileRegistry.get(schema, rankProfileName).getFunctions().get("foo")); + RankProfile rankProfileWithAddedFunction = new RankProfile(rankProfileName, schema, rankProfileRegistry, schema.rankingConstants()); rankProfileWithAddedFunction.addFunction(new ExpressionFunction("foo", RankingExpression.from("1+2")), true); rankProfileRegistry.add(rankProfileWithAddedFunction); - assertNotNull(rankProfileRegistry.get(search, rankProfileName).getFunctions().get("foo")); + assertNotNull(rankProfileRegistry.get(schema, rankProfileName).getFunctions().get("foo")); } } diff --git a/config-model/src/test/java/com/yahoo/searchdefinition/RankProfileTestCase.java b/config-model/src/test/java/com/yahoo/searchdefinition/RankProfileTestCase.java index de43dc94c71..857da3068cd 100644 --- a/config-model/src/test/java/com/yahoo/searchdefinition/RankProfileTestCase.java +++ b/config-model/src/test/java/com/yahoo/searchdefinition/RankProfileTestCase.java @@ -39,18 +39,18 @@ import static org.junit.Assert.fail; * * @author bratseth */ -public class RankProfileTestCase extends SchemaTestCase { +public class RankProfileTestCase extends AbstractSchemaTestCase { @Test public void testRankProfileInheritance() { - Search search = new Search("test"); - RankProfileRegistry rankProfileRegistry = RankProfileRegistry.createRankProfileRegistryWithBuiltinRankProfiles(search); + Schema schema = new Schema("test"); + RankProfileRegistry rankProfileRegistry = RankProfileRegistry.createRankProfileRegistryWithBuiltinRankProfiles(schema); SDDocumentType document = new SDDocumentType("test"); SDField a = document.addField("a", DataType.STRING); a.setRankType(RankType.IDENTITY); document.addField("b", DataType.STRING); - search.addDocument(document); - RankProfile child = new RankProfile("child", search, rankProfileRegistry, search.rankingConstants()); + schema.addDocument(document); + RankProfile child = new RankProfile("child", schema, rankProfileRegistry, schema.rankingConstants()); child.setInherited("default"); rankProfileRegistry.add(child); @@ -249,10 +249,10 @@ public class RankProfileTestCase extends SchemaTestCase { SearchBuilder builder = new SearchBuilder(rankProfileRegistry); builder.importString(sd); builder.build(); - Search search = builder.getSearch(); - AttributeFields attributeFields = new AttributeFields(search); - verifyRankProfile(rankProfileRegistry.get(search, "parent"), attributeFields, deployProperties, termwiseLimit); - verifyRankProfile(rankProfileRegistry.get(search, "child"), attributeFields, deployProperties, termwiseLimit); + Schema schema = builder.getSearch(); + AttributeFields attributeFields = new AttributeFields(schema); + verifyRankProfile(rankProfileRegistry.get(schema, "parent"), attributeFields, deployProperties, termwiseLimit); + verifyRankProfile(rankProfileRegistry.get(schema, "child"), attributeFields, deployProperties, termwiseLimit); } private void verifyRankProfile(RankProfile rankProfile, AttributeFields attributeFields, ModelContext.Properties deployProperties, @@ -291,13 +291,13 @@ public class RankProfileTestCase extends SchemaTestCase { " rank-profile p2 {}", "}")); builder.build(); - Search search = builder.getSearch(); + Schema schema = builder.getSearch(); assertEquals(4, registry.all().size()); - assertAttributeTypeSettings(registry.get(search, "default"), search); - assertAttributeTypeSettings(registry.get(search, "unranked"), search); - assertAttributeTypeSettings(registry.get(search, "p1"), search); - assertAttributeTypeSettings(registry.get(search, "p2"), search); + assertAttributeTypeSettings(registry.get(schema, "default"), schema); + assertAttributeTypeSettings(registry.get(schema, "unranked"), schema); + assertAttributeTypeSettings(registry.get(schema, "p1"), schema); + assertAttributeTypeSettings(registry.get(schema, "p2"), schema); } @Test @@ -318,12 +318,12 @@ public class RankProfileTestCase extends SchemaTestCase { } } - private static RawRankProfile createRawRankProfile(RankProfile profile, Search search) { - return new RawRankProfile(profile, new LargeRankExpressions(new MockFileRegistry()), new QueryProfileRegistry(), new ImportedMlModels(), new AttributeFields(search), new TestProperties()); + private static RawRankProfile createRawRankProfile(RankProfile profile, Schema schema) { + return new RawRankProfile(profile, new LargeRankExpressions(new MockFileRegistry()), new QueryProfileRegistry(), new ImportedMlModels(), new AttributeFields(schema), new TestProperties()); } - private static void assertAttributeTypeSettings(RankProfile profile, Search search) { - RawRankProfile rawProfile = createRawRankProfile(profile, search); + private static void assertAttributeTypeSettings(RankProfile profile, Schema schema) { + RawRankProfile rawProfile = createRawRankProfile(profile, schema); assertEquals("tensor(x[10])", findProperty(rawProfile.configProperties(), "vespa.type.attribute.a").get()); assertEquals("tensor(y{})", findProperty(rawProfile.configProperties(), "vespa.type.attribute.b").get()); assertEquals("tensor(x[5])", findProperty(rawProfile.configProperties(), "vespa.type.attribute.c").get()); @@ -340,13 +340,13 @@ public class RankProfileTestCase extends SchemaTestCase { " rank-profile p2 {}", "}")); builder.build(true); - Search search = builder.getSearch(); + Schema schema = builder.getSearch(); assertEquals(4, registry.all().size()); - assertQueryFeatureTypeSettings(registry.get(search, "default"), search); - assertQueryFeatureTypeSettings(registry.get(search, "unranked"), search); - assertQueryFeatureTypeSettings(registry.get(search, "p1"), search); - assertQueryFeatureTypeSettings(registry.get(search, "p2"), search); + assertQueryFeatureTypeSettings(registry.get(schema, "default"), schema); + assertQueryFeatureTypeSettings(registry.get(schema, "unranked"), schema); + assertQueryFeatureTypeSettings(registry.get(schema, "p1"), schema); + assertQueryFeatureTypeSettings(registry.get(schema, "p2"), schema); } private static QueryProfileRegistry setupQueryProfileTypes() { @@ -365,8 +365,8 @@ public class RankProfileTestCase extends SchemaTestCase { return registry; } - private static void assertQueryFeatureTypeSettings(RankProfile profile, Search search) { - RawRankProfile rawProfile =createRawRankProfile(profile, search); + private static void assertQueryFeatureTypeSettings(RankProfile profile, Schema schema) { + RawRankProfile rawProfile =createRawRankProfile(profile, schema); assertEquals("tensor(x[10])", findProperty(rawProfile.configProperties(), "vespa.type.query.tensor1").get()); assertEquals("tensor(y{})", findProperty(rawProfile.configProperties(), "vespa.type.query.tensor2").get()); assertFalse(findProperty(rawProfile.configProperties(), "vespa.type.query.tensor3").isPresent()); diff --git a/config-model/src/test/java/com/yahoo/searchdefinition/RankPropertiesTestCase.java b/config-model/src/test/java/com/yahoo/searchdefinition/RankPropertiesTestCase.java index 987e92c8c68..e2ca8ac4f65 100644 --- a/config-model/src/test/java/com/yahoo/searchdefinition/RankPropertiesTestCase.java +++ b/config-model/src/test/java/com/yahoo/searchdefinition/RankPropertiesTestCase.java @@ -18,7 +18,7 @@ import static org.junit.Assert.assertEquals; /** * @author bratseth */ -public class RankPropertiesTestCase extends SchemaTestCase { +public class RankPropertiesTestCase extends AbstractSchemaTestCase { @Test public void testRankPropertyInheritance() throws ParseException { @@ -49,12 +49,12 @@ public class RankPropertiesTestCase extends SchemaTestCase { " }", "}")); builder.build(); - Search search = builder.getSearch(); - AttributeFields attributeFields = new AttributeFields(search); + Schema schema = builder.getSearch(); + AttributeFields attributeFields = new AttributeFields(schema); { // Check declared model - RankProfile parent = rankProfileRegistry.get(search, "parent"); + RankProfile parent = rankProfileRegistry.get(schema, "parent"); assertEquals("query(a) = 1500", parent.getRankProperties().get(0).toString()); // Check derived model @@ -64,11 +64,11 @@ public class RankPropertiesTestCase extends SchemaTestCase { { // Check declared model - RankProfile parent = rankProfileRegistry.get(search, "child"); + RankProfile parent = rankProfileRegistry.get(schema, "child"); assertEquals("query(a) = 2000", parent.getRankProperties().get(0).toString()); // Check derived model - RawRankProfile rawChild = new RawRankProfile(rankProfileRegistry.get(search, "child"), + RawRankProfile rawChild = new RawRankProfile(rankProfileRegistry.get(schema, "child"), new LargeRankExpressions(new MockFileRegistry()), new QueryProfileRegistry(), new ImportedMlModels(), @@ -126,8 +126,8 @@ public class RankPropertiesTestCase extends SchemaTestCase { " }", "}")); builder.build(); - Search search = builder.getSearch(); - RankProfile a = rankProfileRegistry.get(search, "a"); + Schema schema = builder.getSearch(); + RankProfile a = rankProfileRegistry.get(schema, "a"); List<RankProfile.ExecuteOperation> operations = a.getExecuteOperations(); assertEquals(3, operations.size()); assertEquals(RankProfile.ExecuteOperation.Phase.onmatch, operations.get(0).phase); @@ -140,7 +140,7 @@ public class RankPropertiesTestCase extends SchemaTestCase { assertEquals("synthetic_attribute_c", operations.get(2).attribute); assertEquals("--", operations.get(2).operation); - AttributeFields attributeFields = new AttributeFields(search); + AttributeFields attributeFields = new AttributeFields(schema); RawRankProfile raw = new RawRankProfile(a, new LargeRankExpressions(new MockFileRegistry()), new QueryProfileRegistry(), new ImportedMlModels(), attributeFields, new TestProperties()); assertEquals(7, raw.configProperties().size()); assertEquals("(vespa.execute.onmatch.attribute, synthetic_attribute_a)", raw.configProperties().get(0).toString()); diff --git a/config-model/src/test/java/com/yahoo/searchdefinition/RankingConstantTest.java b/config-model/src/test/java/com/yahoo/searchdefinition/RankingConstantTest.java index eb71e73be60..8bfd9c898e4 100644 --- a/config-model/src/test/java/com/yahoo/searchdefinition/RankingConstantTest.java +++ b/config-model/src/test/java/com/yahoo/searchdefinition/RankingConstantTest.java @@ -43,9 +43,9 @@ public class RankingConstantTest { "}" )); searchBuilder.build(); - Search search = searchBuilder.getSearch(); + Schema schema = searchBuilder.getSearch(); - Iterator<RankingConstant> constantIterator = search.rankingConstants().asMap().values().iterator(); + Iterator<RankingConstant> constantIterator = schema.rankingConstants().asMap().values().iterator(); RankingConstant constant = constantIterator.next(); assertEquals(TENSOR_NAME, constant.getName()); assertEquals(TENSOR_FILE, constant.getFileName()); @@ -101,8 +101,8 @@ public class RankingConstantTest { "}" )); searchBuilder.build(); - Search search = searchBuilder.getSearch(); - RankingConstant constant = search.rankingConstants().asMap().values().iterator().next(); + Schema schema = searchBuilder.getSearch(); + RankingConstant constant = schema.rankingConstants().asMap().values().iterator().next(); assertEquals("simplename", constant.getFileName()); } @@ -120,8 +120,8 @@ public class RankingConstantTest { "}" )); searchBuilder.build(); - Search search = searchBuilder.getSearch(); - RankingConstant constant = search.rankingConstants().asMap().values().iterator().next(); + Schema schema = searchBuilder.getSearch(); + RankingConstant constant = schema.rankingConstants().asMap().values().iterator().next(); assertEquals(RankingConstant.PathType.URI, constant.getPathType()); assertEquals("http://somewhere.far.away/in/another-galaxy", constant.getUri()); } @@ -140,8 +140,8 @@ public class RankingConstantTest { "}" )); searchBuilder.build(); - Search search = searchBuilder.getSearch(); - RankingConstant constant = search.rankingConstants().asMap().values().iterator().next(); + Schema schema = searchBuilder.getSearch(); + RankingConstant constant = schema.rankingConstants().asMap().values().iterator().next(); assertEquals(RankingConstant.PathType.URI, constant.getPathType()); assertEquals("https://somewhere.far.away:4443/in/another-galaxy", constant.getUri()); } @@ -160,8 +160,8 @@ public class RankingConstantTest { "}" )); searchBuilder.build(); - Search search = searchBuilder.getSearch(); - RankingConstant constant = search.rankingConstants().asMap().values().iterator().next(); + Schema schema = searchBuilder.getSearch(); + RankingConstant constant = schema.rankingConstants().asMap().values().iterator().next(); assertEquals(RankingConstant.PathType.URI, constant.getPathType()); assertEquals("http://somewhere.far.away:4080/in/another-galaxy", constant.getUri()); } @@ -180,8 +180,8 @@ public class RankingConstantTest { "}" )); searchBuilder.build(); - Search search = searchBuilder.getSearch(); - RankingConstant constant = search.rankingConstants().asMap().values().iterator().next(); + Schema schema = searchBuilder.getSearch(); + RankingConstant constant = schema.rankingConstants().asMap().values().iterator().next(); assertEquals(RankingConstant.PathType.URI, constant.getPathType()); assertEquals("http:somewhere.far.away/in/another-galaxy", constant.getUri()); } diff --git a/config-model/src/test/java/com/yahoo/searchdefinition/RankingExpressionConstantsTestCase.java b/config-model/src/test/java/com/yahoo/searchdefinition/RankingExpressionConstantsTestCase.java index 9af026e164d..19a50817f75 100644 --- a/config-model/src/test/java/com/yahoo/searchdefinition/RankingExpressionConstantsTestCase.java +++ b/config-model/src/test/java/com/yahoo/searchdefinition/RankingExpressionConstantsTestCase.java @@ -19,7 +19,7 @@ import static org.junit.Assert.*; /** * @author bratseth */ -public class RankingExpressionConstantsTestCase extends SchemaTestCase { +public class RankingExpressionConstantsTestCase extends AbstractSchemaTestCase { @Test public void testConstants() throws ParseException { @@ -67,7 +67,7 @@ public class RankingExpressionConstantsTestCase extends SchemaTestCase { "\n" + "}\n"); builder.build(); - Search s = builder.getSearch(); + Schema s = builder.getSearch(); RankProfile parent = rankProfileRegistry.get(s, "parent").compile(queryProfileRegistry, new ImportedMlModels()); assertEquals("0.0", parent.getFirstPhaseRanking().getRoot().toString()); @@ -111,7 +111,7 @@ public class RankingExpressionConstantsTestCase extends SchemaTestCase { "\n" + "}\n"); builder.build(); - Search s = builder.getSearch(); + Schema s = builder.getSearch(); try { rankProfileRegistry.get(s, "test").compile(new QueryProfileRegistry(), new ImportedMlModels()); fail("Should have caused an exception"); @@ -142,7 +142,7 @@ public class RankingExpressionConstantsTestCase extends SchemaTestCase { "\n" + "}\n"); builder.build(); - Search s = builder.getSearch(); + Schema s = builder.getSearch(); RankProfile profile = rankProfileRegistry.get(s, "test"); assertEquals("safeLog(popShareSlowDecaySignal,-9.21034037)", profile.getFunctions().get("POP_SLOW_SCORE").function().getBody().getRoot().toString()); } @@ -170,7 +170,7 @@ public class RankingExpressionConstantsTestCase extends SchemaTestCase { "\n" + "}\n"); builder.build(); - Search s = builder.getSearch(); + Schema s = builder.getSearch(); RankProfile profile = rankProfileRegistry.get(s, "test"); assertEquals("safeLog(popShareSlowDecaySignal,myValue)", profile.getFunctions().get("POP_SLOW_SCORE").function().getBody().getRoot().toString()); assertEquals("safeLog(popShareSlowDecaySignal,-9.21034037)", @@ -194,7 +194,7 @@ public class RankingExpressionConstantsTestCase extends SchemaTestCase { "\n" + "}\n"); builder.build(); - Search s = builder.getSearch(); + Schema s = builder.getSearch(); RankProfile profile = rankProfileRegistry.get(s, "test"); assertEquals("k1 + (k2 + k3) / 100000000.0", profile.compile(new QueryProfileRegistry(), new ImportedMlModels()).getFunctions().get("rank_default").function().getBody().getRoot().toString()); @@ -220,7 +220,7 @@ public class RankingExpressionConstantsTestCase extends SchemaTestCase { "\n" + "}\n"); builder.build(); - Search s = builder.getSearch(); + Schema s = builder.getSearch(); RankProfile profile = rankProfileRegistry.get(s, "test"); assertEquals("0.5 + 50 * (attribute(rating_yelp) - 3)", profile.compile(new QueryProfileRegistry(), new ImportedMlModels()).getFunctions().get("rank_default").function().getBody().getRoot().toString()); diff --git a/config-model/src/test/java/com/yahoo/searchdefinition/RankingExpressionInliningTestCase.java b/config-model/src/test/java/com/yahoo/searchdefinition/RankingExpressionInliningTestCase.java index fdc08130375..238e71a6340 100644 --- a/config-model/src/test/java/com/yahoo/searchdefinition/RankingExpressionInliningTestCase.java +++ b/config-model/src/test/java/com/yahoo/searchdefinition/RankingExpressionInliningTestCase.java @@ -23,7 +23,7 @@ import static org.junit.Assert.assertTrue; /** * @author bratseth */ -public class RankingExpressionInliningTestCase extends SchemaTestCase { +public class RankingExpressionInliningTestCase extends AbstractSchemaTestCase { @Test public void testFunctionInliningPreserveArithmeticOrdering() throws ParseException { @@ -66,7 +66,7 @@ public class RankingExpressionInliningTestCase extends SchemaTestCase { "\n" + "}\n"); builder.build(); - Search s = builder.getSearch(); + Schema s = builder.getSearch(); RankProfile parent = rankProfileRegistry.get(s, "parent").compile(new QueryProfileRegistry(), new ImportedMlModels()); assertEquals("7.0 * (3 + attribute(a) + attribute(b) * (attribute(a) * 3 + if (7.0 < attribute(a), 1, 2) == 0))", @@ -126,7 +126,7 @@ public class RankingExpressionInliningTestCase extends SchemaTestCase { "\n" + "}\n"); builder.build(); - Search s = builder.getSearch(); + Schema s = builder.getSearch(); RankProfile parent = rankProfileRegistry.get(s, "parent").compile(new QueryProfileRegistry(), new ImportedMlModels()); assertEquals("17.0", parent.getFirstPhaseRanking().getRoot().toString()); @@ -182,7 +182,7 @@ public class RankingExpressionInliningTestCase extends SchemaTestCase { "\n" + "}\n"); builder.build(); - Search s = builder.getSearch(); + Schema s = builder.getSearch(); RankProfile test = rankProfileRegistry.get(s, "test").compile(new QueryProfileRegistry(), new ImportedMlModels()); assertEquals("attribute(a) + C + (attribute(b) + 1)", test.getFirstPhaseRanking().getRoot().toString()); @@ -216,7 +216,7 @@ public class RankingExpressionInliningTestCase extends SchemaTestCase { " }\n" + "}\n"); builder.build(); - Search s = builder.getSearch(); + Schema s = builder.getSearch(); RankProfile test = rankProfileRegistry.get(s, "test").compile(new QueryProfileRegistry(), new ImportedMlModels()); assertEquals("foo(2)", test.getFirstPhaseRanking().getRoot().toString()); assertTrue("Does not contain expected warning", deployLogger.contains("Function 'foo' replaces " + @@ -246,9 +246,9 @@ public class RankingExpressionInliningTestCase extends SchemaTestCase { return b.toString(); } - private String getRankingExpression(String name, RankProfile rankProfile, Search search) { + private String getRankingExpression(String name, RankProfile rankProfile, Schema schema) { Optional<String> rankExpression = - new RawRankProfile(rankProfile, new LargeRankExpressions(new MockFileRegistry()), new QueryProfileRegistry(), new ImportedMlModels(), new AttributeFields(search), new TestProperties()) + new RawRankProfile(rankProfile, new LargeRankExpressions(new MockFileRegistry()), new QueryProfileRegistry(), new ImportedMlModels(), new AttributeFields(schema), new TestProperties()) .configProperties() .stream() .filter(r -> r.getFirst().equals("rankingExpression(" + name + ").rankingScript")) diff --git a/config-model/src/test/java/com/yahoo/searchdefinition/RankingExpressionShadowingTestCase.java b/config-model/src/test/java/com/yahoo/searchdefinition/RankingExpressionShadowingTestCase.java index f84f3d8c83f..231fa531770 100644 --- a/config-model/src/test/java/com/yahoo/searchdefinition/RankingExpressionShadowingTestCase.java +++ b/config-model/src/test/java/com/yahoo/searchdefinition/RankingExpressionShadowingTestCase.java @@ -21,7 +21,7 @@ import static org.junit.Assert.assertEquals; /** * @author lesters */ -public class RankingExpressionShadowingTestCase extends SchemaTestCase { +public class RankingExpressionShadowingTestCase extends AbstractSchemaTestCase { @Test public void testBasicFunctionShadowing() throws ParseException { @@ -46,7 +46,7 @@ public class RankingExpressionShadowingTestCase extends SchemaTestCase { "\n" + "}\n"); builder.build(); - Search s = builder.getSearch(); + Schema s = builder.getSearch(); RankProfile test = rankProfileRegistry.get(s, "test").compile(new QueryProfileRegistry(), new ImportedMlModels()); List<Pair<String, String>> testRankProperties = createRawRankProfile(test, new QueryProfileRegistry(), s).configProperties(); assertEquals("(rankingExpression(sin@).rankingScript, 2 * 2)", @@ -87,7 +87,7 @@ public class RankingExpressionShadowingTestCase extends SchemaTestCase { "\n" + "}\n"); builder.build(); - Search s = builder.getSearch(); + Schema s = builder.getSearch(); RankProfile test = rankProfileRegistry.get(s, "test").compile(new QueryProfileRegistry(), new ImportedMlModels()); List<Pair<String, String>> testRankProperties = createRawRankProfile(test, new QueryProfileRegistry(), s).configProperties(); assertEquals("(rankingExpression(tan@).rankingScript, 2 * 2)", @@ -133,7 +133,7 @@ public class RankingExpressionShadowingTestCase extends SchemaTestCase { "\n" + "}\n"); builder.build(); - Search s = builder.getSearch(); + Schema s = builder.getSearch(); RankProfile test = rankProfileRegistry.get(s, "test").compile(new QueryProfileRegistry(), new ImportedMlModels()); List<Pair<String, String>> testRankProperties = createRawRankProfile(test, new QueryProfileRegistry(), s).configProperties(); assertEquals("(rankingExpression(sin@).rankingScript, 4.0 * 4.0)", @@ -194,7 +194,7 @@ public class RankingExpressionShadowingTestCase extends SchemaTestCase { " }\n" + "}\n"); builder.build(); - Search s = builder.getSearch(); + Schema s = builder.getSearch(); RankProfile test = rankProfileRegistry.get(s, "test").compile(queryProfiles, new ImportedMlModels()); List<Pair<String, String>> testRankProperties = createRawRankProfile(test, queryProfiles, s).configProperties(); assertEquals("(rankingExpression(autogenerated_ranking_feature@).rankingScript, reduce(query(q) * constant(W_hidden), sum, input) + constant(b_input))", @@ -213,12 +213,12 @@ public class RankingExpressionShadowingTestCase extends SchemaTestCase { testRankProperties.get(8).toString()); } - private static RawRankProfile createRawRankProfile(RankProfile profile, QueryProfileRegistry queryProfiles, Search search) { + private static RawRankProfile createRawRankProfile(RankProfile profile, QueryProfileRegistry queryProfiles, Schema schema) { return new RawRankProfile(profile, new LargeRankExpressions(new MockFileRegistry()), queryProfiles, new ImportedMlModels(), - new AttributeFields(search), + new AttributeFields(schema), new TestProperties()); } diff --git a/config-model/src/test/java/com/yahoo/searchdefinition/RankingExpressionValidationTestCase.java b/config-model/src/test/java/com/yahoo/searchdefinition/RankingExpressionValidationTestCase.java index 66e773cf3f8..a66a4cd8648 100644 --- a/config-model/src/test/java/com/yahoo/searchdefinition/RankingExpressionValidationTestCase.java +++ b/config-model/src/test/java/com/yahoo/searchdefinition/RankingExpressionValidationTestCase.java @@ -11,7 +11,7 @@ import static org.junit.Assert.fail; /** * @author bratseth */ -public class RankingExpressionValidationTestCase extends SchemaTestCase { +public class RankingExpressionValidationTestCase extends AbstractSchemaTestCase { @Test public void testInvalidExpressionProducesException() throws ParseException { @@ -22,8 +22,8 @@ public class RankingExpressionValidationTestCase extends SchemaTestCase { private void assertFailsExpression(String expression) throws ParseException { try { RankProfileRegistry registry = new RankProfileRegistry(); - Search search = importWithExpression(expression, registry); - new DerivedConfiguration(search, registry); // cause rank profile parsing + Schema schema = importWithExpression(expression, registry); + new DerivedConfiguration(schema, registry); // cause rank profile parsing fail("No exception on incorrect ranking expression " + expression); } catch (IllegalArgumentException e) { // Success @@ -31,7 +31,7 @@ public class RankingExpressionValidationTestCase extends SchemaTestCase { } } - private Search importWithExpression(String expression, RankProfileRegistry registry) throws ParseException { + private Schema importWithExpression(String expression, RankProfileRegistry registry) throws ParseException { SearchBuilder builder = new SearchBuilder(registry); builder.importString("search test {" + " document test { " + diff --git a/config-model/src/test/java/com/yahoo/searchdefinition/ReservedWordsAsFieldNamesTestCase.java b/config-model/src/test/java/com/yahoo/searchdefinition/ReservedWordsAsFieldNamesTestCase.java index 9bb588a690b..f893f532f09 100644 --- a/config-model/src/test/java/com/yahoo/searchdefinition/ReservedWordsAsFieldNamesTestCase.java +++ b/config-model/src/test/java/com/yahoo/searchdefinition/ReservedWordsAsFieldNamesTestCase.java @@ -11,14 +11,14 @@ import static org.junit.Assert.assertNotNull; /** * @author bratseth */ -public class ReservedWordsAsFieldNamesTestCase extends SchemaTestCase { +public class ReservedWordsAsFieldNamesTestCase extends AbstractSchemaTestCase { @Test public void testIt() throws IOException, ParseException { - Search search = SearchBuilder.buildFromFile("src/test/examples/reserved_words_as_field_names.sd"); - assertNotNull(search.getDocument().getField("inline")); - assertNotNull(search.getDocument().getField("constants")); - assertNotNull(search.getDocument().getField("reference")); + Schema schema = SearchBuilder.buildFromFile("src/test/examples/reserved_words_as_field_names.sd"); + assertNotNull(schema.getDocument().getField("inline")); + assertNotNull(schema.getDocument().getField("constants")); + assertNotNull(schema.getDocument().getField("reference")); } } diff --git a/config-model/src/test/java/com/yahoo/searchdefinition/SearchImporterTestCase.java b/config-model/src/test/java/com/yahoo/searchdefinition/SchemaImporterTestCase.java index 990fe8cb3d0..1ae44e30c32 100644 --- a/config-model/src/test/java/com/yahoo/searchdefinition/SearchImporterTestCase.java +++ b/config-model/src/test/java/com/yahoo/searchdefinition/SchemaImporterTestCase.java @@ -30,7 +30,7 @@ import static org.junit.Assert.fail; * * @author bratseth */ -public class SearchImporterTestCase extends SchemaTestCase { +public class SchemaImporterTestCase extends AbstractSchemaTestCase { @Test @SuppressWarnings("deprecation") @@ -39,30 +39,30 @@ public class SearchImporterTestCase extends SchemaTestCase { SearchBuilder sb = new SearchBuilder(rankProfileRegistry, new QueryProfileRegistry()); sb.importFile("src/test/examples/simple.sd"); sb.build(); - Search search = sb.getSearch(); - assertEquals("simple",search.getName()); - assertTrue(search.hasDocument()); + Schema schema = sb.getSearch(); + assertEquals("simple", schema.getName()); + assertTrue(schema.hasDocument()); - SDDocumentType document = search.getDocument(); + SDDocumentType document = schema.getDocument(); assertEquals("simple", document.getName()); assertEquals(23, document.getFieldCount()); SDField field; Attribute attribute; - new MakeAliases(search, new BaseDeployLogger(), rankProfileRegistry, new QueryProfiles()).process(true, false); + new MakeAliases(schema, new BaseDeployLogger(), rankProfileRegistry, new QueryProfiles()).process(true, false); // First field field=(SDField) document.getField("title"); assertEquals(DataType.STRING,field.getDataType()); assertEquals("{ input title | tokenize normalize stem:\"BEST\" | summary title | index title; }", field.getIndexingScript().toString()); - assertFalse(search.getIndex("default").isPrefix()); - assertTrue(search.getIndex("title").isPrefix()); - Iterator<String> titleAliases=search.getIndex("title").aliasIterator(); + assertFalse(schema.getIndex("default").isPrefix()); + assertTrue(schema.getIndex("title").isPrefix()); + Iterator<String> titleAliases= schema.getIndex("title").aliasIterator(); assertEquals("aliaz",titleAliases.next()); assertEquals("analias.totitle",titleAliases.next()); assertEquals("analias.todefault", - search.getIndex("default").aliasIterator().next()); + schema.getIndex("default").aliasIterator().next()); assertEquals(RankType.IDENTITY, field.getRankType()); assertEquals(0, field.getAttributes().size()); assertNull(field.getStemming()); @@ -78,7 +78,7 @@ public class SearchImporterTestCase extends SchemaTestCase { field.getSummaryField("dyndesc").getTransform()); assertNull(field.getStemming()); assertTrue(field.getNormalizing().doRemoveAccents()); - assertEquals("hallo",search.getIndex("description").aliasIterator().next()); + assertEquals("hallo", schema.getIndex("description").aliasIterator().next()); // Third field field=(SDField) document.getField("chatter"); @@ -104,24 +104,24 @@ public class SearchImporterTestCase extends SchemaTestCase { assertEquals(1, field.getAttributes().size()); // Seventh field - field= search.getConcreteField("categories"); + field= schema.getConcreteField("categories"); assertEquals("{ input categories_src | lowercase | normalize | tokenize normalize stem:\"BEST\" | index categories; }", field.getIndexingScript().toString()); assertTrue(field.isHeader()); // Eight field - field= search.getConcreteField("categoriesagain"); + field= schema.getConcreteField("categoriesagain"); assertEquals("{ input categoriesagain_src | lowercase | normalize | tokenize normalize stem:\"BEST\" | index categoriesagain; }", field.getIndexingScript().toString()); assertTrue(field.isHeader()); // Ninth field - field= search.getConcreteField("exactemento"); + field= schema.getConcreteField("exactemento"); assertEquals("{ input exactemento_src | lowercase | tokenize normalize stem:\"BEST\" | index exactemento | summary exactemento; }", field.getIndexingScript().toString()); // Tenth field - field = search.getConcreteField("category_arr"); + field = schema.getConcreteField("category_arr"); assertEquals(1, field.getAttributes().size()); attribute = field.getAttributes().get("category_arr"); assertNotNull(attribute); @@ -131,7 +131,7 @@ public class SearchImporterTestCase extends SchemaTestCase { assertTrue(field.isHeader()); // Eleventh field - field = search.getConcreteField("measurement_arr"); + field = schema.getConcreteField("measurement_arr"); assertEquals(1, field.getAttributes().size()); attribute = field.getAttributes().get("measurement_arr"); assertEquals("measurement_arr", attribute.getName()); @@ -139,24 +139,24 @@ public class SearchImporterTestCase extends SchemaTestCase { assertEquals(Attribute.CollectionType.ARRAY, attribute.getCollectionType()); // Rank Profiles - RankProfile profile=rankProfileRegistry.get(search, "default"); + RankProfile profile=rankProfileRegistry.get(schema, "default"); assertNotNull(profile); assertNull(profile.getInheritedName()); assertNull(profile.getDeclaredRankSetting("measurement", RankProfile.RankSetting.Type.RANKTYPE)); assertEquals(RankType.EMPTY, profile.getRankSetting("measurement", RankProfile.RankSetting.Type.RANKTYPE).getValue()); - profile=rankProfileRegistry.get(search, "experimental"); + profile=rankProfileRegistry.get(schema, "experimental"); assertNotNull(profile); assertEquals("default",profile.getInheritedName()); assertEquals(RankType.IDENTITY, profile.getDeclaredRankSetting("measurement", RankProfile.RankSetting.Type.RANKTYPE).getValue()); - profile=rankProfileRegistry.get(search, "other"); + profile=rankProfileRegistry.get(schema, "other"); assertNotNull(profile); assertEquals("experimental",profile.getInheritedName()); // The extra-document field - SDField exact=search.getConcreteField("exact"); + SDField exact= schema.getConcreteField("exact"); assertNotNull("Extra field was parsed",exact); assertEquals("exact",exact.getName()); assertEquals(Stemming.NONE,exact.getStemming()); @@ -178,10 +178,10 @@ public class SearchImporterTestCase extends SchemaTestCase { @Test public void testIdImporting() throws IOException, ParseException { - Search search = SearchBuilder.buildFromFile("src/test/examples/strange.sd"); - SDField idecidemyide=(SDField) search.getDocument().getField("idecidemyide"); + Schema schema = SearchBuilder.buildFromFile("src/test/examples/strange.sd"); + SDField idecidemyide=(SDField) schema.getDocument().getField("idecidemyide"); assertEquals(5,idecidemyide.getId()); - SDField sodoi=(SDField) search.getDocument().getField("sodoi"); + SDField sodoi=(SDField) schema.getDocument().getField("sodoi"); assertEquals(7,sodoi.getId()); } diff --git a/config-model/src/test/java/com/yahoo/searchdefinition/SchemaParsingTestCase.java b/config-model/src/test/java/com/yahoo/searchdefinition/SchemaParsingTestCase.java index 3a6dd00bc0e..984140be12c 100644 --- a/config-model/src/test/java/com/yahoo/searchdefinition/SchemaParsingTestCase.java +++ b/config-model/src/test/java/com/yahoo/searchdefinition/SchemaParsingTestCase.java @@ -15,7 +15,7 @@ import static org.junit.Assert.*; * * @author hmusum */ -public class SchemaParsingTestCase extends SchemaTestCase { +public class SchemaParsingTestCase extends AbstractSchemaTestCase { @Test public void requireThatIndexingExpressionsCanBeParsed() throws Exception { diff --git a/config-model/src/test/java/com/yahoo/searchdefinition/SchemaTestCase.java b/config-model/src/test/java/com/yahoo/searchdefinition/SchemaTestCase.java index c9371160ea5..329d27472c2 100644 --- a/config-model/src/test/java/com/yahoo/searchdefinition/SchemaTestCase.java +++ b/config-model/src/test/java/com/yahoo/searchdefinition/SchemaTestCase.java @@ -1,78 +1,266 @@ // Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. package com.yahoo.searchdefinition; -import com.yahoo.io.IOUtils; +import com.yahoo.searchdefinition.document.Stemming; +import com.yahoo.searchdefinition.parser.ParseException; +import com.yahoo.searchdefinition.processing.ImportedFieldsResolver; +import com.yahoo.searchdefinition.processing.OnnxModelTypeResolver; +import com.yahoo.vespa.model.test.utils.DeployLoggerStub; +import org.junit.Test; -import java.io.BufferedReader; -import java.io.BufferedWriter; -import java.io.IOException; +import static com.yahoo.config.model.test.TestUtil.joinLines; +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertNotNull; +import static org.junit.Assert.assertTrue; +import static org.junit.Assert.fail; -import static helpers.CompareConfigTestHelper.assertSerializedConfigEquals; -import static helpers.CompareConfigTestHelper.assertSerializedConfigFileEquals; +/** + * Schema tests that don't depend on files. + * + * @author bratseth + */ +public class SchemaTestCase { -public abstract class SchemaTestCase { - - protected static void assertConfigFile(String filename, String cfg) throws IOException { - assertSerializedConfigFileEquals(filename, cfg); + @Test + public void testValidationOfInheritedSchema() throws ParseException { + try { + String schema = joinLines( + "schema test inherits nonesuch {" + + " document test inherits nonesuch {" + + " }" + + "}"); + DeployLoggerStub logger = new DeployLoggerStub(); + SearchBuilder.createFromStrings(logger, schema); + assertEquals("schema 'test' inherits 'nonesuch', but this schema does not exist", + logger.entries.get(0).message); + fail("Expected failure"); + } + catch (IllegalArgumentException e) { + assertEquals("schema 'test' inherits 'nonesuch', but this schema does not exist", e.getMessage()); + } } - protected static void assertConfigFiles(String expectedFile, - String cfgFile, - boolean orderMatters, - boolean updateOnAssert) throws IOException { + @Test + public void testValidationOfSchemaAndDocumentInheritanceConsistency() throws ParseException { try { - assertSerializedConfigEquals(readAndCensorIndexes(expectedFile), readAndCensorIndexes(cfgFile), orderMatters); - } catch (AssertionError e) { - if (updateOnAssert) { - BufferedWriter writer = IOUtils.createWriter(expectedFile, false); - writer.write(readAndCensorIndexes(cfgFile)); - writer.newLine(); - writer.flush(); - writer.close(); - System.err.println(e.getMessage() + " [not equal files: >>>"+expectedFile+"<<< and >>>"+cfgFile+"<<< in assertConfigFiles]"); - return; - } - throw new AssertionError(e.getMessage() + " [not equal files: >>>"+expectedFile+"<<< and >>>"+cfgFile+"<<< in assertConfigFiles]", e); + String parent = joinLines( + "schema parent {" + + " document parent {" + + " field pf1 type string {" + + " indexing: summary" + + " }" + + " }" + + "}"); + String child = joinLines( + "schema child inherits parent {" + + " document child {" + + " field cf1 type string {" + + " indexing: summary" + + " }" + + " }" + + "}"); + SearchBuilder.createFromStrings(new DeployLoggerStub(), parent, child); + } + catch (IllegalArgumentException e) { + assertEquals("schema 'child' inherits 'parent', " + + "but its document type does not inherit the parent's document type" + , e.getMessage()); } } - /** - * This is to avoid having to keep those pesky array index numbers in the config format up to date - * as new entries are added and removed. - */ - private static String readAndCensorIndexes(String file) throws IOException { - StringBuilder b = new StringBuilder(); - try (BufferedReader r = IOUtils.createReader(file)) { - int character; - boolean lastWasNewline = false; - boolean inBrackets = false; - while (-1 != (character = r.read())) { - // skip empty lines - if (character == '\n') { - if (lastWasNewline) continue; - lastWasNewline = true; - } - else { - lastWasNewline = false; - } - // skip quoted strings - if (character == '"') { - b.appendCodePoint(character); - while (-1 != (character = r.read()) && character != '"') { - b.appendCodePoint(character); - } - } + @Test + public void testSchemaInheritance() throws ParseException { + String parentLines = joinLines( + "schema parent {" + + " document parent {" + + " field pf1 type string {" + + " indexing: summary" + + " }" + + " }" + + " fieldset parent_set {" + + " fields: pf1" + + " }" + + " stemming: none" + + " index parent_index {" + + " stemming: best" + + " }" + + " field parent_field type string {" + + " indexing: input pf1 | lowercase | index | attribute | summary" + + " }" + + " rank-profile parent_profile {" + + " }" + + " constant parent_constant {" + + " file: constants/my_constant_tensor_file.json" + + " type: tensor<float>(x{},y{})" + + " }" + + " onnx-model parent_model {" + + " file: models/my_model.onnx" + + " }" + + " document-summary parent_summary {" + + " summary pf1 type string {}" + + " }" + + " import field parentschema_ref.name as parent_imported {}" + + " raw-as-base64-in-summary" + + "}"); + String childLines = joinLines( + "schema child inherits parent {" + + " document child inherits parent {" + + " field cf1 type string {" + + " indexing: summary" + + " }" + + " }" + + " fieldset child_set {" + + " fields: cf1, pf1" + + " }" + + " stemming: shortest" + + " index child_index {" + + " stemming: shortest" + + " }" + + " field child_field type string {" + + " indexing: input pf1 | lowercase | index | attribute | summary" + + " }" + + " rank-profile child_profile inherits parent_profile {" + + " }" + + " constant child_constant {" + + " file: constants/my_constant_tensor_file.json" + + " type: tensor<float>(x{},y{})" + + " }" + + " onnx-model child_model {" + + " file: models/my_model.onnx" + + " }" + + " document-summary child_summary inherits parent_summary {" + + " summary cf1 type string {}" + + " }" + + " import field parentschema_ref.name as child_imported {}" + + "}"); - // skip bracket content - if (character == ']') - inBrackets = false; - if (! inBrackets) - b.appendCodePoint(character); - if (character == '[') - inBrackets = true; - } - } - return b.toString(); + System.out.println(parentLines); + SearchBuilder builder = new SearchBuilder(new DeployLoggerStub()); + builder.processorsToSkip().add(OnnxModelTypeResolver.class); // Avoid discovering the Onnx model referenced does not exist + builder.processorsToSkip().add(ImportedFieldsResolver.class); // Avoid discovering the document reference leads nowhere + builder.importString(parentLines); + builder.importString(childLines); + builder.build(true); + var application = builder.application(); + + var child = application.schemas().get("child"); + assertEquals("pf1", child.fieldSets().userFieldSets().get("parent_set").getFieldNames().stream().findFirst().get()); + assertEquals("[cf1, pf1]", child.fieldSets().userFieldSets().get("child_set").getFieldNames().toString()); + assertEquals(Stemming.SHORTEST, child.getStemming()); + assertEquals(Stemming.BEST, child.getIndex("parent_index").getStemming()); + assertEquals(Stemming.SHORTEST, child.getIndex("child_index").getStemming()); + assertNotNull(child.getField("parent_field")); + assertNotNull(child.getField("child_field")); + assertNotNull(child.getExtraField("parent_field")); + assertNotNull(child.getExtraField("child_field")); + assertNotNull(application.rankProfileRegistry().get(child, "parent_profile")); + assertNotNull(application.rankProfileRegistry().get(child, "child_profile")); + assertEquals("parent_profile", application.rankProfileRegistry().get(child, "child_profile").getInheritedName()); + assertNotNull(child.rankingConstants().get("parent_constant")); + assertNotNull(child.rankingConstants().get("child_constant")); + assertTrue(child.rankingConstants().asMap().containsKey("parent_constant")); + assertTrue(child.rankingConstants().asMap().containsKey("child_constant")); + assertNotNull(child.onnxModels().get("parent_model")); + assertNotNull(child.onnxModels().get("child_model")); + assertTrue(child.onnxModels().asMap().containsKey("parent_model")); + assertTrue(child.onnxModels().asMap().containsKey("child_model")); + assertNotNull(child.getSummary("parent_summary")); + assertNotNull(child.getSummary("child_summary")); + assertEquals("parent_summary", child.getSummary("child_summary").getInherited().getName()); + assertTrue(child.getSummaries().containsKey("parent_summary")); + assertTrue(child.getSummaries().containsKey("child_summary")); + assertNotNull(child.getSummaryField("pf1")); + assertNotNull(child.getSummaryField("cf1")); + assertNotNull(child.getExplicitSummaryField("pf1")); + assertNotNull(child.getExplicitSummaryField("cf1")); + assertNotNull(child.getUniqueNamedSummaryFields().get("pf1")); + assertNotNull(child.getUniqueNamedSummaryFields().get("cf1")); + assertNotNull(child.temporaryImportedFields().get().fields().get("parent_imported")); + assertNotNull(child.temporaryImportedFields().get().fields().get("child_imported")); + } + + @Test + public void testSchemaInheritanceEmptyChildren() throws ParseException { + String parentLines = joinLines( + "schema parent {" + + " document parent {" + + " field pf1 type string {" + + " indexing: summary" + + " }" + + " }" + + " fieldset parent_set {" + + " fields: pf1" + + " }" + + " stemming: none" + + " index parent_index {" + + " stemming: best" + + " }" + + " field parent_field type string {" + + " indexing: input pf1 | lowercase | index | attribute | summary" + + " }" + + " rank-profile parent_profile {" + + " }" + + " constant parent_constant {" + + " file: constants/my_constant_tensor_file.json" + + " type: tensor<float>(x{},y{})" + + " }" + + " onnx-model parent_model {" + + " file: models/my_model.onnx" + + " }" + + " document-summary parent_summary {" + + " summary pf1 type string {}" + + " }" + + " import field parentschema_ref.name as parent_imported {}" + + " raw-as-base64-in-summary" + + "}"); + String childLines = joinLines( + "schema child inherits parent {" + + " document child inherits parent {" + + " field cf1 type string {" + + " indexing: summary" + + " }" + + " }" + + "}"); + String grandchildLines = joinLines( + "schema grandchild inherits child {" + + " document grandchild inherits child {" + + " field gf1 type string {" + + " indexing: summary" + + " }" + + " }" + + "}"); + + SearchBuilder builder = new SearchBuilder(new DeployLoggerStub()); + builder.processorsToSkip().add(OnnxModelTypeResolver.class); // Avoid discovering the Onnx model referenced does not exist + builder.processorsToSkip().add(ImportedFieldsResolver.class); // Avoid discovering the document reference leads nowhere + builder.importString(parentLines); + builder.importString(childLines); + builder.importString(grandchildLines); + builder.build(true); + var application = builder.application(); + + assertInheritedFromParent(application.schemas().get("child"), application); + assertInheritedFromParent(application.schemas().get("grandchild"), application); + } + + private void assertInheritedFromParent(Schema schema, Application application) { + assertEquals("pf1", schema.fieldSets().userFieldSets().get("parent_set").getFieldNames().stream().findFirst().get()); + assertEquals(Stemming.NONE, schema.getStemming()); + assertEquals(Stemming.BEST, schema.getIndex("parent_index").getStemming()); + assertNotNull(schema.getField("parent_field")); + assertNotNull(schema.getExtraField("parent_field")); + assertNotNull(application.rankProfileRegistry().get(schema, "parent_profile")); + assertNotNull(schema.rankingConstants().get("parent_constant")); + assertTrue(schema.rankingConstants().asMap().containsKey("parent_constant")); + assertNotNull(schema.onnxModels().get("parent_model")); + assertTrue(schema.onnxModels().asMap().containsKey("parent_model")); + assertNotNull(schema.getSummary("parent_summary")); + assertTrue(schema.getSummaries().containsKey("parent_summary")); + assertNotNull(schema.getSummaryField("pf1")); + assertNotNull(schema.getExplicitSummaryField("pf1")); + assertNotNull(schema.getUniqueNamedSummaryFields().get("pf1")); + assertNotNull(schema.temporaryImportedFields().get().fields().get("parent_imported")); + assertTrue(schema.isRawAsBase64()); } } diff --git a/config-model/src/test/java/com/yahoo/searchdefinition/StemmingSettingTestCase.java b/config-model/src/test/java/com/yahoo/searchdefinition/StemmingSettingTestCase.java index e11efeac31c..d382e29ca4d 100644 --- a/config-model/src/test/java/com/yahoo/searchdefinition/StemmingSettingTestCase.java +++ b/config-model/src/test/java/com/yahoo/searchdefinition/StemmingSettingTestCase.java @@ -16,36 +16,36 @@ import static org.junit.Assert.assertNull; * * @author bratseth */ -public class StemmingSettingTestCase extends SchemaTestCase { +public class StemmingSettingTestCase extends AbstractSchemaTestCase { @Test public void testStemmingSettings() throws IOException, ParseException { - Search search = SearchBuilder.buildFromFile("src/test/examples/stemmingsetting.sd"); + Schema schema = SearchBuilder.buildFromFile("src/test/examples/stemmingsetting.sd"); - SDField artist = (SDField)search.getDocument().getField("artist"); - assertEquals(Stemming.SHORTEST, artist.getStemming(search)); + SDField artist = (SDField) schema.getDocument().getField("artist"); + assertEquals(Stemming.SHORTEST, artist.getStemming(schema)); - SDField title = (SDField)search.getDocument().getField("title"); - assertEquals(Stemming.NONE, title.getStemming(search)); + SDField title = (SDField) schema.getDocument().getField("title"); + assertEquals(Stemming.NONE, title.getStemming(schema)); - SDField song = (SDField)search.getDocument().getField("song"); - assertEquals(Stemming.MULTIPLE, song.getStemming(search)); + SDField song = (SDField) schema.getDocument().getField("song"); + assertEquals(Stemming.MULTIPLE, song.getStemming(schema)); - SDField track = (SDField)search.getDocument().getField("track"); - assertEquals(Stemming.SHORTEST, track.getStemming(search)); + SDField track = (SDField) schema.getDocument().getField("track"); + assertEquals(Stemming.SHORTEST, track.getStemming(schema)); - SDField backward = (SDField)search.getDocument().getField("backward"); - assertEquals(Stemming.SHORTEST, backward.getStemming(search)); + SDField backward = (SDField) schema.getDocument().getField("backward"); + assertEquals(Stemming.SHORTEST, backward.getStemming(schema)); - Index defaultIndex = search.getIndex("default"); + Index defaultIndex = schema.getIndex("default"); assertEquals(Stemming.SHORTEST, defaultIndex.getStemming()); } @Test public void requireThatStemmingIsDefaultBest() throws IOException, ParseException { - Search search = SearchBuilder.buildFromFile("src/test/examples/stemmingdefault.sd"); - assertNull(search.getConcreteField("my_str").getStemming()); - assertEquals(Stemming.BEST, search.getConcreteField("my_str").getStemming(search)); + Schema schema = SearchBuilder.buildFromFile("src/test/examples/stemmingdefault.sd"); + assertNull(schema.getConcreteField("my_str").getStemming()); + assertEquals(Stemming.BEST, schema.getConcreteField("my_str").getStemming(schema)); } } diff --git a/config-model/src/test/java/com/yahoo/searchdefinition/StructTestCase.java b/config-model/src/test/java/com/yahoo/searchdefinition/StructTestCase.java index 9fd5df5e028..b86225110df 100755 --- a/config-model/src/test/java/com/yahoo/searchdefinition/StructTestCase.java +++ b/config-model/src/test/java/com/yahoo/searchdefinition/StructTestCase.java @@ -14,7 +14,7 @@ import static org.junit.Assert.fail; * * @author bratseth */ -public class StructTestCase extends SchemaTestCase { +public class StructTestCase extends AbstractSchemaTestCase { @Test public void testStruct() throws IOException { @@ -37,7 +37,7 @@ public class StructTestCase extends SchemaTestCase { try { DocumenttypesConfig.Builder dt = Deriver.getDocumentTypesConfig("src/test/examples/structanddocumentwithsamenames.sd"); } catch (Exception e) { - fail("Should not have thrown exception " + e.toString()); + fail("Should not have thrown exception " + e); } } diff --git a/config-model/src/test/java/com/yahoo/searchdefinition/SummaryTestCase.java b/config-model/src/test/java/com/yahoo/searchdefinition/SummaryTestCase.java index 35a775d7ad5..e57dcf38fed 100644 --- a/config-model/src/test/java/com/yahoo/searchdefinition/SummaryTestCase.java +++ b/config-model/src/test/java/com/yahoo/searchdefinition/SummaryTestCase.java @@ -27,7 +27,7 @@ public class SummaryTestCase { @Test public void testMemorySummary() throws ParseException { String sd = joinLines( - "search memorysummary {", + "schema memorysummary {", " document memorysummary {", " field inmemory type string {", " indexing: attribute | summary", @@ -45,7 +45,7 @@ public class SummaryTestCase { @Test public void testDiskSummary() throws ParseException { String sd = joinLines( - "search disksummary {", + "schema disksummary {", " document-summary foobar {", " summary foo1 type string { source: inmemory }", " summary foo2 type string { source: ondisk }", @@ -72,7 +72,7 @@ public class SummaryTestCase { @Test public void testDiskSummaryExplicit() throws ParseException { String sd = joinLines( - "search disksummary {", + "schema disksummary {", " document disksummary {", " field inmemory type string {", " indexing: attribute | summary", @@ -95,7 +95,7 @@ public class SummaryTestCase { @Test public void testStructMemorySummary() throws ParseException { String sd = joinLines( - "search structmemorysummary {", + "schema structmemorysummary {", " document structmemorysummary {", " struct elem {", " field name type string {}", @@ -126,7 +126,7 @@ public class SummaryTestCase { @Test public void testInheritance() throws Exception { String sd = joinLines( - "search music {", + "schema music {", " document music {", " field title type string {", " indexing: summary | attribute | index", @@ -185,7 +185,7 @@ public class SummaryTestCase { @Test public void testRedeclaringInheritedFieldFails() throws Exception { String sd = joinLines( - "search music {", + "schema music {", " document music {", " field title type string {", " indexing: summary | attribute | index", @@ -210,7 +210,7 @@ public class SummaryTestCase { SearchBuilder.createFromString(sd, logger); fail("Expected exception"); } catch (IllegalArgumentException e) { - assertEquals("For search 'music', summary class 'title2', summary field 'title': Can not use " + + assertEquals("For schema 'music', summary class 'title2', summary field 'title': Can not use " + "source 'title_short' for this summary field, an equally named field in summary class 'title' " + "uses a different source: 'title'.", e.getMessage()); } @@ -238,6 +238,36 @@ public class SummaryTestCase { } } + @Test + public void testInheritingParentSummary() throws ParseException { + String parent = joinLines( + "schema parent {" + + " document parent {" + + " field pf1 type string {" + + " indexing: summary" + + " }" + + " }" + + " document-summary parent_summary {" + + " summary pf1 type string {}" + + " }" + + "}"); + String child = joinLines( + "schema child inherits parent {" + + " document child inherits parent {" + + " field cf1 type string {" + + " indexing: summary" + + " }" + + " }" + + " document-summary child_summary inherits parent_summary {" + + " summary cf1 type string {}" + + " }" + + "}"); + DeployLoggerStub logger = new DeployLoggerStub(); + SearchBuilder.createFromStrings(logger, parent, child); + logger.entries.forEach(e -> System.out.println(e)); + //assertTrue(logger.entries.isEmpty()); + } + private static class TestValue { private final DocumentSummary summary; diff --git a/config-model/src/test/java/com/yahoo/searchdefinition/derived/AbstractExportingTestCase.java b/config-model/src/test/java/com/yahoo/searchdefinition/derived/AbstractExportingTestCase.java index 5d0e69f9ac4..9ba9f3b37f7 100644 --- a/config-model/src/test/java/com/yahoo/searchdefinition/derived/AbstractExportingTestCase.java +++ b/config-model/src/test/java/com/yahoo/searchdefinition/derived/AbstractExportingTestCase.java @@ -7,9 +7,9 @@ import com.yahoo.config.model.application.provider.MockFileRegistry; import com.yahoo.config.model.deploy.TestProperties; import com.yahoo.document.DocumenttypesConfig; import com.yahoo.document.config.DocumentmanagerConfig; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.SearchBuilder; -import com.yahoo.searchdefinition.SchemaTestCase; +import com.yahoo.searchdefinition.AbstractSchemaTestCase; import com.yahoo.searchdefinition.parser.ParseException; import ai.vespa.rankingexpression.importer.configmodelview.ImportedMlModels; import com.yahoo.vespa.configmodel.producers.DocumentManager; @@ -23,7 +23,7 @@ import java.io.IOException; * * @author bratseth */ -public abstract class AbstractExportingTestCase extends SchemaTestCase { +public abstract class AbstractExportingTestCase extends AbstractSchemaTestCase { private static final String tempDir = "temp/"; private static final String searchDefRoot = "src/test/derived/"; @@ -50,12 +50,13 @@ public abstract class AbstractExportingTestCase extends SchemaTestCase { properties, builder.getRankProfileRegistry(), builder.getQueryProfileRegistry(), - new ImportedMlModels(), new InThreadExecutorService()); + new ImportedMlModels(), + new InThreadExecutorService()); return export(dirName, builder, config); } - DerivedConfiguration derive(String dirName, SearchBuilder builder, Search search) throws IOException { - DerivedConfiguration config = new DerivedConfiguration(search, + DerivedConfiguration derive(String dirName, SearchBuilder builder, Schema schema) throws IOException { + DerivedConfiguration config = new DerivedConfiguration(schema, builder.getRankProfileRegistry(), builder.getQueryProfileRegistry()); return export(dirName, builder, config); @@ -117,8 +118,8 @@ public abstract class AbstractExportingTestCase extends SchemaTestCase { return derived; } - protected DerivedConfiguration assertCorrectDeriving(SearchBuilder builder, Search search, String name) throws IOException { - DerivedConfiguration derived = derive(name, builder, search); + protected DerivedConfiguration assertCorrectDeriving(SearchBuilder builder, Schema schema, String name) throws IOException { + DerivedConfiguration derived = derive(name, builder, schema); assertCorrectConfigFiles(name); return derived; } diff --git a/config-model/src/test/java/com/yahoo/searchdefinition/derived/AttributeListTestCase.java b/config-model/src/test/java/com/yahoo/searchdefinition/derived/AttributeListTestCase.java index c36a682fbd8..f8316c831f1 100644 --- a/config-model/src/test/java/com/yahoo/searchdefinition/derived/AttributeListTestCase.java +++ b/config-model/src/test/java/com/yahoo/searchdefinition/derived/AttributeListTestCase.java @@ -1,9 +1,9 @@ // Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. package com.yahoo.searchdefinition.derived; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.SearchBuilder; -import com.yahoo.searchdefinition.SchemaTestCase; +import com.yahoo.searchdefinition.AbstractSchemaTestCase; import com.yahoo.searchdefinition.document.Attribute; import com.yahoo.searchdefinition.parser.ParseException; import org.junit.Test; @@ -20,15 +20,15 @@ import static org.junit.Assert.assertFalse; * * @author bratseth */ -public class AttributeListTestCase extends SchemaTestCase { +public class AttributeListTestCase extends AbstractSchemaTestCase { @Test public void testDeriving() throws IOException, ParseException { // Test attribute importing - Search search = SearchBuilder.buildFromFile("src/test/examples/simple.sd"); + Schema schema = SearchBuilder.buildFromFile("src/test/examples/simple.sd"); // Test attribute deriving - AttributeFields attributeFields = new AttributeFields(search); + AttributeFields attributeFields = new AttributeFields(schema); Iterator attributes = attributeFields.attributeIterator(); Attribute attribute; attribute = (Attribute)attributes.next(); @@ -71,8 +71,8 @@ public class AttributeListTestCase extends SchemaTestCase { @Test public void fields_in_array_of_struct_are_derived_into_array_attributes() throws IOException, ParseException { - Search search = SearchBuilder.buildFromFile("src/test/derived/array_of_struct_attribute/test.sd"); - Iterator<Attribute> attributes = new AttributeFields(search).attributeIterator(); + Schema schema = SearchBuilder.buildFromFile("src/test/derived/array_of_struct_attribute/test.sd"); + Iterator<Attribute> attributes = new AttributeFields(schema).attributeIterator(); assertAttribute("elem_array.name", Attribute.Type.STRING, Attribute.CollectionType.ARRAY, true, attributes.next()); assertAttribute("elem_array.weight", Attribute.Type.INTEGER, Attribute.CollectionType.ARRAY, false, attributes.next()); @@ -81,8 +81,8 @@ public class AttributeListTestCase extends SchemaTestCase { @Test public void fields_in_map_of_struct_are_derived_into_array_attributes() throws IOException, ParseException { - Search search = SearchBuilder.buildFromFile("src/test/derived/map_of_struct_attribute/test.sd"); - Iterator<Attribute> attributes = new AttributeFields(search).attributeIterator(); + Schema schema = SearchBuilder.buildFromFile("src/test/derived/map_of_struct_attribute/test.sd"); + Iterator<Attribute> attributes = new AttributeFields(schema).attributeIterator(); assertAttribute("str_elem_map.key", Attribute.Type.STRING, Attribute.CollectionType.ARRAY, true, attributes.next()); assertAttribute("str_elem_map.value.name", Attribute.Type.STRING, Attribute.CollectionType.ARRAY, false, attributes.next()); @@ -101,7 +101,7 @@ public class AttributeListTestCase extends SchemaTestCase { @Test public void only_zcurve_attribute_is_derived_from_array_of_position_field() throws ParseException { - Search search = SearchBuilder.createFromString( + Schema schema = SearchBuilder.createFromString( joinLines("search test {", " document test {", " field pos_array type array<position> {", @@ -109,7 +109,7 @@ public class AttributeListTestCase extends SchemaTestCase { " }", " }", "}")).getSearch(); - Iterator<Attribute> attributes = new AttributeFields(search).attributeIterator(); + Iterator<Attribute> attributes = new AttributeFields(schema).attributeIterator(); assertAttribute("pos_array_zcurve", Attribute.Type.LONG, Attribute.CollectionType.ARRAY, true, attributes.next()); assertFalse(attributes.hasNext()); @@ -117,8 +117,8 @@ public class AttributeListTestCase extends SchemaTestCase { @Test public void fields_in_map_of_primitive_are_derived_into_array_attributes() throws IOException, ParseException { - Search search = SearchBuilder.buildFromFile("src/test/derived/map_attribute/test.sd"); - Iterator<Attribute> attributes = new AttributeFields(search).attributeIterator(); + Schema schema = SearchBuilder.buildFromFile("src/test/derived/map_attribute/test.sd"); + Iterator<Attribute> attributes = new AttributeFields(schema).attributeIterator(); assertAttribute("str_map.key", Attribute.Type.STRING, Attribute.CollectionType.ARRAY, true, attributes.next()); assertAttribute("str_map.value", Attribute.Type.STRING, Attribute.CollectionType.ARRAY, false, attributes.next()); diff --git a/config-model/src/test/java/com/yahoo/searchdefinition/derived/CasingTestCase.java b/config-model/src/test/java/com/yahoo/searchdefinition/derived/CasingTestCase.java index 554a78c36ef..080752d260f 100644 --- a/config-model/src/test/java/com/yahoo/searchdefinition/derived/CasingTestCase.java +++ b/config-model/src/test/java/com/yahoo/searchdefinition/derived/CasingTestCase.java @@ -1,9 +1,9 @@ // Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. package com.yahoo.searchdefinition.derived; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.SearchBuilder; -import com.yahoo.searchdefinition.SchemaTestCase; +import com.yahoo.searchdefinition.AbstractSchemaTestCase; import com.yahoo.searchdefinition.parser.ParseException; import org.junit.Test; @@ -16,21 +16,21 @@ import static org.junit.Assert.assertEquals; * * @author vegardh */ -public class CasingTestCase extends SchemaTestCase { +public class CasingTestCase extends AbstractSchemaTestCase { @Test public void testCasing() throws IOException, ParseException { - Search search = SearchBuilder.buildFromFile("src/test/examples/casing.sd"); - assertEquals(search.getIndex("color").getName(), "color"); - assertEquals(search.getIndex("Foo").getName(), "Foo"); - assertEquals(search.getIndex("Price").getName(), "Price"); - assertEquals(search.getAttribute("artist").getName(), "artist"); - assertEquals(search.getAttribute("Drummer").getName(), "Drummer"); - assertEquals(search.getAttribute("guitarist").getName(), "guitarist"); - assertEquals(search.getAttribute("title").getName(), "title"); - assertEquals(search.getAttribute("Trumpetist").getName(), "Trumpetist"); - assertEquals(search.getAttribute("Saxophonist").getName(), "Saxophonist"); - assertEquals(search.getAttribute("TenorSaxophonist").getName(), "TenorSaxophonist"); - assertEquals(search.getAttribute("Flutist").getName(), "Flutist"); + Schema schema = SearchBuilder.buildFromFile("src/test/examples/casing.sd"); + assertEquals(schema.getIndex("color").getName(), "color"); + assertEquals(schema.getIndex("Foo").getName(), "Foo"); + assertEquals(schema.getIndex("Price").getName(), "Price"); + assertEquals(schema.getAttribute("artist").getName(), "artist"); + assertEquals(schema.getAttribute("Drummer").getName(), "Drummer"); + assertEquals(schema.getAttribute("guitarist").getName(), "guitarist"); + assertEquals(schema.getAttribute("title").getName(), "title"); + assertEquals(schema.getAttribute("Trumpetist").getName(), "Trumpetist"); + assertEquals(schema.getAttribute("Saxophonist").getName(), "Saxophonist"); + assertEquals(schema.getAttribute("TenorSaxophonist").getName(), "TenorSaxophonist"); + assertEquals(schema.getAttribute("Flutist").getName(), "Flutist"); } } diff --git a/config-model/src/test/java/com/yahoo/searchdefinition/derived/CombinedAttributeAndIndexSearchTestCase.java b/config-model/src/test/java/com/yahoo/searchdefinition/derived/CombinedAttributeAndIndexSchemaTestCase.java index dcf6a84fd2e..bb339688fcb 100644 --- a/config-model/src/test/java/com/yahoo/searchdefinition/derived/CombinedAttributeAndIndexSearchTestCase.java +++ b/config-model/src/test/java/com/yahoo/searchdefinition/derived/CombinedAttributeAndIndexSchemaTestCase.java @@ -11,7 +11,7 @@ import java.io.IOException; * * @author bratseth */ -public class CombinedAttributeAndIndexSearchTestCase extends AbstractExportingTestCase { +public class CombinedAttributeAndIndexSchemaTestCase extends AbstractExportingTestCase { @Test public void testMultipleSummaries() throws IOException, ParseException { diff --git a/config-model/src/test/java/com/yahoo/searchdefinition/derived/DeriverTestCase.java b/config-model/src/test/java/com/yahoo/searchdefinition/derived/DeriverTestCase.java index 5638aeb92c0..b1fb1962fc8 100644 --- a/config-model/src/test/java/com/yahoo/searchdefinition/derived/DeriverTestCase.java +++ b/config-model/src/test/java/com/yahoo/searchdefinition/derived/DeriverTestCase.java @@ -4,7 +4,7 @@ package com.yahoo.searchdefinition.derived; import com.yahoo.document.DataType; import com.yahoo.document.DocumentTypeManager; import com.yahoo.document.config.DocumentmanagerConfig; -import com.yahoo.searchdefinition.SchemaTestCase; +import com.yahoo.searchdefinition.AbstractSchemaTestCase; import org.junit.Test; import java.util.List; @@ -16,7 +16,7 @@ import static org.junit.Assert.assertEquals; * * @author bratseth */ -public class DeriverTestCase extends SchemaTestCase { +public class DeriverTestCase extends AbstractSchemaTestCase { @Test public void testDeriveDocManager() { diff --git a/config-model/src/test/java/com/yahoo/searchdefinition/derived/EmptyRankProfileTestCase.java b/config-model/src/test/java/com/yahoo/searchdefinition/derived/EmptyRankProfileTestCase.java index 8778a177b81..2633f7881bd 100644 --- a/config-model/src/test/java/com/yahoo/searchdefinition/derived/EmptyRankProfileTestCase.java +++ b/config-model/src/test/java/com/yahoo/searchdefinition/derived/EmptyRankProfileTestCase.java @@ -4,9 +4,9 @@ package com.yahoo.searchdefinition.derived; import com.yahoo.document.DataType; import com.yahoo.search.query.profile.QueryProfileRegistry; import com.yahoo.searchdefinition.RankProfileRegistry; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.SearchBuilder; -import com.yahoo.searchdefinition.SchemaTestCase; +import com.yahoo.searchdefinition.AbstractSchemaTestCase; import com.yahoo.searchdefinition.document.SDDocumentType; import com.yahoo.searchdefinition.document.SDField; import org.junit.Test; @@ -16,22 +16,22 @@ import org.junit.Test; * * @author bratseth */ -public class EmptyRankProfileTestCase extends SchemaTestCase { +public class EmptyRankProfileTestCase extends AbstractSchemaTestCase { @Test public void testDeriving() { - Search search = new Search("test"); - RankProfileRegistry rankProfileRegistry = RankProfileRegistry.createRankProfileRegistryWithBuiltinRankProfiles(search); + Schema schema = new Schema("test"); + RankProfileRegistry rankProfileRegistry = RankProfileRegistry.createRankProfileRegistryWithBuiltinRankProfiles(schema); SDDocumentType doc = new SDDocumentType("test"); - search.addDocument(doc); + schema.addDocument(doc); doc.addField(new SDField("a", DataType.STRING)); SDField field = new SDField("b", DataType.STRING); field.setLiteralBoost(500); doc.addField(field); doc.addField(new SDField("c", DataType.STRING)); - search = SearchBuilder.buildFromRawSearch(search, rankProfileRegistry, new QueryProfileRegistry()); - new DerivedConfiguration(search, rankProfileRegistry); + schema = SearchBuilder.buildFromRawSearch(schema, rankProfileRegistry, new QueryProfileRegistry()); + new DerivedConfiguration(schema, rankProfileRegistry); } } diff --git a/config-model/src/test/java/com/yahoo/searchdefinition/derived/IdTestCase.java b/config-model/src/test/java/com/yahoo/searchdefinition/derived/IdTestCase.java index 1c63b8c462e..b262ef92e1a 100644 --- a/config-model/src/test/java/com/yahoo/searchdefinition/derived/IdTestCase.java +++ b/config-model/src/test/java/com/yahoo/searchdefinition/derived/IdTestCase.java @@ -4,13 +4,15 @@ package com.yahoo.searchdefinition.derived; import com.yahoo.config.model.application.provider.BaseDeployLogger; import com.yahoo.document.DataType; import com.yahoo.searchdefinition.RankProfileRegistry; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.document.SDDocumentType; import com.yahoo.searchdefinition.document.SDField; import com.yahoo.searchdefinition.processing.Processing; import com.yahoo.vespa.model.container.search.QueryProfiles; import org.junit.Test; +import java.util.Set; + import static org.junit.Assert.assertNotNull; import static org.junit.Assert.assertNull; @@ -23,14 +25,15 @@ public class IdTestCase extends AbstractExportingTestCase { @Test public void testExplicitUpperCaseIdField() { - Search search = new Search("test"); + Schema schema = new Schema("test"); SDDocumentType document = new SDDocumentType("test"); - search.addDocument(document); + schema.addDocument(document); SDField uri = new SDField("URI", DataType.URI); uri.parseIndexingScript("{ summary | index }"); document.addField(uri); - new Processing().process(search, new BaseDeployLogger(), new RankProfileRegistry(), new QueryProfiles(), true, false); + new Processing().process(schema, new BaseDeployLogger(), new RankProfileRegistry(), new QueryProfiles(), + true, false, Set.of()); assertNull(document.getField("uri")); assertNull(document.getField("Uri")); diff --git a/config-model/src/test/java/com/yahoo/searchdefinition/derived/InheritanceTestCase.java b/config-model/src/test/java/com/yahoo/searchdefinition/derived/InheritanceTestCase.java index a07eb1015a1..59950cee638 100644 --- a/config-model/src/test/java/com/yahoo/searchdefinition/derived/InheritanceTestCase.java +++ b/config-model/src/test/java/com/yahoo/searchdefinition/derived/InheritanceTestCase.java @@ -4,7 +4,7 @@ package com.yahoo.searchdefinition.derived; import com.yahoo.document.DataType; import com.yahoo.document.config.DocumentmanagerConfig; import com.yahoo.searchdefinition.Index; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.SearchBuilder; import com.yahoo.searchdefinition.document.SDDocumentType; import com.yahoo.searchdefinition.document.SDField; @@ -143,20 +143,20 @@ public class InheritanceTestCase extends AbstractExportingTestCase { @Test public void testIndexSettingInheritance() { SDDocumentType parent = new SDDocumentType("parent"); - Search parentSearch = new Search("parent"); - parentSearch.addDocument(parent); + Schema parentSchema = new Schema("parent"); + parentSchema.addDocument(parent); SDField prefixed = parent.addField("prefixed", DataType.STRING); prefixed.parseIndexingScript("{ index }"); prefixed.addIndex(new Index("prefixed", true)); SDDocumentType child = new SDDocumentType("child"); child.inherit(parent); - Search childSearch = new Search("child"); - childSearch.addDocument(child); + Schema childSchema = new Schema("child"); + childSchema.addDocument(child); prefixed = (SDField)child.getField("prefixed"); assertNotNull(prefixed); - assertEquals(new Index("prefixed", true), childSearch.getIndex("prefixed")); + assertEquals(new Index("prefixed", true), childSchema.getIndex("prefixed")); } } diff --git a/config-model/src/test/java/com/yahoo/searchdefinition/derived/LiteralBoostTestCase.java b/config-model/src/test/java/com/yahoo/searchdefinition/derived/LiteralBoostTestCase.java index 070d7ff5337..507c5370237 100644 --- a/config-model/src/test/java/com/yahoo/searchdefinition/derived/LiteralBoostTestCase.java +++ b/config-model/src/test/java/com/yahoo/searchdefinition/derived/LiteralBoostTestCase.java @@ -6,7 +6,7 @@ import com.yahoo.document.DataType; import com.yahoo.search.query.profile.QueryProfileRegistry; import com.yahoo.searchdefinition.RankProfile; import com.yahoo.searchdefinition.RankProfileRegistry; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.SearchBuilder; import com.yahoo.searchdefinition.document.SDDocumentType; import com.yahoo.searchdefinition.document.SDField; @@ -15,6 +15,7 @@ import com.yahoo.vespa.model.container.search.QueryProfiles; import org.junit.Test; import java.util.Arrays; +import java.util.Set; import static com.yahoo.searchdefinition.processing.AssertIndexingScript.assertIndexing; import static org.junit.Assert.assertTrue; @@ -29,19 +30,20 @@ public class LiteralBoostTestCase extends AbstractExportingTestCase { */ @Test public void testLiteralBoost() { - Search search=new Search("literalboost"); - RankProfileRegistry rankProfileRegistry = RankProfileRegistry.createRankProfileRegistryWithBuiltinRankProfiles(search); - SDDocumentType document=new SDDocumentType("literalboost"); - search.addDocument(document); - SDField field1= document.addField("a", DataType.STRING); + Schema schema = new Schema("literalboost"); + RankProfileRegistry rankProfileRegistry = RankProfileRegistry.createRankProfileRegistryWithBuiltinRankProfiles(schema); + SDDocumentType document = new SDDocumentType("literalboost"); + schema.addDocument(document); + SDField field1 = document.addField("a", DataType.STRING); field1.parseIndexingScript("{ index }"); field1.setLiteralBoost(20); - RankProfile other=new RankProfile("other", search, rankProfileRegistry, search.rankingConstants()); + RankProfile other = new RankProfile("other", schema, rankProfileRegistry, schema.rankingConstants()); rankProfileRegistry.add(other); other.addRankSetting(new RankProfile.RankSetting("a", RankProfile.RankSetting.Type.LITERALBOOST, 333)); - new Processing().process(search, new BaseDeployLogger(), rankProfileRegistry, new QueryProfiles(), true, false); - DerivedConfiguration derived=new DerivedConfiguration(search, rankProfileRegistry); + new Processing().process(schema, new BaseDeployLogger(), rankProfileRegistry, new QueryProfiles(), + true, false, Set.of()); + DerivedConfiguration derived = new DerivedConfiguration(schema, rankProfileRegistry); // Check attribute fields derived.getAttributeFields(); // TODO: assert content @@ -49,11 +51,11 @@ public class LiteralBoostTestCase extends AbstractExportingTestCase { // Check il script addition assertIndexing(Arrays.asList("clear_state | guard { input a | tokenize normalize stem:\"BEST\" | index a; }", "clear_state | guard { input a | tokenize | index a_literal; }"), - search); + schema); // Check index info addition - IndexInfo indexInfo=derived.getIndexInfo(); - assertTrue(indexInfo.hasCommand("a","literal-boost")); + IndexInfo indexInfo = derived.getIndexInfo(); + assertTrue(indexInfo.hasCommand("a", "literal-boost")); } /** @@ -61,50 +63,50 @@ public class LiteralBoostTestCase extends AbstractExportingTestCase { */ @Test public void testNonDefaultRankLiteralBoost() { - Search search=new Search("literalboost"); - RankProfileRegistry rankProfileRegistry = RankProfileRegistry.createRankProfileRegistryWithBuiltinRankProfiles(search); - SDDocumentType document=new SDDocumentType("literalboost"); - search.addDocument(document); - SDField field1= document.addField("a", DataType.STRING); + Schema schema = new Schema("literalboost"); + RankProfileRegistry rankProfileRegistry = RankProfileRegistry.createRankProfileRegistryWithBuiltinRankProfiles(schema); + SDDocumentType document = new SDDocumentType("literalboost"); + schema.addDocument(document); + SDField field1 = document.addField("a", DataType.STRING); field1.parseIndexingScript("{ index }"); - RankProfile other=new RankProfile("other", search, rankProfileRegistry, search.rankingConstants()); + RankProfile other = new RankProfile("other", schema, rankProfileRegistry, schema.rankingConstants()); rankProfileRegistry.add(other); other.addRankSetting(new RankProfile.RankSetting("a", RankProfile.RankSetting.Type.LITERALBOOST, 333)); - search = SearchBuilder.buildFromRawSearch(search, rankProfileRegistry, new QueryProfileRegistry()); - DerivedConfiguration derived = new DerivedConfiguration(search, rankProfileRegistry); + schema = SearchBuilder.buildFromRawSearch(schema, rankProfileRegistry, new QueryProfileRegistry()); + DerivedConfiguration derived = new DerivedConfiguration(schema, rankProfileRegistry); // Check il script addition assertIndexing(Arrays.asList("clear_state | guard { input a | tokenize normalize stem:\"BEST\" | index a; }", "clear_state | guard { input a | tokenize | index a_literal; }"), - search); + schema); // Check index info addition - IndexInfo indexInfo=derived.getIndexInfo(); + IndexInfo indexInfo = derived.getIndexInfo(); assertTrue(indexInfo.hasCommand("a","literal-boost")); } /** Tests literal boosts in two fields going to the same index */ @Test public void testTwoLiteralBoostFields() { - Search search=new Search("msb"); - RankProfileRegistry rankProfileRegistry = RankProfileRegistry.createRankProfileRegistryWithBuiltinRankProfiles(search); - SDDocumentType document=new SDDocumentType("msb"); - search.addDocument(document); - SDField field1= document.addField("title", DataType.STRING); + Schema schema = new Schema("msb"); + RankProfileRegistry rankProfileRegistry = RankProfileRegistry.createRankProfileRegistryWithBuiltinRankProfiles(schema); + SDDocumentType document = new SDDocumentType("msb"); + schema.addDocument(document); + SDField field1 = document.addField("title", DataType.STRING); field1.parseIndexingScript("{ summary | index }"); field1.setLiteralBoost(20); - SDField field2= document.addField("body", DataType.STRING); + SDField field2 = document.addField("body", DataType.STRING); field2.parseIndexingScript("{ summary | index }"); field2.setLiteralBoost(20); - search = SearchBuilder.buildFromRawSearch(search, rankProfileRegistry, new QueryProfileRegistry()); - new DerivedConfiguration(search, rankProfileRegistry); + schema = SearchBuilder.buildFromRawSearch(schema, rankProfileRegistry, new QueryProfileRegistry()); + new DerivedConfiguration(schema, rankProfileRegistry); assertIndexing(Arrays.asList("clear_state | guard { input title | tokenize normalize stem:\"BEST\" | summary title | index title; }", "clear_state | guard { input body | tokenize normalize stem:\"BEST\" | summary body | index body; }", "clear_state | guard { input title | tokenize | index title_literal; }", "clear_state | guard { input body | tokenize | index body_literal; }"), - search); + schema); } } diff --git a/config-model/src/test/java/com/yahoo/searchdefinition/derived/NativeRankTypeDefinitionsTestCase.java b/config-model/src/test/java/com/yahoo/searchdefinition/derived/NativeRankTypeDefinitionsTestCase.java index 3b16ff0221e..8c6bc40ece0 100644 --- a/config-model/src/test/java/com/yahoo/searchdefinition/derived/NativeRankTypeDefinitionsTestCase.java +++ b/config-model/src/test/java/com/yahoo/searchdefinition/derived/NativeRankTypeDefinitionsTestCase.java @@ -1,7 +1,7 @@ // Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. package com.yahoo.searchdefinition.derived; -import com.yahoo.searchdefinition.SchemaTestCase; +import com.yahoo.searchdefinition.AbstractSchemaTestCase; import com.yahoo.searchdefinition.document.RankType; import org.junit.Test; @@ -14,7 +14,7 @@ import static org.junit.Assert.*; * * @author geirst */ -public class NativeRankTypeDefinitionsTestCase extends SchemaTestCase { +public class NativeRankTypeDefinitionsTestCase extends AbstractSchemaTestCase { @Test public void testTables() { assertEquals(NativeTable.Type.FIRST_OCCURRENCE.getName(), "firstOccurrenceTable"); diff --git a/config-model/src/test/java/com/yahoo/searchdefinition/derived/SchemaInheritanceTestCase.java b/config-model/src/test/java/com/yahoo/searchdefinition/derived/SchemaInheritanceTestCase.java new file mode 100644 index 00000000000..254ed26df36 --- /dev/null +++ b/config-model/src/test/java/com/yahoo/searchdefinition/derived/SchemaInheritanceTestCase.java @@ -0,0 +1,35 @@ +// Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. +package com.yahoo.searchdefinition.derived; + +import com.yahoo.config.model.application.provider.MockFileRegistry; +import com.yahoo.config.model.deploy.TestProperties; +import com.yahoo.io.IOUtils; +import com.yahoo.searchdefinition.RankProfileRegistry; +import com.yahoo.searchdefinition.SearchBuilder; +import com.yahoo.searchdefinition.parser.ParseException; +import org.junit.Test; + +import java.io.File; +import java.io.IOException; + +/** + * @author bratseth + */ +public class SchemaInheritanceTestCase extends AbstractExportingTestCase { + + @Test + public void testIt() throws IOException, ParseException { + try { + SearchBuilder builder = SearchBuilder.createFromDirectory("src/test/derived/schemainheritance/", + new MockFileRegistry(), + new TestableDeployLogger(), + new TestProperties()); + derive("schemainheritance", builder, builder.getSearch("child")); + assertCorrectConfigFiles("schemainheritance"); + } + finally { + IOUtils.recursiveDeleteDir(new File("src/test/derived/schemainheritance/models.generated/")); + } + } + +} diff --git a/config-model/src/test/java/com/yahoo/searchdefinition/derived/SearchOrdererTestCase.java b/config-model/src/test/java/com/yahoo/searchdefinition/derived/SchemaOrdererTestCase.java index 00fc07de221..c6f254df798 100644 --- a/config-model/src/test/java/com/yahoo/searchdefinition/derived/SearchOrdererTestCase.java +++ b/config-model/src/test/java/com/yahoo/searchdefinition/derived/SchemaOrdererTestCase.java @@ -5,8 +5,8 @@ import com.yahoo.document.ReferenceDataType; import com.yahoo.document.TemporaryStructuredDataType; import com.yahoo.searchdefinition.DocumentReference; import com.yahoo.searchdefinition.DocumentReferences; -import com.yahoo.searchdefinition.Search; -import com.yahoo.searchdefinition.SchemaTestCase; +import com.yahoo.searchdefinition.Schema; +import com.yahoo.searchdefinition.AbstractSchemaTestCase; import com.yahoo.searchdefinition.document.SDDocumentType; import com.yahoo.searchdefinition.document.SDField; import com.yahoo.searchdefinition.document.TemporarySDField; @@ -26,33 +26,33 @@ import static org.junit.Assert.assertEquals; * @author bratseth * @author bjorncs */ -public class SearchOrdererTestCase extends SchemaTestCase { +public class SchemaOrdererTestCase extends AbstractSchemaTestCase { - private static Map<String, Search> createSchemas() { - Map<String, Search> schemas = new HashMap<>(); + private static Map<String, Schema> createSchemas() { + Map<String, Schema> schemas = new HashMap<>(); - Search grandParent = createSchema("grandParent", schemas); + Schema grandParent = createSchema("grandParent", schemas); - Search mother = createSchema("mother", schemas); + Schema mother = createSchema("mother", schemas); inherit(mother, grandParent); - Search father = createSchema("father", schemas); + Schema father = createSchema("father", schemas); inherit(father, grandParent); createDocumentReference(father, mother, "wife_ref"); - Search daugther = createSchema("daughter", schemas); + Schema daugther = createSchema("daughter", schemas); inherit(daugther, father); inherit(daugther, mother); - Search son = createSchema("son", schemas); + Schema son = createSchema("son", schemas); inherit(son, father); inherit(son, mother); - Search product = createSchema("product", schemas); + Schema product = createSchema("product", schemas); - Search pc = createSchema("pc", schemas); + Schema pc = createSchema("pc", schemas); inherit(pc, product); - Search pcAccessory = createSchema("accessory-pc", schemas); + Schema pcAccessory = createSchema("accessory-pc", schemas); inherit(pcAccessory, product); createDocumentReference(pcAccessory, pc, "pc_ref"); @@ -61,34 +61,34 @@ public class SearchOrdererTestCase extends SchemaTestCase { return schemas; } - private static Search createSchema(String name, Map<String, Search> schemas) { - Search search = new Search(name); + private static Schema createSchema(String name, Map<String, Schema> schemas) { + Schema schema = new Schema(name); SDDocumentType document = new SDDocumentType(name); document.setDocumentReferences(new DocumentReferences(emptyMap())); - search.addDocument(document); - schemas.put(search.getName(), search); - return search; + schema.addDocument(document); + schemas.put(schema.getName(), schema); + return schema; } - private static void inherit(Search inheritee, Search inherited) { + private static void inherit(Schema inheritee, Schema inherited) { inheritee.getDocument().inherit(inherited.getDocument()); } private static void assertOrder(List<String> expectedSearchOrder, List<String> inputNames) { - Map<String, Search> schemas = createSchemas(); - List<Search> inputSchemas = inputNames.stream() - .map(schemas::get) - .map(Objects::requireNonNull) - .collect(toList()); + Map<String, Schema> schemas = createSchemas(); + List<Schema> inputSchemas = inputNames.stream() + .map(schemas::get) + .map(Objects::requireNonNull) + .collect(toList()); List<String> actualSearchOrder = new SearchOrderer() .order(inputSchemas) .stream() - .map(Search::getName) + .map(Schema::getName) .collect(toList()); assertEquals(expectedSearchOrder, actualSearchOrder); } - private static void createDocumentReference(Search from, Search to, String refFieldName) { + private static void createDocumentReference(Schema from, Schema to, String refFieldName) { SDField refField = new TemporarySDField(refFieldName, ReferenceDataType.createWithInferredId(TemporaryStructuredDataType.create(to.getName()))); SDDocumentType fromDocument = from.getDocument(); fromDocument.addField(refField); diff --git a/config-model/src/test/java/com/yahoo/searchdefinition/derived/SimpleInheritTestCase.java b/config-model/src/test/java/com/yahoo/searchdefinition/derived/SimpleInheritTestCase.java index 08d888aba0b..106f5a54cfe 100644 --- a/config-model/src/test/java/com/yahoo/searchdefinition/derived/SimpleInheritTestCase.java +++ b/config-model/src/test/java/com/yahoo/searchdefinition/derived/SimpleInheritTestCase.java @@ -1,7 +1,7 @@ // Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. package com.yahoo.searchdefinition.derived; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.SearchBuilder; import com.yahoo.searchdefinition.parser.ParseException; import org.junit.Test; @@ -24,14 +24,14 @@ public class SimpleInheritTestCase extends AbstractExportingTestCase { builder.importFile(expectedResultsDirName + "child.sd"); builder.build(); - Search search = builder.getSearch("child"); + Schema schema = builder.getSearch("child"); String toDirName = "temp/" + name; File toDir = new File(toDirName); toDir.mkdirs(); deleteContent(toDir); - DerivedConfiguration config = new DerivedConfiguration(search, builder.getRankProfileRegistry()); + DerivedConfiguration config = new DerivedConfiguration(schema, builder.getRankProfileRegistry()); config.export(toDirName); checkDir(toDirName, expectedResultsDirName); diff --git a/config-model/src/test/java/com/yahoo/searchdefinition/derived/SummaryMapTestCase.java b/config-model/src/test/java/com/yahoo/searchdefinition/derived/SummaryMapTestCase.java index adc8b82f5fe..92b7184959c 100644 --- a/config-model/src/test/java/com/yahoo/searchdefinition/derived/SummaryMapTestCase.java +++ b/config-model/src/test/java/com/yahoo/searchdefinition/derived/SummaryMapTestCase.java @@ -15,6 +15,7 @@ import org.junit.Test; import java.io.IOException; import java.util.Iterator; +import java.util.Set; import static com.yahoo.config.model.test.TestUtil.joinLines; import static org.junit.Assert.assertEquals; @@ -25,11 +26,11 @@ import static org.junit.Assert.assertTrue; * * @author bratseth */ -public class SummaryMapTestCase extends SchemaTestCase { +public class SummaryMapTestCase extends AbstractSchemaTestCase { @Test public void testDeriving() throws IOException, ParseException { - Search search = SearchBuilder.buildFromFile("src/test/examples/simple.sd"); - SummaryMap summaryMap=new SummaryMap(search); + Schema schema = SearchBuilder.buildFromFile("src/test/examples/simple.sd"); + SummaryMap summaryMap=new SummaryMap(schema); Iterator transforms=summaryMap.resultTransformIterator(); FieldResultTransform transform = (FieldResultTransform)transforms.next(); @@ -72,14 +73,15 @@ public class SummaryMapTestCase extends SchemaTestCase { } @Test public void testPositionDeriving() { - Search search = new Search("store"); + Schema schema = new Schema("store"); SDDocumentType document = new SDDocumentType("store"); - search.addDocument(document); + schema.addDocument(document); String fieldName = "location"; SDField field = document.addField(fieldName, PositionDataType.INSTANCE); field.parseIndexingScript("{ attribute | summary }"); - new Processing().process(search, new BaseDeployLogger(), new RankProfileRegistry(), new QueryProfiles(), true, false); - SummaryMap summaryMap = new SummaryMap(search); + new Processing().process(schema, new BaseDeployLogger(), new RankProfileRegistry(), new QueryProfiles(), + true, false, Set.of()); + SummaryMap summaryMap = new SummaryMap(schema); Iterator transforms = summaryMap.resultTransformIterator(); @@ -186,7 +188,7 @@ public class SummaryMapTestCase extends SchemaTestCase { assertEquals(expFieldName, override.arguments()); } - private Search buildSearch(String field) throws ParseException { + private Schema buildSearch(String field) throws ParseException { var builder = new SearchBuilder(new RankProfileRegistry()); builder.importString(joinLines("search test {", " document test {", diff --git a/config-model/src/test/java/com/yahoo/searchdefinition/derived/SummaryTestCase.java b/config-model/src/test/java/com/yahoo/searchdefinition/derived/SummaryTestCase.java index e92f81052de..ea801b54b5c 100644 --- a/config-model/src/test/java/com/yahoo/searchdefinition/derived/SummaryTestCase.java +++ b/config-model/src/test/java/com/yahoo/searchdefinition/derived/SummaryTestCase.java @@ -2,9 +2,9 @@ package com.yahoo.searchdefinition.derived; import com.yahoo.config.model.application.provider.BaseDeployLogger; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.SearchBuilder; -import com.yahoo.searchdefinition.SchemaTestCase; +import com.yahoo.searchdefinition.AbstractSchemaTestCase; import com.yahoo.searchdefinition.parser.ParseException; import com.yahoo.vespa.config.search.SummaryConfig; import org.junit.Test; @@ -22,7 +22,7 @@ import static org.junit.Assert.assertTrue; * * @author bratseth */ -public class SummaryTestCase extends SchemaTestCase { +public class SummaryTestCase extends AbstractSchemaTestCase { @Test public void deriveRawAsBase64() throws ParseException { @@ -35,8 +35,8 @@ public class SummaryTestCase extends SchemaTestCase { " }", " }", "}"); - Search search = SearchBuilder.createFromString(sd).getSearch(); - SummaryClass summary = new SummaryClass(search, search.getSummary("default"), new BaseDeployLogger()); + Schema schema = SearchBuilder.createFromString(sd).getSearch(); + SummaryClass summary = new SummaryClass(schema, schema.getSummary("default"), new BaseDeployLogger()); assertEquals(SummaryClassField.Type.RAW, summary.getField("raw_field").getType()); } @@ -50,15 +50,15 @@ public class SummaryTestCase extends SchemaTestCase { " }", " }", "}"); - Search search = SearchBuilder.createFromString(sd).getSearch(); - SummaryClass summary = new SummaryClass(search, search.getSummary("default"), new BaseDeployLogger()); + Schema schema = SearchBuilder.createFromString(sd).getSearch(); + SummaryClass summary = new SummaryClass(schema, schema.getSummary("default"), new BaseDeployLogger()); assertEquals(SummaryClassField.Type.DATA, summary.getField("raw_field").getType()); } @Test public void testDeriving() throws IOException, ParseException { - Search search = SearchBuilder.buildFromFile("src/test/examples/simple.sd"); - SummaryClass summary = new SummaryClass(search, search.getSummary("default"), new BaseDeployLogger()); + Schema schema = SearchBuilder.buildFromFile("src/test/examples/simple.sd"); + SummaryClass summary = new SummaryClass(schema, schema.getSummary("default"), new BaseDeployLogger()); assertEquals("default", summary.getName()); Iterator<SummaryClassField> fields = summary.fieldIterator(); @@ -122,18 +122,18 @@ public class SummaryTestCase extends SchemaTestCase { @Test public void reference_fields_can_be_part_of_summary_classes() throws ParseException { - Search adSearch = buildCampaignAdModel(); + Schema adSchema = buildCampaignAdModel(); - SummaryClass defaultClass = new SummaryClass(adSearch, adSearch.getSummary("default"), new BaseDeployLogger()); + SummaryClass defaultClass = new SummaryClass(adSchema, adSchema.getSummary("default"), new BaseDeployLogger()); assertEquals(SummaryClassField.Type.LONGSTRING, defaultClass.getField("campaign_ref").getType()); assertEquals(SummaryClassField.Type.LONGSTRING, defaultClass.getField("other_campaign_ref").getType()); - SummaryClass myClass = new SummaryClass(adSearch, adSearch.getSummary("my_summary"), new BaseDeployLogger()); + SummaryClass myClass = new SummaryClass(adSchema, adSchema.getSummary("my_summary"), new BaseDeployLogger()); assertNull(myClass.getField("campaign_ref")); assertEquals(SummaryClassField.Type.LONGSTRING, myClass.getField("other_campaign_ref").getType()); } - private static Search buildCampaignAdModel() throws ParseException { + private static Schema buildCampaignAdModel() throws ParseException { SearchBuilder builder = new SearchBuilder(); builder.importString("search campaign { document campaign {} }"); builder.importString(joinLines("search ad {", @@ -173,8 +173,8 @@ public class SummaryTestCase extends SchemaTestCase { assertOmitSummaryFeatures(false, search, "baz"); } - private void assertOmitSummaryFeatures(boolean expected, Search search, String summaryName) { - var summary = new SummaryClass(search, search.getSummary(summaryName), new BaseDeployLogger()); + private void assertOmitSummaryFeatures(boolean expected, Schema schema, String summaryName) { + var summary = new SummaryClass(schema, schema.getSummary(summaryName), new BaseDeployLogger()); var config = new SummaryConfig.Classes(summary.getSummaryClassConfig()); assertEquals(expected, config.omitsummaryfeatures()); } diff --git a/config-model/src/test/java/com/yahoo/searchdefinition/derived/TokenizationTestCase.java b/config-model/src/test/java/com/yahoo/searchdefinition/derived/TokenizationTestCase.java index 6fe367ef6d1..4b9d5386314 100755 --- a/config-model/src/test/java/com/yahoo/searchdefinition/derived/TokenizationTestCase.java +++ b/config-model/src/test/java/com/yahoo/searchdefinition/derived/TokenizationTestCase.java @@ -1,4 +1,4 @@ -// Copyright Verizon media. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. +// Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. package com.yahoo.searchdefinition.derived; import com.yahoo.searchdefinition.parser.ParseException; diff --git a/config-model/src/test/java/com/yahoo/searchdefinition/derived/TypeConversionTestCase.java b/config-model/src/test/java/com/yahoo/searchdefinition/derived/TypeConversionTestCase.java index 2b65361e922..13289d72884 100644 --- a/config-model/src/test/java/com/yahoo/searchdefinition/derived/TypeConversionTestCase.java +++ b/config-model/src/test/java/com/yahoo/searchdefinition/derived/TypeConversionTestCase.java @@ -4,35 +4,38 @@ package com.yahoo.searchdefinition.derived; import com.yahoo.config.model.application.provider.BaseDeployLogger; import com.yahoo.document.DataType; import com.yahoo.searchdefinition.RankProfileRegistry; -import com.yahoo.searchdefinition.Search; -import com.yahoo.searchdefinition.SchemaTestCase; +import com.yahoo.searchdefinition.Schema; +import com.yahoo.searchdefinition.AbstractSchemaTestCase; import com.yahoo.searchdefinition.document.SDDocumentType; import com.yahoo.searchdefinition.document.SDField; import com.yahoo.searchdefinition.processing.Processing; import com.yahoo.vespa.model.container.search.QueryProfiles; import org.junit.Test; +import java.util.Set; + import static org.junit.Assert.assertFalse; /** * Tests automatic type conversion using multifield indices * * @author bratseth */ -public class TypeConversionTestCase extends SchemaTestCase { +public class TypeConversionTestCase extends AbstractSchemaTestCase { /** Tests that exact-string stuff is not spilled over to the default index */ @Test public void testExactStringToStringTypeConversion() { - Search search = new Search("test"); - RankProfileRegistry rankProfileRegistry = RankProfileRegistry.createRankProfileRegistryWithBuiltinRankProfiles(search); + Schema schema = new Schema("test"); + RankProfileRegistry rankProfileRegistry = RankProfileRegistry.createRankProfileRegistryWithBuiltinRankProfiles(schema); SDDocumentType document = new SDDocumentType("test"); - search.addDocument(document); + schema.addDocument(document); SDField a = new SDField("a", DataType.STRING); a.parseIndexingScript("{ index }"); document.addField(a); - new Processing().process(search, new BaseDeployLogger(), rankProfileRegistry, new QueryProfiles(), true, false); - DerivedConfiguration derived = new DerivedConfiguration(search, rankProfileRegistry); + new Processing().process(schema, new BaseDeployLogger(), rankProfileRegistry, new QueryProfiles(), + true, false, Set.of()); + DerivedConfiguration derived = new DerivedConfiguration(schema, rankProfileRegistry); IndexInfo indexInfo = derived.getIndexInfo(); assertFalse(indexInfo.hasCommand("default", "compact-to-term")); } diff --git a/config-model/src/test/java/com/yahoo/searchdefinition/derived/VsmFieldsTestCase.java b/config-model/src/test/java/com/yahoo/searchdefinition/derived/VsmFieldsTestCase.java index 32c22b583d6..55702c9e1a7 100644 --- a/config-model/src/test/java/com/yahoo/searchdefinition/derived/VsmFieldsTestCase.java +++ b/config-model/src/test/java/com/yahoo/searchdefinition/derived/VsmFieldsTestCase.java @@ -6,7 +6,8 @@ import com.yahoo.config.model.deploy.TestProperties; import com.yahoo.config.model.test.MockApplicationPackage; import com.yahoo.document.ReferenceDataType; import com.yahoo.document.TemporaryStructuredDataType; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Application; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.document.SDDocumentType; import com.yahoo.searchdefinition.document.SDField; import com.yahoo.searchdefinition.document.TemporarySDField; @@ -22,13 +23,13 @@ public class VsmFieldsTestCase { @Test public void reference_type_field_is_unsearchable() { - Search search = new Search("test", MockApplicationPackage.createEmpty(), new MockFileRegistry(), new TestableDeployLogger(), new TestProperties()); - search.addDocument(new SDDocumentType("test")); + Schema schema = new Schema("test", new Application(MockApplicationPackage.createEmpty()), new MockFileRegistry(), new TestableDeployLogger(), new TestProperties()); + schema.addDocument(new SDDocumentType("test")); SDField refField = new TemporarySDField("ref_field", ReferenceDataType.createWithInferredId(TemporaryStructuredDataType.create("parent_type"))); refField.parseIndexingScript("{ summary }"); - search.getDocument().addField(refField); + schema.getDocument().addField(refField); - VsmFields vsmFields = new VsmFields(search); + VsmFields vsmFields = new VsmFields(schema); VsmfieldsConfig.Builder cfgBuilder = new VsmfieldsConfig.Builder(); vsmFields.getConfig(cfgBuilder); VsmfieldsConfig cfg = cfgBuilder.build(); diff --git a/config-model/src/test/java/com/yahoo/searchdefinition/document/ComplexAttributeFieldUtilsTestCase.java b/config-model/src/test/java/com/yahoo/searchdefinition/document/ComplexAttributeFieldUtilsTestCase.java index a91911f6fef..449e8054728 100644 --- a/config-model/src/test/java/com/yahoo/searchdefinition/document/ComplexAttributeFieldUtilsTestCase.java +++ b/config-model/src/test/java/com/yahoo/searchdefinition/document/ComplexAttributeFieldUtilsTestCase.java @@ -1,7 +1,7 @@ // Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. package com.yahoo.searchdefinition.document; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.SearchBuilder; import com.yahoo.searchdefinition.parser.ParseException; import org.junit.Test; @@ -17,8 +17,8 @@ public class ComplexAttributeFieldUtilsTestCase { private final ImmutableSDField field; FixtureBase(String fieldName, String sdContent) throws ParseException { - Search search = SearchBuilder.createFromString(sdContent).getSearch(); - field = search.getConcreteField(fieldName); + Schema schema = SearchBuilder.createFromString(sdContent).getSearch(); + field = schema.getConcreteField(fieldName); } public ImmutableSDField field() { diff --git a/config-model/src/test/java/com/yahoo/searchdefinition/processing/AddAttributeTransformToSummaryOfImportedFieldsTest.java b/config-model/src/test/java/com/yahoo/searchdefinition/processing/AddAttributeTransformToSummaryOfImportedFieldsTest.java index f0fab8e4810..715b7b803e2 100644 --- a/config-model/src/test/java/com/yahoo/searchdefinition/processing/AddAttributeTransformToSummaryOfImportedFieldsTest.java +++ b/config-model/src/test/java/com/yahoo/searchdefinition/processing/AddAttributeTransformToSummaryOfImportedFieldsTest.java @@ -6,8 +6,9 @@ import com.yahoo.config.model.deploy.TestProperties; import com.yahoo.config.model.test.MockApplicationPackage; import com.yahoo.document.DataType; import com.yahoo.document.Field; +import com.yahoo.searchdefinition.Application; import com.yahoo.searchdefinition.DocumentReference; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.derived.TestableDeployLogger; import com.yahoo.searchdefinition.document.ImportedField; import com.yahoo.searchdefinition.document.ImportedFields; @@ -34,39 +35,39 @@ public class AddAttributeTransformToSummaryOfImportedFieldsTest { @Test public void attribute_summary_transform_applied_to_summary_field_of_imported_field() { - Search search = createSearchWithDocument(DOCUMENT_NAME); - search.setImportedFields(createSingleImportedField(IMPORTED_FIELD_NAME)); - search.addSummary(createDocumentSummary(IMPORTED_FIELD_NAME, search)); + Schema schema = createSearchWithDocument(DOCUMENT_NAME); + schema.setImportedFields(createSingleImportedField(IMPORTED_FIELD_NAME)); + schema.addSummary(createDocumentSummary(IMPORTED_FIELD_NAME, schema)); AddAttributeTransformToSummaryOfImportedFields processor = new AddAttributeTransformToSummaryOfImportedFields( - search,null,null,null); + schema, null, null, null); processor.process(true, false); - SummaryField summaryField = search.getSummaries().get(SUMMARY_NAME).getSummaryField(IMPORTED_FIELD_NAME); + SummaryField summaryField = schema.getSummaries().get(SUMMARY_NAME).getSummaryField(IMPORTED_FIELD_NAME); SummaryTransform actualTransform = summaryField.getTransform(); assertEquals(SummaryTransform.ATTRIBUTE, actualTransform); } - private static Search createSearch(String documentType) { - return new Search(documentType, MockApplicationPackage.createEmpty(), new MockFileRegistry(), new TestableDeployLogger(), new TestProperties()); + private static Schema createSearch(String documentType) { + return new Schema(documentType, new Application(MockApplicationPackage.createEmpty()), new MockFileRegistry(), new TestableDeployLogger(), new TestProperties()); } - private static Search createSearchWithDocument(String documentName) { - Search search = createSearch(documentName); - SDDocumentType document = new SDDocumentType(documentName, search); - search.addDocument(document); - return search; + private static Schema createSearchWithDocument(String documentName) { + Schema schema = createSearch(documentName); + SDDocumentType document = new SDDocumentType(documentName, schema); + schema.addDocument(document); + return schema; } private static ImportedFields createSingleImportedField(String fieldName) { - Search targetSearch = createSearch("target_doc"); + Schema targetSchema = createSearch("target_doc"); SDField targetField = new SDField("target_field", DataType.INT); - DocumentReference documentReference = new DocumentReference(new Field("reference_field"), targetSearch); + DocumentReference documentReference = new DocumentReference(new Field("reference_field"), targetSchema); ImportedField importedField = new ImportedSimpleField(fieldName, documentReference, targetField); return new ImportedFields(Collections.singletonMap(fieldName, importedField)); } - private static DocumentSummary createDocumentSummary(String fieldName, Search search) { - DocumentSummary summary = new DocumentSummary("mysummary", search); + private static DocumentSummary createDocumentSummary(String fieldName, Schema schema) { + DocumentSummary summary = new DocumentSummary("mysummary", schema); summary.add(new SummaryField(fieldName, DataType.INT)); return summary; } diff --git a/config-model/src/test/java/com/yahoo/searchdefinition/processing/AdjustPositionSummaryFieldsTestCase.java b/config-model/src/test/java/com/yahoo/searchdefinition/processing/AdjustPositionSummaryFieldsTestCase.java index f9796432de8..3b5add5b3d8 100644 --- a/config-model/src/test/java/com/yahoo/searchdefinition/processing/AdjustPositionSummaryFieldsTestCase.java +++ b/config-model/src/test/java/com/yahoo/searchdefinition/processing/AdjustPositionSummaryFieldsTestCase.java @@ -3,7 +3,7 @@ package com.yahoo.searchdefinition.processing; import com.yahoo.document.DataType; import com.yahoo.document.PositionDataType; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.vespa.documentmodel.DocumentSummary; import com.yahoo.vespa.documentmodel.SummaryField; import com.yahoo.vespa.documentmodel.SummaryTransform; @@ -73,7 +73,7 @@ public class AdjustPositionSummaryFieldsTestCase { public void test_pos_default_summary_no_attr_no_rename() { SearchModel model = new SearchModel(false, false, false); model.resolve(); - assertNull(model.childSearch.getSummary("default")); // ImplicitSummaries processing not run in this test + assertNull(model.childSchema.getSummary("default")); // ImplicitSummaries processing not run in this test } @Test @@ -92,8 +92,7 @@ public class AdjustPositionSummaryFieldsTestCase { @Test public void test_pos_summary_no_attr() { exceptionRule.expect(IllegalArgumentException.class); - exceptionRule.expectMessage("For search 'child', field 'my_pos': " - + "No position attribute 'pos_zcurve'"); + exceptionRule.expectMessage("For schema 'child', field 'my_pos': No position attribute 'pos_zcurve'"); SearchModel model = new SearchModel(false, false, false); model.addSummaryField("my_pos", PositionDataType.INSTANCE, null, "pos"); model.resolve(); @@ -102,8 +101,7 @@ public class AdjustPositionSummaryFieldsTestCase { @Test public void test_pos_summary_bad_attr() { exceptionRule.expect(IllegalArgumentException.class); - exceptionRule.expectMessage("For search 'child', field 'my_pos': " - + "No position attribute 'pos_zcurve'"); + exceptionRule.expectMessage("For schema 'child', field 'my_pos': No position attribute 'pos_zcurve'"); SearchModel model = new SearchModel(false, false, true); model.addSummaryField("my_pos", PositionDataType.INSTANCE, null, "pos"); model.resolve(); @@ -112,7 +110,7 @@ public class AdjustPositionSummaryFieldsTestCase { @Test public void test_imported_pos_summary_no_attr() { exceptionRule.expect(IllegalArgumentException.class); - exceptionRule.expectMessage("For search 'child', import field 'my_pos_zcurve': " + exceptionRule.expectMessage("For schema 'child', import field 'my_pos_zcurve': " + "Field 'pos_zcurve' via reference field 'ref': Not found"); SearchModel model = new SearchModel(true, false, false); model.addSummaryField("my_pos", PositionDataType.INSTANCE, null, null); @@ -122,7 +120,7 @@ public class AdjustPositionSummaryFieldsTestCase { @Test public void test_imported_pos_summary_bad_attr() { exceptionRule.expect(IllegalArgumentException.class); - exceptionRule.expectMessage("For search 'child', field 'my_pos': " + exceptionRule.expectMessage("For schema 'child', field 'my_pos': " + "No position attribute 'my_pos_zcurve'"); SearchModel model = new SearchModel(true, false, true); model.addSummaryField("my_pos", PositionDataType.INSTANCE, null, null); @@ -132,7 +130,7 @@ public class AdjustPositionSummaryFieldsTestCase { @Test public void test_my_pos_position_summary_bad_datatype() { exceptionRule.expect(IllegalArgumentException.class); - exceptionRule.expectMessage("For search 'child', field 'my_pos.position': " + exceptionRule.expectMessage("For schema 'child', field 'my_pos.position': " + "exists with type 'datatype string (code: 2)', should be of type 'datatype Array<string> (code: -1486737430)"); SearchModel model = new SearchModel(); model.addSummaryField("my_pos", PositionDataType.INSTANCE, null, null); @@ -143,7 +141,7 @@ public class AdjustPositionSummaryFieldsTestCase { @Test public void test_my_pos_position_summary_bad_transform() { exceptionRule.expect(IllegalArgumentException.class); - exceptionRule.expectMessage("For search 'child', field 'my_pos.position': " + exceptionRule.expectMessage("For schema 'child', field 'my_pos.position': " + "has summary transform 'none', should have transform 'positions'"); SearchModel model = new SearchModel(); model.addSummaryField("my_pos", PositionDataType.INSTANCE, null, null); @@ -154,7 +152,7 @@ public class AdjustPositionSummaryFieldsTestCase { @Test public void test_my_pos_position_summary_bad_source() { exceptionRule.expect(IllegalArgumentException.class); - exceptionRule.expectMessage("For search 'child', field 'my_pos.position': " + exceptionRule.expectMessage("For schema 'child', field 'my_pos.position': " + "has source '[source field 'pos']', should have source 'source field 'my_pos_zcurve''"); SearchModel model = new SearchModel(); model.addSummaryField("my_pos", PositionDataType.INSTANCE, null, null); @@ -175,21 +173,21 @@ public class AdjustPositionSummaryFieldsTestCase { SearchModel(boolean importedPos, boolean setupPosAttr, boolean setupBadAttr) { super(); if (importedPos) { - createPositionField(parentSearch, setupPosAttr, setupBadAttr); + createPositionField(parentSchema, setupPosAttr, setupBadAttr); } - addRefField(childSearch, parentSearch, "ref"); + addRefField(childSchema, parentSchema, "ref"); if (importedPos) { addImportedField("my_pos", "ref", "pos"); } else { - createPositionField(childSearch, setupPosAttr, setupBadAttr); + createPositionField(childSchema, setupPosAttr, setupBadAttr); } } - private void createPositionField(Search search, boolean setupPosAttr, boolean setupBadAttr) { + private void createPositionField(Schema schema, boolean setupPosAttr, boolean setupBadAttr) { String ilScript = setupPosAttr ? "{ summary | attribute }" : "{ summary }"; - search.getDocument().addField(createField("pos", PositionDataType.INSTANCE, ilScript)); + schema.getDocument().addField(createField("pos", PositionDataType.INSTANCE, ilScript)); if (setupBadAttr) { - search.getDocument().addField(createField("pos_zcurve", DataType.LONG, "{ attribute }")); + schema.getDocument().addField(createField("pos_zcurve", DataType.LONG, "{ attribute }")); } } @@ -198,10 +196,10 @@ public class AdjustPositionSummaryFieldsTestCase { } public void addSummaryField(String summaryName, String fieldName, DataType dataType, SummaryTransform transform, String source) { - DocumentSummary summary = childSearch.getSummary(summaryName); + DocumentSummary summary = childSchema.getSummary(summaryName); if (summary == null) { - summary = new DocumentSummary(summaryName, childSearch); - childSearch.addSummary(summary); + summary = new DocumentSummary(summaryName, childSchema); + childSchema.addSummary(summary); } SummaryField summaryField = new SummaryField(fieldName, dataType); if (source != null) { @@ -218,7 +216,7 @@ public class AdjustPositionSummaryFieldsTestCase { } public void assertNoSummaryField(String summaryName, String fieldName) { - DocumentSummary summary = childSearch.getSummary(summaryName); + DocumentSummary summary = childSchema.getSummary(summaryName); assertNotNull(summary); SummaryField summaryField = summary.getSummaryField(fieldName); assertNull(summaryField); @@ -229,7 +227,7 @@ public class AdjustPositionSummaryFieldsTestCase { } public void assertSummaryField(String summaryName, String fieldName, DataType dataType, SummaryTransform transform, String source) { - DocumentSummary summary = childSearch.getSummary(summaryName); + DocumentSummary summary = childSchema.getSummary(summaryName); assertNotNull(summary); SummaryField summaryField = summary.getSummaryField(fieldName); assertNotNull(summaryField); @@ -244,18 +242,17 @@ public class AdjustPositionSummaryFieldsTestCase { } public void resolve() { - resolve(parentSearch); - resolve(childSearch); + resolve(parentSchema); + resolve(childSchema); } - private static void resolve(Search search) { - new CreatePositionZCurve(search, null, null, null).process(true, false); - assertNotNull(search.temporaryImportedFields().get()); - assertFalse(search.importedFields().isPresent()); - new ImportedFieldsResolver(search, null, null, null).process(true, false); - assertFalse(search.temporaryImportedFields().isPresent()); - assertNotNull(search.importedFields().get()); - new AdjustPositionSummaryFields(search, null, null, null).process(true, false); + private static void resolve(Schema schema) { + new CreatePositionZCurve(schema, null, null, null).process(true, false); + assertNotNull(schema.temporaryImportedFields().get()); + assertFalse(schema.importedFields().isPresent()); + new ImportedFieldsResolver(schema, null, null, null).process(true, false); + assertNotNull(schema.importedFields().get()); + new AdjustPositionSummaryFields(schema, null, null, null).process(true, false); } } } diff --git a/config-model/src/test/java/com/yahoo/searchdefinition/processing/AssertIndexingScript.java b/config-model/src/test/java/com/yahoo/searchdefinition/processing/AssertIndexingScript.java index c9f9e8c50f9..8a0c1e2fe71 100644 --- a/config-model/src/test/java/com/yahoo/searchdefinition/processing/AssertIndexingScript.java +++ b/config-model/src/test/java/com/yahoo/searchdefinition/processing/AssertIndexingScript.java @@ -1,7 +1,7 @@ // Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. package com.yahoo.searchdefinition.processing; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.derived.IndexingScript; import com.yahoo.vespa.indexinglanguage.expressions.Expression; import com.yahoo.vespa.indexinglanguage.parser.ParseException; @@ -17,8 +17,8 @@ import static org.junit.Assert.fail; */ public abstract class AssertIndexingScript { - public static void assertIndexing(List<String> expected, Search search) { - assertIndexing(expected, new IndexingScript(search).expressions()); + public static void assertIndexing(List<String> expected, Schema schema) { + assertIndexing(expected, new IndexingScript(schema).expressions()); } public static void assertIndexing(List<String> expected, IndexingScript script) { diff --git a/config-model/src/test/java/com/yahoo/searchdefinition/processing/AttributesExactMatchTestCase.java b/config-model/src/test/java/com/yahoo/searchdefinition/processing/AttributesExactMatchTestCase.java index 7f72cc96ccf..57e01cb9dfc 100644 --- a/config-model/src/test/java/com/yahoo/searchdefinition/processing/AttributesExactMatchTestCase.java +++ b/config-model/src/test/java/com/yahoo/searchdefinition/processing/AttributesExactMatchTestCase.java @@ -1,9 +1,9 @@ // Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. package com.yahoo.searchdefinition.processing; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.SearchBuilder; -import com.yahoo.searchdefinition.SchemaTestCase; +import com.yahoo.searchdefinition.AbstractSchemaTestCase; import com.yahoo.searchdefinition.document.Matching; import com.yahoo.searchdefinition.parser.ParseException; import org.junit.Test; @@ -17,23 +17,23 @@ import static org.junit.Assert.assertFalse; * @author vegardh * */ -public class AttributesExactMatchTestCase extends SchemaTestCase { +public class AttributesExactMatchTestCase extends AbstractSchemaTestCase { @Test public void testAttributesExactMatch() throws IOException, ParseException { - Search search = SearchBuilder.buildFromFile("src/test/examples/attributesexactmatch.sd"); - assertEquals(search.getConcreteField("color").getMatching().getType(), Matching.Type.EXACT); - assertEquals(search.getConcreteField("artist").getMatching().getType(), Matching.Type.WORD); - assertEquals(search.getConcreteField("drummer").getMatching().getType(), Matching.Type.WORD); - assertEquals(search.getConcreteField("guitarist").getMatching().getType(), Matching.Type.TEXT); - assertEquals(search.getConcreteField("saxophonist_arr").getMatching().getType(), Matching.Type.WORD); - assertEquals(search.getConcreteField("flutist").getMatching().getType(), Matching.Type.TEXT); + Schema schema = SearchBuilder.buildFromFile("src/test/examples/attributesexactmatch.sd"); + assertEquals(schema.getConcreteField("color").getMatching().getType(), Matching.Type.EXACT); + assertEquals(schema.getConcreteField("artist").getMatching().getType(), Matching.Type.WORD); + assertEquals(schema.getConcreteField("drummer").getMatching().getType(), Matching.Type.WORD); + assertEquals(schema.getConcreteField("guitarist").getMatching().getType(), Matching.Type.TEXT); + assertEquals(schema.getConcreteField("saxophonist_arr").getMatching().getType(), Matching.Type.WORD); + assertEquals(schema.getConcreteField("flutist").getMatching().getType(), Matching.Type.TEXT); - assertFalse(search.getConcreteField("genre").getMatching().getType().equals(Matching.Type.EXACT)); - assertFalse(search.getConcreteField("title").getMatching().getType().equals(Matching.Type.EXACT)); - assertFalse(search.getConcreteField("trumpetist").getMatching().getType().equals(Matching.Type.EXACT)); - assertFalse(search.getConcreteField("genre").getMatching().getType().equals(Matching.Type.WORD)); - assertFalse(search.getConcreteField("title").getMatching().getType().equals(Matching.Type.WORD)); - assertFalse(search.getConcreteField("trumpetist").getMatching().getType().equals(Matching.Type.WORD)); + assertFalse(schema.getConcreteField("genre").getMatching().getType().equals(Matching.Type.EXACT)); + assertFalse(schema.getConcreteField("title").getMatching().getType().equals(Matching.Type.EXACT)); + assertFalse(schema.getConcreteField("trumpetist").getMatching().getType().equals(Matching.Type.EXACT)); + assertFalse(schema.getConcreteField("genre").getMatching().getType().equals(Matching.Type.WORD)); + assertFalse(schema.getConcreteField("title").getMatching().getType().equals(Matching.Type.WORD)); + assertFalse(schema.getConcreteField("trumpetist").getMatching().getType().equals(Matching.Type.WORD)); } diff --git a/config-model/src/test/java/com/yahoo/searchdefinition/processing/BoldingTestCase.java b/config-model/src/test/java/com/yahoo/searchdefinition/processing/BoldingTestCase.java index 288987e0f2f..d9d8e47338b 100644 --- a/config-model/src/test/java/com/yahoo/searchdefinition/processing/BoldingTestCase.java +++ b/config-model/src/test/java/com/yahoo/searchdefinition/processing/BoldingTestCase.java @@ -2,7 +2,7 @@ package com.yahoo.searchdefinition.processing; import com.yahoo.searchdefinition.SearchBuilder; -import com.yahoo.searchdefinition.SchemaTestCase; +import com.yahoo.searchdefinition.AbstractSchemaTestCase; import com.yahoo.searchdefinition.parser.ParseException; import org.junit.Test; @@ -12,7 +12,7 @@ import static org.junit.Assert.fail; /** * @author bratseth */ -public class BoldingTestCase extends SchemaTestCase { +public class BoldingTestCase extends AbstractSchemaTestCase { private final String boldonnonstring = "search boldnonstring {\n" + diff --git a/config-model/src/test/java/com/yahoo/searchdefinition/processing/BoolAttributeValidatorTestCase.java b/config-model/src/test/java/com/yahoo/searchdefinition/processing/BoolAttributeValidatorTestCase.java index a50af793bc4..fbfb46be558 100644 --- a/config-model/src/test/java/com/yahoo/searchdefinition/processing/BoolAttributeValidatorTestCase.java +++ b/config-model/src/test/java/com/yahoo/searchdefinition/processing/BoolAttributeValidatorTestCase.java @@ -21,7 +21,7 @@ public class BoolAttributeValidatorTestCase { fail("Expected exception"); } catch (IllegalArgumentException e) { - assertEquals("For search 'test', field 'b': Only single value bool attribute fields are supported", + assertEquals("For schema 'test', field 'b': Only single value bool attribute fields are supported", e.getMessage()); } } @@ -33,13 +33,14 @@ public class BoolAttributeValidatorTestCase { fail("Expected exception"); } catch (IllegalArgumentException e) { - assertEquals("For search 'test', field 'b': Only single value bool attribute fields are supported", + assertEquals("For schema 'test', field 'b': Only single value bool attribute fields are supported", e.getMessage()); } } private String getSd(String field) { - return joinLines("search test {", + return joinLines( + "schema test {", " document test {", " " + field, " }", diff --git a/config-model/src/test/java/com/yahoo/searchdefinition/processing/DictionaryTestCase.java b/config-model/src/test/java/com/yahoo/searchdefinition/processing/DictionaryTestCase.java index db87ae698b4..20ff10cb48e 100644 --- a/config-model/src/test/java/com/yahoo/searchdefinition/processing/DictionaryTestCase.java +++ b/config-model/src/test/java/com/yahoo/searchdefinition/processing/DictionaryTestCase.java @@ -3,13 +3,12 @@ package com.yahoo.searchdefinition.processing; import com.yahoo.config.model.test.TestUtil; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.SearchBuilder; import com.yahoo.searchdefinition.derived.AttributeFields; import com.yahoo.searchdefinition.document.Case; import com.yahoo.searchdefinition.document.Dictionary; import com.yahoo.searchdefinition.document.ImmutableSDField; -import com.yahoo.searchdefinition.document.Matching; import com.yahoo.searchdefinition.parser.ParseException; import com.yahoo.vespa.config.search.AttributesConfig; import org.junit.Test; @@ -24,13 +23,13 @@ import static org.junit.Assert.fail; * @author baldersheim */ public class DictionaryTestCase { - private static AttributesConfig getConfig(Search search) { - AttributeFields attributes = new AttributeFields(search); + private static AttributesConfig getConfig(Schema schema) { + AttributeFields attributes = new AttributeFields(schema); AttributesConfig.Builder builder = new AttributesConfig.Builder(); attributes.getConfig(builder); return builder.build(); } - private Search createSearch(String def) throws ParseException { + private Schema createSearch(String def) throws ParseException { SearchBuilder sb = SearchBuilder.createFromString(def); return sb.getSearch(); } @@ -47,20 +46,20 @@ public class DictionaryTestCase { " }", " }", "}"); - Search search = createSearch(def); - assertNull(search.getAttribute("s1").getDictionary()); - assertNull(search.getAttribute("n1").getDictionary()); + Schema schema = createSearch(def); + assertNull(schema.getAttribute("s1").getDictionary()); + assertNull(schema.getAttribute("n1").getDictionary()); assertEquals(AttributesConfig.Attribute.Dictionary.Type.BTREE, - getConfig(search).attribute().get(0).dictionary().type()); + getConfig(schema).attribute().get(0).dictionary().type()); assertEquals(AttributesConfig.Attribute.Dictionary.Type.BTREE, - getConfig(search).attribute().get(1).dictionary().type()); + getConfig(schema).attribute().get(1).dictionary().type()); assertEquals(AttributesConfig.Attribute.Dictionary.Match.UNCASED, - getConfig(search).attribute().get(0).dictionary().match()); + getConfig(schema).attribute().get(0).dictionary().match()); assertEquals(AttributesConfig.Attribute.Dictionary.Match.UNCASED, - getConfig(search).attribute().get(1).dictionary().match()); + getConfig(schema).attribute().get(1).dictionary().match()); } - Search verifyDictionaryControl(Dictionary.Type expected, String type, String ... cfg) throws ParseException + Schema verifyDictionaryControl(Dictionary.Type expected, String type, String ... cfg) throws ParseException { String def = TestUtil.joinLines( "search test {", @@ -72,11 +71,11 @@ public class DictionaryTestCase { " }", " }", "}"); - Search search = createSearch(def); + Schema schema = createSearch(def); AttributesConfig.Attribute.Dictionary.Type.Enum expectedConfig = toCfg(expected); - assertEquals(expected, search.getAttribute("n1").getDictionary().getType()); - assertEquals(expectedConfig, getConfig(search).attribute().get(0).dictionary().type()); - return search; + assertEquals(expected, schema.getAttribute("n1").getDictionary().getType()); + assertEquals(expectedConfig, getConfig(schema).attribute().get(0).dictionary().type()); + return schema; } AttributesConfig.Attribute.Dictionary.Type.Enum toCfg(Dictionary.Type v) { @@ -95,12 +94,12 @@ public class DictionaryTestCase { void verifyStringDictionaryControl(Dictionary.Type expectedType, Case expectedCase, Case matchCasing, String ... cfg) throws ParseException { - Search search = verifyDictionaryControl(expectedType, "string", cfg); - ImmutableSDField f = search.getField("n1"); + Schema schema = verifyDictionaryControl(expectedType, "string", cfg); + ImmutableSDField f = schema.getField("n1"); AttributesConfig.Attribute.Dictionary.Match.Enum expectedCaseCfg = toCfg(expectedCase); assertEquals(matchCasing, f.getMatching().getCase()); - assertEquals(expectedCase, search.getAttribute("n1").getDictionary().getMatch()); - assertEquals(expectedCaseCfg, getConfig(search).attribute().get(0).dictionary().match()); + assertEquals(expectedCase, schema.getAttribute("n1").getDictionary().getMatch()); + assertEquals(expectedCaseCfg, getConfig(schema).attribute().get(0).dictionary().match()); } @Test @@ -145,7 +144,7 @@ public class DictionaryTestCase { try { verifyStringDictionaryControl(Dictionary.Type.HASH, Case.UNCASED, Case.UNCASED, "dictionary:hash"); } catch (IllegalArgumentException e) { - assertEquals("For search 'test', field 'n1': hash dictionary require cased match", e.getMessage()); + assertEquals("For schema 'test', field 'n1': hash dictionary require cased match", e.getMessage()); } } @Test @@ -153,7 +152,7 @@ public class DictionaryTestCase { try { verifyStringDictionaryControl(Dictionary.Type.HASH, Case.UNCASED, Case.UNCASED, "dictionary { hash\nuncased\n}"); } catch (IllegalArgumentException e) { - assertEquals("For search 'test', field 'n1': hash dictionary require cased match", e.getMessage()); + assertEquals("For schema 'test', field 'n1': hash dictionary require cased match", e.getMessage()); } } @Test @@ -166,7 +165,7 @@ public class DictionaryTestCase { verifyStringDictionaryControl(Dictionary.Type.HASH, Case.CASED, Case.CASED, "dictionary { hash\ncased\n}"); fail(); } catch (IllegalArgumentException e) { - assertEquals("For search 'test', field 'n1': Dictionary casing 'CASED' does not match field match casing 'UNCASED'", e.getMessage()); + assertEquals("For schema 'test', field 'n1': Dictionary casing 'CASED' does not match field match casing 'UNCASED'", e.getMessage()); } } @Test @@ -182,13 +181,13 @@ public class DictionaryTestCase { try { verifyStringDictionaryControl(Dictionary.Type.BTREE_AND_HASH, Case.CASED, Case.CASED, "dictionary { btree\nhash\ncased\n}"); } catch (IllegalArgumentException e) { - assertEquals("For search 'test', field 'n1': Dictionary casing 'CASED' does not match field match casing 'UNCASED'", e.getMessage()); + assertEquals("For schema 'test', field 'n1': Dictionary casing 'CASED' does not match field match casing 'UNCASED'", e.getMessage()); } } @Test public void testNonNumericFieldsFailsDictionaryControl() throws ParseException { String def = TestUtil.joinLines( - "search test {", + "schema test {", " document test {", " field n1 type bool {", " indexing: summary | attribute", @@ -200,13 +199,13 @@ public class DictionaryTestCase { SearchBuilder sb = SearchBuilder.createFromString(def); fail("Controlling dictionary for non-numeric fields are not yet supported."); } catch (IllegalArgumentException e) { - assertEquals("For search 'test', field 'n1': You can only specify 'dictionary:' for numeric or string fields", e.getMessage()); + assertEquals("For schema 'test', field 'n1': You can only specify 'dictionary:' for numeric or string fields", e.getMessage()); } } @Test public void testNonFastSearchNumericFieldsFailsDictionaryControl() throws ParseException { String def = TestUtil.joinLines( - "search test {", + "schema test {", " document test {", " field n1 type int {", " indexing: summary | attribute", @@ -218,14 +217,14 @@ public class DictionaryTestCase { SearchBuilder sb = SearchBuilder.createFromString(def); fail("Controlling dictionary for non-fast-search fields are not allowed."); } catch (IllegalArgumentException e) { - assertEquals("For search 'test', field 'n1': You must specify 'attribute:fast-search' to allow dictionary control", e.getMessage()); + assertEquals("For schema 'test', field 'n1': You must specify 'attribute:fast-search' to allow dictionary control", e.getMessage()); } } @Test public void testCasingForNonFastSearch() throws ParseException { String def = TestUtil.joinLines( - "search test {", + "schema test {", " document test {", " field s1 type string {", " indexing: attribute | summary", @@ -240,12 +239,12 @@ public class DictionaryTestCase { " }", " }", "}"); - Search search = createSearch(def); - assertEquals(Case.UNCASED, search.getAttribute("s1").getCase()); - assertEquals(Case.UNCASED, search.getAttribute("s2").getCase()); - assertEquals(Case.CASED, search.getAttribute("s3").getCase()); - assertEquals(AttributesConfig.Attribute.Match.UNCASED, getConfig(search).attribute().get(0).match()); - assertEquals(AttributesConfig.Attribute.Match.UNCASED, getConfig(search).attribute().get(1).match()); - assertEquals(AttributesConfig.Attribute.Match.CASED, getConfig(search).attribute().get(2).match()); + Schema schema = createSearch(def); + assertEquals(Case.UNCASED, schema.getAttribute("s1").getCase()); + assertEquals(Case.UNCASED, schema.getAttribute("s2").getCase()); + assertEquals(Case.CASED, schema.getAttribute("s3").getCase()); + assertEquals(AttributesConfig.Attribute.Match.UNCASED, getConfig(schema).attribute().get(0).match()); + assertEquals(AttributesConfig.Attribute.Match.UNCASED, getConfig(schema).attribute().get(1).match()); + assertEquals(AttributesConfig.Attribute.Match.CASED, getConfig(schema).attribute().get(2).match()); } } diff --git a/config-model/src/test/java/com/yahoo/searchdefinition/processing/FastAccessValidatorTest.java b/config-model/src/test/java/com/yahoo/searchdefinition/processing/FastAccessValidatorTest.java index c889a8ccd34..0a9807e0746 100644 --- a/config-model/src/test/java/com/yahoo/searchdefinition/processing/FastAccessValidatorTest.java +++ b/config-model/src/test/java/com/yahoo/searchdefinition/processing/FastAccessValidatorTest.java @@ -22,14 +22,14 @@ public class FastAccessValidatorTest { SearchBuilder builder = new SearchBuilder(new RankProfileRegistry()); builder.importString( TestUtil.joinLines( - "search parent {", + "schema parent {", " document parent {", " field int_field type int { indexing: attribute }", " }", "}")); builder.importString( TestUtil.joinLines( - "search test {", + "schema test {", " document test { ", " field int_attribute type int { ", " indexing: attribute ", @@ -51,7 +51,7 @@ public class FastAccessValidatorTest { "}")); exceptionRule.expect(IllegalArgumentException.class); exceptionRule.expectMessage( - "For search 'test': The following attributes have a type that is incompatible " + + "For schema 'test': The following attributes have a type that is incompatible " + "with fast-access: predicate_attribute, tensor_attribute, reference_attribute. " + "Predicate, tensor and reference attributes are incompatible with fast-access."); builder.build(); diff --git a/config-model/src/test/java/com/yahoo/searchdefinition/processing/ImplicitSearchFieldsTestCase.java b/config-model/src/test/java/com/yahoo/searchdefinition/processing/ImplicitSchemaFieldsTestCase.java index 5ef2f2842e1..dffc0d968df 100644 --- a/config-model/src/test/java/com/yahoo/searchdefinition/processing/ImplicitSearchFieldsTestCase.java +++ b/config-model/src/test/java/com/yahoo/searchdefinition/processing/ImplicitSchemaFieldsTestCase.java @@ -1,9 +1,9 @@ // Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. package com.yahoo.searchdefinition.processing; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.SearchBuilder; -import com.yahoo.searchdefinition.SchemaTestCase; +import com.yahoo.searchdefinition.AbstractSchemaTestCase; import com.yahoo.searchdefinition.derived.DerivedConfiguration; import com.yahoo.searchdefinition.document.SDDocumentType; import com.yahoo.searchdefinition.parser.ParseException; @@ -14,14 +14,14 @@ import java.io.IOException; import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertNotNull; -public class ImplicitSearchFieldsTestCase extends SchemaTestCase { +public class ImplicitSchemaFieldsTestCase extends AbstractSchemaTestCase { @Test public void testRequireThatExtraFieldsAreIncluded() throws IOException, ParseException { - Search search = SearchBuilder.buildFromFile("src/test/examples/nextgen/extrafield.sd"); - assertNotNull(search); + Schema schema = SearchBuilder.buildFromFile("src/test/examples/nextgen/extrafield.sd"); + assertNotNull(schema); - SDDocumentType docType = search.getDocument(); + SDDocumentType docType = schema.getDocument(); assertNotNull(docType); assertNotNull(docType.getField("foo")); assertNotNull(docType.getField("bar")); @@ -30,10 +30,10 @@ public class ImplicitSearchFieldsTestCase extends SchemaTestCase { @Test public void testRequireThatSummaryFieldsAreIncluded() throws IOException, ParseException { - Search search = SearchBuilder.buildFromFile("src/test/examples/nextgen/summaryfield.sd"); - assertNotNull(search); + Schema schema = SearchBuilder.buildFromFile("src/test/examples/nextgen/summaryfield.sd"); + assertNotNull(schema); - SDDocumentType docType = search.getDocument(); + SDDocumentType docType = schema.getDocument(); assertNotNull(docType); assertNotNull(docType.getField("foo")); assertNotNull(docType.getField("bar")); @@ -43,10 +43,10 @@ public class ImplicitSearchFieldsTestCase extends SchemaTestCase { @Test public void testRequireThatBoldedSummaryFieldsAreIncluded() throws IOException, ParseException { - Search search = SearchBuilder.buildFromFile("src/test/examples/nextgen/boldedsummaryfields.sd"); - assertNotNull(search); + Schema schema = SearchBuilder.buildFromFile("src/test/examples/nextgen/boldedsummaryfields.sd"); + assertNotNull(schema); - SDDocumentType docType = search.getDocument(); + SDDocumentType docType = schema.getDocument(); assertNotNull(docType); assertNotNull(docType.getField("foo")); assertNotNull(docType.getField("bar")); @@ -57,10 +57,10 @@ public class ImplicitSearchFieldsTestCase extends SchemaTestCase { @Test public void testRequireThatUntransformedSummaryFieldsAreIgnored() throws IOException, ParseException { - Search search = SearchBuilder.buildFromFile("src/test/examples/nextgen/untransformedsummaryfields.sd"); - assertNotNull(search); + Schema schema = SearchBuilder.buildFromFile("src/test/examples/nextgen/untransformedsummaryfields.sd"); + assertNotNull(schema); - SDDocumentType docType = search.getDocument(); + SDDocumentType docType = schema.getDocument(); assertNotNull(docType); assertNotNull(docType.getField("foo")); assertNotNull(docType.getField("bar")); @@ -70,10 +70,10 @@ public class ImplicitSearchFieldsTestCase extends SchemaTestCase { @Test public void testRequireThatDynamicSummaryFieldsAreIgnored() throws IOException, ParseException { - Search search = SearchBuilder.buildFromFile("src/test/examples/nextgen/dynamicsummaryfields.sd"); - assertNotNull(search); + Schema schema = SearchBuilder.buildFromFile("src/test/examples/nextgen/dynamicsummaryfields.sd"); + assertNotNull(schema); - SDDocumentType docType = search.getDocument(); + SDDocumentType docType = schema.getDocument(); assertNotNull(docType); assertNotNull(docType.getField("foo")); assertNotNull(docType.getField("bar")); diff --git a/config-model/src/test/java/com/yahoo/searchdefinition/processing/ImplicitStructTypesTestCase.java b/config-model/src/test/java/com/yahoo/searchdefinition/processing/ImplicitStructTypesTestCase.java index fb0f1bc2ba2..b069f62ee92 100644 --- a/config-model/src/test/java/com/yahoo/searchdefinition/processing/ImplicitStructTypesTestCase.java +++ b/config-model/src/test/java/com/yahoo/searchdefinition/processing/ImplicitStructTypesTestCase.java @@ -2,9 +2,9 @@ package com.yahoo.searchdefinition.processing; import com.yahoo.document.*; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.SearchBuilder; -import com.yahoo.searchdefinition.SchemaTestCase; +import com.yahoo.searchdefinition.AbstractSchemaTestCase; import com.yahoo.searchdefinition.document.SDDocumentType; import com.yahoo.searchdefinition.document.SDField; import com.yahoo.searchdefinition.parser.ParseException; @@ -13,22 +13,22 @@ import org.junit.Test; import java.io.IOException; import static org.junit.Assert.*; -public class ImplicitStructTypesTestCase extends SchemaTestCase { +public class ImplicitStructTypesTestCase extends AbstractSchemaTestCase { @Test public void testRequireThatImplicitStructsAreCreated() throws IOException, ParseException { - Search search = SearchBuilder.buildFromFile("src/test/examples/nextgen/toggleon.sd"); - assertNotNull(search); + Schema schema = SearchBuilder.buildFromFile("src/test/examples/nextgen/toggleon.sd"); + assertNotNull(schema); - SDDocumentType docType = search.getDocument(); + SDDocumentType docType = schema.getDocument(); assertNotNull(docType); assertStruct(docType, PositionDataType.INSTANCE); } @Test public void testRequireThatImplicitStructsAreUsed() throws IOException, ParseException { - Search search = SearchBuilder.buildFromFile("src/test/examples/nextgen/implicitstructtypes.sd"); - assertNotNull(search); + Schema schema = SearchBuilder.buildFromFile("src/test/examples/nextgen/implicitstructtypes.sd"); + assertNotNull(schema); - SDDocumentType docType = search.getDocument(); + SDDocumentType docType = schema.getDocument(); assertNotNull(docType); assertField(docType, "doc_str", DataType.STRING); diff --git a/config-model/src/test/java/com/yahoo/searchdefinition/processing/ImplicitSummariesTestCase.java b/config-model/src/test/java/com/yahoo/searchdefinition/processing/ImplicitSummariesTestCase.java index d45d366ee0a..9f4008b5b39 100644 --- a/config-model/src/test/java/com/yahoo/searchdefinition/processing/ImplicitSummariesTestCase.java +++ b/config-model/src/test/java/com/yahoo/searchdefinition/processing/ImplicitSummariesTestCase.java @@ -1,7 +1,7 @@ // Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. package com.yahoo.searchdefinition.processing; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.SearchBuilder; import com.yahoo.searchdefinition.parser.ParseException; import com.yahoo.vespa.documentmodel.SummaryTransform; @@ -29,8 +29,8 @@ public class ImplicitSummariesTestCase { LogHandler log = new LogHandler(); Logger.getLogger("").addHandler(log); - Search search = SearchBuilder.buildFromFile("src/test/examples/implicitsummaries_attribute.sd"); - assertNotNull(search); + Schema schema = SearchBuilder.buildFromFile("src/test/examples/implicitsummaries_attribute.sd"); + assertNotNull(schema); assertTrue(log.records.isEmpty()); } @@ -60,19 +60,19 @@ public class ImplicitSummariesTestCase { @Test public void attribute_combiner_transform_is_set_on_array_of_struct_with_only_struct_field_attributes() throws IOException, ParseException { - Search search = SearchBuilder.buildFromFile("src/test/derived/array_of_struct_attribute/test.sd"); - assertEquals(SummaryTransform.ATTRIBUTECOMBINER, search.getSummaryField("elem_array").getTransform()); + Schema schema = SearchBuilder.buildFromFile("src/test/derived/array_of_struct_attribute/test.sd"); + assertEquals(SummaryTransform.ATTRIBUTECOMBINER, schema.getSummaryField("elem_array").getTransform()); } @Test public void attribute_combiner_transform_is_set_on_map_of_struct_with_only_struct_field_attributes() throws IOException, ParseException { - Search search = SearchBuilder.buildFromFile("src/test/derived/map_of_struct_attribute/test.sd"); - assertEquals(SummaryTransform.ATTRIBUTECOMBINER, search.getSummaryField("str_elem_map").getTransform()); + Schema schema = SearchBuilder.buildFromFile("src/test/derived/map_of_struct_attribute/test.sd"); + assertEquals(SummaryTransform.ATTRIBUTECOMBINER, schema.getSummaryField("str_elem_map").getTransform()); } @Test public void attribute_combiner_transform_is_not_set_when_map_of_struct_has_some_struct_field_attributes() throws IOException, ParseException { - Search search = SearchBuilder.buildFromFile("src/test/derived/map_of_struct_attribute/test.sd"); - assertEquals(SummaryTransform.NONE, search.getSummaryField("int_elem_map").getTransform()); + Schema schema = SearchBuilder.buildFromFile("src/test/derived/map_of_struct_attribute/test.sd"); + assertEquals(SummaryTransform.NONE, schema.getSummaryField("int_elem_map").getTransform()); } } diff --git a/config-model/src/test/java/com/yahoo/searchdefinition/processing/ImplicitSummaryFieldsTestCase.java b/config-model/src/test/java/com/yahoo/searchdefinition/processing/ImplicitSummaryFieldsTestCase.java index b7707f2a517..c9f941087ef 100644 --- a/config-model/src/test/java/com/yahoo/searchdefinition/processing/ImplicitSummaryFieldsTestCase.java +++ b/config-model/src/test/java/com/yahoo/searchdefinition/processing/ImplicitSummaryFieldsTestCase.java @@ -1,9 +1,9 @@ // Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. package com.yahoo.searchdefinition.processing; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.SearchBuilder; -import com.yahoo.searchdefinition.SchemaTestCase; +import com.yahoo.searchdefinition.AbstractSchemaTestCase; import com.yahoo.searchdefinition.parser.ParseException; import com.yahoo.vespa.documentmodel.DocumentSummary; import org.junit.Test; @@ -13,14 +13,14 @@ import java.io.IOException; import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertNotNull; -public class ImplicitSummaryFieldsTestCase extends SchemaTestCase { +public class ImplicitSummaryFieldsTestCase extends AbstractSchemaTestCase { @Test public void testRequireThatImplicitFieldsAreCreated() throws IOException, ParseException { - Search search = SearchBuilder.buildFromFile("src/test/examples/implicitsummaryfields.sd"); - assertNotNull(search); + Schema schema = SearchBuilder.buildFromFile("src/test/examples/implicitsummaryfields.sd"); + assertNotNull(schema); - DocumentSummary docsum = search.getSummary("default"); + DocumentSummary docsum = schema.getSummary("default"); assertNotNull(docsum); assertNotNull(docsum.getSummaryField("rankfeatures")); assertNotNull(docsum.getSummaryField("summaryfeatures")); diff --git a/config-model/src/test/java/com/yahoo/searchdefinition/processing/ImportedFieldsResolverTestCase.java b/config-model/src/test/java/com/yahoo/searchdefinition/processing/ImportedFieldsResolverTestCase.java index 3aeba7cd265..522d4dc2b15 100644 --- a/config-model/src/test/java/com/yahoo/searchdefinition/processing/ImportedFieldsResolverTestCase.java +++ b/config-model/src/test/java/com/yahoo/searchdefinition/processing/ImportedFieldsResolverTestCase.java @@ -3,7 +3,7 @@ package com.yahoo.searchdefinition.processing; import com.yahoo.document.DataType; import com.yahoo.document.TensorDataType; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.document.ImmutableImportedSDField; import com.yahoo.searchdefinition.document.ImmutableSDField; import com.yahoo.searchdefinition.document.ImportedField; @@ -36,9 +36,9 @@ public class ImportedFieldsResolverTestCase { ImportedField myField = model.importedFields.fields().get(fieldName); assertNotNull(myField); assertEquals(fieldName, myField.fieldName()); - assertSame(model.childSearch.getConcreteField("ref"), myField.reference().referenceField()); - assertSame(model.parentSearch, myField.reference().targetSearch()); - ImmutableSDField targetField = model.parentSearch.getField(targetFieldName); + assertSame(model.childSchema.getConcreteField("ref"), myField.reference().referenceField()); + assertSame(model.parentSchema, myField.reference().targetSearch()); + ImmutableSDField targetField = model.parentSchema.getField(targetFieldName); if (targetField instanceof SDField) { assertSame(targetField, myField.targetField()); } else { @@ -60,7 +60,7 @@ public class ImportedFieldsResolverTestCase { @Test public void resolver_fails_if_document_reference_is_not_found() { exceptionRule.expect(IllegalArgumentException.class); - exceptionRule.expectMessage("For search 'child', import field 'my_attribute_field': " + exceptionRule.expectMessage("For schema 'child', import field 'my_attribute_field': " + "Reference field 'not_ref' not found"); new SearchModel().addImportedField("my_attribute_field", "not_ref", "budget").resolve(); } @@ -68,7 +68,7 @@ public class ImportedFieldsResolverTestCase { @Test public void resolver_fails_if_referenced_field_is_not_found() { exceptionRule.expect(IllegalArgumentException.class); - exceptionRule.expectMessage("For search 'child', import field 'my_attribute_field': " + exceptionRule.expectMessage("For schema 'child', import field 'my_attribute_field': " + "Field 'not_existing' via reference field 'ref': Not found"); new SearchModel().addImportedField("my_attribute_field", "ref", "not_existing").resolve(); } @@ -76,7 +76,7 @@ public class ImportedFieldsResolverTestCase { @Test public void resolver_fails_if_imported_field_is_not_an_attribute() { exceptionRule.expect(IllegalArgumentException.class); - exceptionRule.expectMessage("For search 'child', import field 'my_not_attribute': " + exceptionRule.expectMessage("For schema 'child', import field 'my_not_attribute': " + "Field 'not_attribute' via reference field 'ref': Is not an attribute field. Only attribute fields supported"); new SearchModel().addImportedField("my_not_attribute", "ref", "not_attribute").resolve(); } @@ -85,7 +85,7 @@ public class ImportedFieldsResolverTestCase { public void resolver_fails_if_imported_field_is_indexing() { exceptionRule.expect(IllegalArgumentException.class); exceptionRule.expectMessage( - "For search 'child', import field 'my_attribute_and_index': " + + "For schema 'child', import field 'my_attribute_and_index': " + "Field 'attribute_and_index' via reference field 'ref': Is an index field. Not supported"); new SearchModel() .addImportedField("my_attribute_and_index", "ref", "attribute_and_index") @@ -96,55 +96,54 @@ public class ImportedFieldsResolverTestCase { public void resolver_fails_if_imported_field_is_of_type_predicate() { exceptionRule.expect(IllegalArgumentException.class); exceptionRule.expectMessage( - "For search 'child', import field 'my_predicate_field': " + + "For schema 'child', import field 'my_predicate_field': " + "Field 'predicate_field' via reference field 'ref': Is of type 'predicate'. Not supported"); new SearchModel().addImportedField("my_predicate_field", "ref", "predicate_field").resolve(); } static class SearchModel extends ParentChildSearchModel { - public final Search grandParentSearch; + public final Schema grandParentSchema; public ImportedFields importedFields; public SearchModel() { super(); - grandParentSearch = createSearch("grandparent"); - grandParentSearch.getDocument().addField(createField("ancient_field", DataType.INT, "{ attribute }")); - - parentSearch.getDocument().addField(createField("attribute_field", DataType.INT, "{ attribute }")); - parentSearch.getDocument().addField(createField("attribute_and_index", DataType.INT, "{ attribute | index }")); - parentSearch.getDocument().addField(new TemporarySDField("not_attribute", DataType.INT)); - parentSearch.getDocument().addField(createField("tensor_field", new TensorDataType(TensorType.fromSpec("tensor(x[5])")), "{ attribute }")); - parentSearch.getDocument().addField(createField("predicate_field", DataType.PREDICATE, "{ attribute }")); - addRefField(parentSearch, grandParentSearch, "ref"); - addImportedField(parentSearch, "ancient_field", "ref", "ancient_field"); - - addRefField(childSearch, parentSearch, "ref"); + grandParentSchema = createSearch("grandparent"); + grandParentSchema.getDocument().addField(createField("ancient_field", DataType.INT, "{ attribute }")); + + parentSchema.getDocument().addField(createField("attribute_field", DataType.INT, "{ attribute }")); + parentSchema.getDocument().addField(createField("attribute_and_index", DataType.INT, "{ attribute | index }")); + parentSchema.getDocument().addField(new TemporarySDField("not_attribute", DataType.INT)); + parentSchema.getDocument().addField(createField("tensor_field", new TensorDataType(TensorType.fromSpec("tensor(x[5])")), "{ attribute }")); + parentSchema.getDocument().addField(createField("predicate_field", DataType.PREDICATE, "{ attribute }")); + addRefField(parentSchema, grandParentSchema, "ref"); + addImportedField(parentSchema, "ancient_field", "ref", "ancient_field"); + + addRefField(childSchema, parentSchema, "ref"); } protected SearchModel addImportedField(String fieldName, String referenceFieldName, String targetFieldName) { - return addImportedField(childSearch, fieldName, referenceFieldName, targetFieldName); + return addImportedField(childSchema, fieldName, referenceFieldName, targetFieldName); } - protected SearchModel addImportedField(Search search, String fieldName, String referenceFieldName, String targetFieldName) { - super.addImportedField(search, fieldName, referenceFieldName, targetFieldName); + protected SearchModel addImportedField(Schema schema, String fieldName, String referenceFieldName, String targetFieldName) { + super.addImportedField(schema, fieldName, referenceFieldName, targetFieldName); return this; } public void resolve() { - resolve(grandParentSearch); - resolve(parentSearch); - importedFields = resolve(childSearch); + resolve(grandParentSchema); + resolve(parentSchema); + importedFields = resolve(childSchema); } - private static ImportedFields resolve(Search search) { - assertNotNull(search.temporaryImportedFields().get()); - assertFalse(search.importedFields().isPresent()); - new ImportedFieldsResolver(search, null, null, null).process(true, false); - assertFalse(search.temporaryImportedFields().isPresent()); - assertNotNull(search.importedFields().get()); - return search.importedFields().get(); + private static ImportedFields resolve(Schema schema) { + assertNotNull(schema.temporaryImportedFields().get()); + assertFalse(schema.importedFields().isPresent()); + new ImportedFieldsResolver(schema, null, null, null).process(true, false); + assertNotNull(schema.importedFields().get()); + return schema.importedFields().get(); } } diff --git a/config-model/src/test/java/com/yahoo/searchdefinition/processing/ImportedFieldsTestCase.java b/config-model/src/test/java/com/yahoo/searchdefinition/processing/ImportedFieldsTestCase.java index 65010d96ec0..37b7ab0955c 100644 --- a/config-model/src/test/java/com/yahoo/searchdefinition/processing/ImportedFieldsTestCase.java +++ b/config-model/src/test/java/com/yahoo/searchdefinition/processing/ImportedFieldsTestCase.java @@ -1,7 +1,7 @@ // Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. package com.yahoo.searchdefinition.processing; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.SearchBuilder; import com.yahoo.searchdefinition.derived.AttributeFields; import com.yahoo.searchdefinition.document.ImportedComplexField; @@ -24,7 +24,7 @@ public class ImportedFieldsTestCase { @Test public void fields_can_be_imported_from_referenced_document_types() throws ParseException { - Search search = buildAdSearch(joinLines( + Schema schema = buildAdSearch(joinLines( "search ad {", " document ad {", " field campaign_ref type reference<campaign> { indexing: attribute }", @@ -33,9 +33,9 @@ public class ImportedFieldsTestCase { " import field campaign_ref.budget as my_budget {}", " import field person_ref.name as my_name {}", "}")); - assertEquals(2, search.importedFields().get().fields().size()); - assertSearchContainsImportedField("my_budget", "campaign_ref", "campaign", "budget", search); - assertSearchContainsImportedField("my_name", "person_ref", "person", "name", search); + assertEquals(2, schema.importedFields().get().fields().size()); + assertSearchContainsImportedField("my_budget", "campaign_ref", "campaign", "budget", schema); + assertSearchContainsImportedField("my_name", "person_ref", "person", "name", schema); } @Rule @@ -55,9 +55,9 @@ public class ImportedFieldsTestCase { @Test public void fail_duplicate_import() throws ParseException { exception.expect(IllegalArgumentException.class); - exception.expectMessage("For search 'ad', import field as 'my_budget': Field already imported"); - Search search = buildAdSearch(joinLines( - "search ad {", + exception.expectMessage("For schema 'ad', import field as 'my_budget': Field already imported"); + Schema schema = buildAdSearch(joinLines( + "schema ad {", " document ad {", " field campaign_ref type reference<campaign> { indexing: attribute }", " }", @@ -66,14 +66,16 @@ public class ImportedFieldsTestCase { "}")); } - private static Search buildAdSearch(String sdContent) throws ParseException { + private static Schema buildAdSearch(String sdContent) throws ParseException { SearchBuilder builder = new SearchBuilder(); - builder.importString(joinLines("search campaign {", + builder.importString(joinLines( + "schema campaign {", " document campaign {", " field budget type int { indexing: attribute }", " }", "}")); - builder.importString(joinLines("search person {", + builder.importString(joinLines( + "schema person {", " document person {", " field name type string { indexing: attribute }", " }", @@ -84,29 +86,29 @@ public class ImportedFieldsTestCase { } private static void checkStructImport(AncestorStructSdBuilder parentBuilder) throws ParseException { - Search search = buildChildSearch(parentBuilder.build(), new ChildStructSdBuilder().build()); - checkImportedStructFields(search, parentBuilder); + Schema schema = buildChildSearch(parentBuilder.build(), new ChildStructSdBuilder().build()); + checkImportedStructFields(schema, parentBuilder); } private static void checkNestedStructImport(AncestorStructSdBuilder grandParentBuilder) throws ParseException { - Search search = buildChildSearch(grandParentBuilder.build(), - new IntermediateParentStructSdBuilder().build(), - new ChildStructSdBuilder().build()); - checkImportedStructFields(search, grandParentBuilder); - } - - private static void checkImportedStructFields(Search search, AncestorStructSdBuilder ancestorBuilder) { - assertEquals(3, search.importedFields().get().fields().size()); - checkImportedField("my_elem_array.name", "parent_ref", "parent", "elem_array.name", search, ancestorBuilder.elem_array_name_attr); - checkImportedField("my_elem_array.weight", "parent_ref", "parent", "elem_array.weight", search, ancestorBuilder.elem_array_weight_attr); - checkImportedField("my_elem_map.key", "parent_ref", "parent", "elem_map.key", search, ancestorBuilder.elem_map_key_attr); - checkImportedField("my_elem_map.value.name", "parent_ref", "parent", "elem_map.value.name", search, ancestorBuilder.elem_map_value_name_attr); - checkImportedField("my_elem_map.value.weight", "parent_ref", "parent", "elem_map.value.weight", search, ancestorBuilder.elem_map_value_weight_attr); - checkImportedField("my_str_int_map.key", "parent_ref", "parent", "str_int_map.key", search, ancestorBuilder.str_int_map_key_attr); - checkImportedField("my_str_int_map.value", "parent_ref", "parent", "str_int_map.value", search, ancestorBuilder.str_int_map_value_attr); - checkImportedField("my_elem_array", "parent_ref", "parent", "elem_array", search, true); - checkImportedField("my_elem_map", "parent_ref", "parent", "elem_map", search, true); - checkImportedField("my_str_int_map", "parent_ref", "parent", "str_int_map", search, true); + Schema schema = buildChildSearch(grandParentBuilder.build(), + new IntermediateParentStructSdBuilder().build(), + new ChildStructSdBuilder().build()); + checkImportedStructFields(schema, grandParentBuilder); + } + + private static void checkImportedStructFields(Schema schema, AncestorStructSdBuilder ancestorBuilder) { + assertEquals(3, schema.importedFields().get().fields().size()); + checkImportedField("my_elem_array.name", "parent_ref", "parent", "elem_array.name", schema, ancestorBuilder.elem_array_name_attr); + checkImportedField("my_elem_array.weight", "parent_ref", "parent", "elem_array.weight", schema, ancestorBuilder.elem_array_weight_attr); + checkImportedField("my_elem_map.key", "parent_ref", "parent", "elem_map.key", schema, ancestorBuilder.elem_map_key_attr); + checkImportedField("my_elem_map.value.name", "parent_ref", "parent", "elem_map.value.name", schema, ancestorBuilder.elem_map_value_name_attr); + checkImportedField("my_elem_map.value.weight", "parent_ref", "parent", "elem_map.value.weight", schema, ancestorBuilder.elem_map_value_weight_attr); + checkImportedField("my_str_int_map.key", "parent_ref", "parent", "str_int_map.key", schema, ancestorBuilder.str_int_map_key_attr); + checkImportedField("my_str_int_map.value", "parent_ref", "parent", "str_int_map.value", schema, ancestorBuilder.str_int_map_value_attr); + checkImportedField("my_elem_array", "parent_ref", "parent", "elem_array", schema, true); + checkImportedField("my_elem_map", "parent_ref", "parent", "elem_map", schema, true); + checkImportedField("my_str_int_map", "parent_ref", "parent", "str_int_map", schema, true); } @Test @@ -126,35 +128,35 @@ public class ImportedFieldsTestCase { @Test public void check_illegal_struct_import_missing_array_of_struct_attributes() throws ParseException { exception.expect(IllegalArgumentException.class); - exception.expectMessage("For search 'child', import field 'my_elem_array': Field 'elem_array' via reference field 'parent_ref': Is not a struct containing an attribute field."); + exception.expectMessage("For schema 'child', import field 'my_elem_array': Field 'elem_array' via reference field 'parent_ref': Is not a struct containing an attribute field."); checkStructImport(new ParentStructSdBuilder().elem_array_name_attr(false).elem_array_weight_attr(false)); } @Test public void check_illegal_struct_import_missing_map_of_struct_key_attribute() throws ParseException { exception.expect(IllegalArgumentException.class); - exception.expectMessage("For search 'child', import field 'my_elem_map' (nested to 'my_elem_map.key'): Field 'elem_map.key' via reference field 'parent_ref': Is not an attribute field. Only attribute fields supported"); + exception.expectMessage("For schema 'child', import field 'my_elem_map' (nested to 'my_elem_map.key'): Field 'elem_map.key' via reference field 'parent_ref': Is not an attribute field. Only attribute fields supported"); checkStructImport(new ParentStructSdBuilder().elem_map_key_attr(false)); } @Test public void check_illegal_struct_import_missing_map_of_struct_value_attributes() throws ParseException { exception.expect(IllegalArgumentException.class); - exception.expectMessage("For search 'child', import field 'my_elem_map' (nested to 'my_elem_map.value'): Field 'elem_map.value' via reference field 'parent_ref': Is not a struct containing an attribute field."); + exception.expectMessage("For schema 'child', import field 'my_elem_map' (nested to 'my_elem_map.value'): Field 'elem_map.value' via reference field 'parent_ref': Is not a struct containing an attribute field."); checkStructImport(new ParentStructSdBuilder().elem_map_value_name_attr(false).elem_map_value_weight_attr(false)); } @Test public void check_illegal_struct_import_missing_map_of_primitive_key_attribute() throws ParseException { exception.expect(IllegalArgumentException.class); - exception.expectMessage("For search 'child', import field 'my_str_int_map' (nested to 'my_str_int_map.key'): Field 'str_int_map.key' via reference field 'parent_ref': Is not an attribute field. Only attribute fields supported"); + exception.expectMessage("For schema 'child', import field 'my_str_int_map' (nested to 'my_str_int_map.key'): Field 'str_int_map.key' via reference field 'parent_ref': Is not an attribute field. Only attribute fields supported"); checkStructImport(new ParentStructSdBuilder().str_int_map_key_attr(false)); } @Test public void check_illegal_struct_import_missing_map_of_primitive_value_attribute() throws ParseException { exception.expect(IllegalArgumentException.class); - exception.expectMessage("For search 'child', import field 'my_str_int_map' (nested to 'my_str_int_map.value'): Field 'str_int_map.value' via reference field 'parent_ref': Is not an attribute field. Only attribute fields supported"); + exception.expectMessage("For schema 'child', import field 'my_str_int_map' (nested to 'my_str_int_map.value'): Field 'str_int_map.value' via reference field 'parent_ref': Is not an attribute field. Only attribute fields supported"); checkStructImport(new ParentStructSdBuilder().str_int_map_value_attr(false)); } @@ -308,7 +310,7 @@ public class ImportedFieldsTestCase { } } - private static Search buildChildSearch(String parentSdContent, String sdContent) throws ParseException { + private static Schema buildChildSearch(String parentSdContent, String sdContent) throws ParseException { SearchBuilder builder = new SearchBuilder(); builder.importString(parentSdContent); builder.importString(sdContent); @@ -316,7 +318,7 @@ public class ImportedFieldsTestCase { return builder.getSearch("child"); } - private static Search buildChildSearch(String grandParentSdContent, String parentSdContent, String sdContent) throws ParseException { + private static Schema buildChildSearch(String grandParentSdContent, String parentSdContent, String sdContent) throws ParseException { SearchBuilder builder = new SearchBuilder(); builder.importString(grandParentSdContent); builder.importString(parentSdContent); @@ -389,19 +391,19 @@ public class ImportedFieldsTestCase { } private static void checkPosImport(ParentPosSdBuilder parentBuilder, DescendantPosSdBuilder childBuilder) throws ParseException { - Search search = buildChildSearch(parentBuilder.build(), childBuilder.build()); - checkImportedPosFields(search); + Schema schema = buildChildSearch(parentBuilder.build(), childBuilder.build()); + checkImportedPosFields(schema); } private static void checkNestedPosImport(GrandParentPosSdBuilder grandParentBuilder, DescendantPosSdBuilder childBuilder) throws ParseException { - Search search = buildChildSearch(grandParentBuilder.build(), new IntermediateParentPosSdBuilder().build(), childBuilder.build()); - checkImportedPosFields(search); + Schema schema = buildChildSearch(grandParentBuilder.build(), new IntermediateParentPosSdBuilder().build(), childBuilder.build()); + checkImportedPosFields(schema); } - private static void checkImportedPosFields(Search search) { - assertEquals(2, search.importedFields().get().fields().size()); - assertSearchContainsImportedField("my_pos_zcurve", "parent_ref", "parent", "pos_zcurve", search); - assertSearchContainsImportedField("my_pos", "parent_ref", "parent", "pos", search); + private static void checkImportedPosFields(Schema schema) { + assertEquals(2, schema.importedFields().get().fields().size()); + assertSearchContainsImportedField("my_pos_zcurve", "parent_ref", "parent", "pos_zcurve", schema); + assertSearchContainsImportedField("my_pos", "parent_ref", "parent", "pos", schema); } @Test @@ -417,26 +419,26 @@ public class ImportedFieldsTestCase { @Test public void check_pos_import_after_pos_zcurve_import() throws ParseException { exception.expect(IllegalArgumentException.class); - exception.expectMessage("For search 'child', import field 'my_pos_zcurve': Field 'pos_zcurve' via reference field 'parent_ref': Field already imported"); + exception.expectMessage("For schema 'child', import field 'my_pos_zcurve': Field 'pos_zcurve' via reference field 'parent_ref': Field already imported"); checkPosImport(new ParentPosSdBuilder(), new ChildPosSdBuilder().import_pos_zcurve_before(true)); } - private static ImportedField getImportedField(String name, Search search) { + private static ImportedField getImportedField(String name, Schema schema) { if (name.contains(".")) { - assertNull(search.importedFields().get().fields().get(name)); + assertNull(schema.importedFields().get().fields().get(name)); String superFieldName = name.substring(0,name.indexOf(".")); String subFieldName = name.substring(name.indexOf(".")+1); - ImportedField superField = search.importedFields().get().fields().get(superFieldName); + ImportedField superField = schema.importedFields().get().fields().get(superFieldName); if (superField != null && superField instanceof ImportedComplexField) { return ((ImportedComplexField)superField).getNestedField(subFieldName); } return null; } - return search.importedFields().get().fields().get(name); + return schema.importedFields().get().fields().get(name); } - private static void assertSearchNotContainsImportedField(String fieldName, Search search) { - ImportedField importedField = getImportedField(fieldName, search); + private static void assertSearchNotContainsImportedField(String fieldName, Schema schema) { + ImportedField importedField = getImportedField(fieldName, schema); assertNull(importedField); } @@ -444,8 +446,8 @@ public class ImportedFieldsTestCase { String referenceFieldName, String referenceDocType, String targetFieldName, - Search search) { - ImportedField importedField = getImportedField(fieldName, search); + Schema schema) { + ImportedField importedField = getImportedField(fieldName, schema); assertNotNull(importedField); assertEquals(fieldName, importedField.fieldName()); assertEquals(referenceFieldName, importedField.reference().referenceField().getName()); @@ -454,11 +456,11 @@ public class ImportedFieldsTestCase { } private static void checkImportedField(String fieldName, String referenceFieldName, String referenceDocType, - String targetFieldName, Search search, boolean present) { + String targetFieldName, Schema schema, boolean present) { if (present) { - assertSearchContainsImportedField(fieldName, referenceFieldName, referenceDocType, targetFieldName, search); + assertSearchContainsImportedField(fieldName, referenceFieldName, referenceDocType, targetFieldName, schema); } else { - assertSearchNotContainsImportedField(fieldName, search); + assertSearchNotContainsImportedField(fieldName, schema); } } @@ -479,7 +481,7 @@ public class ImportedFieldsTestCase { checkImportedField("entries_from_b.value", "ref_parent_b", "parent_b", "entries.value", child, true); } - private void assertParentContainsEntriesAttributes(Search parent) { + private void assertParentContainsEntriesAttributes(Schema parent) { var attrs = new AttributeFields(parent); assertTrue(attrs.containsAttribute("entries.key")); assertTrue(attrs.containsAttribute("entries.value")); diff --git a/config-model/src/test/java/com/yahoo/searchdefinition/processing/IndexingInputsTestCase.java b/config-model/src/test/java/com/yahoo/searchdefinition/processing/IndexingInputsTestCase.java index bb935fb8a90..725b1b17ff0 100644 --- a/config-model/src/test/java/com/yahoo/searchdefinition/processing/IndexingInputsTestCase.java +++ b/config-model/src/test/java/com/yahoo/searchdefinition/processing/IndexingInputsTestCase.java @@ -16,7 +16,7 @@ public class IndexingInputsTestCase { @Test public void requireThatExtraFieldInputExtraFieldThrows() throws IOException, ParseException { assertBuildFails("src/test/examples/indexing_extra_field_input_extra_field.sd", - "For search 'indexing_extra_field_input_extra_field', field 'bar': Indexing script refers " + + "For schema 'indexing_extra_field_input_extra_field', field 'bar': Indexing script refers " + "to field 'bar' which does not exist in document type " + "'indexing_extra_field_input_extra_field', and is not a mutable attribute."); } @@ -24,21 +24,22 @@ public class IndexingInputsTestCase { @Test public void requireThatExtraFieldInputImplicitThrows() throws IOException, ParseException { assertBuildFails("src/test/examples/indexing_extra_field_input_implicit.sd", - "For search 'indexing_extra_field_input_implicit', field 'foo': Indexing script refers to " + + "For schema 'indexing_extra_field_input_implicit', field 'foo': Indexing script refers to " + "field 'foo' which does not exist in document type 'indexing_extra_field_input_implicit', and is not a mutable attribute."); } @Test public void requireThatExtraFieldInputNullThrows() throws IOException, ParseException { assertBuildFails("src/test/examples/indexing_extra_field_input_null.sd", - "For search 'indexing_extra_field_input_null', field 'foo': Indexing script refers to field " + + "For schema 'indexing_extra_field_input_null', field 'foo': Indexing script refers to field " + "'foo' which does not exist in document type 'indexing_extra_field_input_null', and is not a mutable attribute."); } @Test public void requireThatExtraFieldInputSelfThrows() throws IOException, ParseException { assertBuildFails("src/test/examples/indexing_extra_field_input_self.sd", - "For search 'indexing_extra_field_input_self', field 'foo': Indexing script refers to field " + + "For schema 'indexing_extra_field_input_self', field 'foo': Indexing script refers to field " + "'foo' which does not exist in document type 'indexing_extra_field_input_self', and is not a mutable attribute."); } + } diff --git a/config-model/src/test/java/com/yahoo/searchdefinition/processing/IndexingOutputsTestCase.java b/config-model/src/test/java/com/yahoo/searchdefinition/processing/IndexingOutputsTestCase.java index 4d765127276..5ea4b37991a 100644 --- a/config-model/src/test/java/com/yahoo/searchdefinition/processing/IndexingOutputsTestCase.java +++ b/config-model/src/test/java/com/yahoo/searchdefinition/processing/IndexingOutputsTestCase.java @@ -17,14 +17,14 @@ public class IndexingOutputsTestCase { @Test public void requireThatOutputOtherFieldThrows() throws IOException, ParseException { assertBuildFails("src/test/examples/indexing_output_other_field.sd", - "For search 'indexing_output_other_field', field 'foo': Indexing expression 'index bar' " + + "For schema 'indexing_output_other_field', field 'foo': Indexing expression 'index bar' " + "attempts to write to a field other than 'foo'."); } @Test public void requireThatOutputConflictThrows() throws IOException, ParseException { assertBuildFails("src/test/examples/indexing_output_conflict.sd", - "For search 'indexing_output_confict', field 'bar': For expression 'index bar': Attempting " + + "For schema 'indexing_output_confict', field 'bar': For expression 'index bar': Attempting " + "to assign conflicting values to field 'bar'."); } } diff --git a/config-model/src/test/java/com/yahoo/searchdefinition/processing/IndexingScriptRewriterTestCase.java b/config-model/src/test/java/com/yahoo/searchdefinition/processing/IndexingScriptRewriterTestCase.java index 13581a1e9e6..9cbecddfffa 100644 --- a/config-model/src/test/java/com/yahoo/searchdefinition/processing/IndexingScriptRewriterTestCase.java +++ b/config-model/src/test/java/com/yahoo/searchdefinition/processing/IndexingScriptRewriterTestCase.java @@ -5,9 +5,9 @@ import com.yahoo.config.model.application.provider.BaseDeployLogger; import com.yahoo.document.DataType; import com.yahoo.searchdefinition.Index; import com.yahoo.searchdefinition.RankProfileRegistry; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.SearchBuilder; -import com.yahoo.searchdefinition.SchemaTestCase; +import com.yahoo.searchdefinition.AbstractSchemaTestCase; import com.yahoo.searchdefinition.document.BooleanIndexDefinition; import com.yahoo.searchdefinition.document.SDDocumentType; import com.yahoo.searchdefinition.document.SDField; @@ -21,6 +21,7 @@ import java.util.Arrays; import java.util.OptionalDouble; import java.util.OptionalInt; import java.util.OptionalLong; +import java.util.Set; import static com.yahoo.searchdefinition.processing.AssertIndexingScript.assertIndexing; import static org.junit.Assert.assertEquals; @@ -28,7 +29,7 @@ import static org.junit.Assert.assertEquals; /** * @author Simon Thoresen Hult */ -public class IndexingScriptRewriterTestCase extends SchemaTestCase { +public class IndexingScriptRewriterTestCase extends AbstractSchemaTestCase { @Test public void testSetLanguageRewriting() { @@ -153,9 +154,10 @@ public class IndexingScriptRewriterTestCase extends SchemaTestCase { private static ScriptExpression processField(SDField unprocessedField) { SDDocumentType sdoc = new SDDocumentType("test"); sdoc.addField(unprocessedField); - Search search = new Search("test"); - search.addDocument(sdoc); - new Processing().process(search, new BaseDeployLogger(), new RankProfileRegistry(), new QueryProfiles(), true, false); + Schema schema = new Schema("test"); + schema.addDocument(sdoc); + new Processing().process(schema, new BaseDeployLogger(), new RankProfileRegistry(), + new QueryProfiles(), true, false, Set.of()); return unprocessedField.getIndexingScript(); } diff --git a/config-model/src/test/java/com/yahoo/searchdefinition/processing/IndexingValidationTestCase.java b/config-model/src/test/java/com/yahoo/searchdefinition/processing/IndexingValidationTestCase.java index 15f2b179470..9f8e810f1f8 100644 --- a/config-model/src/test/java/com/yahoo/searchdefinition/processing/IndexingValidationTestCase.java +++ b/config-model/src/test/java/com/yahoo/searchdefinition/processing/IndexingValidationTestCase.java @@ -20,42 +20,42 @@ public class IndexingValidationTestCase extends AbstractExportingTestCase { @Test public void testAttributeChanged() throws IOException, ParseException { assertBuildFails("src/test/examples/indexing_attribute_changed.sd", - "For search 'indexing_attribute_changed', field 'foo': For expression 'attribute foo': " + + "For schema 'indexing_attribute_changed', field 'foo': For expression 'attribute foo': " + "Attempting to assign conflicting values to field 'foo'."); } @Test public void testAttributeOther() throws IOException, ParseException { assertBuildFails("src/test/examples/indexing_attribute_other.sd", - "For search 'indexing_attribute_other', field 'foo': Indexing expression 'attribute bar' " + + "For schema 'indexing_attribute_other', field 'foo': Indexing expression 'attribute bar' " + "attempts to write to a field other than 'foo'."); } @Test public void testIndexChanged() throws IOException, ParseException { assertBuildFails("src/test/examples/indexing_index_changed.sd", - "For search 'indexing_index_changed', field 'foo': For expression 'index foo': " + + "For schema 'indexing_index_changed', field 'foo': For expression 'index foo': " + "Attempting to assign conflicting values to field 'foo'."); } @Test public void testIndexOther() throws IOException, ParseException { assertBuildFails("src/test/examples/indexing_index_other.sd", - "For search 'indexing_index_other', field 'foo': Indexing expression 'index bar' " + + "For schema 'indexing_index_other', field 'foo': Indexing expression 'index bar' " + "attempts to write to a field other than 'foo'."); } @Test public void testSummaryChanged() throws IOException, ParseException { assertBuildFails("src/test/examples/indexing_summary_changed.sd", - "For search 'indexing_summary_fail', field 'foo': For expression 'summary foo': Attempting " + + "For schema 'indexing_summary_fail', field 'foo': For expression 'summary foo': Attempting " + "to assign conflicting values to field 'foo'."); } @Test public void testSummaryOther() throws IOException, ParseException { assertBuildFails("src/test/examples/indexing_summary_other.sd", - "For search 'indexing_summary_other', field 'foo': Indexing expression 'summary bar' " + + "For schema 'indexing_summary_other', field 'foo': Indexing expression 'summary bar' " + "attempts to write to a field other than 'foo'."); } @@ -70,7 +70,7 @@ public class IndexingValidationTestCase extends AbstractExportingTestCase { @Test public void requireThatMultilineOutputConflictThrows() throws IOException, ParseException { assertBuildFails("src/test/examples/indexing_multiline_output_conflict.sd", - "For search 'indexing_multiline_output_confict', field 'cox': For expression 'index cox': " + + "For schema 'indexing_multiline_output_confict', field 'cox': For expression 'index cox': " + "Attempting to assign conflicting values to field 'cox'."); } } diff --git a/config-model/src/test/java/com/yahoo/searchdefinition/processing/IndexingValuesTestCase.java b/config-model/src/test/java/com/yahoo/searchdefinition/processing/IndexingValuesTestCase.java index 64089f6f5fa..25d966b1324 100644 --- a/config-model/src/test/java/com/yahoo/searchdefinition/processing/IndexingValuesTestCase.java +++ b/config-model/src/test/java/com/yahoo/searchdefinition/processing/IndexingValuesTestCase.java @@ -22,7 +22,7 @@ public class IndexingValuesTestCase { @Test public void requireThatInputOtherFieldThrows() throws IOException, ParseException { assertBuildFails("src/test/examples/indexing_input_other_field.sd", - "For search 'indexing_input_other_field', field 'bar': Indexing expression 'input foo' " + + "For schema 'indexing_input_other_field', field 'bar': Indexing expression 'input foo' " + "attempts to modify the value of the document field 'bar'. " + "Use a field outside the document block instead."); } diff --git a/config-model/src/test/java/com/yahoo/searchdefinition/processing/IntegerIndex2AttributeTestCase.java b/config-model/src/test/java/com/yahoo/searchdefinition/processing/IntegerIndex2AttributeTestCase.java index 91f719c4eb8..cd5dd451026 100644 --- a/config-model/src/test/java/com/yahoo/searchdefinition/processing/IntegerIndex2AttributeTestCase.java +++ b/config-model/src/test/java/com/yahoo/searchdefinition/processing/IntegerIndex2AttributeTestCase.java @@ -3,9 +3,9 @@ package com.yahoo.searchdefinition.processing; import com.yahoo.config.model.application.provider.BaseDeployLogger; import com.yahoo.searchdefinition.RankProfileRegistry; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.SearchBuilder; -import com.yahoo.searchdefinition.SchemaTestCase; +import com.yahoo.searchdefinition.AbstractSchemaTestCase; import com.yahoo.searchdefinition.document.SDField; import com.yahoo.searchdefinition.parser.ParseException; import com.yahoo.vespa.model.container.search.QueryProfiles; @@ -20,40 +20,40 @@ import static org.junit.Assert.assertTrue; /** * @author baldersheim */ -public class IntegerIndex2AttributeTestCase extends SchemaTestCase { +public class IntegerIndex2AttributeTestCase extends AbstractSchemaTestCase { @Test public void testIntegerIndex2Attribute() throws IOException, ParseException { - Search search = SearchBuilder.buildFromFile("src/test/examples/integerindex2attribute.sd"); - new IntegerIndex2Attribute(search, new BaseDeployLogger(), new RankProfileRegistry(), new QueryProfiles()).process(true, false); + Schema schema = SearchBuilder.buildFromFile("src/test/examples/integerindex2attribute.sd"); + new IntegerIndex2Attribute(schema, new BaseDeployLogger(), new RankProfileRegistry(), new QueryProfiles()).process(true, false); SDField f; - f = search.getConcreteField("s1"); + f = schema.getConcreteField("s1"); assertTrue(f.getAttributes().isEmpty()); assertTrue(f.existsIndex("s1")); - f = search.getConcreteField("s2"); + f = schema.getConcreteField("s2"); assertEquals(f.getAttributes().size(), 1); assertTrue(f.existsIndex("s2")); - f = search.getConcreteField("as1"); + f = schema.getConcreteField("as1"); assertTrue(f.getAttributes().isEmpty()); assertTrue(f.existsIndex("as1")); - f = search.getConcreteField("as2"); + f = schema.getConcreteField("as2"); assertEquals(f.getAttributes().size(), 1); assertTrue(f.existsIndex("as2")); - f = search.getConcreteField("i1"); + f = schema.getConcreteField("i1"); assertEquals(f.getAttributes().size(), 1); assertFalse(f.existsIndex("i1")); - f = search.getConcreteField("i2"); + f = schema.getConcreteField("i2"); assertEquals(f.getAttributes().size(), 1); assertFalse(f.existsIndex("i2")); - f = search.getConcreteField("ai1"); - assertEquals(search.getConcreteField("ai1").getAttributes().size(), 1); - assertFalse(search.getConcreteField("ai1").existsIndex("ai1")); - f = search.getConcreteField("ai2"); + f = schema.getConcreteField("ai1"); + assertEquals(schema.getConcreteField("ai1").getAttributes().size(), 1); + assertFalse(schema.getConcreteField("ai1").existsIndex("ai1")); + f = schema.getConcreteField("ai2"); assertEquals(f.getAttributes().size(), 1); assertFalse(f.existsIndex("ai2")); } diff --git a/config-model/src/test/java/com/yahoo/searchdefinition/processing/MatchedElementsOnlyResolverTestCase.java b/config-model/src/test/java/com/yahoo/searchdefinition/processing/MatchedElementsOnlyResolverTestCase.java index ab98706fd44..fc44d61541e 100644 --- a/config-model/src/test/java/com/yahoo/searchdefinition/processing/MatchedElementsOnlyResolverTestCase.java +++ b/config-model/src/test/java/com/yahoo/searchdefinition/processing/MatchedElementsOnlyResolverTestCase.java @@ -2,7 +2,7 @@ package com.yahoo.searchdefinition.processing; import com.yahoo.searchdefinition.RankProfileRegistry; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.SearchBuilder; import com.yahoo.searchdefinition.parser.ParseException; import com.yahoo.vespa.documentmodel.SummaryField; @@ -134,7 +134,8 @@ public class MatchedElementsOnlyResolverTestCase { " }", "}"); - var search = buildSearch(joinLines("field my_field type array<string> {", + var search = buildSearch(joinLines( + "field my_field type array<string> {", " indexing: attribute | summary", "}"), documentSummary); @@ -147,7 +148,7 @@ public class MatchedElementsOnlyResolverTestCase { @Test public void unsupported_field_type_throws() throws ParseException { exceptionRule.expect(IllegalArgumentException.class); - exceptionRule.expectMessage("For search 'test', document summary 'default', summary field 'my_field': " + + exceptionRule.expectMessage("For schema 'test', document summary 'default', summary field 'my_field': " + "'matched-elements-only' is not supported for this field type. " + "Supported field types are: array of primitive, weighted set of primitive, " + "array of simple struct, map of primitive type to simple struct, " + @@ -168,11 +169,11 @@ public class MatchedElementsOnlyResolverTestCase { assertEquals(expSourceField, field.getSingleSource()); } - private Search buildSearch(String field) throws ParseException { + private Schema buildSearch(String field) throws ParseException { return buildSearch(field, ""); } - private Search buildSearch(String field, String summary) throws ParseException { + private Schema buildSearch(String field, String summary) throws ParseException { var builder = new SearchBuilder(new RankProfileRegistry()); builder.importString(joinLines("search test {", " document test {", diff --git a/config-model/src/test/java/com/yahoo/searchdefinition/processing/NGramTestCase.java b/config-model/src/test/java/com/yahoo/searchdefinition/processing/NGramTestCase.java index 8d8a19796ba..0e13aacb76f 100644 --- a/config-model/src/test/java/com/yahoo/searchdefinition/processing/NGramTestCase.java +++ b/config-model/src/test/java/com/yahoo/searchdefinition/processing/NGramTestCase.java @@ -1,9 +1,9 @@ // Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. package com.yahoo.searchdefinition.processing; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.SearchBuilder; -import com.yahoo.searchdefinition.SchemaTestCase; +import com.yahoo.searchdefinition.AbstractSchemaTestCase; import com.yahoo.searchdefinition.document.Matching; import com.yahoo.searchdefinition.document.SDField; import com.yahoo.searchdefinition.document.Stemming; @@ -21,22 +21,22 @@ import static org.junit.Assert.fail; /** * @author bratseth */ -public class NGramTestCase extends SchemaTestCase { +public class NGramTestCase extends AbstractSchemaTestCase { @Test public void testNGram() throws IOException, ParseException { - Search search = SearchBuilder.buildFromFile("src/test/examples/ngram.sd"); - assertNotNull(search); + Schema schema = SearchBuilder.buildFromFile("src/test/examples/ngram.sd"); + assertNotNull(schema); - SDField gram1 = search.getConcreteField("gram_1"); + SDField gram1 = schema.getConcreteField("gram_1"); assertEquals(Matching.Type.GRAM, gram1.getMatching().getType()); assertEquals(1, gram1.getMatching().getGramSize()); - SDField gram2 = search.getConcreteField("gram_2"); + SDField gram2 = schema.getConcreteField("gram_2"); assertEquals(Matching.Type.GRAM, gram2.getMatching().getType()); assertEquals(-1, gram2.getMatching().getGramSize()); // Not set explicitly - SDField gram3=search.getConcreteField("gram_3"); + SDField gram3= schema.getConcreteField("gram_3"); assertEquals(Matching.Type.GRAM,gram3.getMatching().getType()); assertEquals(3, gram3.getMatching().getGramSize()); @@ -55,7 +55,7 @@ public class NGramTestCase extends SchemaTestCase { @Test public void testInvalidNGramSetting1() throws IOException, ParseException { try { - Search search = SearchBuilder.buildFromFile("src/test/examples/invalidngram1.sd"); + Schema schema = SearchBuilder.buildFromFile("src/test/examples/invalidngram1.sd"); fail("Should cause an exception"); } catch (IllegalArgumentException e) { @@ -66,7 +66,7 @@ public class NGramTestCase extends SchemaTestCase { @Test public void testInvalidNGramSetting2() throws IOException, ParseException { try { - Search search = SearchBuilder.buildFromFile("src/test/examples/invalidngram2.sd"); + Schema schema = SearchBuilder.buildFromFile("src/test/examples/invalidngram2.sd"); fail("Should cause an exception"); } catch (IllegalArgumentException e) { @@ -77,7 +77,7 @@ public class NGramTestCase extends SchemaTestCase { @Test public void testInvalidNGramSetting3() throws IOException, ParseException { try { - Search search = SearchBuilder.buildFromFile("src/test/examples/invalidngram3.sd"); + Schema schema = SearchBuilder.buildFromFile("src/test/examples/invalidngram3.sd"); fail("Should cause an exception"); } catch (IllegalArgumentException e) { diff --git a/config-model/src/test/java/com/yahoo/searchdefinition/processing/PagedAttributeValidatorTestCase.java b/config-model/src/test/java/com/yahoo/searchdefinition/processing/PagedAttributeValidatorTestCase.java index 038bfc9ddbf..360072c1da7 100644 --- a/config-model/src/test/java/com/yahoo/searchdefinition/processing/PagedAttributeValidatorTestCase.java +++ b/config-model/src/test/java/com/yahoo/searchdefinition/processing/PagedAttributeValidatorTestCase.java @@ -31,13 +31,14 @@ public class PagedAttributeValidatorTestCase { createFromString(getSd(fieldType)); fail("Expected exception"); } catch (IllegalArgumentException e) { - assertEquals("For search 'test', field 'foo': The 'paged' attribute setting is only supported for dense tensor types", + assertEquals("For schema 'test', field 'foo': The 'paged' attribute setting is only supported for dense tensor types", e.getMessage()); } } private String getSd(String type) { - return joinLines("search test {", + return joinLines( + "schema test {", " document test {", " field foo type " + type + "{", " indexing: attribute", diff --git a/config-model/src/test/java/com/yahoo/searchdefinition/processing/ParentChildSearchModel.java b/config-model/src/test/java/com/yahoo/searchdefinition/processing/ParentChildSearchModel.java index 6366be84d8e..a01fdd8725d 100644 --- a/config-model/src/test/java/com/yahoo/searchdefinition/processing/ParentChildSearchModel.java +++ b/config-model/src/test/java/com/yahoo/searchdefinition/processing/ParentChildSearchModel.java @@ -2,16 +2,16 @@ package com.yahoo.searchdefinition.processing; import com.google.common.collect.ImmutableMap; -import com.yahoo.config.application.api.ApplicationPackage; import com.yahoo.config.model.application.provider.MockFileRegistry; import com.yahoo.config.model.deploy.TestProperties; import com.yahoo.config.model.test.MockApplicationPackage; import com.yahoo.document.DataType; import com.yahoo.document.ReferenceDataType; import com.yahoo.document.TemporaryStructuredDataType; +import com.yahoo.searchdefinition.Application; import com.yahoo.searchdefinition.DocumentReference; import com.yahoo.searchdefinition.DocumentReferences; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.derived.TestableDeployLogger; import com.yahoo.searchdefinition.document.SDDocumentType; import com.yahoo.searchdefinition.document.SDField; @@ -22,17 +22,18 @@ import com.yahoo.searchdefinition.document.TemporarySDField; * Fixture class used for ImportedFieldsResolverTestCase and AdjustPositionSummaryFieldsTestCase. */ public class ParentChildSearchModel { - private final ApplicationPackage app = MockApplicationPackage.createEmpty(); - public Search parentSearch; - public Search childSearch; + + private final Application application = new Application(MockApplicationPackage.createEmpty()); + public Schema parentSchema; + public Schema childSchema; ParentChildSearchModel() { - parentSearch = createSearch("parent"); - childSearch = createSearch("child"); + parentSchema = createSearch("parent"); + childSchema = createSearch("child"); } - protected Search createSearch(String name) { - Search result = new Search(name, app, new MockFileRegistry(), new TestableDeployLogger(), new TestProperties()); + protected Schema createSearch(String name) { + Schema result = new Schema(name, application, new MockFileRegistry(), new TestableDeployLogger(), new TestProperties()); result.addDocument(new SDDocumentType(name)); return result; } @@ -47,7 +48,7 @@ public class ParentChildSearchModel { return new TemporarySDField(fieldName, ReferenceDataType.createWithInferredId(TemporaryStructuredDataType.create(parentType))); } - protected static void addRefField(Search child, Search parent, String fieldName) { + protected static void addRefField(Schema child, Schema parent, String fieldName) { SDField refField = createRefField(parent.getName(), fieldName); child.getDocument().addField(refField); child.getDocument().setDocumentReferences(new DocumentReferences(ImmutableMap.of(refField.getName(), @@ -55,11 +56,11 @@ public class ParentChildSearchModel { } protected ParentChildSearchModel addImportedField(String fieldName, String referenceFieldName, String targetFieldName) { - return addImportedField(childSearch, fieldName, referenceFieldName, targetFieldName); + return addImportedField(childSchema, fieldName, referenceFieldName, targetFieldName); } - protected ParentChildSearchModel addImportedField(Search search, String fieldName, String referenceFieldName, String targetFieldName) { - search.temporaryImportedFields().get().add(new TemporaryImportedField(fieldName, referenceFieldName, targetFieldName)); + protected ParentChildSearchModel addImportedField(Schema schema, String fieldName, String referenceFieldName, String targetFieldName) { + schema.temporaryImportedFields().get().add(new TemporaryImportedField(fieldName, referenceFieldName, targetFieldName)); return this; } } diff --git a/config-model/src/test/java/com/yahoo/searchdefinition/processing/PositionTestCase.java b/config-model/src/test/java/com/yahoo/searchdefinition/processing/PositionTestCase.java index bcec2bfacfa..67e47b7f2ae 100644 --- a/config-model/src/test/java/com/yahoo/searchdefinition/processing/PositionTestCase.java +++ b/config-model/src/test/java/com/yahoo/searchdefinition/processing/PositionTestCase.java @@ -4,7 +4,7 @@ package com.yahoo.searchdefinition.processing; import com.yahoo.document.DataType; import com.yahoo.document.DocumentType; import com.yahoo.document.PositionDataType; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.SearchBuilder; import com.yahoo.searchdefinition.document.Attribute; import com.yahoo.searchdefinition.document.FieldSet; @@ -31,20 +31,20 @@ public class PositionTestCase { "src/test/examples/position_base.sd", "src/test/examples/position_inherited.sd")); - Search search = sb.getSearch("position_inherited"); - FieldSet fieldSet = search.getDocument().getFieldSets().builtInFieldSets().get(DocumentType.DOCUMENT); + Schema schema = sb.getSearch("position_inherited"); + FieldSet fieldSet = schema.getDocument().getFieldSets().builtInFieldSets().get(DocumentType.DOCUMENT); assertFalse(fieldSet.getFieldNames().contains(PositionDataType.getZCurveFieldName("pos"))); } @Test public void requireThatPositionCanBeAttribute() throws Exception { - Search search = SearchBuilder.buildFromFile("src/test/examples/position_attribute.sd"); - assertNull(search.getAttribute("pos")); - assertNull(search.getAttribute("pos.x")); - assertNull(search.getAttribute("pos.y")); + Schema schema = SearchBuilder.buildFromFile("src/test/examples/position_attribute.sd"); + assertNull(schema.getAttribute("pos")); + assertNull(schema.getAttribute("pos.x")); + assertNull(schema.getAttribute("pos.y")); - assertPositionAttribute(search, "pos", Attribute.CollectionType.SINGLE); - assertPositionSummary(search, "pos", false); + assertPositionAttribute(schema, "pos", Attribute.CollectionType.SINGLE); + assertPositionSummary(schema, "pos", false); } @Test @@ -53,20 +53,20 @@ public class PositionTestCase { SearchBuilder.buildFromFile("src/test/examples/position_index.sd"); fail(); } catch (IllegalArgumentException e) { - assertEquals("For search 'position_index', field 'pos': Indexing of data type 'position' is not " + + assertEquals("For schema 'position_index', field 'pos': Indexing of data type 'position' is not " + "supported, replace 'index' statement with 'attribute'.", e.getMessage()); } } @Test public void requireThatSummaryAloneDoesNotCreateZCurve() throws Exception { - Search search = SearchBuilder.buildFromFile("src/test/examples/position_summary.sd"); - assertNull(search.getAttribute("pos")); - assertNull(search.getAttribute("pos.x")); - assertNull(search.getAttribute("pos.y")); - assertNull(search.getAttribute("pos.zcurve")); + Schema schema = SearchBuilder.buildFromFile("src/test/examples/position_summary.sd"); + assertNull(schema.getAttribute("pos")); + assertNull(schema.getAttribute("pos.x")); + assertNull(schema.getAttribute("pos.y")); + assertNull(schema.getAttribute("pos.zcurve")); - SummaryField summary = search.getSummaryField("pos"); + SummaryField summary = schema.getSummaryField("pos"); assertNotNull(summary); assertEquals(2, summary.getSourceCount()); Iterator<SummaryField.Source> it = summary.getSources().iterator(); @@ -74,61 +74,61 @@ public class PositionTestCase { assertEquals("pos.y", it.next().getName()); assertEquals(SummaryTransform.NONE, summary.getTransform()); - assertNull(search.getSummaryField("pos_ext.distance")); + assertNull(schema.getSummaryField("pos_ext.distance")); } @Test public void requireThatExtraFieldCanBePositionAttribute() throws Exception { - Search search = SearchBuilder.buildFromFile("src/test/examples/position_extra.sd"); - assertNull(search.getAttribute("pos_ext")); - assertNull(search.getAttribute("pos_ext.x")); - assertNull(search.getAttribute("pos_ext.y")); + Schema schema = SearchBuilder.buildFromFile("src/test/examples/position_extra.sd"); + assertNull(schema.getAttribute("pos_ext")); + assertNull(schema.getAttribute("pos_ext.x")); + assertNull(schema.getAttribute("pos_ext.y")); - assertPositionAttribute(search, "pos_ext", Attribute.CollectionType.SINGLE); - assertPositionSummary(search, "pos_ext", false); + assertPositionAttribute(schema, "pos_ext", Attribute.CollectionType.SINGLE); + assertPositionSummary(schema, "pos_ext", false); } @Test public void requireThatPositionArrayIsSupported() throws Exception { - Search search = SearchBuilder.buildFromFile("src/test/examples/position_array.sd"); - assertNull(search.getAttribute("pos")); - assertNull(search.getAttribute("pos.x")); - assertNull(search.getAttribute("pos.y")); + Schema schema = SearchBuilder.buildFromFile("src/test/examples/position_array.sd"); + assertNull(schema.getAttribute("pos")); + assertNull(schema.getAttribute("pos.x")); + assertNull(schema.getAttribute("pos.y")); - assertPositionAttribute(search, "pos", Attribute.CollectionType.ARRAY); - assertPositionSummary(search, "pos", true); + assertPositionAttribute(schema, "pos", Attribute.CollectionType.ARRAY); + assertPositionSummary(schema, "pos", true); } - private static void assertPositionAttribute(Search search, String fieldName, Attribute.CollectionType type) { - Attribute attribute = search.getAttribute(PositionDataType.getZCurveFieldName(fieldName)); + private static void assertPositionAttribute(Schema schema, String fieldName, Attribute.CollectionType type) { + Attribute attribute = schema.getAttribute(PositionDataType.getZCurveFieldName(fieldName)); assertNotNull(attribute); assertTrue(attribute.isPosition()); assertEquals(attribute.getCollectionType(), type); assertEquals(attribute.getType(), Attribute.Type.LONG); } - private static void assertPositionSummary(Search search, String fieldName, boolean isArray) { - assertSummaryField(search, + private static void assertPositionSummary(Schema schema, String fieldName, boolean isArray) { + assertSummaryField(schema, fieldName, PositionDataType.getZCurveFieldName(fieldName), (isArray ? DataType.getArray(PositionDataType.INSTANCE) : PositionDataType.INSTANCE), SummaryTransform.GEOPOS); - assertSummaryField(search, + assertSummaryField(schema, PositionDataType.getDistanceSummaryFieldName(fieldName), PositionDataType.getZCurveFieldName(fieldName), DataType.INT, SummaryTransform.DISTANCE); - assertSummaryField(search, + assertSummaryField(schema, PositionDataType.getPositionSummaryFieldName(fieldName), PositionDataType.getZCurveFieldName(fieldName), DataType.getArray(DataType.STRING), SummaryTransform.POSITIONS); } - private static void assertSummaryField(Search search, String fieldName, String sourceName, DataType dataType, + private static void assertSummaryField(Schema schema, String fieldName, String sourceName, DataType dataType, SummaryTransform transform) { - SummaryField summary = search.getSummaryField(fieldName); + SummaryField summary = schema.getSummaryField(fieldName); assertNotNull(summary); assertEquals(1, summary.getSourceCount()); assertEquals(sourceName, summary.getSingleSource()); diff --git a/config-model/src/test/java/com/yahoo/searchdefinition/processing/RankModifierTestCase.java b/config-model/src/test/java/com/yahoo/searchdefinition/processing/RankModifierTestCase.java index dc7305f55c3..97eed506c85 100644 --- a/config-model/src/test/java/com/yahoo/searchdefinition/processing/RankModifierTestCase.java +++ b/config-model/src/test/java/com/yahoo/searchdefinition/processing/RankModifierTestCase.java @@ -1,9 +1,9 @@ // Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. package com.yahoo.searchdefinition.processing; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.SearchBuilder; -import com.yahoo.searchdefinition.SchemaTestCase; +import com.yahoo.searchdefinition.AbstractSchemaTestCase; import com.yahoo.searchdefinition.parser.ParseException; import org.junit.Test; @@ -14,9 +14,9 @@ import java.io.IOException; * @author vegardh * */ -public class RankModifierTestCase extends SchemaTestCase { +public class RankModifierTestCase extends AbstractSchemaTestCase { @Test public void testLiteral() throws IOException, ParseException { - Search search = SearchBuilder.buildFromFile("src/test/examples/rankmodifier/literal.sd"); + Schema schema = SearchBuilder.buildFromFile("src/test/examples/rankmodifier/literal.sd"); } } diff --git a/config-model/src/test/java/com/yahoo/searchdefinition/processing/RankProfileSearchFixture.java b/config-model/src/test/java/com/yahoo/searchdefinition/processing/RankProfileSearchFixture.java index 1b8160e0123..7ffb20859e9 100644 --- a/config-model/src/test/java/com/yahoo/searchdefinition/processing/RankProfileSearchFixture.java +++ b/config-model/src/test/java/com/yahoo/searchdefinition/processing/RankProfileSearchFixture.java @@ -12,7 +12,7 @@ import com.yahoo.path.Path; import com.yahoo.search.query.profile.QueryProfileRegistry; import com.yahoo.searchdefinition.RankProfile; import com.yahoo.searchdefinition.RankProfileRegistry; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.SearchBuilder; import com.yahoo.searchdefinition.parser.ParseException; import ai.vespa.rankingexpression.importer.configmodelview.ImportedMlModels; @@ -43,7 +43,7 @@ class RankProfileSearchFixture { new XGBoostImporter()); private final RankProfileRegistry rankProfileRegistry = new RankProfileRegistry(); private final QueryProfileRegistry queryProfileRegistry; - private final Search search; + private final Schema schema; private final Map<String, RankProfile> compiledRankProfiles = new HashMap<>(); private final ExecutorService executor = Executors.newFixedThreadPool(Runtime.getRuntime().availableProcessors()); @@ -79,7 +79,7 @@ class RankProfileSearchFixture { "}"; builder.importString(sdContent); builder.build(); - search = builder.getSearch(); + schema = builder.getSearch(); } public void assertFirstPhaseExpression(String expExpression, String rankProfile) { @@ -106,7 +106,7 @@ class RankProfileSearchFixture { } public RankProfile compileRankProfile(String rankProfile, Path applicationDir) { - RankProfile compiled = rankProfileRegistry.get(search, rankProfile) + RankProfile compiled = rankProfileRegistry.get(schema, rankProfile) .compile(queryProfileRegistry, new ImportedMlModels(applicationDir.toFile(), executor, importers)); compiledRankProfiles.put(rankProfile, compiled); @@ -115,7 +115,7 @@ class RankProfileSearchFixture { /** Returns the given uncompiled profile */ public RankProfile rankProfile(String rankProfile) { - return rankProfileRegistry.get(search, rankProfile); + return rankProfileRegistry.get(schema, rankProfile); } /** Returns the given compiled profile, or null if not compiled yet or not present at all */ @@ -123,6 +123,6 @@ class RankProfileSearchFixture { return compiledRankProfiles.get(rankProfile); } - public Search search() { return search; } + public Schema search() { return schema; } } diff --git a/config-model/src/test/java/com/yahoo/searchdefinition/processing/RankPropertyVariablesTestCase.java b/config-model/src/test/java/com/yahoo/searchdefinition/processing/RankPropertyVariablesTestCase.java index 407a5877976..5c53dbf0c57 100644 --- a/config-model/src/test/java/com/yahoo/searchdefinition/processing/RankPropertyVariablesTestCase.java +++ b/config-model/src/test/java/com/yahoo/searchdefinition/processing/RankPropertyVariablesTestCase.java @@ -5,9 +5,9 @@ import com.yahoo.config.model.application.provider.BaseDeployLogger; import com.yahoo.search.query.profile.QueryProfileRegistry; import com.yahoo.searchdefinition.RankProfile.RankProperty; import com.yahoo.searchdefinition.RankProfileRegistry; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.SearchBuilder; -import com.yahoo.searchdefinition.SchemaTestCase; +import com.yahoo.searchdefinition.AbstractSchemaTestCase; import com.yahoo.searchdefinition.parser.ParseException; import org.junit.Test; @@ -16,21 +16,21 @@ import java.util.List; import static org.junit.Assert.fail; -public class RankPropertyVariablesTestCase extends SchemaTestCase { +public class RankPropertyVariablesTestCase extends AbstractSchemaTestCase { @Test public void testRankPropVariables() throws IOException, ParseException { RankProfileRegistry rankProfileRegistry = new RankProfileRegistry(); - Search search = SearchBuilder.buildFromFile("src/test/examples/rankpropvars.sd", + Schema schema = SearchBuilder.buildFromFile("src/test/examples/rankpropvars.sd", new BaseDeployLogger(), rankProfileRegistry, new QueryProfileRegistry()); - assertRankPropEquals(rankProfileRegistry.get(search, "other").getRankProperties(), "$testvar1", "foo"); - assertRankPropEquals(rankProfileRegistry.get(search, "other").getRankProperties(), "$testvar_2", "bar"); - assertRankPropEquals(rankProfileRegistry.get(search, "other").getRankProperties(), "$testvarOne23", "baz"); - assertRankPropEquals(rankProfileRegistry.get(search, "another").getRankProperties(), "$Testvar1", "1"); - assertRankPropEquals(rankProfileRegistry.get(search, "another").getRankProperties(), "$Testvar_4", "4"); - assertRankPropEquals(rankProfileRegistry.get(search, "another").getRankProperties(), "$testvarFour23", "234234.234"); + assertRankPropEquals(rankProfileRegistry.get(schema, "other").getRankProperties(), "$testvar1", "foo"); + assertRankPropEquals(rankProfileRegistry.get(schema, "other").getRankProperties(), "$testvar_2", "bar"); + assertRankPropEquals(rankProfileRegistry.get(schema, "other").getRankProperties(), "$testvarOne23", "baz"); + assertRankPropEquals(rankProfileRegistry.get(schema, "another").getRankProperties(), "$Testvar1", "1"); + assertRankPropEquals(rankProfileRegistry.get(schema, "another").getRankProperties(), "$Testvar_4", "4"); + assertRankPropEquals(rankProfileRegistry.get(schema, "another").getRankProperties(), "$testvarFour23", "234234.234"); } private void assertRankPropEquals(List<RankProperty> props, String key, String val) { diff --git a/config-model/src/test/java/com/yahoo/searchdefinition/processing/RankingExpressionWithTransformerTokensTestCase.java b/config-model/src/test/java/com/yahoo/searchdefinition/processing/RankingExpressionWithTransformerTokensTestCase.java index dcd7e2426c3..e77787e22ca 100644 --- a/config-model/src/test/java/com/yahoo/searchdefinition/processing/RankingExpressionWithTransformerTokensTestCase.java +++ b/config-model/src/test/java/com/yahoo/searchdefinition/processing/RankingExpressionWithTransformerTokensTestCase.java @@ -8,7 +8,7 @@ import com.yahoo.config.model.test.MockApplicationPackage; import com.yahoo.search.query.profile.QueryProfileRegistry; import com.yahoo.searchdefinition.RankProfile; import com.yahoo.searchdefinition.RankProfileRegistry; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.SearchBuilder; import com.yahoo.searchdefinition.expressiontransforms.RankProfileTransformContext; import com.yahoo.searchdefinition.expressiontransforms.TokenTransformer; @@ -90,8 +90,8 @@ public class RankingExpressionWithTransformerTokensTestCase { SearchBuilder searchBuilder = new SearchBuilder(application, new MockFileRegistry(), new BaseDeployLogger(), new TestProperties(), rankProfileRegistry, queryProfileRegistry); searchBuilder.importString(sdContent); searchBuilder.build(); - Search search = searchBuilder.getSearch(); - RankProfile rp = rankProfileRegistry.get(search, "my_profile"); + Schema schema = searchBuilder.getSearch(); + RankProfile rp = rankProfileRegistry.get(schema, "my_profile"); return new RankProfileTransformContext(rp, queryProfileRegistry, Collections.emptyMap(), null, Collections.emptyMap(), Collections.emptyMap()); } diff --git a/config-model/src/test/java/com/yahoo/searchdefinition/processing/RankingExpressionsTestCase.java b/config-model/src/test/java/com/yahoo/searchdefinition/processing/RankingExpressionsTestCase.java index 27ea680d97a..fc5a2b6a39a 100644 --- a/config-model/src/test/java/com/yahoo/searchdefinition/processing/RankingExpressionsTestCase.java +++ b/config-model/src/test/java/com/yahoo/searchdefinition/processing/RankingExpressionsTestCase.java @@ -9,8 +9,8 @@ import com.yahoo.search.query.profile.QueryProfileRegistry; import com.yahoo.searchdefinition.LargeRankExpressions; import com.yahoo.searchdefinition.RankProfile; import com.yahoo.searchdefinition.RankProfileRegistry; -import com.yahoo.searchdefinition.SchemaTestCase; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; +import com.yahoo.searchdefinition.AbstractSchemaTestCase; import com.yahoo.searchdefinition.SearchBuilder; import com.yahoo.searchdefinition.derived.DerivedConfiguration; import com.yahoo.searchdefinition.derived.AttributeFields; @@ -28,9 +28,9 @@ import java.util.Map; import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertTrue; -public class RankingExpressionsTestCase extends SchemaTestCase { +public class RankingExpressionsTestCase extends AbstractSchemaTestCase { - private static Search createSearch(String dir, ModelContext.Properties deployProperties, RankProfileRegistry rankProfileRegistry) throws IOException, ParseException { + private static Schema createSearch(String dir, ModelContext.Properties deployProperties, RankProfileRegistry rankProfileRegistry) throws IOException, ParseException { return SearchBuilder.createFromDirectory(dir, new MockFileRegistry(), new TestableDeployLogger(), deployProperties, rankProfileRegistry).getSearch(); } @@ -38,8 +38,8 @@ public class RankingExpressionsTestCase extends SchemaTestCase { public void testFunctions() throws IOException, ParseException { ModelContext.Properties deployProperties = new TestProperties(); RankProfileRegistry rankProfileRegistry = new RankProfileRegistry(); - Search search = createSearch("src/test/examples/rankingexpressionfunction", deployProperties, rankProfileRegistry); - RankProfile functionsRankProfile = rankProfileRegistry.get(search, "macros"); + Schema schema = createSearch("src/test/examples/rankingexpressionfunction", deployProperties, rankProfileRegistry); + RankProfile functionsRankProfile = rankProfileRegistry.get(schema, "macros"); Map<String, RankProfile.RankingExpressionFunction> functions = functionsRankProfile.getFunctions(); assertEquals(2, functions.get("titlematch$").function().arguments().size()); assertEquals("var1", functions.get("titlematch$").function().arguments().get(0)); @@ -52,7 +52,7 @@ public class RankingExpressionsTestCase extends SchemaTestCase { assertEquals(0, functions.get("artistmatch").function().arguments().size()); RawRankProfile rawRankProfile = new RawRankProfile(functionsRankProfile, new LargeRankExpressions(new MockFileRegistry()), new QueryProfileRegistry(), - new ImportedMlModels(), new AttributeFields(search), deployProperties); + new ImportedMlModels(), new AttributeFields(schema), deployProperties); List<Pair<String, String>> rankProperties = rawRankProfile.configProperties(); assertEquals(6, rankProperties.size()); @@ -75,8 +75,8 @@ public class RankingExpressionsTestCase extends SchemaTestCase { @Test(expected = IllegalArgumentException.class) public void testThatIncludingFileInSubdirFails() throws IOException, ParseException { RankProfileRegistry registry = new RankProfileRegistry(); - Search search = createSearch("src/test/examples/rankingexpressioninfile", new TestProperties(), registry); - new DerivedConfiguration(search, registry); // rank profile parsing happens during deriving + Schema schema = createSearch("src/test/examples/rankingexpressioninfile", new TestProperties(), registry); + new DerivedConfiguration(schema, registry); // rank profile parsing happens during deriving } private void verifyProfile(RankProfile profile, List<String> expectedFunctions, List<Pair<String, String>> rankProperties, @@ -96,20 +96,20 @@ public class RankingExpressionsTestCase extends SchemaTestCase { } } - private void verifySearch(Search search, RankProfileRegistry rankProfileRegistry, LargeRankExpressions largeExpressions, + private void verifySearch(Schema schema, RankProfileRegistry rankProfileRegistry, LargeRankExpressions largeExpressions, QueryProfileRegistry queryProfiles, ImportedMlModels models, ModelContext.Properties properties) { - AttributeFields attributes = new AttributeFields(search); + AttributeFields attributes = new AttributeFields(schema); - verifyProfile(rankProfileRegistry.get(search, "base"), Arrays.asList("large_f", "large_m"), - Arrays.asList(new Pair<>("rankingExpression(large_f).expressionName", "base.large_f"), new Pair<>("rankingExpression(large_m).expressionName", "base.large_m")), - largeExpressions, queryProfiles, models, attributes, properties); + verifyProfile(rankProfileRegistry.get(schema, "base"), Arrays.asList("large_f", "large_m"), + Arrays.asList(new Pair<>("rankingExpression(large_f).expressionName", "base.large_f"), new Pair<>("rankingExpression(large_m).expressionName", "base.large_m")), + largeExpressions, queryProfiles, models, attributes, properties); for (String child : Arrays.asList("child_a", "child_b")) { - verifyProfile(rankProfileRegistry.get(search, child), Arrays.asList("large_f", "large_m", "large_local_f", "large_local_m"), - Arrays.asList(new Pair<>("rankingExpression(large_f).expressionName", child + ".large_f"), new Pair<>("rankingExpression(large_m).expressionName", child + ".large_m"), + verifyProfile(rankProfileRegistry.get(schema, child), Arrays.asList("large_f", "large_m", "large_local_f", "large_local_m"), + Arrays.asList(new Pair<>("rankingExpression(large_f).expressionName", child + ".large_f"), new Pair<>("rankingExpression(large_m).expressionName", child + ".large_m"), new Pair<>("rankingExpression(large_local_f).expressionName", child + ".large_local_f"), new Pair<>("rankingExpression(large_local_m).expressionName", child + ".large_local_m"), new Pair<>("vespa.rank.firstphase", "rankingExpression(firstphase)"), new Pair<>("rankingExpression(firstphase).expressionName", child + ".firstphase")), - largeExpressions, queryProfiles, models, attributes, properties); + largeExpressions, queryProfiles, models, attributes, properties); } } @@ -120,9 +120,9 @@ public class RankingExpressionsTestCase extends SchemaTestCase { LargeRankExpressions largeExpressions = new LargeRankExpressions(new MockFileRegistry()); QueryProfileRegistry queryProfiles = new QueryProfileRegistry(); ImportedMlModels models = new ImportedMlModels(); - Search search = createSearch("src/test/examples/largerankingexpressions", properties, rankProfileRegistry); - verifySearch(search, rankProfileRegistry, largeExpressions, queryProfiles, models, properties); + Schema schema = createSearch("src/test/examples/largerankingexpressions", properties, rankProfileRegistry); + verifySearch(schema, rankProfileRegistry, largeExpressions, queryProfiles, models, properties); // Need to verify that second derivation works as that will happen if same sd is used in multiple content clusters - verifySearch(search, rankProfileRegistry, largeExpressions, queryProfiles, models, properties); + verifySearch(schema, rankProfileRegistry, largeExpressions, queryProfiles, models, properties); } } diff --git a/config-model/src/test/java/com/yahoo/searchdefinition/processing/ReferenceFieldTestCase.java b/config-model/src/test/java/com/yahoo/searchdefinition/processing/ReferenceFieldTestCase.java index ad38e65e655..41b9d480921 100644 --- a/config-model/src/test/java/com/yahoo/searchdefinition/processing/ReferenceFieldTestCase.java +++ b/config-model/src/test/java/com/yahoo/searchdefinition/processing/ReferenceFieldTestCase.java @@ -5,7 +5,7 @@ import com.yahoo.document.DataType; import com.yahoo.document.Field; import com.yahoo.document.ReferenceDataType; import com.yahoo.searchdefinition.DocumentGraphValidator; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.SearchBuilder; import com.yahoo.searchdefinition.document.SDDocumentType; import com.yahoo.searchdefinition.parser.ParseException; @@ -50,9 +50,9 @@ public class ReferenceFieldTestCase { builder.importString(salespersonSdContent); builder.importString(adSdContent); builder.build(); - Search search = builder.getSearch("ad"); - assertSearchContainsReferenceField("campaign_ref", "campaign", search.getDocument()); - assertSearchContainsReferenceField("salesperson_ref", "salesperson", search.getDocument()); + Schema schema = builder.getSearch("ad"); + assertSearchContainsReferenceField("campaign_ref", "campaign", schema.getDocument()); + assertSearchContainsReferenceField("salesperson_ref", "salesperson", schema.getDocument()); } @Test diff --git a/config-model/src/test/java/com/yahoo/searchdefinition/processing/ReservedDocumentNamesTestCase.java b/config-model/src/test/java/com/yahoo/searchdefinition/processing/ReservedDocumentNamesTestCase.java index 3e61cec54da..6d722138871 100644 --- a/config-model/src/test/java/com/yahoo/searchdefinition/processing/ReservedDocumentNamesTestCase.java +++ b/config-model/src/test/java/com/yahoo/searchdefinition/processing/ReservedDocumentNamesTestCase.java @@ -21,7 +21,7 @@ public class ReservedDocumentNamesTestCase extends AbstractExportingTestCase { assertCorrectDeriving("reserved_position"); fail(); } catch (IllegalArgumentException e) { - assertEquals("For search 'position': Document name 'position' is reserved.", e.getMessage()); + assertEquals("For schema 'position': Document name 'position' is reserved.", e.getMessage()); } } } diff --git a/config-model/src/test/java/com/yahoo/searchdefinition/processing/SearchMustHaveDocumentTest.java b/config-model/src/test/java/com/yahoo/searchdefinition/processing/SchemaMustHaveDocumentTest.java index 4fe03fe0faa..34572280d01 100644 --- a/config-model/src/test/java/com/yahoo/searchdefinition/processing/SearchMustHaveDocumentTest.java +++ b/config-model/src/test/java/com/yahoo/searchdefinition/processing/SchemaMustHaveDocumentTest.java @@ -12,7 +12,7 @@ import static org.junit.Assert.fail; /** * @author hmusum */ -public class SearchMustHaveDocumentTest { +public class SchemaMustHaveDocumentTest { @Test public void requireErrorWhenMissingDocument() throws IOException, ParseException { @@ -21,7 +21,7 @@ public class SearchMustHaveDocumentTest { fail("SD without document"); } catch (IllegalArgumentException e) { if (!e.getMessage() - .contains("For search 'imageconfig': A search specification must have an equally named document inside of it.")) { + .contains("For schema 'imageconfig': A search specification must have an equally named document inside of it.")) { throw e; } } diff --git a/config-model/src/test/java/com/yahoo/searchdefinition/processing/SummaryConsistencyTestCase.java b/config-model/src/test/java/com/yahoo/searchdefinition/processing/SummaryConsistencyTestCase.java index 71b4db60886..e0593aef1e3 100644 --- a/config-model/src/test/java/com/yahoo/searchdefinition/processing/SummaryConsistencyTestCase.java +++ b/config-model/src/test/java/com/yahoo/searchdefinition/processing/SummaryConsistencyTestCase.java @@ -1,7 +1,7 @@ // Copyright 2019 Oath Inc. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. package com.yahoo.searchdefinition.processing; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.SearchBuilder; import com.yahoo.searchdefinition.parser.ParseException; import com.yahoo.vespa.documentmodel.SummaryTransform; @@ -39,7 +39,7 @@ public class SummaryConsistencyTestCase { "", "}" ); - Search search = SearchBuilder.createFromString(sd).getSearch(); - assertEquals(SummaryTransform.ATTRIBUTECOMBINER, search.getSummaryField("elem_array_unfiltered").getTransform()); + Schema schema = SearchBuilder.createFromString(sd).getSearch(); + assertEquals(SummaryTransform.ATTRIBUTECOMBINER, schema.getSummaryField("elem_array_unfiltered").getTransform()); } } diff --git a/config-model/src/test/java/com/yahoo/searchdefinition/processing/SummaryFieldsMustHaveValidSourceTestCase.java b/config-model/src/test/java/com/yahoo/searchdefinition/processing/SummaryFieldsMustHaveValidSourceTestCase.java index adb6cc03d5b..014ca4ec5e9 100644 --- a/config-model/src/test/java/com/yahoo/searchdefinition/processing/SummaryFieldsMustHaveValidSourceTestCase.java +++ b/config-model/src/test/java/com/yahoo/searchdefinition/processing/SummaryFieldsMustHaveValidSourceTestCase.java @@ -4,9 +4,9 @@ package com.yahoo.searchdefinition.processing; import com.yahoo.config.model.application.provider.BaseDeployLogger; import com.yahoo.searchdefinition.RankProfileRegistry; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.SearchBuilder; -import com.yahoo.searchdefinition.SchemaTestCase; +import com.yahoo.searchdefinition.AbstractSchemaTestCase; import com.yahoo.searchdefinition.parser.ParseException; import com.yahoo.vespa.model.container.search.QueryProfiles; import org.junit.Test; @@ -16,7 +16,7 @@ import java.io.IOException; import static org.junit.Assert.assertEquals; import static org.junit.Assert.fail; -public class SummaryFieldsMustHaveValidSourceTestCase extends SchemaTestCase { +public class SummaryFieldsMustHaveValidSourceTestCase extends AbstractSchemaTestCase { @Test public void requireThatInvalidSourceIsCaught() throws IOException, ParseException { @@ -24,7 +24,7 @@ public class SummaryFieldsMustHaveValidSourceTestCase extends SchemaTestCase { SearchBuilder.buildFromFile("src/test/examples/invalidsummarysource.sd"); fail("This should throw and never get here"); } catch (IllegalArgumentException e) { - assertEquals("For search 'invalidsummarysource', summary class 'baz', summary field 'cox': there is no valid source 'nonexistingfield'.", e.getMessage()); + assertEquals("For schema 'invalidsummarysource', summary class 'baz', summary field 'cox': there is no valid source 'nonexistingfield'.", e.getMessage()); } } @@ -34,7 +34,7 @@ public class SummaryFieldsMustHaveValidSourceTestCase extends SchemaTestCase { SearchBuilder.buildFromFile("src/test/examples/invalidimplicitsummarysource.sd"); fail("This should throw and never get here"); } catch (IllegalArgumentException e) { - assertEquals("For search 'invalidsummarysource', summary class 'baz', summary field 'cox': there is no valid source 'cox'.", e.getMessage()); + assertEquals("For schema 'invalidsummarysource', summary class 'baz', summary field 'cox': there is no valid source 'cox'.", e.getMessage()); } } @@ -44,17 +44,17 @@ public class SummaryFieldsMustHaveValidSourceTestCase extends SchemaTestCase { SearchBuilder.buildFromFile("src/test/examples/invalidselfreferringsummary.sd"); fail("This should throw and never get here"); } catch (IllegalArgumentException e) { - assertEquals("For search 'invalidselfreferringsummary', summary class 'withid', summary field 'w': there is no valid source 'w'.", e.getMessage()); + assertEquals("For schema 'invalidselfreferringsummary', summary class 'withid', summary field 'w': there is no valid source 'w'.", e.getMessage()); } } @Test public void requireThatDocumentIdIsAllowedToPass() throws IOException, ParseException { - Search search = SearchBuilder.buildFromFile("src/test/examples/documentidinsummary.sd"); + Schema schema = SearchBuilder.buildFromFile("src/test/examples/documentidinsummary.sd"); BaseDeployLogger deployLogger = new BaseDeployLogger(); RankProfileRegistry rankProfileRegistry = new RankProfileRegistry(); - new SummaryFieldsMustHaveValidSource(search, deployLogger, rankProfileRegistry, new QueryProfiles()).process(true, false); - assertEquals("documentid", search.getSummary("withid").getSummaryField("w").getSingleSource()); + new SummaryFieldsMustHaveValidSource(schema, deployLogger, rankProfileRegistry, new QueryProfiles()).process(true, false); + assertEquals("documentid", schema.getSummary("withid").getSummaryField("w").getSingleSource()); } } diff --git a/config-model/src/test/java/com/yahoo/searchdefinition/processing/TensorFieldTestCase.java b/config-model/src/test/java/com/yahoo/searchdefinition/processing/TensorFieldTestCase.java index c8b0a826d48..cef00346fe7 100644 --- a/config-model/src/test/java/com/yahoo/searchdefinition/processing/TensorFieldTestCase.java +++ b/config-model/src/test/java/com/yahoo/searchdefinition/processing/TensorFieldTestCase.java @@ -26,7 +26,7 @@ public class TensorFieldTestCase { fail("Expected exception"); } catch (IllegalArgumentException e) { - assertEquals("For search 'test', field 'f1': A field with collection type of tensor is not supported. Use simple type 'tensor' instead.", + assertEquals("For schema 'test', field 'f1': A field with collection type of tensor is not supported. Use simple type 'tensor' instead.", e.getMessage()); } } @@ -38,7 +38,7 @@ public class TensorFieldTestCase { fail("Expected exception"); } catch (IllegalArgumentException e) { - assertEquals("For search 'test', field 'f1': A tensor of type 'tensor(x{})' does not support having an 'index'. " + + assertEquals("For schema 'test', field 'f1': A tensor of type 'tensor(x{})' does not support having an 'index'. " + "Currently, only tensors with 1 indexed dimension supports that.", e.getMessage()); } @@ -51,7 +51,7 @@ public class TensorFieldTestCase { fail("Expected exception"); } catch (IllegalArgumentException e) { - assertEquals("For search 'test', field 'f1': An attribute of type 'tensor' cannot be 'fast-search'.", e.getMessage()); + assertEquals("For schema 'test', field 'f1': An attribute of type 'tensor' cannot be 'fast-search'.", e.getMessage()); } } @@ -98,7 +98,7 @@ public class TensorFieldTestCase { fail("Expected exception"); } catch (IllegalArgumentException e) { - assertEquals("For search 'test', field 't1': A tensor that has an index must also be an attribute.", e.getMessage()); + assertEquals("For schema 'test', field 't1': A tensor that has an index must also be an attribute.", e.getMessage()); } } @@ -115,7 +115,7 @@ public class TensorFieldTestCase { fail("Expected exception"); } catch (IllegalArgumentException e) { - assertEquals("For search 'test', field 't1': " + + assertEquals("For schema 'test', field 't1': " + "A tensor that specifies hnsw index parameters must also specify 'index' in 'indexing'", e.getMessage()); } diff --git a/config-model/src/test/java/com/yahoo/searchdefinition/processing/TensorTransformTestCase.java b/config-model/src/test/java/com/yahoo/searchdefinition/processing/TensorTransformTestCase.java index 8714568338f..888cd22c1c3 100644 --- a/config-model/src/test/java/com/yahoo/searchdefinition/processing/TensorTransformTestCase.java +++ b/config-model/src/test/java/com/yahoo/searchdefinition/processing/TensorTransformTestCase.java @@ -3,7 +3,6 @@ package com.yahoo.searchdefinition.processing; import com.yahoo.collections.Pair; import com.yahoo.component.ComponentId; -import com.yahoo.config.model.application.provider.BaseDeployLogger; import com.yahoo.config.model.application.provider.MockFileRegistry; import com.yahoo.config.model.deploy.TestProperties; import com.yahoo.search.query.profile.QueryProfileRegistry; @@ -14,9 +13,9 @@ import com.yahoo.search.query.profile.types.QueryProfileTypeRegistry; import com.yahoo.searchdefinition.LargeRankExpressions; import com.yahoo.searchdefinition.RankProfile; import com.yahoo.searchdefinition.RankProfileRegistry; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.SearchBuilder; -import com.yahoo.searchdefinition.SchemaTestCase; +import com.yahoo.searchdefinition.AbstractSchemaTestCase; import com.yahoo.searchdefinition.derived.AttributeFields; import com.yahoo.searchdefinition.derived.RawRankProfile; import com.yahoo.searchdefinition.parser.ParseException; @@ -28,7 +27,7 @@ import java.util.List; import static org.junit.Assert.assertEquals; import static org.junit.Assert.fail; -public class TensorTransformTestCase extends SchemaTestCase { +public class TensorTransformTestCase extends AbstractSchemaTestCase { @Test public void requireThatNormalMaxAndMinAreNotReplaced() throws ParseException { @@ -199,7 +198,7 @@ public class TensorTransformTestCase extends SchemaTestCase { " }\n" + "}\n"); builder.build(true); - Search s = builder.getSearch(); + Schema s = builder.getSearch(); RankProfile test = rankProfileRegistry.get(s, "test").compile(queryProfiles, new ImportedMlModels()); List<Pair<String, String>> testRankProperties = new RawRankProfile(test, new LargeRankExpressions(new MockFileRegistry()), diff --git a/config-model/src/test/java/com/yahoo/searchdefinition/processing/ValidateFieldTypesTest.java b/config-model/src/test/java/com/yahoo/searchdefinition/processing/ValidateFieldTypesTest.java index c7b9ce16338..08c123b5578 100644 --- a/config-model/src/test/java/com/yahoo/searchdefinition/processing/ValidateFieldTypesTest.java +++ b/config-model/src/test/java/com/yahoo/searchdefinition/processing/ValidateFieldTypesTest.java @@ -6,8 +6,9 @@ import com.yahoo.config.model.deploy.TestProperties; import com.yahoo.config.model.test.MockApplicationPackage; import com.yahoo.document.DataType; import com.yahoo.document.Field; +import com.yahoo.searchdefinition.Application; import com.yahoo.searchdefinition.DocumentReference; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.derived.TestableDeployLogger; import com.yahoo.searchdefinition.document.ImportedField; import com.yahoo.searchdefinition.document.ImportedFields; @@ -35,39 +36,43 @@ public class ValidateFieldTypesTest { @Test public void throws_exception_if_type_of_document_field_does_not_match_summary_field() { - Search search = createSearchWithDocument(DOCUMENT_NAME); - search.setImportedFields(createSingleImportedField(IMPORTED_FIELD_NAME, DataType.INT)); - search.addSummary(createDocumentSummary(IMPORTED_FIELD_NAME, DataType.STRING, search)); + Schema schema = createSearchWithDocument(DOCUMENT_NAME); + schema.setImportedFields(createSingleImportedField(IMPORTED_FIELD_NAME, DataType.INT)); + schema.addSummary(createDocumentSummary(IMPORTED_FIELD_NAME, DataType.STRING, schema)); - ValidateFieldTypes validator = new ValidateFieldTypes(search, null, null, null); + ValidateFieldTypes validator = new ValidateFieldTypes(schema, null, null, null); exceptionRule.expect(IllegalArgumentException.class); exceptionRule.expectMessage( - "For search '" + DOCUMENT_NAME + "', field '" + IMPORTED_FIELD_NAME + "': Incompatible types. " + + "For schema '" + DOCUMENT_NAME + "', field '" + IMPORTED_FIELD_NAME + "': Incompatible types. " + "Expected int for summary field '" + IMPORTED_FIELD_NAME + "', got string."); validator.process(true, false); } - private static Search createSearch(String documentType) { - return new Search(documentType, MockApplicationPackage.createEmpty(), new MockFileRegistry(), new TestableDeployLogger(), new TestProperties()); + private static Schema createSearch(String documentType) { + return new Schema(documentType, + new Application(MockApplicationPackage.createEmpty()), + new MockFileRegistry(), + new TestableDeployLogger(), + new TestProperties()); } - private static Search createSearchWithDocument(String documentName) { - Search search = createSearch(documentName); - SDDocumentType document = new SDDocumentType(documentName, search); - search.addDocument(document); - return search; + private static Schema createSearchWithDocument(String documentName) { + Schema schema = createSearch(documentName); + SDDocumentType document = new SDDocumentType(documentName, schema); + schema.addDocument(document); + return schema; } private static ImportedFields createSingleImportedField(String fieldName, DataType dataType) { - Search targetSearch = createSearch("target_doc"); + Schema targetSchema = createSearch("target_doc"); SDField targetField = new SDField("target_field", dataType); - DocumentReference documentReference = new DocumentReference(new Field("reference_field"), targetSearch); + DocumentReference documentReference = new DocumentReference(new Field("reference_field"), targetSchema); ImportedField importedField = new ImportedSimpleField(fieldName, documentReference, targetField); return new ImportedFields(Collections.singletonMap(fieldName, importedField)); } - private static DocumentSummary createDocumentSummary(String fieldName, DataType dataType, Search search) { - DocumentSummary summary = new DocumentSummary("mysummary", search); + private static DocumentSummary createDocumentSummary(String fieldName, DataType dataType, Schema schema) { + DocumentSummary summary = new DocumentSummary("mysummary", schema); summary.add(new SummaryField(fieldName, dataType)); return summary; } diff --git a/config-model/src/test/java/com/yahoo/searchdefinition/processing/WeightedSetSummaryToTestCase.java b/config-model/src/test/java/com/yahoo/searchdefinition/processing/WeightedSetSummaryToTestCase.java index 3b2875d303c..e9aec9b7293 100644 --- a/config-model/src/test/java/com/yahoo/searchdefinition/processing/WeightedSetSummaryToTestCase.java +++ b/config-model/src/test/java/com/yahoo/searchdefinition/processing/WeightedSetSummaryToTestCase.java @@ -1,9 +1,9 @@ // Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. package com.yahoo.searchdefinition.processing; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.SearchBuilder; -import com.yahoo.searchdefinition.SchemaTestCase; +import com.yahoo.searchdefinition.AbstractSchemaTestCase; import com.yahoo.searchdefinition.parser.ParseException; import org.junit.Test; @@ -12,12 +12,12 @@ import java.io.IOException; import static org.junit.Assert.assertNotNull; /** @author bratseth */ -public class WeightedSetSummaryToTestCase extends SchemaTestCase { +public class WeightedSetSummaryToTestCase extends AbstractSchemaTestCase { @Test public void testRequireThatImplicitFieldsAreCreated() throws IOException, ParseException { - Search search = SearchBuilder.buildFromFile("src/test/examples/weightedset-summaryto.sd"); - assertNotNull(search); + Schema schema = SearchBuilder.buildFromFile("src/test/examples/weightedset-summaryto.sd"); + assertNotNull(schema); } } diff --git a/config-model/src/test/java/com/yahoo/vespa/documentmodel/AbstractReferenceFieldTestCase.java b/config-model/src/test/java/com/yahoo/vespa/documentmodel/AbstractReferenceFieldTestCase.java index 9144ad411b2..e3f16711d73 100644 --- a/config-model/src/test/java/com/yahoo/vespa/documentmodel/AbstractReferenceFieldTestCase.java +++ b/config-model/src/test/java/com/yahoo/vespa/documentmodel/AbstractReferenceFieldTestCase.java @@ -3,7 +3,7 @@ package com.yahoo.vespa.documentmodel; import com.yahoo.document.DocumenttypesConfig; import com.yahoo.document.config.DocumentmanagerConfig; -import com.yahoo.searchdefinition.SchemaTestCase; +import com.yahoo.searchdefinition.AbstractSchemaTestCase; import com.yahoo.vespa.configmodel.producers.DocumentManager; import com.yahoo.vespa.configmodel.producers.DocumentTypes; @@ -12,7 +12,7 @@ import java.io.IOException; /** * Utility functions for testing generated configs for reference/imported fields. */ -public abstract class AbstractReferenceFieldTestCase extends SchemaTestCase { +public abstract class AbstractReferenceFieldTestCase extends AbstractSchemaTestCase { private static String TEST_FOLDER = "src/test/configmodel/types/references/"; diff --git a/config-model/src/test/java/com/yahoo/vespa/documentmodel/DocumentModelBuilderTestCase.java b/config-model/src/test/java/com/yahoo/vespa/documentmodel/DocumentModelBuilderTestCase.java index 04ac16eb999..15de4b8c02e 100644 --- a/config-model/src/test/java/com/yahoo/vespa/documentmodel/DocumentModelBuilderTestCase.java +++ b/config-model/src/test/java/com/yahoo/vespa/documentmodel/DocumentModelBuilderTestCase.java @@ -4,7 +4,7 @@ package com.yahoo.vespa.documentmodel; import com.yahoo.document.DocumenttypesConfig; import com.yahoo.document.config.DocumentmanagerConfig; import com.yahoo.searchdefinition.SearchBuilder; -import com.yahoo.searchdefinition.SchemaTestCase; +import com.yahoo.searchdefinition.AbstractSchemaTestCase; import com.yahoo.searchdefinition.parser.ParseException; import com.yahoo.vespa.configmodel.producers.DocumentManager; import com.yahoo.vespa.configmodel.producers.DocumentTypes; @@ -13,7 +13,7 @@ import java.io.IOException; import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertNotNull; -public class DocumentModelBuilderTestCase extends SchemaTestCase { +public class DocumentModelBuilderTestCase extends AbstractSchemaTestCase { @Test public void testDocumentManagerSimple() throws IOException, ParseException { diff --git a/config-model/src/test/java/com/yahoo/vespa/model/application/validation/NoPrefixForIndexesTest.java b/config-model/src/test/java/com/yahoo/vespa/model/application/validation/NoPrefixForIndexesTest.java index 52a7d383587..5f0296588db 100644 --- a/config-model/src/test/java/com/yahoo/vespa/model/application/validation/NoPrefixForIndexesTest.java +++ b/config-model/src/test/java/com/yahoo/vespa/model/application/validation/NoPrefixForIndexesTest.java @@ -28,7 +28,7 @@ public class NoPrefixForIndexesTest { new VespaModelCreatorWithFilePkg("src/test/cfg/application/validation/prefix_index/").create(); fail(); } catch (IllegalArgumentException e) { - assertEquals("For search 'simple', field 'artist': match/index:prefix is not supported for indexes.", e.getMessage()); + assertEquals("For schema 'simple', field 'artist': match/index:prefix is not supported for indexes.", e.getMessage()); } } @@ -38,7 +38,7 @@ public class NoPrefixForIndexesTest { new VespaModelCreatorWithFilePkg("src/test/cfg/application/validation/prefix_index_and_attribute/").create(); fail(); } catch (IllegalArgumentException e) { - assertEquals("For search 'simple', field 'artist': match/index:prefix is not supported for indexes.", e.getMessage()); + assertEquals("For schema 'simple', field 'artist': match/index:prefix is not supported for indexes.", e.getMessage()); } } } diff --git a/config-model/src/test/java/com/yahoo/vespa/model/application/validation/SearchDataTypeValidatorTestCase.java b/config-model/src/test/java/com/yahoo/vespa/model/application/validation/SchemaDataTypeValidatorTestCase.java index cbea01e3ca3..b7b83dbf68c 100644 --- a/config-model/src/test/java/com/yahoo/vespa/model/application/validation/SearchDataTypeValidatorTestCase.java +++ b/config-model/src/test/java/com/yahoo/vespa/model/application/validation/SchemaDataTypeValidatorTestCase.java @@ -10,7 +10,7 @@ import static org.junit.Assert.fail; /** * @author Simon Thoresen Hult */ -public class SearchDataTypeValidatorTestCase { +public class SchemaDataTypeValidatorTestCase { @Test public void requireThatSupportedTypesAreValidated() { diff --git a/config-model/src/test/java/com/yahoo/vespa/model/application/validation/change/IndexedSearchClusterChangeValidatorTest.java b/config-model/src/test/java/com/yahoo/vespa/model/application/validation/change/IndexedSchemaClusterChangeValidatorTest.java index 44d62b72ce2..7b1474f9754 100644 --- a/config-model/src/test/java/com/yahoo/vespa/model/application/validation/change/IndexedSearchClusterChangeValidatorTest.java +++ b/config-model/src/test/java/com/yahoo/vespa/model/application/validation/change/IndexedSchemaClusterChangeValidatorTest.java @@ -21,7 +21,7 @@ import static com.yahoo.vespa.model.application.validation.change.ConfigChangeTe import static com.yahoo.vespa.model.application.validation.change.ConfigChangeTestUtils.normalizeServicesInActions; import static org.junit.Assert.assertTrue; -public class IndexedSearchClusterChangeValidatorTest { +public class IndexedSchemaClusterChangeValidatorTest { static class Fixture { VespaModel currentModel; diff --git a/config-model/src/test/java/com/yahoo/vespa/model/application/validation/change/StreamingSearchClusterChangeValidatorTest.java b/config-model/src/test/java/com/yahoo/vespa/model/application/validation/change/StreamingSchemaClusterChangeValidatorTest.java index a4080250c60..dddb6a38db1 100644 --- a/config-model/src/test/java/com/yahoo/vespa/model/application/validation/change/StreamingSearchClusterChangeValidatorTest.java +++ b/config-model/src/test/java/com/yahoo/vespa/model/application/validation/change/StreamingSchemaClusterChangeValidatorTest.java @@ -22,7 +22,7 @@ import static com.yahoo.vespa.model.application.validation.change.ConfigChangeTe import static org.hamcrest.CoreMatchers.is; import static org.junit.Assert.assertThat; -public class StreamingSearchClusterChangeValidatorTest { +public class StreamingSchemaClusterChangeValidatorTest { private static class Fixture { VespaModel currentModel; diff --git a/config-model/src/test/java/com/yahoo/vespa/model/application/validation/change/search/DocumentTypeChangeValidatorTest.java b/config-model/src/test/java/com/yahoo/vespa/model/application/validation/change/search/DocumentTypeChangeValidatorTest.java index be47f448e40..0e96420f03c 100644 --- a/config-model/src/test/java/com/yahoo/vespa/model/application/validation/change/search/DocumentTypeChangeValidatorTest.java +++ b/config-model/src/test/java/com/yahoo/vespa/model/application/validation/change/search/DocumentTypeChangeValidatorTest.java @@ -17,6 +17,7 @@ import java.time.Instant; import java.util.Arrays; import java.util.Collections; import java.util.List; +import java.util.Optional; import static com.yahoo.vespa.model.application.validation.change.ConfigChangeTestUtils.newRefeedAction; import static org.junit.Assert.assertEquals; @@ -208,7 +209,7 @@ public class DocumentTypeChangeValidatorTest { return new NewDocumentType( new NewDocumentType.Name("mydoc"), headerfields, - new FieldSets(), + new FieldSets(Optional.empty()), Collections.emptySet(), Collections.emptySet()); } diff --git a/config-model/src/test/java/com/yahoo/vespa/model/builder/xml/dom/DomSearchTuningBuilderTest.java b/config-model/src/test/java/com/yahoo/vespa/model/builder/xml/dom/DomSchemaTuningBuilderTest.java index 7578fc5c8f8..4ed355a7dca 100644 --- a/config-model/src/test/java/com/yahoo/vespa/model/builder/xml/dom/DomSearchTuningBuilderTest.java +++ b/config-model/src/test/java/com/yahoo/vespa/model/builder/xml/dom/DomSchemaTuningBuilderTest.java @@ -18,7 +18,7 @@ import static org.junit.Assert.assertTrue; /** * @author geirst */ -public class DomSearchTuningBuilderTest extends DomBuilderTest { +public class DomSchemaTuningBuilderTest extends DomBuilderTest { private static final double DELTA = 0.000001; diff --git a/config-model/src/test/java/com/yahoo/vespa/model/builder/xml/dom/chains/search/DomSearchChainsBuilderTest.java b/config-model/src/test/java/com/yahoo/vespa/model/builder/xml/dom/chains/search/DomSchemaChainsBuilderTest.java index a34a5c577b7..86c448c403c 100644 --- a/config-model/src/test/java/com/yahoo/vespa/model/builder/xml/dom/chains/search/DomSearchChainsBuilderTest.java +++ b/config-model/src/test/java/com/yahoo/vespa/model/builder/xml/dom/chains/search/DomSchemaChainsBuilderTest.java @@ -35,7 +35,7 @@ import static org.junit.Assert.fail; * * @author Tony Vaagenes */ -public class DomSearchChainsBuilderTest extends DomBuilderTest { +public class DomSchemaChainsBuilderTest extends DomBuilderTest { private SearchChains searchChains; diff --git a/config-model/src/test/java/com/yahoo/vespa/model/container/search/searchchain/Federation2Test.java b/config-model/src/test/java/com/yahoo/vespa/model/container/search/searchchain/Federation2Test.java index 76b5a102155..ac272301445 100644 --- a/config-model/src/test/java/com/yahoo/vespa/model/container/search/searchchain/Federation2Test.java +++ b/config-model/src/test/java/com/yahoo/vespa/model/container/search/searchchain/Federation2Test.java @@ -10,7 +10,7 @@ import static org.junit.Assert.assertEquals; /** * @author bratseth */ -public class Federation2Test extends SearchChainsTestBase { +public class Federation2Test extends SchemaChainsTestBase { @Override Element servicesXml() { return parse( diff --git a/config-model/src/test/java/com/yahoo/vespa/model/container/search/searchchain/FederationTest.java b/config-model/src/test/java/com/yahoo/vespa/model/container/search/searchchain/FederationTest.java index 0c1b2de1c5d..39fcb9d5af2 100644 --- a/config-model/src/test/java/com/yahoo/vespa/model/container/search/searchchain/FederationTest.java +++ b/config-model/src/test/java/com/yahoo/vespa/model/container/search/searchchain/FederationTest.java @@ -14,7 +14,7 @@ import static org.junit.Assert.*; * Test generated config for federation. * @author Tony Vaagenes */ -public class FederationTest extends SearchChainsTestBase { +public class FederationTest extends SchemaChainsTestBase { @Override Element servicesXml() { return parse( diff --git a/config-model/src/test/java/com/yahoo/vespa/model/container/search/searchchain/SearchChainsTest.java b/config-model/src/test/java/com/yahoo/vespa/model/container/search/searchchain/SchemaChainsTest.java index f55bbe4aee0..9ccfa768dc1 100644 --- a/config-model/src/test/java/com/yahoo/vespa/model/container/search/searchchain/SearchChainsTest.java +++ b/config-model/src/test/java/com/yahoo/vespa/model/container/search/searchchain/SchemaChainsTest.java @@ -21,7 +21,7 @@ import static org.junit.Assert.*; * * @author Tony Vaagenes */ -public class SearchChainsTest extends SearchChainsTestBase { +public class SchemaChainsTest extends SchemaChainsTestBase { private ChainsConfig chainsConfig; private ProviderConfig providerConfig; diff --git a/config-model/src/test/java/com/yahoo/vespa/model/container/search/searchchain/SearchChainsTest2.java b/config-model/src/test/java/com/yahoo/vespa/model/container/search/searchchain/SchemaChainsTest2.java index 40182c74be6..04bda0bce21 100644 --- a/config-model/src/test/java/com/yahoo/vespa/model/container/search/searchchain/SearchChainsTest2.java +++ b/config-model/src/test/java/com/yahoo/vespa/model/container/search/searchchain/SchemaChainsTest2.java @@ -15,7 +15,7 @@ import static org.junit.Assert.fail; /** * @author gjoranv */ -public class SearchChainsTest2 { +public class SchemaChainsTest2 { private MockRoot root; diff --git a/config-model/src/test/java/com/yahoo/vespa/model/container/search/searchchain/SearchChainsTestBase.java b/config-model/src/test/java/com/yahoo/vespa/model/container/search/searchchain/SchemaChainsTestBase.java index 8b92704df3d..e654f8deee1 100644 --- a/config-model/src/test/java/com/yahoo/vespa/model/container/search/searchchain/SearchChainsTestBase.java +++ b/config-model/src/test/java/com/yahoo/vespa/model/container/search/searchchain/SchemaChainsTestBase.java @@ -9,7 +9,7 @@ import org.w3c.dom.Element; /** Creates SearchChains model from xml input. * @author Tony Vaagenes */ -public abstract class SearchChainsTestBase extends DomBuilderTest { +public abstract class SchemaChainsTestBase extends DomBuilderTest { @Before public void setupSearchChains() { diff --git a/config-model/src/test/java/com/yahoo/vespa/model/container/xml/SearchBuilderTest.java b/config-model/src/test/java/com/yahoo/vespa/model/container/xml/SchemaBuilderTest.java index 794cfa1f18a..e7901ebd5f0 100644 --- a/config-model/src/test/java/com/yahoo/vespa/model/container/xml/SearchBuilderTest.java +++ b/config-model/src/test/java/com/yahoo/vespa/model/container/xml/SchemaBuilderTest.java @@ -33,7 +33,7 @@ import static org.junit.Assert.fail; /** * @author gjoranv */ -public class SearchBuilderTest extends ContainerModelBuilderTestBase { +public class SchemaBuilderTest extends ContainerModelBuilderTestBase { private ChainsConfig chainsConfig() { return root.getConfig(ChainsConfig.class, "default/component/com.yahoo.search.handler.SearchHandler"); diff --git a/config-model/src/test/java/com/yahoo/vespa/model/content/ContentSearchClusterTest.java b/config-model/src/test/java/com/yahoo/vespa/model/content/ContentSchemaClusterTest.java index 2383c2db407..d6020a96818 100644 --- a/config-model/src/test/java/com/yahoo/vespa/model/content/ContentSearchClusterTest.java +++ b/config-model/src/test/java/com/yahoo/vespa/model/content/ContentSchemaClusterTest.java @@ -31,7 +31,7 @@ import static org.junit.Assert.assertTrue; * * @author geirst */ -public class ContentSearchClusterTest { +public class ContentSchemaClusterTest { private static double EPSILON = 0.000001; diff --git a/config-model/src/test/java/com/yahoo/vespa/model/content/ContentSearchTest.java b/config-model/src/test/java/com/yahoo/vespa/model/content/ContentSchemaTest.java index 12f3d722e5e..952a0b945dd 100644 --- a/config-model/src/test/java/com/yahoo/vespa/model/content/ContentSearchTest.java +++ b/config-model/src/test/java/com/yahoo/vespa/model/content/ContentSchemaTest.java @@ -9,7 +9,7 @@ import static org.junit.Assert.assertNull; /** * @author Simon Thoresen Hult */ -public class ContentSearchTest { +public class ContentSchemaTest { @Test public void requireThatAccessorsWork() { diff --git a/config-model/src/test/java/com/yahoo/vespa/model/content/IndexedSearchNodeNamingTest.java b/config-model/src/test/java/com/yahoo/vespa/model/content/IndexedSchemaNodeNamingTest.java index 55c9853b02b..17e295c1c2a 100644 --- a/config-model/src/test/java/com/yahoo/vespa/model/content/IndexedSearchNodeNamingTest.java +++ b/config-model/src/test/java/com/yahoo/vespa/model/content/IndexedSchemaNodeNamingTest.java @@ -19,7 +19,7 @@ import static org.junit.Assert.assertEquals; * * @author geirst */ -public class IndexedSearchNodeNamingTest { +public class IndexedSchemaNodeNamingTest { private ContentCluster getSingleNodeCluster() throws Exception { String groupXml = joinLines(" <group>", diff --git a/config-model/src/test/java/com/yahoo/vespa/model/content/SearchCoverageTest.java b/config-model/src/test/java/com/yahoo/vespa/model/content/SchemaCoverageTest.java index 8db115aeb10..66184d74251 100644 --- a/config-model/src/test/java/com/yahoo/vespa/model/content/SearchCoverageTest.java +++ b/config-model/src/test/java/com/yahoo/vespa/model/content/SchemaCoverageTest.java @@ -10,7 +10,7 @@ import static org.junit.Assert.fail; /** * @author Simon Thoresen Hult */ -public class SearchCoverageTest { +public class SchemaCoverageTest { @Test public void requireThatAccessorWork() { diff --git a/config-model/src/test/java/com/yahoo/vespa/model/content/cluster/DomContentSearchBuilderTest.java b/config-model/src/test/java/com/yahoo/vespa/model/content/cluster/DomContentSchemaBuilderTest.java index 5222d0a8715..b8788ece7b6 100644 --- a/config-model/src/test/java/com/yahoo/vespa/model/content/cluster/DomContentSearchBuilderTest.java +++ b/config-model/src/test/java/com/yahoo/vespa/model/content/cluster/DomContentSchemaBuilderTest.java @@ -15,7 +15,7 @@ import static org.junit.Assert.assertNull; /** * @author Simon Thoresen Hult */ -public class DomContentSearchBuilderTest { +public class DomContentSchemaBuilderTest { @Test public void requireThatDefaultsAreNull() throws Exception { diff --git a/config-model/src/test/java/com/yahoo/vespa/model/content/cluster/DomSearchCoverageBuilderTest.java b/config-model/src/test/java/com/yahoo/vespa/model/content/cluster/DomSchemaCoverageBuilderTest.java index 7512b6b5469..6c5456e73fc 100644 --- a/config-model/src/test/java/com/yahoo/vespa/model/content/cluster/DomSearchCoverageBuilderTest.java +++ b/config-model/src/test/java/com/yahoo/vespa/model/content/cluster/DomSchemaCoverageBuilderTest.java @@ -15,7 +15,7 @@ import static org.junit.Assert.assertNull; /** * @author Simon Thoresen Hult */ -public class DomSearchCoverageBuilderTest { +public class DomSchemaCoverageBuilderTest { @Test public void requireThatDefaultsAreNull() throws Exception { diff --git a/config-model/src/test/java/com/yahoo/vespa/model/search/test/SearchClusterTest.java b/config-model/src/test/java/com/yahoo/vespa/model/search/test/SchemaClusterTest.java index 660ec9f13f1..27dda6a6bb2 100644 --- a/config-model/src/test/java/com/yahoo/vespa/model/search/test/SearchClusterTest.java +++ b/config-model/src/test/java/com/yahoo/vespa/model/search/test/SchemaClusterTest.java @@ -5,7 +5,7 @@ import com.yahoo.component.ComponentId; import com.yahoo.container.QrSearchersConfig; import com.yahoo.document.DataType; import com.yahoo.search.config.ClusterConfig; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.SearchBuilder; import com.yahoo.searchdefinition.document.Attribute; import com.yahoo.searchdefinition.document.SDDocumentType; @@ -34,31 +34,31 @@ import static org.junit.Assert.assertTrue; * * @author hmusum */ -public class SearchClusterTest { +public class SchemaClusterTest { @Test public void testSdConfigLogical() { // sd1 SDDocumentType sdt1 = new SDDocumentType("s1"); - Search search1 = new Search("s1"); + Schema schema1 = new Schema("s1"); SDField f1 = new SDField("f1", DataType.STRING); f1.addAttribute(new Attribute("f1", DataType.STRING)); f1.setIndexingScript(new ScriptExpression(new StatementExpression(new AttributeExpression("f1")))); sdt1.addField(f1); - search1.addDocument(sdt1); + schema1.addDocument(sdt1); // sd2 SDDocumentType sdt2 = new SDDocumentType("s2"); - Search search2 = new Search("s2"); + Schema schema2 = new Schema("s2"); SDField f2=new SDField("f2", DataType.STRING); f2.addAttribute(new Attribute("f2", DataType.STRING)); f2.setIndexingScript(new ScriptExpression(new StatementExpression(new AttributeExpression("f2")))); sdt2.addField(f2); - search2.addDocument(sdt2); + schema2.addDocument(sdt2); SearchBuilder builder = new SearchBuilder(); - builder.importRawSearch(search1); - builder.importRawSearch(search2); + builder.importRawSearch(schema1); + builder.importRawSearch(schema2); builder.build(); } diff --git a/config-model/src/test/java/com/yahoo/vespa/model/search/test/SearchNodeTest.java b/config-model/src/test/java/com/yahoo/vespa/model/search/test/SchemaNodeTest.java index 16f6a90ac66..cca2d817df4 100644 --- a/config-model/src/test/java/com/yahoo/vespa/model/search/test/SearchNodeTest.java +++ b/config-model/src/test/java/com/yahoo/vespa/model/search/test/SchemaNodeTest.java @@ -29,7 +29,7 @@ import static org.junit.Assert.assertTrue; * * @author geirst */ -public class SearchNodeTest { +public class SchemaNodeTest { private void assertBaseDir(String expected, SearchNode node) { ProtonConfig.Builder builder = new ProtonConfig.Builder(); diff --git a/configserver/src/main/java/com/yahoo/vespa/config/server/application/ApplicationReindexing.java b/configserver/src/main/java/com/yahoo/vespa/config/server/application/ApplicationReindexing.java index 81474158bfd..c1ef4127919 100644 --- a/configserver/src/main/java/com/yahoo/vespa/config/server/application/ApplicationReindexing.java +++ b/configserver/src/main/java/com/yahoo/vespa/config/server/application/ApplicationReindexing.java @@ -2,29 +2,17 @@ package com.yahoo.vespa.config.server.application; import com.yahoo.config.model.api.Reindexing; -import com.yahoo.searchdefinition.Search; -import com.yahoo.searchdefinition.document.SDField; -import com.yahoo.vespa.model.VespaModel; -import com.yahoo.vespa.model.content.cluster.ContentCluster; -import com.yahoo.vespa.model.search.AbstractSearchCluster; -import com.yahoo.vespa.model.search.DocumentDatabase; import java.time.Instant; import java.time.temporal.ChronoUnit; -import java.util.Collection; import java.util.HashMap; import java.util.Map; import java.util.Objects; import java.util.Optional; -import java.util.Set; -import java.util.stream.Collectors; -import java.util.stream.Stream; import static java.util.Objects.requireNonNull; import static java.util.stream.Collectors.toMap; -import static java.util.stream.Collectors.toSet; import static java.util.stream.Collectors.toUnmodifiableMap; -import static java.util.stream.Collectors.toUnmodifiableSet; /** * Pending reindexing: convergence to the stored config generation allows reindexing to start. diff --git a/configserver/src/test/java/com/yahoo/vespa/config/server/http/v2/ApplicationHandlerTest.java b/configserver/src/test/java/com/yahoo/vespa/config/server/http/v2/ApplicationHandlerTest.java index 5706851943e..2df7739e552 100644 --- a/configserver/src/test/java/com/yahoo/vespa/config/server/http/v2/ApplicationHandlerTest.java +++ b/configserver/src/test/java/com/yahoo/vespa/config/server/http/v2/ApplicationHandlerTest.java @@ -296,7 +296,7 @@ public class ApplicationHandlerTest { " \"ready\": {" + " \"bar\": {" + " \"readyMillis\": " + (now - 1000) + - " }," + + " }" + " }" + " }," + " \"foo\": {" + @@ -524,7 +524,7 @@ public class ApplicationHandlerTest { " \"ready\": {\n" + " \"bar\": {},\n" + " \"hax\": {}\n" + - " },\n" + + " }\n" + " },\n" + " \"moo\": {\n" + " \"pending\": {},\n" + diff --git a/container-apache-http-client-bundle/README.md b/container-apache-http-client-bundle/README.md index 99606a95820..d54ecab6872 100644 --- a/container-apache-http-client-bundle/README.md +++ b/container-apache-http-client-bundle/README.md @@ -1,3 +1,4 @@ +<!-- Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. --> # container-apache-http-client-bundle Apache http client 4.x/5.x packaged as bundle diff --git a/container-core/abi-spec.json b/container-core/abi-spec.json index 02d43104a3f..8c0f3e5fd80 100644 --- a/container-core/abi-spec.json +++ b/container-core/abi-spec.json @@ -762,6 +762,19 @@ ], "fields": [] }, + "com.yahoo.container.jdisc.MaxPendingContentChannelOutputStream": { + "superClass": "com.yahoo.container.jdisc.ContentChannelOutputStream", + "interfaces": [], + "attributes": [ + "public" + ], + "methods": [ + "public void <init>(com.yahoo.jdisc.handler.ContentChannel, long)", + "public void send(java.nio.ByteBuffer)", + "public void flush()" + ], + "fields": [] + }, "com.yahoo.container.jdisc.MetricConsumerFactory": { "superClass": "java.lang.Object", "interfaces": [], diff --git a/container-core/src/main/java/com/yahoo/container/jdisc/MaxPendingContentChannelOutputStream.java b/container-core/src/main/java/com/yahoo/container/jdisc/MaxPendingContentChannelOutputStream.java new file mode 100644 index 00000000000..aec4eeecd7b --- /dev/null +++ b/container-core/src/main/java/com/yahoo/container/jdisc/MaxPendingContentChannelOutputStream.java @@ -0,0 +1,92 @@ +// Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. +package com.yahoo.container.jdisc; + +import com.yahoo.jdisc.handler.CompletionHandler; +import com.yahoo.jdisc.handler.ContentChannel; + +import java.io.IOException; +import java.io.InterruptedIOException; +import java.nio.ByteBuffer; +import java.util.concurrent.atomic.AtomicBoolean; +import java.util.concurrent.atomic.AtomicLong; + +/** + * @author baldersheim + */ +public class MaxPendingContentChannelOutputStream extends ContentChannelOutputStream { + + private final long maxPending; + private final AtomicLong sent = new AtomicLong(0); + private final AtomicLong acked = new AtomicLong(0); + + public MaxPendingContentChannelOutputStream(ContentChannel endpoint, long maxPending) { + super(endpoint); + this.maxPending = maxPending; + } + + private long pendingBytes() { + return sent.get() - acked.get(); + } + + private class TrackCompletion implements CompletionHandler { + + private final long written; + private final AtomicBoolean replied = new AtomicBoolean(false); + + TrackCompletion(long written) { + this.written = written; + sent.addAndGet(written); + } + + @Override + public void completed() { + if (!replied.getAndSet(true)) { + acked.addAndGet(written); + } + } + + @Override + public void failed(Throwable t) { + if (!replied.getAndSet(true)) { + acked.addAndGet(written); + } + } + + } + + @Override + public void send(ByteBuffer src) throws IOException { + try { + stallWhilePendingAbove(maxPending); + } + catch (InterruptedException ignored) { + throw new InterruptedIOException("Interrupted waiting for IO"); + } + CompletionHandler pendingTracker = new TrackCompletion(src.remaining()); + try { + send(src, pendingTracker); + } + catch (Throwable throwable) { + pendingTracker.failed(throwable); + throw throwable; + } + } + + private void stallWhilePendingAbove(long pending) throws InterruptedException { + while (pendingBytes() > pending) { + Thread.sleep(1); + } + } + + @Override + public void flush() throws IOException { + super.flush(); + try { + stallWhilePendingAbove(0); + } + catch (InterruptedException e) { + throw new InterruptedIOException("Interrupted waiting for IO"); + } + } + +} diff --git a/container-core/src/main/java/com/yahoo/container/jdisc/ThreadedHttpRequestHandler.java b/container-core/src/main/java/com/yahoo/container/jdisc/ThreadedHttpRequestHandler.java index 0bfe4afe07d..0c3c1e2120b 100644 --- a/container-core/src/main/java/com/yahoo/container/jdisc/ThreadedHttpRequestHandler.java +++ b/container-core/src/main/java/com/yahoo/container/jdisc/ThreadedHttpRequestHandler.java @@ -10,9 +10,6 @@ import com.yahoo.jdisc.handler.ContentChannel; import com.yahoo.jdisc.handler.UnsafeContentInputStream; import com.yahoo.jdisc.handler.ResponseHandler; -import java.io.InterruptedIOException; -import java.util.concurrent.atomic.AtomicBoolean; -import java.util.concurrent.atomic.AtomicLong; import java.util.logging.Level; import java.io.IOException; @@ -253,81 +250,4 @@ public abstract class ThreadedHttpRequestHandler extends ThreadedRequestHandler } - /** - * @author baldersheim - */ - static class MaxPendingContentChannelOutputStream extends ContentChannelOutputStream { - private final long maxPending; - private final AtomicLong sent = new AtomicLong(0); - private final AtomicLong acked = new AtomicLong(0); - - public MaxPendingContentChannelOutputStream(ContentChannel endpoint, long maxPending) { - super(endpoint); - this.maxPending = maxPending; - } - - private long pendingBytes() { - return sent.get() - acked.get(); - } - - private class TrackCompletion implements CompletionHandler { - - private final long written; - private final AtomicBoolean replied = new AtomicBoolean(false); - - TrackCompletion(long written) { - this.written = written; - sent.addAndGet(written); - } - - @Override - public void completed() { - if ( ! replied.getAndSet(true)) { - acked.addAndGet(written); - } - } - - @Override - public void failed(Throwable t) { - if ( ! replied.getAndSet(true)) { - acked.addAndGet(written); - } - } - } - - @Override - public void send(ByteBuffer src) throws IOException { - try { - stallWhilePendingAbove(maxPending); - } catch (InterruptedException ignored) { - throw new InterruptedIOException("Interrupted waiting for IO"); - } - CompletionHandler pendingTracker = new TrackCompletion(src.remaining()); - try { - send(src, pendingTracker); - } catch (Throwable throwable) { - pendingTracker.failed(throwable); - throw throwable; - } - } - - private void stallWhilePendingAbove(long pending) throws InterruptedException { - while (pendingBytes() > pending) { - Thread.sleep(1); - } - } - - @Override - public void flush() throws IOException { - super.flush(); - try { - stallWhilePendingAbove(0); - } - catch (InterruptedException e) { - throw new InterruptedIOException("Interrupted waiting for IO"); - } - } - - } - } diff --git a/container-core/src/test/java/com/yahoo/container/jdisc/ThreadedRequestHandlerTestCase.java b/container-core/src/test/java/com/yahoo/container/jdisc/ThreadedRequestHandlerTestCase.java index d1036ce0e45..a9b16799aea 100644 --- a/container-core/src/test/java/com/yahoo/container/jdisc/ThreadedRequestHandlerTestCase.java +++ b/container-core/src/test/java/com/yahoo/container/jdisc/ThreadedRequestHandlerTestCase.java @@ -1,7 +1,6 @@ // Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. package com.yahoo.container.jdisc; -import com.yahoo.container.jdisc.ThreadedHttpRequestHandler.MaxPendingContentChannelOutputStream; import com.yahoo.jdisc.Request; import com.yahoo.jdisc.Response; import com.yahoo.jdisc.application.ContainerBuilder; @@ -20,11 +19,9 @@ import java.util.concurrent.Executor; import java.util.concurrent.ExecutorService; import java.util.concurrent.Executors; import java.util.concurrent.Future; -import java.util.concurrent.Phaser; import java.util.concurrent.RejectedExecutionException; import java.util.concurrent.TimeUnit; import java.util.concurrent.TimeoutException; -import java.util.concurrent.atomic.AtomicInteger; import static org.junit.Assert.*; diff --git a/container-core/src/test/java/com/yahoo/restapi/RestApiImplTest.java b/container-core/src/test/java/com/yahoo/restapi/RestApiImplTest.java index 151b17a921f..1b03d87f405 100644 --- a/container-core/src/test/java/com/yahoo/restapi/RestApiImplTest.java +++ b/container-core/src/test/java/com/yahoo/restapi/RestApiImplTest.java @@ -1,3 +1,4 @@ +// Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. package com.yahoo.restapi;// Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. import com.fasterxml.jackson.annotation.JsonProperty; @@ -174,4 +175,4 @@ class RestApiImplTest { @JsonProperty("mystring") public String stringValue; @JsonProperty("myinstant") public Instant instantValue; } -}
\ No newline at end of file +} diff --git a/container-search/src/main/java/com/yahoo/prelude/fastsearch/Base64DataField.java b/container-search/src/main/java/com/yahoo/prelude/fastsearch/Base64DataField.java index d51bdc0fad1..18868321d78 100644 --- a/container-search/src/main/java/com/yahoo/prelude/fastsearch/Base64DataField.java +++ b/container-search/src/main/java/com/yahoo/prelude/fastsearch/Base64DataField.java @@ -1,3 +1,4 @@ +// Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. package com.yahoo.prelude.fastsearch; import com.yahoo.data.access.Inspector; diff --git a/container-search/src/main/java/com/yahoo/prelude/hitfield/RawBase64.java b/container-search/src/main/java/com/yahoo/prelude/hitfield/RawBase64.java index 134d0bc902a..87b073b98c5 100644 --- a/container-search/src/main/java/com/yahoo/prelude/hitfield/RawBase64.java +++ b/container-search/src/main/java/com/yahoo/prelude/hitfield/RawBase64.java @@ -1,3 +1,4 @@ +// Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. package com.yahoo.prelude.hitfield; import java.util.Base64; diff --git a/controller-api/src/main/java/com/yahoo/vespa/hosted/controller/api/integration/certificates/EndpointCertificateRequestMetadata.java b/controller-api/src/main/java/com/yahoo/vespa/hosted/controller/api/integration/certificates/EndpointCertificateRequestMetadata.java index 81e04190244..339389105e6 100644 --- a/controller-api/src/main/java/com/yahoo/vespa/hosted/controller/api/integration/certificates/EndpointCertificateRequestMetadata.java +++ b/controller-api/src/main/java/com/yahoo/vespa/hosted/controller/api/integration/certificates/EndpointCertificateRequestMetadata.java @@ -1,4 +1,4 @@ -// Copyright Verizon Media. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. +// Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. package com.yahoo.vespa.hosted.controller.api.integration.certificates; import java.util.List; diff --git a/controller-server/src/main/java/com/yahoo/vespa/hosted/controller/archive/CuratorArchiveBucketDb.java b/controller-server/src/main/java/com/yahoo/vespa/hosted/controller/archive/CuratorArchiveBucketDb.java index ce5869af1a0..8b929cb9d10 100644 --- a/controller-server/src/main/java/com/yahoo/vespa/hosted/controller/archive/CuratorArchiveBucketDb.java +++ b/controller-server/src/main/java/com/yahoo/vespa/hosted/controller/archive/CuratorArchiveBucketDb.java @@ -1,4 +1,4 @@ -// Copyright 2021 Oath Inc. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. +// Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. package com.yahoo.vespa.hosted.controller.archive; import com.yahoo.config.provision.SystemName; diff --git a/controller-server/src/main/java/com/yahoo/vespa/hosted/controller/maintenance/ArchiveAccessMaintainer.java b/controller-server/src/main/java/com/yahoo/vespa/hosted/controller/maintenance/ArchiveAccessMaintainer.java index 1597162bc80..8156a2e9f3b 100644 --- a/controller-server/src/main/java/com/yahoo/vespa/hosted/controller/maintenance/ArchiveAccessMaintainer.java +++ b/controller-server/src/main/java/com/yahoo/vespa/hosted/controller/maintenance/ArchiveAccessMaintainer.java @@ -1,4 +1,4 @@ -// Copyright 2021 Oath Inc. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root +// Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. package com.yahoo.vespa.hosted.controller.maintenance; import com.google.common.collect.Maps; diff --git a/controller-server/src/main/java/com/yahoo/vespa/hosted/controller/restapi/billing/BillingApiHandlerV2.java b/controller-server/src/main/java/com/yahoo/vespa/hosted/controller/restapi/billing/BillingApiHandlerV2.java index 58afed4143f..af4869ef197 100644 --- a/controller-server/src/main/java/com/yahoo/vespa/hosted/controller/restapi/billing/BillingApiHandlerV2.java +++ b/controller-server/src/main/java/com/yahoo/vespa/hosted/controller/restapi/billing/BillingApiHandlerV2.java @@ -1,3 +1,4 @@ +// Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. package com.yahoo.vespa.hosted.controller.restapi.billing; import com.yahoo.config.provision.TenantName; diff --git a/controller-server/src/main/java/com/yahoo/vespa/hosted/controller/restapi/billing/CsvResponse.java b/controller-server/src/main/java/com/yahoo/vespa/hosted/controller/restapi/billing/CsvResponse.java index 5aa993f2727..3643626761d 100644 --- a/controller-server/src/main/java/com/yahoo/vespa/hosted/controller/restapi/billing/CsvResponse.java +++ b/controller-server/src/main/java/com/yahoo/vespa/hosted/controller/restapi/billing/CsvResponse.java @@ -1,3 +1,4 @@ +// Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. package com.yahoo.vespa.hosted.controller.restapi.billing; import com.yahoo.container.jdisc.HttpResponse; diff --git a/controller-server/src/test/java/com/yahoo/vespa/hosted/controller/archive/CuratorArchiveBucketDbTest.java b/controller-server/src/test/java/com/yahoo/vespa/hosted/controller/archive/CuratorArchiveBucketDbTest.java index ab669d6fe14..d6ec01a2d57 100644 --- a/controller-server/src/test/java/com/yahoo/vespa/hosted/controller/archive/CuratorArchiveBucketDbTest.java +++ b/controller-server/src/test/java/com/yahoo/vespa/hosted/controller/archive/CuratorArchiveBucketDbTest.java @@ -1,3 +1,4 @@ +// Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. package com.yahoo.vespa.hosted.controller.archive; import com.yahoo.config.provision.SystemName; @@ -53,4 +54,4 @@ public class CuratorArchiveBucketDbTest { Set.of(new ArchiveBucket("bucketName", "keyArn").withTenant(TenantName.from("firstInZone"))), bucketDb.buckets(ZoneId.from("prod.us-east-3"))); } -}
\ No newline at end of file +} diff --git a/controller-server/src/test/java/com/yahoo/vespa/hosted/controller/maintenance/UserManagementMaintainerTest.java b/controller-server/src/test/java/com/yahoo/vespa/hosted/controller/maintenance/UserManagementMaintainerTest.java index b658e86a575..52cb3ce121f 100644 --- a/controller-server/src/test/java/com/yahoo/vespa/hosted/controller/maintenance/UserManagementMaintainerTest.java +++ b/controller-server/src/test/java/com/yahoo/vespa/hosted/controller/maintenance/UserManagementMaintainerTest.java @@ -1,3 +1,4 @@ +// Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. package com.yahoo.vespa.hosted.controller.maintenance; import com.yahoo.config.provision.ApplicationName; @@ -43,4 +44,4 @@ public class UserManagementMaintainerTest { assertTrue(expectedRoles.containsAll(actualRoles) && actualRoles.containsAll(expectedRoles)); } -}
\ No newline at end of file +} diff --git a/controller-server/src/test/java/com/yahoo/vespa/hosted/controller/maintenance/VcmrMaintainerTest.java b/controller-server/src/test/java/com/yahoo/vespa/hosted/controller/maintenance/VcmrMaintainerTest.java index a2da6e357b6..581d10d4f88 100644 --- a/controller-server/src/test/java/com/yahoo/vespa/hosted/controller/maintenance/VcmrMaintainerTest.java +++ b/controller-server/src/test/java/com/yahoo/vespa/hosted/controller/maintenance/VcmrMaintainerTest.java @@ -1,3 +1,4 @@ +// Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. package com.yahoo.vespa.hosted.controller.maintenance; import com.yahoo.config.provision.HostName; diff --git a/controller-server/src/test/java/com/yahoo/vespa/hosted/controller/persistence/ArchiveBucketsSerializerTest.java b/controller-server/src/test/java/com/yahoo/vespa/hosted/controller/persistence/ArchiveBucketsSerializerTest.java index 17814b12a09..a3e4f6e6f89 100644 --- a/controller-server/src/test/java/com/yahoo/vespa/hosted/controller/persistence/ArchiveBucketsSerializerTest.java +++ b/controller-server/src/test/java/com/yahoo/vespa/hosted/controller/persistence/ArchiveBucketsSerializerTest.java @@ -1,3 +1,4 @@ +// Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. package com.yahoo.vespa.hosted.controller.persistence; import com.yahoo.config.provision.TenantName; diff --git a/controller-server/src/test/java/com/yahoo/vespa/hosted/controller/persistence/ChangeRequestSerializerTest.java b/controller-server/src/test/java/com/yahoo/vespa/hosted/controller/persistence/ChangeRequestSerializerTest.java index 40a045c44cf..29a722be76b 100644 --- a/controller-server/src/test/java/com/yahoo/vespa/hosted/controller/persistence/ChangeRequestSerializerTest.java +++ b/controller-server/src/test/java/com/yahoo/vespa/hosted/controller/persistence/ChangeRequestSerializerTest.java @@ -1,3 +1,4 @@ +// Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. package com.yahoo.vespa.hosted.controller.persistence; import com.yahoo.config.provision.zone.ZoneId; @@ -42,4 +43,4 @@ public class ChangeRequestSerializerTest { assertEquals(changeRequest, reserialized); } -}
\ No newline at end of file +} diff --git a/controller-server/src/test/java/com/yahoo/vespa/hosted/controller/restapi/billing/BillingApiHandlerV2Test.java b/controller-server/src/test/java/com/yahoo/vespa/hosted/controller/restapi/billing/BillingApiHandlerV2Test.java index e733f8e27d6..5cbef8aaba0 100644 --- a/controller-server/src/test/java/com/yahoo/vespa/hosted/controller/restapi/billing/BillingApiHandlerV2Test.java +++ b/controller-server/src/test/java/com/yahoo/vespa/hosted/controller/restapi/billing/BillingApiHandlerV2Test.java @@ -1,3 +1,4 @@ +// Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. package com.yahoo.vespa.hosted.controller.restapi.billing; import com.yahoo.application.container.handler.Request; diff --git a/controller-server/src/test/java/com/yahoo/vespa/hosted/controller/restapi/horizon/HorizonApiTest.java b/controller-server/src/test/java/com/yahoo/vespa/hosted/controller/restapi/horizon/HorizonApiTest.java index b2b5b2286f7..96b08006790 100644 --- a/controller-server/src/test/java/com/yahoo/vespa/hosted/controller/restapi/horizon/HorizonApiTest.java +++ b/controller-server/src/test/java/com/yahoo/vespa/hosted/controller/restapi/horizon/HorizonApiTest.java @@ -1,3 +1,4 @@ +// Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. package com.yahoo.vespa.hosted.controller.restapi.horizon; import com.yahoo.config.provision.SystemName; @@ -61,4 +62,4 @@ public class HorizonApiTest extends ControllerContainerCloudTest { " </filtering>\n" + " </http>\n"; } -}
\ No newline at end of file +} diff --git a/controller-server/src/test/java/com/yahoo/vespa/hosted/controller/restapi/horizon/TsdbQueryRewriterTest.java b/controller-server/src/test/java/com/yahoo/vespa/hosted/controller/restapi/horizon/TsdbQueryRewriterTest.java index d31d9c28c6c..37150b7e6f0 100644 --- a/controller-server/src/test/java/com/yahoo/vespa/hosted/controller/restapi/horizon/TsdbQueryRewriterTest.java +++ b/controller-server/src/test/java/com/yahoo/vespa/hosted/controller/restapi/horizon/TsdbQueryRewriterTest.java @@ -1,3 +1,4 @@ +// Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. package com.yahoo.vespa.hosted.controller.restapi.horizon; import com.yahoo.config.provision.SystemName; @@ -51,4 +52,4 @@ public class TsdbQueryRewriterTest { assertEquals(expectedJson, baos.toString()); } -}
\ No newline at end of file +} diff --git a/document/src/main/java/com/yahoo/document/Field.java b/document/src/main/java/com/yahoo/document/Field.java index 58721c08d37..a45afea6747 100644 --- a/document/src/main/java/com/yahoo/document/Field.java +++ b/document/src/main/java/com/yahoo/document/Field.java @@ -222,6 +222,7 @@ public class Field extends FieldBase implements FieldSet, Comparable, Serializab return getId(); } + @Override public String toString() { return super.toString() + "(" + dataType + ")"; } diff --git a/document/src/main/java/com/yahoo/document/annotation/AnnotationTypeRegistry.java b/document/src/main/java/com/yahoo/document/annotation/AnnotationTypeRegistry.java index dbd77fe7944..d370a26d9c0 100644 --- a/document/src/main/java/com/yahoo/document/annotation/AnnotationTypeRegistry.java +++ b/document/src/main/java/com/yahoo/document/annotation/AnnotationTypeRegistry.java @@ -6,20 +6,22 @@ import java.util.HashMap; import java.util.Map; /** - * A registry of annotation types. This can be set up programmatically or from config. + * A registry of annotation types. This can be set up programmatically or from config. * - * @author <a href="mailto:einarmr@yahoo-inc.com">Einar M R Rosenvinge</a> + * @author Einar M R Rosenvinge */ public class AnnotationTypeRegistry { - private Map<Integer, AnnotationType> idMap = new HashMap<Integer, AnnotationType>(); - private Map<String, AnnotationType> nameMap = new HashMap<String, AnnotationType>(); + + private final Map<Integer, AnnotationType> idMap = new HashMap<>(); + private final Map<String, AnnotationType> nameMap = new HashMap<>(); /** Creates a new empty registry. */ public AnnotationTypeRegistry() { } /** - * Register a new annotation type. WARNING! Only to be used by the configuration system and in unit tests. Not to be used in production code. + * Register a new annotation type. + * WARNING! Only to be used by the configuration system and in unit tests. Not to be used in production code. * * @param type the type to register * @throws IllegalArgumentException if a type is already registered with this name or this id, and it is non-equal to the argument. @@ -39,9 +41,10 @@ public class AnnotationTypeRegistry { } /** - * Unregisters the type given by the argument. WARNING! Only to be used by the configuration system and in unit tests. Not to be used in production code. + * Unregisters the type given by the argument. + * WARNING! Only to be used by the configuration system and in unit tests. Not to be used in production code. * - * @param name the name of the type to unregister. + * @param name the name of the type to unregister * @return true if the type was successfully unregistered, false otherwise (it was not present) */ public boolean unregister(String name) { @@ -54,9 +57,10 @@ public class AnnotationTypeRegistry { } /** - * Unregisters the type given by the argument. WARNING! Only to be used by the configuration system and in unit tests. Not to be used in production code. + * Unregisters the type given by the argument. + * WARNING! Only to be used by the configuration system and in unit tests. Not to be used in production code. * - * @param id the id of the type to unregister. + * @param id the id of the type to unregister * @return true if the type was successfully unregistered, false otherwise (it was not present) */ public boolean unregister(int id) { @@ -69,11 +73,13 @@ public class AnnotationTypeRegistry { } /** - * Unregisters the type given by the argument. WARNING! Only to be used by the configuration system and in unit tests. Not to be used in production code. + * Unregisters the type given by the argument. + * WARNING! Only to be used by the configuration system and in unit tests. Not to be used in production code. * - * @param type the AnnotationType to unregister. + * @param type the AnnotationType to unregister * @return true if the type was successfully unregistered, false otherwise (it was not present) - * @throws IllegalArgumentException if the ID and name of this annotation type are present, but they do not belong together. + * @throws IllegalArgumentException if the ID and name of this annotation type are present, + * but they do not belong together. */ public boolean unregister(AnnotationType type) { if (idMap.containsKey(type.getId()) && nameMap.containsKey(type.getName())) { @@ -85,11 +91,12 @@ public class AnnotationTypeRegistry { idMap.remove(type.getId()); nameMap.remove(type.getName()); } else { - throw new IllegalArgumentException("The ID and name of this annotation type are present, but they do not belong together. Not removing type."); + throw new IllegalArgumentException("The ID and name of this annotation type are present, " + + "but they do not belong together. Not removing type."); } return true; } - //it's not there, but that's no problem + // It's not there, but that's no problem return false; } @@ -113,11 +120,7 @@ public class AnnotationTypeRegistry { return idMap.get(id); } - /** - * Returns an <strong>unmodifiable</strong> {@link Map} of all types registered. - * - * @return an unmodifiable {@link Map} of all types registered. - */ + /** Returns an unmodifiable of all types registered. */ public Map<String, AnnotationType> getTypes() { return Collections.unmodifiableMap(nameMap); } @@ -127,4 +130,5 @@ public class AnnotationTypeRegistry { idMap.clear(); nameMap.clear(); } + } diff --git a/eval/src/tests/instruction/dense_hamming_distance/CMakeLists.txt b/eval/src/tests/instruction/dense_hamming_distance/CMakeLists.txt index 3d18f9613b3..dc45e8ca6df 100644 --- a/eval/src/tests/instruction/dense_hamming_distance/CMakeLists.txt +++ b/eval/src/tests/instruction/dense_hamming_distance/CMakeLists.txt @@ -1,3 +1,4 @@ +# Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. vespa_add_executable(eval_dense_hamming_distance_test_app TEST SOURCES diff --git a/hosted-zone-api/src/main/java/ai/vespa/cloud/Cluster.java b/hosted-zone-api/src/main/java/ai/vespa/cloud/Cluster.java index dba0e3f30bc..218545383e6 100644 --- a/hosted-zone-api/src/main/java/ai/vespa/cloud/Cluster.java +++ b/hosted-zone-api/src/main/java/ai/vespa/cloud/Cluster.java @@ -1,3 +1,4 @@ +// Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. package ai.vespa.cloud; import java.util.Collections; diff --git a/hosted-zone-api/src/main/java/ai/vespa/cloud/Node.java b/hosted-zone-api/src/main/java/ai/vespa/cloud/Node.java index 19ef2757b6c..f745bc29f1e 100644 --- a/hosted-zone-api/src/main/java/ai/vespa/cloud/Node.java +++ b/hosted-zone-api/src/main/java/ai/vespa/cloud/Node.java @@ -1,3 +1,4 @@ +// Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. package ai.vespa.cloud; import java.util.Objects; diff --git a/jdisc_core/src/main/java/com/yahoo/jdisc/refcount/CloseableOnce.java b/jdisc_core/src/main/java/com/yahoo/jdisc/refcount/CloseableOnce.java index f5b41c5ee9f..a542ba448e8 100644 --- a/jdisc_core/src/main/java/com/yahoo/jdisc/refcount/CloseableOnce.java +++ b/jdisc_core/src/main/java/com/yahoo/jdisc/refcount/CloseableOnce.java @@ -1,3 +1,4 @@ +// Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. package com.yahoo.jdisc.refcount; import com.yahoo.jdisc.ResourceReference; diff --git a/linguistics-components/src/main/java/com/yahoo/language/sentencepiece/package-info.java b/linguistics-components/src/main/java/com/yahoo/language/sentencepiece/package-info.java index 3f97277c489..4a8673705ec 100644 --- a/linguistics-components/src/main/java/com/yahoo/language/sentencepiece/package-info.java +++ b/linguistics-components/src/main/java/com/yahoo/language/sentencepiece/package-info.java @@ -1,4 +1,4 @@ -// Copyright 2021 Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. +// Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. @ExportPackage @PublicApi package com.yahoo.language.sentencepiece; diff --git a/messagebus/src/main/java/com/yahoo/messagebus/network/rpc/NamedRPCService.java b/messagebus/src/main/java/com/yahoo/messagebus/network/rpc/NamedRPCService.java index 59cafed1836..0c2e4795a01 100644 --- a/messagebus/src/main/java/com/yahoo/messagebus/network/rpc/NamedRPCService.java +++ b/messagebus/src/main/java/com/yahoo/messagebus/network/rpc/NamedRPCService.java @@ -1,3 +1,4 @@ +// Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. package com.yahoo.messagebus.network.rpc; import com.yahoo.jrt.slobrok.api.IMirror; diff --git a/messagebus/src/main/java/com/yahoo/messagebus/network/rpc/TcpRPCService.java b/messagebus/src/main/java/com/yahoo/messagebus/network/rpc/TcpRPCService.java index e2fae59b429..fc19b786f0d 100644 --- a/messagebus/src/main/java/com/yahoo/messagebus/network/rpc/TcpRPCService.java +++ b/messagebus/src/main/java/com/yahoo/messagebus/network/rpc/TcpRPCService.java @@ -1,3 +1,4 @@ +// Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. package com.yahoo.messagebus.network.rpc; public class TcpRPCService implements RPCService { diff --git a/metrics-proxy/src/main/java/ai/vespa/metricsproxy/http/yamas/YamasResponse.java b/metrics-proxy/src/main/java/ai/vespa/metricsproxy/http/yamas/YamasResponse.java index deeabb59cbe..23b56e94ae4 100644 --- a/metrics-proxy/src/main/java/ai/vespa/metricsproxy/http/yamas/YamasResponse.java +++ b/metrics-proxy/src/main/java/ai/vespa/metricsproxy/http/yamas/YamasResponse.java @@ -1,3 +1,4 @@ +// Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. package ai.vespa.metricsproxy.http.yamas; import ai.vespa.metricsproxy.metric.model.json.JacksonUtil; diff --git a/model-evaluation/src/test/java/ai/vespa/models/evaluation/RankProfilesConfigImporterWithMockedConstants.java b/model-evaluation/src/test/java/ai/vespa/models/evaluation/RankProfilesConfigImporterWithMockedConstants.java index 250ae9a4883..c11f4764678 100644 --- a/model-evaluation/src/test/java/ai/vespa/models/evaluation/RankProfilesConfigImporterWithMockedConstants.java +++ b/model-evaluation/src/test/java/ai/vespa/models/evaluation/RankProfilesConfigImporterWithMockedConstants.java @@ -1,3 +1,4 @@ +// Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. package ai.vespa.models.evaluation; import com.yahoo.config.FileReference; diff --git a/model-integration/src/main/java/ai/vespa/rankingexpression/importer/onnx/ImportedOnnxModel.java b/model-integration/src/main/java/ai/vespa/rankingexpression/importer/onnx/ImportedOnnxModel.java index 714321ec116..b5dbe0b9ef9 100644 --- a/model-integration/src/main/java/ai/vespa/rankingexpression/importer/onnx/ImportedOnnxModel.java +++ b/model-integration/src/main/java/ai/vespa/rankingexpression/importer/onnx/ImportedOnnxModel.java @@ -1,3 +1,4 @@ +// Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. package ai.vespa.rankingexpression.importer.onnx; import ai.vespa.rankingexpression.importer.ImportedModel; diff --git a/node-admin/src/main/java/com/yahoo/vespa/hosted/node/admin/maintenance/identity/AthenzCredentialsMaintainer.java b/node-admin/src/main/java/com/yahoo/vespa/hosted/node/admin/maintenance/identity/AthenzCredentialsMaintainer.java index 280e58c91f1..6fef2cbbb50 100644 --- a/node-admin/src/main/java/com/yahoo/vespa/hosted/node/admin/maintenance/identity/AthenzCredentialsMaintainer.java +++ b/node-admin/src/main/java/com/yahoo/vespa/hosted/node/admin/maintenance/identity/AthenzCredentialsMaintainer.java @@ -20,10 +20,11 @@ import com.yahoo.vespa.athenz.identityprovider.client.CsrGenerator; import com.yahoo.vespa.athenz.identityprovider.client.DefaultIdentityDocumentClient; import com.yahoo.vespa.athenz.tls.AthenzIdentityVerifier; import com.yahoo.vespa.athenz.utils.SiaUtils; -import com.yahoo.vespa.hosted.node.admin.container.ContainerName; import com.yahoo.vespa.hosted.node.admin.component.ConfigServerInfo; +import com.yahoo.vespa.hosted.node.admin.container.ContainerName; import com.yahoo.vespa.hosted.node.admin.nodeagent.NodeAgentContext; import com.yahoo.vespa.hosted.node.admin.nodeagent.NodeAgentTask; +import com.yahoo.vespa.hosted.node.admin.nodeagent.VespaUser; import com.yahoo.vespa.hosted.node.admin.task.util.file.FileFinder; import com.yahoo.vespa.hosted.node.admin.task.util.file.UnixPath; import com.yahoo.vespa.hosted.node.admin.task.util.fs.ContainerPath; @@ -206,7 +207,7 @@ public class AthenzCredentialsMaintainer implements CredentialsMaintainer { EntityBindingsMapper.toAttestationData(signedIdentityDocument), csr); EntityBindingsMapper.writeSignedIdentityDocumentToFile(identityDocumentFile, signedIdentityDocument); - writePrivateKeyAndCertificate(context.userNamespace().vespaUserId(), + writePrivateKeyAndCertificate(context.vespaUser(), privateKeyFile, keyPair.getPrivate(), certificateFile, instanceIdentity.certificate()); context.log(logger, "Instance successfully registered and credentials written to file"); } @@ -234,7 +235,7 @@ public class AthenzCredentialsMaintainer implements CredentialsMaintainer { context.identity(), identityDocument.providerUniqueId().asDottedString(), csr); - writePrivateKeyAndCertificate(context.userNamespace().vespaUserId(), + writePrivateKeyAndCertificate(context.vespaUser(), privateKeyFile, keyPair.getPrivate(), certificateFile, instanceIdentity.certificate()); context.log(logger, "Instance successfully refreshed and credentials written to file"); } catch (ZtsClientException e) { @@ -251,19 +252,19 @@ public class AthenzCredentialsMaintainer implements CredentialsMaintainer { } - private static void writePrivateKeyAndCertificate(int vespaUid, + private static void writePrivateKeyAndCertificate(VespaUser vespaUser, ContainerPath privateKeyFile, PrivateKey privateKey, ContainerPath certificateFile, X509Certificate certificate) { - writeFile(privateKeyFile, vespaUid, KeyUtils.toPem(privateKey)); - writeFile(certificateFile, vespaUid, X509CertificateUtils.toPem(certificate)); + writeFile(privateKeyFile, vespaUser, KeyUtils.toPem(privateKey)); + writeFile(certificateFile, vespaUser, X509CertificateUtils.toPem(certificate)); } - private static void writeFile(ContainerPath path, int vespaUid, String utf8Content) { + private static void writeFile(ContainerPath path, VespaUser vespaUser, String utf8Content) { new UnixPath(path.resolveSibling(path.getFileName() + ".tmp")) .writeUtf8File(utf8Content, "r--------") - .setOwnerId(vespaUid) + .setOwnerId(vespaUser.uid()) .atomicMove(path); } diff --git a/node-admin/src/main/java/com/yahoo/vespa/hosted/node/admin/maintenance/servicedump/VespaServiceDumperImpl.java b/node-admin/src/main/java/com/yahoo/vespa/hosted/node/admin/maintenance/servicedump/VespaServiceDumperImpl.java index 0a9496be0a6..b299e1f3f0d 100644 --- a/node-admin/src/main/java/com/yahoo/vespa/hosted/node/admin/maintenance/servicedump/VespaServiceDumperImpl.java +++ b/node-admin/src/main/java/com/yahoo/vespa/hosted/node/admin/maintenance/servicedump/VespaServiceDumperImpl.java @@ -101,8 +101,8 @@ public class VespaServiceDumperImpl implements VespaServiceDumper { } context.log(log, Level.INFO, "Creating '" + unixPathDirectory +"'."); unixPathDirectory.createDirectory("rwxr-x---") - .setOwner(context.userNamespace().vespaUser()) - .setGroup(context.userNamespace().vespaGroup()); + .setOwner(context.vespaUser().name()) + .setGroup(context.vespaUser().group()); URI destination = serviceDumpDestination(nodeSpec, createDumpId(request)); ProducerContext producerCtx = new ProducerContext(context, directory, request); List<Artifact> producedArtifacts = new ArrayList<>(); diff --git a/node-admin/src/main/java/com/yahoo/vespa/hosted/node/admin/nodeadmin/NodeAdminStateUpdater.java b/node-admin/src/main/java/com/yahoo/vespa/hosted/node/admin/nodeadmin/NodeAdminStateUpdater.java index 53c9e741f59..dda404797d9 100644 --- a/node-admin/src/main/java/com/yahoo/vespa/hosted/node/admin/nodeadmin/NodeAdminStateUpdater.java +++ b/node-admin/src/main/java/com/yahoo/vespa/hosted/node/admin/nodeadmin/NodeAdminStateUpdater.java @@ -3,7 +3,6 @@ package com.yahoo.vespa.hosted.node.admin.nodeadmin; import com.yahoo.concurrent.ThreadFactoryFactory; import com.yahoo.config.provision.HostName; -import com.yahoo.vespa.flags.FlagSource; import com.yahoo.vespa.hosted.node.admin.configserver.noderepository.Acl; import com.yahoo.vespa.hosted.node.admin.configserver.noderepository.NodeRepository; import com.yahoo.vespa.hosted.node.admin.configserver.noderepository.NodeSpec; @@ -13,7 +12,6 @@ import com.yahoo.vespa.hosted.node.admin.nodeagent.NodeAgentContext; import com.yahoo.vespa.hosted.node.admin.nodeagent.NodeAgentContextFactory; import com.yahoo.yolean.Exceptions; -import java.time.Clock; import java.time.Duration; import java.util.ArrayList; import java.util.EnumSet; @@ -59,9 +57,7 @@ public class NodeAdminStateUpdater { NodeRepository nodeRepository, Orchestrator orchestrator, NodeAdmin nodeAdmin, - HostName hostHostname, - Clock clock, - FlagSource flagSource) { + HostName hostHostname) { this.nodeAgentContextFactory = nodeAgentContextFactory; this.nodeRepository = nodeRepository; this.orchestrator = orchestrator; diff --git a/node-admin/src/main/java/com/yahoo/vespa/hosted/node/admin/nodeagent/NodeAgentContext.java b/node-admin/src/main/java/com/yahoo/vespa/hosted/node/admin/nodeagent/NodeAgentContext.java index 8cf8553bc34..d26a1fa6019 100644 --- a/node-admin/src/main/java/com/yahoo/vespa/hosted/node/admin/nodeagent/NodeAgentContext.java +++ b/node-admin/src/main/java/com/yahoo/vespa/hosted/node/admin/nodeagent/NodeAgentContext.java @@ -6,10 +6,10 @@ import com.yahoo.config.provision.HostName; import com.yahoo.config.provision.NodeType; import com.yahoo.config.provision.zone.ZoneApi; import com.yahoo.vespa.athenz.api.AthenzIdentity; -import com.yahoo.vespa.hosted.node.admin.container.ContainerName; import com.yahoo.vespa.hosted.node.admin.component.TaskContext; import com.yahoo.vespa.hosted.node.admin.configserver.noderepository.Acl; import com.yahoo.vespa.hosted.node.admin.configserver.noderepository.NodeSpec; +import com.yahoo.vespa.hosted.node.admin.container.ContainerName; import com.yahoo.vespa.hosted.node.admin.container.ContainerNetworkMode; import com.yahoo.vespa.hosted.node.admin.task.util.fs.ContainerPath; @@ -42,6 +42,9 @@ public interface NodeAgentContext extends TaskContext { ZoneApi zone(); + /** @return information about the Vespa user inside the container */ + VespaUser vespaUser(); + UserNamespace userNamespace(); default boolean isDisabled(NodeAgentTask task) { diff --git a/node-admin/src/main/java/com/yahoo/vespa/hosted/node/admin/nodeagent/NodeAgentContextImpl.java b/node-admin/src/main/java/com/yahoo/vespa/hosted/node/admin/nodeagent/NodeAgentContextImpl.java index 9bcf5d58d6e..037bbc56d1d 100644 --- a/node-admin/src/main/java/com/yahoo/vespa/hosted/node/admin/nodeagent/NodeAgentContextImpl.java +++ b/node-admin/src/main/java/com/yahoo/vespa/hosted/node/admin/nodeagent/NodeAgentContextImpl.java @@ -12,9 +12,9 @@ import com.yahoo.vespa.flags.FetchVector; import com.yahoo.vespa.flags.FlagSource; import com.yahoo.vespa.flags.InMemoryFlagSource; import com.yahoo.vespa.flags.PermanentFlags; -import com.yahoo.vespa.hosted.node.admin.container.ContainerName; import com.yahoo.vespa.hosted.node.admin.configserver.noderepository.Acl; import com.yahoo.vespa.hosted.node.admin.configserver.noderepository.NodeSpec; +import com.yahoo.vespa.hosted.node.admin.container.ContainerName; import com.yahoo.vespa.hosted.node.admin.container.ContainerNetworkMode; import com.yahoo.vespa.hosted.node.admin.task.util.fs.ContainerFileSystem; import com.yahoo.vespa.hosted.node.admin.task.util.fs.ContainerPath; @@ -42,16 +42,14 @@ public class NodeAgentContextImpl implements NodeAgentContext { private final ZoneApi zone; private final ContainerFileSystem containerFs; private final ContainerPath pathToVespaHome; - private final UserNamespace userNamespace; private final double cpuSpeedup; private final Set<NodeAgentTask> disabledNodeAgentTasks; private final Optional<ApplicationId> hostExclusiveTo; public NodeAgentContextImpl(NodeSpec node, Acl acl, AthenzIdentity identity, ContainerNetworkMode containerNetworkMode, ZoneApi zone, - FlagSource flagSource, Path pathToContainerStorage, String pathToVespaHome, - UserNamespace userNamespace, double cpuSpeedup, - Optional<ApplicationId> hostExclusiveTo) { + FlagSource flagSource, ContainerFileSystem containerFs, String pathToVespaHome, + double cpuSpeedup, Optional<ApplicationId> hostExclusiveTo) { if (cpuSpeedup <= 0) throw new IllegalArgumentException("cpuSpeedUp must be positive, was: " + cpuSpeedup); @@ -61,10 +59,9 @@ public class NodeAgentContextImpl implements NodeAgentContext { this.identity = Objects.requireNonNull(identity); this.containerNetworkMode = Objects.requireNonNull(containerNetworkMode); this.zone = Objects.requireNonNull(zone); - this.containerFs = ContainerFileSystem.create(pathToContainerStorage.resolve(containerName.asString()), userNamespace); + this.containerFs = Objects.requireNonNull(containerFs); this.pathToVespaHome = containerFs.getPath(pathToVespaHome); this.logPrefix = containerName.asString() + ": "; - this.userNamespace = Objects.requireNonNull(userNamespace); this.cpuSpeedup = cpuSpeedup; this.disabledNodeAgentTasks = NodeAgentTask.fromString( PermanentFlags.DISABLED_HOST_ADMIN_TASKS.bindTo(flagSource).with(FetchVector.Dimension.HOSTNAME, node.hostname()).value()); @@ -102,8 +99,13 @@ public class NodeAgentContextImpl implements NodeAgentContext { } @Override + public VespaUser vespaUser() { + return containerFs.getUserPrincipalLookupService().vespaUser(); + } + + @Override public UserNamespace userNamespace() { - return userNamespace; + return containerFs.getUserPrincipalLookupService().userNamespace(); } @Override @@ -191,6 +193,7 @@ public class NodeAgentContextImpl implements NodeAgentContext { private ContainerNetworkMode containerNetworkMode; private ZoneApi zone; private UserNamespace userNamespace; + private VespaUser vespaUser; private Path containerStorage; private FlagSource flagSource; private double cpuSpeedUp = 1; @@ -230,6 +233,12 @@ public class NodeAgentContextImpl implements NodeAgentContext { return this; } + public Builder vespaUser(VespaUser vespaUser) { + this.vespaUser = vespaUser; + return this; + } + + /** Sets the file system to use for paths. */ public Builder fileSystem(FileSystem fileSystem) { return containerStorage(fileSystem.getPath(DEFAULT_CONTAINER_STORAGE.toString())); @@ -258,6 +267,14 @@ public class NodeAgentContextImpl implements NodeAgentContext { public NodeAgentContextImpl build() { Objects.requireNonNull(containerStorage, "Must set one of containerStorage or fileSystem"); + UserNamespace userNamespace = Optional.ofNullable(this.userNamespace) + .orElseGet(() -> new UserNamespace(100000, 100000, 100000)); + VespaUser vespaUser = Optional.ofNullable(this.vespaUser) + .orElseGet(() -> new VespaUser("vespa", "vespa", 1000, 100)); + ContainerFileSystem containerFs = ContainerFileSystem.create(containerStorage + .resolve(nodeSpecBuilder.hostname().split("\\.")[0]), userNamespace, vespaUser); + containerFs.createRoot(); + return new NodeAgentContextImpl( nodeSpecBuilder.build(), Optional.ofNullable(acl).orElse(Acl.EMPTY), @@ -285,9 +302,8 @@ public class NodeAgentContextImpl implements NodeAgentContext { } }), Optional.ofNullable(flagSource).orElseGet(InMemoryFlagSource::new), - containerStorage, + containerFs, "/opt/vespa", - Optional.ofNullable(userNamespace).orElseGet(() -> new UserNamespace(100000, 100000, "vespa", "vespa", 1000, 100)), cpuSpeedUp, hostExclusiveTo); } } diff --git a/node-admin/src/main/java/com/yahoo/vespa/hosted/node/admin/nodeagent/UserNamespace.java b/node-admin/src/main/java/com/yahoo/vespa/hosted/node/admin/nodeagent/UserNamespace.java index 1a25b5c3c5e..99529b83374 100644 --- a/node-admin/src/main/java/com/yahoo/vespa/hosted/node/admin/nodeagent/UserNamespace.java +++ b/node-admin/src/main/java/com/yahoo/vespa/hosted/node/admin/nodeagent/UserNamespace.java @@ -8,51 +8,66 @@ import java.util.Objects; */ public class UserNamespace { - /** Total number of UID/GID that are mapped for each container */ - private static final int ID_RANGE = 65_536; // 2^16 - /** * IDs outside the ID range are translated to the overflow ID before being written to disk: - * https://github.com/torvalds/linux/blob/5bfc75d92efd494db37f5c4c173d3639d4772966/Documentation/admin-guide/sysctl/fs.rst#overflowgid--overflowuid */ + * https://github.com/torvalds/linux/blob/5bfc75d92efd494db37f5c4c173d3639d4772966/Documentation/admin-guide/sysctl/fs.rst#overflowgid--overflowuid + * Real value in /proc/sys/fs/overflowuid or overflowgid, hardcode default value*/ private static final int OVERFLOW_ID = 65_534; - private final int uidOffset; - private final int gidOffset; - private final String vespaUser; - private final String vespaGroup; - private final int vespaUserId; - private final int vespaGroupId; + private volatile int uidOffset; + private volatile int gidOffset; + private final int idRangeSize; - public UserNamespace(int uidOffset, int gidOffset, String vespaUser, String vespaGroup, int vespaUserId, int vespaGroupId) { + public UserNamespace(int uidOffset, int gidOffset, int idRangeSize) { this.uidOffset = uidOffset; this.gidOffset = gidOffset; - this.vespaUser = Objects.requireNonNull(vespaUser); - this.vespaGroup = Objects.requireNonNull(vespaGroup); - this.vespaUserId = vespaUserId; - this.vespaGroupId = vespaGroupId; + this.idRangeSize = idRangeSize; } - public int userIdOnHost(int containerUid) { return toHostId(containerUid, uidOffset); } - public int groupIdOnHost(int containerGid) { return toHostId(containerGid, gidOffset); } - public int userIdInContainer(int hostUid) { return toContainerId(hostUid, uidOffset); } - public int groupIdInContainer(int hostGid) { return toContainerId(hostGid, gidOffset); } - - public String vespaUser() { return vespaUser; } - public String vespaGroup() { return vespaGroup; } - public int vespaUserId() { return vespaUserId; } - public int vespaGroupId() { return vespaGroupId; } + public int userIdOnHost(int containerUid) { return toHostId(containerUid, uidOffset, idRangeSize); } + public int groupIdOnHost(int containerGid) { return toHostId(containerGid, gidOffset, idRangeSize); } + public int userIdInContainer(int hostUid) { return toContainerId(hostUid, uidOffset, idRangeSize); } + public int groupIdInContainer(int hostGid) { return toContainerId(hostGid, gidOffset, idRangeSize); } - public int idRange() { return ID_RANGE; } + public int idRangeSize() { return idRangeSize; } public int overflowId() { return OVERFLOW_ID; } - private static int toHostId(int containerId, int idOffset) { - if (containerId < 0 || containerId > ID_RANGE) + // Remove after migration to mapped namespaces is complete, make fields final + public void setOffsets(int idOffset) { + this.uidOffset = idOffset; + this.gidOffset = idOffset; + } + + @Override + public boolean equals(Object o) { + if (this == o) return true; + if (o == null || getClass() != o.getClass()) return false; + UserNamespace that = (UserNamespace) o; + return uidOffset == that.uidOffset && gidOffset == that.gidOffset && idRangeSize == that.idRangeSize; + } + + @Override + public int hashCode() { + return Objects.hash(uidOffset, gidOffset, idRangeSize); + } + + @Override + public String toString() { + return "UserNamespace{" + + "uidOffset=" + uidOffset + + ", gidOffset=" + gidOffset + + ", idRangeSize=" + idRangeSize + + '}'; + } + + private static int toHostId(int containerId, int idOffset, int idRangeSize) { + if (containerId < 0 || containerId > idRangeSize) throw new IllegalArgumentException("Invalid container id: " + containerId); return idOffset + containerId; } - private static int toContainerId(int hostId, int idOffset) { + private static int toContainerId(int hostId, int idOffset, int idRangeSize) { hostId = hostId - idOffset; - return hostId < 0 || hostId >= ID_RANGE ? OVERFLOW_ID : hostId; + return hostId < 0 || hostId >= idRangeSize ? OVERFLOW_ID : hostId; } } diff --git a/node-admin/src/main/java/com/yahoo/vespa/hosted/node/admin/nodeagent/VespaUser.java b/node-admin/src/main/java/com/yahoo/vespa/hosted/node/admin/nodeagent/VespaUser.java new file mode 100644 index 00000000000..78ccca80beb --- /dev/null +++ b/node-admin/src/main/java/com/yahoo/vespa/hosted/node/admin/nodeagent/VespaUser.java @@ -0,0 +1,29 @@ +// Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. +package com.yahoo.vespa.hosted.node.admin.nodeagent; + +import java.util.Objects; + +/** + * Describes Vespa user inside the container user namespace. + * + * @author valerijf + */ +public class VespaUser { + + private final String name; + private final String group; + private final int uid; + private final int gid; + + public VespaUser(String name, String group, int uid, int gid) { + this.name = Objects.requireNonNull(name); + this.group = Objects.requireNonNull(group); + this.uid = uid; + this.gid = gid; + } + + public String name() { return name; } + public String group() { return group; } + public int uid() { return uid; } + public int gid() { return gid; } +} diff --git a/node-admin/src/main/java/com/yahoo/vespa/hosted/node/admin/task/util/fs/ContainerFileSystem.java b/node-admin/src/main/java/com/yahoo/vespa/hosted/node/admin/task/util/fs/ContainerFileSystem.java index 078a60ba7a5..6f0ab5576b7 100644 --- a/node-admin/src/main/java/com/yahoo/vespa/hosted/node/admin/task/util/fs/ContainerFileSystem.java +++ b/node-admin/src/main/java/com/yahoo/vespa/hosted/node/admin/task/util/fs/ContainerFileSystem.java @@ -2,28 +2,35 @@ package com.yahoo.vespa.hosted.node.admin.task.util.fs; import com.yahoo.vespa.hosted.node.admin.nodeagent.UserNamespace; +import com.yahoo.vespa.hosted.node.admin.nodeagent.VespaUser; import java.io.IOException; import java.nio.file.FileStore; import java.nio.file.FileSystem; -import java.nio.file.Files; import java.nio.file.Path; import java.nio.file.PathMatcher; import java.nio.file.WatchService; -import java.nio.file.attribute.UserPrincipalLookupService; import java.util.Set; -import static com.yahoo.yolean.Exceptions.uncheck; - /** * @author valerijf */ public class ContainerFileSystem extends FileSystem { private final ContainerFileSystemProvider containerFsProvider; + private final Path containerRootOnHost; - ContainerFileSystem(ContainerFileSystemProvider containerFsProvider) { + ContainerFileSystem(ContainerFileSystemProvider containerFsProvider, Path containerRootOnHost) { this.containerFsProvider = containerFsProvider; + this.containerRootOnHost = containerRootOnHost; + } + + public Path containerRootOnHost() { + return containerRootOnHost; + } + + public void createRoot() { + provider().createFileSystemRoot(); } @Override @@ -52,7 +59,7 @@ public class ContainerFileSystem extends FileSystem { } @Override - public UserPrincipalLookupService getUserPrincipalLookupService() { + public ContainerUserPrincipalLookupService getUserPrincipalLookupService() { return containerFsProvider.userPrincipalLookupService(); } @@ -86,8 +93,7 @@ public class ContainerFileSystem extends FileSystem { throw new UnsupportedOperationException(); } - public static ContainerFileSystem create(Path containerStorageRoot, UserNamespace userNamespace) { - uncheck(() -> Files.createDirectories(containerStorageRoot)); - return new ContainerFileSystemProvider(containerStorageRoot, userNamespace).getFileSystem(null); + public static ContainerFileSystem create(Path containerStorageRoot, UserNamespace userNamespace, VespaUser vespaUser) { + return new ContainerFileSystemProvider(containerStorageRoot, userNamespace, vespaUser).getFileSystem(null); } } diff --git a/node-admin/src/main/java/com/yahoo/vespa/hosted/node/admin/task/util/fs/ContainerFileSystemProvider.java b/node-admin/src/main/java/com/yahoo/vespa/hosted/node/admin/task/util/fs/ContainerFileSystemProvider.java index 909c6c9cbc1..00dda1d7cd2 100644 --- a/node-admin/src/main/java/com/yahoo/vespa/hosted/node/admin/task/util/fs/ContainerFileSystemProvider.java +++ b/node-admin/src/main/java/com/yahoo/vespa/hosted/node/admin/task/util/fs/ContainerFileSystemProvider.java @@ -2,6 +2,7 @@ package com.yahoo.vespa.hosted.node.admin.task.util.fs; import com.yahoo.vespa.hosted.node.admin.nodeagent.UserNamespace; +import com.yahoo.vespa.hosted.node.admin.nodeagent.VespaUser; import java.io.IOException; import java.net.URI; @@ -31,8 +32,8 @@ import java.util.Iterator; import java.util.Map; import java.util.Set; -import static com.yahoo.vespa.hosted.node.admin.task.util.fs.ContainerAttributeViews.ContainerPosixFileAttributes; import static com.yahoo.vespa.hosted.node.admin.task.util.fs.ContainerAttributeViews.ContainerPosixFileAttributeView; +import static com.yahoo.vespa.hosted.node.admin.task.util.fs.ContainerAttributeViews.ContainerPosixFileAttributes; import static com.yahoo.vespa.hosted.node.admin.task.util.fs.ContainerUserPrincipalLookupService.ContainerGroupPrincipal; import static com.yahoo.vespa.hosted.node.admin.task.util.fs.ContainerUserPrincipalLookupService.ContainerUserPrincipal; import static com.yahoo.yolean.Exceptions.uncheck; @@ -43,17 +44,11 @@ import static com.yahoo.yolean.Exceptions.uncheck; class ContainerFileSystemProvider extends FileSystemProvider { private final ContainerFileSystem containerFs; private final ContainerUserPrincipalLookupService userPrincipalLookupService; - private final Path containerRootOnHost; - ContainerFileSystemProvider(Path containerRootOnHost, UserNamespace userNamespace) { - this.containerFs = new ContainerFileSystem(this); + ContainerFileSystemProvider(Path containerRootOnHost, UserNamespace userNamespace, VespaUser vespaUser) { + this.containerFs = new ContainerFileSystem(this, containerRootOnHost); this.userPrincipalLookupService = new ContainerUserPrincipalLookupService( - containerRootOnHost.getFileSystem().getUserPrincipalLookupService(), userNamespace); - this.containerRootOnHost = containerRootOnHost; - } - - public Path containerRootOnHost() { - return containerRootOnHost; + containerRootOnHost.getFileSystem().getUserPrincipalLookupService(), userNamespace, vespaUser); } public ContainerUserPrincipalLookupService userPrincipalLookupService() { @@ -225,6 +220,16 @@ class ContainerFileSystemProvider extends FileSystemProvider { return value; } + void createFileSystemRoot() { + ContainerPath root = containerFs.getPath("/"); + if (!Files.exists(root)) { + uncheck(() -> { + Files.createDirectories(root.pathOnHost()); + fixOwnerToContainerRoot(root); + }); + } + } + private void fixOwnerToContainerRoot(ContainerPath path) throws IOException { setAttribute(path, "unix:uid", 0); setAttribute(path, "unix:gid", 0); diff --git a/node-admin/src/main/java/com/yahoo/vespa/hosted/node/admin/task/util/fs/ContainerPath.java b/node-admin/src/main/java/com/yahoo/vespa/hosted/node/admin/task/util/fs/ContainerPath.java index 15295ffd087..853646d53b5 100644 --- a/node-admin/src/main/java/com/yahoo/vespa/hosted/node/admin/task/util/fs/ContainerPath.java +++ b/node-admin/src/main/java/com/yahoo/vespa/hosted/node/admin/task/util/fs/ContainerPath.java @@ -32,7 +32,7 @@ public class ContainerPath implements Path { if (!pathOnHost.isAbsolute()) throw new IllegalArgumentException("Path host must be absolute: " + pathOnHost); - Path containerRootOnHost = containerFs.provider().containerRootOnHost(); + Path containerRootOnHost = containerFs.containerRootOnHost(); if (!pathOnHost.startsWith(containerRootOnHost)) throw new IllegalArgumentException("Path on host (" + pathOnHost + ") must start with container root on host (" + containerRootOnHost + ")"); } @@ -173,7 +173,7 @@ public class ContainerPath implements Path { @Override public String toString() { - return containerFs.provider().containerRootOnHost().getFileName() + ":" + pathInContainer(); + return containerFs.containerRootOnHost().getFileName() + ":" + pathInContainer(); } private static ContainerPath resolve(ContainerFileSystem containerFs, String[] currentParts, Path other) { @@ -189,7 +189,7 @@ public class ContainerPath implements Path { } return new ContainerPath(containerFs, - containerFs.provider().containerRootOnHost().resolve(String.join("/", parts)), + containerFs.containerRootOnHost().resolve(String.join("/", parts)), parts.toArray(String[]::new)); } @@ -201,7 +201,7 @@ public class ContainerPath implements Path { public static ContainerPath fromPathOnHost(ContainerFileSystem containerFs, Path pathOnHost) { pathOnHost = pathOnHost.normalize(); - Path containerRootOnHost = containerFs.provider().containerRootOnHost(); + Path containerRootOnHost = containerFs.containerRootOnHost(); Path pathUnderContainerStorage = containerRootOnHost.relativize(pathOnHost); if (pathUnderContainerStorage.getNameCount() == 0 || pathUnderContainerStorage.getName(0).toString().isEmpty()) diff --git a/node-admin/src/main/java/com/yahoo/vespa/hosted/node/admin/task/util/fs/ContainerUserPrincipalLookupService.java b/node-admin/src/main/java/com/yahoo/vespa/hosted/node/admin/task/util/fs/ContainerUserPrincipalLookupService.java index ae65f6a7f7f..8e35bdccc23 100644 --- a/node-admin/src/main/java/com/yahoo/vespa/hosted/node/admin/task/util/fs/ContainerUserPrincipalLookupService.java +++ b/node-admin/src/main/java/com/yahoo/vespa/hosted/node/admin/task/util/fs/ContainerUserPrincipalLookupService.java @@ -2,6 +2,7 @@ package com.yahoo.vespa.hosted.node.admin.task.util.fs; import com.yahoo.vespa.hosted.node.admin.nodeagent.UserNamespace; +import com.yahoo.vespa.hosted.node.admin.nodeagent.VespaUser; import java.io.IOException; import java.nio.file.attribute.GroupPrincipal; @@ -13,16 +14,22 @@ import java.util.Objects; /** * @author valerijf */ -class ContainerUserPrincipalLookupService extends UserPrincipalLookupService { +public class ContainerUserPrincipalLookupService extends UserPrincipalLookupService { private final UserPrincipalLookupService baseFsUserPrincipalLookupService; private final UserNamespace userNamespace; + private final VespaUser vespaUser; - ContainerUserPrincipalLookupService(UserPrincipalLookupService baseFsUserPrincipalLookupService, UserNamespace userNamespace) { + ContainerUserPrincipalLookupService( + UserPrincipalLookupService baseFsUserPrincipalLookupService, UserNamespace userNamespace, VespaUser vespaUser) { this.baseFsUserPrincipalLookupService = Objects.requireNonNull(baseFsUserPrincipalLookupService); this.userNamespace = Objects.requireNonNull(userNamespace); + this.vespaUser = Objects.requireNonNull(vespaUser); } + public UserNamespace userNamespace() { return userNamespace; } + public VespaUser vespaUser() { return vespaUser; } + public int userIdOnHost(int containerUid) { return userNamespace.userIdOnHost(containerUid); } public int groupIdOnHost(int containerGid) { return userNamespace.groupIdOnHost(containerGid); } public int userIdInContainer(int hostUid) { return userNamespace.userIdInContainer(hostUid); } @@ -30,27 +37,27 @@ class ContainerUserPrincipalLookupService extends UserPrincipalLookupService { @Override public ContainerUserPrincipal lookupPrincipalByName(String name) throws IOException { - int containerUid = resolveName(name, userNamespace.vespaUser(), userNamespace.vespaUserId()); - String user = resolveId(containerUid, userNamespace.vespaUser(), userNamespace.vespaUserId()); + int containerUid = resolveName(name, vespaUser.name(), vespaUser.uid()); + String user = resolveId(containerUid, vespaUser.name(), vespaUser.uid()); String hostUid = String.valueOf(userIdOnHost(containerUid)); return new ContainerUserPrincipal(containerUid, user, baseFsUserPrincipalLookupService.lookupPrincipalByName(hostUid)); } @Override public ContainerGroupPrincipal lookupPrincipalByGroupName(String group) throws IOException { - int containerGid = resolveName(group, userNamespace.vespaGroup(), userNamespace.vespaGroupId()); - String name = resolveId(containerGid, userNamespace.vespaGroup(), userNamespace.vespaGroupId()); + int containerGid = resolveName(group, vespaUser.group(), vespaUser.gid()); + String name = resolveId(containerGid, vespaUser.group(), vespaUser.gid()); String hostGid = String.valueOf(groupIdOnHost(containerGid)); return new ContainerGroupPrincipal(containerGid, name, baseFsUserPrincipalLookupService.lookupPrincipalByGroupName(hostGid)); } public ContainerUserPrincipal userPrincipal(int uid, UserPrincipal baseFsPrincipal) { - String name = resolveId(uid, userNamespace.vespaUser(), userNamespace.vespaUserId()); + String name = resolveId(uid, vespaUser.name(), vespaUser.uid()); return new ContainerUserPrincipal(uid, name, baseFsPrincipal); } public ContainerGroupPrincipal groupPrincipal(int gid, GroupPrincipal baseFsPrincipal) { - String name = resolveId(gid, userNamespace.vespaGroup(), userNamespace.vespaGroupId()); + String name = resolveId(gid, vespaUser.group(), vespaUser.gid()); return new ContainerGroupPrincipal(gid, name, baseFsPrincipal); } diff --git a/node-admin/src/test/java/com/yahoo/vespa/hosted/node/admin/integration/ContainerTester.java b/node-admin/src/test/java/com/yahoo/vespa/hosted/node/admin/integration/ContainerTester.java index 08e335f188a..4a26195dd3a 100644 --- a/node-admin/src/test/java/com/yahoo/vespa/hosted/node/admin/integration/ContainerTester.java +++ b/node-admin/src/test/java/com/yahoo/vespa/hosted/node/admin/integration/ContainerTester.java @@ -93,7 +93,7 @@ public class ContainerTester implements AutoCloseable { NodeAgentContextFactory nodeAgentContextFactory = (nodeSpec, acl) -> NodeAgentContextImpl.builder(nodeSpec).acl(acl).fileSystem(fileSystem).build(); nodeAdminStateUpdater = new NodeAdminStateUpdater(nodeAgentContextFactory, nodeRepository, orchestrator, - nodeAdmin, HOST_HOSTNAME, clock, flagSource); + nodeAdmin, HOST_HOSTNAME); loopThread = new Thread(() -> { nodeAdminStateUpdater.start(); diff --git a/node-admin/src/test/java/com/yahoo/vespa/hosted/node/admin/maintenance/servicedump/ArtifactProducersTest.java b/node-admin/src/test/java/com/yahoo/vespa/hosted/node/admin/maintenance/servicedump/ArtifactProducersTest.java index 289f51ca867..4e34d936d85 100644 --- a/node-admin/src/test/java/com/yahoo/vespa/hosted/node/admin/maintenance/servicedump/ArtifactProducersTest.java +++ b/node-admin/src/test/java/com/yahoo/vespa/hosted/node/admin/maintenance/servicedump/ArtifactProducersTest.java @@ -1,4 +1,5 @@ -package com.yahoo.vespa.hosted.node.admin.maintenance.servicedump;// Copyright Verizon Media. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. +// Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. +package com.yahoo.vespa.hosted.node.admin.maintenance.servicedump; import com.yahoo.yolean.concurrent.Sleeper; import org.junit.jupiter.api.Test; @@ -26,4 +27,4 @@ class ArtifactProducersTest { assertEquals(expectedMsg, exception.getMessage()); } -}
\ No newline at end of file +} diff --git a/node-admin/src/test/java/com/yahoo/vespa/hosted/node/admin/nodeadmin/NodeAdminStateUpdaterTest.java b/node-admin/src/test/java/com/yahoo/vespa/hosted/node/admin/nodeadmin/NodeAdminStateUpdaterTest.java index f9b0070a3d6..5436f84f467 100644 --- a/node-admin/src/test/java/com/yahoo/vespa/hosted/node/admin/nodeadmin/NodeAdminStateUpdaterTest.java +++ b/node-admin/src/test/java/com/yahoo/vespa/hosted/node/admin/nodeadmin/NodeAdminStateUpdaterTest.java @@ -3,8 +3,6 @@ package com.yahoo.vespa.hosted.node.admin.nodeadmin; import com.yahoo.config.provision.HostName; import com.yahoo.config.provision.NodeType; -import com.yahoo.test.ManualClock; -import com.yahoo.vespa.flags.InMemoryFlagSource; import com.yahoo.vespa.hosted.node.admin.configserver.noderepository.Acl; import com.yahoo.vespa.hosted.node.admin.configserver.noderepository.NodeSpec; import com.yahoo.vespa.hosted.node.admin.configserver.noderepository.NodeState; @@ -50,11 +48,9 @@ public class NodeAdminStateUpdaterTest { private final Orchestrator orchestrator = mock(Orchestrator.class); private final NodeAdmin nodeAdmin = mock(NodeAdmin.class); private final HostName hostHostname = HostName.from("basehost1.test.yahoo.com"); - private final ManualClock clock = new ManualClock(); - private final InMemoryFlagSource flagSource = new InMemoryFlagSource(); private final NodeAdminStateUpdater updater = spy(new NodeAdminStateUpdater( - nodeAgentContextFactory, nodeRepository, orchestrator, nodeAdmin, hostHostname, clock, flagSource)); + nodeAgentContextFactory, nodeRepository, orchestrator, nodeAdmin, hostHostname)); @Test diff --git a/node-admin/src/test/java/com/yahoo/vespa/hosted/node/admin/nodeagent/UserNamespaceTest.java b/node-admin/src/test/java/com/yahoo/vespa/hosted/node/admin/nodeagent/UserNamespaceTest.java index 73b59a17c37..20e4bad8c31 100644 --- a/node-admin/src/test/java/com/yahoo/vespa/hosted/node/admin/nodeagent/UserNamespaceTest.java +++ b/node-admin/src/test/java/com/yahoo/vespa/hosted/node/admin/nodeagent/UserNamespaceTest.java @@ -11,7 +11,7 @@ import static org.junit.jupiter.api.Assertions.assertThrows; */ class UserNamespaceTest { - private final UserNamespace userNamespace = new UserNamespace(1000, 2000, "vespa", "users", 1000, 100); + private final UserNamespace userNamespace = new UserNamespace(1000, 2000, 10000); @Test public void translates_between_ids() { diff --git a/node-admin/src/test/java/com/yahoo/vespa/hosted/node/admin/task/util/fs/ContainerFileSystemTest.java b/node-admin/src/test/java/com/yahoo/vespa/hosted/node/admin/task/util/fs/ContainerFileSystemTest.java index 4e85052a176..932f56f3a60 100644 --- a/node-admin/src/test/java/com/yahoo/vespa/hosted/node/admin/task/util/fs/ContainerFileSystemTest.java +++ b/node-admin/src/test/java/com/yahoo/vespa/hosted/node/admin/task/util/fs/ContainerFileSystemTest.java @@ -2,6 +2,7 @@ package com.yahoo.vespa.hosted.node.admin.task.util.fs; import com.yahoo.vespa.hosted.node.admin.nodeagent.UserNamespace; +import com.yahoo.vespa.hosted.node.admin.nodeagent.VespaUser; import com.yahoo.vespa.hosted.node.admin.task.util.file.UnixPath; import com.yahoo.vespa.test.file.TestFileSystem; import org.junit.jupiter.api.Test; @@ -24,8 +25,10 @@ class ContainerFileSystemTest { private final FileSystem fileSystem = TestFileSystem.create(); private final UnixPath containerRootOnHost = new UnixPath(fileSystem.getPath("/data/storage/ctr1")); - private final UserNamespace userNamespace = new UserNamespace(10_000, 11_000, "vespa", "users", 1000, 100); - private final ContainerFileSystem containerFs = ContainerFileSystem.create(containerRootOnHost.createDirectories().toPath(), userNamespace); + private final UserNamespace userNamespace = new UserNamespace(10_000, 11_000, 10000); + private final VespaUser vespaUser = new VespaUser("vespa", "users", 1000, 100); + private final ContainerFileSystem containerFs = ContainerFileSystem.create( + containerRootOnHost.createDirectories().toPath(), userNamespace, vespaUser); @Test public void creates_files_and_directories_with_container_root_as_owner() throws IOException { diff --git a/node-admin/src/test/java/com/yahoo/vespa/hosted/node/admin/task/util/fs/ContainerPathTest.java b/node-admin/src/test/java/com/yahoo/vespa/hosted/node/admin/task/util/fs/ContainerPathTest.java index 6bca8c2f0b1..795b3b198f5 100644 --- a/node-admin/src/test/java/com/yahoo/vespa/hosted/node/admin/task/util/fs/ContainerPathTest.java +++ b/node-admin/src/test/java/com/yahoo/vespa/hosted/node/admin/task/util/fs/ContainerPathTest.java @@ -2,11 +2,18 @@ package com.yahoo.vespa.hosted.node.admin.task.util.fs; import com.yahoo.vespa.hosted.node.admin.nodeagent.UserNamespace; +import com.yahoo.vespa.hosted.node.admin.nodeagent.VespaUser; import com.yahoo.vespa.test.file.TestFileSystem; import org.junit.jupiter.api.Assertions; import org.junit.jupiter.api.Test; import org.junit.jupiter.api.function.Executable; +import java.io.IOException; +import java.nio.file.FileSystem; +import java.nio.file.Files; +import java.nio.file.LinkOption; +import java.nio.file.Path; + import static com.yahoo.vespa.hosted.node.admin.task.util.fs.ContainerPath.fromPathInContainer; import static com.yahoo.vespa.hosted.node.admin.task.util.fs.ContainerPath.fromPathOnHost; import static org.junit.jupiter.api.Assertions.assertEquals; @@ -15,19 +22,13 @@ import static org.junit.jupiter.api.Assertions.assertNull; import static org.junit.jupiter.api.Assertions.assertTrue; import static org.mockito.Mockito.mock; -import java.io.IOException; -import java.nio.file.FileSystem; -import java.nio.file.Files; -import java.nio.file.LinkOption; -import java.nio.file.Path; - /** * @author valerijf */ class ContainerPathTest { private final FileSystem baseFs = TestFileSystem.create(); - private final ContainerFileSystem containerFs = ContainerFileSystem.create(baseFs.getPath("/data/storage/ctr1"), mock(UserNamespace.class)); + private final ContainerFileSystem containerFs = ContainerFileSystem.create(baseFs.getPath("/data/storage/ctr1"), mock(UserNamespace.class), mock(VespaUser.class)); @Test public void create_new_container_path() { diff --git a/node-admin/src/test/java/com/yahoo/vespa/hosted/node/admin/task/util/fs/ContainerUserPrincipalLookupServiceTest.java b/node-admin/src/test/java/com/yahoo/vespa/hosted/node/admin/task/util/fs/ContainerUserPrincipalLookupServiceTest.java index bc26cfa73f3..f201f2667cd 100644 --- a/node-admin/src/test/java/com/yahoo/vespa/hosted/node/admin/task/util/fs/ContainerUserPrincipalLookupServiceTest.java +++ b/node-admin/src/test/java/com/yahoo/vespa/hosted/node/admin/task/util/fs/ContainerUserPrincipalLookupServiceTest.java @@ -2,14 +2,15 @@ package com.yahoo.vespa.hosted.node.admin.task.util.fs; import com.yahoo.vespa.hosted.node.admin.nodeagent.UserNamespace; +import com.yahoo.vespa.hosted.node.admin.nodeagent.VespaUser; import com.yahoo.vespa.test.file.TestFileSystem; import org.junit.jupiter.api.Test; import java.io.IOException; import java.nio.file.attribute.UserPrincipalNotFoundException; -import static com.yahoo.vespa.hosted.node.admin.task.util.fs.ContainerUserPrincipalLookupService.ContainerUserPrincipal; import static com.yahoo.vespa.hosted.node.admin.task.util.fs.ContainerUserPrincipalLookupService.ContainerGroupPrincipal; +import static com.yahoo.vespa.hosted.node.admin.task.util.fs.ContainerUserPrincipalLookupService.ContainerUserPrincipal; import static org.junit.jupiter.api.Assertions.assertEquals; import static org.junit.jupiter.api.Assertions.assertThrows; @@ -18,9 +19,10 @@ import static org.junit.jupiter.api.Assertions.assertThrows; */ class ContainerUserPrincipalLookupServiceTest { - private final UserNamespace userNamespace = new UserNamespace(10_000, 11_000, "vespa", "users", 1000, 100); + private final UserNamespace userNamespace = new UserNamespace(10_000, 11_000, 10000); + private final VespaUser vespaUser = new VespaUser("vespa", "users", 1000, 100); private final ContainerUserPrincipalLookupService userPrincipalLookupService = - new ContainerUserPrincipalLookupService(TestFileSystem.create().getUserPrincipalLookupService(), userNamespace); + new ContainerUserPrincipalLookupService(TestFileSystem.create().getUserPrincipalLookupService(), userNamespace, vespaUser); @Test public void correctly_resolves_ids() throws IOException { diff --git a/node-repository/src/main/java/com/yahoo/vespa/hosted/provision/applications/AutoscalingStatus.java b/node-repository/src/main/java/com/yahoo/vespa/hosted/provision/applications/AutoscalingStatus.java index c40408c9109..f44785cfab3 100644 --- a/node-repository/src/main/java/com/yahoo/vespa/hosted/provision/applications/AutoscalingStatus.java +++ b/node-repository/src/main/java/com/yahoo/vespa/hosted/provision/applications/AutoscalingStatus.java @@ -1,3 +1,4 @@ +// Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. package com.yahoo.vespa.hosted.provision.applications; import com.yahoo.vespa.hosted.provision.autoscale.Autoscaler; diff --git a/node-repository/src/main/java/com/yahoo/vespa/hosted/provision/lb/LoadBalancerList.java b/node-repository/src/main/java/com/yahoo/vespa/hosted/provision/lb/LoadBalancerList.java index bad16bf7d12..93c89a7485e 100644 --- a/node-repository/src/main/java/com/yahoo/vespa/hosted/provision/lb/LoadBalancerList.java +++ b/node-repository/src/main/java/com/yahoo/vespa/hosted/provision/lb/LoadBalancerList.java @@ -1,46 +1,28 @@ // Copyright 2020 Oath Inc. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. package com.yahoo.vespa.hosted.provision.lb; +import com.yahoo.collections.AbstractFilteringList; + import java.util.Collection; -import java.util.Iterator; -import java.util.List; -import java.util.Objects; -import java.util.stream.Collectors; -import java.util.stream.Stream; /** * A filterable load balancer list. This is immutable. * * @author mpolden */ -public class LoadBalancerList implements Iterable<LoadBalancer> { - - private final List<LoadBalancer> loadBalancers; +public class LoadBalancerList extends AbstractFilteringList<LoadBalancer, LoadBalancerList> { - private LoadBalancerList(Collection<LoadBalancer> loadBalancers) { - this.loadBalancers = List.copyOf(Objects.requireNonNull(loadBalancers, "loadBalancers must be non-null")); + protected LoadBalancerList(Collection<LoadBalancer> items, boolean negate) { + super(items, negate, LoadBalancerList::new); } /** Returns the subset of load balancers that are in given state */ public LoadBalancerList in(LoadBalancer.State state) { - return of(loadBalancers.stream().filter(lb -> lb.state() == state)); - } - - public List<LoadBalancer> asList() { - return loadBalancers; - } - - private static LoadBalancerList of(Stream<LoadBalancer> stream) { - return new LoadBalancerList(stream.collect(Collectors.toUnmodifiableList())); + return matching(lb -> lb.state() == state); } public static LoadBalancerList copyOf(Collection<LoadBalancer> loadBalancers) { - return new LoadBalancerList(loadBalancers); - } - - @Override - public Iterator<LoadBalancer> iterator() { - return loadBalancers.iterator(); + return new LoadBalancerList(loadBalancers, false); } } diff --git a/node-repository/src/test/java/com/yahoo/vespa/hosted/provision/maintenance/ProvisionedExpirerTest.java b/node-repository/src/test/java/com/yahoo/vespa/hosted/provision/maintenance/ProvisionedExpirerTest.java index 786faae24b4..329e6c833a9 100644 --- a/node-repository/src/test/java/com/yahoo/vespa/hosted/provision/maintenance/ProvisionedExpirerTest.java +++ b/node-repository/src/test/java/com/yahoo/vespa/hosted/provision/maintenance/ProvisionedExpirerTest.java @@ -1,3 +1,4 @@ +// Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. package com.yahoo.vespa.hosted.provision.maintenance; import com.yahoo.config.provision.Cloud; diff --git a/orchestrator/src/test/java/com/yahoo/vespa/orchestrator/resources/ApplicationSuspensionRequestHandlerTest.java b/orchestrator/src/test/java/com/yahoo/vespa/orchestrator/resources/ApplicationSuspensionRequestHandlerTest.java index b32413273bb..1a89c1a4ab4 100644 --- a/orchestrator/src/test/java/com/yahoo/vespa/orchestrator/resources/ApplicationSuspensionRequestHandlerTest.java +++ b/orchestrator/src/test/java/com/yahoo/vespa/orchestrator/resources/ApplicationSuspensionRequestHandlerTest.java @@ -1,4 +1,5 @@ -package com.yahoo.vespa.orchestrator.resources;// Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. +// Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. +package com.yahoo.vespa.orchestrator.resources; import com.fasterxml.jackson.core.type.TypeReference; import com.yahoo.cloud.config.ConfigserverConfig; @@ -34,6 +35,7 @@ import static org.junit.jupiter.api.Assertions.assertEquals; * @author bjorncs */ class ApplicationSuspensionRequestHandlerTest { + private static final String RESOURCE_1 = "mediasearch:imagesearch:default"; private static final String RESOURCE_2 = "test-tenant-id:application:instance"; private static final String INVALID_RESOURCE_NAME = "something_without_colons"; @@ -148,4 +150,4 @@ class ApplicationSuspensionRequestHandlerTest { assertEquals(200, response.getStatus()); return testDriver.parseJacksonResponseContent(response, type); } -}
\ No newline at end of file +} diff --git a/screwdriver/release-java-artifacts.sh b/screwdriver/release-java-artifacts.sh index afb456699b8..a91585d6ad5 100755 --- a/screwdriver/release-java-artifacts.sh +++ b/screwdriver/release-java-artifacts.sh @@ -51,10 +51,10 @@ STG_REPO=$(cat $TMPFILE | grep 'Staging repository at http' | head -1 | awk -F/ rm -f $TMPFILE # Deploy plugins -mvn $COMMON_MAVEN_OPTS --file ./maven-plugins/pom.xml -DskipLocalStaging=true -DskipStagingRepositoryClose=true -DstagingRepositoryId=$STG_REPO deploy +mvn $COMMON_MAVEN_OPTS --file ./maven-plugins/pom.xml -DskipStagingRepositoryClose=true -DstagingRepositoryId=$STG_REPO deploy # Deploy the rest of the artifacts -mvn $COMMON_MAVEN_OPTS --threads 1C -DskipLocalStaging=true -DskipStagingRepositoryClose=true -DstagingRepositoryId=$STG_REPO deploy +mvn $COMMON_MAVEN_OPTS --threads 1C -DskipStagingRepositoryClose=true -DstagingRepositoryId=$STG_REPO deploy # Close with checks mvn $COMMON_MAVEN_OPTS -N org.sonatype.plugins:nexus-staging-maven-plugin:rc-close -DnexusUrl=https://oss.sonatype.org/ -DserverId=ossrh -DstagingRepositoryId=$STG_REPO diff --git a/sd-plugin/README.md b/sd-plugin/README.md index 76a0a53d49f..c9cbaded66c 100644 --- a/sd-plugin/README.md +++ b/sd-plugin/README.md @@ -1,3 +1,4 @@ +<!-- Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. --> This directory holds the code for an IntteliJ plugin for reading SD files. NOTE: This is the source code, not the plugin itself. In order to be able to use the plugin you'll need to download it from JetBrains Marketplace or create a zip file and load it to IntelliJ (details later). @@ -15,5 +16,6 @@ After cloning, you should: Now you should have a "gen" folder next to the "java" folder, and it contains all the parser and lexer code. -Improtant note! After any change in one of this 2 files (bnf, flex) you'll need to generate again. The proper way is to delete the "gen" folder and then do 1-2 again. +Important note! After any change in one of this 2 files (bnf, flex) you'll need to generate again. The proper way is to delete the "gen" folder and then do 1-2 again. +Now, you can run the gradle task "runIde", open a project with some sd file and see how the plugin works on it.
\ No newline at end of file diff --git a/sd-plugin/src/main/java/org/intellij/sdk/language/SdBlock.java b/sd-plugin/src/main/java/org/intellij/sdk/language/SdBlock.java deleted file mode 100644 index d32baf86d7a..00000000000 --- a/sd-plugin/src/main/java/org/intellij/sdk/language/SdBlock.java +++ /dev/null @@ -1,60 +0,0 @@ -package org.intellij.sdk.language; - -import com.intellij.formatting.Alignment; -import com.intellij.formatting.Block; -import com.intellij.formatting.Indent; -import com.intellij.formatting.Spacing; -import com.intellij.formatting.SpacingBuilder; -import com.intellij.formatting.Wrap; -import com.intellij.formatting.WrapType; -import com.intellij.lang.ASTNode; -import com.intellij.psi.TokenType; -import com.intellij.psi.formatter.common.AbstractBlock; -import org.jetbrains.annotations.NotNull; -import org.jetbrains.annotations.Nullable; - -import java.util.ArrayList; -import java.util.List; - -public class SdBlock extends AbstractBlock { - - private final SpacingBuilder spacingBuilder; - - protected SdBlock(@NotNull ASTNode node, @Nullable Wrap wrap, @Nullable Alignment alignment, - SpacingBuilder spacingBuilder) { - super(node, wrap, alignment); - this.spacingBuilder = spacingBuilder; - } - - @Override - protected List<Block> buildChildren() { - List<Block> blocks = new ArrayList<>(); - ASTNode child = myNode.getFirstChildNode(); - while (child != null) { - if (child.getElementType() != TokenType.WHITE_SPACE) { - Block block = new SdBlock(child, Wrap.createWrap(WrapType.NONE, false), Alignment.createAlignment(), - spacingBuilder); - blocks.add(block); - } - child = child.getTreeNext(); - } - return blocks; - } - - @Override - public Indent getIndent() { - return Indent.getNoneIndent(); - } - - @Nullable - @Override - public Spacing getSpacing(@Nullable Block child1, @NotNull Block child2) { - return spacingBuilder.getSpacing(this, child1, child2); - } - - @Override - public boolean isLeaf() { - return myNode.getFirstChildNode() == null; - } - -} diff --git a/sd-plugin/src/main/java/org/intellij/sdk/language/SdChooseByNameContributor.java b/sd-plugin/src/main/java/org/intellij/sdk/language/SdChooseByNameContributor.java index 45e26f401d4..f313bb2be59 100644 --- a/sd-plugin/src/main/java/org/intellij/sdk/language/SdChooseByNameContributor.java +++ b/sd-plugin/src/main/java/org/intellij/sdk/language/SdChooseByNameContributor.java @@ -1,3 +1,4 @@ +// Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. package org.intellij.sdk.language; import com.intellij.navigation.ChooseByNameContributor; @@ -16,6 +17,10 @@ import java.util.ArrayList; import java.util.Collection; import java.util.List; +/** + * This class is used for the extension (in plugin.xml) to enable "Go To Symbol" feature. + * @author shahariel + */ public class SdChooseByNameContributor implements ChooseByNameContributor { @Override diff --git a/sd-plugin/src/main/java/org/intellij/sdk/language/SdCodeStyleSettings.java b/sd-plugin/src/main/java/org/intellij/sdk/language/SdCodeStyleSettings.java index 0719ac7f426..89460099790 100644 --- a/sd-plugin/src/main/java/org/intellij/sdk/language/SdCodeStyleSettings.java +++ b/sd-plugin/src/main/java/org/intellij/sdk/language/SdCodeStyleSettings.java @@ -1,8 +1,13 @@ +// Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. package org.intellij.sdk.language; import com.intellij.psi.codeStyle.CodeStyleSettings; import com.intellij.psi.codeStyle.CustomCodeStyleSettings; +/** + * This class represent a code style settings, and creates an option page in settings/preferences. + * @author shahariel + */ public class SdCodeStyleSettings extends CustomCodeStyleSettings { public SdCodeStyleSettings(CodeStyleSettings settings) { diff --git a/sd-plugin/src/main/java/org/intellij/sdk/language/SdCodeStyleSettingsProvider.java b/sd-plugin/src/main/java/org/intellij/sdk/language/SdCodeStyleSettingsProvider.java index 6ea68655c70..3b8136f92b3 100644 --- a/sd-plugin/src/main/java/org/intellij/sdk/language/SdCodeStyleSettingsProvider.java +++ b/sd-plugin/src/main/java/org/intellij/sdk/language/SdCodeStyleSettingsProvider.java @@ -1,3 +1,4 @@ +// Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. package org.intellij.sdk.language; import com.intellij.application.options.CodeStyleAbstractConfigurable; @@ -10,6 +11,10 @@ import com.intellij.psi.codeStyle.CustomCodeStyleSettings; import org.jetbrains.annotations.NotNull; import org.jetbrains.annotations.Nullable; +/** + * This class is used for the extension (in plugin.xml) to the class SdCodeStyleSettings. + * @author shahariel + */ public class SdCodeStyleSettingsProvider extends CodeStyleSettingsProvider { @Override diff --git a/sd-plugin/src/main/java/org/intellij/sdk/language/SdCommenter.java b/sd-plugin/src/main/java/org/intellij/sdk/language/SdCommenter.java index e0fc30df22b..5d9fbfaa07d 100644 --- a/sd-plugin/src/main/java/org/intellij/sdk/language/SdCommenter.java +++ b/sd-plugin/src/main/java/org/intellij/sdk/language/SdCommenter.java @@ -1,10 +1,13 @@ +// Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. package org.intellij.sdk.language; import com.intellij.lang.Commenter; import org.jetbrains.annotations.Nullable; /** - * This class enables turning a line into a comment with "Code -> Comment with line comment" + * This class is used for the extension (in plugin.xml), to enable turning a line into a comment with + * "Code -> Comment with line comment". + * @author shahariel */ public class SdCommenter implements Commenter { diff --git a/sd-plugin/src/main/java/org/intellij/sdk/language/SdCompletionContributor.java b/sd-plugin/src/main/java/org/intellij/sdk/language/SdCompletionContributor.java index 5dd67e5b6e6..7cf517ba2e9 100644 --- a/sd-plugin/src/main/java/org/intellij/sdk/language/SdCompletionContributor.java +++ b/sd-plugin/src/main/java/org/intellij/sdk/language/SdCompletionContributor.java @@ -1,3 +1,4 @@ +// Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. package org.intellij.sdk.language; import com.intellij.codeInsight.completion.CompletionContributor; @@ -11,6 +12,10 @@ import com.intellij.util.ProcessingContext; import org.intellij.sdk.language.psi.SdTypes; import org.jetbrains.annotations.NotNull; +/** + * This class is used for the extension (in plugin.xml) to enables Auto-Complete. Partially works for now. + * @author shahariel + */ public class SdCompletionContributor extends CompletionContributor { diff --git a/sd-plugin/src/main/java/org/intellij/sdk/language/SdFileType.java b/sd-plugin/src/main/java/org/intellij/sdk/language/SdFileType.java index eeed80ad9f1..0f6189abbe6 100644 --- a/sd-plugin/src/main/java/org/intellij/sdk/language/SdFileType.java +++ b/sd-plugin/src/main/java/org/intellij/sdk/language/SdFileType.java @@ -1,3 +1,4 @@ +// Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. package org.intellij.sdk.language; import com.intellij.openapi.fileTypes.LanguageFileType; @@ -6,6 +7,10 @@ import org.jetbrains.annotations.Nullable; import javax.swing.Icon; +/** + * This class is used for the extension (in plugin.xml), to define SD as a file's type. + * @author shahariel + */ public class SdFileType extends LanguageFileType { public static final SdFileType INSTANCE = new SdFileType(); diff --git a/sd-plugin/src/main/java/org/intellij/sdk/language/SdFormattingModelBuilder.java b/sd-plugin/src/main/java/org/intellij/sdk/language/SdFormattingModelBuilder.java deleted file mode 100644 index 35f57f65c26..00000000000 --- a/sd-plugin/src/main/java/org/intellij/sdk/language/SdFormattingModelBuilder.java +++ /dev/null @@ -1,37 +0,0 @@ -package org.intellij.sdk.language; - -import com.intellij.formatting.Alignment; -import com.intellij.formatting.FormattingContext; -import com.intellij.formatting.FormattingModel; -import com.intellij.formatting.FormattingModelBuilder; -import com.intellij.formatting.FormattingModelProvider; -import com.intellij.formatting.SpacingBuilder; -import com.intellij.formatting.Wrap; -import com.intellij.formatting.WrapType; -import com.intellij.psi.codeStyle.CodeStyleSettings; -import org.intellij.sdk.language.psi.SdTypes; -import org.jetbrains.annotations.NotNull; - -public class SdFormattingModelBuilder implements FormattingModelBuilder { - - private static SpacingBuilder createSpaceBuilder(CodeStyleSettings settings) { - return new SpacingBuilder(settings, SdLanguage.INSTANCE) - .around(SdTypes.SYMBOL) - .spaceIf(settings.getCommonSettings(SdLanguage.INSTANCE.getID()).SPACE_AFTER_COLON) - .before(SdTypes.DOCUMENT_FIELD_DEFINITION) - .none(); - } - - @Override - public @NotNull FormattingModel createModel(@NotNull FormattingContext formattingContext) { - final CodeStyleSettings codeStyleSettings = formattingContext.getCodeStyleSettings(); - return FormattingModelProvider - .createFormattingModelForPsiFile(formattingContext.getContainingFile(), - new SdBlock(formattingContext.getNode(), - Wrap.createWrap(WrapType.NONE, false), - Alignment.createAlignment(), - createSpaceBuilder(codeStyleSettings)), - codeStyleSettings); - } - -} diff --git a/sd-plugin/src/main/java/org/intellij/sdk/language/SdIcons.java b/sd-plugin/src/main/java/org/intellij/sdk/language/SdIcons.java index 8dadda25774..21c46f15e55 100644 --- a/sd-plugin/src/main/java/org/intellij/sdk/language/SdIcons.java +++ b/sd-plugin/src/main/java/org/intellij/sdk/language/SdIcons.java @@ -1,9 +1,14 @@ +// Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. package org.intellij.sdk.language; import com.intellij.openapi.util.IconLoader; import javax.swing.Icon; +/** + * This class is used for defining Icons for the IDE. + * @author shahariel + */ public class SdIcons { public static final Icon FILE = IconLoader.getIcon("icons/sd_icon.png", SdIcons.class); public static final Icon STRUCT_FIELD = IconLoader.getIcon("icons/struct_field_icon.png", SdIcons.class); @@ -16,4 +21,4 @@ public class SdIcons { public static final Icon OVERRIDE_MACRO = IconLoader.getIcon("icons/override_macro_icon.png", SdIcons.class); public static final Icon FIRST_PHASE = IconLoader.getIcon("icons/first_phase_icon.png", SdIcons.class); } -
\ No newline at end of file + diff --git a/sd-plugin/src/main/java/org/intellij/sdk/language/SdLanguage.java b/sd-plugin/src/main/java/org/intellij/sdk/language/SdLanguage.java index 7dbf10b68ac..d283514076d 100644 --- a/sd-plugin/src/main/java/org/intellij/sdk/language/SdLanguage.java +++ b/sd-plugin/src/main/java/org/intellij/sdk/language/SdLanguage.java @@ -1,7 +1,12 @@ +// Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. package org.intellij.sdk.language; import com.intellij.lang.Language; +/** + * This class defines SD as a language. + * @author shahariel + */ public class SdLanguage extends Language { public static final SdLanguage INSTANCE = new SdLanguage(); diff --git a/sd-plugin/src/main/java/org/intellij/sdk/language/SdLanguageCodeStyleSettingsProvider.java b/sd-plugin/src/main/java/org/intellij/sdk/language/SdLanguageCodeStyleSettingsProvider.java index b51ae286201..efe4e9ea0ae 100644 --- a/sd-plugin/src/main/java/org/intellij/sdk/language/SdLanguageCodeStyleSettingsProvider.java +++ b/sd-plugin/src/main/java/org/intellij/sdk/language/SdLanguageCodeStyleSettingsProvider.java @@ -1,3 +1,4 @@ +// Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. package org.intellij.sdk.language; import com.intellij.lang.Language; @@ -5,6 +6,10 @@ import com.intellij.psi.codeStyle.CodeStyleSettingsCustomizable; import com.intellij.psi.codeStyle.LanguageCodeStyleSettingsProvider; import org.jetbrains.annotations.NotNull; +/** + * This class is used for the extension (in plugin.xml), to make the IDE use our plugin's code for coding style. + * @author shahariel + */ public class SdLanguageCodeStyleSettingsProvider extends LanguageCodeStyleSettingsProvider { @NotNull diff --git a/sd-plugin/src/main/java/org/intellij/sdk/language/SdLexerAdapter.java b/sd-plugin/src/main/java/org/intellij/sdk/language/SdLexerAdapter.java index 3363ce730b8..cf4b46ab1d0 100644 --- a/sd-plugin/src/main/java/org/intellij/sdk/language/SdLexerAdapter.java +++ b/sd-plugin/src/main/java/org/intellij/sdk/language/SdLexerAdapter.java @@ -1,8 +1,13 @@ +// Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. package org.intellij.sdk.language; import com.intellij.lexer.FlexAdapter; import org.intellij.sdk.language.lexer.SdLexer; +/** + * This class adapts the JFlex lexer to the IntelliJ Platform Lexer API. + * @author shahariel + */ public class SdLexerAdapter extends FlexAdapter { public SdLexerAdapter() { diff --git a/sd-plugin/src/main/java/org/intellij/sdk/language/SdRefactoringSupportProvider.java b/sd-plugin/src/main/java/org/intellij/sdk/language/SdRefactoringSupportProvider.java index 5867729d54d..88868d324de 100644 --- a/sd-plugin/src/main/java/org/intellij/sdk/language/SdRefactoringSupportProvider.java +++ b/sd-plugin/src/main/java/org/intellij/sdk/language/SdRefactoringSupportProvider.java @@ -1,3 +1,4 @@ +// Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. package org.intellij.sdk.language; import com.intellij.lang.refactoring.RefactoringSupportProvider; @@ -6,6 +7,10 @@ import org.intellij.sdk.language.psi.SdIdentifierVal; import org.jetbrains.annotations.NotNull; import org.jetbrains.annotations.Nullable; +/** + * This class is used for the extension (in plugin.xml), to enable refactoring. + * @author shahariel + */ public class SdRefactoringSupportProvider extends RefactoringSupportProvider { @Override public boolean isMemberInplaceRenameAvailable(@NotNull PsiElement elementToRename, @Nullable PsiElement context) { diff --git a/sd-plugin/src/main/java/org/intellij/sdk/language/SdReference.java b/sd-plugin/src/main/java/org/intellij/sdk/language/SdReference.java index 891d94e15cb..f2f4d475cb2 100644 --- a/sd-plugin/src/main/java/org/intellij/sdk/language/SdReference.java +++ b/sd-plugin/src/main/java/org/intellij/sdk/language/SdReference.java @@ -1,3 +1,4 @@ +// Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. package org.intellij.sdk.language; import com.intellij.codeInsight.lookup.LookupElement; @@ -17,6 +18,10 @@ import org.jetbrains.annotations.Nullable; import java.util.ArrayList; import java.util.List; +/** + * This class represent a reference to a Psi Element. + * @author shahariel + */ public class SdReference extends PsiReferenceBase<PsiElement> implements PsiPolyVariantReference { private final String elementName; diff --git a/sd-plugin/src/main/java/org/intellij/sdk/language/SdSyntaxHighlighter.java b/sd-plugin/src/main/java/org/intellij/sdk/language/SdSyntaxHighlighter.java index ea2e6331c38..ae99b58f11f 100644 --- a/sd-plugin/src/main/java/org/intellij/sdk/language/SdSyntaxHighlighter.java +++ b/sd-plugin/src/main/java/org/intellij/sdk/language/SdSyntaxHighlighter.java @@ -1,3 +1,4 @@ +// Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. package org.intellij.sdk.language; import static com.intellij.openapi.editor.colors.TextAttributesKey.createTextAttributesKey; @@ -14,6 +15,10 @@ import org.jetbrains.annotations.NotNull; import java.util.HashSet; +/** + * This class defines the syntax highlighting of an SD file. + * @author shahariel + */ public class SdSyntaxHighlighter extends SyntaxHighlighterBase { private static final HashSet<IElementType> keyWordsSet = initKeyWordsSet(); @@ -52,9 +57,8 @@ public class SdSyntaxHighlighter extends SyntaxHighlighterBase { return new SdLexerAdapter(); } - @NotNull @Override - public TextAttributesKey[] getTokenHighlights(IElementType tokenType) { + public TextAttributesKey @NotNull [] getTokenHighlights(IElementType tokenType) { if (tokenType.equals(SdTypes.IDENTIFIER_VAL)) { return IDENTIFIER_KEYS; // } else if (tokenType.equals(SdTypes.KEY)) { diff --git a/sd-plugin/src/main/java/org/intellij/sdk/language/SdSyntaxHighlighterFactory.java b/sd-plugin/src/main/java/org/intellij/sdk/language/SdSyntaxHighlighterFactory.java index c4bc4491bf1..b2ace3156e9 100644 --- a/sd-plugin/src/main/java/org/intellij/sdk/language/SdSyntaxHighlighterFactory.java +++ b/sd-plugin/src/main/java/org/intellij/sdk/language/SdSyntaxHighlighterFactory.java @@ -1,3 +1,4 @@ +// Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. package org.intellij.sdk.language; import com.intellij.openapi.fileTypes.SyntaxHighlighter; @@ -6,6 +7,10 @@ import com.intellij.openapi.project.Project; import com.intellij.openapi.vfs.VirtualFile; import org.jetbrains.annotations.NotNull; +/** + * This class is used for the extension (in plugin.xml) to the class SdSyntaxHighlighter. + * @author shahariel + */ public class SdSyntaxHighlighterFactory extends SyntaxHighlighterFactory { @NotNull diff --git a/sd-plugin/src/main/java/org/intellij/sdk/language/SdUtil.java b/sd-plugin/src/main/java/org/intellij/sdk/language/SdUtil.java index 4dd2fe8cac1..45e5d6f4fb3 100644 --- a/sd-plugin/src/main/java/org/intellij/sdk/language/SdUtil.java +++ b/sd-plugin/src/main/java/org/intellij/sdk/language/SdUtil.java @@ -1,3 +1,4 @@ +// Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. package org.intellij.sdk.language; import com.intellij.lang.ASTNode; @@ -10,8 +11,10 @@ import com.intellij.psi.search.FileTypeIndex; import com.intellij.psi.search.GlobalSearchScope; import com.intellij.psi.util.PsiTreeUtil; import com.intellij.util.indexing.FileBasedIndex; +import org.intellij.sdk.language.psi.SdAnnotationFieldDefinition; import org.intellij.sdk.language.psi.SdArgumentDefinition; import org.intellij.sdk.language.psi.SdDeclaration; +import org.intellij.sdk.language.psi.SdDocumentAnnotationDefinition; import org.intellij.sdk.language.psi.SdDocumentDefinition; import org.intellij.sdk.language.psi.SdDocumentFieldDefinition; import org.intellij.sdk.language.psi.SdDocumentStructDefinition; @@ -23,6 +26,7 @@ import org.intellij.sdk.language.psi.SdFunctionDefinition; import org.intellij.sdk.language.psi.SdIdentifier; import org.intellij.sdk.language.psi.SdImportFieldDefinition; import org.intellij.sdk.language.psi.SdRankProfileDefinition; +import org.intellij.sdk.language.psi.SdSchemaAnnotationDefinition; import org.intellij.sdk.language.psi.SdSchemaFieldDefinition; import org.intellij.sdk.language.psi.SdSummaryDefinition; import org.intellij.sdk.language.psi.SdTypes; @@ -33,6 +37,10 @@ import java.util.Collection; import java.util.HashMap; import java.util.List; +/** + * This is the util class for the plugin's code. + * @author shahariel + */ public class SdUtil { public static @NotNull HashMap<String, List<PsiElement>> createMacrosMap(SdFile file) { @@ -110,12 +118,12 @@ public class SdUtil { if (element.getParent() instanceof SdImportFieldDefinition && element.getNextSibling().getNextSibling().getText().equals("as")) { Project project = file.getProject(); - + PsiReference docFieldRef = element.getPrevSibling().getPrevSibling().getReference(); PsiElement docField = docFieldRef != null ? docFieldRef.resolve() : null; SdFieldTypeName fieldType = docField != null ? PsiTreeUtil.findChildOfType(docField, SdFieldTypeName.class) : null; SdIdentifier docIdentifier = fieldType != null ? PsiTreeUtil.findChildOfType(fieldType, SdIdentifier.class) : null; - String docName = docIdentifier != null ? docIdentifier.getName() : null; + String docName = docIdentifier != null ? docIdentifier.getName() : null; if (docName == null) { return result; } @@ -165,8 +173,8 @@ public class SdUtil { curRankProfile = getRankProfileParent((SdRankProfileDefinition) curRankProfile); } } - - for (PsiElement declaration : PsiTreeUtil.collectElements(file, psiElement -> + + for (PsiElement declaration : PsiTreeUtil.collectElements(file, psiElement -> psiElement instanceof SdDeclaration && !(psiElement instanceof SdArgumentDefinition))) { if (name.equals(((SdDeclaration) declaration).getName())) { result.add((SdDeclaration) declaration); @@ -183,30 +191,33 @@ public class SdUtil { public static List<PsiElement> findSchemaChildren(PsiElement element) { return new ArrayList<>(PsiTreeUtil.collectElementsOfType(element, new Class[]{SdDocumentDefinition.class, - SdSchemaFieldDefinition.class, - SdImportFieldDefinition.class, - SdDocumentSummaryDefinition.class, - SdRankProfileDefinition.class})); + SdSchemaFieldDefinition.class, + SdImportFieldDefinition.class, + SdSchemaAnnotationDefinition.class, + SdDocumentSummaryDefinition.class, + SdRankProfileDefinition.class})); + } + + public static List<PsiElement> findAnnotationChildren(PsiElement element) { + return new ArrayList<>(PsiTreeUtil.collectElementsOfType(element, SdAnnotationFieldDefinition.class)); } public static List<PsiElement> findDocumentChildren(PsiElement element) { - return new ArrayList<>(PsiTreeUtil.collectElementsOfType(element, new Class[]{SdDocumentStructDefinition.class, - SdDocumentFieldDefinition.class})); + return new ArrayList<>(PsiTreeUtil.collectElementsOfType(element, new Class[]{SdDocumentStructDefinition.class, + SdDocumentAnnotationDefinition.class, + SdDocumentFieldDefinition.class})); } public static List<PsiElement> findDocumentStructChildren(PsiElement element) { - return new ArrayList<>(PsiTreeUtil.collectElementsOfType(element, - SdDocumentStructFieldDefinition.class)); + return new ArrayList<>(PsiTreeUtil.collectElementsOfType(element, SdDocumentStructFieldDefinition.class)); } public static List<PsiElement> findRankProfileChildren(PsiElement element) { - return new ArrayList<>(PsiTreeUtil.collectElementsOfType(element, - SdFunctionDefinition.class)); + return new ArrayList<>(PsiTreeUtil.collectElementsOfType(element, SdFunctionDefinition.class)); } public static List<PsiElement> findDocumentSummaryChildren(PsiElement element) { - return new ArrayList<>(PsiTreeUtil.collectElementsOfType(element, - SdSummaryDefinition.class)); + return new ArrayList<>(PsiTreeUtil.collectElementsOfType(element, SdSummaryDefinition.class)); } diff --git a/sd-plugin/src/main/java/org/intellij/sdk/language/findUsages/SdDocumentSummaryGroupingRule.java b/sd-plugin/src/main/java/org/intellij/sdk/language/findUsages/SdDocumentSummaryGroupingRule.java index d4d38c8c38a..8047e198d88 100644 --- a/sd-plugin/src/main/java/org/intellij/sdk/language/findUsages/SdDocumentSummaryGroupingRule.java +++ b/sd-plugin/src/main/java/org/intellij/sdk/language/findUsages/SdDocumentSummaryGroupingRule.java @@ -1,3 +1,4 @@ +// Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. package org.intellij.sdk.language.findUsages; import com.intellij.openapi.project.DumbAware; @@ -12,6 +13,10 @@ import org.intellij.sdk.language.psi.SdDocumentSummaryDefinition; import org.jetbrains.annotations.NotNull; import org.jetbrains.annotations.Nullable; +/** + * This class represent a Document Summary that groups elements in the "Find Usages" window. + * @author shahariel + */ public class SdDocumentSummaryGroupingRule extends SingleParentUsageGroupingRule implements DumbAware { @Override diff --git a/sd-plugin/src/main/java/org/intellij/sdk/language/findUsages/SdDocumentSummaryGroupingRuleProvider.java b/sd-plugin/src/main/java/org/intellij/sdk/language/findUsages/SdDocumentSummaryGroupingRuleProvider.java index ffd620c822e..ff519436994 100644 --- a/sd-plugin/src/main/java/org/intellij/sdk/language/findUsages/SdDocumentSummaryGroupingRuleProvider.java +++ b/sd-plugin/src/main/java/org/intellij/sdk/language/findUsages/SdDocumentSummaryGroupingRuleProvider.java @@ -1,3 +1,4 @@ +// Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. package org.intellij.sdk.language.findUsages; import com.intellij.openapi.project.Project; @@ -6,6 +7,10 @@ import com.intellij.usages.rules.UsageGroupingRule; import org.jetbrains.annotations.NotNull; import org.jetbrains.annotations.Nullable; +/** + * This class is used for the extension (in plugin.xml) to the class SdDocumentSummaryGroupingRule. + * @author shahariel + */ public class SdDocumentSummaryGroupingRuleProvider implements FileStructureGroupRuleProvider { @Override diff --git a/sd-plugin/src/main/java/org/intellij/sdk/language/findUsages/SdFindUsagesHandler.java b/sd-plugin/src/main/java/org/intellij/sdk/language/findUsages/SdFindUsagesHandler.java index f00ea940613..6841f608efd 100644 --- a/sd-plugin/src/main/java/org/intellij/sdk/language/findUsages/SdFindUsagesHandler.java +++ b/sd-plugin/src/main/java/org/intellij/sdk/language/findUsages/SdFindUsagesHandler.java @@ -1,3 +1,4 @@ +// Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. package org.intellij.sdk.language.findUsages; import com.intellij.find.findUsages.FindUsagesHandler; @@ -20,6 +21,10 @@ import org.jetbrains.annotations.NotNull; import java.util.HashMap; import java.util.List; +/** + * This class handles creating the "Find Usages" window. + * @author shahariel + */ public class SdFindUsagesHandler extends FindUsagesHandler { protected HashMap<String, List<PsiElement>> macrosMap; diff --git a/sd-plugin/src/main/java/org/intellij/sdk/language/findUsages/SdFindUsagesHandlerFactory.java b/sd-plugin/src/main/java/org/intellij/sdk/language/findUsages/SdFindUsagesHandlerFactory.java index e5a6580c8c2..2ccd7f6ea41 100644 --- a/sd-plugin/src/main/java/org/intellij/sdk/language/findUsages/SdFindUsagesHandlerFactory.java +++ b/sd-plugin/src/main/java/org/intellij/sdk/language/findUsages/SdFindUsagesHandlerFactory.java @@ -1,3 +1,4 @@ +// Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. package org.intellij.sdk.language.findUsages; import com.intellij.find.findUsages.FindUsagesHandler; @@ -7,6 +8,10 @@ import com.intellij.psi.PsiNamedElement; import org.jetbrains.annotations.NotNull; import org.jetbrains.annotations.Nullable; +/** + * This class is used for the extension (in plugin.xml) to the class SdFindUsagesHandler. + * @author shahariel + */ public class SdFindUsagesHandlerFactory extends FindUsagesHandlerFactory { @Override diff --git a/sd-plugin/src/main/java/org/intellij/sdk/language/findUsages/SdFindUsagesProvider.java b/sd-plugin/src/main/java/org/intellij/sdk/language/findUsages/SdFindUsagesProvider.java index eaa5b43641d..4ec407e580d 100644 --- a/sd-plugin/src/main/java/org/intellij/sdk/language/findUsages/SdFindUsagesProvider.java +++ b/sd-plugin/src/main/java/org/intellij/sdk/language/findUsages/SdFindUsagesProvider.java @@ -1,3 +1,4 @@ +// Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. package org.intellij.sdk.language.findUsages; import com.intellij.lang.cacheBuilder.DefaultWordsScanner; @@ -15,6 +16,10 @@ import org.intellij.sdk.language.psi.SdTypes; import org.jetbrains.annotations.NotNull; import org.jetbrains.annotations.Nullable; +/** + * This class is used for the extension (in plugin.xml), to enable "find Usages" window using the plugin code. + * @author shahariel + */ public class SdFindUsagesProvider implements FindUsagesProvider { @Nullable @Override diff --git a/sd-plugin/src/main/java/org/intellij/sdk/language/findUsages/SdRankProfileGroupingRule.java b/sd-plugin/src/main/java/org/intellij/sdk/language/findUsages/SdRankProfileGroupingRule.java index 379f70903df..77de9fd9534 100644 --- a/sd-plugin/src/main/java/org/intellij/sdk/language/findUsages/SdRankProfileGroupingRule.java +++ b/sd-plugin/src/main/java/org/intellij/sdk/language/findUsages/SdRankProfileGroupingRule.java @@ -1,3 +1,4 @@ +// Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. package org.intellij.sdk.language.findUsages; import com.intellij.openapi.project.DumbAware; @@ -12,6 +13,10 @@ import org.intellij.sdk.language.psi.SdRankProfileDefinition; import org.jetbrains.annotations.NotNull; import org.jetbrains.annotations.Nullable; +/** + * This class represent a Rank Profile that groups elements in the "Find Usages" window. + * @author shahariel + */ public class SdRankProfileGroupingRule extends SingleParentUsageGroupingRule implements DumbAware { @Override diff --git a/sd-plugin/src/main/java/org/intellij/sdk/language/findUsages/SdRankProfileGroupingRuleProvider.java b/sd-plugin/src/main/java/org/intellij/sdk/language/findUsages/SdRankProfileGroupingRuleProvider.java index 9ed8628dc44..7527b93c947 100644 --- a/sd-plugin/src/main/java/org/intellij/sdk/language/findUsages/SdRankProfileGroupingRuleProvider.java +++ b/sd-plugin/src/main/java/org/intellij/sdk/language/findUsages/SdRankProfileGroupingRuleProvider.java @@ -1,3 +1,4 @@ +// Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. package org.intellij.sdk.language.findUsages; import com.intellij.openapi.project.Project; @@ -6,6 +7,10 @@ import com.intellij.usages.rules.UsageGroupingRule; import org.jetbrains.annotations.NotNull; import org.jetbrains.annotations.Nullable; +/** + * This class is used for the extension (in plugin.xml) to the class SdRankProfileGroupingRule. + * @author shahariel + */ public class SdRankProfileGroupingRuleProvider implements FileStructureGroupRuleProvider { @Override diff --git a/sd-plugin/src/main/java/org/intellij/sdk/language/findUsages/SdUsageGroup.java b/sd-plugin/src/main/java/org/intellij/sdk/language/findUsages/SdUsageGroup.java index 21f54f86472..9042be84d75 100644 --- a/sd-plugin/src/main/java/org/intellij/sdk/language/findUsages/SdUsageGroup.java +++ b/sd-plugin/src/main/java/org/intellij/sdk/language/findUsages/SdUsageGroup.java @@ -1,3 +1,4 @@ +// Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. package org.intellij.sdk.language.findUsages; import com.intellij.navigation.ItemPresentation; @@ -15,6 +16,10 @@ import org.jetbrains.annotations.Nullable; import javax.swing.*; +/** + * This class represent a group of elements in the "Find Usages" window. + * @author shahariel + */ public class SdUsageGroup implements UsageGroup { private final VirtualFile myFile; private final SmartPsiElementPointer<SdDeclaration> myElementPointer; diff --git a/sd-plugin/src/main/java/org/intellij/sdk/language/hierarchy/SdCallHierarchyBrowser.java b/sd-plugin/src/main/java/org/intellij/sdk/language/hierarchy/SdCallHierarchyBrowser.java index 5960f1241c3..417cab307d4 100644 --- a/sd-plugin/src/main/java/org/intellij/sdk/language/hierarchy/SdCallHierarchyBrowser.java +++ b/sd-plugin/src/main/java/org/intellij/sdk/language/hierarchy/SdCallHierarchyBrowser.java @@ -1,3 +1,4 @@ +// Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. package org.intellij.sdk.language.hierarchy; import com.intellij.ide.hierarchy.CallHierarchyBrowserBase; @@ -21,6 +22,10 @@ import java.util.Map; import javax.swing.JTree; +/** + * This class is a browser for the "Call Hierarchy" window. + * @author shahariel + */ public class SdCallHierarchyBrowser extends CallHierarchyBrowserBase { public SdCallHierarchyBrowser(@NotNull Project project, diff --git a/sd-plugin/src/main/java/org/intellij/sdk/language/hierarchy/SdCallHierarchyNodeDescriptor.java b/sd-plugin/src/main/java/org/intellij/sdk/language/hierarchy/SdCallHierarchyNodeDescriptor.java index 0cf612b0058..5703739aa31 100644 --- a/sd-plugin/src/main/java/org/intellij/sdk/language/hierarchy/SdCallHierarchyNodeDescriptor.java +++ b/sd-plugin/src/main/java/org/intellij/sdk/language/hierarchy/SdCallHierarchyNodeDescriptor.java @@ -1,3 +1,4 @@ +// Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. package org.intellij.sdk.language.hierarchy; import com.intellij.ide.hierarchy.HierarchyNodeDescriptor; @@ -18,6 +19,10 @@ import org.jetbrains.annotations.NotNull; import javax.swing.Icon; +/** + * This class represents a node descriptor to a node in a tree in the "Call Hierarchy" window. + * @author shahariel + */ public class SdCallHierarchyNodeDescriptor extends HierarchyNodeDescriptor { public SdCallHierarchyNodeDescriptor(final NodeDescriptor parentDescriptor, @NotNull final PsiElement element, final boolean isBase) { @@ -68,4 +73,4 @@ public class SdCallHierarchyNodeDescriptor extends HierarchyNodeDescriptor { } -}
\ No newline at end of file +} diff --git a/sd-plugin/src/main/java/org/intellij/sdk/language/hierarchy/SdCallHierarchyProvider.java b/sd-plugin/src/main/java/org/intellij/sdk/language/hierarchy/SdCallHierarchyProvider.java index 44bfdca134f..b15835341d2 100644 --- a/sd-plugin/src/main/java/org/intellij/sdk/language/hierarchy/SdCallHierarchyProvider.java +++ b/sd-plugin/src/main/java/org/intellij/sdk/language/hierarchy/SdCallHierarchyProvider.java @@ -1,3 +1,4 @@ +// Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. package org.intellij.sdk.language.hierarchy; @@ -23,7 +24,10 @@ import com.intellij.openapi.project.Project; import com.intellij.psi.util.PsiTreeUtil; - +/** + * This class is used for the extension (in plugin.xml), to enable "Call Hierarchy" window using the plugin code. + * @author shahariel + */ public class SdCallHierarchyProvider implements HierarchyProvider { @Override diff --git a/sd-plugin/src/main/java/org/intellij/sdk/language/hierarchy/SdCallTreeStructure.java b/sd-plugin/src/main/java/org/intellij/sdk/language/hierarchy/SdCallTreeStructure.java index fb612ed851a..4a007b13f07 100644 --- a/sd-plugin/src/main/java/org/intellij/sdk/language/hierarchy/SdCallTreeStructure.java +++ b/sd-plugin/src/main/java/org/intellij/sdk/language/hierarchy/SdCallTreeStructure.java @@ -1,3 +1,4 @@ +// Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. package org.intellij.sdk.language.hierarchy; import com.intellij.ide.hierarchy.HierarchyNodeDescriptor; @@ -20,6 +21,10 @@ import java.util.HashMap; import java.util.HashSet; import java.util.List; +/** + * This abstract class represents a general tree in the "Call Hierarchy" window. + * @author shahariel + */ public abstract class SdCallTreeStructure extends HierarchyTreeStructure { protected final String myScopeType; protected final SdFile myFile; diff --git a/sd-plugin/src/main/java/org/intellij/sdk/language/hierarchy/SdCalleeTreeStructure.java b/sd-plugin/src/main/java/org/intellij/sdk/language/hierarchy/SdCalleeTreeStructure.java index 977d1948b81..0beca6d5967 100644 --- a/sd-plugin/src/main/java/org/intellij/sdk/language/hierarchy/SdCalleeTreeStructure.java +++ b/sd-plugin/src/main/java/org/intellij/sdk/language/hierarchy/SdCalleeTreeStructure.java @@ -1,3 +1,4 @@ +// Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. package org.intellij.sdk.language.hierarchy; import com.intellij.openapi.project.Project; @@ -15,6 +16,10 @@ import java.util.HashMap; import java.util.HashSet; import java.util.List; +/** + * This class represents a Callee tree in the "Call Hierarchy" window. + * @author shahariel + */ public class SdCalleeTreeStructure extends SdCallTreeStructure { public SdCalleeTreeStructure(Project project, PsiElement element, String currentScopeType) { @@ -61,4 +66,4 @@ public class SdCalleeTreeStructure extends SdCallTreeStructure { } -}
\ No newline at end of file +} diff --git a/sd-plugin/src/main/java/org/intellij/sdk/language/hierarchy/SdCallerTreeStructure.java b/sd-plugin/src/main/java/org/intellij/sdk/language/hierarchy/SdCallerTreeStructure.java index f6b21fc9ead..4a30dd7492f 100644 --- a/sd-plugin/src/main/java/org/intellij/sdk/language/hierarchy/SdCallerTreeStructure.java +++ b/sd-plugin/src/main/java/org/intellij/sdk/language/hierarchy/SdCallerTreeStructure.java @@ -1,3 +1,4 @@ +// Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. package org.intellij.sdk.language.hierarchy; import com.intellij.openapi.progress.ProgressManager; @@ -17,6 +18,10 @@ import java.util.HashSet; import java.util.List; import java.util.function.Consumer; +/** + * This class represents a Caller tree in the "Call Hierarchy" window. + * @author shahariel + */ public class SdCallerTreeStructure extends SdCallTreeStructure { private HashMap<String, HashSet<PsiElement>> macroTreeChildren; @@ -60,4 +65,4 @@ public class SdCallerTreeStructure extends SdCallTreeStructure { return results; } -}
\ No newline at end of file +} diff --git a/sd-plugin/src/main/java/org/intellij/sdk/language/hierarchy/SdHierarchyUtil.java b/sd-plugin/src/main/java/org/intellij/sdk/language/hierarchy/SdHierarchyUtil.java index b3376a85524..c3e59124ef8 100644 --- a/sd-plugin/src/main/java/org/intellij/sdk/language/hierarchy/SdHierarchyUtil.java +++ b/sd-plugin/src/main/java/org/intellij/sdk/language/hierarchy/SdHierarchyUtil.java @@ -1,3 +1,4 @@ +// Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. package org.intellij.sdk.language.hierarchy; import com.intellij.ide.hierarchy.HierarchyBrowserManager; @@ -15,7 +16,10 @@ import org.intellij.sdk.language.psi.SdRankProfileDefinition; import java.util.Comparator; import java.util.HashSet; - +/** + * This class is util class to the Call Hierarchy feature. + * @author shahariel + */ public class SdHierarchyUtil { private static final Comparator<NodeDescriptor<?>> NODE_DESCRIPTOR_COMPARATOR = Comparator.comparingInt(NodeDescriptor::getIndex); diff --git a/sd-plugin/src/main/java/org/intellij/sdk/language/parser/SdParserDefinition.java b/sd-plugin/src/main/java/org/intellij/sdk/language/parser/SdParserDefinition.java index 9d888a89cbf..2a169707a79 100644 --- a/sd-plugin/src/main/java/org/intellij/sdk/language/parser/SdParserDefinition.java +++ b/sd-plugin/src/main/java/org/intellij/sdk/language/parser/SdParserDefinition.java @@ -1,3 +1,4 @@ +// Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. package org.intellij.sdk.language.parser; import com.intellij.lang.ASTNode; @@ -18,6 +19,10 @@ import org.intellij.sdk.language.psi.SdFile; import org.intellij.sdk.language.psi.SdTypes; import org.jetbrains.annotations.NotNull; +/** + * This class is used for the extension (in plugin.xml), to make the parsing process use the plugin code. + * @author shahariel + */ public class SdParserDefinition implements ParserDefinition { public static final TokenSet WHITE_SPACES = TokenSet.create(TokenType.WHITE_SPACE); public static final TokenSet COMMENTS = TokenSet.create(SdTypes.COMMENT); @@ -84,4 +89,4 @@ public class SdParserDefinition implements ParserDefinition { -
\ No newline at end of file + diff --git a/sd-plugin/src/main/java/org/intellij/sdk/language/parser/sd.bnf b/sd-plugin/src/main/java/org/intellij/sdk/language/parser/sd.bnf index a1bc4c548a0..245975f301c 100644 --- a/sd-plugin/src/main/java/org/intellij/sdk/language/parser/sd.bnf +++ b/sd-plugin/src/main/java/org/intellij/sdk/language/parser/sd.bnf @@ -1,3 +1,9 @@ +/** +This file is the SD grammar. +NOTE: This grammar does not enforce zero-or-one occurrences of elements (treats it like zero-to-many) +@author: shahariel + */ + { parserClass="org.intellij.sdk.language.parser.SdParser" // Name and the location of the parser which will be generated. @@ -34,19 +40,15 @@ ] } -// IMPORTANT NOTE: This grammar does not enforce zero-or-one occurrences of elements (treats it like zero-to-many) - -SdFile ::= Schema -Schema ::= search IdentifierVal? '{' SchemaBody '}' +SdFile ::= SchemaDefinition +SchemaDefinition ::= (search | schema) IdentifierVal? '{' SchemaBody '}' SchemaBody ::= SchemaBodyOptions* DocumentDefinition SchemaBodyOptions* // Does not support zero-or-one occurrences private SchemaBodyOptions ::= SchemaFieldDefinition | ImportFieldDefinition | DocumentSummaryDefinition | RankProfileDefinition | FieldSetDefinition | ConstantDefinition | OnnxModelDefinition | StemmingDefinition | - raw-as-base64-in-summary | AnnotationDefinition - - - + raw-as-base64-in-summary | SchemaAnnotationDefinition + SchemaFieldDefinition ::= field IdentifierVal type FieldTypeName '{' SchemaFieldBody '}' { mixin="org.intellij.sdk.language.psi.impl.SdNamedElementImpl" implements=["org.intellij.sdk.language.psi.SdDeclaration" "org.intellij.sdk.language.psi.SdNamedElement"] @@ -95,19 +97,26 @@ OnnxModelBody ::= OnnxModelBodyOptions* private OnnxModelBodyOptions ::= (file ':' FilePath) | (uri ':' UriPath) | ((input | output) (IdentifierVal | STRING) ':' ('.' | '/' | '(' | ')' | IdentifierWithDashVal | WORD_REG)) -AnnotationDefinition ::= annotation IdentifierVal '{' '}' // todo ask Vespa for syntax +SchemaAnnotationDefinition ::= AnnotationDefinition + { mixin="org.intellij.sdk.language.psi.impl.SdNamedElementImpl" + implements=["org.intellij.sdk.language.psi.SdDeclaration" "org.intellij.sdk.language.psi.SdNamedElement"] + methods=[getName setName getType getTypeName getNameIdentifier getPresentation] } + +private AnnotationDefinition ::= annotation IdentifierVal '{' AnnotationFieldDefinition* '}' +AnnotationFieldDefinition ::= field IdentifierVal type FieldTypeName '{' '}' + { mixin="org.intellij.sdk.language.psi.impl.SdNamedElementImpl" + implements=["org.intellij.sdk.language.psi.SdDeclaration" "org.intellij.sdk.language.psi.SdNamedElement"] + methods=[getName setName getType getTypeName getNameIdentifier getPresentation] } //------------------------- //--- Expressions rules --- //------------------------- RankingExpression ::= ParenthesisedExpr | BooleanExpr |ArithmeticExpr | IfFunctionExpr | - QueryDefinitionExpr | FunctionCallExpr | PrimitiveExpr //ReduceExpr | + QueryDefinitionExpr | FunctionCallExpr | PrimitiveExpr IfFunctionExpr ::= "if" '(' (InListRankingExpression | RankingExpression) ',' RankingExpression ',' RankingExpression ')' InListRankingExpression ::= RankingExpression "in" '[' RankingExpression (',' RankingExpression)* ']' -//ReduceExpr ::= reduce '(' RankingExpression ',' (avg|count|max|median|min|prod|sum) (',' RankingExpression)* ')' - BooleanExpr ::= RankingExpression COMPARISON_OPERATOR RankingExpression ArithmeticExpr ::= RankingExpression ARITHMETIC_OPERATOR RankingExpression @@ -132,14 +141,15 @@ private RankProfileBodyOptions ::= MatchPhaseDefinition | NumThreadsDefinition | ignore-default-rank-features | RankPropertiesDefinition | FirstPhaseDefinition | SummaryFeaturesDefinition | RankFeaturesDefinition | SecondPhaseDefinition | ConstantsDefinition | RankDefinition | RankTypeDefinition | MinHitsDefinition | NumSearchPartitionDefinition - // | FieldWeightDefinition | ExecuteDefinition | RankDegradationDefinition // todo check with Vespa if need to add these MatchPhaseDefinition ::= match-phase '{' MatchPhaseBody '}' -MatchPhaseBody ::= MatchPhaseBodyOptions+ // todo check with Vespa- are there more options? +MatchPhaseBody ::= MatchPhaseBodyOptions+ MatchPhaseBodyOptions ::= (attribute ':' IdentifierVal (order ':' (ascending | descending))?) | (max-hits ':' INTEGER_REG) | DiversityDefinition // Does not support zero-or-one occurrences -DiversityDefinition ::= diversity '{' (attribute ':' IdentifierVal min-groups ':' INTEGER_REG) | - (min-groups ':' INTEGER_REG attribute ':' IdentifierVal) '}' +DiversityDefinition ::= diversity '{' DiversityBody '}' +DiversityBody ::= DiversityBodyOptions* +private DiversityBodyOptions ::= (attribute ':' IdentifierVal) | (min-groups ':' INTEGER_REG) | (cutoff-factor ':' FLOAT_REG) | + (cutoff-strategy ':' (strict | loose)) private NumThreadsDefinition ::= num-threads-per-search ':' INTEGER_REG private TermwiseLimitDefinition ::= termwise-limit ':' (FLOAT_REG | INTEGER_REG) @@ -177,9 +187,7 @@ RankFeaturesDefinition ::= rank-features (':' RankFeature+) | ('{' RankFeature* ConstantsDefinition ::= constants '{' (IdentifierVal ':' RankPropertiesValue)* '}' -//******** Rank features ********* RankFeature ::= QueryDefinition | ItemRawScoreDefinition | FunctionCallExpr | (IdentifierWithDashVal ('.' IdentifierWithDashVal)* ) - QueryDefinition ::= "query" '(' IdentifierWithDashVal ')' { mixin="org.intellij.sdk.language.psi.impl.SdNamedElementImpl" implements=["org.intellij.sdk.language.psi.SdDeclaration" "org.intellij.sdk.language.psi.SdNamedElement"] @@ -188,30 +196,7 @@ ItemRawScoreDefinition ::= "itemRawScore" '(' IdentifierVal ')' { mixin="org.intellij.sdk.language.psi.impl.SdNamedElementImpl" implements=["org.intellij.sdk.language.psi.SdDeclaration" "org.intellij.sdk.language.psi.SdNamedElement"] methods=[getName setName getType getTypeName getNameIdentifier getPresentation] } - -// QueryFeature | DocumentFeature | QueryTermFieldMatchFeature | AttributeMatchFeature | RankScoreFeature | -// GlobalFeature | MatchOperatorScoreFeature | UtilityFeature | RankingExpressionMethod - //todo add the rest (| FieldMatchFeature | QueryFieldSimilarityFeature | IdxMultiStrFieldFeature | MultiFieldsAttributesFeature) - -// todo maybe not specify all of the rank features here? Maybe it would be better to write it more generally? like ID '(' ID ')' '.' ID.. so it could handle new features in the future -//QueryFeature ::= QueryDefinition | (term '(' (FLOAT_REG | INTEGER_REG) ')' '.' (significant | weight | connectedness)) // todo add queryTermCount -//DocumentFeature ::= (fieldLength '(' IdentifierVal ')') | (attribute '(' IdentifierVal ')' ('.' count)?) | -// (attribute '(' IdentifierVal ',' (INTEGER_REG | IdentifierVal) ')' ('.' (weight | contains))?) | -// (tensorFromWeightedSet '(' ((attribute'('IdentifierVal')') | QueryDefinition | IdentifierVal) ',' IdentifierVal ')') | -// (tensorFromLabels '(' ((attribute'('IdentifierVal')') | QueryDefinition | IdentifierVal) ',' IdentifierVal ')') -//QueryTermFieldMatchFeature ::= (matchCount'('IdentifierVal')') | (matches'('IdentifierVal (',' INTEGER_REG)? ')') // todo add the rest -//AttributeMatchFeature ::= (attributeMatch'('IdentifierVal')' ('.' (completeness | queryCompleteness | fieldCompleteness | -// normalizedWeight | normalizedWeightedWeight | matches | totalWeight | averageWeight | maxWeight))?) | -// (distance'('IdentifierVal')' ('.' (index | latitude | longitude))?) | -// (distanceToPath'('IdentifierVal')' ('.' (distance | traveled | product))?) | (age'('IdentifierVal')') // todo add closeness and freshness -//RankScoreFeature ::= (nativeDotProduct'('IdentifierVal')') // todo add the rest -//GlobalFeature ::= (random'('IdentifierVal')' '.' match) | (random) // todo add the rest -//MatchOperatorScoreFeature ::= (rawScore'('IdentifierVal')') | ItemRawScoreDefinition -//UtilityFeature ::= (dotProduct'('IdentifierVal ',' IdentifierVal ')') // todo add the rest -//RankingExpressionMethod ::= "rankingExpression" '(' (RankFeature | IdentifierVal) ')' -//***** End of Rank features ***** - - + //------------------------- //---- Document rules ----- //------------------------- @@ -221,7 +206,12 @@ DocumentDefinition ::= document (IdentifierVal (inherits IdentifierVal (',' Iden methods=[getName setName getType getTypeName getNameIdentifier getPresentation] } DocumentBody ::= DocumentBodyOptions* -DocumentBodyOptions ::= DocumentStructDefinition | DocumentFieldDefinition +DocumentBodyOptions ::= DocumentStructDefinition | DocumentFieldDefinition | DocumentAnnotationDefinition + +DocumentAnnotationDefinition ::= AnnotationDefinition + { mixin="org.intellij.sdk.language.psi.impl.SdNamedElementImpl" + implements=["org.intellij.sdk.language.psi.SdDeclaration" "org.intellij.sdk.language.psi.SdNamedElement"] + methods=[getName setName getType getTypeName getNameIdentifier getPresentation] } DocumentStructDefinition ::= struct IdentifierVal '{' DocumentStructBody '}' { mixin="org.intellij.sdk.language.psi.impl.SdNamedElementImpl" @@ -246,7 +236,7 @@ private DocumentFieldBodyOptions ::= StructFieldDefinition | MatchDefinition | I AliasDef | RankDefinition | IndexingRewriteState | QueryCommandDefinition | SummaryDefinition | BoldingDefinition | (id ':' INTEGER_REG) | IndexDefinition | (normalizing ':' IdentifierWithDashVal) | SortingDefinition | StemmingDefinition | (weight ': INTEGER_REG') | WeightedSetDefinition | - RankTypeDefinition | DictionaryDefinition // todo check with Vespa- SummaryToDefinition is needed here? it's deprecated + RankTypeDefinition | DictionaryDefinition //***** Field's body elements ******// // Struct @@ -265,7 +255,7 @@ MatchProperty ::= text | exact | exact-terminator | word | prefix | cased | unca // Indexing IndexingDefinition ::= indexing (':' IndexingStatement) | ('{' IndexingStatement+ '}') IndexingStatement ::= IndexingStatementOptions (('|' IndexingStatementOptions)*) | ((';' IndexingStatementOptions)*) - // Does not support zero-or-one occurrences // todo check with Vespa- Can "input" be here? + // Does not support zero-or-one occurrences IndexingStatementOptions ::= summary | attribute | index | "set_language" // Attribute AttributeDefinition ::= attribute ((':' SimpleAttributeProperty) | ('{' (SimpleAttributeProperty | ComplexAttributeProperty)+ '}')) @@ -278,7 +268,7 @@ AliasDef ::= alias IdentifierVal? ':' IdentifierWithDashVal StemmingDefinition ::= stemming ':' IdentifierWithDashVal // Rank RankDefinition ::= rank ((IdentifierVal? ':' RankingSetting) | ('{' RankingSetting '}')) -RankingSetting ::= filter | normal | literal // todo check with Vespa- is "literal" good here? +RankingSetting ::= filter | normal // Indexing Rewrite IndexingRewriteState ::= indexing-rewrite ':' none // Query Command @@ -330,19 +320,19 @@ IdentifierWithDashVal ::= ID_WITH_DASH_REG | IdentifierVal { implements=["org.in // Those lists of keywords (KeywordOrIdentifier and KeywordNotIdentifier) have to be synchronized with sd.flex file. // If you add a keyword here, you should add it to the sd.flex file as well. -KeywordOrIdentifier ::= search | document | struct | field | type | indexing | input | output | inherits | import | as | - raw | uri | file | annotationreference | array | weightedset | map | - order | ascending | descending | diversity | constants | literal | expression | weight | match | +KeywordOrIdentifier ::= schema | search | document | struct | field | type | indexing | input | output | inherits | + import | as | raw | uri | file | annotationreference | array | weightedset | map | + order | ascending | descending | diversity | constants | expression | weight | match | function | macro | inline | text | exact | word | prefix | cased | uncased | substring | suffix | gram | paged | mutable | alias | sorting | strength | locale | uca | lowercase | primary | secondary | tertiary | quaternary | identical | rank | filter | normal | none | full | dynamic | - source | to | + source | to | strict | loose | bolding | on | off | true | false | id | normalizing | stemming | arity | hnsw | dictionary | hash | btree | fieldset | fields | constant | annotation - // KeywordNotIdentifier ::= (not enforced in this version) | attribute | body | header | index | reference | summary +// Note- in this form, those keywords can't be use as identifier-with-dash! KeywordNotIdentifier ::= struct-field | document-summary | omit-summary-features | from-disk | rank-profile | rank-type | num-threads-per-search | termwise-limit | ignore-default-rank-features | min-hits-per-thread | num-search-partition | match-phase | max-hits | second-phase | rerank-count | min-groups | @@ -351,5 +341,5 @@ KeywordNotIdentifier ::= struct-field | document-summary | omit-summary-features indexing-rewrite | query-command | matched-elements-only | lower-bound | upper-bound | dense-posting-list-threshold | enable-bm25 | max-links-per-node | neighbors-to-explore-at-insert | multi-threaded-indexing | create-if-nonexistent | remove-if-zero | raw-as-base64-in-summary | - onnx-model + onnx-model | cutoff-factor | cutoff-strategy | on-match | on-rank | on-summary
\ No newline at end of file diff --git a/sd-plugin/src/main/java/org/intellij/sdk/language/psi/SdDeclaration.java b/sd-plugin/src/main/java/org/intellij/sdk/language/psi/SdDeclaration.java index c46466ddb84..25ef64b8936 100644 --- a/sd-plugin/src/main/java/org/intellij/sdk/language/psi/SdDeclaration.java +++ b/sd-plugin/src/main/java/org/intellij/sdk/language/psi/SdDeclaration.java @@ -1,9 +1,14 @@ +// Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. package org.intellij.sdk.language.psi; import com.intellij.navigation.NavigationItem; import com.intellij.psi.PsiElement; import com.intellij.psi.PsiNamedElement; +/** + * This interface represents a declaration in the SD language. + * @author shahariel + */ public interface SdDeclaration extends PsiElement, PsiNamedElement, NavigationItem { String getName(); diff --git a/sd-plugin/src/main/java/org/intellij/sdk/language/psi/SdDeclarationType.java b/sd-plugin/src/main/java/org/intellij/sdk/language/psi/SdDeclarationType.java index f832a0d01e2..5ea5968f0bb 100644 --- a/sd-plugin/src/main/java/org/intellij/sdk/language/psi/SdDeclarationType.java +++ b/sd-plugin/src/main/java/org/intellij/sdk/language/psi/SdDeclarationType.java @@ -1,11 +1,18 @@ +// Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. package org.intellij.sdk.language.psi; +/** + * This Enum describes the different declarations' types and their names. + * @author shahariel + */ public enum SdDeclarationType { DOCUMENT("Document"), STRUCT("Struct"), + ANNOTATION("Annotation"), SCHEMA_FIELD("Field (in Schema)"), DOCUMENT_FIELD("Field (in Document)"), STRUCT_FIELD("Struct-Field"), + ANNOTATION_FIELD("Field (in Annotation)"), DOCUMENT_STRUCT_FIELD("Field (in Struct)"), IMPORTED_FIELD("Imported Field"), DOCUMENT_SUMMARY("Document-Summary"), diff --git a/sd-plugin/src/main/java/org/intellij/sdk/language/psi/SdElementDescriptionProvider.java b/sd-plugin/src/main/java/org/intellij/sdk/language/psi/SdElementDescriptionProvider.java index ea588f965e5..3801069d408 100644 --- a/sd-plugin/src/main/java/org/intellij/sdk/language/psi/SdElementDescriptionProvider.java +++ b/sd-plugin/src/main/java/org/intellij/sdk/language/psi/SdElementDescriptionProvider.java @@ -1,3 +1,4 @@ +// Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. package org.intellij.sdk.language.psi; import com.intellij.psi.ElementDescriptionLocation; @@ -8,13 +9,16 @@ import org.intellij.sdk.language.psi.SdDeclaration; import org.jetbrains.annotations.NotNull; import org.jetbrains.annotations.Nullable; - +/** + * This class is used for the extension (in plugin.xml), to enable "find Usages" window take the element description from + * here. Used only for the "target" element. + * @author shahariel + */ public class SdElementDescriptionProvider implements ElementDescriptionProvider { /** * Controls the headline of the element in the "Find Usages" window * @param psiElement the element to describe - * @param elementDescriptionLocation * @return a string with the description to write in the headline */ @Nullable diff --git a/sd-plugin/src/main/java/org/intellij/sdk/language/psi/SdElementFactory.java b/sd-plugin/src/main/java/org/intellij/sdk/language/psi/SdElementFactory.java index 0961704443c..9cb9690cd5b 100644 --- a/sd-plugin/src/main/java/org/intellij/sdk/language/psi/SdElementFactory.java +++ b/sd-plugin/src/main/java/org/intellij/sdk/language/psi/SdElementFactory.java @@ -1,3 +1,4 @@ +// Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. package org.intellij.sdk.language.psi; import com.intellij.openapi.project.Project; @@ -6,6 +7,10 @@ import com.intellij.psi.PsiFileFactory; import com.intellij.psi.util.PsiTreeUtil; import org.intellij.sdk.language.SdFileType; +/** + * This class is a factory of psi elements in the SD PSI tree. + * @author shahariel + */ public class SdElementFactory { private static final String GENERAL_FILE_TEXT = "search {document %s {} rank-profile %s {}}"; diff --git a/sd-plugin/src/main/java/org/intellij/sdk/language/psi/SdElementType.java b/sd-plugin/src/main/java/org/intellij/sdk/language/psi/SdElementType.java index a17ae3c1c07..a091d444d8e 100644 --- a/sd-plugin/src/main/java/org/intellij/sdk/language/psi/SdElementType.java +++ b/sd-plugin/src/main/java/org/intellij/sdk/language/psi/SdElementType.java @@ -1,3 +1,4 @@ +// Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. package org.intellij.sdk.language.psi; import com.intellij.psi.tree.IElementType; @@ -5,6 +6,10 @@ import org.intellij.sdk.language.SdLanguage; import org.jetbrains.annotations.NonNls; import org.jetbrains.annotations.NotNull; +/** + * This class represent a SdElementType. + * @author shahariel + */ public class SdElementType extends IElementType { public SdElementType(@NotNull @NonNls String debugName) { diff --git a/sd-plugin/src/main/java/org/intellij/sdk/language/psi/SdFile.java b/sd-plugin/src/main/java/org/intellij/sdk/language/psi/SdFile.java index 0e0a8b0661c..c044bdeeddd 100644 --- a/sd-plugin/src/main/java/org/intellij/sdk/language/psi/SdFile.java +++ b/sd-plugin/src/main/java/org/intellij/sdk/language/psi/SdFile.java @@ -1,3 +1,4 @@ +// Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. package org.intellij.sdk.language.psi; import com.intellij.extapi.psi.PsiFileBase; @@ -7,6 +8,10 @@ import org.intellij.sdk.language.SdFileType; import org.intellij.sdk.language.SdLanguage; import org.jetbrains.annotations.NotNull; +/** + * This class represent an SD file. + * @author shahariel + */ public class SdFile extends PsiFileBase { public SdFile(@NotNull FileViewProvider viewProvider) { diff --git a/sd-plugin/src/main/java/org/intellij/sdk/language/psi/SdIdentifier.java b/sd-plugin/src/main/java/org/intellij/sdk/language/psi/SdIdentifier.java index 27009e9369c..c88e3e593bb 100644 --- a/sd-plugin/src/main/java/org/intellij/sdk/language/psi/SdIdentifier.java +++ b/sd-plugin/src/main/java/org/intellij/sdk/language/psi/SdIdentifier.java @@ -1,9 +1,14 @@ +// Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. package org.intellij.sdk.language.psi; import com.intellij.psi.PsiElement; import com.intellij.psi.PsiReference; import org.jetbrains.annotations.NotNull; +/** + * This interface represents an identifier in the SD language (regular identifiers and identifiers with dash). + * @author shahariel + */ public interface SdIdentifier extends PsiElement { String getName(); diff --git a/sd-plugin/src/main/java/org/intellij/sdk/language/psi/SdNamedElement.java b/sd-plugin/src/main/java/org/intellij/sdk/language/psi/SdNamedElement.java index 7f4133246a0..bd74a87b551 100644 --- a/sd-plugin/src/main/java/org/intellij/sdk/language/psi/SdNamedElement.java +++ b/sd-plugin/src/main/java/org/intellij/sdk/language/psi/SdNamedElement.java @@ -1,7 +1,13 @@ +// Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. package org.intellij.sdk.language.psi; import com.intellij.psi.PsiNameIdentifierOwner; +/** + * This interface is used to wrap a Psi Element with SdNamedElement interface, which enables the element to be a + * "name owner" (like an identifier). It allows the element to take a part in references, find usages and more. + * @author shahariel + */ public interface SdNamedElement extends PsiNameIdentifierOwner { } diff --git a/sd-plugin/src/main/java/org/intellij/sdk/language/psi/SdTokenType.java b/sd-plugin/src/main/java/org/intellij/sdk/language/psi/SdTokenType.java index ec587ceeb86..98e7494389d 100644 --- a/sd-plugin/src/main/java/org/intellij/sdk/language/psi/SdTokenType.java +++ b/sd-plugin/src/main/java/org/intellij/sdk/language/psi/SdTokenType.java @@ -1,3 +1,4 @@ +// Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. package org.intellij.sdk.language.psi; import com.intellij.psi.tree.IElementType; @@ -5,6 +6,10 @@ import org.intellij.sdk.language.SdLanguage; import org.jetbrains.annotations.NonNls; import org.jetbrains.annotations.NotNull; +/** + * This class represent a SdTokenType. + * @author shahariel + */ public class SdTokenType extends IElementType { public SdTokenType(@NotNull @NonNls String debugName) { diff --git a/sd-plugin/src/main/java/org/intellij/sdk/language/psi/impl/SdNamedElementImpl.java b/sd-plugin/src/main/java/org/intellij/sdk/language/psi/impl/SdNamedElementImpl.java index 999fc4bd290..2901a642904 100644 --- a/sd-plugin/src/main/java/org/intellij/sdk/language/psi/impl/SdNamedElementImpl.java +++ b/sd-plugin/src/main/java/org/intellij/sdk/language/psi/impl/SdNamedElementImpl.java @@ -1,3 +1,4 @@ +// Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. package org.intellij.sdk.language.psi.impl; import com.intellij.extapi.psi.ASTWrapperPsiElement; @@ -5,6 +6,11 @@ import com.intellij.lang.ASTNode; import org.intellij.sdk.language.psi.SdNamedElement; import org.jetbrains.annotations.NotNull; +/** + * This abstract class is used to wrap a Psi Element with SdNamedElement interface, which enables the element to be a + * "name owner" (like an identifier). It allows the element to take a part in references, find usages and more. + * @author shahariel + */ public abstract class SdNamedElementImpl extends ASTWrapperPsiElement implements SdNamedElement { public SdNamedElementImpl(@NotNull ASTNode node) { diff --git a/sd-plugin/src/main/java/org/intellij/sdk/language/psi/impl/SdPsiImplUtil.java b/sd-plugin/src/main/java/org/intellij/sdk/language/psi/impl/SdPsiImplUtil.java index d22caa5adaa..bbe140070fe 100644 --- a/sd-plugin/src/main/java/org/intellij/sdk/language/psi/impl/SdPsiImplUtil.java +++ b/sd-plugin/src/main/java/org/intellij/sdk/language/psi/impl/SdPsiImplUtil.java @@ -1,3 +1,4 @@ +// Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. package org.intellij.sdk.language.psi.impl; import com.intellij.icons.AllIcons; @@ -5,16 +6,16 @@ import com.intellij.lang.ASTNode; import com.intellij.navigation.ItemPresentation; import com.intellij.openapi.util.TextRange; import com.intellij.psi.PsiElement; -import com.intellij.psi.PsiFile; import com.intellij.psi.PsiReference; import com.intellij.psi.util.PsiTreeUtil; import org.intellij.sdk.language.SdIcons; import org.intellij.sdk.language.SdReference; import org.intellij.sdk.language.SdUtil; -import org.intellij.sdk.language.hierarchy.SdHierarchyUtil; +import org.intellij.sdk.language.psi.SdAnnotationFieldDefinition; import org.intellij.sdk.language.psi.SdArgumentDefinition; import org.intellij.sdk.language.psi.SdDeclaration; import org.intellij.sdk.language.psi.SdDeclarationType; +import org.intellij.sdk.language.psi.SdDocumentAnnotationDefinition; import org.intellij.sdk.language.psi.SdDocumentDefinition; import org.intellij.sdk.language.psi.SdDocumentFieldDefinition; import org.intellij.sdk.language.psi.SdDocumentStructDefinition; @@ -31,6 +32,7 @@ import org.intellij.sdk.language.psi.SdImportFieldDefinition; import org.intellij.sdk.language.psi.SdItemRawScoreDefinition; import org.intellij.sdk.language.psi.SdQueryDefinition; import org.intellij.sdk.language.psi.SdRankProfileDefinition; +import org.intellij.sdk.language.psi.SdSchemaAnnotationDefinition; import org.intellij.sdk.language.psi.SdSchemaFieldDefinition; import org.intellij.sdk.language.psi.SdStructFieldDefinition; import org.intellij.sdk.language.psi.SdSummaryDefinition; @@ -41,8 +43,9 @@ import org.jetbrains.annotations.Nullable; import javax.swing.Icon; /** - * I this class there are implementations of methods from rules in the .bnf file. While generating the psi files + * In this class there are implementations of methods of rules in the .bnf file. While generating the psi files * (classes and interfaces) the implementations would be taken from here. + * @author shahariel */ public class SdPsiImplUtil { @@ -68,12 +71,17 @@ public class SdPsiImplUtil { return SdDeclarationType.DOCUMENT; } else if (declaration instanceof SdDocumentStructDefinition) { return SdDeclarationType.STRUCT; + } else if (declaration instanceof SdSchemaAnnotationDefinition || + declaration instanceof SdDocumentAnnotationDefinition) { + return SdDeclarationType.ANNOTATION; } else if (declaration instanceof SdDocumentStructFieldDefinition) { return SdDeclarationType.DOCUMENT_STRUCT_FIELD; } else if (declaration instanceof SdDocumentFieldDefinition) { return SdDeclarationType.DOCUMENT_FIELD; } else if (declaration instanceof SdStructFieldDefinition) { return SdDeclarationType.STRUCT_FIELD; + } else if (declaration instanceof SdAnnotationFieldDefinition) { + return SdDeclarationType.ANNOTATION_FIELD; } else if (declaration instanceof SdQueryDefinition) { return SdDeclarationType.QUERY; } else if (declaration instanceof SdItemRawScoreDefinition) { @@ -119,7 +127,7 @@ public class SdPsiImplUtil { // ################################### // // ##### getName implementations ##### // // ################################### // - + @NotNull public static String getName(SdIdentifier element) { if (element != null) { @@ -136,7 +144,7 @@ public class SdPsiImplUtil { if (declaration instanceof SdImportFieldDefinition) { ASTNode asNode = declaration.getNode().findChildByType(SdTypes.AS); node = declaration.getNode().findChildByType(SdTypes.IDENTIFIER_VAL, asNode); - } else if (declaration instanceof SdRankProfileDefinition || declaration instanceof SdDocumentSummaryDefinition + } else if (declaration instanceof SdRankProfileDefinition || declaration instanceof SdDocumentSummaryDefinition || declaration instanceof SdQueryDefinition) { node = declaration.getNode().findChildByType(SdTypes.IDENTIFIER_WITH_DASH_VAL); } else { @@ -253,7 +261,7 @@ public class SdPsiImplUtil { public String getPresentableText() { if (element instanceof SdFunctionDefinition) { return SdUtil.createFunctionDescription((SdFunctionDefinition) element); - } + } SdRankProfileDefinition rankProfileParent = PsiTreeUtil.getParentOfType(element, SdRankProfileDefinition.class); if (rankProfileParent != null) { if (element instanceof SdQueryDefinition || element instanceof SdItemRawScoreDefinition) { @@ -276,7 +284,8 @@ public class SdPsiImplUtil { if (element instanceof SdFile) { return SdIcons.FILE; } else if (element instanceof SdSchemaFieldDefinition || element instanceof SdDocumentFieldDefinition || - element instanceof SdQueryDefinition || element instanceof SdItemRawScoreDefinition) { + element instanceof SdAnnotationFieldDefinition || element instanceof SdQueryDefinition || + element instanceof SdItemRawScoreDefinition) { return AllIcons.Nodes.Field; } else if (element instanceof SdStructFieldDefinition || element instanceof SdDocumentStructFieldDefinition) { @@ -295,6 +304,9 @@ public class SdPsiImplUtil { return SdIcons.DOCUMENT_SUMMARY; } else if (element instanceof SdDocumentDefinition) { return SdIcons.DOCUMENT; + } else if (element instanceof SdSchemaAnnotationDefinition || + element instanceof SdDocumentAnnotationDefinition) { + return AllIcons.Nodes.ObjectTypeAttribute; } else { return null; @@ -305,7 +317,7 @@ public class SdPsiImplUtil { public static ItemPresentation getPresentation(final SdSummaryDefinition element) { return new ItemPresentation() { - + @Override public String getPresentableText() { return element.getName(); @@ -326,7 +338,7 @@ public class SdPsiImplUtil { public static ItemPresentation getPresentation(final SdFirstPhaseDefinition element) { return new ItemPresentation() { - + @Override public String getPresentableText() { SdRankProfileDefinition rankProfile = PsiTreeUtil.getParentOfType(element, SdRankProfileDefinition.class); diff --git a/sd-plugin/src/main/java/org/intellij/sdk/language/sd.flex b/sd-plugin/src/main/java/org/intellij/sdk/language/sd.flex index c657af4f658..02c14a53358 100644 --- a/sd-plugin/src/main/java/org/intellij/sdk/language/sd.flex +++ b/sd-plugin/src/main/java/org/intellij/sdk/language/sd.flex @@ -1,12 +1,19 @@ package org.intellij.sdk.language.lexer; import com.intellij.lexer.FlexLexer; -import com.intellij.psi.tree.IElementType;import com.intellij.ui.components.MultiColumnList;import org.intellij.sdk.language.psi.SdTokenType; +import com.intellij.psi.tree.IElementType; +import com.intellij.ui.components.MultiColumnList; +import org.intellij.sdk.language.psi.SdTokenType; -import static com.intellij.psi.TokenType.BAD_CHARACTER; // Pre-defined bad character token. -import static com.intellij.psi.TokenType.WHITE_SPACE; // Pre-defined whitespace character token. import static org.intellij.sdk.language.psi.SdTypes.*; // That is the class which is specified as `elementTypeHolderClass` in bnf // grammar file. This will contain all other tokens which we will use. +import static com.intellij.psi.TokenType.BAD_CHARACTER; // Pre-defined bad character token. +import static com.intellij.psi.TokenType.WHITE_SPACE; // Pre-defined whitespace character token. + +/* +* This file is used for the generation of the lexer of the SD language. +* @author shahariel +*/ %% @@ -37,11 +44,19 @@ WORD = \w+ %% <YYINITIAL> { - // In here, we match keywords. So if a keyword is found, this returns a token which corresponds to that keyword. - // These tokens are generated using the 'sd.bnf' file and located in the SdTypes class. - // These tokens are Parsed uses these return values to match token squence to a parser rule. + /** + In here, we match keywords. So if a keyword is found, this returns a token which corresponds to that keyword. + These tokens are generated using the 'sd.bnf' file and located in the SdTypes class. + These tokens are Parsed uses these return values to match token squence to a parser rule. + */ + + /** + This list of keywords has to be synchronized with sd.bnf file. If you add a keyword here, you should add it to the + sd.bnf file as well (to the rule KeywordOrIdentifier / KeywordNotIdentifier). + */ "search" { return SEARCH; } + "schema" { return SCHEMA; } "document" { return DOCUMENT; } "inherits" { return INHERITS; } "struct" { return STRUCT; } @@ -118,6 +133,10 @@ WORD = \w+ "max-hits" { return MAX_HITS; } "diversity" { return DIVERSITY; } "min-groups" { return MIN_GROUPS; } + "cutoff-factor" { return CUTOFF_FACTOR; } + "cutoff-strategy" { return CUTOFF_STRATEGY; } + "loose" { return LOOSE; } + "strict" { return STRICT; } "rank-properties" { return RANK_PROPERTIES; } "first-phase" { return FIRST_PHASE; } @@ -132,7 +151,6 @@ WORD = \w+ "min-hits-per-thread" { return MIN_HITS_PER_THREAD; } "num-search-partition" { return NUM_SEARCH_PARTITION; } "constants" { return CONSTANTS; } - "literal" { return LITERAL; } "second-phase" { return SECOND_PHASE; } "rerank-count" { return RERANK_COUNT; } @@ -170,6 +188,9 @@ WORD = \w+ "rank-type" { return RANK_TYPE; } "onnx-model" { return ONNX_MODEL; } "raw-as-base64-in-summary" { return RAW_AS_BASE64_IN_SUMMARY; } + "on-match" { return ON_MATCH; } + "on-rank" { return ON_RANK; } + "on-summary" { return ON_SUMMARY; } "function" { return FUNCTION; } "macro" { return MACRO; } diff --git a/sd-plugin/src/main/java/org/intellij/sdk/language/structure/SdStructureViewElement.java b/sd-plugin/src/main/java/org/intellij/sdk/language/structure/SdStructureViewElement.java index a5f45e6b26b..493047c0363 100644 --- a/sd-plugin/src/main/java/org/intellij/sdk/language/structure/SdStructureViewElement.java +++ b/sd-plugin/src/main/java/org/intellij/sdk/language/structure/SdStructureViewElement.java @@ -1,3 +1,4 @@ +// Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. package org.intellij.sdk.language.structure; import com.intellij.ide.projectView.PresentationData; @@ -8,61 +9,67 @@ import com.intellij.navigation.ItemPresentation; import com.intellij.psi.NavigatablePsiElement; import com.intellij.psi.PsiElement; import org.intellij.sdk.language.SdUtil; +import org.intellij.sdk.language.psi.SdDocumentAnnotationDefinition; import org.intellij.sdk.language.psi.SdDocumentDefinition; import org.intellij.sdk.language.psi.SdDocumentStructDefinition; import org.intellij.sdk.language.psi.SdDocumentSummaryDefinition; import org.intellij.sdk.language.psi.SdFile; import org.intellij.sdk.language.psi.SdRankProfileDefinition; +import org.intellij.sdk.language.psi.SdSchemaAnnotationDefinition; import org.jetbrains.annotations.NotNull; import java.util.ArrayList; import java.util.List; +/** + * This class is used for the presentation of an element in the "Structure View" window. + * @author shahariel + */ public class SdStructureViewElement implements StructureViewTreeElement, SortableTreeElement { - + private final NavigatablePsiElement myElement; - + public SdStructureViewElement(NavigatablePsiElement element) { this.myElement = element; } - + @Override public Object getValue() { return myElement; } - + @Override public void navigate(boolean requestFocus) { myElement.navigate(requestFocus); } - + @Override public boolean canNavigate() { return myElement.canNavigate(); } - + @Override public boolean canNavigateToSource() { return myElement.canNavigateToSource(); } - + @NotNull @Override public String getAlphaSortKey() { String name = myElement.getName(); return name != null ? name : ""; } - + @NotNull @Override public ItemPresentation getPresentation() { ItemPresentation presentation = myElement.getPresentation(); return presentation != null ? presentation : new PresentationData(); } - + @Override public TreeElement @NotNull [] getChildren() { - List<PsiElement> children = new ArrayList<>();; + List<PsiElement> children = new ArrayList<>(); if (myElement instanceof SdFile) { children = SdUtil.findSchemaChildren(myElement); } else if (myElement instanceof SdDocumentDefinition) { @@ -73,6 +80,9 @@ public class SdStructureViewElement implements StructureViewTreeElement, Sortabl children = SdUtil.findRankProfileChildren(myElement); } else if (myElement instanceof SdDocumentSummaryDefinition) { children = SdUtil.findDocumentSummaryChildren(myElement); + } else if (myElement instanceof SdSchemaAnnotationDefinition || + myElement instanceof SdDocumentAnnotationDefinition) { + children = SdUtil.findAnnotationChildren(myElement); } List<TreeElement> treeElements = new ArrayList<>(children.size()); diff --git a/sd-plugin/src/main/java/org/intellij/sdk/language/structure/SdStructureViewFactory.java b/sd-plugin/src/main/java/org/intellij/sdk/language/structure/SdStructureViewFactory.java index 05590362fcf..6ca4c979ef0 100644 --- a/sd-plugin/src/main/java/org/intellij/sdk/language/structure/SdStructureViewFactory.java +++ b/sd-plugin/src/main/java/org/intellij/sdk/language/structure/SdStructureViewFactory.java @@ -1,3 +1,4 @@ +// Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. package org.intellij.sdk.language.structure; import com.intellij.ide.structureView.StructureViewBuilder; @@ -9,6 +10,11 @@ import com.intellij.psi.PsiFile; import org.jetbrains.annotations.NotNull; import org.jetbrains.annotations.Nullable; +/** + * This class is used for the extension (in plugin.xml) to the class SdStructureViewModel. It make the IDE use our + * plugin's code when creating the "Structure View" window. + * @author shahariel + */ public class SdStructureViewFactory implements PsiStructureViewFactory { @Nullable diff --git a/sd-plugin/src/main/java/org/intellij/sdk/language/structure/SdStructureViewModel.java b/sd-plugin/src/main/java/org/intellij/sdk/language/structure/SdStructureViewModel.java index 549bd811dcf..00cf18565aa 100644 --- a/sd-plugin/src/main/java/org/intellij/sdk/language/structure/SdStructureViewModel.java +++ b/sd-plugin/src/main/java/org/intellij/sdk/language/structure/SdStructureViewModel.java @@ -1,3 +1,4 @@ +// Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. package org.intellij.sdk.language.structure; import com.intellij.ide.structureView.StructureViewModel; @@ -7,6 +8,9 @@ import com.intellij.ide.util.treeView.smartTree.Sorter; import com.intellij.psi.PsiFile; import org.jetbrains.annotations.NotNull; +/** + * This class represents the "Structure View" window. + */ public class SdStructureViewModel extends StructureViewModelBase implements StructureViewModel.ElementInfoProvider { public SdStructureViewModel(PsiFile psiFile) { super(psiFile, new SdStructureViewElement(psiFile)); diff --git a/sd-plugin/src/main/resources/META-INF/plugin.xml b/sd-plugin/src/main/resources/META-INF/plugin.xml index 206b33144e2..7e61f0a8b1e 100644 --- a/sd-plugin/src/main/resources/META-INF/plugin.xml +++ b/sd-plugin/src/main/resources/META-INF/plugin.xml @@ -1,9 +1,10 @@ +<!-- Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. --> <idea-plugin> <id>org.intellij.sdk.language</id> - <name>SdPlugin</name> - + <name>SdReader</name> + <!-- Text to display as company information on Preferences/Settings | Plugin page --> - <vendor email="shahar.ariel@verizonmedia.com">VerizonMedia</vendor> + <vendor email="shahar.ariel@yahooinc.com">Yahoo</vendor> <!-- Product and plugin compatibility requirements --> <depends>com.intellij.java</depends> @@ -20,12 +21,12 @@ <depends>com.intellij.java</depends> <extensions defaultExtensionNs="com.intellij"> - <fileType name="Sd File" implementationClass="org.intellij.sdk.language.SdFileType" fieldName="INSTANCE" + <fileType name="Sd File" implementationClass="org.intellij.sdk.language.SdFileType" fieldName="INSTANCE" language="Sd" extensions="sd"/> <lang.parserDefinition language="Sd" implementationClass="org.intellij.sdk.language.parser.SdParserDefinition"/> <lang.syntaxHighlighterFactory language="Sd" implementationClass="org.intellij.sdk.language.SdSyntaxHighlighterFactory"/> <completion.contributor language="Sd" implementationClass="org.intellij.sdk.language.SdCompletionContributor"/> - + <lang.findUsagesProvider language="Sd" implementationClass="org.intellij.sdk.language.findUsages.SdFindUsagesProvider"/> <findUsagesHandlerFactory implementation="org.intellij.sdk.language.findUsages.SdFindUsagesHandlerFactory"/> <fileStructureGroupRuleProvider implementation="org.intellij.sdk.language.findUsages.SdRankProfileGroupingRuleProvider"/> @@ -33,7 +34,6 @@ <elementDescriptionProvider implementation="org.intellij.sdk.language.psi.SdElementDescriptionProvider"/> <lang.psiStructureViewFactory language="Sd" implementationClass="org.intellij.sdk.language.structure.SdStructureViewFactory"/> - <lang.formatter language="Sd" implementationClass="org.intellij.sdk.language.SdFormattingModelBuilder"/> <codeStyleSettingsProvider implementation="org.intellij.sdk.language.SdCodeStyleSettingsProvider"/> <langCodeStyleSettingsProvider implementation="org.intellij.sdk.language.SdLanguageCodeStyleSettingsProvider"/> <lang.commenter language="Sd" implementationClass="org.intellij.sdk.language.SdCommenter"/> @@ -45,4 +45,4 @@ <actions> <!-- Add your actions here --> </actions> -</idea-plugin>
\ No newline at end of file +</idea-plugin> diff --git a/searchcore/src/apps/vespa-redistribute-bm/vespa_redistribute_bm.cpp b/searchcore/src/apps/vespa-redistribute-bm/vespa_redistribute_bm.cpp index b957226dd8a..f9833b430a0 100644 --- a/searchcore/src/apps/vespa-redistribute-bm/vespa_redistribute_bm.cpp +++ b/searchcore/src/apps/vespa-redistribute-bm/vespa_redistribute_bm.cpp @@ -466,6 +466,7 @@ App::usage() "USAGE:\n"; std::cerr << "vespa-redistribute-bm\n" + "[--async-apply-bucket-diff]\n" "[--bucket-db-stripe-bits bits]\n" "[--client-threads threads]\n" "[--distributor-merge-busy-wait distributor-merge-busy-wait]\n" @@ -481,7 +482,7 @@ App::usage() "[--max-merge-queue-size max-merge-queue-size]\n" "[--max-pending max-pending]\n" "[--max-pending-idealstate-operations max-pending-idealstate-operations]\n" - "[--mbus-distributor-node-max-pending-count] count\n" + "[--mbus-distributor-node-max-pending-count count]\n" "[--mode [grow, shrink, perm-crash, temp-crash, replace]\n" "[--nodes-per-group nodes-per-group]\n" "[--redundancy redundancy]\n" @@ -502,6 +503,7 @@ App::get_options() const char *opt_argument = nullptr; int long_opt_index = 0; static struct option long_opts[] = { + { "async-apply-bucket-diff", 0, nullptr, 0 }, { "bucket-db-stripe-bits", 1, nullptr, 0 }, { "client-threads", 1, nullptr, 0 }, { "distributor-merge-busy-wait", 1, nullptr, 0 }, @@ -532,6 +534,7 @@ App::get_options() { nullptr, 0, nullptr, 0 } }; enum longopts_enum { + LONGOPT_ASYNC_APPLY_BUCKET_DIFF, LONGOPT_BUCKET_DB_STRIPE_BITS, LONGOPT_CLIENT_THREADS, LONGOPT_DISTRIBUTOR_MERGE_BUSY_WAIT, @@ -566,6 +569,9 @@ App::get_options() switch (c) { case 0: switch(long_opt_index) { + case LONGOPT_ASYNC_APPLY_BUCKET_DIFF: + _bm_params.set_async_apply_bucket_diff(true); + break; case LONGOPT_BUCKET_DB_STRIPE_BITS: _bm_params.set_bucket_db_stripe_bits(atoi(opt_argument)); break; diff --git a/searchcore/src/vespa/searchcore/bmcluster/bm_cluster_params.cpp b/searchcore/src/vespa/searchcore/bmcluster/bm_cluster_params.cpp index 3ff10b19164..4a3466f1a51 100644 --- a/searchcore/src/vespa/searchcore/bmcluster/bm_cluster_params.cpp +++ b/searchcore/src/vespa/searchcore/bmcluster/bm_cluster_params.cpp @@ -6,7 +6,8 @@ namespace search::bmcluster { BmClusterParams::BmClusterParams() - : _bucket_db_stripe_bits(4), + : _async_apply_bucket_diff(), + _bucket_db_stripe_bits(4), _disable_queue_limits_for_chained_merges(false), // Same default as in stor-server.def _distributor_merge_busy_wait(10), // Same default as stor_distributormanager.def _distributor_stripes(0), diff --git a/searchcore/src/vespa/searchcore/bmcluster/bm_cluster_params.h b/searchcore/src/vespa/searchcore/bmcluster/bm_cluster_params.h index d365a28b0b6..36b4c22f6a8 100644 --- a/searchcore/src/vespa/searchcore/bmcluster/bm_cluster_params.h +++ b/searchcore/src/vespa/searchcore/bmcluster/bm_cluster_params.h @@ -13,6 +13,7 @@ namespace search::bmcluster { */ class BmClusterParams { + std::optional<bool> _async_apply_bucket_diff; uint32_t _bucket_db_stripe_bits; bool _disable_queue_limits_for_chained_merges; uint32_t _distributor_merge_busy_wait; @@ -44,6 +45,7 @@ class BmClusterParams public: BmClusterParams(); ~BmClusterParams(); + const std::optional<bool>& get_async_apply_bucket_diff() const noexcept { return _async_apply_bucket_diff; } uint32_t get_bucket_db_stripe_bits() const { return _bucket_db_stripe_bits; } bool get_disable_queue_limits_for_chained_merges() const noexcept { return _disable_queue_limits_for_chained_merges; } uint32_t get_distributor_merge_busy_wait() const { return _distributor_merge_busy_wait; } @@ -73,6 +75,7 @@ public: bool needs_distributor() const { return _enable_distributor || _use_document_api; } bool needs_message_bus() const { return _use_message_bus || _use_document_api; } bool needs_service_layer() const { return _enable_service_layer || _enable_distributor || _use_storage_chain || _use_message_bus || _use_document_api; } + void set_async_apply_bucket_diff(bool value) { _async_apply_bucket_diff = value; } void set_bucket_db_stripe_bits(uint32_t value) { _bucket_db_stripe_bits = value; } void set_disable_queue_limits_for_chained_merges(bool value) { _disable_queue_limits_for_chained_merges = value; } void set_distributor_merge_busy_wait(uint32_t value) { _distributor_merge_busy_wait = value; } diff --git a/searchcore/src/vespa/searchcore/bmcluster/bm_node.cpp b/searchcore/src/vespa/searchcore/bmcluster/bm_node.cpp index 545fb08c762..0882153edd6 100644 --- a/searchcore/src/vespa/searchcore/bmcluster/bm_node.cpp +++ b/searchcore/src/vespa/searchcore/bmcluster/bm_node.cpp @@ -382,6 +382,9 @@ struct ServiceLayerConfigSet : public StorageConfigSet stor_bucket_init(), stor_visitor() { + if (params.get_async_apply_bucket_diff().has_value()) { + stor_filestor.asyncApplyBucketDiff = params.get_async_apply_bucket_diff().value(); + } stor_filestor.numResponseThreads = params.get_response_threads(); stor_filestor.numNetworkThreads = params.get_rpc_network_threads(); stor_filestor.useAsyncMessageHandlingOnSchedule = params.get_use_async_message_handling_on_schedule(); diff --git a/staging_vespalib/src/vespa/vespalib/util/process_memory_stats.cpp b/staging_vespalib/src/vespa/vespalib/util/process_memory_stats.cpp index ad143869848..a74010df944 100644 --- a/staging_vespalib/src/vespa/vespalib/util/process_memory_stats.cpp +++ b/staging_vespalib/src/vespa/vespalib/util/process_memory_stats.cpp @@ -182,7 +182,7 @@ ProcessMemoryStats::toString() const ProcessMemoryStats ProcessMemoryStats::create(uint64_t sizeEpsilon) { - constexpr size_t NUM_TRIES = 10; + constexpr size_t NUM_TRIES = 3; std::vector<ProcessMemoryStats> samples; samples.reserve(NUM_TRIES); samples.push_back(createStatsFromSmaps()); @@ -195,7 +195,7 @@ ProcessMemoryStats::create(uint64_t sizeEpsilon) i, (samples.rbegin()+1)->toString().c_str(), samples.back().toString().c_str()); } std::sort(samples.begin(), samples.end()); - LOG(info, "We failed to find 2 consecutive samples that where similar with epsilon of %" PRIu64 ".\nSmallest is '%s',\n median is '%s',\n largest is '%s'", + LOG(debug, "We failed to find 2 consecutive samples that where similar with epsilon of %" PRIu64 ".\nSmallest is '%s',\n median is '%s',\n largest is '%s'", sizeEpsilon, samples.front().toString().c_str(), samples[samples.size()/2].toString().c_str(), samples.back().toString().c_str()); return samples[samples.size()/2]; } diff --git a/storage/src/tests/persistence/apply_bucket_diff_state_test.cpp b/storage/src/tests/persistence/apply_bucket_diff_state_test.cpp index 6e1e62d2080..fe77477fa77 100644 --- a/storage/src/tests/persistence/apply_bucket_diff_state_test.cpp +++ b/storage/src/tests/persistence/apply_bucket_diff_state_test.cpp @@ -1,6 +1,5 @@ // Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. -#include <vespa/storage/persistence/apply_bucket_diff_entry_result.h> #include <vespa/storage/persistence/apply_bucket_diff_state.h> #include <vespa/storage/persistence/merge_bucket_info_syncer.h> #include <vespa/document/base/documentid.h> @@ -26,38 +25,44 @@ document::Bucket dummy_document_bucket(makeDocumentBucket(document::BucketId(0, class DummyMergeBucketInfoSyncer : public MergeBucketInfoSyncer { uint32_t& _sync_count; + vespalib::string _fail; public: DummyMergeBucketInfoSyncer(uint32_t& sync_count) : MergeBucketInfoSyncer(), - _sync_count(sync_count) + _sync_count(sync_count), + _fail() { } + ~DummyMergeBucketInfoSyncer(); void sync_bucket_info(const spi::Bucket& bucket) const override { EXPECT_EQ(bucket, spi::Bucket(dummy_document_bucket)); ++_sync_count; + if (!_fail.empty()) { + throw std::runtime_error(_fail); + } } + void set_fail(vespalib::string fail) { _fail = std::move(fail); } }; -ApplyBucketDiffEntryResult -make_result(spi::Result &spi_result, const DocumentId &doc_id) +DummyMergeBucketInfoSyncer::~DummyMergeBucketInfoSyncer() = default; + +void +make_result(ApplyBucketDiffState& state, spi::Result &spi_result, const DocumentId &doc_id) { - std::promise<std::unique_ptr<spi::Result>> result_promise; - result_promise.set_value(std::make_unique<spi::Result>(spi_result)); - spi::Bucket bucket(makeDocumentBucket(bucket_id_factory.getBucketId(doc_id))); - return ApplyBucketDiffEntryResult(result_promise.get_future(), bucket, doc_id, test_op); + state.on_entry_complete(std::make_unique<spi::Result>(spi_result), doc_id, test_op); } void push_ok(ApplyBucketDiffState &state) { - state.push_back(make_result(spi_result_ok, DocumentId("id::test::0"))); - state.push_back(make_result(spi_result_ok, DocumentId("id::test::1"))); + make_result(state, spi_result_ok, DocumentId("id::test::0")); + make_result(state, spi_result_ok, DocumentId("id::test::1")); } void push_bad(ApplyBucketDiffState &state) { - state.push_back(make_result(spi_result_ok, DocumentId("id::test::0"))); - state.push_back(make_result(spi_result_fail, DocumentId("id::test::1"))); - state.push_back(make_result(spi_result_fail, DocumentId("id::test::2"))); + make_result(state, spi_result_ok, DocumentId("id::test::0")); + make_result(state, spi_result_fail, DocumentId("id::test::1")); + make_result(state, spi_result_fail, DocumentId("id::test::2")); } } @@ -75,15 +80,19 @@ public: { } + ~ApplyBucketDiffStateTestBase(); + std::unique_ptr<ApplyBucketDiffState> make_state() { return std::make_unique<ApplyBucketDiffState>(syncer, spi::Bucket(dummy_document_bucket)); } }; +ApplyBucketDiffStateTestBase::~ApplyBucketDiffStateTestBase() = default; + class ApplyBucketDiffStateTest : public ApplyBucketDiffStateTestBase { public: - std::unique_ptr<ApplyBucketDiffState> state; + std::shared_ptr<ApplyBucketDiffState> state; ApplyBucketDiffStateTest() : ApplyBucketDiffStateTestBase(), @@ -95,13 +104,14 @@ public: state = make_state(); } + void check_failure(std::string expected) { + auto future = state->get_future(); + state.reset(); + std::string fail_message = future.get(); + EXPECT_EQ(expected, fail_message); + } void check_failure() { - try { - state->check(); - FAIL() << "Failed to throw exception for failed result"; - } catch (std::exception &e) { - EXPECT_EQ("Failed put for id::test::1 in Bucket(0xeb4700c03842cac4): Result(5, write blocked)", std::string(e.what())); - } + check_failure("Failed put for id::test::1 in Bucket(0x0000000000000010): Result(5, write blocked)"); } }; @@ -109,7 +119,7 @@ public: TEST_F(ApplyBucketDiffStateTest, ok_results_can_be_checked) { push_ok(*state); - state->check(); + check_failure(""); } TEST_F(ApplyBucketDiffStateTest, failed_result_errors_ignored) @@ -146,4 +156,12 @@ TEST_F(ApplyBucketDiffStateTest, explicit_sync_bucket_info_works) EXPECT_EQ(1, sync_count); } +TEST_F(ApplyBucketDiffStateTest, failed_sync_bucket_info_is_detected) +{ + vespalib::string fail("sync bucket failed"); + syncer.set_fail(fail); + state->mark_stale_bucket_info(); + check_failure(fail); +} + } diff --git a/storage/src/vespa/storage/persistence/CMakeLists.txt b/storage/src/vespa/storage/persistence/CMakeLists.txt index 4b41d3fa778..c737d2bed28 100644 --- a/storage/src/vespa/storage/persistence/CMakeLists.txt +++ b/storage/src/vespa/storage/persistence/CMakeLists.txt @@ -2,7 +2,6 @@ vespa_add_library(storage_spersistence OBJECT SOURCES apply_bucket_diff_entry_complete.cpp - apply_bucket_diff_entry_result.cpp apply_bucket_diff_state.cpp asynchandler.cpp bucketownershipnotifier.cpp diff --git a/storage/src/vespa/storage/persistence/apply_bucket_diff_entry_complete.cpp b/storage/src/vespa/storage/persistence/apply_bucket_diff_entry_complete.cpp index 034089d4eca..1fbe155b16d 100644 --- a/storage/src/vespa/storage/persistence/apply_bucket_diff_entry_complete.cpp +++ b/storage/src/vespa/storage/persistence/apply_bucket_diff_entry_complete.cpp @@ -1,14 +1,17 @@ // Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. #include "apply_bucket_diff_entry_complete.h" +#include "apply_bucket_diff_state.h" #include <vespa/persistence/spi/result.h> #include <cassert> namespace storage { -ApplyBucketDiffEntryComplete::ApplyBucketDiffEntryComplete(ResultPromise result_promise, const framework::Clock& clock, metrics::DoubleAverageMetric& latency_metric) +ApplyBucketDiffEntryComplete::ApplyBucketDiffEntryComplete(std::shared_ptr<ApplyBucketDiffState> state, document::DocumentId doc_id, const char *op, const framework::Clock& clock, metrics::DoubleAverageMetric& latency_metric) : _result_handler(nullptr), - _result_promise(std::move(result_promise)), + _state(std::move(state)), + _doc_id(std::move(doc_id)), + _op(op), _start_time(clock), _latency_metric(latency_metric) { @@ -24,7 +27,7 @@ ApplyBucketDiffEntryComplete::onComplete(std::unique_ptr<spi::Result> result) } double elapsed = _start_time.getElapsedTimeAsDouble(); _latency_metric.addValue(elapsed); - _result_promise.set_value(std::move(result)); + _state->on_entry_complete(std::move(result), _doc_id, _op); } void diff --git a/storage/src/vespa/storage/persistence/apply_bucket_diff_entry_complete.h b/storage/src/vespa/storage/persistence/apply_bucket_diff_entry_complete.h index 6f05ed82fa0..dd2346d9dee 100644 --- a/storage/src/vespa/storage/persistence/apply_bucket_diff_entry_complete.h +++ b/storage/src/vespa/storage/persistence/apply_bucket_diff_entry_complete.h @@ -2,6 +2,7 @@ #pragma once +#include <vespa/document/base/documentid.h> #include <vespa/metrics/valuemetric.h> #include <vespa/persistence/spi/operationcomplete.h> #include <vespa/storageframework/generic/clock/timer.h> @@ -9,19 +10,22 @@ namespace storage { +class ApplyBucketDiffState; + /* * Complete handler for a bucket diff entry spi operation (putAsync * or removeAsync) */ class ApplyBucketDiffEntryComplete : public spi::OperationComplete { - using ResultPromise = std::promise<std::unique_ptr<spi::Result>>; - const spi::ResultHandler* _result_handler; - ResultPromise _result_promise; - framework::MilliSecTimer _start_time; - metrics::DoubleAverageMetric& _latency_metric; + const spi::ResultHandler* _result_handler; + std::shared_ptr<ApplyBucketDiffState> _state; + document::DocumentId _doc_id; + const char* _op; + framework::MilliSecTimer _start_time; + metrics::DoubleAverageMetric& _latency_metric; public: - ApplyBucketDiffEntryComplete(ResultPromise result_promise, const framework::Clock& clock, metrics::DoubleAverageMetric& latency_metric); + ApplyBucketDiffEntryComplete(std::shared_ptr<ApplyBucketDiffState> state, document::DocumentId doc_id, const char *op, const framework::Clock& clock, metrics::DoubleAverageMetric& latency_metric); ~ApplyBucketDiffEntryComplete(); void onComplete(std::unique_ptr<spi::Result> result) override; void addResultHandler(const spi::ResultHandler* resultHandler) override; diff --git a/storage/src/vespa/storage/persistence/apply_bucket_diff_entry_result.cpp b/storage/src/vespa/storage/persistence/apply_bucket_diff_entry_result.cpp deleted file mode 100644 index 9cb5bb1bfd0..00000000000 --- a/storage/src/vespa/storage/persistence/apply_bucket_diff_entry_result.cpp +++ /dev/null @@ -1,44 +0,0 @@ -// Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. - -#include "apply_bucket_diff_entry_result.h" -#include <vespa/persistence/spi/result.h> -#include <vespa/vespalib/stllike/asciistream.h> -#include <cassert> - -namespace storage { - -ApplyBucketDiffEntryResult::ApplyBucketDiffEntryResult(FutureResult future_result, spi::Bucket bucket, document::DocumentId doc_id, const char *op) - : _future_result(std::move(future_result)), - _bucket(bucket), - _doc_id(std::move(doc_id)), - _op(op) -{ -} - -ApplyBucketDiffEntryResult::ApplyBucketDiffEntryResult(ApplyBucketDiffEntryResult &&rhs) = default; - -ApplyBucketDiffEntryResult::~ApplyBucketDiffEntryResult() = default; - -void -ApplyBucketDiffEntryResult::wait() -{ - assert(_future_result.valid()); - _future_result.wait(); -} - -void -ApplyBucketDiffEntryResult::check_result() -{ - assert(_future_result.valid()); - auto result = _future_result.get(); - if (result->hasError()) { - vespalib::asciistream ss; - ss << "Failed " << _op - << " for " << _doc_id.toString() - << " in " << _bucket - << ": " << result->toString(); - throw std::runtime_error(ss.str()); - } -} - -} diff --git a/storage/src/vespa/storage/persistence/apply_bucket_diff_entry_result.h b/storage/src/vespa/storage/persistence/apply_bucket_diff_entry_result.h deleted file mode 100644 index 662656f2dbc..00000000000 --- a/storage/src/vespa/storage/persistence/apply_bucket_diff_entry_result.h +++ /dev/null @@ -1,32 +0,0 @@ -// Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. - -#pragma once - -#include <vespa/document/base/documentid.h> -#include <vespa/persistence/spi/bucket.h> -#include <vespa/metrics/valuemetric.h> -#include <future> - -namespace storage::spi { class Result; } - -namespace storage { - -/* - * Result of a bucket diff entry spi operation (putAsync or removeAsync) - */ -class ApplyBucketDiffEntryResult { - using FutureResult = std::future<std::unique_ptr<spi::Result>>; - FutureResult _future_result; - spi::Bucket _bucket; - document::DocumentId _doc_id; - const char* _op; - -public: - ApplyBucketDiffEntryResult(FutureResult future_result, spi::Bucket bucket, document::DocumentId doc_id, const char *op); - ApplyBucketDiffEntryResult(ApplyBucketDiffEntryResult &&rhs); - ~ApplyBucketDiffEntryResult(); - void wait(); - void check_result(); -}; - -} diff --git a/storage/src/vespa/storage/persistence/apply_bucket_diff_state.cpp b/storage/src/vespa/storage/persistence/apply_bucket_diff_state.cpp index c754796ba51..eb7a5ef5bc6 100644 --- a/storage/src/vespa/storage/persistence/apply_bucket_diff_state.cpp +++ b/storage/src/vespa/storage/persistence/apply_bucket_diff_state.cpp @@ -1,64 +1,50 @@ // Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. #include "apply_bucket_diff_state.h" -#include "apply_bucket_diff_entry_result.h" #include "mergehandler.h" +#include <vespa/document/base/documentid.h> +#include <vespa/persistence/spi/result.h> +#include <vespa/vespalib/stllike/asciistream.h> + +using storage::spi::Result; namespace storage { ApplyBucketDiffState::ApplyBucketDiffState(const MergeBucketInfoSyncer& merge_bucket_info_syncer, const spi::Bucket& bucket) - : _async_results(), - _merge_bucket_info_syncer(merge_bucket_info_syncer), + : _merge_bucket_info_syncer(merge_bucket_info_syncer), _bucket(bucket), - _stale_bucket_info(false) + _fail_message(), + _failed_flag(), + _stale_bucket_info(false), + _promise() { } ApplyBucketDiffState::~ApplyBucketDiffState() { - wait(); try { sync_bucket_info(); - } catch (std::exception&) { - } -} - -bool -ApplyBucketDiffState::empty() const -{ - return _async_results.empty(); -} - -void -ApplyBucketDiffState::wait() -{ - if (!_async_results.empty()) { - _async_results.back().wait(); + } catch (std::exception& e) { + if (_fail_message.empty()) { + _fail_message = e.what(); + } } - for (auto &result_to_check : _async_results) { - result_to_check.wait(); + if (_promise.has_value()) { + _promise.value().set_value(_fail_message); } } void -ApplyBucketDiffState::check() +ApplyBucketDiffState::on_entry_complete(std::unique_ptr<Result> result, const document::DocumentId &doc_id, const char *op) { - wait(); - try { - for (auto& result_to_check : _async_results) { - result_to_check.check_result(); - } - } catch (std::exception&) { - _async_results.clear(); - throw; + if (result->hasError() && !_failed_flag.test_and_set()) { + vespalib::asciistream ss; + ss << "Failed " << op + << " for " << doc_id.toString() + << " in " << _bucket + << ": " << result->toString(); + _fail_message = ss.str(); } - _async_results.clear(); -} - -void -ApplyBucketDiffState::push_back(ApplyBucketDiffEntryResult&& result) -{ - _async_results.push_back(std::move(result)); } void @@ -76,4 +62,11 @@ ApplyBucketDiffState::sync_bucket_info() } } +std::future<vespalib::string> +ApplyBucketDiffState::get_future() +{ + _promise = std::promise<vespalib::string>(); + return _promise.value().get_future(); +} + } diff --git a/storage/src/vespa/storage/persistence/apply_bucket_diff_state.h b/storage/src/vespa/storage/persistence/apply_bucket_diff_state.h index 489e75e4a72..af4174b06d6 100644 --- a/storage/src/vespa/storage/persistence/apply_bucket_diff_state.h +++ b/storage/src/vespa/storage/persistence/apply_bucket_diff_state.h @@ -3,8 +3,13 @@ #pragma once #include <vespa/persistence/spi/bucket.h> +#include <future> +#include <memory> #include <vector> +namespace document { class DocumentId; } +namespace storage::spi { class Result; } + namespace storage { class ApplyBucketDiffEntryResult; @@ -15,19 +20,22 @@ class MergeBucketInfoSyncer; * for one or more ApplyBucketDiffCommand / ApplyBucketDiffReply. */ class ApplyBucketDiffState { - std::vector<ApplyBucketDiffEntryResult> _async_results; const MergeBucketInfoSyncer& _merge_bucket_info_syncer; spi::Bucket _bucket; + vespalib::string _fail_message; + std::atomic_flag _failed_flag; bool _stale_bucket_info; + std::optional<std::promise<vespalib::string>> _promise; + public: ApplyBucketDiffState(const MergeBucketInfoSyncer &merge_bucket_info_syncer, const spi::Bucket& bucket); ~ApplyBucketDiffState(); - void push_back(ApplyBucketDiffEntryResult&& result); - bool empty() const; + void on_entry_complete(std::unique_ptr<storage::spi::Result> result, const document::DocumentId &doc_id, const char *op); void wait(); void check(); void mark_stale_bucket_info(); void sync_bucket_info(); + std::future<vespalib::string> get_future(); }; } diff --git a/storage/src/vespa/storage/persistence/mergehandler.cpp b/storage/src/vespa/storage/persistence/mergehandler.cpp index 274aba303fb..17a16487ac4 100644 --- a/storage/src/vespa/storage/persistence/mergehandler.cpp +++ b/storage/src/vespa/storage/persistence/mergehandler.cpp @@ -3,7 +3,6 @@ #include "mergehandler.h" #include "persistenceutil.h" #include "apply_bucket_diff_entry_complete.h" -#include "apply_bucket_diff_entry_result.h" #include "apply_bucket_diff_state.h" #include <vespa/storage/persistence/filestorage/mergestatus.h> #include <vespa/persistence/spi/persistenceprovider.h> @@ -96,6 +95,17 @@ struct DiffEntryTimestampPredicate { } }; + +void check_apply_diff_sync(std::shared_ptr<ApplyBucketDiffState> async_results) { + auto future = async_results->get_future(); + async_results.reset(); + future.wait(); + auto fail_message = future.get(); + if (!fail_message.empty()) { + throw std::runtime_error(fail_message); + } +} + } // anonymous namespace void @@ -483,27 +493,24 @@ MergeHandler::deserializeDiffDocument( return doc; } -ApplyBucketDiffEntryResult -MergeHandler::applyDiffEntry(const spi::Bucket& bucket, +void +MergeHandler::applyDiffEntry(std::shared_ptr<ApplyBucketDiffState> async_results, + const spi::Bucket& bucket, const api::ApplyBucketDiffCommand::Entry& e, spi::Context& context, const document::DocumentTypeRepo& repo) const { - std::promise<std::unique_ptr<spi::Result>> result_promise; - auto future_result = result_promise.get_future(); spi::Timestamp timestamp(e._entry._timestamp); if (!(e._entry._flags & (DELETED | DELETED_IN_PLACE))) { // Regular put entry Document::SP doc(deserializeDiffDocument(e, repo)); DocumentId docId = doc->getId(); - auto complete = std::make_unique<ApplyBucketDiffEntryComplete>(std::move(result_promise), _clock, _env._metrics.merge_handler_metrics.put_latency); + auto complete = std::make_unique<ApplyBucketDiffEntryComplete>(std::move(async_results), std::move(docId), "put", _clock, _env._metrics.merge_handler_metrics.put_latency); _spi.putAsync(bucket, timestamp, std::move(doc), context, std::move(complete)); - return ApplyBucketDiffEntryResult(std::move(future_result), bucket, std::move(docId), "put"); } else { DocumentId docId(e._docName); - auto complete = std::make_unique<ApplyBucketDiffEntryComplete>(std::move(result_promise), _clock, _env._metrics.merge_handler_metrics.remove_latency); + auto complete = std::make_unique<ApplyBucketDiffEntryComplete>(std::move(async_results), docId, "remove", _clock, _env._metrics.merge_handler_metrics.remove_latency); _spi.removeAsync(bucket, timestamp, docId, context, std::move(complete)); - return ApplyBucketDiffEntryResult(std::move(future_result), bucket, std::move(docId), "remove"); } } @@ -516,7 +523,7 @@ MergeHandler::applyDiffLocally( std::vector<api::ApplyBucketDiffCommand::Entry>& diff, uint8_t nodeIndex, spi::Context& context, - ApplyBucketDiffState& async_results) const + std::shared_ptr<ApplyBucketDiffState> async_results) const { // Sort the data to apply by which file they should be added to LOG(spam, "Merge(%s): Applying data locally. Diff has %zu entries", @@ -527,7 +534,7 @@ MergeHandler::applyDiffLocally( uint32_t addedCount = 0; uint32_t notNeededByteCount = 0; - async_results.mark_stale_bucket_info(); + async_results->mark_stale_bucket_info(); std::vector<spi::DocEntry::UP> entries; populateMetaData(bucket, MAX_TIMESTAMP, entries, context); @@ -565,7 +572,7 @@ MergeHandler::applyDiffLocally( ++i; LOG(spam, "ApplyBucketDiff(%s): Adding slot %s", bucket.toString().c_str(), e.toString().c_str()); - async_results.push_back(applyDiffEntry(bucket, e, context, repo)); + applyDiffEntry(async_results, bucket, e, context, repo); } else { assert(spi::Timestamp(e._entry._timestamp) == existing.getTimestamp()); // Diffing for existing timestamp; should either both be put @@ -578,7 +585,7 @@ MergeHandler::applyDiffLocally( "timestamp in %s. Diff slot: %s. Existing slot: %s", bucket.toString().c_str(), e.toString().c_str(), existing.toString().c_str()); - async_results.push_back(applyDiffEntry(bucket, e, context, repo)); + applyDiffEntry(async_results, bucket, e, context, repo); } else { // Duplicate put, just ignore it. LOG(debug, "During diff apply, attempting to add slot " @@ -610,7 +617,7 @@ MergeHandler::applyDiffLocally( LOG(spam, "ApplyBucketDiff(%s): Adding slot %s", bucket.toString().c_str(), e.toString().c_str()); - async_results.push_back(applyDiffEntry(bucket, e, context, repo)); + applyDiffEntry(async_results, bucket, e, context, repo); byteCount += e._headerBlob.size() + e._bodyBlob.size(); } if (byteCount + notNeededByteCount != 0) { @@ -1204,7 +1211,7 @@ MergeHandler::handleApplyBucketDiff(api::ApplyBucketDiffCommand& cmd, MessageTra tracker->setMetric(_env._metrics.applyBucketDiff); spi::Bucket bucket(cmd.getBucket()); - ApplyBucketDiffState async_results(*this, bucket); + auto async_results = std::make_shared<ApplyBucketDiffState>(*this, bucket); LOG(debug, "%s", cmd.toString().c_str()); if (_env._fileStorHandler.isMerging(bucket.getBucket())) { @@ -1229,8 +1236,7 @@ MergeHandler::handleApplyBucketDiff(api::ApplyBucketDiffCommand& cmd, MessageTra applyDiffLocally(bucket, cmd.getDiff(), index, tracker->context(), async_results); _env._metrics.merge_handler_metrics.mergeDataWriteLatency.addValue( startTime.getElapsedTimeAsDouble()); - async_results.check(); - async_results.sync_bucket_info(); + check_apply_diff_sync(std::move(async_results)); } else { LOG(spam, "Merge(%s): Didn't need fetched data on node %u (%u).", bucket.toString().c_str(), _env._nodeIndex, index); @@ -1289,7 +1295,7 @@ MergeHandler::handleApplyBucketDiffReply(api::ApplyBucketDiffReply& reply,Messag (void) tracker; _env._metrics.applyBucketDiffReply.inc(); spi::Bucket bucket(reply.getBucket()); - ApplyBucketDiffState async_results(*this, bucket); + auto async_results = std::make_shared<ApplyBucketDiffState>(*this, bucket); std::vector<api::ApplyBucketDiffCommand::Entry>& diff(reply.getDiff()); LOG(debug, "%s", reply.toString().c_str()); @@ -1325,8 +1331,7 @@ MergeHandler::handleApplyBucketDiffReply(api::ApplyBucketDiffReply& reply,Messag framework::MilliSecTimer startTime(_clock); applyDiffLocally(bucket, diff, index, s->context, async_results); _env._metrics.merge_handler_metrics.mergeDataWriteLatency.addValue(startTime.getElapsedTimeAsDouble()); - async_results.check(); - async_results.sync_bucket_info(); + check_apply_diff_sync(std::move(async_results)); } else { LOG(spam, "Merge(%s): Didn't need fetched data on node %u (%u)", bucket.toString().c_str(), diff --git a/storage/src/vespa/storage/persistence/mergehandler.h b/storage/src/vespa/storage/persistence/mergehandler.h index 0ff8f3c0ef8..f6e8ddcf306 100644 --- a/storage/src/vespa/storage/persistence/mergehandler.h +++ b/storage/src/vespa/storage/persistence/mergehandler.h @@ -62,7 +62,7 @@ public: std::vector<api::ApplyBucketDiffCommand::Entry>& diff, uint8_t nodeIndex, spi::Context& context, - ApplyBucketDiffState& async_results) const; + std::shared_ptr<ApplyBucketDiffState> async_results) const; void sync_bucket_info(const spi::Bucket& bucket) const override; MessageTrackerUP handleMergeBucket(api::MergeBucketCommand&, MessageTrackerUP) const; @@ -90,10 +90,11 @@ private: * Invoke either put, remove or unrevertable remove on the SPI * depending on the flags in the diff entry. */ - ApplyBucketDiffEntryResult applyDiffEntry(const spi::Bucket&, - const api::ApplyBucketDiffCommand::Entry&, - spi::Context& context, - const document::DocumentTypeRepo& repo) const; + void applyDiffEntry(std::shared_ptr<ApplyBucketDiffState> async_results, + const spi::Bucket&, + const api::ApplyBucketDiffCommand::Entry&, + spi::Context& context, + const document::DocumentTypeRepo& repo) const; /** * Fill entries-vector with metadata for bucket up to maxTimestamp, diff --git a/testutil/pom.xml b/testutil/pom.xml index 26bb3056e2f..dc795a66dab 100644 --- a/testutil/pom.xml +++ b/testutil/pom.xml @@ -37,11 +37,6 @@ <scope>compile</scope> </dependency> <dependency> - <groupId>uk.co.datumedge</groupId> - <artifactId>hamcrest-json</artifactId> - <scope>compile</scope> - </dependency> - <dependency> <groupId>junit</groupId> <artifactId>junit</artifactId> <scope>compile</scope> diff --git a/testutil/src/main/java/com/yahoo/test/json/JsonTestHelper.java b/testutil/src/main/java/com/yahoo/test/json/JsonTestHelper.java index 08afb2fd95f..05532d2a504 100644 --- a/testutil/src/main/java/com/yahoo/test/json/JsonTestHelper.java +++ b/testutil/src/main/java/com/yahoo/test/json/JsonTestHelper.java @@ -5,11 +5,10 @@ import com.fasterxml.jackson.core.JsonProcessingException; import com.fasterxml.jackson.databind.JsonNode; import com.fasterxml.jackson.databind.ObjectMapper; import com.google.common.base.Joiner; -import org.hamcrest.MatcherAssert; import java.io.UncheckedIOException; -import static uk.co.datumedge.hamcrest.json.SameJSONAs.sameJSONAs; +import static org.junit.Assert.assertEquals; /** * @author Vegard Sjonfjell @@ -29,7 +28,13 @@ public class JsonTestHelper { /** Structurally compare two JSON encoded strings */ public static void assertJsonEquals(String inputJson, String expectedJson) { - MatcherAssert.assertThat(inputJson, sameJSONAs(expectedJson)); + try { + JsonNode expected = mapper.readTree(expectedJson); + JsonNode actual = mapper.readTree(inputJson); + assertEquals(expected, actual); + } catch (JsonProcessingException e) { + throw new RuntimeException("Exception when comparing json strings." , e); + } } /** Structurally compare a {@link JsonNode} and a JSON string. */ diff --git a/vespa-documentgen-plugin/pom.xml b/vespa-documentgen-plugin/pom.xml index 7fe11cb1e10..46704533bd1 100644 --- a/vespa-documentgen-plugin/pom.xml +++ b/vespa-documentgen-plugin/pom.xml @@ -34,7 +34,6 @@ <groupId>com.yahoo.vespa</groupId> <artifactId>config-provisioning</artifactId> <version>${project.version}</version> - <scope>compile</scope> </dependency> <dependency> <groupId>com.yahoo.vespa</groupId> @@ -50,18 +49,15 @@ <groupId>com.yahoo.vespa</groupId> <artifactId>vespajlib</artifactId> <version>${project.version}</version> - <scope>compile</scope> </dependency> <dependency> <groupId>org.apache.maven</groupId> <artifactId>maven-plugin-api</artifactId> - <version>3.5.0</version> <scope>provided</scope> </dependency> <dependency> <groupId>org.apache.maven</groupId> <artifactId>maven-project</artifactId> - <version>2.2.1</version> </dependency> <dependency> <groupId>junit</groupId> @@ -71,7 +67,6 @@ <dependency> <groupId>org.apache.maven.plugin-tools</groupId> <artifactId>maven-plugin-annotations</artifactId> - <version>3.2</version> <scope>provided</scope> </dependency> </dependencies> diff --git a/vespa-documentgen-plugin/src/main/java/com/yahoo/vespa/DocumentGenMojo.java b/vespa-documentgen-plugin/src/main/java/com/yahoo/vespa/DocumentGenMojo.java index 39dc6240b84..3a4b36d9bb5 100644 --- a/vespa-documentgen-plugin/src/main/java/com/yahoo/vespa/DocumentGenMojo.java +++ b/vespa-documentgen-plugin/src/main/java/com/yahoo/vespa/DocumentGenMojo.java @@ -17,7 +17,7 @@ import com.yahoo.document.annotation.AnnotationReferenceDataType; import com.yahoo.document.annotation.AnnotationType; import com.yahoo.documentmodel.NewDocumentType; import com.yahoo.documentmodel.VespaDocumentType; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import com.yahoo.searchdefinition.SearchBuilder; import com.yahoo.searchdefinition.document.FieldSet; import com.yahoo.searchdefinition.parser.ParseException; @@ -101,7 +101,7 @@ public class DocumentGenMojo extends AbstractMojo { required = true) private File outputDirectory; - private Map<String, Search> searches; + private Map<String, Schema> searches; private Map<String, String> docTypes; private Map<String, String> structTypes; private Map<String, String> annotationTypes; @@ -149,8 +149,8 @@ public class DocumentGenMojo extends AbstractMojo { } } builder.build(); - for (Search search : builder.getSearchList() ) { - this.searches.put(search.getName(), search); + for (Schema schema : builder.getSearchList() ) { + this.searches.put(schema.getName(), schema); } return builder; } @@ -993,7 +993,7 @@ public class DocumentGenMojo extends AbstractMojo { execute(dir, this.outputDirectory, packageName); } - Map<String, Search> getSearches() { + Map<String, Schema> getSearches() { return searches; } diff --git a/vespa-documentgen-plugin/src/test/java/com/yahoo/vespa/DocumentGenTest.java b/vespa-documentgen-plugin/src/test/java/com/yahoo/vespa/DocumentGenTest.java index 3aaaf52e54a..3879067be2e 100644 --- a/vespa-documentgen-plugin/src/test/java/com/yahoo/vespa/DocumentGenTest.java +++ b/vespa-documentgen-plugin/src/test/java/com/yahoo/vespa/DocumentGenTest.java @@ -4,7 +4,7 @@ package com.yahoo.vespa; import com.yahoo.document.DataType; import com.yahoo.document.StructDataType; import com.yahoo.document.WeightedSetDataType; -import com.yahoo.searchdefinition.Search; +import com.yahoo.searchdefinition.Schema; import org.junit.Test; import java.io.File; @@ -20,7 +20,7 @@ public class DocumentGenTest { public void testMusic() { DocumentGenMojo mojo = new DocumentGenMojo(); mojo.execute(new File("etc/music/"), new File("target/generated-test-sources/vespa-documentgen-plugin/"), "com.yahoo.vespa.document"); - Map<String, Search> searches = mojo.getSearches(); + Map<String, Schema> searches = mojo.getSearches(); assertEquals(searches.size(),1); assertEquals(searches.get("music").getDocument("music").getField("title").getDataType(), DataType.STRING); assertEquals(searches.get("music").getDocument("music").getField("eitheror").getDataType(), DataType.BOOL); @@ -30,7 +30,7 @@ public class DocumentGenTest { public void testComplex() { DocumentGenMojo mojo = new DocumentGenMojo(); mojo.execute(new File("etc/complex/"), new File("target/generated-test-sources/vespa-documentgen-plugin/"), "com.yahoo.vespa.document"); - Map<String, Search> searches = mojo.getSearches(); + Map<String, Schema> searches = mojo.getSearches(); assertEquals(searches.get("video").getDocument("video").getField("weight").getDataType(), DataType.FLOAT); assertEquals(searches.get("book").getDocument("book").getField("sw1").getDataType(), DataType.FLOAT); assertTrue(searches.get("music3").getDocument("music3").getField("pos").getDataType() instanceof StructDataType); @@ -44,7 +44,7 @@ public class DocumentGenTest { public void testLocalApp() { DocumentGenMojo mojo = new DocumentGenMojo(); mojo.execute(new File("etc/localapp/"), new File("target/generated-test-sources/vespa-documentgen-plugin/"), "com.yahoo.vespa.document"); - Map<String, Search> searches = mojo.getSearches(); + Map<String, Schema> searches = mojo.getSearches(); assertEquals(searches.get("video").getDocument("video").getField("weight").getDataType(), DataType.FLOAT); assertTrue(searches.get("book").getDocument("book").getField("mystruct").getDataType() instanceof StructDataType); assertTrue(searches.get("book").getDocument("book").getField("mywsfloat").getDataType() instanceof WeightedSetDataType); diff --git a/vespa-feed-client-cli/src/test/java/ai/vespa/feed/client/CliArgumentsTest.java b/vespa-feed-client-cli/src/test/java/ai/vespa/feed/client/CliArgumentsTest.java index 79aa5b1acfa..622956db530 100644 --- a/vespa-feed-client-cli/src/test/java/ai/vespa/feed/client/CliArgumentsTest.java +++ b/vespa-feed-client-cli/src/test/java/ai/vespa/feed/client/CliArgumentsTest.java @@ -1,4 +1,5 @@ -package ai.vespa.feed.client;// Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. +// Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. +package ai.vespa.feed.client; import org.junit.jupiter.api.Test; @@ -80,4 +81,4 @@ class CliArgumentsTest { assertEquals(expectedHelp, text); } -}
\ No newline at end of file +} diff --git a/vespa-feed-client/README.md b/vespa-feed-client/README.md index cef0815da66..75e556cd3a1 100644 --- a/vespa-feed-client/README.md +++ b/vespa-feed-client/README.md @@ -1,2 +1,3 @@ +<!-- Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. --> # vespa-feed-client Java client library for feeding over HTTP/2 at `/document/v1/` diff --git a/vespa-feed-client/src/test/java/ai/vespa/feed/client/SslContextBuilderTest.java b/vespa-feed-client/src/test/java/ai/vespa/feed/client/SslContextBuilderTest.java index 79d8ea609ce..a74f63f5cd2 100644 --- a/vespa-feed-client/src/test/java/ai/vespa/feed/client/SslContextBuilderTest.java +++ b/vespa-feed-client/src/test/java/ai/vespa/feed/client/SslContextBuilderTest.java @@ -1,4 +1,5 @@ -package ai.vespa.feed.client;// Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. +// Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. +package ai.vespa.feed.client; import org.bouncycastle.cert.jcajce.JcaX509CertificateConverter; import org.bouncycastle.cert.jcajce.JcaX509v3CertificateBuilder; @@ -109,4 +110,4 @@ class SslContextBuilderTest { .getCertificate(jcaCertBuilder.build(contentSigner)); } -}
\ No newline at end of file +} diff --git a/vespa-http-client/pom.xml b/vespa-http-client/pom.xml index 644ee38367e..7028b1802ed 100644 --- a/vespa-http-client/pom.xml +++ b/vespa-http-client/pom.xml @@ -100,8 +100,9 @@ <scope>test</scope> </dependency> <dependency> - <groupId>uk.co.datumedge</groupId> - <artifactId>hamcrest-json</artifactId> + <groupId>com.yahoo.vespa</groupId> + <artifactId>testutil</artifactId> + <version>${project.version}</version> <scope>test</scope> </dependency> <dependency> diff --git a/vespa-http-client/src/test/java/com/yahoo/vespa/http/client/JsonTestHelper.java b/vespa-http-client/src/test/java/com/yahoo/vespa/http/client/JsonTestHelper.java deleted file mode 100644 index b2c8e7c2f4f..00000000000 --- a/vespa-http-client/src/test/java/com/yahoo/vespa/http/client/JsonTestHelper.java +++ /dev/null @@ -1,61 +0,0 @@ -// Copyright 2019 Oath Inc. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. -package com.yahoo.vespa.http.client; - -import com.fasterxml.jackson.core.JsonProcessingException; -import com.fasterxml.jackson.databind.JsonNode; -import com.fasterxml.jackson.databind.ObjectMapper; -import com.fasterxml.jackson.datatype.jdk8.Jdk8Module; -import com.fasterxml.jackson.datatype.jsr310.JavaTimeModule; -import com.google.common.base.Joiner; -import org.hamcrest.MatcherAssert; - -import java.io.UncheckedIOException; - -import static uk.co.datumedge.hamcrest.json.SameJSONAs.sameJSONAs; - -public class JsonTestHelper { - - private static final ObjectMapper mapper = createMapper(); - - private static ObjectMapper createMapper() { - ObjectMapper mapper = new ObjectMapper(); - mapper.registerModule(new Jdk8Module()); - mapper.registerModule(new JavaTimeModule()); - return mapper; - } - - /** - * Convenience method to input JSON without escaping double quotes and newlines - * Each parameter represents a line of JSON encoded data - * The lines are joined with newline and single quotes are replaced with double quotes - */ - public static String inputJson(String... lines) { - return Joiner.on("\n").join(lines).replaceAll("'", "\""); - } - - /** Structurally compare two JSON encoded strings */ - public static void assertJsonEquals(String inputJson, String expectedJson) { - MatcherAssert.assertThat(inputJson, sameJSONAs(expectedJson)); - } - - /** Structurally compare a {@link JsonNode} and a JSON string. */ - public static void assertJsonEquals(JsonNode left, String rightJson) { - try { - String leftJson = mapper.writeValueAsString(left); - assertJsonEquals(leftJson, rightJson); - } catch (JsonProcessingException e) { - throw new UncheckedIOException(e); - } - } - - /** Structurally compare two {@link JsonNode}s. */ - public static void assertJsonEquals(JsonNode left, JsonNode right) { - try { - String leftJson = mapper.writeValueAsString(left); - String rightJson = mapper.writeValueAsString(right); - assertJsonEquals(leftJson, rightJson); - } catch (JsonProcessingException e) { - throw new UncheckedIOException(e); - } - } -} diff --git a/vespa-http-client/src/test/java/com/yahoo/vespa/http/client/runner/JsonReaderTest.java b/vespa-http-client/src/test/java/com/yahoo/vespa/http/client/runner/JsonReaderTest.java index 779e4713ce2..0a5b3771958 100644 --- a/vespa-http-client/src/test/java/com/yahoo/vespa/http/client/runner/JsonReaderTest.java +++ b/vespa-http-client/src/test/java/com/yahoo/vespa/http/client/runner/JsonReaderTest.java @@ -12,9 +12,9 @@ import java.util.ArrayList; import java.util.List; import java.util.concurrent.atomic.AtomicInteger; -import static com.yahoo.vespa.http.client.JsonTestHelper.inputJson; +import static com.yahoo.test.json.JsonTestHelper.inputJson; import static org.hamcrest.core.Is.is; -import static org.junit.Assert.*; +import static org.junit.Assert.assertThat; public class JsonReaderTest { diff --git a/vespa-maven-plugin/src/test/resources/effective-services/dev.xml b/vespa-maven-plugin/src/test/resources/effective-services/dev.xml index b8813f40711..d6343e7ace7 100644 --- a/vespa-maven-plugin/src/test/resources/effective-services/dev.xml +++ b/vespa-maven-plugin/src/test/resources/effective-services/dev.xml @@ -1,5 +1,5 @@ <?xml version="1.0" encoding="UTF-8" standalone="no"?> -<!-- Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. --><services xmlns:deploy="vespa"> +<services xmlns:deploy="vespa"> <container> <component id="good-service-client"> <config> diff --git a/vespa-maven-plugin/src/test/resources/effective-services/prod_us-east-3.xml b/vespa-maven-plugin/src/test/resources/effective-services/prod_us-east-3.xml index 1c71b17a21a..0bab4b7e74a 100644 --- a/vespa-maven-plugin/src/test/resources/effective-services/prod_us-east-3.xml +++ b/vespa-maven-plugin/src/test/resources/effective-services/prod_us-east-3.xml @@ -1,5 +1,5 @@ <?xml version="1.0" encoding="UTF-8" standalone="no"?> -<!-- Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. --><services xmlns:deploy="vespa"> +<services xmlns:deploy="vespa"> <container> <component id="good-service-client"> <config> diff --git a/vespa-maven-plugin/src/test/resources/effective-services/prod_us-west-1.xml b/vespa-maven-plugin/src/test/resources/effective-services/prod_us-west-1.xml index 66b6425a8cb..e29ba231092 100644 --- a/vespa-maven-plugin/src/test/resources/effective-services/prod_us-west-1.xml +++ b/vespa-maven-plugin/src/test/resources/effective-services/prod_us-west-1.xml @@ -1,5 +1,5 @@ <?xml version="1.0" encoding="UTF-8" standalone="no"?> -<!-- Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. --><services xmlns:deploy="vespa"> +<services xmlns:deploy="vespa"> <container> <component id="good-service-client"> <config> diff --git a/vespa-maven-plugin/src/test/resources/effective-services/services.xml b/vespa-maven-plugin/src/test/resources/effective-services/services.xml index de896d5c7bb..184eb2ab48c 100644 --- a/vespa-maven-plugin/src/test/resources/effective-services/services.xml +++ b/vespa-maven-plugin/src/test/resources/effective-services/services.xml @@ -1,5 +1,4 @@ <?xml version='1.0' encoding='UTF-8'?> -<!-- Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. --> <services xmlns:deploy="vespa"> <container> <component id="good-service-client"> diff --git a/vespa-maven-plugin/src/test/resources/effective-services/test_us-east-1.xml b/vespa-maven-plugin/src/test/resources/effective-services/test_us-east-1.xml index 1109e378414..479d4a7166a 100644 --- a/vespa-maven-plugin/src/test/resources/effective-services/test_us-east-1.xml +++ b/vespa-maven-plugin/src/test/resources/effective-services/test_us-east-1.xml @@ -1,5 +1,4 @@ -<?xml version="1.0" encoding="UTF-8" standalone="no"?> -<!-- Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. --> +<?xml version="1.0" encoding="UTF-8" standalone="no"?><!-- Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. --> <services xmlns:deploy="vespa"> <container> <component id="good-service-client"> diff --git a/vespa_feed_perf/src/main/java/com/yahoo/vespa/feed/perf/SimpleFeeder.java b/vespa_feed_perf/src/main/java/com/yahoo/vespa/feed/perf/SimpleFeeder.java index 4f8129dc76d..167e9b338a9 100644 --- a/vespa_feed_perf/src/main/java/com/yahoo/vespa/feed/perf/SimpleFeeder.java +++ b/vespa_feed_perf/src/main/java/com/yahoo/vespa/feed/perf/SimpleFeeder.java @@ -56,6 +56,7 @@ import java.util.concurrent.atomic.AtomicLong; import java.util.concurrent.atomic.AtomicReference; import java.util.logging.Level; import java.util.logging.Logger; +import java.util.stream.Stream; /** * @author Simon Thoresen Hult @@ -441,12 +442,16 @@ public class SimpleFeeder implements ReplyHandler { } } + private static boolean containsFatalErrors(Stream<Error> errors) { + return errors.anyMatch(e -> e.getCode() != DocumentProtocol.ERROR_TEST_AND_SET_CONDITION_FAILED); + } + @Override public void handleReply(Reply reply) { if (failure.get() != null) { return; } - if (reply.hasErrors()) { + if (containsFatalErrors(reply.getErrors())) { failure.compareAndSet(null, new IOException(formatErrors(reply))); return; } diff --git a/vespaclient-container-plugin/src/main/java/com/yahoo/document/restapi/resource/DocumentV1ApiHandler.java b/vespaclient-container-plugin/src/main/java/com/yahoo/document/restapi/resource/DocumentV1ApiHandler.java index aa96b0932c3..23bfde986cf 100644 --- a/vespaclient-container-plugin/src/main/java/com/yahoo/document/restapi/resource/DocumentV1ApiHandler.java +++ b/vespaclient-container-plugin/src/main/java/com/yahoo/document/restapi/resource/DocumentV1ApiHandler.java @@ -8,7 +8,9 @@ import com.yahoo.cloud.config.ClusterListConfig; import com.yahoo.concurrent.DaemonThreadFactory; import com.yahoo.container.core.HandlerMetricContextUtil; import com.yahoo.container.core.documentapi.VespaDocumentAccess; +import com.yahoo.container.handler.threadpool.ContainerThreadPool; import com.yahoo.container.jdisc.ContentChannelOutputStream; +import com.yahoo.container.jdisc.MaxPendingContentChannelOutputStream; import com.yahoo.document.Document; import com.yahoo.document.DocumentId; import com.yahoo.document.DocumentOperation; @@ -86,10 +88,12 @@ import java.util.Optional; import java.util.StringJoiner; import java.util.concurrent.ConcurrentHashMap; import java.util.concurrent.ConcurrentLinkedDeque; +import java.util.concurrent.Executor; import java.util.concurrent.Executors; import java.util.concurrent.Phaser; import java.util.concurrent.ScheduledExecutorService; import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicBoolean; import java.util.concurrent.atomic.AtomicLong; import java.util.concurrent.atomic.AtomicReference; import java.util.concurrent.locks.Lock; @@ -158,6 +162,7 @@ public class DocumentV1ApiHandler extends AbstractRequestHandler { private static final String TIME_CHUNK = "timeChunk"; private static final String TIMEOUT = "timeout"; private static final String TRACELEVEL = "tracelevel"; + private static final String STREAMING = "streaming"; private final Clock clock; private final Duration handlerTimeout; @@ -175,10 +180,12 @@ public class DocumentV1ApiHandler extends AbstractRequestHandler { private final Map<VisitorControlHandler, VisitorSession> visits = new ConcurrentHashMap<>(); private final ScheduledExecutorService dispatcher = Executors.newSingleThreadScheduledExecutor(new DaemonThreadFactory("document-api-handler-")); private final ScheduledExecutorService visitDispatcher = Executors.newSingleThreadScheduledExecutor(new DaemonThreadFactory("document-api-handler-visit-")); + private final Executor defaultExecutor; private final Map<String, Map<Method, Handler>> handlers = defineApi(); @Inject - public DocumentV1ApiHandler(Metric metric, + public DocumentV1ApiHandler(ContainerThreadPool threadPool, + Metric metric, MetricReceiver metricReceiver, VespaDocumentAccess documentAccess, DocumentmanagerConfig documentManagerConfig, @@ -186,12 +193,12 @@ public class DocumentV1ApiHandler extends AbstractRequestHandler { AllClustersBucketSpacesConfig bucketSpacesConfig, DocumentOperationExecutorConfig executorConfig) { this(Clock.systemUTC(), Duration.ofSeconds(5), metric, metricReceiver, documentAccess, - documentManagerConfig, executorConfig, clusterListConfig, bucketSpacesConfig); + documentManagerConfig, executorConfig, clusterListConfig, bucketSpacesConfig, threadPool.executor()); } DocumentV1ApiHandler(Clock clock, Duration handlerTimeout, Metric metric, MetricReceiver metricReceiver, DocumentAccess access, DocumentmanagerConfig documentmanagerConfig, DocumentOperationExecutorConfig executorConfig, - ClusterListConfig clusterListConfig, AllClustersBucketSpacesConfig bucketSpacesConfig) { + ClusterListConfig clusterListConfig, AllClustersBucketSpacesConfig bucketSpacesConfig, Executor defaultExecutor) { this.clock = clock; this.handlerTimeout = handlerTimeout; this.parser = new DocumentOperationParser(documentmanagerConfig); @@ -210,6 +217,7 @@ public class DocumentV1ApiHandler extends AbstractRequestHandler { executorConfig.resendDelayMillis(), executorConfig.resendDelayMillis(), TimeUnit.MILLISECONDS); + this.defaultExecutor = defaultExecutor; } // ------------------------------------------------ Requests ------------------------------------------------- @@ -355,9 +363,10 @@ public class DocumentV1ApiHandler extends AbstractRequestHandler { private ContentChannel getDocuments(HttpRequest request, DocumentPath path, ResponseHandler handler) { enqueueAndDispatch(request, handler, () -> { - VisitorParameters parameters = parseGetParameters(request, path); + boolean streaming = getProperty(request, STREAMING, booleanParser).orElse(false); + VisitorParameters parameters = parseGetParameters(request, path, streaming); return () -> { - visitAndWrite(request, parameters, handler); + visitAndWrite(request, parameters, handler, streaming); return true; // VisitorSession has its own throttle handling. }; }); @@ -573,19 +582,22 @@ public class DocumentV1ApiHandler extends AbstractRequestHandler { private static class JsonResponse implements AutoCloseable { private final BufferedContentChannel buffer = new BufferedContentChannel(); - private final OutputStream out = new ContentChannelOutputStream(buffer); - private final JsonGenerator json = jsonFactory.createGenerator(out); + private final OutputStream out; + private final JsonGenerator json; private final ResponseHandler handler; private ContentChannel channel; - private JsonResponse(ResponseHandler handler) throws IOException { + private JsonResponse(ResponseHandler handler, boolean streaming) throws IOException { this.handler = handler; + out = streaming ? new MaxPendingContentChannelOutputStream(buffer, 1 << 24) + : new ContentChannelOutputStream(buffer); + json = jsonFactory.createGenerator(out); json.writeStartObject(); } /** Creates a new JsonResponse with path and id fields written. */ static JsonResponse create(DocumentPath path, ResponseHandler handler) throws IOException { - JsonResponse response = new JsonResponse(handler); + JsonResponse response = new JsonResponse(handler, false); response.writePathId(path.rawPath()); response.writeDocId(path.id()); return response; @@ -593,15 +605,19 @@ public class DocumentV1ApiHandler extends AbstractRequestHandler { /** Creates a new JsonResponse with path field written. */ static JsonResponse create(HttpRequest request, ResponseHandler handler) throws IOException { - JsonResponse response = new JsonResponse(handler); + return create(request, handler, false); + } + + /** Creates a new JsonResponse with path field written. */ + static JsonResponse create(HttpRequest request, ResponseHandler handler, boolean streaming) throws IOException { + JsonResponse response = new JsonResponse(handler, streaming); response.writePathId(request.getUri().getRawPath()); return response; } /** Creates a new JsonResponse with path and message fields written. */ static JsonResponse create(HttpRequest request, String message, ResponseHandler handler) throws IOException { - JsonResponse response = new JsonResponse(handler); - response.writePathId(request.getUri().getRawPath()); + JsonResponse response = create(request, handler); response.writeMessage(message); return response; } @@ -955,8 +971,10 @@ public class DocumentV1ApiHandler extends AbstractRequestHandler { // ------------------------------------------------- Visits ------------------------------------------------ - private VisitorParameters parseGetParameters(HttpRequest request, DocumentPath path) { - int wantedDocumentCount = Math.min(1 << 10, getProperty(request, WANTED_DOCUMENT_COUNT, integerParser).orElse(1)); + private VisitorParameters parseGetParameters(HttpRequest request, DocumentPath path, boolean streaming) { + int wantedDocumentCount = Math.min(streaming ? Integer.MAX_VALUE : 1 << 10, + getProperty(request, WANTED_DOCUMENT_COUNT, integerParser) + .orElse(streaming ? Integer.MAX_VALUE : 1)); if (wantedDocumentCount <= 0) throw new IllegalArgumentException("wantedDocumentCount must be positive"); @@ -1015,10 +1033,10 @@ public class DocumentV1ApiHandler extends AbstractRequestHandler { /** Called at the start of response rendering. */ default void onStart(JsonResponse response) throws IOException { } - /** Called for every document received from backend visitors — must call the ack for these to proceed. */ + /** Called for every document received from backend visitors—must call the ack for these to proceed. */ default void onDocument(JsonResponse response, Document document, Runnable ack, Consumer<String> onError) { } - /** Called at the end of response rendering, before generic status data is written. */ + /** Called at the end of response rendering, before generic status data is written. Called from a dedicated thread pool. */ default void onEnd(JsonResponse response) throws IOException { } } @@ -1042,7 +1060,7 @@ public class DocumentV1ApiHandler extends AbstractRequestHandler { private void visitAndProcess(HttpRequest request, VisitorParameters parameters, ResponseHandler handler, String route, BiFunction<DocumentId, DocumentOperationParameters, Result> operation) { - visit(request, parameters, handler, new VisitCallback() { + visit(request, parameters, false, handler, new VisitCallback() { @Override public void onDocument(JsonResponse response, Document document, Runnable ack, Consumer<String> onError) { DocumentOperationParameters operationParameters = parameters().withRoute(route) .withResponseHandler(operationResponse -> { @@ -1079,66 +1097,92 @@ public class DocumentV1ApiHandler extends AbstractRequestHandler { }); } - private void visitAndWrite(HttpRequest request, VisitorParameters parameters, ResponseHandler handler) { - visit(request, parameters, handler, new VisitCallback() { + private void visitAndWrite(HttpRequest request, VisitorParameters parameters, ResponseHandler handler, boolean streaming) { + visit(request, parameters, streaming, handler, new VisitCallback() { + final Deque<Runnable> writes = new ConcurrentLinkedDeque<>(); + final AtomicBoolean writing = new AtomicBoolean(); @Override public void onStart(JsonResponse response) throws IOException { + if (streaming) + response.commit(Response.Status.OK); + response.writeDocumentsArrayStart(); } @Override public void onDocument(JsonResponse response, Document document, Runnable ack, Consumer<String> onError) { - response.writeDocumentValue(document); - ack.run(); + writes.add(() -> { + response.writeDocumentValue(document); + ack.run(); + }); + if (writing.compareAndSet(false, true)) // Occupy only a single thread for writing. + defaultExecutor.execute(() -> { + for (Runnable write; (write = writes.poll()) != null; write.run()); + writing.set(false); + }); } @Override public void onEnd(JsonResponse response) throws IOException { + // Wait for other writers to complete, then write what remains here. + while ( ! writing.compareAndSet(false, true)) { + try { + Thread.sleep(1); + } + catch (InterruptedException e) { + log.log(WARNING, "Interrupted waiting for visited documents to be written; this should not happen"); + Thread.currentThread().interrupt(); + } + } + for (Runnable write; (write = writes.poll()) != null; write.run()); response.writeArrayEnd(); } }); } private void visitWithRemote(HttpRequest request, VisitorParameters parameters, ResponseHandler handler) { - visit(request, parameters, handler, new VisitCallback() { }); + visit(request, parameters, false, handler, new VisitCallback() { }); } - private void visit(HttpRequest request, VisitorParameters parameters, ResponseHandler handler, VisitCallback callback) { + private void visit(HttpRequest request, VisitorParameters parameters, boolean streaming, ResponseHandler handler, VisitCallback callback) { try { - JsonResponse response = JsonResponse.create(request, handler); + JsonResponse response = JsonResponse.create(request, handler, streaming); Phaser phaser = new Phaser(2); // Synchronize this thread (dispatch) with the visitor callback thread. AtomicReference<String> error = new AtomicReference<>(); // Set if error occurs during processing of visited documents. callback.onStart(response); VisitorControlHandler controller = new VisitorControlHandler() { @Override public void onDone(CompletionCode code, String message) { - super.onDone(code, message); - loggingException(() -> { - callback.onEnd(response); - switch (code) { - case TIMEOUT: - if ( ! hasVisitedAnyBuckets() && parameters.getVisitInconsistentBuckets()) { - response.writeMessage("No buckets visited within timeout of " + - parameters.getSessionTimeoutMs() + "ms (request timeout -5s)"); - response.respond(Response.Status.GATEWAY_TIMEOUT); - break; - } - case SUCCESS: // Intentional fallthrough. - case ABORTED: // Intentional fallthrough. - if (error.get() == null) { - ProgressToken progress = getProgress() != null ? getProgress() : parameters.getResumeToken(); - if (progress != null && ! progress.isFinished()) - response.writeContinuation(progress.serializeToString()); + defaultExecutor.execute(() -> { + super.onDone(code, message); + loggingException(() -> { + try (response) { + callback.onEnd(response); - if (getVisitorStatistics() != null) - response.writeDocumentCount(getVisitorStatistics().getDocumentsVisited()); - - response.respond(Response.Status.OK); - break; - } - default: - response.writeMessage(error.get() != null ? error.get() : message != null ? message : "Visiting failed"); if (getVisitorStatistics() != null) response.writeDocumentCount(getVisitorStatistics().getDocumentsVisited()); - response.respond(Response.Status.BAD_GATEWAY); - } - }); - visitDispatcher.execute(() -> { + int status = Response.Status.BAD_GATEWAY; + switch (code) { + case TIMEOUT: + if ( ! hasVisitedAnyBuckets() && parameters.getVisitInconsistentBuckets()) { + response.writeMessage("No buckets visited within timeout of " + + parameters.getSessionTimeoutMs() + "ms (request timeout -5s)"); + status = Response.Status.GATEWAY_TIMEOUT; + break; + } + // TODO jonmv: always supply and document continuation? + case SUCCESS: // Intentional fallthrough. + case ABORTED: // Intentional fallthrough. + if (error.get() == null) { + ProgressToken progress = getProgress() != null ? getProgress() : parameters.getResumeToken(); + if (progress != null && ! progress.isFinished()) + response.writeContinuation(progress.serializeToString()); + + status = Response.Status.OK; + break; + } + default: + response.writeMessage(error.get() != null ? error.get() : message != null ? message : "Visiting failed"); + } + if ( ! streaming) + response.commit(status); + } + }); phaser.arriveAndAwaitAdvance(); // We may get here while dispatching thread is still putting us in the map. visits.remove(this).destroy(); }); diff --git a/vespaclient-container-plugin/src/test/java/com/yahoo/document/restapi/resource/DocumentV1ApiTest.java b/vespaclient-container-plugin/src/test/java/com/yahoo/document/restapi/resource/DocumentV1ApiTest.java index 96700f08823..9234f1cfa6f 100644 --- a/vespaclient-container-plugin/src/test/java/com/yahoo/document/restapi/resource/DocumentV1ApiTest.java +++ b/vespaclient-container-plugin/src/test/java/com/yahoo/document/restapi/resource/DocumentV1ApiTest.java @@ -4,6 +4,7 @@ package com.yahoo.document.restapi.resource; import com.yahoo.cloud.config.ClusterListConfig; import com.yahoo.container.jdisc.RequestHandlerTestDriver; import com.yahoo.docproc.jdisc.metric.NullMetric; +import com.yahoo.document.BucketId; import com.yahoo.document.Document; import com.yahoo.document.DocumentId; import com.yahoo.document.DocumentPut; @@ -37,6 +38,7 @@ import com.yahoo.documentapi.UpdateResponse; import com.yahoo.documentapi.VisitorControlHandler; import com.yahoo.documentapi.VisitorDestinationParameters; import com.yahoo.documentapi.VisitorDestinationSession; +import com.yahoo.documentapi.VisitorIterator; import com.yahoo.documentapi.VisitorParameters; import com.yahoo.documentapi.VisitorResponse; import com.yahoo.documentapi.VisitorSession; @@ -134,7 +136,8 @@ public class DocumentV1ApiTest { access = new MockDocumentAccess(docConfig); metric = new NullMetric(); metrics = new MetricReceiver.MockReceiver(); - handler = new DocumentV1ApiHandler(clock, Duration.ofMillis(1), metric, metrics, access, docConfig, executorConfig, clusterConfig, bucketConfig); + handler = new DocumentV1ApiHandler(clock, Duration.ofMillis(1), metric, metrics, access, docConfig, + executorConfig, clusterConfig, bucketConfig, Executors.newFixedThreadPool(2)); } @After @@ -245,19 +248,83 @@ public class DocumentV1ApiTest { "}", response.readAll()); assertEquals(200, response.getStatus()); + // GET at root is a visit. Streaming mode can be specified with &streaming=true + access.expect(tokens); + access.expect(parameters -> { + assertEquals("content", parameters.getRoute().toString()); + assertEquals("default", parameters.getBucketSpace()); + assertEquals(1025, parameters.getMaxTotalHits()); // Not bounded likewise for streaming responses. + assertEquals(100, ((StaticThrottlePolicy) parameters.getThrottlePolicy()).getMaxPendingCount()); + assertEquals("[id]", parameters.getFieldSet()); + assertEquals("(all the things)", parameters.getDocumentSelection()); + assertEquals(6000, parameters.getSessionTimeoutMs()); + // Put some documents in the response + parameters.getLocalDataHandler().onMessage(new PutDocumentMessage(new DocumentPut(doc1)), tokens.get(0)); + parameters.getLocalDataHandler().onMessage(new PutDocumentMessage(new DocumentPut(doc2)), tokens.get(1)); + parameters.getLocalDataHandler().onMessage(new PutDocumentMessage(new DocumentPut(doc3)), tokens.get(2)); + VisitorStatistics statistics = new VisitorStatistics(); + statistics.setBucketsVisited(1); + statistics.setDocumentsVisited(3); + parameters.getControlHandler().onVisitorStatistics(statistics); + parameters.getControlHandler().onDone(VisitorControlHandler.CompletionCode.TIMEOUT, "timeout is OK"); + }); + response = driver.sendRequest("http://localhost/document/v1?cluster=content&bucketSpace=default&wantedDocumentCount=1025&concurrency=123" + + "&selection=all%20the%20things&fieldSet=[id]&timeout=6&streaming=true"); + assertSameJson("{" + + " \"pathId\": \"/document/v1\"," + + " \"documents\": [" + + " {" + + " \"id\": \"id:space:music::one\"," + + " \"fields\": {" + + " \"artist\": \"Tom Waits\"" + + " }" + + " }," + + " {" + + " \"id\": \"id:space:music:n=1:two\"," + + " \"fields\": {" + + " \"artist\": \"Asa-Chan & Jun-Ray\"" + + " }" + + " }," + + " {" + + " \"id\": \"id:space:music:g=a:three\"," + + " \"fields\": {}" + + " }" + + " ]," + + " \"documentCount\": 3" + + "}", response.readAll()); + assertEquals(200, response.getStatus()); + // GET with namespace and document type is a restricted visit. + ProgressToken progress = new ProgressToken(); + VisitorIterator.createFromExplicitBucketSet(Set.of(new BucketId(1), new BucketId(2)), 8, progress) + .update(new BucketId(1), new BucketId(1)); access.expect(parameters -> { assertEquals("(music) and (id.namespace=='space')", parameters.getDocumentSelection()); - assertEquals(new ProgressToken().serializeToString(), parameters.getResumeToken().serializeToString()); + assertEquals(progress.serializeToString(), parameters.getResumeToken().serializeToString()); throw new IllegalArgumentException("parse failure"); }); - response = driver.sendRequest("http://localhost/document/v1/space/music/docid?continuation=" + new ProgressToken().serializeToString()); + response = driver.sendRequest("http://localhost/document/v1/space/music/docid?continuation=" + progress.serializeToString()); assertSameJson("{" + " \"pathId\": \"/document/v1/space/music/docid\"," + " \"message\": \"parse failure\"" + "}", response.readAll()); assertEquals(400, response.getStatus()); + // GET when a streaming visit returns status code 200 also when errors occur. + access.expect(parameters -> { + assertEquals("(music) and (id.namespace=='space')", parameters.getDocumentSelection()); + parameters.getControlHandler().onProgress(progress); + parameters.getControlHandler().onDone(VisitorControlHandler.CompletionCode.FAILURE, "failure?"); + }); + response = driver.sendRequest("http://localhost/document/v1/space/music/docid?streaming=true"); + assertSameJson("{" + + " \"pathId\": \"/document/v1/space/music/docid\"," + + " \"documents\": []," + + //" \"continuation\": \"" + progress.serializeToString() + "\"," + + " \"message\": \"failure?\"" + + "}", response.readAll()); + assertEquals(200, response.getStatus()); + // POST with namespace and document type is a restricted visit with a required destination cluster ("destinationCluster") access.expect(parameters -> { fail("Not supposed to run"); @@ -686,7 +753,8 @@ public class DocumentV1ApiTest { @Test public void testThroughput() throws InterruptedException { DocumentOperationExecutorConfig executorConfig = new DocumentOperationExecutorConfig.Builder().build(); - handler = new DocumentV1ApiHandler(clock, Duration.ofMillis(1), metric, metrics, access, docConfig, executorConfig, clusterConfig, bucketConfig); + handler = new DocumentV1ApiHandler(clock, Duration.ofMillis(1), metric, metrics, access, docConfig, + executorConfig, clusterConfig, bucketConfig, Executors.newFixedThreadPool(2)); int writers = 4; int queueFill = executorConfig.maxThrottled() - writers; @@ -737,7 +805,7 @@ public class DocumentV1ApiTest { replier.schedule(() -> parameters.responseHandler().get().handleResponse(success), 10, TimeUnit.MILLISECONDS); return new Result(0); }); - // Send the rest of the documents. Rely on resender to empty queue of throttled oppperations. + // Send the rest of the documents. Rely on resender to empty queue of throttled operations. for (int i = queueFill; i < docs; i++) { int j = i; writer.execute(() -> { diff --git a/vespajlib/src/main/java/com/yahoo/io/Utf8ByteWriter.java b/vespajlib/src/main/java/com/yahoo/io/Utf8ByteWriter.java index 949cd88e580..c4cf25f34a0 100644 --- a/vespajlib/src/main/java/com/yahoo/io/Utf8ByteWriter.java +++ b/vespajlib/src/main/java/com/yahoo/io/Utf8ByteWriter.java @@ -1,3 +1,4 @@ +// Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. package com.yahoo.io; import com.yahoo.text.Utf8; diff --git a/vespalib/CMakeLists.txt b/vespalib/CMakeLists.txt index 472696131d2..6acac5d3bf5 100644 --- a/vespalib/CMakeLists.txt +++ b/vespalib/CMakeLists.txt @@ -55,6 +55,7 @@ vespa_define_module( src/tests/eventbarrier src/tests/exception_classes src/tests/executor + src/tests/executor_idle_tracking src/tests/explore_modern_cpp src/tests/false src/tests/fiddle diff --git a/vespalib/src/tests/executor_idle_tracking/CMakeLists.txt b/vespalib/src/tests/executor_idle_tracking/CMakeLists.txt new file mode 100644 index 00000000000..5ad8f33db91 --- /dev/null +++ b/vespalib/src/tests/executor_idle_tracking/CMakeLists.txt @@ -0,0 +1,9 @@ +# Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. +vespa_add_executable(vespalib_executor_idle_tracking_test_app TEST + SOURCES + executor_idle_tracking_test.cpp + DEPENDS + vespalib + GTest::GTest +) +vespa_add_test(NAME vespalib_executor_idle_tracking_test_app COMMAND vespalib_executor_idle_tracking_test_app) diff --git a/vespalib/src/tests/executor_idle_tracking/executor_idle_tracking_test.cpp b/vespalib/src/tests/executor_idle_tracking/executor_idle_tracking_test.cpp new file mode 100644 index 00000000000..42faebe45f7 --- /dev/null +++ b/vespalib/src/tests/executor_idle_tracking/executor_idle_tracking_test.cpp @@ -0,0 +1,79 @@ +// Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. + +#include <vespa/vespalib/util/executor_idle_tracking.h> +#include <vespa/vespalib/gtest/gtest.h> + +using namespace vespalib; + +steady_time at(duration d) { return steady_time(d); } + +TEST(ThreadIdleTrackingTest, can_track_idle_time) { + ThreadIdleTracker state; + EXPECT_FALSE(state.is_idle()); // starts in active state + state.set_idle(at(50ms)); + EXPECT_TRUE(state.is_idle()); + EXPECT_EQ(count_ms(state.set_active(at(65ms))), 15); + EXPECT_FALSE(state.is_idle()); + state.set_idle(at(100ms)); + EXPECT_TRUE(state.is_idle()); + EXPECT_EQ(count_ms(state.set_active(at(150ms))), 50); + EXPECT_FALSE(state.is_idle()); +} + +TEST(ThreadIdleTrackingTest, redundant_set_idle_is_handled) { + ThreadIdleTracker state; + state.set_idle(at(50ms)); + state.set_idle(at(100ms)); + EXPECT_TRUE(state.is_idle()); + EXPECT_EQ(count_ms(state.set_active(at(150ms))), 100); +} + +TEST(ThreadIdleTrackingTest, redundant_set_active_is_handled) { + ThreadIdleTracker state; + state.set_idle(at(50ms)); + EXPECT_EQ(count_ms(state.set_active(at(150ms))), 100); + EXPECT_EQ(count_ms(state.set_active(at(200ms))), 0); + EXPECT_FALSE(state.is_idle()); +} + +TEST(ThreadIdleTrackingTest, reset_consumes_idle_time_when_idle) { + ThreadIdleTracker state; + state.set_idle(at(50ms)); + EXPECT_EQ(count_ms(state.reset(at(100ms))), 50); + EXPECT_TRUE(state.is_idle()); + EXPECT_EQ(count_ms(state.set_active(at(150ms))), 50); +} + +TEST(ThreadIdleTrackingTest, reset_does_nothing_when_active) { + ThreadIdleTracker state; + EXPECT_EQ(count_ms(state.reset(at(100ms))), 0); + EXPECT_FALSE(state.is_idle()); +} + +TEST(ExecutorIdleTrackingTest, can_calculate_idle_metric) { + ExecutorIdleTracker state(at(100ms)); + state.was_idle(20ms); + state.was_idle(5ms); + state.was_idle(15ms); + state.was_idle(3ms); + state.was_idle(7ms); // 50 ms total idle + EXPECT_EQ(state.reset(at(120ms), 5), 0.5); // 100 ms total time + EXPECT_EQ(state.reset(at(140ms), 5), 0.0); + state.was_idle(25ms); + EXPECT_EQ(state.reset(at(160ms), 5), 0.25); +} + +TEST(ExecutorIdleTrackingTest, avoids_idle_above_1) { + ExecutorIdleTracker state(at(100ms)); + state.was_idle(100ms); + EXPECT_EQ(state.reset(at(110ms), 1), 1.0); +} + +TEST(ExecutorIdleTrackingTest, avoids_division_by_zero) { + ExecutorIdleTracker state(at(100ms)); + EXPECT_EQ(state.reset(at(100ms), 1), 0.0); + state.was_idle(10ms); + EXPECT_EQ(state.reset(at(100ms), 1), 1.0); +} + +GTEST_MAIN_RUN_ALL_TESTS() diff --git a/vespalib/src/vespa/vespalib/stllike/asciistream.cpp b/vespalib/src/vespa/vespalib/stllike/asciistream.cpp index 8e4a0e7567a..bedd0f119c6 100644 --- a/vespalib/src/vespa/vespalib/stllike/asciistream.cpp +++ b/vespalib/src/vespa/vespalib/stllike/asciistream.cpp @@ -7,7 +7,6 @@ #include <vespa/vespalib/util/size_literals.h> #include <vespa/vespalib/locale/c.h> #include <vespa/fastos/file.h> -#include <algorithm> #include <limits> #include <stdexcept> #include <cassert> @@ -615,9 +614,9 @@ asciistream asciistream::createFromDevice(stringref fileName) FastOS_File file(vespalib::string(fileName).c_str()); asciistream is; if (file.OpenReadOnly()) { - char buf[8_Ki]; - for (ssize_t actual = file.Read(buf, sizeof(buf)); actual > 0; actual = file.Read(buf, sizeof(buf))) { - is << stringref(buf, actual); + MallocPtr buf(64_Ki); + for (ssize_t actual = file.Read(buf, buf.size()); actual > 0; actual = file.Read(buf, buf.size())) { + is << stringref(buf.c_str(), actual); } } return is; diff --git a/vespalib/src/vespa/vespalib/util/CMakeLists.txt b/vespalib/src/vespa/vespalib/util/CMakeLists.txt index 6115c21623d..75ea02d448e 100644 --- a/vespalib/src/vespa/vespalib/util/CMakeLists.txt +++ b/vespalib/src/vespa/vespalib/util/CMakeLists.txt @@ -23,6 +23,7 @@ vespa_add_library(vespalib_vespalib_util OBJECT error.cpp exception.cpp exceptions.cpp + executor_idle_tracking.cpp file_area_freelist.cpp gencnt.cpp generationhandler.cpp diff --git a/vespalib/src/vespa/vespalib/util/executor_idle_tracking.cpp b/vespalib/src/vespa/vespalib/util/executor_idle_tracking.cpp new file mode 100644 index 00000000000..2e3c98c872f --- /dev/null +++ b/vespalib/src/vespa/vespalib/util/executor_idle_tracking.cpp @@ -0,0 +1,6 @@ +// Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. + +#include "executor_idle_tracking.h" + +namespace vespa { +} diff --git a/vespalib/src/vespa/vespalib/util/executor_idle_tracking.h b/vespalib/src/vespa/vespalib/util/executor_idle_tracking.h new file mode 100644 index 00000000000..f85680b9726 --- /dev/null +++ b/vespalib/src/vespa/vespalib/util/executor_idle_tracking.h @@ -0,0 +1,79 @@ +// Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. + +#pragma once + +#include "time.h" + +// Used by various executors to adjust the utilization number reported +// in ExecutorStats. How to use (also see unit test): +// +// (0) Remember that these classes are not thread-safe themselves +// +// (1) Keep one ThreadIdleTracker per worker thread and one +// ExecutorIdleTracker in the executor itself. Note that the +// ExecutorIdleTracker needs the current time as a constructor +// parameter. +// +// (2) Each time a worker thread is blocked; call set_idle with the +// current time right before blocking and call set_active with the +// current time right after waking up again. Pass the result from +// the set_active function to the was_idle function. +// +// (3) Each time stats are sampled, start by sampling the current +// time. Then call ThreadIdleTracker::reset for (at least) all +// blocked worker threads and pass the results to the was_idle +// function. Then call ExecutorIdleTracker::reset with the current +// time and the number of threads as parameters. Subtract this +// result from the utilization of the stats to be reported. + +namespace vespalib { + +class ThreadIdleTracker { +private: + steady_time _idle_tag = steady_time::min(); +public: + bool is_idle() const { return (_idle_tag != steady_time::min()); } + void set_idle(steady_time t) { + if (!is_idle()) { + _idle_tag = t; + } + } + duration set_active(steady_time t) { + if (is_idle()) { + duration how_long_idle = (t - _idle_tag); + _idle_tag = steady_time::min(); + return how_long_idle; + } else { + return duration::zero(); + } + } + duration reset(steady_time t) { + if (is_idle()) { + duration how_long_idle = (t - _idle_tag); + _idle_tag = t; + return how_long_idle; + } else { + return duration::zero(); + } + } +}; + +class ExecutorIdleTracker { +private: + steady_time _start; + duration _total_idle = duration::zero(); +public: + ExecutorIdleTracker(steady_time t) : _start(t) {} + void was_idle(duration how_long_idle) { + _total_idle += how_long_idle; + } + double reset(steady_time t, size_t num_threads) { + double idle = count_ns(_total_idle); + double elapsed = std::max(idle, double(count_ns((t - _start) * num_threads))); + _start = t; + _total_idle = duration::zero(); + return (elapsed > 0) ? (idle / elapsed) : 0.0; + } +}; + +} |