-
Notifications
You must be signed in to change notification settings - Fork 2.4k
[CALCITE-3627] New Row Null policy #2666
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Closed
talatuyarer
wants to merge
7
commits into
apache:master
from
talatuyarer:CALCITE-3627-row-nullpolicy
Closed
[CALCITE-3627] New Row Null policy #2666
talatuyarer
wants to merge
7
commits into
apache:master
from
talatuyarer:CALCITE-3627-row-nullpolicy
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
1. Top-down trait request 2. Bottom-up trait derivation 3. Trait enforcement without AbstractConverter How to use? 1. Enable top-down optimization by setting {VolcanoPlanner#setTopDownOpt(boolean)} or add 'calcite.planner.topdown.opt=true' to saffron.properties config file. 2. Let your convention's rel interface extends {PhysicalNode}, see {EnumerableRel} as an example. 3. Each physical operator overrides any one of the two methods: {PhysicalNode#passThrough(RelTraitSet)} or {PhysicalNode#passThroughTraits(RelTraitSet)} depending on your needs. 4. Choose derive mode for each physical operator by overriding {PhysicalNode#getDeriveMode()}. 5. If the derive mode is {DeriveMode#OMAKASE}, override method {PhysicalNode#derive(List)} in the physical operator, otherwise, override {PhysicalNode#derive(RelTraitSet, int)} or {PhysicalNode#deriveTraits(RelTraitSet, int)}. 6. Mark your enforcer operator by overriding {RelNode#isEnforcer()}, see {Sort#isEnforcer()} as an example. This is important, because it can help {VolcanoPlanner} avoid unnecessary trait propagation and derivation, therefore improve optimization efficiency. 7. Implement {Convention#enforce(RelNode, RelTraitSet)} in your convention, which generates appropriate physical enforcer. See {EnumerableConvention#enforce(RelNode, RelTraitSet)} as example. Simply return null if you don't want physical trait enforcement. How does it work? Let S# denote the seed physical operator in a RelSet after logical and physical rules transformation, P# denote the physical operator generated by passing down parent trait requirements, D# denote the physical operator generated by deriving from child delivered traitSets. The initial rel list state in a RelSet is as follows: cursor | V S1, S2 When we create a task for RelSubset1, the task will immediately pass the subset's traitSet to seed operators, S1 and S2, now we have: cursor | V S1, S2, P1, P2 The subset task will create a optimization task for the relnode pointed by cursor, and move cursor to next available physical operator S2. In the task for S1, it will continue optimize its child nodes, which are RelSubsets. After child inputs optimization is finished, S1 will derive new relnodes from delivered subsets in input RelSet. Once task for S1 is completed, we have: cursor | V S1, S2, P1, P2, D1 The subset task continues scheduling task for S2, P1... until there is no more relnode created for the RelSet, then we have: cursor | V S1, S2, P1, P2, D1, D2, D3, null When a task for another RelSubset2 is created, the task will try to pass down the subset's traitSet to seed operator S1 and S2, now the RelSet looks like: cursor | V S1, S2, P1, P2, D1, D2, D3, P3, P4 The process continues till there is no more subsets or relnodes created for the RelSet. See https://lists.apache.org/thread.html/r38ea71968c069f465921e7197488329c15413b46831c90ad4d48f87e%40%3Cdev.calcite.apache.org%3E Close apache#1953
…ementation[CALCITE-3224]
…tuyarer/calcite into CALCITE-3627-row-nullpolicy � Conflicts: � core/src/main/java/org/apache/calcite/adapter/enumerable/EnumerableConvention.java � core/src/main/java/org/apache/calcite/adapter/enumerable/EnumerableMergeJoin.java � core/src/main/java/org/apache/calcite/adapter/enumerable/EnumerableRel.java � core/src/main/java/org/apache/calcite/adapter/enumerable/EnumerableSortedAggregate.java � core/src/main/java/org/apache/calcite/adapter/enumerable/EnumerableSortedAggregateRule.java � core/src/main/java/org/apache/calcite/adapter/enumerable/NullPolicy.java � core/src/main/java/org/apache/calcite/adapter/enumerable/RexImpTable.java � core/src/main/java/org/apache/calcite/config/CalciteSystemProperty.java � core/src/main/java/org/apache/calcite/plan/Convention.java � core/src/main/java/org/apache/calcite/plan/RelTraitSet.java � core/src/main/java/org/apache/calcite/plan/volcano/RelSet.java � core/src/main/java/org/apache/calcite/plan/volcano/RelSubset.java � core/src/main/java/org/apache/calcite/plan/volcano/VolcanoPlanner.java � core/src/main/java/org/apache/calcite/rel/PhysicalNode.java � core/src/main/java/org/apache/calcite/rel/RelCollations.java � core/src/test/java/org/apache/calcite/test/RelOptTestBase.java � core/src/test/java/org/apache/calcite/test/TopDownOptTest.java � core/src/test/resources/org/apache/calcite/test/TopDownOptTest.xml � core/src/test/resources/saffron.properties � core/src/test/resources/sql/struct.iq � plus/src/test/java/org/apache/calcite/adapter/tpcds/TpcdsTest.java
I only changed 3 files but somehow it shows more than 7 files. |
…Gen Implementation[CALCITE-3224]" This reverts commit 89cb40e.
…ementation[CALCITE-3224]
So sorry I have to close this pr too :( |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
As describe under CALCITE-3627 ticket. Current row policy is wrong. When ROW has a null column. Calcite returns null for ROW value. I am working on Apache Beam SQL Row support. Because of ANY policy RexToLixTranslator generate wrong code. This is my first pr for calcite project. Thanks for your comments in advance.
I recreated my pr because old pr did not puck up updated code.
@DonnyZone could you re-review ? I updated my pr after your RexNode-to-Expression CodeGen Implementation.
Thanks