Skip to content

[CALCITE-3627] New Row Null policy #2666

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed

Conversation

talatuyarer
Copy link

@talatuyarer talatuyarer commented Dec 30, 2021

As describe under CALCITE-3627 ticket. Current row policy is wrong. When ROW has a null column. Calcite returns null for ROW value. I am working on Apache Beam SQL Row support. Because of ANY policy RexToLixTranslator generate wrong code. This is my first pr for calcite project. Thanks for your comments in advance.

I recreated my pr because old pr did not puck up updated code.

@DonnyZone could you re-review ? I updated my pr after your RexNode-to-Expression CodeGen Implementation.

Thanks

hsyuan and others added 5 commits May 10, 2020 16:54
1. Top-down trait request
2. Bottom-up trait derivation
3. Trait enforcement without AbstractConverter

How to use?

1. Enable top-down optimization by setting {VolcanoPlanner#setTopDownOpt(boolean)}
or add 'calcite.planner.topdown.opt=true' to saffron.properties config file.

2. Let your convention's rel interface extends {PhysicalNode}, see
{EnumerableRel} as an example.

3. Each physical operator overrides any one of the two methods:
{PhysicalNode#passThrough(RelTraitSet)} or
{PhysicalNode#passThroughTraits(RelTraitSet)} depending on your needs.

4. Choose derive mode for each physical operator by overriding
{PhysicalNode#getDeriveMode()}.

5. If the derive mode is {DeriveMode#OMAKASE}, override method
{PhysicalNode#derive(List)} in the physical operator, otherwise, override
{PhysicalNode#derive(RelTraitSet, int)} or
{PhysicalNode#deriveTraits(RelTraitSet, int)}.

6. Mark your enforcer operator by overriding {RelNode#isEnforcer()}, see
{Sort#isEnforcer()} as an example. This is important, because it can help
{VolcanoPlanner} avoid unnecessary trait propagation and derivation, therefore
improve optimization efficiency.

7. Implement {Convention#enforce(RelNode, RelTraitSet)} in your convention,
which generates appropriate physical enforcer. See
{EnumerableConvention#enforce(RelNode, RelTraitSet)} as example. Simply return
null if you don't want physical trait enforcement.

How does it work?

Let S# denote the seed physical operator in a RelSet after logical and physical
rules transformation, P# denote the physical operator generated by passing down
parent trait requirements, D# denote the physical operator generated by
deriving from child delivered traitSets.

The initial rel list state in a RelSet is as follows:
    cursor
      |
      V
     S1, S2

When we create a task for RelSubset1, the task will immediately pass the
subset's traitSet to seed operators, S1 and S2, now we have:
    cursor
      |
      V
     S1, S2, P1, P2

The subset task will create a optimization task for the relnode pointed by
cursor, and move cursor to next available physical operator S2. In the task for
S1, it will continue optimize its child nodes, which are RelSubsets. After
child inputs optimization is finished, S1 will derive new relnodes from
delivered subsets in input RelSet. Once task for S1 is completed, we have:
        cursor
          |
          V
     S1, S2, P1, P2, D1

The subset task continues scheduling task for S2, P1... until there is no more
relnode created for the RelSet, then we have:
                                cursor
                                  |
                                  V
     S1, S2, P1, P2, D1, D2, D3, null

When a task for another RelSubset2 is created, the task will try to pass down
the subset's traitSet to seed operator S1 and S2, now the RelSet looks like:
                                cursor
                                  |
                                  V
     S1, S2, P1, P2, D1, D2, D3, P3, P4

The process continues till there is no more subsets or relnodes created for the
RelSet.

See https://lists.apache.org/thread.html/r38ea71968c069f465921e7197488329c15413b46831c90ad4d48f87e%40%3Cdev.calcite.apache.org%3E

Close apache#1953
…tuyarer/calcite into CALCITE-3627-row-nullpolicy

� Conflicts:
�	core/src/main/java/org/apache/calcite/adapter/enumerable/EnumerableConvention.java
�	core/src/main/java/org/apache/calcite/adapter/enumerable/EnumerableMergeJoin.java
�	core/src/main/java/org/apache/calcite/adapter/enumerable/EnumerableRel.java
�	core/src/main/java/org/apache/calcite/adapter/enumerable/EnumerableSortedAggregate.java
�	core/src/main/java/org/apache/calcite/adapter/enumerable/EnumerableSortedAggregateRule.java
�	core/src/main/java/org/apache/calcite/adapter/enumerable/NullPolicy.java
�	core/src/main/java/org/apache/calcite/adapter/enumerable/RexImpTable.java
�	core/src/main/java/org/apache/calcite/config/CalciteSystemProperty.java
�	core/src/main/java/org/apache/calcite/plan/Convention.java
�	core/src/main/java/org/apache/calcite/plan/RelTraitSet.java
�	core/src/main/java/org/apache/calcite/plan/volcano/RelSet.java
�	core/src/main/java/org/apache/calcite/plan/volcano/RelSubset.java
�	core/src/main/java/org/apache/calcite/plan/volcano/VolcanoPlanner.java
�	core/src/main/java/org/apache/calcite/rel/PhysicalNode.java
�	core/src/main/java/org/apache/calcite/rel/RelCollations.java
�	core/src/test/java/org/apache/calcite/test/RelOptTestBase.java
�	core/src/test/java/org/apache/calcite/test/TopDownOptTest.java
�	core/src/test/resources/org/apache/calcite/test/TopDownOptTest.xml
�	core/src/test/resources/saffron.properties
�	core/src/test/resources/sql/struct.iq
�	plus/src/test/java/org/apache/calcite/adapter/tpcds/TpcdsTest.java
@talatuyarer
Copy link
Author

I only changed 3 files but somehow it shows more than 7 files.

Talat Uyarer added 2 commits December 29, 2021 23:03
@talatuyarer
Copy link
Author

So sorry I have to close this pr too :(

@talatuyarer talatuyarer deleted the CALCITE-3627-row-nullpolicy branch December 30, 2021 07:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants