Uploaded image for project: 'Calcite'
  1. Calcite
  2. CALCITE-5893

Wrong NULL operand behavior of ARRAY_CONTAINS/ARRAY_EXCEPT/ARRAY_INTERSECT In Spark Library

    XMLWordPrintableJSON

Details

    Description

      The following are the implementations of some array functions in calcite(Spark Library) that are inconsistent with actual spark behavior.

       

      The reason here is that null and cast (null as xxx) are treated equally, and NullPolicy acts on these two situations at the same time and returns null directly. However in spark, the former needs to throw an exception, and the correct behavior is the latter. (In fact, apache flink also throws an exception.)  we should throw exception in such case to match Spark behavior.

      calcite spark:

      // return null
      select array_contains(array[1, 2], null);
      // return null
      select array_except(array[1, 2, 3], null)
      // return null
      select array_intersect(array[1,2,3], null)

      actual spark:

      // Cannot resolve "array_contains(array(1, 2), NULL)" due to data type mismatch: // Null typed values cannot be used as arguments of `array_contains`
      
      spark-sql (default)> select array_contains(array(1, 2), null); 
      // data type mismatch: Input to function `array_except` should have been two "ARRAY" with same element type, 
      // but it's ["ARRAY<INT>", "VOID"]
      
      spark-sql (default)> select array_except(array(1, 2, 3), null); 
      // data type mismatch: Input to function `array_intersect` should have been two "ARRAY" with same element type, 
      // but it's ["ARRAY<INT>", "VOID"]
      
      spark-sql (default)> select array_intersect(array(1,2,3), null); 

      Attachments

        Issue Links

          Activity

            People

              taoran Ran Tao
              taoran Ran Tao
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: