Details
-
Bug
-
Status: Closed
-
Major
-
Resolution: Fixed
-
1.35.0
Description
The following are the implementations of some array functions in calcite(Spark Library) that are inconsistent with actual spark behavior.
The reason here is that null and cast (null as xxx) are treated equally, and NullPolicy acts on these two situations at the same time and returns null directly. However in spark, the former needs to throw an exception, and the correct behavior is the latter. (In fact, apache flink also throws an exception.) we should throw exception in such case to match Spark behavior.
calcite spark:
// return null
select array_contains(array[1, 2], null);
// return null
select array_except(array[1, 2, 3], null)
// return null
select array_intersect(array[1,2,3], null)
actual spark:
// Cannot resolve "array_contains(array(1, 2), NULL)" due to data type mismatch: // Null typed values cannot be used as arguments of `array_contains` spark-sql (default)> select array_contains(array(1, 2), null);
// data type mismatch: Input to function `array_except` should have been two "ARRAY" with same element type, // but it's ["ARRAY<INT>", "VOID"] spark-sql (default)> select array_except(array(1, 2, 3), null);
// data type mismatch: Input to function `array_intersect` should have been two "ARRAY" with same element type, // but it's ["ARRAY<INT>", "VOID"] spark-sql (default)> select array_intersect(array(1,2,3), null);
Attachments
Issue Links
- links to