Skip to content

[CALCITE-5700] Add ARRAY_SIZE, ARRAY_REPEAT function (enabled in Spar… #3198

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
May 17, 2023

Conversation

liuyongvs
Copy link
Contributor

ARRAY_SIZE - Returns the size of an array.

ARRAY_REPEAT - Returns the array containing element count times.

For more details
https://spark.apache.org/docs/latest/api/sql/index.html

@liuyongvs
Copy link
Contributor Author

liuyongvs commented May 12, 2023

comment:

  1. the NullPolicy should None, because the second arg is null, the result return null
defineMethod(ARRAY_REPEAT, BuiltInMethod.ARRAY_REPEAT.method, NullPolicy.NONE);
  1. the input type should
    override def inputTypes: Seq[AbstractDataType] = Seq(AnyDataType, IntegerType)

https://github.com/apache/spark/blob/1782d0ed9d3c82caee8e57b94229184e308d8b84/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala#L3519

@liuyongvs
Copy link
Contributor Author

hi @JiajunBernoulli could you also help review?

@JiajunBernoulli
Copy link
Contributor

CI failed, please fix them.

@liuyongvs
Copy link
Contributor Author

@JiajunBernoulli ci passed

if (count == null) {
return null;
}
return Collections.nCopies((Integer) count, element);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can it be Long?

I am worried that the Integer will fail for Long.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@JiajunBernoulli it only can be integer in spark, i comments it before.

 override def inputTypes: Seq[AbstractDataType] = Seq(AnyDataType, IntegerType)

"No match found for function signature ARRAY_REPEAT\\(<NUMERIC>, <NUMERIC>\\)", false);

final SqlOperatorFixture f = f0.withLibrary(SqlLibrary.SPARK);
f.checkScalar("array_repeat(1, 2)", "[1, 1]",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If count <= 0, it is easy to understand for error message?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i check the spark it will be to zero.

"No match found for function signature ARRAY_SIZE\\(<INTEGER ARRAY>\\)", false);

final SqlOperatorFixture f = f0.withLibrary(SqlLibrary.SPARK);
f.checkScalar("array_size(array[1])", "1",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as above.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

array type and element type is different.
array type nullable which indicates wether it is nullable, the value is null
element type nullable which indicates wether the element is nullable, the value is [null, ...] like this

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok.

@@ -629,6 +629,7 @@ public enum BuiltInMethod {
SUBMULTISET_OF(SqlFunctions.class, "submultisetOf", Collection.class,
Collection.class),
ARRAY_DISTINCT(SqlFunctions.class, "distinct", List.class),
ARRAY_REPEAT(SqlFunctions.class, "repeat", Object.class, Integer.class),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would Long.class be better?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why?

@liuyongvs
Copy link
Contributor Author

fix all your review @JiajunBernoulli thanks

@liuyongvs liuyongvs requested a review from JiajunBernoulli May 14, 2023 13:42
@JiajunBernoulli
Copy link
Contributor

JiajunBernoulli commented May 15, 2023

Can we declare the type of the element?
image

For example, List<Object>
image

Here is code smells link: https://sonarcloud.io/project/issues?resolved=false&types=CODE_SMELL&pullRequest=3198&id=apache_calcite&open=AYgY9shmOpxb3UapS_aX

@liuyongvs
Copy link
Contributor Author

Can we declare the type of the element? image

For example, List<Object> image

Here is code smells link: https://sonarcloud.io/project/issues?resolved=false&types=CODE_SMELL&pullRequest=3198&id=apache_calcite&open=AYgY9shmOpxb3UapS_aX

fixed, Thank you for your very detailed code review. @JiajunBernoulli

@sonarqubecloud
Copy link

Kudos, SonarCloud Quality Gate passed!    Quality Gate passed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 2 Code Smells

100.0% 100.0% Coverage
0.0% 0.0% Duplication

@liuyongvs
Copy link
Contributor Author

hi @JiajunBernoulli squash the commits and ci passed, will it be merged now, thanks

@JiajunBernoulli JiajunBernoulli added the LGTM-will-merge-soon Overall PR looks OK. Only minor things left. label May 16, 2023
@JiajunBernoulli
Copy link
Contributor

If there are no other suggestions, I will merge tomorrow.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
LGTM-will-merge-soon Overall PR looks OK. Only minor things left.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants