Skip to content

Comments

fix: Don't prevent apply from running given duplicate empty names in data sources. Also fix repeated apply of Spark data source.#2415

Merged
feast-ci-bot merged 15 commits intofeast-dev:masterfrom
adchia:fixApply
Mar 29, 2022
Merged

Conversation

@adchia
Copy link
Collaborator

@adchia adchia commented Mar 17, 2022 •
edited
Loading

What this PR does / why we need it:
This PR does several things:

  • Prints out more warnings on Feast 0.21 needing users to specify unique names or tables for data sources.
  • Converting exceptions to warnings if there are multiple duplicate data source names.
  • Adding the data_source_class_type to the proto so repeated feast apply runs will work on Spark (and other contrib data sources)
  • Disabling feast apply/plan output for data source changes until Feast 0.21
  • Adding tests for this behavior
  • Removing unnecessary logic in DataSource#from_proto. When we apply data sources (incl in feature views), we populate the class type, so it should always fall through in the first if condition.

Which issue(s) this PR fixes:

Fixes #2407 #2409

@adchia adchia requested a review from a team as a code owner March 17, 2022 01:18
@adchia adchia requested review from achals and removed request for a team March 17, 2022 01:18
@adchia adchia mentioned this pull request Mar 17, 2022
Closed
@codecov-commenter
Copy link

codecov-commenter commented Mar 17, 2022 •
edited
Loading

Codecov Report

Merging #2415 (0fe2df8) into master (5acd9c4) will decrease coverage by 26.44%.
The diff coverage is 76.47%.

@@             Coverage Diff             @@
##           master    #2415       +/-   ##
===========================================
- Coverage   84.93%   58.49%   -26.45%     
===========================================
  Files         127      126        -1     
  Lines       10841    10798       -43     
===========================================
- Hits         9208     6316     -2892     
- Misses       1633     4482     +2849
Flag Coverage Δ
integrationtests ?
unittests 58.49% <76.47%> (+0.04%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
...thon/feast/infra/offline_stores/bigquery_source.py 62.93% <ø> (-24.14%) ⬇️
sdk/python/feast/repo_operations.py 55.05% <ø> (ø)
sdk/python/feast/cli.py 38.20% <33.33%> (-0.05%) ⬇️
sdk/python/feast/base_feature_view.py 59.77% <50.00%> (-20.23%) ⬇️
sdk/python/feast/entity.py 91.66% <50.00%> (-1.44%) ⬇️
sdk/python/feast/feature_store.py 70.03% <60.00%> (-22.04%) ⬇️
sdk/python/feast/diff/registry_diff.py 70.37% <80.00%> (-0.63%) ⬇️
sdk/python/feast/data_source.py 66.35% <100.00%> (-0.63%) ⬇️
sdk/python/feast/go_server.py 31.75% <100.00%> (ø)
...thon/feast/infra/offline_stores/redshift_source.py 57.75% <100.00%> (-29.32%) ⬇️
... and 77 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5acd9c4...0fe2df8. Read the comment docs.

@adchia adchia mentioned this pull request Mar 21, 2022
Merged
achals
achals reviewed Mar 21, 2022
View reviewed changes
woop
woop reviewed Mar 21, 2022
View reviewed changes
adchia added 10 commits March 29, 2022 12:03
@adchia
fix: Print more warning statements on requirement for data sources to…
2db0c9a 
… have name in future, but don't prevent apply from running if there are duplicate empty data sources. Also attach class type when applying data sources so repeated feast apply commands properly work for Spark

Signed-off-by: Danny Chiao <[email protected]>
@adchia
typo
2d4fc84 
Signed-off-by: Danny Chiao <[email protected]>
@adchia
typo
88a7434 
Signed-off-by: Danny Chiao <[email protected]>
@adchia
fix
ba97190 
Signed-off-by: Danny Chiao <[email protected]>
@adchia
fix
cdee46c 
Signed-off-by: Danny Chiao <[email protected]>
@adchia
fix
3a4b4cd 
Signed-off-by: Danny Chiao <[email protected]>
@adchia
More tests
7add51a 
Signed-off-by: Danny Chiao <[email protected]>
@adchia
fix
bc9d3ae 
Signed-off-by: Danny Chiao <[email protected]>
@adchia
fix
9d2c5f1 
Signed-off-by: Danny Chiao <[email protected]>
@adchia
fix
bae2e37 
Signed-off-by: Danny Chiao <[email protected]>
adchia added 3 commits March 29, 2022 12:14
@adchia
fix
41ff4d9 
Signed-off-by: Danny Chiao <[email protected]>
@adchia
revert
70d4b8d 
Signed-off-by: Danny Chiao <[email protected]>
@adchia
fix
e17973f 
Signed-off-by: Danny Chiao <[email protected]>
achals
achals reviewed Mar 29, 2022
View reviewed changes
adchia added 2 commits March 29, 2022 14:30
@adchia
fix
0fe2df8 
Signed-off-by: Danny Chiao <[email protected]>
@adchia
fix
f854de4 
Signed-off-by: Danny Chiao <[email protected]>
achals
achals approved these changes Mar 29, 2022
View reviewed changes
Copy link
Member

@achals achals left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@feast-ci-bot
Copy link
Collaborator

feast-ci-bot commented Mar 29, 2022

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: achals, adchia

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@feast-ci-bot feast-ci-bot merged commit b95f441 into feast-dev:master Mar 29, 2022
@adchia adchia deleted the fixApply branch March 29, 2022 21:12
jbvaningen pushed a commit to jbvaningen/feast that referenced this pull request Mar 30, 2022
@adchia
fix: Don't prevent apply from running given duplicate empty names in …
c222364 
…data sources. Also fix repeated apply of Spark data source. (feast-dev#2415)

* fix: Print more warning statements on requirement for data sources to have name in future, but don't prevent apply from running if there are duplicate empty data sources. Also attach class type when applying data sources so repeated feast apply commands properly work for Spark

Signed-off-by: Danny Chiao <[email protected]>

* typo

Signed-off-by: Danny Chiao <[email protected]>

* typo

Signed-off-by: Danny Chiao <[email protected]>

* fix

Signed-off-by: Danny Chiao <[email protected]>

* fix

Signed-off-by: Danny Chiao <[email protected]>

* fix

Signed-off-by: Danny Chiao <[email protected]>

* More tests

Signed-off-by: Danny Chiao <[email protected]>

* fix

Signed-off-by: Danny Chiao <[email protected]>

* fix

Signed-off-by: Danny Chiao <[email protected]>

* fix

Signed-off-by: Danny Chiao <[email protected]>

* fix

Signed-off-by: Danny Chiao <[email protected]>

* revert

Signed-off-by: Danny Chiao <[email protected]>

* fix

Signed-off-by: Danny Chiao <[email protected]>

* fix

Signed-off-by: Danny Chiao <[email protected]>

* fix

Signed-off-by: Danny Chiao <[email protected]>
Signed-off-by: joostvan <[email protected]>
felixwang9817 pushed a commit that referenced this pull request Apr 6, 2022
@adchia @felixwang9817
fix: Don't prevent apply from running given duplicate empty names in …
9baba23 
…data sources. Also fix repeated apply of Spark data source. (#2415)

* fix: Print more warning statements on requirement for data sources to have name in future, but don't prevent apply from running if there are duplicate empty data sources. Also attach class type when applying data sources so repeated feast apply commands properly work for Spark

Signed-off-by: Danny Chiao <[email protected]>

* typo

Signed-off-by: Danny Chiao <[email protected]>

* typo

Signed-off-by: Danny Chiao <[email protected]>

* fix

Signed-off-by: Danny Chiao <[email protected]>

* fix

Signed-off-by: Danny Chiao <[email protected]>

* fix

Signed-off-by: Danny Chiao <[email protected]>

* More tests

Signed-off-by: Danny Chiao <[email protected]>

* fix

Signed-off-by: Danny Chiao <[email protected]>

* fix

Signed-off-by: Danny Chiao <[email protected]>

* fix

Signed-off-by: Danny Chiao <[email protected]>

* fix

Signed-off-by: Danny Chiao <[email protected]>

* revert

Signed-off-by: Danny Chiao <[email protected]>

* fix

Signed-off-by: Danny Chiao <[email protected]>

* fix

Signed-off-by: Danny Chiao <[email protected]>

* fix

Signed-off-by: Danny Chiao <[email protected]>
felixwang9817 pushed a commit that referenced this pull request Apr 6, 2022
@feast-ci-bot
chore(release): release 0.19.4
5659439 
## [0.19.4](v0.19.3...v0.19.4) (2022-04-06)

### Bug Fixes

* Add spark to lambda dockerfile ([#2480](#2480)) ([f7911b7](f7911b7))
* Don't prevent apply from running given duplicate empty names in data sources. Also fix repeated apply of Spark data source. ([#2415](#2415)) ([9baba23](9baba23))
* Fix DataSource constructor to unbreak custom data sources ([#2492](#2492)) ([597c543](597c543))
felixwang9817 pushed a commit that referenced this pull request Apr 6, 2022
@adchia @felixwang9817
fix: Don't prevent apply from running given duplicate empty names in …
88e01a2 
…data sources. Also fix repeated apply of Spark data source. (#2415)

* fix: Print more warning statements on requirement for data sources to have name in future, but don't prevent apply from running if there are duplicate empty data sources. Also attach class type when applying data sources so repeated feast apply commands properly work for Spark

Signed-off-by: Danny Chiao <[email protected]>

* typo

Signed-off-by: Danny Chiao <[email protected]>

* typo

Signed-off-by: Danny Chiao <[email protected]>

* fix

Signed-off-by: Danny Chiao <[email protected]>

* fix

Signed-off-by: Danny Chiao <[email protected]>

* fix

Signed-off-by: Danny Chiao <[email protected]>

* More tests

Signed-off-by: Danny Chiao <[email protected]>

* fix

Signed-off-by: Danny Chiao <[email protected]>

* fix

Signed-off-by: Danny Chiao <[email protected]>

* fix

Signed-off-by: Danny Chiao <[email protected]>

* fix

Signed-off-by: Danny Chiao <[email protected]>

* revert

Signed-off-by: Danny Chiao <[email protected]>

* fix

Signed-off-by: Danny Chiao <[email protected]>

* fix

Signed-off-by: Danny Chiao <[email protected]>

* fix

Signed-off-by: Danny Chiao <[email protected]>
felixwang9817 pushed a commit that referenced this pull request Apr 6, 2022
@feast-ci-bot
chore(release): release 0.19.4
20e7a6f 
## [0.19.4](v0.19.3...v0.19.4) (2022-04-06)

### Bug Fixes

* Add spark to lambda dockerfile ([#2480](#2480)) ([ba22c28](ba22c28))
* Don't prevent apply from running given duplicate empty names in data sources. Also fix repeated apply of Spark data source. ([#2415](#2415)) ([88e01a2](88e01a2))
* Fix DataSource constructor to unbreak custom data sources ([#2492](#2492)) ([2115bd0](2115bd0))
achals pushed a commit that referenced this pull request Apr 14, 2022
@feast-ci-bot
chore(release): release 0.20.0
b2d2cfd 
# [0.20.0](v0.19.0...v0.20.0) (2022-04-14)

### Bug Fixes

* Add inlined data sources to the top level registry ([#2456](#2456)) ([356788a](356788a))
* Add new value types to types.ts for web ui ([#2463](#2463)) ([ad5694e](ad5694e))
* Add PushSource proto and Python class ([#2428](#2428)) ([9a4bd63](9a4bd63))
* Add spark to lambda dockerfile ([#2480](#2480)) ([514666f](514666f))
* Added private_key auth for Snowflake ([#2508](#2508)) ([c42c9b0](c42c9b0))
* Added Redshift and Spark typecheck to data_source event_timestamp_col inference ([#2389](#2389)) ([04dea73](04dea73))
* Building of go extension fails ([#2448](#2448)) ([7d1efd5](7d1efd5))
* Bump the number of versions bumps expected to 27 ([#2549](#2549)) ([ecc9938](ecc9938))
* Create __init__ files for the proto-generated python dirs ([#2410](#2410)) ([e17028d](e17028d))
* Don't prevent apply from running given duplicate empty names in data sources. Also fix repeated apply of Spark data source. ([#2415](#2415)) ([b95f441](b95f441))
* Dynamodb deduplicate batch write request by partition keys ([#2515](#2515)) ([70d4a13](70d4a13))
* Ensure that __init__ files exist in proto dirs ([#2433](#2433)) ([9b94f7b](9b94f7b))
* Fix DataSource constructor to unbreak custom data sources ([#2492](#2492)) ([712653e](712653e))
* Fix default feast apply path without any extras ([#2373](#2373)) ([6ba7fc7](6ba7fc7))
* Fix definitions.py with new definition ([#2541](#2541)) ([eefc34a](eefc34a))
* Fix entity row to use join key instead of name ([#2521](#2521)) ([c22fa2c](c22fa2c))
* Fix Java Master ([#2499](#2499)) ([e083458](e083458))
* Fix registry proto ([#2435](#2435)) ([ea6a9b2](ea6a9b2))
* Fix some inconsistencies in the docs and comments in the code ([#2444](#2444)) ([ad008bf](ad008bf))
* Fix spark docs ([#2382](#2382)) ([d4a606a](d4a606a))
* Fix Spark template to work correctly on feast init -t spark ([#2393](#2393)) ([ae133fd](ae133fd))
* Fix the feature repo fixture used by java tests  ([#2469](#2469)) ([32e925e](32e925e))
* Fix unhashable Snowflake and Redshift sources ([cd8f1c9](cd8f1c9))
* Fixed bug in passing config file params to snowflake python connector ([#2503](#2503)) ([34f2b59](34f2b59))
* Fixing Spark template to include source name ([#2381](#2381)) ([a985f1d](a985f1d))
* Make name a keyword arg for the Entity class ([#2467](#2467)) ([43847de](43847de))
* Making a name for data sources not a breaking change ([#2379](#2379)) ([71d7ae2](71d7ae2))
* Minor link fix in `CONTRIBUTING.md` ([#2481](#2481)) ([2917e27](2917e27))
* Preserve ordering of features in _get_column_names ([#2457](#2457)) ([495b435](495b435))
* Relax click python requirement to >=7 ([#2450](#2450)) ([f202f92](f202f92))
* Remove date partition column field from datasources that don't s… ([#2478](#2478)) ([ce35835](ce35835))
* Remove docker step from unit test workflow ([#2535](#2535)) ([6f22f22](6f22f22))
* Remove spark from the AWS Lambda dockerfile ([#2498](#2498)) ([6abae16](6abae16))
* Request data api update ([#2488](#2488)) ([0c9e5b7](0c9e5b7))
* Schema update ([#2509](#2509)) ([cf7bbc2](cf7bbc2))
* Simplify DataSource.from_proto logic ([#2424](#2424)) ([6bda4d2](6bda4d2))
* Snowflake api update ([#2487](#2487)) ([1181a9e](1181a9e))
* Support passing batch source to streaming sources for backfills ([#2523](#2523)) ([90db1d1](90db1d1))
* Timestamp update ([#2486](#2486)) ([bf23111](bf23111))
* Typos in Feast UI error message ([#2432](#2432)) ([e14369d](e14369d))
* Update feature view APIs to prefer keyword args ([#2472](#2472)) ([7c19cf7](7c19cf7))
* Update file api ([#2470](#2470)) ([83a11c6](83a11c6))
* Update Makefile to cd into python dir before running commands ([#2437](#2437)) ([ca32155](ca32155))
* Update redshift api ([#2479](#2479)) ([4fa73a9](4fa73a9))
* Update some fields optional in UI parser ([#2380](#2380)) ([cff7ac3](cff7ac3))
* Use a single version of jackson libraries and upgrade to 2.12.6.1 ([#2473](#2473)) ([5be1cc6](5be1cc6))
* Use dateutil parser to parse materialization times ([#2464](#2464)) ([6c55e49](6c55e49))
* Use the correct dockerhub image tag when building feature servers ([#2372](#2372)) ([0d62c1d](0d62c1d))

### Features

* Add `/write-to-online-store` method to the python feature server ([#2423](#2423)) ([d2fb048](d2fb048))
* Add description, tags, owner fields to all feature view classes ([#2440](#2440)) ([ed5e928](ed5e928))
* Add DQM Logging on GRPC Server with FileLogStorage for Testing ([#2403](#2403)) ([57a97d8](57a97d8))
* Add Feast types in preparation for changing type system ([#2475](#2475)) ([4864252](4864252))
* Add Field class ([#2500](#2500)) ([1279612](1279612))
* Add support for DynamoDB online_read in batches ([#2371](#2371)) ([702ec49](702ec49))
* Add Support for DynamodbOnlineStoreConfig endpoint_url parameter ([#2485](#2485)) ([7b863d1](7b863d1))
* Add templating for dynamodb table name ([#2394](#2394)) ([f591088](f591088))
* Allow local feature server to use Go feature server if enabled ([#2538](#2538)) ([a2ef375](a2ef375))
* Allow using entity's join_key in get_online_features ([#2420](#2420)) ([068c765](068c765))
* Data Source Api Update ([#2468](#2468)) ([6b96b21](6b96b21))
* Go server ([#2339](#2339)) ([d12e7ef](d12e7ef)), closes [#2354](#2354) [#2361](#2361) [#2332](#2332) [#2356](#2356) [#2363](#2363) [#2349](#2349) [#2355](#2355) [#2336](#2336) [#2361](#2361) [#2363](#2363) [#2344](#2344) [#2354](#2354) [#2347](#2347) [#2350](#2350) [#2356](#2356) [#2355](#2355) [#2349](#2349) [#2352](#2352) [#2341](#2341) [#2336](#2336) [#2373](#2373) [#2315](#2315) [#2372](#2372) [#2332](#2332) [#2349](#2349) [#2336](#2336) [#2361](#2361) [#2363](#2363) [#2344](#2344) [#2354](#2354) [#2347](#2347) [#2350](#2350) [#2356](#2356) [#2355](#2355) [#2349](#2349) [#2352](#2352) [#2341](#2341) [#2336](#2336) [#2373](#2373) [#2379](#2379) [#2380](#2380) [#2382](#2382) [#2364](#2364) [#2366](#2366) [#2386](#2386)
* Graduate write_to_online_store out of experimental status ([#2426](#2426)) ([e7dd4b7](e7dd4b7))
* Make feast PEP 561 compliant ([#2405](#2405)) ([3c41f94](3c41f94)), closes [#2420](#2420) [#2418](#2418) [#2425](#2425) [#2426](#2426) [#2427](#2427) [#2431](#2431) [#2433](#2433) [#2420](#2420) [#2418](#2418) [#2425](#2425) [#2426](#2426) [#2427](#2427) [#2431](#2431) [#2433](#2433)
* Makefile for contrib for Issue [#2364](#2364) ([#2366](#2366)) ([a02325b](a02325b))
* Support on demand feature views in go feature server ([#2494](#2494)) ([6edd274](6edd274))
* Switch from Feature to Field ([#2514](#2514)) ([6a03bed](6a03bed))
* Use a daemon thread to monitor the go feature server exclusively ([#2391](#2391)) ([0bb5e8c](0bb5e8c))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@woop woop woop left review comments

@kevjumba kevjumba kevjumba left review comments

@felixwang9817 felixwang9817 felixwang9817 left review comments

@achals achals achals approved these changes

Assignees

@achals achals

@tsotnet tsotnet

Labels

approved lgtm ok-to-test size/L

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

feast apply fails for Spark Offline Store once the registry has been created

9 participants

@adchia @codecov-commenter @feast-ci-bot @achals @woop @kevjumba @felixwang9817 @pyalex @tsotnet