feat: Offline Store historical features retrieval based on datetime range for spark by aniketpalu · Pull Request #5720 · feast-dev/feast

aniketpalu · 2025-11-13T08:57:32Z

What this PR does / why we need it:

Add support for entity_df=None in SparkOfflineStore.get_historical_features with start_date/end_date.
- Builds a derived entity set by UNION DISTINCT of entity columns from each FeatureView within the window (uses date_partition_column when present), casts entities to STRING, and uses a literal as-of timestamp for PIT joins.
- Why: Enables simple time-window backfills without requiring an entity_df.
Type fixes in spark.py: pass KeysView[str] for entity_df_columns.
- Why: Satisfy mypy and keep PIT join builder signature consistent.
spark_source.py: ensure a SparkSession exists in remote threads.
- Why: Prevent remote mode failures due to missing active session.
arrow_error_handler.py: re-raise non-Feast exceptions on the server.
- Why: Ensure Arrow Flight returns actionable errors instead of None.
Test: add unit test for non-entity retrieval (Spark session mocked).

Which issue(s) this PR fixes:

RHOAIENG-38644

Misc

…ange for spark Signed-off-by: Aniket Paluskar <[email protected]>

jyejare

Reviewewd very high level, looks Good, but could you share the run logs ?

Signed-off-by: Aniket Paluskar <[email protected]>

aniketpalu · 2025-11-25T08:45:41Z

Reviewewd very high level, looks Good, but could you share the run logs ?

Feature Repo used: https://github.com/aniketpalu/feast_examples/tree/main/feature_repo_spark
Testing script: https://github.com/aniketpalu/feast_examples/blob/2d5cef285996f59818ee4d19c595c0fb2efed4da/test_historical_retrieval.py

Attempted to retrieve historical features using entity_df:

📁 Data in Parquet file: 100 records
📅 Timestamp range: 2025-10-13 00:31:16.754605 to 2025-11-12 00:31:16.754605
👥 User IDs range: 1 to 100

📊 Entity data for feature retrieval:
user_id event_timestamp
0 1 2025-11-11 01:31:16.754605
1 2 2025-10-20 01:31:16.754605
2 3 2025-11-06 01:31:16.754605

🔍 Retrieving features for 3 entities...
/Users/apaluska/Work/feast/sdk/python/feast/infra/offline_stores/contrib/spark_offline_store/spark_source.py:96: RuntimeWarning: The spark data source API is an experimental feature in alpha development. This API is unstable and it could and most probably will be changed in the future.
warnings.warn(
/Users/apaluska/Work/feast/sdk/python/feast/infra/offline_stores/contrib/spark_offline_store/spark.py:167: RuntimeWarning: The spark offline store is an experimental feature in alpha development. Some functionality may still be unstable so functionality can change in the future.
warnings.warn(
WARNING: Using incubator modules: jdk.incubator.vector
Using Spark's default log4j profile: org/apache/spark/log4j2-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
25/11/25 12:57:14 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

✅ Historical features retrieved:
📊 Result shape: (3, 4)

📋 Results:
user_id event_timestamp avg_transaction_amount total_transactions
0 1 2025-11-11 07:01:16.754605 773.62 60
1 2 2025-10-20 07:01:16.754605 84.12 369
2 3 2025-11-06 07:01:16.754605 472.25 382

(Failed) Attempt to retrieve historical features using start_date & end_date before implementing:

📁 Data in Parquet file: 100 records
📅 Data timestamp range: 2025-10-13 00:31:16.754605 to 2025-11-12 00:31:16.754605
📅 Query date range: 2025-10-23 00:31:16.754605 to 2025-11-12 00:31:16.754605

🔍 Retrieving features for date range (no entity_df)...

Users/apaluska/Work/feast/sdk/python/feast/infra/offline_stores/contrib/spark_offline_store/spark_source.py:96: RuntimeWarning: The spark data source API is an experimental feature in alpha development. This API is unstable and it could and most probably will be changed in the future.
warnings.warn(
❌ Error in date range-based retrieval: SparkOfflineStore.get_historical_features() got an unexpected keyword argument 'start_date'
Traceback (most recent call last):
File "/Users/apaluska/Work/EAP Demo/feast_example_minimal/test_historical_retrieval.py", line 159, in
historical_features = fs.get_historical_features(
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/apaluska/Work/feast/sdk/python/feast/feature_store.py", line 1218, in get_historical_features
job = provider.get_historical_features(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/apaluska/Work/feast/sdk/python/feast/infra/passthrough_provider.py", line 469, in get_historical_features
job = self.offline_store.get_historical_features(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: SparkOfflineStore.get_historical_features() got an unexpected keyword argument 'start_date'

(Successful) Attempt to retrieve historical features using start_date & end_date:

📁 Data in Parquet file: 100 records
📅 Data timestamp range: 2025-10-13 00:31:16.754605 to 2025-11-12 00:31:16.754605
📅 Query date range: 2025-10-23 00:31:16.754605 to 2025-11-12 00:31:16.754605

🔍 Retrieving features for date range (no entity_df)...
/Users/apaluska/Work/feast/sdk/python/feast/infra/offline_stores/contrib/spark_offline_store/spark_source.py:96: RuntimeWarning: The spark data source API is an experimental feature in alpha development. This API is unstable and it could and most probably will be changed in the future.
warnings.warn(
/Users/apaluska/Work/feast/sdk/python/feast/infra/offline_stores/contrib/spark_offline_store/spark.py:169: RuntimeWarning: The spark offline store is an experimental feature in alpha development. Some functionality may still be unstable so functionality can change in the future.
warnings.warn(

✅ Historical features retrieved:
📊 Result shape: (66, 4)

📋 First 30 rows (showing user_id entity_ts 0 25 2025-11-12 00:31:16.754605 1 7 2025-11-12 00:31:16.754605 2 77 2025-11-12 00:31:16.754605 3 3 2025-11-12 00:31:16.754605 4 57 2025-11-12 00:31:16.754605 5 45 2025-11-12 00:31:16.754605 6 100 2025-11-12 00:31:16.754605 7 35 2025-11-12 00:31:16.754605 8 96 2025-11-12 00:31:16.754605 9 19 2025-11-12 00:31:16.754605 10 29 2025-11-12 00:31:16.754605 11 22 2025-11-12 00:31:16.754605 12 97 2025-11-12 00:31:16.754605 13 81 2025-11-12 00:31:16.754605 14 31 2025-11-12 00:31:16.754605 15 83 2025-11-12 00:31:16.754605 16 40 2025-11-12 00:31:16.754605 17 82 2025-11-12 00:31:16.754605 18 24 2025-11-12 00:31:16.754605 19 73 2025-11-12 00:31:16.754605 20 74 2025-11-12 00:31:16.754605 21 89 2025-11-12 00:31:16.754605 22 95 2025-11-12 00:31:16.754605 23 48 2025-11-12 00:31:16.754605 24 1 2025-11-12 00:31:16.754605 25 50 2025-11-12 00:31:16.754605 26 59 2025-11-12 00:31:16.754605 27 86 2025-11-12 00:31:16.754605 28 21 2025-11-12 00:31:16.754605 29 27 2025-11-12 00:31:16.754605 both event_timestamp and created_timestamp):
avg_transaction_amount total_transactions
NaN NaN
NaN NaN
494.04 127.0
472.25 382.0
NaN NaN
163.35 405.0
NaN NaN
51.20 113.0
NaN NaN
917.33 136.0
NaN NaN
NaN NaN
NaN NaN
NaN NaN
NaN NaN
315.60 313.0
305.82 267.0
NaN NaN
811.43 88.0
NaN NaN
NaN NaN
NaN NaN
65.17 464.0
NaN NaN
773.62 60.0
NaN NaN
921.00 426.0
NaN NaN
376.88 99.0
691.39 447.0

💡 Explanation:

event_timestamp: Point-in-time when features are queried (query timestamp)

created_timestamp_in_source: When the feature was actually created/updated in source data

When using start_date/end_date without entity_df, Feast uses end_date as event_timestamp

This is correct: 'What were the features as of 2025-11-12 00:31:16.754605?

jyejare · 2025-12-02T14:56:08Z

@aniketpalu Did we validate the join works from multiple feature views in a feature service ?

aniketpalu · 2025-12-09T11:03:54Z

sdk/python/tests/unit/infra/offline_stores/contrib/spark_offline_store/test_spark.py

+    retrieval_job = SparkOfflineStore.get_historical_features(
+        config=test_repo_config,
+        feature_views=[test_feature_view2, test_feature_view1],
+        feature_refs=["test_feature_view2:feature2", "test_feature_view1:feature1"],


@aniketpalu Did we validate the join works from multiple feature views in a feature service ?

@jyejare
We have tested this scenario and even added it in test cases.

# [0.58.0](v0.57.0...v0.58.0) (2025-12-16) ### Bug Fixes * Add java proto ([#5719](#5719)) ([fc3ea20](fc3ea20)) * Add possibility to force full features names for materialize ops ([#5728](#5728)) ([55c9c36](55c9c36)) * Fixed file registry cache sync ([09505d4](09505d4)) * Handle hyphon in sqlite project name ([#5575](#5575)) ([#5749](#5749)) ([b8346ff](b8346ff)) * Pinned substrait to fix protobuf issue ([d0ef4da](d0ef4da)) * Set TLS certificate annotation only on gRPC service ([#5715](#5715)) ([75d13db](75d13db)) * SQLite online store deletes tables from other projects in shared registry scenarios ([#5766](#5766)) ([fabce76](fabce76)) * Validate not existing entity join keys for preventing panic ([0b93559](0b93559)) ### Features * Add annotations for pod templates ([534e647](534e647)) * Add Pytorch template ([#5780](#5780)) ([6afd353](6afd353)) * Add support for extra options for stream source ([#5618](#5618)) ([18956c2](18956c2)) * Added matched_tag field search api results with fuzzy search capabilities ([#5769](#5769)) ([4a9ffae](4a9ffae)) * Added support for enabling metrics in Feast Operator ([#5317](#5317)) ([#5748](#5748)) ([a8498c2](a8498c2)) * Configure CacheTTLSecondscache,CacheMode for file-based registry in Feast Operator([#5708](#5708)) ([#5744](#5744)) ([f25f83b](f25f83b)) * Implemented Tiling Support for Time-Windowed Aggregations ([#5724](#5724)) ([7a99166](7a99166)) * Offline Store historical features retrieval based on datetime range for spark ([#5720](#5720)) ([27ec8ec](27ec8ec)) * Offline Store historical features retrieval based on datetime range in dask ([#5717](#5717)) ([a16582a](a16582a)) * Production ready feast operator with v1 apiversion ([#5771](#5771)) ([49359c6](49359c6)) * Support for Map value data type ([#5768](#5768)) ([#5772](#5772)) ([b99a8a9](b99a8a9))

# [0.58.0](feast-dev/feast@v0.57.0...v0.58.0) (2025-12-16) ### Bug Fixes * Add java proto ([feast-dev#5719](feast-dev#5719)) ([fc3ea20](feast-dev@fc3ea20)) * Add possibility to force full features names for materialize ops ([feast-dev#5728](feast-dev#5728)) ([55c9c36](feast-dev@55c9c36)) * Fixed file registry cache sync ([09505d4](feast-dev@09505d4)) * Handle hyphon in sqlite project name ([feast-dev#5575](feast-dev#5575)) ([feast-dev#5749](feast-dev#5749)) ([b8346ff](feast-dev@b8346ff)) * Pinned substrait to fix protobuf issue ([d0ef4da](feast-dev@d0ef4da)) * Set TLS certificate annotation only on gRPC service ([feast-dev#5715](feast-dev#5715)) ([75d13db](feast-dev@75d13db)) * SQLite online store deletes tables from other projects in shared registry scenarios ([feast-dev#5766](feast-dev#5766)) ([fabce76](feast-dev@fabce76)) * Validate not existing entity join keys for preventing panic ([0b93559](feast-dev@0b93559)) ### Features * Add annotations for pod templates ([534e647](feast-dev@534e647)) * Add Pytorch template ([feast-dev#5780](feast-dev#5780)) ([6afd353](feast-dev@6afd353)) * Add support for extra options for stream source ([feast-dev#5618](feast-dev#5618)) ([18956c2](feast-dev@18956c2)) * Added matched_tag field search api results with fuzzy search capabilities ([feast-dev#5769](feast-dev#5769)) ([4a9ffae](feast-dev@4a9ffae)) * Added support for enabling metrics in Feast Operator ([feast-dev#5317](feast-dev#5317)) ([feast-dev#5748](feast-dev#5748)) ([a8498c2](feast-dev@a8498c2)) * Configure CacheTTLSecondscache,CacheMode for file-based registry in Feast Operator([feast-dev#5708](feast-dev#5708)) ([feast-dev#5744](feast-dev#5744)) ([f25f83b](feast-dev@f25f83b)) * Implemented Tiling Support for Time-Windowed Aggregations ([feast-dev#5724](feast-dev#5724)) ([7a99166](feast-dev@7a99166)) * Offline Store historical features retrieval based on datetime range for spark ([feast-dev#5720](feast-dev#5720)) ([27ec8ec](feast-dev@27ec8ec)) * Offline Store historical features retrieval based on datetime range in dask ([feast-dev#5717](feast-dev#5717)) ([a16582a](feast-dev@a16582a)) * Production ready feast operator with v1 apiversion ([feast-dev#5771](feast-dev#5771)) ([49359c6](feast-dev@49359c6)) * Support for Map value data type ([feast-dev#5768](feast-dev#5768)) ([feast-dev#5772](feast-dev#5772)) ([b99a8a9](feast-dev@b99a8a9)) Signed-off-by: Anthonette Adanyin <[email protected]>

…ange for spark (#5720) * feat: Offline Store historical features retrieval based on datetime range for spark Signed-off-by: Aniket Paluskar <[email protected]> * Restructured code, extended existing test cases Signed-off-by: Aniket Paluskar <[email protected]> * Fixed lint issues Signed-off-by: Aniket Paluskar <[email protected]> --------- Signed-off-by: Aniket Paluskar <[email protected]> Signed-off-by: Francisco Javier Arceo <[email protected]>

# [0.58.0](v0.57.0...v0.58.0) (2025-12-16) ### Bug Fixes * Add java proto ([#5719](#5719)) ([fc3ea20](fc3ea20)) * Add possibility to force full features names for materialize ops ([#5728](#5728)) ([55c9c36](55c9c36)) * Fixed file registry cache sync ([09505d4](09505d4)) * Handle hyphon in sqlite project name ([#5575](#5575)) ([#5749](#5749)) ([b8346ff](b8346ff)) * Pinned substrait to fix protobuf issue ([d0ef4da](d0ef4da)) * Set TLS certificate annotation only on gRPC service ([#5715](#5715)) ([75d13db](75d13db)) * SQLite online store deletes tables from other projects in shared registry scenarios ([#5766](#5766)) ([fabce76](fabce76)) * Validate not existing entity join keys for preventing panic ([0b93559](0b93559)) ### Features * Add annotations for pod templates ([534e647](534e647)) * Add Pytorch template ([#5780](#5780)) ([6afd353](6afd353)) * Add support for extra options for stream source ([#5618](#5618)) ([18956c2](18956c2)) * Added matched_tag field search api results with fuzzy search capabilities ([#5769](#5769)) ([4a9ffae](4a9ffae)) * Added support for enabling metrics in Feast Operator ([#5317](#5317)) ([#5748](#5748)) ([a8498c2](a8498c2)) * Configure CacheTTLSecondscache,CacheMode for file-based registry in Feast Operator([#5708](#5708)) ([#5744](#5744)) ([f25f83b](f25f83b)) * Implemented Tiling Support for Time-Windowed Aggregations ([#5724](#5724)) ([7a99166](7a99166)) * Offline Store historical features retrieval based on datetime range for spark ([#5720](#5720)) ([27ec8ec](27ec8ec)) * Offline Store historical features retrieval based on datetime range in dask ([#5717](#5717)) ([a16582a](a16582a)) * Production ready feast operator with v1 apiversion ([#5771](#5771)) ([49359c6](49359c6)) * Support for Map value data type ([#5768](#5768)) ([#5772](#5772)) ([b99a8a9](b99a8a9)) Signed-off-by: Francisco Javier Arceo <[email protected]>

feat: Offline Store historical features retrieval based on datetime r…

6ae5ada

…ange for spark Signed-off-by: Aniket Paluskar <[email protected]>

aniketpalu requested a review from a team as a code owner November 13, 2025 08:57

ntkathole added the ok-to-test label Nov 13, 2025

jyejare reviewed Nov 14, 2025

View reviewed changes

aniketpalu and others added 4 commits November 18, 2025 13:10

Merge branch 'master' into RHOAIENG-38644

759f953

Merge branch 'master' into RHOAIENG-38644

adef6e6

Restructured code, extended existing test cases

4fbf122

Signed-off-by: Aniket Paluskar <[email protected]>

Fixed lint issues

f39ece8

Signed-off-by: Aniket Paluskar <[email protected]>

aniketpalu requested a review from jyejare November 25, 2025 07:22

Merge branch 'master' into RHOAIENG-38644

d7ea5db

aniketpalu added 2 commits December 8, 2025 17:13

Merge branch 'master' into RHOAIENG-38644

964ac47

Merge branch 'master' into RHOAIENG-38644

444698f

aniketpalu commented Dec 9, 2025

View reviewed changes

aniketpalu added 2 commits December 12, 2025 15:29

Merge branch 'master' into RHOAIENG-38644

dc2cbcf

Merge branch 'master' into RHOAIENG-38644

9634c23

jyejare approved these changes Dec 15, 2025

View reviewed changes

ntkathole approved these changes Dec 15, 2025

View reviewed changes

ntkathole merged commit 27ec8ec into feast-dev:master Dec 15, 2025
17 checks passed

jyejare mentioned this pull request Jan 13, 2026

Spark - historical retrieval without entity dataframe #5839

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Offline Store historical features retrieval based on datetime range for spark#5720

feat: Offline Store historical features retrieval based on datetime range for spark#5720
ntkathole merged 10 commits intofeast-dev:masterfrom
aniketpalu:RHOAIENG-38644

aniketpalu commented Nov 13, 2025 •

edited

Loading

Uh oh!

jyejare left a comment •

edited

Loading

Uh oh!

aniketpalu commented Nov 25, 2025 •

edited

Loading

Uh oh!

jyejare commented Dec 2, 2025

Uh oh!

aniketpalu Dec 9, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

aniketpalu commented Nov 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this PR does / why we need it:

Which issue(s) this PR fixes:

Misc

Uh oh!

jyejare left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aniketpalu commented Nov 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jyejare commented Dec 2, 2025

Uh oh!

aniketpalu Dec 9, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

aniketpalu commented Nov 13, 2025 •

edited

Loading

jyejare left a comment •

edited

Loading

aniketpalu commented Nov 25, 2025 •

edited

Loading