Skip to content

feat: Adding Milvus demo to examples#4910

Merged
franciscojavierarceo merged 17 commits intomasterfrom
milvus-demo
Jan 30, 2025
Merged

feat: Adding Milvus demo to examples#4910
franciscojavierarceo merged 17 commits intomasterfrom
milvus-demo

Conversation

@franciscojavierarceo
Copy link
Member

@franciscojavierarceo franciscojavierarceo commented Jan 10, 2025 •
edited
Loading

What this PR does / why we need it:

This pull request introduces several updates and new features to the Feast repository, particularly around the integration of Milvus as an online store. Below is a summary of the key changes:

Documentation Updates

  • Updated Overview of Online Stores: Added Milvus to the matrix of supported functionalities in the overview.md file under docs/reference/online-stores/.

New Example Files

  • RAG Example: Introduced a new example for RAG (Retrieval-Augmented Generation) under examples/rag/.
    • Added __init__.py.
    • Added README.md.
    • Added feature_repo/ directory containing:
      • __init__.py
      • example_repo.py
      • feature_store.yaml
      • test_workflow.py
    • Added milvus-quickstart.ipynb notebook demo containing documentation for data scientists to easily get started

Code Updates

  • Milvus Online Store Integration:
    • Renamed _get_collection method to _get_or_create_collection in milvus.py.
    • Updated method calls to use _get_or_create_collection instead of _get_collection.
    • Added logic to handle vector fields and search metrics.
    • Improved handling of Proto values and serialization in retrieve_online_documents_v2 method.

Detailed Changes

  1. Documentation:

    • Added Milvus to the list of supported online stores in the feature matrix.

  2. Examples:

    • Added new example files under examples/rag/.

  3. Milvus Integration:

    • Method Renaming: Renamed _get_collection to _get_or_create_collection.
    • Vector Field Handling: Added handling for vector fields and search metrics within the _get_or_create_collection method.
    • Method Updates: Updated the following methods to use _get_or_create_collection:
      • online_write_batch
      • update
      • retrieve_online_documents_v2
    • Proto Value Handling: Improved the handling of Proto values and serialization in retrieve_online_documents_v2.

Which issue(s) this PR fixes:

#4364

Misc

N/A

@franciscojavierarceo franciscojavierarceo force-pushed the milvus-demo branch 2 times, most recently from 3731e17 to b7c3426 Compare January 10, 2025 21:59
@franciscojavierarceo franciscojavierarceo force-pushed the milvus-demo branch 2 times, most recently from ce41480 to 8360bbb Compare January 29, 2025 01:02
@franciscojavierarceo franciscojavierarceo marked this pull request as ready for review January 29, 2025 02:32
@franciscojavierarceo franciscojavierarceo requested a review from a team as a code owner January 29, 2025 02:32
dmartinol
dmartinol reviewed Jan 29, 2025
View reviewed changes
examples/rag/README.md Outdated
@@ -0,0 +1,29 @@
# Feast Quickstart
If you haven't already, check out the quickstart guide on Feast's website (http://docs.feast.dev/quickstart), which
Copy link
Contributor

@dmartinol dmartinol Jan 29, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This README seems a generic quickstart guide with no specific connections with the RAG function or the included example.
I assume that it will be more clear once we run it and see what happens, but I believe than a dev experience driven by a notebook may help, WDYT?

Copy link
Member Author

@franciscojavierarceo franciscojavierarceo Jan 29, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good call! I'll update to include a notebook! Will update this README as well.

Copy link
Member Author

@franciscojavierarceo franciscojavierarceo Jan 30, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let me know what you think!

@franciscojavierarceo
checking in progress but this Pr still is not ready yet
468368a 
Signed-off-by: Francisco Javier Arceo <[email protected]>
@franciscojavierarceo
feat: Adding new method to FeatureStore to allow more flexible retrie…
8f26494 
…val of features from vector similarity search

Signed-off-by: Francisco Javier Arceo <[email protected]>
@franciscojavierarceo
Adding requested_features back into online_store
ddc963b 
Signed-off-by: Francisco Javier Arceo <[email protected]>
@franciscojavierarceo
feat: Adding RAG demo displaying Milvus usage for RAG
006e897 
Signed-off-by: Francisco Javier Arceo <[email protected]>
@franciscojavierarceo
uploading sample data and updated yaml
a1a9408 
Signed-off-by: Francisco Javier Arceo <[email protected]>
@franciscojavierarceo
updating workflow
4fba66c 
Signed-off-by: Francisco Javier Arceo <[email protected]>
@franciscojavierarceo
updated example
c4736b7 
Signed-off-by: Francisco Javier Arceo <[email protected]>
@franciscojavierarceo
removing modified files
f7393a7 
Signed-off-by: Francisco Javier Arceo <[email protected]>
@franciscojavierarceo
reverting postgres change
d0a1e30 
Signed-off-by: Francisco Javier Arceo <[email protected]>
@franciscojavierarceo
updating test_workflow
0d4c351 
Signed-off-by: Francisco Javier Arceo <[email protected]>
@franciscojavierarceo
updated and fixed bug
7c006b1 
Signed-off-by: Francisco Javier Arceo <[email protected]>
@franciscojavierarceo
fixing a bad merge/rebase
63aa43b 
Signed-off-by: Francisco Javier Arceo <[email protected]>
@franciscojavierarceo
updated linter because of latest nistall
aa81350 
Signed-off-by: Francisco Javier Arceo <[email protected]>
@franciscojavierarceo
reverting feature store change
e2a1949 
Signed-off-by: Francisco Javier Arceo <[email protected]>
@franciscojavierarceo
adding logging of local milvus back
017ff54 
Signed-off-by: Francisco Javier Arceo <[email protected]>
@franciscojavierarceo
Updating readme and adding notebook
ac96e36 
Signed-off-by: Francisco Javier Arceo <[email protected]>
@franciscojavierarceo
updated readme
c629a42 
Signed-off-by: Francisco Javier Arceo <[email protected]>
@franciscojavierarceo franciscojavierarceo requested review from a team January 30, 2025 14:54
@franciscojavierarceo franciscojavierarceo requested review from HaoXuAI and shuchu and removed request for a team January 30, 2025 14:54
shuchu
shuchu approved these changes Jan 30, 2025
View reviewed changes
Copy link
Collaborator

@shuchu shuchu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lgtm

@franciscojavierarceo franciscojavierarceo merged commit 2daf852 into master Jan 30, 2025
26 checks passed
franciscojavierarceo pushed a commit that referenced this pull request Feb 4, 2025
@feast-ci-bot
chore(release): release 0.44.0
aa7224a 
# [0.44.0](v0.43.0...v0.44.0) (2025-02-04)

### Bug Fixes

* Adding periodic check to fix the sporadic failures of the operator e2e tests.  ([#4952](#4952)) ([1d086be](1d086be))
* Adding the feast-operator/bin to the .gitignore directory. Somehow it… ([#5005](#5005)) ([1a027ee](1a027ee))
* Changed Env Vars for e2e tests ([#4975](#4975)) ([fa0084f](fa0084f))
* Fix GitHub Actions to pass authentication ([#4963](#4963)) ([22b9138](22b9138)), closes [#4937](#4937) [#4939](#4939) [#4941](#4941) [#4940](#4940) [#4943](#4943) [#4944](#4944) [#4945](#4945) [#4946](#4946) [#4947](#4947) [#4948](#4948) [#4951](#4951) [#4954](#4954) [#4957](#4957) [#4958](#4958) [#4959](#4959) [#4960](#4960) [#4962](#4962)
* Fix showing selected navigation item in UI sidebar ([#4969](#4969)) ([8ac6a85](8ac6a85))
* Invalid column names in get_historical_features when there are field mappings on join keys ([#4886](#4886)) ([c9aca2d](c9aca2d))
* Read project data from the 'projects' key while loading the registry state in the Feast UI ([#4772](#4772)) ([cb81939](cb81939))
* Remove grpcurl dependency from Operator ([#4972](#4972)) ([439e0b9](439e0b9))
* Removed the dry-run flag to test and we will add it back later. ([#5007](#5007)) ([d112b52](d112b52))
* Render UI navigation items as links instead of buttons ([#4970](#4970)) ([1267703](1267703))
* Resolve Operator CRD bloat due to long field descriptions ([#4985](#4985)) ([7593bb3](7593bb3))
* Update manifest to add feature server image for odh ([#4973](#4973)) ([6a1c102](6a1c102))
* Updating release workflows to refer to yml instead of yaml ([#4935](#4935)) ([02b0a68](02b0a68))
* Use locally built feast-ui package in dev feature-server image ([#4998](#4998)) ([0145e55](0145e55))

### Features

* Added OWNERS file for OpenshiftCI ([#4991](#4991)) ([86a2ee8](86a2ee8))
* Adding Milvus demo to examples ([#4910](#4910)) ([2daf852](2daf852))
* Adding retrieve_online_documents endpoint ([#5002](#5002)) ([6607d3d](6607d3d))
* Adding support to return additional features from vector retrieval for Milvus db ([#4971](#4971)) ([6ce08d3](6ce08d3))
* Creating/updating the stable branch after the release. ([#5003](#5003)) ([e9b53cc](e9b53cc))
* Implementing online_read for MilvusOnlineStore ([#4996](#4996)) ([92dde13](92dde13))
* Improve exception message for unsupported Snowflake data types ([#4779](#4779)) ([5992364](5992364))
* Operator add feast ui deployment ([#4930](#4930)) ([b026d0c](b026d0c))
* Updating documents to highlight v2 api for Vector Similarity Se… ([#5000](#5000)) ([32b82a4](32b82a4))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dmartinol dmartinol dmartinol left review comments

@shuchu shuchu shuchu approved these changes

@lokeshrangineni lokeshrangineni Awaiting requested review from lokeshrangineni

@HaoXuAI HaoXuAI Awaiting requested review from HaoXuAI HaoXuAI was automatically assigned from feast-dev/maintainers

Assignees

No one assigned

Labels

ok-to-test

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@franciscojavierarceo @shuchu @dmartinol

Comments