Skip to content

feat: Add complex type support (Map, JSON, Struct) with schema validation#5974

Open
ntkathole wants to merge 3 commits intofeast-dev:masterfrom
ntkathole:fix_complex_type
Open

feat: Add complex type support (Map, JSON, Struct) with schema validation#5974
ntkathole wants to merge 3 commits intofeast-dev:masterfrom
ntkathole:fix_complex_type

Conversation

@ntkathole
Copy link
Member

@ntkathole ntkathole commented Feb 16, 2026

What this PR does / why we need it:

This PR adds comprehensive support for complex data types in Feast and implements schema validation across all compute engines:

Commit 1 (970650b): Fix Map/Dict support and implement schema validation

  • Fixed Map/Dict type support across offline stores, online stores, and type mappings
  • Added enable_validation parameter to FeatureView, BatchFeatureView, and StreamFeatureView
  • Implemented schema validation nodes in Local, Ray, and Spark compute engines (missing columns, type mismatch warnings)
  • Added Spark and Milvus map type mappings
  • Updated documentation for Map type backend mappings

Commit 2 (ac28d4d): Add JSON and Struct complex data types

  • Added JSON, JSON_LIST, STRUCT, STRUCT_LIST to the protobuf ValueType enum and Value message
  • Introduced Json primitive type and Struct class (schema-aware structured type with named, typed fields)
  • Implemented full type mapping pipeline: Feast ↔ Proto ↔ PyArrow ↔ backend-native types (PostgreSQL, BigQuery, Snowflake, Redshift, Spark, MSSQL, Athena, DynamoDB, Milvus)
  • Added JSON well-formedness validation at both proto conversion level (always active) and validation node level (when enable_validation=True)
  • Implemented Spark-native type validation using from_feast_to_spark_type() and _spark_types_compatible() — eliminates the PyArrow-to-Spark type mismatch gap
  • Persisted Struct field schemas through the registry via Field tags (feast:struct_schema)
  • Added Go SDK type conversion support for JSON and Struct types
  • Added automatic serialization/deserialization for JSON type (Python dict/list ↔ JSON string)
  • Single _validate_schema() method per engine (missing columns + type checks + JSON content validation)

Commit 3 (a131b73): Modified default template with different types

  • Added Json, Map and Struct in driver template

Open with Devin

@ntkathole ntkathole self-assigned this Feb 16, 2026
@ntkathole ntkathole changed the title feat: Fix Map/Dict support and implement schema validation feat: Add complex type support (Map, JSON, Struct) with schema validation Feb 16, 2026
@ntkathole ntkathole marked this pull request as ready for review February 16, 2026 15:49
@ntkathole ntkathole requested a review from a team as a code owner February 16, 2026 15:49
devin-ai-integration[bot]

This comment was marked as resolved.

devin-ai-integration[bot]

This comment was marked as resolved.

@ntkathole ntkathole force-pushed the fix_complex_type branch 2 times, most recently from a19f25a to 85287f3 Compare February 16, 2026 17:11
devin-ai-integration[bot]

This comment was marked as resolved.

devin-ai-integration[bot]

This comment was marked as resolved.

devin-ai-integration[bot]

This comment was marked as resolved.

devin-ai-integration[bot]

This comment was marked as resolved.

devin-ai-integration[bot]

This comment was marked as resolved.

devin-ai-integration[bot]

This comment was marked as resolved.

devin-ai-integration[bot]

This comment was marked as resolved.

@ntkathole ntkathole force-pushed the fix_complex_type branch 3 times, most recently from f288425 to 4070109 Compare February 19, 2026 05:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

Comments