CREATE a Vector Index

  • Operational Insights
  • reference
Use a CREATE statement to create a vector index for approximate nearest neighbor (ANN) search on a remote or standalone collection.

A vector index, of type VTREE, stores the embeddings from a single vector field and organizes them into a routing tree of leaf clusters. The query optimizer uses a VTREE index to accelerate approximate nearest neighbor (ANN) queries, pruning the search to a subset of clusters instead of scanning every record. For more information about the query-side functions that use this index, see Vector Functions.

A vector index covers only 1 vector field.

Before you create a vector index on a collection, you must run ANALYZE COLLECTION on that collection to gather the samples the index build requires. Creating a vector index on a collection with no samples fails. For more information, see Collecting Samples.

Syntax

CreateVectorIndex EBNF
CreateVectorIndex ::= "CREATE" "INDEX" Identifier ("IF" "NOT" "EXISTS")?
                       "ON" QualifiedName
                       "(" VectorIndexField ")"
                       ("INCLUDE" "(" NestedField ( "," NestedField )* ")")?
                       ("TYPE" "VTREE")?
                       "WITH" ObjectConstructor
CreateVectorIndex Diagram
"CREATE" "INDEX" Identifier ("IF" "NOT" "EXISTS")? "ON" QualifiedName "(" VectorIndexField ")" ("INCLUDE" "(" NestedField ( "," NestedField )* ")")? ("TYPE" "VTREE")? "WITH" ObjectConstructor
Show VectorIndexField Diagram
NestedField "VECTOR"
VectorIndexField
Show NestedField Diagram
Identifier ( "." Identifier )*
NestedField
A vector index does not accept the CAST(…​) modifier or an INCLUDE/EXCLUDE UNKNOWN KEY modifier, both of which are available on a standard secondary index. For more information, see CREATE INDEX Statements.

Examples

The following example creates a vector index on the review_vector_emb field, specifying only the mandatory dimension and similarity parameters and omitting the optional TYPE clause.

 CREATE INDEX review_vector_idx
 ON products.ptype.product (review_vector_emb VECTOR)
 WITH {"dimension": 384, "similarity": "cosine"};
Show additional examples

The following example creates a vector index and uses the INCLUDE clause to replicate the rating and genre fields in the index. A query that filters on rating or genre, orders by vector distance, and selects only the primary key or INCLUDE fields can then be served entirely from the index, without a lookup against the base collection. If the query selects other fields, the filter is still evaluated inside the index search, but each result is then looked up in the collection and re-ranked.

 CREATE INDEX review_vector_idx
 ON products.ptype.product (review_vector_emb VECTOR)
 INCLUDE (rating, genre)
 TYPE VTREE
 WITH {"dimension": 384, "similarity": "cosine"};

The following example fine-tunes the index build and storage behavior by overriding the default settings for train_list_fraction, quantization, and epsilon:

 CREATE INDEX ix_vtree
 ON Entertainment.Movie (embedding VECTOR)
 INCLUDE (title, vote_average, revenue)
 TYPE VTREE
 WITH {
     "dimension": 384,
     "similarity": "cosine",
     "train_list_fraction": 0.2,
     "quantization": "SQ8",
     "epsilon": 0.25
 };

Drop a vector index the same way you drop any other index. For more information, see DROP Statements.

 DROP INDEX Entertainment.Movie.ix_vtree IF EXISTS;

Arguments

Identifier

For the CREATE INDEX statement, the first Identifier is the index name.

QualifiedName

The QualifiedName identifies the collection to build the index on.

VectorIndexField

The VectorIndexField consists of a NestedField that specifies the vector field to index, followed by the VECTOR keyword. The field must contain an array of floating point numbers with the dimensionality specified in the WITH clause. You can have only 1 vector field per vector index.

INCLUDE

The optional INCLUDE clause replicates the listed non-key fields directly in the vector index. When you omit INCLUDE, the index stores only the primary key and the vector field. Replicating fields in the index lets the query optimizer take ANN query paths that filter on those fields from within the closest leaf clusters, instead of looking up each candidate record in the base collection.

Do not specify a type for an INCLUDE field of a vector index. Specifying one causes a compilation error.

TYPE

The optional TYPE clause specifies the index type. VTREE is the only supported vector index type, and it’s the default when you omit TYPE on an index over a VECTOR field.

WITH

The WITH clause is mandatory for a vector index. Its ObjectConstructor represents an object containing key-value pairs, 1 for each parameter. You can define the following parameters:

Name Description Schema

dimension
Required

The dimensionality of the vector field. This specifies the length of the vector arrays stored in the indexed field, such as 384 for a common sentence-transformer model.

integer

similarity
Required

The distance, or similarity, metric used to train and query the index. Accepts one of the following string values:

  • euclidean

  • l2

  • euclidean_squared

  • l2_squared

  • cosine

  • dot (supported only for normalized embeddings in this release)

enum: euclidean, l2, euclidean_squared, l2_squared, cosine, dot

train_list_fraction
Optional

The decimal fraction of dataset records sampled to train the leaf centroid structure during index creation. For example, 0.1 samples 10% of the dataset. Valid values range from greater than 0 to 1. Operational Insights rejects 0.

Default: derived from num_clusters and the number of records in each storage partition, or 1.0 on a small dataset.

float

quantization
Optional

The scalar quantization applied to stored vectors to reduce index size and speed up distance computation, at the cost of a small reduction in recall. Accepts SQ4 (4-bit) or SQ8 (8-bit).

Default: SQ8

enum: SQ4, SQ8

epsilon
Optional

A fractional tolerance, greater than 0, that extends the search radius beyond the closest cluster boundary during index insertion and ANN query execution. A larger epsilon increases recall at the cost of visiting more clusters.

Default: 0.75

float

num_clusters
Optional

The number of leaf clusters created in each storage partition.

Default: the square root of the partition’s record count. If that square root is less than 100, Operational Insights uses the ceiling of the record count divided by 100 instead. The compiler.index.vector.max.num.clusters setting caps this value, and defaults to 65536.

integer

storage-block-compression
Optional

The block-level compression applied to the index’s storage. Takes a nested object with 1 field, scheme, whose value is snappy or none.

Default: the cluster’s default storage compression scheme, snappy unless changed.

object

A num_clusters value greater than the compiler.index.vector.max.num.clusters cap, or greater than the partition’s record count, results in an error. A value more than 10% away from the recommended default draws a warning.
Setting both num_clusters and train_list_fraction is an error. Setting num_clusters derives train_list_fraction from it, but setting train_list_fraction does not derive num_clusters — it keeps its own record-count-based default instead.
Name Description Schema

seed
Optional

A seed value for the index build’s random number generator. Set a specific value to reproduce the same exact index structure each time you create the index.

Default: a random value, unless you specify one

integer

The VTREE index in Operational Insights does not index records whose vector embedding field is any of the following, without raising a warning:
  • missing

  • null

  • The wrong type

  • Empty

  • The wrong length