How to Add new data to an existing Index in Vector Search vertex AI using python library

Question

I am familiar with vectordbs like weaviate and pinecone, and trying to migrate my vectors to gcp. I have uploaded data to a bucket, created index with it and deployed an endpoint using python and following this documentation Vector Search Quickstart.Querying works fine for the index.

Now I want to add new data to the same index, (batch upsertion or equivalent in weaviate or pinecone), which as far as i understand can be done via GUI by doing batch update in the edit Index option, but I am unable to figure out how can i achieve it with python client.

I apologise if this is stupid question, been trying to figure out for a while now, and docs are confusing me more than helping.

I looked at update and rebuild Index and updateIndex, but am unable to figure out if this creates entire index from scratch with the new data or upserts them like weaviate or pinecone.

Answer 1

A staff answered on google cloud forum, here

Welcome and thank you for reaching out to our community. I believe both functionalities do not have batch data processing (upserting). The update index updates the metadata associated with the index, like description, labels, or settings while the actual data content (vectors and document IDs) remains untouched
The rebuild index rebuilds the entire index from scratch, deleting all existing data in the index then recreates it using the data provided in the request I hope I was able to provide you with useful insights.

However I do think its possible, just not with aiplatform as of now (march 2024), There's a method for IndexService in aiplatformv1, not in aiplatform.

Someone pointed this out in .NET, in another reply to the staff:

Hey, not sure about the naming for Python but in .NET there is a class IndexServiceClient that has all methods to manage the index. And it has a method UpsertDatapoints that processes a UpsertDatapointsRequest which represents batch of vectors.