Steps to Add Vector Embeddings

Step 1: Edit entity document

Users need to create and edit entity documents for each entity that they want to generate embeddings for. The user can use a graphical user interface or a text editor to write the templates in the domain-specific language. The users can also preview and test the templates on sample records before saving them. 

Step 2: Storing records in Vector Database

Next initiate the embeddings generation process. The process involves the following sub-steps: 

Sub-step 1

The MemberJunction framework will fetch the records from the source entity and apply the entity document template to each record. The result will be a text document for each record that contains the relevant fields and values in a structured and readable format. 

Sub-step 2

The MemberJunction framework sends the text documents to the embeddings model in batches. The model then processes the text documents and returns the vector embeddings for each document. The embeddings will have a fixed dimensionality and will be normalized to have a unit length. 

Sub-step 3

The MemberJunction framework stores the vector embeddings in the vector database. The database will then index the embeddings and assign a unique identifier to each embedding. The database also stores the metadata of the embeddings, such as the record id, the entity name, and the timestamp. 

Step 3: Vector integration with different applications

Users can access and use the embeddings for various applications. Users can query the vector database from the MemberJunction framework or from a separate interface. The user can also perform tasks such as similarity search, duplicate detection, clustering, and more with the embeddings. Moreover, users can update, delete, or re-generate the embeddings as needed.