Skip to main content

Download notebook · View source · Open in Colab

Bulk Embedding Ingestion

ParallelLoader sends batches of queries concurrently. In this example, a query generator supplies dish embeddings and metadata from the Cookbook dataset. A sentence-transformer model converts dish descriptions to vectors before ingestion.

Connect to ApertureDB​

ApertureDB Cloud The SDK connects with an API key from your Cloud instance. Obtain the key from Connect > Generate API Key and add it to a .env file in this directory:

APERTUREDB_KEY=your_key_here

Local evaluation: follow the deployment guide, then configure your connection profile.

See client configuration options for all connection methods and server setup options for deployment choices.

%pip install --quiet "aperturedb==0.5.1" "python-dotenv==1.1.1" sentence-transformers pandas

# Encode literal data without interpreting a leading dollar sign as a property.
def literal(value):
    if isinstance(value, str) and value.startswith("$"):
        return "$" + value
    if isinstance(value, (list, tuple)):
        return {"array": [literal(item) for item in value]}
    if isinstance(value, dict):
        if len(value) == 1 and ("date" in value or "_date" in value):
            return {"to_date": literal(next(iter(value.values())))}
        return {"json": value}
    return value
# Option A: ApertureDB Cloud
from dotenv import load_dotenv
load_dotenv()  # loads APERTUREDB_KEY from .env into the environment
# For local evaluation, configure the profile from your versioned deployment package.
# https://docs.aperturedata.io/administration/deployment/DockerCompose
from aperturedb.CommonLibrary import create_connector

client = create_connector()
response, _ = client.query([{"GetStatus": {}}])
client.print_last_response()

Load Dataset and Generate Embeddings​

Each input combines dish_name and caption into a description. all-MiniLM-L6-v2 converts those descriptions to 384-dimensional embeddings and can run on a CPU.

import pandas as pd
import numpy as np
from sentence_transformers import SentenceTransformer

dishes = pd.read_csv(
    "https://raw.githubusercontent.com/aperture-data/Cookbook/51b3c3d9f495f8fd77108a270b88fe8ab8cc0004/images.adb.csv"
)
dishes["description"] = dishes["dish_name"] + " - " + dishes["caption"]
print(f"Loaded {len(dishes)} dishes")

model = SentenceTransformer("all-MiniLM-L6-v2")
embeddings = model.encode(dishes["description"].tolist(), normalize_embeddings=True)
print(f"Embedding shape: {embeddings.shape}")

Create the DescriptorSet​

The descriptor set defines the vector dimensions, index engine, and distance metric used for the dish embeddings.

import uuid
SET_NAME = "cookbook_bulk_" + uuid.uuid4().hex[:12]

client.query([{
    "AddDescriptorSet": {
        "name":       SET_NAME,
        "dimensions": 384,
        "engine":     "HNSW",
        "metric":     "CS",
    }
}])
client.print_last_response()

Bulk Ingest with ParallelLoader​

ParallelLoader consumes a Subscriptable that provides (query, blobs) pairs. The DescriptorGenerator below creates those pairs from dish metadata and precomputed embeddings. The loader processes batches of five records with four worker threads.

from aperturedb.ParallelLoader import ParallelLoader

class DescriptorGenerator:
    """Subscriptable generator of (query, blobs) pairs for ParallelLoader."""

    def __init__(self, dishes, embeddings, set_name):
        self.dishes     = dishes
        self.embeddings = embeddings
        self.set_name   = set_name

    def __len__(self):
        return len(self.dishes)

    def __getitem__(self, idx):
        # ParallelLoader calls __getitem__ with a slice for each batch;
        # return a list of (query, blobs) pairs in that case.
        if isinstance(idx, slice):
            return [self[i] for i in range(*idx.indices(len(self)))]
        row = self.dishes.iloc[idx]
        emb = self.embeddings[idx].astype("float32")
        query = [{
            "AddDescriptor": {
                "set": self.set_name,
                "properties": {
                    "dish_name":   row["dish_name"],
                    "cuisine":     row["food_tags"],
                    "caption":     row["caption"],
                },
                "if_not_found": ['$dish_name', '==', literal(row['dish_name'])],
            }
        }]
        return query, [emb.tobytes()]

generator = DescriptorGenerator(dishes, embeddings, SET_NAME)
loader    = ParallelLoader(client)
loader.ingest(generator, batchsize=5, numthreads=4, stats=True)

Verify the Ingestion​

The count query reports the number of descriptors stored in the set.

response, _ = client.query([{
    "FindDescriptor": {
        "set": SET_NAME,
        "results":   {"count": True},
    }
}])
client.print_last_response()

Search the Bulk-Loaded Descriptors​

The query text is embedded with the same model as the dish descriptions. FindDescriptor returns up to three similar dishes with their metadata and distances.

query_text = "creamy tomato curry"
query_emb  = model.encode([query_text], normalize_embeddings=True)[0].astype("float32")

response, _ = client.query([{
    "FindDescriptor": {
        "set":         SET_NAME,
        "k_neighbors": 3,
        "distances":   True,
        "results":     {"all_properties": True},
    }
}], [query_emb.tobytes()])

for entity in response[0]["FindDescriptor"].get("entities", []):
    score = 1 - entity["_distance"]
    print(f"  {entity['dish_name']:<30} [{entity['cuisine']}]  score={score:.3f}")

Cleanup​

Cleanup removes the descriptor set created for this run.

client.query([{"DeleteDescriptorSet": {"with_name": SET_NAME}}])
client.print_last_response()

What's Next​