Skip to main content

Download notebook · View source · Open in Colab

Work with PDFs

ApertureDB stores a PDF as a Blob. Text chunks from the PDF can be embedded and stored as Descriptors connected to that blob. A search can then return matching chunks and their source PDF in one query.

This example stores a recipe PDF with a server-derived _mime_type: "application/pdf", extracts and chunks its text, and generates embeddings with sentence-transformers. The document type enables inline rendering in the ApertureDB UI.

The Embeddings Extraction workflow supports larger ingestion jobs with OCR and parallel processing.

Connect to ApertureDB​

ApertureDB Cloud The SDK connects with an API key from your Cloud instance. Obtain the key from Connect > Generate API Key and add it to a .env file in this directory:

APERTUREDB_KEY=your_key_here

Local evaluation: follow the deployment guide, then configure your connection profile.

See client configuration options for all connection methods and server setup options for deployment choices.

%pip install --quiet "aperturedb==0.5.1" "python-dotenv==1.1.1" sentence-transformers pypdf
# Option A: ApertureDB Cloud
from dotenv import load_dotenv
load_dotenv()  # loads APERTUREDB_KEY from .env into the environment
# For local evaluation, configure the profile from your versioned deployment package.
# https://docs.aperturedata.io/administration/deployment/DockerCompose
from aperturedb.CommonLibrary import create_connector

client = create_connector()
response, _ = client.query([{"GetStatus": {}}])
client.print_last_response()

Download the Sample Recipe PDF​

The sample is a Butter Chicken recipe PDF from the Cookbook repository. pdf_path identifies the local file passed to ingestion.

import os

os.makedirs("data", exist_ok=True)

pdf_path = "data/ButterChicken.pdf"

!wget -q -O "{pdf_path}" \
    https://raw.githubusercontent.com/aperture-data/Cookbook/51b3c3d9f495f8fd77108a270b88fe8ab8cc0004/notebooks/simple/data/ButterChicken.pdf

print(f"Downloaded {pdf_path} ({os.path.getsize(pdf_path)} bytes)")

Store the PDF as a Blob​

AddBlob stores the PDF bytes and metadata. _mime_type: "application/pdf" is computed from its bytes and enables inline rendering in the ApertureDB UI. A unique run marker identifies this example's blob for later connections and cleanup.

from uuid import uuid4

run_id = "pdf-search-" + uuid4().hex
scope = ["$documentation_run", "==", run_id]

query = [{
    "AddBlob": {
        "_ref": 1,
        "properties": {
            "documentation_run": run_id,
            "name":      "butter_chicken_recipe",
            "dish_name": "Butter Chicken",
            "cuisine":   "Indian",
            "format":    "pdf"
        },
        "if_not_found": scope,
    }
}]

with open(pdf_path, "rb") as f:
    pdf_bytes = f.read()

response, _ = client.query(query, [pdf_bytes])
client.print_last_response()

Extract Text, Chunk, and Embed​

pypdf extracts the document text. The example splits it into overlapping chunks and uses sentence-transformers to embed each chunk. Each resulting Descriptor is linked to the source PDF blob.

from pypdf import PdfReader
from sentence_transformers import SentenceTransformer
import numpy as np

# Extract full text
reader = PdfReader(pdf_path)
full_text = " ".join(page.extract_text() for page in reader.pages)
print(f"Extracted {len(full_text)} characters")

# Chunk by words with overlap
def chunk_text(text, chunk_size=60, overlap=15):
    words = text.split()
    chunks = []
    for start in range(0, len(words), chunk_size - overlap):
        chunk = " ".join(words[start : start + chunk_size])
        if chunk:
            chunks.append(chunk)
    return chunks

chunks = chunk_text(full_text)
print(f"Created {len(chunks)} chunks")

# Embed all chunks
model = SentenceTransformer("all-MiniLM-L6-v2")   # 384-dimensional, CPU-friendly
embeddings = model.encode(chunks, normalize_embeddings=True)
print(f"Embedding shape: {embeddings.shape}")

Store Embeddings Linked to the PDF​

A DescriptorSet holds the chunk embeddings. Each descriptor stores its chunk text and connects to the source blob.

import uuid
SET_NAME = "recipe_pdf_chunks_" + uuid.uuid4().hex[:12]

client.query([{"AddDescriptorSet": {
    "name":       SET_NAME,
    "dimensions": 384,
    "engine":     "HNSW",
    "metric":     "CS",
}}])
client.print_last_response()
for i, (chunk, emb) in enumerate(zip(chunks, embeddings)):
    client.query([
        {
            "FindBlob": {
                "_ref": 1,
                "constraints": scope,
                "results": {"all_properties": False},
            }
        },
        {
            "AddDescriptor": {
                "_ref": 2,
                "set": SET_NAME,
                "properties": {
                    "chunk_index": i,
                    "chunk_text":  chunk[:200],
                    "source_name": "butter_chicken_recipe",
                },
            }
        },
        {
            "AddConnection": {
                "src": 2,
                "dst": 1,
                "class": "source_pdf",
            }
        }
    ], [emb.astype("float32").tobytes()])

print(f"Stored {len(chunks)} chunk embeddings linked to the PDF blob")

Semantic Search Over the PDF​

Vector search selects matching chunks. Graph traversal follows their connections to retrieve the source PDF metadata in the same query. Set blobs: True on FindBlob when the PDF bytes are also needed.

query_text = "spices and marinade"
query_emb = model.encode([query_text], normalize_embeddings=True)[0].astype("float32")

q = [{
    "FindDescriptor": {
        "set":         SET_NAME,
        "k_neighbors": 3,
        "distances":   True,
        "_ref":        1,
        "results":     {"all_properties": True},
    }
}, {
    "FindBlob": {
        "is_connected_to": {"ref": 1},
        "results": {"all_properties": True},
    }
}]

response, _ = client.query(q, [query_emb.tobytes()])

print(f"Query: '{query_text}'\n")
chunks_found = response[0]["FindDescriptor"].get("entities", [])
for r in chunks_found:
    score = 1 - r["_distance"]
    print(f"  score={score:.3f}  chunk #{r['chunk_index']}")
    print(f"  {r['chunk_text']}")
    print()

blobs_found = response[1]["FindBlob"].get("entities", [])
if blobs_found:
    print(f"Source PDF: {blobs_found[0]['name']} ({blobs_found[0]['cuisine']})")

Cleanup​

Remove this run's descriptor set and PDF blob. The run marker keeps cleanup scoped to the document created by this example.

# Delete the descriptor set (removes all chunk embeddings)
client.query([{"DeleteDescriptorSet": {"with_name": SET_NAME}}])
client.print_last_response()

# Delete the PDF blob
client.query([{"DeleteBlob": {"constraints": scope}}])
client.print_last_response()

What's Next?​