Skip to main content

Working with Bounding Boxes and Polygons

For more details on all the functionality related to bounding boxes, polygons, other regions, in ApertureDB, please refer to our Image ROI documentation and Video ROI documentation

# Dependencies
from aperturedb.Images import Images
from aperturedb.Utils import Utils, create_connector
from aperturedb import NotebookHelpers as nh
from aperturedb.Constraints import Constraints
import os

import helper

# Connect to the ApertureDB instance.
db = create_connector()
utils = Utils(db)
# Parameters used across queries in this notebook.
demo_name = "coco"
data_source = "coco"
width = 600
height = 600
display_limit = 1
label = "horse"
polygon_label = "dog"
roi_label1 = "tie"
roi_label2 = "person"
image_id = 458992
polygon_id = 226510

Display BoundingBoxes using the Python object mapper API

print("\nShowing bounding boxes overlaid on a subset of images from the dataset: " + data_source)

imgs = Images(db, batch_size=display_limit)
const = Constraints()
const.equal("adb_data_source", data_source)
print("where " + str(imgs.total_results()) + " images matched the query\n")

# It is as simple
# Note: The images are not ordered, and may not have associates bounding boxes.
# those will be diplayed, but not highlighted.
imgs.display(limit=display_limit, show_bboxes=True)
Showing bounding boxes overlaid on a subset of images from the dataset: coco
where 123287 images matched the query


Display Polygons

# Simply switch over to a polygon display
imgs.display(limit=display_limit, show_polygons=True)


Find Images using Labels or Annotations

We use JSON examples here since it allows us to build more complex queries

Retrieve images with a given label being 40% of the image

Graph (metadata) traversal makes it easy to find images and create datasets

print("\nReturning resized version of images with resolution better than: " + str(width) + "x" + str(height) + " with label: " + label + "\n")

q = [ {
"FindBoundingBox": {
# this is key to traversing the graph-based metadata. We are declaring a reference.
"_ref": 1,
"with_label": label,
# ensuring that the box with label is in a given area of the image
"in_rectangle": {
"x": 0,
"y": 100,
"width": 300,
"height": 500
# we would like to see the coordinates of this box
"coordinates": True,
# but not retrieve the pixel area contained within
"blobs": False,
"labels": True,
}, {
"FindImage": {
# we can use the connection between bounding box and image to go back and forth
"is_connected_to": {
"ref": 1,
"uniqueids": True,
# fetch only the images that have a decent resolution
"constraints": {
"width": [">", width],
"height": [">", height]
# also apply some operations on the fly to matching images
"operations": [
"type": "resize",
"width": 224,
"height": 224
"results": {
"limit": display_limit
} ]

responses, blobs = db.query(q)

#Let's make a wrapper with the response from db.query
images_wrapper = Images(db, batch_size=display_limit, response=responses[1]["FindImage"]["entities"])

# and display the results with bounding boxes drawn on top

Returning resized version of images with resolution better than: 600x600 with label: horse


# Show the above again, with polygons instead of bounding boxes


Doing a LOT more with Intersection over Union

Filter and sort regions by their properties

Polygon regions can be queried based on their properties just like any other entity type in ApertureDB. documentation

This example is looking for high-resolution images of the chosen label.

It queries for polygon regions with the expected label, and displays them in descending order by area (measured in pixels).

images, large_dog_polygons = helper.find_polygons_by_property(db, with_label=polygon_label, property="_area", sort="descending", limit=display_limit)

print("displaying the {} largest {} polygons".format(display_limit, polygon_label))

images.display(polygon_constraints=large_dog_polygons, polygon_tag_key="_area", polygon_tag_format="{:.0f} pixels")
displaying the 1 largest dog polygons


Find overlapping polygons

ApertureDB can be used to identify intersecting polygon regions within an image.

This example is looking for overlap between specified roi_label1 and roi_label2.

Basically the query is for any polygon labeled roi_label1 that intersects with a polygon labeled roi_label2.

print("\nReturning images with annotated polygon: " + roi_label1 + " overlapping with polygon: " + roi_label2 + "\n")

images, overlapping_polygons = helper.find_overlapping_polygons(db, roi_label1, roi_label2)

images.display(polygon_constraints=overlapping_polygons, limit=display_limit)
Returning images with annotated polygon: tie overlapping with polygon: person


Find polygons in a fixed location

ApertureDB supports querying for polygon regions based on their location within an image.

For instance, suppose you want to search a security camera feed for people using a specific door, or to flag objects that may extend out of frame.

This example is looking for the given polygon_label extending out of frame past the upper left corner.

It uses a custom "r" shape to describe the upper-left corner of an image (displayed in red), and then queries for polygons labeled polygon_label that intersect that shape.

upper_left_corner_region = [ # This describes the following shape:
[0.0, 0.0], # ____________
[100.0, 0.0], # | _______/
[95.0, 5.0], # | /
[40.0, 5.0], # | |
[5.0, 40.0], # | |
[5.0, 95.0], # | |
[0.0, 100.0], # |/

print("\nReturning images where annotated polygon: " + polygon_label + " spans the upper left corner of the image\n")

images, trains_in_upper_left = helper.find_polygons_within_region(db, with_label=polygon_label, within_region=upper_left_corner_region)


images.display(polygon_constraints=trains_in_upper_left, limit=display_limit)
Returning images where annotated polygon: dog spans the upper left corner of the image


Evaluating an Object Detector by its RoI quality

With ApertureDB, it is easy to compare two sets of regions that describe the same scene.

In this example, we evaluate the similarity between this dataset's polygons and bounding boxes within the same image.

It does this by computing the pairwise intersection-over-union (IoU) values between the individual bounding box and polygon regions. documentation

print("\nIoU matrix for an image with id : " + str(image_id) + "\n")

image = helper.print_iou_matrix_for_image(db, image_id)

IoU matrix for an image with id : 458992

IoU matrix for polygons vs. bounding boxes in image 458992:
PLGN \ BBOX|person| fork |knife |pizza
fork |0.0082|*0.1014|0.0000|0.0047
knife |0.0080|0.0000|*0.5920|0.0044
pizza |0.0070|0.0020|0.0000|*0.7717



Polygon Levels of Detail

Polygon regions in ApertureDB can be retrieved at different of detail levels suitable for a variety purposes.

FindPolygon can be used to retrieve polygon geometry at the following levels of detail, from least to most complex:

  • bounds: Returns the smallest axis-aligned bounding box that contains the region.
  • hulls: Returns the smallest convex polygon that contains the region.
  • vertices: Returns the full-resolution polygon geometry exactly as it was added.
  • decompositions: Returns a breakdown of the region into disjoint convex pieces.

This examples shows all available levels of detail for a single polygon.

print("\nPolygon geometry for a specific polygon with id : " + str(polygon_id) + "\n")

helper.display_polygon_levels_of_detail(db, polygon_id)
Polygon geometry for a specific polygon with id : 226510