CoralNet-Toolbox

Usage

Overview

The CoralNet Toolbox is a Python application built using PyQt5 for image annotation.
This guide provides instructions on how to use the application, including key functionalities and hotkeys

Annotations

• PatchAnnotation: Represents a patch annotation
• RectangleAnnotation: Represents a rectangular annotation
• PolygonAnnotation: Represents a polygonal annotation
  • MultiPolygonAnnotation: Represents multiple, non-overlapping polygonal annotations
• MaskAnnotation: Represents a segmentation mask (one mask per image)

Rasters

Images

• Standard image formats (PNG, JPG, etc.) can be imported directly
• Each image becomes a canvas for annotations and model predictions

Orthomosaics

• GeoTIFF files with geospatial metadata (CRS and transforms)
• Enable spatial analysis and coordinate-based measurements
• Support for GeoJSON export of annotation coordinates
• Useful for drone surveys and mapping workflows

Videos

• Video files can be imported and played frame-by-frame in the Annotation Window
• Each frame is treated as an individual image for annotation purposes
• Frames can be extracted and imported as individual images

Frames from Video

• Extract specific frames from a video file for annotation
• Useful for selective processing without importing entire video
• Extracted frames are added to the project as regular images

Extracted Work Areas (Tiles)

Split an orthomosaic or a large image into smaller images to annotate, then merge the annotations back onto the original:

1. Lay out the work areas

• Put work areas on the raster with Utilities > Work Areas (a tile grid) or the Work Area Tool (by hand)
• For images you plan to merge back, use Edge tiles: Smaller edge tiles with no overlap, so every pixel is in exactly one image

2. Extract

• Right-Click the raster in the Rasters Window, then Work Areas > Extract Work Areas…
  • Works on images and orthomosaics that have work areas, several highlighted rasters at once
• Output Folder: Each raster’s images go in a subfolder named after it, as prefix_x_y_width_height
• Format: tif keeps the georeference of a georeferenced raster; png and jpg do not
• Skip Empty Tiles: Leaves out work areas with no data, such as the blank corners of an orthomosaic, or that are a single flat color
• Include Annotations: Copies the raster’s annotations and mask onto the new images, clipped to each work area; the originals are not changed
• Extract: Writes the images to disk only
• Extract and Import: Also adds them to the project as regular images, linked to the raster they came from
  • Each image gets a copy of the raster’s scale, including one set with the Scale Tool, which takes precedence over the scale a GeoTIFF image would read from its own georeference
  • Overlapping work areas give a warning first: merging back later will ask about annotations in the overlapping strips

3. Annotate

• Extracted images are regular images: annotate them, run inference on them, or use them in Active Learning and Export Dataset
• While its images are in the project, the original is left out of Active Learning and Export Dataset, so the same pixels are not counted twice, and it cannot be extracted again
• Hover over a row to see Extracted: N images (on the original) or Extracted from (on an extracted image)

4. Merge back

• Right-Click the original or any of its images, then Work Areas > Merge Back…; the whole set is merged at once
• Each annotation is compared with how it was when it was extracted:
  • Changed only on the images: the change is applied
  • Changed only on the original: the original is kept
  • Deleted on the images: it is deleted from the original
  • Drawn new on an image: it is added to the original, alongside whatever it overlaps, unless it overlaps an annotation of the same label that was added to the original after extraction (then it is a conflict)
  • Different labels are never treated as the same object: overlapping annotations of two labels are both brought in
  • Pieces of one annotation cut by an image’s edge are joined back together
• Masks merge pixel by pixel the same way
• When both changed: What to do where an annotation changed on both sides (edited on both, relabeled differently on two images, or one of the overlaps above), applied to all of them:
  • Keep new: The images’ version replaces the original’s
  • Keep old: The original’s version stays
  • Keep both: Both are kept
  • Merge: Shapes with the same label are joined into one; different labels, and patches, are kept both
  • Mask pixels changed on both sides take the images’ class, unless you choose Keep old
• After merging:
  • Remove images from project: The images leave the project; their files stay in the output folder
  • Remove images and delete files: Also deletes the images, their tile record and mask snapshots from disk (only files the extraction created); this cannot be undone
  • Keep images as ordinary images: The images stay in the project, unlinked, with their annotations
• Merging clears the undo history, and work areas drawn on the extracted images are not brought back

Other Work Areas actions

• Highlight Extracted Images: Highlights the set’s images in the Rasters Window
• Unlink Extracted Images: Keeps the images as ordinary images and releases the original, without merging anything

Z-Channels (Depth Maps & DEMs)

Z-Channels overlay depth or elevation data on top of images for spatial analysis:

Supported Data Types

• Depth Maps: 2D array representing depth / distance values (typically from sensors or stereo reconstruction)
• Digital Elevation Models (DEMs): Georeferenced elevation data for terrain analysis
• Supports GeoTIFF, PNG, and other raster formats that can be loaded with rasterio

Importing Z-Channel Data

• Right-Click on highlighted images in Rasters Window
• Import Z-Channel: Select depth / elevation file(s) to attach to image(s)
  • Files are automatically resized to match image dimensions if needed
  • Unit of measurement can be specified (meters, feet, centimeters, etc.)
• Batch Import: Apply same Z-Channel to multiple selected images

Z-Channel Visualization

• Overlay appears in the Annotation Window alongside image and annotations
• Colormap Selection: Dropdown in Status Bar to choose visualization style
  • Available colormaps: viridis, plasma, inferno, magma, cividis, twilight, etc.
  • Different colormaps highlight different depth / elevation patterns
• Dynamic Range: Enable / disable dynamic recoloring via Status Bar button
  • When enabled: colormap automatically rescales to current viewport’s min / max values
  • When disabled: colormap uses image’s global min / max (better for comparing multiple images)
• Opacity Control: Adjust Z-Channel overlay transparency via Status Bar slider
  • 0% (transparent) shows only the image
  • 100% (opaque) shows full Z-Channel visualization
• Value Display: Hover over image to see Z value at cursor position in Status Bar (e.g., “Z: 2.45 m”)

Measurement & Analysis

• Scale Tool: Distances and areas automatically account for Z-Channel (depth-corrected measurements)
• Hover Tooltips: View exact Z value when mouse hovers over image
• Nodata Handling: Automatically detects and masks NULL / missing values in Z data

Removing Z-Channel Data

• Right-Click on highlighted images in Rasters Window
• Remove Z-Channel: Clear depth / elevation data from selected image(s)
  • Annotation and other data remain intact

Computer Vision Tasks

• Classification: Assign a label to an image (Patch)
• Detection: Detect objects in an image (Rectangle)
• Instance Segmentation: Segment objects in an image (Polygon)
• Semantic Segmentation: Segment the entire image (Mask)

Thresholds for Computer Vision Tasks

• Max Detections: Maximum number of detections kept after non-max suppression (1-10,000)
  • Lower values reduce clutter and processing time; higher values allow more candidates
• Uncertainty Threshold: Minimum confidence required to accept a prediction as valid (0.00-1.00)
  • Predictions below this value are marked as Review for manual inspection
• IoU Threshold: Intersection-over-Union threshold for non-max suppression (0.00-1.00)
  • Higher values keep more overlapping detections; lower values remove duplicates more aggressively
• Area Threshold: Normalized annotation area filter (min and max as 0.00-1.00 fractions)
  • Min: Objects smaller than this fraction of image area are removed
  • Max: Objects larger than this fraction of image area are removed
• Boundary Detections: Keep or Ignore detections that touch a work-area edge
  • Keep: Retains cut-off objects; Ignore: Removes them to reduce seam duplicates across tiles

Main Window

The main window consists of several dockable components:

• Menu Bar: Contains import, export, and other actions
• Tool Bar: Contains tools for selection and annotation
• Status Bar: Displays the image size, cursor position, view extent, annotation transparency, and thresholds
• Annotation Window: Displays the image, Z-channel, and annotations with interactive annotation tools
• Labels Window: Lists and manages labels with operations for editing, merging, and organizing
• Rasters Window: Displays imported images with filtering and batch operations
• Confidence Window: Displays cropped images and confidence charts for annotation predictions
• Metadata Window: Displays and edits structured metadata for the selected annotation(s)
• Performance Window: Real-time hardware monitor showing CPU, Memory, and GPU usage with historical sparkline graphs
• Timer Window: Session timer with start, stop, and reset controls; tracks total work duration across sessions

Advanced Docking System

All dock windows (Annotation Window, Labels Window, Rasters Window, Confidence Window, Metadata Window, Explorer, Performance, Timer) use an advanced docking system that allows complete layout customization:

Moving & Rearranging Docks

• Grab & Drag: Click and drag the dock title bar to move the window
  • Drag to any dock area (left, right, top, bottom) to dock it
  • Drag to create a floating window (detached from main window)
  • Drag between dock areas to rearrange layout
• Tab Navigation: Multiple docks in the same area appear as tabs
  • Click tabs to switch between docked windows
  • Drag tabs to reorder or move to different areas
• Resize Docks: Drag the borders between docks to resize
  • Works horizontally and vertically
  • Respects minimum and maximum size constraints

Floating Windows

• Float a Dock: Drag title bar away from the main window to float it
  • Floating windows can be positioned anywhere on screen
  • Each floating window has independent window controls (minimize, maximize, close)
  • Windows remember their floating state across sessions
• Dock a Float: Drag floating window title bar back to the main window to re-dock it

Closing & Toggling Visibility

• Close a Dock: Click the ✕ button on the dock title bar
  • Docked windows can be closed / hidden without losing their configuration
  • Closed docks can be reopened via menu or keyboard shortcut
  • Data is preserved when docks are hidden
• Toggle Dock Visibility: View menu contains options to show / hide all docks
  • Use keyboard shortcuts or menu items to toggle dock visibility
  • Useful for maximizing workspace for specific tasks

Default Layout

• A starting arrangement ships with the application and is used until a layout is saved
• Docks that are closed in it, such as Performance and Timer, can be opened from View > Windows

Layout Persistence

• Dock positions, sizes, and floating state are automatically saved on exit and restored on the next launch
• Saved layouts are written to .cache/layout/, next to the other cache directories
• View > Layout > Save: Store the current arrangement under a name of your choosing
• View > Layout > Load: Switch to any previously saved arrangement
• View > Layout > Reset: Return to the default layout that ships with the application

Menu Bar Actions

File

• New Project: Reload CoralNet-Toolbox (loss of data warning)
• Open Project: Open an existing CoralNet-Toolbox project file
  • Supports both binary (.bin) and JSON (.json) formats
  • File dialog allows selection of either format
• Save Project: Save current CoralNet-Toolbox project file
  • Choose between binary (.bin) or JSON (.json) format when saving
  • Binary Format (.bin):
    • Smaller file size (significant space savings for large projects)
    • Faster load / save times
    • Not human-readable (cannot be edited with a text editor)
    • Recommended for production use and large datasets
  • JSON Format (.json):
    • Human-readable text format
    • Can be edited with any text editor for manual fixes
    • Larger file size
    • Better for version control and sharing human-readable backups
• Import:
  • Rasters:
    • Images: Load image files (PNG, JPG, etc.)
    • Videos: Import video files to the project
    • Frames from Video: Extract and import frames from a video file
    • Orthomosaics: Import orthomosaic GeoTIFF files with geospatial metadata
  • Labels:
    • Labels (JSON): Load label data from a JSON file
    • CoralNet (CSV): Load label data from a CoralNet CSV file
    • TagLab (JSON): Load label data from a TagLab JSON file
  • Annotations:
    • Annotations (JSON): Load annotation data from a JSON file
    • CoralNet (CSV): Load annotation data from a CoralNet CSV file
    • TagLab (JSON): Load annotation data from a TagLab JSON file
    • Squidle+ (JSON): Load annotation data from a Squidle+ JSON file
    • Viscore (CSV): Load annotation data from a Viscore CSV file
  • Masks: Import segmentation mask images as MaskAnnotations
  • Metadata: Import metadata field definitions from a YAML file
  • AI-Ready Datasets: Import a YOLO dataset for machine learning (Detect, Segment, Semantic)
    • Dataset Summary: Selecting a data.yaml reports how many images were found, how many carry annotations, and the class count
    • Import Mode: Choose whether the dataset is copied into the project
      • Copy images into a project folder (default): copies every image into the output directory, leaving the originals untouched, so the project is self-contained
        • Every split (train, val, test) is flattened into a single images folder
        • Images sharing a base name across splits would overwrite each other, so a rename option is offered when duplicates are detected
        • An annotations.json is written alongside the copies
      • Import in place: reads the images where they already are and writes nothing to disk
        • Faster for large datasets, and split folders are kept, so base name conflicts cannot occur
        • A saved project references the original dataset folder and breaks if it is moved or deleted
      • Supports PNG, JPG, JPEG, JFIF, BMP, TIF, TIFF, and WebP images
    • Image Import Rule (right side of the dialog): Choose which images are imported
      • Import only images with annotations (default): skips images that have no label file or mask
      • Import all images found in dataset: includes unlabeled images alongside labeled ones
      • Import images only, without annotations: imports every image and ignores its label file or mask; no labels are created
      • Import a random subset: draws that many images from whichever rule is selected
    • Classes to Import: Uncheck classes to leave them out; Select All and Deselect All toggle the whole list
    • Annotation Importing: YOLO format annotations are converted and imported
      • Detection datasets: Bounding boxes converted to Rectangle annotations
      • Instance segmentation datasets: Polygon masks converted to Polygon annotations
      • Semantic segmentation datasets: Single-channel mask images converted to Mask annotations (one per image)
      • Labels automatically created from dataset class names
      • Each image is paired with the sidecar file at the matching path: labels for Detect and Segment, masks for Semantic
    • Dataset Organization: Both standard YOLO directory layouts are recognized automatically
      • Split first: train/images/, train/labels/, val/images/, val/labels/
      • Images first: images/train/, images/val/, labels/train/, labels/val/
      • The train, val, and test entries in the YAML are used to locate the data
      • An entry naming the split folder itself (train: train) resolves to the images folder inside it
      • If those entries point somewhere that does not exist (common when a dataset is moved after being exported with an absolute path), the same folders are looked for alongside the YAML file instead
      • If the YAML has no split entries at all, the dataset folder is scanned for images directories
      • Semantic datasets substitute masks for images in the same way; the folder name is taken from the YAML’s masks_dir entry when present
    • Supported Dataset Types:
      • Detect (bounding boxes): importable as Rectangle or Polygon annotations
      • Segment (polygon masks): importable as Polygon or Rectangle annotations
      • Semantic (single-channel class-ID masks): imported as Mask annotations
    • Semantic Segmentation Specifics:
      • Masks must share the base name of their image (image1.jpg pairs with image1.png)
      • Only lossless mask formats are read (PNG, TIF, TIFF); JPEG compression would corrupt the class values
      • Each pixel value is the class ID, matching the index in the YAML’s names
      • Pixel value 255 is the reserved ignore label and is imported as background
      • A class named background at index 0 is treated as background, and no label is created for it
      • Classes unchecked in Classes to Import are left as background
• Export:
  • Labels:
    • Labels (JSON): Save label data to a JSON file
    • TagLab (JSON): Save label data to a TagLab JSON file
  • Annotations:
    • Annotations (JSON): Save annotation data to a JSON file
    • CoralNet (CSV): Save annotation data to a CoralNet CSV file
    • Viscore (CSV, JSON): Save annotation data in Viscore format
    • TagLab (JSON): Save annotation data to a TagLab JSON file
    • GeoJSON (JSON): Save annotations as GeoJSON for mapping software (only for GeoTIFFs with CRS and Transforms data)
    • Masks (PNG / BMP / TIF / RLE): Save annotations as segmentation mask images
      • Export Modes (choose one):
        • Semantic Segmentation (Integer IDs): Each class assigned a unique integer value (0-255)
          • Best for: Training semantic segmentation models
          • Background value configurable (0 = Background, 255 = Ignore Index)
          • Use 255 for sparse annotations (model ignores unlabeled areas), 0 for exhaustive labeling
        • Structure from Motion (SfM) Binary Mask: Binary foreground / background masks
          • Best for: 3D reconstruction software (Metashape, Agisoft, etc.)
          • Foreground value: 255 (objects to keep), Background: 0 (areas to ignore)
          • Preserves depth information for photogrammetry pipelines
        • Visualization (RGB Colors): Human-readable color-coded masks
          • Best for: Visual inspection, reports, presentations, qualitative analysis
          • Colors automatically assigned from your label definitions
          • Directly viewable without lookup tables
      • File Formats:
        • PNG (recommended): Lossless compression, fast loading, widely compatible
        • BMP: Uncompressed, larger files, maximum compatibility
        • TIF: Supports georeferencing for spatial data preservation
        • RLE (.txt): Run-length encoded text format, minimal storage for sparse masks
      • Georeferencing (TIF format only):
        • Preserve geographic CRS and transform from source images
        • Recommended for orthomosaics and geospatial workflows
      • Annotations to Include:
        • Mask Annotations: Base layer (manually painted masks)
        • Patches: Converted to circular regions
        • Rectangles: Filled rectangular regions
        • Polygons: Precise polygonal regions
        • Negative Samples: Export masks for images with NO annotations (useful for training)
      • Label Configuration:
        • Include / Exclude Labels: Select which labels to include in export
        • Layer Order: Control rendering order for overlapping annotations
          • Labels higher in list drawn first (behind), lower in list drawn last (on top)
          • Important for complex scenes with overlapping annotations
        • Mask Values (Semantic / SfM): Assign integer value (0-255) per label
        • Colors (Visualization): Automatically use label colors or customize
      • Output:
        • Creates a folder with exported masks (one per image)
        • Metadata file included (class_mapping.json or color_legend.json)
        • Handles images with no annotations based on your settings
  • Metadata:
    • Schema (YAML): Save metadata field definitions for reuse in another project
    • Data Table (CSV): Save one row per annotation, combining built-in metrics with custom fields
  • AI-Ready Datasets: Create a YOLO dataset for machine learning
    • Classify: Export classification dataset
      • Exports Patches, Rectangles, and Polygons as classified samples
      • Each annotation becomes a single classified crop
      • Useful for training image classification models
    • Detect: Export object detection dataset
      • Exports Rectangles and Polygons as bounding boxes
      • Creates YOLO detection-formatted annotations
      • Includes coordinate normalization for YOLO format
    • Segment: Export instance segmentation dataset
      • Exports Polygons as instance masks with per-object labels
      • Creates pixel-level segmentation masks
      • Each polygon becomes a labeled instance in the mask
    • Semantic: Export semantic segmentation dataset
      • Exports Mask annotations for semantic segmentation
      • Handles unlabeled / background regions
      • Useful for training pixel-wise segmentation models

    Export Dataset Features (all export types):

    • Train / Validation / Test Split: Configurable ratios (default 70 / 20 / 10)
      • Split by image for static images
      • Split by frame for video data
    • Label Selection: Filter by specific labels or export all
    • Video Frame Sampling: Optional frame stride to sample every Nth frame
    • Automatic Crop Generation: Crops are automatically generated and organized
    • Output Directory: Specify custom export location
    • Format Compliance: Exports follow YOLO format standards for compatibility

Utilities

• Sample Patches: Automatically generate Patch annotations
  • Sampling Method: Choose between Random, Stratified Random, or Uniform distribution
  • Number of Annotations: Specify how many annotations to generate
  • Annotation Size: Set the size of the generated patch annotations
  • Label As: Choose which label to assign to generated annotations
  • Exclude Regions: Option to prevent sampling in areas with existing annotations
  • Margins: Define image boundary constraints for sampling:
    • Set margins in pixels or percentage
    • Configure different values for top, right, bottom, and left edges
    • Annotations will only be placed within these margins
  • Select images by highlighting them in the ImageWindow
• Work Areas: Manage work areas for batch processing
  • Pre-compute and organize multiple work areas / tiles for selected images
  • Edge tiles: How tiles meet the right and bottom edges of the usable area
    • Skip partial tiles: Leave out tiles that would not fit whole
    • Shift inward (full coverage): Add full size tiles slid against the edge, overlapping their neighbors (the default)
    • Smaller edge tiles: Cut the last column and row smaller, so tiles overlap only by the overlap you set; with no overlap, every pixel is in exactly one tile, which is what merging extracted images back needs
  • Work areas can be saved as their own images with Extract Work Areas (see Extracted Work Areas)
  • Select images by highlighting them in the ImageWindow
• Morphological: Whole-image annotation operations, also opened with Ctrl + R in Select tool
  • Bake / Unbake tab: Convert between vector annotations and the mask (see Baking and Unbaking Annotations)
  • Overlaps tab: Resolve overlapping polygon annotations by label
    • Subtract overlap: Cut the overlapping labels out of the labels to change, leaving holes where needed
    • Remove overlapping: Delete annotations of the labels to change that overlap the overlapping labels
    • Merge same label: Combine overlapping annotations that share a label into one shape
    • Preview: Shows per image counts and selects the affected annotations on the current image, changing nothing
    • Double click a row: Opens that image and selects what the preview found on it, keeping the highlighted images
    • Apply: Makes the changes on every highlighted image as a single step, undone with Ctrl + Z
    • Polygons only: rectangles, patches and masks are left alone on both sides
  • Select images by highlighting them in the ImageWindow
• Set Image Scale: Calibrate pixel-to-distance conversion
  • Draw a reference line to establish the scale for measurements
  • Used by the Scale Tool for distance and area calculations

AI-Assist

• Segment Anything (SAM):
  • Deploy Predictor: Deploy SAM, EdgeSAM, MobileSAM, etc. to use interactively (points, box, bounding box)
    • Use the SAM Tool to add points or draw boxes for interactive segmentation
  • Deploy Generator: Deploy FastSAM to automatically segment the entire image
    • Automatically generate segmentations from loaded annotations
• See Anything (YOLOE):
  • Train Model: Train a custom See Anything (YOLOE) model from annotations
  • Deploy Predictor: Deploy a YOLOE model to use interactively within the same image
    • Use the See Anything Tool to draw boxes, type a phrase (Ctrl + T), or click detections to find more / fewer like them
    • Everything added is listed in the Prompt Session panel, grouped by where it came from; untick an example to try the prompt without it
    • Save / Load a session (.npz), or Inspect it to plot the examples and rank phrases against them (Text Alignment)
    • Send to Generator adds the session, with its confidence threshold and image size, to the Generator’s prompt
    • The Status box shows the loaded model, where the session’s examples came from, and the last action
  • Deploy Generator: Deploy a YOLOE model to use like a detector / segmentor across many images
    • Select the YOLOE model and parameters, then load the model
    • Build the prompt from any of:
      • Add from annotations: choose a label, highlight image(s) with rectangles or polygons, then press Add highlighted (one example per image)
        • Images already in the prompt are skipped unless their annotations changed
        • Remove all images from the prompt removes examples added from annotations, keeping tool examples and phrases
      • Add from Tool: the prompt built with the See Anything Tool (adding it twice is harmless)
      • Add phrase: a text prompt
    • Save detections as: the label detections are saved under; it can differ from the label of the annotations used as examples
    • The prompt list groups examples by source (annotations, tool, phrases, files); untick an example to leave it out
    • Save / Load the prompt (.npz only; legacy .pt VPE files are no longer supported)
    • Examples from annotations are re-embedded automatically when the image size changes; examples from the tool cannot be, and the panel warns
    • OK is enabled once a model is loaded, an output label is chosen, and the prompt has at least one enabled example
    • Use the loaded model with the prompt on new images or work areas
• Feature Selector:
  • Deploy Model: Deploy a dense feature extraction model for semantic similarity queries
    • Use the Feature Select Tool to click-to-query semantic similarity across images
    • Supports binary mode (single object) and multi-class mode (multiple classes)
• Active Learning: train on what you have already confirmed, then let the model propose the rest — annotate a little, train, review, repeat
  • Detect / Segment: each task keeps its own session, for as long as the application is open
  • Choose the labels to include and a model, then press Train Round. Training reads the project directly — no dataset is exported — and uses verified annotations only, so a round never trains on its own guesses
  • Browse to start from a model of your own, or from an earlier session’s best. Otherwise leave the default nano model: rounds are only useful if they are cheap enough to run often

  • After a round, predictions land on un-reviewed images and the Rasters Window is filtered to Needs Review. Predictions always arrive unverified

  • Review in the Annotation Gallery. This is the fastest way to work through a round, and it is worth setting up before your first one. Open Gallery, press Apply Filter, and sort by Confidence: the model’s least certain predictions group together at the top, which is exactly where your attention is worth most
    • Ctrl + Space confirms everything selected in one press — select a whole high-confidence bucket and accept it at once
    • Ctrl + Right-Click jumps to an annotation on the canvas when you need to see it in context; Delete removes a wrong one from there
  • You can also review from the session window: Previous / Next walk what is waiting, least confident first, opening each image and centring the annotation. Mark Verified confirms it; Mark as Review parks anything you cannot judge yet
  • An image with no annotations is left out of training: one nobody has annotated yet is not empty. To teach the model an image has nothing on it, highlight it in the Rasters Window (or open it) and press Mark Negative on the Session tab. It then trains as a background image, and rounds do not predict on it. Unmark Negative takes the mark off. Marks last for the session; New Session clears them
  • Image Budget: how many un-reviewed images a round predicts on. Most of it goes to images with nothing on them and the rest to images you have already worked on, so a round looks for new objects and checks itself where you are annotating
  • A round skips the image open on the canvas, images marked negative, and images already carrying predictions you have not reviewed; a round that finds nothing tells you which. Re-run Predictions predicts again at the current thresholds without training, which is the only way a threshold change takes effect before the next round. It reads the Image Budget as it stands, so raising the budget first reaches images the last round did not, and lowering it trims the least promising of its picks
  • Auto Train starts the next round on its own once every included label has gained the given number of newly confirmed annotations
  • A round’s model is kept only if it beat the best round before it, so predictions and Deploy Model always use the best model rather than the newest. The Rounds table reports mAP50 and mAP50-95, and Change is the column to read: it is blank when a round trained on different labels and is not comparable
  • PU Dataset (Detect only) trains the project as positive-unlabeled: regions you have not boxed are treated as unknown rather than as background, so the model is not taught to ignore objects nobody has reached yet. It usually lowers mAP while raising recall, so PU rounds are marked (PU) and only compared with other PU rounds. Best on medium or larger models at large image sizes; not available for RT-DETR, YOLOv10 or YOLO26
  • Stop ends a running round after the current epoch, keeping what it trained
  • New Session forgets the rounds and starts over, and offers to clear out round folders left on disk. Save Session copies the best model, its results and a summary of the session to a folder you choose — rounds are kept in a cache that prunes its own older weights, so this is how a model becomes something you keep
  • The Setup tab warns when your objects are too small for the image size, which is the one problem more annotating will not fix

Machine Learning

• Machine Learning:
  • Merge Datasets: Combine multiple datasets of the same task into a single dataset
    • Classify: Merge classification datasets
    • Detect: Merge detection datasets
    • Segment: Merge instance segmentation datasets
    • Semantic: Merge semantic segmentation datasets
    • Selecting Datasets: Choose each dataset by its data.yaml (Classify uses the dataset folder)
      • An adjacent class_mapping.json is picked up automatically
      • The Merged Classes table previews the combined class list and the ID each source dataset used for it
    • Class Resolution: Classes are matched across datasets by name, not by ID
      • The same ID usually means a different class in each dataset, so every annotation is rewritten to the merged numbering
      • Classification datasets merge by class folder name; no file contents change
    • Filename Collisions: Images sharing a name are renamed rather than overwritten, and their label or mask file is renamed to match
    • Semantic Specifics:
      • A class named background is placed at ID 0, and pixel value 255 stays the reserved ignore label
      • Pixels holding a class that no dataset contributes are set to ignore
    • Output: data.yaml with the merged class list, and class_mapping.json combining the label colors and codes from every source dataset
  • Tile Dataset: Split datasets into tiles for large-image processing
    • Classify: Tile classification datasets
    • Detect: Tile detection datasets
    • Segment: Tile segmentation datasets
    • Semantic: Tile semantic segmentation datasets
  • Pre-Train Model: Pre-train model encoder using self-supervised learning
  • Train Model: Train a machine learning model
    • Classify: Train classification model
    • Detect: Train object detection model
      • PU Dataset: set True when the dataset is only partly annotated (real objects left unboxed). Unboxed regions the model is fairly sure about are dropped from the loss instead of being taught as background. Mosaic is turned off once it is active, and the best-recall epoch is saved as weights/recall_best.pt beside best.pt
    • Segment: Train instance segmentation model
    • Semantic: Train semantic segmentation model
  • Evaluate Model: Evaluate a trained model
    • Classify: Evaluate classification model performance
    • Detect: Evaluate detection model performance
    • Segment: Evaluate segmentation model performance
    • Semantic: Evaluate semantic segmentation model performance
  • Optimize Model: Convert model format for deployment (TensorRT, ONNX, etc.)
  • Deploy Model: Make predictions using a trained model
    • Classify: Deploy for classification inference
    • Detect: Deploy for detection inference
    • Segment: Deploy for segmentation inference
    • Semantic: Deploy for semantic segmentation inference
      • Generates pixel-wise segmentation masks for each class
      • Convert Masks to Polygons: Automatically trace predicted masks into polygon annotations
        • Converts raster mask output to vector polygon format
        • Creates one polygon per connected region per class
        • Polygons are editable after creation for refinement
        • Useful for precise boundary delineation
        • Configurable simplification level for polygon complexity
  • Batch Inference: Run inference on multiple selected images or a single video
    • Input Options: Select multiple images OR one video file (not both, not multiple videos)
    • Type: Standard runs on whole images; Tiled runs only on each raster’s work areas (for example a tile grid from the Work Area Manager), and rasters without work areas are skipped
    • Live Mode: Display inference results in real-time as the model processes
    • Save Annotations: Optional toggle to save generated annotations to the project
      • When disabled: Preview mode shows results without persisting to the project
      • When enabled: Annotations are added to the project for each image / frame
    • Feature Options (Feature model): choose an Action
      • Extract Feature Maps: Cache a dense feature map for each selected image
      • Densify Patches: Fill each image’s mask from its existing labels (patches, polygons and mask classes seed a feature map; Review patches are ignored)
        • Type: Standard densifies the whole image; Tiled densifies only each work area, at full input resolution, and skips rasters without work areas (required for orthomosaics; create a tile grid with the Work Area Manager, ideally with at least 20% overlap). Prototypes are shared across all of an image’s work areas, and each pixel in an overlap takes the work area it sits most centrally in
        • Reject Threshold: Pixels less similar than this to every class stay unlabeled (0, the default, labels every pixel)
        • Minimum Classes: Skip images whose labels seed fewer classes
        • Only Fill Unlabeled Pixels: Keep pixels the mask already labels
        • Replace Matching Patches: Delete each patch whose label the new mask holds at its center (cannot be undone)

CoralNet

• CoralNet:
  • Authenticate: Authenticate with CoralNet
    • Enter your CoralNet username and password to access your sources
    • Authentication is required before downloading any CoralNet data
  • Download: Download data from CoralNet
    • Source ID: Enter the Source ID (or multiple IDs separated by commas)
    • Output Directory: Select where to save downloaded files
    • Download Options: Choose what to download:
      • Metadata: Source information and settings
      • Labelset: All available labels from the source
      • Annotations: Point annotations with their labels
      • Images: Original images from the source
    • Parameters: Configure download settings:
      • Image Fetch Rate: Time between image downloads (seconds)
      • Image Fetch Break Time: Pause duration between batches (seconds)
    • Debug Mode: Toggle headless browser mode for troubleshooting

Explorer

The Explorer is a dual-window system for browsing and analyzing annotations using embeddings (high-dimensional feature vectors). It enables exploration of annotations through two linked views: a gallery viewer and an embedding visualization, with real-time synchronization and feature caching.

Annotation Gallery Window

The gallery displays annotation crops as a scrollable grid of thumbnail images. It includes:

Filtering & Sorting

• Filter… Button: Opens the filter dialog, one tab per filter: Images, Labels, Types and Status (verified / unverified). Its tooltip lists what each filter is set to
  • Every row starts selected, and only annotations in the selected rows pass: click, Ctrl + click, Shift + click or drag to select, Ctrl + A for every row shown. A tab with every row selected also lets through images or labels added later
  • Selected rows within a tab are OR’d; tabs are AND’d
  • The Annotations column is how many annotations each row would match given the other tabs; click a column header to sort by it. The footer shows the total match and how many crops are still needed
  • Search accepts * and ? wildcards; Enter in the search box selects every row shown, and rows selected under an earlier search stay selected
  • OK (Ctrl + Enter) only sets the filter
• Apply Filter Button: Shows the filter that is set (gallery remains in placeholder until applied). It turns bold when the filter set differs from the one shown
• Clear Button: Reset gallery and return to placeholder state

Sorting Options

• None: Display in current order without grouping
• Label: Group by label, sorted by confidence within each label
• Image: Group by image filename
• Confidence: Group by confidence buckets (0-10%, 10-20%, … 90-100%, plus Verified)
• Area: Sort by annotation area (smallest to largest)
• Cluster: Sort by K-Means clustering results from Embedding Viewer (enabled when clusters available)
• Similarity: Sort by distance to the Embedding Viewer’s last similarity anchor (enabled after a neighbour expansion)
  • Anchors are grouped first, then bands of cosine similarity, with unranked annotations last

Display Controls

• Isolate Selection: Show only selected annotations (double-click empty space to exit)
• Size Adjustment: Ctrl + Mouse Wheel to resize gallery thumbnails (32-256px range)
• Sticky Headers: Automatically highlights the current group header as you scroll

Selection & Navigation

• Left-Click: Select a single annotation
• Ctrl + Left-Click: Add / remove from selection
• Shift + Left-Click: Select range between two annotations
• Ctrl + Drag: Box-select multiple annotations
• Ctrl + Right-Click: Navigate to annotation in Annotation Window (zoomed / centered), leaving the current selection intact
• Double-Click: Unselect all, exit isolation view
• Ctrl + A: Select all annotations in current filtered view
• Ctrl + Space: Confirm selected annotations with top machine confidence prediction

Embedding Viewer Window

The embedding viewer displays annotations as points in 2D or 3D space, where proximity indicates semantic similarity. This enables:

Feature Extraction & Dimensionality Reduction

• Model Category: Choose Color Features (HSV-based), YOLO, Transformer, TIMM, OpenCLIP, or Live Models (deployed models)
• Model Selection: Pick specific model from selected category
• Embedding Technique: PCA (fast, linear), LDA (class-aware), TSNE (nonlinear, slow), UMAP (fast nonlinear)
• Dimensionality: 2D (faster) or 3D (interactive rotation)
• Apply Embeddings Button: Extract features and run dimensionality reduction pipeline (shows progress)

Advanced Embedding Settings (expandable “Advanced” panel)

• PCA before reduction: Apply PCA dimensionality reduction before TSNE / UMAP / LDA (auto-disabled for PCA)
• PCA components: Number of intermediate PCA dimensions (default 50; reduces high-dim features before final reduction)
• UMAP settings:
  • n_neighbors: Number of neighbors for UMAP (2-150; default 15; higher = more global structure)
  • min_dist: Minimum point separation in UMAP output (0.00-0.99; lower = tighter packing)
• TSNE settings:
  • Perplexity: Effective neighborhood size (5-50; default 20)
  • Exaggeration: Cluster separation strength (5.0-60.0; default 5.0; higher = more distinct clusters)

Clustering (K-Means)

• Cluster Button: Run K-Means on current embedding to partition points into clusters
• K value: Number of clusters (2-50; default 3)
• Cluster Space: Cluster on 2D projected coordinates or full high-dimensional feature vectors
• Clear Button: Remove cluster boundaries and disable Cluster sort option
• Cluster results are colored and can be sorted via the Annotation Viewer’s “Cluster” sort option

Display & Navigation

• Display Mode Toggle: Switch between dots and sprite / thumbnail view
• Locate Button: Show crosshair pointer to selected annotation
• Center Button: Pan and zoom view to center on selected point(s)
• Home Button: Reset view to fit all points
• Magnifier Lens: Annotations nearest the cursor are always drawn as thumbnails, the rest stay dots
  • Thumbnail size follows the current point / sprite size; the lens radius tightens as thumbnails grow

Interactive Controls

• Left-Click: Select an annotation
• Ctrl + Left-Click: Add / remove from selection
• Ctrl + Left-Click + Drag: Box-select multiple points
• Right-Click + Drag: Pan the view
• Mouse Wheel: Zoom in / out
• Ctrl + Wheel: Adjust point size
• Double-Click: Unselect all, exit isolation mode
• Ctrl + Shift + Wheel: Grow / shrink the selection by similarity to the selected annotation(s)
  • Ranking uses the full feature vectors, falling back to the 2D projection when features are unavailable
  • An overlay reports the neighbour count and the cosine cutoff it implies
  • Escape: Restore the selection the expansion started from
• Ctrl + Right-Click: Navigate to the annotation in the Annotation Window (zoomed / centered), leaving the current selection intact

Mislabel Detection & Analysis

• Outlier detection is performed manually by examining points that are visually distant from their label cluster in the embedding space
• Points far from their label cluster can be selected and examined in detail for potential mislabels
• Isolate Selection: Show only selected points (double-click empty space to exit)
  • Useful for examining specific clusters or suspicious regions in isolation

Feature Caching System (CacheManager)

The Explorer automatically caches extracted features to accelerate re-loading the same models:

How Caching Works

• Features are stored in .cache/embedding/ directory (created automatically)
• Metadata stored in SQLite database (manager.db)
• Feature vectors stored in FAISS indexes (one per model, enabling fast similarity search)
• Cache is model-specific: switching models or changing extraction parameters invalidates the cache

Cache Behavior

• First time running a model: features extracted and automatically cached
• Second time with same model: features loaded from cache (much faster)
• Deleting annotations: cache entries removed automatically
• Changing model / settings: stale cache detected and rebuilt as needed

Multi-Model Support

• Cache supports multiple models simultaneously
• Each model has its own FAISS index file (features_{model_key}.faiss)
• Switching between models uses corresponding cached features

Performance

• Color Features: 70 dims (8×8 Hue-Saturation histogram + H / S / V moments), instant CPU extraction
• YOLO / Transformer / TIMM / OpenCLIP: Model-dependent feature dimensions, GPU-accelerated (if CUDA available)
• Batch Processing: Default batch size 512 annotations; cache flush every 4 batches to balance memory and I/O
• Thread-Safety: Concurrent access via SQLite locks prevents cache corruption during multi-threaded operations
• Feature Storage: Retained feature matrices are reduced to 128 dimensions for ranking and clustering; the cache keeps the full-width vectors

Tips

• Use dual monitors: place Annotation Viewer and Annotation Window side-by-side to review selected annotations
• For exploring mislabels: use Embedding Viewer to identify visual outliers (points far from their label cluster), isolate them, and examine in the gallery
• Larger clusters often indicate common features; smaller / scattered points may be edge cases or mislabels
• Apply filter in Annotation Viewer before running embeddings to focus on specific images / labels
• Use Cluster sort in the Annotation Viewer after running K-Means to organize results by cluster membership
• To gather more of a class: select a known-good annotation, expand by similarity, then switch the Annotation Viewer to Similarity sort and trim the tail where the crops stop matching

Tool Bar Tools

• Select Tool: After selecting the tool
  • Left-Click: Select an annotation drag to move it
  • Ctrl + Left-Click: Add / remove annotation to current selection
  • Ctrl + Delete / Backspace: Remove selected annotation(s)
  • Ctrl + Drag: Create rectangle selection to select multiple annotations
    • Hold still for 2 seconds to show the rectangle’s size (pixels, and real-world units if the image has a scale) at its bottom-left corner; moving hides it
  • Ctrl + Mouse Wheel: Change size of the selected annotation
  • Resize handles: Appear automatically whenever a single annotation is selected; they stay subdued until the cursor comes near, then grow and light up. Drag one to reshape; the cursor shows which way it moves
  • Ctrl + Shift: Show every vertex at full strength (dense polygons normally thin their handles until you zoom in)
  • Ctrl + Shift + Mouse Wheel: Change the number of vertices for a polygon annotation (each tick up doubles the count, so it stops at 2000 vertices; scroll down to simplify first)
  • Ctrl + Space: Confirm prediction for selected annotation with top machine confidence
  • Ctrl + X: Cut a polygon annotation, explode a multi-polygon annotation, or subtract polygon annotations
    • Cutting Rules:
      • Only a single annotation can be selected
      • Press Ctrl-X to enter cutting mode, Left-Click to start, draw line, Left-Click to end
      • Hold Ctrl to create straight line segments
      • Press Backspace or Delete to cancel the current cutting line without making changes
    • Exploding Rules:
      • Select a MultiPolygonAnnotation, press Ctrl + X to explode it into multiple PolygonAnnotations
    • Subtraction Rules:
      • Multiple overlapping annotations must be selected one-by-one
      • The first annotations will be used as the cutters, the last polygon will be used as the base
  • Ctrl + C: Combine each overlapping cluster of selected annotations; anything that doesn’t overlap another selected annotation is left alone
    • Combining Rules:
      • Only annotations within the same overlapping cluster need the same label and to be verified (not machine predictions) – a cluster that fails this check is left untouched rather than blocking the rest
      • RectangleAnnotations can only be combined with other rectangles
      • A MultiPolygonAnnotation is only broken apart if one of its islands actually overlaps something outside it; otherwise it’s left intact
      • PatchAnnotations always combine with the full selection at once (same as Ctrl + Shift + C below), since they can be combined with other patches or polygons to create a polygon
  • Ctrl + Shift + C: Combine the entire selection into a single result, regardless of overlap
    • Same label/verified requirement, but applied to the whole selection at once
    • Use this to deliberately build a MultiPolygonAnnotation out of several non-overlapping polygons
• Scale Tool: Provide scale to the image(s), and measure distances on the current image.
  • Left-Click to set the starting point.
  • Drag to draw a line, then Left-Click again to set the endpoint.
  • Press Backspace to cancel drawing the scale line.
  • The scale will be calculated based on the known provided length and pixel length.
  • Area and Perimeter for an annotation can be viewed when hovering over the Confidence Window.
  • Preferred units can be set in the Status Bar.
• Patch Tool: After selecting the tool
  • Left-Click: Add a patch annotation at the clicked position
  • Ctrl + Mouse Wheel: Adjust the patch size up or down
  • Mouse Movement: Shows a semi-transparent preview of the patch at the cursor position
  • Ctrl + 1: Enable Live Classification mode (requires a loaded classification model)
    • Hover to see real-time predicted labels and confidence scores
• Rectangle Tool: After selecting the tool
  • Left-Click: Start drawing a rectangle click again to finish
  • Mouse Movement: Shows a preview of the rectangle while drawing
  • Backspace: Cancel the current rectangle annotation
• Polygon Tool: After selecting the tool
  • Left-Click (first): Start drawing a polygon
  • Left-Click (subsequent): Add points to the polygon click near the first point to close
  • Ctrl + Left-Click: Enable straight line mode click to add straight line segments
  • Mouse Movement: Shows a preview of the polygon as you draw
  • Backspace: Cancel the current polygon annotation
• Brush Tool: After selecting the tool
  • Left-click and drag to paint brush strokes on the canvas.
  • Hold Ctrl and use the Mouse Wheel to adjust brush size.
  • Press Ctrl + Shift to switch between a circle and square brush shape.
  • A semi-transparent preview shows the brush stroke while drawing.
• Erase Tool: After selecting the tool
  • Left-click and drag to erase pixels.
  • Hold Ctrl and use the Mouse Wheel to adjust eraser size.
  • Press Ctrl + Shift to switch between a circle and square eraser shape.
  • Press Ctrl + (Backspace or Delete) to clear the mask annotation on the current image.
  • A semi-transparent preview shows the eraser while drawing.
• Dropper Tool: After selecting the tool
  • Left-click on a mask annotation region to select the associated label.
• Fill Tool: After selecting the tool
  • Left-click to fill the region under the cursor with the selected label.
• SAM Tool: After a model is loaded
  • Left-Click: Start drawing a work area click again to finish drawing
  • Backspace: Cancel drawing the current work area
  • Space: Create a work area from the current view
    • Space: Set working area confirm prediction finalize predictions and exit working area
    • Left-Click: Start a box press again to end a box
    • Ctrl + Left-Click: Add positive point
    • Ctrl + Right-Click: Add negative point
    • Backspace: Discard unfinalized predictions
• See Anything (YOLOE) Tool: After a model is loaded
  • Left-Click: Start drawing a work area click again to finish drawing
  • Backspace: Cancel drawing the current work area
  • Space: Create a work area from the current view
    • Space: Set working area run prediction finalize predictions and exit working area
    • Left-Click: Start a box press again to end a box
    • Ctrl + T: Set a text prompt (leave empty to clear it)
    • Ctrl + Left-Click: Add the detection under the cursor as a positive example (more like this)
    • Ctrl + Right-Click: Remove the detection and add it as a negative example (fewer like this)
    • Ctrl + Shift + Right-Click: Drop / restore a single detection (hold Ctrl + Shift to reveal dropped ones)
    • Ctrl + Mouse Wheel: Adjust the confidence threshold (add Shift for finer steps)
    • Ctrl + Shift + Backspace: Clear the prompt session
    • Backspace: Discard unfinalized predictions
• Feature Select Tool: After a feature model is deployed
  • Space (or Left-Click, Left-Click): Define a work area first
  • Binary Mode (one object):
    • Ctrl + Left-Click: Add a positive prototype
    • Ctrl + Right-Click: Add a negative prototype
    • Ctrl + Mouse Wheel: Adjust the similarity threshold
    • Colormap dropdown and opacity slider control the heatmap overlay
  • Multi-class Mode (one blob per label):
    • Ctrl + Left-Click: Assign the patch to the selected label (switch labels to add more classes)
    • Ctrl + Right-Click: Undo that label’s last point
    • Ctrl + Mouse Wheel: Adjust the reject threshold (off by default, so every pixel gets a label)
    • Preview is colored per label and tracks the annotation transparency slider
  • Common Controls:
    • Ctrl + Alt: Toggle between Binary and Multi-class mode
    • N: Refresh the yellow crosshair point suggestion (the patch the current points are least sure about)
    • Space: Finalize and create a Polygon / Mask annotation
    • Shift + Space: Finalize to the Mask and delete the patches it agrees with (same label at the patch center), so the mask fills them in. Undo restores both.
    • Backspace: Clear all points and start over
• Work Area Tool: For creating restricted areas for model prediction
  • Left-Click: Start drawing a work area click again to finish drawing
  • Backspace: Cancel drawing the current work area
  • Space: Create a work area from the current view
  • Ctrl + Shift: Show removal buttons on existing work areas (click the “X” to remove)
  • Ctrl + Shift + Backspace: Remove all work areas in the current image
  • Practical Use:
    • Define specific regions where models should make predictions
    • Useful for processing only relevant parts of large images
    • Work areas persist between tool changes and sessions

Status Bar

• Image Size: Displays the image size
• Cursor Position: Displays the cursor position
• View Extent: Displays the view extent
• Annotation Visibility: Show / Hide all existing annotations
• Annotation Transparency: Adjust the annotation transparency
• Scale: Displays the scale dimensions
  • Provides a dropdown to select preferred units (mm, cm, m, km, etc.,)
  • Enabled when a scale is set using the Scale Tool or imported from an image
• Z: Displays the Z-dimension
  • Provides a dropdown to select preferred units (mm, cm, m, km etc.,)
  • Enabled when a Z-channel for the image is imported
  • Select a color map in the dropdown to overlay Z-channel
  • Click dynamic range button to enable dynamic recoloring of Z-channel
• Patch Size: Manipulate Patch Size (only active when using Patch Tool)
• Parameters: Adjust parameters including uncertainty, IoU, and area thresholds

Annotation Window

• Zoom: Use the Mouse Wheel to zoom in and out
• Pan: Right-Click and hold the Mouse Button to pan the image
• Rotate: Ctrl + Right-Click + Drag
  • Drag left / right to rotate; drag up / down to adjust rotation speed
  • Useful for examining annotations at different angles

Baking and Unbaking Annotations

Both live in the Morphological Operations dialog, opened with Ctrl + R in Select tool or from Utilities > Morphological. Its Overlaps tab resolves overlapping polygons by label; see Utilities above.

Baking and Unbaking allows converting between vector annotations and mask annotations:

• Bake: Converts vector annotations (patches, rectangles, polygons) into a mask annotation
  • Converts the geometric outlines into rasterized pixel-based regions
  • Useful for combining multiple overlapping annotations into a single mask
  • Accessed via Ctrl + R in Select tool (opens a dialog)
  • Vector annotations are rasterized and added to the mask
• Unbake: Converts mask regions back into vector polygon annotations
  • Traces mask pixel regions into polygonal outlines
  • Useful for converting mask-based predictions into editable vector form
  • Accessed via Ctrl + R in Select tool (opens a dialog)
  • Each connected mask region becomes a separate polygon annotation
  • Every region comes back as a polygon, whatever its shape: a baked rectangle or patch returns as a polygon

Video Playback Controls

When a video is loaded, additional playback controls appear:

• Play / Pause: Toggle video playback
• Stop: Stop playback and return to paused state
• Step Back / Forward: Move one frame at a time (disabled during playback)
• Jump to First / Last Frame: Jump to the beginning or end of the video
• Jump to Previous / Next Annotated Frame: Navigate between frames with annotations
• Jump to Previous / Next Keyframe: Navigate between marked keyframe frames
• Seek Slider: Scrub through video frames with visual feedback
  • Red tick marks indicate frames with annotations
  • Gold tick marks indicate keyframe frames
  • Ctrl + Hover: Preview frame at cursor position
  • Ctrl + Click: Jump to frame directly
• Keyframe Toggle: Star button to mark / unmark the current frame as a keyframe
  • Keyframes appear as gold tick marks on the slider
  • Useful for marking important frames for review or processing
• Frame Counter: Displays current frame number and total frame count

Labels Window

• Move Label: Right-Click and drag to reorder labels in the window
• Keyboard Stays on the Canvas: Clicking a label leaves the keyboard with the Annotation Window, so hotkeys like Space and Ctrl + C work straight after
• Label Visibility: Each label has a checkbox to show / hide annotations of that label
  • Hidden labels retain their data and can be shown again at any time
  • Transparency slider affects all labels (both visible and hidden)

Label Management Toolbar

Action buttons for label operations:

• Add Label (+ icon): Create a new label with custom name and color
• Delete Label (- icon): Delete the selected label (disabled if no selection)
• Edit Label / Merge Labels: Modify the selected label’s name / color, or merge multiple labels into one (disabled if no selection)
• Map Labels (bulk): Perform bulk mapping to merge multiple source labels into a single target label (disabled if no selection)
• Lock Label (lock icon): Lock the currently selected label to prevent accidental modifications
  • Locked labels remain locked until manually unlocked
• Toggle All (asterisk icon): Quickly toggle visibility of all labels
  • If all labels visible: clicking hides all
  • If any labels hidden: clicking shows all
  • Fast batch operation for many labels

Filter & Search

• Filter Labels: Type in the filter box to search for specific labels by name
• Results update in real-time as you type

Label & Annotation Counts

• Label Count: Displays total number of labels in the project
• Annotation Count: Shows annotation statistics
  • When no annotation selected: Total annotation count
  • When one annotation selected: Its index in the sequence
  • When multiple annotations selected: Number of selected annotations
  • Editable: Click and edit to navigate to a specific annotation by index (when in Select mode)

Metadata Window

The Metadata Window is a property grid showing structured metadata for the currently selected annotation(s). It is docked on the right, tabbed with the Confidence Window, and updates with the selection made in the Annotation Window or the Explorer.

• Selection Driven: Displays the annotation(s) currently selected; the grid is read-only context until a selection exists
• Vector Annotations Only: Patches, Rectangles, Polygons, and MultiPolygons; Mask annotations are excluded (a mask is a per-image raster, not a discrete object)
• Multiple Selection: Fields can be edited across many annotations at once; a single edit writes to every selected annotation

Field Groups

The grid is organized into three collapsible groups, each showing a count of its rows:

• Custom: User-defined typed fields governed by the project’s metadata schema
  • Editable, including across a multi-selection
  • Shows <multiple values> when the selected annotations disagree; viewing a mixed selection never overwrites anything
• Built-in: Read-only values derived from the annotation’s geometry and its raster
  • Identity: Annotation ID, Annotation Type, Label, Full Name
  • Confidence: User Confidence, Machine Confidence, Verified
  • Source: Source Image, Cropped Dimensions
  • Geometry: Area, Perimeter, Scale
  • Morphology: Length, Width, Orientation, Hull Area, Hull Perimeter
  • 3D (requires a Z-Channel): Volume, 3D Surface Area, Z Coverage
  • Recomputed on every refresh, so they can never go stale against the geometry they describe
  • Expressed in the unit selected in the Status Bar; a note is shown when a scale unit cannot be converted to the display unit
• Raw Data: Read-only leftovers from annotation.data that could not be promoted into a typed field (nested structures such as lists and dictionaries)
  • With multiple annotations selected, only the keys shared by all of them are listed

Group expansion is remembered as you move between annotations, and computed rows display <multiple annotations selected> when more than one annotation is selected.

Field Management Toolbar

Action buttons for schema operations:

• Add Field (+ icon): Define a new field for the project
  • Name: Unique key for the field, used as the column name in exports
  • Display Name: Optional label shown in the panel (defaults to the name)
  • Description: Optional text shown as a tooltip on the field row
  • Type: Short Text, Long Text, Yes / No, Whole Number, Decimal Number, Dropdown (one), Checklist (many), or Date
  • Options / Range / Default: Only the settings that apply to the chosen type are shown
• Delete Field (- icon): Delete the selected field and its stored values across the project (disabled if no field selected)
  • Confirmation reports how many annotations hold a value for the field
  • A progress bar appears for projects with more than 200 annotations
• Edit Field: Modify the selected field’s name, type, or settings (disabled if no field selected)
  • Renaming migrates stored values to the new name
  • Changing type re-coerces stored values; values that do not fit the new definition are discarded, and the confirmation appears only when the change actually costs data
• Show / Hide Fields (eye icon): Choose which custom fields the grid shows, and in what order
  • Dual transfer list: move fields between Hidden and Visible, and use the arrows to set display order
  • Hiding affects display only; stored values are kept and still saved with the project
  • Built-in and Raw Data rows are always shown and are not listed here
• Adopt Fields (magic icon): Promote imported data into real typed fields
  • Scalar keys in annotation.data become typed fields, with the type inferred from every value observed
  • Known importer keys (TagLab instance / blob names and notes, Viscore Dot, Reprojection Error, View Index, View Count) adopt a pre-defined type and label
  • Nested values are left behind in the Raw Data group
  • Keys duplicating a built-in value (bbox, centroid, area, perimeter, class name) are dropped rather than stored as a stale copy
  • Importers run this automatically after an import; the toolbar button re-runs it on demand

Filter & Search

• Filter Fields: Type in the filter box to search rows by name across all three groups
• Results update in real-time as you type

Editing Behavior

• Each field type maps to its own editor (line edit, text box, checkbox, spin box, dropdown, checklist, or date picker)
• Edits are committed on a short delay, so holding a key does not trigger one write per keystroke across a large selection
• Pending edits are flushed automatically when the selection changes, when the dock is hidden, and before any field dialog opens
• The annotation selection is cleared when a field dialog opens, since Backspace and Delete would otherwise delete the selected annotation
• Changing the display unit in the Status Bar re-renders the Built-in values in place

Schema Import & Export

The schema (the field definitions) travels separately from the values:

• Import: File > Import > Metadata merges field definitions from a YAML file
  • Merging is by field name; existing definitions are never silently replaced
  • On a collision, choose Replace (overwrite definitions, keeping values that still fit) or Keep (import only the new fields)
• Export Schema: File > Export > Metadata > Schema (YAML) writes the definitions only, as a reusable template for another project
• Export Data Table: File > Export > Metadata > Data Table (CSV) writes one row per annotation
  • Combines built-in metrics computed from geometry with the custom fields you defined
  • Columns: Select from Location, Size, Shape, and 3D metrics, plus your custom fields
  • Filters: Restrict the export by Images, Labels, and Annotation Types
  • Identity columns (annotation id, image name, label codes, color) are always included

Rasters Window

• Select Image: Double-Click on a row to load the image in the annotation window
• Highlight Image: Single-Click on a row to highlight it
  • Ctrl + Left-Click: Select multiple, non-adjacent rows
  • Shift + Left-Click: Select multiple, adjacent rows

Filtering & Search

Multi-select filters and search bars to control which images are displayed:

• Filters (select multiple to combine):
  • Highlighted: Show only highlighted images
  • Image: Show standard image files
  • Ortho: Show orthomosaic GeoTIFF files
  • Video: Show video files
  • Has Z-Channel: Show images with Z-channel data
  • Has Predictions: Show images with model predictions
  • Has Annotations: Show images with annotations
  • No Annotations: Show images without annotations
• Search Images: Filter by filename or file path substring
• Search Labels: Filter by annotation label name (shows images containing that label)

Information & Navigation

• Current Image: Displays the currently selected image in the annotation window
• Highlighted Count: Shows how many images are highlighted
• Total Count: Shows total images in the current filtered view
• Home Button: Click to center the table on the currently selected image
• Toggle Highlighted Button: Quickly highlight all filtered images or unhighlight if all are highlighted

Context Menu

Right-Click on one or more highlighted rows to:

• Check / uncheck highlighted rows
• Batch Inference: Run inference using loaded models (opens Batch Inference Dialog)
• Extract Frames: Extract frames from a single highlighted video (opens Extract Frames dialog)
• Work Areas (images and orthomosaics; see Extracted Work Areas):
  • Extract Work Areas: Save each work area as its own image; shown greyed out with the reason, such as “no work areas” or “already extracted”, when none of the highlighted rows can be extracted
  • Merge Back: Bring a set of extracted images’ annotations back onto the original, one set at a time
  • Highlight Extracted Images: Highlight the images of the set
  • Unlink Extracted Images: Release the original and keep the images as ordinary images
• Z-Channel (not shown for videos):
  • Import: Import Z-channel data for highlighted rows (opens Z-channel import dialog)
  • Remove: Remove Z-channel data, shown when a highlighted row has one
• Features: Remove dense feature maps, shown when a highlighted row has one
• Delete Annotations: Remove all annotations from highlighted rows
• Delete Images: Remove images and associated annotations from the project

Image Preview

• Tooltips: Hover over a row to show image metadata (path, dimensions, annotation count, etc.)
• Thumbnail: Hold Ctrl while hovering over a row to show a preview thumbnail

Confidence Window

• Display Cropped Image: Shows the cropped image of the selected annotation
  • The dimensions shown include both original and scaled sizes when applicable
  • The border of the image is highlighted with the color of the top confident label
• Confidence Chart: Displays a bar chart with confidence scores
  • Top 5 Predictions: Shows up to 5 predictions with their confidence scores
  • Prediction Selection: Click on any confidence bar to change the annotation’s label, verifying it
  • Numerical Keys: Press keys 1-5 to quickly select from the top 5 predictions (Select tool; no need to click the window first)
  • Prev / Next buttons: Cycle through annotations in order of their creation
• Keyboard Stays on the Canvas: Clicking in this window leaves the keyboard with the Annotation Window, so tool hotkeys keep working
• Confidence Mode Toggle:
  • Click the icon button next to the dimensions to toggle between user and machine confidence views
  • User icon shows user-assigned confidence scores
  • Machine icon shows model-predicted confidence scores
  • The toggle is only enabled when both user and machine confidences are available
• Visual Indicators:
  • Each confidence bar shows the label color and confidence percentage
  • Numbered indicators (1-5) show the rank of each prediction
  • Hover over confidence bars to see a pointing hand cursor when selection is possible
• Tooltips: Hover over the window while an annotation is selected to see its metadata

Hotkeys

Tips & Quick Reference

• Hover over any tool button, image row, or window to see helpful tooltips explaining functionality
• Many controls use Ctrl as a modifier key for enhanced functionality
• Work Area and SAM / YOLOE tools use Space as a quick confirm key
• Interactive hotkey map available at Hotkeys Map with visual keyboard layout

General

• Escape: Exit the program
• Ctrl + S: Save the project
• Ctrl + Z: Undo the last addition or deletion of an annotation
• Ctrl + Shift + Z: Redo the previously undone addition or deletion of an annotation

Navigation

• Alt + Up / Down: Cycle through images
• Ctrl + Up / Down: Cycle through labels
• Ctrl + Left / Right: Cycle through annotations

Selection & Annotation Editing

• Ctrl + A: Select all annotations, unselect all annotations (press twice to toggle)
• Ctrl + Left-Click: Add / remove annotation to current selection
• Ctrl + Drag: Create rectangle selection to select multiple annotations
• Ctrl + Delete / Backspace: Remove selected annotation(s)
• Ctrl + Space: Confirm prediction for selected annotation with top machine confidence
• Ctrl + Mouse Wheel: Resize selected annotation or patch preview
• Ctrl + Shift + Mouse Wheel: Change polygon vertex count
• Ctrl + X: Cut a polygon annotation, explode a multi-polygon, or subtract polygons
• Ctrl + C: Combine each overlapping cluster of selected annotations (same type and label); non-overlapping ones are left alone
• Ctrl + Shift + C: Combine the entire selection into a single result regardless of overlap (e.g. to build a MultiPolygonAnnotation)
• Ctrl + Shift: Show every vertex handle at full strength (handles themselves appear on selection)
• Ctrl + R: Open the Morphological Operations dialog (bake vectors into the mask, unbake the mask into vectors, or subtract, remove and merge overlapping polygons by label; Select tool must be active)
• Backspace / Delete: Cancel current drawing (rectangle, polygon, work area, cutting line)

Tool Control

• Ctrl + Alt (either order; holding does not repeat): Switch between tools within the existing tool group
  • Switch from annotation tool to SelectTool
  • Switch from SelectTool back to the tool you came from (e.g., SAM), or else the tool for the selected annotation’s type (e.g., PatchTool when PatchAnnotation selected)
  • Toggle between Brush and Erase tools
  • Toggle Feature Select Tool binary / multi-class mode

Work Areas

• Space: Create a work area from the current view (or define via left-click twice)
• Ctrl + Shift: Show removal buttons on existing work areas
• Ctrl + Shift + Backspace: Remove all work areas in the current image
• Backspace: Cancel work area drawing

Patch Tool

• Ctrl + 1: Enable Live Classification mode (requires loaded classification model)
  • Hover to see real-time predicted labels and confidence scores

SAM / YOLOE Tools

• Space: Set work area, run prediction, finalize, and exit
• Backspace: Discard unfinalized predictions
• Ctrl + Left-Click: Add positive point (SAM Tool) / add detection as a positive example (YOLOE Tool)
• Ctrl + Right-Click: Add negative point (SAM Tool) / remove detection and add it as a negative example (YOLOE Tool)
• Ctrl + T: Set a text prompt (YOLOE Tool)
• Ctrl + Mouse Wheel: Adjust the confidence threshold (YOLOE Tool)
• Ctrl + Shift + Right-Click: Drop / restore a single detection (YOLOE Tool)
• Ctrl + Shift + Backspace: Clear the prompt session (YOLOE Tool)

Feature Select Tool

• Space (or Left-Click, Left-Click): Define work area first
• Ctrl + Alt: Toggle between Binary and Multi-class mode
• Ctrl + Left-Click: Add positive prototype (binary) / assign patch to selected label (multi-class)
• Ctrl + Right-Click: Add negative prototype (binary) / undo last point for label (multi-class)
• Ctrl + Mouse Wheel: Adjust similarity / reject threshold
• N: Refresh the yellow crosshair point suggestion
• Space: Finalize and create Polygon / Mask annotation
• Shift + Space: Finalize to Mask and replace the patches it agrees with
• Backspace: Clear all points and start over

Model Predictions

After a model is loaded, use these shortcuts to run inference:

• Ctrl + 1: Classification - Predict on selected Patch or all Review-labeled patches
• Ctrl + 2: Object Detection - Make predictions using detection model
• Ctrl + 3: Instance Segmentation - Make predictions using segmentation model
• Ctrl + 4: Semantic Segmentation - Make predictions on entire image
• Ctrl + 5: FastSAM - Generate automatic segmentations
• Ctrl + 6: YOLOE / See Anything - Make predictions using See Anything model

Confidence Window

• 1-5: Quick-select from top 5 predictions (Select tool)
• Left-Click on confidence bar: Change annotation’s label and verify it

Mouse Controls

• Left-Click: Select annotation, add point, start / end shape
• Left-Click + Drag: Move selected annotation, draw shape
• Ctrl + Left-Click: Add / remove from selection, add positive point / prototype
• Shift + Left-Click: Select range of items (Rasters Window)
• Ctrl + Drag: Box-select multiple annotations
• Right-Click + Drag: Pan the image viewer
• Right-Click: Open context menu (Rasters Window)
• Ctrl + Right-Click: Center Annotation Window on selected annotation
• Ctrl + Right-Click: Add negative point / prototype
• Double-Click: Load image (Rasters Window)
• Mouse Wheel: Zoom in / out, adjust patch / brush size (with Ctrl)

Annotation Window Navigation

• Mouse Wheel: Zoom in and out
• Right-Click + Hold + Drag: Pan the image
• Ctrl + Right-Click + Drag: Rotate the image (left / right drag for rotation, up / down for speed)

This site is open source. Improve this page.