CoralNet-Toolbox

Usage

Overview

The CoralNet Toolbox is a Python application built using PyQt5 for image annotation This guide provides instructions on how to use the application, including key functionalities and hotkeys

Annotations

PatchAnnotation: Represents a patch annotation
RectangleAnnotation: Represents a rectangular annotation
PolygonAnnotation: Represents a polygonal annotation
- MultiPolygonAnnotation: Represents multiple, non-overlapping polygonal annotations
MaskAnnotation: Represents a segmentation mask (one mask per image)

Computer Vision Tasks

Classification: Assign a label to an image (Patch)
Detection: Detect objects in an image (Rectangle)
Instance Segmentation: Segment objects in an image (Polygon)
Semantic Segmentation: Segment the entire image (Mask)

Thresholds for Computer Vision Tasks

Max Detections: Adjust to limit the number of detections in the status bar
Uncertainty Threshold: Adjust the uncertainty threshold in the status bar
IoU Threshold: Adjust the IoU threshold in the status bar
Area Threshold: Adjust the min and max area threshold in the status bar

Main Window

The main window consists of several components:

Menu Bar: Contains import, export, and other actions
Tool Bar: Contains tools for selection and annotation
Status Bar: Displays the image size, cursor position, view extent, annotation transparency, and thresholds
Annotation Window: Displays the image, Z-channel, and annotations
Label Window: Lists and manages labels
Image Window: Displays imported images
Confidence Window: Displays cropped images and confidence charts

File

New Project: Reload CoralNet-Toolbox (loss of data warning)
Open Project: Open an existing CoralNet-Toolbox project JSON file
Save Project: Save current CoralNet-Toolbox project to JSON file
Import:
- Import Images: Load image files
- Import Frames: Load video frames (Currently not available)
- Import Labels (JSON): Load label data from a JSON file
- Import CoralNet Labels (CSV): Load label data from a CoralNet CSV file
- Import TagLab Labels (JSON): Load label data from a TagLab JSON file
- Import Annotations (JSON): Load annotation data from a JSON file
- Import CoralNet Annotations (CSV): Load annotation data from a CoralNet CSV file
- Import TagLab Annotations (JSON): Load annotation data from a TagLab JSON file
- Import Squidle+ Annotations (JSON): Load annotation data from a Squidle+ JSON file
- Import Viscore Annotations(CSV): Load annotation data from a Viscore CSV file
- Import Dataset: Import a YOLO dataset for machine learning (Detection, Instance Segmentation)
Export:
- Export Labels (JSON): Save label data to a JSON file
- Export TagLab Labels (JSON): Save label data to a TagLab JSON file
- Export Annotations (JSON): Save annotation data to a JSON file
- Export GeoJSON Annotations: Save annotations to GeoJSON file (only for GeoTIFFs with CRS and Transforms data)
- Export Mask Annotations (Raster): Save annotations as segmentation masks
- Export CoralNet Annotations: Save annotation data to a CoralNet CSV file
- Export TagLab Annotations: Save annotation data to a TagLab JSON file
- Export Viscore Annotations: Save annotation data to a Viscore CSV file
- Export Dataset: Create a YOLO dataset for machine learning (Classification, Detection, Instance, Semantic)

Utilities

Sample:
- Sample Annotations: Automatically generate Patch annotations
  - Sampling Method: Choose between Random, Stratified Random, or Uniform distribution
  - Number of Annotations: Specify how many annotations to generate
  - Annotation Size: Set the size of the generated patch annotations
  - Label As: Choose which label to assign to generated annotations
  - Exclude Regions: Option to prevent sampling in areas with existing annotations
  - Margins: Define image boundary constraints for sampling:
    - Set margins in pixels or percentage
    - Configure different values for top, right, bottom, and left edges
    - Annotations will only be placed within these margins
  - Select images by highlighting them in the ImageWindow
Tile:
- Tile Manager: Pre-compute multiple tiles / work areas for selected images
  - Select images by highlighting them in the ImageWindow
- Tile Dataset: Tile existing Classification, Detection or Segmention datasets using yolo-tiling

AI-Assist

SAM:
- Deploy Predictor: Deploy EdgeSAM, MobileSAM, SAM, etc, to use interactively (points, box)
- Deploy Generator: Deploy FastSAM to automatically segment the image
See Anything (YOLOE):
- Deploy Predictor: Deploy a YOLOE model to use interactively within the same image
- Deploy Generator: Deploy a YOLOE model to use like a detector / segmentor, referencing other images’ annotations
  - Select the YOLOE model, parameters, and load it
  - Choose a reference label, then select the image(s) containing reference annotations (must be rectangles or polygons)
  - Generate visual prompt encodings (VPEs) from reference images / annotations (save and show if needed)
  - Use the loaded model w/ VPEs on new images, or work areas
Transformers:
- Deploy Model: Deploy a foundational model
  - Models Available: Grounding DINO, OWLViT

Machine Learning

Machine Learning:
- Merge Datasets: Merge multiple Classification datasets
- Tune Model: Identify ideal hyperparameter values before fully training a model
- Train Model: Train a machine learning model
- Evaluate Model: Evaluate a trained model
- Optimize Model: Convert model format
- Deploy Model: Make predictions using a trained model
- Video Inference: Perform inferencing on videos in real-time, view analytics

CoralNet

CoralNet:
- Authenticate: Authenticate with CoralNet
  - Enter your CoralNet username and password to access your sources
  - Authentication is required before downloading any CoralNet data
- Download: Download data from CoralNet
  - Source ID: Enter the Source ID (or multiple IDs separated by commas)
  - Output Directory: Select where to save downloaded files
  - Download Options: Choose what to download:
    - Metadata: Source information and settings
    - Labelset: All available labels from the source
    - Annotations: Point annotations with their labels
    - Images: Original images from the source
  - Parameters: Configure download settings:
    - Image Fetch Rate: Time between image downloads (seconds)
    - Image Fetch Break Time: Pause duration between batches (seconds)
  - Debug Mode: Toggle headless browser mode for troubleshooting

Explorer

Explorer:
- Annotation Settings: Select images, annotation types, and labels to include / filter press apply
- Model Settings: Use Color Features, a pre-trained, or existing classification model
  - Feature Mode: Embeddings (before classification), or Predictions (after classification)
- Embedding Settings: Map high-dimensional features to 2D space using PCA, TSNE, or UMAP
- Annotation Viewer: View, select, modify labels of annotations
  - Controls:
    - Left-Click: Select an annotation
    - Ctrl + Left-Click: Select multiple annotations
    - Shift + Left-Click: Select in-between annotations
    - Double Left Click: Unselect all annotations, exit from Isolation View
    - Ctrl + Right-Click: Update Annotation Window view, zoomed and centered on selected annotation
  - Toolbar:
    - Isolate Selection: Subset view
    - Sort By: Sort annotations by image name, label, confidence
    - Find Similar: Selects and isolates N nearest annotations to currently selected
    - Size: Slider-bar to control annotation size in Annotation Viewer
- Embedding Viewer: Selected, modify labels of annotations
  - Controls:
    - Left-Click: Select an annotation
    - Ctrl + Left-Click: Select multiple annotations
    - Ctrl + Left-Click + Drag: Select multiple annotations within a draw rectangle
    - Double Left Click: Unselect all annotations, exit from Isolation View
    - Right-Click + Drag: Pan around Embedding Viewer
    - Mouse Wheel: Zoom in and out of Embedding Viewer
  - Toolbar:
    - Isolate Selection: Subset view
    - Find Potential Mislabels: Select potentially incorrectly labeled annotations, based on location
    - Review Uncertain: Select annotations with lower Top-1 confidence scores (requires Predictions)
    - Find Duplicates: Select annotations that are likely duplicates of another (only selects the duplicates)
    - Home: Resets the Embedding Viewer zoom level Tip:
  - Use dual monitors to assess selected annotations in Annotation Viewer, in the Annotation Window

Tool Bar Tools

Select Tool: After selecting the tool
- Left-Click: Select an annotation drag to move it
- Ctrl + Left-Click: Add/remove annotation to current selection
- Ctrl + Delete / Backspace: Remove selected annotation(s)
- Ctrl + Drag: Create rectangle selection to select multiple annotations
- Ctrl + Mouse Wheel: Change size of the selected annotation
- Ctrl + Shift: Show resize handles for the selected annotation
- Ctrl + Shift + Mouse Wheel: Change the number of vertices for a polygon annotation
- Ctrl + Space: Confirm prediction for selected annotation with top machine confidence
- Ctrl + X: Cut a polygon annotation, explode a multi-polygon annotation, or subtract polygon annotations
  - Cutting Rules:
    - Only a single annotation can be selected
    - Press Ctrl-X to enter cutting mode, Left-Click to start, draw line, Left-Click to end
    - Hold Ctrl to create straight line segments
    - Press Backspace or Delete to cancel the current cutting line without making changes
  - Exploding Rules:
    - Select a MultiPolygonAnnotation, press Ctrl + X to explode it into multiple PolygonAnnotations
  - Subtraction Rules:
    - Multiple overlapping annotations must be selected one-by-one
    - The first annotations will be used as the cutters, the last polygon will be used as the base
- Ctrl + C: Combine multiple selected annotations (if same type and label)
  - Combining Rules:
    - All selected annotations must have the same label
    - All selected annotations must be verified (not machine predictions)
    - RectangleAnnotations can only be combined with other rectangles
    - PatchAnnotations can be combined with other patches or polygons to create polygons
    - PolygonAnnotations can be combined with other overlapping polygons to create a polygon
    - MultiPolygonAnnotations can be made with multiple non-overlapping polygons
Scale Tool: Provide scale to the image(s), and measure distances on the current image.
- Left-Click to set the starting point.
- Drag to draw a line, then Left-Click again to set the endpoint.
- Press Backspace to cancel drawing the scale line.
- The scale will be calculated based on the known provided length and pixel length.
- Area and Perimeter for an annotation can be viewed when hovering over the Confidence Window.
- Preferred units can be set in the Status Bar.
Patch Tool: After selecting the tool
- Left-Click: Add a patch annotation at the clicked position
- Ctrl + Mouse Wheel: Adjust the patch size up or down
- Mouse Movement: Shows a semi-transparent preview of the patch at the cursor position
Rectangle Tool: After selecting the tool
- Left-Click: Start drawing a rectangle click again to finish
- Mouse Movement: Shows a preview of the rectangle while drawing
- Backspace: Cancel the current rectangle annotation
Polygon Tool: After selecting the tool
- Left-Click (first): Start drawing a polygon
- Left-Click (subsequent): Add points to the polygon click near the first point to close
- Ctrl + Left-Click: Enable straight line mode click to add straight line segments
- Mouse Movement: Shows a preview of the polygon as you draw
- Backspace: Cancel the current polygon annotation
Brush Tool: After selecting the tool
- Left-click and drag to paint brush strokes on the canvas.
- Hold Ctrl and use the Mouse Wheel to adjust brush size.
- Press Ctrl + Shift to switch between a circle and square brush shape.
- A semi-transparent preview shows the brush stroke while drawing.
Erase Tool: After selecting the tool
- Left-click and drag to erase pixels.
- Hold Ctrl and use the Mouse Wheel to adjust eraser size.
- Press Ctrl + Shift to switch between a circle and square eraser shape.
- Press Ctrl + (Backspace or Delete) to clear the mask annotation on the current image.
- A semi-transparent preview shows the eraser while drawing.
Dropper Tool: After selecting the tool
- Left-click on a mask annotation region to select the associated label.
Fill Tool: After selecting the tool
- Left-click to fill the region under the cursor with the selected label.
SAM Tool: After a model is loaded
- Left-Click: Start drawing a work area click again to finish drawing
- Backspace: Cancel drawing the current work area
- Space: Create a work area from the current view
  - Space: Set working area confirm prediction finalize predictions and exit working area
  - Left-Click: Start a box press again to end a box
  - Ctrl + Left-Click: Add positive point
  - Ctrl + Right-Click: Add negative point
  - Backspace: Discard unfinalized predictions
See Anything (YOLOE) Tool: After a model is loaded
- Left-Click: Start drawing a work area click again to finish drawing
- Backspace: Cancel drawing the current work area
- Space: Create a work area from the current view
  - Space: Set working area run prediction finalize predictions and exit working area
  - Left-Click: Start a box press again to end a box
  - Backspace: Discard unfinalized predictions
Work Area Tool: For creating restricted areas for model prediction
- Left-Click: Start drawing a work area click again to finish drawing
- Backspace: Cancel drawing the current work area
- Space: Create a work area from the current view
- Ctrl + Alt: Create temporary work area from current view (disappears when keys released / pressed again)
- Ctrl + Shift: Show removal buttons on existing work areas (click the “X” to remove)
- Ctrl + Shift + Backspace: Remove all work areas in the current image
- Practical Use:
  - Define specific regions where models should make predictions
  - Useful for processing only relevant parts of large images
  - Work areas persist between tool changes and sessions

Status Bar

Image Size: Displays the image size
Cursor Position: Displays the cursor position
View Extent: Displays the view extent
Annotation Visibility: Show / Hide all existing annotations
Annotation Transparency: Adjust the annotation transparency
Scale: Displays the scale dimensions
- Provides a dropdown to select preferred units (mm, cm, m, km, etc.,)
- Enabled when a scale is set using the Scale Tool or imported from an image
Z: Displays the Z-dimension
- Provides a dropdown to select preferred units (mm, cm, m, km etc.,)
- Enabled when a Z-channel for the image is imported
- Select a color map in the dropdown to overlay Z-channel
- Click dynamic range button to enable dynamic recoloring of Z-channel
Patch Size: Manipulate Patch Size (only active when using Patch Tool)
Parameters: Adjust parameters including uncertainty, IoU, and area thresholds

Annotation Window

Zoom: Use the Mouse Wheel to zoom in and out
Pan: Right-Click and hold the Mouse Button to pan the image

Label Window

Move Label: Right-Click and drag to move labels
Add Label: Click the “Add Label” button to add a new label
Delete Label: Click the “Delete Label” button to delete the selected label
Edit Label: Click the “Edit Label” button to edit the selected label
Lock Label: Click the “Lock Label” button to lock the selected label
Enable / Disable Labels: Enable / disable labels by their checkbox to control visibility
- Toggle All Button (asterisk icon): Click to toggle all label checkboxes at once
  - If all labels are visible, clicking will hide all labels
  - If any labels are hidden, clicking will show all labels
  - Operates as a batch operation for improved performance with many labels and annotations
- Individual label checkboxes control whether annotations of that label are shown or hidden
- Hidden labels retain their data and can be shown again at any time
- Transparency changes apply to all labels, visible or hidden
Filter Labels: Use the filter text box to search for specific labels
Label Count: Displays the total number of labels in the project
Annotation Count: Shows information about the current annotations:
- When no annotation is selected: Shows the total count of annotations
- When a single annotation is selected: Shows the selected annotation’s index
- When multiple annotations are selected: Shows how many annotations are selected
- Can be edited (when in select mode) to navigate to a specific annotation by index

Image Window

Select Image: Double-Click on a row to select and load the image in the annotation window
Highlight Image: Single-Click on a row to highlight one or more rows in the image window
- Ctrl + Left-Click: Select multiple, non-adjacent rows
- Shift + Left-Click: Select multiple, adjacent rows
Open Context Menu:
- Right-Click on a single / multiple highlighted row to:
  - Check / uncheck highlighted rows
  - Batch Inference using loaded models (opens Batch Inference Dialog)
  - Import Z-channel for highlighted rows (opens Z-channel import dialog)
  - Remove Z-channel for highlighted rows
  - Delete annotations for highlighted rows
  - Delete images and annotations for highlighted rows
Search / Filter:
- By Image: Filter for images by name or sub-string
- By Label: Filter images by labels they contain
- No Annotations: Filter images with no annotations
- Has Annotations: Filter images with annotations
- Has Predictions: Filter images with predictions
- Highlighted: Filter highlighted images
Navigation:
- Home Button: Click to center the table on the currently selected image
- Highlight All: Highlight all images in the current filtered view
- Unhighlight All: Unhighlight all images in the current filtered view
Image Preview:
- Tooltips: Hover over a row to show image metadata
- Thumbnail: Hold Ctrl while hovering over a row to show a thumbnail

Confidence Window

Display Cropped Image: Shows the cropped image of the selected annotation
- The dimensions shown include both original and scaled sizes when applicable
- The border of the image is highlighted with the color of the top confident label
Confidence Chart: Displays a bar chart with confidence scores
- Top 5 Predictions: Shows up to 5 predictions with their confidence scores
- Prediction Selection: Click on any confidence bar to change the annotation’s label, verifying it
- Numerical Keys: Press keys 1-5 to quickly select from the top 5 predictions
- Prev / Next buttons: Cycle through annotations in order of their creation
Confidence Mode Toggle:
- Click the icon button next to the dimensions to toggle between user and machine confidence views
- User icon shows user-assigned confidence scores
- Machine icon shows model-predicted confidence scores
- The toggle is only enabled when both user and machine confidences are available
Visual Indicators:
- Each confidence bar shows the label color and confidence percentage
- Numbered indicators (1-5) show the rank of each prediction
- Hover over confidence bars to see a pointing hand cursor when selection is possible
Tooltips: Hover over the window while an annotation is selected to see its metadata

Hotkeys

Escape: Exit the program
Ctrl + S: Save the project
Ctrl + Z: Undo the last addition or deletion of an annotation
Ctrl + Shift + Z: Redo the previously undone addition or deletion of an annotation
Alt + Up/Down: Cycle through images
Ctrl + Up/Down: Cycle through labels
Ctrl + Left/Right: Cycle through annotations
Ctrl + A: Select all annotations, unselect all annotations (when pressed twice)
Ctrl + Alt: Switch between tools within the existing tool group, example:
- When a PatchAnnotation is selected, this switches back to the PatchTool
- When the PatchTool is active, this switches back to the SelectTool
- When the BrushTool is active, this switches to the EraseTool, and vice versa
Machine Learning, SAM, See Anything (YOLOE), and Transformers: After a model is loaded
- Ctrl + 1: Make prediction on selected Patch annotation, else all in the image with Review label using Classification model
- Ctrl + 2: Make predictions using Object Detection model
- Ctrl + 3: Make predictions using Instance Segmentation model
- Ctrl + 4: Make predictions using Semantic Segmentation model
- Ctrl + 5: Make predictions using FastSAM model
- Ctrl + 6: Make predictions using YOLOE model
- Ctrl + 7: Make predictions using Transformers model
Tooltips: Hover over tool buttons, image / annotation rows, and the Confidence Window for additional information

This site is open source. Improve this page.