Class PredictSinglePose

Namespace

Bonsai.Sleap

Assembly

Bonsai.Sleap.dll

Represents an operator that performs markerless, single instance, pose estimation for each image in the sequence using a SLEAP model.

Almost all SLEAP operators afford the detection of multiple instances for each incoming image. However, in certain cases we might be interested in only identifying a single object in the incoming image. This strategy offers multiple advantages, specifically in terms of performance. In Bonsai.SLEAP, this functionality is implemented using the PredictSinglePose operator that implements a single_instance network. Since the centroid detection step is not performed by the network, the operator expects an already centered instance on which it will run the pose estimation. Moreover, the network will always return a single output per incoming frame, even if no valid instances are detected.

The following example workflow highlights how combining basic computer-vision algorithm for image segmentation for centroid detection, with the network-based pose estimation, results in >2-fold increases in performance relative to the previously introduced PredictPoses operator. In this example, the first part of the workflow segments and detects the centroid positions (output of BinaryRegionAnalysis) of all available objects in the incoming frame, which are then combined with the original image to generate centered crops (CropCenter). These images are then pushed through the network that will perform the pose estimation step of the process.

Finally, it is worth noting that PredictSinglePose affords two input overloads. When receiving a single image it will output a corresponding Pose. Since the operator skips the centroid-detection stage, it won't embed a Centroid field in Pose. Alternatively, a batch mode can be accessed by providing an array of images to the operator, instead returning PoseCollection. This latter overload results in dramatic performance gains relative to single images.

public class PredictSinglePose : Transform<IplImage, Pose>

Inheritance

object

Combinator<IplImage, Pose>

Transform<IplImage, Pose>

PredictSinglePose

Inherited Members

Combinator<IplImage, Pose>.Process(IObservable<IplImage>)

object.ToString()

object.Equals(object)

object.Equals(object, object)

object.ReferenceEquals(object, object)

object.GetHashCode()

object.GetType()

object.MemberwiseClone()

Properties

ColorConversion

Gets or sets a value specifying the optional color conversion used to prepare RGB video frames for inference. If no value is specified, no color conversion is performed.

public ColorConversion? ColorConversion { get; set; }

Property Value

ColorConversion?

ModelFileName

Gets or sets a value specifying the path to the exported Protocol Buffer file containing the pretrained SLEAP model.

[FileNameFilter("Protocol Buffer Files(*.pb)|*.pb")]
public string ModelFileName { get; set; }

Property Value

string

PartMinConfidence

Gets or sets a value specifying the confidence threshold used to discard predicted body part positions. If no value is specified, all estimated positions are returned.

[Range(0, 1)]
public float? PartMinConfidence { get; set; }

Property Value

float?

ScaleFactor

Gets or sets a value specifying the scale factor used to resize video frames for inference. If no value is specified, no resizing is performed.

public float? ScaleFactor { get; set; }

Property Value

float?

TrainingConfig

Gets or sets a value specifying the path to the configuration JSON file containing training metadata.

[FileNameFilter("Config Files(*.json)|*.json|All Files|*.*")]
public string TrainingConfig { get; set; }

Property Value

string

Methods

Process(IObservable<IplImage[]>)

Performs markerless, single instance, batched pose estimation for each array of images in an observable sequence using a SLEAP model.

public IObservable<IList<Pose>> Process(IObservable<IplImage[]> source)

Parameters

source IObservable<IplImage[]>

The sequence of image batches from which to extract the poses.

Returns

IObservable<IList<Pose>>

A sequence of Pose collection objects representing the results of pose estimation for each image batch in the source sequence.

Process(IObservable<IplImage>)

Performs markerless, single instance, pose estimation for each image in an observable sequence using a SLEAP model.

public override IObservable<Pose> Process(IObservable<IplImage> source)

Parameters

source IObservable<IplImage>

The sequence of images from which to extract the pose.

Returns

IObservable<Pose>

A sequence of Pose objects representing the result of pose estimation for each image in the source sequence.

See Also

PredictCentroids

PredictPoses

PredictPoseIdentities

GetBodyPart