Finding people and objects

A workflow often needs to know what is in a picture before it decides: is there a person in this room photo, how many people are in the shot, where is the dog, which part of the picture is the subject. Instance Segment (image/instance_segment) answers with a model that runs on Madoo’s own workers: it finds each person, animal or object of the 80 COCO categories and gives it a class, a score, a box and a mask. No provider is called and no credit is spent. It is a prediction, not a measurement — but a fast, repeatable one.

Domain	The question
Hospitality and real estate	is this room photo free of people? which furniture is in it?
Portraits and profiles	is there one person, or several? cut out the main one
Publishing and editorial	measure the subject apart from the background: is the face lit?
Pets and animals	is the dog in the picture, and where, to crop around it?
Events and scenes	how many people, chairs, tables does this picture show?

What it gives

For each instance the node returns its class (a COCO name and id), a score from 0 to 1 — how confident the model is, not a calibrated probability — its box (the tight rectangle of its mask), its area in pixels and the edges of the picture it reaches. Instances come best first and are numbered 1, 2, … within the run.

Output	What it is
`instances`	the whole result as JSON (`madoo.image-instances/v1`): each instance, the model that ran, the share of the picture covered by no instance
`mask`	an image of the picture’s size, white inside the selected instances; absent when nothing is found
`count`	how many instances were found, a number for a Predicate
`found`	true when at least one was found, a boolean for a Condition
`quality_check`	the check, for a Quality Gate: it fails with `NOT_FOUND` when nothing is found

Pixels no instance covers are unassigned, not “background”: the wall and the floor of the room are not classes the model knows, so they are simply not claimed.

What it knows

Six pictures and what the model found in each. A dog: dog 0.97, found. An armchair: chair 0.92, found. A model in a T-shirt and trousers: person 0.96, the clothes are part of the person. A teapot: vase 0.95, there is no teapot class and the nearest one is used. A rocket on its pad: nothing, there is no rocket class. A jacket on a backdrop: nothing, there is no garment class.

The model knows the 80 COCO categories, by these names (the ones Class Filter takes):

Group	Classes
People	person
Animals	bird, cat, dog, horse, sheep, cow, elephant, bear, zebra, giraffe
Vehicles	bicycle, car, motorcycle, airplane, bus, train, truck, boat
Outdoors	traffic light, fire hydrant, stop sign, parking meter, bench
Carried	backpack, umbrella, handbag, tie, suitcase
Sports	frisbee, skis, snowboard, sports ball, kite, baseball bat, baseball glove, skateboard, surfboard, tennis racket
Table	bottle, wine glass, cup, fork, knife, spoon, bowl
Food	banana, apple, sandwich, orange, broccoli, carrot, hot dog, pizza, donut, cake
Furniture	chair, couch, potted plant, bed, dining table, toilet
Electronics	tv, laptop, mouse, remote, keyboard, cell phone
Appliances	microwave, oven, toaster, sink, refrigerator
Indoors	book, clock, vase, scissors, teddy bear, hair drier, toothbrush

Three consequences:

What is not a class is not found, or is called the nearest class. A jacket on a backdrop, a rocket, a logo give nothing; a teapot comes back as a vase. For a product or a garment on its own, use Remove Background to cut it out and Measure Subject to measure it.
A person is one instance, clothes included. The model’s mask covers the whole figure — face, hair, clothes; it has no class for a garment or a face, and finds neither on its own — for faces, see Finding and framing faces.
It identifies no one. “Person” is all it says: not who, not whether two pictures show the same person.

The main subject

Most pictures hold more than one instance. Three settings choose what the result is about:

Class Filter (class_filter) keeps only some classes: ["person"], ["dog", "cat"], or a COCO id. Connect a list to its input to choose per run — the input wins over the parameter.
Mask Of (select) chooses what the mask covers: all instances merged (default), the largest one, or the one with the highest_score. Every instance stays in instances; selected says which ones make the mask.
Region limits the search to a rectangle ({"x": 0, "y": 0, "width": 512, "height": 512}, or a box from Measure Subject or Image Detail); boxes and masks are still given in pixels of the whole picture.

The mask goes straight into the Mask of Image Statistics or Image Detail: the numbers then describe the person, not the room. instances goes straight into Crop’s Region: Crop cuts the box of the first selected instance.

Small and far

A breakfast table by a lake: 25 instances, 9 of them under 2,000 pixels — jars, cups, bottles, a spoon, a knife, and through the window, enlarged, small boats on the lake found at 0.37 to 0.63.

The model sees small things: the boats on the lake behind the breakfast table are real boats, a few dozen pixels wide. Small, far or partly hidden objects score lower, and that is where its mistakes are too. Two settings keep only what is clearly there:

Score Threshold (score_threshold, 0.35): raise it to 0.5 or 0.6 for fewer, surer instances;
Min Area (min_area_px, 16 pixels): raise it to leave out tiny instances.

A score close to the threshold can move by a few hundredths with how the file was saved: an instance right at the edge may appear in a PNG and not in a JPEG of the same picture. Set the threshold with some room below the instances you expect.

How it reads a picture

Upright, in sRGB, 8 bits. As the other measuring nodes: EXIF orientation applied, colour profiles converted, HDR (PQ/HLG) refused with HDR_NOT_SUPPORTED; up to 40 megapixels.
At the model’s resolution. The picture is resized to 504 × 504 for the model, and each mask is computed at 126 × 126, then drawn at the picture’s size. Outlines are smooth but not pixel-exact: on a 4,000-pixel picture an edge is placed to within a few tens of pixels. For an exact outline of a product, use Remove Background.
Locally. The model (RF-DETR-Seg Large, Apache-2.0) runs on the worker’s CPU: about 0.4 to 1 second per picture, whatever its size. The first step on a worker also loads the model, which takes a little longer.

Recipes

Only rooms without people. Room photo → Instance Segment (Class Filter ["person"]) → found into a Predicate (equals false) → a Filter: photos with someone in them are set aside, the others go on to the listing.

Measure the subject, not the scene. Portrait → Instance Segment (["person"], Mask Of largest) → mask into Image Statistics’ Mask, the portrait into its Image: the brightness and colour of the person. The lab does this.

Crop around the animal. Pet photo → Instance Segment (["dog", "cat"], Mask Of highest_score) → instances into Crop’s Region, the photo into Crop’s Image: the box of the animal, ready for a square or a story format.

Require a subject before a paid step. Picture → Instance Segment (["person"], Check ID has-person) → quality_check into a Quality Gate in route mode, the picture into the gate’s value: a picture without a person never reaches the paid step behind it.

Count the furniture. Room photo → Instance Segment (["chair", "couch", "bed", "dining table"]) → count, or instances into the JSON Enumerator (array path instances) to go through each one.

Settings

Setting	Default	What it does
Class Filter (`class_filter`)	every class	keep only these classes: COCO names or ids
Score Threshold (`score_threshold`)	0.35	instances scoring below are left out, 0.05 to 1
Mask Of (`select`)	`all`	the mask covers `all` instances, the `largest` or the `highest_score`
Max Instances (`max_instances`)	50	the most instances kept, best first, up to 200
Min Area (`min_area_px`)	16	instances with a smaller mask are left out
Mask Threshold (`mask_threshold`)	0.5	a pixel is inside when the mask probability is above; lower = looser outlines
Output Masks (`output_masks`)	on	produce the mask image
Untagged Images (`untagged`)	`assume_srgb`	read a picture without a profile as sRGB, or `fail`
Check ID (`check_id`)	`instance-segment`	the check’s name in a Quality Gate

A wrong setting — a class the model does not know, a threshold out of range — is reported when the workflow is saved, with INVALID_INSTANCE_SEGMENT_PARAMETER.

What it does not do

It does not know every object. 80 categories; no garments, generic products, logos, faces or body parts. Faces are found by Face Detect; a given product, person or mark — this one, shown in reference pictures — by Locate Subject.
It does not identify anyone. It finds people, never who they are.
It does not measure. A score is a confidence; a mask is a prediction. To measure what it found, connect its mask to Image Statistics, Image Detail or Measure Subject.
It does not draw exact outlines. For a clean cut-out of a product or a person, use Remove Background.

The lab

The instances lab runs the portrait recipe, at no credit cost: Instance Segment keeps only people and selects the largest — 4 people found, the mask is the man in the foreground; Image Statistics measures 0.081 inside him and 0.217 over the whole photo; Crop cuts his box, 625 × 950.

The lab workflow: the portrait goes to Instance Segment, which finds 4 people and gives the mask of the man; Image Statistics measures the tone of the person with that mask (median 0.081) and of the whole photo (0.217); Crop cuts the person, 625 × 950.

In complete examples

I07 — a hotel’s photos screened for a booking portal: the count of people keeps photos with guests in them off the portal.