Finding and Outlining Objects

Counting the cars needs detection, and measuring how much road each car covers needs segmentation.

The first working method reused the classifier of Part 8 without changing it: cut a rectangle out of the photograph, classify the crop, and repeat.

R-CNN (Girshick, Donahue, Darrell, and Malik, 2013) proposed about 2,000 rectangles per image with selective search, then ran a convolutional network on every one of them, reaching 53.3% mean average precision on PASCAL VOC 2012.

The rate of 20 ms per crop is illustrative; the paper's own timing was tens of seconds per image. Fast R-CNN (Girshick, 2015) shared one pass of convolution across all crops, and Faster R-CNN (Ren, He, Girshick, and Sun, 2015) learned the proposals inside the network, reaching 5 frames per second with 300 proposals per image.

Every version still ran a classifier over hundreds of crops of one photograph.

Each cell predicts B=2 boxes of five numbers each, and one list of C=20 class probabilities shared by both boxes.

The paper states the shape plainly: "Our final prediction is a 7\times7\times30 tensor." One forward pass produces all of it.

The confidence of a box combines two things the network is unsure about, (i) whether any object is there and (ii) how well the box would fit that object.

The class probabilities are conditional, \Pr(\text{Class}_i\mid\text{Object}), so multiplying the two gives the score that ranks a box against every other box in the photograph.

With \Pr(\text{Object})=0.95 and a predicted overlap of 0.84, the confidence is 0.95\times0.84\approx0.80 (illustrative values). A box scoring 0.72 for car and 0.05 for bus is reported as a car.

Every cell reports its two boxes, whether or not anything is there. On PASCAL VOC the network produces 98 boxes for every photograph, and a street scene of the kind drawn earlier holds about six objects (illustrative).

Two steps cut the list down. A score threshold removes the boxes the network doubts, and non maximum suppression removes the duplicates that survive, because several neighboring cells often report the same car.

1. Sort the surviving boxes by score, highest first.
2. Keep the highest, then delete every remaining box whose intersection over union with it passes a threshold, commonly 0.5.
3. Repeat with the highest box that is left, until the list is empty.

The box scoring 0.92 survives. Because the box scoring 0.85 overlaps it by more than the threshold, that box is deleted, while the box scoring 0.40 falls below the score threshold before any comparison begins.

This pruning is written by hand and runs after the network.

Because the whole photograph passes through the network once, the network keeps up with a video stream, so detection reached live camera feeds.

History: the YOLO paper opens on the human visual system, and its title, You Only Look Once, states the reading order it wants, because the network sees the image once. Later versions by other groups continue the line, and Ultralytics released YOLOv8 in 2023 and YOLO11 in 2024.

A single box per cell struggles when a tall pedestrian and a wide bus share one region, so later detectors give each cell a set of anchors, which are box shapes fixed in advance.

• Each anchor starts from its own shape, so the network learns a small correction to a box it already has.
• A cell with three anchors of different aspect ratios can report a pedestrian and a bus at the same place.
• Predictions are read from several layers of the network at once, so a coarse layer catches the bus and a fine layer catches the cyclist far away.

The anchor shapes are chosen by the engineer, often by clustering the box shapes in the training set, which adds one more setting to tune for each new dataset.

By 2019 the detector was one network surrounded by steps that training never adjusted.

DETR removes all three by changing what the network is asked to produce: a fixed size set of predictions, delivered in one pass, with duplicates forbidden by the loss.

DETR (Carion, Massa, Synnaeve, Usunier, Kirillov, and Zagoruyko, ECCV 2020) reads the image as features, mixes them with attention, and lets a fixed number of object queries each ask for one object.

The paper names two ingredients: a global loss over the whole set of predictions, which forces unique predictions through bipartite matching, and a transformer built from an encoder and a decoder.

An object query is a learned vector, one of a hundred, and each one leaves the decoder as a box and a label. The paper trains "all models with N=100 decoder query slots."

Because all hundred slots produce a row for a photograph that holds only two objects, the ninety eight slots left over need an answer of their own, and the paper gives them one: "an additional special class label \varnothing is used to represent that no object is detected within a slot."

Box values are illustrative.

When you ask an assistant to finish a sentence, a transformer decoder produces one token, reads its own output, and produces the next. The DETR decoder is built from the same layers.

The paper marks the difference plainly: DETR "decodes the N objects in parallel at each decoder layer, while Vaswani et al. use an autoregressive model that predicts the output sequence one element at a time", and the queries carry the role that positions carry in text, as "learnt positional encodings that we refer to as object queries".

Because a sentence has an order and the objects in a photograph do not, the assignment step supplies what position supplies for text.

Nothing tells query 1 that it owns the car. Training assigns the objects to the queries by choosing the pairing with the lowest total cost, where the cost of a pair falls as the predicted label agrees with the true label and the predicted box overlaps the true box.

Costs are illustrative. Because each true object takes exactly one query during training, the loss penalizes a second query that claims the same car, which is what removes the pruning step after the network.

Once the assignment is fixed, the training loop of Part 4 runs without change: every prediction has a target, so the loss function measures the error and backpropagation sends it through every layer.

1. A matched query is scored on two counts, (i) the label it gave the object and (ii) the distance between its box and the true box.
2. An unmatched query is scored on one count: it should have said no object.
3. The gradient of that total adjusts every weight, including the hundred query vectors themselves.

The loss itself penalizes duplicates, so no step after the network is needed.

The paper reports accuracy comparable to a well tuned Faster R-CNN on COCO, and the authors report the accuracy separately for large objects and for small objects.

COCO (Lin et al., 2014) is the benchmark behind these numbers: 2.5 million labeled instances across 328,000 images, every instance outlined.

Since then: Deformable DETR (2020) attends to a few sampled points to help small objects, and RT-DETR (Zhao et al., CVPR 2024) reports 53.1% average precision at 108 frames per second on a T4 graphics card, under the title "DETRs Beat YOLOs on Real-time Object Detection."

Every detector so far ends in a layer with one output per trained category, so the list of things it can name was settled before training began. Adding a category means labelling examples of it and training again.

A model trained on COCO answers for a person, a car, and a surfboard, because those are 3 of its 80 categories. The same model has no output at all for a wheelchair, a traffic cone, or a person carrying a surfboard.

Write the cone down in words, and use a model that has read words and pictures together.

CLIP (Radford et al., 2021) trains an image encoder and a text encoder together on 400 million pairs collected from the internet, through "the simple pre-training task of predicting which caption goes with which image."

Training pulls the vector of a photograph toward the vector of its own caption and pushes it away from the captions of the other images in the batch, which is what the field calls contrastive learning.

A trained classifier ends in a matrix with one row per category, and the score for a category is the dot product of its row with the image vector. CLIP replaces the matrix: "the text encoder is a hypernetwork which generates the weights of a linear classifier based on the text specifying the visual concepts that the classes represent."

The box head never has to change, so a YOLO grid, a DETR decoder, and a mask head all take the substitution.

Grounding DINO (Liu et al., 2023) starts from a DETR style detector and mixes the words into it at three places, so a whole sentence can steer the boxes.

The prompt may be a list of category names or a phrase such as "a person carrying a surfboard", which a fixed category list cannot express. The paper reports 52.5 average precision on COCO without training on any COCO data.

Kirillov et al. (CVPR 2019) named panoptic segmentation, which "unifies the typically distinct tasks of semantic segmentation (assign a class label to each pixel) and instance segmentation (detect and segment each object instance)."

Mask R-CNN (He, Gkioxari, Dollár, and Girshick, ICCV 2017) starts from a detector and adds one output: for every region it already found, a small picture of which pixels inside that box belong to the object.

1. The detector supplies a box, as in Faster R-CNN.
2. A mask branch predicts a grid of 28 by 28 values for that box, each between 0 and 1.
3. The grid is stretched to the size of the box, and every value above 0.5 becomes part of the object.

The paper describes it as adding "a branch for predicting an object mask in parallel with the existing branch for bounding box recognition," and reports 5 frames per second. Because the mask is 28 by 28 whatever the object's size, the mask of a bus is stretched over far more pixels than the mask of a distant cyclist, so the bus outline is coarser.

MaskFormer (Cheng, Schwing, and Kirillov, NeurIPS 2021) and Mask2Former (Cheng, Misra, Schwing, Kirillov, and Girdhar, CVPR 2022) apply the set prediction of DETR to pixels: the model returns N masks, each carrying one label, and the pixels a mask marks are the pixels of that object.

Because a mask with a label describes a car, a person, or the road equally well, one trained model answers all three segmentation questions, which is what the authors mean by a universal architecture.

Their addition is masked attention, which restricts each query's attention to the region of its own predicted mask.

Segment Anything (Kirillov et al., ICCV 2023) trains one model to outline whatever a user points at, so the label comes from the person and the outline comes from the model.

• The prompt is a click, a scribble, or a box, and the answer is a mask for the thing under it.
• The image passes through a Vision Transformer once, and each new prompt runs only a small decoder, so clicks feel immediate.
• Training used SA-1B, a dataset of more than 1 billion masks over 11 million licensed images.

The model returns an outline and leaves it unnamed, so a separate labeling step supplies the class, and annotation tools and editing tools use the model in that arrangement.

Grounded SAM (Ren et al., 2024) chains the two models: Grounding DINO reads the phrase and returns boxes, and Segment Anything takes each box as its prompt and returns a mask.

A teacher who types traffic cone receives an outline of every cone in the photograph, with no training run and no labelled cones.

A prediction counts as correct when its label matches the label of a true box and its intersection over union with that box passes a threshold.

1. At the common threshold of 0.5, the two boxes of the earlier worked example overlap by 0.25, so the prediction counts as a false positive and the true box counts as a missed detection.
2. Sweeping the score threshold from high to low traces precision against recall, and the area under that curve is the average precision for one class.
3. Averaging over the classes gives mean average precision; COCO averages further over the thresholds from 0.50 to 0.95.

Segmentation reuses intersection over union with masks, counting the overlap in pixels.

YOLO cuts a 7 by 7 grid and prunes afterwards. DETR cuts regions, mixes them with attention, and forbids duplicates in the loss. Mask2Former does that and returns masks. Grounding DINO does that and reads a sentence beside the image, so the categories arrive when the user types them.

Part 9 works through attention in detail.

1. 0.6\times0.5=0.30.   2. 10{,}000/70{,}000\approx0.14.   3. Ninety seven.   4. The class layer, which becomes the text vector for the word scooter.