Image Coding for Machines via Feature-Preserving RDO

Many images and videos today are primarily processed by computer vision algorithms, with humans inspecting only occasionally. Feature-preserving RDO provides a method to compress such content efficiently while preserving the features that downstream vision models rely on.

Many images and videos today are primarily processed by computer vision algorithms, with humans inspecting only occasionally. When this content requires compression before processing, e.g., in distributed applications, coding methods must optimize for both visual quality and downstream task performance. We first show theoretically that an approach to reduce the effect of compression for a given task loss is to perform rate-distortion optimization (RDO) using the distance between features, obtained from the original and the decoded images, as a distortion metric. Optimizing directly such a rate-distortion objective is computationally impractical because it requires iteratively encoding and decoding the entire image plus feature evaluation for each possible coding configuration. We address this problem by simplifying the RDO formulation to make the distortion term computable using block-based encoders. We apply Taylor’s expansion to the feature extractor, recasting the feature distance as a quadratic metric involving the Jacobian matrix of the neural network. Then, we replace the linearized metric with a block-wise approximation, called input-dependent squared error (IDSE). To make the metric computable, we approximate IDSE using sketches of the Jacobian. The resulting loss can be evaluated block-wise in the transform domain and combined with the sum of squared errors (SSE) to address both visual quality and computer vision performance. Simulations with AVC and HEVC across multiple feature extractors and downstream networks show up to 17% bit-rate savings for the same task accuracy compared to RDO based on SSE, with no decoder complexity overhead and a small encoder complexity increase.