Degradation Process

y: an LR image

x: latent HR image

k: convolution with a blur kernel

s: downsampling operation with scale factor s

n: usually is additive white Gaussian noise (AWGN)

Deep SR Category

SISR: mostly assume that a LR image is bicubicly downsampled from a HR image (bicubic degradation, lots of related work, begin from SRCNN in ECCV 2014)

• Towards real images data
• Towards advanced architecture
• Towards advanced loss

Blind SR: assume degradation process is unknown (only explored in CNNs recently, the first work is claimed in CVPR 2018)

• Consider modeling blur kernel in the network
• Consider zero-shot learning

Exemplar Guided SR: Use an exemplar image as additional information for SR (only found 3 related papers)

• Warping-based
• Patch matching

SISR

Name	Published	Method	Comments
Zoom to Learn, Learn to Zoom	2019/CVPR	introduce a new dataset, SR-RAW, for super-resolution from raw data, with optical ground truth, propose a novel contextual bilateral loss for training	SR-RAW dataset from real sensor, contextual bilateral loss
Towards Real Scene Super-Resolution with Raw Images	2019/CVPR	propose a method to generate realistic training data by simulating the imaging process, develop a dual network architecture for training	generate realistic training data, dual network
Toward Real-World Single Image Super-Resolution: A New Benchmark and A New Model	2019/CVPR	build a real-world SR (RealSR) dataset captured by adjusting the focal length and aliging the image pairs at different resolutions, present a Laplacian pyramid based kernel prediction network (LP-KPN)	real-world SR (RealSR) dataset, LP-KPN
Second-order Attention Network for Single Image Super-Resolution	2019/CVPR	propose SAN with second-order channel attention (SOCA) module that adaptively rescale the channel-wise features and present a non-locally enhanced residual group (NLRG) structure	second-order channel attention module, non-locally enhanced residual group structure
SROBB: Targeted Perceptual Loss for Single Image Super-Resolution	2019/CVPR	propose a loss that penalizes images at different semantic levels according to a segmentation label	targeted perceptual loss

Blind SR

Name	Published	Method	Comments
Learning a Single Convolutional Super-Resolution Network for Multiple Degradations	2018/CVPR	Propose a general framework with dimensionality stretching strategy that enables a SR network to take blur kernel and noise level as input	Consider the kernel as input, but do not estimate the kernel
Blind Image Super-Resolution with Spatially Variant Degradations	2019/SIGA	Use a generator network to synthesize the HR given a LR and the blur kernel, a kernel discriminator to analyze the error in generated HR, an optimization procedure to recover the kernel and the HR by minimizing the error	Estimate spatially varying kernels to handle images with composited content
Blind Super-Resolution With Iterative Kernel Correction	2019/CVPR	Proposed an Iterative Kernel Correction (IKC) framework consists of a SR network given a LR and the blur kernel, a kernel predictor given a LR, and a corrector to estimate kernel error given the HR result and the current kernel	Use an iterative method to correct the kernel gradually and achieve blur kernel estimation in blind SR problem
Kernel Modeling Super-Resolution on Real Low-Resolution Images	2019/ICCV	Generate a GAN-augmented blur kernel pool by extracting real blur kernels with a kernel estimation algorithm, then construct a paired LR-HR training dataset based on the kernel pool	Blur kernel augmentation by GAN
“Zero-Shot” Super-Resolution using Deep Internal Learning	2018/CVPR	Train a small image-specific CNN at test time on examples extracted solely from the input image	Unsupervised SR methods, image-specific kernel is estimated for training dataset, can handle different imaging conditions
Meta-Transfer Learning for Zero-Shot Super-Resolution	2020/CVPR	A three stages training scheme: a large-scale training with bicubic degradation data, a meta-transfer learning with diverse blur kernels data, and a self-supervision meta-test phase	Generate result with a few gradient descent update

Exemplar Guided SR

Name	Published	Method	Comments
Learning Warped Guidance for Blind Face Restoration	2018/ECCV	Predict flow to warp the guided image, and then take LR image and warped guidance as input to produce the result. Use landmark loss and total variation regularization for training	Warp the guidance, use landmark loss and tv loss
Exemplar Guided Face Image Super-Resolution without Facial Landmarks	2019/CVPRW	Warp the guided image to align its contents by a subnetwork, train the network in an adversarial generative manner with identity loss	Warp the guidance, use adversarial loss and identity loss
Image Super-Resolution by Neural Texture Transfer	2019/CVPR	design an deep model which enriches HR details by adaptively transferring the texture from Ref images according to the textural similarity	texture transfer for reference-based SR

Dataset

• Large-scale CelebFaces Attributes (CelebA) Dataset
• VGGFace2: A large scale image dataset for face recognition

Network Strategies

• Learn a mapping that map the vectorized kernel to a low dimensional representation and assemble it at each pixel or region to obtain the kernel maps

• Reduce the dimensionality of the kernel space by principal component analysis (PCA) and stretch the result into kernel maps

• Predict a residual image that is then added to a bicubicly upsampled image to produce the output

• spatial feature transform (SFT) layer: use the kernel maps to predict affine transformation for the input feature maps by a scaling and shifting operation

Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network

• sub-pixel convolution layer

Perceptual Losses for Real-Time Style Transfer and Super-Resolution

• feature loss for preserving image content and overall spatial structure, but not color, texture
• style loss for preserving stylistic features but not spatial structure

Decouple Learning for Parameterized Image Operators

• use a weight learning network to adaptively predict the weights of the base network

Dynamic Convolution: Attention over Convolution Kernels

• dynamic convolution: aggregates multiple convolution kernels dynamically based upon the attentions

Deep Network Interpolation for Continuous Imagery Effect Transition

CFSNet: Toward a Controllable Feature Space for Image Restoration