Augmenting Perceptual Super-Resolution via Image Quality Predictors
Fengjia Zhang*
Samrudhdhi B. Rangrej*
Tristan Aumentado-Armstrong*
Afsaneh Fazly
Alex Levinshtein
AI Center – Toronto, Samsung Electronics
{
f.zhang2,
s.rangrej,
tristan.a,
a.fazly,
alex.lev
}
@samsung.com
Differentiable
Fast
Multimodal Supervision
=
>
>
Ranks
Fast
Precise
Human derived
IQA derived
>
>
>
Image Quality Optimization
score = 0.87
score = 0.75
score = 0.57
score = 0.43
SR
Model
Ground-truth
Preference?
Ranks
Fast
Precise
Enhanced ground-truths
IQA
Metric
Score
SR
Model
Good
Bad
Optimization
metric?
Differentiable
Fast
LR image
SR image
LR image
SR image
SR image
SR image
EGT 01
EGT 02
EGT 03
EGT 04
EGT 03
EGT 02
EGT 01
EGT 04
EGT 03
EGT 02
EGT 01
EGT 04
EGT i
Human Score
IQA Score
IQA
Metric
Score
Figure 1.
Schematics for improving perceptual super-resolution (SR).
Perceptual quality of SR can be improved in two ways:
(Left)
providing supervision through multiple enhanced ground-truths (EGT) or
(Right)
direct optimization for the quality of the super-resolved
image.
In both cases,
human-in-the-loop can greatly improve performance.
However,
manual annotation is tedious,
imprecise,
and non-
differentiable.
An IQA metric can replace a human in rating the enhanced ground-truths or can directly act as a differentiable optimization
objective.
In this paper, we specifically assess whether more practical no-reference (NR) IQA metrics can replace human raters for SR. We
find that combining NR-IQA-based sampling and regularized optimization is sufficient to attain state-of-the-art perceptual image quality,
without
requiring human ratings.
Abstract
Super-resolution
(SR),
a
classical
inverse
problem
in
com-
puter vision, is inherently ill-posed, inducing a distribution
of
plausible
solutions
for
every
input.
However,
the
de-
sired result is not simply the expectation of this distribution,
which is the blurry image obtained by minimizing pixelwise
error, but rather the sample with the highest image quality.
A
variety
of
techniques,
from
perceptual
metrics
to
adver-
sarial
losses,
are
employed
to
this
end.
In
this
work,
we
explore an alternative: utilizing powerful non-reference im-
age quality assessment (NR-IQA) models in the SR context.
We
begin
with
a
comprehensive
analysis
of
NR-IQA
met-
rics
on
human-derived
SR
data,
identifying
both
the
accu-
racy
(human
alignment)
and
complementarity
of
different
metrics.
Then,
we
explore
two
methods
of
applying
NR-
IQA
models
to
SR
learning:
(i)
altering
data
sampling,
by
building
on
an
existing
multi-ground-truth
SR
framework,
and
(ii)
directly
optimizing
a
differentiable
quality
score.
Our results demonstrate a more human-centric perception-
distortion
tradeoff,
focusing
less
on
non-perceptual
pixel-
wise distortion, instead improving the balance between per-
ceptual fidelity and human-tuned NR-IQA measures.
1. Introduction
Many tasks in human and computer vision are naturally for-
mulated
as
ill-posed
inverse
problems
[
65
].
Single-image
super-resolution (SISR), which has many practical applica-
tions to digital photographic zoom, is a well-studied exam-
ple
of
this
(e.g.,
[
4
,
24
]).
In
SISR,
a
given
low-resolution
(LR) image has an associated distribution of high-resolution
(HR) “real” images that could have given rise to it. The fun-
damental
challenge
of
SR
is
therefore
not
just
to
find
any
sample from that distribution,
but instead to find perceptu-
ally plausible one(s).
Early learning-based models, trained
with
pixel-wise
losses
(e.g.,
[
21
,
22
]),
effectively
“aver-
age”
over
possible
solutions
in
pixel-space,
resulting
in
*Equal Contribution.
This CVPR paper is the Open Access version, provided by the Computer Vision Foundation.
Except for this watermark, it is identical to the accepted version;
the final published version of the proceedings is available on IEEE Xplore.
2311
blurry output images with a high peak signal-to-noise ratio
(PSNR). However,
human preferences indicate that a solu-
tion with high image quality is better than this averaged one.
Hence,
numerous
techniques
have
been
devised
to
empha-
size perceptual fidelity, such as perceptual metrics [
19
,
39
]
and adversarial losses (e.g., [
70
]), greatly improving image
quality.
In other words, pixelwise fidelity is a poor measure
of
perceptual
quality.
In
fact,
under
some
conditions,
they
are
directly
oppositional,
forming
a
“perception–distortion
tradeoff” [
5
,
6
].
In theory, the only pixel-space constraint is
given by the LR image;
besides this,
the optimal SR result
(in
terms
of
human
preference)
may
have
very
high
pix-
elwise
distortion
(w.r.t.
the
“real”
ground-truth
generating
image), as long as it has high plausibility with respect to the
LR input and high image quality.
Instead
of
optimizing
pixel-space
distortions,
we
focus
on
improving
perceptual
image
quality.
This
is
commonly
done
using
a
combination
of
perceptual
losses
and
GANs
[
48
,
58
,
61
,
79
,
84
],
enabling
the
SR
model
to
target
a
multi-modal
distribution
rather
than
specific
ground-truth
targets.
The
challenge
of
such
methods,
however,
is
to
produce
perceptually
plausible
outputs
without
introduc-
ing high-frequency artifacts [
51
].
Many full-reference (FR)
[
18
,
27
,
42
,
99
] and non-reference image quality assessment
(NR-IQA) [
41
,
68
,
74
,
75
] metrics were developed to align
with human preferences for identifying perceptually plausi-
ble images.
While some approaches started to replace per-
ceptual losses with FR metrics like LPIPS [
42
] and DISTS
[
18
], for the task of image restoration, NR-IQA metrics are
still used purely for evaluation purposes.
Motivated by the
adoption of human feedback guidance in text-to-image gen-
erative
models
(e.g.,
[
16
,
23
,
67
,
92
]),
we
aim
to
use
NR-
IQA metrics to improve SISR.
Recently,
Chen
et
al.
[
12
]
used
human
feedback
to
im-
prove
SISR.
They
do
so
by
generating
multiple
enhanced
versions of GT (Fig.
1
left), manually rating these different
versions
using
multiple
human
evaluators,
and
fine-tuning
the
model
on
the
positively
ranked
GTs.
While
this
re-
sults
in
significant
perceptual
quality
improvements
with-
out introducing unwanted artifacts,
manual human ranking
is
very
coarse
and
cumbersome.
We
instead
use
an
auto-
matic NR-IQA measure that is well-correlated with human
scores, yielding a more fine-grained ranking, and bypassing
the requirement for having human feedback (Fig.
1
centre).
Additionally, since the measure is fully differentiable, it can
be
used
for
direct
optimization
(e.g.,
replacing
or
comple-
menting GANs), unlike human scores that cannot be used in
this fashion (Fig.
1
right).
Our contributions are as follows:
•
We present a detailed analysis of NR-IQA on two human-
derived SR datasets, thus identifying metrics that are gen-
erally useful for improving SR image quality.
•
We
explore
the
application
of
NR-IQA
to
SISR,
via
two
approaches:
sampling
across
multiple
GTs
weighted
by
NR-IQA, and direct optimization of NR-IQA.
•
We
achieve
SISR
results
that
are
perceptually
on
par
or
better than SISR finetuned with human feedback, but us-
ing an automatic NR-IQA measure instead.
2. Related Work
Deep learning-based SISR:
Deep learning has given a sig-
nificant boost to SISR performance, taking the SOTA man-
tle
from
dictionary-based
methods
[
76
,
77
,
93
]
to
CNN-
based approaches like SRNet and its successors [
2
,
21
,
22
,
44
].
Since
then,
there
came
many
architectural
improve-
ments:
deeper architectures, like RCAN [
102
], hierarchical
processing
[
47
],
advanced
building
blocks
like
in
NAFNet
[
13
] and RRDB from ESRGAN [
84
], and better upsampling
like PixelUnshuffle [
71
], to name a few.
Due to long range
dependencies in SISR, transformer-based methods [
14
,
50
],
and auto-regressive models based on Mamba [
54
], have re-
cently achieved SOTA performance [
15
,
32
].
Perceptual
quality-oriented
SISR:
Early
SISR
ap-
proaches optimized a simple pixel-wise reconstruction loss,
such
as
L2
or
L1,
between
model
output
and
ground-truth
[
21
,
22
].
However,
due
to
the
ill-posedness
of
SISR,
this
yields poor perceptual quality.
Blau and Michaeli [
6
] have
shown that there exists a tradeoff between good perceptual
quality and accurate reconstruction (or fidelity). To improve
perceptual quality, perceptual losses were proposed, such as
SSIM [
86
], which measures patch similarity rather than per-
pixel similarity, and later others [
39
] that measure the simi-
larity between deep VGG features rather than pixel intensi-
ties.
Combined with perceptual losses, GAN-based training
[
30
] was used to improve perceptual quality in SRGAN [
48
]
and
many
follow-up
approaches
[
3
,
58
,
61
,
79
,
84
].
One
challenge of such approaches is to improve perceptual qual-
ity without introducing unwanted hallucinations.
Proposed
solutions
include
more
specialized
discriminators
[
62
]
and
better balancing between various loss terms [
51
,
63
]. Rather
than
changing
the
training
loss
function,
perceptual
qual-
ity can be improved by explicitly generating multiple train-
ing
targets
[
12
,
38
],
or
encouraged
using
specific
architec-
tural designs.
Normalizing flows [
56
,
96
] were used to di-
rectly
output
a
distribution
over
plausible
solutions
rather
than a single SR image.
More recently, diffusion-based ap-
proaches
[
40
,
69
,
83
,
89
–
91
,
95
]
and
alike
[
17
]
have
been
shown
to
achieve
a
better
perceptual
quality
than
GAN-
based methods.
Human guided perceptual quality assessment:
One may
use image quality assessment (IQA) metrics to evaluate the
aesthetic
quality
of
super-resolved
images.
They
can
be
full-reference
(FR)
or
no-reference
(NR)
metrics.
In
real-
world applications where the true HR image is unavailable,
NR-IQA
metrics
are
more
useful.
Early
opinion-unaware
NR-IQA
metrics
used
hand-crafted
features
to
assess
how
closely
the
statistics
of
the
output
images
match
with
nat-
2312
Method
PaQ-2-PiQ [
97
]
NIMA
†
[
74
]
MUSIQ
/
[
41
]
LIQE
~
[
101
]
ARNIQA-TID
?
[
1
]
Q-Align
⇧
[
88
]
TOPIQ-NR [
11
]
Accuracy (%)
76.41
74.91
74.47
74.03
74.03
73.77
73.06
Table 1.
Phase I analysis on SBS180K.
Accuracy of top 7 NR-IQA metrics on a subset (1212 image pairs) from train portion.
We use the
default configuration of the metrics, provided by the
IQA-PyTorch
toolbox, unless stated otherwise.
†
We use NIMA with Inception V2
as base model.
/
We use the default MUSIQ trained on KonIQ [
35
].
~
We use LIQE pretrained on KonIQ [
35
].
?
We use ARNIQA metric
trained on TID2013 [
66
].
⇧
We use Q-Align metric specialized in image quality assessment.
Method
PaQ-2-PiQ [
97
]
NIMA
†
[
74
]
MUSIQ
/
[
41
]
LIQE
~
[
101
]
ARNIQA-TID
?
[
1
]
Q-Align
⇧
[
88
]
TOPIQ-NR [
11
]
Train Acc.
(%)
80.41
79.32
79.96
77.70
77.74
80.00
78.30
Test Acc.
(%)
80.57
81.37
82.73
77.45
77.07
80.68
81.28
Table 2.
Phase II analysis on SBS180K dataset.
Accuracy of top 7 (according to Phase I) NR-IQA metrics on the entire train and test
sets of SBS180K. We exclude image pairs where humans prefer both images equally.
†
/
~
?
⇧
See Table
1
.
ural
scene
statistics,
e.g.,
BRISQUE
[
59
]
and
NIQE
[
98
].
Others developed metrics that align with human preferences
[
68
,
75
].
Recently,
many
developed
deep-learning-based
opinion-unaware approaches such as FID [
34
], and opinion-
aware
approaches
such
as
NIMA
[
74
]
and
MUSIQ
[
41
].
While
the
above
metrics
are
general
purpose,
metrics
such
as
NRQM
[
57
]
and
NeuralSBS
[
43
]
are
specifi-
cally
designed
to
align
with
human
preferences
for
super-
resolution.
Recently,
human
preferences
are
being
widely
incorporated in improving generative models, especially for
text-to-image
generation
[
16
,
52
,
92
].
Yet,
incorporating
human guidance in improving SR models has not received
much attention.
Ding
et al
. [
19
] attempt to use various
full-
reference
IQA
metrics
for
SR
model
finetuning,
and
Chen
et
al
.
[
12
]
propose
to
use
human
guidance
in
GT
selection
process
.
Unlike these works, we attempt to assess the abil-
ity of
no-reference
IQA metrics to (a)
automatically
rate and
select optimal GT, eliminating the need for arduous manual
annotation, and (b) act as a finetuning objective to improve
aesthetic quality of super-resolved images.
3. Analysis of NR-IQA metrics
We analyze the alignment between various NR-IQA metrics
and
human
judgements
for
assessing
the
aesthetic
quality
of
super-resolved
images.
We
report
our
analysis
on
two
publicly available datasets, SBS180K [
43
] and HGGT [
12
].
3.1. Analysis on the SBS180K dataset
We
analyze
various
NR-IQA
metrics
on
SBS180K
[
43
],
a
large scale human preference dataset for super-resolved im-
ages,
containing
167,019
train
and
9,421
test
image
pairs.
Each pair is annotated with a single score in the range [0,1],
depicting
the
fraction
of
human
annotators
preferring
the
aesthetic
quality
of
the
second
image
over
the
first
one.
Each
pair
consists
of
two
super-resolved
versions
of
the
same low-resolution image, with each version generated us-
ing a different SR model. Due to the large size of SBS180K,
we
analyze
the
metrics
in
two
phases.
In
phase
I,
we
ana-
lyze 20 NR-IQA metrics and their variants (total 42 metrics)
on a small subset of the train set. In phase II, we analyze the
top
7
NR-IQA
metrics
from
phase
I
on
the
complete
train
and test sets.
We use
IQA-PyTorch
[
10
], an open-source
toolbox for image quality assessment.
Phase I.
There are 404 unique pairs of compared SR models
in the train set. We randomly select 3 image pairs per model
comparison, yielding a subset of 1212 images.
We evaluate
the following NR-IQA metrics on this subset: Q-Align [
88
],
LIQE [
101
], ARNIQA [
1
], TOPIQ [
11
], TReS [
29
], CLIP-
IQA(+) [
82
], MANIQA [
94
], MUSIQ [
41
], DBCNN [
100
],
PaQ-2-PiQ
[
97
],
HyperIQA
[
72
],
NIMA
[
74
],
WaDIQaM
[
8
],
CNNIQA
[
104
],
NRQM
[
57
],
PI
(Perceptual
Index)
[
7
],
BRISQUE
[
59
],
ILNIQE
and
NIQE
[
98
],
and
PIQE
[
81
].
When
available,
we
also
consider
multiple
variants
of
these
metrics
offered
by
IQA-PyTorch
(e.g.,
metrics
trained on different IQA datasets).
We assess the accuracy
of
each
metric
in
terms
of
whether,
given
an
image-pair,
the metric prefers the same image as the humans prefer col-
lectively
or
not.
Table
1
shows
results
of
only
the
top
7
metrics:
PaQ-2-PiQ, NIMA, MUSIQ, LIQE, ARNIQA, Q-
Align, TOPIQ-NR. Complete results are given in Supp. Ta-
ble
6
.
Phase
II.
We
evaluate
the
top
7
metrics
from
Phase
I
on
the
complete
train
and
test
sets,
excluding
pairs
with
no
consensus among human annotators (score of 0.5).
Results
are given in Table
2
, suggesting that MUSIQ has relatively
higher accuracy on both train and test sets compared to other
metrics. While PaQ-2-PiQ and Q-Align have slightly higher
accuracy
(0.45%
and
0.04%,
respectively)
on
the
train
set,
MUSIQ
outperforms
them
on
the
test
set
by
a
large
mar-
gin
(2.16%
and
2.05%,
respectively).
We
further
analyze
performance
of
the
remaining
six
metrics
on
the
samples
where
MUSIQ
fails
(excluding
pairs
with
a
score
of
0.5).
Results
are
shown
in
Table
3
.
We
find
that
NIMA
and
Q-
Align
achieve
higher
accuracy
on
these
samples
compared
2313
to
other
four
metrics.
Since
they
complement
MUSIQ,
in
§
5
, we report performance on MUSIQ, NIMA and Q-Align.
3.2. Analysis on HGGT dataset
We analyze the seven selected metrics of Phase I above on
the
HGGT
[
12
]
dataset,
containing
20,193
quintuplets
of
HR image patches. Each quintuplet contains an original HR
ground-truth
(GT)
patch
and
four
enhanced
GTs.
Each
of
the
four
enhanced
GTs
in
each
quintuplet
is
annotated
by
human annotators for being better than (‘positive’), similar
to
(‘similar’),
or
worse
than
(‘negative’)
the
original
GT.
While
‘positive’
labels
are
abundant,
‘negative’
labels
are
rare.
Out
of
20,193,
only
1,270
quintuplets
have
at
least
one ‘negative’. We analyze the seven metrics on this subset.
We
evaluate
the
NR-IQA
metrics
based
on
the
average
Spearman rank correlation coefficient, and positive and neg-
ative
misalignment
rates.
Assuming
higher
rank
is
better,
we define positive (negative) misalignment rate as the frac-
tion of quintuplets where at least one positive (negative) GT
is
ranked
lower
(higher)
than
at
least
one
similar
GT.
We
show
results
in
Table
4
.
Note
that
all
metrics
have
poor
negative
misalignment
rate,
leading
to
low
Spearman
cor-
relations.
We
believe
that
NR-IQA
metrics
fail
to
recog-
nize negative GTs, since they may not necessarily have low
quality
(recall
that
all
are
enhanced
GTs),
or
may
have
ar-
tifacts
that
are
unrecognizable
without
a
reference
image.
Nonetheless, MUSIQ has the lowest positive misalignment
rate.
Hence,
we
use
MUSIQ
in
§
4
for
weighted
sampling
of
the
GTs
and
direct
optimization
(see
also
Supp.
§
7.1
).
Since TOPIQ has the 2nd lowest positive misalignment rate
after MUSIQ, we include it as an evaluation metric in §
5
.
4. Methods
We next explore how to improve existing SR methods with
the
results
of
our
findings.
Since
our
interest
is
in
percep-
tual
quality
and
its
use
in
multimodal
SR,
we
build
upon
recent
work,
Human
Guided
Ground-truth
(HGGT)
[
12
],
which
constructs
a
set
of
ground-truth
images
per
input,
with varying quality, and uses human tests to rank their rela-
tive quality.
We begin by reviewing HGGT [
12
] (§
4.1
), and
then
discuss
two
methods
of
applying
neural
IQA
models
to augment it:
altering the choice of ground-truth set based
on an automated IQA weight (§
4.2
) and directly optimizing
the IQA model in a fine-tuning step (§
4.3
) .
4.1. Background
As
discussed
in
§
3.2
,
the
HGGT
dataset
includes
(i)
a
set
of images (“originals”), (ii) a set of four super-resolved ver-
sions
of
each
original
(“enhanced
GTs”),
and
(iii)
human
annotations for each enhanced GT (“positive”, “similar”, or
“negative”
meaning
better
than,
indistinguishable
from,
or
Original GT
MUSIQ = 31.76
Positive GT 1
MUSIQ = 36.13
Positive GT 2
MUSIQ = 54.19
Figure 2.
Fine-Grained Comparison via NR-IQA.
MUSIQ can
differentiate the quality of two images, both marked as “positive”
by
human
annotators.
Higher
MUSIQ
indicates
higher
quality
(zoom
for
details).
Unlike
HGGT
models,
which
utilize
a
uni-
form distribution over positives,
our approach enables differently
weighting them (§
4.2
).
worse than the original).
The set of positives provides
mul-
timodal
supervision,
since
each
one
is
a
disparate
yet
rea-
sonable GT for learning.
The HGGT work [
12
] then shows
that utilizing these synthetic GTs is useful for SR training,
exploring
several
neural
architectures
and
degradation
set-
tings.
While
HGGT
explores
several
variants
for
utilizing
their human labels,
we focus on the simple but highly per-
forming “positives-only” scenario, which performs equiva-
lently or better than the variants utilizing negatives.
In this
scenario,
at
each
training
iteration,
every
input
image
su-
pervises the network with a GT chosen
uniformly randomly
from
the
positives.
As
is
relatively
standard
in
SR
(e.g.,
[
37
,
85
]), HGGT models are trained with a combined loss:
L
(
✓
|
b
I, I
) =
λ
`
1
||
I
−
b
I
||
1
+
λ
P
d
P
(
b
I, I
) +
λ
A
D
(
b
I
)
,
(1)
where
b
I
=
f
✓
(
I
LQ
)
is
the
SR
estimate
of
the
low-
resolution
(or
low-quality)
input
I
LQ
,
via
SR
network
f
✓
,
I
⇠U
I
LQ
[
{
I
1
, . . . , I
n
}
]
is
the
randomly
chosen
GT
(from
the
set
of
positives
corresponding
to
image
I
LQ
),
d
P
is
a
perceptual loss, and
D
is an adversarial discriminator.
However,
HGGT
requires
human
labels,
which
are
dif-
ficult to scale and often domain-dependent.
In contrast, we
explore
the
opportunities
afforded
by
neural
no-reference
image
quality
assessment
(NR-IQA)
models,
which
not
only
eschew
human
labels,
but
also
confer
additional
ca-
pabilities – namely, the ability to provide more fine-grained
non-uniform
sampling
weights
(§
4.2
)
and
to
enable
direct
optimization via differentiability (§
4.3
).
4.2. Reweighted Sampling
We
explore
a
few
straightforward
alternatives
to
uniform
sampling of the positives, via an IQA model.
In particular,
consider the following simple formulation:
I
⇠P
[
S
I
|
SoftMax
⌧
(
Q
(
S
I
))]
,
(2)
Q
(
S
I
) =
{
Q
(
I
1
)
, . . . , Q
(
I
n
)
}
,
(3)
S
I
=
{
I
1
, . . . , I
n
} 2 {
A
I
, P
I
}
(4)
2314
Method
PaQ-2-PiQ [
97
]
NIMA
†
[
74
]
LIQE
~
[
101
]
ARNIQA-TID
?
[
1
]
Q-Align
⇧
[
88
]
TOPIQ-NR [
11
]
Train Acc.
(%)
37.58
44.21
31.50
42.05
42.86
25.71
Test Acc.
(%)
33.10
39.70
30.70
38.60
41.06
30.89
Table
3.
Phase
II
analysis
on
SBS180K
(continued).
Accuracy
of
top
6
NR-IQA
metrics
on
consensus
samples
(from
train
plus
test)
where MUSIQ fails.
†~
?
⇧
See Table
1
.
Method
PaQ-2-PiQ [
97
]
NIMA
†
[
74
]
MUSIQ
/
[
41
]
LIQE
~
[
101
]
ARNIQA-TID
?
[
1
]
Q-Align
⇧
[
88
]
TOPIQ-NR [
11
]
SRC
"
0.10
0.17
0.17
0.03
0.28
0.20
0.09
PM
#
0.26
0.28
0.16
0.58
0.51
0.39
0.21
NM
#
0.85
0.79
0.95
0.64
0.45
0.63
0.97
Table 4.
Analysis on
HGGT
subset.
We
evaluate
Spearman’s rank correlation coefficient (SRC), positive misalignment (PM)
rate,
and
negative misalignment (NM) rate.
†
/
~
?
⇧
.
See Table
1
.
where
I
is
the
sampled
GT,
⌧>
0
is
the
softmax
tem-
perature,
Q
is
the
NR-IQA
model
(higher
is
better),
P
is
a
discrete
distribution
over
elements
of
S
I
(weighted
by
SoftMax
⌧
(
Q
(
S
I
))
),
and
S
I
is
the
set
of
possible
GTs
(ei-
ther
choosing
from
all
candidates,
enhanced
and
original,
denoted
A
I
,
or
just
positive
ones,
P
I
).
The
HGGT
algo-
rithm
simply
uses
S
I
=
P
I
and
⌧
!
1
(i.e.,
the uniform
distribution);
we explore different combinations,
including
⌧
!
0
(the
arg max
choice). We illustrate the utility of NR-
IQA-based sampling (as opposed to uniform) in Fig.
2
, dis-
playing an example that humans rank equivalently as posi-
tive, yet is more precisely distinguished by the neural asses-
sor.
We consider three NR-IQA-based sampling scenarios.
Softmax-All
(SMA).
Given
the
set
of
all
GTs
(i.e.,
S
I
=
A
I
),
we
use
an
IQA-weighted
distribution
over
GTs.
This
setting uses no human data, and simply randomly chooses a
GT at each iteration with a weight proportional to softmax-
rescaled quality.
We set
⌧
to ensure a distribution between
uniform and Kronecker delta (i.e., argmax).
Softmax-Positives
(SMP).
This
approach
actually
builds
on the human data in HGGT, using the softmax-normalized
IQA
scores
but
only
of
the
positives
(i.e.,
S
I
=
P
I
).
This
setting
is
the
most
similar
to
the
HGGT
positives-only
(or
uniform
distribution
on
positives),
just
with
non-uniform
weights
(based
on
⌧
).
We
expect
this
to
outperform
SMA
sampling, as it has access to direct human preferences.
Argmax-online
(AMO).
The
use
of
a
neural
IQA
model
confers
an
additional
capability
that
human
data
lacks:
we
can
dynamically
determine
sampling
weights
for
new
patches
at
training
time.
In
previous
scenarios,
at
training
time, we first pick one GT out of the four (Eq.
3
), followed
by
random
patch
sampling
from
the
selected
GT.
Instead,
in
the
Argmax-online
(AMO)
scenario,
we
first
sample
a
random patch from each GT, followed by selecting the best
patch.
To
be
specific,
we
sample
one
patch
from
the
same
random
location
from
each
GT,
then
run
Q
on
each
patch,
and
choose
the
best
one
(i.e.,
the
arg max
of
Q
values,
so
Figure
3.
Structured
Optimization
Noise.
Optimizing
via
an
NR-IQA metric (MUSIQ [
41
]) generates structured artifacts (left),
similar to an adversarial attack, while utilizing LoRA removes this
noise (right; see §
4.3
and Supp. §
10.2
).
Zoom in for details.
⌧
!
0
).
We
eschew
human
data;
hence,
S
I
=
A
I
.
This
enables a more fine-grained judgment (since quality is com-
puted at the
patch
level), whereas human annotations cannot
necessarily be so easily extrapolated.
4.3. Direct Optimization
Given a differentiable image quality estimator,
Q
, an obvi-
ous
approach
to
improving
our
SR
model
is
to
simply
in-
clude
Q
in our objective function.
To some extent, this has
already been explored for unconditional generative models
(e.g.,
[
16
]).
However,
when
Q
is
a
neural
network
with
many
parameters,
this
is
unlikely
to
succeed;
in
essence,
gradient
descent
will
act
like
an
“adversarial
attack”
on
Q
(e.g.,
[
31
]).
It
is
well-known
that
such
“attacks”
are
often
able
to
dramatically
alter
the
output
of
the
objective
net-
work (say, a classifier), while changing the optimized input
in unintuitive or imperceptible ways (e.g., [
45
,
73
]); in SR,
this could conceivably manifest as artifacts that fool
Q
into
providing
a
high
score,
since
NR-IQA
models
are
known
to be susceptible to attacks (e.g., [
33
,
55
]).
In fact, without
additional regularization, this is precisely what happens:
in
Fig.
3
, we display the artifacts that appear when an SR net-
work is naively fine-tuned with
Q
(see also Supp. §
10.2
).
Thus,
inspired
by
prior
work
[
16
],
we
utilize
low-rank
adaptation (LoRA) [
36
] to regularize the optimization.
For-
mally,
we
continue
training
as
normal,
but
only
on
the
LoRA weights, plus an additional NR-IQA loss term:
e
L
(
φ
|
b
I, I
) =
L
(
φ
|
b
I, I
)
−
λ
Q
Q
(
b
I
)
,
(5)
2315
where
φ
are the LoRA parameters,
b
I
=
f
✓,φ
(
I
LQ
)
,
Q
is an
NR-IQA
model
(where
higher
is
better),
and
L
is
defined
in Eq.
1
.
Unless otherwise specified, we set
λ
A
=
0
when
fine-tuning, since we are already including an image quality
term
(for
which
the
critic
normally
acts;
though
see
§
5.2
).
See Supp. §
8
for additional details.
5. Experiments
Setup.
We consider the HGGT dataset under two degrada-
tion settings for the super-resolution (SR) problem:
(a) the
standard Real-ESRGAN scenario (two random degradation
rounds) [
85
] and (b) a simplified setting with a single round,
used
in
the
HGGT
paper.
We
use
the
ESRGAN
(RRDB-
based) architecture [
84
,
85
] for (a) and SwinIR [
50
] for (b).
Unless
otherwise
stated,
we
use
the
same
training
settings
as HGGT. Based on our analysis in §
3
, unless noted other-
wise,
we
use
MUSIQ
[
41
]
as
our
NR
IQA
model,
Q
,
for
both weighted sampling and direct optimization.
Evaluation.
Following
HGGT
[
12
],
we
utilize
their
Test-
100
held-out
images
for
evaluation.
We
utilize
two
low-
level
distortion
metrics,
PSNR
and
SSIM
[
86
].
We
also
use
three
FR
models,
which
act
as
mid-level
visual
met-
rics: LPIPS [
102
], LPIPS-ST [
28
], and DISTS [
20
]. LPIPS-
ST
[
28
]
is
a
shift-tolerant
form
of
LPIPS
[
102
],
improv-
ing
robustness
to
small
translations
imperceptible
to
hu-
mans but highly damaging to distortion measures.
Finally,
we
apply
four
NR-IQA
metrics.
Since
we
are
interested
in
differentiating
high-quality
images,
we
choose
MUSIQ
[
41
]
and
TOPIQ
[
11
]
as
they
have
the
best
positive
mis-
alignment
scores
(see
Table
4
).
We
also
include
NIMA
[
74
] and Q-Align [
88
] based on their complementarity with
MUSIQ (see §
3.1
), which is directly optimized.
Since Test-
100
has
multiple
positive
GTs
per
image,
evaluations
with
reference-based metrics are averaged across all positives.
Baselines.
We
compare
to
the
SOTA
“positives-only”
model
for
HGGT,
which
we
denote
UPos,
as
it
uses
a
uniform
distribution
over
positive
samples.
We
include
two
human-annotation-free
baselines
that
do
not
incorpo-
rate NR-IQA: “OrigsOnly”, which trains only with original
(non-enhanced) GT, and “Rand”,
which randomly chooses
a supervisory image from among
all
potential GTs.
For our
methods,
we
can
choose
(a)
an
IQA-based
sampling
type
and (b) IQA-based fine-tuning settings.
The different sam-
pling methodologies (SMA, SMP, and AMO) are described
in
§
4.2
,
while
we
denote
the
use
of
fine-tuning
(see
§
4.3
)
with
the
“FT”
moniker.
We
also
consider
two
main
FT
variations, FT
IG
and FT
HP
, described in §
5.2
.
Our primary
method combines the best settings for both IQA-based sam-
pling and optimization:
the AMO+FT scheme.
5.1. Empirical Results
Our results on HGGT Test-100 are displayed in Table
5
and
Fig.
4
.
See Supp. §
11
for RealSR results as well.
Multimodal Training Boosts Performance.
As in HGGT,
we can see the impact of enhanced GTs by comparing Orig-
sOnly to UPos, which has greatly improved perceptual qual-
ity (LPIPS, DIST, and NR-IQA). We also consider the Rand
baseline,
showing
that
even
randomly
sampling
enhanced
GTs
is
helpful
for
perceptual
quality
but
not
sufficient
to
reach
UPos
performance,
the
SoTA
method
from
HGGT,
enabled by
human
filtering of low-quality GTs.
Neural
IQA
Sampling
Outperforms
Human
Rankings.
We
next
investigate
whether
IQA-based
sampling
(SMA,
SMP,
and
AMO)
can
outperform
UPos,
which
relies
on
human
rankings.
On
SwinIR,
LPIPS
and
DISTS
remain
largely
unchanged,
but
LPIPS-ST
and
the
NR
metrics
show
small
improvements,
especially
for
AMO.
On
Real-
ESRGAN,
sampling
with
IQA
improves
both
LPIPS
and
LPIPS-ST, but only AMO shows substantial improvements
on the NR metrics. Surprisingly, despite access to human la-
bels, SMP is very similar to SMA, maintaining nearly iden-
tical performance on low and mid level distortion measures,
with a marginal boost in NR image quality.
In general, we
find
AMO
is
consistently
superior
to
both
SMA
and
SMP,
which
suggests
that
selecting
for
quality
(especially
at
the
fine-grained
level
demanded
in
the
online
setting)
is
more
important
than
simply
having
multiple
GTs;
AMO
is
also
measurably better than UPos in terms of perceptual quality,
despite the lack of access to human annotation.
IQA
Fine-tuning
Improves
both
Human
and
Neural
Sampling.
We
examine
the
impact
of
fine-tuning
(FT)
on
NR-IQA.
When
used
on
top
of
human
data,
denoted
UPos+FT,
LPIPS
and
DISTS
are
the
same
(SwinIR)
or
slightly
worse
(RealESRGAN),
but
NR-IQA
metrics
uni-
formly
improve,
as
well
as,
interestingly,
LPIPS-ST.
This
trade-off
of
mid-level
perceptual
distortion
for
high-level
quality
is
effectively
an
extension
of
the
previously
ob-
served
balance,
between
pixel-level
distortion
versus
per-
ceptual metrics.
Indeed,
we still see the latter compromise
here,
in
that
FT
always
damages
low-level
distortion
met-
rics
(PSNR
and
SSIM),
despite
the
increases
in
perceptual
quality.
This is expected, since NR-based FT does not opti-
mize to a particular SR solution, let alone the one(s) in the
dataset.
In addition, the results with LPIPS-ST indicate that
it
is
more
perceptual
than
LPIPS
or
DIST
(i.e.,
more
NR-
IQA-like,
though
still
full-reference).
Overall,
the
perfor-
mance
boosts
with
FT
suggest
that
useful
information
can
be extracted from neural NR-IQA models, even on top of a
model
with
access
to
human
annotations,
providing
a
sim-
ple mechanism for improving image quality in SR models.
Upper-bounding
NR-IQA
Evaluation
Performance.
In
addition,
we
compute
“gold
standard”
NR-IQA
values
for
the
GT
Test
data,
taking
the
best
score
among
the
original
and enhanced images, providing a soft upper-bound on NR
scores, if one were able to exactly reproduce the “best” GT
via the SR network. We find that altered sampling generally
2316
24.42
56.62
60.86
65.14
73.12
47.59
71.13
74.13
75.78
77.31
LR
Original GT
SwinIR-UPos
SwinIR-AMO
SwinIR-AMO+FT
41.46
52.51
54.92
66.12
77.07
33.52
56.01
63.01
67.50
71.15
LR
Original GT
RESRGAN-UPos
RESRGAN-AMO
RESRGAN-AMO+FT
Figure 4.
Qualitative results with NR-IQA Guidance.
Following the notation of Table
5
,
columns 3-5 are (top 2 rows) SwinIR-UPos,
SwinIR-AMO,
and
SwinIR-AMO
+
FT,
and
(bottom
2
rows)
Real-ESRGAN-UPos,
Real-ESRGAN-AMO,
and
Real-ESRGAN-AMO
+
FT.
We
show
MUSIQ
scores
in
insets.
Qualitatively,
we
see
improved
performance
as
we
move
from
‘UPos’
to
‘AMO’
to
‘AMO-FT’,
showcasing superiority of each method over the previous one.
Zoom in for details.
See also Supp. §
10.1
for additional examples.
does not reach these values, but FT is able to reach and even
surpass them in several scenarios.
Superior
Perceptual
Quality
via
Neural
Sampling
and
Fine-Tuning.
Finally,
we
test
the
natural
unification
of
IQA-based
neural
sampling
with
IQA-driven
FT,
denoted
by the AMO+FT setting.
We find that this combination sur-
passes the SoTA HGGT approach (UPos),
in terms of per-
ceptual
quality,
despite
not
using
any
human
annotations
.
Specifically,
for
SwinIR,
AMO+FT
incurs
a
small
penalty
(
⇠
3-4%)
on
LPIPS
and
DISTS,
but
improves
LPIPS-ST
by
⇠
6%
and
is
superior
to
every
other
method
according
to
NR
metrics.
In
the
RealESRGAN
setting,
compared
to
UPos,
AMO+FT
again
obtains
a
large
improvement
on
LPIPS-ST (
⇠
12%), with neglible changes on the other mid-
level
metrics
(
⇠
2%).
It
also
soundly
surpasses
UPos
ac-
cording
to
every
NR
metric.
Interestingly,
RealESRGAN-
AMO+FT
does
not
outperform
RealESRGAN-UPos+FT;
however,
note
that
the
latter
has
access
to
human
annota-
tions.
We also present a user study in Supp. §
12
, finding a
preference
for
AMO+FT
over
UPos.
Altogether,
these
re-
sults suggest (i) human annotated rankings on multiple GTs,
at
least
when
used
naively
for
SR
training,
can
be
easily
surpassed
via
neural
NR-IQA
scores,
and
(ii)
considerable
improvements
to
the
perceptual
quality
of
SR
models
can
be
attained
through
automated
means,
by
simply
applying
existing NR-IQA models.
2317
Model
FR Low-Lev. Dist.
FR Mid-Lev. Dist.
NR High-Lev.
Perceptual Quality
PSNR
"
SSIM
"
LPIPS
#
LPIPS-ST
#
DISTS
#
MUSIQ
"
NIMA
"
Q-Align
"
TOPIQ
"
Gold Standard
7
–
–
–
–
–
69.64
5.28
3.78
0.69
SwinIR-OrigsOnly
3
22.72
0.652
0.227
0.174
0.162
59.47
4.87
3.17
0.48
SwinIR-Rand
3
22.45
0.650
0.180
0.139
0.131
65.27
5.11
3.52
0.59
SwinIR-UPos
⇤
7
22.30
0.647
0.169
0.129
0.123
66.39
5.16
3.56
0.62
SwinIR-SMA
3
22.27
0.646
0.171
0.129
0.124
66.73
5.16
3.60
0.63
SwinIR-SMP
7
22.29
0.647
0.171
0.130
0.124
66.83
5.17
3.62
0.62
SwinIR-AMO
3
22.08
0.641
0.167
0.124
0.123
68.08
5.21
3.67
0.66
SwinIR-UPos + FT
HP
7
22.17
0.642
0.166
0.123
0.122
68.38
5.23
3.64
0.65
SwinIR-UPos + FT
IG
7
22.03
0.635
0.168
0.122
0.123
69.37
5.24
3.69
0.66
SwinIR-UPos + FT
7
22.01
0.633
0.169
0.123
0.124
69.70
5.26
3.70
0.67
SwinIR-AMO + FT
3
21.77
0.624
0.174
0.121
0.128
70.81
5.29
3.75
0.70
RESRGAN-OrigsOnly
3
22.10
0.618
0.283
0.229
0.185
57.91
4.84
2.99
0.46
RESRGAN-Rand
3
21.66
0.611
0.234
0.190
0.160
64.82
5.18
3.40
0.60
RESRGAN-UPos
⇤
7
21.54
0.608
0.233
0.192
0.158
65.93
5.25
3.47
0.63
RESRGAN-SMA
3
21.46
0.606
0.227
0.182
0.157
65.87
5.23
3.46
0.63
RESRGAN-SMP
7
21.44
0.607
0.226
0.182
0.156
66.66
5.24
3.51
0.64
RESRGAN-AMO
3
21.28
0.602
0.224
0.178
0.156
67.86
5.29
3.56
0.66
RESRGAN-UPos + FT
HP
7
21.30
0.595
0.226
0.175
0.158
70.28
5.32
3.65
0.69
RESRGAN-UPos + FT
IG
7
21.14
0.586
0.236
0.182
0.160
72.01
5.35
3.70
0.70
RESRGAN-UPos + FT
7
21.09
0.580
0.235
0.179
0.163
72.69
5.37
3.69
0.71
RESRGAN-AMO + FT
3
21.02
0.581
0.228
0.169
0.161
71.67
5.35
3.68
0.71
Table 5.
Evaluation on held-out HGGT Test-100.
“FR Low-Lev Dist” refers to full-reference low-level distance metrics; “FR Mid-Lev
Dist”
and
“NR
High-Lev.
Perceptual
Quality”
refer
to
full-reference
and
no-reference
perceptual
metrics,
respectively.
Second
column
(
)
indicates
that
a
method
works
with
no
human
GT
ranking
data
(
3
),
or
requires
such
GT
annotations
(
7
).
“Gold
Standard”
shows
the
average
of
best
metric
value
per
quintuplet
of
test
GTs.
“OrigsOnly”
means
no
multimodal
supervision
(no
enhanced
GT).
“Rand”
signifies random GT choice (from both enhanced and original), which requires no human annotation, while “UPos” denotes the “positives-
only”
scenario
(uniform
sampling
from
human-ranked
positives),
the
SoTA
baseline
method
from
HGGT
(marked
by
⇤
).
“FT”
refers
to
fine-tuning (direct optimization); “FT
HP
” denotes using a higher perceptual loss weight (
λ
P
), and “FT
IG
” the inclusion of GAN loss.
5.2. Ablations and Variations
IQA
Optimization
Refines
the
Perception–Distortion
Trade-off.
As
noted,
our
results
show
a
trade-off
between
mid-level
perceptual
and
NR-IQA
metrics,
reminiscent
of
the classic perception–distortion curve [
5
,
6
] (which we also
observe,
via
PSNR
and
SSIM).
We
can
control
this
mid-
versus-high-level
perceptual
tradeoff,
by
simply
changing
the
FT
loss
weights.
For
instance,
comparing
UPos+FT
to UPos+FT
HP
(which increases
λ
P
), we see that mid-level
metrics all improve, while all NR metrics decline.
Discriminators as IQA.
One can naturally interpret the dis-
criminator (or critic) of a GAN as a form of NR-IQA model
– in fact, it is one specialized to the errors and artifacts of the
SR
function
we
are
training.
However,
the
FT
IG
scenario,
which keeps the GAN loss during FT, does not greatly im-
pact results (compared to the FT setting);
in fact, it largely
induces very slight declines.
We also tried UPos+FT with-
out
an
NR-IQA
model,
instead
simply
upweighting
the
GAN
loss
(treating
the
critic
as
an
IQA
model);
however,
this results in uniformly worse NR scores, with little change
to mid-level metrics (see Supp. §
9
).
Alternative
NR-IQA.
While
we
selected
MUSIQ
based
on
our
analysis
(§
3
),
we
also
tested
FT
with
an
alternative
IQA model,
PaQ-2-PiQ, based on its high score in Table
2
(see Supp. §
9
).
We find that NIMA, Q-Align, TOPIQ, and
(unsurprisingly)
MUSIQ
all
decline,
for
both
SwinIR
and
RealESRGAN. Nevertheless,
it is plausible that a different
IQA
model
(or
combination
thereof),
particularly
if
fine-
tuned for SR, would provide a superior learning signal.
6. Conclusion
As
an
ill-posed
inverse
problem,
SR
struggles
with
the
di-
chotomy between perceptual quality and reference fidelity.
Prior research utilized multiple GTs and human annotations
to
mitigate
this
trade-off.
In
contrast,
herein,
we
focus
on
improving perceptual quality via neural IQA, enabling us to
eschew
human
annotations.
We
first
analyze
existing
NR-
IQA methods, discerning a candidate for adoption in train-
ing, as well as complementary models for evaluation.
Then,
we
devised
two
ways
to
apply
NR-IQA
to
SR
training:
(i)
IQA-weighted multimodal GT sampling and (ii) regularized
optimization
of
NR
quality.
When
jointly
utilized,
our
ap-
proach outperforms the existing SoTA, which relies on hu-
man data, in terms of NR metrics, without sacrificing mid-
level reference-based quality scores.
We hope it enables fu-
ture
investigation
into
NR-IQA
for
SR,
the
connection
be-
tween generative modelling and IQA, and SR with domain
shift, as IQA does not need paired GT.
2318
References
[1]
Lorenzo
Agnolucci,
Leonardo
Galteri,
Marco
Bertini,
and
Alberto
Del
Bimbo.
ARNIQA:
Learning
distortion
mani-
fold for image quality assessment.
In
Winter Conference on
Applications of Computer Vision (WACV)
, 2024.
3
,
5
[2]
Namhyuk
Ahn,
Byungkon
Kang,
and
Kyung-Ah
Sohn.
Fast,
accurate,
and
lightweight
super-resolution
with
cas-
cading residual network.
In
European Conference on Com-
puter Vision (ECCV)
, 2018.
2
[3]
Yuval Bahat and Tomer Michaeli.
Explorable super resolu-
tion.
In
IEEE Conference on Computer Vision and Pattern
Recognition (CVPR)
, 2020.
2
[4]
Simon Baker and Takeo Kanade. Limits on super-resolution
and how to break them.
IEEE Transactions on Pattern Anal-
ysis and Machine Intelligence (PAMI)
, 2002.
1
[5]
Yochai Blau and Tomer Michaeli. The perception-distortion
tradeoff.
In
IEEE Conference on Computer Vision and Pat-
tern Recognition (CVPR)
, 2018.
2
,
8
,
3
,
5
[6]
Yochai
Blau
and
Tomer
Michaeli.
Rethinking
lossy
com-
pression:
The rate-distortion-perception tradeoff.
In
Inter-
national
Conference
on
Machine
Learning
(ICML)
,
2019.
2
,
8
[7]
Yochai
Blau,
Roey
Mechrez,
Radu
Timofte,
Tomer
Michaeli,
and
Lihi
Zelnik-Manor.
The
2018
PIRM
chal-
lenge
on
perceptual
image
super-resolution.
In
European
Conference
on
Computer
Vision
Workshops
(ECCVW)
,
2018.
3
[8]
Sebastian Bosse, Dominique Maniry, Klaus-Robert M¨uller,
Thomas Wiegand,
and Wojciech Samek.
Deep neural net-
works for no-reference and full-reference image quality as-
sessment.
IEEE
Transactions
on
Image
Processing
(TIP)
,
2017.
3
[9]
Jianrui
Cai,
Hui
Zeng,
Hongwei
Yong,
Zisheng
Cao,
and
Lei
Zhang.
Toward
real-world
single
image
super-
resolution:
A
new
benchmark
and
a
new
model.
In
Inter-
national Conference on Computer Vision (ICCV)
, 2019.
3
,
5
[10]
Chaofeng
Chen
and
Jiadi
Mo.
IQA-PyTorch:
Pytorch
toolbox
for
image
quality
assessment.
[Online].
Avail-
able:
https://github.com/chaofengc/IQA-
PyTorch
, 2022.
3
[11]
Chaofeng
Chen,
Jiadi
Mo,
Jingwen
Hou,
Haoning
Wu,
Liang
Liao,
Wenxiu
Sun,
Qiong
Yan,
and
Weisi
Lin.
TOPIQ: A top-down approach from semantics to distortions
for image quality assessment.
IEEE Transactions on Image
Processing (TIP)
, 2024.
3
,
5
,
6
,
1
[12]
Du Chen, Jie Liang, Xindong Zhang, Ming Liu, Hui Zeng,
and Lei Zhang.
Human guided ground-truth generation for
realistic
image
super-resolution.
In
IEEE
Conference
on
Computer
Vision
and
Pattern
Recognition
(CVPR)
,
2023.
2
,
3
,
4
,
6
[13]
Liangyu Chen, Xiaojie Chu, Xiangyu Zhang, and Jian Sun.
Simple baselines for image restoration.
In
European Con-
ference on Computer Vision (ECCV)
, 2022.
2
[14]
Xiangyu
Chen,
Xintao
Wang,
Jiantao
Zhou,
Yu
Qiao,
and
Chao
Dong.
Activating
more
pixels
in
image
super-
resolution
transformer.
In
IEEE
Conference
on
Computer
Vision and Pattern Recognition (CVPR)
, 2023.
2
[15]
Zheng Chen, Zongwei Wu, Eduard Zamfir, Kai Zhang, Yu-
lun
Zhang,
Radu
Timofte,
Xiaokang
Yang,
Hongyuan
Yu,
Cheng Wan, Yuxin Hong, Zhijuan Huang, Yajun Zou, Yuan
Huang,
Jiamin
Lin,
Bingnan
Han,
Xianyu
Guan,
Yong-
sheng
Yu,
Daoan
Zhang,
Xuanwu
Yin,
Kunlong
Zuo,
Jin-
hua
Hao,
Kai
Zhao,
Kun
Yuan,
Ming
Sun,
Chao
Zhou,
Hongyu
An,
Xinfeng
Zhang,
Zhiyuan
Song,
Ziyue
Dong,
Qing Zhao, Xiaogang Xu, Pengxu Wei, Zhi chao Dou, Gui
ling
Wang,
Chih-Chung
Hsu,
Chia-Ming
Lee,
Yi-Shiuan
Chou,
Cansu
Korkmaz,
A.
Murat
Tekalp,
Yubin
Wei,
Xi-
aole
Yan,
Binren
Li,
Haonan
Chen,
Siqi
Zhang,
Sihan
Chen,
Amogh
Joshi,
Nikhil
Akalwadi,
Sampada
Malagi,
Palani
Yashaswini,
Chaitra
Desai,
Ramesh
Ashok
Tabib,
Ujwala
Patil,
Uma
Mudenagudi,
Anjali
Sarvaiya,
Pooja
Choksy,
Jagrit
Joshi,
Shubh
Kawa,
Kishor
Upla,
Sushrut
Patwardhan,
Raghavendra
Ramachandra,
Sadat
Hossain,
Geongi
Park,
S.
M.
Nadim
Uddin,
Hao
Xu,
Yanhui
Guo,
Aman
Urumbekov,
Xingzhuo
Yan,
Wei
Hao,
Minghan
Fu,
Isaac
Orais,
Samuel
Smith,
Ying
Liu,
Wangwang
Jia,
Qisheng
Xu,
Kele
Xu,
Weijun
Yuan,
Zhan
Li,
Wenqin
Kuang, Ruijin Guan, Ruting Deng, Zhao Zhang, Bo Wang,
Suiyi
Zhao,
Yan
Luo,
Yanyan
Wei,
Asif
Hussain
Khan,
Christian Micheloni, and Niki Martinel.
NTIRE 2024 chal-
lenge on image super-resolution (
⇥
4): Methods and results,
2024.
2
[16]
Kevin Clark, Paul Vicol, Kevin Swersky, and David J Fleet.
Directly
fine-tuning
diffusion
models
on
differentiable
re-
wards.
arXiv preprint arXiv:2309.17400
, 2023.
2
,
3
,
5
[17]
Mauricio Delbracio and Peyman Milanfar.
Inversion by di-
rect iteration:
An alternative to denoising diffusion for im-
age restoration.
arXiv preprint arXiv:2303.11435
, 2023.
2
[18]
Keyan Ding, Kede Ma, Shiqi Wang, and Eero P Simoncelli.
Image
quality
assessment:
Unifying
structure
and
texture
similarity.
IEEE Transactions on Pattern Analysis and Ma-
chine Intelligence (PAMI)
, 2020.
2
[19]
Keyan Ding, Kede Ma, Shiqi Wang, and Eero P Simoncelli.
Comparison of full-reference image quality models for op-
timization of image processing systems.
International Jour-
nal of Computer Vision (IJCV)
, 2021.
2
,
3
[20]
Keyan Ding, Kede Ma, Shiqi Wang, and Eero P. Simoncelli.
Image
quality
assessment:
Unifying
structure
and
texture
similarity.
IEEE Transactions on Pattern Analysis and Ma-
chine Intelligence (PAMI)
, 2022.
6
[21]
Chao
Dong,
Chen
Change
Loy,
Kaiming
He,
and
Xiaoou
Tang.
Learning
a
deep
convolutional
network
for
image
super-resolution.
In
European Conference on Computer Vi-
sion (ECCV)
, 2014.
1
,
2
[22]
Chao Dong, Chen Change Loy, and Xiaoou Tang.
Acceler-
ating the super-resolution convolutional neural network.
In
European
Conference
on
Computer
Vision
(ECCV)
,
2016.
1
,
2
[23]
Ying
Fan,
Olivia
Watkins,
Yuqing
Du,
Hao
Liu,
Moonkyung
Ryu,
Craig
Boutilier,
Pieter
Abbeel,
Mo-
hammad
Ghavamzadeh,
Kangwook
Lee,
and
Kimin
Lee.
DPOK:
reinforcement
learning
for
fine-tuning
text-to-
2319
image diffusion models.
In
Neural Information Processing
Systems (NeurIPS)
, 2023.
2
[24]
William
T
Freeman,
Thouis
R
Jones,
and
Egon
C
Pasztor.
Example-based super-resolution.
IEEE Computer graphics
and Applications
, 2002.
1
[25]
Stephanie
Fu,
Netanel
Tamir,
Shobhita
Sundaram,
Lucy
Chai, Richard Zhang, Tali Dekel, and Phillip Isola.
Dream-
Sim:
Learning new dimensions of human visual similarity
using
synthetic
data.
Neural
Information
Processing
Sys-
tems (NeurIPS)
, 2023.
6
[26]
Kevin Galim,
Wonjun Kang,
Yuchen Zeng,
Hyung Il Koo,
and Kangwook Lee. Parameter-efficient fine-tuning of state
space models.
arXiv preprint arXiv:2410.09016
, 2024.
1
[27]
Sara
Ghazanfari,
Siddharth
Garg,
Prashanth
Krishna-
murthy,
Farshad
Khorrami,
and
Alexandre
Araujo.
R-
LPIPS:
An
adversarially
robust
perceptual
similarity
met-
ric.
arXiv preprint arXiv:2307.15157
, 2023.
2
[28]
Abhijay
Ghildyal
and
Feng
Liu.
Shift-tolerant
perceptual
similarity
metric.
In
European
Conference
on
Computer
Vision (ECCV)
, 2022.
6
[29]
S
Alireza
Golestaneh,
Saba
Dadsetan,
and
Kris
M
Kitani.
No-reference
image
quality
assessment
via
transformers,
relative
ranking,
and
self-consistency.
In
Winter
Confer-
ence on Applications of Computer Vision (WACV)
, 2022.
3
[30]
Ian
Goodfellow,
Jean
Pouget-Abadie,
Mehdi
Mirza,
Bing
Xu,
David
Warde-Farley,
Sherjil
Ozair,
Aaron
Courville,
and
Yoshua
Bengio.
Generative
adversarial
nets.
Neural
Information Processing Systems (NeurIPS)
, 2014.
2
[31]
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy.
Explaining
and
harnessing
adversarial
examples.
arXiv
preprint arXiv:1412.6572
, 2014.
5
[32]
Hang
Guo,
Jinmin
Li,
Tao
Dai,
Zhihao
Ouyang,
Xudong
Ren,
and
Shu-Tao
Xia.
MambaIR:
A
simple
baseline
for
image
restoration
with
state-space
model.
In
European
Conference on Computer Vision (ECCV)
, 2024.
2
[33]
Alexander Gushchin, Khaled Abud, Georgii Bychkov, Eka-
terina Shumitskaya, Anna Chistyakova, Sergey Lavrushkin,
Bader
Rasheed,
Kirill
Malyshev,
Dmitriy
Vatolin,
and
Anastasia Antsiferova. Guardians of image quality: Bench-
marking defenses against adversarial attacks on image qual-
ity metrics.
arXiv preprint arXiv:2408.01541
, 2024.
5
[34]
Martin
Heusel,
Hubert
Ramsauer,
Thomas
Unterthiner,
Bernhard Nessler, and Sepp Hochreiter.
GANs trained by a
two time-scale update rule converge to a local Nash equilib-
rium.
Neural
Information
Processing
Systems
(NeurIPS)
,
2017.
3
[35]
Vlad
Hosu,
Hanhe
Lin,
Tamas
Sziranyi,
and
Dietmar
Saupe. KonIQ-10k: An ecologically valid database for deep
learning of blind image quality assessment.
IEEE Transac-
tions on Image Processing (TIP)
, 2020.
3
,
1
[36]
Edward
J
Hu,
Yelong
Shen,
Phillip
Wallis,
Zeyuan
Allen-
Zhu,
Yuanzhi
Li,
Shean
Wang,
Lu
Wang,
and
Weizhu
Chen.
LoRA: Low-rank adaptation of large language mod-
els.
International Conference on Learning Representations
(ICLR)
, 2022.
5
,
1
[37]
Xiaozhong Ji, Yun Cao, Ying Tai, Chengjie Wang, Jilin Li,
and
Feiyue
Huang.
Real-world
super-resolution
via
ker-
nel
estimation
and
noise
injection.
In
IEEE
Conference
on
Computer
Vision
and
Pattern
Recognition
Workshops
(CVPRW)
, 2020.
4
[38]
Younghyun Jo, Seoung Wug Oh, Peter Vajda, and Seon Joo
Kim. Tackling the ill-posedness of super-resolution through
adaptive
target
generation.
In
IEEE
Conference
on
Com-
puter Vision and Pattern Recognition (CVPR)
, 2021.
2
[39]
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual
losses
for
real-time
style
transfer
and
super-resolution.
In
European
Conference
on
Computer
Vision
(ECCV)
,
2016.
2
[40]
Bahjat
Kawar,
Michael
Elad,
Stefano
Ermon,
and
Jiaming
Song.
Denoising
diffusion
restoration
models.
Neural
In-
formation Processing Systems (NeurIPS)
, 2022.
2
[41]
Junjie Ke, Qifei Wang, Yilin Wang, Peyman Milanfar, and
Feng
Yang.
MUSIQ:
Multi-scale
image
quality
trans-
former.
In
International
Conference
on
Computer
Vision
(ICCV)
, 2021.
2
,
3
,
5
,
6
,
1
[42]
Markus
Kettunen,
Erik
H¨ark¨onen,
and
Jaakko
Lehti-
nen.
E-LPIPS:
robust
perceptual
image
similarity
via
random
transformation
ensembles.
arXiv
preprint
arXiv:1906.03973
, 2019.
2
[43]
Valentin
Khrulkov
and
Artem
Babenko.
Neural
side-by-
side:
Predicting human preferences for no-reference super-
resolution
evaluation.
In
IEEE
Conference
on
Computer
Vision and Pattern Recognition (CVPR)
, 2021.
3
,
1
[44]
Jiwon Kim, Jung Kwon Lee, and Kyoung Mu Lee.
Deeply-
recursive convolutional network for image super-resolution.
In
IEEE
Conference
on
Computer
Vision
and
Pattern
Recognition (CVPR)
, 2016.
2
[45]
Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adver-
sarial machine learning at scale.
International Conference
on Learning Representations (ICLR)
, 2017.
5
[46]
Yongchan
Kwon,
Eric
Wu,
Kevin
Wu,
and
James
Zou.
DataInf:
Efficiently
estimating
data
influence
in
LoRA-
tuned
LLMs
and
diffusion
models.
International
Confer-
ence on Learning Representations (ICLR)
, 2024.
1
[47]
Wei-Sheng Lai, Jia-Bin Huang, Narendra Ahuja, and Ming-
Hsuan Yang. Deep Laplacian pyramid networks for fast and
accurate
super-resolution.
In
IEEE
Conference
on
Com-
puter Vision and Pattern Recognition (CVPR)
, 2017.
2
[48]
Christian
Ledig,
Lucas
Theis,
Ferenc
Husz´ar,
Jose
Ca-
ballero,
Andrew
Cunningham,
Alejandro
Acosta,
Andrew
Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et al.
Photo-realistic
single
image
super-resolution
using
a
gen-
erative adversarial network.
In
IEEE Conference on Com-
puter Vision and Pattern Recognition (CVPR)
, 2017.
2
[49]
Xiaoming
Li,
Wangmeng
Zuo,
and
Chen
Change
Loy.
Learning
generative
structure
prior
for
blind
text
image
super-resolution.
In
IEEE Conference on Computer Vision
and Pattern Recognition (CVPR)
, 2023.
6
[50]
Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Luc
Van
Gool,
and
Radu
Timofte.
SwinIR:
Image
restoration
using
swin
transformer.
In
International
Conference
on
Computer Vision (ICCV)
, 2021.
2
,
6
,
1
[51]
Jie Liang, Hui Zeng, and Lei Zhang.
Details or artifacts:
A
locally
discriminative
learning
approach
to
realistic
image
super-resolution.
In
IEEE Conference on Computer Vision
and Pattern Recognition (CVPR)
, 2022.
2
2320
[52]
Youwei
Liang,
Junfeng
He,
Gang
Li,
Peizhao
Li,
Arseniy
Klimovskiy, Nicholas Carolan, Jiao Sun, Jordi Pont-Tuset,
Sarah
Young,
Feng
Yang,
et
al.
Rich
human
feedback
for
text-to-image generation. In
IEEE Conference on Computer
Vision and Pattern Recognition (CVPR)
, 2024.
3
[53]
Liting Lin, Heng Fan, Zhipeng Zhang, Yaowei Wang, Yong
Xu,
and Haibin Ling.
Tracking meets LoRA: Faster train-
ing, larger model, stronger performance.
In
European Con-
ference on Computer Vision (ECCV)
, 2024.
1
[54]
Yue Liu, Yunjie Tian, Yuzhong Zhao, Hongtian Yu, Lingxi
Xie,
Yaowei
Wang,
Qixiang
Ye,
Jianbin
Jiao,
and
Yunfan
Liu.
VMamba:
Visual state space model.
In
Neural Infor-
mation Processing Systems (NeurIPS)
, 2024.
2
[55]
Yujia
Liu,
Chenxi
Yang,
Dingquan
Li,
Jianhao
Ding,
and
Tingting Jiang.
Defense against adversarial attacks on no-
reference
image
quality
models
with
gradient
norm
regu-
larization.
In
IEEE
Conference
on
Computer
Vision
and
Pattern Recognition (CVPR)
, 2024.
5
[56]
Andreas
Lugmayr,
Martin
Danelljan,
Luc
Van
Gool,
and
Radu
Timofte.
SRFlow:
Learning
the
super-resolution
space
with
normalizing
flow.
In
European
Conference
on
Computer Vision (ECCV)
, 2020.
2
[57]
Chao
Ma,
Chih-Yuan
Yang,
Xiaokang
Yang,
and
Ming-
Hsuan
Yang.
Learning
a
no-reference
quality
metric
for
single-image super-resolution.
Computer Vision and Image
Understanding (CVIU)
, 2017.
3
[58]
Cheng
Ma,
Yongming
Rao,
Yean
Cheng,
Ce
Chen,
Jiwen
Lu,
and
Jie
Zhou.
Structure-preserving
super
resolution
with gradient guidance.
In
IEEE Conference on Computer
Vision and Pattern Recognition (CVPR)
, 2020.
2
[59]
Anish
Mittal,
Anush
Krishna
Moorthy,
and
Alan
Conrad
Bovik. No-reference image quality assessment in the spatial
domain.
IEEE
Transactions
on
Image
Processing
(TIP)
,
2012.
3
,
1
[60]
Jim
Nilsson
and
Tomas
Akenine-M¨oller.
Understanding
SSIM.
arXiv preprint arXiv:2006.13846
, 2020.
5
[61]
Xingang
Pan,
Xiaohang
Zhan,
Bo
Dai,
Dahua
Lin,
Chen Change Loy, and Ping Luo.
Exploiting deep genera-
tive prior for versatile image restoration and manipulation.
IEEE Transactions on Pattern Analysis and Machine Intel-
ligence (PAMI)
, 2021.
2
[62]
JoonKyu
Park,
Sanghyun
Son,
and
Kyoung
Mu
Lee.
Content-aware
local
GAN
for
photo-realistic
super-
resolution. In
International Conference on Computer Vision
(ICCV)
, 2023.
2
[63]
Seung
Ho
Park,
Young
Su
Moon,
and
Nam
Ik
Cho.
Perception-oriented
single
image
super-resolution
using
optimal objective estimation.
In
IEEE Conference on Com-
puter Vision and Pattern Recognition (CVPR)
, 2023.
2
[64]
Jonathan
W
Peirce.
Understanding
mid-level
representa-
tions
in
visual
processing.
Journal
of
Vision
(JOV)
,
2015.
6
[65]
Zygmunt Pizlo.
Perception
viewed
as
an
inverse problem.
Vision research
, 41(24):3145–3161, 2001.
1
[66]
Nikolay
Ponomarenko,
Oleg
Ieremeiev,
Vladimir
Lukin,
Karen
Egiazarian,
Lina
Jin,
Jaakko
Astola,
Benoit
Vozel,
Kacem Chehdi, Marco Carli, Federica Battisti, et al.
Color
image database TID2013:
Peculiarities and preliminary re-
sults. In
European workshop on visual information process-
ing (EUVIP)
, pages 106–111. IEEE, 2013.
3
[67]
Mihir
Prabhudesai,
Anirudh
Goyal,
Deepak
Pathak,
and
Katerina
Fragkiadaki.
Aligning
text-to-image
diffusion
models
with
reward
backpropagation.
arXiv
preprint
arXiv:2310.03739
, 2023.
2
[68]
Michele
A
Saad,
Alan
C
Bovik,
and
Christophe
Charrier.
A
DCT
statistics-based
blind
image
quality
index.
IEEE
Signal Processing Letters
, 2010.
2
,
3
[69]
Chitwan
Saharia,
Jonathan
Ho,
William
Chan,
Tim
Sal-
imans,
David
J
Fleet,
and
Mohammad
Norouzi.
Image
super-resolution
via
iterative
refinement.
IEEE
Transac-
tions on Pattern Analysis and Machine Intelligence (PAMI)
,
2022.
2
[70]
Divya
Saxena
and
Jiannong
Cao.
Generative
adversarial
networks
(GANs)
challenges,
solutions,
and
future
direc-
tions.
ACM Computing Surveys (CSUR)
, 2021.
2
[71]
Wenzhe
Shi,
Jose
Caballero,
Ferenc
Husz´ar,
Johannes
Totz, Andrew P Aitken, Rob Bishop, Daniel Rueckert, and
Zehan
Wang.
Real-time
single
image
and
video
super-
resolution using an efficient sub-pixel convolutional neural
network.
In
IEEE Conference on Computer Vision and Pat-
tern Recognition (CVPR)
, 2016.
2
[72]
Shaolin
Su,
Qingsen
Yan,
Yu
Zhu,
Cheng
Zhang,
Xin
Ge,
Jinqiu Sun, and Yanning Zhang. Blindly assess image qual-
ity in the wild guided by a self-adaptive hyper network.
In
IEEE
Conference
on
Computer
Vision
and
Pattern
Recog-
nition (CVPR)
, 2020.
3
[73]
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan
Bruna,
Dumitru
Erhan,
Ian
Goodfellow,
and
Rob
Fergus.
Intriguing
properties
of
neural
networks.
In
International
Conference on Learning Representations (ICLR)
, 2014.
5
[74]
Hossein Talebi and Peyman Milanfar. NIMA: Neural image
assessment.
IEEE Transactions on Image Processing (TIP)
,
2018.
2
,
3
,
5
,
6
[75]
Huixuan Tang, Neel Joshi, and Ashish Kapoor.
Learning a
blind measure of perceptual image quality. In
IEEE Confer-
ence on Computer Vision and Pattern Recognition (CVPR)
,
2011.
2
,
3
[76]
Radu
Timofte,
Vincent
De
Smet,
and
Luc
Van
Gool.
An-
chored
neighborhood
regression
for
fast
example-based
super-resolution.
In
International Conference on Computer
Vision (ICCV)
, 2013.
2
[77]
Radu
Timofte,
Vincent
De
Smet,
and
Luc
Van
Gool.
A+:
Adjusted anchored neighborhood regression for fast super-
resolution.
In
Proceedings
of
the
Asian
Conference
on
Computer Vision (ACCV)
, 2015.
2
[78]
Ilya Tolstikhin, Olivier Bousquet, Sylvain Gelly, and Bern-
hard
Schoelkopf.
Wasserstein
auto-encoders.
arXiv
preprint arXiv:1711.01558v3
, 2018.
6
[79]
Rao
Muhammad
Umer
and
Christian
Micheloni.
Deep
cyclic
generative
adversarial
residual
convolutional
net-
works
for
real
image
super-resolution.
In
European
Con-
ference on Computer Vision Workshops (ECCVW)
, 2020.
2
[80]
International Telecommunication Union.
Methodology for
the
subjective
assessment
of
the
quality
of
television
pic-
2321
tures.
Recommendation
ITU-R
BT.500-13
,
2012.
Geneva,
Switzerland.
5
[81]
Narasimhan
Venkatanath,
D
Praneeth,
Maruthi
Chan-
drasekhar
Bh,
Sumohana
S
Channappayya,
and
Swarup
S
Medasani.
Blind image quality evaluation using perception
based features.
In
2015 Twenty first national conference on
communications (NCC)
. IEEE, 2015.
3
[82]
Jianyi Wang, Kelvin CK Chan, and Chen Change Loy.
Ex-
ploring CLIP for assessing the look and feel of images.
In
Proceedings of the National Conference on Artificial Intel-
ligence (AAAI)
, 2023.
3
[83]
Jianyi Wang, Zongsheng Yue, Shangchen Zhou, Kelvin CK
Chan, and Chen Change Loy.
Exploiting diffusion prior for
real-world image super-resolution.
International Journal of
Computer Vision (IJCV)
, 2024.
2
[84]
Xintao
Wang,
Ke
Yu,
Shixiang
Wu,
Jinjin
Gu,
Yihao
Liu,
Chao
Dong,
Yu
Qiao,
and
Chen
Change
Loy.
ESRGAN:
Enhanced super-resolution generative adversarial networks.
In
European
Conference
on
Computer
Vision
Workshops
(ECCVW)
, 2018.
2
,
6
,
1
[85]
Xintao
Wang,
Liangbin
Xie,
Chao
Dong,
and
Ying
Shan.
Real-ESRGAN: Training real-world blind super-resolution
with
pure
synthetic
data.
In
International
Conference
on
Computer Vision (ICCV)
, 2021.
4
,
6
,
1
[86]
Zhou
Wang,
Alan
C
Bovik,
Hamid
R
Sheikh,
and
Eero
P
Simoncelli.
Image quality assessment:
from error visibility
to
structural
similarity.
IEEE
Transactions
on
Image
Pro-
cessing (TIP)
, 2004.
2
,
6
[87]
Haoning Wu, Erli Zhang, Liang Liao, Chaofeng Chen, Jing-
wen Hou, Annan Wang, Wenxiu Sun, Qiong Yan, and Weisi
Lin.
Exploring video quality assessment on user generated
contents
from
aesthetic
and
technical
perspectives.
In
In-
ternational Conference on Computer Vision (ICCV)
, 2023.
5
[88]
Haoning
Wu,
Zicheng
Zhang,
Weixia
Zhang,
Chaofeng
Chen,
Liang
Liao,
Chunyi
Li,
Yixuan
Gao,
Annan
Wang,
Erli
Zhang,
Wenxiu
Sun,
Qiong
Yan,
Xiongkuo
Min,
Guangtao Zhai, and Weisi Lin.
Q-Align:
Teaching LMMs
for visual scoring via discrete text-defined levels.
In
Inter-
national
Conference
on
Machine
Learning
(ICML)
,
2024.
3
,
5
,
6
,
1
[89]
Rongyuan Wu, Lingchen Sun, Zhiyuan Ma, and Lei Zhang.
One-step
effective
diffusion
network
for
real-world
image
super-resolution.
Neural
Information
Processing
Systems
(NeurIPS)
, 2024.
2
,
5
[90]
Rongyuan
Wu,
Tao
Yang,
Lingchen
Sun,
Zhengqiang
Zhang,
Shuai
Li,
and
Lei
Zhang.
SeeSR:
Towards
semantics-aware
real-world
image
super-resolution.
In
IEEE
Conference
on
Computer
Vision
and
Pattern
Recog-
nition (CVPR)
, 2024.
5
[91]
Bin
Xia,
Yulun
Zhang,
Shiyin
Wang,
Yitong
Wang,
Xing-
long Wu, Yapeng Tian, Wenming Yang, and Luc Van Gool.
DiffIR:
Efficient
diffusion
model
for
image
restoration.
In
International
Conference
on
Computer
Vision
(ICCV)
,
2023.
2
[92]
Jiazheng Xu,
Xiao Liu,
Yuchen Wu,
Yuxuan Tong,
Qinkai
Li,
Ming
Ding,
Jie
Tang,
and
Yuxiao
Dong.
ImageRe-
ward:
Learning and evaluating human preferences for text-
to-image
generation.
Neural
Information
Processing
Sys-
tems (NeurIPS)
, 2024.
2
,
3
[93]
Jianchao Yang, Zhaowen Wang, Zhe Lin, Scott Cohen, and
Thomas
Huang.
Coupled
dictionary
training
for
image
super-resolution.
IEEE Transactions on Image Processing
(TIP)
, 2012.
2
[94]
Sidi
Yang,
Tianhe
Wu,
Shuwei
Shi,
Shanshan
Lao,
Yuan
Gong,
Mingdeng
Cao,
Jiahao
Wang,
and
Yujiu
Yang.
MANIQA:
Multi-dimension
attention
network
for
no-
reference image quality assessment. In
IEEE Conference on
Computer Vision and Pattern Recognition (CVPR)
, 2022.
3
[95]
Tao
Yang,
Rongyuan
Wu,
Peiran
Ren,
Xuansong
Xie,
and
Lei Zhang.
Pixel-aware stable diffusion for realistic image
super-resolution and personalized stylization.
In
European
Conference on Computer Vision (ECCV)
, 2024.
2
,
5
[96]
Jie-En
Yao,
Li-Yuan
Tsao,
Yi-Chen
Lo,
Roy
Tseng,
Chia-
Che Chang,
and Chun-Yi Lee.
Local implicit normalizing
flow
for
arbitrary-scale
image
super-resolution.
In
IEEE
Conference
on
Computer
Vision
and
Pattern
Recognition
(CVPR)
, 2023.
2
[97]
Zhenqiang
Ying,
Haoran
Niu,
Praful
Gupta,
Dhruv
Maha-
jan, Deepti Ghadiyaram, and Alan Bovik.
From patches to
pictures (PaQ-2-PiQ): Mapping the perceptual space of pic-
ture quality.
In
IEEE Conference on Computer Vision and
Pattern Recognition (CVPR)
, 2020.
3
,
5
,
1
[98]
Lin
Zhang,
Lei
Zhang,
and
Alan
C
Bovik.
A
feature-
enriched
completely
blind
image
quality
evaluator.
IEEE
Transactions on Image Processing (TIP)
, 2015.
3
,
1
[99]
Richard
Zhang,
Phillip
Isola,
Alexei
A
Efros,
Eli
Shecht-
man,
and Oliver Wang.
The unreasonable effectiveness of
deep features as a perceptual metric. In
IEEE Conference on
Computer Vision and Pattern Recognition (CVPR)
, 2018.
2
[100]
Weixia Zhang, Kede Ma, Jia Yan, Dexiang Deng, and Zhou
Wang.
Blind image quality assessment using a deep bilin-
ear
convolutional
neural
network.
IEEE
Transactions
on
Circuits and Systems for Video Technology
, 2020.
3
[101]
Weixia Zhang, Guangtao Zhai, Ying Wei, Xiaokang Yang,
and
Kede Ma.
Blind
image quality
assessment
via
vision-
language
correspondence:
A
multitask
learning
perspec-
tive.
In
IEEE Conference on Computer Vision and Pattern
Recognition (CVPR)
, 2023.
3
,
5
[102]
Yulun
Zhang,
Kunpeng
Li,
Kai
Li,
Lichen
Wang,
Bineng
Zhong, and Yun Fu. Image super-resolution using very deep
residual
channel
attention
networks.
In
European
Confer-
ence on Computer Vision (ECCV)
, 2018.
2
,
6
[103]
Yuzhe Zhang, Jiawei Zhang, Hao Li, Zhouxia Wang, Luwei
Hou,
Dongqing
Zou,
and
Liheng
Bian.
Diffusion-based
blind
text
image
super-resolution.
In
IEEE
Conference
on
Computer Vision and Pattern Recognition (CVPR)
, 2024.
6
[104]
Heliang
Zheng,
Huan
Yang,
Jianlong
Fu,
Zheng-Jun
Zha,
and Jiebo Luo.
Learning conditional knowledge distillation
for degraded-reference image quality assessment.
In
Inter-
national Conference on Computer Vision (ICCV)
, 2021.
3
2322
