BlitzMask:
Real-Time Instance Segmentation Approach for Mobile Devices
Vitalii Bulygin
Dmytro Mykheievskyi
Kyrylo Kuchynskyi
Samsung R&D Institute Ukraine
Samsung R&D Institute Ukraine
Samsung R&D Institute Ukraine
Abstract
We
propose
a
fast
and
low
complexity
anchor-
free
instance
segmentation
approach
BlitzMask.
For
the
first
time,
the
approach
achieves
com-
petitive results for real-time inference on mobile
devices.
The
model
architecture
modifies
Cen-
terNet
by
adding
a
new
lite
head
to
the
Center-
Net
architecture.
The
model
contains
only
lay-
ers
optimized
for
inference
on
mobile
devices,
e.g.
batch
normalization,
standard
convolution,
depthwise convolution, and can be easily embed-
ded into a mobile device. The instance segmenta-
tion task requires finding an arbitrary (not a pri-
ori
fixed)
number
of
instance
masks.
The
pro-
posed
method
predicts
the
number
of
instance
masks
separately
for
each
image
using
a
pre-
dicted
heatmap.
Then,
it
decomposes
each
in-
stance mask over a predicted spanning set, which
is
an
output
of
the
lite
head.
The
approach
uses
training
from
scratch
with
a
new
optimization
process
and
a
new
loss
function.
A
model
with
EfficientNet-Lite B4 backbone and
320
×
320
in-
put resolution achieves
28
.
9
mask AP at
29
.
2
fps
on Samsung S21 GPU and
28
.
0
mask AP at
39
.
4
fps on Samsung S21 DSP. This sets a new speed
benchmark
for
inference
for
instance
segmenta-
tion on mobile devices.
1
INTRODUCTION
Several modern Computer Vision tasks have been ported to
mobile devices, such as Semantic Segmentation (T¨urkmen
and
Heikkil¨a,
2019),
(A.
Howard
et
al.,
2019),
(Orsic
et
al., 2019), Object Detection (Cai et al., 2020), (Vasu et al.,
2022),
(A.
Howard
et
al.,
2019),
and
Human
Pose
Esti-
mation
(Bazarevsky
et
al.,
2020),
by
inventing
new
com-
putationally
light
methods
for
each
task.
To
the
best
of
Proceedings
of
the
26
th
International
Conference
on
Artificial
Intelligence
and
Statistics
(AISTATS)
2023,
Valencia,
Spain.
PMLR: Volume 206.
Copyright 2023 by the author(s).
our
knowledge,
Instance
Segmentation
tasks
still
remain
a
challenge,
and
current
methods
do
not
achieve
suffi-
cient accuracy on these tasks for real-time mobile applica-
tions (Minaee et al., 2021),
(H. Liu et al., 2020),
(Zeng and
Sabah, 2020).
In the paper, we are addressing this gap with
an
approach
that
operates
on
low-resolution
input
images
and uses layers well supported on mobile devices,
e.g.
no
Deformable Convolution (Dai et al., 2017), Dynamic Con-
volution
(X.
Wang,
Zhang,
et
al.,
2020),
etc.
In
addition,
as it is hard to implement custom operations on mobile de-
vices, we use simple post-processing and only TFLite sup-
ported and optimized layers.
Adapting existing instance segmentation approaches to
mobile
We were unable to find fully comparable bench-
marks
with
performance
tested
on
regular
mobile
devices
for instance segmentation.
Why is porting an instance seg-
mentation approach to mobile such a problem?
A naive solution is to simply use an existing instance seg-
mentation
approach
with
light
backbones,
such
as
Mo-
bileNet, as well as low-resolution input to achieve real-time
speed
on
mobile.
Suppose
we
are
adapting
well-known
SOLO
(X.
Wang,
Kong,
et
al.,
2020),
or
CenterMask
(Y.
Wang et al., 2020), or a similar approach to mobile.
A sin-
gle output from SOLO or CenterMask has size
S
·
S
·
W
·
H
,
where
S
=
32
,
W
=
H
=
128
.
The corresponding back-
bones have 256 output channels, so that a final
3
×
3
convo-
lution requires
N
= 3
·
3
·
256
·
W
·
H
·
S
·
S
= 38
.
7
billion
multi-adds
(MAdds)
-
this
is
only
for
a
single
layer.
The
total
number
of
operations
for
SOLO
or
CenterMask
will
also
include
calculations
by
the
backbone
and
remaining
head
layers.
Meanwhile,
real-time
computation
for
mo-
bile
object
detection
MobileNetV2-SSDLite
requires
less
than
1
billion MAdds for
300
×
300
input size (Cai et al.,
2020),
(Sandler
et
al.,
2018a).
This
makes
a
single
layer
in the SOLO or CenterMask head nearly
40
times heavier
than
the
entire
MobileNetv2-SSDLite,
and
therefore
it
is
impractical to use such approaches on mobile.
The authors
of SOLOv2 (X. Wang, Zhang, et al., 2020) greatly reduced
the computational complexity of the model head using dy-
namic
convolution.
However,
this
is
a
non-standard
layer
and its adaption is currently not available for mobile; it is a
complex task to produce such an adaptation.
BlitzMask:
Real-Time Instance Segmentation Approach for Mobile Devices
YOLACT (Bolya et al., 2019a), YOLACT++ (Bolya et al.,
2019b) and similar anchor-based instance segmentation ap-
proaches have two main problems:
1) Non-maximum Sup-
pression (NMS), and 2) resolution-specific anchor parame-
ters at each pyramid level.
The former has been addressed
by
GPU-optimized
fast-NMS
(Bolya
et
al.,
2019a),
but
it
is
not
available
for
mobile.
The
latter
brings
up
a
non-
trivial
task
to
change
input
resolution
in
YOLACT,
which
can
dramatically
decrease
accuracy.
For
example,
the
au-
thors
Bolya
et
al.
(2019a)
report
AP
mask
=
29
.
8
for
550
×
550
,
and
AP
mask
=
24
.
9
for
400
×
400
input,
both with ResNet-101 backbone; our own YOLACT train-
ing
for
320
×
320
input
size
gives
only
AP
mask
=
20
.
1
with
ResNet-101
backbone.
The
lightweight
YOLACT
modification
named
mobileYOLACT
(J.
Lee,
S.
Lee,
and
Ko,
2021)
has
low
input
resolution
320
×
320
and
is
de-
signed
for
mobile
environments.
However,
the
best
result
of
the
mobileYOLACT
(J.
Lee,
S.
Lee,
and
Ko,
2021)
is
AP
mask
50
=
23
.
0
,
which
is
too
low,
because
AP
mask
is
usually
near
twice
smaller
than
AP
mask
50
and
as
we
show
below our approach is more than twice more accurate than
mobileYOLACT
and
has
even
faster
speed
on
the
mobile
device
considered
in
the
paper
(J.
Lee,
S.
Lee,
and
Ko,
2021).
All these facts prove that using a lightweight back-
bone and low input resolution for YOLACT approach leads
to
too
low
accuracy.
Furthermore,
YolactEdge
(H.
Liu
et
al., 2020) uses TensorRT optimization for
550
×
550
input
with MobileNetV2 backbone to achieve real-time calcula-
tion on Jetson AGX Xavier with accuracy
AP
mask
= 20
.
8
.
However,
this
model
is
still
too
heavy
for
mobile
devices,
and we show better accuracy for real-time applications on
mobile using the method described below.
In contrast to semantic segmentation, the instance segmen-
tation
task
has
a
variable
number
of
instance
masks
that
have to be presented by an output tensor with a fixed size.
To resolve this problem we express instance masks as lin-
ear combinations
of fixed size
spanning set
with instance-
specific
coefficients.
Our
proposed
method
extends
Cen-
terNet
by
adding
instance
masks
head
in
a
similar
way
as
YOLACT
(H.
Liu
et
al.,
2020)
extends
YOLOv3
(Red-
mon
and
Farhadi,
2018),
so
that
its
numerical
complex-
ity
is
comparable
with
CenterNet
(Zhou,
D.
Wang,
and
Kr¨ahenb¨uhl,
2019).
In
contrast
to
the
YOLACT
(H.
Liu
et
al.,
2020),
our
output
head
is
less
computationally
ex-
pensive and anchor-free, and our post-processing does not
require Non-Maximum Suppression (NMS).
The main contributions of our work are as follows:
•
We propose a novel single-stage anchor-free instance
segmentation
method
for
real-time
execution
on
mo-
bile devices
•
In
our
approach,
CenterNet
(Zhou,
D.
Wang,
and
Kr¨ahenb¨uhl,
2019)
object
detector
is
just
comple-
mented
by
a
lightweight
instance
mask
head,
which
requires
significantly
fewer
resources
than
the
entire
model.
•
We implement a new loss function to train anchor-free
single-stage model for instance segmentation
•
We propose a simple architecture for porting to mobile
devices. It achieves satisfactory accuracy for real-time
applications on mobile devices.
•
We develop a new training process with learning rate
and stochastic gradient momentum schedules that are
dependent on current loss function values.
•
Our
method
even
with
input
resolution
320
×
320
achieves
better
accuracy
and
speed
then
Yolact-
Edge
(H.
Liu
et
al.,
2020)
with
the
same
backbone
and input resolution
550
×
550
.
The
paper
is
organized
as
follows.
Section
2
reviews
re-
lated work.
In Section 3 describes the BlitzMask approach.
Section
4
describes
our
new
model
training
process
and
demonstrates results from our modeling approach.
2
RELATED WORK
Recently, real-time instance segmentation approaches have
achieved great speed and accuracy on a desktop GPU such
as
GTX
2080
TI,
Tesla
V100,
or
Titan
XP
(Bolya
et
al.,
2019a),
(Y.
Wang
et
al.,
2020),
(X.
Wang,
Kong,
et
al.,
2020), (H. Chen et al., 2020), (Xie et al., 2020), (X. Wang,
Zhang,
et
al.,
2020),
(Bolya
et
al.,
2019b),
but
its
accu-
rate
and
fast
implementation
still
remains
a
challenge
for
mobile
devices.
An
anchor-based
single-stage
approach
YOLACT (Bolya et al., 2019a) is one of the first real-time
instance
segmentation
methods.
It
extends
the
object
de-
tector
YOLOv3
(Redmon
and
Farhadi,
2018)
by
adding
two
heads
for
prototype
masks
and
mask
coefficients,
so
that
instance
masks
are
computed
as
linear
combinations
of
the
prototype
masks
with
the
normalized
coefficients
for
each
anchor
box.
The
method
achieves
29.8
mAP
on
MS
COCO
(T.-Y.
Lin,
Maire,
et
al.,
2014)
at
33.5
fps
on
NVIDIA’s Titan Xp graphics card.
A
follow-up
approach
YOLACT++
(Bolya
et
al.,
2019b)
uses deformable convolutions (Dai et al., 2017), optimized
prediction
heads
and
a
fast
mask
re-scoring
head.
These
optimizations improve mask performance to 34.4 mAP on
MS COCO, but reduce speed to 27.3 fps on Titan Xp.
In contrast to the YOLACT approaches, SOLO (X. Wang,
Kong,
et
al.,
2020)
implements
instance
segmentation
di-
rectly, without any dependence on box detection.
It divides
a picture into an
S
×
S
grid and decouples the original mask
prediction
into
semantic
category
prediction
and
instance
mask prediction. The former indicates semantic class prob-
abilities for each cell and has
S
×
S
×
C
values, where
C
is
Vitalii Bulygin,
Dmytro Mykheievskyi,
Kyrylo Kuchynskyi
the number of object classes.
While the latter has
H
×
W
values
for
each
cell
and
S
×
S
×
H
×
W
values
in
total,
where
H
is the output height,
W
is the output width.
Each
category prediction cell
(
i, j
)
,
i, j
= 0
,
1
, . . . , S
−
1
is asso-
ciated with the
k
th
channel of the instance mask prediction,
where
k
=
i
·
S
+
j
. This structure allows SOLO to predict a
variable number of instances using a fixed number of chan-
nels. However, convolution layers with
S
2
×
H
×
W
output
make
SOLO
head
computationally
expensive.
This
slows
down processing to 22.5 fps on NVIDIA’s Tesla V100 with
34.2 mAP on MS COCO.
A
greatly
accelerated
SOLOv2
(in
comparison
to
SOLO) (X. Wang, Zhang, et al., 2020) splits the mask pre-
diction into mask kernel and mask feature map prediction,
so
that
the
output
size
decreases
from
S
2
×
H
×
W
to
2
S
×
H
×
W
.
Each mask kernel is convolved with a fea-
ture map to get an instance mask.
The performance speed
improves to 46.5 fps on Tesla V100 with 34.0 mAP on MS
COCO.
Another
approach
CenterMask
(Y.
Wang
et
al.,
2020)
is
an
anchor-free
method
that
adds
two
heads
to
Center-
Net (Zhou, D. Wang, and Kr¨ahenb¨uhl, 2019):
1) a head for
local instance information with output size
S
2
×
H
×
W
,
and 2) a head for semantic segmentation on the whole im-
age with output size
H
×
W
.
The two outputs are assem-
bled together to produce final instance masks.
Similarly to
SOLO
approach,
the
first
head
is
computationally
expen-
sive, resulting in only 25.2 fps with 32.5 mAP (the Center-
Mask authors do not report computational machine specifi-
cations in their paper).
PolarMask
approach
(Xie
et
al.,
2020)
approximates
in-
stance
mask
contours
using
36
points
derived
intersecting
36
uniformly
emitted
rays
from
the
object
center
with
the
instance
mask
contour.
This
method
is
a
generalization
of
the
FCOS
object
detection
(Tian
et
al.,
2019),
where
bounding boxes are presented as simplest masks with only
4 directions.
The contour approximation in PolarMask can
be
rough,
and
increasing
the
count
of
emitted
rays
does
not
always
converge
to
the
exact
instance
mask
contour.
While
PolarMask
is
a
simple
and
flexible
framework,
it
only achieves 22.9 mAP on MS COCO at 26.3 fps on Tesla
V100,
which is worse than SOLO (X. Wang,
Kong,
et al.,
2020),
YOLACT
(Bolya
et
al.,
2019a),
CenterMask
(Y.
Wang
et
al.,
2020)
and
other
modern
one-stage
instance
segmentation approaches.
There
are
a
few
studies
focusing
on
real-time
in-
stance
segmentation
methods
for
mobile
devices.
For
example,
YolactEdge
(H.
Liu
et
al.,
2020)
improves
YOLACT (Bolya et al., 2019a) by applying TensorRT op-
timization
and
using
MobilenetV2
(Sandler
et
al.,
2018b)
architecture.
It
achieves
20.8
mAP
on
MS
COCO
at
35.7
fps
on
Jetson
AGX
Xavier.
However,
Jetson
AGX
Xavier
device
is much
faster
than
regular
mobile
GPUs,
e.g.
it
is
about several times faster than Samsung Adreno 660 GPU
S21 (see
(Ignatov et al., n.d.), NVIDIA specification web-
site
(
Jetson benchmarks
2022),
(
Mobile Ranking
2022)).
The
instance
segmentation
approach
mobileYOLACT
(J.
Lee, S. Lee, and Ko, 2021) is designed for mobile environ-
ments.
It modifies YOLACT (Bolya et al., 2019a) by using
a
lightweight
backbone,
depthwise
separable
convolution,
a simplified prototype mask generation branch and UINT8
quantization. The mobileYOLACT (J. Lee, S. Lee, and Ko,
2021) has a speed of 21 FPS on Samsung Galaxy S20 with
AP
mask
50
=
23
.
0
(authors
do
not
present
AP
mask
values
in (J. Lee, S. Lee, and Ko, 2021)).
The
lightweight
approach
EOLO
(Zeng
and
Sabah,
2020)
is an anchor-free instance segmentation that works at 16 fps
on Raspberry Pi4 with Google Coral USB Accelerator with
11.7 mAP accuracy on MS COCO.
Lastly,
we
wanted
to
mention
a
real-time
portrait
instance
segmentation
approach
for
mobiles
(L.
Zhu
et
al.,
2019)
that
generates
three
outputs:
person
semantic
segmenta-
tion,
person
bounding
boxes
and
superpixels.
When
there
is no overlap among the bounding boxes, the first and sec-
ond
outputs
are
sufficient
for
instance
segmentation.
Oth-
erwise,
the
method
utilizes
the
third
output
to
resolve
oc-
clusions or overlapping regions.
It is a challenging task to
extend
this
approach
to
80
MS
COCO
classes
or
similar
datasets with a large number of classes.
3
BLITZMASK
3.1
Model architecture
We
propose
a
new
single-stage
anchor-free
instance
seg-
mentation
method
named
BlitzMask.
Its
architecture
is
based
on
CenterNet
(Zhou,
D.
Wang,
and
Kr¨ahenb¨uhl,
2019)
and
FPN
(T.-Y.
Lin,
Doll´ar,
et
al.,
2017)
(Fig.
1).
When
implemented
for
real-time
inference
on
mobile
de-
vices,
the
architecture
uses
separable
convolution
(A.
G.
Howard
et
al.,
2017)
with
bilinear
upsampling
instead
of
a
combination
of
deformable
(Dai
et
al.,
2017)
and
trans-
pose
convolutions
as
described
by
Zhou,
D.
Wang,
and
Kr¨ahenb¨uhl,
2019.
The
reason
is
that,
deformable
convo-
lution is not supported, and transpose convolution has high
latency on mobile devices (Chiang et al., 2020).
Note that we use only well-known layers optimized for mo-
bile
devices
on
TFlite
such
as
Batch
Normalization
(Ioffe
and Szegedy, 2015), depthwise convolution (A. G. Howard
et
al.,
2017)
and
convolutional
layer
(LeCun
et
al.,
1998)
.
Model
output
consists
of
Spanning
Set
B
,
Coefficients
Λ
, Heatmap
Y
and Size
D
.
The output spatial dimensions
are four times less than the input spatial dimensions, chan-
nel
number
K
=
32
(Fig.
1);
we
found
that
this
selection
provides the best trade off between speed and accuracy in a
series of experiments on MS COCO dataset.
Furthermore,
BlitzMask:
Real-Time Instance Segmentation Approach for Mobile Devices
s16, 384
s4, 96
s8, 192
s32, 384
s16, 192
s8, 96
Backbone
merging by
addition
Conv
2x Up-
sample
2x Down-
sample
s4, 4
Spanning
Set
Coeffi-
cients
Heatmap
Size
s4, C
s4, K
s4, K
Figure 1: BlitzMask architecture.
W
and
H
are input width
and
height,
respectively,
C
is
a
number
of
classes,
K
is
number of channels for both Spanning Set and Coefficients.
The solid lines denote
3
×
3
depthwise separable convolu-
tion
(A.
G.
Howard
et
al.,
2017).
L
symbol
stands
for
element-wise summation.
The dashed line is bi-linear up-
sampling by a factor of 2.
The labels inside rectangles de-
note the corresponding stride and number of channels.
we do not use any activation functions for the outputs, ex-
cept for the sigmoid function for Heatmap
Y
.
Each channel
c
of the ground truth for Heatmap
Y
contains
probability
density-like
function
for
bounding
box
center
locations
for
objects
of
class
c
(see
Fig.
2).
So
that
if
a
ground truth bounding box center for an object of class
c
is
located
at
i
0
,
j
0
,
then
the
c
th
channel
of
Heatmap
ground
truth contains scaled Gaussian distribution with maximum
value
1
at
its
center
i
0
,
j
0
with
object
size-adaptive
stan-
dard
deviation
σ
defined
as
described
by
Law
and
Deng
(2018), Zhou, D. Wang, and Kr¨ahenb¨uhl (2019).
Ground
truth
for
Size
D
contains
distances
from
the
ob-
ject
bounding
box
centers
to
its
four
sides
[
w
l
, h
t
, w
r
, h
b
]
similarly
to
Liu
et
al.
(Z.
Liu
et
al.,
2020).
Then,
given
a
Gaussian center in row
i
, column
j
for channel
c
from the
Heatmap
Y
, a bounding box for an object of class
c
can be
computed as follows:
[
x, y, w, h
] = [4
·
j
−
w
l
,
4
·
i
−
h
t
, w
l
+
w
r
, h
t
+
h
b
]
,
(1)
Figure
2:
Left
is
the
original
image
resized
to
the
model
input size
W
,
H
,
right is the single channel of the ground
truth for the output Heatmap
Y
of size
W/
4
,
H/
4
(shown
not
to
scale).
The
Heatmap
contains
scaled
Gaussian
dis-
tributions centered at the object bounding box center loca-
tions
(
i
1
, j
1
)
,
(
i
2
, j
2
)
,
(
i
3
, j
3
)
.
where
(
x, y
)
is
the
box
left
top
point,
(
w, h
)
are
the
box
width,
height,
[
w
l
, h
t
, w
r
, h
b
]
=
D
i,j
.
As
Heatmap
Y
has
four times lower resolution than model input size, we scale
its Gaussian peak locations by a factor of
4
above.
Further-
more,
due
to
the
lower
resolution
of
the
Heatmap,
bound-
ing boxes require specifying four sides, instead of two (i.e.
width
and
height),
to
account
for
possible
offsets.
This
representation allows placing Gaussian center location ev-
erywhere inside bounding box (not only in the box center)
and helps in our experiments with modification of Gaussian
center location in Section 4.
We
express
an
instance
mask
as
a
linear
combination
of
Spanning
Set
B
with
corresponding
Coefficients
Λ
(more
details
are
provided
in
the
following
sections).
We
only
define
ground
truth
for
instance
mask,
and
do
not
require
ground truths for its components used in the derivation, i.e.
Spanning Set and Coefficients.
3.2
Loss function
3.2.1
Object detection loss
The
Heatmap
Y
and
Size
D
outputs
(Fig.
1)
are
com-
pared
to
the
corresponding
ground
truths
using
object
de-
tection
loss
function
L
box
that
consists
of
focal
loss
L
foc
for
Heatmap
and
L
size
for
Size.
The
focal
loss
L
foc
is
a
penalty-reduced
pixelwise
logistic
regression
loss
(T.-Y.
Lin, Goyal, et al., 2017), (Y. Wang et al., 2020) with pertur-
bation
of
the
first
polynomial
coefficient
of
Taylor
expan-
sion similar to (Leng et al., 2022) defined below as follows:
Vitalii Bulygin,
Dmytro Mykheievskyi,
Kyrylo Kuchynskyi
L
foc
=
−
1
N
·
H/
4
X
i
=1
W/
4
X
j
=1
C
X
c
=1
(
A
c
i,j
if
˘
Y
c
i,j
= 1
B
c
i,j
otherwise
,
A
c
i,j
=
1
−
Y
c
i,j
2
·
log
Y
c
i,j
+
ϵ
1
·
1
−
Y
c
i,j
B
c
i,j
=
1
−
˘
Y
c
i,j
4
Y
c
i,j
2
log
1
−
Y
c
i,j
+
ϵ
2
Y
c
i,j
(2)
where
Y
=
Y
c
i,j
H/
4
,W/
4
,C
i
=1
,j
=1
,c
=1
is
the
predicted
Heatmap
and
˘
Y
c
i,j
is the ground truth Heatmap values,
N
is the num-
ber
of
objects
in
the
input
image,
ϵ
1
is
perturbation
coef-
ficient for the first polynomial coefficient of Taylor expan-
sion of the case
˘
Y
c
i,j
=
1
,
ϵ
2
is perturbation coefficient for
the first coefficient for
˘
Y
c
i,j
̸
=
1
.
If
ϵ
1
=
ϵ
2
=
0
then
(2)
is penalty-reduced pixelwise logistic regression loss (T.-Y.
Lin,
Goyal,
et al.,
2017),
however optimal choice of these
coefficients
improves
accuracy
as
it
will
be
shown
below.
For each
n
th
ground truth box center for some class
c
n
at
row
i
n
, column
j
n
,
n
= 1
, . . . , N
we take prediction values
from
the
Size
output
[
w
n
l
, h
n
t
, w
n
r
, h
n
b
]
=
D
c
n
i
n
,j
n
and
com-
pute bounding box
b
n
= [
x
n
, y
n
, w
n
, h
n
]
using (1). We de-
fine the following combination of
l
1
-type loss and UnitBox
loss (Yu et al., 2016) penalizes bounding box predictions:
L
size
=
1
N
·
N
X
n
=1
α
1
·
b
n
−
˘
b
n
−
α
2
·
log
b
n
∩
˘
b
n
b
n
∪
˘
b
n
!
=
α
1
·
L
l
1
+
α
2
·
L
UnitBox
,
(3)
where
the
ground
truth
bounding
box
˘
b
n
=
[˘
x
n
,
˘
y
n
,
˘
w
n
,
˘
h
n
]
, intersection
b
n
∩
˘
b
n
and union
b
n
∪
˘
b
n
as
described by Yu et al. (2016)
Then,
the
object
detection
loss
can
be
expressed
as
a
lin-
ear
combination
of
(2)
and
(3)
averaged
over
a
batch
of
images:
L
box
=
1
b
b
X
i
=1
(
L
i
foc
+
α
1
·
L
i
l
1
+
α
2
·
L
i
UnitBox
)
,
(4)
where
L
i
foc
and
L
i
size
=
α
1
·
L
i
l
1
+
α
2
·
L
i
UnitBox
are loss
functions (2) and (3), respectively, for
i
th
image in a batch.
The
authors
Zhou,
D.
Wang,
and
Kr¨ahenb¨uhl
(2019)
use
the case
ϵ
1
=
ϵ
2
=
α
2
= 0
,
α
1
= 0
.
1
which we call below
as ’default OD loss’
3.2.2
Mask loss
We estimate instance masks using Coefficients
Λ
and Span-
ning Set
B
as described below.
For each ground truth box
center
i
n
,
j
n
,
n
=
1
, . . . , N
,
we
fix
λ
n,k
=
Λ
k
i
n
,j
n
,
k
=
1
, . . . , K
, where
N
is the number of instances and
K
is the
number of channels in both Coefficients and Heatmap out-
puts.
Then, an instance mask is predicted as a linear com-
bination
of
the
Spanning
Set
channels
B
k
,
k
=
1
, . . . , K
with coefficients
λ
n,k
(also see Fig. 3):
M
n
=
K
X
k
=1
λ
n,k
·
B
k
(5)
We
use
bi-linear
upsampling
of
M
by
a
factor
of
3
and
denote
the
result
as
ˆ
M
∈
R
H/
2
,W/
2
.
While
the
bi-
linear
resizing
is
not
required,
it
notably
improved
accu-
racy
in
our
experiments.
Then,
to
define
loss
function
for
instance
masks
as
combination
of
Cross-Entropy
loss
(Jadon,
2020)
and
Dice
loss
(Milletari,
Navab,
and
Ah-
madi,
2016).
Firstly,
let
define
Cross-Entropy
loss
matrix
L
n
=
L
n
i,j
H/
2
,W/
2
i
=1
,j
=1
∈
R
H/
2
,W/
2
:
L
n
=
−
˘
M
n
·
log
σ
ˆ
M
n
−
1
−
˘
M
n
log
1
−
σ
ˆ
M
n
,
(6)
where
σ
is
a
sigmoid
function;
˘
M
n
,
n
=
1
, . . . , N
are
ground
truth
instance
masks;
N
is
a
number
of
instances
in the ground truth.
The ground truth mask equals
0
every-
where, except for instance location, where it equals
1
.
We
use
values
of
L
n
only
inside
the
n
th
ground
truth
bounding box
[
i
n
1
, i
n
2
]
×
[
j
n
1
, j
n
2
]
to define loss function for
a single mask:
L
n
mask
CE
=
1
(
i
n
2
−
i
n
1
)
·
(
j
n
2
−
j
n
1
)
·
i
n
2
X
i
=
i
n
1
j
n
2
X
j
=
j
n
1
L
n
i,j
(7)
The normalization coefficient
1
(
i
n
2
−
i
n
1
)
·
(
j
n
2
−
j
n
1
)
allows
small
and
large
objects
contributing
equally
to
the
loss
function.
The
second
step
is
defining
the
Dice
loss
(Mil-
letari, Navab, and Ahmadi, 2016)
L
n
mask
Dice
=
1
−
2
·
P
˘
M
n
i,j
·
σ
ˆ
M
n
i,j
+ 1
P
˘
M
n
i,j
2
+
P
σ
ˆ
M
n
i,j
2
+ 1
,
(8)
where summarizing indices are as in (7)
i
=
i
n
1
· · ·
i
n
2
,
j
=
j
n
1
· · ·
j
n
2
Then, the loss function for a batch with
b
images,
where
i
th
image
has
N
i
ground
truth
instance
masks,
is
defined as follows:
BlitzMask:
Real-Time Instance Segmentation Approach for Mobile Devices
Table 1:
Brute-forcing loss function parameters using backbone ResNet18**, width
α
= 0
.
5
Table A: Accuracy of object detector for different values of parameters
ϵ
2
, α
1
, α
2
ϵ
2
0.0
0.0
0.0
0.0
0.0
0.0
-0.05
-0.05
-0.1
0.05
0.1
-0.05
0.05
-0.05
α
1
0.1
0.0
0.1
0.2
0.05
0.05
0.05
0.05
0.05
0.05
0.05
0.1
0.1
0.1
α
2
0.0
1.0
1.0
1.0
1.0
2.0
1.0
1.5
1.0
1.0
1.0
0.0
0.0
1.5
AP
box
26.1
26.3
26.3
26.2
26.4
26.3
26.6
26.7
26.5
26.2
26.0
26.2
25.8
26.6
Table B: Accuracy of BlitzMask for different values of parameters
β
1
,
β
2
β
1
1.0
0.0
2.0
0.0
3.0
0.0
4.0
0.0
1.0
2.0
0.5
3.0
β
2
0.0
1.0
0.0
2.0
0.0
3.0
0.0
4.0
2.0
1.0
2.5
1.0
AP
mask
21.8
22.2
21.9
22.3
21.9
22.4
21.7
22.4
22.5
22.3
22.6
22.3
L
mask
=
1
b
b
X
i
=1
1
N
i
·
N
i
X
n
i
=1
β
1
·
L
i,n
i
mask
CE
+
β
2
·
L
i,n
i
mask
Dice
(9)
Let the case of
β
1
=
1
,
β
2
=
0
is
called
as
’default mask
loss’.
Difference between ’default mask loss’ and segmen-
tation part of YOLACT loss (Bolya et al., 2019a) is in us-
ing
the
normalization
coefficient
in
(7)
and
bi-linear
up-
sampling
of
predicted
and
ground
truth
masks
inside
loss
function that improve accuracy essentially.
3.2.3
Overall loss
Finally,
an
overall
loss
function
for
a
batch
of
images
is
defined
as
a
linear
combination
of
box
(4)
and
mask
losses (9):
L
all
=
L
box
+
L
mask
=
L
foc
+
α
1
·
L
l
1
+
α
2
·
L
UnitBox
+
β
1
·
L
mask
CE
+
β
2
·
L
mask
Dice
(10)
Taking
into
account
(2)
we
have
6
unknown
parameters
ϵ
1
, ϵ
2
, α
1
, α
2
, β
1
, β
2
.
The ’default loss’ is the case of
ϵ
1
=
ϵ
2
=
α
2
=
0
,
α
1
=
0
.
1
,
β
1
=
1
,
β
2
=
0
.
It
contains
loss described by Zhou,
D. Wang,
and Kr¨ahenb¨uhl (2019)
and
our
modification
of
the
loss
described
by
Bolya
et
al.
(2019a).
Below, we obtain optimal parameters using brute-
force.
3.3
Post-processing
The
instance
segmentation
masks
are
expressed
using
model outputs - Spanning Set
B
, Coefficients
Λ
, Heatmap
Y
and
Size
D
(see
Fig.1
and
Fig.
3
for
post-processing
a
single
class
”person”).
First,
we
compute
object
cen-
ters
i
n
,
j
n
,
n
=
1
,
2
, . . . , N
by
using
3
×
3
maximum
pooling on the heatmap
Y
as described by Law and Deng
(2018), Zhou,
D. Wang,
and Kr¨ahenb¨uhl (2019).
We only
retain
N
object
centers
with
scores
Y
i
n
,j
n
greater
than
a
Spanning
Set
Coeffi
cients
Size
Heatmap
resize,
threshold
Figure 3:
Instance mask reconstruction using Spanning Set
B
,
Coefficients
Λ
,
Heatmap
Y
and Size
D
for the case of
N
=
3
instance
masks,
class
number
C
=
1
.
Operation
”resize”
means
resize
M
n
,
n
=
1
, . . . , N
to
original
im-
ages size.
The ”threshold” operation assigns
1
to pixels for
which
σ
(
M
n
)
>
0
.
5
and
0
, otherwise.
Vitalii Bulygin,
Dmytro Mykheievskyi,
Kyrylo Kuchynskyi
chosen
threshold.
Then,
for
each
object
center
i
n
,
j
n
,
we
compute
bounding
boxes
b
n
=
[
x
n
, y
n
, w
n
, h
n
]
using
(1)
and
the
corresponding
instance
mask
heatmaps
M
n
,
n
=
1
,
2
, . . . , N
using
(5);
we
bi-linearly
resize
both
the
derived mask heatmaps
M
n
and the boxes
b
n
to the origi-
nal
image
size.
Lastly,
we
define
the
final
instance
masks
as
consisting
of
pixels
that
are
both
within
the
bounding
boxes
b
n
and for which
σ
(
M
n
)
>
0
.
5
(Fig 3).
4
EXPERIMENTS
4.1
Training Details
The
instance
segmentation
models
are
trained
on
MS
COCO
(T.-Y.
Lin,
Maire,
et
al.,
2014)
train2017
dataset
and
evaluated
on
val2017.
We
follow
the
recommenda-
tions of Izmailov et al. (2018) and Smith (2018) to search
for
a
flat
(rather
than
sharp)
minima
on
the
loss
surface,
as
a
sharp
minima
determined
on
train
data
could
result
in
a
large
loss
function
value
on
evaluation
data.
Our
training process is a modification of cyclical learning rates
approach
(Smith,
2018),
it
uses
Stochastic
Gradient
De-
scent (SGD) with momentum and has three phases: Ascent,
Plateau, and Descent.
We will refer to the training process
as ’LRfinder’ below.
At
the
Accent
phase,
LRfinder
uses
momentum
µ
=
0
.
9
and linearly increases the learning rate as much as possible
until the training loss starts deteriorating.
Then we reduce
the rate of the learning increasing twice and load previous
epoch
weights
and
repeat
this
procedure
three
times.
The
maximum
achieved
learning
rate
is
then
denoted
as
η
.
At
the
Plateau
phase,
the
learning
rate
is
fixed
to
η
and
we
count ”Patience”, e.g.
the number of epochs without train-
ing loss improvement.
Each time the Patience increases by
5
,
the momentum value gets reduced to the following val-
ues
µ
=
0
.
63
,
µ
=
0
.
44
,
µ
=
0
.
3
,
and
then
the
Descent
phase starts.
At the Descent phase, the SGD momentum is
µ
= 0
.
3
, learning rate is reduced to
0
.
1
·
η
,
0
.
01
·
η
,
0
.
001
·
η
each time the Patience increases by
5
.
Training is finished
when learning rate becomes
0
.
001
·
η
.
Notably that some-
times
gradient
explosion
occurs
in
Plateau
phase
because
of high learning rate value.
In such case, we load previous
epoch weights and switch to the next step (momentum re-
ducing
or
Descent
phase)
and
it
always
fixes
the
gradient
explosion problem.
The method described by Smith (2018) gave us an idea how
to
modify
the
training
process,
but
our
implementation
is
conceptually different, specifically, we:
1) do not have any
Cyclical Learning rates like described by Smith (2018), 2)
have different rules for SGD momentum changing,
and 3)
follow
a
different
approach
to
decrease
the
learning
rate.
In
contrast
to
(Smith,
2018),
we
find
the
optimal
learning
rate
and
SGD
momentum
only
during
the
training
by
cre-
ating model backups and “look forwards” to what happens
if we change the learning rate or SGD momentum.
More-
over, we use different learning rates ascent speeds and SGD
momentum descent to get the best validation loss.
It essen-
tially
differs
from
the
method
proposed
by
Smith
(2018)
and any other training process published in the literature.
We
use
random
flip,
scaling
(between
0
.
25
to
4
.
0
),
crop-
ping, photometric transformations and aspect ratio modifi-
cation
(from
0
.
5
to
2
.
0
)
as
train
data
augmentation.
All
models
are
trained
on
a
single
Nvidia
V100
GPU
from
scratch,
without
any
pretrained
weights,
with
input
reso-
lution
320
×
320
and batch size
64
.
Models are converted
to
TFlite
and
tested
on
S21
with
Snapdragon
888
system
on a chip with CPU Kryo 680,
GPU Adreno 660.
We use
FLOAT16
quantization
for
S21
GPU
and
UINT8
for
S21
DSP. To save training time, first, we train without mask up-
sampling in (9) and then tune with mask upsampling by a
factor of 3.
4.2
Loss function parameters
To brute force parameters
ϵ
1
, ϵ
2
, α
1
, α
2
, β
1
, β
2
we use ’fast
training’
with
modified
backbone
ResNet18,
without
seg-
mentation mask upsampling by a factor of 3 in (9), a short
”Patience”
=
2
and
switching
of
Accent
phase
to
Plateau
phase just after training loss starts deteriorating (i.e.
with-
out using different learning rates ascent speeds).
The mod-
ified backbone is denoted as ResNet18** and does not con-
tain first stride
2
and
3
×
3
MaxPool and has width
α
= 0
.
5
,
i.e.
only
half
of
the
convolution
layer
filter
number.
We
found
that
BlitzMask
with
such
backbone
required
only
120-140
epochs
to
train,
and
each
epoch
is
near
11
min-
utes on a single Nvidia V100 GPU in contrast to near 350
epochs of full training with MobileNetV2 or EfficientNet-
Lite backbone with near 50 minutes per epoch.
Firstly, we
get parameters
ϵ
1
, ϵ
2
, α
1
, α
2
used in loss functions
L
foc
(2)
and
L
size
(3) for object detection (i.e.
without Coefficients
and Spanning Set outputs in Fig. 1) and then add additional
head for Instance Segmentation to obtain parameters
β
1
, β
2
used in
L
mask
(9).
Our experiments show that parameter
ϵ
1
has negligible in-
fluence
on
accuracy,
therefore
we
fix
ϵ
1
=
0
and
show
some
of
our
results
for
ϵ
2
, α
1
, α
2
in
Table
A
of
Table
1
for object detection case only (i.e.
only Heatmap and Size
outputs in Fig. 1). The second column in Table A of Table 1
is object detection ’default case’ which use the same loss as
described
by
Zhou,
D.
Wang,
and
Kr¨ahenb¨uhl
(2019)
and
the best accuracy is reached with
ϵ
2
=
−
0
.
05
,
α
1
=
0
.
05
,
α
2
= 1
.
5
.
Chosen parameters give
+0
.
6
AP
box
in compar-
ison to ’default case’.
Then the best parameters for object
detection case are fixed and
β
1
,
β
2
are varying in Table B
of Table 1.
Then we add segmentation head (Spanning Set
and Coefficients outputs in Fig. 1), use obtained parameters
of the object detection loss and vary
β
1
,
β
2
inside mask loss
in Table B of Table 1
BlitzMask:
Real-Time Instance Segmentation Approach for Mobile Devices
Table 2:
BlitzMask speed and accuracy of the instance segmentation on Samsung S21 Snapdragon 888
GPU
(FLOAT16
quantization) and
DSP
(UINT8 quantization) for input resolution
320
×
320
, backbones MobileNetV1 (A. G. Howard et al.,
2017), MobileNetV2 (Sandler et al., 2018a), EfficientNet-Lite B0-B4 (M. Tan and Le, 2019), (
EfficientNet-Lite
2022) and
modified ResNet-18 (He et al., 2015)
GPU
DSP
Backbone
Time(ms)
AP
mask
AP
mask
50
Time(ms)
AP
mask
AP
mask
50
MobileNetV1
21.3
21.0
34.3
10.4
20.2
33.1
MobileNetV2
21.8
21.9
35.8
9.2
21.0
34.7
EfficientNet-Lite B0
22.3
25.2
43.4
14.1
24.3
41.8
EfficientNet-Lite B1
24.3
26.2
45.0
16.0
25.4
43.5
EfficientNet-Lite B2
25.2
27.0
45.9
15.7
26.2
44.6
EfficientNet-Lite B3
28.2
27.9
47.3
18.1
27.0
45.9
EfficientNet-Lite B4
34.2
28.9
48.7
25.4
28.0
47.1
ResNet-18*
88.2
29.6
50.1
52.6
29.3
49.4
umbrella
center of box
center of mass
Figure 4:
Difference between center of mass and center of
box position
As
shown
in
Table
A
of
Table
1
mixing
of
Cross
Entropy
and Dice loss with
β
1
= 0
.
5
,
β
2
= 2
.
5
gives
+0
.
8
AP
mask
in
comparison
with
’default
case’
which
is
segmentation
part of YOLACT loss (Bolya et al., 2019a).
4.3
Impact of our improvements
Sometimes,
the
box
center
lies
outsides
the
object,
as
shown
in
Fig.
4).
We
try
to
use
mass
center
of
instance
mask
as
object
centers
for
Heatmap
and
Coefficients
in-
stead of box center, and it improves accuracy a bit (see Ta-
ble 3)
In
Table
3
we
show
influence
of
our
modification
of
”de-
fault”
learning
rate
schedule,
”default”
loss
functions
and
’default’
object
centers
position
on
Heatmap
for
the
case
of MobileNetV2 backbone.
The best results for piece-wise
constant learning rate schedule (which we called ”default”)
we obtain using SGD with momentum
µ
=
0
.
9
, learning rate
η
=
0
.
02
and standard ”ReduceOnPlateau” technique, i.e.
by
10
x
reducing
learning
rate
each
time
”Patience”
=
5
Table 3: Instance segmentation accuracy for cases 1) Piece-
wise
constant
learning
rate
schedule,
2)
LRfinder,
3)
3x
mask upsampling
L
mask
, 4) perturbation of the first poly-
nomial
coefficient
of
the
Taylor’s
expansion
ϵ
2
=
−
0
.
05
,
5)
l
1
loss
and
UnitBox
loss
(Yu
et
al.,
2016)
with
α
1
=
0
.
05
,
β
1
=
1
.
5
, 6) Cross Entropy loss and Dice loss (Mil-
letari, Navab, and Ahmadi, 2016) with
β
1
= 0
.
5
,
β
2
= 2
.
5
,
and 7) using mass center instead of box center.
1)
2)
3)
4)
5)
6)
7)
AP
mask
✓
18.7
✓
19.8
✓
✓
20.7
✓
✓
✓
20.9
✓
✓
✓
✓
21.2
✓
✓
✓
✓
✓
21.6
✓
✓
✓
✓
✓
✓
21.9
epochs
without
accuracy
improvement.
The
cases
1)
and
2)
in
Table
3
shows
that
LRfinder
attains
1
.
1
AP
gains
in
comparison to piece-wise constant learning rate.
Then the
mask
upsampling
in
(9)
(case
3)
in
Table
3)
gives
+0
.
9
AP,
perturbation
of
the
first
polynomial
coefficient
of
the
Taylor’s
expansion
(case
4))
adds
0
.
2
AP,
mixing
l
1
and
UnitBox (Yu et al., 2016) losses with obtained coefficients
in
Table
A
of
Table
1
instead
of
”default
OD
loss”
adds
0
.
3
AP.
Modification
of
”default” mask
loss
with
β
1
=
1
,
β
2
=
0
by
using
obtained
in
Table
B
of
Table
1
param-
eters
achieved
+0
.
4
AP
improvement
and
using
another
Gaussian peak location in Ground Truth Heatmap adds
0
.
3
AP additionally. The whole improvement package achieves
significant
3
.
2
AP instant segmentation accuracy improve-
ment.
Vitalii Bulygin,
Dmytro Mykheievskyi,
Kyrylo Kuchynskyi
4.4
Results
As discussed in the Related work section, there are no pub-
lished results for real-time instance segmentation on smart-
phones
with
adequate
accuracy,
and
hence
Table
2
sets
a
new
benchmark
for
speed
and
accuracy
on
smartphones.
As
elaborated
earlier,
YolactEdge
(H.
Liu
et
al.,
2020)
when
used
with
the
lowest
computing
capability
available
in the literature - Jetson AGX Xavier - achieves
AP
mask
=
20
.
8
on
MS
COCO
val2017
dataset
with
35
.
7
FPS.
How-
ever,
AGX
Xavier
cannot
be
used
in
smartphones,
and
is
superior to Samsung S21 GPUs (Ignatov et al., n.d.).
Fur-
ther,
given
that
YolactEdge
input
resolution
is
550
×
550
,
we achieve better accuracy of
21
.
9
with a smaller input res-
olution
320
×
320
,
but
the
same
backbone
MobileNetV2
(see
Table
2).
Our
results
could
be
compared
with
mobi-
leYOLACT (J. Lee, S. Lee, and Ko, 2021) which achieves
AP
mask
50
= 23
.
0
at the speed of
21
fps on Samsung Galaxy
S20.
In contrast, our accuracy for the case of EfficientNet-
Lite
B4
backbone
is
AP
mask
50
=
47
.
1
at
the
speed
of
39
.
4
fps(see
Table
2)
on
Samsung
Galaxy
S21.
To
allow
comparison with mobile YOLACT, we additionally ran our
method
on
Samsung
S20
and
achieved
36.8
fps
(the
same
accuracy as on S21).
This shows that our model has 2x bet-
ter accuracy and 1.75x faster inference speed than mobileY-
OLACT (J. Lee, S. Lee, and Ko, 2021).
Moreover, another
approach
EOLO
(Zeng
and
Sabah,
2020)
on
Raspberry
Pi4 with Google Coral USB, comparable with smartphone
computing capability, achieves only
AP
mask
= 11
.
7
.
Furthermore, we compare our box detection accuracy with
state-of-the-art object detection on mobile YOLObite (Cai
et al., 2020).
In the case of EfficientNet-Lite B4 backbone,
our approach achieves
AP
mask
= 28
.
9
and
AP
box
= 31
.
4
at
29.2
fps
on
Samsung
S21
Adreno
660
GPU
whereas
YOLObile reaches
AP
box
=
31
.
6
at 19.1 fps on Samsung
S20
Adreno
650
GPU.
Mobile
ranking
(
Mobile
Ranking
2022)
shows
that
S21
GPU
speed
is
similar
to
S20
GPU
speed, so that our approach reaches state-of-the-art in Ob-
ject Detection on mobile (note, we did not target this capa-
bility).
To utilize BlitzMask just for box detection, one can
cut segmentation
head
(i.e.
layers
related
to
Spanning Set
and
Coefficient
outputs
in
Fig.
1)
from
the
model
and
get
AP
box
= 31
.
4
at 30.4 fps. Table 2) shows BlitzMask speed
and accuracy on MobileNetV1 (A. G. Howard et al., 2017),
MobileNetV2 (Sandler et al., 2018a) and EfficientNet-Lite
B0-B4 (M. Tan and Le, 2019), (
EfficientNet-Lite
2022). We
found
EfficientNet-Lite
backbones
are
very
efficient,
and
they
reach
24
.
9
AP
on
44.8
fps
(B0
case)
and
28
.
9
AP
on
29.2
fps
(B4
case)
that
is
significantly
better
than
Mo-
bileNet
backbones.
Lastly,
Table
2)
shows
results
for
our
heavier and more precise model with ResNet-18 backbone
operating
at
11
.
2
FPS
on
Samsung
S21.
While
not
suffi-
cient
for
real-time
instance
segmentation,
it
still
could
be
used for ”near real-time” tasks.
ResNet18* in Table 2) de-
notes
ResNet-18
without
3
×
3
,
stride
2
MaxPool.
This
backbone is significantly heavier than the original ResNet-
18 (He et al., 2015), but in our experiments, it has a better
accuracy/speed trade-off.
5
CONCLUSIONS
We
propose
a
new
fast
anchor-free
instance
segmenta-
tion
approach
BlitzMask
whose
mask
head
requires
sig-
nificantly
less
resources
than
the
entire
model.
We
use
a
modified YOLACT (Bolya et al., 2019a) loss function and
a special training process to increase model accuracy.
One
of the main BlitzMask advantages is its simplicity, which is
very important for a further method development or/and re-
implementation
from
scratch.
Our
experiments
show
that
mixing of
l
1
and UnitBox losses for the box regression part
and
Cross-Entropy
and
Dice
losses
for
the
segmentation
part improves model performance. In addition, using center
mass
of
masks
as
location
of
Gaussian
peaks
in
Heatmap
improves model accuracy.
Simple and fast post-processing
allows
achieving
real-time
speed
on
mobile
devices
with
sufficient
accuracy.
BlitzMask
achieves
AP
mask
=
28
.
9
at
29.2
fps
on
Samsung
S21
GPU.
This
sets
a
new
speed
and
accuracy
benchmark
for
instance
segmentation
infer-
ence
on
mobile
devices.
In
addition,
BlitzMask
reaches
state-of-the-art object detection accuracy for real-time mo-
bile applications.
Acknowledgments
Special
thanks
to
Dr.
Viktor
Porokhonskyy
(Samsung
R&D Institute, Ukraine) for his expertise, assistance, help-
ful comments and wise advice in writing the manuscript.
References
T¨urkmen, Sercan and Janne Heikkil¨a (2019). “An efficient
solution
for
semantic
segmentation:
Shufflenet
v2
with
atrous
separable
convolutions”.
In:
Scandinavian
Con-
ference
on
Image
Analysis
.
Springer,
pp.
41–53
(cit.
on
p. 1).
Howard,
Andrew
et
al.
(2019).
“Searching
for
mo-
bilenetv3”.
In:
Proceedings
of
the
IEEE/CVF
interna-
tional
conference
on
computer
vision
,
pp.
1314–1324
(cit. on p. 1).
Orsic,
Marin
et
al.
(2019).
“In
defense
of
pre-trained
im-
agenet
architectures
for
real-time
semantic
segmenta-
tion
of
road-driving
images”.
In:
Proceedings
of
the
IEEE/CVF Conference on Computer Vision and Pattern
Recognition
, pp. 12607–12616 (cit. on p. 1).
Cai, Yuxuan et al. (2020). “YOLObile: Real-time object de-
tection
on
mobile
devices
via
compression-compilation
co-design”. In:
arXiv preprint arXiv:2009.05697
(cit. on
pp. 1, 9).
Vasu, Pavan Kumar Anasosalu et al. (2022). “An Improved
One
millisecond
Mobile
Backbone”.
In:
arXiv
preprint
arXiv:2206.04040
(cit. on p. 1).
BlitzMask:
Real-Time Instance Segmentation Approach for Mobile Devices
Bazarevsky,
V
et
al.
(2020).
“BlazePose:
On-device
real-
time
body
pose
tracking.
arXiv”.
In:
arXiv
preprint
arXiv:2006.10204
(cit. on p. 1).
Minaee, Shervin et al. (2021). “Image segmentation using
deep learning: A survey”. In:
IEEE transactions on pat-
tern analysis and machine intelligence
(cit. on p. 1).
Liu,
Haotian
et
al.
(2020).
“YolactEdge:
Real-time
In-
stance
Segmentation
on
the
Edge”.
In:
arXiv
preprint
arXiv:2012.12259
(cit. on pp. 1–3, 9).
Zeng,
Longfei
and
Mohammed
Sabah
(2020).
“EOLO:
Embedded
Object
Segmentation
only
Look
Once”.
In:
arXiv preprint arXiv:2004.00123
(cit. on pp. 1, 3, 9).
Dai,
Jifeng
et
al.
(2017).
“Deformable
convolutional
net-
works”.
In:
Proceedings
of
the
IEEE
international
con-
ference on computer vision
, pp. 764–773 (cit. on pp. 1–
3).
Wang,
Xinlong,
Rufeng
Zhang,
et
al.
(2020).
“SOLOv2:
Dynamic
and
fast
instance
segmentation”.
In:
arXiv
preprint arXiv:2003.10152
(cit. on pp. 1–3).
Wang, Xinlong, Tao Kong, et al. (2020). “SOLO: Segment-
ing
objects
by
locations”.
In:
European
Conference
on
Computer
Vision
.
Springer,
pp.
649–665
(cit.
on
pp.
1–
3).
Wang,
Yuqing
et
al.
(2020).
“CenterMask:
single
shot
in-
stance segmentation with point representation”. In:
Pro-
ceedings
of
the
IEEE/CVF
conference
on
computer
vi-
sion
and
pattern
recognition
,
pp.
9313–9321
(cit.
on
pp. 1–4).
Sandler,
Mark
et
al.
(2018a).
“MobileNetV2:
Inverted
residuals and linear bottlenecks”. In:
Proceedings of the
IEEE conference on computer vision and pattern recog-
nition
, pp. 4510–4520 (cit. on pp. 1, 8, 9).
Bolya,
Daniel
et
al.
(2019a).
“YOLACT:
Real-time
in-
stance segmentation”. In:
Proceedings of the IEEE/CVF
International Conference on Computer Vision
, pp. 9157–
9166 (cit. on pp. 2, 3, 6, 8, 9).
–
(2019b).
“YOLACT++:
Better
real-time
instance
seg-
mentation”. In:
arXiv preprint arXiv:1912.06218
(cit. on
p. 2).
Lee,
Juwon,
Seungjae
Lee,
and
Jong
Gook
Ko
(2021).
“Mobile
YOLACT:
Toward
Lightweight
Instance
Seg-
mentation
for
Mobile
Devices”.
In:
2021
International
Conference
on
Information
and
Communication
Tech-
nology Convergence (ICTC)
. IEEE, pp. 1456–1460 (cit.
on pp. 2, 3, 9).
Redmon,
Joseph
and
Ali
Farhadi
(2018).
“YOLOv3:
An
incremental
improvement”.
In:
arXiv
preprint
arXiv:1804.02767
(cit. on p. 2).
Zhou,
Xingyi,
Dequan
Wang,
and
Philipp
Kr¨ahenb¨uhl
(2019).
“Objects
as
points”.
In:
arXiv
preprint
arXiv:1904.07850
(cit. on pp. 2–7).
Chen,
Hao
et
al.
(2020).
“BlendMask:
Top-down
meets
bottom-up
for
instance
segmentation”.
In:
Proceedings
of the IEEE/CVF conference on computer vision and pat-
tern recognition
, pp. 8573–8581 (cit. on p. 2).
Xie,
Enze
et
al.
(2020).
“PolarMask:
Single
Shot
Instance
Segmentation
with
Polar
Representation”.
In:
Proceed-
ings
of
the
IEEE/CVF
conference
on
computer
vision
and pattern recognition
, pp. 12193–12202 (cit. on pp. 2,
3).
Lin,
Tsung-Yi,
Michael
Maire,
et
al.
(2014).
Microsoft
COCO: Common Objects in Context
(cit. on pp. 2, 7).
Tian,
Zhi
et
al.
(2019).
“FCOS:
Fully
Convolutional
One-Stage
Object
Detection”.
In:
Proceedings
of
the
IEEE/CVF international conference on computer vision
,
pp. 9627–9636 (cit. on p. 3).
Sandler,
Mark
et
al.
(2018b).
“MobileNetV2:
Inverted
Residuals
and
Linear
Bottlenecks”.
In:
Proceedings
of
the
IEEE
Conference
on
Computer
Vision
and
Pattern
Recognition (CVPR)
(cit. on p. 3).
Ignatov,
Andrey
et
al.
(n.d.).
“Ai
benchmark:
All
about
deep
learning
on
smartphones
in
2019.
In
2019
IEEE”.
In:
CVF
International
Conference
on
Computer
Vision
Workshop (ICCVW)
, pp. 3617–3635 (cit. on pp. 3, 9).
Jetson
benchmarks
(2022).
Accessed:
2022-02-16.
URL
:
https
:
/
/
developer
.
nvidia
.
com
/
embedded/jetson-benchmarks
(cit. on p. 3).
Mobile
Ranking
(2022).
Accessed:
2022-10-11.
URL
:
https : / / ai - benchmark . com / ranking _
detailed.html
(cit. on pp. 3, 9).
Zhu,
Lingyu
et
al.
(2019).
“Portrait
instance
segmenta-
tion
for
mobile
devices”.
In:
2019
IEEE
International
Conference
on
Multimedia
and
Expo
(ICME)
.
IEEE,
pp. 1630–1635 (cit. on p. 3).
Lin,
Tsung-Yi,
Piotr
Doll´ar,
et
al.
(2017).
“Feature
Pyra-
mid
Networks
for
Object
Detection”.
In:
2017
IEEE
Conference on Computer Vision and Pattern Recognition
(CVPR)
, pp. 936–944.
DOI
:
10.1109/CVPR.2017.
106
(cit. on p. 3).
Howard,
Andrew
G
et
al.
(2017).
“Mobilenets:
Efficient
convolutional
neural
networks
for
mobile
vision
appli-
cations”.
In:
arXiv
preprint
arXiv:1704.04861
(cit.
on
pp. 3, 4, 8, 9).
Chiang,
Cheng-Ming
et
al.
(2020).
“Deploying
image
de-
blurring across mobile devices: A perspective of quality
and latency”. In:
Proceedings of the IEEE/CVF Confer-
ence on Computer Vision and Pattern Recognition Work-
shops
, pp. 502–503 (cit. on p. 3).
Ioffe,
Sergey
and
Christian
Szegedy
(2015).
“Batch
nor-
malization:
Accelerating
deep
network
training
by
re-
ducing internal covariate shift”. In:
International confer-
ence on machine learning
. PMLR, pp. 448–456 (cit. on
p. 3).
LeCun,
Yann
et
al.
(1998).
“Gradient-based
learning
ap-
plied
to
document
recognition”.
In:
Proceedings
of
the
IEEE
86.11, pp. 2278–2324 (cit. on p. 3).
Law,
Hei
and
Jia
Deng
(2018).
“Cornernet:
Detecting
ob-
jects
as
paired
keypoints”.
In:
Proceedings
of
the
Euro-
pean
conference
on
computer
vision
(ECCV)
,
pp.
734–
750 (cit. on pp. 4, 6).
Vitalii Bulygin,
Dmytro Mykheievskyi,
Kyrylo Kuchynskyi
Liu,
Zili
et
al.
(2020).
“Training-time-friendly
network
for
real-time
object
detection”.
In:
Proceedings
of
the
AAAI
Conference
on
Artificial
Intelligence
.
Vol.
34.
07,
pp. 11685–11692 (cit. on p. 4).
Lin,
Tsung-Yi,
Priya
Goyal,
et
al.
(2017).
“Focal
loss
for
dense object detection”. In:
Proceedings of the IEEE in-
ternational
conference
on
computer
vision
,
pp.
2980–
2988 (cit. on pp. 4, 5).
Leng,
Zhaoqi
et
al.
(2022).
“PolyLoss:
A
Polynomial
Ex-
pansion
Perspective
of
Classification
Loss
Functions”.
In:
arXiv preprint arXiv:2204.12511
(cit. on p. 4).
Yu, Jiahui et al. (2016). “Unitbox: An advanced object de-
tection network”. In:
Proceedings of the 24th ACM inter-
national conference on Multimedia
, pp. 516–520 (cit. on
pp. 5, 8).
Jadon,
Shruti
(2020).
“A
survey
of
loss
functions
for
se-
mantic
segmentation”.
In:
2020
IEEE
Conference
on
Computational Intelligence in Bioinformatics and Com-
putational Biology (CIBCB)
. IEEE, pp. 1–7 (cit. on p. 5).
Milletari,
Fausto,
Nassir
Navab,
and
Seyed-Ahmad
Ah-
madi
(2016).
“V-net:
Fully
convolutional
neural
net-
works
for
volumetric
medical
image
segmentation”.
In:
2016
fourth
international
conference
on
3D
vision
(3DV)
. IEEE, pp. 565–571 (cit. on pp. 5, 8).
Tan,
Mingxing
and
Quoc
Le
(2019).
“Efficientnet:
Re-
thinking
model
scaling
for
convolutional
neural
net-
works”.
In:
International
conference
on
machine learn-
ing
. PMLR, pp. 6105–6114 (cit. on pp. 8, 9).
EfficientNet-Lite
(2022).
Accessed:
2022-10-03.
URL
:
https
:
/
/
github
.
com
/
tensorflow
/
tpu
/
tree
/
master
/
models
/
official
/
efficientnet/lite
(cit. on pp. 8, 9).
He,
Kaiming
et
al.
(2015).
“Deep
Residual
Learn-
ing
for
Image
Recognition”.
In:
arXiv
preprint
arXiv:1512.03385
. arXiv:
1512.03385
(cit. on pp. 8,
9).
Izmailov,
Pavel
et
al.
(2018).
“Averaging
weights
leads
to
wider
optima
and
better
generalization”.
In:
arXiv
preprint arXiv:1803.05407
(cit. on p. 7).
Smith, Leslie N (2018). “A disciplined approach to neural
network
hyper-parameters:
Part
1–learning
rate,
batch
size,
momentum,
and
weight
decay”.
In:
arXiv
preprint
arXiv:1803.09820
(cit. on p. 7).
BlitzMask:
Real-Time Instance Segmentation Approach for Mobile Devices
A
TRAINING DETAILS
Here, we describe the training details of how we achieved
AP
mask
=
24
.
9
with backbone EfficientNet-Lite B0.
We use
ϵ
1
=
0
,
ϵ
2
=
−
0
.
05
,
α
1
=
0
.
05
,
β
1
=
1
.
5
inside
loss
function
L
box
and
β
1
=
0
.
5
,
β
2
=
2
.
5
inside
loss
function
L
mask
,
train without predicted mask upsampling using LRfinder and then tune with mask upsampling.
These values are
obtained
by
brute-forcing
loss
function
parameters
in
’fast
training’
described
in
Section
4.2.
Other
backbones
show
a
similar training process, but for larger EfficientNet-Lite number (
B
0
>
B
1
>
B
2
>
· · ·
) training process is a bit longer.
In
the
Accent
phase
(see
Section
4)
we
use
momentum
µ
=
0
.
9
and
the
learning
rate
is
increased
each
epoch
on
value
0
.
0025
linearly.
After
49
epochs
training
loss
starts
deteriorating.
Therefore,
previous
epoch
weights
are
loaded,
and
increasing
step
is
set
to
0
.
00125
.
Analogically,
the
learning
rate
increasing
step
is
reduced
after
57
and
65
epoch
and
at
78 epoch the Plateau phase us started with learning rate
η
= 0
.
1403125
,
AP
mask
= 17
.
8
.
The Plateau phase is continued
till
261
epoch,
e.g.
it
takes
183
epochs.
The
SGD
momentum
is
reduced
to
µ
=
0
.
63
at
120
epoch,
to
µ
=
0
.
44
at
202
epoch,
to
µ
=
0
.
3
at
223
epoch.
The
criteria
of
momentum
reducing
or
switching
to
Descent
phase
(if
µ
=
0
.
3
)
is
absence
of
training
loss
improvement
till
Patience
=
5
epochs.
The
Descent
phase
begins
with
10
time
reduced
learning
rate
η
=
0
.
01403125
,
µ
=
0
.
3
and due to above-mentioned criteria learning rate becomes equal to
0
.
01
·
η
at 282 epoch
and to
0
.
001
·
η
at
301
epoch.
The training process is finished after
312
epoch and accuracy
AP
mask
=
23
.
7
.
Then we
use upsampling by a factor of 3 for predicted and Ground Truth masks as it is described in Section 3.2.2 and tune trained
model with momentum
µ
=
0
.
9
,
learning rate
eta
=
0
.
0001
for
30
epoch and finally achieve
AP
mask
=
24
.
9
.
We just
mention that EfficientNet-Lite B1 requires 332 epochs, B2 - 334 epochs, B3 - 358 epochs, B4 - 391 epochs.
In each case,
the model is tuned for a 30 epoch with learning rate
eta
= 0
.
0001
using predicted mask
3
x
upsampling in loss function.
B
TRAINING FROM SCRATCH
Each
type
of
mobile
devices
has
its
own
computational
specifics,
e.g.
some
operations
have
either
no
or
poor
support,
requiring
model
architecture
modifications.
Traditionally,
each
such
model
modification
is
pre-trained
on
ImageNet
or
another
large
dataset,
making
the
model
development
process
much
more
complex.
To
account
for
this,
we
train
from
scratch similarly to ScratchDet.
Using
pre-trained
weight
such
as
ImageNet
is
not
a
big
problem
for
research
purposes
only.
But
it
could
be
a
problem
in
the
industry
where
a
license
for
a
large
dataset
may
be
too
expensive
and
hence
training
from
scratch
using
a
smaller
private dataset is in demand.
Moreover, if one has a many configurations and backbones and needs to train many models
“on the conveyor” then pretraining step makes the whole process more complex
C
VISUALIZATION OF INSTANCE SEGMENTATION RESULTS
In
Fig.5
we
show
inference
of
BlitzMask,
input
resolution
320
×
320
on
20
random
images
of
MS
COCO
val2017
for
MobileNetV2
backbone
and
EfficientNet-Lite
B4.
One
can
see
that
B4
backbone
gives
much
better
accuracy
than
Mo-
bileNetV2 backbone.
Vitalii Bulygin,
Dmytro Mykheievskyi,
Kyrylo Kuchynskyi
BlitzMask + MobileNetV2
BlitzMask + EfficientNet-Lite B4
Figure 5:
BlitzMask output with backbones MobileNetV2 (left) and EfficientNet-Lite B4(right) for input resolution
320
×
320
,
output
resolution
80
×
80
.
Red
rectangles
denote
predicted
bounding
boxes,
the
corresponding
object
type
and
confidence score are shown at the top of each box, different instance masks are shown by different colors on each image.
All images have the confidence threshold at 0.3.
