Multi-scale hybrid vision transformer and Sinkhorn tokenizer for sewer defect classification

Joakim Bruslund Haurum; Meysam Madadi; Sergio Escalera Guerrero; Thomas B. Moeslund

doi:10.1016/j.autcon.2022.104614

Multi-scale hybrid vision transformer and Sinkhorn tokenizer for sewer defect classification

Joakim Bruslund Haurum, Meysam Madadi, Sergio Escalera Guerrero, Thomas B. Moeslund

Publikation: Bidrag til tidsskrift › Tidsskriftartikel › Forskning › peer review

10 Citationer (Scopus)

330 Downloads (Pure)

Abstract

A crucial part of image classification consists of capturing non-local spatial semantics of image content. This paper describes the multi-scale hybrid vision transformer (MSHViT), an extension of the classical convolutional neural network (CNN) backbone, for multi-label sewer defect classification. To better model spatial semantics in the images, features are aggregated at different scales non-locally through the use of a lightweight vision transformer, and a smaller set of tokens was produced through a novel Sinkhorn clustering-based tokenizer using distinct cluster centers. The proposed MSHViT and Sinkhorn tokenizer were evaluated on the Sewer-ML multi-label sewer defect classification dataset, showing consistent performance improvements of up to 2.53 percentage points.

Originalsprog	Engelsk
Artikelnummer	104614
Tidsskrift	Automation in Construction
Vol/bind	144
ISSN	0926-5805
DOI	https://doi.org/10.1016/j.autcon.2022.104614
Status	Udgivet - dec. 2022

Adgang til dokumentet

10.1016/j.autcon.2022.104614Licens: CC BY 4.0

Open Access articleForlagets udgivne version, 4,87 MBLicens: CC BY 4.0

AUB Link

Søg efter materialet i Aalborg Universitetsbiblioteks søgemaskine

Andre filer og links

Link to publication in Scopus

1 Afsluttet

ASIR: Automated Sewer Inspection Robot
Moeslund, T. B. (PI (principal investigator)), Haurum, J. B. (PI (principal investigator)), Bahnsen, C. H. (PI (principal investigator)) & Hansen, B. D. (PI (principal investigator))
Innovationsfonden
01/11/2018 → 30/04/2022
Projekter: Projekt › Forskning

10 Citationer
2 Konferenceartikel i proceeding
1 Paper uden forlag/tidsskrift
1 Review (oversigtsartikel)

Multi-Scale Hybrid Vision Transformer and Sinkhorn Tokenizer for Sewer Defect Classification
Haurum, J. B., Madadi, M., Møgelmose, A., Guerrero, S. E. & Moeslund, T. B., 2022. 6 s.
Publikation: Konferencebidrag uden forlag/tidsskrift › Paper uden forlag/tidsskrift › Forskning › peer review
Multi-Task Classification of Sewer Pipe Defects and Properties using a Cross-Task Graph Neural Network Decoder
Haurum, J. B., Madadi, M., Guerrero, S. E. & Moeslund, T. B., 2022, Proceedings - 2022 IEEE/CVF Winter Conference on Applications of Computer Vision, WACV 2022. IEEE (Institute of Electrical and Electronics Engineers), s. 1441-1452 12 s. (IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)).
Publikation: Bidrag til bog/antologi/rapport/konference proceeding › Konferenceartikel i proceeding › Forskning › peer review

Åben adgang
8 Citationer (Scopus)
Sewer-ML: A Multi-Label Sewer Defect Classification Dataset and Benchmark
Haurum, J. B. & Moeslund, T. B., 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR): IEEE (Institute of Electrical and Electronics Engineers), s. 13451-13462 12 s. 9577322. (I E E E Conference on Computer Vision and Pattern Recognition. Proceedings).
Publikation: Bidrag til bog/antologi/rapport/konference proceeding › Konferenceartikel i proceeding › Forskning › peer review

Åben adgang
48 Citationer (Scopus)

Sewer-ML
Haurum, J. B. (Ophavsperson) & Moeslund, T. B. (Ophavsperson), sciencedata.dk, 18 jun. 2021
https://forms.gle/hBaPtoweZumZAi4u9 og et link mere, https://vap.aau.dk/sewer-ml/ (vis færre)
Datasæt

Citationsformater

@article{90c6164bd8b54edd8c3cb966692b7f8d,

title = "Multi-scale hybrid vision transformer and Sinkhorn tokenizer for sewer defect classification",

abstract = "A crucial part of image classification consists of capturing non-local spatial semantics of image content. This paper describes the multi-scale hybrid vision transformer (MSHViT), an extension of the classical convolutional neural network (CNN) backbone, for multi-label sewer defect classification. To better model spatial semantics in the images, features are aggregated at different scales non-locally through the use of a lightweight vision transformer, and a smaller set of tokens was produced through a novel Sinkhorn clustering-based tokenizer using distinct cluster centers. The proposed MSHViT and Sinkhorn tokenizer were evaluated on the Sewer-ML multi-label sewer defect classification dataset, showing consistent performance improvements of up to 2.53 percentage points.",

keywords = "Closed-Circuit Television, Convolutional Neural Networks, Sewer Defect Classification, Sewer Inspection, Sinkhorn-Knopp, Vision Transformers",

author = "Haurum, {Joakim Bruslund} and Meysam Madadi and Guerrero, {Sergio Escalera} and Moeslund, {Thomas B.}",

year = "2022",

month = dec,

doi = "10.1016/j.autcon.2022.104614",

language = "English",

volume = "144",

journal = "Automation in Construction",

issn = "0926-5805",

publisher = "Elsevier",

}

TY - JOUR

T1 - Multi-scale hybrid vision transformer and Sinkhorn tokenizer for sewer defect classification

AU - Haurum, Joakim Bruslund

AU - Madadi, Meysam

AU - Guerrero, Sergio Escalera

AU - Moeslund, Thomas B.

PY - 2022/12

Y1 - 2022/12

N2 - A crucial part of image classification consists of capturing non-local spatial semantics of image content. This paper describes the multi-scale hybrid vision transformer (MSHViT), an extension of the classical convolutional neural network (CNN) backbone, for multi-label sewer defect classification. To better model spatial semantics in the images, features are aggregated at different scales non-locally through the use of a lightweight vision transformer, and a smaller set of tokens was produced through a novel Sinkhorn clustering-based tokenizer using distinct cluster centers. The proposed MSHViT and Sinkhorn tokenizer were evaluated on the Sewer-ML multi-label sewer defect classification dataset, showing consistent performance improvements of up to 2.53 percentage points.

AB - A crucial part of image classification consists of capturing non-local spatial semantics of image content. This paper describes the multi-scale hybrid vision transformer (MSHViT), an extension of the classical convolutional neural network (CNN) backbone, for multi-label sewer defect classification. To better model spatial semantics in the images, features are aggregated at different scales non-locally through the use of a lightweight vision transformer, and a smaller set of tokens was produced through a novel Sinkhorn clustering-based tokenizer using distinct cluster centers. The proposed MSHViT and Sinkhorn tokenizer were evaluated on the Sewer-ML multi-label sewer defect classification dataset, showing consistent performance improvements of up to 2.53 percentage points.

KW - Closed-Circuit Television

KW - Convolutional Neural Networks

KW - Sewer Defect Classification

KW - Sewer Inspection

KW - Sinkhorn-Knopp

KW - Vision Transformers

UR - http://www.scopus.com/inward/record.url?scp=85140050969&partnerID=8YFLogxK

U2 - 10.1016/j.autcon.2022.104614

DO - 10.1016/j.autcon.2022.104614

M3 - Journal article

SN - 0926-5805

VL - 144

JO - Automation in Construction

JF - Automation in Construction

M1 - 104614

ER -

Multi-scale hybrid vision transformer and Sinkhorn tokenizer for sewer defect classification

Abstract

Adgang til dokumentet

AUB Link

Andre filer og links

Fingeraftryk

Projekter

ASIR: Automated Sewer Inspection Robot

Publikation

Multi-Scale Hybrid Vision Transformer and Sinkhorn Tokenizer for Sewer Defect Classification

Multi-Task Classification of Sewer Pipe Defects and Properties using a Cross-Task Graph Neural Network Decoder

Sewer-ML: A Multi-Label Sewer Defect Classification Dataset and Benchmark

Forskningsdatasæt

Sewer-ML

Citationsformater