Agglomerative Token Clustering

Joakim Bruslund Haurum; Sergio Escalera; Graham W. Taylor; Thomas B. Moeslund

doi:10.1007/978-3-031-72998-0_12

Agglomerative Token Clustering

Joakim Bruslund Haurum^*, Sergio Escalera, Graham W. Taylor, Thomas B. Moeslund

^*Corresponding author for this work

Research output: Contribution to book/anthology/report/conference proceeding › Article in proceeding › Research › peer-review

2 Citations (Scopus)

Abstract

We present Agglomerative Token Clustering (ATC), a novel token merging method that consistently outperforms previous token merging and pruning methods across image classification, image synthesis, and object detection & segmentation tasks. ATC merges clusters through bottom-up hierarchical clustering, without the introduction of extra learnable parameters. We find that ATC achieves state-of-the-art performance across all tasks, and can even perform on par with prior state-of-the-art when applied off-the-shelf, i.e. without fine-tuning. ATC is particularly effective when applied with low keep rates, where only a small fraction of tokens are kept and retaining task performance is especially difficult.

Original language	English
Title of host publication	Computer Vision – ECCV 2024 : 18th European Conference, Milan, Italy, September 29–October 4, 2024, Proceedings, Part LVII
Editors	Aleš Leonardis, Elisa Ricci, Stefan Roth, Olga Russakovsky, Torsten Sattler, Gül Varol
Number of pages	19
Publisher	Springer
Publication date	2024
Pages	200-218
ISBN (Print)	9783031729973
DOIs	https://doi.org/10.1007/978-3-031-72998-0_12
Publication status	Published - 2024
Event	18th European Conference on Computer Vision, ECCV 2024 - Milan, Italy Duration: 29 Sept 2024 → 4 Oct 2024

Conference

Conference	18th European Conference on Computer Vision, ECCV 2024
Country/Territory	Italy
City	Milan
Period	29/09/2024 → 04/10/2024

Series	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	15115 LNCS
ISSN	0302-9743

Bibliographical note

Publisher Copyright:
© The Author(s), under exclusive license to Springer Nature Switzerland AG 2025.

Keywords

Computer Vision
Vision Transformers
Token Reduction
Clustering
Agglomerative Clustering
Hierarchal Clustering

Access to Document

10.1007/978-3-031-72998-0_12

https://arxiv.org/pdf/2409.11923

AUB Link

Search for the material in Aalborg University Library's search engine

Cite this

Haurum, J. B., Escalera, S., Taylor, G. W., & Moeslund, T. B. (2024). Agglomerative Token Clustering. In A. Leonardis, E. Ricci, S. Roth, O. Russakovsky, T. Sattler, & G. Varol (Eds.), Computer Vision – ECCV 2024: 18th European Conference, Milan, Italy, September 29–October 4, 2024, Proceedings, Part LVII (pp. 200-218). Springer. https://doi.org/10.1007/978-3-031-72998-0_12

Haurum, Joakim Bruslund ; Escalera, Sergio ; Taylor, Graham W. et al. / Agglomerative Token Clustering. Computer Vision – ECCV 2024: 18th European Conference, Milan, Italy, September 29–October 4, 2024, Proceedings, Part LVII. editor / Aleš Leonardis ; Elisa Ricci ; Stefan Roth ; Olga Russakovsky ; Torsten Sattler ; Gül Varol. Springer, 2024. pp. 200-218 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), Vol. 15115 LNCS).

@inproceedings{9658478b526c4892a1a35a9377bb910e,

title = "Agglomerative Token Clustering",

abstract = "We present Agglomerative Token Clustering (ATC), a novel token merging method that consistently outperforms previous token merging and pruning methods across image classification, image synthesis, and object detection \& segmentation tasks. ATC merges clusters through bottom-up hierarchical clustering, without the introduction of extra learnable parameters. We find that ATC achieves state-of-the-art performance across all tasks, and can even perform on par with prior state-of-the-art when applied off-the-shelf, i.e. without fine-tuning. ATC is particularly effective when applied with low keep rates, where only a small fraction of tokens are kept and retaining task performance is especially difficult.",

keywords = "Computer Vision, Vision Transformers, Token Reduction, Clustering, Agglomerative Clustering, Hierarchal Clustering",

author = "Haurum, \{Joakim Bruslund\} and Sergio Escalera and Taylor, \{Graham W.\} and Moeslund, \{Thomas B.\}",

note = "Publisher Copyright: {\textcopyright} The Author(s), under exclusive license to Springer Nature Switzerland AG 2025.; 18th European Conference on Computer Vision, ECCV 2024 ; Conference date: 29-09-2024 Through 04-10-2024",

year = "2024",

doi = "10.1007/978-3-031-72998-0\_12",

language = "English",

isbn = "9783031729973",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer",

pages = "200--218",

editor = "Ale{\v s} Leonardis and Elisa Ricci and Stefan Roth and Olga Russakovsky and Torsten Sattler and G{\"u}l Varol",

booktitle = "Computer Vision – ECCV 2024",

address = "Germany",

}

Haurum, JB , Escalera, S, Taylor, GW & Moeslund, TB 2024, Agglomerative Token Clustering. in A Leonardis, E Ricci, S Roth, O Russakovsky, T Sattler & G Varol (eds), Computer Vision – ECCV 2024: 18th European Conference, Milan, Italy, September 29–October 4, 2024, Proceedings, Part LVII. Springer, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 15115 LNCS, pp. 200-218, 18th European Conference on Computer Vision, ECCV 2024, Milan, Italy, 29/09/2024. https://doi.org/10.1007/978-3-031-72998-0_12

Agglomerative Token Clustering. / Haurum, Joakim Bruslund ; Escalera, Sergio; Taylor, Graham W. et al.
Computer Vision – ECCV 2024: 18th European Conference, Milan, Italy, September 29–October 4, 2024, Proceedings, Part LVII. ed. / Aleš Leonardis; Elisa Ricci; Stefan Roth; Olga Russakovsky; Torsten Sattler; Gül Varol. Springer, 2024. p. 200-218 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), Vol. 15115 LNCS).

Research output: Contribution to book/anthology/report/conference proceeding › Article in proceeding › Research › peer-review

TY - GEN

T1 - Agglomerative Token Clustering

AU - Haurum, Joakim Bruslund

AU - Escalera, Sergio

AU - Taylor, Graham W.

AU - Moeslund, Thomas B.

N1 - Publisher Copyright: © The Author(s), under exclusive license to Springer Nature Switzerland AG 2025.

PY - 2024

Y1 - 2024

N2 - We present Agglomerative Token Clustering (ATC), a novel token merging method that consistently outperforms previous token merging and pruning methods across image classification, image synthesis, and object detection & segmentation tasks. ATC merges clusters through bottom-up hierarchical clustering, without the introduction of extra learnable parameters. We find that ATC achieves state-of-the-art performance across all tasks, and can even perform on par with prior state-of-the-art when applied off-the-shelf, i.e. without fine-tuning. ATC is particularly effective when applied with low keep rates, where only a small fraction of tokens are kept and retaining task performance is especially difficult.

AB - We present Agglomerative Token Clustering (ATC), a novel token merging method that consistently outperforms previous token merging and pruning methods across image classification, image synthesis, and object detection & segmentation tasks. ATC merges clusters through bottom-up hierarchical clustering, without the introduction of extra learnable parameters. We find that ATC achieves state-of-the-art performance across all tasks, and can even perform on par with prior state-of-the-art when applied off-the-shelf, i.e. without fine-tuning. ATC is particularly effective when applied with low keep rates, where only a small fraction of tokens are kept and retaining task performance is especially difficult.

KW - Computer Vision

KW - Vision Transformers

KW - Token Reduction

KW - Clustering

KW - Agglomerative Clustering

KW - Hierarchal Clustering

UR - https://www.scopus.com/pages/publications/85206350597

U2 - 10.1007/978-3-031-72998-0_12

DO - 10.1007/978-3-031-72998-0_12

M3 - Article in proceeding

AN - SCOPUS:85206350597

SN - 9783031729973

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 200

EP - 218

BT - Computer Vision – ECCV 2024

A2 - Leonardis, Aleš

A2 - Ricci, Elisa

A2 - Roth, Stefan

A2 - Russakovsky, Olga

A2 - Sattler, Torsten

A2 - Varol, Gül

PB - Springer

T2 - 18th European Conference on Computer Vision, ECCV 2024

Y2 - 29 September 2024 through 4 October 2024

ER -

Haurum JB , Escalera S, Taylor GW, Moeslund TB. Agglomerative Token Clustering. In Leonardis A, Ricci E, Roth S, Russakovsky O, Sattler T, Varol G, editors, Computer Vision – ECCV 2024: 18th European Conference, Milan, Italy, September 29–October 4, 2024, Proceedings, Part LVII. Springer. 2024. p. 200-218. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), Vol. 15115 LNCS). doi: 10.1007/978-3-031-72998-0_12

Agglomerative Token Clustering

Abstract

Conference

Bibliographical note

Keywords

Access to Document

AUB Link

Other files and links

Fingerprint

Pioneer Centre for AI

Cite this

Agglomerative Token Clustering

Abstract

Conference

Bibliographical note

Keywords

Access to Document

AUB Link

Other files and links

Fingerprint

Projects

Pioneer Centre for AI

Cite this