|
ISMIR2004
Audio Description Contest - Genre / Artist ID Classification Results
Genre
Classification Contest
Results are presented for each participant in the following format:
- The confusion matrix, ie: which tracks are correctly classified or
not depending on the class. It is read as "<column> is
classified as <row>", for instance, if classical is the
first column (and row) and world is the last, the number in the first
column, last row represent the number of classical tracks actually
classified as world music. A perfect matrix only contains numbers in
the diagonal.
- the average results (correctly identified tracks divided by total
number of tracks)
- the results per class and the normalized result with respect to the
probability of each class.
Participants
- Dan Ellis
& Brian Whitman
(Columbia University, MIT)
- Elias
Pampalk
(ÖFAI)
- George
Tzanetakis
(Univ. of Victoria)
- Kris West
(Univ. of East
Anglia)
- Thomas
Lidy & Andreas
Rauber (Vienna University of
Technology)
Results
Dan Ellis
& Brian Whitman
confusion matrix:
[[312
45 11 1 20 77]
[
3 42 1 5
19 12]
[
1 4 13
0 2 0]
[
0 3 0
23 8 1]
[
0 14 0 16
52 6]
[
3 7 1
0 0 27]]
(columns: classical electronic jazz_blues
metal_punk rock_pop world )
469
good
identifications out of 729 queries
averaging 64 % correct answers
total songs per class: [319 115 26 45 101 123]
results per class: [ 0.97806 0.36522 0.5
0.51111 0.51485 0.21951 ]
result
normalized with respect to the probability of each class: 51.48%
Elias Pampalk
confusion matrix:
[[312
1 2 0 3 18]
[
1 84 1 2
10 8]
[ 0
0 21 0 0 1]
[
0 2 0 34 5 0]
[
2 20 1 9 78 12]
[ 4
8 1
0 5 84]]
(columns: classical electronic jazz_blues metal_punk rock_pop world )
613 good identifications out of 729 queries
averaging 84.07 % correct answers
total songs per class: [319 115 26 45 101 123]
results per class: [ 0.97806 0.73043
0.80769 0.75556
0.77228 0.68293 ]
result normalized with respect to the probability of each class: 78.78%
George
Tzanetakis
confusion matrix:
[[301 7
8 1 11 60]
[
0 73 0 2 7 10]
[ 0
0 13 0 0 1]
[
0 0 0 16 6 2]
[
3 16 2 23 74 7]
[
15
19 3 3 3 43]]
(columns: classical electronic jazz_blues metal_punk rock_pop world )
520 good identifications out of 729 queries
averaging 71.33 % correct answers
total songs per class: [319 115 26 45 101 123]
results per class: [ 0.94357 0.63478
0.5 0.35556
0.73267 0.34959 ]
result normalized with respect to the probability of each class: 58.60%
Kris West
confusion matrix:
[[316 3
5 1 10 51]
[
1 98 1 6
13 16]
[ 0
0 16 0 0 1]
[
0 1 0 22 5 0]
[
1 10 3 16 67 4]
[ 1
3 1
0 6 52]]
(columns: classical electronic jazz_blues metal_punk rock_pop world )
571 good identifications out of 729 queries
averaging 78.33 % correct answers
total songs per class: [319 115 26 45 101 123]
results per class: [ 0.99059561 0.85217
0.61538 0.48889
0.66337 0.42276 ]
result normalized with respect to the probability of each class: 67.22%
Thomas
Lidy & Andreas Rauber
confusion matrix:
[[311 4
2 0 7 52]
[
0 80 2 9 12 18]
[ 0
4 11 0 1 2]
[
0 1 2 19 19 1]
[
0 12 4 13 44 2]
[
8
14 5 4 18 48]]
(columns: classical electronic jazz_blues metal_punk rock_pop world )
513 good identifications out of 729 queries
averaging 70.37 % correct answers
total songs per class: [319 115 26 45 101 123]
results per class: [ 0.97492 0.69565
0.42308 0.42222
0.43564 0.39024 ]
result normalized with respect to the probability of each class: 55.70%
>>
back to top
Artist Classification Contest
Due to technical limits, the original aim of running the submissions on
200 classes (artists) could not be achieved, so the submissions were
run both on 30 and 40 classes. The confusion matrices making much less
sense than in genre classification (due to the small number of
instances per class), only the average results will be presented.
Participants
-
Dan Ellis & Brian Whitman (Columbia University, MIT)
-
Thomas Lidy & Andreas Rauber (Vienna University of
Technology)
Results for 30 classes
Dan Ellis
& Brian Whitman
31 good identifications out of 90 queries, averaging 34% good answers
Thomas Lidy
& Andreas Rauber
26 good identifications out of 90 queries, averaging 28% good answers
Results for 40 classes
Dan Ellis
& Brian Whitman
29 good identifications out of 120 queries, averaging 24% good answers
Thomas Lidy
& Andreas Rauber
29 good identifications out of 120 queries, averaging 24% good answers
>>
back to top
Any comments or questions should be sent to Nicolas Wack
|
