Discrepancy between source paper and evaluation results based on Anomalib 1.1.0

[cjy513203427]

Hello Anomalib community,
I just wanted to say a big thank you for all your hard work! I wrote my master's thesis based on the Anomalib library and I found something really interesting. These are the benchmark results with regard to image auroc.

	Screw	Pill	Capsule	Carpet	Grid	Tile	Wood	Zipper	Cable	Toothbrush	Transistor	Metal Nut	Bottle	Hazelnut	Leather	Average	Source
CFA	87.68	93.10	48.17	50.00	84.04	37.50	62.50	40.34	61.96	98.06	45.00	39.78	50.00	60.00	50.00	60.54	99.30
CFLOW	81.57	92.91	95.45	97.27	88.47	100.00	99.39	97.64	94.15	94.17	94.88	100.00	100.00	99.68	100.00	95.71	96.31
CSFLOW	41.58	38.09	60.77	99.16	78.36	90.44	95.75	88.93	67.31	44.72	50.54	74.46	89.92	71.02	100.00	72.74	93.50
DFKDE	69.17	64.18	72.04	68.42	46.69	92.50	81.32	88.26	68.67	78.89	81.12	76.20	92.94	77.57	77.79	75.72	77.40
DFM	77.79	96.89	92.98	90.93	65.41	98.67	97.81	97.51	94.27	96.39	94.79	91.72	100.00	96.82	100.00	92.80	94.30
DRAEM	30.01	74.55	77.02	72.47	80.45	86.80	95.96	78.90	63.98	70.42	90.21	93.55	98.10	77.50	86.65	78.44	98.00
DSR	56.67	70.16	72.72	43.82	97.08	80.70	90.75	78.97	76.96	96.94	91.04	81.04	86.59	81.64	83.97	79.27	98.20
FASTFLOW	65.85	76.60	69.64	93.38	96.32	93.36	98.16	72.69	67.62	72.50	89.62	81.33	99.68	79.86	99.90	83.77	99.40
FRE	58.11	83.82	76.59	94.18	64.75	98.99	98.16	93.91	85.53	95.00	90.92	84.12	99.37	92.29	99.93	87.71	98.40
GANomaly	32.79	59.96	26.57	21.71	57.23	54.42	60.88	41.05	52.47	49.17	33.46	26.30	47.78	53.86	36.68	43.62	42.10
PaDiM	78.95	79.95	86.48	97.99	87.47	94.55	97.46	77.46	85.96	82.50	94.54	98.34	99.52	88.32	100.00	89.97	96.70
PatchCore	98.11	94.76	97.85	99.12	98.08	98.81	98.77	99.21	99.10	100.00	100.00	99.80	100.00	100.00	100.00	98.91	98.10
RD	98.03	97.63	97.93	99.36	95.49	100.00	99.39	97.16	95.45	91.39	97.87	100.00	100.00	100.00	100.00	97.98	98.50
RKDE	50.58	68.77	51.93	-	75.36	67.72	62.54	75.37	85.83	77.17	65.00	90.63	85.07	100.00	100.00	75.43	-
STFPM	77.62	40.78	60.59	98.88	59.23	97.08	98.95	75.39	91.34	47.50	61.88	40.22	43.97	96.50	100.00	72.66	97.00
UFLOW	49.19	94.84	56.72	100.00	99.33	99.39	95.09	89.73	62.67	64.17	81.46	55.77	99.21	90.39	100.00	82.53	98.74

The last column presents the results obtained from the source paper. There is a notable discrepancy between the stated result (99.30) for CFA and the average result (60.54) obtained by our evaluation. However, the results for CSFLOW, DREAM, DSR, FASTFLOW, FRE, PaDiM, STFPM, and UFLOW exhibit a relatively minor discrepancy with our evaluated results. In contrast, CFLOW, DFM, PatchCore, and RD align precisely with our results.

Raw logs are under https://github.com/cjy513203427/IADBE/tree/master/logs/rawlogs.

The models were evaluated on Ubuntu 22.04 with an RTX 3090. As can be seen in the table, there is a discrepancy between the claimed and evaluated results. What is the reason for this discrepancy?