Can't get the paper results for Book-crossing dataset

[Gtarget]

Hi, I just run the code using Booking-crossing dataset for CTR prediction task without modifying any line of code, but it seems I could not reproduce the paper results, see my result:

I don't understand why. Could you kindly examine the default hyperparameter settings and help me reproduce the paper results?
Thanks a lot.

[hwwang55]

I tried on my computer just now and the code seems working well. Here is the result:

I found that the code can hardly reproduce the reported result on Windows. You can try it on Linux. Another thing is did you use the default setting for book-crossing (see main.py)? You cannot just run "python main.py --dataset book". The default setting is for movielens.

[Gtarget]

Thanks for your quick reply.
Yes, I am sure I used the default setting for book-crossing dataset.
But ,from your picture, the best test auc is 0.7223, and the best test acc is 0.6618. But your paper says you got auc 0.840 and acc 0.775 for book-crossing dataset.

This is exactly what makes me confused, could you kindly explain it?

[hwwang55]

That table is from the ealier version of the paper. We have found mistakes when pre-processing the book-crossing dataset and re-run the experiments for RippleNet and all baselines. The correct result can be found in the latest version of this paper in arxiv and the procedding of cikm (may not be available yet).

[Gtarget]

Oh,there. I'll go check the latest version, thx.

[ZJUWeifeng]

Hi, I have run the code under the default settings for the Book-Crossing dataset at least 10 times and my result is similar to Gtarget's.

As you can see, the best auc is 0.6951, which is worse than Wide&Deep's auc.
Could you tell me how to achieve the best auc reported in your paper? Are there any differences between the knowledge graph you use and the one you provide here?

[hwwang55]

I ran the code for 5 times just now, and here are the results:

There is indeed a chance that the model get stuck in AUC of 0.7, but the probability is small. I have checked the parameters and there is no problem. The only thing different is that I anonymized the IDs of entities (i.e., re-indexing) in the released datasets according to the privacy policy of Microsoft, but that should have no effect on the performance. I'll check that later.

[ZJUWeifeng]

Thanks a lot. I will try it again.

[ubear]

Thanks a lot. I will try it again.

how about the result?

[ZJUWeifeng]

Thanks a lot. I will try it again.

how about the result?

By using different random seeds, I get better results, which is 0.72~0.73 on the Book-Crossing dataset. Some random seeds will get stuck, and some will not.

[yorkchu1995]

By using different random seeds, I get better results, which is 0.72~0.73 on the Book-Crossing dataset. Some random seeds will get stuck, and some will not.

The seed is？

[zhenchengchang]

I tried on my computer just now and the code seems working well. Here is the result:

I found that the code can hardly reproduce the reported result on Windows. You can try it on Linux. Another thing is did you use the default setting for book-crossing (see main.py)? You cannot just run "python main.py --dataset book". The default setting is for movielens.

i want to ask why the results in your picture decrease after better results.

[cmaxhao]

I think it may suffer from overfitting.