JSMA never success on CIFAR10

[amosy3]

First of all, I would like to thank you for this incredible work!
I try the following code, to attack CIFAR10 with JSMA. The attack fails all the time (the code works with other attacks).

import os
import pickle
import torch
import torchvision
import torchvision.transforms as transforms
import numpy as np
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from tqdm import tqdm
from advertorch.utils import predict_from_logits
from advertorch_examples.utils import _imshow
from advertorch.attacks import PGDAttack, FGSM, JSMA

def get_test_loader():
transform = transforms.Compose([transforms.ToTensor()])
testset = torchvision.datasets.CIFAR10(root='./data', train=False, download=True, transform=transform)
testloader = torch.utils.data.DataLoader(testset, batch_size=10, shuffle=False, num_workers=20)
return testloader

def get_pretrain_model():
with open('../models/resnetxt_acc_87.pkl', 'rb') as f:
net = pickle.load(f)
return net.module #net is a DataParallel object

testloader = get_test_loader()
net = get_pretrain_model()
adversary = JSMA(net, num_classes=10)

data = next(iter(testloader))
images, labels = data
cln_data, true_label = images.to('cuda'), labels.to('cuda')

adv_untargeted = adversary.perturb(cln_data, true_label)
preds = net(adv_untargeted)
estimate_prob, estimate_class = torch.max(preds .data, 1)

wrong = true_label!=estimate_class
print(wrong)

#output: tensor([0, 0, 0, 0, 0, 0, 0, 0, 0, 0], device='cuda:0', dtype=torch.uint8)

[gwding]

As the original JSMA paper does not contain experiments on CIFAR10, I'm actually not very sure about what performance should be achieved on CIFAR10. Are you trying to replicate any reported results, say from a paper?
One possibility I can think of is that JSMA on CIFAR10 might require different hyperparameters.

@tracyjin could you also comment on this?

[amosy3]

Thank you for your quick response!
I'm not trying to reproduce a result from an article. I checked mainly out of curiosity and was surprised by the result. I tried to change the hyperparameters, but I could not make it work.

[joellliu]

I am trying JSMA on CIFAR10. I guess the issue is due to JSMA is a targeted attack, so when you input true_label in adv_untargeted = adversary.perturb(cln_data, true_label), the attacker will target the adv images to the groundtruth label such that the accuracy on adv images is always (almost) 100%.

[SoEzreal]

I am trying JSMA on CIFAR10. I guess the issue is due to JSMA is a targeted attack, so when you input true_label in adv_untargeted = adversary.perturb(cln_data, true_label), the attacker will target the adv images to the groundtruth label such that the accuracy on adv images is always (almost) 100%.

I found a similar problem in the experimental comparison.I feel that even in non-targeted attacks, the second parameter of the sentence adv_untargeted = adversary.perturb (cln_data, true_label) should not be ‘true label’ but should be the result given by the target model. Do you think so？