nerdctl panic when run inspect --type=image xxxxx with nydus snapshotter

[reich80]

Description

nerdctl panic on running the command to inspect an image on the host with nydus snapshotter enabled. The issue exists in the latest release version. There is no problem if the nydus snapshotter not enabled.

root@xtao7:~# nerdctl inspect --type=image --mode dockercompat registry.servicemgr.xtao:5000/nydus/ubuntu22.04-cuda12.05-torch2.4-huggingface
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xd55373]

goroutine 1 [running]:
github.com/containerd/nerdctl/pkg/inspecttypes/dockercompat.ImageFromNative(0xc0001ef080)
/go/src/github.com/containerd/nerdctl/pkg/inspecttypes/dockercompat/dockercompat.go:296 +0x253
github.com/containerd/nerdctl/pkg/cmd/image.Inspect.func1({0x13b33a0?, }, {{{0xc0005aa000, 0x55}, 0x0, {{0xc000590040, 0x34}, {0xc0005ac000, 0x47}, 0xbc7, ...}, ...}, ...})
/go/src/github.com/containerd/nerdctl/pkg/cmd/image/inspect.go:52 +0x205
github.com/containerd/nerdctl/pkg/idutil/imagewalker.(*ImageWalker).Walk(0xc000679568, {0x13b33a0, 0xc00054c000}, {0x7fffc40cd4bb, 0x4e})
/go/src/github.com/containerd/nerdctl/pkg/idutil/imagewalker/imagewalker.go:81 +0x666
github.com/containerd/nerdctl/pkg/idutil/imagewalker.(*ImageWalker).WalkAll(0x0?, {0x13b33a0, 0xc00054c000}, {0xc0006797c0?, 0x1, 0xedeeb7e7b?}, 0x1)
/go/src/github.com/containerd/nerdctl/pkg/idutil/imagewalker/imagewalker.go:98 +0xea
github.com/containerd/nerdctl/pkg/cmd/image.Inspect({, _}, , {, _, _}, {{0x13ab7a0, 0xc0001ae008}, {0x0, 0x0, ...}, ...})
/go/src/github.com/containerd/nerdctl/pkg/cmd/image/inspect.go:64 +0x1a6
main.inspectAction(0xc0011e0300, {0xc0012d8040, 0x1, 0x4?})
/go/src/github.com/containerd/nerdctl/cmd/nerdctl/inspect.go:147 +0xb7c
github.com/spf13/cobra.(*Command).execute(0xc0011e0300, {0xc0001b0140, 0x4, 0x4})
/go/pkg/mod/github.com/spf13/[email protected]/command.go:940 +0x862
github.com/spf13/cobra.(*Command).ExecuteC(0xc000004300)
/go/pkg/mod/github.com/spf13/[email protected]/command.go:1068 +0x3bd
github.com/spf13/cobra.(*Command).Execute(...)
/go/pkg/mod/github.com/spf13/[email protected]/command.go:992
main.xmain()
/go/src/github.com/containerd/nerdctl/cmd/nerdctl/main.go:136 +0x9f
main.main()
/go/src/github.com/containerd/nerdctl/cmd/nerdctl/main.go:119 +0x19

Steps to reproduce the issue

Describe the results you received and expected

the nerdctl inspect should work on nydus snapshotter enabled, or provide some alternative solution

What version of nerdctl are you using?

nerdctl version 2.0.1

Are you using a variant of nerdctl? (e.g., Rancher Desktop)

None

Host information

No response

[apostasie]

@reich80 stack trace line numbers do match nerdctl 1.7.
Furthermore, nerdctl 2.0 uses cobra 1.8.1, while the stack trace reports cobra 1.7.0 (which was in use with nerdctl 1.7.0).

This is strange.

Would you be able to confirm this is actually nerdctl 2.0.1?

Thanks a lot.

[reich80]

@reich80 stack trace line numbers do match nerdctl 1.7. Furthermore, nerdctl 2.0 uses cobra 1.8.1, while the stack trace reports cobra 1.7.0 (which was in use with nerdctl 1.7.0).

This is strange.

Would you be able to confirm this is actually nerdctl 2.0.1?

Thanks a lot.

I confirm it.

root@xtao8:# nerdctl images
REPOSITORY TAG IMAGE ID CREATED PLATFORM SIZE BLOB SIZE
registry.servicemgr.xtao:5000/nydus/ubuntu22.04-cuda12.05-torch2.5-huggingface latest 83c3c0071b1b 15 hours ago linux/amd64 29.45MB 9.024GB
registry.servicemgr.xtao:5000/test/myredis latest ddce5cdfd3c6 15 hours ago linux/amd64 125.4MB 45.5MB
root@xtao8:# nerdctl --version
nerdctl version 2.0.1
root@xtao8:~# nerdctl inspect --type=image registry.servicemgr.xtao:5000/nydus/ubuntu22.04-cuda12.05-torch2.5-huggingface
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xd1ca48]

goroutine 1 [running]:
github.com/containerd/nerdctl/v2/pkg/inspecttypes/dockercompat.ImageFromNative(0xc0001b0848)
/go/src/github.com/containerd/nerdctl/pkg/inspecttypes/dockercompat/dockercompat.go:343 +0x488
github.com/containerd/nerdctl/v2/pkg/cmd/image.Inspect({_, _}, , {, _, _}, {{0x1486440, 0xc000142058}, {0x0, 0x0, ...}, ...})
/go/src/github.com/containerd/nerdctl/pkg/cmd/image/inspect.go:151 +0x9c5
github.com/containerd/nerdctl/v2/cmd/nerdctl/inspect.inspectAction(0xc000003508, {0xc0003b0e00, 0x1, 0x2?})
/go/src/github.com/containerd/nerdctl/cmd/nerdctl/inspect/inspect.go:153 +0xb34
github.com/spf13/cobra.(*Command).execute(0xc000003508, {0xc0001580e0, 0x2, 0x2})
/go/pkg/mod/github.com/spf13/[email protected]/command.go:985 +0xaaa
github.com/spf13/cobra.(*Command).ExecuteC(0xc0001ae308)
/go/pkg/mod/github.com/spf13/[email protected]/command.go:1117 +0x3ff
github.com/spf13/cobra.(*Command).Execute(...)
/go/pkg/mod/github.com/spf13/[email protected]/command.go:1041
main.xmain()
/go/src/github.com/containerd/nerdctl/cmd/nerdctl/main.go:144 +0x8f
main.main()
/go/src/github.com/containerd/nerdctl/cmd/nerdctl/main.go:127 +0x13

[apostasie]

Thanks a lot @reich80

Your last stack trace above is for 2.0 ^ indeed, and makes much more sense.

Issue is definitely https://github.com/containerd/nerdctl/blob/main/pkg/inspecttypes/dockercompat/dockercompat.go#L343

So, I assume somehow Created is not set in that case, and we should test it.

@AkihiroSuda easy patch.

Clearly affects 1.7 too.

Anyone wants to take it?

[apostasie]

1.7 would be:

https://github.com/containerd/nerdctl/blob/release/1.7/pkg/inspecttypes/dockercompat/dockercompat.go#L296

[reich80]

Thanks a lot @reich80

Your last stack trace above is for 2.0 ^ indeed, and makes much more sense.

Issue is definitely https://github.com/containerd/nerdctl/blob/main/pkg/inspecttypes/dockercompat/dockercompat.go#L343

So, I assume somehow Created is not set in that case, and we should test it.

@AkihiroSuda easy patch.

Clearly affects 1.7 too.

Anyone wants to take it?

Which version will fix the bug? It is currently a block issue for us.

[apostasie]

Thanks a lot @reich80
Your last stack trace above is for 2.0 ^ indeed, and makes much more sense.
Issue is definitely https://github.com/containerd/nerdctl/blob/main/pkg/inspecttypes/dockercompat/dockercompat.go#L343
So, I assume somehow Created is not set in that case, and we should test it.
@AkihiroSuda easy patch.
Clearly affects 1.7 too.
Anyone wants to take it?

Which version will fix the bug? It is currently a block issue for us.

This is open-source software, provided by volunteers giving up their time for free.
Your profile suggests "for us" is a for-profit company.
If "this" is a blocking issue for you and you are in a rush, find and pay someone to fix it.

[reich80]

Thanks a lot @reich80
Your last stack trace above is for 2.0 ^ indeed, and makes much more sense.
Issue is definitely https://github.com/containerd/nerdctl/blob/main/pkg/inspecttypes/dockercompat/dockercompat.go#L343
So, I assume somehow Created is not set in that case, and we should test it.
@AkihiroSuda easy patch.
Clearly affects 1.7 too.
Anyone wants to take it?

Which version will fix the bug? It is currently a block issue for us.

This is open-source software, provided by volunteers giving up their time for free. Your profile suggests "for us" is a for-profit company. If "this" is a blocking issue for you and you are in a rush, find and pay someone to fix it.

thanks. Sorry for my last reply.