[New Feature][1.1.216] Integrated openpose editor

[huchenlei]

Sometimes the controlnet openpose preprocessor does not provide exact result we want. When this happens, we can either:

• Do a pose edit from 3rd party editors such as posex, and use that as input image with preprocessor none
  • Cons: Existing extensions have bad/no support for hand/face. They might not receive the most up to date pose detection code from ControlNet, as most of them copy a version of ControlNet's pose detection code.
• Download the json file provided by controlnet preview and try to correct it in a 3rd party editor
  • Cons: many steps of operation needed: download a file, upload to another tool, download the updated pose back, set it in the input image, choose preprocessor none.

To address this issue. We implemented an embedded openpose editor. Now if you are not satisfied with the pose output, you can click the Edit button on the generated image to send the pose to an editor for edit.

Currently, to use the edit feature, you will need controlnet v1.1.216 and another extension installed:
https://github.com/huchenlei/sd-webui-openpose-editor.
(Note: although the extension is build with Vue.js, the users do not need to setup the build environment. The extension will download the compiled app dist/ from github release.)

Here is a demo showing how to use the feature:
edit_demo.webm

If you don't like the feature, here is a guide on how to disable it:
feature_off_demo.webm

If you meet any problems with the editor, please submit an issue to the editor repo instead of the controlnet extension repo, thanks!

[chekaaa]

Should I install the openpose editor using the extension tab ? because I did and still show the message that I dont have the openpose editor installed.

[huchenlei]

You need to restart the WebUI commandline after installing the extension as it will mount the editor to openpose_editor_index on A1111 start. Clicking restart in the WebUI might not be sufficient.

[yuange250]

这个有用

[Woisek]

Here is a demo showing how to use the feature: [edit_demo.webm](https://github.com/Mikubill/sd-webui-controlnet/assets/20929282

Could you please provide a video in a broadly available format like MP4? Thanks.

[anonghuser]

Rename the WebM to MKV and you have a broadly available format!

[Woisek]

Seems not quite there yet ... 🤔

[Woisek]

Thanks for the reply, but ... scrolling scrolls the page and as soon I press SPACE, the page scrolls again down. 😕
I'm not sure if the space bar is a good choice. Are you aware of canvas zoom? -> https://github.com/richrobber2/canvas-zoom.git
Maybe you can adapt his shortcuts for the pan and zoom. Maybe even do it exactly, so we could use the same shortcuts on both extensions and wouldn't have to rethink every time. 😊

[huchenlei]

Fixed in v0.0.11. The default scrolling behaviour of pressing space is prevented. I know people might have different preference on which key to press. Maybe I will add a setting for keybindings if the list keeps growing. For now I would prefer SPACE as it matches the keybinding of Photoshop.

[Woisek]

Sorry to say but pressing space still triggers scrolling ... 😔

[huchenlei]

I cannot reproduce it locally. Can you submit an issue along with a video showing the behaviour? https://github.com/huchenlei/sd-webui-openpose-editor/issues

[Woisek]

I can submit the issue but a video ... ? When pressing the space bar, the page scrolls down. Nothing more complex about it. 😊
And unfortunately I have nothing to record this, so ... I can just tell you what's going on.

[lybmccree]

640 * 960的图放入controlnet,勾选pixel perfect，点击预览, 然后Edit，打开 窗口，他的512 * 768分辨率，然后我再上传这张素材当背景，完全对不上，可以让素材图片自动 缩到512*768吗

[huchenlei]

https://stackoverflow.com/questions/46468104/how-to-use-subtlecrypto-in-chrome-window-crypto-subtle-is-undefined ok那我看看能不能不调crypto算hash

[lybmccree]

好的，辛苦了

[huchenlei]

Fixed. Now we are using a 3rd party library to do hashing intead of relying on Web Crypto API.

[sdcarterchen]

好像有点不对，背景图是你自己上传的吗？如果是ControlNet点edit直接传过来的图，图的名字会是controlnet input，而且默认是lock状态。

你这边是点击edit没有从ControlNet那边自动带背景图过来吗？

我试着，大部分情况下pose和背景图对不上，需要调那个scale ratio去对上。还有个问题就是，手部分的手指颜色左右手表示的是同一个手指头吗？我看open pose自动生成的和这个插件里面显示的调整节点对不上

[huchenlei]

可以交一个issue到https://github.com/huchenlei/sd-webui-openpose-editor 吗？最好附上截图和复现的图片/json文件。

[katopz]

Good job! Looking forward to this.👍
BTW I'm curious why you chose Openpose over MediaPipe?

[huchenlei]

From the editor perspective, media pipe's face representation has too many keypoints, which is not suitable for manual edits. Openpose at basic body level provides 5 keypoints for head (nose, L/R eye, L/R ear), which is easy to manage.

[2blackbar]

Very good for editing already clean armatures but It needs a way to spawn missing hand instead of clicking points one by one, also premade face points for side,front,4/3 view, thats most used ones, like a library of premade json files ,theres not that many of them really for the most often used angles.
I mean yeah i guess i can make a library and save the hand poses myself but a button "spawn hand" or "spawn face" would be handy, these are the ones we will be fixing most often
Red highlighting on the list with names for the bone im grabbing would be handy , that shouldnt be too hard cause i can see the numbers changing when i move the point, so id make the number red and also its name red when its being moved
what up with black font on black background ?
Clone and flip hand?

Sometimes i get lost /duplicate dots of hands leftovers and cant get rid of them, oh it also happened with face, yeah i think this needs some checks, or a delete button
Wit the missing fingers, i think it wouldnt be THAT hard to approximate position of the missing fingers when theres already at least one finger ? it doesnt have to precise, now i have to manyually input where finger should be and position is kinda easy to figure out with code imo if you have at least one saved hand json file, then we can clean it up after all missing fingers will appear.Id rather have hand restored in wrong position than spend time adding bones one by one
Theres a bug where it will recognize both faces on image with 2 people, once i go into edit, the second face disappears

[2blackbar]

oh thanks, but how to mirror left hand right hand? i think ab utton for that would be handy, i got the hand spawned but i have to mirror it
Its pretty hard to scale the hand once its spawned, i think corners arent reacting that well to drag
But yeah this is already very good

[huchenlei]

There is a section in README explaining group vs selection, and a video demo on the difference. In fabricjs api, grouped objects cannot be scaled or rotated. So what you need to do is ungroup the hand, select the hand keypoints by drag on canvas, and you should be able to scale/rotate/mirror the selection.

I know this is awkward and counter intuitive. I am also working on a solution to it, but for now under my current setup, the grouped object is only moveable.

[2blackbar]

yeah it has its workflow i need to adapt to, i flip the hand once im in hand edit mode cause when i flip it before that then it explodes when going to edit mode.
Imo the editor is quite complete, i was able to fix the hand adding it and moving fingers pretty easily, You did a very good job
Can you lock selecting and moving non hand points when we are in hand editing mode? i dont really want to move wrist or anythin else but hand

[huchenlei]

Grouping is the mechanism for lock movement for other keypoints. You click the icon right besides the eye icon, and the object will be grouped. Once grouped, you cannot operate on individual keypoints within the group.

[huchenlei]

v0.1.2 implemented many features you requested:

• Selected keypoints will be highlighted in the control panel
• Body parts will be lockable
• Grouped objects can be scaled and rotated

[zxczjg123]

GitCommandError: Cmd('git') failed due to: exit code(128) cmdline: git clone -v --filter=blob:none -- #1548 D:\sd-webui-aki-v4.3\tmp\1548 stderr: 'Cloning into 'D:\sd-webui-aki-v4.3\tmp\1548'... warning: missing OAuth configuration for gitcode.net - see https://aka.ms/gcm/gitlab for more information remote: HTTP Basic: Access denied fatal: Authentication failed for 'https://gitcode.net/ranting8323/sd-webui-controlnet/discussions/1548.git/' '