Video generation

[IWULII]

Is there or will there be an opportunity to create videos using a bot?
The same question about the poses.

[wizz13150]

Hey

indeed, i had the same idea.
I found one and kind of working on it, but i'm a pretty bad python (not a) dev, LOL
Using Deforum. But i'm totally stuck with the API at this point. I only get 404 from it, huh

Screenshot:

This part would involve a decent GPU.

Here is the original repo for the Deforum Bot :

https://github.com/kabachuha/deforum-art-bot

If someone succeed, thanks to keep me in touch ! ((:

Cheers ! 🥂

[IWULII]

I believe that you will succeed!
I have 4090 so the performance is enough.

[wizz13150]

ahah, thanks ! ^^

For now, i'm more focus to waste my time to make my Progression thing to work, lol (#206)

I'll add a deforumcog.py soon and post a commit here. But i'm also fighting with the Deforum API.
Dev is NOT a life 😂😂

[wizz13150]

Okay, i'm starting to work on this.
This looks good and already able to generate from it, should be okay soon.

I'll make a functionnal version, and then customize it as well.

(:
Cheers ! 🥂

[IWULII]

Cool!!!
I can't wait for the result!
You will succeed

[wizz13150]

Very first test with default settings :

20230917080406.mp4

👍

[wizz13150]

Okay, here is a first functionnal sketch using the Simple-API :

• Install Deforum, obviously

• Add this argument to your webui-user.bat :
  --deforum-simple-api

• Add these lines to aiya.py (under intents.members = True, line ~15) :
  intents.messages = True
intents.message_content = True

• Also this one to load the new file (line ~30):
  self.load_extension('core.deforumcog')

• And add this file to the core folder :
  https://github.com/wizz13150/aiyabot/blob/current_bot/core/deforumcog.py

As a first version, this is just to take as 'functionnal'.
You can't use "," (commas) in the prompts field for now, so remove them.

You can now use the /Deforum command to generate videos.

Good command:
/deforum prompts: A 3D render of a cyborg cadence: 3 strength: 0.69 width: 832 height: 1216
Bad command (parsing error):
/deforum prompts: A 3D render, of a, cyborg cadence: 3 strength: 0.69 width: 832 height: 1216

Screen:

Will work again on this later.

Cheers ! 🥂

[wizz13150]

Okay, i found all the deforum_settings names to use as option, for the API, and for posterity.
Need to wisely choose 25 of them.

max_frames was bugged in the file above. Will fix this.

Here is a formatted version :

namespace(
W=768, 
H=512, 
show_info_on_ui=True, 
tiling=False, 
restore_faces=False, 
seed_resize_from_w=0, 
seed_resize_from_h=0, 
seed=-1, 
sampler='Euler a', 
steps=25, 
batch_name='2ed18a536c144c4a8382ef7dc72e2e98', 
seed_behavior='iter', 
seed_iter_N=1, 
use_init=False, 
strength=0.8, 
strength_0_no_init=True, 
init_image=None, 
init_image_box=None, 
use_mask=False, 
use_alpha_as_mask=False, 
mask_file='https://deforum.github.io/a1/M1.jpg', 
invert_mask=False, 
mask_contrast_adjust=1.0, 
mask_brightness_adjust=1.0, 
overlay_mask=True, 
mask_overlay_blur=4, 
fill=0, 
full_res_mask=True, 
full_res_mask_padding=4, 
reroll_blank_frames='ignore', 
reroll_patience=10.0, 
motion_preview_mode=False) 

namespace(
animation_mode='3D', 
max_frames=120, 
border='replicate', 
angle='0: (0)', 
zoom='0: (1.0025+0.002*sin(1.25*3.14*t/30))', 
translation_x='0: (0)', 
translation_y='0: (0)', 
translation_z='0: (1.75)', 
transform_center_x='0: (0.5)', 
transform_center_y='0: (0.5)', 
rotation_3d_x='0: (0)', 
rotation_3d_y='0: (0)', 
rotation_3d_z='0: (0)', 
enable_perspective_flip=False, 
perspective_flip_theta='0: (0)', 
perspective_flip_phi='0: (0)', 
perspective_flip_gamma='0: (0)', 
perspective_flip_fv='0: (53)', 
noise_schedule='0: (0.065)', 
strength_schedule='0: (0.65)', 
contrast_schedule='0: (1.0)', 
cfg_scale_schedule='0: (7)', 
enable_steps_scheduling=False, 
steps_schedule='0: (25)', 
fov_schedule='0: (70)', 
aspect_ratio_schedule='0: (1)', 
aspect_ratio_use_old_formula=False, 
near_schedule='0: (200)', 
far_schedule='0: (10000)', 
seed_schedule='0:(s), 1:(-1), "max_f-2":(-1), "max_f-1":(s)', 
pix2pix_img_cfg_scale_schedule='0:(1.5)', 
enable_subseed_scheduling=False, 
subseed_schedule='0: (1)', 
subseed_strength_schedule='0: (0)', 
enable_sampler_scheduling=False, 
sampler_schedule='0: ("Euler a")', 
use_noise_mask=False, 
mask_schedule='0: ("{video_mask}")', 
noise_mask_schedule='0: ("{video_mask}")', 
enable_checkpoint_scheduling=False, 
checkpoint_schedule='0: ("model1.ckpt"), 100: ("model2.safetensors")', 
enable_clipskip_scheduling=False, 
clipskip_schedule='0: (2)', 
enable_noise_multiplier_scheduling=True, 
noise_multiplier_schedule='0: (1.05)', 
resume_from_timestring=False, 
resume_timestring='20230129210106', 
enable_ddim_eta_scheduling=False, 
ddim_eta_schedule='0: (0)', 
enable_ancestral_eta_scheduling=False, 
ancestral_eta_schedule='0: (1)', 
amount_schedule='0: (0.1)', 
kernel_schedule='0: (5)', 
sigma_schedule='0: (1)', 
threshold_schedule='0: (0)', 
color_coherence='LAB', 
color_coherence_image_path='', 
color_coherence_video_every_N_frames=1, 
color_force_grayscale=False, 
legacy_colormatch=False, 
diffusion_cadence=6, 
optical_flow_cadence='None', 
cadence_flow_factor_schedule='0: (1)', 
optical_flow_redo_generation='None', 
redo_flow_factor_schedule='0: (1)', 
diffusion_redo='0', 
noise_type='perlin', 
perlin_w=8, 
perlin_h=8, 
perlin_octaves=4, 
perlin_persistence=0.5, 
use_depth_warping=True, 
depth_algorithm='Midas-3-Hybrid', 
midas_weight=0.2, 
padding_mode='border', 
sampling_mode='bicubic', 
save_depth_maps=False, 
video_init_path='https://deforum.github.io/a1/V1.mp4', 
extract_nth_frame=1, 
extract_from_frame=0, 
extract_to_frame=-1, 
overwrite_extracted_frames=False, 
use_mask_video=False, 
video_mask_path='https://deforum.github.io/a1/VM1.mp4', 
hybrid_comp_alpha_schedule='0:(0.5)', 
hybrid_comp_mask_blend_alpha_schedule='0:(0.5)', 
hybrid_comp_mask_contrast_schedule='0:(1)', 
hybrid_comp_mask_auto_contrast_cutoff_high_schedule='0:(100)', 
hybrid_comp_mask_auto_contrast_cutoff_low_schedule='0:(0)', 
hybrid_flow_factor_schedule='0:(1)', 
hybrid_generate_inputframes=False, 
hybrid_generate_human_masks='None', 
hybrid_use_first_frame_as_init_image=True, 
hybrid_motion='None', 
hybrid_motion_use_prev_img=False, 
hybrid_flow_consistency=False, 
hybrid_consistency_blur=2, 
hybrid_flow_method='RAFT', 
hybrid_composite='None', 
hybrid_use_init_image=False, 
hybrid_comp_mask_type='None', 
hybrid_comp_mask_inverse=False, 
hybrid_comp_mask_equalize='None', 
hybrid_comp_mask_auto_contrast=False, 
hybrid_comp_save_extra_frames=False) 

namespace(
parseq_manifest='', 
parseq_use_deltas=True) 

namespace(
use_looper=False, 
init_images='{\n    "0": "https://deforum.github.io/a1/Gi1.png",\n    "max_f/4-5": "https://deforum.github.io/a1/Gi2.png",\n    "max_f/2-10": "https://deforum.github.io/a1/Gi3.png",\n    "3*max_f/4-15": "https://deforum.github.io/a1/Gi4.jpg",\n    "max_f-20": "https://deforum.github.io/a1/Gi1.png"\n}', 
image_strength_schedule='0:(0.75)', 
blendFactorMax='0:(0.35)', 
blendFactorSlope='0:(0.25)', 
tweening_frames_schedule='0:(20)', 
color_correction_factor='0:(0.075)')

[solareon]

What extension is this based on for video generation?

[wizz13150]

Deforum (available in the extension tab):
https://github.com/deforum-art/sd-webui-deforum

Added some options.

Some others important options (for me) are not there, unfortunately.. like 'soundtrack_path', or 'frames_interpolation'...
Able to use a Parseq Manifest, this is at least a great thing !

Here is the updated deforumcog.py :
(commits: wizz13150@743480f & wizz13150@803cb2a)

Current file (Still a lot to do): EDIT: Bugged as well. Options seems tricky.
https://github.com/wizz13150/aiyabot/blob/current_bot/core/deforumcog.py

Example :

Here is an example of what we can (I love to) do using Deforum + a Parseq Manifest (also using the missing 'soundtrack_path' and 'frames_interpolation' param...) to sync with the music :
https://civitai.com/images/2052050?period=AllTime&periodMode=published&sort=Newest&view=categories&username=_wizz_&withTags=true
(Turn the sound on. Bottom right corner)

Parseq visualization for the video link above :

Parseq link :
https://sd-parseq.web.app/?docId=doc-65c7db22-45b9-4264-b625-e418b0b3cc20

[wizz13150]

Okay, this is not the good approach to make good videos.

I need to rework this from scratch, using the full API.
Everything is in the full API, will be way better.

Source :
https://github.com/deforum-art/sd-webui-deforum/tree/automatic1111-webui/tests

[wizz13150]

Okay, here is a functionnal version of deforumcog.py using --deforum-api instead of --deforum-simple-api :

https://github.com/wizz13150/aiyabot/blob/current_bot/core/deforumcog.py

This version need to submit a full settings file to Deforum to start.

So we need a default_settings.json file, in the main 'aiyabot' folder :

https://github.com/wizz13150/aiyabot/blob/current_bot/default_settings.json
(my current version of default_settings.json, this should be replaced by a real, or custom, Deforum default settings file)
This file is also the list of the available settings for the bot, as options.

A deforum.json file will also be created.

Aiya.py don't need intents.messages = True & intents.message_content = True anymore :

https://github.com/wizz13150/aiyabot/blob/current_bot/aiya.py

• Console Screen:

• Discord Command Screen:

• Discord message:

• Video output (x3 frames interpolation, way longer but better result):

20230918130120.mp4

• Bot Command:

/deforum prompts: 0:(Photo of an unknown alien lifeform with strange pupils and big eyes looking into the camera while sitting on a couch, in his bedroom. In the style of Alyssa Monks, award winning photography, 50mm, blurred background, sharp details, 8K high definition, insanely detailed, elegant, art by David la chapelle and Wayne Barlowe and Alex <lora:Wowifier:0.85> --neg deformed, distorted, disfigured, poorly drawn, bad anatomy, wrong anatomy, extra limb, missing limb, floating limbs, mutated hands and fingers, disconnected limbs, mutation, mutated, ugly, disgusting, blurry, amputation, tattoo, watermark, text) cadence: 3 translation_z: 0:(1) width: 1216 height: 832 fps: 15 max_frames: 120 antiblur_amount_schedule: 0.3 frame_interpolation_engine: FILM

Still a lot to do.
NB: Loras like the Wowifier helps a lot with Deforum, to keep the quality loss to the minimum.
Don't tell anyone the secret, shh.

To be continued...

break

[wizz13150]

Okay, i continue to edit my bot on this branch :

https://github.com/wizz13150/aiyabot/tree/DiscordFullBot

It's way too messy and added too much things to properly separate them in PRs at this point...

I'm still building /Deforum. It's working, but not definitive until i say so.
Spent some time on fixing bugs, lol. I can continue the deforumcog.py now (:
Already 476 lines of pain at this point, and i had to edit some other files too.

To be continued...

Cheers ! 🥂