Stable Diffusion openpose with wood figure experiment

The original coversation comes from Stable Diffusion Thailand

I would like to play openpose controlnet feature of stable diffusion.

Original figure from online shop

Then convert to openpose control figure.

from controlnet_aux import OpenposeDetector
...
image = openpose(image)
image.save("images/openpose_control_image.png")

openpose image.

pytorch is not fully support mps on m2 yet then use cpu version.

from PIL import Image
from diffusers import StableDiffusionControlNetPipeline, ControlNetModel, UniPCMultistepScheduler
import torch
from controlnet_aux import OpenposeDetector
from diffusers.utils import load_image

# device: str = "mps" if torch.backends.mps.is_available() else "cpu"
device: str = "cpu"
openpose = OpenposeDetector.from_pretrained('lllyasviel/ControlNet')

image = load_image("sources/wood_figure.png")

image = openpose(image)
image.save("images/openpose_control_image.png")

controlnet = ControlNetModel.from_pretrained(
    "lllyasviel/control_v11p_sd15_openpose"
).to(device)

pipe = StableDiffusionControlNetPipeline.from_pretrained(
    "runwayml/stable-diffusion-v1-5", controlnet=controlnet,
    requires_safety_checker=False, safety_checker=None
).to(device)

pipe.scheduler = UniPCMultistepScheduler.from_config(pipe.scheduler.config)

image = pipe("chef in the kitchen full body. face looking at the camera",
             image, num_inference_steps=20, negative_prompt="malformed",).images[0]

image.save('images/chef_pose_out.png')

Failed output

The root causes come from many reasons for example

1. wood figure has no background area
2. negative prompt is not cover many words

Let’s try adding more area to the wood man.

A picture of wood man and huge area surrounding

Without adding any negative_prompt get this openpose picture.

Control image of wood man with huge surrounding area

Finally.

Final output.

Conclusion
When work with controlnet remember to give the area to the A.I. since it will actually use exact location of your given input to generate result. And most of the dataset always requires large area to describe scene.

♠♠♠♠♠♠♠♠♠♠♠♠♠♠♠♠♠♠♠♠♠♠♠♠♠♠♠♠♠♠♠♠♠♠♠♠♠♠♠♠♠♠♠♠♠♠♠♠♠♠♠♠♠♠♠♠️️