Motion Brush Example

The motion_brush parameter is used for tasks that involve generating videos from images. It has specific requirements:

This only works in kling v1.0.

The task must be an image-to-video generation task.

The image_tail_url must not be present.

For PPU users, the plan must be Premium.

For BYOA users, the plan must be at least Developer.

There are no additional charges other than plain image-to-video task.

Structure of the `motion_brush`

        "motion_brush": {
            "mask_url": "https://example.com/example.png",
            "static_masks": [
                {"points": []}
            ],
            "dynamic_masks": [
                {"points": [
                        {"x": 510, "y": 309},
                        {"x": 512.8890686035156, "y": 305.3799133300781},
                        {"x": 515.9701843261719, "y": 301.5237731933594},
                        {"x": 518.5692138671875, "y": 297.9819030761719},
                        ...
                        {"x": 587.5043334960938, "y": 107.7113037109375},
                        {"x": 588.9964904785156, "y": 103.77008056640625},
                        {"x": 590.7432556152344, "y": 99.7882080078125},
                        {"x": 592.4257507324219, "y": 95.93536376953125},
                        {"x": 594.3355407714844, "y": 92.24978637695312},
                        {"x": 596.1476440429688, "y": 88.53536987304688},
                        {"x": 598.1287841796875, "y": 85.0318603515625},
                        {"x": 600.0324401855469, "y": 81.4693603515625},
                        {"x": 602, "y": 78}
                ]}
            ]
        }

mask_url: This is the URL of the mask image for the motion_brush. Use a transparent PNG image with the same dimensions as the image_url.

static_masks: This defines the moving direction of the image that do not require motion. You can have at most one static mask.

dynamic_masks: This defines the moving directions of corresponding image areas. You can have up to six dynamic masks (consistent with web interface limits). At least one of the two mask types must be present.

points: This array contains the motion path points for the specified areas. In static_masks, this parameter can be left empty. For dynamic_masks, at least two points are required, with a recommendation of ten or more. The origin (0,0) is the top-left corner of the image, with the x-axis extending to the right and the y-axis extending downward.

Color Indexing for Masks

The index of each mask corresponds to specific fixed color regions in the mask image. The relationship is as follows:

{
  "1": {"r": 114, "g": 229, "b": 40},
  "2": {"r": 171, "g": 105, "b": 255},
  "3": {"r": 0, "g": 170, "b": 255},
  "4": {"r": 240, "g": 38, "b": 173},
  "5": {"r": 255, "g": 225, "b": 29},
  "6": {"r": 255, "g": 34, "b": 0},
  "0": {"r": 0, "g": 0, "b": 0}
}

"0" represents static_masks.

"1" represents the first mask in dynamic_masks, and so on.

Users must ensure that the colors in the mask image match the specified RGB values exactly. The colored areas need to be fully opaque. The uncolored areas need to be fully transparent.

Full Example of Image-to-video Task with Motion Brush

{
    "model": "kling",
    "task_type": "video_generation",
    "input": {
        "prompt": "the boat fly to the sky",
        "negative_prompt": "",
        "cfg_scale": 0.5,
        "duration": 5,
        "image_url": "https://i.ibb.co/MNX5JVP/boat.jpg",
        "image_tail_url": "",
        "mode": "std",
        "version": "1.0",
        "motion_brush": {
            "mask_url": "https://i.ibb.co/WB6YqtH/mask.png",
            "static_masks": [
                {"points": []}
            ],
            "dynamic_masks": [
                {"points": [
                        {"x": 510, "y": 309},
                        {"x": 512.8890686035156, "y": 305.3799133300781},
                        {"x": 515.9701843261719, "y": 301.5237731933594},
                        {"x": 518.5692138671875, "y": 297.9819030761719},
                        {"x": 521.7066345214844, "y": 293.7148132324219},
                        {"x": 523.9982299804688, "y": 290.197509765625},
                        {"x": 527.0696716308594, "y": 285.4500427246094},
                        {"x": 529.1102600097656, "y": 281.9802551269531},
                        {"x": 532.0352783203125, "y": 276.8209533691406},
                        {"x": 533.900146484375, "y": 273.4161376953125},
                        {"x": 536.6403503417969, "y": 267.9842834472656},
                        {"x": 538.3750915527344, "y": 264.6147766113281},
                        {"x": 540.9106750488281, "y": 259.0498352050781},
                        {"x": 542.5351257324219, "y": 255.64447021484375},
                        {"x": 544.8347473144531, "y": 250.0723876953125},
                        {"x": 546.3829345703125, "y": 246.52072143554688},
                        {"x": 548.3993225097656, "y": 241.0670166015625},
                        {"x": 549.939453125, "y": 237.21255493164062},
                        {"x": 551.6444396972656, "y": 231.99563598632812},
                        {"x": 553.2495422363281, "y": 227.67388916015625},
                        {"x": 554.6771850585938, "y": 222.77197265625},
                        {"x": 556.3735656738281, "y": 217.91738891601562},
                        {"x": 557.6281127929688, "y": 213.328857421875},
                        {"x": 559.3721313476562, "y": 208.071533203125},
                        {"x": 560.5928649902344, "y": 203.70278930664062},
                        {"x": 562.2971496582031, "y": 198.332763671875},
                        {"x": 563.603515625, "y": 194.04888916015625},
                        {"x": 565.1891479492188, "y": 188.83755493164062},
                        {"x": 566.6304321289062, "y": 184.55484008789062},
                        {"x": 568.065673828125, "y": 179.58285522460938},
                        {"x": 569.5950622558594, "y": 175.3140869140625},
                        {"x": 570.8982238769531, "y": 170.45913696289062},
                        {"x": 572.3914489746094, "y": 166.26303100585938},
                        {"x": 573.6038818359375, "y": 161.33819580078125},
                        {"x": 574.9257202148438, "y": 157.24008178710938},
                        {"x": 576.082275390625, "y": 152.14614868164062},
                        {"x": 577.1692810058594, "y": 148.11874389648438},
                        {"x": 578.2933349609375, "y": 142.902099609375},
                        {"x": 579.2005920410156, "y": 138.90301513671875},
                        {"x": 580.3246154785156, "y": 133.71231079101562},
                        {"x": 581.2021484375, "y": 129.71713256835938},
                        {"x": 582.3867797851562, "y": 124.717041015625},
                        {"x": 583.3997192382812, "y": 120.72256469726562},
                        {"x": 584.7251586914062, "y": 116.02908325195312},
                        {"x": 585.9737854003906, "y": 112.0469970703125},
                        {"x": 587.5043334960938, "y": 107.7113037109375},
                        {"x": 588.9964904785156, "y": 103.77008056640625},
                        {"x": 590.7432556152344, "y": 99.7882080078125},
                        {"x": 592.4257507324219, "y": 95.93536376953125},
                        {"x": 594.3355407714844, "y": 92.24978637695312},
                        {"x": 596.1476440429688, "y": 88.53536987304688},
                        {"x": 598.1287841796875, "y": 85.0318603515625},
                        {"x": 600.0324401855469, "y": 81.4693603515625},
                        {"x": 602, "y": 78}
                ]}
            ]
        }
    }
}

Motion Brush Example

Structure of the motion_brush#

Color Indexing for Masks#

Full Example of Image-to-video Task with Motion Brush#

Structure of the `motion_brush`

Color Indexing for Masks

Full Example of Image-to-video Task with Motion Brush