AI Can Now Animate Any Monster. Opus 5.5 Built the Tool
Can It Code?
0:00 / 0:00
AI Can Now Animate Any Monster. Opus 5.5 Built the Tool
18 693 просмотра · 8 ч назад
Can It Code?
14,9 тыс. подписчиков
18 693 просмотра · 8 ч назад
My Patronite: / canitcode
A troll, a wyvern, a centaur with a sword. For people there is motion capture and Mixamo. For monsters there is nothing. So Claude Opus 5.5 and I built our own way: Seedance films our monster moving, and our tool finds every joint in that video and puts the move on the monster's own skeleton in Blender, ready for the game.
In this video:
Where it started: a deer from our game, one camera, and legs that kept crossing
Seedance with two cameras in one picture
Every tool we tried and why it broke: point tracking from Google DeepMind, SAM 2, Depth Anything, painted legs, animal pose (AP-10K), MoCapAnything
The rethink: one problem to solve, every joint in every frame
A skeleton for any 3D model with UniRig
DINO from Meta, taught on our troll in about 30 minutes, reads every joint
The exam every change to the tool has to pass
Depth, take two
From A to Z: model, skeleton, Seedance, joints, Blender, game
Before: 4 days with Opus checking every frame. Now: minutes
What it still can't do
Would you like to try it? Tell me in the comments.
My last video, NVIDIA Kimodo: [LINK]
Patreon: / canitcode
Chapters
00:00 Intro
00:42 Every way we tried
01:20 Where it started: the deer
02:27 Seedance with two cameras
03:22 Shopping: point tracking
03:48 SAM 2 outlines
04:06 Depth Anything
04:28 Painted legs
05:00 Animal pose
05:14 MoCapAnything
05:29 The rethink
05:54 A skeleton with UniRig
06:27 DINO reads every joint
07:40 The exam
08:32 Depth, take two
09:08 From A to Z
09:32 4 days vs minutes
09:48 The limits
Sources
AI video: Seedance 2.5 (ByteDance Seed). 3D models: Tripo (tripo3d.ai).
Skeleton: UniRig (github.com/VAST-AI-Research/UniRig). Joint reader: DINOv2 from Meta (github.com/facebookresearch/dinov2).
Tools we tried: BootsTAPIR point tracking from Google DeepMind (github.com/google-deepmind/tapnet), SAM 2 from Meta (github.com/facebookresearch/sam2), Depth Anything V2 (github.com/DepthAnything/Depth-Anything-V2), ViTPose++ with AP-10K, MoCapAnything.
In the intro: NVIDIA Kimodo (github.com/nv-tlabs/kimodo) and Mixamo.
Built in Blender and Godot 4.7. Code and scripts: Claude Opus 5.5. Music: Nocturnal Pads.