mcjKinectKit for Kinect V1 and in June 2018 Kinect V2 - script and exe for DS 3, 4

SimonWM

I have a question? Do you ever get live recording into DAZ Studio. From your instructions I understood "Write to client" in the server would be used to send a live capture into DAZ Studio viewport via the stick figure.

I guess what confuses me the most is that the moment I hit record in the client the stick figure in DAZ Studio viewport stops moving
It still moves in the server window but it stops moving in the DAZ Studio viewport.

mCasual

SimonWM said:

I have a question? Do you ever get live recording into DAZ Studio. From your instructions I understood "Write to client" in the server would be used to send a live capture into DAZ Studio viewport via the stick figure.

I guess what confuses me the most is that the moment I hit record in the client the stick figure in DAZ Studio viewport stops moving
It still moves in the server window but it stops moving in the DAZ Studio viewport.

yes the stick figure is not ( visibly ) updated during capture
well it is updated in the timeline's memory
but the current displayed frame is not changing
so the image doesn't change

in an upcoming version there may be an option to keep it moving

i didn't want to drop capture frames due to intensive CPU use by Daz Studio redrawing the entire image

the Client just waits for poses to arrive on the communication link and stores them in memory as fast as it can
so the communication link is ready to receive more poses from the server
so the server is ready to receive more poses from the kinect drivers

in my initial tests there even was a live preview directly on a figure

so those options will be in the next kinect kit update

but before that i'll roll out the kinect face script

and before that ( according to plan ) i have to complete 3 or 4 other scripts
like the midi music app, the pose tools, the poke-through-fix script

mCasual

TEST 001 Part 2

the setup

at that distance the actress is fully in the kinect's field of view

the kinect can see a large chunk of the floor, so it doesnt have a hard time locating it

the kinect is level, it's motorized base has 0 degrees of tilt

the kinect is around 130cm / 50 inches above the floor

mCasual

TEST 001 Part 3

so in part 1 i did a motion capture with the server in standalone mode

then i loaded a scene containing a Genesis (1?) figure

i delete any left-over stickman from the scene

i select Azura (Genesis/Aiko5)

i start mcjKinectClient

it builds a stickman

i click the Client's load mocap button

i specify i want to load 20150519133454 which was recorded in Part 1

Azura stands above coordinates [0,0,0] of the Daz World

we can see the stickman is facing her

and the stickman stands about 8ft away from her

because i was about 8 ft away from the kinect, facing the kinect

i'm not totally sure why my programming leaves stickmen at this altitude (sunk in the Y=0 floor )

if you select the stickman's root node, you can see there's no keyframes at all on that node

so we can move him ( the whole animation ! ) to floor level, with just 1 keyframe

looks like the stickman is only 4 ft tall! i wonder why and if that was intentional

oops looks like i only recorded 70 frames

i misinterpreted the server's delay-progress bar

the delay's countdown starts when the kinect reports the presence of an actor
during the delay ountdown, the green progress bar, shortens ( like an hourglass of sorts )
and when the delay is elapsed, the yellow LED goes RED
and the capture starts

so i create a new motion capture right now

mCasual

TEST 001 Part 4

now i don't really understand why but this time the stickman appeared at a good altitude

( when the pre-capture-delay ended the Kinect had already been tracking me )

i position and orient Azura's root node where i want her to start Fig 3

i select Azura

i start the mcjKinectClient script

i go in the [ Stickman to figure ] Tab

as you can see (Fig 4) i wont take the collar feet and hands animations which are usually not very good

Fig 5 - i do use the "fill missing key-frames button" because i noticed the server did freeze a few times during capture

( for perfect capture i'd close Chrome and maybe even pause my antivirus )

Fig 6 - surprise surprise Azura jumped over there and we'll figure why

but other than that it's a success!!!

i mean she does the motions and we'll see it on youtube later !!

So, as we saw earlier, the captured motion was - an actor standing about 8 ft in front of the Kinect, facing the Kinect

so Azura is standing 8 ft in front of where she started .... i think that's the (untested) appearances of it all

mCasual

TEST 001 Part 5 ( final )

the first 30 frames at 15 fps

next which won't be shown here i'll use mostly mcjAutoLimb2014 and mcjKeepOrient to improve the feet

since our work so far is moderately precious

we save the scene as moderatelyprecious.duf in case we crash

Fig 8 - i used mcjDecimate to reduce the number of keyframes to 10 fps
and i used mcjCycleFilter Synth/Smooth to smoothen the Shin rotations
note that mcjCycleFilter can also be used to shift the rotation of an entire animation range ( using Synth/Current )
so if the abdomen is BENDing between -5 and -15 degrees due to a bad kinect camera tilt
you could bring it back to -10 to +10 degrees using mcjCycleFilter

important tip about mcjAutoLimb 2014 - set Genesis shin limits to be 0 to 145 degrees instead of -12 to 145 degrees
this way mcjAutoLimb wont even try those bizarre shin angles which completely messes the work
also set the limits-Off on the Thighs rotations

fig 9 fig 10, i anchored the feet using mcjAutoLimb, the anchors were chosen because they were the average positions
mcjKeepOrient was used to stabilize the feet orientations ( because the legs are constantly moving ! )

for the record, Aiko5. leg length, foot to thigh is 92.7cm

script to measure a figure's leg length:

var skel = Scene.getPrimarySelection();
if( skel.inherits( "DzBone" ) )
 skel = skel.getSkeleton();
if( skel.inherits( "DzSkeleton" ) )
{
 var lShin = skel.findBone( "lShin" );
 var ctl = lShin.getXRotControl();
 var old = ctl.getValue();
 ctl.setValue( 0 );
 processEvents();
 var lThigh   = skel.findBone( "lThigh" );
 var lFoot   = skel.findBone( "lFoot" );
 var len = lFoot.getWSPos().subtract( lThigh.getWSPos() ).length();
 ctl.setValue( old );
 MessageBox.information( "left leg length = " + len, "Info", "&OK;" );
}

so right now i'll write a script or use existing ones to correct the hip position so that autolimb can maintain the feet anchors

mCasual

which means i'll soon publish a script ( mcjAutoLimbHelper )

which helps mcjAutoLimb reach its goals

by moving the figure's hip just the very precise minimum distance

so that the feet can reach their anchors ( targets )

and it will be usable for arms-hands

---

if i then smooth the shin animation curves and re-run the mcjAutoLimbHelper script
i'll effectively smooth the hip position curves

mCasual

YOUTUBE VIDEO !!!! yay

https://www.youtube.com/watch?v=UnjaML5B1Yg

well i was at the encoding stage when i noticed she swings her arms across her chest but it doesn't hurt people in that dimension

note i use the script mcjSkinLikeThis to give Aiko5 a skin that looks kin'of good in hardware render mode

SimonWM

Casual said:

SimonWM said:

I have a question? Do you ever get live recording into DAZ Studio. From your instructions I understood "Write to client" in the server would be used to send a live capture into DAZ Studio viewport via the stick figure.

I guess what confuses me the most is that the moment I hit record in the client the stick figure in DAZ Studio viewport stops moving
It still moves in the server window but it stops moving in the DAZ Studio viewport.

yes the stick figure is not ( visibly ) updated during capture
well it is updated in the timeline's memory
but the current displayed frame is not changing
so the image doesn't change

in an upcoming version there may be an option to keep it moving

i didn't want to drop capture frames due to intensive CPU use by Daz Studio redrawing the entire image

the Client just waits for poses to arrive on the communication link and stores them in memory as fast as it can
so the communication link is ready to receive more poses from the server
so the server is ready to receive more poses from the kinect drivers

in my initial tests there even was a live preview directly on a figure

so those options will be in the next kinect kit update

So once you hit record in the client everything happens in memory invisible to the user because there is absolutely no feedback in DAZ Studio.

I think once you have options to keep the skeleton moving once you hit record in the client it will be more intuitive. Right now one wonders why does the stick figure stops moving. It would be great to have an option to see the actual figure moving in realtime for people with a powerful system.

mCasual

SimonWM said:

....

i'll see
maybe re-enabling the live mode for stickman will take me only 10 minutes
if so i'll add a checkbox

live mode with figures is more complicated ( and will almost surely make you drop frames during capture )
so that option may come when i resume work on the kinect kit in June or July
or it will be there today

but the best captures will always be done with the server running in standalone mode

i may even add a write-to-RAM mode to the server ... someday :)

also someday i may make the Kinect plugin for DS3 and DS4 Win-PC 32/64

SimonWM

Casual said:

SimonWM said:

....

i'll see
maybe re-enabling the live mode for stickman will take me only 10 minutes
if so i'll add a checkbox

live mode with figures is more complicated ( and will almost surely make you drop frames during capture )
so that option may come when i resume work on the kinect kit in June or July
or it will be there today

but the best captures will always be done with the server running in standalone mode

i may even add a write-to-RAM mode to the server ... someday :)

also someday i may make the Kinect plugin for DS3 and DS4 Win-PC 32/64

An option for live mode with figures right now would be awesome! You can always turn it off if the system is not strong enough to use it.

mCasual

later today, a checkbox in the mcjKinectClient script
to enable the in-Daz-Studio-scene-stickman animation while recording

mCasual

this one was recorded with the DS-stickman-animation-while-recording option

mCasual

so this evening i'll post a version of the client

with optional Stickman-animation-during-record session

and optional Figure -animation-during-record session

i added reset buttons for the figure and stickman animations

there will be a setting somewhere to decrease the load that represents the Figure-Animation-during-record task
instead of attempting to pose-and-redraw the figure at each frame given to us by the Kinect, it will skip a few frames
( between 1 and 10 i guess )

Fig 2, if someday you find yourself wondering why your actor is all shaky jumpy ...

mCasual

that's captured in Figure-During-Record mode, not sure why it seems to be in 'seated ' mode
i vaguely remember that the client's logic is too quick in its judgement about the seated mode
in an upcoming version the Sever will tell the client if the seated mode is used
instead of having the client guess it

dont forget, after using the Figure-During-Record mode
you need to
1 - clear the figure animationusing the [ Reset Figure ] button
2 - use the [ Fill Missing Keyframes ] button
3 - make sure the transfer range start/enf is what you want ( in the Transfer Tab )
4 - Use the [ Transfer This Range] button

mCasual

Well since it's another very Beta version i wont make a big announcement but

i post it at the bottom of the mcjKinectKit page under the name

mcjKinectClient_VeryBetaV002.dsa

it's the DS3/DS4 version but was only tested on DS3 with mixed results

SimonWM

Wow, I tested it and it seems to be getting so close to something easy to use and understand but it is still confusing. When I hit Transfer this range I see the figure moving and previewing the animation. But when I close the server I don't have any animation in the timeline.

So how it went:

I was expecting that the "Figure During Record" would mean live capture of data into the Timeline while previewing with the Figure (Genesis etc.) and "Stickman During Record" would mean live capture with the stick figure for less powerful systems.

None of my assumptions above were correct. So I capture with the "Figure During Record" and the stick figure moves instead of Genesis as I was wrongly expecting. Then I followed the instructions:

1 - clear the figure animationusing the [ Reset Figure ] button
2 - use the [ Fill Missing Keyframes ] button
3 - make sure the transfer range start/enf is what you want ( in the Transfer Tab )
4 - Use the [ Transfer This Range] button

I see Genesis moving now which is EXCITING!!! I close the server, close the client but now when I check the timeline no animation has been recorded. And I'm confused again, back to reading the instructions.

mCasual

SimonWM said:

... instructions.

normal operation is

0 - go to timeline frame 0

1 - select the figure

2 - start the client

3 - the client builds a stickman if one is not present in the scene

4 - if one is present and contains animation from a previous session,
it's better to clear the stickman animation using the "reset stickman' button

5 if the figure contains animation it's better to clear it using the "reset figure ' button

6 - click the client's 'start server button

7 - click the client's 'record' button

8 - in the server make sure the "write to client" checkbox is checked

9 - in the server check-mark the "delayed button

10 - in the server click the Start button

11 - go on-stage, wait for the green downcount bar to shrink and the server's LED to turn red, move

12 - click the server's Stop button

13 - close the server ( better to close it this way, else the kinect's red/infrared light may stay on - something i may fix someday

14 - click the Client's Stop button ( this is important )

15 - if you were using the preview-figure-while-recording option, it's better to now delete the Figure animation using the "reset figure" button, the stickman animation is precious, and should not be deleted now !

16 - use the "fill missing keyframes" button, the client will finalize the stickman animation by making it 1 keyframe per frame

17 - use the "Transfer this range" button, the script will transfer the animation from the stickman to the figure ( make sure the Transfer Time range covers the section of the stickman animation of interest

SimonWM

Thank you, with this instructions I was able to capture fine. Some problems though, the figure sometimes drop through the floor, not all times.

The toes seem to be kept pointing down. Need to adjust the feet/bend option. And this is more my mistake but as you see in my video my timing predictions were too long, I had 2000 frames but I used much less.

https://www.youtube.com/watch?v=mnyz5k0kTCQ

Perhaps a frame counter on the server or client would help with not having to guess the capture length. Ideally it would be to have realtime figure capture within DAZ Studio viewport with realtime Timeline feedback. But I'm probably dreaming. Still this is an awesome script as it is, and this new beta is twice more awesome than the previous one!

mCasual

SimonWM said:

Thank you, with this instructions I was able to capture fine. Some problems though, the figure sometimes drop through the floor, not all times.

The toes seem to be kept pointing down. Need to adjust the feet/bend option. And this is more my mistake but as you see in my video my timing predictions were too long, I had 2000 frames but I used much less.

https://www.youtube.com/watch?v=mnyz5k0kTCQ

Perhaps a frame counter on the server or client would help with not having to guess the capture length. Ideally it would be to have realtime figure capture within DAZ Studio viewport with realtime Timeline feedback. But I'm probably dreaming. Still this is an awesome script as it is, and this new beta is twice more awesome than the previous one!

actually in your video the feet look better than mine, i usually un-checkmark- the feet and hands (sometimes the collars and buttocks ) so they dont get transfered to the figure

but dont forget to read about the other options in the Transfer tab, because they are related to the floor issues and the feet issues

note that during the capture, i always have the Main Client tab open, not the transfer tab
so i can see the "console" messages

using Daz Studio 3, it takes under a second to build the stickman, with DS4.x it takes many seconds !

jaebea

I am following this thread with great interest! Would like to know if your plugin will also work with Kinect for Windows?

mCasual

jaebea said:

I am following this thread with great interest! Would like to know if your plugin will also work with Kinect for Windows?

if it's the first version of the Kinect For Windows ( there's now a version 2 that, i think, it like the XBOX-One Kinect )
then i think it will work ... but i'm not 100% sure, i did all my tests with an XBOX-360 Kinect
i was careful to buy one that was sold with the "Kinect adventures" pack, because i read somewhere that they were compatible with Brekel
then i bought an $8 USB/Power adapter from Amazon

Tugpsx

Thanks for update.

mCasual

Back in February 2015 i experimented with facial mocap using the kinect for xbox 360 connected to a PC using a $30 adapter

and i'm sure 2 or 3 persons in this forum thread will be interested to know that i'm reactivating this project 

i lost the source code of the "server" app and "client" script in recent hard disk(s) failure(s) but it wont be too hard to re-write it

in fact you can already see i have the head position and orientation covered

http://mcasual.deviantart.com/art/the-return-of-the-kinect-facial-mocap-for-Daz-Stud-580385623

the microsoft kinect driver reports on 6 facial features, which they call Animation Units

to convert this into Daz Studio/Poser face-morph tweakings, i have to write a special mapping system

the result should make it usable for Aiko3 .... but not only her ! for genesis too, even for cats and dogs

 

Here you have Videos ! from Feb. 2015 done using the old Client/Server 

https://www.youtube.com/user/mcasual/search?query=kinect+face

 

SimonWM

mCasual said:

Back in February 2015 i experimented with facial mocap using the kinect for xbox 360 connected to a PC using a $30 adapter

and i'm sure 2 or 3 persons in this forum thread will be interested to know that i'm reactivating this project 

i lost the source code of the "server" app and "client" script in recent hard disk(s) failure(s) but it wont be too hard to re-write it

in fact you can already see i have the head position and orientation covered

http://mcasual.deviantart.com/art/the-return-of-the-kinect-facial-mocap-for-Daz-Stud-580385623

the microsoft kinect driver reports on 6 facial features, which they call Animation Units

to convert this into Daz Studio/Poser face-morph tweakings, i have to write a special mapping system

the result should make it usable for Aiko3 .... but not only her ! for genesis too, even for cats and dogs

 

Here you have Videos ! from Feb. 2015 done using the old Client/Server 

https://www.youtube.com/user/mcasual/search?query=kinect+face

 

Always a genius! Looking forward to this Casual, will it work only on Genesis 1 or Genesis 2 & 3 too?

mCasual

 yes it should work for all figures, the kinect sends us 6 face parameters, 

then there will be a table to map these on the figure

either i will supply a basic mapping table for genesis 1,2,3 and A3,V3,V4 or i'll let you create it

it can work on any figure, even a cat ( which i did in my St-Patrick day Green Cat video )

for now the mapping table looks like this

for this animation

var converters = [

[ AU1_Jaw_Lowerer, "OpenMouth", 2 ], //0

[ AU5_Outer_Brow_Raiser, "BrowUpR", 3 ], //1

[ AU5_Outer_Brow_Raiser, "BrowUpL", 3 ], //2

[ AU4_Lip_Corner_Depressor, "PBMCC_07", -1 ], //3 mcjA3Smile custom smile morph for Aiko3

[ AU2_Lip_Stretcher, "MouthWide", 1 ], //4

[ AU3_Brow_Lowerer, "BrowAngleUp", 1 ],

];

 

SimonWM said:

mCasual said:

 

Always a genius! Looking forward to this Casual, will it work only on Genesis 1 or Genesis 2 & 3 too?

 

Tugpsx

Great to see this back on track. 

mCasual

if all goes well, before the weekend ends i'll post a free Kinect V2 bodymocap-to-DazStudio for-PC-only exe and and a Daz Script script
just like the one i did for the Kinect 360 ....

 

in this image you see the kinect-360-stickman which has 20 bones

the only difference with the kinect one  (V2) seems to be 4 more bones

maybe maybe, i'll make a live poser-figure mode, using a figure like the SecondLife avatar which is a light CPU load to animate or Victoria3/Michael3/Aiko3

SimonWM

V2 meaning it will only work with the new cameras or this is a version 2 of your kinevt script?

mCasual

SimonWM said:

V2 meaning it will only work with the new cameras or this is a version 2 of your kinevt script?

it will be the same thing as the first version - which supports the XBOX 360 kinect

so 1 executable (.exe) app that talks to the kinect and obtains poses and "prints" them out

lets call it the kinect server

and 1 daz studio script that receives those poses and records them while animating a stickman

lets call it the kinect client

then in a second operation, after capture is done, converts it into poses applied to a figure

i dont remember if it was fully compatible with genesis figures

---------------------

initially at least, the daz script for V2 wont be compatible with the V1 server

maybe later