mcjKinectKit for Kinect V1 and in June 2018 Kinect V2 - script and exe for DS 3, 4

12467

Comments

  • 3dOutlaw3dOutlaw Posts: 2,471
    edited December 1969

    Yea, I turned off limits and then it was pretty gross, lol! Twisted really bad. Its hard to tell orientation when they are that twisted. Anyway, at least now I know how to use the interface. It took a few tries to realize I had to click the figure in the scene tab and not the stick figure prior to transfer.

  • mCasualmCasual Posts: 4,607
    edited February 2015

    so today i will make the figure posing more intelligent
    for example, forbid foreArm side-side rotation
    and probably the shin side-side rotation

    maybe i'll borrow some of my mcjAutoLimb code for this

    fig 2 that's more like it !

    there's still minimal side-side rotations on the arms but i'll leave it like that for now
    at least the arm is not over-twisted anymore

    ----

    issue which will be fixed: figure is positioned 154 cm too high

    ----

    figure 3 for the benefit of anyone wanting to write their own Client
    the format of the mocap .txt files

    barbarbar.jpg
    1280 x 720 - 234K
    slowit.gif
    200 x 300 - 618K
    format.jpg
    1280 x 720 - 714K
    ahHA_(2).jpg
    1280 x 720 - 277K
    todayz.jpg
    1280 x 720 - 325K
    Post edited by mCasual on
  • mCasualmCasual Posts: 4,607
    edited February 2015

    progress report

    done

    - (hopefully) fixed the over-twisted left arm issue

    - progress bar during long operations with the possibility of interrupting it

    - warns user if no figure is selected

    todo

    - bend-offset on the feet and neck

    - warn user that locked bones or limits may interfere with poses

    - test with
    - Norm toony ( the zero-posed arms differ from the norm ),
    - TheGirl ( zero-posed legs differ from the norm )
    - Victoria 4
    - Genesis

    fig1one.jpg
    800 x 1000 - 238K
    Post edited by mCasual on
  • 3dOutlaw3dOutlaw Posts: 2,471
    edited December 1969

    Is the download updated with the changes or not yet? I was going to grab it for a test.

  • mCasualmCasual Posts: 4,607
    edited February 2015

    3doutlaw said:
    Is the download updated with the changes or not yet? I was going to grab it for a test.

    here's

    mcjKinectKitSecretPreBetaVersion_v2

    i still have to make tests on different figures and on DS1,2,3,4

    1 - the collar-bone animations are now excluded from the transfer to the figure
    2 - the server now writes mocap archive files(txt) in its own folder instead of in DazStudio's folder
    3 - the “Seated” mocaps are still not really supported
    4 - minor DS3 compatibility enhancement
    5 - the left arm over-twisting issue is possibly fixed
    6 - long operations now display a progress bar with interrupt-ability

    https://sites.google.com/site/mcasualsdazscripts4/mcjkinectkit001

    Post edited by mCasual on
  • mCasualmCasual Posts: 4,607
    edited February 2015

    little problem here Aiko's root bone is about at the same location as Aiko's hip bone

    so that will be fixed somehow in the official 001 release

    in the meantime, you could move the root bone approximately 144 cm down along the Y Axis before transfering the pose(s)

    ------

    Fig 2, Fig3 - the official release will include an "offset" to tweak the figure's hip position

    Fig 4 - Neck Bend adjustment because the kinect's skeleton's neck is always bent forth

    dobz.jpg
    1600 x 800 - 378K
    hiphip.jpg
    1280 x 720 - 296K
    offsets.jpg
    978 x 662 - 182K
    lilpblm.jpg
    800 x 1000 - 233K
    Post edited by mCasual on
  • jimlinjimlin Posts: 146
    edited December 1969

    HI server/Kinect not detecting actor the green light is on .on the Kinect ,server states waiting actor any suggestions

  • mCasualmCasual Posts: 4,607
    edited February 2015

    jimlin said:
    HI server/Kinect not detecting actor the green light is on .on the Kinect ,server states waiting actor any suggestions

    is it an 'old' xbox 360 type of kinect?, because my 'server' application only supports those

    also you need the special power/USB adapter ( and it must be plugged .... you never know :) )

    on mine when the kinect is not in use, i see the green light turn on/off about twice per second

    i suspect this only happens because the drivers are properly installed

    for my setup, my Windows had .NET 4.x installed, which is a pre-requisite for the installation of the drivers

    then i installed the Kinect for Windows drivers,

    the "Runtime" installer installs only the drivers needed to make applications work
    http://www.microsoft.com/en-ca/download/details.aspx?id=40277

    the full sdk is for programmers and contains libraries and header files and the Runtime installer
    http://www.microsoft.com/en-ca/download/details.aspx?id=40278

    it's important to avoid installing the v2 drivers which are for the new kinects

    after installing this, i unplugged and re-plugged the kinect
    Windows detected thr presence of the kinect
    it installed a bunch of drivers, maybe 8 drivers, it took a few minutes to complete

    when you start the mcjKinectServer001.exe application, there's should be a sparkling red light coming out of the kinect,
    i would guess that's an infra-red laser that peppers the room with infra-red dots and makes skeleton detection possible

    if you stand, say, at least 8ft from the kinect you should be detected and appear onscreen

    note that in there's also the "Kinect for Windows Developer Toolkit v1.8"
    which is not required, and is quite big to download, but it contains
    sample applications ( including 3D scanning! )
    http://www.microsoft.com/en-ca/download/details.aspx?id=40276

    Post edited by mCasual on
  • mCasualmCasual Posts: 4,607
    edited February 2015

    well the hopes are high ( for the 2nd or 3rd time i know ), for a stable-ish release this weekend

    incognito.jpg
    890 x 658 - 121K
    momovi.gif
    463 x 600 - 672K
    bob2b.jpg
    1280 x 720 - 432K
    Post edited by mCasual on
  • jimlinjimlin Posts: 146
    edited December 1969

    Casual said:
    jimlin said:
    HI server/Kinect not detecting actor the green light is on .on the Kinect ,server states waiting actor any suggestions

    is it an 'old' xbox 360 type of kinect?, because my 'server' application only supports those

    also you need the special power/USB adapter ( and it must be plugged .... you never know :) )

    on mine when the kinect is not in use, i see the green light turn on/off about twice per second

    i suspect this only happens because the drivers are properly installed

    for my setup, my Windows had .NET 4.x installed, which is a pre-requisite for the installation of the drivers

    then i installed the Kinect for Windows drivers,

    the "Runtime" installer installs only the drivers needed to make applications work
    http://www.microsoft.com/en-ca/download/details.aspx?id=40277

    the full sdk is for programmers and contains libraries and header files and the Runtime installer
    http://www.microsoft.com/en-ca/download/details.aspx?id=40278

    it's important to avoid installing the v2 drivers which are for the new kinects

    after installing this, i unplugged and re-plugged the kinect
    Windows detected thr presence of the kinect
    it installed a bunch of drivers, maybe 8 drivers, it took a few minutes to complete

    when you start the mcjKinectServer001.exe application, there's should be a sparkling red light coming out of the kinect,
    i would guess that's an infra-red laser that peppers the room with infra-red dots and makes skeleton detection possible

    if you stand, say, at least 8ft from the kinect you should be detected and appear onscreen

    note that in there's also the "Kinect for Windows Developer Toolkit v1.8"
    which is not required, and is quite big to download, but it contains
    sample applications ( including 3D scanning! )
    http://www.microsoft.com/en-ca/download/details.aspx?id=40276

    Thanks for your quick and well explained response I have reinstalled using your method now works great thanks for your help and your program this will help me a lot and save me a lot of time

  • TugpsxTugpsx Posts: 738
    edited December 1969

    Very nice progress, thanks for all the work you have done in making the Daz community what it is today. We are very grateful.

  • mCasualmCasual Posts: 4,607
    edited December 1969

    Tugpsx said:
    Very nice progress, thanks for all the work you have done in making the Daz community what it is today. We are very grateful.


    thanks!
    i hope this will help me do more youtube videos where Amy presents ( mostly stand up ) the old and new scripts
    animating a stand-up animation can take days

    so i wont need to re-use sword-wielding or boxing BVHs like in

    https://www.youtube.com/watch?v=7NGVSbhLd_Q

    since many don't do animation, i'll try to add snapshot / posing functionnality

    wield.jpg
    854 x 510 - 127K
  • mCasualmCasual Posts: 4,607
    edited February 2015

    good news i think i fixed most problems with the "seated mode" mocaps

    once i complete this and take care of cases where the kinect loses track of some bones

    i will release the official version 001 !

    flai.gif
    320 x 240 - 781K
    talkshow.jpg
    1024 x 768 - 365K
    Post edited by mCasual on
  • mCasualmCasual Posts: 4,607
    edited February 2015

    impending release of the third secret pre-beta version ... today Sunday evening

    ** Video ** ** Video ** ** Video ** ** Video ** ** Video ** ** Video ** ** Video **

    test of the newly implemented "seated" mode ( it was very shake, so i only kept 10 frames per second )

    https://www.youtube.com/watch?v=wnw1wqfgAwQ

    Post edited by mCasual on
  • mCasualmCasual Posts: 4,607
    edited February 2015

    it's a well kept secret but i uploaded
    mcjKinectKitSecretPreBetaVersion_v3

    i still have things to test later tonight

    note : by default the Hip Offset setting is 120 (cm) , this worked well for the Aiko3 figure under DS3
    but for Genesis2/DS4 it needs to be 0

    note also, it's better to "set Limits off" on the figure before transfering the animation from the stickMan

    https://sites.google.com/site/mcasualsdazscripts4/mcjkinectkit001

    1 - fixed : collarbone now excluded

    2 - fixed : mocap files written in same folder as server's .exe

    3 - fixed : the “Seated” mocaps supported

    4 - fixed : minor DS3 compatibility

    5 - fixed : over-twisted arms issue

    6 - new : bend offsets for the neck and feet

    7 - new : option to lock feet side-side and feet twist

    todo : improve manual

    todo: better handling of bad kinect poses

    todo: keep or discard the list of bones participating in the pose transfer .... for now it's purely decorative

    gibo.jpg
    970 x 578 - 149K
    Post edited by mCasual on
  • mCasualmCasual Posts: 4,607
    edited February 2015

    note : i improved the kinectkit manual a little bit

    ---


    teaser .... that's facial motion capture

    video !! https://www.youtube.com/watch?v=vHd2wHhu274

    i dont know yet if it will be combined with skeletal mocap

    it's facial tracking, i dont know if it will ( or can ) be combined with skeletal mocap

    so you could do facial animations in a second pass

    the set of facial features tracked ( by the SDK's libraries )
    is a bit limited ( see list below )
    the eyelids didn't work in my tests but
    maybe it's a question of setting the kinect resolution

    var coef_UpperLipLift = i++; // 0.110862
    var coef_JawDrop = i++; // 0.179137
    var coef_MouthStretch = i++; // 0.197998
    var coef_BrowLower = i++; //-0.220958
    var coef_MouthCornerLift = i++; // 0.069177
    var coef_OuterBrowRaiser = i++; // 0.098132
    var coef_UpperEyeLid = i++; // 0.000000
    var coef_LowerEyeLid = i++; // 0.000000

    asyoumayknow.gif
    256 x 192 - 1M
    Post edited by mCasual on
  • cwgantcwgant Posts: 16
    edited December 1969

    Really liking the work you're doing. It's much appreciated! Hope my little utils can measure-up someday.

  • 3dOutlaw3dOutlaw Posts: 2,471
    edited February 2015

    Hmm.......I was somehow unsubscribed? Looking good! Beta 3 downloaded, and no plans this evening, so I should be able to give it a good testing! :)

    Did you have a process that you have to smooth out any herky-jerky animation? (using some of your other scripts?) If so, i would love to see your steps.

    Post edited by 3dOutlaw on
  • mCasualmCasual Posts: 4,607
    edited December 1969

    3doutlaw said:
    Hmm.......I was somehow unsubscribed? Looking good! Beta 3 downloaded, and no plans this evening, so I should be able to give it a good testing! :)

    Did you have a process that you have to smooth out any herky-jerky animation? (using some of your other scripts?) If so, i would love to see your steps.

    i use mcjDecimate, it reduces the number of keyframes,
    https://sites.google.com/site/mcasualsdazscripts/mcjdecimate-for-ds-12-3

    for moderately short animations i also use mcjCycleFilter, either the "Filter" (Fourier based) or the "smooth" "Synth"
    using mcjCycleFilter's "Current" "Synth" you can reduce the swing of a rotation or morph
    note that using mcjCycleFilter's Filter you can turn a somewhat-looping animation into a perfectly looping animation
    https://sites.google.com/site/mcasualsdazscripts/mcjcyclefilter

    using the SceneGraf plugin, you can manually move or delete keyframes
    For DS3 https://sites.google.com/site/mcasualsdazscripts/scenegraf
    For DS4 https://sites.google.com/site/mcasualsdazscripts2/mcjscenegrafds45

    for the legs ( keep feet in place ) i use mcjAutolimb or mcjAutoLimb2014
    https://sites.google.com/site/mcasualsdazscripts/mcjautolimb
    https://sites.google.com/site/mcasualsdazscripts3/mcjautolimb2014

    often after "pinning" feet in place using mcjAutoLimb
    i force feet to remain oriented the same way using mcjkeeporient
    https://sites.google.com/site/mcasualsdazscripts/mcjkeeporient

    also often used for animation work
    https://sites.google.com/site/mcasualsdazscripts/setinterpolation-for-ds2-3
    https://sites.google.com/site/mcasualsdazscripts4/mcjmirrorlimbanim
    https://sites.google.com/site/mcasualsdazscripts2/mcjlookatthecamera
    https://sites.google.com/site/mcasualsdazscripts2/mcjholdo
    https://sites.google.com/site/mcasualsdazscripts/mcjmaketarget
    https://sites.google.com/site/mcasualsdazscripts4/mcjrepeataction
    https://sites.google.com/site/mcasualsdazscripts3/mcjparent

  • mCasualmCasual Posts: 4,607
    edited February 2015

    http://www.inb.uni-luebeck.de/publikationen/pdfs/TiBa11b.pdf
    that's a German university paper about gaze / iris tracking
    i may use this method to add eye tracking to the kinect's head/face tracker

    here you can see the X and Y gradients of the image
    ( i actually used a daz script to test this ! but this will be done in the KinectServer, way faster )

    and someone here implemented it https://www.youtube.com/watch?v=aGmGyFLQAFM

    cumul.jpg
    620 x 900 - 132K
    eye100out.jpg
    356 x 148 - 7K
    ooff.jpg
    434 x 366 - 33K
    eye_grad.jpg
    498 x 366 - 124K
    prunelle.jpg
    487 x 473 - 73K
    Post edited by mCasual on
  • TugpsxTugpsx Posts: 738
    edited December 1969

    Very nice, this should definitely improve my old head tracking script. looking forward to the development.
    Thanks for sharing.

  • mCasualmCasual Posts: 4,607
    edited December 1969

    if you have a webcam
    there's an interestin in-browser ( javascript ) gaze tracking demo here

    http://aw204.host.cs.st-andrews.ac.uk/camgaze.js/examples/smiley_face.html

  • 3dOutlaw3dOutlaw Posts: 2,471
    edited February 2015

    OK, so I am testing tonight...when I just run the Server (beta 3) standalone write to file, then I import it, it reads as "seated"? I dont have seated checked, and I can see the whole figure, but it reads in that way via the client? Any ideas?

    OK, it seems to be the beta3 Client, because I can load the Beta 2 Client and it reads all the recorded mocap as "not seated", as expected.

    Another thing I noticed is the thigh bends don't seem to match what I do, but I notice that on the stickman, so maybe that's a Kinect thing. When I sit down, the thighs are only slightly slanted out, like I am leaning back on something.

    Its fun though! :) I noticed some limitation of Kinect are spinning around...and turning sideways

    Post edited by 3dOutlaw on
  • mCasualmCasual Posts: 4,607
    edited December 1969

    3doutlaw said:
    OK, so I am testing tonight...when I just run the Server (beta 3) standalone write to file, then I import it, it reads as "seated"? I dont have seated checked, and I can see the whole figure, but it reads in that way via the client? Any ideas?

    OK, it seems to be the beta3 Client, because I can load the Beta 2 Client and it reads all the recorded mocap as "not seated", as expected.

    Another thing I noticed is the thigh bends don't seem to match what I do, but I notice that on the stickman, so maybe that's a Kinect thing. When I sit down, the thighs are only slightly slanted out, like I am leaning back on something.

    Its fun though! :) I noticed some limitation of Kinect are spinning around...and turning sideways

    thanks i'll look at those issues
    in version 3, the moment the kinect reports a loss of track of the hip node,
    it treats the mocap as "seated"
    i'll change this in the next version

    the microsoft drivers really dont do well with turn-arounds
    eventually, i want to re-write the server but based on OpenNI
    because it seems to cope with turn-arounds https://www.youtube.com/watch?v=sWADWpHIubI

  • mCasualmCasual Posts: 4,607
    edited December 1969

    3doutlaw said:
    OK, so I am testing tonight...when I just run the Server (beta 3) standalone write to file, then I import it, it reads as "seated"? I dont have seated checked, and I can see the whole figure, but it reads in that way via the client? Any ideas?

    OK, it seems to be the beta3 Client, because I can load the Beta 2 Client and it reads all the recorded mocap as "not seated", as expected.

    Another thing I noticed is the thigh bends don't seem to match what I do, but I notice that on the stickman, so maybe that's a Kinect thing. When I sit down, the thighs are only slightly slanted out, like I am leaning back on something.

    Its fun though! :) I noticed some limitation of Kinect are spinning around...and turning sideways

    i did a quick non-seated test and it seems to work

    note that i normally check-mark the server's "delayed" check-box,
    so when the capture really starts i'm fully in view

    maocap.gif
    320 x 240 - 160K
  • 3dOutlaw3dOutlaw Posts: 2,471
    edited December 1969

    thanks! its funny, I was just looking at the OpenNI stuff after I was testing this. I could not get it working with MMD yet, so I uninstalled it all, and am going to try from scratch again tomorrow. I have some MMD VMD's that I ran through LiveAnimation to BVH and then imported, and they came out real good in DS.

  • mCasualmCasual Posts: 4,607
    edited February 2015

    3doutlaw said:
    thanks! its funny, I was just looking at the OpenNI stuff after I was testing this. I could not get it working with MMD yet, so I uninstalled it all, and am going to try from scratch again tomorrow. I have some MMD VMD's that I ran through LiveAnimation to BVH and then imported, and they came out real good in DS.

    an old favorite
    https://www.youtube.com/watch?v=dKhLy3kRcq8


    it's quite possible that the Microsoft drivers interfere with the OpenNI (primesense?) ones
    though i think i saw someone who was using the MS SDK plus the OpenNI one plus the OpenCV all at the same time

    my Breckel to bvh to DazStudio test https://www.youtube.com/watch?v=wYEv7gtg68w

    i also was able to get the MMD plugin to work but didnt post the results
    back then i looked for vmd to bvh converters but didn't get far

    would be nice to be able to export dynamic hair animations

    that type of dynamic hair/clothes effects, similar to what secondLife uses ... is in my backlog of projects :)

    Post edited by mCasual on
  • mCasualmCasual Posts: 4,607
    edited February 2015

    still studying options for face tracking
    the 100 2D points tracked by the kinect ( displayed here in DS3 using an mcjBillboards120 figure by the way )
    i'm trying to see is i can get more motion measurements than the 6 animation/action units supplied by the SDK

    i will probably not support gaze/iris tracking

    but i'd like to add blink/eyelids tracking ( instead of faking eyelid animation based on the eyebrows and jaw )

    nicinico.gif
    240 x 180 - 527K
    ptstrack.gif
    256 x 192 - 102K
    Post edited by mCasual on
  • 3dOutlaw3dOutlaw Posts: 2,471
    edited December 1969

    For the VMD to BVH, this worked great! (link) Just selected the character, Import VMD, export BVH...

  • mCasualmCasual Posts: 4,607
    edited December 1969

    3doutlaw said:
    For the VMD to BVH, this worked great! (link) Just selected the character, Import VMD, export BVH...

    thanks !
    i think there's many MMD animations that can be found in youtube and www.nicovideo.jp/

Sign In or Register to comment.