Loading...
> >

mcjKinectKit for Kinect V1 and in June 2018 Kinect V2 - script and exe for DS 3, 4

12357

Comments

  • 3dOutlaw3dOutlaw Posts: 2,471
    edited December 1969

    Yes, I agree. It would be nice if someone had a library or links page to some of the more popular ones. I will search around during lunch. I will maybe try again tonight with the MMD/Kinect using these steps, as it has the correct Kinect version, and they say they tested on Win7/64, which is what I have: (MikuMikuCapture)

  • mCasualmCasual Posts: 4,607
    edited December 1969

    i think i'll only handle the few facial tracking features from the Kinect SDK
    and leave eyelids detection aside for now

    maybe i'll post the automated blink generator i sometimes used in animations

    testerrr.jpg
    1280 x 720 - 171K
  • mCasualmCasual Posts: 4,607
    edited December 1969

    comparing the kinect-view tracked face points to the "animation units" animation

    just the head rotations and the "jaw"/open mouth morphs

    i'd say there's some lag and loss of data

    loveinahopelessplace.gif
    300 x 225 - 1M
  • ghastlycomicghastlycomic Posts: 2,531
    edited December 1969

    I've downloaded everything needed to run the software and script and installed them, but for some reason the only thing that happens is the green light flashes on the front of my Kinect and the MCJKinectServer just shows whatever was in the background behind it. No stick figure?

    Any idea what's wrong? The Kinect is the old XBox360 one and it works fine on my XBox. The adaptor seems to be working. Is there a simple diagnostic program I can run to make sure everything is talking the way it should be?

    Kinect_Not_Working.jpg
    1056 x 730 - 91K
  • 3dOutlaw3dOutlaw Posts: 2,471
    edited December 1969

    Look under the Start-> Kinect for Windows SDK, and there is a Developer Toolkit Browser (at least there is on my PC), and that has different apps you can use to test the Kinect is working right.

  • ghastlycomicghastlycomic Posts: 2,531
    edited December 1969

    3doutlaw said:
    Look under the Start-> Kinect for Windows SDK, and there is a Developer Toolkit Browser (at least there is on my PC), and that has different apps you can use to test the Kinect is working right.

    Thanks. Running the Kinect Explorer seemed to fix the connection problem. I wonder if I'll have to run that first every time I plan to use it.

  • 3dOutlaw3dOutlaw Posts: 2,471
    edited December 1969

    Yea, I run that to make sure its working...I did not realize it was a pre-req, as I just had done it as a test each time.

  • ghastlycomicghastlycomic Posts: 2,531
    edited February 2015

    I think I need to build a tripod mount for my connect sensor. I put it on an end table to do a test and as a result the Genesis figure is at a weird angle to the flood, like I'm leaning way back. But YAY it WORKS!!!

    [edit] Oh hey! The Kinect Wall Mount has a standard tripod connector on the bottom. Time to get it and an extension cable. Off to Amazon I go!

    Post edited by ghastlycomic on
  • ghastlycomicghastlycomic Posts: 2,531
    edited February 2015

    I was singing "Goody Goody Two Shoes" by Adam Ant when I did my first mocap test.

    Also running the Explorer seems to have permanently fixed the connection problem I was having.

    And you should also see the weird things that happens to the stick man when my dog walks infront of the kinect. I'm gonna have to try and mocap that.

    Kinect_Test_2.gif
    500 x 375 - 828K
    Post edited by ghastlycomic on
  • mCasualmCasual Posts: 4,607
    edited December 1969

    kinect's candid face tracking mask, in DS3

    maskit.jpg
    798 x 607 - 109K
  • mCasualmCasual Posts: 4,607
    edited December 1969

    ghastlycomic said:
    I was singing "Goody Goody Two Shoes" by Adam Ant when I did my first mocap test.

    Also running the Explorer seems to have permanently fixed the connection problem I was having.

    And you should also see the weird things that happens to the stick man when my dog walks infront of the kinect. I'm gonna have to try and mocap that.

    my guess is windows did not finish installing the kinect drivers when you started the server
    after i installed the Kinect SDK Runtime, i unplugged and re-plugged my kinect
    and windows detected the kinect and started installing maybe 10 drivers, which took maybe 5 minutes to complete

  • mCasualmCasual Posts: 4,607
    edited December 1969

    dont know if you'll get that when i publish the kinectface kit for DS

    ( i mean a floating animated kinect mask )

    but this will help me figure how to obtain the exact/correct head rotations applied to the Daz figures
    (
    because in previous examples it was almost correct but not really )

    another thing i don't know is if i'll have a skeleton+face capture mode

    possibly they will be separate processes/passes

    and the other thing i want to do is to make possible the synchronizable recording of the audio

    either by my mcjKinectFaceServer application

    or using a program like audacity, in that case the mcjKinectFaceServer app could generate a beep when the face-mocap starts

    talkinghead.gif
    300 x 225 - 625K
  • mCasualmCasual Posts: 4,607
    edited February 2015

    since Aiko3's head joint is not oriented orthogonally

    making the head do a 30 degrees Y-Rotation as the kinect is asking us to do

    is not as simple as setting the head nodes's Twist channel to 30 degrees

    so we'll use quaternions
    in this test script we select the head node
    get it to Y-Rotate 30 degrees from Aiko's zero-head-pose

    as you can see in the SceneGraf ( free plugin ) graph, all 3 rotation channels of the head contribute to this Y-Rotate movement

    here's the daz script code

    
var node = Scene.getPrimarySelection(); //the head node
var ang = -30; //degrees
var rad = Math.PI / 180;
var angr = ang * rad; //radians
var w = Math.cos( angr / 2 );
var y = Math.sqrt( 1 - w * w ); // keep the quaternion normalized 
var cw = ( angr < 0 )?y:-y; // clockwise/counterclockwise
var R = new DzQuat( 0, cw, 0, w ); 
node.setWSRot( R );

    though i can possibly use

    
void  setValue (Number axis0, Number axis1, Number axis2, DzVec3 angles)


    
void  setValue (0,1,2,DzVec3( angx, angy, angz ) );

    but unfortunately, at least in DS3, this function is unusable because it's overridden by setValue(x,y,z,w)

    robotic.gif
    320 x 180 - 445K
    rotit.jpg
    1920 x 671 - 319K
    30deg.jpg
    960 x 540 - 296K
    Post edited by mCasual on
  • mCasualmCasual Posts: 4,607
    edited February 2015

    ** Worthy Update ** Worthy Update ** Worthy Update ** Worthy Update ** Worthy Update **

    New Kinect Client/Server

    still in Beta but hey

    This version will not misjudge the mocaps as being in "seated mode"

    in the previous version 001v3 , the moment the hip tracking failed, it was deemed in "seated mode"

    in this version 001v4 only sessions without hip poses are deemed in seated mode

    https://sites.google.com/site/mcasualsdazscripts4/mcjkinectkit001

    direct link to the 001v4 package
    https://sites.google.com/site/mcasualsdazscripts4/mcjkinectkit001/mcjKinectKitSecretPreBetaVersion_v4.zip?attredirects=0&d=1

    xfactor.jpg
    1920 x 1080 - 415K
    Post edited by mCasual on
  • mCasualmCasual Posts: 4,607
    edited December 1969

    the kinect face kit is still in the works

    here i created an UpperLipRaiser morph from Aiko3's MouthRaise and MouthOpen morphs
    and the face expressions are more smily!

    in most cases the script will use existing face morphs
    the kinect gives us 6 action/animation values

    UpperLipRaiser --------------------- custom morph
    JawDrop ------------------------------Aiko3 MouthOpen
    MouthStretch ------------------------ Aiko3 MouthWide
    BrowLower --------------------------- Aiko3 BrowUpR and BrowUpL
    MouthCornerLift -------------------- Aiko3 Smile or mcjA3Smile
    OuterBrowRaiser ------------------ not sure yet

    hinhinforum.gif
    320 x 240 - 774K
  • 3dOutlaw3dOutlaw Posts: 2,471
    edited December 1969

    Sorry, I have not tried this again since your last release, but I will try to do so soon. :) Looking good!

  • mCasualmCasual Posts: 4,607
    edited December 1969

    New Youtube Video Test of Kinect face mocap with sound ( recorded by Audacity )

    with moderately not too bad synchro.

    https://www.youtube.com/watch?v=xmfXQ6Y6new

    hinhinforum.gif
    320 x 240 - 774K
  • vincerexvincerex Posts: 2
    edited December 1969

    Hello Very appreciate What you're doing
    I would be grateful if you could do a video tutorial on how to install .
    sorry for my bad English

  • jimlinjimlin Posts: 146
    edited December 1969

    I Cannot record to file I get ERROR unable to open file stream possible due to an invalid path or file name
    mcjkinectserver 001 is in c/documents/mocap/ I have also tried c/username/mocap/ c/user/public/documents/mocap/ .
    I have AutoFile option selected

  • mCasualmCasual Posts: 4,607
    edited March 2015

    jimlin said:
    I Cannot record to file I get ERROR unable to open file stream possible due to an invalid path or file name
    mcjkinectserver 001 is in c/documents/mocap/ I have also tried c/username/mocap/ c/user/public/documents/mocap/ .
    I have AutoFile option selected

    are you launching the server in standalone mode or through a batch file or through the Daz Studio script?

    -just now i placed mcjKinectServer001 in C:\Users\Public\Documents,
    and launched it by double-left-clicking on it
    i had the auto-filename and the write-to-file options check-marked
    and it did record a file named 20150308102609.txt

    when i launch it through the daz script mcjKinectClient
    the location of the server is C:/Users/Public/Documents/mcjKinectServer001.exe
    i notice that in your post the location looked like C/Users/Public/Documents/mcjKinectServer001.exe
    wonder why the : is missing

    proof.jpg
    1444 x 718 - 177K
    Post edited by mCasual on
  • jimlinjimlin Posts: 146
    edited December 1969

    Thanks for your response I was launching from daz script but does not record

    I then launched from c:/user/public/documents/ now records

    the missing : was because I forgot to enter it

    Thanks again working great

  • SimonWMSimonWM Posts: 924
    edited December 1969

    I finally had time to test this script but for some reason my camera is not working. It works fine with ipisoft. Please take a look at the screencap. i'm using an xbox 360 kinect with usb adapter. i've tried USB 3 & 2 but no luck

    Untitled-1.jpg
    1626 x 786 - 265K
  • mCasualmCasual Posts: 4,607
    edited May 2015

    SimonWM said:
    I finally had time to test this script but for some reason my camera is not working. It works fine with ipisoft. Please take a look at the screencap. i'm using an xbox 360 kinect with usb adapter. i've tried USB 3 & 2 but no luck

    i would think the white screen means the app cant initiate a "dialog" with the kinect
    it never receives a detected skeleton pose and therefore doesn't touch the display, so it remains white

    maybe maybe if you did not reboot since your last use of the kinect the kinect thinks it's still under the orders of that other app and therefore refuses to work for us --

    to reset the Kinect
    you could try unplug it from the pc, unplug the power adapter
    replug the power adapter, replug the usb
    look at the lower right corner of your screen and try to see if Windows indeed notices it
    usually it displays a message and since it already installed the drivers before, it only takes a second or 2 before it's ready

    i never tried the "security" apps for kinect ( i see you have one there) , but it's possibly something that's running in the background and stealing the kinect from us. so you could try to use the Windows Task Manager and 'kill' any task or service that could steal 'the precious' i mean the kinect


    the app uses DirectX for the display and usually the display is a black screen background
    so, maybe you have another application running, like a webcam or Mediaplayer that steals the directx availability from us
    ( well it would be surprising but who knows )

    but i'd be more inclined to think it's not working because you did not install the special Microsoft Kinect drivers i mentioned in the instructions, or Ipisoft's drivers sidelined our drivers

    so you could try (re) installing the drivers mentioned in my instructions

    normally the first time you plug your kinect, immediately after installing those drivers
    you see Windows installing a huge (like 12) number of drivers, for the kinect motors, the microphones, the sensor, etc etc

    it's possibly better if webcams are not plugged when using the kinect

    good luck,

    also, make sure the app is not told to write its output to disk in an area where only privileged users are allowed to write

    Post edited by mCasual on
  • SimonWMSimonWM Posts: 924
    edited May 2015

    Thank you Casual, I managed to solve the drivers issues by installing both the Kinect SDK v1.8 setup.exe and the Kinect Developer Toolkit v1.8 setup.exe.

    Now I'm able to see the green stickman moving in the kinect server as well as the stickman moving in the DAZ viewport but nothing records in the timeline. I need to re-read the instructions since I've used both ipisoft and Reallusion Mocap Plugin but this one is great in that it happens inside DAZ Studio. I'm guessing I'm expecting it to work similar to the commercial ones when it doesn't.

    This is all excellent work!!! I really hope you get to do the realtime capture with genesis and more than one Kinect to handle occlusion and we might get something better than my expensive ipisoft rig.

    Post edited by SimonWM on
  • SimonWMSimonWM Posts: 924
    edited May 2015

    Well, several things don't work for me or at least not as I would expect them to work:

    "Write to client" doesn't works for me:

    1-The stick figure the script creates tends to sink down the floor as soon as it starts receiving live data in Studio.
    2- Sometimes I will get two stick figures on the McKinectServer001. The extra figure will be on the left corner of the windows with all the bones in a twitch while the other one moves correctly. I've found two ways to get rid of the extra figure, either by activating seated mode and deactivating it or by moving the real stick figure to that side. It seems to be simply a ghost figure that gets erased that way, a glitch.
    3-Write to client only allows me to preview the moves on the DAZ Studio stick figure. I check delayed and hit start and nothing happens.
    4-If I hit record on the Client then the stick figure in DAZ Studio stops following my moves. Absolutely anything happens with the timeline after I hit record. I would have expected to maybe see the timeline advance while it is recording. I guess it goes to memory? Only it doesn't.
    5-Once I hit stop nothing happens either.
    6-The "Transfer this Range" button opens a window that seems to be loading the captured motion except that nothing happens in the timeline either.
    7-So I stop the Kinect server and absolutely nothing records. The stick figure and the DAZ figure simply stay half thru the floor in the position they took when I went to press the stop button.

    "Write to file" sort of works for me.

    1-I still have the problem that the figure sinks thru the floor.
    2-The server will write a Txt file that I can load using the client "Load Mocap File" button. It will load my captured animation on the stick figure and the "Transfer this range" will put it on Genesis.
    3-No idea how many of those server buttons work. I thought a hip offset value of zero would control the problem of my figure (Genesis) going thru the floor but instead it made the stick figure smaller than my target figure and kind of ruined my results.

    I know this is a great script since I've seen what you can do with it. I just find it extremely difficult to use and understand right now. Its late and I'm frustrated, maybe I'll go thru the instructions one more time in the morning and try to make sense of it.

    Post edited by SimonWM on
  • mCasualmCasual Posts: 4,607
    edited May 2015

    SimonWM said:

    1-The stick figure the script creates tends to sink down the floor as soon as it starts receiving live data in Studio.

    i installed my kinect about 5 ft above the floor, the camera is level,

    i have to stand about 10 ft away to get the kinect to see me head to toe

    i did not include motor controls in the server, i used one of the sample apps (in the bin folder of the Microsoft SDK )
    but you could also use Breckel or ipisoft to set the camera tilt at 0 degrees ( this may all be superfluous tho )

    my kinect has an unobstructed view of the floor it's important because it helps the kinect locate the floor
    the kinect drivers transmit the vertical distance from the camera to the floor and the camera tilt

    when i build the skeleton in the daz scene i place it on Daz Studio's floor
    when i receive the kinect poses i just place it where kinect says it sees it

    my programming efforts on that subject were put in making the floor horizontal ( by un-tilting the coordinates i receive from the kinect )

    once the capture is done, using a parent null node ( or the root node ) i simply moved the whole session animation to ground level

    when you transfer the animation from the skeleton to the figure, there too, the figure will probably not be at daz-studio-floor-level
    so you'll have to move her using her root node or a parent node

    Post edited by mCasual on
  • SimonWMSimonWM Posts: 924
    edited May 2015

    Ok, then the behavior is expected. I'll move the character with a null like you suggest.

    I follow these instructions to capture in DAZ Studio but animation never gets into the DAZ timeline while capturing:

    Typical Motion Capture Session
    A typical motion capture session would go like this:

    - start the mcjKinectClient
    - start the mcjKinectServer using the client's 'Start Server' button

    The server initiated a 'conversation' with the Kinect sensor
    The kinect sensor's red light should now be visible
    The moment the kinect and the driver detects a human shape,
    you should see a green stickman appear in the server's black screen

    - click the client's 'Record' button
    - move the server's window aside so it doesn't obscure Daz Studio's viewport
    - turn on the server's 'delayed' option
    - click the server's 'Start' button
    - stand in front of the Kinect
    - wait for the delay to elapse

    After this step nothing happens in DAZ Studio viewport??!!

    Post edited by SimonWM on
  • mCasualmCasual Posts: 4,607
    edited December 1969

    SimonWM said:

    2- Sometimes I will get two stick figures on the McKinectServer001.

    The kinect can track many actors at once, but mu server and daz script are not written to process them
    in fact the presence of a 2nd actor will probably completely ruin the motion capture session

    sometimes the kinect thinks it sees a 2nd actor, but in fact it's just a pile of clothes and stuff and a cat wearing a hat

    there's even ghost hunting people who present this as proof of the presence of a ghost ( no really, look it up on youtube, kinect ghost )

  • mCasualmCasual Posts: 4,607
    edited May 2015

    SimonWM said:

    3-Write to client only allows me to preview the moves on the DAZ Studio stick figure. I check delayed and hit start and nothing happens.

    when you establish the communication between the daz studio script ( client ) and the server,
    the Client starts to be on the lookout for poses to be sent by the server

    when the kinect detects an actor, it sends the poses to the server
    and the server displays the green skeleton
    but the server doesn't relay the poses to the client, it doesn't record them to disk either.

    when you press the server's Start button, and if the "delayed" option is Off
    the server will transmit the poses to the DS Client, the as soon as the kinect sees an actor

    if the "delayed option is On, you have 5 ( or was it 10 ) seconds to walk on-stage
    this way the Client will receive clean fully formed skeleton poses

    The "write to client" checkbox is only there to let the DS client see what's going on in the server
    or to prevent the DS client from seeing what's going on in the server.

    technically speaking the "write to client" option enables or prevents the "printing" to stdout
    so if you started the server in standalone mode, you'll see the data flow in the DOS window
    or it will be piped to a file if you so chose

    Post edited by mCasual on
  • mCasualmCasual Posts: 4,607
    edited May 2015

    SimonWM said:

    I follow these instructions to capture in DAZ Studio but animation never gets into the DAZ timeline while capturing:

    TEST 001 Part 1

    i'll do a test in DS 4.7 64 bit right now

    ( i didnt do many tests in DS4 because for a very long time, i was almost never in DS4 , it was crashing on me painfully often )
    ( possibly because i did not install the newer versions as i thought i had )

    also note: most of my captures were done in standalone mode meaning

    i start the server "manually" using a shortcut on my Windows task bar
    i click the "write to file" option
    i click the "auto filename" option
    i click the "delayed" option
    i click the start button
    the progress bar fills up with green
    the server's Led is yellow
    i walk 'on-stage'
    the green skeleton appears on-screen
    the green progress bar starts a down-count
    ...waiting ....
    the server's Led turns red, we're recording to disk!
    wave a hand and come back to sit
    i click on stop

    using Windows Explorer, i go to the folder where mcjKinectServer is located

    "C:\Program Files\Microsoft SDKs\ ......... \mcjKinectServer001.exe"

    ah HA

    20150519133454.txt which was created May 19th 2015 at 13:34:54

    i open D|S 4.7

    load a scene containing Azura ( Genesis 1? Aiko ...5 i think )

    azura1.jpg
    960 x 540 - 299K
    Post edited by mCasual on
Sign In or Register to comment.