From a screenshot it is hard to do troubleshooting - so if my post doesn't help, then please send the GH file to me via eMail!
In any case, there seem to be two separate issues:
- The tool geometry should be placed as if the Rhino coordinate system where the tool coordinate system, with Z+ being the normal of the flange, and X+ facing "downwards" when the robot is in its default home position.
- If you define the tool frame with a plane, then the origin should be at the tip of your tool geometry (i.e. somewhere close to the Rhino origin), with the plane oriented the same way you would like it to be positioned - in your case probably Z+ of the plane facing global Z-.
Personally, I would recommend measuring the tool at the robot, then entering the actual XYZABC values in the Custom Tool component and if necessary transforming your tool geometry so that it fits together. Otherwise take care to enter the generated XYZABC values (you can get them e.g. by clicking the grey message balloon on the component) at the robot, so that the simulation is correct.