I want to write a very simple Java 3D editor(for experiment). I know the basic JavaFX usage, and I know enough OpenGL knowledge. But all my OpenGL experience is from working with C/C++.
Could I make a 'canvas' in JavaFx application and map OpenGL viewport on it?
Internally, JavaFX can use OpenGL as a rendering pipeline, so some care on integration between the two is required to avoid conflicts.
OpenGLNode in JavaFX
Richard Bair, JavaFX team lead posted on the openjfx development mailing list:
One thing I want to see done (for example) for the 8 update is to have an OpenGLNode or NativeSurfaceNode or something along those lines so that if you are doing your own D3D / OpenGL you can have a way to send those raw commands down to the graphics card but still have your node composited in the scene graph.
So a future JavaFX release update might include an OpenGLNode. Such a feature would probably not see inclusion in a JavaFX general availability release until the next JavaFX feature release after the initial Java 8 release (my guess is that would put it at about September 2014).
3rd Party OpenGL/JavaFX integration
You don't need to wait so long to start integrating JavaFX and OpenGL. All of the required source code to start an implementation is open in the OpenJFX repository, so you could try building a custom integration of that code with a library such as lwjgl or jogl.
This answer will get dated as developers start performing integrations of JavaFX with existing Java wrappers for OpenGL apis. Some developers have started such work already - run a google search of
lwjgl javafx or
jogl javafx to find out about current integration projects and their status.
The simplest integration is probably to have a 3rd party library render to an off screen buffer then transfer the pixels from the buffer to a JavaFX WritableImage or Canvas as required to get the OpenGL rendered graphics composited into the JavaFX scene graph.
JavaFX 3D API Alternative
JavaFX has its own lightweight 3D api that provides the ability to composite phong shaded 3d models into the JavaFX scene graph. The JavaFX 3D api is not going to provide all the power of a full OpenGL api integration, however using the JavaFX 3D api is relatively simple. A description, with code examples of the 3D features in Java 8 is on the open-jfx wiki.
Interactive Mesh provides a free 3D model importer for JavaFX, that allows you to very simply bring complex shaded and textured 3D models into a JavaFX scene graph.