It seems wired to ask this, I would think that using FXML to write our custom components is the obviously the right way to go.
But as we can see from ControlsFX, JFXextras and even the book 'Mastering JavaFX8 controls do not use or mention the use of FXML in custom controls.
Despite that, the official documentation to say to go on that route, and create JavaFX controls via FXML.
What is the more correct way and why ?
There are two kinds of custom controls in JavaFX:
fx:root based custom controls: These custom controls are styleable (support CSS), but are not skinnable.
If you're an application developer, this is what you want to use most of the time.
This kind of controls are usually application or organization specific. Additional custom skins for these controls are not needed, CSS is enough.
Usually they just group some controls together and provide a nicer API and hide the internals.
They are much easier to write than skinnable custom controls.
I'm thinking of this as an alternative of the JPanel subclasses in the Swing-world.
I suggest to break down your GUI into small self-contained fx:root based controls (high cohesion, low coupling). Use the Mediator Pattern to combine these controls (the parent control manages/ configures its child controls and listens to the events of its child controls. The child controls don't know about each other).
You can create whole hierarchies of such controls. Breaking them down into small controls will make maintenance easier (e.g. if the layout or the user interaction has to change).
skinnable custom controls: skinnable custom controls are not only styleable (via CSS) but their look and feel can be changed completely by providing a custom skin.
If you're a controls library developer then most likely you want to provide skinnable custom controls to allow the users of the library to get most out of JavaFX.
This kind of controls are usually very basic, highly reusable controls and often (but not always) use some custom painting as well.
They are harder to get right than fx:root based custom controls.
There is no correct way. But we may break it down to this:
Do you know the amount of JavaFX elements to display before your application has started? If yes, you can build your GUI with SceneBuilder or directly in XML writing the FXML File. This is true for most GUI screen setups where the overall layout (MenuBar, ToolBar, content, sidebar, ...) is well known and does not change over time.
Now, custom components do not know how to display itself. There are many options to correspond for and most custom components are build up dynamically or can change drastically as the user changes these options. You can not write a single FXML file but instead would have to write many single files, each would build a small part of the custom component. This is not really handy to manage, so most developers choose to do it without FXML, directly in code.
And on top of that: There are many custom components which use a special layout which is not built in into JavaFX, to achieve that you have to write your own containers and wire them up with custom components to get the perfect result. In the end, custom components are built (or better: should be built) to be used inside FXML / SceneBuilder.