It can take a lot of work to create a high quality applet compared to the amount of time it takes someone to create a high quality flash application. Building a presentation layer can be tedious without the right tools.
Furthermore, Oracle/Sun gave up on applets a while ago. They still support them but are not actively trying to improve them. Oracle is currently putting all of its weight behind JavaFX. JavaFX is a technically viable solution and has the advantage of allowing you to leverage your Java knowledge. The problem is most web designers don't have much Java knowledge and don't want to bother with it so JavaFX has never been very popular. If a web language isn't popular it can annoy your users when they have to download plugins to use it.
Another thing to keep in mind is that Flash itself is also on the way out. If you're going to be developing and maintaining this project yourself then feel free to pick up JavaFX.
"it isn't used widely now"
I disagree with this perception... some examples:
Mindterm is a Java based ssh freeware client supporting ssh2 can be run embedded as applet google for it
Mizu web phone is a Java and open standards based SIP VOIP client which can be embedded as an Applet -google for it-
OpenSignX is an opensource Java based document and form signing Java applet for PKI X.509 certificates. google for it
Coolsmile is a Java based IRC client that can be run as app or embedded Applet -google for it-
JFTerm is a Java telnet client. Can be run as desktop app or applet. Supports telnet (23), ssl and ssh -google for it-
J3Dworkbench is a Java 3D game-design authoring tool. delivery via Webstart or as Applets -google for it-
There´s also thevirtualheart dot org a Java based heart simulator also an applet
or Geocaching browser (JavaFX) at canoo dot com
You cannot do everything in HTML5... for instance take a look at complex apps like JITSI (videocomerencing, voip app) or MuCommander -google it- or Sweet Home 3D, or Art of Illusion... all java-based. Java serves a purpose... and Microsoft and its friends have been trying to discredit it, replace it with something else, and it continues being relevant. ask yourself why. it´s because it serves a purpose and its level playing field software ecosystem is HUGE.
Do a seach of code dot google.com for Java-related projects... and see.
Plus, to the commenter above who says JavaFX has limited appeal let me remind you that Java is NOT ONLY a programming language but actually 3 things: a programming language, a runtime environment, and a level playing field software ecosystem. You can use Java libs without writing a single code of Java language ... using Jython (Python for Java), JRuby (Ruby on Java), NetRexx (open source Rexx on Java, see www.netrexx.org), xRuby (ruby to Java bytecode compiler) or Jabaco (Basic to java bytecode compiler)... just to name a few of the many JVM languages... http://en.wikipedia.org/wiki/List_of_JVM_languages