Remake

Allan Quatermain - Pierce Brosnan

Mina Harker - Michelle Dockery

Captain Nemo - Naveen Andrews

Dr. Jekyll/Mr. Hyde -Harry Treadaway/Andy Serkis

Invisible Man - Ben Barnes

M/Professor Moriarty - Charles Dance

Casting The League Of Extraordinary Gentlemen Movie Reboot

https://screenrant.com/league-extraordinary-gentlemen-casting-reboot-movie/

With a new adaptation of Alan Moore's The League of Extraordinary Gentlemen on the way, these actors could potentially play the key roles.

Allan Quatermain - Jason Isaacs

Captain Nemo - Dev Patel

Dr. Henry Jekyll/Edward Hyde - Jamie Foxx

Wilhelmina "Mina" Murray - Lily Collins

Hawley Griffin/Invisible Man - Damian Lewis

M - Charles Dance

Fu Manchu (Aka The Doctor) - Donnie Yen

If we're talking about a more faithful adaptation to the comics series, then I think Gerard Butler and Jason Isaacs are out as Quartermain - in my opinion. Both would be too powerful, too much of action heroes to work right for the faded adventurer that Moore and O'Neill envisioned.

I'd go with Ben Kingsley for Quartermain.
Your suggestion of Rebecca Ferguson does look the part for Mina, but I like HarlemEagle's choice: Michelle Dockery.
Nemo in the comics was portrayed as very much an Indian Hindu, and I think the film should honour that. Honestly, with hair and makeup, I don't think the original actor, Naseeruddin Shah, has aged out of it and he did a smashing job.
I'd cast Jekyll and Hyde with different actors. I like the idea of Daniel Radcliffe for Jekyll, but he's a bit young, so I'd go with Jude Law, I guess? I think he'd be good. Tom Hardy for Hyde, using camera tricks to make him look massive.
Brannagh is a great choice for M!
I do like the "Bond connection" you have, but I'd probably cast Mark Hamill.