What would have happened if indeed the reactor exploded, BUT the ship had deliberately been sunk to the bottom of the ocean first? Like a suicide by the crew to prevent the misunderstanding and radiation disaster. At worst I imagine a small earthquake would be registered? Certainly such a disturbance couldn't be mistaken for a full-on first strike.
Just Google " Underwater Nuclear Explosion ". There will be a fairly large mushroom cloud followed by a shock wave. While big, both of these would not be enormous enough to damage severely that U.S. Navy destroyer or the relatively close by NATO base.
This would also be an A-bomb level type of explosion unless the very unfortunate occurred and the initial explosion set off the H-bombs onboard. I am doubtful that would happen as a very precise set of events have to occur to set off an H-bomb type of explosion.
It might be possible to off-load all the nuclear bombs and have the sub some distance away. The implosion from going below crush depth would probably rather instantly cool the reactor components and avoid a nuclear explosion from the reactor itself.
Since time was of the essence, quickly going below crush depth would minimize the chances of a nuclear explosion.
There are few things that really need to addressed here.
I am no expert, I didn't study nuclear technology or physics, so feel free to cross-check, but this is what I know:
While in the movie a crew member notes that the loss of coolant will lead to a chain reaction and therefore "thermonuclear explosion", this is not the case. However, this is also no movie mistake. People back then indeed assumed this would happen, and the crew member indeed said this, according to the reports following the incident.
What would have happened is pretty much exactly what happened to Chernobyl's reactor number 4 back in 1986, a "core melt accident".
In a core melt accident ("meltdown") the reactor's fuel (uranium in this case) starts to melt under extreme heat due to loss of coolant, ultimately leading to the destruction/breach of the reactor core and the release of contaminated material, either by corium (the molten fuel and all it contains) melting through the reactor's hull, or due to a steam explosion when the molten metal(s) react with the hot steam inside the reactor (the remains of the coolant).
That sort of explosion is NOT to be confused with that of an atomic bomb, however!
A reactor is not build like an atomic bomb, which needs a very specific setup and ingredients to do what it does - but I do not understand that fully myself yet. I know for sure though: A reactor can not explode like a nuclear bomb, ever.
So, what would have happened?
The boat would have been ripped apart by a steam explosion, releasing a plethora of radiation and contaminated material into the North Atlantic with severe ecological consequences I do not dare to imagine or predict.
The US destroyer mentioned, would be "fine" aside from the fallout it would receive, but that can be dealt with.
For example, the USS Ronald Reagan conducted a 'wash down' after the Fukushima Daiichi power plant explosion as it was rather close to it at the time and measured radiation aboard.
But didn't they mention that nuclear warheads or at least conventional warheads were on the ship? They could have exploded through the steam explosion.
The steam explosion would probably disrupt the interior of the boat, pushing interior bulkheads forward and aft on either side of it, but it would also, in fractions of a second, breach the hull expending the force outward while spreading the reactors innards around. Any nuclear torpedo warhead at the bow or stern would be smashed about but probably not even broken open. And, while there have been near detonations in nuclear weapons that were involved in air crashes, weapons normally require a lot of specific things to occur in a specific order or they are just radioactive junk, or at worst, radioactive junk spread over a larger area. Pretty much any incidental detonation of conventional weapons near nuclear ones would just lead to that sort of mess.
No.
Actually detonating a nuclear device is not as easy as just "blowing it up", it takes more. That is why not ever did a nuclear weapon detonate despite being lost/destroyed (in various aircraft crashes throughout the decades for example).
There is no way for a nuclear reactor of that type to undergo the same kind of nuclear explosion as a bomb. The fuel density is far too low. It is possible that the crew could bring it prompt-critical. This is a condition in which the control rods are withdrawn as fast as possible (still rather slow) while the reactor is cooled down . The power will eventually rise so fast that the coolant will vaporize and cause a steam explosion. This would shut down the reactor immediately, but no nuclear explosion.
Lots of fissionable material and fission products would be released, but as long as the sub was submerged, the radioactive particles would not go airborne. An explosion would be heard by other ships equipped with sonar and by any underwater sonar array, but they would know it was not a nuclear bomb being detonated.