That is exactly correct. FSM is not stored on the flash. This happened to me on my DEX console before.
I updated to OFW 4.11 DEX, to play Twisted Metal from original disc. A couple days later, I was gonnal mess around in FSM. And test some remarry files that were someone said would restore blu ray movie playback on DEX. He was full of it, but that is beside the point.
Anyway, I entered FSM right away with my PSP. Then it just booted to the XMB, when I tried the usb with the remarry files. And I had that big red box FACTORY/SERVICE MODE.
To fix it, I flashed patched CEX dump and went to do downgrade again. Then convert back to DEX. When I got to what should of been the "connect controller screen", it would just power off.
That is when I realized also, that the ps3 powers off at that step when you are in FSM. It was looking for the Lv2diag.self, with what syscall to use. When it didn't find it, it shut off and looked like a brick.
I tried to just put the USB drive in with the .self and PUP, then it installed.
I was surprised. I booted the console and it was on Rogero 3.55 in FSM.
EDIT - O. k. I think I missed something now, while typing that long story. LOL