You need a protocol analyzer setup between the dongle and a PS3 running TB CFW. Even in that case, you won't get anywhere as the handshake is a challenge/response ie: dongle says hello, CFW sends a random string challenge and expects a response encrypted with keys (which the dongle has in it's firmware)
To put it plainly, the PS3 sends for example "0123456789ABCDEF" and the dongle takes that, encrypts it with the key, and sends it back as authentication. Once authenticated, CFW loads the payload-enabled portion of itself and boots. Dongle probably has sections of the CFW in it, which it sends encrypted to the PS3.
It's mathematically impossible to reverse the algo/keys using this though. As I've said before, the dongle is a red herring. If you remove that from the equation entirely, you're left with the CFW and the eboots. The CFW is also a red herring, because if 3.60+ games run on their CFW, they run on regular 3.55 CFW. So that leaves the eboots. The TB eboots are regular 3.60+ eboots decrypted and resigned with TB DRM. In order to eliminate TB, we have to figure out how they are decrypting retail eboots. This might be done with hardware, or they might have paid someone for Sony keys. If it's done via hardware, then we can replicate it. If they have keys, they win. Simple as.