I recently ran into an bizarre case. We have a WCF service which needs to route requests to another WCF service that is protected using SSL hosting and certificate authentication (two separate certificates). We set up everything as expected i.e. added certificate to LocalMachine's Trusted Root Certificate Authority, added required elements in configuration file to ensure service is working after loading certificates and the target WCF service is accessible from the machine. Everything worked like charm. Good.
The next day we updated the binaries as part of daily deployment process. And...drum rolls...all of the WCF service methods that routed requests to target WCF service started to fail with following message :
The request was aborted: Could not create SSL/TLS secure channel
I checked Application error logs for any details in inner exceptions and entries in Application Event Log and found nothing. Then I installed Fiddler on the machine to identify any specific HTTP responses that may help but it returned nothing - every request succeeded with 200 OK. Frustrating.
Then I did what any sane person will do. Dig some more :). I decided to look into categories other than Application (Security, Setup and System). I found an interesting entry in System category whenever the service call failed.
A fatal error occurred when attempting to access the SSL client credential private key. The error code returned from the cryptographic module is 0x8009030d. The internal error state is 10003.
Most of the answers/suggestions available on the Internet that are related to the above error pertained to hosting the SSL enabled WCF service which was not applicable to my case. However, i tried one simple thing - I manually removed the certificate and added it again. Voila - everything works again.
So we included a step to remove and add the certificate after updating binaries to our deployment process. The solution is quite trivial but the problem itself surprises me.
