My team was recently performing performance tests on an integration solution which pushed messages on MSMQ queues hosted on a remote server via MSMQ send adapter of BizTalk Server. The functionality used the most plain vanilla implementation possible for transactional queues and it worked without a hitch on development and test environment.
It was a different story on the load test environment though. We were in for a surprise. After we had completed the first round of load testing of the system, we noticed quite a few warnings and errors in the event viewer and all of those pointed to "insufficient resources". The exact error message was -
There are insufficient resources to complete the send operation.
For example, this could happen when you are trying to send a large message (message larger than 4095 KB) to a non-transactional queue. Large messages can be sent only to transactional queues.
This is actually a very misleading message, something we established soon after we inspected the messages in the suspended messages queue of BizTalk Server. Most of the messages were in the range of 2 KB - 20 KB. So size of the message was not an issue at all.
More intrigue awaited us as we were not able to send even a single message even after we cleared all the suspended messages from BizTalk Server. We checked the size of message log and directory of MSMQ on the destination server - everything was within the bounds of quota limit set of MSMQ and Queue respectively. Changing the destination queue did not help either - we got the same error message but that helped in establishing that the issue was not with the destination server. Somehow the servers that were running the send handler of MSMQ were choked as far as MSMQ transport was concerned.
So we did what every sane person does - turn to search engines. One of the top search results was "Insufficient Resources? Run away, run away!". It has 11 possible reasons listed in it. In our case, it turned out to be 3rd reason. When we checked the folders at "%windir%\system32\msmq\" on the machines that were running host instances of MSMQ send ports, it turned out that there were lot of MSMQ messages that could not be sent (because of our system's issues) and the cumulative size of MSMQ messages had reached beyond 1.7 GB. Once we cleared these folders on the client machines, the process started working again.
Learning - beware of misleading error messages.
It was a different story on the load test environment though. We were in for a surprise. After we had completed the first round of load testing of the system, we noticed quite a few warnings and errors in the event viewer and all of those pointed to "insufficient resources". The exact error message was -
There are insufficient resources to complete the send operation.
For example, this could happen when you are trying to send a large message (message larger than 4095 KB) to a non-transactional queue. Large messages can be sent only to transactional queues.
This is actually a very misleading message, something we established soon after we inspected the messages in the suspended messages queue of BizTalk Server. Most of the messages were in the range of 2 KB - 20 KB. So size of the message was not an issue at all.
More intrigue awaited us as we were not able to send even a single message even after we cleared all the suspended messages from BizTalk Server. We checked the size of message log and directory of MSMQ on the destination server - everything was within the bounds of quota limit set of MSMQ and Queue respectively. Changing the destination queue did not help either - we got the same error message but that helped in establishing that the issue was not with the destination server. Somehow the servers that were running the send handler of MSMQ were choked as far as MSMQ transport was concerned.
So we did what every sane person does - turn to search engines. One of the top search results was "Insufficient Resources? Run away, run away!". It has 11 possible reasons listed in it. In our case, it turned out to be 3rd reason. When we checked the folders at "%windir%\system32\msmq\" on the machines that were running host instances of MSMQ send ports, it turned out that there were lot of MSMQ messages that could not be sent (because of our system's issues) and the cumulative size of MSMQ messages had reached beyond 1.7 GB. Once we cleared these folders on the client machines, the process started working again.
Learning - beware of misleading error messages.
THANKS!!! What I struggled with exact same error.
ReplyDelete