The website for Boomershoot entries uses an email API to automatically send email to entrants and for sending out mass emails to the announcement list. This had worked fine for Boomershoot 2023. But as I was ready to announce Boomershoot 2024 in early summer it failed with an error:
Azure.RequestFailedException: The underlying connection was closed: An unexpected error occurred on a send. —>
System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a send. —>
System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. —>
System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
I worked on the problem for quite a while (many hours off and on over several days). Then I tried using a different provider for the email service. The same exact error occurred.
Frustrated, I would set it aside for a month or so and worked on it again on some weekends. No luck.
People realized that the web site actually accepted entries even if it had not been announced. Then, a couple months ago, to my great surprise, I discovered that about 10% of the time the email did work. WHAT?!! Is it a timing issue? I did lots of experiments without success.
Yesterday, I started doing web searches with various search parameters to try and find the solution. Somewhat buried on one web page I found a suggestion, put this in your C# program anyplace before you attempt to make your connection with the other server:
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12;
Since the web page was several years old I first tried using Tls13, newer is better, right?
No luck.
So I tried the original suggestion, Tls12. This worked.
There was great joy in Boomershoot land!
I’m making the blog post about this so perhaps other people can find the answer more easily than I was able to.
The complete error (exception call stack) is below the fold.
Exception: System.AggregateException: Retry failed after 4 tries. Retry settings can be adjusted in ClientOptions.Retry or by configuring a custom retry policy in ClientOptions.RetryPolicy. —> Azure.RequestFailedException: The underlying connection was closed: An unexpected error occurred on a send. —> System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a send. —> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. —> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
— End of inner exception stack trace —
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
at System.Net.FixedSizeReader.ReadPacket(Byte[] buffer, Int32 offset, Int32 count)
at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ForceAuthentication(Boolean receiveFirst, Byte[] buffer, AsyncProtocolRequest asyncRequest, Boolean renegotiation)
at System.Net.Security.SslState.ProcessAuthentication(LazyAsyncResult lazyResult)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Net.TlsStream.ProcessAuthentication(LazyAsyncResult result)
at System.Net.TlsStream.Write(Byte[] buffer, Int32 offset, Int32 size)
at System.Net.PooledStream.Write(Byte[] buffer, Int32 offset, Int32 size)
at System.Net.ConnectStream.WriteHeaders(Boolean async)
— End of inner exception stack trace —
at System.Net.HttpWebRequest.GetRequestStream(TransportContext& context)
at System.Net.HttpWebRequest.GetRequestStream()
at Azure.Core.Pipeline.HttpWebRequestTransport.<ProcessInternal>d__8.MoveNext()
— End of inner exception stack trace —
at Azure.Core.Pipeline.HttpWebRequestTransport.<ProcessInternal>d__8.MoveNext()
— End of stack trace from previous location where exception was thrown —
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Azure.Core.Pipeline.HttpWebRequestTransport.Process(HttpMessage message)
at Azure.Core.Pipeline.HttpPipelineTransportPolicy.Process(HttpMessage message, ReadOnlyMemory`1 pipeline)
at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory`1 pipeline)
at Azure.Core.Pipeline.RequestActivityPolicy.ProcessNextAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
at Azure.Core.Pipeline.RequestActivityPolicy.Process(HttpMessage message, ReadOnlyMemory`1 pipeline)
at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory`1 pipeline)
at Azure.Core.Pipeline.ResponseBodyPolicy.<ProcessAsync>d__5.MoveNext()
— End of stack trace from previous location where exception was thrown —
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Azure.Core.Pipeline.ResponseBodyPolicy.Process(HttpMessage message, ReadOnlyMemory`1 pipeline)
at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory`1 pipeline)
at Azure.Core.Pipeline.LoggingPolicy.Process(HttpMessage message, ReadOnlyMemory`1 pipeline)
at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory`1 pipeline)
at Azure.Communication.Pipeline.HMACAuthenticationPolicy.Process(HttpMessage message, ReadOnlyMemory`1 pipeline)
at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory`1 pipeline)
at Azure.Core.Pipeline.RedirectPolicy.<ProcessAsync>d__7.MoveNext()
— End of stack trace from previous location where exception was thrown —
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Azure.Core.Pipeline.RedirectPolicy.Process(HttpMessage message, ReadOnlyMemory`1 pipeline)
at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory`1 pipeline)
at Azure.Core.Pipeline.RetryPolicy.<ProcessAsync>d__5.MoveNext()
— End of inner exception stack trace —
at Azure.Core.Pipeline.RetryPolicy.<ProcessAsync>d__5.MoveNext()
— End of stack trace from previous location where exception was thrown —
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Azure.Core.Pipeline.RetryPolicy.Process(HttpMessage message, ReadOnlyMemory`1 pipeline)
at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory`1 pipeline)
at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.Process(HttpMessage message, ReadOnlyMemory`1 pipeline)
at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory`1 pipeline)
at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.Process(HttpMessage message, ReadOnlyMemory`1 pipeline)
at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory`1 pipeline)
at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.Process(HttpMessage message, ReadOnlyMemory`1 pipeline)
at Azure.Core.Pipeline.HttpPipeline.Send(HttpMessage message, CancellationToken cancellationToken)
at Azure.Communication.Email.EmailRestClient.Send(EmailMessage message, Nullable`1 operationId, CancellationToken cancellationToken)
at Azure.Communication.Email.EmailClient.SendEmailInternal(WaitUntil wait, EmailMessage message, CancellationToken cancellationToken)
at Azure.Communication.Email.EmailClient.Send(WaitUntil wait, EmailMessage message, CancellationToken cancellationToken)
at FlashTek.BoomershootEntry.Lib.Email.SendNow(String hostName)
at FlashTek.BoomershootEntry.Lib.MailSender.SendEmail()
Damn Russians. It’s always the Russians. Unless it’s the Chinese. Or the Brady Bunch.
Don’t know if it’s pertinent:
There exists MITM exploits such as “tcprst” and “tcpkill” that can interfere with the TCP/IP protocol to make it appear as though the remote side has terminated the connection.
The old axiom that error messages should be helpful is ignored when related to security.
Hopefully someday simply dropping connections will forever go down as a serious mistaken belief in security by obscurity.
“Security by Obscurity”
That’s my method for making a cheat sheet of websites, my various user ID’s and my passwords. I read how someone formerly in some American TLA (three letter agency) recorded her user ID’s and passwords that way.
I had to do this in one of my programs.
Yeah, a lot of security changes over the years have required such settings to allow for fallback. Sites are set up now to reject older protocols outright unless the clients are configured to negotiate explicitly.
Good on you for figuring it out. E-mail arrived, by the way. I was one of those early entrants who just “knew” Boomershoot opened for entry in November. I didn’t think anything of it and no one complained. Being old hat at this, getting access to the entry wasn’t an issue even without the e-mail.