How to resolve Sporadic 503 UC Errors in Istio

Open-source Istio service mesh is a powerful tool for ensuring resiliency and improving the reliability of your multicloud and multicluster infrastructure. Some issues can cause SREs and the DevOps team to search for solutions incessantly. That’s why we at the IMESH team would start to post about various topics and their resolutions to help adopt the Istio service mesh.  

Issue Summary wrt Istio 

Recently, we encountered one of our clients’ instances where their SREs observed sporadic 503 UC Errors on the Istio sidecar. This issue persisted for all the inbound requests across environments such as production, integration, and staging.

The team encountered the following symptoms: 

• 503 UC Errors: The upstream connection terminated before the response started.
• High-load Errors: Errors occur intermittently across multiple services, primarily under high-load conditions.

Once the IMESH team was notified, we quickly acknowledged and started diagnosing the issue. 

Root Cause Analysis of 503 UC Errors

We analyzed application logs, proxy logs, and tcpdump of the loopback interface to identify the root cause accurately. 

The issue stems from reusing expired or inactive HTTP connections by the Istio proxy when communicating with upstream services.

Below is a quick technical breakdown of our observation.

1. Istio Proxy Behaviour:
   1. Istio proxies establish multiple TCP connections to upstream services.
   2. These connections are reused for various requests to enhance performance and reduce connection overhead.
   3. By default, Istio proxies maintain these connections for up to 1 hour.
2. Upstream HTTP Keep-Alive Timeout:
   1. Upstream services enforce their HTTP keep-alive timeout, often configured to shorter durations (e.g., 5 seconds) to close idle connections and free up resources.
   2. After the keep-alive timeout elapses, the upstream service sends a FIN packet to close the connection. We can observe this in the below tcpdump.

Note: HTTP Keep-alive timeout: 10s

3. Reuse of Expired Connections:
   1. Istio sidecar may attempt to reuse these expired idle connections for new requests, unaware that the connection is no longer.
4. Race Condition Between FIN and New Requests:
   1. A race condition arises when:
      1. The FIN packet from the upstream service (indicating connection closure) is delayed(or in process) or not yet processed by the proxy.
      2. The Istio proxy sends a new request on the same expired connection.
   2. In this scenario, the upstream service rejects the new request with RST packet, terminating the connection and leading to a 503 UC error.
   3. We can observe the race condition in below tcpdump.

Note: HTTP Keep-alive timeout: 5s

The impact of such configurations is:

1. Increased error rates (503 UC errors in Istio), particularly during high load or traffic spikes.
2. Reduced reliability of services, with potential degradation in the overall user experience.
3. Operational overhead due to debugging and incident handling.

Proposed Solutions by IMESH

1. Connection Timeout Synchronization:
   1. Reduce the Istio proxy’s default connection timeout (currently 1 hour) to align with upstream services’ HTTP keep-alive timeout (e.g., 60 seconds).
   2. This ensures the proxy proactively closes connections before the upstream server does. 

OR

3. Increase application HTTP keep-alive timeout (e.g., 60 seconds or higher). The higher the HTTP keep-alive value, the lower the chances of idle connection timeout expiry.
4. Note: The above two can be combined to eliminate race conditions.

2. Disable Keep-Alive
   1. Altogether disable HTTP KeepAlive (tested with Sanic), causing the application to close the connection after each request. This may degrade performance, as a new connection will be established for every request.
3. Error Handling and Retries:
   1. Enhance Istio retry policies to gracefully retry failed requests caused by connection reuse issues with the help of Envoy Filter. This poses risk of idempotency issues, however, depending on the particular server framework.
   2. Implement circuit breakers to reduce the load on problematic upstream services during traffic spikes.
4. Monitoring Enhancements: 
   1. Set up alerts to monitor for spikes in 503 UC errors and respond promptly.

Hassle-free Istio support and maintenance by IMESH

As you adopt open-source Istio more in an enterprise setup, you may encounter some issues or bugs. The Istio community is already doing a fantastic job creating new features and solving bugs and issues. However, there can be instances in which problems are critical and can disrupt services; feel free to contact us for dedicated and quick help for enterprise Istio support.