Tenant API hostname resolves to stale/unreachable IPs causing intermittent connect timeouts: implement retry logic instead of IP pinning

Android

When making API calls to your Esper tenant endpoint, roughly half of all requests fail with connection timeouts while others succeed normally.

Why this happens

Your Esper tenant API hostname (for example, krntxg-api.esper.cloud) uses DNS-based load balancing with a 60-second TTL, meaning the active IP addresses rotate frequently. If your API client caches a resolved IP address and reuses it across requests or long-lived sessions, it will eventually try to connect to a retired IP that is no longer active. Because only some IPs are retired at any given time, roughly half of requests fail while the rest succeed.

Before you begin

Confirm that stale DNS resolution is the cause before making changes. Use a DNS lookup tool that does not cache results to check the currently active IP addresses for your tenant hostname:

• DNS Checker — enter your full tenant hostname (for example, krntxg-api.esper.cloud)
• Dig Web Interface — query your tenant hostname using nameserver 9.9.9.10

If the tool returns different IP addresses on repeated lookups, DNS rotation is active and IP pinning will cause intermittent failures.

Steps

1. Remove any IP address pinning from your client configuration. Hard-coded or pinned IPs become unreachable as the load balancer rotates addresses. Remove all of the following if present:
   • --add-host flags in container configurations that map your tenant hostname to a static IP
   • Static entries in /etc/hosts that override your tenant hostname
   • Hard-coded IP addresses in integration scripts or runbooks
   Always reference the hostname directly (for example, krntxg-api.esper.cloud) rather than any resolved IP address.
2. Configure your API client to re-resolve the hostname on every new connection. Do not reuse a resolved IP address across multiple requests or across long-lived sessions. Allow your system DNS resolver to query the hostname fresh on each connection attempt. Consult your HTTP client library's documentation for the setting that controls DNS caching or connection reuse — common options include disabling connection pooling per-host or setting a maximum connection age aligned to the 60-second TTL.
3. Add retry logic that triggers a fresh DNS lookup on TCP connection timeout. When a connection attempt fails, retrying the request causes your client to re-resolve the hostname, which will return a currently active IP. A strategy of 2–3 retry attempts with a short delay (for example, 500 ms) is sufficient to recover from the IP rotation window without adding meaningful latency to successful requests.

Verify: After deploying these changes, monitor your API call success rate over at least 10 minutes of normal traffic. Requests that previously timed out should now self-recover on the first or second retry with no manual intervention required. Sustained timeout rates above 5% indicate the issue has not been fully resolved.

If this doesn't resolve it

Two additional causes can produce the same symptom after IP pinning is removed:

• DNS resolver ignoring TTL: Some corporate or cloud DNS resolvers enforce a minimum cache TTL (for example, 300 seconds) that overrides the 60-second TTL on the Esper hostname. If you suspect this, configure your resolver to honor low TTLs, or route your API hostname lookups through a public resolver such as 9.9.9.10 that respects the published TTL.
• Network egress filtering: If your environment restricts outbound traffic by destination IP, newly rotated IPs may be blocked at the network layer before your client can connect. Work with your network team to confirm that all IP addresses returned for your tenant hostname are reachable from your environment, and switch to hostname-based egress rules rather than IP-based allowlists.

When contacting Esper Support, collect the following before reaching out:

• The full tenant hostname you are querying
• Output from at least three consecutive DNS lookups of your tenant hostname, taken 60 seconds apart
• Your HTTP client library, language, and the connection/DNS settings currently in use
• A sample of failed request logs showing the destination IP and the exact timeout error

Contact Esper Support at support.esper.io with the above information.

--- **Improved title:** API calls to Esper tenant endpoint timing out — disable IP pinning and add retries

Still need help?

If you're still experiencing connection timeouts to the Tenant API after implementing retry logic, please submit a support ticket and include your application logs showing the timeout errors, the specific API endpoints you're calling, and details about your retry configuration.