yhzhu的随手记
0%

Troubleshooting APISIX OAuth2 Redirect Port Issue at Gateway Layer

发表于 分类于 ，
A detailed walkthrough of troubleshooting OAuth2 redirect URLs with incorrect ports in a multi-layer proxy architecture using APISIX, and the complete solution via configuring trusted_addresses.

🌐 Language: English Version | 中文版

TL;DR

Problem: OAuth2 redirect URL contains incorrect port http://example.com:9080
Root Cause: APISIX trusted_addresses configuration too restrictive, causing X-Forwarded-* headers to be overwritten
Solution: Add upstream proxy network segments to trusted_addresses

Problem Background

In a typical multi-layer proxy architecture:

flowchart TD
    Browser[Browser
HTTPS:443] --> Nginx[Nginx
TLS Termination]
    Nginx --> APISIX[APISIX
Docker:9080]
    APISIX --> OAuth2[Spring OAuth2
Authorization Server]

Users access the system via https://public.example.com. When not logged in, accessing the OAuth2 authorization endpoint /oauth2/authorize should theoretically return:

1
2
HTTP/1.1 302 Found
Location: https://public.example.com/login

But the actual response was:

1
2
HTTP/1.1 302 Found
Location: http://public.example.com:9080/login

When the browser accesses https://public.example.com:9080/login, it throws ERR_CONNECTION_CLOSED, preventing users from logging in.

Problem Analysis

Browser Network Request Chain

1
2
3
GET https://public.example.com/oauth2/authorize → 302
GET http://public.example.com:9080/login → 307 Internal Redirect
GET https://public.example.com:9080/login → FAILED

Key Application Log Information

1
2
3
4
5
requestUrl=http://public.example.com:9080/oauth2/authorize
scheme=http serverName=public.example.com serverPort=9080
Saved request http://public.example.com:9080/oauth2/authorize?...
Redirecting to /login
location=http://public.example.com:9080/login

From the logs, we can see Spring Security already sees the wrong request view:

  • scheme recognized as http
  • port recognized as 9080
  • SavedRequest URL is corrupted

Troubleshooting Directions

Initially, we suspected several directions:

  1. Spring Security’s SavedRequest logic had issues
  2. APISIX’s proxy-rewrite plugin configuration was incorrect
  3. Application didn’t properly enable forward-headers-strategy
  4. Entry Nginx wasn’t passing X-Forwarded-* headers

These hypotheses could explain some symptoms, but none were the root cause.

Troubleshooting Timeline

Phase Hypothesis Verification Method Conclusion
1 Spring Security issue Check SavedRequest logic ❌ Ruled out
2 proxy-rewrite config Compare plugin configs ❌ Ruled out
3 forward-headers-strategy Check app configuration ❌ Ruled out
4 Upstream Nginx config Check X-Forwarded-* passing ❌ Ruled out
5 APISIX log diagnostics Add diagnostic logs ✅ Root cause identified

Converging on APISIX

The real breakthrough came after adding diagnostic logs to APISIX, revealing:

1
2
3
4
5
6
7
8
9
{
  "host": "public.example.com",
  "server_port": "9080",
  "http_x_forwarded_host": "public.example.com",
  "http_x_forwarded_proto": "http",
  "http_x_forwarded_port": "9080",
  "sent_http_location": "http://public.example.com:9080/login",
  "upstream_http_location": "http://public.example.com:9080/login"
}

Key findings:

  • The incorrect http/9080 was already formed inside APISIX
  • It wasn’t the application “breaking it again” during the return phase
  • The wrong request view the application received was passed in by the gateway itself

Easily Confused Points

During troubleshooting, the most confusing aspect was mixing “user real IP” chain with “public origin” chain. These are not the same thing:

User Real IP Chain

Used for risk control, auditing, rate limiting, and log location, typically relying on:

  • X-Real-IP
  • X-Forwarded-For

Public Origin Chain

Used for OAuth2 redirects, absolute URL generation, and callback address validation, typically relying on:

  • X-Forwarded-Proto
  • X-Forwarded-Host
  • X-Forwarded-Port

The essence of this problem wasn’t “user IP not being passed through,” but APISIX corrupting the second chain after trust judgment failed.

APISIX Source Code Analysis

The problem ultimately converged on the trusted_addresses configuration.

APISIX request processing logic:

  1. Calculate previous hop address (usually via X-Real-IP or Remote Addr)
  2. Match this address against apisix.trusted_addresses
  3. If not trusted, overwrite:
    • X-Forwarded-Proto
    • X-Forwarded-Host
    • X-Forwarded-Port
  4. Overwrite values are not the external public entry, but what APISIX observes:
    • scheme
    • host
    • server_port

Pseudocode representation:

1
2
3
4
5
6
7
8
9
10
11
12
13
-- Check if previous hop address is in trusted list
local addr_is_trusted = trusted_addresses_util.is_trusted(api_ctx.var.realip_remote_addr)

if not addr_is_trusted then
    -- When not trusted, overwrite X-Forwarded-* headers with gateway's own view
    local proto = api_ctx.var.scheme        -- actual value: http
    local host = api_ctx.var.host            -- actual value: public.example.com
    local port = api_ctx.var.server_port     -- actual value: 9080

    core.request.set_header(api_ctx, "X-Forwarded-Proto", proto)
    core.request.set_header(api_ctx, "X-Forwarded-Host", host)
    core.request.set_header(api_ctx, "X-Forwarded-Port", port)
end

This explains why the application kept seeing http and 9080:

  • Once APISIX judges the upstream proxy as “not trusted”
  • It downgrades external origin information to its own listening view

For more configuration details, refer to APISIX Official Documentation - trusted_addresses

Solution

Quick Verification

The simplest control variable experiment:

1
2
3
4
apisix:
  trusted_addresses:
    - 0.0.0.0/0
    - ::/0

After reloading, verify the OAuth2 authorization endpoint:

1
Location: https://public.example.com/login

The problem disappears immediately. This experiment itself isn’t the final configuration, but it pins down the root cause: the problem is at the APISIX trusted_addresses layer.

Production Configuration

More appropriate approach for production:

1
2
3
4
5
6
7
8
apisix:
  node_listen: 9080
  enable_ipv6: false
  trusted_addresses:
    - 127.0.0.1           # Local loopback
    - "::1"               # IPv6 loopback
    - 172.17.21.0/24     # Docker network (e.g., Docker Desktop)
    - 192.168.10.0/24     # Internal network (where upstream proxy resides)

With upstream Nginx configuration:

1
2
3
4
5
6
7
8
9
10
location / {
    proxy_pass http://apisix;

    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-Host $host;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header X-Forwarded-Port $server_port;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}

Why Application Layer Patches Aren’t the Root Solution

During troubleshooting, several workarounds were attempted at the application layer:

  • Custom RequestCache, forcibly changing SavedRequest to public address
  • Custom login redirect URL generation
  • Wrapping HttpServletRequest in filters to override scheme/host/port

These methods can alleviate symptoms in the short term, but aren’t suitable as formal solutions:

  1. The problem won’t appear in just one service
  2. Absolute URL generation doesn’t only occur in OAuth2 login flows
  3. Future Swagger, email links, frontend API base URLs, callback addresses, and download links may encounter similar issues

If the public access view isn’t uniformly governed at the gateway layer, it will eventually become patchwork for each service.

Differences Between Dev and Production Environments

Another reality of this problem: development and production network topologies differ.

Common dev environment chain:

1
Host Nginx → Docker Desktop port mapping → APISIX container

More common production pattern:

1
Nginx node → Internal network → APISIX node

In both environments, the “previous hop address” that APISIX actually sees can be completely different.

This causes a phenomenon:

  • In dev environments, using a very narrow trusted_addresses may not stably reproduce production behavior
  • Because the address object APISIX uses for trust judgment may not be the expected upstream proxy address

This is why in some local environments, temporarily widening trusted_addresses immediately restores functionality, while continuing to guess CIDRs can lead to endless loops.

Applicable Scenarios

This record applies to similar scenarios:

  • APISIX deployed behind a reverse proxy
  • Public unified domain and TLS termination at upstream layer
  • Downstream applications generate absolute URLs based on scheme/host/port
  • OAuth2 / OIDC login flows sensitive to URL view consistency

If any of the following phenomena appear in the chain:

  • 302 Location contains internal port
  • SavedRequest shows internal addresses or container ports
  • Frontend callbacks or logout addresses mix in internal ports
  • Mixed Content points to internal HTTP services

It’s worth checking if APISIX is overwriting X-Forwarded-*.

Quick Checklist

When encountering similar issues, check in order:

  • Is server_port in APISIX logs showing the internal port
  • Does trusted_addresses include upstream proxy IPs
  • Is upstream Nginx properly setting X-Forwarded-* headers
  • Is application enabled forward-headers-strategy
  • Is OAuth2 redirect URL using internal addresses

For more on Forwarded Headers configuration, refer to Spring Security Official Documentation.

Summary

This problem isn’t essentially an OAuth2 protocol issue, nor is it Spring Security’s special behavior.

It’s more like a typical gateway trust boundary problem:

  • External entry is HTTPS
  • Gateway internal is HTTP 9080
  • If the gateway doesn’t stably preserve the public access view
  • Downstream services will naturally expose internal ports in absolute URLs

And APISIX’s trusted_addresses is the key switch in this chain.

The problem wasn’t solved by a single “magic configuration,” but through a complete convergence:

  • First admitting application patches aren’t the root solution
  • Then converging the problem to the gateway
  • Then using source code and control variables to pin down the root cause
  • Finally distinguishing dev environment temporary recovery from production formal configuration principles

If the runtime chain contains both upstream proxies, gateways, and applications that need to generate absolute URLs, this type of problem is worth prioritizing from the trust boundary rather than business code.

These solutions can help maintain trusted_addresses correctness in more complex environments, avoiding repeated issues due to IP changes.