Routing Priority Mixup in Kong 3.4

[benhamill]

Is there an existing issue for this?

• I have searched the existing issues

Kong version ($ kong version)

3.4.2

Current Behavior

I have encountered what I think is a bug in service/routing prioritization when using regex routes with explicit priorities set.

With this minimal kong.yaml and DB mode off:

---
_format_version: '3.0'
services:
- name: generic-webhook
  url: http://example.com
  routes:
  - name: generic-webhook-production-us
    paths:
    - "~/webhook/([a-zA-Z0-9-]+)"
    # headers:
    #   Environment-Region:
    #   - production-us
    strip_path: false
    regex_priority: 1

- name: specific-webhook
  url: http://example.com/api
  routes:
  - name: specific-webhook-production-us
    paths:
    - "~/(?i)webhook/foo/bar/?$"
    # headers:
    #   Environment-Region:
    #   - production-us
    strip_path: false
    regex_priority: 2

When the headers section are commented out, requests with the path /webhook/foo/bar route to the specific webhook service, but with the headers sections present, it always routes to the generic webhook service, despite the specific one having the higher regex priority.

Here is the kong.conf we are using:

# nginx
nginx_daemon=off
nginx_worker_processes=2
proxy_listen = 0.0.0.0:8000 reuseport backlog=16384
admin_listen = 0.0.0.0:8001 reuseport backlog=16384

# this allows kong.client.get_forwarded_ip have the original client's ip
# instead of the load balancer's.
trusted_ips = 10.0.0.0/8,172.16.0.0/12,192.168.0.0/16
real_ip_header = X-Forwarded-For

# logging
log_level = debug
proxy_access_log=/dev/stdout
admin_access_log=/dev/stdout
status_access_log=/dev/stdout
proxy_error_log=/dev/stdout
admin_error_log=/dev/stdout
status_error_log=/dev/stdout


# disable all headers
headers = off


# database
database=off


# cache
mem_cache_size = 2048m


# plugins
plugins = bundled


# paths
prefix = /usr/local/kong/

allow_debug_header = true

Expected Behavior

I expect two services with regex paths to be selected based on the higher regex priority.

Steps To Reproduce

1. Place the kong.yaml and kong.conf files in the appropriate places (see mounting below).
2. Uncomment the header sections of both routes in the provided kong.yaml.
3. Run Kong 3.4 from the base Kong Ubuntu Docker image:

   docker run \
   --publish 8000:8000 \
   --volume "$PWD/etc/kong":/etc/kong \
   --volume "$PWD/kong/declarative":/usr/local/kong/declarative \
   --env KONG_DECLARATIVE_CONFIG='/usr/local/kong/declarative/kong.yaml' \
   kong:3.4-ubuntu \
   kong start
   

4. Make a request:

   curl -i \
   -X POST \
   -H "Environment-Region:production-us" \
   -H "Kong-Debug:1" \
   http://localhost:8000/webhook/foo/bar
   

5. See a response header of Kong-Service-Name: generic-webhook.
6. Stop your kong docker container.
7. Comment out or delete the header section from the two routes in the provided kong.yaml.
8. Rerun your kong docker container with the command in step 3.
9. Make the same request from step 4.
10. See a response header of Kong-Service-Name: specific-webhook.

Anything else?

We have a custom plugin that sets the environment-region header based on authenticated data in the request during the rewrite-phase and use it to route between different deployments of services, but I have eliminated those parts of our use of Kong in the interest of a minimally reproducible setup. Just not having that header in our routes would constitute a giant change in our infrastructure, so I'd prefer to understand why having the header requirements is causing the generic route to "win" the priority race, rather than hear "just don't have the header".

[Oyami-Srk]

This is a known bug for the router priority. Could you please try switching to traditional flavor instead of the default traditional_compatible? Router Flavor.

[outsinre]

Another solution is to keep the router flavour, but use a larger regex priority (e.g. 8 vs 16).

[benhamill]

Oh wow. Thanks, folks! Is there an existing issue about this that I should link to in a comment here in case it helps others find it and/or know it's relevance when I did not earlier?

[Oyami-Srk]

We have an internal ticket for this. I think we can use this issue (#14457) for acknowledgment to other users. Thanks again for your very detailed report! 🚀