bug: traffic-split err

[xianshun163]

Issue description

the traffic-split make some error, as the error log show

------ some times, the weighted_upstreams value is differrent， if it is like the second， it will not make err

1、weighted_upstreams:[{"upstream_id":"382981255605519758","weight":2},{"upstream":"plugin#upstream#is#empty","weight":1}],
2、weighted_upstreams:[{"upstream_id":"382981255605519758","weight":2},{"weight":1}]

if it like the first， the follow code will make the error:
210 upstream_obj.upstream.vid = i
I have add some log to the source code, you will see it at the error log area

Environment

• apisix version (cmd: apisix version): 2.10.1
• OS (cmd: uname -a): CentOS Linux release 7.6.1810 (Core)
• OpenResty / Nginx version (cmd: nginx -V or openresty -V):nginx version: openresty/1.19.9.1
• etcd version, if have (cmd: run curl http://127.0.0.1:9090/v1/server_info to get the info from server-info API):"etcd_version":"3.4.0",
• apisix-dashboard version, if have: 2.10.1
• the plugin runner version, if the issue is about a plugin runner (cmd: depended on the kind of runner):
• luarocks version, if the issue is about installation (cmd: luarocks --version):

Steps to reproduce

1、I create a rule like this: the upstream is a simple java app server
curl http://127.0.0.1:9080/apisix/admin/routes/xs-app-r1 -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
{
"uri": "/index",
"name": "蓝绿发布",
"host": "local.xsapp.org",
"plugins": {
"traffic-split": {
"rules": [
{
"match": [
{
"vars": [
["http_release","==","new_release"]
]
}
],
"weighted_upstreams": [
{
"upstream_id": "382981255605519758",
"weight": 2
},
{
"weight": 1
}
]
}
]
}
},
"upstream_id": "382981468860712334"
}'

2、I open a global plugin key-auth in the dashboard

3、I test it like this : curl http://127.0.0.1:9080/index -H 'release: new_release' -H 'Host: local.xsapp.org' -H "apikey: key-of-sam"

4、sometimes it work ， some time it make a error， as the error log show。

5、And then I go to print some error log in the lua file, you can see it at the error log.

Actual result

as the error log show

Error log

2021/11/25 14:03:47 [warn] 24989#24989: *39085296 [lua] lrucache.lua:92: lrucache(): ----xs---begin to create_obj_fun---,key:table: 0x7f76788384f8, client: 127.0.0.1, server: _, request: "GET /index HTTP/1.1", host: "local.xsapp.org"
2021/11/25 14:03:47 [error] 24989#24989: *39085296 [lua] traffic-split.lua:202: create_obj_fun(): ---xs-- upstream_obj:{"upstream_id":"382981255605519758","weight":2}, client: 127.0.0.1, server: _, request: "GET /index HTTP/1.1", host: "local.xsapp.org"
2021/11/25 14:03:47 [error] 24989#24989: *39085296 [lua] traffic-split.lua:205: create_obj_fun(): -----xs--- upstream_id:382981255605519758, client: 127.0.0.1, server: _, request: "GET /index HTTP/1.1", host: "local.xsapp.org"
2021/11/25 14:03:47 [error] 24989#24989: *39085296 [lua] traffic-split.lua:202: create_obj_fun(): ---xs-- upstream_obj:{"upstream":"plugin#upstream#is#empty","weight":1}, client: 127.0.0.1, server: _, request: "GET /index HTTP/1.1", host: "local.xsapp.org"
2021/11/25 14:03:47 [error] 24989#24989: *39085296 [lua] traffic-split.lua:209: create_obj_fun(): ----xs--- upstream is not null:plugin#upstream#is#empty, client: 127.0.0.1, server: _, request: "GET /index HTTP/1.1", host: "local.xsapp.org"
2021/11/25 14:03:47 [error] 24989#24989: *39085296 lua entry thread aborted: runtime error: /data/apisix-2.10.1/apisix/plugins/traffic-split.lua:210: attempt to index field 'upstream' (a string value)
stack traceback:
coroutine 0:
/data/apisix-2.10.1/apisix/plugins/traffic-split.lua: in function 'create_obj_fun'
/data/apisix-2.10.1/apisix/core/lrucache.lua:93: in function 'lrucache'
/data/apisix-2.10.1/apisix/plugins/traffic-split.lua:268: in function 'phase_func'
/data/apisix-2.10.1/apisix/plugin.lua:683: in function 'run_plugin'
/data/apisix-2.10.1/apisix/init.lua:465: in function 'http_access_phase'
access_by_lua(nginx.conf:264):2: in main chunk, client: 127.0.0.1, server: _, request: "GET /index HTTP/1.1", host: "local.xsapp.org"

--------- this is my traffic-split.lua
198 local function new_rr_obj(weighted_upstreams)
199 local server_list = {}
200 -- core.log.error("---xs--upstream_obj:",core.json.delay_encode(weighted_upstreams,true))
201 for i, upstream_obj in ipairs(weighted_upstreams) do
202 core.log.error("---xs-- upstream_obj:",core.json.delay_encode(upstream_obj,true))
203
204 if upstream_obj.upstream_id then
205 core.log.error("-----xs--- upstream_id:",upstream_obj.upstream_id)
206 server_list[upstream_obj.upstream_id] = upstream_obj.weight
207 elseif upstream_obj.upstream then
208 -- Add a virtual id field to uniquely identify the upstream key.
209 core.log.error("----xs--- upstream is not null:",tostring(upstream_obj.upstream))
210 upstream_obj.upstream.vid = i
211 server_list[upstream_obj.upstream] = upstream_obj.weight
212 else
213 -- If the upstream object has only the weight value, it means
214 -- that the upstream weight value on the default route has been reached.
215 -- Mark empty upstream services in the plugin.
216 core.log.error("----xs--- all null")
217 upstream_obj.upstream = "plugin#upstream#is#empty"
218 server_list[upstream_obj.upstream] = upstream_obj.weight
219
220 end
221 end
222
223 return roundrobin:new(server_list)
224 end

------ some times, the weighted_upstreams value is differrent， if it is like the second， it will not make err
1、weighted_upstreams:[{"upstream_id":"382981255605519758","weight":2},{"upstream":"plugin#upstream#is#empty","weight":1}],
2、weighted_upstreams:[{"upstream_id":"382981255605519758","weight":2},{"weight":1}]
if it like the first， the follow code will make the error:
210 upstream_obj.upstream.vid = i

Expected result

do not make error

[tzssangglass]

can you test this on master branch? we have solved a bug releated to this:https://github.com/apache/apisix/pull/5414/files

[xianshun163]

I will do it later

[xianshun163]

can you test this on master branch? we have solved a bug releated to this:https://github.com/apache/apisix/pull/5414/files
I have found the bug fix you tell me at the changelog, it is fix as 2.10.2 version. Maybe I only change the traffic-split.lua use the new one.

[tzssangglass]

I cannot reproduce your error on the master branch, here are the steps I took to reproduce it

1. add upstream

curl --location --request PUT 'http://127.0.0.1:9080/apisix/admin/upstreams/100' \
--header 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' \
--header 'Content-Type: application/json' \
--data-raw '{
    "type":"roundrobin",
    "nodes":{
        "httpbin.org:80": 1
    }
}'

2. add route

curl --location --request PUT 'http://127.0.0.1:9080/apisix/admin/routes/1' \
--header 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' \
--header 'Content-Type: application/json' \
--data-raw '{
    "plugins": {
        "traffic-split": {
            "rules": [
                {
                    "match": [
                        {
                            "vars": [
                                [
                                    "http_release",
                                    "==",
                                    "new_release"
                                ]
                            ]
                        }
                    ],
                    "weighted_upstreams": [
                        {
                            "upstream_id": "100",
                            "weight": 2
                        },
                        {
                            "weight": 1
                        }
                    ]
                }
            ]
        }
    },
    "upstream": {
        "nodes": {
            "127.0.0.1:1980": 1
        },
        "type": "roundrobin"
    },
    "uri": "/*"
}'

3. my backend 1980 is https://github.com/apache/apisix/blob/master/benchmark/server/conf/nginx.conf

4. test

curl http://127.0.0.1:9080/get -H 'release: new_release'

5. no attempt to index field 'upstream' (a string value) in logs/error.log

[tzssangglass]

is your upstream 382981468860712334 or 382981255605519758 exist?

[tzssangglass]

the upstream is 382981255605519758 looks strange, what have you done anything with this upstream?

[xianshun163]

the upstreamid are exist, they are create by the apisix auto.

I can't reproduce the issue now，

[jujiale]

I also suffered such a situation. but I can not reproduce it. the error log is :
`

/traffic-split.lua:209: attempt to index field 'upstream' (a string value)

`

according the source code. when the traffic-split plugin config "weighted_upstreams" . it's element has no property "upstream". but have property "upstream_id", it should not invode the follwing code (line 209, because elseif upstream_obj.upstream then return false )

so I confused why line 209 is invoked.

traffic_split config is below:
`

{
"traffic-split": {
	"disable": false,
	"rules": [{
		"weighted_upstreams": [{
			"weight": 0
		}, {
			"upstream_id": "466807194923303750",
			"weight": 100
		}, {
			"upstream_id": "466807194721978186",
			"weight": 0
		}]
	}]
}

}

`

[nic-chen]

fixed by #11932