Wrong record test fixture for clents/library.yml #7633

sanchezzzhak · 2024-03-21T13:41:40Z

https://github.com/matomo-org/device-detector/blob/c9952fc4d31dd249d2bf7dc88ca9963c7a2739b7/Tests/Parser/Client/fixtures/library.yml#L621C1-L643C18

In some parsers, this is perceived as a float/number and not as a string

full fix

---
-
  user_agent: Wget/1.10+devel
  client:
    type: library
    name: Wget
    version: "1.10"
-
  user_agent: Wget/1.11.4 Red Hat modified
  client:
    type: library
    name: Wget
    version: "1.11.4"
-
  user_agent: Wget/ (linux-gnu)
  client:
    type: library
    name: Wget
    version:
-
  user_agent: curl/7.21.0 (i386-redhat-linux-gnu) libcurl/7.21.0 NSS/3.12.10.0 zlib/1.2.5 libidn/1.18 libssh2/1.2.4
  client:
    type: library
    name: curl
    version: "7.21.0"
-
  user_agent: PycURL/7.19.3.1 libcurl/7.26.0 GnuTLS/2.12.20 zlib/1.2.7 libidn/1.25 libssh2/1.4.2 librtmp/2.3
  client:
    type: library
    name: curl
    version: "7.26.0"
-
  user_agent: python-requests/1.2.0 CPython/2.7.3 Linux/3.8.0-33-generic
  client:
    type: library
    name: Python Requests
    version: "1.2.0"
-
  user_agent: python-requests/1.2.0 CPython/2.7.5 Windows/7
  client:
    type: library
    name: Python Requests
    version: "1.2.0"
-
  user_agent: Python-urllib/2.6
  client:
    type: library
    name: Python urllib
    version: "2.6"
-
  user_agent: Mozilla/5.0 (Python-urllib2)
  client:
    type: library
    name: Python urllib
    version:
-
  user_agent: Java/1.7.0_51
  client:
    type: library
    name: Java
    version: "1.7.0"
-
  user_agent: Java1.1.4
  client:
    type: library
    name: Java
    version: "1.1.4"
-
  user_agent: libwww-perl/5.69
  client:
    type: library
    name: Perl
    version: "5.69"
-
  user_agent: perlclient/1.0
  client:
    type: library
    name: Perl
    version: "1.0"
-
  user_agent: Guzzle/3.9.3 curl/7.38.0 PHP/5.6.14-0+deb8u1
  client:
    type: library
    name: Guzzle (PHP HTTP Client)
    version: "3.9.3"
-
  user_agent: HTTP_Request2/2.3.0 (http://pear.php.net/package/http_request2) PHP/5.3.3
  client:
    type: library
    name: HTTP_Request2
    version: "2.3.0"
-
  user_agent: Mechanize/2.7.3 Ruby/1.9.3p551 (https://github.com/sparklemotion/mechanize/)
  client:
    type: library
    name: Mechanize
    version: "2.7.3"
-
  user_agent: Python/3.5 aiohttp/1.0.5
  client:
    type: library
    name: aiohttp
    version: "1.0.5"
-
  user_agent: Google-HTTP-Java-Client/1.17.0-rc (gzip)
  client:
    type: library
    name: Google HTTP Java Client
    version: "1.17.0-rc"
-
  user_agent: WWW-Mechanize/1.73
  client:
    type: library
    name: WWW-Mechanize
    version: "1.73"
-
  user_agent: Faraday v0.9.1
  client:
    type: library
    name: Faraday
    version: "0.9.1"
-
  user_agent: Go-http-client/1.1
  client:
    type: library
    name: Go-http-client
    version: "1.1"
-
  user_agent: Go-http-client/2.0
  client:
    type: library
    name: Go-http-client
    version: "2.0"
-
  user_agent: Go 1.1 package http
  client:
    type: library
    name: Go-http-client
    version: "1.1"
-
  user_agent: urlgrabber/3.9.1 yum/3.2.29
  client:
    type: library
    name: urlgrabber (yum)
    version: "3.9.1"
-
  user_agent: urlgrabber/3.10 yum/3.4.3
  client:
    type: library
    name: urlgrabber (yum)
    version: "3.10"
-
  user_agent: libdnf/0.11.1
  client:
    type: library
    name: libdnf
    version: "0.11.1"
-
  user_agent: HTTPie/1.0.2
  client:
    type: library
    name: HTTPie
    version: "1.0.2"
-
  user_agent: rest-client/2.0.2 (linux-gnu x86_64) ruby/2.5.3p105
  client:
    type: library
    name: REST Client for Ruby
    version: "2.0.2"
-
  user_agent: RestSharp/106.2.1.0
  client:
    type: library
    name: RestSharp
    version: "106.2.1.0"
-
  user_agent: scalaj-http/2.4.0
  client:
    type: library
    name: ScalaJ HTTP
    version: "2.4.0"
-
  user_agent: REST::Client/273
  client:
    type: library
    name: Perl REST::Client
    version: "273"
-
  user_agent: node-fetch/1.0 (+https://github.com/bitinn/node-fetch)
  client:
    type: library
    name: Node Fetch
    version: "1.0"
-
  user_agent: ReactorNetty/0.9.10.RELEASE
  client:
    type: library
    name: ReactorNetty
    version: "0.9.10"
-
  user_agent: PostmanRuntime/7.9.1
  client:
    type: library
    name: Postman Desktop
    version: "7.9.1"
-
  user_agent: insomnia/2022.6.0
  client:
    type: library
    name: Insomnia REST Client
    version: "2022.6.0"
-
  user_agent: Jakarta Commons-HttpClient/3.1
  client:
    type: library
    name: Jakarta Commons HttpClient
    version: "3.1"
-
  user_agent: Mozilla/4.0 (compatible; Win32; WinHttp.WinHttpRequest.5)
  client:
    type: library
    name: WinHttp WinHttpRequest
    version: "5"
-
  user_agent: Embarcadero URI Client/1.0
  client:
    type: library
    name: Embarcadero URI Client
    version: "1.0"
-
  user_agent: Mikrotik/6.x Fetch
  client:
    type: library
    name: Mikrotik Fetch
    version: "6"
-
  user_agent: akka-http/10.1.11
  client:
    type: library
    name: Akka HTTP
    version: "10.1.11"
-
  user_agent: BTWebClient/2020(19648)
  client:
    type: library
    name: uTorrent
    version:
-
  user_agent: aria2/1.33.1
  client:
    type: library
    name: Aria2
    version: "1.33.1"
-
  user_agent: gvfs/1.34.1
  client:
    type: library
    name: gvfs
    version: "1.34.1"
-
  user_agent: uclient-fetch
  client:
    type: library
    name: uclient-fetch
    version:
-
  user_agent: cpprestsdk/2.8.0
  client:
    type: library
    name: C++ REST SDK
    version: "2.8.0"
-
  user_agent: lua-resty-http/0.10 (Lua) ngx_lua/10019
  client:
    type: library
    name: LUA OpenResty NGINX
    version: "0.10"
-
  user_agent: unirest-java/3.1.00
  client:
    type: library
    name: Unirest for Java
    version: "3.1.00"
-
  user_agent: GuzzleHttp/6.5.5 curl/7.58.0 PHP/7.4.21
  client:
    type: library
    name: Guzzle (PHP HTTP Client)
    version: "6.5.5"
-
  user_agent: Mozilla/5.0 (X11; Linux x86_64; rv:21.0) Gecko/20100101 SlimerJS/0.7
  client:
    type: library
    name: SlimerJS
    version: "0.7"
-
  user_agent: Mozilla/5.0 (win32) AppleWebKit/537.36 (KHTML, like Gecko) jsdom/15.2.1
  client:
    type: library
    name: jsdom
    version: "15.2.1"
-
  user_agent: Java-http-client/11.0.13
  client:
    type: library
    name: Java HTTP Client
    version: "11.0.13"
-
  user_agent: WinHTTP
  client:
    type: library
    name: Windows HTTP
    version:
-
  user_agent: 'Mozilla/5.0 (compatible; U; AnyEvent-HTTP/2.21;  http://software.schmorp.de/pkg/AnyEvent)'
  client:
    type: library
    name: AnyEvent HTTP
    version: "2.21"
-
  user_agent: Apache-HttpClient/4.5.3 (Java/1.8.0_292)
  client:
    type: library
    name: Apache HTTP Client
    version: "4.5.3"
-
  user_agent: Cpanel-HTTP-Client/1.0
  client:
    type: library
    name: cPanel HTTP Client
    version: "1.0"
-
  user_agent: PHP-Curl-Class/9.5.1 (+https://github.com/php-curl-class/php-curl-class)
  client:
    type: library
    name: PHP cURL Class
    version: "9.5.1"
-
  user_agent: geoipupdate/4.9.0
  client:
    type: library
    name: GeoIP Update
    version: "4.9.0"
-
  user_agent: fasthttp
  client:
    type: library
    name: fasthttp
    version:
-
  user_agent: python-httpx/0.19.0
  client:
    type: library
    name: HTTPX
    version: "0.19.0"
-
  user_agent: Python-httplib2/0.18.1 (gzip)
  client:
    type: library
    name: httplib2
    version: "0.18.1"
-
  user_agent: hackney/1.7.1
  client:
    type: library
    name: hackney
    version: "1.7.1"
-
  user_agent: network-okhttp3/3.12.10.301
  client:
    type: library
    name: OkHttp
    version: "3.12.10.301"
-
  user_agent: okhttp3-3.12.1.hw.191202
  client:
    type: library
    name: OkHttp
    version: "3.12.1"
-
  user_agent: grpc-java-okhttp/1.32.2
  client:
    type: library
    name: gRPC-Java
    version: "1.32.2"
-
  user_agent: go-resty/2.7.0 (https://github.com/go-resty/resty)
  client:
    type: library
    name: Resty
    version: "2.7.0"
-
  user_agent: pa11y/6.1.1
  client:
    type: library
    name: Pa11y
    version: "6.1.1"
-
  user_agent: ultimate_sitemap_parser/0.5
  client:
    type: library
    name: Ultimate Sitemap Parser
    version: "0.5"
-
  user_agent: Artifactory/7.21.5 72105900
  client:
    type: library
    name: Artifactory
    version: "7.21.5"
-
  user_agent: containers/5.17.0 (github.com/containers/image)
  client:
    type: library
    name: containers
    version: "5.17.0"
-
  user_agent: containerd/v1.4.13-k3s1
  client:
    type: library
    name: Containerd
    version: "1.4.13"
-
  user_agent: containerd/1.5.11+azure-1
  client:
    type: library
    name: Containerd
    version: "1.5.11"
-
  user_agent: cri-o/1.16.1 go/go1.12.17 os/linux arch/amd64
  client:
    type: library
    name: cri-o
    version: "1.16.1"
-
  user_agent: go-containerregistry/v0.11.0
  client:
    type: library
    name: go-container registry
    version: "0.11.0"
-
  user_agent: Buildah/1.23.1
  client:
    type: library
    name: Buildah
    version: "1.23.1"
-
  user_agent: buildkit/0.10
  client:
    type: library
    name: BuildKit
    version: "0.10"
-
  user_agent: buildkit/v0.10
  client:
    type: library
    name: BuildKit
    version: "0.10"
-
  user_agent: BSRPC 0.9.1
  client:
    type: library
    name: Open Build Service
    version: "0.9.1"
-
  user_agent: skopeo/1.7.0
  client:
    type: library
    name: Skopeo
    version: "1.7.0"
-
  user_agent: Helm/3.9.2
  client:
    type: library
    name: Helm
    version: "3.9.2"
-
  user_agent: harbor-registry-client
  client:
    type: library
    name: Harbor registry client
    version: ""
-
  user_agent: libcurl/7.79.1 r-curl/4.3.2 httr/1.4.3
  client:
    type: library
    name: r-curl
    version: "4.3.2"
-
  user_agent: Dart/2.16 (dart:io)
  client:
    type: library
    name: Dart
    version: "2.16"
-
  user_agent: req/v3 (https://github.com/imroc/req)
  client:
    type: library
    name: req
    version: "3"
-
  user_agent: azure-data-factory/2.0
  client:
    type: library
    name: Azure Data Factory
    version: "2.0"
-
  user_agent: quic-go-HTTP/3
  client:
    type: library
    name: quic-go
    version: ""
-
  user_agent: 'Typhoeus - https://github.com/typhoeus/typhoeus'
  client:
    type: library
    name: Typhoeus
    version: ""
-
  user_agent: got (https://github.com/sindresorhus/got)
  client:
    type: library
    name: got
    version: ""
-
  user_agent: axios/1.2.0
  client:
    type: library
    name: Axios
    version: "1.2.0"
-
  user_agent: HTMLParser/1.6
  client:
    type: library
    name: HTML Parser
    version: "1.6"
-
  user_agent: python-urllib3/1.26.9
  client:
    type: library
    name: Python urllib
    version: "1.26.9"
-
  user_agent: msray-plus
  client:
    type: library
    name: Msray-Plus
    version: ""
-
  user_agent: Slim Framework
  client:
    type: library
    name: Slim Framework
    version: ""
-
  user_agent: Fuzz Faster U Fool v1.5.0-dev
  client:
    type: library
    name: FFUF
    version: "1.5.0"
-
  user_agent: cpp-httplib/0.10.9
  client:
    type: library
    name: cpp-httplib
    version: "0.10.9"
-
  user_agent: Definitely-Not-Requests
  client:
    type: library
    name: Requests
    version: ""
-
  user_agent: Stealer 0.2.0
  client:
    type: library
    name: Stealer
    version: "0.2.0"
-
  user_agent: Mandrill-PHP/1.0.55
  client:
    type: library
    name: Mandrill PHP
    version: "1.0.55"
-
  user_agent: babashka.http-client/0.4.14
  client:
    type: library
    name: Babashka HTTP Client
    version: "0.4.14"
-
  user_agent: http.rb/4.0.0
  client:
    type: library
    name: http.rb
    version: "4.0.0"
-
  user_agent: node-superagent/3.8.2
  client:
    type: library
    name: superagent
    version: "3.8.2"
-
  user_agent: CakePHP
  client:
    type: library
    name: CakePHP
    version: ""
-
  user_agent: request.js
  client:
    type: library
    name: request
    version: ""
-
  user_agent: qbhttp/1.0.0
  client:
    type: library
    name: QbHttp
    version: "1.0.0"
-
  user_agent: kiwi-tcms/13.0
  client:
    type: library
    name: Kiwi TCMS
    version: "13.0"
-
  user_agent: kiwi-tcms/12.6.1-Enterprise
  client:
    type: library
    name: Kiwi TCMS
    version: "12.6.1"
-
  user_agent: tcms-api/12.9.1/Python 3.11.8 (main, Feb  7 2024, 04:02:05) [GCC 11.4.0]
  client:
    type: library
    name: Kiwi TCMS API
    version: "12.9.1"
-
  user_agent: tcms-api/12.7/Java
  client:
    type: library
    name: Kiwi TCMS API
    version: "12.7"
-
  user_agent: electron-fetch/1.0 electron (+https://github.com/arantes555/electron-fetch)
  client:
    type: library
    name: Electron Fetch
    version: "1.0"

sanchezzzhak · 2024-03-21T13:53:12Z

Problem 2
https://github.com/matomo-org/device-detector/blob/c9952fc4d31dd249d2bf7dc88ca9963c7a2739b7/Tests/fixtures/clienthints-app.yml#L2578C1-L2578C16

The version must be declared as a string

ref matomo-org#7633

sanchezzzhak · 2024-03-21T21:43:24Z

Problem 3
The UA Mozilla/5.0 (X11; Linux x86_64; rv:21.0) Gecko/20100101 SlimerJS/0.7
result Firefox for full parsing misc/test.php

If you use the Library parser, the result is client type "library"

# Firefox
- regex: '(?<!like )Gecko(?:/(?:\d+[\.\d]+) (?!SlimerJS))'
  name: 'Firefox'
  version: ''
  engine:
    default: 'Gecko'

Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 10.0; Win64; x64; Anonymisiert durch AlMiSoft Browser-Maulkorb 39663422; Trident/7.0; .NET4.0C; .NET4.0E; .NET CLR 2.0.50727; .NET CLR 3.0.30729; .NET CLR 3.5.30729; Tablet PC 2.0; Browzar)

If we use full parsing, we get the application, but if we use the Browser Parser separately, we get the browser.

liviuconcioiu · 2024-03-21T21:57:06Z

@sanchezzzhak coincidence or not, I wanted to talk about this a few days ago. I use Spyc::YAMLDump to generate the tests. Depending on version it escapes it or not in double quotes, I see there's a code in misc folder to rewrite them, so technically all version and engine_version should be escaped in double quotes.

Same here:

device-detector/Tests/Parser/fixtures/oss.yml

Lines 2250 to 2257 in c9952fc

    
           - 
        
             user_agent: '[FBAN/Orca-Android;FBAV/215.1.0.21.101;FBPN/com.facebook.orca;FBLC/en_GB;FBBV/154685928;FBCR/Banglalink;FBMF/Xiaomi;FBBD/xiaomi;FBDV/Redmi Note 5;FBSV/8.1.0;FBCA/armeabi-v7a:armeabi;FBDM/{density=2.75,width=1080,height=2030};FB_FW/1;]' 
        
             os: 
        
               name: Android 
        
               short_name: AND 
        
               version: 8.1.0 
        
               platform: ARM 
        
               family: Android

What should I do with platform: for example. Should it be platform: ""? Same for platform: x64 or platform: "x64". I want to fix them in the next days.

What I see, is empty strings should have double quotes, otherwise without double quotes (except versions).

sanchezzzhak · 2024-03-22T10:49:28Z

wrap quotes only for numbers \d+ and float (\d.\d) (if we expect to get the correct result in tests.)

platform: x64

it is not necessary to escape, since there are no values that start with a number

ref #7633

liviuconcioiu added a commit to liviuconcioiu/device-detector that referenced this issue Mar 21, 2024

Add double quotes

8019aae

ref matomo-org#7633

liviuconcioiu mentioned this issue Mar 21, 2024

Add double quotes #7634

Merged

sanchezzzhak closed this as completed in #7634 Mar 22, 2024

sanchezzzhak pushed a commit that referenced this issue Mar 22, 2024

Add double quotes (#7634)

1dab2b3

ref #7633

liviuconcioiu mentioned this issue Mar 23, 2024

Clean test files #7636

Closed

sgiehl added the not-in-changelog label Apr 8, 2024

sanchezzzhak mentioned this issue Aug 1, 2024

Cleanup tests #2 #7753

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Wrong record test fixture for clents/library.yml #7633

Wrong record test fixture for clents/library.yml #7633

sanchezzzhak commented Mar 21, 2024

sanchezzzhak commented Mar 21, 2024 •

edited

Loading

sanchezzzhak commented Mar 21, 2024 •

edited

Loading

liviuconcioiu commented Mar 21, 2024 •

edited

Loading

sanchezzzhak commented Mar 22, 2024 •

edited

Loading

Wrong record test fixture for clents/library.yml #7633

Wrong record test fixture for clents/library.yml #7633

Comments

sanchezzzhak commented Mar 21, 2024

sanchezzzhak commented Mar 21, 2024 • edited Loading

sanchezzzhak commented Mar 21, 2024 • edited Loading

liviuconcioiu commented Mar 21, 2024 • edited Loading

sanchezzzhak commented Mar 22, 2024 • edited Loading

sanchezzzhak commented Mar 21, 2024 •

edited

Loading

sanchezzzhak commented Mar 21, 2024 •

edited

Loading

liviuconcioiu commented Mar 21, 2024 •

edited

Loading

sanchezzzhak commented Mar 22, 2024 •

edited

Loading