Product Sync fails on fetch, if a key contain " in it.

[markus-mw]

I'm trying to sync products (v3.12.0 and v3.11.0) between projects, but I'm getting this error:

{"timestamp":"2021-04-15T07:44:31.713Z","severity":"ERROR","loggerName":"com.commercetools.project.sync.product.ProductSyncer","message":"Error when trying to sync product. Existing key: <<not present>>. Update actions: []","stackHash":"1e0fe24d","stackTrace":"i.s.s.c.ErrorResponseException: detailMessage: Malformed parameter: where: Syntax error while parsing 'where'. Invalid input '-', expected comma or ')' (line 1, column 1194):\

[ahmetoz]

Hey, @markus-mw thanks for reaching us and opening the issue.

To understand it better, It looks like the issue coming through where a query and escaping issue, and we will check how we are querying on the ProductSycner.

Are you using any query for products like productQueryParameters on your command?

Does this error happen when you change the version from v3.11.0 to v3.12.0? More input would be helpful to understand the problem better.

[markus-mw]

I'm using the docker way ;). Don't know how to use any query.
As already mentionend, with both versions 3.11 and 3.12. Same result.

docker run -e SOURCE_PROJECT_KEY="abc" -e SOURCE_CLIENT_ID="" -e SOURCE_CLIENT_SECRET="" -e TARGET_PROJECT_KEY="cbe" -e TARGET_CLIENT_ID="" -e TARGET_CLIENT_SECRET="" commercetools/commercetools-project-sync:3.12.0 -s products -f

[ahmetoz]

@markus-mw Thanks for the reply,
Could you please provide more details, you are claiming that when a product has a key "-" character it's failing right? Do you have a clear use case, such as a product sample or so we could reproduce and trace the bug in which condition that it happens?

[markus-mw]

Right, I'm trying to sync products from one project to another. Products have keys with "-" characters.

Here's an anonymized impex example:

{
  "limit": 20,
  "offset": 0,
  "count": 1,
  "total": 1,
  "results": [
    {
      "id": "",
      "version": 85,
      "lastMessageSequenceNumber": 35,
      "createdAt": "2021-03-11T14:56:09.441Z",
      "lastModifiedAt": "2021-04-15T12:45:20.868Z",
      "lastModifiedBy": {
        "isPlatformClient": true
      },
      "createdBy": {
        "isPlatformClient": true
      },
      "productType": {
        "typeId": "product-type",
        "id": ""
      },
      "masterData": {
        "current": {
          "name": {
            "de": "product name"
          },
          "description": {
            "de": "product name"
          },
          "categories": [
            {
              "typeId": "category",
              "id": ""
            }
          ],
          "categoryOrderHints": {},
          "slug": {
            "de": "product-slug-with-minus-2020"
          },
          "metaTitle": {
            "de": "meta title"
          },
          "metaDescription": {
            "de": "meta description"
          },
          "masterVariant": {
            "id": 1,
            "sku": "4352",
            "key": "4352_1",
            "prices": [
              {
                "value": {
                  "type": "centPrecision",
                  "currencyCode": "EUR",
                  "centAmount": 249999,
                  "fractionDigits": 2
                },
                "id": "",
                "channel": {
                  "typeId": "channel",
                  "id": ""
                }
              },
              {
                "value": {
                  "type": "centPrecision",
                  "currencyCode": "EUR",
                  "centAmount": 219999,
                  "fractionDigits": 2
                },
                "id": "",
                "channel": {
                  "typeId": "channel",
                  "id": ""
                }
              }
            ],
            "images": [
              {
                "url": "",
                "label": "product_01",
                "dimensions": {
                  "w": 0,
                  "h": 0
                }
              }
            ],
            "attributes": [
			
            ],
            "assets": [],
            "availability": {
              "channels": {
                "": {
                  "isOnStock": false,
                  "availableQuantity": 0
                }
              }
            }
          },
          "variants": [],
          "searchKeywords": {
            "de": [
              
            ]
          }
        },        
        "published": true,
        "hasStagedChanges": false
      },
      "key": "product-key-with-minus-2020",
      "taxCategory": {
        "typeId": "tax-category",
        "id": ""
      },
      "lastVariantId": 1
    }
  ]
}

[markus-mw]

It's possible that the issue only happens with force parameter. I can't retry it at the moment.

[ahmetoz]

Hey, @markus-mw I don't know how to reproduce the issue, I tried with a similar product that you have shared with the same parameters but did not face the issue that you had described, it would be useful if you could share more context maybe a whole stack trace of when and why this issue happens in the code, otherwise for us a bit hard to analyze the issue. Thanks in advance 🙏🏽

[markus-mw]

There isn't so much more I could give you.

I'm getting for all of the products the same error:

{"timestamp":"2021-04-19T05:23:47.085Z","severity":"ERROR","loggerName":"com.commercetools.project.sync.product.ProductSyncer","message":"Error when trying to sync product. Existing key: <<not present>>. Update actions: []","stackHash":"1e0fe24d","stackTrace":"i.s.s.c.ErrorResponseException: detailMessage: Malformed parameter: where: Syntax error while parsing 'where'. Invalid input '-', expected comma or ')' (line 1, column 99):
key in (\"product-bass-sl-20-bright-turquoise-2020\", \"kypu-lr-m-kypu-color-2021\", \"product-pirat-18\"-wave-blue-2021\", \"product-e-postman-luxe-norma-500-wave-stealth-black-2020\", \"kypu-steel-18-lovely-pink-2021\", \"kypu-cyke-24-3-alu-schwarz-2021\", \"product-tour-nuxes-24-lagoon-green-2020\", \"product-savoie-lady-stealth-black-2020\", \"product-tour-nuxes-27-5-cosmos-black-2020\", \"product-postman-luxe-gent-grau-2020\", \"product-c-3-0-diamant-blue-2021\", \"product-corby-8-xxl-comfort-2021\", \"product-boulevard-luxe-norma-schwarz-2020\", \"product-run-&-ride-blue-2021\", \"product-e-triton-pt5-gent-500-diamant-grau-2020\", \"product-fx-407-nd-diamant-black-2021\", \"product-e-triton-45-gent-625-diamant-schwarz-2020\", \"kypu-lr-1-br-die-maus-2021\", \"product-l-4-0-cargo-wave-black-2021\", \"product-l-4-0-cargo-wave-turquoise-2021\", \"kypu-lr-light-blau-2021\", \"rimoson-e-6-0-wave-dark-blue-2021\", \"product-e-tremalzo-27-5-500-diamant-weiss-2020\", \"rimoson-tucano-sport-trapez-electric-grey-2021\", \"rimoson-t-7-0-trapez-dark-blue-black-2021\", \"product-e-agnello-29-625-diamant-grau-2020\", \"product-5x-lady-grau-2020\", \"kypu-ceety-cat-s6-bronze-2021\", \"kypu-steel-12-lovely-pink-2021\", \"kypu-lr-1-l-blau-2021\")
                                                                                                  ^
summary: GET https://api.europe-west1.gcp.commercetools.com/myshop-stage/products?where=key+in+%28%22product-bass-sl-20-bright-turquoise-2020%22%2C+%22kypu-lr-m-kypu-color-2021%22%2C+%22product-pirat-18%22-wave-blue-2021%22%2C+%22product-e-postman-luxe-norma-500-wave-stealth-black-2020%22%2C+%22kypu-steel-18-lovely-pink-2021%22%2C+%22kypu-cyke-24-3-alu-schwarz-2021%22%2C+%22product-tour-nuxes-24-lagoon-green-2020%22%2C+%22product-savoie-lady-stealth-black-2020%22%2C+%22product-tour-nuxes-27-5-cosmos-black-2020%22%2C+%22product-postman-luxe-gent-grau-2020%22%2C+%22product-c-...
"}
{"timestamp":"2021-04-19T05:23:47.120Z","severity":"ERROR","loggerName":"com.commercetools.project.sync.product.ProductSyncer","message":"Error when trying to sync product. Existing key: <<not present>>. Update actions: []","stackHash":"1e0fe24d","stackTrace":"i.s.s.c.ErrorResponseException: detailMessage: Malformed parameter: where: Syntax error while parsing 'where'. Invalid input '-', expected comma or ')' (line 1, column 104):
key in (\"product-jam-1.0-trapez-2021\", \"product-e-compact-1-0-black-grey-2021\", \"product-mermaid-12\"-wave-turquoise-2021\")
                                                                                                       ^
summary: GET https://api.europe-west1.gcp.commercetools.com/myshop-stage/products?where=key+in+%28%22product-jam-1.0-trapez-2021%22%2C+%22product-e-compact-1-0-black-grey-2021%22%2C+%22product-mermaid-12%22-wave-turquoise-2021%22%29&limit=250&withTotal=false failed  with response code 400 with X-Correlation-ID `myshop-stage/b140a7e8-252e-4b75-b905-af70e578edbf` on 2021-04-19T05:23:47.116677Z
http response normatted body: {
  \"statusCode\" : 400,
  \"message\" : \"Malformed parameter: where: Syntax error while parsing 'where'. Invalid input '-', expected comma or ')' (line 1, column 104):\
key in (\\\"product-jam-1.0-trapez-2021\\\", \\\"product-e-compact-1-0-black-grey-2021\\\", \\\"product-mermaid-12\\\"-wave-turquoise-2021\\\")\
                                                                                                       ^\",
  \"errors\" : [ {
    \"code\" : \"InvalidInput\",
    \"message\" : \"Malformed parameter: where: Syntax error while parsing 'where'. Invalid input '-', expected comma or ')' (line 1, column 104):\
key in (\\\"product-jam-1.0-trapez-2021\\\", \\\"product-e-compact-1-0-black-grey-2021\\\", \\\"product-mermaid-12\\\"-wave-turquoise-2021\\\")\
                                                                                                       ^\"
  } ]
}
http response: io.sphere.sdk.http.HttpResponseImpl@7d4b01a[statusCode=400,headers={date=[Mon, 19 Apr 2021 05:23:47 GMT], content-length=[811], server=[istio-envoy], x-envoy-upstream-service-time=[3], Alt-Svc=[clear], access-control-allow-headers=[Accept, Authorization, Content-Type, Origin, User-Agent, X-Correlation-ID], x-correlation-id=[myshop-stage/b140a7e...
"}

[markus-mw]

@ahmetoz : I found the problem... The error message is misleading... There is a " in the key! Sorry for the inconvenience!

[ahmetoz]

Hi @markus-mw, no problem even with that I will check the queries created by us in case if this issue could be resolved when the key has " character in it.

[ahmetoz]

I was not expecting the key would be possible with " for a product, because mostly it's not allowed by commercetools platform but it's different for the product resource I tried and could create a key with " in it. And of course, the query we have does not have any escaping for such characters and it fails in where query parameter. I'm glad that you have found this issue so we might improve the query creation.

[praveenkumarct]

Fixed as part of version 4.0.1