Send mail failed with v3 API and multi-byte contents.

[vvakame]

Issue Summary

We can't send a mail via v3 API with multibyte content.

Steps to Reproduce

func TestSendGrid_bare(t *testing.T) {
    m := mail.NewV3Mail()
    m.SetFrom(mail.NewEmail("日本語名1", "[email protected]"))
    m.Subject = "日本語タイトル"
    m.AddContent(mail.NewContent("text/html", "０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９"))

    p := mail.NewPersonalization()
    p.AddTos(mail.NewEmail("日本語名2", "[email protected]"))
    m.AddPersonalizations(p)

    request := sendgrid.GetRequest("<my api key>", "/v3/mail/send", "https://api.sendgrid.com")
    request.Method = "POST"
    request.Body = mail.GetRequestBody(m)

    response, err := sendgrid.API(request)
    if err != nil {
        t.Error(err)
    } else if response.StatusCode < 200 || 300 <= response.StatusCode{
        t.Error(response.StatusCode)
        t.Error(response.Body)
        t.Error(response.Headers)
    } else {
        t.Log(response.StatusCode)
        t.Log(response.Body)
        t.Log(response.Headers)
    }
}

func TestSendGrid_emoji(t *testing.T) {
    m := mail.NewV3Mail()
    m.SetFrom(mail.NewEmail("foo", "[email protected]"))
    m.Subject = "title"
    m.AddContent(mail.NewContent("text/plain", "🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸🐱😸"))

    p := mail.NewPersonalization()
    p.AddTos(mail.NewEmail("fizzbuzz", "[email protected]"))
    m.AddPersonalizations(p)

    request := sendgrid.GetRequest("<my api key>", "/v3/mail/send", "https://api.sendgrid.com")
    request.Method = "POST"
    request.Body = mail.GetRequestBody(m)

    response, err := sendgrid.API(request)
    if err != nil {
        t.Error(err)
    } else if response.StatusCode < 200 || 300 <= response.StatusCode{
        t.Error(response.StatusCode)
        t.Error(response.Body)
        t.Error(response.Headers)
    } else {
        t.Log(response.StatusCode)
        t.Log(response.Body)
        t.Log(response.Headers)
    }
}

=== RUN   TestSendGrid_bare
--- FAIL: TestSendGrid_bare (0.89s)
    sendgrid_test.go:30: 415
    sendgrid_test.go:31: {"errors":[{"message":"Unrecognized UTF8 Byte at position 511","field":null,"help":"http://sendgrid.com/docs/API_Reference/Web_API_v3/Mail/errors.html#-Encoding-Errors"}]}
    sendgrid_test.go:32: map[Server:[nginx] Date:[Thu, 23 Jun 2016 04:21:42 GMT] Content-Type:[application/json] Content-Length:[171] Connection:[keep-alive] X-Frame-Options:[DENY]]
=== RUN   TestSendGrid_emoji
--- FAIL: TestSendGrid_emoji (0.09s)
    sendgrid_test.go:59: 415
    sendgrid_test.go:60: {"errors":[{"message":"Unrecognized UTF8 Byte at position 511","field":null,"help":"http://sendgrid.com/docs/API_Reference/Web_API_v3/Mail/errors.html#-Encoding-Errors"}]}
    sendgrid_test.go:61: map[Content-Length:[171] Connection:[keep-alive] X-Frame-Options:[DENY] Server:[nginx] Date:[Thu, 23 Jun 2016 04:21:42 GMT] Content-Type:[application/json]]

Technical details:

• sendgrid-go Version: master (latest commit: [21f0869])
• Go Version: 1.6.2

I'm posted this issue to SendGrid support. Request # 691762

[vvakame]

any update?

[thinkingserious]

This is in our queue to be fixed ASAP. I'll post any updates here.

Thanks!

[vvakame]

any update?
I need a solution for my project. likes #65
Our project uses gb. but our sub projects can't use it.
CI system update github.com/sendgrid/sendgrid-go automatically.
Please consider using a gopkg.in .

[csummers]

@thinkingserious For clarity, is this a Sendgrid V3 server-side bug?

I think I am running into the same issue using the latest sendgrid-java client. Mail that used to go through V2 fails in V3 with the "Unrecognized UTF8 Byte" error message.

Update: Maybe it's not server-side. This looks like the relevant pull request, though it was admittedly harder to find since the corresponding sendgrid-java issue had already been closed: sendgrid/java-http-client#5 Sorry for the noise here.

[thinkingserious]

@csummers,

It is a client side bug. The issue is that when we send the request body JSON, I did not consider the multi-byte characters when calculating the length, which causes it to be truncated.

Thanks for speaking up though because every voice helps bump the ticket up our queue to be fixed :)

[vvakame]

@thinkingserious I think this issue is not client side bug.
It can reproduce by cURL.

$ curl --request POST --url https://api.sendgrid.com/v3/mail/send --header 'Authorization: Bearer <my api key>' --header 'Content-Type: application/json' --data '{"from":{"name":"unit test","email":"[email protected]"},"subject":"unit test","personalizations":[{"to":[{"email":"[email protected]"}]}],"content":[{"type":"text/plain","value":"テキストだよ"},{"type":"text/html","value":"０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４ ５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９ ０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４ ５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９ ０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４ ５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９"}]}'
{"errors":[{"message":"Unrecognized UTF8 Byte at position 1023","field":null,"help":"http://sendgrid.com/docs/API_Reference/Web_API_v3/Mail/errors.html#-Encoding-Errors"}]}

[thinkingserious]

Thanks for the additional information @vvakame!

[thinkingserious]

If I do:

curl -X "POST" "https://api.sendgrid.com/v3/mail/send" \
        -H "Authorization: Bearer $SENDGRID_API_KEY"  \
        -H "Content-Type: application/json" \
        -d @post.json

where post.json is:

{
  "from": {
    "name": "unit test",
    "email": "[email protected]"
  },
  "subject": "unit test",
  "personalizations": [{
    "to": [{
      "email": "[email protected]"
    }]
  }],
  "content": [{
    "type": "text/plain",
    "value": "テキストだよ"
  }]
}

The email is delivered as expected.

There appears to be some issue in that string of numbers. Is there some sort of non utf-8 special character in there?

[thinkingserious]

@vvakame,

I've been able to confirm that it is an API client issue and a bug has been filed. When it get's fixed, I'll post here. Thanks for your patience and help with tracking this issue down.

[norbergj]

@thinkingserious

Do you have an ETA when this will be fixed?
I am currently evaluating if I can wait for the fix or if I should use v2 as described in #65

[lon9]

@thinkingserious I got the same problem when I try to send email written in Japanese then email was not delivered and the error was appeared.

I think it is caused by the length of body. Please see the examples below.

package main

import (
    "fmt"
    "github.com/sendgrid/sendgrid-go"
    "github.com/sendgrid/sendgrid-go/helpers/mail"
    "os"
)

func main() {
    from := mail.NewEmail("sender", "[email protected]")
    to := mail.NewEmail("receiver", "[email protected]")
    content := mail.NewContent("text/plain", "こんにちは")
    m := mail.NewV3MailInit(from, "テスト", to, content)

    request := sendgrid.GetRequest(os.Getenv("SENDGRID_API_KEY"), "/v3/mail/send", "https://api.sendgrid.com")
    request.Method = "POST"
    request.Body = mail.GetRequestBody(m)
    resp, err := sendgrid.API(request)
    if err != nil {
        panic(err)
    }
    fmt.Println(resp.StatusCode)
    fmt.Println(resp.Body)
    fmt.Println(resp.Headers)
}

It's ok. but

package main

import (
    "fmt"
    "github.com/sendgrid/sendgrid-go"
    "github.com/sendgrid/sendgrid-go/helpers/mail"
    "os"
)

func main() {
    from := mail.NewEmail("sender", "[email protected]")
    to := mail.NewEmail("receiver", "[email protected]")
    content := mail.NewContent("text/plain", "こんにちはああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああああ!!!!")
    m := mail.NewV3MailInit(from, "テスト", to, content)

    request := sendgrid.GetRequest(os.Getenv("SENDGRID_API_KEY"), "/v3/mail/send", "https://api.sendgrid.com")
    request.Method = "POST"
    request.Body = mail.GetRequestBody(m)
    resp, err := sendgrid.API(request)
    if err != nil {
        panic(err)
    }
    fmt.Println(resp.StatusCode)
    fmt.Println(resp.Body)
    fmt.Println(resp.Headers)
}

It outputs the error

415
{"errors":[{"message":"Unrecognized UTF8 Byte at position 1023","field":null,"help":"http://sendgrid.com/docs/API_Reference/Web_API_v3/Mail/errors.html#-Encoding-Errors"}]}
map[Server:[nginx] Date:[Thu, 14 Jul 2016 11:06:56 GMT] Content-Type:[application/json] Content-Length:[172] Connection:[keep-alive]]

And, it's appeared in a byte of 2 of the multiplier - 1, for example 511 and 1023.

I hope it will be your help, and I use v2.0.0 with glide until it be fixed.

[thinkingserious]

@norbergj,

I don't have an ETA, but the continued comments and votes help move it up the queue. Also, I will be advocating again for a quick fix today at our stand up.

@Rompei,

Thanks! I'll pass this along to the team.

[alxbog]

@thinkingserious Come on, guys! It's a blocker bug on server side that is known for a month already. I guess a lot of your clients believe their email are delivered when they are not.

[thinkingserious]

@alxbog,

I sincerely wish it was fixed 3 months ago :(

The good news is that it's currently in this sprint and is now being actively worked on. I'm looking forward to announcing the fix very soon.

[thinkingserious]

The fix for this has been deployed :)

Thank you for your patience!