proposal: Go 2 error values

[jba]

This issue is for discussion of our Go 2 error values proposal, which is based on our draft designs for error information and formatting.

This proposal will follow the process outlined in the "Go 2, here we come" blog post: we will have everything ready to go and checked in at the start of the Go 1.13 cycle (Feb 1), we will spend the next three months using those features and soliciting feedback based on actual usage, and then at the start of the release freeze (May 1), we will make the "launch decision" about whether to include the work in Go 1.13 or not.

Update 2019-05-06: See launch decision here: #29934 (comment)

[mvdan]

Assuming this is accepted and implemented before generics, would the design be adapted in a potential redesign of std with generics?

[mpvl]

@mvdan We think the design not using generics resulted in a slightly nicer API in the end. So we will probably not do so. That said, it can't be ruled out there will be additional API using generics.

[networkimprov]

Errors constructed with errors.New and fmt.Errorf will display differently with %+v

Could a change in output break consumers which compare it with a hard-coded string, e.g. test code?

[neild]

Errors constructed with errors.New and fmt.Errorf will display differently with %+v

Could a change in output break consumers which compare it with a hard-coded string, e.g. test code?

Conceivable, but since the fmt package currently treats %+v identically to %v for any type which implements the error interface there isn't a strong reason for users to use the former to format errors at the moment.

[Sherlock-Holo]

I wonder why xerrors.Errorf("new error: %w", baseErr) %w must at the end, I think when print the error, it should be

baseErr: some error

not

some error: baseErr

[networkimprov]

Agreed it's odd that the format string is an output-type controller, rather than a flag in the verb like %$v.

Also, have the authors considered providing this, which is conspicuously absent in package fmt:

func Error(a ...interface{}) error

This works like errors.New(fmt.Sprint(a, b))
Just as Errorf works like errors.New(fmt.Sprintf(s, a, b))

[neild]

Agreed it's odd that the format string is an output-type controller, rather than a flag in the verb like %$v.

What are the arguments for a flag over a verb? I don't see a strong argument for one over the other and %w has some mnemonic value, but perhaps I'm missing something.

func Error(a ...interface{}) error

I think this is orthogonal to the rest of the proposal, but it doesn't seem unreasonable. On the other hand, I don't think I've ever felt the absence of this function.

[networkimprov]

I'm referring to this output-type control via the format string: "if ... the format string ends with : %s, : %v, or : %w, then the returned error will implement FormatError." Why not a flag for %s & %v?

The easiest way to print an error is: fmt.Println("this happened: ", err)
The easiest way to make an error should be: fmt.Error("this happened: ", err)

[jba]

Why not a flag for %s & %v?

The only existing flags that don't have a meaning for %s and %v are space and 0. But those both feel wrong, so we'd need to make up a new flag. That means one could potentially use that flag with any verb, but it would be meaningless except for %s and %v. That feels wasteful—flags and verbs should combine orthogonally to (almost) always produce a useful result (with space and 0 themselves being the notable exceptions).

[ericlagergren]

To be clear: does this mean each call to fmt.Errorf will collect a stack trace?

[neild]

To be clear: does this mean each call to fmt.Errorf will collect a stack trace?

A single stack frame, not a full trace.

[tandr]

To be clear: does this mean each call to fmt.Errorf will collect a stack trace?

A single stack frame, not a full trace.

Would it be possible to make it "adjustable" somehow? In our home-made logging wrapper we have noticed that we have to pull a couple or 3 frames to get to "the caller of interest", so printout looks more or less pointing to the place where logwrap.Print(err) function was called.
If this is becoming a part of the standard, in case when wrapper (or utility method) is created to simplify an error creation, I would like to report the place where that method was called from, and not the utility method itself.

We did it by (arguable not so elegant) way as adding a parameter "how big is the stack offset" to the logging function, but I am not sure if that's the only option here. Adding special format symbol for stack, with number of frames interesting might be one of possible approaches.

[neild]

The intended usage is that additional frames be added as necessary by annotating or wrapping the error, possibly with additional context.

For example,

func inner() error { return errors.New("inner error") }
func outer() error { return fmt.Errorf("outer error: %w", inner()) }

fmt.Fprintf("%+v", outer())
// outer error:
//     /path/to/file.go:123
//   - inner error:
//     /path/to/file.go:122

To attach information about where a utility function was called from, you do so in the same way as today: By annotating the error with additional information.

We considered capturing a full stack rather than a single frame, but think the single-frame approach has some advantages: It's light-weight enough to be feasible to do for every error and it deals well with error flows that pass between goroutines.

That said, this proposal makes it easy for error implementations to add whatever detail information they want in a compatible way, be it full stacks, offset stack frames, or something else.

[networkimprov]

[the flag] would be meaningless except for %s and %v. That feels wasteful—flags and verbs should combine orthogonally to (almost) always produce a useful result (with space and 0 themselves being the notable exceptions).

Erm, I think you've refuted your own argument by raising exceptions :-)

: %s is magical, invisible, and readily broken without compiler complaint. And no other format string has similar effect. That's not Goish to my eye.

[neild]

: %s is magical, invisible, and readily broken without compiler complaint. And no other format string has similar effect. That's not Goish to my eye.

I'm not certain if you're making an case about verbs vs. flags (%v/%w vs. %v/%$v) or about using fmt.Errorf to annotate/wrap errors in general. Could you clarify your understanding of the proposal and what you're suggesting instead?

[networkimprov]

As I said above, I'm referring to this output-type control in the format string:

if ... the format string ends with : %s, : %v, or : %w, then the returned error will implement FormatError.

The control should be a flag e.g. %$s & %$v, and not the contents of the format string.

[neild]

The control should be a flag e.g. %$s & %$v, and not the contents of the format string.

Thanks; I think I understand you now.

The current design of applying special-case handling in fmt.Errorf to a : %v et al. suffix is informed by a couple of factors.

Firstly, we want to permit existing code to take as much advantage of the new error formatting features as possible. So, for example, errors.New and fmt.Errorf now capture the caller's stack frame by and print this information in detail mode. Existing code that creates errors does not need to change to include this information. But what about annotation of errors? Consider the following:

func f1() error { return errors.New("some error") }
func f2() error { return fmt.Errorf("f2: %v", f1()) }

func main() {
  fmt.Printf("%+v", f2())
}

We want this to print:

some error:
    main.go:1
  - f2:
    main.go:2

But if annotation is only enabled with a special format directive like %$v, then this will instead print:

some error: f2
    main.go:2

The innermost stack frame is lost.

The other consideration is that error annotation is linear. The errors.Formatter interface allows an error to format itself and return the next error to format. By design, it does not provide a way for an error to interleave its output with that of another error. This limitation makes the formatted output more consistent and predictable, but it means that there is no way to insert a hypothetical %$v into the middle of a format string:

// What would this error's `FormatError` method do?
return fmt.Errorf("some information about error: %$v (with some more information)", err)

These two considerations led to the current design of automatically annotating errors produced by fmt.Errorf calls that follow the most common pattern of adding information to an error, but only when the original error appears at the end of the string.

[networkimprov]

Re the stack frame, annotation could be enabled by any use of an error argument with Errorf(), instead of a magical, brittle format string.

there is no way to insert a hypothetical %$v into the middle of a format string

Would a format string "abc %v 123" work as expected given annotation enablement on use of an error argument? If not, I believe you need to rethink this...

[jba]

Would a format string "abc %v 123" work as expected given annotation enablement on use of an error argument?

No; we will only treat an error specially if the format string ends with ": %s" or ": %v".

If not, I believe you need to rethink this...

Can you explain why? Our goal with this feature is to instantly and painlessly bring the new formatting into most existing Go code. We'd rethink that if, for example, it turned out that most existing calls to fmt.Errorf that included an error did not put the error at the end with ": %s" or ": %v". If that is true, we'd like to know.

As Damien pointed out, if you do like to put errors in the middle of your format strings, you're going to have bigger problems with our proposal than its fmt.Errorf behavior. Even if you roll your own formatting with the FormatError method, you're going to have a hard time getting wrapped errors to display in the middle.

One more point: while we do expect people to continue to write fmt.Errorf calls that wrap errors, we also encourage custom error-creating functions that build specific errors and might incorporate other features, like frame depth (as in @tandr's request). E.g.

type MyError ...
func MyErrorf(wrapped error, frameDepth int, format string, args ...interface{}) *MyError

[tandr]

No; we will only treat an error specially if the format string ends with ": %s" or ": %v".

My concern here would be that
a) not every message fits this format, and
b) not every message is going to be in English.

"Something happened: This is What" is an ok way to produce a short human-readable error, but... Even in English, if I would want some text explaining "What to do now?" (remediation suggestion, "contact support" etc), that : %v might not be the last item in the formatting string. And the whole "at the end" logic will be completely broken for languages that are using Right-to-Left writing order (Arabic, Hebrew, Farsi, Urdu etc.)

Would it be easier to introduce a special formatter "print this as 'error'" and a modifier to make it "print this as an error with a stack"? I think there are enough letters in English alphabet left to cover one more case.

Also, if I may ask - no magic, please.

[glibsm]

The Unwrap function is a convenience for calling the Unwrap method if one exists.

// Unwrap returns the result of calling the Unwrap method on err, if err implements Unwrap.
// Otherwise, Unwrap returns nil.
func Unwrap(err error) error

It would also be nice to have a standard helper that can get to the root cause, i.e. the last error in the chain.

Similar to https://godoc.org/github.com/pkg/errors#hdr-Retrieving_the_cause_of_an_error.

[neild]

There are two points here, and it's worth considering them independently. The first is the manner in which error annotations compose.

The proposal adds a new errors.Formatter interface which errors may optionally implement. This interface provides two main pieces of functionality: Error detail in which an error may provide additional information when formatted with the %+v directive, and wrapping in which one error may add additional information to another.

A primary goal of the errors.Formatter interface is to allow different error implementations to interoperate. You should be able to create an error with errors.New, add some information to it with "github.com/pkg/errors".Wrap, add more information to that with "gopkg.in/errgo.v2/errors".Wrap, and get reasonable behavior.

Detail formatting of an error (%+v) produces multi-line output, with each step in the error chain represented as a section of one or more lines. For example:

could not adumbrate elephant:  - these three lines are the outermost error
    (some additional detail)   |
    labrynth.go:1986           /
  - out of peanuts             - and these two lines are the innermost error
    elephant.go:15             / 

Displaying a chain of errors in a consistent fashion requires making certain decisions about formatting: Do we print errors from the outermost to innermost, or vice-versa? How do we indicate boundaries between errors? If we leave these formatting decisions up to individual error implementations, we are likely to have situations where, for example, part of an error chain is printed inner-to-outer and another part is outer-to-inner; very confusing and not a good user experience.

To avoid confusion and simplify the work required to write a custom error type, we put the formatting of the error chain under the control of the error printer. Errors provide three pieces of information: The error text, detail text, and the next error in the chain. The printer does the rest. So, for example, the outermost error in the above example might have an ErrorFormatter implementation like:

func (e errType) FormatError(p Printer) (next error) {
  p.Print("could not adumbrate elephant") // error text
  if p.Detail() {
    p.Print("(some additional detail)") // error text only visible in detail (%+v)
    p.frame.FormatError(p) // print stack frame as detail
  }
  return e.wrappedErr // return the next error in the chain
}

Note again that an important property of this design is that an error doesn't choose how to format the error that it wraps; it returns that error to the printer, which then determines how to format the error chain. Not only does this simplify the error implementation and ensure consistency, but this property is important for localization of errors in RTL languages (as mentioned by @tandr), because it permits a localizing printer to adjust the output order as appropriate. (Most users will print errors using the fmt package, but note that this design deliberately allows for other implementations of errors.Printer.)

Given the above, the question then is how to make it simple for users to produce errors that satisfy these interfaces, which is where we get to the special handling of a suffix : %v. It will be helpful if critiques of this proposal are clear about whether a comment is about the above formatting interfaces, or fmt.Errorf specifically.

[neild]

It would also be nice to have a standard helper that can get to the root cause, i.e. the last error in the chain.

The equivalent in this proposal is the errors.Is function:

if errors.Is(err, io.EOF) { ... }

We believe this has several nice properties over a function returning the last element in a chain:

• Harder to accidentally discard context (e.g., return errors.Cause(err)).
• An error can implement an Is(target error) bool) method to declare that it is equivalent to another error.
• Nicely analogous to errors.As, which converts an error to a type.

[velovix]

I like the motivation of the new Errorf behavior being a way to grandfather in old code. Near as I can tell, it could only make old code better. However, I'm less into the idea of Errorf being the main way to wrap errors. I could see programmers writing some error handling logic and, due to force of habit, accidentally writing fmt.Errorf("context: %v", err) instead of fmt.Errorf("context: %w", err) and missing out on potentially valuable information or effecting the intended error handling path. Since errors often find themselves in uncommon circumstances, it's possible to miss problems like these until an error in production occurs and you have less diagnostic information than you were hoping. For that reason, I think having a stronger, more compiler friendly wrapping solution available makes sense. Maybe something like github.com/pkg/errors.Wrap(err, message)?

To be a bit more concrete, I have more than once found myself in a similar situation with Python, where I raise an exception using f-strings but forget to put the "f".

raise RuntimeError("oh no! Something happened: {important_information}")

... instead of ...

raise RuntimeError(f"oh no! Something happened: {important_information}")

And since there's nothing syntactically wrong with what I did, I might not know until later when I need important_information that I've made a mistake.

[networkimprov]

Our goal with this feature is to instantly and painlessly bring the new formatting into most existing Go code.

Although that seems like a laudable goal, modifying the way existing code creates output is inherently risky. I suspect most of us would rather have an opt-in mechanism. Let project owners trial a feature and switch over (via go fix) when convenient for them.

IOW, new error-construction functions are needed, which always return errors.Formatter.

[PieterD]

While the fmt.Errorf changes to wrap the error is neat and conserves backwards compatibility with previous versions, a slightly nicer option going forward might be good?

When using pkg/errors, errors.Wrapf(err, "error reticulating spline %d", splineID) is the most used pattern by far. If that is also introduced, both of the most common patterns are maintained (aside from a switch from %v to %w in fmt.Errorf, which might cause trouble if one is missed.)

[fatih]

We will produce a revised proposal for error formatting improvements for Go 1.14 [link]

Go 1.14 is out and I was curious if any one these made into Go v1.14 but couldn't find anything in the changelog. Can we assume it's now postponed to 1.15? What is the latest status on capturing and including stack frames inside Errors?

[changkun]

@fatih There are no more plans for error handling. It is delayed at least after generics. You need to wait years.

[fatih]

@fatih There are no more plans for error handling. It is delayed at least after generics. You need to wait years.

@changkun where is the decision for the delay? Can you please give more details.

[changkun]

@fatih

https://blog.golang.org/go1.15-proposals

... we have not further pursued changes to error handling for now. ... -- Robert Griesemer

https://golang.org/issue/33162#issuecomment-559183019

I think we should decline all the error handling proposals until we are ready to revisit the topic (perhaps not for a few years). -- Russ Cox

[JavierZunzunegui]

@fatih

FYI zerrors is a new library that offers precisely what go 1.13 didn't -> frame info & error formatting.

[fatih]

Thank you @changkun for the links.

@jba @rsc based on the links that @changkun provided, can we assume that error handling work is suspended (or paused)? If that is the case we should update the issue with a notice reflecting the latest status. There are many people (including me) assuming this is still and on-going work.

I was assuming that we would receive some of the functionality, such as stack frames and was planning accordingly for that internally at GitHub. I need to revisit my planning and inform teams at GitHub that the work on errors is delayed (or suspended) if that is the case.

[fatih]

@JavierZunzunegui thanks for the link. Is there any additional benefit compared to xerrors, which already includes frame info ?

[JavierZunzunegui]

@JavierZunzunegui thanks for the link. Is there any additional benefit compared to xerrors, which already includes frame info ?

(+) It provides much more formatting flexibility. Skip want you want, encode the others any way you want, if you want to you can even reverse the encoding order.
(+) Mixing non-zerrors with zerrors fares better than mixing non-xerrors with xerrors. Formatting only messes up locally (versus messing up the entire rest of the error in xerrors), and you always have the option to skip ayways (not so in xerrors).
(+) Easier to use. Your errors don't have to implement anything besides Error() string, the library provides everything else.
(?) performance haven't done the benchmark comparison yet, but will do.
(?) minimal API. Comes with being new, it makes it easier to understand but the desired 'utilities' are yet to come, with usage feedback.
(-) new and untested in production. You won't find libraries using it (yet?).
(-) v0.1.0. I'll try not to break things but that's not set in stone...

[flibustenet]

About including frame, there are a lot of lib implementing this, and it's very easy to do, i did it myself to. But all of them are unusable in library if there is not at least a common interface to know if an error already embed the frame and how to print it.

[JavierZunzunegui]

About including frame, there are a lot of lib implementing this, and it's very easy to do, i did it myself to. But all of them are unusable in library if there is not at least a common interface to know if an error already embed the frame and how to print it.

@flibustenet of course, the ideal place for any such solution is the standard library, but that is not happenning any time soon. We are in the same place with frames / formatting that we were with wrapping pre-1.13: multiple, sometimes poorly compatible frameworks. But that is the way to experiment and ultimately show, through experience, what is best, and guide the decision of what makes it to the standard library. That was very much the story of wrapping too.

[knz]

The issue with zerrors linked above is that the error objects are not suitable for transporting over the wire in a distributed application. It's not possible to reliably inspect stack traces and frame information across servers.

See crdb's own errors replacement which supports network transport and is battle-tested in production.

[knz]

the crdb errors package also has its own formatting protocol for %+v which aims to fix many shortcomings of xerrors.

[ianlancetaylor]

@changkun We are unlikely to do anything about error handling any time soon. But this proposal is not about error handling. It is about error values. They are different things. See, e.g., the separate discussions of error values and error handling at https://go.googlesource.com/proposal/+/master/design/go2draft.md.

[gophun]

Shouldn't this issue be closed? The proposal was accepted and shipped with Go 1.13. As @ianlancetaylor pointed out this issue was about the error values proposal, not about error handling in general. This is currently the most commented open issue, so I was expecting something interesting going on here, but it's already done.