Value Refinements

A new approach for cty to refine the possible range of an unknown value beyond just a type constraint.

COMPATIBILITY.md

@@ -0,0 +1,88 @@
+# `cty` backward-compatibility policy
+
+This library includes a number of behaviors that aim to support "best effort"
+partial evaluation in the presence of wholly- or partially-unknown inputs.
+Over time we've improved the accuracy of those analyses, but doing so changes
+the specific results returned by certain operations.
+
+This document aims to describe what sorts of changes are allowed in new minor
+releases and how those changes might affect the behavior of dependents after
+upgrading.
+
+Where possible we'll avoid making changes like these in _patch_ releases, which
+focus instead only on correcting incorrect behavior. An exception would be if
+a minor release introduced an incorrect behavior and then a patch release
+repaired it to either restore the previous correct behavior or implement a new
+compromise correct behavior.
+
+## Unknown Values can become "more known"
+
+The most significant policy is that any operation that was previously returning
+an unknown value may return either a known value or a _more refined_ unknown
+value in later releases, as long as the new result is a subset of the range
+of the previous result.
+
+When using only the _operation methods_ and functionality derived from them,
+`cty` will typically handle these deductions automatically and return the most
+specific result it is able to. In those cases we expect that these changes will
+be seen as an improvement for end-users, and not require significant changes
+to calling applications to pass on those benefits.
+
+When working with _integration methods_ (those which return results using
+"normal" Go types rather than `cty.Value`) these changes can be more sigificant,
+because applications can therefore observe the differences more readily.
+For example, if an unknown value is replaced with a known value of the same
+type then `Value.IsKnown` will begin returning `true` where it previously
+returned `false`. Applications should be designed to avoid depending on
+specific implementation details like these and instead aim to be more general
+to handle both known and unknown values.
+
+A specific sensitive area for compatibility is the `Value.RawEquals` method,
+which is sensitive to all of the possible variations in values. Applications
+should not use this method for normal application code to avoid exposing
+implementation details to end-users, but might use it to assert exact expected
+results in unit tests. Such test cases may begin failing after upgrading, and
+application developers should carefully consider whether the new results conform
+to these rules and update the tests to match as part of their upgrade if so. If
+the changed result seems _not_ to conform to these rules then that might be a
+bug; please report it!
+
+## Error situations may begin succeeding
+
+Over time the valid inputs or other constraints on functionality might be
+loosened to support new capabilities. Any operation or function that returned
+an error in a previous release can begin succeeding with any valid result in
+a new release.
+
+## Error message text might change
+
+This library aims to generate good, actionable error messages for user-facing
+problems and to give sufficient information to a calling application to generate
+its own high-quality error messages in situations where `cty` is not directly
+"talking to" an end-user.
+
+This means that in later releases the exact text of error messages in certain
+situations may change, typically to add additional context or increase
+precision.
+
+If a function is documented as returning a particular error type in a certain
+situation then that should be preserved in future releases, but if there is
+no explicit documentation then calling applications should not depend on the
+dynamic type of any `error` result, or should at least do so cautiously with
+a fallback to a general error handler.
+
+## Passing on changes to Go standard library
+
+Some parts of `cty` are wrappers around functionality implemented in the Go
+standard library. If the underlying packages change in newer versions of Go
+then we may or may not pass on the change through the `cty` API, depending on
+the circumstances.
+
+A specific notable example is Unicode support: this library depends on various
+Unicode algorithms and data tables indirectly through its dependencies,
+including some in the Go standard library, and so its exact treatment of strings
+is likely to vary between releases as the Unicode standard grows. We aim to
+follow the version of Unicode supported in the latest version of the Go standard
+library, although we may lag behind slightly after new Go releases due to the
+need to update other libraries that implement other parts of the Unicode
+specifications.

README.md

@@ -42,6 +42,9 @@ For more details, see the following documentation:
 * [Conversion to and from native Go values](./docs/gocty.md)
 * [JSON serialization](./docs/json.md)
 * [`cty` Functions system](./docs/functions.md)
+* [Compatibility Policy for future Minor Releases](./COMPATIBILITY.md): please
+  review this before using `cty` in your application to avoid depending on
+  implementation details that may change.
 
 ---
 

cty/convert/conversion.go

@@ -43,7 +43,7 @@ func getConversion(in cty.Type, out cty.Type, unsafe bool) conversion {
 			out = out.WithoutOptionalAttributesDeep()
 
 			if !isKnown {
-				return cty.UnknownVal(dynamicReplace(in.Type(), out)), nil
+				return prepareUnknownResult(in.Range(), dynamicReplace(in.Type(), out)), nil
 			}
 
 			if isNull {
@@ -199,3 +199,64 @@ func retConversion(conv conversion) Conversion {
 		return conv(in, cty.Path(nil))
 	}
 }
+
+// prepareUnknownResult can apply value refinements to a returned unknown value
+// in certain cases where characteristics of the source value or type can
+// transfer into range constraints on the result value.
+func prepareUnknownResult(sourceRange cty.ValueRange, targetTy cty.Type) cty.Value {
+	sourceTy := sourceRange.TypeConstraint()
+
+	ret := cty.UnknownVal(targetTy)
+	if sourceRange.DefinitelyNotNull() {
+		ret = ret.RefineNotNull()
+	}
+
+	switch {
+	case sourceTy.IsObjectType() && targetTy.IsMapType():
+		// A map built from an object type always has the same number of
+		// elements as the source type has attributes.
+		return ret.Refine().CollectionLength(len(sourceTy.AttributeTypes())).NewValue()
+	case sourceTy.IsTupleType() && targetTy.IsListType():
+		// A list built from a typle type always has the same number of
+		// elements as the source type has elements.
+		return ret.Refine().CollectionLength(sourceTy.Length()).NewValue()
+	case sourceTy.IsTupleType() && targetTy.IsSetType():
+		// When building a set from a tuple type we can't exactly constrain
+		// the length because some elements might coalesce, but we can
+		// guarantee an upper limit. We can also guarantee at least one
+		// element if the tuple isn't empty.
+		switch l := sourceTy.Length(); l {
+		case 0, 1:
+			return ret.Refine().CollectionLength(l).NewValue()
+		default:
+			return ret.Refine().
+				CollectionLengthLowerBound(1).
+				CollectionLengthUpperBound(sourceTy.Length()).
+				NewValue()
+		}
+	case sourceTy.IsCollectionType() && targetTy.IsCollectionType():
+		// NOTE: We only reach this function if there is an available
+		// conversion between the source and target type, so we don't
+		// need to repeat element type compatibility checks and such here.
+		//
+		// If the source value already has a refined length then we'll
+		// transfer those refinements to the result, because conversion
+		// does not change length (aside from set element coalescing).
+		b := ret.Refine()
+		if targetTy.IsSetType() {
+			if sourceRange.LengthLowerBound() > 0 {
+				// If the source has at least one element then the result
+				// must always have at least one too, because value coalescing
+				// cannot totally empty the set.
+				b = b.CollectionLengthLowerBound(1)
+			}
+		} else {
+			b = b.CollectionLengthLowerBound(sourceRange.LengthLowerBound())
+		}
+		b = b.CollectionLengthUpperBound(sourceRange.LengthUpperBound())
+		return b.NewValue()
+	default:
+		return ret
+	}
+
+}

cty/convert/public_test.go

@@ -1615,6 +1615,161 @@ func TestConvert(t *testing.T) {
 				})),
 			}),
 		},
+
+		// Object to map refinements
+		{
+			Value: cty.UnknownVal(cty.EmptyObject),
+			Type:  cty.Map(cty.String),
+			Want: cty.UnknownVal(cty.Map(cty.String)).Refine().
+				CollectionLength(0).
+				NewValue(),
+		},
+		{
+			Value: cty.UnknownVal(cty.EmptyObject).RefineNotNull(),
+			Type:  cty.Map(cty.String),
+			Want:  cty.MapValEmpty(cty.String),
+		},
+		{
+			Value: cty.UnknownVal(cty.Object(map[string]cty.Type{"a": cty.String})),
+			Type:  cty.Map(cty.String),
+			Want: cty.UnknownVal(cty.Map(cty.String)).Refine().
+				CollectionLength(1).
+				NewValue(),
+		},
+		{
+			Value: cty.UnknownVal(cty.Object(map[string]cty.Type{"a": cty.String})).RefineNotNull(),
+			Type:  cty.Map(cty.String),
+			Want: cty.UnknownVal(cty.Map(cty.String)).Refine().
+				NotNull().
+				CollectionLength(1).
+				NewValue(),
+		},
+
+		// Tuple to list refinements
+		{
+			Value: cty.UnknownVal(cty.EmptyTuple),
+			Type:  cty.List(cty.String),
+			Want: cty.UnknownVal(cty.List(cty.String)).Refine().
+				CollectionLength(0).
+				NewValue(),
+		},
+		{
+			Value: cty.UnknownVal(cty.EmptyTuple).RefineNotNull(),
+			Type:  cty.List(cty.String),
+			Want:  cty.ListValEmpty(cty.String),
+		},
+		{
+			Value: cty.UnknownVal(cty.Tuple([]cty.Type{cty.String})),
+			Type:  cty.List(cty.String),
+			Want: cty.UnknownVal(cty.List(cty.String)).Refine().
+				CollectionLength(1).
+				NewValue(),
+		},
+		{
+			Value: cty.UnknownVal(cty.Tuple([]cty.Type{cty.String})).RefineNotNull(),
+			Type:  cty.List(cty.String),
+			Want:  cty.ListVal([]cty.Value{cty.UnknownVal(cty.String)}),
+		},
+
+		// Tuple to set refinements
+		{
+			Value: cty.UnknownVal(cty.EmptyTuple),
+			Type:  cty.Set(cty.String),
+			Want: cty.UnknownVal(cty.Set(cty.String)).Refine().
+				CollectionLength(0).
+				NewValue(),
+		},
+		{
+			Value: cty.UnknownVal(cty.EmptyTuple).RefineNotNull(),
+			Type:  cty.Set(cty.String),
+			Want:  cty.SetValEmpty(cty.String),
+		},
+		{
+			Value: cty.UnknownVal(cty.Tuple([]cty.Type{cty.String})),
+			Type:  cty.Set(cty.String),
+			Want: cty.UnknownVal(cty.Set(cty.String)).Refine().
+				CollectionLength(1).
+				NewValue(),
+		},
+		{
+			Value: cty.UnknownVal(cty.Tuple([]cty.Type{cty.String})).RefineNotNull(),
+			Type:  cty.Set(cty.String),
+			Want:  cty.SetVal([]cty.Value{cty.UnknownVal(cty.String)}),
+		},
+		{
+			Value: cty.UnknownVal(cty.Tuple([]cty.Type{cty.String, cty.String})),
+			Type:  cty.Set(cty.String),
+			Want: cty.UnknownVal(cty.Set(cty.String)).Refine().
+				CollectionLengthLowerBound(1).
+				CollectionLengthUpperBound(2).
+				NewValue(),
+		},
+		{
+			Value: cty.UnknownVal(cty.Tuple([]cty.Type{cty.String, cty.String})).RefineNotNull(),
+			Type:  cty.Set(cty.String),
+			Want: cty.UnknownVal(cty.Set(cty.String)).Refine().
+				NotNull().
+				CollectionLengthLowerBound(1).
+				CollectionLengthUpperBound(2).
+				NewValue(),
+		},
+
+		// Collection to collection refinements
+		{
+			Value: cty.UnknownVal(cty.List(cty.String)).Refine().
+				CollectionLengthLowerBound(2).
+				CollectionLengthUpperBound(4).
+				NewValue(),
+			Type: cty.Set(cty.String),
+			Want: cty.UnknownVal(cty.Set(cty.String)).Refine().
+				CollectionLengthLowerBound(1).
+				CollectionLengthUpperBound(4).
+				NewValue(),
+		},
+		{
+			Value: cty.UnknownVal(cty.List(cty.String)).Refine().
+				NotNull().
+				CollectionLengthLowerBound(2).
+				CollectionLengthUpperBound(4).
+				NewValue(),
+			Type: cty.Set(cty.String),
+			Want: cty.UnknownVal(cty.Set(cty.String)).Refine().
+				NotNull().
+				CollectionLengthLowerBound(1).
+				CollectionLengthUpperBound(4).
+				NewValue(),
+		},
+		{
+			Value: cty.UnknownVal(cty.Set(cty.String)).Refine().
+				CollectionLengthLowerBound(2).
+				CollectionLengthUpperBound(4).
+				NewValue(),
+			Type: cty.List(cty.String),
+			Want: cty.UnknownVal(cty.List(cty.String)).Refine().
+				CollectionLengthLowerBound(2).
+				CollectionLengthUpperBound(4).
+				NewValue(),
+		},
+		{
+			Value: cty.UnknownVal(cty.Set(cty.String)).Refine().
+				NotNull().
+				CollectionLengthLowerBound(2).
+				CollectionLengthUpperBound(4).
+				NewValue(),
+			Type: cty.List(cty.String),
+			Want: cty.UnknownVal(cty.List(cty.String)).Refine().
+				NotNull().
+				CollectionLengthLowerBound(2).
+				CollectionLengthUpperBound(4).
+				NewValue(),
+		},
+
+		// General unknown value refinements
+		{
+			Value: cty.UnknownVal(cty.Bool).RefineNotNull(),
+			Type:  cty.String,
+			Want:  cty.UnknownVal(cty.String).RefineNotNull(),
+		},
 	}
 
 	for _, test := range tests {

cty/ctystrings/doc.go

@@ -0,0 +1,26 @@
+// Package ctystrings is a collection of string manipulation utilities which
+// intend to help application developers implement string-manipulation
+// functionality in a way that respects the cty model of strings, even when
+// they are working in the realm of Go strings.
+//
+// cty strings are, internally, NFC-normalized as defined in Unicode Standard
+// Annex #15 and encoded as UTF-8.
+//
+// When working with [cty.Value] of string type cty manages this
+// automatically as an implementation detail, but when applications call
+// [Value.AsString] they will receive a value that has been subjected to that
+// normalization, and so may need to take that normalization into account when
+// manipulating the resulting string or comparing it with other Go strings
+// that did not originate in a [cty.Value].
+//
+// Although the core representation of [cty.String] only considers whole
+// strings, it's also conventional in other locations such as the standard
+// library functions to consider strings as being sequences of grapheme
+// clusters as defined by Unicode Standard Annex #29, which adds further
+// rules about combining multiple consecutive codepoints together into a
+// single user-percieved character. Functions that work with substrings should
+// always use grapheme clusters as their smallest unit of splitting strings,
+// and never break strings in the middle of a grapheme cluster. The functions
+// in this package respect that convention unless otherwise stated in their
+// documentation.
+package ctystrings

cty/ctystrings/normalize.go

@@ -0,0 +1,14 @@
+package ctystrings
+
+import (
+	"golang.org/x/text/unicode/norm"
+)
+
+// Normalize applies NFC normalization to the given string, returning the
+// transformed string.
+//
+// This function achieves the same effect as wrapping a string in a value
+// using [cty.StringVal] and then unwrapping it again using [Value.AsString].
+func Normalize(str string) string {
+	return norm.NFC.String(str)
+}