Allow string operations on nodes with custom tags

[Slooz]

Please describe your feature request.
I have to modify a CloudFormation template that uses intrinsic functions. Intrinsic functions are prepended with an exclamation point. yq treats the intrinsic functions as custom tags and prevents me from performing string operations.

Describe the solution you'd like
For example:

Resources:
  S3Bucket:
    Type: AWS::CloudFormation::Stack
    Properties:
      BucketName: !Ref MyBucketName

And we run a command:

yq eval '.Resources.S3Bucket.Properties.BucketName += "A"'

it would output

Resources:
  S3Bucket:
    Type: AWS::CloudFormation::Stack
    Properties:
      BucketName: !Ref MyBucketNameA

Describe alternatives you've considered
Convert to a !!str and then back to a !Ref

Additional context
yq shows an error:

Error: !Ref cannot be added to !!str

I think yq should act as if custom tags were strings. From my point of view, !Ref MyBucketName may as well just be a string, but yq treats !Ref as a tag and MyBucketName as a value of type !Ref, which prevents me from performing any string operations, even though !Ref MyBucketName is really just a string.

[mikefarah]

I'm not convinced about having a default string behavior for custom tags, however I have added a new with operator in 4.13.0 that should make convert to/from tags less cumbersome

yq e `with(.Resources.S3Bucket.Properties.BucketName` ; . tag= "!!str" | . += "A" | .tag = "!Ref")' file.yaml

[Slooz]

Thank you for your response.

The with operator does look useful, and it allows me to simplify my query, but I was already using the as operator, which is now the ref operator from #934, which basically gave me the same functionality.

I was hoping to avoid the . tag= "!!str" | ... | .tag = "!Ref" syntax. However, after doing some more reading about YAML tags, I agree that there shouldn't be a default string behavior for custom tags.

Maybe there could be functionality to declare a custom tag as a type of another tag within the query? As a crude example:

yq e '"!Ref": "!!str" | .Resources.S3Bucket.Properties.BucketName += "A" file.yaml

The Red Hat YAML Language Server has similar functionality that is used for CloudFormation development:

yaml.customTags: Array of custom tags that the parser will validate against. It has two ways to be used. Either an item in the array is a custom tag such as "!Ref" and it will automatically map !Ref to scalar or you can specify the type of the object !Ref should be e.g. "!Ref sequence". The type of object can be either scalar (for strings and booleans), sequence (for arrays), map (for objects).

https://github.com/redhat-developer/yaml-language-server#language-server-settings

I am by no means a YAML expert, so I may not understand the full context of this issue and will defer to you 😃

[mikefarah]

That's a possibility - I'm going to leave this open to see how common this sort of problem is

[mikefarah]

Closing due to inactivity

[Slooz]

FWIW, I think the fact that the Red Hat YAML Language Server implemented a feature for this serves as enough proof for the value of this feature. The only reason that this issue hasn't had any activity is because not many people try to use yq with CloudFormation. Besides, is inactivity a good reason to close an issue? The following issues haven't had activity yet remain open:
#301
#317
#407

Instead of being able to declare a custom tag as a type of another tag, how about a feature to disable yq from parsing custom tags in the first place?

In YAML, ! indicates a custom type. But in CloudFormation, ! is simply a short form for CloudFormation intrinsic functions:

https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/intrinsic-function-reference-ref.html#intrinsic-function-reference-ref-syntax.yaml

In CloudFormation, there is no difference between the following syntaxes:

!Ref MyBucketName

Ref: MyBucketName

The CloudFormation documentation never refers to these intrinsic functions as YAML custom tags. You can blame CloudFormation for abusing the YAML syntax, but the truth is that YAML custom tags are barely used by anyone. Yet yq is deliberately making the more common use case more difficult.

It's even more interesting because yq started supporting custom tags because of CloudFormation:
#303

The above issue asked about "performing writes whilst preserving string styles and tags." But instead, the implemented feature prevented writes to custom tags.

I searched through existing yq issues, and I couldn't find any issues that even mention custom tags. So yq has extra logic to prevent something that nobody asked for in the first place?

I appreciate your work, but I disagree with how yq handles this. The current implementation is more faithful to the original intent of the YAML schema, but in reality I think that this implementation just causes headaches for most people, especially those using CloudFormation.

[mikefarah]

Those are still open as I think they make sense to have in a yaml processing tool - the #303 fix was to stop dropping custom tags - again something that a yaml processing tool should do.

I'm not sure what you mean about extra logic to prevent something? The + operation has logic for maps, arrays, strings and number types.

It still seems very weird to me to treat custom types as strings, I'm happy to keep this issue open - if more people with headaches complain too then I'll consider myself outvoted :)

[mikefarah]

What's the redhat feature that you mentioned?

[Slooz]

I haven't looked at the code, but yq prevents the use of the + operation on custom types. I understand the intent behind this, and it makes complete reasonable sense. But the issue is that this really limits how you can interact with CloudFormation YAML without jumping through hoops.

I do understand the rationale behind the current implementation. In YAML, !Ref is a custom type. Thus, the current implementation makes sense.

However, in CloudFormation, there really is no such thing as custom types. CloudFormation uses !Ref as a notation for dynamic operations at runtime. CloudFormation sees !Ref MyBucketName and replaces it with BucketNameA. In CloudFormation, !Ref isn't a custom type, it's just the syntax that CloudFormation chose for these kinds of operations.

To further demonstrate this, the !Sub intrinsic function actually accepts an array or a string. !Sub doesn't indicate a certain data type, it's just notation to run a function at runtime:


https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/intrinsic-function-reference-sub.html#intrinsic-function-reference-sub-syntax.yaml

I realize that this poses a problem for yq. Let's use the following cases as an example. We have a case where !Sub is used with a string and one where !Sub is used with an array:

BucketName: !Sub foo

BucketName: !Sub
  - foo1
  - foo2
  - foo3

What should yq do in the following case?

.BucketName += "bar"

Well, what would yq do if the !Sub wasn't there? With a string, it would result in:

BucketName: foobar

With an array, it would result in:

BucketName:
  - foo1
  - foo2
  - foo3
  - bar

I would argue that this is the desired behavior. The !Sub shouldn't affect behavior, it's just that the !Sub should be preserved:

BucketName: !Sub foobar

BucketName: !Sub
  - foo1
  - foo2
  - foo3
  - bar

So it would be great if yq had an option to ignore but preserve "custom types". I think this kind of implementation would be more intelligent than the current implementation, which restricts you from using + unless you temporarily cast it to another type.

[Slooz]

Here are the discussions of the Red Hat YAML Language Server when they addressed the same challenges posed by CloudFormation:
redhat-developer/yaml-language-server#20
redhat-developer/yaml-language-server#77

[mikefarah]

Ok thinking more about this now that I'm back from my break, I think I've got a good approach.

I could make it so that before I add the nodes together, I check the tags. If they are custom, then I'll decode their values and add those instead. Decoding will automatically treat strings values as strings, and numbers as numbers.

I've got it working locally - just wanted to hear your thoughts first though:

Custom types: that are really strings

when custom tags are encountered, yq will try to decode the underlying type.

Given a sample.yml file of:

a: !horse cat
b: !goat _meow

then

yq eval '.a += .b' sample.yml

will output

a: !horse cat_meow
b: !goat _meow

Custom types: that are really numbers

when custom tags are encountered, yq will try to decode the underlying type.

Given a sample.yml file of:

a: !horse 1.2
b: !goat 2.3

then

yq eval '.a += .b' sample.yml

will output

a: !horse 3.5
b: !goat 2.3

I will be able to get this to work for maps and arrays with custom types too...

[Slooz]

I love it. Thank you so much for your work. I truly appreciate it.

[mikefarah]

Thanks for all your feedback and links 👍🏼

[mikefarah]

I'll (aim) have this our in the next release 4.17.3

[mikefarah]

Ok released in 4.18.1 - and thanks for the sponsorship, much appreciated!!