Validation of Deferred multivalued parameters fails #9438

geek876 · 2024-07-30T10:17:23Z

Describe the Bug

Upgraded puppet agent from 7.30.0 to 8.7.0. Started to get this error for deferred functions

Error: Failed to apply catalog: Parameter environment failed on Exec[/usr/sbin/create-admin.sh]: Validate method failed for class environment: undefined method `=~' for #<Puppet::Pops::Evaluator::DeferredValue:0x00007ff6913f7908 @proc=#<Proc:0x00007ff6913f7930 /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/deferred_resolver.rb:210>> (file: /<path>/manifests/init.pp, line: 65)
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/type/exec.rb:308:in `block (4 levels) in <module:Puppet>'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/type/exec.rb:307:in `each'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/type/exec.rb:307:in `block (3 levels) in <module:Puppet>'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parameter.rb:479:in `validate'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parameter.rb:514:in `value='
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/type.rb:690:in `[]='
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/type.rb:2451:in `block in set_parameters'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/type.rb:2446:in `each'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/type.rb:2446:in `set_parameters'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/type.rb:2343:in `initialize'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource.rb:526:in `new'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource.rb:526:in `to_ral'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:611:in `block in to_catalog'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:603:in `each'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:603:in `to_catalog'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:495:in `to_ral'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:120:in `block in convert_catalog'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:518:in `block in thinmark'
/opt/puppetlabs/puppet/lib/ruby/3.2.0/benchmark.rb:311:in `realtime'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:517:in `thinmark'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:114:in `convert_catalog'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:485:in `run_internal'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:338:in `block in run'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:64:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:288:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:337:in `run'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/agent.rb:85:in `block (6 levels) in run'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:64:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:288:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/agent.rb:84:in `block (5 levels) in run'
/opt/puppetlabs/puppet/lib/ruby/3.2.0/timeout.rb:189:in `block in timeout'
/opt/puppetlabs/puppet/lib/ruby/3.2.0/timeout.rb:196:in `timeout'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/agent.rb:83:in `block (4 levels) in run'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/agent/locker.rb:23:in `lock'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/agent.rb:73:in `block (3 levels) in run'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/agent.rb:164:in `with_client'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/agent.rb:69:in `block (2 levels) in run'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/agent.rb:129:in `run_in_fork'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/agent.rb:68:in `block in run'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:174:in `controlled_run'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/agent.rb:49:in `run'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/agent.rb:437:in `onetime'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/agent.rb:394:in `block in run_command'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:64:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:288:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/agent.rb:391:in `run_command'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:423:in `block in run'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:706:in `exit_on_fail'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:423:in `run'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:145:in `run'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:79:in `execute'
/opt/puppetlabs/puppet/bin/puppet:6:in `<main>'

Error persists even after setting preprocess_deferred=false within Agent's puppet.conf
Tried Agent 8.8.1 but same issue.

Expected Behavior

Puppet Apply should finish without any errors

Steps to Reproduce

The following puppet code fails

  $admin_password_sensetive = Deferred('test::lookup', ["path1/path2/${::role}/admin_password", 'content'])
  $admin_password = Deferred('unwrap', [$admin_password_sensetive])

  file { '/usr/sbin/create-admin.sh':
    owner  => 'root',
    group  => 'root',
    mode   => '0755',
    source => 'puppet:///modules/some_module/create-admin.sh',
  } -> exec { '/usr/sbin/create-admin.sh':
    timeout     => 600,
    environment => [
      Deferred('inline_epp', ['ADMIN_PASSWORD=<%= $admin_password %>', { 'admin_password' => $admin_password }]),
    ],
    creates     => '/etc/some_file',
  }

Environment

Agent-Version: 8.7.0
Puppet-Server Version: puppetserver-7.13.0-1.el9.noarch

Additional Context

Add any other context about the problem here.

The text was updated successfully, but these errors were encountered:

joshcooper · 2024-07-31T19:11:55Z

The issue occurs because environment is set to an array of strings so the special handling we added in a76f498 doesn't detect that the value contains a deferred value. Passing the environment value as a string works as expected, but triggers a warning due to the parameter containing a Sensitive value, so it can't be automatically redacted:

❯ cat defer3.pp 
$admin_password = Deferred('new', [Sensitive, 'opensesame'])
exec { '/usr/bin/echo $ADMIN_PASSWORD':
  logoutput => true,
  environment => 
    Deferred('inline_epp', ['ADMIN_PASSWORD=<%= $admin_password %>', { 'admin_password' => $admin_password }])
}
❯ bundle exec puppet apply defer3.pp
Notice: Compiled catalog for localhost in environment production in 0.05 seconds
Warning: /Exec[/usr/bin/echo $ADMIN_PASSWORD]: Unable to mark 'environment' as sensitive: environment is a parameter and not a property, and cannot be automatically redacted.
Notice: /Stage[main]/Main/Exec[/usr/bin/echo $ADMIN_PASSWORD]/returns: opensesame
Notice: /Stage[main]/Main/Exec[/usr/bin/echo $ADMIN_PASSWORD]/returns: executed successfully
Notice: Applied catalog in 0.03 seconds

We should correctly detect when puppet parameters are mulltivalued and at least one of the value is Deferred

github-actions · 2024-07-31T19:12:33Z

Migrated issue to PUP-12068

geek876 · 2024-07-31T23:19:05Z

@joshcooper thanks.
Can I get some clarification on this.
v7.32.1 and v8.7.0 both has default value of true. So what does it exactly mean by
The previous behavior can be enabled by setting preprocess_deferred=false in puppet.conf on each agent

I actually checked the default value of preprocess_deferred on v8 and it turns out it is false (not true as per the doc)

[x]# grep preprocess_deferred /etc/puppetlabs/puppet/puppet.conf
#preprocess_deferred = true
[x]# systemctl restart puppet
[x]# puppet config print preprocess_deferred
false
[x]# puppet --version
8.7.0

joshcooper · 2024-07-31T23:31:48Z

The puppet 8 docs are wrong puppetlabs/puppet-docs#1166

If an `exec` resource's `environment` parameter was specified as an array of values, where at least one of the values was deferred, then validation failed with: undefined method `=~' for #<Puppet::Pops::Evaluator::DeferredValue:0x00007ff6913f7908 This occurred because the transaction's `resolved_resolve` assumed the parameter's value was single-valued. As a result, the deferred value was never resolved before calling the type's `validate` method. This updates the transaction to support multi-valued parameters. All deferred values will be resolved before the type's `validate` method is called. Fixes puppetlabs#9438

geek876 added the bug Something isn't working label Jul 30, 2024

joshcooper changed the title ~~Puppet-Agent v8.7.0 failing for Deferred Functions~~ Validation of Deferred multivalued parameters fails Jul 31, 2024

joshcooper added the triaged Jira issue has been created for this label Jul 31, 2024

joshcooper mentioned this issue Jul 31, 2024

Deferred functions sometimes resolve too late with preprocess_deferred=false #9423

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Validation of Deferred multivalued parameters fails #9438

Validation of Deferred multivalued parameters fails #9438

geek876 commented Jul 30, 2024 •

edited by joshcooper

Loading

joshcooper commented Jul 31, 2024

github-actions bot commented Jul 31, 2024

geek876 commented Jul 31, 2024 •

edited

Loading

joshcooper commented Jul 31, 2024

Validation of Deferred multivalued parameters fails #9438

Validation of Deferred multivalued parameters fails #9438

Comments

geek876 commented Jul 30, 2024 • edited by joshcooper Loading

Describe the Bug

Expected Behavior

Steps to Reproduce

Environment

Additional Context

joshcooper commented Jul 31, 2024

github-actions bot commented Jul 31, 2024

geek876 commented Jul 31, 2024 • edited Loading

joshcooper commented Jul 31, 2024

geek876 commented Jul 30, 2024 •

edited by joshcooper

Loading

geek876 commented Jul 31, 2024 •

edited

Loading