Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor(formula): align with template-formula v3.0.0 #12

Merged
merged 9 commits into from
Jul 19, 2019
Merged

refactor(formula): align with template-formula v3.0.0 #12

merged 9 commits into from
Jul 19, 2019

Conversation

noelmcloughlin
Copy link
Member

@noelmcloughlin noelmcloughlin commented Jun 16, 2019

This PR is a migration to template-formula v3.0.0. One extra feature added was "optionally install from source tarball". Some unit tests were added.

Verified on Ubuntu with/without pillars. No regression. formula needs further improvements.

sysstat

  ID: sysstat-package-install-pkg-installed
    Function: pkg.installed
        Name: sysstat
      Result: True
     Comment: All specified packages are already installed
     Started: 13:00:56.185501
    Duration: 467.652 ms
     Changes:   
----------
          ID: sysstat-config-file-file-managed-config_file
    Function: file.managed
        Name: /etc/default/sysstat
      Result: True
     Comment: File /etc/default/sysstat updated
     Started: 13:00:56.655645
    Duration: 29.12 ms
     Changes:   
              ----------
              diff:
                  --- 
                  +++ 
                  @@ -8,3 +8,11 @@
                   # will be overwritten by debconf!
                   ENABLED="true"
                   
                  +# Additional options passed to sa1 by /etc/init.d/sysstat
                  +# and /etc/cron.d/sysstat
                  +# By default contains the `-S DISK' option responsible for
                  +# generating disk statisitcs.
                  +SA1_OPTIONS="-S DISK"
                  +
                  +# Additional options passed to sa2 by /etc/cron.daily/sysstat.
                  +SA2_OPTIONS=""
              mode:
                  0640
----------
          ID: sysstat-config-file-file-managed-environ_file
    Function: file.managed
        Name: /etc/default/sysstat.sh
      Result: True
     Comment: File /etc/default/sysstat.sh is in the correct state
     Started: 13:00:56.684914
    Duration: 16.641 ms
     Changes:   
----------
          ID: sysstat-service-running-service-running
    Function: service.running
        Name: sysstat
      Result: True
     Comment: Service sysstat has been enabled, and is running
     Started: 13:00:56.702387
    Duration: 426.275 ms
     Changes:   
              ----------
              sysstat:
                  True

Summary for local
------------
Succeeded: 4 (changed=2)
Failed:    0

from source tarball

          ID: sysstat-package-source-install-file-directory
    Function: file.directory
        Name: /tmp/sysstat/sysstat-12.1.5
      Result: True
     Comment: Directory /tmp/sysstat/sysstat-12.1.5 updated
     Started: 13:28:15.297047
    Duration: 805.568 ms
     Changes:   
              ----------
              mode:
                  0755
----------
          ID: sysstat-package-source-install-source-extracted
    Function: archive.extracted
        Name: /tmp/sysstat/sysstat-12.1.5
      Result: True
     Comment: All files in archive are already present
     Started: 13:28:16.103208
    Duration: 284.55 ms
     Changes:   
----------
          ID: sysstat-package-source-install-cmd-run-make-install
    Function: cmd.run
        Name: ./configure
      Result: True
     Comment: Command "./configure" run
     Started: 13:28:16.388545
    Duration: 1878.988 ms
     Changes:   
              ----------
              pid:
                  6481
              retcode:
                  0
              stderr:
                  ./configure: line 4088: --variable=systemdsystemunitdir: command not found
              stdout:
                  .
                  Check programs:
                  .
                  checking for gcc... gcc
                  checking whether the C compiler works... yes
                  checking for C compiler default output file name... a.out
                  checking for suffix of executables... 
                  checking whether we are cross compiling... no
<< CUT >>>
                  checking whether documentation should be installed... yes
                  checking whether object files should be stripped... yes
                  .
                  Now create files:
                  .
                  configure: creating ./config.status
                  config.status: creating sa1
                  config.status: creating sa2
                  config.status: creating cron/crontab
                  config.status: creating sysstat.sysconfig
                  config.status: creating version.h
                  config.status: creating sysconfig.h
                  config.status: creating cron/sysstat.cron.daily
                  config.status: creating cron/sysstat.cron.hourly
                  config.status: creating cron/sysstat.crond
                  config.status: creating cron/sysstat.crond.sample.in
                  config.status: creating sysstat
<< CUT >>>
                  config.status: creating tests/rt00020
                  config.status: creating Makefile
                  
                     Sysstat version:		12.1.5
                     Installation prefix:		/usr/local
                     rc directory:		/etc
                     Init directory:		/etc/init.d
                     Systemd unit dir:		
                     Configuration directory:	/etc/sysconfig
                     Man pages directory:		${datarootdir}/man
                     Compiler:			gcc
                     Compiler flags:		-g -O2
----------
          ID: sysstat-package-source-install-cmd-run-make-install
    Function: cmd.run
        Name: make
      Result: True
     Comment: Command "make" run
     Started: 13:28:18.267837
    Duration: 7820.508 ms
     Changes:   
              ----------
              pid:
                  8014
              retcode:
                  0
              stderr:
                  ar: creating librdstats.a
                  ar: creating librdsensors.a
                  ar: creating librdstats_light.a
                  ar: creating libsyscom.a
                  svg_stats.c: In function 'svg_print_pwr_cpufreq_stats':
                  svg_stats.c:4296:25: warning: '%d' directive writing between 1 and 10 bytes into a region of size 8 [-Wformat-overflow=]
                       sprintf(item_name, "%d", i - 1);
                                           ^~
              ..... <<< CUT >>>
                  gcc -o cifsiostat.o -c -g -O2 -Wall -Wstrict-prototypes -pipe -O2  -DSA_DIR=\"/var/log/sa\" -DSADC_PATH=\"/usr/local/lib/sa/sadc\"   -DHAVE_SYS_SYSMACROS_H -DHAVE_SYS_PARAM_H cifsiostat.c
                  gcc -o cifsiostat -g -O2 -Wall -Wstrict-prototypes -pipe -O2 cifsiostat.o librdstats_light.a libsyscom.a -s
----------
          ID: sysstat-package-source-install-cmd-run-make-install
    Function: cmd.run
        Name: make install
      Result: True
     Comment: Command "make install" run
     Started: 13:28:26.088653
    Duration: 61.539 ms
     Changes:   
              ----------
              pid:
                  8159
              retcode:
                  0
              stderr:
              stdout:
                  mkdir -p /usr/local/share/man/man1
                  mkdir -p /usr/local/share/man/man5
                  mkdir -p /usr/local/share/man/man8
                  rm -f /usr/local/share/man/man8/sa1.8*
                  install -m 644 -g man man/sa1.8 /usr/local/share/man/man8
                  rm -f /usr/local/share/man/man8/sa2.8*
                  install -m 644 -g man man/sa2.8 /usr/local/share/man/man8
                  rm -f /usr/local/share/man/man8/sadc.8*
                  install -m 644 -g man man/sadc.8 /usr/local/share/man/man8
                  rm -f /usr/local/share/man/man1/sar.1*
                  install -m 644 -g man man/sar.1 /usr/local/share/man/man1
                  rm -f /usr/local/share/man/man1/sadf.1*
                  install -m 644 -g man man/sadf.1 /usr/local/share/man/man1
                  rm -f /usr/local/share/man/man5/sysstat.5*
                  install -m 644 -g man man/sysstat.5 /usr/local/share/man/man5
                  rm -f /usr/local/share/man/man1/iostat.1*
                  install -m 644 -g man man/iostat.1 /usr/local/share/man/man1
                  rm -f /usr/local/share/man/man1/tapestat.1*
                  install -m 644 -g man man/tapestat.1 /usr/local/share/man/man1
<< CUT >>>

----------
          ID: sysstat-config-file-file-managed-config_file
    Function: file.managed
        Name: /etc/default/sysstat
      Result: True
     Comment: File /etc/default/sysstat is in the correct state
     Started: 13:28:26.150536
    Duration: 18.719 ms
     Changes:   
----------
          ID: sysstat-config-file-file-managed-environ_file
    Function: file.managed
        Name: /etc/default/sysstat.sh
      Result: True
     Comment: File /etc/default/sysstat.sh updated
     Started: 13:28:26.169422
    Duration: 18.078 ms
     Changes:   
              ----------
              diff:
                  --- 
                  +++ 
                  @@ -2,3 +2,4 @@
                   # File managed by Salt at <salt://sysstat/files/default/sysstat.sh.jinja>.
                   # Your changes may be overwritten.
                   ########################################################################
                  +export PATH=${PATH}:/usr/local/bin
----------
          ID: sysstat-service-running-service-running
    Function: service.running
        Name: sysstat
      Result: True
     Comment: The service sysstat is already running
     Started: 13:28:26.422392
    Duration: 27.741 ms
     Changes:   

Summary for local
------------
Succeeded: 8 (changed=5)

** clean **

          ID: sysstat-service-clean-service-dead
    Function: service.dead
        Name: sysstat
      Result: True
     Comment: The service sysstat is already dead
     Started: 12:39:53.706593
    Duration: 27.452 ms
     Changes:   
----------
          ID: sysstat-package-clean-pkg-removed
    Function: pkg.removed
        Name: sysstat
      Result: True
     Comment: All specified packages are already absent
     Started: 12:39:53.861096
    Duration: 438.859 ms
     Changes:   
----------
          ID: sysstat-package-source-clean-file-absent
    Function: file.absent
        Name: /tmp//tmp
      Result: True
     Comment: File /tmp//tmp is not present
     Started: 12:39:54.302153
    Duration: 0.304 ms
     Changes:   
----------
          ID: sysstat-package-source-clean-file-absent
    Function: file.absent
        Name: /usr/local/lib/sa
      Result: True
     Comment: File /usr/local/lib/sa is not present
     Started: 12:39:54.302595
    Duration: 0.208 ms
     Changes:   
----------
          ID: sysstat-package-source-clean-file-absent
    Function: file.absent
        Name: /usr/local/bin/sar
      Result: True
     Comment: File /usr/local/bin/sar is not present
     Started: 12:39:54.302884
    Duration: 0.251 ms
     Changes:   
----------
          ID: sysstat-package-source-clean-file-absent
    Function: file.absent
        Name: /usr/local/bin/sadf
      Result: True
     Comment: File /usr/local/bin/sadf is not present
     Started: 12:39:54.303224
    Duration: 0.196 ms
     Changes:   
----------
          ID: sysstat-package-source-clean-file-absent
    Function: file.absent
        Name: /usr/local/bin/iostat
      Result: True
     Comment: File /usr/local/bin/iostat is not present
     Started: 12:39:54.303500
    Duration: 0.187 ms
     Changes:   
----------
          ID: sysstat-package-source-clean-file-absent
    Function: file.absent
        Name: /usr/local/bin/tapestat
      Result: True
     Comment: File /usr/local/bin/tapestat is not present
     Started: 12:39:54.303767
    Duration: 0.187 ms
     Changes:   
----------
          ID: sysstat-package-source-clean-file-absent
    Function: file.absent
        Name: /usr/local/bin/mpstat
      Result: True
     Comment: File /usr/local/bin/mpstat is not present
     Started: 12:39:54.304031
    Duration: 0.184 ms
     Changes:   
----------
          ID: sysstat-package-source-clean-file-absent
    Function: file.absent
        Name: /usr/local/bin/pidstat
      Result: True
     Comment: File /usr/local/bin/pidstat is not present
     Started: 12:39:54.304293
    Duration: 0.19 ms
     Changes:   
----------
          ID: sysstat-package-source-clean-file-absent
    Function: file.absent
        Name: /usr/local/bin/cifsiostat
      Result: True
     Comment: File /usr/local/bin/cifsiostat is not present
     Started: 12:39:54.304558
    Duration: 0.183 ms
     Changes:   
----------
          ID: sysstat-package-source-clean-file-absent
    Function: file.absent
        Name: /etc/sysconfig/sysstat.ioconf
      Result: True
     Comment: File /etc/sysconfig/sysstat.ioconf is not present
     Started: 12:39:54.304819
    Duration: 0.183 ms
     Changes:   
----------
          ID: sysstat-package-source-clean-file-absent
    Function: file.absent
        Name: /etc/sysconfig/sysstat/sysstat
      Result: True
     Comment: File /etc/sysconfig/sysstat/sysstat is not present
     Started: 12:39:54.305092
    Duration: 0.179 ms
     Changes:   
----------
          ID: sysstat-package-source-clean-file-absent
    Function: file.absent
        Name: /usr/local/share/doc/sysstat-12.1.5
      Result: True
     Comment: File /usr/local/share/doc/sysstat-12.1.5 is not present
     Started: 12:39:54.305347
    Duration: 0.179 ms
     Changes:   
----------
          ID: sysstat-package-source-clean-file-absent
    Function: file.absent
        Name: /usr/local/share/man/man8/sa1.8*
      Result: True
     Comment: File /usr/local/share/man/man8/sa1.8* is not present
     Started: 12:39:54.305599
    Duration: 0.181 ms
     Changes:   
----------
          ID: sysstat-package-source-clean-file-absent
    Function: file.absent
        Name: /usr/local/share/man/man8/sa2.8*
      Result: True
     Comment: File /usr/local/share/man/man8/sa2.8* is not present
     Started: 12:39:54.305854
    Duration: 0.178 ms
     Changes:   
----------
          ID: sysstat-package-source-clean-file-absent
    Function: file.absent
        Name: /usr/local/share/man/man8/sadc.8*
      Result: True
     Comment: File /usr/local/share/man/man8/sadc.8* is not present
     Started: 12:39:54.306108
    Duration: 0.178 ms
     Changes:   
----------
          ID: sysstat-package-source-clean-file-absent
    Function: file.absent
        Name: /usr/local/share/man/man1/sar.1
      Result: True
     Comment: File /usr/local/share/man/man1/sar.1 is not present
     Started: 12:39:54.306362
    Duration: 0.179 ms
     Changes:   
----------
          ID: sysstat-package-source-clean-file-absent
    Function: file.absent
        Name: /usr/local/share/man/man1/sadf.1
      Result: True
     Comment: File /usr/local/share/man/man1/sadf.1 is not present
     Started: 12:39:54.306614
    Duration: 0.217 ms
     Changes:   
----------
          ID: sysstat-package-source-clean-file-absent
    Function: file.absent
        Name: /usr/local/share/man/man5/sysstat.5
      Result: True
     Comment: File /usr/local/share/man/man5/sysstat.5 is not present
     Started: 12:39:54.306954
    Duration: 0.257 ms
     Changes:   
----------
          ID: sysstat-package-source-clean-file-absent
    Function: file.absent
        Name: /usr/local/share/man/man1/iostat.1
      Result: True
     Comment: File /usr/local/share/man/man1/iostat.1 is not present
     Started: 12:39:54.307331
    Duration: 0.255 ms
     Changes:   
----------
          ID: sysstat-package-source-clean-file-absent
    Function: file.absent
        Name: /usr/local/share/man/man1/tapestat.1
      Result: True
     Comment: File /usr/local/share/man/man1/tapestat.1 is not present
     Started: 12:39:54.307706
    Duration: 0.254 ms
     Changes:   
----------
          ID: sysstat-package-source-clean-file-absent
    Function: file.absent
        Name: /usr/local/share/man/man1/mpstat.1
      Result: True
     Comment: File /usr/local/share/man/man1/mpstat.1 is not present
     Started: 12:39:54.308080
    Duration: 0.255 ms
     Changes:   
----------
          ID: sysstat-package-source-clean-file-absent
    Function: file.absent
        Name: /usr/local/share/man/man1/pidstat.1
      Result: True
     Comment: File /usr/local/share/man/man1/pidstat.1 is not present
     Started: 12:39:54.308454
    Duration: 0.24 ms
     Changes:   
----------
          ID: sysstat-package-source-clean-file-absent
    Function: file.absent
        Name: /usr/local/share/man/man1/cifsiostat.1
      Result: True
     Comment: File /usr/local/share/man/man1/cifsiostat.1 is not present
     Started: 12:39:54.308819
    Duration: 0.198 ms
     Changes:   
----------
          ID: sysstat-package-source-clean-file-absent
    Function: file.absent
        Name: /var/log/sa
      Result: True
     Comment: File /var/log/sa is not present
     Started: 12:39:54.309093
    Duration: 0.191 ms
     Changes:   

Summary for local
-------------
Succeeded: 26

@baby-gnu
Copy link
Contributor

Hello.

Thanks for your work @noelmcloughlin

I must admin than reviewing 54 files change in one commit is quite many to me.

I'll take time to do it locally in my GNU Emacs.

Regards.

@myii
Copy link
Member

myii commented Jun 19, 2019

@noelmcloughlin Could this be done in separate blocks? I.e. One PR specific to semantic-release, another for the formula changes, etc.?

@noelmcloughlin
Copy link
Member Author

noelmcloughlin commented Jun 19, 2019

@baby-gnu, I did not change 54 files - that is misleading claim. The community changed nearly ~50 files in template-formula over past few months so adopting the template formula updates a repo.

I changed 7, added 5, deleted 2 approximately.

Yes, the PR could have been split.
PR1) Migrate to template-formula v3.0.0 . (~7 files modified, ~2 deleted)
PR2) Add archive handling (~5 files added)

An additional ~5 files have modifications beyond template-formula v3.0.0 state. I'm not familiar with both sysstat and template formulas. Some issues maybe template-formula (i.e. saltstack-formulas/prometheus-formula#3) and some sysstat.

The formula has no CI/CD setup so testing is manual and takes a long time.
Breaking up work items like is possible - sometimes I am very good at small PR & sometimes not.

If migrating to template-formula vX.X.X requires reviewing code and patterns already reviewed in template-formula that's is double work - and the effort is doomed.

I will try to avoid the bigger PR but its a balance between ideal PR V.S. life, circumstances, and supporting multiple OS/features and expectations.

@myii
Copy link
Member

myii commented Jun 19, 2019

@noelmcloughlin I appreciate what you're saying. Reviewing PRs is also hard work and the same situations apply. I've been really busy recently and I can't find the time to review in the way I usually do. So PRs that only do one thing are much easier to deal with.

I've collected a list of all of the semantic-release PRs so far, with the hope that some documentation can be collected by comparing the common bits in each one. See the list under saltstack-formulas/template-formula#21.

In the meantime, let's leave this PR as-is and I'll try to get through it over a few days, going through each part.

@noelmcloughlin
Copy link
Member Author

noelmcloughlin commented Jun 19, 2019

I will update the PR description with inventory of where the changes originate from.

Filenames originating from template-formula technically do not need a review;

During migration I'm raising issues at template-formula and trying to workaround them. I need to raise one issue (saltstack-formulas/prometheus-formula#3) at template-formula because it is a generic issue. just have not got around to it.

I apologise for not splitting my commits better ;-)
I will try to enforce this practice on myself going forward.

Copy link
Member

@myii myii left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@noelmcloughlin This initial review is based on a Travis run and a diff with the template-formula. I haven't checked the main formula changes themselves.

  • https://travis-ci.org/myii/sysstat-formula/jobs/550186931#L1322
    • jinja2.exceptions.TemplateNotFound: sysstat/osmap.yaml
  • Unnecessary since created automatically:
    • docs/AUTHORS.rst
    • docs/CHANGELOG.rst
  • Add:
    • bin/kitchen and add section to README
    • docs/CONTRIBUTING.rst -- relevant sections
    • docs/TOFS_pattern.rst
  • Merge:
    • .gitignore
    • .travis.yml
    • Gemfile
    • kitchen.yml

You may want to change the commit to be of type feat since this is more of a minor bump than a patch.

@noelmcloughlin
Copy link
Member Author

@myii I fixed unit test and tested manually on Ubuntu (sysstat and systat.clean without and with true/false pillars) - looks good.

One Debian 9 Travis Job is failing: https://travis-ci.com/saltstack-formulas/sysstat-formula/jobs/210822474 and issue seems to be identical to closed: saltstack/salt#44980 (comment) Anyone ideas? @javierbertoli maybe

@noelmcloughlin
Copy link
Member Author

One Debian 9 Travis Job is failing: https://travis-ci.com/saltstack-formulas/sysstat-formula/jobs/210822474 and issue seems to be identical to closed: saltstack/salt#44980 (comment) Anyone ideas? @javierbertoli maybe

Four travis CI jobs pass. Two have that error.

.gitignore Outdated Show resolved Hide resolved
@myii
Copy link
Member

myii commented Jun 26, 2019

@noelmcloughlin This commit will help you get the other two instances working:

* myii@edbf72c

You can see the result here:

* https://travis-ci.org/myii/sysstat-formula/builds/550601334

UPDATE: Thanks to @javierbertoli, these are now provided by default in the pre-salted images.


The default-opensuse-leap-42-2018-3-py2 is now only failing on one of the inspec tests since the service isn't being started.

@myii
Copy link
Member

myii commented Jun 28, 2019

@noelmcloughlin As can be seen from the latest run, the final state sysstat-service-running-service-running is not running for the default-opensuse-leap-42-2018-3-py2 instance. Probably a change required in map.jinja.

@noelmcloughlin
Copy link
Member Author

Probably a change required in map.jinja.

thanks @Aboe and @myii for reviewing. Let me check - might be #14 related

@noelmcloughlin
Copy link
Member Author

thanks @Aboe and @myii for reviewing. Let me check - might be #14 related

Yes the osfamilymap.yaml was wrong - fixed.

Copy link
Member

@myii myii left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some changes required, still only general formula structure, more to do with semantic-release PRs. The actual formula changes can be checked after these changes are made.

Alongside these changes, a couple still from this comment:

  • Unnecessary since created automatically:
    • docs/AUTHORS.rst
    • docs/CHANGELOG.rst

docs/_static/css/custom.css Outdated Show resolved Hide resolved
docs/CONTRIBUTING.rst Show resolved Hide resolved
docs/CONTRIBUTING_DOCS.rst Outdated Show resolved Hide resolved
docs/README.rst Outdated Show resolved Hide resolved
docs/README.rst Outdated Show resolved Hide resolved
docs/conf.py Outdated Show resolved Hide resolved
docs/index.rst Outdated Show resolved Hide resolved
FORMULA Outdated Show resolved Hide resolved
Copy link
Member

@myii myii left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for all of the changes, just one that went wrong because my explanation was bad (sorry!). In the meantime, I'll start checking the main formula changes themselves.

docs/CONTRIBUTING.rst Show resolved Hide resolved
@myii
Copy link
Member

myii commented Jun 29, 2019

@noelmcloughlin Thanks, just need to find some time to check the main formula changes themselves...

@myii
Copy link
Member

myii commented Jul 3, 2019

@noelmcloughlin Like saltstack-formulas/deepsea-formula#10, I don't want to hold you up with the merge. I've managed to check the semantic-release components but I'll leave the rest in your capable hands. If anybody else is able to check the formula changes, that would be great. Thanks for your work!

@noelmcloughlin
Copy link
Member Author

Thanks @myii and @Aboe for reviewing.

@noelmcloughlin noelmcloughlin merged commit 43923a4 into saltstack-formulas:master Jul 19, 2019
@noelmcloughlin noelmcloughlin deleted the template-formula branch July 19, 2019 21:15
@saltstack-formulas-travis

🎉 This PR is included in version 0.2.0 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants