v2.1.0

Summary

The v2.1.0 release of f1 is closer to a complete rewrite, than a feature release. Includes massive
performance, security, usability and maintainability improvements.

Transforming f1 into a modern, high-quality and efficient package.

Highlights

• Massive performance improvements, by redesigning the work scheduling and removing per-iteration
  memory allocations.
  • Performance no longer degrades with time.
  • Significantly more iterations per second, with significantly less resources used:
    • users low(2) concurrency: Up to 33 times more iterations per second (390k -> 12.8 mil) with 1.5x less user cpu time
      and 23x less system CPU time.
    • users high(100) concurrency: Up to 28 times more iterations per second (173k -> 5mil) with 34x
      less system CPU time and better CPU utilisation.
    • constant high(1000) concurrency: can sustain 10,000 IPS with 1000 workers, without dropped
      iterations(previously 20% dropped) while using 300x less CPU time.
• Rework output formatting and support structured logging, including JSON logs with LOG_FORMAT=json,
  log level support with LOG_LEVEL, and providing custom loggers to f1. Migrate logger to
  log/slog with logrus compatibility layer.
• Remove archived, abandoned, single-use and unnecessary dependencies to reduce the package footprint
  and mitigate supply chain attacks.
• Secure the CI pipeline with strict permissions, SHA version locking and CodeQL.
• Test suite now takes seconds(11s) instead of minutes(1m 5s) with better coverage.
• Features:
  • Require go 1.22
  • Support Bash v2 completions
  • Support CPU & Memory profiling with filename completion
  • Support custom start-time for staged tests
• Fixes:
  • exit after a timeout on stuck iterations
  • don't panic on unknown scenario name
  • change t.Log level to Info from Error
  • Fix rounding on iteration-distribution calculations

Detailed

Breaking

• remove tracer
• Remove fluentd integration
• remove --verbose-fail flag

Deprecations

• deprecate GetMetrics()
• deprecate --verbose-fail, t.Logger() and t.NewT

Performance

• allocations: change testing.T iteration value
• allocations: remove per iteration allocations
• allocations/cpu: remove performance impact of progress reporting
• allocations/cpu: don't start a goroutine for each iteration
• cpu: rewrite iteration management to improve throughput
• cpu: rewrite work scheduling to improve performance

Features

• Require go 1.22
• Support Bash v2 completions
• Support CPU & Memory profiling with filename completion
• Support custom start-time for staged tests

Fixes

• exit after a timeout on stuck iterations
• don't panic on unknown scenario name
• change t.Log level to Info from Error
• Fix rounding on iteration-distribution calculations

Usability

• Remove panic logs and replace with errors

Testing

• rewrite prometheus test code to allow parallel tests
• allow running tests in parallel
• fix randomly failing TestRandomDistribution test
• run tests in parallel
• add example cmd to be used for benchmarking
• more readable run_test failures

CI

• Update GH actions
• Use gorelease to check for breaking API changes
• Enable CodeQL
• Ignore cache when running tests

Dependencies

• Remove archived https://github.com/pkg/errors dependency and use stdlib error handling

• Remove archived https://github.com/giantswarm/retry-go test dependency:

  • replace retry lib with avast/retry-go
  • remove avast/retry-go dependency

• Remove github.com/asaskevich/govalidator dependency

• Remove freeport dependency

• Remove obsolete go-atamanand replace with stdlib

• Remove obsolete ggtimer dependecy and replace with stdlib

• Remove go-gaussian and replace with custom and faster implementation

• Remove durafmt dependency and replace with stdlib usage

• Remove google/uuid dependency and replace with stdlib

• Remove logrus_fluent dependency

• Update dependencies to latest versions:

  • update dependencies & remove vendor folder
  • bump dependencies
  • bump github.com/prometheus/common from 0.54.0 to 0.55.0
  • bump github.com/spf13/cobra from 1.8.0 to 1.8.1
  • bump github.com/prometheus/common from 0.53.0 to 0.54.0
  • bump github.com/prometheus/client_golang from 1.19.0 to 1.19.1
  • bump github.com/prometheus/common from 0.52.3 to 0.53.0
  • bump github.com/prometheus/common from 0.52.2 to 0.52.3
  • bump github.com/prometheus/common from 0.51.1 to 0.52.2
  • bump github.com/prometheus/client_model from 0.6.0 to 0.6.1
  • bump github.com/guptarohit/asciigraph from 0.6.0 to 0.7.1
  • bump github.com/prometheus/common from 0.50.0 to 0.51.1
  • bump github.com/guptarohit/asciigraph from 0.5.6 to 0.6.0

Linting

Run golangci-lint as part of CI and enable most linters:

• use golangci-lint & fix formatting
• enable more linters
• enable gochecknoglobals linter
• enable more linters
• bump golangci-lint to 1.57.1
• update golangci-lint to 1.57.2
• bump golangci-lint to 1.58.0
• security: check golangci-lint signatures on installing
• bump golangci-lint to 1.58.2
• enable copyloopvar linter
• bump golangci-lint to 1.59.0
• bump golangci-lint to 1.59.1
• enable testpackage linter

Documentation:

• Cleanup README.md
• Document ENV variables

Cleanup

• remove unused snyk file
• consistent ENV setting handling
• testing: simplify goleak check
• simplify template rendering
• metrics: remove magic strings & panic
• redo gauss calculations
• inject printer to allow output stream redirect
• use constants for flag names
• [allow usi...

v2.0.22

What's Changed

• test: fix grouped labels in the fake prometheus by @damiankaminski-form3 in #146
• feat: optional id label for all metrics by @damiankaminski-form3 in #154

Full Changelog: v2.0.21...v2.0.22

v2.0.21

What's Changed

• feat: add configurable namespace label to prometheus metrics by @damiankaminski-form3 in #145

New Contributors

• @damiankaminski-form3 made their first contribution in #145

Full Changelog: v2.0.20...v2.0.21

v2.0.20

What's Changed

• chore: update all deps to latest by @gkoutsoumpakis-form3 in #144

New Contributors

• @gkoutsoumpakis-form3 made their first contribution in #144

Full Changelog: v2.0.19...v2.0.20

v2.0.19

What's Changed

• Fix gaussian test to properly test and graph non default repeat by @carl-stlaurent-form3 in #137
• Add max failures option (#107) by @ken-che-form3 in #139

New Contributors

• @carl-stlaurent-form3 made their first contribution in #137
• @ken-che-form3 made their first contribution in #139

Full Changelog: v2.0.18...v2.0.19

v2.0.18

What's Changed

• Remove default case to reduce cpu consumption by @andykuszyk-form3 in #132
• feat: gaussian test peak-rate parameter by @blaze-form3 in #134
• fix: bug that made some calculations wrong by @blaze-form3 in #135

New Contributors

• @blaze-form3 made their first contribution in #134

Full Changelog: v2.0.17...v2.0.18

Security fixes

Merge pull request #123 from form3tech-oss/chore-fix-image-cve

fix: updates golang.org/x/image  CVE-2022-41727

v2.0.16

What's Changed

• Bump github.com/guptarohit/asciigraph from 0.5.2 to 0.5.3 by @dependabot in #94
• Bump github.com/prometheus/common from 0.31.0 to 0.32.1 by @dependabot in #89
• Bump go.uber.org/goleak from 1.1.10 to 1.1.12 by @dependabot in #86
• Bump github.com/spf13/cobra from 1.2.1 to 1.3.0 by @dependabot in #90
• Bump github.com/prometheus/client_golang from 1.11.0 to 1.12.1 by @dependabot in #93
• Bump github.com/stretchr/testify from 1.7.0 to 1.7.1 by @dependabot in #96
• update yaml package by @rkorkosz-f3 in #110
• Update readme with the output description by @bpiekny-form3 in #111
• [skip ci] Updating CODEOWNERS file by @jeeves-form3 in #113
• Replace deprecated ioutil by @dmytro-vovk-f3 in #115

New Contributors

• @rkorkosz-f3 made their first contribution in #110
• @bpiekny-form3 made their first contribution in #111
• @dmytro-vovk-f3 made their first contribution in #115

Full Changelog: v2.0.15...v2.0.16

Made metrics usable from outside of the internal package

Use metrics.GetMetrics() to gain access to the internal metrics, and analyse performance programmatically.

Add godoc documentation

Merge pull request #87 from form3tech-oss/andyk-add-godoc

Added package and struct level documentation