Skip to content

remove excessive backup decompression logging#11479

Merged
deepthi merged 5 commits intovitessio:release-15.0from
planetscale:fixbackuplogging
Oct 15, 2022
Merged

remove excessive backup decompression logging#11479
deepthi merged 5 commits intovitessio:release-15.0from
planetscale:fixbackuplogging

Conversation

@rsajwani
Copy link
Contributor

@rsajwani rsajwani commented Oct 13, 2022

Signed-off-by: Rameez Sajwani rameezwazirali@hotmail.com

Description

This is an enhancement where we are doing excessive logging during decompression. Instead of logging warning for every file we moved the warning to every restore we run.

Logs before this change

/usr/local/go/bin/go tool test2json -t /private/var/folders/3q/njsb_8v13978qdmq4j5c25gh0000gn/T/GoLand/___backup_only_test_go.test -test.v -test.paniconexit0 -test.run ^\QTestTabletInitialBackup\E|\QTestTabletBackupOnly\E$
=== RUN   TestTabletInitialBackup
E1014 19:35:33.155605   78850 query.go:83] ExecuteFetch(ALTER TABLE _vt.reparent_journal CHANGE COLUMN master_alias primary_alias VARBINARY(32) NOT NULL) failed: Unknown column 'master_alias' in 'reparent_journal' (errno 1054) (sqlstate 42S22) during query: ALTER TABLE _vt.reparent_journal CHANGE COLUMN master_alias primary_alias VARBINARY(32) NOT NULL
W1014 19:35:57.194316   80804 compression.go:196] engine "pargzip" doesn't support decompression, using "pgzip" instead
W1014 19:35:57.194366   80804 compression.go:196] engine "pargzip" doesn't support decompression, using "pgzip" instead
W1014 19:35:57.194395   80804 compression.go:196] engine "pargzip" doesn't support decompression, using "pgzip" instead
W1014 19:35:57.194483   80804 compression.go:196] engine "pargzip" doesn't support decompression, using "pgzip" instead
W1014 19:35:57.254949   80804 compression.go:196] engine "pargzip" doesn't support decompression, using "pgzip" instead
W1014 19:35:57.255043   80804 compression.go:196] engine "pargzip" doesn't support decompression, using "pgzip" instead
...
<redacted>
...
W1014 19:35:59.288801   80804 mysqld.go:283] MySQL version has built-in upgrade, skipping RunMySQLUpgrade
--- PASS: TestTabletInitialBackup (86.19s)
=== RUN   TestTabletBackupOnly
E1014 19:36:59.548629   85230 backup.go:306] no backup to restore on BackupStorage for directory ks/0. Starting up empty.
--- PASS: TestTabletBackupOnly (45.60s)
PASS

Logs after this change

/usr/local/go/bin/go tool test2json -t /private/var/folders/3q/njsb_8v13978qdmq4j5c25gh0000gn/T/GoLand/___backup_only_test_go.test -test.v -test.paniconexit0 -test.run ^\QTestTabletInitialBackup\E|\QTestTabletBackupOnly\E$
=== RUN   TestTabletInitialBackup
E1014 19:30:32.856072   68553 query.go:83] ExecuteFetch(ALTER TABLE _vt.reparent_journal CHANGE COLUMN master_alias primary_alias VARBINARY(32) NOT NULL) failed: Unknown column 'master_alias' in 'reparent_journal' (errno 1054) (sqlstate 42S22) during query: ALTER TABLE _vt.reparent_journal CHANGE COLUMN master_alias primary_alias VARBINARY(32) NOT NULL
W1014 19:30:58.207658   70513 builtinbackupengine.go:612] engine "pargzip" doesn't support decompression, using "pgzip" instead
W1014 19:31:00.338380   70513 mysqld.go:283] MySQL version has built-in upgrade, skipping RunMySQLUpgrade
--- PASS: TestTabletInitialBackup (86.84s)
=== RUN   TestTabletBackupOnly
E1014 19:32:01.249594   74926 backup.go:306] no backup to restore on BackupStorage for directory ks/0. Starting up empty.
W1014 19:32:14.666754   74926 vtbackup.go:248] Failed to remove temporary tablet directory: unlinkat /Users/rameezsajwani/Code/fork/vitess/vtdataroot/vtroot_8901/vt_1817924007: directory not empty
--- PASS: TestTabletBackupOnly (47.91s)
PASS

Related Issue(s)

closes #11480

Checklist

  • "Backport me!" label has been added if this change should be backported
  • Tests were added or are not required
  • Documentation was added or is not required

Deployment Notes

Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com>
@vitess-bot
Copy link
Contributor

vitess-bot bot commented Oct 13, 2022

Review Checklist

Hello reviewers! 👋 Please follow this checklist when reviewing this Pull Request.

General

  • Ensure that the Pull Request has a descriptive title.
  • If this is a change that users need to know about, please apply the release notes (needs details) label so that merging is blocked unless the summary release notes document is included.

If a new flag is being introduced:

  • Is it really necessary to add this flag?
  • Flag names should be clear and intuitive (as far as possible)
  • Help text should be descriptive.
  • Flag names should use dashes (-) as word separators rather than underscores (_).

If a workflow is added or modified:

  • Each item in Jobs should be named in order to mark it as required.
  • If the workflow should be required, the maintainer team should be notified.

Bug fixes

  • There should be at least one unit or end-to-end test.
  • The Pull Request description should include a link to an issue that describes the bug.

Non-trivial changes

  • There should be some code comments as to why things are implemented the way they are.

New/Existing features

  • Should be documented, either by modifying the existing documentation or creating new documentation.
  • New features should have a link to a feature request issue or an RFC that documents the use cases, corner cases and test cases.

Backward compatibility

  • Protobuf changes should be wire-compatible.
  • Changes to _vt tables and RPCs need to be backward compatible.
  • vtctl command output order should be stable and awk-able.
  • RPC changes should be compatible with vitess-operator
  • If a flag is removed, then it should also be removed from VTop, if used there.

@rsajwani rsajwani self-assigned this Oct 13, 2022
@rsajwani rsajwani added Forwardport to: main This will forward port the PR to the main branch Type: Enhancement Logical improvement (somewhere between a bug and feature) Component: Cluster management labels Oct 13, 2022
Copy link
Collaborator

@deepthi deepthi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

mysql80 unit test is fixed in #11475 (not merged yet)

…gging

Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com>
@rsajwani rsajwani marked this pull request as ready for review October 13, 2022 17:20
@rsajwani rsajwani requested a review from mattlord as a code owner October 13, 2022 17:20
@frouioui frouioui mentioned this pull request Oct 14, 2022
100 tasks
Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com>
…gging

Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com>
Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com>
@deepthi
Copy link
Collaborator

deepthi commented Oct 15, 2022

Can you include the log snippet in the description? That will be verification that this is working as intended.

@rsajwani
Copy link
Contributor Author

Can you include the log snippet in the description? That will be verification that this is working as intended.

I have updated description with the logs...

@deepthi deepthi merged commit d0d9831 into vitessio:release-15.0 Oct 15, 2022
@deepthi deepthi deleted the fixbackuplogging branch October 15, 2022 17:37
@vitess-bot
Copy link
Contributor

vitess-bot bot commented Oct 15, 2022

I was unable to forwardport this Pull Request to the following branches: main.

rsajwani added a commit to planetscale/vitess that referenced this pull request Oct 26, 2022
* remove excessive logging

Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com>

* code feeback

Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com>

* fixing typo

Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com>

Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com>
@mattlord mattlord changed the title remove excessive logging remove excessive backup decompression logging Oct 28, 2022
mattlord pushed a commit that referenced this pull request Oct 28, 2022
* remove excessive logging

Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com>

* code feeback

Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com>

* fixing typo

Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com>

Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com>

Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Component: Backup and Restore Component: Cluster management Forwardport to: main This will forward port the PR to the main branch Type: Enhancement Logical improvement (somewhere between a bug and feature)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants