Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix multi cache dir #2639

Merged
merged 8 commits into from
Feb 27, 2023

Conversation

zwwhdls
Copy link
Member

@zwwhdls zwwhdls commented Feb 23, 2023

Ⅰ. Describe what this PR does

fix multi cache dir in juicefs runtime

Ⅱ. Does this pull request fix one issue?

fixes #2641

Ⅲ. List the added test cases (unit test/integration test) if any, please explain if no tests are needed.

Ⅳ. Describe how to verify it

Ⅴ. Special notes for reviews

Signed-off-by: zwwhdls <[email protected]>
@zwwhdls
Copy link
Member Author

zwwhdls commented Feb 23, 2023

/test fluid-e2e

// community doc: https://juicefs.com/docs/community/command_reference/#juicefs-mount
// enterprise doc: https://juicefs.com/docs/cloud/commands_reference#mount
// /mnt/disk1/bigboot or /mnt/disk1/bigboot:/mnt/disk2/bigboot
storagePath = tiredStoreLevel.Path
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please fix unit test. thanks.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

Signed-off-by: zwwhdls <[email protected]>
@zwwhdls
Copy link
Member Author

zwwhdls commented Feb 24, 2023

/retest

@codecov
Copy link

codecov bot commented Feb 24, 2023

Codecov Report

Merging #2639 (7f45972) into master (ee79af5) will increase coverage by 0.05%.
The diff coverage is 88.70%.

❗ Current head 7f45972 differs from pull request most recent head 428d28e. Consider uploading reports for the commit 428d28e to get more accurate results

@@            Coverage Diff             @@
##           master    #2639      +/-   ##
==========================================
+ Coverage   67.58%   67.63%   +0.05%     
==========================================
  Files         369      369              
  Lines       21540    21579      +39     
==========================================
+ Hits        14558    14596      +38     
  Misses       5274     5274              
- Partials     1708     1709       +1     
Impacted Files Coverage Δ
pkg/ddc/eac/shutdown.go 35.95% <0.00%> (ø)
pkg/ddc/juicefs/operations/base.go 80.35% <75.00%> (-0.74%) ⬇️
pkg/ddc/eac/data_load.go 78.84% <100.00%> (ø)
pkg/ddc/eac/node.go 57.14% <100.00%> (ø)
pkg/ddc/eac/utils.go 75.93% <100.00%> (+1.13%) ⬆️
pkg/ddc/eac/worker.go 67.81% <100.00%> (ø)
pkg/ddc/juicefs/shutdown.go 61.11% <100.00%> (+4.96%) ⬆️
pkg/ddc/juicefs/transform_fuse.go 78.45% <100.00%> (ø)
pkg/ddc/juicefs/transform_volume.go 94.11% <100.00%> (-0.06%) ⬇️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

@zwwhdls
Copy link
Member Author

zwwhdls commented Feb 24, 2023

/test fluid-e2e

@TrafalgarZZZ
Copy link
Member

/hold
hold because multi-dir path won't be cleaned up when shutting down juicefs engine

@zwwhdls
Copy link
Member Author

zwwhdls commented Feb 24, 2023

/test fluid-e2e

Signed-off-by: zwwhdls <[email protected]>
Signed-off-by: zwwhdls <[email protected]>
@fluid-e2e-bot
Copy link

fluid-e2e-bot bot commented Feb 25, 2023

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: cheyang

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

return
}
return
}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To prevent data loss from incorrect directory parameters, I suggest using regular expressions to validate before using "rm -rf" command. Is it possible for this case?

@cheyang cheyang self-requested a review February 25, 2023 12:00
@TrafalgarZZZ
Copy link
Member

/test fluid-e2e

@TrafalgarZZZ
Copy link
Member

/retest

2 similar comments
@TrafalgarZZZ
Copy link
Member

/retest

@TrafalgarZZZ
Copy link
Member

/retest

@zwwhdls
Copy link
Member Author

zwwhdls commented Feb 27, 2023

/test fluid-e2e

// cache dir check
match := ValidCacheDir(dir)
if !match {
err = fmt.Errorf("dir %s is not valid", dir)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about using the following info?

err = fmt.Errorf("invalid cache directory %s, skip cleaning up", dir)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@zwwhdls The code now returns an error but I think we should ignore the error if it's not a valid cache dir.

@TrafalgarZZZ
Copy link
Member

/unhold
/lgtm

@fluid-e2e-bot fluid-e2e-bot bot added the lgtm label Feb 27, 2023
@fluid-e2e-bot fluid-e2e-bot bot merged commit 9834eca into fluid-cloudnative:master Feb 27, 2023
cheyang pushed a commit to cheyang/fluid that referenced this pull request Mar 23, 2023
* fix multi cache dir

Signed-off-by: zwwhdls <[email protected]>

* fix unit test

---------

Signed-off-by: zwwhdls <[email protected]>
cheyang pushed a commit to cheyang/fluid that referenced this pull request Mar 23, 2023
* fix multi cache dir

Signed-off-by: zwwhdls <[email protected]>

* fix unit test

---------

Signed-off-by: zwwhdls <[email protected]>
Signed-off-by: cheyang <[email protected]>
cheyang added a commit that referenced this pull request Mar 27, 2023
* [juicefs] fix worker cache when set option (#2563)

* fix worker cache when set option

Signed-off-by: zwwhdls <[email protected]>

* update changelog in chart

Signed-off-by: zwwhdls <[email protected]>

* fix unittest

Signed-off-by: zwwhdls <[email protected]>

---------

Signed-off-by: zwwhdls <[email protected]>
Signed-off-by: cheyang <[email protected]>

* fix multi cache dir (#2639)

* fix multi cache dir

Signed-off-by: zwwhdls <[email protected]>

* fix unit test

---------

Signed-off-by: zwwhdls <[email protected]>
Signed-off-by: cheyang <[email protected]>

* [Enhancement]CSI plugin checks mount point liveness before binding mount points (#2703)

* Clean up broken mount point when NodeStageVolume

Signed-off-by: dongyun.xzh <[email protected]>

* Check mount point aliveness when NodePublishVolume

Signed-off-by: dongyun.xzh <[email protected]>

* Clean up broken mount point when NodeStageVolume

Signed-off-by: dongyun.xzh <[email protected]>

* Fix cleaning logic

Signed-off-by: dongyun.xzh <[email protected]>

---------

Signed-off-by: dongyun.xzh <[email protected]>
Signed-off-by: cheyang <[email protected]>

* Prettify error messages for exec.Commands in Fluid (#2718)

* Prettify error log message when calling NodePublishVolume

Signed-off-by: dongyun.xzh <[email protected]>

* Prettify error logs when calling helm-related funcs

Signed-off-by: dongyun.xzh <[email protected]>

* Use instead `errors.As`

Signed-off-by: dongyun.xzh <[email protected]>

* Use instead `errors.As`

Signed-off-by: dongyun.xzh <[email protected]>

* Set higher log level for helm exec

Signed-off-by: dongyun.xzh <[email protected]>

---------

Signed-off-by: dongyun.xzh <[email protected]>
Signed-off-by: cheyang <[email protected]>

* update mount to check mountinfo, To #48327952

Signed-off-by: cheyang <[email protected]>

* update mount to check mountinfo, To #48327952

Signed-off-by: cheyang <[email protected]>

* Build docker images for v0.8.5, To #48327952

Signed-off-by: cheyang <[email protected]>

---------

Signed-off-by: zwwhdls <[email protected]>
Signed-off-by: cheyang <[email protected]>
Signed-off-by: dongyun.xzh <[email protected]>
Co-authored-by: Weiwei <[email protected]>
Co-authored-by: TzZtzt <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[BUG]JuiceFSRuntime has multi cache dir issue
3 participants