Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Remove files from cloud storage when version is wiped. #6186

Closed
wants to merge 9 commits into from
Closed

Remove files from cloud storage when version is wiped. #6186

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
e9b1406
remove files from cloud storage
dojutsu-user Sep 16, 2019
9fabd5e
add logger
dojutsu-user Sep 16, 2019
393c9bf
refactoring
dojutsu-user Sep 17, 2019
ee05e15
update test
dojutsu-user Sep 17, 2019
d23a05e
add faq
dojutsu-user Sep 17, 2019
4619ea5
add improvements
dojutsu-user Sep 17, 2019
6e0f81f
revert test name
dojutsu-user Sep 17, 2019
fc8b26a
typo fixes
dojutsu-user Sep 18, 2019
cbbb92e
fix func name
dojutsu-user Sep 18, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions docs/faq.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -245,6 +245,13 @@ What commit of Read the Docs is in production?
We deploy readthedocs.org from the `rel` branch in our GitHub repository. You can see the latest commits that have been deployed by looking on GitHub: https://github.com/readthedocs/readthedocs.org/commits/rel


How can I remove stale results from the search?
-----------------------------------------------

If the search results contains stale/outdated results,
you can update the search index by :doc:`wiping the build environment <guides/wipe-environment>` and then rebuilding your docs.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure about this. I mean, if that is happening I would like that users report that to us, since it is a bug from our side. We could remove this or add something like if this is happening very often, report it.

@davidfischer what do you think?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@stsewd
I think this is a bug from Sphinx, if a .rst file is deleted, then the .html file for that shouldn't be created or deleted in the next build.

I think it would be better if a user tries wiping the environment and rebuilding the docs, if that doesn't work -- then report to us.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, as we move toward each build being totally fresh and not relying on a previous environment, this problem should naturally go away. It is possible that we need to detect when files aren't generated as part of the build and remove them. Even with this change, that won't be done exactly.



How can I avoid search results having a deprecated version of my docs?
----------------------------------------------------------------------

Expand Down
24 changes: 23 additions & 1 deletion readthedocs/core/utils/general.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,15 +2,24 @@

import os

from django.conf import settings
from django.core.files.storage import get_storage_class
from django.shortcuts import get_object_or_404

from readthedocs.core.utils import broadcast
from readthedocs.projects.tasks import remove_dirs
from readthedocs.builds.models import Version
from readthedocs.projects.tasks import remove_build_storage_paths


def wipe_version_via_slugs(version_slug, project_slug):
stsewd marked this conversation as resolved.
Show resolved Hide resolved
"""Wipes the given version of a given project."""
"""
Wipes the given version of a project.

It does two things:
* Clears the `checkouts`, `envs`, and `conda` directories (if exist).
* Removes the html files from cloud storage.
"""
version = get_object_or_404(
Version,
slug=version_slug,
Expand All @@ -23,3 +32,16 @@ def wipe_version_via_slugs(version_slug, project_slug):
]
for del_dir in del_dirs:
broadcast(type='build', task=remove_dirs, args=[(del_dir,)])

_clear_html_files_from_media_storage(version)


def _clear_html_files_from_media_storage(version):
"""Removes html files from media storage (cloud or local) for a given version of a project."""

storage_path = version.project.get_storage_path(
type_='html',
version_slug=version.slug,
include_file=False,
)
remove_build_storage_paths.delay([storage_path])
7 changes: 6 additions & 1 deletion readthedocs/rtd_tests/tests/test_core_utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -200,7 +200,11 @@ def test_slugify(self):
)

@mock.patch('readthedocs.core.utils.general.broadcast')
def test_wipe_version_via_slug(self, mock_broadcast):
@mock.patch('readthedocs.core.utils.general.remove_build_storage_paths')
@mock.patch.object(Project, 'get_storage_path')
def test_wipe_version_via_slugs(self, mock_get_storage_path, mock_remove_build_storage_paths, mock_broadcast):
mock_get_storage_path.return_value = 'test/path/'

wipe_version_via_slugs(
version_slug=self.version.slug,
project_slug=self.version.project.slug
Expand All @@ -219,6 +223,7 @@ def test_wipe_version_via_slug(self, mock_broadcast):
],
any_order=False
)
mock_remove_build_storage_paths.delay.assert_called_once_with(['test/path/'])

@mock.patch('readthedocs.core.utils.general.broadcast')
def test_wipe_version_via_slug_wrong_param(self, mock_broadcast):
Expand Down