Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add command to download and cache lists. #18

Merged
merged 1 commit into from
Oct 29, 2021

Conversation

jmhobbs
Copy link
Contributor

@jmhobbs jmhobbs commented Oct 28, 2021

I took a pass at #4, would love to iterate on this if you have feedback.

  • Add an option to the cli
  • When the option is given, all the repository information is downloaded to the cache recursively
  • Inform user that the operation will take a long time
  • Give the number of repositories cached when the operation is finished
  • Display progress information (the repository being fetched etc)
  • Use go routines to make the repository fetch in parallel (4 go routines)

Here's a demo run starting with a mostly complete cache:

cache

}

root := parser.ParseIndex(fetched)
urls := getAllChildrenURLs(root.GetChildren())
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From what I could tell, only the root level awesome list has links to other lists. Is that correct? If not, how can I differentiate list links from regular link nodes?

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, this is correct. The cli totally relies on this. Because, the owner of awesome list root repo pushes it with contribution guideline.

internal/package/fetcher/all.go Outdated Show resolved Hide resolved
internal/package/fetcher/all.go Outdated Show resolved Hide resolved
@umutphp umutphp linked an issue Oct 29, 2021 that may be closed by this pull request
6 tasks
@jmhobbs
Copy link
Contributor Author

jmhobbs commented Oct 29, 2021

Since you merged #19 I rebased and typos should be good now 👍

@jmhobbs jmhobbs requested a review from umutphp October 29, 2021 04:16
@umutphp
Copy link
Owner

umutphp commented Oct 29, 2021

@jmhobbs Many thanks for this great PR :)

@umutphp umutphp merged commit 4fbd891 into umutphp:master Oct 29, 2021
@jmhobbs jmhobbs deleted the cache-all branch October 29, 2021 18:51
@jmhobbs
Copy link
Contributor Author

jmhobbs commented Oct 29, 2021

My pleasure, it's a neat project!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Option To Download Everything to the Cache
2 participants