Queueing System/Secondary Storage Tier

[brandongalbraith]

While grabbing lunch at the Internet Archive recently, tubeup came up. I was provided a feature request by the Internet Archive staff: the ability for tubeup to operate in a mode where content is retrieved, but stored locally instead of being immediately uploaded to the Internet Archive. On future runs, if a multimedia artifact was detected as no longer being available (for example, a Youtube video was deleted), that would trigger the upload to the Internet Archive.

I am still putting thought into this request, but wanted to share the high level concept for socialization amongst project users.

[vxbinaca]

I think I know who told you that because that person and I discussed it a few months ago when talking about another matter.

I don't really control Tubeup. I'm just here for a shift sort of directing things. If you explicitly need by blessing (for some odd reason), the person who I talked to at IA already got it. I'd talk to @antonizoon though.

I think you have balls for bringing up the tool there it's caused them heartburn totally unintentional by me. If I ever took a tour of IA I'd do it stealth and not mention my full name or especially the tool. I have a chance of not being recognized because theres almost no pictures of me on the internet.

Edit 2: My stance: Give IA whatever they want. The tool exists because they tolerate it. Talk to Ant.

Edit 3: Rips are about to become extinct anyway due to odious Google web DRM.. We've already seen it in other extractors like TikTok or Instagram who aggressively prevent ripping.

[antonizoon]

This actually seems to strike the right balance between user anxiety due to the general state of the internet let alone youtube, and only uploading what is absolutely necessary because there is only so much space at the IA.

In the past the situation may have been different but nowadays users are now mass uploading ahead of any problems and this time compared to all other situations, the user for once might be right.

If your team knows exactly what to do, go ahead and write up the code for everyone to review in a pull request. We also can approve additional Github contributors as necessary.

If your team really needs some development assistance, we may have some motivated volunteers in light of the special situation, but I think for a tool that is a significant pipeline to the Internet Archive let's let you all see what you can make first.