using s3 buckets?

[chichicuervo]

there are a few mentions of S3 bucket support in 2015. Does it actually work, and if so, how do I configure so it will actually work?

[ldko]

HI @chichicuervo, the support for S3 buckets was contributed from someone outside of IIPC via #188 and #189, but as far as I know, there was no documentation provided for it. The code to support it is still in ResourceFactory. It indicates credentials would go into $HADOOP_CONF/core-site.xml. Other than that, I would try to configure it per the CDX instructions in the wiki and using a FlatFileResourceFileLocationDB (path-index.txt) as described there, that had the addresses of the WARCs in the S3 buckets.

[chichicuervo]

yeah... the lack of documentation is kind of the problem. Particularly since the hadoop stuff neither makes any sense to me (not a java dev) or does the link mentioned in previous issues point to anything but a 404

[ldko]

It looks like the Internet Archive has an archived copy of that page from around the time that shows key configuration.

[chichicuervo]

for reference's sake... this didn't work.. attempting to go the CDX route