Notice on ship drop data storage and dumping

[KagamiChan]

Updated Nov 2024

• we've moved data storage to Cloudlfare R2, and the new monthly data url will be https://db.poi.moe/ without dump pathname

---

Updated Jan 2021

According to poooi/poi#1843, there'll be some changes on the strategy of ship drop data storage and dumping.

• ship drop data will be wiped monthly after they are dumped
• the monthly data dump will be available at https://db.poi.moe/
• https://api.poi.moe/dump provides some known data sets

[KagamiChan]

pinging @Acrone because you've posted a question before

pinging @Gizeta

[Gizeta]

Got it.

[Acrone]

Exciting stuff. Thanks for the ping!

[yukixz]

Ping @Gizeta @Acrone, the server url is updated to http://poidb.0u0.moe/. The old http://citrus.dazzyd.org may be unusable someday.

[gakada]

@yukixz There are few issues with the site / dumps that just pop up recently.

1. SSL certificate is expired on poi.io and poi.0u0.moe

$ curl -vIk https://poi.io 2>&1 | grep 'expire date'
*  expire date: Sep 15 02:53:34 2019 GMT

$ curl -vIk https://poi.0u0.moe 2>&1 | grep 'expire date'
*  expire date: Sep 15 02:53:34 2019 GMT

# just in case, not expired yet
$ curl -vIk https://poidb.0u0.moe 2>&1 | grep 'expire date'
*  expire date: Oct  4 02:06:07 2019 GMT

2. https://poidb.0u0.moe/dump/ has no dropshiprecords dump for August.

3. Current https://poi.0u0.moe/dump/dropshiprecords.gz is just records from Aug 31 ~ Sep 03, missing the rest of the month's records.

[yukixz]

@gakada so terrible...

1. fixed.
2. seem data of august is lost for unknown reason.
3. ->2

[gakada]

Seems like there is a significant decline in data due to decline in reports from poi.

[gakada]

@KagamiChan can it be due to poooi/poi@dd8ab77? Right now plugin-report is sending data via http://api.poi.moe -> https://api.poi.moe redirect, while previously it was (and still can be) http://poi.0u0.moe, maybe that is affecting some users in a weird way.

[KagamiChan]

Thanks for catching this issue

It turns out that request won't follow the redirection for non GET requests, and won't throw either, thus we were not aware of this.

I'll update the reporter plugin to use https instead

[KagamiChan]

@gakada The new plugin is live, could you please help monitor 1 or 2 days later?

[gakada]

Yeah, I will check if % goes back up in next few dumps.

[KagamiChan]

@gakada Thanks, I've confirmed that we have data from newer version of poi

[gakada]

It is back to normal mostly, 2-3x more reports from poi and 2x for the percentage.

[gakada]

@yukixz having problems with the current dump for the last few days:

tmp $ curl -sI https://poi.moe/dump/dropshiprecords.gz | grep content-length
content-length: 128815104
tmp $ wc -c dropshiprecords.gz
128815104 dropshiprecords.gz
tmp $ sha1sum dropshiprecords.gz
a6c6b569f9e287340e2293a2db3868653c7f6765  dropshiprecords.gz
tmp $ gunzip dropshiprecords.gz

gzip: dropshiprecords.gz: unexpected end of file

[yukixz]

@gakada Dumping again now.

[gakada]

Thanks, now it is fine.

[Tibowl]

@yukixz It looks like current dumps are just giving home page HTML.

For example, https://poi.moe/dump/passeventrecords.gz just shows the poi home page. Happens both in browser and with curl.

[KagamiChan]

Hi @Tibowl

please try https://api.poi.moe/dump/passeventrecords.gz

we've made a separation for website and api server, I thought the change has been announced but seems not yet. sorry for that

Btw, it is a known issue that due to network incompatibility, we have less reports since last December

[gakada]

I added some question marks in https://github.com/poooi/poi-server/wiki

as those seem to be enabled, but I don't know what are the links to their current dumps (two of them have monthly archived dumps).

[KagamiChan]

Thanks @gakada

We disabled file listing for security reasons, but I can still log in to the machine and check the files. I'll make it up later and next time you may just let me update that.

[KagamiChan]

As a follow-up of data loss issue in Dec 2019, it turns out that a POST request could become a GET request with 301 redirection, this may prevent some clients which are still using http scheme

Returning 307 code could solve the issue.
https://rtfm.co.ua/en/http-redirects-post-and-get-requests-and-lost-data/

Also an update for data loss starting from Dec 2020, we've added event and tracing reports in both client side and server side, but for now we don't have any new findings yet

[gakada]

@KagamiChan it seems quests are not being reported? https://api.poi.moe/dump/quests.gz only has January quests and https://api.poi.moe/api/report/v3/known_quests is empty.

Also reminding that https://github.com/poooi/poi-server/wiki is missing quest reward dumps, etc.

[KagamiChan]

@gakada ack, will take a look later

[KagamiChan]

@gakada There were multiple issues, I've fixed them and restored back some data sets

quests should be ready with next daily dump, and I'll add rewards later