Update tr.wikipedia-on-ipfs.org #60

lidel · 2019-09-09T14:32:56Z

create new (test) snapshot
- couldn't extract fully, some files fail with other os error (couldn't extract ... other os error dignifiedquire/zim#3)
- fixed in extract_zim v0.2.0
ensure canonical link is correct (Block internet search engines from indexing the mirror #48 (comment) + Add rel="canonical" for search engines #65)
- present in wikipedia_tr_all_maxi_2019-10.zim
ensure footer is updated ([BOUNTY] Fix script responsible for preparing IPFS mirror #64
identify landing page parameter for execute-changes.sh
fix any broken JS by execute-changes.sh
recreate snapshot if needed
pin
- set up collaborative pinning cluster (Set up collaborative pinning clusters #68)
update DNSLink at tr.wikipedia-on-ipfs.org

This could be done manually or as a part of #58

The text was updated successfully, but these errors were encountered:

lidel · 2019-10-28T11:30:05Z

Couldn't extract fully, some files fail with other os error described in dignifiedquire/zim#3:

$  extract_zim --skip-link wikipedia_tr_all_maxi_2019-10.zim --out distributed-wikipedia-mirror/out2
Mon 28 Oct 12:26:11 CET 2019                                                                                                                       
Extracting file: wikipedia_tr_all_maxi_2019-10.zim to distributed-wikipedia-mirror/out2

  Creating map
  Extracting entries: 4808
  Spawning 4808 threads
couldn't create distributed-wikipedia-mirror/out2/A/Eternity: other os error
couldn't create distributed-wikipedia-mirror/out2/A/Eternity: other os error
couldn't create distributed-wikipedia-mirror/out2/A/Eternity: other os error
thread '<unnamed>' panicked at 'failed retry: couldn't create distributed-wikipedia-mirror/out2/A/Eternity: other os error', extract_zim.rs:133:17
note: Run with `RUST_BACKTRACE=1` environment variable to display a backtrace.
couldn't create distributed-wikipedia-mirror/out2/A/Karacaoğlan: other os error
couldn't create distributed-wikipedia-mirror/out2/A/Karacaoğlan: other os error
couldn't create distributed-wikipedia-mirror/out2/A/Karacaoğlan: other os error
thread '<unnamed>' panicked at 'failed retry: couldn't create distributed-wikipedia-mirror/out2/A/Karacaoğlan: other os error', extract_zim.rs:133:17
couldn't create distributed-wikipedia-mirror/out2/A/Dört/Mazi_Kalbimde: other os error
couldn't create distributed-wikipedia-mirror/out2/A/Dört/Mazi_Kalbimde: other os error
couldn't create distributed-wikipedia-mirror/out2/A/Dört/Mazi_Kalbimde: other os error
thread '<unnamed>' panicked at 'failed retry: couldn't create distributed-wikipedia-mirror/out2/A/Dört/Mazi_Kalbimde: other os error', extract_zim.rs:133:17
couldn't create distributed-wikipedia-mirror/out2/A/HIV: other os error
couldn't create distributed-wikipedia-mirror/out2/A/HIV: other os error
couldn't create distributed-wikipedia-mirror/out2/A/HIV: other os error
thread '<unnamed>' panicked at 'failed retry: couldn't create distributed-wikipedia-mirror/out2/A/HIV: other os error', extract_zim.rs:133:17
couldn't create distributed-wikipedia-mirror/out2/A/Fredrikstad/Sarpsborg: other os error
couldn't create distributed-wikipedia-mirror/out2/A/Fredrikstad/Sarpsborg: other os error
couldn't create distributed-wikipedia-mirror/out2/A/Fredrikstad/Sarpsborg: other os error
thread '<unnamed>' panicked at 'failed retry: couldn't create distributed-wikipedia-mirror/out2/A/Fredrikstad/Sarpsborg: other os error', extract_zim.rs:133:17
...

Fixing this would be the first step to unblock this.

kelson42 · 2019-11-07T17:54:35Z

I want to make a side remark here. It is a pity that openZIM seems to not to provide an official tool you can just use to extract the ZIM content. I'm not sure about your exact requirement, but If I get them, I will seriously consider to do something to fix that problem for you. I want to really encourage you to create a feature request here https://github.com/openzim/zim-tools

lidel · 2019-11-07T22:17:39Z

@kelson42 thank you for bringing zim-tools to my attention!
I was not around when tweaked extract_zim was created, but been told it was created either because zimdump was simply not around yet and original one was missing some features.
I'll run some tests with zimdump from zim-tools and report back.

Update: I think we could switch, but some things need to be fixed first. See #66 :)

dignifiedquire · 2020-01-19T23:38:39Z

this error should be fixed now in the latest version of dignifiedquire/zim

dignifiedquire · 2020-01-20T00:58:02Z

Just ran the extraction, all fixed now

$time ./target/release/extract_zim --skip-link ~/Downloads/wikipedia_tr_all_maxi_2019-10.zim --out ./out
Extracting file: /Users/dignifiedquire/Downloads/wikipedia_tr_all_maxi_2019-10.zim to ./out

  Creating map
  Extracting entries: 4808
  Spawning 4808 tasks across 16 threads
  Extraction done in 47453ms
  Main page is Kullanıcı:The_other_Kiwix_guy/Landing
./target/release/extract_zim --skip-link  --out ./out  81.81s user 218.73s system 631% cpu 47.560 total

lidel · 2020-01-20T12:45:52Z

Thank you @dignifiedquire, this is great!

I took it for a spin and initial results are pretty good (wip in #67):

Unpacking wikipedia_tr_all_maxi_2019-10.zim took less than two minutes.
Adding ~10GB to IPFS (0.5.0-dev with sharding+badgerds) in --offline mode took under 10 minutes.

Next step is to figure out #64 and landing page for execute-changes.sh

@kelson42 do you know why .zim file states that the Main page is Kullanıcı:The_other_Kiwix_guy/Landing?
Is providing custom page a new convention in kiwix project?

lidel · 2020-01-22T15:40:10Z

Original landing page at ./out/A/Anasayfa.html seems to be truncated, the page includes only "article of the week" section, making it pretty bad landing page overall.
Tried to finish snapshot creation but scripts no longer work.
JS and the directory structure changed so much, that entire execute-changes.sh needs to be redone.
I also noticed ./out/-/j/js_modules/jsConfigVars.js is invalid, its contents being: ( (the single character) file has the same contents when unpacked with zimdump, so its not a bug in extract_zim
Retested with wikipedia_tr_all_maxi_2019-12.zim, same results

Update: I created a bounty for remaining work: #64

lidel · 2020-04-09T23:37:30Z

#70 fixed the most painful blockers, and I was able to produce a new snapshot from wikipedia_tr_all_maxi_2020-04.zim:

https://bafybeih3pdooy7tghdiuibphro6cwlyxp4henmmap7f2qm6fyitgpqafle.ipfs.dweb.link/

It has a pretty nice footer with useful information about mirror and its sources.

Two cosmetic issues remain:

Main page at /wiki/Anasayfa.html is fetched for the date of unpacking ZIM, instead of the day ZIM was created (This seems to be specific to Turkish wiki)
Some links on the Main page are broken, for example ones in header (/wiki/Portal:Matematik.html etc)
- Not a problem, as the old version did not have links there anyway

Both can be fixed manually by patching HTML in /wiki/Anasayfa.html, but if anyone has time to fix them programmatically, that would be useful for other languages.

momack2 · 2020-04-09T23:41:28Z

The view on mobile isn’t great - is that expected or a regression?

lidel · 2020-04-10T00:24:59Z

@momack2 Yes, if we want "original" landing page this is something we need to fix manually (I believe the one in old snapshot was also crafted by hand).

Context:

Original Main page is truncated or not included in many ZIM, so we have no "mobile friendly" version.
We download original HTML from wikipedia itself and add it to the unpacked ZIM snapshot, which as seen above requires some work.

FYI ZIM files often use a custom page provided by a contributor that makes more sense for offline use (example). You can see it does not have "topic of the day", instead its a dry list of wide topics ready to explore.

If we keep it, very little or no manual fixes may be needed because it is already simple enough to be mobile friendly – see this build where I left the original landing page from ZIM archive: https://bafybeieoya74422ovlmx23i5bxpuw2szsdrhsjwenfxkqoknw34jigcoua.ipfs.dweb.link/wiki/Anasayfa.html (added only the landing page, links may not resolve)

momack2 · 2020-04-10T01:30:35Z

Oh yeah - much better. Not perfect, but "good enough" for smooth browsing.

lidel · 2021-02-15T14:18:17Z

Ok friends, I've picked up the ball in #77 and produced a brand new snapshot from wikipedia_tr_all_maxi_2021-02.zim. If this goes well we will do the same for English (#61)

Highlights:

fixed favicon (the most important! 😅)
we now use upstream https://github.com/openzim/zim-tools for unpacking
native-looking footer
source ZIM is added to the root directory and linked from the footer – this is extremely important for both auditability and kicks-off ZIM-on-IPFS efforts from Question: Use directly ZIM files? #42 and ZIM on IPFS: maximizing deduplication #71
article names no longer have .html, so paths look exactly the same as on original Wikipedia 🤯 ✨
we no longer overwrite original landing page from the day snapshot was taken:
- /wiki/ uses "main page" from ZIM (date-agnostic, usually better suited for offline/exploratory use)
- the original one is still available at /wiki/Anasayfa (matching https://tr.wikipedia.org/wiki/Anasayfa)

👉 take it for a spin and comment if you find any issues:

https://dweb.link/ipfs/bafybeiexiynkovpcka22bmsbc4z4kvwtoggvyx7vzf5xfy5zo3vlpvyn2m/
(be mindful this is not pinned anywhere, only full copy is on my box for now, so ymmv)

kelson42 · 2021-02-16T03:56:52Z

"Kiwix" in place of "kiwix" in the footer would be better. With a link to https://kiwix.org, even better :)

lidel · 2021-02-16T23:27:00Z

@kelson42 added this in #82 and created updated version:

https://dweb.link/ipfs/bafybeieuutdavvf55sh3jktq2dpi2hkle6dtmebe7uklod3ramihyf3xa4

@mburns mind switching https://tr.dev.wikipedia-on-ipfs.org to the above CID?

mburns · 2021-02-17T00:10:23Z

done. :)

lidel · 2021-02-18T23:43:27Z

Ok, this should be good enough for now.

DNSLink is updated and https://tr.wikipedia-on-ipfs.org now points at:
bafybeieuutdavvf55sh3jktq2dpi2hkle6dtmebe7uklod3ramihyf3xa4 (generated from wikipedia_tr_all_maxi_2021-02)

I'm now shifting focus to English one, tracking that in #61

Updating Turkish CID to #60 (comment)

lidel added the language language-specific issues label Sep 9, 2019

lidel mentioned this issue Sep 10, 2019

[BOUNTY] Fix script responsible for preparing IPFS mirror #64

Closed

13 tasks

lidel added the P1 High: Likely tackled by core team if no one steps up label Oct 28, 2019

lidel mentioned this issue Nov 8, 2019

Switch to zimdump from zim-tools #66

Closed

6 tasks

lidel mentioned this issue Jan 20, 2020

Update en.wikipedia-on-ipfs.org #61

Closed

9 tasks

lidel mentioned this issue Jan 22, 2020

Make execute-changes script more generally usable #21

Closed

4 tasks

lidel added the Epic label Mar 9, 2020

This was referenced Feb 8, 2021

refactor: build against ZIMs from 2021 #77

Merged

zimdump produces invalid path in HTML-based redirect openzim/zim-tools#224

Closed

This was referenced Feb 15, 2021

Add rel="canonical" for search engines #65

Closed

Block internet search engines from indexing the mirror #48

Closed

About the Contents Retrieve and Errors Corrections #57

Closed

lidel pinned this issue Feb 15, 2021

lidel mentioned this issue Feb 16, 2021

style: link to kiwix.org from footer #82

Merged

lidel closed this as completed Feb 18, 2021

lidel unpinned this issue Feb 18, 2021

lidel mentioned this issue Feb 19, 2021

Set up collaborative pinning clusters #68

Closed

3 tasks

lidel added a commit that referenced this issue Feb 19, 2021

Update snapshot-hashes.yml

fda3239

Updating Turkish CID to #60 (comment)

ipfs deleted a comment from VIP0000fa Aug 21, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update tr.wikipedia-on-ipfs.org #60

Update tr.wikipedia-on-ipfs.org #60

lidel commented Sep 9, 2019 •

edited

Loading

lidel commented Oct 28, 2019 •

edited

Loading

kelson42 commented Nov 7, 2019

lidel commented Nov 7, 2019 •

edited

Loading

dignifiedquire commented Jan 19, 2020

dignifiedquire commented Jan 20, 2020

lidel commented Jan 20, 2020 •

edited

Loading

lidel commented Jan 22, 2020 •

edited

Loading

lidel commented Apr 9, 2020

momack2 commented Apr 9, 2020

lidel commented Apr 10, 2020 •

edited

Loading

momack2 commented Apr 10, 2020

lidel commented Feb 15, 2021 •

edited

Loading

kelson42 commented Feb 16, 2021 •

edited

Loading

lidel commented Feb 16, 2021

mburns commented Feb 17, 2021

lidel commented Feb 18, 2021 •

edited

Loading

Update tr.wikipedia-on-ipfs.org #60

Update tr.wikipedia-on-ipfs.org #60

Comments

lidel commented Sep 9, 2019 • edited Loading

lidel commented Oct 28, 2019 • edited Loading

kelson42 commented Nov 7, 2019

lidel commented Nov 7, 2019 • edited Loading

dignifiedquire commented Jan 19, 2020

dignifiedquire commented Jan 20, 2020

lidel commented Jan 20, 2020 • edited Loading

lidel commented Jan 22, 2020 • edited Loading

lidel commented Apr 9, 2020

momack2 commented Apr 9, 2020

lidel commented Apr 10, 2020 • edited Loading

momack2 commented Apr 10, 2020

lidel commented Feb 15, 2021 • edited Loading

kelson42 commented Feb 16, 2021 • edited Loading

lidel commented Feb 16, 2021

mburns commented Feb 17, 2021

lidel commented Feb 18, 2021 • edited Loading

lidel commented Sep 9, 2019 •

edited

Loading

lidel commented Oct 28, 2019 •

edited

Loading

lidel commented Nov 7, 2019 •

edited

Loading

lidel commented Jan 20, 2020 •

edited

Loading

lidel commented Jan 22, 2020 •

edited

Loading

lidel commented Apr 10, 2020 •

edited

Loading

lidel commented Feb 15, 2021 •

edited

Loading

kelson42 commented Feb 16, 2021 •

edited

Loading

lidel commented Feb 18, 2021 •

edited

Loading