Running a second instance of Gitea on the same storage path hangs

[segevfiner]

• Gitea version (or commit ref): 1.6.4
• Git version: 2.15.3
• Operating system: Docker Image of Alpine 3.7
• Database (use [x]):
  • PostgreSQL
  • MySQL
  • MSSQL
  • SQLite
• Can you reproduce the bug at https://try.gitea.io:
  • Yes (provide example URL)
  • No
  • Not relevant
• Log gist: N/A

Description

Running a second instance of Gitea using the same storage path hangs on startup on locking issues.bleve. This probably didn't happen to us on 1.5 due to the issue fixed by #5563.

Stack trace

 0  0x000000000043c4a9 in runtime.gopark
    at .usr/local/go/src/runtime/proc.go:292
 1  0x000000000043c55e in runtime.goparkunlock
    at .usr/local/go/src/runtime/proc.go:297
 2  0x000000000045a6db in time.Sleep
    at .usr/local/go/src/runtime/time.go:102
 3  0x0000000000dbb9b1 in code.gitea.io/gitea/vendor/github.com/boltdb/bolt.flock
    at .go/src/code.gitea.io/gitea/vendor/github.com/boltdb/bolt/bolt_unix.go:38
 4  0x0000000000dc4891 in code.gitea.io/gitea/vendor/github.com/boltdb/bolt.Open
    at .go/src/code.gitea.io/gitea/vendor/github.com/boltdb/bolt/db.go:186
 5  0x0000000000df8e2e in code.gitea.io/gitea/vendor/github.com/blevesearch/bleve/index/store/boltdb.New
    at .go/src/code.gitea.io/gitea/vendor/github.com/blevesearch/bleve/index/store/boltdb/store.go:77
 6  0x0000000000cd8911 in code.gitea.io/gitea/vendor/github.com/blevesearch/bleve/index/upsidedown.(*UpsideDownCouch).Open
    at .go/src/code.gitea.io/gitea/vendor/github.com/blevesearch/bleve/index/upsidedown/upsidedown.go:308
 7  0x0000000000e691a5 in code.gitea.io/gitea/vendor/github.com/blevesearch/bleve.openIndexUsing
    at .go/src/code.gitea.io/gitea/vendor/github.com/blevesearch/bleve/index_impl.go:168
 8  0x0000000000e67bd9 in code.gitea.io/gitea/vendor/github.com/blevesearch/bleve.Open
    at .go/src/code.gitea.io/gitea/vendor/github.com/blevesearch/bleve/index.go:248
 9  0x0000000000e90503 in code.gitea.io/gitea/modules/indexer.openIndexer
    at .go/src/code.gitea.io/gitea/modules/indexer/indexer.go:84
10  0x0000000000e90815 in code.gitea.io/gitea/modules/indexer.InitIssueIndexer
    at .go/src/code.gitea.io/gitea/modules/indexer/issue.go:55
11  0x00000000011b0211 in code.gitea.io/gitea/models.InitIssueIndexer
    at .go/src/code.gitea.io/gitea/models/issue_indexer.go:21
12  0x00000000014d03ff in code.gitea.io/gitea/routers.GlobalInit
    at .go/src/code.gitea.io/gitea/routers/init.go:72
13  0x0000000001805186 in code.gitea.io/gitea/cmd.runWeb
    at .go/src/code.gitea.io/gitea/cmd/web.go:121
14  0x00000000012f986e in code.gitea.io/gitea/vendor/github.com/urfave/cli.HandleAction
    at .go/src/code.gitea.io/gitea/vendor/github.com/urfave/cli/app.go:471
15  0x00000000012fb0e8 in code.gitea.io/gitea/vendor/github.com/urfave/cli.Command.Run
    at .go/src/code.gitea.io/gitea/vendor/github.com/urfave/cli/command.go:191
16  0x00000000012f7427 in code.gitea.io/gitea/vendor/github.com/urfave/cli.(*App).Run
    at .go/src/code.gitea.io/gitea/vendor/github.com/urfave/cli/app.go:241
17  0x000000000181804f in main.main
    at .go/src/code.gitea.io/gitea/main.go:53
18  0x000000000043c070 in runtime.main
    at .usr/local/go/src/runtime/proc.go:198
19  0x000000000046c081 in runtime.goexit
    at .usr/local/go/src/runtime/asm_amd64.s:2361

cc @zvikagart

[techknowlogick]

Gitea expects to be able claim the lock for indexing. Are you running a second install of Gitea using same path for High Availability reasons?

[segevfiner]

Yeah. For a rolling upgrade without downtime.

[zvikagart]

@techknowlogick any suggestions or outline of how to solve this? would setting a per-instance index file work?

[lunny]

I will test it.

[jeblair]

We're working on deploying a multi-master Gitea using CephFS as shared storage for both git repos and blevesearch indexes. We seemed to have the system working on 1.6.0, but suspect that we're hitting this issue after upgrading to 1.7.1. The system is not in production yet, and we don't have much in the way of historical data to contribute to this bug report. But if you do come up with a solution to this, we would be happy to help out with some fairly high-volume testing.

[lunny]

@jeblair any logs could you give?

[jeblair]

I ran strace on the second gitea process I started and came to the same conclusion as @segevfiner, when I saw an open file handle to issues.bleve and flock syscalls. Other than that, I didn't see much of use. Is there anything I should look for?

[emonty]

Looking at the indexer code, it seems to be a nice clean interface (yay for interfaces!) What would people think about adding something like an optional elasticsearch driver that people trying to do larger installations could use, but keep the bleve implementation in as the default so that people don't have to install elasticsearch just to get a gitea?

[techknowlogick]

@emonty that's exactly the plan 😉

[emonty]

@techknowlogick sweet! that sounds like an excellent plan. 👍

[lunny]

@emonty #5363 just merged yesterday. The issues Indexer interface has been there and it's simple enough that somebody could send a PR to implement elasticsearch. And I will send a PR to refactor code search indexer similar with issue indexer.

[lunny]

I have sent #6144, then if you change ISSUE_INDEXER_TYPE to db will resolve the problem when you have a small deployment. And I will send another PR to add elasticsearch and redis.