-
Notifications
You must be signed in to change notification settings - Fork 91
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Syncthing often gets stuck on "Starting" (VPN use case) #957
Comments
same problem on Android 13, Google Pixel 6.
Syncthing-Fork v1.23.0.0 from F-Droid |
Yeah, like I said, force stopping manually is a major impairment of auto syncing, which is the main purpose of Syncthing. You might find yourself expecting on files having been transferred from one device to another, only to find out Syncthing hasn't transferred anything because it's been stuck since the last time you had a chance to check it. At which point, if the device is out of range, has bad/spotty connectivity or the source device has gone offline or has connectivity issues, you're - excuse the French - SOL. Hence, the only reliable solution is not Force Stopping, but rather leaving Syncthing running 24/7, as this minimizes the chance of the issue arising, but also has a significant impact on battery life. In light of this, I consider this issue as a fairly major one. PS: having updated to 1.23 shortly after opening this ticket, I can also confirm updating did not bring any relevant change. |
Same thing here, been since the last few weeks. Pixel 6 Pro Android 13, Syncthing fork V1.23.0. I am using rules to have it only run when I'm at home on home WIFI - it works OK, but its like when i leave and return it getting stuck re-starting after the pause based on rules. As it hasn't started I cant access the logs in app to see anything, not sure how else to contribute to issue. |
You might be able to collect some useful information from |
I'm currently running Syncthing manually and then going "exit" after reaching the 100% sync as I only have small data deltas to catch up every day. If someone has an idea why this happens on newer Androids, I'm all ears. |
In my case I'm seeing this behavior on a OnePlus 6 Android 10, Syncthing fork 1.23.0.0. There is kind of a pattern where it gets stuck on starting if I turn on the VPN when I'm not on my home WiFi. So If I never turn the VPN on it works fine but if I turn it on sometimes it gets stuck and sometimes doesn't. If I remember correctly the first time I saw this behavior was on version 1.21.0.3. |
FWIW I'm also using a VPN. |
Same issue on Oneplus Nord (Android 12) Syncthing-Fork v1.23.0.0 from F-Droid. |
I have been having a similar problem, and it seems to be related to this upstream issue: syncthing/syncthing-android#1799 |
Same issue on Pixel 4a, Android 13, Syncthing Fork Wrapper Version v1.23.0.0. Configured to run on WiFi only, and cannot access the log while stuck "Syncthing is starting". |
One odd behavior I've noticed is that even when the GUI thinks What tells the GUI that Syncthing has fully started ( What calls that method? Considering the multiple paths to call And worse yet, reproducing this bug is tricky. In my real-world usage, I think I would get this bug due to switching Wi-Fi networks (which breaks wireless debugging) or entering/exiting power saver mode (which requires unplugging the power cable, breaking wired debugging). I may try "Enable "Run according to time schedule" so Syncthing starts and stops every hour", switching Wi-Fi networks with USB debugging plugged in, or even starting/stopping by hand. Is there a good way to save all logcat logs of a specific app to a file on storage (possibly a new filename with timestamp every time the Android app starts, to avoid unbounded file size growth like EDIT: I tried switching from one Wi-Fi network to another, and after seeing the "onServiceStateChange: from DISABLED to STARTING" message switching Wi-Fi networks again. It's been stuck on Starting for minutes now. I think I've found a way to reproduce the bug (possibly through human-timescale state management errors alone, without even a high-speed heisenbug race condition). I'll debug what's going on tomorrow. |
I've found one way that Syncthing-Android gets stuck on "Starting".
Quickly the logcat will start being spammed by errors:
These errors proceed even while Syncthing itself is making sync progress.
Oddly enough, I can access https://127.0.0.1:8384 on an Android browser (which asks for a username and password), even while the network thread is throwing ENONET. Is Volley trying to connect to a stale disconnected Wi-Fi network (even though all Wi-Fi networks share a Linux network interface)? IDK, again I don't understand how Volley works. Hopefully @Catfriend1 or an upstream developer can fix the problem from here. Can I safely upload the full logcat, or are device IDs sensitive and I should redact them first? I have not tried reproducing this issue in upstream Syncthing. If the bug reproduces there as well, should it be fixed there or in this fork? |
@nyanpasu64 thanks, and wow. Good find :). I'll take a look, hopefully next week. Did you try to turn off "bind to active network" in settings? |
After unchecking "Behaviour > bind to active network" and force quitting Syncthing, I was not able to reproduce the endless ENONET errors. I was however able to get Syncthing into a state where it would Additionally I also managed to enter an endless cycle of Do you need logs for these bugs? Honestly at this point, with the number of different bugs I've seen, the only way I know to prevent more from popping up after stacking attempted fixes, is to informally verify the correctness of the state management algorithms. You'd have to prove that no matter the starting state of the Java GUI and Go services, the Syncthing daemon will be appropriately stopped or started bound to the correct network, and the Java GUI will correctly reflect the state of the daemon and can communicate with it if running, in bounded time without hitting deadlocks or nondeterministic failure. Then you'd have to rewrite/rearchitect any pieces of the code which we can't understand, cannot prove correct, or proved to cause incorrect behavior under some circumstances. |
In which case what you're looking at might be unrelated to the problem I (and possibly others) have been having. As pointed out previously, when it gets stuck on "Syncthing is starting", files get out of sync -- which is why I noticed in the first place, as I don't usually pay much attention to what the service notification is saying. Currently, I have set Syncthing to run only when a charger is connected. Since I do partial cycles, this means Syncthing gets started once or twice per day. Fewer starts mean less of a chance for it to get stuck. It's less then ideal but borderline tolerable. |
In a previous comment I said:
Quoting from my logcat dump:
To me, it looks like I have been able to get Syncthing to not sync while the GUI is stuck on starting, but only after disabling "Bind to active network". It may be possible to reach this state without "Bind to active network" as well (possibly by switching Wi-Fi networks more times after triggering the endless ENONET errors), but I have not attempted (and unfortunately I'm working on other projects now). I think there are a lot of intertwined bugs, all caused by one or more (I don't know how many) state machine errors in the Syncthing-Android service management code. IMO it's valid to debug them all in this issue. |
I would agree as long as one solution fixes them all in one fell swoop. Things might get messy if this doesn't turn out to be the case, though ? Anyways, I think the only difference under Run Conditions between fork and upstream is the addition of the Run on a Schedule option. If that's the case, the Wi-Fi switching trick might mess up upstream, too, OTOH I could not find an equivalent bug report over there. |
BTW, for those who have Developer Options unlocked, it should be possible to figure out whether the Syncthing executable is running even though the wrapper claims it isn't, without having to resort to external debug facilities. Under Running Services (the description might be different in different Android versions), locate Syncthing-Fork and tap If the only thing currently running is the wrapper, you should notice the memory consumption to be low (mid 50s for me). When the Syncthing executable is also active, memory consumption will increase. The figure for me is mid 80s but the it may realistically vary depending on how many folders or devices you have and what tasks Syncthing is currently executing at the moment. I have switched the Schedule option back on and keeping an eye on what Syncthing is doing, will report if my findings deviate from the situation described in this post. |
I also have this issue, where the Syncthing service starts - either automatically or if I use "force start" (and I can see it connects and syncs on my other devices) but the UI shows "Syncthing is starting" and never gets the the point that it sees that the service has started. If I use "force stop" (or just let it timeout), the service itself will not stop (I can still see it connected on my other devices, and also I can access the local web UI from a browser), the UI and notification still say "Syncthing is starting", and the log has the message I have tried with both "Bind to active network" and without. I think that the problem may be - at least for me - that the wrapper can't figure out the address for the REST API and starts
Device: Pixel 4a running Android 13 build 230305 |
I tried to debug this, but after installing a debug package - which cleared the app storage and I needed to set things up again - I can no longer reproduce the problem: the wrapper correctly sets up the REST API URL and can connect to it. I should have used the export configuration option, but I kind of forgot it exists... |
This happens to me constantly, several times a day, every time there's a network change (like when i leave the area of WiFi or enter it back), but ONLY on my phone (original Pocophone), the tablet (a Samsung tablet) under the same circumstances doesn't get the issue. Not sure what's the difference. |
@jherazob I've got decent luck lately keeping the Bind Active Network option disabled. If you haven't done that already, you should go through each and every option and make sure they are exactly the same on either device. Make sure to leave no stone unturned. |
Checked and the phone and the tablet have the exact same settings, so it's (probably) not one of the options. Also tried it, the Bind Active Network option made no difference, whether it's enabled or not it hangs "Starting" so probably it isn't that. |
I haven't had the motivation to dig further into this bug. I wonder if instead of relying on log prints, we could semi-automatically log and visualize traces of events, to get more observability into how the various states should operate under normal operation, and what happens when the race condition hangs Wi-Fi handling. I've been trying out the Tracy profiler to track control flow and input latency on Dolphin Emulator, but since Syncthing-Android is a Java application, you'd need to use a different library (eg. https://www.google.com/search?client=firefox-b-1-d&q=java+tracing+library).
|
@jherazob Probably not going to like the sound of this very much, however I have seen indication that sometimes dismantling a folder, or even the whole misbehaving client, and setting it up again from scratch can fix obscure issues with Syncthing. If you decide to go down that road, it may be worth to start from as much of a blank slate as possible. PS: Don't shoot the messenger! |
I've found one race condition:
If you enter the starting state (by connecting to Wi-Fi), then disconnect from Wi-Fi while still in the starting state,
One way to fix this bug is to abort deferred calls to I'm not sure if EDIT:
For a more "correct by design" and comprehensive fix, it may be possible to schedule future calls to |
To add another data point: I've been struggling with the same issue for a while. I set up Syncthing to only run on wifi. Now, it always gets stuck on "Starting", though, and then uses up ridiculous amounts of battery to the point where my phone gets incredibly hot. At that point, changing the run conditions (e.g. "force stop / ignore run conditions") doesn't do anything anymore and I need to reboot my phone for the settings to take effect. I will try to look into the logcat output next time this happens. |
Happens to me too, seems to be caused by the VPN app. Before that, I never had those hang problems. With VPN running or started while the app already is running, I get this error behaviour. |
Just to let you know, I have this startup problem too but without VPN. |
Also occurs on Samsung Galaxy Tab A8 SM-X200, syncthing-fork version 1.27.6.0. No VPN. |
Description of the issue
Syncthing gets frequently stuck on Starting. The "Syncthing is starting" message can be seen on both the notification and inside the app under the Status tab.
This issue breaks the ability to run on a schedule, because the app needs manual intervention (Force Stop) for syncing to resume. The last sync indicator also breaks as the app appears to incorrectly assume that the Syncthing starting status is tantamount to synchronization having taken place.
Reproduction Steps
Enable "Run according to time schedule" so Syncthing starts and stops every hour.
Version Information
Device platform info
Will fill in later if deemed necessary.
Android Log
Will fill in later if deemed necessary.
The text was updated successfully, but these errors were encountered: