Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Driver for PlayerOne Ares-M Pro keeps failing #822

Open
gnthibault opened this issue Aug 9, 2023 · 35 comments
Open

Driver for PlayerOne Ares-M Pro keeps failing #822

gnthibault opened this issue Aug 9, 2023 · 35 comments
Labels
bug Something isn't working

Comments

@gnthibault
Copy link

Describe the bug
affected program: indi_playerone_ccd
The following error keeps appearing very often, making the camera very hard to use:
[ERROR] Error connecting to the CCD (POA_ERROR_OPERATION_FAILED).

To Reproduce

  1. Run driver:
    indiserver indi_playerone_ccd
  2. Perform action
    Nothing specific, it seems more or less random
  3. See error
    In Device manager you will see: [ERROR] Error connecting to the CCD (POA_ERROR_OPERATION_FAILED).

Expected behavior
I expect the driver to work properly

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

  • OS:
    Description: Manjaro ARM Linux
    Release: 23.02
  • Version: v2.0.2
@gnthibault gnthibault added the bug Something isn't working label Aug 9, 2023
@knro
Copy link
Collaborator

knro commented Aug 10, 2023

@hiro3110i can you please check?

@hiro3110i
Copy link
Contributor

@gnthibault Thank you for reporting.
One chance, could you please upgrade indilib and drivers to v2.0.3 which is included in the latest KStars v3.6.6?
v2.0.2 uses PlayerOneCamera SDK v3.4.0 while v2.0.3 uses the latest v3.4.1.
Ares-M Pro is a very new device, so it may need the latest SDK.

Upgrading KStars to v3.6.6 will install indilib v2.0.3.
Alternatively, you can download the v2.0.3 indi-3rdparty source code from GitHub and build it as described in the readme.

@gnthibault
Copy link
Author

Thank you so much for your feedback.
First, this morning I finally found how to reproduce the error in a seemingly deterministic way:

1: power both usb and 12V for the camer
2: start indiserver indi_playerone_ccd
3: click the connect device button in indi device manager (launched from kstars in my case)
4: disconnect / or unpower the usb cable (which I do programmatically using pegasus ultimate powerbox)
5: try to reconnect or use any button from device manager on the camera, you will see an error (that's expected)
6: Now re-plug / repower the camera usb (which I do programmatically using pegasus ultimate powerbox)
7: You will see that you cannot reconnect to the camera though device manager, it will exhibit the poa error without exception
8: Try to restart the playerone driver from indiwebmanager, and then reconnect the camera from device manager, which is a workflow I perform routinely on other devices to programmatically reset them: you will still see the error

From there, the only way I found to reconnect to the camera, is to STOP the whole indiserver, and then start it again, then I manage to get rid of the POA error and reconnect properly.
I would like to avoid restarting the whole server, or use a separate server solely for the playerone camera, as it comes with additional complexity.

I am now trying to upgrade indi to v2.0.3 to see if it fixes the issue, but I have the feeling that it's independant of indi/sdk.
Thank you again for your support, I will report next experiment results here

@hiro3110i
Copy link
Contributor

@gnthibault According to the changelog of SDK, v3.4.1 has fixed some bugs around USB.
I hope it's fixed in the latest SDK.

@gnthibault
Copy link
Author

gnthibault commented Aug 10, 2023 •
edited
Loading

@hiro3110i
I updated Indi to v2.0.3
I updated indi-3rd-party libs to v2.0.3
I updated indi-3rd-party indi_playerone_ccd to v2.0.3

And now, I do not even see the device appearing in device manager, when I try to run indi_getprop for my camera, there is not return (No PlayerOne CCD Ares-M PRO.. from [...])

Return from indiserver in verbose mode is:

indiserver -vvv indi_playerone_ccd
2023-08-10T09:11:15: startup: indiserver -vvv indi_playerone_ccd
2023-08-10T09:11:15: Driver indi_playerone_ccd: pid=5013 rfd=6 wfd=6 efd=7
2023-08-10T09:11:15: listening to port 7624 on fd 5
2023-08-10T09:11:15: Local server: listening on local domain at: @/tmp/indiserver
2023-08-10T09:11:15: Driver indi_playerone_ccd: sending msg nq 1:

2023-08-10T09:11:30: Client 9: new arrival from 127.0.0.1:40876 - welcome!
2023-08-10T09:11:30: Client 9: read getProperties
2023-08-10T09:11:30: Driver indi_playerone_ccd: queuing responsible for
2023-08-10T09:11:30: Driver indi_playerone_ccd: sending msg nq 1:

I also tried with indi_playerone_single_ccd, but basically got the same result. This is much worse than expected. I am downgrading to v2.0.2 to check if the camera is back again to rule-out an hardware problem (the other camera driver seems to be working fine).

EDIT: Downgraded everything to v2.0.2, and camera cannot connect now... I really start to suspect a hardware problem now for that specific issue. will keep you up to date with further test results

@gnthibault
Copy link
Author

gnthibault commented Aug 10, 2023 •
edited
Loading

Okey, so I think I found the main rootcause of the issue (not 100% sure though).
The Playerone Ares-M Pro camera seems to rely on a very specific initializion/deinitialization order:

USB 3 input needs always to be powered first,
Then main 12V input needs to be powered afterwards, with apparently a small delay (I use 5 seconds)

I have done some tests where 12V is powered first, and I never managed to get the camera to work properly.
When the camera is in this failed state (no usb communication), then one needs to

  • unpower 12V, then unpower USB, then power USB then wait ~5 seconds, then power 12V

Interestingly, the following, that seemed equivalent to me doesn't seems to work:

  • unpower USB, then unpower 12V, then power USB then wait ~5 seconds, then power 12V

It took me quite some time to go back and forth between software versions, start/stop procedures, so I am not 100% sure that what I report here is fully deterministic. But to me it looks like a purely hardware/firmware camera related issue.
I'd be very glad if another player one Ares-M owner or company developer could report its test report.

@gnthibault
Copy link
Author

Ok another aspect that I didn't managed to mention earlier, is that, in order to get a successful "reset", I need to restart completely the indiserver, just restarting the driver itself (through indiwebmanager) is not enough.
I reported the problem in the indiwebmanager project, but eventually I am wondering wether it wouldn't make sense to have it here:
knro/indiwebmanager#55

@gnthibault
Copy link
Author

I have contacted playerone support to get some feedback hopefully.

@gnthibault
Copy link
Author

Ok, so I didn't get any relevant feedback from PlayerOne, but I had the opportunity to test-out the camera under windows with their driver + Ascom driver.
Interestingly Sharpcap didn't managed to connect to the camera, but PhD2 did, and managed to stream frames from the device.

So in the end it seems that, it is not a hardware problem on the camera side, I will loop with PlayerOne with this information. Now the issue can still be either on the sdk side or the indi driver side.

@gnthibault
Copy link
Author

I have implemented a setup where I consistently restart the indiserver as well as implement a proper power-on/power-off strategy.
I seeems this way the camera is working as expected, I still need to do further testing, but for me this is definitely a software issue specifically on linux either with sdk or indi driver

@hiro3110i
Copy link
Contributor

@gnthibault Your deep investigations are very helpful, thank you.
I'm also discussing with PlayerOne engineer.
They're always cooperative and quick.
I'd like to expect effective countermeasures.

@knro
Copy link
Collaborator

knro commented Aug 14, 2023

We received more reports about this failure, could it be some SDK issue?

@hiro3110i
Copy link
Contributor

@knro I don't know sure at this moment.
I hope that SDK will solve the stability issues around USB.
On the other hand, it may be necessary to consider separately whether it is possible to avoid the problem with indi driver.

@gnthibault
Copy link
Author

I got some feedback from playerone saying that their hardware engineers are looking into the problem, I will report any other news I receive here.

@hiro3110i
Copy link
Contributor

@gnthibault I also received messages from Lei-san (PlayerOne engineer).
There seems to be a little more complicated than the Saturn-C issue #824.
I'd like to wait for further information.

@gnthibault
Copy link
Author

Interestingly, I cam across this interesting post from Pegasus support: https://pegasusastro.com/forum/viewtopic.php?p=387#p387
That was for ZWO camera though, but the symptoms seems much similar to the one I am experiencing

@hiro3110i
Copy link
Contributor

@gnthibault Today, I updated libplayerone and indi-playerone with the latest SDK v3.5.0.
I'm not sure if your problem will be fixed, but I'd appreciate if you could test it.
Thank you for your cooperation.

@gnthibault
Copy link
Author

Thanks, this weekend I cannot, but I'll do next week

@hiro3110i
Copy link
Contributor

@gnthibault I had a quick chat with Lei-san tonight.
Unfortunately, this update cannot solve the problem, but engineers are still looking into it.
The current best practice is to connect the USB first and then 12V power later, as you pointed out.
I apologize for your inconvenience.

@gnthibault
Copy link
Author

I have more tests to do, but it seemed like in my context this might not be an issue as problematic as I initially thought.
I am working on a fully robotic observatory and I can make the startup/stop routine perfectly reproducible.
My main pain point was simply to update indiweb manager to systematize indiserver restart for each camera. But it seems I am more or less here (will push my last commit to the PR today)

@glundby
Copy link

glundby commented Nov 4, 2023

Title | Player One Saturn-C SQR driver indi-playerone-ccd v1.12 interface 6 Crashes system in many ways
WDBT-002982
StellarMate
StellarMate OS
19 mins ago
Unassigned
App not used
1.7.7
SkyHunter mount, Player One Saturn-C, ASI462MC

Player One Saturn-C SQR driver indi-playerone-ccd v1.12 interface 6 Crashes system in many ways

Sometimes it kills connection to mount, frequently it disconnects memory stick, now and then it takes down Kstars. If not it stopps in capture of second sub. This happens always when exposure is longer than 1s, shorter may get through several cakptures and downloads.To check my camera I have tested it on USB3 from PC with FireCapture and SharpCap and it works perfect.It used to work well before my last upgrade to Kstars 3.6.7 Stable

@knro
Copy link
Collaborator

knro commented Nov 5, 2023

I don't have this camera so can't test. If you have SM OS, you can file a ticket so we can do remote support to diagnose the issue further.

@glundby
Copy link

glundby commented Nov 5, 2023

Sorry! Just found out that the RPI4b cannot handle two USB3 cameras at the same time. It overpowers the USB busses.
I need to add a powered USB bus.

@hiro3110i
Copy link
Contributor

@glundby Thank you for reporting. Could you please give me more information about your issue?

For your reference, I'm always using KStars/Ekos with Poseidon-C and Sedna-M which is connected to USB3.0 port on Raspberry Pi4 at the same time. Or another case, Uranus-C and Neptune-C II is working well at the same time in Ekos.
Each of use case is long exposure with short exposure guiding. (not high frame rate like a video capture because RasPi4 doesn't have enough performance for video capture)

@glundby
Copy link

glundby commented Nov 6, 2023 via email

@glundby
Copy link

glundby commented Nov 6, 2023 via email

@hiro3110i
Copy link
Contributor

@glundby One point, could you please check your device settings of Saturn-C on your Ekos?
The latest version of Ekos has experimental functionality of "Fast Exposure" in the "Options" tab.
It's useful for long exposure but interval of each exposure is very short.
You can turn off before exposure loop.
Device settings are saved in ~/.indi folder. You can try to delete old settings in this folder.
Another check point is "USBBandwidthLimit" in the "Controls" tab. Less than 35 is suitable for dual camera system.

If the behavior is still unstable, I recommend rolling back to a previous stable working version.
ScreenShot 2023-11-07 21 54 09
ScreenShot

@glundby
Copy link

glundby commented Nov 7, 2023 via email

@hiro3110i
Copy link
Contributor

@glundby Some new features have been added with the update, but they are not critical for astrophotography.
When connecting multiple peripheral devices such as cameras to Raspberry Pi, power supply capacity is often questioned.
I'm getting very stable operation using Jackery's high capacity mobile power supply.
We also need to choose a high quality USB3.0 cable.
On the other hand, reset problems around USB are often caused by problems with the firmware of the USB chip.
I apologize for any inconvenience, but I recommend that you use a stable version for a while.

@glundby
Copy link

glundby commented Nov 10, 2023 via email

@hiro3110i
Copy link
Contributor

@glundby Most of the power of Image sensor and USB bus are consumed at the read out timing.
If the camera and/or bus hangs while reading out the first picture and is unable to take the second picture,
we cannot deny the possibility that the power supply is the cause.
Because I can shoot stably all night without any problems.
I suspect it's more of a hardware issue (power, timing, noise, temperature something like that) rather than a logical software issue.
Of course, there might be a software solution to avoid these problems, but it is not a fundamental.

@glundby
Copy link

glundby commented Nov 11, 2023 via email

@hiro3110i
Copy link
Contributor

@glundby I received Saturn-C camera and tested it with the latest indi_playerone_ccd driver v1.13.
It works very well on my Raspberry Pi4/4GB (Ubuntu Mate 20.04).
Could you try it again with the latest driver?

You should contact to PlayerOne support if you still have problem on Saturn camera.

@glundby
Copy link

glundby commented Jan 16, 2024 via email

@glundby
Copy link

glundby commented Jan 16, 2024 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@gnthibault @knro @hiro3110i @glundby