windows access denied while trying to remove file

[mathisloge]

Hey, I don't really know if this is the place to ask a question, but I'm currently adding github actions testing workflow for mapnik and while doing so, I'm experiencing boost::filesystem::remove: Access is denied error.
This only happens on the windows-latest ci machine. The file which should get removed is created by the test itself and it is checked before removal if the file exists.

Normally I would use process explorer to see which program hold the lock of the file, but as far as i know there is not a possibility to check this on the CI.

In summary it works on linux and macos but not on window (only ci). On my local windows machine everything works as expected.

Do anyone of you have a hint what I can do about it or how to debug this further?

The workflow:
https://github.com/mathisloge/mapnik/blob/gh_ci/.github/workflows/build_and_test.yml
latest run:
https://github.com/mathisloge/mapnik/actions/runs/1466056507

[dsame]

@mathisloge
I see the test/data directory is the link to the mapnik/test-data directory.

I not quite understand the workflow yet, but while am still investigate the the problem,

can you please doublecheck

• mapnik/test-data is cloned to the CI machine
• test/data is correct link to mapnik/test-data

and next try to fix "access denied" with the following cmd step:

TAKEOWN /F your_dirictory

Also it was noted the similar problem could happen with some old version of boost:

https://stackoverflow.com/questions/55982063/having-a-problem-with-boostfilesystemremove-all-in-windows-10

[mathisloge]

Hi @dsame
first thanks for investigating!

The test data is correctly copied to expected paths.
I'm using latest boost(1.77) so the linked bug should be fixed.

Regarding the takeown command:
The file to remove (in this case test/data/shp/boundaries.index) is created by shapeindex.exe before. I've downloaded the whole build dir and checked if the file exists (the runtime code checks it too) and the file was created.
How would I use takeown when the file don't exists before the test. Normally the file will be deleted after the test run.

On my local windows machine this all works. (windows 10 & 11 using vs2019 and vs2022)

[mathisloge]

I've also used std::filesystem::remove with c++20 instead of boost::filesystem::remove this change also worked on github actions windows-latest. Unfortunately the c++ min version is c++14, so we can't use std::filesystem.

So the normal flow should be correct so it has to do something with boost::filesystem

[dsame]

@mathisloge
Due to the private nuget repo i am not able to reproduce the problem, but still i am almost sure we have some problem with the submodules, so can you please to try in your repo

TAKEOWN /F test\data before running the ctest step
or alternatively can you please generate some short-living tokens for nuget repo and email them to me?

[mathisloge]

So I've now added the takeown cmds to the workflow:

• workflow file
• mathisloge/mapnik@7bf448f
• workflow run with takeown

the dir build\windows-ci\out\test is the copied test/data submodule.
But the issue is still there.

I'm not sure if this is a real access problem, since std::filesystem::remove works. Maybe I have to open an issue at boost filesystem.

Do you have any tips on how to debug such things better?

[dsame]

@mathisloge

Yes, the boost support could understand the problem better than i do

But i'd try the following tools to find out the process which holds a lock on the file:

https://docs.microsoft.com/en-gb/sysinternals/downloads/handle
https://lockhunter.com/

Is not it possible to get the credentials for nuget repo in order i can reproduce and investigate the problem?

[mathisloge]

I'll send you a PSA that includes read rights later. Note however that you won't be able to write to the repo. (I'm currently not at home on my PC)

I've also created a issue at boost/filesystem boostorg/filesystem#216 and tried to create a small reproducable example but i couldn't reproduce the error here https://github.com/mathisloge/boost-repro

Can you run github actions locally?

Thanks so much for your help!

[mathisloge]

I've also add handle <path> to be called before boost::filesystem::remove(<path>)
https://github.com/mathisloge/mapnik/runs/4244078451?check_suite_focus=true

looks like the unit test still holds a handle. But I can't find a thing why it has a handle.
IIRC the msvc stl lib reuses handles which are used by the same process. If I look at the code of boost it doesn't look like that. Will investigate that further

[dsame]

@mathisloge just for clarification: i am requesting the read-only token for nugets repo.

Having not this token causes my build to fail on "Setup NuGet Credentials" step

https://github.com/akv-demo/mapnik/runs/4237293494?check_suite_focus=true

[mathisloge]

I have narrowed it further down. Seems some interplay of the memory mapped file and the resulting file.

[mathisloge]

i'll mark this as answered as it is related to memory mapped files and some special handling of stl versions on the ci.
Related boost filesystem issue: boostorg/filesystem#216