Simulation not progressing

[SFC-CA]

Hi all,

I have some simulations with multiple meshes being run on Sabalcore. Some of them have seemed to have halted, no output files are being updated. For example: one run with identical geometry except for fire location has run 1200s in the time it has taken others to run 65s and 185s.

I have sprinklers with particles in the files, but the last outputted temperatures in the csv do not indicate that sprinkler activation is imminent, and i did not have problems with activation in the other one.

I am using hvac to model leakage in a warehouse, don't know if that is relevant.

The last entry in the out file gives time step as about 0.008s. This is on par with the simulation that ran successfully.

What steps should I take to identify the issue here, particularly between if it is my model inputs causing the problem or a hardware issue?

What outputs should I be looking at?

Would FDS hang like this if it couldn't communicate with one of the nodes, or would the run terminate with an error code?

Thanks in advance.

From out file:
Current Date : February 25, 2025 08:01:24
Revision : FDS-6.9.1-0-g889da6a-HEAD
Revision Date : Sun Apr 7 17:05:06 2024 -0400
Compiler : Intel(R) Fortran Compiler for applications running on Intel(R) 64, Version 2024.1.0 Build 20240308
Compilation Date : Jul 22, 2024 21:27:27

Number of MPI Processes: 42
Number of OpenMP Threads: 2

MPI version: 3.1
MPI library version: Intel(R) MPI Library 2021.12 for Linux* OS

[mcgratta]

This is a bit like me asking you why my car did not start this morning. I would need an input file (one only) and I would run it on my computer.

[SFC-CA]

Update on some things I've tried.

I have tried removing the HVAC components from the model, modelling leakage as holes instead, but my simulation still halted.

Next, I have removed particles from the sprinklers (with the above modification as well) and the simulation appears to be continuing to run.

So maybe something to do with the particles in my model?

They have been defined as:

&PART ID= 'WATERDROPS',
SPEC_ID= 'WATER VAPOR',
DIAMETER= 1000.,
AGE= 2.5, // water particles to be used by the sprinklers

&PROP ID= 'Sprinkler',
QUANTITY= 'SPRINKLER LINK TEMPERATURE',
RTI= 50.,
OPERATING_PRESSURE= 6.2,
K_FACTOR= 202,
SMOKEVIEW_ID= 'SPRINKLER_PENDENT',
SPRAY_ANGLE= 30.,80.,
ORIFICE_DIAMETER= 0.020,
ACTIVATION_TEMPERATURE= 100.,
PART_ID= 'WATERDROPS',

From my understanding, this should result in a flow rate of 500 LPM (0.008m3/s), with an orifice diameter of 20mm -> velocity of 26.5m/s -> timestep of around 0.008 seconds with a cell dimension of 0.2m, which is roughly what is indicated in the out file.

When using the K factor, you don't need to include any additional discharge coefficient for deriving velocity from volumetric flow, is that correct? That's what I've assumed in the above.

I'd still like to know if there was a hardware issue with communication between nodes if that would cause this freeze, or a termination with an error message? I am more naive about the hardware side of things. Without knowing this, I can't rule out that I have just gotten lucky with the nodes I got on this simulation or if it is a problem with my particle definition.

[drjfloyd]

I am running your case with 6.9.1 on our cluster.

[drjfloyd]

With the file you attached around what time did you start noticing things slow down?

[SFC-CA]

Thanks, Jason.

This was the last output in the .out file.

Time Step: 13200, Simulation Time: 184.32 s, Step Size: 0.00845 s, Pressure Iterations: 2

[drjfloyd]

Your case is out to 450 s on our linux cluster. I am not seeing any unusual time behavior. The wall time per simulation time increases at first as the fire grows (expected as velocities are increasing) but then becomes linear. I am seeing the same time step that you did at 184 s of ~0.08 s. We did find a memory bug with particles causing a memory leak. If that was happening in this case it could cause a slowdown. You could try the latest nightly build on the test bundle page (https://github.com/firemodels/test_bundles). However, in my run the first sprinkler wasn't until 189 s or after the time you indicate things started slowing down. It could be some issue with hardware. We have seen that sometimes communication hangs without dropping.

[SFC-CA]

Thanks for that, Jason, much appreciated.

So I guess at this stage I can't really rule anything out.

However, in my run the first sprinkler wasn't until 189 s or after the time you indicate things started slowing down.

With a variant of this model where the only thing I have changed is the criteria for freezing the HRR, I have had it stall at around 30-40s of run time. So it seems independent of the sprinkler activation.

I might try the nightly; I am using a third party for hardware, so I'll have to get them to sort that out. I don't personally have access to hardware to run this size model in reasonable time.

You say you were running it on 6.9.1, I'm assuming that is the official release, or was it a nightly with a fix for the potential memory leak ?

If I have particles in my model is there memory allocation for them before they are injected into the domain? Could I still get a memory leak problem before the sprinklers are activated?

We have seen that sometimes communication hangs without dropping

Do you mean an indefinite hang, or just a delay? In my case, when the stall occurs it will last for days, so I assume its indefinite.

Thanks again!

[drjfloyd]

I am running as well.

[SFC-CA]

Apologies, I should have mentioned the threads per MPI initially.

[SFC-CA]

In regards to dividing the domain into meshes, I've always aimed for about 250,000 cells per mesh and 2-4 threads per mesh. I've felt this is an ok balance between resources and run time. I've tried to limit the total number of meshes mainly due to this comment in section 6.3.4 in the UG:

... the exchange of information across mesh boundaries is not yet as accurate as cell to cell exchanges within one mesh

But reading section 3.3.1 MPI Efficiency, it discusses dividing the domain into very small meshes. I'm guessing this has been done for the purposes of the scaling analysis without necessarily a focus on result accuracy.

I've worried that the more meshes would compound inaccuracies in communication between meshes. Does anyone have any comment on concerns for dividing the domain into very small meshes? If the issue is to do with assigning more than one OPENMP thread per mesh, I can sub divide my domain more and only assign one thread. Doesn't get to the bottom of the problem, but it would help me in a practical way.

How concerned should I be about the mesh to mesh inaccuracies? I primarily model bulk smoke movement in large environments, so there is already recognised inherent uncertainty and safety factors are applied. I understand that the responsibility for justification of the results lies with the end user, but any guidance is appreciated.

[SFC-CA]

Update:

The runs of my version of the model with no HVAC and no particles have also stalled, at different stages on different hardware on Sabalcore.

So at the moment it seems the only common factors are the multiple OPENMP threads and Sabalcore. And my input file, obviously, but I don't think I'm doing anything but the basics in it.

[drjfloyd]

While there is some error at mesh boundaries, in many cases it is not significant. For something like a smoke control simlation in a large building, I would not be very concerned with errors at mesh boundaries.

The simulation running with two threads per mesh ran got to 444 s overnight. It has not stalled out; however, compared with the non-OpenMP run above it is not running as quickly.

[mcgratta]

This is the current state of my run, with 42 MPI processes, 2 OpenMP threads per process

[drjfloyd]

You are seeing similar times to me for 42 x 2. What happens if you run with just 42 x 1?

[mcgratta]

There may not be a speed-up for the case with 2 OpenMP threads per MPI process because when we compile FDS with OpenMP directives, there is decreased optimization of the 3-D loops because, in general, it is more difficult to implement optimization strategies for OpenMP. That is, if the compiler cannot be guaranteed that only one process is used to compute the loop, it cannot do certain things to make the computation faster.

In any event, I'll run the case with 1 OpenMP thread and compare to that with 2.

[mcgratta]

It seems that the 2 thread case runs slower than the 1 thread case, probably for the reasons I mentioned. I suggest that you run with 1 OpenMP thread.

[SFC-CA]

Thanks for all your help, I'll go ahead with one OPENMP thread from here on out.

So neither Kevin or Jason had the simulation hang? Whereas, I've had it hang everytime but at different simulation times. I've been running a version with one OPENMP thread and it is up to 900 odd out of 1200s. One of the previous runs made it this far before stalling, though...

Is not the speed of simulation a separate issue to the stalling? I guess it's hard for you to comment on the stalling when the issue hasn't been replicated.

[SFC-CA]

Update, I have also had stalling when OMP_NUM_THREADS=1.

I am currently doing a run with FDS 6.8.0 as per a suggestion by Sabalcore. I noticed that when I start the run with only one mesh per MPI process the .out file specifically says OPENMP disabled, but this line is not included in FDS 6.9.* .outs. Does this mean OPENMP is still being utilised in some way?

[rmcdermo]

At some point, I think it must have been between 6.8 and 6.9, though I am not seeing this noted in the release notes, we started releasing two executables, one compiled without openmp (fds) and one compiled with openmp (fds_openmp). This was because the overhead of going into openmp loops with 1 thread had a significant cost. So, you have to specifically ask for fds_openmp now. If you were running fds only then no openmp information would be printed.

[SFC-CA]

Thanks, Randy,

In the user guide, I can see under running on windows you must invoke fds_openmp to use multiple threads, however this is not specified under the linux and macOS section. Is this still the case?