We have identified that a VI that contains a majority of the DDS calls is leaking memory. There are several loops in this VI that either read, write, or read and write. In an effort to narrow down which calls are having issues I have created a stand alone application and am doing each call individually.
The first one test was what we call action response. Below are screen shots of the loop and its states.
As you can see it contiually reads from the FGV and publishes the results
In 1 hour and 17 minutes it consumed 7.5 MB of memory.
cRIO 9045, LabView RT 2020, latest version of DDS for LabView
Hi FFLy
Can you provide the ctl of your DataType? Are you doing blocking or polling read? If so, Can you try with polling read to check if the issue happens?
Regards
Ismael
Ismael,
This is a write not a read so blocking/polling don't apply.
Attached is the control.
QoS is this "BuiltinQosLibExp::Generic.KeepLastReliable"
Imael,
Some testing results on a read. Still leaking memory in both polling and blocking reads.
Blocking: 3.3MB/20 minutes
Polling: 2.8 MB/20 minutes
QoS: BuiltinQosLibExp::Generic.KeepLastReliable
I have attached the control
Thanks for the attachments. I'll try to reproduce the issue on my environment.
Any updates on this?
Hi, Ismael is OOO Wednesday - Friday this week. I'm sure he'll take a look as soon as he's back on Monday.
Thanks,
Ken
Hi FFly,
I'm having a look at this. I haven't found any memory leak at this moment (testing the vechicleactionresponsestamped writer). I'm going to continue investigating this and keep this thread updated.
Regards,
Ismael
Are you testing in Windows or on a cRIO with RT Linux?
I'm trying to reproduce it on Windows. The behaviour is the same in both platforms and it is easier for us to detect a memory leak on Windows.
There is memory leak in 3.1.1 when using blocking read that is fixed in the incomming release 3.2.0. I have not observed any leak in the write operation of both types neither in the read when using polling read. I'm doing basic read/write operations based on the generated examples. Similar as you do for what I see in the snapshots. Is there something else I need to consider?
Please have alook at the following questions:
- Does the memory consumption keeps growing or it stabilizes after some time? LabVIEW manages the memory at its convenience and that memory grow you observe might be part of the LabVIEW normal operations.
- Does it happen with the default QoS?
- Does the lvrt process memory returns after you close the VI?
I'll try to reproduce the scenario in a cRIO and see the memory behaviour.
- Does the memory consumption keeps growing or it stabilizes after some time? LabVIEW manages the memory at its convenience and that memory grow you observe might be part of the LabVIEW normal operations.
It keeps growing
- Does it happen with the default QoS?
Have not tired that. I will giveit a try and let you know.
- Does the lvrt process memory returns after you close the VI?
Not sure. I have never checked that. I will check and let you know
Is there something else I need to consider?
Would it be helpful if I sent the test VI's I am using?
If you could send the problematic VIs would be a great help
Another question. Do you use keys in the types you attached?
No
I generated examples using the types you attached with Complex Type Generator. Then modified them to populate all the fields using the iteration number of the main loop.
I haven't found any memory leak in Windows when doing polling reading and writing. Then I tried in the cRIO-9031 LabVIEW RT 2020. This is what I've done.
I ran the writer vechicleactionresponsestamped (VehicleActionresponseStamped Writer Example.vi) using QoS profile BuiltinQosLibExp::Generic.KeepLastReliable for 102 minutes approx (61692 iterations). I register some readings of the time and memory usage. The memory was taken using the "Get memory Status.vi" and output "memory allocated for LabVIEW". Also run the "top -p $(pidof lvrt)" in the cRIO while running. The "memory allocated by LabVIEW" from Get memory Status.vi and the RES value of the top command are the same.
I observed a memory growth from 67832 KB to 71564 KB in 60 minutes. Then it kept constant for the next 40 minutes until I stopped the vi. The gif in attached file memw.zip (cannot upload images larger than 2MB) is around 6.5 min long. It shows how the memory allocated for LabVIEW parameter remains constant. Also in the background top command is running in the cRIO.
With the HeartbeatStamped type I see a similar case. Not memory leaks detected in Windows when running the reader and writer examples at the same time. Then run reader and writer examples in the cRIO for 144 minutes. In this case memory fluctuated from 69552 KB until it reached 77528 KB were it kept constant (see gif in memrw2.zip) during 32 minutes until I stopped the vi.
This might not be a memory leak. A memory leak would lead to a constant memory growth over time since we are executing the same operation in a loop. However, in this case the memory reaches a stabilization point and then it keeps constant. This would happen for buffers fragmentation, LabVIEW internal memory management or some other reason. Can you try it running for long time?
Just did a 24 hour test of polling instead of blocking read. Still leaking.
Start: 1334 MB available
End: 1265 MB available
I have attached the test VI/project
After running for 7 days it is now down to 801.5MB free
Sorry for the delay. I've been out for Christmas. I'll have a closer look to this. In your long execution test, did you have a parallel writer running or just the rti_test.vi? I observed you are measuring the free physical memory and free virtual memory. Those measurements refers to the whole system not just an specific application. Do you have any other process running on the cRIO that might be consuming memory?
I'll do a long execution measuring the memory with the Get Memory Status VI. that VI measures the memory allocated only by the LabVIEW process.
The only thing running was the VI I sent you
Hi,
I've running your test VI and found the following. The vi "heartbeatStamped Create Advanced reader.vi" has been modified. I don't recommend modifying the generated VIs. It would lead to unexpected behaviour. Apart from that, in order to isolate the reason of memory growth I run some tests and found the following measuring the memory allocated by the LabVIEW process in the cRIO:
- The memory grows up event if no read operation is done in the loop.
- It also happens with the default QoS.
- In my environment, it happens in domain 0. If I test it in domain 145 the memory kept constant for 102 minutes. I only saw a memory grow when I created another participant in the same domain on a different machine of the same network.
The domain fact gives me a clue. Do you have in your network other participants in domain 0 publishing discovery traffic? Could be that the memory increasing is the information stored about other participants (or other entities) in the same domain along the time? If so, can you try running it on a different domain?
When you say the generated VI has been modified what do you mean. This VI is lock with a PW and it has not been modified so I am not sure how it could have been.
The operating environment is isolated to specific vehicles. They are isolated from one another so there should not be any other participants on domain 0. I will verify that with our network guru's just to be sure. I will run some more tests this weekend on different domain.
I have verified that we are indeed running on a totally isolated network both on the vehicle and the test setup.
A question for you: Is domain ID 0 a "special" (like IP 127.0.0.1)domain? You indicated you changed to 145. Is there a reason you selected that domain? Should we not be using 0?
The Domain Id 0 is used as the default domain where participants announces themselves. Despite a participant in domain 0 should ignore announcements of participants in domains different than 0. It is a good practice using a domain different than 0. I chose 145 as a random one different than 0. No special reason for it.
If I open the "HeartbeatStamped Create Advanced reader.vi" included in the rtitest.zip I see one of the input parameters in the Call Library Funcion is not correct. Then, if I check the revision history of that VI the following entry is shown: "rev. 61 Thu, Dec 21, 2023 6:23:06 PM XXXX Type Definitions used by this VI were modified.". I replaced the name with XXXX. If the Type Definition changes is necesary to generate new VIs for the new Type Definition. Don't try to modify the previous ones.