out of memory error comparing large files

differ · Post by **differ** » Fri Sep 17, 2004 3:43 am

Hi.

During compare of two 70 MB .wav files, ExamDiff Pro (v3.2c) fails and reports:

"Out of memory while trying to allocate 290462608 bytes: The access code is invalid. (Error 12)."

Machine is WinXP-H with 1 GB RAM and 3 GB swap.

While 70 MB files are big, I'm surprised ExamDiff Pro has trouble with them. I'm not sure how the compare algorithms work, but it seems to me that both files could be completely read into memory no problem and compared there.

Bug or limitation?

Thanks.

Post by **psguru** » Fri Sep 17, 2004 5:08 pm

Let me first state that the error message is incorrect; it should've said:
"Not enough memory". This will be fixed in the next release.

Now, why does EDP need so much memory? If you use HEX binary comparison
(meaning that you want to see details of what is different in binary files),
EDP allocates as much as 3 x FileSize -- since it needs to compare bytes
in their HEX presentation (1 extra byte is used for separation). That's why
you see a need for 290462608 byte for a 70 MB file. At some point such
buffers are copied form one to another, so during such short time periods
memory consumption can reach as much as 6 x FileSize.

That said, HEX comparison is really meant for files smaller than 70 MB. Even
if you had every byte shown, it's hardly useful for such large files. So
generally you want to check "Message about different binary files" under
Options | Misc | Messages.

As to why Windows wouldn't let EDP allocate so much memory, watch the
Task Manager: at some point you'll see that the amount of available memory
goes below the needs of EDP.

differ · Post by **differ** » Sat Sep 18, 2004 6:31 am

Thanks for the explanation. Indeed, checking "Message about different binary files" allows the comparison to complete, even though I cannot display the differences.

I must admit, however, I'm still baffled by why the memory cannot be allocated. Even according to your worst case, 2 files * 70 MB * 6 = 840 MB << 4 GB (physical + swapfile).

I have been monitoring memory usage with both Task Manager and Everest. Everest seems to be reporting that EDP is using 1786 MB! (see Used Virtual Memory) Even so, I don't understand why physical memory is being depleted while plenty of swap space remains available.

Windows memory management has always baffled me.

Does any of this make sense to you?

note, values are before / after running EDP

Physical Memory
---------------
Total 1023 MB / 1023 MB
Used 224 MB / 1014 MB
Free 799 MB / 9 MB
Utilization 22 % / 99 %

Swap Space
----------
Total 3926 MB / 3926 MB
Used 126 MB / 1123 MB
Free 3799 MB / 2803 MB
Utilization 3 % / 29 %

Virtual Memory
--------------
Total 4949 MB / 4949 MB
Used 351 MB / 2137 MB
Free 4598 MB / 2812 MB
Utilization 7 % / 43 %

Post by **psguru** » Sat Sep 18, 2004 1:29 pm

Windows memory management is indeed a complicated subject. What gets reported in Task Manager is not always accurate, for precise measurements you should use Performance Monitor. Also, Virtual Memory reported by the tools is not what EDP allocated but rather what the OS marked as possibly needed to be committed in the future.

I found reading of Raymond Chen http://weblogs.asp.net/oldnewthing/) blog explains a lot about Windows internals, including memory management.