Out of memory error

[JeremyNicoll]

V5.0.1.14

I tried to compare two roughly 32 MB .exe files (installers for ZoneAlarmPro); EDP reported

out of memory while trying to allocate 256.0 MB,
do you want to compare these files in binary mode byte-by-byte ... ?

The PE plugins were turned on.


Before anwering the question, I looked in temp hoping to find the plugin result files, and found two recent large files. I was surprised to see that they were called

ED96EB~1.EXE and EDB96D~1.EXE

because I'd assumed that they'd be text files. I looked at one in my text editor and indeed it WAS a text file. These files were both about 157,000 KB in size and a little more than 2 million lines long.

I thought the out-of-memory message was a bit ambiguous when it referred to "these files", as it wasn't altogether clear whether it meant the original user-specified files, or the intermediate files generated by the plugins.

I answered "yes" to the question, and got the progress box telling me that plugins were running again. It would be useful if that progress box specified which plugins and on what file... Anyway at the end of that I was told the files were different. Maybe I pressed the wrong button, but I wasn't told what the difference was nor where it was.

It seems to me that rerunning the plugins (which took ages) was a waste of time. Why weren't the results files from the first attempted compare used again? The original files had not changed.

I know from running compares on much smaller EXE files with the PE plugin active that it's normal for identical .EXE files to be shown as different because one line near the top of the plugin results file shows the full path of the file being dumped - and obviously these are different in the two PE-produced files. I don't know if there's some way to have PE not generate those lines, or have EDP ignore them, but it's a pain. As it is I suspect that that may be the only difference in the compares I ran but don't yet know that for certain.

Obviously at some stage EDP deletes the plugin-results files. I applaud that as good practice; so few programmers bother to delete temporary files. But maybe the point where you choose to delete them needs changed, so that in circumstances like mine where EDP offers to re-do a compare they don't have to be generated again?

Or perhaps EDP instead of just putting temporary files in %temp% like:

%temp%\xyz.pqr

could put them in

%temp%\EDP\yyyymmddhhmmss\xyz.pqr

where yyyymmddhhmmss represents the time that this instance of EDP was started. I'd like it if EDP didn't delete any of the files it puts into that directory until that instance of EDP is about to close, and/or also only deleted temporary files as it goes along if they are smaller than some user-specified threshhold size - so that big ones don't have to be regenerated.

I suppose it's possible that if someone was comparing two enormous file hierarchies that that could mean too many temporary files were being held, which might be a problem.

As it stands for my problem, because EDP has deleted the intermediate files, I can't do anything manually to see what the difference(s) that EDP told me exist(s) actually is/are.



I did try again with "treat binary files as text" turned on - and the plugins were run again which surprised me. I guess this means that the plugin processing occurs before EDP's other options take effect? If that's so, then doesn't the labelling of the plugin-results files as .EXE confuse EDP when it is about to compare them?

[psguru]

Perhaps you could explain what you are trying to achieve. Running the Dump PE32 plug-in on such large files will produce huge text files, which will include raw hex dumps of the executables. Why not turn off this plug-in and use the binary comparison instead? If you don't want to change your options, you can also specify this in the Browse dialog.

[JeremyNicoll]

I was unaware that the PE plugin shouldn't be enabled for routine .EXE compares, and have indeed now turned it and the others off. I can't recall when I turned the plugins on, but it might have been during evaluation when I looked at what they did to small test files of the appropriate sorts.

I take your point about altering the option, which I eventually did, however I was invoking EDP using the shell context menu 'specify first item' and 'compare to "first item"' options which seem to dive straight into executing the compare with - I suppose - the currently set options applied. Is there a way to have that type of use stop at the Compare window with both filepaths filled in but allowing the user to specify the options to be used? I'd certainly prefer it, because I tend to use EDP in bursts and then not for days by which time I don't recall what options I had set last time I used it.

I also just noticed that if you try to set "specify first item" against an empty file, it doesn't get set. I suppose that's sensible, but I might have expected Windows to play the 'stupid user' sound...


I think my general points about not needlessly regenerating intermediate files is still worth commenting on though...

[psguru]

Is there a way to have that type of use stop at the Compare window with both filepaths filled in but allowing the user to specify the options to be used?

You can use the "Compare to..." command to get the Compare dialog.

I think my general points about not needlessly regenerating intermediate files is still worth commenting on though...

It's not easy to change the behavior there, and it works in the majority of cases. In fact, it works in all cases correctly, so the only argument here is performance. However, optimizing this area may lead to caching problems, where a changed file is not compared correctly because an older temp file was reused.

[JeremyNicoll]

You can use the "Compare to..." command to get the Compare dialog.

Yes I know, but that ignores the filepath I've just saved away as 'first item'. Instead I get the thing I chose 'Compare To' for, and the second file - so far as I can tell - is whatever was last used as a second file.

...so the only argument here is performance. However, optimizing this area may lead to caching problems, where a changed file is not compared correctly because an older temp file was reused.

Yes, well performance is the point isn't it? Of course your app would need to be careful to check that the source file was the same as it had been when the plugin was last initiated, but your app does already have code that (if it's on) notices changes to files being compared.

[psguru]

Yes I know, but that ignores the filepath I've just saved away as 'first item'. Instead I get the thing I chose 'Compare To' for, and the second file - so far as I can tell - is whatever was last used as a second file.

So why not use the "Compare To..." to on your first file (as it's designed to work) in order to the select your second file from the Compare dialog?

Yes, well performance is the point isn't it? Of course your app would need to be careful to check that the source file was the same as it had been when the plugin was last initiated, but your app does already have code that (if it's on) notices changes to files being compared.

As I said, it's not easy to do. We have a large list of features planned for the next version, and implementing this kind of optimization has to be weighed against other things we want to do. But I do appreciate your suggestions.

[JeremyNicoll]

So why not use the "Compare To..." to on your first file (as it's designed to work) in order to the select your second file from the Compare dialog?

Because being able to select both filenames via the shell is significantly easier if you have both directory viewers open already - there's no tedious navigation required to find the second file's path and then choose it.

I've found a way that seems to work, though: specifying "/sd" in View -> Options -> Global -> CLI options

[psguru]

I've found a way that seems to work, though: specifying "/sd" in View -> Options -> Global -> CLI options

Good point.

[MSpagni]

May I refresh the request for a "stupid" binary compare?
When the files are binary and huge this can be done with less memory and proved for me much more useful than the simple message "files are different".

[psguru]

If you are referring to the "Implement fast binary comparison mode (byte-by-byte comparison)" feature, it is planned for the next release. However, it will not take significantly less memory since both files will still be shown in HEX format for detailed comparison results.

[MSpagni]

Yes, right that. Good.
And you're right: the comparing algorithm is far simpler, but the memory requirements are not so different.

So, what can I do to compare huge binary files?
I think I'll do a porting to windows of my old DOS "COMP" clone...

[psguru]

So, what can I do to compare huge binary files?

Not much until you get more RAM and use the 64-bit build of EDPro. Again, you can still find out whether files are different today, and having binary details for really huge files can be overwhelming.

[MSpagni]

having binary details for really huge files can be overwhelming.

For sure. Indeed my "COMP" allowed to specify the maximum number of differences (i.e. different bytes) allowed, stopping if it's reached, and also an option to simply count the number of different bytes without showing anything else.
Even without knowing each different byte, I find very useful to know the total number of differences.

That program had no file size limit (other than the FAT16 or FAT32 intrinsic one) because it worked loading in memory only chunks of both files at a time.

Maybe EDP can do the same.
Since there is no need to "resynchronise" the views because they are always forcefully in synch, there is no need to convert in hex the whole of both files in advance.
The visualizer can work by simply loading and converting to hex (and comparing) only the data to fill a screen each time.

To store the address of every different byte can be very memory hungry, so the "find next difference" and "find previous difference" command can be done by scanning the files in real time.
Of course, if the next (or previous) difference is at the other end of the file the response time can be a bit slow, but that's always better than nothing.

[AlexL]

...So, what can I do to compare huge binary files?
I think I'll do a porting to windows of my old DOS "COMP" clone...

You should do this: viewtopic.php?f=3&t=1208#p3142 (see the last post in that thread).

[MSpagni]

Ouch!
I know FC very well (and COMP, and DIFF...).
I forgot that all these commands are still present in win 2000 and XP and they are already ported to 32 bit native.
Thank you for reminding me.