Massive Batch processing optimizations?

[TechMage89]

I use Neat image to process video saved as an image sequence, because Neat Image is so effective with VHS noise and the like.

However, processing 95,000 frames (at 720x480 pixels) is a nightmare, because NeatImage doesn't seem to have a way to handle it. Firstly, if I queue more than about 2,000 images (even at this low resolution), weird things start happening. Either NeatImage freezes processing the first image (actually not using CPU or hard drive at all), or gives me an out-of-memory error (although its only used about half of my paging file.

The other problem I noticed was with performance: when one image is finished and the next image begins processing, the CPU useage drops for a few seconds (a big issue when processing so many images). My guess is that it is programmed so the second image isn't requested from memory until the first is done.

These problems, I think, could be fixed by limiting the number of images loaded into memory at a time (probably at 1,000) and load an additional image each time an image is deleted. The second could be fixed by begining the request from memory for the next image about two seconds before the current one is estimated to be done.

Even better would be an actual video filter with the Neat Image Noise Removal System

[andewid]

I think there is a thread about using NI's command line interface instead. You could do a real batch process and call NI for each file.

Use something like this:
NeatImage.exe input output profile preset

You could then do a for loop that did this for every image, or you can use AviSynth to render a single frame as a bitmap, call ni, then render next frame and so on. That way you don't even need to put thousands of pictures on your disk

[NITeam]

Thank you for the first hand experience, comments and suggestions. Yes, we are considering and working on improvements like those you suggested and are going to implement quite many of them. On the other hand, it really may make sense to use the method described in another thread in this forum, at least until those improvements are implemented.

Once again, thank you very much for your feedback.

Vlad

[TechMage89]

I'm sorry, I really don't know very much about batch files. Could someone explain in greater detail how to write a batch file to do this?

[DaveQ]

After several weeks of intense coding, I am finally ready to present v.1 to the world!

The reason for all this work, was the need to process very large numbers of video frames with Neat Image (119,000 +). I soon realized that I could add features to the barebones code and make the tools useful to others, who had also expressed an interest in this usage.

http://www.isochroma.com/Testfiles/NB/neatbatch.htm

Partial list of features:

> Open Source
> AVI or Image File input
> Very large batches (up to 2,147,483,646 images)
> Priority setting
> Input/Output sequence name & number remapping
> Input & Output image format conversion
> Automatic AVS generation for video output
> DNP, NFP, all switches supported
> Automated cleanup of input, output files

All done on-the-fly... with only 1 frame overhead.