Very long loading of the .AUX-file

[Alex]

Hi Robert,

I am currently trying GE to create mosaics using pre-compiled AUX file. Everything seems to be fine except one thing:

After loaging the .AUX file with thousands of lines in it (say, thousands of described tiles) - GE starts to check something on each single tile, which takes it's time. I bet it checks for the resolution or something like this...Robert, in some cases the whole check takes SOOOOO long - even 3-4 days sometimes, prior I am able to start the encoding (while the encoding will take its time to access each tile, too). Sometimes even longer if we use a network storage (which is much slower comparing to HDD).

No any errors are given after the check of course, as the tiles and the AUX file itself are valid - it just takes days because of the record's number in the AUX file.

Robert, is there any option to disable such of check? I am pretty sure about all my tiles and the geometry described in AUX file, there is no any reason to have this extra check, this is really boring me and wasting my time ALOT...It would be nice just to load the AUX file and be able to start the encoding immediately. In case of errors (IF any errors will be found) the encoding process will stop anyway...

Thanks,
Alex

[rparker]

If you use the command line you can use your aux mosaic file and save time. Install the command line help for more details. Here is a sample mosaic command using an aux (*.txt) file.

mrsidgeoencoder -i mosaic.txt -tiff -mos -o mosaic.sid

[Alex]

[quote="rparker"]If you use the command line you can use your aux mosaic file and save time. Install the command line help for more details. Here is a sample mosaic command using an aux (*.txt) file.

mrsidgeoencoder -i mosaic.txt -tiff -mos -o mosaic.sid[/quote]

Hi Robert,
I am sorry, but your suggestion makes no difference:

on small mosaics:
----------
G:\bin>mrsidgeoencoder.exe -mos -input output_small.aux -o output.sid
Using local license
<***a few minutes here***>
Output file name: mos.sid
Output format: MrSID Generation 3
Estimated memory required: 103,5 MB
Encoder version: 7.0.2.2183
Encode start time: Wed Jun 30 20:30:57 2010
Encode process: 0.......100%
Encode finish time: Wed Jun 30 21:10:43 2010
Total encode time: 39 minutes, 46 seconds
Input image size: 3,0 GB (3221225472 bytes)
Output file size: 153,6 MB (161061271 bytes)
Target encode ratio: 20,00:1
Actual encode ratio: 20,00:1
----------
those few minutes right before the encoding are the question.

But on large mosaics, it is even worse:
----------
G:\bin>mrsidgeoencoder.exe -mos -input output_large.aux -o output.sid

Using local license
<***a few hours passed***>

This application has requested the Runtime to terminate it in an unusual way.
Please contact the application's support team for more information.
---------------------------

Using GUI, the large ones at least being processed. Very slow but without errors. :(

Actually I am able to stitch my mosaics to one RAW file by some scripts, after that convert RAW->TIFF, and after that just encode TIFF->SID. But...is there any simpler and faster way to encode large mosaics using GE7 app alone?

[Alex]

bump up

[rparker]

Is the auxfile just a list of filenames? Could you please share an example line, showing how you've formatted the input files?

Are all of the source tiles in the same Coordinate Reference System (or is this a multiCRS mosaic)?

Do all of the source tiles have the same resolution?

How many tiles are there?

What are the dimensions of the source tiles?

How large is the output image (GeoExpress GUI reports this)?

What is the format of the input tiles?

It is unusual that several thousand tiles in an aux file would take days to process.

[Alex]

>Is the auxfile just a list of filenames?
Filenames and X\Y position in the mosaic

>Could you please share an example line, showing how you've formatted the input files?
Sure.

Code: Select all

"\\10.1.1.4\shared\MARS\z8\0\x0\0\y0.jpg" 0 32512
"\\10.1.1.4\shared\MARS\z8\0\x0\0\y1.jpg" 0 32256
"\\10.1.1.4\shared\MARS\z8\0\x0\0\y2.jpg" 0 32000
"\\10.1.1.4\shared\MARS\z8\0\x0\0\y3.jpg" 0 31744
"\\10.1.1.4\shared\MARS\z8\0\x0\0\y4.jpg" 0 31488
"\\10.1.1.4\shared\MARS\z8\0\x0\0\y5.jpg" 0 31232
"\\10.1.1.4\shared\MARS\z8\0\x0\0\y6.jpg" 0 30976
"\\10.1.1.4\shared\MARS\z8\0\x0\0\y7.jpg" 0 30720
"\\10.1.1.4\shared\MARS\z8\0\x0\0\y8.jpg" 0 30464
"\\10.1.1.4\shared\MARS\z8\0\x0\0\y9.jpg" 0 30208


etc. It is just the part of the 1st column in the mosaic. The second column will have "256" in the X offset:

Code: Select all

"\\10.1.1.4\shared\MARS\z8\0\x1\0\y8.jpg" 256 30464
"\\10.1.1.4\shared\MARS\z8\0\x1\0\y9.jpg" 256 30208


etc.

>Are all of the source tiles in the same Coordinate Reference System
Yes.

>Do all of the source tiles have the same resolution?
Yes.

>How many tiles are there?
Small mosaic - 2...8092, tested, processed OK. AUX file was loaded within 3-4min.
Huge mosaic - 4'194'304, never been processed due to extremally slow loading of the AUX file

>What are the dimensions of the source tiles?
All are 256x256

>What is the format of the input tiles?
All are in JPG format.

>How large is the output image (GeoExpress GUI reports this)?
For a huge mosaic, I never see this report (GUI seems to be hanged for days while "checking" all of the tiles, and that is the question of this topic).

Mathematically, 4M 256x256 tiles must come to the square of 2048x2048 tiles, and to the image of 524'288 x 524'288 pixels.
Plus RGB's 32bit on each single pixel, if you need the bitmap's size in bytes.

[rparker]

Hello again,

With 4 million tiles we do expect the behavior you are seeing.

[Alex]

With 4 million tiles we do expect the behavior you are seeing.

Well, Robert, is there a way to work around this issue? It is really, REALLY disturbing to be awaiting for days before the encoding starts...and finally fails. :evil:

Or may you recommend me the application which can do easier mosaicking and save to a _single_ lossless file? Than I'll use GE to convert that single file to the lossless .sid...

Thanks,
Alex

[rparker]

That is quite a bit of data to process. I am not aware of any other applications that would handle that as quickly as you desire.

You could, perhaps, convert each file to MrSID and try a composite mosaic when each has been converted.

However, why do you want one overly large mosaic (is it of the entire planet of Mars?)?

You could keep these as individual MrSID files and serve them using Express Server to any application or over the web. Express Server would mosaic on the fly with no problem.

[rparker]

Side note: I notice you are using commas in place of decimal points. This is know to cause some problems reading world files and perhaps other issues. Please see the following KB:

h_t_t_p://w_w_w.lizardtech.com/support/kb/vi ... .php?t=102

[Alex]

That is quite a bit of data to process. I am not aware of any other applications that would handle that as quickly as you desire.

Well, Robert, we are not talking about the time while converting the data itself. We are talking about the delay BEFORE starting the convertsion of the 1st tile.

The delay caused by GE while [unnecessary in my case] checking all the tiles PRIOR to the converting.

Load AUX file ->Load OK->{!!!DELAY FOR DAYS!!!}->starting conversion to a .sid (also requires for a few days or even few weeks sometimes, but that's ok)
There is no question regarding the conversion's time itself, Robert - but how to avoid unnecessary delay PRIOR the conversion.

Firstly, the converter is accessing ALL the millions of tiles to read the tile's headers to "check" for something (caused us a delay for days). Secondly, it starts over and accessing each tile AGAIN just to convert'em.
Could GE "check" each tile on the fly while converting, or something like this? Or could we simply disable the initial "checking" - in our case, we are to guarantee the AUX file and each tile all together are valid, and nothing needs to be checked?

You could, perhaps, convert each file to MrSID and try a composite mosaic when each has been converted.

Well, how to do that in batch mode?
Say, I have tons of graphical tiles and the AUX file describing them in the mosaic.

You could keep these as individual MrSID files and serve them using Express Server to any application or over the web.

Some of our projects require standalone single file as a source. Changing them to multi-source will be a different pain.

I notice you are using commas in place of decimal points.

Nope, I am not. All dec.points are "dots" in my case. :)

PS: anyway, thanks for the great converter and the file-format itself. Yet I know only one providing such of great but still lossless compression!

[Alex]

Bump uppp