[closed] How to work around corrupted cache files IDX, DAT?

[Rob]

Hello,
I'm trying to back up an Offline Maps workspace from my Android phone's San Disk SD card to an external disk. There are problems with an .IDX file and a .DAT1 file: using the Windows Explorer on Win8, these files are copied to the destination drive with a greatly reduced file size, as if only part of each file was copied. Similar story when using my phone's on-board file manager: the copy process is started, but at some point an error message appears ("Couldn't copy file. Error unknown.") and the process is aborted.

I suspect that somehow these files became corrupted, although disk-checking tools show that my SD card is not damaged. And Offline Maps does continue to work normally with these files: I did not notice any missing or scrambled areas when viewing the stored map in offline mode. Also, when adding/forcibly updating map areas, the "corrupted" files on the SD card are indeed updated by Offline Maps. When adding new map areas, the .IDX file increases in size, just as one would expect. But when I try once more to copy these updated files to the external disk, nothing changes: only partial files are copied, with the exact same reduced file sizes as before. As if the copy process for each file is cut off at a certain cluster.

Assuming that the .IDX and .DAT1 files are corrupted, how can this be fixed without losing the other non-corrupted files? The map is stored in several files .DAT ... .DAT7. If I delete the seemingly broken .DAT1 files and the .IDX file, the whole chain will probably be broken. I remember the same problem was mentioned 5 years ago:

And do not remove a cache file by yourself, it will break the entire storage (the index is built when data is added, and is not reconstructed).

As soon as the new update is available, I'll have more time to work again on the store area tool and improve it, I'm sure we'll found a suitable solution.

Any solution by now? Thanks for helping!

[Rob]

Additional information:
I did a physical check and repair of the SD card using the windows command "chkdsk /r". That found and repaired some bad sectors on the SD card. After the repair, I was able to copy the Offline Maps files in their correct full size to an external drive. Previously I had tried "chkdsk /f" which didn't help, as that command doesn't check the disk for bad sectors. I also noticed that after the repair, some areas of the map were missing. Obviously, the repair excluded the map areas stored on bad sectors, but allowed me to backup the rest.

The question remains: what can a user do if an IDX or DAT file becomes corrupted? One solution is obviously to always do a backup as soon as you have downloaded a large map area (several GB in size) and revert to that backup if needed. But what happens if you are in the middle of nowhere many days from the nearest town and an IDX file gets corrupted due to bad sectors on the disk. Does that mean that the application will not be able to access the downloaded map any more? Likewise, what happens if the map is stored in several files .DAT, .DAT1, .DAT2... .DAT7 and the .DAT1 file gets corrupted. Will that break the chain and none of the map areas will be accessible any more?

Users who do serious wilderness hiking need to understand if a few bad disk sectors will affect the display of the map as a whole, or just the map area affected by the bad sectors won't be displayed any more.

[Psyberia-Support]

Hi and sorry for the delay.

I understand your concern, this is an important topic.
Unfortunately it will never be possible to be sure that your device will work fine in the wild at all time. It could be a file/memory corruption like you got, but it can also be so much other things, like your device taking water, a shock, broken screen, etc. So it's always recommenced to have a backup plan like a paper map, a compass and some knowledge of your area.
For a recent one week trek in Nepal, I used the very light garmin inreach mini, I found it a great companion to my smartphone and really help to feel more safe (you can send/receive texts/locations using iridium satellites).
Also, I personally use the internal device memory as app data storage instead of the SD card which I feel less robust (and slower). But corruption remains rare, I use the app on almost a daily basis since ten years, and only got one problem with a cheap SD card on an older device I kept using while outdoor...

Concerning your particular problem, yes backup is always a good solution anyway, but as you point out they're not always available if you're away. But on the contrary of placemarks, you can always re-download maps.
Anyway to copy a lot of data from your SD card, do not connect your device, instead try to remove the SD card and insert it in your PC directly (using an MicroSD/SD adapter), it will be much faster and stable (but it seems it's what you did since you used the windows check disk utility).

Finally, to answer your questions:

what can a user do if an IDX or DAT file becomes corrupted?

It depends, but not much. Corruption only occurs on small parts, so you may even not notice it. I always try to improve the error handling, usually the application will discard corrupted data and try to replace it with newly downloaded one, but it's not always possible. If you're at home, the best think is to clean up the data and download fresh one.

Does that mean that the application will not be able to access the downloaded map any more?

The multiple zoom levels of a same map are not usually saved in a contiguous location, so even if you have some corruption and data not available at a zoom level, you may still be able to display the the same location on the next or previous zoom level. So unless you have a complete data corruption, you should still be able to access some part of the data.

Likewise, what happens if the map is stored in several files .DAT, .DAT1, .DAT2... .DAT7 and the .DAT1 file gets corrupted. Will that break the chain and none of the map areas will be accessible any more?

No, it should not, as long as the DAT1 file keeps the same size. If you store new data in let's say the DAT7 file, and only use this new data, the application will not access the other files at all.

[Rob]

Thank you for your answers. You are right, Inreach, paper map and compass are must-haves. We go wilderness trekking and rafting for 2 months at a time without meeting any people, and your app is most useful on such trips. Actually, the more useful it gets, the more I worry what would happen if the app stops working due to some hardware failure. But your answers are, at least partly, good news.

While in the wilderness, the landmarks are always backed up to my phone's internal memory, so that there is a chance to retrieve them if the landmark folder on the SD card is ruined by bad clusters. But the maps (.DAT files) are too big for that. What about the other files that are not as big and could be backed up to internal memory? I see files .AQX, .IDX, .PREVIEW. Are all of these crucial in order to correctly access the stored maps? Then it would make sense to back them up, too.

[Psyberia-Support]

The .AQX file is the configuration file of the map, so it's needed. This is the file downloaded when you add a new map from the community maps list.
The .IDX file is part of the stored data just like the .DAT files, so only needed (mandatory) if you have the .DAT files.
The .PREVIEW is just the preview image, not mandatory. The app will try to (re)generate it when needed.

[Rob]

Thanks a lot!