How To Create a Comprehensive OSGB Map Cache For The UK

Post a reply

This question is a means of preventing automated form submissions by spambots.
:D :) ;) :( :o :shock: :? 8-) :lol: :x :P :oops: :cry: :evil: :twisted: :roll: :!: :?: :idea: :arrow: :| :mrgreen: :geek: :ugeek:

BBCode is ON
[img] is ON
[flash] is OFF
[url] is ON
Smilies are ON

Topic review

If you wish to attach one or more files enter the details below.

Expand view Topic review: How To Create a Comprehensive OSGB Map Cache For The UK

Re: How To Create a Comprehensive OSGB Map Cache For The UK

by Yemble » Wed May 13, 2015 4:34 pm

AlpineQuest wrote:Thanks for the info and update.
It seems you're busy these last days!
Also it's good to know the app can manage such a large storage...
Agreed, there is no appreciable drop in performance of AQ on my Note Pro when using this very large cached map, which is testament to your great coding :P

Re: How To Create a Comprehensive OSGB Map Cache For The UK

by Psyberia-Support » Tue May 12, 2015 7:43 pm

Thanks for the info and update.
It seems you're busy these last days!
Also it's good to know the app can manage such a large storage...

Re: How To Create a Comprehensive OSGB Map Cache For The UK

by Yemble » Mon May 11, 2015 11:49 am

The OSGB cache sizes at various stages of the build are as follows:

1:50K Z12 - All UK - 1.55Gb
1:50K Z13 - All UK - 5.41Gb
1:50K Z14 - All UK - 18.1Gb
1:25K Z15 - North England - 24.8Gb
1:25K Z15 - North England + Midlands - 32.2Gb
1:25K Z15 - North England + Midlands + Scotland - 47.9Gb
1:25K Z15 - North England + Midlands + Scotland + Wales - 51.1Gb
1:25K Z15 - All UK - 56.8Gb

56.8Gb is a bit less than I had predicted previously and maybe due to a more methodical approach to the tile downloads (95% via Store Area, 5% panning around the map). This might just fit on a 64Gb card, but without much headroom for anything else.

How To Create a Comprehensive OSGB Map Cache For The UK

by Yemble » Tue May 05, 2015 12:11 pm

Detailed topo walking maps (1:50K & 1:25K) are very expensive to purchase as off-line maps for large areas and are generally tied to specific applications, such as ViewRanger and MemoryMap. Certain on-line map servers, such as BING, provide the same level of map detail, made available to a wider range of mobile mapping applications such as Alpine Quest and Back Country Navigator, but only on a tile by tile basis. This is fine if you have a WiFi signal to hand, but utterly useless when you are out in the wilderness with no phone signal.

Luckily, there is a compromise available, which involves building your own map cache from downloaded tiles at home (with a WiFi connection) and then operating this cached map in off-line mode when out trekking. The map cache database of AlpineQuest has the capability of storing and handling extremely large map areas, without any appreciable degradation in performance of the application. It achieves this by segmenting the cache database into 2Gb sub-files which are allocated as required, up to the maximum capacity of the device storage!

However, there are some caveats to building your own cached map in this manner:

1) It can take a very long time to accumulate all of the map tiles at all zoom levels for a large map area and a significant amount of operator effort is required.
2) The Store Area function in AQ is NOT bullet proof and can fail if given too much work to do.
3) As we have seen recently with the BING OSGB debacle, on-line cached maps may be subject to tile “corruption” from the on-line source. A Basic BING Maps key is essential to mitigate against this.
4) The map-cache database is very large when compared to a purchased off-line map for the same area. For example, a 128Gb flash card is required to store the OSBG community map at the default 1:50K and 1:25K zoom levels – 64Gb is NOT enough! The final size is likely to be around the 75Gb to 80Gb mark. This is around three times the size of the equivalent Memory-Map QCT map circa 2004, so storage efficiency is clearly not the primary goal of the AQ map cache.
5) Cache database corruption is likely to occur if the storage runs out during Store Area processing!
6) Taking backup of the map-cache from time to time is highly advisable, but takes a long time. If you are doing this via an external card reader on a PC, then USB3 is essential when shipping tens of gigabytes.

I am happy to report that it IS viable to build a comprehensive OSGB cached map in AQ for the entire UK at the four zoom levels specified in the OSGB community map (ie. three at 1:50K and one at 1:25K). This map will not fit on a 64Gb flash card, so prepare to invest in a 128Gb card, formatted exFat – a 45Gb/s read speed is OK. Make sure that your device supports the SDXC standard first. Samsung devices (eg. S3, S4, S5 and Notes) are fine with this.

Here is my method…

1) Start with the basic OSGB community map with no cached tiles and switch AQ to off-line mode.
2) Generate a Basic BING Maps key on a PC (with no expiry) and apply it to the AQ OSGB map config file ( ref. viewtopic.php?f=4&t=2139&start=30 ).
3) Make sure that your WiFi connection to the device is solid and as fast as it can be.
4) Use the Store Area function to process just one zoom level at a time for the entire UK, however, avoid selecting a download area that results in the slider zoom turning red. The maximum area will reduce as each zoom level is processed and the amount of operator effort increases accordingly!
5) As each zoom level has been completed, stop downloading and carefully verify that all of the tiles have been captured. AQ has a brilliant feature for this process, so turn on the Show Storage Coverage option in the Maps menu. The green squares show downloaded tiles, whereas the red squares show missing tiles. These indicators are cascaded to represent up to four zoom levels at a time. Pan around the map looking for any red squares at zoom levels that should all be indicated by green squares.
6) Re-download any missing tiles either with Store Area or by switching to on-line mode.
7) DO NOT be tempted to pan around the map in on-line mode downloading tiles whilst at the same time using Store Area to download large numbers of tiles. Although this may be OK for a while, it WILL eventually cause AQ to lock up. Then you may need to reboot your device, or even mount the flash card under Windows in order to make it writeable again… you have been warned! Also, if you are operating in on-line mode, you have a much greater chance of accidentally downloading tiles for the “wrong” zoom level.
8) Take a backup of the AQ cache to a PC once each zoom level has been completed and verified. Keep these backups separate and give them sensible names as they may be useful later on.
9) Whereas the 1:50K downloads might be regarded as a middle distance effort, the 1:25K downloads are a full marathon… so get set for the long haul. Keep yourself sane by targeting specific areas of the UK map and work diligently on these manageable chunks, including verification. Save your map as significant achievements are reached, for example, Scotland, Wales, South West, etc.
10) Avoid overcommitting the AQ Store Area function as it can result in lock ups. Also, avoid numerous consecutive runs of Store Area as it will eventually just report errors rather than downloading tiles. A periodic restart of AQ both prevents and recovers from, this issue.
11) Take your time and don’t cut corners. Remember that this is a one off exercise that should pay dividends for many years and potentially over a number of devices, so it is worthwhile putting the effort in at this point to get it right.
12) If you do spot any cache corruption whilst verifying a zoom region (such as red, grey, or duplicated, tiles) then it might be prudent to revert to your previously saved cache set from the PC and re-process the area(s) in AQ. At present, there is no (published) means to repair a corrupted AQ cache, but hopefully, this capability will be available in a future version.
13) Once your map is complete, you should keep AQ in off-line mode when viewing it. This will prevent any possibility of BING inspired corruption from reaching your precious map. Obviously, if you are aware that new OSGB tiles are available, then you can switch it on on-line mode to download them. You should always have your latest backup to revert to should the worst happen!

BTW, don’t take any notice of anyone saying that such large scale tile downloads might put the BING Maps service at risk. This is complete and utter nonsense as even 80Gb, spread over several days / weeks, is extremely small beer when compared to the terabytes of tile data that BING ship out every day.