Making the Most of your GPS Data
As you’ve probably seen on my posts and statistics page, I like to keep an accurate log of where I’ve travelled. Part Excel Spreadsheet that logs all the necessary information such as End location, Food, Drink, Lodging, and Misc. costs, along with brief ride notes, the magic comes by means of recording data with my Garmin Edge 705 GPS. It’s an older device that has been superseded by newer fancier models with touch screens such as the Garmin Edge 810.
The GPS has multiple functions – 1) being that it can display various maps generated from Garmin directly, 3rd party manufacturers such as Tracks4Africa, and even free maps from OpenStreetMap, as detailed on this website nearly 4 years back. 2) It can serve as a wonderful statistics recording device, recording your movements that can later be used on online services, loaded in tools like Google Earth, and displayed on a website. Not only does it track the movements, but it also tracks metrics such as distance travelled, speed, elevation ascended/descended, calories burned, and even can track things like your heart rate, and cadence if you care to follow that sort of thing. For touring I left the heart rate and cadence monitor behind, but find great use with all of the other metrics. While the unit isn’t perfect, it works wonderfully lasting near 3 days of cycle touring recording these statistics, and can be recharged easily enough off the bike by means of a USB port, or even while riding with my Pedal Powered Electronics Charging System.
I’m going to outline some of the techniques and tools that I use to manipulate some of the data that this device records. Even if you don’t have one of the above listed models, or even brands it’s likely your unit will record statistics into some flat file that can be accessed through your computer. I’ll discuss on how to make the file smaller/web friendly while keeping the majority of the data, dealing with when a rare occurrence the files get corrupt and how to fix, and also some tricks to display more data than was intended on the device. If someone has interest in some of the more fancier techniques I use to output my data onto this website, feel free to contact me privately or discuss in the comments and I can provide snippets of my code that makes the process near automated. Those who have been following this website for a while may remember my complaining that the recording/analyzing/output process to this website took near 45 minutes each night – It’s less than 1 minute now, and I can focus on relaxing at the end of a hard day by not sitting in front of a computer. Read On..
There are a number of tools that I use to perform this analytics – most are all available for both Windows and Apple computers, along with some other tools even available for Linux, and even mobile phones running IOS or Android. First off are the Garmin branded products:
Training Center – Sadly, now discontinued with no updates since 2011, this is the statistics recording application that one can use at the end of the day to import data from the device. Available for Windows Machines and Apple OSX, it offers a clean interface with multiple panes which allow you to store your entire rides in a database detailing the ride, graph charts of your metrics, and a map with our movements overlaid. Designed only for use with the Garmin EDGE Series – it’s a fantastic piece of software that can be used offline and in the tent at night. Recently, Garmin has moved the same technology to an online only service known as Garmin Connect. It’s function is to display similar statistics as Training Center, but with a social aspect tied in, allowing rides and courses to be shared and even integrated into your own website. Since internet access is hard to come by – I’ll stick with the Training Center.
Basecamp – An update to Garmin’s well loved MapSource application, Basecamp allows you to perform trip planning, analytics of past maps, surveying terrain, and the ability to load maps onto your Garmin Device. Lately in the past year updates have been many adding more social aspects to it. Those who want to stay away from these I suggest finding a copy of the Basecamp 3.x series, as opposed to the present day 4.x versions. All work the same. Available both for Windows and Apple Operating Systems. Those who find building routes with their handheld GPS will love this program and be able to transfer their work directly to their GPS for usage. Usage of BaseCamp is out of the scope of this write-up other than a few simple features.
When you start your ride each day, you’ll want to reset your reset your GPS to 0 to provide accurate stats. If you don’t reset, sometimes the data can get mangled when importing into Training Center, and you don’t have an accurate day by day account of where you travelled (this can be fixed if you have however, a bit later on I’ll show you). For the Garmin 705 it’s as simple as holding down the bottom left most key that is titled “Lap” for a few seconds, while the display counts down giving 3 seconds of safety to make sure you really want to do this.
Once everything has been reset to zero, you would simply press the bottom button on the right “Start/Stop”. Recording now begins and you can go and do your cycle. When I stop for any longer than 15 minutes, I simply hold down the power button, which stops the recording to conserve battery power. Once powered on, don’t forget to press Start again like I do sometimes.
Note: There is an option inside the settings screen on these devices called ‘Adaptive Recording’. This handy function only records data when it detects that your unit is moving, thus saving battery life and cutting down the file size. It’s recommended to turn this on, as it’s off by default.
Importing into Training Center
After your day is complete, and you get access to a computer, the fun starts here, plugging it in and importing the data into Training Center. Plugging the cable into the Garmin unit and into your USB port you’d can now head to the top menu under File –> Receive from Device. Depending on how much data you have stored on it it may take a while to import into the software.
You’ll then be presented with data populated in many panes as shown below:
- Pane 1 – Shows your history of files imported fro the device
- Pane 2 – Is the detail of the days ride. In this example I have 2/1/2014 Highlighted
- Pane 3 – Shows Detailed graphs of elevation, speed, distance, anything you like selectable by the drop down menus.
- While not a pain – 4 is a tab allowing you to switch to a Map view, showing the route travelled, and add notes for the day.
In Pane 1 I’ve changed the view removing fields that I do not normally care about. Things like Average Speed, Heart Rate and laps seems a bit useless when touring, obviously with lots of start and stops. This is about as far as we’re going to with the interface other than Pane 1.
I generally keep the “My Activities” folder populated for as long of a time that I am in a country. Right Clicking on the ‘My Activates’ folder allows one to export the data into a single .GPX file for later usage. When I’m through with the country, I simply open up the ”Imported Activities” folder and drag each week into a new country folder for reference. I keep these files exported as well in the event that Training Center breaks on me. One can even export the entire “Imported Activities” folder to create an entire continent as opposed to exporting each separate country and joining them with a third party product. Handy stuff. Finally, You can see total details in Pane two for each of these/all folders.
Reducing Size / Optimizing the Data
We’ve figured out the exporting routine and are left with a bunch of files that can now be used. But are awfully large if they were to be shared/placed on the internet. Even if you are in a country with unmetered internet and the fastest speeds, this is a bit excessive for public display. There is a handy multi purpose tool we can use named GPSBabel which will let us manipulate our data any which way we want. Designed to be able to import GPS file format regardless of manufacturer, it allows us to export the new file into as well any format we like. Other features allow for changing the dates and times of entries inside the file, merging the data so that tracks are now viewed as opposed to date but by distance (i.e., split it every 100km), clean some of the unnecessary bits from the file, but most importantly would be the ‘Simplify’ and the ‘Radius’. Inside these GPX files are thousands of marks that are written to the file upon each change in direction, geocoordinates, speed, and so on, hence the large file size. Simplify allows one to find the most useful track points and only display 100 (or whatever value you define). Radius allows you to fudge some of the data, specifically the start and end points purely for privacy purposes. Sometimes I stay at peoples houses who I have met, and I want to keep it private where their exact location is. The Radius is helpful by changing some of these coordinates based on a set of values. GPSBabel can be downloaded here for Windows, Apple, and Linux distributions. It’s an open source product and if you have a few bucks, they’d appreciate a donation or two to keep the project alive.
GPSBabel can be used from the command line, meaning that you can type a bunch of commands into a box and have it perform your optimizations and changes for you, very useful in scripts for automation, but for beginners it’s best to use the included user interface. In this example I’m going to simplify and optimize my ‘Africa-zw.gpx’ file into something that can be more readable on the internet.
Since we’re using the universal ‘GPX’ format we’ll need to change the drop down box near Number 1 to ‘GPX XML’, and as well do the same for Number 2. As mentioned before you can mix and match your formats of files, but we’ll let you play with that if you have a device that doesn’t support GPX. Clicking the Filename in Number 1 allows you to select 1 or multiple files to act as the source file. In this case I’ve selected the Zimbabwe GPX file shown above, and in Number 2 I’ve set it to save on my Desktop with a similar filename, although I’ve padded “-optimized” to the end of it. Now we need to change some options.
Clicking the “Filters” button (I have it marked number 3) brings up a series of options. A lot of this is out of the scope of the document, read GPSBabels thorough documentation if you want to dive deep into some of them. I generally only use the Start / End Date, Packing/Merging / Splitting at specific Dates/Distances, and of course the Simplify option.
Note in the third image I have the program set to simplify to 100 track points. You can go higher or lower, lower meaning less detail will be shown, and higher with more detail but bigger file size. In the 4th image I nuke Routes and Waypoints, as sometimes this is private and personal data, that I don’t want the world to see. Once you’ve made your necessary changes click Ok and you’ll be brought back to the main GPS Babel Screen. Clicking “More Options” (I have it listed as number 4) has a nice little feature I like to have checked that brings up a map from Google Maps showing your route. Obviously if you have no internet, turn this option off. Now that’s only left to do is click ‘Apply’ in the main GPSBabel screen and let it do it’s thing. It’s fast, and unless you have a corrupt file, should give you your output file immediately and in our case, show the map. You’ll also see that it spits out some output in the “whitespace” in the program. This is the commands that can be used using the command line version of the program. For most it’s gobblety goop, but handy to for those who need it.
As you can see in the map, it still shows accurately the basics on where I travelled, minus the parts where I went into the bush and camped, stopped to take a pee, and peoples houses. Now take a peek at the file size difference – the results are huge.
Now with this optimized file I can display it in tools that can read the GPX format (Google Earth, Basecamp, Google Maps even lets you load GPX files as well if they are stored on the internet. You can also get plugins that will read the files in your blog and display fancy output similar to what you see on one of my blog posts. You are on your own to find what works for you.
Dealing with Corrupt Files
Sometimes when you import files using Training Center, they don’t appear properly in the screen. It’s a rare occasion, but sometimes the file doesn’t get saved correctly upon reset, the power off command garbles some data, or just because it feels like it. The Garmin Edge saves its files as TCX format – saving far more data than the traditional GPX format (Calories, and such), so you’ll have to work with the problematic file from the device itself. Garmin stores the files in the “Garmin/History” folder as shown below. I had a problem with one file ‘2014-01-27-06-34-12.tcx’ that wouldn’t read properly.
You can go and download that file here if you want to follow along and try the steps.
There are a few ways you can solve this:
The easiest is to download TCX Converter which has a function in there called “Repair TCX”. Most of the time this will repair it and ask you to resave the file, thus allowing import. You would select the ‘Open File’ setting, load the TCX file in question and select Fix Bad TCX under the ‘Track Modify’ tab. In this case, ours doesn’t work. So we need to find a different option.
FirstObject XML Editor is a harder approach at trying to find where the corruption is in your files. As detailed here the TCX file s a series of data wrapped inside Open and Closing tags. Programmers will make more sense of it, but everything that starts i.e. <Distance>123 must have a closing </Distance> tag. First ObjectXML editor allows you to open the corrupted file, and on the left hand tree will show a Yellow box where the corruption resides. You just need to expand it and modify the erroneous entry and you should be able to resave the file ready for reimportation. I spent about 5 minutes trying to find where the problem resided, but still had no luck. Not wanting to give up, I tried another solution.
Next, I opened up the TCX file in Microsoft Excel, knowing that it can read XML files properly, and it may be a way to find the error. I was presented with this error message on the left hand side. Excel couldn’t open it, but provided some very useful information upon clicking the details screen. What it said was on Line 2, “End Tag ‘LatitudeDuow’ does not match the start tag ‘LatitudeDegrees’. Now I knew that it was definitely corrupt, and had to do things the hard way. I opened up my notepad editor, and searched for the string of ‘LatitudeDuow’ (I like using Notepad++)
Sure enough, I found it in short order, with a whole bunch of junk trailed onto the end. I got to work holding down the delete key until I found something legible, and then made sure that the <LatitudeDegrees> tag had a closing </LatitudeDegrees> tag after the values of ‘-19.564332’. I resaved the file, tried to import it into Training Centre, and it worked! This was an exceptionally rare case of corruption, that I’ve seen once or twice in 1500 or so usages of TCX / GPX files. Hopefully this can be of assistance to you.
Using your GPS to GeoTag your Photos
Since you are riding all day long and have your GPS running, you likely are taking photos at the same time. In my case I take so many that if I went back a few days later I couldn’t remember exactly as to where they were taken. If you’ve got your GPS and Camera’s time in Sync, you can have this data merged in with your photos, and then stored inside the photos description tag, very useful for Photo Galleries like Windows Live Photo Gallery, Apple iPhoto, and even for online services like Picasa and Flickr. This is a huge feature I’d sorely miss after using it for many years. Some cameras have a GPS chip embedded inside them, but if it doesn’t you can use the free program called GeoSetter. Sadly, it’s only in Windows, and I’m not sure what a suitable replacement would be for the Apple users. Garmin Basecamp also has the same abilities, but I find its user interface clunky.
Once you load GeoSetter, and aim it a folder of images, you would then select the individual image, or highlight the whole lot of them, and select ‘Synchronize with GPS Data Files’.
An options screen will appear, where you can set various options. Since every time I use Training Centre, I export the data into a folder I simply point it at that folder and let it find any photos that match with the GPX files in question.
A screen will appear asking you to confirm which that you would like to lookup Data on the photos that were found in ‘sync’. What GeoSetter will do is connect to the internet and find out actual names for the GPS Coordinates. Countries, Cities, Districts, States, and also repair any erronous Altitude information if your GPS sometimes is recording bad values (My Garmin is known to do this every once in a while). Once complete, you would click save, and then the details would be saved inside the file. If you wanted to add more detail, selecting a file and clicking “Edit” will let you change the fields as well.
Simply close GeoSetter and it will ask you to save the modified files, and you now have this data populated in your photos, and can now use it for searching, displaying on maps, and other things. Very Cool!
Merging Multiple Maps on your Device
The Garmin Units can be a bit frustrating if you own multiple maps. For example I utilize the Garmin 2013 Southern Africa Maps, which are very detailed for cities, but not so good for rural areas, as well as using the wonderful community supported Tracks4Africa map sets, which aren’t so hot for cities, but fantastic when travelling on dirt roads. As well, I like to have an idea of what the elevation is going to be in areas. The problem is, is that Garmin only lets you load one map per storage device at one time. So, even with an extra MicroSD card in your unit that lets you only load 2 maps at once – as the data is stored in a file called “gmapsupp.img” located in the /Garmin/ folder on the GPS. I wanted to merge the best pieces of maps into a custom made map yet struggled with the BaseCamp map installation software and looked for a solution.
Luckily, there are tools that allow you to modify and muck with your installed maps and this one, GMapTool is a comprehensive GUI that lets you modify some of the options that Garmin didn’t really want you to do. Here’s how I manage to put multiple maps on one device, and even on one card.
Load Basecamp and select the Install Maps Function. Make sure your GPS is connected, as it will want to verify what unit you are using, if you are authorized, and what space is available on the device. Somewhere around on the internet is a program called “BaseCamp Patcher” which removes this restriction and lets you use an actual SD card that’s plugged into your computer. Use your searching skills to find it, as I’m not going to post it here.
Select the map you wish to install from, and highlight the region areas that you wish to copy to your device, before clicking continue. It could take 30 minutes for the maps to be loaded onto your device, so go outside and get some fresh air or brew a coffee or something.
Once complete, on your GPS in the “Garmin” folder you’ll have a brand spanking new ‘gmapsupp.img’ file. Copy it to somewhere on your Desktop, and rename it something – in this case I’m going to call mine t4a.img. Next, go back and do the same MapInstall process all over again, picking a new map provider, regions, and let it copy to your GPS. Copy the file again to your desktop, and rename it to something other than gmapsupp.img. I’ll call this next one GarminDEM.img. Then, if you want to do another one, go through the same steps. I’ll call the final map file I want to merge “garminssa.img”.
Now you should have 3 (or however many) .img files on your Desktop. It’s time to load GMapTool, and merge them together. Click the ‘Add Files’ Button, and select your 3 .img files.
Next, Move to the ‘Join’ tab, and click “Select Output File”. Here is where I am going to save on my desktop a brand new ‘gmapsupp.img’ file. In the Mapset Name, I’m going to just call it something random, like “Daves cool merged maps”. Then, you would click ‘Join All’ and you should have a gmapsupp.img file on your desktop that equals the size of the ones that you had made earlier with BaseCamp and MapInstall. All you need to copy that newly made’ gmapsupp.img’ file into the Garmin folder on your GPS, and then power it up. In the Settings menu on my Edge 705 under Maps, it now shows the 3 mapsets that I can toggle on or off, or my favorite, have them all work together at the same time. Problem solved!