FME and the art of the API

By April 6, 2020 No Comments

Far from the typical transfer of shapefiles and spreadsheets, a new era of web resources and APIs is coming. As GIS professionals, this has typically been a skillset not needed within our domain – but with the increase in web based technologies and a need to constantly pull resources from external sources, APIs are definitely something worth exploring.

What is an API

An API can vary greatly, depending on what it’s for. But put simply, it’s a set of functions and procedures, allowing the creation of applications that access the features or data of an operating system, application, or other service. These days pretty much any data service you’re accessing will have an API in order for you to query the system for data that you need.

What format can I access?

The format of an API’s data depends largely on the system you’re accessing, but JSON is likely to be the most common format you’ll come across. For this reason, we’ll explore a few API examples which return their data in JSON. The key challenge, is in turning a large chunk of text into tabular data, separated into distinct columns and records. Once you’re comfortable with this, using FME it’s really simple to write that data into a typical GIS format like a shapefile, or a File Geodatabase.

Example: Auckland Transport Developer Portal

The Auckland Transport Developer Portal is a great place to start exploring the use of APIs. In this example, we’re going to look at the Realtime Transit Feed (GTFS) which gives users the potential to query the locations of Auckland Transport’s vehicles in realtime.

 Most developer portals have the option to try the API and return sample data in order for the user to verify it’s what they’re looking for. This is a great place to start, as users can make a sample request with correct syntax, and check out what data is returned really easily.

Here you can see with a very simple request, and using an API key unique to the user, you can return a lot of information – including spatial attributes. This response is formatted as JSON, which is returned as a single chunk of text.

Using FME to access an API

Within FME, you can replicate exactly the same request made above via the developer portal. Using the HTTP Caller transformer, you can make web requests to return data like above, and then use a series to unpack the data into the correct columns and rows.

This chain of transformers takes the nested JSON and flattens the attributes into legitimate attributes. Next it explodes the list into individual features, which you can see by the feature counts changing dramatically. At this step, one feature enters which is exploded into the 1150 features that the JSON text actually makes up. 

Lastly, we use a Vertex Creator and Coordinate System Setter to create geometry using the attributes that the API returns, and sets the coordinate system in order to display it correctly on a map. The result, is that the JSON we originally started with has been split into individual records which can be displayed on a map and queried, like any other dataset. From here, it’s simple to output the data to any format that FME supports. 

APIs are a great way to keep some of your datasets up to date, as you can build processes which are repeatable, run on a regular schedule, and require no manual processes due to the automatic transfer of data over the web. For the next dataset or data service you come across, why not try and master the art of the API! 

Article Author: Kieran O’Donnell, FME Data Solutions, Locus

More about Locus …Locus is a team of business practitioners and FME technical experts ready to help you maximise the value of your business through better data management. As a Safe Software Platinum Partner and Value Added Reseller with offices in New Zealand and Australia, we hold the unique position of being engaged solely in the sale, support and servicing of FME products; this is our competitive advantage.

More about FME…Built by Safe Software and used by over 20,000 organisations worldwide, FME is a data translation and transformation tool for solving problems of data interoperability, without the need for coding. With support for 400+ formats and applications, FME is the integration platform that converts, transforms and automates data whenever and however it’s needed.