All Posts By

Angela Worsley

FME 2020.2 Delivers a Sneak Peak of FME 2021

The release of FME 2020.2 delivers some usability enhancements and additional format support. It also gives a sneak preview into FME’s 2021 release.   The Locus team reviews the highlights below.

FME Server

FME Server automations now have the ability to determine how your automations will run if the trigger fails by adding a failure outport port to each trigger. Reducing the need for you to watch your workflow as actions are taken automatically if the trigger fails.

FME Data Express and FME AR

Using the FME Data Express app to run and monitor FME Server jobs and Server apps from your mobile device, you now have the ability to search repositories using the new search bar, so you can easily find what you are looking for. Additionally, a new satellite map is now available in the location picker when selecting the geometry parameter and improvements have been made to the map scale and zoom level. Enhancements to FME AR include adding a compass so you can see your model’s orientation as you view it in the ‘real world’ and stabilization of geolocated models and better anchor handling. All making for a better experience and giving you more control.

New Application Support

With each FME release comes additional application support: With the 2020.2 release comes support for the following applications:

ArcGIS portal

Further to the release of the ArcGISOnlineConnector earlier in the year, you can now use this to access your ESRI ArcGIS Portal as a file storage service and access your stored ArcGIS content to upload, update, download, delete, or list information about your items using your ArcGIS credentials. Giving you the ability to manage your ArcGIS Portal data and automate your processes.

Shopify

Safe Software like to experiment and listen to customer feedback and ideas, which they can look to support new applications. At the request of some customers, Shopify is now in the Safe Lab (lab development stage) and can be used in FME by customers who use Shopify as their e-commerce platform. Simply by using the Shopify API, managing the backend data for your business can be easily done with the ability to automatically update your product catalogue, manage inventory and upload or update products. With FME you can easily sync between other applications, while automating your workflow.

Sneak Peak at the 2021.0 Release: Event Streaming Services

In the 2021 release FME will make it easier to integrate IoT and streaming data, creating a better user experience for event stream processing capability. This enables you to process and integrate continuous, real-time data of any volume by processing the data in your workflow before storing it. Providing a great alternative to bulk/batch workflows.

Support is now available on the FME hub for the following:

Article Author: Pip Norris, FME Data Solutions, Locus

More about Locus …Locus is a team of business practitioners and FME technical experts ready to help you maximise the value of your business through better data management. As a Safe Software Platinum Partner and Value Added Reseller with offices in New Zealand and Australia, we hold the unique position of being engaged solely in the sale, support and servicing of FME products; this is our competitive advantage.

More about FME…Built by Safe Software and used by over 20,000 organisations worldwide, FME is a data translation and transformation tool for solving problems of data interoperability, without the need for coding. With support for 400+ formats and applications, FME is the integration platform that converts, transforms and automates data whenever and however it’s needed.

Data Integration Workflows Power Utility Operations

An organization works with  many systems that each hold their own data, often in different data formats. When data comes from multiple sources, data integration is an integral part of a team’s processes.  Often the data is siloed in systems, existing independent of one another within a business. Bringing that data together means it can be used in meaningful ways and to help inform business decisions. Integration and automation also give your team time to focus on tasks that are beyond the everyday functions of the business, validating the quality of the data and delivering an up-to-date single source of truth, saving you time and resources.

Applying FME to Utilities

Utility companies often have large amounts of data from multiple sources that need to be converted, transformed and validated. FME allows these companies to accomplish a variety of automation tasks across multiple workstreams, some of these include:

  • Design and Planning
  • Company Operations
  • Asset Management
  • Service Outages and Leaks
  • Distribution Networks
  • Cross Team Communication and Collaboration
  • Information Sharing Between the Public, Shareholders and Others

There are a range of ways FME can be used to integrate data across multiple applications, let’s take a look at some common uses across the utility industry.

(1) Data Migration

Utility companies often need to migrate incoming data or data from a legacy system, converting it and integrating it with new data, as a repeatable task.

  • Task: Data migration between systems where integration from GIS to CAD or CAD to GIS needs to be carried out on a regular basis, as a repeatable migration workflow and which can be easily automated using FME
  • Solution: Building a migration workflow that connects to your systems
  • Result: Data is written out to where it needs to be

 

(2) Data Consolidation and Quality Control

Utility companies regularly collect and analyze data from different sources whether it be from out in the field or from multiple departments within the organisation to meet stakeholder requirements. Running an automated self-serve validation portal process allows your organisation to attain accurate, complete and compliant data

  • Task: Data validation and quality control
  • Problem: Data needs to be good quality and of value to an organisation
  • Solution: Implement data portal self-serve for data validation purposes. Workspaces can use FME’s validation or data quality transformers – which includes the geometry validator or attribute validator. Giving the ability to then generate reports, PDFs, Spreadsheets or HTML pages that can be automatically delivered via email
  • Result: Data is now correct, complete and compliant to industry or organisation standards and can be used throughout the business

(3) Synchronising Data and Integration of Data

Decision makers across businesses require a complete picture of all available information to reach conclusions and make recommendations. This can be achieved through the integration of multiple systems and the delivery of automated workflows. Data integration is carried out between systems and applications from different parts of a business, external and internal sources or geospatial and business software. The output of these systems including; GIS, CAD, Bentley and Microsoft are brought together to a central repository (or kept separate and synchronized) while needing to be kept up to date and coordinated.

  • Task: Synchronize data across different systems and formats e.g. GIS, CAD, Bentley, Microsoft
  • Problem: Difficult to make decisions if you don’t have all the data available to you
  • Solution: Use FME to build data and application integration workflows
  • Result: Synchronized and up to date data from across systems that is used to make informed decisions

A workflow that builds integration between data and applications can run as an automation on FME server to synchronize the data automatically.

(4) Information Delivery and Distribution

Utility companies often have self-service portals for stakeholders to access data.  This provides high quality data in the format required by the general public, customers, design and contractors or other clients and is used to inform the public about scheduled or unscheduled outages or in times when emergency management is needed.

  • Task: Data needs to be accessible and distributed to shareholders and the public
  • Problem: Critical information demands to be delivered and distributed in a range of formats suitable to the end user
  • Solution: Data integration workflow to collect data into appropriate formats and hosted on an FME Cloud portal for sharing
  • Result: User friendly data formats that are accessible at the self-service portal

Generate Report Workflow: Water Utility Company

The FME platform is used to generate reports for technicians, where information is captured about network issues. The workspace pulls together detail about the leaks and locations of the water mains. The extent and severity are also captured while information around the nearest located valve is included so they can be turned off.

A simple workflow like this can run as an automation on FME server. The automation would be set up to listen for events or triggers that initiate the workspace. This can be done in a host of ways i.e. scheduled to run at a particular time, activate when a file is added to a  Dropbox account, via a Webhook or a directory and sending the output to an email.

(5) Beyond Basic Data Types

FME allows you to easily experiment with integrating data and delivering the output to different formats/functions or run analytics.  Some of the exciting data sources that could be leveraged include:

  • Drones/UAV
  • Augmented Reality
  • Satellite Imagery
  • Game Engines
  • Cognitive Services and Machine Learning
  • Virtual Reality and Interactive Web Visualisations
  • Sensor Data

Many businesses have similar workflows that are often repeated.  FME gives you the power to automate simple tasks and perform more complex data integration modelling that improve operational efficiency and enable data-based decision making.

There are FME licensing and subscription options to suit businesses and budgets of all sizes.  Chat to Locus to get started.

This article is a synopsis of the Safe Software Webinar; How Utilities Can Improve Operations Using Data Integration Workflows (recorded June 2020), reviewed and repurposed by Pip Norris, FME Data Solutions, Locus

More about Locus …Locus is a team of business practitioners and FME technical experts ready to help you maximise the value of your business through better data management. As a Safe Software Platinum Partner and Value Added Reseller with offices in New Zealand and Australia, we hold the unique position of being engaged solely in the sale, support and servicing of FME products; this is our competitive advantage.

More about FME…Built by Safe Software and used by over 20,000 organisations worldwide, FME is a data translation and transformation tool for solving problems of data interoperability, without the need for coding. With support for 400+ formats and applications, FME is the integration platform that converts, transforms and automates data whenever and however it’s needed.

A little Bit of FME Goes a Long Way for OXFAM

Oxfam is a world-wide development organisation that mobilises the power of people against poverty.

The Background

Oxfam New Zealand (Oxfam) works in the Pacific and Southeast Asia working with communities to tackle the root causes of poverty. Oxfam helps people to access safe water and sanitation, to build a sustainable livelihood, to provide education for their children and to live free from persecution and violence.

The Oxfam Trailwalker is the primary fundraising event for Oxfam to meet its objectives with a target of raising NZD$1 million per annum. Originally set up by the legendary elite Queen’s Gurkha Signals Regiment in 1981 as a military exercise to test teamwork, endurance and determination. Participants now tackle 100km in 36 hours, or 50km in 18 hours, through the beauty of the Bay of Plenty alongside their team of 4 and raise funds to fight poverty.

The Challenge

Oxfam Trailwalker is the ultimate team challenge.  Physically and mentally demanding, 300+ teams of 4 complete a gruelling 50km or 100km walk all in the name of fundraising.  Supporters pledge their financial aid and every year the New Zealand Oxfam Trailwalker event raises up to $1mil to help bring about positive change in people’s lives.

Every Oxfam Trailwalker team must commit to raising at least $2,500 to support Oxfam New Zealand’s work in the Pacific and around the world.   A Trailwalker team’s web page is the primary vehicle for recording and broadcasting fundraising progress.  Moreover, every team page features a team map that is instrumental in recording team progress through the course, stimulating donations, and providing a much needed safety net for team progression and well-being over the 100km course.    Driven by FME and delivered by Locus, all participating teams adopt a map and its associated functionality and share this with its supporter and donators database.   Fundraising endeavours are incentivised with rewards provided by event sponsors and teams can upgrade their fundraising commitment on their team page at any time to strive for a greater fundraising target.

The Results

Utilising FME, Locus delivered 300 highly functional Oxfam Trailwalker team maps, the purpose of which is to monitor team progress over the course of the event, promote supporter engagement and fundraising participation and make provision for a very visible mechanism by which team safety and well-being can be observed – all without the requirement for complex or expensive tracking equipment – a little bit of FME can go a long way!

Locus prepared Elevation profiles, created kilometre markers for the trail, enabled team tracking via Twitter and built a race officials dashboard to ensure no team was lost.

Map functionality is delivered by Locus without the requirement for expensive or complex tracking equipment and leverages the use of core FME componentry, including built-in Twitter search and regular expression transformers, to expose latitude and longitude attributes and highlight distance markers and associated tweets.

This FME and Twitter collaboration supports 300+ teams, of 4 people per team, tweeting every kilometre delivering much needed team and supporter engagement which in turns raises the profile of the event and attracts greater $$ support.

The Future

In providing its expertise, Locus collaborates with the Oxfam Events team, Whatakate District Council and Wireless Nation.   Locus provides its services free of charge, cherishes the longstanding relationship it holds with Oxfam and looks forward to supporting this event for many more years.

More about Locus …Locus is a team of business practitioners and FME technical experts ready to help you maximise the value of your business through better data management. As a Safe Software Platinum Partner and Value Added Reseller with offices in New Zealand and Australia, we hold the unique position of being engaged solely in the sale, support and servicing of FME products; this is our competitive advantage.

More about FME…Built by Safe Software and used by over 20,000 organisations worldwide, FME is a data translation and transformation tool for solving problems of data interoperability, without the need for coding. With support for 400+ formats and applications, FME is the integration platform that converts, transforms and automates data whenever and however it’s needed.

Using the Power of FME to Manage Asset Data at Powerco

Powerco distributes electricity to 336,000 urban and rural households, businesses and major industrial and commercial sites in New Zealand’s North Island. Networks are in the Taranaki, Wanganui, Rangitikei, Manawatu and Wairarapa regions, including the urban centres of New Plymouth, Wanganui, Palmerston North and Masterton, as well as Tauranga and the surrounding rural areas and the eastern and southern Waikato, Thames and Coromandel regions.

The Background

Powerco is New Zealand’s largest electricity distributor in terms of network length (36,000km) and has the second largest number of electricity connections (336,000).  The company also has the second largest gas distribution network (5815km) and the second highest number of gas connections (107,000).

PowerCo initiated an Asset Modelling Project to implement a software solution to provide regulatory reporting of PowerCo’s electricity assets and improve inputs into asset management planning.

The Challenge

PowerCo required a 2-part solution; a geo-processing component and an ETL modelling component.  The geo-processing component needed to interact with PowerCo spatial systems and execute spatial operations to prepare data for the next component.  The ETL modelling component would have to provide flexible and auditable modelling, analytics and reporting.

The Results

Locus was engaged to create the modelling component of the Asset Modelling Tool, which was needed for drawing information from a number of sources and calculating models of asset data.

The modelling process is based on the principle of keeping source data intact whilst adding new attributes that are based on the source data and any other created attribute.

FME takes the Electricity Network data from an ESRI Geodatabase and other support datasets, in various formats, and creates a database in MS SQL Server containing both the input data and computed tables.

The process connects to and extracts data from the GIS database, runs spatial operations to form relationships, and outputs results to a staging database. Literally the beating heart of the asset management system. Our solution included all of the desired outcomes, including a feature to enable map displays and further spatial analysis, in additional to the primary data tables being updated and populated with all of the necessary fields from all of the disparate data sources.

The Future

Powerco continues to explore new opportunities for the application of FME across the business

More about Locus …Locus is a team of business practitioners and FME technical experts ready to help you maximise the value of your business through better data management. As a Safe Software Platinum Partner and Value Added Reseller with offices in New Zealand and Australia, we hold the unique position of being engaged solely in the sale, support and servicing of FME products; this is our competitive advantage.

More about FME…Built by Safe Software and used by over 20,000 organisations worldwide, FME is a data translation and transformation tool for solving problems of data interoperability, without the need for coding. With support for 400+ formats and applications, FME is the integration platform that converts, transforms and automates data whenever and however it’s needed.

Network Waitaki Overcomes Stakeholder Data Accessibility Challenges

Network Waitaki is an electricity distribution company that distributes electricity from the national grid to consumers’ properties.  Network Waitaki’s core business is the provision, operation, and management of electricity lines infrastructure.

The Background

Network Waitaki were encumbered by an older web service technology implemented using the SOAP (Simple Object Access Protocol) protocol designed for the exchange of structured information but without a current interface had resulted in data interoperability challenges.

The Challenge

Network Waitaki has a web service provided by Technology One that was intended to be used for extracting asset details to synchronise with the Geospatial Information Systems (GIS) network layers.

The web service was implemented using the SOAP protocol which is complex, relatively slow and has been superseded by newer, more lightweight protocols. There are no current interfaces for SOAP, so Network Waitaki was at a “dead end”.

The Results

As SOAP uses XML for all requests and responses, a request can be composed in XML and using the power of FME, Locus was able to return the data to a tabular format for use in a modern and accessible database.

A further complication encountered by Locus were the numerous user extensions that added extra fields that were customised for the obsolete SOAP output. Here Locus was able to create a unique FME transformer, with conditional capability, and intelligently mapped the anonymous fields back into an equivalent set.

These results enabled meant the client was able to successfully extract the legacy asset information and compare it with the information in its GIS.

Through Locus’s solution, Network Waitaki achieved quality assurance for its stakeholders, the effective and robust workflow updating its GIS with missing attributes, ensuring a future of guaranteed data access for Network Waitaki and its customers.

The Future

Network Waitaki continues to work with FME to deliver ongoing productivity gains.

More about Locus …Locus is a team of business practitioners and FME technical experts ready to help you maximise the value of your business through better data management. As a Safe Software Platinum Partner and Value Added Reseller with offices in New Zealand and Australia, we hold the unique position of being engaged solely in the sale, support and servicing of FME products; this is our competitive advantage.

More about FME…Built by Safe Software and used by over 20,000 organisations worldwide, FME is a data translation and transformation tool for solving problems of data interoperability, without the need for coding. With support for 400+ formats and applications, FME is the integration platform that converts, transforms and automates data whenever and however it’s needed.

Auckland Council Delivers a Single-Automated Solution for its Digital RFS App.

Auckland Council is the local authority that is responsible for all local government decisions and responsibilities in the Auckland region. In New Zealand, central government has created a local government structure for every region and district in New Zealand.

The Background

Auckland Council is responsible for managing thousands of assets across New Zealand’s most populated city.  From rubbish bins to leisure centres and everything in between. Each one of these assets must be maintained with the help of various contractors, often relying on reported problems from the public to trigger this work to take place. Locus was recently engaged to assist in a project that automates the end-to-end processing of these requests for service, significantly reducing administrative overhead, and improving customer service.

Locus’ role in this project was to develop a data processing model in FME that could manipulate dozens of GIS asset and contextual feature classes with over a million records, into a useable format. The transformed data allows the public and customer services agents to select assets in a web map interface and pass attributes through to SAP and other business systems for processing. This process is now automated with FME Server to ensure the Report A Problem web application for asset maintenance is always up to date with the latest assets from the enterprise system.

The Challenge

The Digital RFS (Request for Servive)  app needed a single automated ETL solution that would remove data inconsistencies, manual processing and maintain the latest assets. The data had its quirks, repositories, update schedules and owners. Geometries were stored in GIS while most attributes were stored in SAP, and refreshed every week using FME server. Only the latest information would be useful and so the consolidated data needed to be updated automatically after the weekly refresh.

“FME never ceases to amaze with how it can simplify and create order out of extremely complicated data processing operations. It’s simple enough to dive into for anyone to have a go. However, we’ve certainly appreciated having Kieran O’Donnell from Locus engaged on our projects and make full use of its capabilities”

Auckland Council

The Results

The following key project deliverables/outcomes were achieved:

  • A single solution to read, process, publish data for the Digital RFS solution
  • Reads 20 enterprise feature classes from Auckland Council’s SDE
  • ‘Cookie-Cut’ process to identify all feature intersections and assigns the classification based on the asset hierarchy
  • FME Server used to process data into a single Query Layer, Query Table, 19 reference layers
  • FME Server manages a python script to automate service publishing to keep ArcGIS Online hosted service in-sync with the data on the premises.

The Future

Auckland Council plans to refresh the data structure soon and will continue to utilise the power of FME to update the data.  There are plans to integrate more data sources and widen the scope of the solution.

More about Locus …Locus is a team of business practitioners and FME technical experts ready to help you maximise the value of your business through better data management. As a Safe Software Platinum Partner and Value Added Reseller with offices in New Zealand and Australia, we hold the unique position of being engaged solely in the sale, support and servicing of FME products; this is our competitive advantage.

More about FME…Built by Safe Software and used by over 20,000 organisations worldwide, FME is a data translation and transformation tool for solving problems of data interoperability, without the need for coding. With support for 400+ formats and applications, FME is the integration platform that converts, transforms and automates data whenever and however it’s needed.

Our FME 2020.0 Top Picks!

The latest version of FME is here, with some significant improvements to the 2020 release giving you greater performance, flexibility and accessibility to the power of FME.

We asked our Locus FME Certified technical team to take a closer look at some of the 2020.0 enhancements.

FME Server

Access to FME Server has never been easier, with new options for authentication and the ability to run jobs from a mobile device. This gives you the power to check on or run jobs remotely, monitor logs, or even build a data capture service via mobile. Download the FME Data Express mobile application now to get started.

FME Server Applications can now use authentication, giving you greater flexibility with security whilst still opening up the power of FME Server to a wider audience. This can be controlled on the application level, letting you choose which apps are shared with different individuals or even roles or groups.

A key enhancement to FME Server apps is also the ability for users to input geometry with a map picker as a published parameter. This enhances the user experience of running server apps, and opens your workspaces to an even wider audience by allowing non-technical staff to select areas of interest, straight out of the box.

 

FME Desktop

As always, FME Desktop has undergone significant performance enhancements. When upgrading your version of FME Desktop, always check to look for upgradeable transformers which will improve the efficiency of your workspace. The Statistics Calculator is roughly 100x faster than it’s last version, so it’s definitely worth upgrading sooner rather than later.

Two of the most used formats; Shapefile and Excel reader/writers are now lightning fast. These popular formats use new underlying data structures, so your workflows that connect to these formats will run more efficiently than ever.  After upgrading, when you open your workspaces that use these formats, make sure you update the reader/writers to the newer version.

You also now have the ability to apply custom connection line styles in your FME Workspaces to enhance the user experience when building up your workflows.

Like any release of FME there has been a series of new formats added for both Reading and Writing. Connect to Big Data systems, spatial data and 3D models, XML and JSON formats, and more.

New readers, writers, connectors, and transformers in FME 2020.0 include:

  • Azure and Google cognitive services for natural language processing and computer vision
  • CityJSON (Tech Preview)
  • Entwine Point Tile (EPT)
  • Google BigQuery
  • Mapbox Vector Tiles (MVT)
  • Microsoft DirectDraw Surface (DDS)
  • NIfTI (for MRI data)
  • OGC S-121 GML
  • PROJ transformers (leverage more coordinate systems by connecting to the PROJ reprojection library)
  • Revit (including floor plans and easy wall simplification)
  • Snowflake
  • UK MasterMap variants: Water, Highways, Sites, Topography
  • XSD-driven XML

Article Author: Kieran O’Donnell, FME Data Solutions, Locus

More about Locus …Locus is a team of business practitioners and FME technical experts ready to help you maximise the value of your business through better data management. As a Safe Software Platinum Partner and Value Added Reseller with offices in New Zealand and Australia, we hold the unique position of being engaged solely in the sale, support and servicing of FME products; this is our competitive advantage.

More about FME…Built by Safe Software and used by over 20,000 organisations worldwide, FME is a data translation and transformation tool for solving problems of data interoperability, without the need for coding. With support for 400+ formats and applications, FME is the integration platform that converts, transforms and automates data whenever and however it’s needed.

Integrating FME with ArcGIS Online

ArcGIS Online is a great tool for hosting your data online, and building web maps or apps which you can use to share data with your users. Using FME, you can make sure you’re getting the most from your ArcGIS Online subscription by building some really great workflows to keep your data up to date, pull data down into your organisation, or even administer your ArcGIS Online account. 

ArcGIS Online Feature Service Writer 

The ArcGIS Online Feature Writer is one of many ways you can use FME to integrate with ArcGIS Online. Using this writer, you can create, update or remove data which is in your ArcGIS Online organization. For many organisations publishing datasets to ArcGIS Online creates a bit of a data management headache, as Hosted Feature Services are duplicates of data which is maintained internally.  

Using FME however, you can create a simple process to read from your internally maintained dataset, and write directly to the feature service. By running this process on a regular basis, the data hosted in ArcGIS Online will stay up to date and you can have confidence in all your data without a burdensome process to maintain. 

ArcGIS Online Feature Service Reader 

Using the ArcGIS Online Reader, you can also access any Hosted Feature Service you have published. By adding this into an FME Workspace, you can read the data like any other format. This is particularly useful if you have projects utilising ESRI’s Collector application, and FME even allows you to download the attachments stored against each feature. 

Just like using the Feature Service Writer, you can keep two datasets in-sync without needing to constantly process the data yourself. In this case, the master dataset is held in the ArcGIS Online Feature Service, and an internal dataset is the replica which you use to display in your in-house GIS systems. 

ArcGIS REST API 

Another way to integrate FME with ArcGIS Online is using the REST API. This is slightly more advanced, so users who are unfamiliar with accessing APIs in FME should check out our FME and the art of the API blog article. We’ve also put together a sample workspace you can use to get started. Some instructions are included in the workspace to set this up to work with your own ArcGIS Online account. 

The ArcGIS REST API has a wealth of functions you can use to perform geospatial, mapping and administrative functions using your ArcGIS Enterprise organisation or ArcGIS Server. Using FME, you can make HTTP Calls to the ArcGIS REST API to perform these functions in an automated fashion. 

In the sample workspace, the process takes you through a fairly simple workflow to perform a query on your ArcGIS Online organisational account. Like most APIs you need to authenticate with ArcGIS Online first to fetch a token, in order to validate you have permissions to perform this function. 

Next, the workspace performs another request to the REST API, querying the portal with the user parameter you have set. This could be for content owned by a specific user, containing a certain tag, or a combination of criteria – the potential is endless. After making the request, the JSON is unpacked and split into individual records. In this case, the output is written to a spreadsheet of items which do or don’t contain tags, but again you can perform any function you wish. Querying the organisation is a great way to keep on top of data or items hosted in ArcGSI Online, in order to ensure duplicates aren’t kept and metadata standards are adhered to over time. 

The functions aren’t just limited to querying the portal though. You can use FME to publish services, create new items, or migrate content from one user account to another. If you’re willing to spend a little time developing these processes, you can get some great functionality which will save you plenty of time and effort in administering your organisational account, and ensure it’s always kept up to date. More information on the ArcGIS REST API. 

Article Author: Kieran O’Donnell, FME Data Solutions, Locus

More about Locus …Locus is a team of business practitioners and FME technical experts ready to help you maximise the value of your business through better data management. As a Safe Software Platinum Partner and Value Added Reseller with offices in New Zealand and Australia, we hold the unique position of being engaged solely in the sale, support and servicing of FME products; this is our competitive advantage.

More about FME…Built by Safe Software and used by over 20,000 organisations worldwide, FME is a data translation and transformation tool for solving problems of data interoperability, without the need for coding. With support for 400+ formats and applications, FME is the integration platform that converts, transforms and automates data whenever and however it’s needed.

FME and the art of the API

Far from the typical transfer of shapefiles and spreadsheets, a new era of web resources and APIs is coming. As GIS professionals, this has typically been a skillset not needed within our domain – but with the increase in web based technologies and a need to constantly pull resources from external sources, APIs are definitely something worth exploring.

What is an API

An API can vary greatly, depending on what it’s for. But put simply, it’s a set of functions and procedures, allowing the creation of applications that access the features or data of an operating system, application, or other service. These days pretty much any data service you’re accessing will have an API in order for you to query the system for data that you need.

What format can I access?

The format of an API’s data depends largely on the system you’re accessing, but JSON is likely to be the most common format you’ll come across. For this reason, we’ll explore a few API examples which return their data in JSON. The key challenge, is in turning a large chunk of text into tabular data, separated into distinct columns and records. Once you’re comfortable with this, using FME it’s really simple to write that data into a typical GIS format like a shapefile, or a File Geodatabase.

Example: Auckland Transport Developer Portal

The Auckland Transport Developer Portal is a great place to start exploring the use of APIs. In this example, we’re going to look at the Realtime Transit Feed (GTFS) which gives users the potential to query the locations of Auckland Transport’s vehicles in realtime.

 Most developer portals have the option to try the API and return sample data in order for the user to verify it’s what they’re looking for. This is a great place to start, as users can make a sample request with correct syntax, and check out what data is returned really easily.

Here you can see with a very simple request, and using an API key unique to the user, you can return a lot of information – including spatial attributes. This response is formatted as JSON, which is returned as a single chunk of text.

Using FME to access an API

Within FME, you can replicate exactly the same request made above via the developer portal. Using the HTTP Caller transformer, you can make web requests to return data like above, and then use a series to unpack the data into the correct columns and rows.

This chain of transformers takes the nested JSON and flattens the attributes into legitimate attributes. Next it explodes the list into individual features, which you can see by the feature counts changing dramatically. At this step, one feature enters which is exploded into the 1150 features that the JSON text actually makes up. 

Lastly, we use a Vertex Creator and Coordinate System Setter to create geometry using the attributes that the API returns, and sets the coordinate system in order to display it correctly on a map. The result, is that the JSON we originally started with has been split into individual records which can be displayed on a map and queried, like any other dataset. From here, it’s simple to output the data to any format that FME supports. 

APIs are a great way to keep some of your datasets up to date, as you can build processes which are repeatable, run on a regular schedule, and require no manual processes due to the automatic transfer of data over the web. For the next dataset or data service you come across, why not try and master the art of the API! 

Article Author: Kieran O’Donnell, FME Data Solutions, Locus

More about Locus …Locus is a team of business practitioners and FME technical experts ready to help you maximise the value of your business through better data management. As a Safe Software Platinum Partner and Value Added Reseller with offices in New Zealand and Australia, we hold the unique position of being engaged solely in the sale, support and servicing of FME products; this is our competitive advantage.

More about FME…Built by Safe Software and used by over 20,000 organisations worldwide, FME is a data translation and transformation tool for solving problems of data interoperability, without the need for coding. With support for 400+ formats and applications, FME is the integration platform that converts, transforms and automates data whenever and however it’s needed.

How Wellington Water is using FME to deliver 12D Compatible As-Built

Wellington Water manage drinking water, stormwater and wastewater services in the Wellington region.  As-Built drawings are used to show the location and details of existing or new services and typically concentrate on underground services; stormwater, water and sewer.

The Background

Wellington Water manage drinking water, stormwater and wastewater services in the Wellington region.

With a philosophy for continuous improvement, Wellington Water has focused on the automation of data capture and transfer to their Asset Management Systems.  This centre of interest has resulted in a review not only of their Regional Standards & Specifications for Water Services but also the Regional As-Built and Drafting Specifications.

In reviewing not only documents but also the process of receiving as-builts drawings and other asset data and information, a key outcome was to examine Wellington Water’s Asset Management System (AMS), key shareholders (Councils) and internal customers to define the Asset Attributes used to Operate, Maintain, Model and Value those Assets.

The Challenge

Inconsistent data capture and manual processing proved challenging for Surveyors in the field to provide As-Built that met the needs of Wellington Water’s data schemas.

  • Wellington Water use CAD for their As-Built output and convert to GIS ESRI formats
  • As-Built data is captured by external surveyors in the field using the 12D format
  • Wellington Water can’t read 12D and the 12D application (application and format 12D xml) does not handle CAD or ESRI (Technically 12D can read CAD but due to its loose format it can’t enforce data integrity such as types out in the field – a significant limitation)
  • For Surveyors in the field to provide an As-Built which would meet the Wellington Water requirements a Schema Template needs to be created.  This template must take the form of an .XSD (an ADAC type format which is an industry standard)

“A Vision in Action!” The moment when Locus and 12D presented the solution to a problem that I’ve been thinking out. These guys put together a great outcome with a solution that will save us many hours of data entry and validation

Steve RobsonWellington Water

The Results

To facilitate the goal of automatic data capture and entry meant developing a consistent Asset Data Schema that could be used for AutoCad drawings or 12d Model geospatial files. This work has resulted in the creation of a ‘Wellington Water 12DModel XSD file’ that both Wellington Water and 12D will roll out across the Wellington region in early 2020.

Working with FME, Wellington Water together with Kieran O’Donnell of Locus, the team of data evangelists developed a FME workspace which could generate the .XSD required; a template that could be loaded into the 12D application and provide 12D with all the required information on types etc.  Surveyors were then able to select from drop-downs in the field ensuring 12D supplied data met the precise requirements of Wellington Water.  The current .XSD consists of 3 waters-schema with the opportunity to scale-up as required.

The Future

Wellington Water continues to work with Locus and FME to refine the automatic data loading routines.

With plans to convert the 12D data supplied to an ESRI format and additionally once the As-Built has been pressed it may at times be necessary to send this data back as 12D; > a 12D to ESRI – > 12D.   The latter will utilise the 12D Reader/Writer FME transformer scheduled for the upcoming FME release (2020).

More about Locus …Locus is a team of business practitioners and FME technical experts ready to help you maximise the value of your business through better data management. As a Safe Software Platinum Partner and Value Added Reseller with offices in New Zealand and Australia, we hold the unique position of being engaged solely in the sale, support and servicing of FME products; this is our competitive advantage.

More about FME…Built by Safe Software and used by over 20,000 organisations worldwide, FME is a data translation and transformation tool for solving problems of data interoperability, without the need for coding. With support for 400+ formats and applications, FME is the integration platform that converts, transforms and automates data whenever and however it’s needed.