All Posts By

Angela Worsley

Our FME 2020.0 Top Picks!

The latest version of FME is here, with some significant improvements to the 2020 release giving you greater performance, flexibility and accessibility to the power of FME.

We asked our Locus FME Certified technical team to take a closer look at some of the 2020.0 enhancements.

FME Server

Access to FME Server has never been easier, with new options for authentication and the ability to run jobs from a mobile device. This gives you the power to check on or run jobs remotely, monitor logs, or even build a data capture service via mobile. Download the FME Data Express mobile application now to get started.

FME Server Applications can now use authentication, giving you greater flexibility with security whilst still opening up the power of FME Server to a wider audience. This can be controlled on the application level, letting you choose which apps are shared with different individuals or even roles or groups.

A key enhancement to FME Server apps is also the ability for users to input geometry with a map picker as a published parameter. This enhances the user experience of running server apps, and opens your workspaces to an even wider audience by allowing non-technical staff to select areas of interest, straight out of the box.

 

FME Desktop

As always, FME Desktop has undergone significant performance enhancements. When upgrading your version of FME Desktop, always check to look for upgradeable transformers which will improve the efficiency of your workspace. The Statistics Calculator is roughly 100x faster than it’s last version, so it’s definitely worth upgrading sooner rather than later.

Two of the most used formats; Shapefile and Excel reader/writers are now lightning fast. These popular formats use new underlying data structures, so your workflows that connect to these formats will run more efficiently than ever.  After upgrading, when you open your workspaces that use these formats, make sure you update the reader/writers to the newer version.

You also now have the ability to apply custom connection line styles in your FME Workspaces to enhance the user experience when building up your workflows.

Like any release of FME there has been a series of new formats added for both Reading and Writing. Connect to Big Data systems, spatial data and 3D models, XML and JSON formats, and more.

New readers, writers, connectors, and transformers in FME 2020.0 include:

  • Azure and Google cognitive services for natural language processing and computer vision
  • CityJSON (Tech Preview)
  • Entwine Point Tile (EPT)
  • Google BigQuery
  • Mapbox Vector Tiles (MVT)
  • Microsoft DirectDraw Surface (DDS)
  • NIfTI (for MRI data)
  • OGC S-121 GML
  • PROJ transformers (leverage more coordinate systems by connecting to the PROJ reprojection library)
  • Revit (including floor plans and easy wall simplification)
  • Snowflake
  • UK MasterMap variants: Water, Highways, Sites, Topography
  • XSD-driven XML

Article Author: Kieran O’Donnell, FME Data Solutions, Locus

More about Locus …Locus is a team of business practitioners and FME technical experts ready to help you maximise the value of your business through better data management. As a Safe Software Platinum Partner and Value Added Reseller with offices in New Zealand and Australia, we hold the unique position of being engaged solely in the sale, support and servicing of FME products; this is our competitive advantage.

More about FME…Built by Safe Software and used by over 20,000 organisations worldwide, FME is a data translation and transformation tool for solving problems of data interoperability, without the need for coding. With support for 400+ formats and applications, FME is the integration platform that converts, transforms and automates data whenever and however it’s needed.

Integrating FME with ArcGIS Online

ArcGIS Online is a great tool for hosting your data online, and building web maps or apps which you can use to share data with your users. Using FME, you can make sure you’re getting the most from your ArcGIS Online subscription by building some really great workflows to keep your data up to date, pull data down into your organisation, or even administer your ArcGIS Online account. 

ArcGIS Online Feature Service Writer 

The ArcGIS Online Feature Writer is one of many ways you can use FME to integrate with ArcGIS Online. Using this writer, you can create, update or remove data which is in your ArcGIS Online organization. For many organisations publishing datasets to ArcGIS Online creates a bit of a data management headache, as Hosted Feature Services are duplicates of data which is maintained internally.  

Using FME however, you can create a simple process to read from your internally maintained dataset, and write directly to the feature service. By running this process on a regular basis, the data hosted in ArcGIS Online will stay up to date and you can have confidence in all your data without a burdensome process to maintain. 

ArcGIS Online Feature Service Reader 

Using the ArcGIS Online Reader, you can also access any Hosted Feature Service you have published. By adding this into an FME Workspace, you can read the data like any other format. This is particularly useful if you have projects utilising ESRI’s Collector application, and FME even allows you to download the attachments stored against each feature. 

Just like using the Feature Service Writer, you can keep two datasets in-sync without needing to constantly process the data yourself. In this case, the master dataset is held in the ArcGIS Online Feature Service, and an internal dataset is the replica which you use to display in your in-house GIS systems. 

ArcGIS REST API 

Another way to integrate FME with ArcGIS Online is using the REST API. This is slightly more advanced, so users who are unfamiliar with accessing APIs in FME should check out our FME and the art of the API blog article. We’ve also put together a sample workspace you can use to get started. Some instructions are included in the workspace to set this up to work with your own ArcGIS Online account. 

The ArcGIS REST API has a wealth of functions you can use to perform geospatial, mapping and administrative functions using your ArcGIS Enterprise organisation or ArcGIS Server. Using FME, you can make HTTP Calls to the ArcGIS REST API to perform these functions in an automated fashion. 

In the sample workspace, the process takes you through a fairly simple workflow to perform a query on your ArcGIS Online organisational account. Like most APIs you need to authenticate with ArcGIS Online first to fetch a token, in order to validate you have permissions to perform this function. 

Next, the workspace performs another request to the REST API, querying the portal with the user parameter you have set. This could be for content owned by a specific user, containing a certain tag, or a combination of criteria – the potential is endless. After making the request, the JSON is unpacked and split into individual records. In this case, the output is written to a spreadsheet of items which do or don’t contain tags, but again you can perform any function you wish. Querying the organisation is a great way to keep on top of data or items hosted in ArcGSI Online, in order to ensure duplicates aren’t kept and metadata standards are adhered to over time. 

The functions aren’t just limited to querying the portal though. You can use FME to publish services, create new items, or migrate content from one user account to another. If you’re willing to spend a little time developing these processes, you can get some great functionality which will save you plenty of time and effort in administering your organisational account, and ensure it’s always kept up to date. More information on the ArcGIS REST API. 

Article Author: Kieran O’Donnell, FME Data Solutions, Locus

More about Locus …Locus is a team of business practitioners and FME technical experts ready to help you maximise the value of your business through better data management. As a Safe Software Platinum Partner and Value Added Reseller with offices in New Zealand and Australia, we hold the unique position of being engaged solely in the sale, support and servicing of FME products; this is our competitive advantage.

More about FME…Built by Safe Software and used by over 20,000 organisations worldwide, FME is a data translation and transformation tool for solving problems of data interoperability, without the need for coding. With support for 400+ formats and applications, FME is the integration platform that converts, transforms and automates data whenever and however it’s needed.

FME and the art of the API

Far from the typical transfer of shapefiles and spreadsheets, a new era of web resources and APIs is coming. As GIS professionals, this has typically been a skillset not needed within our domain – but with the increase in web based technologies and a need to constantly pull resources from external sources, APIs are definitely something worth exploring.

What is an API

An API can vary greatly, depending on what it’s for. But put simply, it’s a set of functions and procedures, allowing the creation of applications that access the features or data of an operating system, application, or other service. These days pretty much any data service you’re accessing will have an API in order for you to query the system for data that you need.

What format can I access?

The format of an API’s data depends largely on the system you’re accessing, but JSON is likely to be the most common format you’ll come across. For this reason, we’ll explore a few API examples which return their data in JSON. The key challenge, is in turning a large chunk of text into tabular data, separated into distinct columns and records. Once you’re comfortable with this, using FME it’s really simple to write that data into a typical GIS format like a shapefile, or a File Geodatabase.

Example: Auckland Transport Developer Portal

The Auckland Transport Developer Portal is a great place to start exploring the use of APIs. In this example, we’re going to look at the Realtime Transit Feed (GTFS) which gives users the potential to query the locations of Auckland Transport’s vehicles in realtime.

 Most developer portals have the option to try the API and return sample data in order for the user to verify it’s what they’re looking for. This is a great place to start, as users can make a sample request with correct syntax, and check out what data is returned really easily.

Here you can see with a very simple request, and using an API key unique to the user, you can return a lot of information – including spatial attributes. This response is formatted as JSON, which is returned as a single chunk of text.

Using FME to access an API

Within FME, you can replicate exactly the same request made above via the developer portal. Using the HTTP Caller transformer, you can make web requests to return data like above, and then use a series to unpack the data into the correct columns and rows.

This chain of transformers takes the nested JSON and flattens the attributes into legitimate attributes. Next it explodes the list into individual features, which you can see by the feature counts changing dramatically. At this step, one feature enters which is exploded into the 1150 features that the JSON text actually makes up. 

Lastly, we use a Vertex Creator and Coordinate System Setter to create geometry using the attributes that the API returns, and sets the coordinate system in order to display it correctly on a map. The result, is that the JSON we originally started with has been split into individual records which can be displayed on a map and queried, like any other dataset. From here, it’s simple to output the data to any format that FME supports. 

APIs are a great way to keep some of your datasets up to date, as you can build processes which are repeatable, run on a regular schedule, and require no manual processes due to the automatic transfer of data over the web. For the next dataset or data service you come across, why not try and master the art of the API! 

Article Author: Kieran O’Donnell, FME Data Solutions, Locus

More about Locus …Locus is a team of business practitioners and FME technical experts ready to help you maximise the value of your business through better data management. As a Safe Software Platinum Partner and Value Added Reseller with offices in New Zealand and Australia, we hold the unique position of being engaged solely in the sale, support and servicing of FME products; this is our competitive advantage.

More about FME…Built by Safe Software and used by over 20,000 organisations worldwide, FME is a data translation and transformation tool for solving problems of data interoperability, without the need for coding. With support for 400+ formats and applications, FME is the integration platform that converts, transforms and automates data whenever and however it’s needed.

How Wellington Water is using FME to deliver 12D Compatible As-Built

Wellington Water manage drinking water, stormwater and wastewater services in the Wellington region.  As-Built drawings are used to show the location and details of existing or new services and typically concentrate on underground services; stormwater, water and sewer.

The Background

Wellington Water manage drinking water, stormwater and wastewater services in the Wellington region.

With a philosophy for continuous improvement, Wellington Water has focused on the automation of data capture and transfer to their Asset Management Systems.  This centre of interest has resulted in a review not only of their Regional Standards & Specifications for Water Services but also the Regional As-Built and Drafting Specifications.

In reviewing not only documents but also the process of receiving as-builts drawings and other asset data and information, a key outcome was to examine Wellington Water’s Asset Management System (AMS), key shareholders (Councils) and internal customers to define the Asset Attributes used to Operate, Maintain, Model and Value those Assets.

The Challenge

Inconsistent data capture and manual processing proved challenging for Surveyors in the field to provide As-Built that met the needs of Wellington Water’s data schemas.

  • Wellington Water use CAD for their As-Built output and convert to GIS ESRI formats
  • As-Built data is captured by external surveyors in the field using the 12D format
  • Wellington Water can’t read 12D and the 12D application (application and format 12D xml) does not handle CAD or ESRI (Technically 12D can read CAD but due to its loose format it can’t enforce data integrity such as types out in the field – a significant limitation)
  • For Surveyors in the field to provide an As-Built which would meet the Wellington Water requirements a Schema Template needs to be created.  This template must take the form of an .XSD (an ADAC type format which is an industry standard)

“A Vision in Action!” The moment when Locus and 12D presented the solution to a problem that I’ve been thinking out. These guys put together a great outcome with a solution that will save us many hours of data entry and validation

Steve RobsonWellington Water

The Results

To facilitate the goal of automatic data capture and entry meant developing a consistent Asset Data Schema that could be used for AutoCad drawings or 12d Model geospatial files. This work has resulted in the creation of a ‘Wellington Water 12DModel XSD file’ that both Wellington Water and 12D will roll out across the Wellington region in early 2020.

Working with FME, Wellington Water together with Kieran O’Donnell of Locus, the team of data evangelists developed a FME workspace which could generate the .XSD required; a template that could be loaded into the 12D application and provide 12D with all the required information on types etc.  Surveyors were then able to select from drop-downs in the field ensuring 12D supplied data met the precise requirements of Wellington Water.  The current .XSD consists of 3 waters-schema with the opportunity to scale-up as required.

The Future

Wellington Water continues to work with Locus and FME to refine the automatic data loading routines.

With plans to convert the 12D data supplied to an ESRI format and additionally once the As-Built has been pressed it may at times be necessary to send this data back as 12D; > a 12D to ESRI – > 12D.   The latter will utilise the 12D Reader/Writer FME transformer scheduled for the upcoming FME release (2020).

More about Locus …Locus is a team of business practitioners and FME technical experts ready to help you maximise the value of your business through better data management. As a Safe Software Platinum Partner and Value Added Reseller with offices in New Zealand and Australia, we hold the unique position of being engaged solely in the sale, support and servicing of FME products; this is our competitive advantage.

More about FME…Built by Safe Software and used by over 20,000 organisations worldwide, FME is a data translation and transformation tool for solving problems of data interoperability, without the need for coding. With support for 400+ formats and applications, FME is the integration platform that converts, transforms and automates data whenever and however it’s needed.

Fully automate your Data Integration with Web Feature Services

Web Feature Services (WFS) are a great way of accessing GIS resources over the web without the need to download static files. The WFS Standard provides an interface for allowing requests for geographical features across the web, using platform-independent calls. As opposed to Web Map Services (WMS), Web Feature Services gives you access to the actual data which drive maps, and not just display it.

FME and the WFS Reader
FME has a WFS reader enabling a connection to Web Feature Services. You can input the URL, use authentication (if required), and configure the reader to use the WFS for whatever way you need it.

Performance
One of the main issues you can face when using a WFS is accessing large datasets. Try some of these tips to limit the features you pull from the server, to ensure a stable connection that you can rely on.

Geographic Filter
If you’re only trying to access features within a certain geographic extent, you can apply a bounding box on the reader, to only return features within your extent. This works particularly well if you’re a local government with a constant area of interest, or pulling features from an international service but are only interested in data for New Zealand.

Attribute Filter Expressions
Alternatively if you’re interested in a specific set of data which you can define using an attribute, you can enter this in the WFS parameters. This is particularly useful if you’re looking for data within a certain date range.

Multipart Transfer
Sometimes it’s not possible to  filter data using an attribute, or if you’re trying to access a table there’s no geographic extent to limit your features by.  When accessing very large datasets, this can cause issues with connection timeouts and lead to an unstable process. Using the WFS Reader’s constraints, you can make multiple requests to the WFS in order to pull smaller and more manageable chunks of data which make up a complete dataset. By setting the Count field to a smaller number, rather than asking for the entire dataset in a single request, FME batches up the requests based on your count size.

Using these constraints, a request is sent to the server for only features 1-5000. After this request is made, another will be made for features 5001-10,000 – and so on until the entire dataset is received. By using this method, smaller requests to the service are much more reliable, resulting in a more stable process with greatly reduced risk of timeout errors.

Did you know?… The LINZ Data Service has support for WFS on it’s datasets. This can save you plenty of time by taking away the need to download datasets manually, and allows for fully automated updates using FME as your ETL tool.

On the LINZ data service, check out the Services tab on any of the datasets you need to download on a regular basis. Using some of the tips and tricks above, you can set up FME workspaces to run on a regular schedule, pulling only the data that you and your organization need, when you need it.  Read more about how to use the LINZ WFS.

Read more on the FME Web Feature Service reader

Article Author:
Kieran O’Donnell
FME Data Solutions
Locus

More about Locus …Locus is a team of business practitioners and FME technical experts ready to help you maximise the value of your business through better data management. As a Safe Software Platinum Partner and Value Added Reseller with offices in New Zealand and Australia, we hold the unique position of being engaged solely in the sale, support and servicing of FME products; this is our competitive advantage.

More about FME…Built by Safe Software and used by over 20,000 organisations worldwide, FME is a data translation and transformation tool for solving problems of data interoperability, without the need for coding. With support for 400+ formats and applications, FME is the integration platform that converts, transforms and automates data whenever and however it’s needed.

Ready to upgrade to FME 2019.1

With the release of 2019.1 on both FME Desktop and Server some impressive new enhancements deliver visual and performance based gains.

What’s new in 2019?

FME Server Automations: Released in 2019.0, Automations are a revolutionary new tool in FME Server where you can use a graphical builder to create event based workflows. You can build workflows by defining triggers and actions, such as receiving an email, watching a directory or using a schedule. Automations are revolutionary because you can build workflows that leverage all the power of FME Server, in an easy to use drag and drop interface.

 

Visual Preview: This feature allows you to preview your data from within a dockable pane inside FME Workbench, without the need to open a data inspector window. Our users find this particularly helpful when working with tricky logic and minor tweaks  to their workspace, and definitely makes building and debugging your workspaces more efficient.

Performance Enhancements: With all FME releases there are some enhancements to performance, but this version in particular brought some really great improvements. Shapefiles have had a redevelopment to become incredibly fast and efficient, which is great news for the most popularly used format. The Dissolver transformer has also been seriously enhanced, running as much as 75x faster than before.

New Formats: As always the number of formats supported in FME has increased, with this year the number reaching more than 450. Of note is a new native Revit reader, where you can read your Revit files directly without needing to use RVZ intermediate files. Some users will also be interested that there is now a 12d Reader released in beta, with expected support in FME 2020.

More about Locus …Locus is a team of business practitioners and FME technical experts ready to help you maximise the value of your business through better data management. As a Safe Software Platinum Partner and Value Added Reseller with offices in New Zealand and Australia, we hold the unique position of being engaged solely in the sale, support and servicing of FME products; this is our competitive advantage.

More about FME…Built by Safe Software and used by over 20,000 organisations worldwide, FME is a data translation and transformation tool for solving problems of data interoperability, without the need for coding. With support for 400+ formats and applications, FME is the integration platform that converts, transforms and automates data whenever and however it’s needed.

Allow external access to FME Server

We recently had a support request asking “How would I go about getting our FME Server to be externally facing – ie to allow “client name” to be able to post a notification to a topic I have set up? “.

Gary Nicholson, Locus’s FME Server Certified Professional answers:

When creating demos or prototypes using FME Server I quite often have to open up access to a development FME Server to be able to show the demo. Below are some of the ways I have used to achieve this and some ways that could be used in a production environment.

I have the flexibility to implement whatever method I choose but you are likely to be restricted by corporate IT policies so your first step in opening up access to FME Server should be to talk to your IT department. Here are some suggestions to take to them:

  • Open up port 80 through your firewall so external browsers can access FME Server directly.   If by some miracle this is allowed by IT then you should at least install an SSL certificate and use port 443.
  • Use a reverse proxy that sits in your DMZ to take the external requests and forward them to FME Server. This proxy can be locked down to only allow certain authorized requests through to FME Server.
  • Have a web server in the DMZ that handles the request and uses the FME Server API to trigger workspaces.
  • I have installed NGROK.com on the FME Server machine and used it to forward requests from a public URL and it works quite well. Please ask your IT department before installing this!
  • For a limited number of external users then a VPN could work.  I use Hamachi from LogMeIn to network all my machines together and that also works well.
  • Set up a separate FME Server either in the DMZ or on a cloud VM.  This would depend if the workspace needed to access internal data.
  • Investigate if FME Cloud would be an appropriate solution.
  • Rather than giving access direct to FME Server look at some of the other ways that Automations can fire a workspace.  Maybe the external user can send an email or put a file in a DropBox folder?

If you have any other ideas to add to this list please let us know.

FME Cloud SignupGet started with a free $250 credit.


Locus Achieves Safe Software Platinum Status

December 2018.  Locus, market leaders in the provision of certified FME training and consultancy has been awarded Platinum Value Added Reseller (VAR) status by Safe Software Inc. the developers of FME.

Founded in 2008 by John Arnerich, Locus is a team of business practitioners and FME technical experts engaged in the sale, support and servicing of FME products including FME Desktop, FME Server and FME Cloud.

With its breadth of FME knowledge, technical understanding and expertise in New Zealand and Australia, Locus has assisted over 200 organisations transform the way they manage their data. The powerful data integration capabilities, flexible and scaleable licencing editions and models make FME a natural choice for many organisations, from the small operator to enterprise level organisation.

“We are pleased to award Locus with Platinum Partner, a highly regarded position within the FME partner community,” says Sonia Hobbs, VP, Business Development and Strategy, Safe Software. “Their business acumen, technical expertise, and customer-centric approach make Locus an exemplary partner and we look forward to seeing their success and growth in the future.”

“We couldn’t be more thrilled to be awarded Platinum status in Safe Software’s partner program, commented John Arnerich, Group Director, Locus. Our relationship with Safe spans more than 10 years and this accolade further deepens our commitment to Safe and the FME suite of products. Our customers rely on the extensive capabilities of this flexible software in solving a variety of both every day and often more complex data interoperability challenges. We believe in the power of FME in improving organisational productivity and ultimately saving you time, and money,” added John.

Visit Locus at www.locus.co.nz and www.locusglobal.com.au to learn more about the FME product suite, FME certified training courses and software support.

More about Locus …Locus is a team of business practitioners and FME technical experts ready to help you maximise the value of your business through better data management. As a Safe Software Platinum Partner and Value Added Reseller with offices in New Zealand and Australia, we hold the unique position of being engaged solely in the sale, support and servicing of FME products; this is our competitive advantage.

More about FME…Built by Safe Software and used by over 20,000 organisations worldwide, FME is a data translation and transformation tool for solving problems of data interoperability, without the need for coding. With support for 400+ formats and applications, FME is the integration platform that converts, transforms and automates data whenever and however it’s needed.