All Posts By

Angela Worsley

How Wellington Water is using FME to deliver 12D Compatible As-Built

Wellington Water manage drinking water, stormwater and wastewater services in the Wellington region.  As-Built drawings are used to show the location and details of existing or new services and typically concentrate on underground services; stormwater, water and sewer.

The Background

Wellington Water manage drinking water, stormwater and wastewater services in the Wellington region.

With a philosophy for continuous improvement, Wellington Water has focused on the automation of data capture and transfer to their Asset Management Systems.  This centre of interest has resulted in a review not only of their Regional Standards & Specifications for Water Services but also the Regional As-Built and Drafting Specifications.

In reviewing not only documents but also the process of receiving as-builts drawings and other asset data and information, a key outcome was to examine Wellington Water’s Asset Management System (AMS), key shareholders (Councils) and internal customers to define the Asset Attributes used to Operate, Maintain, Model and Value those Assets.

The Challenge

Inconsistent data capture and manual processing proved challenging for Surveyors in the field to provide As-Built that met the needs of Wellington Water’s data schemas.

  • Wellington Water use CAD for their As-Built output and convert to GIS ESRI formats
  • As-Built data is captured by external surveyors in the field using the 12D format
  • Wellington Water can’t read 12D and the 12D application (application and format 12D xml) does not handle CAD or ESRI (Technically 12D can read CAD but due to its loose format it can’t enforce data integrity such as types out in the field – a significant limitation)
  • For Surveyors in the field to provide an As-Built which would meet the Wellington Water requirements a Schema Template needs to be created.  This template must take the form of an .XSD (an ADAC type format which is an industry standard)

“A Vision in Action!” The moment when Locus and 12D presented the solution to a problem that I’ve been thinking out. These guys put together a great outcome with a solution that will save us many hours of data entry and validation

Steve RobsonWellington Water

The Results

To facilitate the goal of automatic data capture and entry meant developing a consistent Asset Data Schema that could be used for AutoCad drawings or 12d Model geospatial files. This work has resulted in the creation of a ‘Wellington Water 12DModel XSD file’ that both Wellington Water and 12D will roll out across the Wellington region in early 2020.

Working with FME, Wellington Water together with Kieran O’Donnell of Locus, the team of data evangelists developed a FME workspace which could generate the .XSD required; a template that could be loaded into the 12D application and provide 12D with all the required information on types etc.  Surveyors were then able to select from drop-downs in the field ensuring 12D supplied data met the precise requirements of Wellington Water.  The current .XSD consists of 3 waters-schema with the opportunity to scale-up as required.

The Future

Wellington Water continues to work with Locus and FME to refine the automatic data loading routines.

With plans to convert the 12D data supplied to an ESRI format and additionally once the As-Built has been pressed it may at times be necessary to send this data back as 12D; > a 12D to ESRI – > 12D.   The latter will utilise the 12D Reader/Writer FME transformer scheduled for the upcoming FME release (2020).

More about Locus …Locus is a team of business practitioners and FME technical experts ready to help you maximise the value of your business through better data management. As a Safe Software Platinum Partner and Value Added Reseller with offices in New Zealand and Australia, we hold the unique position of being engaged solely in the sale, support and servicing of FME products; this is our competitive advantage.

More about FME…Built by Safe Software and used by over 20,000 organisations worldwide, FME is a data translation and transformation tool for solving problems of data interoperability, without the need for coding. With support for 400+ formats and applications, FME is the integration platform that converts, transforms and automates data whenever and however it’s needed.

Fully automate your Data Integration with Web Feature Services

Web Feature Services (WFS) are a great way of accessing GIS resources over the web without the need to download static files. The WFS Standard provides an interface for allowing requests for geographical features across the web, using platform-independent calls. As opposed to Web Map Services (WMS), Web Feature Services gives you access to the actual data which drive maps, and not just display it.

FME and the WFS Reader
FME has a WFS reader enabling a connection to Web Feature Services. You can input the URL, use authentication (if required), and configure the reader to use the WFS for whatever way you need it.

Performance
One of the main issues you can face when using a WFS is accessing large datasets. Try some of these tips to limit the features you pull from the server, to ensure a stable connection that you can rely on.

Geographic Filter
If you’re only trying to access features within a certain geographic extent, you can apply a bounding box on the reader, to only return features within your extent. This works particularly well if you’re a local government with a constant area of interest, or pulling features from an international service but are only interested in data for New Zealand.

Attribute Filter Expressions
Alternatively if you’re interested in a specific set of data which you can define using an attribute, you can enter this in the WFS parameters. This is particularly useful if you’re looking for data within a certain date range.

Multipart Transfer
Sometimes it’s not possible to  filter data using an attribute, or if you’re trying to access a table there’s no geographic extent to limit your features by.  When accessing very large datasets, this can cause issues with connection timeouts and lead to an unstable process. Using the WFS Reader’s constraints, you can make multiple requests to the WFS in order to pull smaller and more manageable chunks of data which make up a complete dataset. By setting the Count field to a smaller number, rather than asking for the entire dataset in a single request, FME batches up the requests based on your count size.

Using these constraints, a request is sent to the server for only features 1-5000. After this request is made, another will be made for features 5001-10,000 – and so on until the entire dataset is received. By using this method, smaller requests to the service are much more reliable, resulting in a more stable process with greatly reduced risk of timeout errors.

Did you know?… The LINZ Data Service has support for WFS on it’s datasets. This can save you plenty of time by taking away the need to download datasets manually, and allows for fully automated updates using FME as your ETL tool.

On the LINZ data service, check out the Services tab on any of the datasets you need to download on a regular basis. Using some of the tips and tricks above, you can set up FME workspaces to run on a regular schedule, pulling only the data that you and your organization need, when you need it.  Read more about how to use the LINZ WFS.

Read more on the FME Web Feature Service reader

Article Author:
Kieran O’Donnell
FME Data Solutions
Locus

More about Locus …Locus is a team of business practitioners and FME technical experts ready to help you maximise the value of your business through better data management. As a Safe Software Platinum Partner and Value Added Reseller with offices in New Zealand and Australia, we hold the unique position of being engaged solely in the sale, support and servicing of FME products; this is our competitive advantage.

More about FME…Built by Safe Software and used by over 20,000 organisations worldwide, FME is a data translation and transformation tool for solving problems of data interoperability, without the need for coding. With support for 400+ formats and applications, FME is the integration platform that converts, transforms and automates data whenever and however it’s needed.

Ready to upgrade to FME 2019.1

With the release of 2019.1 on both FME Desktop and Server some impressive new enhancements deliver visual and performance based gains.

What’s new in 2019?

FME Server Automations: Released in 2019.0, Automations are a revolutionary new tool in FME Server where you can use a graphical builder to create event based workflows. You can build workflows by defining triggers and actions, such as receiving an email, watching a directory or using a schedule. Automations are revolutionary because you can build workflows that leverage all the power of FME Server, in an easy to use drag and drop interface.

 

Visual Preview: This feature allows you to preview your data from within a dockable pane inside FME Workbench, without the need to open a data inspector window. Our users find this particularly helpful when working with tricky logic and minor tweaks  to their workspace, and definitely makes building and debugging your workspaces more efficient.

Performance Enhancements: With all FME releases there are some enhancements to performance, but this version in particular brought some really great improvements. Shapefiles have had a redevelopment to become incredibly fast and efficient, which is great news for the most popularly used format. The Dissolver transformer has also been seriously enhanced, running as much as 75x faster than before.

New Formats: As always the number of formats supported in FME has increased, with this year the number reaching more than 450. Of note is a new native Revit reader, where you can read your Revit files directly without needing to use RVZ intermediate files. Some users will also be interested that there is now a 12d Reader released in beta, with expected support in FME 2020.

More about Locus …Locus is a team of business practitioners and FME technical experts ready to help you maximise the value of your business through better data management. As a Safe Software Platinum Partner and Value Added Reseller with offices in New Zealand and Australia, we hold the unique position of being engaged solely in the sale, support and servicing of FME products; this is our competitive advantage.

More about FME…Built by Safe Software and used by over 20,000 organisations worldwide, FME is a data translation and transformation tool for solving problems of data interoperability, without the need for coding. With support for 400+ formats and applications, FME is the integration platform that converts, transforms and automates data whenever and however it’s needed.

Allow external access to FME Server

We recently had a support request asking “How would I go about getting our FME Server to be externally facing – ie to allow “client name” to be able to post a notification to a topic I have set up? “.

Gary Nicholson, Locus’s FME Server Certified Professional answers:

When creating demos or prototypes using FME Server I quite often have to open up access to a development FME Server to be able to show the demo. Below are some of the ways I have used to achieve this and some ways that could be used in a production environment.

I have the flexibility to implement whatever method I choose but you are likely to be restricted by corporate IT policies so your first step in opening up access to FME Server should be to talk to your IT department. Here are some suggestions to take to them:

  • Open up port 80 through your firewall so external browsers can access FME Server directly.   If by some miracle this is allowed by IT then you should at least install an SSL certificate and use port 443.
  • Use a reverse proxy that sits in your DMZ to take the external requests and forward them to FME Server. This proxy can be locked down to only allow certain authorized requests through to FME Server.
  • Have a web server in the DMZ that handles the request and uses the FME Server API to trigger workspaces.
  • I have installed NGROK.com on the FME Server machine and used it to forward requests from a public URL and it works quite well. Please ask your IT department before installing this!
  • For a limited number of external users then a VPN could work.  I use Hamachi from LogMeIn to network all my machines together and that also works well.
  • Set up a separate FME Server either in the DMZ or on a cloud VM.  This would depend if the workspace needed to access internal data.
  • Investigate if FME Cloud would be an appropriate solution.
  • Rather than giving access direct to FME Server look at some of the other ways that Automations can fire a workspace.  Maybe the external user can send an email or put a file in a DropBox folder?

If you have any other ideas to add to this list please let us know.

FME Cloud SignupGet started with a free $250 credit.


Locus Achieves Safe Software Platinum Status

December 2018.  Locus, market leaders in the provision of certified FME training and consultancy has been awarded Platinum Value Added Reseller (VAR) status by Safe Software Inc. the developers of FME.

Founded in 2008 by John Arnerich, Locus is a team of business practitioners and FME technical experts engaged in the sale, support and servicing of FME products including FME Desktop, FME Server and FME Cloud.

With its breadth of FME knowledge, technical understanding and expertise in New Zealand and Australia, Locus has assisted over 200 organisations transform the way they manage their data. The powerful data integration capabilities, flexible and scaleable licencing editions and models make FME a natural choice for many organisations, from the small operator to enterprise level organisation.

“We are pleased to award Locus with Platinum Partner, a highly regarded position within the FME partner community,” says Sonia Hobbs, VP, Business Development and Strategy, Safe Software. “Their business acumen, technical expertise, and customer-centric approach make Locus an exemplary partner and we look forward to seeing their success and growth in the future.”

“We couldn’t be more thrilled to be awarded Platinum status in Safe Software’s partner program, commented John Arnerich, Group Director, Locus. Our relationship with Safe spans more than 10 years and this accolade further deepens our commitment to Safe and the FME suite of products. Our customers rely on the extensive capabilities of this flexible software in solving a variety of both every day and often more complex data interoperability challenges. We believe in the power of FME in improving organisational productivity and ultimately saving you time, and money,” added John.

Visit Locus at www.locus.co.nz and www.locusglobal.com.au to learn more about the FME product suite, FME certified training courses and software support.

More about Locus …Locus is a team of business practitioners and FME technical experts ready to help you maximise the value of your business through better data management. As a Safe Software Platinum Partner and Value Added Reseller with offices in New Zealand and Australia, we hold the unique position of being engaged solely in the sale, support and servicing of FME products; this is our competitive advantage.

More about FME…Built by Safe Software and used by over 20,000 organisations worldwide, FME is a data translation and transformation tool for solving problems of data interoperability, without the need for coding. With support for 400+ formats and applications, FME is the integration platform that converts, transforms and automates data whenever and however it’s needed.