OS Open Names - Ordnance Survey

Products and services Getting started guide

OS Open Namesv1.1

OS Names Open – Getting Started Guide | 2

Preface

Purpose and disclaimer

This ‘getting started’ guide is provided for guidance only and does not constitute any warranty, representation, undertaking, commitment or obligation (express or implied) about the product or its suitability for any particular or intended purpose. Any warranties, representations, undertakings, commitments and obligations given by Ordnance Survey about the product and/or its suitability for any particular or intended purpose are set out in your customer contract. It is your responsibility to ensure that this product is suitable for your intended purpose.

Ordnance Survey does not accept any liability (whether for breach of contract, negligence or otherwise) for any loss or liability you or any third party may suffer in relying on this guide and any guidance, suggestion, advice or explanation provided in it. Any liability that Ordnance Survey has to you in relation to the product, its supply, use, accuracy, data supplied, functionality or any other liability arising out of, or in connection with, the product is limited as set out in your customer contract.

We may change the information in this guide at any time without notice.

We do not accept responsibility for the content of any third-party websites referenced or accessed.

Data copyright and other intellectual property rights

The Crown (or, where applicable, Ordnance Survey’s suppliers) owns the intellectual property rights in Ordnance Survey digital map data.

Full details of the terms and conditions under which Ordnance Survey digital map data may be processed and/or manipulated or copied by a customer – whether or not for use on PCs or workstations or for making hard copies – are available from the Customer Service Centre, please see Further Information. You should check the terms and conditions with us before using the data. It is also the responsibility of the holder of the digital map data to ensure that any plotted or printed output contains the required copyright and database acknowledgements in a conspicuous position.

The data in AddressBase® is provided by GeoPlace®, a limited liability partnership between the Local Government Association and Ordnance Survey.

© Local Government Information House Limited copyright and database rights 2015

© Hawlfraint a hawliau cronfa ddata cyfyngedig Tŷ Gwybodaeth ar Lywodraeth Leol 2015

PAF® data are copyright material of Royal Mail® Group Limited.

Contains information provided by the Valuation Office Agency under the Open Government Licence.

Preface

How do I manage the downloaded data?

How do I manage the data that comes on a CD/DVD?

How do I use FME to manage the data?

How do I load and view the data using QGIS?

How do I use the CSV files in QGIS

How do I use the GML files in QGIS

How do I load and view the CSV data using PostGIS?

How do I load and view the data using Microsoft Access?

How do I load and view the data using Netezza?

How do I load and view the data using MapInfo?

How do I load and view the data using ESRI ArcGIS?

Further information


OS Names Open - Getting Started Guide

The contents of this Getting Started Guide are designed to assist with the management, viewing and use of the OS Open Names product.

1 Managing the data

1.1 Download

The Comma Separated Variable (.CSV) data downloaded from the OpenData web site is provided in 20x20km tiles. These tiles appear in the folder DATA with sub-folders containing the 100km grid reference, for example, su. The 20x20km tiles need to be merged to allow the required area to be loaded. Additionally, the data is provided without column headers attached. These headers will need to be added to the merged file.

The Geographic Mark-up Language (.GML) data downloaded from the OpenData web site is provided in 20x20km tiles. This data will need to be loaded into a suitable GI application to view and merge the data. It cannot be merged nor have the headers added using a simple batch file process.

1.1.1 Merging the data and adding the headers

In the case of the .CSV supply option, copy the individual .CSV files from the download folders into a new folder. Ensure that this folder contains only the .CSV files to be merged. Copy the following script into a text editor such as Notepad.

copy *.csv merged_names.csv

Save this file in Notepad with a suitable name such as ‘mergeOSnames.bat’. This creates a batch file which will merge the .CSV files and create a merged file called merged_names.csv. Double-click the batch file to run. The merged file called merged_names.csv will appear in the folder.

Remove all the original .CSV files from this folder, leaving just the newly created file. Copy the file called ‘OS_Open_Names_Header.csv from the DOC folder which downloaded with the data into this folder which contains the merged file. Copy the following script into a text editor such as Notepad.

copy OS_Open_Names_Header.csv+ merged_names.csv OS_Names.csv

Save this file in Notepad with a suitable name such as ‘Append_OSNames_Header.bat’. This creates a batch file which will take the information from the header file and add it to the top of the merged file saving it with a name ‘OS_Names.csv. Double-click the batch file to run. The file created called OS_Names.csv contains all of the necessary information and can now be loaded into a GI application or database. The file called merged_names.csv can now be deleted from the folder as it may cause confusion when loading the .CSV data.

GML files can be viewed in a text editor such as Notepad or in a web browser window. However these files cannot easily be merged into one file covering the whole area required, outside of a GI application enabled to read .GML files natively.

Preface












Further information


1.2 Media

The Comma Separated Variable (.CSV) data supplied on hard media after ordering from the OpenData web site is provided in 20x20km tiles. These tiles appear in the folder called ‘CSV’ on the hard media. Another folder called DOC contains the header file and the licence information. The 20x20km tiles need to be merged to allow the required area to be loaded. The data is provided without column headers attached. These headers will need to be added to the merged file.

The Geographic Mark-up Language (.GML) data supplied on hard media is provided in 20x20km tiles ending in .GML. These tiles appear in the folder GML on the hard media. Another folder called DOC contains the header file and the licence information. This data will need to be loaded into a suitable GI application to view and merge the data. It cannot be merged nor have the headers added using a simple batch file process.

1.2.1 Merging the data and adding the headers

For the .CSV supply option, copy the individual .CSV files from the CSV folder on the hard media into a new folder. Ensure that this folder contains only the .CSV files to be merged. Copy the following script into a text editor such as Notepad.

copy *.csv merged_names.csv

Save this file in Notepad with a name such as ‘mergeOSnames.bat’. This creates a batch file which will merge the .CSV files and create a merged file called merged_names.csv. Double-click the batch file to run. The merged file called ‘merged_names.csv’ will appear in the folder.

Remove all the original .CSV files from this folder, leaving just the newly created file. Copy the file called ‘OS_Names_Header.csv from the DOC folder which was supplied in the folder called DOC into this folder which contains the merged file. Copy the following script into a text editor such as Notepad.

copy OS_Open_Names_Header.csv+ merged_names.csv OS_Names.csv

Save this file in Notepad with a name such as ‘Append_OSNames_Header.bat’. This creates a batch file which will take the information from the header file and add it to the top of the merged file saving it with a name ‘OS_Names.csv’. Double-click the batch file to run. The file created called OS_Names.csv contains all of the necessary information and can now be loaded into a GI application or database. The file called merged_names.csv can now be deleted from the folder as it may cause confusion when loading the .CSV data.

The GML files supplied on hard media can be viewed in a text editor such as Notepad or in a web browser window. However, these files cannot easily be merged into one file covering the whole area required, outside of a GI application enabled to read .GML files natively.

1.3 Using FME

The OS_Names data, whether supplied via the download option from the OS OpenData web site or on hard media can be converted into various GI formats using the FME application by Safe Software. This applies to both the .CSV and .GML variants of the data. The .GML variant will require a new schema for the data to be released before the translation will work. It will be up to the user as to how the translation in FME is configured, depending upon requirements.

Preface












Further information


2 Loading OS Open Names

2.1 QGIS

The procedures described below will work for both data obtained via the download service or on hard media. It is assumed that for the .CSV variant the procedures described in points 1.1.1 and 1.2.1 above have been followed. The .GML data is treated slightly differently.

2.1.1 CSV

Open QGIS. In the tools options on the left hand side select the ‘open delimited text file’ option.

In the window, browse to the location where the merged .CSV file resides.

Figure 1 Importing text files into QGIS

N.B. OS_Gazetteer.csv in figure 1 should be renamed to OS_Names.csv

Ensure that the geometry fields are correctly defined and the ‘first record has field names’ box is ticked. Providing that the default co-ordinate reference system in QGIS has been set to British National Grid (EPSG 27700), the .CSV data should display correctly.

Preface












Further information


Figure 2 Visually check that the data has loaded correctly

To improve performance, create a spatial index. Right click on the OS Names table in the left hand pane and select ‘properties’.

Figure 3 Creating a spatial index in the Layer Properties window

Click on the ‘create spatial index’ then click ‘OK’. The data will then contain a spatial index which will provide much improved rendering performance.

Preface












Further information


2.1.2 GML

Open QGIS. In the tools options on the left hand side select the ‘add vector layer’ option.

In the window, browse to where the .GML files are located. Ensure that the .GML option is selected in the box at the bottom.

Figure 4 Selecting GML files

Select the file(s) to be loaded. Click ‘open’ twice – in the two open windows.

The data will load into QGIS.

Preface












Further information


Figure 5 OS Open Names data loaded into QGIS

Depending upon how many .GML files are opened, a number of tables will appear in the layers window. Each tile will be assigned a different colour style which can be subsequently changed. The performance of the rendering of the .GML data cannot be improved by adding a spatial index to the layers as with the .CSV options. Each table will need to be saved out of QGIS into shapefile format to allow the data to be spatially indexed to improve performance.

2.2 PostGIS CSV

Using PostgreSQL, the free open-source database application, with the spatially enabled PostGIS extension added it is possible to load the OS Open Names data into a database. It will then be possible to open the data from a GI application such as QGIS (or ESRI ArcGIS with the correct plugins installed).

Open PGAdmin from the Windows start menu;

You will be presented with the PGAdmin window. A new database and schema within that application will need to be created to contain the data. Instructions for doing this are available in the material available on the PostgreSQL help pages on the web site; http://www.postgresql.org/docs/manuals/

Preface












Further information

http://www.postgresql.org/docs/manuals/


For the purposes of this guide a new database called ‘osdata’ has been created to store the data. A schema has also been created within this database called ‘osnames’ into which the data will be loaded. All of the code examples listed below follow the protocol that the schema being used is called ‘osnames’ with the table name called ‘names’.

As the names data comes in a .CSV file, there is no tool or GUI to load a CSV file into a PostgreSQL database table. As a result, these following steps for loading this type of data must be followed.

1. Create the database table

2. Load the data

3. Attribute indexes

4. Create geometries

5. Create spatial Indexes

2.2.1 Create the table

To create an empty table into which we can load the data, we can use a simple SQL command to create the table declaring the number of columns, their data types and field lengths. Open a SQL window in PGAdmin using the following button;

Use the code listed below in the SQL editor window in PGAdmin in PostgreSQL.

CREATE TABLE osnames.names

(

id character varying NOT NULL,

names_uri character varying,

name1 character varying,

name1language character varying,

name2 character varying,

name2language character varying,

type character varying,

local_type character varying,

geometry_x numeric,

geometry_y numeric,

most_detail_view_res integer,

least_detail_view_res integer,

Preface












Further information


mbr_xmin numeric,

mbr_ymin numeric,

mbr_xmax numeric,

mbr_ymax numeric,

postcode_district character varying,

postcode_district_uri character varying,

populated_place character varying,

populated_place_uri character varying,

populated_place_type character varying,

district_borough character varying,

district_borough_uri character varying,

district_borough_type character varying,

county_unitary character varying,

county_unitary_uri character varying,

county_unitary_type character varying,

region character varying,

region_uri character varying,

country character varying,

country_uri character varying,

related_spatial_object character varying,

same_as_dpedia character varying,

same_as_geonames character varying,

CONSTRAINT id_pk PRIMARY KEY (id)

)

WITH (

OIDS=FALSE

);

COMMIT;

Preface












Further information


Once the code has been copied and pasted, the user should see something similar to what’s displayed in the diagram shown below. Once run, this code will have created an empty table called ‘names’ in the schema ‘osnames’. This table will then need to be loaded with the data from the merged .CSV file created by the procedures outlined earlier in this guide.

Figure 6 Create table code in PostGIS

N.B.

The schema is called osnames and the table is called names.

gazetteer_uri is now names_uri

sames_as_dbpedia comes before same_as_geonames

2.2.2 Data Loading

Copy the code below to load the contents of the merged .csv file to be loaded into the schema called osnames in the database newgaz.

COPY osnames.names FROM 'D:/Vector/osnames/OSNames.csv' DELIMITER ',' CSV;

It is important to note that the files used in this example outlined in italics will need to be changed to whatever file names are being used by the user along with the path to the merged .CSV file being loaded. Depending on whether you have properly setup and configured your database this load will take approximately 4 minutes.

Preface












Further information


2.2.3 Add an Index

To make queries run faster we can index the column name(1). Copy the code below into the SQL window changing the schema name if necessary.

CREATE INDEX osnames_name1_idx ON osnames.names (name1);

COMMIT;

2.2.4 Creating geometry from XY fields

To map the locations of the features in OS Open Names, the fields in the table that contain the geometry must be specified.

There are two methods for creating geometry from fields within a table.

The preferred method is to

1. Add a geometry column

2. Populate it based on the XY co-ordinates

The code is;

a) Add the geometry column;

SELECT AddGeometryColumn ('osnames', 'names', 'geom', 27700, 'POINT', 2);

a) Then populate it from the XY co-ordinates;

UPDATE osnames.names SET geom = ST_GeomFromText('POINT(' || x || ' ' || y || ')', 27700 );

2.2.5 Create Spatial Index

Create a spatial index on the geometry columns in order to make queries on the data run faster and speed up the data rendering within the GI application in which we will read the data.

Copy the code below to the SQL window in PGAdmin to create the spatial index.

CREATE INDEX osnames_geom_idx ON osnames.names USING GIST (geom);

COMMIT;

The data can then be opened in QGIS and other GI applications enabled to link to a PostGIS database.

Preface












Further information


2.3 Access

Load the text file into Access.

External Data>Get External Data>Navigate to text file

Import source data into a new table in your database.

Using the Import Text Wizard; Select Delimited > Set the delimiter to | (pipe) > first row contains headers > text qualifier is “

(double quote)

Set an Index if required and check the Data Types for each column.

Add a primary key as necessary

Name the table

2.4 Netezza

2.4.1 Create table

CREATE TABLE OSNAMES(

ID CHARACTER VARYING(39),

NAMES_URI CHARACTER VARYING(100),

NAME1 CHARACTER VARYING(100),

NAME1_LANG CHARACTER VARYING(3),

NAME2 CHARACTER VARYING(100),

NAME2_LANG CHARACTER VARYING(3),

TYPE CHARACTER VARYING(50),

LOCAL_TYPE CHARACTER VARYING(50),

GEOMETRY_X DOUBLE PRECISION,

GEOMETRY_Y DOUBLE PRECISION,

MOST_DETAIL_VIEW_RES BIGINT,

LEAST_DETAIL_VIEW_RES BIGINT,

MBR_XMIN DOUBLE PRECISION,

Preface












Further information

JTerry

Cross-Out


MBR_YMIN DOUBLE PRECISION,

MBR_XMAX DOUBLE PRECISION,

MBR_YMAX DOUBLE PRECISION,

POSTCODE_DISTRICT CHARACTER VARYING(4),

POSTCODE_DISTRICT_URI CHARACTER VARYING(60),

POPULATED_PLACE CHARACTER VARYING(80),

POPULATED_PLACE_URI CHARACTER VARYING(60),

POPULATED_PLACE_TYPE CHARACTER VARYING(80),

DISTRICT_BOROUGH CHARACTER VARYING(80),

DISTRICT_BOROUGH_URI CHARACTER VARYING(80),

DISTRICT_BOROUGH_TYPE CHARACTER VARYING(80),

COUNTY_UNITARY CHARACTER VARYING(80),

COUNTY_UNITARY_URI CHARACTER VARYING(80),

COUNTY_UNITARY_TYPE CHARACTER VARYING(80),

REGION CHARACTER VARYING(30),

REGION_URI CHARACTER VARYING(60),

COUNTRY CHARACTER VARYING(30),

COUNTRY_URI CHARACTER VARYING(60),

SAME_AS_GEONAMES_URI CHARACTER VARYING(100),

SAME_AS_DBPEDIA_URI CHARACTER VARYING(100)

RELATED_SPATIAL_OBJECT CHARACTER VARYING(20),

)

DISTRIBUTE ON RANDOM;

2.4.2 Import data

Right click on your table and select Import data. Browse to your file.

Preface












Further information


Import All.csv to OSNAMES

INSERT INTO OSNAMES

SELECT * FROM

-- N.B. The following settings apply;

comma delimited, DMY, Quotes = Double, Skip 1 row

EXTERNAL 'C:\\DataStructure\\Product\\Media\\DATA\\CSV_NATIONAL\\All.csv'

USING

(

DELIMITER ','

DATESTYLE 'DMY'

QUOTEDVALUE 'DOUBLE'

Y2BASE 2000

ENCODING 'internal'

SKIPROWS 1

REMOTESOURCE 'ODBC'

ESCAPECHAR '\'

);

2.4.3 Set geometry

Add geometry column and populate from the GEOMETRY_X and GEOMETRY_Y columns.

ALTER TABLE OSNAMES

ADD COLUMN SHAPE CHARACTER VARYING (80)

;

UPDATE OSNAMES

SET SHAPE = st_point(GEOMETRY_X, GEOMETRY_Y,1234)

- Where 1234 is your geometry type (likely to be 27700 for British National Grid)

Preface












Further information


2.5 Mapinfo

Open Mapinfo > File Open > navigate to your Names data.

Set delimiter to ‘,’ (comma) > Check the Use First Line for Column Titles box > select OK.

N.B. Due to number of rows, this could take up to 15 minutes.

The data will open in a feature browser.

Creating points for the features;

Go to the Table tab and Create Points. Select your symbol, ensure Get X coordinates from Column is pointing to the GEOMETRY_X attribute and Get Y coordinate from Column is pointing to the GEOMETRY_Y attribute. Ensure you have the correct Projection (Coordinate Reference System (CRS) selected This will be British National Grid or 27700. Click OK.

N.B. this will take a few minutes to process.

2.6 ESRI ArcGIS

2.6.1 CSV Direct Load

Open ArcMap. Click on the ‘add data’ icon in the toolbar at the top of the window.

Navigate to the folder where the merged .CSV files reside, whether they have been created from downloaded data or supplied on hard media.

N.B. Any reference to ‘Gazetteer’ in the image above is now OS_Names.

Preface












Further information


Click ‘add. The table will appear in the table of contents window. The table will need to have a feature class added based upon X-Y co-ordinated before it can be displayed as a map.

Right click on the table so that the context menu is displayed. Select ‘display XY data’.

In the next window make sure that the X and Y references point to the X and Y co-ordinate columns in the OS_Names table.

Preface












Further information


N.B. OS_Gazetteer.csv is now OS_Names.csv.

Click the edit button at the bottom to set the co-ordinate reference system of the table. This should be projected co-ordinate systems>National Grids>Europe>British National Grid.

Preface












Further information


Click OK. A message comes up warning that this table cannot be queried as it does not have an object ID.

The data should now display correctly in ArcMap.

Preface












Further information


To interrogate this data, export the data into a shapefile using the method described in the dialog box displayed above.

2.6.2 Create a File geodatabase in Arc

Create a file geodatabase in Arc in Catalog

Right click on the File gdb and click Import > Table (single) which opens the Table to Table import window

Input - Browse to your features.txt. Output – set output file

Click OK

N.B. It’ll take about 10-15 minutes to import the data.

Go to File>Open to input the table into ArcMap

2.6.3 GML

The data supplied will not open as .GML in ArcMap either directly or using the quick import in data interoperability. There is no schema file yet available for this data. When that is released, ArcMap should be able to import the data.

Preface












Further information

General information General enquiries Textphonewww.os.uk/contactus +44 (0)3456 05 05 05 +44 (0)23 8005 6146This document has been screened in accordance with the requirements set out in Ordnance Survey’s Equality Scheme. If you have difficulty reading this information in its current format and would like to find out how to access it in a different format (Braille, large print, computer disk or in another language), please contact us on: +44 (0)3456 05 05 05.

GeoPlace is a public sector limited liability company partnership between the Local Government Association and Ordnance Survey. Ordnance Survey, the OS Symbol, AddressBase, MasterMap and TOID are registered trademarks and GeoPlace is a trademark of Ordnance Survey, the national mapping agency of Great Britain. PAF and Royal Mail are registered trademarks of Royal Mail Group plc. WinZip is a registered trademark of WinZip International LLC. Ordnance Survey © Crown copyright

D12783_0315

OS Open Names - Ordnance Survey

Documents

Transcript of OS Open Names - Ordnance Survey