dopecarpet

A toolset to generate 4 digit GeoHash tile sized Parquet/GeoJSON/CBOR files from OpenStreetMap PBF files. It comes with a custom tag filter and a minimal GET API to download the data in different formats.

Dependencies

Installation (Fedora)

sudo dnf install osmium-tool jq python3-devel pip
pip install quackosm[cli] psycopg2-binary pyosmium geoalchemy2 geojson shapely pygeohash pyarrow pandas click

Basic Use

1. Customize Filter

Edit the tags.json to specify which OSM tags you want to include in the metadata. This affects what geometries are extracted from the OSM data and has a direct influence on the file sizes.

{
    "name:en": true,
    "name": true,
    "aeroway": true,
    "boundary": ["aboriginal_lands", "border_zone", "forest", "hazard", "national_park", "protected_area", "disputed"],
    "geological": true,
    "highway": ["motorway", "trunk", "primary", "raceway"],
    "historic": true,
    "landuse": true,
    "leisure": ["dog_park", "garden", "miniature_golf", "nature_reserve", "park", "playground", "stadium", "swimming_pool", "water_park"],
    "military": ["airfield", "base", "bunker", "barracks", "danger_area", "nuclear_explosion_site", "range", "training_area"],
    "natural": true,
    "place": ["plot", "farm", "island", "islet", "sea", "ocean"],
    "power": ["plant"],
    "railway": ["abandoned", "construction", "disused", "rail"],
    "tourism": ["attraction", "yes"],
    "water": true,
    "waterway": true
}

2. Generate GeoHash Files

This extracts the desired data from the PBF file and generates four digit geohash parquet files in the geohash/ directory.

just process <osm_data.pbf>

3. Lookup Coordinate Metadata

Prints metadata found for a coordinate as JSON object list.

just lookup <latitude>, <longitude>

Sample Output: just lookup 52.496846793890256 13.435128880554208

[
  {
    "type": "polygon",
    "leisure": "park",
    "name": "Görlitzer Park"
  },
  {
    "type": "polygon",
    "landuse": "forest"
  },
  {
    "type": "polygon",
    "name": "Luisenstadt"
  },
  {
    "type": "polygon",
    "name": "Jugendverkehrsschule Wiener Straße"
  },
  {
    "type": "polygon",
    "name": "Friedrichshain-Kreuzberg"
  },
  {
    "type": "polygon",
    "name": "Kreuzberg"
  }
]

4. Web API

API on port 8000 with endpoint get_geodata that takes the parameter geojson (min 4 digits) and format (parquet|geojson|cbor) and returns a file in the requested format. Supports optional Gzip compression during transfer.

Start server:

just serve

Request example:

curl -H "Accept-Encoding: gzip" -o geodata.cbor.gz -v http://localhost:8000/get_geodata/?geohash=svc5&format=geojson

Advanced Use

Extract and Filter Data

To just extract the filtered data from an OSM PBF file and save it as parquet file

just extract <osm_data.pbf> <extracted_data.parquet>

Download OSM Data File for Region

Automatically download a file from Geofabrik based on the name of a region, filter it, and save it as parquet file.

just download '<name of the region>' <extracted_data.parquet>

The original PBF file remains saved in the files/ folder.

Split Parquet File to GeoHash Parquet Files and add Sea Polygons

just convert <extracted_data.parquet> <water_polygons.parquet> <geohash_parquet_folder>

Convert Shapefile to Parquet

Input file needs to be 'EPSG:4326'

just shp <shapefile.shp> <output.parquet>

Plot Parquet File as PNG

Plot the geometry of a parquet file as a PNG with the same filename.

just plot <file.parquet>

Sample Output:

TO DO

• GeoHash parquet file generation
• Append data to existing parquet files
• Create web API server
• Dockerize it
• Scheduled update routine