152 lines
No EOL
3.8 KiB
Markdown
152 lines
No EOL
3.8 KiB
Markdown
<image src="https://blog.therugseller.co.uk/wp-content/uploads/2017/10/2-10.png" width="400" />
|
|
<p> <p>
|
|
|
|
# dopecarpet
|
|
|
|
A toolset to generate 4 digit GeoHash tile sized Parquet/GeoJSON/CBOR files from OpenStreetMap PBF files. It comes with a custom tag filter and a minimal GET API to download the data in different formats.
|
|
|
|
## Dependencies
|
|
|
|
Installation (Fedora)
|
|
|
|
```bash
|
|
sudo dnf install osmium-tool jq python3-devel pip
|
|
pip install quackosm[cli] psycopg2-binary pyosmium geoalchemy2 geojson shapely pygeohash pyarrow pandas click
|
|
```
|
|
|
|
## Basic Use
|
|
|
|
### 1. Customize Filter
|
|
|
|
Edit the `tags.json` to specify which OSM tags you want to include in the metadata. This affects what geometries are extracted from the OSM data and has a direct influence on the file sizes.
|
|
|
|
```json
|
|
{
|
|
"name:en": true,
|
|
"name": true,
|
|
"aeroway": true,
|
|
"boundary": ["aboriginal_lands", "border_zone", "forest", "hazard", "national_park", "protected_area", "disputed"],
|
|
"geological": true,
|
|
"highway": ["motorway", "trunk", "primary", "raceway"],
|
|
"historic": true,
|
|
"landuse": true,
|
|
"leisure": ["dog_park", "garden", "miniature_golf", "nature_reserve", "park", "playground", "stadium", "swimming_pool", "water_park"],
|
|
"military": ["airfield", "base", "bunker", "barracks", "danger_area", "nuclear_explosion_site", "range", "training_area"],
|
|
"natural": true,
|
|
"place": ["plot", "farm", "island", "islet", "sea", "ocean"],
|
|
"power": ["plant"],
|
|
"railway": ["abandoned", "construction", "disused", "rail"],
|
|
"tourism": ["attraction", "yes"],
|
|
"water": true,
|
|
"waterway": true
|
|
}
|
|
```
|
|
|
|
### 2. Generate GeoHash Files
|
|
|
|
This extracts the desired data from the PBF file and generates four digit geohash parquet files in the `geohash/` directory.
|
|
|
|
```bash
|
|
just process <osm_data.pbf>
|
|
```
|
|
|
|
### 3. Lookup Coordinate Metadata
|
|
|
|
Prints metadata found for a coordinate as JSON object list.
|
|
|
|
```bash
|
|
just lookup <latitude>, <longitude>
|
|
```
|
|
|
|
Sample Output: `just lookup 52.496846793890256 13.435128880554208`
|
|
|
|
```json
|
|
[
|
|
{
|
|
"type": "polygon",
|
|
"leisure": "park",
|
|
"name": "Görlitzer Park"
|
|
},
|
|
{
|
|
"type": "polygon",
|
|
"landuse": "forest"
|
|
},
|
|
{
|
|
"type": "polygon",
|
|
"name": "Luisenstadt"
|
|
},
|
|
{
|
|
"type": "polygon",
|
|
"name": "Jugendverkehrsschule Wiener Straße"
|
|
},
|
|
{
|
|
"type": "polygon",
|
|
"name": "Friedrichshain-Kreuzberg"
|
|
},
|
|
{
|
|
"type": "polygon",
|
|
"name": "Kreuzberg"
|
|
}
|
|
]
|
|
```
|
|
|
|
### 4. Web API
|
|
|
|
API on port `8000` with endpoint `get_geodata` that takes the parameter `geojson` (min 4 digits) and `format` (`parquet|geojson|cbor`) and returns a file in the requested format. Supports optional Gzip compression during transfer.
|
|
|
|
Start server:
|
|
|
|
```bash
|
|
just serve
|
|
```
|
|
|
|
Request example:
|
|
|
|
```bash
|
|
curl -H "Accept-Encoding: gzip" -o geodata.cbor.gz -v http://localhost:8000/get_geodata/?geohash=svc5&format=geojson
|
|
```
|
|
|
|
## Advanced Use
|
|
|
|
### Extract and Filter Data
|
|
|
|
To just extract the filtered data from an OSM PBF file and save it as parquet file
|
|
|
|
```bash
|
|
just extract <osm_data.pbf> <extracted_data.parquet>
|
|
```
|
|
|
|
### Download OSM Data File for Region
|
|
|
|
Automatically download a file from Geofabrik based on the name of a region, filter it, and save it as parquet file.
|
|
|
|
```bash
|
|
just download '<name of the region>' <extracted_data.parquet>
|
|
```
|
|
|
|
The original PBF file remains saved in the `files/` folder.
|
|
|
|
### Split Parquet File to GeoHash Parquet Files
|
|
|
|
```bash
|
|
just convert <extracted_data.parquet> <geohash_parquet_folder>
|
|
```
|
|
|
|
### Plot Parquet File as PNG
|
|
|
|
Plot the geometry of a parquet file as a PNG with the same filename.
|
|
|
|
```bash
|
|
just plot <file.parquet>
|
|
```
|
|
|
|
Sample Output:
|
|
|
|
|
|
## TO DO
|
|
|
|
- [X] GeoHash parquet file generation
|
|
- [X] Append data to existing parquet files
|
|
- [X] Create web API server
|
|
- [ ] Dockerize it
|
|
- [ ] Scheduled update routine |