readme

2025-02-26 14:51:25 +00:00 · 2025-02-26 14:51:25 +00:00 · fc00e77b33
commit fc00e77b33
parent fc91d3c884
3 changed files with 136 additions and 94 deletions
--- a/readme.md
+++ b/readme.md
@ -0,0 +1,136 @@
+<image src="https://blog.therugseller.co.uk/wp-content/uploads/2017/10/2-10.png" width="400" />
+<p>&nbsp;<p>
+
+# dopecarpet
+
+Generate GeoHash parquet files from OpenStreetMap PBF files with custom tag filter to look up metadata for coordinates.
+
+## Dependencies
+
+Installation (Fedora)
+
+```bash
+sudo dnf install osmium-tool jq python3-devel pip
+pip install quackosm[cli] psycopg2-binary pyosmium geoalchemy2 geojson shapely pygeohash pyarrow pandas click
+```
+
+## Basic Use
+
+### 1. Customize Filter
+
+Edit the `tags.json` to specify which OSM tags you want to include in the metadata. This affects what geometries are extracted from the OSM data and has a direct influence on the file sizes.
+
+```json
+{
+    "name:en": true,
+    "name": true,
+    "aeroway": true,
+    "boundary": ["aboriginal_lands", "border_zone", "forest", "hazard", "national_park", "protected_area", "disputed"],
+    "geological": true,
+    "highway": ["motorway", "trunk", "primary", "raceway"],
+    "historic": true,
+    "landuse": true,
+    "leisure": ["dog_park", "garden", "miniature_golf", "nature_reserve", "park", "playground", "stadium", "swimming_pool", "water_park"],
+    "military": ["airfield", "base", "bunker", "barracks", "danger_area", "nuclear_explosion_site", "range", "training_area"],
+    "natural": true,
+    "place": ["plot", "farm", "island", "islet", "sea", "ocean"],
+    "power": ["plant"],
+    "railway": ["abandoned", "construction", "disused", "rail"],
+    "tourism": ["attraction", "yes"],
+    "water": true,
+    "waterway": true
+}
+```
+
+### 2. Generate GeoHash Files
+
+This extracts the desired data from the PBF file and generates four digit geohash parquet files in the `geohash/` directory.
+
+```bash
+just process <osm_data.pbf>
+```
+
+### 3. Lookup Coordinate Metadata
+
+Prints metadata found for a coordinate as JSON object list.
+
+```bash
+just lookup <latitude>, <longitude>
+```
+
+Sample Output: `just lookup 52.496846793890256 13.435128880554208`
+
+```json
+[
+  {
+    "type": "polygon",
+    "leisure": "park",
+    "name": "Görlitzer Park"
+  },
+  {
+    "type": "polygon",
+    "landuse": "forest"
+  },
+  {
+    "type": "polygon",
+    "name": "Luisenstadt"
+  },
+  {
+    "type": "polygon",
+    "name": "Jugendverkehrsschule Wiener Straße"
+  },
+  {
+    "type": "polygon",
+    "name": "Friedrichshain-Kreuzberg"
+  },
+  {
+    "type": "polygon",
+    "name": "Kreuzberg"
+  }
+]
+```
+
+## Advanced Use
+
+### Extract and Filter Data
+
+To just extract the filtered data from an OSM PBF file and save it as parquet file
+
+```bash
+just extract <osm_data.pbf> <extracted_data.parquet>
+```
+
+### Download OSM Data File for Region
+
+Automatically download a file from Geofabrik based on the name of a region, filter it, and save it as parquet file.
+
+```bash
+just download '<name of the region>' <extracted_data.parquet>
+```
+
+The original PBF file remains saved in the `files/` folder.
+
+### Split Parquet File to GeoHash Parquet Files
+
+```bash
+just convert <extracted_data.parquet> <geohash_parquet_folder>
+```
+
+### Plot Parquet File as PNG
+
+Plot the geometry of a parquet file as a PNG with the same filename.
+
+```bash
+just plot <file.parquet>
+```
+
+Sample Output:
+![snippet](u33d.png)
+
+## TO DO
+
+- [X] GeoHash parquet file generation
+- [X] Append data to existing parquet files
+- [ ] Create web API server
+- [ ] Dockerize it
+- [ ] Scheduled update routine