dopecarpet/readme.md

# dopecarpet

A toolset to generate 4 digit GeoHash tile sized Parquet/GeoJSON/CBOR files from OpenStreetMap PBF files. It comes with a custom tag filter and a minimal GET API to download the data in different formats.

## Dependencies

Installation (Fedora)

```bash
sudo dnf install osmium-tool jq python3-devel pip
pip install quackosm[cli] psycopg2-binary pyosmium geoalchemy2 geojson shapely pygeohash pyarrow pandas click
```

## Basic Use

### 1. Customize Filter

Edit the `tags.json` to specify which OSM tags you want to include in the metadata. This affects what geometries are extracted from the OSM data and has a direct influence on the file sizes.

```json
{
    "name:en": true,
    "name": true,
    "aeroway": true,
    "boundary": ["aboriginal_lands", "border_zone", "forest", "hazard", "national_park", "protected_area", "disputed"],
    "geological": true,
    "highway": ["motorway", "trunk", "primary", "raceway"],
    "historic": true,
    "landuse": true,
    "leisure": ["dog_park", "garden", "miniature_golf", "nature_reserve", "park", "playground", "stadium", "swimming_pool", "water_park"],
    "military": ["airfield", "base", "bunker", "barracks", "danger_area", "nuclear_explosion_site", "range", "training_area"],
    "natural": true,
    "place": ["plot", "farm", "island", "islet", "sea", "ocean"],
    "power": ["plant"],
    "railway": ["abandoned", "construction", "disused", "rail"],
    "tourism": ["attraction", "yes"],
    "water": true,
    "waterway": true
}
```

### 2. Generate GeoHash Files

This extracts the desired data from the PBF file and generates four digit geohash parquet files in the `geohash/` directory.

```bash
just process <osm_data.pbf>
```

### 3. Lookup Coordinate Metadata

Prints metadata found for a coordinate as JSON object list.

```bash
just lookup <latitude>, <longitude>
```

Sample Output: `just lookup 52.496846793890256 13.435128880554208`

```json
[
  {
    "type": "polygon",
    "leisure": "park",
    "name": "Görlitzer Park"
  },
  {
    "type": "polygon",
    "landuse": "forest"
  },
  {
    "type": "polygon",
    "name": "Luisenstadt"
  },
  {
    "type": "polygon",
    "name": "Jugendverkehrsschule Wiener Straße"
  },
  {
    "type": "polygon",
    "name": "Friedrichshain-Kreuzberg"
  },
  {
    "type": "polygon",
    "name": "Kreuzberg"
  }
]
```

### 4. Web API

API on port `8000` with endpoint `get_geodata` that takes the parameter `geojson` (min 4 digits) and `format` (`parquet|geojson|cbor`) and returns a file in the requested format. Supports optional Gzip compression during transfer.

Start server:

```bash
just serve
```

Request example:

```bash
curl -H "Accept-Encoding: gzip" -o geodata.cbor.gz -v http://localhost:8000/get_geodata/?geohash=svc5&format=geojson
```

## Advanced Use

### Extract and Filter Data

To just extract the filtered data from an OSM PBF file and save it as parquet file

```bash
just extract <osm_data.pbf> <extracted_data.parquet>
```

### Download OSM Data File for Region

Automatically download a file from Geofabrik based on the name of a region, filter it, and save it as parquet file.

```bash
just download '<name of the region>' <extracted_data.parquet>
```

The original PBF file remains saved in the `files/` folder.

### Split Parquet File to GeoHash Parquet Files and add Sea Polygons

```bash
just convert <extracted_data.parquet> <water_polygons.parquet> <geohash_parquet_folder>
```

### Convert Shapefile to Parquet

Input file needs to be 'EPSG:4326'

```bash
just shp <shapefile.shp> <output.parquet>
```

### Plot Parquet File as PNG

Plot the geometry of a parquet file as a PNG with the same filename.

```bash
just plot <file.parquet>
```

Sample Output:
![snippet](u33d.png)

## TO DO

- [X] GeoHash parquet file generation
- [X] Append data to existing parquet files
- [X] Create web API server
- [ ] Dockerize it
- [ ] Scheduled update routine