A revised breaking protobuf schema for Geobuf

[mourner]

The more I find ideas to improve the sizes of geobuf, the more I realize that we would need to completely rewrite the schema to support the improvements, breaking compatibility. So I'm opening the ticket to start a discussion about what a perfect Geobuf schema would look like (not to say this is a priority, but still a good thing to discuss).

I wrote a prototype schema with all the improvements I could think of here: https://gist.github.com/mourner/3c6ddca04c9772593302

The main difference is utilizing the power and flexibility of the new oneof statement to create a better and more compact schema.

Features:

• the data itself contains information whether it's Feature, FeatureCollection, Geometry or GeometryCollection, so you don't need to guess this when decoding
• keys and values for properties are stored separately in the top-level Data object, and features only store indexes to them (like vector-tile-spec does); keys and possibly values are reduced to unique values Fancy encoding for properties #26 — for much better properties packing
• geometry coordinates are a oneof set of different fields (depending on type), which solves the ambiguity with null values since a oneof field can be empty (without a default value), and also makes it easier to understand and work with
• feature has a oneof of either geometry or geometry collection instead of repeated geometries
• coordinates are stored as delta-encoded sint32 to take full advantage of varint and zig-zag encoding Switch to sint32 for coords #24 — for much more compact geometries
• Value message is also a oneof of different value types
• the property value int type is sint32 instead of int64 — it's more compact and JS doesn't actually handle int64 values; in addition, uint type becomes uint32
• the data contains optional flag indicating whether it contains altitude (third coordinate), since this is a global-level setting
• the data also contains optional precision information (6 by default) Configurable precision #25

@tmcw @springmeyer what do you think?

[springmeyer]

Awesome to learn about oneof. Overall your ideas look excellent. I'll comment more when I've had time to digest the proposed schema.

[brendan-ward]

This looks like a great direction.

What about letting the feature id field leverage oneof to allow encoding as uint32 values (much more common than string IDs in many cases we deal with):

message Feature {
    oneof geometry_type {
        Geometry geometry = 1;
        GeometryCollection geometry_collection = 2;
    }
    repeated uint32 properties = 3 [packed = true];
    optional ID id = 4;
}

message ID {
    oneof value {
        string string_value = 1;
        uint32 uint_value = 2;
    }
}

[mourner]

@brendan-ward yes, excellent suggestion! Added it to the proto gist, except including oneof directly in the Feature message, since oneof fields are already optional by nature.

[mourner]

also cc @mick @rclark @sgillies in case you're interested

[mourner]

OK guys, today I wrote the encoder of my proposed format in Python to test it out, and the results are pretty mind-blowing.

Here's the comparison for the 100Mb 5.4-million points (1e4 precision) US zip codes GeoJSON (that I used for testing geojson-vt):

	normal	gzipped
GeoJSON	101.85 MB	26.67 MB
current geobuf	73.53 MB	31.09 MB
my geobuf 1e6	22.27 MB	16.39 MB
my geobuf 1e4	12.43 MB	10.62 MB

Here's another sample, Idaho from geojson-fixtures:

	normal	gzipped
GeoJSON	10.93 MB	2.58 MB
current geobuf	5.53 MB	2.07 MB
my geobuf 1e6	1.38 MB	1.18 MB

The keys of properties stored are unique. I think I can squish out just a bit more by making string values unique too.

[tmcw]

Cool. To be clear, "your geobuf" can be "our geobuf" :)

[mourner]

@tmcw sure, I just wanted to share the motivation before suggesting that I completely rewrite this repo from scratch while breaking compatibility, which can be very annoying. :)

[tmcw]

I don't mind anything being rewritten ever as long as it's better :) But the people who might feel the pain of this would be @mick basically, because it's used in cardboard afaik

[mick]

@mourner I will feel the pain. But that's is ok, this like a great rewrite. I can migrate my usage of geobuf to the new version.

[mourner]

Reference Python encoder here: https://github.com/mapbox/pygeobuf
Working on decoder and some tests, and after that we can start reworking the JS implementation.

[kkaefer]

Any comparisons to TopoJSON?