How To Serialize Information Successfully with Protocol Buffers – Grape Up

In a world of microservices, we frequently must go data between functions. We serialize knowledge right into a format that may be retrieved by each side. One of many serialization options is Protocol Buffers (Protobuf) – Google’s language-neutral mechanism. Messages will be interpreted by a receiver utilizing the identical or totally different language than a producer. Many languages are supported, equivalent to Java, Go, Python, and C++.

An information construction is outlined utilizing impartial language by means of .proto information. The file is then compiled into code for use in functions. It’s designed for efficiency. Protocol Buffers encode knowledge into binary format, which reduces message measurement and improves transmission velocity.

Defining Message Format

This .proto the file represents geolocation data for a given car.

1 syntax = "proto3";
3 package deal com.grapeup.geolocation;
5 import "google/kind/latlng.proto";
6 import "google/protobuf/timestamp.proto";
8 message Geolocation 
9  string vin = 1;
10  google.protobuf.Timestamp occurredOn = 2;
11  int32 velocity = 3;
12  google.kind.LatLng coordinates = 4;
1 syntax = "proto3";

Syntax refers to Protobuf model, it may be proto2 or proto3.

1package com.grapeup.geolocation;

Package deal declaration prevents naming conflicts between totally different tasks.

1 message Geolocation 
2  string vin = 1;
3  google.protobuf.Timestamp occurredOn = 2;
4  int32 velocity = 3;
5  google.kind.LatLng coordinates = 4;

Message definition accommodates a reputation and a set of typed fields. Easy knowledge sorts can be found, equivalent to bool, int32, double, string, and so on. You can even outline your individual sorts or import them.

1google.protobuf.Timestamp occurredOn = 2;

The = 1, = 2 markers establish the distinctive tag. Tags are a numeric illustration for the sector and are used to establish the sector within the message binary format. They must be distinctive in a message and shouldn’t be modified as soon as the message is in use. If a subject is faraway from a definition that’s already used, it should be reserved.

Area sorts

Apart from scalar sorts, there are numerous different kind choices when defining messages. Listed below are few, however you could find all of them within the Language Information Language Guide (proto3)  |  Protocol Buffers  |  Google Developers .

Effectively Identified Sorts

1 import "google/kind/latlng.proto";
2 import "google/protobuf/timestamp.proto";
4 google.protobuf.Timestamp occurredOn = 2;
5 google.kind.LatLng coordinates = 4;

There are predefined sorts obtainable to make use of Overview  |  Protocol Buffers  |  Google Developers . They’re often known as Effectively Know Sorts and must be imported into .proto .

LatLng represents a latitude and longitude pair.

Timestamp is a particular time limit with nanosecond precision.

Customized sorts

1 message SingularSearchResponse 
2  Geolocation geolocation = 1;

You need to use your custom-defined kind as a subject in one other message definition.


1 message SearchResponse 
2  repeated Geolocation geolocations = 1;

You’ll be able to outline lists through the use of repeated key phrase.


It could possibly occur that in a message there’ll all the time be just one subject set. On this case, TelemetryUpdate will include both geolocation, mileage, or gasoline degree data.

This may be achieved through the use of oneof. Setting worth to one of many fields will clear all different fields outlined in oneof.

1 message TelemetryUpdate 
2  string vin = 1;
3  oneof replace 
4    Geolocation geolocation = 2;
5    Mileage mileage =3;
6    FuelLevel fuelLevel = 4;
10 message Geolocation 
11  ...
14 message Mileage 
15  ...
18 message FuelLevel 
19  ...

Take into accout backward-compatibility when eradicating fields. If you happen to obtain a message with oneof that has been faraway from .proto definition, it won’t set any of the values. This conduct is similar as not setting any worth within the first place.

You’ll be able to carry out totally different actions primarily based on which worth is ready utilizing the getUpdateCase() technique.

1 public Non-compulsory<Object> getTelemetry(TelemetryUpdate telemetryUpdate) 
2        Non-compulsory<Object> telemetry = Non-compulsory.empty();
3        change (telemetryUpdate.getUpdateCase()) 
4            case MILEAGE -> telemetry = Non-compulsory.of(telemetryUpdate.getMileage());
5            case FUELLEVEL -> telemetry = Non-compulsory.of(telemetryUpdate.getFuelLevel());
6            case GEOLOCATION -> telemetry = Non-compulsory.of(telemetryUpdate.getGeolocation());
7            case UPDATE_NOT_SET -> telemetry = Non-compulsory.empty();
9        return telemetry;

Default values

In proto3 format fields will all the time have a price. Due to this proto3 can have a smaller measurement as a result of fields with default values are omitted from payload. Nevertheless this causes one difficulty – for scalar message fields, there is no such thing as a method of telling if a subject was explicitly set to the default worth or not set in any respect.

In our instance, velocity is an non-obligatory subject – some modules in a automobile would possibly ship velocity knowledge, and a few may not. If we don’t set velocity, then the geolocation object may have velocity with the default worth set to 0. This isn’t the identical as not having velocity set on messages.

In an effort to cope with default values you should utilize official wrapper sorts protobuf/wrappers.proto at main · protocolbuffers/protobuf . They permit distinguishing between absence and default. As an alternative of getting a easy kind, we use Int32Value, which is a wrapper for the int32 scalar kind.

1 import "google/protobuf/wrappers.proto";
3 message Geolocation 
4  google.protobuf.Int32Value velocity = 3;

If we don’t present velocity, it will likely be set to nil.

Configure with Gradle

When you’ve outlined your messages, you should utilize protoc, a protocol buffer compiler, to generate courses in a selected language. The generated class can then be used to construct and retrieve messages.

In an effort to compile into Java code, we have to add dependency and plugin in construct.gradle

1 plugins 
2    id '' model '0.8.18'
5 dependencies 
6    implementation ''

and setup the compiler. For Mac customers an osx particular model must be used.

1 protobuf 
2    protoc 
3        if (osdetector.os == "osx") 
4            artifact = "$protobuf_version:osx-x86_64"
5         else 
6            artifact = "$protobuf_version"

Code will probably be generated utilizing generateProto process.

The code will probably be situated in construct/generated/supply/proto/foremost/java in a package deal as laid out in .proto file.

We additionally want to inform gradle the place the generated code is situated

1 sourceSets 
2    foremost 
3        java 
4            srcDirs 'construct/generated/supply/proto/foremost/grpc'
5            srcDirs 'construct/generated/supply/proto/foremost/java'

The generated class accommodates all the required strategies for constructing the message in addition to retrieving subject values.

1 Geolocation geolocation = Geolocation.newBuilder()
2            .setCoordinates(LatLng.newBuilder().setLatitude(1.2).setLongitude(1.2).construct())
3            .setVin("1G2NF12FX2C129610")
4            .setOccurredOn(Timestamp.newBuilder().setSeconds(12349023).construct())
5            .construct();
7 LatLng coordinates = geolocation.getCoordinates();
8 String vin = geolocation.getVin();

Protocol Buffers – Abstract

As proven protocol buffers are simply configured. The mechanism is language agnostic, and it’s straightforward to share the identical .proto definition throughout totally different microservices.

Protobuf is well paired with gRPC, the place strategies will be outlined in .proto information and generated with gradle.

There’s official documentation obtainable Protocol Buffers  |  Google Developers and guides Language Guide (proto3)  |  Protocol Buffers  |  Google Developers .