Spring REST API with Protocol Buffers

1. Overview

Protocol Buffers is a language and platform neutral mechanism for serialization and deserialization of structured data, which is proclaimed by Google, its creator, to be much faster, smaller and simpler than other types of payloads, such as XML and JSON.

This tutorial guides you through setting up a REST API to take advantage of this binary-based message structure.

2. Protocol Buffers

This section gives some basic information on Protocol Buffers and how they are applied in the Java ecosystem.

2.1. Introduction to Protocol Buffers

In order to make use of Protocol Buffers, we need to define message structures in .proto files. Each file is a description of the data that might be transferred from one node to another, or stored in data sources. Here is an example of .proto files, which is named maixuanviet.proto and lives in the src/main/resources directory. This file will be used in this tutorial later on:

syntax = "proto3";
package maixuanviet;
option java_package = "com.maixuanviet.protobuf";
option java_outer_classname = "VietMXTraining";

message Course {
    int32 id = 1;
    string course_name = 2;
    repeated Student student = 3;
}
message Student {
    int32 id = 1;
    string first_name = 2;
    string last_name = 3;
    string email = 4;
    repeated PhoneNumber phone = 5;
    message PhoneNumber {
        string number = 1;
        PhoneType type = 2;
    }
    enum PhoneType {
        MOBILE = 0;
        LANDLINE = 1;
    }
}

In this tutorial, we use version 3 of both protocol buffer compiler and protocol buffer language, therefore the .proto file must start with the syntax = “proto3” declaration. If a compiler version 2 is in use, this declaration would be omitted. Next comes the package declaration, which is the namespace for this message structure to avoid naming conflicts with other projects.

The following two declarations are used for Java only: java_package option specifies the package for our generated classes to live in, and java_outer_classname option indicates name of the class enclosing all the types defined in this .proto file.

Subsection 2.3 below will describe the remaining elements and how those are compiled into Java code.

2.2. Protocol Buffers With Java

After a message structure is defined, we need a compiler to convert this language neutral content to Java code. You can follow the instructions in the Protocol Buffers repository in order to get an appropriate compiler version. Alternatively, you may download a pre-built binary compiler from the Maven central repository by searching for the com.google.protobuf:protoc artifact, then picking up an appropriate version for your platform.

Next, copy the compiler to the src/main directory of your project and execute the following command in the command line:

protoc --java_out=java resources/maixuanviet.proto

This should generate a source file for the VietMXTraining class within the com.maixuanviet.protobuf package, as specified in the option declarations of the maixuanviet.proto file.

In addition to the compiler, Protocol Buffers runtime is required. This can be achieved by adding the following dependency to the Maven POM file:

<dependency>
    <groupId>com.google.protobuf</groupId>
    <artifactId>protobuf-java</artifactId>
    <version>3.0.0-beta-3</version>
</dependency>

We may use another version of the runtime, provided that it is the same as the compiler’s version. For the latest one, please check out this link.

2.3. Compiling a Message Description

By using a compiler, messages in a .proto file are compiled into static nested Java classes. In the above example, the Course and Student messages are converted to Course and Student Java classes, respectively. At the same time, messages’ fields are compiled into JavaBeans style getters and setters inside those generated types. The marker, composed of an equal sign and a number, at the end of each field declaration is the unique tag used to encode the associated field in the binary form.

We will walk through typed fields of the messages to see how those are converted to accessor methods.

Let’s start with the Course message. It has two simple fields, including id and course_name. Their protocol buffer types, int32 and string, are translated into Java int and String types. Here are their associated getters after compilation (with implementations being left out for brevity):

public int getId();
public java.lang.String getCourseName();

Note that names of typed fields should be in snake case (individual words are separated by underscore characters) to maintain the cooperation with other languages. The compiler will convert those names to camel case according to Java conventions.

The last field of Course message, student, is of the Student complex type, which will be described below. This field is prepended by the repeated keyword, meaning that it may be repeated any number of times. The compiler generates some methods associated with the student field as follows (without implementations):

public java.util.List<com.maixuanviet.protobuf.VietMXTraining.Student> getStudentList();
public int getStudentCount();
public com.maixuanviet.protobuf.VietMXTraining.Student getStudent(int index);

Now we will move on to the Student message, which is used as complex type of the student field of Course message. Its simple fields, including id, first_name, last_name and email are used to create Java accessor methods:

public int getId();
public java.lang.String getFirstName();
public java.lang.String getLastName();
public java.lang.String.getEmail();

The last field, phone, is of the PhoneNumber complex type. Similar to the student field of Course message, this field is repetitive and has several associated methods:

public java.util.List<com.maixuanviet.protobuf.VietMXTraining.Student.PhoneNumber> getPhoneList();
public int getPhoneCount();
public com.maixuanviet.protobuf.VietMXTraining.Student.PhoneNumber getPhone(int index);

The PhoneNumber message is compiled into the VietMXTraining.Student.PhoneNumber nested type, with two getters corresponding to the message’s fields:

public java.lang.String getNumber();
public com.maixuanviet.protobuf.VietMXTraining.Student.PhoneType getType();

PhoneType, the complex type of the type field of the PhoneNumber message, is an enumeration type, which will be transformed into a Java enum type nested within the VietMXTraining.Student class:

public enum PhoneType implements com.google.protobuf.ProtocolMessageEnum {
    MOBILE(0),
    LANDLINE(1),
    UNRECOGNIZED(-1),
    ;
    // Other declarations
}

3. Protobuf in Spring REST API

This section will guide you through setting up a REST service using Spring Boot.

3.1. Bean Declaration

Let’s start with the definition of our main @SpringBootApplication:

@SpringBootApplication
public class Application {
    @Bean
    ProtobufHttpMessageConverter protobufHttpMessageConverter() {
        return new ProtobufHttpMessageConverter();
    }

    @Bean
    public CourseRepository createTestCourses() {
        Map<Integer, Course> courses = new HashMap<>();
        Course course1 = Course.newBuilder()
          .setId(1)
          .setCourseName("REST with Spring")
          .addAllStudent(createTestStudents())
          .build();
        Course course2 = Course.newBuilder()
          .setId(2)
          .setCourseName("Learn Spring Security")
          .addAllStudent(new ArrayList<Student>())
          .build();
        courses.put(course1.getId(), course1);
        courses.put(course2.getId(), course2);
        return new CourseRepository(courses);
    }

    // Other declarations
}

The ProtobufHttpMessageConverter bean is used to convert responses returned by @RequestMapping annotated methods to protocol buffer messages.

The other bean, CourseRepository, contains some test data for our API.

What’s important here is that we’re operating with Protocol Buffer specific data – not with standard POJOs.

Here’s the simple implementation of the CourseRepository:

public class CourseRepository {
    Map<Integer, Course> courses;
    
    public CourseRepository (Map<Integer, Course> courses) {
        this.courses = courses;
    }
    
    public Course getCourse(int id) {
        return courses.get(id);
    }
}

3.2. Controller Configuration

We can define the @Controller class for a test URL as follows:

@RestController
public class CourseController {
    @Autowired
    CourseRepository courseRepo;

    @RequestMapping("/courses/{id}")
    Course customer(@PathVariable Integer id) {
        return courseRepo.getCourse(id);
    }
}

And again – the important thing here is that the Course DTO that we’re returning from the controller layer is not a standard POJO. That’s going to be the trigger for it to be converted to protocol buffer messages before being transferred back to the Client.

4. REST Clients and Testing

Now that we had a look at the simple API implementation – let’s now illustrate deserialization of protocol buffer messages on the client side – using two methods.

The first one takes advantage of the RestTemplate API with a pre-configured ProtobufHttpMessageConverter bean to automatically convert messages.

The second is using protobuf-java-format to manually transform protocol buffer responses into JSON documents.

To begin, we need to set up the context for an integration test and instruct Spring Boot to find configuration information in the Application class by declaring a test class as follows:

@RunWith(SpringJUnit4ClassRunner.class)
@SpringApplicationConfiguration(classes = Application.class)
@WebIntegrationTest
public class ApplicationTest {
    // Other declarations
}

All code snippets in this section will be placed in the ApplicationTest class.

4.1. Expected Response

The first step to access a REST service is to determine the request URL:

private static final String COURSE1_URL = "http://localhost:8080/courses/1";

This COURSE1_URL will be used for getting the first test double course from the REST service we created before. After a GET request is sent to the above URL, the corresponding response is verified using the following assertions:

private void assertResponse(String response) {
    assertThat(response, containsString("id"));
    assertThat(response, containsString("course_name"));
    assertThat(response, containsString("REST with Spring"));
    assertThat(response, containsString("student"));
    assertThat(response, containsString("first_name"));
    assertThat(response, containsString("last_name"));
    assertThat(response, containsString("email"));
    assertThat(response, containsString("john.doe@maixuanviet.com"));
    assertThat(response, containsString("richard.roe@maixuanviet.com"));
    assertThat(response, containsString("jane.doe@maixuanviet.com"));
    assertThat(response, containsString("phone"));
    assertThat(response, containsString("number"));
    assertThat(response, containsString("type"));
}

We will make use of this helper method in both test cases covered in the succeeding sub-sections.

4.2. Testing With RestTemplate

Here is how we create a client, send a GET request to the designated destination, receive the response in the form of protocol buffer messages and verify it using the RestTemplate API:

@Autowired
private RestTemplate restTemplate;

@Test
public void whenUsingRestTemplate_thenSucceed() {
    ResponseEntity<Course> course = restTemplate.getForEntity(COURSE1_URL, Course.class);
    assertResponse(course.toString());
}

To make this test case work, we need a bean of the RestTemplate type to be registered in a configuration class:

@Bean
RestTemplate restTemplate(ProtobufHttpMessageConverter hmc) {
    return new RestTemplate(Arrays.asList(hmc));
}

Another bean of the ProtobufHttpMessageConverter type is also required to automatically transform the received protocol buffer messages. This bean is the same as the one defined in sub-section 3.1. Since the client and server share the same application context in this tutorial, we may declare the RestTemplate bean in the Application class and re-use the ProtobufHttpMessageConverter bean.

4.3. Testing With HttpClient

The first step to use the HttpClient API and manually convert protocol buffer messages is adding the following two dependencies to the Maven POM file:

<dependency>
    <groupId>com.googlecode.protobuf-java-format</groupId>
    <artifactId>protobuf-java-format</artifactId>
    <version>1.4</version>
</dependency>
<dependency>
    <groupId>org.apache.httpcomponents</groupId>
    <artifactId>httpclient</artifactId>
    <version>4.5.2</version>
</dependency>

For the latest versions of these dependencies, please have a look at protobuf-java-format and httpclient artifacts in Maven central repository.

Let’s move on to create a client, execute a GET request and convert the associated response to an InputStream instance using the given URL:

private InputStream executeHttpRequest(String url) throws IOException {
    CloseableHttpClient httpClient = HttpClients.createDefault();
    HttpGet request = new HttpGet(url);
    HttpResponse httpResponse = httpClient.execute(request);
    return httpResponse.getEntity().getContent();
}

Now, we will convert protocol buffer messages in the form of an InputStream object to a JSON document:

private String convertProtobufMessageStreamToJsonString(InputStream protobufStream) throws IOException {
    JsonFormat jsonFormat = new JsonFormat();
    Course course = Course.parseFrom(protobufStream);
    return jsonFormat.printToString(course);
}

And here is how a test case uses private helper methods declared above and validates the response:

@Test
public void whenUsingHttpClient_thenSucceed() throws IOException {
    InputStream responseStream = executeHttpRequest(COURSE1_URL);
    String jsonOutput = convertProtobufMessageStreamToJsonString(responseStream);
    assertResponse(jsonOutput);
}

4.4. Response in JSON

In order to make it clear, JSON forms of the responses we received in the tests described in previous sub-sections are included herein:

id: 1
course_name: "REST with Spring"
student {
    id: 1
    first_name: "John"
    last_name: "Doe"
    email: "john.doe@maixuanviet.com"
    phone {
        number: "123456"
    }
}
student {
    id: 2
    first_name: "Richard"
    last_name: "Roe"
    email: "richard.roe@maixuanviet.com"
    phone {
        number: "234567"
        type: LANDLINE
    }
}
student {
    id: 3
    first_name: "Jane"
    last_name: "Doe"
    email: "jane.doe@maixuanviet.com"
    phone {
        number: "345678"
    }
    phone {
        number: "456789"
        type: LANDLINE
    }
}

5. Conclusion

This tutorial quickly introduced Protocol Buffers and illustrated the setting up of a REST API using the format with Spring. We then moved to client support and the serialization-deserialization mechanism.

The implementation of all the examples and code snippets can be found in a GitHub project.