The joys of RPCs and web servers in java

So this will be about signald and the ongoing work of trying to find a way to provide as many client libraries in as many programming languages as possible with the least effort possible. You will quickly see that I just can't stand doing things half-assed and that "oh it works" isn't satisfactory.

So, first lets take a step back and answer the following question

What is signald?

If you haven't heard of Signal yet, it is an opensource alternative to Facebook's Whatsapp that is continuing to take flak for invading user's privacy. For reasons I do not fully comprehend, Signal hasn't decided to become the open-source solution to secure and decentralized communications free from the threat of government intervention. Why do I say this?

Well, Signal doesn't:

provide an easy deployment guide for their server
the server isn't federated
the server doesn't provide any alternative to the hardcoded proprietary dependencies (e.g SMS)
third-party clients are frowned upon
there is no simple programmatic client, lib or framework to help build applications that use Signal

The last point is where signald comes in. It stands for Signal Daemon, is written in Java and is basically a local application that provides a much simpler interface to make calls to the signal server. The interface uses JSON as the data format and a UNIX socket as a transport. What it doesn't do however is provide clients.

The initial idea

Since signald :

uses JSON as the data format
doesn't provide any real security besides file permissions to the UNIX socket
the configuration is stored in the folder of the process's user

I set out looking for an easy method to provide an API and client libraries. My search brought me to OpenAPI generator. Using a specification file written in JSON or YAML (maybe TOML too?) it can then generate client libraries that interact with a HTTP REST API. So, happy with the prospect of saving lots of work, I went ahead and started adding a HTTP server to signald.

Not so fast!

The first problem I ran into was Java. I hadn't written a single line of Java (Groovy for Jenkins CI doesn't count) in nearly 10 years. Suffice to say I was a little rusty. Luckily, the project uses Java 8, which doesn't seem to have evolved much besides adding better handling for async code as well as streams (which wasn't being used in the project).

Great! That means I could get started!

...except... not really. Signald has a SocketHandler that handles input to UNIX socket. It's a huge class that handles every single method supported by the interface. All in one class.
That, had to be taken apart and put into classes.

Once that was done the next problem was around the corner

Picking a lightweight Java web server

Once I got into the task, I knew exactly why I had switched over to Ruby, then Python, then Javascript for server-side web-development so long ago.

Java web-servers as great, monolithic monstrosities with Java Glassfish top among them. And they aren't actually webservers, they are application servers. These operate using "servlets" which, as far as I understand, are .wars ("web .jars) that abide by a protocol allowing them to be loaded into the application server.

There are lots of names out there, but Glassfish from Java EE, and Tomcat stand out.

These were of course a no-go. You don't want a huge, hunking application server running in a daemon just to provide an API.

Java's inbuilt HTTP server

Seems like a good idea, but isn't. All it really does is give you juuuust enough material to start-up something, but then reinvent the wheel for everything: path handling, GET params handling, POST params handling, uploads, downloads, security, etc.

Another NOGO.

Viable candidates

I finally settled upon Spark in favor of undertow. Both provide the same, basic functionality that I needed (a simple server able to handle POST requests using handlers)

import static spark.Spark.*;

public class HelloWorld {
    public static void main(String[] args) {
        get("/hello", (req, res) -> "Hello World");
    }
}

Simple, right?

But wait, there's more

With the UNIX socket handlers separated into classes and the HTTP server moving slowly forward, I went ahead and wrote an OpenAPI spec.
A good example of a spec is on the Swagger Editor.

After writing one single POST handler for Spark, I got annoyed. Why write all the input checks manually when they already exist in the spec? Everything is there: params, types, count, regex patterns, etc.

Lo and behold, OpenAPI generator does have generators for Java webservers and servlets!

So let's go ahead aaaaand this is an example output:

public class PathHandlerProvider implements HandlerProvider {

    public HttpHandler getHandler() {
        HttpHandler handler = Handlers.routing()


            .add(Methods.POST, "/v2/pet", new HttpHandler() {
                        public void handleRequest(HttpServerExchange exchange) throws Exception {
                            exchange.getResponseSender().send("addPet");
                        }
                    })
         ;
        return handler;
    }
}

noooo

Where are the params and input checks??

How can I customize the return value without the code being overwritten everytime I add a new API endpoint?

Fuck this, let's see what else there is

I wasn't at all in the mood of writing a template for the OpenAPI generator to output abstract Handlers that I could subclass and somehow register with the HanderProvider. Nor was I in the mood of learning undertow to deal with input checking.

So I looked around and long story short, I found gRPC, google's solution to RPCs, which surprisingly uses HTTP/2. Honestly though, I should've expected it. They are a web company after-all and why use anything but a hammer when it's served you so well so far?

Enter gRPC

It uses a DSL called Protocol Buffers to declare services, their endpoints, their input and output data structures and can even define security access profiles. Most importantly, they check input and the generated Handlers need only be subclassed with the right naming convention in order to work.

Great!

Here's an example of a service definition

syntax = "proto3";
service HelloService {
  rpc SayHello (HelloRequest) returns (HelloResponse);
}

message HelloRequest {
  string greeting = 1;
}

message HelloResponse {
  string reply = 1;
}

Not so fast!

I thought I was on the home stretch, but being Google being Google, they only write tools that serve their usecases. If your usecase doesn't fit into theirs somehow, well, you're piss outta luck. There are a few examples of this (maybe I'll find them later again), but this time it bit me in the behind.

So what's the problem, you may ask?

Composition over inheritance

signald has many methods that require the "username" / the telephone number of the sender. It's a mandatory parameter when sending and receiving messages, adding, updating and deleting contacts and groups, etc.

So, it would make sense to define one SecureBaseMessage and make subclasses for each other message e.g

syntax = "proto2";


service signald {
    rpc AddContact(AddContactMessage) returns (Response);
    rpc DeleteContact(DeleteContactMessage) returns (Response);
}

message SecureBaseMessage {
    required string username = 1;
}

message Contact {
    required string phone_number = 1;
    required string name = 2;

}

message AddContactMessage {
    // Only supported in protobuf 2
    extend SecureBaseMessage {
        required Contact contact = 2;
    } 
}

message DeleteContactMessage {
    // Only supported in protobuf 2
    extend SecureBaseMessage {
        required string contact_number = 2;
    } 
}

That way, each service endpoint would have a specific type of message. But notice that the version in this example is version 2, which is the one being deprecated. 🙄Awesome🙄

The version 3 solution would be

syntax = "proto3";

service signald {
    rpc AddContact(InputMessage) returns (Response);
    rpc DeleteContact(InputMessage) returns (Response);
}

message InputMessage {
    string username = 1;
    oneof inner {
        AddContactMessage add_contact_message = 2;
        DeleteContactMessage delete_contact_message = 3;
    }
}

message Response {
    string reply = 1;
}

message Contact {
    required string phone_number = 1;
    required string name = 2;

}

message AddContactMessage {
    Contact contact = 1;
}

message DeleteContactMessage {
    string contact_number = 1;
}

There are multiple problems with that:

The API signature is basically the same for all messages requiring a username
Adding another message to InputMessage::inner has to be tacked on with a different ID and most likely not in alphabetical order (not that bad)
Removing a message means removing an ID from InputMessage::inner which protobuf really doesn't like
The generated code will require checking for the correct message type in inner, which does away with automatic input checking

For those reasons, gRPC fell through and another solution had to be found

Off to a new search

So now I'm back on the road again looking for an API generator for signald.

OpenAPI generator might be solution with some work. I only recently looked closer at the generated code and it uses networknt's light4j which is a framework written around undertow. It has a lot of functionality including input verification based on an openapi spec, but the doc is lacking and I'd have to dig into the code - a prospect I'm not really looking forward to after multiple weeks spent on this.

Another candidate might be Apache Thrift, which initially was a Facebook project but is now under the Apache Foundation's wing.


MRW facebook provides a lib

I'll try to put my prejudice aside and seriously consider it after playing around, but I guess that'll be it for now.

Until next time.

The Bug Factory🐞

What is signald?

The initial idea

Not so fast!

Picking a lightweight Java web server

Java's inbuilt HTTP server

Viable candidates

But wait, there's more

Fuck this, let's see what else there is

Enter gRPC

Not so fast!

Off to a new search