Having entity schemas defined in a central, language-neutral format
has many benefits as the scale of software engineering organizations increase. To do this, many organizations use Protocol Buffers as their interface definition language (IDL). In addition, gRPC,
a Protobuf-based RPC framework modeled after Google's internal Stubby is becoming increasingly popular due to its efficiency and code-generation capabilities.
Being an IDL, gRPC does not prescribe any specific guidelines on implementing the data access layer so implementations vary greatly. Ent is a natural candidate for building the data access layer in any Go application and so there is great potential in integrating the two technologies together.
Today we announce an experimental version of
entproto, a Go package, and a command-line tool to add Protobuf and gRPC support for ent users. With
entproto, developers can set up a fully working CRUD gRPC server in a few minutes. In this post, we will show exactly how to do just that.
The final version of this tutorial is available on GitHub, you can clone it if you prefer following along that way.
Let's start by initializing a new Go module for our project:
Next we use
go run to invoke the ent code generator to initialize a schema:
Our directory should now look like:
Next, let's add the
entproto package to our project:
Next, we will define the schema for the
User entity. Open
ent/schema/user.go and edit:
In this step, we added two unique fields to our
ent.Schema is just the definition of the schema, to create usable production code from it we need to run Ent's code generation tool on it. Run:
Notice the a bunch of new files were created from our schema definition now:
At this point, we can open a connection to a database, run a migration to create the
users table, and start reading and writing data to it. This is covered on the Setup Tutorial, so let's cut to the chase and learn about generating Protobuf definitions and gRPC servers from our schema.
Generating Go Protobufs with
As ent and Protobuf schemas are not identical, we must supply some annotations on our schema to help
entproto figure out exactly how to generate Protobuf definitions (called "Messages" in protobuf lingo).
The first thing we need to do is to add an
entproto.Message() annotation. This is our opt-in to Protobuf schema generation, we don't necessarily want to generate proto messages or gRPC service definitions from all of our schema entities, and this annotation gives us that control. To add it, append to
Next, we need to annotate each field and assign it a field number. Recall that when defining a protobuf message type, each field must be assigned a unique number. To do that, we add an
entproto.Field annotation on each field. Update the
Notice that we did not start our field numbers from 1, this is because
ent implicitly creates the
ID field for the entity, and that field is automatically assigned the number 1. We can now generate our protobuf message type definitions. To do that, we will add to
go:generate directive that invokes the
entproto command-line tool. It should now look like this:
Let's re-generate our code:
Observe that a new directory was created which will contain all protobuf related generated code:
ent/proto. It now contains:
Two files were created. Let's look at their contents:
Nice! A new
.proto file containing a message type definition that maps to our
User schema was created!
generate.go file was created with an invocation to
protoc, the protobuf code generator instructing it how to generate Go code from our
.proto file. For this command to work, we must first install
protoc as well as 3 protobuf plugins:
protoc-gen-go (which generates Go Protobuf structs),
protoc-gen-go-grpc (which generates Go gRPC service interfaces and clients), and
protoc-gen-entgrpc (which generates an implementation of the service interface). If you do not have these installed, please follow these directions:
After installing these dependencies, we can re-run code-generation:
Observe that a new file named
ent/proto/entpb/entpb.pb.go was created which contains the generated Go structs for our entities.
Let's write a test that uses it to make sure everything is wired correctly. Create a new file named
pb_test.go and write:
To run it:
Hooray! The test passes. We have successfully generated working Go Protobuf structs from our Ent schema. Next, let's see how to automatically generate a working CRUD gRPC server from our schema.
Generating a Fully Working gRPC Server from our Schema
Having Protobuf structs generated from our
ent.Schema can be useful, but what we're really interested in is getting an actual server that can create, read, update, and delete entities from an actual database. To do that, we need to update just one line of code! When we annotate a schema with
entproto.Service, we tell the
entproto code-gen that we are interested in generating a gRPC service definition, from the
protoc-gen-entgrpc will read our definition and generate a service implementation. Edit
ent/schema/user.go and modify the schema's
Now re-run code-generation:
Observe some interesting changes in
entproto added a service definition to
In addition, two new files were created. The first,
ent_grpc.pb.go, contains the gRPC client stub and the interface definition. If you open the file, you will find in it (among many other things):
The second file,
entpub_user_service.go contains a generated implementation for this interface. For example, an implementation for the
Not bad! Next, let's create a gRPC server that can serve requests to our service.
Creating the Server
Create a new file
cmd/server/main.go and write:
Notice that we added an import of
github.com/mattn/go-sqlite3, so we need to add it to our module:
Next, let's run the server, while we write a client that will communicate with it:
Creating the Client
Let's create a simple client that will make some calls to our server. Create a new file named
cmd/client/main.go and write:
Our client creates a connection to port 5000, where our server is listening, then issues a
Create request to create a new user, and then issues a second
Get request to retrieve it from the database. Let's run our client code:
Observe the output:
Amazing! With a few annotations on our schema, we used the super-powers of code generation to create a working gRPC server in no time!
Caveats and Limitations
entproto is still experimental stage and lacks some basic functionality. For example, many applications will probably want a
Find method on their service, but these are not yet supported. In addition, some other issues we plan to tackle in the near future:
- Currently only "unique" edges are supported (O2O, O2M).
- The generated "mutating" methods (Create/Update) currently set all fields, disregarding zero/null values and field nullability.
- All fields are copied from the gRPC request to the ent client, support for configuring some fields to be unsettable via the service by adding a field/edge annotation is also planned.
We believe that
ent + gRPC can be a great way to build server applications in Go. For example, to set granular access control to the entities managed by our application, developers can already use Privacy Policies that work out-of-the-box with the gRPC integration. To run any arbitrary Go code on the different lifecycle events of entities, developers can utilize custom Hooks.
Do you want to build gRPC servers with
ent? If you want some help setting up or want the integration to support your use case, please reach out to us via our Discussions Page on GitHub or in the #ent channel on the Gophers Slack.
For more Ent news and updates: