Skip to main content

Β· 5 min read

Joining an existing project with a large codebase can be a daunting task.

Understanding the data model of an application is key for developers to start working on an existing project. One commonly used tool to help overcome this challenge, and enable developers to grasp an application's data model is an ER (Entity Relation) diagram.

ER diagrams provide a visual representation of your data model, and details each field of the entities. Many tools can help create these, where one example is Jetbrains DataGrip, that can generate an ER diagram by connecting to and inspecting an existing database:

Datagrip ER diagram

DataGrip ER diagram example

Ent, a simple, yet powerful entity framework for Go, was originally developed inside Facebook specifically for dealing with projects with large and complex data models. This is why Ent uses code generation - it gives type-safety and code-completion out-of-the-box which helps explain the data model and improves developer velocity. On top of all of this, wouldn't it be great to automatically generate ER diagrams that maintain a high-level view of the data model in a visually appealing representation? (I mean, who doesn't love visualizations?)

Introducing entviz

entviz is an ent extension that automatically generates a static HTML page that visualizes your data graph.

Entviz example output

Entviz example output

Most ER diagram generation tools need to connect to your database and introspect it, which makes it harder to maintain an up-to-date diagram of the database schema. Since entviz integrates directly to your Ent schema, it does not need to connect to your database, and it automatically generates fresh visualization every time you modify your schema.

If you want to know more about how entviz was implemented, checkout the implementation section.

See it in action

First, let's add the entviz extension to our entc.go file:

go get github.com/hedwigz/entviz
info

If you are not familiar with entc you're welcome to read entc documentation to learn more about it.

ent/entc.go
import (
"log"

"entgo.io/ent/entc"
"entgo.io/ent/entc/gen"
"github.com/hedwigz/entviz"
)

func main() {
err := entc.Generate("./schema", &gen.Config{}, entc.Extensions(entviz.Extension{}))
if err != nil {
log.Fatalf("running ent codegen: %v", err)
}
}

Let's say we have a simple schema with a user entity and some fields:

ent/schema/user.go
// Fields of the User.
func (User) Fields() []ent.Field {
return []ent.Field{
field.String("name"),
field.String("email"),
field.Time("created").
Default(time.Now),
}
}

Now, entviz will automatically generate a visualization of our graph everytime we run:

go generate ./...

You should now see a new file called schema-viz.html in your ent directory:

$ ll ./ent/schema-viz.html
-rw-r--r-- 1 hedwigz hedwigz 7.3K Aug 27 09:00 schema-viz.html

Open the html file with your favorite browser to see the visualization

tutorial image

Next, let's add another entity named Post, and see how our visualization changes:

ent init Post
ent/schema/post.go
// Fields of the Post.
func (Post) Fields() []ent.Field {
return []ent.Field{
field.String("content"),
field.Time("created").
Default(time.Now),
}
}

Now we add an (O2M) edge from User to Post:

ent/schema/post.go
// Edges of the User.
func (User) Edges() []ent.Edge {
return []ent.Edge{
edge.To("posts", Post.Type),
}
}

Finally, regenerate the code:

go generate ./...

Refresh your browser to see the updated result!

tutorial image 2

Implementation

Entviz was implemented by extending ent via its extension API. The Ent extension API lets you aggregate multiple templates, hooks, options and annotations. For instance, entviz uses templates to add another go file, entviz.go, which exposes the ServeEntviz method that can be used as an http handler, like so:

func main() {
http.ListenAndServe("localhost:3002", ent.ServeEntviz())
}

We define an extension struct which embeds the default extension, and we export our template via the Templates method:

//go:embed entviz.go.tmpl
var tmplfile string

type Extension struct {
entc.DefaultExtension
}

func (Extension) Templates() []*gen.Template {
return []*gen.Template{
gen.MustParse(gen.NewTemplate("entviz").Parse(tmplfile)),
}
}

The template file is the code that we want to generate:

{{ define "entviz"}}

{{ $pkg := base $.Config.Package }}
{{ template "header" $ }}
import (
_ "embed"
"net/http"
"strings"
"time"
)

//go:embed schema-viz.html
var html string

func ServeEntviz() http.Handler {
generateTime := time.Now()
return http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) {
http.ServeContent(w, req, "schema-viz.html", generateTime, strings.NewReader(html))
})
}
{{ end }}

That's it! now we have a new method in ent package.

Wrapping-Up

We saw how ER diagrams help developers keep track of their data model. Next, we introduced entviz - an Ent extension that automatically generates an ER diagram for Ent schemas. We saw how entviz utilizes Ent's extension API to extend the code generation and add extra functionality. Finally, you got to see it in action by installing and use entviz in your own project. If you like the code and/or want to contribute - feel free to checkout the project on github.

Have questions? Need help with getting started? Feel free to join our Slack channel.

For more Ent news and updates:

Β· 10 min read

Observability is a quality of a system that refers to how well its internal state can be measured externally. As a computer program evolves into a full-blown production system this quality becomes increasingly important. One of the ways to make a software system more observable is to export metrics, that is, to report in some externally visible way a quantitative description of the running system's state. For instance, to expose an HTTP endpoint where we can see how many errors occurred since the process has started. In this post, we will explore how to build more observable Ent applications using Prometheus.

What is Ent?

Ent, is a simple, yet powerful entity framework for Go, that makes it easy to build and maintain applications with large data models.

What is Prometheus?

Prometheus is an open source monitoring system developed by engineering at SoundCloud in 2012. It includes an embedded time series database and many integrations to third-party systems. The Prometheus client exposes the process's metrics via an HTTP endpoint (usually /metrics), this endpoint is discovered by the Prometheus scraper which polls the endpoint every interval (typically 30s) and writes it into a time-series database.

Prometheus is just an example of a class of metric collection backends. Many others, such as AWS CloudWatch, InfluxDB and others exist and are in wide use in the industry. Towards the end of this post, we will discuss a possible path to a unified, standards-based integration with any such backend.

Working with Prometheus

To expose an application's metrics using Prometheus, we need to create a Prometheus Collector, a collector collects a set of metrics from your server.

In our example, we will be using two types of metrics that can be stored in a collector: Counters and Histograms. Counters are monotonically increasing cumulative metrics that represent how many times something has happened, commonly used to count the number of requests a server has processed or errors that have occurred. Histograms sample observations into buckets of configurable sizes and are commonly used to represent latency distributions (i.e how many requests returned in under 5ms, 10ms, 100ms, 1s, etc.) In addition, Prometheus allows metrics to be broken down into labels. This is useful for example for counting requests but breaking down the counter by endpoint name.

Let’s see how to create such a collector using the official Go client. To do so, we will use a package in the client called promauto that simplifies the processes of creating collectors. A simple example of a collector that counts (for example, total request or number or request error):

package example

import (
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/promauto"
)

var (
// List of dynamic labels
labelNames = []string{"endpoint", "error_code"}

// Create a counter collector
exampleCollector = promauto.NewCounterVec(
prometheus.CounterOpts{
Name: "endpoint_errors",
Help: "Number of errors in endpoints",
},
labelNames,
)
)

// When using you set the values of the dynamic labels and then increment the counter
func incrementError() {
exampleCollector.WithLabelValues("/create-user", "400").Inc()
}

Ent Hooks

Hooks are a feature of Ent that allows adding custom logic before and after operations that change the data entities.

A mutation is an operation that changes something in the database. There are 5 types of mutations:

  1. Create.
  2. UpdateOne.
  3. Update.
  4. DeleteOne.
  5. Delete.

Hooks are functions that get an ent.Mutator and return a mutator back. They function similar to the popular HTTP middleware pattern.

package example

import (
"context"

"entgo.io/ent"
)

func exampleHook() ent.Hook {
//use this to init your hook
return func(next ent.Mutator) ent.Mutator {
return ent.MutateFunc(func(ctx context.Context, m ent.Mutation) (ent.Value, error) {
// Do something before mutation.
v, err := next.Mutate(ctx, m)
if err != nil {
// Do something if error after mutation.
}
// Do something after mutation.
return v, err
})
}
}

In Ent, there are two types of mutation hooks - schema hooks and runtime hooks. Schema hooks are mainly used for defining custom mutation logic on a specific entity type, for example, syncing entity creation to another system. Runtime hooks, on the other hand, are used to define more global logic for adding things like logging, metrics, tracing, etc.

For our use case, we should definitely use runtime hooks, because to be valuable we want to export metrics on all operations on all entity types:

package example

import (
"entprom/ent"
"entprom/ent/hook"
)

func main() {
client, _ := ent.Open("sqlite3", "file:ent?mode=memory&cache=shared&_fk=1")

// Add a hook only on user mutations.
client.User.Use(exampleHook())

// Add a hook only on update operations.
client.Use(hook.On(exampleHook(), ent.OpUpdate|ent.OpUpdateOne))
}

Exporting Prometheus Metrics for an Ent Application

With all of the introductions complete, let’s cut to the chase and show how to use Prometheus and Ent hooks together to create an observable application. Our goal with this example is to export these metrics using a hook:

Metric NameDescription
ent_operation_totalNumber of ent mutation operations
ent_operation_errorNumber of failed ent mutation operations
ent_operation_duration_secondsTime in seconds per operation

Each of these metrics will be broken down by labels into two dimensions:

  • mutation_type: Entity type that is being mutated (User, BlogPost, Account etc.).
  • mutation_op: The operation that is being performed (Create, Delete etc.).

Let’s start by defining our collectors:

//Ent dynamic dimensions
const (
mutationType = "mutation_type"
mutationOp = "mutation_op"
)

var entLabels = []string{mutationType, mutationOp}

// Create a collector for total operations counter
func initOpsProcessedTotal() *prometheus.CounterVec {
return promauto.NewCounterVec(
prometheus.CounterOpts{
Name: "ent_operation_total",
Help: "Number of ent mutation operations",
},
entLabels,
)
}

// Create a collector for error counter
func initOpsProcessedError() *prometheus.CounterVec {
return promauto.NewCounterVec(
prometheus.CounterOpts{
Name: "ent_operation_error",
Help: "Number of failed ent mutation operations",
},
entLabels,
)
}

// Create a collector for duration histogram collector
func initOpsDuration() *prometheus.HistogramVec {
return promauto.NewHistogramVec(
prometheus.HistogramOpts{
Name: "ent_operation_duration_seconds",
Help: "Time in seconds per operation",
},
entLabels,
)
}

Next, let’s define our new hook:

// Hook init collectors, count total at beginning error on mutation error and duration also after.
func Hook() ent.Hook {
opsProcessedTotal := initOpsProcessedTotal()
opsProcessedError := initOpsProcessedError()
opsDuration := initOpsDuration()
return func(next ent.Mutator) ent.Mutator {
return ent.MutateFunc(func(ctx context.Context, m ent.Mutation) (ent.Value, error) {
// Before mutation, start measuring time.
start := time.Now()
// Extract dynamic labels from mutation.
labels := prometheus.Labels{mutationType: m.Type(), mutationOp: m.Op().String()}
// Increment total ops counter.
opsProcessedTotal.With(labels).Inc()
// Execute mutation.
v, err := next.Mutate(ctx, m)
if err != nil {
// In case of error increment error counter.
opsProcessedError.With(labels).Inc()
}
// Stop time measure.
duration := time.Since(start)
// Record duration in seconds.
opsDuration.With(labels).Observe(duration.Seconds())
return v, err
})
}
}

Connecting the Prometheus Collector to our Service

After defining our hook, let’s see next how to connect it to our application and how to use Prometheus to serve an endpoint that exposes the metrics in our collectors:

package main

import (
"context"
"log"
"net/http"

"entprom"
"entprom/ent"

_ "github.com/mattn/go-sqlite3"
"github.com/prometheus/client_golang/prometheus/promhttp"
)

func createClient() *ent.Client {
c, err := ent.Open("sqlite3", "file:ent?mode=memory&cache=shared&_fk=1")
if err != nil {
log.Fatalf("failed opening connection to sqlite: %v", err)
}
ctx := context.Background()
// Run the auto migration tool.
if err := c.Schema.Create(ctx); err != nil {
log.Fatalf("failed creating schema resources: %v", err)
}
return c
}

func handler(client *ent.Client) func(w http.ResponseWriter, r *http.Request) {
return func(w http.ResponseWriter, r *http.Request) {
ctx := context.Background()
// Run operations.
_, err := client.User.Create().SetName("a8m").Save(ctx)
if err != nil {
http.Error(w, err.Error(), http.StatusBadRequest)
return
}
}
}

func main() {
// Create Ent client and migrate
client := createClient()
// Use the hook
client.Use(entprom.Hook())
// Simple handler to run actions on our DB.
http.HandleFunc("/", handler(client))
// This endpoint sends metrics to the prometheus to collect
http.Handle("/metrics", promhttp.Handler())
log.Println("server starting on port 8080")
// Run the server
log.Fatal(http.ListenAndServe(":8080", nil))
}

After a few times of accessing / on our server (using curl or a browser), go to /metrics. There you will see the output from the Prometheus client:

# HELP ent_operation_duration_seconds Time in seconds per operation
# TYPE ent_operation_duration_seconds histogram
ent_operation_duration_seconds_bucket{mutation_op="OpCreate",mutation_type="User",le="0.005"} 2
ent_operation_duration_seconds_bucket{mutation_op="OpCreate",mutation_type="User",le="0.01"} 2
ent_operation_duration_seconds_bucket{mutation_op="OpCreate",mutation_type="User",le="0.025"} 2
ent_operation_duration_seconds_bucket{mutation_op="OpCreate",mutation_type="User",le="0.05"} 2
ent_operation_duration_seconds_bucket{mutation_op="OpCreate",mutation_type="User",le="0.1"} 2
ent_operation_duration_seconds_bucket{mutation_op="OpCreate",mutation_type="User",le="0.25"} 2
ent_operation_duration_seconds_bucket{mutation_op="OpCreate",mutation_type="User",le="0.5"} 2
ent_operation_duration_seconds_bucket{mutation_op="OpCreate",mutation_type="User",le="1"} 2
ent_operation_duration_seconds_bucket{mutation_op="OpCreate",mutation_type="User",le="2.5"} 2
ent_operation_duration_seconds_bucket{mutation_op="OpCreate",mutation_type="User",le="5"} 2
ent_operation_duration_seconds_bucket{mutation_op="OpCreate",mutation_type="User",le="10"} 2
ent_operation_duration_seconds_bucket{mutation_op="OpCreate",mutation_type="User",le="+Inf"} 2
ent_operation_duration_seconds_sum{mutation_op="OpCreate",mutation_type="User"} 0.000265669
ent_operation_duration_seconds_count{mutation_op="OpCreate",mutation_type="User"} 2
# HELP ent_operation_error Number of failed ent mutation operations
# TYPE ent_operation_error counter
ent_operation_error{mutation_op="OpCreate",mutation_type="User"} 1
# HELP ent_operation_total Number of ent mutation operations
# TYPE ent_operation_total counter
ent_operation_total{mutation_op="OpCreate",mutation_type="User"} 2

In the top part, we can see the histogram calculated, it calculates the number of operations in each β€œbucket”. After that, we can see the number of total operations and the number of errors. Each metric is followed by its description that can be seen when querying with Prometheus dashboard.

The Prometheus client is only one component of the Prometheus architecture. To run a complete system including a scraper that will poll your endpoint, a Prometheus that will store your metrics and can answer queries, and a simple UI to interact with it, I recommend reading the official documentation or use the docker-compose.yaml in this example repo.

Future Work on Observability in Ent

As we’ve mentioned above, there is an abundance of metric collections backends available today, Prometheus being just one of many successful projects. While these solutions differ in many dimensions (self-hosted vs SaaS, different storage engines with different query languages, and more) - from the metric reporting client perspective, they are virtually identical.

In cases like these, good software engineering principles suggest that the concrete backend should be abstracted away from the client using an interface. This interface can then be implemented by backends so client applications can easily switch between the different implementations. Such changes are happening in recent years in our industry. Consider, for example, the Open Container Initiative or the Service Mesh Interface: both are initiatives that strive to define a standard interface for a problem space. This interface is supposed to create an ecosystem of implementations of the standard. In the observability space, the exact same convergence is occurring with OpenCensus and OpenTracing currently merging into OpenTelemetry.

As nice as it would be to publish an Ent + Prometheus extension similar to the one presented in this post, we are firm believers that observability should be solved with a standards-based approach. We invite everyone to join the discussion on what is the right way to do this for Ent.

Wrap-Up

We started this post by presenting Prometheus, a popular open-source monitoring solution. Next, we reviewed β€œHooks”, a feature of Ent that allows adding custom logic before and after operations that change the data entities. We then showed how to integrate the two to create observable applications using Ent. Finally, we discussed the future of observability in Ent and invited everyone to join the discussion to shape it.

Have questions? Need help with getting started? Feel free to join our Slack channel.

For more Ent news and updates:

Β· 7 min read

It has been almost 4 months since our last release, and for a good reason. Version 0.9.0 which was released today is packed with some highly-anticipated features. Perhaps at the top of the list, is a feature that has been in discussion for more than a year in a half and was one of the most commonly requested features in the Ent User Survey: the Upsert API!

Version 0.9.0 adds support for "Upsert" style statements using a new feature flag: sql/upsert. Ent has a collection of feature flags that can be switched on to add more features to the code generated by Ent. This is used as both a mechanism to allow opt-in to some features that are not necessarily desired in every project and as a way to run experiments of features that may one day become part of Ent's core.

In this post, we will introduce the new feature, the places where it is useful, and demonstrate how to use it.

Upsert

"Upsert" is a commonly-used term in data systems that is a portmanteau of "update" and "insert" which usually refers to a statement that attempts to insert a record to a table, and if a uniqueness constraint is violated (e.g. a record by that ID already exists) that record is updated instead. While none of the popular relational databases have a specific UPSERT statement, most of them support ways of achieving this type of behavior.

For example, assume we have a table with this definition in an SQLite database:

CREATE TABLE users (
id integer PRIMARY KEY AUTOINCREMENT,
email varchar(255) UNIQUE,
name varchar(255)
)

If we try to execute the same insert twice:

INSERT INTO users (email, name) VALUES ('rotem@entgo.io', 'Rotem Tamir');
INSERT INTO users (email, name) VALUES ('rotem@entgo.io', 'Rotem Tamir');

We get this error:

[2021-08-05 06:49:22] UNIQUE constraint failed: users.email

In many cases, it is useful to have write operations be idempotent, meaning we can run them many times in a row while leaving the system in the same state.

In other cases, it is not desirable to query if a record exists before trying to create it. For these kinds of situations, SQLite supports the ON CONFLICT clause in INSERT statements. To instruct SQLite to override an existing value with the new one we can execute:

INSERT INTO users (email, name) values ('rotem@entgo.io', 'Tamir, Rotem')
ON CONFLICT (email) DO UPDATE SET email=excluded.email, name=excluded.name;

If we prefer to keep the existing values, we can use the DO NOTHING conflict action:

INSERT INTO users (email, name) values ('rotem@entgo.io', 'Tamir, Rotem') 
ON CONFLICT DO NOTHING;

Sometimes we want to merge the two versions in some way, we can use the DO UPDATE action a little differently to achieve do something like:

INSERT INTO users (email, full_name) values ('rotem@entgo.io', 'Tamir, Rotem') 
ON CONFLICT (email) DO UPDATE SET name=excluded.name || ' (formerly: ' || users.name || ')'

In this case, after our second INSERT the value for the name column would be: Tamir, Rotem (formerly: Rotem Tamir). Not very useful, but hopefully you can see that you can do cool things this way.

Upsert with Ent

Assume we have an existing Ent project with an entity similar to the users table described above:

// User holds the schema definition for the User entity.
type User struct {
ent.Schema
}

// Fields of the User.
func (User) Fields() []ent.Field {
return []ent.Field{
field.String("email").
Unique(),
field.String("name"),
}
}

As the Upsert API is a newly released feature, make sure to update your ent version using:

go get -u entgo.io/ent@v0.9.0

Next, add the sql/upsert feature flag to your code-generation flags, in ent/generate.go:

package ent

//go:generate go run -mod=mod entgo.io/ent/cmd/ent generate --feature sql/upsert ./schema

Next, re-run code generation for your project:

go generate ./...

Observe that a new method named OnConflict was added to the ent/user_create.go file:

// OnConflict allows configuring the `ON CONFLICT` / `ON DUPLICATE KEY` clause
// of the `INSERT` statement. For example:
//
// client.User.Create().
// SetEmailAddress(v).
// OnConflict(
// // Update the row with the new values
// // the was proposed for insertion.
// sql.ResolveWithNewValues(),
// ).
// // Override some of the fields with custom
// // update values.
// Update(func(u *ent.UserUpsert) {
// SetEmailAddress(v+v)
// }).
// Exec(ctx)
//
func (uc *UserCreate) OnConflict(opts ...sql.ConflictOption) *UserUpsertOne {
uc.conflict = opts
return &UserUpsertOne{
create: uc,
}
}

This (along with more new generated code) will serve us in achieving upsert behavior for our User entity. To explore this, let's first start by writing a test to reproduce the uniqueness constraint error:

func TestUniqueConstraintFails(t *testing.T) {
client := enttest.Open(t, "sqlite3", "file:ent?mode=memory&cache=shared&_fk=1")
ctx := context.TODO()

// Create the user for the first time.
client.User.
Create().
SetEmail("rotem@entgo.io").
SetName("Rotem Tamir").
SaveX(ctx)

// Try to create a user with the same email the second time.
_, err := client.User.
Create().
SetEmail("rotem@entgo.io").
SetName("Rotem Tamir").
Save(ctx)

if !ent.IsConstraintError(err) {
log.Fatalf("expected second created to fail with constraint error")
}
log.Printf("second query failed with: %v", err)
}

The test passes:

=== RUN   TestUniqueConstraintFails
2021/08/05 07:12:11 second query failed with: ent: constraint failed: insert node to table "users": UNIQUE constraint failed: users.email
--- PASS: TestUniqueConstraintFails (0.00s)

Next, let's see how to instruct Ent to override the existing values with the new in case a conflict occurs:

func TestUpsertReplace(t *testing.T) {
client := enttest.Open(t, "sqlite3", "file:ent?mode=memory&cache=shared&_fk=1")
ctx := context.TODO()

// Create the user for the first time.
orig := client.User.
Create().
SetEmail("rotem@entgo.io").
SetName("Rotem Tamir").
SaveX(ctx)

// Try to create a user with the same email the second time.
// This time we set ON CONFLICT behavior, and use the `UpdateNewValues`
// modifier.
newID := client.User.Create().
SetEmail("rotem@entgo.io").
SetName("Tamir, Rotem").
OnConflict().
UpdateNewValues().
// we use the IDX method to receive the ID
// of the created/updated entity
IDX(ctx)

// We expect the ID of the originally created user to be the same as
// the one that was just updated.
if orig.ID != newID {
log.Fatalf("expected upsert to update an existing record")
}

current := client.User.GetX(ctx, orig.ID)
if current.Name != "Tamir, Rotem" {
log.Fatalf("expected upsert to replace with the new values")
}
}

Running our test:

=== RUN   TestUpsertReplace
--- PASS: TestUpsertReplace (0.00s)

Alternatively, we can use the Ignore modifier to instruct Ent to keep the old version when resolving the conflict. Let's write a test that shows this:

func TestUpsertIgnore(t *testing.T) {
client := enttest.Open(t, "sqlite3", "file:ent?mode=memory&cache=shared&_fk=1")
ctx := context.TODO()

// Create the user for the first time.
orig := client.User.
Create().
SetEmail("rotem@entgo.io").
SetName("Rotem Tamir").
SaveX(ctx)

// Try to create a user with the same email the second time.
// This time we set ON CONFLICT behavior, and use the `Ignore`
// modifier.
client.User.
Create().
SetEmail("rotem@entgo.io").
SetName("Tamir, Rotem").
OnConflict().
Ignore().
ExecX(ctx)

current := client.User.GetX(ctx, orig.ID)
if current.FullName != orig.FullName {
log.Fatalf("expected upsert to keep the original version")
}
}

You can read more about the feature in the Feature Flag or Upsert API documentation.

Wrapping Up

In this post, we presented the Upsert API, a long-anticipated capability, that is available by feature-flag in Ent v0.9.0. We discussed where upserts are commonly used in applications and the way they are implemented using common relational databases. Finally, we showed a simple example of how to get started with the Upsert API using Ent.

Have questions? Need help with getting started? Feel free to join our Slack channel.

For more Ent news and updates:

Β· 11 min read

When we say that one of the core principles of Ent is "Schema as Code", we mean by that more than "Ent's DSL for defining entities and their edges is done using regular Go code". Ent's unique approach, compared to many other ORMs, is to express all of the logic related to an entity, as code, directly in the schema definition.

With Ent, developers can write all authorization logic (called "Privacy" within Ent), and all of the mutation side-effects (called "Hooks" within Ent) directly on the schema. Having everything in the same place can be very convenient, but its true power is revealed when paired with code generation.

If schemas are defined this way, it becomes possible to generate code for fully-working production-grade servers automatically. If we move the responsibility for authorization decisions and custom side effects from the RPC layer to the data layer, the implementation of the basic CRUD (Create, Read, Update and Delete) endpoints becomes generic to the extent that it can be machine-generated. This is exactly the idea behind the popular GraphQL and gRPC Ent extensions.

Today, we would like to present a new Ent extension named elk that can automatically generate fully-working, RESTful API endpoints from your Ent schemas. elk strives to automate all of the tedious work of setting up the basic CRUD endpoints for every entity you add to your graph, including logging, validation of the request body, eager loading relations and serializing, all while leaving reflection out of sight and maintaining type-safety.

Let’s get started!

Getting Started

The final version of the code below can be found on GitHub.

Start by creating a new Go project:

mkdir elk-example
cd elk-example
go mod init elk-example

Invoke the ent code generator and create two schemas: User, Pet:

go run -mod=mod entgo.io/ent/cmd/ent init Pet User

Your project should now look like this:

.
β”œβ”€β”€ ent
β”‚ β”œβ”€β”€ generate.go
β”‚ └── schema
β”‚ β”œβ”€β”€ pet.go
β”‚ └── user.go
β”œβ”€β”€ go.mod
└── go.sum

Next, add the elk package to our project:

go get -u github.com/masseelch/elk

elk uses the Ent extension API to integrate with Ent’s code-generation. This requires that we use the entc (ent codegen) package as described here. Follow the next three steps to enable it and to configure Ent to work with the elk extension:

1. Create a new Go file named ent/entc.go and paste the following content:

// +build ignore

package main

import (
"log"

"entgo.io/ent/entc"
"entgo.io/ent/entc/gen"
"github.com/masseelch/elk"
)

func main() {
ex, err := elk.NewExtension(
elk.GenerateSpec("openapi.json"),
elk.GenerateHandlers(),
)
if err != nil {
log.Fatalf("creating elk extension: %v", err)
}
err = entc.Generate("./schema", &gen.Config{}, entc.Extensions(ex))
if err != nil {
log.Fatalf("running ent codegen: %v", err)
}
}

2. Edit the ent/generate.go file to execute the ent/entc.go file:

package ent

//go:generate go run -mod=mod entc.go

3/. elk uses some external packages in its generated code. Currently, you have to get those packages manually once when setting up elk:

go get github.com/mailru/easyjson github.com/masseelch/render github.com/go-chi/chi/v5 go.uber.org/zap

With these steps complete, all is set up for using our elk-powered ent! To learn more about Ent, how to connect to different types of databases, run migrations or work with entities head over to the Setup Tutorial.

Generating HTTP CRUD Handlers with elk

To generate the fully-working HTTP handlers we need first create an Ent schema definition. Open and edit ent/schema/pet.go:

package schema

import (
"entgo.io/ent"
"entgo.io/ent/schema/field"
)

// Pet holds the schema definition for the Pet entity.
type Pet struct {
ent.Schema
}

// Fields of the Pet.
func (Pet) Fields() []ent.Field {
return []ent.Field{
field.String("name"),
field.Int("age"),
}
}

We added two fields to our Pet entity: name and age. The ent.Schema just defines the fields of our entity. To generate runnable code from our schema, run:

go generate ./...

Observe that in addition to the files Ent would normally generate, another directory named ent/http was created. These files were generated by the elk extension and contain the code for the generated HTTP handlers. For example, here is some of the generated code for a read-operation on the Pet entity:

const (
PetCreate Routes = 1 << iota
PetRead
PetUpdate
PetDelete
PetList
PetRoutes = 1<<iota - 1
)

// PetHandler handles http crud operations on ent.Pet.
type PetHandler struct {
handler

client *ent.Client
log *zap.Logger
}

func NewPetHandler(c *ent.Client, l *zap.Logger) *PetHandler {
return &PetHandler{
client: c,
log: l.With(zap.String("handler", "PetHandler")),
}
}

// Read fetches the ent.Pet identified by a given url-parameter from the
// database and renders it to the client.
func (h *PetHandler) Read(w http.ResponseWriter, r *http.Request) {
l := h.log.With(zap.String("method", "Read"))
// ID is URL parameter.
id, err := strconv.Atoi(chi.URLParam(r, "id"))
if err != nil {
l.Error("error getting id from url parameter", zap.String("id", chi.URLParam(r, "id")), zap.Error(err))
render.BadRequest(w, r, "id must be an integer greater zero")
return
}
// Create the query to fetch the Pet
q := h.client.Pet.Query().Where(pet.ID(id))
e, err := q.Only(r.Context())
if err != nil {
switch {
case ent.IsNotFound(err):
msg := stripEntError(err)
l.Info(msg, zap.Error(err), zap.Int("id", id))
render.NotFound(w, r, msg)
case ent.IsNotSingular(err):
msg := stripEntError(err)
l.Error(msg, zap.Error(err), zap.Int("id", id))
render.BadRequest(w, r, msg)
default:
l.Error("could not read pet", zap.Error(err), zap.Int("id", id))
render.InternalServerError(w, r, nil)
}
return
}
l.Info("pet rendered", zap.Int("id", id))
easyjson.MarshalToHTTPResponseWriter(NewPet2657988899View(e), w)
}

Next, let’s see how to create an actual RESTful HTTP server that can manage your Pet entities. Create a file named main.go and add the following content:

package main

import (
"context"
"fmt"
"log"
"net/http"

"elk-example/ent"
elk "elk-example/ent/http"

"github.com/go-chi/chi/v5"
_ "github.com/mattn/go-sqlite3"
"go.uber.org/zap"
)

func main() {
// Create the ent client.
c, err := ent.Open("sqlite3", "./ent.db?_fk=1")
if err != nil {
log.Fatalf("failed opening connection to sqlite: %v", err)
}
defer c.Close()
// Run the auto migration tool.
if err := c.Schema.Create(context.Background()); err != nil {
log.Fatalf("failed creating schema resources: %v", err)
}
// Router and Logger.
r, l := chi.NewRouter(), zap.NewExample()
// Create the pet handler.
r.Route("/pets", func(r chi.Router) {
elk.NewPetHandler(c, l).Mount(r, elk.PetRoutes)
})
// Start listen to incoming requests.
fmt.Println("Server running")
defer fmt.Println("Server stopped")
if err := http.ListenAndServe(":8080", r); err != nil {
log.Fatal(err)
}
}

Next, start the server:

go run -mod=mod main.go

Congratulations! We now have a running server serving the Pets API. We could ask the server for a list of all pets in the database, but there are none yet. Let’s create one first:

curl -X 'POST' -H 'Content-Type: application/json' -d '{"name":"Kuro","age":3}' 'localhost:8080/pets'

You should get this response:

{
"age": 3,
"id": 1,
"name": "Kuro"
}

If you head over to the terminal where the server is running you can also see elks built in logging:

{
"level": "info",
"msg": "pet rendered",
"handler": "PetHandler",
"method": "Create",
"id": 1
}

elk uses zap for logging. To learn more about it, have a look at its documentation.

Relations

To illustrate more of elks features, let’s extend our graph. Edit ent/schema/user.go and ent/schema/pet.go:

ent/schema/pet.go
// Edges of the Pet.
func (Pet) Edges() []ent.Edge {
return []ent.Edge{
edge.From("owner", User.Type).
Ref("pets").
Unique(),
}
}

ent/schema/user.go
package schema

import (
"entgo.io/ent"
"entgo.io/ent/schema/edge"
"entgo.io/ent/schema/field"
)

// User holds the schema definition for the User entity.
type User struct {
ent.Schema
}

// Fields of the User.
func (User) Fields() []ent.Field {
return []ent.Field{
field.String("name"),
field.Int("age"),
}
}

// Edges of the User.
func (User) Edges() []ent.Edge {
return []ent.Edge{
edge.To("pets", Pet.Type),
}
}

We have now created a One-To-Many relation between the Pet and User schemas: A pet belongs to a user, and a user can have multiple pets.

Rerun the code generator:

go generate ./...

Do not forget to register the UserHandler on our router. Just add the following lines to main.go:

[...]
r.Route("/pets", func(r chi.Router) {
elk.NewPetHandler(c, l, v).Mount(r, elk.PetRoutes)
})
+ // Create the user handler.
+ r.Route("/users", func(r chi.Router) {
+ elk.NewUserHandler(c, l, v).Mount(r, elk.UserRoutes)
+ })
// Start listen to incoming requests.
fmt.Println("Server running")
[...]

After restarting the server we can create a User that owns the previously created Pet named Kuro:

curl -X 'POST' -H 'Content-Type: application/json' -d '{"name":"Elk","age":30,"owner":1}' 'localhost:8080/users'

The server returns the following response:

{
"age": 30,
"edges": {},
"id": 1,
"name": "Elk"
}

From the output we can see that the user has been created, but the edges are empty. elk does not include edges in its output by default. You can configure elk to render edges using a feature called "serialization groups". Annotate your schemas with the elk.SchemaAnnotation and elk.Annotation structs. Edit ent/schema/user.go and add those:

// Edges of the User.
func (User) Edges() []ent.Edge {
return []ent.Edge{
edge.To("pets", Pet.Type).
Annotations(elk.Groups("user")),
}
}

// Annotations of the User.
func (User) Annotations() []schema.Annotation {
return []schema.Annotation{elk.ReadGroups("user")}
}

The elk.Annotations added to the fields and edges tell elk to eager-load them and add them to the payload if the " user" group is requested. The elk.SchemaAnnotation is used to make the read-operation of the UserHandler request " user". Note, that any fields that do not have a serialization group attached are included by default. Edges, however, are excluded, unless configured otherwise.

Next, let’s regenerate the code once again, and restart the server. You should now see the pets of a user rendered if you read a resource:

curl 'localhost:8080/users/1'
{
"age": 30,
"edges": {
"pets": [
{
"id": 1,
"name": "Kuro",
"age": 3,
"edges": {}
}
]
},
"id": 1,
"name": "Elk"
}

Request validation

Our current schemas allow to set a negative age for pets or users and we can create pets without an owner (as we did with Kuro). Ent has built-in support for basic validation. In some cases you may want to validate requests made against your API before passing their payload to Ent. elk uses this package to define validation rules and validate data. We can create separate validation rules for Create and Update operations using elk.Annotation. In our example, let’s assume that we want our Pet schema to only allow ages greater than zero and to disallow creating a pet without an owner. Edit ent/schema/pet.go:

// Fields of the Pet.
func (Pet) Fields() []ent.Field {
return []ent.Field{
field.String("name"),
field.Int("age").
Positive().
Annotations(
elk.CreateValidation("required,gt=0"),
elk.UpdateValidation("gt=0"),
),
}
}

// Edges of the Pet.
func (Pet) Edges() []ent.Edge {
return []ent.Edge{
edge.From("owner", User.Type).
Ref("pets").
Unique().
Required().
Annotations(elk.Validation("required")),
}
}

Next, regenerate the code and restart the server. To test our new validation rules, let’s try to create a pet with invalid age and without an owner:

curl -X 'POST' -H 'Content-Type: application/json' -d '{"name":"Bob","age":-2}' 'localhost:8080/pets'

elk returns a detailed response that includes information about which validations failed:

{
"code": 400,
"status": "Bad Request",
"errors": {
"Age": "This value failed validation on 'gt:0'.",
"Owner": "This value is required."
}
}

Note the uppercase field names. The validator package uses the structs field name to generate its validation errors, but you can simply override this, as stated in the example .

If you do not define any validation rules, elk will not include the validation-code in its generated output. elks` request validation is especially useful if you'd wanted to do cross-field-validation.

Upcoming Features

We hope you agree that elk has some useful features already, but there are still many exciting things to come. The next version of elk will include::

  • Fully working flutter frontend to administrate your nodes
  • Integration of Ent’s validation in the current request validator
  • More transport formats (currently only JSON)

Conclusion

This post has shown just a small part of what elk can do. To see some more examples of what you can do with it, head over to the project’s README on GitHub. I hope that with elk-powered Ent, you and your fellow developers can automate some repetitive tasks that go into building RESTful APIs and focus on more meaningful work.

elk is in an early stage of development, we welcome any suggestion or feedback and if you are willing to help we'd be very glad. The GitHub Issues is a wonderful place for you to reach out for help, feedback, suggestions and contribution.

About the Author

MasseElch is a software engineer from the windy, flat, north of Germany. When not hiking with his dog Kuro (who has his own Instagram channel 😱) or playing hide-and-seek with his son, he drinks coffee and enjoys coding.

Β· 10 min read

Locks are one of the fundamental building blocks of any concurrent computer program. When many things are happening simultaneously, programmers reach out to locks to guarantee the mutual exclusion of concurrent access to a resource. Locks (and other mutual exclusion primitives) exist in many different layers of the stack from low-level CPU instructions to application-level APIs (such as sync.Mutex in Go).

When working with relational databases, one of the common needs of application developers is the ability to acquire a lock on records. Imagine an inventory table, listing items available for sale on an e-commerce website. This table might have a column named state that could either be set to available or purchased. avoid the scenario where two users think they have successfully purchased the same inventory item, the application must prevent two operations from mutating the item from an available to a purchased state.

How can the application guarantee this? Having the server check if the desired item is available before setting it to purchased would not be good enough. Imagine a scenario where two users simultaneously try to purchase the same item. Two requests would travel from their browsers to the application server and arrive roughly at the same time. Both would query the database for the item's state, and see the item is available. Seeing this, both request handlers would issue an UPDATE query setting the state to purchased and the buyer_id to the id of the requesting user. Both queries will succeed, but the final state of the record will be that the user who issued the UPDATE query last will be considered the buyer of the item.

Over the years, different techniques have evolved to allow developers to write applications that provide these guarantees to users. Some of them involve explicit locking mechanisms provided by databases, while others rely on more general ACID properties of databases to achieve mutual exclusion. In this post we will explore the implementation of two of these techniques using Ent.

Optimistic Locking

Optimistic locking (sometimes also called Optimistic Concurrency Control) is a technique that can be used to achieve locking behavior without explicitly acquiring a lock on any record.

On a high-level, this is how optimistic locking works:

  • Each record is assigned a numeric version number. This value must be monotonically increasing. Often Unix timestamps of the latest row update are used.
  • A transaction reads a record, noting its version number from the database.
  • An UPDATE statement is issued to modify the record:
    • The statement must include a predicate requiring that the version number has not changed from its previous value. For example: WHERE id=<id> AND version=<previous version>.
    • The statement must increase the version. Some applications will increase the current value by 1, and some will set it to the current timestamp.
  • The database returns the amount of rows modified by the UPDATE statement. If the number is 0, this means someone else has modified the record between the time we read it, and the time we wanted to update it. The transaction is considered failed, rolled back and can be retried.

Optimistic locking is commonly used in "low contention" environments (situations where the likelihood of two transactions interfering with one another is relatively low) and where the locking logic can be trusted to happen in the application layer. If there are writers to the database that we cannot ensure to obey the required logic, this technique is rendered useless.

Let’s see how this technique can be employed using Ent.

We start by defining our ent.Schema for a User. The user has an online boolean field to specify whether they are currently online and an int64 field for the current version number.

// User holds the schema definition for the User entity.
type User struct {
ent.Schema
}

// Fields of the User.
func (User) Fields() []ent.Field {
return []ent.Field{
field.Bool("online"),
field.Int64("version").
DefaultFunc(func() int64 {
return time.Now().UnixNano()
}).
Comment("Unix time of when the latest update occurred")
}
}

Next, let's implement a simple optimistically locked update to our online field:

func optimisticUpdate(tx *ent.Tx, prev *ent.User, online bool) error {
// The next version number for the record must monotonically increase
// using the current timestamp is a common technique to achieve this.
nextVer := time.Now().UnixNano()

// We begin the update operation:
n := tx.User.Update().

// We limit our update to only work on the correct record and version:
Where(user.ID(prev.ID), user.Version(prev.Version)).

// We set the next version:
SetVersion(nextVer).

// We set the value we were passed by the user:
SetOnline(online).
SaveX(context.Background())

// SaveX returns the number of affected records. If this value is
// different from 1 the record must have been changed by another
// process.
if n != 1 {
return fmt.Errorf("update failed: user id=%d updated by another process", prev.ID)
}
return nil
}

Next, let's write a test to verify that if two processes try to edit the same record, only one will succeed:

func TestOCC(t *testing.T) {
client := enttest.Open(t, "sqlite3", "file:ent?mode=memory&cache=shared&_fk=1")
ctx := context.Background()

// Create the user for the first time.
orig := client.User.Create().SetOnline(true).SaveX(ctx)

// Read another copy of the same user.
userCopy := client.User.GetX(ctx, orig.ID)

// Open a new transaction:
tx, err := client.Tx(ctx)
if err != nil {
log.Fatalf("failed creating transaction: %v", err)
}

// Try to update the record once. This should succeed.
if err := optimisticUpdate(tx, userCopy, false); err != nil {
tx.Rollback()
log.Fatal("unexpected failure:", err)
}

// Try to update the record a second time. This should fail.
err = optimisticUpdate(tx, orig, false)
if err == nil {
log.Fatal("expected second update to fail")
}
fmt.Println(err)
}

Running our test:

=== RUN   TestOCC
update failed: user id=1 updated by another process
--- PASS: Test (0.00s)

Great! Using optimistic locking we can prevent two processes from stepping on each other's toes!

Pessimistic Locking

As we've mentioned above, optimistic locking isn't always appropriate. For use cases where we prefer to delegate the responsibility for maintaining the integrity of the lock to the databases, some database engines (such as MySQL, Postgres, and MariaDB, but not SQLite) offer pessimistic locking capabilities. These databases support a modifier to SELECT statements that is called SELECT ... FOR UPDATE. The MySQL documentation explains:

A SELECT ... FOR UPDATE reads the latest available data, setting exclusive locks on each row it reads. Thus, it sets the same locks a searched SQL UPDATE would set on the rows.

Alternatively, users can use SELECT ... FOR SHARE statements, as explained by the docs, SELECT ... FOR SHARE:

Sets a shared mode lock on any rows that are read. Other sessions can read the rows, but cannot modify them until your transaction commits. If any of these rows were changed by another transaction that has not yet committed, your query waits until that transaction ends and then uses the latest values.

Ent has recently added support for FOR SHARE/ FOR UPDATE statements via a feature-flag called sql/lock. To use it, modify your generate.go file to include --feature sql/lock:

//go:generate go run -mod=mod entgo.io/ent/cmd/ent generate --feature sql/lock ./schema 

Next, let's implement a function that will use pessimistic locking to make sure only a single process can update our User object's online field:

func pessimisticUpdate(tx *ent.Tx, id int, online bool) (*ent.User, error) {
ctx := context.Background()

// On our active transaction, we begin a query against the user table
u, err := tx.User.Query().

// We add a predicate limiting the lock to the user we want to update.
Where(user.ID(id)).

// We use the ForUpdate method to tell ent to ask our DB to lock
// the returned records for update.
ForUpdate(
// We specify that the query should not wait for the lock to be
// released and instead fail immediately if the record is locked.
sql.WithLockAction(sql.NoWait),
).
Only(ctx)

// If we failed to acquire the lock we do not proceed to update the record.
if err != nil {
return nil, err
}

// Finally, we set the online field to the desired value.
return u.Update().SetOnline(online).Save(ctx)
}

Now, let's write a test that verifies that if two processes try to edit the same record, only one will succeed:

func TestPessimistic(t *testing.T) {
ctx := context.Background()
client := enttest.Open(t, dialect.MySQL, "root:pass@tcp(localhost:3306)/test?parseTime=True")

// Create the user for the first time.
orig := client.User.Create().SetOnline(true).SaveX(ctx)

// Open a new transaction. This transaction will acquire the lock on our user record.
tx, err := client.Tx(ctx)
if err != nil {
log.Fatalf("failed creating transaction: %v", err)
}
defer tx.Commit()

// Open a second transaction. This transaction is expected to fail at
// acquiring the lock on our user record.
tx2, err := client.Tx(ctx)
if err != nil {
log.Fatalf("failed creating transaction: %v", err)
}
defer tx.Commit()

// The first update is expected to succeed.
if _, err := pessimisticUpdate(tx, orig.ID, true); err != nil {
log.Fatalf("unexpected error: %s", err)
}

// Because we did not run tx.Commit yet, the row is still locked when
// we try to update it a second time. This operation is expected to
// fail.
_, err = pessimisticUpdate(tx2, orig.ID, true)
if err == nil {
log.Fatal("expected second update to fail")
}
fmt.Println(err)
}

A few things are worth mentioning in this example:

  • Notice that we use a real MySQL instance to run this test against, as SQLite does not support SELECT .. FOR UPDATE.
  • For the simplicity of the example, we used the sql.NoWait option to tell the database to return an error if the lock cannot be acquired. This means that the calling application needs to retry the write after receiving the error. If we don't specify this option, we can create flows where our application blocks until the lock is released and then proceeds without retrying. This is not always desirable but it opens up some interesting design options.
  • We must always commit our transaction. Forgetting to do so can result in some serious issues. Remember that while the lock is maintained, no one can read or update this record.

Running our test:

=== RUN   TestPessimistic
Error 3572: Statement aborted because lock(s) could not be acquired immediately and NOWAIT is set.
--- PASS: TestPessimistic (0.08s)

Great! We have used MySQL's "locking reads" capabilities and Ent's new support for it to implement a locking mechanism that provides real mutual exclusion guarantees.

Conclusion

We began this post by presenting the type of business requirements that lead application developers to reach out for locking techniques when working with databases. We continued by presenting two different approaches to achieving mutual exclusion when updating database records and demonstrated how to employ these techniques using Ent.

Have questions? Need help with getting started? Feel free to join our Slack channel.

For more Ent news and updates:

Β· 7 min read

TL;DR

We added a new integration to the Ent GraphQL extension that generates type-safe GraphQL filters (i.e. Where predicates) from an ent/schema, and allows users to seamlessly map GraphQL queries to Ent queries.

For example, to get all COMPLETED todo items, we can execute the following:

query QueryAllCompletedTodos {
todos(
where: {
status: COMPLETED,
},
) {
edges {
node {
id
}
}
}
}

The generated GraphQL filters follow the Ent syntax. This means, the following query is also valid:

query FilterTodos {
todos(
where: {
or: [
{
hasParent: false,
status: COMPLETED,
},
{
status: IN_PROGRESS,
hasParentWith: {
priorityLT: 1,
statusNEQ: COMPLETED,
},
}
]
},
) {
edges {
node {
id
}
}
}
}

Background

Many libraries that deal with data in Go choose the path of passing around empty interface instances (interface{}) and use reflection at runtime to figure out how to map data to struct fields. Aside from the performance penalty of using reflection everywhere, the big negative impact on teams is the loss of type-safety.

When APIs are explicit, known at compile-time (or even as we type), the feedback a developer receives around a large class of errors is almost immediate. Many defects are found early, and development is also much more fun!

Ent was designed to provide an excellent developer experience for teams working on applications with large data-models. To facilitate this, we decided early on that one of the core design principles of Ent is "statically typed and explicit API using code generation". This means, that for every entity a developer defines in their ent/schema, explicit, type-safe code is generated for the developer to efficiently interact with their data. For example, In the Filesystem Example in the ent repository, you will find a schema named File:

// File holds the schema definition for the File entity.
type File struct {
ent.Schema
}
// Fields of the File.
func (File) Fields() []ent.Field {
return []ent.Field{
field.String("name"),
field.Bool("deleted").
Default(false),
field.Int("parent_id").
Optional(),
}
}

When the Ent code-gen runs, it will generate many predicate functions. For example, the following function which can be used to filter Files by their name field:

package file
// .. truncated ..

// Name applies the EQ predicate on the "name" field.
func Name(v string) predicate.File {
return predicate.File(func(s *sql.Selector) {
s.Where(sql.EQ(s.C(FieldName), v))
})
}

GraphQL is a query language for APIs originally created at Facebook. Similar to Ent, GraphQL models data in graph concepts and facilitates type-safe queries. Around a year ago, we released an integration between Ent and GraphQL. Similar to the gRPC Integration, the goal for this integration is to allow developers to easily create API servers that map to Ent, to mutate and query data in their databases.

Automatic GraphQL Filters Generation

In a recent community survey, the Ent + GraphQL integration was mentioned as one of the most loved features of the Ent project. Until today, the integration allowed users to perform useful, albeit basic queries against their data. Today, we announce the release of a feature that we think will open up many interesting new use cases for Ent users: "Automatic GraphQL Filters Generation".

As we have seen above, the Ent code-gen maintains for us a suite of predicate functions in our Go codebase that allow us to easily and explicitly filter data from our database tables. This power was, until recently, not available (at least not automatically) to users of the Ent + GraphQL integration. With automatic GraphQL filter generation, by making a single-line configuration change, developers can now add to their GraphQL schema a complete set of "Filter Input Types" that can be used as predicates in their GraphQL queries. In addition, the implementation provides runtime code that parses these predicates and maps them into Ent queries. Let's see this in action:

Generating Filter Input Types

In order to generate input filters (e.g. TodoWhereInput) for each type in your ent/schema package, edit the ent/entc.go configuration file as follows:

// +build ignore

package main

import (
"log"

"entgo.io/contrib/entgql"
"entgo.io/ent/entc"
"entgo.io/ent/entc/gen"
)

func main() {
ex, err := entgql.NewExtension(
entgql.WithWhereFilters(true),
entgql.WithConfigPath("../gqlgen.yml"),
entgql.WithSchemaPath("<PATH-TO-GRAPHQL-SCHEMA>"),
)
if err != nil {
log.Fatalf("creating entgql extension: %v", err)
}
err = entc.Generate("./schema", &gen.Config{}, entc.Extensions(ex))
if err != nil {
log.Fatalf("running ent codegen: %v", err)
}
}

If you're new to Ent and GraphQL, please follow the Getting Started Tutorial.

Next, run go generate ./ent/.... Observe that Ent has generated <T>WhereInput for each type in your schema. Ent will update the GraphQL schema as well, so you don't need to autobind them to gqlgen manually. For example:

ent/where_input.go
// TodoWhereInput represents a where input for filtering Todo queries.
type TodoWhereInput struct {
Not *TodoWhereInput `json:"not,omitempty"`
Or []*TodoWhereInput `json:"or,omitempty"`
And []*TodoWhereInput `json:"and,omitempty"`

// "created_at" field predicates.
CreatedAt *time.Time `json:"createdAt,omitempty"`
CreatedAtNEQ *time.Time `json:"createdAtNEQ,omitempty"`
CreatedAtIn []time.Time `json:"createdAtIn,omitempty"`
CreatedAtNotIn []time.Time `json:"createdAtNotIn,omitempty"`
CreatedAtGT *time.Time `json:"createdAtGT,omitempty"`
CreatedAtGTE *time.Time `json:"createdAtGTE,omitempty"`
CreatedAtLT *time.Time `json:"createdAtLT,omitempty"`
CreatedAtLTE *time.Time `json:"createdAtLTE,omitempty"`

// "status" field predicates.
Status *todo.Status `json:"status,omitempty"`
StatusNEQ *todo.Status `json:"statusNEQ,omitempty"`
StatusIn []todo.Status `json:"statusIn,omitempty"`
StatusNotIn []todo.Status `json:"statusNotIn,omitempty"`

// .. truncated ..
}
todo.graphql
"""
TodoWhereInput is used for filtering Todo objects.
Input was generated by ent.
"""
input TodoWhereInput {
not: TodoWhereInput
and: [TodoWhereInput!]
or: [TodoWhereInput!]

"""created_at field predicates"""
createdAt: Time
createdAtNEQ: Time
createdAtIn: [Time!]
createdAtNotIn: [Time!]
createdAtGT: Time
createdAtGTE: Time
createdAtLT: Time
createdAtLTE: Time

"""status field predicates"""
status: Status
statusNEQ: Status
statusIn: [Status!]
statusNotIn: [Status!]

# .. truncated ..
}

Next, to complete the integration we need to make two more changes:

1. Edit the GraphQL schema to accept the new filter types:

type Query {
todos(
after: Cursor,
first: Int,
before: Cursor,
last: Int,
orderBy: TodoOrder,
where: TodoWhereInput,
): TodoConnection
}

2. Use the new filter types in GraphQL resolvers:

func (r *queryResolver) Todos(ctx context.Context, after *ent.Cursor, first *int, before *ent.Cursor, last *int, orderBy *ent.TodoOrder, where *ent.TodoWhereInput) (*ent.TodoConnection, error) {
return r.client.Todo.Query().
Paginate(ctx, after, first, before, last,
ent.WithTodoOrder(orderBy),
ent.WithTodoFilter(where.Filter),
)
}

Filter Specification

As mentioned above, with the new GraphQL filter types, you can express the same Ent filters you use in your Go code.

Conjunction, disjunction and negation

The Not, And and Or operators can be added using the not, and and or fields. For example:

{
or: [
{
status: COMPLETED,
},
{
not: {
hasParent: true,
status: IN_PROGRESS,
}
}
]
}

When multiple filter fields are provided, Ent implicitly adds the And operator.

{
status: COMPLETED,
textHasPrefix: "GraphQL",
}

The above query will produce the following Ent query:

client.Todo.
Query().
Where(
todo.And(
todo.StatusEQ(todo.StatusCompleted),
todo.TextHasPrefix("GraphQL"),
)
).
All(ctx)

Edge/Relation filters

Edge (relation) predicates can be expressed in the same Ent syntax:

{
hasParent: true,
hasChildrenWith: {
status: IN_PROGRESS,
}
}

The above query will produce the following Ent query:

client.Todo.
Query().
Where(
todo.HasParent(),
todo.HasChildrenWith(
todo.StatusEQ(todo.StatusInProgress),
),
).
All(ctx)

Implementation Example

A working example exists in github.com/a8m/ent-graphql-example.

Wrapping Up

As we've discussed earlier, Ent has set creating a "statically typed and explicit API using code generation" as a core design principle. With automatic GraphQL filter generation, we are doubling down on this idea to provide developers with the same explicit, type-safe development experience on the RPC layer as well.

Have questions? Need help with getting started? Feel free to join our Slack channel.

For more Ent news and updates:

Β· 3 min read

A few months ago, we announced the experimental support for generating gRPC services from Ent Schema definitions. The implementation was not complete yet but we wanted to get it out the door for the community to experiment with and provide us with feedback.

Today, after much feedback from the community, we are happy to announce that the Ent + gRPC integration is "Ready for Usage", this means all of the basic features are complete and we anticipate that most Ent applications can utilize this integration.

What have we added since our initial announcement?

  • Support for "Optional Fields" - A common issue with Protobufs is that the way that nil values are represented: a zero-valued primitive field isn't encoded into the binary representation. This means that applications cannot distinguish between zero and not-set for primitive fields. To support this, the Protobuf project supports some "Well-Known-Types" called "wrapper types" that wrap the primitive value with a struct. This wasn't previously supported but now when entproto generates a Protobuf message definition, it uses these wrapper types to represent "Optional" ent fields:

    // Code generated by entproto. DO NOT EDIT.
    syntax = "proto3";

    package entpb;

    import "google/protobuf/wrappers.proto";

    message User {
    int32 id = 1;

    string name = 2;

    string email_address = 3;

    google.protobuf.StringValue alias = 4;
    }
  • Multi-edge support - when we released the initial version of
    protoc-gen-entgrpc, we only supported generating gRPC service implementations for "Unique" edges (i.e reference at most one entity). Since a recent version, the plugin supports the generation of gRPC methods to read and write entities with O2M and M2M relationships.

  • Partial responses - By default, edge information is not returned by the Get method of the service. This is done deliberately because the amount of entities related to an entity is unbound.

    To allow the caller of to specify whether or not to return the edge information or not, the generated service adheres to Google AIP-157 (Partial Responses). In short, the Get<T>Request message includes an enum named View, this enum allows the caller to control whether or not this information should be retrieved from the database or not.

    message GetUserRequest {
    int32 id = 1;

    View view = 2;

    enum View {
    VIEW_UNSPECIFIED = 0;

    BASIC = 1;

    WITH_EDGE_IDS = 2;
    }
    }

Getting Started

  • To help everyone get started with the Ent + gRPC integration, we have published an official Ent + gRPC Tutorial (and a complimentary GitHub repo).
  • Do you need help getting started with the integration or have some other question? Join us on Slack.
For more Ent news and updates:

Β· 3 min read

Migrating to a new ORM is not an easy process, and the transition cost can be prohibitive to many organizations. As much as we developers are enamoured by "Shiny New Things", the truth is that we rarely get a chance to work on a truly "green-field" project. Most of our careers, we operate in contexts where many technical and business constraints (a.k.a legacy systems) dictate and limit our options for moving forward. Developers of new technologies that want to succeed must offer interoperability capability and integration paths to help organizations seamlessly transition to a new way of solving an existing problem.

To help lower the cost of transitioning to Ent (or simply experimenting with it), we have started the "Schema Import Initiative" to help support many use cases for generating Ent schemas from external resources. The centrepiece of this effort is the schemast package (source code, docs) which enables developers to easily write programs that generate and manipulate Ent schemas. Using this package, developers can program in a high-level API, relieving them from worrying about code parsing and AST manipulations.

Protobuf Import Support

The first project to use this new API, is protoc-gen-ent, a protoc plugin to generate Ent schemas from .proto files (docs). Organizations that have existing schemas defined in Protobuf can use this tool to generate Ent code automatically. For example, taking a simple message definition:

syntax = "proto3";

package entpb;

option go_package = "github.com/yourorg/project/ent/proto/entpb";

message User {
string name = 1;
string email_address = 2;
}

And setting the ent.schema.gen option to true:

syntax = "proto3";

package entpb;

+import "options/opts.proto";

option go_package = "github.com/yourorg/project/ent/proto/entpb";

message User {
+ option (ent.schema).gen = true; // <-- tell protoc-gen-ent you want to generate a schema from this message
string name = 1;
string email_address = 2;
}

Developers can invoke the standard protoc (protobuf compiler) command to use this plugin:

protoc -I=proto/ --ent_out=. --ent_opt=schemadir=./schema proto/entpb/user.proto

To generate Ent schemas from these definitions:

package schema

import (
"entgo.io/ent"
"entgo.io/ent/schema/field"
)

type User struct {
ent.Schema
}

func (User) Fields() []ent.Field {
return []ent.Field{field.String("name"), field.String("email_address")}
}
func (User) Edges() []ent.Edge {
return nil
}

To start using protoc-gen-ent today, and read about all of the different configuration options, head over to the documentation!

Join the Schema Import Initiative

Do you have schemas defined elsewhere that you would like to automatically import in to Ent? With the schemast package, it is easier than ever to write the tool that you need to do that. Not sure how to start? Want to collaborate with the community in planning and building out your idea? Reach out to our great community via our Slack channel or start a discussion on GitHub!

For more Ent news and updates:

Β· 11 min read

ent + gRPC

Introduction

Having entity schemas defined in a central, language-neutral format has many benefits as the scale of software engineering organizations increase. To do this, many organizations use Protocol Buffers as their interface definition language (IDL). In addition, gRPC, a Protobuf-based RPC framework modeled after Google's internal Stubby is becoming increasingly popular due to its efficiency and code-generation capabilities.

Being an IDL, gRPC does not prescribe any specific guidelines on implementing the data access layer so implementations vary greatly. Ent is a natural candidate for building the data access layer in any Go application and so there is great potential in integrating the two technologies together.

Today we announce an experimental version of entproto, a Go package, and a command-line tool to add Protobuf and gRPC support for ent users. With entproto, developers can set up a fully working CRUD gRPC server in a few minutes. In this post, we will show exactly how to do just that.

Setting Up

The final version of this tutorial is available on GitHub, you can clone it if you prefer following along that way.

Let's start by initializing a new Go module for our project:

mkdir ent-grpc-example
cd ent-grpc-example
go mod init ent-grpc-example

Next we use go run to invoke the ent code generator to initialize a schema:

go run -mod=mod entgo.io/ent/cmd/ent init User

Our directory should now look like:

.
β”œβ”€β”€ ent
β”‚Β Β  β”œβ”€β”€ generate.go
β”‚Β Β  └── schema
β”‚Β Β  └── user.go
β”œβ”€β”€ go.mod
└── go.sum

Next, let's add the entproto package to our project:

go get -u entgo.io/contrib/entproto

Next, we will define the schema for the User entity. Open ent/schema/user.go and edit:

package schema

import (
"entgo.io/ent"
"entgo.io/ent/schema"
)

// User holds the schema definition for the User entity.
type User struct {
ent.Schema
}

// Fields of the User.
func (User) Fields() []ent.Field {
return []ent.Field{
field.String("name").
Unique(),
field.String("email_address").
Unique(),
}
}

In this step, we added two unique fields to our User entity: name and email_address. The ent.Schema is just the definition of the schema, to create usable production code from it we need to run Ent's code generation tool on it. Run:

go generate ./...

Notice the a bunch of new files were created from our schema definition now:

β”œβ”€β”€ ent
β”‚Β Β  β”œβ”€β”€ client.go
β”‚Β Β  β”œβ”€β”€ config.go
// .... many more
β”‚Β Β  β”œβ”€β”€ user
β”‚Β Β  β”œβ”€β”€ user.go
β”‚Β Β  β”œβ”€β”€ user_create.go
β”‚Β Β  β”œβ”€β”€ user_delete.go
β”‚Β Β  β”œβ”€β”€ user_query.go
β”‚Β Β  └── user_update.go
β”œβ”€β”€ go.mod
└── go.sum

At this point, we can open a connection to a database, run a migration to create the users table, and start reading and writing data to it. This is covered on the Setup Tutorial, so let's cut to the chase and learn about generating Protobuf definitions and gRPC servers from our schema.

Generating Go Protobufs with entproto

As ent and Protobuf schemas are not identical, we must supply some annotations on our schema to help entproto figure out exactly how to generate Protobuf definitions (called "Messages" in protobuf lingo).

The first thing we need to do is to add an entproto.Message() annotation. This is our opt-in to Protobuf schema generation, we don't necessarily want to generate proto messages or gRPC service definitions from all of our schema entities, and this annotation gives us that control. To add it, append to ent/schema/user.go:

func (User) Annotations() []schema.Annotation {
return []schema.Annotation{
entproto.Message(),
}
}

Next, we need to annotate each field and assign it a field number. Recall that when defining a protobuf message type, each field must be assigned a unique number. To do that, we add an entproto.Field annotation on each field. Update the Fields in ent/schema/user.go:

// Fields of the User.
func (User) Fields() []ent.Field {
return []ent.Field{
field.String("name").
Unique().
Annotations(
entproto.Field(2),
),
field.String("email_address").
Unique().
Annotations(
entproto.Field(3),
),
}
}

Notice that we did not start our field numbers from 1, this is because ent implicitly creates the ID field for the entity, and that field is automatically assigned the number 1. We can now generate our protobuf message type definitions. To do that, we will add to ent/generate.go a go:generate directive that invokes the entproto command-line tool. It should now look like this:

package ent

//go:generate go run -mod=mod entgo.io/ent/cmd/ent generate ./schema
//go:generate go run -mod=mod entgo.io/contrib/entproto/cmd/entproto -path ./schema

Let's re-generate our code:

go generate ./...

Observe that a new directory was created which will contain all protobuf related generated code: ent/proto. It now contains:

ent/proto
└── entpb
β”œβ”€β”€ entpb.proto
└── generate.go

Two files were created. Let's look at their contents:

// Code generated by entproto. DO NOT EDIT.
syntax = "proto3";

package entpb;

option go_package = "ent-grpc-example/ent/proto/entpb";

message User {
int32 id = 1;

string user_name = 2;

string email_address = 3;
}

Nice! A new .proto file containing a message type definition that maps to our User schema was created!

package entpb
//go:generate protoc -I=.. --go_out=.. --go-grpc_out=.. --go_opt=paths=source_relative --go-grpc_opt=paths=source_relative --entgrpc_out=.. --entgrpc_opt=paths=source_relative,schema_path=../../schema entpb/entpb.proto

A new generate.go file was created with an invocation to protoc, the protobuf code generator instructing it how to generate Go code from our .proto file. For this command to work, we must first install protoc as well as 3 protobuf plugins: protoc-gen-go (which generates Go Protobuf structs), protoc-gen-go-grpc (which generates Go gRPC service interfaces and clients), and protoc-gen-entgrpc (which generates an implementation of the service interface). If you do not have these installed, please follow these directions:

After installing these dependencies, we can re-run code-generation:

go generate ./...

Observe that a new file named ent/proto/entpb/entpb.pb.go was created which contains the generated Go structs for our entities.

Let's write a test that uses it to make sure everything is wired correctly. Create a new file named pb_test.go and write:

package main

import (
"testing"

"ent-grpc-example/ent/proto/entpb"
)

func TestUserProto(t *testing.T) {
user := entpb.User{
Name: "rotemtam",
EmailAddress: "rotemtam@example.com",
}
if user.GetName() != "rotemtam" {
t.Fatal("expected user name to be rotemtam")
}
if user.GetEmailAddress() != "rotemtam@example.com" {
t.Fatal("expected email address to be rotemtam@example.com")
}
}

To run it:

go get -u./... # install deps of the generated package
go test ./...

Hooray! The test passes. We have successfully generated working Go Protobuf structs from our Ent schema. Next, let's see how to automatically generate a working CRUD gRPC server from our schema.

Generating a Fully Working gRPC Server from our Schema

Having Protobuf structs generated from our ent.Schema can be useful, but what we're really interested in is getting an actual server that can create, read, update, and delete entities from an actual database. To do that, we need to update just one line of code! When we annotate a schema with entproto.Service, we tell the entproto code-gen that we are interested in generating a gRPC service definition, from the protoc-gen-entgrpc will read our definition and generate a service implementation. Edit ent/schema/user.go and modify the schema's Annotations:

func (User) Annotations() []schema.Annotation {
return []schema.Annotation{
entproto.Message(),
+ entproto.Service(), // <-- add this
}
}

Now re-run code-generation:

go generate ./...

Observe some interesting changes in ent/proto/entpb:

ent/proto/entpb
β”œβ”€β”€ entpb.pb.go
β”œβ”€β”€ entpb.proto
β”œβ”€β”€ entpb_grpc.pb.go
β”œβ”€β”€ entpb_user_service.go
└── generate.go

First, entproto added a service definition to entpb.proto:

service UserService {
rpc Create ( CreateUserRequest ) returns ( User );

rpc Get ( GetUserRequest ) returns ( User );

rpc Update ( UpdateUserRequest ) returns ( User );

rpc Delete ( DeleteUserRequest ) returns ( google.protobuf.Empty );
}

In addition, two new files were created. The first, ent_grpc.pb.go, contains the gRPC client stub and the interface definition. If you open the file, you will find in it (among many other things):

// UserServiceClient is the client API for UserService service.
//
// For semantics around ctx use and closing/ending streaming RPCs, please refer to https://pkg.go.dev/google.golang.org/grpc/?tab=doc#ClientConn.NewStream.
type UserServiceClient interface {
Create(ctx context.Context, in *CreateUserRequest, opts ...grpc.CallOption) (*User, error)
Get(ctx context.Context, in *GetUserRequest, opts ...grpc.CallOption) (*User, error)
Update(ctx context.Context, in *UpdateUserRequest, opts ...grpc.CallOption) (*User, error)
Delete(ctx context.Context, in *DeleteUserRequest, opts ...grpc.CallOption) (*emptypb.Empty, error)
}

The second file, entpub_user_service.go contains a generated implementation for this interface. For example, an implementation for the Get method:

// Get implements UserServiceServer.Get
func (svc *UserService) Get(ctx context.Context, req *GetUserRequest) (*User, error) {
get, err := svc.client.User.Get(ctx, int(req.GetId()))
switch {
case err == nil:
return toProtoUser(get), nil
case ent.IsNotFound(err):
return nil, status.Errorf(codes.NotFound, "not found: %s", err)
default:
return nil, status.Errorf(codes.Internal, "internal error: %s", err)
}
}

Not bad! Next, let's create a gRPC server that can serve requests to our service.

Creating the Server

Create a new file cmd/server/main.go and write:

package main

import (
"context"
"log"
"net"

_ "github.com/mattn/go-sqlite3"
"ent-grpc-example/ent"
"ent-grpc-example/ent/proto/entpb"
"google.golang.org/grpc"
)

func main() {
// Initialize an ent client.
client, err := ent.Open("sqlite3", "file:ent?mode=memory&cache=shared&_fk=1")
if err != nil {
log.Fatalf("failed opening connection to sqlite: %v", err)
}
defer client.Close()

// Run the migration tool (creating tables, etc).
if err := client.Schema.Create(context.Background()); err != nil {
log.Fatalf("failed creating schema resources: %v", err)
}

// Initialize the generated User service.
svc := entpb.NewUserService(client)

// Create a new gRPC server (you can wire multiple services to a single server).
server := grpc.NewServer()

// Register the User service with the server.
entpb.RegisterUserServiceServer(server, svc)

// Open port 5000 for listening to traffic.
lis, err := net.Listen("tcp", ":5000")
if err != nil {
log.Fatalf("failed listening: %s", err)
}

// Listen for traffic indefinitely.
if err := server.Serve(lis); err != nil {
log.Fatalf("server ended: %s", err)
}
}

Notice that we added an import of github.com/mattn/go-sqlite3, so we need to add it to our module:

go get -u github.com/mattn/go-sqlite3

Next, let's run the server, while we write a client that will communicate with it:

go run -mod=mod ./cmd/server

Creating the Client

Let's create a simple client that will make some calls to our server. Create a new file named cmd/client/main.go and write:

package main

import (
"context"
"fmt"
"log"
"math/rand"
"time"

"ent-grpc-example/ent/proto/entpb"
"google.golang.org/grpc"
"google.golang.org/grpc/status"
)

func main() {
rand.Seed(time.Now().UnixNano())

// Open a connection to the server.
conn, err := grpc.Dial(":5000", grpc.WithInsecure())
if err != nil {
log.Fatalf("failed connecting to server: %s", err)
}
defer conn.Close()

// Create a User service Client on the connection.
client := entpb.NewUserServiceClient(conn)

// Ask the server to create a random User.
ctx := context.Background()
user := randomUser()
created, err := client.Create(ctx, &entpb.CreateUserRequest{
User: user,
})
if err != nil {
se, _ := status.FromError(err)
log.Fatalf("failed creating user: status=%s message=%s", se.Code(), se.Message())
}
log.Printf("user created with id: %d", created.Id)

// On a separate RPC invocation, retrieve the user we saved previously.
get, err := client.Get(ctx, &entpb.GetUserRequest{
Id: created.Id,
})
if err != nil {
se, _ := status.FromError(err)
log.Fatalf("failed retrieving user: status=%s message=%s", se.Code(), se.Message())
}
log.Printf("retrieved user with id=%d: %v", get.Id, get)
}

func randomUser() *entpb.User {
return &entpb.User{
Name: fmt.Sprintf("user_%d", rand.Int()),
EmailAddress: fmt.Sprintf("user_%d@example.com", rand.Int()),
}
}

Our client creates a connection to port 5000, where our server is listening, then issues a Create request to create a new user, and then issues a second Get request to retrieve it from the database. Let's run our client code:

go run ./cmd/client

Observe the output:

2021/03/18 10:42:58 user created with id: 1
2021/03/18 10:42:58 retrieved user with id=1: id:1 name:"user_730811260095307266" email_address:"user_7338662242574055998@example.com"

Amazing! With a few annotations on our schema, we used the super-powers of code generation to create a working gRPC server in no time!

Caveats and Limitations

entproto is still experimental stage and lacks some basic functionality. For example, many applications will probably want a List or Find method on their service, but these are not yet supported. In addition, some other issues we plan to tackle in the near future:

  • Currently only "unique" edges are supported (O2O, O2M).
  • The generated "mutating" methods (Create/Update) currently set all fields, disregarding zero/null values and field nullability.
  • All fields are copied from the gRPC request to the ent client, support for configuring some fields to be unsettable via the service by adding a field/edge annotation is also planned.

Next Steps

We believe that ent + gRPC can be a great way to build server applications in Go. For example, to set granular access control to the entities managed by our application, developers can already use Privacy Policies that work out-of-the-box with the gRPC integration. To run any arbitrary Go code on the different lifecycle events of entities, developers can utilize custom Hooks.

Do you want to build gRPC servers with ent? If you want some help setting up or want the integration to support your use case, please reach out to us via our Discussions Page on GitHub or in the #ent channel on the Gophers Slack.

For more Ent news and updates:

Β· 5 min read

Over the past few months, there has been much discussion in the Ent project issues about adding support for the retrieval of the foreign key field when retrieving entities with One-to-One or One-to-Many edges. We are happy to announce that as of v0.7.0 ent supports this feature.

Before Edge-field Support

Prior to merging this branch, a user that wanted to retrieve the foreign-key field for an entity needed to use eager-loading. Suppose our schema looked like this:

// ent/schema/user.go:

// User holds the schema definition for the User entity.
type User struct {
ent.Schema
}

// Fields of the User.
func (User) Fields() []ent.Field {
return []ent.Field{
field.String("name").
Unique().
NotEmpty(),
}
}

// Edges of the User.
func (User) Edges() []ent.Edge {
return []ent.Edge{
edge.From("pets", Pet.Type).
Ref("owner"),
}
}

// ent/schema/pet.go

// Pet holds the schema definition for the Pet entity.
type Pet struct {
ent.Schema
}

// Fields of the Pet.
func (Pet) Fields() []ent.Field {
return []ent.Field{
field.String("name").
NotEmpty(),
}
}

// Edges of the Pet.
func (Pet) Edges() []ent.Edge {
return []ent.Edge{
edge.To("owner", User.Type).
Unique().
Required(),
}
}

The schema describes two related entities: User and Pet, with a One-to-Many edge between them: a user can own many pets and a pet can have one owner.

When retrieving pets from the data storage, it is common for developers to want to access the foreign-key field on the pet. However, because this field is created implicitly from the owner edge it was automatically accessible when retrieving an entity. To retrieve this from the storage a developer needed to do something like:

func Test(t *testing.T) {
ctx := context.Background()
c := enttest.Open(t, dialect.SQLite, "file:ent?mode=memory&cache=shared&_fk=1")
defer c.Close()

// Create the User
u := c.User.Create().
SetUserName("rotem").
SaveX(ctx)

// Create the Pet
p := c.Pet.
Create().
SetOwner(u). // Associate with the user
SetName("donut").
SaveX(ctx)

petWithOwnerId := c.Pet.Query().
Where(pet.ID(p.ID)).
WithOwner(func(query *ent.UserQuery) {
query.Select(user.FieldID)
}).
OnlyX(ctx)
fmt.Println(petWithOwnerId.Edges.Owner.ID)
// Output: 1
}

Aside from being very verbose, retrieving the pet with the owner this way was inefficient in-terms of database queries. If we execute the query with the .Debug() we can see the DB queries ent generates to satisfy this call:

SELECT DISTINCT `pets`.`id`, `pets`.`name`, `pets`.`pet_owner` FROM `pets` WHERE `pets`.`id` = ? LIMIT 2 
SELECT DISTINCT `users`.`id` FROM `users` WHERE `users`.`id` IN (?)

In this example, Ent first retrieves the Pet with an ID of 1, then redundantly fetches the id field from the users table for users with an ID of 1.

With Edge-field Support

Edge-field support greatly simplifies and improves the efficiency of this flow. With this feature, developers can define the foreign key field as part of the schemas Fields(), and by using the .Field(..) modifier on the edge definition instruct Ent to expose and map the foreign column to this field. So, in our example schema, we would modify it to be:

// user.go stays the same

// pet.go
// Fields of the Pet.
func (Pet) Fields() []ent.Field {
return []ent.Field{
field.String("name").
NotEmpty(),
field.Int("owner_id"), // <-- explictly add the field we want to contain the FK
}
}

// Edges of the Pet.
func (Pet) Edges() []ent.Edge {
return []ent.Edge{
edge.To("owner", User.Type).
Field("owner_id"). // <-- tell ent which field holds the reference to the owner
Unique().
Required(),
}
}

In order to update our client code we need to re-run code generation:

go generate ./...

We can now modify our query to be much simpler:

func Test(t *testing.T) {
ctx := context.Background()
c := enttest.Open(t, dialect.SQLite, "file:ent?mode=memory&cache=shared&_fk=1")
defer c.Close()

u := c.User.Create().
SetUserName("rotem").
SaveX(ctx)

p := c.Pet.Create().
SetOwner(u).
SetName("donut").
SaveX(ctx)

petWithOwnerId := c.Pet.GetX(ctx, p.ID) // <-- Simply retrieve the Pet

fmt.Println(petWithOwnerId.OwnerID)
// Output: 1
}

Running with the .Debug() modifier we can see that the DB queries make more sense now:

SELECT DISTINCT `pets`.`id`, `pets`.`name`, `pets`.`owner_id` FROM `pets` WHERE `pets`.`id` = ? LIMIT 2

Hooray πŸŽ‰!

Migrating Existing Schemas to Edge Fields

If you are already using Ent with an existing schema, you may already have O2M relations whose foreign-key columns already exist in your database. Depending on how you configured your schema, chances are that they may be stored in a column by a different name than the field you are now adding. For instance, you want to create an owner_id field, but Ent auto-created the column foreign-key column as pet_owner.

To check what column name Ent is using for this field you can look in the ./ent/migrate/schema.go file:

PetsColumns = []*schema.Column{
{Name: "id", Type: field.TypeInt, Increment: true},
{Name: "name", Type: field.TypeString},
{Name: "pet_owner", Type: field.TypeInt, Nullable: true}, // <-- this is our FK
}

To allow for a smooth migration, you must explicitly tell Ent to keep using the existing column name. You can do this by using the StorageKey modifier (either on the field or on the edge). For example:

// In schema/pet.go:

// Fields of the Pet.
func (Pet) Fields() []ent.Field {
return []ent.Field{
field.String("name").
NotEmpty(),
field.Int("owner_id").
StorageKey("pet_owner"), // <-- explicitly set the column name
}
}

In the near future we plan to implement Schema Versioning, which will store the history of schema changes alongside the code. Having this information will allow ent to support such migrations in an automatic and predictable way.

Wrapping Up

Edge-field support is readily available and can be installed by go get -u entgo.io/ent@v0.7.0.

Many thanks πŸ™ to all the good people who took the time to give feedback and helped design this feature properly: Alex Snast, Ruben de Vries, Marwan Sulaiman, Andy Day, Sebastian Fekete and Joe Harvey.

For more Ent news and updates: