Overview

Xorma is a synchronous, reactive, in-memory database powered by mobx.

I have found it to be amazing for building editor-like experiences that maintain complex object graphs with lots of write operations.

I hacked together the first version of xorma back in 2022 when I set out to build Diode, a web-based 3d circuit simulator.

It worked remarkably well and I have continued to use this foundation when building other editor-like experiences.

Getting Started

Firstly, we'll need to install xorma along with mobx.

npm i xorma mobx

Next up, we'll need to define some models.

Each model will operate on a specific object type, so for example the UserModel will operate on the User object type. To get started we'll have to implement loadJSON and toJSON methods for each of our model types.

Take note of the fields that we mark as observable and computed. This is a necessary step to make our fields reactive and is standard practice when working with mobx.

import { DataType, Model } from "xorma";
import { observable, computed } from "mobx";

interface BaseData {
  id: string;
}

interface User extends BaseData {
  first_name: string;
  last_name: string;
}

interface Project extends BaseData {
  name: string;
}

interface Task extends BaseData {
  name: string;
  project_id: string;
  creator_id: string;
  assignee_id: string | null;
  created_at: number;
}

class BaseModel extends Model.withType(DataType<BaseData>()) {
  /**
   *  This is actually the default implementation of idSelector so this step
   *  is unnecessary, but you would want to override idSelector for any model
   *  where the primary key field is something other than `id`.
   */
  static idSelector(data: BaseData) {
    return data.id;
  }
}

class UserModel extends BaseModel.withType(DataType<User>()) {
  firstName: string;
  lastName: string;

  constructor(data: User) {
    super(data);
    this.loadJSON(data);
    makeObservable(this, {
      firstName: observable,
      lastName: observable,
      name: computed,
      createdTasks: computed,
      assignedTasks: computed,
    });
  }

  get name(): string {
    return `${this.firstName} ${this.lastName}`;
  }

  get createdTasks(): TaskModel[] {
    return TaskModel.getAll().filter((task) => task.creatorId === this.id);
  }

  get assignedTasks(): TaskModel[] {
    return TaskModel.getAll().filter((task) => task.assigneeId === this.id);
  }

  loadJSON(data: User) {
    this.firstName = data.first_name;
    this.lastName = data.last_name;
  }

  toJSON(): User {
    return {
      id: this.id,
      first_name: this.firstName,
      last_name: this.lastName,
    };
  }
}

class ProjectModel extends BaseModel.withType(DataType<Project>()) {
  name: string;

  constructor(data: Project) {
    super(data);
    this.loadJSON(data);
    makeObservable(this, {
      name: observable,
      tasks: computed,
    });
  }

  get tasks(): TaskModel[] {
    return TaskModel.getAll().filter((task) => task.projectId === this.id);
  }

  toJSON(): Project {
    return {
      id: this.id,
      name: this.name,
    };
  }

  loadJSON(data: Project) {
    this.name = data.name;
  }
}

class TaskModel extends BaseModel.withType(DataType<Task>()) {
  name: string;
  projectId: string;
  creatorId: string;
  assigneeId: string | null;
  createdAt: number;

  constructor(data: Task) {
    super(data);
    this.loadJSON(data);
    makeObservable(this, {
      name: observable,
      assigneeId: observable,
      creator: computed,
      assignee: computed,
    });
  }

  get creator(): UserModel {
    return UserModel.getById(this.creatorId)!;
  }

  get assignee(): UserModel | null {
    return this.assigneeId ? UserModel.getById(this.assigneeId)! : null;
  }

  toJSON(): Task {
    return {
      id: this.id,
      name: this.name,
      project_id: this.projectId,
      creator_id: this.creatorId,
      assignee_id: this.assigneeId,
      created_at: this.createdAt,
    };
  }

  loadJSON(data: Task) {
    this.name = data.name;
    this.projectId = data.project_id;
    this.creatorId = data.creator_id;
    this.assigneeId = data.assignee_id;
    this.createdAt = data.created_at;
  }
}

Next up, we need to connect our models to a store. The store will manage our model instances in a centralized place and will give us behavior such as undo/redo.

import { Store } from "xorma";

/* ... */

const store = new Store({
  models: {
    BaseModel,
    ProjectModel,
    UserModel,
    TaskModel,
  },
});

Here's a live example that ties together the setup we just ran through.

Create
Task
Assignee
Complete Project Proposal
Jane Smith
Review Code
John Doe

Reactivity

Before we dive deeper into xorma, let's briefly cover how mobx enables reactivity.

Observables and observers

At its core, mobx is comprised of observables and observers.

When an observable value is accessed inside of an observer, the observer will subscribe to the observable and will be re-executed whenever the observable is assigned a new value.

0

Here our little counter ticks away because we access the observable property, count, inside of an observer, AutoCount. Because of this, AutoCount is subcribed to all future changes to the count property and will re-render whenever count is assigned a new value.

Computeds

Computed properties are a second powerful type of observable state at our disposal. When computed properties are accessed inside of an observer, the observer again will subscribe to any future changes. However, we cannot assign values to computed properties. Instead, they take the form of readonly getters which accept no arguments.

0

You may question why we need to make doubled a computed getter. In this case, the reality is we don't. However, in more complex cases where the computed field logic is expensive, we can avoid executing the getter every time we access it and rather, mobx will cache its return value until the observables it depends on are modified.

Actions

The last concept we'll cover is actions. Actions offer a way to batch changes. Observers will not be notified of changes to observables until an action terminates. This means we can assign new values to multiple observables inside of an action and we don't have to worry about observers reacting to the changes until the action terminates.

Increment & decrementDo nothing

Notice how the doNothing method seemingly does nothing because it's wrapped in an action, however the incrementAndDecrement method quickly triggers two toasts in succession since it is not wrapped in an action.

To read up more about mobx, you can check out their docs site.

Store

In xorma, our state is held in a centralized store which we can think of as a reactive in-memory database.

Each model in this store has a corresponding collection and each collection holds model instances.

Model instances are special however, because each model instance must be tied to a unique id and only one instance will ever exist per id.

Consider the following scenario.

Init users

Note that even though we attempt to create 3 users, we only end up with 2. This is because xorma guarantees that only one model instance exists per id.

When we attempt to create the 2nd user, what happens under the hood is xorma sees the existing UserModel instance associated with id user-1 and updates the instance's data by calling instance.loadJSON({ id: "user-1", first_name: "Maxime", last_name: "Heckel" }).

The other important thing to note in the above example is that our component automatically re-renders when we create users. This is because the UserModel.getAll() function accesses values from an observable collection.

So as long as we wrap our component in the observer hoc, the component will automatically render whenever instances are added/removed.

History

The store also provides us a history api. At any point we can call store.history.commit() to push a snapshot onto the history stack. And then we can navigate forward and backward via store.history.undo() and store.history.redo().

UndoRedo
Austin Malerba
Maxime Heckel
Guillermo Rauch

Sandboxing

Sometimes it's useful to dry-run some logic, but ultimately we don't want to change the store state. For example, consider a chess game in which we want to check a pieces valid moves. A natural way to check if a move is valid is simply to try moving the piece and then check whether the move puts the player in check. With sandboxing we can easily dry-run the move and revert the store to its previous state before we notify any observers of the changes.

Below is a dumb example that demonstrates a less practical implementation of sandboxing.

Maybe add user

The callback passed to store.sandbox(callback) is run immediately and the default behavior is to revert, so there's no need to explicitly revert. However, if we wish to commit the changes made within the sandbox callback, then we need to call commit.

Relationships

One of the powerful things we can do with xorma is create getters to access related model instances.

Foreign keys

Consider the following one-to-many relationship between projects and tasks. It's easy to look up relevant model instances via the Model.getById and Model.getAll functions.

import { Model, DataType } from "xorma";
import { makeObservable, computed } from "mobx";

interface Project {
  id: string;
  name: string;
}

interface Task {
  id: string;
  project_id: string;
  name: string;
  done: boolean;
}

class ProjectModel extends Model.withType(DataType<Project>) {
  /** ... */

  constructor(data: Project) {
    super(data);
    this.loadJSON(data);
    makeObservable(this, {
      tasks: computed,
    });
  }

  /**
   * At first glance this might seem unperformant, but as long as we make
   * this field computed, its return value will be cached and it will only
   * be re-evaluated when its observable dependencies change. In this case,
   * our only observable dependency is the array of TaskModel instances and
   * this array will only change when we create/delete tasks. In practice,
   * this means that the logic inside of this getter will rarely run 🙌.
   */
  get tasks(): TaskModel[] {
    return TaskModel.getAll().filter((task) => task.projectId === this.id);
  }

  /** ... */
}

class TaskModel extends Model.withType(DataType<Task>) {
  /** ... */

  projectId: string;

  constructor(data: Task) {
    super(data);
    this.loadJSON(data);
    makeObservable(this, {
      project: computed,
    });
  }

  get project(): ProjectModel {
    return ProjectModel.getById(this.projectId)!;
  }

  /** ... */
}

Embedded data

In some cases, especially when working with nosql, our relationships are modeled as embedded subdocuments rather than via foreign keys.

Xorma supports this as well, though it generally leads to a clunkier experience, requires more maintenance logic, and decreases the viability of optimizations like sending patches to the server.

import { Model, DataType } from "xorma";
import { makeObservable, computed } from "mobx";

interface Project {
  id: string;
  name: string;
  tasks: Task[];
}

interface Task {
  id: string;
  name: string;
  done: boolean;
}

class ProjectModel extends Model.withType(DataType<Project>) {
  /** ... */

  tasks: TaskModel[];

  constructor(data: Project) {
    super(data);
    this.loadJSON(data);
    makeObservable(this, {
      tasks: observable.shallow,
    });
  }

  loadJSON(data: Project) {
    /** ... */
    this.tasks = data.tasks.map((task) => TaskModel.create(task));
  }

  toJSON() {
    return {
      /** ... */
      tasks: this.tasks.map((task) => task.toJSON()),
    };
  }

  /** ... */
}

class TaskModel extends Model.withType(DataType<Task>) {
  /** ... */

  constructor(data: Task) {
    super(data);
    this.loadJSON(data);
    makeObservable(this, {
      project: computed,
    });
  }

  get project(): ProjectModel {
    return ProjectModel.getAll().find((project) =>
      project.tasks.some((task) => task.id === this.id)
    )!;
  }

  /** ... */
}

With that said, it is recommended when working with xorma that you model your data in a highly normalized way (sql), as opposed to a denormalized/embedded way (nosql).

Inheritance

Another powerful facet of xorma is class inheritance. You'll frequently want to build out model behavior incrementally via inheritance chains.

Let's consider a BaseModel that all other models will inherit from. This is a great place to implement custom behaviors like soft deletes and cloning.

Clone user
Austin Malerba

Because we define the clone method on the BaseModel, all derrived classes will inherit it as well.

An example of a more complex hierarchy would be a file viewer. You could imagine the model hierarchy for a file viewer might resemble the following.

import { Model, DataType } from "xorma";

type BaseData {
  id: string
  created_at: number
  deleted_at: number | null
}

interface Node {
  type: 'file' | 'folder'
  parent_id: string | null
  name: string
}

interface FileNode extends Node {
  type: 'file'
  size: number
  content: string
  content_type: string
}

interface FolderNode extends Node {
  type: 'folder'
}

class BaseModel extends Model.withType(DataType<BaseData>()) {
  /** ... */
}

class NodeModel extends BaseModel.withType(DataType<Node>()) {
  /** ... */
}

class FileNodeModel extends NodeModel.withType(DataType<FileNode>()) {
  /** ... */
}

class FolderNodeModel extends NodeModel.withType(DataType<FolderNode>()) {
  /** ... */
}

This way we could put a large amount of shared logic in the NodeModel which the FileNodeModel and FolderNodeModel would then inherit.

Persistence

The easiest way to persist the store state is to call store.toJSON() inside of a mobx reaction and then do a debounced save to our backend.

Below is a simplified example of what this process could look like.

import { useEffect } from "react";
import { Model, Store } from "xorma";
import { reaction } from "mobx";
import { observer } from "mobx-react";
import axios from "axios";
import { debounce } from "lodash";
import { ProjectEditor } from "./editor";

class ProjectModel extends Model {
  /** ... */
}

const store = new Store({
  schemaVersion: 0,
  models: {
    ProjectModel,
    /** ... */
  },
});

const saveProject = debounce(
  (projectId, snapshot) => axios.post(`/api/project/${projectId}`, snapshot),
  3000
);

const ProjectPage = observer(({ projectId }) => {
  useEffect(() => {
    axios.get(`/api/projects/${projectId}`).then((data) => {
      store.reset();
      store.loadJSON(data);
    });
  }, [projectId]);

  useEffect(() => {
    return reaction(
      () => store.toJSON(),
      (snapshot) => saveProject(projectId, snapshot)
    );
  }, [projectId]);

  const project = ProjectModel.getById(params.projectId);

  if (!project) return null;

  return <ProjectEditor project={project} />;
});

Migrations

Another key part of persistence is migrations. When initializing the store, we can pass a schemaVersion parameter.

This version number will also be present in the serialized data returned by store.toJSON.

If the schemaVersion of the data ingested by store.loadJSON does not match the schemaVersion specified when initializing the store, the store will throw an error.

Whenever we modify our models in a way that alters the shape of the serialized store data, we should bump the store's schemaVersion and implement a migration function capable of reshaping older snapshots into the new format.

Let's assume we have the following TaskModel implementation where the status of the task is represented as a boolean done state.

import { DataType, Model, Store } from "xorma";
import { observable, computed } from "mobx";

interface Task {
  id: string;
  name: string;
  done: boolean;
}

class TaskModel extends BaseModel.withType(DataType<Task>()) {
  name: string;
  done: boolean;

  constructor(data: Task) {
    super(data);
    this.loadJSON(data);
    makeObservable(this, {
      name: observable,
      done: observable,
    });
  }

  toJSON(): Task {
    return {
      id: this.id,
      name: this.name,
      done: this.done,
    };
  }

  loadJSON(data: Task) {
    this.name = data.name;
    this.done = data.done;
  }
}

const store = new Store({ schemaVersion: 0, models: { TaskModel } });

But we're now feeling this isn't experessive enough and instead of a boolean done field, we decide we want a status field which can be one of ["idle", "in-progress", "done"].

We can update our type and model to accomodate this.

import { DataType, Model, Store } from "xorma";
import { observable, computed } from "mobx";

interface Task {
  id: string;
  name: string;
  status: "idle" | "in-progress" | "done";
}

class TaskModel extends BaseModel.withType(DataType<Task>()) {
  name: string;
  status: "idle" | "in-progress" | "done";

  constructor(data: Task) {
    super(data);
    this.loadJSON(data);
    makeObservable(this, {
      name: observable,
      status: observable,
    });
  }

  toJSON(): Task {
    return {
      id: this.id,
      name: this.name,
      status: this.status,
    };
  }

  loadJSON(data: Task) {
    this.name = data.name;
    this.status = data.status;
  }
}

// Note that we've bumped the schemaVersion
const store = new Store({ schemaVersion: 1, models: { TaskModel } });

But now if we fetch a project that was already save to the backend with schemaVersion 0, obviously we can't load that into the store because it has a done field and status will be undefined.

So we can write a migration that will run lazily whenever we fetch a project from the backend and we can make sure that it's been adapted to meet the store's current assumptions about the shape of the data.

import { Migration, Migrations, mapValues } from "xorma";

const migrations = new Migrations([
  new Migration({
    toVersion: 1,
    run: (data) => {
      return {
        ...data,
        TaskModel: mapValues(data.TaskModel, (task) => {
          const nextTask = {
            ...task,
            status: task.done ? "done" : "idle",
          };
          delete nextTask["done"];
          return nextTask;
        }),
      };
    },
  }),
]);

// Yay, now our old snapshot will meet the expectations of our latest store logic.
const newSnapshot = migrations.run(oldSnapshot);

API Reference

Store

The central class that manages all model instances.

interface StoreData<Data extends Record<string, Record<string, object>>> {
  schemaVersion: number;
  data: Data;
}

interface StorePatch {
  schemaVersion: number;
  data: Record<string, Record<string, object | null>>;
}

class Store<Models> {
  constructor(params: { schemaVersion?: number; models: Models });

  // Methods
  sandbox<T>(
    callback: (options: { commit: () => void; revert: () => void }) => T
  ): T;
  reset(): void;
  toJSON(): StoreData<Models>;
  loadJSON(data: StoreData<Models>): void;
  loadPatch(patch: StorePatch): void;

  // Properties
  history: {
    commit(options?: { replace?: boolean }): void;
    undo(): void;
    redo(): void;
    clear(): void;
    readonly items: StoreData<Models>[];
    readonly activeIndex: number;
    readonly activeItem: StoreData<Models> | undefined;
  };
}

Model

Base class for creating and managing models.

class Model {
  static idSelector(data: any): string;
  static create<T extends Model>(data: any): T;
  static getById<T extends Model>(
    id: string,
    options?: { includeChildren?: boolean }
  ): T | undefined;
  static getAll<T extends Model>(options?: { includeChildren?: boolean }): T[];
  static withType<T extends DefaultData>(): RetypedModelClass<typeof Model, T>;

  // Instance methods
  delete(): this;
  toJSON(): any;
  loadJSON(data: any): void;

  // Properties
  readonly id: string;
  readonly isDeleted: boolean;
  readonly isDetached: boolean;
}

Utilities

// Deep clone an object, ensuring that all observable properties
// are accessed and also that all proxies are stripped.
function dereference<T>(data: T): T;

// Compare two objects for deep equality
function isDeepEqual(obj1: any, obj2: any): boolean;

// Map values of an object
function mapValues<T, U>(
  obj: { [K in keyof T]: T[K] },
  mapper: (value: T[keyof T], key: keyof T) => U
): { [K in keyof T]: U };

// Calculate the difference between two store states
function diff(
  prev: StoreData<Record<string, Record<string, object>>>,
  next: StoreData<Record<string, Record<string, object>>>
): StorePatch;

class Migration {
  constructor(params: { toVersion: number; run: (data: any) => any });
}

class Migrations<T extends StoreData<any>> {
  constructor(migrations: Migration[]);
  run(snapshot: StoreData<any>): T;
}

class PromiseManager<T extends (...args: any[]) => Promise<any>> {
  constructor(asyncFunction: T);

  // Methods
  execute(...args: Parameters<T>): Promise<ReturnType<T>>;

  // Properties
  readonly status: "idle" | "pending" | "fulfilled" | "rejected";
  readonly isPending: boolean;
  readonly isFulfilled: boolean;
  readonly isRejected: boolean;
}

Examples

Below are some examples built with xorma.

Chess