Mediachain v1.0

(aka Mediachain Developer Update XIII)

Just over five weeks ago, we published a post announcing a radical new technical direction for Mediachain. Based on experiments with the prototype, we’ve moved away from a global consensus model to a pluralistic, eventually-consistent universe that can handle the kind of data volume and throughput that the global media ecosystem requires — read more about our motivations here.

Some 500 commits, 43 PRs, and countless pots of tea (we like Assam Meleng) later, we are ready to ship the first production builds of concat and aleph, as well as to set up the first two public nodes and a directory. The nodes are loaded with normalized Creative Commons image data we’ve gathered from around the web, available to explore, manipulate and extend (or just browse through the Attribution Engine interface). The CC dataset is a perfect microcosm of how we envision data providers, aggregators and consumers across different industries and academic fields to interact in an open, universal media library.

This is all live, right now: scroll down to HOWTO below if you can’t wait to play with it, or read on for more details.

Architecture: Foundations

Home and Away, Again

The basic building blocks of the new architecture are concat nodes. Each node can work as a self-contained database to answer queries or receive writes, but they also can (crucially) natively query remote peers through the p2p protocol, and transparently merge peer data into the local db.

A node contains a datastore, which holds arbitrary metadata objects as CBOR/IPLD blobs, and a statement database. Statements are a kind of meta-metadata, including timestamps, publisher identity, signature, etc. They can refer to a single metadata object (currently implemented), a list of objects, or even an archive of millions (TODO). These statements serve as an index for the underlying metadata, but also as an announce channel for its publication.

Suppose I represent a small museum that wishes to make their collection data open to the public: I can simply add the item descriptions to the datastore, then publish a single statement announcing the publication of this dataset.

Discovery, Discovery, Discovery, Discovery

These nodes discover each other through directory servers (mcdir). At this stage discovery is very rudimentary: we have an open registry (“universe” mode) running at 54.173.71.205. Expanding this functionality, like adding authoritative peers for a particular namespace, is where a lot of the upcoming work will focus

Advantages to Both

The third piece of the ecosystem is aleph. Its role has shifted somewhat since the previous post: it’s now a set of lightweight clients that communicate with a concat. The aleph executable itself is a kind of lightweight node that speaks to a remote concat through libp2p. It can be wrapped by an Electron app for visual exploration and manipulation of the data (similar to how geth is used by Mist). The mcclient script speaks to and controls a local concat instance, much like how the ipfs client communicates with the ipfs daemon.

How did we do?

Let’s quickly revisit the main areas of concern from the “Looking Backwards, Looking Forwards” post:

• Throughput: the copycat-based design struggled to accept over 100 writes/s. concat currently receives over 5000 writes/s per node, with the bottleneck being the pure js CBOR encoder in the client (an optimized C implementation like tinycbor is over 100x faster). This brings us to the kinds of throughput expected in serious production systems, and the native sharding capabilities take us planet-scale.
• Scale: previously, we were hitting scaling issues at ~10MM objects in toto; with concat we’re not seeing performance degradation with ~60MM objects/node and expect this performance to hold north of 100MM. This means many organizations can run a single node on a modestly sized cloud instance, with room to grow.
• Translation: we’ve simplified our treatment of translators, but we still have schema validation and semantic versioning. This removes more complicated normalization questions from protocol scope.
• Assigning identifiers: we’re using a compact internal identifier, but all out-of-system IDs are also preserved (look at any image page in the indexer). This means we are no longer an arbiter of object similarity relationships, but give that power to the users.

Of course, the astute reader will point out that this is single-node performance; in effect, we are benchmarking a relatively thin golang layer around RocksDB/SQLite. This is exactly by design: eliminating the overheads of consensus has allowed us to achieve a highly performant system.

Architecture: Coming up Next

Many distributed systems, including Ethereum and Blockstack, have opted for full global state replication as a very simple answer to the question of data routing and discovery. Others, like IPFS, use a flat global content-addressed routing space, with object coordinates announced to a DHT.

Because of the size (storing the current test set as Ethereum state would cost tens of millions of dollars) and cardinality (already ~100MM records, vs ~1BN for the entire IPFS DHT) of our dataset, and the need to efficiently address individual objects, we cannot use these approaches.

Fortunately, media data has a naturally hierarchical structure according to its source, organization, field, etc; this means we can leverage this structure for efficient routing by topic, while keeping some of the nice self-certifying properties of a system like IPFS (and, in fact, full interoperability with it). To achieve this, we’ll need to significantly expand the functionality of the directory servers, and introduce query merging and processing logic in aleph, such that queries can be routed to authoritative peers for the appropriate namespaces (i.e. you will not need to specify the -r $peerId option to mcclient).

The counterpart to routing is aggregation and publishing. The current implementation supports a basic pull-based aggregation mechanism (I can replicate your data and statements into my own db based on a query); more sophisticated modes include Github-style pull requests, subscriptions/polling through multicast pubsub, and so on. An in-depth exploration of these options will be made in an upcoming blog post; we’re eager to hear about which ones will best support your usecases.

It may be helpful to think of this system as a sort of modern, data-oriented treatment of ideas explored in RSS and usenet: a combination of permissioned and permissionless spaces that speak a common protocol on top of which a variety of aggregation, discovery, subscription, archiving, etc topologies can be constructed.

HOWTO

Let’s explore the 70MM creative commons image dataset! To get started, grab the 1.0 release of concat and follow the node setup instructions (be sure to configure the node to connect to the public directory at 54.173.71.205). You’ll also need aleph from npm.

We can start by exploring the directory:

As you can see, one node holds the flickr dataset, while another holds a combined 500px/pexels/DPLA set (for simplicity, since each one is relatively small at the moment). You can imagine each of these organizations running its own node, synced to its API.

Let’s look up an image, for example this slightly unnerving clown:

I found it by searching for “tophat” in the Attribution Search Engine, which gave me the complete attribution data, as well as the “WKI” (Well-Known Identifier, or established ID) of “flickr100mm_3733230430”. Let’s use this to look up the complete record through the CLI (you can also get this invocation from the “Explore data in Mediachain” link on the page):

To get this data, your node connected to the flickr peer through the libp2p protocol and issued the query, but the results are available to you just as if they were stored in your own node.

Suppose that you represent, let’s say, the World Clown Association and wish to aggregate and preserve cultural artefacts related to your members’ work. To do this, you can merge the remote data about this photo into your own store:

Your node has now replicated the remote statement and underlying data!

For a somewhat less contrived example, consider an organization like CC, DPLA, or Europeana, which aggregate the works of member institutions or other like-minded contributors. The merge functionality is what will allow them to seamlessly bring together these datasets; right now the merge operation will bring in all referenced objects, but we’re working on a variety of replication modes, e.g. a lightweight reference where a weekly update pointer is copied over while the underlying data lives only on the remote peer.

(By the way, because you configured the directory at the beginning of this HOWTO, you can make your data available for exploration by others simply by taking your node public with mcclient status public — see the README for instructions on publishing your own data and other administrative commands)

Next Steps

The lion’s share of upcoming work will focus on routing and discovery questions discussed above. One prerequisite is integrating with an external identity provider, like Keybase or Blockstack, which is necessary to establish real-world authority. With that in place, we can start building out authoritative routing (so directories can send CC-related queries to the CC peers) and better aggregation tools like Pull Requests.

We’re also very excited for possibilities of bulk operations, like distributing millions of metadata items at once, packed in RocksDB SST files and shipped through a torrent-like mechanism for seeding large datasets or performing large-scale updates.

We are optimistic that this design will carry Mediachain into being a truly planetary scale media library. Please join us in our Slack for this journey.

If you believe in the decentralization revolution, hit the heart button below to help spread the word and tell your friends about Mediachain!