JUnit tests in this module require an appropriate environment to be
setup for EO. The MariaDB instance specified in the environment
configuration will be exercised.

## Deployment Notes

### Hostname Scheme

* ms[n] - MongoDB Storage Server (Shard or Config)
* ma[n] - MongoDB Router

## Implementation Notes

* Streaming JSON type; application/json-seq

## Why Not Just Use Mongo

* With user-defined business schemas it is not possible to pick a good sharding key since the available keys are not
  known ahead of time. The only fiends which are available for an object are id, ctime, and type. Of those, id makes
  the best case for a shard id. In MongoDB the ID has the property that it includes 4 bytes of timestamp, so ids for
  newly created objects are ever-increasing. In other words, using id as a shard key results in newer objects
  concentrating on new shards. REBUTTAL: With later versionsof MongoDB it is now possible to use a hash-based key
  so using ID becomes practica in this sense.
* Changing the shard key results in the entire collection re-balancing, and 1.2x space is required across the cluster
  to perform the rebalance operation, plus all the IO to perform the work (the entire collection must be read and)
  written again). There are other approaches to generating MongoDB ids such as counter documents but they all have
  significant drawbacks.

### DIY with MySQL

* The IDs can be generated and mapped to a shard server by way of a route map instead of a hash algorithm. For example,
  a shard can be chosen randomly or deterministically, and then a new ID generated on that particular shard, which is
  suffixed to the shard ID, like a network and a host part on an IP address. This allows objects to be located using
  what is essentially a route map.
* Objects can be associated with their shards forever. In this case, it is the shards themselves that move around.
  More shards than necessary can be created early on (for example, 1000x shards per server) which allow hot shards to
  be detected later and moved onto more resource-available hardware.
* The thing MongoDB has going for it is that it naturally supports filtering based on the document contents, whereas
  some kind of query & filter language inevitably is going to have to be provided to support a DIY solution. However,
  it is straightforward from both a cost & billing solution to push this down onto the client. For example, it's hard to
  understand the billing impact of SQL queries since they have such varied consequences, but it's easy to just meter
  object store based on rads/writes of objects, especially when those objects have maximum sizes. If the task performing
  the filtering is a) streaming and b) network-adjacent c) read criteria is well indexed to produce a stream of objects
  for the client, this could work quite well. S3 took this approach and became wildly successful.

## Things that are tough

### Visit All The Things

Finding all objects of a certain type requires visiting all shards. There really is no way around this. Although
a simple enough concept, there are practical implications here - are all shards online, ok probably not, so then when
you visit a bad shard during a streaming operation, what then.

Once potential answer to this is to say that - no, in fact, you can't visit all the things, at least not with a stream.
The problem with a batch approach to this solution is that it requires the space to buffer the result, which would
potentially be out of control.

### Delete All The Things

Deleting all the things (all objects of a given type) requires visiting all the shards, just like streaming all of
them. Similarly to the visitation problem there are availability problems here. Probably what would need to happen
is a job would need to be published to every shard that says hey, delete all the objects of type X from the store.

### Huge assoc lists

Since an association list is stored always on the same shard as the object it originates from, there are implicit
limits in the number of associations which can be stored for any object; given 64-bit sharding, then only 64-bits
worth of objects may be created, or about 4 billion associations.

I think the natural reaction to this is "when would you ever want to store let alone process 4 billion associations",
and the natural answer is when you are mis-using an object association to serve as a directory (for example, all
items in inventory, all people in the world, etc). It might be tempting for an application programmer to do this when
the system does not provide a way to iterate "all" things.

## Business-leven Consistency

If relaxed consistency is the only mode that is provided, it may become difficult for certain types of business
processing. Bank transactional information, for example, seems on the surface to recommend strong consistency or else
an attacker could exploit delayed consistency to gain an advantage (in fact, isn't this the heart of many forms of
check fraud). MongoDB has the advantage here because it provides "both".