LumoSQL protects data on mobile phones using a new data storage technology which is highly compatible with most existing devices. With LumoSQL, the device owner has ultimate right to decide who can read or change their data… and this decision continues to be enforced even after it has been copied off the phone to (for example) a bank or insurance company for processing with their in-house database software. In contrast, the situation at present is that device owners are rarely in control of the privacy of their own data, despite many laws relating to privacy.

If a criminal or government officer takes a phone away from its owner, LumoSQL data cannot be read without the consent of either the phone owner or someone(s) to whom the phone owner has granted access. This is fine-grained, meaning different levels of permission can be granted.

Why this matters

A typical mobile phone stores all its non-streaming data — contacts, messages, health records, browsing history, app settings — in several hundred small databases, all using the same software: SQLite. SQLite is very likely the most-deployed software in the world, by a factor of at least four zeros. It is likely the only trillion-scale software in existence. Web browsers use it. Operating systems use it. Vehicles use it. It is a standard data format relied on by companies such as Airbus whose products last decades.

And yet SQLite stores all of this data unencrypted. Features such as whole-device encryption do not really address the problem — once the phone is unlocked, every app and every intruder can read every database. It is unacceptable to store personal data this way, but SQLite works so well and is so ubiquitous that people can’t imagine an alternative.

LumoSQL wants to be that alternative. It modifies SQLite (rather than replacing it) so that existing apps continue to work, but the data they store gains real protection. The key idea is a Lumion — a piece of data that carries its own access rules with it, like a letter in a locked envelope where only the people named on the envelope can open it, even after it’s been forwarded. A Lumion attached to an email, saved to a cloud backup, or copied into a corporate database still enforces its owner’s permissions. The phone owner decides who can read or change each piece of data, and that decision travels with the data wherever it goes.

The detailed strategy document LumoSQLMotivation-1.0.pdf explains the social, business and technical pressures behind LumoSQL. As the strategy document says, “LumoSQL assumes software development should never be relied on and is getting worse”.

Surprising facts about SQLite

The following facts often surprise people:

  • SQLite is a full-featured database, supporting the standard SQL language despite being tiny compared to all the other mainstream SQL databases.
  • SQLite is open source, exceptionally well-maintained, mostly by just 3 people. Many more people contribute occasionally to SQLite, and the community of deeply technical users is very large.
  • SQLite version 3 is a standard data format, and relied on by companies such as Airbus whose products last many decades.
  • SQLite is exceptionally reliable, given the policy decisions not to change certain fundamentals.

The corollaries to these unusual facts are significant:

  • This is uncharted territory for Computer Science: is SQLite’s ultra-conservative compatibility commitment to its (at least) hundreds of billions of installations the right choice? Is SQLite’s fast-moving support of formal database standards the best way forward? It does certainly work very well.
  • Of the many forks of SQLite, none seem to have more than a relatively trivial deployed base (likely at most a few handfuls of millions.) This includes seemingly very useful forks. It appears the nature of the SQLite project ensures its success and discourages replacements, but also constrains its future in important ways.
  • The obvious strategic problems with SQLite are not widely discussed. But why? Perhaps because SQLite works so well and is so ubiquitous that people can’t imagine an alternative. The world needs a new option, compatible with SQLite. LumoSQL wants to be that option, and more.

Funding and contributors

LumoSQL exists due to the volunteer effort contributed by many skilled people. The Vrije Universiteit Brussel funded valuable contributions in cryptography and mathematical analysis. The NLnet Foundation funded Phase I and Phase II.

For developers

For even more detail see the code development page.

LumoSQL is a modification (not a fork) of the SQLite embedded data storage library. It uses Predicate Based Encryption to implement Lumions — the encrypted, self-governing data units described above. LumoSQL offers multiple key-value backend storage systems selectable by the user, and features not found in any other mainstream database:

  • ability to checksum every row on write and verify on read
  • ability to trigger arbitrary functions on per-row read and write
  • a general test suite for benchmarking precisely how LumoSQL (or SQLite) is performing and the full context of that benchmark run. For some reason database benchmark is very poorly done, including by the TCP-C consortium founded for solely that purpose.
  • a general build system able to mix and match multiple versions of the database with multiple versions of multiple backends. Never before has it been possible to compare the different strategies of various Key-Value stores with the same database frontend.

If you are an SQLite user familiar with C development wanting an easier way to benchmark and measure SQLite, or if you are wanting features only available in other key-value storage engines, then you will find that LumoSQL offers new features even in its prototype stage.

In Phase II LumoSQL is implementing at-rest encryption and privacy using the features developed in Phase I, and readying LumoSQL for more general testing.

Notable outcomes from LumoSQL already include:

  • The only mainstream database with swappable Key-Value stores, where all stores are peers rather than one store having special knowledge that gives it technical advantages. The two stores we have concentrated on so far are (1) the existing SQLite store with optional, binary-compatible modifications for encrypted rows and tables and the associated metadata and (2) the LMDB memory-based store which may have advantages when uses with the most modern high-performance RAM-based storage hardware. We look forward to integrating other stores, and to prove the point we also supply the ancient Oracle BDB backend as an example third store.
  • The only mainstream database optionally without a Write-ahead Log, when using the LMDB storage backend.
  • preliminary Benchmarking results
  • The Not Forking tool, which avoids forks in both simple and complicated source code

Exciting things in progress:

  • Lumions are described in a draft RFC for universal encrypted blobs with authentication. For the first time, a piece of data can have all the security rights of a full database, even it is called “mydata.txt” and attached to an email. LumoSQL uses Lumions as rows and tables in SQLite but is just one use case. As soon as the cryptographic design has settled we will update this RFC and consult even more widely
  • Documented API for arbitrary key-value stores
  • Documented API for accessing the key-value stores via the SQLite library, instantly making the SQLite key-value store the most widely-distributed key-value store. Nothing calls the SQLite key-value store today except SQLite