The Future of esProc

esProc’s long-term targets are the replacement of databases and the data warehouses. One day, it will confront the traditional database and data warehouse products in the market.

Here we’ll talk about databases and data warehouses separately. Databases are mainly for performing OLTP while data warehouses are used for performing OLAP. At present many commercial databases encapsulate the two into one product. The problem is that they are contradictory in certain aspects. OLTP requires transaction consistency and high concurrency but features small data scale for each task and relatively simple computations while OLAP have the opposite requirements and features. Implementing them in one product could cause unbalanced handling.

They need to be split to be implemented in two products, thus each can be well handled. That is what we’re going to do.

Data warehouses mainly deal with computations. Their storage strategies serve the computations. High efficiency computations require a matching storage mechanism. esProc, as a computing engine, already possesses relatively complete computing capabilities and offers high efficient storage plans. On the basis of these advantages, only an excellent storage management mechanism is needed to make a data warehouse product. Anyway, the whole computing system won’t be absolutely open (data still needs to be preprocessed), but it is sure to be much better (with the ability of computing external data) than traditional data warehouses.

Yet, esProc not only aims high but is intended to last. Even after the data warehouse product is successfully developed, esProc will still be useful as a computing engine and be maintained and updated continually. Since applications generate large amounts of external data but the data warehouse doesn’t have an equally open computing ability and is hard to be embedded in an application, the open and integration-friendly computing ability will be always in demand.

The database product targets cloud applications. The cloud application will break the wall between users. A cloud application vendor doesn’t deploy software for each user as a traditional application vendor does, but provides one system for all users in the cloud. Vendors don’t upgrade applications version by version; instead they offer hot upgrade, which requires the application be online forever. Users have different data structures and the data structure of one user varies according to time. Huge amounts of data of different structures will thus be accumulated. In that case, the storage and computations are completely beyond the capability of relational databases.

NoSQL products do support various structures to some extent but they don’t support transaction consistency. Relational databases are the opposite. The cloud database developed from esProc integrates both features – ensuring transaction consistency while supporting different data structures – and uses esProc to provide the strong computing ability.