The Future of esProc

esProc’s long-term targets are the database and the data warehouse.  One day, it will confront the traditional database and data warehouse products in the market.

Here we’ll talk about databases and data warehouses separately. Databases are mainly for performing OLTP while data warehouses are used for performing OLAP. At present many commercial databases encapsulate the two into one product. The problem is that they are contradictory in certain aspects. OLTP requires transaction consistency and high concurrency but features small data scale for each task and relatively simple computations while OLAP have the opposite requirements and features. Implementing them in one product could cause unbalanced handling.

They need to be split to be implemented in two products. Thus each can be well handled. That is what we’re going to do.

Data warehouses mainly deal with computations. Their storage strategies serve the computations. Since esProc, as a computing engine, already possesses relatively complete computing capabilities, it needs an appropriate storage plan, which is necessary for achieving efficient computations, to make a database product. But it’s not suitable to include a too demanding storage strategy in case its openness is compromised too much. Anyway, the whole computing system won’t be absolutely open (data still needs to be preprocessed), but it is sure to be much better (with the ability of computing external data) than traditional data warehouses.

Yet, esProc not only aims high but is intended to last. Even after the data warehouse product is successfully developed, esProc will still be useful as a computing engine and be maintained and updated continually. Since applications generate large amounts of external data but the data warehouse doesn’t have an equally open computing ability and is hard to be embedded in an application, the open and integration-friendly computing ability will be always in demand.

The database product targets cloud applications. The cloud application will break the wall between users. A cloud application vendor doesn’t deploy software for each user as a traditional application vendor does, but provides one system for all users in the cloud. Vendors don’t upgrade applications version by version; instead they offer hot upgrade, which requires the application be online forever. Users have different data structures and the data structure of one user varies according to time. Huge amounts of data of different structures will thus be accumulated. In that case, the storage and computations are completely beyond the capability of relational databases.

NoSQL products do support various structures to some extent but they don’t support transaction consistency. Relational databases are the opposite. The cloud database developed from esProc integrates both features – ensuring transaction consistency while supporting different data structures – and uses esProc to provide the strong computing ability.