esProc is the middleware designed specifically to undertake the computing jobs between database and application. For the application, esProc provides the easy-to-use JDBC interface; For the database, esProc is powerful in parallel computing. By implementing the computing outside the database and in external storage, esProc alleviates the computing, storage and concurrency pressure on the database; effectively cuts down the cost of database software and hardware; and greatly optimizes database management.
Alleviating computing pressure. Too much computing will occupy a lot of CPU utilization and memory resources of the database server. Such cases include the data-intensive computing and computation-intensive tasks involving only a small amount of data. Consequently, long response time will deteriorate user experience and the overloading will shorten the life of the server. esProc has a complete and well-defined computing system, making it fully capable of sharing the workload for the database, and easily undertake various computing tasks, difficulty or easy, from the application. In addition, esProc supports parallel computing across multiple nodes, enabling it to equally distribute the task involving massive data or computations to multiple average servers or inexpensive PCs. In this way, users can achieve an overall outstanding performance much more superior to database servers while keeping a far lower provision of node hardware than that of the database servers.
Alleviating storage pressure. For the sake of safety, performance and usability, all kinds of application systems will not manipulate the core business data directly. Instead, the core business data will be processed into the form of temporary table/intermediate table for the direct use by application. Such redundant data and the associated SP would occupy a great many of storage spaces in the database, resulting in great storage pressure on database. esProc features the mechanism of external memory computing, which stores the intermediate database data into files on inexpensive HDDs and thus remove the need for database storage spaces. Through esProc JDBC, applications can access to the data stored in external storages and perform the standard structured data operations like query, compute, and modify. esProc also supports the external storage file retrieval and write-back to HDFS, which has a much greater storage space than that of any high capacity database.
Alleviating concurrency pressure. The enterprise level report applications or Web applications generally involve a great number of concurrent accesses and ultimately incur great pressure on database, which would reject the computing requests from users because all connection quotas have been used up. On the one hand, it is an effective and expensive means to solve the problem by upgrading the database hardware and buying more user licenses. On the other hand, limiting the number of concurrent connections and reducing the response will cut down the database cost while compromising the user experience and productivity. esProc solves the dilemma that the database cost can be lowered while alleviating the pressure of concurrency on database by supporting the parallel computing. The compute-intensive task can now be equally divided and allocated to multiple average PCs. Each node only needs to undertake a small amount of computing jobs.
Data administration tree. A database is a flat directory structure where all data tables are placed together linearly. Such situation is also true with SP. What’s worse, the intermediate table and its corresponding SP can only be stored separately in different directories. The flat directory structure is the root of disorder in data management/permission and application management. The public and private SPs become entangled and the intermediate tables for the sales or the production application get messed up. By comparison, esProc features the trouble-free external memory computing that manages source files and script files with an organized file system in a directory tree, where data can be placed in different directories according to the category and hierarchy and various sub-directories will be arranged for various modules, names of directories clearly indicate and differentiate the data from the new and the legacy systems, the script file and its source data can be stored in the same place, and the import, export, backup and recovery can be done with several simple actions of copy & paste. With esProc, data management becomes simple and clear.
Cutting database cost. With esProc users can store the core data in the database, with relatively low cost in management and hardware and fewer user licenses for maintaining and calculating important data and security management. The intermediate tables and script files derived from the core data are all stored outside the database. By doing so, the high-concurrency, big data parallel computing can be implemented on the inexpensive PCs. The data expansion can be solved easily with the HDFS and inexpensive HDDs. By reasonably leverage resources, the workload pressure on database can be significantly reduced, and database cost kept under control, management problems solved effectively, and various database applications handled with ease. These applications include the real-time high performance application, non-real-time big data application, desktop BI, report application, and ETL.