It would be better to lessen the dependence on stored procedures by storing algorithms in a file system to reduce their coupling with the database. Since the database computing resources are limited, we should first move data out of the database before we perform the batch processing. Then esProc will take the place of the database to perform the data manipulation, saving the database resources and reducing the number of intermediate tables to make a slim database.
- Non-database computing speeds batch processing
- File-based algorithm storage lessens dependence on stored procedure
- Processing data outside creates a slim database
- Independent computing engine speeds up report development
Often the data obtained from the data center comes from multiple heterogeneous sources, yet both SQL and the stored procedure can’t process such type of data. esProc, able to handle operations on heterogeneous databases, can provide data service for the data interface by performing data cleansing, data masking and data manipulation beforehand.
- Prepare mixed data obtained from heterogeneous sources
- Easy to develop data service
SQL is good at set-orientation, but not at discreteness, making it awkward in describing complex computations. Based on the discreteness data set model, esProc integrates set orientation and discreteness, and goes further in set orientation. This makes post-grouping subsets, order-related aggregation, and inverse grouping possible and convenient, facilitates order-related calculations, making easier inter-row reference, order-related grouping and position-based operations, and enables the computation and analysis of large-scale data using stepwise cursor processing, program cursor and ordered cursor.
- Discrete data set model
- Functional syntax
- Complex analytic functions
esProc can pre-process data coming from a variety of sources, including txt, csv, xlsx, NoSQL, Json and xml, and then send it into the database. Extracting data into a specialized database for analysis, which isn’t a data warehouse is designed for, also wastes precious computing resources. With esProc, we can extract data from the disparate, heterogeneous databases as a temporary, intermediate file for cleansing, transformation and integration, and then load it onto the data warehouse.
- Pre-processing heterogeneous data sources out of the database
- Perform ETL through programming
- Data migration tools