Data analysis and query are essential for data computing. Report data source preparation and data management & ETL also involve data computing. Although SQL, R, Java, C language and other current solutions have powerful computing ability, coding for complex computing is rather cumbersome. Even R language is agile, but it’s too difficult to understand for nontechnical people. In order to enable convenient complex computing, esProc Dev comes in users’ sight.

The development environment of esProc is very easy to use. The script is written in the grid, which avoids formatting and the script is clear and easy to read. For instance, the script will be auto-aligned according to the gridline. For long scripts, such as judgment statement, loop statement, etc., indention can directly represent the scope of computing. Besides, the step by step computing mode enables complex computing goal to be divided into multiple simple steps, which greatly simplifies complex problems. With grid-style interface, esProc can easily realize step-by-step computing and results reference.

Better debugging features of esProc brings development efficiency. esProc provides the perfect debugging functions, with support for the breakpoint, single step, and run to cursor functions. Unlike many long SQL/SP statements, which can only be composed and comprehended as a whole, esProc enables separate script writing. In this case, the internal working details can be monitored. For example, for grouping and summarizing, esProc can implement them in 2 steps, while SQL can’t. What’s more, esProc supports explicit set, ordered computation, object reference, real grouping and cross-database computing, which makes it have more efficient development capability than SQL.

esProc has powerful data processing ability. It’s known to all that Java is not good at complex computing, especially the structured data computing. But esProc provides a richer and more complete system of structured data computing and easily achieves different complex computing demands. It can also seamlessly integrate with the main program in the form of standard JDBC embedded into Java applications. In addition, for a system where Java reporting tool is adopted, esProc is ideal to perform the complex computation, compute with multiple data sources, and clean the dirty data sources. The reporting tool can receive the result returned by esProc via JDBC interface by taking esProc as a database.

Related Article

How to Easily Calculate Sales Data with Spreadsheet?

Both R language and esProc are good at agile syntax. Their difference can be illustrated with the below examples:

Take computing the quadratic sum of various vector members for example. Although both R and esProc provide some functions to compute the quadratic sum, we are not going to use the existing functions in this example. Instead, the most fundamental functions will be adopted to implement this function.

R solution:
01 A4<-c(1,2,3)
02 Reduce(function(x,y) x+y*y, c(0,A4))
esProc solution:
A1 =[1,2,3]
A2 =A1.loop@s( 0; ~~+~*~ )

Comments:
In the example codes of R, the function Reduce is a quite useful function to implement a great many of advanced functions. In addition, the lambda syntax is also adopted for R users to construct the simple functions easily with concise codes. In addition, to facilitate the computation, one extra 0 is required to fill into the original vector.

esProc users can use the basic looping function loop to implement the similar function, not requiring the lambda syntax (although it is supported), and not having to add extra 0 (although it still requires a 0 as initial). Comparatively, the esProc code is much easier to understand. In addition, esProc can simply use ~ to represent the “member in computation” directly, not requiring an additional function to represent it. While the R users cannot represent it directly, they will have to construct a function and use xy to represent the “member in computation”. It is obviously very inconvenient for R users. Let’s take the moving averages computation below for example:

Based on the Orders table from Northwind, compute the moving averages of freightage in three days.

R solution:
filter(result$Freight/3, rep(1, 3),sides = 1)
esProc solution:
=A1.( ~{-1,1}.( Freight).avg())

Comments:

Because R cannot represent the “member in computation”, it is hard to compute the moving average with the basic functions. To reduce the difficulty, we will have to establish a customized advanced function filter at the cost of spending extra time of users to study the new function.

As for esProc users, to make up the advanced function to compute the moving averages, they only need the basic moving average function avg (of course the similar functions are also available in R) plus the featured member representations.

In this case, the {-1,1} is to represent the relative position in esProc. The -1 indicates the previous member, and the 1 indicates the next member. The {-1,1} indicates the range of members: there are 3 in total. Representing the relative positions in this way is easy to understand and use. Comparatively, the R users can represent the absolute position directly but hard to represent the relative position. This is not a minor obstacle. Let’s check the below example.

It is an example of period-on-period comparison from another essay I composed, Computation after Grouping – R Language vs. esProc. To demonstrate it in a more simplified way, let’s suppose the data is sorted by time. In this case, only compute the freightage compared with the previous period.

esProc solution:
=A1.((Freight-Freight[-1])/Freight[-1])
R solution:
c(0,result$Freight[-1]-result$Freight[-length(result$Freight)])/result$Freight[-length(result$Freight)])
Comments: esProc solution is quite intuitive and straightforward, that is, ”(current-previous)/previous”, in which the [-1] indicates the previous one.

R users can use the [-1] but to represent the “remains after removing the first record in the vector”, instead to represent the “previous”. Because no representation of relative position is available, the row-moving method is adopted, that is, to construct a new column and then move the data of each row downward by one row. R users will need to convert the computation between rows to that between columns. As can be seen, although the result is correct, the algorithm is quite perplexing to those who are not expert at R.

These examples prove that the “current member” and “relative position” of esProc is very characteristic and impressive to simplify the data computation greatly. Thus users only need to memorize few basic functions to implement a great many of advanced functions, saving lots of effort to learn. This is good news for lazy bone like me. Undoubtedly, there are quite a lot of similar syntax, such as the function options and cascade parameters.

The function option is a way to use the symbol to expand the common functions/features of functions, such as to change the type of return value and to determine whether to sort the results. For example, the group is a function for grouping, group@o indicates not to sort out the result, group@z indicates to sort out in reverse order, and group@1 indicates that only the first record of each group will be retrieved. Then, the result of this type is not a set but a TSeq (corresponds to the list and dataframe of R). The common functions/features refer to those can be implemented through other functions by adding @o@z@1, for example, the filter function select.

R uses various function names and parameters to expand the functions/features of functions, such as tapply, sapply, lapply, by and other functions. Although they are in effect just the extension to the cycling and computing function apply, you will have to grasp 5 functions at least before you can declare that the cyclic computation is grasped. To this point, it is self evident which way requires less functions to memorize. You may already have an idea of which solution demands less to learn.

As for the cascade parameters, this document will not discuss it in details. The interested readers can research it for themselves.

Finally, R excels in the matrix computation with a great many of fixed analysis algorithms. Plus, the customized library functions are also excellent. I am fascinated with these advantages. However, I must admit that esProc beat R for the basic characteristics of being easy-to-learn/use as well as the syntax agility.

With JDBC package, esProc can be called by Java application as an embedded database, and then return the results to main program to further process and utilize. Its method of invoking conforms to the public JDBC specifications. The result will be returned in the form of stored procedures, just as same as a standard stored procedure. esProc can receive parameters before called, and return multiple results after called.

public void testDataServer() {
      Connection con = null;
      com.esproc.jdbc.InternalCStatement st;
      try {
            Class.forName("com.esproc.jdbc.InternalDriver");
            con = DriverManager.getConnection("jdbc:esproc:local://");
            st = (com.esproc.jdbc.InternalCStatement)
            con.prepareCall("call findState(?)");
            st.setObject(1, "TX");
            st.execute();
            ResultSet set = st.getResultSet();
            ……
      }catch (Exception e) {
            System.out.println(e);
      }finally {
            if (con != null) {
                  try {
                  con.close();
                  }catch (Exception e) {
                        e.printStackTrace();
                  }
            }
      }
}

• Plug-and-use desktop BI tool to access datasource by connecting database or importing files
• Rich and perfect format setup and pagination features to support various charts
• Excel-style interface for easier report design and more convenient operation
• Rich data analysis features to complete various analysis operations like statistics, grouping, sorting, and filtering.
• System Requirements:

  JDK version: 1.5 or above
  Memory: 256 MB or above
  Support all RDBMS via JDBC

Related Readings:
A Desktop Application of Install-and-Play Design

• Explicit set
• Multi-level structured table
• Multi-level association
• Homocell model

esCalc has implemented the complete set-lization for you to conveniently perform the set operations, such as intersection, complement, and union.With the multi-level structure, the practical business can be described in a better way to implement the real grouping. Based on the multi-level structure, various data file can be associated to implement the join and union operations between tables, and thus the data can be integrated. The data at the same business level will be stored in the homocells. The formula in cells will be analyzed intelligently according to its position, auto adjusted and pasted, and the calculations in all homocells will be completed instantly.

Calculator-style Operation Mode

• Step by step operation, concrete presentation of the result of each step, and the next step calculation can be based on the previous one.
• Unnecessary to design the whole route beforehand. What’s the next step should do is determined tentatively. Non-technical person needs to be involved in the whole process.
• Different the normal calculator for numerics, esCalc works on the whole data sheet instead of a single value.

esCalc adopts the calculator style reporting calculation mode. The step by step calculation mode makes the objective for each step clearer, and no modeling beforehand is required. The result is presented in details, intuitive and easy-to-understand, users don’t need abstract ability to solve them.

esCalc is not a tool to calculate the numeric value, but to handle the data in the spreadsheet, calculate and summarize the data, or perform various analysis operations such as sorting, filtering, and grouping.

• esProc is a developer tool for business computing.
• In esProc, the computing result can be accessed through standard JDBC interface.
• Able to act as the layer of data computation of Java application
• Able to act as the datasource for reporting tool.
• Support any Java Web Server, and platform independently.
• System Requirements:

  JDK version: 1.5 or above
  Memory: 256 MB or above
  Support all RDBMS JDBC

Related Reading:
Being accessed by Report Tool via JDBC
JDBC Interface Support to Invoke External Java Application

Desktop Environment

• Desktop analyzer without any service end.
• Execute instantly when entering the expression, the result is available meanwhile.
• Support various parameters and macro to perform the analysis procedures repeatedly.
• System Requirements:

  OS: Windows (all versions), including Windows XP and the latest Windows 8
  JDK version: 1.5 or above
  Memory: 256 MB or above
  Support all RDBMS JDBC via JDBC

Related Reading:
Instant Scripting & Instant Analysis
A Desktop Application of Install and Play Design

Command Line

• Support direct scheduling at specified time with the schedule function of operating system
• Support the data management operation
• Support the lightweight ETL procedure
• System Requirements:

  JDK version: 1.5 or above
  Memory: 256 MB or above
  Support all RDBMS JDBC via JDBC

Related Reading:
Calling from Command Line

• Cellset-style script
  esProc adopts the cellset-style script, the cells are natural ly located and formatted. The computation steps can be spreaded in the cells neatly and clearly. The results of previous computations can be referenced easily with the cell name.
• Agile syntax
  The structure of esProc syntax is simple and clear, with concise and rich expression.
• Instant Scripting
  esProc supports instant operation. The computation in the latter step can reference the results from the previous and current steps. Users can monitor, analyze, and script at the same time.

Related Readings:
Neat and clear codes
Instant Scripting & Instant Analysis

Innovative Model Transcending Computing Level of SQL

• Step-by-Step
  The step by step computation belongs to the scientific methodology, that is, a complex objective can be divided to several simple and small objectives.
• Explicit set
  esProc has implemented the set-lization completely for you to conveniently perform the set operations, such as intersection, complement, and union.
• Ordered computation
  The computing of the top N, period-on-period comparison, and comparison with previous period is the typical algorithms related to “being ordered”. The ordered computation can solve a great many tough problems of data analysis.
• Object reference
  In esProc, ”.” is used to reference the associated tables and fields. Such expression is simple and easy-to-understand.

Related Readings:
Omnipresent sets
Locate by serial numbers
Compare over the previous period for consecutive intervals
Reference of the record associated by foreign key

Developer tool for business computing
esProc is a developer tool for data computing with higher development efficiency, better debugging features, easier codes maintenance, and Big Data support. It is database computing script with more advanced core model, and specializes in complex computing objects.

• Independent data-computing layer for Java application
  The JDK provides few functions for structured data computing, while esProc can effectively enhance the ability of Java in this respect. esProc provides a richer and more complete system of structured data computing than SQL, easily achieve various of complex computing demands, and seamlessly integrates with the main program in the form of standard JDBC embedded into Java applications.

  esProc separates the complex computations from applications and databases, thus effectively reducing the burden of database (The costs of database expansion are high). esProc can also be applied in situations where there are no databases but still need to do batch computing.

• Datasource for reporting tool
  For a system where Java reporting tool is adopted, esProc is ideal to perform the complex computation, compute with multiple data sources, and clean the dirty data sources. The reporting tool can receive the result returned by esProc via JDBC by taking esProc as a database.

  esProc supports various data sources such as the database driven by JDBC and the non-database source like Excel, Txt, etc.. esProc can access the data from multiple and diversified data sources for interactive computing. And the result is exported as a single data source that can be invoked by reporting tools or other external applications.

  Related Articles:
  Called by Reporting Tool via JDBC
  Fit for the Heterogeneous Data Environment

• Lightweight ETL
  Although esProc is not a professional ETL tool, it can be used to save you from the cumbersome SQL/SP and provides the application system with ready-to-use data. esProc has powerful data processing ability. The unordered data can become clean and usable through Extraction, Transformation, and Load (ETL).

Desktop BI Tool
esProc is a database script, enabling agile and easy-to-use statement for the interactive analysis of structured data, and is especially good at dealing with complex, flexible or occasional data.

• Plug-and-use, without any deployment
  esProc is a desktop BI tool to help users complete a series of computing independently, especially for complex analytic goals.

  Related Readings:
  A Desktop Application of Plug and use Design

• Without modeling beforehand
  esProc does not require data modeling in advance, allowing users to freely conduct data analysis. It can conveniently reference and process the prior calculation results and realize multi-step complicated analysis, capable for real-time computing and analysis.

  Related Readings:
  Prepare Test Data for Sales Management System
  Statistics on Sales Values of the Top 3 Salespersons Distributed in Respective Product Categories

For example, the sales details of a Volkswagen 4S shop, which are the purchase records of customers in various periods is presented in the below table.

We need to compute the link relative ratio of each month (Supposing the current month is December of 2012), and the year-on-year basis of sales volume of each month. We keep the detail data for the next computing. The result will be like this:

Users can easily implement the computing in the red box, since they only need to filter, sort, summarize in groups, and fold the data. But the link relative ratio calculation (i.e. column LRR) is not so simple. It seems that you only need to write =C458/C4 in the F458 cell, drag or copy it to the column F or other cells. Actually, you’ll find problems since the formula in the cell F890 will be changed into “=C890/C436” but not the expected “=C890/C458”. The reason for this is that the traditional spreadsheets only mechanically calculate the offset when copying the formula. For instance, 458 – 4 = 454, when you bias 890 by 454, the result will be 436. Therefore, you have to manually input formula to have a correct computing. But when the data is huge, the workload will be greater.

In addition, the meaningless formula will definitely appear in the detail cells like F1324, because the common business spreadsheet software cannot differentiate the summary section and the detail section. If dragging formula to the summary section, these formulas will be copied and pasted to the detail section automatically. But such “automation” is obviously not expected, and we have to input the formula manually since there is no other choice.

There is also similar situation when calculating the Year-on-Year (i.e. YOY) column: undistinguishable summary and detail, incorrect formula paste, and wrong formula in the section of details data.

However, esCalc can solve the above mentioned problems efficiently. It is the business spreadsheet software with the “homocell” functions. With multi-level structure of this powerful business spreadsheet software, mass computing is realized easily. For instance, in the Summary section, any formula entered will be copied and pasted to the cell with the same business status (i.e. other summary sections), without any impact on the detailed data. By entering the formula once, the homocells will be adjusted according to the business logics automatically. For example, if you write “=C458/C4” in cell F458, “=C890/C458” will appear automatically in the cell F890. In this case, only two formulas can solve such kind of problems with esCalc, the business spreadsheet software. The formula for link relative ratio is showed in the following figure:

The year-on-year basis is:

Related Article

How Well Do You Know Business Spreadsheet Software?

Report calculator for desktop BI should support the export formats generally supported by the reporting tools, for example, the text file and Excel spreadsheet. It should be able to support various intuitive data manipulation methods, allowing users to process the data conveniently and rapidly. Plus, such tools are expected to provide business-personnel-oriented UI, making it possible for users to complete the computation independently without having to rely on technicians. esCalc is just such a report calculator.

The application scenarios of report calculator are given in the below example. In a certain business report system, the query and presentation of orders in a given period is provided. The reporting results include Order number, Order date, Sales person, Clients, and Order value. Now, it is time to make statistics on the big clients accounting for 60% of sales in the recent half year. Because such calculation result is not provided in the report directly, and it is too pressing to request the developers to calculate in this case, business personnel will have to calculate all by themselves.

Business personnel will firstly query on the order data of the recent half year in the reporting system; secondly, export report results and import them to esCalc; thirdly, with several simple steps, achieve the computational goal, and print it out with the pie chart.

esCalc is also fit to calculate the below example:

Based on the reporting result in an Excel spreadsheet, calculate the increment of the year-on- year monthly sales.

Based on the text file exported from the sales report, calculate the sales persons whose monthly sales are ranking the top 3.

Through the table of website visitor volume, calculate the channels whose visitor volumes are among the top 10 in the consecutive 3 weeks.

Based on the report results of a telecommunications company, calculate the changing tendency of the average contribution of each customer during the Christmas season.

As the report calculator for business personnel, esCalc empowers the users with the below features:

1.Support the typical formats of report results

esCalc supports the tabular data import of Excel, including the Excel 97 version to the latest Excel 2010 version. Still the above example, esCalc users can export the order report from business system as Excel, and then import into esCalc, as the shown in the below screenshot:

esCalc also supports the tabular data import from text file, as shown in the below figure:

The text file and Excel spreadsheet are the commonest file formats. Almost all reporting tools support these two patterns, and most reporting results are the tabular data. Therefore, esCalc can handle the various reporting tools that the business personnel encountered in their daily work.

2.Support the calculator style operation

esCalc supports the calculator-style operation, which means that users can monitor the current data, perform an action according to the data characteristics, monitor the execution results, and then perform the next actions according to the current result. Unlike programmers who prefer to solve problems with VBA and development tools alike, business personnel prefer the simple and easy calculator-style operation to solve their problems

Still the above case, to make statistics on the big clients accounting for 60% of total sales for the company in the recent half year, users can calculate the sales achieved from each client, and then sort the sales data descendingly. The steps of this part are shown below:

Firstly, group by client. Right click on any cell (except for the header) of the Client column and then select Group. The result is shown in the below figure:

Secondly, calculate the sales achieved from each client. Enter the formula in the summary section of each group data. For example, when entering the formula ={E3}.sum in E2, other corresponding cells with the same business senses (called as homocells) will be generated and filled to E6, E8, and other cells automatically, as shown in the below figure:

Thirdly, fold the bands to the summary row. Because the detailed data is not currently needed, users can click the level number to hide them to have a clear view of summarized data, as shown in the below figure

Fourthly, sort. In any cell of column E (except for the column header), select Desc on the right-click menu and the data will be sorted descendingly, as shown in the below figure.

Each step described above can be completed in one action. In each step, users can monitor the result of the previous step, based on which, the computation can be carried on. This operation mode is designed on the basis of the findings on business personnel operation habits through a great many of studies, and is referred as Calculator-style Operation.

The Calculator-style Operation allows users to calculate independently without having to resort to technicians. Users can thus consider the problem according to the natural train of thoughts, and decompose a complex problem into several simple steps.

3.Support the arbitrary data processing

Considering the arbitrarily data processing, first and foremost, the processing method must be easy-to-use and strong. Users should be able to perform any processing on the data, and required to perform the minimized types of operations. Through a huge volume of demonstrations and tests, esCalc implements this functions through visualizing and proceduralizing the SQL syntax, including sorting, grouping, query, distinct filtering, and vertical or horizontal merges. The greatest difference from SQL is that the data is processed through the detailed and visualized menu and operations. The complex and hard-to-understand scripts language is not a must any longer.

The arbitrary data processing also refers to free transition between various processing methods. For example, for Excel, the level of difficulties between sorting-before-grouping and grouping-before-sorting vary greatly, whereas they are both simple for esCalc. Similarly, esCalc allows users to add the calculated column to the grouped data, or conduct the further summarizing. In addition, esCalc also supports the filtering, sorting, and other operations on the grouped data arbitrarily.

In the above case, clients are ranked by its contribution to the total sales descendingly. To compute the great client who contributes 60% of the total sales for the company, esCalc users only need to summarize the sales top-down and compare it with the 60% of the total sales. The steps are as follows:

Step 1: Calculate the cumulative sales for each client. Users only need to enter the formula into the column F for once, for example, enter this in F5: =F2+E5. Then, the correspondent formulas will be generated automatically in other homocells, as shown in the below figure:

From this step, we can see that esCalc not only supports the grouping, computational column, inter-row computation, but also the arbitrary inter-row computations on the grouped summary value, or adding the computational columns.

In the step 2, calculate the standard for comparison: the 60% of total sales. In this case, you are only required to enter the formula ={E2}.sum()*0.6, as shown in the below figure:

As can be seen from the above steps, esCalc supports not only the summarization expression, but also the further summarizing on the grouped summery value arbitrarily.

Step 3: Lastly, with the filter criteria just calculated, filter the cumulative value. To do this, right-click on any cell (except for the header) of column F and choose Filter, with the filter criteria @

From this step, we can see that esCalc not only supports the filtering, but also supports the further filtering on the summery value after grouping.

For esCalc, it adopts the more advanced tabular computational module, which enables the business personnel without technical background to leverage the data processing capabilities as strong as SQL. In addition, it guarantees the smooth transition between any operating steps. The arbitrary data processing allows business personnel to operate on the data freely in a natural brain of thoughts. This greatly strengthens the analysis ability of business personnel to confront to the complex and pressing computational task.

4.Support Business-personnel-oriented UI

As the report calculator tailored for business personnel, esCalc has investigated the use habit of many business personnel and provided the similar UI of Excel style. The comparison chart of Excel and esCalc is as shown below:

As can be seen from the above figure, esCalc has the column named after letters and the row named after numbers. Every cell has a unique cell name by which they can be referenced to each other. The formula syntax is similar to that of Excel. When constructing the levels, the cellset can be folded to the left or expanded. One thing to note is that the direct calculation of the summary value of column F is not supported in Excel, so that there is no column F in the Excel sheet shown above. Excel users will have to copy and paste it to another sheet to calculate over again.

Just like Excel, esCalc also allows the style settings, format painter, statistics charts, and print and export. Besides the similar appearances, esCalc also supports the normal and special copies & pastes of formulas like Excel. The only difference is that esCalc additionally supports the auto formula copy and intelligent formula adaptive adjustment, as shown in the below figure: