The problems that report data source faces

Complex logic SQL is hard to write

  • SQL is order-less, is not completely set oriented, and is difficult in procedure computing
  • Many projects do not allow report stored procedures
  • The grammar is not universal and difficult to migratet

Multiple data sources are difficult to handle

  • Special data sources such as Excel and NoSQL
  • Cross data source, cross database
  • Some reporting tools only support single data source

Custom data source (Java) is more difficult to write

  • Lack of structured class library and tedious code
  • High coupling and difficult to maintain

Introducing Data Computing Layer - esProc

Note: SPL is the scripting language used by esProc. DFX is the script file, which is equivalent to an external stored procedure. Report uses the JDBC interface to pass in SPL statements to realize structured calculation or script execution.

Simplify complex SQL / stored procedures

Count the longest consecutively rising trading days for a stock.

SQL

1 select max(continuousDays)-1
2 from (select count(*) continuousDays
3 from (select sum(changeSign) over(order by tradeDate) unRiseDays
4 from (select tradeDate,
5 case when closePrice>lag(closePrice) over(order by tradeDate)
6 then 0 else 1 end changeSign
7 from stock) )
8 group by unRiseDays)

Can you do it in a more natural way of thinking?

SPL

A
1 =stock.sort(tradeDate)
2 =0
3 =A1.max(A2=if(closePrice>closePrice[-1],A2+1,0))

Syntax suitable for describing a natural way of thinking!

External algorithm reduces stored procedures

Using stored procedure to implement data preparation algorithm will cause the coupling between report and database.

  • Stored procedure and report are stored in different locations, which makes it very difficult to be consistent.
  • Stored procedure modification needs to allocate corresponding database privileges, and there are security risks.
  • Stored procedures are easily used by other applications, resulting in coupling between multiple applications.
  • Many projects do not allow the creation of stored procedures for reports for security and permission purposes.
It will greatly reduce the number of stored procedures by replacing stored procedures with esProc to prepare data for report. An algorithm will be stored and managed along with the report template in file system and become a part of the reporting module. This will reduce its coupling with the other parts of the application or other applications.

Avoid complex Java programming

esProc adopts the set-oriented syntax, and the code is shorter than Java which does not directly provide structured computing.

Write faster and shorter

  • esProc provides a higher class library and method based on Java.

Easy to understand and troubleshoot

  • The proportion of pseudo real code is only about 1:1.5, and most report data preparation algorithms can be displayed in one screen.
  • More code can be seen in one page, and the meaning of code can be understood more completely, thus the troubleshooting is easy.

Interpreted execution reduces application coupling

To prepare data for report by using JAVA and esProc has the following differences:

JAVA

    Difficulties in modularization

  • Java programs must be compiled and packaged with the main application, which is highly coupled.
  • Hot switching is almost impossible

  • The modified report data preparation algorithm written in Java will lead to the recompilation and deployment of the whole application, and it is difficult to achieve hot switching.
  • Few class libraries

  • There are few class libraries in structured and semi-structured data computing in JAVA, and all algorithms need hard coding.

esProc

    Simple modularization

  • The esProc script file can be managed and maintained together with the report template, so that the report function can be modularized.
  • Hot switching is easy

  • esProc is the interpretive execution language , and it is easy to do hot switching.
  • Rich class libraries

  • There are rich syntax and class libraries, making the computing of structured data more efficient.

Support multiple data sources

The computing power of reporting tools are not competent for diverse data sources.

Computing layer processes diverse data sources:

  • RDBMS:Oracle、MS SQL Server、DB2、Informix、MySQL、PostgreSQL
  • NOSQL:MongoDB、Redis、Cassandra、ElasticSearch
  • Hadoop:HDFS、HIVE、HBase
  • Application software:SAP ECC、BW
  • File:Excel、Json、XML、TXT
  • Others:Http Restful、Web Services、OLAP4j 、...

Direct use of multiple data sources and cross database computing

Dynamic Data Source/Set

Dynamic Data Source

Switch database dynamically according to parameters

 ${pds}.query("select * from T where F=?",pF)

Dynamic Data Set

Generate algorithm dynamically. Concat dynamic SQL.

A B
1 =sums.array().("sum("+~+") as "+~).string() /sum(a) as a, sum(b) as b
2 =db.query("select G,"+A1+" from T group by G")
General code to implement algorithm, no need to modify during migration

Capacity control for result set

A B C
1 =db.cursor("select * from T") =A1.fetch(1000)
2 if B1.fetch@0(1) >B1.insert(0,"Continue") /Insert markers when uncompleted
3 >A1.close() return B1

Parallel access improves performance

Databases generally have a bad JDBC performance, but the reporting performance relies heavily on the data retrieval performance. esProc can create multiple database connections to retrieve data in segments by using multithreaded parallel processing, and the performance can be increased by several times.

A B C
1 fork 4 =connect(db) /There are four threads, and each connection is established separately.
2 =B1.query@x("select * from T where part=?",A1) /Retrieve each segment separately
3 =A1.conj() /Merge the results

esProc as BIRT data source – Dynamically parse csv files

esProc as JasperReport data source – Cross database join

External JAVA programs call esProc scripts through JDBC

JDBC class stored procedure calls SPL script file

...
Connection con = null;
Class.forName("com.esproc.jdbc.InternalDriver");
con= DriverManager.getConnection("jdbc:esproc:local://");
// Calling stored procedures ,CountName is the file name of dfx
st =(com. esproc.jdbc.InternalCStatement)con.prepareCall("call 
CountName()");
// Execute stored procedures
st.execute();
//Get  result set
ResultSet rs = st.getResultSet();
...

JDBC query files directly using SQL

...
Connection con = null;
Class.forName("com.esproc.jdbc.InternalDriver");
con= DriverManager.getConnection("jdbc:esproc:local://");
// Calling stored procedures ,CountName is the file name of dfx
st =(com. esproc.jdbc.InternalCStatement)con.createStatement();
//Query files using SQL,get result set
ResultSet rs =  st.executeQuery("$select name,count(*) from 
/home/user/duty.txt group by name");
...

Resource Link