Data computing in Java programs

For various reasons, we often need to compute data in Java programs.
According to system design requirements, business logic is separated from data storage.
Data is scattered among different data sources.
Computing logic is complex and it is difficult to implement in SQL.

The Pain Points of Implementing Data Computing in Java

Large amount of code
Difficult to implement complex algorithms
High coupling
Difficult to maintain

A JAVA example to implement grouping

Such a long code only realizes the function of single field grouping and single field aggregation

Solution:Java computing layer - esProc

Note: SPL is the scripting language used by esProc. DFX is the script file, which is equivalent to an external stored procedure. Java uses the JDBC interface to import SPL statements to realize structured calculation or script execution.

Professional Data Computing Language SPL

  • Combination of discreetness and set orientation
  • More than 300 Structured Computing Functions
  • Characteristic cell-style coding makes layer clear
  • Memory, External Storage Cursor, Cluster Operation
  • Loop and Branch Adapts Procedural Computation
  • Step-by-step calculation and perfect debugging function

Abundant data interfaces

  • RDB:Oracle,DB2,MS SQL, MySQL, PG,….
  • MongoDB,REDIS,…
  • Hadoop:HDFS,HIVE,HBASE
  • TXT/CSV,JSON/XML,EXCEL
  • HTTP, ALI-OTS
  • File SQL interface

Professional Jar packages

  • Standard JAR package, easy to integrate
  • Multithread Parallel Computing
  • Perfect exception handling

Examples of implementing grouping with SPL

Conventional grouping

Summarize the number of days on duty for each person
A
1 =file("/Users/test/duty.xlsx").importxls@tx()
2 =A1.groups(name;count(name):count)

Such simple operation can also be written directly in SQL :

SELECT name,count(name) FROM user/test/duty.xlsx GROUP BY name

Top N of each group

Obtain overtime records for each month, each person, and the first three days
A
1 =file("/Users/test/duty.xlsx").importxls@tx()
2 =A1.groups(month(workday):mon,name;~.top(3):top3)

Alignment grouping

List in order the number of countries using Chinese, English and French as official languages
A
1 =connect("mysql")
2 =A1.query@x("select * from world.countrylanguage where isofficial='T'")
3 [Chinese,English,French]
4 =A2.align@a(A3,Language)
5 =A4.new(A3(#):name,~.len():cnt)

Rich class library

Designed specifically for structured data tables

Reduce Java coupling

JAVA hard coding

Compile and execute

Hard coding algorithm must be compiled and packaged together with the main application

Difficult to modularize

Hard coding algorithm and main application are class dependent, hard to isolate and highly coupled

Hard to hot switch

After the hard coding algorithm is modified, the whole application needs to be recompiled, deployed and restarted, so it is difficult to achieve hot switching

esProc

Interpreted execution

esProc algorithm does not need to be compiled in advance

Easy to modularize

The script file of esProc is independent of the main application and can be placed externally in Java and maintained separately

Easy to hot switch

No need to compile and no need to restart, hot switching is easily achieved

Diversity data source support

Java computing ability is not competent for diverse data sources.

The built-in interface of esProc can easily access:

  • Commercial RDBMS:Oracle, MS SQL Server, DB2, Informix
  • Open source RDBMS:MySQL, PostgreSQL
  • Open source NOSQL:MongoDB, Redis, Cassandra, ElasticSearch
  • Hadoop:HDFS, HIVE, HBase
  • Application software:SAP ECC, BW
  • File:Excel, Json, XML, TXT
  • Others:Http Restful, Web Services, OLAP4j, ...

Cross database computing support

Standard invocation interface

Java code

...
Connection con = null;
Class.forName(com.esproc.jdbc.InternalDriver");
con= DriverManager.getConnection("jdbc:esproc:local://");
// Calling stored procedures ,CountName is the file name of dfx
st =(com. esproc.jdbc.InternalCStatement)con.prepareCall("call CountName()");
// Execute stored procedures
st.execute();
// Get  result set
ResultSet rs = st.getResultSet();
...		
Hot deployment of external algorithm reduces coupling effectively

Resource Link