esProc
Agile Data Computing Engine

esProc - Agile Data Computing Engine

What’s esProc?

  • Report Data Source
  • JAVA Computing Layer
  • Offline Free Computing

What does esProc solve?

Report Data Source

  • The SQL for preparing data for reports is several K bytes long, which is difficult to write and more difficult to maintain.
  • For large amount of data, report presentation is too slow, sometimes overflow happens. Users often complain.
  • ...

JAVA Computing Layer

  • Java computing is too cumbersome. It takes hundreds of lines to write a simple filter grouping.
  • If the algorithm logic is changed, the whole application will be restarted and it will be difficult to provide continuous service.
  • ...

Offline Free Computing

  • A large number of TXT/CSV/XLS have to be processed, and SQL is NOT available. Everything is very complicated.
  • SQL is not easy to write, Python is also difficult to learn, it always requires professional programmers to do, and the cost is high.
  • ...

Report data source

more

JAVA Computing Layer

more

Offline Free Computing

more

APPLICATION

case 1 An Insurance Company - Storage Procedures outside the database

Vertica does not support stored procedures. Complex business logic can only be written in Java, which is cumbersome and difficult to maintain.

User Evaluation

  • The best use for us is to pass parameters to the Vertica database.
  • Each cell becomes a data array that are easy to use, compare and manipulate. It is very logical and you have made it user friendly.

Application Effect

  • Make Vertica support stored procedures outside the database.
  • The procedural algorithm is simple and friendly to implement.
  • BIRT data preparation is simple and efficient.

case 2A Bank-Computing Middleware

There are 44 kinds of Excel, which are parsed by Java and stored in database. The development cycle is long and hard-coding is difficult.

Reducing Development Cost

  • All 44 kinds of Excel parses have been reduced from 32 person/day to 6 person/day

Improving Development Efficiency

  • Each Excel parsing code volume changes from 100 lines to 3 lines

Reducing maintenance cost

  • Except for the short code size and easy maintenance, the hot deployment of the script takes effect as soon as the modification is made.

How does esProc do it?

Openness

Independent of database computing power

= lighter architecture

Language

New syntax adapts to complex Procedure computing

= Codes are much simpler

Performance

Easy to implement high performance algorithms and parallelism

= Stronger computing power

Application

Standard Interface for Embedded Integration

= Application deployment is easier

Independent Computing Engine

It does not depend on the computing power of the database and has a complete agile syntax system.

No database is needed in the computing process

It’s not converted to SQL for database execution.

Computing can be performed for any data source.

Agile Programming Language

SPL(Structured Process Language)

Combining the advantages of SQL and Java
  • Adapt to set batch operations (SQL Advantage)
  • Good at step-by-step process computing (Java advantage)
Well-designed rich library functions and consistency grammar
  • It's easier to master and performs better than Python

eg.Count the longest consecutively rising trading days for a stock?

Python

1 import pandas as pd
2 def iterate(col):
3 prev = 0;
4 res = 0;
5 val = 0;
6 for curr in col:
7 if curr – prev > 0:
8 res += 1;
9 else:
10 res = 0;
11 prev = curr;
12 if val < res:
13 val = res;
14 return val;
15 data = pd.read_excel(‘D:/Stock.xlsx’,sheet_name=0). sort_values(‘tradeDate’).groupby(‘Company’)[‘closePrice’].apply(iterate);

SQL

1 select max(continuousDays)-1
2 from (select count(*) continuousDays
3 from (select sum(changeSign) over(order by tradeDate) unRiseDays
4 from (select tradeDate,
5 case when closePrice>lag(closePrice) over(order by tradeDate)
6 then 0 else 1 end changeSign
7 from stock) )
8 group by unRiseDays)

SPL

  A
1 =stock.sort(tradeDate)
2 =0
3 =A1.max(A2=if(closePrice>closePrice[-1],A2+1,0))

Syntax suitable for describing a natural way of thinking!

Professional grammar in the field of set operation, code is more concise for the same process!

Application Integration and Management

esProc is developed in JAVA and provides standard application interfaces for seamless integration.