Chapter 1 Gettting Started

Next
Previous
Related :
  • 1. Installation and Basic Use
  • 2. Cell Editing
  • 3. Basic Data Types
  • 4. Constants
  • 5. Types of Cells
  • 6. Multilayer Parameters
  • 7. Option Syntax
  • 8. Use of Common Data
  • 9. Basic Usage of JDBC
    • Copyright® 2015 RAQSOFT


1. Installation and Basic Use

esProc is a tool for programmed data computing. It can execute various types of data analysis and structured data computing, as well as get access to the database freely and perform online data analysis.

Next
Previous
Related :
  • Chapter 1 Gettting Started
  • 1.1 Installation
  • 1.2 Basic Use of esProc
  • 1.3 Installation and Operation under Linux System


1.1 Installation

You can download esProc installation package on the following official website:

http://www.raqsoft.com/esproc-download.html

To install esProc, please run the installer according to the instructions below step by step:

1) Run the installer

2) Click Next to continue, accept the agreement

3) Select the installation directory, click Install

4) Complete the installation

If you are familiar with the configuration of running environment of Java, and JDK1.5 or higher version has been installed locally, you can also choose to launch the esProc installer that will not install JDK automatically but prompt you to navigate to the directory of JDK in the local machine.

Next
Previous
Related :
  • 1. Installation and Basic Use


1.2 Basic Use of esProc

Run esProc main program to open esProcand click [image] button to create the cellset file.

On the left is the cellsetsection, the active cellsetfiles you are operating are shown here; on the right, the upper area is the valueview section and the lower area is the variable section. Value view section and variable section can be retracted and expanded easily.

In esProc, if a cell has a value, then its name can be used to reference the cell value in a computational cell or an executable cell. A cell possessing a value can be a constant cell, a computational cell or other types of cells got assigned by the executable cell. The name of a cell includes the letter representing the column in which the cell is located and the integer – the serial number of the row it resides. In the above cellset, the expression in cell A2 calls the value of cell A1. The following is another example:

But different to Excel, in esProc, the letter in the cell name must be capital.

esProc provides lots of functions to handle various kinds of data computing. For example:

And auxiliary function for editing functions can make code-writing in cells more easily. During editing expressions in cells, press Alt+↓to open/close the auxiliary function for editing functions. On the auxiliary interface for editing functions, you can see function syntax, function reference and other information:

Next
Previous
Related :
  • 1. Installation and Basic Use


1.3 Installation and Operation under Linux System

At present, ways of installing esProc under Linux are as follows:

1. deb file

2. rpm file

3. zip file

4. Download and install esProc by updating the software source

Only Linux systems that support deb files can use this kind of installation document, such as Debian and Ubuntu. For detailed information about installing and uninstalling esProc, please refer to Installation Instructions on esProc for Linux System.

By default, esProc will be installed in the path /usr/share/raqsoft. To start esProc, just execute startup-en.sh in the path esProc/bin under the installation directory:

To start the demo database, you need to execute startDataBase.sh under the above directory.

Next
Previous
Related :
  • 1. Installation and Basic Use


2. Cell Editing

Next
Previous
Related :
  • Chapter 1 Gettting Started
  • 2.1 Cell Copy and Paste
  • 2.2 Insert and Delete like Text-editing


2.1 Cell Copy and Paste

InesProc, you can pressCtrl+C and Ctrl+V to copy data or an expression insome cell and paste it on the targetedone. But unlike EXCEL, the copy and paste operations on the cell will not perform the adjust-paste onthe expression automatically.

If you need to adjust the cell quoted by the expression, press Ctrl-Alt-V to paste the cell. The expression will undergo the adjust-paste like what will happen in EXCEL.

Like the EXCEL, esProc set it as a rule that the row/column number starting with $ will never get adjust-pasted.

Next
Previous
Related :
  • 2. Cell Editing


2.2 Insert and Delete like Text-editing

On the cell, press Ctrl+Enter, Ctrl+BackSpace or Ctrl+Del will result in a similar effect of carriage return, backspace or deleting the current cell. This is similar to pressing Enter, Backspaceor Delete keys in a text editor. Pressing Ctrl+Insert will insert a cell in this row.

1) Pressing Ctrl+Enter will cause the effect similar to a carriage return.

2) Pressing Ctrl+BackSpace will cause a forward delete.

3) Pressing Ctrl+Delete will delete the current cell. The cellsfollowing it will move leftward to fill the space.

4) Pressing Ctrl+Insert will insert a cell, and the cells behindit will move rightward to make room for the newly inserted.

5) When the copy area is a whole row/column, pressing Ctrl+B will automatically insert the row/column first and then paste the content.

PressCltr+Alt+B, the expression will be adjust-pasted in a way similar to pressingCtrl+Alt+C.

Next
Previous
Related :
  • 2. Cell Editing


3. Basic Data Types

Next
Previous
Related :
  • Chapter 1 Gettting Started
  • 3.1 Data types in esProc
  • 3.2 Judge the data type


3.1 Data types in esProc

In esProc, the following basic data types are available:

Integer

Any integer between -231 and 231-1, i.e. the value range is -2147483648~2147483647. The type conversion function int() can be used to convert other types of data into integer.

Long integer

-263~263-1, which is a value range bigger than that for integer type. The type conversion function long() can be used to convert other types of data into long integer.

Long integer can be specially represented by appending a capital letter L to the integer. Compared with an integer, a long integer has a bigger value range; strings starting with 0x can be used to represent the hexadecimal long integers:

 | A
1 | =123456789L*1000
2 | =123456789*1000
3 | =12345678900*10
4 | 0x00FF

Since the value range of common integers is -231~231-1, that is, -2147483648~2147483647, the result in A2 is beyond the boundary. While by using a long integer in A1, the value range is increased to -263~263-1 and correct result can be obtained. In A3, the number 12345678900 is beyond the value range of common integers and will be parsed automatically into a long integer. It is a hexadecimal constant that cell A4 contains. Values of A1, A2, A3 and A4 are as follows:

As can be seen from the computed results of A1 and A3, if one of the operands involved is a long integer during the operation of a certain step, the result will be a long integer.

Floating-point number

64-bit floating-point number is the commonest data type in esProc. Almost all decimals-related computations are performed with this data type. The type conversion function float() can be used to convert other types of data to floating-point number. Because the floating-point number is used to store data according to the binary rule, there could be errors in the computation.

 | A
1 | 43.41
2 | -4.45E13
3 | 3%
4 | =float(54)

Values of A1, A2, A3 and A4 are as follows:

The floating-point number in A2 is represented by scientific notation; A3 contains a floating-point number represented by a percentage, which, with this writing form, can only be used as a constant instead of being used in an expression.

Big decimal

Big decimal can be used to store any real number error-freely, but more memory could be consumed when big decimal is used for computation and the computational efficiency is relatively low. The type conversion function decimal() can be used to convert other types of data (such as the string) into the big decimal.

 | A
1 | 5.2
2 | =A1*A1
3 | =decimal("5.2")
4 | =A3*A3

The 5.2 in A1 is a floating-point number; the code in A3 converts the string “5.2” into a big decimal. The computed results in A2 and A4 are as follows:

It can be seen that the operation using a floating-point number that works according to the binary rule could result in accuracy errors, while the operation using a big decimal is much more accurate.

Real number

The real number covers four data types: integer, long integer, floating-point number and big decimal. number() is the type conversion function available to convert other types of data to real number.

Boolean

It includes true and false

 | A
1 | true
2 | false
3 | =4>2
4 | =13%3==0

Values of A1, A2, A3 and A4 are as follows:

String

Double quoted in an expression, and the right slash \ is used as the escape character. If the string is defined in a constant cell directly, the double quotation marks are omitted. string() function can be used to convert other types of data to strings. When concatenating two strings x and y in an expression, you can add a space between them, that is, x y.

The escape rule of strings is the same as that of JAVA. For details, please refer to the manual.

 | A
1 | Amy
2 | ="State:\tTX"
3 | ="Texas" "Dallas"

Values of A1, A2 and A3 are as follows:

Datetime

Written in the format of yyyy-mm-dd and hh:mm:ss. Type conversion functions such as date(), time() and datetime() can be used to convert the string or long integer to date, time, or datetime.

 | A
1 | 2014-10-5
2 | 22:15:30
3 | =date("2014" "-10" "-5")

Values of A1, A2 and A3 are as follows:

Click Tool -> Options menu item, and set the format of time/date data and the character code, etc. on the Environment tab.

Precision may be sacrificed when a type of data is converted into another one.

Next
Previous
Related :
  • 3. Basic Data Types


3.2 Judge the data type

The type of data can be judged with the following functions:

ifnumber(x)

Judge if x is a real number

ifstring(x)

Judge if x is a string

ifdate(x)

Judge whether x is a date or a datetime

iftime(x)

Judge if x is a time

 | A
1 | =ifnumber(3.5)
2 | =ifstring(now())

In A1, the floating-point number 3.5 is a real number. now() function in A2 computes the current date and time and the resut is not a string. The results in A1 and A2 are as follows:

Next
Previous
Related :
  • 3. Basic Data Types


4. Constants

In esProc computing, we use constants frequently or sometimes store the data in the cellset directly. In this article, let’s learn the usage of constants in the esProc.

Next
Previous
Related :
  • Chapter 1 Gettting Started
  • 4.1 Directly using constant in the expressions
  • 4.2 Constant cells
  • 4.3 The sequence of constant cells


4.1 Directly using constant in the expressions

Constants can be directly used in the esProc expressions:

 | A
1 | =64*32
2 | =round(1.45*5.5)
3 | =isupper("Tom Sawyer")

When constants are used in the expressions, the integer and the float can be used directly, but the string must be enclosed in quotes. For the Boolean constants true and false, they can be used directly in the expressions, but there are few cases of using them. The results in A1, A2 and A3 are as follows:

In particular, the capitalized L can be appended to the integer to indicate the long integer. Compared with the integer, the long integer has a wider range of value; the hexadecimal long integer can be represented with a string whose first two characters are 0x:

 | A
1 | =123456789L*100
2 | =123456789*100
3 | =0x00FF

Because the value range of normal integers is -231~231-1, that is, -2147483648~2147483647, the result in A2 exceeds the value range of integer. In A1, a long integer is used, and the value range is increased to -263~263-1. By this way, the correct result can be obtained. The computed results in A1, A2 and A3 are as follows:

As can be noticed, during a certain step of the computation, if one of the operands involved in the integer computing is the long integer, the result is the long integer.

In a string used in an expression, if there are special characters like " and \, an escape character \ is required as an indicator before each special character:

 | A
1 | ="C:\\Program Files\\"+"Java"
2 | ="\"I can come today,\" she said, \"but not tomorrow.\""

After execution, the strings in A1 and A2 are as follows:

The time and date constants are not allowed in the expressions. Instead, only the type conversion functions, such as date(), time(), and datetime(), can be used to convert strings or long integers:

 | A
1 | =interval@y(date("1944-6-6"),now())

The result in A1 is as follows:

Next
Previous
Related :
  • 4. Constants


4.2 Constant cells

When the data is written to the cell directly, if the cell can interpret the string as a constant, then this cell can be regarded as a constant cell. Its cell value is the constant.

The default text in the constant cell is pink. Based on the data in the cell, the constant will be parsed into various data types. If the data is unidentifiable, it will be interpreted as a string.

The double quotation marks are not necessary when a string is defined in the constant cell.

In addition, in the constant cell, the constant can be represented as a percentage, like 5%:

 | A
1 | 5%
2 | =A1*1000

After the computation, the cell values of A1 and A2 are as follows:

In the constant cell A1, 5% will be converted to the corresponding float value 0.05. Note: The writing style of 5% cannot be used in the expressions. It is only valid in the constant cells.

In particular, the value in the constant cell can also be true, false or null:

 | A
1 | true
2 | false
3 | null

As can be seen from above, the constant cell value is the corresponding Boolean value or null value at this point:

The string constant cell can be used to specify the value of a cell as a string. The cell string in a string constant cell starts with the single quotation mark ' of English font. The string constant cell is used to define the strings containing special characters or those composed of figures:

 | A
1 | 'abc\d
2 | '1234.5
3 | 1234.5

In the above, the value of the constant cell A1 is the string abc\d; the cell value of A2 is the string 1234.5; the cell value of A3 is the number 1234.5:

In the constant string cells, the escape character \ is not necessary if any special character exists in the string:

 | A
1 | 'C:\Program Files\Java
2 | '"I can come today," she said, "but not tomorrow."

Values of strings in A1and A2 are as follows:

In a constant cell, we can also define the sequence constant with the brackets [], and use commas to separate the sequence members. For example,

 | A
1 | [2,3,5,7,11]
2 | [Gold,Silver,Bronze]

The sequence constants in A1 and A2 are as follows:

In defining a sequence constant, the double quotation marks are also unnecessary for indicating the string members in a sequence.

In an expression, we can use the cell name to reference the constant cell:

 | A
1 | 1944-6-6
2 | =interval@y(A1,now())

The result in A2 is as follows:

Next
Previous
Related :
  • 4. Constants


4.3 The sequence of constant cells

In esProc, once constants are saved in multiple cells, these cells are often combined to form a constant sequence for use in the expression. For example,

 | A | B | C | D | E | F
1 | 1 | 2 | 4 | 8 | 16 | 32
2 | =[A1:F1] |  |  |  |  |

In A2, the expression [A1:F1] indicates a sequence composed of the values of all cells in the A1~F1 area. The sequence is as follows:

The cells in the sequence can also be in a continuous area:

 | A | B | C
1 | 1 | 2 | 4
2 | 8 | 16 | 32
3 | =[A1:C2] |  |

In A3, expression [A1:C2] indicates a sequence composed of values of cells in an area from A1 to C2, and in which the members follow the order of first-row-then-column.

In esProc, an area of constant cells is often used to make up a TSeq:

 | A | B | C | D | E
1 | German | 2 | 1 | 0 | 7
2 | American | 1 | 1 | 1 | 4
3 | Portugal | 1 | 1 | 1 | 4
4 | GHana | 0 | 1 | 2 | 1
5 | =create(Team,W,D,L,Points) |  |  |  |
6 | >A5.record([A1:E4]) |  |  |  |

In A5, a new TSeq is created. In A6, the record function is used to fill the TSeq with the values of the constant cells in the area of A1~E4.

With such method, the dfx file can be used to store the tabular data.

Next
Previous
Related :
  • 4. Constants


5. Types of Cells

In esProc, cells are automatically classified into different types according to their values or the expressions they contain. In a cellset, each type of cell has its own appearance, which highlights the code in the cells.

Next
Previous
Related :
  • Chapter 1 Gettting Started
  • 5.1 Types of cells
  • 5.2 Setthe appearance of different types of cells


5.1 Types of cells

Next
Previous
Related :
  • 5. Types of Cells
  • 5.1.1 Constant cell
  • 5.1.2 String constant cell
  • 5.1.3 Computational cell
  • 5.1.4 Executable cells
  • 5.1.5 Comment cells


5.1.1 Constant cell

A cell in which the cell string can be interpreted as the constant is called a Constant Cell. Its cell value is the constant.

The constant cell has the following types of values:

The text of a constant cell is in pink by default. Depending on the data in the cell, its value will be interpreted as various data types. For the unrecognized data, the value will be taken as the string.

Note:The representation style of 3% is unacceptable in any expression, but it is acceptable in the constant cell.

Please note the common reserved words below and keep in mind that they arecase-sensitive!

null

Null value; the cell value of any empty cell is always the null

true

True

false

False

Next
Previous
Related :
  • 5.1 Types of cells


5.1.2 String constant cell

A string constant cell is the cell whose cell string starts with single quote ', indicating it is a constant cell with a string value. The cell value is a string composed of those characters after the single quote. In this case, all characters following the single quote will be parsed as string, not requiring any additional quotes, escape characters, etc.

'abc\d String abc\d。

'1234.5 String 1234.5。

Next
Previous
Related :
  • 5.1 Types of cells


5.1.3 Computational cell

A computational cell is the cell whose cell string starts with =, and that is used for computing an expression. Its cell value is the computed result of the expression, which can reference cell values of other cells.

=5*(3+2)

=A1*3

=A1/B2+B1

The text of the computational cell is in black by default.

Click [image] to execute the code and you will find that the background of the computational cell turns to light yellow, indicating its cell value has been computed.Click the cell with cell value. In the value view section, the current cell value will be displayed for you to view the computed result.

In the expression, you can use various basic operators to compute cell value:

a+ba-ba*ba/b

Add, subtract, multiply, and division operations.

=3*4 12

=1+2*3 7

a%b

Seek the remainder of a divided by b.

=121%11 0, which indicates 121 is a multiple of 11, and is divisible by 11

=100%7 2

a\b

Integer division, in which the integer parts of both a andbare kept. The integer part of the result will be returned after division.

=11\4 2, cut off the decimal part of the result

=6.1\2.9 3, which is equal to 6\2, the result is 3

s1+s2

Concatenate string s1 and s2. Note that the string will be ignored if added to a real number.

="Stephen"+" "+"Rolfe" String Stephen Rolfe

="A"+3 Integer 3, string "A" is ignored

="A"+string(3) String A3. If you do need to concatenate the string and the real number, please convert the real number to string first.

Next
Previous
Related :
  • 5.1 Types of cells


5.1.4 Executable cells

An Executable cell is the cell whose cell string starts with >, in which certain actions will be executed. The executable cell has no cell value. If an executable cell starts with one of the reserved words, like if, for, etc. then >can be omitted.

The code in an executable cell can be used to modify the value of another cell.

>A1=5

>B1=A1+3

In the executable cells, the cells can be used as variables.

Next
Previous
Related :
  • 5.1 Types of cells


5.1.5 Comment cells

A comment cell is the cell whose cell string starts with /.Its cell value is null.

Comment cells will be ignored during execution.

The text of comment cells is in green by default.

Next
Previous
Related :
  • 5.1 Types of cells


5.2 Setthe appearance of different types of cells

esProc provides the users with options to set the font styles and the foreground/background color of different types of cells.Click the Tool ->Options. On the Appearance tab, you can view and modify settings.

Next
Previous
Related :
  • 5. Types of Cells


6. Multilayer Parameters

esProc provides a large number of functions, many of which use many parameters. In order to clearly judge the positions of these parameters and make writing and reading easier, esProc is specially equipped with multilayer separators of function parameters.

Next
Previous
Related :
  • Chapter 1 Gettting Started
  • 6.1 Separators of function parameters
  • 6.2 Omission of function parameters


6.1 Separators of function parameters

Colon (:), comma (,) and semicolon (;) are used as separators of function parameters in esProc. Their priority decreases in turn.

The most common method is to use commas to separate parameters, which is in line with function syntax in most programming languages. For example:

 | A | B
1 | Math | 92
2 | Writing | 84
3 | =if(B1>=80 && B2>=60,"Pass","Fail") |
4 | =create(Math,Writing,Result) |
5 | >A4.insert(0,B1,B2,A3) |

In this example, parameters of if(), create() and T.insert() function in A3, A4 and A5 are separated from each other by commas. After the code in A3 is executed, the result of A4 is as follows:

Some functions have “coupled” parameters, which are closely related or work together. In this case, a colon is often used to separate them.

 | A | B
1 | Math | 92
2 | Writing | 84
3 | =if(B1+B2>=180:"A",in(B1+B2,150:180):"B", B1+B2>=120:"C",B1+B2<120:"D") |
4 | =create(Math,Writing,Rank) |
5 | >A4.insert(0,A3:Rank,B1:Math,B2:Writing) |

For example, for if() function in A3, each condition corresponds to a returned result and colons are used to separate the results. In the in() function used in a condition, 150 and 180 are also separated by a colon and together they form a numerical interval [150,180]. In A5’s T.insert() function, field value and field name also come in pairs with colons in between to separate them. After execution, the result in A4 is as follows:

In some functions, indicative parameters can be added to certain parameters to change the computational method relating to them. In this case, colons are usually used as separators. See below:

 | A
1 | =demo.query("select * from CITIES")
2 | =A1.sort(STATEID,NAME)
3 | =A1.sort(STATEID,NAME:-1)

Both A2 and A3 sort records of cities according to state ID first, then sort by name if cities belong to the same state. Difference is that -1 is appended after NAME in A3’s function, meaning that sorting by name is in a descending order. The results in A2 and A3 are as follows:

Sometimes, parameters in a function can be divided into different parts according to their roles. Semicolons are usually used to separate these parts.

 | A
1 | =demo.query("select * from CITIES")
2 | =A1.groups(STATEID; sum(POPULATION):Population)
3 | =A2.top(-Population;5)

In A2’s groups() function, the parameter before the semicolon is used for grouping, and those after it are for summarizing computation, whose parameters are separated by a colon to define name of the summarizing field. The result in A2 is as follows:

In A3’s top() function, the parameter after the semicolon defines that the top 5 records are fetched. The result is as follows:

In some functions, parameters are quite many. Usually these parameters are divided into several groups which are separated by semicolons:

 | A
1 | =demo.query("select * from STATES")
2 | =demo.query("select EID, NAME, STATE, DEPT, SALARY from EMPLOYEE")
3 | =A2.groups(DEPT;count(~):Count)
4 | =A2.switch(STATE,A1:NAME; DEPT,A3:DEPT)

T.switch() function in A4 transforms different fields into records of another table sequence, and a semicolon is used here. Besides, comma, colon and semicolon are all used in A4 as separators. This kind of code writing creates clear layers for function parameters. The result in A4 is as follows:

Next
Previous
Related :
  • 6. Multilayer Parameters


6.2 Omission of function parameters

Some esProc parameter functions have default values and, therefore, can be omitted, making functions more concise.

The parameter after a colon is generally used to complement another’s computational model. If default mode is used, the parameter can be omitted. For example:

 | A | B
1 | Math | 92
2 | Writing | 84
3 | =if(B1>=80 && B2>=60,"Pass","Fail") |
4 | =create(Math,Writing,Result) |
5 | >A4.insert(0,A3:Result,B1:Math,B2:Writing) |
6 | >A4.insert(0,B1,B2,A3) |

In A5, parameters after the colons are used to designate field names corresponding to certain values when inserting records. Parameters for designating field names can be omitted in A6 because they use default field names to set field values one by one. Two records are inserted respectively in A5 and A6, then the result in A4 is as follows:

But colons cannot be omitted when they are used to separate intervals:

 | A | B
1 | Math | 92
2 | Writing | 84
3 | =if(in(B1+B2,180:):"A",in(B1+B2,150:180):"B", in(B1+B2,120:150):"C",in(B1+B2,:120):"D") |

In A3, in(B1+B2,180:) and in(B1+B2,:120) represent respectively B1+B2>=180 and B1+B2<=120, in which the colons cannot be omitted. The result in A3 is as follows:

For parameters separated by semicolons, the semicolon can be omitted if there is no parameter after it. For example, if n is not set in A.top() function and only the top one is needed; or when the transformation of a certain field is not needed in T.switch() function.

When commas are used as separators, they should generally be retained if parameters are set by default. For example:

 | A
1 | [a,b,c,d,e,f,g]
2 | =A1.to(4,)
3 | =A1.to(,4)

Expression in A2 is equal to =A1.to(4,A1.len()), expression in A3 is equal to =A1.to(1,4). Let’s compare the results in A2 and A3:

Next
Previous
Related :
  • 6. Multilayer Parameters


7. Option Syntax

Next
Previous
Related :
  • Chapter 1 Gettting Started
  • 7.1 esProc’s function options
  • 7.2 Common function options
  • 7.3 Use multiple options simultaneously


7.1 esProc’s function options

Many functions in esProc can use function options with which the same function can have different work patterns. The basic format of function options is f@o(…) in which o is f function’s option. For example:

 | A | B
1 | 2004-5-5 | 2014-7-7
2 | =interval(A1,B1) |
3 | =interval@y(A1,B1) |
4 | =interval@m(A1,B1) |

We can use interval function to compute the number of days between two dates. In expressions in A3 and A4, @y and @m are interval function’s options. With these options, the function will use the year and the month as computational units in computing time interval. With function options, a function can meet multiple needs, extend its role and avoid too many function names or function parameters. The number of days, years and months between the two dates are computed respectively in A2, A3 and A4, as shown below:

Next
Previous
Related :
  • 7. Option Syntax


7.2 Common function options

Some function options are common in esProc. They can be used by many functions.

@1 and @a

@1 option and @a option are frequently used by position, select and join functions , such as A.pos(), A.select(), A.pmax(), A.pselect(), A.minp(), P.align(), etc.

Use of @a option can make functions that return the first query result by default return all eligible results. Contrary to @a, functions that return multiple members by default will return only the first query result by using @1 option.

We’ll look at the use of these two options through some examples:

 | A
1 | [1,2,3,4,3,2,1,2,3,2,1]
2 | =A1.pos(2)
3 | =A1.pos@a(2)
4 | =A1.pselect(~==1)
5 | =A1.pselect@a(~==1)

The results in A2, A3, A4 and A5 are as follows:

 | A
1 | =demo.query("select * from CITIES")
2 | =A1.select(left(NAME,1)=="C")
3 | =A1.select@1(left(NAME,1)=="C")
4 | =A2.align([6,35,40],STATEID)
5 | =A2.align@a([6,35,40],STATEID)

The cities whose names start with a C are selected in A2:

The first city whose name starts with a C is selected in A3:

The first cities whose STATEIDs are 6, 35 and 40 respectively are selected in A4:

All cities whose STATEIDs are 6, 35 and 40 are selected and grouped by states in A5:

By the way, since digit 1 is difficult to distinguish from the lowercase letter l, in most cases, the former is used in esProc’s options.

One other thing need to be made known is that the same option in different functions can have different meanings. For example, @a used in position functions means returning all results, while in functions for file writing, like f.write() and f.export(), it means appending.

@z

@z option is often used in some functions related to order, like sort, position, select, etc. Such as A.rank(), A.sort(), A.pos(), A.pselect(), A.select() and so on.

 | A
1 | [1,2,3,4,3,2,1,2,3,2,1]
2 | =A1.pos@z(2)
3 | =A1.sort@z()
4 | =demo.query("select * from CITIES")
5 | =A4.select@z(STATEID:5)

With @z option, positioning or selecting data in a sequence or a table sequence will be executed from back to front. Thus in this example, A2 returns the position of the last 2:

A3 returns results of sorting in descending order:

Records obtained in A5 are also sorted in descending order:

@b

@b option is often used in functions to position, select, etc., like A.pos(), A.pselect(), A.select(), and so on. The use of @b option is usually accompanied by binary search algorithm, which is more efficient in query, with the prerequisite that A is ordered; otherwise results may be wrong.

 | A
1 | =demo.query("select * from CITIES")
2 | =A1.select@b(STATEID:5)
3 | =A1.sort(STATEID)
4 | =A3.select@b(STATEID:5)

Since data in A1 is not sorted according to state ID, only one result is obtained in A2 with the binary search when @b option is used:

while A4 obtains the correct results because data is sorted in A3:

But it is another thing when @b is used in functions to read and write, such as f.import(), f.export(), f.cursor(). In this case, it is the binary files that being read in or written out. In esProc, binary files use less memory space and have faster access speed, therefore the use of binary files will bring higher efficiency.

Next
Previous
Related :
  • 7. Option Syntax


7.3 Use multiple options simultaneously

Multiple options can be used simultaneously in esProc when needed. There is no specific order among these options. For example:

 | A
1 | [1,2,3,4,3,2,1,2,3,2,1]
2 | =A1.pos@az(2)
3 | =A1.pos@za(2)
4 | =demo.query("select * from CITIES")
5 | =A4.select@1z(STATEID:5)

Because both A 2 and A3 get positions of all the 2 in A1 from back to front, their results are identical:

A5 gets the first city whose STATEID is 5 from back to front:

Note that some function options are mutually exclusive , thus cannot be used simultaneously. Such as @a and @1, or @t and @b, options of f.import().

Next
Previous
Related :
  • 7. Option Syntax


8. Use of Common Data

To make data analysis, first you need to load the original data. The data used most frequently comes from the text files or the databases. In esProc, you can load data from the text files or the databases easily and quickly.

Next
Previous
Related :
  • Chapter 1 Gettting Started
  • 8.1 Text file data
  • 8.2 Database data


8.1 Text file data

esProc can loaddata from a text file as a table sequence. For example, the text file empolyee.txt contains employee information as follows:

EID NAME SURNAME GENDER STATE BIRTHDAY HIREDATE DEPT SALARY

1 Rebecca Moore F California 1974-11-20 2005-03-11 R&D 7000

2 Ashley Wilson F New York 1980-07-19 2008-03-16 Finance 11000

3 Rachel Johnson F New Mexico 1970-12-17 2010-12-01 Sales 9000

4 Emily Smith F Texas 1985-03-07 2006-08-15 HR 7000

5 Ashley Smith F Texas 1975-05-13 2004-07-30 R&D 16000

6 Matthew Johnson M California 1984-07-07 2005-07-07 Sales 11000

7 Alexis Smith F Illinois 1972-08-16 2002-08-16 Sales 9000

8 Megan Wilson F California 1979-04-19 1984-04-19 Marketing 11000

9 Victoria Davis F Texas 1983-12-07 2009-12-07 HR 3000

…

import function is used in esProc to import data from files:

 | A
1 | =file("employee.txt")
2 | =A1.import@t()
3 | =A1.import()

In A2, the use of option @t in import function means that, during data importing, the first row of the text file will be regarded as the column names in the imported table sequence. The data in A2 is as follows:

Let’s look at what it will be like if option @t isn’t used. The imported table sequence in A3 is as follows:

Next
Previous
Related :
  • 8. Use of Common Data


8.2 Database data

esProc can access various kinds of databases through JDBC. Click menu item Datasource Connection in Tool to view the datasource manager:

You can connect to or disconnect from a certain datasource, as well as configure the database to be connected through the datasource manager. demo is esProc’s built-in datasource which is launched by executing esProc\bin\startDataBase.bat under the installation directory. Once it connects to the datasource, esProc gains access to the database and fetches data using SQL:

 | A
1 | =demo.query("select * from CITIES")
2 | $select * from CITIES

query function can be used to get the result set after executing the SQL command and retrieve it as a table sequence, like the code in A1. When the database is connected, SQL statement can be written directly behind $, like what is shown in A2. The results in A1 and A2 are the same, as shown below:

Besides using the datasource manager, you can also use connect function to connect to a datasource. In this case, you should close the connection using close function after data is retrieved from the database:

 | A
1 | =connect("demo")
2 | =A1.query("select * from CITIES")
3 | >A1.close()

The same method is used in A2 to import the table sequence of cities information.

Next
Previous
Related :
  • 8. Use of Common Data


9. Basic Usage of JDBC

esProc can be embedded into Java program. So the latter can call the cellset program written in esProc using a way of connection such as JDBC. The method of calling the esProc program is the same as that of calling the stored procedure. The following is a brief introduction to esProc JDBC.

Next
Previous
Related :
  • Chapter 1 Gettting Started
  • 9.1 Description of the jars of esProc JDBC
  • 9.2 Basic usage of esProc JDBC


9.1 Description of the jars of esProc JDBC

esProc JDBC is like an incomplete database JDBC driver without physical tables. It can be regarded simply as a database that only supports the stored procedure. In addition, it is a built-in computing engine, thus no standalone servers are needed.

esProc JDBC has three basic jars, which are all situated in \esProc\ lib in installation directory:

dm.jar esProc computing engine and JDBC driver

icu4j_3_4_5.jar handle internationalization

dom4j-1.6.1.jar parse the configuration files

If other databases are to be used as the datasources of esProc JDBC, then the driver jars of these databases are required to be in place. For example, hsqldb.jar is necessary to use the demo database. Please note the esProc JDBC requires JDK1.6 or higher versions.

Next
Previous
Related :
  • 9. Basic Usage of JDBC


9.2 Basic usage of esProc JDBC

In the cellset code, the result set is returned by result statement.

 | A
1 | =connect("demo")
2 | =A1.query("select * from EMPLOYEE where EID=?",arg1)
3 | >A1.close()
4 | result A2

In the application code, arg1 is a cellset parameter. This dfx file will be named as test.dfx.

Note: The result set of dfx is returned by result statement. When dfx is called, the parameter names that receive the parameters won’t be used; the values of parameters will be assigned according to their order instead.

Load the jars to be used. Load the basic jars of esProc JDBC mentioned above while launching the Java application. These jars can be put in the directory of WEB-INF/lib under a web application.

Deploy dfxConfig.xml, config.xml and the dfx file

Prepare file config.xml, which contains the basic configuration information of esProc, such as registration code, searching path, datasource configuration, etc. The file can be found in the path esProc\config in esProc’s installation directory. The information in it is the same as that set in the esProc Option page. The configuration is allowed to be adjusted before deployment (like modifying the Searching path which is used to search the dfx file):

Or the datasources necessary for dfx can be configured in the Data Source Explorer:

After the modification, config.xml and dfxConfig.xml, which are situated in esProc\classes in the esProc’s installation directory, will be saved in the class path of the application that will use them.

Put the test.dfx created in Step 1 in the class path of the application, or put it in the path specified by file dfxConfig.xml’s <paths/> node (i.e. the above-mentioned Searching path).

Further configure file dfxConfig.xml manually if necessary. For detailed operation, please refer to related documents. Please note the name of configured file should still be dfxConfig.xml and must not be changed.

Call dfx in Java program.

public void testDataServer(){

Connection con = null;

com.esproc.jdbc.InternalCStatement st;

com.esproc.jdbc.InternalCStatement st2;

try{

//create a connection

Class.forName("com.esproc.jdbc.InternalDriver");

con= DriverManager.getConnection("jdbc:esproc:local://");

//call the stored procedure in which test is the name of dfx file

st =(com. esproc.jdbc.InternalCStatement)con.prepareCall("call test(?)");

//set the parameters

st.setObject(1,"3");

//the result obtained by executing the following code is the same as that obtained using the above calling method

st =(com. esproc.jdbc.InternalCStatement)con.prepareCall("call test(3)");

//execute the stored procedure

st.execute();

//get result set

ResultSet set = st.getResultSet();

}

catch(Exception e){

System.out.println(e);

}

finally{

//close the connection

if (con!=null) {

try {

con.close();

}

catch(Exception e) {

System.out.println(e);

}

}

}

}

To know more about calling methods and configuration, please refer to documents that cover a more in-depth discussion at this point.

Next
Previous
Related :
  • 9. Basic Usage of JDBC


Chapter 2 Program Logic

Next
Previous
Related :
  • 1. Set Operators
  • 2. Branches
  • 3. Loops
  • 4. Sequence and Table Sequence
  • 5. Sequences
  • 6. Operations of Table Sequences and Record Sequences
  • 7. Modify Table Sequences
  • 8. Parameters and Variables


1. Set Operators

Since sets are commonly used in esProc, the latter provides comprehensive set operations.

Next
Previous
Related :
  • Chapter 2 Program Logic
  • 1.1 Binary Operation on Sets
  • 1.2 Alignment Arithmetic Operations on Sets
  • 1.3 A Comparison of Sets


1.1 Binary Operation on Sets

The most basic set in esProc is sequence. Let’s look at some basic binary operations between two sequences A and B.

Ø A|B

Concatenation: Concatenate the two sequences straightforwardly. The members of B are added behind the members of A. If A or B or both A and B are single-value instead of sequences, then they will be handled as single member sequences.

Ø A&B

Union: Join the two sequences and remove the members of B that already exist in A. If one of A and B or both are single- value instead of sequences, then you can take them as single member sequences to process.

Ø A^B

Intersection: Intersection set of A and B. Get a sequence composed of members that not only exist in A but also exist in B.

Ø A\B

Complement: Members in A but not in B. If B is not a sequence, then treat it as the single member sequence.

Ø k*A

A|A|…|A, which is to concatenate the k same sequences, that is, copy A k times. Positions of k and A are interchangeable in the expression.

 | A | B
1 | [a,b,1,2,3,4] | [d,b,10,12,4,3]
2 | =A1|B1 |
3 | =A1&B1 |
4 | =A1^B1 |
5 | =A1\B1 |
6 | =2*A1 |
7 | =[1]*8 |

A2, A3, A4 and A5 compute respectively the concatenation, union, intersection and complement of the sequence in A1 and the sequence in B1:

In A6 and A7, sequence A1 is multiplied by an integer. The result is as follows:

These basic operations - concatenation, complement, intersection and union - can also be used to deal with more complex set operations. Such as, to seek the “inverse intersection” of sets A and B, i.e., the set composed of all that is not the common members of A and B.

 | A | B
1 | [2,3,5,8,13,21,34] | [3,9,15,21,27]
2 | =(A1&B1)\(A1^B1) |

The result of A2 is as follows:

The “inverse intersection” of two sets is the complement of their union and intersection. With this kind of transformation, more complex binary operations will be completed.

Next
Previous
Related :
  • 1. Set Operators


1.2 Alignment Arithmetic Operations on Sets

We can perform the alignment arithmetic operations on two sequences of the same length according to their members and return a new sequence.

Ø A++B

[A(1)+B(1),A(2)+B(2),…]

Ø A--B

[A(1)-B(1),A(2)-B(2),…]

Ø A**B

[A(1)*B(1),A(2)*B(2),…]

Ø A//B

[A(1)/B(1),A(2)/B(2),…]

Ø A%%B

[A(1)%B(1),A(2)%B(2),…], the % here is the Mod computation.

Ø A\\B

[A(1)\B(1),A(2)\B(2),…], the slash here means integer division.

 | A | B
1 | [1,2,3] | [4,1,2]
2 | =A1++B1 |
3 | =A1--B1 |
4 | =A1//B1 |
5 | =A1%%B1 |

The results of A2, A3, A4 and A5 are as follows:

Next
Previous
Related :
  • 1. Set Operators


1.3 A Comparison of Sets

In esProc, the function cmp(A,B) can be used to compare the sequence A and B.

Ø cmp(A,B)

Compare the values of members of the two sequences one by one in alignment. When the first pair of unequal members appear, return 1 if the member in A is greater than that in B; otherwise return -1. If A is identical to B, then return 0. Specifically, cmp(A) or cmp(A,0) will compare A and the sequence with the same length and with its all members being 0, i.e. cmp(A,[0,0,…,0]).

 | A
1 | =cmp(["a","b","c"],["a","b","c"])
2 | =cmp([1,3,5,7],[1,3,7,5])
3 | =cmp([7,6,5,4],[7,6,4,10,11])

The results of A1, A2 and A3 are as follows:

The comparison of two sequences can be briefly expressed as A==B, A>B.

Note that a sequence in esProc is an ordered set, so order plays an important role in comparing if the two sequences A and B are equal. A.eq(B) is used to see if the two sequences have common members.

 | A
1 | [Tom,Jerry,Tuffe,Tyke]
2 | [Jerry,Tuffe,Tom,Tyke]
3 | =A1==A2
4 | =A1.eq(A2)

Because the member order in A1 and A2 is different, result in A3 is false, showing the two sequences are not equal:

While result in A4 is true, showing the two sequences have the same members:

Next
Previous
Related :
  • 1. Set Operators


2. Branches

When using branch statements, we need to check one or multiple conditions, execute different code according to the results or return different results. In esProc, the if/else statement is used to realize branching. It has the following forms:

Next
Previous
Related :
  • Chapter 2 Program Logic
  • 2.1 if x … else … in the same line
  • 2.2 Function if( x , a , b )
  • 2.3 if x … else … in the code block
  • 2.4 Multiple blocks of if x … else if y …
  • 2.5 Nested Branch Statements


2.1 if x … else … in the same line

When condition x after if is true, the statement after it will be executed; otherwise the statement after else will be executed. The else part can be omitted. else and if must be in the same row. After execution, the value of the cell in which if is located is the computed result of x.

 | A | B | C | D
1 | -14 |  |  |
2 | if A1>=0 | >A3=A1 | else | >A3=-A1
3 |  |  |  |

The result of A2 is as follows:

Since the result of A2 is false, the else part, that is, the code in D2, will be executed. The result of A3 is as follows after execution:

Actually, the result of A3 is the absolute value of the number in A1.

Sometimes, there is no else part. We only need to decide whether the code after if statement should be executed:

 | A | B | C
1 | =demo.query("select * from EMPLOYEE") |  |
2 | for A1 |  |
3 |  | if age(A2.BIRTHDAY)>=40 | >A4=A4+1
4 |  |  |

The code in A2 checks the information of every employee by loop and the code in B3 judges it. If the age of an employee is not less than 40, he/she will be counted in A4. The computation continues thus until the number of all employees whose ages are not less than 40 are worked out.

The following logical connectors can be used in the judgment statements:

a&&b

For the "a and b", the result is true only on condition that both a and b are true.

a||b

For the "a or b", the result is true only on condition that either a or b is true.

!a

For the "not a", the result is true only on condition that a is false.

 | A | B | C
1 | =demo.query("select * from EMPLOYEE") |  |
2 | for A1 |  |
3 |  | if age(A2.BIRTHDAY)>=40 && A2.GENDER=="M" | >A4=A4+1
4 |  |  |

In A4, the number of male employees whose age is not less than 40 is computed:

Next
Previous
Related :
  • 2. Branches


2.2 Function if(x,a,b)

If x is true, expression a will be computed and the result be returned; otherwise, expression b will be computed and the result be returned.

Sometimes, the previously mentioned if…else… code can be replaced by if function in order to make the statement more concise.

 | A
1 | -14
2 | =if(A1>=0,A1,-A1)

Next
Previous
Related :
  • 2. Branches


2.3 if x … else … in the code block

If x is true, then the if block of code will be executed ; otherwise, the else block of code will be executed. You can certainly ignore the else part. else and if must be in the same column. Different from some other programming languages, esProc employs code block, instead of symbols such as {} or statements like end if, to determine the statement’s valid scope.

 | A | B | C
1 | =demo.query("select * from EMPLOYEE") |  |
2 | Male | [] | 0
3 | Female | [] | 0
4 | for A1 | if A4.GENDER=="M" | >B2=B2|A4.(NAME+" " +SURNAME)
5 |  |  | >C2+=1
6 |  | else | >B3=B3|A4.(NAME+" " +SURNAME)
7 |  |  | >C3+=1

As shown in the example, based on the judging results of B4, the if block of code composed of C4 and C5 and the else block of code composed of C6 and C7 are executed alternatively. They will compute respectively the name lists of male and female employees, as well as the numbers of male and female employees. Results can be seen in B2, B3, C2 and C3 after the computations are over:

Next
Previous
Related :
  • 2. Branches


2.4 Multiple blocks of if x … else if y …

This is code for multi-branch statements in esProc, which can be written repeatedly forever. The if/else must be in the same cell. Again the point is emphasized that there is no such corresponding statement of end if, and esProc employs the scope of code block to determine when if statement will be over.

 | A | B | C
1 | 4 | * | 7
2 | if B1=="+" |  |
3 |  | >B12=A1+C1 |
4 | else if B1=="-" |  |
5 |  | >B12=A1-C1 |
6 | else if B1=="*" |  |
7 |  | >B12=A1*C1 |
8 | else if B1=="/" |  |
9 |  | >B12=A1/C1 |
10 | else |  |
11 |  | >B12="Error" |
12 | Result: |  |

A1, B1 and C1 combine to form an expression, which is computed in B12. The result is as follows:

Next
Previous
Related :
  • 2. Branches


2.5 Nested Branch Statements

Branch statements can be used to make further judgment in the if or else block of code.

 | A | B | C
1 | 2016-2 |  |
2 | =A1.array("-") | >year=A2(1) | >month=A2(2)
3 | if [1,3,5,7,8,10, 12].pos (month)>0 | >A8=31 |
4 | else if [4,6,9,11].pos (month )>0 | >A8=30 |
5 | else if month==2 | if year%400==0 | >A8=29
6 |  | else if year%100!=0 && year%4==0 | >A8=29
7 |  | else | >A8=28
8 |  |  |

In the above example, a character string of year and month, with a separator “-”, is entered in A1, and the code after it computes the total days of the month. The computation should first judge which month it is. For February, further judgment is needed to determine whether it is a leap year. The computed result in A8 is as follows:

Next
Previous
Related :
  • 2. Branches


3. Loops

With esProc, the code for a loop is mainly written with for statement. The for statement will repeatedly execute the code block with for being the master cell. There are different formats of for statements, as listed below:

Next
Previous
Related :
  • Chapter 2 Program Logic
  • 3.1 for loop
  • 3.2 for A
  • 3.3 for n
  • 3.4 for a, b, s
  • 3.5 for x
  • 3.6 Nested loops


3.1 for loop

An unconditional loop is the one where the values of the master cell are, in turn, the counts of the current loop, which can usually be terminated by a break command.

The unconditional loop or endless loop is the simplest loop structure, for example:

 | A | B | C
1 | =demo.query("select * from EMPLOYEE") | [] |
2 | for | =A1(A2) |
3 |  | if B2.STATE=="Texas" |
4 |  |  | >B1=B1|B2
5 |  | if B1.len()==10 |
6 |  |  | break

In the above example, A2 executes an unconditional loop. B2 retrieves one employee record based on each circle of loop in A2. The ensuing code saves the information of Texas employees in B1.After the information of the first 10 Texas employees are retrieved, the loop will be terminated with a break statement in C6. Alternatively, a break C command can also be used to replace the break statement to exit from the loop body with cell C being the master cell.

After execution, B1 contains the following results:

Note that when using the unconditional loop, we need to make sure that the break statement can be executed properly, otherwise the program will loops endlessly.

Next
Previous
Related :
  • 3. Loops


3.2 for A

Loop members of the sequence A. The master cell values will be the current members of A in turn.

In esProc, the most commonly used loop is to loop on every member in a sequence. For example:

 | A | B
1 | =demo.query("select * from EMPLOYEE") | 0
2 | =A1.select(STATE=="Texas") |
3 | for A2 | =age(A3.BIRTHDAY)
4 |  | >B1=max(B1,B3)

In this example, all Texas employees are selected in A2. The loop is executed in A3 for each Texas employee to compute their age, and then store the oldest age in B1. When the loop is complete, we can get the maximum age of all Texas employees from B1:

For relatively simple loop statements, sometimes we can use the loop function for sequences to achieve the same result. For example:

 | A
1 | =demo.query("select * from EMPLOYEE")
2 | =A1.select(STATE=="Texas")
3 | =A2.max(age(BIRTHDAY))

The result of A3 is the same as that in the above example:

Next
Previous
Related :
  • 3. Loops


3.3 for n

Loops n times. The master cell values are the current loop counts in turn.

In addition to performing loop on all members in a sequence, specifying the times of loop is also very common.

For example, suppose we have a piece of paper which is 1000mm*800mm in size and of 0.1mm thickness. If we fold it ten times, what will be the final length, width and thickness?

 | A | B | C
1 | 1000 | 800 | 0.1
2 | for 10 | >A1=A1/2 | >C1=C1*2
3 |  | if B1>A1 | =B1
4 |  |  | >B1=A1
5 |  |  | >A1=C3

In the above example, the folding is repeated ten times. After each folding the length will be reduced by half, and the thickness will be doubled. If after the folding the width of the paper is greater than the length, the length and width will be switched. After execution, A1, B1 and C1 contain the length, width and thickness of the paper as follows:

The for n statement can also be seen as a simple form of for to(1, n) , which means "loop from 1 to n".

Next
Previous
Related :
  • 3. Loops


3.4 for a, b, s

Loop from a to b, at the step of s. This means "to loop on each member of array to(a,b).step(s)". The value of s will be 1 by default.

Sometimes the loop does not start from 1, and the step value is not 1. We can then use for a, b, s to execute the loop. For example:

 | A | B
1 | for 1,100,2 | >A2=A2+A1
2 |  |

A2 sums up all odd numbers less than 100 and the result is as follows:

For any n digits number, if the sum of the nth power of each of its digits equals its own, the number can be called a daffodil number. In the following example, all three digits daffodil numbers are to be identified:

 | A | B | C | D
1 | [] |  |  |
2 | for 100,999 | =A2\100 | =(A2\10)%10 | =A2%10
3 |  | =power(B2,3) | =power(C2,3) | =power(D2,3)
4 |  | if B3+C3+D3==A2 | >A1=A1|A2 |

In the loop code in A2, the loop is excecuted only on all 3 digits numbers. After the execution of the program, we can see the result in A1:

Next
Previous
Related :
  • 3. Loops


3.5 for x

Loop when x is true. The value of the master cell is the computed result for x.

By specifying the loop condition, we can control when to end the loop. For example, the issue in section 1 can also be solved using for statement with specific loop conditions:

 | A | B | C
1 | =demo.query("select * from EMPLOYEE") | [] |
2 | for B1.len()<10 | =A1(#A2) |
3 |  | if B2.STATE=="Texas" |
4 |  |  | >B1=B1|B2

In the master cell of for x loop, the cell value is either true or false, which is the result of computing the loop condition every time. At this point we cannot use the values of the master cell as the count of loop. Therefore to retrieve employee records in B2, we use #C to get the current number of loops. In #C, C is the master cell of the loop body. After the code is executed, the result is the same as that in section 1.

Next
Previous
Related :
  • 3. Loops


3.6 Nested loops

In the loop body, we can still use the loop statements. This is what we call nested loops, or multi-layer loops. For example:

 | A | B | C | D
1 | [] |  |  |
2 | for 2,1000 | for A1 | if A2%B2==0 | next A2
3 |  | >A1=A1|A2 |  |

In the example, A1 stores the prime number sequence we found; A2 loops the integers from 2 to 1000. The prime number sequence is looped in B2. If the number in A2 is divisible by certain prime number, it will be replaced by the next number in A2 with the command of next A2. If the number in A2 is not divisible by all integer numbers in current prime number list, a new prime number has thus been found and it will be added to the sequence in A1. The next C command in D2 specifies that the rest part of the loop body whose master cell is C should be skipped and the next round of loop begins. After execution, the sequence of prime numbers within 1,000 in A1 is as follows:

When using the next statement, we don’t have to specify the master cell C, but to skip the level of the loop where the cell of next statement is located.

As with the following one hundred chickens puzzle, each rooster is worth 5 dollars, and each hen 3 dollars. Three chicks are worth one dollar. If we can buy 100 chickens with 100 dollars, how many roosters, hens and chicks do we buy actually? We can resolve the issue in the following way:

 | A | B | C | D
1 | [] |  |  |
2 | for 100 | if 5*A2>100 | break |
3 |  | for 100-A2 | =100-A2-B3 | =5*A2+3*B3+C3/3
4 |  |  | if D3>100 | next A2
5 |  |  | else if D3<100 | next
6 |  |  | >A1=A1|[[A2,B3,C3]] |

In the code, A2 loops on the total number of roosters. We need to judge in B2: if the total price of the roosters is more than 100, there must be too many roosters. Then we can use break command in C2 to end the loop. In B3 we continue the looping to compute the possible number of hens. If the current total price of chicken is more than 100 in D4,we know that there are too many hens. We can use next A2 to increase the number of roosters and try again. In D5, the current total price of chicken is still less than 100, and we can continue to try increasing the total number of hens. The next statement here can skip the code for the deepest level of loop, which is the loop having B3 as the master cell. The result in A1 is as follows:

Next
Previous
Related :
  • 3. Loops


4. Sequence and Table Sequence

Sequence, table sequence and record sequence are commonest data types of esProc. This article tries to expound their respective characteristics as well as relations between them.

Next
Previous
Related :
  • Chapter 2 Program Logic
  • 4.1 A sequence is an ordered generic set
  • 4.2 A table sequence is a structured sequence
  • 4.3 A record sequence is the reference of table sequence records


4.1 A sequence is an ordered generic set

Next
Previous
Related :
  • 4. Sequence and Table Sequence
  • 4.1.1 Collectivity
  • 4.1.2 Genericity
  • 4.1.3 Orderliness


4.1.1 Collectivity

A sequence consists of multiple data, which is called members of the sequence. The members can be any type of data, such as string, integer, float and date, or empty. A sequence has the general characteristics of a set, and can perform set operations. For example:

 | A
1 | []
2 | [5,6,7]
3 | [red,blue,yellow]
4 | =["blue","yellow","white"]
5 | =A3^A4

In the above cellset, the values of A1, A2 and A3 are as follows:

The contents in A1, A2 and A3 are sequences. There is an empty sequence in A1. In A2, the members of the sequence are intergers, thus it is also called an integer sequence. The sequence in A3 has string members; it is also the value of the cell.

In A4, an expression is used to compute a sequence. Different from the constant sequence in A3, the strings in the expression should be placed in double quotes. A5 computes the intersection of the sequence in A3 and the one in A4, with a result being a sequence too. The results in A4 and A5 are as follows:

Next
Previous
Related :
  • 4.1 A sequence is an ordered generic set


4.1.2 Genericity

A sequence is a generic set, which is allowed to have members of various data types. A member can also be a sequence. For example:

 | A | B
1 | [] | =[1,date(“2014-05-01”)]
2 | [5,6,7] | =[“blue”,[],[5,6,7]]
3 | [red,blue,yellow] | =[“blue”,A1,A2]
4 | =[“blue”,”yellow”,”white”] |
5 | =A3^A4 |

In the cellset, members of the sequence in B1 are an integer and a date; members of the sequence in B2, which equals the expression in B3, contain a sequence. The results in B1, B2 and B3 are as follows:

You can click the sequence members, which are displayed in blue, and see the details.

Next
Previous
Related :
  • 4.1 A sequence is an ordered generic set


4.1.3 Orderliness

Generally, a set is unordered, that is, two sets with same members of different order are equal. While a sequence is ordered, which means two sequences with same members of different order are not equal. For example:

 | A | B
1 | [Mike,Tom] | [Tom,Mike]
2 | =A1==B1 |

In A2, expression =A1==B1 is used to judge if the two sequences are equal. The result is false:

Orderliness is a common feature of business data. For example, that Mike comes before Tom may mean that Mike has done a better job in school study; sorting monthly sales can clearly present its changing rule. It is more convenient to use a sequence to perform ordered computation. For example:

 | A
1 | [Mike,Tom]
2 | =A1(2)
3 | =A1.m(-1)
4 | =A1.pos("Tom")
5 | =A1.rvs()

A2 fetches the second member of the sequence. The operation can also be expressed as =A1.m(2). A3 fetches the last member of the sequence. A4 gets the sequence number of the member Tom from the sequence. A5 reverses the sequence. The results in A2, A3, A4 and A5 are as follows:

In addition, there are operations like insert, delete, modify, copy, compare, aggregate, sub-sequence, sort, rank, sets computation, mutual transformation of strings and sequences, etc.

An integer sequence is a sequence whose members are integers. It has a more detailed access method. For example:

 | A
1 | =to(2,5)

The sequence generated in A1 is as follows:

If the sequence begins with 1, such as =to(1,5), it can be abbreviated to =to(5).

Next
Previous
Related :
  • 4.1 A sequence is an ordered generic set


4.2 A table sequence is a structured sequence

esProc inherits the data table concept of relational database, and defines it as table sequence. Consistent with the concept of relational database, every table sequence also has its own data structure, which consists of several fields. Members of a table sequence are called as records.

Next
Previous
Related :
  • 4. Sequence and Table Sequence
  • 4.2.1 Structured two-dimensional data objects
  • 4.2.2 A table sequence is a special sequence


4.2.1 Structured two-dimensional data objects

Members of a sequence can be any types of data, such as ordinary types, sequences or records; while members of a table sequence must be records of the same structure. For example, the data objects in the following figure constitute a table sequence:

Because a table sequence is a structured two-dimensional data object, it is usually created from SQL, text files, binary files, Excel files or an empty table sequence. A1, B1 and C1 in the following are table sequences:

 | A
1 | =file("Order_Books.txt").import@t()
2 | =demo.query("select * from CITIES")
3 | =create(OrderID,Client,SellerId,Amount,OrderDate)

A1 creates a table sequence, which is the one in the above, from a text file. B1 creates a table sequence using SQL, and C1 creates an empty table sequence by specifying the field names. The data in B1 and C1 is as follows:

A great deal of structured data operations, such as query, sort, sum, average value, merging repeated records, can be performed with a table sequence. For example:

 | A
1 | =file("Order_Books.txt").import@t()
2 | =A1.select(Amount>=20000 && month(Date)==5)
3 | =A1.sort(SalesID,Date:-1)
4 | =A1.groups(SalesID, month(Date); sum(Amount), count(~))

A2 selects records whose Amount is greater than or equal to 20,000 and Date is the month of May:

A3 sorts the records by SalesID in ascending order. Records with the same SalesID will be sorted by Date in descending order:

A4 sums up Amount of each group of data and count up orders of each group according to SalesID and the month:

Next
Previous
Related :
  • 4.2 A table sequence is a structured sequence


4.2.2 A table sequence is a special sequence

A table sequence is still a sequence. The collectivity, orderliness and functions of a sequence apply in a table sequence. A table sequence hasn’t the feature of genericity in the sense of a sequence because its members must be records of the same structure. But, the field values of the records can be generic data, which in this sense is another form of genericity. Thanks to these features, a table sequence is better at handling complicated computation than traditional programming language does.

For example, based on orderliness, we can compute the growth rate of each month compared with the previous one with a table sequence. The approach is as follows:

 | A
1 | =file("Order_Books.txt").import@t()
2 | =A1.groups(month(Date); sum(decimal(string(Amount))):TotalAmount, count(~):Count)
3 | =A2.derive(round(TotalAmount/TotalAmount[-1]-1,4): compValue)

A2 computes the total sales of each month:

A3 computes the final result:

Another instance based on collectivity. Assume that a big contract in business is the one whose quantity ordered is greater than 1,000 and an important contract is the one in which each amount of order is more than 10,000. Please select: 1. Contracts that are both big and important; 2. The other contracts:

 | A | B
1 | =file("Order_Books.txt").import@t() |
2 | Big | =A1.select(Quantity>1000)
3 | Importance | =A1.select(Amount>10000)
4 | =B2^B3 | =A1\A4

B2 selects the big contracts, and then B3 selects the important ones. A4 computes the common members of the two kinds of contracts, which are the answer to question 1:

B4 answers question 2 by computing the complement of all orders and the results of question 1:

Note: In the above code, A1 contains a table sequence, while, in B2, B3, A4 and B4, there are all record sequences originating from table sequences. The difference and relation between a table sequence and a record sequence will be explained in the following.

Next
Previous
Related :
  • 4.2 A table sequence is a structured sequence


4.3 A record sequence is the reference of table sequence records

Obviously, if each computation on a table sequence is to produce a new table sequence, a great deal of the memory will be used. For instance, a table sequence, Order_Books, has 50,000 records, and 30,000 records are obtained by query. If new table sequences are to be created, there will be 80,000 records in the memory. In fact, as the queried records are part of the original table sequence, it is unnecessary to create table sequences anew. We simply store their references using certain data objects. These data objects are a record sequence.

Next
Previous
Related :
  • 4. Sequence and Table Sequence
  • 4.3.1 Transparency of record sequences
  • 4.3.2 A table sequence has a one-way influence on a record sequence


4.3.1 Transparency of record sequences

Usually, it’s not necessary for users to differentiate between record sequences and table sequences, as what they do with references and physical data. For instance, operations like query, sort and intersection in the previous example can be performed on both record sequences and table sequences with the same syntax:

 | A | B
1 | =file("Order_Books.txt").import@t() | =A1.dup()
2 | =A1.select(Amount>=20000 && month(Date)==5) | =B1.select(Amount>=20000 && month(Date)==5)
3 | =A1.sort(SalesID,Date:-1) | =B1.sort(SalesID,Date:-1)

Note: A1 is a table sequence, from which the record sequence in B1 originates. The computed results in A2, B2, A3 and B3 are all record sequences.

Sets operation can be performed between record sequences, for example, the computation of orders in 2.2 A table sequence is a special sequence. But there isn’t any practical significance to perform sets operation between table sequences, for members of different table sequences are always different objects and the intersection of them is always empty.

When data structure changes, esProc will automatically create new table sequences, like what happened in the preceding example when groups function is used to group and summarize data, or new fields are added to a table sequence using derive function.

Next
Previous
Related :
  • 4.3 A record sequence is the reference of table sequence records


4.3.2 A table sequence has a one-way influence on a record sequence

Different table sequences represent different physical records, so modifying a table sequence will not affect the other ones. But, a record sequence is the reference of the records of a table sequence and the two have the same physical data, any change of the table sequence will affect the corresponding record sequence. For example:

 | A
1 | =file("Order_Books.txt").import@t()
2 | =A1.select(SalesID==363)
3 | >A1.modify(3, 1:SalesID)

Click [image] in the toolbar for stepping through the program, and you will see the table sequence, in which SalesID of the third record is 363, in A1 as follows:

A2 selects from the table sequence all the records whose SalesID is 363, and creates the corresponding record sequence as follows:

A3 modifies the third record’s SalesID of the table sequence in A1 into 1. Check the table sequence afer the code is executed and you can see that the third record has really been modified:

Then check the record sequence in A2 and you will find the corresponding record has been modified too, though it wasn’t modified directly. This is because the records of the record sequence originate from the table sequence:

If there are multiple record sequences that originate from this same table sequence, their data will change accordingly.

In most case, this is not what a programmer wants to see. Take the above case as an example, in the selected data in A2 there appeared records in which SalesID is not equal to 363. This is because the modification of the table sequence data in A3 was executed after A2 had selected the required data (In this case, the corresponding reference didn’t change accordingly). Sometimes this kind of operation would cause an error in business

In order to avoid the error, users should finish modifying the table sequence before the record sequence is created. For example:

TSeq.modify(5, 1000: Amount) / Modify the field Amount of the records in which OrderID equals 5 into 1,000, i.e., less than 2,000.

RSeq =TSeq.select(Amount>2000) / The records in which OrderID equals 5 won’t appear in the RSeq.

 | A
1 | =file("Order_Books.txt").import@t()
2 | >A1.modify(3, 1:SalesID)
3 | =A1.select(SalesID==363)

First modify the data in A2, and then select the desired data. In this way, the record sequence obtained in A3 is as follows:

This computed result is what the business is required. It’s only natural for the above computing order as long as the reference relation between a table sequence and a record sequence is understood.

It’s important to note that operations to modify the original table sequence, like modify function, are not accepted in a record sequence. The relation between the two is a safe one-way influence. So users can safely use record sequences and table sequences.

Next
Previous
Related :
  • 4.3 A record sequence is the reference of table sequence records


5. Sequences

A sequence is an ordered set consisting of some data, which are called members of the sequence. A sequence is similar to an array in high-level languages, but the data types of its members not necessarily the same. The following will explain its basic computation through creation, access, operators and functions.

Next
Previous
Related :
  • Chapter 2 Program Logic
  • 5.1 Creation
  • 5.2 Accessing
  • 5.3 Operators
  • 5.4 Functions


5.1 Creation

Next
Previous
Related :
  • 5. Sequences
  • 5.1.1 Create with Constants
  • 5.1.2 Create with functions
  • 5.1.3 Create by computation


5.1.1 Create with Constants

Bracketing one or more constants with “[]” represents a sequence constant. Bracketing the members in an expression with “[]” makes a sequence. e.g.

 | A | B | C
1 | 1. 1 | 2. red | 3. 2013-06-04
2 | 4. 2 | 5. blue | 6. 27.49
3 | 7. 3 | 8. yellow | 9. Tom
4 | 10. [15.2,b,1] | =[A1:C3] | =[3,A4,B4]
5 | 11. [1,2,3,3] | 12. [] | 13. [[]]

In the cellset, A4, A5, B5 and C5 contain respectively a sequence constant. B4 and C4 contain sequences obtained by computing the expressions.

Members of the sequence in A4 are of various data types, including float, string and integer. Members in B4 are from cell values. In C4, the members even include a sequence. And there are identical members in the sequence in A5. In turn, the data in A4, B4, C4 and C5 are as follows:

Let’s compare the two values in B5 and C5:

It can be seen that the value in B5 is an empty sequence, while the value in C5 is a non-empty one, whose only member is an empty sequence.

Note: Members of a sequence can be of any data types, including basic types, sequences, records, etc. A sequence that all its members are integers is called an integer sequence.

Next
Previous
Related :
  • 5.1 Creation


5.1.2 Create with functions

 | A
1 | =to(2,6)
2 | ="1,a,b,c".array()
3 | =periods@y("2012-08-10",date(now()),1)
4 | =file("sales.txt").import@t()

The code in A1 represents a sequence composed of the consecutive integers from 2 to 6; the code for the sequence can be abbreviated as to(6) if the sequence begins at 1. The code in A2 splits a string into a sequence. And the code in A3 creates a date sequence representing a time period whose starting and ending point are in two adjacent years respectively. The results of A1, A2 and A3 are as follows:

A4 imports records from a structured text file and creates a sequence. The values of the sequence are as follows:

A sequence whose members are records is a table sequence, which is often used to perform structured data computing. It’s not the focus of this article, for more information please refer to “esProc Program: Table Sequence Operation”.

Next
Previous
Related :
  • 5.1 Creation


5.1.3 Create by computation

The code in the following cellset is for importing the text file sales.txt into the table sequence A1. Fetch column STATE to create sequence A2; group the records according to STATE to create sequence A3:

 | A
1 | =file("sales.txt").import@t()
2 | =A1.(STATE)
3 | =A1.group(STATE)

After computing, the sequence in A2 is as follows:

The sequence in A3 is as follows:

It can be seen that members of the sequence in A3 are sequences, whose members are records.

Next
Previous
Related :
  • 5.1 Creation


5.2 Accessing

Next
Previous
Related :
  • 5. Sequences
  • 5.2.1 Access members according to sequence numbers
  • 5.2.2 Assignment and modification
  • 5.2.3 Add members
  • 5.2.4 Delete members


5.2.1 Access members according to sequence numbers

 | A | B
1 | 14. [a,b,c,d,e,f,g] | 15.
2 | =A1(2) | =A1.m(2)
3 | =A1([2,3,4]) | =A1(to(2,4))
4 | =A1.m(-1) |

The expressions in A2 and B2 are equal. Both of them fetch the second member from the sequence. The value is as follows:

A3 fetches members of 2 ~4 from the sequence, with the value being a sequence. Please note [2,3,4] is also a sequence (integer sequence). The results of the expressions in B3 and A3 are the same:

A4 fetches the last member of the sequence. Please note A.m() function must be used when members are fetched backwards and the expression cannot be abbreviated to A1(-1). The result is as follows:

Next
Previous
Related :
  • 5.2 Accessing


5.2.2 Assignment and modification

 | A
1 | 16. [a,b,c,d,e,f,g]
2 | >A1(2)="r"
3 | >A1([3,4])=["r","s"]
4 | >A1.modify(4,[ "r","s"])

In order to specify the change of the statement in e ach cell, click [image] in the tool bar to execute the code step by step.

A2 modifies the second member into r; A3 goes on to modify the third and fourth members in A1. A4 continues the modification in order from the fourth member. The expression in A4 equals >A1([4,5])=["r","s"]. With the step-by-step execution, the changes of sequence in A1 are shown as follows:

Next
Previous
Related :
  • 5.2 Accessing


5.2.3 Add members

 | A
1 | 17. [a,b,c,d,e,f,g]
2 | >A1.insert(0,"r")
3 | >A1.insert(2,["r","s","t"])

A2 inserts a new member at the end of the sequence. A3 continues to insert three members consecutively before the second member. When executed step by step, the sequence in A1 has undergone the following changes:

Next
Previous
Related :
  • 5.2 Accessing


5.2.4 Delete members

 | A
1 | 18. [a,b,c,d,e,f,g]
2 | > A1.delete(2)
3 | > A1.delete([2,4])

A2 deletes the second member. A3 continues to delete the second and the fourth members. When executed step by step, the sequence in A1 has undergone the following changes:

Next
Previous
Related :
  • 5.2 Accessing


5.3 Operators

Next
Previous
Related :
  • 5. Sequences
  • 5.3.1 Sets operations
  • 5.3.2 Alignment arithmetic operations
  • 5.3.3 Boolean operation


5.3.1 Sets operations

Sets operations include ^ (intersection), & (union), \ (complement), and | (concatenation), etc. For example:

 | A | B
1 | 19. [a,b,1,2,3,4] | 20. [d,b,10,12,3,4]
2 | =A1^B1 | =A1\B1
3 | =A1&B1 | =A1|B1

The sequences in A1 and B1 are as follows:

A2, B2, A3 and B3 compute respectively the intersection, difference, union and concatenation of the two sequences. After the computation is finished, the results of A2,B2, A3 and B3 are as follows:

Note: Both union and concatenation are created by combining members of two sequences in order. Their common members only appear once in union while, in concatenation, all the duplicates will appear.

Next
Previous
Related :
  • 5.3 Operators


5.3.2 Alignment arithmetic operations

Two sequences of the same length can perform alignment operation according to members and return a sequence. The operation includes ++ (addition), -- (subtraction), ** (multiplication), // (division) and %% (remainder). For example:

 | A | B
1 | 21. [1,2,3,4] | 22. [10,12,3,4]
2 | =A1++B1 | =A1--B1
3 | =A1**B1 | =A1//B1

The sequences in A1 and B1 are as follows:

A2, B2, A3 and B3 perform addition, subtraction, multiplication and division respectively on the two sequences in alignment. After computing, the results of A2, B2, A3 and B3 are as follows:

Next
Previous
Related :
  • 5.3 Operators


5.3.3 Boolean operation

Two sequences can be compared in alignment. The result is of Boolean type. For example:

 | A
1 | =[1,2,3]==[1,2,3]
2 | =[1,"B",3]<=[1,"b",4]
3 | =[1,2,3]<[1,3,4]

The result in A1 is true, meaning the two sequences are equal. The result in A2 is true, because B is less than b. The result in A3 is true because the second member of the first sequence -2- is less than the one -3- in the second sequence:

Note: A.in(B) function is used to judge the inclusion relation between sequences.

Next
Previous
Related :
  • 5.3 Operators


5.4 Functions

Next
Previous
Related :
  • 5. Sequences
  • 5.4.1 aggregate function
  • 5.4.2 loop function


5.4.1 aggregate function

aggregate functions include A.sum(),A.avg(), A.max() and A.variance() etc., respectively for computing sum, average, maximum value and variance. The following is an example:

 | A
1 | [2,4,6]
2 | =A1.sum()
3 | =A1.sum(~*~)

A2 computes the sum of the sequence’s members. A3 computes the sum of every member’s square. Results of A2 and A3 are as follows:

Next
Previous
Related :
  • 5.4 Functions


5.4.2 loop function

loop functions can perform computation on each member of a sequence. Simple as they are, the loop functions can be used to simplify the complex loop statements, which include loop operation, filter, locate, query, rank and sort, etc. For example:

 | A | B | C
1 | 23. [2,4,-6] | =A1.(~+1) | 24.
2 | =A1.select(~>1) | =A1.pselect@a(~>1) |
3 | =A1.rank() | =A1.sort() | =A1.sort(~:-1)

A2 adds 1 to each member of the sequence. The results are as follows:

A2 selects members that are greater than 1. B2 selects the sequence numbers of members that are greater than 1. C2 queries the sequence numbers of member -6 and member 2 in sequence A1. The results of A2, B2 and C2 are as follows:

A3 computes the rankings of each member in the sequence. B3 sorts members of the sequence in ascending order, and C3 does it in descending order. The results of A3, B3 and C3 are as follows:

Next
Previous
Related :
  • 5.4 Functions


6. Operations of Table Sequences and Record Sequences

The esProc table sequence is a structured two-dimensional table, having concepts of field, record, primary key and reference. These concepts originate from the data table of relational database. A table sequence is also an explicit set of genericity and orderliness, which can perform structured data computing more flexibly. A record sequence is the reference of table sequences. They are closely related and their usages are basically the same. The article will explain their basic computations from aspects of access, loop function, aggregate function and sets operations.

Next
Previous
Related :
  • Chapter 2 Program Logic
  • 6.1 Access
  • 6.2 Loop functions
  • 6.3 Aggregate function
  • 6.4 Operations between sets


6.1 Access

Next
Previous
Related :
  • 6. Operations of Table Sequences and Record Sequences
  • 6.1.1 Creation
  • 6.1.2 Access field values
  • 6.1.3 Access column data
  • 6.1.4 Access row data


6.1.1 Creation

In thecellset below, read two-dimensional structured data from a file, create the table sequence object and store it in cell A1. Create the record sequence object by referring to A1 and store it in B1:

 | A | B
1 | =file("Order_Books.txt").import@t() | =A1.select(Amount>20000)

The following is the table sequence in A1 after computing. Only part of the data is displayed in the window. You can drag the scrollbar on the right to check the complete records:

The record sequence in B1 is as follows:

Note: The table sequence object can be created based on a database or a file, or be created by inserting records into an empty object. A record sequence originates from a table sequence but it doesn’t store physical records. It only stores the references of some records in the table sequence.

Next
Previous
Related :
  • 6.1 Access


6.1.2 Access field values

In the cellset below, get the field PName of the twentieth recordin table sequence object A1 and store it in cell A2, andget the field PName of the second record in record sequence object B1 and store it in cell B2:

 | A | B
1 | =file("Order_Books.txt").import@t() | =A1.select(Amount>20000)
2 | =A1(20).PName | =B1(2).PName

The results in A2 and B2 are as follows:

It can be seen that, as the second record in B1 is identical to the twentieth record in A1, so A2 and B2 have the same computedresults. As can also be seen from the expressions of A2and B2, both table sequence and record sequence have completely the same syntax for accessing fields.

A field name can be replaced by the field’s sequence number and the result won’t change. For instance:=A1(20).#3. Because this kind of replacement is employed universally in esProc, we won’t go into details about it.

Next
Previous
Related :
  • 6.1 Access


6.1.3 Access column data

In the cellset below, fetch column PName according to the column name from table sequence A1 and store it in A2, and again from table sequence A1, fetch column PName and column Amountaccording to the column names and store them in B2. The record sequence and table sequence have the same expression when accessing the column data, so only the latter is selected for our illustration:

 | A | B
1 | =file("Order_Books.txt").import@t() | =A1.select(Amount>20000)
2 | =A1.(PName) | =A1.new(PName,Amount)

The results in A3 and B3 are as follows:

Using the syntax of T.(x), you can only fetch one column of data, and the computed result is a sequence without structured column name. With T.new()function, however, you can fetch one or more columns of data and the computed result is a table sequence with structured column names.

Whether the computing object is a table sequence or a sequence, both T.new() function and A.new() function will create a new table sequence. This means the computed result of B1.new(PName, Amount)is also a table sequence.

Next
Previous
Related :
  • 6.1 Access


6.1.4 Access row data

In the following, fetch the first two records from table sequence A1 according to row number and store them in A2, and fetch the first two records from table sequence B1 according to row number and store them in B2. Both the record sequence and table sequence in this example have the same expression for accessing row data:

 | A | B
1 | =file("Order_Books.txt").import@t() | =A1.select(Amount>20000)
2 | =A1([1,2]) | =B1([1,2])

The result of A2 is as follows:

The result of B2 is as follows:

Next
Previous
Related :
  • 6.1 Access


6.2 Loop functions

Loop functions can compute each record of a table sequence/record sequence, express complex loop statements with simple function. For instance, select is used to make query, sort to sort, id to merge repeated records, pselect to fetch sequence numbers of eligible records and maxif to read the maximum value from eligible records. Here the most basic ones - select function and sort function- will be illustrated. For more information about loop functions, please refer to esProc Program: Loop Operation.

Next
Previous
Related :
  • 6. Operations of Table Sequences and Record Sequences
  • 6.2.1 Query
  • 6.2.2 Sort


6.2.1 Query

In the following cellset, query out records whose Amount field is greater than or equal to 20,000 and whose Date is the month of March. As both record sequence and table sequence have the same expression for querying data, only the latter is selected for our illustration:

 | A | B
1 | =file("Order_Books.txt").import@t() | =A1.select(Amount>=20000 && month(Date)==3)

The result of B1 is as follows:

Whether the computing object is a table sequence or a record sequence, the computed result of select function will always be a record sequence, that is, the references of records instead of the physical records.

Next
Previous
Related :
  • 6.2 Loop functions


6.2.2 Sort

In the following cellset, sort records in ascending order according to SalesID field, and for the recordshaving the same SalesID, sort them in descending order according to Date field. The record sequence and table sequence have the same expression for sorting. Here we’ll take the record sequence in B1 as an example:

 | A | B
1 | =file("Order_Books.txt").import@t() | =A1.select(Amount>20000)
2 | =B1.sort(SalesID,Date:-1) |

The computed result is:

Whether the computing object is a table sequence or a record sequence, the computed result of sort function will always be a record sequence. In fact, most of the functions for table sequences and record sequences can be employed universally unless the records are modified.

Next
Previous
Related :
  • 6.2 Loop functions


6.3 Aggregate function

Next
Previous
Related :
  • 6. Operations of Table Sequences and Record Sequences
  • 6.3.1 Seek maximum value
  • 6.3.2 Sum by grouping


6.3.1 Seek maximum value

Seek the maximum value of Amount field. The record sequence and table sequence in this example have the same expression, so we only take table sequence A1 as an example:

 | A | B
1 | =file("Order_Books.txt").import@t() | =A1.select(Amount>20000)
2 | =A1.max(Amount) |

The computed result of A2 is as follows:

Similar functions include min(minimum value), sum(summation), avg(average value), count(count), etc.

Next
Previous
Related :
  • 6.3 Aggregate function


6.3.2 Sum by grouping

In the following cellset, sum Amount in each group of data according to SalesIDand the month, and count orders of each group. The record sequence and table sequence in this example have the same expression, so we only take the table sequence A1 as an example:

 | A | B
1 | =file("Order_Books.txt").import@t() | =A1.select(Amount>20000)
2 | =A1.groups(SalesID, month(Date); sum(Amount), count(~)) |

The computed result of A2 is as follows:

Note: groups function will create a new table sequence.“~” in expression count(~) represents the current group. count(~) can also be written as count(ID). Besides, we don’t designate field names of computed results in writing code, so default field names like month(Date)will appear. A colon could be used in designating field names, such as =A1.groups(SalesID, month(Date):Month;sum(Amount),count(ID)).

Next
Previous
Related :
  • 6.3 Aggregate function


6.4 Operations between sets

Operations between sets include intersection “^”, union “&”, complement “\” and concatenate “|”, etc.

Next
Previous
Related :
  • 6. Operations of Table Sequences and Record Sequences
  • 6.4.1 Intersection and union operations
  • 6.4.2 Union and concatenation operations


6.4.1 Intersection and union operations

In the following cellset, store orders whose Amount is greater than and equal to 20,000 in the month of March in A2, and store those whose SalesID is equal to 540 or 992 in B2. Now seek the intersection and complement of A2 and A3, and store results respectively in A4 and A5:

 | A | B
1 | =file("Order_Books.txt").import@t() |
2 | =A1.select(Amount>=20000 && month(Date)==3) | =A1.select(SalesID==540 || SalesID==992)
3 | =A2^B2 | =A2\B2

The record sequence in A2 is as follows:

The record sequence in B2 is as follows:

A3 computes the intersection and gets a record sequence as the result:

B3 computes the complement. The result is still a record sequence after members of B2 have been removed from A2:

Let’s move on to the intersection and complement operations between table sequences. First create new table sequences from sequences or record sequences using dup@t() function, and then compute the intersection and the complement:

 | A | B
1 | =file("Order_Books.txt").import@t() |
2 | =A1.select(Amount>=20000 && month(Date)==3) | =A1.select(SalesID==540 || SalesID==992)
3 | =A2.dup@t() | =B2.dup@t()
4 | =A3^B3 | =A3\B3

The table sequence data in A3 and B3 are the same as the datain A2 and B2 in the previous example.

A4 computes the intersection of A3 and B3. The result is an empty set:

This means, as the set of physical members, different table sequences have different members and the result of intersection operation between them will always be empty. Thus the operation has no practical significance in business.

B4 computes the complement of A3 and B3. As members of two table sequences are always different, the computed result of complement operation is still A3:

Only sets operations between record sequences originating from the same table sequence have practical significance in business. Usually, the intersection and complement operations between different table sequences or record sequences originating from different table sequences make no sense in business.

Next
Previous
Related :
  • 6.4 Operations between sets


6.4.2 Union and concatenation operations

In the following cellset, select some records from the record sequence in B1 to create another record sequence, store orders in which SalesID equals 540 and 992 in A2, and store those in which SalesID equals 540 and 668 in B2. Now seek the union and concatenation of A2 and B2and store the results respectively in A3 and B3:

 | A | B
1 | =file("Order_Books.txt").import@t() | =A1.select(Amount>=20000 && month(Date)==3)
2 | =B1.select(SalesID==540 || SalesID==992) | =B1.select(SalesID==540 || SalesID==668)
3 | =A2&B2 | =A2|B2

The record sequence in A2 is as follows:

The record sequence in B2 is as follows:

A3 computes the union of A2 and B2. Members of A2 and B2 will combine in order and the repeated members will be removed.

B3 computes the concatenation of A2 and B2. Members of A2 and B2 will combine in order and repeated members won’t be removed.

Next let’s look at the the corresponding operations between table sequences:

 | A | B
1 | =file("Order_Books.txt").import@t() | =A1.select(Amount>=20000 && month(Date)==3)
2 | =B1.select(SalesID==540 || SalesID==992) | =B1.select(SalesID==540 || SalesID==668)
3 | =A2.dup@t() | =B2.dup@t()
4 | =A3&B3 | =A3|B3

The table sequence data in A3 and B3 are the same as the data in A2 and B2 in the previous example.

A4 computes the union of A3 and B3. As members of the two table sequences are totally different, the union operation means a simple union-all of the two, with the same result as that of the concatenation operation:

B4 computes the concatenation of A3 and B3. The result is completely the same as that in A4, which computes the union:

Next
Previous
Related :
  • 6.4 Operations between sets


7. Modify Table Sequences

Table sequences are universally used in esProc. During data analysis and computing, we often need to add records to a table sequence or delete them from it, or modify the data in it. This article will illustrate the modification and maintenance of the table sequences through operations of adding, deleting and modifying records, as well as of resetting a table sequence and adding fields to it. Meanwhile, it will introduce the different ways of handling these operations in a record sequence.

Next
Previous
Related :
  • Chapter 2 Program Logic
  • 7.1 Add records
  • 7.2 Delete records
  • 7.3 Modify records
  • 7.4 Reset table sequences
  • 7.5 Add fields


7.1 Add records

In the following cellset, append a record r1 to the table sequence in A1 and insert a record r2 at the second row. r1 is to set field values in order and r2 is to set field values by specifying field names. Finally, insert 3 records r3, r4 and r5 consecutively after the first row. Of the three records, ID field and NAME field have values while values of other fields are null:

 | A
1 | =demo.query("select * from STUDENTS")
2 | >A1.insert(0,111,"Jessica","F",18)
3 | >A1.insert(2,25:ID,"William":NAME,20:AGE,"M":GENDER)
4 | >A1.insert(2:3,25+#:ID,"William"+string(#):NAME)

The operations of adding records to the table sequence in A1 are performed successively in A2, A3 and A4. Click [image] in the tool bar to execute the code step by step and you will see the changes of the original table sequence as follows:

As can be seen from the above code, T.insert() function is always used to add records in a table sequence. It can specify positions for the records to be inserted, or set fields for new values. If fields haven’t been set, their values are null.

Different from a table sequence, a record sequence is a sequence composed of references of the records. So you can only add new references of the records to a record sequence:

 | A
1 | =demo.query("select * from STUDENTS")
2 | =A1.dup()
3 | >A1.insert(0,111,"Jessica","F",18)
4 | >A2.insert(2, A1(8))

The record sequence in A2 is created from the records of the table sequence in A1 using dup function. To create another table sequence by duplicating the original table sequence, you need to use T.dup@t() function. Both A1 and A2 and the table sequence in A1 in the previous example have the same data. The program in the cellset will still be executed step by step. After the code in A3 is executed, the data in A1 and A2 are respectively as follows:

It can be seen that a new record has been added to the table sequence, but the original records haven’t been modified. So the record sequence in A2 remains the same. A4 inserts the new record into the record sequence, but the position into which the record is inserted is not the corresponding one of the record’s position in the table sequence. When the code is executed, the record sequence in A2 becomes as follows:

There are a lot of differences between the syntax of table sequences and that of record sequences for adding records. The newly added records in a table sequence are physical ones, so you can use insert function directly in table sequence A1. But you can only add the references of records to a record sequence, so the physical records for referencing must exist first. In the example, the physical records are stored in A1. (Or they can be the records of other table sequences. But the records of the same record sequence should be of the same structure; otherwise the computed result could be wrong.)

That the first parameter of insert function is 0 means appending records at the end of the original records. If not, insert the records in the specified positions. The rule applies to both table sequences and records sequences.

Next
Previous
Related :
  • 7. Modify Table Sequences


7.2 Delete records

In the following cellset, first delete the second record, then delete the first and second records, and finally delete the records of students who are older than 15:

 | A
1 | =demo.query("select * from STUDENTS")
2 | >A1.delete(2)
3 | >A1.delete([1,2])
4 | >A1.delete(A1.select(AGE>15))

The operations of deleting records from the table sequence in A1 are performed successively in A2, A3 and A4. After the code is executed step by step, you can see the changes of the table sequence in A1 as follows:

delete function is used to delete records from a table sequence. With the function, you can specify the sequence numbers, and a sequence composed of the sequence numbers; or you can delete all the records of the corresponding record sequence from the table sequence. The code in A4 can also be written as >A1.delete(A1.pselect@a(AGE>15)), which has the same effect.

Let’s look at the case in a record sequence:

 | A
1 | =demo.query("select * from STUDENTS")
2 | =A1.dup()
3 | >A2.delete(2)
4 | >A2.delete([1,2])

The record sequence in A2 and the table sequence in the previous example have the same original data. First A3 deletes a record, and then A4 deletes two records. After the code is executed step by step, you can see the changes of the record sequence in A2:

By checking the data in A1, you’ll find that the original data in the table sequence remain unchanged:

It is the physical records that have been deleted in the table sequence; while it is the references of the records that have been deleted in the record sequence, which won’t affect the original table sequence.

Next
Previous
Related :
  • 7. Modify Table Sequences


7.3 Modify records

In the following cellset, first modify the GENDER field into Female and the NAME field into Violet in the second record; and then add 10 to each ID of the three consecutive records after the first:

 | A
1 | =demo.query("select * from STUDENTS")
2 | >A1.modify(2, "Female":GENDER, "Violet":NAME)
3 | >A1. modify (2:3,ID+10:ID)

The original data of the table sequence in A1 are the same as those in the above example. A2 and A3 modify the table sequence. After the code is executed step by step, you can see the changes of the data in A1:

modify function is used to modify records in a table sequence. It is also used to modify data in a single record.

It is forbidden to modify records in a record sequence. The modification must be performed in the original table sequence.

You can also modify the field values of a record into those of another record according to their positions:

 | A
1 | =demo.query("select * from STUDENTS")
2 | >A1(5).paste(A1(1))

Record pasting is the operation of modifying records using paste function. After the record pasting operation in A2 is executed, the fifth record in the table sequenc has been modified:

The records being pasted can be from the same table sequence, or other table sequences. The record pasting is performed on the function of the records, not on table sequences, so it applies to record seqences. The operation can modify the original table sequence when executed. In esProc, P.paste(P') can be used to paste multiple records of a record sequence simultaneously.

Next
Previous
Related :
  • 7. Modify Table Sequences


7.4 Reset table sequences

Using reset function, you can empty a table sequence while retaining its data structure:

 | A
1 | =demo.query("select * from STUDENTS")
2 | >A1.reset()

Resetting table sequences can remove useless records and release the memory:

Next
Previous
Related :
  • 7. Modify Table Sequences


7.5 Add fields

It is not allowed to add fields to a table sequence or delete them from it. To add new fields to a table sequence, such as computed columns, you can create a new table sequence using derive function:

 | A
1 | =demo.query("select * from STUDENTS")
2 | =A1.derive(left(NAME,1):INIT)

The data are as follows after A2 adds new fields to the table sequence:

derive function can also be used in record sequences. No matter the computing object is a table sequence or a sequence, the function will always create a new table sequence. If all the original fields are not needed in the newly-created table sequence, new function can be used to create the table sequence. If the data in the original table sequence A1 are of no use, you’d better empty the cell value of A1, or reset the table sequence using reset function.

Next
Previous
Related :
  • 7. Modify Table Sequences


8. Parameters and Variables

In esProc, constants can be directly stored in cells and be referenced in expressions. For the basic usage of constants, please refer to esProc Getting Started: Constants. Actually, when a constant stored in a cell is referenced, it means the value of the cell is used as a variable. The parameters and variables in esProc can be any data types and can be changed freely if necessary. But you should be careful when calling them.

Now let’s learn to use the various parameters in esProc.

Next
Previous
Related :
  • Chapter 2 Program Logic
  • 8.1 Cell parameters
  • 8.2 Program parameters
  • 8.3 Cellset variables
  • 8.4 Cellset constants
  • 8.5 The priority of parameters, variables and constants


8.1 Cell parameters

In esProc, if a cell has value, the cell name can be used directly to reference the cell value in a computational cell or an executable cell. A cell with value can be a constant cell, a computational cell or any other cell which is assigned value by an executable cell. A cell name is composed of the letter for the column and the row number. For example:

 | A | B
1 | 3 | =3*5
2 | =A1*B1 |
3 | =B2 | >B2=3.14
4 | =B2/2 |

A1 is a constant cell, B1 is an expression. Both values of A1 and B1 can be used as parameters. The results of B1 and A2 are as follows:

There is neither a constant nor an expression in B2. Thus when A3 calls the cell value of B2, the result is null. In fact the initial value of any cell is null.

B3 sets value for cell B2. So when B2 is called in A4, its value becomes 3.14. And the result of A4 is as follows:

It can be seen that if a cell’s value is not null, its background will be light yellow by default.

In esProc, you can directly use a cell parameter without declaring in advance or defining the parameter name. Moreover, the cell parameters are easy to look up. Therefore they have been widely used.

The following program, for example, uses cell parameters to store and compute the accumulated value:

 | A | B | C
1 | =demo.query("select * from CITIES") |  |
2 |  |  |
3 | for A1 | if A3.POPULATION>500000 | >A2=A2+1
4 |  | else | >B2=B2+1

A3 traverses the data of cities. If a city’s population is greater than 500,000, it will be counted in A2; otherwise it will be counted in B2. After computing, the results of A2 and B2 are as follows:

Next
Previous
Related :
  • 8. Parameters and Variables


8.2 Program parameters

In a cellset file, you can also define the program parameters, which can be used in the current cellset.

Click Parameter in the Program menu item on the menu bar, you can view the setting of program parameters:

You can set the parameters used in a cellset file in the Program parameter dialog box. A program parameter name is not allowed to contain spacing and signs, or numbers alone; the parameter value, which can be constants of any data type, will be automatically parsed into corresponding data type according to the set value. Please note that expressions cannot be used as program parameters.

If the Set arguments before run option is checked, the Input argument window will pop up before the cellset file is computed:

You can modify the program parameter value before computation starts, or use the default value. To use a program parameter, call it directly by its name:

 | A
1 | =pi*4
2 | =arg1+" world!"
3 | =arg2.(~*~)

After computing the results ofA1, A2 and A3 are respectively as follows:

A program parameter should be defined before it is used. The use of an undefined program parameter in an expression will make the expression unable to be parsed and thus cause an error.

Since the names of program parameters are user-defined, they are often used to store some constants for computing, such as Pi. In addition, program parameters are used to set those data needed to be modified before computing. For example:

 | A | B | C | D
1 | Big | Small | SAME |
2 |  |  |  |
3 | for num | =rand(6)+1 | =rand(6)+1 | =rand(6)+1
4 |  | if B3==C3 && B3==D3 | >C2+=1 |
5 |  | else if B3+C3+D3>10 | >A2+=1 |

num is a preset program parameter, which sets the number of throwing dice. Count the points of 3 random dices. If the total points are greater than 10, mark them as big and store them in A2; if the total points is less than or equal to 10, mark them as small and store them in B2; if the 3 dices have the same points, store the points in C2 without marking them as big or small. Then set num as 100 and perform the computation, results of A2, B2 and C2 are as follows:

In the cellset, the expression >C2+=1 is equal to >C2=C2+1. For expressions like a=a?x, they can be abbreviated to a?=x.

Next
Previous
Related :
  • 8. Parameters and Variables


8.3 Cellset variables

Next
Previous
Related :
  • 8. Parameters and Variables
  • 8.3.1 The use of cellset variables
  • 8.3.2 Judge and delete variables


8.3.1 The use of cellset variables

Besides using cell names directly and calling predefined program parameters, you can also use variables having names, which are called cellset variables. Cellset variables are automatically created through value assignment, without declaring in advance. They are valid in the whole cellset once the cellset gets assigned. But the referencing of an unassigned variable will cause errors. For example:

 | A | B
1 | >a=5 | >b=[1,3,2]
2 | =a+3 | =b.(~+a)

A1 and B1 define cellset variables a and b respectively. The variables can be called in A2 and B2. After computing, the results of A2 and B2 are as follows:

The cellset variables defined in a cellset and their values can be viewed in page Cellset variable in the bottom right of the interface:

The program parameters defined by the cellset will also be displayed in this list.

Next
Previous
Related :
  • 8.3 Cellset variables


8.3.2 Judge and delete variables

ifv function is used in esProc to judge if a variable has been defined. If a defined variable has done its work, it can be removed using rmv function. For example:

 | A | B
1 | >a=5 | >b=[1,3,2]
2 |  |
3 | if ifv(a) | >A2=b.(~+a)
4 | >rmv(a) |
5 | if ifv(a) | >B2=b.(~+a)

The third and fifth lines of code are to assign values to A2 and B2 respectively. Since A4 has removed the variable, only A2 has result:

But B2 is empty. The use of ifv in A5 for judgment avoids errors in parsing the expression in B5.

Next
Previous
Related :
  • 8.3 Cellset variables


8.4 Cellset constants

You can set parameters used repeatedly in cellset computing as the cellset constants, the usage of which is the same as that of the cellet parameters, that is, calling them by their names. But the cellset constants are different from the program parameters in that you can set data types for cellset constants and are not allowed to assign value to them in a cellset, while the program parameters can get assigned.

To view the cellset constants, you can click the Cellset constant button on the Tool menu:

Constants can be various data types, like string, integer, date and sequence, etc. You can directly call them by their names for use:

 | A
1 | =param1+" Smith"
2 | =round(pi*3*3, 2)
3 | =param2.sum()

After computing the results of A1, A2 and A3 are as follows:

Next
Previous
Related :
  • 8. Parameters and Variables


8.5 The priority of parameters, variables and constants

In the above you learned the usages of cell parameters, program parameters and cellset constants. But what the situation will be like if parameters, variables or constants of various types are used at the same time?

Let’s first look at the following situation, in which the value of Pi is set as both the cellset constant and the program parameter:

At this point, the cellset is as follows:

 | A
1 | =pi
2 | >pi=3
3 | =pi

After computing, the results in A1 and A3 are as follows:

As can be seen from this situation, when the namesake cellset constant and program parameter are defined at the same time, the latter has the priority. Thus the value of A1 is 3.14. But if there is a cellset variable of the same name in the cellset, this cellset variable will gets the priority. Therefore the value of A3 is 3 assigned by A2.

Another situation:

 | A
1 |
2 | =A1
3 | >A1=3
4 | =A1

As there are no data in A1 originally, the result of A2 is empty. But both the "A1" defined as a cellset constant and that as a program parameter cannot be identified. So cell names cannot be used to set cellset constants and program parameters.

After A3 assigns value to "A1", the background of cell A1 becomes light yellow. But in the bottom right of the interface there isn’t such a cellset constant as "A1", which means A3 has actually assigned a value to the cell. Now the data in both A1 and A4 are the same:

Hence, in esProc, the program parameters, cellset constants and variables defined in a cellset should all avoid using names like a cell name.

Next
Previous
Related :
  • 8. Parameters and Variables


Chapter 3 Related Operation

Next
Previous
Related :
  • 1. Create and Derive New Table Sequences
  • 2. Data Grouping and Summarizing
  • 3. Sorting and Ranking Data
  • 4. Alignment Grouping and Enumeration Grouping
  • 5. Foreign Key and Relationship between Tables
  • 6. Primary Keys and Index Function


1. Create and Derive New Table Sequences

Table sequences are frequently used in esProc. A table sequence can be retrieved from a database, created based on data in a sequence or another table sequence using new function, or derived by using derive function to add a single or multiple columns to an existing record sequence or table sequence.

Next
Previous
Related :
  • Chapter 3 Related Operation
  • 1.1 Create a table sequence based on existing data
  • 1.2 Create a sequence based on existing data
  • 1.3 Derive a new table sequence based on a sequence or a record sequence
  • 1.4 Create a table sequence using new function and derive function consecutively


1.1 Create a table sequence based on existing data

A.new() function can be used to create a new table sequence based on the data of an existing sequence or table sequence. For example:

 | A
1 | [1,2,3,4,5]
2 | $(demo) select * from EMPLOYEE
3 | =A2.select(STATE=="Texas")
4 | =A1.new(~,~*~,~*~*~)
5 | =A2.new(EID,NAME+" "+SURNAME,GENDER,age(BIRTHDAY))
6 | =A3.new(EID,NAME+" "+SURNAME,GENDER,STATE)

A1 is a simple sequence. A2 is a table sequence of employee information got from a database. A3 is a sequence consisting of records of employees of the state of Texas. A4, A5 and A6 create new table sequences respectively based on the sequence, table sequence and record sequence using new function.

The table sequence in A4 is:

The table sequence in A5 is:

The table sequence in A6 is:

As can be seen from the above results, A.new() function computes the expression by looping each member in A and enter each result into the new table sequence. new function can reference members of A; if A is a table sequence or a record sequence, field values of its records can be used for the computation. By default, the field names of the new table sequence are the expressions used for generating the table sequence, for instance, age(BIRTHDAY). If the fields in a new table sequence are directly from the old one, the old field names will be used in the case that no new names are given to the fields, like EID and GENDER, which are actually expressions for generating the new fields. Data can be selected as needed to create a new table sequence.

It is inconvenient to use expressions as the field names of the new table sequence. Usually new names will be given to the fields of the new table sequence. For example:

 | A
1 | [1,2,3,4,5]
2 | $(demo) select * from EMPLOYEE
3 | =A2.select(STATE=="Texas")
4 | =A1.new(~:Value,~*~:Square,~*~*~:Cube)
5 | =A2.new(EID,NAME+" "+SURNAME:FullName,GENDER, age(BIRTHDAY):Age)
6 | =A3.new(EID,NAME+" "+SURNAME:NAME,GENDER,STATE)

Results of A4, A5 and A6 are as follows:

It can be seen that, in order to rename the new fields while using new function, one just add the desired names after the expressions for generating the corresponding fields, with a colon separating the name and the expression. For instance, NAME+" "+SURNAME:FullName represents that employees’ full names are computed according to the NAME and SURNAME fields of the records in the original table sequence and the corresponding field in the new table sequence is named FullName. In this case, new function works on the basis of the old sequence, table sequence or record sequence, so it cannot use the field names of the resulting table sequence, like FullName.

By the way, if it is a sequence consisting of continuous numbers that according to which the new table sequence is created, such as the sequence in A1, the expression can be simply written as n.new(). For example:

 | A
1 | =5.new(~:Value,~*~:Square,~*~*~:Cube)
2 | $(demo) select * from EMPLOYEE
3 | =100.new(A2(~).(NAME+" "+SURNAME):Name,rand(5)+1:Group)

A1 is the abbreviated form of the expression in A4 in the previous example, and both have the same results:

A3 executes new function with numbers likewise and generates 100 records. Put the records of the first 100 employees in the employee table into five groups randomly and the resulting table sequence is as follows:

When new function is used to create a new table sequence, the fields of the new table sequence are allowed to be referenced in its expression. For example:

 | A
1 | $(demo) select * from EMPLOYEE
2 | =A1.new(EID,NAME+" "+SURNAME:FullName, FullName[-1]:PREV)
3 | =A1.new(EID,NAME+" "+SURNAME:NAME)
4 | = A1.new(EID,NAME+" "+SURNAME:NAME,NAME[-1]:PREV)
5 | =A1.new(EID,NAME+" "+SURNAME:NAME, NAME[-1]+" "+ SURNAME[-1]:PREV)

So A2 computes the employees’ full names and references the full name of the previous employee in the following field when using new function to create a new table sequence. Its result is as follows:

Sometimes the new field names are allowed to be the same as those in the original table sequence. For instance, NAME is still used in A3 to be the name of the field of employees’ full names:

But notice that a problem thus arises in A4’s code, in which NAME[-1]:PREV is used to reference the name of the previous employee and name the new field PREV. Since the original table sequence in A1 also has a NAME field, which takes the priority over the one in the new sequence if the two have the same names, A4 gets the following result:

It can be seen that A4 only gets the employee names in the original table sequence, rather than the newly computed full names.

In this case, one can only get the PREV field using the field in A1, like what A5 does. Result is as follows:

Next
Previous
Related :
  • 1. Create and Derive New Table Sequences


1.2 Create a sequence based on existing data

The loop function A.(x) can also be used to generate data using an existing sequence, table sequence or record sequence. Let’s learn more about it and compare it with A.new(). For example:

 | A
1 | $(demo) select * from EMPLOYEE
2 | =A1.(NAME+" "+SURNAME)
3 | =A1.( age(BIRTHDAY))
4 | =5.([~,~*~,~*~*~])

Results of A2 and A3 are as follows:

It can be seen that the results are sequences instead of table sequences. Generally one can only compute a single field with the loop function A.(x).

If multiple fields are to be computed, the code should include these multiple fields, like what A4 does. Result of A4 is as follows:

Next
Previous
Related :
  • 1. Create and Derive New Table Sequences


1.3 Derive a new table sequence based on a sequence or a record sequence

Sometimes it is no need to create a new table sequence. One just needs to add data as required to the existing ones. In this case, derive function can be used to add computed fields to a table sequence or a record sequence and thus derive a new one. For example:

 | A
1 | $(demo) select EID,NAME,SURNAME,BIRTHDAY,GENDER from EMPLOYEE
2 | =A1.derive(age(BIRTHDAY))
3 | =A2.derive(NAME+" "+SURNAME:FullName)
4 | =A1.derive(:FullName)

Data in the original table sequence in A1 are as follows:

A2 derives a new table sequence by using derive function to add an age field to the original table sequence. Result is as follows:

It can be seen that an age field which is computed according to BIRTHDAY field has been appended to the original fields. The new field hasn’t been renamed and so the expression age(BIRTHDAY) for generating it is its name. Actually derive function can be considered as the simplified writing style of new function, it also generates a new table sequence when executed. But the new table sequence in A2 won’t have any impact on the table sequence in A1.

A3 adds the FullName field to the table sequence in A2 and gets the following table sequence:

It is clear that all the fields of the table sequence in A2 have been copied, only with a new field being appended. In the expression in A3, the new field is named FullName. Similarly, the operation in A3 won’t affect the result of A2.

A4 reuses the original table sequence in A1 and adds a new field named FullName to it, but generates no expression. Result is as follows:

It can be seen that, without an expression, the program will generate a null field and return it.

 | A
1 | $(demo) select EID,NAME,SURNAME,BIRTHDAY,GENDER from EMPLOYEE
2 | =A1.derive(age(BIRTHDAY),NAME+" "+SURNAME:FullName)
3 | =A1.derive(age(BIRTHDAY):Age,string(Age)+GENDER:Group)

Multiple fields can be added simultaneously through derive function. In this example, result of A2 is as follows:

It is the same as the result got by adding fields step by step.

A3 adds an Age field and a Group field which is computed with Age field and GENDER field to the original table sequence. Result is as follows:

The expression in A3 not only adds two fields to the original sequence simultaneously, but references the data of the new Age field in the second field – Group. With derive function a generated field can be referenced by a field generated later than it. But please note that, in esProc, if a number needs to be concatenated with a string, first it should be converted to a string.

Another thing worthy of note is that, like new function, derive function will also create a new table sequence rather than purely add a column to the existing one. Each record will be recreated in the new table sequence during the execution. Thus both derive function and new function are not sufficiently efficient. But the efficiency can be much increased if one executes derive only once when adding multiple fields. So the coding method in A2 is better than the step-by-step method in the previous example. Even some data cannot be specified at the beginning, null fields are allowed to be generated during the execution of derive function and they can get assigned by T.run() function later. For example:

 | A
1 | $(demo) select EID,NAME,SURNAME,BIRTHDAY,GENDER from EMPLOYEE
2 | =A1.derive(age(BIRTHDAY),:FullName)
3 | >A2.run(FullName=NAME+" "+SURNAME)

With this method of executing derive function, it is unnecessary to recreate the records in the new table sequence and only value assignment is needed. So this method is more efficient than the two-step execution.

Next
Previous
Related :
  • 1. Create and Derive New Table Sequences


1.4 Create a table sequence using new function and derive function consecutively

In some cases, a new table sequence requires adding some computed columns but, at the same time, abandoning all the fields of the original table sequence or adjusting the order of the new fields. The consecutive use of new function and derive function can deal with these operations. For example:

 | A
1 | $(demo) select EID,NAME,SURNAME,BIRTHDAY,GENDER from EMPLOYEE
2 | =A1.derive(NAME+" "+SURNAME:FullName).new(EID,FullName,GENDER, BIRTHDAY)
3 | =A1.derive(NAME+" "+SURNAME:FullName)
4 | =A3.new(EID,FullName,GENDER,BIRTHDAY)

After FullName is computed according to the employees’ NAME and SURNAME in the table sequence in A1, NAME and SURNAME are not needed any more. Besides, the fields need to be rearranged. Move FullName forward, for example, as the fields generated by derive function will always come last. To do this A2 uses derive function and new function together. Result of A2 is as follows:

A2 is equal to the effect of two-step execution that includes A3 and A4. When they are used consecutively, new function is actually executed based on the result of executing derive function, like the result A3 gets:

So new function can reference the fields in the original table sequence, like EID, or the new field – FullName generated by derive function.

It is more common to first add fields using derive function and then arrange the result using new function when they are used consecutively. But one can also arrange the desired fields first using new function and then generate the computed fields using derive function. For example:

 | A
1 | $(demo) select EID,NAME,SURNAME,BIRTHDAY,GENDER from EMPLOYEE
2 | =A1.new(EID,NAME+" "+SURNAME:FullName,GENDER, age(BIRTHDAY):Age).derive(string(Age)+GENDER:Group)

In this example, Group field needs to be computed using the new Age field, so A2 first arranges the table sequence in A1 using new function and then computes Group field using derive function. Result of A2 is as follows:

It needs to be made clear that this time derive function is executed based on the result of new function, and thus no data of the original table sequence in A1 can be referenced, like NAME, SURNAME, and etc.

Next
Previous
Related :
  • 1. Create and Derive New Table Sequences


2. Data Grouping and Summarizing

It is a common task to group records in a table sequence as needed and to summarize the data in each group during data analysis and statistics. This includes data summarizing like summing up and averaging, as well as the listing of the top n records, and etc. This article will explore how to group and summarize the data of a table sequence in esProc.

Next
Previous
Related :
  • Chapter 3 Related Operation
  • 2.1 Compute distinct values
  • 2.2 Equal grouping
  • 2.3 Grouping and summarizing


2.1 Compute distinct values

Of the fields of all the records, there are some that have different values, like sequence numbers, though lots of them could have duplicate values. Sometimes all the different values are required to be listed, and, in this case, id function will be used. For example:

 | A
1 | =file("Order_Books.txt").import@t()
2 | =A1.id(SalesID)
3 | =A1.id(PName)

Only part of the data of the table sequence in A1is displayed in the following window:

A2 gets all the different IDs of salespersons, i.e. SalesID; A3 gets all the different names of books, i.e. PName. Results of A2 and A3 are as follows:

As can be seen from the results, id function will sort values in ascending order when it is used to compute distinct values.

Actually, id function returns a result similar to that returned by the distinct statement in SQL.

 | A
1 | =demo.query("select * from EMPLOYEE")
2 | =A1.id(STATE)
3 | =demo.query("select distinct STATE from EMPLOYEE")

The following is a table sequence of employee information A1 selects from the database:

A2 gets all different states from the table sequence in A1 using id function; while A3 gets them from the database using the distinct statement. Results of A2 and A3 are as follows:

We can see that the two methods return the same result. The difference is that id function returns the result in sequence, while the SQL statement returns a result of table sequence from the database.

But sometimes we want to keep the original order of the data. To achieve this, @o option can be used. For example:

 | A
1 | =demo.query("select * from EMPLOYEE")
2 | =A1.id@o(STATE)

Now the result of A2 is like this:

It can be seen that the id@o function doesn’t sort the data, but it directly removes the neighboring data of the same value instead. So there could be duplicate values left in the result, like Texas, California and others in the above result.

Using id function, we can also list all the different results of a certain expression rather than use fields:

 | A
1 | =demo.query("select * from EMPLOYEE")
2 | =A1.id@o(age(BIRTHDAY))

A2 lists the ages of employees:

Next
Previous
Related :
  • 2. Data Grouping and Summarizing


2.2 Equal grouping

We often need to group data according to a certain value, such as group employees by departments, group people by gender, and etc. In esProc, group function is used to group data of a table sequence or of a record sequence and return a sequence composed of groups. For example:

 | A
1 | =demo.query("select * from EMPLOYEE")
2 | =A1.group(DEPT)
3 | =A1.group@o(DEPT)
4 | =A1.group(year(BIRTHDAY))

A2 groups employees by DEPT. The result is shown below:

It can be seen that the result is a sequence whose members are groups, each of which is a record sequence, consisting of employee data of the same department, and is sorted by the department in ascending order.

group function can be used with @o option, which means keeping the original order and putting the neighboring data of the same values into one group. This is what A3 does. It only puts the neighboring employees of the same department into one group when grouping data by department. The result is as follows:

During data grouping, we can create a group only containing data that make a certain expression produce the same result. For example, A4 groups employees by the birth year. Result is as follows:

The operation of group function is similar to that of group by in SQL. Its return value is a sequence consisting of groups. Its final result is a sequence composed of sequences. The summary value of each group can be used for further computation based on the grouping result. This is different from SQL that hasn’t the explicit set data types and cannot store the grouping result. In SQL, each group of data must be summarized immediately when group by is executed, and after that the grouping result will be dumped and thus cannot be reused.

In esProc, we can summarize the result of each group after data grouping. For example:

 | A
1 | =demo.query("select * from EMPLOYEE")
2 | =A1.group(DEPT)
3 | =A2.new(DEPT,count(~):Count,~.sum(SALARY):TotalSalary)
4 | =A2.new(DEPT,~.count(GENDER=="M"):Male, ~.count(GENDER=="F"):Female)

Using the grouping result, A3 computes the number of employees and the total salary of each department. A4 computes the number of male employees and female employees in each department. The result of grouping and summarizing will be returned as a table sequence. Results of A3 and A4 are as follows:

We can see that A3 and A4 perform different summarizing operations on the same grouping result of A2. That the grouping result can be reused is one of the main features of esProc.

An explanation for the expression =A2.new(DEPT,count(~):Count,~.sum(SALARY):TotalSalary) in A3: As A2 is the grouping result of the A1’s table sequence, each member of A2 is a set of records, that is, a record sequence. Therefore when new function is used to perform loop computation on A2, the targets of its expressions are the record sequences. For example, DEPT means getting the value of DEPT field of the first record of the current member; ~.sum(SALARY) means summing up the SALARY field of the current record sequence.

Similar to the group by in SQL, group function also supports grouping multiple fields (expressions) simultaneously. In that case, only fields that have the same values can be grouped together.

Next
Previous
Related :
  • 2. Data Grouping and Summarizing


2.3 Grouping and summarizing

In the previous section, we explained how to group the records in a table sequence and then summarize the records of each group using the grouping result. Alternatively we can use group function to directly get the grouping result, saving the trouble of computing it step by step. For example:

 | A
1 | =demo.query("select * from EMPLOYEE")
2 | =A1.group(DEPT;~.count():Count,~.sum(SALARY):TotalSalary)
3 | =A1.group(DEPT;~.count(GENDER=="M"):Male, ~.count(GENDER=="F"):Female)
4 | =A1.group(DEPT;round(~.avg(age(BIRTHDAY)),2):AverageAge)

Both A2 and A3 directly group and summarize the data by DEPT field, getting the same result as that got by step-by-step computation:

A4 computes the average age of employees in each department:

Actually esProc also provides groups function for directly computing the result of data grouping and summarizing. For example:

 | A
1 | =demo.query("select * from EMPLOYEE")
2 | =A1.groups(DEPT;count():Count,sum(SALARY):TotalSalary)
3 | =A1.groups(DEPT;count(GENDER=="M"):Male, count(GENDER=="F"):Female)

A2 and A3 group and summarize the data by DEPT field directly. Notice that "~." is abandoned here in the code. This is slightly different from the code in the above, but the result is the same as that got by grouping first and then summarizing:

The returned result of groups function for grouping and summarizing data directly is also a table sequence. groups function for performing data grouping and summarizing won’t record the data in each group, it only accumulates the results got by computing the expression. But compared with group function used in the above, groups function is more efficient.

Note: The use of groups function for directly grouping and summarizing data through accumulation requires that only simple aggregate functions, such as sum/count/max/min/topx/avg, can be used in summarizing expressions. For other more complicated ones, it is group function, instead of groups function, that should be used.

One thing worthy of note is that topx, one of the summarizing functions, can be used to compute the n smallest values. For example:

 | A
1 | =demo.query("select * from EMPLOYEE")
2 | =A1.groups(DEPT;topx(age(BIRTHDAY),3))

A2 computes the ages of three youngest persons in each department after data grouping:

topx can only computes the n smallest results. But if the ages of three oldest persons are needed, we can use the following method:

 | A
1 | =demo.query("select * from EMPLOYEE")
2 | =A1.groups(DEPT;topx(BIRTHDAY,3))
3 | =A2.new(DEPT,#2.(age(~)))
4 | =A1.groups(DEPT;topx(-age(BIRTHDAY),3))
5 | =A4.new(DEPT,#2.(-~))

A2 first computes the 3 smallest birthdays in each department, and then A3 further computes the corresponding ages. Results of A2 and A3 are as follows:

Or we can make it by adding a negative sign to topx. For example, A4 gets the 3 smallest ages by negative values, and then A5 computes the desired ages. Results of A4 and A5 are as follows:

Notice that the results of A3 and A5 are the same.

Next
Previous
Related :
  • 2. Data Grouping and Summarizing


3. Sorting and Ranking Data

We often need to sort or rank data during data statistics and analysis. The operations include problems like sort sales records according to the dates and rank the results of salespersons for assessing their performance.

Next
Previous
Related :
  • Chapter 3 Related Operation
  • 3.1 Sort and rank members in a sequence
  • 3.2 Sort records in a data table
  • 3.3 Compute rankings in a table sequence


3.1 Sort and rank members in a sequence

In esProc, functions, such as psort, sort, ranki rank, and etc. are used to sort or rank data. Now let’s look at how to realize the operations.

 | A
1 | [8,1,2,7,0,5,3]
2 | =A1.sort()
3 | =A1.psort()
4 | =A1.rank()
5 | =A1.ranki(5.5)

The codes from A2 to A5 sorts data and ranks data based on the sequence in A1. Results of A1 and A2 are respectively as follows:

In these codes, A2 sorts members in the sequence in ascending order and A3 computes the sequence numbers of the sorted members in the original sequence. For instance, the sequence number 0, which is of the smallest value, in the original sequence is 5; the sequence number 8, which is of the biggest value, in the original sequence is 1. Actually the result of the expression in A2 is the same as that of =A1(A1.psort()). So psort function can be used to generate an index sequence during data computing.

Results of A3 and A4 are respectively as follows:

The result of A3 is the rankings in descending order of members in the original sequence. A4 computes the ranking of 3.5 compared with each member of the original sequence.

In real business situations, it is seldom to sort data based on an existing sequence of constants. Usually the sequence waiting to be sorted is got by computing. For example:

 | A | B | C
1 | =demo.query("select * from EMPLOYEE where EID<6") |  |
2 | =A1.(NAME+" "+SURNAME) | =A2.sort() | =A2.rank()
3 | =A1.(BIRTHDAY) | =A3.sort() | =A3.rank()
4 | =A1.(age(BIRTHDAY)) | =A4.sort@z() | =A4.rank@z()

For the convenience of comparison, A1 only selects the top five records of employees for computing. A2 computes their full names, B2 sorts them and C2 computes the rankings of these full names. Results of A2, B2 and C2 are as follows:

It can be seen that these strings are sorted alphabetically. In practice, strings are sorted and ranked according to each character’s ASCII code.

A3 computes employees’ birthdays, B3 sorts them and C3 computes the rankings of these birthdays. Results A3, B3 and C3 are as follows:

It can be seen that the data of date type are sorted by the date in ascending order. Please note that you can click Tool>Option to set the format of the date on page Environment. The result of sorting or ranking data of date type or date/time type is irrelevant to the format.

A4 computes the ages of employees. B4 sorts these ages using @z option, which means sorting data in descending order; C4 also uses this option to compute the rankings of these ages, meaning the rankings are ordered in ascending order. Similarly, psort function and ranki function can also use the @z option. Results of A4, B4 and C4 are as follows:

We can notice these two points. One is that, by adding @z option, the orders according to which sort function and rank function get their results respectively are opposite; the other is that there are duplicate values in the age data. As shown in the resulting rankings, the rankings of both 39 are 3 while the ranking of 43 becomes 5 when it should be ranked next to 39. This means a same ranking will occupy a position by default. We’ll discuss data ranking in detail in the third section.

The way that strings are sorted according to each character’s ASCII code may cause an error if the target strings are not in English language. In that case, A.sort(…;loc) is needed to specify the language with the parameter loc. For example:

 | A
1 | [Íker,Álvaro,Estela,Alba,César,Sancho]
2 | =A1.sort()
3 | =A1.sort(;"esp")

The sequence in A1 contains names in Spanish. A2 and A3 sort them in different ways. A3 specifies "esp" as the language parameter, meaning the to-be-sorted strings are in Spanish. Results of A1, A2 and A3 are as follows:

As can be seen from the results, a correct sorting result can be got by specifying the language parameter.

For the languages and charsets esProc supports, please refer to the Function Reference.

Next
Previous
Related :
  • 3. Sorting and Ranking Data


3.2 Sort records in a data table

We often need to sort the records in a data table as needed during data computing. For example:

 | A
1 | =demo.query("select EID,NAME,SURNAME,GENDER, BIRTHDAY from EMPLOYEE where EID<6")
2 | =A1.sort(NAME+" "+SURNAME)
3 | =A1.sort(BIRTHDAY)
4 | =A1.sort@z(age(BIRTHDAY))

A1 selects some of the employee information:

A2 sorts the records according to the employees’ full names. Result is as follows:

A3 sorts the records again by birthdays. Result is as follows:

Then A4 sorts the records by ages. Result is as follows:

So we can see that the records in a table sequence or a record sequence can be sorted conveniently as needed by using sort function and a new record sequence will be generated and returned.

Let’s look at the result of A2, which has been sorted by full names. But the sorting process is usually that the records of employees are sorted first by surnames and then those of employees who have the same surnames are sorted again by names. Multiple fields are needed for sorting data in this situation.

 | A
1 | =demo.query("select EID,NAME,SURNAME,GENDER, BIRTHDAY from EMPLOYEE")
2 | =A1.sort(SURNAME,NAME)
3 | =A1.sort(GENDER:1,SURNAME:-1,NAME:-1)

A1 selects all the employee records, as shown below:

A2 sorts the data first by SURNAME and then by NAME. Result is as follows:

We have learned from the first section that, when using functions like sort, psort, rank, ranki and so on, @z option can be added to them to adjust the sorting order. But in the case that the data are sorted according to multiple fields, @z option cannot be used to because the sorting order for each field may not be the same, though we can add a certain mark to each field to specify the sorting order. For example, in the expression in A3, GENDER:1 means sorting data by genders in ascending order and SURNAME:-1 means sorting data by surnames in descending order. If there is no such kind of marks, the data will be sorted in ascending order. The sorting result of A3 is as follows:

If the records waiting to be sorted are non-English strings, we can also specify the language parameter in sort function.

Sometimes we sort the records in a table sequence just for getting the top few ones. And top function or topx function can be used for this purpose. For example:

 | A
1 | =demo.query("select EID,NAME,SURNAME,GENDER, BIRTHDAY from EMPLOYEE")
2 | =A1.top(age(BIRTHDAY);5)
3 | =A1.topx(age(BIRTHDAY);5)
4 | =A1.topx(-age(BIRTHDAY);5)
5 | =A4.(-~)

A2 gets the records of five youngest employees and A3 computes the five youngest ages. Results of A2 and A3 are as follows:

@z option cannot be used in either top function or topx function. When computing five oldest ages, we should add a negative sign before the expression for data sorting, as the code in A4 shows. The final result will be got after the data arrangement in A5. Results of A4 and A5 are as follows:

The same as we deal with the numerical data in the above example, the method of getting the top few rows in descending order by adding a negative sign before the expression for sorting can also apply to the data of date/time type and of string type. For example:

 | A
1 | =demo.query("select EID,NAME,SURNAME,GENDER, BIRTHDAY from EMPLOYEE")
2 | =A1.top(-BIRTHDAY;5)
3 | =A1.topx(-NAME;5)
4 | =A3.(-~)

A2 gets the records of the five youngest employees:

A3 gets the last five employee names that appear in alphabetical order. The result needs to be rearranged in A4 for viewing. The final result is as follows:

It can be seen that all the names are Zarchary.

Next
Previous
Related :
  • 3. Sorting and Ranking Data


3.3 Compute rankings in a table sequence

There are two common ways for computing rankings: rankings with/without duplicate values. For example, when there are two players tied for first place in a competition and if duplicate rankings are to be counted, the next player following them will be the one getting the third place. If duplicate rankings won’t be counted, the second place will follow without having to consider the extra ranking. In esProc, rank is used to compute the rankings and ranki to compute the ranking of a certain member. The use of @i option is to remove the duplicate values before the rankings are computed. For example:

 | A
1 | =demo.query("select * from EMPLOYEE")
2 | =A1.rank(SALARY)
3 | =A1.rank@i(SALARY)
4 | =A1.ranki(10000,SALARY)
5 | =A1.ranki@i(10000,SALARY)

A2 computes the rankings of salaries of all employees. A3 computes them without duplicate values. Results of A2 and A3 are as follows:

Now let’s look at the employee data in A1:

It can be seen by comparison that the employees with the same salary, such as Rebecca and Emily, get the same ranking for their salaries. But obviously they rank much higher in the result of A3, which is caused by the method of ranking without duplicate values, with which the same rankings won’t affect the rankings following them. In fact it is the result in A2 that reflects the rankings more objectively.

A4 computes the ranking of 10,000 in employee’s salaries; A5 computes the ranking of 10,000 by removing the duplicate values. Results of A4 and A5 are as follows:

And we also get a higher ranking with the method of rankings without duplicate values. rank@i and ranki@i compute rankings in a way similar to that in which id function is used to list all the different rankings before the result is computed.

Next
Previous
Related :
  • 3. Sorting and Ranking Data


4. Alignment Grouping and Enumeration Grouping

In Related Computing in esProc – Data Grouping and Summarizing, we learned how to group data in a table as needed and to summarize the grouped data. Please note that, in data grouping and summarizing, data will be grouped by a single or multiple fields or an expression in ascending order based on equivalent values. Alignment grouping or enumeration grouping will be used in data analysis when sometimes data need to be sorted according to a specified order, or to be grouped by specified conditions.

Next
Previous
Related :
  • Chapter 3 Related Operation
  • 4.1 Alignment
  • 4.2 Alignment grouping
  • 4.3 Find satisfied condition(s)
  • 4.4 Enumeration grouping


4.1 Alignment

Usually sort function is used to sort data in a sequence or a table sequence, with a result being listed either in ascending order or descending order. If data need to be sorted in a specified order, esProc provides align function. For example:

 | A | B
1 | [three,one,four,six,two] | [one,two,three,four,five,six]
2 | =A1.sort() | =A1.align(B1)
3 | =demo.query("select EID,NAME,GENDER, STATE,DEPT from EMPLOYEE") | [California,Texas,Florida,Illinois]
4 | =A3.align(B3,STATE) |

The sequence in A1 is composed of string members, and B1 provides a base sequence for alignment:

Both A2 and B2 re-sort members of the sequence in A1 according to the sequence in B1. A2 uses the common sort function, while B2 uses align function. Results of A2 and B2 are as follows after computing:

sort function only sorts the members alphabetically in ascending or descending order, while align function sorts them according to the positions of members of the base sequence. If there is not a corresponding one for a member in the base sequence, the corresponding position will be null.

align function is generally used to sort records of a table sequence or a record sequence. For example, A3 selects some employees, and A4 gets from them the employees of some states according to the specified order. The data in A3 are as follows:

According to the order of the sequence in B3, A4 gets the employees of some states:

With the alignment operation, the first employees of these states have been retrieved and a record sequence composed of these records has been returned.

Next
Previous
Related :
  • 4. Alignment Grouping and Enumeration Grouping


4.2 Alignment grouping

In the previous example, only the first eligible row for each state has been got after the alignment operation. In many cases, however, we need to get all the eligible data. This requires the alignment grouping which can be realized with align@a. For example:

 | A | B
1 | =demo.query("select EID,NAME,GENDER, STATE,DEPT from EMPLOYEE") | [California,Texas,Florida,Illinois]
2 | =A1.align@a(B1,STATE) |
3 | =A2.new(STATE,~.count(GENDER=="M"):Male,~.count(GENDER=="F"):Female) |
4 | =A1.align@n(B1,STATE) |

By adding @a option to the align function, A2 gets all eligible records for each state:

The return value of alignment grouping is a sequence consisting of all the groups, as the result of A2, which is a sequence composed of sub-sequences. Similar to the result of data grouping using group function, the result of alignment grouping using align@a is often used for further computation. For example, A3 further summarizes the data in A2 to get the number of male and female employees:

In alignment grouping, there may be some data that cannot be matched with members of the base sequence. For this problem, esProc provides another option for alignment grouping – align@n, the use of which will put all members that haven’t corresponding ones in the base sequence into a separate group. As the grouping result of A4 shown below:

By comparing the results of alignment grouping of A2 and A4, we can notice that the result of A4 has one more group which stores the records the employees who are not from the four specified states.

The base sequence for alignment grouping can be got by computing. For example:

 | A
1 | =demo.query("select EID,NAME,GENDER, STATE,DEPT from EMPLOYEE")
2 | =A1.groups(STATE;count(~):Count)
3 | =A2.sort(Count:-1)
4 | =A1.align@a(A3:STATE,STATE)

A2 computes the number of employees of each state using groups function for data grouping and summarizing, and then A3 sorts the result by the number of employees in descending order. Results of A2 and A3 are as follows:

A4 gets the states from the result of A3 and computes the base sequence for alignment. The result is as follows:

Another method of alignment grouping is to divide data into n groups using P.align(n,y), and then directly compute the group numbers corresponding to these groups according to the grouping expression y. For example:

 | A
1 | =demo.query("select EID,NAME,GENDER, STATE,DEPT from EMPLOYEE")
2 | =A1.align@a(26,asc(left(NAME,1))-64)
3 | [HR,R&D,Finance,Marketing]
4 | =A1.align@a(A3.len(),A3.pos(DEPT))

In the expression =A1.align@a(26,asc(left(NAME,1))-64) in A2, left(NAME,1) gets the initials of the names of the employees, which are converted by asc function into ASCII code, and, with the subtraction of 64 from it, the initials will correspond to the group numbers from 1 to 26. When the code for alignment grouping is executed, the following result will be got:

The employees in the same group have the same initials.

Sometimes position functions, like pos and pmax, are used to compute the group numbers while P.align(n,y) is used to perform alignment grouping. A4 groups the employees in alignment according to the sequence of specified departments. The result is as follows:

With group function for data grouping, each group of data will be sorted by values of the field for grouping in ascending order; while the alignment grouping can produce a result sorted in specified order.

Next
Previous
Related :
  • 4. Alignment Grouping and Enumeration Grouping


4.3 Find satisfied condition(s)

Sometimes data are to be grouped not by the equivalent values but by some conditions, like the range of the order amount and the employees’ age groups.

The grouping conditions can be represented by strings, like "?>3", "[2,3,5,7].pos(?)>0", and etc. penum function is used to judge which condition(s) a number can satisfy in a sequence formed by strings of conditional expressions. For example:

 | A
1 | [?>=85,?>=70,?>=60]
2 | =A1.penum(100)
3 | =A1.penum(66)
4 | =A1.penum(54)

There is a sequence consisting of strings of conditional expressions in A1:

Respectively, A2 and A3 find the condition(s) 100 and 66 satisfy. Results are as follows:

As can be seen from the result, when a certain number satisfy multiple conditions, only the sequence number of the first satisfied condition will be returned. 100, for example, can satisfy all the three conditions, but the returned result is 1.

As the number 54 in the expression in A4 doesn’t satisfy any of the conditions, the returned result will be null.

Similar to the alignment grouping, penum function also uses @n option, which returns k+1 when data don’t satisfy any of the k conditions. In particular, if a conditional expression is null, it can be satisfied by any data. For example:

 | A
1 | [?>=85,?>=70,?>=60]
2 | =A1.penum@n(54)
3 | [?>=85,?>=70,?>=60,null]
4 | =A1.penum(54)

Results of A2 and A4 are as follows:

The number 54 in A2 doesn’t satisfy any of the three conditions, so penum@n function is used to return a result of 4. But in A4, it satisfies the fourth condition – a null. Please note if the null is listed in the conditions, it must be put at the end.

penum function also uses @r option to represent that the satisfied conditions for different numbers can be identical. In this case, a sequence composed of the sequence numbers of all satisfied conditions will be returned. For example:

 | A
1 | [?>=85,?>=70,?>=60,null]
2 | =A1.penum(100)
3 | =A1.penum(66)
4 | =A1.penum(54)

A2, A3 and A4 return respectively a sequence composed of the sequence number(s) of the condition(s) 100, 66 and 54 satisfy. Results are as follows:

Note: Though the condition null is defined to be satisfied by any data, its sequence number won’t appear in the result if one (or above) more conditions can also be satisfied when penum@r function is doing its job. You can also notice that, when @r option works, the returned result is a sequence rather than a number even if there is only one satisfied condition.

Next
Previous
Related :
  • 4. Alignment Grouping and Enumeration Grouping


4.4 Enumeration grouping

Suppose there is a sequence E consisting of strings of conditional expression. You can group members of another sequence P using P.enum(E,y) function. The operation will compute the expression y using each member of sequence P, judge which condition(s) each computed result satisfies, and group members of P according to conditions they satisfy. By default, members of P will be grouped according to their first satisfied conditions. For example:

 | A
1 | =demo.query("select EID,NAME,GENDER, BIRTHDAY,SALARY from EMPLOYEE")
2 | [?>=15000,?>=12000,?>=9000]
3 | =A1.enum(A2,SALARY)

A1 gets a table sequence and A2 is the sequence of enumeration conditions. They are as follows:

According to the sequence of conditions, the result of enumeration grouping in A3 is as follows:

It can be seen that the data of each group produced by enumeration grouping don’t have the same value of SALARY, but they satisfy the same condition. Similarly, though a row may satisfy both the first condition and the other two conditions, it will, by default, be put into the group where only the first one is satisfied. For the data that don’t satisfy any of the conditions, like the first and the fourth row , they won’t be grouped.

Similar to the case where penum function is used, you can append a null condition at the end, under which all data that don’t satisfy the previous conditions can be placed, while performing enumeration grouping. Or you can use enum@n, which plays a similar role. For example:

 | A
1 | =demo.query("select EID,NAME,GENDER, BIRTHDAY,SALARY from EMPLOYEE")
2 | [?>=15000,?>=12000,?>=9000]
3 | =A1.enum@n(A2,SALARY)
4 | [?>=15000,?>=12000,?>=9000,null]
5 | =A1.enum(A2,SALARY)

Results of A3 and A5 are same:

It can be seen by comparing the result with that of the previous example, both the use of enum@n function and the appending of null at the end of the sequence of conditions will put all data that don’t satisfy the other conditions into a separate group.

By default, it is assumed that there are no overlapped groups when enum function is used to compute, that is, each member of P won’t satisfy two conditional expressions at the same time. @r option is needed if some members need to be grouped more than once. For example:

 | A
1 | =demo.query("select EID,NAME,GENDER, BIRTHDAY,SALARY from EMPLOYEE")
2 | [?>=15000,?>=12000,?>=9000]
3 | =A1.enum@r(A2,SALARY)

A3 performs overlapped enumeration grouping on members using enum@r. Result is as follows:

As can be seen from the result, members of the first group appear in all the three groups and members of the second group appear repeatedly in the latter two groups after the data are overlappedly grouped.

Next
Previous
Related :
  • 4. Alignment Grouping and Enumeration Grouping


5. Foreign Key and Relationship between Tables

In relational databases, a foreign key is often used to identify the relationship between tables. In esProc, this relationship of correspondence can be expressed by the foreign key field.

Next
Previous
Related :
  • Chapter 3 Related Operation
  • 5.1 Computed columns and foreign key fields
  • 5.2 The usage of foreign key fields
  • 5.3 Join operations


5.1 Computed columns and foreign key fields

T.derive() function is used in esProc to add computed columns to a table sequence, for example:

 | A
1 | =demo.query("select EID, NAME, SURNAME, BIRTHDAY from EMPLOYEE")
2 | =A1.derive(NAME+" "+SURNAME: FULLNAME, age(BIRTHDAY):AGE)

A1 gets employee information. A2 adds computed columns - FULLNAME and AGE - to A1’s table sequence, returns a new table sequence, and computes the employees’ full names and ages.

The following is the table sequence in A1:

After the computed columns are added, A2 gets a new table sequence as follows:

Having known about how to add computed columns to a table sequence, we’ll explore the link between the computed columns and the relationship between tables.

Usually, there are relationships between tables in a database. In esProc the relationship is expressed by using the records references to represent data of a table sequence, which makes the data search and presentation simpler and the data structure clearer.

Using T.derive() function, we can add one or more computed columns to a table sequence so that the records or the record sequence in another table can be referenced in the data type of these columns, thus a single or more foreign key fields are created to identify the relationship between the two tables. For example:

 | A
1 | =demo.query("select * from CITIES")
2 | =demo.query("select STATEID, NAME, ABBR, CAPITAL from STATES")
3 | =A1.derive(A2.select@1(STATEID==A1.STATEID):State)
4 | =A1.derive(State.ABBR:SA)
5 | =A2.derive(A1.select(STATEID==A2.STATEID):Cities)

A1 and A2 retrieve the data from database tables - STATES and CITIES - respectively:

CITIES is related to STATES through STATEID field. This kind of storage pattern in databases can keep data consistency, make data easy to maintain and, at the same time, save storage spaces.

A3 adds State field to CITIES to create a foreign key for storing the records of states in which cities located. A4 adds SA field to CITIES to list abbreviations of these states, for the convenience of being looked up. A5 adds Cities field to STATES as a foreign key for storing records of cities of every state.

Execute the program and the data of A4 are as follows:

In the above figure, State field contains records of states information. You can double-click it to see details.

Data of A5 are as follows:

Cities field contains records of cities of each state, double-click it to see details.

It can be seen that, through foreign keys, data of both A4 and A5 have obtained all information of the two tables: STATES and CITIES, in the original database, thus a relationship between both tables has been established. Note that the data types of these foreign key fields are different. The data of foreign key State in A4 are records while those of the foreign key Cities in A5 are record sequences, i.e. a sequence of records.

Actually the STATEID field in table STATES is usually the primary key. switch function can be used to relate a second table, like CITIES, to STATES through the primary key. For the use of the function, please refer to Related computing in esProc – Primary Key and Index Function.

Next
Previous
Related :
  • 5. Foreign Key and Relationship between Tables


5.2 The usage of foreign key fields

During querying or presenting a table sequence, the usage of foreign key fields is the same as that of common fields. But we should be careful towards the data types while using them.

For example, select from CITIES the states whose names contain “la”. Here you can directly call the NAME field of the foreign key State in the filtering expression to filter data:

 | A
1 | =demo.query("select * from CITIES")
2 | =demo.query("select STATEID, NAME, ABBR, CAPITAL from STATES")
3 | =A1.derive(A2.select@1(STATEID==A1.STATEID):State)
4 | =A1.derive(State.ABBR:SA)
5 | =A2.derive(A1.select(STATEID==A2.STATEID):Cities)
6 | =A4.select(like(State.NAME,"*la*"))

Note that here the data type of the foreign key field is records. Computed results of A6 are as follows:

Foreign key fields can also be used as sorting criterion. For example, sort the states data in descending order according to the number of cities of the current state:

 | A
1 | =demo.query("select * from CITIES")
2 | =demo.query("select STATEID, NAME, ABBR, CAPITAL from STATES")
3 | =A1.derive(A2.select@1(STATEID==A1.STATEID):State)
4 | =A1.derive(State.ABBR:SA)
5 | =A2.derive(A1.select(STATEID==A2.STATEID):Cities)
6 | =A5.sort(Cities.len():-1)

Note that here the data type of the foreign key field is record sequence. Sorting results of A6 are as follows:

If the records referenced by a certain foreign key field of a table sequence still contain foreign key fields, these records can be referenced again. For example, list the cities that the states in which they are located happen to contain three cities:

 | A
1 | =demo.query("select * from CITIES")
2 | =demo.query("select STATEID, NAME, ABBR, CAPITAL from STATES")
3 | =A2.derive(A1.select(STATEID==A2.STATEID):Cities)
4 | =A1.derive(A3.select@1(STATEID==A1.STATEID):State)
5 | =A4.select(State.Cities.len()==3)

Here A4 uses states information in A3 containing foreign key fields, instead of the original states information while adding states information to the CITIES. In this way, results we need can be seen in A5:

Next
Previous
Related :
  • 5. Foreign Key and Relationship between Tables


5.3 Join operations

Another type of relationship between tables is the join operations in SQL. esProc’s join function can realize the similar operations. However, with the methods for referencing records and record sequences as well as the align function, and etc., the operations become less used.

Next
Previous
Related :
  • 5. Foreign Key and Relationship between Tables
  • 5.3.1 Equal join
  • 5.3.2 Alignment Join
  • 5.3.3 Cross join


5.3.1 Equal join

The foreign key matches a field in a table with the record references in a second one based on their equivalence, while the join realized with join function matches record references from two or more tables according to the rule of equivalence. For example:

 | A
1 | =demo.query("select STATEID,NAME,POPULATION,ABBR from STATES")
2 | =demo.query("select * from CITIES")
3 | =A1.select(NAME>"C")
4 | =A2.groups(STATEID:ID;count(~):Count)
5 | =join(A3:StateInfo,STATEID;A4:CityCount,ID)
6 | =join@1(A3:StateInfo,STATEID;A4:CityCount,ID)
7 | =join@f(A3:StateInfo,STATEID;A4:CityCount,ID)

A3 selects the states whose spellings begin with letter C or letters after it:

A4 counts the number of the big cities in each state according to the data of CITIES – the table of big cities:

Note:A3 doesn’t include those states whose names begin with letter “A” and letter “B” and STATEID has eligible data only greater than or equal to 6. Not all states have the records of big cities, for in A4 there are no statistics of states whose IDs are 4, 7, 8 and etc.

A5 joins the records in A3 and A4 together according to their respective STATEID. Result is as follows:

It can be seen that, by default, the result of join using join function only include the records of the two tables whose STATEIDs match. The result can be regarded as a table sequence composed of foreign keys fields, each of which is the record references.

Generally the operation of join function amounts to the following code in SQL:

select Ai.* as Fi,… from Ai,… where Ai.xj =… and …

That is the most frequently used code for join operations in SQL.

There are other types of joins. For example, join@1 in A6 represents left join. Please note the “1” in option @1 uses the number 1 instead of the lowercase letter l, which is not used in esProc in case confusion is caused. Result of A6 is as follows:

At this point, all the state information in A3 have got into the result, though there are no corresponding statistics for some of the states in A4, such as those states whose IDs are 7, 8 and etc.

The join@f in A7 represents the full join. Result of A7 is as follows:

Now all the state information of A3 and A4 has been returned to the result. Still, some of the states haven’t the corresponding statistics in A3, such as those states whose IDs are 1, 2, 3 and etc. Some cannot find the corresponding statistics in A4, such as the states with IDs being 7, 8, and etc.

Next
Previous
Related :
  • 5.3 Join operations


5.3.2 Alignment Join

It is based on the equivalence that join function works to join record references of different table sequences. Besides, table sequences can be joined in alignment. In that case, pjoin - the function for joining records in alignment - will be used. For example:

 | A
1 | =demo.query("select STATEID,NAME,POPULATION,ABBR from STATES order by STATEID")
2 | =demo.query("select * from CITIES")
3 | =A2.groups(STATEID:ID;count(~):Count)
4 | =pjoin(A1:StateInfo,A3:CityCount)

A3 counts the number of big cities in each state according to the data of CITIES – the table of big cities. Notice that it lacks the statistics of some states:

A4 joins all the state information in A1 with the statistical results in A3 in alignment, and gets the following result:

The number of records in the result of this join is determined by the table sequence involved in the operation with the shortest data length. This join merely joins the records of two table sequences according to their positions. If not all the records to be joined match, you’ll get a result with not all data being related correctly, like the above result, in which the data of the state of Arkansas, whose ID is 4, is joined mistakenly to the statistic of the city whose ID is 5.

In fact, pjoin is only used to join tables that have the same data length and order. For example:

 | A
1 | =demo.query("select * from STATENAME")
2 | =demo.query("select * from STATECAPITAL")
3 | =demo.query("select * from STATEINFO")
4 | =pjoin(A1:Name,A2:Capital,A3:Info)

The data in A1, A2 and A3 are respectively a part of all the state information and have the same order:

In this case, they can be joined together using pjoin, like what A4 does:

Thus we’ve got a correct result. For the situation in the previous example, the records from the two tables can be joined in alignment using align function.

Next
Previous
Related :
  • 5.3 Join operations


5.3.3 Cross join

A cross join is a JOIN operation that combines each member from the first sequence with each member from the second (or up to nth) sequence. It is the most basic JOIN operation. Without any filtering condition, it will list all possible combinations. xjoin function can be used to realize the cross join operation. For example:

 | A
1 | [1,2,3,4]
2 | [a,b,c]
3 | =xjoin(A1:Number;A2:Letter)

Result of A3 is as follows:

It can be seen that the records in the resulting table sequence are all the possible combinations of the numbers and the letters.

xjoin function can also be used to execute the cross join on records in a table sequence or a record sequence. The records can be filtered according to one or more specified conditions. For example:

 | A
1 | =demo.query("select * from STATES")
2 | [M,F]
3 | =demo.query("select * from EMPLOYEE")
4 | =xjoin(A1:State;A2:Gender;A3:Employee,STATE==State.NAME&& GENDER==Gender)

A1 selects records of all the states. A2 specifies a sequence of genders. A3 selects records of all the employees. In A4, only records of specified states and genders are selected according to the filtering conditions from the table sequence in A3 for performing the cross join operation. Result of A4 is as follows:

Next
Previous
Related :
  • 5.3 Join operations


6. Primary Keys and Index Function

In an esProc table sequence, a single or multiple fields can be used as a primary key. We can make query based on the primary key using some special functions, which can both simplify the code and increase computational performance effectively.

Next
Previous
Related :
  • Chapter 3 Related Operation
  • 6.1 find and pfind
  • 6.2 Primary key and the index table
  • 6.3 switch function


6.1 find and pfind

Primary keys are common used in database tables. The value of a primary key field is used to uniquely identify a record, so primary keys must not contain identical values, which is a mandatory rule in many databases.

In esProc, it is assumed that the primary key has unique value among the records. But no mandatory check will be executed and there won’t be an error reporting even if there are identical primary key values. Both T.pfind(v) function and T.find(v) function can be used to query records in table sequence T according to v, the primary key value. find returns the first record found and pfind returns the sequence number of this record.

It is not a must to define a primary key. When no primary key is defined in esProc, the first field will be used as the primary key. If specified fields are needed to be defined as the primary key, T.primary(Fi,…) function, which means defining Fi,… as the primary key of table sequence T, will be used.

Usually, conventional position functions T.select() and T.pselect() are used to query records in a table sequence. Now we’ll compare usages of this pair of functions with pfind and find.

Here table EMPLOYEE in demo database is the table sequence on which query will be executed. Add FullName field to it so that the employees’ full names can be used to make query:

In order to show the performance advantages of using the primary key to query data, 10,000 full names will be generated randomly and pselect and pfind will be used respectively to make query according to these full names. Compute the time these two methods will take:

 | A
1 | =demo.query("select * from EMPLOYEE")
2 | =A1.derive(NAME+" "+SURNAME:FullName)
3 | =10000.(A2(rand(A2.len())+1).FullName)
4 | =now()
5 | =A3.(A2.pselect(FullName:A3.~))
6 | =interval@ms(A4,now())
7 | =now()
8 | >A2.primary(FullName)
9 | =A3.(A2.pfind(A3.~))
10 | =interval@ms(A7,now())

Based on the same data, A5 and A9 use pselect and pfind respectively to query positions of the records in the table sequence. In A9, primary function is used to set the primary key for the table sequence before pfind starts to work. A6 and A10 compute respectively the milliseconds the queries will take:

But the query results in A5 and A9 are same:

Using similar method, we can compare select function and find function. To keep in line with find function, @1 option is used in select function to get the first result and return it:

 | A
1 | =demo.query("select * from EMPLOYEE")
2 | =A1.derive(NAME+" "+SURNAME:FullName)
3 | =10000.(A2(rand(A2.len())+1).FullName)
4 | =now()
5 | =A3.(A2.select@1(FullName==A3.~))
6 | =interval@ms(A4,now())
7 | =now()
8 | >A2.primary(FullName)
9 | =A3.(A2.find(A3.~))
10 | =interval@ms(A7,now())

A6 and A10 compute respectively the milliseconds the queries will take:

Still, the query results in A5 and A9 are same:

It can be seen easily from the comparison that the query functions based on the primary key are much more efficient than conventional position functions.

Next
Previous
Related :
  • 6. Primary Keys and Index Function


6.2 Primary key and the index table

Why, in esProc, will the efficiency increase significantly when the primary key is used to make query? The reason is that the index table of the primary key has been used for computing.

During calling a primary function, or making query based on the primary key in a table sequence without an index table for the first time, an index table will be generated according to the primary key. While the index table is being generated, a hash table will be created according to all values of the primary key, which will divide primary key values into many groups by their hash values. These hash values are the corresponding group numbers.

Normally, when we query a certain record in a table sequence according to the field value, we need to examine the records one by one until the target is found. For a table sequence containing n records, an average of n/2 examinations are needed.

Thanks to the index table, it would be different to query a certain record in a table sequence according to the value of the primary key field. The hash values will be computed first according to the primary key values, which enable us to find out directly the corresponding groups in the index table. Then we just need to examine records of the same group. In the same way, for a table sequence containing n records, if its primary key values are distributed in k groups according to hash values, only an average of n/2k comparisons are needed. Using this method, despite hash values must be computed before an index table is generated and the query is executed, the number of comparisons is reduced significantly, and in particular, the index table needs to be generated only once. Therefore, the more the data in a table sequence and the times needed for a query, the higher the efficiency.

During computing, T.index(n)function can be used to create an index table for T’s primary key in advance. n represents the length of the index table. Default value will be used if there is not a defined length.

What we should know is that the reason why the functions for query based on primary key values, such as find and pfind, can enhance computational performance effectively is that an index table has been created for the primary key in a table sequence. Therefore, if the primary key itself can be used as the index to locate records, it is unnecessary to create an index table. EID field itself in the above-mentioned table EMPLOYEE, for example, represents the positions of records in the table sequence, thus it is more efficient to use it to query the corresponding records:

 | A
1 | =demo.query("select * from EMPLOYEE")
2 | =10000.(A1(rand(A1.len())+1).EID)
3 | =now()
4 | =A2.(A1(A2.~))
5 | =interval@ms(A3,now())
6 | =now()
7 | =A2.(A1.find(A2.~))
8 | =interval@ms(A6,now())

This time 10,000 sequence numbers of employees are generated randomly. A4 finds the corresponding records directly according to these sequence numbers; A7 still uses find function to find records. A5 and A8 compute respectively the time the two methods will take:

We can see that it is much faster to locate records using sequence numbers directly. Because this method doesn’t compare field values, nor does it compute the hash values and create an index table. The query results in A4 and A7 are same:

Thus it can be seen that suitability should be taken into consideration if the index function of the primary key in a table sequence is to be used to increase efficiency in esProc.

Next
Previous
Related :
  • 6. Primary Keys and Index Function


6.3 switch function

Besides find function and pfind function, switch function query records according to the primary values too. It will also use the index table of the primary key automatically in dealing with the operation. For example:

 | A
1 | =file("D:/files/txt/PersonnelInfo")
2 | =A1.import@b()
3 | =A1.import@b()
4 | =demo.query("select * from STATES")
5 | =now()
6 | >A2.(State=A4.select@1(ABBR:A2.State))
7 | =interval@ms(A5,now())
8 | =now()
9 | >A3.switch(State,A4:ABBR)
10 | =interval@ms(A8,now())

Both A2 and A3 contain personnel information imported from binary text file PersonnelInfo, as shown below:

A4 contains states information:

In both A6 and A9, State field of PersonnelInfo is switched into corresponding states information. Their difference is that A6 uses select@1 function while A9 uses switch function. A7 and A9 compute respectively the time the two methods will take:

After the code in the cellset is executed, values of A2 and A3 are same, as shown below：

State field has been switched into corresponding records in states information table.

Before switch is executed, an index table will also be created for corresponding fields in the table sequence. In this example, an index table is created for ABBR field of states information table in A4 to increase the matching efficiency. So switch function should be properly used when foreign key fields for referencing are to be generated.

Next
Previous
Related :
  • 6. Primary Keys and Index Function


Chapter 4 Database

Next
Previous
Related :
  • 1. Using SQL
  • 2. Database Configuration
  • 3. Data Connection Management
  • 4. Updating Data
  • 5. Calling Database Stored Procedures


1. Using SQL

In esProc, we can use not only the SQL to retrieve data from databases, but also the preliminary database query results to perform further analyses and operations to solve some complicated problems which are difficult to deal with in SQL alone.

Next
Previous
Related :
  • Chapter 4 Database
  • 1.1 Database connection and disconnection
  • 1.2 Using simple SQL
  • 1.3 SQL that returns no results
  • 1.4 Use SQL directly in cells
  • 1.5 Usage of query results of SQL
  • 1.6 Comparison between common SQL statements and esProc syntax


1.1 Database connection and disconnection

When SQL accesses databases, it should first connect to the designated database. Usually, there are two ways to connect to the database: direct connection in the data source manager or calling functions to connect in cellsets.

 | A
1 | =connect("demo")

In practice, a database connection in the data source manager can be referenced by directly using data source name and is valid while the connection is on; for a database connected through functions, the connection object will be stored as cell value, and it can be referenced by using the cell name and is valid before the referenced connection object closes.

 | A
1 | =connect("demo")
2 | =demo.query("select * from STATES")
3 | =A1.query("select * from STATES")
4 | >A1.close()

Similarly, there are two ways for database disconnection as well. Besides calling db.close() function to close the connection mentioned above, another way is to close the specified data source in data source manager:

Next
Previous
Related :
  • 1. Using SQL


1.2 Using simple SQL

Using db.query() function can execute SQL commands in designated database. SQL statements can contain various query sub-clauses and database functions.

 | A
1 | =connect("demo")
2 | =A1.query("select * from STATES where ABBR like 'N%' order by POPULATION desc")
3 | >A1.close()

A2 queries states whose abbreviations begin with N, and sorts them by population in descending order. Results are as follows:

A SQL statement can also uses other data in the cellset as parameters:

 | A
1 | =connect("demo")
2 | [CA,ME,NM,SC,LA]
3 | =A1.query("select * from STATES where ABBR in (?) order by AREA",A2)
4 | >A1.close()

A4 queries states whose abbreviations fall into designated sequences and sorts them by area in ascending order. Results are as follows:

In particular, we can make query function return only the first record by using @1 option with it. In this case, the query result will be returned as a sequence whose members are values of columns of the first record. If the record has only one field, a single value will be returned. For example:

 | A
1 | =demo.query@1("select * from CITIES")
2 | =demo.query@1("select NAME from CITIES where STATEID=5")

Results of A1 and A2 will be those as follows when @1 option is used:

Note that the return value is a single value or a sequence, instead of a table sequence, by using this method.

Next
Previous
Related :
  • 1. Using SQL


1.3 SQL that returns no results

If SQL statements that return no results are to be used to access a database, like create, update, delete and so on, then db.execute() function should be used. Meanwhile, since it is not necessary to assign values to cells, expressions begin with ">" instead of "=". For example:

 | A
n1 | =connect("demo")
2 | >A1.execute("update STATES set ABBR='CAA' where ABBR='CA'")
3 | =A1.query("select * from STATES where NAME = 'California'")
4 | >A1.close()

After the statement in A2 modifies records in database table STATES, query result of A3 is as follows：

SQL that returns no results can use parameters, too:

 | A
1 | =connect("demo")
2 | CA
3 | CAA
4 | >A1.execute("update STATES set ABBR=? where ABBR=?",A2,A3)
5 | =A1.query("select * from STATES where NAME = 'California'")
6 | >A1.close()

The statement in A4 restores the modified record in database STATES to its original values, and query result of A5 is as follows:

Next
Previous
Related :
  • 1. Using SQL


1.4 Use SQL directly in cells

Besides db.query(sql) function and db.execute(sql) function, esProc can also use $(db)sql;… to directly execute SQL. If (db) i s omitted, connect to the last database used. If parameters are used in the sql statement, put them after the semicolons. This method doesn’t add an equal sign before the sql statement and surround the statement with double quotation marks; nor does it support @1 option any more. In this case, execute function or query function is not needed to judge whether the statement returns a result set or not. select statement will return a result set, but other statements will return different values respectively. For example:

 | A
1 | $(demo)select * from STATES where ABBR like 'N%' order by POPULATION desc
2 | [CA,ME,NM,SC,LA]
3 | $select * from STATES where ABBR in (?) order by AREA;A2

The code has the same effect as that in section 2: Using simple SQL and gets the same result when executed. The code in A3 doesn’t specify the datasource name and thus use the last database used, which is still (demo).

Another situation:

 | A
1 | $(demo) create table TESTTEMP (ID int, NAME varchar(20))
2 | $(demo) insert into TESTTEMP values (2,'Tom Smith')
3 | =demo.query("select * from TESTTEMP")
4 | $(demo)update TESTTEMP set NAME='Jane White' where ID=2
5 | =demo.query("select * from TESTTEMP")
6 | $(demo)drop table TESTTEMP

A1 creates a new table sequence. A2 inserts a record into it. A4 modifies the records and A6 drops the table sequence. The updating of the database in A3 and A5 is as follows:

This seems different from the case in the above section. Their return values are as follows:

Values of A1 and A6 mean that the SQL statements don’t return result sets. Values of A2 and A4 mean one record has been updated respectively.

Code will become more concise using the format of $(db)sql;…. Please note that parameters should be separated from each other by semicolons and different returned results need to be identified.

Next
Previous
Related :
  • 1. Using SQL


1.5 Usage of query results of SQL

Query results of SQL can be used in esProc to perform operations, like filtering, sorting and combination, etc., to increase query efficiency or solve some complicated problems.

In the following examples, executions are performed by connecting to data source demo in data source manager based on query results of cell A1:

 | A
1 | =demo.query("select * from STATES order by POPULATION desc")

For example, filter the data and select states with designated abbreviations:

 | A
1 | =demo.query("select * from STATES order by POPULATION desc")
2 | [CA,ME,NM,SC,LA]
3 | =A1.select(A2.pos(ABBR)>0)

Also, aggregation computations can be performed on the data. For example, count the number of states whose abbreviations begin with C:

 | A
1 | =demo.query("select * from STATES order by POPULATION desc")
2 | =A1.count(left(ABBR,1)=="C")

More significantly, we can group data in databases according to certain requirements, for example, group the data according to initials of abbreviations:

 | A
1 | =demo.query("select * from STATES order by POPULATION desc")
2 | =A1.group(left(ABBR,1))

A2 groups the data according to the initial of each state’s abbreviation. Double-click each group and see details.

It is thus clear that, different from the SQL grouping method that doesn’t provide real grouping and summarizing function, grouping with esProc is the real one, on which further computations can be performed. For example, select groups that contain three or more states, compute the total number and population of the states in each of these groups:

 | A
1 | =demo.query("select * from STATES order by POPULATION desc")
2 | =A1.group(left(ABBR,1))
3 | =A2.select(~.count()>=3)
4 | =A3.new(left(ABBR,1):Initial,~.count():Count,~.sum(POPULATION):TotalPopulation)

The final results of A4 are:

Next
Previous
Related :
  • 1. Using SQL


1.6 Comparison between common SQL statements and esProc syntax

1) Select * from

 | A
1 | =demo.query("select * from EMPLOYEE")

Query results are as follows:

2) Select … from

 | A
1 | =demo.query("select * from EMPLOYEE")
2 | =A1.new(EID, NAME, SURNAME, GENDER, BIRTHDAY, DEPT)
3 | =demo.query("select EID, NAME, SURNAME, GENDER, BIRTHDAY, DEPT from EMPLOYEE")

Get designated fields from the table. A2 and A3 have the same query results as follows:

3) As

 | A
1 | =demo.query("select * from EMPLOYEE")
2 | =A1.new(EID, NAME+" "+SURNAME: FULLNAME, GENDER, age(BIRTHDAY):AGE, DEPT)
3 | =demo.query("select EID, NAME+' '+SURNAME as FULLNAME, GENDER, year(now())-year(BIRTHDAY) as AGE, DEPT from EMPLOYEE")

Compute FULLNAME according to NAME and SURNAME, and meanwhile, compute AGE according to BIRTHDAY. Basically, A2 and A3 have the same query results as follows:

Note that AGE is computed in A3 by simply subtracting years and exact computation will be complicated since SQL hasn’t functions to directly compute age.

4) Where

 | A
1 | =demo.query("select * from EMPLOYEE")
2 | =A1.new(EID, NAME+" "+SURNAME: FULLNAME, GENDER, age(BIRTHDAY):AGE, DEPT)
3 | =A2.select(AGE<30)
4 | =demo.query("select EID, NAME+' '+SURNAME as FULLNAME, GENDER, year(now())-year(BIRTHDAY) as AGE, DEPT from EMPLOYEE where year(now())-year(BIRTHDAY)<30")

Select employees who are younger than 30 years old. In esProc, we can perform further computations using the existing results. Query results of A3 are as follows:

Query the same results in A4 with SQL but the syntax is much more complicated. And we cannot get accurate results due to the inexact method of computing age.

5) Count, sum, avg, max and min

 | A
1 | =demo.query("select * from EMPLOYEE")
2 | =A1.new(EID, NAME+" "+SURNAME: FULLNAME, GENDER, age(BIRTHDAY):AGE, DEPT)
3 | =A2.count(AGE<30)
4 | =demo.query("select count(EID) from EMPLOYEE where year(now())-year(BIRTHDAY)-(case when month(now())<month(BIRTHDAY) then 1 WHEN month(now())=month(BIRTHDAY) and day(now())<day(BIRTHDAY) then 1 else 0 end)<30")

Count the total number of employees who are younger than 30 years old. In esProc, the result can be used to perform further computation. Query result of A3 is as follows:

This time, A4 uses a more exact method to compute AGE and gets a query result that is consistent with that of A3. But with this method, we cannot use the existing results and the statements are more complicated.

The usage of SQL functions, such as sum, avg, max and min, is similar to that of count.

6) Distinct

 | A
1 | =demo.query("select * from EMPLOYEE")
2 | =A1.id(DEPT)
3 | =demo.query("select distinct DEPT from EMPLOYEE")

Query which departments the employee information come from. A2 and A3 have the same results. Query results are as follows:

7) Order by

 | A
1 | =demo.query("select * from EMPLOYEE")
2 | =A1.new(EID, NAME+" "+SURNAME: FULLNAME, GENDER, age(BIRTHDAY):AGE, DEPT)
3 | =A2.select(AGE<30).sort(AGE:-1, FULLNAME ).new(FULLNAME,AGE)
4 | =demo.query("select FULLNAME, AGE from (select NAME+' '+SURNAME as FULLNAME, year(now())-year(BIRTHDAY)-(case when month(now())<month(BIRTHDAY) then 1 WHEN month(now())=month(BIRTHDAY) and day(now())<day(BIRTHDAY) then 1 else 0 end) as AGE from EMPLOYEE) where AGE<30 order by AGE desc, FULLNAME")

Select employees who are younger than 30 years old, sort them by age in descending order; meanwhile, sort employees of the same age by FULLNAME in ascending order. A3 and A4 have the same query results as follows:

In SQL, it is complicated to compute age and existing results cannot be used, so this time A4 tries to simplify the statements with nested query. However, the process is still complex.

8) And, or, not and <>

 | A
1 | =demo.query("select * from EMPLOYEE")
2 | =A1.new(EID, NAME+" "+SURNAME: FULLNAME, GENDER, age(BIRTHDAY):AGE, DEPT)
3 | =A2.select(AGE<30 && left(FULLNAME,1)=="S").new(FULLNAME, AGE)
4 | =demo.query("select FULLNAME, AGE from (select NAME+' '+SURNAME as FULLNAME, year(now())-year(BIRTHDAY)-(case when month(now())<month(BIRTHDAY) then 1 WHEN month(now())=month(BIRTHDAY) and day(now())<day(BIRTHDAY) then 1 else 0 end) as AGE from EMPLOYEE) where AGE<30 and left(FULLNAME, 1)='S'")

Select employees who are younger than 30 years old and whose initials of full names are S. Results are as follows:

It can be seen that and is represented in esProc by the operator && and two equal signs == are used to judge whether things are equal or not. These are in line with the customs of many program languages. Similarly, in esProc, or is represented by the operator "||", not by"!", and<>by"!=".

9) Like

 | A
1 | =demo.query("select * from EMPLOYEE")
2 | =A1.select(like(NAME,"*a")).new(NAME+" "+SURNAME:FULLNAME)
3 | =demo.query("select NAME+' '+SURNAME as FULLNAME from EMPLOYEE where NAME like '%a'")

Query the full names of employees whose names are ended by a. Query results are as follows:

The use of like function requires different wildcard characters in different databases. In this example, for instance, percent "%" is used to represent zero or multiple arbitrary characters; while in some other databases, asterisk "*" is used to represent the same things. esProc defines the same syntax for any database.

10) Group

 | A
1 | =demo.query("select * from EMPLOYEE")
2 | =A1.new(NAME+" "+SURNAME: FULLNAME, DEPT).group(DEPT)
3 | =A1.groups(DEPT;count(~):ECOUNT)
4 | =demo.query("select DEPT, count(*) as ECOUNT from EMPLOYEE group by DEPT order by DEPT")

group function can be used in esProc to group records of employees by departments as shown below:

It can be seen that the result of grouping with esProc is that records are divided into multiple groups. These groups can be used to perform further computations as required in esProc.

A3 directly computes data grouping and summarizing with the esProc function while A4 does the job with SQL. They get the same results. SQL doesn’t have the real "group" concept, so it can only perform aggregate computations during query by groups. Results are as follows:

Next
Previous
Related :
  • 1. Using SQL


2. Database Configuration

esProc supports a variety of heterogeneous datasources, including the databases. This article will introduce the methods of esProc’s connection to various databases.

Next
Previous
Related :
  • Chapter 4 Database
  • 2.1 Configure datasource manager
  • 2.2 Retrieve data from databases
  • 2.3 Configure other databases
  • 2.4 esProc JDBC


2.1 Configure datasource manager

To connect to a relational database, please follow these steps:

Click Datasource connection on Tool options menu to open the datasource manager.

Click New to create a datasource and select its type.

Select information like database type, charset, and etc. Set connection parameters for the datasource and name them.

Connect to the database. The datasource manager will display the result of connection and supports connecting to multiple databases.

esProc has its own system datasource – demo, which is located in esProc\bin in IDE installation directory. Find it and run startDataBase.bat to launch the datasource demo.

esProc’s Integration Development Environment (IDE) provides JDBC configure prompts for many types of databases, including SQL server, Oracle, DB2, Sybase, Access, MySQL, HSQL, Teradata, PostgreSQL and etc. esProc programmers need to prepare these databases’ JDBC drivers, which are provided by database service providers, by themselves and put them into \common\jdbc in esProc’s IDE installation directory.

Next
Previous
Related :
  • 2. Database Configuration


2.2 Retrieve data from databases

After the drivers are put in place and configured, programmers can connect esProc to the databases in IDE conveniently and select desired data from them. For example:

 | A
1 | =demo.query("select * from CITIES")
2 | $ select * from CITIES
3 | =connect("demo")
4 | =A3.query("select * from CITIES")
5 | =interval@ms(A6,now())

In the cellset, A1 uses the connected datasource - "demo" – in the datasource manager to query data. A2 uses the database connected by default to query data. A4 uses the database connection created by A3 to query data. Programmers can also control the created database connection with program and for further discussion, please refer to esPorc Accesses Databases: Data Connection Management. A1, A2 and A4 return the same results:

Here SQL is used to return directly the result of querying data from a database. Please refer to esProc Accesses Databases: Using SQL for details.

Next
Previous
Related :
  • 2. Database Configuration


2.3 Configure other databases

esProc can also connect to a database through JDBC-ODBC bridge. The ODBC driver also needs to be prepared by programmers. The esProc IDE’s ODBC configuration interface is as follows:

Select the ODBC datasource for configuration from the list and set related parameters including user name and password.

If the database to which one wants to connect is outside of the list of databases provided by esProc, one can choose Other type for connection. The configuration interface is as follows:

In the same way, put the JDBC driver used here into \common\jdbc in esProc’s IDE installation directory and enter the name of the driver, database URL, and other information into the configuration page.

Next
Previous
Related :
  • 2. Database Configuration


2.4 esProc JDBC

esProc JDBC takes esProc as a built-in application. In other web applications, programmers can call a cellset program using a JDBC-like connection. With esProc JDBC, a cellset program is packaged as a stored procedure, so the method of calling a cellset program is the same as that of calling a stored procedure.

The cellset code for an esProc file will return the result set using result function, like the test.dfx file shown below:

 | A
1 | =connect("demo")
2 | =A1.query("select * from EMPLOYEE where EID=?",arg1)
3 | >A1.close()
4 | result A2

In the above code, arg1 is a cellset argument, which should get assigned when dfx file is called. The computed result is a table sequence which is returned using a result statement. Please note that some of the files called by JDBC may need to connect to other datasources, like "demo" in this example. Programmers also need to set parameters for these datasources, and they should handle the datasource configuration in esProc JDBC in configuration file config.xml or dfxConfig.xml.

1) Method One for configuring datasources: Set connection parameters for the database datasource directly in config.xml:

<?xml version="1.0" encoding=" UTF-8"?>

< Config Version="1">

<DBList>

<!-- datasource name, which must be the same as that in the dfx file-->

<DB name="demo">

<property name="url" value="jdbc:hsqldb:hsql://127.0.0.1/demo"/>

<property name="driver" value="org.hsqldb.jdbcDriver"/>

<property name="type" value="HSQL"/>

<property name="user" value="sa"/>

<property name="password" value=""/>

<property name="batchSize" value="1000"/>

<!--

Automatically connect or not. If the value is true, db.query() function can be used directly to access the database; if it is false, the connection won’t be established automatically and thus connect(db) statement must be used to establish the connection first.

-->

<property name="autoConnect" value="true"/>

<property name="useSchema" value="false"/>

<property name="addTilde" value="false"/>

</DB>

</DBList >

</dfxConfig>

2) Method Two for configuring datasources: Configure Application Server connection pool in dfxConfig.xml.

<?xml version="1.0" encoding=" UTF-8"?>

<dfxConfig>

<!-- specify the datasource name here -->

<jndi-ds-configs>

<!—prefix jndi -->

<jndi-prefix>java:comp/env</jndi-prefix>

<!-- datasource name, which must be the same as that in the dfx file -->

<jndi-ds-config>

<name>demo</name>

<dbType>HSQL</dbType>

<dbCharset>ISO-8859-1</dbCharset>

<clientCharset>ISO-8859-1</clientCharset>

<needTranContent>false</needTranContent>

<needTranSentence>false</needTranSentence>

<!--

Automatically connect or not. If the value is true, db.query() function can be used directly to access the database; if it is false, the connection won’t be established automatically and thus connect(db) statement must be used to establish the connection first.

-->

<autoConnect>true</autoConnect>

</jndi-ds-config>

</jndi-ds-configs>

</dfxConfig>

During configuration, the names of configuration files must be config.xml and dfxConfig.xml and are not allowed to be changed. While configuring information for connecting to a database, reentering is forbidden and esProc JDBC itself cannot be used as a data source and configured in a database connection.

This article only introduces the methods for configuring datasources in esProc JDBC. For using and configuring esProc JDBC, please refer to Integration and Use of esProc JDBC.

Once the datasource has been configured, the above-mentioned dfx can be called by a Java program:

public void testDataServer(){

Connection con = null;

com.esproc.jdbc.InternalCStatement st;

com.esproc.jdbc.InternalCStatement st2;

try{

//create a connection

Class.forName("com.esproc.jdbc.InternalDriver");

con= DriverManager.getConnection("jdbc:esproc:local://");

// call the stored procedure, in which test is the name of file dfx

st =(com. esproc.jdbc.InternalCStatement)con.prepareCall("call test(?)");

// set parameters

st.setObject(1,"3");

// the result of the following statement is the same as that got by using the above calling methods

st =(com. esproc.jdbc.InternalCStatement)con.prepareCall("call test(3)");

// execute the stored procedure

st.execute();

// get the result set

ResultSet set = st.getResultSet();

// create a Statement directly

st2=(com. esproc.jdbc.InternalCStatement)con.createStatement();

// execute the cellset file directly and get the same result as that of the call test(3) in the above

set=st2.executeQuery("test 3");

// execute the statement immediately and query data from a specified datasource

set=st2.executeQuery("$(demo)select * from STUDENTS");

}

catch(Exception e){

System.out.println(e);

}

finally{

// close the database connection

if (con!=null) {

try {

con.close();

}

catch(Exception e) {

System.out.println(e);

}

}

}

}

In the above code, the connecting string of esProc JDBC is "jdbc:esproc:local://" and in this case, default configuration will be used; if ...?config=...; is used in the connecting string, use the configuration of a specified .xml file and ignore the definitions in the config.xml.

For example, con= DriverManager.getConnection("jdbc:esproc:local:/?config=myconfig.xml"); should use the configuration of myconfig.xml.

Next
Previous
Related :
  • 2. Database Configuration


3. Data Connection Management

During our dealing with the database transactions, some operations may produce errors which could bring unexpected results, particularly during batch processes. In order to prevent this from happening, we need to manage the data connection in a right way and handle those errors properly.

Next
Previous
Related :
  • Chapter 4 Database
  • 3.1 Database error messages
  • 3.2 Commit and rollback transaction management


3.1 Database error messages

First let’s look at the database error messages. We’ll use an Access file - DbCon.accdb – as the target database, and then create an ODBC datasource in esProc. Use the data of the file directly and enter the connecting string DRIVER=Microsoft Access Driver (*.mdb, *.accdb);DBQ=D:\\files\\DbCon.accdb :

Create an empty table CityBak in the Access file - DbCon.accdb:

In the table, ID is the primary key. A limitation on POPULATION that the data of it should be greater than 1,000,000 has been set.

The following will introduce how to enter the data of table CITIES of demo database in the CityBak:

 | A
1 | =demo.query("select * from CITIES")
2 | =connect("DbCon")
3 | =A2.query("select * from CityBak")
4 | >A2.update(A1,CityBak,ID:CID,CITY:NAME,POPULATION,STATE:STATEID)
5 | >A2.close()

A1 selects the data of CITIES from demo database:

A2 creates a connection to database DbCon. A3 gets the data of CityBak. But as a newly-created table, it hasn’t any records:

The program reports an error when A4 is executed. According to the limitation that population should be greater than 1,000,000, the 10th record Detroit is ineligible. By default the program ends when an error message is printed out.

In this case, we can view the data written to the CityBak in another dfx file:

 | A
1 | =connect("DbCon")
2 | =A1.query("select * from CityBak")
3 | >A1.close()

The query result of A2 is as follows:

It can be seen that, though the execution of the program in the first cellset failed because of the limitation on the data of CityBak, still some data had been written to the targeted table successfully before the first ineligible record appeared.

The program will stop when a database error occurs during the batch database update. In order to avoid this interruption, @e option can be used at the time of creating the connection to allow programmer to handle the error by codes. For example:

 | A | B
1 | =demo.query("select * from CITIES") |
2 | =connect@e("DbCon") |
3 | >A2.execute("delete from CityBak") |
4 | >A2.update(A1,CityBak,ID:CID,CITY:NAME,POPULATION,STATE:STATEID) | =A2.error()
5 | =A2.query("select * from CityBak") |
6 | >A2.close() |

In A2, @e option is added to the connect function to let programmer handle the error message if there is one, without interrupting the program. A2 deletes the existing data from CityBak to maintain a data consistency. When the program is executed, the query result of A5 is the same as that of A2 in the above:

While A4 is updating the data of CityBak, an operation that is not in line with the above requirement is performed. So we can see the error code in B4:

Note: Once an error occurs during the batch database update, the program will stop and report an error. By modifying the expression in B4 into =A2.error@m(), we can see the error message, as shown below:

Let’s go on with our discussion:

 | A | B
1 | =demo.query("select * from CITIES") |
2 | =connect@e("DbCon") |
3 | >A2.update@a(A1.select(CID<6), CityBak,ID:CID,CITY:NAME, POPULATION,STATE:STATEID) | =A2.error()
4 | =A2.query("select * from CityBak") |
5 | >A2.close() |

By adding @a option to db.update(), A3 will delete the existing data from the database table before it execute the update, saving us the trouble of specially writing code for deleting the records. A3 selects only the top 5 records and add them to CityBak. Since all these records meet the requirement, the statement in A3 has been executed smoothly and the error code in B3 is 0:

And as can be seen from A5, the records have been written to the table successfully:

Next
Previous
Related :
  • 3. Data Connection Management


3.2 Commit and rollback transaction management

We cannot know beforehand if the batch update will be completely executed, or which record will produce an error. By default the update of each record will be committed automatically and the result is unpredictable. This is not good for database management.

Sometimes whether a transaction should be committed or canceled depends on different situations. And db.commit() and db.rollback() are used to determine which operation we will perform.

By default, a statement in esProc will be committed automatically after it is executed. In this case, the situation is out of our control. In order to control the transaction commit through such code as db.commit() and db.rollback(), we need to add @k option to the execute or update statement. Then the validity of the data can be determined according to the error code. As shown in the following example, if an error occurs during batch update, all the data will become invalid. In this way, the database can be saved from the unpredictable results:

 | A | B
1 | =demo.query("select * from CITIES") |
2 | =connect@e("DbCon") |
3 | >A2.execute("delete from CityBak") |
4 | >A2.update@k(A1,CityBak,ID:CID,CITY:NAME,POPULATION,STATE:STATEID) |
5 | if A2.error()==0 | >A2.commit()
6 | else | >A2.rollback()
7 | =A2.query("select * from CityBak") |
8 | >A2.update@k(A1.to(5),CityBak,ID:CID, CITY:NAME,POPULATION,STATE:STATEID) |
9 | if A2.error()==0 | >A2.commit()
10 | else | >A2.rollback()
11 | =A2.query("select * from CityBak") |
12 | >A2.close() |

A3 deletes the data from CityBak. It will commit the transaction automatically due to the lack of @k option in db.execute().

An error occurs in A4 when batch update is executed, which is shown in the result of A5:

That’s why the rollback transaction in B6, instead of the commit transaction in B5, has been executed. The query result in A6 is as follows:

If no error is reported during batch update with the use of @k option, like the statement executed in A8, we can still determine the data validity according to the error code. The value of A9 is as follows:

In this case, the commit transaction in B9 will be executed and the data will be written to the database. The query result of A11 is as follows:

Next
Previous
Related :
  • 3. Data Connection Management


4. Updating Data

Besides being used to query data in a database, SQL is also used to update them. In esProc, you can use the SQL commands directly to return a query result, or use update function to modify a database with the data from a table sequence or a sequence.

Next
Previous
Related :
  • Chapter 4 Database
  • 4.1 Update a database with SQL commands
  • 4.2 Update a database with data from a table sequence or a sequence


4.1 Update a database with SQL commands

In esProc, db.execute(sql) function can be used to execute the SQL commands in a database, so as to realize various update operations. For example:

 | A | B
1 |  | /=demo.query("select * from TEST")
2 | >demo.execute("create table TEST(ID int,FULLNAME varchar(50), STATE varchar(20))") | =demo.query("select * from TEST")
3 | >demo.execute("insert into TEST values (1,'Tom Smith','Florida')") |
4 | >demo.execute("insert into TEST values (2,'Michael Jones','Montana')") | =demo.query("select * from TEST")
5 | >demo.execute("update TEST set STATE='New York' where ID=1") | =demo.query("select * from TEST")
6 | >demo.execute("delete from TEST where STATE='Montana'") | =demo.query("select * from TEST")
7 | >demo.execute("drop table TEST") | /=demo.query("select * from TEST")

In this cellset, SQL commands are used to create a new table – TEST, insert records into it, update records in it and delete records from it, and drop it finally. Column B provides the viewing of the data in TEST after each step is executed. In B2 you can see the structure of the table in which there are no records:

A3 and A4 insert a record respectively into the table using SQL commands. The result can be seen in B4:

A5 modifies a record and the result can be seen in B5:

It can be seen that the order of records in the database has been changed after A5 modified the state in which Tom Smith stays. In fact, the records stored in various databases are usually not in a fixed order, so the data updating will change the original order, which will bring some trouble.

A6 deletes the record of the state of Montana and the result is as follows:

A7 drops the table finally. Because query function cannot retrieve data before the table is created and after it is dropped, the program will report error when query commands are executed in B1 and B7.

Parameters can also be used when execute statement is used to perform the SQL commands. For example:

 | A | B
1 |  | /=demo.query("select * from TEST")
2 | >demo.execute("create table TEST(ID int,FULLNAME varchar(50), STATE varchar(20))") | =demo.query("select * from TEST")
3 | >demo.execute("insert into TEST values (?,?,?)",1,"Tom Smith","Florida") |
4 | >demo.execute("insert into TEST values (?,'Michael Jones','Montana')",2) | =demo.query("select * from TEST")
5 | >demo.execute("update TEST set STATE='New York' where ID=?",1) | =demo.query("select * from TEST")
6 | >demo.execute("delete from TEST where STATE=?","Montana") | =demo.query("select * from TEST")
7 | >demo.execute("drop table TEST") | /=demo.query("select * from TEST")

While adding parameters to an execute statement, the positions corresponding to the parameters should be replaced with question marks and the parameters will be entered after the SQL statement in order. As the code in A3 shows, for example, values to be inserted can be all replaced by parameters, or as the code in A4 shows, not all of them will be replaced. Please note that a string in esProc’s functions should be enclosed by double quotation marks ("…"), which is different from the syntax of SQL.

When an execute statement is executed, the result will be committed automatically to the database by default. Once an error occurs, the program will stop. For example:

 | A | B | C
1 | 1 | Tom Smith | Florida
2 | 2 | Michael Jones | Montana
3 | three | William Bush | New York
4 | 4 | Violet Taylor | Montana
5 | >demo.execute("create table TEST(ID int,FULLNAME varchar(50), STATE varchar(20))") |  |
6 | =create(Fd1,Fd2,Fd3).record(A1:C4) |  |
7 | for A6 | =demo.execute("insert into TEST values (?,?,?)",A7.Fd1,A7.Fd2,A7.Fd3) |
8 |  | =demo.query("select * from TEST") |
9 | =demo.execute("drop table TEST") |  |

The table sequence in A6 is as follows:

The code block in A7 performs loop on the above-mentioned table sequence, insert records into TEST of the database and query the result in B8. The program reports an error and stops while trying to insert the third record into the table, because three is not an integer and thus cannot be inserted. At this point, the data in TEST can be viewed in B8:

This means only some of the data have been inserted into the database and the code for dropping the table in A9 hasn’t been executed. Obviously, this is not the result we expected.

Therefore, we need to get the error code under control and commit the transaction using commands. To achieve the target, default database connection should be given up and connect@e() will be used to create a connection to the database instead; meanwhile execute@k will be used to execute the SQL command without committing the result automatically:

 | A | B | C
1 | 1 | Tom Smith | Florida
2 | 2 | Michael Jones | Montana
3 | three | William Bush | New York
4 | 4 | Violet Taylor | Montana
5 | =connect@e("demo") | >A5.execute("create table TEST(ID int,FULLNAME varchar(50), STATE varchar(20))") |
6 | =create(Fd1,Fd2,Fd3).record([A1:C4]) |  |
7 | for A6 | >A5.execute@k("insert into TEST values (?,?,?)",A7.Fd1,A7.Fd2,A7.Fd3) |
8 | if A5.error()==0 | >A5.commit() |
9 | else | >A5.rollback() |
10 | =A5.query("select * from TEST") | >A5.execute("drop table TEST") | >A5.close()

In this cellset, @e option is used in connect function to connect to the database. In this way, when an error occurs during the loop in B7 for updating the database, it will be recorded instead of causing an interruption to the program. Finally, we can see in A8 if the error code is a zero:

The error code is not a zero, which means an error occurred during the execution. Since @k option is used in the execute statement in B7, the result won’t be committed automatically. Thus when the rollback in B9 is executed, we can see in B10 that all the records haven’t been committed as an error occurred:

For details of the data connection control, please refer to esProc Accesses Databases: Data Connection Management.

Next
Previous
Related :
  • 4. Updating Data


4.2 Update a database with data from a table sequence or a sequence

In the above example, for statement for looping was used to update the data of a table sequence into another table sequence. While, in esProc, update function can be used to directly update the data from a sequence or a table sequence into another table sequence. For example:

 | A | B | C
1 | 1 | Tom Smith | Florida
2 | 2 | Michael Jones | Montana
3 | 3 | William Bush | New York
4 | 4 | Violet Taylor | Montana
5 | =connect@e("demo") | >A5.execute("create table TEST(ID int,FULLNAME varchar(50), STATE varchar(20))") |
6 | =create(Fd1,Fd2,Fd3).record([A1:C4]) |  |
7 | >A5.update(A6,TEST,ID:Fd1,FULLNAME:Fd2,STATE:Fd3;ID) |  |
8 | =A5.query("select * from TEST") | >A5.execute("drop table TEST") | >A5.close()

The table sequence in A6 is as follows:

update function is used in A7 to update the data of the table sequence in A6 into table TEST in the database. The function first specifies a source table - A6 and a target table – TEST, then defines the correspondence between fields of the two tables and the primary key – ID – for updating the database. If the primary key is not specified, it will be got automatically from the database. The query result of A8 is as follows:

It can be seen that the result of updating a database with update function is the same as that using a for loop.

When the primary key for updating the database has been defined and there are already the data in the database that use the same primary key, what will we do then? Let’s look at the following example:

 | A | B | C
1 | 1 | Tom Smith | Florida
2 | 2 | Michael Jones | Montana
3 | 3 | William Bush | New York
4 | 4 | Violet Taylor | Montana
5 | =connect@e("demo") | >A5.execute("create table TEST(ID int,FULLNAME varchar(50), STATE varchar(20))") |
6 | >demo.execute("insert into TEST values (1,'TOM SMITH','FL')") | >demo.execute("insert into TEST values (20,'MICHAEL','MT')") | =demo.query("select * from TEST")
7 | =create(Fd1,Fd2,Fd3).record([A1:C4]) |  |
8 | >A5.update(A7,TEST,ID:Fd1,FULLNAME:Fd2,STATE:Fd3;ID) |  |
9 | =A5.query("select * from TEST") | >A5.execute("drop table TEST") | >A5.close()

First, A6 and B6 insert two records respectively into TEST. Then C6 selects all the data before the database was updated using update function from TEST, as shown below:

After that, update function is executed in A8 and the data in TEST can be viewed in A9:

As can be seen from the above, the records that use the same primary key have been updated, but the record – MICHAEL – which has a different primary key remain unchanged.

Some options can be used in update function to represent different operations: @u means updating the data without inserting new ones; @i means inserting data without updating the existing values; @a means deleting all the records from the target table before updating; @1 means that the first field is auto-increment and won’t get assigned during updating.

If we modify the statement in A8 in the previous example into =A5.update@u(A7,TEST,ID:Fd1,FULLNAME:Fd2,STATE:Fd3;ID), the query result in A9 will be as follows:

update@u in A8 is executed to only modify the existing records.

But if the statement in A8 in the previous example is modified into =A5.update@a(A7,TEST,ID:Fd1,FULLNAME:Fd2,STATE:Fd3;ID), the query result in A9 will be as follows:

In A8, the original records in the table will be deleted before update@a is executed. The use of update@a can ensure a consistency between the current data in the table and the source data used for updating.

Similar to the direct execution of a SQL statement, @k option can also be used in update function. In this case the data connection control should be considered. For more information, please refer to esProc Accesses Databases: Data Connection Management.

Next
Previous
Related :
  • 4. Updating Data


5. Calling Database Stored Procedures

Besides data tables, store procedures are also very important for a database. esProc can call database stored procedures conveniently. This article will illustrate in detail how to call various stored procedures with db.proc() function in esProc.

Next
Previous
Related :
  • Chapter 4 Database
  • 5.1 Call stored procedures without return values
  • 5.2 Call stored procedures which return a single value
  • 5.3 Call stored procedures which return a single result set
  • 5.4 Call stored procedures which return multiple result sets
  • Appendix: Definition of Parameter Types


5.1 Call stored procedures without return values

Stored procedures are used to return query results or only to implement the database operations without returning results. Let’s first look at how to call stored procedures without return values. Take an oracle stored procedure as an example. The stored procedure proc1 has only one input parameter, and no output parameter:

create or replace procedure pro1

(pid IN VARCHAR)

as

begin

insert into emp values(pid,'mike');

update emp set name='rose' where id=pid;

commit;

end;

execute function or proc function can be used in esProc to call this stored procedure:

 | A
1 | 1. =connect("ora")
2 | 2. >A1.execute("{call proc1(?)}",4)
3 | 3. >A1.close()

A1 connects to the database. A2 uses execute function o call the stored procedure, with the input parameter value being 4. When the stored procedure is executed, connection will be closed in A3.

 | A
1 | 4. =connect("ora")
2 | 5. >A1.proc("{call proc1(?)}",4:0:"i":)
3 | 6. >A1.close()

A1 connects to the database and A3 closes the connection. Different from the above code, a description is inserted after the parameter when A2 calls the stored procedure proc1. esProc’s proc function can be used to call stored procedures without returned parameters, like proc1, though it is mainly used to call stored procedures that return values and result sets.

Define the complete format of each parameter as a:t:m:v, in which a represents the parameter value, t represents the parameter’s data type, m represents the input/output model of the parameter and v represents the parameter name of the returned result if the stored procedure has one, when calling a stored procedure. In this piece of code, 4:0:"i": defines an input parameter, in which 4 is the input parameter value, 0 represents that the type of input parameter is automatically identified by esProc and "i" represents the input type. If the parameter type needs to be specified manually, the code can be written as 4:1:"i":, in which the 1 in the middle represents Integer(int). For more information about parameter types supported by esProc, please see Appendix: Definition of Parameter Types.

Next
Previous
Related :
  • 5. Calling Database Stored Procedures


5.2 Call stored procedures which return a single value

Stored procedures with returned values can be called only by proc function. The stored procedure below will return one parameter value, so proc function, instead of execute function, will be used.

create or replace procedure testb

(para1 in varchar2,para2 out varchar2)

as

begin

select name into para2 from emp where id= para1;

end testb;

The code for calling this stored procedure by esProc is:

 | A
1 | 7. =connect("ora")
2 | 8. =A1.proc("{call testb(?,?)}",1:0:"i":,:11:"o":name)
3 | 9. =name
4 | 10. >A1.close()

A1 connects to the database and A4 closes the connection.

A2 uses proc function to call the stored procedure testb. Here two parameters are used: one is 1:0:”i” representing that the value is 1 and the type of the input parameter is automatically identified; the other is 11:”o”:name in which 11 represents string type (see Appendix: Definition of Parameter Types for more), ”o” represents the type of output parameter and name defines an esProc variable to receive the return value.

A3 gets the value of variable name, which is the output result of the stored procedure. The type of the variable value is determined by the stored procedure. In this example, name is used to return the name of the employee of specified number and the data type of the returned result is string.

There are 3 input/output types: input parameter, output parameter as well as input/output parameter, which are represented respectively by "i","o" and "io". The input/output parameter in a stored procedure can both input data and get assigned by the stored procedure.

Next
Previous
Related :
  • 5. Calling Database Stored Procedures


5.3 Call stored procedures which return a single result set

In more cases, stored procedures return result sets. Stored procedure RQ_TEST_CUR returns a single result set:

CREATE OR REPLACE PROCEDURE RQ_TEST_CUR

(

V_TEMP OUT TYPE.RQ_REF_CURSOR,

PID IN VARCHAR

)

AS

BEGIN

OPEN V_TEMP FOR SELECT * FROM TEST WHERE ID =PID;

END RQ_TEST_CUR;

The stored procedure has an input parameter and returns a result set using a cursor. The code for calling the stored procedure in esProc is as follows:

 | A
1 | 11. =connect("ora")
2 | 12. =A1.proc("{call RQ_TEST_CUR(?,?)}",:101:"o":table1,1:0:"i":)
3 | 13. =table1
4 | 14. >A1.close()

A2 uses proc function to call the stored procedure: =A1.proc("{call RQ_TEST_CUR(?,?)}", :101:"o":table1, 1:0:"i":). The following will explain the input parameters of proc function.

The SQL strings "{call RQ_TEST_CUR(?,?)}" contains the name of the stored procedure. The question marks represent SQL’s parameters.

Parameters in the stored procedure:

:101:"o":table1 defines an output parameter. That it begins directly with a colon means it hasn’t an input value. 101 represents that its data type is cursor, ”o” represents that it is an output parameter and table1 defines a variable which is used to reference the returned result.

1:0:"i": defines an input parameter, in which 1 is the value of the input parameter, 0 represents that the parameter type is automatically identified by esProc , "i" represents that it is an input parameter and that it ends directly with a colon means there is no need to output the result.

A3 uses output variable in A2 to reference the execution result of the stored procedure. The computed result is a table sequence containing query results from table TEST, which is the same as that in A2.

Next
Previous
Related :
  • 5. Calling Database Stored Procedures


5.4 Call stored procedures which return multiple result sets

Stored procedures can also return multiple result sets. For example, the following oracle stored procedure returns two result sets using the cursor:

create or replace procedure proAA

(

out_var out sys_refcursor,

out_var2 out sys_refcursor

)

as

begin

openout_var for select * from emp;

open out_var2 for select * from test;

end;

The simple stored procedure returns result sets of two tables: emp and test. Call the stored procedure in esProc and the program for receiving the two result sets is as follows:

 | A
1 | 15. =connect("ora")
2 | 16. =A1.proc("{call proAA(?,?)}",:101:"o":a,:101:"o":b)
3 | 17. =A2(1)
4 | 18. =a
5 | 19. =b
6 | 20. >A1.close()

A2 uses proc function to call the stored procedure: =A1.proc("{call proAA(?,?)}",:101:"o":a,:101:"o":b) , which returns two result sets (table sequences) to form a sequence, i.e., a set of table sequences. The sequence assigns value to A2. The following will explain the input parameters in proc function.

SQL string "{call proAA(?,?)}" contains the stored procedure’s name. The question marks represent the SQL’s parameters.

Output parameter 1

:101:"o":a defines an output parameter, in which 101 represents that its data type is cursor, ”o” represents it is an output parameter and a defines a variable which is used to output the returned results to this variable.

Output parameter 2

:101:"o":b defines an output parameter, in which 101 represents that its data type is cursor, ”o” represents it is an output parameter and b defines a variable which is used to output the returned results to this variable.

A3 returns A2’s first table sequence (table emp’s result set)

A4 and A5 use respectively the output variables a and b in A2 to obtain the execution results of corresponding stored procedures. a corresponds to data in table emp and assigns value to A4; b corresponds to data in table test and assigns value to A5. Actually, the result of A2 is the sequence composed of the table sequences in A4 and A5.

Next
Previous
Related :
  • 5. Calling Database Stored Procedures


Appendix: Definition of Parameter Types

Values of type in stored procedures are:

public final static byte DT_DEFAULT = (byte) 0; //default, automatically identify

public final static byte DT_INT = (byte) 1; //a 32-bit integer

public final static byte DT_LONG = (byte) 2; //a 64-bit long integer

public final static byte DT_SHORT = (byte) 3; //a 16-bit short integer

public final static byte DT_BIGINT = (byte) 4; //big integer

public final static byte DT_FLOAT = (byte) 5; //a 32-bit floating point

public final static byte DT_DOUBLE = (byte) 6; //a 64-bit double-precision floating point

public final static byte DT_DECIMAL = (byte) 7; //a big decimal

public final static byte DT_DATE = (byte) 8; //date

public final static byte DT_TIME = (byte) 9; //time

public final static byte DT_DATETIME = (byte) 10; //date/time

public final static byte DT_STRING = (byte) 11; //string

public final static byte DT_BOOLEAN = (byte) 12; //boolean

public final static byte DT_INT_ARR = (byte) 51; //a sequence of integer

public final static byte DT_LONG_ARR = (byte) 52; //a sequence of long integer

public final static byte DT_SHORT_ARR = (byte) 53; // a sequence of short integer

public final static byte DT_BIGINT_ARR = (byte) 54; //a sequence of big integer

public final static byte DT_FLOAT_ARR = (byte) 55; //a sequence of floating point

public final static byte DT_DOUBLE_ARR = (byte) 56; //a sequence of double-precision floating point

public final static byte DT_DECIMAL_ARR = (byte) 57; //a sequence of big decimal

public final static byte DT_DATE_ARR = (byte) 58; //a sequence of date

public final static byte DT_TIME_ARR = (byte) 59; // a time sequence

public final static byte DT_DATETIME_ARR = (byte) 60; //a sequence of data/time

public final static byte DT_STRING_ARR = (byte) 61; //a string sequence

public final static byte DT_BYTE_ARR = (byte) 62; //a sequence of byte

public final static byte DT_CURSOR = (byte) 101; //cursor

public final static byte DT_AUTOINCREMENT = (byte) 102; //self-increment field

Next
Previous
Related :
  • 5. Calling Database Stored Procedures


Chapter 5 Advanced Code

Next
Previous
Related :
  • 1. Code Block
  • 2. Long Statement
  • 3. Creative Usage of Strings
  • 4. Clear Cell Values
  • 5. Special Identifiers


1. Code Block

In esProc Getting Started: Types of Cells, we learned that cells in esProc have many types, including constant cell, computational cell, executable cell and comment cell, and etc. The code blocks into which code is written in esProc also have the similar types: computational block, assignment block, executable block and comment block.

Next
Previous
Related :
  • Chapter 5 Advanced Code
  • 1.1 Computational block
  • 1.2 Assignment block and executable block
  • 1.3 Comment block


1.1 Computational block

In esProc advanced coding: Long Statements, we learned long statements and computational sub-statements. Instead of =, the cell strings in their master cells begin with ==, which tells esProc to regard the code block with this master cell as a single statement which cannot be separated and thus compute it as a whole. The program will ignore other cells after it executes the statement as required. For example:

 | A | B | C
1 | ==demo.query( | "select NAME as CITY, |
2 |  | STATEID as STATE from CITIES") |
3 | ==A1.select(??) | =demo.query( | "select * from STATES where STATEID=?",
4 |  |  | STATE)
5 |  | if left(B3.ABBR,1)=="N" | >STATE=B3.NAME
6 |  |  | =true
7 |  | else | =false

In the above code, the first two lines are a long statement with A1 being the master cell. It selects the desired cities information as follows:

The 3~7 lines are a computational sub-statement with A3 being the master cell. It selects from CITIES the cities that meets the criterion that the abbreviations of states to which they belong begin with "N" and assigns the names of the states as values to their STATE field. Result of A3 is as follows:

This block of statement beginning with "==" is also called as a computational block, which is actually a computational statement. A long statement or a computational sub-statement written in a single line can be also regarded as a computational block.

The master cell of a code block is its starting cell. Now let’s look at what a code block cell is.

The format of a code block should meet the following requirement: In the lines following the line where the master cell resides, the cells in the master cell column and those in the column to the left should be remain blank until the line where a cell in the master cell column or any cell to its left is non-blank. This means that, from this line on, cells no longer belong to the code block.

 | A | B | C | D
1 |  |  |  |
2 |  | >Code begin |  |
3 |  |  |  |
4 |  |  |  |
5 |  |  |  |
6 | Not null | Not null |  |

In the above cellset, suppose B2 is the code block’s master cell. If the cells in the green area are blank, then we call the lines from the second to the fifth as the master cell’s code block. As neither A6 nor B6 in the red area is blank, regardless of the code cell, computational cell, executable cell, constant cell or comment cell, the code block ends from the sixth line.

A code block not only holds a single statement, it can also put a statement in its master cell in order to execute a loop or a branch, or other instructions. For example:

 | A | B | C
1 | =demo.query("select NAME as CITY, STATEID as STATE from CITIES") | [] |
2 | for A1 | =demo.query("select * from STATES where STATEID=?",A2.STATE) |
3 |  | if left(B2.ABBR,1)=="N" | >A2.STATE=B2.NAME
4 |  |  | >B1=B1|A2

The above code uses code blocks instead of long statements or computational sub-statements. The code block with A2 being its master cell executes a loop and the one with B3 being the master cell executes a judgment instruction. B1 also gets the cities meeting the criterion that the abbreviations of states to which they belong begin with "N".

However, the code blocks under discussion don’t include the case that statements are used to execute various instructions, but those cases where a statement is written in multiple cells according to sequential cells rule and thus should not be executed separately. The rule allows the code written in a computation cell or an execution cell to automatically move on to the next cell if it ends with characters, including ",", ";" and "(", until it doesn’t ends with these characters or reaches the end of the row (or the code block).

Next
Previous
Related :
  • 1. Code Block


1.2 Assignment block and executable block

The double equal sign == at the head of the computational block has the similar function to that in a computational cell. If the expression in the code block doesn’t return a result, the code block will begin with double greater-than symbol >> and will still follow the sequential cells rule, meaning that the code block with the master cell is a single statement. For example:

 | A | B | C
1 | ==demo.query("select NAME as CITY, | POPULATION from CITIES") |
2 | >>C1=A1.select( | left(CITY,1):"C", |
3 |  | POPULATION>500000:true) |

The long statement in A2 doesn’t return a result. It is used to select the data of cities whose names begin with "C" and whose population is greater than 500,000, and assign the result to C1. The computed result of C1 is as follows:

This code block for value assignment is called as an assignment block.

Some other long statements are used to execute certain instructions. For example:

 | A | B
1 | =connect@e("demo") | =A1.query("select * from EMPLOYEE")
2 | >A1.execute("drop table EMPLOYEE1") |
3 | >>A1.execute("create table EMPLOYEE1( | EID int,FULLNAME varchar(30),
4 |  | GENDER varchar(10))")
5 | >>A1.execute(B1, | "insert into EMPLOYEE1(EID,
6 |  | FULLNAME,GENDER) values(?,?,?)",
7 |  | EID,NAME+" "+SURNAME, GENDER)
8 | =A1.query("select * from EMPLOYEE1") | >A1.close()

In this cellset, the code block in A3 is used to create a table – EMPLOYEE1- in the database. The code block in A5 is used to insert records into EMPLOYEE1 according to the data of B1. We can see the result of inserting records in A8 as follows:

The code blocks in A3 and A5 are called as execution blocks. The execution block and assignment block are defined according to their functions.

Next
Previous
Related :
  • 1. Code Block


1.3 Comment block

Similarly, if the string in a cell starts with //, the code block with this cell being the master cell holds comments and it is thus called as a comment block. esProc will skip the whole comment block when encountering it. For example:

 | A | B
1 | //comment | 1.note...
2 |  | 2.note...
3 |  | 3.note...
4 |  | =1+1
5 | =1+1 |

In the code block whose master cell is A1, all cells are naturally regarded as comment cells, without the need of being marked with a slash / in the front like the way a comment cell is handled. B4 is handled as a comment cell likewise. The non-blankness of A5 means the end of the code block, and the expression in A5 will be parsed and computed normally.

Different from a computational block, an assignment block and an executable block, there are no expressions in a comment block to be computed, so it doesn’t need the sequential cells rule.

Next
Previous
Related :
  • 1. Code Block


2. Long Statement

esProc has some special ways to produce more orderly and more readable code, among which long statements and computational sub-statements will be illustrated in this article.

Next
Previous
Related :
  • Chapter 5 Advanced Code
  • 2.1 Long statements
  • 2.2 Computational sub-statements


2.1 Long statements

If a long expression cannot be fully displayed when written in a single cell, one can write it in multiple cells according to sequential cells rule, which, in a computational cell or an execution cell, allows the code to automatically move on to the next cell if it ends with characters including ",", ";" and "(" until it doesn’t ends with these characters or reaches the end of the row (or the code block). For example:

 | A | B | C
1 | ==["one","two", | "three","four", | "five"]
2 | ==create(Field1, | Field2, | Field3)
3 | ==demo.query( | "select NAME, | STATEID from CITIES")
4 | ==A3.groups( | STATEID; | count(~):Count)
5 | >>C5=A4.( | Count) |

In the example, each row is a long statement. A long statement begins with double equals sign "==" or double greater-than symbol ">>" and tells esProc to write the expression according to sequential cells rule. The computed result of an expression beginning with double equals sign will be stored in the first cell. Results of A1, A2, A3 and A4 are as follows:

The long statement in A5 is used to assign value to C5, which is shown as follows:

As can be seen from the example, the long statements can use various functions. The only thing one should note is that, except the last one, all cells should end with ",", ";" and "(".

A long statement can show the structure of an expression clearer when one uses functions, particularly if and case, to write code. For example:

 | A | B | C | D
1 | Position | F |  |
2 | ==case(B1, | "C":"Center", | "F":"Forward", | "G":"Guard")
3 | Month | 7 |  |
4 | ==if(B3<=3:"Q1", | B3<=6:"Q2", | B3<=9:"Q3", | "Q4")

A2 changes the abbreviations for different positions of the players in a competition to the corresponding words. A4 computes which quarter the month in B3 belongs to. Results of A2 and A4 are respectively as follows:

Besides being in a single row, a long statement can be also used in a code block. In this case, the first cell of the long statement is the master cell, and all the cells in the code block are extended cells. For example:

 | A | B | C
1 | Air Quality Index | [59,164,50,104,93] |
2 | =B1.(func(A3,~)) |  |
3 | func | ==if( | A3>300:"Hazardous",
4 |  |  | A3>200:"Very Unhealthy",
5 |  |  | A3>150:"Unhealthy",
6 |  |  | A3>100:"Unhealthy for Sensitive Groups",
7 |  |  | A3>50:"Moderate" ,
8 |  |  | "Good")
9 |  | return B3 |

In this example, the subroutine in A3 uses a long statement to return the corresponding levels according to the air quality indexes (AQI). B3 is the master cell of the cell block in which the long statement resides and C3~C8 includes all the extended cells. A2 calls the subroutine to compute the quality grades corresponding to the AQI data. The result of A2 is as follows:

esProc will handle the code block that takes the first cell of the long statement as its master cell as a single statement; otherwise the program will execute each cell in the code block and may produces an error.

Next
Previous
Related :
  • 2. Long Statement


2.2 Computational sub-statements

A subroutine was used in the above to compute the grades for air quality corresponding to the AQI data. Its function is to compute the grade for air quality corresponding to each piece of AQI data. One can also write the expression in A2 directly as =B1.(if(~>300:"Hazardous",~>200:"Very Unhealthy",~>150:"Unhealthy",~>100:"Unhealthy for Sensitive Groups",~>50:"Moderate","Good")) rather than use the subroutine, but the code seems complicated and overloaded and not easy to understand. Besides the subroutine, a computational sub-statement can be used in esProc to handle the computation:

 | A | B | C
1 | Air Quality Index | [59,164,50,104,93] |
2 | ==B1.(??) | =if( | A3>300:"Hazardous",
3 |  |  | A3>200:"Very Unhealthy",
4 |  |  | A3>150:"Unhealthy",
5 |  |  | A3>100:"Unhealthy for Sensitive Groups",
6 |  |  | A3>50:"Moderate",
7 |  |  | "Good")

A2 gets the same result as the code in the above does. One can notice that the coding form of the computational sub-statement is similar to that of a subroutine. The former’s master cell begins with double equals sign "==", which means it is not necessary to add the double equal sign or double greater-than symbol before a long statement used in the code block that uses the same master cell. The double question marks "??" is used in the master cell of the computational sub-statement, and the value of the computational cell finally executed in the code block will be returned automatically and no return statement is needed. In addition, a subroutine can be called from anywhere in a cellset, but a computational sub-statement can be called only in the master cell.

The computational sub-statement in the above only consists of a single long statement which uses the if() function. But the code block of a computational sub-statement can also hold multiple statements. For example:

 | A | B | C
1 | Air Quality Index | [59,164,50,104,93] |
2 | ==B1.(??) | if ~>300 | ="Hazardous"
3 |  | else if ~>200 | ="Very Unhealthy"
4 |  | else if ~>150 | ="Unhealthy"
5 |  | else if ~>100 | ="Unhealthy for Sensitive Groups"
6 |  | else if ~>50 | ="Moderate"
7 |  | else | ="Good"

In this case, the code block of a computational sub-statement includes more than one computational cell. Each computation will return the value of the computational cell finally executed. Take 93 as an example, ="Moderate" in the computational cell C5 is executed finally in the code block, so the returned result is Moderate. Note that expressions, instead of constants, should be entered into the cells in Column C. A2 gets the same result as the A2 in the previous cellset:

Functioning as a special long statement, a computational sub-statement is to compute the expression in the master cell.

One can handle some complicated computations using the computational sub-statements. For example, based on the state data table – STATES – and big cities data table – CITIES, find the states where the total population of both the two most populated cities is greater than 2,000,000. The solution is as follows:

 | A | B | C
1 | =demo.query("select STATEID,ABBR, NAME from STATES order by STATEID") |  |
2 | ==A1.select(??) | =demo.query("select * from CITIES where STATEID=?",STATEID) |
3 |  | if B2.len()<1 | =false
4 |  | else if B2.len()==1 | =B2(1).POPULATION>2000000
5 |  | else | =B2.sort(POPULATION:-1)
6 |  |  | =C5(1).POPULATION+ C5(2).POPULATION>2000000

First A1 selects all the data from STATES. A2 uses the computational sub-statements to judge if each state meets the condition according to the results in the code block. In the computing process, B2 selects the big cities in a state and makes further judgment according to the result. If no big cities are found, use =false in C3 to return the result, meaning that no data meet the condition; if only one city is found, judge in C4 if this city’s population is greater than 2,000,000; if two or more cities are found, sort them in C5 by population in descending order and judge in C6 if they meet the condition. Therefore, by using the computational sub-statements, the structure of the complicated query becomes clearer. Result of A2 is as follows:

Next
Previous
Related :
  • 2. Long Statement


3. Creative Usage of Strings

In esProc, strings are not merely a data type; they can also be used as macros or string constants to construct expressions, or employed by eval() function to generate expressions dynamically as needed.

Next
Previous
Related :
  • Chapter 5 Advanced Code
  • 3.1 Macro
  • 3.2 eval function
  • 3.3 String constants


3.1 Macro

Macros in esProc refer to strings for composing expressions, only that these strings are usually generated dynamically. Macros are values of parameters or computed results of expressions. In practical use, a macro should be enclosed by signs ${}. esProc will first compute the expression enclosed by ${…} and then replace ${} with computed results which act as macro string values. For example:

 | A | B
1 | [+,-,*,/] | =A1(rand(4)+1)
2 | =rand(1000) | =rand(1000)
3 | =A2${B1}B2 |

B1 gets an arithmetic operator randomly from the operators list in A1; A2 and B2 generate randomly two integers less than 1,000, then A3 gets the arithmetic result.

Results of B1, A2 and B2 are as follows:

So, what A3 gets is the result of computing 79*994:

In this case, ${B1} is the macro used to construct the expression dynamically according to different signs.

Please note that if a macro is used in a string enclosed by double quotation marks, the replacement will not be executed. For example:

 | A | B
1 | A"+"A | "
2 | ="aabb${A1}ccdd" |
3 | =${B1}aabb${A1}ccdd" |

${A1} in A2 hasn’t been replaced, and the result in A2 is still the string in quotation marks:

In A3, the left double quotation marks are changed to another macro, and then the two macros are outside the quotation marks. Now the macros are validated and can be used to construct the expression ="aabbA"+"Accdd". Computed result in A3 is as follows:

It can be seen from the above example that, different from string parameters, macros are part of the expressions, and usually play a role in dynamically generating expressions.

Often, Macros are used to call dynamically different cell values, as well as various functions according to names. They can also employ dynamically various parameters and do some other jobs. Macros are really a convenient choice because these jobs are difficult to be done by using fixed expressions. For example:

 | A | B
1 | 4779736 | AL
2 | 710231 | AK
3 | 6392017 | AZ
4 | 2915918 | AR
5 | 37253956 | CA
6 | =[A1:A5].pselect(~>5000000) |
7 | =B${string(A6)} |

In A6, find the row in which the first record satisfying the condition of POPULATION >5,000,000 is located; with the expression in A7, get the name of the corresponding cell in column B according to the row number, so as to get the abbreviation of the state. Please note that the expression in the macro must return a string, therefore, string() function is needed in A7 to convert the row number in A6 into a string when the macro is used to generate the cell name. Results of A6 and A7 are as follows:

Also please note that if a macro is used in a cell, it will be parsed only in its first use. Therefore macros are not suitable for loop. For example:

 | A | B
1 | 4779736 | AL
2 | 710231 | AK
3 | 6392017 | AZ
4 | 2915918 | AR
5 | 37253956 | CA
6 | =[A1:A5].pselect@a(~>5000000) |
7 | =A6.(B${string(A6.~)}) |

Here A6 returns rows in which all records whose population is greater than 5,000,000 are located:

A7 tries every time to get the abbreviation of corresponding state according to the sequence loop in A6. Results are as follows:

As strings in the macro are only converted when the expression is first parsed, all abbreviations of the states after loop are AZ, which is the result when the macro is first replaced.

It should be clear that macros are used for generating expressions, not for generating command statements, like for, func, etc.

Next
Previous
Related :
  • 3. Creative Usage of Strings


3.2 eval function

eval(x, …) function in esProc can convert strings computed by x into an expression and compute its value. Its usage is similar to that of a macro, for both of them can generate an expression. Their difference lies that a macro can only convert a partial expression while eval function can generate a new expression during computing and return the computed result, and use parameters in the expression.

Since eval function needs to generate an expression in computing, the computed result of expression x must be strings. For example:

 | A | B
1 | [+,-,*,/] | =A1(rand(4)+1)
2 | =rand(1000) | =rand(1000)
3 | =eval("A2"+B1+"B2") |

Still, an arithmetic expression is generated randomly and computed result can be seen in A3.

Results of B1, A2 and B2 are as follows:

In A3, it can be seen that the use of eval() function is different from that of the macro in the above. At this point, the result of A3 is that of computing 893+466:

eval function is more powerful than the macro. It can achieve functions similar to those of a self-defining function when it uses parameters in the generated expression. For example:

 | A
1 | =eval("?.count(~>5)",[2,3,4,5,6,7,8])
2 | =eval("if (?1>0 && ?2>0 && ?3>0 && ?1+?2>?3 && ?1+?3>?2 && ?2+?3>?1)",3,2,5)

A1 computes the number of members that are greater than 5 in a sequence by using eval function. The question mark ? in the expression generated by eval represents a parameter. A2 judges whether the three parameters qualify as the lengths of the three sides of a triangle. ?1, ?2 and ?3 respectively represent the first, the second and the third parameter. Results of A1 and A2 are as follows:

In using eval function, the generated expression will be parsed each time it is computed, so it can be used by loop. For example:

 | A | B
1 | 4779736 | AL
2 | 710231 | AK
3 | 6392017 | AZ
4 | 2915918 | AR
5 | 37253956 | CA
6 | =[A1:A5].pselect@a(~>5000000) |
7 | =A6.(eval("B"+string(A6.~))) |

Now the results in A7 are what we want:

Next
Previous
Related :
  • 3. Creative Usage of Strings


3.3 String constants

The string constants being discussed are neither the strings simply enclosed by double quotation marks, nor the string-typed constants, they are the strings represented by $[…].

$[…] and "…" have similar usages. Both of them refer to strings represented by … in expressions. For example:

 | A
1 | ="abc"+$[ABC]
2 | ="\"[abc]\""+$["[ABC\]"]

Computed results in A1and A2 are as follows:

It can be seen that the usage of $[…] is basically the same as that of "…". Strings enclosed by both of them use a backslash \ as the escape character to mark ambiguous characters, like the quotation mark " in a common string and right bracket ] in a string constant.

In fact, the string constant $[…] is often used by macros and eval function. In the esProc’s IDE, the cell name in the expression in $[…] will change along with different editing operations, and the expression can be adjusted during operations like copy, paste, etc. Therefore, when a string constant returns a cell name as needed, these characters of it can prevent expression errors if there is any change in cellset structure. For example:

 | A | B
1 | select * from STATES | =demo.query(A1)
2 | POPULATION>10000000 | =demo.query($[${A1} where ${A2}])
3 | left(ABBR,1)='C' | =demo.query($[${A1} where ${A2} and ${A3}])

Both B2 and B3 generate query function’s query strings with cell values by using nested macros in string constants. B2 queries states whose population is greater than 10,000,000:

B3 queries states whose population is greater than 10,000,000 and whose abbreviations begin with letter C:

Now if an empty row is inserted before the second row, the cellset will be like this:

 | A | B
1 | select * from STATES | =demo.query(A1)
2 |  |
3 | POPULATION>10000000 | =demo.query($[${A1} where ${A3])
4 | left(ABBR,1)='C' | =demo.query($[${A1} where ${A3} and ${A4}])

It can be seen that the cell names in expressions in both B3 and B4 change accordingly. As a string represented by "…" cannot use macros, and the content of "…" will not change along with the editing operation, it is not applicable in certain cases.

Next
Previous
Related :
  • 3. Creative Usage of Strings


4. Clear Cell Values

In esProc, the storage of cellset variables is ubiquitous. Cell values are convenient references during computation, but they could occupy too much memory. Data can be cleared from cells to reduce memory footprint after they accomplish their mission in computation. It should be particularly noted that, when the intermediate data are obtained and further complicated computations are needed, useless cell values should be deleted to reduce memory usage in order to effectively avoid memory overflow.

Next
Previous
Related :
  • Chapter 5 Advanced Code
  • 4.1 Set cell values as null and reset a table sequence
  • 4.2 Clear cell values with clear statement


4.1 Set cell values as null and reset a table sequence

In the following task, you are asked to list top 200 transaction records of all household appliances orders and food orders according to total order amount, and sort them by product names. Order records come from two text files: Order_Appliances.txt and Order_Foods.txt. First summarize the data in the two sheets, get top 200 order records by total order amount, and then sort them by product names.

 | A
1 | =file("Order_Appliances.txt").import@t()
2 | =file("Order_Foods.txt").import@t()
3 | =A1|A2
4 | =A3.top(-Amount;200).new(ID, PName,Type,Date,Amount)
5 | =A4.sort(PName)

Computed results of all cells are as follows:

The table sequence in A1 contains order records of Order_Appliances.txt:

The table sequence in A2 contains order records of Order_Foods.txt:

A3 concatenates records of the two table sequences for the use of filtering in the next step.

A4 first selects top 200 order records according to total sales amount and then selects from them the desired fields to generate a new table sequence. As the data need to be sorted by sales amount in descending order, a negative sign is added before the Amount in top() function and thus you get the results sorted by sales amount:

A5 sorts the top 200 order records in sales amount by product names as required:

In fact, what we really need in the final result is the data in A5. After A4 gets all the necessary information of order records, the information in original tables in A1 and A2 becomes useless. Deleting these useless data after getting intermediate data can release memory and make the operation steadier.

Therefore, the cellset program can be modified according to the following method:

 | A | B | C
1 | =file("Order_Appliances.txt").import@t() |  |
2 | =file("Order_Foods.txt").import@t() |  |
3 | =A1|A2 | >A1.reset() | >A2=null
4 | =A3.top(-Amount;200).new(ID, PName,Type,Date,Amount) | >A3=null |
5 | =A4.sort(PName) | >A4=null |

If the cell value is set as null, the data in the cell would be deleted, as shown by statements in C3 and B4. After statement in C3 clears the cell value from A2 and B4 clears the references of records of food orders from A3, the original table sequences of food orders will be removed from memory.

While T.reset() function in B3, which is a little different, will delete all records from the table sequence but will retain its data structure. After B3 is executed, value of A1 is as follows:

You can choose the method for clearing cell values as needed. Setting cell values as null is more commonly used. T.reset() is used only when the table sequence’s data structure is worth retaining.

Note that though the statement in B5 sets the values of A4 as null, it cannot reduce memory footprint. Because the result A5 returns is a record sequence in which the record references come from the table sequence in A4, these records cannot be deleted and will be still referenced in A5 even if A4 is set as null. Therefore, when the method of setting cell values as null is to be used, you must find out whether data in the original cells are being in use or not.

In addition, A5 sorts records in A4, but the execution won’t produce new records. What A5 stores is merely the references resulted from sorting records, which have a limited memory footprint and won’t increase memory usage.

Next
Previous
Related :
  • 4. Clear Cell Values


4.2 Clear cell values with clear statement

After a piece of code is executed, probably there is a batch of cells whose values need to be cleared and only the final result is deserved to be retained, like the following cellset shows:

 | A | B
1 | =file("Order_Appliances.txt").import@t() |
2 | =file("Order_Foods.txt").import@t() | =A1|A2
3 | =B2.top(-Amount;200).new(ID, PName,Type,Date,Amount) |
4 | =A3.sort(PName) |
5 | >A1=null | >A2=null
6 | >B2=null | >A3=null

Similar to the previous example, A4 gets the top 200 records of orders according to the total amounts and sorts them by product names. In order to reduce memory usage after computing, A5, B5, A6 and B6 clear values from the useless cells. Obviously the code for accomplishing this is a little tedious. In esProc, clear command is specially provided to do this job:

 | A | B
1 | =file("Order_Appliances.txt").import@t() |
2 | =file("Order_Foods.txt").import@t() | =A1|A2
3 | =B2.top(-Amount;200).new(ID, PName,Type,Date,Amount) |
4 | =A3.sort(PName) |
5 | clear A1,A2,B2,A3 |

The effect of executing this cellset is the same as the above. The clear command in A5 can remove values from multiple cells at a time. In addition, the clear command can delete cell values by specifying an area of cells. For example:

 | A | B
1 | =file("Order_Appliances.txt").import@t() |
2 | =file("Order_Foods.txt").import@t() | =A1|A2
3 | =B2.top(-Amount;200).new(ID, PName,Type,Date,Amount) |
4 | =A3.sort(PName) |
5 | clear A1:B3 |

In the clear command in A5, A1:B3 represents a rectangular area from A1 to B3, including A1, B1, A2, B2, A3 and B3. The clear command will clear all these cells of values. You are allowed to write the command using single cells and the an area of cells together, for example, you can write the clear command in A5 as clear A1:B2,A3, and get the same result.

Another cellset:

 | A | B
1 | [Order_Appliances.txt,Order_Foods.txt] | =[0]*A1.len()
2 | for A1 | =file(A2).import@t()
3 |  | =B2.top(-Amount;100)
4 |  | =B3.sort(PName)
5 |  | >B1(#A2)=B4
6 | =B1.merge(PName) |
7 | =A6.new(ID, PName,Type,Date,Amount) |

The operation is a little different. It imports data from the two files according to the file names in A1, selects the top 100 records from each file according to order amounts, sorts them respectively by product names and stores the results in B1 respectively as a member of the sequence. As the results have been sorted, they will be merged in A6 by product names. The final result of table sequence can be seen in A7:

In this piece of code, the useless cell values also need to be deleted after A7 computes the final result. Besides by specifying cells or a cell area, you can also clear cell values by specifying a master cell, as shown in the following cellset:

 | A | B
1 | [Order_Appliances.txt,Order_Foods.txt] | =[0]*A1.len()
2 | for A1 | =file(A2).import@t()
3 |  | =B2.top(-Amount;100)
4 |  | =B3.sort(PName)
5 |  | >B1(#A2)=B4
6 | =B1.merge(PName) |
7 | =A6.new(ID, PName,Type,Date,Amount) | clear A1:B1,A2:,A6

In B7’s clear command, A2:’s writing style is similar to a cell area’s, but it doesn’t specify the end cell. This means clearing values from all the cells in a code block whose master cell is A2; that is, deleting values from the cells from the second to fifth line. After the code is executed, all the cell values are cleared except the result of A7.

Next
Previous
Related :
  • 4. Clear Cell Values


5. Special Identifiers

The strings in esProc sometimes include some special identifiers that may cause ambiguity about the expression. So does the names of fields in a table sequence sometimes. The following will teach you how to deal with them properly.

Next
Previous
Related :
  • Chapter 5 Advanced Code
  • 5.1 Special identifiers in strings
  • 5.2 Special identifiers in field names


5.1 Special identifiers in strings

A string may include various signs, some of which could bring trouble to the use of the string. First let’s look at problems you may encounter when using constant strings. For example:

 | A | B | C
1 | =,>,<,+,- | ===== dividing line ===== | >100
2 | / is division sign | 34.55 | end

Strings in A1 and B1 begin with "=" and will be interpreted as wrong expressions instead of constants if they are entered directly in cells. This can cause errors in computation. The string in C1 begins with ">" and won’t be handled as what it is; instead the cell will be regarded as an executable cell. The string beginning with "/" in A2 will be handled as a comment statement when written directly in the cell. The constant in B2 will be parsed into an integer automatically instead of a string. Being a reserved word, the string in C2 also won’t be handled as a string constant. In these cells, only cell B2 has a value; but the value is not of string type.

The above problems that string constants cannot be identified correctly are caused by two factors. In most cases the string itself is ambiguous, in other cases the string’s first character is a special identifier, such as "=", ">", "/" and etc. In esProc, you just need to add a single quotation mark (’) before the string constant to make it recognized normally.

 | A | B | C
1 | '=,>,<,+,- | '===== dividing line ===== | '>100
2 | '/ is division sign | '34.55 | 'end

Values of A1, B1 and C1 are as follows. The single quotation mark at the front serves only as the mark of string constant cell, it won’t appear in the value:

Values of A2, B2 and C2 are as follows. Notice that the data in B2 are left-aligned and of string type:

A string used in an expression needs to be surrounded by double quotation marks (like "…"), thus the above problematic strings can be identified correctly when used in expressions. Besides, for a string including " in it, put an escape character (\) before the double quotation marks of the string; the escape character is also needed before the back slash (\) of the string. For example:

 | A | B | C
1 | ="===== dividing line =====" | ="/ is division sign" | ="34.55"
2 | ="\"How is Frank?\" he asked." | =$[$\{B1}"C:\\test"] | =$[${B1}"C:\\test"]

In the cellset, A1, B1 and C1 show the proper method of handling the special strings mentioned in the above. The use of double quotation marks in an expression is to indicate the type of the string and thus to remove the ambiguity. Values of cell A1, B1 and C1 are as follows:

The string in A2 includes double quotation marks, so put an escape character before each double quotation mark. Do the same thing before \ if a string has one. Value of cell A2 is as follows:

B2 uses the string constant $[…], instead of the double quotation marks, to represent a string. In this case, the double quotation marks won’t cause ambiguity and escape characters are not necessary; but an escape character is still needed before \. Note that there is another problem about the string constants: As a macro in the format of ${…} can be used in a string constant, an escape character should be added to distinguish the string from a macro when the latter is not wanted. You can see this more clearly by comparing values of cell B2 and cell C2:

It can be seen that when the expression in C3 is computed, ${B1} is used as a macro and B1’s value is entered in the string.

Next
Previous
Related :
  • 5. Special Identifiers


5.2 Special identifiers in field names

In a table sequence, a field name including non-letter and non-number special identifiers, like %, (,), " or a blank space, could cause ambiguity when the expression is parsed. For example:

 | A
1 | $(demo) select * from STATES order by STATEID
2 | =A1.new(NAME:STATE,POPULATION,AREA:AREA(sq. mi.))
3 | =A2.(round(POPULATION/AREA(sq. mi.),3))

The following is the table sequence generated in A2, in which the AREA field is further explained by adding units to it:

When A3 computes the population density using the data in A2, something goes wrong when parsing the field name in the expression. To prevent this from happening, single quotation marks can be used to surround the name of the field in the table sequence (like '…') so as to distinguish the identifier from the specific area of signs. For example:

 | A
1 | $(demo) select * from STATES order by STATEID
2 | =A1.new(NAME:STATE,POPULATION,AREA:AREA(sq. mi.))
3 | =A2.(round(POPULATION/'AREA(sq. mi.)',3))

By doing so, esProc will handle the characters surrounded by the single quotation marks (as with 'AREA(sq. mi.)') as a whole and A3 can get the desired result:

Besides the names of fields in a table sequence, the special identifiers represented in the form of '…' can also be used when parameters with special names are called.

Of course the fields in a table sequence can be referenced by their positions to avoid the trouble brought by directly using the field names. For example:

 | A
1 | $(demo) select * from STATES order by STATEID
2 | =A1.new(NAME:STATE,POPULATION,AREA:AREA(sq. mi.))
3 | =A2.derive(round(POPULATION/#3,3):Density)

A3 adds the column of Density to the table sequence in A2. The corresponding code uses #3 to reference A2’s second field so that the problem caused by special field name is solved. Result of A3 is as follows:

When a field in a table sequence is called with special identifiers represented by '…', the field name is not allowed to contain single quotation marks; otherwise it can only be called according to the position as with the example above.

Next
Previous
Related :
  • 5. Special Identifiers


Chapter 6 Integration&Application

Next
Previous
Related :
  • 1. Command Line
  • 2. Java Invocation
  • 3. Deploying JDBC
  • 4. License and Encryption
  • 5. Integration by BIRT
  • 6. Integration with JasperReport
  • 7. User-defined Functions


1. Command Line

The command line refers to using commands in the form of strings in DOS-like environment to execute programs. All operating systems support the command lines, like the Linux console and the Windows command line window.

The file, esprocx.exe, is needed when the command line is used to execute an esProc cellset file. The file can be found in esProc\bin path in esProc’s installation directory.

Next
Previous
Related :
  • Chapter 6 Integration&Application
  • 1.1 Cellset files without return results
  • 1.2 Cellset files with return results


1.1 Cellset files without return results

Use the esprocx command to execute the cellset file at command line. Take the following cellset file, D:\files\output1.dfx, as an example:

 | A | B
1 | =create(ID,Amount) |
2 | for 100 | >A1.insert(0,#A2,rand(100*100))
3 | =file("D:\\files\\Data1.txt") | >A3.export@t(A1)

Cellset files without return results are usually used to access files or databases. In the above cellset, for example, some simulation data are created and stored in a text file, Data1.txt. In using an esprocx command, enter the to-be-executed dfx file after it; during execution, prompts of start and end will be displayed on the screen, as shown below:

After the execution, the randomly generated data will be written into Data1.txt, as shown below:

If the path in which the esprocx file is located is included into the operating system’s path list, run the file directly instead of running it in esProc\bin path of the installation directory. But the to-be-called dfx file must be entered in its full path, or should be located in the current path. For example:

If the running cellset file uses parameters, enter their values, separated by spaces, after the dfx file when executing esprocx. Take the following cellset file D:\files\output2.dfx as an example:

 | A | B
1 | =file("D:\\files\\Data1.txt") | =A1.import@t()
2 | for IDs | >B1.modify(A2,Value:Amount)
3 | >A1.export@t(B1) |

The file uses two parameters: IDs and Value, as shown below:

The cellset is to import the Data1.txt file just created, modify some of the records with specified sequence numbers so as to make their Amount fields are equal to the specified Value and store them in the file again. The prompts displayed on the screen during execution are as follows:

Two parameters, one is a sequence, [2,3,5,7,11] and the other is an integer, 99999, are input while the cellset file, output2.dfx, is called. Similar to a cross-cellset call, the input parameters will assign value to the cellset parameters according to their orders instead of their names during the command line call. In this case [2,3,5,7,11] assigns value to the first parameter, IDs and 99999 assigns value to the second parameter, Value. After the command line is executed, data in Data1.txt will have been modified:

Next
Previous
Related :
  • 1. Command Line


1.2 Cellset files with return results

If the esprocx command is used directly at the command line, the result won’t be displayed on the screen by default. If the return result of the dfx file is to be displayed, -R option needs to be used in execution. Similar to the cellset file called by the call function, the dfx file must return the result using result statement. Take the following cellset file, D:\files\calculate1.dfx, as an example:

 | A | B
1 | =days(Day) | =string(Day,"EEE")
2 | result A1 | result B1

This cellset will return the total number of days of the month in which the specified date occurs and compute what day the specified date is. It uses Day as one of its parameters:

Let’s first look at the execution with -R option:

Here no parameters have been set during the execution and a default parameter, 01/01/2000, is used for the computation. You can see that A2 and B2 respectively return a result using result statement. The results will be listed respectively after the computation is finished.

If a parameter is to be set, its data type should be date. As a parameter of date or time type should be in some kind of format, you need to set its format in the configuration file, config.xml, which is located in esProc\config path in esProc’s installation directory:

<dateFormat>MM/dd/yyyy</dateFormat>

<timeFormat>HH:mm:ss</timeFormat>

<dateTimeFormat>MM/dd/yyyy HH:mm:ss</dateTimeFormat>

The execution is what the following shows if a parameter is set for the call:

At this point, you get the total number of days of the month in which February 22, 2012 occurs and what day this date is.

Different from the previous cellset file, the following one, D:\files\calculate2.dfx, returns the results as a sequence:

 | A | B
1 | =days(Day) | =string(Day,"EEE")
2 | result [A1,B1] |

The execution is as follows:

It can be seen that members of the resulting sequence are listed respectively, each having a row.

Here comes the last sample of the cellset file - D:\files\calculate3.dfx:

 | A | B
1 | $(demo) select * from CITIES | $select * from STATES
2 | =A1.switch(STATEID,B1) | =A2.new(CID,NAME:City,STATEID.NAME:State)
3 | =B2.select(left(City,1)==Letter) |
4 | result A3 |

The cellset file uses a parameter - Letter:

This cellset uses the database data. In this case the database connection needs to be configured in config.xml in the esProc\config path in the esProc installation directory:

<DBList>

<DB name="demo">

<property name="url" value="jdbc:hsqldb:hsql://127.0.0.1/demo"/>

<property name="driver" value="org.hsqldb.jdbcDriver"/>

<property name="type" value="HSQL"/>

<property name="user" value="sa"/>

<property name="password" value=""/>

<property name="batchSize" value="1000"/>

<property name="autoConnect" value="true"/>

<property name="useSchema" value="false"/>

<property name="addTilde" value="false"/>

</DB>

</DBList >

Please refer to related documents for detailed configuration method.

When the program is executed, the result is as follows:

To set a parameter of string type, it’s not necessary to surround it with double quotation marks. Cities whose names start with B are listed in the above result. For the result of the type of a table sequence or a record sequence, each record is a row and fields are separated from each other by tabs.

The above configuration file, config.xml, is also used by the Integration Development Environment (IDE), so information, like date format and datasource configuration, can be edited on the IDE interface. Click Tool>Option on the menu bar and configure the information on the Environment of Option window:

Click Tool>Datasource connection and configure information, like the database connection parameter, in the datasource manager. For detailed configuration method, please refer to esProc Accesses Databases: Database Configuration.

Next
Previous
Related :
  • 1. Command Line


2. Java Invocation

esProc can be embedded in a Java application. Invoking an esProc cellset program is like accessing a database via JDBC. During the invocation the cellset program is packaged as a stored procedure, so it is called in the same way as that the latter is called.

Next
Previous
Related :
  • Chapter 6 Integration&Application
  • 2.1 Basic usage
  • 2.2 Call different cellset files
  • 2.3 No result or multiple results
  • 2.4 Execute statements directly


2.1 Basic usage

esProc JDBC resembles a database JDBC driver without a physical table and can be simply regarded as a database with only the stored procedure. Besides, esProc JDBC is a fully embedded computing engine that all computations can be performed within it, whereas the JDBC provided by ordinary databases is only an interface and computations are performed in an independent database server.

First deploy the application as the article esProc Integration & Application: Deploying JDBC introduces.

Next
Previous
Related :
  • 2. Java Invocation
  • 2.1.1 Cellset files used by esProc JDBC
  • 2.1.2 Basic method of Java invocation


2.1.1 Cellset files used by esProc JDBC

Similar to the cross-cellset call with call function, the cellset code used in an esProc JDBC will return a result set through result statement, as shown in the following cellset file - createTable1.dfx：

 | A | B
1 | =create(ID,Amount) |
2 | for 100 | >A1.insert(0,#A2,rand(100*100))
3 | result A1 |

The computation in this cellset is simple: A table sequence holding 100 records and with an orderly-set ID field and a randomly-generated Amount field is created. In A3, the result statement returns A1’s table sequence.

This cellset file will be used in the following to explain how to make a call via esProc JDBC in Java.

Next
Previous
Related :
  • 2.1 Basic usage


2.1.2 Basic method of Java invocation

Before calling esProc to execute the cellset file, you need to configure related information according to the following instructions:

Load the necessary jars when launching a Java application (see esProc Integration & Application: Deploying JDBC for the introduction of jars) . With a WEB application, these jars can be put into WEB-INF/lib folder.

Deploy dfxConfig.xml and config.xml

config.xml contains esProc’s basic configuration information, like registration code, searching path, main path, datasource configuration and so on. The file is located in esProc’s [installation directory]\esProc\config folder, in which the information is the same as the settings on esProc’s Option page. Its configuration can be adjusted before the deployment.

During the deployment, config.xml is allowed to be renamed esprocJDBCconfig.xml in case there are already config.xml files for other purposes under classpath. esProc JDBC driver will find and use esprocJDBCconfig.xml first in preference to config.xml, which will be used when the former does not exist.

Configure in dfxConfig.xml the concurrency value of esProc, information about logs, connection pool, and etc. This file is located in esProc’s [installation directory]\esProc\classes folder.

Note: The configuration files should be copied and put into the application project’s classpath, their names must remain dfxConfig.xml and config.xml, or esprocJDBCconfig.xml , and must not be changed. For detailed explanation about the files, please refer to esProc Integration & Application: Deploying JDBC.

Deploy dfx file

The above createTable1.dfx can be put into either the application project’s claspath or the searching path specified by config.xml’s <paths/> node, or the main path specified by <mainPath/>.

Call dfx in Java

public void testDataServer(){

Connection con = null;

com.esproc.jdbc.InternalCStatement st;

try{

// establish a connection

Class.forName("com.esproc.jdbc.InternalDriver");

con= DriverManager.getConnection("jdbc:esproc:local://");

// call the stored procedure; createTable1 is the name of dfx file

st =(com. esproc.jdbc.InternalCStatement)con.prepareCall("call createTable1()");

// execute the stored procedure

st.execute();

// get the result set

ResultSet rs = st.getResultSet();

// process the result set simply by printing out its field names and data

ResultSetMetaData rsmd = rs.getMetaData();

int colCount = rsmd.getColumnCount();

for ( int c = 1; c <= colCount;c++) {

String title = rsmd.getColumnName(c);

if ( c > 1 ) {

System.out.print("\t");

}

else {

System.out.print("\n");

}

System.out.print(title);

}

while (rs.next()) {

for (int c = 1; c<= colCount; c++) {

if ( c > 1 ) {

System.out.print("\t");

}

else {

System.out.print("\n");

}

Object o = rs.getObject(c);

System.out.print(o.toString());

}

}

}

catch(Exception e){

System.out.println(e);

}

finally{

// close the connection

if (con!=null) {

try {

con.close();

}

catch(Exception e) {

System.out.println(e);

}

}

}

}

Use "call createTable1()" statement to run the cellset file - createTable1.dfx and return the result as the ResultSet object when calling an esProc file. In the subsequent program, simply print out the data of the result set. After the program is executed, the printed out result is as follows:

Next
Previous
Related :
  • 2.1 Basic usage


2.2 Call different cellset files

Basic method of Java invocation in 1.3 explained how to call a cellset file in a Java program. But what will you do with different cellset files? Take the following cellset file - createTable2.dfx – as an example:

 | A | B
1 | =connect("demo") |
2 | =A1.query("select * from EMPLOYEE") | =A2.select(month(BIRTHDAY)==month(Date)&&day(BIRTHDAY)==day(Date))
3 | >A1.close |
4 | if B2.len()>1 | result B2.new(EID,NAME+ " "+ SURNAME:NAME, GENDER, BIRTHDAY)
5 | else | result "None"

In this cellset file, data are got from demo, the datasource, and a date parameter – Date – is used:

The datasource demo used in the cellset needs to be configured in config.xml. In this respect, please refer to esProc Integration & Application: Deploying JDBC for details. The connection to the datasource established in the cellset should be closed using db.close() function after it accomplishes its mission.

For the call from Java, the code for establishing a connection and outputting the result is the same as the previous example. The difference lies in the method of calling the file:

// call the stored procedure; createTable2 is the name of dfx file and ? represents the to-be-set parameters

st =(com. esproc.jdbc.InternalCStatement)con.prepareCall("call createTable2(?)");

// set the parameters

java.util.Calendar calendar = java.util.Calendar.getInstance();

calendar.set(1980, 0, 1);

st.setObject(1, calendar.getTime());

// execute the stored procedure

st.execute();

// get the result set

ResultSet rs = st.getResultSet();

This cellset file uses a parameter, so, to call the cellset, use "call createTable2(?)", in which ? represents the parameter waiting to be set. In this case, st.setObject() needs to be used to set the parameters according to their order; this is irrelevant to the parameter names in the cellset. In this cellset file, employees whose birthdays are on the day presented in the parameter will be listed. Note that the month will start with 0 when the date is set for Calendar. The printed out result is as follows:

During parameter setting, the parameter objects can be directly set according to different parameter types; but the date parameter can be entered in the form of a string, which will be automatically parsed by esProc:

// set the parameter

st.setObject(1, "1/1/1980");

Note that the format of the string you entered should be in consistent with the date format set in config.xml. After the program is executed, the result is the same as that of the previous example.

In addition to the above calling method, the parameters can be fixed in the statement. For example:

// call the stored procedure; createTable2 is the name of dfx file and the parameter is fixed in the statement

st =(com. esproc.jdbc.InternalCStatement)con.prepareCall("call createTable2(\"2/29/1980\")");

// execute the stored procedure

st.execute();

// get the result set

ResultSet rs = st.getResultSet();

With this calling method, the parameter objects cannot be set according to different parameter types; the parameters can only be either the numerical type or the string type which will be parsed automatically by esProc. After the above code is executed, the result is as follows:

Since no employees were born on February 29, B5 returns a string – None – using result statement. As can be seen from this result, the column name will be automatically generated for the result set, making it a standard Result for returning, even if the data returned from the cellset is a single value.

Next
Previous
Related :
  • 2. Java Invocation


2.3 No result or multiple results

It is allowed to return no result during calling a cellset file. Take the following cellset file - outputData1.dfx - as an example:

 | A
1 | $(demo) select * from CITIES
2 | =A1.select(Arg.pos(left(NAME,1))>0)
3 | =file("Cities.txt")
4 | >A3.export@t(A2)

The file uses Arg, a parameter of string sequence type:

A1 directly calls the demo database to query data, instead of establishing a connection using connect statement. In this case, the property of demo datasource’s autoConnect in config.xml’s database configuration should be true:

<property name="autoConnect" value="true"/>

Please refer to esProc Integration & Application: Deploying JDBC for detailed configuration. To call the cellset in Java, the code should be as follows:

// call the stored procedure; outputData1 is the name of dfx file

st =(com. esproc.jdbc.InternalCStatement)con.prepareCall("call outputData1(?)");

// set the parameters

com.raqsoft.dm.Sequence seq = new com.raqsoft.dm.Sequence();

seq.add("A");

seq.add("B");

seq.add("C");

st.setObject(1, seq);

// execute the stored procedure

boolean hasResult = st.execute();

// cannot get the result set as there is no return result; hasResult is false

Note: The cellset file uses a parameter of sequence type, which can only be set by generating a Sequence object. As with this example, a sequence parameter [A,B,C] is set for writing information of cities whose initials are A, B and C to the file Cities.txt. If no path has been specified for storing the output file in cellset, store it in the main path, which is specified by config.xml’s <mainPath/> node. Data of the file are as follows:

It is probably that a cellset file returns multiple result sets, like the following cellset file - createTable3.dfx:

 | A | B
1 | $(demo) select EID, NAME+' '+SURNAME FULLNAME, GENDER, STATE, BIRTHDAY from EMPLOYEE |
2 | =A1.select(STATE == state && age(BIRTHDAY)<age) |
3 | =A2.select(GENDER=="F") | =A2\A3
4 | result A3 | result B3

This cellset returns the data of employees who are younger than the specified age and who, at the same time, come from the specified state as two table sequences according to genders respectively. The cellset uses two parameters, respectively specifying the employees’ oldest age and the state from which the employees come from:

This cellset file will return multiple result sets when executed.

// call the stored procedure; createTable3 is the name of dfx file and ? represents the parameters

st =(com. esproc.jdbc.InternalCStatement)con.prepareCall("call createTable3(?,?)");

// set the parameters in order

st.setObject(1, 30);

st.setObject(2, "California");

// execute the stored procedure

boolean hasResult = st.execute();

// get the multiple result sets and print them out

while (hasResult) {

ResultSet rs = st.getResultSet();

// print out the results

ResultSetMetaData rsmd = rs.getMetaData();

int colCount = rsmd.getColumnCount();

for ( int c = 1; c <= colCount;c++) {

String title = rsmd.getColumnName(c);

if ( c > 1 ) {

System.out.print("\t");

}

else {

System.out.print("\n");

}

System.out.print(title);

}

while (rs.next()) {

for (int c = 1; c<= colCount; c++) {

if ( c > 1 ) {

System.out.print("\t");

}

else {

System.out.print("\n");

}

Object o = rs.getObject(c);

System.out.print(o.toString());

}

}

System.out.println();

// check if there are other return results

hasResult = st.getMoreResults();

}

When the code is executed, data of the female and male employees who are younger than 30 and who come from state of California will be fetched respectively. The printed out result is as follows:

In the cellset file - createTable3.dfx, a single statement – result A3,B3 - returns two record sequences. The case in Java is the same as here.

Next
Previous
Related :
  • 2. Java Invocation


2.4 Execute statements directly

Besides calling a cellset file, a Java program also uses esProc to execute statements directly. Actually, serving as a call statement, the invocation can be directly executed. Let’s look at in detail how other statements can be executed. For example:

// execute the statement directly and return the result set

st = (com. esproc.jdbc.InternalCStatement)con.createStatement();

ResultSet rs1 = st.executeQuery("=age(date(\"1/1/1990\"))");

ResultSet rs2 = st.executeQuery("=5.(~*~)");

Here executeQuery is used to directly execute the esProc expression starting with = and return the result as a result set. Note that an escape character needs to precede the double quotation marks in the expression. Use the similar code to print out the result sets as follows:

Notice that if the return result is a sequence, JDBC will return a one-column-and-multiple-row result set. Or another method can be used to call the statement:

// first judge if there exists a result set; get it if there is one

boolean hasResult1 = st.execute("=age(date(\"1/1/1990\"))");

ResultSet rs1,rs2;

if (hasResult1) {

rs1 = st.getResultSet();

}

boolean hasResult2 = st.execute("=5.(~*~)");

if (hasResult2) {

rs2 = st.getResultSet();

}

Based on the statement being executed, the execute function will return a value to make clear whether there is a result set. The effect of the corresponding code is the same as the previous one.

If parameters are needed in the statement, use the operator - (x1, x2,…) to compute the expression successively and return the last result. For example:

st = (com. esproc.jdbc.InternalCStatement)con.createStatement();

ResultSet rs1 = st.executeQuery("=(pi=3.14,r=4,r*r*pi)");

The above code computes the area of a circle. The output result set is as follows:

The data set query statement starting with $ can also be used in esProc JDBC. For example:

// execute the data set query statement

st = (com. esproc.jdbc.InternalCStatement)con.createStatement();

ResultSet rs1 = st.executeQuery("$(demo) select * from CITIES where POPULATION > 2000000");

Note that demo, the data set used here, needs to be configured as automatically connected. The result is as follows:

Use the file name to call a cellset file so that call function can be omitted. For example:

st = (com. esproc.jdbc.InternalCStatement)con.createStatement();

boolean hasResult1 = st.execute("createTable3 30,\"California\"");

The cellset file - createTable3.dfx will be called during the execution. If parameters need to be set for the file, they should be fixed in the statement and separated from the file name with a space. The above statement also returns multiple result sets, which are the same as that got using the other method in the above:

Next
Previous
Related :
  • 2. Java Invocation


3. Deploying JDBC

Java programs can call esProc via JDBC. It is necessary for programmers to know how to deploy the esProc JDBC and familiarize themselves with the detailed configuration information.

Next
Previous
Related :
  • Chapter 6 Integration&Application
  • 3.1 Basic steps
  • 3.2 Configuring config.xml
  • 3.3 Configuring dfxConfig.xml


3.1 Basic steps

The following instructions need to be followed to deploy the esProc JDBC:

1) Load necessary jars

Load the jars needed by esProc when launching a Java application. These jars can be placed under the WEB-INF/lib directory in the Web application environment. There are three basic jars necessary for esProc JDBC, all of them can be found in [instllation]\esProc\lib directory:

dm.jar esProc computing engine and JDBC driver

icu4j_3_4_5.jar Process internationalization

dom4j-1.6.1.jar Parse the configuration files

Besides the basic jars, there are more jars specifically for some certain functions,they can all be found in the \esProc\lib directory:

To use other databases as the data sources in esProc JDBC, their driver jars are required; hsqldb.jar is specifically used for database demo;

To access Microsoft Office files in esProc JDBC, geronimo-stax-api_1.0_spec-1.0.jar, poi*.jar and xmlbeans-2.3.0.jar are required;

To use the graphic function in esProc JDBC, jars for SVG-typed image processing, including batik*.jar, js.jar, pdf-transcoder.jar, xalan-2.6.0.jar, xerces_2_5_0.jar, xml-apis.jar and xml-apis-ext.jar are required;

Note that JDK1.6 or any of the higher versions of esProc JDBC is required.

2) Deploy config.xml file and dfxConfig.xml file

You can find the two configuration files - config.xml and dfxConfig.xml –respectively in [installation directory]\esProc\lib and [installation directory]\esProc\classes. They need to be copied and placed under the classpath of the application project. During the deployment, config.xml is allowed to be renamed esprocJDBCconfig.xml in case there are already config.xml files for other purposes under classpath. esProc JDBC driver will find and use esprocJDBCconfig.xml first in preference to config.xml, which will be used when the former does not exist. Names of these configuration files’ names should be remain what they are specified here and must not be changed. Detailed information about the two files will be discussed in the second section – Configuring config.xml, and the third section – Configuring dfxConfig.xml.

3) Deploy dfx file

Place the necessary dfx file in either the application project’s classpath or the searching path specified by config.xml’s <paths/> node, or the main path specified by <mainPath/>.

4) Invoke esProc JDBC in Java

About the method of invoking a dfx file in Java and of executing the code directly, please refer to esProc Integration & Application: Java Invocation.

Next
Previous
Related :
  • 3. Deploying JDBC


3.2 Configuring config.xml

config.xml file contains configuration information, including license information, esProc’s main path, dfx file’s searching path, and etc. They are as follows:

<?xml version="1.0" encoding=" UTF-8"?>

< Config Version="1">

<!—esProc register code; leave it blank for a free version -->

< regCode>W2r@7-8cLMJ-GVU33-BhTIB3E</regCode>

<!-- esProc charset-->

<charSet>ISO-8859-1</charSet>

<!--

Configure dfx file’s searching path, which is an absolute path. Multiple paths, separated by "semicolon(s)", are allowed. The dfx file can also be placed in the application project’s classpath, which has the priority over the searching path for loading the file.

-->

<dfxPathList>

< dfxPath>D:\files\dfx;D:\files\txt;D:\files</dfxPath>

</dfxPathList>

<!-- Date, time and other data types -->

<dateFormat>MM/dd/yyyy</dateFormat>

<timeFormat>HH:mm:ss</timeFormat>

<dateTimeFormat>MM/dd/yyyy HH:mm:ss</dateTimeFormat>

<!-- The first method for datasource configuration: configure connection pool in the application server and specify the datasource name -->

<DBList>

<!-- Datasource name must be the same as that in the dfx file -->

<DB name="demo">

<property name="url" value="jdbc:hsqldb:hsql://127.0.0.1/demo"/>

<property name="driver" value="org.hsqldb.jdbcDriver"/>

<property name="type" value="HSQL"/>

<property name="user" value="sa"/>

<property name="password" value=""/>

<property name="batchSize" value="1000"/>

<!--

Automatically connect or not. If the value is true, use db.query() function or SQL statements starting with $ directly to access the database; if it is false, there is no automatic connection; you should first establish the connection with connect(db) statement before using the function or the statements.

-->

<property name="autoConnect" value="true"/>

<property name="useSchema" value="false"/>

<property name="addTilde" value="false"/>

</DB>

<DB name="hsql">

<property name="url" value="jdbc:hsqldb:hsql://localhost/demo"/>

<property name="driver" value="org.hsqldb.jdbcDriver"/>

<property name="type" value="HSQL"/>

<property name="user" value="sa"/>

<property name="password"/>

<property name="batchSize" value="0"/>

<property name="autoConnect" value="false"/>

<property name="useSchema" value="false"/>

<property name="addTilde" value="false"/>

</DB>

</DBList >

<!-- esProc main path, which is a single absolute path -->

<mainPath> D:\files</mainPath>

<!-- The save path for temporary files; it can be an absolute path. If using a relative path, put it under the main path -->

<tempPath>\temp</tempPath>

<bufSize>65536</bufSize>

<localHost>192.168.0.86</localHost>

<localPort>8282</localPort>

<logLevel>DEBUG</logLevel>

<callxParallel>4</callxParallel>

<nodeParallel>4</nodeParallel>

</Config>

The config.xml file is shared by esProc JDBC and the Integrated Development Environment (IDE), so the above information can be configured on the IDE interface.

First, to set the registration code, select Help>Registration from the menu bar on the IDE interface to enter or query the registration code. Leave the registration code blank for a free version esProc. After the registration code is entered, you can click on Scan button to see the license information. Then click Register to complete the registration procedure:

Click Tool>Options on the menu bar to edit main configuration information. The key information related to JDBC, like paths, format of date and time and so on, can be edited on the Environment page:

General page also provides JDBC-related settings, such as log level and number of parallel tasks:

Besides, datasource information of the config.xml file can be configured in the datasource manager. Click Tool>Datasource connection on the menu bar to open the datasource manager window:

You can add, delete or modify the datasource configuration information in the datasource manager. Click Edit to edit the connection parameter of the datasource:

In the datasource manager, you can set the information about the driver used by the datasource, datasource URL, user name, password, batch size, and etc. Note that demo is the system datasource, whose connection URL cannot be changed. In addition, esProc JDBC itself cannot be used as a datasource during the datasource connection configuration.

config.xml file will be loaded automatically in Java application’s classpath for use of esProc JDBC but cannot be renamed at this point. If esProc JDBC’s connecting string uses this format (...?config=...), the .xml file specified by config in the connecting string will be loaded for use in the configuration and the file defined in the default configuration will be ignored.

For example, based on a connecting string like con= DriverManager.getConnection("jdbc:esproc:local:/?config=myconfig.xml"), the configuration in myconfig.xml will be used.

Next
Previous
Related :
  • 3. Deploying JDBC


3.3 Configuring dfxConfig.xml

The dfxConfig.xml file has the configuration information about concurrency, log property, connection pool, and so on. Take the following file as an example:

<dfxConfig>

<!--

Maximum amount of concurrency. Set the allowed max. number of concurrent tasks; tasks performed in parallel within the same connection are handled as the concurrent ones. The number should be within the licensing limit.

-->

<maxConcurrents>10</maxConcurrents>

<!--

Maximum wait time. If the number of concurrent tasks exceeds the maximum amount of concurrency, the extra tasks will be in wait state. Exceptions will be thrown if the wait time exceeds its specified limit.

-->

< maxWaitTime>5000</maxWaitTime>

<!-- Files of log configuration property -->

< log>logger.properties</log>

<!-- Configure connection pool in the application server and specify the datasource name -->

<jndi-ds-configs>

<!—jndi is a prefix -->

<jndi-prefix>java:comp/env</jndi-prefix>

<!-- Datasource name must be the same as that in the dfx file -->

<jndi-ds-config>

<name>olap</name>

<dbType>ACCESS</dbType>

<dbCharset>ISO-8859-1</dbCharset>

<clientCharset>ISO-8859-1</clientCharset>

<needTranContent>false</needTranContent>

<needTranSentence>false</needTranSentence>

<!--

Automatically connect or not. If the value is true, use db.query() function directly to access the database; if it is false, there is no automatic connection; you should first establish the connection with connect(db) statement before using the function.

-->

<autoConnect>true</autoConnect>

</jndi-ds-config>

</jndi-ds-configs>

</dfxConfig>

If a datasource connection needs to be established for the dfx file invoked through the esProc JDBC, the related information should be edited in config.xml’s datasource configuration or dfxConfig.xml’s connection pool configuration. A point worth particular note is that, if db.query() function or $(db) sql accesses the database directly, the property of autoConnect of this database should be true. For the usage of esProc JDBC, please refer to esProc Integration & Application: Java Invocation.

Next
Previous
Related :
  • 3. Deploying JDBC


4. License and Encryption

Next
Previous
Related :
  • Chapter 6 Integration&Application
  • 4.1 File encryption
  • 4.2 License and registration code


4.1 File encryption

esProc provides ways of encrypting cellset files to protect them from unauthorized access or use. There are two levels of encryption:

Execution privilege：By this privilege, users can set parameters, execute the cellset and see the cell values; they can also view the cellset descriptions and cell tips but cannot modify them. With only the execution privilege, users can neither see strings in the cells nor any cellset variables.

Full control privilege：Users have full access to the cellset file and can take any action.

Take the following cellset as an example:

 | A | B
1 | Minimum Population | 500000
2 | $(demo) select * from CITIES | /Get data from table CITIES.
3 | $(demo) select STATEID, ABBR from STATES | /Get data from table STATES.
4 | =A3.select@1(ABBR==state).STATEID | /Find the STATEID of the given state.
5 | =A2.select(POPULATION>B1&& STATEID==A4) | /Find the cities.
6 | result A5 |

In the above cellset, the information in the comment cells of column B is the tips waiting to be set for cells of column A. Notice the difference between comment cells and cell tips. Parameter state is used in the cellset:

Click Tool>Cellset password on the menu bar to set the password on the pop-up window:

There are two parts for password settings: Set the password for full control privilege after Input password and set the privilege without a password after None password privilege.

To set the password, just enter it in the box, like setting abc as the password for the full access to the cellset in the above.

Select in the drop-down menu either “For execution” or “For full control” for setting the privilege without a password:

If For full control is selected, then users are granted the highest full control privilege without a password and the input password set before will be cleared, which means the cellset encryption has been removed.

Suppose set abc as the password for the full control privilege, select For execution as the None password privilege and then click OK and close the cellset, the system will open the cellset according to the None password privilege next time it is required to do so. For example:

It can be seen that, after the cellset is encrypted, the code is invisible, even the information in the comment cells won’t be displayed, with only the execution privilege, except that the cell tips still work. Moreover, users with the execution privilege cannot perform operations which can only be performed with full control privilege. So users can view the cellset descriptions by clicking Tool>Cellset description, but they cannot modify them:

The values of constant cells of the cellset can be viewed in Value View Section and the cellset parameter and its default value can be viewed in Cellset Variables List which is on the bottom right corner of the dialog box:

By execution privilege, users can click [image] on the menu bar to execute the cellset. If cellset parameters need to be set before the execution, the Input argument window will pop up for the setting.

After the parameter is set and the cellset is executed and the result is returned, the cells holding the code will change their color though the code still remains encrypted:

At this point, click one of the cells and its value will be displayed on the right. Following is A5’s value:

With execution privilege, click Tool>Unlock advanced privilege on the menu bar to unlock the cellset and get back the full control privilege with the password being entered correctly:

Users are allowed, as a full control privilege, to see or modify the code in the cellset, and to encrypt or unencrypt the cellset file, etc.

Next
Previous
Related :
  • 4. License and Encryption


4.2 License and registration code

Different esProc editions correspond to different registration codes and support different functions. About each edition’s feature and the way of getting a registration code, please visit the official website http://www.raqsoft.com/product-esproc.

With a registration code, users can conduct registration in the esProc according to following steps:

Launch esProc and click Help>Registration on the menu bar:

Enter the registration code and click Scan button to check its validity and the corresponding authorized functions. After confirming its validity, click Register to begin the registration procedure. By the way, leave the registration code blank if the esProc edition is free.

Click Check points to see the authorized functions provided by this edition:

Click More Editions to visit the official website for information of more editions.

When the registration is completed, information about the registration code will be saved automatically in esProc’s configuration file config.xml, which is placed in esProc’s [installation directory]\esProc\config directory by default. Some of the license information is as follows:

<?xml version="1.0" encoding=" UTF-8"?>

< Config Version="1">

<!-- esProc licensing code; leave it blank for a free version-->

< regCode>W2r@7-8cLMJ-GVU33-BhTIB3E</regCode>

</Config>

Users can edit the config.xml file by themselves, instead of using the Integrated Development Environment (IDE).

When the command line is used to call an esProc cellset file, the config.xml file residing in a default directory will be loaded automatically to read the license information. Please refer to esProc Integration & Application: Command Line for the use of the command line.

To use the esProc JDBC, the configuration file needs to be copied to the classpath of the Java application project. The license information will be read from the file while esProc JDBC is called. Besides using the default configuration, users can also specify the configuration file with the format ...?config=... in the connecting string. In this case, registration code will be got from the <regCode> node of the specified file. For instance, according to con= DriverManager.getConnection("jdbc:esproc:local:/?config=myconfig.xml");, the license information need to be read from myconfig.xml file instead of the default config.xml file. About the deployment of esProc JDBC, please see esProc Integration & Application: Deploying JDBC.

Next
Previous
Related :
  • 4. License and Encryption


5. Integration by BIRT

esProc provides standard JDBC to integrate with BIRT and other reporting tools. Here we’ll explain the process of integration of esProc and BIRT in development environment and WEB application environment.

Next
Previous
Related :
  • Chapter 6 Integration&Application
  • 5.1 Integration in development environment
  • 5.2 Deploy web server-side


5.1 Integration in development environment

Next
Previous
Related :
  • 5. Integration by BIRT
  • 5.1.1 Create configuration files
  • 5.1.2 Create a report and load the driver jars
  • 5.1.3 Deploy esProc script
  • 5.1.4 Invocation in BIRT


5.1.1 Create configuration files

Create the two configuration files – config.xml and dfxConfig.xml – necessary for esProc JDBC.

Next
Previous
Related :
  • 5.1 Integration in development environment
  • 5.1.1.1 config.xml
  • 5.1.1.2 dfxConfig.xml


5.1.1.1 config.xml

<?xml version=”1.0” encoding=”UTF-8”?>

<Config Version=”1”>

<!—Registration codeà

<regCode>Hojw6-9da4x-NXkD6-089IX@w</regCode>

<harset>ISO-8859-1</harset>

<!—Configure dfx file’s searching path, which is an absolute path. There can be multiple paths separated by semicolon(s). The dfx file can also be placed in the application’s classpath which has the priority over a searching path for loading the fileà

<dfxPathList>

<dfxPath>D:/dfxs</dfxPath>

</dfxPathList>

<dateFormat>yyyy-MM-dd</dateFormat>

<timeFormat>HH:mm:ss</timeFormat>

<dateTimeFormat>yyyy-MM-dd HH:mm:ss</dateTimeFormat>

<!—Method one for datasource configuration: configure connection pool in the application server and specify the datasource nameà

<DBList>

<!—Datasource name must be consistent with that in the dfx file. Here HSQL database is usedà

<DB name=”demo”>

<property name=”url” value=”jdbc:hsqldb:hsql://127.0.0.1/demo”/>

<property name=”driver” value=”org.hsqldb.jdbcDriver”/>

<property name=”type” value=”HSQL”/>

<property name=”user” value=”sa”/>

<property name=”password” value=””/>

<property name=”batchSize” value=”1000”/>

<property name=”autoConnect” value=”false”/>

<property name=”useSchema” value=”false”/>

<property name=”addTilde” value=”false”/>

</DB>

</DBList>

<mainPath/>

<tempPath>temp</tempPath>

<bufSize>65536</bufSize>

<localHost/>

<localPort>0</localPort>

<logLevel>DEBUG</logLevel>

<callxParallel>1</callxParallel>

<nodeParallel>1</nodeParallel>

</Config>

Next
Previous
Related :
  • 5.1.1 Create configuration files


5.1.1.2 dfxConfig.xml

<?xml version=”1.0” encoding=”iso-8859-1”?>

<dfxConfig>

<maxConcurrents>10</maxConcurrents>

<maxWaits>10</maxWaits>

<log></log>

</dfxConfig>

Next
Previous
Related :
  • 5.1.1 Create configuration files


5.1.2 Create a report and load the driver jars

Create a report, add JDBC Data Source to the list of “DataSources”, then move on to the next step.

Click “Manage Drivers” on the New Data Source window to add esProc JDBC’s driver jars.

Three basic jars – dm.jar, icu4j_3_4_5.jar and dom4j-1.6.1.jar – need to be added. They can be found in esProc IDE’s [installation directory]\esProc\lib directory. If esProc also uses other databases as the datasources, then the driver jars of these databases need to be added too. For instance, hsqldb.jar has been added here. About the jars for activating other esProc functions (like accessing Office files), see esPrc Integration & Application: Deploying JDBC.

Place the two configuration files – config.xml and dfxConfig.xml – created in the first step in the classpath’s root path (no function of setting a directory-style classpath is provided here and the configuration files can be compressed into a root directory of any of the jars above like dm.jar). config.xml is allowed to be renamed esprocJDBCconfig.xml in case there are already config.xml files for other purposes under classpath. esProc JDBC driver will find and use esprocJDBCconfig.xml first in preference to config.xml, which will be used when the former does not exist. Names of these configuration files’ names should be remain what they are specified here and must not be changed.

After that, select com.esproc.jdbc.InternalDriver ( v1.0) in the drop-down list of the Driver Class and enter the database URL jdbc:esproc:local://. Leave both the user name and the password blank.

esProc JDBC is a fully-embedded computing engine, enabling all computations to be performed in its embedded package, so the database URL is local. But as an incomplete database, it has neither a user name nor a password.

esProc amounts to a database without the storage mechanism. The JNDI connection pool similar to that used with other databases can also be used when BIRT is connecting to the esProc. Configurations vary according to different types of WEB container. For example the configuring of JNDI connection pool under Tomcat requires that Tomcat’s JNDI URL be entered into the Edit Data Source window, as java:comp/env/demo shown in the following figure:

When creating a report in the Integrated Development Environment (IDE), we did the test that a database is connected through JDBC. Under running environments like Tomcat, the convenient JNDI connection pool is employed for application development.

Note: About the configuration of connecting esProc to a database, see esProc Integration & Application: Deploying JDBC.

Having finished setting the above information, click on “Test Connection” to check if the connection has been successfully established.

Next
Previous
Related :
  • 5.1 Integration in development environment


5.1.3 Deploy esProc script

Copy the esProc script (dfx) to the dfxPath configured in config.xml. Suppose to query the closing stock prices of a certain month, the script stocks.dfx is as follows:

 | A
1 | =connect(“demo”)
2 | =A1.query(“select * from stockrecords where year(date)+’-’month(date)=?”,arg1)
3 | >A1.close()
4 | Result A2

Next
Previous
Related :
  • 5.1 Integration in development environment


5.1.4 Invocation in BIRT

Create a new data set by selecting the datasource just configured. The data set type is stored procedure.

Then define a query text – {call stocks(?)}. The way of invoking an esProc script is similar to that of invoking a stored procedure. “stocks” is the dfx’s name and “?” represents a passed parameter, which is the month being queried.

Set the parameter.

Click on Preview Results to see the query results. At this point, the invocation of esProc script hasbeen completed.

The above is the process of integration of esProc and BIRT development environment. Now let’s look at how to deploy both of them in the WEB server-side.

Next
Previous
Related :
  • 5.1 Integration in development environment


5.2 Deploy web server-side

Select the appropriate application server. Here Tomcat6.0 is used as an example to deploy the application.

Next
Previous
Related :
  • 5. Integration by BIRT
  • 5.2.1 Prepare web distribution package
  • 5.2.2 Create configuration files
  • 5.2.3 Copy necessary jars
  • 5.2.4 Publish the report


5.2.1 Prepare web distribution package

Download BIRT4.3.0 distribution package birt-runtime-4_3_0.zip. The link is http://www.eclipse.org/downloads/download.php?file=/birt/downloads/drops/R-R1-4_3_0-201306131152/birt-runtime-4_3_0.zip .

Copy the WebViewerExample directory to Tomcat’s webapps directory and rename it birt.

Next
Previous
Related :
  • 5.2 Deploy web server-side


5.2.2 Create configuration files

Two configuration files – config.xml and dfxConfig.xml – are needed. The configuration information is the same as that for integration in development environment.

Next
Previous
Related :
  • 5.2 Deploy web server-side


5.2.3 Copy necessary jars

Copy dm.jar, icu4j_3_4_5.jar and dom4j-1.6.1.jar necessary for the above “integration in development environment” to the application’s WEB-INF\lib directory and compress the above two configuration files into dm.jar.

Next
Previous
Related :
  • 5.2 Deploy web server-side


5.2.4 Publish the report

When the report (like stocks.rptdesign) is completed, copy it to the application’s root directory, launch Tomcat and access the website http://127.0.0.1:8080/birt/frameset?report=stocks.rptdesign where we can see the result as displayed below. Now we can call it a successful deployment.

Next
Previous
Related :
  • 5.2 Deploy web server-side


6. Integration with JasperReport

esProc provides standard JDBC interface to be easily integrated with reporting tools. This article will explain the integration of esProc and JasperReport respectively in development environment and WEB server-side deployment.

Next
Previous
Related :
  • Chapter 6 Integration&Application
  • 6.1 Integration in development environment
  • 6.2 Deploy web server-side


6.1 Integration in development environment

Access esProc JDBC to call the esProc script using JasperReport iReport Designer through the following steps:

Next
Previous
Related :
  • 6. Integration with JasperReport
  • 6.1.1 Create configuration files
  • 6.1.2 Load driver jars
  • 6.1.3 Create a datasource
  • 6.1.4 Deploy esProc script
  • 6.1.5 Call esProc script in IDE


6.1.1 Create configuration files

Create the two configuration files – config.xml and dfxConfig.xml – necessary for esProc JDBC.

Next
Previous
Related :
  • 6.1 Integration in development environment
  • 6.1.1.1 config.xml
  • 6.1.1.2 dfxConfig.xml


6.1.1.1 config.xml

<?xml version="1.0" encoding="UTF-8"?>

<Config Version="1">

<!-- Registration code -->

<regCode>HOjw6-9da4x-NXkD6-089IX@w</regCode>

<charSet>iso-8859-1</charSet>

<!-- Configure dfx file’s searching path, which is an absolute path. There can be multiple paths separated by semicolon(s). The dfx file can also be placed in the application’s classpath which has the priority over a searching path for loading the file -->

<dfxPathList>

<dfxPath>D:/dfx</dfxPath>

</dfxPathList>

<dateFormat>yyyy-MM-dd</dateFormat>

<timeFormat>HH:mm:ss</timeFormat>

<dateTimeFormat>yyyy-MM-dd HH:mm:ss</dateTimeFormat>

<!-- Method one for datasource configuration: configure connection pool in the application server and specify the datasource name -->

<DBList>

<!-- Datasource name must be consistent with that in the dfx file. Here MySQL database is used -->

<DB name="mysql">

<property name="url" value="jdbc:mysql://127.0.0.1:3306/test"/>

<property name="driver" value="com.mysql.jdbc.Driver"/>

<property name="type" value="MYSQL"/>

<property name="user" value="root"/>

<property name="password" value=""/>

<property name="batchSize" value="1000"/>

<property name="autoConnect" value="false"/>

<property name="useSchema" value="false"/>

<property name="addTilde" value="false"/>

</DB></DBList>

<mainPath/>

<tempPath>temp</tempPath>

<bufSize>65536</bufSize>

<localHost/>

<localPort>0</localPort>

<logLevel>DEBUG</logLevel>

<callxParallel>1</callxParallel>

<nodeParallel>1</nodeParallel>

</Config>

Next
Previous
Related :
  • 6.1.1 Create configuration files


6.1.1.2 dfxConfig.xml

<?xml version="1.0" encoding="iso-8859-1"?>

<dfxConfig>

<maxConcurrents>10</maxConcurrents>

<maxWaits>10</maxWaits>

<log></log>

</dfxConfig>

Next
Previous
Related :
  • 6.1.1 Create configuration files


6.1.2 Load driver jars

esProc JDBC driver includes three basic jars - dm.jar, icu4j_3_4_5.jar and dom4j-1.6.1.jar. They can be found in esProc IDE’s [installation directory]\esProc\lib directory. If esProc also uses other databases as the datasources, then the driver jars of these databases need to be added too. For instance, mysql-connector-java-5.1.5-bin.jar has been added here. Place all these jars in classpath’s root path. About the jars for activating other esProc functions (like accessing Office files), see esPrc Integration & Application: Deploying JDBC.

Except for these jars, the directory (such as D:\lib\config) of the two configuration files – config.xml and dfxConfig.xml – created in the first step needs to be placed in the classpath’s root path too. Or compress the configuration files into the root directory of any of the jars above. config.xml is allowed to be renamed esprocJDBCconfig.xml in case there are already config.xml files for other purposes under classpath. esProc JDBC driver will find and use esprocJDBCconfig.xml first in preference to config.xml, which will be used when the former does not exist. Names of these configuration files’ names should be remain what they are specified here and must not be changed.

Configure the classpath on the Tool Option bar, as shown in the following figure:

Next
Previous
Related :
  • 6.1 Integration in development environment


6.1.3 Create a datasource

Create a new datasource – esproc – on the datasource configuration window. Select the JDBC driver com.esproc.jdbc.InternalDriver and enter the JDBC URL as jdbc:esproc:local://. Leave the username and password blank.

esProc JDBC is a fully embedded computing engine. All computations are performed in this embedded package, thus the URL is local. On the other hand, since it isn’t a complete database, it has neither the username nor the password.

After finishing all necessary information, click on “Test” to see whether the connection to the datasource has been successfully established.

Next
Previous
Related :
  • 6.1 Integration in development environment


6.1.4 Deploy esProc script

Copy the esProc script (dfx) to dfxPath configured in config.xml (such as D:/dfx configured in the above). Here we’ll take the query of the closing stock price in a certain month as an example. The esProc script is as follows:

 | A
1 | =connect(“mysql”)
2 | =A1.query("select * from STOCKRECORDS where year(tradingdate)*100+month(tradingdate)=?",arg1)
3 | Result A2

Next
Previous
Related :
  • 6.1 Integration in development environment


6.1.5 Call esProc script in IDE

Create a report and set the query

Create a new report – esProcReport, write the query statement and call the esProc script using call dfxName(), in which dfx name is stocks.

Select all the fields and execute the code by clicking on Next Step to the end.

Because the esProc script needs to receive the parameter representing the year and the month, it is necessary to set the parameter for use in the query for the report after it is created. In the following figure, add the parameter representing the year and the month and edit the query for the report.

Then drag the parameter to the specified position of the query statement, thus the query becomes as follows:

Edit the report

Drag all fields under Fields to the report’s detail band and complete the edition, as shown below:

Preview the report

Click on Preview and enter the parameter to see the query result:

This is the integration of esProc script and JasperReport in the development environment. Next we’ll move on to explain how to deploy them in the web server-side.

Next
Previous
Related :
  • 6.1 Integration in development environment


6.2 Deploy web server-side

Select a suitable application server. For this purpose, Tomcat6.0 is used here for illustration.

Next
Previous
Related :
  • 6. Integration with JasperReport
  • 6.2.1Create a web application
  • 6.2.2 Create configuration files
  • 6.2.3 Deploy necessary jars
  • 6.2.4 Copy the report
  • 6.2.5 Define and Publish the JSP report
  • 6.2.6 Preview the report


6.2.1Create a web application

Create a J2EE application, which is named, say, JasperReport, and copy it to Tomcat’s webapps directory.

Next
Previous
Related :
  • 6.2 Deploy web server-side


6.2.2 Create configuration files

Two configuration files – config.xml and dfxConfig.xml – are needed. The configuration information is the same as that for integration in development environment.

Next
Previous
Related :
  • 6.2 Deploy web server-side


6.2.3 Deploy necessary jars

JasperReport driver jar

Deploy jars necessary for JasperReport into the application’s WEB-INF\lib directory. The JasperReport jar can be found in ireport\modules\ext directory of iReport Designer’s installation directory.

esProc JDBC’s driver jar

Copy the jars – dm.jar, icu4j_3_4_5.jar and dom4j-1.6.1.jar – used in the above integration in the development environment and MYSQL’s driver jar – mysql-connector-java-5.1.5-bin.jar – to the application’s WEB-INF\lib directory. Then copy config.xml and dfxConfig.xml to the application’s WEB-INF\classes directory.

Next
Previous
Related :
  • 6.2 Deploy web server-side


6.2.4 Copy the report

Copy the completed report (whose file extension is .jasper) to the corresponding directory of the application. For example, copy esProcReport.jasper to the application’s reportFiles directory.

Next
Previous
Related :
  • 6.2 Deploy web server-side


6.2.5 Define and Publish the JSP report

The following code is to define and publish the JSP report, such as showReport.jsp:

<%@ page language="java" contentType="text/html; charset=UTF-8"

  pageEncoding="UTF-8"%>

<%@ page

  import="java.io.*,

  net.sf.jasperreports.engine.*,

  net.sf.jasperreports.engine.util.*,

  java.util.*,java.sql.*,

  net.sf.jasperreports.engine.export.*"%>

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">

<html>

<head>

<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">

<title>Show JasperReport</title>

</head>

<body>

  <%

  String report = request.getParameter("report");

  File reportFile = new File(application.getRealPath("reportFiles/"

  + report));

  //Load report object

  JasperReport jasperReport = (JasperReport) JRLoader

  .loadObject(reportFile.getPath());

  //Pass report parameter through Map

  Map parameters = new HashMap();

  //Assign value to parameter p1 of the report

  parameters.put("p1", 200901);

  //Connect to esProc

  Class.forName("com.esproc.jdbc.InternalDriver");

  Connection conn = DriverManager.getConnection(

  "jdbc:esproc:local://", "", "");

  //Export the report to the page

  JasperPrint jasperPrint = JasperFillManager.fillReport(

  jasperReport, parameters, conn);

  JRHtmlExporter exporter = new JRHtmlExporter();

  exporter.setParameter(JRExporterParameter.JASPER_PRINT, jasperPrint);

  exporter.setParameter(JRExporterParameter.OUTPUT_WRITER, out);

  exporter.setParameter(

  JRHtmlExporterParameter.IS_USING_IMAGES_TO_ALIGN,

  Boolean.FALSE);

  exporter.exportReport();

  out.flush();

  conn.close();

  %>

</body>

</html>

esProc offers standard JDBC interface. Besides, as a database without the storage mechanism, esProc allows users to connect it by configuring the connection pool. In the above code, esProc is connected through JDBC:

Class.forName("com.esproc.jdbc.InternalDriver");

Connection conn = DriverManager.getConnection(

"jdbc:esproc:local://", "", "");

With JNDI connection pool, the code will become this under Tomcat:

Context c = new InitialContext();

DataSource ds = (DataSource)c.lookup("java:comp/env/esproc");

Connection conn = ds.getConnection();

Note: About the configuration of connecting esProc to a database, see esProc Integration & Application: Deploying JDBC.

Next
Previous
Related :
  • 6.2 Deploy web server-side


6.2.6 Preview the report

Launch Tomcat and access the website http://127.0.0.1:8080/JasperReport/reportJsp/showReport.jsp?report=esProcReport.jasper to preview the report as shown below, which means the deployment is completed.

Next
Previous
Related :
  • 6.2 Deploy web server-side


7. User-defined Functions

Besides the system-provided functions, esProc supports invoking user-defined functions to handle some special operations or the encapsulation of certain computations. This article will briefly explain how to invoke user-defined functions with invoke function and how to use parameters and return the results as needed.

Next
Previous
Related :
  • Chapter 6 Integration&Application
  • 7.1 Basic method of invoking user-defined functions
  • 7.2 Return results as needed
  • 7.3 Parameters of sequence type


7.1 Basic method of invoking user-defined functions

invoke function can be used to invoke static method(s) in a specified user-defined Java class. Take the following Java class test.Calc01 as an example:

package test;

public class Calc01 {

public static Double distance1(Number loc) {

double len = Math.abs(loc.doubleValue());

len = Math.round(len*1000)/1000d;

return Double.valueOf(len);

}

}

In this simple class, the static method distance1 computes the distance between a given coordinate and the origin, with the value being rounded to three decimal places. The methods being invoked in esProc must be both static and public. Before invocation, the class of user-defined functions needs to be placed in esProc’s classpath, [installation directory]\esProc\classes path; if invoked in the web server-side, it needs to be placed in WEB-INF/classes path correspondingly. After that, the user-defined functions can be invoked by invoke function in the cellset:

 | A
1 | -12.34567
2 | =invoke(test.Calc01.distance1,A1)
3 | =invoke(test.Calc01.distance1, -512)

As shown in this example, when using invoke function, first specify the full path of the class being invoked and the static method name and then list the necessary parameters in order. The parameters can be cell values in a cellset and cellset parameters, or can be typed in. After computation, results of A2 and A3 are as follows:

There can be multiple user-defined functions in a single class. Now add another method:

public static Double distance2(Number loc1, Number loc2) {

double len = Math.abs(loc1.doubleValue()-loc2.doubleValue());

len = Math.round(len*1000)/1000d;

return Double.valueOf(len);

}

The newly-added distance2 computes the distance between two coordinates in a number axis. As esProc invokes methods by their names, different user-defined functions need to have different method names. Once the class(es) has been place in the application’s classpath, the user-defined functions can be invoked in the same cellset, no matter they are from a single class or multiple classes:

 | A
1 | =invoke(test.Calc01.distance1,-12.3456)
2 | =invoke(test.Calc01.distance2,12,-12)

In the invoke function, the number and type of parameters need to match the methods being invoked. Results of A1 and A2 are as follows:

Next
Previous
Related :
  • 7. User-defined Functions


7.2 Return results as needed

The user-defined functions may or may not return results. For example:

public static void distance3(Number loc1, Number loc2) {

double len = Math.abs(loc1.doubleValue()-loc2.doubleValue());

len = Math.round(len*1000)/1000d;

System.out.println(Double.toString(len));

}

When the computation with static method distance3 is completed, the output data will be handled with print spooling and won’t be returned. Such functions are similar to the executable cells in a cellset, in which invoke function can start with >. For example:

 | A
1 | >invoke(test.Calc01.distance3,-12.3,15)

Now the output data can be viewed in the console. To access the console, click Tool>Console on the menu bar. The result is as follows:

But if the user-defined functions return results, they need to use data types supported in esProc; otherwise errors may occur during presentation or invocation. Following is the correspondence between commonly-used data types in esProc and those in Java:

Integer java.lang.Integer

Long integer java.lang.Long

Floating point number java.lang.Double

Big decimal java.math.BigInteger

Real number java.lang.Number

Boolean java.lang.Boolean

String java.lang.String

Date java.sql.Date

Time java.sql.Time

Date/time java.sql.TimeStamp

Sequence com.raqsoft.dm.Sequence

Table sequence com.raqsoft.dm.Table

In the previous examples, all the returned data are Double object, which corresponds to the floating point number in esProc. User-defined function can return results as needed. For instance, add another two user-defined functions to test.Calc01 and return results of different data types:

public static String distance4(Number loc1, Number loc2) {

double len = Math.abs(loc1.doubleValue()-loc2.doubleValue());

len = Math.round(len*1000)/1000d;

return Double.toString(len);

}

public static Sequence distance5(Number loc1, Number loc2) {

double len = Math.abs(loc1.doubleValue()-loc2.doubleValue());

len = Math.round(len*1000)/1000d;

com.raqsoft.dm.Sequence result = new com.raqsoft.dm.Sequence();

result.add(loc1);

result.add(loc2);

result.add(Double.valueOf(len));

return result;

}

These two static methods are still used to compute the distance between two points in a number axis. distance4 returns the result of string type and distance5 returns the result in a form of a sequence in which the coordinates of the two points and the distance between them are stored. invoke function is still used to invoke the two methods in a cellset:

 | A
1 | =invoke(test.Calc01.distance2,-12.3,15)
2 | =invoke(test.Calc01.distance4,-12.3,15)
3 | =invoke(test.Calc01.distance5,-12.3,15)

For the convenience of comparison, A1 calls distance2 used in the previous example. After the code is executed, results of A1~A3 are as follows:

Notice that the results of A1 and A2 use different data types and are displayed differently. The return results of user-defined functions can be used for later computation.

Next
Previous
Related :
  • 7. User-defined Functions


7.3 Parameters of sequence type

The parameters used in a user-defined function come from esProc. The correspondence between their data types and the Java objects is the same as that mentioned above. Note that the parameters used during the invocation of user-defined functions need to be consistent with the parameters in the static methods.

A special point worth noting is that the sequence is the most frequently used data type in esProc. Besides returning a sequence as the result, user-defined functions can also use parameters of sequence type. For example:

public static Double distance6(com.raqsoft.dm.Sequence seq1, com.raqsoft.dm.Sequence seq2) {

int len1 = seq1.length();

int len2 = seq2.length();

double x1 = len1 > 0 ? ((Number) seq1.get(1)).doubleValue(): 0;

double x2 = len2 > 0 ? ((Number) seq2.get(1)).doubleValue(): 0;

double y1 = len1 > 1 ? ((Number) seq1.get(2)).doubleValue(): 0;

double y2 = len2 > 1 ? ((Number) seq2.get(2)).doubleValue(): 0;

double len = Math.sqrt((x1-x2)*(x1-x2)+(y1-y2)*(y1-y2));

len = Math.round(len*1000)/1000d;

return Double.valueOf(len);

}

The user-defined function distance6 computes the distance between two points in a rectangular coordinate system. Coordinates of the two points need to be input using parameters of sequence type during invocation. For example:

 | A
1 | [1,1]
2 | [3,3]
3 | =invoke(test.Calc01.distance6,A1,A2)

After computation, result of A3 is as follows:

Next
Previous
Related :
  • 7. User-defined Functions


Chapter 7 External Memory Computation

Next
Previous
Related :
  • 1. Concept of Cursor
  • 2. Basic Usages of the Cursor
  • 3. Aggregate Operations with Cursor
  • 4. Merge and Join Cursor Data
  • 5. Principle of Grouping
  • 6. Principle of Sorting
  • 7. Text Files
  • 8. Binary Files
  • 9. Group Cursor
  • 10. Cross-cellset Cursor


1. Concept of Cursor

The concept of cursor is very important for the database. With the cursor, data can be manipulated more flexibly and returned from the data table by rows. esProc supports many types of cursor, like database cursor, file cursor and in-memory record sequence cursor, to satisfy various needs in data fetching and processing. In this article, we’ll introduce what the cursor used in esProc is and its three types.

Next
Previous
Related :
  • Chapter 7 External Memory Computation
  • 1.1 What is cursor
  • 1.2 Database cursor
  • 1.3 External file cursor
  • 1.4 In-memory record sequence cursor


1.1 What is cursor

Using the cursor in the database enables returning the result in batches, instead of trying to retrieve all of it into the memory at a time. The cursor is like a pointer, it fetches one row of data each time from the result set by moving its own position. This mechanism of retrieving data by rows is usually used to process big data table, which cannot be entirely loaded into the limited memory.

Compared with directly returning all result, the database cursor is flexible enough to avoid the memory overflow but its speed of processing is not very fast.

There are some differences, however, between esProc cursor and database cursor:

esProc cursor is only used to get data without modifying the result set.

Data are traversed only once from front to back during fetching data from the esProc cursor, which is similar to the setting of TYPE_FORWARD_ONLY in JDBC.

esProc cursor can be created based on either the database or the data file, or the in-memory record sequence.

Different types of esProc cursors have similar usages and we can mix and match them to suit the requirements.

Next
Previous
Related :
  • 1. Concept of Cursor


1.2 Database cursor

With db.cursor(sql) function, data returned from the database can be converted into the cursor. For example:

 | A
1 | =demo.cursor("select * from EMPLOYEE")
2 | =A1.fetch(100)
3 | =A1.fetch(100)
4 | >A1.close()

A1 creates a database cursor. A2 fetches the first 100 records from it, as shown below:

Data fetched from the cursor with cs.fetch() will be returned as a table sequence.

A3 will follow on to fetch another 100 records in order, for esProc allows only a single traversal of cursor data from front to back:

As can be seen from EID field, A3 fetches the records from the 101th employee. In this example, cs.fetch() is used merely to fetch data from the cursor. For more usages of the esProc cursor, please refer to esProc External Memory Computing: Basic Usages of the Cursor.

If the data in the cursor haven’t been all fetched out when data fetching is over, use cs.close() to close the cursor. It is very important to close the cursor, particularly the database cursor, timely. Because the database cursor gets data from the database, the existence of it means the connection to the database will remain, causing unnecessary memory usage and even the overrun of cursors allowed to be created by the database.

Similar to the use of db.query() function in data query, the parameter is allowed in creating a database cursor. For example:

 | A
1 | =demo.cursor("select EID, NAME, SURNAME, GENDER, STATE from EMPLOYEE where GENDER=? and EID>?","F",100)
2 | =A1.fetch()

In SQL statements, parameters can be represented by the question mark (?). They are entered in a certain order after it and separated by the comma. A1’s cursor will return data of female employees whose EID is greater than 100. A2 returns all data of A1’s cursor, as shown below:

The number of rows of data to be returned is not specified for the cs.fetch() function used in A2, thus all data in the cursor will be returned. When data in a cursor have all been fetched out, it will close automatically with no need of calling cs.close().

Next
Previous
Related :
  • 1. Concept of Cursor


1.3 External file cursor

In practice, particularly big data processing, data are often generated from the file rather than the database. Given this, esProc provides the function of creating the external file cursor for data fetching based on the structured file. The following text file PersonnelInfo.txt holds the personnel information:

In the file, each row holds a record and columns are separated by tabs. The external file cursor can be created according to the file for data fetching. For example:

 | A
1 | =file("PersonnelInfo.txt")
2 | =A1.cursor@t()
3 | >A2.skip(50000)
4 | =A2.fetch(100)
5 | >A2.close()

A2 creates a cursor based on the external file and uses @t option to make the file’s first row the column names of the result set. A3 skips the first 50,000 rows and then A4 fetches 100 rows from A2’s cursor, as shown below:

As the database cursor, data are fetched from the external file cursor through a single one-way traversal. After A3 skips certain rows using cs.skip(), the data fetching in A4 will begin from the 50,001th row.

While creating a cursor based on a certain file, it is not necessary to use all its fields. We can specify fields to create the cursor, for example:

 | A
1 | =file("PersonnelInfo.txt")
2 | =A1.cursor@t(ID, Name, State, Gender)
3 | >A2.skip(50000)
4 | =A2.fetch(100)
5 | >A2.close()

The code in the above cellset is almost the same as that the previous one, except that A2 creates the external file cursor using specified fields.

Note: It’s only after field names are imported using @t option that they can be used to specify fields. Without the option, fields can only be specified directly by their positions, like =A1.cursor(#1, #2, #6, #3).

About the external file cursor, we’ll make a deeper explanation in esProc External Memory Computing: Text Files.

Next
Previous
Related :
  • 1. Concept of Cursor


1.4 In-memory record sequence cursor

In esProc, the result table sequence or record sequence can be converted into the cursor to be used with other types of cursor. For example:

 | A
1 | $(demo) select STATEID, NAME, ABBR from STATES order by STATEID
2 | =A1.cursor()

The following is data of the states A1 selects:

A2 creates a cursor using the table sequence:

Generally the in-memory record sequence is converted into the cursor to perform some operations based on the cursor, like merging and joining the cursor data. For details in this respect, please refer to esProc External Memory Computing: Merge and Join Cursor Data.

Besides, the cursor can be created based on the result table sequence or record sequence got in another cellset file. Take cellset file createTable.dfx as an example:

 | A | B
1 | =create(ID,Amount) |
2 | for 1000 | >A1.insert(0,#A2,rand(100*100))
3 | result A1 |

The result of the cellset file returned by the result statement can be converted into a cursor, as shown below:

 | A
1 | =pcursor("createTable.dfx")
2 | =A1.fetch(100)
3 | >A1.close()

A2 fetches the first 100 rows from the A1’s cursor:

This is only a simple conversion of a result table sequence got in another cellset file into the in-memory record sequence cursor, which is a kind of cross-cellset cursor in this case. More usages of the cross-cellset cursor will be illustrated in the esProc External Memory Computing: Cross-cellset Cursor.

Next
Previous
Related :
  • 1. Concept of Cursor


2. Basic Usages of the Cursor

esProc supports importing big data in batches with the cursor, which is the usual method used in big data computing. Usages of cursors, including external file cursor, database cursor and in-memory record sequence cursor, are basically the same. This article will take the external file cursor as the example to explain basic usages of the cursor in esProc.

Next
Previous
Related :
  • Chapter 7 External Memory Computation
  • 2.1 Data fetching
  • 2.2 Cursor loop
  • 2.3 Computation with the cursor


2.1 Data fetching

When a cursor is created, its data can be fetched by cs.fecth() function. Different from the way data of an ordinary table are imported, data in the cursor are fetched by specifying the number of rows to be returned, or one or more conditions on which data will be fetched, rather than being fetched all at once. For example:

 | A
1 | =file("Order_Books.txt")
2 | =A1.cursor@t()
3 | =A2.fetch(1000)
4 | >A2.close()
5 | =A2.fetch(1000)

A2 creates an external file cursor. A3 fetches the first 1,000 rows from it, as shown below:

With fetch function, each batch of data fetched from the cursor will be returned as a table sequence. A4 calls cs.close() function to close the cursor. When a cursor is closed, data cannot be fetched from it. That is why the result of A5 is null:

When the cursor has accomplished its mission and is not needed any more, it ought to be closed through calling cs.close() function to release the memory.

Besides fetching data from the file or the database result set with the cursor, cs.skip() function can be used to skip the specified number of rows in the cursor. For example:

 | A
1 | =file("Order_Books.txt")
2 | =A1.cursor@t()
3 | >A2.skip(45000)
4 | =A2.fetch()
5 | =A2.fetch(1000)

A2 creates the cursor and A3 skips 45,000 rows. Then A4 fetches data from the cursor as follows:

As can be seen from the transaction ID, the data fetching begins from the 45001th row. Notice that the fetch function in A4 doesn’t specify the number of rows to be returned. In this case, all the rest of the data in the cursor will be fetched. Once data in the cursor are completely fetched, the cursor closes automatically, without the necessity of calling cs.close() function.

So A5 fetches nothing from A2’s cursor:

In fact, both cs.skip() function and cs.fecth() function will only traverse the cursor data once from front to back. When the data fetching is over or cs.close() function is executed, the cursor becomes useless.

Another type of fetching data from the cursor using cs.fecth() function is cs.fetch(;x), which specifies the condition on which data fetching will continue interruptedly until the value of expression x changes. For example:

 | A
1 | =file("Order_Books.txt")
2 | =A1.cursor@t()
3 | =A2.fetch(;Date)
4 | =A2.fetch(1000)
5 | >A2.close()

A2 creates the cursor. A3 fetches data from it until the value of Date changes. The fetched data are as follows:

It can be seen that A3 fetches all data whose Date value is 2013-01-01. When A4 goes on with fetching data from A2’s cursor, it will begin from the date 2013-01-02:

Note that we cannot specify both the number of rows to be returned and the condition for data fetching at the same time; otherwise only the former is valid. Besides, since not all the data in A2’s cursor have been fetched, the cursor needs to be closed in A5 after doing its job.

Similarly, we can also specify condition for cs.skip() function to skip certain records. For example:

 | A
1 | =file("Order_Books.txt")
2 | =A1.cursor@t()
3 | >A2.skip(;Date)
4 | >A2.skip(;Date)
5 | >A2.skip(;Date)
6 | =A2.fetch(1000)
7 | >A2.close()

A2 creates the cursor. A3, A4 and A5 respectively skip records of one day. Then A6 will fetch data from the cursor from the fourth day:

Similarly, if not all the data have been fetched from the cursor, call cs.close() function in A7 to close the cursor.

Next
Previous
Related :
  • 2. Basic Usages of the Cursor


2.2 Cursor loop

When fetching data from the cursor, use for statement to process the rows returned from the cursor by loop. For example:

 | A | B
1 | =file("Order_Books.txt") |
2 | =A1.cursor@t() | 0
3 | for A2,1000 | =A3.select(SalesID==1).sum(Amount)
4 |  | >B2=B2+B3

A2 creates the cursor. By loop, A3 processes 1,000 records returned by the cursor each time. After all records are traversed, the cursor closes automatically. In A3’s code block, based on the table sequence returned by the cursor each time, the code computes the total sales amount of the salesperson whose ID number is 1 and stores the result in B2. The value is shown as follows:

Sometimes not all data need to be traversed during the cursor loop. For example:

 | A | B | C
1 | =file("Order_Books.txt") |  |
2 | =A1.cursor@t() | [] |
3 | for A2,1000 | =A3.select(SalesID==1) | >B2=B2|B3
4 |  | if B2.len()>=5 | >B2=B2(to(5))
5 |  |  | break
6 | >A2.close() |  |

A2 creates the cursor and A3 executes cursor loop, fetching 1,000 records each time. In A3’s code block, according to the batches of data fetched from the cursor, the code selects the sale records of the salesperson whose ID number is 1 and stores them in B2. Exit the loop after the first 5 records are fetched. At this point the result seen in B2 is as follows:

One point worthy of special note: When the computation is over and the loop is terminated by break function, there are data still remaining in the cursor. Different from the previous example, in this case the cursor won’t close automatically, thus close function is called in A6 to close it and release the memory.

Similar to cs.fecth() and cs.skip(), the condition can be specified for the cursor loop, so that the data fetching will stop as soon as the value of the conditional expression changes. For example:

 | A | B | C
1 | =file("Order_Books.txt") |  |
2 | =A1.cursor@t() | =create(Date,Count,Amount) |
3 | for A2;Date | =A3.count() | =round(A3.sum(Amount))
4 |  | >B2.insert(0,A3.Date,B3,C3) |

With the condition, A3 fetches the sales data of one day each time, computes the number of transactions and the total sales amount in this day in its code block and stores the results in B2’s table sequence. After the code is executed, B2’s result table sequence is as follows:

Next
Previous
Related :
  • 2. Basic Usages of the Cursor


2.3 Computation with the cursor

Besides data fetching, esProc offers the use of cursor in performing common forms of data computing by directly using certain functions. The following example is to select data from the cursor according to the condition:

 | A | B
1 | =file("Order_Books.txt") | =A1.cursor@t()
2 | =B1.select(SalesID==1) | =A2.fetch()
3 | =A2.fetch() | =B1.fetch()

B1 creates the external file cursor and A2 selects the sales data of the salesperson whose ID number is 1 from B1’s cursor. A2’s result is as follows:

It can be seen that, different from the table sequence or the record sequence, the select function used with the cursor returns the cursor as the result. Actually A2 hasn’t really got data from the cursor. It is in B2 where cs.fecth() function is being executed that the eligible data are fetched from B1’s cursor according to the condition. Result of B2 is as follows:

No number of rows to be returned or the condition for data fetching is specified for cs.fecth() function used in B2. All data, or all the sales data of the salesperson, in A2’s cursor will be returned. After data of the cursor in A2 have all been fetched out, the cursor will close automatically. Since the cursor in A2 is generated from the one in B1, the data query is in fact executed by traversing data of the cursor in B1 and the completion of fetching data from the cursor in A2 means the same thing to the cursor in B1, which will close at the same time. Therefore the effort of A3 and B3 trying to fetch data from the cursors in either A2 or B1 will definitely fail.

The cursor can also be used to create new records, or append new fields. For example:

 | A | B
1 | =file("employee.txt") | =A1.cursor@t(EID, NAME,SURNAME, GENDER, STATE,BIRTHDAY)
2 | =B1.new(EID, NAME+" "+SURNAME: FullName, GENDER,STATE) | =B1.derive(age(BIRTHDAY):Age)
3 | =A2.fetch(100) | =B2.fetch(100)
4 | >A2.close() | =B2.fetch()
5 | >B2.close() |

B1 creates a file cursor using some of the fields of the external file employee.txt. In A2, the employees’ full names are generated by joining NAME field and SURNAME field in B1’s cursor together. B2 computes each employee’s age according to the BIRTHDAY field in B1’s cursor and insert the results to Age field. Similar to cs.select(), both cs.new() and cs.derive() return cursor as the result, rather than returning data directly. When executed, A2 and B2 have the following results:

A3 fetches the first 100 rows from A2’s cursor:

Generating data with cs.new() function means executing new statement on every record.

B3 fetches the first 100 rows from B2’s cursor:

Generating data with cs.derive() function means appending computed columns into the returned table sequence.

It can be seen that B3’s data begin not from the first employee, but from the 101th one. That’s because fetching data from cursors in A2 and B2 is, in reality, fetching data from the cursor in B1 and cursor data will be traversed only once from front to back. After A3 is executed, the first 100 rows of data in B1’s cursor have been traversed. So B3 can only go on with the traversal in order in the execution of cs.fecth() function by beginning with the 101th employee.

Because data haven’t been all fetched out, the cursors need to be closed using cs.close() function. As A4 closes the cursor in A2, the cursor in B1 is also closed simultaneously, for the former is generated by the latter. This affects the cursor in B2, from which B4 is trying to fetch data but to no avail.

In practical use, we ought to avoid using the same cursor to generate multiple ones as possibly as we can; the same problem as the one with B3 in fetching data, or as another one that cursors will be unavailable when the source cursor closes, will arise otherwise.

What’s more, cs.run() can be used to modify values of a certain field of cursor data. For example:

 | A | B
1 | =file("employee.txt") | =A1.cursor@t(EID, NAME,SURNAME, GENDER, STATE,BIRTHDAY)
2 | =B1.new(EID, NAME+" "+SURNAME: FullName, GENDER,STATE) | =B1.run((a=STATE,STATE=A2.select@1(NAME==a).ABBR))
3 | =B2.fetch(100) | >B2.close()

B2 replaces names of the states in the returned data by their abbreviations and still returns the cursor as the result:

A3 fetches the first 100 rows as follows:

Similar to a table sequence, the cursor allows the conversion of a certain field of its data into the corresponding records in another table sequence by using cs.switch() function. For example:

 | A | B
1 | =file("employee.txt") | =A1.cursor@t(EID, NAME,SURNAME, GENDER, STATE,BIRTHDAY)
2 | =demo.query("select STATEID, NAME, ABBR from STATES") | =B1.switch(STATE,A2:NAME)
3 | =B2.fetch(100) | >B2.close()

B2 uses cs.switch() to switch certain data of the cursor. Its result is also a cursor:

The data A3 fetches are as follows：

It can be seen that, in the return result, STATE field has been converted into the corresponding state records.

cs.switch() can convert more than one field of the cursor. For example:

 | A | B
1 | =file("employee.txt") | =A1.cursor@t(EID, NAME,SURNAME, GENDER, STATE,BIRTHDAY)
2 | [F,Female,M,Male] | =create(ID,Gender).record(A2)
3 | =B1.new(EID, NAME+" "+SURNAME: FullName, GENDER,STATE) | =B1.switch(GENDER,B2;STATE,A3:NAME)
4 | =B3.fetch(100) | >B3.close()

B3 switches records of several fields of the cursor to the corresponding ones in another table sequence. A4 fetches the first 100 records:

In the article – Related Computing in esProc – Primary Keys and Index Function, we explained the usage of switch in the ordinary table sequence, which is similar to that of cs.switch() in external memory computing. Both functions will create corresponding index table for certain field(s) of the dimension table in order to increase efficiency. It is more important to adopt this processing method while using the cursor because the data under processing are usually big.

Next
Previous
Related :
  • 2. Basic Usages of the Cursor


3. Aggregate Operations with Cursor

The data volume of the big data table is usually quite huge, which makes it impossible to retrieve all data from the big data table at once. In view of this, the data processing over big data table is usually to serve two purposes: With cs.fetch(), retrieve partial data each time or group & aggregate the data in the big data table.

Next
Previous
Related :
  • Chapter 7 External Memory Computation
  • 3.1 Group and aggregate data of a cursor
  • 3.2 Aggregate data from multiple cursors
  • 3.3 Big result sets


3.1 Group and aggregate data of a cursor

The commonest grouping & aggregate operations are count and summation. In addition, the aggregate operation includes computing the maximum/minimum value and top-n queries. For the aggregate operation over the cursor, you can use function cs.groups(). The aggregate functions corresponding to various aggregate operations include count/sum/max/min/topx. For example:

 | A
1 | =file("Order_Wines.txt")
2 | =file("Order_Electronics.txt")
3 | =file("Order_Foods.txt")
4 | =file("Order_Books.txt")
5 | =[A1:A4].(~.cursor@t())
6 | =A5.merge@x(Date)
7 | =A6.groups(Type;count(~):OrderCount,max(Amount):MaxAmount,sum(round(decimal(Amount),2)):Total)

In A7, make statistics on the total number of orders, maximum trade amount, and total amount of different kinds of products. One thing to note is that the precision of the double-precision data is usually not enough. In this case, decimal() is used to convert the data into the big decimal type for computing. Regarding the grouping and aggregate computation with the cursor, all data will be traversed just once. During the traversal, each row of data will be aggregated based on the group to which it belongs. Multiple aggregate values can be computed in one traversal, and the cursor will be closed automatically once the traversal is completed.

The result in A7 is as follows:

As can be seen from the result, in the text data for test, each product has 50,000 records of orders. Data of every file and the data of A6’s cursor are sorted by Date, but not sorted by Type. Seen from the use of cs.groups() function in the example, the function groups and summarizes data in the cursor without requiring them to have been sorted. After data grouping and aggregation, result will be sorted by the grouping value. In addition, because cs.groups() returns a table sequence, the result set must not exceed the memory size when the function is used to handle the grouping and aggregate operations on the cursor. Thus it is not fit for the cases in which the result set holds big data.

To find the n greatest/smallest data values is also one of the aggregate operations. In this case, use the aggregate function topx:

 | A
1 | =file("Order_Wines.txt")
2 | =file("Order_Electronics.txt")
3 | =file("Order_Foods.txt")
4 | =file("Order_Books.txt")
5 | =[A1:A4].(~.cursor@t())
6 | =A5.merge@x(Date)
7 | =A6.groups(Type;topx(Amount;3):MinAmount,topx(-Amount;3):MaxAmount)

A7computes the bottom 3 orders with the lowest total amount and the top 3 orders with the highest total amount among the orders of each kind of product. Please remember to add the negative sign to the aggregate function when computing the top 3 orders, that is, topx(-Amount:3). The result is as follows:

To aggregate all records, simply omit the grouping expression from cs.groups function:

 | A
1 | =file("Order_Wines.txt")
2 | =file("Order_Electronics.txt")
3 | =file("Order_Foods.txt")
4 | =file("Order_Books.txt")
5 | =[A1:A4].(~.cursor@t())
6 | =A5.merge@x(Date)
7 | =A6.groups(;count(~):Count,sum(round(decimal(Amount),2)):TotalAmount)

The result in A7 is as follows:

This shows that the aggregation of all records can be regarded as a special kind of grouping and aggregate operation.

Next
Previous
Related :
  • 3. Aggregate Operations with Cursor


3.2 Aggregate data from multiple cursors

When handling the aggregate operations over multiple cursors, you can compute the aggregate result of each cursor first:

 | A
1 | =file("Order_Wines.txt")
2 | =file("Order_Electronics.txt")
3 | =file("Order_Foods.txt")
4 | =file("Order_Books.txt")
5 | =[A1:A4].(~.cursor@t())
6 | =A5.(~.groups(Date;count(~):Count,sum(round(decimal(Amount),2)):TotalAmount))

A6 groups and aggregates each file cursor to compute the daily aggregated statistics of each kind of product:

Because the data will be sorted by grouping value when using the cs.groups() function to aggregate and group the cursor data, the result table sequence of aggregating the order data of each kind of product is always ordered by the date, whatever the order of original data is. So, the result can be merged further to compute the daily order data for all products:

 | A
1 | =file("Order_Wines.txt")
2 | =file("Order_Electronics.txt")
3 | =file("Order_Foods.txt")
4 | =file("Order_Books.txt")
5 | =[A1:A4].(~.cursor@t())
6 | =A5.(~.groups(Date;count(~):Count,sum(round(decimal(Amount),2)):TotalAmount))
7 | =A6.merge(Date)
8 | =A7.groups@o(Date;sum(Count):Count,sum(round(decimal(TotalAmount),2)):TotalAmount)

In A8, you will see the summarizing result:

Since the order data of each kind of product in every data file have been sorted by date, cursors can be merged first, and then aggregated:

 | A
1 | =file("Order_Wines.txt")
2 | =file("Order_Electronics.txt")
3 | =file("Order_Foods.txt")
4 | =file("Order_Books.txt")
5 | =[A1:A4].(~.cursor@t())
6 | =A5.merge@x(Date)
7 | =A6.groups@o(Date;count(~):Count,sum(round(decimal(Amount),2)):TotalAmount)

As you can see, this method leads to the same result as that got in the above. For cursors each of which data are ordered, it would be simpler and more effective to merge first, then group and aggregate. Note the difference between merge and merge@x. A.merge() is to merge the members of multiple sequences or records of multiple table sequences in proper order, and return a sequence. The computation is completed during the execution; CS.merge@x() is to merge the records from multiple cursors in proper order, return a cursor, and start the computing when fetching data from the cursor. Both functions require that the data in A and CS are ordered. There are detailed explanations about the usage of merge@x in esProc External Memory Computing: Merge and Join Cursor Data.

Next
Previous
Related :
  • 3. Aggregate Operations with Cursor


3.3 Big result sets

In the above section, we’ve learned that the data of each cursor can be grouped and aggregated first to have a group of ordered result table sequences, which then will be merged to get the final result. With such method, we can solve the grouping and aggregating problems related to the big result sets.

The big result set means not only the source data for computation is of huge amount, but the computed result has huge data too. Owing to this, the computed result set cannot be read into the memory all at once. It has to be read out step by step. For example, the monthly bill for each customer of telecommunication company and the sales of each product of the B2C website. These statistical results may contain more than several millions of records.

In esProc, you can use cs.groupx() function to perform the grouping and aggregate operations over the big result set. Here, we will take the daily statistic on product orders as the example to illustrate the processing of big result sets. In order to have a better understanding about the memory limit, we retrieve 100 records each time. Thus, if we need to compute the daily order by grouping and aggregating, we should use groupx function:

 | A
1 | =file("Order_Wines.txt")
2 | =file("Order_Electronics.txt")
3 | =file("Order_Foods.txt")
4 | =file("Order_Books.txt")
5 | =[A1:A4].(~.cursor@t())
6 | =A5.conj@x()
7 | =A6.groupx(Date;count(~):Count,sum(round(decimal(Amount),2)):TotalAmount;100)
8 | =A7.fetch(100)
9 | =A7.fetch(100)
10 | =A7.fetch(100)
11 | =A7.fetch(100)

In A6, the data in each cursor are concatenated in proper order, instead of being sorted by date. A7 returns the result of the grouping and aggregate operation as a cursor instead of a table sequence. By doing so, the data can be retrieved step by step in A8~A11. The data are shown below:

In which, each of A8~A10 retrieves the statistical result for 100 days, and A11 retrieves the remaining data. After the data of A7’s cursor are all fetched out, the cursor will close automatically.

In A7, with groupx function, the data are grouped and aggregated by date, and the number of buffer rows is set to 100. By doing so, when executed, A7 will fetch data from A6’s cursor and meanwhile, split and aggregate them. Once the result of aggregating 100 rows is produced, it will be buffered to a temporary file, and the rest can be done in the same manner. The computed result in A7 is a cursor composed of these temporary files:

Through step-by-step execution, you can view the generated temporary files in the system temporary directory after A7 is executed:

In order to see the contents of each of these binary files, you can retrieve the data from them, for example:

 | A
1 | =file("temp/tmpdata1372633253707222252").import@b()
2 | =file("temp/tmpdata2669923936558108709").import@b()
3 | =file("temp/tmpdata3937021394605697131").import@b()
4 | =file("temp/tmpdata8767777617311441973").import@b()

The data in A1 and A3 are shown below:

The name of these temporary files are generated randomly. Judging from the data retrieved from some temporary files, each temporary file stores the grouping and aggregate result of some consecutive original records, which are already sorted by the date. In fact, based on the number of buffer rows specified in the function, each temporary file, except for the last one, contains partial summarizing data for just 100 days. For a cursor generated with groupx, its data will be fetched with fetch by merging the data of all temporary files in proper order.

After cursor data are fetched out completelyby using the temporary file cursors or these cursors are closed, the corresponding temporary files will be deleted automatically. About the external memory grouping, please refer to esProc External Memory Computing: Principle of Grouping.

Next
Previous
Related :
  • 3. Aggregate Operations with Cursor


4. Merge and Join Cursor Data

During data computing based on the table sequence, we can combine data from multiple table sequences together for use in the analysis and computation. For instance, use A.merge() to combine records of multiple table sequences in a certain order, or A.conj() to union them in order into a grand table, or JOIN functions, like join, pjoin and xjoin, to establish certain relationship between multiple table sequences by joining their data together and perform query or other computations based on this relation.

Cursors for processing big data can be merged and joined likewise. This article aims to teach the method of merging and joining data of the cursor with functions including CS.conj@x(), CS.merge@x(x) and join@x(), etc.

Next
Previous
Related :
  • Chapter 7 External Memory Computation
  • 4.1 Merge data in order
  • 4.2 Aligned joining
  • 4.3 Unconditional joining


4.1 Merge data in order

Most of the time, data could be stored in multiple data tables, for example, sales records of multiple products and employee profiles of each department. In this case, we need to merge the data from multiple data tables together for a better use. For multiple normal homogeneous table sequences, A.conj() or A.merge(x) can be used to merge the records of these table sequence into a record sequence. For big data, use CS.conj@x() and CS.merge@x(x) to combine data in every cursor of cursor sequence CS, thus the data can be fetched as a whole.

Data in the cursor will only be traversed once, so it is impossible to re-sort them after they are merged and fetched out. In view of this, the data in each cursor must be ordered in the case of merging data from multiple cursors.

Now let’s learn about the usages of and differences between CS.conj@x() and CS.merge@x(x). Firstly, let’s look at the case of simple union.

 | A | B | C
1 | =file("Order_Wines.txt") |  |
2 | =file("Order_Electronics.txt") |  |
3 | =file("Order_Foods.txt") |  |
4 | =file("Order_Books.txt") |  |
5 | =[A1:A4].(~.cursor@t()) |  |
6 | =A5.conj@x() | 300 |
7 | for | =A6.fetch(B6) |
8 |  | if B7==null | break
9 |  | =B7(1).Type | =B7(B6).Type
10 |  | if B9!=C9 | break
11 | >A6.close() |  |

There are four text files respectively recording the order information about wines, electrical appliances, foods, and books. In A6, the four file cursors are concatenated. To find out the order in which the data are retrieved, the following code retrieves 300 records each time and data retrieval is terminated once the record of a different kind of goods appear. The result table sequence in B7 is as follows:

As can be learned from the result, regarding the unioned cursor, after all wine order data are retrieved from the 1st text data table, the electrical appliance data from the 2nd text data table follows. In other words, after the simple union by using CS.conj@x() function, records in the result cursor will be retrieved in the same order as each cursor is arranged in the cursor sequence CS.

In most cases, we need more than just the union of the records from each data table one after another. We want to merge them in certain order we need. To serve this purpose, CS.merge@x(x) function can be used. Please note that the records in each cursor of the cursor sequence CS must be already sorted according to expression x when using this function. For example, sort and merge the order data of various products by the sales date:

 | A | B
1 | =file("Order_Wines.txt") |
2 | =file("Order_Electronics.txt") |
3 | =file("Order_Foods.txt") |
4 | =file("Order_Books.txt") |
5 | =[A1:A4].(~.cursor@t()) |
6 | =A5.merge@x(Date) | 300
7 | for | =A6.fetch(B6)
8 |  | break
9 | >A6.close() |

In this case, we intend to have a clear view of the order in which the records are retrieved from the cursor after they are merged in certain order. In order to do this, only the first 300 entries are retrieved. The table sequence in B7 is shown below:

It can be seen that the data are retrieved by Date in a specified order . Once all wine order data of January 1st is retrieved, the retrieval of all electrical appliance order data of the same day will start. Because retrieving data from the cursor is a forward-only operation, the order data in each cursor must have been ordered by the date. After data of multiple cursors are merged using function CS.merge@x() in the specified order, the result data will be retrieved from the grand cursor according to the order of one of the member cursors of sequence CS, which is selected by comparing the current expression values of the data tables. In this way, we can ultimately get the result arranged in the specified order. In data retrieving, each cursor will still traverse the records in each data table for once.

Merging the data in multiple cursors in a certain order simply creates a single grand cursor, and adjusts the orders in which data are retrieved from each cursor, without increasing or decreasing any records.

If the cursor data are not ordered, they must be sorted before merging, as illustrated in the following example:

 | A | B
1 | =file("Order_Wines.txt") |
2 | =file("Order_Electronics.txt") |
3 | =file("Order_Foods.txt") |
4 | =file("Order_Books.txt") |
5 | =[A1:A4].(~.cursor@t().sortx(PID)) |
6 | =A5.merge@x(PID) | 2000
7 | for | =A6.fetch(B6)
8 |  | break
9 | >A6.close() |

Before cursors are merged in proper order by the product number, we must ensure the data in each cursor are ordered against the product number. To do so, A5 uses cs.sortx() function to sort cursors of various products.

Please note that the cursor and the table sequence are sorted differently. Because there is usually a great amount of data in the cursor, they cannot be loaded into the memory all at once for sorting. Therefore, the data retrieval will be performed along with the data sorting. The sorted data will be saved as a temporary data file when they are accumulated to a certain amount. Once all data are retrieved and sorted, all temporary data files will be merged in proper order and returned as the cursor. For more details about sorting cursor data, please see esProc External Memory Sorting: Principle of Sorting.

In B7, the retrieved records are shown below:

It can be seen that the ordered merging is accomplished once the data in each cursor have been sorted.

Next
Previous
Related :
  • 4. Merge and Join Cursor Data


4.2 Aligned joining

When making the statistics, sometimes we need to consolidate the data from multiple cursors, which is similar to joining the data from multiple tables together. If the data in cursor need to point to a normal table sequence, then we can use cs.switch().

What if the data to be joined are all from the cursor? As we know, it is usually impossible to retrieve all data from the cursor. How can we join these data then? In esProc, we can use join@x() function to join the data from multiple cursors together. For example:

 | A
1 | =file("Order_Wines.txt")
2 | =file("Order_Electronics.txt")
3 | =file("Order_Foods.txt")
4 | =file("Order_Books.txt")
5 | =A1.groupx(Type,Date;count(~):Count,sum(round(decimal(Amount),2)):TotalAmount;100)
6 | =A2.groupx(Type,Date;count(~):Count,sum(round(decimal(Amount),2)):TotalAmount;100)
7 | =A3.groupx(Type,Date;count(~):Count,sum(round(decimal(Amount),2)):TotalAmount;100)
8 | =A4.groupx(Type,Date;count(~):Count,sum(round(decimal(Amount),2)):TotalAmount;100)
9 | =join@x1(A5:Wines,Date;A6:Electronics,Date;A7:Foods,Date;A8:Books,Date)
10 | =A9.fetch(25)
11 | >A9.close()

In A5~A8, perform aggregate operations over each kind of product, and return the cursor of respective temporary files (Usage of groupx is explained in detail in esProc External Memory Computing: Principle of Grouping). In A9, the daily sales data of each kind of products will be aligned and joined by the date. A10 fetches the statistical results of the first 25 days, as shown below:

After cursors are joined in alignment, a grand cursor will be returned, from which the fetched result is similar to the result of joining table sequences together, that is, every field in every row is composed of the record. Thus we must note that the joined data take up more memory than those in the normal cases. In addition, since data are composed of records instead of values, please note the writing form of the expression, particularly when re-joining the result cursors used for computation.

The result of aligned joining of cursors can also be reused for filtering or generating data, for example:

9 | =join@x1(A5:Wines,Date;A6:Electronics,Date;A7:Foods,Date;A8:Books,Date)
10 | =A9.select(Foods.TotalAmount>Wines.TotalAmount)
11 | =A10.new(Foods.Date:Date,Foods.TotalAmount:Foods,Wines.TotalAmount:Wines)
12 | =A11.fetch(100)
13 | >A11.close()

From the joined cursor, A10 selects records in which the total amount of food orders is greater than that of wine orders. Then A11 generates the result cursor. A12 returns the first 100 records:

In performing the aligned joining of cursors, we must remember that the data in the cursor cannot be fetched into and store in the memory. They can only be traversed once from the first to the last. Therefore the data in each cursor must be sorted for aligned joining, which is different from handling the database multi-table join and the join() operation for normal table sequences. As shown in the above example, the data in A5~A8 are already sorted by date to ensure a correct computation.

In order to better explain this point, we create a cursor using two in-memory table sequences in which data are not big:

 | A
1 | $select * from CITIES
2 | $select * from STATES
3 | =A1.cursor()
4 | =A2.cursor()
5 | =join@x(A3:City,STATEID;A4:State,STATEID)
6 | =A5.fetch()

The table sequences in A1 and A2 are shown below:

In A5, we will see the aligned joining result:

The cursor data are different to those in the normal table sequence. During the operation of aligned joining, when looking for the New York state corresponding to the New York city, the cursor for the State data has already moved to entry 32 and the records before it are unavailable for later computations. So most cities cannot find their corresponding states. Furthermore, options @1 and @a are not used in function join@x() to specify the left join or full join. Only data of few cities finding out corresponding states are returned.

But if having the city data sorted first, we can have the normal result:

 | A
1 | $select * from CITIES order by STATEID
2 | $select * from STATES
3 | =A1.cursor()
4 | =A2.cursor()
5 | =join@x(A3:City,STATEID;A4:State,STATEID)
6 | =A5.fetch()

In A1, the data are sorted by STATEID:

In A6, we can see the joining result:

The joining of data from multiple cursors doesn’t require the same cursor type. For example:

 | A
1 | =file("PersonnelInfo.txt")
2 | =A1.cursor@t().sortx(State)
3 | $(demo) select STATEID, NAME, ABBR from STATES order by ABBR
4 | =A3.cursor()
5 | =join@x(A1:PersonnelInfo,State;A4:State,ABBR)
6 | =A5.fetch(100)
7 | >A5.close()

A5 joins personnel data with the corresponding state data. What’s important is that the cursor data have been sorted. A6 fetches the first 100 rows of data from A5’s cursor:

In fact the aim of joining data from different cursors is usually to get the related detail information from them. In the above example, there’s no need to join records from the two cursors together if the purpose is only to get the full names of the states where employees come from. Because the state data actually come from a table sequence, cs.switch() can be used to convert a certain field of cursor cs into the corresponding records of the table sequence. There are explanations about the function’s usage in esProc External Memory Computing: Basic Usages of the Cursor. Difference between join@x() and cs.switch() is that the former creates a mutual reference between different cursors, whereas the latter establishes a relationship, through the foreign key, between one or more certain fields of a cursor and a table sequence/sequence, which is regarded as a dimension table in the process. join@x() for joining cursor data requires the data be sorted by the field according to which they are joined. cs.switch() hasn’t such a requirement as the dimension table in use can be referenced directly from the memory.

Sometimes, in a foreign-key relationship, the composite foreign key is needed to reference certain data of a table sequence or a cursor. In this case cs.join() can be used to append fields to the cursor. For example:

 | A
1 | =file("PersonnelInfo.txt")
2 | =A1.cursor@t()
3 | $(demo) select STATEID, NAME,ABBR from STATES
4 | =create(Month,State,GroupID)
5 | >A3.(12.((m=~,A4.insert(0,m,A3.ABBR,A3.ABBR+string(m)))))
6 | =A2.join(month(Birthday):State,A4:Month:State,GroupID:Group)
7 | =A6.fetch(100)
8 | >A6.close()

A4 creates a table sequence based on the month and the states’ abbreviations and A5 inserts detailed data into it. Then A4’s table sequence is as follows:

A6 appends fields to A2’s cursor using cs.join(). Equally, the Month field holding employees’ birthday information and the State field of A2’s cursor are joined with the Month and State field of A4’s cursor. Then fetch GroupID field from A4’s cursor and append it to State field of A2’s cursor after the joining. A7 fetches the first 100 records from A6’s cursor:

With cs.join(), we can join not only the cursor, but the table sequence, with the field in another cursor. Different from join@x(),cs.join() doesn’t require data in the cursor cs be sorted. But if the dimension table in use comes from the cursor, its data must be fetched into the memory all at one. With the requirement of joining two fields of the cursor cs and the dimension table, this example cannot be handled with cs.switch().

Next
Previous
Related :
  • 4. Merge and Join Cursor Data


4.3 Unconditional joining

In the example in the previous section, the statistical data of orders of different products are joined by the dates. In reality the number of rows of data of every kind of order is always the same and so is the data’s order in terms of the date. Thus we don’t have to specify a criterion for sorting. We simply join the data together using pjoin function, without any condition.

 | A
1 | =file("Order_Wines.txt").cursor@t()
2 | =file("Order_Electronics.txt").cursor@t()
3 | =file("Order_Foods.txt").cursor@t()
4 | =file("Order_Books.txt").cursor@t()
5 | =A1.groupx(Type,Date;count(~):Count,sum(round(decimal(Amount),2)):TotalAmount;100)
6 | =A2.groupx(Type,Date;count(~):Count,sum(round(decimal(Amount),2)):TotalAmount;100)
7 | =A3.groupx(Type,Date;count(~):Count,sum(round(decimal(Amount),2)):TotalAmount;100)
8 | =A4.groupx(Type,Date;count(~):Count,sum(round(decimal(Amount),2)):TotalAmount;100)
9 | =[A5:A8].pjoin()
10 | =A9.fetch(100)
11 | >A9.close()

A9 joins cursors of sales data of four products together, without specified order. A10 fetches the first 100 records:

We can see from the result the way fields from different cursors are joined. The result can be rearranged as follows:

9 | =[A5:A8].pjoin()
10 | =A9.new(Date,#3:WineCount,#4:WineAmount,#7:ElecCount,#8:ElecAmount, #11:FoodCount,#12:FoodAmount,#15:BookCount,#16:BookAmount)
11 | =A10.fetch(100)
12 | >A10.close()

A11 fetches the first 100 records:

Next
Previous
Related :
  • 4. Merge and Join Cursor Data


5. Principle of Grouping

After data are imported from a data table, we often need to group them as required, or work out grouping and summarizing result. In esProc, groups function is used to compute the result of data grouping and summarizing; or group function can be used to first group the data, then perform further analysis and computation.

But the case will be different in processing huge data, for the records cannot be loaded to the memory all together and distributed into each group. Other times the number of groups is huge and the grouping and summarizing result cannot even be returned all at once. In these two occasions, the external memory grouping is required.

Next
Previous
Related :
  • Chapter 7 External Memory Computation
  • 5.1 Grouping with cursor by directly specifying group numbers
  • 5.2 Grouping and summarizing result sets of huge data


5.1 Grouping with cursor by directly specifying group numbers

Let’s create a big, simple data table containing employee information, which includes three fields: employee ID, state and birthday. The employee IDs are generated in order and the states are written in abbreviations obtained arbitrarily from the STATES table of demo database; birthdays are the dates selected arbitrarily within 10,000 days before 1994-1-1.The data table will be stored as a binary file for convenience.

 | A | B | C
1 | 1000000 | 1000 | =A1/B1
2 | =file("BirthStateRecord") | =create(ID,Birthday,State) | =demo.query("select ABBR from STATES")
3 | 1994-1-1 | 0 | =C2.(ABBR)
4 | for C1 | for B1 | >B3=B3+1
5 |  |  | =after(A3,-rand(10000))
6 |  |  | =C3(rand(C3.len())+1)
7 |  |  | >B2.insert(0,B3,C5,C6)
8 |  | >A2.export@ab(B2) |
9 |  | >B2.reset() |
10 | =A2.cursor@b() | >A10.skip(50000) | =A10.fetch(1000)
11 | >A10.close() |  |

Altogether 1,000,000 rows of data are generated. The result of reading the 50,001th ~51,000th rows of data with cursor can be seen in C10 as follows:

In the following, we’ll take the generated data file, BirthStateRecord, as an example to explore how to group data in cursor computing by directly specifying group numbers. Because the data of the big data table cannot be loaded all together into the memory, we cannot group them as we do with the data from an ordinary table sequence. To solve this problem, esProc offers cs.groupn(x) function which can distribute the records in cursor cs according to the computed results of expression x into groups with specified serial numbers and return a sequence of cursors. For example:

 | A
1 | =file("BirthStateRecord")
2 | =A1.cursor@b()
3 | =demo.query("select * from STATES order by STATEID")
4 | =A3.(ABBR)
5 | =A2. groupn (A4.pos(State))
6 | =A5(1).fetch()
7 | =A5(22).groups(State;count(~):Count)
8 | >A5.(~.close())

To explain the way in which cs.groupn(x) function performs grouping by using external memory to specify group numbers, Click [image] in the debug area of the toolbar to execute the code step by step until A6. A2 creates a cursor with the binary data file BirthStateRecord. A4 creates a sequence using the states’ abbreviations in STATES table.A5 uses groupn function to group data of the cursor; in this process, we need to find the corresponding serial numbers of the states in A4 and make them the group numbers. During the execution of groupn function in A5, a temporary file will be generated for each group to store the grouping result and a sequence of temporary file cursors will be returned as follows:

While the code in A5 is executed, external files are generated in the directory of temporary files:

In the execution of groupn function, the number of temporary files equals that of the groups of records. We can import data from one of the temporary files in another cellset:

 | A
1 | =file("/temp/tmpdata997124882597229639")
2 | =A1.import@b()

The data A2 imports are as follows:

It can be seen that the data of a temporary file is, in fact, the employee information of a state. Here it is the employee information of the state of Missouri. The name of a temporary file is generated arbitrarily by the esProc program and managed by it.

Click [image] on the toolbar in the previous cellset file and go on with the execution of this cellset. In A8, when all file cursors are closed, the temporary files will be deleted automatically. A6 fetches from the first file cursor the employee information of the state of Alabama as follows:

A7 works out the grouping and summarizing result of the 22nd group using groups function, that is, the number of employees from the state of Michigan:

Actually A6 and A7 fetch data from two groups respectively. When the data of a file cursor have been fetched, the corresponding temporary file will be deleted automatically.

It is thus clear that a sequence consisting of temporary file cursors will be returned in grouping records of cursors using directly specified group numbers. Each file cursor contains the records of a group which can be further processed.

Next
Previous
Related :
  • 5. Principle of Grouping


5.2 Grouping and summarizing result sets of huge data

When grouping data of cursors, most of the time we needn’t to get the detailed data of each group. What we only need is the grouping and summarizing result. To get the number of employees of each state from BirthStateRecord, for example, we use groups function to compute the grouping and summarizing result:

 | A
1 | =file("BirthStateRecord")
2 | =A1.cursor@b()
3 | =A2.groups(State;count(~):Count)
4 | >A2.close()

Thus we can get the result in A3:

Here we notice that the groups function for grouping and summarizing will return a table sequence as the result after the computation is completed. In processing massive data, sometimes they need to be divided into a lot of groups and the result set of grouping and summarizing is too big to be returned all at once. This includes the following cases: the telecom company makes statistics of each customer’s bill; online shopping malls make statistics by categories about the sales of each kind of commodity, and the like. In these cases, the use of groups function may result in a memory overflow. We can use groupx(x:F,…;y:F,…;n) function instead to perform grouping and summarizing by making use of external memory. In the function, n represents the number of rows in buffer area. For example:

 | A
1 | =file("BirthStateRecord")
2 | =A1.cursor@b()
3 | =A2.groupx(Birthday;count(~):Count;1000)
4 | =A3.fetch(1000)
5 | >A2.close()

Still, the code is executed step by step until A4. In A3, groupx function uses external memory to perform grouping and summarizing. In cursor computing, groupx function is used in the operations of both grouping and summarizing with external memory and grouping by directly specified group numbers. The difference of the two operations lies in the parameters. A3 performs grouping by employees’ birthdays, then summate the number of employees born on each of the dates. During the computation, the number of rows in buffer area is set as 1,000. The result returned by A3 is a cursor as follows:

After the code in A3 is executed, external files will be generated in the directory of temporary files:

The data of one of the temporary files can be imported:

 | A
1 | =file("/temp/tmpdata1786998866046507792")
2 | =A1.import@b()
3 | =A2.count()

The data A2 imports are as follows:

The data of A3 is as follows:

It can be seen that each temporary file is the result of grouping and summarizing a part of the data according to employees’ birthdays. A larger cursor composed of all temporary files will be merged and returned by esProc. When the temporary files are generated, esProc will select a group number suitable for computing, so the rows of data in the temporary files will be a little more than the number of rows we set in buffer area. Special attention is needed in this point.

Go on with the execution of the cellset in the previous cellset file. When cursors are closed in A5, the temporary files will be automatically deleted. A4 fetches the first 1,000 birthdays from the cursor generated in A3 and counts the number of employees born on each birth date, as shown below:

Next
Previous
Related :
  • 5. Principle of Grouping


6. Principle of Sorting

It is common to sort records in the table during data analysis and computing. In esProc, sort function is used to sort data in the sequence or the table sequence. External memory sorting is required when data being sorted are massive and cannot be loaded into memory all together, for the ordinary sorting method cannot handle this situation.

Next
Previous
Related :
  • Chapter 7 External Memory Computation
  • 6.1 Massive data sorting with external memory
  • 6.2 Applications of external memory sorting


6.1 Massive data sorting with external memory

In data statistics, cursors are usually used to fetch massive data. This applies in esProc, which also processes big data with the cursor. In esProc, the function of a cursor, which reads one or more records each time according to the position(s) marked by it and won’t return all data all at once, is similar to that in a database stored procedure.

A cursor can only fetch part of the data every time, thus operations like sorting and grouping all data in the cursors cannot be executed directly. esProc uses external memory to handle these operations on massive data. Each time it reads some of the data for computing and records the result temporarily in the external memory. Later it will merge all the sub-results into a cursor and works out the final result.

Let’s create a data table with huge data in which the dates and 8-digit phone numbers (the first and the last digit should not be zero) are generated arbitrarily. The data table will be stored in the format of a binary file for convenience.

 | A | B | C
1 | 100000a | 1000 | =A1/B1
2 | =file("PhoneRecord") | =create(ID,Date,PhoneNum) |
3 | =date(now()) | 0 |
4 | for C1 | for B1 | >B3=B3+1
5 |  |  | =after(A3,-rand(1000))
6 |  |  | =rand(90000000)+10000000
7 |  |  | >B2.insert(0,B3,C5,C6)
8 |  | >A2.export@ab(B2) |
9 |  | >B2.reset() |
10 | =A2.cursor@b() | >A10.skip(50000) | =A10.fetch(1000)
11 | >A10.close() |  |

Altogether 100,000 rows of data are generated. Read data from the 50,001th row to the 51,000th row using the cursor. The result can be seen in C10 as follows:

We’ll take PhoneRecord, the generated data file, as an example to explore how to perform external sorting in esProc.

Using cs.sortx(x…;n) function in external memory sorting, we can sort the data in cursor cs in ascending order according to the computed result of expression x…and set the number of rows in buffer area by defining n to determine the number of records fetched each time for generating a temporary file. For example:

 | A
1 | =file("PhoneRecord")
2 | =A1.cursor@b()
3 | =A2.sortx(PhoneNum;20000)
4 | =A3.fetch(1000)
5 | >A3.close()

In order to understand how the external memory is used in esProc to sort data, we click [image] in the debug area on the toolbar to execute the code step by step until A5. A2 uses the binary data file PhoneRecord to create cursors. A3 uses sortx function to sort data of the cursors. The sorting result, in fact, will be a large cursor merged orderly by many temporary cursor files. The result of A3 is as follows:

A4 fetches the first 1,000 records from this result cursor as follows:

While the code in A3 is executed, external files, which are also called as temporary files, are generated in the directory of temporary files:

Because the number of rows in buffer area was set as 20,000 while using sortx function, five temporary files have been generated for the 100,000 records in the cursor. The data of one of them will be imported:

 | A
1 | =file("/temp/tmpdata984048609330992507")
2 | =A1.import@b()
3 | =A2.count()

A2 imports the data as follows:

A3 works out the number of rows in this temporary file as follows:

By comparing the data in A2 with the final sorting result previously obtained, we can see that the former is, actually, the result obtained by sorting a part of the data. This indicates that each temporary file is the sorting result of some data fetched according to the number of rows in buffer area.

Then go on to execute the previous cellset file. We may find that the temporary files will be deleted automatically when the cursor is closed.

sortx function can also be used to sort multiple fields. For example:

 | A
1 | =file("PhoneRecord")
2 | =A1.cursor@b()
3 | =A2.sortx(Date,-PhoneNum;20000)
4 | =A3.fetch(1000)
5 | >A3.close()

A3 sorts data by Date and –PhoneNum , meaning sorting data by date in ascending order, and then sorting those of the same date by phone number in descending order. In programming, it is common to precede an expression for sorting data in descending order by a negative sign. A4 fetches the first 1,000 records from the result as follows:

Next
Previous
Related :
  • 6. Principle of Sorting


6.2 Applications of external memory sorting

In fact, from the operation of external memory sorting we can see one of the uses of the cursor-style sorting, that is, the sorted data can be used in orderly merging. The operation of orderly merging gets data from multiple cursors and computes the sorting expression. Records will be fetched from the cursor that currently has the smallest (or biggest) result. Apparently, this method can only be used when the data of every cursor are properly ordered. In addition, joining records of cursors in alignment with join@x() function also requires that data in every cursor be sorted. About the usage of join@x() function, please refer to esProc External Memory Sorting: Merge and Join Cursor Data.

When the data of a cursor are properly ordered, it can be specified that we fetch data continuously from the cursor until the expression is changed. For example:

 | A
1 | =file("PhoneRecord")
2 | =A1.cursor@b()
3 | =A2.sortx(Date,-PhoneNum;20000)
4 | =A3.fetch(;Date)
5 | >3.(A3.skip(;Date))
6 | =A3.fetch(;Date)
7 | >A3.close()

A4 fetches data from the cursor in A3 until the Date is changed, meaning the data of the first day will be fetched; A5 skips data of consecutive three days; A6 fetches the data of the fifth day. The results of A4 and A6 are as follows:

Next
Previous
Related :
  • 6. Principle of Sorting


7. Text Files

Sources of data used for analysis usually fall into two categories: the database source and the file source. Compared with the database data, the file data are simple to deploy and publish. The problem is that, since the file data generally need to be used as a whole and thus need to be loaded into the memory all at once, they are not so easy to handle when being big. esProc offers solution to this problem by fetching them with the cursor. This makes the use of file data more convenient.

This article will explore how to manipulate file data in esProc through the case of text data - the most common external file data.

Next
Previous
Related :
  • Chapter 7 External Memory Computation
  • 7.1 Basic usage
  • 7.2 Batch processing of big data
  • 7.3 Access big text files with the cursor


7.1 Basic usage

file function and import function can be used to read data from text files. The text file employee.txt holds information of 500 employees, as shown below:

EID NAME SURNAME GENDER STATE BIRTHDAY HIREDATE DEPT SALARY

1 Rebecca Moore F California 1974-11-20 2005-03-11 R&D 7000

2 Ashley Wilson F New York 1980-07-19 2008-03-16 Finance 11000

3 Rachel Johnson F New Mexico 1970-12-17 2010-12-01 Sales 9000

4 Emily Smith F Texas 1985-03-07 2006-08-15 HR 7000

5 Ashley Smith F Texas 1975-05-13 2004-07-30 R&D 16000

6 Matthew Johnson M California 1984-07-07 2005-07-07 Sales 11000

7 Alexis Smith F Illinois 1972-08-16 2002-08-16 Sales 9000

8 Megan Wilson F California 1979-04-19 2004-04-19 Marketing 11000

9 Victoria Davis F Texas 1983-12-07 2009-12-07 HR 3000

…

esProc sets rules for the format of the text data used in the data table: Records must be separated by the carriage return and fields should be separated by the tab. Text data can be imported using simple code:

 | A
1 | =file("employee.txt")
2 | =A1 .import@t()
3 | =A1 .import()

In esProc, import function is used to import the file data as the table sequence. In the above cellset, A1 creates a file object. If the path hasn’t been specified in the file name, it can be found in the main path on the settings page. Click on Tool>Options on the menu bar to view or set the main path on the Environment page of Option window:

A2 imports the file as a table sequence. The use of @t option in the import function will import the text file’s first row as the column names. For a clearer comparison, the option is omitted in A3’s data import. After the code is executed, A2’s table sequence is as follows:

Not all files necessarily use tab as the field separator. The CSV file, for instance, uses commas. We can specify a separator for a file when using import. Thus A2’s code can be modified as =A1 .import@t(;",").

A3 generates a table sequence as follows:

It can be seen that, when @t option is omitted, the table sequence’s field names will be automatically generated and named _1,_2,_3 according to their positions respectively.

When the text data are being imported, their data type will be automatically parsed according to the data type of the first row. As the first row of employee.txt holds field names, all data of A3’s fields will be parsed into strings, like column _1 and column _9 in the result above, which are left-justified. While the type of data in A2’s EID field and SALARY field is integer, and the data are right-justified.

We can import only some of the fields to generate a table sequence. For example:

 | A
1 | =file("employee.txt")
2 | =A1 .import@t(EID,NAME,BIRTHDAY,SALARY)
3 | =A1 .import(#1,#2,#6,#9)

The use of @t option in A2 means that we can directly use names to specify fields to be imported. Result of A2 is as follows:

Without @t option, A3 specifies the fields with their serial numbers in the format #i. The result table sequence is as follows:

As mentioned above, during text data import, data type will be automatically parsed according to that of the first row. If a particular data type is wanted, it can be defined for the specified fields to be imported. For example:

 | A
1 | =file("employee.txt")
2 | =A1 .import@t(EID:string,NAME,BIRTHDAY:date,SALARY:int)
3 | =A1 .import(#1:string,#2,#6:date,#9:int)

The following is the set of settings used in esProc to specify the data type for the field:

Settings | string | int | float | long | decimal | date | datetime | time | bool
Data types | String | Integer | Floating point number | Long integer | Big decimal | Date | Date/time | Time | Boolean

To learn more about the data type, please refer to esProc Getting Started: Basic Data Types.

A2 specifies data types for EID field, BIRTHDAY field and SALARY field – string, date and integer respectively. When executed, A2’s result table sequence is as follows:

A3 specifies the same data types for the fields as in A2. The result table sequence is as follows:

It can be seen that if certain data of a field cannot be parsed into the specified data type, they will be parsed automatically, like those of the title row, which have been automatically parsed into strings.

About the subsequent computations after data are imported, please see the article esProc Program: Operations on Table Sequences and Record Sequences.

In esProc, not only text data can be imported as a table sequence, in-memory data can also be exported to a text file. The to-be-exported data are usually those of the table sequence or the record sequence, or sometimes, members of the sequence. export function is used to export data to the file. For example:

 | A
1 | =file("employee1.txt")
2 | =create(EID,NAME).record([1,"Frank",2,"Harry"])
3 | >A1.export(A2)

A1 creates a file object, employee1.txt, to save data. But at this point the file data are still stored in the report’s main path.

A2 creates a table sequence and inserts two records into it:

A3 exports the data of the table sequence to file employee1.txt:

We can use @t option to export the column names. And we can specify certain fields to be exported. By modifying A3’s code into >A1.export@t(A2,NAME), the exported data in employee1.txt are as follows:

As in import, tab is used as the default field separator in export. Of course we can also specify the field separator used in the function.

Next
Previous
Related :
  • 7. Text Files


7.2 Batch processing of big data

When a file holds big data, to load it all together to the memory may cause the memory overflow. In this case, only some of the data ought to be imported each time. import function can be used to realize this by specifying the beginning byte and the ending byte. For example:

 | A
1 | =file("PersonnelInfo.txt")
2 | =A1 .import@t(;,10000:20000)
3 | =A1 .import@t(;,20000:30000)

The text file PersonnelInfo.txt holds the personnel information. A2 specifies the range of data import - from the 10,000th byte to the 20,000th byte. Since only the complete record has meaning in the result table sequence, esProc will adjust the positions of beginning byte and ending byte to ensure they are respectively in a record’s starting point and another record’s ending point. The imported data in A2 are as follows:

As we can see, @t option can still be used to import field names while importing data through specifying the range.

A3 continues to import data from the 20,000th byte to 30,000th byte. Result is as follows:

The two results show that the personnel information in A2 and A3 is continuous. esProc specifies the range for data import in a uniform way to ensure the continuity, integrity and uniqueness of the imported data. A text file with big data can thus be imported and processed in batches to avoid the memory overflow.

In addition to directly specifying beginning and ending positions for data import, @z option can be used in import to import data segmentally. For example:

 | A
1 | =file("PersonnelInfo.txt")
2 | =A1 .import@tz(;,100:500)
3 | =A1 .import@tz(;,101:500)

A2 and A3 use both @t and @z in the import function to import column names as well as the data in segments. That means we can use multiple options simultaneously in a function. A2 divides data into 500 segments and imports the 100th segment. Its result is as follows:

A3 continues to import the 101th segment. Its result is as follows:

It can be seen that, during the batch import, esProc is also able to ensure that complete records are imported by setting appropriate positions for beginning and ending points, and at the same time, the continuity, integrity and uniqueness of the imported data.

Note: With import function in which the file is imported in batches by specifying the beginning and ending bytes or dividing it into segments, the number of imported rows of data is different for each batch because of the variation of the number of bytes each record holds. This also explains why a specified row of data cannot be got directly. To locate the record in a specified positon, all records before this position need to be traversed. This is not efficient at all. Yet, the external file cursor can be used to access the target record precisely, which will be explained in the next section.

The export of big data to a text file can also be hindered by the limited memory. Because data cannot be loaded into the memory all at once, they cannot be exported to a certain file by simply using export. But similarly, they can be exported in batches. Use @a option in export function to append data to the existing data at each batch of data export. For example:

 | A | B
1 | =file("PersonnelInfo.txt") | =file("PersonnelInfo1.txt")
2 | for 500 | =A1 .import@tz(ID,Name,State;,A2:500)
3 |  | >B1.export@at(B2)

The loop in A2 imports data from PersonnelInfo.txt in 500 segments and specifies certain fields to be written to a new file PersonnelInfo1.txt. @at options are used in export to export both column names and data by appending each batch export to the previous data. When the code is executed, PersonnelInfo1.txt gets data as follows:

But without @a option, the existing data in the file will be removed at each time of export.

Next
Previous
Related :
  • 7. Text Files


7.3 Access big text files with the cursor

A more convenient tool to process text files containing big data is the cursor. Cursor functions can be called conveniently after the external file cursor is generated based on the text file. For example:

 | A
1 | =file("PersonnelInfo.txt")
2 | =A1.cursor@t(ID,Name,Gender,State)
3 | >A2.skip(10000)
4 | =A2.fetch(1000)
5 | =A2.fetch(1000)
6 | >A2.close()

A2 creates a file cursor using cursor function, in which @t option is used to make the file’s first row the column names. A3 skips the first 10,000 rows. A4 and A5 respectively fetch 1,000 rows from A2’s cursor:

Data can be fetched conveniently and quickly by their position from the cursor. As with f.import() in section one, fields can be specified for creating a cursor and the data type of a certain field can be defined with f.cursor().

With the external file cursor, we are able to handle various operations easily. About its usages, please refer to esProc External Memory Computing: Basic Usages of the Cursor and other articles on esProc external memory computing.

Also through the cursor, data can be exported to a file. For example:

 | A
1 | =file("employee2.txt")
2 | =demo.cursor("select * from EMPLOYEE")
3 | >A1.export@t(A2,EID,NAME+" "+SURNAME:FullName,GENDER,STATE)

A2 creates a database cursor. A3 gets the expected result by computing the cursor data and then export it to the file. After the code is executed, employee2.txt holds data as follows:

Exporting data through the cursor is convenient and the code is concise. To retain the existing data during the data export, use @a option to append each batch export to the previous data.

Besides the text file, esProc also supports the binary file. The latter takes up less memory space and makes the computation more efficient. For its usages, please see article esProc External Memory Computing: Binary Files.

Next
Previous
Related :
  • 7. Text Files


8. Binary Files

Two kinds of data file - the normal text file and the binary file – are most used in esProc, of which the binary file adopts compressed encoding of low CPU consumption, meaning that it takes less space than an uncompressed text file, and the data reading efficiency will be higher. Thus, we can conclude that the binary file is a better choice when you need to use data files.

In esProc External Memory Computing: Text Files, we learned how to manipulate the text data. This article will deal with the use of binary data.

Next
Previous
Related :
  • Chapter 7 External Memory Computation
  • 8.1 Comparison between binary files and text files
  • 8.2 Fetch data segmentally with the cursor
  • 8.3 Columnar storage of binary file


8.1 Comparison between binary files and text files

Their usages are almost the same, except that @b option cannot be omitted in functions like import, export and cursor used with the binary file. Let’s look at this with an example:

The above two files - PersonnelInfo and PersonnelInfo.txt – hold the same personnel information, respectively stored in binary format and text format. Each file includes 100,000 records of 6 fields. As can be seen, in the hard disk, size of a binary file is less than that of a text file.

Then, let’s check out how data are retrieved from these two kinds of files:

 | A
1 | =now()
2 | =file("PersonnelInfo")
3 | =A2.cursor@b()
4 | =A3.groups(State;count(~):Count)
5 | =interval@ms(A1,now())

In the cellset, A3 creates a cursor using the binary file. A5 computes the time (millisecond) consumed for performing the grouping and aggregate operation over the binary data. The result is shown below:

 | A
1 | =now()
2 | =file("PersonnelInfo.txt")
3 | =A2.cursor@t()
4 | =A3.groups(State;count(~):Count)
5 | =interval@ms(A1,now())

In the same way, the text data are used to perform the same operation. The computing time is shown below:

In the above two cellsets, the binary file and the text file are respectively used to perform the same grouping and aggregate computation to count the number of employees of each state, and then compute the time (millisecond) consumed for doing this in A5. As can be seen from the result, the binary data are retrieved at a speed significantly higher than the speed at which the text data are retrieved.

In a word, it is more convenient to store data in a binary file in esProc.

Next
Previous
Related :
  • 8. Binary Files


8.2 Fetch data segmentally with the cursor

During big data computing, it is a common solution to first split the data into segments and then compute them respectively. When using the file data, both the text file and the binary file can be imported segmentally using @z option. The article – esProc External Memory Computing: Text Files – explains how to import text data segmentally for creating a table sequence. Similarly, data can be fetched through the cursor. For example:

 | A
1 | =file("PersonnelInfo")
2 | =A1.cursor@bz(;,1:5)
3 | =A2.fetch()
4 | =A1.cursor@bz(;,2:5)
5 | =A4.fetch()

In both A2 and A4, in generating the cursor, @z option is used to divide the data in the cursor into 5 parts based on parameters. The 1st part is returned by A2, and the 2nd is returned by A4. A3 and A5 fetch all data, as shown below:

In the file PersonnelInfo, there are 100,000 records in total. We can see that they are roughly divided into 5 parts based on approximate sizes, instead of being divided precisely according to the number of records. When retrieving data by segment, esProc will adjust the range of data retrieval automatically so as to ensure the data integrity, as with the 1st and the 2nd segment of data. This can ensure the continuity of the data and prevent duplicate data during the process of computing.

The usage of @z option with the text file is exactly the same as that with the binary file, except that @b option will be omitted for the former.

While retrieving data by segment from a binary file, the file can be specifically divided into multiple segments corresponding to different groups of data. For detailed information in this respect, see esProc External Memory Computing: Group Cursor. This is a function specially applied to binary files, so it cannot be used for text files.

Next
Previous
Related :
  • 8. Binary Files


8.3 Columnar storage of binary file

If an access-intensive big data table contains multiple fields, then you can use the columnar storage of binary file to store the data table into multiple files by fields. In this way, you can select the data file of the desired fields to generate the cursor, so as not to read the unnecessary data. For example:

 | A
1 | =file("PersonnelInfo")
2 | =file("P_ID")
3 | =file("P_Name")
4 | =file("P_Gender")
5 | =file("P_Birthday")
6 | =file("P_City")
7 | =file("P_State")
8 | =A1.cursor@b()
9 | >[A2:A7].export(A8)
10 | =[A2,A3,A7,A4].cursor()
11 | =A10.fetch(100)
12 | >A10.close()

In A9, the data in cursor are saved as multiple binary files in a columnar format, and each file only stores one column of data. In A10, according to the desired fields, select the corresponding files to build the cursor jointly, which can be used conveniently as the normal cursor, while keeping the system resources from consuming by the extra data. From A11, retrieve the first100 records, as shown below:

However, because retrieving data by block is based on dividing the data volume of the file itself instead of the number of records, the consistency cannot be ensured for the multiple files in a columnar format. Therefore, the access by segment is not allowed regarding the file cursor composed of multiple columnar files.

If accessing multiple files simultaneously on the mechanical disks and the file buffer is small, then the data retrieval efficiency will be reduced greatly due to the frequent access to different files. Thus the file buffer must be set to a greater one, like 16M. However, please note that the memory overflow may be incurred if the file buffer is over-sized or there are too many parallel threads. If using the SSD instead of the mechanical hard disk, then you will not encounter the great decrease in the data retrieval speed. Just set the default file buffer settings, i.e.64K/65536.

Next
Previous
Related :
  • 8. Binary Files


9. Group Cursor

During big data computing, besides data traversal and grouping and aggregate operations, sometimes we need to retrieve one group of data each time to analyze. For example, analyze sales data by dates, plot sales curve for each product, and study the purchase habit of each client.

Next
Previous
Related :
  • Chapter 7 External Memory Computation
  • 9.1 Fetch data by groups according to the expression
  • 9.2 Fetch data from file cursor segmentally


9.1 Fetch data by groups according to the expression

In esProc, we can use cs.fetch(;x) function or cs.skip(;x) function to get or skip records till the value of expression x is changed. By doing so, a group of consecutive data can be obtained. For example, retrieve the data of one product each time and examine the sales:

 | A | B
1 | =file("Order_Wines.txt") |
2 | =file("Order_Electronics.txt") |
3 | =file("Order_Foods.txt") |
4 | =file("Order_Books.txt") |
5 | =[A1:A4].(~.cursor@t().sortx(PID)) |
6 | =A5.merge@x(PID) |
7 | for 19 | =A6.skip(;PID)
8 | =A6.fetch(;PID) |
9 | >A6.close() |

The cursor in A6 contains the sales data of four kinds of products, which have been sorted by PID. From B7, records of the 20th piece of commodity have been retrieved:

Note that the traversal of cursor data in esProc is executed just once in a one-way street. Thus the data in cursor must be in order when retrieving a single group of records each time as necessary.

Next
Previous
Related :
  • 9. Group Cursor


9.2 Fetch data from file cursor segmentally

Both articles – esProc External Memory Computing: Text Files and esProc External Memory Computing: Binary Files – deal with the usage of cs.fetch@z(). As we know, @z option can be used to retrieve file or fetch data from cursor segmentally (or by block). However, when fetching data segmentally, it is esProc that determine how the data are divided and sometimes problems may arise.

First, let’s create a text file to save the above used data which are already sorted by PID as a new binary file Order_Products:

 | A
1 | =file("Order_Wines.txt")
2 | =file("Order_Electronics.txt")
3 | =file("Order_Foods.txt")
4 | =file("Order_Books.txt")
5 | =[A1:A4].(~.cursor@t().sortx(PID))
6 | =A5.merge@x(PID)
7 | =file("Order_Products")
8 | >A7.export@b(A6)

In the later computation, if retrieving data by segment, we will get the situation given below:

 | A
1 | =file("Order_Products")
2 | =A1.cursor@bz(;,1:100)
3 | =A2.fetch()
4 | =A1.cursor@bz(;,2:100)
5 | =A4.fetch()

After all data are divided into 100 segments, retrieve data from the 1st segment in A3, and from the 2nd segment in A5, as shown below:

Now a problem appears. For the product with the number B1445, its sales records appear in both groups. If we need to perform the aggregate operation after each time data are fetched, then duplicate product numbers will appear in the return result. Thus re-aggregation will be necessary to get the final result. Such segmental computation is quite common for the parallel computation over big data, but the above problem will make the computation more complicated. To avoid it, we should store data through the way of segmentation by groups.

When storing the binary data with the cursor, simply use the @g option. In this case, the data written into the cursor will be segmented by groups. By doing so, the data from a same group is sure to be fully fetched all at once when retrieving data segmentally. For example:

 | A
1 | =file("Order_Wines.txt")
2 | =file("Order_Electronics.txt")
3 | =file("Order_Foods.txt")
4 | =file("Order_Books.txt")
5 | =[A1:A4].(~.cursor@t().sortx(PID))
6 | =A5.merge@x(PID)
7 | =file("Order_Products_G")
8 | >A7.export@bg(A6;PID)

Save the data that have been sorted by the product numbers as a binary file Order_Products_G and segment it by groups according to PID. This is slightly different to the method we adopted previously to write the data to the file of Order_Products. Please note that segmental storage is only valid for the binary file. The field by which the data are written into a binary file through the way of segmentation by groups is determined by the parameter used in export@bg() function. It is specified in A8 that A6’s cursor is segmented by groups according to PID, for instance.

At this point, result will be different if we fetch data by segments:

 | A
1 | =file("Order_Products_G")
2 | =A1.cursor@bz(;,1:100)
3 | =A2.fetch()
4 | =A1.cursor@bz(;,2:100)
5 | =A4.fetch()

The binary data stored like this is fetched using the cursor in the same way as that of other binary data being fetched with the cursor. But their data fetching results are different as a result of segmenting data by groups during file saving in the former case. Data fetched in A3 and A5 are as follows:

Therefore records of the product whose ID is B1445 will all be fetched for segment one and the data for segment two will be fetched from the next product. It can be seen that, after we specify that data are written into the binary file through the way of segmentation by groups, data of the same group will be put in the same segment for being fetched from the cursor. With segmentation by group, the integrity of the data in each group is guaranteed, and the segmental computation over big data becomes simpler and easier.

Next
Previous
Related :
  • 9. Group Cursor


10. Cross-cellset Cursor

In the article esProc External Memory Computing: Concept of Cursor, we only touched on the basic usage of the cross-cellset cursor. Here we’ll delve into more issues about it.

Next
Previous
Related :
  • Chapter 7 External Memory Computation
  • 10.1 Basic usage
  • 10.2 Cellset that returns multiple results
  • 10.3 Cellset that returns data by loop
  • 10.4 Applications


10.1 Basic usage

The cross-cellset cursor is typically used to handle big data analysis and computing, but it doesn’t impose a minimum limit on data volume. First we’ll explain its usage using relatively small data volume. Cellset D:\files\FindEmployees1.dfx is as follows:

 | A
1 | $(demo) select * from EMPLOYEE where DEPT = ?;arg1
2 | >output("before result")
3 | result A1

The program in the cellset is simple. It retrieves data of employees in a certain department and returns a table sequence in A3 using a result statement. The arg1 used in A1 is a cellset parameter specifying the department name. The parameter can be set by clicking Program>Parameter on the menu bar:

The article esProc Advanced Coding: Subroutines discusses the method of executing program in another cellset file via call function. The cross-cellset cursor has a similar usage, except that it will return a cursor using pcursor. It calls a cellset file in the master cellset as follows:

 | A
1 | =pcursor("FindEmployees1.dfx";"Sales")
2 | >output("before fetch")
3 | =A1.fetch()

While creating a cross-cellset cursor using pcursor, directly specify the name of the to-be-invoked dfx file. Enter the parameter, if needed, after the file and use a semicolon to separate them. After the program is executed, the result of A1 is a cross-cellset cursor that is used in the same way as the ordinary cursor:

Using fetch, A3 returns a table sequence of employees of sales department:

Both cellsets use output function that outputs characters to the console during the execution, thus we can clearly see the execution order in the cellset. Click Tool>Console on the menu bar to see the output information:

It can be seen that the program in a specified cellset will only be invoked to kick off the cross-cellset computation when fetch starts to fetch data.

A1 calls the dfx file without using its full path. But this requires that the file have been placed into esProc’s main path or searching path, whose configuration is the same as that in the cross-cellset call by call function. We’ll reemphasize it by explaining the configuration process again.

Click Tool>Options on the menu bar to configure the main path and the searching path on the Environment page, as shown below:

The dfx file can be invoked only by its name without the necessity of writing the full path, regardless of being placed in the main path, or under any directory of the searching path in the above configuration.

In the case of being integrated, the main path and searching path of the dfx file need to be configured in the configuration file config.xml:

<dfxPathList>

<dfxPath>E:\tools\raqsoft\esProc\demo\Case\Structural;D:\files\txt;D:\files</dfxPath>

</dfxPathList>

<mainPath>D:\files\demo</mainPath>

Next
Previous
Related :
  • 10. Cross-cellset Cursor


10.2 Cellset that returns multiple results

The cellset file to be invoked through the cross-cellset cursor could return multiple results, as D:\files\FindEmployees2.dfx shown below:

 | A
1 | $(demo) select * from EMPLOYEE where DEPT = ?;arg1
2 | result A1.select(GENER=="F")
3 | result A1.select(GENER=="M")

In this cellset, the program queries the database to get data of employees in a certain department, and returns two record sequences, composed respectively of data of female employees and of those of male ones, in A2 and A3 separately using two result statements. It sets the same cellset parameter as the previous section did.

Then the cross-cellset cursor is called in the master cellset as follows:

 | A
1 | =pcursor("FindEmployees2.dfx")
2 | =A1.fetch()

While creating a cross-cellset cursor by invoking pcursor, use the default value of the cellset parameter if it is omitted. That is, in this instance, to find data of employees in HR department. After execution A2 returns result as follows:

The first half of the result table sequence is data of female employees in HR department and the second half is those of male ones in the department. We can see that the cross-cellset cursor will join in due order the multiple results that are returned via multiple result statements by the cellset being invoked.

During the invocation for the cellset that returns multiple results, all these results must have the same data structure because they will be returned altogether by the cross-cellset cursor. Otherwise errors will occur.

In order to see the detailed execution process, use output statements in the cellset to be invoked, as shown in the following cellset D:\files\FindEmployees3.dfx:

 | A
1 | $(demo) select * from EMPLOYEE where DEPT = ?;arg1
2 | >output("return GENDER F")
3 | result A1.select(GENER=="F")
4 | >output("return GENDER M")
5 | result A1.select(GENER=="M")

In this cellset, the program queries the database to get data of employees in a certain department, and returns two record sequences, composed respectively of data of female employees and of those of male ones, in A3 and A5 separately using two result statements. Here the same cellset parameter is used.

The cross-cellset cursor is called as follows:

 | A | B
1 | =pcursor("FindEmployees3.dfx") | []
2 | for A1,5 | >output("fetch"+string(#A2))
3 |  | >B1=B1|A2

In our examples, the cross-cellset cursor only contains a very small volume of data, but this is sufficient for explaining the cursor invocation. A2 loops through data in the cursor and fetches five records each time. B2 outputs the execution information during loops. B3 concatenates the records returned from the cursor into B1’s record sequence. After the program is executed, B1gets the same result as the previous instance did – records of female and male employees in HR department. The output information received by the console is as follows:

As can be seen from the output information, during the invocation of FindEmployees3.dfx by the master cellset, the program will be executed step by step according to the specified number of records to be returned. That’s why it is only after all the records of female employees in A3 have been returned that the fetching of records of male employees begins.

Next
Previous
Related :
  • 10. Cross-cellset Cursor


10.3 Cellset that returns data by loop

When a large volume of data need to be returned from a cross-cellset cursor, result statements can be used to return them through multiple steps as multiple results that will be concatenated in the master cellset. But a more commonly used way to handle this is to return data from the cursor by loop in multiple results in the cellset to be invoked. These results will be automatically concatenated as they are being retrieved in the master cellset. D:\files\Order1.dfx is such a cellset file for invocation:

 | A | B
1 | =file("Order_Books.txt") | =A1.cursor@t()
2 | for B1,5000 | result A2

A2 fetches data from the cursor by loop and B2 returns each batch of data as a table sequence using the result statement. This is not the way used in the previous example that the whole table sequence or record sequence is returned at a time.

In the master cellset, the cross-cellset cursor works in completely the same way as those in the above:

 | A
1 | =pcursor("Order1.dfx")
2 | =A1.fetch(1000)
3 | >A1.close()

The cross-cellset cursor returns huge data. A2 only fetches the first 1,000 records as follows:

Since not all data in the cross-cellset cursor have been fetched out, it needs to be closed deliberately in A3. Once the cross-cellset cursor closes, the cursors in the cellset being invoked will close simultaneously.

Cursor data can also be returned as multiple cursors using multiple result statements. In this case data in every cursor will be concatenated together and fetched as a whole. This is the same as the case in which multiple record sequences are returned. So the data in every cursor need to have the same data structure.

Next
Previous
Related :
  • 10. Cross-cellset Cursor


10.4 Applications

By invoking a cellset file, the cross-cellset cursor converts the file’s return result into a cursor. Thus the cellset file can be used to deal with some complicated data processing tasks. During the big data analysis and processing, for example, text data are often used as the source data. But as they are not as orderly as the database data, they need to be rearranged by a routine. In this case the cross-cellset cursor is used to call this separate routine.

D:\files\EmployeeMul.txt contains employee data, as shown below:

In this text file, data of each employee hold information of ID, name, gender, entry date, birthday and city, etc. in three lines. It is common for a text file to save a record with many fields in multiple lines. The task is to find the data of Philadelphian employees from the file.

The text file cannot be used directly for computation, because esProc requires that each line of data in the file be regarded as a record when directly importing text data as the table sequence or the cursor. Data in the file EmployeeMul.txt need to be reorganized before they are used for analysis and computing. The program will become more intuitive if we choose to use a cross-cellset cursor. First reorganize the data in mergeRecord.dfx:

 | A | B | C | D
1 | =file("EmployeeMul.txt").cursor() |  |  |
2 | for A1,5000*3 | =A2. step(3,1) | =A2. step(3,2) | =A2. step(3,3)
3 |  | =pjoin(B2:L1,C2:L2,D2:L3) | =B3.new(L1.#1:ID,L1.#2:Name,L1.#3:Gender,L1.#4:EntryDate,L2.#1:Birthday,L2.#2:City,L2.#3:State,L2.#4:Dept,L3.#1:SuperiorID,L3.#2:MailAdress) |
4 |  | result C3 |  |

A1 imports the text file as a cursor. A2 fetches 5000*3 records each time and joins the fetched data together within its statement block. For every three records, B2, C2 and D2 respectively fetch the first one, the second one and the third one to create a record sequence. The table sequences they create with the last loop are separately shown below:

Only the first two columns of data in D2 are valid. B3 joins the three table sequences together using pjoin function:

After all this C3 can generate the desirable records using those source data:

Then the computation required by the task will be easily handled by invoking mergeRecord.dfx through the cross-cellset cursor.

 | A
1 | =pcursor("mergeRecord.dfx")
2 | =A1.select(City=="Philadelphia")
3 | =A2.fetch()

A cross-cellset cursor allows preprocessing the complicated multi-line records in a sub-cellset, making the program focused, concise and readable. It is particularly handy in performing analysis and computation based on big data file. Data volume is not large in this example, so A3 fetches all data of Philadelphian employees at once:

Sometimes the text data have a more complicated format besides the need of combining multiple lines of data into a record, as the following D:\files\mailInfo.txt shows:

The file contains emails including recipient addresses, sender addresses and contents, which start respectively with RECIPIENT:, SENDADDRESS: and CONTENT:. As the number of lines in each mail’s content is different, it makes it impossible to import and combine a specified number of rows of data and to use them directly. Yet esProc supports the cross-cellset cursor through which the data are allowed to be first reorganized in another cellset - readMail.dfx:

 | A | B | C
1 | =file("mailInfo.txt").cursor@s() | >A1.skip(2) | =""
2 | for A1,5000 | =C1+A2.(#1).string@d("\r\n") | =B2.array("RECIPIENT:")
3 |  | if A1.fetch@0(1) | =C2.len()
4 |  |  | >C1="RECIPIENT:"+C2(C3)+"\r\n"
5 |  |  | >C2=C2.to(2,C3-1)
6 |  | else | >C2=C2.to(2)
7 |  | =C2.regex("(.+)[\\s\\S]+SENDADDRESS:(.+)[\\s\\S]+CONTENT:([\\s\\S]+)";Recipient,SendAddress,Content) |
8 |  | result B7 |

A1 creates a cursor based on the text file. B1 skips the first two rows of meaningless data. For the text data, we haven’t decided how many rows are needed to form a record. It is a question under consideration. C1 stores the unprocessed data left by each loop.

By loop, A2 fetches 5,000 rows from the cursor each time. B2 concatenates the fetched data into a big string, placing the unprocessed data left by the previous loop at the head of each batch of data. Since each email starts with RECIPIENT:, C2 splits the big string into a sequence by it. The first member of the newly-created sequence will be an empty row and we cannot confirm if its last member has been processed. So the data will continue to be processed from the 3rd to the 7th line. @0 option in cs.fetch@0() in B3 is used to judge if the cursor data have all been fetched out, rather than really fetching data from the cursor. If not all has been fetched out, the remaining data will be stored in C1. For the first batch of data, results of B2 and C2 are as follows:

B7 performs regular expression parsing on each row of data in C2 using regex function. According to another two key words SENDADDRESS: and CONTENT:, it fetches the desirable data and makes them the field values. The following figure shows the parsing result of the first batch of data in B7:

B8 returns the parsing result of each batch to the main program.

Despite the complexity of the parsing process, the main program fetches data simply using a cross-cellset cursor, whose usage is nothing different from an ordinary cursor.

 | A
1 | =pcursor("readMail.dfx")
2 | =A1.fetch(1000)
3 | >A1.close()

A2 fetches the first 1,000 rows:

Next
Previous
Related :
  • 10. Cross-cellset Cursor


Chapter 8 Parrellel Computation

Next
Previous
Related :
  • 1. Multithreading
  • 2. The Built-in Parallelism
  • 3. The Server
  • 4. Cluster Computing
  • 5. The Remote Cursor


1. Multithreading

Sequential computation was the simplest and most intuitive processing method for handling computational tasks. Contemporary servers and PCs, however, have gained the multi-tasking ability with multi-core CPUs. Under the circumstances, sequential computation cannot make the most use of the computational power the CPU offers. Thus parallel computing is the best choice for implementing complex computational tasks and processing big data using multiple threads, or even multicomputers.

We’ll discuss multithreading – the most basic form of parallel computing – in this article.

Next
Previous
Related :
  • Chapter 8 Parrellel Computation
  • 1.1 Multithreaded execution with fork
  • 1.2 Performance advantage of multithreading
  • 1.3 Set reasonable number of threads


1.1 Multithreaded execution with fork

Multithreading allows subtasks to have their respectively independent threads to compute concurrently during the implementation of a computational task. esProc uses the fork statement to execute multiple threads. Its usage will be illustrated through the following example:

 | A | B
1 | $(demo) select * from EMPLOYEE | [Sales,R&D,Finance,Production]
2 | fork B1 | =A1.select(DEPT==A2)
3 |  | =B2.minp(BIRTHDAY)
4 |  | result B3

A1 retrieves data from the database table EMPLOYEE:

A2 performs the multithreaded operation using fork statement to select from each department the eldest employee. When executed, A2 gets result as follows:

During the code execution of multiple threads, fork statement is used to loop through the sequence parameter, and according to the length of the parameter sequence, allows multiple threads running in parallel to execute the code block where fork statement resides. The result of every thread returned through the result statement from the code block will be concatenated into a sequence in the main thread.

Though looking like a loop computation, the execution of fork statement is different from that of for statement. Loops specified in the code block of for statement are executed by a single thread, while for a fork statement, subtasks separated by the sequence parameter are allocated to multiple threads running in parallel for implementation.

In order to further understand the execution of each thread, in A2’s code block output function is used to output the execution information to the console:

 | A | B | C
1 | $(demo) select * from EMPLOYEE | [Sales,R&D,Finance,Production] |
2 | fork B1 | >output(A2+"-begin") | =A1.select(DEPT==A2)
3 |  | =B2.minp(BIRTHDAY) | >output(A2+"-end")
4 |  | result B3 |

During the execution of A2’s code block information is output from B2 and C3, respectively as the computation stars and ends. The output result after execution is as follows:

As can be seen from the output result, different subtasks can be implemented concurrently but some may need to wait for being processed when resources are available. System runs, distributes and controls the multiple threads of execution according to the CPU capacity. The computing time for each subtask is affected by its computation volume and the thread by which it is being performed. The implementation of subtasks with multiple threads differs according to different environments.

Multiple threads of execution can use multiple parameters. For example:

 | A | B | C
1 | $(demo) select * from EMPLOYEE | [Sales,R&D,Finance,Production] | [Texas,Illinois,Ohio,Florida]
2 | fork B1,["F","F","F","F"],C1 | =A1.select(DEPT==A2(1) && GENDER==A2(2) && STATE==A2(3)) |
3 |  | =B2.minp(BIRTHDAY) |
4 |  | result B3 |

A2 uses three parameters in executing multiple threads. The parameters will be sequentially matched for computation, i.e. [Sales,F,Texas], [R&D,F,Illinois], [Finance,F,Ohio] and [Production,F,Florida]. In this case, department, gender and state are specified for selecting the eldest employees. A2’s result is as follows:

The parameters used in the multiple threads of execution must have the same lengths.

We notice that in the above code the genders specified for selecting employees from every department are same. So the corresponding parameter can be written as the single value, as shown in the following cellset:

 | A | B | C
1 | $(demo) select * from EMPLOYEE | [Sales,R&D,Finance,Production] | [Texas,Illinois,Ohio,Florida]
2 | fork B1,"F",C1 | =A1.select(DEPT==A2(1) && GENDER==A2(2) && STATE==A2(3)) | >A1=B2
3 |  | result B2 |

While a multithreaded operation is invoked, the single-value parameter will be copied by each of the multiple threads for using the same parameter value. By the way, the result statement at the code block returns a record sequence instead of a single record each time, so the result of A2 is a sequence composed of records sequences:

Cell C2 at the code block updates A1’s data by modifying them into a record sequence, which is the result of the subroutine derived from A1. Check A1’s data after the code block is executed and we can find that they haven’t been modified at all by the code block holding the multithreaded operation. Actually subroutines executed with multiple threads will respectively duplicate the current cellset, thus they are mutually independent and do not influence each other, let alone updating data of the main routine.

Next
Previous
Related :
  • 1. Multithreading


1.2 Performance advantage of multithreading

The execution of multithreaded code appears similar to that of the loop code. Why should we use multithreading? The answer lies in the following comparison between them:

 | A | B
1 | =file("PersonnelInfo.txt") |
2 | =now() |
3 | fork to(4) | =A1.import@zt(;,A3:4)
4 |  | result B3.select(City=="San Diego")
5 | =A3.conj() | =interval@ms(A2,now())
6 |  |
7 | =now() |
8 | for 4 | =A1.import@zt(;,A8:4)
9 |  | >A10=A10|B8.select(City=="San Diego")
10 | [] | =interval@ms(A7,now())

This example requires finding out data of employees coming from San Diego from the text file PersonnelInfo.txt. We’ll use multithreading from the 2nd to the 5th line and run loops from the 7th to the 10th line. Both B3 and B8 import data in segments during the computation. Note that fork to(4) cannot be simplified for multithreaded execution. We get the same results in A5 and A10:

B5 and B10 get the estimated computing time the two methods spend:

It can be seen that multithreading spends much less time than the loop thanks to a better utilization of CPU.

Next
Previous
Related :
  • 1. Multithreading


1.3 Set reasonable number of threads

Since multiple threads of execution can increase efficiency, then will the performance be improved more significantly with more threads? Let’s make an experiment.

 | A | B
1 | =file("PersonnelInfo.txt") |
2 | =now() |
3 | fork to(4) | =A1.import@zt(;,A3:4)
4 |  | result B3.select(City=="San Diego")
5 | =A3.conj() | =interval@ms(A2,now())
6 |  |
7 | =now() |
8 | fork to(400) | =A1.import@zt(;,A3:400)
9 |  | result B3.select(City=="San Diego")
10 | =A8.conj() | =interval@ms(A7,now())

Multithreaded program with 4 threads is executed from the 2nd to 5th line while another multithreaded program with 400 threads is executed from the 7th to the 10th line. A5 and A10 get the same results as those in the previous section.

Now let’s check their computing time in B5 and B10:

As we can see, A8’s efficiency doesn’t increase in spite of using so many threads in multithreaded execution; on the contrary, the efficiency decreases. The computer has limited CPU capacity, but too many threads have to wait in queue for being executed and they also need to use some CPU resources. That’s why the efficiency lags rather than increases. In practice, good performance can be achieved with the number of threads being a little smaller than that of CPU cores.

Next
Previous
Related :
  • 1. Multithreading


2. The Built-in Parallelism

In esProc Parallel Computing: Multithreading, we learned how to increase efficiency through executing multithreaded computation. But besides using fork in the cellset code to achieve the multithreading type of parallel processing, esProc also builds the approach of parallel processing into some functions. Let’s find out things about this in the following part.

Next
Previous
Related :
  • Chapter 8 Parrellel Computation
  • 2.1 Parallel data retrieval
  • 2.2 Joining cursors
  • 2.3 Merging cursors in certain order


2.1 Parallel data retrieval

We can retrieve data from data tables through multithreaded processing, if the order of records is irrelevant to the computed result. This type of processing can make full use of the system resources, thereby enhancing the efficiency. Add @m option to certain functions to use the parallel multithreaded processing. The functions include CS.conj@mx() and f.import@m().

Let’s start from the case where we retrieve data using the sequence of file cursors:

 | A | B
1 | =file("Order_Wines.txt") | =file("Order_Foods.txt")
2 | =file("Order_Electronics.txt") | =file("Order_Books.txt")
3 | =now() | 0
4 | =[A1:B2].(~.cursor@t()) | =A4.conj@x()
5 | for B4,10000 | =A5
6 |  | >B3=B3+B5.count()
7 | =now() | 0
8 | =[A1:B2].(~.cursor@t()) | =A8.conj@mx()
9 | for B8,10000 | =A9
10 |  | >B7=B7+B9.count()
11 | =now() |
12 | =string(interval@ms(A3,A7))+"/"+string(interval@ms(A7,A11)) |

There are four text files in which ordering information of different kinds of products are stored. We want to retrieve all the records of product ordering into a sequence of file cursors, concatenate and summarize them. We use two methods to count the total number of orders. From line 3 to line 6, the cursors in the sequence are simply concatenated, thus records will be fetched in order. From line 7 to line 10, we use CS.conj@mx() to concatenate the cursors. In this way data will be fetched from cursors via multithreads; each cursor in the sequence uses an independent thread and these threads for record retrieval run simultaneously.

A12 compares the two methods to find how long it takes each of them to do this. The result is as follows:

We can see obviously that the multithreaded processing using @m option uses much less time and has a higher efficiency.

But the results of counting total number of orders in A3 and A7 with two different methods are the same:

Despite the same result, the two methods retrieve records in different orders. After the computations are finished, we can see in B5 and B9 respectively the 10,000 rows last retrieved. Now we compare them:

With the first method, data is fetched from every cursor in the sequence in normal order, so the last 10,000 rows in B5 come from the last data file Order_Books.txt. While with the second method of fetching data through parallel multithreaded processing, the returned data comes from every cursor. Therefore, in B9 data includes ordering information of every kind of product, the order in which it is fetched is irregular and data is mixed.

In spite of the higher efficiency we can get by retrieving data using parallel multithreaded processing, we can only use the method in computations whose result sets are irrelevant to orders, like summarizing and counting, because of the irregular order in which records are retrieved.

The parallel multithreaded processing also applies to cases where data is imported from a single data file, as shown below:

 | A | B
1 | =file("PersonnelInfo.txt") | =now()
2 | =A1.import@t() | =now()
3 | =A1.import@mt() | =now()
4 | =string(interval@ms(B1,B2))+"/"+string(interval@ms(B2,B3)) |

A2 imports data directly; while A3 uses f.import@m() to import data with multithreads. A4 compares the time the two methods use and gets the following result:

Importing data in parallel can significantly improve performance.

While a single data file is being retrieved with multithreads, the multithreaded system will divide the file into multiple segments and every segment will be retrieved with a file cursor. These file cursors use separate threads during data retrieval, which resembles the case where multithreaded data retrieving was done in the above example. For more information about retrieving data by segments in esProc, read esProc External Memory Computing: Binary Files.

Likewise, because multithreads are used while retrieving data from a single data file by parallel processing, the order of records in the result is irregular too. This can be proved by comparing results of A2 and A3:

With f.import@m(), the number of threads it uses is determined by the pre-specified Parallel number of task. Set this parameter in General page by clicking Tool>Option on esProc’s menu bar:

If the cellset program is run on parallel servers, the number of threads can be set through callxTask – the number of parallel tasks. For details about the setting, please see esProc Parallel Computing: The Server.

Next
Previous
Related :
  • 2. The Built-in Parallelism


2.2 Joining cursors

Parallel computing is used more than to improve the computational efficiency. It is a necessity for handling particular operations, for instance, joining cursors in a sequence together. Related functions include CS.pjoin() and F.cursor().

First let’s look at the case where file cursors in a sequence are joined:

 | A | B
1 | =file("P_ID").cursor@b() | =file("P_Name").cursor@b()
2 | =file("P_State").cursor@b() | =file("P_Gender").cursor@b()
3 | =[A1:B2].pjoin() | =A3.fetch(10000)
4 | >A3.close() |

Every cursor in A1, B1, A2 and B2 constitutes a column in the result. A3 generates a new cursor using CS.pjoin(). To fetch data from the new cursor, data from every cursor must form a record. B3 fetches the first 10,000 rows as follows:

In this case, retrieving data from A3’s cursor requires that every cursor containing a column of data needs to play a role concurrently. In other words, only multithreads can do this processing. esProc automatically employs multithreaded processing to join cursors, without the need of adding @m option. The number of thread it uses is determined by the length of the cursor sequence.

The same thing will happen when retrieving data from binary files stored in columnar format:

 | A | B
1 | =file("P_ID") | =file("P_Name")
2 | =file("P_State") | =file("P_Gender")
3 | =[A1:B2].cursor() | =A3.fetch(10000)
4 | >A3.close() |

In A3, each column in the cursor is composed of data from a binary file. Similarly, every file will be retrieved as a cursor to be handled through parallel processing. The process is the same as that in the previous example, so is the result of B3.

Next
Previous
Related :
  • 2. The Built-in Parallelism


2.3 Merging cursors in certain order

There is an operation that specifically requires parallel processing. That is merging cursors in certain order, i.e. CS.merge@x(). For example:

 | A | B
1 | =file("Order_Wines.txt").cursor@t() | =file("Order_Foods.txt").cursor@t()
2 | =file("Order_Electronics.txt").cursor@t() | =file("Order_Books.txt").cursor@t()
3 | =[A1:B2].merge@x(Date) | =A3.fetch(10000)
4 | >A3.close() |

A3 merges the ordering data from four text files by dates and generates a new cursor.

During merging cursors in certain order, we need to judge which cursor the data should be fetched from according to the sorting expression, whenever necessary. This means cursors should be in place simultaneously throughout the computation, which is a problem also need to be approached by multithreading. Equally, esProc provides automatic use of parallel processing for this kind of situation, namely, no need of @m option.

B3 fetches the first 10,000 rows as follows:

If you want to learn more about the topic, please see esProc External Memory Computing: Merge and Join Cursor Data.

Next
Previous
Related :
  • 2. The Built-in Parallelism


3. The Server

Parallel computing solves a computational problem by breaking it apart into discrete subtasks and distributing them among multiple servers on which they will be implemented concurrently. Each subtask returns its own computational result to the master task for combination. Through the simultaneous use of multiple compute resources, parallel computing can effectively enhance the computational speed and the data processing ability.

There are two forms of parallel computing - the multithreading executed on a single computer and the parallel system established through a computer cluster consisting of a network of multiple stand-alone computers. Nowadays when we talk about parallel computing, usually it is the latter that we refer to.

This article discusses the configuration of parallel servers in esProc and their application, giving you more basic ideas about the parallel computing.

Next
Previous
Related :
  • Chapter 8 Parrellel Computation
  • 3.1 esProc parallel server
  • 3.2 The launch
  • 3.3 Configuration files
  • 3.4 Data storage
  • 3.5 Application


3.1 esProc parallel server

In a parallel system, a server, or a process, that gives instructions is the main node. Other servers that receive the instructions are sub-nodes. During computation each server in this system uses an independent process to execute program instructions. Parallel servers in operation receive computational instructions, execute local cellset files accordingly and return results to the main node. In a computer cluster, a server can be run on multiple computers and a computer can have several servers running; each server permits multiple threads to execute its computational subtask. All running servers constitute the parallel computing server system.

esProc provides server class - com.raqsoft.parallel.UnitServer – to get addresses and ports according to configuration files and to launch the parallel servers. A computer is allowed to run multiple UnitServer processes.

esProc parallel system hasn’t a certain single "manager" node centrally conducting a group of "worker" nodes. An available computer cluster is designated to serve as the provisional nodes for execution of each parallel task.

Each parallel task has its logical center – the main node – from which instructions are given and to which results are returned for being combined. The task fails if the main node malfunctions; but if one of the sub-nodes malfunctions, the main node will redistribute the subtask to another one that is available. To know more about the execution of parallel tasks in esProc, see esProc Parallel Computing: Cluster Computing.

The data file needed by a sub-node to perform its own task should be stored in this sub-node beforehand. We can store the data in multiple sub-nodes, which is called redundant storage, so that all of them have the potential to perform the same subtask. The main node selects an idle sub-node to which the subtask will be distributed. Besides, redundant storage guarantees that, when a sub-node breaks down due to disasters caused by system failure, memory over flow and network anomaly, etc., there are able sub-nodes to one of which the main node can redistribute the subtask. Therefore the use of redundant storage ensures both fault-tolerance and the steadiness in performance. About the design of redundancy, the article esProc Parallel Computing: Data Redundancy will cover more details.

Data can also be stored on a Network File System (NFS), such as HDFS, over which they can be accessed by nodes. The NFS management is simpler than storing data on nodes for fault-tolerance. But compared with accessing files stored locally, it may sacrifice some performance due to the network transmission.

Next
Previous
Related :
  • 3. The Server


3.2 The launch

Run the esprocs.exe file under esProc installation directory’s esProc\bin path to launch parallel servers. The jars needed by the file will be automatically loaded under the installation directory. Here one thing to note is that configuration files – config.xml and unit.xml – and log configuration properties file must be under esProc installation directory’s esProc\config path. The setting of configuration files will be introduced in the next section.

Click on Start on UnitServer window to run parallel servers. Run-time information will be written to the console according to the log configuration.

At the launch of UnitServer, we also need to specify the server’s main path through the parameter start.home and ensure that the parallel server configuration files exist in the config directory under this main path. For example:

On the above command prompt, parameter java.ext.dirs is set to specify the loading path for Java’s jar and parameter start.home to specify the main path of parallel servers. Other parameters can be set too, for instance user.language specifies which language is to be used. The multiple server threads initiated in a single computer can either share the same main path and configuration files or have their respective main path and configuration files specified for them.

Under Linux, run startunit.sh to launch the parallel service class:

Next
Previous
Related :
  • 3. The Server


3.3 Configuration files

As mentioned in the above section, three configuration files are required to run parallel servers. They are config.xml, unit.xml and the log configuration file.

config.xml file comes first. The file contains esProc’s basic configuration information, like registration code, searching path, main path and data source configuration, etc. It resides on the esProc\config path of esProc’s installation directory. Click Tool>Options on the menu bar to configure config.xml on the Environment page, as shown in the following figure:

For the parallel computing, a dfx file can be called directly by its name, rather than by the full path, wherever it exists – the main path or any directory of the searching path.

For the data source configuration, click Tool>Datasource connection on the menu bar, then view or modify the data sources in the data source manager. To know details about the configuration, see esProc and Databases: Database Configuration.

Then comes another configuration file for parallel servers – unit.xml.

Names of the two configuration files must not be changed and the files should be placed under the config directory of home specified at the launch of servers.

The following is the configuration information of unit.xml:

<?xml version="1.0" encoding="UTF-8"?>

<Server>

<TempTimeOut>600</TempTimeOut>

<Interval>6</Interval>

<ProxyTimeOut>30</ProxyTimeOut>

<UnitList loggerProperties="raqsoftLog.properties">

<Unit host="192.168.0.86" port="4001" nodeTask="4" callxTask="2"/>

<Unit host="192.168.0.86" port="4004" nodeTask="4" callxTask="4"/>

<Unit host="192.168.1.84" port="4001" nodeTask="4" callxTask="1"/>

<Unit host="192.168.0.86" port="4006" nodeTask="4" callxTask="2"/>

</UnitList>

</Server>

The sub-node list UnitList includes all local machine’s IP addresses and ports that can potentially be used to run the servers. Servers will automatically search the list for the vacant IP address and ports at the launch. Note that the IP addresses should be the local machine’s real IP addresses. You can set multiple IP addresses when using multiple network adapters. TempTimeOut specifies the length of the life cycle of a temporary file (measured by seconds); interval is the time interval between two expiration checks, its value must be positive; proxyTimeout is the proxy life cycle, i.e. the life cycle of remote cursor and task space (measured by seconds). If either TempTimeOut or proxyTimeout is not given a value, or if either of them is 0, do not perform the expiration check.

In the allocation of sub-nodes, host is the server’s IP address, port is the specified port number and nodeTask is the number of computers running in parallel on which a server allows to be run. callxTask is the maximum number of parallel subtasks each callx is allowed to run on a sub-node and typically should be not greater than the number of CPU cores a physical machine has. This number of parallel subtasks is usually less than the number of computers running in parallel.

loggerProperties is the server’s log configuration properties file, in which log levels and other information can be set. raqsoftLog.properties - the log configuration properties file is as follows:

// Log levels include OFF, ERROR, WARN, INFO, DEBUG and ALL, whose priority levels decrease from left to right. Level OFF is intended to turn off logging

// Log messages. Level INFO indicates that messages of levels ERROR, WARN and INFO will be exported; each level exports messages likewise

// Specify name and log level for Logger

// Format: Level (can be omitted; default level is INFO), log name 1, log name 2

Logger=LOG1

// Output log messages to the console. There are only two forms of message logging: console and file

LOG1=Console

// Log message level; messages whose levels are lower than it in priority will be ignored. If it is omitted, level INFO is used by default

LOG1.Level=DEBUG

// Output logs to the specified file

// Specify the full path of LOG2. Default full path is the application’s current working path

LOG2=C:/raqsoft.log

// Default mode of logging is appending if not specified

LOG2.Append=true

// Maximum number of bytes in a log file. The default value is infinite if not specified

LOG2.MaxFileSize=10MB

// Maximum number of backup files. The default value is 1 if not specified

LOG2.MaxBackupIndex=2

//LOG2.Level=DEBUG

Next
Previous
Related :
  • 3. The Server


3.4 Data storage

esProc provides storage service to manage parallel servers in a cluster and the subtasks run on every server.

Under esProc installation directory’s esProc\bin path, click on datastore.exe file (datastore.sh under Linux) to open a window as follows:

On the Data Store interface, click node search button [image] to search the current cluster for servers and list each of them. Select an operating server and a list of subtasks run on it will be displayed on the right part of the interface. You are allowed to force a termination of a certain subtask. When the subtask is aborted, the operation performed on the main node stops too.

Next
Previous
Related :
  • 3. The Server


3.5 Application

callx instruction is used in a cellset to distribute subtasks among running servers. parallel01.dfx is such a cellset:

 | A
1 | =file("D:/files/txt/PersonnelInfo.txt")
2 | =A1.import@tz(;,pPart:pAll)
3 | =A2.select(State==pState)
4 | result A3

This piece of program imports a segment of data from the personnel information file PersonnelInfo.txt and selects from them data of employees from the specified state. The cellset parameters used in it are as follows:

The master program invokes parallel.dfx through cluster computing to find out all employees from Ohio:

 | A
1 | [192.168.0.86:4001,192.168.0.86:4004,192.168.1.84:4001]
2 | =callx("D:/files/dfx/parallel01.dfx","OH",to(20),20;A1)
3 | =A2.conj()

A1 specifies a list of parallel servers for computing. A2 invokes these servers to execute the parallel computing. When executed, A2’s result is as follows:

It can be seen that each parallel server returns a result set, thus A2’s result is a sequence composed of multiple record sequences. A3 joins all results together:

Through this form of parallel computing, master program divides a complicated computational task or an operation involving big data into multiple subtasks, distributes them to multiple servers to compute separately and then joins every result together. The topic of parallel computing will continue in the article esProc Parallel Computing: Cluster Computing.

Next
Previous
Related :
  • 3. The Server


4. Cluster Computing

Cluster computing can be used in esProc for performing complicated analytic and processing tasks. A cluster system consists of multiple parallel servers running on independent computers in a network. Every single computer in this network can send a parallel computing request to the cluster.

A cluster system enhances computational performance significantly by gathering the computational power of multiple servers. It expands its size as needed whenever necessary without paying the high cost associated with the mainframe.

Next
Previous
Related :
  • Chapter 8 Parrellel Computation
  • 4.1 Perform cluster computation through callx function
  • 4.2 Handle large-scale computation


4.1 Perform cluster computation through callx function

The article esProc Parallel Computing: The Server introduced how to launch parallel servers. After multiple parallel servers are started and a cluster system is created, cluster computation can be performed through callx function. In esProc, every server will carry out the parallel computing tasks by computing the specified dfx file and returning the result.

Cluster computing requires the preparation of the cellset file to be used in performing the subtasks, like CalcStock.dfx shown below:

 | A | B
1 | >output@t("calc begin, SID="+string(arg1) +". ") | =file("StockRecord.txt").cursor@t()
2 | =B1.select(SID==arg1) | =A2.fetch()
3 | =B2.count() | =B2.max(Closing)
4 | =B2.min(Closing) | =round(B2.avg(Closing),2)
5 | >output@t("calc finish, SID="+string(arg1) +". ") | result A3,B3,A4,B4

The file StockRecord.txt contains closing prices of some stocks in a certain period of time. Compute transaction information of specified stock codes in each subtask, including the total number of trading days, the highest and lowest closing prices and the average closing price. The average closing price will be simply computed by trading days, without taking the number of transactions into consideration. In order to know the detailed information about the execution of subtasks, output@t function is used at the beginning of the computation and before result returning to output times and prompts. In the cellset file, arg1 is a parameter for passing stock codes in and needs to be set before the parallel computation. Select Set arguments before run to set the parameter:

Parallel computing requires that in computers that run servers the cellset file CalcStock.dfx should exist and be placed in mainPath of every server’s configuration.

This example configures three servers in two computers, whose IPs and port numbers respectively are: Server I (192.168.0.86:4001), Server II (192.168.0.66:4001) and Server III (192.168.0.86:4004). Set different callxTask – number of parallel tasks - 2, 1, and 4 respectively, for the three servers in running. nodeTask – the number of parallel machines - is 4.

Then we can start performing parallel computation using callx function in the main program on any computer in the network:

 | A
1 | [192.168.0.86:4001,192.168.0.66:4001,192.168.0.86:4004]
2 | [124051,128857,131893,136760,139951,145380]
3 | =callx("CalcStock.dfx",A2;A1)
4 | =A3.new(A2(#):SID,~(1):Count,~(2):Maximum,~(3):Minimum,~(4):Average)

About callx’s parameters in A3’s expression, the one before the semicolon is used for computing the cellset file. The parallel computing will divide the computational task into multiple subtasks according to the sequence parameter passed through it. The length of the sequence parameter represents the number of subtasks; multiple parameters are separated with commas, in which the single-value parameter will be duplicated to every subtask. callx’s parameter after the semicolon is the sequence of servers, each of which is represented by a string in the format of “IP:port number”, for example “192.168.0.86:4001”. After the computations on the parallel servers are completed, the result will be returned by the result statement; multiple results will be returned in the form of a sequence.

In this example, we need to get the data of six stocks. Their codes are 124051,128857,131893,136760,139951,145380, which correspond subtasks a, b, c, d, e, f that will be distributed among the three servers for performing parallel computation. The distribution is conducted according to the order of the sequence of servers specified in the function. If one of the servers with specified IP and port number is available, then give it one subtask; if the server hasn’t been started or the number of parallel subtasks on it reaches its maximum, then move on to the next server on the list. We can read the distribution and execution messages of the subtasks on each of their System Output window:

As can be seen from the output information, Server I is handling subtasks a and d; Server II is handling subtask b; and Server III is handling subtasks c and e. After the first three subtasks a, b and c have been distributed in order among the servers, the second round of distribution begins by giving subtask d to Server I. Subtask e is given to Server III for execution because the callxTask set for Server II is 1 and the server has already been given a task. Server III also receives subtask f as both Server II and Server III are fully loaded.

After the computation is completed, we can see the result in A4:

This example shows that, by splitting apart the data-intensive main computational task and having each server processing one of the parts, cluster computing effectively avoids the memory overflow caused by big data processing.

Note that as subtasks are executed separately on each server, they are independent of each other. The dfx file must exist in each server. If data sources or data files are needed in the computation, they should be configured for or stored on the related servers.

Let’s look at another example. Still we’ll get the statistical data of the six stocks, but this time each subtask only gets data of a certain year. The corresponding cellset file CalcStock2.dfx is as follows:

 | A | B
1 | >output@t("calc begin, year: "+string(arg1)+". ") | =file("StockRecord"+string(arg1)+".txt")
2 | if !(B1.exists()) | end "File of year "+string(arg1)+" is not found!"
3 | =B1.cursor@t() | [124051,128857,131893,136760,139951,145380]
4 | =A3.select(B3.pos(SID)>0).fetch() | =A4.groups(SID;count(~):Count,max(Closing):Maximum,min(Closing):Minimum,sum(Closing):Sum)
5 | >output@t("calc finish, year: "+string(arg1)+". ") | result B4

Data of stocks of different years are stored in their respective data files. For example the data of 2010 are stored in StockRecord2010.txt. The five data files used for the computation are stored on three servers. Data of 2010 and 2014 are stored on Server I, data of 2011 and 2013 are on Server II and data of 2012 are on Server III. Each subtask computes the transaction data of the six stocks in a specified year. As in the above example, output@t function is used to output times and prompts as the computation begins and before the result is returned. Since each data file is only stored on one certain server, A2 judges if the file exists on the current server. If it is false, the result will be returned by end statement and the subtask need to be redistributed.

In the cellset file, arg1 is a parameter that passes a specified year whose data are to be computed:

Then perform cluster computation in the master program as follows:

 | A
1 | [192.168.0.86:4001,192.168.0.66:4001,192.168.0.86:4004]
2 | [2010,2011,2012,2013,2014]
3 | =callx("CalcStock2.dfx",A2;A1)
4 | =A3.merge(SID).groups(SID;sum(Count):Count,max(Maximum):Maximum,min(Minimum):Minimum,sum(Sum):Sum)
5 | =A4.new(SID,Count,Maximum,Minimum,round(Sum/Count,2):Average)

A3 performs cluster computation, A4 merges its results and A5 generates the final result. Here to get the statistics of five years, cluster computing gives five subtasks to three servers. The distribution and execution messages about the subtasks will be displayed on each server’s System Output window:

As can be seen from these output messages, the first three subtasks – 2010, 2011 and 2012 - are distributed to the servers in order, and they execute normally. Then the subtask of 2013 is distributed to Server I. The subtask of 2014 is given to Server III as the number of parallel subtasks on Server II has reached the maximum. But as Server I hasn’t the data file of 2013 and Server III hasn’t the data file of 2014, both subtasks of 2013 and 2014 cannot find the files they needed for computation and throw exceptions through end statement; then the two subtasks are redistributed by the main program. At this time the computation on Server II hasn’t completed, so subtask of 2013 is distributed to Server III and subtask of 2014 is distributed to Server I. The latter executes normally while the former throws exception again. The redistribution of subtask of 2013 suspends due to the non-availability of both Server I and Server III; it has to stay waiting until Server II accomplishes its current subtask and the redistribution resumes.

After all computations are completed, A5’s result is as follows:

Thus when a server can only perform certain subtasks, redistribution of subtasks may be unavoidable when exceptions are thrown. In this case use @a option in callx function to distribute each task to the server strictly in order, as shown in the following cellset:

 | A
1 | [192.168.0.86:4001,192.168.0.66:4001,192.168.0.86:4004, 192.168.0.66:4001,192.168.0.86:4001]
2 | [2010,2011,2012,2013,2014]
3 | =callx@a("CalcStock2.dfx",A2;A1)
4 | =A3.merge(SID).groups(SID;sum(Count):Count,max(Maximum):Maximum,min(Minimum):Minimum,sum(Sum):Sum)
5 | =A4.new(SID,Count,Maximum,Minimum,round(Sum/Count,2):Average)

To use callx@a, the length of the sequence of servers should be the same as the number of the subtasks, and duplicated servers are allowed. Thus each subtask will be sure to be distributed to a specified server, on which if the parallel subtasks are full, this subtask will wait until it is available and the distribution continues. The System Output window displays information about distribution and execution of the subtasks:

The use of @a option can avoid the loss of efficiency caused by redistribution of subtasks.

From this example we can see that troubles will arise if a subtask can only be performed on a certain server. This may even result in the failure of the whole task if one of the servers goes wrong. The problem can be solved by storing data files on multiple servers to increase the steadiness of the system. See esProc Parallel Computing: Data Redundancy for related information.

Next
Previous
Related :
  • 4. Cluster Computing


4.2 Handle large-scale computation

In the previous examples, cluster computing is used to handle data-intensive computational task. By processing part of the data in each subtask, it makes the most use of the limited memory. In the other scenario, the task involves extremely complicated computation that can also be split apart into multiple subtasks through cluster computing that are distributed among multiple servers for execution. Results of the subtasks will be merged by the master program.

Take the following subroutine CalcPi.dfx as an example:

 | A | B | C
1 | 1000000 | 0 | >output@t("Task "+ string(arg1)+ " start...")
2 | for A1 | =rand() | =rand()
3 |  | =B2*B2+C2*C2 |
4 |  | if B3<1 | >B1=B1+1
5 | >output@t("Task "+ string(arg1)+ " finish.") | result B1 |

Parameter arg1 is used to record numbers of subtasks:

This subroutine is created to estimate the value of π - ratio of circumference to diameter - using probability theory. Look at the following figure:

There is the quarter-circle in a square with side length of 1. Area of the square is 1 and area of the sector isπ/4. The probability of a point of the square that falls in the sector is the ratio of their areas, i.e. π/4. This subroutine randomly generates 1,000,000 points whose x, y coordinates are within the interval [0,1), computes the distances of each of these points from the origin, records the number of points that fall in the sector and then estimates the value of π.

The main program is as follows:

 | A | B
1 | [192.168.0.86:4001,192.168.0.66:4001,192.168.0.86:4004] | 20
2 | =callx("CalcPi.dfx",to(B1);A1) | =A2.sum()*4/(1000000d*B1)

By invoking the subroutine 20 times, cluster computing distributes the task of computing 20,000,000 points into servers (they are those used in the previous section). Distribution and execution information of the subtasks can be seen on each server’s System Output window:

It can be seen that the number of subtasks exceeds the maximum number of subtasks the servers can receive. After the first 7 subtasks are distributed, all the three servers have reached their maximum number of parallel subtasks set for them. The subsequent subtasks need to wait to be distributed until a server is available. In this case a subtask will be given to any server that has the idle resource, so distribution solutions may vary. Seen from the output information of Server III, subtasks executed on the same server have separate threads and each computational process is an independent one. A subtask added behind another one may be accomplished earlier, which will not affect the result of parallel processing.

The approximate value of π is computed in B2. Because the computation is performed based on probability, every result is slightly different from other ones.

Next
Previous
Related :
  • 4. Cluster Computing


5. The Remote Cursor

Parallel computing allows executing a computation on big data by distributing it to nodes. If the data distributed to each node is still in large volume, it can be returned from the node with the remote cursor. We’ll now learn the usage of the remote cursor as well as its features.

Next
Previous
Related :
  • Chapter 8 Parrellel Computation
  • 5.1 The usage
  • 5.2 The auto-close
  • 5.3 Merge remote cursors


5.1 The usage

The servers the remote cursor uses are the same as those used in a common cluster computation. You can see esProc Parallel Computing: The Server for details about their running and configuration. After servers are started, the remote cursor service will be able to return cursors during parallel computation using the result statement.

For example, the following StockCursor.dfx reads the stock record file of a certain year and returns it as the cursor:

 | A | B
1 | >output@t("calc begin, year: "+string(arg1)+". ") | =file("StockRecord"+string(arg1)+".txt")
2 | if !(B1.exists()) | end "File of year "+string(arg1)+" is not found!"
3 | =B1.cursor@t() | >output@t("calc finish, year: "+string(arg1)+". ")
4 | result A3 |

Judge if the data file for computation exists in the current server. Return the result by end statement if it is false, so that the main program can redistribute the computational task to another server. If the result is true, then return the file cursor. In the cellcet program, arg1 is the parameter for passing in the year of which the statistics are to be calculated.

In this example, three servers are running on two computers and stock records for different years are saved in multiple data files, whose distribution goes like this: Server I (192.168.0.86:4001) stores data of 2010 and 2014; Server II (192.168.0.66:4001) stores data of 2011 and 2013 and Server III (192.168.0.86:4004) stores data of 2012. The number of parallel tasks – callxTask – set for each of the three servers is different, respectively 2, 1 and 4; while the number of machines for the parallel processing – nodeTask – is 4.

The main program calls the cellset file as follows:

 | A
1 | [192.168.0.86:4001,192.168.0.66:4001,192.168.0.86:4004]
2 | [2010,2011,2012,2013,2014]
3 | =callx("StockCursor.dfx",A2;A1)
4 | =A3.conj@x()
5 | =A4.select(SID==124051)
6 | =A5.fetch()

In this cellset, A3 executes parallel computation via callx to run StockCursor.dfx on the servers. The file must exist in the mainPath configured for each server. Since every subtask returns a cursor, the result of A3 will be a sequence of remote cursors after the whole computation is completed:

Cursors in the cursor sequence are concatenated in A4 with conj@x, so that data fetching will go through every cursor one after another. A5 selects from the cursors the one in which data of stock 124051 is stored. A6 fetches data from it, the result is as follows:

As can be seen from the result, data of every year is concatenated orderly.

Then we’ll look at the information output from each server.

Subtasks will be assigned to servers according to the order of the corresponding parameters. If one of the servers has already had the callxTask set for it, then move on to the next server for the distribution. If the data file of a certain year for the computation of StockCursor.dfx does not exist in the server, the information will be returned to the main program via the end statement and the subtask will be redistributed to another one. As can be seen from the output information, a subtask is considered completed when a remote cursor is returned to the main program. So each subtask is quickly completed despite the fact that data hasn’t been really returned from the cursor.

In order to avoid distributing a subtask to a wrong server where the required data file does not exist, we can prescribe strictly which servers a subtask should use through callx@a. It is the same approach used in a common parallel computation. You can find concrete examples in esProc Parallel Computing: Cluster Computing.

While the main program is executing fetch statement, remote cursors are still working on servers, on which data is fetched from each cursor and returned to the main program. When every cursor’s data is all fetched out, remote cursors are automatically closed.

Next
Previous
Related :
  • 5. The Remote Cursor


5.2 The auto-close

In the main program, if cursors are generated but data is not fetched, or not completely fetched, as the following cellset shows:

 | A
1 | [192.168.0.86:4001,192.168.0.66:4001,192.168.0.86:4004]
2 | [2010,2011,2012,2013,2014]
3 | =callx("StockCursor.dfx",A2;A1)
4 | =A3.conj@x()
5 | =A4.select(SID==124051)
6 | /=A5.fetch()

Comment out A6’s fetch statement to merely generate cursors in A5. In this case the information output from each server is as follows:

We can see that remote cursors will commit self-destruction if they are not accessed within a long time after the creation. In the server’s configuration file unit.xml, we can set the expiration time for the remote cursor with ProxyTimeOut. If the expiration time is zero, or hasn’t been set, the program will not perform an expiration check. The related setting is introduced in esProc Parallel Computing: The Server.

After a remote cursor closes because of a time-out, fetching data from it will result in an error. The program considers the server’s returning of a cursor to the main program as the accomplishment of a subtask, so if the cursor closes like that the corresponding subroutine will not be run again. To avoid such a problem, we should appropriately set the expiration time as needed.

Next
Previous
Related :
  • 5. The Remote Cursor


5.3 Merge remote cursors

Parallel computation using the remote cursor will return a sequence of remote cursors that will be combined by the main program. In the previous example, the remote cursors returned from multiple servers were concatenated in order, but they can be merged as well

The following StockCursor2.dfx file retrieves stock record data of a certain year and sorts them by stock number SID before returning a cursor:

 | A | B
1 | >output@t("calc begin, year: "+string(arg1)+". ") | =file("StockRecord"+string(arg1)+".txt")
2 | if !(B1.exists()) | end "File of year "+string(arg1)+" is not found!"
3 | =B1.cursor@t() | >output@t("calc finish, year: "+string(arg1)+". ")
4 | result A3.sortx(SID) |

A remote cursor can be various types, such as file cursor, database cursor and in-memory record sequence cursor. The remote cursor A4 returns to the main program is actually a temporary file cursor generated during the execution of sortx.

In the main program, cursor data of every year is merged orderly by SID:

 | A
1 | [192.168.0.86:4001,192.168.0.66:4001,192.168.0.86:4004]
2 | [2010,2011,2012,2013,2014]
3 | =callx("StockCursor2.dfx",A2;A1)
4 | =A3.merge@x(SID)
5 | =A4.fetch(;SID)
6 | >A4.close()

Note that data in every cursor of the cursor sequence should be sorted before cursors are merged; otherwise error will occur. A5 only fetches the transaction data of the first stock in five years:

As data in the remote cursor hasn’t been completely fetched out, A6 needs to call close to close it. If not closed, it will perform auto-close when its time expires.

Next
Previous
Related :
  • 5. The Remote Cursor


Chapter 9 Heterogeneous Datasource

Next
Previous
Related :
  • 1. Excel
  • 2. JSON and HTTP
  • 3. Hadoop
  • 4. MongoDB


1. Excel

esProc can import the data from and write them into Excel files by making the Excel one of its various datasources. Two simple examples will show how this works.

First let’s look at the data importing. The target files are data.xlsx and data.xls (excel97-2000 format) and have the same content. sheet0 is named employee and sheet1 is named orders.

The following is employee:

And this is orders:

esProc script for importing the data from the Excel files:

 | A
1 | =file("D:/files/work/excel/data.xlsx")
2 | =file("D:/files/work/excel/data.xls")
3 | =A1.importxls@tx()
4 | =A2.importxls@t(ORDERID,SELLERID,AMOUNT,ORDERDATE;"orders",3:-3)

A1: Define a file object; the path and name is D:/files/work/excel/data.xlsx。

A2: Define a file object; the path and name is D:/files/work/excel/data.xls。

A3: Import the data from A1’s file object using importxls function. @tx is function options. t represents making the first row the column names; without it, esProc will name the columns “_1,_2…” automatically. x represents that the file format is xlsx; without it, esProc will try to determine the format according to the filename extension; but if the format can’t be determined, the file will be imported in xls format. Without input parameters, importxls function will import all data from the first sheet, as shown below:

A4: Import A2’s file object using importxls function. @t represents making the first row the column names. The input parameters - ORDERID, SELLERID, AMOUNT and ORDERDATE – represent that only these specified columns will be imported. The parameter “orders” specifies the name of the to-be-imported sheet. Parameter “3:-3” represents importing data from the third row to the third-last row. Similarly, you can import data from the 3rd row to the 100th row, which is 3:100.

The result of data importing is as follows:

Then let’s look at how to write data into an Excel file. Suppose the results of A3 and A4 in the script above need to be written into result1.xlsx, a sheet named re-emp, and result2.xls, a sheet named r-orders, respectively.

esProc script for writing them into the Excel files:

 | A
1 | =file("D:/files/work/excel/data.xlsx")
2 | =file("D:/files/work/excel/data.xls")
3 | =A1.importxls@tx()
4 | =A2.importxls@t(ORDERID,SELLERID,AMOUNT,ORDERDATE;"orders",3:-3)
5 | =file("D:/files/work/excel/result1.xlsx")
6 | =file("D:/files/work/excel/result2.xls")
7 | =A5.exportxls@tx(A3,EID:EMPLOYEEID,NAME;"r-emp")
8 | =A6.exportxls (A4;"r-orders")

A5, A6: Define two file objects as required.

A7: Export the result of A3 to A5’s file object using exportxls function. @tx option means the exported file format is xlsx and the first exported row is the field names. The input parameter A3 is the to-be-exported table sequence. The parameters - EID:EMPLOYEEID and NAME – are specified to-be-exported fields, in which EID:EMPLOYEEID means field EID will be renamed EMPLOYEEID when exported. Parameter "r-emp" specifies the name of the exported sheet. The exported Excel file is as follows:

A8: Export the result of A4 to A6’s file object using exportxls function. Without the @tx option, the exported file format will be xls determined by the filename extension and the first row won’t be exported. The input parameter A4 is the to-be-exported table sequence. All fields will be exported as none is specified. The parameter "r-orders" specifies the name of the to-be-exported sheet. The exported Excel file is as follows:

Next
Previous
Related :
  • Chapter 9 Heterogeneous Datasource


2. JSON and HTTP

JSON is a common data-interchange format. JSON data come from external Java programs, local files or the HTTP server. esProc can easily parse the JSON strings and compute them.

The following examples will show in detail how to write an esProc script for doing this.

The first case is the one in which the JSON string is transferred from the external Java program in the form of a parameter.

Suppose there is a string holding employee information in JSON format and including the following fields – EID, NAME, SURNAME, GENDER, STATE, BIRTHDAY, HIREDATE and DEPT. Now parse the string and find out the female employees born after January 1, 1981 inclusive. The string is as follows:

[{EID:1,NAME:"Rebecca",SURNAME:"Moore",GENDER:"F",STATE:"California ",BIRTHDAY:1974-11-20,HIREDATE:2005-03-11,DEPT:"R&D",SALARY:7000},

{EID:2,NAME:"Ashley",SURNAME:"Wilson",GENDER:"F",STATE:"New York",BIRTHDAY:1980-07-19,HIREDATE:2008-03-16,DEPT:"Finance",SALARY:11000},

{EID:3,NAME:"Rachel",SURNAME:"Johnson",GENDER:"F",STATE:"New Mexico",BIRTHDAY:1970-12-17,HIREDATE:2010-12-01,DEPT:"Sales",SALARY:9000},…]

Solution: Call the esProc script using the Java program and meanwhile input the JSON string; parse and filter the JSON data and then return the result in the form of JSON string to the Java program.

To select female employees born after January 1, 1981 inclusive, esProc script inputs two parameter – “jsonstr” and “where” – from the external program, as shown below：

“where” is a string, its value is BIRTHDAY>=date(1981,1,1) && GENDER=="F".

esProc script for doing this task:

 | A |
1 | =jsonstr.import@j() |
2 | =A1.select(${where}) |
3 | =export@j(A2) | [{EID:4,NAME:"Emily",SURNAME:"Smith",GENDER:"F",STATE:"Texas",BIRTHDAY:1985-03-07,HIREDATE:2006-08-15,DEPT:"HR",SALARY:7000}…]
4 | result A3 |

A1: Parse the JSON data into a table sequence. The result can be displayed in esProc’s Integration Development Environment (IDE), as shown in the right part of the figure above.

A2: Filter the data of the table sequence. A macro is used to dynamically parse the expression, in which “where” is the input parameter. esProc will compute the expression in ${…}, take the computed result as the macro string value and replace ${…} with it, and then parse and execute the expression. The final code for execution is =A1.select(BIRTHDAY>=date(1981,1,1) && GENDER=="F").

A3: Export the data of the filtered table sequence in the form of a JSON string.

A4: Return the eligible result set to the external program.

Please refer to esProc Tutorial for the method of calling an esProc script in Java program.

Another case is that the JSON data comes from the HTTP server. Suppose to return a JSON string of employee information with a testServlet, then you can get and process the data as follows:

 | A
1 | =httpfile("http://localhost:6080/myweb/servlet/testServlet?table=employee&type=json")
2 | =A1.read()
3 | =A2.import@j()
4 | =A3.select(${where})
5 | =export@j(A4)
6 | result A5

A1: Define a httpfile object, URL is

http://localhost:6080/myweb/servlet/testServlet?table=employee&type=json

A2: Read the result returned by the httpfile object.

A3: Parse the JSON string and generate a table sequence.

A4: Filter the data according to the specified condition.

A5: Convert the filtered table sequence to a JSON string.

A6: Return the result of A5 to the Java program invoking the esProc script.

What’s more, if the JSON data include nested object, esProc can still parse and compute them. For example, there are nested comments in the blog information. esProc can parse them into fields which can be referenced. The blogs.json file is as follows:

[

{ "id" : 1000,

"content" : "It is too hot",

"comment" :

[

{

"author" : "joe",

"score" : 3,

"comment" : "just so so!"

},

{

"author" : "jimmy",

"score" : 5, "comment" : "cool! good!"

}

]

},

{ "id" : 1001,

"content" : "It is too cold",

"comment" :

[

{

"author" : "james",

"score" : 1,

"comment" : "yes!" },

{

"author": "jimmy",

"score" : 5,

"comment" : "cool!"

}

]

},

{ "id" : 1002,

"content" : "It is windy day today",

"comment" :

[

{

"author" : "tom",

"score" : 3,

"comment" : "I do not thinkso!"

},

{

"author" : "jimmy",

"score" : 5,

"comment" : "cool!"

}

]

}

]

Use the esProc editor to write a script in which the JSON file will be imported, data are parsed and computed:

 | A
1 | =file("D:/files/work/txt/blogs.json").read().import@j()

Get the resulting table sequence and we can see that comment field acts as a referencing field to reference a table sequence, as shown below:

Double-click the blue part on the first row to see the referenced table sequence in detail, as shown below:

esProc can further perform more complicated set operations based on this table sequence with a referencing field.

Next
Previous
Related :
  • Chapter 9 Heterogeneous Datasource


3. Hadoop

esProc’s Hadoop datasource includes mainly HDFS files and Hive database. The way of accessing Hive database is similar to that of accessing other databases, so we won’t discuss it here. esProc has the built-in method of accessing HDFS files, which resembles that of accessing files from an ordinary file system. We’ll introduce how to write an esProc script through the following example.

The HDFS text file, employee.txt, has the employee data. We are asked to import the data and find out female employees born after January 1, 1981 inclusive and export the result to a file in a compressed format.

The data of empolyee.txt are as follows:

EID NAME SURNAME GENDER STATE BIRTHDAY HIREDATE DEPT SALARY

1 Rebecca Moore F California 1974-11-20 2005-03-11 R&D 7000

2 Ashley Wilson F New York 1980-07-19 2008-03-16 Finance 11000

3 Rachel Johnson F New Mexico 1970-12-17 2010-12-01 Sales 9000

4 Emily Smith F Texas 1985-03-07 2006-08-15 HR 7000

5 Ashley Smith F Texas 1975-05-13 2004-07-30 R&D 16000

6 Matthew Johnson M California 1984-07-07 2005-07-07 Sales 11000

7 Alexis Smith F Illinois 1972-08-16 2002-08-16 Sales 9000

8 Megan Wilson F California 1979-04-19 1984-04-19 Marketing 11000

9 Victoria Davis F Texas 1983-12-07 2009-12-07 HR 3000

10 Ryan Johnson M Pennsylvania 1976-03-12 2006-03-12 R&D 13000

11 Jacob Moore M Texas 1974-12-16 2004-12-16 Sales 12000

12 Jessica Davis F New York 1980-09-11 2008-09-11 Sales 7000

13 Daniel Davis M Florida 1982-05-14 2010-05-14 Finance 10000

…

To develop and debug a program in esProc’s Integration Development Environment (IDE), first copy Hadoop’s core package and configuration package, like commons-configuration-1.6.jar, commons-lang-2.4.jar, and hadoop-core-1.2.1.jar (Hadoop1.2.1) into “esProc installation directory\esProc\lib” directory.

esProc code for processing the file:

 | A
1 | =hdfsfile("hdfs://192.168.1.11:9000/user/employee.txt","UTF-8")
2 | =A1.cursor@t()
3 | = A2.select(BIRTHDAY>=date(1981,1,1) && GENDER=="F")
4 | =hdfsfile("hdfs://192.168.1.11:9000/user/emp-result.gz").export@t(A3)
5 | =hdfsfile("hdfs://192.168.1.11:9000/user/emp-result.gz").import@t()

A1: Define an HDFS file object. “UTF-8” is a charset for encoding the file; use the JVM charset by default. We can see that an HDFS file object is defined in a similar way to defining one from an ordinary file system. Subsequent computations, like declaring a cursor and importing data, can also be performed with no difference.

A2: A cursor is defined according to A1’s file object, whose first row is imported as the column names with tab being the default separator. With cursors, we can process big files by importing data segmentally.

A3: Filter cursor data according to the specified condition.

A4: Export the filtering results to emp-result.gz, which will be compressed automatically into gzip format in Hadoop because of its .gz extension. Hadoop also supports other compressed formats, like lzo and lz4 (see Hadoop-related documents for details).

A5: Import all data from emp-result.gz, which will be decompressed automatically into an ordinary text file according to the gzip format because of its .gz extension. import function is used to import the whole data into the memory.

The final result is as follows:

Next
Previous
Related :
  • Chapter 9 Heterogeneous Datasource


4. MongoDB

It is convenient to establish and close a connection to MongoDB in esProc , as well as to call the database to query and count the data, perform distinct and aggregate operations.

Next
Previous
Related :
  • Chapter 9 Heterogeneous Datasource
  • 4.1 Preparation for connecting to MongoDB
  • 4.2 Create and close a connection to MongoDB
  • 4.3 Common data query and selection
  • 4.4 Query and filter nested arrays and documents
  • 4.5 Count the data, perfor distinct and aggregate operations


4.1 Preparation for connecting to MongoDB

The MongoDB Java driver (like mongo-java-driver-2.12.2.jar), which esProc does’nt provide, should be put into [esProc installation directory]\common\jdbc beforehand when esProc designer is used to access the database. The MongoDB Java driver’s download link is https://github.com/mongodb/mongo-java-driver/releases .

To access MongoDB by calling the esProc script through esProc JDBC in a Java program, the above MongoDB Java driver needs to be put into the classpath of the Java program.

Next
Previous
Related :
  • 4. MongoDB


4.2 Create and close a connection to MongoDB

In an esProc script, mongodb(con) function is used to establish a connection to the MongoDB database and mdb.close() function is used to close it, as shown by the following code:

 | A
1 | =mongodb("mongo://localhost:27017/test?user=test&password=test")
2 | =A1.close()

A1: Connect to MongoDB. IP and port are localhost:27017; database name is test, and so are the user name and the password.

A2: Close the MongoDB connection.

Next
Previous
Related :
  • 4. MongoDB


4.3 Common data query and selection

Let’s take the employee, a collection in the database whose name is test, of MongoDB as an example to illustrate the esProc script for accessing MongoDB. Some of the data of employee are as follows:

{ "_id" : ObjectId("546aea22ee62ea452bdef4fe"), "EID" : 1, "NAME" : "Rebecca", "

SURNAME" : "Moore", "GENDER" : "F", "STATE" : "California", "BIRTHDAY" : "1974-1

1-20", "HIREDATE" : "2005-03-11", "DEPT" : "R&D", "SALARY" : 7000 }

{ "_id" : ObjectId("546aea22ee62ea452bdef4ff"), "EID" : 2, "NAME" : "Ashley", "S

URNAME" : "Wilson", "GENDER" : "F", "STATE" : "New York", "BIRTHDAY" : "1980-07-

19", "HIREDATE" : "2008-03-16", "DEPT" : "Finance", "SALARY" : 11000 }

{ "_id" : ObjectId("546aea22ee62ea452bdef500"), "EID" : 3, "NAME" : "Rachel", "S

URNAME" : "Johnson", "GENDER" : "F", "STATE" : "New Mexico", "BIRTHDAY" : "1970-

12-17", "HIREDATE" : "2010-12-01", "DEPT" : "Sales", "SALARY" : 9000 }

…

The following is a sample of esProc script for querying data of the collection of MongoDB:

 | A
1 | =mongodb("mongo://localhost:27017/test?user=test&password=test")
2 | =A1.find("employee","{STATE:\"California\",SALARY:{$gt:4000}}","{GENDER:0}").fetch()
3 | =A1.close()

A2: Filter and query the data of employee. find function’s first parameter is employee, the collection’s name; its second parameter is the filtering criterion: STATE=” California” and SALARY >4,000; the third one specifies which properties should be fetched (here GENDER won’t be fetched). Noticed that esProc uses the same parameter format as that used in MongoDB’s find statement.

The function returns a cursor, which can fetch data in batch for computing when the result set is big, so as to avoid the memory overflow. The usage of cursor has been explained in detail in esProc-related documents. Here suppose the data are not big and use fetch function to fetch all the results for observation:

Next
Previous
Related :
  • 4. MongoDB


4.4 Query and filter nested arrays and documents

Nested arrays and docements are multilevel data structure specific to MongoDB. The Java application doesn’t support this multilevel data structure when it accesses MongoDB through JDBC. Because esProc supports referencing data by fields, like referencing a record sequence by a field, it can fetch the nested arrays and documents of MongoDB.

For a nested array, let’s look at a collection – fruitshop:

{ "_id" : ObjectId("548a58c19cfbc9dd702eb71e"), "name" : "big fruit", "fruits" :

[ "apple", "pear", "orange" ] }

{ "_id" : ObjectId("548a58c19cfbc9dd702eb71f"), "name" : "good fruit", "fruits"

: [ "banana", "pear", "orange" ] }

{ "_id" : ObjectId("548a58c29cfbc9dd702eb720"), "name" : "my fruit", "fruits" :

[ "banana", "apple", "tomato" ] }

And for a nested docment, let’s look at this collection – blogs:

{ "_id" : ObjectId("548a59fa9cfbc9dd702eb721"), "content" : "It is too hot", "co

mment" : [ { "author" : "joe", "score" : 3, "comment" : "just so so!" }, { "auth

or" : "jimmy", "score" : 5, "comment" : "cool! good!" } ] }

{ "_id" : ObjectId("548a59fa9cfbc9dd702eb722"), "content" : "It is too cold", "c

omment" : [ { "author" : "james", "score" : 1, "comment" : "yes!" }, { "author"

: "jimmy", "score" : 5, "comment" : "cool!" } ] }

{ "_id" : ObjectId("548a59fb9cfbc9dd702eb723"), "content" : "It is windy day tod

ay", "comment" : [ { "author" : "tom", "score" : 3, "comment" : "I do not think

so!" }, { "author" : "jimmy", "score" : 5, "comment" : "cool!" } ] }

esProc script for querying and filtering data of the two collections:

 | A
1 | =mongodb("mongo://localhost:27017/test?user=test&password=test")
2 | =A1.find("fruitshop","{fruits:\"pear\"}").fetch()
3 | =A1.find("blogs","{comment.author:\"jimmy\"}").fetch()
4 | =A1.close()

A2: The filtering criterion of find function includes the pear insead of being equal to pear, for the fruits is an array. The return result of find funciton is a cursor, from which the resulting table sequence is fetched using fetch function and you can see fruits field is a referencing field, which references a sequence consisiting of various fruits names, as shown below:

Click the blue part of the first row to see the detailed information of the referenced sequence, as shown below:

Based on this table sequence with a referencing field, esProc can further perform more complicated set operations.

A3: comment is a collection of documents, and the filtering criterion is the comment’s documents whose author is jimmy. find function returns a cursor, from which the table sequence is fetched using fetch function. comment field of the table sequence is a referencing field that references a table sequence, as shown below:

Click the blue part of the second row to see the detailed information of the table sequence, as shown below:

Based on this table sequence with a referencing field, esProc can further perform more complicated set operations. Here further discussion is omitted.

Next
Previous
Related :
  • 4. MongoDB


4.5 Count the data, perfor distinct and aggregate operations

With more powerful ability in computing and flow control, usually esProc is used to perform the complicated multi-step operation and MongoDB, a highly-efficient storage solution, is responsible for providing data without participating in the computation. But considering the performance loss during the data transmission, you can use MongoDB alone to perform some simple aggregate operations.

esProc provides the way of calling MongoDB interface to count the data, perform distinct and aggregate operations. Different from the find function for querying and filtering data, the return results of the three operations are table sequences instead of cursors. A sample code is as follows:

 | A
1 | =mongodb("mongo://localhost:27017/test?user=test&password=test")
2 | =A1.count("employee","{STATE:\"California\"}")
3 | =A1.distinct("employee","DEPT")
4 | =A1.aggregate("employee","[{$group : {_id : \"$DEPT\", num_eid : {$sum : 1}}}]")
5 | =A1.aggregate("employee","[{$group : {_id : \"$DEPT\", num_eid : {$sum : 1},avg_salary:{$avg:\"$SALARY\"}}}]")
6 | =A1.close()

A2: Count the number of eligible files in employee, a collection, using count function. The function’s first parameter is employee, the collection’s name; its second parameter is the filtering criterion: STATE=” California”. The result is count value:

A3: Get the distinct values from employee using distinct function. The function’s first parameter is employee, the collection’s name, and the second parameter is one of the properties, DEPT. The return result is a table sequence consisting of employee’s DEPT without duplicate members:

A4: Group and summarize employee. The first parameter of aggregate function is employee, the collection’s name, and the second one includes the grouping property and aggregate property, which means the data of the collection will be grouped by DEPT and then count the number of files in each group. aggregate function returns a cursor as well, whose data will be fetched all at once using fetch function.

A5: Group and summarize employee, which is the same as the above. But another aggregate property is added to the second parameter of aggregate function, which means grouping the data of the set by DEPT and computing the average value of SALARY in each group.

Next
Previous
Related :
  • 4. MongoDB


Chapter 10 Deep Subject

Next
Previous
Related :
  • 1. Set Thinking
  • 2. Referencing Thinking
  • 3. Using Sequence Numbers and Locating Data
  • 4. Application of Index Sequences
  • 5. Using Cursor


1. Set Thinking

Next
Previous
Related :
  • Chapter 10 Deep Subject
  • 1.1 Sequences and sets in esProc
  • 1.2 Parameters in a loop function
  • 1.3 Order of loop
  • 1.4 Sequence computing
  • 1.5 Impure sets
  • 1.6 Sets consisting of subsets
  • 1.7 Understanding group operation
  • 1.8 Non-equal grouping


1.1 Sequences and sets in esProc

Unlike traditional programming languages, esProc employs sets commonly. In fact, the sequence in esProc belongs to the field of set. So it’s quite important to deeply understand the concept of sets when using esProc. Like an integer and a string, the set is a basic esProc data type; it could be a variable value, a computation of an expression, and a return value of a function.

For this data type, esProc provides operators for operations between two sets A and B, like intersection, concatenation, union, difference: A^B，A|B，A&B，A\B. It is recommended users should deeply understand and master these set operations, in order to start approaching problems more by set thinking. The set-based thinking approach can make good use of existing data to bring about an easier solution.

The following is an illustration of using set operations to simplify code:

 | A
1 | =demo.query("select * from EMPLOYEE")
2 | =A1.select(GENDER=="M")
3 | =A1.select(STATE=="California")
4 | =A2^A3
5 | =A1.select(GENDER=="M" && STATE=="California")
6 | =A2&A3
7 | =A1.select(GENDER=="M" || STATE=="California")
8 | =A2\A3
9 | =A1.select(GENDER=="M" && STATE!="California")

A4 and A5 use different methods to find male employees from the state of California; A6 and A7 select male employees and employees from the state of California using different approaches; A8 and A9 finds male employees from states outside of California in different ways. Note that A6 and A7, despite having got the same employee information, have different orders in which records appear in the results.

Unlike a mathematical set, a set in esProc is arranged orderly and allows duplicate members in it, like a sequence or a table sequence.

 | A
1 | [1,2,3,4]
2 | [1,3,3,2]
3 | =[1,2,3]==[1,3,2]

For instance there are duplicate membes in A2’s sequence. Members in A3’s two sequences have different orders, which makes us believe they are not equal and thus the comparison result is false:

Mathematically, the intersection operation and union operation are both commutative; in other words, A∩B º B∩A and A∪B º B∪A. However, this commutative property is not valid for esProc because the member order in a sequence counts. For esProc, the result of the intersection / union operation is required to be arranged according to the order of the left operand.

 | A
1 | [1,2,3]
2 | [3,1,5]
3 | =A1^A2
4 | =A2^A1
5 | =A1&A2
6 | =A2&A1

The computed results of A3, A4, A5 and A6 are separately listed below:

Because of this ordering feature of sequence members, we should adopt A.eq(B) function to judge whether two sequences have the same members instead of simply using the comparison operator == :

 | A
1 | =[1,2,3]==[3,2,1]
2 | =[1,2,3]==[3,2,1].sort()
3 | =[1,2,3].eq([3,2,1])
4 | =[1,2,3].eq([3,2,2])
5 | =[1,2,2,3].eq([3,2,1,2])
6 | =[1,2,2,3].eq([3,2,3,1])

A1 and A2 judge if the two sequences are equal. The results are:

A3, A4, A5 and A6 judge if the two sequences have the same members. Results are respectively as follows:

Next
Previous
Related :
  • 1. Set Thinking


1.2 Parameters in a loop function

With the data type of set, you can handle many operations on members of sets in a single line of code, without the need of loop code.

 | A
1 | [3,4,1,3,6]
2 | =A1.sum()
3 | =A1.avg()
4 | =A1.max()-A1.min()

A2 calculates the sum of members in the sequence; A3 calculates their average; and A4 finds the difference between the maximum value and the minimum one. Their results are respectively as follows:

Sometimes, the loop functions will not process the set members but the values computed based on these members. In this case, you can use parameters to construct an expression in a function, in which "~" represents the current member.

 | A
1 | [3,4,1,3,6]
2 | =A1.sum(~*~)
3 | =demo.query("select * from EMPLOYEE")
4 | =A3.min(~.BIRTHDAY)
5 | =A3.min(BIRTHDAY)
6 | =A3.avg(age(BIRTHDAY,HIREDATE))

A2 calculates the sum of every sequence member’s square. The result is:

A4, A5 and A6 perform computations based on the table sequence of employee information queried by A3. A4 finds the birth date of the oldest employee. The result is:

"~" in A4’s expression can be omitted; A5 gets the same result as A4 does.

A6 calculates the average hire age of all the employees. The result is:

For the nested loop functions, "~" is interpreted as a member in an inner sequence; in such cases, if you want to reference a member in an outer sequence, precede "~" with the name of the outer sequence.

 | A
1 | [1,2,3]
2 | [-1,-2,-3]
3 | =A1.sum(A2.sum(~*~))
4 | =A1.sum(A2.sum(A1.~*~))
5 | =A1.sum(~*A2.sum(~))
6 | =A1.sum(A1.sum(A1.~*~))

A3 calculates the result of multiplying 3 by the sum of squares of A2’s members. A4 calculates the sum of cross products of members in sequence A1 and sequence A2; A5 is another way of calculating the sum of cross products of members of the two sequences. In A6, the expression in the inner loop fails to reference members of the sequence in the outer loop, so the result is 3 times of the sum of squares of A1’s members. The following are results of A3~A6:

The above rule is also applicable to the field reference where ~ is omitted. In such a case, a field will be first considered as one in an inner record sequence; if such a field cannot be found in the inner record sequence, esProc will search for it in the outer record sequence.

Next
Previous
Related :
  • 1. Set Thinking


1.3 Order of loop

The arguments in a loop function will compute according to order of members in the original sequence. This is a feature of which we can make full use.

 | A
1 | [1,3,2,5,4,8,7]
2 | 0
3 | =A1.(A2=A2+~)
4 | [1,1,0,0,1,0,0,0,1,0,1,0,0,0]
5 | 0
6 | =A4.max(if(~==0,A5=A5+1,A5=0))

By running a loop, A3 calculates a sequence of cumulative sums of members in A1’s sequence:

A6 calculates the maximum number of successive 0s in A4’s sequence:

In many cases, we can use just a single expression to implement the function which can be achieved by simple loop code.

Next
Previous
Related :
  • 1. Set Thinking


1.4 Sequence computing

Different from such loop functions as sum and avg that return a single aggregate value, the A.(x) function for computation on a sequence returns a set. Often, a new set can be created by sequence computing, in addition to using the basic set operations such as union, intersection and difference.

 | A
1 | [1,2,3]
2 | =A1.(~*~)
3 | =A1.(~)
4 | =A1.()
5 | =A1.(1)
6 | =A1.(if(~%2==0,~,0))
7 | I love you
8 | =len(A7).(mid(A7,~,1))
9 | =A8.countif(;~=="o")

The code from A2 to A6 perform computations based on A1’s sequence: A2 calculates each member’s square; both A3 and A4 list sequence members sequentially; A5 lists 1 cyclically; In A6’s loop computation, if the member in A1 is an odd number, return 0; otherwise return the member value. Their results are respectively as follows:

A8 presents A7’s string as a sequence of single characters; A9 counts the number of times letter o appears. Their results are:

The execution of an aggregate function with arguments can be divided into two steps:

Use arguments to produce computed column;

Aggregate the result.

That is, A.f(x) = A.(x).f()

The new function is used to return a table sequence by sequence computing.

 | A
1 | [1,2,3,4,5]
2 | =A1.new(~:Origin,~*~:Square)
3 | =demo.query("select * from EMPLOYEE")
4 | =A3.new(NAME,age(BIRTHDAY):Age)
5 | =A3.new(NAME)
6 | =A3.(NAME)

A2 returns a new table sequence by looping through and computing members of A1’s sequence. “~” in the expression represents the sequence member on the current loop. The result is:

Based on A3’s table sequence, A4 and A5 respectively create a new table sequence with two fields and one with only one field. Their results are:

A6 runs a loop to get a sequence composed of values of NAME field, based on A3’s table sequence. Note that the result is different from that of A5:

In addition, there is a run function for computing sequence. This function returns the original sequences themselves instead of the result of alignment computation. Generally, it is used to assign values to fields in a record sequence (table sequence).

 | A
1 | =demo.query("select * from EMPLOYEE")
2 | =A1.new(NAME,age(BIRTHDAY):Age)
3 | =A2.run(Age=Age+1)

A2 creates a new table sequence, enters names of employees in it and calculates their ages. Based on this table sequence, A3 adds 1 to each employee’s age. As it is the data of original table sequence in A2 that run function modifies, the results of A2 and A3 are the same. The following step-by-step execution shows how A2’s table sequence changes:

Next
Previous
Related :
  • 1. Set Thinking


1.5 Impure sets

esProc has no restriction on the consistency of sequence member types, that is, a sequence may consist of numbers, strings, and records.

 | A
1 | [1,a3,2,5.4,$[4.5],2011-8-8]
2 | =[A1,4]

A1’s sequence includes members of various data types. A2’s sequence is composed of A1’s sequence and a single-value member. The following lists data in A1 and A2:

However, in many cases, it would be of little practical significance to arrange data of various types in one sequence. Therefore, it is not necessary to pay too much attention to it.

On the other hand, a record sequence – a sequence consisting of records – may consist of records from different table sequences; in that case, this feature offers concrete convenience.

 | A
1 | =demo.query("select * from EMPLOYEE")
2 | =demo.query("select * from FAMILY")
3 | =A1|A2
4 | =A3.count(left(GENDER,1)=="F")

A4 counts the number of females in employees and their families. We can perform the computation provided that both employee table and family table contain GENDER field, regardless of their different data structures.

Under esProc, it is not necessary that records in a record sequence originate from the same table sequence. As long as they have the same field names, the records can be processed uniformly. Here, merits of esProc include simpler program writing, higher efficiency and reduced occupation of memory. For SQL, however, two tables with different structures must be united into a new one by using the UNION clause before performing any operations.

Next
Previous
Related :
  • 1. Set Thinking


1.6 Sets consisting of subsets

Since the member of a set in esProc is less restrictive by nature, a set may be a member of another set. If A is a set consisting of other sets, functions A.conj(), A.union(), A.diff(), A.isect() could be employed to compute concatenation, union, difference and intersection between subsets of A.

 | A
1 | [[1,2,3,4,5],[1,3,5,7,9],[2,3,5,7]]
2 | =A1.conj()
3 | =A1.isect()
4 | =A1.(~.sum())
5 | =A1.(~.(~*~))

A1 is a sequence consisting of other sequences. According to A1’s sequence, A2, A3, A4 and A5 respectively obtain a concatenation sequence, an intersection sequence, a result sequence containing sums of members in every sub-sequence, and a new sequence including multiple sequences, members of each of which are squares of each member in every original sub-sequence. These result sequences are as follows:

Also, a record sequence may be a member of a sequence.

 | A
1 | =demo.query("select * from EMPLOYEE")
2 | =A1.select(STATE=="California")
3 | =A1.select(STATE=="Indiana")
4 | =A1.select(STATE=="Florida")
5 | =[A2,A3,A4]
6 | =A5.(~.count())
7 | =A5.(~(1).STATE)
8 | =A5.(STATE)
9 | =A5.new(STATE,~.count():Count)

A2, A3 and A4 respectively retrieve data of employees from California, Indiana and Florida. A5 generates a sequence consisting of these three record sequences. It is a set of sets.

A6 counts the number of employees from the three states, with the following result:

A7 gets name of each of the states. ~(1) in the expression can be omitted, so expressions in A8 and A7 are equal and their results are the same:

A9 creates a table sequence, based on A5’s sequence, to count number of employees from each state. The result is:

Next
Previous
Related :
  • 1. Set Thinking


1.7 Understanding group operation

The group operation is commonly used in SQL. However, not everyone can grasp its essence. By nature, it is to split a set into multiple subsets according to a certain rule. In other words, the return value of a group operation shall be a set consisting of subsets. However, people often do not need to use such a set; instead, they may need to view a part of aggregate values of its subsets. Therefore, group operations are often followed by aggregate operations on subsets.

This is what SQL does. The GROUP BY clause is always followed by an aggregate operation. Because there is no explicit set data type in SQL, a set consisting of subsets cannot be returned directly. Therefore, there is no other choice but to append aggregate operations to group operations.

This explains why one has got used to the conception that the group operations should go hand in hand with the aggregate operations and forgotten that they are in fact two independent operations.

However, sometimes we would still be interested in these grouped subsets, instead of the aggregate values. Even if we might only be interested in the aggregate values, these subsets are worth retaining for reuse in subsequent computations, rather than being discarded once an aggregate operation is completed and being re-generated if needed.

This requires us to understand group operations in its true sense. With set thinking embedded, esProc can achieve this goal. In fact, the basic group functions in esProc are used only for data grouping by removing the aggregate operations.

 | A | B
1 | =demo.query("select * from EMPLOYEE") |
2 | =A1.group(month(BIRTHDAY),day(BIRTHDAY)) | /Group by Birthday
3 | =A2.select(~.len()>1) | /Employees with same birthday as others
4 | =A3.conj() |
5 | =A1.group(STATE) | /Group by State
6 | =A5.new(~(1).STATE:State,~.count():Count) | /Use the group result to create a TSeq and counts number of employees from each state
7 | =A5.new(STATE,~.avg(interval@y(BIRTHDAY,now())):Age) | /Use the group result again to create another TSeq and calculates the average age of employees from each state

The group result is a set consisting of subsets, and the subset can be grouped too. Members in the group result are also sets, which can also be grouped. Both of the two operations will produce a multilayer set.

 | A | B
1 | =demo.query("select * from EMPLOYEE") |
2 | =A1.group(year(BIRTHDAY)) | /Group by the year of birth
3 | =A2.group(int(year(~(1).BIRTHDAY)%100/10)) |
4 | =A2.group(int(year(BIRTHDAY)%100/10)) |
5 | =A2.(~.group(month(BIRTHDAY))) | / Group members of the group result; A3/A4/A5 returns a sequence consisting of record sequences

Because these results have too many layers, they are rarely used in practice. The above example is only to show you the set-based thinking pattern and the nature of the set operations.

Next
Previous
Related :
  • 1. Set Thinking


1.8 Non-equal grouping

Besides the conventional group function, esProc provides align@a() function for perfoming alignment grouping and enum() function for performing enumeration grouping.

The data grouping implemented by the group function is called equal grouping with the following features:

1) Any member in the original set must and can only be in one subset;

There is no empty subset.

The above two features are unavailable for the alignment grouping or the enumeration grouping.

Alignment grouping is an operation that calculates grouping expressions with members in a set and matches the result subsets with values in a specified sequence. For the alignment group, the following steps are required:

2) Specify a group of values;

In the set to be grouped, put members whose certain property values are equal to the specified values to one subset;

Each result subset must correspond to a pre-defined value.

It is possible that a certain member exists in none of the subsets, or an empty set appears, or a member exists in both subsets.

The following case will group employees by specified states sequence:

 | A
1 | =demo.query("select * from EMPLOYEE")
2 | [California,Florida,Chicago]
3 | =A1.align@a(A2,STATE)

A3 performs alignment grouping by A2’s sequence. It matches name of each state with members in A1. This type of grouping may produce the result that certain employees belong to none of the groups, or empty groups that contain not a member at all. A3’s result is as follows:

Enumeration grouping is defined as follows: First, specify a group of conditions, take the members in the set to be grouped as arguments and compute these conditions, members satisfying these conditions will be grouped into a subset; each subset is corresponding to a pre-defined condition. A member may be in none of these subsets, or in two subsets at the same time; besides, an empty set may appear.

The following case will group employees by specified age groups:

 | A
1 | =demo.query("select * from EMPLOYEE")
2 | [?<=30,? <=40,?>40]
3 | =A1.enum(A2,age(BIRTHDAY))
4 | [?<=30,?>20 && ?<=40,?>50]
5 | =A1.enum@r(A4,age(BIRTHDAY))

A3 performs enumeration grouping by A2’s conditional sequence. An employee record in A1 can be be in none of the subgroups but is not allowed to exist in two subgroups. A3’s result is as follows:

In this case, an employee may be put into the group satisfying the first condition. Though employees who are below 30 satisfy the condition that employees who are 40 or below, they will not be distributed to the second group.

A5 also performs enumeration grouping by A2’s conditional sequence. Here @r option is used to allow repeated grouping. So an employee record may appear in more than one group. A4’s result is as follows:

Although it seems that these two functions differ greatly with group function, you will understand, having the nature of group operations in mind, that the three functions actually do the same thing – to split a set into multiple subsets. The only difference is that they split sets in different ways.

Next
Previous
Related :
  • 1. Set Thinking


2. Referencing Thinking

Next
Previous
Related :
  • Chapter 10 Deep Subject
  • 2.1 Foreign key referencing
  • 2.2 Referencing sets


2.1 Foreign key referencing

The foreign key is a common concept for various relational databases. In a relational table, you can define one or multiple fields as the foreign key. The foreign key field values can be associated with the data in another table. For example, in the game scoring report, player scores are usually recorded with the player ID. To get the details of this player from another player profile table using the player ID, you can query in the player profile table according to the player ID. In this case, the player ID field can be set as the foreign key.

The foreign key can associate two relational tables and help ensure the data consistency, so as to facilitate some cascade operations. Take the above game scoring report as an example. If defining the player ID as the foreign key field, then the player name, country, age, and other data can be queried at the same time while getting the game scoring record of the player.

In this example of foreign key definition, it is actually equal to referencing the records in the player data table from the game scoring record table. The core idea of foreign key concept is to reference another record.

In esProc, unlike the database, there is no particular requirement on the data type of record fields. In other words, esProc users can get the value at will. Thanks to this convenience, in the esProc TSeq, fields can be directly assigned with records for referencing. The foreign key associating can thus be established naturally and straightforwardly.

For example, create an employee education background table comprising the employee ID field and education field. This example is designed to illustrate the way the esProc foreign key references, and the education field is simply set through the looping in proper order:

 | A | B | C
1 | =create(ID,Education) | [BS,BA,MS,MA,MBA,MEE,MPA,PhD] | =B1.len()
2 | for 500 | =B1((A2-1)%C1+1) |
1. 3 |  | >A1.insert(0,A2,B2) |

At this point, associate this TSeq with employee table EMPLOYEE. A foreign key field Info can be added to the education table so as to reference the records in the table EMPLOYEE with the foreign key field values:

 | A | B | C
1 | =create(ID,Education) | [BS,BA,MS,MA,MBA,MEE,MPA,PhD] | =B1.len()
2 | for 500 | =B1((A2-1)%C1+1) |
3 |  | >A1.insert(0,A2,B2) |
2. 4 | $ select * from EMPLOYEE | =A1.derive(A4.select@1(EID==ID):Info) |

To this point, the TSeq in B4 is as follows. In this TSeq, the Info field is the employee record on which you can double-click to query.

With the record reference, the foreign-key-related query becomes more convenient to implement. The query statement gets much more clear and intuitive. The record fields can be used in the same way of using the normal fields or records. For example, list the data of serial number, full name, education background, state, and department:

 | A | B | C
1 | =create(ID,Education) | [BS,BA,MS,MA,MBA,MEE,MPA,PhD] | =B1.len()
2 | for 500 | =B1((A2-1)%C1+1) |
3 |  | >A1.insert(0,A2,B2) |
4 | $ select * from EMPLOYEE | =A1.derive(A4.select@1(EID==ID):Info) |
3. 5 | =B4.select(Info.STATE== "Ohio") | =A5.new(ID,Info.NAME+" "+ Info.SURNAME:FullName,Education, Info.STATE:State,Info.DEPT:Dept) |

The following is B5’s result:

In esProc, not only the record of other TSeq but also that of the current TSeq can be assigned to the foreign key field.

In esProc, besides the function A.derive() for assigning a certain field of records with the objects of entire records when adding fields to the newly-generated TSeq; the function A.run() can be used to assign value to the existing fields. For example:

 | A | B | C
1 | =create(ID,Education) | [BS,BA,MS,MA,MBA,MEE,MPA,PhD] | =B1.len()
2 | for 500 | =B1((A2-1)%C1+1) |
3 |  | >A1.insert(0,A2,B2) |
4. 4 | $ select * from EMPLOYEE | >A1.run(ID=A4.select@1(EID==A1.ID)) |

In B4, modify the employee serial number field in the Education Background TSeq to the corresponding records. In A1, the TSeq is as follows:

If the primary key of the corresponding table is a single field, you can also use the function A.switch():

 | A | B | C
1 | =create(ID,Education) | [BS,BA,MS,MA,MBA,MEE,MPA,PhD] | =B1.len()
2 | for 500 | =B1((A2-1)%C1+1) |
3 |  | >A1.insert(0,A2,B2) |
5. 4 | $ select * from EMPLOYEE | >A1.switch(ID,A4) |

The TSeq in A1 is the same to that in the previous example, but much more concise and clear. One thing to note is that the switch function can be used to have the primary key and specified fields in theTSeq associated automatically. If the primary key of TSeq is not the first field, you need to set it with the function T.primary(). For example, the STATE field of employee table is associated with the STATE data table:

 | A | B | C
1 | =create(ID,Education) | [BS,BA,MS,MA,MBA,MEE,MPA,PhD] | =B1.len()
2 | for 500 | =B1((A2-1)%C1+1) |
3 |  | >A1.insert(0,A2,B2) |
4 | $ select * from EMPLOYEE | >A1.switch(ID,A4) |
6. 5 | $ select * from STATES | >A4.switch(STATE,A5.primary(NAME)) |

In this case, the State information can be further viewed in the employee data of TSeq in A1:

To this point, the query or computation based on multi-table association can be further processed. For example, find the state, in which the employees are located, has a population of below 1,000,000:

 | A | B | C
1 | =create(ID,Education) | [BS,BA,MS,MA,MBA,MEE,MPA,PhD] | =B1.len()
2 | for 500 | =B1((A2-1)%C1+1) |
3 |  | >A1.insert(0,A2,B2) |
4 | $ select * from EMPLOYEE | >A1.switch(ID,A4) |
5 | $ select * from STATES | >A4.switch(STATE,A5.primary(NAME)) |
7. 6 | =A1.select(ID.STATE.POPULATION< 1000000) |  |

As can be seen, the record-referencing foreign key can make the query syntax of multi-table association more concise and readable with an improved computing speed.

Next
Previous
Related :
  • 2. Referencing Thinking


2.2 Referencing sets

In assignment, esProc fields can be assigned with records or sets in a similar way - referencing the set in the foreign key field. In most cases, what is referenced in the foreign key fields is the RSeq composed of records from other TSeqs. Please notice that the “Foreign Key Field” is not the same thing as the foreign key field in the database. It is much more like the query to reference the sub table data from the primary table with SQL.

Like the foreign key referencing, fields can be added when generating the TSeq. With the function derive, the foreign key field can be assigned with RSeqs, for example:

 | A
1 | $ select NAME,ABBR from STATES
2 | $ select * from EMPLOYEE
8. 3 | =A1.derive(A2.select(STATE==A1.NAME):Employees)

In A3, add foreign key field Employees to A1, and assign this field with the employees in corresponding states, then return a new TSeq. By doing so, all values of Employees fields in the TSeq are RSeq, as shown below:

For the foreign key field referencing sets, they can be called directly as the fields of TSeq. The values of foreign key field can be taken as the set for direct use in the computation. For example, add the field Count and compute the total employees in each state:

 | A
1 | $ select NAME,ABBR from STATES
2 | $ select * from EMPLOYEE
3 | =A1.derive(A2.select(STATE==A1.NAME):Employees)
9. 4 | =A3.derive(Employees.count():Count)

The total number of employees can be computed with the foreign key, and the resulting TSeq is shown in A4:

Like records being referenced, the sets or RSeqs are referenced in the foreign key field. The expressions on foreign key field can also be used to solve the multi-table-association query, filtering, and other problems. For example, find the states in which there are more than 50 employees:

 | A
1 | $ select NAME,ABBR from STATES
2 | $ select * from EMPLOYEE
3 | =A1.derive(A2.select(STATE==A1.NAME):Employees)
4 | =A3.derive(Employees.count():Count)
10. 5 | =A4.select(Employees.len()>50)

By referencing sets in foreign key field, the syntax of multi-table-association query can be more clear and intuitive. In the TSeq of A4, the sole purpose of introducing the field Count is to have a more clear view of the result. This field is not used when filtering records in A5.

When referencing sets in a TSeq field, the switch function cannot be used, but the function A.run() can be used to assign values to the existing fields. For example:

 | A
1 | =12.new(#:ID,string(date("2000-"+string(#)+"-1"),"MMMM"):Month,Employees)
2 | $ select EID,NAME+' '+SURNAME FULLNAME, BIRTHDAY from EMPLOYEE
11. 3 | >A1.run(Employees=A2.select(month(BIRTHDAY)==A1.ID))

In A3, assign the null-value field Employees with the records of employees born in each month. Once executed, the TSeq in A1 is as follows:

In fact, the result of grouping an RSeq is just a sequence composed of sets. It is similar to the field referencing single sets. For example, if grouped by the birth month of employee, the code is:

 | A
1 | $ select EID,NAME+' '+SURNAME FULLNAME, BIRTHDAY from EMPLOYEE
12. 2 | =A1.group(month(BIRTHDAY))

The grouping result is as follows:

Like the result in the above example, the result of grouping can be further used in various computations if necessary:

 | A
1 | $ select EID,NAME+' '+SURNAME FULLNAME, BIRTHDAY from EMPLOYEE
2 | =A1.group(month(BIRTHDAY))
3 | =A2.new(#:Month,~.count():Count)
13. 4 | =A2.new(#:Month,~.minp(BIRTHDAY).FULLNAME:Oldest)

In A3, count the total number of employees who were born in each month. The oldest employees born in each month are listed in A4. Once executed, A3 and A4 get the following results:

Next
Previous
Related :
  • 2. Referencing Thinking


3. Using Sequence Numbers and Locating Data

Next
Previous
Related :
  • Chapter 10 Deep Subject
  • 3.1 Accessing members
  • 3.2 Accessing subsets
  • 3.3 Locating data using loop functions
  • 3.4 Alignment access
  • 3.5 Aligning sequences
  • 3.6 Interval integer sequence
  • 3.7 ISeq consisting of sequence numbers
  • 3.8 Locating computation


3.1 Accessing members

Members in a set of esProc (sequence) are organized in order. Therefore, you can reference a member in a set with its sequence number. The more flexible use of sequence numbers, the better esProc functions to make operations simpler and more efficient. Thus sequence numbers or an integer sequence (ISeq) of sequence numbers are recommended to work with certain esProc functions, such as delete() function for deleting records, and compose() function for re-sorting a table sequence (TSeq).

The simplest application is to access members with their sequence numbers directly; this is the same as what you would do to access an array with a programming language.

 | A
1 | [1,3,5,7,9]
2 | =A1(1)
3 | =A1(3)
4 | >A1(2)=4
5 | >A1(4)=8

A2 and A3 get members by specified positions. Their results are:

A4 and A5 modified A1’s sequence. The following shows the changing step by step:

You can use m(i) function to get members backwards or in a loop manner. It is a serviceable complement to A(i) function.

 | A
1 | [1,3,5,7,9]
2 | =A1.m(3)
3 | =A1.m(-2)
4 | =A1.m@r(6)
5 | =A1.m@r(12)
6 | =A1.m(6)

A2 and A3 get values of members by specified sequence numbers, using A.m() function. -2 represents the second-to-last member. With @r option, A4 and A5 will get members cyclically if the specified sequence number exceeds the number of members. For example the sequence number 12 will get the second member after it circulates through A1’s members twice. Computations from A2 to A5 get the following results:

Without the @r option, A6 returns the null value as its specified sequence number 6 exceeds the sequence’s length.

In addition, esProc provides a series of p-headed functions for finding sequence numbers of members, as given below:

 | A
1 | [3,5,1,9,7]
2 | =A1.pos(5)
3 | =A1.pmin()
4 | =A1.pmax()
5 | =A1.pselect(~%5==0)

A2 finds the sequence number by a specified member value. If there are multiple members with the same value, return only the sequence number of the first member. A3 and A4 respectively return sequence numbers of members of the maximum and minimum value. A5 finds the sequence number of the first member that is a multiple of 5. Computations from A2 to A5 are as follows:

pos function will return null if a specified member is not found in a sequence. This function can be used to judge whether a member is in a certain set or not.

 | A
1 | [3,5,1,9,7]
2 | =A1.pos(1)!=null
3 | =A1.pos(2)!=null

Computations of A2 and A3 are respectively as follows:

Next
Previous
Related :
  • 3. Using Sequence Numbers and Locating Data


3.2 Accessing subsets

With sequence numbers, you can access the subsets of a set.

 | A
1 | [3,5,4,6,1]
2 | =A1([1,3,5])
3 | =A1([3,5,2])
4 | =A1([4,1,3,1])
5 | >A1([1,3,5])=[12,43,28]
6 | >A1([2,4,3])=0

A2, A3 and A4 get subsets of the sequence. Different from getting members, here the operations use ISeqs composed of sequence numbers as their parameters. Results are as follows:

A5 and A6 modify members of the sequence in one go using sequence parameters. A1’s sequence changes step by step as follows

You can also use the m() function to access a subset by specifying the corresponding sequence numbers.

 | A
1 | [3,5,4,6,1]
2 | =A1.m([1,-1])
3 | =A1.m@r([1,6,12])
4 | =A1.m@0([1,6,3])

You can use a negative number in A.m() function’s ISeq parameter to represent a position counted backwards, or you can use @r option with the function to indicate a turn-back if a specified sequence number exceeds the boundary. Besides, @0 option means the corresponding null value of a sequence number that is beyond the sequence, if any, will not appear in the result. Computations of A2, A3 and A4 are:

If there is the @a option in a locate function, the sequence numbers of all members satisfying the specified condition will be returned.

 | A
1 | [3,2,1,9,6,9,1,2,8]
2 | =A1.pos@a(2)
3 | =A1.pmin@a()
4 | =A1.pmax@a()
5 | =A1.pselect@a(~%2==0)

With @a option, A2 returns all positions of 2 in A1’s sequence; A3 returns sequence numbers of all members having the smallest value; A4 returns sequence numbers of all members having the biggest value; and A5 returns sequence numbers of all members that are multiples of 2. Their computations are:

If you want to get the positions of multiple members at one time using pos function, the @i option may be required in certain cases.

 | A
1 | [3,2,1,9,6,9,1,2,8]
2 | =A1.pos@i([2,9,8])
3 | =A1.pos@i([3,1,1,1])
4 | =A1.pos@i([1,2,3])
5 | =A1.pos([1,2,3])
6 | =A1.pos([1,1,2,2])
7 | =A1.pos([1,1,1,2,2,2,3,3])

A.pos@i() finds positions of members given in the sequence parameter in one-way sequence. Without @i, A.pos () simply judges if every member of the parameter sequence is included in sequence A. Computations from A2 to A7 are:

Both A3 and A4 return nulls, because A3 didn’t find the third 1 specified by the parameter sequence in A1’s sequence and A4 couldn’t find 1, 2, 3 in a one-way sequence. A.pos@i() returns only an increasing ISeq; if not all members are found, it returns null.

The pos@i function returns null if a certain given member is not found in a sequence. Considering the order it requires and the duplicate members, you cannot simply use this function to judge if the specified subset is contained; instead, you should use an intersection operation.

 | A
1 | [3,2,1,9,6,9,1,2,8]
2 | =A1.pos@i([1,9,6])!=null
3 | =A1.pos@i([1,2,3])==null
4 | =A1.pos([1,2,3,4])==null
5 | =A1.pos([1,2,3,3])!=null
6 | [1,1,2,2]
7 | [1,1,2,2,3,3]
8 | =A6^A1==A6
9 | =A7^A1==A7

Computations from A2 to A4 are:

When A.pos@i(B) is used to query and judge data, the non-null result indicates that members of B can be found in A sequentially and that A must contain B. While a null result only indicates that B cannot be found in A sequentially, it does not necessarily mean A does not contain B. One example is A3’s case.

If result of A.pos () is null, there must be certain members of B that you cannot find in A and it is necessary that A does not contain B. But from the non-null result you cannot deduce that A contains B, if there are duplicate members in B. A5 is such an example.

The following are computations of A8 and A9:

It is feasible to judge if A contains B using the formula B^A==B. Results of A8 and A9 prove that A1 contains A6 but it does not contain A7. Note that the operands in the formula should not be commuted, otherwise the order of members in computation of A^B may be different from that in B and the judgment cannot be rightly made.

Next
Previous
Related :
  • 3. Using Sequence Numbers and Locating Data


3.3 Locating data using loop functions

Like the symbol ~, # in a loop function represents the sequence number of the current member.

 | A
1 | [5,4,3,2,1]
2 | =A1.(#)
3 | =A1.(#+~)
4 | =A1.select(#%3==2)
5 | =A1.group(int((#-1)/2))

A2 gets an ISeq consisting of sequence numbers; A3 gets a result sequence by adding each member and its sequence number. Using select function, A4 finds the second ones of every three members in A1’s sequence, i.e. members whose sequence numbers are 2, 5, 8,…, and creates a sequence with them. A5 groups A1’s sequence every two members. The following lists their computations:

In a loop function, esProc allows using the symbol [] to access members in relative positions.

 | A
1 | [1,2,3,4,5]
2 | =A1.(~[0])
3 | =A1.(~[1])
4 | =A1.((~-~[-1])/~[-1])
5 | =demo.query("select * from STOCKRECORDS where STOCKID=000062").sort(DATE)
6 | =A5.((CLOSING-CLOSING[-1])/CLOSING[-1])
7 | 0
8 | =A5.max(if(CLOSING>CLOSING[-1],A7=A7+1,A7=0))

A2 lists each member of the sequence; A3 gets every member that is next to the current member; A4 calculates the ratio between each member and the previous one. Their computations are:

A5 finds records of the stock with a specified ID; A6 calculates growth rate of every day’s stock price; then A8 calculates the maximum number of successive rising days. Computations of A6 and A8 are:

In addition, you can use the symbol {} to get members of subsets by relative positions.

 | A
1 | [1,2,3,4,5]
2 | =A1.(~{-1,1})
3 | =A1.(~{-1,1}.avg())
4 | =A1.(~{1-#,0}.sum())
5 | =A1.(~{,0}.sum())
6 | =A1.(~{0,}.sum())

For each position, A2 gets the current member, the member before it and the one after it. A3 calculates moving average for each position. Both A4 and A5 calculate the cumulative sum. A6 calculates the reversed cumulative sum, that is, the sum of a decreasing number of members. Computations from A2 to A6 are:

Next
Previous
Related :
  • 3. Using Sequence Numbers and Locating Data


3.4 Alignment access

As we know, the symbol # in a loop function represents the sequence number of the current member. In fact, it is nothing but a number which can be calculated as any others. In particular, it can be used as a sequence number to access a member in another sequence, which makes alignment access possible.

 | A
1 | [1,2,3,4,5]
2 | =A1.(A1(#))
3 | =A1.(A1.m(#-1))
4 | [5,4,3,2,1]
5 | =A1.(~+A4(#))
6 | =A1++A4
7 | =10.(if(#%2==1,A1((#-1)/2+1),A4(#/2)))

In loop computations, you can use # in an expression to represent the current sequence number. Computations of A2, A3, A5, A6 and A7 are respectively as follows:

When independent sequences are arranged in the same order, the use of alignment access may create an effect similar to accessing fields of a record.

 | A
1 | [Bray,Jacob,Michael,John]
2 | [65,87,98,72]
3 | [76,82,78,88]
4 | =A1.rank(A2(#)+A3(#))
5 | =A1.new(~:name,A4(#):rank)

A4 calculates ranks of total scores, getting related scores by positions. A5 creates a table sequence with name field and rank field by linking two sequences together according to member positions. Computations of A4 and A5 are:

Next
Previous
Related :
  • 3. Using Sequence Numbers and Locating Data


3.5 Aligning sequences

Before an alignment access is executed, it is necessary that all the sequences are arranged in the same order. However, in practice, sequences are not always in the same order. Under such circumstances, you should use the align function to re-order sequences according to the order of a certain sequence to make them in the same order.

 | A | B
1 | =demo.query("select * from EMPLOYEE") | 3) /The EMPLOYEE table
2 | =demo.query("select * from ATTENDANCE").align(A1:EID,EMPLOYEEID) | 4) /The ATTENDANCE table is aligned with the EID field of the EMPLOYEE table
3 | =demo.query("select * from PERFORMANCE").align(A1: EID, EMPLOYEEID) | 5) /The PERFORMANCE table is aligned with the EID field of the EMPLOYEE table
4 | =A1.new(NAME,SALARY*(1+A2(#).ABSENCE+A3(#).EVALUATION):salaryPaid) | 6) /A new table sequence is created to calculate salaries(here, A1, A2 and A3 are organized in the same order)
5 | =demo.query("select * from GYMSCORE where EVENT='Vault'") | 7) /The score table for Vault
6 | =demo.query("select * from GYMSCORE where EVENT='Floor'").align(A5:NAME,NAME) | 8) / The score table for Floor; it is aligned with NAME
7 | =A5.(SCORE*0.6+A6(#).SCORE *0.4) | 9) / Calculate weighted scores
8 | =A7.rank() | 10) /Return a ranking of weighted scores
9 | =A5.new(NAME,A7(#):score,A8(#):rank) | 11) / Create a table sequence to get athlete names, weighted scores and ranks

Both table sequences in A2 and A3 have been aligned to A1’s employee IDs. A4 gets the table sequence of employees’ salaries:

In A6, data is aligned with names of the athletes in A5’s table sequence. Then A7 calculates weighted scores and A8 gets the ranking of weighted scores. Finally, A9 organizes the data and gets the result table sequence:

In fact, an align function align@a will also return a sequence aligned with the standard sequence. With members being sets, it can also be accessed with the alignment access approach.

 | A | B
1 | =demo.query("select * from EMPLOYEE") | 12) /The EMPLOYEE table
2 | [California,Texas,Pennsylvania] | 13)
3 | =A1.align@a(A2,STATE) | 14) /Align grouping by A2’s STATE field
4 | =A3.new(A2(#):STATE,~.count():Count,round(~.avg(age(BIRTHDAY)),2):Age) | 15) /Use # to search for, in A2, the values of the fields by group numbers in A3

When computations are completed, A4 gets the following result:

Without option, align function fetches the first member corresponding to each member of the standard sequence from the source sequence and returns a set consisting of these first members, instead of returning a set consisting of subsets. If we already know there is only one member in each subgroup, using this function is to sort these members according to a standard sequence.

Similarly, the alignment access can be used in enumeration group operation; here, enum@1 is not commonly used.

 | A
1 | =demo.query("select * from EMPLOYEE")
2 | [AgeGroup1,AgeGroup2,AgeGroup3]
3 | [?<=30,?>30 && ?<=40,?>40]
4 | =A1.enum(A3,age(BIRTHDAY))
5 | =A4.new(A2(#):AgeInterval,~.count():Count)

A5 calculates the number of employees for each of the three age groups:

Next
Previous
Related :
  • 3. Using Sequence Numbers and Locating Data


3.6 Interval integer sequence

An integer sequence is a special set that is applicable to all the set operations. In addition, it can be used as sequence numbers for accessing subsets in another sequence. A flexible use of integer sequences is vital for starting approaching problems with sequence numbers.

 | A
1 | =to(10)
2 | =to(3,8)
3 | =A1.step(3,2)
4 | =20.step(4,2,3)

to function generates a sequence consisting of consecutive integers. With step function, you can set intervals between members of an ISeq and other parameters. Computations from A1 to A4 are:

You can process subsets by an integer sequence consisting of the subsets’ positions in the original set.

 | A | B
1 | =to(100) | 16) /Return a sequence consisting of numbers 1, 2, …100
2 | =A1(100.step(14,7))=0 | 17) /Assign 0 to the members whose sequence numbers are multiple of 7 (from the 14th member)
3 | =A1.run(if(~>1,A1(100.step(~,~+~))=0,0)) | 18)
4 | =A1.select(~>1) | 19) /Generate a list of prime numbers; then, assign 0 to the members whose sequence numbers are composite numbers; other non-zero numbers are prime numbers
5 | =100.(rand()) | 20) /Generate 100 random numbers
6 | =A5(to(50)) | 21)
7 | =A5(to(51,100)) | 22)
8 | >A5(100.step(2,1))=A6 | 23)
9 | >A5(100.step(2,2))=A7 | 24) /Shuffle A5; that is, alternatively exchange members in the first half and the second half

Next
Previous
Related :
  • 3. Using Sequence Numbers and Locating Data


3.7 ISeq consisting of sequence numbers

After a sequence is ordered, the previous order of the members in the sequence will be lost. However, in certain conditions, this order information may be required. For example, we may need to know the entry order of the three oldest employees in the company, the amount of increase of a share’s price for the three trading days when the share prices are on the highest level, and so on.

This problem can be solved by using the psort function in esProc; the function returns the previous order of the ordered members.

 | A
1 | [c,b,a,d]
2 | =A1.psort()
3 | =A1(A2)
4 | =A1.sort()
5 | =A3==A4

Computations from A2 to A5 are:

In plain words, in an integer sequence returned by the psort function, the first number is, relative to the original sequence, the sequence number of the member which should be placed in the first place in the sorted sequence; the second member is, relative to the original sequence, the sequence number of the member which should be placed in the second place; the rest may be deduced by analogy.

For the sequence resulting from the sequence number ISeq, you can also use the inv() function to get the inverse ISeq of the sequence number ISeq, which can be used to undo operations.

 | A
1 | [c,b,d,a]
2 | =A1.sort()
3 | =A1.psort().inv()
4 | =A2(A3)
5 | =A4==A1

Computations from A2 to A5 are:

You can use the psort function to solve the above problem which requires that the original sequence numbers should be kept.

 | A | B
1 | =demo.query("select * from EMPLOYEE ").sort(HIREDATE) | 25)
2 | =A1.psort(BIRTHDAY:-1) | 26) /Return an ISeq of sequence numbers of the records ordered by Birthday
3 | =A2(to(3)) | 27) /The youngest three employees’ positions in A1
4 | =demo.query("select * from STOCKRECORDS where STOCKID=000062").sort(DATE) | 28)
5 | =A4.psort(CLOSING:-1) | 29) /Return an ISeq of sequence numbers of the records reversely sorted by ClosingPrice
6 | =A5(to(3)) | 30) /The positions of the highest three CLOSING in A4
7 | =A6.(A4(~).CLOSING/A4.m@0(~-1).CLOSING-1) | 31) /The increase for these three days; the increase shall be computed according to the positions of records sorted by TradeDate, that is, the positions in A4
8 | =A6.(A4.calc(~,(CLOSING-CLOSING[-1])/CLOSING[-1])) | 32) /Use the calc function to simplify A7’s expression

A binary search is widely recognized for its high efficiency; however, it requires that an original sequence is sorted by keywords. So, before a binary search is executed, the original sequence must be sorted. However, this is not suitable for all. For example, if you want to search for a member in a sequence, you can of course run the sort function before the searching; but if you want to search for an index of a member, running the sort function before searching would damage order; in this case ,you should use psort() function.

 | A | B
1 | =demo.query("select * from EMPLOYEE ").sort(HIREDATE) | 33)
2 | =A1.psort(NAME) | 34) /The integer sequence of sequence numbers, after the EMPLOYEE table is sorted by NAME
3 | =A1(A2) | 35) /The record sequence formed after the EMPLOYEE table is sorted by NAME
4 | =A3.pselect@b(NAME:"David") | 36) / Use the binary search to search for the sequence number of David in A3
5 | =A2(A4) | 37) / David’s sequence number in the original record sequence

In this case, psort creates a binary search index for the sequence; there could be one or more search indexes, depending on different keywords, for a single sequence.

An alignment group function can also return an ISeq consisting of sequence numbers, instead of the sequence aligned.

 | A | B
1 | =demo.query("select * from SALES").sort(AMOUNT:-1) | 38) /The ranking of AMOUNT
2 | [QUICK,ERNSH,HANAR,SAVEA] | 39)
3 | =A1.align@1p(A2,CLIENT) | 40) /Alignment grouping by A2’s sequence of clients; sequence numbers are returned
4 | =A3.new(A2(#):NAME,A1(~).AMOUNT: Amount,~:Rank) | 41) / Use A3’s group number to search for the corresponding amount and ranks

Next
Previous
Related :
  • 3. Using Sequence Numbers and Locating Data


3.8 Locating computation

After working out sequence numbers of records needed, we can compute the required results with locating computation A.calc().The locating computation can avoid unnecessary computation and increase efficiency.

 | A
1 | =file("VoteRecord")
2 | =A1.import@b()
3 | [Califonia,Ohio,Illinois]
4 | =A2.pselect@a(A3.pos(State)>0)
5 | =A2.calc(A4,Votes[-1]-Votes+1)

Computations of A2, A4 and A5 are as follows:

In this case, the binary file VoteRecord stores poll results, with a descending sort of the votes. A4 is the computed result of employee ID sequence of a specified state. A5 calculates the number of votes they need in order to moving up. For example, Ryan Williams, now ranking 3rd, needs another 69 votes to move up one place. Here inter-row operation is needed, because the computation needs more than the data of selected employees.

Next
Previous
Related :
  • 3. Using Sequence Numbers and Locating Data


4. Application of Index Sequences

In databases, creating appropriate indexes for tables can greatly increase query efficiency. Similarly, index sequences can be created for record sequences or table sequences in esProc to increase efficiency in querying data repeatedly.

For example, we need to access the food order Order_Foods.txt repeatedly.

1 | 2 A | 3 B | 4 C
5 1 | =file("Order_Foods.txt").import@t() | 50000 | 1000
2 | =A1(C1.(rand(B1)+1)).new(PName, Quantity) |  |

Records of food orders queried by A1 are as follows (altogether 50,000 records):

A2 gets 1,000 random records of food orders with only their product names and purchase quantities. The two fields will be query conditions for use in the later test query (here repetitions are allowed). Data in A2 are as follows:

In the following, in order to explore the role of index sequences, we’ll query the data of food orders in A1 according to the 1,000 product names in A2 with and without an index sequence respectively.

First let’s look at the situation without an index sequence. Since records in Order_Foods.txt are sorted by Date, i.e. the order dates, binary search cannot be used when searching by product names, otherwise errors will occur.

6 | 7 A | 8 B | 9 C
10 1 | =file("Order_Foods.txt").import@t() | 50000 | 1000
11 2 | =A1(C1.(rand(B1)+1)).new(PName, Quantity) | =now() |
3 | =A2.(A1.select@1(PName==A2.PName && Quantity ==A2.Quantity)) | =interval@ms(B2,now()) |

Expressions in B2 and B3 get the current time through now() function and roughly estimate query time (millisecond). Query results are stored in B3 as follows:

Estimated time computed in B3 is as follows:

Then let’s move to the situation where an index sequence is used:

12 | 13 A | 14 B | 15 C
16 1 | =file("Order_Foods.txt").import@t() | 50000 | 1000
17 2 | =A1(C1.(rand(B1)+1)).new(PName, Quantity) | =now() |
18 3 | =A2.(A1.select@1(PName==A2.PName && Quantity ==A2.Quantity)) | =interval@ms(B2,now()) | =now()
19 4 | =A1.psort(PName, Quantity) |  |
5 | =A2.(A1(A4.select@b1(cmp(A1(~).([PName,Quantity]), A2.~.([PName,Quantity]))))) | =interval@ms(C3,now()) |

First create an index sequence corresponding to PName and Quantity, thus binary search can be used to make query by making use of indexes. In order to compare the efficiency of the two situations, time for creating an index sequence is also included. The index sequence created in A4 is as follows:

As binary search is used in A5 to query data, the query condition should be modified to mode x==0. Results are the same as those in A3:

Estimated time computed in B5 is as follows:

By comparing results in B3 and B5, it can be seen that the second method is much more efficient. That is to say, query speed can be significantly increased by using binary search on the condition that an index sequence is created. Note that computations are involved to create index sequences. The more we use an index sequence to query, the more efficient the query becomes. It is not so necessary to create an index sequence if the computation is not query-intensive.

For specific databases and query modes, it is unnecessary to create the index sequence each time query is executed. The index sequence can be stored after it is created. For example:

20 | 21 A
22 1 | =file("Order_Foods.txt").import@t()
23 2 | =A1.psort(PName, Quantity).new(~:Pos)
3 | >file("OF_Index_PQ").export@b(A2)

Thus there is no need to recreate the index sequence for the next query. Simply import the index file.

24 | 25 A | 26 B | 27 C
28 1 | =file("Order_Foods.txt").import@t() | 50000 | 1000
29 2 | =A1(C1.(rand(B1)+1)).new(PName, Quantity) | =now() |
30 3 | =file("OF_Index_PQ").import@b() |  |
4 | =A2.(A1(A3.select@b1(cmp(A1(A3.Pos).([PName, Quantity]), A2.~.([PName,Quantity]))).Pos)) | =interval@ms(B2,now()) |

In this way, the query speed is faster than creating a new one.

During establishing the foreign key relationship with switch function for a table sequence or a cursor, each record needs to find its corresponding reference in the dimension table. This is similar to the query-intensive situation in the above example. esProc will create the index sequence automatically for the execution of switch function. For example:

 | A | B
1 | =file("PersonnelInfo.txt") | =demo.query("select STATEID, NAME, ABBR from STATES")
2 | =now() |
3 | =A1.cursor@t() | =A3.switch(State,B1:ABBR)
4 | for 50 | =B3.fetch(1000)
5 | >B3.close() | =interval@ms(A2,now())
6 | =now() |
7 | =A1.cursor@t() |
8 | for 50 | =A7.fetch(1000)
9 |  | >B8.run(State=B1.select@1(ABBR==State))
10 | >A7.close() | =interval@ms(A6,now())

From the 2nd to the 5th line, switch function is used to relate personnel information stored in the cursor to corresponding states information. 1,000 rows are fetched each of the 50 times by loop. B5 computes the time consumed to perform the processing. From the 6th to the 10th line, run function is used to query the corresponding states data by loop, and similarly, 1,000 rows are fetched each of the 50 times of loop. B10 computes the time consumed for this processing. The results of B5 and B10 are shown below:

In this case, the processing efficiency is apparently higher with switch function, despite that there is only a small amount of states data. Actually the bigger the data in the dimension table, the more efficient the processing will be by using switch function.

Next
Previous
Related :
  • Chapter 10 Deep Subject


5. Using Cursor

With esProc, you can handle the structured big data easily, and perform data computing and analysis without difficulty. Because the big data table cannot be loaded into the memory all at once, the basic solution is using cursor to handle it.

Next
Previous
Related :
  • Chapter 10 Deep Subject
  • 5.1 Basic principle of cursor
  • 5.2 Inheritance feature of the cursor
  • 5.3 Basic cursor computations


5.1 Basic principle of cursor

The esProc cursor is like the database cursor in the stored procedure: In fact, the cursor is a pointer that points to a row of data. Records can thus be retrieved one by one according to its position, yet will not be totally returned. Through the cursor, the data retrieval can be completed progressively to ensure the system memory won’t become overburdened.

In order to facilitate the usage, esProc cursor does not need to access records one by one. However, esProc also makes rules for the use of cursor. In esProc cursor, moving backward is forbidden, only moving forward is allowed, which is similar to TYPE_FORWARD_ONLY type of result set in JDBC. In this way, esProc cursor can only traverse the records in a table sequence once, and the related computations will be completed in this traversal. This differs from the computational mode of normal table sequences. Once a traversal is completed, the cursor will close automatically and become invalid. So, records can no longer be fetched from it anymore.

In esProc,a cursor can be generated from a big database table or a big data file. Sometimes, you can generate a cursor using the existing TSeq in the memory for the computation. For example:

 | A
1 | =file(“PersonnelInfo.txt”)
2 | $select * from CITIES
3 | =A1.cursor()
4 | =demo.cursor("select * from EMPLOYEE")
5 | =A2.cursor()

In which, A3 holds a file cursor, A4 holds a database cursor, and A5 holds an in-memory record sequence cursor.

When using the cursor, the function cs.fetch() can be used to retrieve one or multiple consecutive rows according to the position(s) specified by the cursor for computing or debugging. You can see the article esProc External Memory Computing: Concept of Cursor to learn different types of cursor and their differences.

Next
Previous
Related :
  • 5. Using Cursor


5.2 Inheritance feature of the cursor

When using cursor to handle data computation, the result returned by cursor is usually the records from a data table or result set. However, the data in cursor is usually not the desired final result, and needs reprocessing.

When handling the query result from the cursor, you may need to perform filtering, summarizing, sorting, and other jobs, sometimes even data combining or merging for multiple cursors. To do these jobs, in esProc, you can directly use a cursor to generate a new one, yet with no need to execute data query. The data retrieval will be performed when you actually fetch the data. For example, to complete a certain data processing job, use cursor A to generate new cursor B. Cursor B will use A to retrieve data, process the retrieved data and return the result. Based on cursor B, you can generate cursor C as necessary, and further process the data in B……For join and merge operations, a new cursor can be generated from multiple cursors.

In the mode of cascaded inheritances, the complex data processing job can be divided into multiple steps to accomplish, with a specific purpose for each step. Each cursor retrieves data through the original cursor when doing the processing. So, data is fetched actually from the original cursor for each data computation. Such mode ensures that the original data can only be traversed once. This avoids data conflict as well as reduces the frequency of database access while keeping a high computational efficiency.

The cascaded inheritance approach processes data in the cursor without extra operations. In addition, each stage of cursor object only takes very limited amount of memory. So, this mode will not compromise the performance of accessing the external storage, and has little impact on the computational performance by adding more inheritance stages.

Next
Previous
Related :
  • 5. Using Cursor


5.3 Basic cursor computations

Next
Previous
Related :
  • 5. Using Cursor
  • 5.3.1 Basic usage of cursor
  • 5.3.2 Filtering cursor
  • 5.3.3 Creating and modifying cursor
  • 5.3.4 Foreign key in cursor


5.3.1 Basic usage of cursor

Once the cursor is created, the cs.fetch() function can be used to retrieve data from data table. You can also use cs.skip() function to skip multiple rows . To close the cursor before all data is traversed, you can use cs.close(). For example:

 | A | B | C | D
1 | =file("PersonnelInfo.txt") |  |  |
2 | =A1.cursor@t() |  |  |
3 | for | =A2.fetch(1) |  |
4 |  | if B3==null | break |
5 |  | else | =age(B3.Birthday) |
6 |  |  | if C5>40 | break
7 | >A2.close() |  |  |
8 | =if(B3==null,null,B3.Name) |  |  |

This cellset is designed to find the name of the first employee who is above 40. Once the file cursor is generated in A2, run a loop in A3, and retrieve one employee record from B3 each time. If all records are traversed with the cursor, then the loop will break in C4; otherwise C6 will judge if the employee is over 40. If the condition is met, then exit the loop and close the cursor. In A8, you will see the result:

Usually a cursor retrieves multiple records at one time:

 | A | B | C
1 | =file("PersonnelInfo.txt") |  |
2 | =A1.cursor@t() | 0 |
3 | for | =A2.fetch(100) |
4 |  | if B3==null | break
5 |  | else | =B3.count(age(B3.Birthday)>40)
6 |  |  | >B2=B2+C5
7 | =B2 |  |

In the above program, all data in the text file cursor in A2 will be traversed, retrieving 100 rows each time and computing the total number of employees above 40. The final result can be viewed in A7:

Because the cursor automatically close when all data is traversed, there is no need to use cs.close(). Please note that the cs.fetch() returns a table sequence each time, instead of returing an RSeq composed of records from a same TSeq.

Next
Previous
Related :
  • 5.3 Basic cursor computations


5.3.2 Filtering cursor

When you need to filter the desired data out from the cursor, you can use cs.select() function . Thus, the above problem can be solved with the following method:

 | A | B | C
1 | =file("PersonnelInfo.txt") |  |
2 | =A1.cursor@t() | 0 |
3 | =A2.select(age(Birthday)>40) |  |
4 | for | =A3.fetch(100) |
5 |  | if B4==null | break
6 |  | else | >B2=B2+B4.len()
7 | =B2 |  |

The final result in A7 is the same as above:

Because generally the result of filtering big data in the cursor is also the big data, cs.select() will return a cursor too. Further processing of the returned big data often includes aggregate operation and others. If there is small amount of data in the result, then all can be returned at once. For example, find female employees from California who are above 40:

 | A
1 | =file("PersonnelInfo.txt")
2 | =A1.cursor@t()
3 | =A2.select(age(Birthday)>40)
4 | =A3.select(State=="CA")
5 | =A4.select(Gender=="F")
6 | =A5.fetch()

In A6, you will see the results:

In the computation, multiple inheritance of cursor is used for filtering step by step. Data can be retrieved from text data table and computed to get the required result only if they are retrieved in A6. This method almost will not affect performance.

Next
Previous
Related :
  • 5.3 Basic cursor computations


5.3.3 Creating and modifying cursor

Sometimes, the data retrieved from big data table is not the final result you need, and further computation is required to get it. In such cases, you can use the cs.new() function to generate a new cursor:

 | A
1 | =file("PersonnelInfo.txt")
2 | =A1.cursor@t()
3 | =A2.new(ID,Name,age(Birthday):Age,Gender,State)
4 | =A3.select(Age>40)
5 | =A4.select(State=="CA")
6 | =A5.select(Gender=="F")
7 | =A6.fetch()

Likewise, data will be fetched from text data table at A7’s instruction and computed to get the required data. The result of A7 is shown below:

Besides the cs.new()for generating new data, cs.run() function can be used to modify the fields of records directly. For example:

 | A
1 | =file("PersonnelInfo.txt")
2 | =A1.cursor@t()
3 | =A2.new(ID,Name,age(Birthday):Age,Gender,State)
4 | =A3.select(Age>40)
5 | =A4.select(State=="CA")
6 | =A5.select(Gender=="F")
7 | =A6.run(Gender="Female",State="California")
8 | =A7.fetch()

The result in A8 is shown below:

Next
Previous
Related :
  • 5.3 Basic cursor computations


5.3.4 Foreign key in cursor

Sometimes, the big data table requires foreign keys to relate to data in other tables. In this case, you can use cs.switch() function, for example:

 | A
1 | =file("PersonnelInfo.txt")
2 | =A1.cursor@t()
3 | =A2.select(age(Birthday)>40)
4 | =A3.select(left(Name, 1)=="W")
5 | =A4.select(Gender=="F")
6 | $select ABBR, NAME, CAPITAL from STATES
7 | =A5.switch(State, A6)
8 | =A7.fetch()

In this example, find the profiles for all female employees whose name initials are W and above 40. A7 sets the State field as the foreign key, and references the records of table sequence in A6.The result of A8 is shown below:

Next
Previous
Related :
  • 5.3 Basic cursor computations