SIREN — Interactive and visual geospatial redescription mining

Formats

Data formats

For redescription mining, one considers entities discribed by variables divided into two sets, hereafter arbitrarily called left-hand side and right-hand side. This can be seen as a pair of data matrices, where entities are identified with rows and variables with columns. Both sets of variables describe the same entities, hence, the matrices have the same number of rows.

In Siren, data include:

Variables:
The variables describing the entities are divided in two sets. They can be of three types:
1. Boolean,
2. categorical,
3. or real-valued.
Obviously, this is required.
Variable names:
Optional additional information, providing names for the variables.
Coordinates:
Additional location information, geographic coordinates of the entities. This makes the data geospatial.

Data can be imported to Siren via the interface menu File → Import → Import Data, either as a set of separate files (Menu: Import from separate files) or as a single XML file (Menu: Import from XML file). Below, we present the data formats supported by Siren.

Multiple data files

In case the data is imported as a set of files, the program expects:

Left-hand side variables file:
file containing the left-hand side variables.
Right-hand side variables file:
file containing the right-hand side variables.
Coordinates file:
file containing the entities coordinates.

Variables formats:

Right-hand side and left-hand side variables are stored in separate files. This import mode does not allow for mixed types of variables, contrarily to the single XML data import mode. Several formats are available:

Dense format means variables are stored one per line, with values separated by commas. This corresponds roughly to storing your data matrix transposed, in csv format.
Sparse format means each row has the form
```
entity_id [SPC] variable_id [SPC] value
```
To make sure no empty entity or variable is omitted, the first row must be
```
nb_of_entities [SPC] nb_of_variables [SPC] 0
```
This correspond roughly to Matlab representation of sparse matrices.
Dat format means variables are stored one per line, only the ids of entities for which the variable holds true are stored, separated by commas.

The format is determined from the filename extension, below is the table of extensions, - means the format is not available for the given variable type:

	dense	sparse	dat
1. Boolean	.densebool	.sparsebool	.datbool
2. categorical	.densecat	-	-
3. numerical	.densenum	.sparsenum	-

Coordinates format:

Each entity should be located by a pair of coordinates (i.e. latitude and longitude). Coordinates are stored one entity per line, with the two coordinates separated by a comma. If the number of coordinates found in the coordinates file does not match the number of entities, coordinates will be ommitted.

Names:

Finally, the files containing the names of variables are obtained from the corresponding data filename by replacing its extension by .names. For example, if the LHS variables data file is some_path/LHS_file.densenum the program expect to find the LHS variables names in a file called some_path/LHS_file.names. Names of variables are stored one variable per line. Names may contain spaces and unicode characters but no equality or comparison signs (i.e. =,>,<). If the number of names found in the names file does not match the number of variables on the given side, names will be ommitted.

Single XML file

Here is an example of a simple XML data file. In this example, the data consist of two Boolean variables on the left-hand side, one categorical variable and two numerical variables on the right-hand side, describing a total of ten geolocated entities.

Each variable is stored between variable tags, with it name, type_id, number of entities (numbers of entities should match accross all variables on both sides), etc.

For Boolean variables (cf. A_red and B_red in example above) the ids of entities for wich the variable holds true are separated by commas and stored between rows tags (cf. dat format).
For categorical variables (cf. A_blue in example above) the categories of the entities are separated by commas and stored in order between values tags (cf. dense format).
Numerical variables can be stored either in sparse or dense format, depending on convenience. The format used should be indicated using store_type tags.
- Dense format (cf. B_blue in example above) means the values are separated by commas and stored in order between values tags (cf. dense format).
- Sparse format (cf. C_blue in example above) means the pairs (entity id:value) are stored only for non-default values, separated by commas between values tags.

Inside a siren package, the data is stored as a single XML file.

Redescriptions formats

The product of redescription mining is a list of redescriptions. A redescription consist of a pair of queries over the variables describing the entities, one query for each set. The two sets of variables are arbitrarily called left-hand side and right-hand side, and so are the corresponding queries.

The support of a query is the set of entities for which the query holds. Any given redescription partitions the entities into four sets:

E_{1,0} is the set of rows for which only the left hand side query holds,
E_{0,1} is the set of rows for which only the right hand side query holds,
E_{1,1} is the set of rows for which both queries hold,
and E_{0,0} is the set of rows for which neither of the queries hold.

Redescriptions can be imported to Siren via the interface menu File → Import → Import Redescriptions. More importantly, they can be exported via the interface menu File → Export Redescriptions. Below, we present the redescription formats supported by Siren.

Queries

A query is formed by combining literal using Boolean operators.

For several reasons, Siren evaluates the queries from left to right irrelevant of the operator precedence. In other words, it supports only queries that can be parsed in linear order, without trees. For example, (a ∧ b) ∨ ¬c is supported, but (a ∧ b) ∨ (c ∧ d) is not. Parenthesis delimiting groups of literals combined with the same operator can be added to ease readability.

We consider three types of literals, defined over a Boolean, categorical or numerical variable respectively.

Below is an unformal grammar of Siren's query language, parenthesis denote optional elements. The preferred syntax for editing queries is marked with bold.

conjunction operator	AND	&, ∧, \land
disjunction operator	OR	\|, ∨, \lor
operator	OP	AND, OR
negation	NEG	!, ¬, \neg
variable	VAR	integer, name
category	CAT	integer
interval bound	IBD	float with at least one decimal precision
less-than sign	LEQ	<, ≤, \leq{}
Boolean literal	BLIT	(NEG) VAR
categorical literal	CLIT	(NEG) VAR = CAT
		VAR (\not)\in CAT
		VAR ∈ CAT
		VAR ∉ CAT
numerical literal	NLIT	(NEG) [ (IBD LEQ) VAR (LEQ IBD) ]
numerical literal	NLIT	(NEG) VAR (> IBD) (< IBD)
literal	LIT	BLIT, CLIT, NLIT
query	QRY	LIT (OP LIT)*

Naturally, the type of literal and the type of variable should match, i.e., [4.0 < Va < 8.32] is a valid numerical literal only if the corresponding variable Va is a numerical variable. Furthermore, the upper bound of a numerical variable should always be greater or equal to the lower bound and either of them should be specified.

Redescription statistics

The statistics of a redescription include:

accuracy, as measured by Jaccard coefficient |E_{1,1}| / (|E_{1,0}|+|E_{1,1}|+|E_{0,1}|),
p-value,
cardinality of the support sets E_{1,0}, E_{0,1}, E_{1,1}, E_{0,0} (sometimes also referred to as alpha, beta, gamma and delta, respectively).

Exporting Redescriptions

Redescriptions can be exported in three formats, determine by the extension of the provided filename:

.xml (default)
Each redescription is stored between redescription tags, with both queries and basic statistics.
.queries
One redescription per line, with both queries and basic statistics tab separated.
.tex
Produces a tex file that can be compiled to obtain a table of the redescriptions

When exporting redescriptions in the latter two formats, diabled redescriptions will not be printed.

Importing Redescriptions

Both .queries and .xml format can be imported into Siren, .tex cannot.

Inside a siren package, the redescriptions are stored in a XML file together with display order and disabled status.

Siren --- Last modified: Wed Aug 1 2012, galbrun@cs.helsinki.fi

Formats

Data formats

Variables:

Variable names:

Coordinates:

Multiple data files

Left-hand side variables file:

Right-hand side variables file:

Coordinates file: