For example in Telecom graphs can be built using phone calls data. The people you call are your first circle. They are relatives, colleagues or friends. You value those people and listen to their opinions. If one of your friends uses mobile internet the telecom operator can offer this service to you with a high probability of purchase.

Social networks like Facebook can find out your first circle using your “friends list” or monitoring the personal pages you visit. The advertising you saw on Facebook could be shown to you because one of your friends clicked on it earlier.

Social Networks are also important in debt collection. The colleagues and friends can influence the debtor and make him pay the debt. This is why banks and collection agencies do actively collect contact information of your friends, neighbors and colleagues. It sometimes happens that the debt is payed by the friends or relatives, not the debtor.

For software companies their experts are the most valued asset. Every person in the company should have access to the expert’s knowledge. The social network graph can show whether the expert is actively helping other colleagues or is he isolated from others. This graph can use data on internal mail and phone conversations.

For example the graph above is based on internet forum SQL.ru → OLAP and DWH. The nodes are the forum members and the edges show whether the member took part in other member’s thread. Every edge has a weight that equals the number of member A posts in member’s B thread plus number of member B posts in member’s A thread.

At first I made a list of all the 3 000+ forum members and added edges. The graph looked like a black spot on a monitor. I removed all the edges with the weights less than 10 and deleted all the members left without edges. That is the last graph in the video. Then I continued to delete edges with a minimal weights till there was only one edge left. That is the first graph in the video. Then I put the graphs in the video in the reverse order, starting with the smallest graph to the biggest one.

The video bellow is based on the forum SQL.ru → Просто треп (just chat).

Consumers with good credit histories can get a new loan with a lower interest rate. But one has to know which CB stores its credit history and what banks can request that history from CB. If your credit history is poor for example you had delinquent loans you better look for a bank that don’t request your credit history.

Russian Central Bank (RCB) web site allows finding the CB’s where ones histories are stored. According to RCB web site there are 800+ credit organizations and 30+ credit bureaus in Russia. Most of the credit histories are stored in the five biggest CB’s: Equifax, Expirian-Interfax, NBKI, Infocredit and MBKI.

Banks don’t like to publish the list of CB’s they work with. But some information is available online. My task was to develop the schema of banks and credit bureaus cooperation. For instance if one sentence contains both bank and credit bureau names it is very probable they do exchange information with each other. But I had to exclude all the sentences containing more than one bank or CB names because that sentence can be a list of some forum participants.

The schema above was developed using 5 000+ html pages that contain at least one of the five biggest CB’s. Unfortunately schema doesn’t show the direction of the information exchange. The bank can request information from one CB and report credit histories to the other. The calculation of data exchange direction is a next task as well as the increase of the number of banks and CB’s.

I couldn’t simply delete the gender because it is often an important factor in the model. Thus I decided to replace the gender with a new column calculated using clients’ patronymics. The thing is that most Russian male patronymics are derived from father’s name by adding “ich” like “Ivanovich” and “Ilyich”. Most female patronymics end in “na” like “Ivanovna” and “Ilyinichna”.

Most of the rest patronymics belong to Turkic peoples. Their surnames end in “-ogli” meaning son or “-kizi” meaning daughter. Thus one can recalculate the gender using SAS the following way:

data test;
  input patronymic $ 1-20;

  length gender $10;

  if prxmatch('/ich$|ogli$/', lowcase(trim(patronymic))) > 0 then gender = 'male';
    else if prxmatch('/na$|kizi$/', lowcase(trim(patronymic))) > 0 then gender = 'female';
	  else gender = 'unknown';

datalines;
Ivanovich
Ilyich
Ivanovna
Ilyinichna
Hamzat-ogli
Suleyman-kizi
;
run;

The Oracle code:

drop table names;
create table names(patronymic varchar2(20));

insert into names values('Ivanovich');
insert into names values('Ilyich');
insert into names values(‘Ivanovna’);
insert into names values('Ilyinichna’);
insert into names values('Hamzat-ogli ');
insert into names values(‘Suleyman-kizi’);
select patronymic,
       case when REGEXP_LIKE(lower(patronymic), 'ich$|ogli$') then 'male'
            when REGEXP_LIKE(lower(patronymic), 'na$|kizi$') then 'female'
            else                                                  'unknown'
       end as gender
from names;

This algorithm recalculated the gender for the 99% of the clients. Patronymics column contained missing for the rest 1% of the clients. But the algorithm can be improved to take into account names and family names.

proc sql;
  create table test as
  select *
  from dictionary.columns;
quit;

  “Well you rashly gave me access to your product DB and now I can get any information out of it.”
  “Hey, be careful!”
  “For instance I can get all the column names in your database.”
  “Hm… Nice!”

I was also impressed but not with the query.

  “Looks like some of your columns have Russian names.”
  “So what?”
  “I thought that SAS columns should contain English characters only.”
  “Well, seems this interview is not a waste of time for you. You’ve learned something new.”

This feature really impressed me. At SAS courses I learned that column names can contain only English letters, numbers or underscores. But Russian names didn’t fit this rule.

Later I forgot about this incident till once I had to import a lot of Excel files with Russian names. The IMPORT procedure converts Russian names into F1, F2, F3, … according to the order of names in an Excel file. But VALIDVARNAME option allows columns with Russian names for example:

options validvarname=any;
data work.test;
  'да - это русское имя'n = 1;
run;

One can also import Excel files with Russian names and use them in a code:

proc sql;
  select sum('да - это русское имя'n)
  from test;
quit;

Looks like SAS has its own double standards.

The first thing one should start with is indexes. They decrease table read time if one of the columns in the where statement is indexed. They also decrease tables join/merge time if one of the ID columns is indexed. The list of DWH indexes is available in the system table DICTIONARY.INDEXES:

proc sql;
  create table work.indexes_list as
  select *
  from dictionary.indexes;
quit;

Indexes can be simple and combined. Simple index is created on one column. Combined index involves several columns. The main difference is that combined index is more efficient then simple index when query involves filter or join based on several columns. SAS compiler decides what index to use depending on the query code and indexes available.

Indexes have advantages and draw backs. Indexes do reduce query time but it takes time and additional disk space to calculate and store them. Ideally indexes should be created on every column that can be used in a query where or join statements. But calculation of all possible indexes can take too much time. One has to rely on his own experience and gut feel to decide what columns to index.

Index renewal

DWH is updated with new data every day and old indexes become useless. That is why indexes are modified each time the table updates. There are two ways of updating indexes:

• remove the index and calculate it again
• update the existing index without removal

The first way takes the most time but results in better index quality. The second approach updates index with the new information that takes less time but results in worse index quality. SAS “optimizer” can delete and recalculate the index automatically if it decides that its quality is too low.

Compress option

SAS file format is very rarefied. One can compress the SAS table 10 times using RAR or ZIP archiver. SAS also has built in algorithm that is activated with compress option:

data work.test(compress = yes);
  ...
run;

or in the libname statement:

libname example 'c:\dwh\data\example' compress = yes;

SPDE

SPDE (Scalable Performance Data Engine) is a new SAS library type. The old and default type is V9. But SPDE can process data faster than V9. That is done by splitting SAS tables into several files and using of several CPU to process the table. SPDE does not require additional license and comes with Base SAS starting from SAS 9.1. More about SPDE here.

Defragmentation

One can imagine hard disk as a huge line. When new file is saved the beginning of the file is written in any free segment of the line. When the file is too big to fit the segment the file is being split into several fragments. The more fragments the more time it takes to read the file. When the DWH is updated a lot of new files are being written to the disk.

Defragmentation is a process that rewrites the files to reduce the number of fragmented ones. It is probably the most simple and the most efficient DWH optimization technique.

Optimizing hard disk usage

DWH server hard disks as a rule have different characteristics. The most important for the DWH are the size and read/write speed. The fastest disk should be used for the WORK library as it is the most extensively used by users.

The slowest disks should contain OS files, DWH software (SAS) and the tables being copied from sources as network speed is usually slower than the disk write speed.

One can also store tables within one library but on different disks. It is useful when library size grows bigger than any disk available:

libname example ('c:\DWH\data\example' 'd:\DWH\data\example');
libname sample (sasuser sashelp);

Metrics

The article started with the optimization methods. But one should start optimization project with defining the DWH metrics. Good examples are process load and the volume of data red/written to the DWH. Metrics are used to assess whether the optimization methods work or not. A lot of metrics are available in the Windows Performance Monitor: Start → Control Panel → Administrative Tools → Performance.