%macro _list(n,pre=ff);
    %if &n=1 %then &pre.1;
     %else %_list(%eval(&n-1)),&pre.&n;
%mend _list;

%put %_list(3); *produces ff1, ff2, ff3;

But when I read one of Ian Whitlock’s papers, Names, Names, Names – Make Me a List (SGF 2007, SESUG 2008),  I say: stop! I’m gonna use Ian’s %range and I create Github page to hold it (with minimum modifications due to personal preference).

I once posted My Collection of SAS Macro Repositories credited to some SAS gurus like Richard DeVenezia. When facing a programming challenge, there is always a trade-off: should I take a look at what others wrote, or I just write from the scratch? Searching also needs lots of efforts, so I plan to utilize Github pages to minimum my own searching efforts and hope it would be helpful for you (no intelligence waste anymore!). I begin with SAS list processing:

https://github.com/Jiangtang/Programming-SAS/tree/master/ListProcessing

I got most of such utilities macros (with detailed comments, examples and sources) from papers, blogs and other websites and honors belong to their authors! Sometimes I will also add my own if I think there are some holes to fill up. To get start, you may read a READ ME (will keep updated) first. Besides the individual macros, a combined file (trigged by a simple Dos command) is also available.

[NOTE2013-05-30: This post is permanently suspended. In this post, I only focused on the initialization effect of the %local statement and totally ignored its function to avoid variable collisions as Quentin mentioned in comment box.]

A piece of SAS codes to create a list of all variables within a dataset (a nice programming trick from Art Carpenter, 2004):

%macro getvars(dset) ;
%local varlist ;
%let fid = %sysfunc(open(&dset)) ;
%if &fid %then %do ;
%do i=1 %to %sysfunc(attrn(&fid,nvars)) ;
%let varlist= &varlist %sysfunc(varname(&fid,&i));
%end ;
%let fid = %sysfunc(close(&fid)) ;
%end ;
&varlist
%mend ;

%put %getvars(%str(sashelp.iris));

One question would be: why declare macro variable varlist as local explicitly by a %local statement? Since it was defined within the macro %getvar, it just went to the local macro table automatically and the %local statement is not necessary anymore!

Such argument is definitely right. But the %local statement above was not intent to localize a macro variable, actually, it’s used to initiate it. If you just delete it, you will get a warning then an error:

WARNING: Apparent symbolic reference VARLIST not resolved.
ERROR: The text expression &VARLIST SPECIES contains a recursive reference to the macro variable VARLIST.  The
macro variable will be assigned the null value.

This is because the macro variable &varlist in right side of the second %let statement was not declared before. To be clear, you can also replace the %local statement  with an explicit initiating statement:

%let varlist=;

In this example, the %local statement and the %let statement simply do the same job and the choice is totally subject to programmers’ preference(well I prefer the later).

So, take home question: any comments on the following snippet to count the numbers of word in a string(also from Art Carpenter, 2004; sorry Carpenter, your books and papers are the most familiar sources for me to learn SAS macro programming):

%macro wordcount(list);
    %local count;
%let count=0;
%do %while(%qscan(&list,&count+1,%str( )) ne %str());
%let count = %eval(&count+1);
%end;
&count
%mend wordcount;

%put %wordcount(a b cd);

I had a new baby! He was delivered on time (and on budget!), lions tigers and bears, oh my… His brother is Tiger so I named him, Leo.

And I got the latest SAS 9.3 (TS1M2) installed! SAS is jus getting much beautiful.

OpenCDISC had the latest release, Version 1.4 with the new SDTM 3.1.3 validation checks,—and yes CDISC itself also had some significant updates:

SDTMv1.3 and SDTMIGv3.1.3 now have the machine readable metadata online. It’s a nice improvement (last year I just posted The Great, Open, Vendor-neutral, Platform-independent Data Standards, . . . Yet in PDF Formats).

Define-XML now turns to 2.0 (finally).

R had its final 2.* release, Version 2.15.3 and Version 3.0.0 will just come soon. RStudio also had a update recently. RStudio is the best IDE (not just R IDE) I used.

Google will shut down Google Reader, the best RSS reader ever. It’s a huge loss and I tell you, for example, the famous SAS and statistical blogger Wensui Liu, once frequently posted on Windows Live Space, and then Blogspot and finally WordPress. The former two blogs were closed and Google Reader feed is the only way to archive these lost posts!

To get started, Wendy McHenry posted a nice introductory blog, Why I love SAS metadata and Angela Hall, How did you find that metadata?, Jennifer Parks, Two methods for editing SAS metadata.

To communicate with SAS Metadata Server programmingly, you can use Java or Base SAS  which are both well documented. Chris Hemedinger recently shared bunch of posts on how to use PowerShell to talk with SAS Metadata Server.

Industry gurus Paul Homes and Gregory Nelson also begin to blog in SAS User Groups on this subject (Paul’s own website is a great source too).

A community blog “BI Notes” also has some interesting posts on SAS metadata. Check it out.

In Art Carpenter’s latest book, Carpenter’s Guide to Innovative SAS Techniques, a data step approach to transpose data (from long to wide) works like (Ch2.4.2):

data tst;
    input type $ grp value $3.;
datalines;
A 1 a
A 2 aa
A 3 aaa
B 1 b
B 2 bb
B 3 bbb
C 1 c
C 2 cc
C 3 ccc
;

data art(keep=type grp1-grp3);
   set tst;
   by type;
   retain grp1-grp3 ;
   array grps {3} $ grp1-grp3;
   if first.type then do i = 1 to 3;
      grps{i} = " ";
   end;

   grps{grp} = value;
   if last.type then output art ;
run;

And such logic can be best demonstrated by a DoW Loop:

data dow(keep=type grp1-grp3);
     array grps[3] $ grp1-grp3;
     do _n_ = 1 by 1 until(last.type);
        set tst;
        by type;
        grps[grp]=value;
     end;
run;

/*Note*/

1. The traditional PROC TRANSPOSE approach:

proc transpose data=tst
               out=trans(drop=_:)
               prefix=grp;
   by type;
   id grp;
   var value;
run;

2. Why use data step approach (both Art and DoW) to transpose data against the TRANSPOSE procedure:

• it’s much faster since data step array used
• save codes when complex transformation needed
• last but not least, it’s cool!

3. Arthur Tabachneck maintains a general data step transposing macro, %transpose and you can call it like:

%transpose(data=tst, out=mac,
            by=type, var=value,
            id=grp)

data a;
b=1;
run;

proc datasets noprint;
delete a;
run;
quit;

and the dataset was deleted as I expected:

32   proc datasets noprint;
33       delete a;
34   run;

NOTE: Deleting WORK.A (memtype=DATA).
35   quit;

But when tested in a SAS 9.2 machine, I got errors

25   proc datasets noprint;
——-
22
202
NOTE: Enter RUN; to continue or QUIT; to end the procedure.
ERROR 22-322: Syntax error, expecting one of the following: ;, ALTER, DD, DDNAME, DETAILS,
FORCE, GENNUM, KILL, LIB, LIBRARY, MEMTYPE, MT, MTYPE, NODETAILS, NOFS, NOLIST,
NOWARN, PROTECT, PW, READ.
ERROR 202-322: The option or parameter is not recognized and will be ignored.
26       delete a;
27   run;

NOTE: Statements not processed because of errors noted above.
27 !     quit;

NOTE: The SAS System stopped processing this step because of errors.

Then I checked the SAS HelpDoc, there is no such “noprint” option in PROC DATASETS statement, but in CONTENTS statement in the PROC DATASETS PROCEDURE. Actually I added the “noprint” option just because I expected it act like the same option in PROC FREQ statement to suppresses output. And (un)fortunately, it worked in my machine…

UPDATE(2013-02-27): Just got feedback from a SAS developer this “noprint” will be documented in the upcoming SAS 9.4; actually it works like the existing “nolist” option.

During my initial stay in US last year, one of the interesting exercises was to write a check. The Arabic numerals (the universal language!) were pretty intuitive while I didn’t feel much comfortable on spelling the face value in words: I never played the game!

But the good side of this story was, as a SAS programmer and I used a WORDFw. Format:

data _null_;
    money=8.15;
    put money wordf100.;
run;

and got:

Another check with a bigger value:

2

I paid specially interested on SAS Format recently because of the introducing Perl Regular Expression into PROC FORMAT invalue statement since SAS 9.3(an old procedure plays a new game!). For example, a labeled TIME8. informat created by

proc format;
    invalue xxx (default=20)
      ‘/(\d+):(\d\d)(?:\.(\d+))?/‘ (REGEXP) = [time8.];
run;

For more, see Rick Langston’s paper, Using the New Features in PROC FORMAT (2012).

But after viewing one of WG5’s working reports, Sharing Standard Statistical Scripts and getting to know why they finally chose Google Code (rather than Github which was also tested by WG5 members), I think it’s necessary to clarify some misunderstanding against Github where I’m also an occasional user.

As stated in Slide 11 in the report mentioned before, Github,

Too complicated an interface
Too much overhead for simple development
Too much training and education needed

designed for classic programming languages like C and Java (not for things like R and SAS)

For the first point regarding interface, it seems only Git command line tested, and it may be too complicated to “classic statistical programming users”. Actually, Github offers a great GUI tool, for example, GitHub for Windows to help users visually clone repositories, commit changes and other management tasks without typing Git commands:

It’s also worthy to mention that with GitHub for Windows, users don’t need to install any separated version control software like Git, CVS or SVN. GitHub for Windows already includes a fully functional version of msysGit. It just makes users’ life much simpler. To use Google Code, you must install and configure something like TortoiseSVN.

The second, is Github suitable for “things like R and SAS”? It’s true that all hosts including Github are dominated by “classic programming languages like C and Java”. For SAS, SAS programmers as a whole are just not active in  any social coding activities, but for R, actually it is one of the mostly used languages in Github.

Google Code is good and a “Google Code vs Github” question is just mostly subjective. It seems to me the pickup of Google Code by WG5 rather than Github was based on incomplete information. I personally prefer Github and there are also some good reasons:

• Use the GUI tool, GitHub for Windows to maintain a minimum Git/SVN/CVS setup.
• Github supplies much richer statistics reports, including charts.
• Github is more social oriented which makes it cool in this Web2.0 world.

data t1;
    input ID $ ;
datalines;
T20
T4
T3
T1
;

and want to get such intuitive orderings (files sorting in Window 7 directory):

But when apply the default sorting:

proc sort data=t1 out=t2;
    by ID;
run;

I get:

T1
T20
T3
T4

To produce what expected, add a SORTSEQ option:

proc sort data=t1 out=t3  
    SORTSEQ=LINGUISTIC(NUMERIC_COLLATION=ON)
          ;
    by ID;
run;

T1
T3
T4
T20

In the first block of code, the default sorting is determined by their characters’ appearance in EBCDIC or the ASCII tables (according to OS). To change this default collating sequences, a specific linguistic collation (numeric collation) option added.

For details, see the corresponding part in SAS SORT Procedure and Collating Sequence in SAS(R) 9.3 National Language Support (NLS) with a great paper.

I personally nominated SAS ODS Report Writing Interface was the one of the best technology I found in SAS in 2012. It can generates reports cell by cell and row by row and has much flexibility to produce highly customized reports. Basically, to use it,

• first assign an ODS destination. Nothing new, and I prefer HTML like

ods html   file="output.html" style=sasweb;

• then create a object(instance) based on the ODS Report Writing class, odsout,

declare odsout myout();

ODS Report Writing Interface holds some so called object oriented features. Odsout is a predefined class (or, a SET, a CONTAINER), then you take from it a instance or an object, myout(or any other legal SAS variable names). Suppose Odsout is a class for apples, then myout is a specific apple.

• apply methods associated with the Odsout class, like CELL, ROW, TABLE to generate reports.

Roughly methods are just the functions (subroutines, procedures) in the object oriented world. For example, in SAS Data Steps, you use function weight(apple) to get the weight of the apple; in object oriented world, you use apple.weight() to return the same thing. In ODS Report Writing Interface, if you want to get a table, use TABLE methods:

• myout.TABLE_START() to start a table
• myout.TABLE_END to end a table

Then all we need to do next is to use the flexible SAS data steps to leverage the ODS Report Writing Interface methods (see docs). The codes, you can see below, are pretty verbose compared to its counterparts, PROC REPORT, but that’s why it gains power to build highly customized reports. It’s also very structural (and easy to build, like playing blocks):

/*making a sortable HTML table*/
%macro ods_html_sort_table;
    <script src=’http://goo.gl/Pg0GB’></script>
    <script src=’http://goo.gl/ruKEb’></script>
    <script>$(document).ready(function(){$(‘.table’).tablesorter({widgets: ['zebra']});});</script>
%mend;

title ;
ods listing close;
ods html   file="a:\test\iris.html" style=sasweb headtext="%ods_html_sort_table";

data _null;
    set sashelp.iris;
    by Species;
   
/*    create an object, obj, based on the ODS Report Writing class, odsout*/
    if _n_ = 1 then do;
        dcl odsout obj();
    end;

    if (first.Species) then do; *by group processing;
       obj.title(text: "Fisher’s Iris Data Set by Species"); *title;

/*       start a table*/
       obj.table_start();
               obj.row_start();
              if (Species = "Setosa") then
                 obj.image(file: "Iris_setosa.jpg" );*insert image;
              else if (Species = "Versicolor") then
                 obj.image(file: "Iris_versicolor.jpg" );
              else if (Species = "Virginica") then
                 obj.image(file: "Iris_virginica.jpg" );
            obj.row_end();

            obj.row_start();
                obj.format_cell(text: "Iris Species",  overrides: "fontweight=bold just=right" );
                obj.format_cell(text: Species, column_span: 3, overrides: "just=left");
            obj.row_end();

            obj.row_start();
                obj.format_cell(text: "Unit",  overrides: "fontweight=bold just=right" );
                obj.format_cell(text: "(mm)", column_span: 3, overrides: "just=left");
            obj.row_end();
       obj.table_end();

       /* start another table */
       obj.table_start();
            obj.head_start();
                obj.row_start();
                    obj.format_cell(text: "Sepal Length" , overrides: "fontweight=bold");
                    obj.format_cell(text: "Sepal Width" , overrides: "fontweight=bold");
                    obj.format_cell(text: "Petal Length" , overrides: "fontweight=bold");
                    obj.format_cell(text: "Petal Width" , overrides: "fontweight=bold");
                obj.row_end();
            obj.head_end();
    end;

        obj.row_start();
            obj.format_cell(data: SepalLength);
            obj.format_cell(data: SepalWidth);
            obj.format_cell(data: PetalWidth);
            obj.format_cell(data: SepalLength);
        obj.row_end();

    if (last.Species) then do;
        obj.table_end();

        obj.note(data: "Note: These Tables are Sortable."); *note;

        obj.foot_start(); *footer;
            obj.row_start();
                obj.cell_start();
                    obj.format_text(data: "Footer: Data from SAS V&sysver at &sysscp &sysscpl Sashelp.iris",just:"C");
                obj.cell_end();
            obj.row_end();
        obj.foot_end();

        obj.page();
    end;
run;

ods html close;
ods listing;

Note:

• A flavor added to get a sortable HTML report. Thanks to Charlie Huang and then Andrew Z to introduce a Javascript library JQury to SAS HTML report.
• The full report, see here.
• If column spannings needed, use the following codes as header (and the report here):

obj.head_start();
    obj.row_start();
        obj.format_cell(text: "Sepal" , overrides: "fontweight=bold",column_span: 2);
        obj.format_cell(text: "Petal" , overrides: "fontweight=bold",column_span: 2);
    obj.row_end();

    obj.row_start();
        obj.format_cell(text: "Length" );
        obj.format_cell(text: "Width" );
        obj.format_cell(text: "Length" );
        obj.format_cell(text: "Width" );
    obj.row_end();
obj.head_end();

• ODS Report Writing Interface will get rid of its preproduction hat since SAS 9.4, but you can use it somehow since SAS 9.1.3. For more, see the draft SAS 9.4 ODS documentation with ODS Report Writing Interface.
• To get started, see the developer’s paper.

I spent almost all of my blogging time last month to follow an online course in Coursera, Computing for Data Analysis (with R) by Dr. Roger  Peng of Johns Hopkins, Biostatistics Department. I already checked out bunch of Coursera courses just to take a look at what else MOOC look like, this R course was the first and only one I took seriously: I reviewed all slides (although didn’t watch the videos since I didn’t have enough time) and most important, finished all programming assignments(and got full score!).

Not a frequent R user, I used some R packages for learning purpose (mostly in data mining). This Computing for Data Analysis course focused mostly on data manipulation and that’s why I chose it to dig into R. R by design is not a tool for data management (where SAS excels), but it’s nice to have (no hurt since it’s free). Those days I put most of my efforts on R data structures(vector, matrix, array, list, data frame) and by grouping processing(*ply functions). Few resource I found extremely useful as a starter:

• http://stackoverflow.com/questions/tagged/r  when I google a specific R programming technique, mostly I will reach to stackoverflow website, a new generation Q&A hub to replace user forum, mailing list.
• An introduction to R by Longhow Lam, a free book. I like its Chapter 2, Data Objects, including R data types and data structures.
• Data Manipulation with R by Phil Spector, one of the Use R! books. I used its Chapter 8, Data Aggregation on R by group processing.
• RStudio, a must have R IDE, the best of best.
• Package plyr, tools for splitting, applying and combining data. Actually, it is awful to use R to manipulate data from a SAS programmer point of view. Use plyr to alleviate the pain.
• Package sqldf, like SAS Proc SQL against data frame. Reason, ibid. Use sqldf won’t enhance your R programming skill, but sometime when you feel down at R, you may want to launch piece of SQL scripts you really like.

This is for for SAS programmers who might be reluctant to check out a GUI tool like SAS Data Integration Studio (DIS for short, which is graphic tool to implement ETL processes: extract, transform, load). DIS translates all users dragged nodes, transformations and process into SAS codes which are traditionally written by SAS programmers. I find one of the benefits of using DIS is that I can package all the ETL work to other players(technical or nontechnical) then they can replay the job by dragging and clicking.

Actually besides translating all the visual nodes to SAS codes, DIS can also reversely incorporate your SAS codes to the graphic job shown as above. So SAS programmers can easily jump into DIS by importing their codes to get the graphic workflow (to get the first impression). Here is an one-minute demo:

Save the following codes in a file(from SAS SQL onlinedoc),

%let dir=C:\;

libname source BASE "&dir";
libname target BASE "&dir";

data source.one;
    input x y;
datalines;
1 2
2 3
;

data source.two;
    input x y;
datalines;
2 5
3 6
4 9
;

proc sql;
    create table target.three as
        select o.x,o.y
        from source.one as o, source.two as t
        where o.x=t.x
        ;
quit;

proc print data=target.three;
run;

Just create a test folder in DIS (this demo was created in a working repository, in a Windows DIS 4.21 machine ) then right click mouse to “Import”-“Import SAS Code” to import the file created above then run the job generated and all set (you will get all stuff showed above).

Note that the SAS libraries must be registered first in the SAS Metadata Server then you can use it. This demo omit this process only for demo purpose.

You can check out the SQL Join by double clicking the SQL node:

and your codes in Code panel (still yours!):

Furthermore, without getting know bunch of the tool first, you can jump into DIS quickly by creating customized Transformations which are basically SAS codes with macro variables as the perimeters in the visual options box. Pretty neat? Just check it out!

The MONOTONIC() function is not supported in PROC SQL. Using the MONOTONIC() function in PROC SQL can cause missing or non-sequential values to be returned.

Here question is, why stick to the function? Actually there is a long-existing Proc SQL option (at least since SAS 9.1.3 which was my first SAS version), NUMBER to return the row numbers:

proc sql number outobs=5;
    select Species
    from sashelp.iris
    ;
quit;

See a new column "Row" was created in response to this NUMBER option.

Now my turn (with Tagsets.ExcelXP, I can at least get rid of DDE)…

0. SAS Templates window

A visual way to browse all ODS templates is to use windows command “ODSTEMPLATES” to invoke a SAS template window.

1.  Check the current ExcelXP tagset version

filename temp temp;
ods tagsets.ExcelXP file=temp options(doc=’help’);
ods tagsets.ExcelXP close;

Scroll down the Log window to the end and you get something like:

NOTE: This is the Excel XP tagset (Compatible with SAS 9.1.3 and above, v1.122, 01/04/2011).

2. Get the source codes of current ExcelXP tagset and library it’s stored

proc template;
    source Tagsets.ExcelXP;
run;

The Log window shows the template codes:

15   proc template;
16       source Tagsets.ExcelXP;
define tagset Tagsets.ExcelXP;
   parent = Tagsets.ExcelBase;
end;
NOTE: Path ‘Tagsets.ExcelXP’ is in: SASHELP.TMPLMST.

Note Tagsets.ExcelXP is subject to Tagsets.ExcelBase.

The template store path uses a two level name in which SASHELP is a SAS library and TMPLMST, a name assigned to distinguish a set of templates. You can also store your customized templates in SASHELP.mytpl1, WORK.forTLF and such.

If want a the source codes in a separated file rather than in Log window, try

proc template;

source Tagsets.ExcelXP/file="a:\test\ExcelXP.tpl";
run;

The extension doesn’t matter and you can use anything you like (including .SAS).

3. List all ODS template store path

ods path show;

Log:

Current ODS PATH list is:

1. SASUSER.TEMPLAT(UPDATE)
2. SASHELP.TMPLMST(READ)

This order also indicates the ODS template searching priority, first SASUSER.TEMPLAT then SASHELP.TMPLMST. If two templates with same name exist in both library, the one in SASUSER.TEMPLAT will be used.

Templates in SASHELP.TMPLMST can’t be modified(read only).

4. Install latest version in SASUSER library

The last version of ExcelXP tagset is v1.127 by far according to SAS ODS MARKUP portal:

• http://support.sas.com/rnd/base/ods/odsmarkup/

filename excltags url ‘http://support.sas.com/rnd/base/ods/odsmarkup/excltags.tpl’;
%include excltags;

log:

NOTE: TAGSET ‘Tagsets.ExcelBase’ has been saved to: SASUSER.TEMPLAT
NOTE: Overwriting existing template/link: Tagsets.Config_debug
NOTE: TAGSET ‘Tagsets.Config_debug’ has been saved to: SASUSER.TEMPLAT
NOTE: TAGSET ‘Tagsets.ExcelXP’ has been saved to: SASUSER.TEMPLAT

Since there is already an ExcelXP tagset in SASHELP.TMPLMST and it can’t be overwritten, this new added ExcelXP tagset is thrown to SASUSER.TEMPLAT by adding a STORE option (check the source code) and will be used when statement “ODS tagsets.ExcelXP” invoked.

5. List all templates

proc template;
    list / store=sasuser.templat;
    list / store=sashelp.tmplmst;
run;

Get output:

If only type “Tagset” need (to save space!):

proc template;
list tagsets/store=sasuser.templat;
run;

6. Delete template

proc template;
    delete Tagsets.ExcelBase;
    delete Tagsets.ExcelXP;
run;

It’s always safe to perform delete statements since items in SASHELP.TMPLMST are read only.  In this case, the ExcelXP tagset in SASUSER.TEMPLAT will be deleted.

log:

8617   proc template;
8618       delete Tagsets.ExcelBase;
NOTE: ‘Tagsets.ExcelBase’ has been deleted from: SASUSER.TEMPLAT
8619       delete Tagsets.ExcelXP;
NOTE: ‘Tagsets.ExcelXP’ has been deleted from: SASUSER.TEMPLAT

check again following step #5 and get what’s in SASUSER.TEMPLAT (no ExcelXP tagset anymore):

7 Install in other specified library

Install ExcelXp tagset in a other library. It will be deleted if current SAS session is killed:

libname excltags ‘a:\test’;
ods path (prepend) excltags.templat(update);
filename excltags url ‘http://support.sas.com/rnd/base/ods/odsmarkup/excltags.tpl’;
%include excltags;

log:

NOTE: Overwriting existing template/link: Tagsets.ExcelBase
NOTE: TAGSET ‘Tagsets.ExcelBase’ has been saved to: EXCLTAGS.TEMPLAT
NOTE: Overwriting existing template/link: Tagsets.Config_debug
NOTE: TAGSET ‘Tagsets.Config_debug’ has been saved to: EXCLTAGS.TEMPLAT
NOTE: Overwriting existing template/link: Tagsets.ExcelXP
NOTE: TAGSET ‘Tagsets.ExcelXP’ has been saved to: EXCLTAGS.TEMPLAT

show path as step #3:

Current ODS PATH list is:

1. EXCLTAGS.TEMPLAT(UPDATE)
2. SASUSER.TEMPLAT(UPDATE)
3. SASHELP.TMPLMST(READ)

If apply step #6 to delete ExcelXP tagset, the items in EXCLTAGS.TEMPLAT will be deleted(as same as the searching priority).

8. Hello World

Adopted from one of Vince DelGobbo’s papers(2012):

title ‘The CLASS Data Set’;
footnote ‘(From the SASHELP Library)’;

*Set some "global" tagset options that affect all worksheets;

ods tagsets.ExcelXP path="a:\test\" file=’MyWorkbook.xml’ style=Printer
                    options(embedded_titles=’yes’
                            embedded_footnotes=’yes’
                            print_header=’&C&A&RPage &P of &N’
                            print_footer=’&RPrinted &D at &T’
                            autofilter=’2′);

%macro doit;

proc sort data=sashelp.class (keep=sex) out=class nodupkey;
    by sex;
run;

data _null_;
    set class end=eof;
    II=left(put(_n_,2.));
    call symputx(‘sex’||II,compress(sex));
    if eof then call symputx(‘total’,II);
run;

%do i=1 %to &total;
    ods tagsets.ExcelXP options(sheet_name="Students_&&sex&i");

    proc print data=sashelp.class noobs style(Header)=[just=center];
      where (sex eq "&&sex&i");
      var name age      / style(Column)=[background=#99ccff];
      var height weight / style(Column)=[background=#99ccff tagattr='format:#.0'];
    run;

%end;

%mend doit;

%doit

ods tagsets.ExcelXP close;

This .XML file can be opened by Office Excel:

9. Get Help

As a new comer, I found the following materials useful and accessible:

• SAS ODS MARKUP homepage
• SAS developer Vince DelGobbo’s homepage on ExcelXP
• Vince DelGobbo’s ExcelXP Tagset Paper Index

• Quick Reference for the TAGSETS.EXCELXP Tagset

• Try This Demo: The ExcelXP Tagset and Microsoft Excel

• Modifying ODS Statistical Graphics Templates in SAS 9.2 by Warren F. Kuhfeld

Top 10 countries/territories and cities

• Most of my readers come from US and I’m glad my Chinese fellows still fork me since I moved to US this year (today is my one year anniversary in US).
• To my surprising there are much more readers from South Korea than from Japan(#12) and Singapore(#22).
• Readers from a secret city (#7) are watching me!

Top 10 referral social networks

• I’m happy that people navigating through Facebook also click this technical blog!

Top 10 mobile devices

• I think myself contributed a lot via Google Nexus 7 (#7) visiting my own blog (one upon a time in 2012, I had two new released Nexus 7 tablets at hand until one dropped down and another sold out…).

Top 10 most viewed posts

• Sublime Text 2 for SAS Programmers: A Quick Note
• Tony Barr and the Early History of SAS
• Face Off: Review OpenCDISC XML files
• My SAS Books: Shopping List 2012
• Statistical Notes (4): Dragon’s Teeth and Fleas: Hypothesis Testing in Plain English
• Statistical Notes (1): Geometric Mean and Geometric Mean Ratio
• Column Mode in SAS 9.3 Enhanced Editor
• Weekend Clip: Data Scientist
• Statistical Notes (2): Equivalence Testing and TOST (Two One-Sided Test)
• Is There Any Better Way? Publishing Process For CDISC Standards Documentation

Thank you my fellow readers and happy new year (if you like to have another break, follow me to celebrate Chinese new year at Feb 10, 2013 )!

XML File Reading: SAS XML Mapper

SAS XML Mapper itself is not an elegant tool from software design perspective (an example: I keep multiple versions because the latest version seems not carry out all the functionalities from old ones), but it is best XML file processing tool for SAS programmers like me. The sweet part of SAS XML Mapper is that you can use it to get SAS datasets directly with a automatically generated XML mapping file. I use it to import XML files from

CDISC ODM based files like define.xml

metadata querying results (XML) returned by SAS Metadata Server

and it works pretty well and I just can’t live without it!

Graphics Facility: ODS Graphics

ODS Graphics is the raising star among SAS products family since its advent and I also fall into love with her. This system contains

five procedures with “SG” in names,

a template language(GTL), and

two GUI tools, ODS Graphics Editor, and ODS Graphics Designer

in Base SAS and ODS Statistical Graphics in SAS/STAT (and in SAS/QC which I didn’t check out). ODS Graphics just makes SAS graph  much beautiful and graphics task much fun (and elegant!).

Btw, it may not be a fair game but still nice to check out a SAS ODS graph and some random pick up R graphs:

This is from Rick Wicklin with 3 lines of codes including one “RUN” statement:

proc sgscatter data=sashelp.iris; 
matrix SepalLength--PetalLength /group=Species diagonal=(histogram kernel);
run;

and this is the R homepage graph(and the R codes):

and this is R Graph Gallery homepage:

I must say SAS ODS Graphics rocks!

Statistical Procedure: PROC TTEST and PROC FREQ

In this category I list two because they are equally extremely relevant and important for me as statistical SAS programmer.

PROC TTEST makes equivalence test (which is extremely popular in clinical research) much more accessible by adding a TOST(two one-side test) option. Years ago SAS programmers might use PROC MIXED or other methods for this kind of statistical test(I also took a note on this topic, see here).

The new on PROC FREQ I checked out is to support much richer methods on calculating confidence intervals for binomial proportion (my note here) and confidence intervals for difference between independent binomial proportions (my note here) which are also extremely important in clinical research and I programmed a lot.

Report Writing: ODS Report Writing Interface

My first SAS version was 9.1.3 where PROC REPORT is the primary SAS reporting writing tool (with ODS) while PROC TABULATE not in fashion anymore (and you may merely hear the arguments among these two procedures since then). I used PROC REPORT for all my production work for reporting since recently I tried the ODS Report Writing Interface in a project for non-rectangular tables. It’s great and everyone was happy!

Basically it is an ODS enhanced DATA _NULL_ reporting writing method (DATA _NULL_ with FILE PRINT statements: it’s an even older way for me);a new ODS output object declared within DATA _NULL_:

dcl odsout obj();

I like this kind of reporting method: you can control your report line by line and cell by cell (although with more lines of structured codes!).

Metadata Querying Tool: PROC METADATA

In this category, the other two strong candidates are JAVA interface and SAS Data Step Functions. I like PROC Metadata against JAVA because it holds the same full functionality (while SAS Data Step Functions, no yet) while I can still work in SAS to produce reports (just add a new line to start to use ODS Report Writing Interface:)). Furthermore, I feel much comfortable working with SAS!

PROC Metadata uses XML as inputs and outputs: it may be not admirable compared to Data Step Functions. Since I use SAS XML Mapper, it’s not a problem anymore!

Documentation

I like the totally new SAS help and documentation system both online and offline since SAS 9.3.

First in Base SAS, lots of files were separated from “SAS Language Dictionary”,

and in all procedures guide, the tab view looks great:

Today Jozef (Jos) Aerts from XML4Pharma posted his frustration of copying and pasting metadata from only PDF-formed SDTM-IG 3.1.4.  I hate to complain the volunteer work by the CDISC team but it is worth a discussion, is there any better way to publish CDISC Standards?

1. What is A Data Scientist Anyway?

2. You Just Can’t Be Replaced by Yourselves!

3. We are all Data Scientists!

Last week I got another SAS certification, SAS Certified Data Integration Developer for SAS 9 and I must say it can’t be better as a SAS holiday gift for myself!

I took all my SAS certificates when employed (when in Sanofi Pasteur, 2010, I got two SAS programming certificates, Base and Advanced). You may say it’s kind of certificate for sake of certificate and not for applying a job like college students. You are right and few reasons I like to earn such certificates at work:

• It doesn’t hurt! Little better than nothing and holding a certificate will never make a thing worse off.
• It’s highly relevant. Only few additional efforts needed to take such SAS certificate examinations when worked as SAS programmer and solution consultant.
• It’s necessary(somehow). When worked as a SAS programmer at pharma,  I played data manipulations techniques heavily, but never used a indexing method because it’s not needed due to limited cases of clinical data: recruiting a thousand subjects is even not a easy task! Then I took SAS Advance test and I filled the hole of SAS index.
• It’s fun; actually I mean the certificate fee was paid by my employers (thanks!)…I got this from Prof. N. Gregory Mankiw:

Most people who pursue an academic career do so because they are fascinated by their subject. It is for this reason that professors report among the highest rates of job satisfaction of all professions. Professors have found what they like to do, and they have found someone to pay them to do it.

(then change “people/professor” to “SAS Programmer”)

I’m always enamored of playing with SAS in any kind of forms: writing SAS codes, configuring SAS products, reading SAS books, taking SAS certificates, and very important, I’m paid to do these!

Last year I threw away all my SAS books (to friends and colleagues in Beijing) before moving to US. You might agree that it’s not economically bound to transport such heavy books intercontinentally! Now I just start to build my SAS library one by one.

I will most probably not buy more SAS books as Christmas gifts for myself, so it’s time to take a 2012 snapshot of shopping:

I Programming Books by Art Carpenter

1 Carpenter’s Complete Guide to the SAS Macro Language, 2nd Edition, 2004

Best of best on SAS (dynamic) programming. I also had a hardcopy in Beijing. ★★★★★

2 Carpenter’s Guide to Innovative SAS Techniques, 2012

It’s new by Carpenter and nice to check out.

3 Carpenter’s Complete Guide to the SAS REPORT Procedure, 2007

Use Proc Report! It is the workhorse procedure for SAS report writing and Proc Tabulate is not in fashion anymore!

Btw, The historically called SAS DATA _NULL_ report writing now has a brand new name, ODS Report Writing Interface, due to the incorporated ODS facility(which is cool). I think it deserves a new title.

PS: I got it free from the latest RTSUG (Research Triangle SAS Users’ Group) meeting.

II Programming Miscellaneous

4 Web Development with SAS by Example, Third Edition by Frederick Pratter, 2011

Want to play HTML/htmSQL/AJAX/Javascript… within SAS. no comments by far.

5  SAS Hash Object Programming Made Easy by Michele Burlew, 2012

Totally new title on this topic. Nice to approach to the new area.

III ODS Graphics

6 Statistical Graphics Procedures by Example: Effective Graphs Using SAS by Sanjay Matange and Dan Heath, 2011

SAS ODS Graphics is the rising Star! I also read other title on ODS statistical graphics in Beijing,

Statistical Graphics in SAS: An Introduction to the Graph Template Language and the Statistical Graphics Procedures by Warren Kuhfeld

You may read plenty of ODS Graphics  papers by these authors/developers from SAS Institute.

IV Statistical Books

7 Common Statistical Methods for Clinical Research with SAS Examples, Third Edition by Glenn Walker, Jack Shostak, 2010

Best for SAS clinical programmers. Relevant, comprehensive and easy to access. I had a second version in Beijing. ★★★★★

8 Multiple Comparisons and Multiple Tests Using SAS, Second Edition by Peter H. Westfall, Randall D. Tobias, Russell D. Wolfinger, 2011

Multiple comparison is an advanced topic and is beyond my limited statistical knowledge. My short term goal is just to follow the codes then interprete results.

V CDISC

9 Implementing CDISC Using SAS: An End-to-End Guide by Chris Holland and Jack Shostak, 2012

Still on the way. The first title on CDISC + SAS and worth of waiting. I read Jack Shostak’s another book when applying a SAS programmer job at Sanofi Pasteur Beijing R&D in 2009:

SAS Programming in the Pharmaceutical Industry

VI My Wish List 2013

10 Building Web Applications with SAS/IntrNet: A Guide to the Application Dispatcher by Don Henderson

SAS web programming again. I might read this title before #4. Don Henderson was the former SAS developer and SAS/IntrNet was just his baby!

11 Categorical Data Analysis Using SAS, Third Edition by Maura Stokes, Charles Davis, and Gary Koch

I had a second version hardcopy in Beijing and it’s still cool to play with 2*2 table in this so called BIG DATA world! ★★★★★

12 Cody’s Collection of Popular SAS Programming Tasks and How to Tackle Them by Ron Cody

I read and like Cody’s book on SAS functions and it’s always nice to check out his new title.

13 The Global English Style Guide: Writing Clear, Translatable Documentation for a Global Market by John R. Kohl

I read the free chapter and it would be good book to polish my English as a technical blog writer.

14 The Little SAS Book: A Primer, Fifth Edition by Lora Delwiche and Susan Slaughter

I forget the version I read back to the time I first approached to SAS and it’s nice to hold a latest version in shelf and I’d rather learn the effective writing! Best for beginners. ★★★★★

One blog post:

Statisticians aren’t the problem for data science. The real problem is too many posers

One job advertisement:

SAS Data Scientist ?(!)

A joke:

The biggest joke about data scientist is that the Google query “data scientist joke” returns nothing interesting.

2 cents add to this note:

First, we do not need to write codes to get to know the machine version for a catalog(formats.sas7bcat or sasmacr.sas7bcat). Just open the catalog using a text editor, Notepad++, then you get

a SAS format catalog created by SAS 9.1 in a Windows XP Pro machine:

and a SAS macro catalog by SAS 9.3 in a 64 bits Windows 7 Pro machine:

Second, if do want programming (for curiosity purpose; I like this idea), we can also improve the sample codes since the position and length information were hard coded in Line 21 (the question is: how can we know that?):

test=substr(theline,210,20);

To get rid of hard coding, I use Perl Regular Expression. Just copy the archived codes for my SESUG 2012 talk with tiny modification (I always use the same programming blocks if possible):

https://github.com/Jiangtang/SESUG2012/blob/master/read_SAP.sas

The following codes can be also used to run against SAS macro catalog:

filename fmt "a:formats.sas7bcat";

data fmt;
   infile fmt lrecl=1000 truncover;
   input line $1000.;
run;

data _null_;
    set fmt;
         if _n_=1 then do;
            retain queVer;

            ver="/(d+).[dw]+_[dw]+/";
            queVer  = prxparse(ver);

            if missing(queVer) then do;
               putlog "ERROR: Invalid regexp" ver;
               stop;
            end;
      end;

      queVerN  = prxmatch(queVer ,line);

      if queVerN > 0  then do ;
              call PRXsubstr(queVer,line,position,length);
            version = compress(substr(line, position, length));
            output;
      end;   

      put position= length=;
      put version=;
run;

Test it against five Windows machines:

position=221 length=16
version=9.0301M1X64_7PRO

position=217 length=15
version=9.0202M2NET_SRV

position=221 length=17
version=9.0202M3X64_SRV08

position=217 length=14
version=9.0100A0XP_PRO

position=217 length=15
version=9.0101M3NET_SRV

P.S.: The pattern (d+).[dw]+_[dw]+  can be read as

number(s)                       +
.                                      +
some digits and words     +
_                                     +
some digits and words

Since the underscore “_” is also included in meta-character w, it seems OK to simplify the pattern above to

(d+).[dw]+

But in a Window Server 2003 machine with SAS 9.2, it also returns the following unrelated information:

position=998 length=3
version=2.S

The website encountered an error while retrieving http://www.jiangtanghu.com/blog/. It may be down for maintenance or configured incorrectly.

I checked web log and it says:

[24-Nov-2012 02:58:49] PHP Parse error:  syntax error, unexpected T_STRING, expecting ‘,’ or ‘;’ in ../public_html/jiangtanghu/blog/wp-content/plugins/wp-conditional-captcha/wp-conditional-captcha.php on line 401

So the problem comes from a WordPress plugin “Conditional CAPTCHA for WordPress” . The line 401 in wp-conditional-captcha.php is

<li><input type="radio" name="pass_action" id="pass_action_hold" value="hold" <?php checked( $opts['pass_action'], ‘hold’);?> /> <label for="pass_action_hold"><?php _e(‘Queue the comment for moderation’, ‘wp-conditional-captcha’);?></label></li>

I use WordPress v3.4.2 and “Conditional CAPTCHA for WordPress” v3.2.6. I didn’t notice this error before and it may be a compatibility issue (I know 0 about PHP). So my solution: delete the plugin.

Furthermore, I plan to switch to Github+jekyll and get rid of WordPress (sorry, you are wonderful; I’m just an emotional customer in this crazy Black Friday) .

• BASE SAS
• Enterprise Guide: create a new “File-New-Program”
• SAS Data Integration Studio: create a new “Tool-Code Editor”

%let species="Setosa" "Versicolor";

data a;
    set sashelp.iris;
    where species in (&species);
run;

Then create a Transformation in SAS Data Integration Studio (DIS for short; I use version 4.3 in a Win 7 machine) using the codes above as source code (remember deleting first line) and create a simple Prompt to assign the macro variable &species with default values as "Setosa" "Versicolor":

Drag this Transformation in a single node job and run, then you will get such errors:

127        data a;
128            set sashelp.iris;
129            where species in (&species);
NOTE: Line generated by the macro variable "SPECIES".
129        "Setosa", "Versicolor"
           _
           22
            _
            76
ERROR: Syntax error while parsing WHERE clause.
ERROR 22-322: Syntax error, expecting one of the following: a quoted string, a numeric constant, a datetime constant,
              a missing value, -. 

ERROR 76-322: Syntax error, statement will be ignored.

130        run;

So, what happened inside DIS since such codes go well in BASE SAS, SAS EG and the Code Editor in DIS itself? The default value "Setosa" "Versicolor" was assigned to macro variable &species in the DIS Prompt (see picture above) and you would expect the following effect like what I wrote in open codes in BASE SAS:

%let species="Setosa" "Versicolor";

Actually NO. In DIS, this action was translated into such clause:

%let species = %nrquote(%"Setosa%", %"Versicolor%");

Here is the point: since there quotation symbols were masked by %nrquote function, they are no longer valid syntax characters (then we got errors!).

The solution: use a %unquote function on the macro variable &species to reverse the masking effort by %nrquote(note in our open codes example in BASE SAS, both functions were not used):

data a;
    set sashelp.iris;
    where species in (%unquote(&species));
run;

Notes on Macro Variable Generated by DIS

%let species = %nrquote(%"Setosa%", %"Versicolor%");

1)%nrquote function used while it is dated for a long time… %nrquote and %quote were replaced by %BQUOTE and %NRBQUOTE Functions.

2)All quotation symbols(“) were preceded by a percent sign (%). That’s %nrquote and %quote needed and why they are outdated any more.

%let var=1,2,3;
%put &=var;

or

%macro test(var);
    %put &=var;
%mend;
%test(%str(1,2,3))

and you will get in Log window

VAR=1,2,3

You can read from SAS 9.3 online doc:

If you place an equal sign between the ampersand and the macro variable name of a direct macro variable reference, the macro variable’s name displays in the log along with the macro variable’s value.

In SAS 9.2 and prior, you should type the macro variable twice:

%let var=1,2,3;
%put var=&var;

PS: it is called a “Tip” in SAS 9.3 online doc, but I’d rather call it a trick: not critical but nice to have (save your typing!). You may be also interested in Rick Wicklin’s differentiation on Tip and Technique.

see, it’s hip to be square
‘cuz SAS has a new PROC that’s GROOVY
-Chris Hemedinger, Poetry on our own terms   

These days I played Proc Groovy (new in SAS 9.3) for a while because Groovy natively supports JSON (JavaScript Object Notation) data format. I downloaded much JSON data in the past few weeks(Github archive for example).

——–I’m a line separator ———

DO wish SAS can also support JSON in the following release, in a way similar to the existing XML Libname engine, ODS XML markup and SAS XML Mapper!

UPDATE 20121030: I submitted a ticket on SASware Ballot on this idea and get a feedback that SAS will support JSON since 9.4, both reading and writing. That’s cool.

——–I’m a line separator end———

Groovy is a dynamic language running in a JVM (Java Virtual Machine) and looks like a lightweight version of JAVA. Don’t know why Groovy was chosen but it is nice to have (at least to parse JSON data) and it’s in Base SAS!

Set UP

Back to Proc Groovy. To make it work, you should first set a JDK(Java Development Kit; or at least a JVM, also called JRE, Java Runtime Environment) and install Groovy. I test in a Windows 7 machine with SAS 9.3:

• Install a JDK (or JVM). Get a proper version for your machine. I have a JDK7 installed in C:Program FilesJavajdk1.7.0_09. By the way, if your SAS 9.3 works well in a64 bit of Windows 7, there must be one JRE in C:Program Files (x86)Javajre1.6.0_24.
• Set up the windows environment variable for JRE. First, create a system variable JAVA_HOME with value of C:Program FilesJavajdk1.7.0_09, then put %JAVA_HOME%bin to the existing system variable Path. If you use a JVM, replace the value of JAVA_HOME as C:Program FilesJavajre7 or other proper directory.
• Install Groovy using the latest version (current v2.05). Accept all the default settings and it will set the environment variable for Groovy. You will have the Groovy installed in C:Program Files (x86)GroovyGroovy-2.0.5.
• Locate the sasv9.cfg file in C:Program FilesSASHomeSASFoundation9.3nlsen, find option –JREOPTIONS and add a line inside

-Dtkj.app.class.path=C:Program Files (x86)GroovyGroovy-2.0.5embeddablegroovy-all-2.0.5.jar

That’s it.

Hello World in Proc Groovy

proc groovy;
    submit;
        def name=’World’;
        println "Hello $name!"
    endsubmit;
quit;

Read JSON data

Use a simple example from JSON in wikipedia (save it to a file, test.json):

{
    "firstName": "John",
    "lastName": "Smith",
    "age": 25,
    "address": {
        "streetAddress": "21 2nd Street",
        "city": "New York",
        "state": "NY",
        "postalCode": "10021"
    },
    "phoneNumber": [
        {
            "type": "home",
            "number": "212 555-1234"
        },
        {
            "type": "fax",
            "number": "646 555-4567"
        }
    ]
}

The following codes simple demonstrate how to:

• read JSON file
• print JSON file
• extract JSON values
• output values into SAS macro variables

options nosource;

proc groovy;

    submit;

        import groovy.json.*

        def input=new File(‘a:testtest.json’).text

        def output = new JsonSlurper().parseText(input)

        println output   
        println ""

        output.each {println it}  

        println ""

        println output.address.streetAddress

        println "Street Address: $output.address.streetAddress"

        println output.address["streetAddress"]       

        exports = [fName1:output['firstName']]   
        exports.put(‘fName2′, output['firstName']) 
        
        endsubmit;

quit;

%put fName1: &fName1;

%put fName2: &fName2;

The output in LOG window:

Happy Groovying!

References

Parse JSON in SAS with Groovy – part 2 by Simon Dawson

http://groovy.codehaus.org/Documentation

Overview: GROOVY Procedure in support.sas.com

SAS historians (not all SAS users!) know Anthony James Barr (aka Tony Barr or Jim Barr) was the designer and implementor of SAS language (1966-1979) and the first Chairman and President of SAS Institute (1976-1979). In this picture was Tony Barr in the first SUGI (January 26-28, 1976, Kissimmee, FL), taken by the first SUGI chair Julian Horwich. It’s said in 1970s, everyone had long hair!

I was particularly interested in the early history of SAS when traced back the SAS file system to FFS (Formatted File System) in Tony Barr’s statement, SAS Related History. It’s not relational in the sense of RDBMS, but definitely a database management system(DBMS). Then you may understand why some people still call SAS library/datasets “database”. They are right in this non-relational point of view. Actually there are lots of parallel efforts to develop non-relational databases even after 1970 (when Codd introduced the concept of relational model). Google’s BigTable for example is not  relational.

1960s-1970s was the emerging period for statistical computing technology (also golden age of technology). SAS as software was conceived in 1966, implemented in 1968 and got first massive distributed release in 1972 and then as a company incorporated in 1976 (you may know Apple also founded in this year; Microsoft 1975; Oracle, 1977; SPSS also in the same period). Around this time, PL/1 first appeared in 1964 (SAS was initially written by PL/1), C developed during 1969-1973 (SAS rewritten by C in 1980s), SQL 1978, S language 1976.

Putting Tony Barr’s work on SAS in such exciting historical background (they were in their 20-30s!), then you will also be interested to dig into such materials on early history of SAS:

Barr, Anthony J. (2006), Professional History (Tony Barr’s profile; also in Wikipedia)

Barr, Anthony J. (2006), SAS History (the early history of SAS, the software and company)

Barr, Anthony J. (2010), SAS Beginnings (talk in NCSU; also in Youtube)

The early issues of SAS Communications, achieved by CodeCrafters

SASCommunity.org, early SUGI papers collection (also archived by Lex Jansen)

http://www.sas.com/company/about/history.html, the official history of SAS

SAS (software) in Wikipedia

SAS (company) in Wikipedia

Jim Goodnight in Wikipedia

John Sall in Wikipedia

I’m not the SAS manual guy (I used SAS since 9.1.3). The early issues of SAS manuals are definitely good resources for SAS historians (I’m waiting for a yard sale . . .).  For paper books, I only found

Hidden Value by Charles A. III O’Reilly, had one chapter for SAS where we can learn bits of its early history.

Object Lessons by Tom Love: few pages for SAS that we can know the initial price of SAS software was $35.

Will keep updating and thank you for any inputs.

    Once upon a midnight dreary,

While I pondered, weak and weary,

   Over many a quaint and curious

        Volume of SAS-L galore,

    While I coded, nearly snapping,

    Suddenly there came a tapping

    As of someone gently wrapping

      Code around my mental core.

“’Tis’ Do-Whitlock Loop”, I muttered,

  “Wrapped around my brain core” -

     This it was, and so much more! 

  –Paul Dorfman, A Bit of DoW-History

It’s great to have Paul Dorfman’s demo of DoW-loop (with a poem! Thanks to Paul to send me a copy.) in this SESUG 2012 conference. First posted in SAS-L by Whitlock, promoted by Dorfman,  DoW-loop is widely known as “Whitlock Do Loop” or ” Dorfman-Whitlock Do Loop”. In Dorfman’s presentation, the following three forms of DoW-loop were constructed:

• Henderson-Whitlock Original Form of DoW-loop
• Dorfman’s Short Form of DoW-loop
• Double DoW-loop: the DeVenezia-Schreier Form

Using a test data set

data a;
    input id $ var;
datalines;
A 1
A 2
B 3
B 4
B 5
;

the Henderson-Whitlock Original Form of DoW-loop looks like:

data b;
    count= 0;
    sum = 0 ;

    do until ( last.id ) ;
        set a ;
        by id ;

        count+1;
        sum+var;
    end ;

    mean = sum / count ;
run ;

Ian Whitlock first used such kind of do loop in a SAS-L post and after its rising, Paul Dorfman, the main promoter of DoW-loop, found that Don Henderson also made use such DO UNTIL() structure in a NESUG 1988 paper, The SAS Supervisor. That’s why he named it as Henderson-Whitlock form of DoW-loop. I then read from a DoW-loop page in sascommunity.org that Don Henderson taught such concept in class where Ian Whitlock was a student. This is a nice story.

Paul Dorfman himself also contributed a short form:

data c ;
    do n = 1 by 1 until ( last.id ) ;
        set a ;
        by id ;

        count = sum (count, 1) ;
        sum = sum (sum, var) ;
    end ;

    mean = sum / count ;
run ;

or even shorter:

data c ;
    do _n_ = 1 by 1 until ( last.id ) ;
        set a ;
        by id ;

        sum = sum (sum, var) ;
    end ;

    mean = sum / _n_ ;
run ;

These kinds of form of DoW-loop utilizes the SUM function, automatic variable _N_ and an increment trick in DO UNTIL() structure then the initializations before the loop are not needed any more. Besides such a short form, Paul Dorfman’s work on DoW-loop includes the invention of the dynamic file splitting method combining the DoW-loop and the hash object (also showed up in the meeting).

The double DoW-loop is under the name Howard Schreier and Richard DeVenezia (DeVenezia-Schreier Form; I should do more literature research on it!):

data d ;
    do n = 1 by 1 until ( last.id ) ;
        set a ;
        by id ;

        count = sum (count, 1) ;
        sum = sum (sum, var) ;
    end ;

    mean = sum / count ;

    do n = 1 by 1 until ( last.id ) ;
        set a ;
        by id ;
        output;
    end ;
run ;

/*********************/

/*****update 1*********/

Thanks to Quentin’s message, Don Henderson’s The SAS Supervisor can be even traced back to 1983.

/*****update 2*********/

I used a Star Wars style of opening crawl to render Paul Dorfman’s DoW-loop verses simply because the finding of Don Henderson looks slightly like a prequel for me (although I have no intention to make up a Star Wars parody). Actually, as Paul stated, the honor belongs to an American author, Edgar Allan Poe and one of his poems, The Raven:

Once upon a midnight dreary, while I pondered, weak and weary,
Over many a quaint and curious volume of forgotten lore —
While I nodded, nearly napping, suddenly there came a tapping,
As of some one gently rapping, rapping at my chamber door.
"’Tis some visiter," I muttered, "tapping at my chamber door —
            Only this and nothing more."

/*****update 3*********/

A double DoW demo in recent SAS-L(Sat, 27 Oct 2012).

You may find such SAS desktop shortcuts useful:

• click icon A then you get a SAS session titled “Project A” (you may notice your SAS session with title “SAS” by default). In this session, the current working folder is just where you put Project A (the default may be like C:usersyourID). Also, some autoexec codes run including libname, fileref, formats search, macro loading and other customized options/settings.
• Keep session A alive, then click icon B: you get your SAS session specified for Project B. And very important, there are no such annoying logs (you will most likely get such warnings when hitting your own SAS desktop icon twice):

NOTE: Unable to open SASUSER.REGSTRY. WORK.REGSTRY will be opened instead.
NOTE: All registry changes will be lost at the end of the session.
WARNING: Unable to copy SASUSER registry to WORK registry. Because of this,
WARNING: you will not see registry customizations during this session.
NOTE: Unable to open SASUSER.PROFILE. WORK.PROFILE will be opened instead.
NOTE: All profile changes will be lost at the end of the session.

Here is the how-to demo (tested in a Window 7 machine with SAS 9.3; the so called project folder is my GitHub programming folder,  C:\Users\jhu\Documents\GitHubProgramming-SAS):

• From C:\Users\jhu\Documents\My SAS Files\9.3, copy user profile file profile.sas7bcat to  the project folder. This is used to prevent the annoying message above.
• Throw a autoexec.sas file to the project folder with anything you like; or if you do not like, skip it
• Put in the project folder a configuration file sasv9.cfg with the following settings (delete –autoexec option if you want):

-config “C:\Program Files\SASHome\SASFoundation\9.3\nls\ensasv9.cfg”
-sasinitialfolder “C:\Users\jhu\Documents\GitHubProgramming-SAS”
-awstitle “My SAS Programming@GitHub”
-sasuser “C:\Users\jhu\Documents\GitHubProgramming-SAS”
-autoexec “C:\Users\jhu\Documents\GitHubProgramming-SASautoexec.sas”

• Send C:\Program Files\SASHome\SASFoundation\9.3\sas.exe to a desktop shortcut (I renamed it as “GitHub”).
• In desktop, find this icon and get its properties: in Target, replace all the text with

“C:\Program Files\SASHome\SASFoundation\9.3\sas.exe” -config “C:\Users\jhu\Documents\GitHubProgramming-SAS\sasv9.cfg”

Save the setting and exit then you click this icon and get:

Note:

Lex Jansen made a good trick to start SAS in any specific folder by modifying the registry table in Windows; then you can right click mouse to get in a SAS session:

It’s simple: just save the following text to a file with extension as REG and run it (the configuration also in Windows 7 with SAS 9.3):

Windows Registry Editor Version 5.00

[HKEY_CLASSES_ROOTDirectoryshellSAS_here_v93]
@=”&SAS here V9.3″
[HKEY_CLASSES_ROOTDirectoryshellSAS_here_v93command]
@=”"C:Program FilesSASHomeSASFoundation9.3sas.exe” -sasinitialfolder “%V” -nosplash -CONFIG “C:Program FilesSASHomeSASFoundation9.3nlsensasv9.cfg”"

• IML itself, seems have no intention to replace SAS/Stat, SAS/ETS, SAS/QC and other SAS statistical facilities in matrix way like R or Matlab. Instead, by enhancing the abilities to call SAS data steps(including macros), SAS Procedures and R inside,  IML seems to be glad as glue to leverage multiple programming elements.
• BASE SAS is a collection of lots of programming elements: data step, macro, SQL, array, hash, Java objects, ODS, XML and bunches of procedures. IML is clearly a programming tool rather than a specific application(SAS/Stat and such) and if it is added to BASE SAS, then the SAS programmers  will have one more data structure (matrix) to play with besides SAS table (R copies this idea as data frame), array and hash table (dictionary).
• Actually it is an emergent need (for a matrix). Processing different rows in variable(s) is not an easy task in SAS data steps. Mostly SAS programmers utilize arrays (plus some tricky functions like LAG and DIF) and Dow-loop after its advent. There are really advanced programming techniques, but in matrix with index, such manipulation is pretty easy and straightforward (and saving your loops).
• Although IML can communicate with BASE SAS, but for portable purpose (IML is not expected in every SAS session), most SAS programmers prefer to submit pure BASE codes for data processing. Furthermore, we can even understand why SAS programmers in general invest less in IML. It’s not good: just think about a BASE SAS programmer without a handy matrix manipulation tool (and think about R)!

Personally I played IML for a while and still keep eyes on its updates. But since I had some SAS machines without IML, I realize that IML can’t be used as a production language in my clinical SAS programming life: just can’t afford the latency  to ask IT administrators (including in clients side) to manage it!

SAS PRX is an old school topic somehow. After dumping the paper, I think I should restructure my talk in a more accessible way. Regular expression is the de facto standard tool for text processing (you can get help from the whole world besides SAS!), and SAS 9.3 even incorporates it to Proc Format. My talk will go like this:

• No picnic! You are already using Regex: remember *.sas?
• Where to start? what you should do is only to fill the programming block
• Supporting tools: visually play with Regex
• Resources: how to get the Regex for immediate use without digging into the “hairy” metacharacters for days and weeks
• A demo
• A Joke

See you then!

I also used a free Powershell GUI product, PowerGUI with similar interface like ISE. The latest PowerGUI (3.2.0.2237) supports Powershell 2.0 by default (while you can force it to play with PS 3.0). I’m not a script guy but bits of shell do make life easy.

Getting Started:

• a quick look of Powershell, Windows CMD and Unix shell
• Powershell 3.0 offical portal
• a wonderful book(FREE), Master Powershell with Dr. Tobias Weltner
• Powershell Tips
• For SAS users, blogger Chris Hemedinger introduced some useful examples.

The concept of current working directory in SAS is not as important as it sounds. Most SAS users point to file reference (fileref) and library reference (libname, a “namespace”-like mechanism to access files ), which are much convenient in large project with multiple folders.

Sometimes I push files to the current folder in relatively small task because I don’t want to

• jump to C:UsersjhuappdataLocalTempSAS Temporary Files_TD2820_like folder to get the temporary datasets (in WORK library)
• hard code a folder to receive ODS outputs.

The demo codes(a SAS system autocall macro %xlog used):

/*display current directory in LOG window*/
%xlog(cd);

/*change current working directory */
x "cd a:test";

/*display current directory in LOG window*/
%xlog(cd);

/*output a dataset in current directory */
data "iris";
    set sashelp.iris;
run;

/*output a pdf file in current directory */
ods pdf file="iris.pdf";
proc print data="iris";
run;
ods pdf close;

/*show files in current directory */
%xlog(dir);

And the LOG window may like:

Path
—-
C:Usersjhu

Path
—-
A:test

    Directory: A:test
Mode        LastWriteTime   Length Name
—-        ————-   —— —-
-a— 10/8/2012  12:49 PM   113665 iris.pdf
-a— 10/8/2012  12:49 PM    13312 iris.sas7bdat

NOTES:

• You can get autocall macro %xlog in (if SAS 9.3 in Windows 7)

C:Program FilesSASHomeSASFoundation9.3coresasmacro

An equivalent macro %xlst generates the shell command outputs to LIST window.

• If prefer Windows Powershell, a similar macro can be used (put it to autocall if needed):

%macro pslog(cmd);
  option nonotes;
  filename pslogcmd pipe "powershell &cmd";
  data _null_;
     file log;
     infile pslogcmd;
     input;
     put _INFILE_;
  run;
  option notes;
%mend pslog;

Test the following commands:

%pslog(gl);
%pslog(get-location);
%pslog(get-host)

1. LogFilter: a SAS log checker by rtsl

http://www.ratcliffe.co.uk/rest_logfilt.htm

2. A SAS format viewer by Frank Poppe

http://frank-poppe.xs4all.nl/SAS/edit_format.html

Yep, I am a SAS programmer: during the day time, I use SAS for my work; at the evening, I use R for entertainment.—

Charlie Huang

Getting start to play with R (it is really a wonderful learning tool):

12. SAS IML Studio and R Integration (not tested)

11. SAS IML and R Integration (not tested)

10. SAS Base and R Integration: PROC R

also a blog post.

9. Data transfer among R and SAS(xpt)

SAS: SAS Macros that Convert a Directory of Transport Files

R: use read.xport function in Foreign package and write.xport in SASXport package to read/write xpt file

8. Data transfer among R and SAS(CSV)

SAS: use %DS2CSV Macro to export SAS dataset to CSV file

R: use read.csv /write.csv function to read/write CSV file

7. The R Graph Gallery

6. R blogs

5. R search

4. R Task View: packages recommended

3. All R packages

2. Install RStudio, a much better R IDE

1. Get R software

A guy notices a bunch of targets scattered over a barn wall, and in the center of each, in the "bulls-eye," is a bullet hole. "Wow," he says to the farmer, "that’s pretty good shooting. How’d you do it?" "Oh," says the farmer, "it was easy. I painted the targets after I shot the holes." – An Old Joke

Note on how to get the difference between independent binomial proportions using SAS PROC FREQ (tested in SAS 9.3M1).  Also, benchmarks took from Prof. Newcombe ’s paper:

Interval estimation for the difference between independent proportions: comparison of eleven methods.

Key findings:

1. SAS Proc Freq offers 8 ( including 4 of total 11 in Prof. Newcombe ’s paper) methods to compute confidence intervals for difference between independent binomial proportions:

• Wald #1
• Wald (Corrected) #2
• Exact
• Exact (FM Score)
• Newcombe Score #10
• Newcombe Score (Corrected) #11
• Farrington-Manning
• Hauck-Anderson

Note that #1 method is the most popular one in textbook, #10 might  be the most wildly used method in industry.

2. There is a big discrepancy in the outputs in the  so called “exact” method among  SAS Proc Freq and Prof. Newcombe ’s paper. Seems they used different methods (in the same “exact” families) under same name. Needs further investigation.

The SAS codes:

/*
2*2 table

        response0 response1
group1      56      14
group2      48      32
*/

data ci;
    input grp res count;
datalines;
1 0 56
1 1 14
2 0 48
2 1 32
;

/*
methods:
Exact
Farrington-Manning
Hauck-Anderson
Newcombe Score
Wald
*/
ods select  PdiffCLs;
ods output PdiffCLs=ci1;
proc freq data=ci order=data;
    tables grp*res /riskdiff (CL=(EXACT FM HA WILSON WALD));
    weight count;
    exact riskdiff;
run;

/*
methods:
Exact (FM Score)
*/
ods select  PdiffCLs;
ods output PdiffCLs=ci2;
proc freq data=ci order=data;
    tables grp*res /riskdiff (CL=(EXACT));
    weight count;
    exact riskdiff(fmscore);
run;

/*
methods:
Newcombe Score (Corrected)
Wald (Corrected)
*/
ods select  PdiffCLs;
ods output PdiffCLs=ci3;
proc freq data=ci order=data;
    tables grp*res /riskdiff (CORRECT CL=(WILSON WALD));
    weight count;
    exact riskdiff;
run;

/*put them all*/
data CI_all;
    set ci3 ci2 ci1;
    CI=’(‘||compress(put(LowerCL,8.4))||’, ‘||compress(put(UpperCL,8.4))||’)';
run;

proc sql;
    select
    case Type
        when "Wald" then "a"
        when "Wald (Corrected)" then "b"
        when "Exact" then "c"
        when "Exact (FM Score)" then "d"
        when "Newcombe Score" then "e"
        when "Newcombe Score (Corrected)" then "f"
        when "Farrington-Manning" then "g"
        when "Hauck-Anderson" then "h"
        else "i"
    end as order,
        Type "method",
        CI "95% Confidence Limits"
    from CI_all
    order by order
        ;
quit;

———————————-

You may be also interested in this post on calculating confidence intervals for single proportion:

Statistical Notes (3): Confidence Intervals for Binomial Proportion Using SAS

using the following 11 methods:

• 1.  Simple asymptotic, Without CC | Wald
• 2.  Simple asymptotic, With CC      
• 3.  Score method, Without CC | Wilson       
• 4.  Score method, With CC    
• 5. Binomial-based, ‘Exact’ | Clopper-Pearson              
• 6.  Binomial-based, Mid-p      
• 7.  Likelihood-based            
• 8.  Jeffreys    
• 9.  Agresti-Coull,z^2/2 successes          
• 10. Agresti-Coull,2 successes and 2 fail           
• 11. Logit                                               

There is an important difference in philosophy between S (and hence R) and the other main statistical systems. In S a statistical analysis is normally done as a series of steps, with intermediate results being stored in objects. Thus whereas SAS and SPSS will give copious output from a regression or discriminant analysis, R will give minimal output and store the results in a fit object for subsequent interrogation by further R functions.

For SAS, it’s true, but not exactly the whole story. Actually all outputs generated by SAS procedures are stored in datasets besides the “copious output” in Output Window for further processing if needed. It is totally controllable like object in R. Take a regression example, You can use ODS Select statement to select what you want to be displayed in Output Window, and ODS Output statement to fetch the output datasets(test on SAS 9.3M1 in Windows 7):

ods select Anova ResidualPlot;
ods output Anova=Anova;

proc reg data=fitness;
   model Oxygen=Age Weight RunTime RunPulse RestPulse MaxPulse / selection=forward; 
run;
quit;

or, you can use ODS Exclude statement to just drop some outputs in Output Window:

ods exclude ResidualPlot;

proc reg data=fitness;
   model Oxygen=Age Weight RunTime RunPulse RestPulse MaxPulse / selection=forward; 
run;
quit;

To get a list all the ODS outputs like Anova, ResidualPlot, you can trigger on ODS Trace ON statement:

ods trace on;

proc reg data=fitness;
   model Oxygen=Age Weight RunTime RunPulse RestPulse MaxPulse
         / selection=forward; 
run;
quit;

ods trace off;

then you get a full list in Log Window like:

————-
Name:       NObs
Label:      Number of Observations
Template:   Stat.Reg.NObs
Path:       Reg.MODEL1.SelectionMethod.Oxygen.NObs
————-

Output Added:
————-
Name:       ANOVA
Label:      ANOVA
Template:   Stat.REG.ANOVA
Path:       Reg.MODEL1.SelectionMethod.Oxygen.Step1.ANOVA
————-

Output Added:
————-
Name:       SelParmEst
Label:      Parameter Estimates
Template:   Stat.REG.SelParmEst
Path:       Reg.MODEL1.SelectionMethod.Oxygen.Step1.SelParmEst
————-

Confidence Intervals for Proportions and Related Measures of Effect Size

This book also supplies a set of Excel spreadsheets to calculating CIs, see

http://www.crcpress.com/product/isbn/9781439812785

Other users may be interested on a SAS or R implementation.

A story goes with this post, that Bob won 20 times out of total 30 rounds with Alice in a children card game, “War”. War itself is a totally random game, so everyone is supposed to have 50% chance to win in a fair game. Since Bob won 20 out of 30, the question raised: is he cheating?

To answer this question, just follow the Dragon’s Teeth and Fleas logic of hypothesis testing:

• First, assume they are dragon’s teeth; in this case, we assume it is a fair game that everyone has probability of 0.5 to win.
• Under this assumption, we know the the number of winning follows binomial distribution with probability of 0.5. Now we should check Bob’s 20 winning in 30 in this distribution: is this output extreme, or is it a flea?
• We then get the p-Value calculated by P(X>=20) which means the chance of winning 20 times and more in 30 games (as extreme or even more extreme ), is 0.0494. Here we only think about winning, so it is a one-sided test. Since this p-Value is less than the default significance level of 0.05 (IT IS A FLEA!), we reject the null and think this is more likely not a fair game (Bob is cheating in this statistical point of view).

That’s all the story. Then this post goes, “Although the procedure is mathematically well defined, it is nonsense”. The following is the first reason listed in the post:

To calculate the probability of Bob’s winning at least 20 games, you use not only the data that you saw (that Bob won 20 games out of 30) but also the imaginary data of Bob winning 21 out of 30 games, or 22 out of 30 games, all the way up to Bob winning all 30 games. The idea of imaginary data is oxymoronic.

In my opinion, the so called “imaginary data” are just the data filling out the binomial distribution (under the assumption of true null hypothesis) we got before and they are not oxymoronic at all (sorry I use “data” as plural). 

What do you think? I can’t go far away from here due to my limit knowledge of Bayesian.

————-appendix: SAS Codes and Outputs——————

data a;
    input output count;
datalines;
0 20
1 10
;

proc freq data=a;
    tables output /binomialc (p=0.5) alpha=0.05;
    exact binomial;
    weight count;
run;

Statisticians aren’t the problem for data science. The real problem is too many posers — Cathy O’Neil

you actually do need to understand how to invert a matrix at some point in your life if you want to be a data scientist. — Cathy O’Neil

I was asked in several different occasions to explain hypothesis testing to non-technical people in plain English. Now I think I got a pretty neat one while honors belong to three great Germans, Friedrich Engels, Karl Marx, and Heinrich Heine. Engels wrote that Marx once quoted a saying from Heine that “I have sown dragon’s teeth and harvested fleas” (but I didn’t find the original source):

In terms of hypothesis testing, say, if you plant dragon’s teeth, you are not supposed to harvest the fleas; but if you do get the fleas, then most likely they are not dragon’s teeth at all!

To make up the story as simple as possible while keep most of the key messages, here take a one-sample T-test example from SAS TTEST Procedure User Guide, where the investigated data is the Degree of Reading Power (DRP, a measurement of children’s reading skill) from 44 third-grade children. The goal is to test if the mean score of DRP is equal to 30:

Null        Hypothesis (H0): μ = 30

Alternative Hypothesis (H1): μ ≠ 30

For the following discussion, three statistical measures needed (number of cases, mean and standard error):

proc means data=read maxdec=4 n mean stderr;
    var score;
    freq count;
run;

and the output:

Here we go:

1. Suppose H0 is true that the mean score is equal to 30 (population mean μ = 30, we assume they are dragon’s teeth!).

2.  We know the sample mean is the best estimate of the population mean, here it is 34.8636.

3. This sample mean is derived from a sample (namely, the 44 children). We’d like to know if this sample (with mean of 34.8636) really come from the population (with mean of 30).

4. To get to know this, we can repeat this survey (or trial, experiment), for example, take another 99 samples (also with 44 children in each sample) and calculate their sample means (we then have 100 different means from the whole 100 samples).

5.  We may know these means (got from such mechanism), with some necessary mathematical transformation,  follow a t-distribution with degree of freedom of 43 (df = 44 – 1):

The denominator as a whole is the standard error.

6. Now we can check our interested sample (with mean of 34.8636) in this t-distribution. There is a corresponding t-value of this sample in the distribution, (34.8636 – 30) / 1.6930 =  2.8728:

7. In this distribution, there are only 0.63% (0.0063) of t-values either above 2.8728 or below -2.8728 (symmetry of t-distribution). What does this mean?  Obviously, our interested sample (with mean of 34.8636 and t-value of 2.8728) is not in the mainstream of the whole distribution. We can even think it is  very “extreme”  because, there are only 0.63% of elements in the whole that are more (or at least as equal) extreme than it.

8. We call this number, 0.63% (0.0063) the P-value. According to wikipedia:

the p-value is the probability of obtaining a test statistic at least as extreme as the one that was actually observed, assuming that the null hypothesis is true.

Here the test statistic that was actually observed is 2.8728, the t-value from our interested sample. And we got from this t-distribution that P{|t|>2.8728} = 0.0063.

9. A question is, how extreme is extreme? We can’t define “extreme” exactly, but probably, we can. The threshold is the significance level α (usually 0.05). As a rule of thumb, if p-value is less than the predefine significance level α, then  we think the event associated  with this p-value is pretty extreme.

10. Retell our story: first we assume the population mean is 30 just as the null hypothesis claims (assume they are dragon’s teeth), and under this assumption,  we built a t-distribution; we then got that our interested sample is an extreme case (it is a flea!). It is so extreme (much less than 0.05) that we can even think about that our assumption (the null hypothesis) is far away from true (most likely they are not dragon’s teeth at all!). We reject the null hypothesis and take the alternative that the mean score is different from 30.

11. Formally, it just follows a simple logic (taken from Glenn Walker and Jack Shostak, P.18):

In our story, P is the null hypothesis, while Q is the extreme case: if H0 is true, most likely such extreme case should not happen; now we observe the extreme case (a t-value of 2.8728 in a t-distribution with degree of freedom of 43), then it is reasonable to question the null hypothesis itself.

12. To end this analysis, we can take a look at the output of SAS PROC TTEST:

proc ttest data=read h0=30;
   var score;
   freq count;
run;

The same result (P-value 0.0063 < 0.05, reject H0):

SAS Notes

12. In this example, input data take a form of cell count data, not as raw as the case-record data, so in both PROC MEANS and PROC TTEST, a “FREQ” statement added.

13. You can compute this P-value in step-6 with SAS using ProbT function:

data;
    t    = 2.8727938571;
    df   = 43;
    tail = 2;
    P    = tail*(1-probt(t,df));
    put P = ;
run;

According to the symmetry of t-distribution, you should multiple 2 to get the two-sided probability:

14. Long long ago, a so called “critical value” is calculated to support the decision. That’s the calculation method using TINV function:

data;
    alpha = 0.05;
    tail  = 2;
    df    = 43;

    tCritic = tinv(1-alpha/tail,df);
    put tCritic=;
run;

We get this critical value of 2.0166921992:

These two shaded zones are called “rejection zones”. The rule is, if the t-value we observed lies within the rejection zones, we should then reject the null hypothesis and take the alternative. In step-6, the t-value we calculated is 2.8728, which is bigger than this critical value, so we should reject H0.

Critical value approach is equivalent with the p-value approach we discussed before, but slightly old fashioned. P-value is pretty intuitive and can be also easily calculated by computer. In days when computing power was not wildly available, people just took the t-distribution table to look up the critical values.

Most statistical packages including SAS software don’t report the critical value (but still alive in statistics text books).

Reference

SAS PROC TTEST USER GUIDE

Common Statistical Methods for Clinical Research with SAS Examples by Glenn Walker and Jack Shostak

Hypothesis Testing: A Primer (in Chinese)

Two sides T-distribution calculator

Single side t-distribution calculator

Online Latex Equation Editor

A guy notices a bunch of targets scattered over a barn wall, and in the center of each, in the "bulls-eye," is a bullet hole. "Wow," he says to the farmer, "that’s pretty good shooting. How’d you do it?" "Oh," says the farmer, "it was easy. I painted the targets after I shot the holes." – An Old Joke

Last year I dumped piece of SAS codes to compute confidence intervals for single proportion using 11 methods including 7 presented by one of Prof. Newcombe‘s  most wildly cited papers, Two-sided confidence intervals for the single proportion: comparison of seven methods (r is the number of event in total cases of n):

To get such results using my macro, just submit the following SAS codes:

filename CI url ‘https://raw.github.com/Jiangtang/Programming-SAS/master/CI_Single_Proportion.sas’;
%include CI;
%CI_Single_Proportion(r=81,n=263);

And the outputs(plus 4 additional methods):

In SAS 9.2 and 9.3, 6 of the 11 intervals can be calculated by PROC FREQ. First, method 1, 3, 9 ,8, 5 via a binomial option:

data test;
    input grp outcome $ count;
datalines;
1 f 81
1 u 182
;

ods select BinomialCLs;
proc freq data=test;
   tables outcome / binomial(all);
   weight Count;
run;

The outputs:

And you can get another Wald interval with a continuity correction (method 2) via binomialC option:

ods select BinomialCLs;
proc freq data=test;
   tables outcome / binomialc(all);
   weight Count;
run;

get:

R Notes

These two R packages contain a bunch of CI calculations:

Binom

PropCIS

For SAS macro %CI_Single_Proportion, I borrowed a lot from this R implementation:

http://www2.jura.uni-hamburg.de/instkrim/kriminologie/Mitarbeiter/Enzmann/Software/prop.CI.r

Additional Notes

If interested, a nice paper on binomial intervals with ties, the paper Confidence intervals for a binomial proportionin the presence of ties,  and the program.

Programmers Need to Learn Statistics Or I will Kill Them All –Zed A. Shaw

In an equivalence testing example against lognormal data,  a TOST (Two One-Sided Test)  option used in SAS TTEST procedure:

proc ttest data=auc dist=lognormal tost(0.8, 1.25);
   paired TestAUC*RefAUC;
run;

And the output:

Since the 90% (who not 95%? see below) limit of the geometric mean ratio (0.9412) , [0.8634,1.0260], lies in the FDA endorsed equivalence bounds [0.8,1.25],  then we can make a conclusion of “Equivalent” among these Test and Reference drugs. Following are the notes on how this so called TOST (Two One-Sided Test) works.

In its original form, the hypothesis of equivalence testing is written as difference on means (equivalence by definition: not above the upper limit, and not below the lower limit):

Since our AUC example with lognormal data, the difference on means is transformed into a ratio,  geometric mean ratio: if no difference in means, then the ratio simply equals 1. The two one-sided tests are (with form of ratio, not difference anymore):

I) upper one-sided T-test with a null ratio as the left bound of predefined limits, here is 0.8

II) lower one-sided T-test with a null ratio as the right bound of predefined limits, here is 1.25.

T-test I can be done as (with option sides=U, where U means upper one-sided test. This option also only goes with SAS 9.2 and later)

proc ttest data=auc dist=lognormal h0=0.8  sides=U;
   paired TestAUC*RefAUC;
run;

And the output:

T-test II, lower one-sided (with option sides=L, where L means lower one-sided test):

proc ttest data=auc dist=lognormal h0=1.25 sides=L;
   paired TestAUC*RefAUC;
run;

And the output:

So, the two one-sided tests together get a limit of [0.8634,1.0260], which is exactly as same as we got before using TOST option, except the former noted as “95%” and the later, “90%”. That’s because, the 0.1 (1-90%) should be dividend by 2 for the two one-sided tests to ensure they can both get their 0.05 significance respectively. Namely, at significant level α=5%, the equivalence test gets a  100(1-2α)% = 90% interval.

At last, the overall P-value, will take the maximum one in these two one-sided tests (here we get max{0.0031,<0.0001} = 0.0031).

SAS Notes

Just did a quick search, PROC MIXED seemed more popular for statisticians to perform equivalence test years before:

Cookbook SAS Codes for Bioequivalence Test in 2x2x2 Crossover Design, by Chunqin Deng (2012)

The Paired T-Test: Does PROC MIXED Produce the Same Results as PROC TTEST? by Jack Nyberg (2004)

Section 8.2 Bioequivalence Testing, in Pharmaceutical Statistics Using SAS: A Practical Guide, by Alex Dmitrienko, Christy Chuang-Stein, and Ralph D’Agostino (2007)

Reference

SAS TTEST Procedure User Guide

Beyond the t-Test

Programmers Need to Learn Statistics Or I will Kill Them All –Zed A. Shaw

Just read since SAS 9.2, the TTEST procedure also natively supports Equivalence Test by simply adding a TOST option (Two one-sided tests). In a example, TTEST procedure reports a geometric mean as 0.9412, which is the geometric mean of a ratio, TestAUC/RefAUC. It can also be calculated manually by

proc sql;
    select exp(mean(log(TestAUC/RefAUC)))
            "geometric mean of TestAUC/RefAUC"
    from auc;
quit;

Actually this kind of geometric mean (of a ratio) is more often called geometric mean ratio(a ratio of two geometric means, in this case, geometric mean of TestAUC and geometric mean of RefAUC). Similarly,

proc sql;
    select exp(mean(log(TestAUC))) as gmTest
                "geometric mean of TestAUC",
           exp(mean(log(RefAUC)))  as gmRef 
                "geometric mean of RefAUC" ,
           CALCULATED  gmTest / CALCULATED  gmRef as gmr
                "geometric mean ratio"
    from auc;
quit;

78.315/83.2077 (the ratio of two geometric means) simply gets 0.941199 (geometric mean ratio), which can be also derived by calculating the geometric mean of a ratio (see above). You can prove it mathematically by playing some log-transformations:

geomean(A) / geomean(B) = geomean(A / B)

In vaccine trials (I worked before) where the interested values are antibody titers,  the geometric mean is also called Geometric Mean Titer (GMT), while geometric mean ratio  referred as Geometric Mean Titer Ratio (GMTR, also named “n-fold rise”). Both GMT and GMTR are wildly presented in statistical analysis reports.

Usually, AUC data mentioned above (Area Under Curve, which is based on blood levels) and antibody titer values are log-normal distributed(source data, noted as X), which means, if you take a log-transformation against X, namely Y=log(X), then Y follows the normal distribution (then Y has good properties to perform further statistical analysis). Suppose two treatment groups available, 1 and 2.

Few extremely important conclusions, just as the codes above suggested, exp[mean(Y)]=exp{mean[log(X)]} is just geometric mean of X, and the geometric mean ratio, geomean (X1) / geomean(X2), simply equals exp[mean(Y1) – mean(Y2)]—This is the mathematical equivalence of ratio and difference.

Back to the AUC example, the FDA endorsed equivalence bounds offered,  0.8 and 1.25, which are both ratio values. So, in the analysis, the original values are transformed to geometric mean ratio (0.9412 with 90% limit of [0.8634 , 1.0260]) to compare with the predefined bounds. For details of equivalence testing, I may extend it into a separated note.

SAS Notes

In SAS/Base, GEOMEAN function is used to calculate the geometric mean. It takes a list of values as argument, which means, it can only perform on rows, but not on columns. So, if need the geometric mean of a variable, TestAUC for example, you should transpose the rows to columns or similarly, use a array:

/*1. transpose*/
proc transpose data=auc (keep=TestAUC) out=test prefix=t;
    var TestAUC;
run;

data _null_;
    set test;
    gmTest=geomean (of t1-t12);
    put gmTest=;
run;

/*2. array*/

data _null_;
    if 0 then set auc(keep=TestAUC) nobs=nobs;
    call symputx(‘nobs’,nobs);
    stop;
run;

data _null_;
    set auc(keep=TestAUC) end=eof;
    array gm{&obs} _temporary_;
    gm{_n_}=TestAUC;
    if eof then do;
       gmTest=geomean(of gm{*});
       put gmTest=;
    end;
run;

Both methods return to the geometric mean of TestAUC as 78.314997096. The SQL solution I used above, exp(mean(log(TestAUC))), is much simpler, and most important, it reminds us  the applying of log-transformation.

Reference

Appendix F – Data Transformations in Common Statistical Methods for Clinical Research with SAS Examples by Glenn Walker and Jack Shostak supplies a concise discussion on log-transformation.  Btw, this book is the No.1 statistical book I found ever for SAS programmers without statistical background.

1. The Pain

I read from Lex Jansen (@LexJansen) that CDISC SDTM v1.3 and SDTMIG v3.1.3 were newly released. It’s pretty nice since CDISC SDTM was supposed to be released semiannually in the new publishing cycle. We can see the team put great efforts on this new version, but frankly speaking, this delivery (the way to display, not the content itself) is far away elegant.

The new SDTM Implementation Guide (IG) v1.3 is just a temporary workaround shipment, as an embedded file “How to Use SDTMIG 3.1.3” indicates,

SDTMIG 3.1.3 is presented as an annotated version of SDTMIG 3.1.2. This approach was taken for SDTMIG in order for the document to be released quickly without an extensive rewrite. The content presented as annotations will be incorporated into a single version of documentation in a future release.

What does “annotated” mean? When you replace “should” to “must” in the file,

• strikethrough the word “should”
• insert the replacement “must”, and
• add a sticky note to indicate the change above

This is annoying. There are 143 sticky notes throughout the whole documentation, including replacement, deleting, files attachment and such and the reason, is said to ensure “the document to be released quickly without an extensive rewrite”. BUT 143 sticky notes in a PDF file! it’s already huge editing effort ever!

2. The Reason (or The Conjecture)

Almost everybody complains of Microsoft Office Excel and Word, but Ура(!), they are still dominant in our working spaces (especially heavy in clinical world? I’m not sure). I didn’t have any personal connection with CDISC publishing team, but from the documentation released, I’m pretty confident that these files (SDTMIG v3.1.3 and others) were edited in Word and then published into PDF via Adobe products (very common practice, isn’t it?).

Now you may understand why CDISC publishing team delivered this “annotated”  version due to limited time and human resource (although editing 143 sticky notes was also a big work load). The clue is Word! Word! Word!

Microsoft Word is extremely popular for its WYSIWYG (What You See Is What You Get), but it can’t separate contents from formats and it will a disaster when maintaining a frequent updated Word file by multiple users. In this CDISC SDTMIG case,  there are about 143 content updates supplied by CDISC community worldwide, but when applying such content updates to the original Word file, you are always reasonable to worry about that such updates would change something(yes SOMETHING) unexpectedly! The biggest concern for CDISC standard files, I guess, again with confidence, is if such updates destroy the in-text links  or other cross references which offers the nice navigation throughout the documentation.

So, this “annotated” version at least is safe (and SAFE is much more important than what it looks): no links proven worktable in v3.1.2 will broken in this time pushing new release, and things would get better in the future (from the same source, “How to Use SDTMIG 3.1.3”):

CDISC is currently discussing how future documentation will be published ensure documentation is easy to navigate and read and at the same time easy to maintain.

3. The Prospective

Yes I will end with a (set of) suggestion(s). The bottom line is no Word anymore and I promise no additional cost and pain compared to digging into Microsoft Word and Adobe Acrobat.

Take SDTM IG v3.1.3 as a demo project:

• Convert all the contents of SDTM IG v3.1.2 (from PDF, or original Word) to a text based format. Personally I prefer Markdown and reStructuredText. Actually it doesn’t matter which one is chosen for test purpose, because such text based formats can be easily transferred (much easier than from Word/PDF). The benefits of these two formats are separation of contents and formats, and very intuitive to learn (much easier than HTML; almost WYSIWYG). This task is machine doable somehow but also needs manually modification. But all in all,  it is not a big deal, it is only about 300 pages.
• Edit these text files according to the new SDTM IG v3.1.3 updates.
• Distribute these text files (and rendered output files in PDF/ HTML formats) to a vendor supported or self hosted collaborating site, like GitHub.
• Call for CDISC team members and users to report any issues and even encourage them to directly edit them online (don’t worry, it won’t be mess; we are in a version control system like GitHub). 
• Then the next version will come out naturally (and peacefully).

then I’m looking forward to hearing your ideas.

4. Additional Notes

The markup standards mentioned above in my proposal,  Markdown and reStructuredText, are not replacement for CDISC metadata standard, ODM and its XML derivatives Define.xml. Instead, they are better formats to get rid of Microsoft Word for community collaborating of editing the “narrative” parts of models (the PDF files we read from CDISC), for example, SDTMIG we discussed before.

1. File Explorer: Clover

http://ejie.me/

There are lots of fancy/complicated file/folder management system beyond Windows Explore (Total Commander might be the most famous one) while Clover, as its name indicates, is just as simple as I need:

• Ctrl+T: open new tab
• Ctrl+W: close tab
• Ctr+tab: switch among tabs

And these shortcuts also work at the following applications (so, nothing new need to learn).

2. Web Browser: Chrome

https://www.google.com/intl/en/chrome/browser/

While IE, as a saying goes, is a best browser to download Chrome(or Firefox if you like).

3. PDF Reader: Foxit Reader

http://www.foxitsoftware.com/Secure_PDF_Reader/

Get rid of Adobe Reader. Foxit Reader is no doubt the best of best PDF reader as it shows (and it’s free).

4. Office Suite: Office Tab

http://www.extendoffice.com/product/office-tab.html

Office Tab is a Microsoft Office add-in and it works (to launch Word, Excel and PPT files to tabs)!

5. Text Editor: Notepad++/Sublime Text 2

http://notepad-plus-plus.org/

http://www.sublimetext.com/

I use Notepad++ for a long time and love it, but when I take a look at Sublime Text 2, I should admit that it has much of potentials to serve as my default text editor in the future:

/****update: 2012-10-16, Roy  Pardee maintains a nice bundle (here), so the following bundle will be not updated any more.***********/

Also created a Github repository to hold all my SAS configuration files for Sublime Text 2. Fork me at

https://github.com/Jiangtang/sas.tmbundle

1. SAS syntax highlighting

Sublime Text 2 doesn’t support SAS syntax natively. I got a workaround so I didn’t need to write my own to play with it in the validation stage. Since  Sublime Text 2 supports TextMate’s syntax configuration, Just borrowed a Textmate’s SAS syntax coloring theme from Jakob Stoeck (thanks a lot man!).

This SAS coloring theme is Github hosted, so you can clone the files to the user created “SAS” folder (C:Program FilesSublime Text 2dataPackagesSAS; I use a Win 7 machine) if your Git is installed properly:

git clone git://github.com/jakob-stoeck/sas.tmbundle SAS.tmbundle

or you can just simply save and copy this file to the SAS folder above:

https://raw.github.com/jakob-stoeck/sas.tmbundle/master/Syntaxes/SAS.tmLanguage

Also can get from my repository forked from Jakob:

https://github.com/Jiangtang/sas.tmbundle/tree/master/Syntaxes

Then you get

Pretty nice, isn’t it? One of the neat functionalities of Sublime Text 2 is to open a folder as a project (showed in the left panel).

2. VI(M) Emulation

We can also easily launch Vi(M) mode in Sublime Text 2, see

http://www.sublimetext.com/docs/2/vintage.html

https://github.com/SublimeText/VintageEx

3. Add Comments

Use this file from my repository, Comments (SAS).tmPreferences, then you can use the Sublime Text 2 default key shortcuts

Ctrl+/ to add *comment; style comments(toggle), and

Shift + Ctrl+/  to add /*comment*/ style comments (toggle block)

Note that in SAS Enhanced Editor, we use Ctrl+/ to add toggle block comments.

4. Code Autocomplete

I added a few snippets in my repository to support some code autocomplete.

For example, when type “s” and few alternatives show up,

Hit RETURN and get

The neat thing is that your cursor is just in the shadow area /* variable(s)*/ after hitting RETURN where you can overwrite.

5. Run SAS within Sublime Text 2

Copy the following system build file to SAS package folder (mine, C:UsersjhuAppDataRoamingSublime Text 2PackagesSAS):

{
“cmd”: ["SAS","-sysin","$file"],
“selector”: “source.sas”
}

https://github.com/Jiangtang/sas.tmbundle/blob/master/SAS.sublime-build

Then you can run SAS code in Sublime Text 2 (by clicking “Build” or using shortcut Ctrl+B):

Actually it is batch run mode and you will get all the output files (.log, .lst and .pdf, .png and such) in the same folder you hold you SAS codes.

I’m still working on how to set up interactive run mode and please leave any suggestions, comments and hints!

I remember when blogging was cool.

Before the specializing and monetizing and Twitter-izing.

                                      —Peter Dewolf

Well I think blogging is still cool (and awesome and awesome …). The most appealing personal reason is, blog posts are Google searchable and suitable for archive while Tweets NOT. Admittedly I hold some sort of  Existentialism 2.0:

if it is not Google searched, it doesn’t exit!

Last month I placed a post on how to keep pace with CDISC from its official channels and I feel cool to add an appendix of source from the awesome blogosphere. Fortunately or not, CDISC is still in the niche market of topics and it takes few efforts to get the list(update me if someone else available! if you are a Google Reader user, just simply import this file, my Google Reader subscription on CDISC):

1. Blog @ Assero by Dave Iberson-Hurst (“Dave IH”)

http://www.assero.co.uk/category/blog/

Insightful and full of humor. I retweeted all of its latest posts and you can feel somehow on these tittles (YES on CDISC):

What I Want, What I Really Really Want

Churchill, the FDA and a Fall

Mad March and the FDA

Btw, I write blogs casual way while it is very impressive to read IH reminding me the George Orwell style.

2. d-Wise Technologies Blog

http://www.d-wise.com/blog/

It is my employer’s official blog site where Chris Decker is the key contributor to CDISC. You can check out his latest posts on FDA/PhUSE Annual Computational Science Symposium where he served as committee lead:

Overcoming Industry Challenges: A Shift to Collaboration

Validation and Quality: Are They the Same?

I will also commit to update this blog as my understanding on clinical standards goes. Here is the saying:

look to the master,
follow the master,
walk with the master,
see through the master,
become the master.

3.  XML4Pharma Blog

http://cdiscguru.blogspot.com/

with industry news and hard (while cool) way writing on XML (CDISC ODM, define.xml).

4. eClinical Trends by Clinovo

http://blog.clinovo.com/category/cdisc/

Clinovo jumps to this topic by launching a CDISC SDTM convertor CDISC Express.

5. eClinicalOpinion

http://eclinicalopinion.blogspot.com/

This blog is most focused on EDC, the clinical data management part. I like its series discussion on CDISC ODM.

6. eCTD Regulatory Submissions Network

http://ectdregulatorysubmissionsnetwork.blogspot.com/

This is a personal blog by Shakul Hameed. I read it mostly to get some information on submission requirements from European regulatory.

7. HL7 Watch

http://hl7-watch.blogspot.com/

while it is not CDISC directly related (#6 also), it’s nice to get some voice of HL7 which would be the future of CDISC.

8. From a Logical Point of View-CDISC

http://www.jiangtanghu.com/blog/category/cdisc/

Yes this one, my 2 cents. I will keep recording my personal immersion and understanding on CDISC and related clinical standards. (while it is privilege to cross reference oneself in his/her own blog! Keep awesome, keep blogging.)

9. Linked Data and URI:s for Enterprises

http://kerfors.blogspot.com/

Look at the colon (:) in the title of this blog and you’re right this blog plays (at least) with XML. I find it is good resource (thanks @kerfors for referencing!) to learn ODM, the foundation of CDISC while the latest post is

Semantic models for CDISC based standard and metadata management

P.S.: Blogger Chris Hemedinger maintains a nice list of SAS bloggers (blogs by SAS employees, and blogs by SAS customers, consultants, and the analytics community).

This picture (by newsobserver.com) took from Wake County Library Book Fair in North Carolina Fairground where I also went to pick up some books.

I was and still is a big book (almost paper books) fan. But when I was standing among the 450,000 used books in the book fair, I felt depressed and reluctantly had an impression: the paper  books will and should go away in the future.  eBooks will not fade away and they also have the economical, ecological and even aesthetical advantages.

Books are good(hundred and thousand years ago, our ancestors read books in silk, bamboo, shell, stone, . . .), new paper books are good too. But the massive used books are burdens. I like the smell of ink of paper books but only for the new. The 450, 000 used books only have some recycling values: you can get a bag of books only for only 1 buck in the last exhibition day!

Digital Life and Personal Data Analysis

eBook is one part of our digital library. We also have eMails, eBills, Tweets . . . and these data are born for analysis: complete, accurate, well formatted while in time(again, analytical advantage!).

Stephen Wolfram, the founder and CEO of Wolfram Research, posted an analysis of his personal emails as above. That’s cool and you might know that years ago a writer would transcribe at least two copies of his/her mail, one for sending out and another for archiving (and publishing).

Liza Lucas of SAS JMP R&D also contributed a page on her graphical financial life while the data come from a online personal financial management system:

Back to reading. Below is my daily readings via Google Reader (in laptop and phone) in the past 30 days:

Haa, I should clean up my subscribes now.

Here is a simple approach. Just submit the following one line of codes in a shell,

wget http://evs.nci.nih.gov/ftp1/CDISC/ -r –no-parent  -l 3

and you will get all the CDISC Control Terminology files (plus historical versions) with proper folder structures in your local driver (the current directory of your shell).

For syntax details, you can refer to its online manual:

-r: recursive retrieving

–no-parent: only fetch files under the URL

-l 3: set maximum depth level of 3  

If you are a Windows user, you might install Wget for Windows (it is a native tool in Unix/Linux under GNU)and add it into your environmental variable, Path. You can also save the above scripts in a notepad and save it as a .bat file(test.bat for example). Next time you just click the test.bat to get all the updates.

Wget is a very powerful tool. For me, the download speed is pretty acceptable (depends on internet connection) and almost no difference between a Windows 7 and a Ubuntu 11 machine:

1. people

Almost fresh faces for me. It’s nice to meet Jack Shostak of Duke Clinical Research Institute again. I visited him in Duke last year after SAS Global Forum in Las Vegas. Jack has a forthcoming book on SAS and CDISC, Implementing CDISC Using SAS: An End-to-End Guide. It’s the first book on this topic and worth waiting!

I also met (unexpectedly and exciting) a Chinese friend Chunmao in the meeting. Very interesting: after introduction, then we got that we emailed on CDISC mapping before! Chunmao just moved from DC to Triangle as SAS programmer weeks before(a side message: Triangle is hiring!). Big bonus to attend this meeting.

My colleague Chris Decker of d-Wise Technologies also showed up in this meeting. Actually he and Jack both serve as committee members in RTP CDISC User’s Group (they are also core members in CDISC community worldwide).

Tom Soeder of Cato (venue supplier for this meeting) kindly served as host while Jeff Abolafia of Rho the moderator.

2. agenda

Jeff and another key member of this group introduced some important updates from CDISC. One of the most interesting messages for me is the regular release cycle of SDTM Model and Implementation Guide. SDTM will be released semiannually, so we will get SDTMIG 3.1.3 in this summer, 3.1.4 at the end of year which will mainly hold the recently updates of Trial Summary, an amendment,  and CDISC Devise domains respectively.

SDTM is the flagship model of CDISC. SDTMIG 3.1.1 published in 2005 while 3.1.2 in 2008. It’s nice to see from the new more frequent release schedule that the CDISC community is expanding (and more organized and expected).

Recently SDTM does have lots of updates, including a copy of the Metadata Submission Guideline (MSG). CDISC organization will also offer periodic webinars on updates.

Chris then gave a summary on latest FDA/PhUSE Computational Science Symposium (CSS while Chris organized it). You may get more information on Chris’s blog, and CDISC blog. It’s better to keep CSS in watch list.

Jack and Jeff had comments on working the FDA/PhUSE working groups.

Peter Schaefer of Certara released the outputs of latest CDISC user network servey where SDTM and ADaM are still on the top of user’s list.

Final part (most practical), group exercises! Three groups were assigned to map some challenging CRF pages to SDTM. Some users also took some CRF pages from their own companies for public discussion (nice to have some flavors!).

3. Links

RTP CDISC User’s Group on Yahoo Group (the traffic is low but still informative):

http://tech.groups.yahoo.com/group/rtp_cdisc/

CDISC official site:

http://www.cdisc.org/

Now we have more reasons to visit CDISC website frequently for new updates models (e.g., Control Terminology also released semiannually) and webinar postings.

FDA/PhUSE working groups Wiki:

http://www.phusewiki.org/wiki/index.php?title=PhUSE_Wiki

Lots of action followed by the six working groups.

Chris is one of the core members to promote CDISC among industry and regulator  and he is also the most active blog writer on d-Wise blog and you can get informed:

http://d-wise.com/blog/

One of the interesting observations in my first few months in US: there is no price printed in the new mail stamps!

It is interesting because as a former student of economics, I think US Post system did a nice attempt to beat the so called “menu cost” which should honor to Harvard economist Mankiw. Literally the menu cost is the cost of a restaurant to print new menus due to price adjustment. In this story, it is the cost of US Post to print new stamps with adjusted price.

Why US Post need to print new stamps with new price? It is because of the inflation! I took a quick search, the first class postage rate was 5 cents in 1963 and never changed until 1968. But during this 5 years, the CPI (consumer price index, one of the robust measures of inflation; data see here, codes see below) increased from 30.6 to 33.4, which means, the actual price (see red line below) of a postage in 1967 was even less than in 1963!

Why US Post didn’t adjust the postage rate accordingly in these years? In a free market, prices are supposed to be adjusted immediately upon market demands and supplies. A school of economics argues that even in free markets, the prices can be sticky. One of causes of such price stickiness is the menu cost. In this case, US Post knew that the actual cost of posting a mail was up, but it still didn’t will to print new stamps with new price due to printing cost itself (suppose there were lots of stocks) and even the cost to inform people(and such small costs).

Now US Post doesn’t want to lose money by supplying mail services with a price under market value while also want to avoid so called menu cost, then a stamp without price printed seems good enough to solve such dilemma!

/*———–codes for graphics go here and it’s nice to post the first SGPLOT in this blog while watching NCAA game of NCSU vs Kansas————–*/

data rate;
    input Year Nominal Real CPI;
datalines;
1866    3    44.0    15.4
1867    3    47.4    14.3
1868    3    49.1    13.8
(data see here)
;

proc means data=rate;
    var Nominal Real CPI;
run;

proc sgplot data=rate;
    title "History of United States CPI";
    series x=year y=cpi;
    xaxis values= (1870 to 2010 by 10);
    refline 100 / transparency = 0.5;
run;

proc sgplot data=rate;
    title ‘History of United States Postage Rates’;
    series x=Year y=Nominal;
    series x=Year y=Real;
    xaxis values= (1870 to 2010 by 10);
    yaxis label=’Rates’;
    refline 10 44 / transparency = 0.5 
          label = (‘Nominal (Mean)’ ‘Real (Mean)’);
run;

Then it turns that I was wrong. My friend Venkat Veerisetty just told me that this feature was available since SAS v8. I also found a machine with SAS v9.2 and it works.

While not-documented-not-exist is obviously an arbitrary assumption regardless of facts, I still hold it somehow when emphasizing the importance of documentation.  Chris Hemedinger gave a reason why there are some undocumented “behaviors” in SAS. It makes sense, while in this case, it would be just happened to be missed in other documents and then newly added in v9.3.

To determine if 2012 is a leap year, use the straightforward logic expression (demo is SAS, follow the pseudo code in the wiki page of leap year):

if mod(year,4) = 0 then do;  
    if mod(year,100) = 0 then do;          
        if mod(year,400) = 0 then is_leap = 1;
        else is_leap = 0;
    end;
    else is_leap = 1;
end;
else leap=0;

while using logic operators (Boolean expression), only one line of codes needed:

is_leap = (mod(year,4) = 0) and
          (
            (mod(year,100) ^= 0) or (mod(year,400) = 0)
          );

or using SAS bitwise logical operations:

is_leap=bor(band(mod(year,4) = 0,mod(year,100) ^= 0), mod(year,400) = 0)

https://github.com/Jiangtang

Currently I only created one repo as personal SAS code repository. To kill weekend time, I uploaded piece of codes to count the OpenCDISC validation rules by models. To use it:

filename CDISC url “https://raw.github.com/Jiangtang/Programming-SAS/master/Rules_Count_OpenCDISC_XML.sas”;

%include CDISC;

%Rules_Count_OpenCDISC_XML(dir=C:tempOpenCDISCsoftwareopencdisc-validatorconfig)

while get:

Happy weekend and happy programming.

Currently this column mode only supports selecting (highlighting) but not editing. Hoo, it’s better than nothing.

Update: the column selection mode was available since SAS v8, see

http://www.jiangtanghu.com/blog/2012/02/29/not-documented-not-exist/

• SDTM 3.1.2
• SDTM 3.1.1
• Define.xml 1.0
• ADaM 1.0

You can get the corresponding configuration files (validation rules) online or in the software folder (in ..opencdisc-validatorconfig with extension of .xml). Since SDTM 3.1.2 has the most rich set of validation rules from Janus, WebSDM and of course additional  OpenCDISC rules by itself, its configuration file (config-sdtm-3.1.2.xml) deserves more attention. Better understanding of config-sdtm-3.1.2.xml is the first step to customize the software according to business needs. Followings are some personal tips and tricks to play and even “torture” the file, using Notepad++, web browsers (IE and Firefox), Excel with MSXML and SAS XML Mapper.

1. DON’T use the Windows default Notepad to open and edit the xml file

while the reason:

if you use Notepad to open a XML file, almost you get nothing but strings and strings.

and another supporting reason, see bellowing picture.

2. USE Notepad++ or other REAL text editors to open and edit it

Notepad++ makes the difference. It supports multiple tabs view, XML syntax highlighting and XML tags match and other fancy stuff never in the plain Notepad. And like OpenCDISC, it’s free, both in sense of free beer and free speech.

Other real text editor, include Vim, UltraEdit and such, but for most users, I still think Notepad++ is the most handy one.

3. At first, use a web browsers to review it

It is the web view of config-sdtm-3.1.2.xml. The secret is a style file, define-1.0.xsl in ..opencdisc-validatorconfigschematron. This is another story of dichotomy. The config-sdtm-3.1.2.xml file itself is only used to store metadata (machine-readable), while the style file (also a XML file) used to instruct how to display it (human-readable). Within some proper internal interface, web browsers (I tested in IE and Firefox; Google Chrome doesn’t work). Excel can also render this XML file well (only test on Excel 2010 and 2007) while Web view is much better:

4. The real awesome job: use Microsoft XML parser or other XML parsers to dig into XML structure

I use Excel 2010 with Microsoft XML parser (MSXML 6.0. You can get the version of your MSXML by visiting this website in IE and you will get the different results when switching to other web browsers because Firefox and Chrome use other parsers).

You can also get a instance of each XML tag:

5. The real awesome job: use SAS XML Mapper to get the tabulation view

And you may want to exact all the tables in the XML file with tabulation view, ideally, in SAS dataset:

For example, the first few rows in config-sdtm-3.1.2.xml:

and the corresponding SAS dataset:

Actually you can put all the data in XML into a big dataset but with lots of redundancies. To use SAS XML Mapper (the latest version is 9.3), you should design a mapping file to tell the structure of the XML file. For the simple ODM dataset, you indicate the table name, column name and path, type and length:

It never be fun to play with XML files. SAS XML Mapper is supposed to read CDISC ODM based XML files automatically (OpenCDISC XML files are called ODM compliant), but at least for this config-sdtm-3.1.2.xml, it failed and that’s why we should create a mapping file (see above) by ourselves. Fortunately you don’t need to write it from scratch (it would be thousands of lines of codes):

• find a CDISC ODM based XML file that SAS XML Mapper can read automatically, e.g., in http://www.cdisc.org/define-xml, a file named define-example1.xml works well.
• use AutoMap function in SAS XML Mapper to get the mapping file.
• modify the mapping file to fit your needs.
• for details, refer SAS XML mapping syntax.

6. Final Notes for Excel

Right click config-sdtm-3.1.2.xml then open with “Microsoft Excel”:

Option 2 will go to section 3. If go with option 1:

Option 1-1 and 1-2:   tabulation view in section 5

Option  1-3:  tag view in section 4.

Today, Jan 23, is the first day of Chinese New Year, and it is Monday, the first day of work week–it is always joyful to have such coincidenceJ. HAPPY NEW YEAR!

I had a big move this year. Actually I passed through multiply new years of 2012: I took my flight from Beijing in Jan 1 (happy new year!) and landed in Raleigh, North Carolina also in the same day due to the time difference among China and US where I take a new job as a Life Sciences Consultant at d-Wise Technologies. I’m not supposed to be a pure SAS programmer any more, but I don’t want to change the theme of my blog (“Hello World by A SAS Programmer“) since my most handy language is still SAS while my role will expose me more SAS products and solutions.

Chinese New Year is also called “Spring Festival” and “spring” is always the key word for the holiday:

(Chinese Characters for “the Beauty of Spring”. I wrote these years before in Beijing)

And I also want to spread my spring greetings with few sentences from Walt Whitman, These I Singing in Spring:

THESE, I, singing in spring, collect for lovers,

. . .

Collecting, I traverse the garden, the world—but soon

I pass the gates,

Now along the pond-side—now wading in a little, fearing not the wet,

Now by the post-and-rail fences, where the old stones

thrown there, pick’d from the fields, have accumulated,

(Wild-flowers and vines and weeds come up through

the stones, and partly cover them—Beyond these I pass,)

Far, far in the forest, before I think where I go,

Solitary, smelling the earthy smell, stopping now and then in the silence,

Alone I had thought—yet soon a troop gathers around me,

Some walk by my side, and some behind, and some embrace my arms or neck,

They, the spirits of dear friends, dead or alive—thicker

they come, a great crowd, and I in the middle,

Collecting, dispensing, singing in spring, there I wander with them,

Plucking something for tokens—tossing toward whoever is near me;

. . .

F3: run SAS codes (in batch mode)
F4: close other two windows (the current active window is Log window after F3 running; F4 jump to SAS file with full window)

F5: jump to SAS file
F6: jump to Log file
F7: jump to lst file (list output)
F8: keep only the current window (full window)

****************

Details and Credits

1. The first post on Vim and SAS I read is by Xiaowei Wang in Chinese.

The original SAS syntax file took from Zhenhuan Hu.

Kent Nassen also maintains some Vim functions to run SAS codes and check log.

2. To run SAS codes using F3:

map <F3> :w<CR>:!SAS % -CONFIG “C:Program FilesSASSASFoundation9.2nlsenSASV9.CFG“<CR>:sp  %<.lst<CR>:sp  %<.log<CR>

3. Close other windows using F4:

map <F4> :close<CR>:close<CR>

4. Keep only current window using F8:

map <F8> : only<CR>

5. Jump to SAS file using F5:

map <F5> :e %<.sas<CR>

6. Jump to Log file using F6:

map <F6> :e %<.log<CR>

7. Jump to Lst file using F7:

map <F7> :e %<.lst<CR>

Followings are my personal collections of SAS macro Repositories (I will keep it update according to new sites reached and your inputs). Most of them are rich, well documented and friendly for navigation and review:

/***General***/

1. SAS Macros by Richard DeVenezia

http://www.devenezia.com/downloads/sas/macros/index.php

Richard is a very active contributor in SAS-L. He also plays with Java, Perl, PHP and JavaScript and you can find all these codes in his homepage. Besides the well organized macros, there are some interesting utilities:

http://www.devenezia.com/downloads/sas/samples/

2. Roland‘s SAS Macros

http://www.datasavantconsulting.com/roland/Spectre/maclist2.html

Roland, a proficient SAS programmer from Europe, also supply two SAS applications:

Spectre – a Practical and Educational Clinical Trials Reporting Engine
http://www.datasavantconsulting.com/roland/Spectre/index.html

RGPP -Graphical Patient Profiler
http://www.datasavantconsulting.com/roland/RGPP/rgpp.html

And some tips:
http://www.datasavantconsulting.com/roland/sastips.html

3. Chris‘s SAS Macros

http://sas.cswenson.com/downloads/macros

I just found Chris’ site weeks before and Chris is a pretty cool programmer: in one of his fun macros, an error message will be such kind of form:

ERROR: MWA HA HA! You fool! You are cursed with leprosy!

4. sconsig SAS Coding Tips and Techniques 

http://www.sconsig.com/sastip.htm

rich while badly for navigating and review

5. Arnold Schick’s macros

http://schick.tripod.com/macros.html

also some macros collected from SAS-L :

http://schick.tripod.com/p-index.html

6. Rodney A. Sparapani ‘s Macro

http://www.mcw.edu/PCOR/Education/SASMacros.htm

Rodney is best known for his contribution to SAS support in ESS (Emacs Speaks Statistics) as a cool programmer and WinBugs and Bayesian then he works as a statistician.

Rodney’s site also contains lots of statistical stuff.

/***Statistics***/

1. Mayo Clinic Locally Written SAS Macros
http://cancercenter.mayo.edu/mayo/research/biostat/sasmacros.cfm

or in

http://mayoresearch.mayo.edu/biostat/sasmacros.cfm

2. Paul D. Allison
http://www.ssc.upenn.edu/~allison/#Macros
new site: http://www.pauldallison.com/Download3.html

Paul is a prolific writer with books on SAS and statistics.

3. MCHP SAS Macros

http://mchp-appserv.cpe.umanitoba.ca/viewConcept.php?conceptID=1048

4. Ralph O’Brien ,UnifyPow: A SAS Module for Sample-Size Analysis
http://www.bio.ri.ccf.org/Power/

5. Usual Dietary Intakes: SAS Macros for the NCI Method
http://riskfactor.cancer.gov/diet/usualintakes/macros.html

6. Clinician’s corner: SAS macros
http://www.medicine.mcgill.ca/epidemiology/Joseph/PBelisle/sas-macros.html

/***Graph***/

1. Robert Allison’s SAS/Graph Examples!
http://robslink.com/SAS/Home.htm

My first stop for SAS graphics.

2. SAS Graphic Programs and Macros by Michael Friendly
http://www.datavis.ca/sasmac/

more popular in academia.

WPS uses a lite version of Eclipse as GUI(WPS Workbench; the “lite” means WPS Workbench can’t be extensible as the original Eclipse but really with shorter response time). Besides its Project Explore for folders and files management(left panel), I also love its Outline in right panel to show the SAS programming elements and errors information in log window:

Then I’d like to switch to SAS itself. Frankly speaking, at least in IDE part, WPS looks pretty better than the current corresponding SAS:

Of course I hold the principle of “substance over form”, but if available, the form itself also make people comfortable and enjoyable (for example, the Apple products…). As far as I know, the new version of SAS DI Studio and Enterprise Miner both have pretty much improvement in GUI from ergonomic point of view. Even for code editor, Enterprise Guide Editor  is now more superior than so called SAS Enhanced Editor. But as a SAS programmer (not only the SAS user), I may spend all my day in the Base SAS window!

I also spent some time to configure VIM as a relative simple SAS IDE as a temporally replacement (F3 to run, F5 to jump to program window, F6 to log window, F7 to output window just as the same as in SAS IDE):

It’s simple but always can do the job as SAS itself while looks really cool to comfort myself as a programmer. So, what’s going on in the next release? Still wait.

#count number of lines of
#single file
def lineCount(fileName):
countSingle=0
for line in open(fileName):
countSingle += 1
return countSingle

#count number of lines of
#directory and subdirectories
def dirCount(dir,extension):
countTotal=0
for r,d,f in os.walk(dir):
for files in f:
if files.endswith(extension):
fileName=os.path.join(r,files)
countSingle=lineCount(fileName)
countTotal += countSingle
return countTotal

a=dirCount(“C:/Program Files/CDISC Express/”,”.sas”)

print a
[/cc]

I use python-2.7.2, the final Python 2.x release most because of the various modules support for learning purpose. The book helps me to get the quick review of Python is Think Python: How to Think Like a Computer Scientist by Allen Downey.

Also, I begin to use CodeColorer for this blog to insert codes.

draco dormiens nunquam titillandus

You can get the meaning using Google search of course—but not in Google Translator (Google Translator can’t recognize all of such Latin words!). Jian posted a concise Perl way to parse webs which contain this Latin phrase and key words “mean”,  “means” and such and you can always find page like

“draco dormiens nunquam titillandus,” which means “never tickle a sleeping dragon.”

My SAS approach can’t return a human readable sentence like this one but a 100% word to word machine translation and you can use it to translate any Latin sentence which happens not appear in any singe web page. The usage is also very simple:

filename L2E url ‘http://jiangtanghu.com/docs/en/Latin2Eng.sas’;
%include L2E;

%Latin2Eng(draco dormiens nunquam titillandus)

and you get:

Obs   draco         dormiens                               nunquam                         titillandus

1    dragon    sleep, rest                 at no time, never            tickle, titillate, provoke
2     snake     be/fall asleep          not in any circumstances     stimulate sensually
3                   behave as if asleep
4                   be idle, do nothing

and also (2*4*2*2=) 32 Cartesian combinations to feel the meaning if needed.

Then you can also test the words by Julius Caesar:

%Latin2Eng(Veni Vidi Vici)

and get:

Obs   Veni       Vidi                                                                     Vici

1     come    see, look at                                                     conquer, defeat, excel
2                 consider                                                           outlast
3                (PASS) seem, seem good, appear, be seen      succeed

This SAS translator is based on WORDS (version 1.97FC) by William Whitaker and the codes still needs some modifications when any unexpected special symbols popped up in the translating page.

MapReduce is a programming framework and really language independent, so SAS users can also get the basic idea from their daily programming practices and here is just a simple illustration using data step array (not array in Proc FCMP or matrix in IML). Data step array in SAS is fundamentally not a data structure but a convenient way of processing group of variables, but it can also be used to play some list operations like in Python and other rich data structure supporting languages(an editable version can be founded in here):

Follow code above, the programming task is to capitalize a string “Hadoop” (Line 2) and the “master” method is just to capitalize the string in buddle(Line 8): just use a master machine to processing the data.

Then we introduce the idea of “big data” that the string is too huge to one master machine, so “master method” failed. Now we distribute the task to thousands of low cost machines (workers, slaves, chunk servers,. . . in this case, the one dimensional array with size of 6, see Line 11), each machine produces parts of the job (each array element only capitalizes a single letter in sequence, see Line 12-14). Such distributing operation is called “map”. In a MapReduce system, a master machine is also needed to assign the maps and reduce.

How about “reduce”?  A “reduce” operation is also called “fold”—for example, in Line 17, the operation to combine all the separately values into a single value: combine results from multiple worker machines.

Tom Davenport reported an observation that Silicon Valley is becoming more analytical since companies in the Valley such as Google, Facebook, eBay, LinkedLn all have strong presences in analytics. Besides such predominant companies, I’d also like to add Yahoo to the list although Yahoo is no longer in its peak. Yahoo is the largest sponsor and contributor of Hadoop, an open source framework for distributed processing of so called “big data”. When taking a look at the outstanding Facebook data team or LinkedIn data team, we can see that Hadoop is also one of the most overwhelmingly successful technical factors. Such Valley companies themselves are the huge consumers of big data and have strong incentives to develop analytical solutions beyond their high technology product pipelines.

Analytical staffs in LinkedLn also helps a lot to promote the widely usage of the term “data scientist”. They identify themselves as data scientists and that’s really cool. Now more and more statisticians are also very glad to accept this brand new title. According to a survey in JSM (2011, Miami), more than 85% (164) statisticians there considered themselves “data scientists”.

McKinsey also released a report this May on big data and the huge gap of qualified analytical talents. You know when a management consulting firm begins to talk something technical, it is no longer a fashion to follow the discussion of the concept. To embrace the challenge of big data, one or the team needs multidiscipline background—basically speaking, computer science and statistics (and data mining or machine learning is just an interdisciplinary subject of them). Here is an ambitious list on “How do I become a data scientist”:

http://www.quora.com/Educational-Resources/How-do-I-become-a-data-scientist

For these learning plans, just feel the meaning and don’t take it too seriously. Check yourself and set up your own priority.

Notes for SAS Programmers

For SAS programmers, I read an exciting post besides High Performance Computing that SAS will also play with Hadoop by introducing some functionality in SAS/Access and SAS Data Integration Studio.

For SAS programmers with no IT background, it is not a good idea to jump into algorithms and data structures and other hard core computer courses immediately. Instead I recommend to take the full advantages of SAS language and system itself to dive into computer world gradually:

1. Learn and practice and practice SAS Proc SQL which is compliant with the SQL-92 standard. SQL is the common language in database world and SAS Proc SQL can help you switch smoothly to Oracle SQL, Teradata SQL, MySql SQL and other SQL implementations although there are some non-critical differences in details.

2. Dig into the operating system specific documentation of SAS, for example in SAS 9.3,  SAS 9.3 Companion for Windows or SAS 9.3 Companion for UNIX Environments or others depending the OS you are working on. They are the critical important documentations but unfortunately often missed in SAS programmers’  reading list.

Such docs will help SAS programmers to deal with the machines and expose to the wide computer world in a way that a SAS programmer can understand. You can’t expect to be an expert on computer via such docs, but at least you can communicate fluently with internal IT staff.

3. Then you get all the confidences to play with computer and can switch to any other topics interested in the list above!

In the previous post, I just mentioned that Fisher’s Iris Data is embedded officially in SASHELP library in SAS 9.2. Note that even in SAS 9.1.3, you can also find this data with several instances from some demos in user guide (just search "Iris" in "SAS Help and Documentation" accompany with you SAS product), for example, in SAS 9.1.3 IML.

Iris dataset is so important and popular that researchers round the world use it as benchmark to test and compare their algorithms and also as pedagogical purpose. It is also the overwhelming No. 1 dataset considering popularity in UCI Machine Learning Repository. Here 4 errors in SASHELP.iris listed for your consideration if interested and if you find some slightly differences in outputs following some demos out of SAS using this data:

Error 1: Line 35, the PetalWidth of Setosa should be 2 mm, not 1 mm;

Error 2: Line 38, the SepalWidth of Setosa should be 36 mm, not 31 mm;

Error 3: Line 38, the PetalLength of Setosa should be 14 mm, not 15 mm;

Error 4: Line 119, the PetalLength of Virginica should be 69 mm, not 70 mm.

For errors 1-3, there is also an interesting story in statistical literature. In 1936, Fisher the Great published his famous paper, The use of multiple measurements in taxonomic problems and the Iris data also attached (called Fisher Version in this post). In the following years (until today), people cited this paper and the Iris data Fisher Version is also replicated and distributed worldwide and then a version with above errors 1-3 might gain a very dominant popularity (I don’t know the source of there errors). In UCI Machine Learning Repository, the dataset iris.data is the one with such 3 errors (called UCI Version as well).

We could see that the duplicated UCI Version is even more popular in some extension than its original Fisher Version (SASHELP.iris also seems to be copied from UCI Version). Story goes on. In 1998, James Bezdek and other scholars just found the three discrepancies between Iris Fisher Version and UCI Version (and in some published papers using the same version of data). You can read it in Will the Real Iris Data Please Stand Up?

Bezdek then proposed to use the original Fisher Version of Iris, and UCI Machine Learning Repository also documented these three errors and added new dataset called bezdekIris.data (Bezdek Version) which is exactly Fisher Version (iris.data kept and I think it is because now the so called error version is also valuable).

Return to error 4 and I can’t figure out why and I might as well call it Iris SAS Version. Note that the unit in SAS Version is millimeter (mm), while others version all use centimeter (cm).

The interesting part is that I also check the Iris data in SAS 9.1.3 IML mentioned before and not surprising, it is exactly the Fisher Version (you can also find a right one in a demo from SAS 9.2 IML Studio 3.2).

The following codes generate several Iris versions:

iris_uci: UCI Version with both CM and MM as unit

bezdekiris_uci: Bezdek Version or Fisher Version with both CM and MM as unit

iris_mm: UCI Version with MM as unit and attributes alike SASHELP.iris, SAS Version

bezdekiris_mm: Bezdek Version or Fisher Version with MM as unit and attributes alike SASHELP.iris, SAS Version

filename iris URL "http://archive.ics.uci.edu/ml/machine-learning-databases/iris/";

%macro getIris(input);
data &input._uci;
    length _Species $15.;
    length  Species $10.;
    infile iris(&input..data) dlm=’,';
    input _SepalLength _SepalWidth _PetalLength _PetalWidth _Species $;

    Species=propcase(scan(_Species,2));

    array iris {*}  _SepalLength _SepalWidth _PetalLength _PetalWidth;
    array iris_mm{4} SepalLength  SepalWidth  PetalLength  PetalWidth;

    do i=1 to dim(iris);
        iris_mm{i}=iris{i}*10;
    end;

    drop i;

    label   _sepallength=’Sepal Length (cm)’
            _sepalwidth =’Sepal Width (cm)’
            _petallength=’Petal Length (cm)’
            _petalwidth =’Petal Width (cm)’
            _Species =’Iris Species’;

    label   sepallength=’Sepal Length (mm)’
            sepalwidth =’Sepal Width (mm)’
            petallength=’Petal Length (mm)’
            petalwidth =’Petal Width (mm)’
            Species =’Iris Species’;
run;

data &input._mm (label="Fisher’s Iris Data (1936)");
    set &input._uci;
    drop _:;
run;

%mend getIris;

%getIris(iris)
%getIris(bezdekIris)

Tell me something about Alfred, male or female? age? height and weight?

Oracle database (version 9 and below) had a well known default demo account SCOTT with a password, TIGER (and TIGER was the name of the real person Bruce Scott ’s cat, see) and in this account, there are some tables named DEPT, EMP, BONUS and SALGRADE (you can read their meaning). Almost every Oracle DBA learn SQL using these database and an joke just says that in DBA’s meetings, people just  warm up saying “how about Smith?” And you should know that in the database, Smith is a clerk and his boss is Ford (whose boss is Jones)!

In the beginning I also raise a question for SAS programmers: who is Alfred? Don’t give quick answer such that “Alfred who”. Actually, you should already go through with Alfred very well as a SAS programmer:

proc print data=sashelp.class;
    where name="Alfred";
run;

As a clinical SAS programmer, I play with data, get acquaintance with the data and subjects and then subjects are no longer “subject”. They have identities and  Alfred is a 14 years old boy. I have such habit mostly because in clinical world, data are very expensive (not like the massive transaction data in financial industry) and should be took more care.

I dare say that “class” is the most famous SAS dataset in sashelp library and then in the SAS world. The first dataset used for demo is almost this “class”. I just did a quick Google search, “sas sashelp.class” returns about 44,400 results. Hope you can find any other SAS datasets to beat it.

Alfred in “class” pops into my mind because today, I do find a strong candidate. In SAS 9.2 (and 9.3), the sashelp library has a new member, Iris. YES, it is the “Fisher Iris Flower Data”, which can be safely considered the most famous and most  used dataset in machine learning and data mining papers and statistical applications. Currently it has only 859 hits in Google, I think the number will reach high accompany with the wide use of SAS 9.2 and above, and to enforce my prediction, I will definitely play with the Iris data in the following future!

You Could be a SAS® Nerd If . . .

Here I’m glad to find that I am roughly a 20% SAS nerd (12 matched in all 57 lists):

8. You blog SAS-related comments and technical solutions frequently.

9. You have more than five SAS blogs in your RSS feed.

10. Your home page is support.sas.com, sasCommunity.org, SAS-L, or LexJansen.com.

11. You know more than ten SAS keyboard shortcuts.

12. You get excited when you find a new match-merge technique that performs better than the one you developed the week before.

21. You have more than one version of SAS on your machine or network so you can compare and contrast program, processing and output differences.

28. You spend your Friday evenings and weekends responding to SAS-L posts, entering sasCommunity blog entries, and reading the latest “hot” SAS topic on LexJansen.com.

38. The first thing you read in the morning is the “Tip of the Day”.

45. You subscribe to five or more SAS groups on LinkedIn, sasCommunity, and Facebook and you use a tabbed browser so you can be online with all of them at the same time.

47. You spend your evenings and weekends SAS-L’ing, Googling and Binging looking for elegant SAS technical solutions.

50. You proudly proclaim that you’re a SAS programmer when asked by a fellow passenger, “What do you do for a living?”

51. You’re amazed when your fellow airline passenger replies, “What is a “SAS programmer?”.

I also asked Kirk if he is a 100% SAS nerd and Kirk replied, NO. He said he is a 99% SAS nerd:)

As a learner, I benefit frequently from lots of high quality blogs. But I’m also slightly lazy as a reader: I read almost all blogs in Google Reader rather than returning to the original web pages and adding some comments. Here I’d like to post a series of SAS bloggers and their blogs (that I subscribed and read frequently) to extent my thanks, share with friends and sometimes amuse the bloggers themselves:)

In this page, honor belongs to Jian Dai and his Yahoo group, “SAS Academy”:

http://tech.groups.yahoo.com/group/sas_academy/

Wait. You may ask: a Yahoo group?

Jian actually is a blogger for a collective blog, Clinical Trial New Technologies & Best Practices. But here I’d like to refer this Yahoo group which actually serves as his own blog and I also use Google Reader to subscribe its RSS feed:

launched at December of 2008

444 posts till July 2011, averagely 14 posts per month (and roughly 1 post for every 2 days!)

And all in all, most (and almost every) posts are under Jian’s name and few posts come from other’s hands are just outlier-like(less than 10). Note that I just read all the historical messages in a weekend and got the impressions. In the following page, I might retrieve all the data from the SAS Academy and supply accurate reports.

Recently Jian posted a C implementation of a SAS tokenizer based on Finite-state Machine(FSM). A tokenizer is a lexical analyzer of a specific programming language and in this case you can just imagine it is a small part of SAS Supervisor to recognize all tokens. In SAS Academy, Jian posted lots wonderful (and beautiful) implementation of lots of general or specific problems using C, Perl, JScript, JavaScript, PowerShell, VBS/VBA, . . . and even ProLog, a general logic programming language!

Wait wait. You may wonder: is Jian a SAS programmer?

I don’t know how Jian identify himself as a programmer and it would be best to leave this question to himself. But don’t worry. There is no such phrase engraved at the door of Jian’s SAS Academy:

Let no one ignorant of C/Perl enter.

Actually Jian is a SAS/statistical/clinical programmer by job and he of course has lots of interesting SAS codes posted, including SAS implementation of Lambda Expression and recursion(Jian is also an active conference speaker on this topic). –Some clinical programmers without any computer background would find it difficult to understand the CS part even of Jian’s SAS codes, for example, Lambda Expression. Suggestion: Google it and you will enter a totally new joy world.

Jian plays like a geek, and he is always making efforts to extend the horizon and scope of SAS programming language by taking advantage from computer science(actually he had some posts titled play like a CS pro). I have great experience by exploring Jian’s SAS Academy and very often, a popular book title pops up in my mind:

Programming Pearls

If you want sometimes programming for fun, just for fun, that’s it.

First step: define the scope

For my purpose, I only need the titles and publish dates of Rick’s posts. It is so called the “metadata” of the blog. I do not need all the post contents. By the way, if all information needed, you can use a blog backup tool, or write codes to retrieve all the pages of http://blogs.sas.com/iml at the maximum depth, or simply, you can write to Rick and say: hey Rick, could you please send me all the contents of your blog? And Rick may go to the management console of his own blog, export all the contents to an XML file and get back to you.

Second step: analyze the web pages

Browse to the right panel of Rick’s blog, in the ARCHIVES frame, click “Older”:

And you get

This page just gives a big picture of Rick’s blog (ARCHIVE section is always a good place to get such metadata, for example, archives for my blog). But we need more. Click “view topics” for example of September, 2010:

This page is exactly what we want with titles and dates. Open an editor to write codes immediately to read all the information in this page?—wait. Currently this blog has posts across 11 months and you can expect the increase. You should design a dynamic method to read all the topics pages: Sep 2010, Oct 2010, … and, today().

Return to the archives page. RCM (right click your mouse) and select “View page source” if you use Google Chrome web browser (“View Source” in IE; “View Page Source” in Firefox) and you get all the HTML scripts (Note: you DO not need any knowledge of HTML to understand this post). Copy and paste them into a text editor supporting HTML syntax highlighting (such as Notepad++). Search all instances of “view topics” we mentioned before:

We are lucky. They are 11 instances of “view topics” accompanying with 11 hyperlinks for the currently 11 months’ archives of Rick’s blog. We can read such 11 hyperlinks to a macro array for dynamic retrieval.

Then we return to the single topics page, for example of September, 2010. Review the HTML source file. Search for “posted_by_date” and we get 14 instances which is same as the number of posts in September 2010:

We should also need to locate all the instances of titles. Search “/iml/index.php?/archives/” and we get 17 responses:

We see 3 instances at end of the finding results don’t contain any titles. You can check other pages to confirm such pattern. Yes we can use regular expressions to parse the HTML pages to locate more exactly for the titles. But for a quick job and due to the relative simple HTML pages, some basic SAS character functions are enough for our purpose. In the following codes, limited regular expressions are used only to remove HTML tags such as “<a href=”.

After such explorative search of HTML scripts, we can get the basic idea where can we find the interested information. Then we begin to coding work.

Third step: Coding at last!

For our purpose, we should first read the archive page to get all the topics links to a macro array, then read the all the topics pages dynamically. Finally, we should also add the all the calendar dates with holidays. Some friends may find that they met piece of the following codes before. Yes, such codes just assembled some skills what I learned from Art Carpenter, Richard DeVenezia, Jian Dai and lots of programmers before!

3-1: read archive page

%let URL=http://blogs.sas.com/iml/index.php?/archive;

filename archive URL "&URL";

data archive;
    length text $1024;
    infile archive lrecl=1024;
    input text $;
    text= _infile_;
    if index(text, ">view topics<") then output;
run;

data  archive1;
    set archive;
    summary=scan(text,4,’"’);
run;

3-2: read all topics pages

data _null_;
    set  archive1 end=eof;
    I+1;
    II=left(put(I,2.));
    call symputx(‘summary’||II,summary);
    if eof then call symputx(‘total’,II);
run;

%macro readit;
    %do i=1 %to &total;
        filename f&i URL "&&summary&i";   

        data f&i;
            length text $1024;
            infile f&i lrecl=1024;
            input text $;
            text= _infile_;
            if index(text, "/iml/index.php?/archives/") or index(text, "posted_by_date") then output;
        run;

    /*remove HTML tags;*/
    data ff&i;
        set f&i; 
        prx=prxparse("s/<.*?>//");
          call prxchange(prx,99,text);
        drop prx;

        flag=ifn(mod(_n_,2),1,2);
        grpn=&i;
        if index(text,"201") and length(text)<10 then delete;/*be carefull! hard coding;*/
        seq=_n_;
     run;

     /*transpose data;*/
     data fff&i;
         set ff&i;
        by grpn seq flag;

        retain title;
        if first.flag then title=lag(text);
        if flag=1 then delete;
     run;

    %end;

%mend readit;
%readit

%macro getall;
    data Rick;
        set %do i=1 %to &total; fff&i %end; ;
        seq=seq/2;   
        drop flag;
    run;   
%mend getall;
%getall

data rick2;
    set rick;

    datetime=scan(text,2,",");     
    year=scan(text,2,".");       
    month=scan(scan(text,2,","),1);
    day=scan(scan(text,2,","),2);     
    week=scan(text,6);            
    dt=compress(catx("",day,substr(month,1,3),year));  
    worddat=input(dt,date9.);                        
    format worddat  ddmmyy10.;                        

    m=scan(put(worddat,ddmmyy10.),2);             
    my=compress(catx("",year,m));                     
run;

proc sort ;
    by  worddat descending seq ;
run;

It is also interesting to add additional information for further analysis, such as all calendar dates during Rick’s blogging history and holidays.

3-3: get all calendar dates

proc sort data=rick2 out=rick3 nodupkey;
    by my;
run;

data _null_;
    set rick3 end=eof;
    I+1;
    II=left(put(I,2.));
    call symputx(‘year’||II,year);
    call symputx(‘month’||II,m);
    if eof then call symputx(‘total’,II);
run;

%macro getCalendar;
    %do i=1 %to &total;
        data calendar&i;
            date1 = mdy (&&month&i,1,&&year&i);
            date2 = intnx (‘month’, date1, 1) – 1;

            do worddat = date1 to date2;
                  wim = intck (‘week’, date1, worddat);
                  dim = worddat-date1+1;               
                  output;
            end;

            format worddat  ddmmyy10.;           
            keep   worddat  dim ;
      run;
    %end;
%mend getCalendar;
%getCalendar;

%macro allCalendar;
    data Calendar;
        set %do i=1 %to &total; calendar&i %end; ;   
    run;   
%mend allCalendar;
%allCalendar

3-4: get holidays

I have no specific idea about US holidays and just referenced the two pages:

http://support.sas.com/kb/24/655.html 

http://www.opm.gov/Operating_Status_Schedules/fedhol/2011.asp

Also, the holidays and observances should manully modified according to personal working schedule. So this section serves only for demonstration.

filename hld url "http://jiangtanghu.com/docs/en/US_holiday.sas";

%include hld;

%US_holiday(2010)
%US_holiday(2011)

data holiday;
    set holiday2010 holiday2011;
run;

proc sort ;
    by  worddat;
run;

3-5: put all together

data rick_all;
    merge rick2 calendar holiday;
    by worddat; 
    if worddat <’03Sep2010′d then delete;    
    if worddat >’15Jul2011′d then delete;
    drop today;
run;

A pooled version (beta) also available at

http://jiangtanghu.com/docs/en/SAS_Blogs.sas

non-fiction on  rose,

fiction on blue,

poetry on yellow

The SAS blog writer, author of Statistical Programming with SAS/IML Software, Rick Wicklin of SAS Institute,  also leads a strong “color revolution” in SAS blog community:

In an interesting personal statement, Blogging, Programming, and Johari Windows, Rick summarizes his rich and colorful blogging rhythms according to the above Johari window:  

• Mondays, writes introductory notes (corresponding to the upper right quadrant of Johari window). 
• Wednesdays, experimental articles on sampling, simulation and other statistical programming topics(lower left quadrant).
• Fridays, on explorative analysis of data (upper left quadrant).

So what about the lower right quadrant? Rick rediscovers and exposes what he once knew. Just suppose that, Rick picks up some codes he wrote before (ten years ago maybe) with big surprise: oh, who on earth wrote such damned clever beautiful codes? He or she must be in his/her aggressive youth. –then Rick wrote them all in blog.

Here I produced a summary table for Rick’s blogging activities (numbers per month per weekday; before July 16, 2011 Beijing time; next following post would introduce how to use SAS to analyze data from website such as Rick’s blog):

Key findings:

• Rick is really a frequent and productive blogger with averagely 0.5 posts per day!
• Rick DOES keep his words. Most of the posts are published in Friday, Wednesday and Monday(44, 44, 42 posts respectively).
• None posted in Saturdays and Sunday.

Rick began his writing since September  3. 2010, Friday.  Up to July 15. 2011, Friday, there are 48 Fridays, 46 Wednesdays and 46 Mondays.  Only 10 colored weekdays (4 Fridays, 2 Wednesdays and 4 Mondays) passed with no posts and most all them are due to national holidays:

06/09/2010 , Monday       : Labor Day
24/11/2010 , Wednesday: round Thanksgiving Day
26/11/2010 , Friday         : round Thanksgiving Day
22/12/2010 , Wednesday
24/12/2010 , Friday         : Christmas Day
27/12/2010 , Monday
31/12/2010 , Friday         : New Year’s Day
30/05/2011 , Monday      : Memorial Day
10/06/2011 , Friday
04/07/2011 , Monday      : Independence Day

At least in 4 holidays (most in Monday), Rick was also active in writing:

11/10/2011, Monday, Columbus Day: How Do You Reshape a Matrix?
11/11/2010, Thursday, Veterans Day: It’s Here!
17/01/2011, Monday, Birthday of Martin Luther King, Jr.: On the Flip Side: Exchanging Rows and Columns
21/02/2011, Monday, Washington’s Birthday: How to Build a Vector from Expressions

Amazing Rick keeps a fixed writing pattern and in next following post, detailed analysis and SAS codes will be presented so you can also keep eyes on the metadata of your favorite bloggers’ writing and may rise a question like:

Hey Rick, what’s up in Jun 10, 2011, Friday?

My finger hurts,

I can not dance

Tango in Bastille.

So… any ideas?

…

… …

… … …

OK. I am a terrible Haiku writer. The sentences are not self explained without  /*commentaries*/ . So the story behind the scene…

One day my remote workstation was very very slow due to some network issues. But I still needed to write and run SAS codes in the server which is located in France. It really hurt my fingers from the ergonomic point of view.

“Tango” was my project code. But why Bastille? oh, that day happened to be round the Bastille Day in France. The performance of the remote server that time was just as terrible as the Bastille I want to break so I could do my project work more smoothly!

—————————–

Note that after my completing this Tango Haiku, I did a Google search and found that there is really a French book called Bastille tango!

• Two-sided confidence intervals for the single proportion: comparison of seven methods.
  Newcombe RG, Stat Med , Volume 17 , 8 (April 1998) pp.857-872
• Interval estimation for the difference between independent proportions: comparison of eleven methods.
  Newcombe RG, Stat Med , Volume 17 , 8 (April 1998) pp.873-890

This post serves as an implementation using SAS accompany with the first paper on confidence intervals for single proportion(method 1-7). Additional 4 methods also provided:

1.  Simple asymptotic, Without CC                       
2.  Simple asymptotic, With CC                          
3.  Score method, Without CC                            
4.  Score method, With CC                               
5.  Binomial-based, ‘Exact’                             
6.  Binomial-based, Mid-p                               
7.  Likelihood-based                                    
8.  Jeffreys                                            
9.  Agresti-Coull,z^2/2 successes                       
10. Agresti-Coull,2 successes and 2 fail                
11. Logit

The codes are available at

http://jiangtanghu.com/docs/en/CI_Single_Proportion.sas 

It is a purely SAS/Base(data step and SQL) approach. I prefer data steps because it can be seamlessly incorporate into production work which looks like:

/*r is the observed number of events/responders in n observations;*/

%macro _Wald(r,n,alpha=0.05);

      p=&r/&n;
      z = probit (1-&alpha/2);
      sd=sqrt(&n*p*(1-p)); *Standard Deviation ;
      se=sd/&n;            *standard error;
      me=z*se;             *margin of error;
      p_CI_low = p-me;
      p_CI_up  = p+me;
     p_CI=compress(catx("","[",put(round(p_CI_low,0.0001),6.4),",",put(round(p_CI_up,0.0001),6.4),"]"));
%mend _Wald;

data test;
      input r n;
     %_Wald(r,n);
      keep r n p p_CI;
datalines;
81 263
15 148
0 20
1 29
29 29
;  
run;

proc print data=test;run;

But when simulation required (in methods 6-7), some SQL clauses pop up. Admit that It makes the codes less elegant. In this situation, matrix operation language such as SAS/IML or R will do a better job.

Note that in SAS 9.2, PROC FREQ can produce the following five intervals(method 1, 3, 9 ,8, 5):

Wald (also with method 2)                  
Wilson                
Agresti-Coull         
Jeffreys              
Clopper-Pearson (Exact)

For the five intervals in SAS 9.2, a good reference is Confidence Intervals for the Binomial Proportion with Zero Frequency by Xiaomin He and Shwu-Jen Wu:

www.pharmasug.org/download/papers/SP10.pdf

Also, an equivalent R version of the eleven methods by Abteilung Kriminologie is also available at

http://www2.jura.uni-hamburg.de/instkrim/kriminologie/Mitarbeiter/Enzmann/Software/prop.CI.r

Such R repository is also a good resource for SAS users. I almost “translate” R implementations directly to SAS for this post.

Dive into CDISC Express (1): Introductory

Dive into CDISC Express (2): Create a New Study

Dive into CDISC Express (3): Navigate mapping file

Dive into CDISC Express (4): Data manipulation techniques

A more friendly PDF version of these all CDISC Express series is also available in

http://jiangtanghu.com/docs/en/CDISCExpress.pdf

The following tasks, such as generating SDTM domains and define.xml, need just some clicking button work in CDISC Express using a well designed mapping file. Few words needed due to the software.

5. Step 3 of 6: Validate mapping file (Validate_Mapping_File.sas)

It would be back and forth to design, validate then modify and re-validate the mapping file. And sure finally, you will get all the work done, at least no syntax error (how to avoid semantic errors is upon your domain knowledge). A validated mapping file, named mapping.xls will be copied to … docMapping file – validated version from the working file, tmpmaping.xls. You will see

The corresponding log file in folder … log

A report in …resultsMapping Validation, named Mapping_validation.html

Also the temporary datasets in …tempdata and …temp:

6. Step 4 of 6: Generate SDTM datasets (generate_SDTM.sas)

If mapping file is OK, generating SDTM domains is just clicking the button. After submitting the codes, you will see the log file, reports, SDTM datasets and temporary datasets in corresponding folders:

7. Step 5 of 6: Validate SDTM datasets (Validate_SDTM_Domains.sas)

The outputs files of validating SDTM datasets are all located in C:Program FilesCDISC ExpressSDTM Validation:

8. Step 6 of 6: Generate Define.xml and xpt (generate_Definexml.sas)

Get the final define.xml file and SAS transport files (.xpt):

9. Recommended reading and action taken

For a quick start and deep understanding, you could read the official documentations in the following sequence:

C:Program FilesCDISC ExpressdocumentationFAQ.htm

C:Program FilesCDISC ExpressdocumentationQuick Start.htm

C:Program FilesCDISC ExpressdocumentationUser guide.htm

A video tutorial would be also helpful:

C:Program FilesCDISC Expressdocumentationvideotutorial.htm

A must-read conference paper, An Excel Framework to Convert Clinical Data to CDISC SDTM Leveraging SAS Technology by Sophie McCallum and Stephen Chan of Clinovo, supplies a wonderful discussion the architectures of CDISC Express:

http://www.lexjansen.com/pharmasug/2011/ad/pharmasug-2011-ad08.pdf

Of course, you do not need to review all the pages then get the confidence to use the software. Learn by doing and just dive into it. There is an opportunity for your practicing and you will also have a change to win an iPad2 from Clinovo’s CDISC Express Contest:

http://www.clinovo.com/cdisc/game

The due day is July 15^th and I already submitted my work. That’s fun.

Dive into CDISC Express (1): Introductory

Dive into CDISC Express (2): Create a New Study

Dive into CDISC Express (3): Navigate mapping file

4.3 Data manipulation techniques in CDISC Express

CDISC Express supplies relative rich sets of data manipulation techniques assembling with SAS languages used for data mapping. Following is a not limited listing and I will keep it updated.

4.3.1 Reference one dataset

A raw dataset name appear in “Dataset” column indicate a “set” operation in SAS.

All dataset options can be used when referencing a dataset, such as

siteinv(drop=invcode)

siteinv(rename=(invcode=inv))

siteinv(where=(invcode ne “”))

You can also reference an external dataset. You should incorporate the external file in spreadsheet with name beginning with an underscore, “_”, and “_visits” in this case:

Then you can use it in any domains needed, e.g., TV domain:

There is a macro %cpd_importlist used to import the external dataset, “_visits”. Again, this macro roots in C:Program FilesCDISC Expressmacrosfunction_library.

Using a macro call to re-sharp or modify an input dataset offers great flexibility referencing data. We will also discuss the benefits later on.

4.3.2 Assignment

You can assign a number, string and a dataset variable with any valid SAS functions to a SDTM domain variable in “Expression” column.

Sometimes a temporary variable needed for later calculation. You can produce such temporary variable in “Dataset” column with an assignment in the “Expression” column just similar with any other domain variables. Two differences: first, such temporary variables named begin with an asterisk, “*”; second, all temporary variables will not be included in the final domain. Once created, such temporary variables can be used for any other expressions.

There are three special symbols used in “Dataset” column of CDISC Express. Asterisk, “*” indicates a temporary variable, while other two are

Tilde, “~” : indicate a variable used for supplemental domain (SUPPQUAL).

Number sign, “#”: indicate a variable used for comments domain (CO).

Another symbol, at sign, “@”, used in “Expression” column, indicated referencing a variables produced before:

In this case, “AGEU” uses “AGE” as input, while “AGE” is calculated before. “@AGE” just indicates the dependency. In concept, it looks like the “calculated” option in SAS PROC SQL:

proc sql ;

select (AvgHigh – 32) * 5/9 as HighC ,

(AvgLow – 32) * 5/9 as LowC ,

(calculated HighC – calculated LowC)

as Range

from temps;

quit;

4.3.3 Match-merging

We already got a math-merging example before. If “all” appears as a dataset in the “Dataset” column, all the previous datasets should be merged first for later processing by the common key specified in “Merge Key” column. If no key assigned, patient ID is used by the system.

CDISC Express also supports two types of join, inner join and outer join (left, right, full) using data steps. The implementation has slightly difference with standard SQL, but the ideas are same.

We add a new column, “Join”, usually beside the “Merge Key” column.

There are two values for “Join”, “O” or “I” while “O” stands for “outer join” and “I”, “inner join”. A join indicator “I” equals a dataset option “in=” in action while “O” means no. Use the above as illustration, the corresponding SAS codes behind look like

data temp;

merge demog(in=a) siteinv(in=b);

by sitecode;

if b;

run;

This is so called “right outer join”. The combination of “I” and “O” in these two datasets can perform all the four types of join, one inner join and three outer join:

As we could see, if no “Join” column specified, CDISC Express will perform inner join by default.

So far CDISC Express cannot support multiply merge keys. For example, the following file is illegal currently:

Dataset	Merge Key
arm	siteid, grpno
armdescri	siteid, grpno

The developer Romain indicated that such enhancements would be raised to the next round of product road map and he also proposed a work around. To use multiple keys for merging, we can create a temporary variable holding such multiple keys as a concatenation then this temporary variable can be used as a single merging key.

4.3.4 Concatenating

Above we discussed lots about “merge” operation in CDISC Express. This section dedicated for “set” operation. We already know how to “set” one dataset for referencing, but how to “set” multiple datasets, i.e, “Concatenating”?

Symmetrically, an “all” appears in “Dataset” column indicating merging operation, an “all (stack)” indicates concatenating operation:

The above file can be also translated to SAS codes for better understanding:

data height;

set vtsigns(where=(height ne .));

VSTESTCD="HEIGHT";

VSTEST ="Height";

VSORRES =put(height,best12.);

VSORRESU="cm";

VSSTRESC=put(height,best12.);

VSSTRESN=height;

VSSTRESU="cm";

run;

data weight;

set vtsigns(where=(weight ne .));

VSTESTCD="WEIGHT";

VSTEST ="Weight";

VSORRES =put(weight,best12.);

VSORRESU="kg";

VSSTRESC=put(weight,best12.);

VSSTRESN=weight;

VSSTRESU="cm";

run;

data vs;

set height weight;

STUDYID =study;

DOMAIN =&domain;

USUBJID =%CONCATENATE(_variables=study sitecode patid);

VSSEQ =%SEQUENCE();

. . .

run;

4.3.5 Transpose

Clinical SAS programmers do lots of transpose operation to re-sharp the raw data to fit the CDISC standards. Currently there is no explicit guide in CDISC Express on how to transpose, but this is not the end of story.

There are two types of transpose:

Type I: from a wide dataset (more variables, less observations) to a long dataset (less variables, more observations), e.g. transposing a one-row-per-subject datasets to a multiple-row-per-subject dataset

Type II: from a long dataset (less variables, more observations) to a wide dataset (more variables, less observations), e.g. transposing a multiple-row-per-subject dataset to a one-row-per-subject datasets

As good practices, in SAS we always use data steps with “output” statement to perform type I transpose and use PROC TRANSPOSE for type II. Although CDISC Express doesn’t support transpose operation in an explicit way, at least you can perform type I transpose and surprisingly we already saw it before!

Just back to section of concatenating. The example is taken from C:Program FilesCDISC Expressstudiesexample2.

We can see the input data vtsigns is typical wide table (more variables, less observations):

And the final domain VS is a typical long table (less variables, more observations):

So obviously, such concatenating operation just did a wonderful type I transpose, from a wide table to a long table! More often, the compact SAS codes for type I transpose look like:

data vs;

set vtsigns;

if height ne . then do;

VSTESTCD="HEIGHT";

VSTEST ="Height";

VSORRES =put(height,best12.);

VSORRESU="cm";

VSSTRESC=put(height,best12.);

VSSTRESN=height;

VSSTRESU="cm";

output;

end;

if weight ne . then do;

VSTESTCD="WEIGHT";

VSTEST ="Weight";

VSORRES =put(weight,best12.);

VSORRESU="kg";

VSSTRESC=put(weight,best12.);

VSSTRESN=weight;

VSSTRESU="cm";

output;

end;

. . .

run;

4.3.6 All others: use macro!

Now we discussed almost all the common data derivation techniques in programmers’ daily life and the corresponding implementation in CDISC Express. At least we have one question unsolved: how to perform type II transpose, i.e. from a long table to a wide table?

It would be an open question for the developers of the application. But we can also solve this problem in current framework: use macro, customized macro. You can use macros in “Expression” and “Dataset” column. Macro used in “Dataset” column returns a dataset, while macro in “Expression” column returns series of string: that’s the basic structure you should consider when customize your own macros. For more, you can reference the macros in C:Program FilesCDISC Expressmacrosfunction_library. For example, &concatenate used in “Expression” column; &cpd_importlist in “Dataset” column.

So it would be convenient to create temporary datasets using macros imbedded type II transpose operation in “Dataset” column. Every thing SAS can do, you can also implement it in CDISC Express. Just use macros, in “Expression” and “Dataset” column accordingly.

The raw data varies according to trial design and clinical data capture system and procedures. It is impossible and impractical to anticipate the CDISC SDTM converter such as CDISC Express to map all the data just clicking a button. The introducing of CDISC Express doesn’t keep programmers away. It just keeps most of the trivial work away from programmers’ daily life and let them more concentrated on creative work and be productive and efficient.

Following would be the close of such pages.

Dive into CDISC Express (1): Introductory

Dive into CDISC Express (2): Create a New Study

4. Step 2 of 6: Generate mapping file

Generating template (blank) mapping file only needs pieces of effort by submitting generate_mapping_template.sas. The toughest one is to fill it with mapping rules according to specified study.

4.1 Get the blank template mapping file (generate_mapping_template.sas)

To get the blank template mapping file, just fill the one line of macro call in generate_mapping_template.sas:

%createmapping(filespec=SDTM_Specs_3_1_1.xls, Dom=CM AE TV, req=YES, perm=YES, exp=YES);

Also, you can specify SDTM implementation version, 3.1.1 or 3.1.2. For domains (&Dom), DM, CO and SUPPQUAL will be created automatically; you should list others accordingly:

SDTM 3.1.1: SV CM EX AE DS MH DV EG IE LB PE QS SC VS TV TI XD SU XR XS XE TR (Total: 22)

SDTM 3.1.2: AE CE CM DA DS DV EG EX FA IE LB MB MH MS PC PE PP QS SC SE SU SV TA TE TI TS TV VS (Total: 28)

You should also choose the “CORE” variable (REQUIRED, PERMISSIBLE and EXPECTED) by triggering &req, &perm, and &exp to “YES” or “NO”. Note that

REQUIRED and EXPECTED variables must always be included (req=YES, exp=YES);

PERMISSIBLE variables included if needed (perm=YES or perm=NO)

Submit generate_mapping_template.sas and you can get a blank template mapping file tmpmapping.xls in C:Program FilesCDISC Expresstemp.

Copy it to C:Program FilesCDISC ExpressstudiesCLINCAPdocMapping file – working version for example used for study “CLINCAP” and then fill all the blank columns (it NEEDS efforts!).

If this mapping file passes the validation process, a final version named mapping.xls will be copied automatically to C:Program FilesCDISC ExpressstudiesCLINCAPdocMapping file – validated version for later processing.

Note that if you already have some validated mapping file for other studies, it would serve as a good start rather than using the blank template from the scratch.

4.2 Navigate mapping file

Let’s take a look at the “real” worked mapping file for a demo study first, in C:Program FilesCDISC Expressstudiesexample1docMapping file – working version.

The first sheet is a welcome dashboard:

Then StudyMetadata sheet, a XML metadata specification used to generate define.xml. you need only add some information in “Values” column:

The FORMAT sheet:

Such format structure is similar with the one we export the format from a format catalog using

proc format library=library cntlout=format_out;

run;

In most production environment, programmers get formats from clinical data management group. If the entire formats are assigned into proper libraries (work or library), you don’t need to export such formats into this spreadsheet. Of course in the format sheet, you can type some customized format.

A typical domain sheet (AT LAST!) that needs efforts and our understanding of the software, DM for example:

From the ‘Dataset’ column, three raw datasets from C:Program FilesCDISC Expressstudiesexample1source needed to map into DM domain, demog, siteinv and eligassess. Note that you can use any data step options such as drop=, rename=, where= for the input datasets.

At the last of ‘Dataset’ column, “all” indicates that all the previous datasets mentioned above should be merged together for final processing.

In the ‘Merge Key’ column, ‘sitecode’ is designed to datasets demog and siteinv which means demog and siteinv should be merged by the common key, ‘sitecode’.

As we mentioned, all the previous datasets should be merged at last. But there is no common key settled in the ‘Merge Key’ column. It is a common rule: if no key specified for merge, USUBJID is used by default.

The third column is ‘CDISC variable’, which list all the needed variables according to implementation version. An important note: you do not need to implement all the variables according to the order as they appear in the blank template mapping file. In the previous blank file, “AGE” in DM domain is ordered in Line 12, but in this working file, “AGE” is calculated in the second last order. The variable order of final DM domain will be as same as the blank one.

It makes sense in practice. For example, the sequential variable, e.g. AESEQ is ordered after USUBJID, but you can only get the sequential number when all other variables well done. So SEQ variables are always computed in the final stage in a working mapping file.

“Expression” column specify the mapping rule from raw datasets to SDTM domains. Assignments, expressions and macro calls (rooted in C:Program FilesCDISC Expressmacrosfunction_library) are allowed in this column and most of them are straightforward. We will discuss more in the following section.

Sum up, we can “translate” this mapping sheet to SAS codes for better understanding of CDISC Express architecture:

data tem1;

    set demog;

    STUDYID=study;

    DOMAIN =&domain;

    USUBJID=%CONCATENATE(_variables=study sitecode patid);

    SUBJID =patid;

RFSTDTC=%D_RFST(_dataset=trtinf,_date=trtinfdt,_key=patid,_ivrsds=ivrs,_ivrsdt=randdat);

RFENDTC=%SORTLOOKUP(_dataset=disc,_variable=fupdat,_key=patid,_sort_variable=fupdat,_keep=last);

    SITEID =sitecode;

    INVID =invcode;

   BRTHDTC=%FORMAT(_variable=brthdat,_format=yymmdd10);

   SEX =%GENDER(_gender=sex);

   RACE =%D_RACE();

   ETHNIC =upcase(ethnic);

   COUNTRY="USA";

    DMDTC =%FORMAT(_variable=formdat,_format=yymmdd10);

   ARMCD ="";

    ARM ="";

run;

 

data tem2;

    merge tem1 siteinv(drop=invcode);

    by sitecode;

    INVNAM=trim(lname)||","||fname;

run;

 

data dm;

    merge tem2 eligassess;

    by patid;

  AGE =year(infcondt)-year(brthdat)-(month(brthdat)>month(infcondt))-(month(brthdat)=month(infcondt) and day(brthdat)>day(infcondt));

AGEU=%CONVERTIF(_if_variable=@AGE,_if_value=.,_then_value=, _else_value=YEARS);

run;

Following will be some explore the data manipulation techniques in CDISC Express, such as merge, transpose.

Dive into CDISC Express (1): Introductory

3. Step 1 of 6: Create a new study (create_new_study.sas)

Open create_new_study.sas in C:Program FilesCDISC Expressprograms, you can see only one line of a macro call:

%addnewstudy(studyname=my new study);

Just assign a study name to the macro variable, &studyname, e.g, “CLINCAP”:

%addnewstudy(studyname= CLINCAP);

Submit the codes, you can find a folder named “CLINCAP” with the same structure as the two demo studies imbedded in this application(example1 and example2) in C:Program FilesCDISC Expressstudies, see(the left and right panels are folders and files before and after the execution of create_new_study.sas. The following the same):

Folder ‘doc’ is used to hold the mapping files;

Folder ‘log’ used to hold log files generated by following macro calls, such as generating SDTM domains;

Folder ‘results’ and its subfolder will hold all the outputs, such as define.xml, SAS transport file, validation reports and SDTM datasets;

Folder ‘source’ holds all the clinical raw data used as inputs for SDTM domains;

Folder ‘tempdata’ holds all the temporary datasets generated by following macro calls.

Also, a configuration file named CLINCAP_configuration.sas put in C:Program FilesCDISC Expressprogramsstudy configuration. This file is used to set some study level parameters, such as lab and toxicity specifications (details in C:Program FilesCDISC ExpressspecsLab specs).

Two versions of SDTM implementation guides are supported by CDISC Express, CDISC SDTM Implementation Guide Version 3.1.1 and Version 3.1.2. You can find the corresponding specification files in C:Program FilesCDISC ExpressspecsSDTM specs:

SDTM_Specs_3_1_1.xls

SDTM_Specs_3_1_2.xls

The choosing of SDTM implementation version is also coded in the configuration file, in Line 41:

%LET SDTMSPECFILE=SDTM_Specs_3_1_1.xls;

Version 3.1.1 is used by default. You can also choose Version 3.1.2 if needed:

%LET SDTMSPECFILE=SDTM_Specs_3_1_2.xls;

Assign a study name and choose a SDTM implementation version. That’s all needed in step 1. Let’s take few minutes to navigate the software. CDISC Express is a set of macros and Excel files. It is important to know the file structure first:

C:Program FilesCDISC Express

├─documentation :FAQ, Quick Start, User Guide

├─macros

│ ├─ClinMap :system level macros

│ └─function_library :study level macros

├─programs :"action taken" macros

│ ├─study configuration :study parameters configuration

├─SDTM Validation :For validation of SDTM domains

│ └─study1

├─specs :specification files

│ ├─Excel engine :ExcelXP tagset file

│ ├─Lab specs :lab and toxicity

│ ├─Mapping validation :validation rules

│ ├─SDTM specs :hold two versions of SDTM implementation

│ └─SDTM Terminology :SDTM codelist(including NCI terminology)

├─studies

│ ├─example1

└─temp :hold temporary data not specified to any studies

As we already got, all the “action taken” programs such as create_new_study.sas are located in C:Program FilesCDISC Expressprograms. In create_new_study.sas, one macro is called, %addnewstudy, which is in C:Program FilesCDISC ExpressmacrosClinMap.

Note that in C:Program FilesCDISC Expressmacros, there are two sets of macros in different folders:

C:Program FilesCDISC ExpressmacrosClinMap: this folder holds all “system” level macros used by the application only. No modification encouraged.

C:Program FilesCDISC Expressmacrosfunction_library: macros used for mapping among studies. You can also create you own macro in this folder. The application imbedded macros also documented in user guide.

Following will be the most important part, mapping file.

Really she is the strangest creature in the world, far from heroic, variable as a weathercock, “bashful, insolent; chaste, lustful; prating, silent; laborious, delicate; ingenious, heavy; melancholic, pleasant; lying, true; knowing, ignorant; liberal, covetous, and prodigal”— in short, so complex, so indefinite, …

            –Virginia Woolf, The Common Reader, First Series (1925)

I don’t know if Ernest Hemingway is still one of the recognized dominant writers in colleges English education. At least once  a time he WAS. In an extreme form, he used only nouns and verbs to construct sentences.

In my personal English education(as Second Language), admittedly that there is also an absence of adjectives. It is just wonderful, nice, great, cool, weird, awesome,  and all in all, everything is OK or not OK, good or not good. In writing, my sentences lack of tone and shades. I write only technical articles in English and people can often well manage the so-called technical borings when acquiring information, knowledge, and opinions.

In reading when I try to just read for the sake of reading itself, I also find it is difficult to dig into pure literature pages where rich adjectives assembled heavily. I only read smoothly technical papers. So when I happened to have a paper book of Woolf and also loaded a corresponding public domain e-book in Kindle, I read the most intensive instances of adjectives ever. It is really totally different experiences.

There would be three types in any languages. For English:

in the top, pure literature, Shakespeare-like;

middle, which could be called the universal or international English; it is be the dominate English among nations in business, technology and even academia; most of the popular writers also utilize such sort of English to extent their global reputation;

bottom, the street language, slangs, talk-show.

I penetrate the English through the middle like almost all of the ESL learners. It is the most convenient and effective way in a very short run. But for a leap in long run, some friends just suggest that I should go down the street or climb up to the top. Ok I am on the way. Virginia Woolf is the first stop and I keep the first notes.

CDISC Express is the first and only practically usable open source CDISC converter I even met. I wrote a post a month ago when I first tested it with great interests and reported some issues to its fix system. Then I also had the great opportunity to discuss the software via email with its core developer, Romain Miralles. This post is just my personal notes on how to use and dig into the software, and will be best serve as a working documentation. You can return to me for any questions and comments.

By the way, there is an opportunity for your practicing and you will also have a change to win an iPad2 from Clinovo’s CDISC Express Contest:

http://www.clinovo.com/cdisc/game

The due day is July 15^th and I already submitted my work. That’s fun.

1. Download and Installation

You can get CDISC Express for free in

http://www.clinovo.com/cdisc/download

It is a window application and will be installed by default in

C:Program FilesCDISC Express

After installation, this path will be coded as a macro variable &CDISCPATH in the following six SAS files which are all located in C:Program FilesCDISC Expressprograms:

create_new_study.sas

generate_Definexml.sas

generate_mapping_template.sas

generate_SDTM.sas

Validate_Mapping_File.sas

Validate_SDTM_Domains.sas

The macro variable reads as

%LET CDISCPATH = C:Program FilesCDISC Express;

If you change the destination folder at the installation stage, e.g., to D:CDISC Express, the value of the macro variable &CDISCPATH will be changed accordingly in the six files mentioned before:

%LET CDISCPATH = D:CDISC Express;

Note that if you want copy the whole folder of files to another destination, you should at least manually change the value of &CDISCPATH in such six files or add some codes to capture the path accordingly. From this point of view, the path setting of CDISC Express is not completely portable. Recommend that if you have such needs, just re-install the software in any destination you want. It will not write any records into registry and you can have many copies in one machine.

The following discussion assumes the software roots in C:Program FilesCDISC Express.

2. Working Flow

You can follow all the 6 action steps one by one coded in

C:Program FilesCDISC Expressprograms

1) Create a new study (create_new_study.sas)

Simple and easy. Just assign a new study name in a macro call and run.

2) Generate mapping file (generate_mapping_template.sas)

This is the critical and most time consuming part. You should design mapping rules for every domain needed in Excel spreadsheets (the MAPPING FILE). If done, all other tasks, such as generate SDTM datasets, SAS transport files, define.xml and validation, can be well done by just clicking buttons

.

3) Validate mapping file (Validate_Mapping_File.sas)

For validating the mapping file, just click the button. As mentioned, the most important work is designing mapping file. It would be back and forth to design mapping file and validate it.

4) Generate SDTM datasets (generate_SDTM.sas)

If mapping file is OK, click the button.

5) Validate SDTM datasets (Validate_SDTM_Domains.sas)

Click the button.

6) Generate Define.xml (generate_Definexml.sas)

Click the button.

Following part will dig into the software step by step.

It’s wonderful. It’s awful. It’s wonderfully awful and awfully wonderful. –Frommer’s Las Vegas 2010

June at last! June, July and August (great thanks to French colleagues!) would be the best time to prepare a paper for next year’s SAS Global Forum (SGF, 2012 in Orlando, Florida). SGF2012 will hold at the “real” Walt Disney at Orlando against Las Vegas, the adults’ Walt Disney for SGF2011. Lots of children will be anticipated to register as guests for this meeting and I may contribute one:).

Before moving to 2012, I need to add some notes on 2011. It was my first time to present at SAS Global Forum, my first time to attend the Forum and even my first time out of China. First of all, I’d like to extend my special thanks to

Nancy Brucken of i3, section chair of Planning and Support who served as my key contact of the conference committee for all the coordination stuff

Duke Owen of Westat, my mentor of this year’s SAS Global Forum Presenter Mentoring Program. We had intensive communication via email, telephone and WebEx, see

01/25/2011 initial call.  review abstract and title
02/01/2011 final abstract submission, no call
02/08/2011 no call – Chinese New Year celebration continues
02/15/2011 call – review paper
02/22/2011 call – finalize paper for submission
02/25/2011 submission of pdf (on or before this date)
03/01/2011 call – first draft of slides, or at least the outline of your plan
03/08/2011 call – refine the slides and preview of what your script for presentation
03/15/2011 call – "live" presentation for Duke.   This can be done just on our pc’s or as a WebEx
03/22/2011 call – WebEx for a wider audience.  We can discuss who to invite and adjust time as needed.
03/29/2011 call – Finalize slides/script for you to send to Nancy Brucken
04/02/2011 Duke arrives in Las Vegas
04/03/2011 John arrives in Las Vegas (I think)
04/04/2011 Rehearsal room practice and final slides to Nancy
04/05/2011 Show Time:  Tuesday 3:00-3:20 pm    ,  drinks on me at the kickback
04/06/2011 It is all in the past
04/08/2011 Duke leaves Las Vegas

Last but not least, I also want to thank my boss and my company to support and sponsor me the trip. It is not a world-peace speech. Economy is tough worldwide and few SAS programmers in China could get the opportunity to attend the Forum. In my observation, even in US, relative fewer programmers in pharmaceutical company and other industries attended SGF2011 compared former years.

That might be also one of the reasons why in this year, SAS Institute had a stronger presence at Las Vegas. For me, it’ not bad. I had the opportunity to meet and talk with some of my former colleagues at SAS R&D Beijing and Cary (even when we’re all in Beijing, we have few chance to meet together). What’s more, SGF2011 had wonderful SAS Support and Demo Area and Hands-on Workshop fully supported by the Institute. It was definitely great opportunity to learn almost all parts of SAS system by talking with R&D staff directly. Admitted that I spent more time in the demo room instead of the user presentation room (I can read all the user papers after the Forum, but there is only one chance I can talk with R&D staff about the software itself!)

My personal presence at Las Vegas:

1.  Monday, April 4, attended the pre-conference training, Data Simulation for Evaluation Statistical Methods in SAS by Rick Wicklin. Rick is a famous blog writer on IML(The DO Loop), SAS’s matrix language. His blog is just as popular as Chris Hemedinger’s The SAS Dummy, a blog on SAS Enterprise Guide. It’s wonderful to talk with these two bloggers face to face!

2. Tuesday, April 5, I was invited to a panel discussion on SAS Business Visualization and Mobile Computing in a SAS BI Product Development Focus Group, hosted by 3 SAS directors, developer Justin Choy, product manager Pyan Schmidt and marketing Jo Fisher. Visualization and mobile computing (in iPad, iPhone, …) are two of most hottest topics in Las Vegas. This Focus Group also wanted to gather feedbacks from end users.

3. Tuesday, April 5, my presentation, Work Smarter Rather than Harder- Tools for Growing up A SAS® Programmer.

4. Wednesday, April 6, one or two seconds personal show up in Gregory Nelson and Neil Howard’s featured presentation in lunch, Revenge of the Semicolon People: Business Analytics at Work.  !!!!!

That time Duke and me were in the same table and Greg and Neil were just showing they slides. Suddenly I found a picture of Duke, Shane and me was in the presentation for SGF2011 after some historical pictures for 2010. It just flashed past but it was enough to excite around the table!

The core of CDISC Express is an Excel configuration file called Mapping File which defines all the metadata and mapping rules required by CDISC SDTM standard. Then a set of SAS macros is used to

• validate the SDTM mapping file
• generate SDTM datasets
• generate define.xml file
• validate the SDTM datasets

I found its architecture is simple, concise and powerful. When mentioned “simple”, I mean it is easy to understand and implement. It is still in its version 1.0, I wish following enhancements in the future according my personal user experiences:

• More web browsers supports for define.xml file and documentation
• More merge or set types
• Debug mechanism enhancement

look to the master,
follow the master,
walk with the master,
see through the master,
become the master.

The SAS Global Forum 2011 (Apr 4-7, Las Vegas) papers are now available on line:

http://support.sas.com/resources/papers/proceedings11/TOC.html

There are lots of Web 2.0 elements in this year’s SGF, including a Twitter hashtag #SASGF11, a Facebook account, a blog, a LinkedIn group, and of course, sasCommunity.org. I will give a talk at this conference (Tuesday, April 5, 05:30- 05:50 PM, Promenade Level, Neopolitan IV) to discuss how to get informed with little efforts from such multiple channels for SAS programmers using Gmail and Good Reader to create a personal information portal. That is part of my first SAS paper, Work Smarter Rather than Harder- Tools for Growing up A SAS® Programmer.

It will also be my first presence in SGF. Just can’t wait to walk with SAS masters there!

1. Google calculator

As Google fanatics, I first try to search the answer via Google:

Whoops, nothing interested returned! Type “170!” and get the output:

Why kinda things happened in this calculator? 171! is just equal to 171*170!.

2. Excel

Switch to Excel spreadsheet. Function fact(*) used:

Oo, interesting. The same.

3. SAS

Google and Excel may be the niche players in calculators’ family. Why not try to use some programming languages?

As a SAS programmer, my handy tool is SAS of course.

First, I use SAS data step with its build-in function fact(*):

data _null_;
    x=fact(170);
    y=fact(171);
    put x= y=;
run;

and I get

NOTE: Invalid argument to function FACT at line 49 column 7.
x=7.257416E306 y=.
x=7.257416E306 y=. _ERROR_=1 _N_=1
NOTE: Mathematical operations could not be performed at the following places. The results of the operations have been set to missing values.

Expected or unexpected? I don’t know how this fact(*) function is defined, and  try to define a function to calculate the factorials by myself. In SAS 9.2, you can use PROC FCMP(also available at 9.1.3 as a experimental procedure):

proc fcmp outlib = work.funcs.math ;
    function factorial(k) ;
        if k = 0 then return(1) ;
        z = k ; *preserve k ;
        x = factorial(k-1) ;
        k = z ; *recover k ;
        k = k * x ;
        return(k) ;
    endsub ;
quit ;

options cmplib=work.funcs ;

Use this self-defined function to get 170!

proc fcmp ;
    x = factorial (170) ;
    put x = ;
run ;

The FCMP procedure returns

x=7.257416E306

Try to calculate 171! ?

proc fcmp ;
    y = factorial (171) ;
    put y = ;
run ;

Just get the overflow error. The interaction stops at 170!:

ERROR: An overflow occurred during execution in function ‘factorial’ in statement number 7 at   line 10 column 1.
       The statement was:
    1      (10:1)    k = (k=171) * (x=7.257416E306)

The above function definitions use recursion. Recursion may have some limitation on efficiency. We could try the loop without recursion. SAS/IML doesn’t support recursion. Let SAS/IML to the court:

proc iml;
    start factorial (n);
        fact=1;
        do i=1 to n;
        fact=fact*i;
        end;
        return (fact);
    finish factorial;

    x= factorial (170);
    print x;

    y= factorial (171);
    print y;
quit;

Again, I get 170!

    x
7.257E306

and a overflow error for 171!

            y= factorial (171);
ERROR: Overflow error in *.

Turing, Von Neumann and Tony, what happened?

4. R

When SAS failed, lots of voices pop up: use R! OK, Rction!

> x=factorial(170);x
[1] 7.257416e+306
> y=factorial(171);y
Warning message:
In factorial(171) : value out of range in ‘gammafn’
[1] Inf