LISTSERV mailing list manager LISTSERV 16.0

Help for STAT_873 Archives


STAT_873 Archives

STAT_873 Archives


STAT_873@LISTSERV.UNL.EDU


View:

Message:

[

First

|

Previous

|

Next

|

Last

]

By Topic:

[

First

|

Previous

|

Next

|

Last

]

By Author:

[

First

|

Previous

|

Next

|

Last

]

Font:

Proportional Font

LISTSERV Archives

LISTSERV Archives

STAT_873 Home

STAT_873 Home

STAT_873  November 2005, Week 5

STAT_873 November 2005, Week 5

Subject:

Re: Mismatch in output

From:

Chris Bilder <[log in to unmask]>

Reply-To:

STAT 873 Listserv <[log in to unmask]>

Date:

Tue, 29 Nov 2005 12:55:28 -0600

Content-Type:

text/plain

Parts/Attachments:

Parts/Attachments

text/plain (105 lines)

Jeremy,

It is disappointing that SAS does not do the ROC curve using
crossvalidation information.  However, I would not discount its use.  Note
that ALL cutoff probabilities are used with resubstution.  Therefore when
comparing different choices for cutoff probabilities, you are comparing it
using resubstitution always.  Also, resubstitution will still give similar
results to crossvalidation for a LARGE sample size - see the placekicking
data set for an example.

So, how could you create an ROC curve plot in SAS using crossvalidation?
I have not tried this yet, but here is one method.  Set a lot of cutoff
probabilities in the MODEL statement of PROC LOGISTIC.  Use the Output
Delivery System (ODS) to output all of the classification table to a data
set.  Plot the sensitivity and specificity in the data set to produce the
curve.

We have used the ODS a little in this class.  For more information on it,
please see my SAS course lecture notes at
http://www.chrisbilder.com/stat4091/schedule.htm.

Chris

On Tue, 29 Nov 2005 09:12:18 -0600, Jeremy Penn <[log in to unmask]>
wrote:

>So the ROC curve isn't that helpful in selecting a cutoff probability
level,
>because it uses results from resubstitution.  Is there an easy way that
you
>know of to create an ROC curve from the crossvalidation data?
>
>Jeremy Penn
>University of Nebraska-Lincoln
>Graduate Assistant
>402-325-8598
>
>
>
>-----Original Message-----
>From: STAT 873 Listserv [mailto:[log in to unmask]] On Behalf Of
>Chris Bilder
>Sent: Tuesday, November 29, 2005 8:53 AM
>To: [log in to unmask]
>Subject: Re: Mismatch in output
>
>Jeremy,
>
>Please see my message board posting from last Wednesday for the answer to
>your question: http://listsrv.unl.edu/cgi-bin/wa?
>A2=ind0511d&L=stat_873&T=0&F=&S=&P=170.
>
>Chris
>
>On Tue, 29 Nov 2005 08:44:49 -0600, Jeremy Penn <[log in to unmask]>
>wrote:
>
>>Our group is having an unusual error, and we can't figure out why it is
>>occurring.
>>We are trying to complete the last question for problem #1.  When we run
>the
>>original proc logistic, we get some output for particular cutoff
>probability
>>levels.
>>However, when we look in the out_roc file (located in the 'work' folder),
>we
>>find that the values in out_roc do NOT match with what is appearing in
the
>>output.  The same cutoff probability values in the two different
locations
>>have different sensitivity and 1-specificity values.  For example, (this
>is
>>just an estimate of what I remember) for a cutoff of .78, the output says
>>that 162 events were predicted correctly.  But in the out_roc file, for a
>>cutoff probability of about .78, it says that 165 events were predicted
>>correctly.
>>We can't figure out why the two places are giving us different results.
>>
>>Here's the SAS code we are using:
>>
>>title2 'Logistic Regression Model with 5 Variables';
>>proc logistic data=set1 descending;
>>        model PASS =EFFORT      LGOAL EFFICACY ANXIETY DISORG/ lackfit
>>alpha=0.10 clodds=wald ctable
>>pprob=0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9
>>        outroc=out_roc;
>>run;
>>
>>proc gplot data=out_roc;
>>goptions htext=1;
>>title2 'Samples ROC Curve';
>>   plot _sensit_*_1mspec_ / vaxis=axis1 haxis=axis2 frame grid;
>>   axis1 label = (a=90 'Sensitivity')
>>          length = 30
>>          order = (0 to 1 by .1);
>>   axis2 label = ('1-Specificity')
>>          length = 50
>>          order = (0 to 1 by .1);
>>   symbol1 i=join v=none c=blue;
>>run;
>>
>>Jeremy Penn
>>University of Nebraska-Lincoln
>>Graduate Assistant
>>402-325-8598

Top of Message | Previous Page | Permalink

Advanced Options


Options

Log In

Log In

Get Password

Get Password


Search Archives

Search Archives


Subscribe or Unsubscribe

Subscribe or Unsubscribe


Archives

September 2014, Week 1
August 2014, Week 4
August 2014, Week 3
August 2014, Week 2
December 2013, Week 3
December 2013, Week 2
December 2013, Week 1
November 2013, Week 5
November 2013, Week 4
November 2013, Week 3
November 2013, Week 2
November 2013, Week 1
October 2013, Week 5
October 2013, Week 4
October 2013, Week 3
October 2013, Week 2
October 2013, Week 1
September 2013, Week 5
September 2013, Week 4
September 2013, Week 3
September 2013, Week 2
September 2013, Week 1
August 2013, Week 5
August 2013, Week 4
August 2013, Week 3
August 2013, Week 2
December 2005, Week 3
December 2005, Week 2
December 2005, Week 1
November 2005, Week 5
November 2005, Week 4
November 2005, Week 3
November 2005, Week 2
November 2005, Week 1
October 2005, Week 4
October 2005, Week 3
October 2005, Week 2
October 2005, Week 1
September 2005, Week 5
September 2005, Week 4
September 2005, Week 3
September 2005, Week 2
September 2005, Week 1
August 2005, Week 5
August 2005, Week 4
August 2005, Week 3
February 2005, Week 4
December 2004, Week 3
December 2004, Week 2
December 2004, Week 1
November 2004, Week 5
November 2004, Week 4
November 2004, Week 3
November 2004, Week 2
November 2004, Week 1
October 2004, Week 4
October 2004, Week 2
October 2004, Week 1
September 2004, Week 4
September 2004, Week 3
September 2004, Week 2
September 2004, Week 1
August 2004, Week 5
August 2004, Week 4

ATOM RSS1 RSS2



LISTSERV.UNL.EDU

CataList Email List Search Powered by the LISTSERV Email List Manager