Caret:Documentation:Statistics
From Van Essen Lab
(→WARNING) 
(→Performing Inferential Statistical Tests in Caret) 

Line 31:  Line 31:  
== Performing Inferential Statistical Tests in Caret ==  == Performing Inferential Statistical Tests in Caret ==  
  Inferential statistical tests in Caret are performed on metric or surface shape files. All of the data (metric or shape files) must be on a coregistered surface so that all data files have the same number of nodes and each node number ''i'' is  +  Inferential statistical tests in Caret are performed on metric or surface shape files. All of the data (metric or shape files) must be on a coregistered surface so that all data files have the same number of nodes and each node number ''i'' is "in register" across subjects (i.e., all subjects' surfaces have undergone surfacebased registration using Caret, Freesurfer, CIVET, or other software). 
The goal is to find clusters (regions) that are statistically different between the groups of input data. That is, one can reject the null hypothesis which states that the metric/shape values at each node are essentially the same.  The goal is to find clusters (regions) that are statistically different between the groups of input data. That is, one can reject the null hypothesis which states that the metric/shape values at each node are essentially the same. 
Revision as of 18:28, 20 February 2009
Contents

WARNING
THIS DOCUMENT IS IN DEVELOPMENT AND DESCRIBES FUTURE VERSIONS OF CARET
Descriptive Statistics
Descriptive statistics provide information about the data such as the mean (average), median (middle value), mode (most common value), standard deviation, and variance. When computing the standard deviation, one must know if the data values represent the entire population in which case division is by N (number of items) or the data values are a subsample of the population in which case division is by N  1.
Population Descriptive Statistics
 Population Mean
 Population Standard Deviation OR
 Population Variance = σ^{2}
 Standard Deviation of the Mean
Sample Descriptive Statistics
 Sample Mean
 Sample Standard Deviation OR
 Sample Variance = S^{2}
 Standard Error of the Mean
Miscellaneous Descriptive Statistics
 ZScore
Inferential Statistic Tests
The purpose of the inferential statistic is to take the input files, perform a statistical test at each node, and create a new file containing one or more statistical measurements (F, T, Z, etc) at each node.
Performing Inferential Statistical Tests in Caret
Inferential statistical tests in Caret are performed on metric or surface shape files. All of the data (metric or shape files) must be on a coregistered surface so that all data files have the same number of nodes and each node number i is "in register" across subjects (i.e., all subjects' surfaces have undergone surfacebased registration using Caret, Freesurfer, CIVET, or other software).
The goal is to find clusters (regions) that are statistically different between the groups of input data. That is, one can reject the null hypothesis which states that the metric/shape values at each node are essentially the same.
The steps in Caret are:
 Run the input files through an inferential statistical test.
 Perform a randomization process to identify the pvalue for a cluster based upon the surface area of the cluster.
 Assign PValues to the output of the original inferential statistical test.
Parametric Inferential Tests
For parametric tests, the data is assumed to be in a specific probability distribution, typically the normal (gaussian) distribution.
ANOVA (Analysis of Variance), One Way
A oneway ANOVA determines if the mean values at each node for two or more groups of subjects are statistically different.
K = Number of Groups
N = Total number of observations
N_{i} = Number of items in group i.
= Variance for group i.
Mean of group i,
Grand Mean,
Variance of distribution of means,
Mean Square between groups,
Mean Square within groups,
df_{Between}(numerator) = K − 1
df_{Within}(denominator) = N − K
TTest, OneSample
A onesample TTest determines if the mean value at each node is statistically different than a specified value, often zero.
t =
TTest, Paired (Dependent Means)
A paired TTest determines if mean at each node is statistically different for two measurements (X and Y) on one group of subjects.
t =
TTest, TwoSample (Independent Means)
A twosample TTest determines if the means at each node for two groups of subjects are statistically different.
Equal (Pooled) Variances
df = N_{1} + N_{2} − 2
Unequal (Unpooled) Variances
NonParametric (Distribution Free) Inferential Statistic Tests
For nonparametric tests, no assumptions are made about the distribution of the data.
PValue Determination
In Caret, PValues are determined using clusterbased techniques.
SupraThresholding
With clusterbased thresholding, the user must select a threshold, or cutoff value. Clusters are formed from connected groups of nodes whose metric values are greater than or equal to the threshold. There is no one "correct" threshold value. Smaller values result in larger clusters and higher thresholds result in smaller clusters.
ThresholdFree Cluster Enhancement (TFCE)
With thresholdfree cluster enhancement, the user does not need to select a threshold, or cutoff value.
Consider a spherical representation of the cortex with radius R. For each node adjust its radius by adding the node's raw statistical value. For each node with a positive raw statistical value, modify the radius of the surrounding nodes, so, no matter what path is followed to a node with a statistical value less than or equal to zero, the radius of the node decreases. What remains is the "supporting section". Essentially, "jagged terrain" becomes a roughly conical shape.
Positive and negative metrics are handled separately. When determining positive TFCE, all metrics less than zero are set to zero. When determining negative TFCE, all metrics greater than zero are set to zero, the negatives are then multiplied by 1, and the positive TFCE algorithm is used.
Algorithm that uses a breadthfirst search.
 For each triangle in the fiducial surface, calculate its surface area. Scale the triangle's area by the average of the triangle's three node's areal distortion.
 Copy original metric column with statistical values at each node.
 If negative searching, multiply all metric values by 1.
 Set all nodes with metric less than zero to zero.
 For all nodes:
 Copy metric column.
 Mark all nodes with positive value as unvisited and all nodes with zero values as visited.
 Add start node number to queue.
 Loop until queue is empty
 Get current node from queue.
 If node unvisited
 Mark node visited.
 If the nodes metric value is greater than zero
 Get node's neighbors sorted by smallest metric value. All new neighbors must be BELOW any previous neighbors.
 For all neighbor nodes
 If the neighbor is unvisited
 Set neighbor's metric value to minimum of its value and the current node's metric value.
 Add the neighbor to the queue.
 If the neighbor is unvisited
 For all triangles with visited nodes, multiply the triangle's surface area by the triangle's height . We will estimate the height by using the average of the original statistical values at each of the triangle's three nodes. Add these TFCE scores to the start node's TFCE value.
My concern is that the need to perform the above algorithm on many nodes may be slow. When considering the randomization process, the time required may be significant. For example, a user on the "neuromultcomp" list claims FSL's randomise requires more than three days for 5000 iterations using TFCE. An algorithm that runs randomization in parallel will help.
Visualize as a flat surface where Z = metric value
Randomization
Randomization testing is used to determine the PValues. In this process, the subjects are pooled and then randomly split into groups. The statistical test is performed on this new grouping and the largest cluster is identified. This process is iterated many times and the largest cluster from each grouping is saved. At the end, the largest clusters are sorted by decreasing surface area. The surface area of the 95% percent largest cluster (for a pvalue of 0.05) indicates the surface area of a cluster with 0.05 significance.
The original statistical metric column's clusters are assigned pvalues based upon their ranking, by surface area, in the sorted, randomlycreated clusters.
Randomization With One Group of Subjects
When there is one group of subjects, such as in a onesample TTest, it is not possible to randomize among groups. So, the randomization is performed by randomly flipping the signs of the values for each subject. The statistical test is then run on each of these randomizations and the largest clusters are identified.
Randomization With Multiple Groups of Subjects
With multiple groups of subjects, all of subjects are placed into a pool. Subjects are then randomly drawn from the pool and placed into new groups. The statistical tests are then run on each of these randomizations and the largest clusters are identified.
When creating the randomized groups, combinations may result in the creation of a randomized group that was already created. Permutations prevent the repeating of randomized groups. For example, the sets {A, B} and {B, A} are two unique combinations but are considered identical permutations.
gifti_statistics
gifti_statistics is a command line program that performs statistical operations on GIFTI surface data files.
Descriptive Statistical Operations
Descriptive statistic operations are performed on the values associated with each node. The parameters may be specified in any order.
gifti_statistics descriptive \
 datafile <datafilename.gii> \
 outputfile <outputfilename.gii> \
 <descriptivestatistics>
There may be more than one input file. Specify each input data file with the datafile option.
Population Descriptive Statistics
 popmean <newpopulationmeandataarrayname>
 popsd <newpopulationstandarddeviationdataarrayname>
 popvar <newpopulationvariancedataarrayname>
 popsdm <newpopulationstandarddeviationofthemean>
Sample Descriptive Statistics
 samplemean <newsamplemeandataarrayname>
 samplesd <newsamplestandarddeviationdataarrayname>
 samplevar <newsamplevariancedataarrayname>
 samplesem <newsamplestandarderrorofthemeandataarrayname>
Inferential Statistical Operations
gifti_statistics <inferentialtest>
 datafilegroup <groupname> <datafilename.gii> \
 outputfile <outputfilecontainingstatistics.gii>
 <surfaceinformation> \
 [Multithreading] \
 [PValueDetermination] \
 [VarianceSmoothing] \
inferentialtest is one of:
 inferentialttestonesample <meanvalue>
 inferentialttesttwosamplepooledvariance
 inferentialttesttwosampleunpooledvariance
 inferentialttestpaired
 inferentialanovaoneway
PValue Determination
PValues may be determined using either clusterbased thresholding or thresholdfree cluster enhancement.
PValue Methods
 cluster clusternegativethreshold clusterpositivethreshold
 tfce <positive  negative both>
PValue Options
 iterations specifies the number of iterations. The default is 50.
 permute During the randomization process, using this option ensures that each random grouping is different than any previous groupings.
 randomseed <valuelargepositiveinteger> This is the seed that initializes random number generation sequence.
PValue Output
 outputdof newdataarrayname
 outputpvalue newdataarrayname
 outputoneminuspvalue dataarrayname
 outputrandomization filename.gii* This new file will contain the randomized data.
The PValue, 1PValue, and the DOF is added to the output file.
Variance Smoothing
 variancesmoothing iterations strength
Iterations must be greater than zero and strength ranges from 0.0 to 1.0.
Surface Information
The user may specify either a gifti surface file or a coordinate file and a topology file.
 surface filename.gii.surf
 coordinate filename.gii.coord topology filename.gii.topo
If there surface has been distorted in some way, a gifti data array may contain data that corrects for the distortion.
 surfacedistortion functional_distortion.gii* dataarraynameor number
MultiThreading
If the computer being used has either multiple processors or multicore processors, running multiple threads will likely reduce the execution time of some operations.
 threads numberofthreadstorun
The number of threads must be one or larger.
gifti_statistics plan
 Translate the statistics code from C++ to Java. I have found Java development much easier than C++ primarily due to far better error checking earlier in the process and a much smarter development environment. Most of the existing statistics algorithms will convert from C++ to Java fairly quickly.
 Development of TFCE, in particular, will be faster if done in Java.
 Multithreading will be easier to use.
 Input files must be in GIFTI format. However, if REALLY needed, Caret formats can be read. caret_command can convert files between Caret and GIFTI formats.
 Use NIFTI intent codes for newly created metric columns
 Rewriting the statistical operations will simplify the addition of new statistical methods.
 Add GIFTI statistics to NITRC?
References
Journal Articles
 Nonparametric Permutation Test For Functional Neuroimaing: A Primer with Examples. Thomas E. Nichols and Andrew P. Holmes. Human Brain Mapping 15:1
 ThresholdFree Cluster Enhancement: Addressing problems of smoothing, threshold dependence and localisation in cluster inference. Stephen M. Smith and Thomas E. Nichols.NeuroImage 2009 44(1)
Web Sites
Glossary
 Clusterbased Thresholding  Groups of connected nodes with attribute values greater than a threshold (ie: t > 3.0) are identified.
 Family Wise Error  Probability of making Type I Errors (rejecting the null hypothesis when the null hypothesis is true). Also called alpha error.
 Gaussian Field Theory
 Nonparametric Statistics  The test contains no requirement that the data fit a probability distribution.
 Permutation Testing  A type of nonparametric test. http://en.wikipedia.org/wiki/Resampling_(statistics)
 Parametric Statistics  The test requires the data to fit a probability distribution, typically the normal distribution.
 ROC (Receiver Operating Characteristic) A plot that shows the tradeoff of true positive and false positive as the treshold is varied.
 Spatial Smoothing
 Supporting Section
 Threshold Free Cluster Enhancement (TFCE)