12 Sep Save parameters and seeds of simulations

Today, Robert posted the nice idea to save seeds of simulations. He recommends storing seeds with results, alternatively, store seeds with starting parameter values if you run numerical optimization. It is also useful to create an external file which stores parameters and seeds with a unique identifier which one can use for naming results folders and file organization. Th is post contains a code to do the latter.

Update

R tipp of the day: Renaud Gaujoux' package 'doRNG' is the equivalent of set.seed(1234) for your parallelized code. Like this: foreach(i=1:4, .options.RNG=1234) %dorng% > https://t.co/linBcqqow4

— Jana Jarecki (@JanaJarecki) August 28, 2018

The package save the associated seed as attribute to your (paralelized) simulation result.

Seeds and storing seeds

Imagine your code uses random draws with e.g., the runif() function. A seed is a number telling R the specific string of pseudorandom number R should use. Usually we set a seed before we run simulations such that others can replicate our simulation results exactly without the variation through “your current draw’s randomness”. Usually, we repeat our simulation with different seeds to make sure the results are robust and independent of the luck of the specific random draw. R-bloggers recommends to store all those seeds with the results.

R simulates with a pre-defined seed, you can look at it via
[code lange = “R”]set.seed(237)
.Random.seed
[/code]

Define one specific seed by this command before running the simulation. With the same seed all subsquent results will be replicable exactly.

[code lang = “R”]set.seed(273) #273 could be any number
# … simulation …
[/code]

Instead of hardcoding setting the seed we use R’s pre-defined seed but store it with the result (this is adapted from the R-blogger code)

[code lang = “R”]iter <- 10 #number of simulation replications
seeds <- list(NULL) #initialize list object to store seeds
for(i in 1:iter) {
seeds[[i]] <- .Random.seed
# … simulation function …
# store results, seeds
}
[/code]

Store the seed and the parameter

Similarly it makes sense to store parameter combinations with seeds. I add the seed to the starting parameters of my simulation and store it as I go along.

I run numerical simulations with many different combinations of parameters, for example to try out a plausible range for one parameter. Each result needs a unique identifier where the straightforward identifier is the unique parameter combination. But it is tedious to name the results with an long name holding all the parameter values. Therefore it makes sense to assign an identifier number to each parameter combination and store the parameter values in an external file. The below achieves this

[code lang = “R”]# This may be your parameters including a seed
P = list(p1=1, p2=2, seed=3)

# Store the parameter values in a file called “ConditionNumbers.csv”

# Transform parameter into a string
par = paste(sep=””,gsub(“_”,””,names(P)), #parameter as a _-separated string
lapply(P, paste,collapse=”,”), collapse=”_”)

# Check if the file already exists
exi = file.exists(“ConditionNumbers.csv”)
# If it does not, create is
if(!exi) {
write.table(data.frame(nr=1,cond=par), file.path(getwd(),”ConditionNumbers.csv”), sep=”;”, row.names=F)
# If it exists, make sure you did not already assing an identifier to theis parameter combination
} else if(exi) {
ConNr = read.table(“ConditionNumbers.csv”, sep=”;”, header=T)
if (!any(ConNr$cond==par)) { #if new combination: append it
write.table(data.frame(nr=max(ConNr$nr)+1, cond=par), “ConditionNumbers.csv”, sep=”;”, append=T, col.names=F, row.names=F)
} else if (any(ConNr$cond==par)) {
stop(“This parameter condition has already a number.”)
}
}
[/code]

Imagine you changed your code and you want to re-run one of these simulations with a specific parameter combination. The new results should overwrite the previous results from the old code. Therefore here is code for a function that can both write or get the parameter combination number to identify if we have already run a simulation with the current parameter values
[code lange = “R”]fcn <- function(P, type){ #get condition number
par = paste(sep=””,gsub(“_”,””,names(P)),
lapply(P, paste,collapse=”,”), collapse=”_”)
exi = file.exis(“ConditionNumbers.csv”)

if(!exi) { #check
if(type==”g”) {
stop(“Parameter Number file ‘ConditionNumbers.csv’ does not exis.”)
}
if(type==”w”) {
write.table(data.frame(nr=1,cond=par), file.path(getwd(),”ConditionNumbers.csv”), sep=”;”, row.names=F)
}
} else if(exi) {
ConNr = read.table(“ConditionNumbers.csv”, sep=”;”, header=T)

if(type==”g” & !any(ConNr$cond==par)) { #check
stop(“Parameters in the parameter number file don’t match current parameters.”)
}

if(type==”g”) { #get parameter-combination-no. from condition number file
cn = ConNr$nr[which(ConNr$cond==par)]
cat(“\nThe condition number is “, cn, “\n”)
cn
} else if (type==”w” & !any(ConNr$cond==par)) { #if new combination: append it
write.table(data.frame(nr=max(ConNr$nr)+1, cond=par), “ConditionNumbers.csv”, sep=”;”, append=T, col.names=F, row.names=F)
} else if (type==”w” & any(ConNr$cond==par)) { #if old combination: do nothing
stop(“This parameter condition has already a number.”)
}
}
[/code]