Configuration files

Format:

{ "Configuration-file": {
    "group-1":{
            "configuration1String": "",
            "configuration2String": "",
            "configuration3Int": 5
        },
        
    "group-2":{
            "configuration4String": "",
            "configuration5Float": 0.01,
            "configuration6Bool": true,
            "configuration6List": [0 1 2 3 4]
        }
}
}

Engine configuration

Required configuration to initialize the engine and execute training and inference:

{ "Configuration-file": {
    "module-configuration":{
        "configInference": "(String) Path to the inference configuration file",
        "configDataset": "(String) Path to the dataloader configuration file"
    },
    "model-build":{
        "inferenceModel": "(String) Inference model",
        "signalModel": "(String) Signal model",
        "likelihoodModel": "(String) Likelihood model",
        "datasetModel": "(String) Dataloader model",
        "lossFunction": "(String) Training loss function ['MSE', 'L1', 'NLLLoss', 'SmoothL1']",
        "trainerModule": "(String) Engine used to train a model ['emcqmri', 'monai']",
        "estimatorModule": "(String) Engine used to estimate using a pre-trained model ['emcqmri']",
        "optimizer": "(String) Training optimiser ['Adam', 'AdaDelta', 'RMSProp', 'SGD']",
        "useCUDA": "(Bool) Wether to use GPU to train/estimation",
        "runBenchmark": "(Bool) Wether to run GPU benchmarking before training",
        "allowCMDoverride": "(Bool) Wether to  allow override of configuration through command line arguments (it requires additional Python files for configuration)",
        "displayConfigurations": "(Bool) Wether to display all loaded configurations"
    },
    "datasets":{
        "mode": "(String) Set the execution mode of the EMCqMRI ['training', 'testing']",
        "trainingDataPath": "(String) The path to the training data folder:",
        "runValidation": "(Bool) Wether to run validation during training",
        "validationDataPath": "(String) The path to the validation data folder",
        "testingDataPath": "(String) The path to the testing data folder",
        "fileExtension": "(String) File extension of the data files ['pkl', 'nii']",
        "preLoadData": "(Bool) Whether to load the entire data folder or individual files",
        "shuffleData": "(Bool) Whether to shuffle the data at each epoch"
    },
    "training-engine":{
        "learningRate": "(Float) The initial learning rate during training",
        "batchSize": "(Int) The mini-batch size used during training",
        "epochs": "(Int) The number of training epochs",
        "numWorkers": "(Int) The number of dataloader workers used to load the data",
        "loadCheckpoint": "(Bool) Whether to load a pre-trained model",
        "loadCheckpointPath": "(String) Full path to a pre-trained network model (*.pth)",
        "saveResults": "(Bool) Whether to save the output of the model",
        "saveResultsPath": "(String) Path to folder where the data should be saved",
        "saveCheckpoint": "(Bool) Whether to save the a trained-model at each epoch (Only in 'training' mode)",
        "saveCheckpointPath": "(String) Path to folder where pre-trained models should be saved",
        "sulfixCheckpoint": "(String) Suffix of the checkpoint name",
        "logging": "(Bool) Whether to log results at each iteration/epoch (it requires additional custom logging class)"
    }
    }
} 

Inference configuration

The configurations used in a custom inference model are defined in this file. The EMCqMRI automatically connects the JSON configuration to a variable in the Signal, Inference and Likelihood custom modules:

{ "Inference-configuration-file": {
    "inferenceModel":{
            "inferenceSteps": 10
        },

    "signalModel":{
            "measurementUnit": "ms"
        },

    "likelihoodModel":{
            "reduction": "mean"
        }
}
}

The value set for inferenceSteps can be assessed through the global configuration object:

from core.base import base_inference_model

class MyInferenceModel(base_inference_model.InferenceModel, object):
    """
        Class implementing the inference model for estimation of parameters.
    """
    def __init__(self, configObject):
        super(MyInferenceModel, self).__init__()
        self.__name__ = 'MyInferenceModel'
        self.args = configObject.args

    def forward(self, input_signal):
        # Variable from configuration file
        inference_steps = self.args.inference.inferenceSteps
        ...

Variables in Signal and Likelihood models are accessed in a similar way:

from EMCqMRI.core.base import base_signal_model


class MySignalModel(base_signal_model.SignalModel):
    """
        Class implementing the forward model of a parametric signal model.
    """
    def __init__(self, configObject):
        super(MySignalModel, self).__init__()
        self.__name__ = 'MySignalModel'
        self.args = configObject.args
        self.independent_var = []

        # Variable from configuration file
        self.unit_of_measurement = self.args.inference.measurementUnit

    def forward(self, dependent_var, *extra_args):
        ...

Dataloader configuration

The dataloader configuration file is used to define the variables used in a custom dataloader module. In a similar way as before, we first create the configuration file:

{ "dataloader-configuration-file": {
    "data":{
            "applyNoise": true,
            "noiseSTD": 0.01
        }
}
}

The variables are accessed as follows:

from EMCqMRI.core.base.base_dataset import Dataset


class MyDatasetModel(Dataset):
    def __init__(self, configObject):
        super(MyDatasetModel, self).__init__()
        self.__name__ = 'MyDatasetModel'
        self.args = configObject.args

    def get_signal(self, idx):
        ...
        if self.args.dataset.applyNoise:
            noisy_data = self.args.engine.signal_model.applyNoise(data, self.args.dataset.noiseSTD)
        ...
        

The EMCqMRI Configuration Object

The EMCqMRI engine parses all configuration options from the .TXT files into a global object, which is passed to all standard and custom modules linked to the EMCqMRI. Additionally, all loaded modules (e.g. Signal Model) are made available through the Configuration Object.

Alongside all configuration options (”engine_configuration.txt”, “inference_configuration.txt”, etc.), below you find a list of the variables in the Configuration Object the user has access to:

signal_model : Loaded Signal Model class
inference_model : Loaded Inference Model class
likelihood_model : Loaded Likelihood Model class
dataset_model : Loaded Dataloader Model class
device : Indentified device (e.g. ‘cpu’, ‘cuda:0’)
optimizer : Contains the optimizer used to train the model
objective_fun : Object containing standard (e.g. ‘MSE’) or custom loss function.
log_training_fun : Object containing the logging function (Only if custom class is provided)
prepare_batch : Contains a function to prepare the batched data (after the dataloader)

Linking and overriding configurations via command line

The EMCqMRI also offers the ability to override the configuration options via the command line, when executing the train.py or estimate.py. This functionallity is implemented for all required configuration options in the Engine Configuration file. For example, to override the value of a configuration option, you can use:

Python3 train.py --configurationFile custom_configuration/engine_configuration.txt -inferenceModel custom_inference_model -learningRate 0.01

However, additional Python files are required to enable this functionality for the Inference and Dataloader Configuration files. For example, to be able to override the dataloader configurations “applyNoise” and “noiseSTD”, the Python file below must be created:

from EMCqMRI.core.utilities import configuration_utilities


class Configuration(object):
    def __init__(self, configObject):
        self.configObject = configObject
        self.required = False

    def parse_configuration(self):
        group_task = self.configObject.parser.add_argument_group('dataloader_config')

        group_task.add_argument('-applyNoise', required=self.required, type=configuration_utilities.str2bool, help='If true, applies noise to the weighted images')
        group_task.add_argument('-noiseSTD', required=self.required, type=float, help='Set the standard deviation of the noise.')
                
        config_args, _ = self.configObject.parser.parse_known_args()
        configuration = configuration_utilities.convert_argparse_to_attr(config_args)
        return configuration

The only required modifications to this file are the arguments added to ‘group_task’, which must match the configurations in the .TXT file. Finally, you must save the Python file in the same folder as the corresponding configuration file and with similar name (e.g. ‘./configuration/data_configuration.txt’, ‘./configuration/data_configuration.py’) Now, you can override the dataloader configurations:

Python3 train.py --configurationFile custom_configuration/engine_configuration.txt -inferenceModel custom_inference_model -learningRate 0.01 -applyNoise true -noiseSTD 1.0