Imputations
A set of functions for imputing missing data provided as helpers.
This is a script containing functions that perform imputations of missing values in a dataset. The imputations are built off of the sklearn library and are designed to handle missing values in a dataset. For specific documentation, visit the sklearn documentation: https://scikit-learn.org/stable/modules/impute.html The following imputation methods are implemented:
- Mean imputation: missing values are replaced with the mean of the observed values for each feature.
- Median imputation: missing values are replaced with the median of the observed values for each feature.
- KNN imputation: missing values are replaced with the mean of the k-nearest neighbors for each feature.
- Regression imputation: missing values are replaced with predictions from a regression model trained on the observed values for each feature.
- PCA_imputation: missing values are replaced with predictions from a PCA model in an iterative manner.
Other imputation methods, along with cross-validation and hyperparameter tuning may be added in the future. All methods require the data to be a pandas DataFrame, and the output will also be a pandas DataFrame with the same index and columns as the input data.
mean_imputation(data)
Impute missing values with the mean of each feature.
median_imputation(data)
Impute missing values with the median of each feature.
knn_imputation(data, n_neighbors=5)
Impute missing values using k-nearest neighbors.
regression_imputation_feature(data, iter=False)
Impute missing values using regression models trained on all other features.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
data
|
DataFrame
|
The input dataset with missing values. |
required |
iter
|
bool
|
If True, perform iterative imputation until convergence or a maximum number of iterations is reached. If False, perform a single imputation step. |
False
|
Returns:
| Type | Description |
|---|---|
|
pd.DataFrame: The dataset with imputed values. |
regression_imputation_covariates(data, covariates, iter=False)
Impute missing values using regression models with covariates.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
data
|
DataFrame
|
The input dataset with missing values. |
required |
covariates
|
list
|
List of column names to be used as covariates for imputation. |
required |
Returns:
| Type | Description |
|---|---|
|
pd.DataFrame: The dataset with imputed values. |
nets_svds(X, n)
Python equivalent of MATLAB nets_svds from https://git.fmrib.ox.ac.uk/steve/fslnets_matlab This is included as a helper function for PCA_imputation and is not used elsewhere in the codebase.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
X
|
ndarray
|
Input data matrix with shape (m, n). |
required |
n
|
int
|
Number of components to compute. If n <= 0, compute rank+n components. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
U |
ndarray
|
Left singular vectors with shape (m, n). |
S |
ndarray
|
Singular values with shape (n, n). |
V |
ndarray
|
Right singular vectors with shape (p, n). |
PCA_imputation(impute_input, Npca, Nrand=0)
Impute missing values using PCA-based imputation. Based on original code in MATLAB by Stephen Smith, modified for python: https://git.fmrib.ox.ac.uk/steve/fslnets_matlab/
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
impute_input
|
array - like
|
Input data with missing values (NaNs). |
required |
Npca
|
int
|
Number of principal components to use for imputation. If Npca > 0, use Npca components. If Npca < 0, use -Npca components. |
required |
Nrand
|
int
|
Number of random initializations for imputation. If Nrand > 0, perform Nrand random initializations and average the results. If Nrand = 0, perform a single imputation. |
0
|
Returns:
| Name | Type | Description |
|---|---|---|
impute_replace |
ndarray
|
Imputed data with missing values replaced. |
impute_noreplace |
ndarray
|
Imputed data without replacing missing values. |
pcaU |
ndarray
|
Left singular vectors from PCA. |
pcaS |
ndarray
|
Singular values from PCA. |
pcaV |
ndarray
|
Right singular vectors from PCA. |