# Kernel Ridge Regression¶

Kernel ridge regression is a non-parametric form of ridge regression. The aim is to learn a function in the space induced by the respective kernel $$k$$ by minimizing a squared loss with a squared norm regularization term.

The solution can be written in closed form as:

$\alpha = \left({\bf K}+\tau{\bf I}\right)^{-1}{\bf y}$

where $${\bf K}$$ is the kernel matrix and $$\alpha$$ is the vector of weights in the space induced by the kernel. The learned function can then be evaluated as $$f(x)=\sum_{i=1}^N\alpha_ik(x,x_i)$$.

See Chapter 17 in [Bar12] for a detailed introduction.

## Example¶

Imagine we have files with training and test data. We create CDenseFeatures (here 64 bit floats aka RealFeatures) and CRegressionLabels as

features_train = RealFeatures(f_feats_train)
features_test = RealFeatures(f_feats_test)
labels_train = RegressionLabels(f_labels_train)
labels_test = RegressionLabels(f_labels_test)

features_train = RealFeatures(f_feats_train);
features_test = RealFeatures(f_feats_test);
labels_train = RegressionLabels(f_labels_train);
labels_test = RegressionLabels(f_labels_test);

RealFeatures features_train = new RealFeatures(f_feats_train);
RealFeatures features_test = new RealFeatures(f_feats_test);
RegressionLabels labels_train = new RegressionLabels(f_labels_train);
RegressionLabels labels_test = new RegressionLabels(f_labels_test);

features_train = Shogun::RealFeatures.new f_feats_train
features_test = Shogun::RealFeatures.new f_feats_test
labels_train = Shogun::RegressionLabels.new f_labels_train
labels_test = Shogun::RegressionLabels.new f_labels_test

features_train <- RealFeatures(f_feats_train)
features_test <- RealFeatures(f_feats_test)
labels_train <- RegressionLabels(f_labels_train)
labels_test <- RegressionLabels(f_labels_test)

features_train = shogun.RealFeatures(f_feats_train)
features_test = shogun.RealFeatures(f_feats_test)
labels_train = shogun.RegressionLabels(f_labels_train)
labels_test = shogun.RegressionLabels(f_labels_test)

RealFeatures features_train = new RealFeatures(f_feats_train);
RealFeatures features_test = new RealFeatures(f_feats_test);
RegressionLabels labels_train = new RegressionLabels(f_labels_train);
RegressionLabels labels_test = new RegressionLabels(f_labels_test);

auto features_train = some<CDenseFeatures<float64_t>>(f_feats_train);
auto features_test = some<CDenseFeatures<float64_t>>(f_feats_test);
auto labels_train = some<CRegressionLabels>(f_labels_train);
auto labels_test = some<CRegressionLabels>(f_labels_test);


Choose an appropriate CKernel and instantiate it. Here we use a CGaussianKernel.

width = 1.0
kernel = GaussianKernel(features_train, features_train, width)

width = 1.0;
kernel = GaussianKernel(features_train, features_train, width);

double width = 1.0;
GaussianKernel kernel = new GaussianKernel(features_train, features_train, width);

width = 1.0
kernel = Shogun::GaussianKernel.new features_train, features_train, width

width <- 1.0
kernel <- GaussianKernel(features_train, features_train, width)

width = 1.0
kernel = shogun.GaussianKernel(features_train, features_train, width)

double width = 1.0;
GaussianKernel kernel = new GaussianKernel(features_train, features_train, width);

auto width = 1.0;
auto kernel = some<CGaussianKernel>(features_train, features_train, width);


We create an instance of CKernelRidgeRegression classifier by passing it $$\tau$$, the kernel and labels.

tau = 0.001
krr = KernelRidgeRegression(tau, kernel, labels_train)

tau = 0.001;
krr = KernelRidgeRegression(tau, kernel, labels_train);

double tau = 0.001;
KernelRidgeRegression krr = new KernelRidgeRegression(tau, kernel, labels_train);

tau = 0.001
krr = Shogun::KernelRidgeRegression.new tau, kernel, labels_train

tau <- 0.001
krr <- KernelRidgeRegression(tau, kernel, labels_train)

tau = 0.001
krr = shogun.KernelRidgeRegression(tau, kernel, labels_train)

double tau = 0.001;
KernelRidgeRegression krr = new KernelRidgeRegression(tau, kernel, labels_train);

auto tau = 0.001;
auto krr = some<CKernelRidgeRegression>(tau, kernel, labels_train);


Then we train the regression model and apply it to test data to get the predicted CRegressionLabels.

krr.train()
labels_predict = krr.apply_regression(features_test)

krr.train();
labels_predict = krr.apply_regression(features_test);

krr.train();
RegressionLabels labels_predict = krr.apply_regression(features_test);

krr.train
labels_predict = krr.apply_regression features_test

krr$train() labels_predict <- krr$apply_regression(features_test)

krr:train()
labels_predict = krr:apply_regression(features_test)

krr.train();
RegressionLabels labels_predict = krr.apply_regression(features_test);

krr->train();
auto labels_predict = krr->apply_regression(features_test);


After training, we can extract $$\alpha$$.

alpha = krr.get_alphas()

alpha = krr.get_alphas();

DoubleMatrix alpha = krr.get_alphas();

alpha = krr.get_alphas

alpha <- krr$get_alphas()  alpha = krr:get_alphas()  double[] alpha = krr.get_alphas();  auto alpha = krr->get_alphas();  Finally, we can evaluate the CMeanSquaredError. eval = MeanSquaredError() mse = eval.evaluate(labels_predict, labels_test)  eval = MeanSquaredError(); mse = eval.evaluate(labels_predict, labels_test);  MeanSquaredError eval = new MeanSquaredError(); double mse = eval.evaluate(labels_predict, labels_test);  eval = Shogun::MeanSquaredError.new mse = eval.evaluate labels_predict, labels_test  eval <- MeanSquaredError() mse <- eval$evaluate(labels_predict, labels_test)

eval = shogun.MeanSquaredError()
mse = eval:evaluate(labels_predict, labels_test)

MeanSquaredError eval = new MeanSquaredError();
double mse = eval.evaluate(labels_predict, labels_test);

auto eval = some<CMeanSquaredError>();
auto mse = eval->evaluate(labels_predict, labels_test);


## References¶

Wikipedia: Kernel_method

 [Bar12] D. Barber. Bayesian reasoning and machine learning. Cambridge University Press, 2012.