Library

previous arrow
next arrow
Slider

McTorch

Manifold optimization library for deep learning

The software is available at Github.

McTorch is a python-based manifold optimization library for deep learning, which adds manifold optimization functionality to PyTorch. McTorch builds on top of PyTorch and supports all PyTorch functions in addition to manifold optimization. This is done to ensure researchers and developers using PyTorch can easily experiment with McTorch functions without writing a single line of extra code. It supports several manifolds including the Stiefel manifold, the positive definite manifold, and the Hyperbolic manifold. The optimizer can be selected among SGD, Adagrad and conjugate gradient (CG).

SGDLibrary

Stochastic optimization algorithm library

The software is available at Github.

The SGDLibrary is a pure-MATLAB library of a collection of stochastic optimization algorithms. Let f: \mathbb{R}^d \rightarrow \mathbb{R} be a smooth real-valued function on  w \in \mathbb{R}^d, where d is the dimension. This solves an unconstrained minimization problem of the form,

 \displaystyle{\min_{w \in \mathbb{R}^d} f(w) := \frac{1}{n} \sum_{i=1}^n f_i(w)} = \frac{1}{n} \sum_{i=1}^n \ \underbrace{L(w, x_i, y_i)}_{\text{\scriptsize{loss function}}}  \  + \ \lambda \underbrace{R(w)}_{\text{\scriptsize{regularizer}}}.

See the detailed explanation of the stochastic optimization here.

Supported stochastic optimization algorithms

Here is the list of the supported optimization algorithms.

Supported stochastic optimization algorithms.

An example of sample codes


Results of of cost, optimality gap, and classification for SGD, SVRG, Adam, and IQN
in L2-norm regularized softmax classification problem.

The SGDLibrary is also operable on GNU Octave (Free software compatible with many MATLAB scripts). Note that this SGDLibrary internally contains the GDLibrary.

NMFLibrary

The software is available at Github.

The NMFLibrary is a pure-MATLAB library of a collection of algorithms of non-negative matrix factorization (NMF). Nonnegative matrix factorization (NMF) is a fundamental problem for discovering nonnegative latent factors and/or performing dimensionality reduction. NMF has been successfully applied in diverse technical fields, such as pattern recognition, image/video analysis, text mining, bioinformatics and Web analysis because non-negativity of the obtained factors  gives understandable interpretations of data of interest. NMF approximates a nonnegative matrix {\bf V} as a product of two nonnegative matrices {\bf W} and {\bf H}. More concretely, given {\bf V}\in \mathbb{R}^{F \times N}_{+}, we calculate {\bf W}\in \mathbb{R}^{F \times K}_{+} and {\bf H}\in \mathbb{R}^{K \times N}_{+} with respect to the Euclidean distance as

 \displaystyle{\min_{\scriptsize {\bf W}, {\bf H}} \ \ \ \frac{1}{2} \| {\bf V} - {\bf WH}\|_F^2},

\displaystyle{{\rm subject\ to} \ \ \ [{\bf W}]_{f,k} \geq 0, [{\bf H}]_{k,n} \geq 0, \ \ \ \ \  \forall f,n,k},

 where K is usually chosen such that K \ll \min\{F, N\}, that is, {\bf V} is approximately represented by the two low-rank matrices.

ClassifierToolbox

The software is available at Github.

The ClassifierToolbox is a pure-MATLAB toolbox for classifier. This package provides various tools for classification, e.g., image classification, face recognition, and related applications.

SparseGDLibrary

The software is available at Github.

The SparseGDLibrary is a pure-MATLAB library of a collection of unconstrained optimization algorithms for sparse modeling. This package includes various solvers such as

  • APG (Accelerated gradient descent, i.e., Nesterov AGD)
  • ISTA (Iterative shrinkage-thresholding algorithm)
  • FISTA (Fast iterative shrinkage-thresholding algorithm)
  • CD (Coordinate descent) for Lasso and Elastic Net
  • ADMM (The alternating direction method of multipliers) for Lasso.

RSOpt

Riemannian stochastic optimization library

The software is available at Github.

Let f: \mathcal{M} \rightarrow \mathbb{R} be a smooth real-valued function on a Riemannian manifold \mathcal{M}. The target problem concerns a given model variable w on \mathcal{M}, and is expressed as

 \displaystyle{\min_{w \in \mathcal{M}} f(w) := \frac{1}{n} \sum_{i=1}^n f_i(w)},

where n is the total number of the elements.This RSOpt package provides the MATLAB implementation codes dedicated to those stochastic algorithms above.

Note that various manifold algorithms on various manifolds are implemented in MATLAB toolbox Manopt. The RSOpt codes are compliant to Manopt. Also, please see here for more comprehensive explanation of optimization algorithms on matrix manifolds.

See additional detailed explanations here.