Library

programming-1873854_1920

pc-1768018_1920

McTorch

Manifold optimization library for deep learning

The software is available on GitHub.

McTorch is a python-based manifold optimization library for deep learning, which adds manifold optimization functionality to PyTorch. McTorch builds on top of PyTorch and supports all PyTorch functions in addition to manifold optimization. This is done to ensure researchers and developers using PyTorch can easily experiment with McTorch functions without writing a single line of extra code. It supports several manifolds including the Stiefel manifold, the positive definite manifold, and the Hyperbolic manifold. The optimizer can be selected among SGD, Adagrad, and conjugate gradient (CG).

SGDLibrary

Stochastic optimization algorithm library

The software is available on GitHub.

The SGDLibrary is a pure-MATLAB library of a collection of stochastic optimization algorithms. Let $f: \mathbb{R}^d \rightarrow \mathbb{R}$ be a smooth real-valued function on $w \in \mathbb{R}^d$ , where $d$ is the dimension. This solves an unconstrained minimization problem of the form,

$\displaystyle{\min_{w \in \mathbb{R}^d} f(w) := \frac{1}{n} \sum_{i=1}^n f_i(w)} = \frac{1}{n} \sum_{i=1}^n \ \underbrace{L(w, x_i, y_i)}_{\text{\scriptsize{loss function}}} \ + \ \lambda \underbrace{R(w)}_{\text{\scriptsize{regularizer}}}.$

See the detailed explanation of stochastic optimization here.

Supported stochastic optimization algorithms

Here is the list of the supported optimization algorithms.

An example of sample codes

Results of cost, optimality gap, and classification for SGD, SVRG, Adam, and IQN
in L2-norm regularized softmax classification problem.

The SGDLibrary is also operable on GNU Octave (Free software compatible with many MATLAB scripts). Note that this SGDLibrary internally contains the GDLibrary.

NMFLibrary

The software is available on GitHub.

The NMFLibrary is a pure-MATLAB library of a collection of non-negative matrix factorization algorithms (NMF). Nonnegative matrix factorization (NMF) is a fundamental problem for discovering nonnegative latent factors and/or performing dimensionality reduction. NMF has been successfully applied in diverse technical fields, such as pattern recognition, image/video analysis, text mining, bioinformatics, and Web analysis. The non-negativity of the obtained factors gives understandable interpretations of the data of interest. NMF approximates a nonnegative matrix ${\bf V}$ as a product of two nonnegative matrices ${\bf W}$ and ${\bf H}$ . More concretely, given ${\bf V}\in \mathbb{R}^{F \times N}_{+}$ , we calculate ${\bf W}\in \mathbb{R}^{F \times K}_{+}$ and ${\bf H}\in \mathbb{R}^{K \times N}_{+}$ with respect to the Euclidean distance as

$\displaystyle{\min_{\scriptsize {\bf W}, {\bf H}} \ \ \ \frac{1}{2} \| {\bf V} - {\bf WH}\|_F^2},$

$\displaystyle{{\rm subject\ to} \ \ \ [{\bf W}]_{f,k} \geq 0, [{\bf H}]_{k,n} \geq 0, \ \ \ \ \ \forall f,n,k},$

where $K$ is usually chosen such that $K \ll \min\{F, N\}$ , that is, ${\bf V}$ is approximately represented by the two low-rank matrices.

ClassifierToolbox

The software is available on GitHub.

The ClassifierToolbox is a pure-MATLAB toolbox for classifiers. This package provides various tools for classification, e.g., image classification, face recognition, and related applications.

SparseGDLibrary

The software is available on GitHub.

The SparseGDLibrary is a pure-MATLAB library of a collection of unconstrained optimization algorithms for sparse modeling. This package includes various solvers such as

APG (Accelerated gradient descent, i.e., Nesterov AGD)
ISTA (Iterative shrinkage-thresholding algorithm)
FISTA (Fast iterative shrinkage-thresholding algorithm)
CD (Coordinate descent) for Lasso and Elastic Net
ADMM (The alternating direction method of multipliers) for Lasso.

RSOpt

Riemannian stochastic optimization library

The software is available on GitHub.

Let $f: \mathcal{M} \rightarrow \mathbb{R}$ be a smooth real-valued function on a Riemannian manifold $\mathcal{M}$ . The target problem concerns a given model variable $w$ on $\mathcal{M}$ , and is expressed as

$\displaystyle{\min_{w \in \mathcal{M}} f(w) := \frac{1}{n} \sum_{i=1}^n f_i(w)},$

where $n$ is the total number of elements. This RSOpt package provides the MATLAB implementation codes dedicated to those stochastic algorithms above.

Note that various manifold algorithms on various manifolds are implemented in MATLAB toolbox Manopt. The RSOpt codes are compliant with Manopt. Also, please see here for a more comprehensive explanation of optimization algorithms on matrix manifolds.

See additional detailed explanations here.