.. DO NOT EDIT. .. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY. .. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE: .. "auto_ch7\ch7_AAD.py" .. LINE NUMBERS ARE GIVEN BELOW. .. only:: html .. note:: :class: sphx-glr-download-link-note :ref:`Go to the end ` to download the full example code. .. rst-class:: sphx-glr-example-title .. _sphx_glr_auto_ch7_ch7_AAD.py: ==================================== 7.06 Automatic Differentiation ==================================== We reproduce here the figures 7.9, 7.10 & 7.11 of the book. Utilitary functions can be found next to this file. Here, we only define codpy-related functions. .. GENERATED FROM PYTHON SOURCE LINES 10-12 Necessary Imports ------------------------ .. GENERATED FROM PYTHON SOURCE LINES 12-28 .. code-block:: Python import os import sys import matplotlib.pyplot as plt import torch try: CURRENT_DIR = os.path.dirname(os.path.abspath(__file__)) except NameError: CURRENT_DIR = os.getcwd() data_path = os.path.join(CURRENT_DIR, "data") PARENT_DIR = os.path.abspath(os.path.join(CURRENT_DIR, "..")) sys.path.insert(0, PARENT_DIR) from utils.ch7.ch7_utils import testAAD, differentialMlBenchmarks, taylor_test, get_param21,get_param22 .. GENERATED FROM PYTHON SOURCE LINES 29-32 AAD ------------------------ The first figure displays the computation of first- and second-order derivatives of a function $f(X)=\frac{1}{6}X^3$ using AAD. .. GENERATED FROM PYTHON SOURCE LINES 32-35 .. code-block:: Python testAAD(lambda x: 1/6 * x**3, torch.randn((100,1), requires_grad=True)) plt.show() .. image-sg:: /auto_ch7/images/sphx_glr_ch7_AAD_001.png :alt: cubic, 1st derivative, 2nd derivative :srcset: /auto_ch7/images/sphx_glr_ch7_AAD_001.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 36-41 1 dimensional differential machines ------------------------------------------------ Here, we illustrate a general multi-dimensional benchmark of two differential machines methods. The first one uses the kernel gradient operator. The second one uses a neural network defined with Pytorch together with AAD tools. .. GENERATED FROM PYTHON SOURCE LINES 41-44 .. code-block:: Python differentialMlBenchmarks(D=1,N=500) plt.show() .. image-sg:: /auto_ch7/images/sphx_glr_ch7_AAD_002.png :alt: training set, ground truth values, Pytorch f, Codpy f, exact grad, codpy grad, pytorch grad-1, pytorch grad-2 :srcset: /auto_ch7/images/sphx_glr_ch7_AAD_002.png :class: sphx-glr-single-img .. rst-class:: sphx-glr-script-out .. code-block:: none Sequential( (0): Linear(in_features=1, out_features=128, bias=True) (1): ReLU() (2): Dropout(p=0.0, inplace=False) (3): Linear(in_features=128, out_features=128, bias=True) (4): ReLU() (5): Dropout(p=0.0, inplace=False) (6): Linear(in_features=128, out_features=128, bias=True) (7): ReLU() (8): Dropout(p=0.0, inplace=False) (9): Linear(in_features=128, out_features=128, bias=True) (10): ReLU() (11): Dropout(p=0.0, inplace=False) (12): Linear(in_features=128, out_features=1, bias=True) ) 0/500 loss 0.83568 torch nabla: 1.3428218364715576 Sequential( (0): Linear(in_features=1, out_features=128, bias=True) (1): ReLU() (2): Dropout(p=0.0, inplace=False) (3): Linear(in_features=128, out_features=128, bias=True) (4): ReLU() (5): Dropout(p=0.0, inplace=False) (6): Linear(in_features=128, out_features=128, bias=True) (7): ReLU() (8): Dropout(p=0.0, inplace=False) (9): Linear(in_features=128, out_features=128, bias=True) (10): ReLU() (11): Dropout(p=0.0, inplace=False) (12): Linear(in_features=128, out_features=1, bias=True) ) 0/500 loss 0.84137 torch nabla: 1.2531850337982178 Sequential( (0): Linear(in_features=1, out_features=128, bias=True) (1): ReLU() (2): Dropout(p=0.0, inplace=False) (3): Linear(in_features=128, out_features=128, bias=True) (4): ReLU() (5): Dropout(p=0.0, inplace=False) (6): Linear(in_features=128, out_features=128, bias=True) (7): ReLU() (8): Dropout(p=0.0, inplace=False) (9): Linear(in_features=128, out_features=128, bias=True) (10): ReLU() (11): Dropout(p=0.0, inplace=False) (12): Linear(in_features=128, out_features=1, bias=True) ) 0/500 loss 0.83807 torch projection: 1.1627514362335205 codpy nabla: 0.07037615776062012 codpy projection: 0.0065724849700927734 RMSE delta codpy: 0.7808704291070304 RMSE delta pytorch: 0.47247325248239247 .. GENERATED FROM PYTHON SOURCE LINES 45-48 2 dimensional differential machines ------------------------------------------------ We perform the same benchmark in 2 dimensions below: .. GENERATED FROM PYTHON SOURCE LINES 48-51 .. code-block:: Python differentialMlBenchmarks(D=2,N=500) plt.show() .. image-sg:: /auto_ch7/images/sphx_glr_ch7_AAD_003.png :alt: training set, ground truth values, Pytorch f, Codpy f, exact grad, codpy grad, pytorch grad-1, pytorch grad-2 :srcset: /auto_ch7/images/sphx_glr_ch7_AAD_003.png :class: sphx-glr-single-img .. rst-class:: sphx-glr-script-out .. code-block:: none Sequential( (0): Linear(in_features=2, out_features=128, bias=True) (1): ReLU() (2): Dropout(p=0.0, inplace=False) (3): Linear(in_features=128, out_features=128, bias=True) (4): ReLU() (5): Dropout(p=0.0, inplace=False) (6): Linear(in_features=128, out_features=128, bias=True) (7): ReLU() (8): Dropout(p=0.0, inplace=False) (9): Linear(in_features=128, out_features=128, bias=True) (10): ReLU() (11): Dropout(p=0.0, inplace=False) (12): Linear(in_features=128, out_features=1, bias=True) ) 0/500 loss 1.00956 torch nabla: 1.116265058517456 Sequential( (0): Linear(in_features=2, out_features=128, bias=True) (1): ReLU() (2): Dropout(p=0.0, inplace=False) (3): Linear(in_features=128, out_features=128, bias=True) (4): ReLU() (5): Dropout(p=0.0, inplace=False) (6): Linear(in_features=128, out_features=128, bias=True) (7): ReLU() (8): Dropout(p=0.0, inplace=False) (9): Linear(in_features=128, out_features=128, bias=True) (10): ReLU() (11): Dropout(p=0.0, inplace=False) (12): Linear(in_features=128, out_features=1, bias=True) ) 0/500 loss 0.99518 torch nabla: 1.1354081630706787 Sequential( (0): Linear(in_features=2, out_features=128, bias=True) (1): ReLU() (2): Dropout(p=0.0, inplace=False) (3): Linear(in_features=128, out_features=128, bias=True) (4): ReLU() (5): Dropout(p=0.0, inplace=False) (6): Linear(in_features=128, out_features=128, bias=True) (7): ReLU() (8): Dropout(p=0.0, inplace=False) (9): Linear(in_features=128, out_features=128, bias=True) (10): ReLU() (11): Dropout(p=0.0, inplace=False) (12): Linear(in_features=128, out_features=1, bias=True) ) 0/500 loss 0.99291 torch projection: 1.1650512218475342 codpy nabla: 0.1067359447479248 codpy projection: 0.009524106979370117 RMSE delta codpy: 0.7197046151870157 RMSE delta pytorch: 0.3861852949724002 .. GENERATED FROM PYTHON SOURCE LINES 52-56 Taylor expansions and differential machines ------------------------------------------------ We approximate up to second order $\nabla f(x)$,$\nabla^2 f(x)$ with $$\nabla f_x = \nabla_Z \mathcal{P}_m\big(X,Y,Z=x,f(X)\big), \nabla^2 f_x = \nabla^2_Z \mathcal{P}_m\big(X,Y,Z=x,f(X)\big)$$ .. GENERATED FROM PYTHON SOURCE LINES 56-58 .. code-block:: Python taylor_test(**get_param22(),taylor_order = 2) plt.show() .. image-sg:: /auto_ch7/images/sphx_glr_ch7_AAD_004.png :alt: z, fz (AAD ord.)2, Codpy ord.2, Pytorch ord.2 :srcset: /auto_ch7/images/sphx_glr_ch7_AAD_004.png :class: sphx-glr-single-img .. rst-class:: sphx-glr-script-out .. code-block:: none Sequential( (0): Linear(in_features=1, out_features=128, bias=True) (1): ReLU() (2): Dropout(p=0.0, inplace=False) (3): Linear(in_features=128, out_features=128, bias=True) (4): ReLU() (5): Dropout(p=0.0, inplace=False) (6): Linear(in_features=128, out_features=128, bias=True) (7): ReLU() (8): Dropout(p=0.0, inplace=False) (9): Linear(in_features=128, out_features=128, bias=True) (10): ReLU() (11): Dropout(p=0.0, inplace=False) (12): Linear(in_features=128, out_features=1, bias=True) ) 0/500 loss 0.82311 Sequential( (0): Linear(in_features=1, out_features=128, bias=True) (1): ReLU() (2): Dropout(p=0.0, inplace=False) (3): Linear(in_features=128, out_features=64, bias=True) (4): ReLU() (5): Dropout(p=0.0, inplace=False) (6): Linear(in_features=64, out_features=1, bias=True) ) 0/500 loss 0.84781 .. rst-class:: sphx-glr-timing **Total running time of the script:** (0 minutes 10.574 seconds) .. _sphx_glr_download_auto_ch7_ch7_AAD.py: .. only:: html .. container:: sphx-glr-footer sphx-glr-footer-example .. container:: sphx-glr-download sphx-glr-download-jupyter :download:`Download Jupyter notebook: ch7_AAD.ipynb ` .. container:: sphx-glr-download sphx-glr-download-python :download:`Download Python source code: ch7_AAD.py ` .. container:: sphx-glr-download sphx-glr-download-zip :download:`Download zipped: ch7_AAD.zip ` .. only:: html .. rst-class:: sphx-glr-signature `Gallery generated by Sphinx-Gallery `_