More Complex Functions#

If our function does not have a single maximum/minimum it makes the search more difficult, and is probably more true to life than our simple functions so far. This is when it is necessary to use the assistance of scipy.

Start with a simple minimum \((5.0 + x)^2\) use a bit of overkill with the brent method.

../_images/convex_plot.png

Convex function, has a minimum#

The result is the red point

The result of minimize_scalar can show whether the outcome was successful or not success, the evaluation fun at the output x, in a total number of evaluations nfev in the number of iterations nit.

A graph is plotted which shows the minimum.

Show/Hide Code convex_function.py

# https://machinelearningmastery.com/univariate-function-optimization-in-python/

# optimize convex objective function
from numpy import arange
from scipy.optimize import minimize_scalar
import matplotlib.pyplot as plt
import seaborn as sns

sns.set_theme()

# objective function
def objective(x):
 return (5.0 + x)**2.0

# minimize the function
result = minimize_scalar(objective, method='brent')
# summarize the result
opt_x, opt_y = result['x'], result['fun']
print('Optimal Input x: %.6f' % opt_x)
print('Optimal Output f(x): %.6f' % opt_y)
print('Total Evaluations n: %d' % result['nfev'])
# define the range
r_min, r_max = -10.0, 10.0
# prepare inputs
inputs = arange(r_min, r_max, 0.1)
# compute targets
targets = [objective(x) for x in inputs]
# plot inputs vs target
plt.plot(inputs, targets, '--', label='function')
# plot the optima
plt.plot([opt_x], [opt_y], 's', color='r', label='result')

plt.title('Convex Function '+ r'$(5.0 + x)^2$')
plt.xlabel(r'$x$')
plt.ylabel(r'$F(x)$')
plt.legend()
# show the plot
plt.show()
#plt.savefig('../../figures/convex_plot.png')

Multi-Extrema#

The next example has a local and a global minimum, separated by a local maximum. Using the brent method in scipy as before the script found the global minimum using no additional conditions, even though no limits were set.

../_images/non_convex_plot.png

Non-Convex univariate function, has more than one extrema#

The result is the red point

Show/Hide Code brent_non_convex.py

# optimize non-convex objective function
from numpy import arange
from scipy.optimize import minimize_scalar
import matplotlib.pyplot as plt
import seaborn as sns

sns.set_theme()

# objective function
def objective(x):
 return (x - 2.0) * x * (x + 2.0)**2.0

# minimize the function
result = minimize_scalar(objective, method='brent')
# summarize the result
opt_x, opt_y = result['x'], result['fun']
print('Optimal Input x: %.6f' % opt_x)
print('Optimal Output f(x): %.6f' % opt_y)
print('Total Evaluations n: %d' % result['nfev'])
# define the range
r_min, r_max = -3.0, 2.5
# prepare inputs
inputs = arange(r_min, r_max, 0.1)
# compute targets
targets = [objective(x) for x in inputs]
# plot inputs vs target
plt.plot(inputs, targets, '--', label='function')
# plot the optima
plt.plot([opt_x], [opt_y], 's', color='r', label='result')

plt.title('Non-Convex Univariate Function \n $x(x - 2.0)(x+2.0)^2$')
plt.xlabel(r'$x$')
plt.ylabel(r'$F(x)$')
plt.legend()
# show the plot
#plt.show()
plt.savefig('../../figures/non_convex_plot.png')
../_images/non_convex_equal.png

Non-Convex univariate with equal extrema#

The results are the red and magenta points

In the special case where both minima are equal the optimize function may require help:

.......
def objective(x):
    return x ** 4 - x ** 2
......
result = minimize_scalar(objective, method='brent')
....

returns x: 0.707107, changing the minimize_scalar call to:

result = minimize_scalar(objective, method='brent', bracket(-1,0))

does not change the result, so choose the different method 'bounded':

result = minimize_scalar(objective, method='bounded', bounds=(-1,0))

which requires bounds rather than bracket. This returns x: -0.707107.

Show/Hide Code non_convex_univariate_equal.py

# https://machinelearningmastery.com/univariate-function-optimization-in-python/
# https://realpython.com/python-scipy-cluster-optimize/

# optimize non-convex objective function
from numpy import arange
from scipy.optimize import minimize_scalar
import matplotlib.pyplot as plt
import seaborn as sns

sns.set_theme()

# objective function
def objective(x):
	return x ** 4 - x ** 2

# minimize the function
result = minimize_scalar(objective, method='bounded', bounds=(-1,0))
# summarize the result
opt_x, opt_y = result['x'], result['fun']
print('Optimal Input x: %.6f' % opt_x)
print('Optimal Output f(x): %.6f' % opt_y)
print('Total Evaluations n: %d' % result['nfev'])
# define the range
r_min, r_max = -1.5, 1.5
# prepare inputs
inputs = arange(r_min, r_max, 0.1)
# compute targets
targets = [objective(x) for x in inputs]
# plot inputs vs target
plt.plot(inputs, targets, '--', label='function')
# plot the optima
plt.plot([opt_x], [opt_y], 's', color='r', label='result')
plt.plot(0.707, -0.25, 's', color='m', label='alternative')

plt.title('Non-Convex Univariate Function \n$x^4-x^2$')
plt.xlabel(r'$x$')
plt.ylabel(r'$F(x)$')
plt.legend()
# show the plot
plt.show()
#plt.savefig('../../figures/non_convex_equal.png')

Scipy optimize is a far ranging library that is best read to ascertain its scope . As seen scipy can be also used for root finding, but the methods used would not have been explained so easily.

On this page

This Page