Practical Problems, Math and AI
At the end of the first post I wrote that next we’d be learning a basic neural network, but later I realized that there should come another section, which discusses artificial intelligence from the mathematical aspect. So that is the subject of this small section, which will be followed by a post about perceptrons.
The ability to formulate a problem in the form of equations in our head, on a piece of paper, or in the computer, allows us to seek solutions and translate them to actions in the real world. If you throw a ball in vacuum there is a closed-form solution:
that allows you to anticipate the position of the ball at any moment in the future, as a function of the initial conditions, and take actions based on this knowledge.
Many common practical cases, are problems that are described by well known equations, but have no closed-form solution like x(t) = f(initial conditions), because the evolution of the system with time and the interaction between its components are just too complicated to solve or because those equations depend on data that is partially unknown. To name a few examples:
- Let’s say you want to launch a satellite in orbit around the Earth and you want to hard-code the missile’s steering commands before the launch. Theoretically you could do it, because the motion of your missile is described by well-known equations of classical mechanics, the thermodynamics of the expanding propellant and aerodynamics. But in practice we don’t know how to fully express the missile’s position by just using the initial conditions. This is the reason we give it a programmed trajectory, but also equip it with expensive sensors (IMU — Inertial Navigation Unit and other types) to have accurate measurements which the vehicle can respond to by steering.
- Determining the quality of an antenna just by the shape of it and the materials it’s made of is not an easy task to do just by solving a set of equations, even though electromagnetic theory is very well formulated. When the shape is irregular very quickly the equations become impossible to solve exactly and we need to either approximate parts of the antenna as “blocks” interacting with each other, or simulate a wave passing through the antenna, solve for a specific moment in time and incrementally advance time and solve the equations again. — A harder version of the problem — which brings us to the third type of problems, is the case where you have a set of constraints for the antenna — let’s say — performance at a given range of frequencies, dimensions, material etc., and you need to come up with a design. Can you formulate a thought process?
Yet a third realm of practical problems, is those that we know can be represented as some set of equations, and that can even be solved (because we keep solving them without thinking about it), but have no idea what those equations are.
- Determining whether the face in a photo belongs to your friend, or whether that object in front of you is a horse,
- Identifying the hand-written digits in a postal ZIP code for sorting,
- Finding the best translation of a paragraph from English to French
- Formulating the rules that enable a bipedal robot to walk. The exact motion and response required at every given second given the exact posture, velocity, and terrain, and maintaining balance while doing it, is a hard problem to formulate! And we humans do it innately!
- Designing a new antenna given a set of constraints.
I should say that the successful handling of tasks such as these is not restricted to humans. Every living organism reacts to sensory stimuli it receives from its environment and the pattern of stimulus-response quickly becomes too complex for us to fully formulate into equations.
Enter AI.
The essence of artificial intelligence algorithms is the ability to find — and then solve — an approximate equation for those problems to which we can’t formulate an equation. In many AI approaches, we construct an objective function (another name for it is a loss function), which is minimized when our AI algorithm performs well, and the goal of the training phase is to find the global minimum of that objective function, without solving explicitly the original set of equations of the problem, since it is (and will remain) unknown.
Thanks to the computational power of modern computers, we can construct models that are rich enough in parameters (some computer vision neural networks have billions of parameters), that they can approximate extremely complicated equations, that is — detect relations between different features in the input, that have an effect on the output. In neural networks, those relations are established through the training phase of the AI model. The training stage is designed in a way that will give the AI model a positive feedback when it performs correctly, and a negative feedback when it makes a mistake, and that feedback results in a change in the model parameters pushing it to perform better in the next iteration. An AI model is said to have trained well if it is able to generalize — that is — to correctly solve not only the very same cases that were presented to it during training, but also other cases it never encountered before. When it does this, it means it actually found profound relations between different features in the data, that are essential to the (unknown) exact function describing the problem.
