Insight 3: Deductive reasoning is preferable to inductive reasoning
TLDR: The simplest solution should be tried first, because it will set a baseline and it is often the optimal solution.
In our third insight “Deduction is preferable to induction” we argue why data scientists should favour a top-down approach to a bottom-up approach and strive to develop data independent solutions when possible.
A primer on the philosophy of logic
Deductive reasoning, or the top-down approach, start with premises and generalizes over principles to reason about specific instances. As long as the premises hold, the conclusion is true.
Given the premise “Socrates is a man” and the principle “All men are mortal”, we can deduct the knowledge “Socrates is mortal”.
Inductive reasoning, or the bottom-up approach, start with observations of specific instances and uses the information to construct general principles. If the observations are representative of the general case, the conclusion is true.
If we have observed thousands of swans, and every single one of them was white, we can construct the hypothesis that “All swans are white”.
In the context of software
Most problems in software are tackled with a deductive approach. Our code represents general rules so that they can tackle specific cases.
For example, y = 2 * x is a general rule that can tackle all kinds of specific instances of x.
It’s only recently that machine learning has paved the way for a set of induction-based solutions. While statisticians have used the bottom-up approach for ages, machine learning practitioner’s pragmatic, non-mathematically rigorous, big-data approach has become a whole new paradigm.
Instead of writing y = 2 * x, we provide the computer with specific examples (e.g. “If x is 3 then y is 6”) and use machine learning algorithms to construct models that are generally applicable.
The software 2.0 stack
Andrej Karpathy, the Director of AI at Tesla, introduced the term Software Stack 2.0 to describe what we like to call “inductive programming”. We prefer the term “inductive programming”, because unlike “software 2.0”, it does not imply that this type of programming is superior to deductive programming (aka. “software 1.0”).
We believe the “software 1.0 and 2.0” terminology is the product of the recent hype surrounding AI. The choice between deductive and inductive programming should depend on the problem at hand and the available tools – one is not superior to the other.
The reason for the boom in induction-based programming is not that it is inherently better than the deductive approach, but rather that it yields great performance in domains where computers used to lack behind humans. Visual, auditory and textual recognition used to be human-only domains, but induction-based methods have made computers catch up quickly.
As a data scientist, it is easy to forget, but important to remember, that induction-based approaches have several inherent disadvantages. By neglecting to acknowledge these limitations, we risk using induction-based approaches when a deductive approach is the better choice.
The disadvantages include:
- Collecting enough data is time-consuming and cumbersome.
- Making sure that the data is representative is difficult.
- Managing the collected data takes time and resources.
- Reasoning based on the data is difficult.
- Computing the solution is more computationally expensive.
- The outputted results always have a degree of uncertainty.
- Reconfiguring the problem based on new knowledge is cumbersome.
- Since you are not designing the rules, you miss out on insights otherwise gained from iteratively working on the problem.
Because of these reasons, deductive solutions are almost always easier to implement than their inductive counterparts. Building inductive solutions might get easier with the introduction of new tools, but the inherent problems of induction cannot be mitigated.
An incremental approach
Inexperienced data scientists will often approach problems with the most advanced tools available, neglecting deductive methods entirely. We encourage data scientists to adopt an incremental approach, moving from deductive solutions towards more inductive approaches. We believe that one of the key responsibilities of data scientists is to determine which solution of use for a given problem.
The image below has proven useful when explaining our approach to laymen. The explanation goes something like this, moving from left to right:
We start with plain old code (Hard-coded). When this becomes too inflexible, we let the user specify the rules themselves (Rule based). When this shows poor performance, or becomes too complex to maintain, we gather data and create a statistical model (Statistics). If the problem is too complex for common statistics, we sacrifice the mathematical rigor and focus solely on the performance of our models (shallow machine learning). If the problem at hand is very complex, we stack multiple models on top of each other (Deep Learning).
Deductive solutions should always serve as baselines for more sophisticated approaches. If induction-based approaches cannot beat the baseline, there is no reason to move on. It is often the case that the best solution to industry problems lie at the beginning of this process.
By using an incremental approach, we save time and avoid introducing unnecessary complexity to our systems. Even if inductive approaches yield better performance, deduction might be preferable due to the ease of interpretability and reconfiguration.
Do you think making a distinction between induction and deduction-based programming is sensible? Do you have any experiences with projects what failed to provide a deduction-based baseline? We look forward to your thoughts and insights.
To sum up
In our third insight “Deduction is preferable to induction” we argue for the importance of creating simple solutions before bringing out the big guns. Creating simple solutions have a number of desirable properties such as constituting a good baseline and ease of maintenance. In data science, the top-down approach is usually simpler than the bottom-up approach, thus engineers should strive to develop a data independent solution when possible.
In our fourth insight “Manipulate the problem-model-data-metric equation” we go into detail about what distinguishes pragmatic data science in the industry from academic and competitive data science.
Get ‘The Pragmatic Data Scientist’ as a whitepaper