.svg){kind=icon}
.svg){kind=icon}
.svg){kind=icon}
Get your free ebook
One common challenge faced by businesses lies in distinguishing between research and engineering projects. Often, companies initiate a research project with the expectations typical of an engineering one. They assume that by hiring data scientists will bring them a fully functional model within a specified timeframe. However, when the model fails to meet expectations, the managers start to have doubts about the endeavor. They question whether the failure stems from the project's feasibility, the team's competence, or the model's suitability for the particular case.
Interestingly, neither managers nor engineers can pinpoint the root cause, as engineers often lack clarity themselves. They attempt to resolve the issue by implementing new solutions found online, aiming to demonstrate progress to the management while overlooking the fundamental distinction: they are engaged in research, not engineering. The intended outcome is not a functional model ready for production but rather a comprehensive report outlining what works and how it operates. Based on this information they can proceed to designing solution architecture and start an engineering project that aims to make this solution production-ready.
On the other hand, there can be an instance when a lot of engineering work needs to be done, but the project is treated as research. Such a situation can occur when the research that needs to be conducted is demanding in terms of resources, infrastructure, and the complexity of the problem. It is necessary to prepare appropriate research infrastructure to ensure results reproducibility, provide data quickly and of adequate quality, and allow for the rapid implementation of new ideas and their reliable testing. The preparation of such infrastructure usually is not a task suitable for researchers, who often are not aware of how demanding the task of preparing it is. As a result, the situation arises where research seems to be progressing, but the results from previous iterations are constantly being questioned, making it difficult to carry out further iterations because the system is unstable. That is why, aligning expectations and understanding the nature of the project is crucial for its success.
The primary goal of a research project is to explore, investigate, or discover new network architecture, algorithms, or technologies. It aims to advance the understanding of a particular problem or domain and solve what has not been solved so far. Alternatively, it can also try to see how already existing technology works under the given conditions.
Research projects often involve experimentation, literature review, hypothesis testing, and analysis of findings. The entire process can look like this:
Of course, each iteration involves further development and in some cases, testing the hypothesis alone requires a lot of development.
The most important aspect of the research project is that it ends with a report that summarizes everything that has been done. The result of a research project is knowledge of what works how and what potential it has. Based on such a report, we can, for example, decide to build a product using a given new technology. This also means that a research project can end with a statement like "It can't be done" or “We didn’t achieve any improvement” and it doesn’t mean that something was done wrong.
Research projects often involve higher levels of uncertainty and risk due to the exploration of unknown territories and the possibility of failure. There's no guarantee of immediate practical outcomes, and the research may not always lead to a successful solution.
An AI engineering project focuses on the design, development, implementation, and maintenance of Artificial Intelligence systems or solutions based on established principles, specifications, and requirements. These projects may include tasks such as data collection and preprocessing, model selection and training, software engineering, deployment, monitoring, and ongoing optimization. Its main objective is to create and implement AI-powered systems or applications that solve specific problems or enhance existing processes without the need for research.
The goal can also be to use the technology in a particular place, e.g. using this technology in a mobile application.
Engineering Project: Engineering projects typically follow a systematic development process that includes:
Engineers focus on building robust, scalable, and maintainable software solutions that meet functional and non-functional requirements.
It’s important to note that here, too, it may turn out that in a given place a given technology does not work because, for example, there is not enough data to train an appropriate model.
However, the engineering team is not expected to come up with a solution to make the technology work to determine precisely why it doesn’t work.
An engineering project does not, as a rule, create the risk of not "getting it right," although sometimes it may turn out that in a given situation a given technology does not work as well as in others, and it is necessary to launch a research project to find out why or how to solve a given problem.
An engineering project is typically more focused on delivering tangible results within defined constraints, such as time, budget, and functionality. While there are still risks involved, they are often more manageable and predictable compared to research projects.
In the real world, it's rare for a project to be purely research-based or purely engineering-based. Typically, AI projects lie somewhere between these two. For example, the technology might be ready, but it needs to be adapted to a specific use case. In such cases, some degree of research is necessary, but the risk of failure is low. On the other hand, research might require a significant amount of development of infrastructure needed to conduct the research, which must be prepared through a more engineering-focused project.
Conducting research at the beginning of a project, to ensure that a given problem is being addressed at the appropriate level under the given data conditions, is often the case in AI projects. Later, the research is transformed into engineering, where specific and tested solutions from the "lab" are deployed effectively in production.
Startups are an exception to this rule, where research blends with engineering due to the specificity of the business and their limited resources. This can yield amazing results in terms of cost-effectiveness, but it can also end in complete failure.
After deployment to production, in most cases, further iterations are necessary to ensure that the model operates properly. These post-deployment iterations typically are a blend of research and engineering.
If you're looking for an experienced AI team that can audit your solution, conduct research, or bring your engineering project to completion, contact us.