INESC TEC adopts output evaluation model of the European Space Agency
INESC TEC’s management model evaluated in TRL – Technology Readiness Level – considered ‘exceptional’ by the FCT
With this method, INESC TEC wanted to complement the evaluation of its scientific results, measured previously by number of publications, PhDs and projects developed. The results obtained have shown that the INESC TEC projects cover the entire value chain, with 60% of outputs in Knowledge Production and Applied Research, and 40% in Development and Product. In FCT’s evaluation, and following the use of this methodology, INESC TEC’s management model was considered ‘exceptional’.
What has been done and how?
A TRL-based evaluation focuses on the project’s end result and not on its scientific quality or progress. For that reason, with this methodology it is possible to assess the maturity of the technology resulting from the different projects developed at INESC TEC, and to understand trends relatively to the work conducted by the different centres and clusters.
This model consists of nine TRL, which means that there are nine levels of readiness for a technology. In their turn, these levels are part of four domains: Knowledge Production, Applied Research, Development and Product.
The TRL can be applied to hardware projects and systems, as well as software projects, consulting projects, and even PhDs.
TRL of systems
The following table contains a decision tree that summarises the main evaluation criteria that make it possible to characterise the TRL of a project.
TRL 1 focuses on the basic principles observed; it is the lowest technology readiness level. At this level, the basic research principles have been observed and reported. These principles can be translated into research and development. TRL 2 formulates the technological concept, but the system is still incipient, as there is no experimental evidence or detailed analysis that supports the technology.
TRL 3 refers to the proof of concept. In this TRL we are already transitioning to applied research. In TRL 4, the technology should be validated in a lab. The tests should prove that the basic technological elements involved in an invention can be integrated in a system and/or testbeds according to the specifications defined.
TRL 5 refers to validation in a relevant environment. This TRL requires the component and/or the breadboard to be validated in a relevant environment. In TRL 6, the technology or the system should be demonstrated in a relevant environment and a qualified prototype with all functionalities should be presented.
TRL 7 is a significant step in the maturing process after TRL 6, requiring a real demonstration of the prototype in the expected operational environment. TRL 8 assumes that there is a complete and qualified system, implemented with all functionalities defined and tested according to the standards of the product or sector.
In the last level, TRL 9, the system has already been approved under operational conditions. This readiness level required the system to be operated in an initially planned environment and its performance features should meet the system’s requirements.
TRL for software projects
The following table contains a decision tree that summarises the main evaluation criteria that make it possible to characterise the TRL of a project.
TRL 1 refers to the detailed mathematical formulation containing the results achieved, that is, the basic observed and reported principles. In TRL 2 there should be documentation on the algorithm’s implementation because a technological concept is being formulated.
In TRL 3, the architecture of the prototype’s critical features should be designed. In its turn, in TRL 4 it is necessary to design the architecture and the functionalities implemented in the ALPHA version.
TRL 5 focuses on the BETA version, and documentation should include test reports and application examples. The functionalities are fully implemented. The product release is contemplated in TRL 6. At this level there should be documentation according to the quality standards required in an SW product.
The early adopter version, TRL 7, includes documentation with the results of the first product uses. At this stage, the system is demonstrated in an operational environment. TRL 8, the general product, refers to a complete and qualified system that is ready to be used in a real environment; the documentation includes the specification, design, verification and validation.
Lastly, TRL 9, or live product, is a system that has been approved in operational conditions and is ready to be used in a real environment. It includes documentation as in level 8, as well as an instruction manual and an operating help desk.
TRL for consulting projects
In consulting projects, the evaluation of TRL should be made taking into consideration the time at which the consulting takes place and according to the following table:
Consulting services provided and evaluated with TRL 1 to 3 aim to assess technologies and trends, corresponding, in most cases, to technology monitoring studies. Levels 4 and 5 assess the most suitable tools and computational technologies to solve the problem. Consulting provided at development stage with TRL 6 and 7 focus on studying application architectures or databases, methodologies to develop applications and systems, and methodologies to test and guarantee reliable software. Consulting with TRL 8 or 9 corresponds to work where the goal is to optimise manufacturing processes, logistics, or to organise layouts, business process reengineering and product benchmarking.
TRL for PhDs
The TRL of PhDs will be assessed according to the specifications and methodology defined for system or software projects, and according to the nature of the PhD.
Results obtained by INESC TEC
This TRL evaluation has made it possible to conclude that the INESC TEC projects cover the entire value chain, with 60% of outputs in research stages and 40% in the development stages. The following table shows the distribution of projects relatively to the TRL, in the period between 2008 and 2013.
The following chart shows the distribution of TRLs associated with the results obtained by each INESC TEC Centre. It is possible to observe that the profiles are quite heterogeneous, which is explained by each Centre's position in the knowledge production value chain. The aggregation by clusters shows that each Centre is balanced.
Professionalising science management
Vladimiro Miranda, Director of INESC TEC, sees these indicators in a very positive way. “Their systematically application is a result of the professionalisation of science management, and this only happens because INESC TEC believes that the research conducted by its different groups is strategic. Thus is the value of institutions, and we have to acknowledge it because so often it is disregarded.” And he adds: “the identification of technology readiness levels revealed a profile that, for INESC TEC, surprisingly coincided with what we were hoping: 60% of activity generating results in science, and 40% in application. This means that we have an added responsibility to maintain this in the future.”
And he concludes: “The social relevance cannot be achieved only by promoting science of excellence. It is necessary to have excellent science management. If the FCT rated our management model as exceptional, which we really appreciated, it would be important for the national policy of science to follow that inspiration – generating adequate incentives, not only for scientific groups, but also for the institutions that are part of these groups in a broader mission, thus recognising their pivotal role.”
Impact and perspectives for the future
According to Augustin Olivier, the person in charge of the Industry Partnership Service (Serviço de Apoio a Parcerias Empresariais, SAPE) which conducted this study, “the TRL is a powerful instrument to manage science at INESC TEC, because it makes it possible to understand the institution’s position, and the position of its different centres.”
The manager considers, on the basis of a first analysis, that INESC TEC’s global score demonstrates that the institution’s relationship with the entrepreneurial environment is fundamental, with 40% of projects being conducted for companies. Augustin Olivier also adds that the results confirm “the importance of having activities upstream in order to feed the pipeline of knowledge that is required in order to work with companies.”
Lastly, he mentioned that “analysing the centres using TRL demonstrates their strategic position in the knowledge production value chain. Using this analysis, together with other data, INESC TEC created the concept of cluster. By combining competences and creating synergies, clusters make it possible to use a more homogenous value chain and to understand which areas need improvement. Ultimately, this tool can be used to assess researchers individually.”
INESC TEC, January 2015
