Osmo Kuusi, Kerstin Cuhls and Karlheinz Steinmüller 16.2.2015 Quality Criteria for Scientific Futures Research The article discusses quality criteria of the futures research in the ‘Futures Map’ frame. The Futures Map is comprehensive description of the outcomes of a futures research process. It comprises all relevant pictures of the future identified during the process and all relations between these pictures. The article suggests that the key task of scientific futures research is to improve the validity of Futures Maps. The high internal validity means that futures mapping process is well-organized. The high external validity means that the constructed Futures Map is good. The key contribution of the article is six pragmatic criteria of external validity of the Futures Maps. The suggested criteria of internal and external validity are compared with the quality criteria defined by the German Netzwerk Zukunftsforschung. The prospects of futures research as a field of science depend first of all on its research methods. The suitable methods do not, however, guarantee a high quality research effort. Based on the defined validity criteria, the article discusses the choice of the research methods as well as the evaluation of particular applications of the methods in practical research projects. Key words: futures research, science, quality criteria, validity, futures map, mapping, method 1. Background and main aims of the article In recent years, programs and projects that study the future from various methodological perspectives and focuses have multiplied. These studies have had quite different headings: futures research, futures studies, regional and technological foresight, forward-looking studies, some were even performed under the heading of corporate governance, to mention only the most widespread names. There is, however, no common understanding about the quality of these studies – not even about the criteria according to which the quality of foresight can be assessed or evaluated. Exceptions are found in the evaluation literature which mainly covers criteria of performance (Amanatidou and Guy 2008 Georghiou and Keenan 2006; Cuhls and Georghiou 2003) In this article, we will discuss quality criteria of futures studies. The lack of common understanding about the quality of futures studies is connected to the lack of a coherent conceptual frame of the field that is vaguely defined by the concepts of ‘futures studies’, ‘Foresight’ and ‘futures research’. A number of similar approaches under different names or with the same names can be found, e.g. scenario method or scenario technique (and sometimes even in short ‘scenarios’ where the term for the result is used for the method) for completely different procedures although the general idea behind them is the same. In this paper, besides the quality criteria discussion, we suggest a frame for the various concepts of Futures Research. The quality criteria we suggest are connected to what is called ‘Futures Map’ frame. There are various reasons – some old and some recent – that motivate our effort: · Practical work of futurists requires the distinction between high quality and low quality work, especially how to improve the use of methods in futures research · Because creative imagination is an integral ingredient of futures studies, it is especially important to define what forms of creative thinking are acceptable in scientific futures research and in which phases and frames they are acceptable. Research without open minded creative imagination is like bones without flesh or pictures without colors. On the other hand, futures research without fact-based justifications is like flesh without bones or colors without a picture. · For some years, training programs in futures studies have started in several European universities. The unclear position of futures studies in the field of sciences has seriously hindered its development towards an established discipline. · Quality problems have also been an often discussed theme in the futurist network of the global Millennium Project. The quality requirement is a special challenge for the Global Futures Intelligence System GFIS (http://www.millennium-project.org/millennium/GFIS.html). The authors of this article belong to or are connected to the Millennium Nodes in Europe. We will use the concepts ‘futures studies’, ‘foresight’ and ‘futures research’ in the following way: Futures studies refers to all kinds of approaches studying the future or futures. The concept foresight has a similar broad content though the foresight stresses more the pragmatic side of the futures studies and is defined as a systematic debate about different futures (see e.g. Martin 1995, or Cuhls 2003a or 2010). The concept ‘futurist’ refers to all ‘scientific’/ ‘research’ practitioners in the field of futures studies or foresight as long as their work is of a serious nature and not simply fantasy. The main focus of this article is the field of futures research. We reserve the concept futures research for those futures studies that are looking for pragmatically valid knowledge concerning possible futures. Validity means that this knowledge is based on facts, assumptions and reasons (including methodological approaches) that can be justified in discussions with other people, i. e. supported by well-founded argumentation.[footnoteRef:1] It combines external validity (supported by facts and well-established theories) with internal validity (consistent reasoning, appropriate methodology).[footnoteRef:2] [1: Comp. de Jouvenel’s concept of conjecture as ‘justified speculation about the future” (de Jouvenel 1964).] [2: Comp. Grunwald’s two aspects of a scientific approach to futures studies: validity in terms of ‘ingredients” and ‘composition” (Grunwald 2015).] According to the conventional definition, knowledge about a topic is justified true belief concerning the topic, Because there is no way to directly ascertain the truth of anticipation before its defined realization time, knowledge concerning possible futures can be nothing else than well-justified or well-argued beliefs. However, some futures studies simply present a selection of recent facts (developments, trends, weak signals…) that justify a particular future without taking into account similar facts that point in another direction. According to our interpretation, these types of one-sided justifications do not deserve the name ‘futures research’ and even less the name ‘scientific futures research’. Futures research should always be based on the ‘whole picture‘ of relevant futures. In this article, we will suggest six pragmatic criteria of external validity of the futures map. They define practical criteria for the validation of the ‘whole picture‘. It is important to realize that the ‘whole picture’ is produced and adapted for certain customers or specific users. Thus, there are two basic challenges in futures research. The first main challenge is scientific rigor: Does the ‘whole picture’ meet scientific criteria? Does it deserve the name of ‘research’ or is it just based on trend searching and arbitrary interpretation in a ‘closed shop’ and by an untransparent way? The other main challenge concerns the customers or the target group of users of the ‘whole picture’. Does the ‘whole picture’ serve their interests? Is it relevant for them? What is feasible from their point of view and what does the client or sponsor really want? 2. Futures Map and related concepts Wendell Bell (1997 p. 73) defined the purposes of futures studies as follows: The purposes of the futures studies are to discover or invent, examine or evaluate, and propose possible, probable and preferable futures. We consider that Bell’s definition characterizes the identity of futures studies. The purposes mentioned are well in line with the purposes of two main approaches of futures research: the scenario approach and Delphi surveys. The scenario approach and the related methods, first of all the morphological matrix- method (or the futures table-method), are used to discover, to invent or to propose possible futures. Scenario methods also provide systematic frames for the examination and evaluation of futures. You can also use some versions of Delphi surveys to invent futures but most Delphi versions are mainly used for other purposes (e.g. sense-making, assessments, see Cuhls 2012 or 2013). Experts use Delphi studies to evaluate how probable and preferable or important some specified futures are. The Futures Map provides a conceptual frame that helps to evaluate how futures researchers have proceeded in the promotion of the purposes defined by Bell. In that way, it is also a suitable frame for the discussion about quality criteria of futures research, since the quality of research results should be measured not against some singled-out elements but the ‘whole picture’ that the future study paints. For this ‘whole picture’ Malaska et al. (2009) have introduced the concept of Futures Map[footnoteRef:3] as follows: [3: The concept of futures map is not to be confused with general bibliometric or science maps.] … map is a source of information about the scenery, a symbolic replica of some characters of it. There is a relationship between the map’s designs and symbols and the real scenery at some level of coarseness and vagueness…In geographical mapping the elementary symbols and patterns of the map represent different elements of the scenery, e.g. trees, lakes, meadows…In the same way a futures manifold (or map) is a symbolic representation of the future… Based on the introduction of Malaska (2009), we define: A Futures Map is the comprehensive description of the outcomes of a futures research process. It comprises all relevant pictures of the future identified during the process and all relations between these pictures and between them and the present state as well as assessments about time frames, desirability and possibility of these pictures. The Futures Map includes all possible futures (‘futuribles’ [footnoteRef:4]) as identified during the research process. Any future or picture of the future is characterized by two parameters or dimensions: when it is assumed to realize (time of realisation) and how the users of the map will appreciate its realization (desirability, preferred futures), see Figure 1. The higher a point on the y-axle the more desirable the picture of the future is. That means normative as well as non-normative questions are considered. Another parameter (not shown in the illustration below) is possibility: Whether a picture of the future may – according to our best knowledge – realize or whether it is only an abstract option that does not fit reality (utopia, pipe dream…). We reserve the third dimension for the free arrangement of pictures of the future and pathways connecting them. This third dimension spans the ‘depth’ of the terrain. [4: In the terminology of de Jouvenel (1964) and Malaska et al. (2009): futurible.] Any picture of the future, when assessed according to time frame and desirability, has a definite place in the Futures Map. It may even occur that two pictures of the future occupy the same point in the two dimensions of time and desirability since they have identical assessments. Nevertheless, they can be completely different according to depth and content. We can place them differently in the third dimension. Pictures of the future are descriptions of a future state of affairs, sometimes rather lengthy narrative ones, sometimes rather short ones with only some key figures about society, economy etc. When placing the pictures of the future into the map, many points in the Futures Map may be left empty since there is no corresponding picture of the future. The scenario is a specified path on the Futures Map connecting the present state to at least one picture of the future. Sometimes also broader contextual pictures of the future are called scenarios though this is a bit confusing.[footnoteRef:5] In figure 1, the straight lines with decision (bifurcation) points illustrate possible scenarios. The original definition of the term ‘scenario’ in futures research as given by Herman Kahn is still a good way to describe its function in the Futures Map (Kahn – Wiener 1967 p. 6): [5: One may distinguish diachronic (or developmental) scenarios (often also called roadmap, but we use this term only linked to planning) from synchronic (or static) scenarios that coincide with our concept of picture of the future.] Scenarios are hypothetical sequences of events constructed for the purpose of focusing attention on causal processes and decision-points. They answer two kinds of questions: (1) precisely how might some hypothetical situation develop, step by step and (2) what alternatives exist, for each actor, at each step, for preventing, diverting, or facilitating the process. Figure 1: Scenario paths, a trend, a road map[footnoteRef:6], a vision and acceptable futures on the Futures Map [6: Whereas a trend is just a continuous development that can be more or less desirable and is not defined by desirability, the roadmap leads towards a desirable future, one that is chosen actively for planning (goal, vision). Nevertheless, one can ask – as Andrew Stirling asked at the FTA Conference November 2014 in Brussels: ‘Why do we need a map if there is only one road?”] An important task of futures studies is to challenge expectations. A way to do that is to construct unexpected developments. Futurists suggest various pictures of possible futures or simply possible futures. According to a rather common interpretation among futurists a future is possible if it does not include elements that are in contradiction with the basic laws of nature and behavioral possibilities of human beings. In evidence-based futures research, it is, however, reasonable to call a future possible just if there is a causal chain or a storyline that connects the present evidence - at least recent weak signals - and the picture of future. If a possibility lacks even a weak signal-based piece of evidence it can be called ‘abstract possibility’.[footnoteRef:7]We suggest that any possible future requires a kind of looking backwards from the picture of the future, a kind of backcasting from the (picture of) the future to the present evidence. [footnoteRef:8] [7: Already Hegel 1ntroduced an analogous distinction (Hegel 1817, 1947).] [8: In Futures research, the backcasting method implies a broader approach where all kinds of scenario paths starting from some picture(s) of the future are discussed.] A way to describe the range of possible futures is the scenario funnel. The basic idea of the scenario funnel is that the farther we gaze from the today’s standpoint toward the future the more possibilities are open (e.g. Kosow and Gassner 2008)[footnoteRef:9] In Figure 1, this idea of the funnel is taken into account in the scenario paths of the illustration. They all start from a “now” point and the range of the scenario paths increases with time. Scenario funnels are often used in that way that the most probable scenario is in the middle of the funnel. This idea does not function in Figure 1 where the possible scenarios are presented in the order in which the most desirable scenarios are at the top. [9: Methods of Future and Scenario Analysis, http://www.die-gdi.de/uploads/media/Studies_39.2008.pdf] A difference between a geographical map and the Futures Map is that the Futures Map depends on the capacities and purposes of actors. When we use a geographical map for navigation, our final destination depends on our choices. The same holds true for the Futures Map: The final position, the picture of the future where one arrives, depends on the choices the actors make on the way, on their purposes and capacities. Theodore Gordon (1989) expressed this important starting point of futures research in the following way: ‘There is a future without action, and a different one with it. Thus futures research and predestination are, at least on the surface, antithetical’. In other words: people decide on the future path by action or inactivity – there is always an alternative. During the futures mapping process, the actor (e.g. an organization) obtains information that changes its expectations of the ranges of desirability and of probability of possible future developments. The range of acceptable futures[footnoteRef:10] – and the related aspiration level of the actor – are based on these two ranges. If the evaluations of the probabilities and desirability are valid, the rational aspiration level is defined by the maximum expected value (expected desirability x expected probability). Another objective may be to avoid the worst possible (inacceptable) outcomes/ futures or to maximize the expected desirability x expected probability over all pictures of the future. Taking into account that the reach of desired futures in the Futures Map requires actions of actors it is reasonable to replace the concept of the probability with the concept of accessibility. Noticing also needed resources and uncertainties related to the results of the actions, acceptable futures are those futures that represent high enough expected value (expected desirability x expected accessibility). [10: We use the term “acceptable” in a restricted sense, as it is done in most business circumstances where insufficient ambition is “not acceptable”. We exclude therefore “acceptable” in the sense of “just tolerable” states of affair.] In practice, it is often reasonable just to divide pictures of the future of the Futures Map into those that are easy to achieve, difficult to achieve or impossible to achieve with available resources. Taking into account uncertainties, similar crude classification is often reasonable concerning expectations concerning the desirability. In the illustrating picture, the trend or the business-as-usual scenario is not acceptable (in the sense of supportable) as such (because its desirability is too low or unknown) but the map assumes that acceptable futures are accessible with action. The illustration suggests three points of time when actions might break the trend in that way. Of course, sometimes the trend development can also be acceptable. An important concept of the futures mapping is vision. There are many interpretations of this concept (e.g. Carey 1999, Masini 2002, Stevenson 2006). According to Jim Dator, vision is ‘the best possible real world you can imagine and strive for, always re-evaluating your preferences as you struggle towards it’ (Stevenson 2006). Futures Maps are typically constructed for organizations. Ideally the vision is shared by the members of the organization and vision processes looking for joint thinking based on common vision are more and more popular in companies (Weisbord and Janoff 2005 and 2000; zur Bonsen 2015) The following interpretation combines this feature with the definition: the vision of an organization is a future where the organization or members of the organization like to be, their shared dream future. Hard work might realize a dream but one has to accept and to be prepared for the fact that there is a highly probable possibility (with high probability) that a dream future will not be realized. The assumption is that the more desired a vision is, the more people work on it to make things real – until now, the empirical evidence of the realization in these cases is missing. The acceptable futures are situated in Figure 2 in the desirability between the vision and those bad futures that the people of the shared vision believe they can avoid. As we mentioned above, they are expected to be accessible. It means that if available resources are allocated to their realization they will – in case of reasonable probabilities and low uncertainties – be realized. Two important concepts of the Futures Maps are the mapping horizon and the planning horizon. In most cases both time horizons are defined during the bounding and framing phase of the futures research process. The mapping horizon is the anticipation horizon of the Futures Map (of the possible futures). Like in the picture, a scenario path might de facto end already before the mapping horizon. For example in the Futures Map of a company, the bankruptcy of the company might be an end point for the individual company, but in fact, the scenario without this company might be going on, unconsidered by the managers of the company. However, the time horizon of most scenarios is defined by the mapping horizon. As in the picture, there might be many scenario paths to the same end point of the mapping horizon. We connect the planning horizon of the Futures Map to the concept and method of a roadmap. In the picture, the dotted line that starts from the ‘now’ box (the present state of affairs) illustrates a roadmap. During the time frame of the planning horizon, the involved actors are committed to follow the specified road of the map – the roadmap[footnoteRef:11]. For example, a company decides that during the next two years all mobile phones they produce follow the same specified standards. A roadmap requires a vision or some other orienting picture of the future in the mapping horizon, planning horizon (time frame) and a planning horizon goal. [11: Nevertheless, one can ask – as Andrew Stirling asked at the FTA Conference November 2014 in Brussels: ‘Why do we need a map if there is only one road?”] Typically, it is reasonable to create and design the road map for a shorter period than the mapping horizon of the Futures Map. In the illustration, when one has followed the road of the map to the end of the dotted line, the world is not anymore the same as in the ‘now’ situation – it has changed and every new action starts under different conditions. At the end of the planning horizon, new evaluations of the vision and the acceptable futures are needed. Earlier adaptations are possible and recommended (e.g. in yearly strategic meetings adaptations in the roadmap can be made). 3. Pragmatic validity criteria for futures mapping processes Basic dictionary meanings of valid’ are sound, just or well-founded (Webster 1996). In logic it means that conclusions are based on established / agreed-on premises. In empirical sciences, the validity of generalizing theories or statements is, however, a matter of degree because you can never be sure that a general statement is true. Testing the theory, you can possibly falsify some of its conclusions but if no better theory is available you are justified to conclude that the theory is more valid than that of your competitors. The suitable definition for pragmatic validity in empirical sciences is the following: valid proposition, inference or conclusion is the best available approximation to the truth (compare http://www.socialresearchmethods.net/kb/introval.php). Following the above definition of validity, we conclude that the key task of scientific futures research is to improve the pragmatic validity of Futures Maps. [footnoteRef:12] This task is closely related to high quality standards for futures studies, [12: Kuusi (2011) made similar suggestion in the article Futures Research and the Think-Tanks ( in Finnish).] Any empirical science defines its own criteria of pragmatic validity.[footnoteRef:13] A common distinction in empirical sciences is the distinction between internal and external validity. Internal validity means that the research results are obtained using sound research methods. Besides the sound use of its various methods, we assume that the internal validity in the futures research requires well-organized processes that ‘shape the future’ as pragmatic and organizational approaches. The EFFLA Group (2013) has suggested that Foresight (in the sense of ‘applied futures research’ but including open debates[footnoteRef:14]) should always be a process that integrates Strategic intelligence, sense-making activities and the link to the Policy Cycle (see figure 2). [13: For example in psychology, validity encompasses the entire experimental concept and establishes whether the results obtained meet all of the requirements of the scientific practice e.g. there must have been randomization of the sample groups (https://explorable.com/validity-and-reliability)] [14: Foresight is defined very broadly as a systematic process of debating different futures (Cuhls 2010) as a further development of the definitions from Martin 1995.] Figure 2: The elements of a strategy process (own configuration (Cuhls) of figure from EFFLA Brief no. 14) Like the EFFLA Group (2013), we consider that a good way to promote the internal validity of the futures research process is to answer the following practical questions in the starting phase of the process (EFFLA Policy Brief no. 14, p. 2-3). Answers to these questions help when planning of the internally valid futures research process that also promotes externally valid results (high quality Futures Map, see below). a) What is the objective of the whole foresight activity? Are there hidden agendas? b) What type of activity has to be considered for what type of issues/time spans/ knowledge? c) What is the scope of foresight? What is the scope of relevant intelligence and sense-making? Is there specific strategic intelligence or or are there sense-making projects to be launched? How focused or wide should their scope be? d) What is an appropriate set of/ combination of/ methods to make use of the strategic intelligence of the specific actors? And how can this be organized? e) What are the intended outcomes of the different stages in the process? In general, reports are written but often, the activity as such is an outcome. How are the results presented? Campbell and Stanley (1966) have presented a classic definition of the external validity in behavioral sciences: External validity asks the question of generalizability: To what populations, settings, treatment variables and measurement variables can this effect be generalized? In the case of futures research, the external validity means that there are sound reasons to generalize – or to make abduction[footnoteRef:15] - from past and present facts to futures relevant conclusions. The Futures Map summarizes these generalizations or abductions. External validity therefore means that the research results, the Futures Map, is supported by facts and observations like existing trends, weak signals etc. and well-established theories. The pragmatic validity of the futures map increases if relevant actors are able to use it. If most relevant actors do not get the map or do not understand its messages some opportunities, which are otherwise accessible and are identified in the map, might not belong to accessible possibilities. This concerns, of course, also the avoidance of bad possibilities. [15: According to Charles Peirce, new ideas cannot be originated by deduction or induction but only by abduction; "[a]bduction furnishes all our ideas concerning real things, beyond what are given in perception" (Peirce CP 8.209, c. 1905),] For the pragmatic description of the validity of futures research results, we suggest six validity criteria we suggest six validity criteria of the Futures Maps of the Futures Maps.[footnoteRef:16] Let us have two Futures Maps FM1 and FM2 that have the same topical focus. Ceteris paribus or when FM1 and FM2 are equally valid in other criteria, any following criterion implies that the FM1 is more valid than FM2 from a pragmatic point of view. [16: These criteria were first suggested by Osmo Kuusi in research seminar of the Finnish Futures Academy, Helsinki 11.10.2011. The name of his presentation was ‘The Identity of the Futures Research among Sciences and Its Empirical Implications”] 1) FM1 suggests more possible futures than FM2 that might be relevant from the point of view of the vision or acceptable futures (wide scope of possibly relevant paths) 2) FM1 is able to identify most relevant futures better than FM2 (important relevant futures) 3) FM1’s scenarios are causally in line with more futures’ relevant facts than FM2’s scenarios (more interpreted causally relevant facts ) 4) FM1’s number of facts that get causal interpretation in scenarios divided by the number of scenarios is higher than in FM2 (effectively with scenarios interpreted facts) 5) FM1 is understood by more customers than FM2 (many understand) 6) FM1 is better understood by those customers who understand FM2 (better understood) Futures Maps are made for some customers or targeted users. Besides the quality criteria 5 and 6, this is referred to in the quality criteria 1 and 2 by the concept ‘relevant future’. The objective validity comparison between the maps FM1 and FM2 is possible just when the customers of FM1 and FM2 or their interests[footnoteRef:17] are the same.[footnoteRef:18] [17: Kuusi (1999) suggests that we should speak here ‘genuine interests” that the customers will not regret later.] [18: In the introduction of the Futures Research Methodology 3.0 of the Millennium Project Jerome Glenn argued that ‘futures research is not a science because it does not have controlled experiments like physics and chemistry and because two groups with different values, experience and knowledge using the same methods to explore the future of the same subject will produce different results”. However, if the customers of the futures map are defined Glenn’s argument makes little sense. ] For the identity of futures studies the first criterion is especially important. According to Futures Research Methodology 3.0 of the Millennium Project, ‘perhaps the most commonly understood reason for the use of futures methods is to help identify what you do not know, but need to know, to make more intelligent decisions.’ (http://www.millennium-project.org/millennium/GFIS.html). The identification of new possible scenario paths uses to show relevant gaps in our knowledge and in that way the validity of the Futures Map increases whenever it is able to evoke more (relevant) possible futures. [footnoteRef:19] The validity improves even without critical fact based examination of the suggested new possible paths (criterion 1). However, if the map includes, ceteris paribus, the most relevant path it improves is the validity of the map (criterion 2). [19: Metaphorically, a futures study focused on the criterion 1 is like a panorama photograph in which a picture of the future gives a particular perspective. ] As Malaska et al. (2009) remarked, futures are just partly determined by the known facts. It is possible that just some weak signals give indications of most relevant futures. So it is important to take into account all causally relevant past or recent facts (criterion 3). But if you are able to construct a map of a few scenario paths that takes account of a given set of facts and observations and gives an interpretation of their effects, it is ceteris paribus better than a map that needs more scenarios for the interpretation (criterion 4).[footnoteRef:20] A nice example of the use of criterion 4 (from another field) is linear regression analysis that interprets with a trend the variance of the past evidence assuming that the trend will continue also in future. [20: One has to notice that criterion 4 is in conflict with criterion 1 that suggests a preference to a futures map with a higher number of scenarios. Conflicts between criteria are not uncommon and sometimes helpful.] Even if a futures researcher has identified causally relevant possible futures and has taken into account even weak future signals something more is needed for a valid Futures Map. The validity of the Futures Map requires more than the correspondence of the Futures Map and past or present facts. Assuming that users or customers of the Futures Map themselves are best experts of choices that they will not regret (compare Kuusi 1999), they have to understand its relevant messages (the criterion 5). The criterion 5 is especially relevant when the common understanding of possibilities and relevant past facts are important for common choices of actors. However, in some situations it might be important that just key customers understand the map and those who have an interest to hinder most favorable futures of the map’s customers do not understand it (the criterion 6). There are connections between the suggested criteria of external validity and the internal validity of futures research processes. Typically, the identification of potentially important paths (criteria 1 and 2) belongs to the first phase of the futures mapping process. In Figure 2, it is Strategic Intelligence. The collection of potentially futures-relevant facts belongs also to the Strategic intelligence phase but the interpretation of their relevance in various scenarios belongs to Sense Making (criteria 3 and 4). The criteria 5 and 6 are related to all stages but they are especially important in the phases Sense Making and Selecting Priorities. A good scenario is not just possible and consistent but it should also be believable, trustworthy and interesting. This is an especially difficult challenge concerning those possibilities that challenge recent values and beliefs of the customers of the Futures Map. In order to manage this challenge, the futures researcher has to get the customers to understand their prejudices. To this end, the Causal Layered Analysis is a suggested approach. In practice, the management of this challenge is highly related to the use of interesting and convincing metaphors. The validity criteria function pairwise so that criteria 1 and 2; 3 and 4; and 5 and 6 define basic dimensions in validity evaluations. In his article The Science of ‘What-if’ Jerome Ravetz (1997) classified styles of research around ‘leading questions’ that nicely illustrate the three basic dimensions of the validity criteria. According to Ravetz the leading question is ‘what/how’ for research, the outcome of which is a statement intended to be factual. What/how research combines substance and agency: ‘what is this made of’ or ‘how does this cause that’. These questions are relevant especially concerning the validity of criteria 3 and 4. Statistical anticipation methods - e.g. regression analysis - are useful in predictions based on causal connections between specified variables. In these kinds of anticipations, criterion 3 means that scenario paths of FM1 take into account more past trends of possibly relevant variables than the scenario paths of FM2. The validity of criterion 4 means in these kinds of anticipations that FM1 causally interprets the past variance of facts with fewer ‘how the future might develop’ –trends than FM2. For example, think about weather forecasts. In the 1950s, the forecasts provided many possible scenarios for the next few days. Now we have a much more narrow scope of possible weather scenarios for the next three days, based on much richer past evidence and rather better models than in the 1950s. So recent weather Futures Maps are more validated based on both criterion 3 and criterion 4. Ravetz’ second style of research is focused on ‘how/why’ questions. This style accepts concepts like ‘final course’, ‘function’ or ‘purpose’. Here the point is the design that is able to perform a given function, to do its job. ‘How/why’ questions are reasonable only in the framework of actors with their interests (compare Kuusi 1999). The ‘why’-questions are especially relevant for the validity criteria 5 and 6. If an actor is not able to understand how his or her interests are connected to a possible future he or she is not interested in promoting that future. Concerning the validity criteria 1 and 2, Ravetz’ third research style is the most relevant. According to Ravetz, ‘what-if?’ expresses the spirit of creativity, of inventiveness, forays into an unknown that is passive and expectant. For Ravetz the role of ‘what-if’ questions was first of all the management of treating ignorance related to the impacts of new technologies and environmental hazards e.g. possible hazards of new chemical plants. We can, however, generalize the ‘what-if’ questions to concern also positive futures that are possibly just based on weak signals. ‘What-if?’ questions challenge prevailing anticipations and action routines. Based on them, actors realize new options for action. The value of the ‘what-if?’ approach is actually at least as much in new questions as in the suggested answers. This idea is nicely stressed by Michel Godet (2010) who cited Woody Allen: ‘The answer is yes, but what was the question?’ 4. Standards and quality criteria for projects in Futures Research Next we will compare the validity criteria of the Futures Map with other suggested quality criteria of Futures Research. As we mentioned in the beginning, there are few explicitly defined quality criteria in and for Futures Research. There is one clear exception that will be our focus in comparisons: quality criteria for futures studies suggested by the German Netzwerk Zukunftsforschung. Surely the quality problem on the general level is discussed in many books that are used in academic teaching or as guides of practical work of futurists. The main theme of Wendell Bell’s Foundation of Futures Studies (Bell 1997) is the nature of futures studies as a scientific activity.[footnoteRef:21] For the practical part directly linked to policy-making, the ‘‘Handbook of Foresight’ (Georghiou et al. 2008) is the major reference for ‘practitioners’ internationally. [21: Bell (1997) is extensively used in the university education of futures studies. During the last 17 years, it has been the book that all students of Finland Futures Academy have read. The Finland Futures Academy is a Finnish network of 10 universities, facilitating academic education and research programmes in Futures studies.] The journals of the field, especially Technological Forecasting and Social Change, Futures, Foresight and The Journal of Futures Studies and the new European Journal of Futures Research or the ZFZ (Zeitschrift für Zukunftsforschung) which are published in German have served as platforms for discussion of the good practice of Futures Research. The international Millennium Project has an acknowledged role in the development of shared understanding of the field. Though the Millennium Project is open for both scientific futures research and artistic futures studies, the quality issues are met especially in its Delphi related activities. The criteria defined by the Netzwerk Zukunftsforschung[footnoteRef:22] are, however, the best concise list which covers quality criteria of Futures Research projects that we know. The lack of commonly accepted standards and quality criteria for futures studies was widely recognized by members of NZF. In 2010 a group of NZF members united to initiate a discourse about ways to improve that situation. Seven professionals from academia and practice formed what they called ‘‘Task Force Standards’ (Taskforce zu Gütekriterien und Standards, TFS), and established a list of basic requirements futures studies should comply with.[footnoteRef:23] Ten more futures studies experts joined in to write a manual of standards, modeled on similar handbooks in social sciences. [22: The Netzwerk Zukunftsforschung (NZF) is the association of futures studies professionals of German speaking countries.] [23: See Gerhold (2015). For an outline of the history and a short exposition of the standards see Gerhold et al (2012).] The TFS –standards fall into three groups: · First, standards that ensure that Futures Research takes into account that the usual scientific procedures of verification or falsification can hardly be applied to future issues since their specific (future) subject is not accessible and requires therefore a special approach in knowledge creation. In short: The standards of this group make sure that Futures Research becomes really Futures Research, distinct from other scientific pursuits. · Second, there are also well-established norms and standards of good scientific research that are effectively applicable to futures studies. In short: The standards of this group ensure that futures studies become really Futures Research, distinct from other activities generating statements about future issues (like clairvoyance or science fiction). · Third, standards which guarantee that futures studies serve the purposes they are launched for, that the results of the studies have the best possible practical value. In short: The standards of this group ensure the relevance and effectiveness of Futures Research. The TFS standards relate primarily to the research process or internal validity of Futures Research, whereas the validity criteria of the Futures Map apply to the results or external validity of Futures Research. As we have already discussed previously, there is a close connection between procedure and product and therefore the two sets of criteria/ standards are interrelated. They are not identical but support each other – as will be discussed in the following. The ‘‘TFS list of standards’ suggests the following standards that result from the specific character of futures studies in distinction to other forms of research: 1.1 Principle ‘Images of the Future’’ reflects that statements about the future are constructions 1.2 Modality aspect of the Futures Research: possible, probable and preferable futures 1.3 Argumentative verification: images of the future have to be open to scrutiny in a debate about their ingredients and composition 1.4 Reference to action : futures studies are to inform decision making and action 1.5 Interdisciplinarity 1.6 Transdisciplinarity What do these standards mean and how are they related to the six validity criteria of the Futures Maps? The concept of the Image of the Future is broad and has nearly the same content as a belief concerning the future. For example, Wendell Bell (1997) considers that nearly equivalent concepts as ‘image of the future’ are ‘a developmental construct’’, ‘expectations’’, ‘anticipations’’, ‘hopes’’ and ‘fears’’. Bell considers that actors’ images of the future are among the causes of their present behaviour. Standard 1.1 of the TFS list requires that futures researchers should always make clear that they construct their statements based solely on present knowledge, not on any direct ‘fore-knowledge’’ of future objects and that their research results do not represent the future as it will be not true for some time, but present assumptions about the future (images, constructs). For scientific Futures research the crucial question is under which conditions a statement concerning the future represents knowledge. As was remarked at the beginning of the article, Futures researchers can just evaluate if the actor (e.g. an expert) has good justifications for the statement.[footnoteRef:24] This is subject to the standard 1.3 ‘Argumentative verification’: The assumptions underlying the statement about the future and the methodological integration of these assumptions should, in principle, be open to discourse, to examination, to approval or refusal by others. [24: Compare Bertrand de Jouvenel’s basic concept of ‘conjecture”: A conjecture is a justified hypothesis about the future.] When expert judgments are the source of knowledge concerning futures it is important to certify that the expert really believes his or her judgments. This is a highly relevant issue e.g. related to Delphi processes. Sometimes an expert is not ready to provide her/ his true belief. According to Kuusi (1999), the expert’s information policy defines how (s)he informs others about his or her beliefs. The expert is able to give two main kinds of justifications for the image of the future/ a belief concerning a future. The first kind of justification is based on natural objects and their behavioural rules. The second kind of justification is based on choices of actors. Actors are able to realize or construct the future anticipated by the image of the future. A justified belief concerning futures is related to all six criteria of the validation of the Futures Map: · Criterion 1: The coverage of possible futures (Pᵢ i= 1,..,n) gives a good justification for the disjunction of the possibilities (P1 ∨P2 ∨…∨Pn) · Criterion 2: If FM1 and FM2 identify the same possible futures (P1…Pn) but if F1 is able to classify the futures according to their relevancy it is a better justification than just (P1 ∨P2 ∨…∨Pn) · Criterion 3: Let us assume that FM1 takes better into account than FM2 future impacts of all kinds of causal processes that explain a set of given facts (Fi i= 1, …, k). Possible futures (P1…Pn) with explained facts derived from the past F1 &F2&…&Fk&(P1 ∨P2 ∨…∨Pn) · is surely a better justification than possible futures that are not based on development hypotheses which also explain a given range of facts. · Criterion 4: If FM1 and FM2 identify the same possible futures (P1…Pn) and are based on the interpretation of the same facts (F1,…, Fn) but FM1 is able to show better than FM2 which futures provide better interpretations of given facts than others, then FM1 is a better justification. A methodological application is e.g. the regression analysis that tries to find variables that explain the past variance of the explained variables in the best way. · Criterion 5: The justification given by this criterion is based on impacts on decision making. Futures are described in the Futures Map so that many can be used in decision making. Understandable descriptions might, however, be too superficial because they have to use metaphoric comparisons and skip complicated but relevant issues. Actors might later regret their choices. · Criterion 6: The justification given by this criterion is based especially on choices that actors do not need to regret. Based on the Futures Map, some actors understand possible futures so well that they understand causal chains behind the possible futures and impacts of their choices. The criteria 5 and 6 might be in contradiction if the better informed customers of the Futures Map can benefit from the ignorance of others. The standard 1.2 or the modal distinction between possible, probable and preferable futures is a key feature of scientific futures research and in line with Bell’s earlier cited purposes of futures studies (1997, Vol. I p. 73). The distinction between possible, probable and preferable futures is also the background distinction of the criteria 1-6. The first two criteria are focused on possible futures. The criteria 3 and 4 define probable futures based on facts of the past. In order to define preferable futures from their points of view, possibilities presented in the Futures Map have to be understandable for them (criteria 5 and 6). The Futures Maps are generated for actors and their decision making. The validation procedure with Futures Maps serves in this way for decision making and action focus of the standard 1.4. An important choice of scientific Futures Research is the choice between interdisciplinary and transdisciplinary research approaches (standards 1.5 and 1.6). Both aspects are important from the point of view of scientific futures research. Transdisciplinarity suggests a research strategy that crosses many disciplinary boundaries to create a holistic approach and even goes beyond that. Transdisciplinary research, as a rule, includes non-scientific experts from relevant domains of practice, who bring in their perspectives and knowledge. Interdisciplinarity accepts different specific scientific perspectives in the pursuit of a common task. More valid Futures Maps define a distinct transdisciplinary field of futures research. In this way, futures research is proceeding towards the target suggested by Bell (1997). According to Bell, the futures studies should not only ‘futurize’ existing fields of research but should incorporate more principles of scientific futures research into the perspectives, theories, teaching, and research of ‘old sciences’. Bell (1997 Vol. I p.59) considered that without this type of distinct transdisciplinary field, the ‘futurizing’ of other fields of science will be difficult or even impossible. The ‘TFS list of standards’ suggests the following standards that make scientific futures research different from other forms of futures thinking: 2.1 Explicit aims and framework conditions 2.2 Transparency and comprehensibility: third parties should be able to follow each step of reasoning 2.3 Theoretical foundation: a sound theoretical basis for the construction of images of the future 2.4 Appropriate choice and combination of research methods 2.5 Conceptual quality, including procedure according to the state of the art 2.6 Scientific relevance 2.7 Code of conduct Any kind of scientific work requires explicit aims and framework conditions as well as transparency and comprehensibility. The standards 2.1 and 2.2 are, however, difficult to achieve without a sound theoretical basis (standard 2.3). Especially sociological theories about the emergence of new forms of social life etc. and useful transdisciplinary theories about ‘futurogenesis”, the generation or emergence of futures, is missing. Wendell Bell (1997) suggested that the epistemological approach of critical realism is able to provide a sound theoretical basis for scientific Futures research. In natural sciences, you can typically put your belief to a strict intersubjective test that either keeps it acceptable or makes it inacceptable (falsifies it). The following basic principle of critical realism suggested by Alan Musgrave (1993, p.281) works well: A belief is reasonable if and only if it has withstood serious criticism. Typically in natural sciences, there are just few or only one belief or theory that is able to manage serious criticism of the scientific community. Possible futures should not contradict the rules of natural sciences. In that trivial sense, critical realism is also the sound theoretical basis for the construction of images of the future. But how to take into account actors who learn and make choices based on their interests? If an actor is informed about a prediction concerning its behaviour the actor can make any prediction or theory concerning its behaviour in the limits of its resources untrue. In this paper, we will not discuss this basic theoretical challenge to futures studies. Like Kuusi (1999) we just stress the importance of learning in any basic theory of futures research. The Futures Map is a tool for learning - for customers or decision makers. Unlike many methods of sociology that are looking for invariant (or ‘reliable’) patterns of behavior, the methods of futures research are used to promote learning and anticipative thinking. For example, many rounds of the Delphi method promote the common learning of panelists We consider that the prospects of scientific futures research will depend first of all on scientific methods that organize research processes well (the internal validity) and result in more valid Futures Maps (the external validity) (the standard 2.4). In the next chapter, we shortly evaluate how different methods are suitable for different purposes and what their strengths and weaknesses in the validation of the Futures Maps are. The standards 2.5, 2.6 and 2.7 - i.e. conceptual quality, including procedure according to the state of the art; scientific relevance and code of conduct - are closely related to the methodological choices. The ‘TFS list’ suggests the following standards that ensure that futures studies serve the purpose they are launched for: 3.1 Practical relevance, usability and impact 3.2 Understanding of the addressees, their types, roles and peculiarities 3.3 Transfer and communication: results should have a format suited to transfer 3.4 Identification of (general) lines of action 3.5 Project and process management Futures Maps are customer specific. If the Futures Map takes into account customers’ interests and key customers understand it, the validity of the Futures Map increases. In this sense, futures research is first of all an applied science serving actors in their decision making. Relevant actors have to understand the validated Futures Maps (validation criterion 5). In some cases, it is required that just key customers of the study understand the map (validation criterion 6). Sometimes these key customers might even require that e.g. their competitors are not able to use the map. The standards 3.1-3.5 are especially important for those who are practitioners in applied futures research. However, besides good project management, any futures study has to take very seriously the standards 3.2 and 3.3 into account. If panellists of a Delphi study notice that the study has practical relevance, usability and impact they are typically more ready to provide high quality arguments and judgments. 6. How to Promote High Quality Use of Futures Research Methods? The prospects of futures research as a field of science depend first of all on their research methods. It is highly important that the community of futures researchers is able to define the internal validity criteria of the futures research processes. To define the external validity criteria is therefore highly important. In this article, we have suggested six validation criteria of the Futures Map. We consider that they are ways to evaluate the external validity of the use of futures research methods. They are able to give the systematic frame for the scientific evaluation of the results of the futures research projects. The high quality use of the methods requires on the one hand that the research problem is handled with suitable methods. On the other hand, the high quality application of the selected methods is needed. In the choice of suitable methods, the classification of the methods can help. Cuhls (2008) suggests this kind of classification. She has classified the methods of technology foresight based on their suitability for different kinds of research problems – derived from practical projects. The suitable methods do not, however, guarantee a high quality research effort. We need quality criteria or validity criteria both for the evaluation of the research methods as such and for the evaluation of particular applications of the methods in practical research projects. Though there are only few lists of quality criteria of futures research many classifications of futures studies are available. Some are derived from the functions of the methods (see e.g. Steinmüller 1997), others from existing processes. Cuhls’ (2008) classification is based on different countries’ national foresight processes and how they are performed. A later project for the German Federal Ministry of Research and Education confirms these findings in the “Treasury of Experiences” (unpublished). Though the processes or the internal validity was in the forefront of her classification this approach also takes into account targets or functions of the foresight processes. A different criterion for the classification is the type of evidence (with a specific definition of output-oriented “evidence” in mind) or data that is used by the methods. This was the main classification criterion in the ‘Diamond of Methods’ of Rafael Popper (2007). Wendell Bell’s (1997) main classification criterion was the traditional distinction between qualitative and quantitative methods. In this paper, we comment on various classifications mainly from the point of view of external validity criteria of the Futures Map. We briefly discuss Bell’s and Popper’s classifications and focus more attention on Cuhls’ classification. Its focus on research problems is besides the internal validity also especially promising for the evaluation of the external validity of various methods and their use. Bell’s (1997) distinction between qualitative and quantitative futures research methods does not function very well, because e.g. Delphi surveys are quantitative in their data processing but qualitative in the analysis of different comments or open questions. When mixed methods or method combinations are applied – and that is meanwhile a high percentage of all future studies or foresight processes – quantitative and qualitative methods are combined or even integrated into each other. From the validation criteria of the Futures Map, quantitative methods are important especially concerning criterion 4. They are also more in line with criteria 2 and 5 than with criteria 1 and 3. They are clearly problematic concerning criterion 5. The Foresight Diamond of Rafael Popper (2008) distinguishes four data sources of the methods. Popper’s creativity-based methods are focused especially on the validity criteria 1 and 2 of the Futures Map. According to Popper (2008) creativity-based methods normally require a mixture of original and imaginative thinking, often provided by technology ‘gurus’, via genius forecasting, backcasting or essays. From the point of view of the futures research process framework of figure 2 (or EFFLA 2013), the inspiration which emerges from groups of people involved in brainstorming or wild cards sessions is mainly applied in the intelligence-gathering phase of Foresight. Evidence-based methods are focused especially on criteria 3 and 4. According to Popper (2008) evidence-based methods attempt to explain and/or forecast a particular phenomenon with the support of reliable documentation and means of analysis. These activities are particularly helpful for understanding the actual state of the development of the research issue. For this reason, quantitative methods (e.g. benchmarking, bibliometrics, data mining and indicators work) have become popular given that they are supported by statistical data or other types of indicators. All of them are lacking the explorative “outlook” character because they are based on historical or present data, which are prolonged into the future. Interaction-based methods help especially in managing criteria 5 and 6. According to Popper (2008) interaction-based methods feature in Foresight for at least two reasons – one is that expertise gains if experts challenge the expertise of each other or share their viewpoints (or the views of non-expert stakeholders); and the other is that Foresight activities are taking place in societies where democratic ideals are widespread, and legitimacy involves ‘bottom-up’, participatory and inclusive activities, not just reliance on evidence and experts (which are liable to be used selectively!). Scenario workshops, voting and polling are among the most widely used methods here. The third reason for using these methods – not mentioned by Popper in this connection - is that they often touch upon topics, for which explicit knowledge is not yet available. But how to interpret the role of expertise-based methods in the scientific mapping of futures? According to Popper (2008) they rely on the skill and knowledge of individuals in a particular area or subject. Common examples are expert panels and Delphi, but methods like roadmapping, relevance trees, logic charts, morphological analysis, key technologies or even focus groups and SMIC are essentially based on expertise. The foci of these basic methods of scientific futures research seem to be all six validity criteria. This shows the special data role of the expert information in the scientific futures research. Cuhls (2008) classified research problems of technology Foresight processes using the following dimensions of the research problems: 1. Explorative vs. selective 2. Long-term vs. short-term 3. Participative vs. analytic foresight approaches 4. Focused on general themes vs. on specific themes We consider that it is highly important to continue the discussion opened by Cuhls (2008). How relevant are the six validity criteria of the Futures Map in different research questions? Is it possible to combine futures research methods and research questions better if they are analyzed using internal or external validity criteria? There are some evident links between the six validation criteria and the types of research problems. In explorative studies, the criteria 1 and 3 are especially important. In selective studies the criteria 2 and 4 as well as the understanding of key customers of the study (criterion 6) are especially relevant. In the long-term studies the rich description of the possibilities (criteria 1 and 3) is the asset but also the most relevant lines of development (criterion 4). In short-term choices, the evidence-base (criterion 3 and 4) as well as the most relevant options that key customers understand (criteria 2 and 6) are especially important. For participation focused research problems, criterion 5 is especially important together with the open discussion about various possibilities (criterion 1). The analytic approach is based on the evidence base (criterion 3 and 4). Especially the focus is on criterion 4, on most effective explanations. When the theme is general, the rich view of the theme area is especially important (criteria 1 and 3). In wicked problems, in which very different kinds of stakeholders are involved, it is especially important to find a shared conceptual frame for communication and cooperation(criterion 5). In specific problems, the relevancy of the choices from the perspective of the key customers (criteria 2 and 6) and the search for best explanations based on data (criterion 4) are especially important. When the above remarks and Cuhls’ (2008) classification of methods are compared, this results in the hypotheses concerning validity criteria, on which the different methods especially focus. Cuhls considered that scenarios and monitoring (e.g. combined with technology mining) are most suitable for explorative studies. In line with the above conceptual frame of the Futures Map, she considered that roadmapping belongs to selective methods and is even at the border between foresight and planning. We can conclude that all planning horizon methods are selective, e.g. also the SWOT and the trend impact analysis. The planning horizon methods are observed to be more short-term (e.g. trend analysis) than mapping horizon methods, first of all the scenario methods. It is however important to realize that in the case of very rapidly changing environments the mapping horizon might be very short e. g. just one year. This means that also scenarios are performed just for one year. Delphi and simulation models are suitable both for explorative and selective as well as both short-term and long-term studies. Various kinds of futures conferences – especially the futures workshop of Robert Jungk – are participative methods. Also large panel and especially open access Delphi surveys are ways to promote participation and for many stakeholders to hear of an issue (source Cuhls 2000, Cuhls and Jaspers 2004). Small carefully selected expert Delphi panels of the Argument Delphi (Kuusi 1999) represent, however, rather the analytic approach as well as the trend analysis, the economic analysis (especially the regression analysis) and simulation models. The peculiar methods of futures research - especially the scenario methods and the Delphi – are able to challenge the statistical or economic methods, e.g. the regression analysis in complex issues. Complex issues are often broad issues, only in rare cases one finds sector specific issues. In principle, most futures research methods are, however, suitable also for sector specific studies though especially trend analysis is especially suitable for short-term sector specific studies. In table 1, we summarize the focused validity criteria of various futures research methods mentioned by Cuhls (2008). We can suppose that any specific research problem defines specific validity requirements for the needed Futures Map. For example, some explorative research problem requires high validity in criteria 1, 3 and 6 but requires less in criteria 2, 4 and 5. It means that a method or a combination of methods that has a similar profile is an especially suitable research method for this specific research problem. If the strength of the method does not match, it can be regarded as ineffective in the research problem. In the table, we use the scale typical for universities 0-6.[footnoteRef:25] The numbers illustrate the good practice of the method. In the poor practice some numbers are lower. [25: The interpretation of the scale would be, for example, the following 0= not at all noticed, 1= poorly noticed, 2= noticed, 3 =taken into account, 4= taken rather well into account, 5=taken well into account, 6= taken especially well into account] Table 1 Assessment of some futures research Methods according to the validity criteria 1 to 6 1: Wide scope of possible future paths 2: Most relevant futures paths 3: Covering interpretation of past ‘facts’ 4: Effective interpretation of past ‘facts’ 5: Many people understand the map, e.g. simple visualisation 6: Relevant experts understand the map Scenario writing 6 3 5 2 6 4 Futures conferences 2 5 4 2 6 5 Delphi survey 4 5 5 2 4 4 Argument Delphi 5 3 6 3 3 4 Trend analysis 2 4 2 5 5 6 Economic regression analysis 2 5 3 6 2 5 Simulation modelling 4 5 4 5 2 5 Road mapping 3 6 3 5 5 5 The evaluations of table 1 are focused on very broad methods and are, of course, very questionable and subjective. Any futures research project can, however, define the main targets of its futures map in a table like table 1. It is possible to compare this target structure with the special strengths of various methods. When the research project proceeds it is also possible to evaluate how the study has proceeded according to different validity criteria of the Future Map. 7. Conclusion Although we observe a lot of futures research, new projects and new actors that are coming up, we are still unclear about classifications, criteria what is ‘good’ and what is ‘not’ futures research in the sense of scientific rigor or transparency. The projects are just performed, some are even evaluated (see Cuhls and Georghiou 2004 or Georghiou and Keenan 2006) but the way we evaluate is rather arbitrary and the criteria chosen according to the case. In order to know what ‘good foresight’ or ‘good futures research’ really is, how it can be measured and which criteria have to be observed in the beginning and planning phase of a study or project, we still need more discussion, agreement and definition work. This paper is an attempt to stimulate this discussion and make proposals for quality criteria for Futures Maps and criteria for the choice of a certain methodology in a certain case. We need this discussion especially for those who organize processes in the futures fields – but also for education. We meanwhile have some lectures and chairs in futures research, especially in Germany and in Finland, and for them, a little more clarity in wording, criteria and methodological rigor would do the discipline an immense favor. We hope to contribute to this debate – otherwise, soon, the confusion will be perfect, the quality of futures research will decrease and the – still fragile – field might disappear again. This would be a pity, because: We need futures thinking, we need it now, we need it with a long-term view and we need insistence and continuity. References Amanatidou, Effie and Guy, Ken, 2008. Interpreting foresight process impacts: Steps towards the development of a framework conceptualising the dynamics of ‘foresight systems’, in: Technological Forecasting and Social Change, vol. 75, no. 4, May 2008, pp. 539–557. Bell, Wendell 1997. Foundations of futures studies, Vol I-II, New Brunswick: Transaction Books. Carey, John (ed.) 1999. The Faber Book of Utopias. London: Faber and Faber. Cuhls, Kerstin 2000: Opening up Foresight Processes, in: Économies et Sociétés, Série Dynamique technologique et organisation, no. 5, pp. 21-40 Cuhls, Kerstin 2003a. From Forecasting to Foresight processes – New participative Foresight Activities in Germany. In: Cuhls, Kerstin and Salo, Ahti (Guest Editors): Journal of Forecasting, Wiley Interscience, Special Issue, no. 22 (2003) pp. 93-111 (original paper presented at the EASST, Wien 2000). Cuhls, Kerstin 2003b. Delphi and Survey Techniques, Seminar Workbook. Ankara: UNIDO. Cuhls, Kerstin und Georghiou, Luke 2004. Evaluating a participative foresight process: 'Futur – the German research dialogue', in: Research Evaluation, vol. 13, no. 3, December 2004, pp. 143-153 Cuhls, Kerstin und Jaspers, Michael 2004 (eds.). Participatory Priority Setting for Research and Innovation Policy, Stuttgart: IRB, ISBN 3-8167-6468-1 Cuhls, Kerstin 2008. Methoden der Technikvorausschau – eine internationale Übersicht. Karlsruhe: FhG-ISI. Cuhls, Kerstin 2010. Schnittstellen von Foresight und Innovationsmanagement, in: Tiberius, Victor (ed.): Zukunftsorientierung in der Betriebswirtschaftslehre. Berlin: Springer, pp. 189-201. Cuhls, Kerstin 2012: Zu den Unterschieden zwischen Delphi-Befragungen und “einfachen” Zukunftsbefragungen, in: Popp, R. (Hg.): Zukunft und Wissenschaft. Wege und Irrwege der Zukunftsforschung, Heidelberg: Springer de Jouvenel, Bertrand 1964. L’Art de la Conjecture. Monaco: Éditions du Rocher. EUROPEAN FORUM ON FORWARD LOOKING ACTIVITIES – EFFLA 2013: EFFLA POLICY BRIEF no. 14. Towards standards in Forward Looking Activities for the EC (main author: Cuhls, Kerstin); http://ec.europa.eu/research/innovation-union/pdf/expert-groups/effla-reports/effla_pb_14_-_towards_a_foresight_standard.pdf (access 1/10/2014) Georghiou, Luke / Harper, Jennifer Cassingena / Keenan, Michael / Miles, Ian / Popper, Rafael (eds.) 2008. The Handbook of Technology Foresight. Concepts and Practice. Cheltenhem: Elgar. Georghiou, Luke and Keenan. Michael 2006: Evaluation of national foresight activities: Assessing rationale, process and impact, in: Technological Forecasting and Social Change, Volume 73, Issue 7, September 2006, Pages 761–777 Gerhold, Lars / Holtmannspötter, Dirk / Neuhaus, Christian / Schüll, Elmar / Schulz-Montag, Beate / Steinmüller, Karlheinz (eds.) 2015. Standards und Gütekriterien der Zukunftsforschung. Ein Handbuch für Wissenschaft und Praxis. Wiesbaden: Springer Fachmedien. Gerhold, Lars / Holtmannspötter, Dirk / Neuhaus, Christian / Schüll, Elmar / Schulz-Montag, Beate / Steinmüller, Karlheinz 2012: Qualitätsstandards für die Zukunftsforschung. In: Reinhold Popp (ed..): Zukunft und Wissenschaft. Wege und Irrwege der Zukunftsforschung. Berlin Heidelberg: Springer, pp. 203 – 209. Glenn, Jerome C. 2009. Introduction to the Futures Research Methods Series. The Millennium Project Futures Research Methodology, V3.0. Godet, Michel (2010) Future Memories, Technological Forecasting & Social Change 77 (2010) 1457–1463 Gordon T.J. (1989) Futures Research: Did it Meet its Promise? Technological Forecasting and Social Change, Vol. 34, No.1-2 Grunwald, Armin 2015. Argumentative Prüfbarkeit. In: Gerhold et. al. (eds.) 2015, pp. 40-51. Hegel, Georg Wilhelm Friedrich. Die Wissenschaft der Logik Vol. 2. (1817) Leipzig: Meiner , 1947 Inayatullah, Sohail (ed.) 2004. The Causal Layered Analysis (CLA) Reader, Theory and Case Studies of an Integrative and Transformative Methodology. Tapei: Tamkang University Press (http://www.metafuture.org/cla%20papers/Inayatullah%20%20Causal%20layered%20analysis%20-%20theory,%20historical%20context,%20and%20case%20studies.%20Intro%20chapter%20from%20The%20CLA%20Reader.pdf). Kahn, Herman and Wiener, Anthony 1967. The Year 2000. A Framework for Speculation on the Next Thirty-Three Years. New York: MacMillan. Kosow, Hannah and Gassner, Robert 2008. Methods of Future and Scenario Analysis. Overview, Assessment, and Selection Criteria. Bonn: DIE – Deutsches Institut für Entwicklungspolitik (http://www.die-gdi.de/uploads/media/Studies_39.2008.pdf). Kuusi, Osmo 1999. Expertise in the Future Use of Generic Technologies, Government Institute for Economic Research, Research Reports 59, Helsinki, J-Paino Oy. Kuusi, Osmo 2011. Futures Research and the Think-Tanks (in Finnish language). In: Futura 4/2011, pp.12-18 Malaska, Pentti and Virtanen, Ilkka 2009. Theory of Futuribles and Historibles. In Futura 1/2009, pp. 65-84. Martin, Ben R. 1995. Foresight in Science and Technology. Technology Analysis & Strategic Management, 7, no. 2., pp. 139-168. Masini, Eleonora Barbieri 2002. A vision of futures studies. In: Futures vol. 34, pp. 249-259. Musgrave, Alan 1993. Common Sense, Science and Scepticism: A Historical Introduction to the Theory of Knowledge. Cambridge: Cambridge University Press. Popper, Rafael 2008. Foresight Methodology. In: Georghiou et al. 2008, pp. 44-88. Ravetz, Jerome 1997. The Science of ‘What-if’. In: Futures Vol. 29, No. 6, August 1997, pp. 533-539. Steinmüller, Karlheinz 1997. Grundlagen und Methoden der Zukunftsforschung, SFZ-WerkstattBericht Nr. 21. Gelsenkirchen: Sekretariat für Zukunftsforschung. Stevenson, Tony 2006. From Vision into Action. In: Futures vol. 38, pp. 667-672. Webster 1996. Encyclopedic Unabridged Dictonary of the English Language, Avenel, New York Weisbord, Marvin 2000. Future Search: An Action Guide to Finding Common Ground in Organizations and Communities. 2nd Edition. San Francisco: Berrett-Koehler. Weisbord, Marvin and Janoff, Sandra 2005. Faster, Shorter, Cheaper May be Simple. It’s Never Easy. In: The Journal of Applied Behavioral Science, Vol. 41, No. 1, pp. 70-82. Zur Bonsen, Matthias 2010. Leading With Life: Lebendigkeit im Unternehmen freisetzen. Wiesbaden: Gabler. 1 image1.png image2.png