The Evaluation of Open Learning Scenarios that we spoke at, at FH Joanneum in Graz (Sept 2014) was a wonderful opportunity to engage with the teaching and research community at FHJ on open learning, Footprints, and evaluation.

In particular, Denny Seiger and I had an extensive discussion on reflection and data, or how reflective practice, such as Footprints can produce research data. The answer is that it depends on what you want to do with the research: how you want to go about it, what kind of data you want to create, and what you want to use it for. In other words, what kinds of affordances you are interested in. Here are some ideas ...

1. Collaborative reflection and design and research
Footprints of Emergence were developed primarily as a tool for surfacing, articulating and sharing deep tacit knowledge about what happens when people learn. The visualisation tools is a probe (i.e. a tool for creative exploration, and for new affordances). And probes are open too (the best probes are), which means they can (and are) used by learners, designers, researchers, teachers, and managers - individually or collaboratively.

The point is that they enable people to create rich visualisations of what happens in learning - what the designer / teacher envisaged, what the learner experienced, what the researcher identified, etc. What this yields is a range of possibilities:

  • If you are working within a complex adaptive systems framework, Footprints yield powerful outlier examples of learning and personal development. What this means is simply that if you regard learning as an adaptive, social, ecological activity, in which new ideas, techniques, paradigms, skills, etc are introduced, but, also, in which the learners also adapt, develop, changes, grow, within their own personal, social and professional narratives ..
  • Then ... the 'central tendencies' so beloved of statistical means, modes, standard deviations (etc) are less important than what is happening on the fringes, the edge of chaos (at more than 1 or even 2 standard deviations, if you insist) - those are the outlier activities and behaviours, many of which extend beyond the expected learning, the core curriculum, the norms.
  • And ... Footprints, because they articulate and map out the dynamics of change across a learning event, and the dynamics of different responses (different 'adaptations') by different learners, which are in turn often different from the designer's expectations ...
  • ... and offer rich, empirical, primary data for collaborative conversations between learners, designers, teachers, researchers, evaluators, about all these differences, across design, teaching and learning as a dynamic, adaptive, collaborative activity.
  • This in turn can lead to new collaborative design, research, teaching, learning, reflection (de capo).

2. Enriched reflections
Footprints are the product of personal reflection and introspection - not only about learning as an 'educational' event, but also as a space for personal, professional, cultural, social, and political adaptation, change and development. (See "a Footprint tells you about the micro-narrative of the learning event, as well as about the macro-narratives of the user's context of application", here ...).

So, from a rich data perspective, it would make sense to capture additional, rich, primary data about each of the factors that have been used in a particular footprint. This has been done in several footprints, in which the person creating the footprint has used the extreme right hand column of a mapping sheet to enter some thoughts about how they responded to whether that particular factor was 'pulled' towards the prescriptive centre of the footprint, or 'pushed' towards the emergent edge (of chaos).

Mapping sheet2.png
Mapping Sheet: Open / Structure
It is of course possible to ask the creator of the footprint to fill in much more detail, and even better, to record audio comments as well.

3. Email interviews
These additional comments (in text and/or audio) can be used for much more traditional quantitiative data analysis, preferably using critical discourse analysis, Hallidayian functional grammar analysis, or software such as Nvivo.

The important thing from the point of view of these visualisations, and reflective practice, is that it is important to allow space for reflection to take place on its own terms, before anything else. In other words, to make this as open as possible, and NOT to allow research or analytic frameworks anywhere near the person creating the visualisations.

Then, later, and separately, you can shift into more traditional research practices, if you wish, and conduct interviews. eMail interviews are a good option, as they yield ready-transcribed texts, but can be quite interactive - up to three emails and replies is typical, on a small number of themes - not too many questions!)

Conclusion
The real advantage of reflective methods like Footprints is in the personal and professional reflection, and in the opportunities for a wide range of people involved in the design and teaching and learning (and research) to be involved in more collaborative reflection and analysis. But as long as you create a safe space for reflection, first, it is possible to add other layers of more traditional qualitative analysis.