Beyond Artistic Merit: Assessing Rich Pictures for VUCA Problem-Solving with PPRESS

Research Paper Recommended citation: Cooke, N., Elphick, J., & Blumenfeld, M. (2025). Beyond Artistic Merit: Assessing Rich Pictures for VUCA Problem-Solving with PPRESS. In Kangaslampi, R., Langie, G., Järvinen, H.-M., & Nagy, B. (Eds.), SEFI 53rd Annual Conference. European Society for Engineering Education (SEFI), Tampere, Finland. DOI: 10.5281/zenodo.17631840. This Conference Paper is brought to you for open access by the 53rd Annual Conference of the European Society for Engineering Education (SEFI) at Tampere University in Tampere, Finland. This work is licensed under a Creative Commons Attribution-NonCommercial-Share Alike 4.0 International License. BEYOND ARTISTIC MERIT: ASSESSING RICH PICTURES FOR VUCA PROBLEM-SOLVING WITH PPRESS Neil Cooke a, 1 , Jon Elphick b, Marcelo Blumenfeld c, a University of Birmingham, UK, 0000-0003-2247-0663 b WSP Global Inc., Newcastle, UK. c University of Birmingham, UK, 0000-0003-2865-4519 Conference Key Areas: Engineering skills, professional skills, and transversal skills, Sustainability and society in engineering Keywords: Rich Pictures, Soft Systems Methodology, Assessment, VUCA, Challenge-Based Learning. ABSTRACT The Rich Picture is a key component of Soft Systems Methodology which has long been used to understand ill-structured problems or problematic situations across many fields. This case study explores the effectiveness of these pictures to foster students’ shared understanding of problems within contemporary interdisciplinary engineering challenge-based learning, where problems are situated in a volatile, uncertain, complex, and ambiguous world (VUCA) which requires a holistic consideration of diverse stakeholders. Student groups (n=65) created Rich Pictures for a 6-week project. We adapt an existing evaluation instrument– the Subjective Analysis of Group Assessment - to the engineering education context to assess Rich Pictures, devising the PPRESS criteria (Problem, Process, Relationships, Elements, Scope, and Stakeholders). An analysis revealed a moderately positive correlation between the quality of the pictures scored by PPRESS and the quality of the groups' final solutions. However, students demonstrated weaknesses in representing stakeholder views and relationships. Student feedback indicated mixed views on the usefulness of Rich Pictures, although they were seen as relatively beneficial for collaboration. These findings offer valuable insights for educators aiming to incorporate them into engineering curricula. The proposed PPRESS assessment criteria can aid in standardising scoring and reducing inherent positive marking bias towards artistry and aesthetics. 1 NJ Cooke [email protected] 1 INTRODUCTION Engineering Educators should encourage students to effectively understand and then structure complex problem situations before solving them. In a contemporary setting, problems arise from our increasingly volatile, uncertain, complex, ambiguous (VUCA) world (Fernandes & Afonso, 2021). This means they often include sociopolitical aspects and lack a definitive solution. For engineers to devise solutions to these ill-posed problems (or problematic situations), they should be skilled in systems thinking; a language and tool set that enables engineers to think spatially, expansively, nonlinearly, and holistically (Monat & Gannon, 2015). This critical skill is particularly needed by professionally registered engineers, who are expected to find optimal solutions through clearly identifying a problem’s root causes. In this VUCA world, problem identification itself is a problem; how can multiple stakeholder perspectives and scenarios be captured and related in such a way that a group of novice engineers can accurately and richly articulate a problematic situation? In this study we consider students’ articulation of complex problems in a group project using Rich Pictures (RP) – a diagrammatic tool from Soft Systems Methodology (SSM). We propose a scoring criterion and elicit students views on their use. The following research questions are posed: • RQ1 RP aspect score distribution: To what extent do engineering students' scores on different RP aspects vary in their successful representation of a complex problem? • RQ2 Relationship of RP quality to solution quality: How does the quality of RPs, as assessed using the proposed scoring criteria, correlate with the overall performance of group solutions presented in the final report? • RQ3 Student opinion: What are engineering students’ sentiments about the utility of RPs for understanding complex problems, as captured by a short survey? 2 CONTEXT AND PRACTICAL WORK 2.1 Rich Pictures and Soft Systems Methodology Peter Checkland developed soft systems methodology (SSM) in the 1970s with a focus on working with problematic situations by firstly gaining an appreciation of the problem through an unstructured means of appreciation – a Rich Picture (RP) - to avoid imposing premature constraints on thinking (Flood, 2000). It’s use by engineers gained popularity following realisations around the complexity of contemporary projects which not only require multidisciplinary structures but are situated in a world of nuanced social, economic, political, and cultural perspectives compounded by an increasing pressure of environmental urgency. Consequently, decision making within engineering projects needed systems thinking skills on top of traditional technical expertise. RPs embody the systems view of the world as a network of interconnected and interdependent elements (Meadows, 2008). As an open exploratory modelling process, they grant more freedom to articulate these nuanced stakeholder interactions when compared to more structured modelling techniques rooted in the systems engineering domain. RPs differ from conceptual exploratory tools such as mind maps due to their distinct epistemological emphasis. While the mind map is an analytical tool meant for brainstorming and decomposing a system, RPs embrace rather than avoid the complex environments within and around the system. Once a picture is drawn, the engineer follows SSM methodology to describe the problem more formally through “root definitions” to capture multiple stakeholder perspectives using the “CATWOE” mnemonic: Customers, Actors, Transformations, Worldviews, Owners, and the Environment (Flood, 2000). Conceptual models of the problematic situations and solution are then developed using SSM skills. These skills require the application of concepts such as events, feedback, flow, hierarchy, emergence, structure, and hierarchy (Kim, 1999). Despite its maturity, SSM is still used in research e.g. it was recently applied in an Industry 4.0 context (Kumar et al., 2024). 2.2 Assessment Schemes for Rich Pictures Assessing RP quality is inherently subjective. As a result, scoring is liable to be biased towards artistic merit. To mitigate this potentiality, we devise a marking scheme “PPRESS” for engineering education which builds upon the “Subjective Analysis of Group Assessment” (SAGA) instrument (Bell & Morse, 2013) which is itself built on a more general framework for art critique (Carney, 1994). This addresses artistic bias through coding the level of colour relevance, variation in lines and symbols, and mood expression. SAGA was also recently adapted to assess collaborative drawing skills of digital media communications and business students (Bowen, 2024). The PPRESS assessment criteria are detailed in Table 1. It is designed for those familiar with systems/process/software engineering “box and line” diagrams such as architecture diagrams, functional flow, and state-transition diagrams. Table 1 Proposed assessment criteria PPRESS criteria Description to derive marking rubrics Relationship to SAGA (Bowen, 2024) Problem Communication: How clearly the problematic situation is conveyed. A prominent and easily understandable symbol or shape representing the problematic situation, possibly accompanied by a brief label or description to ensure clarity for the intended audience. Visual metaphor Processes Depiction of activities and flows associated with the problematic situation. Resources and activities represented by distinct symbols. Different types of arrows (e.g., solid, dashed, coloured) show the flow of resources and the sequence of activities. Kinetic activity: arrows, dotted lines, connectors. Relationships Connections between different elements of the problematic situation. Lines and arrows connecting different elements to represent relationships and feedback loops. Different line styles, thicknesses, and labels can be used to distinguish relationship types, their strength, or direction of influence. Kinetic activity: arrows, dotted lines, connectors Elements How key elements influencing the problematic situation and their relative importance are represented Symbols or shapes representing key elements positioned around the problematic situation, with size or colour indicating perceived importance, and proximity suggesting the strength or type of influence. Graphic objects identified, Colour use: spatial organisation Scope The extent of the analysis and consideration of the wider environment. The RP clearly considers the wider environment, including wider social and political stakeholders, processes and elements, and how this effects the problematic situation. Complexity: detail, abstraction. Stakeholders Representation of stakeholder views, concerns, and needs. Key stakeholders represented by distinct symbols e.g., stick people. Their perspectives, concerns, motivations, or goals are represented through labels, speech bubbles, thought clouds, or emoticons. Expression: facial, thoughts, symbolic Overall Score for ranking Equally weighted sum of all criteria. 2.3 Context: Integrated Design Projects The case study context is a 12-week interdisciplinary, group-based project worth 20credits (200 hours of study) run in 2025. Participants include 341 students from aerospace, electrical, computer, mechanical, mechatronic, and civil engineering working in 65 teams. Each team comprises of up to 6 students, supported by 24 academic staff and 12 teaching assistants. During the first 6 weeks, students collaborate in multidisciplinary teams to develop a conceptual design for a proxy Moon base built on Earth to serve both local needs as well as to contribute to future space exploration. The design problem is intentionally ill-posed. It requires students to define the local needs and impacts, and incorporate these into their design process, alongside researching relevant emerging technologies. Thus, the challenge is not only creating a design solution but also forecasting emerging technologies and defining the problem itself. To provide distinction between groups, each is assigned a unique country to provide variations in social, environmental, economic, and government needs and constraints. Students are taught general systems design engineering, sustainability, risk management, project management, teamwork, and business skills. Following an SSM teaching sequence like (Wyatt et al., 2025), in week 1, students are introduced to RPs as an icebreaker activity. They draw an initial RP of their problematic situation (the group-based project) which is formatively peer assessed so that all students review at least 3 other groups’ understanding of the problem and can reflect on their RP skills through comparison. Team bonding is strengthened by making this exercise an ungraded competition, with academics awarding prizes to the groups with the best pictures in week 5. In this later session, students receive a 2-hour lecture on SSM. They then develop a final version of their picture (figure 1), incorporating enhanced understanding of the problem gained from approximately 100 hours of study. The final pictures are submitted for summative assessment against the PPRESS criteria, alongside their final reports. 3 METHODOLOGY To answer the three RQs stated in Section 1, two experienced academics with industry experience systems engineering (the authors) independently assess the pictures and reports respectively submitted by 65 student groups. Reports are scored using the criteria: project, research, requirements, design, risk management, and sustainability. Pictures are scored using PPRESS criteria described in Table 1. Total scores for both submissions are calculated as equally weighted sums. An anonymous survey is used to capture student perceptions on the RPs with respondents giving informed consent to use their responses. It employs a 5-point Likert scale, ranging from 'Strongly Disagree' to 'Strongly Agree,' for 3 statements relating to their understanding of the problem, effectiveness in facilitating collaboration, usefulness as a tool, as well as space for a free-text comment. Report and picture score distributions are visualised to check whether the data is approximately normally distributed in order to decide whether to use Pearson or Spearman rank correlation to analyse their relationship. The Shapiro-Wilks test indicated that while reports show no significant difference (p=0.24), RP scores significantly deviate (p<0.001). Consequently, Spearman’s rank correlation coefficient was selected. 4 RESULTS AND INSIGHTS 4.1 RQ1: Rich Picture aspect score distribution An analysis of the average marks for each RP category (Table 2) reveals the highest average score is for problem communication (42.4%) and the lowest averages and variances are for stakeholders and relationships (both <4% and <2% respectively). The highest variance was for scope (17.0%). Fig. 1 Examples of Rich Pictures. Table 2 Breakdown of average (n=65) rich picture scores Criteria Mean (%) Variance (%) Problem Communication 42.4 7.9 Elements 33.3 9.5 Scope 28.0 17.0 Processes 10.6 5.0 Stakeholders 3.8 1.8 Relationships 3.0 1.4 Average 20.2% 3.2% These results suggest that students found it challenging to identify and represent stakeholder views, concerns, and needs, as well as to connect different elements. In contrast, they found it easier to understand the scope of the problem and the wider environment. The higher variance for scope potentially serves as a useful criterion for discriminative assessment. 4.2 RQ2: Relationship of RP quality to solution quality The relationship between RP and report score ranks are visualised with a scatter plot (figure 2). There is a positive linear regression (r2=0.36) between RP scores and report scores. This is supported by the Spearman rank correlation (0.60) indicating a moderately positive monotonic relationship. Fig. 2 Scatterplot of Report Rank vs Rich Picture Rank (n=65) with linear regression line (r2=0.36) and 95% confidence interval (shaded). The relationship between pictures and report are visualised using a heat map (figure 3) of the Spearman rank correlation pairs. Overall, the correlations between the RP and report aspects are positive, ranging from weak (<0.3) to moderate (<0.5). Notably, there are no moderate or strong negative correlations. Correlations between RP criterion were moderately high for Elements with Communication (0.61) and Scope (0.53). This suggests some dependency and/or similar judgement being exercised by the assessor with the potential for scope to combine these categories. Correlations between the RP and report criterion were strongest for Sustainability and Scope (0.39), Sustainability and Elements (0.37), and for Research and Processes (0.39). Given that students score relatively low on processes (10.6%, table 2), this latter result suggests that groups who are doing better research will better consider activities and flows associated with a problem when revising their pictures. The moderately high positive correlation between Sustainability and Scope might highlight that problem extents and boundaries are enriched by better consideration of sustainable development goals and lifecycle analysis. Notable weak yet negative correlations between the report and pictures were Requirements and Design with Stakeholders and Relationships respectively. This might suggest that RPs are better suited for broader, humanistic, and conceptual relationships rather than the specific technical details of engineering. Fig. 3 Heatmap of Spearman Correlation Matrix of Report and Rich Picture categories 4.3 RQ3: Student opinion The survey received 21 responses to the Likert scale questions representing a response rate of 6.2%. Table 3 reveals that there was negative sentiment overall expressed about their usefulness, with 12 (57% - row 4) of the respondents strongly disagreeing, contrasting with 2 (10%) strongly agreeing. “The rich picture improved our groups collaboration” attracted the most positive response (row 2). Table 3 Student questionnaire results (n=21) Statement Strongly Disagree (%) Disagree (%) Neutral (%) Agree (%) Strongly Agree (%) Creating the rich picture helped me understand the problem better. 61.9 19 4.8 9.5 4.8 The rich picture improved our group's collaboration. 47.6 9.5 19.0 19.0 4.8 I found the rich picture a useful tool for this project. 66.7 14.3 9.5 4.8 4.8 Overall, I am satisfied with using rich pictures in this project. 57 29 0 5 10 Fourteen students left free-text comments. Two of them acknowledged the usefulness in understanding the problem. In contrast, 8 considered RPs too superficial to help solve engineering problems, with 3 specifically referring to the second RP as unneeded. A lack of familiarity in RPs was conveyed in comments – 2 sought more instruction on creating them despite their intention as an unstructured tool, and another expressed the sentiment that being marked on artistic skill is not engineering. This suggests that despite assurances by the instructors that drawing skills were not assessed, there is an instinctive and not altogether unreasonable assumption that when assessing pictures, markers will be biased towards aesthetically pleasing imagery. 5 CONCLUSIONS AND IMPLICATIONS Rich Pictures present a dichotomy: Their unstructured nature can help students to understand ill-posed problems, yet structured assessment criteria are needed for grading. They also present a paradox: The positive correlation between RP and solution success is not reflected in student sentiment. We proposed the PPRESS criteria to standardise assessment and mitigate potential biases that favour subjective artistic qualities over content. The comparison of RP scores and report scores revealed a moderate but not strong positive correlation. With this caveat firmly in mind, almost every group found it challenging to identify and represent stakeholder views and concerns. Groups who did better research were more likely to better consider activities and flows, while groups who scored better on sustainability were more likely to have wider defined problem extents. Despite these positive correlations, student sentiment was mostly negative about their usefulness with worries about marking bias towards aesthetics. We envisage as a time-limited activity early in the project supported by explicit instruction of PPRESS principles to help reduce these concerns. The case study is limited to a single intermediate-level group design module for one cohort, and assessment of RP and solutions each by one independent expert. Therefore, to generalise, the work should be repeated in other contexts by multiple markers. Future studies could focus on how to develop learning outcomes to improve students’ RPs and perceived value. 6 ACKNOWLEDGEMENTS This work is part of the "Maximising Academic and Social Outcomes in Engineering Education" (MASOEE) project with ethical approval ERN_2023-0385.