AI, Critical Thinking, and Risk – Are We Learning More While Understanding Less?

Tim: Cool, excellent. We are live. We’re going to give people a little while to join. Welcome to our Redflags summer series, which is the excuse for me wearing a ridiculous hat, sunglasses and summery shirt. So we’re doing a summer series while everyone else is on holiday. We thought we’d carry on talking about behavioural science and I’m going to give people a little bit longer to join and we’ll probably start at about two minutes past two. Join in the chat, let us know where you’re from, what the weather’s like, where you are hot and possibly on fire. We’re all relieved it’s not quite so hot as yesterday. Apparently it is raining in Nottingham. Wow, we need some of that rain in Cambridge. It is not raining in Cambridge. I just discovered a tree I’d planted that has died because I have not watered it. It’s sad.

Tim: So welcome everybody once again to our Redflags Summer series. We’re just pausing a little bit, a couple of minutes past the hour for people to join. This is our summer series where we thought, well, risk doesn’t go on holiday, so we carry on talking about cyber security and risk and hence it’s the summer. So I’m wearing silly clothes to celebrate that fact. I will take these sunglasses off soon so I can read my screen. Excellent, right well, let’s crack on and welcome everybody. Thank you for joining us. This is our AI critical thinking and risk webinar, part of our Redflags summer series. And I’m really excited about this panel because it’s a little bit different to what we normally do at Redflags. We’re moving away a bit from our normal topics, but in some respects it’s the same behavioural science, the same issues behind the scenes that are actually driving us to use AI in certain ways and to create risk. And obviously we’ll be digging into that as we go.

Tim: So let me stop showing my background slide and introduce who is here with us. So I’m absolutely delighted to be joined by two stellar people. First, we have Bec McKeown, one of the leading psychologists working at the intersection of human performance, decision making and cyber security. So Bec’s a chartered psychologist, founder of Mind Science, with more than two decades of experience spanning defence, aviation, cyber security, helping organisations understand how people really make decisions and how they perform under pressure. So a leading voice on the psychology of cyber security, operational resilience and human performance in complex environments. And certainly we live in them nowadays. So second, we’ve got Redflags’ very own Melanie Knight, a learning designer and specialist in taking complex knowledge, transforming it into accessible, impactful learning experiences that really actually change behaviour. So with a background in education, educational leadership, she brings a perspective on how people learn that spans from childhood in the classroom all the way into adulthood and workplace learning. So Melanie lives and breathes our Redflags content and applies behavioural psychology and learning techniques to make it as impactful as possible. So welcome Bec and welcome Melanie.

Bec: Thank you.

Tim: So to set a bit of context around our interest in this topic at Redflags. So our mission at Redflags is to empower people to protect themselves against cyber threats that target the person. So we’re all about trying to understand behaviour and learning and we spend a lot of time thinking about the psychology of why people behave the way they do, and why they get caught by social engineering or why they make data loss mistakes. And we try and use that understanding of cognitive bias to help protect people. It’s very rarely an individual’s fault that they get caught out. It’s more of a behavioural or a system failure. And so that’s our background. That’s why we’re interested in this topic. But today we talk about AI and learning and critical thinking and let’s start big. So Bec, how do you see that AI has transformed access to information and workforce learning?

Bec: It’s just massive, isn’t it? I think back to when the time was, we used to have Encyclopedia Britannica on our shelves and then Google appeared. And this is just that extra, isn’t it? I mean, access to huge amounts of information has never been easier. And that’s brilliant in some ways. I think also it does get a little bit tricky sometimes because just because you’ve got access to information doesn’t mean you actually understand it or how it’s all fitting together. So, pluses and minuses, pluses and minuses.

Tim: And Melanie, I think there is a problem here, isn’t there, around that difference between acquiring information and actually developing that understanding of the information?

Melanie: Definitely. There’s a huge difference between getting a nice easy answer and actually understanding the topic and getting the meaning and the relationships and the context and underlying principles around the topic. Anyone can now produce a really polished looking article about a topic. But studies have shown that using AI to offload that thinking means that people felt less capable, less confident, and they have less ownership of the ideas. Because AI skips that formative struggle, which is we need that struggle to build the judgement and the confidence and the self efficacy. So I think we come to solutions a bit more later, but one thing that’s been suggested is using AI instead of to get quick answers, using AI to have more of a Socratic dialogue, using AI to ask questions, which develop your understanding of principles on the topic rather than just, I’ve got the information I need, I’m moving on.

Tim: Absolutely. I think we’ve all discussed in our prep for this the study where I think they looked at how much people could remember about an essay they’d written when they did or didn’t use AI. And it was quite worrying that using AI, well, you simply haven’t formulated any of those sentences yourself. So how are you going to remember them?

Tim: Absolutely. So there’s an aspect here that again, we’ll come onto a little bit more through the talk around cognitive load and that you’ve explored the concept of cognitive load in your work. Can you talk us through it? What is cognitive load and why is it relevant?

Bec: I mean, it’s usually relevant in high performance organisations because if you’re cognitively overloaded, basically it means that you’ve got nothing left for anything else that’s coming in. And so to give it a more everyday example, if you think about when you’re on, I don’t know, maybe your route to work and you drive the same route every day for donkeys years. So you know where you’re going. You don’t have to think about it. You can have a conversation with somebody sitting in the car. You could be listening to the radio. You could be thinking about the day ahead and what’s going to happen at work, your shopping lists, what type of dog has got to go to the vet, all of those things. And that’s fine.

Bec: And then you go and drive through an unfamiliar city, it’s rush hour, it’s chucking it down with rain and you’re trying to follow a diversion. And at that point, you’re not going to be doing any of the chatting or listening to the radio or thinking about anything else because all of your cognitive resources are utterly consumed with not crashing on that journey. So it’s really about how full the tank is and what you have left to work with.

Bec: And I think for me, that’s kind of where AI is providing us with some answers because generally the brain’s a limited capacity information processor. Any chance it has to reduce mental effort, it’s going to grab it. And look at what AI does. It can do a huge amount of all sorts of tasks, and it just gives you that little bit of capacity back because you’ve either got the job done quicker or it’s doing it while you’re doing something else. So with cognitive load, it’s really about mental effort. And the more capacity you have for mental effort, the more effective you’re going to be at work generally.

Tim: And is that essentially why we’re all diving on AI? We’re just trying to offload as much as we can?

Bec: I think there’s that, but there’s also some sort of curiosity involved in that because we’re all starting to use AI in different ways because we weren’t sure what it could do. I mean, my first use of it was to make up a story about mine and a colleague’s dogs who were police dogs and they were off doing exciting things, so it was a proper rabbit hole and still is to a certain extent. So I think we’re naturally curious. People enjoy learning about new things, or some people do. But I think that for a large part of it is about what can I do to make myself quicker, more efficient. And I think especially when you’ve got huge workloads, I mean, everybody’s trying to do more with less these days. So anything that we can do that makes it easier and makes us more comfortable and actually gives you a little bit of confidence that you’re back in control.

Tim: And I think the reason to touch on that aspect of cognitive load is that it sounds like it’s one of the drivers that gets us to lean on these tools and other tools. But Melanie, that concept of cognitive load, we see that impacting security behaviours, don’t we as well? How does that play out?

Melanie: Definitely. But it’s also just that although the technology has changed, obviously human attention, human behaviours haven’t changed. So we’re still dealing with the same cognitive biases that we’ve always been dealing with around phishing, data loss, etcetera, just applying them in different ways. So we can still look at things like security fatigue when people are overloaded with warnings and policies and then they start bypassing those protocols and ignoring the warnings and choosing the easy option, which is often AI.

Melanie: And we sometimes try and combat that with lots of fear based messaging around the risks. And this is really dangerous and don’t do it. But that actually produces more of a feeling of helplessness instead of protective action because that threat is unavoidable. So we definitely see cognitive load making us make much quicker decisions. You’re under pressure. You just need a quick answer. And you’re not thinking about that training you had six months ago or the policy you read three months ago. You’re just going for the easy option and the fix for this isn’t more training. Part of this needs to be an environment fix about making it simpler to make the right choice and helping us to make fewer decisions rather than adding more rules, more expectations, more training, more things to do.

Tim: It’s interesting. So it’s some of the same behavioural issues that we have around the brain being overloaded. It’s a bit of a cognitive miser and we’re working in complex environments, that’s driving us to grab these new tools, but it’s also what makes us make these mistakes. When a well crafted social engineering email comes in, the cues are playing to our quick and easy decision making and so we make these mistakes. And so, back more to the broader AI topic, but there is some good here in, you know, it’s reducing the effort required to develop expertise. So can you become an expert? Can it help you?

Bec: Well, I think when you say, does it help you become an expert, maybe it does, maybe it doesn’t. That’s a typical psychology answer is maybe yes and no, all of that sort of thing. I think that generally, if you’re not an expert in the topic or you don’t have much knowledge or skills, we tend to overestimate our confidence because we lack the expertise to actually recognise what our own limitations are. I think that’s amplified by AI because you’ve put whatever prompt that you’ve put in and you’ve got some stuff back and actually it looks fabulous, but is it fabulous because you’re not expert enough to know if it’s right or not, or if there’s anything subtle that’s missing?

Bec: But also, as Melanie said earlier, they understand the explanation, they can read the words, but they don’t necessarily understand the underlying concept. So I would say that from that perspective, it’s almost like success without expertise. You can produce something that looks fabulous, you get good feedback from it, so your confidence is going up faster than your actual genuine competence is developing. So you tend to have that feeling that I can do this rather than I can produce something that looks as good as this. And there’s a very subtle difference between those two things.

Tim: OK, so we’re almost getting self efficacy. We feel like we’ve done the task, but we haven’t really. We’ve offloaded the task and the output created the output.

Bec: Yes. And that happens to me all the time when I’m doing marketing stuff for my business because I’m obviously not an expert in marketing, but you can create some wild stuff until somebody reads it and says so. I think that confidence is a really funny thing because there’s that lack of confidence, there’s overconfidence. Where do you calibrate in between that? And I think also the way AI is set up, it just butters you up all the time, doesn’t it? It doesn’t say, oh, by the way, don’t do that or I challenge this unless you particularly ask it to. It doesn’t tell you I don’t know and it doesn’t say you haven’t understood this.

Bec: So what you’re not getting is feedback about any mistakes that you’re making, the level of knowledge, which is all that really important stuff that happens in learning design. Because if you’re not getting feedback about the mistake you’ve made, you don’t know you’ve made it or what to do differently. So if you’re not getting all of those signals that you’re reaching the limits of your knowledge, then confidence just tends to grow. So I think it’s a really interesting paradox between the two situations for that.

Tim: It’s always creating that stronger understanding of the difference between knowledge and that kind of, there was a time when the government felt that it was all about rote learning and we just needed a pile of knowledge, and that difference in knowledge and curiosity, problem solving skills. And it’s exacerbating that challenge really, isn’t it? And Melanie, why do we get this false confidence? Bec’s touched on some of those issues, but why do we believe what the AI is telling us?

Melanie: I think it’s definitely things like the authority. A lot of these are developed by big companies that we trust. We trust Google. We’ve been using them for a search engine and our emails for years. And so that gives us a trust in them. Also, we anthropomorphise these tools. We treat them like people. We talk about the people, he talks, Claude, he told me to do this. And that again builds our trust. And if we have limited understanding of the tools, that means that we can often underestimate those risks, like misinformation or bias or privacy concerns.

Melanie: And it’s also really interesting that we’d expect to see the Dunning Kruger effect here, that people with low ability in an area will overestimate their knowledge. But actually, in the case of AI, one study showed that it’s the opposite. People with higher AI literacy became almost risk blind and underestimated that risk because they’ve cognitively offloaded decision making. And so they’re not critically evaluating their reasoning like Bec was saying. They’re not thinking about how much they know.

Tim: Well, they’ve been dumbed down slightly. Because at that point, we’re thinking, oh, the AI is going to make that decision or do that for me. We almost disengage that critical thinking at that point.

Bec: I think it’s also easier because it’s given you something that looks very, very credible instantly. You’re busy. You want to get this task out the way so you can move on with something else. So that looks great. Off it goes.

Tim: It’s said with authority as well, isn’t it? It’s not just that we see the authority, and I know some people have been trying to tailor it to say just give me the answer straight, but it gives you that chat around the edges, like that’s a brilliant question. And it’s making you feel good about the whole dialogue with it, which is interesting. Someone somewhere made a decision to build in that slightly creepy over friendliness.

Bec: Is that why I always say thank you to mine? Because I just don’t want to fail the algorithm.

Melanie: And that fluent, decisive language also feeds into that authority bias. We mistake that for, oh, it’s produced a really quick answer, it sounds really confident, therefore I can trust it. It must be right.

Bec: But that’s not even a conscious thing. And I think that’s where the problem is. You don’t even realise that that’s what you’re doing. And it just made me think about metacognition, which is basically thinking about the way you think, or it’s about being very self aware of your own cognitive processes, what you know, what you don’t know, when to verify, when to challenge. And I think that is something that really probably needs to come back to the fore.

Bec: You know, we’re offloading a lot of things to AI, but I think there’s also an increasing requirement for the more human skills to actually be ramped up a bit because that’s a really interesting concept. I mean, we don’t teach people metacognition, do we? I didn’t know about it until I started teaching it at master’s level. That’s how I learned, not in school 100 years ago. So I think there’s a whole bunch of skills that are really going to start coming to the fore, whereas the focus has very much been on technology for quite some time. I hope that those human skills and how to be human are going to become more important the more ubiquitous AI gets.

Tim: I think there is also a need for that kind of, like quite a lot of dyed in the wool cyber security people don’t make mistakes because they are obsessively paranoid, and I think we all need to be a little bit more paranoid and scared. You can start to see how the AI has been designed, not the technology capability, but that sort of, yes, that’s a great question. But also it’s almost like hooking you.

Tim: I got quite rude to my AI yesterday, don’t tell the others, they’ll gang up on me, but it was helping me write something and each time it got to a final draft, it would say, oh, but actually you could also add this. And I was like, well, put that back in the draft. And it did it and then said, oh, and now you can have it. It’s like, just give me the final answer that I can review with my own. And that’s not helping me. That’s designed for the vendor to keep you on the platform using tokens, isn’t it? I think we need to almost breed some more cynicism. I mean, I’m jumping around in the questions here, but does that tie into gender difference here? Sorry, an age difference here. Is there an age expertise difference in the way that we should and can use these tools?

Bec: I don’t know if it’s age related. I think sometimes we like to, you know, young people can do it, old people can’t, because that’s a nice little neat distinction. And I think that the novice expert is possibly more the way to look at it because different people have different levels of expertise. So I think that’s probably a more useful way of looking at it. But I don’t know. What do you think, Melanie? Because I think Melanie could mention something about this.

Melanie: I agree. And I think it’s also the difference between expertise in AI usage and expertise in the domain itself. Practitioners who are experienced in their domain are more likely to recognise the limitations of AI and to check, and more able to check. For example, I was preparing for this webinar and I’d done the research myself and then put my notes into AI to just sort of tidy it up and spot the gaps. And it changed, it made one sort of the complete opposite of what it actually said. And if I just relied on AI, I would be sitting here presenting incorrect information. It was only because I’d read the source material myself that I knew that was incorrect and I could go back to that source and find the quote and challenge it.

Melanie: And we’ve seen things like an experienced coder is much more likely to spot the errors or security flaws in AI generated code, whereas a novice coder might be really experienced in using AI but doesn’t have the coding knowledge to recognise those errors and to be able to verify it. So I think we need to make sure we’re keeping that human knowledge of topics and not relying on AI for all of the information.

Bec: That’s part of the learning pattern though, isn’t it? Because I can’t remember when I first started using it, well over a year ago, but not long ago. But I’ve learned very quickly and it’s easier to learn quickly about AI when you’re the expert in the thing that you’re using it to do because like you say, you spot the mistake. That’s not quite true. We haven’t linked that with that. And actually that depends, and all of that sort of thing. It’s very easy. But if you haven’t got that level of experience, then it’s just back to that credibility thing again, isn’t it?

Melanie: There was also a study recently which showed that it’s actually younger generations who see more of the risk in AI and they were much more likely to self limit their use of AI. They had high confidence in their ability to use it, but they were choosing to limit their use mostly because of things like privacy risks that they were concerned about.

Bec: I think that’s partly about becoming digital natives. I think the expression is, because I’m definitely not one of those. We didn’t even have a computer at school. We could all touch when I went to school. So I think maybe they’re more tuned into that sort of thing than others. I don’t know.

Tim: I think it’s going to be a growing area of research because there’s previous research on cyber that seems to suggest that young people are maybe slightly more trusting of the tech and more likely to, certainly when they come into the world of business because they’ve been used to using it in a social way. And then in the world of business, people are trying to send them dodgy emails and they’re just a bit too trusting. So we’ve been really dancing around this idea of verification and critical thinking. Bec, how do you see that play into organisational risk? Is it creating problems for organisations?

Bec: I think it is generally because mostly human capability is largely ignored. And I think that the focus is all on what kind of tech do you have. Have we got the right processes in place, is the governance in place. We have policies and procedures and all this sort of thing. But I don’t really see that much emphasis on human capability. Do we ever measure how well people perform under pressure? Do we ever look at their decision making? I mean, you say that was a bad decision. Well, you can only know that in hindsight. So how do you, you know, what about sort of decision making skills where you consciously walk through and red team yourself. The critical thinking, critical reasoning, hugely important now. But like I say, I don’t know. How do we know what people’s capabilities are when what we measure is something completely different?

Tim: That’s a really interesting. I mean, we moan in the cyber security world that people aren’t thinking about the human enough. But you’ve taken that in a slightly different direction. It’s just like they say people, process, technology, but there’s just so little focus on the people, on the people side of the decision making.

Bec: And that’s something that people talk about when it comes to resilience all of the time, but you don’t talk about the people. So you might have resilient technology and resilient processes, but if you haven’t got resilient people, then you haven’t got a resilient system. And I think it’s just not that mainstream thinking. I mean, I’ve come across this numerous times in years of consultancy work where somebody will ring you up. So I need human factors evaluation. So, oh yeah, we’ve given you two weeks out of the 20 month project. And it’s like they expect me to come in with this wallpaper roll of human factors and just wipe people down with it and then write it all up.

Bec: They have no understanding that it’s about the actual behaviour in those circumstances, in this particular context. So I think there’s a huge misunderstanding and maybe that’s our fault as a profession for not really communicating it well enough. But you can’t just do the human factors because there is so much variation in the way that people behave and across context. And I think it’s just complicated. It’s just easier to think or just assume, yeah, people can do that because, well, that’s what people do, isn’t it?

Tim: That’s what people do. But it’s not, it’s a bit qualitative and not very quantitative. And Melanie, that kind of reduced verification, reduced critical thinking, that’s a problem for cyber security and decision making, isn’t it? I mean, that’s going to come through and cause us problems as well.

Melanie: Oh, definitely. I think as we said, when people are under pressure, they’re not thinking about the risks of, for example, I need to analyse this data, what’s the quickest way to do this. And then not thinking about whether they should be giving that information, that sensitive personal data to AI, or they’re not thinking about is AI the right tool to be making this decision.

Melanie: There was an interesting study recently where 25% of cyber professionals said they were expected to act on AI outputs that they didn’t understand. They didn’t understand where that information had come from, but just trust the system. And 89% of those professionals had experienced a wrong AI recommendation, but most of the time it was the human that was blamed. Despite the fact that they were under this pressure to just take what the AI told them, they were still blamed in the end. And some of them were saying that they now spend more time. So AI is actually creating more work because they have to spend more time checking the outputs than if they just did it themselves.

Melanie: What I think is also really interesting is that the messaging that pushes people towards AI usage is using the same tactics as the phishing tactics that we’ve been trying to protect them from. That fear, it’s going to take your job, the urgency, you’ve got to keep up with the pace of change, social pressure, the authority. And so this produces those same behavioural modes, either reflexive compliance, I’ll just adopt it without evaluating because that’s what I’m told to do, or I’ll just ignore it because I don’t understand. I’m too scared.

Melanie: And we get that contrast of some people being under pressure to use AI for the sake of it. Big companies that always token maximally towards, it’s just AI for AI’s sake, and other people hiding their use of AI. There was an interesting study where one in three people said they use AI covertly because it feels like cheating and they’re worried that people won’t take them as seriously if they say they’ve used AI. And this leads to this risk of shadow AI where people are using tools that the organisation don’t know about and can’t govern, which then creates huge risks because you just don’t know what’s going on. So I think this definitely has huge implications for security.

Tim: And we’re kind of touching on the AI side of shadow AI. It’s that kind of people trying to speed themselves up by not really thinking about where they’re putting data. And of course the other side of that is that AI is helping the bad guys make the bad things they already do better. So you’ve got the old threats on steroids and then you’ve got some new threats. What you said there managed kind of leads us on. You need to dig a bit deeper into that kind of underneath all of this, Bec. It’s the same humans. It’s still us. And I mean, is this creating new behaviours or is it really just amplifying what we used to do?

Bec: I think it’s just amplifying. I mean, it’s showing them up in a different manner because I think you mentioned at the start that the human brain has been the same for oh, ever. So we’re not, the way the brain works isn’t changing, the context it works within absolutely is. And I think that because it’s all new as well. And I was thinking, as you were talking, Melanie, you were saying about sort of pressure to use AI. And I remember seeing a comment on a LinkedIn post a couple of weeks ago. They said, oh, well, we’re now, the board has mandated use of AI and there was some sort of sarcastic comment beneath it around, but they haven’t thought about how to use it, how people are not going to use it and all of that sort of thing.

Bec: So I think it’s almost like it’s another kind of industrial revolution and we’re all just trying and grabbing onto things and doing various things with it. But nobody’s really sat and thought about how we’re going to use it because we’ve all been too excited to just get on and use it. And I think that it’s a bit like social media now. I mean, we’ve started to see reports about is it under sixteens not having smartphones and all of that sort of thing. Well, phones have been around since the mid 90s. So here we are a couple of decades later now finally managing to start thinking about the consequences. And I think that this is going to be a very similar thing. You know, we’ll all get on and do various things and then things will start to crumble and then bad things will happen and then we’ll start to think a little bit more about what we’re doing.

Bec: And I think also that sort of set off in mind then, the culture of the workplace and how that’s used. Because like you said, some people will be right full on off down that road. Some people will be, no, I’m not doing it. Some people, I’m going to do it, but I don’t want anybody to know I’m going to do it. So what is this going to do to the culture of the organisation? Because I think we’ve talked many times about culture and cyber security awareness. And I think that there’s equally a big part for psychological safety to play when we come into this. Well, we’re in the AI age.

Tim: That’s really interesting. And you talked more about people earlier, about people not really taking the human into account. But I think that broader company culture, psychological safety, generally, not enough thought goes into that generally. And the way that it will impact our working environment, the way we use tools and the organisational risk.

Melanie: We’ve seen things around phishing. We’re trying to encourage people to report and encourage people to report if they’ve clicked on a link by accident. We need to have that same culture around AI where it’s safe to be able to say I’ve made a mistake, I accidentally put some data in that I shouldn’t have done or I’m not sure about this tool. Can we have a discussion about this? We need that culture of psychological safety, where people feel safe to talk about that and don’t feel like they’re just going to be punished or that there isn’t a space to talk about that.

Tim: It’d be interesting to see if the slightly more regulated industries, maybe finance, where people have gone a little bit slow with the AI because they couldn’t, they had to kind of like, but we’re going to keep it blocked until we have a policy and we know how to use it. And they may have had a proper change project to actually say right now we’re ready to use it and this is how we’re going to use it. Whether those sorts of organisations and the people working there have more effective outcomes. I’m not sure how you would measure that, but I’ve been quite sticking into that. Because it’s like any IT adoption, isn’t it? Like it ought to have a change project rather than that little two weeks that you talked about on the human factor side of it. And do you think there’s also, I mean, we’ve talked about cognitive load, we’ve talked about the trust issue. Do you think that we’re more likely to trust these vaguely plausible answers because we’re under cognitive load? So that just adds to the problem.

Bec: Absolutely. Because that’s an immediate cue to the brain that you know it’s under pressure. And so unconsciously we’re making these decisions. And yes, you do trust it because you just want to get on. It’s that, what is called premature closure, I think, is the cognitive bias there, is that you just stop thinking of, that’ll do.

Bec: And we will do it. I mean, I did it yesterday. I was looking at going through Adobe Photos thing, trying to find bits for a piece I’m working on. And after an hour and a half of this, I am so done, you know, I am not interested in this task. It’s getting towards the end of the day and I’m bored with it. So basically I’m just like offski and I’ll come back to it. But it’s knowing that I’m doing that, thinking actually, do you know what? I’m going to dump this task to tomorrow and go off and do some bookkeeping because even that’s more appealing, which really never is.

Bec: So I think it goes back to that, how do we know ourselves? How do we understand that actually what I’ve just done, instead of finishing it and sending this thing to the designer, which I will regret two days later when I’ve just made a stupid decision, knowing that that’s what I’m doing is I’m fed up. My brain does not want to do this, but it doesn’t mean it’s finished.

Bec: And it’s, you know, if you know, it’s a bit left of bang. This is a phrase we used in the military. Is that what was happening before the thing? And I can feel my frustration getting up and rushing through the tasks. I recognise that they’re my left of bang cues, that now I’m switched off. I haven’t finished. I’ve just switched off. So therefore put it over there to go back to it some other time. And I think having the knowledge to run through that sort of framework is helpful.

Tim: And I think that is there a case for saying to people about building verification into their mental workflow when they’re working? So you know, you’re looking at something. How do we know what evidence supports this? What have we missed? What assumptions are we making? Again, it’s going back to that red teaming, but red teaming yourself, I don’t know what you call it, red teaming one, but yeah.

Tim: And that’s fascinating, isn’t it? Because that is also what you need for that cyber security behaviour. Like we talk a lot about stop trying to train people to look out for sort of tiny little cues in the URL being malformed and stuff. Just look for the tells like manipulation, urgency, scarcity. And so that metacognition of like, oh, I’m feeling anxious. That’s why I’m about to click this. That ability to, I mean, it’s very difficult because that’s exactly designed to play to your cognitive bias and make you click it, but that ability to slow down and think about how you’re feeling is going to help with AI, is going to help with cyber threats.

Bec: Absolutely. I mean, I had a phishing email and it arrived on a Friday afternoon and it arrived I think just after I’d done my first VAT return unsupervised because I’ve always had my bookkeeper around to supervise me. I’d done it unsupervised and it was a, oh, there’s been an error. You need to check, click this link. It will take you to your HMRC account, blah, blah, blah.

Bec: It was a Friday afternoon. I was massively busy trying to get something else finished, saw it, absolute panic ensued because HMRC scares the living daylights out of me. And what I did was write, I’m seeing my bookkeeper, forwarded it to her and said put this on the list to deal with and moved on. Get a phone call and it’s Kate and she just says do not click anything, don’t do anything. It’s gonna happen. I’ve just forwarded it to you. She’s no, no, no, it’s a scam and I’m still at, what?

Bec: And when I actually looked at it, the email address it came from was absolutely ridiculous. It was just so obviously a scam. RedFlags, pardon the pun, were waving all over the place at me and yet I didn’t see them. But that lesson and reflecting on that, to think, right, what happened, why did that happen enabled me to learn something about myself and how I act when I’m under pressure. And it’s the same thing again with, as we’ve just been talking about, that metacognition. You can do it, but it happens through reflection. But how many times do any of us ever want to spend the time after something’s happened doing that reflection?

Tim: Well, especially with a phishing sim where you realise you feel stupid, you feel humiliated, you feel embarrassed. And Melanie, I mean, you alluded to earlier, but our sense is that people maybe need a little bit of help at that point to kind of knock them into that more deliberative thinking about. Can you talk a bit about that?

Melanie: I think quite often we’re making those decisions in system one thinking, very quick reflexive action. Sometimes we need something that just nudges us into that slower, more thoughtful decision making that you’re saying about phishing sims, and also things like, I finished that task, great, move on to the next thing. Whereas we need something that helps slow down, actually no, first I need to check this output. And that’s some of the work we’ve been doing with our Redflags, particularly the trackers and the nudges, where what we’re hoping is that it starts to build things into your everyday workflow as habits.

Melanie: So when you’re downloading a new AI tool and you’re just rushing to, yep, click the button, click, click yes, agree to the policy, agree to, yep, give it permission, you get a little nudge that appears and says, don’t forget, only give it permission that it needs. And that just hopefully is enough to slow you down a little bit. And then you remember there was some information I read a couple of weeks ago about not giving up all the permissions it asks for to all the files and the company and all the data. And it just makes you pause and reflect.

Melanie: Or for example, the trackers can also then help with the visibility of behaviours. So things like tracking if people are signing into AI with non corporate credentials so that at least the organisation knows what’s happening and can start to think about what they need to do to change that. So I think it’s a combination of having the capability and the motivation the training can provide, the why it’s important and what the risks are. But we know that knowing the risks doesn’t actually translate to action at the time that that risk happens. Sometimes people need just that little bit of help to kind of wake up.

Melanie: And yes, nudges can supply that in the moment, prompt at the right opportunity when you’re making that decision under cognitive load, just a little nudge to get you on the right track, hopefully.

Tim: Take a breath, isn’t it really? And so just, I mean, we’ve been a little bit doom and gloom about AI. So what do we do about this? We want to build AI literacy, but we don’t want to lose critical thinking. We want to use these tools to speed us up. Is there a model of usage that works best, a way that you can use AI that makes it impactful, but it doesn’t kind of lead to some of these issues?

Bec: I think it’s not about the way you use AI, it’s about the people using it. It’s as simple as that. And I think that organisations that deliberately develop the human capabilities to manage AI and that AI can’t reliably replace are probably the ones that are going to be more successful with it. So thinking about judgement and critical thinking, do people actually know what judgement means? Do they know what critical thinking means? Do they know how to do it? What sort of verification things do you want to build into your habits?

Bec: So I think really, and again, it’s so similar to everything to do with security awareness really. We talk about security awareness process like, you know, it’s knowledge based. You learn this stuff, you remember it, you understand it, and then you start to work with it a little bit more, get a bit more complicated with it. And that’s all based on Bloom’s taxonomy, which is a knowledge of cognitive domain. Not many people understand that Bloom et al also wrote that affective domain. This to me is equally important.

Bec: So awareness, it’s just about, all of a sudden we know that this is a thing. So we’re aware of it. And then I start playing around with how to respond to it. Can I do, you know, can I do stopping the phishing links? Can I look at something critically? And then the more aligned you get with that, then you start to value the importance of it. Once you start to have that, it’s all about a mindset change and then you, it sort of starts to, you know, you value the importance of it. So you’re starting to look out for it and try and spot it because you know it’s important. Then that becomes automatic. It’s that habit that develops. And then you start preaching to other people about it. You know, it’s this evangelist, kind of. But it’s getting somebody from that one spot of just understanding it to evangelising it.

Bec: And there’s a number of steps. And interestingly, you see these things a lot in the research that has come out. There’s been a lot in clinical psychology about people who are addicted. So what is their path out of addiction? You see the same sort of path in counterterrorism, you know, so there’s paths of getting people to engage with material. And I think that’s again something else that is super helpful. But with security awareness, we tend to mill around in the sort of receive, respond, bottom two parts of that taxonomy without pushing it that little bit further. So I think people more trained, make it more like Netflix, and then they won’t complain too much.

Bec: I always think though, it sounds really stupid, but one thing that really struck me, I think it was last year when, was it a Netflix documentary about the Post Office scandal and everything that went wrong. All of a sudden it just exploded again. This thing’s been going on for donkeys years, but it was because people understood the human story behind it and it was in national knowledge, and then all of a sudden people looked at it from a very different perspective. So I think I’m not sure if I’m saying that we need to get this on EastEnders or something like that, but you know, it’s just explaining the narrative as well as the facts and the knowledge. Sorry, I went off on a bit then.

Tim: No, it’s interesting. And you’ve tied it really well into the awareness side. And we, I mean, I suppose we’ve talked a bit about, some of this is about helping build those habits in. So it’s not just about the knowledge side, it’s about the learned behaviours and embedding those habits into workflows. But I mean, you talked, I think, I can’t remember who said, you talked about that kind of didactic thing as well. So if people are going to go away from here and we made them think, oh God, AI is evil. Like there’s a way to use it, isn’t it? It’s maybe as a coach or a knowing thinking, rather than as a kind of, I’m going to fully offload my critical thinking to you.

Bec: I think that sort of, it could be a tutor because it can’t, you know, it’s a brainstorming partner. It can be a thinking partner. I think it’s more about learning to engage with it rather than just be fully on the accepting side. And I think that that’s something, as I said a bit earlier, it sort of comes the more you use it, the more you understand not necessarily the background of how it works, but what it’s going to do that’s going to annoy you and why is that annoying you and all of that sort of thing. So I think that the more we use it, the more used to it you get, but people are already using it and that’s not going to stop. So I think the biggest question for me is how do we strengthen that human judgement while people are using AI? So it’s not really so much the AI, the people again, isn’t it?

Tim: A bit of backpedalling. So we have had some questions and comments come in. I’m just going to try and make sense of them. So we’ve got here from Jared, isn’t the rise of AI another iteration of information digital literacy from an information consumer standpoint? We touched on Google in the beginning. We trusted anything Google told us. And then we started learning about SEO and keywords. And maybe we started to become a bit more cynical. So we need to learn how to use AI to kind of be sure that it’s giving us valid information. I think that’s a fair point. Any comments on that?

Bec: Absolutely. I think also it’s what to use it for and if how, isn’t it? It’s not just about blindly doing everything on it. I mean, AI, Google AI searching is brilliant. If you’re shopping for something or if you want to change your electricity supply, absolutely, bomb on. But that’s one thing. Using it to make big decisions on information that’s changing, you know, it’s something else.

Tim: It’s the critical thinking and the expertise aspect, isn’t it, of kind of are people being encouraged to do that challenging and are they or are they just taking it all on trust? I mean, I’ve sorry, Melanie.

Melanie: I think it’s building on what we’ve already started to doing around things like media literacy. You know, don’t just trust everything you read online. Don’t trust everything that is fed to you on social media. Like I think you mentioned earlier, maintaining a healthy amount of cynicism and it’s just expanding that into also AI. Don’t trust everything that AI tells you. I think it’s not a new skill, but it’s one that has been underdeveloped for quite some time. I think it’s fairly recently that schools have started to teach that media literacy and digital literacy.

Tim: So I think we need to, I think that there’s, and it’s not like one size or a simple answer, isn’t it? Because I think you could say, well, if it’s a simple problem, then you can trust the AI. But I was trying to look up something fairly simple. I couldn’t find the manual for something and it was how to reset something, and I was getting told this answer, which was completely wrong. And you drill into it and I was trying to use one bit of technology from one vendor. And it was finding the answers from another manual from another vendor. And so the steps it confidently told me, press do ABC, just simply were complete nonsense. And now it’s quite a simple task. And yet maybe we trust him more on the complicated ones.

Tim: Let’s see what other questions we’ve got here. OK, AI has lots of gaps. Technical detail needs to be given using clear, non conflated prompts. This requires you to know your stuff. It needs directing to outcomes versus just working. Absolutely.

Bec: OK, I found Prompt Cowboy. He’s very useful for that.

Tim: Cowboy I haven’t come across. Tell us more about that.

Bec: I know somebody I know who’s rather into AI. So you’ve got to use Prompt Cowboy because actually when I did a prompt engineering course, I really need to bother, but I put my clunky prompt into this is what I want to know. And it turns it out into a proper prompt.

Tim: Oh, wow. OK, because it’s helping you write AI to help you write your prompts.

Bec: Yes. But the output has been far superior to anything I could have managed.

Tim: OK, I’ll start using that one. That sounds a good one. But sorry, really interesting. No, no, really interesting, almost philosophical question here, which I love. So is AI going to ultimately, is it going to create more space for human creativity and deeper thinking, which perhaps some of the people selling it promise us? Or is the convenience actually going to make it harder for people to kind of tolerate that messy bits of learning, the uncertainty, the struggle, and not immediately knowing the answer? So is it going to help us be more creative? Or is it actually going to just make life too simple so we won’t bother?

Melanie: I think it’s really interesting seeing how we’ve already come from the beginning when the promise was that AI will give you more time to be creative. And then suddenly AI was doing the creative stuff. AI was making the art and the music. That’s not the thing I want AI to do. So we’ve already seen a shift there. But then I think we’re already starting to see a shift towards more considered use. I think eventually the study where 42% of particularly the younger respondents were already limiting their use of AI. So I think that we’re already starting to see people realising that just adopting AI for everything is not the answer. And what we need now is to start thinking about how do we retain the skills that we want to retain and not offload those skills.

Bec: I think I have a slightly different view on creative thinking because to me, in working in organisations with decision making and judgement and all of that sort of thing, creative thinking isn’t necessarily the typical way of creating, I would say about art and music and that sort of thing. Creative thinking is something that’s really important with problem solving and judgement and decision making. So actually what I was, that I worked with the senior leaders in the military, was creative thinking.

Bec: There’s four aspects to it. And one of those aspects is what we call incubation. So that is time. So when you want a solution to something, you can rush and come up with one. But if you ask you or whoever you’ve delegated to, don’t come back to me this afternoon at 4:00, come back to me tomorrow lunchtime, because that knowledge will be settling down and consciously you’re working on it. And then when you look at it in the morning, it will be different. You’ve probably had that experience when you’ve written a document and then you go back to it two days later before you send it off and there is like 100 things wrong with it. So stepping back gives you that ability.

Bec: And I think for me, you can still do creative thinking using AI, but it’s again, how you use it. So do you have that session where you do some brainstorming, you come up with lots of different ideas and then you stop and you come back to it later on in the week. And then actually your brain will have been incubating and cogitating on it subconsciously and then you’ll see it in a different light. So for me, that’s where the creative thinking comes in. Yes, it can happen with AI, but you do need to take, you have to do it in the same way you do your normal thinking. So you have to engage with it by leaving it be and then coming back and seeing it from a different angle.

Melanie: It’s also about the way we craft the prompts, whether we’re using it, just give me the answer, or whether we can use it to ask me questions about this topic to help me develop my understanding. That sort of more Socratic dialogue approach where it’s actually as a tool for learning, not just a tool for understanding. Am I using it to just get some information or am I asking it to help me understand and question me rather than understanding the difference between those things, which takes us all the way back to the beginning and with self awareness.

Tim: There’s an interesting question of whether, if the AI is better at thinking, whatever that means, that it doesn’t need that incubation period. And maybe it depends on what you’re thinking about. And that one of the things that I found, there’s a funny meme I’ve seen going around this idea that Mary Shelley would never have written Frankenstein, AI would have pushed her towards the generic average of what was being produced by women writing fiction at that point in time. And Einstein would have just been told no, Newtonian physics says this, shut up. And so it’s just kind of these lateral thinking, these jumps where it’s not average and it’s relating to things that don’t seem connected. That’s where the human intelligence comes. Whereas that won’t come unless you add seeds, some randomness into the AI, that’s just not going to come from there. So maybe it’s types of thinking that’s interesting.

Melanie: Because I think we also need to remember what these large language models actually are and what they’re not. They’re actually just delivering the next statistically likely piece of information based on what they already know. They’re not actually doing that lateral thinking that you were talking about, Tim, you know, discovering new concepts or certainly the sort of generative AI that most of us are using isn’t. So it’s about how they’re used. And also it’s retaining what is that human thing that only we can do?

Tim: Brilliant, right. I’m going to have to wrap that up because we’ve got 4 minutes left. So look, I mean, some key takeaways, I think that from all of that. I mean, I think one and the kind of the grounding of it is that we’re cognitive misers and we’ve always sought to offload things and that actually leads us to make cyber mistakes. But it’s also AI makes that offloading really easier and more compelling. So that’s kind of one of our key themes.

Tim: I think you’re more likely to fail to check the AI answers under pressure, just as we see with cyber risky behaviours. And obviously one of the focuses there is, well, how do you help people prompt them into that kind of thinking, prompt them into the metacognition that Bec’s highlighted, which is a fantastic kind of concept, I think, because there are these real dangers of over trust, trust without verification.

Tim: But I hope we ended on a bit of a positive that actually there are ways that we can use AI properly. We can kind of use it as a tutor coach, a brainstorming partner, not a replacement thinker, but someone to support us. And I hope, I suppose your key point, Bec, was that we need to encourage organisations to think about that side of things and train people to do that. Otherwise it’s really not going to happen.

Tim: Fantastic, thank you so much and I really appreciate. That’s been fantastic having you here, Bec and Melanie, and thank you to our audience and your engaging questions. So if I didn’t get to all of the comments and questions, thank you for joining us on our summer series. I can put my sunglasses and hat back on and get back out into the sunshine. So thank you very much. It’s been really good to have you here.

Bec: Thank you.

Melanie: Thank you.

Tim: Cheers. Bye, bye bye.