Research and considerations on the use of AI in coaching.
Some Poetry
There once was a great coach named Lir
Whose advice was always sincere
But ChatGPT came along
With its knowledge so strong
And they both coached their clients with cheer!
Yes, that was written by ChatGPT, not by me. And while it’s not a brilliant limerick – that last line is particularly painful – it’s good enough.
Does the same apply to coaching? Are bots ‘good enough’ to improve human’s lives?
Can a machine “partner with clients in a thought-provoking and creative process that inspires them to maximize their personal and professional potential” (ICF coaching definition)?
Short answer – kinda.
- Bot coaching is better than no coaching, and even has some advantages over us humans.
- It’s also better than self coaching – as you’re not constrained only to the questions that you think of yourself; an external source pushing you to answer outside questions will usually be better.
- But human coaches still win. Bots just don’t have the ‘genius’ of human interaction – they can respond correctly and sensibly to a pattern, but don’t have the flair of insight. And we don’t value interaction with a bot as much as we do with a human coach.
I’m safe, at least for now 🙂.
Chat GPT for beginners
Background for the sensible majority who know little about Chat GPT: the Chat Generative Pre-Trained Transformer – Chat GPT to its friends – launched just months ago at the end of Nov 22. And the tech world gasped in awe.
It’s an artificial intelligence (AI) chatbot (bot). You’ve probably come across these many times over the years. If you ever wanted to throttle Microsoft’s Clippy as it patronisingly said “It looks like you’re trying to write a letter”; or if you’ve sworn down the phone as the voice recognition system totally failed to understand the simplest of requests – that was what you were dealing with.
Enter Chat GPT. Every other AI bot in the world blushes, shuffles feet and plans its retirement – because this is a whole new beast. Interacting with Chat GPT is, if anything, better than interacting with the average support line human over chat. Its responses seem to show understanding; it generates well formed answers to the most peculiar questions – such as the request to write a limerick on the competition between a human and bot coach. The difference in quality of answers between this and all predecessors is, simply, stunning.
If you watched the film “The Imitation Game”, you might remember the Turing test: while the question of whether machines can think is “meaningless”, the definition of when a machine can be considered “intelligent” is when a human cannot distinguish whether they are talking to a human or a bot. Chat GPT passes this test with flying colours: the bot is clearly “intelligent” by this definition. It can pass exams and write essays to an extent that is causing academics globally to wring their hands.
And causing the rest of us to sit up and think about the impact on our own careers. Developers worldwide have discovered with delight – or fear – that it can accelerate their work a hundred fold, stubbing out hours of work, in mere seconds. Those less at the forefront of technology are catching on also – see this Forbes article, for example, on how business leaders can use Chat GPT to improve their business – customer service, content creation, report generation and many other areas.
So, brilliant. But it comes with significant limitations that are worth keeping in mind before we move on to the impact on coaching:
- It is – in Benedict Evans words – a ‘confident bullshitter”. It ‘hallucinates’ entirely plausible answers, without factual grounding. For example, I asked it for research on the topic of coaching and Chat GPT. It gave me some great sounding stuff from reputable journals, neatly summarised. But none of the research actually existed (see transcript for details).
- Ethics, privacy and other fundamental concerns. This topic deserves a whole article. Open AI – the company who created this – has a supposedly benevolent mission “to ensure that artificial general intelligence benefits all of humanity”. But founders and sponsors like Tesla’s Elon Musk, Paypal’s Peter Thiel, and significant investment from Microsoft give cause for concern. And even without them, the point where AI is smarter than humans seems worryingly close; even a benevolently intended misstep feels likely to be dangerous.
With that in mind, fire ahead and play with Chat GPT yourself – it’s free, though occasionally gets too busy so you can’t always get in.
So – how good a coach is Chat GPT?
I’ve tested the bot with a few typical questions and scenarios – full transcripts below.
The first questions I asked received recommendations rather than actual coaching – it felt like reading typical articles on the topic. Reasonably helpful information – like reading an article on the specific topic of interest – but not ‘coaching’, as you’ll see in the transcript.
I then instructed it on how to coach me, telling it to ask a sequence of questions going deeper into each response. This was reasonably successful with a decent practical outcome for a practical topic – confidence in giving presentations. Transcript below.
Then, using the same instructions on how to coach via questions, I went into an actual coaching topic, something of emotional importance to me, which a human coach might well have helped me to reach much deeper into myself about. While I’m not willing to share the transcript for that, I overall felt it had more in common with the ‘practical’ topic of ‘confident presentations’ above, than a ‘real’ coaching interaction. It definitely helped me articulate some slightly buried issues more clearly – but I am certain a human coach would have led to a much more significant outcome.
The fundamental barriers with Chat GPT as a coach stay the same:
- Text based. The coaching is typing based, rather than spoken. This might change – I can imagine a video call with an AI-generated human-seeming coach in our near future – but for now, it’s limited to text input.
- This fundamentally shifts the coaching experience – the words you type, are different to how you’d express it in a spoken interaction.
- That said – having to write your coaching responses is not wholly bad; you can spend more reflective time and articulate a thing more precisely than you would in the flow of conversation. That’s why journaling is generally regarded as a beneficial practice, and having tough questions to answer in your writing does help, whether it’s from a bot or not.
- However, most people would agree that spoken interactions are superior. A human coach, in person or over video, can pick up on the client’s tone and body language and whole way of being – all of which are missing if they’re coaching over text.
- Responsive. The bot responds only to what’s ‘there’ in your text and has no capability of sifting through the context to identify what’s important that’s missing. It is a very literal machine, and works with the words you give it.
- It could potentially improve over time, to pick up on subtle cues and spot patterns within what’s said better than humans, having perfect recall.
- Human coaches, however, are likely to stay far better at interpreting information from the wider – human – context, and spotting what the coachee is avoiding, minimising or deflecting. For now, at least..
- Empathy. It’s possible that as technology improves, we’ll have more respect for bots – and they’ll have earned it, demonstrating something that feels like empathy. At this point – no; it’s impersonal and generic. You don’t respect the questions and you’re less likely to keep the coach’s wise words in mind if it’s a bot.. And that has impact on the coaching experience.
- At its best – the bot’s questions prompted me to think, and to reach into myself to articulate a difficult answer. That’s pretty good.
- But ultimately – it felt ‘soulless’. I was conscious all the time that I was dealing with a bot which didn’t fundamentally ‘get’ me; it doesn’t empathise with me as a person and only reacts to words. This is a barrier to transformational coaching – though, notably, less of a barrier to practical coaching. I can’t see myself taking into myself the words a bot used – whereas I do, with a coach. Human interaction responding to our unique selfs, simply makes a bigger impression on us than words on a page by a pattern-spotting bot.
- However – it’s worth bearing in mind that this could sometimes be a strength. Some people might be willing to open up to bots in a way that they mightn’t for a human. An interesting 2014 paper (Gratch) suggests that in a medical context, “participants who believed they were interacting with a computer reported lower resistance to self-disclosure… than those who believed they were interacting with a human operator”
- Imagination. The big fuss is because it’s the first time bots can ‘generate’ human sounding text. And it does follow very sensible patterns; can imitate well – see this example of Bing’s AI doing creative writing, then using Vonnegut’s rules to improve its own work. But it entirely lacks human’s creative flair and imagination – Vonnegut’s actual output would have been far superior. Similarly, I asked Chat GPT to draft the outline for this article after I’d done so myself. It drafted a sensible outline – but I think mine was better. And, in the creative human way, I proceeded to largely ignore the outline as I wrote, relying more on inspiration and the creative rabbit hole as I explored the topic.
- The impact of this lack of creativity on coaching means that the coaching questions it asks are very much ‘by the book’. Nothing wrong – but nothing surprising or insightful.
- Again, however – ‘by the book’ is still pretty good. Even using this approach, a human can still make some progress. They’re just less likely to achieve breakthroughs or rapid progress.
What does the research say?
Most research is pre-Chat GPT release, of course, but still fairly relevant; it’s mostly over the last few years that AI has been good enough for any form of coaching so still very new. Some papers that I particularly liked:
- A 2023 Stanford working paper by Kapoor, Narayanan and Manchanda indicates humans are still winning – “access to human coaches leads to higher weight loss than with AI coaches alone”. This was a pretty wide study, on 65k participants – so fairly significant.
- This 2022 paper by Terblanche, Molyn, de Haan and. Nilsson suggests parity – it compared two different studies, one of which used AI and the other used human coaches – and found the AI coach was as effective as human coaches at the end of the trials.
- Graßmann and Schermuly’s 2020 paper suggests that “AI coaching seems to encounter the greatest difficulties in the clients’ problem identification and in delivering individual feedback, which may limit its effectiveness.” I’d argue this is a failure of both imagination and empathy. Some things still need humans. However, as they say, AI is capable of guiding clients through many steps in the coaching process – a bot can indeed help.
- Interesting 2022 paper by Mai, Neef & Richert shows that for a practical topic – exam anxiety – a chatbot was reasonably successful, and that even a click-based ‘select your response’ approach had value.
The good coachbot
So, while I have plenty of reservations about AI coaching – the lack of empathy and imagination (and humanity!), the medium of text, and that it can only respond to what’s there rather than what’s missing – there are some brilliant things about AI coaching too:
- Cheap. Anything that brings more help to more people is good. This is a hugely scalable way of making practical coaching affordable.
- Practical. Research so far indicates bot coaching is fairly good for well defined problems, where you have a clear goal and are motivated to change – lots of common issues, from weight loss to exam anxiety, could be helped. The evidence all indicates it is better than no coaching, and there are no indications of damage or negative outcomes.
- Available. It’s there when you want it, for as long as you want it – instead of having to set aside an hour of predefined time once a fortnight.
- Consistent. Even the best human coaches might have bad days – and not all coaches are good or highly skilled. Machines do good consistency – and people will trust in their reliability.
So, there is the potential for the AI to do much better as a coach in the very near future. There are some benefits – some people may find disclosure easier; the AI’s focus on goals is good; and it should be better than humans at detecting clients’ patterns over time.
What next?
Here’s what I’d love to see next
- Dedicated, freely available coachbot for all – potentially part of the NHS, in improving health / behavioural outcomes.
- Chat GPT as-is is a bit too tricky to really use for coaching – I had to tweak my inputs a lot to start getting good coaching output.
- The bot would need to be trained on coaching specifics and transcripts, with human safeguarding to prevent abuse and ensure it flags up worrying conversations – this would be an essential precursor to a public system
- The system would also need to offer support for professionals who are well placed to spot those in need of this – eg HR, educators, health workers etc. So they can recommend it when appropriate, with a clear idea of its capabilities and limitations.
- Coach support bot
- The bot could be brilliant between-session support for coaches, prompting the coachee with actions or questions between sessions, and helping them prepare for the next session by clarifying their goals for it. Its constant availability would be a bonus too, allowing the client access to constant support.
- The bot could do transcriptions of the coaching sessions, summarise and analyse them – looking for patterns of words or topics, which machines are particularly good at.
- The bot could personalised per client by the coach, and use different approaches to interaction for each client, so they feel more ‘human’ and give a better experience
- A bot with access to all the latest greatest insights in coaching and psychology research, could help inspire a coach with personalised suggestions and ideas, for their general practice and for individual clients. Most coaches read a lot on the topic; this could be better than random browsing or serendipitous discoveries.
I suspect that the bots will fill the ‘free/cheap’ gap and help many people that way, and that there is still going to be plenty of demand for human coaches for many years still. And hopefully, it will help us coach our clients even better.
Chat GPT’s response transcripts:
First attempts – confidence and stress management
Question:
Coach me on how to feel more confident.
Response:
Feeling more confident is a common goal for many people, and it’s great that you’re looking for ways to achieve this. Here are some strategies that can help you feel more confident:
Identify your strengths: Make a list of your strengths and accomplishments, and focus on them. Acknowledging your positive qualities and what you’ve achieved can help boost your self-esteem and confidence.
Practice self-care: Taking care of yourself both physically and mentally can help you feel more confident. This includes getting enough sleep, eating a healthy diet, and exercising regularly. It can also mean practicing self-compassion and avoiding negative self-talk.
Set achievable goals: Setting and achieving goals can help you feel more confident in your abilities. Start by setting small, achievable goals, and gradually work towards larger goals as you build confidence.
Learn new skills: Learning new skills and taking on challenges can help you develop your self-confidence. Consider taking a course or workshop in a new area, or challenging yourself to try something you’ve never done before.
Surround yourself with positive people: Spending time with supportive and positive people can help boost your confidence. Seek out friends and mentors who will encourage and support you, and who believe in your abilities.
Remember, building confidence takes time and effort. Be patient with yourself, and don’t be afraid to seek support and guidance as you work towards your goals. As your coach, I’m here to help you navigate any challenges you may encounter along the way.
I tried again – stress management:
Question:
I need help developing healthier habits for managing stress.
Response:
Of course, I’d be happy to help! Developing healthy habits to manage stress is an important part of taking care of your overall well-being. Here are a few strategies that may be helpful for you:
Practice mindfulness: Mindfulness is the practice of being present in the moment and fully engaged in your surroundings. When you feel stressed, taking a few minutes to practice mindfulness can help you feel more centered and calm. You can practice mindfulness through deep breathing exercises, meditation, or simply by focusing on your breath and letting go of distracting thoughts.
Exercise regularly: Exercise is a great way to manage stress and improve your overall health. Aim to get at least 30 minutes of moderate exercise most days of the week. This can include activities like jogging, swimming, or cycling, or even just taking a brisk walk around your neighborhood.
Get enough sleep: Lack of sleep can increase stress levels and make it harder to manage your emotions. Aim to get 7-8 hours of sleep per night to help you feel more rested and less stressed.
Eat a healthy diet: Eating a balanced diet that’s rich in fruits, vegetables, lean protein, and whole grains can help you manage stress and improve your overall health. Avoiding processed foods and excess sugar and caffeine can also help you feel more balanced and less anxious.
Connect with others: Spending time with loved ones and participating in social activities can help you feel more supported and less stressed. Make time for regular social activities, whether that’s joining a club or group, meeting up with friends, or simply spending time with family.
Remember, developing healthy habits takes time and effort. Start by incorporating one or two of these strategies into your daily routine and gradually build on them as you feel comfortable. Don’t hesitate to reach out to me for further support and guidance as you work towards your goals.
Third attempt – teach it to coach using questions:
My prompt:
Coaching is best done by asking someone questions, rather than offering them suggestions. Coach me, using questions, to feel more confident. Await my response to each question, and then ask another question that builds on my response.
Chat GPT responses:
Great, let’s try coaching through questions! Here are some questions to help you explore your confidence:
What does confidence mean to you?
When do you feel most confident in yourself?
What situations tend to make you feel less confident?
What self-talk do you use when you feel less confident?
What are some examples of times when you have successfully overcome challenges?
What small steps can you take to build your confidence gradually?
How can you celebrate your successes, no matter how small they may seem?
Who are the people in your life that you feel most supported by and can turn to for encouragement?
What can you do to practice self-compassion and avoid negative self-talk?
What would you like your confidence level to be in the future, and what steps can you take to get there?
Remember, the answers to these questions are personal to you, and there are no right or wrong answers. As your coach, my goal is to help you explore your thoughts and feelings around confidence and develop strategies that work best for you.
My prompt:
Instead of asking me all these questions in a list, please ask a single question. Review my response, then ask another question that goes deeper into my previous response.
Chat GPT:
Sure, let’s try a different approach. Here’s a question to start with:
What is one situation or task that you feel particularly unconfident about right now?
Me:
I have to give a big presentation to a lot of people soon
Chat GPT:
Thanks for sharing that with me. It’s natural to feel nervous about giving a big presentation. Let’s explore this further – can you tell me more about what specifically makes you feel unconfident about the presentation?
Me:
I’ll make lots of mistakes and look like a fool
Chat GPT:
It sounds like you’re concerned about making mistakes and how that might impact how others perceive you. Mistakes can be an opportunity to learn and improve, but it’s understandable to worry about how others might react.
With that in mind, let me ask you this – what steps can you take to prepare for the presentation and feel more confident about your ability to deliver it effectively?
Me:
I can prepare thoroughly and rehearse it many times
Chat GPT:
Preparing thoroughly and rehearsing your presentation multiple times is a great way to build confidence.
Now, let’s dive a bit deeper – what specific steps can you take to ensure you are fully prepared for the presentation? For example, do you need to create an outline, gather data, or practice your delivery with someone else? What steps can you take to feel fully prepared?
I abandoned the conversation at this point – it’s an ok practical coaching conversation.
Chat GPT’s ‘research’ on this topic
CAVEAT: None of these seem to exist, and all this ‘research’ predates Chat GPT’s existence. Note that Chat GPT is limited to older data sources and is usually a few months out of date, so it probably couldn’t have found the few articles that do exist anyway as they’re all very recent.
Me:
What research has been done into using chat gpt for coaching?
Chat GPT:
A study published in the Journal of Educational Psychology (2021) explored the use of a Chat GPT-based coaching system for improving writing skills in college students. The researchers found that students who received coaching from the Chat GPT system made significant improvements in their writing skills compared to a control group.
Another study published in the Journal of Experimental Psychology: General (2020) examined the use of a Chat GPT-based virtual coach for helping people set and achieve goals. The researchers found that the virtual coach was able to effectively guide participants towards achieving their goals, and that participants reported high levels of satisfaction with the coaching experience.
A study published in the International Journal of Human-Computer Studies (2020) explored the use of a Chat GPT-based conversational agent for providing mental health coaching to college students. The researchers found that the conversational agent was able to provide effective coaching and support for a range of mental health concerns.
A study published in the Journal of Medical Internet Research (2021) examined the use of a Chat GPT-based virtual coach for helping people with diabetes manage their condition. The researchers found that the virtual coach was able to provide personalized coaching and support, and that participants reported high levels of engagement and satisfaction with the coaching experience.
Overall, these studies suggest that Chat GPT-based coaching systems have the potential to be effective and well-received by users. However, more research is needed to explore the effectiveness of Chat GPT coaching across different domains and populations, as well as to address potential ethical concerns.
Me:
Where are these studies – give me links to the original sources
Chat GPT:
Study on using Chat GPT for writing coaching: https://psycnet.apa.org/record/2020-70169-007
Study on using Chat GPT for goal-setting coaching: https://doi.org/10.1037/xge0000997
Study on using Chat GPT for mental health coaching: https://doi.org/10.1016/j.ijhcs.2019.06.008
Study on using Chat GPT for diabetes coaching: https://doi.org/10.2196/21140