According to the M365 Copilot monitoring dashboard made available in the trial, an average of 72 M365 Copilot actions were taken per user.
“Based on there being 63 working days during the pilot, this is an average of 1.14 M365 Copilot actions taken per user per day,” the study says. Word, Teams, and Outlook were the most used, and Loop and OneNote usage rates were described as “very low,” less than 1 percent and 3 percent per day, respectively.
Yeah that probably won’t have the intended effect…this basically just shows that AI assistants provide no benefit when they’re not used and nothing else.
Maybe, maybe not we actually have no idea as the article doesn’t mention it. Nevertheless, doing infrequent queries is an equally likely scenario, given that people are really bad at changing their habits and existing workflows regardless of potential benefits.
Thats complete speculation on your part though. It could equally be people hardly used it at first then started to use it more as they found ways it was helpful. Unless you see the data there’s no reason to say one or the other.
Probably, my point was that you cant say if its increasing, decreasing or staying constant just from the number of times it’s been used. It could be that for most people its completely useless but for a small group its very usefull and they are using it more and more. Or as suggested it could be that everyone tried it a bit at first found it useless and stopped using it. Or that its kinda useful in very specific cases so it gets constantly used a tiny bit.
What differece does it make if a small group finds it useful? They’re saying as an organization, there was no overall improvement in productivity. It’s such a weird point to make and to defend. They did an assessment and it didn’t deliver, that’s it.
I’m not defending it or attacking it, mearly saying that
They probably did multiple queries per day at the beginning, found out it isn’t worth it and stopped using it …
Isnt supported by the information given. The GP gave a story they made up about how usage would be falling based on nothing at all, I gave two other alternate stories about how it could be either rising in usage or remaining flat to demonstrate that we cannot say anything about rate of change from a single average.
If you force someone to use a given solution, they’re going to push back until they see value. And people aren’t going to put in the effort to learn something they have initial misgivings about.
LLMs can be useful tools, or they can be a terrible waste of time. It all comes down to what tasks you use it for and how motivated you are to get it to work. I use it occasionally (maybe 2-3x/week), and I find it saves me time when I do. I don’t use it for tasks I know it’ll be poor at, because that’s a waste of time.
You assume the average usage is representative of the actual usage. You averaged the actions over the time period, nothing’s says the users didn’t performed the averaged 72 actions within the first three days or any time restricted window within the whole period of time and got bored with it seeing no or low value.
It’s also possible a handful of power users use it a ton and found value, while the quiet majority only used it a few times because they were required to and didn’t see value.
We need more details to draw conclusions. For example:
what types of tasks did people use it for? What roles did they have?
what does the distribution of usage look like? What’s the median number of uses? What’s the average of users within one standard deviation?
were they forced to meet some quota of uses, or were they left to choose on their own?
I’m not a programmer, so it’s got nothing to offer me. Mostly my job is to write documentation for propriety software and hardware, stuff the AI knows nothing about, not everyone in the world can mak use of AI, and it doesn’t require a PhD and 30 days of constant usage to work that out.
I’m not saying AI specifically is useful, just that people in general tend to resist change in their work methods regardless of what they are.
I also work with a lot of proprietary knowledge, chemical and infrastructure in my case, and AI still can be useful when used properly. We use a local model and have provided it with all our internal docs and specs, and limited answers to knowledge from these, so we can search thousands of documents much faster, and it links to the sources for it’s answers.
Doesn’t do my job for me, but it sure as shit makes it easier to have a proper internal search engine that can access information inside documents and not just the titles.
Then maybe it’s not useful for you. That doesn’t mean AI isn’t useful for a number of other roles.
I’m a software developer and find its code generation to be awful, but I also find that it’s great at looking up technical information. Maybe I’m looking for a library to accomplish a task, and I want to compare features. Or maybe I’m having trouble finding usage examples for a relatively niche library. Those are task the AI is great at, because it can look at tons of blog posts, stack overflow questions, etc, and generate me something reasonable that I can verify against official docs.
If my workflow was. mostly email and internal documentation, yeah, AI wouldn’t be that useful. If my workflow relies on existing documentation that’s perhaps a little hard to find or a bit poor, then AI is great. Find the right use case and it can save time.
“Regular office work” is a pretty broad category. Yeah, it’s probably not useful in retrieving records for someone or processing forms, but it should be useful for anything that requires research.
I don’t know much about copilot, but some quick research shows it uses GPT-5 for the chat feature. I assume that’s what’s meant by the average queries in the article.
Having been part of multiple projects introducing new software tools (not AI) to departments before, people are usually just stubborn and don’t want to change their ways, even if it enables a smoother work-flow with minimal training/practice. So yeah, basically people are so set in their ways,it is often hard to convince them something new will actually make their job easier.
The devil is in the details… what you describe screams to me what I call the “new boss syndrome”. New boss comes in and they feel the need to pee on everyone to mark their territory so they MUST bring in some genius change.
99% of the time, they are bringing in some forced change for the sake of change or something that worked on their previous place without taking into consideration the context.
I do not know anyone who prefers to work harder… either the changes proposed make no sense (or it’s too complex for people to understand the benefit) or the change is superfluous. That is usually where resistance to change comes from.
In all your software deployments did you blame the users for not getting it or did you redesign the software because it sucked (according to your users)?
I’ve occasionally been part of training hourly workers on software new to them. Having really, really detailed work instructions and walking through all the steps with themthe first time has helped me win over people who were initially really opposed to the products.
My experience with salaried workers has been they are more likely to try new software on their own, but if they don’t have much flexible time they usually choose to keep doing the established less efficient routine over investing one-time learning curve and setup time to start a new more efficient routine. Myself included - I have for many years been aware of software my employer provides that would reduce the time spent on regular tasks, but I know the learning curve and setup is in the dozens of hours, and I haven’t carved out time to do that.
So to answer the question, neither. The problem may be neither the software nor the users, but something else about the work environment.
That’s not what I’m asking. You designed or built something for some users. They didn’t like it, or didn’t use it as you expected. Was your response to change the software or blame the users for not using it correctly?
That depends on the issue. Sometimes it’s a lack of training, sometimes it’s obtuse software. That’s a call the product owner needs to make.
For something like AI, it does take some practice to learn what it’s good at and what it’s not good at. So there’s always going to be some amount of training needed before user complaints should be taken at face value. That’s true for most tools, I wouldn’t expect someone to jump in to my workflow and be productive, because many of the tools I use require a fair amount of learning to use properly. That doesn’t mean the tools are bad, it just means they’re complex.
We have it on our system at work. When we asked what management expected it to be used for they didn’t have an answer.
We have a shell script that ingests a list of user IDs and resets their active directory passwords, then locks the account, then sends them an email telling them to contact the support desk to unlock the account. It a cron job that runs ever Monday morning.
Why do a need an AI for when we can just use that? A script that can be easily read understood and upgraded, with no concerns about it going off-piste and doing something random and unpredictable.
So yeah, they don’t use it, because it won’t work.
Well yeah, AI shouldn’t replace existing, working solutions, it should be used in the research phase for new solutions as a companion to existing tools.
Yeah that probably won’t have the intended effect…this basically just shows that AI assistants provide no benefit when they’re not used and nothing else.
People probably tried it, found out that it’s crap and stopped using it.
Its hardly possible to actually test it properly in relation to your work and changes in productivity with a single query per day. It
They probably did multiple queries per day at the beginning, found out it isn’t worth it and stopped using it …
Maybe, maybe not we actually have no idea as the article doesn’t mention it. Nevertheless, doing infrequent queries is an equally likely scenario, given that people are really bad at changing their habits and existing workflows regardless of potential benefits.
Thats complete speculation on your part though. It could equally be people hardly used it at first then started to use it more as they found ways it was helpful. Unless you see the data there’s no reason to say one or the other.
Ok but if it was actually useful, wouldn’t people actively engage with it?
Probably, my point was that you cant say if its increasing, decreasing or staying constant just from the number of times it’s been used. It could be that for most people its completely useless but for a small group its very usefull and they are using it more and more. Or as suggested it could be that everyone tried it a bit at first found it useless and stopped using it. Or that its kinda useful in very specific cases so it gets constantly used a tiny bit.
What differece does it make if a small group finds it useful? They’re saying as an organization, there was no overall improvement in productivity. It’s such a weird point to make and to defend. They did an assessment and it didn’t deliver, that’s it.
I’m not defending it or attacking it, mearly saying that
Isnt supported by the information given. The GP gave a story they made up about how usage would be falling based on nothing at all, I gave two other alternate stories about how it could be either rising in usage or remaining flat to demonstrate that we cannot say anything about rate of change from a single average.
On my work, of I’m provided free software that makes my work easier, I’ll use it. If the users arent seeing the value, then the value is not there.
Simply not true, if this was the case, then change management wouldn’t be a thing.
While I agree that people are resistant to change, all the studies ive seen show negative or minimal benefit.
So either people are being poorly trained by the change management or the product is poor and doesn’t love up to its marketing.
Yes this happens a lot, and IT-habits are notoriously difficult to change in a work-setting.
Exactly!
If you force someone to use a given solution, they’re going to push back until they see value. And people aren’t going to put in the effort to learn something they have initial misgivings about.
LLMs can be useful tools, or they can be a terrible waste of time. It all comes down to what tasks you use it for and how motivated you are to get it to work. I use it occasionally (maybe 2-3x/week), and I find it saves me time when I do. I don’t use it for tasks I know it’ll be poor at, because that’s a waste of time.
What do you think Change Management is?
You assume the average usage is representative of the actual usage. You averaged the actions over the time period, nothing’s says the users didn’t performed the averaged 72 actions within the first three days or any time restricted window within the whole period of time and got bored with it seeing no or low value.
It’s also possible a handful of power users use it a ton and found value, while the quiet majority only used it a few times because they were required to and didn’t see value.
We need more details to draw conclusions. For example:
I’m not a programmer, so it’s got nothing to offer me. Mostly my job is to write documentation for propriety software and hardware, stuff the AI knows nothing about, not everyone in the world can mak use of AI, and it doesn’t require a PhD and 30 days of constant usage to work that out.
I’m not saying AI specifically is useful, just that people in general tend to resist change in their work methods regardless of what they are.
I also work with a lot of proprietary knowledge, chemical and infrastructure in my case, and AI still can be useful when used properly. We use a local model and have provided it with all our internal docs and specs, and limited answers to knowledge from these, so we can search thousands of documents much faster, and it links to the sources for it’s answers.
Doesn’t do my job for me, but it sure as shit makes it easier to have a proper internal search engine that can access information inside documents and not just the titles.
Then maybe it’s not useful for you. That doesn’t mean AI isn’t useful for a number of other roles.
I’m a software developer and find its code generation to be awful, but I also find that it’s great at looking up technical information. Maybe I’m looking for a library to accomplish a task, and I want to compare features. Or maybe I’m having trouble finding usage examples for a relatively niche library. Those are task the AI is great at, because it can look at tons of blog posts, stack overflow questions, etc, and generate me something reasonable that I can verify against official docs.
If my workflow was. mostly email and internal documentation, yeah, AI wouldn’t be that useful. If my workflow relies on existing documentation that’s perhaps a little hard to find or a bit poor, then AI is great. Find the right use case and it can save time.
Case in point, as per the article, AI is pretty useless for regular office work
“Regular office work” is a pretty broad category. Yeah, it’s probably not useful in retrieving records for someone or processing forms, but it should be useful for anything that requires research.
not sure there is any research done by people using office suite…
it sounds like you are conflating LLM in general with the crappy copilot that MS offers with the office suite
an LLM could be useful for research of large (large) datasets… Copilot would not be
I don’t know much about copilot, but some quick research shows it uses GPT-5 for the chat feature. I assume that’s what’s meant by the average queries in the article.
so you think they may be useful but people just like to work harder? or perhps, they tried and saw no benefit at all and moved on?
Having been part of multiple projects introducing new software tools (not AI) to departments before, people are usually just stubborn and don’t want to change their ways, even if it enables a smoother work-flow with minimal training/practice. So yeah, basically people are so set in their ways,it is often hard to convince them something new will actually make their job easier.
The devil is in the details… what you describe screams to me what I call the “new boss syndrome”. New boss comes in and they feel the need to pee on everyone to mark their territory so they MUST bring in some genius change.
99% of the time, they are bringing in some forced change for the sake of change or something that worked on their previous place without taking into consideration the context.
I do not know anyone who prefers to work harder… either the changes proposed make no sense (or it’s too complex for people to understand the benefit) or the change is superfluous. That is usually where resistance to change comes from.
In all your software deployments did you blame the users for not getting it or did you redesign the software because it sucked (according to your users)?
I’ve occasionally been part of training hourly workers on software new to them. Having really, really detailed work instructions and walking through all the steps with themthe first time has helped me win over people who were initially really opposed to the products.
My experience with salaried workers has been they are more likely to try new software on their own, but if they don’t have much flexible time they usually choose to keep doing the established less efficient routine over investing one-time learning curve and setup time to start a new more efficient routine. Myself included - I have for many years been aware of software my employer provides that would reduce the time spent on regular tasks, but I know the learning curve and setup is in the dozens of hours, and I haven’t carved out time to do that.
So to answer the question, neither. The problem may be neither the software nor the users, but something else about the work environment.
I was one of the users, these are my observations with my colleagues reactions, and sometimes also myself.
That’s not what I’m asking. You designed or built something for some users. They didn’t like it, or didn’t use it as you expected. Was your response to change the software or blame the users for not using it correctly?
That depends on the issue. Sometimes it’s a lack of training, sometimes it’s obtuse software. That’s a call the product owner needs to make.
For something like AI, it does take some practice to learn what it’s good at and what it’s not good at. So there’s always going to be some amount of training needed before user complaints should be taken at face value. That’s true for most tools, I wouldn’t expect someone to jump in to my workflow and be productive, because many of the tools I use require a fair amount of learning to use properly. That doesn’t mean the tools are bad, it just means they’re complex.
Are you the person (alt) I was asking this of?
No. Is that a problem?
We have it on our system at work. When we asked what management expected it to be used for they didn’t have an answer.
We have a shell script that ingests a list of user IDs and resets their active directory passwords, then locks the account, then sends them an email telling them to contact the support desk to unlock the account. It a cron job that runs ever Monday morning.
Why do a need an AI for when we can just use that? A script that can be easily read understood and upgraded, with no concerns about it going off-piste and doing something random and unpredictable.
So yeah, they don’t use it, because it won’t work.
Well yeah, AI shouldn’t replace existing, working solutions, it should be used in the research phase for new solutions as a companion to existing tools.
Worth noting the average includes the people who did use it a lot too.
So you can conclude people basically did not use it at all.