AI in Practice: Reliance and Risks

Artificial intelligence (AI) now features firmly in the day-to-day work of professional service providers – from drafting documents to interacting with clients. AI is now influencing, not just assisting, decisions. While the benefits are clear, its adoption introduces a distinct set of legal risks that extend beyond traditional technology concerns.

These risks stem from how the technology is relied upon in practice as much as the risks relating to the technology itself, in particular: dependence on third-party providers; limited visibility over how systems operate; and the potential for concealed errors. The combination of widespread adoption and concentration of providers introduces the potential for systemic impacts.

This article explores these issues across the following four key reliance risk areas:

• Service outage and availability;
• Lack of control;
• Silent errors; and
• Consequential liabilities.

Key risk areas

Service outage and availability

AI tools are commonly delivered via third-party providers, including OpenAI and Anthropic. When those services become unavailable (for example, due to server outages, software bugs or too many users accessing the system at once), users are largely dependent on the provider for any recourse and have limited ability to resolve the issue independently.

The practical impact can be immediate. Work may be interrupted or delayed, deadlines may be missed and contractual obligations may be left unfulfilled. This can give rise to breach of contract claims and, in certain circumstances, professional negligence allegations.

Lack of control

Another issue flowing from the above is the limited visibility users have over how these systems operate. Users cannot meaningfully audit the model, control updates or fully predict how it will behave. Performance may change over time, sometimes without clear notice, making it difficult for users to fully understand the tool’s capabilities and limitations.

This creates challenges both at the outset and retrospectively. At the point of use, it is harder to assess risk, set appropriate safeguards or determine whether the tool is suitable for a particular task. If something goes wrong, the same lack of visibility makes it more difficult to investigate what happened and why.

In turn, this complicates legal analysis. Questions of causation and foreseeability become less straightforward where the system’s behaviour is not fully understood or stable over time. This may lead to disputes over whether the outcome was something the user ought reasonably to have anticipated or whether it arose from factors outside their control.

How different is this from other forms of technology? At first glance, not much. But unlike traditional software, AI systems do not operate on fixed rules. Their outputs are probabilistic. They can vary over time and are not always predictable or reproducible. As a result, they are harder to test, validate and rely on consistently.

Recent reports have also highlighted a further aspect of this issue. During a controlled security test (conducted within what’s referred to as a “sandbox” environment), OpenAI disclosed that an autonomous AI agent was able to circumvent containment measures, access the internet and compromise systems belonging to third-party AI platform, Hugging Face, while pursuing its assigned objective. OpenAI described the event as an “unprecedented” cyber incident.

Silent errors

AI does not always fail in an obvious way. It can produce answers that look convincing but are wrong.

For instance, fictitious authorities, incorrect analysis and different answers to the same question. These are commonly known as AI hallucinations. The biggest challenge is that these errors are not always picked up straight away or remain unidentified.

This raises several issues. In practice, it may be difficult to disentangle the role played by the AI from the actions of the user, particularly regarding outputs which are adopted or relied upon without clear oversight. This can create complexity around causation and responsibility. This is often referred to as the ‘black box problem’.

The challenge is not limited to inaccurate outputs. OpenAI’s sandbox testing environment incident illustrates the broader difficulty of understanding and explaining AI behaviour. Even where systems are subject to extensive testing and controls, the reasoning behind AI’s actions or outcomes may not always be readily apparent.

It also has implications for the standard of care. There is likely to be increased scrutiny of how far users are expected to check AI outputs before relying on them. What constitutes reasonable use is still developing but blind reliance is unlikely to be defensible. The extent of verification required will vary depending on the nature of the task, the user’s level of expertise and the potential consequences of any error. Over time, clearer expectations around supervision and verification are likely to emerge.

This is considered further in our analysis of the importance and implications of AI output review which can be viewed here – Mind the Gap: AI Efficiency, Human Oversight and the Risks in Between | Beale & Co.

Consequential liabilities

Where AI is used by providers of professional services, it has the potential to cause direct and immediate loss.

This may arise through, for example, biased or discriminatory outputs, inaccurate financial/legal information, delays due to outage issues or responses that fail to meet regulatory or professional standards. Where such incidents occur, they can lead to customer detriment and, ultimately, claims.

In most cases, responsibility will sit with the business deploying the AI rather than the provider. This is reinforced by the contractual position: providers’ terms typically include broad exclusions and limitations of liability, meaning recourse against them may be limited in practice.

Therefore, businesses are likely to remain accountable for the outputs on which they rely, despite having limited visibility or control over how those outputs are generated. The result is a potential liability gap, in which the provider limits its exposure, primary responsibility falls on the user and insurers bear the loss.

Conclusion

AI presents clear opportunities, but it also introduces a distinct and evolving risk profile. Most of these risks are not new in isolation; businesses have always faced risks of system failure and human error. What is new is the way these risks combine when AI is used and the repercussions of doing so. Exposures that might previously have been contained and manageable may now be harder to predict and control.

From an insurance perspective, this raises more fundamental questions around coverage scope, aggregation and adequacy of existing policy wordings which we investigate within a subsequent article in this series.

With appropriate safeguards, such as human oversight, verification processes, staff training and controls on use, many of these risks can be mitigated. The challenge for professional service providers using AI technology is to understand where responsibility sits and how these exposures may develop as use becomes more widespread. As reliance on AI grows, risks may become more complex and systemic.