AI and Employment Risk: Benefits vs Bias

Since its breakout year in 2023, generative artificial intelligence (AI)  is proving to  be an unstoppable force with applications in every field, and in just four short years, AI’s market size is predicted to reach US$407 bn by 2027.¹ 

Business leaders and boardrooms are recognising the opportunities of AI, from its ability to innovate at speed, to its role in driving efficiency among workforces. As firms of all sizes look to adopt AI technology, many are finding opportunities to streamline their workforces and increase outputs. This streamlining effect is reflected in our Risk & Resilience research undertaken earlier this year, which saw 68% of the 3,500 business leaders surveyed believing that AI will lead to jobs being replaced in their company this year. 

Yet, the deployment of AI is not without risks. Our research also found that a quarter of boardrooms (25%) believe AI and other disruptive technologies are the biggest risk they face this year. 

Workforce in motion

While the benefits of AI in the workplace have been broadly welcomed, we are seeing new challenges emerge that are fuelling employment liability practices (EPL) risks for firms. 

This potential for AI to displace jobs and skills is increasingly an ‘elephant in the room’ within discussions around these new technologies. The automation of routine tasks in certain sectors is likely to lead to some jobs no longer needing a human to carry out tasks. For firms, there is a potential here to focus resources on other business areas where human skills are needed or reduce the overall headcount of the business.

Developments like this present future challenges for companies as they look to build out their talent pipeline of senior staff. Furthermore, impacted staff who might lose their job or need to retrain could question the fairness of these decisions and whether they are getting the appropriate support to reskill in the workplace or find other employment. If employees believe that they have been dismissed unfairly, they could challenge this and bring forward a claim against their employer.

Rising risks for HR teams

AI is further creating new risks for human resources (HR) professionals who are increasingly using the technology in the management and recruitment of staff. 
Bias in AI systems is a very topical issue. A recent study by Bloomberg found that candidates for roles were being filtered out by ChatGPT 3.5 because of an individual’s racial background.² The investigation concluded that ‘that using generative AI for recruiting and hiring poses a serious risk for automated discrimination at scale’. A global beauty brand recently settled a court case with a former employee who believed that they had been discriminated against by AI tools that sifted through job applications on behalf of the company.³ The firm deployed AI facial analysis technology to score each candidate and decide if they would go on to the next round.

Firms need to be particularly careful in how they handle employee data with existing rules like the Genetic Information Nondiscrimination Act (GINA) in the US requiring businesses to keep genetic information confidential.⁴ The regulatory landscape is also evolving with New York City implementing Local Law 144, which requires companies to audit and test automated employment decision-making tools.⁵ All of these rules place greater pressure on employers to ensure they are handling and processing employee data in a safe, secure, and fair manner.

Managing new risks

For businesses deploying AI to analyse and manage their human capital, there is a real opportunity to create efficiencies, especially when they operate in an organisation with a large talent pool or are advertising for a popular vacancy which generates thousands of applications. 

That said, AI can also create new risks, and a challenge for organisations is ensuring that AI-based systems do not discriminate against or show bias towards certain groups or demographics. While this is difficult to prove, increasingly, there are claims that AI technology in the workplace has unintended consequences and is showing bias against gender, race, older staff, those with disabilities, or those from a disadvantaged background.⁶

There is a clear risk for businesses that their deployment of AI could lead to accusations of unfairness in the workplace or discrimination which could lead to an EPL claim. In larger organisations where managers might be using AI to manage their team but not have appropriate training to ensure this is compliant and fair, it could further heighten the risk of an employment claim. 

As AI adoption becomes more common in the workplace, businesses need to ensure the coverage provided under their EPL policy is fit for purpose and that it provides adequate protections from AI-related risks. For employers, understanding the potential exposures associated with the technology, alongside the opportunities, is vital to ensuring compliance and reducing the risk of a claim.