AI and the Future of Skills – Healthy human adults share some basic skills that AI systems do not have

Artificial intelligence (AI) and robotics are major breakthrough technologies that are transforming the economy and society. To understand and anticipate this transformation, policy makers must first understand what these technologies can and cannot do. The OECD launched the Artificial Intelligence and the Future of Skills project to develop a programme that could assess the capabilities of AI and robotics and their impact on education and work. This report represents the first step in developing the methodological approach of the project. It reviews existing taxonomies and tests in psychology and computer science, and discusses their strengths, weaknesses and applicability for assessing machine capabilities.

Assessing AI and robotics capabilities is a necessary foundation for understanding their implications for education, work and the larger society.

An ongoing programme of assessment for AI and robotics will add a crucial component to the OECD’s set of international comparative measures that help policy makers understand human skills. The Programme for International Student Assessment (PISA) describes the link between the education system and the development of human skills, while the Programme for the International Assessment of Adult Competencies (PIAAC) links those skills to work and other key adult roles. A programme for assessing AI and robotics capabilities will relate human skills to these pivotal technologies, thereby providing a bridge from AI and robotics to their implications for education and work, and the resulting social transformations in the decades to come.

There are a number of taxonomies and tests of human skills. These provide different perspectives and opportunities for understanding AI capabilities.

Taxonomies stemming from the cognitive psychology literature are hierarchical models of broad cognitive abilities, such as fluid intelligence, general memory/learning, visual and auditory perception, assessed by factor analysis of cognitive ability tests. These tests have been widely used and validated for assessing human skills.

Research interest in social and emotional skills is growing and their testing is advancing. These skills are focused on individuals’ personality, temperament, attitudes, integrity and personal interaction. Recent research considers not just individual abilities but also collective ones. This emerging literature studies the factors of “collective intelligence” and is developing tests to measure them.

Education research has also contributed to defining and shaping the understanding of human skills. This domain focuses on subject-specific knowledge (e.g. in mathematics, biology and history), basic skills such as literacy and numeracy, and more complex transversal skills such as problem solving, collaboration, creativity, digital competence and global competence. A wide range of tests is available from international and national large-scale educational assessments.

Skills can be linked to work tasks and occupations, and measured through complex vocational tests.

Another major area – industrial-organisational psychology – links abilities to tasks specific to particular occupations. The resulting comprehensive occupation taxonomies classify occupations by work tasks, and the required skills, knowledge and competences. The most widely used classifications are the Occupational Network (O*NET) database of the US Department of Labor and the European classification of skills, competences, qualifications and occupations (ESCO). Assessments in this domain comprise a variety of vocational and occupational tests.

Healthy human adults share some basic skills that AI systems do not have.

Many taxonomies for assessing skills overlook ubiquitous low-level or basic cognitive skills. These are rarely assessed in human adults because there are few meaningful individual differences in the absence of severe disability. However, AI systems do not necessarily have these skills (e.g. navigating in a complex physical environment, understanding basic language or knowing basic rules of the world). Taxonomies and assessments for these skills are found in the fields of animal cognition, child development and neuropsychology. A recently emerging field assesses basic (low-level) skills of AI systems drawing on these fields of psychology.

Evaluating AI and robotics systems is challenging and applying human tests can be misleading.

AI assessment focuses on functional components of intelligent mechanisms, such as knowledge representation, reasoning, perception, navigation and natural language processing. These are strongly linked to the underlying technique used by the mechanism. Many components overlap with the ability categories developed in psychology for humans, but the match is not exact. In addition, many capabilities that AI is developing – such as language identification and the generation of realistic images – are not well covered by human skill taxonomies or tests.

Moreover, the design of human tests takes for granted that the test takers all share basic features of human intelligence, which might be radically different from AI. For example, integrating basic skills, such as natural language understanding and object recognition, is easy for humans. However, most AI systems are trained to perform a specific narrow task, but they are not (or are rarely) able to integrate and apply these to perform a different type of task. This makes it difficult to generalise from an AI system’s performance on a specific human ability test to an underlying AI skill, let alone infer general intelligence.

Different types of empirical assessments can gauge AI capabilities, but these are scattered and not systematic.

A multitude of benchmarks and competitions assess and compare AI systems empirically. However, these have not yet been systematically classified. Increasingly, more institutions carry out rigorous evaluation campaigns to assess the capabilities of AI and robotics systems. These include the evaluation of individual functions, i.e. self-contained units of capability, such as self-localisation. They also include evaluation of complete tasks that constitute a meaningful activity, such as autonomous driving and text summarisation. Evaluation of AI systems is particularly well developed in certain areas, such as language understanding. Machine translation, in particular, is a field that holds many lessons for assessing AI.

A systematic assessment of AI demands a comprehensive framework that covers all human skills necessary for work and life.

Providing valid, reliable and meaningful measures of AI and robotic capabilities requires a comprehensive approach that brings together different research traditions and complementary methodologies. The goal should be to address the full range of relevant human capabilities; the extra capabilities needed to consider for AI (because they are difficult for AI and often neglected in lists of human skills); and the full range of valued tasks that appear in education, work and daily life.

A robust methodology involves understanding how AI and robotics systems are assessed and bringing together different assessment approaches.

A multidisciplinary approach needs theoretical underpinning that considers the challenges linked to assessing AI and robotic capabilities with regard to human skills. The different disciplinary approaches can be organised across two dimensions. One relates to whether skill taxonomies and tests measure primarily human or primarily AI capabilities. The second dimension is whether they measure single (isolated) capabilities or complex tasks that require multiple capabilities. Future systematic assessment of AI capabilities should bring together different assessments along these two dimensions and skilfully integrate their potential to draw valid implications for the future of work and education.

---

Volume 1. Capabilities and Assessments. The volume consists of expert contributions that review skills taxonomies and tests in different domains of psychology, and efforts in computer science to assess AI and robotics. It provides extensive discussion on the strengths and weaknesses of different approaches, and outlines directions for the project.
Volume 2. Methods for Evaluating AI Capabilities. The report discusses the advantages and challenges in using these approaches and describes how they will be integrated into developing indicators of AI capabilities.

Chosen excerpts by Job Market Monitor. Read the whole story @  AI and the Future of Skills, Volume 1: Capabilities and Assessments | en | OECD

Related Posts

Skills and AI – Millions of workers may need to be not just upskilled but reskilled

In the coming decades, as the pace of technological change continues to increase, millions of workers may need to be not just upskilled but reskilled—a profoundly complex societal challenge that will sometimes require workers to both acquire new skills and change occupations entirely. Companies have a critical role to play in addressing this challenge, but … Continue reading →

Artificial Intelligence (AI) – Tasks will be augmented, not automated

Despite technological strides forward, this year’s Future of Jobs report suggests that businesses are becoming more sceptical about the potential for artificial intelligence to fully automate work tasks. Executives estimate that 34% of tasks are already automated – just one percentage point ahead of the figure reported in the Future of Jobs Report 2020. Future … Continue reading →

The Future of Work – Artificial Intelligence (AI) won’t replace most jobs but people using it are starting to replace people who don’t

As AI is increasingly applied to knowledge work, a significant shift will likely take place in the workplace, affecting many jobs in the Western middle class. Contrary to recent dire predictions about AI’s effect on employment, our survey suggests cautious optimism. Most respondents, for example, do not expect that AI will lead to a reduction … Continue reading →

New Technologies and Artificial intelligence – How they are transforming tasks : analyzing 170 million online job postings

Technology has long brought change to the nature of work, and to the skills required for the most desirable, best-paying jobs. But until recently, new technology – even robotics – has tended to mean automating repetitive or arduous tasks, while often leading to new types of tasks for workers. The emergence of artificial intelligence (AI) … Continue reading →

AI and Jobs? Workers seem more hopeful than concerned about the impact of AI

Which U.S. workers are more exposed to AI on their jobs? In 2022, 19% of U.S. workers were in jobs in which the most important activities may be replaced or assisted by artificial intelligence tools. Higher-paid workers, as well as those who are women, Asian or college-educated, are more exposed to AI than other groups. … Continue reading →

AI – Has rapidly become a part of the cultural and business zeitgeist LinkedIn finds

The Future of Work Report: AI at Work, November 2023 uncovers the latest real-time trends we see around how AI is ushering a new world of work, grounded in LinkedIn’s unique labor market insights, based on one billion professionals and 67 million companies. Explore our latest findings and insights around: Professionals’ conversations and sentiments towards … Continue reading →

GenAI and Upskilling – Look at how employee roles and skills must evolve

Generative AI presents both benefits and challenges to organizations. BCG’s GenAI Workforce Productivity Diagnostic helps companies define a clear path to successfully implement this transformative technology. Key Takeaways While the value and transformation potential of GenAI are real, so too are the technical, implementation, and change management challenges. BCG’s GenAI Workforce Productivity Diagnostic helps companies … Continue reading →

AI and Ageing Revolutions – The extent to which they alter our societies for good or for ill will very much depend on how we choose to respond

The world of work is getting older in advanced economies and increasingly in many emerging ones. Today, people between theages of 45 and 64 account for some 40 percent of the working-age population in OECD countries, up from 28 percent in 1990. Two powerful and persistent trends – longer lifespans and lower birth rates – … Continue reading →

AI in the Workplace – What do workers and employers think and what to do about it

In 2022, the OECD gathered data on the impact of AI on people and their workplaces, in the manufacturing and finance sectors of seven countries. The findings show that AI use at work can lead to positive outcomes for workers around job satisfaction, health and wages. Yet there are also risks around privacy, work intensity … Continue reading →

Employment Outlook – OECD countries may be on the brink of an AI revolution while wages have not kept up with inflation

While firms’ adoption of AI is still relatively low, rapid progress including with generative AI (e.g. ChatGPT), falling costs and the increasing availability of workers with AI skills suggest that OECD countries may be on the brink of an AI revolution. It is vital to gather new and better data on AI uptake and use … Continue reading →

Artificial Intelligence (AI) – It will challenge economists in coming years

After a number of AI-winters, AI is back with a boom. There are concerns that it will disrupt society. The immediate concern is whether labor can win a ‘race against the robots’ and the longer-term concern is whether an artificial general intelligence (super-intelligence) can be controlled. This paper describes the nature and context of these … Continue reading →

The Future of Work – Artificial Intelligence (AI) impacts on learning, teaching, and education

At the November 2017 Gothenburg Summit, the Commission presented the Communication ‘Strengthening European Identity through Education and Culture’, that set out a vision for a European Education Area and announced a dedicated Digital Education Action Plan, which aims to foster digital skills and competences for all citizens. The Action Plan focuses on implementation and the … Continue reading →

Artificial Intelligence – Creating both fear and hope among workers

What will be the effects of artificial intelligence on the workplace? Our survey respondents expect AI will have a large impact on the skills employees will need on the job. At the same time, they remain cautiously optimistic about AI’s overall effect on the workforce. Another year of AI experimentation and learning has left opinions … Continue reading →

Future of Jobs – Views about robots and artificial intelligence, Sweden vs US

Eighty percent of Swedes express positive views about robots and artificial intelligence, according to a survey this year by the European Commission. By contrast, a survey by the Pew Research Center found that 72 percent of Americans were “worried” about a future in which robots and computers substitute for humans. In the United States, where … Continue reading →