How AI can help choose your next career and stay ahead of automation

By Nik Dawson, Marian-Andrei Rizoiu, Mary-Anne Williams

The typical Australian will change careers five to seven times during their professional lifetime, by some estimates. And this is likely to increase as new technologies automate labour, production is moved abroad, and economic crises unfold.

Jobs disappearing is not a new phenomenon – have you seen an elevator operator recently? – but the pace of change is picking up, threatening to leave large numbers of workers unemployed and unemployable.

New technologies also create new jobs, but the skills they require do not always match the old jobs. Successfully moving between jobs requires making the most of your current skills and acquiring new ones, but these transitions can falter if the gap between old and new skills is too large.

We have built a system to recommend career transitions, using machine learning to analyse more than 8 million online job ads to see what moves are likely to be successful. The details are published in PLOS ONE.

Our system starts by measuring similarities between the skills required by each occupation. For example, an accountant could become a financial analyst because the required skills are similar, but a speech therapist might find it harder to become a financial analyst as the skill sets are quite different.

Next, we looked at a large set of real-world career transitions to see which way around these transitions usually go: accountants are more likely to become financial analysts than vice versa.

Finally, our system can recommend a career change that’s likely to succeed – and tell you what skills you may need to make it work.

Measure the similarity of occupations

Our system uses a measure economists call “revealed comparative advantage” (RCA) to identify how important an individual skill is to a job, using online job ads from 2018. The map below visualises the similarity of the top 500 skills. Each marker represents an individual skill, coloured according to one of 13 clusters of highly similar skills.

Once we know how similar different skills are, we can estimate how similar different professions are based on the skills required. The figure below visualises the similarity between Australian occupations in 2018.

Each marker shows an individual occupation, and the colours depict the risk each occupation faces from automation over the next two decades (blue shows low risk and red shows high risk). Visibly similar occupations are grouped closely together, with medical and highly skilled occupations facing the lowest automation risk.

Mapping transitions

We then took our measure of similarity between occupations and combined it with a range of other labour market variables, such as employment levels and education requirements, to build our job transition recommender system.

Our system uses machine learning techniques to “learn” from real job transitions in the past and predict job movements in the future. Not only does it achieve high levels of accuracy (76%), but it also accounts for asymmetries between job transitions. Performance is measured by how accurately the system predicts whether a transition occurred, when applied to historic job transitions.

The full transitions map is big and complicated, but you can see how it works below in a small version that only includes transitions between 20 occupations. In the map, the “source” occupation is shown on the horizontal axis and the “target” occupation on the vertical axis.

If you look at a given occupation at the bottom of the map, the column of squares shows the probability of moving from that occupation to the one listed at the right-hand side. The darker the square, the higher the probability of making the transition.

Artificial intelligence-powered job recommendations Sometimes a new career requires developing new skills, but which skills? Our system can help identify those. Let’s take a look at how it works for “domestic cleaners”, an occupation where employment has shrunk severely during Covid-19 in Australia.

First, we use the transitions map to see which occupations it is easiest for a domestic cleaner to transition to. The colours split occupations by their status during the Covid-19 crisis – blue occupations are “essential” jobs that can continue to operate during lockdown, and red are “non-essential”.

We identify top recommended occupations, as seen on the right side of the flow diagram (bottom half of the image), sorted in descending order by transition probability. The width of each band in the diagram shows the number of openings available for each occupation. The segment colours represent whether the demand has increased or decreased compared with the same period of 2019 (pre-Covid).

The first six transition recommendations for are all “non-essential” services, which have unsurprisingly experienced decreased demand. However, the seventh is “aged and disabled carers”, which is classified as “essential” and grew significantly in demand during the beginning of the Covid-19 period.

Since your prospects of finding work are better if you transition to an occupation in high demand, we select “aged and disabled carers” as the target occupation for this example.

What skills to develop for new occupations

Our system can also recommend skills that workers need to develop to increase their chances of a successful transition. We argue that a worker should invest in developing the skills most important to their new profession and which are most different from the skills they currently have.

For a “domestic cleaner”, the top-recommended skills needed to transition to “aged and disabled carer” are specialised patient care skills, such as “patient hygiene assistance”.

On the other hand, there’s less need to develop unimportant skills or ones that are highly similar to skills from your current occupation. Skills such as “business analysis” and “finance” are of low importance for an “aged and disabled carer”, so they should not be prioritised. Similarly, skills such as “ironing” and “laundry” are required for the new job but it is likely that a “domestic cleaner” already possesses these skills (or can easily acquire them).

The benefit of smoother job transitions

While the future of work remains unclear, change is inevitable. New technologies, economic crises and other factors will continue to shift labour demands, causing workers to move between jobs.

If labour transitions occur efficiently, there are significant productivity and equity benefits for everyone. If transitions are slow, or fail, it will have significant costs for both individuals and the state and the individual. The methods and systems we put forward here could significantly improve the achievement of these goals.

(Dawson and Rizoiu are with University of Technology Sydney and Williams is with University of New South Wales)