The University of Melbourne

Name (Individual/Organisation)

The University of Melbourne

Responses

Q1. How could the purpose in the ARC Act be revised to reflect the current and future role of the ARC?

For example, should the ARC Act be amended to specify in legislation:
(a) the scope of research funding supported by the ARC
(b) the balance of Discovery and Linkage research programs
(c) the role of the ARC in actively shaping the research landscape in Australia
(d) any other functions?

If so, what scope, functions and role?

If not, please suggest alternative ways to clarify and define these functions.

Executive Summary

The University of Melbourne (‘the University’) welcomes the opportunity to make a submission to the independent review of the Australian Research Council Act 2001 (the ARC Review) and how it establishes the role and purpose of the Australian Research Council (ARC) within the Australian research system. After two decades of ARC operation this is a timely review of its legislative foundations and is a significant opportunity to strengthen and streamline ARC organisational responsibilities and processes.
Research is core to the progress of society, and Australia has built a reputation over many years for research excellence. From basic, curiosity-driven scholarship to the innovation of new research applications, the advancement of knowledge brings untold benefits to Australian and global communities. Basic and applied research are interdependent and both are crucial to Australia’s research and development ecosystem, which underscores much of our national productivity and wellbeing. Australian Government funding plays an essential role in driving Australian research excellence, including through the operations of the ARC as the independent statutory agency that provides expert advice and administration of the National Competitive Grants Program (NCGP).
The University recognises the ARC’s concerted focus in recent times to meet expectations through consulting, reviewing and updating its processes and procedures, including through changes to the National Interest Test and Linkage Projects Scheme 2023 and the recent appointment of a Chief Research Officer. This submission from the University of Melbourne includes extensive feedback on existing processes (under the ARC legislation, or otherwise) and provides recommendations on additional ways to enhance and uplift the cyclical processes of grant application and evaluation.
In this submission the University proposes that the ARC should be mandated, through legislative changes where appropriate, to take up a clearer leadership role, along with other publicly funded research agencies, in the overall research landscape through adoption of a stronger governance structure, evaluative capabilities, and stated strategic objectives. While the ARC will continue to be closely informed by the stated priorities of government, it also must be equipped and resourced to undertake its independent, expert, and influential role as the leading national funding agency for non-medical research.
The ARC should be empowered in law and policy, and resourced appropriately, to be a major influence proactively shaping the national research landscape in pursuit of a long-term vision, in accordance with a commitment to foundational research principles of trust, integrity, excellence and accountability. To achieve this, the ARC will require: a strong and transparent evidence base about how its programs are managed, delivered and evaluated; a strengthened social licence won through more explicit engagement with civil society stakeholders on the core value of research; and a collaborative relationship with the university research sector, industry, and government.
This review is confined to operational issues and the legislative underpinnings of the ARC. However, there are broader issues that are significant and interlinked with the ARC’s management of competitive grants programs, including the true cost of research, the quantum of university funding and the development of new knowledge and skills capability through higher education in Australia. To this point, the University will closely engage with the University Accord process over the next year.
The University notes that the adequacy of research funding provided by the Australian Government is not within the remit of this review. However, further consideration of the broader context of research funding is relevant and advisable given the ARC has a key role to play in monitoring and enabling the success of Australian research and providing advocacy and leadership to ensure its sustainability and global competitiveness.
For more information or to discuss this submission, please contact [Redacted]

Recommendations

The University of Melbourne offers the following recommendations on both legislative amendments and non-legislative process changes to better support Australian research excellence and strengthen the ARC’s standing as an investable proposition.
1. The ARC Act should explicitly establish the ARC’s broader strategic remit, independence, and key role in shaping world-leading research through grant program management, engagement, advice to government, and sector leadership.
2. The ARC Act should be clarified to redefine the ARC’s funding scope as being for non-medical research that is conducted by universities and their partners.
3. The ARC Act should require an appropriate balance of funding for basic and applied research that is weighted towards basic research and distributed equitably across disciplines, with explicit in-principle support for the essential role of basic research within the research ecosystem. This is an essential function of the ARC and will help secure a pipeline of translatable discoveries and new scholarship.
4. The ARC Act should specify the ARC’s core obligations with regards to nurturing researcher development through fellowship and other programs to help maintain broad capability across different disciplines and strong evaluative capability to ensure the equity and effectiveness of its funding schemes.
5. The ARC Act should establish a renewed Board of the ARC, with contemporary modifications and legislated clarity on its membership requirements, functions, authority, and transparency measures.
6. The University recommends the establishment of a light-touch umbrella governance forum of Australia’s major publicly funded research funding agencies to bring greater coherence and coordination to the sector and contribute to an integrated national research strategy.
7. The ARC Act should state a requirement that the CEO and Executive Directors have term limits of approximately five years and be of esteemed academic/research background.
8. The ARC Act should enshrine a form of the Haldane Principle and affirm that the evaluation of the research merit of individual applications will be conducted through peer review (e.g., through the ARC College of Experts).
9. In the case that the Ministerial veto power is retained in the ARC Act, it should be clearly defined and limited to extraordinary cases, and accompanied by a statutory timeframe for informing the public about the decision through tabling of reasons in Parliament by the Minister.
10. In place of the National Interest Test, the University recommends refining Applicant Summary sections of the application to require a plain language explanation of the broader value of the research (which explicitly can include scholarly, social, economic, and national benefits).
11. The ARC, potentially under the guidance of its new Board, should undertake high-level engagement and enhance understanding amongst key policymakers about the value and benefits of the research it funds, as part of its sectoral leadership and pivotal role in strengthening the social licence for research, across Discovery and Linkage schemes.
12. While ERA has been influential in uplifting quality, it should be succeeded by a data asset (such as a National Research Capability Map) that uses latest techniques to consolidate university and non-university data, which can underpin regular structured evaluation between Australian Government and universities on outcomes and return on research investment.

Q1 Response

Purpose and strategic remit of the ARC
As noted in the Consultation Paper, the Objects (s3) of the Australian Research Council Act 2001 (ARC Act) confer a narrow set of practical responsibilities to the ARC, aimed at the provision of advice to the Minister and administration of research funding. By contrast, the National Health and Medical Research Council Act 1992 (NHMRC Act) at s3 establishes a national body to ‘pursue activities’ in pursuit of broadly aspirational and strategic purposes, such as fostering medical research and training, and uplifting public health in Australia.
Like the NHMRC, the ARC should be mandated in its enabling legislation to pursue broader strategic purposes that support, shape and drive Australian research, support international collaboration, and complement the development of research workforce capability. The amended purpose in the ARC Act should include, but go beyond, advising government on research matters and distributing research funding, and should bring it, at a minimum, into equivalence with the drafting of the NHMRC Act.
The University supports amending the ARC Act to reflect the ARC’s contemporary activities; its unique leadership positioning in the research sector; and its strategic functions. The ARC’s role should be defined broadly enough to avoid undue prescription and emphasise the ARC’s modern role as outlined in the most recent ARC Strategic Plan, being to ‘help shape the Australian research system for the benefit of the nation by enabling world-leading research, fostering research quality, translation and impact, and safeguarding research integrity’.
The University further supports amendments to the ARC Act to ensure that its purpose statement captures the ARC’s role in shaping the research landscape as outlined in the Consultation Paper. The ARC’s refreshed purposes should also include a requirement for its Executive and Board (if this is established) to develop strategic programs and foci though engagement and dialogue with key stakeholders, including the universities and the research sector, professional associations and learned academies.
The University supports changes that give the ARC flexibility and capacity to be responsive to global research dynamics, while explicitly strengthening and enshrining its independence (this latter aspect is discussed in more detail in response to Question 4).
Recommendation 1: The ARC Act should explicitly establish the ARC’s broader strategic remit, independence, and key role in shaping world-leading research through grant program management, engagement, advice to government, and sector leadership.
Scope of research funded by the ARC
Any clarifications of the ARC’s research funding scope should aim to strengthen the ARC’s critical role as the only national funding agency for non-medical research. This is crucial in ensuring adequate funding support across the breadth of Australian research and the diverse national benefits this brings. This is relevant not only to humanities and social sciences (HASS) disciplines but to all the non-medical fields of research across the science, technology, engineering and mathematics (STEM) disciplines.
Additionally, the definition and scope of research funded by the ARC should be clarified to explicitly identify and prioritise universities and their research partners in order to ensure national support for a diverse and broad range of disciplines. The ARC is a key public funder for university research and is fundamental to driving basic research and innovation across multiple disciplines. While there is in-principle value in allowing research institutes to apply for research funding of fundamental, non-medical research through their affiliation with universities, this could not be supported without increasing the ARC funding envelope to meet the new demand and avoid further reduction in funding of research in non-medical disciplines. Clarifying this research funding prioritisation and nuance in broad terms within the scope of the ARC Act would give a valuable signal and direction to ARC administrators and policy-makers in the future.
Consideration should also be given to interaction with the functions and funding remit of other national research funding agencies, including the NHMRC and MRFF. The exclusion of health and medical research under the ARC Act should remain, with clearer delineations in practice between funding bodies to minimise any unintended impacts on eligibility for pioneering and potentially cross-over fields, such as biomedical engineering, digital health, or health economics.
Recommendation 2: The ARC Act should be clarified to redefine the ARC’s funding scope as being for non-medical research that is conducted by universities and their partners.
Balance of Discovery and Linkage research programs
To ensure consistency in terminology, in this submission we use the descriptors ‘basic’ and ‘applied’ as adopted in the ARC Review Consultation Paper (p5) and for simplicity we use basic to refer to both pure basic and strategic basic. In doing so, we acknowledge that the distinctions are not always clear cut, that modes of research inquiry can overlap, and that there are numerous synonyms that capture these distinctions.
The ARC should be tasked in its enabling legislation with providing a steady safety-net for basic research and strengthening a culture of respect for basic research, as this is in the long-term interest of the nation. Investment in basic research provides the pre-condition for any future applications or research translation and builds the foundational workforce capability without which further research innovation, application and translation would not be possible. As the political and policy focus has increasingly emphasised research translation and commercialisation, often with an accent on science, technology, engineering, mathematics and medicine (STEMM) research, there is a clear need for the ARC to have a leadership role as an impartial public guarantor of basic research funding, ensuring the maintenance of a core level of competitively selected, peer reviewed, high-quality basic research across all non-medical disciplines.
It is critical that the ARC implements an appropriately balanced distribution of funding for basic and applied research (currently loosely mapped onto the Discovery/Linkage programs). Approximately 60 percent of the ARC’s research funding is currently distributed through the Discovery Program, with some basic research also partially supported in projects funded through the Linkage program (e.g. Centres of Excellence). While the recent increase in Linkage expenditure (and overall ARC expenditure) through Industry Fellowships was welcome, the drive towards applied research was accompanied by a gradual decrease in funding for fundamental research in Australia since 1992 (Australian Academy of Science, drawing on ABS data).
The University recommends that the ARC Act should emphasise the need for an appropriate balance of funding for basic and applied research that is weighted towards basic research, reflecting in-principle support for the essential role basic research plays across the whole research ecosystem, as outlined above. This would give security and enduring protection to funding for basic research in Australia. It would not prohibit future Australia Governments from creating, and resourcing, new mission-driven or Linkage-style schemes in addition to the funding of basic research. The University does not believe that a specific percentage should be enshrined in the ARC Act; rather, a process to determine such a split could be a role given to a new ARC Board (see response to Question 2 on ARC governance).
In addition to enshrining the need for an appropriate balance of basic and applied funding, the ARC Act should require that this funding is allocated across all non-medical disciplines in a way that maintains broad research capability. This would have the additional and crucial effect of protecting the long-term viability and quality of publicly funded research in HASS fields, as well as technical and scientific fields including engineering, physics, maths, and computing, for whom the ARC is the only public funding body. An ARC Board could conduct a review of the criteria for allocation of funding to discipline panels as a mechanism to ensure appropriate distribution that promotes the sustainability of all eligible disciplines. The design of particular schemes within the Discovery and Linkage programs also needs to be attuned to supporting the quality and sustainability of Australia’s non-medical research workforce and research capability across all disciplines funded by the ARC (see response at Question 10).
The University supports the role of the Linkage Program in encouraging cooperative approaches to research and promoting research partnerships but recommends that this role be carefully scoped in the context of other Government schemes with similar aims. This includes ensuring, potentially through an ARC Board, that scheme design, priorities and application processes support collaboration with a broad range of external partners such as, non-government, public sector, industry, cultural heritage sector, community and not for profit groups. This breadth would also promote participation of researchers across a wide range of disciplines.
Recommendation 3: The ARC Act should require an appropriate balance of funding for basic and applied research that is weighted towards basic research and distributed equitably across disciplines, with explicit in-principle support for the essential role of basic research within the research ecosystem. This is an essential function of the ARC and will help secure a pipeline of translatable discoveries and new scholarship.
ARC leadership and shaping the research sector
The ARC can be a key point of influence in the Australian research sector by demonstrating best practice in the administration and design of equitable research funding and strong evaluative capability. It can also influence sectoral change through the setting of standards, norms, and expectations. As discussed in the next section, a well-designed and influential new ARC Board or Council will assist with building the social licence and sectoral momentum for programmatic reforms and cultural change where this is demonstrably necessary.
The University agrees that a core function of the ARC is to refine its own programs and practices to address any imbalances in outcomes across disciplines, and to recognise the effect of disruption on research careers, particularly where these result in gender or other inequities. Importantly, equity reforms and program improvements should be based on a strong and transparent evidence base and involve collaboration and consultation with the research sector and be made publicly available. Similarly, a diversity and inclusion analysis and report on the assessor pool and the College of Experts should be published at regular intervals.
A strong model for this collaborative approach is the process led by the NHMRC over 2022, leading to the NHMRC’s announcement of significant changes to the structure of its Investigator Grants scheme to ensure greater gender equity. To achieve this, the NHMRC initially released key data and rationale; modelled and released numerous options for equity interventions for public consideration; and conducted a national consultation with the research sector to test and build support for gender equity interventions. The University supported the rigour, transparency and collaborative spirit of this approach and stands ready to work with the ARC on similar reforms of its programs and outcomes, which could have the additional benefit of influencing swifter equity uplifts within institutions. (See also Question 9 on the evaluation capacity of ARC).
Recommendation 4: The ARC Act should specify the ARC’s core obligations with regards to nurturing researcher development through fellowship and other programs to help maintain broad capability across different disciplines and strong evaluative capability to ensure the equity and effectiveness of its funding schemes.

Q2. Do you consider the current ARC governance model is adequate for the ARC to perform its functions?

If not, how could governance of the ARC be improved? For example, should the ARC Act be amended to incorporate a new governance model that establishes a Board on the model outlined in the consultation paper, or another model.

Please expand on your reasoning and/or provide alternative suggestions to enhance the governance, if you consider this to be important.

To improve organisational governance and strategic capability, the University concurs with the proposition that the ARC Act should establish an influential ARC Board, with contemporary modifications and legislated specificity about the mix of membership and functions.
Broadly, the renewed ARC Board should be designed with close reference and alignment to the NHMRC Council and its principal committees – notwithstanding the broader role of the NHRMC beyond the funding of research. The NHMRC Council has wide representation of respected researchers, and experts from a range of perspectives and from all states and territories, which affords greater protection and assurance to the CEO of the NHMRC. A renewed design of an ARC Board will create greater alignment and coordination between Australia’s two major public research funding bodies and provide both Boards with equivalent seniority and distance from political pressure.
The renewed ARC Board should have representatives and prominent connections to the Australian researcher base, with membership composition drawing on diverse groups of scholars, assured humanities and social sciences representation, industry experts and other research end-users. This will lift perceptions of the ARC’s independence and accountability and reduce any sense in the research sector or broader community that the ARC is remote or disconnected. Establishing robust governance that includes a strong researcher voice will also assist in rebuilding trust in the ARC among the academy.
Further, the University recommends:
• The draft functions of the renewed ARC Board as outlined in the Consultation Paper at p7 are appropriate as a starting point. Additionally, the ARC Board could draw upon the Terms of Reference of the Board of the United Kingdom Research and Innovation (UKRI), which require that the UKRI Board is also responsible for ‘engagement with relevant research stakeholders’.
• Terms of appointment, responsibilities, and lines of reporting should be clearly set out in the ARC Act.
• For accountability and transparency (and excluding necessary redactions for national security or commercial in confidence matters) the Minutes of the renewed ARC Board (and Board Committee) should be made publicly available. The Minutes of Meetings of both the NHMRC Council, and the UKRI Board, are publicly available.
On the Board membership, the University recommends the requirement of an expanded membership list beyond that proposed in the Consultation Paper, to ensure crucial links to academic societies and scientific/policy expertise. Members appointed by the Minister should have representation across the research spectrum as far as possible, and include:
• Representation from the Australian Council of Learned Academies;
• Australia’s Chief Scientist;
• At least two distinguished researchers in HASS disciplines, and one researcher who is at an earlier career stage (amongst the ‘4-6 members with a combination of skills’ listed at bullet point 2 in the Consultation Paper);
• One person with expertise in research by or relating to Aboriginal persons and Torres Strait Islanders or the advancement of Indigenous knowledge (additional to those already listed at bullet point 2);
• Two distinguished ‘industry’ leaders, meaning business, cultural sector or not-for-profit (separate to those already listed at bullet point 2);
• One person from the Australian public sector with relevant expertise, such as in international collaboration.
The University further recommends the establishment of a light-touch umbrella governance forum of the publicly funded research agencies to contribute to the development of an integrated national research strategy and bring greater coherence and coordination to the sector in the long-term service of the nation. This approach was adopted in the UK following the 2015 review by Sir Paul Nurse review leading to the creation of the UKRI as the national funding agency investing in science and research; and bringing together the seven Research Councils, Innovate UK, and Research England.
While the University is not suggesting the creation of a UKRI-equivalent in Australia, there would be enormous potential benefits of better coordination across the sector. A light-touch governance forum of funding bodies would also be well-placed to work with peak bodies, universities, research professional associations, government, departments and research institutes to contribute to the development of a more integrated national research strategy with removal of duplication and joint effort in key areas.
Recommendation 5: The ARC Act should establish a renewed Board of the ARC, with contemporary modifications and legislated clarity on its membership requirements, functions, authority, and transparency measures.
Recommendation 6: The University recommends the establishment of a light-touch umbrella governance forum of Australia’s major publicly funded research funding agencies to bring greater coherence and coordination to the sector and contribute to an integrated national research strategy.

Q3. How could the Act be improved to ensure academic and research expertise is obtained and maintained to support the ARC?

How could this be done without the Act becoming overly prescriptive?

Greater visibility of academic expertise will rebuild trust in ARC processes and instil greater confidence in the agency’s research leadership amongst the sector. The need for research expertise within the ARC should be required by the ARC Act so that it is prioritised in staff recruitment, particularly at the executive and senior management levels. The University recommends that the CEO of the ARC and the Executive Directors should all be required by the ARC Act to have a senior academic background and an esteemed profile as a researcher. The University notes the recent appointment of a Chief Research Officer to the ARC, which signals a welcome direction.
To maintain currency of expertise, the ARC Act should impose term limits of five years on key roles. This would ensure that the ARC CEO and Executive Directors continue to reflect contemporary academic practices and remain connected to academic cultures. As far as possible, the ARC should also aim to recruit a variety of disciplinary expertise into its senior management, such that each Panel (Humanities and Creative Arts, Social Behavioural and Economic Science, etc) is supported by an Executive Director with aligned academic expertise.
Recommendation 7: The ARC Act should state a requirement that the CEO and Executive Directors have term limits of approximately five years and be of esteemed academic/research background.

Q4. Should the ARC Act be amended to consolidate the pre-eminence or importance of peer review?

Please provide any specific suggestions you may have for amendment of the Act, and/or for non-legislative measures.

Prioritise peer review in the ARC Act
The quality, integrity, and good management of Australian research is intrinsically tied to the established system of peer review of research funding, a process whereby academic experts independently evaluate the quality, significance, and expected benefits of proposed research projects. This is articulated in the globally recognised Haldane Principle dating from 1918, which is itself enshrined at s102 of the UK’s equivalent legislation, the Higher Education and Research Act 2017. As defined in that Act, the Haldane Principle is that ‘decisions on individual research proposals are best taken following an evaluation of the quality and likely impact of the proposals (such as a peer review process)’. Peer review ensures that decisions about the merit of research proposals are led by specialists and subject to rigorous scrutiny and consideration from multiple perspectives.
According to a 2022 article, former Australian Ministers of Education have vetoed 32 ARC recommended grants since 2005, (in 2006, 2011, 2021) while the Minister for Health has never vetoed a recommended NHMRC grant. The veto decisions affecting ARC grants, overwhelmingly in the HASS disciplines, were marked deviations from the long-established system of peer review and academic freedom and put Australia out of step with the research funding approach of comparable countries, such as the United Kingdom and the United States. Ministerial vetoes of recommended grants have a lasting corrosive effect on people's preparedness to apply for ARC grants, particularly in the disproportionately affected HASS disciplines, where confidence in the integrity of ARC processes has been thoroughly undermined.
The Ministerial veto at s51–53 of the ARC Act is not necessary as a quality assurance mechanism for research excellence or value for taxpayer money, due to parallel processes and assurances that exist elsewhere in the grants assessment process. Research proposals recommended by the ARC for funding have typically already received up to six or more independent and separate evaluations. Individual assessor reports for each grant are moderated by the College of Experts and the merits and final ranking of projects are debated and determined in meetings of the relevant ARC Selection Advisory Committee. Crucially, the proposed ARC Board (equivalent to the NHMRC Council) will be able to provide further confidence and assurance to the Minister around compliance with any statements on social and national benefits or equivalent.
In line with good governance and the Haldane Principle, the ARC Act should be clear that research funding recommendations and decisions on the ranking of grants will remain with the ARC College of Experts, who are senior researchers in relevant disciplines equipped to make informed judgements about the significance, quality, and benefits of research proposals. The University unequivocally recommends that the Haldane Principle should be enshrined in the ARC Act and the Minister’s power to veto grants removed or constrained to limited circumstances (see discussion in next section).
Recommendation 8: The ARC Act should enshrine a form of the Haldane Principle and affirm that the evaluation of the research merit of individual applications will be conducted through peer review (e.g., through the ARC College of Experts).
Currently the exercise of the veto power is opaque. There is no process for appeal or explanation; and no legislative or parliamentary trigger for the disclosure of when the Ministerial veto has been exercised. Yet, it is both reasonable and practical for the research community and the Australian public to be advised when, and why, the veto has been exercised. In the scenario of the 2018 Ministerial vetoes, applications recommended by the College of Experts but rejected by the Minister were subsequently placed high on a list of unsuccessful grants. This led applicants to believe, erroneously, that their application had been unsuccessful based on rankings from the peer review process.
The University recommends that, in the outcome that the Ministerial veto is retained in the ARC Act, it should be amended to ensure it can only be utilised in limited and exceptional circumstances, and new provisions inserted that require parliamentary notification by the Minister within a stated time frame after the exercise of a veto.
Exceptional circumstances should be defined categorically and set at a high threshold, for example where there is risk posed by the research that cannot be mitigated through bespoke arrangements as appropriate to the recommended research proposal. The ARC Act should include an explicit statement that the National Interest Test alone does not satisfy extraordinary circumstances.
Recommendation 9: In the case that the Ministerial veto power is retained in the ARC Act, it should be clearly defined and limited to extraordinary cases, and accompanied by a statutory timeframe for informing the public about the decision through tabling of reasons in Parliament by the Minister.
Non-legislative measures to strengthen peer review
There are several non-legislative measures that would strength peer review through the ARC but do not require amendment to the ARC Act.
• The peer review process itself would benefit from a close review by the renewed ARC Board to ensure the conduct of peer review is, as far as possible, conducted by experts in the field or sub-discipline who are accustomed to making rigorous and fair academic judgements. This could include provision of training for all new detailed assessors, adapted from what is currently provided to College of Experts members. This is particularly important in relation to addressing conflicts of interest, unconscious bias, and calibration of ranking of applications.
• It is important that the ARC maintains its panel system, i.e. the College of Experts and its systematic processes of moderation, to guard against the views of single reviewers determining grant outcomes. As part of standard evaluation processes, review of its operation could be undertaken, overseen by the ARC Board which would be well-placed to consider and lead reforms in this regard.
Further and more detailed suggestions on grants application and assessment processes are covered in the University’s responses to Question 6.

Q5. Please provide suggestions on how the ARC, researchers and universities can better preserve and strengthen the social licence for public funding of research?

Enhance public value explanations in grant applications in place of the NIT
The Australian Government and the public legitimately expect that the considerable public investment in research will deliver the gains expected of it. To communicate the public value of proposed research, particularly for the Minister’s information at the final point of grant approval, a practice has emerged in recent years to channel an explanation of public value into an additional application requirement known as the National Interest Test statement (NIT). Concurrently, the NIT became one of the four requirements that a successful grant needed to satisfy, effectively placing it on par with recommendations from the College of Experts. The Expert Panel will be aware of the sector-wide concerns about requests from the ARC CEO for revisions to individual NIT statements.
The University notes the ARC CEO’s announcement in early December (mid this review process) that the NIT statement process will be streamlined and enhanced, including 1) certification through DVCRs, and 2) use by assessors when they are considering the benefit and impact of the proposed research in the ARC peer assessment process. While these changes are a step in the right direction, they do not go far enough.
There is still insufficient detail on what it means for the NIT to be considered as part of the peer review assessment process, particularly given its modest length, competing functions and that the anticipated benefits of the research are also expected to be demonstrated in the main body of the application. Of further concern is the conflation of purpose in both the original and revised NIT statement process; namely, its function in demonstrating national benefit/national interest and its function in communicating in plain language what the research is about and what it aims to achieve. The latter is critical to enhancing the social licence of research.
The University’s view is that the case for the scholarly, social, and national benefits of the proposed research should be demonstrated in the main text of the application, removing the need for the NIT as currently framed to be part of the formal assessment process. As noted in the Consultation Paper, the value and potential benefit of the research to the community were already within the selection criteria prior to and after the introduction of the NIT in 2018.
In place of the NIT, the University recommends refining Applicant Summary sections of the application form to require a plain language explanation of the broader research value. The relevant provision should make it clear that the plain language description can include scholarly, social and national benefits. It should be an accessible explanation of the community value of the project and be findable, when needed, to assist assessors, policy makers and non-academic readers to understand the aims and benefits of the funded research.
Taken as a whole, the grant application provides a more substantial and detailed explanation of the aims and intended impact of the research, including its feasibility and likely benefits. Proper and informed assessment of the full application against the selection criteria is necessary to make a considered and defensible judgement as to the value and significance of the research and whether a particular project is in the national interest. Were it to remain a separate requirement, the NIT should not displace or override the formal selection criteria for grant programs and should not be relied on as a determinant of research funding. As recommended in the response to Question 4, Ministerial veto based solely on the NIT should be explicitly excluded by the ARC Act.
Recommendation 10: In place of the National Interest Test, the University recommends refining Applicant Summary sections of the application to require a plain language explanation of the broader value of the research (which explicitly can include scholarly, social, economic, and national benefits).
Build social licence for investment in research
The University concurs that much more should be done to build social licence for public investment in research. However, building social licence with the Australian community is distinct from building political licence, which seems more the focus of the NIT statements to date. Social licence will not be gained through perfecting the NIT process. Rather, a stronger social licence from the Australian community will require a substantial new accent on the ARC to assure, advocate and engage as a key funder of basic research. Social licence, while assisted by a case-by-case demonstration of value in individual applications, will require sustained socialising of the value of collective research and the wider acceptance and understanding – politically and in the community – about the critical role of basic and curiosity-driven research.
The University sees great utility in the ARC being authorised to undertake more expert engagement and advocacy/outreach with government and the public about the aggregate value of the research it funds. The ARC would need consistent and principled support, and appropriate resourcing, in this endeavour from the Australian Government, as the funder of research, and collaborative support from university researchers, peak associations and learned academies. Likewise, universities are seeking to increasingly remove barriers to knowledge transfer and build wider public understanding of and connection to their research, innovation, teaching and learning activities.
Recommendation 11: The ARC, potentially under the guidance of its new Board, should undertake high-level engagement and enhance understanding amongst key policymakers about the value and benefits of the research it funds, as part of its sectoral leadership and pivotal role in strengthening the social licence for research, across Discovery and Linkage schemes.

Q6. What elements of ARC processes or practices create administrative burdens and/or duplication of effort for researchers, research offices and research partners?

The University recognises the efforts made by the ARC over the years to scrutinise and improve procedural issues in the administration of its schemes. More recently, we acknowledge the extensive changes announced by the ARC to the Linkage Projects Scheme 2023 aimed at improving ‘pain points’ for both applicants and assessors.
The Consultation Paper includes a wide range of actual or perceived administrative challenges on which the Panel has already received feedback. Below, the University responds to the listed challenges and provides feedback on additional issues.
Elaboration on challenges to process and practice challenges listed in the Consultation Paper
The University concurs with the challenge areas listed in the Consultation Paper and offers the following elaboration under those points:
• Delays to, and uncertainty regarding, announcements;
Given the scale of operations at the University of Melbourne (across multiple funding schemes), certainty of timing and processes is essential for planning. As a priority, we also recommend streamlining processes and systems across major funders (ARC, NHMRC and the Medical Research Future Fund) (MRFF) to eliminate duplication and co-ordinating the timing of submission rounds and outcomes these of three funders.
• Impediments to international research collaboration;
The University recommends that the role of the ARC in supporting international collaboration be explicitly recognised in the Council’s purpose statement. The University has previously given feedback to the ARC on the differences between US funding sources (largely open to Australian researchers); grants from UK Research Councils (up to 30% of project budgets are open to Australian researchers in some areas); and Australian research funds (largely do not accommodate active and funded international participants). The ARC should explore approaches to funding allocation that actively support and encourage international collaboration, including joint funding opportunities. Direct arrangements with international funding agencies are important to Australian research and would be best managed with coordinated single point peer review.
• Prescriptive financial requirements, variations, and approvals;
Overly prescriptive financial requirements are a major point of dissatisfaction for our researchers. These include problems with pre-award budget development and post-award management. These issues could be greatly reduced through adoption of the approach employed by the NHMRC: one-line budgets that can be spent in ways that vary from the original budget, as long as all expenditure is still for the funded project, leads to project delivery and complies with rules related to allowable expenditure.
Slow approval of variations, restrictive contracting requirement and onerous reporting requirements are further pain points. Specific examples include: several months delay in approval of project extensions; requests for Partner Organisation Contribution Reports for Fellowships, where the Partner Organisations are not listed in applications nor are a requirement of the scheme; and requiring all Multi-Institutional Agreements (MIAs) to be fully executed before any work can commence on a grant. In contrast, while the NHMRC requires that MIAs must be in place before any partner or non-lead organisations can commence work, a lead organisation can commence work before execution of any MIA.
• Duplication of national security requirements and processes outside the University Foreign Interference Taskforce (UFIT);
Enhanced whole-of-government coordination of national security requirements would reduce duplication and inefficiencies where they are occurring. The University recommends a coordinated, streamlined, and simplified approach to national security across the Australian Government.
Additional feedback on process and practice challenges and opportunities in ARC processes
• Streamlining the time intensive grant process through development of a two-stage process
Proposals for ARC funding have become extremely long and require an enormous time commitment from researchers for a very low likelihood of success. One option for reducing the time commitment required is implementing a two-stage process involving an Expression of Interest (EOI) round assessed by expert panel, prior to preparation of full applications to go to College of Expert assessors. This would also be more efficient from the perspective of reviewers, by improving the quality and consistency of review processes and resulting in fewer applications to scrutinise in greater detail. A two-stage process would need to be carefully designed to avoid compounding system delays and adding further administrative burden at applicant and assessor end.
Options for a two-stage process could be considered by the new ARC Board and carefully scoped to ensure there is no inadvertent blow out of review times and processes; i.e. to avoid unintended administrative burden and imposts on reviewers, or undue lengthening of the timeframe facing academics from submission of EOI to notification of outcome of full application.
The use of an EOI stage has proven to be valuable and effective process in the development of ARC Centre of Excellence bids. Looking globally, the National Institutes of Health (NIH) and the National Science Foundation in the US have processes in place in which there are outside reviewers and a panel for each division in the national foundation. Reviews are passed on to applicants, who can also receive individual feedback. This is essential to help researchers hone ideas to fit funding priorities and to triage ideas that are unlikely to be funded. Also, the NIH offers applicants the option to resubmit rejected proposals.
• Simplifying proposals and curriculum vitaes (CVs)
ARC proposals have become too long and complex for 4–7 reviewers to read and compare with others in an efficient and comprehensive way. Clarity and brevity are critical skills for researchers; shorter applications will give applicants and assessors a better chance of making fair assessments across multiple applications. Simplified budgets and reduced CV requirements that target quality would help reduce the overall length of proposals.
As an example of international best practice, the main grant scheme administered by the Swiss National Science Foundation includes CVs of the applicants and straight-forward descriptions of their expertise and roles in the proposed project. Also, the Dutch Research Council (NWO) has piloted a narrative CV format in its Early Career scheme, its major funding instrument for early career researchers. The format advances showcasing diverse types of talent and encourages assessment of quality rather than quantity and has only two categories: Academic Profile and Key Output. They describe their position in their (inter)national academic field, their motivation for conducting research and the academic and societal potential of their work. The Key Output section allows a maximum of ten output items.
• Imposing re-submission limits
The ever-growing competition on limited research funding means that more researchers are submitting applications for each call every year, putting pressure on the peer review process, and placing an administrative burden on research councils. Previously rejected applications are one reason for the rising number of applications and restrictions on resubmissions could be one way to manage demand and quality control. Such a policy would need to be coupled with improved mechanisms and opportunities for feedback on unsuccessful applications.

Q7. What improvements could be made:

(a) to ARC processes to promote excellence, improve agility, and better facilitate globally collaborative research and partnerships while maintaining rigour, excellence and peer review at an international standard?

(b) to the ARC Act to give effect to these process improvements, or do you suggest other means?

Please include examples of success or best practice from other countries or communities if you have direct experience of these.

• Funding of international collaboration
To promote excellence in global research and allow researchers to be more agile in their development of global research collaboration, the ARC should explore opportunities for supporting international collaboration through the NCGP. As noted above, the ARC’s purpose statement should include its role in supporting international collaboration.
A newly established ARC Board could be empowered to explore how international collaboration is best supported through funding allocations. Given that broadened models of collaboration may need new funding, any expansion in format should be underpinned by a commitment to value for investment and systematic arrangements to drive international collaboration while anchoring Australia’s national research effort.
International collaboration may be enhanced through mechanisms such as engaging in more joint funding programs or permitting international co-investigators to be funded from ARC grant programs. Another option would be to bring back special purpose funding schemes to support international research networks. These are utilised in Europe and work as more explicit and properly resourced mechanisms to support and resource fast-paced internationally collaborative research.
With regards to industry collaboration, we note it is important to provide sufficient time (between the announcement of scheme guidelines and submission deadlines) for companies to be involved in co-design of the proposal, review of budgets, and sign-off through the company’s internal investment processes.
• Aligned and flexible timelines for collaboration
The University notes that the ARC does not control all aspects of the timeframe for announcing grant outcomes, given the Minister is responsible for the final stage and timing of announcements. Notwithstanding this, ARC processes could be improved by significantly reducing the timelines of the grant application processes to ensure Australia does not lag in fast-moving fields of research. Examples of shorter timeframes (of 4 to 5 months) include the Marie Curie Fellowship from the European Commission, and the Lise Meitner fellowship from the Austrian Science Fund (FWF).
This issue is important because international research facilities (such as synchrotron beamlines) operate on much shorter program funding timelines, making it difficult to coordinate research efforts while waiting on an Australian funding outcome. The longer timescale also detrimentally affects Australian researchers’ ability to plan their research career and living arrangements, influencing some to choose to leave Australia for positions elsewhere. Aligning Australian research funding timelines with the global research funding environment is an important way of retaining our most outstanding talent and pipeline researchers, while also ensuring the research program stays cutting-edge technologically. These timelines also make it difficult to maintain industry engagement and commitment, particularly from small and medium enterprises, where corporate decisions are typically quicker and more agile.

Q8. With respect to ERA and EI:

(a) Do you believe there is a need for a highly rigorous, retrospective excellence and impact assessment exercise, particularly in the absence of a link to funding?

(b) What other evaluation measures or approaches (e.g. data driven approaches) could be deployed to inform research standards and future academic capability that are relevant to all disciplines, without increasing the administrative burden?

(c) Should the ARC Act be amended to reference a research quality, engagement and impact assessment function, however conducted?

(d) If so, should that reference include the function of developing new methods in research assessment and keeping up with best practice and global insights?

a. Do you believe there is a need for a highly rigorous, retrospective excellence and impact assessment exercise, particularly in the absence of a link to funding?

Research evaluation and demonstration of impact and returns on investment are important and ongoing parts of ensuring university research is an investible proposition for research funders, including government. The University agrees with the observation in the Consultation Paper that ERA has been influential in assisting researchers to focus on quality of outputs. We also strongly believe that ERA has achieved its initial purpose, and that the time and resources involved across the university sector should now be directed in other ways.
Successive ERA rounds have enabled the development of national assets, namely a systematic and comprehensive, albeit retrospective, collection of information about national research. The ARC also has a wealth of other data, such as the reports on all completed grants, and annual reports from Centres of Excellence, that form part of the national understanding of research capability. While much of the interest in the ERA reports to date has focused on individual institutional performance, such a central asset can also be interrogated to inform an understanding of national strengths in various disciplines. Modern techniques mean that by centrally maintaining and expanding to the non-university sector, a comprehensive national metadata collection on staffing, grants and outputs could occur, and would provide a valuable input to government decision making relating to the research and innovation sector.
Importantly, as modern techniques enable such a data asset to include non-university contributions, such an asset (e.g. a National Research Capability Map [NRCM]) would enable a rich picture of sovereign capability and international connections. Viewing the national research capability in this way provides a natural channel of advice on future priorities for investment, on protection of critical infrastructure, and other matters of cross-departmental government interest.
With an NRCM in place, the current focus of ERA and EI on individual institutional outcomes could, for example, be replaced by regular targeted discussions in the context of Government Compact considerations with each university, informed by their data as held in the NRCM. This process could be linked to new or existing funding for university research through the ARC, ensuring development of longitudinal measures for evaluating research, impacts, benefits, and sector-wide returns on public investment.
At Appendix A, the University has included information about administrative challenges and areas for improvement in ERA processes, in the case that ERA or a similar process it is continued by the ARC.
Recommendation 12: While ERA has been influential in uplifting quality, it should be succeeded by a data asset (such as a National Research Capability Map) that uses latest techniques to consolidate university and non-university data, which can underpin regular structured evaluation between Australian Government and universities on outcomes and return on research investment.
b. What other evaluation measures or approaches (e.g. data driven approaches) could be deployed to inform research standards and future academic capability that are relevant to all disciplines, without increasing the administrative burden?

The University’s view is that any evaluation measure or fine-grained data driven approach to ERA and EI – that is fair across all disciplines and for diverse institutions – would inevitably add substantially to the existing administrative burden on universities and would not be likely to deliver the intended commensurate increase in benefit. Recent reports from the UK on methods for research evaluation add to the weight of evidence that finding the right positioning for data-driven methods in the assessment process still requires careful analysis.
c. Should the ARC Act be amended to reference a research quality, engagement and impact assessment function, however conducted?
The ARC should be recognised as a national funding agency with responsibility to ensure allocations are fair and directed to the benefit of the nation across the dimensions of scholarly, societal and economic value. The ARC Act should acknowledge this responsibility, but express it via a set of principles to be addressed in ways that will change over time, guided by shifting contemporary expectations and changing capabilities for automation.
d. If so, should that reference include the function of developing new methods in research assessment and keeping up with best practice and global insights?

We do not see this level of detail as appropriate for inclusion in the ARC Act. Without question, the ARC should keep up with best practice and global insights, as noted in our response to Q9. We do not see it as inevitable that the ARC itself should be responsible for developing new methods in research assessment.

Q9. With respect to the ARC’s capability to evaluate research excellence and impact:

(a) How can the ARC best use its expertise and capability in evaluating the outcomes and benefits of research to demonstrate the ongoing value and excellence of Australian research in different disciplines and/or in response to perceived problems?

(b) What elements would be important so that such a capability could inform potential collaborators and end-users, share best practice, and identify national gaps and opportunities?

(c) Would a data-driven methodology assist in fulfilling this purpose?

The Consultation Paper recognises that the ARC has developed in-house capability relating to research assessment with potential to be of value more broadly across government stakeholders. Internationally, in the last decade there has been spectacular growth in activities associated with the assessment of research excellence and identification of the impact of research in scholarly, societal, and economic dimensions. Regardless of decisions on the future of the current ERA and EI assessments, we believe it is important for the ARC, as a national agency, to continue to develop a sophisticated understanding of this fast-moving field. This should include ongoing engagement with other funding agencies, nationally and internationally, but also with the fast-developing community of practitioners in the domestic research system – who can be found in universities, in publicly funded research agencies, and within other research providers.
As a next step, the ARC could position itself for sectoral leadership in this community by planning to convene, on a regular (e.g., biennial) basis, a meeting/workshop, bringing together key players on this topic, and use that to inform not only the ARC’s workplan, but to identify opportunities for collaborative engagement across government.

Q10. Having regard to the Review’s Terms of Reference, the ARC Act itself, the function, structure and operation of the ARC, and the current and potential role of the ARC in fostering excellent Australian research of global significance, do you have any other comments or suggestions?

As a standing commitment, the University of Melbourne would welcome being part of continuing collaborative discussions at an organisational or sectoral level with the ARC on new ideas and ways to improve existing processes.
In terms of areas for possible future exploration, perhaps through further consultation or review, the University suggests:
• The new ARC Board, if or when it is established, should be commissioned to undertake a review of the range of ARC schemes with a view to ensuring they remain fit for purpose. This includes considering whether they continue to meet the requirements to support research workforce capability and pipeline and build research strength across the diversity of disciplines, with attention to issues of equity, effectiveness and efficiency;
• A national consultation on research capability across the Australian research system, including a close analysis of the role of the ARC in building the researcher pipeline with a view to long-sighted national interest and innovation priorities;
• Consideration by the ARC of a new ‘partnership grants program’ or similar to support the seeding of new collaborations between academics, and/or with external partners, which could work as modest ‘start-up’ funding for fertile research ideas (and noting that relevant models for this exist in UKRI programs);
• Prioritise starting work on harmonising the IT systems across NHMRC and ARC, which would bring administrative and efficiency benefits as well as rich analytical opportunities.

Appendix A

Recommended process improvements if ERA is maintained
Feedback on administrative challenges and potential improvements to ERA
The following responses all pertain to administrative challenges and areas for improvement in ERA processes, in the case that ERA is maintained.
• Providing metadata for all creators
Page 26 of the ERA 2023 Technical Specifications notes the following:
For each research output across all types, the institution must provide the list of all creators named in the research output. However, where the research output consists of a large number of creators, the institution must provide a list of all creators in the order that they appear on the publication, up to the last staff creator claimed by the institution.
This requirement is technically difficult (the largest number of authors the University of Melbourne was to report for ERA 2023 was 3,614) and the value of the information is unclear. It is worth noting that reporting up to the last staff creator did not streamline the workload. The ARC could instead source this information via DOI or allow universities to upload the author data into its own table, which would be technically more streamlined than creating a single block of XML. Providing total number of authors would also have been a more manageable requirement.
To illustrate the issue with this requirement, the article ‘Multi-messenger Observations of a Binary Neutron Star Merger’ has 3,614 authors according to Web of Science. Of those, only seven are at the University of Melbourne. There were ten Australian universities associated with the output. In total there were 145 authors from Australian universities on the paper. Assuming all authors are eligible, that would result in 35,995 lines of repeated, redundant XML for this single paper. Noting that the University of Melbourne alone was to report 359 journal articles with more than 1,000 authors, SEER would have hundreds of thousands, if not millions of lines of redundant data that does not offer benefit in assessing the quality of research. It should be noted that it was estimated that the University of Melbourne submission was likely to be around 4 million lines of XML of which 35% was due to reporting all authors.
• Employment level being a mandatory field
All eligible academics were required to have an employment level (page 22 of the ERA Technical Specifications). This is required even though the submission guidelines explicitly state unpaid staff are eligible on page 21. However unpaid staff do not have an employment level as they are not paid by the institution. The example on page 56 of the submission guidelines notes for the fictitious academic Emeritus Professor Omega that “while unpaid, their Employment Level is equivalent to Level E – Professor”. There is no guidance on how this equivalency is determined. This could be done via title but not all honorary appointments have a title that corresponds to an Employment Level (e.g. ‘Reverend’ does not directly map to an Employment Level). A status of “Unpaid/Honorary” could deal with this issue.


• Field of Research (FoR) Code required for all eligible academics
The ERA rules assume that all eligible academics will have research outputs, and for the purposes of ERA, all academic staff are required to have at least one FoR code. However, some eligible academics do not have FoR codes. The workaround was to have a default code assigned to these people. However, it is our view that applying a field of research code to someone who is not research active is not recommended.
At the University of Melbourne, academic FoR codes were determined by the FoR codes assigned to their research output and proportioned out accordingly. This could be done centrally by the ARC and would ensure all universities were treated equally in the assessment process. Those staff who do not have research output could be flagged as “Not Active”.
• BR133 – Reassignment of FoR Codes
As stated on page 45 of the ERA 2023 Business rules and Verification document:
If the FoR code assigned to a journal article does not match the FoR code(s) specified in the ERA 2018 (sic) Submission Journal List then the apportionment value of that FoR code must be at least 66% Note: Indigenous Studies (FoR45) is exempt from this rule
This business rule is confusing and its purpose unclear. This business rule cannot accommodate two FoR codes not matching those on the ERA Journal List (as the apportionment must be 66%). The exception for Indigenous Studies confuses the matter further and reduces the impact of the Indigenous component of the research. For example, if an article is in the Journal of Indigenous Research, and 50% of that article are to be reported under 4410 Sociology, the percentages would have to be changed to accommodate Sociology (where the rule would still apply) resulting in only 33% of the research being reported under code 45. This rule requires a great deal of manipulation of the data in order to meet its requirements.
• Research Statements for Research Reports
Under previous ERA rules, a research statement explaining the research content of a research report was a mandatory data element. In general, this is not needed as the research content of a report is clear: most research reports have an abstract or executive summary. Providing a research statement for a piece of art or a dance performance is helpful ,but providing a research statement for a research report seems unnecessary and this requirement should be removed as part of any future assessment.
• Meta-data requirements
Universities should still provide data on the different output types: books, book chapters, journal articles, conference publications and non-traditional research outputs. However, reducing the metadata requirements would substantially reduce the workload on universities – for instance, using DOIs to extract all the relevant metadata is a method that should be explored.
In terms of standardising publisher names, there were over 10,000 different publishers listed in the ERA Publisher List. However, the majority of the books and books chapters in the University of Melbourne data were concentrated in around 60 publishers. Standardising the most common publishers would be enough to provide information on the publishing patterns of a discipline. There is a massively long tail of outputs where only one publisher is evident in the dataset. The value of standardising every small publisher is not clear. Along similar lines, standardising the name of every conference is not required. Conference publications are most relevant to the Computer Science and Engineering Disciplines. Standardised naming of the conference organiser (e.g. IEEE) rather than the conference itself should be enough to ascertain the publishing patterns in these disciplines, rather than standardising the name of every particular conference. Conference names are not standard and a large amount of work goes into standardising the names for ERA purposes.


• IT Systems should not dictate business rules
ERA was introduced to the sector in 2009 and the fundamental approach has not changed since then. Universities were still expected to produce a single XML file for ingest and validation. When ERA was introduced, large author lists were not common. The research landscape has changed dramatically since then. For instance, the University of Melbourne was going to report over 1,205 publications with over 50 authors – 359 of these outputs had over 1,000 authors. It is technically very difficult to de-normalise this data using SQL. This is partly because the Oracle listagg function (which is required to loop through the data) stops working at 4,000 characters. The alternative is using the XMLAGG function which does not have a character limit. However, this will create the list of contributors as a clob field which cannot be exported from Oracle Developer. It needs to be copied and pasted into a text editor and have all non-printing characters removed. As the data is already normalised at source, having to de-normalise the data is labour-intensive and technically very difficult. The data could have been uploaded table by table, which would not have required a complicated XML schema. It is worth noting that the tagging service provided by Clarivate as part of the exercise did allow upload of data via flat file rather than using a complicated XML schema.
An added issue with SEER was the method of validation. Validation occurred on the entire data set – a university could not, for instance, upload just its publication data and have the data checked against business rules. It had to upload an entire submission. If there was an issue with the staff data, then the validation would stop at that point. This made it very difficult for universities to ensure that their data was compliant with the business rules.
• Academic involvement should be reduced
ERA has been a highly labour intensive academic intensive exercise. The expectation of a Research Evaluation Committee (REC) member is to secure 36 days of their time to participate in the panel. For a REC Chair, the commitment expected is 42 days. In addition to these formal academic roles, universities must draw on discipline level committees or leaders to ensure coding decisions are discipline appropriate for making peer review selections, in the writing of background statements and to prepare research statements for Non-Traditional Research Outputs.
• External data should be used where possible
There have been many advances in the bibliometric space since ERA was introduced in 2009. Yet the same business rules and methodology that were introduced at ERA’s inception remain. For example, the use of Digital Object Identifiers (DOIs) is now commonplace. The ARC could extract metadata directly to reduce duplication across the sector. Furthermore, using DOIs to populate metadata in SEER would ensure greater consistency of data.
• One Central Peer Review Database
Under previous ERA exercises, each Australian university was required to grant access to peer-reviewers to items selected for assessment via their own repository or database. This creates multiple points of failure across the system as well as a lot of duplicated effort – if universities are reporting the same output for peer review it is a duplication of effort to have that same output available in different locations. Having one central database for the collation of peer review items addresses this issue. This approach could also lessen the load on universities as the ARC could source outputs themselves using DOI – universities could then provide the outputs that could not be sourced centrally.
• Use of a third-party benchmarking citation service such as InCites or SciVal
Using a third-party provider will not only reduce the burden on universities, but will also address many of the issues associated with ‘gaming’ the system. Issues of equity (such as the ability to ‘model’ an ERA submission) would be addressed. The University of Melbourne has conducted an analysis, which it is willing to share with the ARC, demonstrating it is possible to obtain similar patterns of results at the two-digit level using SciVal data for 21 of the 23 divisions.
Additional advantages to using a third-party provider include:
1. An output authored by academics at different institutions will be treated in exactly the same way.
2. Balancing out workloads across universities of different sizes.
3. This approach would facilitate benchmarking against international leaders in the field without requiring a separate approach or rating scale.
• Report assessment outcomes differently for different methodologies
The well documented discrepancies between ratings for peer reviewed disciplines compared to those for citation-based disciplines is suggestive of a fundamental difference in the assessment methodologies, and one that many commentators have identified as problematic. The growth in five ratings has been much greater in the citation-based disciplines than the peer reviewed disciplines. An analysis of Field Weighted Citation Impact taken from SciVal shows minimal movement for the nation over the past 12 years – so the results of ERA do not correlate with a well-known and validated measure of citation performance. The likely explanation is the ability to model the citation-based disciplines using the citation benchmarks (either provided by the ARC or external providers) rather than an increase in the quality of research being produced by Australian universities.
A possible approach to recognise methodology differences is to have different assessment outcome descriptors for the different methodologies, as illustrated below. A further benefit of adopting different assessment approaches is that it would dissuade the creation of league tables.
For instance, ratings for citation analysis could be categorised by the data itself:
1. High Volume, High Quality
2. Low Volume, High Quality
3. High Volume, Low Quality
4. Low Volume, Low Quality
We acknowledge that there would need to be more nuanced assessment – the above example is used to illustrate that the assessment could be rooted in the analysis of the data itself.
For peer review, the assessments should reflect the qualitative nature of the exercise. For instance:
1. High quality research with international applicability
2. High quality research with national applicability
3. Low quality research with international applicability
4. Low quality with national applicability
Again, the point of this example is to illustrate a possible assessment based on peer review – more work would need to be conducted to determine appropriate classifications.
• Two vs Four Digit Assessment
Given the wide disparity of four-digit codes that underpin many of the two-digit codes, a discussion needs to be had as to whether there is enough discipline expertise available to make robust judgements at the four-digit level. An alternative could be for RECs to rate the research at a two-digit level and note the contribution/spread of four-digit codes.
• Remove FoR Code Apportionment
The current requirement to assign percentages/apportionments to FoRs associated with outputs is a disincentive to report cross-disciplinary research. Using apportionment also encourages gaming in order to maximise citation performance using benchmarks (either from the ARC or via external providers). As the citation analysis could be a separate exercise, the incentive to report journal articles under one FoR code with an apportionment of 100% would be removed. This would also allow the ARC to get a much clearer understanding of the inter-disciplinarity of Australian research.
This issue is of particular concern with the introduction of the 45 Indigenous codes. If a university wanted to maximise its citation performance, it could do so by not indicating the Indigenous component of a citation-based discipline (e.g. Public Health) hence not losing 50% of the citations to the peer reviewed 45 code.
• Make use of existing sources
Much of the data required for an assessment of engagement can be found in existing ERA data. For instance, Category 2 and 3 research income and research reports are two metrics that can be derived from the data. In addition, tools such as SciVal can provide data on Academic-Corporate Collaboration in the publication space. The qualitative aspect of engagement could be addressed in the Explanatory Statement.
Project reports provided to the ARC at the conclusion of funded grants, if provided in helpful formats, could also be used as the basis for other forms of outcome reporting.
• Assessment of publications should be comparable across institutions
A paper should be assessed in the same manner regardless of which institution reports it. However, this has not been the case in previous ERA rounds. As a hypothetical example, consider a journal article with 20 citations and co-authored by academics at two universities. One institution reports that paper under a particular FoR code that has a normalised world benchmark of 10 – that paper will be assessed at two times the world average (i.e. it will have an RCI of 2.0 – 20 cites/10). The other institution reports the paper under two FoR codes with a 50% split and both those codes have a normalised world benchmark of 12 – the same paper is now assessed as having two RCIs of .833 (20 cites*50%/12) and now is seen as below world average. This points not only to a lack of reliability but again to the point that universities that are more adept at modelling their submission can affect their ratings. This also means there is a strong disincentive to report cross-disciplinary research.
• Adjust the methodology to account for institutional scale in a discipline
ERA business rules treat universities as though they are all the same size. This is best reflected in two areas: the volume threshold and the percentage of items required for peer review. In terms of the volume threshold, having a standard 50 outputs approach means large institutions have less opportunity to “hide” their lower rated research. To illustrate this, consider a university which has 100 outputs reported under a particular FoR code and they deem 50% of that research is low quality. They could move 49 of those outputs into a code which won’t reach the threshold for assessment and therefore exempt them from assessment. Consider another institution which has 1,000 outputs under the same FoR code and they deem 20% of their research is low quality. That equates to 200 outputs – they cannot move 200 outputs to a code that will be below the threshold limit. They can move 50 out but now have 150 outputs that are of lower quality remaining. The result is the smaller institution has a stronger submission than the larger institution despite having an overall higher percentage of lower quality research.
• Greater effort to reduce the likelihood of ERA and EI being used to create league tables
An approach to make post-hoc league table construction by third parties more challenging could be to provide outcomes in a more complex format, for example showing where institutions cluster on several metrics rather than a simple rating. The University of Melbourne stands willing to provide examples of this type of visualisation.

Submission received

16 December 2022 (extension given)

Publishing statement

Yes, I would like my submission to be published and my name and/or the name of the organisation to be published alongside the submission. Your submission will need to meet government accessibility requirements.