AI Bibliography

Engaged and taking on responsibility earlier, more decisively, and more substantially – that is how German foreign and security policy has come to be envisaged over the last few years. However, it is an open and ongoing debate how Germany will meet its growing responsibilities, especially with regard to its armed forces – the Bundeswehr. This report argues that there is an emerging nexus of security and new technologies which provides a litmus test regarding the fundamental norms and values that Ger- many and the Bundeswehr heed whilst taking on additional responsibilities. It specif- ically focuses on the military use of artificial intelligence (AI) and robotics in so-called autonomous weapon systems (AWS).

The report adopts a working definition of AWS that, in accordance with definitions provided by the International Committee of the Red Cross (ICRC) and the US Depart- ment of Defense, puts emphasis on autonomy in a weapon system’s critical functions, that is, target selection and engagement. However, the report reframes this issue in terms of 5 levels of human supervisory control. This allows for closely examining the governance of AWS and teasing out what is meant by notions such as «appropriate levels of human judgement», «human oversight», «human in» or «on the loop» and «meaningful human control».

The report argues that human supervisory control over critical functions must be subject to careful scrutiny considering the legal, technical, moral/ethical, and security implications of AWS.

Legally, all existing evidence indicates that the deployment of AWS could not comply with International Humanitarian Law (IHL) for at least the foreseeable future, and that they pose as yet unresolved problems regarding accountability and responsi- bility for the use of violent force.

Technically, autonomous weapons lack the necessary components to ensure compliance with the IHL requirements of distinction and proportionality. Their behavior is inherently unpredictable, particularly in scenarios where multiple AWS would interact.

Morally, the guiding principle of respect for human dignity, enshrined in Ger- man basic law («Grundgesetz») Article 1 (1) as well as in International Human Rights Law, dictates that machines should not be making life or death decisions regarding humans.

In terms of global security, the development of AWS poses serious dangers for regional and global stability and provides incentives for their proliferation, including their use by actors not accountable to legal frameworks governing the use of force. In light of this, the report develops the following recommendations to the German government : develop a national policy on the use of autonomy in weapon systems and make that document publicly available; adopt a simple, ICRC-like definition of autonomous weapon systems, based on the system’s «task autonomy» in the performance of the critical functions of selecting and engaging targets; stipulate the legal requirement of «meaningful human control» over all Bun- deswehr weapon systems, avoiding autonomy in target selection beyond level 3 («software selects the target and a human must approve it before the attack») thereby effectively prohibiting «fully autonomous weapon systems» at the national level; continuously investigate and map out levels of human control for autonomous functionality in future Bundeswehr weapon systems to allow the human reason- ing and control needed (that is, levels 1-3 for critical functions), and train Bunde- swehr personnel accordingly with regard to the required tactics, techniques and procedures; regulate the Bundeswehr’s continued use of defensive SARMO (Sense and React to Military Objects) systems to satisfy the strict targeting limitations and con- straints on design and operations specified in this report; continue and intensify support for an international, legally binding and verifiable ban on fully autonomous weapon systems, that is, weapon systems with auton- omy in the performance of the critical functions of selecting and engaging targets.

Regarding the international prohibition on AWS, adopting a leadership role and work- ing even more actively and decisively toward this goal at the United Nations would represent an exemplary milestone in Germany’s new foreign and security policy. It would send a strong signal that Germany is heeding its fundamental norms and val- ues whilst living up to its newly grown international responsibilities.