Georgetown Law
Technology Review

Abstract

Justice system agencies around the globe are rapidly adopting digital technologies to administer a justice system or service, create access to that system or service, or increase the agency of justice system-involved people through online support. However, in many places, procurement processes—how governments buy goods and services—are too narrow or outdated to competently vet the newest technology. This leaves technologies and vendors under-investigated, which creates human rights harms for people in the justice system and the public at-large. To ameliorate the problem, this Essay proposes the use of the sustainable procurement model to create a human rights-based approach to buying justice technologies. To accomplish this, the Essay provides questions that agencies can ask to meet human rights obligations by better planning technology projects and vetting potential bids.

Introduction

Public procurement can make better public policy.

Called sustainable procurement, this approach goes beyond the transactional nature of public procurement, the government’s purchase of goods and services, by considering economic, environmental, and social factors when determining which bid should be selected for a specific contract. Over the last decade, sustainable procurement has primarily focused on either (1) the environmental and labor practices used to create a product or service sold to governments or (2) the demographic makeup of companies bidding on a contract to increase opportunities for minority-owned businesses and marginalized communities.

At the same time, civil and criminal justice systems around the globe have rapidly digitized, a trend accelerated by the pandemic. These so-called justice technologies refer to technology or data projects or digital services purchased by a public agency to administer a justice system or service, create access to that system or service, or increase the agency of justice system-involved people through online support. The technology used by justice agencies, such as police, prosecutors, corrections, and courts, today creates substantive obligations for human rights. Unprepared to buy technology with these implications attached, the current state of procurement has opened the door to privacy violations and surveillance, injecting bias into judicial processes, and limiting the ability of justice agencies (like courts and prisons) to be open and transparent. Although these problems are largely inflicted on justice systems with higher levels of digitization, these products and their harms crop up in the developing world as well, making this a global issue.

To rectify this deficiency, this Essay proposes a human rights-based approach to purchasing justice technology that improves the sophistication and ability of government buyers to safeguard the rights of justice system-involved people, such as criminal defendants or civil litigants, and the openness of democratic institutions.

The goal of this Essay is two-fold: (1) to carve out a place in the sustainable procurement framework to protect the human rights of justice system-involved people and (2) to prompt questions that could improve justice agency sophistication of technology procurement.

To accomplish these goals, this Essay first explains the harms created by current justice technologies and procurement practices. Second, it explains sustainable procurement and how it is used to improve respect for human rights by government agencies. Third, the Essay outlines questions that a procuring agency can use to better evaluate their potential projects, requirements, and vendors. The Essay concludes by drawing attention to other areas of research needed.

I. The Harms and Shortcomings of the Status Quo

The justice system is rapidly adopting technology.

Whether digitizing paper documents in case or record management systems, providing legal information online, or setting up video conferencing for judicial proceedings, the software that makes these systems operate is primarily created by private businesses. At the same time, justice agencies often lack the technical and legal expertise needed to vet potential projects, vendors, and risks appropriately. This sophistication gap between suppliers and buyers is, in part, perpetuating human rights issues in the justice system.

For the purpose of this Essay, justice technology refers to technology or data projects or digital services purchased by a public agency to administer a justice system or service, create access to that system or service, or increase the agency of justice system-involved people through online support. This Essay does not include explicit surveillance technologies bought and used by law enforcement agencies. State actors buying this technology include courts, administrative agencies, police, prosecutors, legal aid and defender services, probation and parole officers, and corrections departments.

As the complexity and potential for justice technologies create a need for more sophisticated procurement practices, many procurement systems are not keeping up. Even in jurisdictions that have adopted sustainable procurement frameworks, substantive rights implications of new technologies in the justice system are not always considered, which leaves open the potential for problems. For example, the increased use of data and software, like algorithmic risk assessment tools used to aid decision making, like bail decisions, in the criminal justice system “are blurring contemporary regulatory boundaries, undercutting the safeguards built into regulatory regimes, and abolishing subjectivity and case-specific narratives.” Far-reaching concerns have been found in the civil justice system, as well, which impacts how people access housing, public benefits and even workers’ rights.

These challenges which are created by justice technologies impact numerous human rights obligations, including equal protection under the law, privacy, access to opportunity, and access to public information. Not just being pointed to by advocates, these impacts have been noted by the United Nations’ Secretary- General, the Office of the High Commission for Human Rights, the European Union Agency for Fundamental Rights, the American Bar Association, and the Law Society of England and Wales, among others.

A more holistic procurement process is a check on the fact that justice technologies can create or exacerbate problems for the following rights, as described in the UN Declaration of Human Rights:

1. Equal Protection Under the Law

Guarantees of equal protection under the law and the right to a fair trial free from discrimination are undercut by algorithmic tools and biased data.

Algorithmic tools can inject bias into the judicial process, which can reinforce, create, or automate discrimination against disadvantaged groups. Currently, algorithmic tools called risk assessments are used to assist the determination of pretrial detention, sentencing, and parole. The data the tools are built on have been shown to be biased against communities that have historically been over-policed. Similarly, while the algorithms may include “static” inputs, like age and criminal history, researchers have found that the tools still recommend harsher judicial responses for racialized communities. Similarly, predictive policing algorithms, which recommend areas to increase police presence  reinforce discriminatory policing trends and arrests in disadvantaged communities. Many of these algorithms are privately held, and judges, defense counsel, advocates, policymakers, and the public are not allowed to see how the tool works or independently validate their accuracy, allowing bias to go unchecked. Only the company knows what’s going on inside the algorithm.

Tools like these encode and reinforce bias in the justice system, and make it difficult or impossible to see how they function. This automates discrimination and discards with equal protection under the law.

2. Privacy

The collection, sale, and breach of justice system data—that is, data created by justice system agencies or data generated on account of a person’s contract with the justice system—exposes justice-involved people to privacy violations.

Justice system data is being collected by private vendors on behalf of government agencies with no clear policy regarding how that data is used. At the same time, courts and other justice agencies do not proactively or retroactively inform data subjects of how their data may be handled. While some agencies have proactively protected personal identifying information, others sell or give away their data without a complete understanding of the data’s value or power. In other instances, there are explicit trades between companies and governments for citizens’ data. For example, governments in Angola, Ethiopia, and Zimbabwe traded away their citizens’ faces for access to facial recognition software. In other instances, legal confidentiality between a person’s attorney, spouse, or clergy is contracted away when prisons buy communications technology from certain vendors, whether or not there’s legal grounds to do so.

Digitization of justice system data also increases vulnerability to cybersecurity lapses and breaches. In one example, the U.S. federal courts were breached during the sprawling SolarWinds hack, which compromised electronic filing and potentially sensitive documents. Courts in Brazil and elsewhere have also been hacked. In Chicago recently, GPS ankle monitoring information, including the names and whereabouts of the monitored individuals, was accidentally leaked online. Similarly, it seems a major court case management vendor left open access to non-public databases, allowing another company to scrape and publish that data, leading to the publication of non-public data.

Leaks like this hurt the data subject, like through identity theft,  and undermine the credibility of the justice system by giving citizens less reason to trust it. When this information is published online without controls or sold to data brokers by a public agency or its vendor, it can be used in ways that impact the data subject’s access to opportunity. In more extreme examples, the breach of this data can threaten the lives of data subjects, as illustrated by the Afghan government’s biometric database falling into the hands of the Taliban in 2021.

3. Access to Opportunity

The sale and breach of justice system data directly hurts people’s access to public benefits, work, and housing.

While the single sale of a person’s information may seem merely inconvenient, the future harms and society-wide impacts are real. As justice system data about a person is collected and sold, it is used alongside other data to build profiles on people, which can have significant consequences  As algorithmic decision-making becomes more common in public benefits processes, the opportunity for people to be miscategorized as someone not qualified for and rejected from benefits they are entitled to grows. Access to private markets, like employment and housing, are also impacted. In the U.S., mugshots—historically police administrative data—are reposted by private companies online, which potential employers see as grounds not to hire someone, regardless if the case’s outcome was favorable to the applicant. This depresses the earning potential of those who have been arrested. Going further, mugshot websites will also extort the data subjects for money in exchange for taking down the photo. Similarly, and more subversively, social media sites use private data, like that sold or produced by justice agencies, to discriminate against marginalized groups by limiting their access to housing and employment.

Leaked, breached, or sold justice system data can be used to limit people’s access to basic social needs.

4. Access to Information & Government Transparency

By outsourcing data management from the government agency to software companies and adopting algorithmic tools, private entities are limiting public and government agency access to information.

Private entities are limiting public and agency access to public data because those private entities can make significant profit by controlling the data. For example, police record management vendors attempt to limit public and agency access to raw data collected by the agency by not creating a report feature or up-charging for the publication of reports and data exports. In another instance, a major vendor’s terms-of-use policy for a crime map built with public data collected through a public contract requires puts limits on that data by stating “[t]he content of the Site cannot be used, posted, sold, transmitted, distributed, modified or transferred for public or commercial purposes.” Similarly, a court’s public website has terms of use that limit  public access to data through the portal. One company filed a protective order against a person for requesting public information from and about a police record management system. In a separate instance, a vendor that managed a public police data portal sued a company that used the portal to download data under that country’s anti-hacking statute.

Monopolizing access to public data also results from a technical issue. Vendors who run case and records management systems make it hard for other vendors or researchers to use public data by limiting interoperability, the ability for software to “talk” to other software. A recent report looked at court technology adoption in multiple countries and found that information systems are being deployed, but not integrated with older systems, creating silos and new ways to data-sharing. As prisons digitize their services, some vendors are putting forward  solutions under what are effectively state-granted monopolies. This means that every product used, and all the data generated, in that facility would be managed by a single vendor, creating a monopoly on access to prisoners. When vendors don’t build their tools with accessibility in mind, disabled people who rely on screen readers or other accessibility aids to function online are kept from valuable information and services.

Further, many of these tools, hiding behind intellectual property protections, obscure public legal processes. The inability to look inside a company’s software has led to defendants being unable to confront the accusations against them and multi-million dollar public contracts that later found that there was no software at all.

As demonstrated, government contracting of certain justice technologies are linked to adverse human rights impacts on people and communities—but they don’t have to be. Improving the sophistication of government buyers and incorporating sustainable procurement practices can help countries fulfill their human rights obligations.

II. Sustainable Procurement and Human Rights Protection

Nations have an obligation to protect human rights. This includes protecting citizens from abuses by third parties, including private companies, according to the UN Guiding Principles on Business and Human Rights (UNGP). These rights are vested in  laws and treaties which require governments to take or refrain from particular acts. In the past decade, there has been a growing consensus that when government procurement balances cost, quality, and sustainability, it can serve as a lever to improve both public and private fulfillment of human rights obligations.

This rights-based approach to procurement is called sustainable procurement. It expands the scope and impact of government procurement processes, which have traditionally focused on a transactional “value for money” standard, a check against corruption, and anti-competition measures. The sustainable approach goes beyond that by leveraging a government’s buying power to promote the responsible production and consumption of goods and services that help governments meet the United Nations’ Sustainable Development Goals. It also requires that private companies take actions they would otherwise not, because they are not incentivized to do so.

The sustainability model is built on environmental, economic, and social pillars. Since the environmental and economic pillars are not relevant to this argument, I focus on the social pillar here. The social pillar of sustainable procurement is informed by the Universal Declaration of Human Rights (UDHR) as codified by the International Covenant on Civil and Political Rights (ICCPR); the International Covenant on Economic, Social and Cultural Rights; and the International Labor Organization’s Declaration on Fundamental Principles and Rights at Work. In application, this work has focused on protecting labor rights and supporting disadvantaged groups, such as minority business owners.

Leading countries on sustainable procurement have expanded what the social pillar covers to improve communal and societal policy goals, such as increased charitable contributions and access to broadband. To date, however, this pillar has yet to be expanded to include the rights of justice-involved people, even though the UDHR and ICCPR speak explicitly to their rights.

III. Increasing the Sophistication of Justice Tech Procurement

Due to the changing nature of technology in the justice system and its substantive impact on people’s rights, the social pillar of sustainable procurement must be built upon. Doing so, governments can more proactively protect the rights of people impacted by the justice system and bring the government and the private sector in line with UNGP and other international human rights standards. Already proposed in the various documents outlining the governance of Artificial Intelligence (AI), this approach should apply to any software, such as those discussed in the previous Section.

This approach has numerous benefits. A procurement system that considers the impacts, access, and accessibility of technology is optimized for human rights protection and puts the focus on citizens involved in justice system, resulting in a people-centered-approach. For governments, a human rights-focused procurement process will help avoid the negative experiences discussed in earlier sections for people with justice system contact. This will bolster the credibility of the justice system and the rule of law, and avoid legal and political liabilities. Last, having rights-abiding justice technologies supports advocates for other Sustainable Development Goals.

The movement toward sustainable procurement may require enabling laws, ordinances, or resolutions. However, justice agencies in almost any regulatory environment can do a better job evaluating their own requirements and the vendors bidding on contracts. Below are 18 questions that agencies and procurement officers can use to improve the scope and design of new projects, write requests for information and proposals, and ultimately vet vendor bids.

The application of these questions is country specific. Local administrative, contract, anti- corruption, and privacy or data protection laws, for example, may guide or supersede some of the questions. These questions are pulled from existing research, especially regarding AI, interviews conducted for this Essay, and best practices from technology development literature. They are organized by theme. The first group of questions relates to the early-stage planning of a technology project. The second set helps assess potential vendors and solutions. The last collection of questions regarding liability.

1. Planning a Technology Project

As the preceding sections have illustrated, even the most banal technologies can have major repercussions for human rights. When planning technology projects, agency due diligence needs to consider whether technology is the solution to the problem, the potential positive and negative knock-on effects of the proposed plan, and the maintenance considerations of a new tool or process.

When considering a new technology project, the agency should ask itself:

• How can this technology impact the rights of people engaging with or trying to access the justice system? The use of technology is not alone a good thing. Considering the knock- on effects of the proposed project will put human rights in the center of its design and improve due diligence. This should include interviews with people impacted by this technology.
• Does the problem the project looks to solve stem from a technology issue, or is it something else? Many justice systems use antiquated processes and rules. Merely digitizing a byzantine process will not fix the problem and may, in fact, make an unjust system more efficient, exacerbating existing harms.
• Does the agency have the human and technical capacity to maintain the proposed project? Beyond the purchase of the software and its implementation, the agency’s human, monetary, and technical resources will need to be sufficient to onboard the project and keep it operating and updated. Without these supports, a project’s potential is diminished.

2. Assessing Potential Vendors & Bids

Technology projects exist in an ecosystem of people, process, and technology. This interconnected reality requires that regulators look beyond the narrow goals of a specific project and consider broader technical and legal issues, including issues of interoperability as well as how data is collected and processed. As justice agencies collect more data, they need to consider themselves fiduciaries of that data. Failing to safeguard data subject privacy can cause public mistrust and harm.

Interoperability considerations:

• Does the vendor provide an open application programming interface (API) with clear documentation? Does the vendor charge a fee to access the API? If so, to whom? APIs allow technologies to “speak” with other technologies. Requiring them will help avoid data and technology silos within agencies and across governments. Without requiring APIs with clear documentation (the instructions on how an API works), agencies run the risk of being charged extra by vendors and limiting their own access to the data.
• Does the vendor require a closed network that only it and its subsidiaries can provide services? Closed networks are one way that vendors lock-in agencies and create a monopoly, stifling innovation and increasing costs.
• Does the software meet accessibility standards? Disabled people that use screen readers or cannot navigate with a mouse require particular software or specific web design to make digital content The Web Content Accessibility Guidelines (WCAG 2.2) are the international standard for web accessibility. Vendors should be expected to meet them.

Data management and processing considerations:

• Is the agency or vendor collecting, or requiring to be collected, more data than is needed for the purpose of a project? Because digital data collection makes it easy to collect copious amounts of data, it is important to scope data collection strictly to what is needed. This is the principle of data Following this principle decreases risk to data subjects and their privacy, especially if the data is breached in a security incident.
• How long will the data be retained and at what point will it be deleted? For both the agency and the vendor, transparent data retention and deletion policies should be established to clearly manage data and protect data subjects.
• Who owns the data collected and processed under the contract? The public agency is creating public data, regardless of whether the agency is aided by a private That data, as determined by local law and practice, must remain in the public domain and must not be allowed to be privatized by the vendor.
• Who has access to the data created, stored, and transferred by the contract? Setting contractual standards on which outside parties can access the data is a good way to keep the data secure and diminish opportunities for abuse. Similarly, agencies should create permissions to internally control access to the data.
• In what format will the vendor make the data available? To avoid vendor control over data and reporting, the agency should have access to all data collected by a vendor in a usable format, such as the comma separated values (CSV) format.
• How, if at all, will the vendor profit from the collection of public data outside of the immediate contract? Companies often have other revenue streams in mind for the data they collect through contracts with government agencies. It is incumbent on the agency to ascertain how the company intends to use the data and agree to a contract that creates appropriate limits that protect data subject privacy, for example not allowing the data to be accessed or sold by data brokers.
• What happens to data and other digital artifacts if the vendor is sold, closes, or changes business models? The promises a vendor makes may depend on the vendor’s continued stability. Planning ahead for future changes to the vendor’s operations help prevent misuse or loss of

Transparency considerations:

• Is the software’s source code, training data, operation, and validation public or reviewable by a qualified party? Software systems, especially AI, should only be adopted if the stakeholders of that system have access to pertinent information about its design, development, operation, and validation. For anything short of this, government agencies run the risk of buying uncertain technology, creating biased outcomes, or limiting public access to previously public processes.

3. Defining Liability, Remedies, & Responsibilities

Holding companies accountable for their impacts on human rights only works if liability is applied and there is an available remedy.

• What is the remedy for persons who have their rights violated? Without remedies and access to them, a human rights standard for justice technology procurement will not be successful. Remedies are determined by local law and the regulation of the contracting agency. According to the UNGP, remedies “may include apologies, restitution, rehabilitation, financial or non-financial compensation and punitive sanctions (whether criminal or administrative, such as fines), as well as the prevention of harm through, for example, injunctions or guarantees of non-repetition.”
• What knowledge and skills are required to understand, explain, and operate the program? Technology creators need to specify the competencies needed to effectively operate the technology. Doing so, technology creators may increase the efficacy of the technology and limit liability risks for both the vendor and agency.
• Who is responsible for any rights violations created by the use of the software? Most countries have laws that hold companies accountable under criminal and civil actions, including torts, breach of contract, and consumer protection. The liability of a technology vendor, when their product is used by a government agency to the detriment of someone’s human rights, may or may not be clear. In other instances, companies try to sign away their liability in the contract. Answering this question early will define responsibilities.
• For public-facing products, who is responsible for managing customer service issues when they arise? The success of a public-facing project depends in large part on the public’s experience. Defining who is responsible for specific customer needs will create a clear division of labor and responsibility that increases customer success and satisfaction.

IV. Topics in Need of Research

The research surrounding technology and procurement in justice systems remains under-researched. For that reason, a deeper look is needed into the following questions:

• What impacts do vendors’ terms of service, privacy policies, and boilerplate contract language have on human rights and justice agency functionality?The harms discussed in Section III of this Essay provide a snapshot of the current landscape. The next step is to collect government contracts for justice technologies to find commonalities across good and bad terms. Doing so will provide a significant pool of information for researchers, advocates, and government agencies to pinpoint troubling clauses and propose counter language that protects human rights.
• Should the justice sector require human rights and algorithmic impact assessments? If so, what should they look like? The UNGP calls for “human rights due diligence,” which takes traditional business risk analysis and applies it to the potential human rights risks a company creates through its product or practices. There have been numerous human rights and algorithmic impact assessment frameworks developed. Their use and application has been studied and adaptations have been proposed. Existing frameworks will need to be changed to address the specific concerns regarding the justice system.
• Do justice agencies need external standards or third-party certifications to improve their buying sophistication? If so, what should they look like?The sustainable “green” and labor procurement movements (i.e., sustainable procurement focused on environmental and labor rights) have been aided by third-party certifications of vendors. These third-party certifications, like the Global Recycling Standard and Designated Suppliers Program, help buyers know a product or vendor meets a particular standard. Applying this approach to justice technology could create standards for transparency, validation, and accountability, for example.
• How can justice technology vendors be incorporated into the development of sustainable procurement policies and practices?The integration of a human rights-based approach to procuring justice technologies will come with trade-offs. Specifically, certain standards may lead to increased prices or ward off certain companies from bidding. Vendors should not drive the conversation about sustainable procurement, but they need to be included in it. Working with government vendors in the development of sustainable procurement practices may hone new processes and increase the understanding and acceptance of new standards and expectations.

Appendix

Unless cited in the Essay, no idea or section is attributed to an interviewee or their organization. Interviews occurred over video or voice call between February and April 2022. Each entry is cited by name, relevant title, and organization.

Nikki Archer, head of procurement and commercial policy and strategy, Scottish Government. Maurits Barendrecht, research director, Hague Institute for Innovation of Law (HiiL).

Cathrine Bloch Veiberg senior advisor, Danish Human Rights Institute. David Bonebreak, program counsel, Legal Services Corporation.

Bob Bullock, senior counsel, US Department of Justice.

Matthew Burnett, senior program officer, American Bar Foundation.

Natalie Byrom, director of research and learning, The Legal Education Foundation. MJ Cartwright, mentor-in-residence, University of Michigan.

Michele Gilman, professor, University of Baltimore School of Law. Laura Guzmán, co-deputy director, the Engine Room.

John C. Havens, sustainability practice lead, IEEE Standards Association. Danielle Hirsch, interim court services director, National Center for State Courts. Hunt La Cascia, senior procurement specialist, World Bank.

Reilly Martin, senior program manager, Open Contracting Partnership. Daniel Morris, advisor, Danish Human Rights Institute.

Juan Ortiz Freuler, affiliate, Berkman Klein Center, Harvard University. Dory Reiling, retired senior judge, Dutch Judiciary.

Jane Ribadeneyra, program analyst for technology, Legal Services Corporation.

Sarah Rodriguez, researcher for digital service network portfolio, Beeck Center, Georgetown University.

Gilbert Sendugwa, executive director, Africa Freedom of Information Centre.

Brittany Suszan, vice president, SpotCrime.

Jason Vail, division director, American Bar Association. Justin Valentine, strategy and policy consultant, World Bank.

Nicole Vander Meulen, senior advocacy counsel, International Corporate Accountability Roundtable.

Hillary Viita, head of department for law enforcement information systems, Estonia. Bianca Wylie, senior fellow, Centre for International Governance Innovation.

Zach Zarnow, principal court management consultant, National Center for State Courts.