The integration of artificial intelligence into clinical research is accelerating, but it brings a thorny question to the forefront: when a CRO’s AI system sends a query, who is actually accountable for the response? This issue, highlighted in a recent report by The Clinical Trial Vanguard, is becoming a critical point of friction as AI tools take on more responsibilities in trial management.

The AI Query Conundrum

Clinical Research Organizations (CROs) are increasingly deploying AI to handle data queries, patient monitoring, and even regulatory submissions. These systems can process vast amounts of information in seconds, flagging anomalies or asking for clarifications that would take human staff hours to identify. But with this efficiency comes a new layer of complexity: determining liability when an AI-generated query leads to a wrong decision or a compliance breach.

The issue is not hypothetical. In practice, AI systems are already sending queries about patient eligibility, data inconsistencies, and protocol deviations. When these queries are based on flawed algorithms or incomplete datasets, the consequences can be severe—ranging from delayed trials to regulatory penalties. Yet, the current regulatory frameworks are largely silent on the question of AI accountability, leaving sponsors, CROs, and technology vendors to navigate a murky legal landscape.

Who Bears the Responsibility?

There are several potential parties who could be held accountable for AI-generated queries:

  • The CRO – As the entity deploying the AI, the CRO may be seen as ultimately responsible for its outputs, especially if the AI is used as a substitute for human oversight.
  • The Sponsor – The pharmaceutical or biotech company funding the trial might bear responsibility for ensuring that all vendors, including AI providers, meet regulatory standards.
  • The AI Vendor – The company that developed the AI system could be liable if the algorithm is proven to be defective or biased.
  • The Human Operator – The individual who reviews and acts on AI-generated queries may also be in the firing line, especially if they fail to exercise due diligence.

Each of these stakeholders has a different perspective on where accountability should lie, and the lack of clear guidance is causing significant concern across the industry.

Regulatory Gaps and Ethical Dilemmas

Regulatory bodies like the FDA and EMA have issued guidance on the use of AI in drug development, but these are often high-level and do not address specific scenarios such as AI-generated queries. This leaves a vacuum that is being filled by contractual agreements and internal policies, which are rarely consistent across organizations.

From an ethical standpoint, the question is even more nuanced. If an AI system makes a mistake, can it be said to have 'intent'? And if not, how do we apply principles of accountability that are rooted in human agency? Some experts argue that the 'human in the loop' approach is essential, but even that does not fully resolve the issue, as human oversight can be superficial if the AI's reasoning is opaque.

The Need for Transparency

One of the key challenges is the 'black box' nature of many AI algorithms. If a query is generated, the reasoning behind it may not be easily explainable, making it difficult to audit or challenge. This lack of transparency not only complicates accountability but also undermines trust in the system.

To address this, there is a growing call for 'explainable AI' in clinical trials. This would require AI systems to provide clear, human-readable justifications for their queries, allowing stakeholders to understand and verify the logic. While this is technically challenging, it is seen as a necessary step toward responsible AI deployment.

Practical Steps Forward

While the regulatory landscape evolves, there are practical steps that CROs, sponsors, and vendors can take to mitigate risk:

  • Establish clear contracts that specify who is responsible for AI outputs and under what circumstances.
  • Implement robust validation processes for AI systems, including testing on historical data to identify potential biases.
  • Maintain human oversight at critical decision points, ensuring that AI is used as a tool, not a replacement for professional judgment.
  • Document everything – keep logs of AI queries, actions taken, and the rationale behind decisions.
  • Engage with regulators early to seek clarity on specific use cases and align with evolving expectations.

These measures can help reduce ambiguity, but they are not a substitute for systemic change. The industry needs a broader conversation about how to integrate AI into clinical research in a way that is both innovative and accountable.

Conclusion: A Shared Responsibility?

As the article in The Clinical Trial Vanguard suggests, the question of accountability for AI-generated queries is not one that can be answered by a single party. Instead, it points to a shared responsibility model, where CROs, sponsors, vendors, and regulators all have a role to play in ensuring that AI is used safely and ethically.

Ultimately, the goal should not be to assign blame, but to create systems that are transparent, reliable, and aligned with the core principles of clinical research. Until then, the question of who is accountable may remain open, but the industry is clearly moving toward a more thoughtful integration of AI—one that balances innovation with responsibility.