AI Product Design

How might AI-driven interventions support and sustain positive behavioral changes in patients following psychedelic-assisted therapy?

This case study contains highlights of my capstone project for the MIT certification in Designing and Building AI Products and Services.

Drawn to innovation that improves mental health and well-being, I decided to explore the intersection of artificial intelligence and psychedelics. Several research lab projects and companies inspired my work, such as Behavidence, because of the work they are doing to make a difference for mental healthcare practitioners and their clients.

Challenge

Psychedelic-assisted therapy is gaining recognition as a powerful tool for mental health treatment, offering profound insights and emotional breakthroughs. However, the integration process—where patients translate these insights into goals to achieve lasting behavioral changes—presents a unique challenge. In the critical weeks following a therapy session, interactions between patients and practitioners are often infrequent, leading to a loss of momentum and diminished motivation to act on the therapeutic insights gained.

To address this issue, I propose the development of an AI-driven application designed to support patients throughout their integration process. This application aims to bridge the momentum gap by providing an accountability framework that is both user-friendly for patients and easily monitored by practitioners. By leveraging AI, we can enhance the integration process, ensuring that the transformative potential of psychedelic therapy is fully realized in patients' lives.

Project Background

Need identification: practitioners cannot consistently monitor client progress between office visits beyond self-administered survey data that clients might complete. There is a need for more "wrap-around" support for psychedelic-assisted therapy during the integration phase. For example, there might be 6 treatment sessions spanning multiple weeks as in the case of ketamine-assisted therapy, compared to the integration phase for medicines like MDMA and Psilocybin which extends from a few weeks to several months. Without technology-based interventions in the process of the integration phase, practitioners do not have real-time insight into the behavioral change patients are experiencing.

Hypothesis: To best support the client's integration process using AI, combine Machine Learning (ML) and digital phenotyping to gather and assess data from a user’s mobile device and present information back in the form of recommendations in an app and/or platform so that both client and practitioner can monitor treatment progression.

Value proposition: Tailoring the periods between formal psychedelic integration sessions so that clients have a strong sense of agency in achieving their goals. Real-time assessment of improvement in overall mental health and progress toward the desired behavioral change can be monitored remotely by the practitioner so the treatment plan can be adjusted accordingly.

Prototype

Click here for a larger view of the Customer Journey Map.

AI Algorithm: CatBoost

Since the proposed solution uses digital phenotyping to track mood and mental state through social media usage on a user’s smartphone, I have explored a multifaceted approach to sentiment analysis to account for the data extraction on a personal digital device to measure aspects of an individual’s behavior and mental health. Based on my research of medical journal articles, the most appropriate AI algorithm to use for psychological and behavioral disorders, such as depression and anxiety, is category boosting (CatBoost): “All five machine learning models can identify the occurrence of depression in the NHANES data set through social demographics, lifestyle, laboratory data and other data of middle-aged and elderly people, and among five models, the CatBoost model performed best” (Zhang et al.). I have broken down my rationale for using CatBoost and my assessment of appropriate AI algorithms by function below:

• Handling Categorical Data: Social media data often include categorical features (like types of posts, reactions, etc.). CatBoost can effectively process these without extensive pre-processing.

• High Dimensionality: CatBoost can efficiently manage high-dimensional data, which is typical when integrating various types of social media content and interaction metrics.

• Complex Feature Interactions: CatBoost automatically handles complex interactions between features, which is crucial in understanding the multifaceted nature of social media behavior.

• Large Dataset Scalability: It is optimized for performance on large datasets, making it suitable for the voluminous data generated by social media platforms.

• Robust to Overfitting: CatBoost has built-in mechanisms to prevent overfitting, which is crucial when dealing with complex and noisy social media data.

AI ALGORITHM: ARIMA

To analyze interaction patterns, such as frequency of posts, timing, and type of content shared or interacted with, a model like ARIMA (AutoRegressive Integrated Moving Average) or LSTM (Long Short-Term Memory) would be best to track changes in social media activity levels, which can be indicative of mood changes.

Training

I propose using transfer learning to overcome the limitation of having insufficient training samples. Transfer learning is effective because it leverages the generic learning from the model's initial training and applies it to a more focused domain. Using a pre-trained model allows us to bypass the need for foundational learning so the model can focus on the more specialized patterns found in behavioral health data. My proposal outlines the need for pre-trained models, fine-during, and review.

Pre-trained Models:

Start with pre-trained models. The initial layers of a neural network model learn to recognize basic patterns, which are often universal across different types of data. In the realm of behavioral health data analysis, models pre-trained on large datasets from related fields, such as general health records, psychiatric evaluations, or even broader text and data analysis tasks, can serve as a solid foundation. These models have already learned a rich set of features that can apply to behavioral health, such as patterns of language used in clinical notes, relationships between symptoms and diagnoses, or trends in patient progress over time. For my capstone project, I explored pre-trained models based on the BERT architecture (Bidirectional Encoder Representations from Transformers) like ClinicalBERT. ClinicalBERT has been further trained on clinical text like electronic health records and medical publications and therefore is a better fit for behavioral health solutions.

Fine-tuning:

During the process of fine-tuning, we would adjust the pre-trained ClinicalBERT model to make it more relevant to specific data analysis tasks and workflows. I propose retraining some of the layers of the model with a proprietary behavioral health dataset to specialize its understanding and predictions to the nuances of behavioral health. A proprietary dataset would include information like notes from therapy sessions and patient journal entries. My argument for fine-tuning is that this process requires relatively small datasets to achieve significant improvements in model performance; therefore, it is highly suitable for situations where extensive labeled data is scarce.

Review, testing, and concept drift mitigation:

Once fine-tuning is complete, then we must review and test the fine-tuning comprehensively on an ongoing basis. For example, concept drift will affect this model because the distribution of data will change as new diagnosis criteria emerge, treatment modalities change, and patient populations evolve. Having concept drift detection in place will be critical in monitoring the model’s performance. Combining transfer learning with online learning algorithms allows for real-time adjustments as new data patterns and distributions emerge. Online learning is effective at handling concept drift; by continuously updating the model with new data, online learning helps maintain the model's performance even when the underlying data distribution shifts.

Data Gathering

Due to the sensitive and complex nature of the types of data involved, this phase of thinking through considerations for data gathering, infrastructure, storage, ETL (Extract, Transform, Load), and data cleaning is critical.

• Mapping out the data privacy regulations in the geographies served by the product, such as HIPAA and GDPR, will help ensure data collection efforts are compliant with ethical guidelines and privacy regulations.

• Since this data can contain unique behavioral patterns and physiological responses, it needs to be anonymized to remove any identifiers that could link the data back to specific individuals. Anmol Arora and Ananya Arora (2022) discuss the potential benefits of using GANs in a medical context, highlighting that “it is increasingly possible to fully anonymise datasets to the point where no data point is traceable to any real individual. This can be used to expand and balance datasets as well as to replace the use of real patient data in certain contexts.”

• Examples of information collected for training GANs (Generative Adversarial Networks) include recordings during the therapy sessions and behavioral data from wearable devices to train GANs in generating synthetic data that mimics complex patient states.

• Examples of information collected for training NLP (Natural Language Processing) models include textual data from therapy session transcripts, patient journals, and digital communication (from mobile devices, specifically social media apps). This data is crucial for training NLP models to understand and analyze patient sentiment, progress, and response to therapy. The application of NLPs is growing in the mental health space, as discussed by Malgaroli, Hull, Zech, and Althoff (2023) in their systematic review on the use of natural language processing for mental health interventions: “[Mental Health Interventions] MHIs rely on linguistic exchanges and so are well suited for NLP analysis that can specify aspects of the interaction at utterance-level detail for extremely large numbers of individuals, a feat previously impossible.”

Considerations

Decision Tracing:

Decision tracing involves understanding how the model makes its decisions, which can help identify during review and testing why biases are occurring. My recommendation is to employ model interpretability tools and techniques, such as LIME (Local Interpretable Model-agnostic Explanations) or SHAP (SHapley Additive exPlanations), to trace back the decision-making process. If there is gender bias detected in the model for example, then these techniques can help identify if and how gender-related features are influencing outcomes. The model or its inputs can be adjusted based on these insights to prevent gender from being a decision factor.

Bias Mitigation:

Bias mitigation involves taking steps to actively reduce and eliminate the biases identified in the model's decisions. Techniques such as re-sampling the training data to balance gender representation, using synthetic data to augment underrepresented groups, or applying algorithmic fairness interventions (e.g., equal opportunity, equalized odds) can help mitigate bias.

Integrating GANs for an Imbalanced Dataset:

GANs are a creative solution to the problem of imbalanced datasets because they generate synthetic samples of the minority class (like attack instances). GANs can be used to augment the minority class by generating additional synthetic attack samples. This will increase the size of the attack dataset without losing information from the majority class (non-attack samples). By training on a more balanced dataset that includes synthetic attack instances, machine learning models can learn a more accurate representation of attack behaviors. This leads to improved detection rates (performance) of actual attacks and reduces bias toward the majority class. Synthetic samples generated by GANs can cover a broader range of potential attack scenarios, including evolving attack types that are not represented in the original dataset. This helps models generalize better to new and unseen attack methods (enhanced generalization). I acknowledge that implementing and training GANs can be resource-intensive because implementation requires careful tuning and monitoring to ensure that the generated samples are both diverse and representative, but highlight here as a method for mitigating bias.

Professional Oversight:

AI is not a panacea. Keeping humans in the loop, especially mental healthcare professionals, enhances the reliability and ethical consideration of the AI system and also ensures that decisions are made with a level of understanding and compassion that AI alone cannot provide. Careful consideration to the language and wording of recommendations served through the app will need to ensure that final decisions default the professional. Continuous input from mental health professionals on the AI's performance can guide adjustments and improvements, ensuring the system remains aligned with clinical needs and standards. Having a strong Customer Advisory Board consisting of participating professionals will support this continuous feedback and improvement loop. Ideally, tap into existing clinical science review boards or committees to review and oversee the AI system's ethical use, including considerations around privacy, consent, and bias.

References

Arora, A., & Arora, A. (2022). Generative adversarial networks and synthetic patient data: current challenges and future perspectives. Future Healthcare Journal, 9(2), 190–193. https://doi.org/10.7861/fhj.2022-0013

Malgaroli, M., Hull, T. D., Zech, J. M., & Althoff, T. (2023). Natural language processing for mental health interventions: A systematic review and research framework. Translational Psychiatry, 13, Article 309. https://doi.org/10.1038/s41398-023-02592-2

 Zhang, Chenyang, et al. "Using CatBoost Algorithm to Identify Middle-Aged and Elderly Depression, National Health and Nutrition Examination Survey 2011–2018." Psychiatry Research, vol. 306, Dec. 2021, Elsevier, p. 114261. Science Direct. https://www.sciencedirect.com/science/article/abs/pii/S0165178121005564

Definitions

Practitioners: Licensed professionals who provide therapeutic services to individuals, groups, or communities. These practitioners may include psychologists, psychotherapists, psychoanalysts, counselors, social workers, and other trained specialists who apply evidence-based practices and therapeutic techniques to promote mental health and well-being.

Background: Psychedelic-assisted therapy 

Psychedelic-assisted therapy is a therapeutic approach that combines traditional psychotherapy with the use of psychedelic substances to enhance the therapeutic process and facilitate deeper insights, emotional breakthroughs, and psychological healing. It is primarily used for treating mental health conditions such as depression, anxiety, post-traumatic stress disorder (PTSD), and addiction.

The most commonly used medicines in psychedelic-assisted therapy include psilocybin (“magic mushrooms”), MDMA, LSD (Lysergic acid diethylamide), and ketamine. Although not a classic psychedelic, ketamine is used in a psychotherapeutic setting to treat depression, including treatment-resistant depression.

Legal use of psychedelic medicines varies widely and the legal landscape is rapidly evolving. Supervised psilocybin use is legal in Oregon. Ketamine is widely used in treatment centres across the US and Canada. Experts anticipate that the Food and Drug Administration will approve MDMA for PTSD treatment by mid 2024, which would make the drug available for PTSD diagnoses through doctors across the US.