SOJO : A Music Therapy Toolbox Powered By Gemini

• 

  Our Logo

Inspiration

SOJO: A toolbox that makes music-therapist-patient interactions easier before, between, and after sessions.

What it does

The SOJO platform is designed with an easy-to-use interface that allows users to input their preferences and feedback. Besides machine learning algorithms, the digital platform is utilized to collect and analyze the user’s feedback concerning their personalized music playlist.

How we built it

On the other hand, the collected data can also be used to train machine learning models to more accurately predict the user’s music preference. This model enables the platform to adapt the music played during therapy sessions, aligning with the user’s psychological and emotional state. Over time, the platform learns the user’s preferences and responses to different music styles and pieces, enabling it to better tailor future sessions to the user’s needs.

Challenges we ran into

Accomplishments that we're proud of

1. We Migrate our platform from Heroku to the Google Cloud app engine

2. we migrate our storage option from Cloudinary to Google Bucket

3. We utilize Google Speech API to generate transcripts for therapy sessions

def transcribe(
        project_id: str = "sojo-api-380801",
        model: str = "gemini",
        audio_file: str = "gs://cloud-samples-data/speech/brooklyn_bridge.raw",
):
    # Instantiates a client
    client = SpeechClient()

    # Reads a file as bytes
    with open(audio_file, "rb") as f:
        content = f.read()

    config = cloud_speech.RecognitionConfig(
        auto_decoding_config=cloud_speech.AutoDetectDecodingConfig(),
        language_codes=["en-US"],
        model=model,
    )

    request = cloud_speech.RecognizeRequest(
        recognizer=f"projects/{project_id}/locations/global/recognizers/_",
        config=config,
        content=content,
    )

    # Transcribes the audio into text
    response = client.recognize(request=request)

    for result in response.results:
        print(f"Transcript: {result.alternatives[0].transcript}")

    return response


class SessionRecord(models.Model):
    start_time = models.DateTimeField(default=None)
    end_time = models.DateTimeField(default=None)
    patient = models.ForeignKey(Patient, on_delete=models.CASCADE, null=True)
    doctor = models.ForeignKey(Doctor, on_delete=models.CASCADE, null=True)
    notes = models.CharField(max_length=500, blank=False)
    transcripts = models.CharField(max_length=500, blank=False)
    voice_recording_url = models.CharField(max_length=500, blank=False)

    def transcript(self):
        return transcribe(audio_file=self.voice_recording_url)

1. We utilize google Gemini API and Document AI to generate summary and prefill the medical forms

What we learned

1. Gemini is a super useful tool

2. It is importance for us to do the right thing for the AI in healthcare, use the best practice and follow the HIPAA guidance and FHIR standards

What's next for SOJO : A Music Therapy Toolbox Powered By Gemini

1. we would use medical grade database and Google Healthcare API

2. start pilot study with clinics and Hospitals

Built With

• bootstrap
• cloudinary
• django
• gemini
• github
• heroku
• postgresql
• redis-to-go