landing page
dashboard 1

Project Lullaby

Team: Erik, Matt, Joey, Mike

Sleep Reimagined.

What if parents and babies could get a better night's sleep? With Lullaby, we designed a web application that helps babies and parents get a better night's sleep by notifying them when their baby is crying and providing them with data to optimize their sleep strategies.

Inspiration

Inspired by Google's human voice library https://developers.google.com/assistant/tools/sound-library/human-voices and their voice API, as well as Mike's experience with his own kids.

Primary clients:

Parents/caregivers
Babies - newborn & infants

Main Tasks:

Detect Cries
1. Capture audio
2. Process captured audio
3. Notify guardian
Handle Responses
1. Determine gaurdian for response
2. Address reasons for wake up

How it Works:

Devices:

1.A: An IoT or Home Device for baby's room
1.B: Caregiver's IoT / Home device / mobile
Cloud

Implementation details

IOT / Home Device

Trained Machine Learning to recognize / distinguish infant crying sounds using Keras and Tensorflow
- Record & track sounds coming from baby's room
- Process audio
- Generate label (string ? with label identifying sound)
- Send label to appropriate devices
  - Back to device in baby room
    - change white noise (frequency / volume)
  - Alert to caregiver(s)
  - Record event in database with details ()
    - DETAILS:
      - length of cry
      - type of sound (other baby generated sounds / movement learnable)
      - which caregiver alerted
Caregiver device:
- Alert - baby is crying
- Trigger recording
  - length of time before they wake to respond
  - Time of wake
  - time to sooth baby / get back to sleep
  - number of times woken up
    - that evening
    - aggregatable
- NOTE: could be audio and video

TODO

device use cycle(s) - arrows that show pathway of "objects" & "actions"
Notes / quotes on research / importance of sleep
DataViz for Dashboard

Website

https://lullabyzzz-20191013090820.azurewebsites.net/

Documentation

https://hackmd.io/MPnwtfkkT0aCi30UjaEahQ?view

Getting Started

In the root directory of the project...

Install node modules yarn install or npm install.
Start development server yarn start or npm start.

In Windows:

Install ffmpeg at 'https://www.ffmpeg.org/'
Change Path to C:/file/location/bin/ffmpeg.exe

File Structure

The front-end is based on create-react-app.

The back-end is based on Express Generator.

The front-end is served on http://localhost:3000/ and the back-end on http://localhost:3001/.

.
├── audio-classification/ - Express server that provides API routes and serves front-end
│ ├── clean/ - Handles all interactions with the cosmos database
│ ├── models/
│ ├── oggfiles/ - Adds middleware to the express server
│ ├── pickles/
│ ├── wavfiles/ - Handles API calls for routes
│ ├── cries.csv - input file for training
│ ├── demo.csv - input file for demo
│ ├── prediction.csv - results of prediction.py
│ ├── cfg.py - configuration options for the program
│ ├── model.py - creates a trained ML model to predict sound
│ ├── prediction.py - classifies sound files using the trained model
│ └── requirements.txt - modules required for running the program
├── server/ - Express server that provides API routes and serves front-end
│ ├── mongo/ - Handles all interactions with the cosmos database
│ ├── routes/ - Handles API calls for routes
│ ├── app.js - Adds middleware to the express server
│ ├── sampleData.js - Contains all sample text data for generate pages
│ ├── constants.js - Defines the constants for the endpoints and port
│ └── server.js - Configures Port and HTTP Server
├── src - React front-end
│ ├── components - React components for each page
│ ├── App.jsx - React routing
│ └── index.jsx - React root component
├── .env - API Keys
└── README.md

Additional Documentation

React - https://reactjs.org/
React Router - https://reacttraining.com/react-router/
Bootstrap CSS - https://getbootstrap.com/
Express - https://expressjs.com/
Mongo/Mongoose - https://mongoosejs.com/docs/guide.html
Cosmos DB - https://docs.microsoft.com/en-us/azure/cosmos-db/mongodb-mongoose

Accomplishments that we're proud of

Coming together as group - none of us knew each other before yesterday.
Trained a Tensorflow model to recognize and distinguish baby cries from other baby noises.