Fleximinder

Inspiration

One of the things that gets difficult with age is remembering stuff - significantly affecting one's self-confidence and state of being organized. To alleviate this, of course, people resort to taking notes; an activity that becomes increasingly important and more frequent as people get older. A hassle-free note taking and retrieval experience should help make the lives of the aging better.

What it does

Fleximinder attempts to create a novel note taking experience. It makes it quick, handsfree and, as you can see in the details below, provides rich and precise retrieval experience based on natural language. It is a sophisticated keeper of users' todo's and forgettables that provides a natural way to say and to ask back for them (in English). A typical interaction goes like

📝 My hotel room number is 432

Alexa: Noted. Your hotel room number is 432 [This is to confirm correct interpretation]

❔ What is my hotel room number?

Alexa: Your hotel room number is 432

How It Works

Input from Alexa is sent to Parsey McParseface, a natural language parser based on SyntaxNet. The parse result is analyzed to extract meaning in some form. The representation of the original input is then stored in a database (DynamoDb table) or, if it were a question, it will result in a query against same database.

How I built it

Fleximinder is based on Google's Parsey McParseface (SyntaxNet) for natural language understanding (NLU), Docker containers, AWS Fargate and load balancers for container management, Amazon Alexa and Alexa.NET.

Challenges I ran into

Overall, attempting to extract, store and query meaning out of natural language is much easier said than done.

Accomplishments that I'm proud of

Quite a toddler the product is, but it is performing to my satisfaction.

What's Next for Fleximinder

Allow more relaxed expressions.
Update to newer Syntaxnet engines if any for improvement
Companion app or website to manage longer lists
Support time zones in dates. Currently all dates and time are processed in UTC.
Use APL and add feature to select an item on display device and say "Delete this note".
Export notes to email
Batch input, at least a few statements at a time.
More natural answers e.g.
Implied in its name but conspicuously missing in this skill is the means to save or execute reminders. In the current architecture, these are just commands the NLU unit should interpret (just like it does when doing "delete this note"). When eventually implemented, reminder setting and querying will enjoy the same level of flexibility that notes and todo's enjoy currently.
Simple yes/no Assuming you already noted I have emailed John about the meeting asking Have I emailed John about the meeting? should be answered as Yes, you have instead of You have emailed John about the meeting)
Extract relevant portion of answers Having already noted down I have emailed Mary and John about the meeting, saying Tell me about John should answer You have emailed John about the meeting, along with other facts about John, instead of the whole You have emailed Mary and John about the meeting.
Make it take anything thrown at it, or stand up against it (like it does with invalid dates).

Advanced Details

Notes should be both grammatically correct and simple. While this might be somehow limiting, it makes it precise during retrieval.

📝 Open Fleximinder.

FM What's on your mind?

📝 My driving license expires on June 7

FM Noted. My driving license expires on June 7.

📝 John's insurance expires on June 8

FM ...

📝 John has borrowed my battery yesterday

FM ...

📝 Sophia's insurance expires on July 7

FM ...

📝 What expires in June? (Here you can't say 'What is expiring...')

FM There are 2 records. My driving license...John's insurance...

📝 What expires? (So you must ask a little too directly/literally like this)

FM There are 3 records...

📝 Tell me about John.

FM John has borrowed... (One result only. "John's insurance" is not "John")

📝 Tell me about insurance.

FM There are 2 records...

📝 She bought two pressure cookers. ("she" is not the Sophia mentioned above, not yet)

FM ...

📝 How many cookers did she buy?

FM She bought...

Retrieving Notes

[1] Just ask e.g. given She bought two pressure cookers

She bought five slow cookers

She bought three cats

You can ask questions like:

How many cats did she buy?

How many slow cookers did she buy?

How many cookers did she buy?

How many bottles did she buy?

How many did she buy? (Does general search for any of her purchases mentioning numbers)

[2] List command.

List my notes (from the last 5 minutes, yesterday, etc)

List all notes loads all notes. It is useful if you want to issue a subsequent delete these notes to clear everything.

[3] Tell me command. Tell me about John. This will get everything mentioning John.

Deleting Notes

Immediately after a note is saved, say Delete (this) note
After a certain number of notes have been displayed, say Delete these notes. A list gets displayed by requests such as Tell me about John and List my notes from yesterday.
Say List all notes and then Delete these notes to delete everything.

About Dates

All dates should be in Coordinated Universal Time or UTC. To get the current time in UTC, ask "what is the time?" or "what time is it?". Time zones are ignored or may cause errors in some cases.

If a date expression is not valid, you will be told so. For example, as of the year 2022, "last year 2019" will fail.

You can enjoy robust date checks as long as your note contains valid date. If, on a Monday, you say

Parcel arrives on Tuesday

then you're protected from duplicating it in the following way

Parcel arrives tomorrow

You can specify exact date if you wish to be precise, as in Parcel arrives tomorrow Tuesday twelfth of February 2019

If a date is structurally invalid (or correctly formed but not make sense e.g. "next morning yesterday"), Fleximinder explicitly reports it as invalid and will not save notes containing them.