Inspiration

I have always wanted to create assistive natural language agent that recognizes complex grammar and precise meaning. That seemed achievable with help of Alexa and I set out to create a skill that can keep notes intelligently.

What it does

Fact Bin allows you to store notes and retrieve them using natural language. However, your facts (notes) should be both grammatically correct and simple. While this might be really limiting, it makes it precise during retrieval. Check the following conversation for instance.

You: Open Fact Bin.

Fact Bin: What's on your mind?

You: My driving license expires on June 7 (Alexa-enabled devices with a screen will show the recognized date)

Fact Bin: Noted. My driving license expires on June 7.

You: John's insurance expires on June 8

Fact Bin: ...

You: John has borrowed my battery yesterday

Fact Bin: ...

You: Sophia's insurance expires on July 7

Fact Bin: ...

You: What expires in June?

Fact Bin: There are 2 records. My driving license...John's insurance...

You: What expires? (Sorry but you must ask a little too directly/literally like this)

Fact Bin: There are 3 records...

You: Tell me about John.

Fact Bin: John has borrowed... (One result only. "John's insurance" is not "John")

You: Tell me about insurance.

Fact Bin: There are 2 records...

You: She bought two pressure cookers. (Hmm..no, "she" is not the Sophia mentioned above, not yet)

Fact Bin: ...

You: How many cookers did she buy?

Fact Bin: She bought...

Options to retrieve notes

  1. Just ask e.g. given She bought two pressure cookers

She bought five slow cookers

She bought three cats

You can ask questions like:

How many cats did she buy?

How many slow cookers did she buy?

How many cookers did she buy?

How many bottles did she buy?

How many did she buy? (Does general search for any of her purchases mentioning numbers)

  1. List command.

List my notes (from the last 5 minutes, yesterday, etc)

List all notes loads all notes. It is useful if you want to issue a subsequent delete notes to clear everything.

  1. Tell me command. Tell me about John. This command is a good trick to alleviate the need to be literal.

Options to delete notes

  1. Immediately after a note is saved, say Delete (this) note
  2. After a certain number of notes have been displayed, say Delete these notes. A list gets displayed by requests such as Tell me about John and List my notes from yesterday.
  3. Say List all notes and then Delete these notes to delete everything.

Multi-modal Experience

If you have Alexa-enabled devices with a screen, you will get more details on your display in most cases. This is especially useful for questions resulting in large number of matches. You probably would rather scroll through a long list of notes instead of having it read one by one.

Also shown is interpreted date ranges, if any. For example, what "last month" translates into is clearly shown as something like "01/01/2019 12:00 AM".

About Dates

All dates should be in Coordinated Universal Time or UTC. To get the current time in UTC, ask "what is the time?" or "what time is it?". Time zones are ignored or may cause errors in some cases.

If a date expression is not valid, you will be told so. For example, as of the year 2019, "last year 2015" will fail.

You can enjoy robust date checks as long as your note contains valid date. If, on a Monday, you say

Parcel arrives on Tuesday

then you're protected from duplicating it in the following way

Parcel arrives tomorrow

You can specify exact date if you wish to be precise, as in Parcel arrives tomorrow Tuesday twelfth of February 2019

If recognized date is empty on your display device, then it means your date is structurally invalid (in contract to correctly formed date that still does not make sense e.g. "next morning yesterday". Fact Bin explicitely reports such dates as invalid and will not save notes containing them.)

How I built it

Fact Bin was created using Google's Parsey McParseface, AWS, Amazon Alexa and Alexa.NET.

Challenges I ran into

Overall, attempting to extract, store and query meaning out of natural language is much easier said than done. Creating .NET models for supporting APL was also a challenge.

Accomplishments that I'm proud of

Quite a toddler the product is, but it is performing to my satisfaction.

What's next for Fact Bin

  • Support time zones in dates. Currently all dates and time are processed in UTC.
  • Add feature to select an item on display device and say "Delete this note".
  • Export notes to email
  • Batch input, at least a few statements at a time.
  • More natural answers e.g.

Simple yes/no Assuming you already noted I have emailed John about the meeting asking Have I emailed John about the meeting? should be answered as Yes, you have instead of You have emailed John about the meeting)

Extract relevant portion of answers Having already noted down I have emailed Mary and John about the meeting, saying Tell me about John should answer You have emailed John about the meeting, along with other facts about John, instead of the whole You have emailed Mary and John about the meeting.

  • Make it take anything thrown at it, or stand up against it (like it does with invalid dates).

How to try it now

It seems APL-based skills take much longer to certify and Fact Bin got no special treatment. Actually, I wish the skill was live to link to the published description so as to make this project's description shorter, but I have duplicated here anyway. But, while waiting for the long certification cycles, I came up with another idea of a skill that can use the same engine that was built for Fact Bin. The skill, Talking Machine (http://www.amazon.com/dp/B07NHLFKPW), was certified more easily and you can check it out in the meantime to get a feel of the upcoming skill. It offers similar conversational interface but Talking Machine is a game by design and does not promise long term data storage.

Built With

Share this project:
×

Updates