REST2DDP

Inspiration

At OK GROW! most apps we build need to integrate with a legacy REST API of some sort. Meteor of course provides simple functions for making raw HTTP calls but once you've been working with DDP for a while you don't want to go back.

We picture a world where developers demand a DDP API (remember when developers overthrew their SOAP overlords in the Great RESTful API Revolution of 2006?) and we want to make it easier for legacy apps to produce a live updating DDP API.

What it does

You have a service which provides a REST API but the world wants it as a live updating DDP API. With just some simple configuration you can add a DDP API to your site.

http://rest2ddp.meteor.com/

You can create configurations for REST API endpoints, and the system makes a DDP subscription available. You define the following:

• URL for the REST API endpoint
• A JSONPath expression that describes where to find an array in that response
• Some variables that can be substituted at the time of subscription (e.g. ids, url parameters, auth tokens, etc).
• The name of the collection that the data will go into
• How often to poll for updates

How we built it

It's a Meteor app that lets you store descriptions of REST API endpoints (URL, etc). For each one it provides a publication that can be subscribed to by any DDP client.

The core of the app is a Meteor.publish function that connects to a REST API. It loads the data, parses out an array and converts that array into documents that get published to the client.

It then continues to poll the REST endpoint until the subscription is stopped (on a frequency that you define), comparing the new results with the previous and sending just the changes over the DDP connection to the subscribed client.

Meteor packages

accounts-github accounts-ui aldeed:autoform aldeed:collection2 aldeed:simple-schema check http kadira:blaze-layout kadira:flow-router less momentjs:moment mquandalle:jade twbs:bootstrap random reactive-var

Npm packages

deep-diff jsonpath moniker

Challenges we ran into

Efficiency

We used an object diffing library that should be able to efficiently give us just the changes so that we send the minimal changes to the subscribed client. But due to time constraints we're not taking full advantage of the diff library right now and our DDP updates are not as efficient as they should be. There are two issues:

1. Unfortunately we can't tell yet that an object was inserted into the array and the following items just shifted down, currently all items after the inserted one will appear as changes. Very inefficient.
2. Changes should be just the (top-level) field that changed. Right now we send the whole object (all fields). The diff library can tell us exactly which fields changed but we're not using it.

Usability

It's challenging to design a UI that's self-explanatory enough for a casual user or hackathon judge to grasp immediately. We didn't have time to iterate on the UI.

Accomplishments that we're proud of

1. We built something useful in 24 hours!
2. Variable substitution allows creating publications that are general and can be used by a variety of clients without hardcoding authentication, etc.
3. Live previewing of the API call results is pretty cool.

What we learned

• There is a lot of variety and very little standard in the ways that REST APIs do authentication.
• 24 hours is a short time but with Meteor we can get a lot done :-)

What's next for REST2DDP

Improvements to the app

• More robust error handling and general bug fixing
• Support XML-based REST APIs
• Allow setting custom HTTP headers and auth
• Allow specifying a field from the response to use as a unique id
• Extract the core functionality into a package that can be used by Meteor apps without needing to use our service.
• Improve the diffing and fix the inefficient publishing of changes
• UI improvements, e.g. grouping publications together, changing the route for each saved configuration.
• Allow keeping some publications private
• Clean up some nasty parts of the code
• Support APIs that return only a single item (currently we expect an array)
• Scaling & performance
• Reduce load on REST APIs: ensure that multiple subscriptions to the exact same API with the same variables share the same HTTP requests to the REST API.

Create a package

We would like to see Meteor come to mind immediately for developers who are looking to put a new UI in front of a legacy app with a RESTful API, rather than being considered for new app development only. Front-end only frameworks tend to grow in popularity much faster than full-stack frameworks because there are so many developers with legacy apps that want to update the UI without rebuilding the entire app.

A hosted service is good for an API provider who wants to provide a DDP API without doing more work, it can be thought of like a proxy. But a developer who wants to consume a REST API might not want to go through a hosted proxy and might want to be able to subscribe to REST API's directly within their app.

So we would like to extract the core functionality into a package so that any Meteor app can easily subscribe to a REST API and receive live updates over DDP.

Built With

• meteor.js