Object storage has emerged as an alternative to traditional File System storage for large quantities of unstructured data. Whereas traditional hierarchical File System structures can get cumbersome as they grow very large, Object storage brings a “flat” structure with equal access to all objects held, making it eminently suitable for large volumes (i.e. billions or trillions of objects) of unstructured data. Further, Object storage objects in general support a richer set of custom metadata than the files and folders of a traditional File System can. This potentially makes the object data of Object storage better-suited for analytic use cases.
However, Object storage lacks the locking mechanism that enables concurrency that File System-based approaches support. In addition, Object storage tends to be the worst performing of all storage modes in part because of the heavier metadata overheads (although implementations such as CORTX Object storage are changing this!). This tends to make raw Object storage not well-suited to more time-critical operations such as transactional processes.
But perhaps the biggest problem with Object storage affecting Enterprise mass adoption is that the vast majority of existing Enterprise applications are File System centric and cannot work with Object storage without significant modifications. File systems are a key part of traditional on premise Enterprise computing systems, and remain the most common way that applications and operating systems store data. In contrast, Object storage was born in the cloud and provides an abstraction model for data that supports a broad range of customizable metadata. However, Object storage is not directly accessible to conventional applications, creating a barrier to entry into the Enterprise.
What if there was a way to unify the best of File and Object storage models, allowing all the data of the most important Enterprise business applications to be stored in objects automatically and seamlessly?
What it does
CortxNTFS is a Windows NTFS compatible “virtual drive” technology that unifies traditional File with modern Object storage seamlessly, on premise, in the cloud or a hybrid of both. CortxNTFS is a traditional kernel-mode File System driver that is compatible with the low-level system API calls of the World’s #1 client Operating System – Windows. It is provides block-level file i/o, not to local disk, but to objects stored in a CORTX Object storage instance.
When the CortxNTFS software is installed, Windows sees a “virtual” drive backed by one or more CORTX Object storage instances, in a similar way to the way it sees a USB thumb drive as a connected storage drive. This drive can be up to 8.0 EBs (8 Million Terabytes) in size and can be dynamically grown as needed by adding more CORTX instances. The CortNTFS vDrive is NTFS security compatible which means it automatically supports the vast majority of Windows file-based applications.
CortxNTFS unifies File and Object storage by allowing the Enterprise to have all of their applications “just work” with CORTX Object storage without the need to either re-factor them, or use slow gateways that translate between file and object.
How we built it
The technology used to build CortxNTFS came out of the larger World Computer project I have been working on for several years (see WorldComputer.org) which features a fully operational NTFS vDrive that is backed by Microsoft’s Azure Blob Storage service. For this hackathon I focused on refactoring that NTFS vDrive technology into a standalone piece of software and then added support for the CORTX S3 layer. To complete the effort I added a software installer for CortxNTFS as well. The tech stack is C++ and C#.
Challenges we ran into
Most of my problems revolved around CloudShare as I was using the CORTX instance running in CloudShare provided to me by Seagate for this hackathon. The main issue was getting CloudShare instance to remain running as it tended to “hibernate” when not in use (i.e.; not receiving traffic for a period of time). And once hibernated, frequently the CORTX S3 Instance would not resume reliably. However, a shout out to Patrick Hession of Seagate technical support who went above and beyond late into the evening and on his weekend to help get me sorted out. You rock Patrick!!
The other issue I found was that under extreme load (specifically, heavy parallel i/o that certain applications I tested generate) the CORTX instance sometimes behaved in an irregular way (pauses in response, or outright i/o failures) as compared to other S3 instances such as AWS.
Accomplishments that we're proud of
Seeing all of the amazing advanced use cases that I tried working with CortxNTFS! (Check out the video that accompanies this submission for demos!).
What we learned
File and Object can be unified successfully and CORTX is the perfect Object storage system to support this unification as it is specifically designed to run on premise where the vast majority of traditional File System i/o takes place.
What's next for CortxNTFS
Hopefully work with Seagate to stress test CortxNTFS against popular Enterprise applications.