{"id":677,"date":"2012-03-21T10:49:29","date_gmt":"2012-03-21T01:19:29","guid":{"rendered":"http:\/\/www.clearchain.com\/blog\/?p=677"},"modified":"2012-03-21T10:54:44","modified_gmt":"2012-03-21T01:24:44","slug":"the-intransit-filesystem","status":"publish","type":"post","link":"https:\/\/www.clearchain.com\/blog\/posts\/the-intransit-filesystem","title":{"rendered":"The InTransit Filesystem"},"content":{"rendered":"

\"\"<\/a><\/p>\n

A Long time ago I was watching a story about data and one fact that rang in my mind was how much data was in transit at anyone point in time. A classic way to think about this is a transatlantic fibre cable. Even at the speed of light, the amount of data that is bouncing in that cable grows as the distance increases. It occurred to me you could actually use the in transit bandwidth as a filesystem.<\/p>\n

There’s a number of reasons you might want to do this?<\/p>\n

    \n
  1. A really secure filesystem\u00a0 – since you have to continually retransmit the data, you simply unplug your PC and the data disappears<\/li>\n
  2. A common storage system that multiple people can use<\/li>\n
  3. A Storage system that grows as the amount of diverse paths increases<\/li>\n<\/ol>\n

     <\/p>\n

    Below are some notes I made on a recent international flight.. at the time I considered myself as the in transit filesystem!!<\/p>\n

    The filesystem to this day has not been implemented though the thought about finishing the design and building it still lingers in some parts of my brain.<\/p>\n

    —-<\/p>\n

     <\/p>\n

    The In Transit Filesystem<\/p>\n

    Concept: A file system that exists in the propagation delay of a transit media<\/p>\n

    Examples: There exists many international fibre cables. These cables can
    \ntransmit large amounts of data in a relatively short amount of time. If a cable
    \nis capable of 1Tb\/s and there is a latency of 1ms to the other end of the cable
    \nthen there can be approximately 1Tb\/0.001 bits of data in transit at any one
    \ntime. The aim of the intransit filesystem is to store data in this transmission
    \nspace. Constant retransmission of data maintains the live filesystem. If the
    \nnetwork goes down or the data is not retransmitted the filesystem goes down\/is
    \nlost.<\/p>\n

    Why: Reason behind the In Transit Filesystem<\/p>\n

    The In Transit filesystem does away with a physical storage medium attached
    \nto any one machine. Instead the storage medium is simply propagation delay.
    \nThis sort of storage has a number of applications. These include security
    \nrelated keys which exist with a limited time. Ie consider the concept of
    \nemail that decays. The keys to the email exist in the InTransit Filesystem
    \nThe email client ask for the keys and if available decodes the message. The
    \nsender is the pingback for the filesystem hence the sender decides if they
    \nretransmit the key or not. If they don’t the email is now undecodable (easily
    \nanyway). The filesystem can also be used to establish common tables. Consider a
    \nrouter that needs to build mac tables. Instead of sending a broadcast arp
    \nrequest to find a ip for a machine, it can query the in transit filesystem, this
    \nnot only allows quicker recovery of the ip but also reduces traffic.<\/p>\n

    Objective: The aim of this brief is to asses the requirements needed to not only
    \nbuild the In Transit Filesystem but also to make it resilent.<\/p>\n

    Problems & Solutions:<\/p>\n

    There is a number of issues when building a filesystem of this type. These are
    \nlisted below and discussed afterwards with possible solutions.<\/p>\n

    \to Calculating the Filesystem Size
    \n\to Latency to packets\/files in read\/writing
    \n\to Dealing with packet drop
    \n\to Retransmission
    \n\to Error checking and redundancy
    \n\to transport medium<\/p>\n

    The problems discussed:<\/p>\n

    o Calculating filesystem size<\/p>\n

    With the storage medium live the filesystem size is not only hard to calculate
    \nbut may be continually changing. The aim is to be able to guarantee a minimum
    \nfile system size so data can be recovered.<\/p>\n

    A ongoing calculation supports this by calculating packet latency over time. A
    \nsimple average gives an upper limit to the available data space. Of course
    \nadding another route gives the filesystem increased space. For advanced
    \nfilesystem calculations packet loss including RED and similar algorithms, window
    \nsizing and other medium fallbacks must be considered. These algorithms are not
    \nyet determined.<\/p>\n

    o Latency to packets\/files in read\/writing<\/p>\n

    The speed of the filesystem is limited to at most 2*the propagation delay of the
    \nmedium link. This delay can often be up to 10 seconds ( on large long haul links
    \nwith slow transmission speeds or congestion. This delay is unacceptable. To
    \nincrease speed local caching can be applied though really converts the
    \nfilesystem from an in transit system to an in-memory system. Delays can be also
    \nreduced by secondary links and forward propagation, ie splitting up the blocks
    \nso a file is transmitted in a fragmented with multiple packets containing the
    \nsame part of a file. Of cause this reduces the available size of the filesystem.<\/p>\n

    o Dealing with Packet Drop<\/p>\n

    Packet drop in the in Transit Filesystem is not just bad but devastating. The
    \npacket without some sort of redundancy will cause filesystem corruptions. To deal
    \nwith this there has to be redundancy. <\/p>\n

    o Error checking and redundancy<\/p>\n

    Whilst initially the filesystem will be implemented with traditional algorithms
    \nto support redundancy, ie md5, forward, backward crc’s, etc long term the use of
    \nsuch files are not scalable. Data is growing at such a large rate. Much which
    \ndoesn’t need to be archived but is anyway. We take an approach that is used in
    \nthe zfs filesystem, that being block deduping. Ie files with a block of data that
    \nis the same need only store that block once. <\/p>\n

    We also introduce a new concept that the current modern age of filesystems
    \ndoesn’t support. That being guaranteed reliability. Whilst the computer age has
    \nbrought us digital data, the laying to store that data relies on consistency at
    \nthe filesystem level. Ie A file is always 100% intact. If any bits of that file
    \nare not intact bad things happen. This places lots of constraints on the
    \nfilesystem in regard to the management of data. Whilst I don’t believe
    \nfilesystems can’t have reliability, I believe that applications can perform some
    \nredundancies of their own formats. They then ask the filesystem for a guarantee
    \nof a level of consistency to a file. For example: Modern day application
    \nrequests the creation of a file with a guaranteed consistency of 100%.
    \nApplication X (which say stores a lossy image format) requests block x of file
    \nto be 100% guaranteed (ie a file header) then places a 70% guarantee request on
    \nall other blocks (image data). Hence if data is missing of the filesystem
    \nbegins to fill up the filesystem has the option of dropping packets to
    \naccommodate expansion.<\/p>\n","protected":false},"excerpt":{"rendered":"

    A Long time ago I was watching a story about data and one fact that rang in my mind was how much data was in transit at anyone point in time. A classic way to think about this is a transatlantic fibre cable. Even at the speed of light, the amount of data that is [..more..]<\/font><\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[34],"tags":[416,412,415,413,414,82],"class_list":["post-677","post","type-post","status-publish","format-standard","hentry","category-computers","tag-bandwidth","tag-filesystem","tag-in-transit","tag-propagation","tag-propogation","tag-storage"],"yoast_head":"\nThe InTransit Filesystem - ClearChain<\/title>\n<meta name=\"description\" content=\"An article describing a new but yet to be implemented filesystem using propagation delay (intransit) nature of links as the storage system\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.clearchain.com\/blog\/posts\/the-intransit-filesystem\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Benjamin Close\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"5 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.clearchain.com\/blog\/posts\/the-intransit-filesystem#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.clearchain.com\/blog\/posts\/the-intransit-filesystem\"},\"author\":{\"name\":\"Benjamin Close\",\"@id\":\"https:\/\/www.clearchain.com\/blog\/#\/schema\/person\/aef6faa2c32188398139db9270ca1c98\"},\"headline\":\"The InTransit Filesystem\",\"datePublished\":\"2012-03-21T01:19:29+00:00\",\"dateModified\":\"2012-03-21T01:24:44+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.clearchain.com\/blog\/posts\/the-intransit-filesystem\"},\"wordCount\":1075,\"commentCount\":1,\"image\":{\"@id\":\"https:\/\/www.clearchain.com\/blog\/posts\/the-intransit-filesystem#primaryimage\"},\"thumbnailUrl\":\"http:\/\/www.clearchain.com\/blog\/images\/\/2012\/03\/UV-FibreOptic-Ends-150x150.jpg\",\"keywords\":[\"bandwidth\",\"filesystem\",\"in transit\",\"propagation\",\"propogation\",\"storage\"],\"articleSection\":[\"Computers\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/www.clearchain.com\/blog\/posts\/the-intransit-filesystem#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.clearchain.com\/blog\/posts\/the-intransit-filesystem\",\"url\":\"https:\/\/www.clearchain.com\/blog\/posts\/the-intransit-filesystem\",\"name\":\"The InTransit Filesystem - ClearChain\",\"isPartOf\":{\"@id\":\"https:\/\/www.clearchain.com\/blog\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/www.clearchain.com\/blog\/posts\/the-intransit-filesystem#primaryimage\"},\"image\":{\"@id\":\"https:\/\/www.clearchain.com\/blog\/posts\/the-intransit-filesystem#primaryimage\"},\"thumbnailUrl\":\"http:\/\/www.clearchain.com\/blog\/images\/\/2012\/03\/UV-FibreOptic-Ends-150x150.jpg\",\"datePublished\":\"2012-03-21T01:19:29+00:00\",\"dateModified\":\"2012-03-21T01:24:44+00:00\",\"author\":{\"@id\":\"https:\/\/www.clearchain.com\/blog\/#\/schema\/person\/aef6faa2c32188398139db9270ca1c98\"},\"description\":\"An article describing a new but yet to be implemented filesystem using propagation delay (intransit) nature of links as the storage system\",\"breadcrumb\":{\"@id\":\"https:\/\/www.clearchain.com\/blog\/posts\/the-intransit-filesystem#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.clearchain.com\/blog\/posts\/the-intransit-filesystem\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.clearchain.com\/blog\/posts\/the-intransit-filesystem#primaryimage\",\"url\":\"http:\/\/www.clearchain.com\/blog\/images\/\/2012\/03\/UV-FibreOptic-Ends-150x150.jpg\",\"contentUrl\":\"http:\/\/www.clearchain.com\/blog\/images\/\/2012\/03\/UV-FibreOptic-Ends-150x150.jpg\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.clearchain.com\/blog\/posts\/the-intransit-filesystem#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.clearchain.com\/blog\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"The InTransit Filesystem\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.clearchain.com\/blog\/#website\",\"url\":\"https:\/\/www.clearchain.com\/blog\/\",\"name\":\"ClearChain\",\"description\":\"-= Daily Happenings =-\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.clearchain.com\/blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.clearchain.com\/blog\/#\/schema\/person\/aef6faa2c32188398139db9270ca1c98\",\"name\":\"Benjamin Close\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.clearchain.com\/blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/19dca0aa372edfa901b93c556dfda2e78ad4434558fe4d139598e086315d714a?s=96&d=mm&r=pg\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/19dca0aa372edfa901b93c556dfda2e78ad4434558fe4d139598e086315d714a?s=96&d=mm&r=pg\",\"caption\":\"Benjamin Close\"},\"sameAs\":[\"http:\/\/www.clearchain.com\"]}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"The InTransit Filesystem - ClearChain","description":"An article describing a new but yet to be implemented filesystem using propagation delay (intransit) nature of links as the storage system","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.clearchain.com\/blog\/posts\/the-intransit-filesystem","twitter_misc":{"Written by":"Benjamin Close","Est. reading time":"5 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.clearchain.com\/blog\/posts\/the-intransit-filesystem#article","isPartOf":{"@id":"https:\/\/www.clearchain.com\/blog\/posts\/the-intransit-filesystem"},"author":{"name":"Benjamin Close","@id":"https:\/\/www.clearchain.com\/blog\/#\/schema\/person\/aef6faa2c32188398139db9270ca1c98"},"headline":"The InTransit Filesystem","datePublished":"2012-03-21T01:19:29+00:00","dateModified":"2012-03-21T01:24:44+00:00","mainEntityOfPage":{"@id":"https:\/\/www.clearchain.com\/blog\/posts\/the-intransit-filesystem"},"wordCount":1075,"commentCount":1,"image":{"@id":"https:\/\/www.clearchain.com\/blog\/posts\/the-intransit-filesystem#primaryimage"},"thumbnailUrl":"http:\/\/www.clearchain.com\/blog\/images\/\/2012\/03\/UV-FibreOptic-Ends-150x150.jpg","keywords":["bandwidth","filesystem","in transit","propagation","propogation","storage"],"articleSection":["Computers"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.clearchain.com\/blog\/posts\/the-intransit-filesystem#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.clearchain.com\/blog\/posts\/the-intransit-filesystem","url":"https:\/\/www.clearchain.com\/blog\/posts\/the-intransit-filesystem","name":"The InTransit Filesystem - ClearChain","isPartOf":{"@id":"https:\/\/www.clearchain.com\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.clearchain.com\/blog\/posts\/the-intransit-filesystem#primaryimage"},"image":{"@id":"https:\/\/www.clearchain.com\/blog\/posts\/the-intransit-filesystem#primaryimage"},"thumbnailUrl":"http:\/\/www.clearchain.com\/blog\/images\/\/2012\/03\/UV-FibreOptic-Ends-150x150.jpg","datePublished":"2012-03-21T01:19:29+00:00","dateModified":"2012-03-21T01:24:44+00:00","author":{"@id":"https:\/\/www.clearchain.com\/blog\/#\/schema\/person\/aef6faa2c32188398139db9270ca1c98"},"description":"An article describing a new but yet to be implemented filesystem using propagation delay (intransit) nature of links as the storage system","breadcrumb":{"@id":"https:\/\/www.clearchain.com\/blog\/posts\/the-intransit-filesystem#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.clearchain.com\/blog\/posts\/the-intransit-filesystem"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.clearchain.com\/blog\/posts\/the-intransit-filesystem#primaryimage","url":"http:\/\/www.clearchain.com\/blog\/images\/\/2012\/03\/UV-FibreOptic-Ends-150x150.jpg","contentUrl":"http:\/\/www.clearchain.com\/blog\/images\/\/2012\/03\/UV-FibreOptic-Ends-150x150.jpg"},{"@type":"BreadcrumbList","@id":"https:\/\/www.clearchain.com\/blog\/posts\/the-intransit-filesystem#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.clearchain.com\/blog"},{"@type":"ListItem","position":2,"name":"The InTransit Filesystem"}]},{"@type":"WebSite","@id":"https:\/\/www.clearchain.com\/blog\/#website","url":"https:\/\/www.clearchain.com\/blog\/","name":"ClearChain","description":"-= Daily Happenings =-","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.clearchain.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/www.clearchain.com\/blog\/#\/schema\/person\/aef6faa2c32188398139db9270ca1c98","name":"Benjamin Close","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.clearchain.com\/blog\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/19dca0aa372edfa901b93c556dfda2e78ad4434558fe4d139598e086315d714a?s=96&d=mm&r=pg","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/19dca0aa372edfa901b93c556dfda2e78ad4434558fe4d139598e086315d714a?s=96&d=mm&r=pg","caption":"Benjamin Close"},"sameAs":["http:\/\/www.clearchain.com"]}]}},"amp_enabled":true,"_links":{"self":[{"href":"https:\/\/www.clearchain.com\/blog\/wp-json\/wp\/v2\/posts\/677","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.clearchain.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.clearchain.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.clearchain.com\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.clearchain.com\/blog\/wp-json\/wp\/v2\/comments?post=677"}],"version-history":[{"count":7,"href":"https:\/\/www.clearchain.com\/blog\/wp-json\/wp\/v2\/posts\/677\/revisions"}],"predecessor-version":[{"id":1073,"href":"https:\/\/www.clearchain.com\/blog\/wp-json\/wp\/v2\/posts\/677\/revisions\/1073"}],"wp:attachment":[{"href":"https:\/\/www.clearchain.com\/blog\/wp-json\/wp\/v2\/media?parent=677"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.clearchain.com\/blog\/wp-json\/wp\/v2\/categories?post=677"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.clearchain.com\/blog\/wp-json\/wp\/v2\/tags?post=677"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}