On Dat://

I spent four months using Beaker actively: running this website on the network,
messing with the different libraries, trying out the different
apps—and then
I hit a wall. Had a good time, for sure. And I kept seeding my Dats—kept my
sites on the network. The technology was just lovely.

But: you can’t yet edit a website from a different browser (like on a
different computer). This is called multi-writer support—and there is some
talk about this landing by the end of the year. But this is, from what I can
see, the single major omission in Beaker. (It’s not a problem with Dat itself—you
can use a Hyperdb for that—but Beaker hasn’t settled the details.)

So I left Dat alone. I figured: they need time to work this problem out. Beaker
has remained remarkably effortless to use—I’d hate for multi-writer to be tacked on,
complicating the whole thing.

An Imperfect Dat—Cool?

Recently, it occured to me that maybe I don’t need multi-writer. And maybe I
should really be sure that the rest of Dat is as perfect as I think it is.
So I started working on a limited (but full-featured) app for Beaker, with the
intention of writing up a full ‘review’/‘teardown’ of everything I discover
in the process.

This is my review—and the app is
Duxtape.

It occured to me that a Muxtape clone would be a perfect tracer
bullet for me to push Beaker.
(Muxtape was a 2008 website for sharing mixtapes—minimal design, suddenly
became very prominent, and then was promptly DEMOLISHED by the music industry.)

1. Muxtape was shut down because it was centralized. If Muxtape had been
   distributed, it would be much more difficult (perhaps impossible) to
   shutter.

2. Muxtape did some file processing. Reading music file metadata
   (title, artist’s name) and loading music into the browser’s music player.
   Could the app handle this?

3. The Muxtape home page listed recent mixtapes. This would give me a chance
   to use datPeers—a way of talking to others that are using the same site.

4. Storing song information and order. I don’t have a database, so where
   do I put this stuff?

5. A more general question: What if I upgrade the code? How do I handle
   upgrading the mixtapes too?

I also didn’t want to think in terms of social networks. Many of Beaker’s most
advanced apps (like Fritter and Rotonde) are ‘messaging’/‘social’ apps. I
specifically wanted a creation tool that spit out something that was easy to
share.

How would Beaker do with that kind of tool?

A Teardown of The Network

Ok, so how does Dat work exactly? It is simply a unique address attached to
a folder of files (kind of like a ZIP file.) You then share that folder on the
network and others can sync it to their system when they visit the unique
address.

In the case of Duxtape, the address is
dat://df1cc…40.

The full folder contents can be viewed
here
at datBase.

So when you visit Duxtape, all that stuff is
downloaded. Beaker will show you the index.html, which simply lets you
create a new mixtape and lists any that you’ve encountered.

Now, you can’t edit my Dat—so how do you create a mixtape?? And how does it
keep track of other mixtapes?? Teardown time!

CREATING A MIXTAPE

This creates a new Dat (new folder on your computer)
with just index.html inside. I actually copy the tape.html
from my Dat into that folder, your mixtape. That HTML file will load its
images and Javascript and such from MY Duxtape dat! (This means I can
upgrade my original Dat—and upgrade YOUR Dat automatically—cool, but…
dangerous.)

DISCOVERING A MIXTAPE

When you hit someone else’s mixtape link, the Javascript loads the Duxtape
home page in an HTML iframe—passing the link to that page. The link is then
stored in ‘localStorage’ for that page. So, those are kept in a kind of a
cookie. Nothing very server-like about any of that.

But furthermore: when you are on the Duxtape homepage, your browser will
connect to other browsers (using datPeers) that are viewing the homepage.
And you will trade mixtapes there. Think about this: you can only discover
those who happen to be around when you are! It truly acts like a street
corner for a random encounter.

ALTERING A MIXTAPE

Where are song titles and song ordering kept? Well, heh—this is just kept in
the HTML—in your index.html. Many Beaker apps keep stuff like this in a
JSON file. But I felt that there was no need for duplication. (I think the
IndieWeb has fully corrupted me.) When I want to read the mixtape title, I
load the index.html and find the proper tags in the page.
(Like: span.tape-title, for instance.)

PUBLISHING A MIXTAPE

Beaker has a special technique you can use for batching up edits before you
publish them. (See the
checkout method.)
Basically, you can create a temporary Dat, make your changes to it, then
either delete it or publish it.

However, I didn’t go this route. It turned out that I could batch up all my
changes in the browser before saving them. This includes uploaded files! I
can play files in the browser and read their data without copying them to the
Dat. So no need to do this. It’s a neat feature—for a different app.

So this allows you to work on your mixtape, add and delete songs, get it
perfect—then upload things to the network.

This all worked very well—though I doubt it would work as well if you had
1,000 songs on your mixtape. In that case, I’d probably recommend using a
database to store stuff rather than HTML. But it still might work well for 1,000
songs—and maybe even 1,000,000. This is another advantage to not having a
server as a bottleneck. There is only so much that a single person can do to
overload their browser.

For reading song metadata, I used the
music-metadata-browser
library—yes, I actually parse the MP3 and OGG files right in the browser! This
can only happen in modern times: Javascript has become a competent technology on
the server, now all of that good stuff can move into the browser and the whole
app doesn’t need a server—in fact, WebAssembly makes Dat even more compelling.

Special Feature: The DatArchive Object

Lastly, here are some calls that I used which are specific to the Beaker
Browser—these are the only differences between running Duxtape in plain Chrome
and running it distributed:

1. stat: I use this to
   check if a song file has already been uploaded.

2. readFile: To read
   the index.html when I need to get song information.

3. writeFile: To save
   changes to songs—to publish the index.html for your mixtape.

4. unlink: To delete
   songs—NOTE: that songs are still in the Dat’s history and may be downloaded.

5. getInfo and
   configure: Just to
   update the name of the mixtape’s Dat if the name of the mixtape is changed by
   you. A small touch.

6. isOwner: The getInfo() above also tells me if you are the owner of this
   mixtape. This is crucial! I wanted to highlight this—I use this to enable
   mixtape editing automatically. If you don’t own the mixtape, you don’t see
   this. (All editor controls are removed when the index.html is saved back
   to disk.)

So this should give you a good idea of what Dat adds. And I just want to say:
I have been wondering for awhile why Dat has its own special format rather than
just using something like Git. But now I see: that would be too complex. I am
so glad that I don’t have to pull() and commit() and all that.

I spent most of my time working on the design and on subtle niceties—and
that’s how it should be.

Peeling Back the Good and Bad

It’s clear that there are tremendous advantages here: Dat is apps without
death. Because there is no server, it is simple to both seed an app (keep it
going) and to copy it (re-centralize it). I have one central Duxtape right now
(duxtape.kickscondor.com), but you could easily fork that one (using Beaker’s
‘make editable copy’ button) and improve it, take it further.

The roots of ‘view source’ live on, in an incredibly realized form. (In Beaker, you
can right-click on Duxtape and ‘view source’ for the entire app. You can do this
for your mixtapes, too. Question: When was the last time you inspected the code
hosting your Webmail, your blog, your photo storage? Related question: When was
the first time?)

In fact, it now becomes HARD:IMPOSSIBLE to take down an app. There is no app
store to shut things down. There is no central app to target. In minutes, it can
be renamed, rehashed, reminified even (if needed)—reborn on the network.

This has a fascinating conflict with the need to version and centralize an app.
Many might desire to stay with the authoritative app—to preserve their data,
to stay in touch with the seeders of that central app. But this is a good
tension, too—it INSISTS on backwards compatibility. I am pressured to keep
Duxtape’s conventions, to preserve everyone’s mixtapes. It will be difficult to
upgrade everything that is possibly out there.

This same pressure is reminiscent of the Web’s own history: HTML that ran in
1995 often still runs today—Flash and Quicktime are quite the opposite, as
will be all of the native apps of today. (Think of apps you’ve bought that are
already outmoded.) The ‘view source’ keeps compatibility in check. If Beaker
is able to keep their APIs firm, then there is real strength here.

Still, Dat is limited. Where is it short? Can we accept these?

• It truly RESISTS centralization. This becomes starkly apparent when you are
  working on your app—you cannot connect to a REST web service. You need to
  rethink everything. This is good—but it is painful.
• Discovery suffers. This is related: I cannot just advertise published
  mixtapes to a central web server that stays up all night showing off how
  busy things are. But, as I mentioned above (in the PUBLISHING A MIXTAPE section,) the
  datPeers feature has really helped assuage this sore spot.
• Not everything can be stored in the browser. How does a search engine
  work on this network? Or is this type of centralization something we should
  resist? (I do offer search on my Dat-version of this website, by leaning on
  Elasticlunr.js.)
• Inter-app communication is hard. Earlier I mentioned that I need to use
  an HTML iframe to communicate with the Duxtape home page—there is no need
  to use the Fetch
  API (AJAX) in
  Beaker, ever. DatArchive deprecates it. (Though I would be interested to see
  a use for the Fetch API—if a Dat could house a GUI-less service, to
  negate the need for iframes.)
• The multi-writer problem. Again: you cannot edit a Dat from a second
  machine.

But—think about this: I don’t have to take on cloud hosting! I don’t need to
scale the app! This is a huge relief. URGENT QUESTION: Why are we trying to even do this?

I also mentioned not needing the multi-writer feature. Obviously, multi-writer
demands some centralization. A central Dat needs to authorize other Dats. But I
think this centralization could be moved to the DNS resolution—basically, if
I edit Duxtape on a second machine, it will have a new unique address—and I
can point duxtape.kickscondor.com to that new address. This means I can never
get locked out of the Dat—unless I am locked out of the DNS. (So there is a
way forward without any new features.)

Still, these downsides are pure side effects of a distributed Web. These are the
realities we’re asking for—for me, it’s time to start accepting them.

Dat Uptake

Several months had passed since I last used Dat—how was it doing with adoption?

Well, it seems, no different. But it’s hard to say for a distributed network.
Every Dat runs in secret—they are difficult to find. The discovery problems
are perhaps the most urgent ones.

But there is good recent work:

• Cabal: Not a browser project. Just an IRC-like network
  on Dat. There is very active work on all of
  these projects.
• Data Terra Nemo and the DWeb Camp
  show that ‘distributed web’ stuff has momentum. Beaker seems to have a solid
  presence at these.
• Unwalled Garden: The developer of Beaker is
  dabbling with… social networks. This is probably needed, though. But I would
  hope for more work on multi-writer, on sparse downloading, on different modes
  of seeding (like it would be cool to have a ‘vacuum’ type mode—where you
  only seed the latest,) or on BitTorrent integration. (I wish I could just
  serve large files with BitTorrent and mix dat: and magnet: links!)
• The Dat project’s blog has all kinds of
  academic and hobbyist work going on.

These are all cool—but Dat has a long way to go. With the corpypastas (or CorpASAs) taking
up all the attention, adoption is terribly slow. What Beaker may need most of
all is a mobile version. But, hey, I’ll write my article here and make my
dent—if you feel stimulated to noise about, then please join in. I mean: using
a new web browser is just very low effort—perhaps the lowest. You need to use
one anyway!

I think
HTTPS has proven itself well for the centralized stuff. Perhaps there is a
future for HTTPS as simply a collection of centralized REST APIs for things
like search and peer discovery. I think the remaining apps could migrate to this
fertile garden emerging on the Dat network.