Local First Patterns in our Apps

General
1

TL;DR: When you are building something on top of a p2p network (e.g. Waku), think about making it Local First to improve UX whenever the network is unreliable or unavailable.

I’ve been thinking and talking about local first (https://localfirstweb.dev/) for some time, but mostly in particualar contexts of a particular app or a project. I am no expert on local first development or architecture, but it is something that heavily resonates with me - especially in context of p2p networks.

Early internet apps started as Local First - you’d do things locally and since early on you’d rarely have a 24/7 stable connection, you’d sync your stuff whenever possible - anyone tried Git?:slight_smile:

We evolved internet and our tools to be always online and always in sync with everyone else - and we mainly achieved this from the perspective of tools by turning them into services (SaaS) and centralizing them. Turning our fat clients (MS Word) into thin clients (Google Docs/Office 365). It is amazing to have access to your data and services from any device at any point. But there are huge buts.

We all probably understand well enough the risks of relying on centralized services, so this post is not about that - and many centralized services (Google Docs) allow you to work offline and hence “local first” as well.

We build and use peer-to-peer networks and as we established through various experiences in Waku, Status, Codex… p2p networks are inherently unreliable. As much as we’d love to add all the reliability guarantees of centralized systems - we cannot. The only way to overcome this is to always approach our apps as Local First and then realy on p2p network for syncing.

I am absolutely one to blame for not following this approach as well:) Qaku is not local first and will not work without connection to Waku, shame on me! The main reason is my laziness and also that Q&A kinda assumes a live interaction (at least in my head).

But we have another great examples where Local First approach should be very much the go-to mantra - OpChan and WhisperBox. Let’s focuse on OpChan because that is something we need and want to put in front of people.

We should probably consider it in our FURPS (probably in Usability?) to add items specifically talking about “when the network connection is not available, …” - I should be able to read threads that I have already synced from the network, but I should be also able to upvote, reply and create new threads while being offline - the content / messages should be cached and synced with the network when I am reconnected to Waku.

There are also existing libraries and specs which already help with the Local First approach - namely CRDTs - e.g. GitHub - yjs/yjs: Shared data types for building collaborative software is used in Fileverse (ddocs.new) and we could consider integrating Waku (either as a transport for sync or signaling approach to establish direct WebRTC/p2p connection).

Call to action: If you are interested in this, or have an experience with Local First approach, or have an app that you have built with this approach in mind, please do share - apps, insights, libraries, thoughts, coutner-arguments. I believe it would be extremely valuable to know about and leverage Local First in anything we build, but I also might be wrong or missing some key perspective.

5 Likes
2

I totally support the local-first approach!

My two cents from a user’s perspective. Local-first apps give me a comforting feeling of stability: what I see on the screen will be there again when I relaunch the app. Modern cloud-first apps, in contrast, shift towards shuffling content before my eyes, making me feel like nothing is really there, that I don’t control what I see and can only passively consume what I’m fed by our big-tech overlords.

Two local-first apps that I name as good examples:

  • Obsidian: a knowledge based build on plain markdown files. It’s so reassuring to know that I’m not locked-in in a particular app. Even though the app is closed-source (not ideal, but fair enough, I’m respecting the devs’ choice), my files are perfectly portable and stored locally.
  • Organic Maps: an Openstreetmap-based app, a fork of Maps dot Me (the original developers forked off after their project was acquired, ironically, by a blockchain firm that started agressively monetizing it by introducing a crypto wallet). What I enjoy most about this app is that you download a city once (few tens of megabytes), and then the map just works, renders instantly, online and offline, with things showing up where they always are, unaffected by ad algorithms or whatnot. Good for battery life too.
3 Likes
3

P.S. Another local-first app I’ve heard about but never tried myself is Anytype - another knowledge base AFAICT.

1 Like
4

Yes.

2000×1443 125 KB

The “Application Cache” box matters :slight_smile:

1 Like
5

You are right to highlight the change of mindset in terms of development decentralized applications @vpavlin .

To facilicate onboarding, it seems fair to make it explicit that one should develop a local-first app.

I wonder how this should be reflected in our code. for native apps, the expectations is that the application brings a DB that implement specific interface.

Building native app for Status does biaised us toward an existing app, that already has a DB and other patterns.

When building a brand new app, web or native, it would make sense to provide a good default being an out of the box DB. Which can then be swapped for something else once the app developer proceed with further optimization.

I wonder in terms of basic usage of Waku and the messaging API, how much would it change the development patterns? Is it useful for the developer to have access to local waku messages previously received? I guess they could “load” messages that are stored locally at app start to render and display.

1 Like
6

There are 2 basic approaches that I could imagine - both valid and both having their own pros and cons

  1. Cache received/published Waku messages locally
  2. Store app state locally

Both can also co-exist, but each will probably want to choose one approach to not to duplicate data.

Cache Waku Messages

This is what Waku Dispatcher (and hence Qaku and anything else I build) does. This approach basically relies on Event Sourcing pattern and influences how you design the protocol. Each message represents a state change and the final state is “computed” by (re)processing all the evenvts. IMO SDS does play really well with this approach.

All the successfully processed (i.e. valid/accepted) messages per “session” (whatever that session is - a group chat, Q&A…) are stored locally and then re-processed on load to provide the current state for/in the application.

This means lower layers (Waku, Messaging API, Waku Dispatcher…) can take care of the data storage and the app developer’s life is a bit easier.

It also means some computation overhead - if the messages are encrypted, we’ll probably want to store them encrypted, hence we might need to decrypt potentially long log of messages to start the app - but things can be optimized.

It works well with Store/History API as well given that updating state is simply adding latest messages in the event log.

This underlying database can also be used to persist outgoing messages - both published and to-be-published and the libraries working with the DB can handle delayed/lazy publishing or retries easily.

Store app state locally

Here we let the app to take care of storing its state - basically your app needs to take care of CRUD operations as only the app understands the data structures it can operate with. There is not much we can do here to help the developer - other than recommending technologies that we deem appropriate. The app developer needs to setup the schema and figure out what indexes and queries he needs for the DB.

It gives more control about what is stored and how, but also puts more pressure on the developer - I can imagine ChatSDK coming with prebaked schema and DB, but not Messaging API.

As you can probably guess, my preferred approach is #1 as it makes underlying libs more complicated, but also makes my life as a dev super simple - I don’t have to worry about any DBs and the state is/will be reconstructed for me when I initialized Waku Dispatcher / Messaging API /…

Good think also is that it can work the same way in any implementation - all you need is a Key/Value store with indexes (e.g. IndexedDB in browser, LevelDB on desktop/backend) andt the interface for the app developer is absolutely consistent - basically get/set/getAll/delete :slight_smile:

This can also be a configurable option to enable, so having such caching can be optional and the app dev could opt-out and simply use their own higher-level DB as well (or use both if there is any reason)

7

To me it sounds like a evolution. Where #1 comes out of the box, but then a developer can opt in #2.

When #2 is opted in, maybe #1 remains enabled but the local caching can have some lower limits (ie, only 1 day of data is kept in waku cache instead of all times).