Lessons Learned from Building Blockchain Consortia Apps

A consortium is an association of business companies. It includes companies that may be competitors or partners—or both—that need to work together to do business. It's one of the most exciting and interesting aspects of the blockchain landscape, and there are many use cases for business consortia to use blockchain to enable new and innovative sways of exchanging data.

Kaleido is a leader in this space and has extensive experience with building, hosting, and running applications for consortia. A few of the projects that we've worked on in the enterprise space include the Riskstream Collaborative for insurance providers, Synaptic in the healthcare space, and Covantis in trade settlement. These are all groups of companies that are exchanging data and using blockchain to solve real problems in their industries.

On our YouTube channel we have walkthroughs to help you set up a consortium, add members, and set up how they control their piece of the application. In this article, we’re going to zoom out and talk about not just the foundational level where we exchange data on a blockchain, but also what a web3 consortium really looks like, the pieces that go into making it work for all parties, and some of the lessons I’ve learned in building these sorts of applications.

The Anatomy of a Decentralized Business Network

Looking at the anatomy of a decentralized business network, we have a set of members in a particular industry that need to exchange data with each other. In most cases, each of them has some existing core systems.

In the image below, you can see these core systems on the left and the right, which have probably been around for years—decades even—and are well established and running these businesses. Our goal is to find a new way to exchange data, to help these systems communicate, and build a decentralized architecture that allows each member to run their own piece of infrastructure.

In most cases, businesses on a blockchain are going to want at least two ways of exchanging data. They’ll want to share some data privately, directly from one member to another. This data is shared off-chain. Then there is the central piece that we all like to talk about—an immutable ledger or blockchain, where we have public verifiable data and interesting constructs like digital assets and tokenization.

With this picture in mind, I’m going to share my personal journey and some of the lessons that I've learned as I moved from a Web 2.0 developer to a web3 developer to build blockchain-backed business networks. I’m sharing these lessons through the consortium lens to help you build applications that make it easier for businesses to exchange data.

1. Solve a solvable problem

First, you need to pick a question for which blockchain is the answer. This seems like an obvious one, but it’s not always obvious when people are first approaching blockchain. Blockchain, like any tech, doesn't solve all problems. It solves some problems really well, and you have to start from a point of picking a problem that blockchain can solve.

Of course, this implies that you have an understanding of the businesses involved and what their problems might be, and also that you have an understanding at some level of how blockchains work and what problems blockchains are good at solving. Then you can find an intersection. If you don't start on the right foot you're going to find yourself quickly going down the wrong road.

With that said, blockchain-backed apps are excellent for solving problems like:

• Gaps in trust between parties collaborating on shared data.
• Lack of traceability and verification in cross-company workflows.
• Difficulty reconciling multiple views of a single object/person/event.
• Overly centralized data storage and validation.
• Indeterminate ordering of events in multi-input systems.

Starting from one of these problems is always a good way to frame your planning of a consortium. To make it a little bit more concrete, let’s compare two different problems that you may try to solve using blockchain to illustrate a good use case versus a poor use case.

‍

Problem 1

Say you have an existing business process. Why not just take that business process, convert it to a smart contract, put it on the blockchain, and therefore get all these benefits from the blockchain?

You can certainly do this, but it usually ends up being a very high effort endeavor. It takes a long time, you need to do a lot of work to get other people in the industry to buy into it, and you're likely going to end up with a lot of privacy concerns. In the end, you may do a lot of work and end up with something that's not all that different from what you started with.

Problem 2

On the other hand, imagine you have a problem along the lines of this: you have a network of providers, and an individual participant in this network can only send three Fizz per 24 hours regardless of the provider he uses.

Now, this is a very concrete problem. It's arbitrary, but you'll find analogs to this in banking, finance, insurance, and all sorts of industries where businesses work together but have some sort of legal or regulatory requirements that everyone has to abide by. It's a defined pain point between members. There’s a piece we need to define around identity resolution—how we determine that we're all talking about the same John—but once you solve that and find a way to agree on who we're talking to, there's actually very little data that's needed on the chain. This is where you can really get value out of a blockchain application, as you bring it in to supplement an existing process and drive value.

Taking things like Problem 1 and shifting them to make them more like Problem 2 are the key to success in this space. The goal is to take something that’s big and abstract and find a single point where blockchain can provide a real value.

2. It’s not all about the ledger

This is where I might lose some blockchain experts, but in reality this is a key point. It's easy to get enthusiastic about the blockchain and feel like we have this great way to exchange data, so let's put as much as possible into the chain. Here, it's important to take a step back.

There are actually a lot of things you can't put on the blockchain. It’s not a replacement for existing data stores. It’s not going to replace an entire back office system that you've built for decades. Of course the blockchain has some inherent things that are good about it, but they can also be a liability.

Namely, it's permanent and public to all the members of the network. Anything you put on there can never be deleted, so you have to think about what you want to make public. The reality is that you actually need to think about what data goes on chain, what data goes off chain, and how you’re going to bind those things together.

So after defining a problem for your consortium, you need to decide on how you’re bucketing the data. This should be considered even before deciding on a ledger—often people get married to a protocol or component too early; they decide they have to use Ethereum or Fabric before even deciding what the architecture of the application looks like. It’s helpful to sort out the data before you pick the tech.

As you think about your data and how you’re going to handle it, you can keep these key points in mind:

• Every application will be a combination of on- and off-chain pieces
• The ledger is not a full replacement for an existing database
• Assume on-chain data is public to all and can never be deleted
• Pick the minimum data, rules, or proofs that need to be on chain
• Sort out your data before you settle on a specific ledger technology
• Have a reliable way to bind on- and off-chain elements together

Another way to look at it is to pick what box you're going to put things in. On-chain data may include:

• Data public to the network (identities, schemas, announcements)
• Proofs/hashes of off-chain payloads
• Static rules and logic with no dependency on private data
• Tokens, NFTs, and “digital twins” to represent real world items

Off-chain data may include:

• Any data not visible to all members
• Any data that is subject to deletion
• Business rules that require interaction with private data
• Interfaces to integrate with existing software/databases

It’s really important to think about which pieces of the solution go into which box and how they interact and tie back and forth before moving on to the tech. There are so many pieces to a consortium solution outside of the ledger itself. With that said, though…

3. It’s all about the ledger

On the flip side, it's actually all about the ledger. Once you reconcile the idea that some things are going to be on-chain and some things are going to be off-chain, you realize that you can start to solve real problems with the blockchain without having to put every bit of data on the chain.

We know that some of the critical paths where the blockchain can serve as the source of truth include:

• Verifying the sender of a piece of data
• Verifying the integrity of a piece of data
• Tracking uniqueness and ownership of objects/data
• Using tokens to track transfer of value
• Determining the order of events

So if we take this knowledge, a simple recipe for a lot of consortia is to think about one point in a workflow where blockchain fits. You don't want to replace your whole workflow; what you want to do is isolate a function that happens between your company and another company and defer a decision to the blockchain. That’s where you want to use the blockchain to be the source of truth—whether it's an integrity decision, maybe a uniqueness or ownership decision. You want to supplement your workflow by moving one thing to the blockchain. Then, if you move one thing, you can start to see other things that you can solve.

In this picture, we look at where we can insert Hyperledger FireFly to help. It’s a really powerful way to manage data that takes a lot of different forms.

‍

If I'm sending data from Company A to Company B, it's probably going to be a combination of on- and off- chain data. As I’ve said, it's rare in a business case that all of your data for a transaction can go onto your side chain for every member of your consortium to see. Assuming we have some combination of on- and off-chain data, I can use Firefly to pin those things together and be assured that the off-chain data is always going to be verifiable because of a hash that was put on-chain.

If you plug in your existing core systems on both sides of this, you can start to build working flows that supplement but don't replace your existing back office. We’ve seen this to be really powerful in enabling people to move gradually onto a blockchain system and trust it.

4. Put your things in order

Building blockchain applications is fundamentally different from building traditional applications—whether it's Web 2.0 or on-prem or any other type of applications that you've built before. It all centers around in-order, event-driven processing.

That’s because blockchain applications have a couple of unique characteristics:

• Blockchains are slow by design.
• Every node is guaranteed to emit events in the same order.
• Decentralized apps will each run their processing in isolation.
• All changes in application state should be driven by async events.

Within the context of a decentralized app, it’s critical to track the difference between “proposed” and “confirmed” items. Local operations on my app cannot be processed differently from anyone else’s—they are not “true” until some record of them is sent to the blockchain and comes back confirmed.

A diagram that I I like to show is the one below with two members, A and B, that are cooperating on a set of data through blockchain. They're linked together with blockchain nodes. Member A might be proposing some changes to that data, and B might be proposing some changes to that data, but neither one of these can process their own changes in isolation because they would end up with different results.

‍

But if you put some proof of each of these items onto the blockchain and then you wait for the blockchain to confirm it, and you each run your processing in your own nodes only after the confirmation, you can be guaranteed of the ordering and the authorship of the data.

This was a difficult lesson to wrap my head around, but it's a really powerful and almost magical thing about building apps on top of blockchain. You can each run your own copy of an app, and even though it's not directly talking to the other app, it ends up in the same state by running the same rules on the same events in the same order.

5. Build to last

The final lesson I’ve learned is to pick tools and designs that scale. It's easy to think about web3 as a one-time cost, but it's not a one-time cost. It's a different way of thinking about exchanging data and it can be really powerful. But you have to build a system that can scale up and bring on other members. A consortium doesn't work in isolation. While there are problems you can solve by yourself with blockchain, you get unique benefits when you can build a consortium and find ways to add people to that consortium and exchange data.

Use open source

Hyperledger FireFly was built for exactly this purpose, because we saw that there are so many big problems when you zoom out. It's not just the blockchain, there are all these integration pieces and middleware pieces and event synchronization pieces, and we've tried to build FireFly into a toolkit that makes all of these ten times easier.

Leverage skills that your team has

You don't have to do everything in Solidity or a blockchain-specific language. The off-chain bits can be written in JavaScript, Java, Go, or whatever your team feels most comfortable with. Whatever is present in your company is always going to accelerate the building of a consortium, if you can allow members to build their own integrations in the language of their choice and just agree on data formats for the network.

Minimize the dependency on niche skills

If you have to bring a consortium member on board and they need a fully scaled out team of blockchain developers, a fully scaled out team of cloud engineers, and an IT team for support, it's going to be cost-prohibitive. It’s harder to build a working consortium if every member needs to spend a lot to operate.

Choose a deployment that scales

You're always going to have a few of the players that want to own things and have their own IT department and their own specific security restrictions. They want to own the blockchain nodes and own the data and operate things at a granular level. But then you're often going to have other players that would prefer a more hands-off approach, something that’s hosted and managed for them, which is a great place to plug in Kaleido and our Consortium-as-a-Service to help build working consortia and solve real business problems.

In Conclusion

In the web3 world, whether you're a newcomer or a seasoned veteran, keep your perspective on why you want to use blockchain technology. It’s important to clearly define your problem, understand how you want to handle your data, then move into technical discussions about the ledger and the other tech you stack on top of it. Remember, you're here to solve a business problem, and you need to define your parameters well to be able to truly measure ROI on the backend.

If you’re ready to get started, partnering with Kaleido means you have an expert on your side to manage all the plumbing, technical challenges, and hosting related to your consortium. This allows you to spend your DevOps and engineering cycles on the business logic and the user-facing experience—ensuring you deliver value to customers.