A year ago we launched Kaleido with a goal to make blockchain adoption easy for enterprise. We've had an exciting year partnering with you, bringing projects to production, and helping modern business networks get the technology they need to build shared IT solutions.
In the engineering team we've been working hard, delivering continuously while retaining the enterprise grade reliability our production networks require. Last week on the first anniversary of our launch at NYC Blockchain Week, we did something a bit different... our biggest single drop of new function since we launched.
It's been quite a year!
So I wanted to spend a little time reflecting back on our engineering journey so far, and why we believe we have now delivered the first full stack blockchain solution for modern business networks.
The post is a reflection of how much we’ve added and how proud I am of the quantity and quality of capabilities the team has delivered.
A key focus for us has been choice and flexibility. A business network building decentralized technology across enterprise participants needs to be flexible and consumable by the IT teams of each organization, while retaining the integrity of a critical transaction processing system. That is what shared IT is all about.
In March 2019 Kaleido was proud to become the only enterprise SaaS blockchain service spanning Azure and AWS with a truly seamless, native experience. Kaleido’s borderless blockchain capability offers a single API, security, operation and tenancy model which makes for a dramatically simple, reliable, and cost-effective solution. This was the cumulation of a huge engineering project in the team re-architecting our networking platform and Kubernetes infrastructure for federated, firewall isolated, per-environment multi-cloud networks.
Last week we announced two new offerings that build upon these infrastructure and platform engineering advances:
Spinning up a blockchain network in minutes across multiple clouds is great... but alone it is not nearly enough to give your projects the acceleration you need. While blockchain is the beating heart of a decentralized solution, you have unanimously told us it is only 5-10% of the development needed to deliver your overall solution.
So since the middle of last year we have been filling in the core capabilities that need to be in place to support innovative use cases, and accelerate your projects.
We started back in the middle of last year close to the chain, by delivering the first Kaleido service (ethconnect) - a solution for streaming transactions reliably into the network. The throttling, ordering and nonce-management needed to operate a production blockchain network in production, all backed by the best-of-breed Apache Kafka messaging backbone.
Then we expanded out from the chain to other decentralized solutions and 3rd party services from industry-leading partners such as OpenLaw, Chainlink and Clause, - launching our Marketplace back in November. We've been adding Kaleido Services continuously such as an IPFS decentralized filesystem, a Hierarchical Deterministic (HD) Wallet, and the second generation of our Public Ethereum Tether. We also delivered native AWS/Azure Cloud integrations for Key Protection, Backup, Log streaming, and Private Networking.
With the new services delivered last week, we believe we've reached the tipping point where the majority of enterprise projects can build apps on Kaleido without needing custom plumbing infrastructure. In short...
Let's run through the layers and take a look at the range of technologies a modern business network needs to build transformational projects with blockchain at their core.
We'll start where you live as developers, in the Decentralized Application (DApp), and then move down the stack all the way to the data layer.
Programming a DApp is fundamentally different to programming a traditional centralized database application. The core constructs are different. SQL Rows / NoSQL Documents and Stored Procedures... these are replaced by constructs like Tokens and Smart Contracts. The low level interfaces you come across are alien to traditional development, like JSON/RPC with RLP (Recursive Length Prefix) encoded payloads, abstracted by thick client libraries like Web3/Nethereum. Then throw in some advanced cryptography, and you could be sent spinning.
... but not to fear. Application development can be streamlined with the right building blocks, and in Kaleido we've been working to give you all the tools you need to do productive enterprise application development in this new world.
Tokens are a core concept in the vast majority of Blockchain projects, and the awesome thing about the Ethereum ecosystem is they are a *hardened and native construct.
This is why I'm personally so excited by the suite of tokenization features we released last week.
Key Management is probably the hottest topic in any Enterprise business network, and rightly so. After all the key you use to submit transactions represents your identity, it is intended to be irrefutable proof that the transaction came from you.
In Kaleido, we have continually focused on allowing you to manage and own your own keys off-platform. This has always been possible when submitting transactions, by signing them externally in the application before sending. However, the complexity added to an application by that approach is not always justified. So we've done the work to link our new managed Wallet to keys that never leave your HSM. So you get the best of both worlds. Azure Key Vault is the first HSM we are enabling, with AWS CloudHSM coming soon.
In some use cases, it’s more appropriate to hide the identity of the transaction submitter, or at least make it impossible for other members to analyze your trading patterns. A popular technique is using one-time signing keys derived from a master secret. Hierarchical Deterministic Wallets are specifically designed for this purpose. Kaleido supports the HD Wallet as a service so you can easily use REST API calls to sign your transactions with any of the millions of derived keys available in the wallet using a simple index. And each of your transactions would appear to be from a completely separate account.
The first time you try out Blockchain programming, you might be surprised how much code it takes to perform simple tasks. Signing and sending a transaction, querying data, subscribing to events. Even with thick client libraries you can end up with many lines of code, and error handling. As most modern developments are microservices based, projects almost always push this logic to an API abstraction layer.
So in our latest enhancement to the REST API Gateway we now generate full REST APIs for you. Just pass us the zip of your Solidity + dependencies and we give you an API that does everything from installing your contract, to subscribing to events. It's a standard API with standard REST types, so now any application or integration technology can interact with the chain with ease.
And this isn't just for tinkering. POST your transactions in async mode, and transactions streamed over our reliable Kafka backbone with nonce-management and throttling - so you have what you need from first-use through to production.
Of course that kernel of on-chain smart contract logic is the core of the transactional system, and whether you are trying it for the first time, or an EVM / Solidity specialist, you need a comprehensive set of developer tools. You need a development experience from initial test on your laptop, to continuous integration builds for smart contract code changes, through to debugging/replaying transaction failures in production.
The great news here is that the Ethereum ecosystem is recognized as the most active and engaged development community in Blockchain. There are many thousands of DApp developers out there across public and private chains, and some amazing projects capturing the experience of how to design, code and unit test Solidity source code (and other EVM compatible languages).
We've been really active recently building developer to developer relationships between the Kaleido team and a number of these projects. Our friends in Truffle, Metamask and Remix in particular. So you will samples and instructions on how you can use these together with a private Kaleido chain to go right from unit testing with Javascript and Ganache, to stepping through debugging steps of transactions in a production chain.
Each participant in a business network has their own core IT infrastructure, their own systems of record. The new transactional systems being built in a decentralized way need to be integrated with these back-ends, individually by each IT team participating in the network.
The business transaction itself is seldom performed exclusively on the blockchain in smart contract logic, instead there are many off-chain interactions that happen as part of the end-to-end transaction, or as side-effects to it.
This is the reason we added the programming model of Chainlink to our marketplace last year. As the leading decentralized Oracle technology, Chainlink allows on-chain logic to initiate off-chain actions and pulls external data on-chain. Coordinated in a decentralized network where the members of the business network manage the Chainlink nodes.
Sometimes smart contracts just need quality data from authoritative sources. For these use cases we added curated oracles like Rhombus that provide feeds of data spanning many industries like insurance, supply chain management, and finance.
However, blockchain based technologies are not enough. Much of the data that needs to be exchanged to coordinate the transaction is simply too sensitive, too high volume, or too large to put on-chain. It's common to see two or more parties needing to exchange 10s of messages with each other's Applications for each transaction that touches the chain.
Of course physical networking, identity and permissioning are just as critical to these off-chain exchanges as the on-chain ones. The exchanges need to be secure, reliable, scalable, and coupled to the business network.
So Kaleido has baked end-to-end encrypted messaging into the platform. It is built upon our rugged Apache Kafka backbone, that has been in the platform since last year. Messages are delivered in a PKCS#7 envelope, encrypted for the target recipient, and signed by the sender. The identities of members and destinations are registered on-chain, and pinned to off-chain PKI identity via the on-chain registry (which we talk about later).
And don't worry, you don't need to become messaging specialists in your applications to use it. We've layered an intuitive socket.io interface on top, so in the programming language of your choice you can build secure message-driven communications with ease.
[video width="800" height="600" mp4="https://kaleido.io/wp-content/uploads/2019/05/kaleido_app2app_messaging.mp4" loop="true" autoplay="true"][/video]
... those of you who know me, won't be surprised to find I'm particularly excited by this new enterprise messaging feature and its ability to accelerate development of transactional use cases in business networks.
When you have simple REST APIs, asynchronous messaging transports, and event subscription baked in, it becomes pretty easy to couple your new decentralized projects to your existing systems and integration technologies (ESB etc.). However, we're not stopping there. We have developed a set of simple connectors that let you integrate without needing to write a custom glue layer at all.
The first three on their way are:
Of course the critical central component of your business network is the shared ledger, and the data submitted to it via the decentralized participants. You need that business data to flow reliably via mutually agreed on-chain Smart Contract logic, with the right amount of privacy.
This means you need to agree what the logic is between the participants, and which parties are able to see the data.
It is important to know the source code associated with each smart contract being used by the business network. Usually versions of this smart contract code are carefully reviewed by participants in the business network.
Post deployment verification that a deployed contract matches a particular source code version is complex, and difficult to automate. We often see enterprise projects compile and deploy a new smart contract instance for each transaction, allowing the business logic to evolve over time.
So the Kaleido REST Gateway now allows you to deploy the smart contracts via a simple REST API, generated from a Solidity source package. This assures the same bytecode is deployed every time, and makes it simpler to track an on-chain instance back to the source.
We've talked about how Kaleido includes transaction streaming and nonce-management for reliably delivering and throttling transactions as they arrive onto the chain (over our Kafka backbone). But what about the other direction?
The Ethereum programming model is designed to be event-driven, with really simple programming constructs to define and emit events. These events might need to trigger processing in one or more participants systems - remember smart contract logic must be deterministic, so it cannot make external calls (the Ethereum EVM enforces this rule strictly, preventing programming mistakes).
However, building the infrastructure to reliably detect and process these events can be complex.... so Kaleido has now added reliable event streams into the platform.
You simply define the target to deliver the events to, and we take care of the rest.
While every project will have some data that is visible to all participants of the private chain, that might only be a subset of the data that needs to be mutually agreed by multiple parties via shared business logic.
This is where an additional layer of privacy is needed, such as a Private Transaction Manager as specified in the Enterprise Ethereum specification. These give each participant a separate private store of data, which is updated by the same Ethereum programming model as the all-participant chain. The installation of smart contracts and submissions of transactions to these private enclaves are targeted to a subset of the participants. For example it could be a bilateral transaction private to two parties. The only thing recorded on the main all-participant chain for these transactions is a hash.
Kaleido has been managing the Quorum Constellation implementation of a private transaction manager for some time, which was the first to be developed (in Haskell, with NaCl cryptography). Kaleido has a close collaborative partnership with the Quorum team, who have been such an innovative force in the Enterprise Ethereum space since its inception.
In Kaleido's most recent protocol release (V1.0.11) Kaleido has updated from Constellation to Tessera - the next generation OSS project from the Quorum team. We automatically migrate all existing private transaction data. As well as being a much easier to maintain Java open source codebase, Tessera has some exciting new enhancements. The first significant change is the decoupling of the encrypted data store via JDBC - we use a H2 database in Kaleido (with reliable HA failover obviously!).
Another significant enhancement is the ability to perform external transaction signing off-platform before submitting transactions. This enhancement means you can choose to have no transaction signing keys within Kaleido even when using private transactions.
We also did some significant under-the-covers engineering here to onboard Tessera. The data for the transactions is sent via an off-chain mutual TLS protected transport, and previously we had been using Trust on First Use (TOFU) verification of those TLS certificates auto-generated by Constellation. When we migrate you to Tessera we move over to CA verification - automatically issuing all certificates for an environment centrally, with a dedicated signing certificate for each environment.
We'll be discussing other approaches to privacy such as zero-knowledge proofs in upcoming blogs
So now we arrive in the engine room of the solution, the core protocol layer where a distributed network of nodes independently managed by different members of the network participate in consensus and add blocks to the chain.
Our specialist protocol engineering team have been hard at work here too, providing updates and contributing fixes back to the OSS communities, while keeping the thousands of chains we are custodians for humming along. Our curated forks are at the time of writing at 1.8.27 of go-ethereum (Geth) and 2.2.3 of Quorum. Stay tuned for an automated fork to Constantinople, using the procedures we pioneered on our journey from 1.7 to 1.8.
The new exciting news though, is the addition of a brand new protocol option!
The Kaleido and PegaSys dev teams have been actively collaborating since the very beginning of the Kaleido project. PegaSys engineering are hugely active members of the Ethereum community in both the public Ethereum community as one of the core teams implementing Ethereum 2.0, and the Enterprise Ethereum Alliance (EEA) as one of the vendors of Enterprise Ethereum client implementation compliant to EEA client spec.
What's special about PegaSys Pantheon is that it has been built from scratch (in Java), with a modular architecture, and Enterprise use cases in mind. This positions the team for exciting innovations tailored to the Enterprise.
Proven to be fully main-net compliant giving you the confidence it's able to cope with any EVM byte code you can throw at it, and available with a choice of Istanbul Byzantine Fault Tolerance (IBFT) or Clique PoA consensus algorithms, give Pantheon 1.1.1 a try today!
Also a shout out to the PegaSys team for creating ethsigner, which was crucial to on-boarding the Pantheon client into Kaleido, and provides the foundation for our Kaleido managed wallet.
Great work team PegaSys!
Wait... there's more???
You will find that in a modern business network, even the Blockchain needs to sit on foundations of data, and maybe most importantly identity. While public chains might be trust-less, private chains are all about knowing who you are trusting, and what data you are sharing with them.
Attempting to store large amounts of data on a shared ledger is impractical and inefficient. Forming permissioned consensus with finality across many independently managed parties, with ordered execution of smart contract logic, is significantly more expensive than writing data to a disk or a centralized database.
Then you need to consider that data on the chain itself is persisted indefinitely. How big will your chain be after one year? Two years?
Some data cannot be persisted indefinitely for regulatory reasons, at least not without additional layers of cryptography to allow cryptographic deletion.
So the many business networks manage the bulk of data off-chain, and then pin that data to the chain using a hash or other unique identifier in transactions and state.
There are numerous ways to store and distribute off-chain data. We talked about using a document store in the application layer, and sometimes the data is independently managed by each member of the network in their own core systems of record and local databases.
A popular additional option to partner with blockchain, is a decentralized peer-to-peer file-sharing technology - such as the Inter-Planetary File-System (IPFS). These have almost as much network, orchestration and permissioning complexity to set up as the blockchain itself! So, as you would expect Kaleido provides this layer managed for you too.
It's fitting to end our story of a year of engineering, with our work the most important foundational concept of permissioned chains - organizational identity. This is maybe the area where there is the most difference between the usage of the Ethereum protocol in the main-net, and in private permissioned networks.
It might be counter-intuitive, but in enterprise use cases the individual identity is often less important than the organizational identity. Individual employee identity and access management matters a huge amount within the existing IT security infrastructure of an individual participant... but that's a solved problem. It's not always very interesting to the other members of the business network how it was solved, or even who the individual is.
It simply matters that the organization decided to sign a transaction.
This area where our head of protocol Jim Zhang has been very active in the Enterprise Ethereum Alliance (EEA), and I encourage you to check out our protocol engineering team's work on pinning an external PKI established identity to an on-chain identity that backs the membership identity system of the Kaleido product.
As with every feature in Kaleido we give you an easy button to give the system a spin. With a click of a button we generate you a self-signed on Kaleido identity. Then I suggest you try out the app 2 app messaging system, to see it in action proving the identity owning multiple separate signing keys for the individual destinations. Then once you get out of Proof of Concept into Pilot and Production phases, members can jump through the hoops with their security team to get a full identity backed by the PKI trust chain of their organization.
I hope you've enjoyed this tour of our year of engineering in Kaleido, and technical overview of some of our most recent announcements. It's been an exciting year for our team being a technology partner for some of the most interesting production projects delivering value today, and capturing that experience into a full-stack of technology.
We hope you'll find the Kaleido B2B Tech Stack is the accelerator your business network needs to capitalize on the potential of Blockchain.
Want to hear a bit more? Then I suggest you check out our main stage session from Consensus 2019 from our CEO Steve Cerveny, along with Joseph Lubin (who needs no introduction), and a panel of some of our production customers: Komgo (Souleima Baddi - CEO); Project I2I (Justo Ortiz - Chairman, Union Bank of Philippines); Greenfence Consumer (Jonas Hudson - Co-Founder).
Not used the platform before? No problem, Kaleido has a Starter Plan where you can get started in minutes and the majority of features I've described can be tried for free.
Your guide to everything from asset tokenization to zero knowledge proofs
Download Now
Learn how Swift, the world’s leading provider of secure financial messaging services, utilizes Kaleido in its CBDC Sandbox project.
Download Now
