I just started my career at SAP but have had extensive exposure to various emerging technologies in my university career and see a desire within SAP and it's customers to better understand these technologies and how they might be optimally implemented. To that end, I hope this blog post can bring some clarity into the complex world of blockchain.
Putting aside all the hype surrounding blockchains or distributed ledgers, the most important question to ask when deciding what technology to use for a certain facet of an application, whether we are talking about a database, a programming language or a platform, is: Why this technology and not something else? Merely selecting something because you haven’t researched alternatives or you just assume it’s the best option based on the amount of media coverage it has received recently will inevitably lead to poorly designed and poorly optimized final products. And I know SAP does not want to deliver poorly designed or poorly optimized products. In that vein, I will try to examine the weaknesses and strengths of distributed ledger technology and come to what can be called a scope of sensible implementations for distributed ledger technology in an enterprise context.
NOTE: I will be referring to blockchains as distributed ledgers throughout this post to assist in differentiating from cryptocurrency implementations. This post also assumes a basic understanding of some keyterms used in the blockchain/distributed ledger sphere. If this is the first article you are reading about blockchains/distributed ledgers I recommend you keep http://www.blockchaintechnologies.com/blockchain-glossary open.
What is a Distributed Ledger?
Let’s start with a definition of a distributed ledger (DL). A DL is a consensus of information and the history of that information that has been wholly distributed between all partaking parties without a central administrator or central data storage. A DL does not require a blockchain specifically, but blockchains are the most functional implementation as of now.
As we are interested in an enterprise context, let us focus on a permissioned distributed ledger. Permissioned here refers to the fact that the blockchain is not public and only shared between selected parties, no doubt a prerequisite for any company that wishes to use blockchains for data storage and doesn’t want unknown actors connecting to and hijacking their network. To achieve true consensus without a central authority, it is thus required that each transaction is verified as being valid given the current world state of assets through a consensus method, usually requiring an endorsement by at least a majority of all partaking nodes. Every transaction proposal and verification step is also signed to guarantee actor authenticity. After verification, this transaction is ordered along with other verified and signed transactions and added as a packaged block to the blockchain. The transactions and blocks are also hashed during these processes, usually following a merkle tree (https://brilliant.org/wiki/merkle-tree/). I’ve deliberately attempted to keep the language here vague as the specifics of the permissioning, verification, consensus, ordering and broadcasting methods will naturally vary from implementation to implementation but the underlying concepts are fairly static. A clean graphic showing the typical flow of steps taken to create a transaction is depicted in the graphic below, taken from the Hyperledger Fabric documentation. All further discussion will loosely use Hyperledger Fabric as a point of reference as this is the implementation SAP has chosen to base their solution on. Those interested in the exact architecture of Hyperledger Fabric should take some time to read through their documentation: https://hyperledger-fabric.readthedocs.io/en/release/arch-deep-dive.html
Source: http://hyperledger-fabric.readthedocs.io/en/release/txflow.html
Strengths and Weaknesses
So, what properties does this provide a DL? Let’s start with the weaknesses. DLs possess multiple weaknesses including their difficulty to be implemented, their complexity, their low transaction throughput, and the difficulty to change their structure/rules, but I find the primary weakness that must be addressed is performance. I’m going to posit that a DL will always perform worse than a traditional distributed database (I say distributed database because if you are attempting to use a DL to replace a single-node central database, you have some fundamental misconceptions). This is due to the fact that practically all popular distributed databases (Cassandra, CouchDB, Datomic) use some kind of replication/broadcasting method to distribute data which assumes that the data in the originating database is correct and can be simply broadcast to all other nodes in the network. A DL on the other hand requires verification from at least a majority of all other nodes before broadcasting. Furthermore, during this verification of the transaction, all nodes are checking the validity of the signatures of all packets they receive and once a transaction has been verified by at least a majority or a set threshold (and probably checked by even more nodes), it is sent to be ordered where it waits until the designated transaction amount for a block has been gathered so a new block can be created and broadcast back to all nodes in the network. All of this work needs to be done on top of the “classical DB operation”. Thus, due to the nature of its design, it is inevitable that a DL will perform worse than a traditional update & broadcast distributed DB schema, with the added effect that as the network and number of verifying nodes grows, the performance will DECREASE. Effectively, we’re talking about negative network scalability which doesn’t exactly provide a good advertisement slogan.
If a DL has such performance issues, why use it at all? Because it provides features that do not exist in any other distributed database technology. Since every transaction has to be verified by a majority of endorsers, it is impossible to push fraudulent transactions onto a network where a party does not control more than a majority of the endorsing nodes. Since the hashing method is so strong, it becomes neigh impossible to alter transactions that have already been written to the blockchain, and even if you recalculate all the hashes, you would have to supply the correct signatures and replace a majority of the existing blockchains with your fake. Since smart contracts are programmed to auto-execute and are pre-determined and agreed upon by the involved parties before being stored on the blockchain, all internal transaction logic is also effectively immutable and isolated against malicious tampering. Since the control of read/write access granting and addition of new nodes to the network is truly decentralized and succeeds democratically, no party can flood the system with nodes to achieve a majority, allowing them to circumvent the previously mentioned guards. As should be obvious, the common thread through all of these capabilities that no other distributed database can perform is trust. Distributed ledgers are only of use in an environment where multiple parties exist and there is less than complete trust between parties. If any party has an incentive to alter transactions/database entries or the scope of the application requires absolute data security against malicious tampering, both from within and from without the network, a DL could be a good fit. This is also the primary selling point of DLs and should not be seen as trivial. DLs solve a problem that no previous database could solve and thus open up a new realm of business solutions that simply could not have existed before.
Smart Contracts
The other primary use-case often touted for DL technology is smart contracts. As previously stated, smart contracts allow you to submit immutable, ready-to-execute database updates that run automatically when the desired conditions are met. Thus, in my opinion, they can be split into two primary categories: simple and complex. Simple smart contracts are those that solely use information already present on the DL and are a fantastic resource to automate and safeguard simple processes. Some examples from the world of finance would be taking a percentage of every transaction to send to tax authorities, keeping funds in a timelock, or lotteries. Complex smart contracts are those that require information currently not present on the DL. Examples would include paying out insurance if a crop yield was destroyed due to weather, offering kickbacks or discounts based on tested product quality, or automatic execution of legal contracts after terms have been fulfilled. This type of smart contract is slightly trickier as it requires oracles where users can submit information not already present on the DL, onto the DL. This still has the perceived weakness of being susceptible to incorrect information being submitted. Whether IoT sensors are used, external DBs are queried, or data is submitted manually, there always exists the possibility for incorrect or falsified data to be entered. This problem exists with all DBs though and will never be solved unless a group of mutually independent auditors watch and verify every single piece of information submitted, which, in my opinion, will never happen. Despite this, DLs provide us with the next best safeguards, which are democratized data input and traceable, immutable data. If information being added can be obtained from multiple sources with equal authority, it becomes exceedingly difficult for incorrect information to be accepted. And even if a single point of information entry exists, a pointed discouragement for submitting falsified information is present as all information accepted by a well-designed DL is immutable. If this information is found to be falsified, and it most likely will be discovered as we are generally operating in an environment with only semi-trusted parties, all subsequent legal processes should be trivial and the malicious parties would have their reputation ruined permanently.
Looking Forward
It is imperative to closely examine all submitted blockchain/DL use-cases and ensure that the problem of trust is a central one. Without this aspect, one of the other distributed database implementations such as CouchDB, Cassandra, Datomic, a DB owned by a trusted authority, or just simply reconciling data between non-connected company DBs will surely be better. Between them, it should be fairly simple to find one that fulfills all of the criteria and with the added bonus of being much more performant than any DL.
I hope that this post has been able to scrub away some of the muddled unclarities surrounding blockchain and assist in the formation of a more sharply defined mindset that can be used when thinking of DL solutions. It is imperative to have a clean and clear idea of what a technology’s strengths and weaknesses are to be able to recognize excellent use cases. Keep these in mind and I am certain your next project will be a success.
Sources/Relevant Links:
http://hyperledger-fabric.readthedocs.io/en/release/
http://couchdb.apache.org/
http://cassandra.apache.org/
http://www.datomic.com/
Putting aside all the hype surrounding blockchains or distributed ledgers, the most important question to ask when deciding what technology to use for a certain facet of an application, whether we are talking about a database, a programming language or a platform, is: Why this technology and not something else? Merely selecting something because you haven’t researched alternatives or you just assume it’s the best option based on the amount of media coverage it has received recently will inevitably lead to poorly designed and poorly optimized final products. And I know SAP does not want to deliver poorly designed or poorly optimized products. In that vein, I will try to examine the weaknesses and strengths of distributed ledger technology and come to what can be called a scope of sensible implementations for distributed ledger technology in an enterprise context.
NOTE: I will be referring to blockchains as distributed ledgers throughout this post to assist in differentiating from cryptocurrency implementations. This post also assumes a basic understanding of some keyterms used in the blockchain/distributed ledger sphere. If this is the first article you are reading about blockchains/distributed ledgers I recommend you keep http://www.blockchaintechnologies.com/blockchain-glossary open.
What is a Distributed Ledger?
Let’s start with a definition of a distributed ledger (DL). A DL is a consensus of information and the history of that information that has been wholly distributed between all partaking parties without a central administrator or central data storage. A DL does not require a blockchain specifically, but blockchains are the most functional implementation as of now.
As we are interested in an enterprise context, let us focus on a permissioned distributed ledger. Permissioned here refers to the fact that the blockchain is not public and only shared between selected parties, no doubt a prerequisite for any company that wishes to use blockchains for data storage and doesn’t want unknown actors connecting to and hijacking their network. To achieve true consensus without a central authority, it is thus required that each transaction is verified as being valid given the current world state of assets through a consensus method, usually requiring an endorsement by at least a majority of all partaking nodes. Every transaction proposal and verification step is also signed to guarantee actor authenticity. After verification, this transaction is ordered along with other verified and signed transactions and added as a packaged block to the blockchain. The transactions and blocks are also hashed during these processes, usually following a merkle tree (https://brilliant.org/wiki/merkle-tree/). I’ve deliberately attempted to keep the language here vague as the specifics of the permissioning, verification, consensus, ordering and broadcasting methods will naturally vary from implementation to implementation but the underlying concepts are fairly static. A clean graphic showing the typical flow of steps taken to create a transaction is depicted in the graphic below, taken from the Hyperledger Fabric documentation. All further discussion will loosely use Hyperledger Fabric as a point of reference as this is the implementation SAP has chosen to base their solution on. Those interested in the exact architecture of Hyperledger Fabric should take some time to read through their documentation: https://hyperledger-fabric.readthedocs.io/en/release/arch-deep-dive.html
Source: http://hyperledger-fabric.readthedocs.io/en/release/txflow.html
Strengths and Weaknesses
So, what properties does this provide a DL? Let’s start with the weaknesses. DLs possess multiple weaknesses including their difficulty to be implemented, their complexity, their low transaction throughput, and the difficulty to change their structure/rules, but I find the primary weakness that must be addressed is performance. I’m going to posit that a DL will always perform worse than a traditional distributed database (I say distributed database because if you are attempting to use a DL to replace a single-node central database, you have some fundamental misconceptions). This is due to the fact that practically all popular distributed databases (Cassandra, CouchDB, Datomic) use some kind of replication/broadcasting method to distribute data which assumes that the data in the originating database is correct and can be simply broadcast to all other nodes in the network. A DL on the other hand requires verification from at least a majority of all other nodes before broadcasting. Furthermore, during this verification of the transaction, all nodes are checking the validity of the signatures of all packets they receive and once a transaction has been verified by at least a majority or a set threshold (and probably checked by even more nodes), it is sent to be ordered where it waits until the designated transaction amount for a block has been gathered so a new block can be created and broadcast back to all nodes in the network. All of this work needs to be done on top of the “classical DB operation”. Thus, due to the nature of its design, it is inevitable that a DL will perform worse than a traditional update & broadcast distributed DB schema, with the added effect that as the network and number of verifying nodes grows, the performance will DECREASE. Effectively, we’re talking about negative network scalability which doesn’t exactly provide a good advertisement slogan.
If a DL has such performance issues, why use it at all? Because it provides features that do not exist in any other distributed database technology. Since every transaction has to be verified by a majority of endorsers, it is impossible to push fraudulent transactions onto a network where a party does not control more than a majority of the endorsing nodes. Since the hashing method is so strong, it becomes neigh impossible to alter transactions that have already been written to the blockchain, and even if you recalculate all the hashes, you would have to supply the correct signatures and replace a majority of the existing blockchains with your fake. Since smart contracts are programmed to auto-execute and are pre-determined and agreed upon by the involved parties before being stored on the blockchain, all internal transaction logic is also effectively immutable and isolated against malicious tampering. Since the control of read/write access granting and addition of new nodes to the network is truly decentralized and succeeds democratically, no party can flood the system with nodes to achieve a majority, allowing them to circumvent the previously mentioned guards. As should be obvious, the common thread through all of these capabilities that no other distributed database can perform is trust. Distributed ledgers are only of use in an environment where multiple parties exist and there is less than complete trust between parties. If any party has an incentive to alter transactions/database entries or the scope of the application requires absolute data security against malicious tampering, both from within and from without the network, a DL could be a good fit. This is also the primary selling point of DLs and should not be seen as trivial. DLs solve a problem that no previous database could solve and thus open up a new realm of business solutions that simply could not have existed before.
Smart Contracts
The other primary use-case often touted for DL technology is smart contracts. As previously stated, smart contracts allow you to submit immutable, ready-to-execute database updates that run automatically when the desired conditions are met. Thus, in my opinion, they can be split into two primary categories: simple and complex. Simple smart contracts are those that solely use information already present on the DL and are a fantastic resource to automate and safeguard simple processes. Some examples from the world of finance would be taking a percentage of every transaction to send to tax authorities, keeping funds in a timelock, or lotteries. Complex smart contracts are those that require information currently not present on the DL. Examples would include paying out insurance if a crop yield was destroyed due to weather, offering kickbacks or discounts based on tested product quality, or automatic execution of legal contracts after terms have been fulfilled. This type of smart contract is slightly trickier as it requires oracles where users can submit information not already present on the DL, onto the DL. This still has the perceived weakness of being susceptible to incorrect information being submitted. Whether IoT sensors are used, external DBs are queried, or data is submitted manually, there always exists the possibility for incorrect or falsified data to be entered. This problem exists with all DBs though and will never be solved unless a group of mutually independent auditors watch and verify every single piece of information submitted, which, in my opinion, will never happen. Despite this, DLs provide us with the next best safeguards, which are democratized data input and traceable, immutable data. If information being added can be obtained from multiple sources with equal authority, it becomes exceedingly difficult for incorrect information to be accepted. And even if a single point of information entry exists, a pointed discouragement for submitting falsified information is present as all information accepted by a well-designed DL is immutable. If this information is found to be falsified, and it most likely will be discovered as we are generally operating in an environment with only semi-trusted parties, all subsequent legal processes should be trivial and the malicious parties would have their reputation ruined permanently.
Looking Forward
It is imperative to closely examine all submitted blockchain/DL use-cases and ensure that the problem of trust is a central one. Without this aspect, one of the other distributed database implementations such as CouchDB, Cassandra, Datomic, a DB owned by a trusted authority, or just simply reconciling data between non-connected company DBs will surely be better. Between them, it should be fairly simple to find one that fulfills all of the criteria and with the added bonus of being much more performant than any DL.
I hope that this post has been able to scrub away some of the muddled unclarities surrounding blockchain and assist in the formation of a more sharply defined mindset that can be used when thinking of DL solutions. It is imperative to have a clean and clear idea of what a technology’s strengths and weaknesses are to be able to recognize excellent use cases. Keep these in mind and I am certain your next project will be a success.
Sources/Relevant Links:
http://hyperledger-fabric.readthedocs.io/en/release/
http://couchdb.apache.org/
http://cassandra.apache.org/
http://www.datomic.com/
- SAP Managed Tags:
1 Comment
