From Part 3...
Wow it's been just over two months since my last blog post in this series and it seems like a year has passed. During this time I've been anxious to delve into the Theta blockchain's off-chain micropayments feature.
While other blockchains handle this type of feature by utilizing "side-chain" approaches like the lightning network, Theta implements it in the core protocol so this simplifies the number of systems you will likely need to integrate with. In addition to this, the feature is optimize for distributed content distribution where there might be several "providers" of the same data (likely identical copies of a stream segment of a popular video). If we simplify this to one provider and one consumer, the mechanism still retains the merit of protecting the provider's ability to get paid while not incurring lengthy(~15-20 second) on-chain settlement times. Enterprises can benefit from this quality even when other forms of settlement are available. We'll get into the details here later in this post.
When I was looking to explore the off-chain micropayments feature, I first read up on the white-paper. When you're doing this, try to first put yourself in the mind of the author. What is important to them and what use-case are they addressing. Understand their context before trying to apply that understanding to your own. Naturally I looked around at the documentation for an API I could use to test this out. Upon digging, I realized that the published NodeJS API didn't cover this feature. The ThetaLabs team publishes the Theta blockchain(ledger) node code as an open source project. The source code is written in the Go programming language. Included in the source is an implementation of a command line program for interacting with the blockchain. Digging into that I realized that it also was lacking in implementation of the off-chain micropayment features, but seemed further along than the NodeJS code. BTW, unless a javascript library is obfuscated, you can always look at it. It is after all an interpreted language.
Should I wait for the ThetaLabs folks to publish complete versions of their APIs? I didn't want to put to much time into it myself if they were on the brink of releasing what I needed, so a call was in order.
I reached out to their lead developer and discussed my intentions. Incidentally, my interest in their project was picked up in one of their posts that came out a few days later. He was going to take my request back to his team to get an idea of when these features might see release. In the meantime things seemingly got very busy for them. They introduced a major update to their platform, teamed up with a premier talent agency, announced more Etherium smart contract compatibility, were awarded key DRM patents, launched an NFT(Non-Fungible Token) ecosystem, another key update, and now another governance token(TDrop). So I can't say that I wasn't surprised when I hadn't heard back from them(I also told them not to worry about getting back to me in subsequent emails). Another friendly follow up call confirmed this. The development is has their plate full and my humble needs were in the pile, just not near to top.
This sort of dilemma isn't that uncommon in the development world. It used to be that you'd be stuck at this point waiting for a vendor to put out a new release an hoping your issues/feature requests would be addressed. But as I already hinted, you can look directly into the NodeJS code and the Go project was already open source. Sometimes you have to take more on that you initially expected. The key point here is to assess really how stuck are you. If I have access to the code and a good debugger than I'm not really that stuck at all. I've always felt the debuggers are often underutilized when it comes to understanding someone else's code when you don't have access to them. This was the situation I faced and I really wanted to wrap of this blog series.
The GoLang command line was closer to what I needed but that presented a different dilemma to me. SAP doesn't consider Go as a primarily supported language so I can't expect that an SAP developer coming to this topic to understand the pertinent points without getting confused by the syntax of an unfamiliar language. I'm committed to providing an actionable working code sample wth my blog posts. The ThetaLabs provided NodeJS library would be more preferable, but after a few days, I couldn't get the encoding and signing of the payloads of the payment requests to work properly. I switched my efforts to the Go code, and was able to enhance the command line tool to implement the key calls by doing a bit of cut/paste/refactor magic and studying(by debugging) it enough to fill in the gaps. Since the project is open source I was able to fork it and freely modify it to my needs. I contributed a PR(pull request) of my enhancements to the team but as of this writing, my contribution hasn't been merged into the mainline code. If you want to follow what I've done for the command line tool, then you may need to fork the project and apply my pull request yourself.
If you look at my PR you'll see a single batch file(test/batch.sh) that tests the functionality that I added to the cli and does some performance tests as well. At first I was just going to publish those results in this series and be done with it. However, I felt that would be an injustice to the SAP developer community. While Go is a supported language in BTP via the cloud foundry(CF) community buildpack mechanism and since you can put literally anything you want into a docker container and launch it into the Kyma(Kubernetes) environment, I settled on providing an example module written in Go that can easily be added as a starting point to a BTP cloud foundry Multi-Target Application(MTA).
This is actually one of the side-benefits of following a micro-services approach to your development strategy. There is nothing that prevents you from implementing different modules in different languages. You can isolate a part of your solution that needs to scale out or up with the language that make the most sense for that purpose. In my case here, it was easier to break out enough of the theta ledger source code and package it as independent libraries so that I could build a Go module that relies on them without having to bring along the entire project.
Grab the latest version of the sample code project and switch to the v4.0 branch.
You'll find the steps needed to build and deploy this project to SAP BTP in the COMMANDS.md file.
Once deployed, you should see these four modules running.
The NodeJS module called theta-trustee you'll notice from prior posts contains a few non-functional links. This is my work-in-progress. Go to the URL of the AppRouter. In this case it's
You'll know from my examples that I'm not into fancy UI's in my sample code. I don't want the ui code or some fancy framework to distract from the core code points I'm trying to illustrate. You can adapt the code to your own project situation as needed.
Click on the offchain link that is handled by the Go module.
Click on the acct link. The index and form links are from where I was trying to make sure I had a functional go app before getting into the Theta stuff (crawl, walk, run). This will do an account query against the blockchain of a hard-coded account address that already exists and already has ample Theta and Theta Fuel (TFuel) tokens. By default we are talking to our own private node, not the public mainnet. If you wanted to test this on the public mainnet, you'd need an account that's funded with real TFuel tokens purchased with real money(fiat currency).
You should see this very plain looking response. This give the current status and balance of the account with has a wallet address of xxx. The thing to note is that the reserve_funds array is currently empty. Click on the link to create one. A Reserve Fund is like setting aside some money for a particular purpose only that a reserve requires you set aside a bit more than 2 times the amount you plan on spending. This is to twart what's know as double-spend. I won't go into this here. The details can be found in the white-paper.
If you're curious as to what's going on with your privatenet note, you can ssh into it and watch the logs. Once connected, attach to the existing screen session with this.
Back to the creating of our Reserve Fund. You should now get this response. Note that there isn't much of interest here. The hash is the identifier for the creation of the Reserve Fund on the account and the Height is the current block height where this transaction exists. Note that in blockchain terminology the block height is the size of the blockchain which is always growing at slightly different rates. Sometimes things happen at a future time when a particular blockchain height is reached. This is the case of the Reserve Fund. If the you haven't settled your transactions against it by the time(blockheight) is reached, it expires. The value represented by your transactions are lost. Also note that it took about 15 seconds for the creation to occur. Click the account link to query the account again.
Now there is more to see. The reserved_funds array is filled(you can have more than one going at once) and we have more details. The collateral is what you can loose if you misbehave, the inital_fund is the limit of what can be spent and the end_block_height is when the reserve is closed automatically. There are currently no transfer_records(transactions or Service Payments in Theta speak). The TFuel amounts are expressed in WEI which is the smallest unit of division. 1^18 to the token. Click the off-chain payment link to create one.
Now we can see a preview of what the payment will look like. There is a fee section that shows that it will cost 0.3TFuel (currently) to process this payment but it isn't levied until an on-chain payment is submitted. There is a source section with the address of the sender and the amount of TFuel to be sent. Also a target section showing the address of the wallet where the funds will go. The payment_sequence, reserve_sequence, and resource_id determine which Reserve Fund will be used and if the request is in the proper sequence. Notice that it took almost no time to generate the off-chain payment request. That's because the only thing done was the calculation of the source signature. This is like making out a check and signing the front of it and giving it to someone else for payment. It's not until the back of the check is endorsed and you present it at the bank that you get the funds it represents.
While it's not really shown here, in a real application a mechanism for conveying a price, agreeing on a price, receiving the item of value, and sending the signed off-chain payment for it is dependent upon the particulars of the implementation. All that is critical is that the receiver of the payment present it for on-chain settlement by keeping the details of it and the signature unchanged and signing it (endorsing) for on-chain settlement.
The receiver doesn't have to "bank" the payment right away. In fact, it's in the receiver's interest that if he's expecting additional payment to hold on to them until he gets the last one he expects. Unlike real checks, the amount on the face is the accumulated value since the Reserve Fund was created. This way he only has to redeem the last payment on-chain to realize the full value. In fact there are protection mechanisms built into the blockchain that prevents multiple payments once one has been processed. Each on-chain payment will cost him 0.3 TFuel deducted from the amount in the payment so it makes sense to do as few as possible (within the lifespan of the Reserve Fund).
Try clicking on the second link a couple times. You'll see that the generated signature changes each time since the details are changing. Again, this happens almost instantly.
Now click on the first long link. This will submit the payment as an on-chain micropayment. It will take again about 15 seconds to return. Click on the acct link to again query the account.
Now we see that the reserve_fund has a transfer_record with the details of which account it was sent from and which to and the detail of which fund was used and the fees incurred. Also the used_fund section accumulates what's been paid out. At this point the receiver will have the payment minus the fee in their account.
Circling back to the point of all this.
While I've made a lot of assumptions to keep the example small and represent both sides of the transaction, the real beauty of the off-chain micropayments feature is that many small signed payments can be issued with nearly no latency. It's up to the receiver to decide how often to "redeem" them at a later time on-chain at a cost of the fee. This way the overhead of settling doesn't get in the way of facilitating the delivering of value in the solution and the paying for that value.
I don't think this can be overstated. Let me explain with another example. Say you have a habit of glancing at the story headlines of the Wall Street Journal online. You occasionally start reading an article that you think is interesting but if the first few lines don't catch you attention so you move on. When you do find an article that you want to read in full, you scroll down and get hit with the "your not subscribed" pop-up message. Now most of us will just abandon the article instead of whipping out our credit card and signing up for some monthly charge that we will forget about ever having approved and feel bad that we don't make enough use of it every time we see the charge on our credit card bill.
What if when you scroll down in that article it just silently charged your Theta account 0.1 TFuel (0.03 USD as of this writing) per page of article. You finish the 5 page article and it's charged you 15 cents without bothering you about it. You feel informed. The WSJ has been paid for something they wouldn't likely have been. Everyone is happy and the experience is convenient.
What about charging by API call, by volume of data returned. What about charging for the accuracy of a sales forecast. What about charging only when the value provided is useful, creates a net profit, or just makes you feel good. What about charging only for jokes that make you laugh? OK, that last one was a bit silly, but the point is to start thinking outside the box on how we think about charging for the value we as enterprise solution providers create with our products and services.
I'm going to extend this feature in my example to cover an SAP S/4 Hana Financial enterprise integration in the next blog post. I'll keep the details a secret for now, but as a teaser I'll leave you with this hint.
Let me know if you have and questions or issues by leaving me a question below or better yet, asking it on the SAP community.
-Andrew
I’ll re-commit to my promise of demonstrating the very intriguing off-chain micropayments feature of the Theta blockchain in my next installment of this blog series.
Wow it's been just over two months since my last blog post in this series and it seems like a year has passed. During this time I've been anxious to delve into the Theta blockchain's off-chain micropayments feature.
While other blockchains handle this type of feature by utilizing "side-chain" approaches like the lightning network, Theta implements it in the core protocol so this simplifies the number of systems you will likely need to integrate with. In addition to this, the feature is optimize for distributed content distribution where there might be several "providers" of the same data (likely identical copies of a stream segment of a popular video). If we simplify this to one provider and one consumer, the mechanism still retains the merit of protecting the provider's ability to get paid while not incurring lengthy(~15-20 second) on-chain settlement times. Enterprises can benefit from this quality even when other forms of settlement are available. We'll get into the details here later in this post.
Wait or forge ahead?
When I was looking to explore the off-chain micropayments feature, I first read up on the white-paper. When you're doing this, try to first put yourself in the mind of the author. What is important to them and what use-case are they addressing. Understand their context before trying to apply that understanding to your own. Naturally I looked around at the documentation for an API I could use to test this out. Upon digging, I realized that the published NodeJS API didn't cover this feature. The ThetaLabs team publishes the Theta blockchain(ledger) node code as an open source project. The source code is written in the Go programming language. Included in the source is an implementation of a command line program for interacting with the blockchain. Digging into that I realized that it also was lacking in implementation of the off-chain micropayment features, but seemed further along than the NodeJS code. BTW, unless a javascript library is obfuscated, you can always look at it. It is after all an interpreted language.
Should I wait for the ThetaLabs folks to publish complete versions of their APIs? I didn't want to put to much time into it myself if they were on the brink of releasing what I needed, so a call was in order.
I reached out to their lead developer and discussed my intentions. Incidentally, my interest in their project was picked up in one of their posts that came out a few days later. He was going to take my request back to his team to get an idea of when these features might see release. In the meantime things seemingly got very busy for them. They introduced a major update to their platform, teamed up with a premier talent agency, announced more Etherium smart contract compatibility, were awarded key DRM patents, launched an NFT(Non-Fungible Token) ecosystem, another key update, and now another governance token(TDrop). So I can't say that I wasn't surprised when I hadn't heard back from them(I also told them not to worry about getting back to me in subsequent emails). Another friendly follow up call confirmed this. The development is has their plate full and my humble needs were in the pile, just not near to top.
This sort of dilemma isn't that uncommon in the development world. It used to be that you'd be stuck at this point waiting for a vendor to put out a new release an hoping your issues/feature requests would be addressed. But as I already hinted, you can look directly into the NodeJS code and the Go project was already open source. Sometimes you have to take more on that you initially expected. The key point here is to assess really how stuck are you. If I have access to the code and a good debugger than I'm not really that stuck at all. I've always felt the debuggers are often underutilized when it comes to understanding someone else's code when you don't have access to them. This was the situation I faced and I really wanted to wrap of this blog series.
The GoLang command line was closer to what I needed but that presented a different dilemma to me. SAP doesn't consider Go as a primarily supported language so I can't expect that an SAP developer coming to this topic to understand the pertinent points without getting confused by the syntax of an unfamiliar language. I'm committed to providing an actionable working code sample wth my blog posts. The ThetaLabs provided NodeJS library would be more preferable, but after a few days, I couldn't get the encoding and signing of the payloads of the payment requests to work properly. I switched my efforts to the Go code, and was able to enhance the command line tool to implement the key calls by doing a bit of cut/paste/refactor magic and studying(by debugging) it enough to fill in the gaps. Since the project is open source I was able to fork it and freely modify it to my needs. I contributed a PR(pull request) of my enhancements to the team but as of this writing, my contribution hasn't been merged into the mainline code. If you want to follow what I've done for the command line tool, then you may need to fork the project and apply my pull request yourself.
Proving it works, not enough.
If you look at my PR you'll see a single batch file(test/batch.sh) that tests the functionality that I added to the cli and does some performance tests as well. At first I was just going to publish those results in this series and be done with it. However, I felt that would be an injustice to the SAP developer community. While Go is a supported language in BTP via the cloud foundry(CF) community buildpack mechanism and since you can put literally anything you want into a docker container and launch it into the Kyma(Kubernetes) environment, I settled on providing an example module written in Go that can easily be added as a starting point to a BTP cloud foundry Multi-Target Application(MTA).
This is actually one of the side-benefits of following a micro-services approach to your development strategy. There is nothing that prevents you from implementing different modules in different languages. You can isolate a part of your solution that needs to scale out or up with the language that make the most sense for that purpose. In my case here, it was easier to break out enough of the theta ledger source code and package it as independent libraries so that I could build a Go module that relies on them without having to bring along the entire project.
NOTE: My intention here is to not provide a long term Go library for these features. The implementation is rough and not intended for any sort of production use. The code does not do a lot of error checking or query parsing for security purposes. It's intended to prove out the viability of the off-chain micropayment features in an enterprise solution environment. In time ThetaLabs will replace the function of these libs with their own and you should always use official sources when possible.
Get on with it already!
Grab the latest version of the sample code project and switch to the v4.0 branch.
git clone https://github.com/SAP-samples/btp-blockchain-theta.git
git checkout v4.0You'll find the steps needed to build and deploy this project to SAP BTP in the COMMANDS.md file.
Once deployed, you should see these four modules running.
The NodeJS module called theta-trustee you'll notice from prior posts contains a few non-functional links. This is my work-in-progress. Go to the URL of the AppRouter. In this case it's
theta-dev-theta-app.cfapps.eu10.hana.ondemand.com
You'll know from my examples that I'm not into fancy UI's in my sample code. I don't want the ui code or some fancy framework to distract from the core code points I'm trying to illustrate. You can adapt the code to your own project situation as needed.
Click on the offchain link that is handled by the Go module.
Click on the acct link. The index and form links are from where I was trying to make sure I had a functional go app before getting into the Theta stuff (crawl, walk, run). This will do an account query against the blockchain of a hard-coded account address that already exists and already has ample Theta and Theta Fuel (TFuel) tokens. By default we are talking to our own private node, not the public mainnet. If you wanted to test this on the public mainnet, you'd need an account that's funded with real TFuel tokens purchased with real money(fiat currency).
You should see this very plain looking response. This give the current status and balance of the account with has a wallet address of xxx. The thing to note is that the reserve_funds array is currently empty. Click on the link to create one. A Reserve Fund is like setting aside some money for a particular purpose only that a reserve requires you set aside a bit more than 2 times the amount you plan on spending. This is to twart what's know as double-spend. I won't go into this here. The details can be found in the white-paper.
If you're curious as to what's going on with your privatenet note, you can ssh into it and watch the logs. Once connected, attach to the existing screen session with this.
screen -x thetaBack to the creating of our Reserve Fund. You should now get this response. Note that there isn't much of interest here. The hash is the identifier for the creation of the Reserve Fund on the account and the Height is the current block height where this transaction exists. Note that in blockchain terminology the block height is the size of the blockchain which is always growing at slightly different rates. Sometimes things happen at a future time when a particular blockchain height is reached. This is the case of the Reserve Fund. If the you haven't settled your transactions against it by the time(blockheight) is reached, it expires. The value represented by your transactions are lost. Also note that it took about 15 seconds for the creation to occur. Click the account link to query the account again.
Now there is more to see. The reserved_funds array is filled(you can have more than one going at once) and we have more details. The collateral is what you can loose if you misbehave, the inital_fund is the limit of what can be spent and the end_block_height is when the reserve is closed automatically. There are currently no transfer_records(transactions or Service Payments in Theta speak). The TFuel amounts are expressed in WEI which is the smallest unit of division. 1^18 to the token. Click the off-chain payment link to create one.
Now we can see a preview of what the payment will look like. There is a fee section that shows that it will cost 0.3TFuel (currently) to process this payment but it isn't levied until an on-chain payment is submitted. There is a source section with the address of the sender and the amount of TFuel to be sent. Also a target section showing the address of the wallet where the funds will go. The payment_sequence, reserve_sequence, and resource_id determine which Reserve Fund will be used and if the request is in the proper sequence. Notice that it took almost no time to generate the off-chain payment request. That's because the only thing done was the calculation of the source signature. This is like making out a check and signing the front of it and giving it to someone else for payment. It's not until the back of the check is endorsed and you present it at the bank that you get the funds it represents.
While it's not really shown here, in a real application a mechanism for conveying a price, agreeing on a price, receiving the item of value, and sending the signed off-chain payment for it is dependent upon the particulars of the implementation. All that is critical is that the receiver of the payment present it for on-chain settlement by keeping the details of it and the signature unchanged and signing it (endorsing) for on-chain settlement.
The receiver doesn't have to "bank" the payment right away. In fact, it's in the receiver's interest that if he's expecting additional payment to hold on to them until he gets the last one he expects. Unlike real checks, the amount on the face is the accumulated value since the Reserve Fund was created. This way he only has to redeem the last payment on-chain to realize the full value. In fact there are protection mechanisms built into the blockchain that prevents multiple payments once one has been processed. Each on-chain payment will cost him 0.3 TFuel deducted from the amount in the payment so it makes sense to do as few as possible (within the lifespan of the Reserve Fund).
Try clicking on the second link a couple times. You'll see that the generated signature changes each time since the details are changing. Again, this happens almost instantly.
Now click on the first long link. This will submit the payment as an on-chain micropayment. It will take again about 15 seconds to return. Click on the acct link to again query the account.
Now we see that the reserve_fund has a transfer_record with the details of which account it was sent from and which to and the detail of which fund was used and the fees incurred. Also the used_fund section accumulates what's been paid out. At this point the receiver will have the payment minus the fee in their account.
Circling back to the point of all this.
While I've made a lot of assumptions to keep the example small and represent both sides of the transaction, the real beauty of the off-chain micropayments feature is that many small signed payments can be issued with nearly no latency. It's up to the receiver to decide how often to "redeem" them at a later time on-chain at a cost of the fee. This way the overhead of settling doesn't get in the way of facilitating the delivering of value in the solution and the paying for that value.
I don't think this can be overstated. Let me explain with another example. Say you have a habit of glancing at the story headlines of the Wall Street Journal online. You occasionally start reading an article that you think is interesting but if the first few lines don't catch you attention so you move on. When you do find an article that you want to read in full, you scroll down and get hit with the "your not subscribed" pop-up message. Now most of us will just abandon the article instead of whipping out our credit card and signing up for some monthly charge that we will forget about ever having approved and feel bad that we don't make enough use of it every time we see the charge on our credit card bill.
What if when you scroll down in that article it just silently charged your Theta account 0.1 TFuel (0.03 USD as of this writing) per page of article. You finish the 5 page article and it's charged you 15 cents without bothering you about it. You feel informed. The WSJ has been paid for something they wouldn't likely have been. Everyone is happy and the experience is convenient.
What about charging by API call, by volume of data returned. What about charging for the accuracy of a sales forecast. What about charging only when the value provided is useful, creates a net profit, or just makes you feel good. What about charging only for jokes that make you laugh? OK, that last one was a bit silly, but the point is to start thinking outside the box on how we think about charging for the value we as enterprise solution providers create with our products and services.
For next time.
I'm going to extend this feature in my example to cover an SAP S/4 Hana Financial enterprise integration in the next blog post. I'll keep the details a secret for now, but as a teaser I'll leave you with this hint.
"What is MicroStrategy leading the corporate world in doing?"
Let me know if you have and questions or issues by leaving me a question below or better yet, asking it on the SAP community.
-Andrew
| Partners: If you have a question, click here to ask it in the SAP Community . Be sure to tag it with Partnership and leave your company name in the question so that we can better assist you. |
- SAP Managed Tags:
2 Comments
