Technical Articles
Escaping EINTEGRITY Hell
If you use NodeJS for anything in your project(and it’s hard not to) you may find yourself stuck in EINTEGRITY hell. What do I mean?
When it happens:
If you’re deploying an Multi-Target Application(MTA) to SAP Cloud Platform Cloud Foundry, you may find that it can fail if there is a mismatch in the files your sending as part of your project and the system’s attempt to pull in additional dependent files during the deployment. This can happen with full deploys where the entire project is zipped up, or partial “push” deploys of single NodeJS modules(Including the app-router, and HDI DB modules which are themselves specialized NodeJS apps).
How you know it’s happening:
You’ll usually see the module itself fail(doing a push) or the full deploy will retry starting the module 3 times before giving up. Look at the output and find which module is failing. The default output will not give you enough information to figure this out. You need to inspect the logs of the module itself.
cf logs failing-module --recent
Scroll backwards through the log output looking for the message “npm ERR! code EINTEGRITY”. Here’s some sample output.
2020-07-06T19:40:39.05-0400 [STG/0] OUT -----> Building dependencies
2020-07-06T19:40:39.05-0400 [STG/0] OUT Installing node modules (package.json + package-lock.json)
2020-07-06T19:41:00.45-0400 [STG/0] OUT npm ERR! code EINTEGRITY
2020-07-06T19:41:00.45-0400 [STG/0] OUT npm ERR! Verification failed while extracting @sap/node-jwt@1.6.11:
2020-07-06T19:41:00.45-0400 [STG/0] OUT npm ERR! Verification failed while extracting @sap/node-jwt@1.6.11:
2020-07-06T19:41:00.45-0400 [STG/0] OUT npm ERR! sha512-1WWg93DFtt+cZZCuW68LK3PxmyJesWkeMiLzLYvq+eSg0kj4PfUGEBR/oK/4LN2qdMiqtn2/6tpED/vL7REwrw== integrity checksum failed when using sha512:
wanted sha512-1WWg93DFtt+cZZCuW68LK3PxmyJesWkeMiLzLYvq+eSg0kj4PfUGEBR/oK/4LN2qdMiqtn2/6tpED/vL7REwrw==
but got sha512-Wr3ARQWQAWoZqVF3fM/s1z+B5n9Ne+4l/BBncb8OckwYB3P6t3aV0CjaA35dcLo4vLG8aT/Lf+E7c0Lga3e6KQ==. (4939364 bytes)
2020-07-06T19:41:08.13-0400 [STG/0] OUT npm ERR! A complete log of this run can be found in:
2020-07-06T19:41:08.13-0400 [STG/0] OUT npm ERR! /tmp/cache/final/.npm/_logs/2020-07-06T23_41_00_464Z-debug.log
2020-07-06T19:41:08.14-0400 [STG/0] OUT **ERROR** Unable to build dependencies: exit status 1
2020-07-06T19:41:08.43-0400 [STG/0] ERR Failed to compile droplet: Failed to run all supply scripts: exit status 14
2020-07-06T19:41:08.46-0400 [STG/0] OUT Exit status 223
Why it’s happening:
When your NodeJS file are being prepared for a deploy, the packaging tool will perform an “npm install –production” for you in each NodeJS modules to create a node_modules folder with all the dependencies(and dependencies dependencies) listed in the package.json file and an accompanying package-lock.json file that is synchronized with it. All the files are then zipped up into the .mtar file and sent for deployment.
If you run “npm install” within a single module and manually “cf push” it, the effect is similar, just that you did the “npm install” part yourself.
When the module is received by the cloud platform and deployed, the NodeJS buildpack will look at what was sent and look to see if additional dependencies are needed that weren’t sent along.
THIS IS WHERE THE PROBLEM OCCURS!
If it’s determined(and I’m not entirely sure exactly what triggers this) that the buildpack needs to check the dependencies, it will read the package-lock.json file that you sent along and do an integrity check by generating a checksum on the package(the but got checksum) you sent in node_modules OR one that it determined it needed to download from the package repo it wants to pull from and compare it against what was listed in the package-lock.json file(the wanted checksum). If it finds they don’t match, you’ll get the “npm ERR! code EINTEGRITY” error and the deploy/push will fail.
What can you do about it?
It seems that the reason this occurs is due to some hiccup that happens when the package is published to the repository. The tiniest glitch will result in a checksum mismatch.
- Wait for the owner of the package to notice the issue and re-publish the package so that the checksum is correct.
- If you are pushing the single module manually, you can edit the package-lock.json, find the package and replace the (but got checksum for the wanted checksum) and save the package-lock.json file and re-push the module.
But what if you’re trying to get the whole app initially deployed and the packaging tool keeps re-running “npm install” and putting what it thinks is the proper checksum in the package-lock.json file before whisking it away during the deploy(Case #2). You could unzip the mtar file, unzip the offending module, fix the package-lock.json file, re-zip it back up and then deploy the mtar file, but what a pain that is.
What if you’re on a deadline(or just want to move forward) and can’t wait for the package to be fixed in the repo(Case #1). What then? (Also see the addendum at the end of this post.)
What you CAN do in this situation is to tell npm on your local machine to NOT generate the package-lock.json file. If you look back at the logs, you’ll see this line.
Installing node modules (package.json + package-lock.json)
You can do this by running this command.
npm config set package-lock false
Delete the offending package-lock.json file and re-package the mtar file and deploy.
Now this time, the buildpack won’t have the package-lock.json to compare against and the side-effect will be that any dependent packages that it pulls, it will trust that the checksum is correct and allow the installation to continue.
When you do this confirm that it’s working by looking for the following in the logs.
Installing node modules (package.json)
This can get you unstuck when nothing else will.
A different kind of hell:
I’d be remiss to completely ignore a huge issue with ALL of these approaches.
In keeping with my metaphor… “Better the devil you know than the one you don’t”.
Other than Case #1 above, these approaches open up the opportunity for someone to be spoofing the npm repo that’s being used and supplying a poisoned version of the packag with malicious code.
These solutions should be TEMPORARY at best. It’s YOUR decision to use them or not but at least you know what your options are. If you do end up turning off package-lock, set a reminder in your calendar to turn it back on and try again. You may be able to notify the publisher of the package and bring the issue to their attention in order to get the repo fixed sooner, but that may not be enough for your immediate needs.
npm config set package-lock true
I hope this saves someone out there.
Addendum:
After running into this again today, I dug deeper and found another approach that may work in some circumstances. The trick here is to configure the behavior of the mbt packaging tool so as to NOT pull package dependencies or include them. For each module that is giving you issues, you can add the following in the module definition.
- name: module-app
type: nodejs
path: app
build-parameters:
ignore: ["package-lock.json", "node_modules/"]
# Suppress the calling of npm install by listing no commands
builder: custom
commands: []
This also has the side benefit of making your mtar file a LOT smaller and thus improves upload speeds. The deployer will have to pull the dependencies during the buildpack step, but hopefully the repo that it’s pulling from will not have the checksum issues described above.
-Andrew
Partners: If you have a question, click here to ask it in the SAP Community . Be sure to tag it with Partnership and leave your company name in the question so that we can better assist you. |
Great blog Andrew Lunde.,
I think I've encountered this error before. I struggled for a while, but eventually I got to the same conclusion as you.
However, I believe I've managed it differently (since I didn't quite know what was going on). I disabled the packaging of the nodes_module folder on my mtar. This forced the whole project to download the files again on CF while deploying. But I remember I also deleted the whole node_modules folder locally together with the lock file before building it again.
Once it deployed than I understood what went wrong.
Best regards,
Ivan
Ivan,
Great point.
Yes, I've had to do that as well.
The MBT tools seems to obey these build-parameters and not package the node_modules but the earlier java based tool didn't.
-Andrew
Andrew,
Thanks for this post.
It seems to be very timely given the recent availability of SAP NPM packages in the default public registry. By that, I mean I think I’ve already hit some of these integrity issues you described when I switched my local development environment to use the public registry.
The problem isn’t necessarily the local development but more when I attempt to deploy to SAP Cloud Platform Cloud Foundry. What’s happening is that I’ll have everything built successfully locally but when I try to deploy I’ll hit those integrity errors during application staging.
Is it possible that during app staging on SAP Cloud Platform Cloud Foundry, it is using the old private SAP registry and therefore introducing a high probability of integrity errors given slight variations in the files present in the public npm registry and the private sap registry?
For example, I’ve been having trouble with is the same
@sap/node-jwt
module you shared, specifically the most recent version1.6.11
. When I install the app using the private repository I find this in thepackage-lock.json
file:Yet when I install it using the new public registry it is listed as:
Note the different values for the
resolved
andintegrity
fields.For the time being, I’m just using an older version of the module (
1.6.6
) as a workaround, but I’m curious if you have any insight into how the npm registry config works in CF deployments as I feel that this is very likely the culprit.Thanks again,
Brian
Hi Brian,
How did you fix the issue with the node-jwt module? I have the same issue: everything works fine when I deploy from my laptop but it fails when I try to deploy from a GitHub action.
Andrew Lunde thanks for sharing this, as Brian mentioned this is very helpful with the transition from npm.sap.com to the public npm.
Thanks,
Pierre
I had same problem, in my case the build stopped reffering to @sap/node-jwt@^1.3.0.
I solved it pinpointing which direct-module-usage caused me the error by removing the dependencies in package.json half by half. In the end I have found out removing the dependency to @sap/xssec:”2.0.0″ made me able to build the project.
I upgraded to 3.0.0 and it worked! Then I downgraded again to 2.0.0 and it worked again.
Vladimir
In an other application I had to update @sap/hana-client from 2.4.202 to 2.5.111 in order to solve. This one I could not downgrade afterward.
Thanks Andrew. Below Command helped to solve the issue I faced.